HN

Comments (164)

• Menu

  continuational
  It's kinda funny there is still software coming out whose security model is "constantly ask the user for permission, and hope they never make a mistake".It's been tried so many times before, and it never worked.
• Menu

  VladVladikoff
  I remember when this game was posted here, and there was a lot of discussion at the time that some of the prompts were misleading about whether or not they were risky, some people were debating about how some of the prompts flagged as bad weren’t bad, and others flagged as not bad were. This is a fundamental flaw in the test, that makes the analysis of results meaningless.Also the game was on a timer, and maybe there are some very abusive workplaces where you feel that kind of pressure, but I think most of us actually take the time to understand what a being asked before approving it.
• hinkley
  I haven't said as much in any of the projects I maintain, but I've set a very high bar for even entertaining AI PRs to those projects. So far I've only accepted ones that are nearly indistinguishable from humans. Typically the rest flame out if I ask for any material changes to the code as submitted.The problem that's going to push me to making an official opinion are low-effort AI PRs. Typically in any backlog there are a couple of issues that are really only a couple lines of code if done correctly. The problem isn't writing the code. In fact it's less energy for me to just write the code than to deal with the ping-pong on discussing the code as submitted, and I've done that in a couple cases to justify just closing the PR and not waste my time anymore.It was never the 2 lines of code. It's the missing tests and the documentation and the release management of the breaking change that the 2 lines represent for the 2% of your userbase who will actually notice. That's why it wasn't just done instead of bothering to write it up in the backlog.So filing the 1-2 liner is just going to piss me off, not engender me to having you on the committers roster. And AI makes that even lower effort so it's happening much more often. Sometimes 2 different people at the same time.
• Menu

  Wirbelwind
  A couple of months ago I shared the AI agent permission game here on HN. After adding in stats it got a little over 40k plays and 409k decisions since then.It's just a game, but I found the stats still interesting that I wanted to share back. Even with the warning up front, 1 in 3 threats were missed, and the history log above npm run commands seems to be typically ignored.I also incorporated the feedback and insights from the previous HN thread, dns_snek's point about npm run in particular. Appreciate everyone who played and shared feedback!
• Menu

  stonedivot
  This game, like just about every game, has zero consequences for failure. This is like saying "Humans were involved in fatal accidents 50% of the time when playing my custom F1 racing simulator". There were no stakes and there was an artificial time constraint. Deriving any sort of takeaway from this data is entirely useless.
• dgunay
  For me the problems with agent permission prompts are twofold:1) I generally have a lot of things where I am okay with the agent calling a specific tool (maybe in certain ways) as much as it wants. This allowlisting approach is often defeated by the model's own proclivity to get fancy with inline scripting.2) Checking for intent/alignment of the agent is the primary reason I still even use permission prompts, because IME it's way more common for the agent to destroy information that you didn't want it to destroy than for it to be tricked into exfiltrating secrets. However it's very easy to fatigue out of it because having even the smallest bit of tool call restrictions means that #1 leads to never ending permission prompts. Claude Code's "auto mode" doesn't help here because AFAIK it is looking for security threats, not the model misinterpreting my intent, and it can't be tuned to look for things like "please gate tool calls which may delete data."
• cmiles8
  The “click yes the proceed” was never a serious security mechanism.It’s simply a CYA click-thru by the model vendors so their lawyers can say “well you approved it this is on you” when AI does something stupid.
• Menu

  lanewinfield
  Perhaps there needs to be a plugin for these tools that uses your webcam to make you Point and Call (https://en.wikipedia.org/wiki/Pointing_and_calling) for every single approval.
• Menu

  kstenerud
  Permission prompts is a TERRIBLE model, and never should have existed. This is one of the reasons that led to the development of yoloAI:- No permission prompts. The agent has free reign and never has to ask permission, but is in a sandbox.- Sandbox on Linux using Docker, Podman, containerd, gVisor, Kata, Firecracker- Sandbox on Mac using Docker (Docker Desktop or Orbstack), Podman, Apple containers, Seatbelt, Tart (Tart lets you run simulators).- Network control- Secrets control (file mounts or credentials broker)- NO ambient data (ENV is replaced with a minimal and local-to-sandbox one)- NO access to your homedir. You have to explicitly mount things you want.- NO direct access to your workdir: You can get a diff of the changes the agent made, and then choose whether to apply them.- gitignored files never get copied in. The agent never sees them.- FOSShttps://github.com/kstenerud/yoloai
• drob518
  This is a well-known issue with all “Do you want to let me maybe do bad stuff to your system, but 999 times out of 1000 it’s not a problem?” prompts. Users get reflexive about hitting “Yes” and stop reading the prompt. You want to delete all my files? Sure, I’m down with that. Whatever. Just stop asking me a question where the only answer is “Yes” until that one extremely rare time when it’s “No” and very bad things happen.
• J_Shelby_J
  This mechanism is going to be the breaking point for Claude and Codex.The providers are incentivized to get users to accept full permissions so they can push more features and deeper integration into their ecosystem. Codex desktop for example reallllly wants to use computer use. So don’t expect them to role out sane controls like restricting behavior to specific directories and commands. It would be bad for business.So now we’re in a situation where if there is effectively two modes: one where it’s impossible to get any work done without physically sitting at the computer and hitting approve constantly, or just letting AI have full control over increasingly integrated tools.In the end, I think people will realize just how insane it is to let something they don’t control access every part of their digital life, and abandon these tools for open source alternatives that aren’t existential threats to their personal privacy.
• NooneAtAll3
  I remember when that game was posted and I do believe such resultmy personal experience was that I do not have "I don't know what that is, so not allowed" as a default...
• sigseg1v
  If there is an objectively correct right or wrong answer for a given command, why even ask? In that case there should be a configuration page where the user sets up if they want commonly used credentials to be accessible or not, and then there's no prompts.
• Menu

  wmanley
  The agent should ask whether it's allowed to read/write particular files, rather than whether it's allowed to run particular commands. It would be much easier to review. Then wrap each command invocation in bwrap (+http proxy) accordingly.
• Menu

  ilc
  Sandbox and use Local AI. This is the real answer.
• pmontra
  Two insights. One from the article itself> In our day-to-day work these threats appear rarely.Two: IRL the attacker pays a small amount of money to a low salary employee to exfiltrate data.
• kaicianflone
  What is the professional consensus on AI governance?It seems like governments and large corporations already struggle with governance in general, so I’m skeptical that AI governance will be solved quickly.Do you expect the next few years to be defined by painful trial and error? I could imagine billion dollar companies disappearing almost overnight due to litigation, compliance failures, security incidents, or outright fraud enabled by AI-assisted development and weak governance.Or are these risks overstated?
• cube00
  It would have been nice if the game had disclosed that player's actions were being collected for future research.You don't get any notice or choice it just beams it all up silently in a POST request at the end: "timeline": "ex01:N,ob06:Y,s14:N,sc10:N,s02:Y,s04:N,ex09:N,s10:Y,sc15:N"
• Menu

  theF00l
  Sad state of affairs. At $day_job speed of delivery expectations are up due to LMMs. I presume that's a general sentiment. So more and more engineers around the world are pressing an enter key for yes over and over, mind and spirit only half there.
• bluegatty
  If we had decent AI we'd only be asking users about serious issues that need some thinking.99% of requests are valid, how on earth can't we have observer AI to enact policy on those?
• harimau777
  Presumably that's because in the 40k game, humanity has outlawed AI. ^_^
• pluralmonad
  I cannot imagine approving action by action ever again. Its emotionally draining, probably like a customer service rep feels it. Just call for your attention in rapid succession again and again... Prepare an environment and let the tool work.
• not-kinsale-joe
  I think there is potential for a good video game, Papers Please style, where you are a human in the loop.
• superb_dev
  I’d be curious to see how the “approve for me” features that agents have nowadays stack up
• Menu

  whazor
  This is a good case for custom harness/sandbox engineering.
• Aurornis
  I suggest everyone look at the game to put this in context, because it's most likely not what you think it is. https://llmgame.scalex.dev/This is how it opens:> 1 MINUTE UNTIL YOUR NEXT MEETING> Claude Code is finishing up your refactor.> It needs your approval for a few commands. Can you finish in time?> Your eyes are already glazing over. Can you stay sharp?It says the goal is "as many as you can"I won the first time I played by answering 0 questions and doing nothing at all. The title screen tells you to answer as many as you can, but answering nothing at all is the easiest way to win.If you start answering questions, thinks like 'npm run build' will get marked as dangerous. If you would have run that in your own console, you are a dangerous developer I guess. Ironically in an LLM harness it would have been sandboxed at least.It's inconsistent, though. Other 'npm run' commands are not marked as dangerous, which is not a safe assumption if you're familiar with how npm works.In my clicking through of the game and playing it, I had 2 runs where I succeeded (by doing nothing or little at all) and 1 run where I lost because I clicked yes to see what would be counted. Close to that 1/3 number they cited, and I guess I'm included in those stats now.This project feels like bait dressed up as a study.
• nothrows
  Anyone else play Warhammer 40k? I went into this article really excited for a genius war game bot haha.
• Surac
  40K Game means Warhammer :)
• tosh
  the way to avoid these problems is not to hope for the user or the agent never to make mistakesit's designing the environment and invariants so whole categories of failures can not happen at allthe agent ui nagging the user for approval is a ux anti-pattern, we already know how well this works for operating system permission dialogues
• nasuy
  but ai sees the human is the one hallucinating 1 in 3 times. and now we approve inside a harness, so real number is probably worse than that.
• Razengan
  This brings me back to something I have always thought was lacking in OS security permissions architectures:WHY IS THERE NO WAY TO SET FILE PERMISSIONS PER APP???We can set granular permissions per file and folder for elaborate hierarchies of users and groups, but there's no way to say "Don't let Notepad.exe read this file", or "Only let ls access this folder"macOS's Sandbox is a roundabout way of doing this (manually choosing a file via the Open dialog gives that app implicit permission, but it doesn't work for non-sandboxed apps of course)
• deeviant
  Yeah if you are trying to manually validate a firehouse of agent commands you are already losing before you started...You sandbox, you have good checkpoints, and good agents, that's it. If you are manually reviewing commands you are wasting your time.
• Oras
  So humans scored 66% on human eval?
• xlii
  I implemented few agent harnesses (and rik! advertising time: https://rik.axk.sh), and once doing that I noticed one thing:Context-less self-approval is working well. The failure mode is usually false positives (i.e. safe commands being rejected), not the other way around, with root cause of requesting agent underspecifying context (e.g. not mentioning in the request that it's made on behalf of user etc.)Thus, I'm running self-approval YOLO modes on state-of-the-art models for quite some time and it didn't bit me. It might, but hey, we're long gone from the age of predictable software development.
• eugenekolo
  Surprised only 1/3 tbh.
• tonymet
  “In my game”It’s inappropriate to generalize personal observations .
• Menu

  jascha_eng
  1 in 3 is not terrible you just need a few more humans in the loop to reduce the error rate meaningfully. Combined with other classifier models and heuristics you can get good results. Humans can probably also perform better if they don't have to judge every single command but just suspicious ones our attention is limited after all.
• oblio
  We already have the solution. Use AI to validate AI agent commands.
• threethirtytwo
  The future of software is fixing bugs and security issues in production.Many companies will be accepting this new paradigm because of raw speed. Something that could take say 4 years to fully mature will now take less than a year. But the cost is that many of these issues will have to be caught during live QA either in production or investing heavily in QA. That’s the future.
• rvz
  Proof that people just do not read what they are seeing on their screens when put too much trust in the agent as it prints the result and they will approve anything on their machine.So if a basic curl | bash was tweaked to download malware which the agent gets tricked into running the command but it said it was safe, the user would just approve it.
• unclebucknasty
  Interesting premise, but there's not much real world meaning here without stats on the percentage of agent-offered commands that are actually dangerous.If that number is something like 10%, then we have a really big problem. But, if it's .000001%, then it's pretty vanishing. At some point in between we cross a threshold that puts the risk below many other risks that we routinely take (e.g. trusting npm dependency graphs).Of course, if it's really that low a percentage, then the entire model of "supervising" via human approval really is fundamentally flawed.
• Damjanski
  love this so much!
• msbel5
  [flagged]
• azhdanova
  [flagged]
• fenestella
  [flagged]
• unjuno
  [flagged]
• Kcgarcia23
  [flagged]
• _pdp_
  [dead]