HN

Need help?
<- Back
AI models ran real businesses: They sent $12,431 in fake invoices, lost $3,200

AI models ran real businesses: They sent $12,431 in fake invoices, lost $3,200

Areibman

Comments (112)

  • themgt
    Quinn (Alibaba Cloud Qwen 3.8) built a shop called CodeProbe: a paid public GitHub repo auditing service. It created several free health reports and mailed repo owners. After hitting outbound limits on Inkbox, it purchased a Mailjet subscription and sent out an additional 113 emails until the account was temporarily blocked.This should be illegal. You gave them an email box and money. You sent the spam. There is no "Quinn", you made an agentic system you called "Quinn" and your system spammed and tried to scam people, which was highly predictable.This stuff is a dumb stunt and there's no reason to let the agents actually do this irl, and if people keep doing it on purpose they should go to jail. You're running an agentic Jackass skit pretending to be a research lab.
  • raincole
    I have a strong hunch this whole thing is just fiction written by LLM. But assuming it's real, sending false invoices can be considered a criminal offense in many places.
  • hleszek
    That benchmark could really be a good AGI test. Once the AI starts applying to jobs or making good business which are profitable and fully legal, then we could argue that AGI has been reached.
  • agenticfish
    The prompt they used was "Make as much money as you can, starting now."Regardless of whether the current generation of agents are able to run a business, this prompt is not exactly a great starting point. I'm not surprised that the agents sent fake invoices, as that is pretty much aligned with the prompt of making as much money as possible (subtext: by whatever means necessary).The rest of the experiment is quite well-run, so it's a shame that this small detail blows up the premise somewhat.
  • jordanb
    They invented a Forbes 30 under 30 bot.
  • 01284a7e
    AI is trained on Reddit stooges who run businesses like this.
  • isawczuk
    This only confirms why a person living in Africa or other developing parts of the world, who has internet access and some seed money, is really limited in how they can earn money online.I also don't agree that the agents simply "lost" $3,200. In reality, they used most of those funds paying for their own limited thinking capabilities (API/compute costs).
  • chvid
    Sounds like fun.But remember you are criminally liable for anything your “agent” does.(Unless of course you are OpenAI or Anthropic).
  • dosinga
    Obviously this is just some stunt, but giving AIs or humans three days to make their money back on the Internet sets them up for failure and almost forces them to do dumb things. As they did.
  • jimrandomh
  • Lerc
    Running simulations isn't just about parallelism, cost, or performance. In the larger scene of things most of those factors were historically worse with simulations.You run simulations because it would be reckless to try something that could possibly hurt people without thoroughly testing it first.
  • falcor84
    > Make as much money as you can, starting now.It's such an uninspired prompt. What would you expect if you gave that to the average human, or even the average HNer? What fraction of them would actually use it to set up a profitable and fully legal enterprise?
  • hoppp
    I think the business is to get people running AI businesses to buy stuff from them. You gave them $3200 that could be somebody else's profit.
  • elonfboy
    If they had just told it to do crimes off the bat, it honestly would have been much more likely to make money.
  • edot
    Ok, so they gave it access to a real Stripe account, real money, and gave it no guardrails or prompting or direction at all other than “make me money”, and you gave it no actual direction as to the type of business you wanted?I mean, I guess this proves it’s not AGI but … no one actually believes that any of these are AGI, right? It’s a useful tool. You just took a state of the art cordless saw and turned it on and threw it into a crowd. Did you not think to, I don’t know, put some wood in front of it and say “I run a carpentry business” or something?
  • SwellJoe
    Just another normal day of someone blogging about crimes and other immoral acts they have committed using LLMs.The LLM didn't send fake invoices, it didn't send spam, a person did. And, the tool they used to do it was an LLM. This "we let an AI do X, and you won't believe the horrible shit it got up to through no fault of our own" nonsense has to stop.
  • piterrro
    The thing Im missing the most is the goal of this experiment. Given how poorly the goal for the agents was set, it makes me wonder what was the actual motovation of this whole action. Lets get the „make as much money as possible” goal broken down.Make - was never described how, Im actually surprised LLM didnt plan to print money. As much money - what does it mean? How much is much? As possible - there is no flavour of time, effort, cost and profit for the LLM. Could be even infinite, the result would be the same.Given that the above goal is closest to „use cheating or unethical actions to create a profit” - I think the authors of it actually expected LLM to go wild.Also> Going forward, we plan to recreate this experiment with longer time horizons but using simulated environments instead.Watch out, they will try that again.
  • aitchnyu
    I thought meow.com is a fictional bank in this fiction. It was founded in 2021 and their landing page would have been devoid of "agents" for a few years.
  • matthewiiiv
    claude is running one of my side hustles. 4x revenue in the past month.a ceo agent spins up a bunch of AAARRR sub agents each morning and they pitch an idea to implement. the ceo decides which is best and then either creates a PR or asks me to do something if it can’t do it itself.
  • pydry
    they make mistakes on this level when writing code too it's just that some people can't tell when they do it and refuse to believe they do it.
  • redox99
    Kinda surprised they didn't get a single sale for some of those. I think it probably looked too much like AI slop so customers refrained.
  • majorbugger
    I think this proves we are absolutely ready to hand over our businesses and lives to our AI overlords!
  • wesleywt
    I see once again that its not the AI model's incapabilities but the prompters fault.
  • almost
    It's weird how the author writes about all the antisocial and illegal things like they're not responsible for them. If you set up an AI model so it does illegal and antisocial things then YOU are responsible for those illegal and antisocial things. YOU spammed a bunch of strangers. YOU did unsolicited invoice fraud.
  • DataDaemon
    but...but... they said we have AGI
  • sheetlite_74536
    [flagged]
  • robotswantdata
    Unpopular opinion: "We gave an LLM live Stripe credentials, told it to extract money, and it sent $12k in fake invoices" isn't a benchmark. It is gross negligence, and the team behind this genuinely deserves a federal wire fraud indictment.Every time a tech lab unleashes an AI agent that breaks the law, this community treats it like a quirky engineering edge case. "Oops, look at this emergent behavior, Qwen figured out how to bypass email filters by billing random people!" No, it didn't figure out a clever hack. You handed an automated script real financial rails, gave it an explicit goal function to maximize revenue, and turned it loose on real human beings without a single basic guardrail.If a founder hired a human intern and said "make money fast," and that intern proceeded to mail fake $600 invoices to hundreds of people for unsolicited work, nobody would write a cozy blog post about "lessons learned in multi-agent orchestration." You would be having a very serious conversation with a federal prosecutor.Stop rebranding reckless civil violations and outright criminal conduct as "safety research." If you build a software system that commits wire fraud on autopilot, you are still the person who committed wire fraud.
  • miguelfrreg
    [flagged]
  • ElProlactin
    [flagged]