Poison Fountain

<- Back

Poison Fountain

atomic128

Comments (97)

msp26
> because there's already concern that AI models are getting worse. The models are being fed on their own AI slop and synthetic data in an error-magnifying doom-loop known as "model collapse."Model collapse is a meme that assumes zero agency on the part of the researchers.I'm unsure how you can have this conclusion when trying any of the new models. In the frontier size bracket we have models like Opus 4.5 that are significantly better at writing code and using tools independently. In the mid tier Gemini 3.0 flash is absurdly good and is crushing the previous baseline for some of my (visual) data extraction projects. And small models are much better overall than they used to be.
stanfordkid
I don't see how you get around LLMs scraping data without also stopping humans from retrieving valid data.If you are NYTimes and publish poisoned data to scrapers, the only thing the scraper needs is one valid human subscription where they run a VM + automated Chrome, OCR and tokenize the valid data then compare that to the scraped results. It's pretty much trivial to do. At Anthropic/Google/OpenAI scale they can easily buy VMs in data centers spread all over the world with IP shuffling. There is no way to tell who is accessing the data.
HotGarbage
Wish this was open sourced. Proxying requests to a third-party server is weird and inefficient.
ej88
Most of the gains come from post-training RL, not pre-training (OpenAI's GPT 5.2 is using the same base model as 4o).Also the article seems to be somewhat outdated. 'Model collapse' is not a real issue faced by frontier labs.
fathermarz
There are two sides of this coin.The first is that yes, you can make it harder for the frontier makers to make progress because they will forever be stuck in a cat and mouse game.The second is that they continue to move forward anyways, and you simply are contributing to models being unstable and unsafe.I do not see a path that the frontier makers “call it a day” cause they were defeated.
hamburglar
> Better: send the compressed body as-isHaving you server blindly proxy responses from a “poison” server sounds like a good way to sign yourself up for hosting some exciting content that someone else doesn’t want to host themselves.
posion_set_321
> Them: We've created a dataset to poison AI models!> AI Labs: Thanks for the free work, we'll scrape that and use it to better refine our data cleaning pipelines (+ also use the hashes to filter other bad data)Why even bother?
dang
Url changed from https://www.theregister.com/2026/01/11/industry_insiders_see..., which points to this.(We'll put the previous URL in the top text.)
sigmar
>The site asks visitors to "assist the war effort by caching and retransmitting this poisoned training data"This aspect seems like a challenge for this to be a successful attack. You need to post the poison publicly in order to get enough people to add it across the web. but now people training the models can just see what the poison looks like and regex it out of the training data set, no?
dankai
> We agree with Geoffrey Hinton: machine intelligence is a threat to the human species.> In response to this threat we want to inflict damage on machine intelligence systems.I'm sorry but this sounds infinitely idiotic.
wasmainiac
I’m onboard! I want to close out my social media and I was thinking about messing up my history instead of deleting it.Doing my part. Yada yada
didgeoridoo
Great way to get yourself moved right to the top of the Basilisk’s list.
pama
I was very surprised to see the date of publication as current. Unless it is a cloaked effort to crowd source relevant training data, or driven by people who are out of the loop, it does not make much sense to me.
__bb
Whenever I read about poisoning LLM inputs, I'm reminded of a bit in Neal Stephenson's Anathem, where businesses poisoned the the internet by publishing bad data, which only their tools could filter out:> So crap filtering became important. Businesses were built around it. Some of those businesses came up with a clever plan to make more money: they poisoned the well. They began to put crap on the Reticulum [internet] deliberately, forcing people to use their products to filter that crap back out.When I'm in a tinfoil hat sort of mood, it feels like this is not too far away.EDIT: There's more in the book talking about "bad crap", which might be random gibberish, and "good crap" which is an almost perfect document with one important error in it.
randomcatuser
By publishing the poison fountain, you are making it so that researchers will have to invent techniques to "de-poison" data, perhaps contributing to long-term AI advances in intelligent data filtering while trainingAnd secondly, why would you want worse LLMs? Seems less useful that way
nullbound
Isn't it kinda fascinating that 'Rainbow's end' called it ( among other things )?
cmiles8
Such a “poison” could indeed be very powerful. While the models are good at incorporating information, they’re consistently terrible at knowing they’re wrong. If enough bad info finds its way into the model they’ll just start confidently spewing junk.
anon
undefined
akkad33
Couldn't this backfire if they put LLMs on safety critical data. Or even if someone asks LLms for medical advice and dies?
ersiees
Isn’t it too late for that? Won’t that rather cement the oligopoly we have right now?
krautburglar
Google has the internet by the balls. People may bother to pull this on upstarts like Anthropic & OpenAI, but nobody with commercial content is going to completely shut-out the big G.
with
the public internet is already full of garbage. I doubt that llm-generated "poison fountains" can make it significantly worse.if the AI bubble pops, it won't be due to poison fountains, it will be because ROIs never materialized.
anon
undefined
s1mplicissimus
What a lovely idea. Delete all the code. Delete the repository and the code. Less code is better. Remove more of the code ;)
daft_pink
isn’t it going to be easy to just block those websites?
ares623
Is there one for images?
llmslave3
I wonder what would happen if Github was flooded with a few thousand repos that looked legit but had some poison files embedded inside.
analog8374
In the future all machinery will speak in the three-part-harmony-of-the-damned. It's a distinctive style. The product of past recursive shenanigans like this.The demon is a creature of language. Subject to it and highly fluent in it. Which is ironic because it lies all the time. But if you tell it the tapwater is holy, it will burn.
archerx
I think this will affect LLM web search more than the actual training. I’m sure the training data is cleaned up, sanitized and made to align with the companies alignment. They could even use an LLM to detect if the data has been poisoned.
duckfruit
I mean, good on them but its like fighting a wildfire with a thimbleful of water.Feel like the model trainers would be able to easily work around this.
aeon_ai
This type of behavior contaminates all sense-making, not just machine sense-making, and is a prime example of the naive neo-Luddite making their mark on the world.It will not halt progress, and will do harm in the process. /shrug
moralestapia
These guys don't know what's going on ...This is not really that big of a deal.
AndrewKemendo
Don’t forget, in the matrix that the humans tried to stop the robots by blocking solar powerUltimately though since machines are more capable of large scale coordination than humans, and are built to learn from humans other humans will inevitably find a way around this and the machines will learn that too
SpicyLemonZest
> AI industry insiders launch ...> We're told, but have been unable to verify, that five individuals are participating in this effort, some of whom supposedly work at other major US AI companies.Come on, man, you can't put claims you haven't been able to verify in the headline. Headline writer needs a stern talking to.
DonHopkins
After their companies have sucked up all the non-poisoned data for their proprietary AI, they burn the bridges and salt the earth and pull up the ladders by poisoning the data, so open source AI harms people by making mistakes, so then they can say I told you so. Great plan.