<- Back
Comments (205)
- maxgeeOur account was recently banned from OpenAI for a CSAM violation. We had an autonomous agent using an image classifier switch to OpenAI (instead of a local model it should have been using) which was classifying user-uploaded content. After classifying over 100,000 images, OpenAI banned our account.We asked for further information so we could report the uploader to the NCMEC and remove the content from our systems. They refused to give us any information about this incident, denied our appeal, and have so far, illegally refused our privacy requests for data about our account.OpenAI does not care about fighting child sexual abuse, this is completely performative.EDIT: These image uploads are up to 20+ years old, and are being served through Cloudflare with their CSAM scanning product enabled. We've received no notification of any problems from Cloudflare. Something on OpenAI's system thinks that an image we sent to them for classification is CSAM, and they aren't willing to elaborate or communicate about it at all. For reference, our website is https://ytmnd.com
- gwdSo I've been working on having Claude using GPT images to illustrate various fairy tales. At some point there was a Russian one of a girl left in the forest, but in the first cut she looked peaceful and happy. I told Claude I didn't think that was realistic, and it said that it had already asked GPT images to show the girl "scared", and it refused 12/12 times. It also refused to have Frost (who in the fairy tale ends up giving her a coat and a chest of gold) standing over her; he had to be at her level, an arm's length away. (Showing the girl as "brave" and then "nervous" had better results.)Gave me quite a funny feeling, tripping innocently over the empty shape that they've had to carve out. They wouldn't have spent the effort to do that if it wasn't pretty bad.
- nolist_policyRelatedly, Openrouter now does CSAM scanning and retention on so called "Zero Data Retention" endpoints as well.https://openrouter.ai/blog/announcements/csam-screening/
- DonaldIt is kind of wild reading these criminal affidavits around these National Center for Missing and Exploited Children integrations. The criminal uploads an image to their email, photo storage service, or gen AI service, it's detected as CSAM, automatically routes an alert to NCMEC, NCMEC generates a tip to the user's local law enforcement jurisdiction, and they're arrested within a short time period, like one or two weeks.
- eutropiaThis is a positive development. People working for the NCMEC have had to deal with an insane increase in CSAM content (like upwards of 1,000% increase in 2025 alone) due to diffusion models dramatically multiplying the supply of CSAM.When there's an active and urgent case to try and locate and rescue children in these situations, the volume of AI-generated content makes it that much harder to perform forensic investigations on background materials (often how they locate children being abused is by prosaic items like electrical outlets, household goods, etc. in the background of abuse photos and videos) because now they have to rule out whether an image is even real -- a task that often requires close attention and scrutiny to details of traumatizing and disturbing content (sometimes all the more traumatizing due to the 'creative freedom' of AI-generated material)A model capable of shielding investigators from additional trauma is an excellent thing.
- itakeMy colleagues in Singapore trained pornography detection model.Pornography is illegal to download or save in Singapore… I wonder how they were able to train a model without the data.
- AmazingTurtlereminds me of the case where a dude made pics of his kid to show the doctor and his google account got permanently banned for CSAM (google photos involved)
- callcCurious what people here think as digital technology makes it easier to spot indicators that someone may commit a crime.I’m thinking about how it used to be extremely impractical to track most of the population, the the enforcement comes after the crime, and there is no pressure to have detected it before it happened and stop it.How should society/law change in response? Is it purely a give and take between surveillance/right-to-not-be-watch and trying to prevent all bad things from happened before they happen?
- RIMRThis kind of thing is always an excuse for creating surveillance tools that scan your files and report you to the government. It starts with CSAM because they know you don't want to argue against CSAM detection, but you absolutely should.I deal with a lot of user-generated content, and I have been using OpenAI as a tool to help find offending content. Now I see that if CSAM ever makes it into one of my systems, OpenAI is going to send the authorities after me if their model sees it too, so that's a dealbreaker for that use case, and it was a big one!It's also wild that they are trying to detect and report AI-generated CSAM. This does absolutely nothing to help protect actual children.
- stickfigureHow is this different from omni-moderation, which OpenAI already has?https://developers.openai.com/api/docs/models/omni-moderatio...
- anonundefined
- andai>It is well established that combatting online child predators requires coordination across many organizations.It requires all devices integrated into one single consciousness.
- avazhiNCMEC is cancer.Remember when they wanted Apple to scan everybody’s devices locally?I’m not sure what the solution is to deal with pedophiles but I’m sure it starts with parents monitoring their kids better and keeping young children from being isolated from real world social groups. I’m also sure it doesn’t involve using classifiers on a person’s own device to detect ‘possible’ child porn, which is where this is headed. Who can forget the father who had photos of his sick baby on Google Drive for his paediatrician to see and got reported to the police.If it really is all about protecting kids then maybe the NCMEC should be lobbying to pass laws banning kids from the internet. Think about how many harms you could reduce instantly (bye bye Instagram, Snapchat, etc).
- nrmitchiFrankly, and I don't mean this in a bad way about the problem, I do not think OpenAI actually gives a shit about the underlying problem. Especially as an OpenAI-hosted-service-only, this seems more like a data grabbing play. "Scan all your images through our CSAM detector" means "send all images to us, including the 99.99% that are not CSAM!"Don't get me wrong: these are very necessary tools for the modern internet to function, and I'm sure _drastically_ reduce the review burden on humans, but the skeptic in me does not believe that OpenAI is being entirely altruistic here.
- eastboundFinally Sam Altman found something good to do in terms of my security, I'm relieved. It fits perfectly well in a world where Saudi Arabia lead the UN Commission on the Status of Women (CSW).
- kittikittiThis is just public relations and no reasonable person would believe OpenAI will do anything to fight against online child sexual abuse. In fact, it's much more reasonable to assume they will continue to make the problem much worse.
- bearjawsIf only there was a huge data mine of child abusers sitting on some sort of email service they could go after. Maybe some sort of jeevacation inbox.Could be a great start for "rogue AI"Jokes aside, I'm glad they are giving tools to help fight this as currently your options are limited because every provider will simply return a content warning if you try to use an LLM as a moderator.
- anonundefined
- rfgplk[flagged]
- haickernews[flagged]