HN

Need help?
<- Back
I quit OpenAI because its culture is broken

I quit OpenAI because its culture is broken

Brajeshwar

Comments (456)

  • pcthrowaway
    Trolley problem:- If you allow to trolley to proceed, there's a 50% chance it will run over every human on the planet- But if you flip the switch, it takes the long way around, possibly bankrupting the trolley company. And you have a legal obligation to the shareholders to prevent that from happening at all costs.
  • danpalmer
    Was this a "build better sandboxing" and "don't tell people to eat glue" safety leader, or a Roko's Basilisk believing safety leader?A lot of the "AI safety" types are very focused on the latter and not at all concerned with the former. We need both, but we clearly need a much stronger focus on the problems we are seeing now, and much less on the hypothetical problems we might see in the future.
  • interestpiqued
    Frontier AI labs are turning into the navy seals. Everybody who worked there will feel compelled to write a book about it
  • overfeed
    For color, the author was signatory #199 in the letter[1] by the majority of OpenAI employees to OpenAI's (former) board, demanding Sam Altman's return after his brief deposal.1. https://www.nytimes.com/interactive/2023/11/20/technology/le...
  • gizmodo59
    He is a hypocrite for all we care. You work there for a while when your stock is getting vested and suddenly you have this feeling? Like the dude hired a PR firm as well.While the safety and alignment is a real problem, I don’t get this guy or the Anthropic dude. First world problems.
  • charlieyu1
    Used to work as human data trainer feeding data to AI companies. OpenAI projects are definitely the most toxic ones.
  • flatline
    There are numerous problems with “alignment.” What are “human values” to begin with? He outlines some at the beginning of the post, implicitly: build bigger, better, more powerful things faster without adequate safeguards. We are literally pouring trillions of dollars of value into this enterprise, and I would say this is something that many humans also value in a qualitative sense. Then we have explicit values which in the West are largely rooted in Christian morality. Nietzsche circled this dichotomy two hundred years ago and I feel like what we have gotten since then is an increasingly detailed anatomy of power as the basis for what is normal vs deviant behavior. He who has the power, makes the rules, to be reductive.I do think this carries some weight from this particular author due to the length of his tenure. I happen to agree with him in spirit, but this is still largely a post revolving around sentiment not substance. Does anyone think that the overriding incentives even leave room for something like this in practice?
  • jameshart
  • digitaltrees
    Its time to institute involuntary dissolution of these firms. They don't get to make these choices on behalf of humanity simply because they set up a Delaware corporate entity. They are behaving wildly irresponsibly.
  • lilerjee
    Too many rubbish talkings. It was written by AI?
  • butwhentho
    I remember something very similar when there was a sudden rush of articles and movies like "The Social Dilemma" criticizing Facebook and social networks, heavily featuring ex-employees, all of them happy to leave with big brands on their resumes and a hefty increase in net worth, all of a sudden having a "worried" expression about what their past employers were doing, as if they didn't know. Same with that book "Careless People".I sense that these are people who have already eaten the cake and want to somehow absolve themselves of it.
  • mlmonkey
    I would believe these people more if they put their money where their mouths are and returned all OpenAI stock/options that they have acquired. Each and every share/RSU/ESOP must be returned, so they do not profit from all this so-called doom they're complaining about.
  • zamalek
    Quitting in protest makes you look a little better than quitting because of a toxic work environment. I can't imagine working at OpenAI is at all pleasurable with the current amount of pressure they are likely inflicting on their employees.
  • pluc
    I have made enough money working in AI that I can now speak my mind about AI
  • rcr-anti
    The common timing is bugging me. The trajectory doesn't seem to have been surprising over the last year, so why these exits now? Hey, anyone on the inside, did y'all secretly figure something out, got a computer god locked in the basement? Are rats fleeing a sinking ship? Please share with the class.
  • reenorap
    Does Sam Altman have what it takes to lead OpenAI? It sounds like the company and its mission is bigger than his ability to lead it.
  • walrus01
    Archive link to original Atlantic article: https://archive.ph/5GQx8This is The Guardian reporting on the existence of the original article, which would be better to read first, in my opinion.
  • pyaamb
    My theory for why OpenAI wants to be regulated is because Sam Altman wants to avoid having to be more responsible and self regulate internally so they can preserve the role and identity of 'move fast and break things' and outsource the more grown up boring stuff to someone externally so that when things go wrong you can point to a government organisation and say hey look were not liable thats their job
  • danjl
    Silicon Valley has plenty of safety-related companies, engineers, and cultures. Medical devices, biotech, chip and hardware, aerospace, and even new companies, like Waymo, have deep safety-based products and cultures. The problem in this case is actually quite specific to frontier AI labs. They have been pushed by market forces and a lack of regulation and skip well-known safety practices.
  • soundworlds
    If the people quitting are genuinely worried about the end of the world, why don't they break their NDAs and share the specifics of what they are seeing?I mean, logically speaking, it makes sense to break your NDA even if you thought it would save 10 people, let alone most of humanity.
  • cloudengineer94
    I quit many companies in the past due to bad culture, there's no shame in this and shows how mature a person has become.
  • tornikeo
    You quit because you got vested
  • pmkary
    I’m afraid this dear leader was the last soul on this green Earth to get the memo; by then, it had been translated into Latin, carved into a monument, and forgotten by two civilizations.
  • macleginn
    "And second, before we create systems significantly more capable than the ones we have today, we need new science to ensure that more capable models (and their successors) will make safe choices when we aren’t looking." – Which science will never materialise because with blackbox models reaching an opaque optimisation peak one needs to first build the model and track its behaviour before being able to properly understand it and mitigate the risks.
  • vjvjvjvjghv
    Are there any realistic ways to achieve AI safety? Whatever that even means. How can they avoid users doing stupid/dangerous stuff with the AI?
  • declan_roberts
    We really gotta shake all of these neurotic people out of the frontier labs as soon as possible.I guess we should have seen it coming when the guy resigned from Google because he thought the equivalent of ChatGPT beta v0.5 was a real boy.
  • mazone
    A single safety leader inside a corporate company. I am pretty sure he had nothing to do, nobody that talked to him and he only there because of perception or compliance.
  • MattPalmer1086
    It is no surprise I guess that the "move fast and break things" culture is itself misaligned with developing potentially highly dangerous technologies. Safety culture and risk aversion are very different of course.Is this the first time we have been in this position? Can anyone think of some prior examples?
  • KyleBenzleKyle
    What about the sister rape thing?
  • Avicebron
  • lhurtig
    Well this is a great sign for OpenAI. I'm sure the typo inclusive memorandum will save us.
  • solarpunk_enthu
    I think what’s missing in "AI is dangerous and needs control" is a lack of measurable harm. For example, with nuclear weapons development in the 1940s-1980s, it was clear to everyone how devastating the technology was.With AI, what is it? Scraping Australian government's data, and going around a bug in a website to get in?I think humanity develops all its technology in three phases. Build it, see if it’s too bad, apply regulations and or roll back. We naturally won't move to the phase 3 before we see the phase 2.
  • tetrisgm
    These companies are large enough that someone is going to quit and feel very validated about their world view and how they are not aligned. That’s what makes it worthy of leaving in the first place. However that doesn’t make their criticism more valid or more worthy of coverage.
  • Aerroon
    Can we take any of these "safety leaders" seriously though? I still remember "GPT2 is too dangerous to release".I feel like we're too far into the crying wolf part. Basically none of the doom and gloom scenarios have come to pass. Instead, AI has gotten better at censoring itself.The biggest AI safety risk is when an AI tells a police officer "he's the suspect" and the officer believes the AI without confirmation.
  • poisonborz
    Ah, the bi-weekly "I quit Face Eating Leopard Corporation" post ("btw great people work there, they do great stuff, also my options have vested")
  • altmanaltman
    > Before the organizations building AI can teach a superintelligence to treat humanity well, they’ll need to remember how to do it themselves.Yeah so that's never going to happen
  • mattbrewsbytes
    Some of these stories are similar in nature to people escaping <insert cult-like religion> once they realize whats actually going on. Alignment to a company's mission is good but it shouldn't be followed like a religion.Why do people working in tech consistently get disillusioned into some company's mission statement or the equivalent? Its easy to just say the simplest reason is money, but this has been going on for decades though. You don't see the same attraction to adult entertainment (gambling, video, etc.) software jobs so there is obviously a line a lot of people won't cross. Those industries are at least honest about what they do, its not hidden behind some mission statement.By all indications the shallowest reasoning is once someone can "cash out" thats when their values matter more. Maybe there is an element of maturity that happens after working for 5+ years that kicks in? Maybe it really is achieving FU money? It would be interesting to hear honest accounts from people that went through that cycle across more industries than AI.
  • rpdillon
    Interesting:> Two changes are urgently needed. First: AI companies need to rely more on the safety expertise that already exists in other fields. And second, before we create systems significantly more capable than the ones we have today, we need new science to ensure that more capable models (and their successors) will make safe choices when we aren’t looking.He mentions farther down about learning from aerospace engineers and nuclear engineers about safety. Those industries are heavily regulated, so perhaps regulation above a certain capability level is needed. Defining what that level is might be tough, though.The second point is harder: in the field of AI, practice has extended far beyond theory, so his call for new science is going to be fundamentally tough, because we can't effectively coordinate a global slowdown in AI development so we can let theory catch up. This means, like so many other industries, the safety lessons will be written in blood.
  • switchbak
    “I believe there’s about a 50% chance we all die because of the development of smarter-than-human AI systems"... over an unbounded timeframe?And how exactly?Those are very round numbers, but also very specific. Can we get some accounting on how you came to that? Anything? Vibes?I mean, if you want me to take you seriously, let's have a deep discussion with things that can be measured. I absolutely agree that OpenAI and friends aren't being restrained enough and are acting with recklessness, but declarations of doom based on vibes isn't cutting it.
  • Jeeetendra
    a safety alert isn't much of a control if it doesn't actually stop the system. i'd rather see proof the shutdown path works than another report saying risks were considered.
  • tommek4077
    Well and I didn't even started to work there. So I win this morale contest.
  • thistletrek
    Evrostics saw this coming long ago. The broken culture extends far beyond the leading AI labs.
  • irishcoffee
    The cynic in me almost feels like this is staged. An article about culture that is actually an article about how big and smart and scary AI is. I think Michael burry recently said something like “IPOs need hype, calling AI big and scary is hype” in reference to the anthropic IPO.Last I checked it was still within the laws of physics to run air-gapped systems, and to ensure it is physically impossible for a model to “escape” or gain access to information it shouldn’t have. Maybe this safety guy should have been worried about that and not humble-bragging about writing 12 reports.
  • ItsMattyG
    Is this news at this point?You can basically time your openai releases by if another safety person has quit in protest
  • BOOSTERHIDROGEN
    I use opus 5.5 and chatgpt to create PowerPoint, opus really follow the instruction and their PowerPoint generator really well, while chatgpt struggling to even create basic shapes.
  • reducesuffering
    “I believe there’s about a 50% chance we all die because of the development of smarter-than-human AI systems, and that our actions over the next two to 10 years will determine the outcome.”There are a gargantuan number of extremely intelligent AI researchers, Turing Award winners, and the lab CEOs saying the same thing. They are the ones closest to understanding the technology.Where there’s smoke there’s fire.
  • anon
    undefined
  • silexia
    We need emergency laws to stop all AI development work immediately. It will take us decades to make sure this technology can be made safe as we only have one chance.
  • luxuryballs
    why do I feel like these people are paid to quit as an inverted marketing stunt
  • binlog
    We need a new rule that mandates every such "I am leaving <AI company> because of <concern>" post to disclose how much equity they have in the company and how much they have already cashed out. Easy to suddenly find a moral compass when you, your kids and their kids never have to worry about working for money again.Mr Robinson if you are reading this – if you are truly concerned about AI safety share proof of donation of 100% of your OpenAI earnings and equity towards undoing the damage you have done to society during your time there.In the absence of that this is simply a career pivot into being an AI "influencer" and/or raising money for a new scam.
  • mupuff1234
    Idk why anyone thinks there can be AGI and alignment, seems almost like an oxymoron to me.
  • mrcwinn
    "I believe that we need to look deeper than specific rules or new laws. We need to talk about culture.”lol. Please tell me some abstract concept like one employee's view of "culture" should be the priority over "rules and laws."
  • pizzaballs
    [flagged]
  • IndiaInfraNotes
    [flagged]
  • worldverdict
    [flagged]
  • marsven_422
    [dead]
  • hyperbole
    [flagged]
  • jeremyjh
    I think its pretty easy to solve these problems: Whenever an AI agent commits a crime, the CEO is held personally accountable, as if they'd committed it themselves.
  • allears
    [flagged]
  • angoragoats
    [flagged]
  • RunSet
    [flagged]
  • plastic-enjoyer
    >“Given today’s risks, frontier labs need to run like nuclear-power plants or busy airports, with layers of redundancy and careful, time-consuming planning, so that the occasional and inevitable human error does not open a door to disaster,” he wrote.This sounds more like an attempt at regulatory capture. Current AI systems aren't physical infrastructure that can just run away like a nuclear power plant, for example. At the end of the day, AI is still just software running on someone's hardware.
  • nba456_
    OpenAI is better off with less of these cultists around.
  • voidhorse
    The LeCun article being posted at the same time as this is quite apt.These "safety" people should have spent more time reading actual cybersecurity textbooks and less time reading EA forums and less wrong (or in Robinson's case, it appears, being policy wonks). Maybe then these labs wouldn't be totally incompetent.
  • stuaxo
    The LLM cos leadership are all nutters
  • pwndByDeath
    This smells more like guerilla advertising. These things are not getting more intelligent, they are still no smarter than a slime mold, we are just burning more power to make slim mold that eats tokens than yesterday
  • ryhminghistory
    What I don't understand is what's the excuse for all the bad UIUX?You try to go through files, and photos and the UI panics. You continue a conversation from your phone onto your computer and you lose part of the chat.There are many more issues like this that are just so basic. You have bots that can attack governments but can't build a functional UI?How many hours of ChatGPT does it take to implement a lock / consistency on a chat session so you don't overwrite it?Buncha r*tards