HN

Comments (73)

• Menu

  simonw
  Firing people for bad architectural decisions is generally a terrible idea - especially decisions that shipped and ran in production for several years.This article also doesn't make a convincing case for this being a huge mistake. Companies like Uber change their architectural decisions while they scale all the time. Provided it didn't kill the company stuff like this becomes part of the story of how they got to where they are.Related: the classic line commonly attributed to original IBM CEO Thomas John Watson Sr:“Recently, I was asked if I was going to fire an employee who made a mistake that cost the company $600,000. No, I replied, I just spent $600,000 training him. Why would I want somebody to hire his experience?”https://blog.4psa.com/quote-day-thomas-john-watson-sr-ibm/
• Menu

  Aurornis
  > A redesign that gets replaced 2 years later is a catastrophe.> Somebody Should Have Been Fired For ThisThis person is not a good resource. Uber was a very fast growing company, both in terms of their product and staff. Turnover in architecture happens. Calling this a catastrophe and click baiting about firing engineers over a rounding error in Uber’s overall finances is gross.I understand this person is trying to grow their Substack with these inflammatory claims but I hope HN readers aren’t falling for it. This person’s takes are bad and they’re doing it to try to get you to become a subscriber. This is hindsight engineering from someone who wasn’t there.
• paphillips
  This article does not make convincing arguments to match the strong criticism. Engineering decisions are difficult, especially in high-growth orgs, where one must balance many constraints and risks including opportunity cost. Handling payments is part of Uber's core product.The financial criticism ('napkin math') appears to estimate DynamoDB costs of USD $8 million for 2017 to 2020. Uber revenue for the same period is roughly USD $42.5 billion, thus this cost weighs in at about 0.02%, or 1/50th of one percent. This is a rounding error for a high growth company, and not something that warrants a witch-hunt and firing. It's easy to blow more than $2 million per year on software engineers in pursuit of an alternative high-scalability solution.I'm also not on board with the 'resume driven development' criticism as the explanation for solution churn. Perhaps that is actually what happened. I wasn't there and don't know, but if that is being asserted I expect to see evidence presented to support it.
• Menu

  websap
  > A redesign that gets replaced 2 years later is a catastrophe.People forget how quickly Uber scaled, and the user impact of not being able to track your trips could be catastrophic to retention. There's a class of tech-influencer who think they can dissect past decisions on a blog post without being in the room when the technical constraints were being laid out. This is Monday morning quaterbacking at it's most grotesque.
• Menu

  kshacker
  Can I do a mea culpa? This is more than 3 decades back. I was a junior programmer (2-3 years in industry) sent to a client site in europe. You can imagine the state of systems those days. I wrote (or rather updated) a fix which would updated the discount and tax rates on orders based on new terms. It would run every day to account for ... whatever. You pick the values from master file, update all the orders and move on.I wiped out VAT on all orders and for the next month the paper invoices were sent without VAT. So the invoice is $100, VAT is $20, the invoice should be $120, but they were sent as $100.100s of invoices every day would be my guess.Nobody noticed.For a month.Millions of dollars of revenue and IIRC millions of dollars of VAT.Until a customer complained to the CEO.We had a firefight to fix it, not just technical but legal and managerial. We can send a new invoice just for tax. We can redo the invoice. We can send a debit memo. What is the right decision? But what if customers does not pay? What about returns? How will we track returns? Of course we were doing the technical solutions and the client company was front-ending how to handle it business wise.And the managerial firefight - who did it, what are the safeguards in future? We had a company exec visit the client site to manage the issue.I was in the hot seat but I was protected by my managers from any fallout. Just do the work. Do not screw up again. (Test every row every column even if you did not change it)A month later the sales director at the client company got fired.The grapevine is that this was just the tipping point, but you never know. BTW these were paper invoices printed onsite and mailed out, but I do not know if someone had the job to scrutinize them.PS: True story, going by old memory, although such legends remain fresh in your mind, forever. Not sure it belongs here, but the mention of firing for a multi-million dollar mistake pulled this into cache memory.
• colinbartlett
  A single engineer should not get fired for an architectural decision that clearly had buy in from many people.
• 4fterd4rk
  Alleged expert hawking some kind of blog/newsletter thing doesn't know that no major company is ever going to fire an employee for an honest mistake, especially one that would have had multiple sign offs from around the organization.
• junon
  > Every rewrite was someone’s promotion project.At least when I worked at Uber, that wasn't really how it worked. The eng org was so big that it was nearly impossible to track all the projects people worked on, and you'd get micro-ecosystems of tools because of it.Some grew large, others stayed quite "local".
• tibbar
  $8M sounds like a lot, but (a) the cost of making a material financial mistake c an easily dwarf this, and (b) the cost of the engineers maintaining the system was likely about this expensive anyway. And infra is expensive when you're Uber. It all seems rather overblown to me.
• barbazoo
  > Nobody Got Fired for Uber's $8 Million Ledger Mistake?> Somebody Should Have Been Fired For This> And nobody got fired for this?Not once do they explain what firing someone would actually improve?!
• infecto
  I have a hard time following the thesis. I know Uber is famous for overbuilding in house but this example does not feel defensible. 2020 revenue was $11bn. 2024 revenue is $44bn. $8mm is not very material in the grand scheme. Could it be optimized? Maybe but I don’t know the full surface area and this article is overly aggressive with opinions. $250k a year just for writes sounds cheap to me when your top line is in the billions.
• gadders
  $8m is a lot of money for us working stiffs, but it's about 0.03% of their 2025 profits.Hindsight is 20/20. Not saying they did the right thing, but they may have had specific performance reasons for originally going with DynamoDB.
• kayodelycaon
  Who exactly is supposed to be fired for this?If you don’t have price controls, it’s easy to run up a bill.If no single person had the responsibility to check the cost, then no one actually failed at their assigned job. So you either fix the system or fire everyone involved in the decision.What you’re doing now is looking a scapegoat to beat up. You’re angry and you’re going to make someone pay for pissing you off.
• dasil003
  This seems like dramatically overstating the mistake. Yeah it was expensive, and yes this could easily been foreseen, but that’s really small potatoes compared to mistakes I’ve seen. I mean I’ve seen promos off shit that never even fully worked beyond pilot scale and had to be rolled back because it was fundamentally flawed on purely technical level.
• Menu

  taspeotis
  This is an ad
• siliconc0w
  8M is kinda small potatoes - which is essentially the AWS business model. Sure you could build a cheaper thing but that is hard and this thing is right here, easy(ish) to use, and your company won't regret using it until you've moved into a different role.
• Esophagus4
  Hate to say it but kind of a lousy article... zippy writing but lots of Monday Morning Quarterbacking for something the author doesn't seem to show much knowledge of. Maybe this is his style to gin up subscribers, but I'm not a fan.> But nobody was optimizing for cost. They were optimizing for their next promotion. Each rewrite was a new proposal, a new design doc, a new system to put on a resume. The incentive was never to pick the boring, correct choice — it was to pick the complex, impressive one....I guess it could be possible nobody thought about cost at all, and this was all misaligned incentives and resume-driven development, but I find that kind of hard to believe? As someone who has made cost mistakes in the cloud, this claim seems a bit silly.Not to detract from his experience, but I didn't actually see much payments experience at all on his resume, so I'm curious why he's branding himself as a payments guru. Kind of tech content creation fluff, I guess.
• teling
  It's helpful to rewrite software every 2 years as your team turns over. The code you understand best is code you wrote yourself, and no one likes maintaining other people's legacy code.
• askbjoernhansen
  The author clearly hasn't worked for a big company. An $8m mistake? Meh, not great, but it probably seemed like a good idea at the time. Seen worse.Oh, it was $8m over several years? That's a couple of projects that didn't pan out, or a small team that wasn't firing on all cylinders for a stretch.Nobody got fired because there was nothing unusual to fire anyone for.
• Menu

  skizm
  In general is there any practical way to fix the issue of "Every rewrite was someone's promotion project"? There doesn't seem to be any incentive for employees to care about projects long term. Keeping something running smoothly is never rewarded the same as launching something new or fixing something broken.
• Menu

  blenderob
  > With each trip generating multiple ledger entries, and Uber as a whole processing 15 million trips per day, it didn’t matter that DynamoDB was great because of high throughput at global scale. The proverbial bean counter should’ve stopped this madness from happening.> At Uber’s scale, DynamoDB became expensive. Hence, we started keeping only 12 weeks of data (i.e., hot data) in DynamoDB and started using Uber’s blobstore, TerraBlob, for older data (i.e., cold data). TerraBlob is similar to AWS S3. For a long-term solution, we wanted to use LSG.Honest question. Why do people go for this kind of complicated solution? Wouldn't Postgres work? Let's say each trip creates 10 ledger entries. Let's say those are 10 transactions. So 150 million transactions in a day. That's like 2000 TPS. Postgres can handle that, can't it?If regional replication or global availability is the problem, I've to ask. Why does it matter? For something so critical like ledger, does it hurt to make the user wait a few 100 milliseconds if that means you can have a simple and robust ledger service?I honestly want to know what others think about this.
• paulbjensen
  What's an $8m mistake to a company like Uber? They made $9.8bn in profits in 2024.
• AbrahamParangi
  the author overestimates how much ~$5M/yr actually is. a business like uber isn't happy about that but it's not even in the top 10 of things they're wasting money on. moreover this isn't the engineer's sole fault it is more the fault of whoever actually approved the expense.
• tajd
  > A redesign that gets replaced 2 years later is a catastropheI mean, given how quickly things can change I think the language and sentiment here isn't quite right, it's just how businesses can change and we can't necessarily control that.
• sailfast
  Submarine article.Outside of that, it sounds like the system worked perfectly. They launched, they paid DB costs (the 8M was not a ledger mistake) and then they rebuilt after they wanted more cost savings. Also a bunch of folks got promoted.The 8M came from VCs lighting money on fire. Honestly this seems like the system worked as planned to me, not a case study in how not to do things.
• Menu

  lozenge
  Spotify was another big proponent of DynamoDB, does anybody know how that went?
• fontain
  Everything is a good idea until it isn’t. The entire industry was enamoured with microservices for far too long. We can look at these mistakes in hindsight and learn from them but we can’t judge them without the context of the time. Software was very different even just 10 years ago. $8m is a rounding error.
• Aboutplants
  Imagine firing everyone that made a mistake, and not keeping the people that learned a valuable lesson?
• refulgentis
  This is horrible slop, and I gave it a long chance. Gave up after handwringing about how DynamoDB would be $300 a day for Uber. Should have gave up when it framed each DB evolution as a “promo project”
• wat10000
  The breathless tone of this article is irritating. Was this a bad decision? Maybe. Is $8 million some vast amount that merits this sort of wide-eyed crazy ranting? Uber's 2020 revenue was around $11 billion, so I'm going to say no, not really. Obviously you don't want to burn millions willy-nilly, but for such a critical component, this isn't so terrible.
• arc_light
  [dead]