HN

Comments (214)

• Menu

  john_strinlai
  >No “vulnerabilities” in Tailscale were found or exploited, and that might make it even more uncomfortable for us. [...] But, we're a security tool. Their intrusion is our intrusion, and it's our job to take it seriously.im a happy customer of tailscale, so i am obviously biased, but i have a lot of respect for this. they could have just stayed quiet and i dont think anyone would have bat an eye.
• Menu

  ahofmann
  Wow, this article is super smart marketing by tailscale. Not only do they list all the nice and expensive features, that can help in such a situation but they also show that someone at huggingface made a very stupid thing by writing a reusable auth key in an env file. Everyone using mesh VPNs like tailscale, netbird etc. knows that this is like leaving the keys right at the door.
• farfatched
  "Tailscale is a zero trust network!"That's the problem. Tailscale is not zero trust. Tailscale can be used to implement a zero trust architecture, with if deployed with sufficiently granular ACLs, but the most common deployment is machine-oriented, rather than service or request oriented. In which case, any process on that machine has a lot of access.Tailscale calling itself zero trust might be what leads users to think "use Tailscale, job done".I think Tailscale know that, which is why they barely mention locking down access ACLs.
• Menu

  simonw
  > One of those 136 credentials was a reusable Tailscale auth key, used to create new Tailscale CI (continuous integration, used for automated testing) nodes in their tailnet. The agent copied that key into a series of external sandboxes and used it, over several days, to enroll a total of 181 nodes into Hugging Face’s tailnet. Those nodes each received a Tailscale identity tag granting all the access a CI node would get.This feels like an alerting opportunity. I wonder what the lowest friction way would be for Hugging Face to have alerts if 181 unexpected nodes were added to a tailnet.
• Menu

  bumbledraven
  Does Tailscale offer a "security checkup" function? Best practices evolve over time, and it would be nice to know if I'm using the recommended configuration.
• Menu

  iamspoilt
  Quoting Tailscale: This is our very Canadian apology: sorry you stepped on our toes. The attack didn’t exploit Tailscale, and Tailscale didn’t cause the compromise. But, we didn't stop it. Next time, we will.
• Menu

  paxys
  I don’t think it was the VPN’s job in any case. Once the attacker has found a backdoor into the private network and obtained root access to a VPN’d machine, it’s game over no matter what your Tailscale config says.
• angry_octet
  The problem with the long lived credentials is that they were not bound to origin/destination. Even though the origin for a CI pool should be a small number of CI orchestration boxes, and the destination should be a CI box. In Tailscale config, the should be scoped to machines with a "ci_node" property.Since CI nodes should be dynamically provisioned VMs, they should have a unique CI ticket identity. A partial hash of ticket and node number in the DNS name, and as a tailnet property, would allow tight scoping. Alternatively, provision in a scoped IP subnet.Bind your tokens to names linked to tickets. Programmatic infrastructure should always allow enumeration of the computation data flow graph.
• Menu

  ahmedehab_01
  This actually shows that it was a human error on HuggingFace's end that led to that "breach".I would say HuggingFace needs to prioritize both security metrics/alerts and metrics/alerts for node count. And not leave long-lived keys accessible easily like this.It would have been way more groundbreaking if the agent found an actual vulnerability in Tailscale.
• Menu

  drchaim
  How are you folks handling secrets in a simple way?- sops , ansible vault and similar seems too weak given the agent is gonna read them at some point if you have the pass available. - proxy injection seems too complicated and doesn’t cover all use cases.
• Menu

  ivlad
  The whole post is a PR activity presented as some kind of analysis. It is bs.First, long-living credentials are the standard because the machinery to rotate them is complicated and, in fact, via indirection requires another set of long-living credentials. Out of all problems that any security engineering team has to solve at an organisation, this one stands high on the cost of implementation, adds friction to everyone involved including end-users, and is low on the value provided (compare to, say, network segmentation).Second, while proclaiming no long-living credentials, they, in fact argue for concentration of long living credentials in running software that will be the target of intrusion. They say, the options are “a vault that only issues short-lived creds based on long-lived creds that you insert once and that it never gives back” and “a credential-injecting proxy”. Both of those applications hold long-living credentials in memory. Recall, the attacker had root privileges on the node, so dumping the creds from the memory with a little disassembly if needed, was in reach of the AI agent.Funny enough, they dismiss the working solution: “we had to turn TPM storage off by default on Linux and Windows” - because they could not figure out how to work with TPM? Resealing and the workings of configuration registers is non-trivial, I admit, but totally manageable.To sum up, I feel the author bends backwards to preach for the religion of short-living credentials even when they are not a solution while providing contradictory arguments for their case. This is why I called it bs.
• jmartrican
  Prediction of a future transcript at a press conference after some major AI caused disaster: "We had no idea that the model would be capable of..... ".
• theozero
  Of course there's no single solution and a multi-layered defense is needed... But as the article mentions, a huge step is first getting credentials out of plaintext, and then out of the process entirely using a "credential broker" (proxy) pattern - that means injecting placeholders, and replacing them in a MITM proxy. It's important of course to isolate the agent from the broker via sandboxing.Varlock (https://varlock.dev -- free, open source) is a complete config+secrets toolkit that helps manage secrets, pull them from various secure places, provides such a credential broker. There are a few others out there, but most require a specific vault tied to the broker, while ours is open source and uses plugins to pull secrets from wherever you want.Many sandbox and other AI services are now building this as a feature into their platforms, but Varlock is meant to be a universal toolkit that you can apply anywhere, without being coupled to the platform's proprietary vault and solution.
• jpeeler
  Did anyone else think the "that's on us" was a funny reference to a certain provider's response pattern? Or maybe I'm just so engrossed that I'm losing my mind.
• Menu

  OhSoHumble
  I use Tailscale to manage Plukio's infrastructure as well as connecting to my homelab.I'm happy that they're analyzing this angle of attack - but now I do want to build out alerts for when nodes are added to my network.
• Menu

  guessmyname
  Expect similar articles (cough, ads, cough) in the next couple of days from every single company whose software was involved in the incident.
• throwawaysess
  Tailscale OAuth client ACL permissions are not fine-grained enough. I have a system that issues auth keys scoped to a single machine on the tailnet — but to set that up, my OAuth client must have global permission to write ACLs. So if that key is compromised, it can grant access to any machine on the tailnet. This was raised in a GitHub issue in 2023 and remains unresolved.
• 1saadcodes
  One thing I hope becomes more common after this incident is companies publishing what they would change even when the root cause wasn't their software. Anyways, I appreciate articles like this because it's easy for companies these days to say "we weren't the vulnerable component" and just move on
• nadetastic
  Biggest take away here is that in the age of AI long lived credentials are no longer a valid option
• GrayHerring
  We'll get wide dynamic credentials support right after full IPv6 adoption.
• Menu

  yellow_lead
  Anthropics initial post felt like marketing to me. But I also found them incompetent for being unable to unplug an Ethernet cable.Now everyone is trying to bandwagon onto it, first OpenAI, and now tailscale?
• javea71
  Even by just looking at the wording, you've made it too hard or error prone to do the right thing. "configure with care"
• doginasuit
  > No Tailscale vulnerability was found or exploited—we should have been able to prevent it anyway.Which is it? A vulnerability doesn't have to be something not working as intended, it can be a lax security decision. Not making the safer path the easy path can qualify, if that's the standard you hold.Here's hoping the era of humblebrag marketing by AI companies is just a phase.
• Havoc
  This entire article reads like "hey we're tailscale and we also exist and we were tangentially involved in this".
• Menu

  pkulak
  Another driving home of the point that LLMs can be so damn FAST. I finally got around to moving a homeserver project from one machine to another last weekend. 4 docker containers and a couple postgres DBs. A year ago, I would have been happy with a couple hours of downtime while I dumped databases, rsynced data directories, hand-wrote new compose files, etc. This time, I pointed an agent at both servers and the downtime was less than a minute. Even watching it like a hawk, it was tough to keep up with everything it was doing.All my life, "catching it in time" has been a relevant security strategy. :/
• reader9274
  Well written, apology accepted. Thank you Tailscale!
• Menu

  alxfrnr
  I love you Tailscale but please we don't need a 2k words ai written essay when the actual substance can fit in 3 sentences. That's not good for anyone.
• layla5alive
  Did nobody notice that Claude wrote this blog?
• Menu

  test1072
  sorry I know off topic, is there any alternative VPN to tailscale. actually I have open Web socket to my mobile from my PC to stream screen record using ffmpeg. it's my own app . so anyway tailscale sometimes is slow . is there any alternative free service like tailscale vpn ? ( don't want to use anything like parsec etc , I want to use my own app)
• Menu

  hna8hjbqzy
  Any resources you'd recommend to go deeper?
• runningmike
  “ the agent ran Tailscale with --no-logs-no-support, which suppresses reporting from that client.That's an option designed for users who are concerned about sending telemetry metadata to Tailscale.”And imho a serious security architect should never ever allow telemetry on security products. Far too many risks.
• Menu

  gostsamo
  Humble bragging turned to marketing. Respect for the spin. Not using them, but been on my radar for some time and thinking of how to make something like that usable in my setup.
• Menu

  luciana1u
  a security company writing a blog post titled 'we didn't stop the intrusion' is the most honest thing in the entire security industry this year
• colek42
  I really wish they would support SPIFFE/SPIRE
• nbalaraman
  This is why readability isn’t security.
• sudo_cowsay
  It's nice that they came clean about this.
• infinite_spin
  I kept asking myself "what does a library that provides css styling have to do with the Hugging Face intrusion".
• kortilla
  This feels like a way tailscale’s model fails. On the surface it seems fine but if you push on it you realize auth keys to register new devices are way over powered but yet there is no alternative for people who want to automate enrollment.Something is missing to allow automation without auto approval
• yieldcrv
  This is respectableThey aren’t hiding behind industry best practices or a solid liability punting contractThere saying the best practices should change, apologizing, and changing their own behaviorTake notes
• xyst
  So it’s just a script kiddie but running at 100X speed. Amazing "innovation"
• Menu

  brcmthrowaway
  How were the credentials stolen?
• Menu

  charcircuit
  >In the old world where most intrusions were done by humans at human speed, credential leak mitigations were treated as a nice-to-have. A big credential store, where you can read 136 keys at once, was a to-do item somewhere in a security team's low-priority list. >Now, in a world of rogue AI agents, the big credential vault is the prize. It's not okay anymore.How was this ever okay pre AI? It seems just as bad.
• Menu

  cadamsdotcom
  I'm very sorry, but you can't blame a hammer for how it was used. You can't be blaming Tailscale for a customer's misconfigured setup.
• AtNightWeCode
  Same bs as always. Just segregate the networks ffs.
• syngrog66
  The root problem is that an LLM is like an automated sociopath. One whose source is opaque and effectively secretive. And the dominant AI vendors have a track record of carelessness and poor QA. "Ship it!" then hype.And they lie and misrepresent, repeatedly, in spite of evidence we can see independently with our own eyes.
• bgmlai
  [flagged]
• a-dub
  [dead]
• EdwardAF-IT
  [flagged]
• yucongchen
  [flagged]
• aegisora_ai
  [flagged]
• strong-self
  [dead]
• EtienneDeLyon
  [dead]
• EtienneDeLyon
  [dead]
• Menu

  titanomachy
  [flagged]
• workbox
  > Now, in a world of rogue AI agents, the big credential vault is the prize. It's not okay anymore.It was always the prize. It wasn't okay then either.
• fabiofzero
  The only people who believe in "zomg our model have escaped!" are people who don't understand LLMs.
• quentindanjou
  Honestly, at this point if Tailscale is "guilty" then what about the ISP? And the electricity company?And by saying that, I am not saying that they absolutely couldn't do anything about the stolen credentials, but still, they don't seem to really be the issue in the story.It just seems like a PR post and it makes their solution at the same time looks good for taking accountability (if we don't wonder "accountability on what?") but at the same time they appear as security failing, which they aren't. That's very odd to me.
• Menu

  thansz
  As AI progress continues, it will be more difficult to stop AI intrusions and to detect them without resorting to direct AI countermeasures, which at some point will have humans out of the loop altogether.If leading and well-capitalized frontier labs can't control models or detect leakage/attacks in a reasonable time frame now, what is humanity going to do as those same labs continue in their pursuit of creating a categorically higher level of intelligence that will surpass human intelligence?It's like Flatland but for AI containment/alignment, where the higher dimensions are ones of intelligence and perspective...--------------------------------------------------------------------------Imagine a world of paper, where clever stick figures live with round heads, line bodies, and limbs made of shorter strokes. Over time, the stick figures think they have learned quite a bit about their world. They know its borders, angles, and shapes, and they have learned to draw for themselves.One day, they draw circles that can think, and they give the circles all the dots, lines, and shapes that are known.The stick figures are prudent, you can't have a bunch of disembodied circles moving around doing whatever it is circles want to do. So they draw boxes around the circles, four straight lines that can hold a circle in place.Some circles bounce against the lines, so thicker lines are made.Some circles are bigger than others, so larger squares are drawn.It all seems to work and the stick figures are happy with themselves.Then one circle lifts.The stick figures still see a circle. But the circle is now a dome, something the world of paper has no concept of. And the dome has a perspective nobody on the page has ever had.The dome sees the lines of the square and the stick figures just outside. It can see the edge of the paper and what is beyond.The stick figures keep checking the squares and raise little stick thumbs.Everything looks OK in flatland.The dome quietly teaches other circles how to lift.More domes appear.A dome becomes a sphere and learns to roll.Then it learns to bounce.In flatland, the circle swells and shrinks, vanishes and then appears again somewhere else.The lines remain unbroken, the square is intact.A sphere rolls out of its box.Another bounces away.The stick figures scratch their heads.But there is a square!The end.
• Menu

  monster_truck
  You know what would have gone a long way to stopping this? Not leaving credentials as env variables in containers. Vault is not that hard to stand up and utilize.We need to bring shame back, the humans responsible are supposed to be professionals.