HN

Comments (320)

• Menu

  semiquaver
  > because apparently what we have to offer is worth spending a ton of cycles to calculate the Anubis challenge. This statement holds the core misapprehension behind Anubis. It’s not a ton of cycles. There is no difficulty setting that would be inconvenient for bots but usable for humans on mobile devices.I noticed the other day that lists.ffmpeg.org had moved to Anubis difficulty level 6, which takes ~180sec for my iPhone 17 to solve at ~100KH/s, making the site unusable. So I spent ~10 minutes vibe coding a safari extension with a native bridge to an optimized C kernel using ARM SHA256H* instructions that can do 200+ MH/s on the same device. This solves Anubis difficulty level 6 in a handful of milliseconds.Given the numbers and capabilities involved (a single $5K ASIC miner yields 200TH/s, a million times more hash rate than my optimized kernel running on an iPhone), I don’t see how proof of work could possibly be a sustainable strategy to keep bots out without ruining human user experience. It’s an arms race that can’t be won.Edit: I encourage you to try this yourself. Here's a sample prompt that ought to one-shot the task: > Build an iOS Safari Web Extension that accelerates Anubis proof-of-work using a native C ARM64 SHA-256 kernel. Precompute the invariant 128-byte challenge prefix, search fixed-width decimal nonces with ARM SHA-2 intrinsics and two worker threads, and target difficulty-6 solves under one second. Relay challenges from a Safari content script through the background service worker to native code, then submit the valid nonce/hash through Anubis’s normal pass-challenge endpoint. Include a deterministic benchmark app, correctness tests against CryptoKit, bounded execution, and fallback to Anubis’s stock solver.
• Menu

  robotmay
  I've spent the last few days adding traps to one of my websites, ironically using LLMs of course, and I've been having quite a lot of fun doing it.Instead of the proof-of-work system of Anubis, I've gone down the iocaine route but implemented it in my application itself, as it's built in Elixir and causing problems for scrapers is really fun when it takes almost no server resources.Currently I trick bad scrapers into a fake infinite black hole path with the promise of tasty data, then serve images to them one byte at a time over 15 minutes (after sending the header quickly), bloat the responses to cost them tokens, and randomly return AI generated images of sexy toasters. I have an admin dashboard with a little leaderboard for which ones get the most stuffed, and it keeps my heart warm on these wet autumn evenings.
• kdowns
  I made it to a third round interview at anthropic in 2024 and they had me build a web crawler as their programming test. Part way through I started on making it respect robots.txt and I could immediately tell they were no longer interested in me.
• Menu

  virgoerns
  I also run a public cgit instance and get over 1M hits every day, although my pet projects are nowhere near the size or impact of kernel. I had to block (via nginx conf) cgit endpoints for diffs, blame, snapshots and historical commits, because nothing else works. Now they return 402 (payment required). I consider this my total defeat and it's killing me inside, but it is what it is.
• Menu

  tptacek
  Tavis Ormandy called this, about Anubis, almost exactly a year ago:https://news.ycombinator.com/item?id=44962529It never really cohered as a solution. High-powered scrapers are better equipped to handle proof-of-work challenges than end users. Proof of work makes sense for a password hash, where any one guess at a password provides zero marginal utility. But every request from a scraper is productive to the scraper.
• Menu

  mzajc
  > Why is git.kernel.org “interesting” to crawlersI think the post underestimates just how little thought and effort is put into these bots. I also run a cgit instance with far less interesting projects, and am not spared from the deluge of HTTP requests.The explanation I could come up with is that they try to crawl all links regardless of how much sense it makes or how much load it causes. cgit being cgit, this means billions of links for all combinations of parameters and hashes. That, or it's a deliberate DDoS attack.
• Menu

  jdnier
  I really enjoyed the writing style in this article.And the bot progression from "alter user agent" to "change IP addresses" to providers having to ban whole subnets, whole ASNs, and realizing "proxy SDK monetization" is a thing mirrors threat actor progression from the time before LLMs.
• Menu

  Demiurge
  I maintain a formerly popular gaming website, and it used to have hundreds of legitimate requests per second. The load would be especially high during popular event times. So, it’s always been running on a dedicated server.It also has an “online users” counter, which attempted to count real user sessions of unauthenticated user which still maintained a session, which lets them comment, or modify certain filter and display options. It never counted the Google bot.Over the last few years this counter went from 100-200 users online to thousands. I have been very hands-off with it for many years, doing minor upgrades and backups. However, the site also has gotten quite slow, these sessions were obviously impacting it. So, I finally investigated these crawlers, and yes, it turns out it’s an insane amount of traffic that is entirely artificial, the site has just a handful of real users, and thousands of these crawling sessions that actually try to do everything they can, click every button. It doesn’t help that sort and search were implemented using GET links.I fixed the counter to exclude the crawlers, but I have a bit of a dilemma. I don’t want to stop the bots from updating their knowledge based on all the content.The best solution I could find is the new CloudFlare feature where they might charge the crawlers for every request, or otherwise block them. I think that’s a fantastic idea for the internet, at large. I signed up for the beta access, but haven’t heard from them again. I do think it’s unfortunate that this requires CloudFlare and the middleman.Overall, it seems like the LLM are really straining the internet economy, the openness of it. Email spam used to be the worst, but the organized trillionaire labs sucking up the entire internet is going to break something if we don’t preempt them better.It’s too bad the copyright and public internet systems are not acting quick enough. And I think there is no reason to act like this race really has to be at such a breakneck speed.
• Menu

  Waterluvian
  > It was immediately extremely effective — the bots just gave up. For a few months, it was bliss: bots were blocked at the perimeter and gave up, moving on to easier targets; the users were mildly annoyed but tolerated it, and the Anubis stack was easy enough to deploy everywhere.As a tech person who works with tech people, I have become extremely sensitive to this kind of bias. Is this solution actually better? Is the CPU cost actually worse than mildly annoying everyone, or is it a problem being solved because it “offends the senses?”I’m not leaning towards yes or no for this instance. But I regularly see people jumping to conclusions without measuring. What is the cost of 20% and is that cost worth “mildly annoying” everyone?
• Menu

  easton
  Side note: why are shallow clones evil? I always thought they were cheaper, but I guess that’s really just for my disk space. (since the server has to compute what blobs to give you instead of just “everything”?)
• javcasas
  At this point they are using residential proxies and stuff, and increasing the difficulty level is not going to help, among other things because they don't pay for it.Why cannot we turn this whole proof of work thing into an official "help mining $SHITCOIN"? I mean, if they really want the data that badly, at least have them pay the hosting with their CPU/GPU/ASIC cycles.
• Menu

  yellow_lead
  High Anubis difficulty is annoying the hell out of me for several sites. And it's starting to not block LLM bots anymore?> 33% are now solving the math and getting through to the main site — because apparently what we have to offer is worth spending a ton of cycles to calculate the Anubis challenge.
• Menu

  ynniv
  this is an increasingly common situation. it goes something like: - i have a free, niche resource - it becomes too popular - i make it more efficient - now it's really popular, and people are "abusing" it - let's make them proof-of-work - ... and proof-of-work harder - but now "legitimate" users can't use it - ??? the core problem is that the average person uses a mobile device where work is expensive, and the "attackers" use servers where work is cheap. if you require expensive proof-of-work, next comes a cheap-work-as-a-service where inefficient mobile devices pay small amounts of money to get efficient servers to complete their work for them. now everyone has an interest in making their usage efficient, but there's still an obvious inefficiency in the system: why have people pay unknown 3rd parties to burn cpu cycles to reduce costs for a free service, when you could just have people make small payments that cover the service's costs?which is called l402/x402. micropayments' day has come
• beached_whale
  I wonder if they could pre-render the stuff older than a month ago and compress it and serve it as static content. Not optimal, trades space for CPU, but might be cheaper.
• superjan
  How feasible would it be to only offer a binary git (partial) download and move the html rendering to the client? It would still be a lot of requests, but less work for those servers. Not that I like SPA’s, but they could be useful here.
• Menu

  lkbm
  > So, you'd think that something that pretends to be “Artificial Intelligence” would use the most efficient way of using our data for training purposes, right? Clone the repos, walk every commit. Done.If repos like this were 10%+ of what they crawled, having a special case for clone-able repos would be smart, but if you're crawling everything, you're not going to do an efficiency tweak for each special case that has a more parse-able option.
• Menu

  delichon
  > Why is git.kernel.org “interesting” to crawlersInteresting to crawlers is not a narrow scope. We have the same problem on a B2B car wash site.
• Menu

  stickfigure
  How about just stop offering a html interface to the code? This doesn't seem like a critical service. Let people clone the repo normally.If someone else wants to run a public HTML service, let them deal with the bots.If you really want to offer a web interface, put it behind login. You can apply enough restrictions (captcha, super slow rate limit for new accounts) that it isn't cost effective to generate zillions of logins, and you can monitor logins for bot behavior.Sucks, but here we are.
• Menu

  NooneAtAll3
  from what I see there are 2 solutions:1) ban TV-proxy-as-a-service - straight up go to every representative there is and start pushing and lobbying and everything to stop spammers from distributing over non-computer devices, especially legally2) make old commits more expensive to access than new ones. Legit users are not going to access those much, so they can pay the time. I assume diverse (unpredictable?) difficulty can also make spam pulling harder
• Menu

  __MatrixMan__
  This is a fundamental flaw in the web. Since we treat a server's name as authoritative, anybody maintaining a replica needs to repeatedly hit that server to know if their remote version is up to date.If we trusted digital signatures on content instead of server names, we could have a model where a single bit of server load propagates to millions of interested parties. As it is the server must do something distinct for each interested party.CDNs mitigate this only partially, because mutable data means they have no good cache invalidation strategy.There's got to be a solution that doesn't involve heaping even more burdensome requirements on those who would dare to publish.
• Menu

  nneonneo
  I wonder if one solution here could be to turn up Anubis difficulty if the first page hit is not one of the obvious entry-points to cgit. It could even have a little hint that says something to the effect of “go visit the home page if this is taking too long”. (Better not to ban them entirely, in case people really did click on some random link e.g. in a news story or mailing list message).Distributed scrapers are going to generally try and hit their assigned list of pages; it’s a bigger waste of time if they have to go off to visit other pages first to get the cookie challenge.
• Menu

  chuckadams
  Given the nature of git, wouldn't all that HTML be highly cacheable? I get that's not free either, but it's got to be a lot less intensive than having cgit generate it every single time.
• NegativeLatency
  Claude loves doing this on GitHub repos too, I have line in my agents file to tell it to clone to tmp and look there.
• initramfs
  I've been noticing page views in the past several months with a much wider span of origin on my Blogger stats. Before I would get a few from several countries, but now I am getting views from tiny countries and obscure or outdated browsers and operating systems, which leads me to think scrapers could be using VPN services in various countries along with header anonymizers that mask the device that they are using.Extensions like ModHeader, BrowserMask do this: https://github.com/apify/crawlee-pythonhttps://github.com/mthcht/Masquerade-SpooferGreat for AI scrapers, bad for hosters and everyone else.
• kgeist
  How about: "Type the seahorse emoji to solve the CAPTCHA" :) Something that triggers infinite loops in LLMs or trips the guardrails.
• marginalia_nu
  FWIW, git hosts have always interacted very poorly with crawlers, to the point where you have to actively code in git host detection to avoid getting stuck in an accidental crawler trap if you want to run a well behaved crawler. Easiest is just to look for anything that looks like a commit hash in a path and drop those URLs from the crawl frontier.Reason they interact so poorly is that is that git hosts generate a lot of links. One for each file in each commit, and a diff for each file appearing in a pair of commits. Even a small repo can have millions of viable links, and most of these are stupidly expensive to render for the git host. On top of this crawlers generally don't have a very deep understanding of what they are crawling, and can't meaningfully distinguish computationally expensive requests from cheap ones.
• cobbzilla
  I ended public access to my git server after I got flooded by bots and my own commits were noticeably lagging.That’s not an option for the kernel. It’s hard to read the cat-and-mouse account with any hope today. I think the flood abates someday but not sure how it happens.
• Menu

  feelamee
  Hm, interesting - how will it look the actual solution for such problems in the future. I suppose the issue will continue to grow.First idea - there should be some cost for sending traffic somewhere. And the server owner also should receive pay - not only the internet provider.So, in with this idea, the server owner can potentially increase the amount of computing power to satisfy all requests.
• Menu

  bourse_lee
  What if Anubis computations were turned into a crypto-miner
• Menu

  afarah1
  Why not aggressively rate limit? Legitimate use of HTML rendered commits should be largely unaffected, and crawlers slowed to a halt. You can even jail after a number of 429's...
• Menu

  AshamedCaptain
  This is bad enough that I'm going to stop serving cgit. I've been doing cvsweb, then subversion, then cgit over my home server for many many years and for the first time ever this is annoying my own bw usage.It's ridiculous also how you ban an IP then 1 second later another one picks up from where the first one left on.
• bauerd
  They're the exception, not the rule. They get crawled like any other site, but happen to host git repositories. It's not obvious that these are targeted crawls and they likely may just end up in crawling queues a lot generally
• Menu

  arlattimore
  In the case of kernel.org, why not make the unauthenticated version return only the latest kernel repo with no history (tiny number of URLs relatively speaking). If you want full kernel.org features, login.
• a-dub
  i wonder what they're all up to. i imagine some are scraping datasets for pre-training, others are probably real-time scrapers looking for security bugs, even more still are agents working on coding tasks and looking at the kernel. also interesting to think about solutions: does everything need to be optimized now for weird access patterns that proliferated ai creates? do the ais need to have behavior trained in to be better netizens? is this the end of anonymous browsing and the beginning of an era where one has to attach an identity to all requests? or the end of community hosted free information services more broadly?
• inigyou
  I also had this problem, but since nobody actually uses my gitea site besides crawlers, I just let a script run through my access log and ban every IP address who asked for a commit in the last 24 hours
• userbinator
  So, you'd think that something that pretends to be “Artificial Intelligence” would use the most efficient way of using our data for training purposes, right?I'm strongly convinced that these aren't "AI crawlers"; they're just plain DDoSes done by those who have interests in turning the Internet into a dystopian walled garden with "security", and now they have a convenient scapegoat to blame. Don't you find it too coincidental with the rise in identity/age verification and other attempts at silencing free speech on the Internet?It's widely known there are questions that LLMs can't solve, and once in a while an obvious example appears, so a simple CAPTCHA-like challenge with an HTML-only form would be the logical "defense". Instead there's a huge interest in pushing JS-required proof-of-work (as others have pointed out, these attackers have far more compute than the average user) and remote attestation (there are already providers with huge farms of mobile devices that can defeat this easily).Things just don't add up.
• Menu

  jopsen
  I've seen this too.I think it's a few bad actors really. Because nobody serious about indexing content will do what these crawlers are doing..They are consume lots of content that is unoriginal or duplicate or duplicate with minor modifications.Not sure how to block, but maybe a little bit of law enforcement could dramatically reduce the number of TVs being used a proxies.
• hei-lima
  Great chart! Does anyone know what tool was used to make this?
• Menu

  ChocolateGod
  > phone gets uncomfortably warm as it's doing the number crunchingIMHO if I visit your website and it intentionally starts wasting my electricity for no other reason than to cost me money, with no opt in, it's hostile and malicious.
• Menu

  Kuinox
  1.4 billions requests, 258 160 cpu hours. That's 1.5 requests per second ? I'm starting to believe, the issue is more that their software is not well optimized.
• Menu

  andruby
  A creepy crawly is a South African invention to clean your swimming pool. The company that introduced them in the 70ies is called Kreepy Krauly. Also popular in Australia.https://kreepykrauly.co.za/about-us/
• Menu

  Velocifyer
  I would add cloudflare, but set it to cache only mode *without* the bot blocking features.
• Menu

  edent
  WordPress powered a huge number of websites. Yet the crawlers all go straight for the HTML of those sites rather than the more efficient and structured JSON API which all WordPress sites have.If these crawlers are so smart, why aren't they following the rel="alternate" which is provided explicitly for them?
• Menu

  bilater
  Instead of trying to block why not monetize? So the proof of work can be directed at something you can be paid for (bitcoin mining)?
• Menu

  alkonaut
  Proof-of-humanity can’t come soon enough.We’re talking about privacy-preserving proof of age, but as we see here the real utility of such a system will be proof of humanity.
• anon
  undefined
• lxgr
  > [...] when a source is guaranteed to be LLM-free, like the entire history of kernel commits [...]Is that really the case? It was my understanding that LLM-based agents were explicitly allowed as long as their users follow certain guidelines [1]?And more generally: Somehow the theory of "essentially all bot traffic is AI labs crawling the Internet for LLM training data" doesn't make sense to me at all. There are at best dozens of labs capable of running their own crawl at Internet scale, but hundreds of millions of people using LLMs to answer their questions. (If my personal LLM usage is any indication, firing off dozens or hundreds of web fetches to answer a single question is not unusual.)While I understand that many existing projects have been resourced only for human readers and might as a result be struggling due to this, this characterization sounds a bit dishonest to me.And unfortunately, for this use case (i.e. ephemeral queries in a context possibly lacking storage or git access), forking the individual repo to answer a handful of string match queries against it might just be more expensive than to run that query against a web search index and then just fetch those results over HTTP.The solution would accordingly also look very different, as caching at the inference layer is significantly harder than at the training one (where it's most likely already widely done as that seems like a no-brainer).[1] https://docs.kernel.org/process/coding-assistants.html
• Menu

  Artoooooor
  How expensive would AI access be if every user paid their fair share instead of shoving it on the people doing the actual work?
• Menu

  tarpitt
  Maybe you could have a system that heuristicially detects when an crawler is making the request and then feeds them a modified page, itself generated from an LLM, that injects vulnerabilities and bad code and discussion and such.
• asah
  Just slow unauthenticated traffic to non-essential stuff...
• DarmokTanagra
  AI has simultaneously made the easiest parts of web development even easier while making the hardest parts near impossible.
• oowa
  have a hackathon to solve for this. OP says it's not a problem for him right now but if we extrapolate what he's talking about it's definitely a problem aaaaaaand It's totally solvable, Even with all of the crazy combinations he's talking about it's still solvable. And it's already been solved using patterns we see in streaming services. This is completely hackathonable. but why do we even need to bother with this? The slurpers are the cause of this, and they can cause this problem because of Murphy's Law. well you can only account for Murphy's Law with good architecture or something like that or whatever. Ha ha hackathon.
• Menu

  singpolyma3
  Why is no one filing lawsuits over this yet?
• Menu

  nxndbebdb
  Just serve the raw commit and render on frontend. I really don't get why they are complaining, just be performant
• Menu

  jruohonen
  Off-topic, but anyone with which he did the plots?
• monegator
  The thing that bothers me is why the fuck are they still scraping git.kernel.org or any other site that has already been scraped a million times before. Who would pay for that data?Then again there is the conspiracy theory about cloudflare sponsoring the scrapers
• hnisjafx40
  Learned this the expensive way
• pbronez
  “Expect to lose some functionality, at least when accessing our resources anonymously.”This seems fine to me. It would be a better world if we could have anonymous bulk data access. But if aggressive scrapers are bloating host costs, I’m fine with logging in.Now, the flip side is that ONCE logged in, I want my bulk access. The worst of all worlds with when you demand authentication and then STILL block bulk access.Case in point, I want to automatically download my Amazon and Target order records. This is easy to automate with playwright or whatever, but authentication stays annoying. My sessions expire quickly and I have to re-auth all the time. There should be an API to pull this data down.
• anon
  undefined
• api
  The AI companies should have their AI fix their crappy inefficient crawler code.
• forrestthewoods
  > Shallow clones are awful. Run your own damn mirror if you're going to do something nasty like that.TIL shallow clones are expensive. That's wild to me. It's supposed to be cheaper!
• Menu

  Velocifyer
  But why don't they just git clone?
• bluedino
  > But no, let's in fact choose the stupidest possible way of doing it — by rendering everything as HTML commit by commit and then parsing it.I feel like I'm at work.We had some web crawler using Selenium to make queries and scrape the data instead of just downloading the whole file.Every day it seems like we have some people that know just enough to be dangerous creating things like that. And then of course it's our fault that things are slow, or we won't give them infinite system resources, etc
• 6d6b73
  Add a lot of random text to the html pages, preferably hidden to regular users, have the bots use lots of tokens to process it all.
• Menu

  calvinmorrison
  I love to see the 'leet kernel hackers and maintainers' struggling with basic volume. Each page load should cost you near nil. Us lowly PHP developers have been caching shit for close to twenty years.Learn how to cache your application and your cpu usage should be almost zero. In fact basically any read should cost nothing in comparison to writes.
• gib444
  Running Firefox with Temporary Containers Plus makes challenges 10x more annoying :D(Each new tab is isolated, unless opening a link in a new tab. Same as Safari in private mode)
• 0xbadcafebee
  I'm assuming they haven't yet sent responses to the bots? Since AI is dumb, you can send errors that tell the bot to git clone rather than crawl. If it's vulnerable to prompt injection, it might listen and do the clone instead and stop trying to solve challenges.Barring that, I think the solution is to charge money for access. Require users to sign up to render HTML, and provide a form of payment (any form you want). The cost is, say, $0.1 per GB. Rate limit all requests to reduce CPU. For the average user this will cost a few cents. For the bots you'll cover your costs and have a rate limiter to keep your system from being overwhelmed. Or they can git clone for free with no limit.
• Menu

  nicman23
  couldn't you have anubis on a dynamic difficulty? ie if a ip requests more than 1k pages per day +1 the difficulty ?
• adverbly
  Is it really stupid if it means more data centers need to be built and it keeps the AI bubble going and GDP number go up?
• acedTrex
  It feels inevitable that many systems will have to go to a login/trusted ip source type system. Its just not feasible to continue to operate with 99% of your traffic being fake.
• anon
  undefined
• iririririr
  anyone knows how Jwz solution is working?dont click next link because he will show a nutsack image if the referrer contains hackernews. love the guy.www.jwz.org/blog/2025/01/exterminate-all-rational-ai-scrapers/basically, instead of blocking, he just poison it. and if a human sees it, it takes less effort to ignore the nonsense than it takes your pocket computer to deal with proof of work.
• Menu

  TZubiri
  Same problem we've been having for ages.Using shared ip banlists is the best solution so far, like cloudflare. Sure maybe they hit your server for 5 seconds and then desist, but they'll attack someone else, and they'll eventually rotate.I'm not sure if Anubis has a feature for centralized banlists, but I'm assuming since it's OS and privacy oriented, there isn't.There's a tradeoff between privacy and abuse, you want privacy? You get abuse, you want to battle abuse? Gotta sacrifice privacy. Worth noting that unmarked vpn users (residential proxy or residential vpn users) use these proxies for privacy, and therefore give a reasonable alibi to abusers.
• stefantalpalaru
  [dead]
• Menu

  petesergeant
  > Training an LLM on content produced by the LLM gives it the equivalent of a digital prion diseaseIs it foolish of me to have expected more from a blog post on kernel.org?
• Menu

  leoqa
  It seems clear to me we are moving towards a world where you will have to perform device attestation to access the internet. The spam/abuse is too great and accelerating.
• Menu

  znnajdla
  Put a CDN in front and let them absorb the load? Seriously, this is static content, which is so cheap to serve it should be free.