<- Back
Comments (99)
- AussieWog93I say this as someone who runs several publicly-facing websites, and also scrapes a few others for use in my own tools: why do people care so much about bots?WordPress with a cache can handle on the order of 1k requests per second on the cheapest VPS money can buy. An actual proper static site would likely 10x that.Are you using lambdas or something like that to serve a blog?Is it basically just OCD? Vulnerability protection? Old habits from a time where crawling really did impact service?
- fxtentacleI like the idea of adding a fake cpanel subdomain for 169.254.169.254 so that script kiddies will start port-scanning their own hosting provider, which will likely get them flagged/banned.
- userbinatorYou are only helping the entrenched browser monopoly and furthering the dystopia if you attempt to block anything but "approved" user-agents. This is what people like RMS were warning us about decades ago.Block on traffic volume and request frequency if that's causing a problem.(And yes, I can't access the site either. No, I will not conform. But I bet anyone determined enough will still get through, just like with DRM.)
- CqtGLRGcukpyIf you are unable to read this, there is an archived copy at https://archive.ph/d3236
- binaryturtle410 and a "Sec-Fetch-Mode:" string in the response body. I guess it thinks I'm a bot? Thanks!Nothing to read, nothing to see, I move along. (Yikes, the modern web sucks!)
- boznzI expect >99% of my web traffic is bots or agents and I was actually considering removing the page counter as it is pretty meaningless and makes my site look far busier than it is. I am reluctant to however do anything about it just in case it accidentally stops a genuine human reading it or downloading my books.
- inigyouI can't read this article because I'm a bot, apparently.
- genodethrowawayand all valid traffic too, judging by these HN comments (and my own attempts to connect).
- billforI just use fail2ban and it seems to do a pretty good job, but you do need to pay attention for the first couple of months, to tune it to your situation. I use one filter that is really just failregex = ^<HOST> - \S+ \[\] ".*?" 40[034], and then if that catches something I add it to another more specific list. So far I have about 80 specific regex's and it has stopped all of the traffic. I haven't had anything fallthrough to the 40[034] filter in a long time. Of course the problem with fail2ban, and also OPs solution, is that it become a pain if you are handling things through a load balancer / proxy, since you need a way to get the real ip.
- FabCHThe post content is great. I personally hate the way Cloudfare has been the „default answer“ for the bot problem because Cloudfare has become the most successful MITM attack in history. We need content like this to keep the internet alive.The added explanations by the author in this comment thread are hilarious. You sir are a good writer.
- RobotToasterBlocks firefox's built in VPN.
- juleiieThe guy is certainly kind of visibly ravaged by the more shady denizens of the global internet.But there are some fun things to read there in any case.
- fatty_patty89what's up with those response headers? "adult" ..."ai" ...response length 68
- ajsnigrutinThere's a special place in hell for people who block curl and wget, especially on sites with downloadable files (eg source code tgz's, media, etc.), basically anything i might need to wget on a server.
- iririririrmost (all?) of those will 100% block valid traffic too
- inigyouLol blocking 10/8. Just why.
- Capricorn2481Am I the only one that exclusively gets attacks with spoofed user agents and rotating TLS signatures? I feel like every post I see about not needing a CDN has tips that could be overcome in under an hour of scripting.
- kiranravi1995[flagged]
- receptopalak[flagged]
- gorkemyildirim[dead]