jwz gave the game away, so i’ll reveal:
the One Weird Trick for this week is that the bots pretend to be an old version of Chrome. So you can block on useragent
so I blocked old Chrome from hitting the expensive mediawiki call on rationalwiki and took our load average from 35 (unusable) to 0.8 (schweeet)
caution! this also blocks the archive sites, which pretend to be old chrome. I refined it to only block the expensive query on mediawiki, vary as appropriate.
nginx code:
# block some bot UAs for complex requests # nginx doesn't do nested if, so we set a test variable # if $BOT is both Complex and Old, block as bot set $BOT ""; if ($uri ~* (/w/index.php)) { set $BOT "C"; } if ($http_user_agent ~* (Chrome/[2-9])) { set $BOT "${BOT}O";} if ($http_user_agent ~* (Chrome/1[012])) { set $BOT "${BOT}O";} if ($http_user_agent ~* (Firefox/3)) { set $BOT "${BOT}O";} if ($http_user_agent ~* (MSIE)) { set $BOT "${BOT}O";} if ($BOT = "CO") { return 503;}you always return “503” not “403”, because 403 says “fuck off” but the scrapers are used to seeing 503 from servers they’ve flattened.
I give this trick at least another week.
Re the blocking of fake useragents, what people could try is see if there are things older useagents do (or do wrong) which these do not. I heard of some companies doing that. (Long ago I also heard of somebody using that to catch mmo bots in a specific game. There was a packet that if the server send it to a legit client, the client crashed, a bot did not). I’d assume the specifics are treated as secret just because you don’t want the scrapers to find out.
You could probably do something by getting into the weeds of browser updates, at least for web traffic. Like, if they’re showing themselves as an older version of chrome send a badly formatted cookie to crash it? Redirect to /%%30%30?
Yes, there I heard there is some javascript that various older versions of chrome/firefox don’t properly execute for example. So you can use that to determine which version they are (as long as nobody shares that javascript with the public. So this might even not be javascript, I honestly know nothing about it just heard it).
It’s a constant cat and mouse atm. Every week or so, we get another flood of scraping bots, which force us to triangulate which fucking DC IP range we need to start blocking now. If they ever start using residential proxies, we’re fucked.
I have a tiny neocities website which gets thousands of views a day, there is no way that anyone is viewing it often enough for that to be organic.
quickly, add some ad revenue :P
From ai vendors. Let them pay you for scraping you lol
at least OpenAI and probably others do currently use commercial residential proxying services, though reputedly only if you make it obvious you’re blocking their scrapers, presumably as an attempt on their end to limit operating costs
Oh never heard of that. I have blocked their scrapers via agents but I haven’t felt residential proxy pain.
here’s a mastodon post and linked blog post with some details on what currently sets it off
Infinite-garbage-maze does seem more appealing than “proof-of-work” (the crypto parentage is yuckish enough ^^) as a countermeasure, though I would understand if some would not feel confortable with direct sabotage—say for example a UN organization.
Daym, I should set me up some iocane as well I think
PS: Looks like that sync issue between our instances is resolved now?
yep, it seems so! I haven’t put the permanent fix for the nodeinfo bug into place yet but it’ll be live as soon as I’m able to give it an appropriate level of testing.
They have a botnet on residential devices?
the term of art is “residential proxy” and there’s a ton of them
for example: it’s the flipside of Bright’s free VPN service - through Bright Data they sell people access proxied via some user’s connection
And companies like honey that pay you (a pittance) to proxy people’s requests to porn sites.
Count them as ad visits, to make big tech pay for better hardware or line?
That opens you up to getting accused of click fraud, as AdNauseam found out the hard way but its worth it if you can squeeze some cash out of them before that happens.
