Background: I’m using Ungoogled Chromium, which I do not keep up to date because non-Debian packages are a big hassle to maintain. I expect this risk to be mine alone.
Since websites are secretive¹ about their reasons for blocking, it has been quite painful (as a Tor user) that many services are not blocking Tor, but in fact blocking old versions of Chrome. WTF? Why does a website block old Chrome versions? My gut thought was the website wants to nanny users, to try to force users to secure themselves (“for the user’s own protection”), because surely a user with old chrome does not realistically threaten the server’s security.
The answer: it turns out that AI bots are strangely using old Chrome versions for a user-agent header. So webmasters are reacting by blocking old versions of Chrome. What I don’t understand is why devs of AI bots would try to masquerade as old Chrome versions.
And to a lesser extent I wonder why webmasters are reckless enough to take this collective-punishment-is-ok stance. It would be like a store owner banning customers wearing blue hats from their shop because some shoplifters were caught wearing blue hats.
¹ Exceptionally, codeberg.org actually respects our dignity enough to preduce an error message that says they are blocking old chrome. But they are the exception.
dgdft@lemmy.world 2 days ago
Yeah, it’s completely normal.
The AI bots often use curl, which is commonly blocked by UA filtering, so they started sidestepping that by setting a fake chrome UA (usually the latest chrome version as of model training cutoff).
You just set a fake UA of your own to fix it.
daveyOsborn@infosec.pub 2 days ago
Unless I am using frogfind.com, which discriminates against recent UAs because it is meant to serve those with old gear.
The UA setting has become relevant to enough instances of enshitification that perhaps we really need a DB that tracks which UA gets the best treatment per website. I would like a browser that chooses the UA string that will minmize enshification. I suspect the answer may be a string that impersonates Google’s crawler would be ideal, particularly with paywalls.
dgdft@lemmy.world 2 days ago
Unless you wanna make life hard on yourself for no good reason, use your real UA where possible and set the closest-to-accurate UA as a site-specific override where needed.
Fake googlebots are extremely easy to spot and will get treated as malicious. Cloudflare will flag you if you’re spoofing browser identity in any obvious way.