Comment on Trend of websites blocking old versions of Chrome -- is this sensible?
dgdft@lemmy.world 2 days ago
Yeah, it’s completely normal.
The AI bots often use curl, which is commonly blocked by UA filtering, so they started sidestepping that by setting a fake chrome UA (usually the latest chrome version as of model training cutoff).
You just set a fake UA of your own to fix it.
daveyOsborn@infosec.pub 2 days ago
Unless I am using frogfind.com, which discriminates against recent UAs because it is meant to serve those with old gear.
The UA setting has become relevant to enough instances of enshitification that perhaps we really need a DB that tracks which UA gets the best treatment per website. I would like a browser that chooses the UA string that will minmize enshification. I suspect the answer may be a string that impersonates Google’s crawler would be ideal, particularly with paywalls.
dgdft@lemmy.world 2 days ago
Unless you wanna make life hard on yourself for no good reason, use your real UA where possible and set the closest-to-accurate UA as a site-specific override where needed.
Fake googlebots are extremely easy to spot and will get treated as malicious. Cloudflare will flag you if you’re spoofing browser identity in any obvious way.
daveyOsborn@infosec.pub 2 days ago
I’m not sure you understand the transparency problem. Codeberg is the /only/ service that I have ever encountered that honestly, transparently states that they blocked me on the basis of old chrome. All other blocks on the order of 10s or 100s per day give no reason. As a Tor user, I assume Tor is the problem. But now there are increasingly non-Tor criteria emerging. I cannot investigate every blockade. Hence why establishing a UA string db would be useful, which could then be incorporated into browsers.
I boycott Cloudflare. CF already blocks most of my access attempts and I am grateful for it. What disturbs me is when I discover that I am on a CF website that whitelisted Tor because such cases could lead to me accidentally patronising a Cloudflare supporter. I know there are plugins that would help me avoid those instances but if CF has another trigger to block me from their side, then it is mutually appreciated.
frongt@lemmy.zip 1 day ago
If your point is transparency, and the given justification is to block scrapers, then it makes sense they wouldn’t be transparent about it. You don’t want to give your adversary instructions about how to bypass your security.
dgdft@lemmy.world 2 days ago