Comment

Comment on Cloudflare and (friendly!) bots

I’m not using any headless browser or anything. I’m using Go’s “net/http” package from the standard library which just… makes HTTP requests. I’m making requests to endpoints like https://lemmy.world/api/v3/user/login and https://lemmy.world/api/v3/post. The basic process is that every time it runs, it:

Gets the community id by name.
Logs in (getting the JWT token).
And then posts to the community in question (using the JWT token).

The user agent it uses is Go-http-client/1.1 if that helps any.

source

Sort:hotnew top

MrKaplan@lemmy.world ⁨1⁩ ⁨day⁩ ago
Then I don’t know where you’re seeing Cloudflare CAPTCHAs. There shouldn’t be any on any of the API paths.

I’ve checked both URLs you mentioned above and only saw a Python user agent getting blocked with a browser integrity check on one of them in our CF logs. I’ve now added a dedicated rule to skip browser integrity checks on all API paths.

source
- TootSweet@lemmy.world ⁨1⁩ ⁨day⁩ ago
  Hrm, well, now I’m noticing that I’m getting inconsistent results.
  
  I keep getting {“error”:“rate_limit_error”} from the login endpoint, and I’ve made… maybe 10 requests to that endpoint over the course of the day? And I’ve had that block this user (like, my @TootSweet account) from being able to log in. But I really don’t think I’m sending any noticeable amount more traffic than I would be without the bot testing.
  
  But sometimes I get past login and get a Cloudflare page back on another call. (In the unhappy path it tries to DM me, and it’s tried that and failed.)
  
  Maybe I’m just not waiting long enough between requests? I had it set to wait 3 seconds between requests, but maybe I’ll bump it up to 5.
  
  source
  - MrKaplan@lemmy.world ⁨1⁩ ⁨day⁩ ago
    You should persist the auth token, rate limits are by IP. They also don’t exclude successful logins currently. This is a limitation in Lemmy. We have currently configured it to allow 5 logins per IP per day, but if you’re lucky you can exceed that, as the consumption is not synchronized between backends at this time.
    
    source
    TootSweet@lemmy.world ⁨1⁩ ⁨day⁩ ago
    Ok. That helps a lot. Thanks much! I’ll take that back to my drawing board and I’d imagine that’ll fix things for me.
    
    source