Comment on Cloudflare and (friendly!) bots

MrKaplan@lemmy.world ⁨1⁩ ⁨day⁩ ago

• TootSweet@lemmy.world ⁨1⁩ ⁨day⁩ ago

  I’m not using any headless browser or anything. I’m using Go’s “net/http” package from the standard library which just… makes HTTP requests. I’m making requests to endpoints like https://lemmy.world/api/v3/user/login and https://lemmy.world/api/v3/post. The basic process is that every time it runs, it:

  • Gets the community id by name.
  • Logs in (getting the JWT token).
  • And then posts to the community in question (using the JWT token).

  The user agent it uses is Go-http-client/1.1 if that helps any.

  source

  • MrKaplan@lemmy.world ⁨1⁩ ⁨day⁩ ago

    Then I don’t know where you’re seeing Cloudflare CAPTCHAs. There shouldn’t be any on any of the API paths.

    I’ve checked both URLs you mentioned above and only saw a Python user agent getting blocked with a browser integrity check on one of them in our CF logs. I’ve now added a dedicated rule to skip browser integrity checks on all API paths.

    source

    • TootSweet@lemmy.world ⁨1⁩ ⁨day⁩ ago

      Hrm, well, now I’m noticing that I’m getting inconsistent results.

      I keep getting {“error”:“rate_limit_error”} from the login endpoint, and I’ve made… maybe 10 requests to that endpoint over the course of the day? And I’ve had that block this user (like, my @TootSweet account) from being able to log in. But I really don’t think I’m sending any noticeable amount more traffic than I would be without the bot testing.

      But sometimes I get past login and get a Cloudflare page back on another call. (In the unhappy path it tries to DM me, and it’s tried that and failed.)

      Maybe I’m just not waiting long enough between requests? I had it set to wait 3 seconds between requests, but maybe I’ll bump it up to 5.

      source

      • MrKaplan@lemmy.world ⁨1⁩ ⁨day⁩ ago

        You should persist the auth token, rate limits are by IP. They also don’t exclude successful logins currently. This is a limitation in Lemmy. We have currently configured it to allow 5 logins per IP per day, but if you’re lucky you can exceed that, as the consumption is not synchronized between backends at this time.

        source

        • -> View More Comments