Comment on Bots dominate internet activity, account for nearly half of all traffic
elgordio@kbin.social 2 months ago
My CDN bill recently went from about $5 a month to over $200. Turned out it was Tictok’s spider relentlessly scraping the same content over and over again.
It was ignoring robots.txt. In the end I just had to ban their user agent in the CDN config.