Comment on Bots dominate internet activity, account for nearly half of all traffic

elgordio@kbin.social ⁨8⁩ ⁨months⁩ ago

My CDN bill recently went from about $5 a month to over $200. Turned out it was Tictok’s spider relentlessly scraping the same content over and over again.

It was ignoring robots.txt. In the end I just had to ban their user agent in the CDN config.

source
Sort:hotnewtop