Comment on What are You Working on Wednesday
dubbel@discuss.tchncs.de 2 years ago
Private project, not really security related: Crawling robots.txts to gather some statistics on which bots people are most often excluding - weirdly I couldn’t find any recent/regularly updated stats on this.
PaddleMaster@beehaw.org 2 years ago
dubbel@discuss.tchncs.de 2 years ago
It started with a popular mastodon posts on how to block openai crawlers I think, and I’d like to know whether people are actually implementing it.