robots.txt
is a bit like asking bots to not visit my site; with.htaccess
, you’re not asking
Blocking Bots with Nginx
Submitted 3 weeks ago by poVoq@slrpnk.net to selfhosting@slrpnk.net
https://rknight.me/blog/blocking-bots-with-nginx/
Comments
Zachariah@lemmy.world 3 weeks ago
nek0d3r@lemmy.world 3 weeks ago
My robots.txt has definitely only partially helped. I’ll have to see if I can do the same with my Caddy setup.