Comment on Google is redesigning its search engine — and it’s AI all the way down
Bishma@discuss.tchncs.de 6 months agoWhen I was running a site, I had special rules in my firewall to look for things that said they were googlebot but which didn’t come from one of googles published public IPs.
ptz@dubvee.org 6 months ago
…yeah.
I had that thought after I replied when I realized that most of the reputable search crawlers will publish the IPs/ranges they use in addition to the UA. The disreputable ones (cough Bytedance cough Xiaomi cough) will just spoof Chrome on Windows 10.
That said, I may still continue looking into working with one of the actual search engines (vs meta search) and see how well that works.