They’re spamming all web logs too with an advertisement for their services in the user agent. I decided to ban them from all my websites because the logs took up too much space.
They’re saying who they are, what they do, and are linking to their website and sometimes sends hundreds of requests in a minute. It might not say "For only €49.99 you can get your very own thing!”, but that does not mean they aren’t throwing their name up in every website owner’s arse whether they like it or not.
That could have been done by just having a single entry called GoogleBot or BingBot, not an entire sentence explaining their product offering let alone hundreds of times a minute.
They’re spamming all web logs too with an advertisement for their services in the user agent. I decided to ban them from all my websites because the logs took up too much space.
Yeah fuck them.
lmao that’s not an ad, dude.
They’re saying who they are, what they do, and are linking to their website and sometimes sends hundreds of requests in a minute. It might not say "For only €49.99 you can get your very own thing!”, but that does not mean they aren’t throwing their name up in every website owner’s arse whether they like it or not.
It’s so you know who is scanning you.
That could have been done by just having a single entry called GoogleBot or BingBot, not an entire sentence explaining their product offering let alone hundreds of times a minute.