• darth_helmet
    link
    fedilink
    English
    arrow-up
    2
    ·
    1 year ago

    It’s conventional to do this, but a user agent string is entirely up to the client, and robots.txt is just a suggestion.

    So for the best results, you probably want to mock Google’s crawler because it’s suicidal to block that if you want search traffic.