We have paused all crawling as of Feb 6th, 2025 until we implement robots.txt support. Stats will not update during this period.

  • JustAnotherKay@lemmy.worldEnglish
    1·
    20 hours ago

    The point was “don’t add another bot into the pile of millions of bots that hit people’s servers every day unless you’re gonna be polite about it”