Lemmy newb here, not sure if this is right for this /c.

An article I found from someone who hosts their own website and micro-social network, and their experience with web-scraping robots who refuse to respect robots.txt, and how they deal with them.

  • sugar_in_your_tea
    link
    fedilink
    English
    arrow-up
    2
    ·
    23 days ago

    The admin could use a CDN and not worry about it, if it’s just static content.

    • klu9@lemmy.caOP
      link
      fedilink
      English
      arrow-up
      4
      ·
      22 days ago

      I believe using a CDN would defeat the author’s goal of not being reliant on third-party service providers.