I also don’t know if a law that says search engines have to honor a robots.txt file. I guess we will see what happens if Bing or some other service decides to ignore it.
I guess they can make it hard to index by scraping by rate limiting or requiring login to view content etc and only provide Google the api to bypass the restrictions
I guess Reddit is permitted to only let Google index it
Are they though?
I don’t know of any law that says that they can’t.
I also don’t know if a law that says search engines have to honor a robots.txt file. I guess we will see what happens if Bing or some other service decides to ignore it.
You can just require a log in to view content, or just flat out auto ban indexing robots.
deleted by creator
I guess they can make it hard to index by scraping by rate limiting or requiring login to view content etc and only provide Google the api to bypass the restrictions
There’s probably a lot of ways to do it