bathalumang_peppa@lemmy.world to Technology@lemmy.worldEnglish · 2 years agoApple considered switching to DuckDuckGo from Google for Safari - Bloomberg Newswww.reuters.comexternal-linkmessage-square291fedilinkarrow-up11.2Karrow-down122cross-posted to: [email protected]
arrow-up11.17Karrow-down1external-linkApple considered switching to DuckDuckGo from Google for Safari - Bloomberg Newswww.reuters.combathalumang_peppa@lemmy.world to Technology@lemmy.worldEnglish · 2 years agomessage-square291fedilinkcross-posted to: [email protected]
minus-squaredarth_helmetlinkfedilinkEnglisharrow-up2·2 years agoIt’s conventional to do this, but a user agent string is entirely up to the client, and robots.txt is just a suggestion. So for the best results, you probably want to mock Google’s crawler because it’s suicidal to block that if you want search traffic.
It’s conventional to do this, but a user agent string is entirely up to the client, and robots.txt is just a suggestion.
So for the best results, you probably want to mock Google’s crawler because it’s suicidal to block that if you want search traffic.