Cynicus Rex@lemmy.ml to Privacy@lemmy.mlEnglish · 3 months agoHow to block AI Crawler Bots using robots.txt filewww.cyberciti.bizexternal-linkmessage-square63fedilinkarrow-up1108arrow-down132
arrow-up176arrow-down1external-linkHow to block AI Crawler Bots using robots.txt filewww.cyberciti.bizCynicus Rex@lemmy.ml to Privacy@lemmy.mlEnglish · 3 months agomessage-square63fedilink
minus-squareJackbyDev@programming.devlinkfedilinkEnglisharrow-up1arrow-down5·3 months agoIt’s better than saying something like “there’s no point in robots.txt because bots can disobey is” though.
minus-squarethanks_shakey_snake@lemmy.calinkfedilinkarrow-up3·3 months agoMaybe? But it’s not like that’s the only alternative thing to say, lol
It’s better than saying something like “there’s no point in robots.txt because bots can disobey is” though.
Maybe? But it’s not like that’s the only alternative thing to say, lol