Furbland@lemmy.world to 196@lemmy.blahaj.zone · 1 month agorulebots.txtlemmy.worldimagemessage-square33fedilinkarrow-up1321arrow-down10
arrow-up1321arrow-down1imagerulebots.txtlemmy.worldFurbland@lemmy.world to 196@lemmy.blahaj.zone · 1 month agomessage-square33fedilink
minus-squaremorgunkorn@discuss.tchncs.delinkfedilinkarrow-up43·1 month agoThat’s the catch, Google made a deal with Reddit and remains the only search engine allowed to access its data for indexing. It cuts off every other search engine
minus-squarevortic@lemmy.worldlinkfedilinkarrow-up27·1 month agoTell me that there is an anti trust suit over this.
minus-squareFurbland@lemmy.worldOPlinkfedilinkarrow-up26·1 month agoThere’s a suit over google in general so this may well be part of it
minus-squareTriflingToad@lemmy.worldlinkfedilinkarrow-up3·1 month agoreally? ddg will show me reddit links, did they have to make a webscraper or something
minus-squaremorgunkorn@discuss.tchncs.delinkfedilinkarrow-up4·1 month agoThere’s a cutoff date, anything indexed before the robots.txt was changed stays in the index
That’s the catch, Google made a deal with Reddit and remains the only search engine allowed to access its data for indexing. It cuts off every other search engine
Tell me that there is an anti trust suit over this.
There’s a suit over google in general so this may well be part of it
really? ddg will show me reddit links, did they have to make a webscraper or something
There’s a cutoff date, anything indexed before the robots.txt was changed stays in the index