skilledtothegills@lemmy.world to Technology@lemmy.worldEnglish · 1 year agoX updates its Terms to prohibit crawling/scraping of its datastackdiary.comexternal-linkmessage-square39fedilinkarrow-up1339arrow-down121file-textcross-posted to: hakendran@feddit.dehackernews@derp.foo
arrow-up1318arrow-down1external-linkX updates its Terms to prohibit crawling/scraping of its datastackdiary.comskilledtothegills@lemmy.world to Technology@lemmy.worldEnglish · 1 year agomessage-square39fedilinkfile-textcross-posted to: hakendran@feddit.dehackernews@derp.foo
minus-squaredingleberry@discuss.tchncs.delinkfedilinkEnglisharrow-up15arrow-down2·1 year agoJust update robot.txt coward!
minus-squareIwasondigg@lemmy.onelinkfedilinkEnglisharrow-up6·1 year agoTook a look at their robots.txt, it appears to block all bots except Google.
Just update robot.txt coward!
Took a look at their robots.txt, it appears to block all bots except Google.