• 0 Posts
  • 17 Comments
Joined 1 year ago
cake
Cake day: October 26th, 2023

help-circle












  • WGET is awesome, I have scraped tons with it. So many options, you can even spoof all the request header info to get around sites that try to limit auto downloaders. Here is the manual: https://www.gnu.org/software/wget/manual/wget.html

    1. webp or any file extension will work. (note on webp, most sites actually have jpgs still, but convert and serve webp to save bandwidth if the browser says it accepts them. There is a header you can disable in firefox to not accept webp unless it is the only option:

    https://addons.mozilla.org/en-CA/firefox/addon/dont-accept-webp/

    Wget is not behaving identically to a browser so im unsure what this part of the request looks like or if it needs modification. If it isnt working let me know.

    1. 5 might be enough, but maybe not. Scroll down in my first link comments, they show how to set to infinite: “-l inf”.

    For future scraping, look at the mirror command. It sets recursion to infinite and will make a full copy of the site. You can also use the --convert-links option, which changes all the links to point to the locally downloaded files. It then behaves the same as the real website.

    You cant go too deep unless you use --span-hosts, it can grab external files from different domains to make the mirrored site a true copy, but yea, you often don’t need that. You also want to be more careful with recursive depth here - it can go too deep and you end up with too much data.

    1. I’m not sure about this. I think you can turn on logging, but I’m not sure what that gets you. I’ve used the no-clobber command to run wget again, without re-downloading existing files. This is handy for resuming or filling in gaps that were missed due to timeout, etc.

    Some sites also need to use the wait or random-wait command to avoid detection.




  • plunki@alien.topBtoData Hoarder@selfhosted.forumSSD vs. USB
    link
    fedilink
    English
    arrow-up
    1
    ·
    1 year ago

    External SSD is way better than USB key. I just got a Samsung T7 and compared to name brand keys it is far superior. Both the controller and the chips are better quality. The speed is consistently fast compared with keys that always seem to slow down / hang / fluctuate on big transfers. I don’t know what the actual stats hare but have heard SSD just has higher IOPS