
Grateful: Internet archive slow download speed
Internet archive slow download speed | 843 |
Internet archive slow download speed | 905 |
Internet archive slow download speed | 34 |
Internet archive slow download speed | 49 |
hartator / wayback-machine-downloader
Generally speaking, it's good etiquette to crawl slowly. You want to avoid hurting the Internet Archive's servers by overloading them with too many requests for too much data in too little time. It could interfere with their normal operations, e.g. serving snapshots to actual humans via the Wayback Machine. If this happens too often, it might prompt them to take measures to block downloaders such as this one.
That said, I don't know the Internet Archive's stance on people mass-scraping their snapshots, nor the capabilities of their infrastructure. It might be the case that they are perfectly fine with all of this. But generally, you don't want your scrapers to put unusual load on people's servers. That kind of behavior can get you blocked, and in the long run, might contribute to anti-scraping legislation.
0 thoughts to “Internet archive slow download speed”