[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
[Bug-wget] Async webcrawling
|
From: |
James Read |
|
Subject: |
[Bug-wget] Async webcrawling |
|
Date: |
Tue, 31 Jul 2018 17:39:47 +0100 |
Hi,
how much work would it take to convert wget into a fully fledged
asynchronous webcrawler?
I was thinking something like using select. Ideally, I want to be able to
supply wget with a list of starting point URLs and then for wget to crawl
the web from those starting points in an asynchronous fashion.
James
- [Bug-wget] Async webcrawling,
James Read <=