[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: [Bug-wget] Async webcrawling
|
From: |
Tim Rühsen |
|
Subject: |
Re: [Bug-wget] Async webcrawling |
|
Date: |
Tue, 31 Jul 2018 19:22:01 +0200 |
|
User-agent: |
Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Thunderbird/52.9.1 |
On 31.07.2018 18:39, James Read wrote:
> Hi,
>
> how much work would it take to convert wget into a fully fledged
> asynchronous webcrawler?
>
> I was thinking something like using select. Ideally, I want to be able to
> supply wget with a list of starting point URLs and then for wget to crawl
> the web from those starting points in an asynchronous fashion.
>
> James
>
Just use wget2. It is already packaged in Debian sid.
To build from git source, see https://gitlab.com/gnuwget/wget2.
To build from tarball (much easier), download from
https://alpha.gnu.org/gnu/wget/wget2-1.99.1.tar.gz.
Regards, Tim
signature.asc
Description: OpenPGP digital signature