bug-wget
[Top][All Lists]
Advanced

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Bug-wget] New to list and program


From: Darshit Shah
Subject: Re: [Bug-wget] New to list and program
Date: Mon, 25 Feb 2019 08:50:29 +0100
User-agent: NeoMutt/20180716

Hi Annette,

That error would happen if the web server you're trying to backup doesn't like
it. One thing you can do to fix it is to throttle your requests, of course,
this will slow down the entire process. In order to do this, use the options:
"--wait=1 --random-wait".

This will cause Wget to wait between 0.5 and 1.5 seconds between each page it
downloads. That should ideally prevent any "429 Too Many Requests" errors.

If it still causes problems, you could attempt to add the following options as
well:
"--retry-on-http-error=429 --wait-retry=20"
This will cause Wget to wait for a few seconds when the server complains before
resuming its job.

You can also pass the "-k -K" options to Wget so that it converts the links in
the downloaded pages to local links. That is, the links will no longer take you
back to the actual website on the web.

When trying again, you should also use "-c" to ask Wget to try and continue the
download rather than doing it all again. It may or may not help, but it can't
hurt.


So, finally the options you need are:

$ wget -m -c -k -K --wait=1 --random-wait <link to the board>

* Crusade 36 <address@hidden> [190224 17:23]:
> Thank you so much for your reply!
> I did check the: 
> Internet Archive 
> 
> The didn't have most of the board backed up, mostly a surface page, then a 
> link to where the page is on the web.
> I did try as you suggested:
> $ wget -m <link to the board>
> It worked wonders,  to start, then:
> connected.
> HTTP request sent, awaiting response... 429 Too Many Requests
> 2019-02-24 10:56:46 ERROR 429: Too Many Requests.
> 
> It saved the surface page, some pages, but then the links would take you back 
> to the actual website on the web. 
> 
> I  was wondering if you had any ideas, on what I might be done next. 
> 
> Thank you so much!
> 
> Annette
>  
> 
>     On Friday, February 22, 2019 5:59 PM, Darshit Shah <address@hidden> wrote:
>  
> 
>  Hi Annette,
> 
> This is absolutely the perfect place for you to ask questions and get help.
> We'd be glad to help you archive your message board.
> 
> With wget, the basic command you should need is:
> 
> $ wget -m <link to the board>
> 
> This will invoke Wget in the mirror mode which tries to make a perfect local
> copy of the website. If it doesn't work to your taste, you can come back to us
> and we'll help you to tweak the options to get it just right.
> 
> However, before you attempt to do this, may I suggest you go through The
> Internet Archive (https://archive.org). They might already have a full backup
> of the website which you can browse and even download locally.
> 
> * Crusade 36 <address@hidden> [190222 23:53]:
> > Hello,
> > I've subscribed to this list, because it said it was for bugs or for 
> > getting help with the program.
> > To be upfront, I was a participant / owner,  for 17 years on a message 
> > board, first Ezboard, then Yuku, now its own by Taptalk, and it was so bad 
> > we moved the board. But there is 17 years of data, role playing, that I 
> > would love to back up. 
> > 
> > 99.9999% of the customization of the board is gone, but the data is still 
> > there:Solaris Humanus
> > 
>
> > |  
> > |  
> > |  
> > |  |    |
> > 
> >    |
> > 
> >  |
> > |  
> > |  |  
> > Solaris Humanus
> >  Solaris Humanus  |  |
> > 
> >  |
> > 
> >  |
> > 
>
> > 
> > But have no clue on how to do it. Which is why I am writing for help, I am 
> > computer illiterate,  I know the basic things, but when I looked at the 
> > document page, how to page, and starting reading at the top, I got so 
> > confused not longer after. 
> > 
> > Help would be appreciated, as I would love to back up the entire site, but 
> > at the same time, if I writing this inappropriate, then please forgive me.
> > 
> > Thank you
> > Annette
> > 
> > 
> > 
> > 
> 
> -- 
> Thanking You,
> Darshit Shah
> PGP Fingerprint: 7845 120B 07CB D8D6 ECE5 FF2B 2A17 43ED A91A 35B6
> 
>    

-- 
Thanking You,
Darshit Shah
PGP Fingerprint: 7845 120B 07CB D8D6 ECE5 FF2B 2A17 43ED A91A 35B6

Attachment: signature.asc
Description: PGP signature


reply via email to

[Prev in Thread] Current Thread [Next in Thread]