Re: [Bug-wget] wget ignores --mirror option on openWrt's Luci interface

[Olivier Diotte]

On Thu, Apr 11, 2013 at 5:19 AM, Tim Ruehsen <address@hidden> wrote:
> Hi Olivier,
>
> I got openWRT running.
>
> And I can reproduce the problem.
>
> Wget -r seems to miss some <a href URLs.
>
> I have a conference right now which might take the whole day...
>
> But maybe someone else could have a quick look at the html file.
>
> It is (attached)
> 192.168.1.1/cgi-
> bin/luci/;stok=5ddf8fa64dbdd8ffb5c878e8d2339567/admin/network/index.html
>
> and it contains e.g.
> <a href="/cgi-
> bin/luci/;stok=5ddf8fa64dbdd8ffb5c878e8d2339567/admin/network/routes/">S
> tatic Routes</a>
>
> but wget doesn't recognize/download that URL.
>
> Regards, Tim

Hi Tim,

Thanks for your interest in my report.

I am not sure whether my original report was correct though: I
originally thought wget missed href URLs, but I now think my problem
is with the authentication.
Attached is the simple script I use to do my tests. Based on those
tests (and having not had a look at wget's code yet) here is what I
gather:
-The --save-cookies invocation creates the cookies.txt file (which, as
far as I can tell, contains the correct cookie information) and the
./index.html file which is a logged-in /cgi-bin/luci/index.html file
with the href tags and all
-The --load-cookies invocation doesn't use the ./index.html file (it
can be deleted prior without changing the behaviour) and it creates a
hierarchy in the "172.16.1.1" folder (or whatever your router's
hostname is)
-All files downloaded by the --load-cookies invocation seem to be the
login page and the requisite files of that page
-It seems not to be possible to use any combination of '-l', '-r' or
'-m' 'with the '--post-data' option (with or without the
--save-cookies option)

Now, that is the behaviour I get on
address@hidden:~/Downloads/wget-1.14/foo/usr/local/bin,0$ ./wget --version
GNU Wget 1.14 built on linux-gnu.

+digest +https +ipv6 -iri +large-file +nls -ntlm +opie +ssl/gnutls

Wgetrc:
    /usr/local/etc/wgetrc (system)
Locale: /usr/local/share/locale
Compile: gcc -DHAVE_CONFIG_H -DSYSTEM_WGETRC="/usr/local/etc/wgetrc"
    -DLOCALEDIR="/usr/local/share/locale" -I. -I../lib -I../lib -O2
    -Wall
Link: gcc -O2 -Wall -lgnutls -lgcrypt -lgpg-error -lz -lz -lrt ftp-opie.o
    gnutls.o ../lib/libgnu.a

Copyright (C) 2011 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later
<http://www.gnu.org/licenses/gpl.html>.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Originally written by Hrvoje Niksic <address@hidden>.
Please send bug reports and questions to <address@hidden>.


Do you get the same behaviour on your end? Or do you have a
combination of options which allow you to get the target/main page of
a --load-cookies invocation to download correctly?

I have saved wireshark logs of Firefox and wget traffic. I'll report
any finding regarding those.

Regards,
Olivier

run.sh
Description: Bourne shell script