Re: [Bug-wget] Invoking wget via cli v. python3

[Darshit Shah]

Hi Pierre,

Could you please share the Python script you wrote? We have a functioning
Python 3 script that invokes Wget with whatever parameters you want. It's a
part of the Test Suite that currently resides in the parallel-wget branch.

Check out the code in testenv/WgetTest.py. It's not the cleanest code, and
having all the various features didn't help either. But you'll get a gist
of how to invoke Wget through Python.

I'm not completely sure of the problem you are facing with your invokation
of Wget. The exact command you provided and wgetrc file cause Wget to work
perfectly for me, just as described by you. If you were to share your
Python script it'd help in debugging to find the problem a little better.



On Tue, Feb 18, 2014 at 10:57 PM, Pierre Fortin <address@hidden> wrote:

> Hi all,       [pulling my hair out over this one...]
>
> I have a python3 script that builds the .wgetrc* file and invokes wget...
> it even prints out the command as invoked...  should be easy, right?
>
> If I issue the generated/printed command via cli, it works exactly as
> expected.
>
> However, when python3 issues the command, all looks well until it gets
> and saves .listing, then....  it removes and replaces it with index.html
> and silently quits...
>
> Under what conditions would .listing get written, erased and replaced by
> an index.html file, then, wget quits?
>
> Thanks for any clue stick,
> Pierre
>
>
> =========== details follow ===========
>
>
> The abridged output looks like this:
>
> $ getMageia2.1 -d
> Processing /home/pfortin/bin/getMageia2.d/accept...
> Processing /home/pfortin/bin/getMageia2.d/reject...
> Ignoring missing file: /home/pfortin/bin/getMageia2.d/include_directories
> Processing /home/pfortin/bin/getMageia2.d/exclude_directories...
> wgetdebug = --debug
>
>   Executing command:
>     wget --debug --config=/home/pfortin/.wgetrc.getMageia2.x86_64.4
>   --no-host-directories ftp://mirror.nexcess.net/mageia/distrib/4/x86_64/
>
> Retrieving files...
> Setting --config (chooseconfig)
>   to /home/pfortin/.wgetrc.getMageia2.x86_64.4
> Setting --host-directories (addhostdir) to 0
> DEBUG output created by Wget 1.14 on linux-gnu.
> URI encoding = 'UTF-8'
> Using '.listing' as listing tmp file.
> [...]
> Created socket 4.
> done.    ==> LIST ...
> --> LIST -a
> [...]
> 226 Transfer complete
> 2014-02-18 16:14:06 (224 KB/s) - '.listing' saved [939]
> [...file & perms listed]
> Removed '.listing'.                                  <======= ???
> Wrote HTML-ized index to 'index.html' [1807].        <======= ???
> Closed fd 3
>
>
> The generated .wgetrc* file contains accept/reject rules and this:
> mirror = on
> remove-listing = off
> no_parent = on
> add_hostdir = off
> dir_prefix =
> netrc = off
> recursive = on
> cut-dirs=4
> [...rules here]
>
>
>
> $ wget -V
> GNU Wget 1.14 built on linux-gnu.
>
> +digest +https +ipv6 +iri +large-file +nls +ntlm +opie +ssl/openssl
>
> Wgetrc:
>     /etc/wgetrc (system)
> Locale: /usr/share/locale
> Compile: gcc -DHAVE_CONFIG_H -DSYSTEM_WGETRC="/etc/wgetrc"
>     -DLOCALEDIR="/usr/share/locale" -I. -I../lib -I../lib -O2 -g -pipe
>     -Wformat -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2
>     -fstack-protector --param=ssp-buffer-size=4 -fPIC
> Link: gcc -O2 -g -pipe -Wformat -Werror=format-security
>     -Wp,-D_FORTIFY_SOURCE=2 -fstack-protector --param=ssp-buffer-size=4
>     -fPIC -Wl,--as-needed -Wl,--no-undefined -Wl,-z,relro -Wl,-O1
>     -Wl,--build-id -Wl,--enable-new-dtags /usr/lib64/libssl.so
>     /usr/lib64/libcrypto.so /usr/lib64/libz.so -ldl -lz -lz -lidn
>     ftp-opie.o openssl.o http-ntlm.o ../lib/libgnu.a
>
>
>


-- 
Thanking You,
Darshit Shah