Issue with multi-tasking in wget2.exe

[Michael D. Setzer II]

Sent a message to Tim Rühsen and got reply from
Tim Rühsen (@rockdaboot) <gitlab@mg.gitlab.com>

That said to sent email to this link.
Have a program that uses wget2 under linux and it has
no problems with using 13 threads to download 13 pages.

Have found that in test with wine it has failures with
threads set above 2, but they vary. Also, did some earlier
test on windows 10 machine, and with native windows it
wasn't getting errors till higher tests, but was doing
whole process manually on windows..

Under Fedora 35 I ran this test script.

for a in 1 2 3 4 5 6 7 8 9 10 11 12 13; do
mkdir $a
cd $a
time wine wget2 --restrict-file-names=windows --max-threads=$a 
--secure-protocol=PFS -q
--base="https://ssb-prod.ec.guamcc.edu/PROD/"; -i ../testlist.sav
cd ..
done
cd 1
for b in 2 3 4 5 6 7 8 9 10 11 12 13; do for a in *; do echo $a; cmp $a 
../$b/$a ; done; done>>../output
2>../output2

Conents of testlist.sav
bwpkedir.P_NameDirectory?dir_indx=&next_row=001&prev_row=000&searchtype=N&ButtonSelected=
bwpkedir.P_NameDirectory?dir_indx=&next_row=026&prev_row=025&searchtype=N&ButtonSelected=
bwpkedir.P_NameDirectory?dir_indx=&next_row=051&prev_row=050&searchtype=N&ButtonSelected=
bwpkedir.P_NameDirectory?dir_indx=&next_row=076&prev_row=075&searchtype=N&ButtonSelected=
bwpkedir.P_NameDirectory?dir_indx=&next_row=101&prev_row=100&searchtype=N&ButtonSelected=
bwpkedir.P_NameDirectory?dir_indx=&next_row=126&prev_row=125&searchtype=N&ButtonSelected=
bwpkedir.P_NameDirectory?dir_indx=&next_row=151&prev_row=150&searchtype=N&ButtonSelected=
bwpkedir.P_NameDirectory?dir_indx=&next_row=176&prev_row=175&searchtype=N&ButtonSelected=
bwpkedir.P_NameDirectory?dir_indx=&next_row=201&prev_row=200&searchtype=N&ButtonSelected=
bwpkedir.P_NameDirectory?dir_indx=&next_row=226&prev_row=225&searchtype=N&ButtonSelected=
bwpkedir.P_NameDirectory?dir_indx=&next_row=251&prev_row=250&searchtype=N&ButtonSelected=
bwpkedir.P_NameDirectory?dir_indx=&next_row=276&prev_row=275&searchtype=N&ButtonSelected=
bwpkedir.P_NameDirectory?dir_indx=&next_row=301&prev_row=300&searchtype=N&ButtonSelected=

The times don't vary much with 1 task taking about 7
seconds, and 2 being about 6. Above 2 the times tend to
take longer, except for the ones with failures where those
pages are not downloded completely.

The output2 shows these errors. 1, 2, 4, and 5 seemed to
download all files the same, but others had differences.
Only ran once, so not sure if results would always be
same. Running under linux files are always identicle
regardless of multi-task.

cmp: EOF on
../3/bwpkedir.P_NameDirectory%3Fdir_indx=&next_row=026&prev_row=025&searchtype=N&ButtonSelected=
after byte 24464, in line 544
cmp: EOF on
../6/bwpkedir.P_NameDirectory%3Fdir_indx=&next_row=001&prev_row=000&searchtype=N&ButtonSelected=
after byte 15816, in line 322
cmp: EOF on
../7/bwpkedir.P_NameDirectory%3Fdir_indx=&next_row=026&prev_row=025&searchtype=N&ButtonSelected=
after byte 15816, in line 319
cmp: EOF on
../8/bwpkedir.P_NameDirectory%3Fdir_indx=&next_row=001&prev_row=000&searchtype=N&ButtonSelected=
after byte 8080, in line 125
cmp: EOF on
../8/bwpkedir.P_NameDirectory%3Fdir_indx=&next_row=126&prev_row=125&searchtype=N&ButtonSelected=
after byte 8080, in line 121
cmp: EOF on
../9/bwpkedir.P_NameDirectory%3Fdir_indx=&next_row=201&prev_row=200&searchtype=N&ButtonSelected=
after byte 32656, in line 756
cmp: EOF on
../10/bwpkedir.P_NameDirectory%3Fdir_indx=&next_row=076&prev_row=075&searchtype=N&ButtonSelected=
after byte 8080, in line 125
cmp: EOF on
../10/bwpkedir.P_NameDirectory%3Fdir_indx=&next_row=176&prev_row=175&searchtype=N&ButtonSelected=
after byte 8080, in line 125
cmp: EOF on
../11/bwpkedir.P_NameDirectory%3Fdir_indx=&next_row=126&prev_row=125&searchtype=N&ButtonSelected=
after byte 8080, in line 121
cmp:
../11/bwpkedir.P_NameDirectory%3Fdir_indx=&next_row=176&prev_row=175&searchtype=N&ButtonSelected=:
No such file or directory
cmp: EOF on
../12/bwpkedir.P_NameDirectory%3Fdir_indx=&next_row=126&prev_row=125&searchtype=N&ButtonSelected=
after byte 8080, in line 121
cmp: EOF on
../12/bwpkedir.P_NameDirectory%3Fdir_indx=&next_row=276&prev_row=275&searchtype=N&ButtonSelected=
after byte 8080, in line 118
cmp: EOF on
../13/bwpkedir.P_NameDirectory%3Fdir_indx=&next_row=026&prev_row=025&searchtype=N&ButtonSelected=
after byte 15816, in line 319
cmp: EOF on
../13/bwpkedir.P_NameDirectory%3Fdir_indx=&next_row=126&prev_row=125&searchtype=N&ButtonSelected=
after byte 32656, in line 740
cmp: EOF on
../13/bwpkedir.P_NameDirectory%3Fdir_indx=&next_row=201&prev_row=200&searchtype=N&ButtonSelected=
after byte 16211, in line 335

Windows 10 machine is current disconnected, so haven't
done tests on a native windows. Last time, the errors
didn't show till higher numbers. Once got up to 10
threads with no errors, but then ran again at 10, and had
errors.
With the windows native it was a manual process since
scripting doesn't work as nice.



+------------------------------------------------------------+
 Michael D. Setzer II - Computer Science Instructor (Retired)
 mailto:mikes@guam.net
 mailto:msetzerii@gmail.com
 Guam - Where America's Day Begins
 G4L Disk Imaging Project maintainer
 http://sourceforge.net/projects/g4l/
+------------------------------------------------------------+