>From 7fef56c4f815359047347eee4356b9f8971f1df5 Mon Sep 17 00:00:00 2001 From: Matthew White Date: Sun, 21 Aug 2016 18:45:09 +0200 Subject: [PATCH 11/25] New: Metalink/XML and Metalink/HTTP file naming safety rules * src/metalink.h: Add declaration of function append_suffix_number() * src/metalink.c: Add function append_suffix_number() append number to string * src/metalink.c (retrieve_from_metalink): Safer Metalink/XML and Metalink/HTTP download naming system, opt.trustservernames based * doc/metalink-standard.txt: Update doc. Explain new Metalink/XML and Metalin/HTTP download naming system and --trust-server-names role * testenv/Makefile.am: Add new files * testenv/Test-metalink-xml.py: Update test. Metalink/XML naming tests * testenv/Test-metalink-xml-trust.py: New file. Metalink/XML naming tests with --trust-server-names * testenv/Test-metalink-xml-abspath.py: Update test. Metalink/XML absolute path tests * testenv/Test-metalink-xml-abspath-trust.py: New file. Metalink/XML absolute path tests with --trust-server-names * testenv/Test-metalink-xml-relpath.py: Update test. Metalink/XML relative path tests * testenv/Test-metalink-xml-relpath-trust.py: New file. Metalink/XML relative path tests with --trust-server-names * testenv/Test-metalink-xml-homepath.py: Update test. Metalink/XML home path and ~ (tilde) tests * testenv/Test-metalink-xml-homepath-trust.py: New file. Metalink/XML home path and ~ (tilde) tests with --trust-server-names * testenv/Test-metalink-xml-prefix.py: New file. Metalink/XML naming tests with --directory-prefix * testenv/Test-metalink-xml-prefix-trust.py: New file. Metalink/XML naming tests with --directory-prefix and --trust-server-names * testenv/Test-metalink-xml-absprefix.py: New file. Metalink/XML absolute --directory-prefix tests * testenv/Test-metalink-xml-absprefix-trust.py: New file. Metalink/XML absolute --directory-prefix tests with --trust-server-names * testenv/Test-metalink-xml-relprefix.py: New file. Metalink/XML relative --directory-prefix tests * testenv/Test-metalink-xml-relprefix-trust.py: New file. Metalink/XML relative --directory-prefix tests with --trust-server-names * testenv/Test-metalink-xml-homeprefix.py: New file. Metalink/XML home --directory-prefix tests * testenv/Test-metalink-xml-homeprefix-trust.py: New file. Metalink/XML home --directory-prefix tests with --trust-server-names The option --trust-server-names allows to use the file names parsed from a Metalink/XML file. Without --trust-server-names, the safety mechanism provides secure and predictable file names. --- doc/metalink-standard.txt | 59 ++++++-- src/metalink.c | 107 +++++++++++--- src/metalink.h | 1 + testenv/Makefile.am | 14 +- testenv/Test-metalink-xml-abspath-trust.py | 130 +++++++++++++++++ testenv/Test-metalink-xml-abspath.py | 61 ++++++-- testenv/Test-metalink-xml-absprefix-trust.py | 194 +++++++++++++++++++++++++ testenv/Test-metalink-xml-absprefix.py | 194 +++++++++++++++++++++++++ testenv/Test-metalink-xml-homepath-trust.py | 195 +++++++++++++++++++++++++ testenv/Test-metalink-xml-homepath.py | 126 ++++++++++++++-- testenv/Test-metalink-xml-homeprefix-trust.py | 194 +++++++++++++++++++++++++ testenv/Test-metalink-xml-homeprefix.py | 194 +++++++++++++++++++++++++ testenv/Test-metalink-xml-prefix-trust.py | 194 +++++++++++++++++++++++++ testenv/Test-metalink-xml-prefix.py | 194 +++++++++++++++++++++++++ testenv/Test-metalink-xml-relpath-trust.py | 193 +++++++++++++++++++++++++ testenv/Test-metalink-xml-relpath.py | 126 ++++++++++++---- testenv/Test-metalink-xml-relprefix-trust.py | 194 +++++++++++++++++++++++++ testenv/Test-metalink-xml-relprefix.py | 194 +++++++++++++++++++++++++ testenv/Test-metalink-xml-trust.py | 197 ++++++++++++++++++++++++++ testenv/Test-metalink-xml.py | 126 ++++++++++++++-- 20 files changed, 2803 insertions(+), 84 deletions(-) create mode 100755 testenv/Test-metalink-xml-abspath-trust.py create mode 100755 testenv/Test-metalink-xml-absprefix-trust.py create mode 100755 testenv/Test-metalink-xml-absprefix.py create mode 100755 testenv/Test-metalink-xml-homepath-trust.py create mode 100755 testenv/Test-metalink-xml-homeprefix-trust.py create mode 100755 testenv/Test-metalink-xml-homeprefix.py create mode 100755 testenv/Test-metalink-xml-prefix-trust.py create mode 100755 testenv/Test-metalink-xml-prefix.py create mode 100755 testenv/Test-metalink-xml-relpath-trust.py create mode 100755 testenv/Test-metalink-xml-relprefix-trust.py create mode 100755 testenv/Test-metalink-xml-relprefix.py create mode 100755 testenv/Test-metalink-xml-trust.py diff --git a/doc/metalink-standard.txt b/doc/metalink-standard.txt index d00c384..18acaaa 100644 --- a/doc/metalink-standard.txt +++ b/doc/metalink-standard.txt @@ -29,18 +29,61 @@ paths or descend/escalate to a relative path unexpectedly. 2.1 Metalink/XML implemented tests ================================== -* testenv/Test-metalink-xml.py: Accept safe paths -* testenv/Test-metalink-xml-abspath.py: Reject absolute paths -* testenv/Test-metalink-xml-relpath.py: Reject relative paths -* testenv/Test-metalink-xml-homepath.py: Reject home paths +See testenv/Makefile.am (METALINK_TESTS). + +2.2 Metalink/HTTP implemented tests +=================================== + +See testenv/Makefile.am (METALINK_TESTS). 3. Download file name ********************* -Computing the file name to wrote from the followed urls only leads to -uncertainty. Reason why an unique name shall be used. Respectively, it -shall be the metalink:file "name" field for Metalink/XML and a derived -cli's url for Metalink/HTTP. +The download file name shall be decided by precise rules which prevent +any naming uncertainty and security issues. + +3.1 Naming rules +================ + +The final name of downloaded files is computed starting from a trusted +name, which is then combined with the "Directory Options". The result +is verified and eventually made safer following security rules. If the +final name isn't found safe enough, then the file isn't downloaded. + +Depending on the options used, a suffix could be appended to the final +name to not overwrite existing files. + +3.1.1 The trusted name +====================== + +The option --trust-server-names decides what is the trusted name. + +Any Metalink/XML element with an unsafe metalink:file "name" field is +ignored, see '1. Security features'. + +3.1.1.1 Without --trust-server-names +==================================== + +When --trust-server-names is off, the basename of the --input-metalink +file, if available, or of the mother URL is trusted. + +The files described by a Metalink/XML file will be named sequentially +applying a suffix to the trusted name. + +3.1.1.2 With --trust-server-names +================================= + +When --trust-server-names is on, the metalink:file "name" field parsed +from Metalink/XML files is trusted. When no Metalink/XML is available, +the mother URL is trusted. + +3.1.2 The final name +==================== + +The "Directory Options" are combined with the trusted name. The result +is evaluated again by the '1. Security features'. If the path is found +unsafe, only the basename of the final name is considered. If this is +found unsafe too, the file is not downloaded. 4. Metalink/XML *************** diff --git a/src/metalink.c b/src/metalink.c index a8b047b..356a68b 100644 --- a/src/metalink.c +++ b/src/metalink.c @@ -68,7 +68,16 @@ retrieve_from_metalink (const metalink_t* metalink) bool _output_stream_regular = output_stream_regular; char *_output_document = opt.output_document; - DEBUGP (("Retrieving from Metalink\n")); + /* metalink file counter */ + unsigned mfc = 0; + + /* metalink retrieval type */ + const char *metatpy = metalink->origin ? "Metalink/HTTP" : "Metalink/XML"; + + /* metalink mother source */ + char *metasrc = metalink->origin ? metalink->origin : opt.input_metalink; + + DEBUGP (("Retrieving from Metalink %s\n", quote (metasrc))); /* No files to download. */ if (!metalink->files) @@ -86,6 +95,8 @@ retrieve_from_metalink (const metalink_t* metalink) { metalink_file_t *mfile = *mfile_ptr; metalink_resource_t **mres_ptr; + char *planname = NULL; + char *trsrname = NULL; char *filename = NULL; char *basename = NULL; char *safename = NULL; @@ -103,30 +114,76 @@ retrieve_from_metalink (const metalink_t* metalink) output_stream = NULL; + mfc++; + /* The directory prefix for opt.metalink_over_http is handled by src/url.c (url_file_name), do not add it a second time. */ if (!metalink->origin && opt.dir_prefix && strlen (opt.dir_prefix)) - filename = aprintf("%s/%s", opt.dir_prefix, mfile->name); + planname = aprintf("%s/%s", opt.dir_prefix, mfile->name); else - filename = xstrdup (mfile->name); + planname = xstrdup (mfile->name); - DEBUGP (("Processing metalink file %s...\n", quote (mfile->name))); + /* With Metalink/HTTP, trust the metalink file name (from cli). + With --trust-server-names, trust the Metalink/XML file name, + otherwise, use the basename of --input-metalink followed by + the metalink file counter as suffix. */ + if (metalink->origin || opt.trustservernames) + { + trsrname = xstrdup (mfile->name); + } + else + { + trsrname = xstrdup (get_metalink_basename (opt.input_metalink)); + append_suffix_number (&trsrname, ".#", mfc); + } + + /* Add the directory prefix for opt.input_metalink. */ + if (!metalink->origin && opt.dir_prefix && strlen (opt.dir_prefix)) + filename = aprintf("%s/%s", opt.dir_prefix, trsrname); + else + filename = xstrdup (trsrname); /* Enforce libmetalink's metalink_check_safe_path(). */ basename = get_metalink_basename (filename); safename = metalink_check_safe_path (filename) ? filename : basename; - if (filename != safename) - logprintf (LOG_NOTQUIET, - _("Unsafe metalink file %s. Stripping directory...\n"), - quote (filename)); - - if (!basename) + DEBUGP (("Processing metalink file %s...\n", quote (mfile->name))); + DEBUGP (("\n")); + DEBUGP ((" %s\n", metatpy)); + DEBUGP (("\n")); + DEBUGP ((" --trust-server-names %s\n", opt.trustservernames ? "true" : "false")); + DEBUGP ((" --directory-prefix %s\n", quote (opt.dir_prefix ? opt.dir_prefix : ""))); + DEBUGP (("\n")); + DEBUGP ((" Counted metalink file %u\n", mfc)); + DEBUGP ((" Planned metalink file %s\n", quote (planname ? planname : ""))); + DEBUGP ((" Trusted metalink file %s\n", quote (trsrname ? trsrname : ""))); + DEBUGP ((" Current metalink file %s\n", quote (filename ? filename : ""))); + DEBUGP ((" Cleaned metalink file %s\n", quote (basename ? basename : ""))); + DEBUGP ((" Secured metalink file %s\n", quote (safename ? safename : ""))); + DEBUGP (("\n")); + + /* Verify if the planned metalink file name is safe. */ + if (!safename || strcmp (planname, safename)) { logprintf (LOG_NOTQUIET, - _("Rejecting metalink file. Invalid basename.\n")); - xfree (filename); - continue; + _("[--trust-server-names %s, --directory-prefix=%s]\n"), + (opt.trustservernames ? "true" : "false"), + quote (opt.dir_prefix ? opt.dir_prefix : "")); + logprintf (LOG_NOTQUIET, + _("Planned metalink file: %s\n"), + quote (planname ? planname : "")); + logprintf (LOG_NOTQUIET, + _("Secured metalink file: %s\n"), + quote (safename ? safename : "")); + if (!safename) + { + logprintf (LOG_NOTQUIET, + _("Rejecting metalink file. Unsafe name.\n")); + xfree (planname); + xfree (trsrname); + xfree (filename); + continue; + } } /* Resources are sorted by priority. */ @@ -189,12 +246,6 @@ retrieve_from_metalink (const metalink_t* metalink) /* Avoid recursive Metalink from HTTP headers. */ bool _metalink_http = opt.metalink_over_http; - /* FIXME: could be useless. */ - if (strcmp (url->file, basename)) - logprintf (LOG_VERBOSE, - _("URL file name %s and Metalink file name %s are different.\n"), - quote_n (0, url->file), quote_n (1, basename)); - /* If output_stream is not NULL, then we have failed on previous resource and are retrying. Thus, continue with the next resource. Do not close output_stream @@ -664,6 +715,8 @@ gpg_skip_verification: } xfree (destname); xfree (filename); + xfree (trsrname); + xfree (planname); } /* Iterate over files. */ /* Restore original values. */ @@ -696,6 +749,22 @@ get_metalink_basename (char *name) return metalink_check_safe_path (basename) ? basename : NULL; } +/* + Append a separator and a numeric suffix to a string. + + The string is permanently modified. +*/ +void +append_suffix_number (char **str, const char *sep, wgint num) +{ + char *new, buf[24]; + + number_to_string (buf, num); + new = aprintf("%s%s%s", *str ? *str : "", sep ? sep : "", buf); + xfree (*str); + *str = new; +} + /* Append the suffix ".badhash" to the file NAME, except without overwriting an existing file with that name and suffix. */ void diff --git a/src/metalink.h b/src/metalink.h index 4846e84..0cfc948 100644 --- a/src/metalink.h +++ b/src/metalink.h @@ -50,6 +50,7 @@ int metalink_res_cmp (const void *res1, const void *res2); int metalink_check_safe_path(const char *path); char *get_metalink_basename (char *name); +void append_suffix_number (char **str, const char *sep, wgint num); void badhash_suffix (char *name); void badhash_or_remove (char *name); diff --git a/testenv/Makefile.am b/testenv/Makefile.am index 569d762..92a3934 100644 --- a/testenv/Makefile.am +++ b/testenv/Makefile.am @@ -31,7 +31,19 @@ if METALINK_IS_ENABLED Test-metalink-xml.py \ Test-metalink-xml-relpath.py \ Test-metalink-xml-abspath.py \ - Test-metalink-xml-homepath.py + Test-metalink-xml-homepath.py \ + Test-metalink-xml-trust.py \ + Test-metalink-xml-relpath-trust.py \ + Test-metalink-xml-abspath-trust.py \ + Test-metalink-xml-homepath-trust.py \ + Test-metalink-xml-prefix.py \ + Test-metalink-xml-relprefix.py \ + Test-metalink-xml-absprefix.py \ + Test-metalink-xml-homeprefix.py \ + Test-metalink-xml-prefix-trust.py \ + Test-metalink-xml-relprefix-trust.py \ + Test-metalink-xml-absprefix-trust.py \ + Test-metalink-xml-homeprefix-trust.py else METALINK_TESTS = endif diff --git a/testenv/Test-metalink-xml-abspath-trust.py b/testenv/Test-metalink-xml-abspath-trust.py new file mode 100755 index 0000000..161e06f --- /dev/null +++ b/testenv/Test-metalink-xml-abspath-trust.py @@ -0,0 +1,130 @@ +#!/usr/bin/env python3 +from sys import exit +from test.http_test import HTTPTest +from misc.wget_file import WgetFile +import re +import hashlib + +""" + This is to test if Metalink/XML forbids absolute paths. + + With --trust-server-names, trust the metalink:file names. + + Without --trust-server-names, don't trust the metalink:file names: + use the basename of --input-metalink, and add a sequential number + (e.g. .#1, .#2, etc.). + + Strip the directory from unsafe paths. +""" +############# File Definitions ############################################### +bad = "Ouch!" + +File1 = "Would you like some Tea?" +File1_lowPref = "Do not take this" +File1_sha256 = hashlib.sha256 (File1.encode ('UTF-8')).hexdigest () + +File2 = "This is gonna be good" +File2_lowPref = "Not this one too" +File2_sha256 = hashlib.sha256 (File2.encode ('UTF-8')).hexdigest () + +MetaXml = \ +""" + + + GNU Wget + + + GNU GPL + http://www.gnu.org/licenses/gpl.html + + Wget Test Files + 1.2.3 + Wget Test Files description + + + + {{FILE1_HASH}} + + + http://{{SRV_HOST}}:{{SRV_PORT}}/wrong_file + http://{{SRV_HOST}}:{{SRV_PORT}}/404 + http://{{SRV_HOST}}:{{SRV_PORT}}/File1_lowPref + http://{{SRV_HOST}}:{{SRV_PORT}}/File1 + + + + + {{FILE2_HASH}} + + + http://{{SRV_HOST}}:{{SRV_PORT}}/wrong_file + http://{{SRV_HOST}}:{{SRV_PORT}}/404 + http://{{SRV_HOST}}:{{SRV_PORT}}/File2_lowPref + http://{{SRV_HOST}}:{{SRV_PORT}}/File2 + + + + +""" + +wrong_file = WgetFile ("wrong_file", bad) + +# rejected by libmetalink +File1_orig = WgetFile ("File1", File1) +File1_nono = WgetFile ("File1_lowPref", File1_lowPref) + +File2_orig = WgetFile ("File2", File2) +File2_down = WgetFile ("File2", File2) +File2_nono = WgetFile ("File2_lowPref", File2_lowPref) + +MetaFile = WgetFile ("test.meta4", MetaXml) + +WGET_OPTIONS = "--trust-server-names --input-metalink test.meta4" +WGET_URLS = [[]] + +Files = [[ + wrong_file, + File1_orig, File1_nono, + File2_orig, File2_nono, +]] +Existing_Files = [MetaFile] + +ExpectedReturnCode = 0 +ExpectedDownloadedFiles = [ + File2_down, + MetaFile +] + +################ Pre and Post Test Hooks ##################################### +pre_test = { + "ServerFiles" : Files, + "LocalFiles" : Existing_Files +} +test_options = { + "WgetCommands" : WGET_OPTIONS, + "Urls" : WGET_URLS +} +post_test = { + "ExpectedFiles" : ExpectedDownloadedFiles, + "ExpectedRetcode" : ExpectedReturnCode +} + +http_test = HTTPTest ( + pre_hook=pre_test, + test_params=test_options, + post_hook=post_test, +) + +http_test.server_setup() +### Get and use dynamic server sockname +srv_host, srv_port = http_test.servers[0].server_inst.socket.getsockname () + +MetaXml = re.sub (r'{{FILE1_HASH}}', File1_sha256, MetaXml) +MetaXml = re.sub (r'{{FILE2_HASH}}', File2_sha256, MetaXml) +MetaXml = re.sub (r'{{SRV_HOST}}', srv_host, MetaXml) +MetaXml = re.sub (r'{{SRV_PORT}}', str (srv_port), MetaXml) +MetaFile.content = MetaXml + +err = http_test.begin () + +exit (err) diff --git a/testenv/Test-metalink-xml-abspath.py b/testenv/Test-metalink-xml-abspath.py index 62aabb9..0c9570d 100755 --- a/testenv/Test-metalink-xml-abspath.py +++ b/testenv/Test-metalink-xml-abspath.py @@ -6,12 +6,27 @@ import re import hashlib """ - This is to test if Metalink XML file escapes current directory. + This is to test if Metalink/XML forbids absolute paths. + + With --trust-server-names, trust the metalink:file names. + + Without --trust-server-names, don't trust the metalink:file names: + use the basename of --input-metalink, and add a sequential number + (e.g. .#1, .#2, etc.). + + Strip the directory from unsafe paths. """ ############# File Definitions ############################################### +bad = "Ouch!" + File1 = "Would you like some Tea?" File1_lowPref = "Do not take this" File1_sha256 = hashlib.sha256 (File1.encode ('UTF-8')).hexdigest () + +File2 = "This is gonna be good" +File2_lowPref = "Not this one too" +File2_sha256 = hashlib.sha256 (File2.encode ('UTF-8')).hexdigest () + MetaXml = \ """ @@ -22,36 +37,63 @@ MetaXml = \ GNU GPL http://www.gnu.org/licenses/gpl.html - Wget Test File 1 + Wget Test Files 1.2.3 - Wget Test File 1 description + Wget Test Files description - + {{FILE1_HASH}} - http://broken.example/File1 + http://{{SRV_HOST}}:{{SRV_PORT}}/wrong_file + http://{{SRV_HOST}}:{{SRV_PORT}}/404 http://{{SRV_HOST}}:{{SRV_PORT}}/File1_lowPref http://{{SRV_HOST}}:{{SRV_PORT}}/File1 + + + {{FILE2_HASH}} + + + http://{{SRV_HOST}}:{{SRV_PORT}}/wrong_file + http://{{SRV_HOST}}:{{SRV_PORT}}/404 + http://{{SRV_HOST}}:{{SRV_PORT}}/File2_lowPref + http://{{SRV_HOST}}:{{SRV_PORT}}/File2 + + """ -A_File = WgetFile ("File1", File1) -B_File = WgetFile ("File1_lowPref", File1_lowPref) +wrong_file = WgetFile ("wrong_file", bad) + +# rejected by libmetalink +File1_orig = WgetFile ("File1", File1) +File1_nono = WgetFile ("File1_lowPref", File1_lowPref) + +File2_orig = WgetFile ("File2", File2) +File2_down = WgetFile ("test.meta4.#1", File2) +File2_nono = WgetFile ("File2_lowPref", File2_lowPref) + MetaFile = WgetFile ("test.meta4", MetaXml) WGET_OPTIONS = "--input-metalink test.meta4" WGET_URLS = [[]] -Files = [[A_File, B_File]] +Files = [[ + wrong_file, + File1_orig, File1_nono, + File2_orig, File2_nono +]] Existing_Files = [MetaFile] ExpectedReturnCode = 0 -ExpectedDownloadedFiles = [MetaFile] +ExpectedDownloadedFiles = [ + File2_down, + MetaFile +] ################ Pre and Post Test Hooks ##################################### pre_test = { @@ -78,6 +120,7 @@ http_test.server_setup() srv_host, srv_port = http_test.servers[0].server_inst.socket.getsockname () MetaXml = re.sub (r'{{FILE1_HASH}}', File1_sha256, MetaXml) +MetaXml = re.sub (r'{{FILE2_HASH}}', File2_sha256, MetaXml) MetaXml = re.sub (r'{{SRV_HOST}}', srv_host, MetaXml) MetaXml = re.sub (r'{{SRV_PORT}}', str (srv_port), MetaXml) MetaFile.content = MetaXml diff --git a/testenv/Test-metalink-xml-absprefix-trust.py b/testenv/Test-metalink-xml-absprefix-trust.py new file mode 100755 index 0000000..1bbaabc --- /dev/null +++ b/testenv/Test-metalink-xml-absprefix-trust.py @@ -0,0 +1,194 @@ +#!/usr/bin/env python3 +from sys import exit +from test.http_test import HTTPTest +from misc.wget_file import WgetFile +import re +import hashlib + +""" + This is to test Metalink/XML absolute directory prefix support in Wget. + + With --trust-server-names, trust the metalink:file names. + + Without --trust-server-names, don't trust the metalink:file names: + use the basename of --input-metalink, and add a sequential number + (e.g. .#1, .#2, etc.). + + Strip the directory from unsafe paths. +""" +############# File Definitions ############################################### +bad = "Ouch!" + +File1 = "Would you like some Tea?" +File1_lowPref = "Do not take this" +File1_sha256 = hashlib.sha256 (File1.encode ('UTF-8')).hexdigest () + +File2 = "This is gonna be good" +File2_lowPref = "Not this one too" +File2_sha256 = hashlib.sha256 (File2.encode ('UTF-8')).hexdigest () + +File3 = "A little more, please" +File3_lowPref = "That's just too much" +File3_sha256 = hashlib.sha256 (File3.encode ('UTF-8')).hexdigest () + +File4 = "Maybe a biscuit?" +File4_lowPref = "No, thanks" +File4_sha256 = hashlib.sha256 (File4.encode ('UTF-8')).hexdigest () + +File5 = "More Tea...?" +File5_lowPref = "I have to go..." +File5_sha256 = hashlib.sha256 (File5.encode ('UTF-8')).hexdigest () + +MetaXml = \ +""" + + + GNU Wget + + + GNU GPL + http://www.gnu.org/licenses/gpl.html + + Wget Test Files + 1.2.3 + Wget Test Files description + + + + {{FILE1_HASH}} + + + http://{{SRV_HOST}}:{{SRV_PORT}}/wrong_file + http://{{SRV_HOST}}:{{SRV_PORT}}/404 + http://{{SRV_HOST}}:{{SRV_PORT}}/File1_lowPref + http://{{SRV_HOST}}:{{SRV_PORT}}/File1 + + + + + {{FILE2_HASH}} + + + http://{{SRV_HOST}}:{{SRV_PORT}}/wrong_file + http://{{SRV_HOST}}:{{SRV_PORT}}/404 + http://{{SRV_HOST}}:{{SRV_PORT}}/File2_lowPref + http://{{SRV_HOST}}:{{SRV_PORT}}/File2 + + + + + {{FILE3_HASH}} + + + http://{{SRV_HOST}}:{{SRV_PORT}}/wrong_file + http://{{SRV_HOST}}:{{SRV_PORT}}/404 + http://{{SRV_HOST}}:{{SRV_PORT}}/File3_lowPref + http://{{SRV_HOST}}:{{SRV_PORT}}/File3 + + + + + {{FILE4_HASH}} + + + http://{{SRV_HOST}}:{{SRV_PORT}}/wrong_file + http://{{SRV_HOST}}:{{SRV_PORT}}/404 + http://{{SRV_HOST}}:{{SRV_PORT}}/File4_lowPref + http://{{SRV_HOST}}:{{SRV_PORT}}/File4 + + + + + {{FILE5_HASH}} + + + http://{{SRV_HOST}}:{{SRV_PORT}}/wrong_file + http://{{SRV_HOST}}:{{SRV_PORT}}/404 + http://{{SRV_HOST}}:{{SRV_PORT}}/File5_lowPref + http://{{SRV_HOST}}:{{SRV_PORT}}/File5 + + + + +""" + +wrong_file = WgetFile ("wrong_file", bad) + +File1_orig = WgetFile ("File1", File1) +File1_down = WgetFile ("File1", File1) +File1_nono = WgetFile ("File1_lowPref", File1_lowPref) + +# rejected by libmetalink +File2_orig = WgetFile ("File2", File2) +File2_nono = WgetFile ("File2_lowPref", File2_lowPref) + +# rejected by libmetalink +File3_orig = WgetFile ("File3", File3) +File3_nono = WgetFile ("File3_lowPref", File3_lowPref) + +# rejected by libmetalink +File4_orig = WgetFile ("File4", File4) +File4_nono = WgetFile ("File4_lowPref", File4_lowPref) + +File5_orig = WgetFile ("File5", File5) +File5_down = WgetFile ("File5", File5) +File5_nono = WgetFile ("File5_lowPref", File5_lowPref) + +MetaFile = WgetFile ("test.meta4", MetaXml) + +WGET_OPTIONS = "--trust-server-names --directory-prefix /dir --input-metalink test.meta4" +WGET_URLS = [[]] + +Files = [[ + wrong_file, + File1_orig, File1_nono, + File2_orig, File2_nono, + File3_orig, File3_nono, + File4_orig, File4_nono, + File5_orig, File5_nono +]] +Existing_Files = [MetaFile] + +ExpectedReturnCode = 0 +ExpectedDownloadedFiles = [ + File1_down, + File5_down, + MetaFile +] + +################ Pre and Post Test Hooks ##################################### +pre_test = { + "ServerFiles" : Files, + "LocalFiles" : Existing_Files +} +test_options = { + "WgetCommands" : WGET_OPTIONS, + "Urls" : WGET_URLS +} +post_test = { + "ExpectedFiles" : ExpectedDownloadedFiles, + "ExpectedRetcode" : ExpectedReturnCode +} + +http_test = HTTPTest ( + pre_hook=pre_test, + test_params=test_options, + post_hook=post_test, +) + +http_test.server_setup() +### Get and use dynamic server sockname +srv_host, srv_port = http_test.servers[0].server_inst.socket.getsockname () + +MetaXml = re.sub (r'{{FILE1_HASH}}', File1_sha256, MetaXml) +MetaXml = re.sub (r'{{FILE2_HASH}}', File2_sha256, MetaXml) +MetaXml = re.sub (r'{{FILE3_HASH}}', File3_sha256, MetaXml) +MetaXml = re.sub (r'{{FILE4_HASH}}', File4_sha256, MetaXml) +MetaXml = re.sub (r'{{FILE5_HASH}}', File5_sha256, MetaXml) +MetaXml = re.sub (r'{{SRV_HOST}}', srv_host, MetaXml) +MetaXml = re.sub (r'{{SRV_PORT}}', str (srv_port), MetaXml) +MetaFile.content = MetaXml + +err = http_test.begin () + +exit (err) diff --git a/testenv/Test-metalink-xml-absprefix.py b/testenv/Test-metalink-xml-absprefix.py new file mode 100755 index 0000000..fb35e4e --- /dev/null +++ b/testenv/Test-metalink-xml-absprefix.py @@ -0,0 +1,194 @@ +#!/usr/bin/env python3 +from sys import exit +from test.http_test import HTTPTest +from misc.wget_file import WgetFile +import re +import hashlib + +""" + This is to test Metalink/XML absolute directory prefix support in Wget. + + With --trust-server-names, trust the metalink:file names. + + Without --trust-server-names, don't trust the metalink:file names: + use the basename of --input-metalink, and add a sequential number + (e.g. .#1, .#2, etc.). + + Strip the directory from unsafe paths. +""" +############# File Definitions ############################################### +bad = "Ouch!" + +File1 = "Would you like some Tea?" +File1_lowPref = "Do not take this" +File1_sha256 = hashlib.sha256 (File1.encode ('UTF-8')).hexdigest () + +File2 = "This is gonna be good" +File2_lowPref = "Not this one too" +File2_sha256 = hashlib.sha256 (File2.encode ('UTF-8')).hexdigest () + +File3 = "A little more, please" +File3_lowPref = "That's just too much" +File3_sha256 = hashlib.sha256 (File3.encode ('UTF-8')).hexdigest () + +File4 = "Maybe a biscuit?" +File4_lowPref = "No, thanks" +File4_sha256 = hashlib.sha256 (File4.encode ('UTF-8')).hexdigest () + +File5 = "More Tea...?" +File5_lowPref = "I have to go..." +File5_sha256 = hashlib.sha256 (File5.encode ('UTF-8')).hexdigest () + +MetaXml = \ +""" + + + GNU Wget + + + GNU GPL + http://www.gnu.org/licenses/gpl.html + + Wget Test Files + 1.2.3 + Wget Test Files description + + + + {{FILE1_HASH}} + + + http://{{SRV_HOST}}:{{SRV_PORT}}/wrong_file + http://{{SRV_HOST}}:{{SRV_PORT}}/404 + http://{{SRV_HOST}}:{{SRV_PORT}}/File1_lowPref + http://{{SRV_HOST}}:{{SRV_PORT}}/File1 + + + + + {{FILE2_HASH}} + + + http://{{SRV_HOST}}:{{SRV_PORT}}/wrong_file + http://{{SRV_HOST}}:{{SRV_PORT}}/404 + http://{{SRV_HOST}}:{{SRV_PORT}}/File2_lowPref + http://{{SRV_HOST}}:{{SRV_PORT}}/File2 + + + + + {{FILE3_HASH}} + + + http://{{SRV_HOST}}:{{SRV_PORT}}/wrong_file + http://{{SRV_HOST}}:{{SRV_PORT}}/404 + http://{{SRV_HOST}}:{{SRV_PORT}}/File3_lowPref + http://{{SRV_HOST}}:{{SRV_PORT}}/File3 + + + + + {{FILE4_HASH}} + + + http://{{SRV_HOST}}:{{SRV_PORT}}/wrong_file + http://{{SRV_HOST}}:{{SRV_PORT}}/404 + http://{{SRV_HOST}}:{{SRV_PORT}}/File4_lowPref + http://{{SRV_HOST}}:{{SRV_PORT}}/File4 + + + + + {{FILE5_HASH}} + + + http://{{SRV_HOST}}:{{SRV_PORT}}/wrong_file + http://{{SRV_HOST}}:{{SRV_PORT}}/404 + http://{{SRV_HOST}}:{{SRV_PORT}}/File5_lowPref + http://{{SRV_HOST}}:{{SRV_PORT}}/File5 + + + + +""" + +wrong_file = WgetFile ("wrong_file", bad) + +File1_orig = WgetFile ("File1", File1) +File1_down = WgetFile ("test.meta4.#1", File1) +File1_nono = WgetFile ("File1_lowPref", File1_lowPref) + +# rejected by libmetalink +File2_orig = WgetFile ("File2", File2) +File2_nono = WgetFile ("File2_lowPref", File2_lowPref) + +# rejected by libmetalink +File3_orig = WgetFile ("File3", File3) +File3_nono = WgetFile ("File3_lowPref", File3_lowPref) + +# rejected by libmetalink +File4_orig = WgetFile ("File4", File4) +File4_nono = WgetFile ("File4_lowPref", File4_lowPref) + +File5_orig = WgetFile ("File5", File5) +File5_down = WgetFile ("test.meta4.#2", File5) +File5_nono = WgetFile ("File5_lowPref", File5_lowPref) + +MetaFile = WgetFile ("test.meta4", MetaXml) + +WGET_OPTIONS = "--directory-prefix /dir --input-metalink test.meta4" +WGET_URLS = [[]] + +Files = [[ + wrong_file, + File1_orig, File1_nono, + File2_orig, File2_nono, + File3_orig, File3_nono, + File4_orig, File4_nono, + File5_orig, File5_nono +]] +Existing_Files = [MetaFile] + +ExpectedReturnCode = 0 +ExpectedDownloadedFiles = [ + File1_down, + File5_down, + MetaFile +] + +################ Pre and Post Test Hooks ##################################### +pre_test = { + "ServerFiles" : Files, + "LocalFiles" : Existing_Files +} +test_options = { + "WgetCommands" : WGET_OPTIONS, + "Urls" : WGET_URLS +} +post_test = { + "ExpectedFiles" : ExpectedDownloadedFiles, + "ExpectedRetcode" : ExpectedReturnCode +} + +http_test = HTTPTest ( + pre_hook=pre_test, + test_params=test_options, + post_hook=post_test, +) + +http_test.server_setup() +### Get and use dynamic server sockname +srv_host, srv_port = http_test.servers[0].server_inst.socket.getsockname () + +MetaXml = re.sub (r'{{FILE1_HASH}}', File1_sha256, MetaXml) +MetaXml = re.sub (r'{{FILE2_HASH}}', File2_sha256, MetaXml) +MetaXml = re.sub (r'{{FILE3_HASH}}', File3_sha256, MetaXml) +MetaXml = re.sub (r'{{FILE4_HASH}}', File4_sha256, MetaXml) +MetaXml = re.sub (r'{{FILE5_HASH}}', File5_sha256, MetaXml) +MetaXml = re.sub (r'{{SRV_HOST}}', srv_host, MetaXml) +MetaXml = re.sub (r'{{SRV_PORT}}', str (srv_port), MetaXml) +MetaFile.content = MetaXml + +err = http_test.begin () + +exit (err) diff --git a/testenv/Test-metalink-xml-homepath-trust.py b/testenv/Test-metalink-xml-homepath-trust.py new file mode 100755 index 0000000..9637048 --- /dev/null +++ b/testenv/Test-metalink-xml-homepath-trust.py @@ -0,0 +1,195 @@ +#!/usr/bin/env python3 +from sys import exit +from test.http_test import HTTPTest +from misc.wget_file import WgetFile +import re +import hashlib + +""" + This is to test if Metalink/XML forbids the home path and names + beginning with the ~ (tilde) character. + + With --trust-server-names, trust the metalink:file names. + + Without --trust-server-names, don't trust the metalink:file names: + use the basename of --input-metalink, and add a sequential number + (e.g. .#1, .#2, etc.). + + Strip the directory from unsafe paths. +""" +############# File Definitions ############################################### +bad = "Ouch!" + +File1 = "Would you like some Tea?" +File1_lowPref = "Do not take this" +File1_sha256 = hashlib.sha256 (File1.encode ('UTF-8')).hexdigest () + +File2 = "This is gonna be good" +File2_lowPref = "Not this one too" +File2_sha256 = hashlib.sha256 (File2.encode ('UTF-8')).hexdigest () + +File3 = "A little more, please" +File3_lowPref = "That's just too much" +File3_sha256 = hashlib.sha256 (File3.encode ('UTF-8')).hexdigest () + +File4 = "Maybe a biscuit?" +File4_lowPref = "No, thanks" +File4_sha256 = hashlib.sha256 (File4.encode ('UTF-8')).hexdigest () + +File5 = "More Tea...?" +File5_lowPref = "I have to go..." +File5_sha256 = hashlib.sha256 (File5.encode ('UTF-8')).hexdigest () + +MetaXml = \ +""" + + + GNU Wget + + + GNU GPL + http://www.gnu.org/licenses/gpl.html + + Wget Test Files + 1.2.3 + Wget Test Files description + + + + {{FILE1_HASH}} + + + http://{{SRV_HOST}}:{{SRV_PORT}}/wrong_file + http://{{SRV_HOST}}:{{SRV_PORT}}/404 + http://{{SRV_HOST}}:{{SRV_PORT}}/File1_lowPref + http://{{SRV_HOST}}:{{SRV_PORT}}/File1 + + + + + {{FILE2_HASH}} + + + http://{{SRV_HOST}}:{{SRV_PORT}}/wrong_file + http://{{SRV_HOST}}:{{SRV_PORT}}/404 + http://{{SRV_HOST}}:{{SRV_PORT}}/File2_lowPref + http://{{SRV_HOST}}:{{SRV_PORT}}/File2 + + + + + {{FILE3_HASH}} + + + http://{{SRV_HOST}}:{{SRV_PORT}}/wrong_file + http://{{SRV_HOST}}:{{SRV_PORT}}/404 + http://{{SRV_HOST}}:{{SRV_PORT}}/File3_lowPref + http://{{SRV_HOST}}:{{SRV_PORT}}/File3 + + + + + {{FILE4_HASH}} + + + http://{{SRV_HOST}}:{{SRV_PORT}}/wrong_file + http://{{SRV_HOST}}:{{SRV_PORT}}/404 + http://{{SRV_HOST}}:{{SRV_PORT}}/File4_lowPref + http://{{SRV_HOST}}:{{SRV_PORT}}/File4 + + + + + {{FILE5_HASH}} + + + http://{{SRV_HOST}}:{{SRV_PORT}}/wrong_file + http://{{SRV_HOST}}:{{SRV_PORT}}/404 + http://{{SRV_HOST}}:{{SRV_PORT}}/File5_lowPref + http://{{SRV_HOST}}:{{SRV_PORT}}/File5 + + + + +""" + +wrong_file = WgetFile ("wrong_file", bad) + +# rejected by libmetalink +File1_orig = WgetFile ("File1", File1) +File1_nono = WgetFile ("File1_lowPref", File1_lowPref) + +# rejected by libmetalink +File2_orig = WgetFile ("File2", File2) +File2_nono = WgetFile ("File2_lowPref", File2_lowPref) + +# rejected by libmetalink +File3_orig = WgetFile ("File3", File3) +File3_nono = WgetFile ("File3_lowPref", File3_lowPref) + +File4_orig = WgetFile ("File4", File4) +File4_down = WgetFile ("dir/File4~", File4) +File4_nono = WgetFile ("File4_lowPref", File4_lowPref) + +File5_orig = WgetFile ("File5", File5) +File5_down = WgetFile ("dir/~/File5", File5) +File5_nono = WgetFile ("File5_lowPref", File5_lowPref) + +MetaFile = WgetFile ("test.meta4", MetaXml) + +WGET_OPTIONS = "--trust-server-names --input-metalink test.meta4" +WGET_URLS = [[]] + +Files = [[ + wrong_file, + File1_orig, File1_nono, + File2_orig, File2_nono, + File3_orig, File3_nono, + File4_orig, File4_nono, + File5_orig, File5_nono +]] +Existing_Files = [MetaFile] + +ExpectedReturnCode = 0 +ExpectedDownloadedFiles = [ + File4_down, + File5_down, + MetaFile +] + +################ Pre and Post Test Hooks ##################################### +pre_test = { + "ServerFiles" : Files, + "LocalFiles" : Existing_Files +} +test_options = { + "WgetCommands" : WGET_OPTIONS, + "Urls" : WGET_URLS +} +post_test = { + "ExpectedFiles" : ExpectedDownloadedFiles, + "ExpectedRetcode" : ExpectedReturnCode +} + +http_test = HTTPTest ( + pre_hook=pre_test, + test_params=test_options, + post_hook=post_test, +) + +http_test.server_setup() +### Get and use dynamic server sockname +srv_host, srv_port = http_test.servers[0].server_inst.socket.getsockname () + +MetaXml = re.sub (r'{{FILE1_HASH}}', File1_sha256, MetaXml) +MetaXml = re.sub (r'{{FILE2_HASH}}', File2_sha256, MetaXml) +MetaXml = re.sub (r'{{FILE3_HASH}}', File3_sha256, MetaXml) +MetaXml = re.sub (r'{{FILE4_HASH}}', File4_sha256, MetaXml) +MetaXml = re.sub (r'{{FILE5_HASH}}', File5_sha256, MetaXml) +MetaXml = re.sub (r'{{SRV_HOST}}', srv_host, MetaXml) +MetaXml = re.sub (r'{{SRV_PORT}}', str (srv_port), MetaXml) +MetaFile.content = MetaXml + +err = http_test.begin () + +exit (err) diff --git a/testenv/Test-metalink-xml-homepath.py b/testenv/Test-metalink-xml-homepath.py index 85c6df9..03e6c5a 100755 --- a/testenv/Test-metalink-xml-homepath.py +++ b/testenv/Test-metalink-xml-homepath.py @@ -6,12 +6,40 @@ import re import hashlib """ - This is to test if Metalink XML file escapes current directory. + This is to test if Metalink/XML forbids the home path and names + beginning with the ~ (tilde) character. + + With --trust-server-names, trust the metalink:file names. + + Without --trust-server-names, don't trust the metalink:file names: + use the basename of --input-metalink, and add a sequential number + (e.g. .#1, .#2, etc.). + + Strip the directory from unsafe paths. """ ############# File Definitions ############################################### +bad = "Ouch!" + File1 = "Would you like some Tea?" File1_lowPref = "Do not take this" File1_sha256 = hashlib.sha256 (File1.encode ('UTF-8')).hexdigest () + +File2 = "This is gonna be good" +File2_lowPref = "Not this one too" +File2_sha256 = hashlib.sha256 (File2.encode ('UTF-8')).hexdigest () + +File3 = "A little more, please" +File3_lowPref = "That's just too much" +File3_sha256 = hashlib.sha256 (File3.encode ('UTF-8')).hexdigest () + +File4 = "Maybe a biscuit?" +File4_lowPref = "No, thanks" +File4_sha256 = hashlib.sha256 (File4.encode ('UTF-8')).hexdigest () + +File5 = "More Tea...?" +File5_lowPref = "I have to go..." +File5_sha256 = hashlib.sha256 (File5.encode ('UTF-8')).hexdigest () + MetaXml = \ """ @@ -22,36 +50,112 @@ MetaXml = \ GNU GPL http://www.gnu.org/licenses/gpl.html - Wget Test File 1 + Wget Test Files 1.2.3 - Wget Test File 1 description + Wget Test Files description - + {{FILE1_HASH}} - http://broken.example/File1 + http://{{SRV_HOST}}:{{SRV_PORT}}/wrong_file + http://{{SRV_HOST}}:{{SRV_PORT}}/404 http://{{SRV_HOST}}:{{SRV_PORT}}/File1_lowPref http://{{SRV_HOST}}:{{SRV_PORT}}/File1 + + + {{FILE2_HASH}} + + + http://{{SRV_HOST}}:{{SRV_PORT}}/wrong_file + http://{{SRV_HOST}}:{{SRV_PORT}}/404 + http://{{SRV_HOST}}:{{SRV_PORT}}/File2_lowPref + http://{{SRV_HOST}}:{{SRV_PORT}}/File2 + + + + + {{FILE3_HASH}} + + + http://{{SRV_HOST}}:{{SRV_PORT}}/wrong_file + http://{{SRV_HOST}}:{{SRV_PORT}}/404 + http://{{SRV_HOST}}:{{SRV_PORT}}/File3_lowPref + http://{{SRV_HOST}}:{{SRV_PORT}}/File3 + + + + + {{FILE4_HASH}} + + + http://{{SRV_HOST}}:{{SRV_PORT}}/wrong_file + http://{{SRV_HOST}}:{{SRV_PORT}}/404 + http://{{SRV_HOST}}:{{SRV_PORT}}/File4_lowPref + http://{{SRV_HOST}}:{{SRV_PORT}}/File4 + + + + + {{FILE5_HASH}} + + + http://{{SRV_HOST}}:{{SRV_PORT}}/wrong_file + http://{{SRV_HOST}}:{{SRV_PORT}}/404 + http://{{SRV_HOST}}:{{SRV_PORT}}/File5_lowPref + http://{{SRV_HOST}}:{{SRV_PORT}}/File5 + + """ -A_File = WgetFile ("File1", File1) -B_File = WgetFile ("File1_lowPref", File1_lowPref) +wrong_file = WgetFile ("wrong_file", bad) + +# rejected by libmetalink +File1_orig = WgetFile ("File1", File1) +File1_nono = WgetFile ("File1_lowPref", File1_lowPref) + +# rejected by libmetalink +File2_orig = WgetFile ("File2", File2) +File2_nono = WgetFile ("File2_lowPref", File2_lowPref) + +# rejected by libmetalink +File3_orig = WgetFile ("File3", File3) +File3_nono = WgetFile ("File3_lowPref", File3_lowPref) + +File4_orig = WgetFile ("File4", File4) +File4_down = WgetFile ("test.meta4.#1", File4) +File4_nono = WgetFile ("File4_lowPref", File4_lowPref) + +File5_orig = WgetFile ("File5", File5) +File5_down = WgetFile ("test.meta4.#2", File5) +File5_nono = WgetFile ("File5_lowPref", File5_lowPref) + MetaFile = WgetFile ("test.meta4", MetaXml) WGET_OPTIONS = "--input-metalink test.meta4" WGET_URLS = [[]] -Files = [[A_File, B_File]] +Files = [[ + wrong_file, + File1_orig, File1_nono, + File2_orig, File2_nono, + File3_orig, File3_nono, + File4_orig, File4_nono, + File5_orig, File5_nono +]] Existing_Files = [MetaFile] ExpectedReturnCode = 0 -ExpectedDownloadedFiles = [MetaFile] +ExpectedDownloadedFiles = [ + File4_down, + File5_down, + MetaFile +] ################ Pre and Post Test Hooks ##################################### pre_test = { @@ -78,6 +182,10 @@ http_test.server_setup() srv_host, srv_port = http_test.servers[0].server_inst.socket.getsockname () MetaXml = re.sub (r'{{FILE1_HASH}}', File1_sha256, MetaXml) +MetaXml = re.sub (r'{{FILE2_HASH}}', File2_sha256, MetaXml) +MetaXml = re.sub (r'{{FILE3_HASH}}', File3_sha256, MetaXml) +MetaXml = re.sub (r'{{FILE4_HASH}}', File4_sha256, MetaXml) +MetaXml = re.sub (r'{{FILE5_HASH}}', File5_sha256, MetaXml) MetaXml = re.sub (r'{{SRV_HOST}}', srv_host, MetaXml) MetaXml = re.sub (r'{{SRV_PORT}}', str (srv_port), MetaXml) MetaFile.content = MetaXml diff --git a/testenv/Test-metalink-xml-homeprefix-trust.py b/testenv/Test-metalink-xml-homeprefix-trust.py new file mode 100755 index 0000000..1a2d9f2 --- /dev/null +++ b/testenv/Test-metalink-xml-homeprefix-trust.py @@ -0,0 +1,194 @@ +#!/usr/bin/env python3 +from sys import exit +from test.http_test import HTTPTest +from misc.wget_file import WgetFile +import re +import hashlib + +""" + This is to test Metalink/XML home directory prefix support in Wget. + + With --trust-server-names, trust the metalink:file names. + + Without --trust-server-names, don't trust the metalink:file names: + use the basename of --input-metalink, and add a sequential number + (e.g. .#1, .#2, etc.). + + Strip the directory from unsafe paths. +""" +############# File Definitions ############################################### +bad = "Ouch!" + +File1 = "Would you like some Tea?" +File1_lowPref = "Do not take this" +File1_sha256 = hashlib.sha256 (File1.encode ('UTF-8')).hexdigest () + +File2 = "This is gonna be good" +File2_lowPref = "Not this one too" +File2_sha256 = hashlib.sha256 (File2.encode ('UTF-8')).hexdigest () + +File3 = "A little more, please" +File3_lowPref = "That's just too much" +File3_sha256 = hashlib.sha256 (File3.encode ('UTF-8')).hexdigest () + +File4 = "Maybe a biscuit?" +File4_lowPref = "No, thanks" +File4_sha256 = hashlib.sha256 (File4.encode ('UTF-8')).hexdigest () + +File5 = "More Tea...?" +File5_lowPref = "I have to go..." +File5_sha256 = hashlib.sha256 (File5.encode ('UTF-8')).hexdigest () + +MetaXml = \ +""" + + + GNU Wget + + + GNU GPL + http://www.gnu.org/licenses/gpl.html + + Wget Test Files + 1.2.3 + Wget Test Files description + + + + {{FILE1_HASH}} + + + http://{{SRV_HOST}}:{{SRV_PORT}}/wrong_file + http://{{SRV_HOST}}:{{SRV_PORT}}/404 + http://{{SRV_HOST}}:{{SRV_PORT}}/File1_lowPref + http://{{SRV_HOST}}:{{SRV_PORT}}/File1 + + + + + {{FILE2_HASH}} + + + http://{{SRV_HOST}}:{{SRV_PORT}}/wrong_file + http://{{SRV_HOST}}:{{SRV_PORT}}/404 + http://{{SRV_HOST}}:{{SRV_PORT}}/File2_lowPref + http://{{SRV_HOST}}:{{SRV_PORT}}/File2 + + + + + {{FILE3_HASH}} + + + http://{{SRV_HOST}}:{{SRV_PORT}}/wrong_file + http://{{SRV_HOST}}:{{SRV_PORT}}/404 + http://{{SRV_HOST}}:{{SRV_PORT}}/File3_lowPref + http://{{SRV_HOST}}:{{SRV_PORT}}/File3 + + + + + {{FILE4_HASH}} + + + http://{{SRV_HOST}}:{{SRV_PORT}}/wrong_file + http://{{SRV_HOST}}:{{SRV_PORT}}/404 + http://{{SRV_HOST}}:{{SRV_PORT}}/File4_lowPref + http://{{SRV_HOST}}:{{SRV_PORT}}/File4 + + + + + {{FILE5_HASH}} + + + http://{{SRV_HOST}}:{{SRV_PORT}}/wrong_file + http://{{SRV_HOST}}:{{SRV_PORT}}/404 + http://{{SRV_HOST}}:{{SRV_PORT}}/File5_lowPref + http://{{SRV_HOST}}:{{SRV_PORT}}/File5 + + + + +""" + +wrong_file = WgetFile ("wrong_file", bad) + +File1_orig = WgetFile ("File1", File1) +File1_down = WgetFile ("File1", File1) +File1_nono = WgetFile ("File1_lowPref", File1_lowPref) + +# rejected by libmetalink +File2_orig = WgetFile ("File2", File2) +File2_nono = WgetFile ("File2_lowPref", File2_lowPref) + +# rejected by libmetalink +File3_orig = WgetFile ("File3", File3) +File3_nono = WgetFile ("File3_lowPref", File3_lowPref) + +# rejected by libmetalink +File4_orig = WgetFile ("File4", File4) +File4_nono = WgetFile ("File4_lowPref", File4_lowPref) + +File5_orig = WgetFile ("File5", File5) +File5_down = WgetFile ("File5", File5) +File5_nono = WgetFile ("File5_lowPref", File5_lowPref) + +MetaFile = WgetFile ("test.meta4", MetaXml) + +WGET_OPTIONS = "--trust-server-names --directory-prefix ~/dir --input-metalink test.meta4" +WGET_URLS = [[]] + +Files = [[ + wrong_file, + File1_orig, File1_nono, + File2_orig, File2_nono, + File3_orig, File3_nono, + File4_orig, File4_nono, + File5_orig, File5_nono +]] +Existing_Files = [MetaFile] + +ExpectedReturnCode = 0 +ExpectedDownloadedFiles = [ + File1_down, + File5_down, + MetaFile +] + +################ Pre and Post Test Hooks ##################################### +pre_test = { + "ServerFiles" : Files, + "LocalFiles" : Existing_Files +} +test_options = { + "WgetCommands" : WGET_OPTIONS, + "Urls" : WGET_URLS +} +post_test = { + "ExpectedFiles" : ExpectedDownloadedFiles, + "ExpectedRetcode" : ExpectedReturnCode +} + +http_test = HTTPTest ( + pre_hook=pre_test, + test_params=test_options, + post_hook=post_test, +) + +http_test.server_setup() +### Get and use dynamic server sockname +srv_host, srv_port = http_test.servers[0].server_inst.socket.getsockname () + +MetaXml = re.sub (r'{{FILE1_HASH}}', File1_sha256, MetaXml) +MetaXml = re.sub (r'{{FILE2_HASH}}', File2_sha256, MetaXml) +MetaXml = re.sub (r'{{FILE3_HASH}}', File3_sha256, MetaXml) +MetaXml = re.sub (r'{{FILE4_HASH}}', File4_sha256, MetaXml) +MetaXml = re.sub (r'{{FILE5_HASH}}', File5_sha256, MetaXml) +MetaXml = re.sub (r'{{SRV_HOST}}', srv_host, MetaXml) +MetaXml = re.sub (r'{{SRV_PORT}}', str (srv_port), MetaXml) +MetaFile.content = MetaXml + +err = http_test.begin () + +exit (err) diff --git a/testenv/Test-metalink-xml-homeprefix.py b/testenv/Test-metalink-xml-homeprefix.py new file mode 100755 index 0000000..1ee6c11 --- /dev/null +++ b/testenv/Test-metalink-xml-homeprefix.py @@ -0,0 +1,194 @@ +#!/usr/bin/env python3 +from sys import exit +from test.http_test import HTTPTest +from misc.wget_file import WgetFile +import re +import hashlib + +""" + This is to test Metalink/XML home directory prefix support in Wget. + + With --trust-server-names, trust the metalink:file names. + + Without --trust-server-names, don't trust the metalink:file names: + use the basename of --input-metalink, and add a sequential number + (e.g. .#1, .#2, etc.). + + Strip the directory from unsafe paths. +""" +############# File Definitions ############################################### +bad = "Ouch!" + +File1 = "Would you like some Tea?" +File1_lowPref = "Do not take this" +File1_sha256 = hashlib.sha256 (File1.encode ('UTF-8')).hexdigest () + +File2 = "This is gonna be good" +File2_lowPref = "Not this one too" +File2_sha256 = hashlib.sha256 (File2.encode ('UTF-8')).hexdigest () + +File3 = "A little more, please" +File3_lowPref = "That's just too much" +File3_sha256 = hashlib.sha256 (File3.encode ('UTF-8')).hexdigest () + +File4 = "Maybe a biscuit?" +File4_lowPref = "No, thanks" +File4_sha256 = hashlib.sha256 (File4.encode ('UTF-8')).hexdigest () + +File5 = "More Tea...?" +File5_lowPref = "I have to go..." +File5_sha256 = hashlib.sha256 (File5.encode ('UTF-8')).hexdigest () + +MetaXml = \ +""" + + + GNU Wget + + + GNU GPL + http://www.gnu.org/licenses/gpl.html + + Wget Test Files + 1.2.3 + Wget Test Files description + + + + {{FILE1_HASH}} + + + http://{{SRV_HOST}}:{{SRV_PORT}}/wrong_file + http://{{SRV_HOST}}:{{SRV_PORT}}/404 + http://{{SRV_HOST}}:{{SRV_PORT}}/File1_lowPref + http://{{SRV_HOST}}:{{SRV_PORT}}/File1 + + + + + {{FILE2_HASH}} + + + http://{{SRV_HOST}}:{{SRV_PORT}}/wrong_file + http://{{SRV_HOST}}:{{SRV_PORT}}/404 + http://{{SRV_HOST}}:{{SRV_PORT}}/File2_lowPref + http://{{SRV_HOST}}:{{SRV_PORT}}/File2 + + + + + {{FILE3_HASH}} + + + http://{{SRV_HOST}}:{{SRV_PORT}}/wrong_file + http://{{SRV_HOST}}:{{SRV_PORT}}/404 + http://{{SRV_HOST}}:{{SRV_PORT}}/File3_lowPref + http://{{SRV_HOST}}:{{SRV_PORT}}/File3 + + + + + {{FILE4_HASH}} + + + http://{{SRV_HOST}}:{{SRV_PORT}}/wrong_file + http://{{SRV_HOST}}:{{SRV_PORT}}/404 + http://{{SRV_HOST}}:{{SRV_PORT}}/File4_lowPref + http://{{SRV_HOST}}:{{SRV_PORT}}/File4 + + + + + {{FILE5_HASH}} + + + http://{{SRV_HOST}}:{{SRV_PORT}}/wrong_file + http://{{SRV_HOST}}:{{SRV_PORT}}/404 + http://{{SRV_HOST}}:{{SRV_PORT}}/File5_lowPref + http://{{SRV_HOST}}:{{SRV_PORT}}/File5 + + + + +""" + +wrong_file = WgetFile ("wrong_file", bad) + +File1_orig = WgetFile ("File1", File1) +File1_down = WgetFile ("test.meta4.#1", File1) +File1_nono = WgetFile ("File1_lowPref", File1_lowPref) + +# rejected by libmetalink +File2_orig = WgetFile ("File2", File2) +File2_nono = WgetFile ("File2_lowPref", File2_lowPref) + +# rejected by libmetalink +File3_orig = WgetFile ("File3", File3) +File3_nono = WgetFile ("File3_lowPref", File3_lowPref) + +# rejected by libmetalink +File4_orig = WgetFile ("File4", File4) +File4_nono = WgetFile ("File4_lowPref", File4_lowPref) + +File5_orig = WgetFile ("File5", File5) +File5_down = WgetFile ("test.meta4.#2", File5) +File5_nono = WgetFile ("File5_lowPref", File5_lowPref) + +MetaFile = WgetFile ("test.meta4", MetaXml) + +WGET_OPTIONS = "--directory-prefix ~/dir --input-metalink test.meta4" +WGET_URLS = [[]] + +Files = [[ + wrong_file, + File1_orig, File1_nono, + File2_orig, File2_nono, + File3_orig, File3_nono, + File4_orig, File4_nono, + File5_orig, File5_nono +]] +Existing_Files = [MetaFile] + +ExpectedReturnCode = 0 +ExpectedDownloadedFiles = [ + File1_down, + File5_down, + MetaFile +] + +################ Pre and Post Test Hooks ##################################### +pre_test = { + "ServerFiles" : Files, + "LocalFiles" : Existing_Files +} +test_options = { + "WgetCommands" : WGET_OPTIONS, + "Urls" : WGET_URLS +} +post_test = { + "ExpectedFiles" : ExpectedDownloadedFiles, + "ExpectedRetcode" : ExpectedReturnCode +} + +http_test = HTTPTest ( + pre_hook=pre_test, + test_params=test_options, + post_hook=post_test, +) + +http_test.server_setup() +### Get and use dynamic server sockname +srv_host, srv_port = http_test.servers[0].server_inst.socket.getsockname () + +MetaXml = re.sub (r'{{FILE1_HASH}}', File1_sha256, MetaXml) +MetaXml = re.sub (r'{{FILE2_HASH}}', File2_sha256, MetaXml) +MetaXml = re.sub (r'{{FILE3_HASH}}', File3_sha256, MetaXml) +MetaXml = re.sub (r'{{FILE4_HASH}}', File4_sha256, MetaXml) +MetaXml = re.sub (r'{{FILE5_HASH}}', File5_sha256, MetaXml) +MetaXml = re.sub (r'{{SRV_HOST}}', srv_host, MetaXml) +MetaXml = re.sub (r'{{SRV_PORT}}', str (srv_port), MetaXml) +MetaFile.content = MetaXml + +err = http_test.begin () + +exit (err) diff --git a/testenv/Test-metalink-xml-prefix-trust.py b/testenv/Test-metalink-xml-prefix-trust.py new file mode 100755 index 0000000..dfc07ed --- /dev/null +++ b/testenv/Test-metalink-xml-prefix-trust.py @@ -0,0 +1,194 @@ +#!/usr/bin/env python3 +from sys import exit +from test.http_test import HTTPTest +from misc.wget_file import WgetFile +import re +import hashlib + +""" + This is to test Metalink/XML relative directory prefix support in Wget. + + With --trust-server-names, trust the metalink:file names. + + Without --trust-server-names, don't trust the metalink:file names: + use the basename of --input-metalink, and add a sequential number + (e.g. .#1, .#2, etc.). + + Strip the directory from unsafe paths. +""" +############# File Definitions ############################################### +bad = "Ouch!" + +File1 = "Would you like some Tea?" +File1_lowPref = "Do not take this" +File1_sha256 = hashlib.sha256 (File1.encode ('UTF-8')).hexdigest () + +File2 = "This is gonna be good" +File2_lowPref = "Not this one too" +File2_sha256 = hashlib.sha256 (File2.encode ('UTF-8')).hexdigest () + +File3 = "A little more, please" +File3_lowPref = "That's just too much" +File3_sha256 = hashlib.sha256 (File3.encode ('UTF-8')).hexdigest () + +File4 = "Maybe a biscuit?" +File4_lowPref = "No, thanks" +File4_sha256 = hashlib.sha256 (File4.encode ('UTF-8')).hexdigest () + +File5 = "More Tea...?" +File5_lowPref = "I have to go..." +File5_sha256 = hashlib.sha256 (File5.encode ('UTF-8')).hexdigest () + +MetaXml = \ +""" + + + GNU Wget + + + GNU GPL + http://www.gnu.org/licenses/gpl.html + + Wget Test Files + 1.2.3 + Wget Test Files description + + + + {{FILE1_HASH}} + + + http://{{SRV_HOST}}:{{SRV_PORT}}/wrong_file + http://{{SRV_HOST}}:{{SRV_PORT}}/404 + http://{{SRV_HOST}}:{{SRV_PORT}}/File1_lowPref + http://{{SRV_HOST}}:{{SRV_PORT}}/File1 + + + + + {{FILE2_HASH}} + + + http://{{SRV_HOST}}:{{SRV_PORT}}/wrong_file + http://{{SRV_HOST}}:{{SRV_PORT}}/404 + http://{{SRV_HOST}}:{{SRV_PORT}}/File2_lowPref + http://{{SRV_HOST}}:{{SRV_PORT}}/File2 + + + + + {{FILE3_HASH}} + + + http://{{SRV_HOST}}:{{SRV_PORT}}/wrong_file + http://{{SRV_HOST}}:{{SRV_PORT}}/404 + http://{{SRV_HOST}}:{{SRV_PORT}}/File3_lowPref + http://{{SRV_HOST}}:{{SRV_PORT}}/File3 + + + + + {{FILE4_HASH}} + + + http://{{SRV_HOST}}:{{SRV_PORT}}/wrong_file + http://{{SRV_HOST}}:{{SRV_PORT}}/404 + http://{{SRV_HOST}}:{{SRV_PORT}}/File4_lowPref + http://{{SRV_HOST}}:{{SRV_PORT}}/File4 + + + + + {{FILE5_HASH}} + + + http://{{SRV_HOST}}:{{SRV_PORT}}/wrong_file + http://{{SRV_HOST}}:{{SRV_PORT}}/404 + http://{{SRV_HOST}}:{{SRV_PORT}}/File5_lowPref + http://{{SRV_HOST}}:{{SRV_PORT}}/File5 + + + + +""" + +wrong_file = WgetFile ("wrong_file", bad) + +File1_orig = WgetFile ("File1", File1) +File1_down = WgetFile ("dir/subdir/File1", File1) +File1_nono = WgetFile ("File1_lowPref", File1_lowPref) + +# rejected by libmetalink +File2_orig = WgetFile ("File2", File2) +File2_nono = WgetFile ("File2_lowPref", File2_lowPref) + +# rejected by libmetalink +File3_orig = WgetFile ("File3", File3) +File3_nono = WgetFile ("File3_lowPref", File3_lowPref) + +# rejected by libmetalink +File4_orig = WgetFile ("File4", File4) +File4_nono = WgetFile ("File4_lowPref", File4_lowPref) + +File5_orig = WgetFile ("File5", File5) +File5_down = WgetFile ("dir/subdir/File5", File5) +File5_nono = WgetFile ("File5_lowPref", File5_lowPref) + +MetaFile = WgetFile ("test.meta4", MetaXml) + +WGET_OPTIONS = "--trust-server-names --directory-prefix dir --input-metalink test.meta4" +WGET_URLS = [[]] + +Files = [[ + wrong_file, + File1_orig, File1_nono, + File2_orig, File2_nono, + File3_orig, File3_nono, + File4_orig, File4_nono, + File5_orig, File5_nono +]] +Existing_Files = [MetaFile] + +ExpectedReturnCode = 0 +ExpectedDownloadedFiles = [ + File1_down, + File5_down, + MetaFile +] + +################ Pre and Post Test Hooks ##################################### +pre_test = { + "ServerFiles" : Files, + "LocalFiles" : Existing_Files +} +test_options = { + "WgetCommands" : WGET_OPTIONS, + "Urls" : WGET_URLS +} +post_test = { + "ExpectedFiles" : ExpectedDownloadedFiles, + "ExpectedRetcode" : ExpectedReturnCode +} + +http_test = HTTPTest ( + pre_hook=pre_test, + test_params=test_options, + post_hook=post_test, +) + +http_test.server_setup() +### Get and use dynamic server sockname +srv_host, srv_port = http_test.servers[0].server_inst.socket.getsockname () + +MetaXml = re.sub (r'{{FILE1_HASH}}', File1_sha256, MetaXml) +MetaXml = re.sub (r'{{FILE2_HASH}}', File2_sha256, MetaXml) +MetaXml = re.sub (r'{{FILE3_HASH}}', File3_sha256, MetaXml) +MetaXml = re.sub (r'{{FILE4_HASH}}', File4_sha256, MetaXml) +MetaXml = re.sub (r'{{FILE5_HASH}}', File5_sha256, MetaXml) +MetaXml = re.sub (r'{{SRV_HOST}}', srv_host, MetaXml) +MetaXml = re.sub (r'{{SRV_PORT}}', str (srv_port), MetaXml) +MetaFile.content = MetaXml + +err = http_test.begin () + +exit (err) diff --git a/testenv/Test-metalink-xml-prefix.py b/testenv/Test-metalink-xml-prefix.py new file mode 100755 index 0000000..816f0ec --- /dev/null +++ b/testenv/Test-metalink-xml-prefix.py @@ -0,0 +1,194 @@ +#!/usr/bin/env python3 +from sys import exit +from test.http_test import HTTPTest +from misc.wget_file import WgetFile +import re +import hashlib + +""" + This is to test Metalink/XML relative directory prefix support in Wget. + + With --trust-server-names, trust the metalink:file names. + + Without --trust-server-names, don't trust the metalink:file names: + use the basename of --input-metalink, and add a sequential number + (e.g. .#1, .#2, etc.). + + Strip the directory from unsafe paths. +""" +############# File Definitions ############################################### +bad = "Ouch!" + +File1 = "Would you like some Tea?" +File1_lowPref = "Do not take this" +File1_sha256 = hashlib.sha256 (File1.encode ('UTF-8')).hexdigest () + +File2 = "This is gonna be good" +File2_lowPref = "Not this one too" +File2_sha256 = hashlib.sha256 (File2.encode ('UTF-8')).hexdigest () + +File3 = "A little more, please" +File3_lowPref = "That's just too much" +File3_sha256 = hashlib.sha256 (File3.encode ('UTF-8')).hexdigest () + +File4 = "Maybe a biscuit?" +File4_lowPref = "No, thanks" +File4_sha256 = hashlib.sha256 (File4.encode ('UTF-8')).hexdigest () + +File5 = "More Tea...?" +File5_lowPref = "I have to go..." +File5_sha256 = hashlib.sha256 (File5.encode ('UTF-8')).hexdigest () + +MetaXml = \ +""" + + + GNU Wget + + + GNU GPL + http://www.gnu.org/licenses/gpl.html + + Wget Test Files + 1.2.3 + Wget Test Files description + + + + {{FILE1_HASH}} + + + http://{{SRV_HOST}}:{{SRV_PORT}}/wrong_file + http://{{SRV_HOST}}:{{SRV_PORT}}/404 + http://{{SRV_HOST}}:{{SRV_PORT}}/File1_lowPref + http://{{SRV_HOST}}:{{SRV_PORT}}/File1 + + + + + {{FILE2_HASH}} + + + http://{{SRV_HOST}}:{{SRV_PORT}}/wrong_file + http://{{SRV_HOST}}:{{SRV_PORT}}/404 + http://{{SRV_HOST}}:{{SRV_PORT}}/File2_lowPref + http://{{SRV_HOST}}:{{SRV_PORT}}/File2 + + + + + {{FILE3_HASH}} + + + http://{{SRV_HOST}}:{{SRV_PORT}}/wrong_file + http://{{SRV_HOST}}:{{SRV_PORT}}/404 + http://{{SRV_HOST}}:{{SRV_PORT}}/File3_lowPref + http://{{SRV_HOST}}:{{SRV_PORT}}/File3 + + + + + {{FILE4_HASH}} + + + http://{{SRV_HOST}}:{{SRV_PORT}}/wrong_file + http://{{SRV_HOST}}:{{SRV_PORT}}/404 + http://{{SRV_HOST}}:{{SRV_PORT}}/File4_lowPref + http://{{SRV_HOST}}:{{SRV_PORT}}/File4 + + + + + {{FILE5_HASH}} + + + http://{{SRV_HOST}}:{{SRV_PORT}}/wrong_file + http://{{SRV_HOST}}:{{SRV_PORT}}/404 + http://{{SRV_HOST}}:{{SRV_PORT}}/File5_lowPref + http://{{SRV_HOST}}:{{SRV_PORT}}/File5 + + + + +""" + +wrong_file = WgetFile ("wrong_file", bad) + +File1_orig = WgetFile ("File1", File1) +File1_down = WgetFile ("dir/test.meta4.#1", File1) +File1_nono = WgetFile ("File1_lowPref", File1_lowPref) + +# rejected by libmetalink +File2_orig = WgetFile ("File2", File2) +File2_nono = WgetFile ("File2_lowPref", File2_lowPref) + +# rejected by libmetalink +File3_orig = WgetFile ("File3", File3) +File3_nono = WgetFile ("File3_lowPref", File3_lowPref) + +# rejected by libmetalink +File4_orig = WgetFile ("File4", File4) +File4_nono = WgetFile ("File4_lowPref", File4_lowPref) + +File5_orig = WgetFile ("File5", File5) +File5_down = WgetFile ("dir/test.meta4.#2", File5) +File5_nono = WgetFile ("File5_lowPref", File5_lowPref) + +MetaFile = WgetFile ("test.meta4", MetaXml) + +WGET_OPTIONS = "--directory-prefix dir --input-metalink test.meta4" +WGET_URLS = [[]] + +Files = [[ + wrong_file, + File1_orig, File1_nono, + File2_orig, File2_nono, + File3_orig, File3_nono, + File4_orig, File4_nono, + File5_orig, File5_nono +]] +Existing_Files = [MetaFile] + +ExpectedReturnCode = 0 +ExpectedDownloadedFiles = [ + File1_down, + File5_down, + MetaFile +] + +################ Pre and Post Test Hooks ##################################### +pre_test = { + "ServerFiles" : Files, + "LocalFiles" : Existing_Files +} +test_options = { + "WgetCommands" : WGET_OPTIONS, + "Urls" : WGET_URLS +} +post_test = { + "ExpectedFiles" : ExpectedDownloadedFiles, + "ExpectedRetcode" : ExpectedReturnCode +} + +http_test = HTTPTest ( + pre_hook=pre_test, + test_params=test_options, + post_hook=post_test, +) + +http_test.server_setup() +### Get and use dynamic server sockname +srv_host, srv_port = http_test.servers[0].server_inst.socket.getsockname () + +MetaXml = re.sub (r'{{FILE1_HASH}}', File1_sha256, MetaXml) +MetaXml = re.sub (r'{{FILE2_HASH}}', File2_sha256, MetaXml) +MetaXml = re.sub (r'{{FILE3_HASH}}', File3_sha256, MetaXml) +MetaXml = re.sub (r'{{FILE4_HASH}}', File4_sha256, MetaXml) +MetaXml = re.sub (r'{{FILE5_HASH}}', File5_sha256, MetaXml) +MetaXml = re.sub (r'{{SRV_HOST}}', srv_host, MetaXml) +MetaXml = re.sub (r'{{SRV_PORT}}', str (srv_port), MetaXml) +MetaFile.content = MetaXml + +err = http_test.begin () + +exit (err) diff --git a/testenv/Test-metalink-xml-relpath-trust.py b/testenv/Test-metalink-xml-relpath-trust.py new file mode 100755 index 0000000..b9b8287 --- /dev/null +++ b/testenv/Test-metalink-xml-relpath-trust.py @@ -0,0 +1,193 @@ +#!/usr/bin/env python3 +from sys import exit +from test.http_test import HTTPTest +from misc.wget_file import WgetFile +import re +import hashlib + +""" + This is to test if Metalink/XML forbids relative paths. + + With --trust-server-names, trust the metalink:file names. + + Without --trust-server-names, don't trust the metalink:file names: + use the basename of --input-metalink, and add a sequential number + (e.g. .#1, .#2, etc.). + + Strip the directory from unsafe paths. +""" +############# File Definitions ############################################### +bad = "Ouch!" + +File1 = "Would you like some Tea?" +File1_lowPref = "Do not take this" +File1_sha256 = hashlib.sha256 (File1.encode ('UTF-8')).hexdigest () + +File2 = "This is gonna be good" +File2_lowPref = "Not this one too" +File2_sha256 = hashlib.sha256 (File2.encode ('UTF-8')).hexdigest () + +File3 = "A little more, please" +File3_lowPref = "That's just too much" +File3_sha256 = hashlib.sha256 (File3.encode ('UTF-8')).hexdigest () + +File4 = "Maybe a biscuit?" +File4_lowPref = "No, thanks" +File4_sha256 = hashlib.sha256 (File4.encode ('UTF-8')).hexdigest () + +File5 = "More Tea...?" +File5_lowPref = "I have to go..." +File5_sha256 = hashlib.sha256 (File5.encode ('UTF-8')).hexdigest () + +MetaXml = \ +""" + + + GNU Wget + + + GNU GPL + http://www.gnu.org/licenses/gpl.html + + Wget Test Files + 1.2.3 + Wget Test Files description + + + + {{FILE1_HASH}} + + + http://{{SRV_HOST}}:{{SRV_PORT}}/wrong_file + http://{{SRV_HOST}}:{{SRV_PORT}}/404 + http://{{SRV_HOST}}:{{SRV_PORT}}/File1_lowPref + http://{{SRV_HOST}}:{{SRV_PORT}}/File1 + + + + + {{FILE2_HASH}} + + + http://{{SRV_HOST}}:{{SRV_PORT}}/wrong_file + http://{{SRV_HOST}}:{{SRV_PORT}}/404 + http://{{SRV_HOST}}:{{SRV_PORT}}/File2_lowPref + http://{{SRV_HOST}}:{{SRV_PORT}}/File2 + + + + + {{FILE3_HASH}} + + + http://{{SRV_HOST}}:{{SRV_PORT}}/wrong_file + http://{{SRV_HOST}}:{{SRV_PORT}}/404 + http://{{SRV_HOST}}:{{SRV_PORT}}/File3_lowPref + http://{{SRV_HOST}}:{{SRV_PORT}}/File3 + + + + + {{FILE4_HASH}} + + + http://{{SRV_HOST}}:{{SRV_PORT}}/wrong_file + http://{{SRV_HOST}}:{{SRV_PORT}}/404 + http://{{SRV_HOST}}:{{SRV_PORT}}/File4_lowPref + http://{{SRV_HOST}}:{{SRV_PORT}}/File4 + + + + + {{FILE5_HASH}} + + + http://{{SRV_HOST}}:{{SRV_PORT}}/wrong_file + http://{{SRV_HOST}}:{{SRV_PORT}}/404 + http://{{SRV_HOST}}:{{SRV_PORT}}/File5_lowPref + http://{{SRV_HOST}}:{{SRV_PORT}}/File5 + + + + +""" + +wrong_file = WgetFile ("wrong_file", bad) + +# rejected by libmetalink +File1_orig = WgetFile ("File1", File1) +File1_nono = WgetFile ("File1_lowPref", File1_lowPref) + +# rejected by libmetalink +File2_orig = WgetFile ("File2", File2) +File2_nono = WgetFile ("File2_lowPref", File2_lowPref) + +# rejected by libmetalink +File3_orig = WgetFile ("File3", File3) +File3_nono = WgetFile ("File3_lowPref", File3_lowPref) + +# rejected by libmetalink +File4_orig = WgetFile ("File4", File4) +File4_nono = WgetFile ("File4_lowPref", File4_lowPref) + +File5_orig = WgetFile ("File5", File5) +File5_down = WgetFile ("File5", File5) +File5_nono = WgetFile ("File5_lowPref", File5_lowPref) + +MetaFile = WgetFile ("test.meta4", MetaXml) + +WGET_OPTIONS = "--trust-server-names --input-metalink test.meta4" +WGET_URLS = [[]] + +Files = [[ + wrong_file, + File1_orig, File1_nono, + File2_orig, File2_nono, + File3_orig, File3_nono, + File4_orig, File4_nono, + File5_orig, File5_nono +]] +Existing_Files = [MetaFile] + +ExpectedReturnCode = 0 +ExpectedDownloadedFiles = [ + File5_down, + MetaFile +] + +################ Pre and Post Test Hooks ##################################### +pre_test = { + "ServerFiles" : Files, + "LocalFiles" : Existing_Files +} +test_options = { + "WgetCommands" : WGET_OPTIONS, + "Urls" : WGET_URLS +} +post_test = { + "ExpectedFiles" : ExpectedDownloadedFiles, + "ExpectedRetcode" : ExpectedReturnCode +} + +http_test = HTTPTest ( + pre_hook=pre_test, + test_params=test_options, + post_hook=post_test, +) + +http_test.server_setup() +### Get and use dynamic server sockname +srv_host, srv_port = http_test.servers[0].server_inst.socket.getsockname () + +MetaXml = re.sub (r'{{FILE1_HASH}}', File1_sha256, MetaXml) +MetaXml = re.sub (r'{{FILE2_HASH}}', File2_sha256, MetaXml) +MetaXml = re.sub (r'{{FILE3_HASH}}', File3_sha256, MetaXml) +MetaXml = re.sub (r'{{FILE4_HASH}}', File4_sha256, MetaXml) +MetaXml = re.sub (r'{{FILE5_HASH}}', File5_sha256, MetaXml) +MetaXml = re.sub (r'{{SRV_HOST}}', srv_host, MetaXml) +MetaXml = re.sub (r'{{SRV_PORT}}', str (srv_port), MetaXml) +MetaFile.content = MetaXml + +err = http_test.begin () + +exit (err) diff --git a/testenv/Test-metalink-xml-relpath.py b/testenv/Test-metalink-xml-relpath.py index e146b97..579a835 100755 --- a/testenv/Test-metalink-xml-relpath.py +++ b/testenv/Test-metalink-xml-relpath.py @@ -6,12 +6,39 @@ import re import hashlib """ - This is to test if Metalink XML file escapes current directory. + This is to test if Metalink/XML forbids relative paths. + + With --trust-server-names, trust the metalink:file names. + + Without --trust-server-names, don't trust the metalink:file names: + use the basename of --input-metalink, and add a sequential number + (e.g. .#1, .#2, etc.). + + Strip the directory from unsafe paths. """ ############# File Definitions ############################################### +bad = "Ouch!" + File1 = "Would you like some Tea?" File1_lowPref = "Do not take this" File1_sha256 = hashlib.sha256 (File1.encode ('UTF-8')).hexdigest () + +File2 = "This is gonna be good" +File2_lowPref = "Not this one too" +File2_sha256 = hashlib.sha256 (File2.encode ('UTF-8')).hexdigest () + +File3 = "A little more, please" +File3_lowPref = "That's just too much" +File3_sha256 = hashlib.sha256 (File3.encode ('UTF-8')).hexdigest () + +File4 = "Maybe a biscuit?" +File4_lowPref = "No, thanks" +File4_sha256 = hashlib.sha256 (File4.encode ('UTF-8')).hexdigest () + +File5 = "More Tea...?" +File5_lowPref = "I have to go..." +File5_sha256 = hashlib.sha256 (File5.encode ('UTF-8')).hexdigest () + MetaXml = \ """ @@ -22,76 +49,111 @@ MetaXml = \ GNU GPL http://www.gnu.org/licenses/gpl.html - Wget Test File 1 + Wget Test Files 1.2.3 - Wget Test File 1 description + Wget Test Files description - - - {{FILE1_HASH}} - - - http://{{SRV_HOST}}:{{SRV_PORT}}/File1 - - - + {{FILE1_HASH}} + http://{{SRV_HOST}}:{{SRV_PORT}}/wrong_file + http://{{SRV_HOST}}:{{SRV_PORT}}/404 + http://{{SRV_HOST}}:{{SRV_PORT}}/File1_lowPref http://{{SRV_HOST}}:{{SRV_PORT}}/File1 - + - {{FILE1_HASH}} + {{FILE2_HASH}} - http://{{SRV_HOST}}:{{SRV_PORT}}/File1 + http://{{SRV_HOST}}:{{SRV_PORT}}/wrong_file + http://{{SRV_HOST}}:{{SRV_PORT}}/404 + http://{{SRV_HOST}}:{{SRV_PORT}}/File2_lowPref + http://{{SRV_HOST}}:{{SRV_PORT}}/File2 - + - {{FILE1_HASH}} + {{FILE3_HASH}} - http://{{SRV_HOST}}:{{SRV_PORT}}/File1 + http://{{SRV_HOST}}:{{SRV_PORT}}/wrong_file + http://{{SRV_HOST}}:{{SRV_PORT}}/404 + http://{{SRV_HOST}}:{{SRV_PORT}}/File3_lowPref + http://{{SRV_HOST}}:{{SRV_PORT}}/File3 - + - {{FILE1_HASH}} + {{FILE4_HASH}} - http://{{SRV_HOST}}:{{SRV_PORT}}/File1 + http://{{SRV_HOST}}:{{SRV_PORT}}/wrong_file + http://{{SRV_HOST}}:{{SRV_PORT}}/404 + http://{{SRV_HOST}}:{{SRV_PORT}}/File4_lowPref + http://{{SRV_HOST}}:{{SRV_PORT}}/File4 - + - {{FILE1_HASH}} + {{FILE5_HASH}} - http://broken.example/File1 - http://{{SRV_HOST}}:{{SRV_PORT}}/File1_lowPref - http://{{SRV_HOST}}:{{SRV_PORT}}/File1 + http://{{SRV_HOST}}:{{SRV_PORT}}/wrong_file + http://{{SRV_HOST}}:{{SRV_PORT}}/404 + http://{{SRV_HOST}}:{{SRV_PORT}}/File5_lowPref + http://{{SRV_HOST}}:{{SRV_PORT}}/File5 """ -A_File = WgetFile ("File1", File1) -B_File = WgetFile ("File1_lowPref", File1_lowPref) +wrong_file = WgetFile ("wrong_file", bad) + +# rejected by libmetalink +File1_orig = WgetFile ("File1", File1) +File1_nono = WgetFile ("File1_lowPref", File1_lowPref) + +# rejected by libmetalink +File2_orig = WgetFile ("File2", File2) +File2_nono = WgetFile ("File2_lowPref", File2_lowPref) + +# rejected by libmetalink +File3_orig = WgetFile ("File3", File3) +File3_nono = WgetFile ("File3_lowPref", File3_lowPref) + +# rejected by libmetalink +File4_orig = WgetFile ("File4", File4) +File4_nono = WgetFile ("File4_lowPref", File4_lowPref) + +File5_orig = WgetFile ("File5", File5) +File5_down = WgetFile ("test.meta4.#1", File5) +File5_nono = WgetFile ("File5_lowPref", File5_lowPref) + MetaFile = WgetFile ("test.meta4", MetaXml) WGET_OPTIONS = "--input-metalink test.meta4" WGET_URLS = [[]] -Files = [[A_File, B_File]] +Files = [[ + wrong_file, + File1_orig, File1_nono, + File2_orig, File2_nono, + File3_orig, File3_nono, + File4_orig, File4_nono, + File5_orig, File5_nono +]] Existing_Files = [MetaFile] ExpectedReturnCode = 0 -ExpectedDownloadedFiles = [WgetFile ("dir/subdir/File1", File1), MetaFile] +ExpectedDownloadedFiles = [ + File5_down, + MetaFile +] ################ Pre and Post Test Hooks ##################################### pre_test = { @@ -118,6 +180,10 @@ http_test.server_setup() srv_host, srv_port = http_test.servers[0].server_inst.socket.getsockname () MetaXml = re.sub (r'{{FILE1_HASH}}', File1_sha256, MetaXml) +MetaXml = re.sub (r'{{FILE2_HASH}}', File2_sha256, MetaXml) +MetaXml = re.sub (r'{{FILE3_HASH}}', File3_sha256, MetaXml) +MetaXml = re.sub (r'{{FILE4_HASH}}', File4_sha256, MetaXml) +MetaXml = re.sub (r'{{FILE5_HASH}}', File5_sha256, MetaXml) MetaXml = re.sub (r'{{SRV_HOST}}', srv_host, MetaXml) MetaXml = re.sub (r'{{SRV_PORT}}', str (srv_port), MetaXml) MetaFile.content = MetaXml diff --git a/testenv/Test-metalink-xml-relprefix-trust.py b/testenv/Test-metalink-xml-relprefix-trust.py new file mode 100755 index 0000000..f891583 --- /dev/null +++ b/testenv/Test-metalink-xml-relprefix-trust.py @@ -0,0 +1,194 @@ +#!/usr/bin/env python3 +from sys import exit +from test.http_test import HTTPTest +from misc.wget_file import WgetFile +import re +import hashlib + +""" + This is to test Metalink/XML relative directory prefix support in Wget. + + With --trust-server-names, trust the metalink:file names. + + Without --trust-server-names, don't trust the metalink:file names: + use the basename of --input-metalink, and add a sequential number + (e.g. .#1, .#2, etc.). + + Strip the directory from unsafe paths. +""" +############# File Definitions ############################################### +bad = "Ouch!" + +File1 = "Would you like some Tea?" +File1_lowPref = "Do not take this" +File1_sha256 = hashlib.sha256 (File1.encode ('UTF-8')).hexdigest () + +File2 = "This is gonna be good" +File2_lowPref = "Not this one too" +File2_sha256 = hashlib.sha256 (File2.encode ('UTF-8')).hexdigest () + +File3 = "A little more, please" +File3_lowPref = "That's just too much" +File3_sha256 = hashlib.sha256 (File3.encode ('UTF-8')).hexdigest () + +File4 = "Maybe a biscuit?" +File4_lowPref = "No, thanks" +File4_sha256 = hashlib.sha256 (File4.encode ('UTF-8')).hexdigest () + +File5 = "More Tea...?" +File5_lowPref = "I have to go..." +File5_sha256 = hashlib.sha256 (File5.encode ('UTF-8')).hexdigest () + +MetaXml = \ +""" + + + GNU Wget + + + GNU GPL + http://www.gnu.org/licenses/gpl.html + + Wget Test Files + 1.2.3 + Wget Test Files description + + + + {{FILE1_HASH}} + + + http://{{SRV_HOST}}:{{SRV_PORT}}/wrong_file + http://{{SRV_HOST}}:{{SRV_PORT}}/404 + http://{{SRV_HOST}}:{{SRV_PORT}}/File1_lowPref + http://{{SRV_HOST}}:{{SRV_PORT}}/File1 + + + + + {{FILE2_HASH}} + + + http://{{SRV_HOST}}:{{SRV_PORT}}/wrong_file + http://{{SRV_HOST}}:{{SRV_PORT}}/404 + http://{{SRV_HOST}}:{{SRV_PORT}}/File2_lowPref + http://{{SRV_HOST}}:{{SRV_PORT}}/File2 + + + + + {{FILE3_HASH}} + + + http://{{SRV_HOST}}:{{SRV_PORT}}/wrong_file + http://{{SRV_HOST}}:{{SRV_PORT}}/404 + http://{{SRV_HOST}}:{{SRV_PORT}}/File3_lowPref + http://{{SRV_HOST}}:{{SRV_PORT}}/File3 + + + + + {{FILE4_HASH}} + + + http://{{SRV_HOST}}:{{SRV_PORT}}/wrong_file + http://{{SRV_HOST}}:{{SRV_PORT}}/404 + http://{{SRV_HOST}}:{{SRV_PORT}}/File4_lowPref + http://{{SRV_HOST}}:{{SRV_PORT}}/File4 + + + + + {{FILE5_HASH}} + + + http://{{SRV_HOST}}:{{SRV_PORT}}/wrong_file + http://{{SRV_HOST}}:{{SRV_PORT}}/404 + http://{{SRV_HOST}}:{{SRV_PORT}}/File5_lowPref + http://{{SRV_HOST}}:{{SRV_PORT}}/File5 + + + + +""" + +wrong_file = WgetFile ("wrong_file", bad) + +File1_orig = WgetFile ("File1", File1) +File1_down = WgetFile ("File1", File1) +File1_nono = WgetFile ("File1_lowPref", File1_lowPref) + +# rejected by libmetalink +File2_orig = WgetFile ("File2", File2) +File2_nono = WgetFile ("File2_lowPref", File2_lowPref) + +# rejected by libmetalink +File3_orig = WgetFile ("File3", File3) +File3_nono = WgetFile ("File3_lowPref", File3_lowPref) + +# rejected by libmetalink +File4_orig = WgetFile ("File4", File4) +File4_nono = WgetFile ("File4_lowPref", File4_lowPref) + +File5_orig = WgetFile ("File5", File5) +File5_down = WgetFile ("File5", File5) +File5_nono = WgetFile ("File5_lowPref", File5_lowPref) + +MetaFile = WgetFile ("test.meta4", MetaXml) + +WGET_OPTIONS = "--trust-server-names --directory-prefix ../dir --input-metalink test.meta4" +WGET_URLS = [[]] + +Files = [[ + wrong_file, + File1_orig, File1_nono, + File2_orig, File2_nono, + File3_orig, File3_nono, + File4_orig, File4_nono, + File5_orig, File5_nono +]] +Existing_Files = [MetaFile] + +ExpectedReturnCode = 0 +ExpectedDownloadedFiles = [ + File1_down, + File5_down, + MetaFile +] + +################ Pre and Post Test Hooks ##################################### +pre_test = { + "ServerFiles" : Files, + "LocalFiles" : Existing_Files +} +test_options = { + "WgetCommands" : WGET_OPTIONS, + "Urls" : WGET_URLS +} +post_test = { + "ExpectedFiles" : ExpectedDownloadedFiles, + "ExpectedRetcode" : ExpectedReturnCode +} + +http_test = HTTPTest ( + pre_hook=pre_test, + test_params=test_options, + post_hook=post_test, +) + +http_test.server_setup() +### Get and use dynamic server sockname +srv_host, srv_port = http_test.servers[0].server_inst.socket.getsockname () + +MetaXml = re.sub (r'{{FILE1_HASH}}', File1_sha256, MetaXml) +MetaXml = re.sub (r'{{FILE2_HASH}}', File2_sha256, MetaXml) +MetaXml = re.sub (r'{{FILE3_HASH}}', File3_sha256, MetaXml) +MetaXml = re.sub (r'{{FILE4_HASH}}', File4_sha256, MetaXml) +MetaXml = re.sub (r'{{FILE5_HASH}}', File5_sha256, MetaXml) +MetaXml = re.sub (r'{{SRV_HOST}}', srv_host, MetaXml) +MetaXml = re.sub (r'{{SRV_PORT}}', str (srv_port), MetaXml) +MetaFile.content = MetaXml + +err = http_test.begin () + +exit (err) diff --git a/testenv/Test-metalink-xml-relprefix.py b/testenv/Test-metalink-xml-relprefix.py new file mode 100755 index 0000000..8718ed1 --- /dev/null +++ b/testenv/Test-metalink-xml-relprefix.py @@ -0,0 +1,194 @@ +#!/usr/bin/env python3 +from sys import exit +from test.http_test import HTTPTest +from misc.wget_file import WgetFile +import re +import hashlib + +""" + This is to test Metalink/XML relative directory prefix support in Wget. + + With --trust-server-names, trust the metalink:file names. + + Without --trust-server-names, don't trust the metalink:file names: + use the basename of --input-metalink, and add a sequential number + (e.g. .#1, .#2, etc.). + + Strip the directory from unsafe paths. +""" +############# File Definitions ############################################### +bad = "Ouch!" + +File1 = "Would you like some Tea?" +File1_lowPref = "Do not take this" +File1_sha256 = hashlib.sha256 (File1.encode ('UTF-8')).hexdigest () + +File2 = "This is gonna be good" +File2_lowPref = "Not this one too" +File2_sha256 = hashlib.sha256 (File2.encode ('UTF-8')).hexdigest () + +File3 = "A little more, please" +File3_lowPref = "That's just too much" +File3_sha256 = hashlib.sha256 (File3.encode ('UTF-8')).hexdigest () + +File4 = "Maybe a biscuit?" +File4_lowPref = "No, thanks" +File4_sha256 = hashlib.sha256 (File4.encode ('UTF-8')).hexdigest () + +File5 = "More Tea...?" +File5_lowPref = "I have to go..." +File5_sha256 = hashlib.sha256 (File5.encode ('UTF-8')).hexdigest () + +MetaXml = \ +""" + + + GNU Wget + + + GNU GPL + http://www.gnu.org/licenses/gpl.html + + Wget Test Files + 1.2.3 + Wget Test Files description + + + + {{FILE1_HASH}} + + + http://{{SRV_HOST}}:{{SRV_PORT}}/wrong_file + http://{{SRV_HOST}}:{{SRV_PORT}}/404 + http://{{SRV_HOST}}:{{SRV_PORT}}/File1_lowPref + http://{{SRV_HOST}}:{{SRV_PORT}}/File1 + + + + + {{FILE2_HASH}} + + + http://{{SRV_HOST}}:{{SRV_PORT}}/wrong_file + http://{{SRV_HOST}}:{{SRV_PORT}}/404 + http://{{SRV_HOST}}:{{SRV_PORT}}/File2_lowPref + http://{{SRV_HOST}}:{{SRV_PORT}}/File2 + + + + + {{FILE3_HASH}} + + + http://{{SRV_HOST}}:{{SRV_PORT}}/wrong_file + http://{{SRV_HOST}}:{{SRV_PORT}}/404 + http://{{SRV_HOST}}:{{SRV_PORT}}/File3_lowPref + http://{{SRV_HOST}}:{{SRV_PORT}}/File3 + + + + + {{FILE4_HASH}} + + + http://{{SRV_HOST}}:{{SRV_PORT}}/wrong_file + http://{{SRV_HOST}}:{{SRV_PORT}}/404 + http://{{SRV_HOST}}:{{SRV_PORT}}/File4_lowPref + http://{{SRV_HOST}}:{{SRV_PORT}}/File4 + + + + + {{FILE5_HASH}} + + + http://{{SRV_HOST}}:{{SRV_PORT}}/wrong_file + http://{{SRV_HOST}}:{{SRV_PORT}}/404 + http://{{SRV_HOST}}:{{SRV_PORT}}/File5_lowPref + http://{{SRV_HOST}}:{{SRV_PORT}}/File5 + + + + +""" + +wrong_file = WgetFile ("wrong_file", bad) + +File1_orig = WgetFile ("File1", File1) +File1_down = WgetFile ("test.meta4.#1", File1) +File1_nono = WgetFile ("File1_lowPref", File1_lowPref) + +# rejected by libmetalink +File2_orig = WgetFile ("File2", File2) +File2_nono = WgetFile ("File2_lowPref", File2_lowPref) + +# rejected by libmetalink +File3_orig = WgetFile ("File3", File3) +File3_nono = WgetFile ("File3_lowPref", File3_lowPref) + +# rejected by libmetalink +File4_orig = WgetFile ("File4", File4) +File4_nono = WgetFile ("File4_lowPref", File4_lowPref) + +File5_orig = WgetFile ("File5", File5) +File5_down = WgetFile ("test.meta4.#2", File5) +File5_nono = WgetFile ("File5_lowPref", File5_lowPref) + +MetaFile = WgetFile ("test.meta4", MetaXml) + +WGET_OPTIONS = "--directory-prefix ../dir --input-metalink test.meta4" +WGET_URLS = [[]] + +Files = [[ + wrong_file, + File1_orig, File1_nono, + File2_orig, File2_nono, + File3_orig, File3_nono, + File4_orig, File4_nono, + File5_orig, File5_nono +]] +Existing_Files = [MetaFile] + +ExpectedReturnCode = 0 +ExpectedDownloadedFiles = [ + File1_down, + File5_down, + MetaFile +] + +################ Pre and Post Test Hooks ##################################### +pre_test = { + "ServerFiles" : Files, + "LocalFiles" : Existing_Files +} +test_options = { + "WgetCommands" : WGET_OPTIONS, + "Urls" : WGET_URLS +} +post_test = { + "ExpectedFiles" : ExpectedDownloadedFiles, + "ExpectedRetcode" : ExpectedReturnCode +} + +http_test = HTTPTest ( + pre_hook=pre_test, + test_params=test_options, + post_hook=post_test, +) + +http_test.server_setup() +### Get and use dynamic server sockname +srv_host, srv_port = http_test.servers[0].server_inst.socket.getsockname () + +MetaXml = re.sub (r'{{FILE1_HASH}}', File1_sha256, MetaXml) +MetaXml = re.sub (r'{{FILE2_HASH}}', File2_sha256, MetaXml) +MetaXml = re.sub (r'{{FILE3_HASH}}', File3_sha256, MetaXml) +MetaXml = re.sub (r'{{FILE4_HASH}}', File4_sha256, MetaXml) +MetaXml = re.sub (r'{{FILE5_HASH}}', File5_sha256, MetaXml) +MetaXml = re.sub (r'{{SRV_HOST}}', srv_host, MetaXml) +MetaXml = re.sub (r'{{SRV_PORT}}', str (srv_port), MetaXml) +MetaFile.content = MetaXml + +err = http_test.begin () + +exit (err) diff --git a/testenv/Test-metalink-xml-trust.py b/testenv/Test-metalink-xml-trust.py new file mode 100755 index 0000000..d916bc5 --- /dev/null +++ b/testenv/Test-metalink-xml-trust.py @@ -0,0 +1,197 @@ +#!/usr/bin/env python3 +from sys import exit +from test.http_test import HTTPTest +from misc.wget_file import WgetFile +import re +import hashlib + +""" + This is to test Metalink/XML file support in Wget. + + With --trust-server-names, trust the metalink:file names. + + Without --trust-server-names, don't trust the metalink:file names: + use the basename of --input-metalink, and add a sequential number + (e.g. .#1, .#2, etc.). + + Strip the directory from unsafe paths. +""" +############# File Definitions ############################################### +bad = "Ouch!" + +File1 = "Would you like some Tea?" +File1_lowPref = "Do not take this" +File1_sha256 = hashlib.sha256 (File1.encode ('UTF-8')).hexdigest () + +File2 = "This is gonna be good" +File2_lowPref = "Not this one too" +File2_sha256 = hashlib.sha256 (File2.encode ('UTF-8')).hexdigest () + +File3 = "A little more, please" +File3_lowPref = "That's just too much" +File3_sha256 = hashlib.sha256 (File3.encode ('UTF-8')).hexdigest () + +File4 = "Maybe a biscuit?" +File4_lowPref = "No, thanks" +File4_sha256 = hashlib.sha256 (File4.encode ('UTF-8')).hexdigest () + +File5 = "More Tea...?" +File5_lowPref = "I have to go..." +File5_sha256 = hashlib.sha256 (File5.encode ('UTF-8')).hexdigest () + +MetaXml = \ +""" + + + GNU Wget + + + GNU GPL + http://www.gnu.org/licenses/gpl.html + + Wget Test Files + 1.2.3 + Wget Test Files description + + + + {{FILE1_HASH}} + + + http://{{SRV_HOST}}:{{SRV_PORT}}/wrong_file + http://{{SRV_HOST}}:{{SRV_PORT}}/404 + http://{{SRV_HOST}}:{{SRV_PORT}}/File1_lowPref + http://{{SRV_HOST}}:{{SRV_PORT}}/File1 + + + + + {{FILE2_HASH}} + + + http://{{SRV_HOST}}:{{SRV_PORT}}/wrong_file + http://{{SRV_HOST}}:{{SRV_PORT}}/404 + http://{{SRV_HOST}}:{{SRV_PORT}}/File2_lowPref + http://{{SRV_HOST}}:{{SRV_PORT}}/File2 + + + + + {{FILE3_HASH}} + + + http://{{SRV_HOST}}:{{SRV_PORT}}/wrong_file + http://{{SRV_HOST}}:{{SRV_PORT}}/404 + http://{{SRV_HOST}}:{{SRV_PORT}}/File3_lowPref + http://{{SRV_HOST}}:{{SRV_PORT}}/File3 + + + + + {{FILE4_HASH}} + + + http://{{SRV_HOST}}:{{SRV_PORT}}/wrong_file + http://{{SRV_HOST}}:{{SRV_PORT}}/404 + http://{{SRV_HOST}}:{{SRV_PORT}}/File4_lowPref + http://{{SRV_HOST}}:{{SRV_PORT}}/File4 + + + + + {{FILE5_HASH}} + + + http://{{SRV_HOST}}:{{SRV_PORT}}/wrong_file + http://{{SRV_HOST}}:{{SRV_PORT}}/404 + http://{{SRV_HOST}}:{{SRV_PORT}}/File5_lowPref + http://{{SRV_HOST}}:{{SRV_PORT}}/File5 + + + + +""" + +wrong_file = WgetFile ("wrong_file", bad) + +File1_orig = WgetFile ("File1", File1) +File1_down = WgetFile ("File1", File1) +File1_nono = WgetFile ("File1_lowPref", File1_lowPref) + +File2_orig = WgetFile ("File2", File2) +File2_down = WgetFile ("File2", File2) +File2_nono = WgetFile ("File2_lowPref", File2_lowPref) + +File3_orig = WgetFile ("File3", File3) +File3_down = WgetFile ("File3", File3) +File3_nono = WgetFile ("File3_lowPref", File3_lowPref) + +File4_orig = WgetFile ("File4", File4) +File4_down = WgetFile ("File4", File4) +File4_nono = WgetFile ("File4_lowPref", File4_lowPref) + +File5_orig = WgetFile ("File5", File5) +File5_down = WgetFile ("File5", File5) +File5_nono = WgetFile ("File5_lowPref", File5_lowPref) + +MetaFile = WgetFile ("test.meta4", MetaXml) + +WGET_OPTIONS = "--trust-server-names --input-metalink test.meta4" +WGET_URLS = [[]] + +Files = [[ + wrong_file, + File1_orig, File1_nono, + File2_orig, File2_nono, + File3_orig, File3_nono, + File4_orig, File4_nono, + File5_orig, File5_nono +]] +Existing_Files = [MetaFile] + +ExpectedReturnCode = 0 +ExpectedDownloadedFiles = [ + File1_down, + File2_down, + File3_down, + File4_down, + File5_down, + MetaFile +] + +################ Pre and Post Test Hooks ##################################### +pre_test = { + "ServerFiles" : Files, + "LocalFiles" : Existing_Files +} +test_options = { + "WgetCommands" : WGET_OPTIONS, + "Urls" : WGET_URLS +} +post_test = { + "ExpectedFiles" : ExpectedDownloadedFiles, + "ExpectedRetcode" : ExpectedReturnCode +} + +http_test = HTTPTest ( + pre_hook=pre_test, + test_params=test_options, + post_hook=post_test, +) + +http_test.server_setup() +### Get and use dynamic server sockname +srv_host, srv_port = http_test.servers[0].server_inst.socket.getsockname () + +MetaXml = re.sub (r'{{FILE1_HASH}}', File1_sha256, MetaXml) +MetaXml = re.sub (r'{{FILE2_HASH}}', File2_sha256, MetaXml) +MetaXml = re.sub (r'{{FILE3_HASH}}', File3_sha256, MetaXml) +MetaXml = re.sub (r'{{FILE4_HASH}}', File4_sha256, MetaXml) +MetaXml = re.sub (r'{{FILE5_HASH}}', File5_sha256, MetaXml) +MetaXml = re.sub (r'{{SRV_HOST}}', srv_host, MetaXml) +MetaXml = re.sub (r'{{SRV_PORT}}', str (srv_port), MetaXml) +MetaFile.content = MetaXml + +err = http_test.begin () + +exit (err) diff --git a/testenv/Test-metalink-xml.py b/testenv/Test-metalink-xml.py index 1659165..7807720 100755 --- a/testenv/Test-metalink-xml.py +++ b/testenv/Test-metalink-xml.py @@ -6,12 +6,39 @@ import re import hashlib """ - This is to test Metalink as XML file support in Wget. + This is to test Metalink/XML file support in Wget. + + With --trust-server-names, trust the metalink:file names. + + Without --trust-server-names, don't trust the metalink:file names: + use the basename of --input-metalink, and add a sequential number + (e.g. .#1, .#2, etc.). + + Strip the directory from unsafe paths. """ ############# File Definitions ############################################### +bad = "Ouch!" + File1 = "Would you like some Tea?" File1_lowPref = "Do not take this" File1_sha256 = hashlib.sha256 (File1.encode ('UTF-8')).hexdigest () + +File2 = "This is gonna be good" +File2_lowPref = "Not this one too" +File2_sha256 = hashlib.sha256 (File2.encode ('UTF-8')).hexdigest () + +File3 = "A little more, please" +File3_lowPref = "That's just too much" +File3_sha256 = hashlib.sha256 (File3.encode ('UTF-8')).hexdigest () + +File4 = "Maybe a biscuit?" +File4_lowPref = "No, thanks" +File4_sha256 = hashlib.sha256 (File4.encode ('UTF-8')).hexdigest () + +File5 = "More Tea...?" +File5_lowPref = "I have to go..." +File5_sha256 = hashlib.sha256 (File5.encode ('UTF-8')).hexdigest () + MetaXml = \ """ @@ -22,36 +49,115 @@ MetaXml = \ GNU GPL http://www.gnu.org/licenses/gpl.html - Wget Test File 1 + Wget Test Files 1.2.3 - Wget Test File 1 description + Wget Test Files description {{FILE1_HASH}} - http://broken.example/File1 + http://{{SRV_HOST}}:{{SRV_PORT}}/wrong_file + http://{{SRV_HOST}}:{{SRV_PORT}}/404 http://{{SRV_HOST}}:{{SRV_PORT}}/File1_lowPref http://{{SRV_HOST}}:{{SRV_PORT}}/File1 + + + {{FILE2_HASH}} + + + http://{{SRV_HOST}}:{{SRV_PORT}}/wrong_file + http://{{SRV_HOST}}:{{SRV_PORT}}/404 + http://{{SRV_HOST}}:{{SRV_PORT}}/File2_lowPref + http://{{SRV_HOST}}:{{SRV_PORT}}/File2 + + + + + {{FILE3_HASH}} + + + http://{{SRV_HOST}}:{{SRV_PORT}}/wrong_file + http://{{SRV_HOST}}:{{SRV_PORT}}/404 + http://{{SRV_HOST}}:{{SRV_PORT}}/File3_lowPref + http://{{SRV_HOST}}:{{SRV_PORT}}/File3 + + + + + {{FILE4_HASH}} + + + http://{{SRV_HOST}}:{{SRV_PORT}}/wrong_file + http://{{SRV_HOST}}:{{SRV_PORT}}/404 + http://{{SRV_HOST}}:{{SRV_PORT}}/File4_lowPref + http://{{SRV_HOST}}:{{SRV_PORT}}/File4 + + + + + {{FILE5_HASH}} + + + http://{{SRV_HOST}}:{{SRV_PORT}}/wrong_file + http://{{SRV_HOST}}:{{SRV_PORT}}/404 + http://{{SRV_HOST}}:{{SRV_PORT}}/File5_lowPref + http://{{SRV_HOST}}:{{SRV_PORT}}/File5 + + """ -A_File = WgetFile ("File1", File1) -B_File = WgetFile ("File1_lowPref", File1_lowPref) +wrong_file = WgetFile ("wrong_file", bad) + +File1_orig = WgetFile ("File1", File1) +File1_down = WgetFile ("test.meta4.#1", File1) +File1_nono = WgetFile ("File1_lowPref", File1_lowPref) + +File2_orig = WgetFile ("File2", File2) +File2_down = WgetFile ("test.meta4.#2", File2) +File2_nono = WgetFile ("File2_lowPref", File2_lowPref) + +File3_orig = WgetFile ("File3", File3) +File3_down = WgetFile ("test.meta4.#3", File3) +File3_nono = WgetFile ("File3_lowPref", File3_lowPref) + +File4_orig = WgetFile ("File4", File4) +File4_down = WgetFile ("test.meta4.#4", File4) +File4_nono = WgetFile ("File4_lowPref", File4_lowPref) + +File5_orig = WgetFile ("File5", File5) +File5_down = WgetFile ("test.meta4.#5", File5) +File5_nono = WgetFile ("File5_lowPref", File5_lowPref) + MetaFile = WgetFile ("test.meta4", MetaXml) WGET_OPTIONS = "--input-metalink test.meta4" WGET_URLS = [[]] -Files = [[A_File, B_File]] +Files = [[ + wrong_file, + File1_orig, File1_nono, + File2_orig, File2_nono, + File3_orig, File3_nono, + File4_orig, File4_nono, + File5_orig, File5_nono +]] Existing_Files = [MetaFile] ExpectedReturnCode = 0 -ExpectedDownloadedFiles = [A_File, MetaFile] +ExpectedDownloadedFiles = [ + File1_down, + File2_down, + File3_down, + File4_down, + File5_down, + MetaFile +] ################ Pre and Post Test Hooks ##################################### pre_test = { @@ -78,6 +184,10 @@ http_test.server_setup() srv_host, srv_port = http_test.servers[0].server_inst.socket.getsockname () MetaXml = re.sub (r'{{FILE1_HASH}}', File1_sha256, MetaXml) +MetaXml = re.sub (r'{{FILE2_HASH}}', File2_sha256, MetaXml) +MetaXml = re.sub (r'{{FILE3_HASH}}', File3_sha256, MetaXml) +MetaXml = re.sub (r'{{FILE4_HASH}}', File4_sha256, MetaXml) +MetaXml = re.sub (r'{{FILE5_HASH}}', File5_sha256, MetaXml) MetaXml = re.sub (r'{{SRV_HOST}}', srv_host, MetaXml) MetaXml = re.sub (r'{{SRV_PORT}}', str (srv_port), MetaXml) MetaFile.content = MetaXml -- 2.7.3