Re: [Bug-wget] [PATCH] Print expected total download size in spider mode

[Darshit Shah]

Hi Rohit,

Not all servers send a Content-Length header. Some servers send a
Content-Length header on certain resources, but not on others. What do
you think is the best way to handle this scenario if we want to print
the total expected download size in spider mode?

Also I think the total_download_size variable name is easy to confuse
with other meanings. Maybe we should change it to something that is a
little more verbose about what it holds. Otherwise, I think the patch
is mostly fine.

However, we are currently in a feature freeze till the next version of
Wget is released. A couple of weeks I should expect. We can only merge
your changes after that.

On Sat, Feb 14, 2015 at 1:30 AM, Rohit Mathulla
<address@hidden> wrote:
> Hi,
>
> I'd sent a similar patch about a year back which seems to have got lost in 
> the ether.
>
> Anyway, as the subject says this patch prints out the total expected download 
> size when using the --spider option.
> Its useful when you have a list of links to download but don't know the size 
> of the downloads.
>
> Please let me know if any changes need to be done.
>
> Thanks,
> Rohit
>
> ---
>  src/http.c | 1 +
>  src/http.h | 1 +
>  src/main.c | 8 ++++++++
>  3 files changed, 10 insertions(+)
>
> diff --git a/src/http.c b/src/http.c
> index bc2cfd0..edb5af5 100644
> --- a/src/http.c
> +++ b/src/http.c
> @@ -2866,6 +2866,7 @@ read_header:
>            if (contlen != -1)
>              {
>                logputs (LOG_VERBOSE, number_to_static_string (contlen + 
> contrange));
> +              total_download_size += contlen;
>                if (contlen + contrange >= 1024)
>                  logprintf (LOG_VERBOSE, " (%s)",
>                             human_readable (contlen + contrange, 10, 1));
> diff --git a/src/http.h b/src/http.h
> index 21f1ed5..4b83f16 100644
> --- a/src/http.h
> +++ b/src/http.h
> @@ -44,6 +44,7 @@ typedef struct {
>    const char *b, *e;
>  } param_token;
>  bool extract_param (const char **, param_token *, param_token *, char, bool 
> *);
> +SUM_SIZE_INT total_download_size;
>
>
>  #endif /* HTTP_H */
> diff --git a/src/main.c b/src/main.c
> index 7675455..cba5aa5 100644
> --- a/src/main.c
> +++ b/src/main.c
> @@ -1025,6 +1025,7 @@ main (int argc, char **argv)
>    double start_time = ptimer_measure (timer);
>
>    total_downloaded_bytes = 0;
> +  total_download_size = 0;
>    program_name = argv[0];
>
>    i18n_initialize ();
> @@ -1749,6 +1750,13 @@ outputting to a regular file.\n"));
>                     human_readable (opt.quota, 10, 1));
>      }
>
> +  /* Print size when --spider is passed except when spidering websites */
> +  if (opt.spider && total_downloaded_bytes == 0)
> +    logprintf (LOG_NOTQUIET,
> +               _("Total expected download size: %s\n"),
> +               total_download_size > 0 ? human_readable 
> (total_download_size, 10, 1)
> +               : "unspecified");
> +
>    if (opt.cookies_output)
>      save_cookies ();
>
> --
> 2.3.0
>
>



-- 
Thanking You,
Darshit Shah