Re: [Bug-wget] [PATCH] Update HSTS info

[Daniel Kahn Gillmor]

Hi Ander--

Thanks for this!  having the concise, human-readable description of the
intended behavior is really useful.

a couple comments on the proposed documentation below:

On Sat 2015-08-08 13:50:17 -0400, Ander Juaristi wrote:

> Those two patches add two extra debug traces for HSTS, and most importantly, 
> update the documentation to include information about HSTS.
 [...]
> address@hidden address@hidden
> +By default, Wget stores its HSTS database in @file{~/.wget-hsts}.
> +You can use @samp{--hsts-file} to override this. Wget will use
> +the supplied file as the HSTS database. Such file must conform to the
> +correct HSTS database format used by Wget. If Wget cannot parse the provided
> +file, the behaviour is unspecified.

There is no pointer or reference here to what the "correct HSTS database
format used by Wget" is.  providing a brief pointer (e.g. "look at file
doc/FOO in the wget source" or "see https://example.org/wget-hsts-db";)
would be useful.

> +Be aware though, that Wget may modify the provided file if any change occurs
> +between the HSTS policies requested by the remote servers and those in the
> +file. When Wget exists, it effectively updates the HSTS database by rewriting
> +the database file with the new entries.

Is it possible that a user would want to decouple reading from
(respecting) an HSTS file and writing to it?  some examples:

 * A wget process might have no ability to modify the filesystem (e.g.
   a constrained wget -O- in a pipeline), but wants to respect a
   system-provided HSTS ruleset.  This might want to read an hsts file
   without trying to write to it (or at least without raising an error
   on an attempted modification).

 * A wget process used as part of network testing suite might want to
   disobey any system-provided HSTS rules (it needs to emulate behavior
   of "ignorant" clients), but might also want to collect the HSTS rules
   it sees for later review.  This might want to update an hsts file
   without trying to respect its contents.

I don't know if these use cases warrant the additional option complexity
of accomodating them, but it'd be good to make that decision explicitly
(sorry if that's already been done and i've just missed the discussion.

> +Care is taken not to override possible changes made by other Wget processes 
> at
> +the same time over the HSTS database. Before dumping the updated HSTS entries
> +on the file, Wget will re-read it and merge the changes.

This sounds like there might still be a race condition where one
process's changes get clobbered.  can we make any atomicity guarantees
for users who might be concerned about this?

> +Using a custom HSTS database and/or modifying an existing one is discouraged.
> +For more information about the potential security threats arised from such 
> practice,
> +see section 14 "Security Considerations" of RFC 6797, specially section 14.9
> +"Creative Manipulation of HSTS Policy Store".

I'd normally characterize these threats as "privacy concerns", not
necessarily "security concerns".  users of wget might understand them
most closely as offering "cookie-equivalent" mechanisms for some
http/https clients.

sorry these questions and considerations are in text form and not as
patches.  feel free to take from them whatever you might find useful.

Regards,

        --dkg