Re: [Bug-wget] Signature verification support in wget?

[Darshit Shah]

Hi Ludovic,

Thanks for the detailed write-up on this feature. It indeed is a fairly 
good list of questions that we must answer as this feature is 
implemented.

Given the complexity of implementing this and the lack of time on the 
part of the contributors, I think it's safe to assume that the 
--verify-signature switch and all discussions pertaining to it will 
apply only to Wget2.

My thoughts come in-line...

* Ludovic Courtès <address@hidden> [170830 14:53]:
> Hello!
>
> Following the GNU Hackers Meeting there was a discussion about the
> ability to add signature verification support directly in wget, which
> I’ll try to summarize here to get the ball rolling.
>
> Darshit was suggesting having this:
>
>  wget --verify-signature \
>    https://ftp.gnu.org/gnu/recutils/recutils-1.7.tar.gz
>
> whereby wget would automatically download recutils-1.7.tar.gz.sig and
> run gpgv or similar.  Having something along these lines would be great
> because it could help make things “secure by default”, as the marketing
> folks would say.  :-)

Yes, this is the high level idea I have in my mind. However, to be 
specific, it would not be calling `gpg` or `gpgv` but rather the checks 
should be done using the GPGME library. It is more robust to use the 
library since it does not rely on parsing the output of `gpg` which is 
highly fragile.

> The devil is in the detail though, and I was wondering whether having
> that feature within wget might raise another set of issues, and
> whether/how these could be solved.  Here are some examples:
>
>  • Is the file named .sig, .sign, or .asc?
>
>  • Is it the compressed tarball that’s signed or the uncompressed one
>    (as on kernel.org)?
>
>  • For GNU specifically, should we somehow honor the keyring that’s
>    published on ftp.gnu.org?
>
>  • What should wget do when a file is signed by an unknown OpenPGP key?
>    Should it offer to import it in the user’s keyring?  Or abort?
>
>  • How would --verify-signature report errors in a way that is
>    intelligible to the user?
>
>
> We dealt with some of these in the “guix import”¹ and “guix refresh”²
> tools.  For example, the kernel.org and GNU updaters and importers work
> slightly differently due to the different conventions being used.  These
> commands also have a --key-download option to specify how unknown
> OpenPGP keys should be handled.
These are interesting and important questions. As far as possible, I'd 
like to avoid having any specific knowledge of the GNU toolchain / 
ecosystem embedded within the codebase. That makes changes in the future 
extremely hard, both for Wget and the GNU System. Instead, we should 
rely on flexibility through command line switches for fine tuning the 
behavior.

Based on my current thoughts, this is how it ought to be implemented:

        * New command line switch, `--verify-signature`. Enabling this
	switch will cause Wget2 to download a "${URL}.sig" file for every 
	file it downloads with Content-Type == "application/*".

	* I'm keeping the ".sig" as the default extension for signature 
	files. However, it can be modified using the 
	`--signature-extension=sig,asc,sign` flag. In such a case, Wget2 
	will attempt to download a ".sig" first, and failing that a ".asc" 
	and lastly a ".sign" file. Any other extensions can be added to this 
	list as well. We don't keep the list populated with all three by 
	default since it could cause lots of 404 responses.

	* In the future, we could leverage the plugin mechanism to embed 
	knowledge about certain domains. Such as downloads from foo.com 
	always have signatures in .asc files.

	* The default is that the file being downloaded is what is signed. I 
	wasn't aware that the kernel tarballs need to be uncompressed first.  
	This is a more complicated scenario since it requires Wget2 to 
	uncompress the data on disk first and kind of goes beyond its scope.  
	We already have the logic for decompression, so we could in theory 
	add a `--decompress-before-verify` switch (something named better 
	than that) to handle the case for a few formats. However, I think 
	this is best leveraged through the plugin mechanism. Where it 
	attempts to identify the compression algorithm and extracts it and 
	then verifies the uncompressed tarball.

	* As of now, I'm not too keen on embedding the data about the GNU 
	keyring into Wget2 itself. It could be an external plugin that the 
	user explicitly installs. This way, the user knows that the specific 
	knowledge of the GNU keyring is not a part of Wget, allowing updates 
	to be simpler.

	* Unknown PGP keys are always an interesting problem due to the 
	various differences in how people would like to deal with it. By 
	default, I would suggest that Wget reports a warning about a missing 
	key and continues. There would be yet another switch 
	"--import-missing-keys" which would cause Wget2 to attempt to 
	contact keyservers and import the key.

	* How Wget2 reports warnings and issues is again a little 
	complicated. The high level idea is that if all signatures are 
	correctly verified, it is a normal successful exit ($? == 0). If one 
	or more keys were missing, Wget2 prints a warning message that some 
	keys were missing and exits with $? = 11. If any signature 
	verification failed, Wget2 exits with a dedicated code $? == 12 for 
	that. I'm still conflicted on whether Wget2 should continue 
	downloading files if a signature verification fails.


--
Thanking You,
Darshit Shah
PGP Fingerprint: 7845 120B 07CB D8D6 ECE5 FF2B 2A17 43ED A91A 35B6

signature.asc
Description: PGP signature