Re: [Help-tar] Reproducibility of tar archives

[Yann E. MORIN]

Jakob, All,

On 2019-04-01 23:56 +0200, Jakob Bohm spake thusly:
> On 01/04/2019 22:15, Yann E. MORIN wrote:
> >On 2019-04-01 12:12 +0200, Jakob Bohm spake thusly:
> >>On 31/03/2019 14:08, Yann E. MORIN wrote:
> >>>So, here's my question: starting with tar-1.32 (the latest release as of
> >>>today), is the gnu tar format considered stable now, or is there no
> >>>guarantee about the gnu tar format stability?
[--SNIP--]
> >>The 3rd option, consistent with how reproducible builds are
> >>otherwise done, is to treat tar as part of the tool chain, thus
> >>making the exact build or source version of tar part of the list
> >>of exact tool versions needed to reproduce a specific build (just
> >>like there is already a requirement to use exact versions of gcc,
> >>autotools etc.), doing so would also allow the historic hash values
> >>to remain valid, as they are each tied to the tar version they were
> >>historically built with.
> >
> >The problem is that today, Buildroot uses tar-1.29, so all hashes are
> >generated with that "gnu-1.29" format, and they eventually percolate to
> >our source mirror (aka backup): http://sources.buildroot.org/
> 
> The problem that I don't understand is this:
> 
> In which situations does Buildroot recreate a tar file that doesn't
> contain built/generated files and expect it to be exactly the same tar
> archive as a different build configuration?

So, a bit of background: Buildroot is a cross-compilation build system,
in the same spirit as OpenEmebedded or OpenWrt. As such, it downloads
the source code from various projects, and compiles it

Most packages provides readily-made archives, and that is what we
download. The archive is extracted, the code is built and installed. The
archive is eventualyl used as-is and copied to a "legla-info" landing
area.

But some packages only have a git, an Hg, an svn, or a cvs repository.
For those, we eventually need to generate the archive that lands in the
"legal-info".

This is the case where we need to generate an archive that contains
actual source code, and for which we want reproducibility of the
archive.

> Do those situations incorporate other computed files, such as the
> result of running autotools on a Configure.in file in an upstream
> source?

No it does not.

>  If so, the generated tar content already depends on the
> versions of tools (such as autotools) used, and tar would belong to
> the same version control as those tools.
> 
> Do those situations really need to recreate the tar file instead of
> downloading it from sources.buildroot.org and checking the hash?

Hashes are bundled in Buildroot, but people are free not to use our
mirror. Especially, enterprise-class users would typically want to grab
the sources from the real, official upstream, rather than use our
mirror, just in case they are worried archives there would be trojaned.

Still, they want to be sure that what they get from upstream is indeed
what Buildroot expects to build, so they want the hashes to match.

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'