Re: [Libreboot-dev] Suggestions and proposal for easier Libreboot development

[Paul Kocialkowski]

Le vendredi 22 avril 2016 à 18:06 +0100, Francis Rowe a écrit :
> On 20/04/16 23:21, Paul Kocialkowski wrote:
> > Hi,
> > 
> > Le mercredi 20 avril 2016 à 21:55 +0100, Ministry of Freedom a écrit :
> > > Op 20/04/16 om 17:19 schreef Paul Kocialkowski:
> > > > After spending some hours updating C201 support in Libreboot, here
> > > > are some suggestions I'd like to submit to make Libreboot
> > > > development easier and more comfortable.
> > > > 
> > > > First, there were talks about moving the build system to something
> > > > other than scripts. Is it still planned? What is the new
> > > > alternative going to be? I would strongly urge you to keep a
> > > > script-based approach. It's a perfect fit for such a project and a
> > > > Makefile-based approach should like a train-wreck for this task, if
> > > > doable at all.
> > > the current build system is fine and will be kept, but we want to
> > > start using Makefiles around that. see libreboot.org/gnuI could start
> > > looking into something Makefile-based, too. It would be nice for
> > handling dependencies but might also cause a lot of duplication, where
> > scripts
> > are much more flexible. I'm not saying Makefiles are necessarily a bad idea,
> > but
> > it'll be much trickier to get it right.
> > 
> > > > Regarding the current script-based build system, I noticed how
> > > > multiple versions of coreboot are handled, to have a different
> > > > revision per payload-device association. I think it's a very nice
> > > > thing to have, but the way it's done current seems quite painful to
> > > > me. Instead of duplicating the tree, I'd suggest using git and
> > > > branches, for each devices (even ones that don't need it).
> > > > 
> > > impossible, because of blobs in coreboot git history. this is why we
> > > create directories for each revision of coreboot used, and then run
> > > git-init again and apply patches for each board, making a branch per
> > > board per revision, with patches applied per branch(board)
> > I don't see what the problem is with blobs in the history. As long as blobs
> > are
> > removed from the tree and the git history is not shipped with source code
> > releases, there should be no problem.
> 
> There is a problem when the release archives include coreboot already,
> and need to be able to be built without any .git directories in the
> release archives.

That makes me wonder: what is fundamentally wrong with distributing blobs as
part of the history? As far as Libreboot is concerned, they are not distributed
as files, and not really as part of the source code. This seems a bit
borderline.

Also, do you want the build from the released source code to work exactly the
same as the "freshly-downloaded" source or can there be different
scripts/instructions for those (suggestion below).

We could still ensure that Libreboot builds normally offline, after downloading
"fresh" sources and using git. Then, releases would contain one tree per device,
without .git and with adapted build instructions (could be as simple as a
keeping the very same build infrastructure but using different make target or
having an env varialbe to set). Then, Libreboot would only release files without
blobs in them (even in the history) but keep using the flexibility of git (avoid
the need for duplicating the repositories and the hassle of removing the git
history) when build from a fresh download instead of releases.

> > > > Also, the various scripts would be much easier to handle with some
> > > > included library, instead of duplicating lots of aspects in each
> > > > script, making it hard to maintain each and creating disparity
> > > > between projects.
> > > > 
> > > > For instance, a library that handles generic project-related
> > > > aspects for all the other scripts would be very welcome. This would
> > > > also make Libreboot more consistent. Such functions could handle: *
> > > > Cloning a particular project from given URLs (with a primary URL
> > > > and mirrors)
> > > This would be nice
> > > 
> > > > * Creating a branch for each payload-device pair and reverting it
> > > > to the appropriate revision. Similarly, having a branch for
> > > > "modules", used for building host tools (crossgcc, cbfstool in
> > > > coreboot, flashrom, bucts, etc). * Patching each branch with the
> > > > appropriate patches, both with patches common to the payload and
> > > > specific to the device. * Releasing each project's source code for
> > > > each payload-device pair, making a copy for each and removing the
> > > > git history.
> > > > 
> > > This needs further thought.
> > > 
> > > I agree with the overall premise; the current build system is very
> > > inflexible.
> > > 
> > > > This way, all downloaded projects would only have one copy and a
> > > > branch for each needed source tree. For coreboot, it would imply
> > > > keeping the git history. Removing it is very inconvenient when
> > > > developing and there is about no need to do it prior to releasing.
> > > > 
> > > > In order to make everything even more simply, I think it would be
> > > > best to switch to a project-based structure, where each project
> > > > simply defines a few functions with given names, such as
> > > > "download", "build", with relevant parameters. The global scripts
> > > > would simply include the appropriate scripts and call the functions
> > > > with the right arguments. This would severely reduce the
> > > > dissipation of files for a particular project. Each file would
> > > > actually be quite simple thanks to the use of a library for common
> > > > operations.
> > > Could you whip up a basic proof of concept for all of this? Even just
> > > diagrams/concepts are fine, I find it hard to understand text because
> > > we're talking about massive changes to the build system.
> > Of course, I'll let you know when it does something significant. I'll target
> > building images for veyron_speedy with it.
> 
> I suggest holding off for now. The priority is a stable release. We need
> to also finish the talks listed on https://libreboot.org/gnu/
>
> Can you work on that?

Let me know if you need me for anything regarding the release, especially on
CrOS devices. I have a lot on my plate currently, but I can try to find time for
critical issues that would be blocking the release.

I'll look into a Makefile-wrapped approach for this new build system next time I
spend some time on it, so that it also helps reach the GNU guidelines.

> > > > Projects could either be tools (flashrom, bucts), the bootup
> > > > software (coreboot) or a payload (GRUB, depthcharge). Since one
> > > > might surely depend on the other, each project's script would have
> > > > a "depends" function calling other scripts and building the
> > > > required components.
> > > 
> > > 
> > > > Regarding the naming of projects, I find some to be quite
> > > > confusing. Libreboot sometimes refers to the whole build system or
> > > > to the deblobbed coreboot, which is sometimes also called
> > > > "libreboot-libre". This kind of inconsistency makes it hard to
> > > > figure out exactly which component does what.
> > > > 
> > > > Overall, here is a suggested directory structure example matching
> > > > these changes: ressources/projects/coreboot/libreboot (script with
> > > > the functions) 
> > > > ressources/projects/coreboot/config/depthcharge/veyron_speedy/config
> > > > (coreboot configuration) 
> > > > ressources/projects/coreboot/config/depthcharge/veyron_speedy/revision
> > > > (coreboot revision) 
> > > > ressources/projects/coreboot/patches/depthcharge/0001-foo.patch 
> > > > ressources/projects/coreboot/patches/depthcharge/veyron_speedy/0001-ba
> > > r.patch
> > > > 
> > > ressources/projects/vboot/libreboot
> > > > ressources/projects/vboot/config/depthcharge/veyron_speedy/revision
> > > > 
> > > > 
> > > ressources/libs/git (git-related library)
> > > > ressources/libs/common (common operations library)
> > > > 
> > > I think rather, we just need better documentation for maintenance.
> > > Having a more logical structure makes more sense. I've considered
> > > something like what you're proposing.
> > > 
> > > We're focussing on a release at the moment. Organisational/structural
> > > changes to the project are not on the table at the moment, but we'll
> > > revisit this discussion thread later.
> > Of course, I'm not asking that you or anyone else spend their time on it.
> > I'd be
> > happy to contribute to it myself to the extent I can. I really care about
> > Libreboot and see the current build system as somewhat inconvenient. Thus
> > the
> > naturel thing to do from my perspective is to try and improve it. Hopefully,
> > others will also agree that it's an improvement.
> > 
> > > > Also, it would be best to build each project out-of-tree, or to
> > > > ensure that the tree is cleaned between different builds (something
> > > > that could be done in a clean() step, either called before each
> > > > build() for in-tree build or skipped for out-of tree build).
> > > > 
> > > > This is really just a mockup at this point, but if you don't hate
> > > > the idea or see a reason not to do this, I'll probably be spending
> > > > some time working in that direction, because I find the Libreboot
> > > > build system quite painful to deal with as it is now and it won't
> > > > scale well for more projects anyway.
> > > > 
> > > > Cheers,
> > > > 
> > > See above. You hit the nail on the head; it's a mockup. Let's get a
> > > new stable release done first, before thinking about changing the
> > > entire structure of the libreboot project.
> > Well, things are ready on my side for the release (veyron_speedy-wise), so
> > I'll
> > work on that build system change on my own for now and ping you back about
> > it
> > when the release is out of the way.
> > 
> > > (this response is hastily written, because paulk pinged me on IRC and
> > > told me to respond quickly)
> > Thanks for sharing your thoughts under such short notice, it is greatly
> > appreciated.
> > 
>

signature.asc
Description: This is a digitally signed message part