Re: [lmi] Dependencies and submodules (was: Tests PR 174)

[Greg Chicares]

On 3/25/21 1:00 PM, Vadim Zeitlin wrote:
> On Thu, 25 Mar 2021 10:10:39 +0000 Greg Chicares <gchicares@sbcglobal.net> 
> wrote:
[...]
> GC> I guess the conclusion is that using external libraries introduces
> GC> complexities; and bringing such libraries into git submodules does make
> GC> those complexities somewhat less difficult to manage, but does not
> GC> magically eliminate them.
> 
>  No, but it makes it much simpler to use different versions of the same
> dependency, including one with project-specific custom changes, and switch
> between them. Of course, you could -- and you did -- do it before with
> scripts applying patches to the downloaded files, but using submodules
> simplifies this a lot.

Exactly.

Excuse the stream-of consciousness posts--I've been thinking out loud.
I started with the hypothesis that git submodules have inconvenienced
me, not just because of my ignorance, but in some material way. That's
an emotional reaction to yesterday's snafu--I cherry-picked some changes
that included an update to some submodule, and then I innocently did
what git suggested, but that was a mistake and I had to revert it. Is my
instinctive emotional response justified--can I conclude that my life
was better without git submodules?

Apparently not. Formerly, if something changed in a library, I had to
 - change a script, to point to the library change
 - run the script, to fetch the change and rebuild the library
and 'git revert' couldn't magically "just work" because of all the things
I was doing outside of git. Now, it's semi-automated: that's one step in
a positive direction, though only one step. But one positive step can't
make my life worse than it was.

So nothing is git's fault--except, arguably, that 'git status' could
give more helpful advice:

  Changes not staged for commit:
    (use "git add <file>..." to update what will be committed)
    (use "git restore <file>..." to discard changes in working directory)
+       modified: [list only non-submodule changes here]
-       modified:   third_party/wx (new commits)
+   (use "git submodule update --init --recursive" to update submodules)
+       modified:   third_party/wx (new commits)

>  Also note that I've tried to change the existing use of dependencies and
> of lmi build system in general as little as possible, so its use of
> submodules is a bit idiosyncratic: it doesn't actually use any files from
> the submodules, other than in the install_*.sh scripts that build and
> install the dependencies from their sources. It's much more typical to use
> the headers directly and build the libraries directly in the build
> directory of the main project being built, which makes things a bit
> simpler. But I don't propose to change this because the existing system
> seems to work well enough and I don't think we'd gain that much from
> changing it.

It does work well enough, and that might mean that we should choose not
to expend effort on changing it. But that doesn't mean it's ideal. It
evolved in a very different world, with much less powerful hardware.

But consider what might ideally be done (and may be possible today).
Suppose we use libraries {wx, wxPdf, libxml2, xmlwrapp} "directly" as
you describe above. Then a continuous-integration job could pull all
available library updates nightly, and find upstream issues immediately.
That would be nice to have.

(Of course, we can already do that, by running 'install_msw.sh' in
a CI environment--without changing anything in lmi's repository.)

> GC> Well, a magical solution is possible--just
> GC> redesign the build system so that all libraries are dependencies--but
> GC> that's so much costly magic that we wouldn't want to do it.
> 
>  I'm not sure why do you think this is so costly? And don't we practically
> do this already? I'm afraid I might be missing your point here.

I'm not really sure, myself. Let's think this through.

If nothing upstream changes, then the incremental cost is just the
time it takes 'make' to determine that it doesn't have to do anything.
I don't know what that cost is, though maybe you do. If it takes even
ten seconds, then that might actually be meaningfully inconvenient.

Probably wx is our most active library. Let's suppose it changes on
most days, and maybe several times a day. Rebuilding wx really is
costly; perhaps it would double our build times. That's inconvenient.
And continuously integrating wx means we'd be subject to any mistake
that any wx developer might make. As a routine nightly test on github,
that might be very worthwhile, but for the lmi rebuilds that I'm doing
many, many times a day, it would not make sense.

Maybe the ideal is to use all libraries "directly" as above, but
update them infrequently, and only by deliberate intention (for lmi
development only, not for the continuous-integration-of-everything
scenario mentioned a few paragraphs above). What would that mean?

 - For headers: instead of installing them explicitly, we'd just use
   them where they are in the submodules. The only real difference is
   that we'd save the trouble of installing them. But that doesn't
   seem very significant: the code to install headers is small and
   inexpensive to maintain.

 - For source and binaries: we'd still have to rebuild, infrequently.
   These libraries use autotools, and we're never going to integrate
   them into lmi's makefiles: way too much work, for no real benefit,
   AFAICS.

My tentative conclusion is that we should do nothing, as there's
approximately nothing to be gained here. Making lmi's build system
marginally less idiosyncratic in this one respect is no real benefit.

>  FWIW the current setup is quite close to ideal from my point of view. The
> repository is practically self-contained (with the exception of build
> tools, OS images, schematics for constructing the semiconductor factories
> needed to produce the CPUs all this runs on and other minor details) and
> updating the dependencies is as simple and painless as I can imagine it to
> be. What problems do you see with it, other than having to run "git
> submodule update" after switching branches/commits?

Having to remember to ignore git's advice to git-add submodule updates.
(Someone who uses smiley faces would insert one here.) But that's
addressed by understanding the reason why things are the way they are.

>  I'm probably too set up in my own ways of working and could be missing
> something obvious, so I'm genuinely curious to understand your point of
> view.

It's not as though a different POV had crystallized for me. Rather,
whenever anything goes wrong (like that commit which I had to revert),
I step back, look for the root cause, and ask how I can prevent the
problem from happening again.

But now it crystallizes: we should do nothing here, and now that I
understand better, there's no difference between our viewpoints.

Still, the root-cause analysis did lead me to ask about embedding
the SHA1 in the wx soname (separate post). It's good to take a big
step back, from time to time, and look for unnoticed anomalies.