bug#63827: Some --fallback logic seems inconsistent or bad, and other substitution issues

[Maxime Devos]

According to the manual (6.3.6 Substitution Failure):

> Even when a substitute for a derivation is available, sometimes the
> substitution attempt will fail.

Inconvenient, but expected.

> This can happen for a variety of reasons: the substitute server
> might be offline, the substitute may recently have been deleted,

So, according to this paragraph, it is possible that simultaneously

  (1) a substitute is available
  (2) the substitution attempt fails
  (3) and the substitute server is offline

but (1) and (3) contradict each other -- if the substitute server is 
offline, then from the client's perspective, the substitute might as 
well not exist, so then substitutes are unavailable, and likewise, if 
the substitute is deleted, it doesn't exist anymore, so it isn't available.

This inconsistency is the specification of Guix substitution is the 
first bug I'd like to point out.

(The paragraph also mentions ‘the connection might have been 
interrupted, etc.’, which I do not take any issue with.)

This is not only a documentation issue, because it impacts --fallback:

> When substitutes are enabled and a substitute for a derivation is
> available, but the substitution attempt fails, Guix will attempt to
> build the derivation locally depending on whether or not ‘--fallback’
> was given (*note common build option ‘--fallback’: fallback-option.

I think its reasonable that if a substitute is unavailable (*), then 
Guix will built it locally.  Often, this is already the case (e.g. think 
of all the times you locally updated a package and prepared a patch), 
yet this is apparently not always the case when the substitute server is 
offline or when the substitute previously existed but was deleted.

This ‘Yet this is apparently not always the case when [...]’ is the 
second bug.

(*) Unavailability caused by interruption is its own can of worms, which 
will be covered separately.

Now, consider the following output of "guix home build [...]":

> substitute: lijst van substituten van ‘https://ci.guix.gnu.org’ aan het 
> bijwerkesubstitute: lijst van substituten van ‘https://ci.guix.gnu.org’ aan het 
> bijwerkesubstitute: lijst van substituten van ‘https://ci.guix.gnu.org’ aan het 
> bijwerkesubstitute: lijst van substituten van ‘https://ci.guix.gnu.org’ aan het 
> bijwerkesubstitute: lijst van substituten van ‘https://ci.guix.gnu.org’ aan het 
> bijwerken... 100.0%

Third bug: unless you're being informal and putting the NL in nl_NL, 
it's "bijwerken", not "bijwerke". Also, newlines are missing.
Looking at po/guix/nl.po, the translation does, in fact, write 
"bijwerken" instead of "bijwerke" and it has the same format stuff at 
the end as the English translation, so there appears to be a bug in the 
code (instead of the translation).

> 21,2 MB will be downloaded:

Fourth bug: inconsistent units -- in some substitute output, Guix uses 
binary prefixes (see, e.g., #63784) and in other places it used metric 
prefixes.

>   /gnu/store/gh1yr17j1amkjf653k8l31wqyzliqnmm-python-pytest-qt-3.3.0
>   /gnu/store/y1918l95a3wjij79456n9qdhlk8b8nja-python-qtconsole-5.3.0
>   /gnu/store/6x5rzyhqi2pchfb8w4nc7rs0jnxi5mil-python-ipykernel-6.13.0
>   /gnu/store/c536wjwfxdx3z7s9bwwhj1vngpq50aq6-python-ipyparallel-bootstrap-8.2.1
>   /gnu/store/y0qskmhmy4kzh02rlj2r238r7p39svwf-python-ipykernel-bootstrap-6.13.0
>   /gnu/store/viqz0fvfy3bdspzl31s6qv27sma8cbhs-python-numpydoc-1.5.0
>   /gnu/store/ifk6bsddap0smgq7x6lgkw3174hy29gz-python-ipython-8.5.0
>   /gnu/store/dsd9dnf54571qggry3g5qwihx2f3v23m-python-jupyter-client-7.2.2
>   /gnu/store/55zn484d18xdy0iarwcfxc5lxzsrjszf-qtconnectivity-5.15.8
>   /gnu/store/zsxcgdxfr10k1f8jg32axk60wzi8fyny-qtsvg-5.15.8
>   /gnu/store/kzg742h88nd03yagry7ddhpk5plqhnlj-qtmultimedia-5.15.8
>   /gnu/store/j3i78xw93xa5fwsg7l84x6p7abfl0xh3-qtserialport-5.15.8
>   /gnu/store/0dxzxs9yza70ypmw2hw07s9dqghmlda3-qtlocation-5.15.8
>   /gnu/store/nb5wjfx56bsw0m4bvgjh2aas4g3zzq2w-qtwebchannel-5.15.8
>   /gnu/store/9vpx7hb9dg9r9410sg2r2lmw6yx82ixq-qtsensors-5.15.8
>   /gnu/store/m1nw1w8qhkmnj4vmn97bwy6dvhi1awli-qttools-5.15.8
>   /gnu/store/9rmh4afafhc77q16gbf1080ffdc7cz1l-qtxmlpatterns-5.15.8
>   /gnu/store/fwkxw2ylbx0bd5vx1rnw1zkk40mfx0yn-qtwebsockets-5.15.8
>   /gnu/store/1l3xi0xcsf09i2ilsbzbrfvpla0pgya8-qtdeclarative-5.15.8
>   /gnu/store/c9nq0sbciv9jcw02b61fp4xf82dnn53a-python-pyqt-5.15.9
>   /gnu/store/a9r3n52w9f224mlj7mgrw6avl45s3kxn-python-seaborn-0.12.0
> /gnu/store/y0qskmhmy4kzh02rlj2r238r7p39svwf-python-ipykernel-bootstrap-6.13.0 
> vervangen...
> /gnu/store/gh1yr17j1amkjf653k8l31wqyzliqnmm-python-pytest-qt-3.3.0 vervangen...
> /gnu/store/6x5rzyhqi2pchfb8w4nc7rs0jnxi5mil-python-ipykernel-6.13.0 vervangen...
> /gnu/store/viqz0fvfy3bdspzl31s6qv27sma8cbhs-python-numpydoc-1.5.0 vervangen...
> /gnu/store/a9r3n52w9f224mlj7mgrw6avl45s3kxn-python-seaborn-0.12.0 vervangen...
> /gnu/store/1l3xi0xcsf09i2ilsbzbrfvpla0pgya8-qtdeclarative-5.15.8 vervangen...
> /gnu/store/j3i78xw93xa5fwsg7l84x6p7abfl0xh3-qtserialport-5.15.8 vervangen...
> /gnu/store/zsxcgdxfr10k1f8jg32axk60wzi8fyny-qtsvg-5.15.8 vervangen...

(For context, these are the ‘substituting FOO’ messages)

> guix substitute: waarschuwing: tijdens het binnenhalen van 
> https://ci.guix.gnu.org/nar/lzip/y0qskmhmy4kzh02rlj2r238r7p39svwf-python-ipykernel-bootstrap-6.13.0:
>  de server is een beetje traag
> guix substitute: waarschuwing: probeer ‘--no-substitutes’ als het probleem 
> hardnekkig is

(For context, this is the ‘server is somewhat slow’ message.)

> retrying download of 
> '/gnu/store/y0qskmhmy4kzh02rlj2r238r7p39svwf-python-ipykernel-bootstrap-6.13.0' 
> with other substitute URLs...
> guix substitute: waarschuwing: tijdens het binnenhalen van 
> https://ci.guix.gnu.org/nar/lzip/viqz0fvfy3bdspzl31s6qv27sma8cbhs-python-numpydoc-1.5.0:
>  de server is een beetje traag
> guix substitute: waarschuwing: tijdens het binnenhalen van 
> https://ci.guix.gnu.org/nar/lzip/a9r3n52w9f224mlj7mgrw6avl45s3kxn-python-seaborn-0.12.0:
>  de server is een beetje traag
> guix substitute: waarschuwing: probeer ‘--no-substitutes’ als het probleem 
> hardnekkig is
> guix substitute: waarschuwing: probeer ‘--no-substitutes’ als het probleem 
> hardnekkig is
> retrying download of 
> '/gnu/store/viqz0fvfy3bdspzl31s6qv27sma8cbhs-python-numpydoc-1.5.0' with other 
> substitute URLs...
> retrying download of 
> '/gnu/store/a9r3n52w9f224mlj7mgrw6avl45s3kxn-python-seaborn-0.12.0' with other 
> substitute URLs...
> guix substitute: waarschuwing: tijdens het binnenhalen van 
> https://ci.guix.gnu.org/nar/lzip/1l3xi0xcsf09i2ilsbzbrfvpla0pgya8-qtdeclarative-5.15.8:
>  de server is een beetje traag
> guix substitute: waarschuwing: probeer ‘--no-substitutes’ als het probleem 
> hardnekkig is
> retrying download of 
> '/gnu/store/1l3xi0xcsf09i2ilsbzbrfvpla0pgya8-qtdeclarative-5.15.8' with other 
> substitute URLs...
> guix substitute: waarschuwing: tijdens het binnenhalen van 
> https://ci.guix.gnu.org/nar/lzip/j3i78xw93xa5fwsg7l84x6p7abfl0xh3-qtserialport-5.15.8:
>  de server is een beetje traag
> guix substitute: waarschuwing: probeer ‘--no-substitutes’ als het probleem 
> hardnekkig is
> retrying download of 
> '/gnu/store/j3i78xw93xa5fwsg7l84x6p7abfl0xh3-qtserialport-5.15.8' with other 
> substitute URLs...
> guix substitute: waarschuwing: tijdens het binnenhalen van 
> https://ci.guix.gnu.org/nar/lzip/6x5rzyhqi2pchfb8w4nc7rs0jnxi5mil-python-ipykernel-6.13.0:
>  de server is een beetje traag
> guix substitute: waarschuwing: probeer ‘--no-substitutes’ als het probleem 
> hardnekkig is
> retrying download of 
> '/gnu/store/6x5rzyhqi2pchfb8w4nc7rs0jnxi5mil-python-ipykernel-6.13.0' with 
> other substitute URLs...
> guix substitute: waarschuwing: tijdens het binnenhalen van 
> https://ci.guix.gnu.org/nar/lzip/zsxcgdxfr10k1f8jg32axk60wzi8fyny-qtsvg-5.15.8: 
> de server is een beetje traag
> guix substitute: waarschuwing: probeer ‘--no-substitutes’ als het probleem 
> hardnekkig is
> retrying download of '/gnu/store/zsxcgdxfr10k1f8jg32axk60wzi8fyny-qtsvg-5.15.8' 
> with other substitute URLs...
> guix substitute: waarschuwing: tijdens het binnenhalen van 
> https://ci.guix.gnu.org/nar/lzip/gh1yr17j1amkjf653k8l31wqyzliqnmm-python-pytest-qt-3.3.0:
>  de server is een beetje traag
> guix substitute: waarschuwing: probeer ‘--no-substitutes’ als het probleem 
> hardnekkig is
> retrying download of 
> '/gnu/store/gh1yr17j1amkjf653k8l31wqyzliqnmm-python-pytest-qt-3.3.0' with other 
> substitute URLs...
> guix substitute: waarschuwing: tijdens het binnenhalen van 
> https://ci.guix.gnu.org/nar/lzip/y0qskmhmy4kzh02rlj2r238r7p39svwf-python-ipykernel-bootstrap-6.13.0:
>  de server is een beetje traag
> guix substitute: waarschuwing: probeer ‘--no-substitutes’ als het probleem 
> hardnekkig is
> guix substitute: fout: failed to find alternative substitute for 
> '/gnu/store/y0qskmhmy4kzh02rlj2r238r7p39svwf-python-ipykernel-bootstrap-6.13.0'
> vervanging van 
> /gnu/store/y0qskmhmy4kzh02rlj2r238r7p39svwf-python-ipykernel-bootstrap-6.13.0 
> mislukt
> guix home: fout: some substitutes for the outputs of derivation `/gnu/store/nvynln5vrrd8983slpais1c4dqn397j4-python-jupyter-client-7.2.2.drv' failed (usually happens due to networking issues); try `--fallback' to build derivation from source 

OK, substituting ipykernel-bootstrap failed. Let's look what happened to 
ipytkernel-bootstrap:

> 21,2 MB will be downloaded:
>   [...]
> 
/gnu/store/y0qskmhmy4kzh02rlj2r238r7p39svwf-python-ipykernel-bootstrap-6.13.0
>   [...]
> [...]
> 
/gnu/store/y0qskmhmy4kzh02rlj2r238r7p39svwf-python-ipykernel-bootstrap-6.13.0 
vervangen...
> guix substitute: waarschuwing: tijdens het binnenhalen van 
https://ci.guix.gnu.org/nar/lzip/y0qskmhmy4kzh02rlj2r238r7p39svwf-python-ipykernel-bootstrap-6.13.0: 
de server is een beetje traag
> guix substitute: waarschuwing: probeer ‘--no-substitutes’ als het 
probleem hardnekkig is
> retrying download of 
'/gnu/store/y0qskmhmy4kzh02rlj2r238r7p39svwf-python-ipykernel-bootstrap-6.13.0' 
with other substitute URLs...
> [...]
> guix substitute: waarschuwing: tijdens het binnenhalen van 
https://ci.guix.gnu.org/nar/lzip/y0qskmhmy4kzh02rlj2r238r7p39svwf-python-ipykernel-bootstrap-6.13.0: 
de server is een beetje traag
> guix substitute: waarschuwing: probeer ‘--no-substitutes’ als het 
probleem hardnekkig is
> guix substitute: fout: failed to find alternative substitute for 
'/gnu/store/y0qskmhmy4kzh02rlj2r238r7p39svwf-python-ipykernel-bootstrap-6.13.0'
> vervanging van 
/gnu/store/y0qskmhmy4kzh02rlj2r238r7p39svwf-python-ipykernel-bootstrap-6.13.0 
mislukt
> guix home: fout: some substitutes for the outputs of derivation 
`/gnu/store/nvynln5vrrd8983slpais1c4dqn397j4-python-jupyter-client-7.2.2.drv' 
failed (usually happens due to networking issues); try `--fallback' to 
build derivation from source

I.e.:

  1. Let's substitute python-pykernel-bootstrap-6.13.0
  2. I'm downloading from ci.guix.gnu.org, but it's going a bit slow,
     patience!
  3. For some reason I'm not disclosing, let's download from
     somewhere else instead.
  4. Again, I'm downloading from ci.guix.gnu.org, but again it's
     going a bit slow.
     (Also, I lied, instead of downloading from somewhere else,
     I'm downloading from the same location!)
  5. I couldn't find any substitute at another location.

There are multiple bugs here:

  5. The warning message is badly constructed:
     ‘during [...]: the server is a bit slow’ is technically English
     I suppose, but there exist much better ways to phrase this
     (e.g.: ‘Warning: the download of https://[...] is proceeding
     slowly.’)
  6. When substituting from ci.guix.gnu.org failed, it then used
     the same location again as ‘other substitution location’ --
     that message is a lie, and it also isn't going to accomplish
     anything (except, maybe, by brute force).
  7. It doesn't give information on why it (tries to) switch to
     another substitution server: did the first server remove the
     substitute that Guix expected? Is the TLS certificate out-of-date?
     A time-out? A prematurely closed connection? A hash mismatch?

     Depending on the cause, the user should do nothing or report
     an issue to whoever manages the server.

     My best guess is that Guix decided that the old server is too
     slow, but it would be better if Guix actually said that,
     e.g. ‘warning: the download of https://[...] is proceeding too
     slowly; download aborted.’
  8. It looks like no server had a substitute available
     (or it was technically available, only very slowly, but for
     practical purposes that counts as unavailable), so Guix
     should have built it from source without having to add
     --fallback.

About --fallback, I think that too much unexpected network behaviour is 
only handled by --fallback, and that more --fallback behaviour should be
done unconditionally.  I think that, by default, Guix should always 
build from source when a substitute is unavailable, except when the 
cause of the unavailability is a symptom of some kind of wrongness.

The option --fallback would then also build from source even when the 
cause of unavailability is a symptom of wrongness.

More concretely:

  * Being offline is inconvenient, but not wrong.
  * A substitute being deleted isn't wrong either.
    -- it might a symptom of TTL not being properly
    implemented, but it could also just be the server
    administrator going ‘OOps, I need more space, I'll
    need to delete some substitutes earlier than expected.’

    (That's kind of a wrongness, but it's a wrongness
    that the client doesn't have to report.)

    (Improperly implemented TTL is wrong IMO, but it's not
    really problematic and Guix has bigger problems to handle.)
  * Slowness can be caused by peering problems, but that's
    not really anything that the client or server can do
    much about, so it doesn't count as wrongness.
  * I don't know about interrupted connections, because
    I don't know what it usually is a symptom of.  (There
    can exist multiple causes, but I don't have a clue
    about their frequencies.)
  * Likewise, I don't know about timeouts.
  * Hash mismatches count as wrongness (likely symptom
    of FS corruption!)
  * TLS errors: wrongness, likely indicates misconfiguration
    or MITM attempt.

Best regards,
Maxime.

OpenPGP_0x49E3EE22191725EE.asc
Description: OpenPGP public key

OpenPGP_signature
Description: OpenPGP digital signature