bug#67696: 30.0.50; Help deal with multiple versions in load-path

[Stefan Monnier]

>> With packages being available both as bundled with Emacs and as ELPA
>> packages, it has become a lot more common place to have two versions of
>> a package in the `load-path` and to have to deal with situations
>> where the "incorrect" version has been loaded before `load-path`
>> was changed.
>>
>> These kinds of problems manifest in various ways and we try to
>> circumvent them in `package.el` in some cases but that can't cover
>> all cases.
>>
>> I suggest we introduce a new function to help packages susceptible to
>> those problems.  The patch below introduces a new function which
>> I tentatively called `require-with-check` and shows how it could be used
>> in the case of `eglot.el` (which relies on several core packages also
>> distributed via GNU ELPA and currently uses a hack which slows it down
>> unnecessarily in the normal case).
> Will this be useful only for :core packages?

AFAIK it's useful only to detect and/or work around problems linked to
interleavings of loading files and changing `load-path`.

I can't think of any reason why this would be limited to :core packages,
but it's a problem that shows up more commonly for :core packages, indeed.

> If so, it would be nice to not have to introduce more functions and
> extra complexity just to deal with this situation.  It seems like
> a problem we should be able to fix without it leaking into our API.

Another option is to tweak `require` directly: the new function works
basically "identically" to `require` expect for the added checks.

> I didn't think deeply about this so here's a probably naive question:
> could we make `require' reload the file, if a newer one is detected
> earlier in the load-path?

By default I made the function signal an error rather than reload,
because reloading is not a really reliable solution (`defvar` won't be
updated to the new value, renamed functions will linger, ...).

But yes, we could change `require` to behave more like
`require-with-check`.  The downside is that it's more risky, and it may
come with a performance cost (it means that calling `require` repeatedly
isn't as cheap as `(memq F features)`).

> Are there any other alternative approaches that you considered?

Many other approaches have been mentioned/considered in the long
discussion around Org mode's attempt to deal with similar problems.

I found this one approach to be the clea[rn]est because it seemed to get
to the actual core of the problem and deals with a known problem (file
shadowing) that's reasonably easy&cheap to check reliably at runtime.

As you can see in my patch, Eglot took the other approach to blindly
reload the files (which kind of works but is less efficient and suffers
from further subtle differences with `require` such as the presence of
a load message, or the absence of an entry in `load-history`).

Org took yet another approach (signal an error if the loaded version is
different from the file's own notion of version), but it suffers from
annoying false positives and requires more ad-hoc support (it relies on
having a version, on hard-coding that version info in the .elc files,
and on someone manually updating that version info when needed), so it's
harder to turn it into a generic solution.


        Stefan