Re: bug#30854: 27.0.50; Speeding up package.el startup

[Arthur Miller]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> Could this strategy work: "site-autoloads.el" is used just to create
>> "user-autoloads.el" (I am just making up names, hope it is clear what I
>> mean with them) on first start. User can now disable/enable whichever
>> from sysadmin installed ones, as if they were present in users packages,
>> and regenerate autoloads.el as the docs describe. Emacs checks mtime of
>> site-autoloads.el and if it is changed it recreates new autoloads file
>> with respect to disabled packages.
>
> The code in package.el already handles the case I described.
> What doesn't work yet is to automatically detect when
> `package-quickstart.el` needs to be recreated.
>
>> You mean to automate, so the user does not need to run
>> package-quickstart-refresh manually?
>
> Exactly.  That's what needs to happen before we can consider enabling
> `package-quickstart` by default.
>
>> It means Emacs would need to iterate all the directories in site
>> packages and user packages somewhere at some time; probably in some
>> idle timer function or do you have something else in mind?
>
> To be correct, it needs to be done before we load
> `package-quickstart.el`.
>
>> If sysadmin updated site packages then autoloads for the site would
>
> There's no such thing as "autoloads for the site" currently.
Yes, I know. That was a proposal to structure it that way; also to
introduce site-autoloads.el for potential sysadmins. Similar to
site-start/load/init.el.

> But I don't think we need to create that either.
> Installing/removing/upgrading a package will change the mtime of the
> parent directory (i.e. the directory that's in
> `package-directory-list`), so we just need to check the mtime of
> those directories.  While there can easily be hundreds of packages,
> `package-directory-list` is usually a short list, holding typically less
> than 5 elements, so it should not impact startup time noticeably.
Indeed; we are on same page there.

What I was thinking of was that autoloads would be generated when
packages are updated anyway, so checking just mtime for that file is
synonymous to checking directories in that list. Discovering change is
only needed for site installed packages. If there existed one central
autoload in system elpa directory, then autoloads for all those packages
would be calculated only once: when sysadmin update packages. User Emacs
could just check that file, and if it is newer then some last time stamp,
simply update the user autoloads. But what I am thinking of now is
that sysadmin my mount storage locations under different path than how
it is mounted for a user; so site autoloads would need to be emited with
relative paths, while user's would need to be absolute and updated when
user file is generated. Not hard but but I scratch the idea if there is
extra work performed anyway. Just check mtime of the repo directory,
and generate new user autoloads; works just fine too :-). 

For the user directories there is no need to check them, since autoloads
file would be touched automatically when packages are updated.

I have come up to think of a thing when quickstart is creating
autoloads. It currently loops through all packages avialable and
generates autoloads for them all, and adds them all to the path, it does
not seem to care about what user activated or not, at least how I
understand package-quickstart-refresh function.

If there was a blacklist of some kind, then it could be checked when
autoloads file is created and those packages simply not emitted to
autoloads so emacs does not read them on the start, or do I
missunderstand how active/inactive packages works? I think it would be
needed if user is to disable site installed packages.