Recursion problems with package management

[Rusi]

Collecting some thoughts and threads on package management here.
tl;dr at bottom

On Friday, August 14, 2015 at 2:01:08 AM UTC+5:30, Stefan Monnier wrote:
> On Thursday, August 13, 2015 at 12:55:26 PM UTC+5:30, Tomás wrote:
> > FWIW, and based on a recent experience of mine, yes, I think both ways
> > are needed/useful and complement each other. Recently, I installed some
> > package from ELPA (magit) and it failed to byte-compile. I've yet to
> > investigate what went wrong (perhaps my Emacs version is too new/old,
> > what have you), but I now find myself wrangling with the complexities
> > of the package itself *and* those of the packaging system.
> > 
> > So some "wholly integrated solution" makes life easier only when things
> > work out (duh ;-) Otherwise it makes life harder, and what's more important
> > in my view -- it tends to make a stronger separation of "outer circle"
> > and "inner circle", making it more difficult to get involved.

> My crystal ball tells me that you upgraded Magit from an Emacs where the
> previous Magit was already loaded, and the old (loaded) version
> interfered while byte-compiling the new version.
> [ The typical interference, looks like this: Emacs tries to compile
>   foo-1.el which does (require 'foo-2) to get some macros, but (the old)
>   `foo-2' is already loaded so we end up using the old version of the
>   macros (or the macros may simply not be defined at all).  ]

Here that we are at the root of the problem I think -- emacs package-managing 
emacs is a recursion problem.

In short, if emacs package-manager is weak enough it will work...  at
the cost of being 1/2-assed. If it is full-featured enough to be a basic,
all-round useful building block, recursion/meta-circularity needs to
be a (significant) factor in its design.

Obvious instance of the above: Imagine async.el is a package.
How to install async asyncly without async?

Here are some random (non emacs) instances of the above issue

1. apt-get upgrade apt-get works
What about 'apt-get install apt-get' 

2. Linux needs a shell, including to startup
So there is dash in addition to bash

3. Everything runs on top (transitively) the kernel.
But the kernel is a package that can be upgraded.
kernel install requires restart.
Which needs help of apt/grub etc.

4. The general idea, need, issues for bootloading-OS, bootstrapping-compiler

5. firefox plugin-install requires restart firefox. Why?


Coming to emacs package management

Edward's req-package usage-pattern has exemplified use thus
[see https://github.com/edvorg/emacs-configs ]
There is an init.el containing
(add-hook 'after-init-hook (lambda () (load "~/.emacs.d/init-real.el")))
And the real stuff is in init-real.el.

Related to the formidable list shown by Robert Thorpe
[thread "How can I change the default loading buffer"]

On Sunday, August 2, 2015 at 9:26:13 PM UTC+5:30, Robert Thorpe wrote:
> Emanuel Berg writes:
> > OK, so what is the method to find out (or rule of
> > thumb) what you shouldn't put directly into .emacs but
> > instead into `emacs-startup-hook'?
> 
> Have a look at (info "(elisp)Startup Summary") and see if any of it
> applies to you.  Check from the bottom of the list up.
> 
> You can only delay things as far at step 24, so there's little
> use in worrying much about the earlier steps.  For people who use the
> session manager and --daemon they could cause problems though.
> 
> This is how it is for me, going backwards, last step first:
> 
> * 24.  `emacs-startup-hook' and `term-setup-hook' are run.
> * 23.  initial-buffer-choice.  Because the initial-buffer-choice is set
> here, what happens in emacs-startup-hook can change it.  What happens in
> the init file can't though.  I use this feature.  I setup
> initial-buffer-choice to my ToDo file.  In emacs-startup-hook I look for
> a bookmark into my init file, if it exists then I jump to it.
> * 22.  Exit point for --batch.  If you use Emacs a lot in batch mode
> it's worth doing a lot of visual stuff in emacs-startup-hook because
> that way it won't slow down your batch programs.
> * 21.  Other command line options.  I think this includes files
> specified on the command line.
> * 20.  Initial echo area message.  Not very important.
> * 19.  Terminal specific lisp library.  This can be important if you use
> terminal emulation and need one of these.  I've never had to worry about
> this.
> * 18.  Set the mode of *scratch*.
> * 17.  Run after-init-hook.
> * 16.  Set after-init-time.  I think this is used in measuring how fast
> Emacs init is.  If you use after-init-hook or emacs-startup-hook then
> the stuff in those isn't measured.  In different situations you may want
> that or not want it.  You can always put everything into the init file
> temporarily just to see how long the full init takes.
> * 15.  Initialize packages.  This is important, packages aren't
> initialized in the init file.  There has been some discussion about
> changing this.  Many people change it by putting package-initialize in
> their init file.
> * 14.  Load abbrevs.  Abbrevs are rarely used programmatically so I
> don't think this matters much.
> * 13.  Load "default" library.  I've never used that.
> * 12.  Loading of the init file.

--------------------------------------------------------
tl;dr
Emacs package-managing emacs is thhe usual lure of Turing complete language can 
do anything does not mean doing it is appropriate.

If we must use elisp for it, it would be good to have a separate
'binding-time'. Just as in the C world compile time and runtime are
rigorously separated, it would be neat IMHO if elisp running in
package-mgmt mode did it in batch-mode and required restart of emacs.

Perhaps req-package is in the earlier binding time
And use-package the later one?