bug#67455: (Record source position, etc., in doc strings, and use this in *Help* and backtraces.)

[Alan Mackenzie]

Hello, Stefan.

On Sun, Apr 07, 2024 at 23:16:13 -0400, Stefan Monnier wrote:
> > Pretending the problem doesn't exist won't solve it.  In the ;POS...
> > structures for a lambda, there are two pointers - one to the definition
> > of the lambda, the other to the point of use.

> Fancy.  Could you give me an example where I see this in play?
> [ To help me understand also what you mean by "definition of the
>   lambda" and "point of use"?  ]

> I looked around but all I could see where position info like

>     [foo foo.el 41 nil]

> which point to "the definition" of the function.

Apologies.  I was thinking of my latest not yet committed version, where
I've added a fifth element into the position information, the buffer
containing the lambda.  This should enable buttons to be set on the
interactive backtrace, pointing back at the two source code positions.

> >> BTW, AFAIK the above is conceptually what the byte-compiler does (except
> >> it performs a few more transformations between `macroexp--expand-all`
> >> and `strip-all-symbol-positions`).
> > It is a bad idea to conflate these two radically different uses of SWPs.

> In what way are they radically different uses of SWPs?

You're confused in precisely the way I feared.  "conceptually what the
byte-compiler does" is what it does to strip the SWPs used as WARNING
POSITIONS.  When the SWPs are used for function position structures
(whether in interpreted or compiled code) the handling is radically
different - the P in the SWP is stripped as soon as possible after its
use.  In the warning pos use, the positions are preserved for as long as
possible.

> >> Is it the case that `cl-defmethod` generates a function whose source
> >> position (partly) points to `generic.el:403` if `cl-generic.el` was
> >> interpreted but not if it compiled?
> > No, the intention is that the source positions are independent of whether
> > the code is compiled.

> Good.  So why do the interpreted and compiled cases need to be
> "radically different uses of SWPs"?

They're not.  It is the use for warning positions that differs from the
use for putting pos info into the doc string.

> >> > (defmacro foo (lambda bar)
> >> >   `(cons ,lambda ,bar))

> >> > expands to

> >> > (macro closure (t) (lambda bar) ";POS^^^A^A^A [foo *scratch* 158 nil]
> >> > " (list 'cons lambda bar))

> >> IIUC your reader will make the `lambda` formal argument into an SWP.
> >> Where is that SWP stripped?

> > In macroexp--expand-all in the "guard arm" near the end.

> How?  `macroexp--expand-all` will not be passed this `lambda` because
> it's not an *expression*.

Well, when I commented out that pcase arm, the lambda no longer got
stripped.  I'm not sure what you mean by expression.

> `eval-buffer` of a buffer containing the above defmacro does:

> 1 -> (macroexp--expand-all (defalias 'foo (cons 'macro #'{foo} (lambda 
> (#<symbol lambda at 46> bar) ";POS

 [foo foo.el 41 nil]\n" `(cons ,lambda 
> ,bar)))))
> | 2 -> (macroexp--expand-all 'foo)
> | 2 <- macroexp--expand-all: 'foo
> | 2 -> (macroexp--expand-all (cons 'macro #'{foo} (lambda (#<symbol lambda at 
> 46> bar) ";POS

 [foo foo.el 41 nil]\n" `(cons ,lambda ,bar))))
> | | 3 -> (macroexp--expand-all 'macro)
> | | 3 <- macroexp--expand-all: 'macro
> | | 3 -> (macroexp--expand-all #'{foo} (lambda (#<symbol lambda at 46> bar) 
> ";POS

 [foo foo.el 41 nil]\n" `(cons ,lambda ,bar)))
> | | | 4 -> (macroexp--expand-all ";POS

 [foo foo.el 41 nil]\n")
> | | | 4 <- macroexp--expand-all: ";POS

 [foo foo.el 41 nil]\n"
> | | | 4 -> (macroexp--expand-all `(cons ,lambda ,bar))
> | | | | 5 -> (macroexp--expand-all 'cons)
> | | | | 5 <- macroexp--expand-all: 'cons
> | | | | 5 -> (macroexp--expand-all lambda)
> | | | | 5 <- macroexp--expand-all: lambda
> | | | | 5 -> (macroexp--expand-all bar)
> | | | | 5 <- macroexp--expand-all: bar
> | | | 4 <- macroexp--expand-all: (list 'cons lambda bar)
> | | 3 <- macroexp--expand-all: #'{foo} (lambda (#<symbol lambda at 46> bar) 
> ";POS

 [foo foo.el 41 nil]\n" (list 'cons lambda bar))
> | 2 <- macroexp--expand-all: (cons 'macro #'{foo} (lambda (#<symbol lambda at 
> 46> bar) ";POS

 [foo foo.el 41 nil]\n" (list 'cons lambda bar)))
> 1 <- macroexp--expand-all: (defalias 'foo (cons 'macro #'{foo} (lambda 
> (#<symbol lambda at 46> bar) ";POS

 [foo foo.el 41 nil]\n" (list 'cons 
> lambda bar))))

> So we see that indeed it returns code where the formal argument `lambda`
> is (incorrectly) a SYMPOS.  Yet somehow the sympos is stripped after
> macroexpansion somewhere since `(symbol-function 'foo)` shows the
> resulting function doesn't have any symposes in it.

OK, this needs clearing up.

> >> > so it is clear this case is getting handled OK.  I'm afraid I can't
> >> > point out the exact place in the code at the moment where this is
> >> > getting done.
> >> I think it would be good to know, so as to be able to decide whether
> >> it'll indeed always work right, or we just got lucky this time.
> > See above.

> Yes, please, see above 🙂

> >> Could you explain what you think makes it intrinsically complex?
> > The mass of detail that needs dealing with that Emacs has collected over
> > the decades.  As a counter question, why do you think the exercise ought
> > to be simple (assuming you do think this)?

> Because you solved the hard part when you added the symposes for the compiler.

OK, but that's not the way things turned out.

>         Stefan

-- 
Alan Mackenzie (Nuremberg, Germany).