Re: On generating compilation databases

[Arsen Arsenović]

Hi,

Paul Smith <psmith@gnu.org> writes:

> It's really not at all clear what you are talking about here, until far
> down your email where finally you say:
>
>> ... dumped into compile_commands.json.
>
> and we discover you're trying to build a compile_commands.json file.

Ah - sorry, I should've known better than to trust my writing abilities
in the late night :-).  Hopefully I can clear this up.

>> So, in conclusion, what do you think is the right approach?
>
> I'm sorry but I'm not sure I understood the approach that you are
> referring to here.  Are you talking about the idea of "setting CC, CXX
> ... variables to a tool"?
>
> That seems like a reasonable way to go in my opinion.

Yes, that's the generic make option.  I also (tried to) talk about a
possibility of an automake-enabled autoconf project generating the file
all at once at configure time, based on the same logic that determines
what parts of Makefile.in are enabled and which are commented, seeing as
they're two specifications of the same (or, at least, very similar)
data.

Automake may have a simpler time generating this data (since it has data
more abstract than just opaque shell code), but would be less generic as
a result.

To expand on the non-automake option, I'd expect that it'd be safe to
re-run all targets if all the commands they invoked got set to
$LIBEXECDIR/make/collect-rule or whatever.  For a concrete example,
let's suppose we have the following trivial makefile:

  CC = gcc
  CCLD = gcc
  MKDIR_P = mkdir -p
  COMPVARS = CC CCLD  # New syntax.
  PROGVARS = MKDIR_P  # ^^^^^^^^^^^
  %.o: %.c
        $(CC) -o $@ -Wall -W -std=c11 $<
  out/test: a.o b.o c.o
        $(MKDIR_P) out
        $(CCLD) -o $@ $<

... the hypothetical tool would set all the COMPVARS (short for
``compiler variables'') to ``$LIBEXEC/make/collect-rule /tmp/some_socket
original_value'', all the variables in PROGVARS (short for ``program
variables'', with the implicit exclusion of compilers) to ``:'', MAKE to
an invocation of gmake that'd do the same, and then run the rules with
all targets considered out of date (as if -W was set on all files).

Since it's often useful to also build a program rather than just dump
how it's built, it could be possible to have an alternative mode that
doesn't alter PROGVARS, and passes --invoke to
``$LIBEXEC/make/collect-rule'', to instruct it to also run the command
it collects.

The "trick" with ignoring dependency and target age is part of why I
think this might work well if part of make itself.

(segment moved from further down in the message)
> The big problem, as with all make-based tools, is that if you don't
> actually build the target then your compiler invocation won't run and
> that source file and compile command won't be added to the JSON file.
>
> If I were designing a tool such as the putative "report-calls" I would
> have it load the compile_commands.json file into memory, then merge in
> any new or different information based on the current compiler
> invocation, then write it out again.  Of course this would require some
> amount of locking, or else using some kind of server model.

The "augmented" execution environment combined with a well specified
makefile, would lead to all targets being pure with respect to the
filesystem (i.e. it'd be safe to rerun many times or in partially built
trees).  Presumably, Automake could help migrate many existing projects
so that they have a well specified makefile, with far less manual work
than normally.

Maybe it'd also be worth to augment the shell environment slightly to
alias touch and other coreutils to ``:'', to catch more makefiles.

Besides that, it is likely a good idea to merge flags in the way you
described anyway, yes (and here another fault with compile_commands
shines: merging files is somewhat involved), since the implementation I
described would allow you to only run one target and it's dependencies
(e.g. make --generate-compdb src/tee && make --generate-compdb src/tail
would generate a compile_commands.json for src/{tee,tail}).

>> Do you think a feature like this should even exist in the GNU Build
>> System (or, Make more broadly)?
>
> Well, I personally think that compile_commands.json specifically is a
> poor format; for example it provides no information on header files,
> which are clearly needed in order to have a well-functioning LSP server
> (I mean, we edit header files too).  Similar to the LSP spec itself, it
> suffers from having been designed by one group for their needs without
> much thought, at least initially, to how to make the format generic and
> widely useful.  That leads to bad decisions being made up-front then
> hard to get away from (for example, LSP's requirement for UTF-16 which
> betrays its origins as a purely Microsoft-generated specification).
>
> However I have nothing against having compile_commands.json being
> generated, certainly.

I fully agree with everything you said here.  It's a shame that this is
the point we arrived at.  Hopefully we have something better some day.

These problems aren't theoretical either: editing a header will, indeed,
often result in flags not being specified properly, and using clangd
often requires specifying some tweaks in .clangd files.

But beyond that, I think it'd still be valuable to help contributors
which are trying to get into contributing to GNU by supporting some
tools they might be familiar with, and I don't feel the cost here being
too high.

>> Do you have any implementation ideas?
>
> Not really.  I see no feasible way that that generating
> compile_commands.json could be built into make itself, so it would need
> to be some kind of add-on package.  For people who are willing or able
> to modify makefiles and reset appropriate variables such as CC or CXX,
> that seems like quite a reasonable way forward.

I think formalizing and implementing the idea inside make could simplify
the implementation, even if it can be an add-on (though, maybe actually
trying to implement it would change my mind ;) ).

Apologies again for the unclear initial message, have a great day.
-- 
Arsen Arsenović

signature.asc
Description: PGP signature