Re: [PATCH] Project out of sources compilation

[Ergus]

Hi:

On Mon, Apr 22, 2024 at 10:20:32PM GMT, Mohsin Kaleem wrote:
> Ergus <spacibba@aol.com> writes:
>
> Howdie howdie,
>
> I'm the maintainer of projection. Looks like it's a similar external
> package to what we're trying to formalize in Emacs core project.el .
>
> I'm just going to highlight 2 things which I think would help you Ergus.
> Ninja multi-config generators [1] which basically lets you configure a
> project for Debug, Release, ASAN, UBSAN, etc. simultaneously. And CMake
> presets [2] which is a standard for loading common configuration in CMake
> like build directories but also compiler flags, environment variables,
> etc.
>
> [1]: https://cmake.org/cmake/help/latest/generator/Ninja%20Multi-Config.html
> [2]: https://cmake.org/cmake/help/latest/manual/cmake-presets.7.html
I'll give it a look. I am aware of cmake presets, but we don't even
support cmake or out of sources compilation yet, so I tink that
supporting the most basic cmake setup will be already a step forward.

> I'll add my 2-cents on some of the discussed points.
>
> Firstly, I disagree with the goal of making the build directory option
> customizable in project.el itself. Most build tools I've encountered
> (including all supported by projection) take an extra flag to point to
> the build directory and I find this a nicer UX because it's easily
> inspectable and editable with `C-u M-x recompile`. I noticed projectile
> supported this feature of "run somewhere else" as well but it was only
> for one project type "meson" and even that has this option to move
> directories so I don't really see the value of complicating the
> interface to support it. Another pain point with a generalised
> "build-directory" option is that it's not generic for all possible
> commands. Configuration often runs before the build directory exists
> (because it creates it) so then you need an exemption for one command
> type but not another. I think the project-root as a standard location
> for commands to run is a good common default and changing it unless
> absolutely necessary is un-needed :-).
Yes, that's actually the current approach in my POC code. after some
fight with rust projects. However, the difference on moving or not is so
simple that it doesn't even represent a difficulty.

> Secondly I agree with Dmitry about defining an alternate orthoganal API
> to project.el for this. It probably belongs more in a project-types.el
> package. The motivation being that project.el is for more bare-bones
> mechanics, things like "am I in a project", "where is the root of the
> project", "show me all the files in this project". Project types is more
> "how do I build this project", "how do I test it", "what artifacts does it
> produce". The former is a set of functionalities which is relatively
> common and static across all projects, the latter is very specific to
> what kind of a project we're dealing with and how you interface with it.
> For example python projects probably don't have a configure step or
> build directory, node projects may treat "npm install" as a configure
> step but omit building, CMake projects support all steps through to
> packaging and installing. Things don't generalise well here.
This implies removing many of the features project.el already has
(project-compile, project-eshell, project-compile-buffer-name-function)
+ somehow create another level of abstraction over project.el...

The initial basic commands (config, build, test/run) seems to be common,
even in python projects it usually needs to get dependencies, create
virtualenvs and so on. However if a project does not support a build
command a simple message can solve it easily.

Initially we are not even supporting the generate/configure, and the
functionality is limited to: recognize compile and test already generated
projects; but using some very primitive heuristics. More complex setups
I thing must be left to more specialized external projects.

These seems limiting, but it is much more than what we already have now.

> I think Ergus you've actually touched on a fun corner case. You've got a
> project with sub-projects and define a regex in a marker file to detect
> this. I'll admit I haven't touched on this before, I know you can have a
> CMake project with different sub-directories being considered their own
> projects and buildable independently, but I've never seen this practice
> in my day-to-day work. I'd be inclined to treat this as a project.el
> feature, something akin to git sub-modules. My only concern with it as a
> feature is its not trivial to check like git submodules. Basically every
> directory in a CMake project can have a CMakeLists.txt file and
> recursively searching upwards for one that might coincidentally also be
> a project root marker isn't ideal :-(.
FWIK every directory may have a CMakeLists.txt, but they need to have a
project() entry to be considered a project's CMake. The search is
actually unavoidable and the regex check actually solves the race
conditions.

I know that the search is not efficient, but with a couple of
optimizations if works very well even with tramp.

Nested projects is something very common these days and supporting them
in project.el is part of another parallel discussion going on already
(very slowly btw).

For now it is save enough to find the top-most one as it is what the
default backend does. It will probably fit a high percent of all
projects around.

> While we're discussing interfaces for this project-type abstraction I'll
> mention what I did for projection. In projection a project-type is an
> eioio class. It has some slots like :predicate to check if the current
> project matches that type and :build, :test, etc. for various project
> specific commands that can be run. The list of all actively checked-for
> project types is just a flat hook. I opted for this for 2 reasons:
>
> 1. It's easy to modify. Every project-type can be edited directly you
> can deregistered one from the monitored set of types by just removing it
> from the collection.
> 2. It enforces a general ordering concept which is important for
> determining a primary project type. I think moving the properties into a
> generic function though would just decouple them and make it harder to
> reason about and modify them.

More or less the same idea; I use with plist. I think they a bit
simpler, but the same underground. I tend to avoid eioio because I have
heard some performance complains (don't know if they are justified) but
also because plists are somehow more flexible in some aspects.

> In regards to interactivity, I outsourced a lot to another package I
> maintain called compile-multi [3]. At its core its just a menu for
> predicates and compile commands. When predicate X, you have the option
> to choose command Y. I've plugged it into projection so that through it
> you can see project type specific options. For example in a CMake
> project this menu shows you all compilation targets and all CTest
> targets. Running a selected target will run the compile command that
> builds that target. If you want to permanently override one that runs
> with the standard projection-commands-build-project or
> projection-commands-test-project interactive commands (this is the same
> as setting what you're wanting to build going forward) I recommend using
> of embark [4]. What I normally do is run `M-x projection-multi-compile`
> narrow to the build or test target I care about and then run `M-x
> embark-act p c` (or p t for a test) and next time I run
> projection-commands-build-project it'll use that selected command. If I
> want to reset this to the project default I can use `M-x
> projection-cache-clear` which shows all cached project options and lets
> me select one or more to prune. Next time I run any command it'll be
> the default for the current project.
>
> [3]: https://github.com/mohkale/compile-multi
> [4]: https://github.com/oantolin/embark
Actually I am not trying to do anything so fancy. We cannot go from 0 to
100 in a single pass. If we start 1. recognizing projects backends and
giving a better compile-command for them in the right place and to the
right build-dir is a good step forward compared to what we have. Then we
talk about test and finally we can consider the generate one.

> I think generalising this latter feature to a common interface will be
> tricky. To begin with just knowing a project has build target X doesn't
> really tell you how to build X, just that you can. You need to feed that
> selection back into the project-type or export it directly as is. I
> recently added a feature to list build artifacts (for debugger
> integration) and that shows just how much variety there is here.
>
> Lastly, just a general point (sorry for writing so much), I think any
> feature for this shouldn't assume only a single project type is valid.
> There should be a concept of a primary type just to make commands like
> configure or build make sense relative to each other, but nothing stops
> a project from matching two types at once and at least IME that's the
> more natural state of things. I commonly work on C++ projects but we
> also have a package.json in those projects file so we can depend on
> prettier and other non-C++ specific linters. The way projection gets
> around this is by always matching all defined project types. Users can
> add to the list of matched types for the current project and can cycle
> the primary project type. Certain interfaces that aren't single project
> specific like the aforementioned projection-multi-compile will source
> compilation targets from all applicable project types. This provides an
> extremely rich interface for interacting with projects (at least IMO).
At the moment project.el only handles one backend at the time. I made a
simple workaround to extend it a bit, but that is a bit tricky and at
this point simplicity matters. Otherwise we end up with an inefficient
and complex interface easy to break and hard to make it work.

> Let me know if anyone has any thoughts or questions about projection.
> I'm not averse to re-aligning across something in Emacs mainline. The
> only minor concern is how specific it is to the kinds of projects I work
> on and that may not generalise well to others.
>
> --
> Mohsin Kaleem
Ergus