bug#63870: 29.0.90; project.el can't dynamically populate the project list

[Spencer Baugh]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Spencer Baugh <sbaugh@janestreet.com>
>> Cc: 63870@debbugs.gnu.org
>> Date: Wed, 28 Jun 2023 08:05:23 -0400
>> 
>> Eli Zaretskii <eliz@gnu.org> writes:
>> 
>> > Beware of watching a tree recursively: file notifications are not very
>> > scalable, for more than one reason.  For example, the inotify backend
>> > consumes a file descriptor and a slot in the descriptor set monitored
>> > by pselect per each file/directory you watch.  And watching many
>> > directories can overwhelm Emacs if some program (even unrelated to
>> > Emacs) performs many file operations in that directory; VCS programs
>> > are notorious in this regard, e.g., when you update from upstream.
>> 
>> Absolutely.  I am trying to be careful about this: project-watch
>> shouldn't create watches on VCS directories.
>
> But below you explicitly give an example where it will.  And given the
> fact that the majority of project.el projects use VCS as its backend,
> I'd say we are already there...

No: the watch on a directory is removed once the directory becomes a VCS
directory.

>> > Are you sure this feature justifies the risks?  When would someone
>> > want to use it, while simultaneously limiting the value of RECURSIVE
>> > to some small integer?  (And what is considered "small" for these
>> > purposes?)
>> 
>> Imagine, for example, that a user has a directory ~/src.  They make all
>> their VCS clones directly under ~/src: ~/src/emacs, ~/src/glibc, etc.
>> And when they work on a new project, they create that new clone under
>> ~/src.
>> 
>> If the user wanted all these VCS clones to show up in Emacs as soon as
>> they're made, they could run (project-watch "~/src" 1).  This would
>> create a watch on ~/src, which would create watches on new empty
>> directories under ~/src (e.g. ~/src/gdb); the watch on ~/src/gdb would
>> stop if and when ~/src/gdb becomes a project (as defined above).
>> 
>> So in the steady state, if ~/src contains only projects, Emacs would run
>> exactly one watch, the one on ~/src.  This is definitely okay.
>> 
>> If, instead, ~/src has a two-level structure, where ~/src/emacs is not
>> itself a clone but instead contains a clone for each branch,
>> e.g. ~/src/emacs/emacs-29 and ~/src/emacs/trunk, then a user might run
>> (project-watch "~/src" 2).  Then in the steady state there would be one
>> watch on ~/src and one watch on each subdirectory of ~/src,
>> e.g. ~/src/emacs.  (This is the setup I personally have.)
>
> If you want to support one or two levels of recursion, let's support
> just that and remove the too-general RECURSIVE argument.  If you think
> there might be important use cases where there's more than one or two
> levels of recursion, please describe them.

Hm, well, I assume some users might use even more structure than this;
for example, some might have ~/src/gnu/emacs/emacs-29.  Then they'd want
3 levels of recursion.

> Once again, this is dangerous; users could easily shoot themselves in
> the foot, because not many are aware of the pitfall of using file
> notifications for many directories.  It makes no sense to warn against
> something and at the same time let callers easily stumble upon that.

I agree with that, I suppose.  Personally I would be fine with a
mandatory 1 or 2 levels of recursion, since I only need 2.  Do you have
a suggestion for what that interface could look like?  It feels a bit
awkward...