Re: LLM Experiments, Part 1: Corrections

[Karthik Chikmagalur]

Hi Andrew,

Having worked on similar problems in gptel for about nine months now
(without much success), here are some thoughts.

> Question 1: Does the llm-flows.el file really belong in the llm 
> package?  It does help people code against llms, but it expands 
> the scope of the llm package from being just about connecting to 
> different LLMs to offering a higher level layer necessary for 
> these more complicated flows.  I think this probably does make 
> sense, there's no need to have a separate package just for this 
> one part.

I think llm-flows works better as an independent package for now,
since it's not clear yet what the "primitive" operations of working
with LLMs look like.  I suspect these usage patterns will also be a
matter of preference for users, as I've realized from comparing how I
use gptel to the feature requests I get.

> Question 2: What's the best way to write these flows with multiple 
> stages, in which some stages sometimes need to be repeated? It's 
> kind of a state machine when you think about it, and there's a 
> state machine GNU ELPA library already (fsm). I opted to not model 
> it explicitly as a state machine, optimizing instead to just use 
> the most straightforward code possible.

An FSM might be overkill here.  At the same time, I'm not sure that
all possible interactions will fit this multi-step paradigm like
rewriting text does.

> Question 3: How should we deal with context? The code that has the 
> text corrector doesn't include surrounding context (the text 
> before and after the text to rewrite), but it usually is helpful. 
> How much context should we add? The llm package does know about 
> model token limits, but more tokens add more cost in terms of 
> actual money (per/token billing for services, or just the CPU 
> energy costs for local models). Having it be customizable makes 
> sense to some extent, but users are not expected to have a good 
> sense of how much context to include. My guess is that we should 
> just have a small amount of context that won't be a problem for 
> most models. But there's other questions as well when you think 
> about context generally: How would context work in different 
> modes? What about when context may spread in multiple files? It's 
> a problem that I don't have any good insight into yet.

I see different questions here:

1.  What should be the default amount of context included with
requests?
2.  How should this context be determined? (single buffer, across
files etc)
3.  How should this be different between modes of usage, and how
should this be communicated unambiguously?
4.  Should we track token costs (when applicable) and communicate them
to the user?

Some lessons from gptel, which focuses mostly on a chat interface:

1.  Users seem to understand gptel's model intuitively since they
think of it like a chat application, where the context is expected to
be everything in the buffer up to the cursor position.  The only
addition is to use the region contents instead when the region is is
active.  This default works well for more than chat, actually.  It's
good enough when rewriting-in-place or for continuing your prose/code.

2.  This is tricky, I don't have any workable ideas yet.  In gptel
I've experimented with providing the options "full buffer" and "open
project buffers" in addition to the default, but these are both
overkill, expensive and rarely useful -- they often confuse the LLM
more than they help.  Additionally, in Org mode documents I've
experimented with using sparse trees as the context.  This is
inexpensive and can work very well but the document has to be
(semantically) structured a certain way.  This becomes obvious after a
couple of sessions, but the behavior has to be learned nevertheless.

3.  For coding projects I think it might be possible to construct a
"sparse tree" with LSP or via treesitter, and send (essentially) an
"API reference" along with smaller chunks of code.  This should make
copilot-style usage viable.  I don't use LSP or treesitter seriously,
so I don't know how to do this.

3b. Communicating this unambiguously to users is a UI design question,
and I can imagine many ways to do it.

4.  I think optionally showing the cumulative token count for a
"session" (however defined) makes sense.

> Question 5: Should there be a standard set of user behaviors about 
> editing the prompt? In another demo (one I'll send as a followup), 
> with a universal argument, the user can edit the prompt, minus 
> context and content (in this case the content is the text to 
> correct). Maybe that should always be the case. However, that 
> prompt can be long, perhaps a bit long for the minibuffer. Using a 
> buffer instead seems like it would complicate the flow. Also, if 
> the context and content is embedded in that prompt, they would 
> have to be replaced with some placeholder. I think the prompt 
> should always be editable, we should have some templating system. 
> Perhaps emacs already has some templating system, and one that can 
> pass arguments for number of tokens from context would be nice.

Another unsolved problem in gptel right now.  Here's what it uses
currently:

- prompt: from the minibuffer
- context and content: selected region only

The main problem with including context separate from the content here
is actually not the UI, it's convincing the LLM to consistently
rewrite only the content and use the context as context.

> Question 6: How do we avoid having a ton of very specific 
> functions for all the various ways that LLMs can be used? Besides 
> correcting text, I could have had it expand it, summarize it, 
> translate it, etc. Ellama offers all these things (but without the 
> diff and other workflow-y aspects). I think these are too much for 
> the user to remember.

Yes, this was the original reason I wrote gptel -- the first few
packages for LLM interaction (only GPT-3.5 back then) wanted to
prescribe the means of interaction via dedicated commands, which I
thought overwhelmed the user while also missing what makes LLMs
different from (say) language checkers like proselint and vale, and
from code refactoring tools.

> It'd be nice to have one function when the 
> user wants to do something, and we work out what to do in the 
> workflow. But the user shouldn't be developing the prompt 
> themselves; at least at this point, it's kind of hard to just 
> think of everything you need to think of in a good prompt.  They 
> need to be developed, updated, etc. What might be good is a system 
> in which the user chooses what they want to do to a region as a 
> secondary input, kind of like another kind of 
> execute-extended-command.

I think having users type out their intention in natural language into
a prompt is fine -- the prompt can then be saved and added to a
persistent collection.  We will never be able to cover
(programmatically) even a reasonable fraction of the things the user
might want to do.

The things the user might need help with is what I'd call "prompt
decoration".  There are standard things you can specify in a prompt to
change the brevity and tone of a response.  LLMs tend to generate
purple prose, summarize their responses, apologize or warn
excessively, etc.  We can offer a mechanism to quick-add templates to
the prompt to stem these behaviors, or encourage other ones.

Karthik