Re: Calling readline() a second or subsequent time and retaining state from previous calls

[Sam Habiel]

I fixed the double free() issue by doing rl_free_line_state  BEFORE 
rl_save_state. That's fixed. However, I still have a leak of 5 bytes according 
to LSAN, which I think is some history structure not getting freed. 
Unfortunately LSAN isn't showing me the trace beyond two levels, and all that 
says is that it happened in xalloc.c, nothing else.

--Sam

 ---- On Fri, 29 Sep 2023 11:24:53 -0400  Sam Habiel  wrote --- 
 > Thank you Chet.
 > 
 > I proceeded with implementing something that is split up across calls to 
 > readline. It's not perfect, and I am writing to you about the problems I 
 > faced. I totally understand if you think it's crazy and think I should 
 > abandon the effort, but I mostly got it to work.
 > 
 > The current source code is here: 
 > https://gitlab.com/shabiel/YDB/-/blob/ydb83_readline/sr_unix/readline.c (it 
 > will change a bit in the next few days, but not materially, so you can still 
 > refer to it).
 > 
 > My approach can be summarized as follows:
 > 
 > When I receive a SIGUSR1, siglongjmp to code that does the following:
 > - save the current state using rl_save_state
 > - Copy out the state->buffer and state->prompt
 > - NULL out rl_prompt (otherwise, it gets freed by readline on the next call)
 > - free state->buffer
 > - free state->prompt
 > - Put my buffer and prompt back into the state structure (but I did the 
 > allocation and they won't disappear)
 > - unwind the C stack
 > - Call readline again with the following hooks
 > -- startup hook: restore state using rl_restore_state, and set 
 > rl_already_prompted to 1.
 > -- redisplay hook: no-op: The text is already on the screen and we don't 
 > want it displayed again.
 > 
 > I compiled readline from master with -g -O3 so I can see what's going on.
 > 
 > Here are the two major issues I faced:
 > 1. I want to call rl_free_line_state after saving state, but doing so, after 
 > restoring state, I get a double free warning from ASAN. The undo list is 
 > attempted to be freed again:
 > 
 > ==68415==ERROR: AddressSanitizer: attempting double-free on 0x603000001ae0 
 > in thread T0:
 >     #0 0x7f5e7c27edb8 in __interceptor_free (/usr/lib64/libasan.so.4+0xdbdb8)
 >     #1 0x7f5e763cb117 in _rl_free_undo_list ../undo.c:111
 >     #2 0x7f5e763cb178 in rl_free_undo_list ../undo.c:122
 >     #3 0x7f5e763ac32c in readline_internal_teardown ../readline.c:512
 >     #4 0x7f5e763ada1a in readline_internal ../readline.c:734
 >     #5 0x7f5e763ada1a in readline ../readline.c:387
 >     #6 0x7f5e76d2c7ef in readline_read_mval 
 > /home/sam/work/gitlab/YDB/sr_unix/readline.c:428
 >     #7 0x7f5e77fa53b7 in op_dmode 
 > /home/sam/work/gitlab/YDB/sr_port/op_dmode.c:126
 >     #8 0x7f5e776f0c50  (/home/sam/work/builds/YDB/libyottadb.so+0xfaac50)
 > 
 > 0x603000001ae0 is located 0 bytes inside of 32-byte region 
 > [0x603000001ae0,0x603000001b00)
 > freed by thread T0 here:
 >     #0 0x7f5e7c27edb8 in __interceptor_free (/usr/lib64/libasan.so.4+0xdbdb8)
 >     #1 0x7f5e763cb117 in _rl_free_undo_list ../undo.c:111
 > 
 > previously allocated by thread T0 here:
 >     #0 0x7f5e7c27f110 in malloc (/usr/lib64/libasan.so.4+0xdc110)
 > 
 > 2. If I bypass the readline redisplay, I cannot see any characters being 
 > typed. I tried calling rl_reset_terminal and rl_tty_set_echoing to no avail. 
 > I think it has to do with internal readline state (maybe it thinks my cursor 
 > is somewhere else), but the re-display code is so complex that it's hard for 
 > me to tell.
 > 
 > Let me know if you have any advice; and if not, that's fine too... I do know 
 > what I am doing is crazy, and maybe when we switch in the future to the 
 > callback interface, things would be easier and I won't need all of these 
 > shenanigans.
 > 
 > --Sam
 > 
 > 
 >  ---- On Mon, 25 Sep 2023 09:30:44 -0400  Chet Ramey  wrote --- 
 >  > On 9/21/23 12:34 PM, Sam Habiel wrote:
 >  > > Hello there,
 >  > > 
 >  > > Quick Intro: We have an interactive CLI for a language called M used in 
 > banking and healthcare... I am working on adding readline integration as a 
 > nice to have feature. We are an open source project located over here: 
 > https://gitlab.com/YottaDB/DB/YDB.
 >  > > 
 >  > > I sent an earlier message, and I thank Chet for the reply.
 >  > > 
 >  > > I am dealing with the unpleasant for anybody to handle signals. One of 
 > our signals suspends execution, completely unwinds the C stack (yes, we have 
 > lots of assembly code for that), and then reissues the prompt. An example 
 > will help:
 >  > > 
 >  > > YDB>write "fo  <<< Receive SIGUSR1
 >  > > 
 >  > > Once a SIGUSR1 is received, in the background, execution is suspended, 
 > a custom function is executed, and the entire prompt code is actually 
 > processed, but the user sees absolutely nothing of that. In the example 
 > above, the cursor will stay between f and o. Our current code just sets up 
 > all the prompt data structures from a saved structure when the signal is 
 > received.
 >  > > 
 >  > > Is this possible with readline()? I naively tried rl_save_state() and 
 > rl_restore_state(), but that doesn't seem to work.
 >  > > 
 >  > 
 >  > SIGUSR1 isn't one of the signals readline handles by default, but it has
 >  > enough API functions available so that you can do it in your application.
 >  > 
 >  > All the necessary functions are in signals.c. The basic idea is that you
 >  > install your signal handler as usual, call the readline functions you
 >  > want to restore the terminal and anything else from the handler when it's
 >  > safe (look at _rl_signal_handler() for examples), then restore before your
 >  > handler returns (look at rl_cleanup_after_signal/rl_restore_after_signal).
 >  > 
 >  > If you want to split it up across calls to readline, you'll have to save
 >  > and restore a lot of state. I don't know if the information in
 >  > struct readline_state is enough for you -- it depends on your application
 >  > needs -- and it's not really intended to save and restore across calls
 >  > to readline(). It's mostly intended as a checkpoint during a single call,
 >  > so the interaction between the initialization readline does on each
 >  > invocation and the state restored isn't well exercised.
 >  > 
 >  > If you want to go further with that approach, you'll need at least a
 >  > startup hook to restore the state and do a redisplay before going on
 >  > in the second (post-signal) call to readline.
 >  > 
 >  > Chet
 >  > 
 >  > -- 
 >  > ``The lyf so short, the craft so long to lerne.'' - Chaucer
 >  >          ``Ars longa, vita brevis'' - Hippocrates
 >  > Chet Ramey, UTech, CWRU    chet@case.edu    http://tiswww.cwru.edu/~chet/
 >  > 
 >  > 
 >