Hi,
sorry for the previous long mail.
Here is quite a short solution:-)
In edif2.cc
replace
// int inotify_rc = inotify_add_watch(inotify_fd, dir,
IN_CREATE | IN_MODIFY );
by
int inotify_rc = inotify_add_watch(inotify_fd, dir,
IN_CLOSE_WRITE);
So far it solved the problem.
Explanation:
IN_NOTIFY gets fired if as soon as a file gets created.
IN_CLOSE_WRITE as soon as the file content got fully written and
the file got closed. So reading the file content is save
Best Regard
Hans-Peter
Am 02.12.20 um 16:27 schrieb
Hans-Peter Sorge:
Hi Chris,
I have done some inotify tests to see what happens using
different editors.
Editing a FUNCTION in apl-ws will
1. start edif2 and create a file FUNCTION.apl - this file create
will directly trigger inotify messages
inotify_test/ CREATE FUNCTION.apl
inotify_test/ OPEN FUNCTION.apl
inotify_test/ MODIFY FUNCTION.apl
inotify_test/ CLOSE_WRITE,CLOSE FUNCTION.apl
based on the events one FX
FUNCTION is being processed by edif2 - even if two monitors
(CREATE and MODIFY) are set.
I did not dig into this whether edif2 receives combined CREATE,
MODIFY events as one event.
2. edif2 then starts the editor "emacs FUNCTION.apl" which in
turn keeps inotify rather busy:
--- start emacs
inotify_test/ OPEN FUNCTION.apl
inotify_test/ CLOSE_NOWRITE,CLOSE FUNCTION.apl
inotify_test/ OPEN FUNCTION.apl
inotify_test/ CLOSE_NOWRITE,CLOSE FUNCTION.apl
inotify_test/ OPEN FUNCTION.apl
inotify_test/ ACCESS FUNCTION.apl
inotify_test/ CLOSE_NOWRITE,CLOSE FUNCTION.apl
inotify_test/ OPEN,ISDIR
inotify_test/ CLOSE_NOWRITE,CLOSE,ISDIR
inotify_test/ OPEN,ISDIR
inotify_test/ CLOSE_NOWRITE,CLOSE,ISDIR
inotify_test/ OPEN,ISDIR
inotify_test/ CLOSE_NOWRITE,CLOSE,ISDIR
inotify_test/ OPEN,ISDIR
inotify_test/ CLOSE_NOWRITE,CLOSE,ISDIR
inotify_test/ OPEN,ISDIR
inotify_test/ CLOSE_NOWRITE,CLOSE,ISDIR
--- when start editing
inotify_test/ CREATE .#FUNCTION.apl
--- save from editor
inotify_test/ MODIFY FUNCTION.apl --
inotify messages do not guarantee a stable order of messages
from events
inotify_test/ OPEN FUNCTION.apl -- inotify
messages do not guarantee a stable order of messages from events
inotify_test/ MODIFY FUNCTION.apl --
inotify messages do not guarantee a stable order of messages
from events
inotify_test/ CLOSE_WRITE,CLOSE FUNCTION.apl
inotify_test/ OPEN FUNCTION.apl
inotify_test/ CLOSE_WRITE,CLOSE FUNCTION.apl
inotify_test/ DELETE .#FUNCTION.apl
inotify_test/ OPEN FUNCTION.apl
inotify_test/ CLOSE_NOWRITE,CLOSE FUNCTION.apl
--- again edif2 produces only one FX FUNCTION despite 2xMODIFY
events
--- leaving the editor -
no message
----- for slickedit ---------
--- open buffer
inotify_test/ OPEN FUNCTION.apl
inotify_test/ ACCESS FUNCTION.apl
inotify_test/ CLOSE_WRITE,CLOSE FUNCTION.apl
--- start editing
--- no event
--- save buffer
inotify_test/ OPEN FUNCTION.apl -- inotify
messages do not guarantee a stable order of messages from events
inotify_test/ MODIFY FUNCTION.apl --
inotify messages do not guarantee a stable order of messages
from events
inotify_test/ MODIFY FUNCTION.apl --
inotify messages do not guarantee a stable order of messages
from events
inotify_test/ CLOSE_WRITE,CLOSE FUNCTION.apl
inotify_test/ OPEN FUNCTION.apl
inotify_test/ ACCESS FUNCTION.apl
inotify_test/ OPEN FUNCTION.apl
inotify_test/ ACCESS FUNCTION.apl
inotify_test/ CLOSE_NOWRITE,CLOSE FUNCTION.apl
inotify_test/ CLOSE_WRITE,CLOSE FUNCTION.apl
--- leaving the buffer -
no message
* -- inotify messages do not guarantee a stable order of
messages from events.
I observed events in different order:
OPEN, MODIFY, MODIFY or
MODIFY, OPEN, MODIFY or even
OPEN, MODIFY, MODIFY, MODIFY.
More diverse message are being sent by different editors.
btw. I have setup emacs config to not create a backup file.
Handling of backup FUNCTION.apl~ would be even more verbose.
Coming back to where things go wrong:
in APL WS edit function:
- 'EDITOR' edif2 'FUNCTION'
edif2 starts processing:
- A create a file from function
(- write file content from function)
- inotify gets triggered - MODIFY
- B read file function
(- write file content from function)
- fx function
- open external EDITOR FUNCTION.apl
in EDITOR do nothing
then close EDITOR
The race is on between A and B.
If the file content got written before B all is fine.
If not, the file read happens during the write, by that
truncating the FUNCTION.
Because of the verbosity of inotify and the editors handling of
files
it seems to be tricky to rely on a specific inotify message or
even rely
on some specific sequence of inotify messages for a consistent
file processing.
Some possible solutions:
1- Serialize the events: stop inotify, file write, file
edit, start inotify
The first FX caused by
FUNCTION-write-file/inotify-MODIFIED could be prevented
However - if MODIFIED creates a message, it
seems (my feeling is), that MODIFIED
does not guaranty, that the full FUNCTION content got written by
that time.
It could be that some things are going
on at the discretion of the editor ( or even the OS) to run
unrelated processes in any order.
2- Set a timer to .... ? NO.. NEVER EVER:-)
3- Make file_name unique. e.g. by adding a full timestamp
(yyyymmddhhmmssmmm.FUNCTION.apl). Or create a unique path for
file_name.
This could even open an way for experiments -
open the same function twice for editing.
make changes to instance A of FUNCTION - test in APL
make changes to instance B of FUNCTION - test in APL
decide which is better ...
I am still not sure, that the race condition could be
resolved by adding a timestamp.
4- Add some "intelligent" message filter.
Lots of work with little hope for success.
5- As edif2 tries to sync two unrelated processes (APL,
EDITOR) based on events outgoing from either APL or EDITOR at
any point in time
there needs to be a an arbiter which has the control.
APL -> EDIF2 -- I want to edit FUNCTION
EDIF2 -- OK I'll set up the environment
-- start EDITOR
EDIF2 -- pick FUNCTION from APL
-- create file for editing
-- let EDITOR pickup the file
But then from here on the same a sync inotify problems
remain.
6- Let the editor "directly" talk to EDIF2.
Create a save macro that generates a file FUNCTION.svd
Editor-save writes file FUNCTION.svd
EDIF2 reacts to inotify FUNCTION.svd only.
And again - inotify might even fire MODIFIED as soon as
the file name got established
and the content was not written in total.
vi is even more active on all sorts of backup files, attribute
changes.
nano, to the contrary, has the least amount inotify events.
Having gone through 1..6 and the inotify events from different
editors, finally the following state handling is likely to fix
the situation:
while receive inotify events {
if (MODIFY FUNCTION.apl) {
while receive inotify events {
if (CLOSE_WRITE,CLOSE FUNCTION.apl ) {
read file FUNCTION.apl
FX FUNCTION
break;
}
if (CLOSE_NOWRITE,CLOSE FUNCTION.apl)
break
if (ANY_OTHER_INOTIFY_EVENT) continue // ignore
}
}
With all four editors the inotify sequence
MODIFY, ..., CLOSE_WRITE,CLOSE
identifies a save file.
... denotes inotify events depending on EDITOR's behavior.
The inotify events CLOSE_NOWRITE,CLOSE so far seems to be a save
event to escape the loop.
THE events happen too when the editors get closed w/o
editing/saving.
Finally:
The tricky part. There is one state loop for possibly more than
one file being edited.
soooo - Item 5 comes into play again.
Currently edif2 is state less.
If the state analyzer is being implemented one might need to
implement a file status state too.
// some very rough ideas .....
// warning - untested code ...
// more warning not 100% C++
class inotifyFileStateVector( ){
// initialize for each instance
bool modify=false;
bool clnwc = false;
string FN = fn;
public:
inotifyFileStateVector( fn);
~ inotifyFileStateVector( );
}
bool
inotifyFileStateVector::checkFN(event){
int eventT = event.type()
string fn = evetn.fileName()
if ( FN != fn ) return (false)
if (eventT == MODIFY) modify=true;
if ((eventT ==CLOSE_NOWRITE || CLOSE) && modify )
clnwc=true;
if ( modify && clnwc ) {
modify=false;
clnwc = false;
return(true)
}
return (false);
}
-------------------------------
inotifyFileStateVector iFSV
// keep instances of function states in a vector
...
// check - only one entry for each FUNCTION_name allowed.
bool exists=false;
for (iFSV) {
if ( FUNCTION_name == iFSV.FN) exists=true;
}
if (! exists){
......
// append new instance
iFSV.append(new iFSV(FUNCTION_name))
...
} else {
// ignore new edit request
// maybe return message to user "Being edited."
// maybe bring editor window to foreground.
}
.......
// check each message against the vector of function states
while receive inotify events {
for (ifsv in iFSV) {
bool aSaveOccured = ifsv.checkFN(inotify_event);
if ( aSaveOccured ) {
.....
FX ifsv.FN
.....
}
}
}
---------------------------------------
Best Regards
Hans-Peter
Am 30.10.20 um 16:50 schrieb Chris
Moller:
Looks
like it might be a concurrency problem--edif2 forks, IIRC,
twice, serially, and I never was completely sure that I had
the waitpid()s right. I'll take a look.
Chris
On 10/30/20 11:22 AM, Hans-Peter Sorge wrote:
Hi,
working with an external editor under some circumstances
some function content gets destroyed:
)wsid
IS CLEAR WS
'libedif2.so' ⎕FX 'edif2'
edif2
'emacs' edif2 'FN1' - adding two lines, save and
exit
'emacs' edif2 'FN1' - body OK
'emacs' edif2 'FN2' - no entry, just exit
'emacs' edif2 'FN1' - last line got lost
'emacs' edif2 'FN3' - no entry, just exit
'emacs' edif2 'FN1' - just the first line is left
over.
.. and the user interface (KDE) is unresponsive for a
moment
Best Regards
Hans-Peter