Emacs deadlocks on sub-process reads and writes

[speech . free]

I am debugging a Clang based code completer (irony-mode). Unfortunately the 
transfer between Emacs and the Clang sub-process is slow on Windows. We are 
experiencing frequent freeze-ups due to what looks like a deadlock. The 
deadlock happens under the following conditions:

1. Emacs reading a large transfer from the sub-process (2+MB, around 2 seconds 
to finish on Windows after setting w32-pipe-read-delay to 0);

2. While 1. is ongoing Emacs side send a new request to the sub-process. The 
request is regular source file sized -- it includes the buffer content being 
edited -- sufficient to cause a block on the write but nothing too big.

When these two conditions are met, Emacs UI reliably locks up until the 
sub-process is killed. The error message complaining about writing to a dead 
process after killing the sub-process shows that the Emacs was deadlocked 
waiting for the write to finish. The process buffer is half filled showing that 
the process filter function was no longer called. The write to the sub-process, 
which blocks the UI, can't proceed as it exceeds the size of the small write 
buffer. The sub-process will not read on its side until it finishes the 
transfer to Emacs. But for some reason Emacs also stopped reading from the pipe.

After poking around the Emacs code base a little bit it looks to me that when 
you call process-send-* in lisp Emacs eventually calls send_process in 
process.c. If the send can't complete it calls wait_reading_process_output, 
which calls read_and_dispose_of_process_output, which if there is new text 
received, calls the lisp process-filter. The logic in 
wait_reading_process_output is too complicated for me to understand if somehow 
it may decide not to call read_process_output, causing a deadlock. So I am 
asking for advice here.

A previous related question asked about transfer speed on Windows 
https://groups.google.com/forum/#!topic/gnu.emacs.help/Y0MYAz8WZUU I believe 
the slowness is the reason why this manifests itself on Windows but it may also 
appear on Linux under the right conditions. For a very long thread of 
discussion on this issue see https://github.com/Sarcasm/irony-mode/issues/131

Please feel free to forward to any Emacs list that may be relevant to solving 
this issue.