bug#22768: Crash safety

[Paul Eggert]

On 03/01/2016 10:13 AM, Antonio Diaz Diaz wrote:
> [1] http://news.mit.edu/2015/crash-tolerant-data-storage-0824

FSCQ is not even close to ready for prime-time, I'm afraid. Its 
prototype is slow compared to conventional file systems (it assumes a 
single-threaded kernel, it issues many more writes than ext4 does to 
implement a commit, its has been publicly tested only on flash drives, 
etc.). The FSQC authors would like to add support for fsync/fdatasync to 
get some of that performance back, which seems reasonable -- but at that 
point, applications like gzip would still need to call fsync/fdatasync 
to avoid losing data.

You may well be right that eventually file system designers will figure 
this stuff out so that well-written POSIX applications will not lose 
data even if they don't use fsync/fdatasync. However, if FSCQ is any 
indication, we're many years away from that. In the meantime 
fsync/fdatasync is all we have.

> My point is that if gzip has run unsafely for decades without a 
> reported failure, maybe all those who take these things seriously are 
> already using file systems safe enough to guarantee that well behaved 
> tools like gzip do not lose data.

That will be true for many users. Still, I imagine that non-experts 
would have a good deal of trouble connecting the dots between lost data 
and any gzip invocation that lost the data, and could chalk it up to a 
system crash losing data for other reasons.  (After all, things are 
somewhat chaotic during a crash...)  One can find examples on the net 
like "How to recover lost/deleted Gzip compressed gz file" (this is for 
BYclouder, a commercial tool) that talk about system crashes, and which 
indicate (though do not prove) that a real problem exists with gzip.

My sources:

Chen H, Ziegler D, Chajed T, Chlipapa A, Kaashoek MF, Zeldovich N. Using 
Crash Hoare logic for certifying the FSCQ file system. SOSP 2015. 
https://people.csail.mit.edu/nickolai/papers/chen-fscq.pdf

How to recover lost/deleted Gzip compressed gz file. BYclouder. 
2013-04-23. 
http://www.byclouder.com/help/recovery/file/archive/how-to-recover-gzip-gz.html