[Gluster-devel] preventing gfid-mismatches because of crashes in afr

[Pranith Kumar Karampuri]

hi,

   Traditionally afr just remembers which of the directories are good vs stale 
in extended attributes and then at the time of self-heal, does full directory 
scan and deletes stale entries and creates new entries. There are two problems 
with this approach
1) even creating/deleting/renaming one entry requires full scan of the 
directory.
2) If both bricks crash at the same time while a rename is going on, then it 
can lead to same-name, different gfid split-brains.
   Example:
            0) dir1 has file 'a' with gfid-a, dir2 has file 'b' with gfid-b.
            1) user executes rename dir1/a -> dir2/b on the mount over-writing 
the original file b.
            2) On brick-0 rename succeeds so the end result is dir1 does not 
have 'a' and dir2 has file 'b' with gfid-a
            3) at this point both the brick processes go down or data center 
shutdown happens etc, so brick-1 still has dir1 with file 'a' with 'gfid-a' and 
dir2 with file 'b' with 'gfid-b'.
            4) Now when both bricks are back up, dir1 can be healed 
conservatively where 'a' will be recreated with 'gfid-a' and heal it from 
brick-1 to brick-0 (incorrectly undoing the rename).
            5) But for dir2 on brick-0 there is a file 'b' with gfid-a where as 
on brick-1 there is a file 'b' with 'gfid-b', afr at the moment doesn't store 
any information to figure out which one is correct.

To address this issue, granularity of preop/postop of the entry operations need 
to be incremented.
a filename inside a directory can be uniquely identified by the entry-tuple 
(parent-gfid, entryname, entry-gfid).
Example: For dir2/b in the example above we can represent it as (gfid-of-dir2, 
b, gfid-b) on brick-1

So we need to remember such information for every entry fop along with whether 
that entry is coming 'in' to the directory or going 'out' of the directory.
So in the previous example we would have remembered dir2/b with gfid-b is going 
out of that directory so that entry could be deleted and dir2/b with gfid-a can 
be healed from brick-0.

The solution that we come up with should have the following functionalities 
broadly:
1) Given an entry-tuple it should be able to remember that it is going in or 
out of that directory.
2) Given an existing entry-tuple it should be able to forget it.
3) Given an entry-tuple, we should be able to query if that entry-tuple is 
going in/out.

This is one possible way to address this issue:
0) Create directory .glusterfs/indices/entry and two files 'in', 'out' in that 
directory and
1) Every time creat/mknod/symlink/link/mkdir happens create a hardlink from 
following path .glusterfs/indices/entry/pargfid/gfid/filename to 
'.glusterfs/indices/entry/in' as part of pre-op
2) Every time unlink/rmdir happens create a hardlink from following path inside 
.glusterfs/indices/entry/pargfid/gfid/filename to 
'.glusterfs/indices/entry/out' as part of pre-op
3) Every time rename happens create the following 2/3 hardlinks
   - .glusterfs/indices/entry/old-pargfid/gfid/old-filename to 
'.glusterfs/indices/entry/out'
   - .glusterfs/indices/entry/new-pargfid/gfid/new-filename to 
'.glusterfs/indices/entry/in'
and if the destination exists:
   - .glusterfs/indices/entry/new-pargfid/exisiting-file-gfid/new-filename to 
'.glusterfs/indices/entry/out'
4) Delete the same files as part of post-op.

To improve upon the solution we can do some optimizations:
Max filename is 255 bytes. And pargfid, gfid can take 16 bytes each.
So
1) If the file that is created/deleted/renamed is <= 223 (filename-max-len(255) 
- twice-gfid-len(32) = 223) then instead of representing the entry-tuple as 
pargfid/gfid/filename (i.e. two directories and a filename) it can be 
represented as modified-filename: pargfidgfidfilename i.e. first 16 bytes 
pargfid next 16 bytes as gfid and the rest as filename. Instead use this 
filename as link to 'in', 'out'. (2 mkdirs are saved)
2) If the file that is created/deleted/renamed is <= 249 and > 223 then we can 
probably use pargfid/gfidfilename as the link. (1 mkdir is saved)

Let me know your thoughts and do let me know If there is an easier way which 
can satisfy all the functionalities I listed above.

Thanks to Niels for listening to the initial approach and reading the initial 
draft :-).

Pranith