Re: [Gluster-devel] file version on glusterfs using libgit

[Niels de Vos]

On Mon, Mar 11, 2013 at 08:34:00AM +0530, Kaleb KEITHLEY wrote:
> On 03/08/2013 04:50 PM, Niels de Vos wrote:
> >On Fri, Mar 08, 2013 at 06:00:24AM -0500, Shishir Gowda wrote:
> >>Hi Niels,
> >>
> >>Thinking out aloud, I think the snaps(in file version context) can be
> >>displayed as branches (list).
> >
> >Well, I am not sure if branches are really needed. Isn't linear history
> >sufficient? Every change should be committed to the master branch
> >anyway. Branches may be useful for switching between versions, but
> >nothing prevents you from checking out (or "git ls-files") with a commit
> >or a date.
> 
> I think snapshots can be implemented with tags, and had also
> envisioned a virtual directory scheme like Niels has described.
> There could/should be a cron job that makes yearly, monthly, daily,
> and/or even hourly snapshots.

A cronjob would not be needed if this git-xlator creates the tags on 
a readdir/lookup/... Some of the tags do not even need to exist, and can 
be generated on the fly. Personally I prefer as few additional scripts 
and configuration files as possible. Of course, giving users the ability 
to create their own tags would be most welcome too.

> I could imagine using branches for a kind of auto-dedupe copy
> mechanism. If I copy a file to another location, a branch is made,
> and the new copy is a magic symlink that points to the branch.

Ah, yeah, that is something branches could help with. I'm not sure how 
practical the implementation would be, but I'll leave that to Shishir :D

Cheers,
Niels

> >
> >I'm thinking of a virtual .snaps directory:
> >
> >$ cd $VOLUME/.snaps
> >$ ls
> >2013-03-07/
> >2013-03-06/
> >.....
> >current/
> >changelog
> >yesterday -> 2013-03-07/
> >
> >This makes it possible to do something like:
> >$ cat changelog
> >    - virtual file, showing the contents of 'git log'
> >    - find a commit you're interested in
> >$ mkdir $GIT_COMMIT_ID
> >$ ls $GIT_COMMIT_ID/
> >    - get the state just like 'git checkout $GIT_COMMIT_ID'
> >
> >Maybe it would be helpful to be able to create tags inside this .snaps
> >directory. But I would refrain from branches for now (unless there is
> >a clear use-case).
> >
> >Cheers,
> >Niels
> >
> >>Once the user cd's into any one of them, we could do a git checkout of the 
> >>branch.
> >>
> >>That should mimic the behaviour.
> >>
> >>With regards,
> >>Shishir
> >>
> >>----- Original Message -----
> >>From: "Shishir Gowda" <address@hidden>
> >>To: "Niels de Vos" <address@hidden>
> >>Cc: address@hidden
> >>Sent: Friday, March 8, 2013 4:04:42 PM
> >>Subject: Re: [Gluster-devel] file version on glusterfs using libgit
> >>
> >>Hi Niels,
> >>
> >>My inclination too is to load git ontop of posix xlator.
> >>
> >>I was thinking of making previous versions (based on some policy) to be 
> >>treated a new branch.
> >>
> >>We could see how to export these branches as user visible dirs.
> >>
> >>With regards,
> >>Shishir
> >>
> >>----- Original Message -----
> >>From: "Niels de Vos" <address@hidden>
> >>To: "Shishir Gowda" <address@hidden>
> >>Cc: address@hidden
> >>Sent: Friday, March 8, 2013 3:25:57 PM
> >>Subject: Re: [Gluster-devel] file version on glusterfs using libgit
> >>
> >>On Thu, Mar 07, 2013 at 12:54:41AM -0500, Shishir Gowda wrote:
> >>>Hi All,
> >>>
> >>>Was playing around with git on glusterfs volume, to provide was of file 
> >>>version support.
> >>>
> >>>And initial run is encouraging.
> >>>
> >>>A brief overview what was tried:
> >>>
> >>>Approach 1: Glusterfs volume as a git repo
> >>>
> >>>1. created a 2 brick distribute volume
> >>>2. inited a git repo on fuse volume
> >>>3. created files, committed them in git.
> >>>4. Modified files, and committed them again
> >>>5. Did branch check-outs, to simulate versions @ point in time
> >>>6. reset branch heads, and was able access older version of files (after a 
> >>>stash).
> >>>7. Was able to create files/dirs/symlinks/hardlinks
> >>>8. Both NFS/FUSE clients were used.
> >>>
> >>>Approach 2: Glusterfs bricks as git repo's
> >>>
> >>>1. created a 2 brick distribute volume
> >>>2. inited git repo on brick1
> >>>3. inited git repo on brick2
> >>>4. created files, committed the relevant brick's git.
> >>>5. Modified files, and committed them again on brick's git
> >>>6. Did branch check-outs, to simulate versions @ point in time on 
> >>>individual bricks
> >>>7. reset branch heads, and was able access older version of files (after a 
> >>>stash).
> >>>8. Was able to create files/dirs/symlinks/hardlinks
> >>>9. Both NFS/FUSE clients were used.
> >>>
> >>>Buoyed by this, will start prototyping integration of libgit2 as xlator 
> >>>for file version support.
> >>>
> >>>There are 3 approaches to consider:
> >>>
> >>>1. Load git xlator on clients volfiles
> >>>2. Load git xlator on server volfiles
> >>>3. Replace posix interface with git interface.
> >>>
> >>>Please provide feedback, on what would be more desirable.
> >>
> >>Very interesting! Option 2 makes most sense to me, the posix xlator
> >>contains some access checks and such, which you probably should not need
> >>to duplicate.
> >>
> >>Have you thought about making the previous version accessible through
> >>the glusterfs/nfs mount? Other vendors seem to have a .snapshot
> >>directory with previous versions, would something like that be possible?
> >>Users would be able to recover deleted files themselves that way.
> >>
> >>Also, I do not know if git stores xattrs and their changes...
> >>
> >>Cheers,
> >>Niels
> >>
> >>_______________________________________________
> >>Gluster-devel mailing list
> >>address@hidden
> >>https://lists.nongnu.org/mailman/listinfo/gluster-devel
> >
> 
> 
> _______________________________________________
> Gluster-devel mailing list
> address@hidden
> https://lists.nongnu.org/mailman/listinfo/gluster-devel

-- 
Niels de Vos
Sr. Software Maintenance Engineer
Support Engineering Group
Red Hat Global Support Services