Using a shared Guix store (was RE: [Bio-packaging] testing out guix)

[Cook, Malcolm]

Ricardo, Pj, et al,


... Hi - I am resending this message as I omitted Guix-devel <address@hidden> 
at first and would like to cast a broader net....


I am interested in understanding details behind Ricardo's observation: "Guix is 
not designed to be run in a centralised manner. A Guix daemon is supposed to 
run on each system as root and it listens to RPCs from local users only. In an 
environment with multiple clusters and multiple workstations this approach 
requires considerable effort to make it work correctly and securely. Instead we 
opted to run the Guix daemon on a single dedicated server, writing profile data 
and store items onto an NFS share. . The cluster nodes and workstations mount 
this share read-only." (http://elephly.net/posts/2015-04-17-gnu-guix.html)

Can anyone elaborate a little on what are the obstacles to having  `/gnu` 
mounted read-write network wide?

Can it be partially characterized as:

        Multi-process contention over un-coordinated access to the store 
(especially write access necessitated by supporting the `build` action)

If so, might this be mitigated using a variant off of "Using the Offload 
Facility" 
(http://www.gnu.org/software/guix/manual/guix.html#Daemon-Offload-Setup) in 
which builds would still be offloaded (and thus subject to coordination), with 
the elimination of the need for " Missing prerequisites for the build are 
copied over SSH to the target machine, which then proceeds with the build; upon 
success the output(s) of the build are copied back to the initial machine" 
since they would be done through the shared file system?

Do I understand correctly that in your setup, Ricardo, that absolutely no 
`guix` commands are executed on any host other than the "single dedicated 
server".   What about `guix environment p1 p2 p3` when p1 p2 p3 are already 
available in /gnu.  If I understand correctly, in such a case, nothing need be 
written to /gnu... and so should not present any challenge to running guix off 
a shared mount.   Or am I missing an aspect of what is going on?

Lacking the ability to, from any host, dynamically alter the runtime 
environment to include _already installed_  scientific applications seems like 
the most important aspect of guix that would be untolerable to my users were I 
to user guix.    I do hope I am mis-understanding the trade-off you made at 
your site.

I think the second-worst trade-off this compromise takes is that a user cannot 
even alter their own profile unless connected to the master guix host.  For 
instance, a user switching her default emacs  between two already built & 
installed versions only requires editing the profiles/per-user symlink farm; 
couldn't such commands be put behind a per-user semaphore,  allowing 
guix/profiles/per-user to be made available +rw on a shared network mount?

I really appreciate everyone's assistance in helping me understand these 
trade-offs, how guix works, and if there is any new or different thinking about 
strategies for deploying guix.

Thanks,

Malcolm Cook



Hi Malcolm,

> Pleased to e-meet you.  It was your
> http://elephly.net/posts/2015-04-17-gnu-guix.html that got on this 
> path.  I'm so glad I found it.

Oh, great to read that it reached someone who might benefit from it :)

> Great.  I have a trove of recipes (in either bash and a
> brew-derivative) which I intend to move into guix over time, unless 
> I'm beaten to any of them (I'm sure I already have been for some):>

[...]

I just went through the list and found a few that looked familiar.
Below are some comments.

Non-free, won't package:

> gatk
> jimKentUtils

On my list (possibly non-free):

> igv
> igvtools
> meme
> picard (requires more Java toolchain stuff to build from source)
> soapdenovo2
> trimmomatic

Non-free stuff, packaged in my private guix-nonfree repo:

> macs14
> tophat
> viennaRNA

Free stuff, packaged and available through Guix:

> bamtools
> bedtools
> blast+  (WIP, it's really big)
> bowtie2
> cufflinks
> fastqc
> fastx_toolkit
> graphviz
> ncbi_sra_toolkit
> ngsutils
> rsem
> star
> tabix

These look familiar but I'm not sure if I already have them:

> bam2fastq
> bcftools

[...]

> Gonna be a party!

Indeed!  I would be very happy if you decide for Guix in the end.  There isn't 
a lot of bioinfo stuff there yet, but that can be fixed by having more people 
contribute recipes.

Cheers,
Ricardo