[GWL] (random) next steps?

[zimoun]

Dear Guixers,
... or at least some of them :-)

Here, I would like to collect some discussions or ideas about the Guix
Workflow Language (GWL) and the next steps of this awesome tool!

For those who do not know.
About workflow language, Wikipedia says:
https://en.wikipedia.org/wiki/Scientific_workflow_system

About GWL, basically the idea is to apply Guix functional principles
to data processing. More details here:
https://www.guixwl.org/
Roel Janssen is the original author of the GWL and now the project is
part of GNU.


Well, I narrow the Ludo's notes from the Paris' meeting and add
commentaries as it was suggested. :-)

** HPC, “workflows”, and all that

  - overview & status
  - supporting “the cloud”
    + service that produces Docker/Singularity images
    + todo: produce layered Docker images like Nix folks
  - workflows
    + snakemake doesn’t handle software deployment
      + or does so through Docker
    + GWL = workflow + deployment
      + add support for Docker
      + add “recency” checks
      + data storage: IRODS?
  - Docker arguments
    + security: handling patient data with untrusted “:latest” images
    + Guix allows for “bisect”


1.
Even if I am not a big fan of WISP because I remember difficulties to
catch "parenthesis closing" issue last time I tried, now I am fan of
what Ricardo showed!
Let push out the wisp-way of GWL... or not. :-)

What are the opinions ?

(pa (ren (the (sis))))

vs

pa:
  ren:
    the:
      sis

With wisp, the workflow description seems close to CWL, which is an argument ;-)
Last time, I check, CWL files seems flat yaml-like files and they lack
programmable extension; as Snakemake provides with Python for example.
And GWL-wisp will have both: easy syntax and programmable stuff,
because it is still lisp under the hood.

https://www.draketo.de/proj/wisp/
https://www.commonwl.org/
https://snakemake.readthedocs.io/en/stable/getting_started/examples.html


2.
One of the lacking feature of GWL is kind-of content addressable store
(CAS) for data. Another workflow language named FunFlow (baked as
Haskell-DSL) implements such kind of ideas. To quote explanations by
Ricardo: "we could copy for the GWL (thus avoiding the need for
recency checks).  The GWL lacks a data provenance story and a CAS
could fit the bill."

https://github.com/tweag/funflow


3.
The project OpenMole about parametric explorations seems implementing
an IPFS way to deal with the data. Not sure what does it mean. :-)

https://openmole.org/

Talking about data, Zenodo is always around. ;-)
https://zenodo.org/


4.
Some GWL scripts are already there.
Could we centralize them to one repo?
Even if they are not clean. I mean something in this flavor:
https://github.com/common-workflow-language/workflows


5.
I recently have discovered the ELisp package `s.el` via the blog post:
http://kitchingroup.cheme.cmu.edu/blog/2018/05/14/f-strings-in-emacs-lisp/
or other said:
https://github.com/alphapapa/elexandria/blob/master/elexandria.el#L224

Does it appear to you a right path to write a "formater" in this
flavour instead of the `string-append` ?
I mean, e.g.,
  `(system ,(string-command "gzip  ${data-inputs}  -c >  ${outputs}"))
instead of e.g.,
  `(system ,(string-append "gzip " data-inputs " -c > " outputs))

It seems more on the flavour of  Snakemake.


6.
The graph of dependencies between the processes/units/rules is written
by hand. What should be the best strategy to capture it ? By files "à
la" Snakemake ? Other ?


7.
Does it appear to you a good idea to provide a command `guix workflow pack` ?
to produce an archive with the binaries or the commit hashes of the
channels, etc.



Last, the webpage [1] points to gwl-devel mailing list which seems broken.
Does the gwl-devel need to be activated and the GWL discussion will
append there or everything stay here to not scattered too much.

[1] https://www.guixwl.org/community



What do you think?
What is do-able? Science-fiction dream?


Thank you.

Have a nice week-end,
simon

--

Then, just pointers/threads (that I am aware of) to remind what the
list already discussed.

https://lists.gnu.org/archive/html/help-guix/2016-02/msg00019.html
https://lists.gnu.org/archive/html/guix-devel/2016-05/msg00380.html
https://lists.gnu.org/archive/html/guix-devel/2016-10/msg00947.html
https://lists.gnu.org/archive/html/guix-devel/2016-10/msg01248.html
https://lists.gnu.org/archive/html/guix-devel/2018-01/msg00371.html
https://lists.gnu.org/archive/html/guix-devel/2018-02/msg00177.html
https://lists.gnu.org/archive/html/help-guix/2018-05/msg00241.html