Re: [Rd] using R as SHELL in gnu make

[Simon Urbanek]

Malcom,

On Feb 3, 2014, at 12:43 PM, Cook, Malcolm <address@hidden> wrote:

> Simon,
> Reviving an old thread … and moving it to your preferred forum for such 
> discussion ….
> I am re-invigorated to improve my old approach to allowing GNU-make's recipes 
> to be written in R language and evaluated by a running and configured Rserve.
> Having unix local sockets working and the new RS.eval makes this make more 
> reasonable.
> I can now for instance start Rserve using a socket path in the current 
> directory, like this:
> (R_DEFAULT_PACKAGES=MASS;R CMD Rserve --vanilla --silent --RS-socket 
> ${PWD}/.rs-socket --RS-workdir ${PWD})
> Rserv started in daemon mode.
> ...and then query it from the command line like this:
> Rscript --default-packages=RSclient -e 
> 'RS.eval(RS.connect(port=0,host=".rs-socket"),sessionInfo())'
> R version 3.0.2 (2013-09-25)
> Platform: x86_64-unknown-linux-gnu (64-bit)
>  
> locale:
> [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C               LC_TIME=en_US.UTF-8 
>        LC_COLLATE=en_US.UTF-8     LC_MONETARY=en_US.UTF-8    
> LC_MESSAGES=en_US.UTF-8    LC_PAPER=en_US.UTF-8       LC_NAME=C               
>    LC_ADDRESS=C               LC_TELEPHONE=C             
> LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C      
>  
> attached base packages:
> [1] utils     stats     graphics  grDevices base    
>  
> other attached packages:
> [1] MASS_7.3-29
> This works great and as documented.
> However, in order to make the Rserve process evaluate a parsed stream of 
> expressions, as needed to have it be the back-end for GNU-make recipes, I 
> feel I am going through too many contortions. The following example works, 
> but "feels wrong" (tm)
> echo $'print("hello")\nprint("world")\nsetwd("..")\ngetwd()' | Rscript 
> --default-packages=RSclient 
> -e'lines=readLines("stdin");expr.list=parse(text=lines);RS.eval(RS.connect(port=0,host=".rs-socket"),as.call(list(quote(eval),expr.list)),lazy=FALSE)'
> [1] "hello"
> [1] "world"
> [1] "/home/mec/local/mecsrc/mkr"
> Can you advise if there is a simpler, more direct approach than the above.

I'm not sure what would constitute a more direct approach. You could push the 
parsing to the remote side as well if that helps, but it will make the code 
more ugly since you have to use eval(paste(text=...)). 

Note, however, that the output of the print() statements goes out of the Rserve 
process, not the calling process, so if you want that, you probably have to use 
capture.output() as well.


> Also, my recipes will want to generate output files which of course are being 
> placed in the working directory, which is named 'connNNNNN' subordinate to 
> the –RS-workdir. Other than explicitly calling `setwd("..")`, is there 
> perhaps a value in having a flag to RServe which avoids creating connection 
> specific directories?

If you set the workdir setting to '' then Rserve won't manage the working 
directory.

There is, however, one additional issue - the daemon version of Rserve changes 
the working directory to / as part of the daemonization process, so you won't 
get to pwd in that case. One way to deal with that is to use in-R Rserve 
process, e.g. start it via

Rscript -e 'Rserve::run.Rserve(workdir="",socket=".rs-socket")'

then you'll and up in the working directory.

Cheers,
Simon




> Thanks for your consideration, and, thanks for Rserve!
> ~Malcolm Cook
>  
> >-----Original Message-----
> >From: Cook, Malcolm
> >Sent: Thursday, September 22, 2011 9:14 AM
> >To: 'Simon Urbanek'
> >Subject: RE: [Rd] using R as SHELL in gnu make
> >
> >Simon,
> >
> >sorry about the old link.  I am using the latest from sourceforge, but when 
> >I composed the email I took top hit from Google.  My
> >mistake.  Won't happen again.
> >
> >I use Rscript all the time to write stand alone scripts, but don't see the 
> >advantage here.  Can you elaborate what you see it to be?
> >Make does not pass recipes to its SHELL via stdin, rather, as options which 
> >must be preceeded by '-e'.
> >
> >Cheers,
> >
> >~Malcolm
> >
> >
> >-----Original Message-----
> >From: Simon Urbanek [mailto:address@hidden
> >Sent: Thursday, September 22, 2011 9:01 AM
> >To: Cook, Malcolm
> >Subject: Re: [Rd] using R as SHELL in gnu make
> >
> >
> >On Sep 22, 2011, at 9:29 AM, Cook, Malcolm wrote:
> >
> >> OK,
> >>
> >> I now have a working version of setting SHELL=R in a GNU make script, 
> >> allowing make’s recipes to be written in R, that works out
> >most of the complications.
> >>
> >
> >Shouldn't that be SHELL=Rscript?
> >
> >
> >> I plan to clean it up a little more and blog about it soon here: 
> >> http://malcook-gedanken.blogspot.com/ - I’ll follow up here  when I
> >do.
> >>
> >> The approach optionally allows evaluating the recipes using a running 
> >> Rserve (http://rosuda.org/Rserve/), avoiding initialization time
> >and allowing pre-loading of R libraries common to multiple recipes.
> >>
> >
> >Thank you for referencing Rserve, but can you, please, use links that are 
> >not 7 years out of date? I have left Ausgburg (rosuda) in 2004
> >...
> >
> >Thanks,
> >Simon
> >
> >
> >> The approach however does NOT provide any special mechanism to preserve 
> >> state between recipes,
> >> Rather, recipes may create the make rule’s target as a state dump by 
> >> setting with address@hidden  in a call to `save` (or `save.image`,
> >`dump`, as desired).
> >> Other make rules may then call `load('$<')` when the previously saved dump 
> >> is a pre-requisite to the rule.
> >>
> >> I’m still not sure if it is not more of an amusement and am interested in 
> >> all thoughts on this, and welcome any suggestions for
> >example applications that I might include when I go to blog it up...
> >>
> >> Cheers,
> >>
> >> ~Malcolm
> >>
> >>
> >> -----Original Message-----
> >> From: Paul Gilbert [mailto:address@hidden
> >> Sent: Tuesday, September 20, 2011 8:32 AM
> >> To: Cook, Malcolm; 'address@hidden'; 'address@hidden'
> >> Subject: RE: using R as SHELL in gnu make
> >>
> >> Other than the RServe part, I do this all the time. It works well. Perhaps 
> >> we can put together some notes off-line and then bring it
> >back to the list.
> >>
> >> Paul
> >>
> >>> -----Original Message-----
> >>> From: address@hidden [mailto:address@hidden
> >>> project.org] On Behalf Of Cook, Malcolm
> >>> Sent: September 19, 2011 6:35 PM
> >>> To: 'address@hidden'; 'address@hidden'
> >>> Subject: [Rd] using R as SHELL in gnu make
> >>>
> >>> I am intrigued by the possibility of using R as the SHELL in a (Gnu)
> >>> makefile (instead of /bin/sh).  (c.f.
> >>> http://www.gnu.org/software/make/manual/make.html#Choosing-the-Shell)
> >>>
> >>> Well, rather, I would like the makefile's SHELL to be a command which
> >>> communicated with an R process.
> >>>
> >>> The makefile targets/prerequistes would, as always, be OS files, which
> >>> would be written/read using standard R file IO.
> >>>
> >>> The makefile's "recipe"s would be written in R (instead of the usual
> >>> shell).
> >>>
> >>> The R process would be able to be initiated by `load`ing one or more R
> >>> datasets, libraries or entire images.
> >>>
> >>> The R process would be able to accumulate state as the makefile
> >>> progressed.  The recipe's would be able to refer to that state,
> >>> allowing conditional execution.
> >>>
> >>> The R process would optionally be saved as an image of on job
> >>> termination/completion.
> >>>
> >>> The R process might be managed using the RServe package, and would need
> >>> to be initiated once only, when the makefile was first invoked.
> >>>
> >>> I would appreciate learning if anyone had any success, informative
> >>> failures, or other lore that may help in (or dissuade me from)
> >>> embarking on attempt this.
> >>>
> >>> Thanks,
> >>>
> >>> Malcolm Cook
> >>> Stowers Institute for Medical Research
> >>>
> >>> ______________________________________________
> >>> address@hidden mailing list
> >>> https://stat.ethz.ch/mailman/listinfo/r-devel
> >> ====================================================================================
> >>
> >> La version française suit le texte anglais.
> >>
> >> ------------------------------------------------------------------------------------
> >>
> >> This email may contain privileged and/or confidential information, and the 
> >> Bank of
> >> Canada does not waive any related rights. Any distribution, use, or 
> >> copying of this
> >> email or the information it contains by other than the intended recipient 
> >> is
> >> unauthorized. If you received this email in error please delete it 
> >> immediately from
> >> your system and notify the sender promptly by email that you have done so.
> >>
> >> ------------------------------------------------------------------------------------
> >>
> >> Le présent courriel peut contenir de l'information privilégiée ou 
> >> confidentielle.
> >> La Banque du Canada ne renonce pas aux droits qui s'y rapportent. Toute 
> >> diffusion,
> >> utilisation ou copie de ce courriel ou des renseignements qu'il contient 
> >> par une
> >> personne autre que le ou les destinataires désignés est interdite. Si vous 
> >> recevez
> >> ce courriel par erreur, veuillez le supprimer immédiatement et envoyer 
> >> sans délai à
> >> l'expéditeur un message électronique pour l'aviser que vous avez éliminé 
> >> de votre
> >> ordinateur toute copie du courriel reçu.
> >> ______________________________________________
> >> address@hidden mailing list
> >> https://stat.ethz.ch/mailman/listinfo/r-devel