Re: Make and Clusters

[Tim Murphy]

Hi,

Firstly I must warn you that I am an interested party since I do consulting
related to this product but I did use it purely as a customer for several
years.

Electric Accelerator is pretty much the pièce de résistance for running GNU
Make and Ninja makefiles on clusters and there is a free evaluation version.

It solves several of your problems:
1)  Making sure the files you need (including tools and source) are
available to every machine in your cluster.  You never have to manage this
or debug whether all your machines have been kept uptodate or provide NFS
mounts everywhere or manage NIS authentication or any of that stuff.
2) It spots problems in your makefiles where tasks are run in the wrong
order - this usually happens because there isn't enough dependency
information in the makefiles.
3) There is a tool called Electric Insight which lets you visualise the
build across your cluster and the analysis tools and reporting on this are
incredible for helping to fix performance or correctness problems.

http://electric-cloud.com/products/electricaccelerator/

Scroll down to the "Use it for free" link.   There are a few limitations to
the free version but they don't generally matter too much if you're not
supporting 100s of users and if you find it very useful you can now get
team licenses priced so that you don't have to be a huge company to afford
it which is a welcome new change.

Regards,

Tim




On 11 October 2017 at 03:15, Martin d'Anjou <address@hidden>
wrote:

> Hi David,
>
> I explored this problem a while ago. I captured my musings in a blog:
> https://ninjaverification.wordpress.com/2008/10/21/
> overriding-gnu-make-shell-variable-for-massive-parallel-make/
>
> In the end the idea is to have make call out to a job scheduler as part
> of the target recipe. This is the method I use and it works really well.
>
> One problem I still have though is that if one of the parallel jobs
> aborts and compromises the whole build, make will wait for all other
> in-flight jobs to finish before itself returning. At the time I had
> manually patched my local copy of GNU Make, but the patch no longer
> applies and I have not gotten back to it. Here is a reference in case
> you find it useful:
> http://savannah.gnu.org/bugs/?41781
>
> Best,
> Martin
>
> On 17-10-10 09:53 AM, David Delahaye wrote:
> > Hello,
> >
> > I wonder how the make command is compatible with clusters. I know the
> > "-j" option, which allows us to run several jobs simultaneously and
> > uses the several cores of a given processor. But if we have several
> > nodes of computation of a cluster, is the command "make -j" able to
> > run over these nodes (transparently for the user)? I know there are
> > specific commands to run jobs on a cluster, I would like to know how
> > "make -j" is compatible with these commands. Thanks in advance for
> > your help.
> >
> > Best regards,
> >
> > David Delahaye.
> >
>
>
> _______________________________________________
> Help-make mailing list
> address@hidden
> https://lists.gnu.org/mailman/listinfo/help-make
>