[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: Parallelization of shell scripts for 'configure' etc.
From: |
madmurphy |
Subject: |
Re: Parallelization of shell scripts for 'configure' etc. |
Date: |
Tue, 14 Jun 2022 04:12:05 +0100 |
Four months ago I had created a couple of m4 macros to allow
parallelization of the configure process:
https://github.com/madmurphy/not-autotools/blob/master/m4/not-parallel-configure.m4
If you look at the source code you will find also a sample usage:
# Start a new unnamed thread....
NC_THREAD_NEW([
AC_MSG_NOTICE([Starting a parallel test...])
# Do something expensive here
sleep 3
# ...
AS_VAR_SET([TEST_RESULT_1], [foo])
AC_MSG_NOTICE([Parallel test 1 has terminated])
])
# Start another thread, named this time....
NC_THREAD_NEW([my_named_thread], [
# Do something expensive here
# ...
sleep 1
# ...
AS_VAR_SET([TEST_RESULT_2], [bar])
AC_MSG_NOTICE([Parallel test 2 has terminated])
])
# This message will appear immediately, without waiting for the tests
# to be completed
AC_MSG_NOTICE([Hello world])
# Do other unrelated things here
# ...
AC_MSG_NOTICE([Wait until all the threads have returned...])
NC_JOIN_THREADS
AC_MSG_NOTICE([The result of the first test was ${TEST_RESULT_1}])
AC_MSG_NOTICE([The result of the second test was ${TEST_RESULT_2}])
AC_OUTPUT
Or you can see the example above in action under
not-autotools/examples/not-parallel-configure
<https://github.com/madmurphy/not-autotools/tree/master/examples/not-parallel-configure>
.
--madmurphy
On Tue, Jun 14, 2022 at 3:00 AM Alex Ameen <alex.ameen.tx@gmail.com> wrote:
> You can try to use the `requires` toposort routine to identify "Strongly
> Connected Sub-Components", which is where I imagine you'll get the
> best results. What you'll need to watch out for is undeclared ordering
> requirements that parallelism would break.
>
> The `m4sh` and `m4sugar` source code is documented in a lot of detail. The
> manuals exclude that type of documentation because it's internal; but you
> could keep yourself occupied for at least a month or two before you ran out
> of topics to explore.
>
> On Mon, Jun 13, 2022, 8:45 PM Dale R. Worley <worley@alum.mit.edu> wrote:
>
> > Paul Eggert <eggert@cs.ucla.edu> writes:
> > > In many Gnu projects, the 'configure' script is the biggest barrier to
> > > building because it takes soooo long to run. Is there some way that we
> > > could improve its performance without completely reengineering it, by
> > > improving Bash so that it can parallelize 'configure' scripts?
> >
> > It seems to me that bash provides the needed tools -- "( ... ) &" lets
> > you run things in parallel. Similarly, if you've got a lot of small
> > tasks with a complex web of dependencies, you can encode that in a
> > "makefile".
> >
> > It seems to me that the heavy work is rebuilding how "configure" scripts
> > are constructed based on which items can be run in parallel. I've never
> > seen any "metadocumentation" that laid out how all that worked.
> >
> > Dale
> >
> >
>
- Parallelization of shell scripts for 'configure' etc., Paul Eggert, 2022/06/13
- Re: Parallelization of shell scripts for 'configure' etc., Michael Orlitzky, 2022/06/14
- Re: Parallelization of shell scripts for 'configure' etc., Nick Bowler, 2022/06/14
- Re: Parallelization of shell scripts for 'configure' etc., Nick Bowler, 2022/06/14
- Re: Parallelization of shell scripts for 'configure' etc., Richard Purdie, 2022/06/14
- Re: Parallelization of shell scripts for 'configure' etc., Demi Marie Obenour, 2022/06/16
- Re: Parallelization of shell scripts for 'configure' etc., Alex Ameen, 2022/06/16
- Re: Parallelization of shell scripts for 'configure' etc., Bob Friesenhahn, 2022/06/17
- Re: Parallelization of shell scripts for 'configure' etc., Paul Eggert, 2022/06/14
- Re: Parallelization of shell scripts for 'configure' etc., Bruno Haible, 2022/06/15
- Re: Parallelization of shell scripts for 'configure' etc., Warren Young, 2022/06/16