Re: [Fab-user] Wait for parallel task to complete

[Morgan Goose]

Hmm, I think perhaps I may have been wrong that it takes all env
params. Will look and see if there is a bug or feature request for
that. But decorating @parallel should still be honored.

-goose

On Thu, May 10, 2012 at 11:16 AM, anatoly techtonik <address@hidden> wrote:
> Hi Morgan,
>
> Thanks for the advice, but keyword arguments from the environment
> don't work with execute(). The code below gives a traceback:
>
> @task
> def test():
>  pass
> @task
> def deploy():
>   execute(test, parallel=True)
>
> Traceback (most recent call last):
>  File "/usr/lib/python2.7/site-packages/fabric/main.py", line 712, in main
>    *args, **kwargs
>  File "/usr/lib/python2.7/site-packages/fabric/tasks.py", line 327, in execute
>    results['<local-only>'] = task.run(*args, **new_kwargs)
>  File "/usr/lib/python2.7/site-packages/fabric/tasks.py", line 112, in run
>    return self.wrapped(*args, **kwargs)
>  File "/prdir/fabfile.py", line 95, in deploy
>    execute(test, parallel=True)
>  File "/usr/lib/python2.7/site-packages/fabric/tasks.py", line 327, in execute
>    results['<local-only>'] = task.run(*args, **new_kwargs)
>  File "/usr/lib/python2.7/site-packages/fabric/tasks.py", line 112, in run
>    return self.wrapped(*args, **kwargs)
> TypeError: test() takes no arguments (1 given)
>
>
> On Wed, May 2, 2012 at 2:02 AM, Morgan Goose <address@hidden> wrote:
>> Notice the section in
>> http://docs.fabfile.org/en/1.4.1/api/core/tasks.html#fabric.tasks.execute
>> where it mentions that "Any other arguments or keyword arguments will
>> be passed verbatim into task when it is called, so execute(mytask,
>> 'arg1', kwarg1='value') will (once per host) invoke mytask('arg1',
>> kwarg1='value')."
>>
>> It is refering, albeit not clearly, to the fact that any keyword that
>> is given will be used as if it were a member of the env dict. So you
>> can use any of these env var names to specify things, eg:
>> parallel=True; 
>> http://docs.fabfile.org/en/1.4.1/usage/env.html#full-list-of-env-vars
>>
>> Also I fumbled the name of the decorator. It's serial:
>> http://docs.fabfile.org/en/1.4.1/api/core/decorators.html#fabric.decorators.serial
>>
>> You can also specify custom args to fab tasks:
>> http://docs.fabfile.org/en/1.4.1/usage/fab.html#per-task-arguments
>>
>> And with that you could make the deploy task take a role name, or list
>> of hosts to then use as the lists, local and remote, to use in your
>> script.
>>
>> -goose
>>
>> On Tue, May 1, 2012 at 12:01 PM, anatoly techtonik <address@hidden> wrote:
>>> Hi Morgan,
>>>
>>> I've tried your example, which is:
>>>
>>>  local = ['a','b']
>>>  remote = ['c','d']
>>>
>>>  def test:pass
>>>  def update:pass
>>>
>>> address@hidden
>>> address@hidden
>>>  def deploy():
>>>     execute(test, hosts=local+remote)
>>>     execute(update, hosts=local+remote)
>>>
>>> At first it didn't run at all with Fabric 1.4.1, unable to call
>>> @sequential decorator:
>>>
>>>  Traceback (most recent call last):
>>>    ...
>>>    File "/checkout83/fabfile.py", line 20, in <module>
>>>     address@hidden
>>>  NameError: name 'sequential' is not defined
>>>
>>> I commented the @sequential, after which `fab deploy` run sequentially:
>>>  local test
>>>  remote test
>>>  local update
>>>  remote update
>>>
>>> That's much better, but it still impossible to specify hosts from
>>> command line. `fab -H local deploy` still runs additional command on
>>> the remote, and `fab -H local,remote deploy` run everything twice on
>>> both hosts.
>>>
>>> I couldn't find parallel among params for execute -
>>> http://docs.fabfile.org/en/1.4.1/api/core/tasks.html#fabric.tasks.execute
>>> so I've just decorated functions:
>>>
>>>  local = ['a','b']
>>>  remote = ['c','d']
>>>
>>> address@hidden
>>> address@hidden
>>>  def test:pass
>>> address@hidden
>>> address@hidden
>>>  def update:pass
>>>
>>> address@hidden
>>>  def deploy():
>>>     execute(test, hosts=local+remote)
>>>     execute(update, hosts=local+remote)
>>>
>>> I must admit that although not ideally, but this works. I just need to
>>> make sure that deploy is always called with empty host list and I need
>>> to find a way to specify hosts from command line for the subtasks.
>>> --
>>> anatoly t.
>>>
>>>
>>> On Fri, Apr 27, 2012 at 11:48 PM, Morgan Goose <address@hidden> wrote:
>>>> The example I gave you and how to run it did all of those things, sans
>>>> the parallel. Execute has parallel as a param. Use that in your master
>>>> task's executes, and you'll have everything. If that still isn't what
>>>> you're looking for, we're going to need more information. Eg, like you
>>>> did before where you said how you're seeing it run, and how you'd
>>>> instead like it to run, complete with how you ran it with fab, and how
>>>> the fabfile looks.
>>>>
>>>> -goose
>>>>
>>>> On Thu, Apr 26, 2012 at 10:32 PM, anatoly techtonik <address@hidden> wrote:
>>>>> Thanks for the explanation. I read the tutorial. Still I see I can't
>>>>> construct the system that will satisfy the following usability
>>>>> requirements:
>>>>>
>>>>>  1. Define (or override) hosts from command line
>>>>>  2. Execute each scheduled task in parallel
>>>>>  3. Wait until previous task completes successfully on all servers
>>>>> before moving to the next one
>>>>>  4. Ability to run each task separately or use master task as a helper 
>>>>> command
>>>>>
>>>>> Command line hosts (1) is needed to test new nodes and control
>>>>> deployment from 3rd party application.
>>>>> Parallel execution (2) and (3) is critical to minimize servers downtime.
>>>>> Master task (4) is also highly desired, because full deployment process
>>>>> contains a lot of steps.
>>>>>
>>>>> It seems that this could be real with the @mastertask decorator that
>>>>> would make task run without servers at all. With it the following
>>>>> could work as expected.
>>>>>
>>>>> @task
>>>>> @parallel
>>>>> def test:pass
>>>>>
>>>>> @task
>>>>> @parallel
>>>>> def update:pass
>>>>>
>>>>> @mastertask
>>>>> def deploy():
>>>>>   execute(test)
>>>>>   execute(update)
>>>>>
>>>>> --
>>>>> anatoly t.
>>>>>
>>>>>
>>>>> On Fri, Apr 27, 2012 at 1:28 AM, Morgan Goose <address@hidden> wrote:
>>>>>> I just read that last line, where you said you wanted to run them with
>>>>>> hosts defined at runtime. You can either use functions to generate the
>>>>>> hosts lists or with said example i gave change it up to get rid of the
>>>>>> deploy task, and then run the fabfile like:
>>>>>>    $ fab -H local,remote test update
>>>>>>
>>>>>> As that will be sequential and honor the host list as well. It will
>>>>>> also run the test on all hosts before moving to the next task listed,
>>>>>> update.
>>>>>>
>>>>>> -goose
>>>>>>
>>>>>> On Thu, Apr 26, 2012 at 3:25 PM, Morgan Goose <address@hidden> wrote:
>>>>>>> First have you read the tutorial? This is the section that will get
>>>>>>> you in on host list defining, and point you the the more detailed
>>>>>>> information that I link to below it:
>>>>>>> http://docs.fabfile.org/en/1.4.1/tutorial.html#defining-connections-beforehand
>>>>>>> http://docs.fabfile.org/en/1.4.1/usage/execution.html#host-lists
>>>>>>>
>>>>>>> As to what you want to do. Each task will loop over it's list of hosts
>>>>>>> to run on. So top down it's TaskA over all HostListA's hosts. Then if
>>>>>>> no errors move to the other task with the same or unique host list
>>>>>>> (defined either globally or per task).
>>>>>>>
>>>>>>> From what you're wanting to do you will construct the fabfile like this:
>>>>>>>
>>>>>>> local = ['a','b']
>>>>>>> remote = ['c','d']
>>>>>>>
>>>>>>> def test:pass
>>>>>>> def update:pass
>>>>>>>
>>>>>>> @task
>>>>>>> @sequential
>>>>>>> def deploy():
>>>>>>>    execute(test, hosts=local+remote)
>>>>>>>    execute(update, hosts=local+remote)
>>>>>>>
>>>>>>>
>>>>>>> Then all hosts will have the test task run on them before moving to
>>>>>>> looping over the two hosts lists and running the update. You then call
>>>>>>> it simply with:
>>>>>>>
>>>>>>> $ fab deploy
>>>>>>>
>>>>>>> And you never run the test and update tasks themselves. Nor do you
>>>>>>> define anything to be parallel. As parallel runs on the task would
>>>>>>> test on both hosts in parallel, and then deploy to both hosts in
>>>>>>> parallel. This was, running sequential and explicitly defined to do
>>>>>>> so, will fail fast on the test hosts if one dies. Because the host
>>>>>>> list's order is honored.
>>>>>>>
>>>>>>> -goose
>>>>>>>
>>>>>>> On Wed, Apr 25, 2012 at 1:13 PM, anatoly techtonik <address@hidden> 
>>>>>>> wrote:
>>>>>>>> On Sat, Apr 21, 2012 at 7:51 PM, Jeff Forcier <address@hidden> wrote:
>>>>>>>>> On Fri, Apr 20, 2012 at 5:31 AM, anatoly techtonik <address@hidden> 
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Is it possible in fabric to wait until a subtask completes on all
>>>>>>>>>> servers successfully before moving to next step?
>>>>>>>>>
>>>>>>>>> You need to use execute() to treat subroutines as if they were full
>>>>>>>>> fledged tasks. execute() is the machinery that says "take this
>>>>>>>>> function and run it once per host in this list of hosts." Right now
>>>>>>>>> that machinery is just applying implicitly to your deploy() task and
>>>>>>>>> you're probably using env.hosts to set your host list.
>>>>>>>>>
>>>>>>>>> Remove env.hosts and maybe make that host list a role in env.roledefs.
>>>>>>>>> Then you can do this:
>>>>>>>>>
>>>>>>>>>    env.roledefs = {'myrole': ['a', 'b', 'c']}
>>>>>>>>>
>>>>>>>>>    def test(): ...
>>>>>>>>>    def update(): ...
>>>>>>>>>
>>>>>>>>>    def deploy():
>>>>>>>>>        execute(test, role='myrole')
>>>>>>>>>        execute(update, role='myrole')
>>>>>>>>>
>>>>>>>>> That should have the effect you want: "fab deploy" => first test()
>>>>>>>>> runs once per host, and when it's all done, update() will run once per
>>>>>>>>> host. deploy() itself will end up running only one time total -- it's
>>>>>>>>> just a "meta" task now.
>>>>>>>>
>>>>>>>> It took time to realize the that execute() iterates over the global
>>>>>>>> list of hosts. I expected that the following two to be equivalent, but
>>>>>>>> they were not:
>>>>>>>>
>>>>>>>>  1. fab -H local,remote test update
>>>>>>>>  2. fab -H local,remote deploy
>>>>>>>>
>>>>>>>> I used the script without roles:
>>>>>>>>
>>>>>>>>  def test(): ...
>>>>>>>>  def update(): ...
>>>>>>>>  def deploy():
>>>>>>>>      execute(test)
>>>>>>>>      execute(update)
>>>>>>>>
>>>>>>>> 1st execution variant is fully synchronous (i.e. next task doesn't
>>>>>>>> start until previous finishes) and gave the sequence:
>>>>>>>>
>>>>>>>>  local test
>>>>>>>>  local update
>>>>>>>>  remote test
>>>>>>>>  remove update
>>>>>>>>
>>>>>>>> but the 2nd variant with subtasks was confusing (I indented to see
>>>>>>>> what's going on):
>>>>>>>>
>>>>>>>>  local deploy
>>>>>>>>    local test
>>>>>>>>    remote test
>>>>>>>>    local update
>>>>>>>>    remote update
>>>>>>>>  remote deploy
>>>>>>>>    local test
>>>>>>>>    remote test
>>>>>>>>    local update
>>>>>>>>    remote update
>>>>>>>>
>>>>>>>> I found fabric pretty counter-intuitive in this case. I tried to fix
>>>>>>>> that without roles by explicitly passing current host:
>>>>>>>>
>>>>>>>>  def test(): ...
>>>>>>>>  def update(): ...
>>>>>>>>  def deploy():
>>>>>>>>      execute(test, host=env.host_string)
>>>>>>>>      execute(update, host=env.host_string)
>>>>>>>>
>>>>>>>> This gives:
>>>>>>>>  local deploy
>>>>>>>>    local test
>>>>>>>>    local update
>>>>>>>>  remote deploy
>>>>>>>>    remote test
>>>>>>>>    remove update
>>>>>>>>
>>>>>>>> Still not the desired behavior. The desired is:
>>>>>>>>  deploy
>>>>>>>>    local test
>>>>>>>>    remote test
>>>>>>>>    wait
>>>>>>>>    local update
>>>>>>>>    remote update
>>>>>>>>
>>>>>>>> I've tried using @parallel decorator for deploy task and it seemed to
>>>>>>>> work fine at first.
>>>>>>>>  local deploy
>>>>>>>>  remote deploy
>>>>>>>>  remote test
>>>>>>>>  local test
>>>>>>>>  local update
>>>>>>>>  remote update
>>>>>>>>
>>>>>>>> But the test step didn't not synchronize - local update executed while
>>>>>>>> remote test was still running. It looks like the roles is the only
>>>>>>>> chance, but I'd like to avoid hardcoding server names at all costs. Is
>>>>>>>> it possible?
>>>>>>>>
>>>>>>>> --
>>>>>>>> anatoly t.
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> Fab-user mailing list
>>>>>>>> address@hidden
>>>>>>>> https://lists.nongnu.org/mailman/listinfo/fab-user