Re: [Fab-user] rsync_project

[Christian Vest Hansen]

On Fri, Apr 3, 2009 at 11:01 PM, Paul Baumgart <address@hidden> wrote:
> Yeah, I took the change you pushed and added to that support for more than
> one exclude option, because otherwise the exclude parameter isn't nearly as
> useful as it could be.

Ok, applied. Thanks!

>
> Sorry if I wasn't clear.
>
> Paul
>
> On Fri, Apr 3, 2009 at 1:41 PM, Christian Vest Hansen <address@hidden>
> wrote:
>>
>> I think you misunderstood me!
>>
>> I had already pushed a change. Or maybe you're building on top of that?
>>
>> On Fri, Apr 3, 2009 at 10:34 PM, Paul Baumgart <address@hidden> wrote:
>> > Ok, fair enough. :-)
>> >
>> > How's this then:
>> >
>> > @operation
>> > def rsync_project(remotedir, exclude=[], delete=False, extra_opts='',
>> > **kwargs):
>> >     """
>> >     Uploads the current project directory using rsync.
>> >     By using rsync, only changes since the last upload are actually sent
>> > over
>> >     the wire, rather than the whole directory like when using
>> > upload_project.
>> >
>> >     Requires the rsync command-line utility to be available both on the
>> > local
>> >     and the remote machine.
>> >
>> >     Parameters are:
>> >         remotedir:          the directory on the remote machine to which
>> > to
>> >                             rsync the current project. The project
>> > directory
>> >                             becomes a subdirectory of the remotedir.
>> >         exclude (optional): values passed to rsync's --exclude option.
>> >                             If the parameter object is iterable, (that
>> > is,
>> > it
>> >                             defines an __iter__ method), each of its
>> > elements is
>> >                             passed to a separate --exclude option.
>> > Otherwise,
>> >                             the object is passed as a string to a single
>> >                             --exclude option.
>> >                             See the rsync manpage for details on
>> > specifying
>> >                             filter rule arguments for --exclude.
>> >         delete (optional):  True or False, whether to delete remote
>> > files
>> > that
>> >                             don't exist locally. Defaults to False.
>> >         extra_opts (optional): Additional command-line options to set
>> > for
>> > rsync.
>> >
>> >     The rsync command is built from the options as follows:
>> >         rsync [--delete] [--exclude exclude[0][, --exclude[1][, ...]]]
>> > \\
>> >             -pthrvz [extra_opts] ../<project dir>
>> > <fab_user>@<host>:<remotedir>
>> >     """
>> >     username = ENV.get('fab_user')
>> >
>> >     if not hasattr(exclude, '__iter__'):
>> >         exclude = [exclude]
>> >
>> >     exclude_opts = ' --exclude "%s"' * len(exclude)
>> >     exclusions = tuple([str(s).replace('"', '\\\\"') for s in exclude])
>> >
>> >     options_map = {
>> >         "delete"  : '--delete' if delete else '',
>> >         "exclude" : exclude_opts % exclusions,
>> >         "extra"   : extra_opts
>> >     }
>> >     options = "%(delete)s%(exclude)s -pthrvz %(extra)s" % options_map
>> >     cwd = '../' + os.getcwd().split(os.sep)[-1]
>> >     userhost = "$(fab_user)@$(fab_host)"
>> >     rdir = _lazy_format(remotedir, ENV)
>> >
>> >     cmd = "rsync %s %s %s:%s" % (options, cwd, userhost, rdir)
>> >     local_per_host(cmd, **kwargs)
>> >
>> >
>> > Besides reducing the length of lines, I also made it more pythonic by
>> > using
>> > duck typing to determine how to handle the exclude parameter. So now you
>> > can
>> > pass integers, complex numbers, or anything else with a __str__ method
>> > and
>> > it will work.
>> >
>> > I also cleaned up the documentation a bit and made some parts of it more
>> > precise.
>> >
>> > Paul
>> >
>> >
>> >
>> >
>> >
>> > On Fri, Apr 3, 2009 at 10:50 AM, Christian Vest Hansen
>> > <address@hidden> wrote:
>> >>
>> >> Good idea. But as you may have guessed, I'm not fond of long lines :)
>> >>
>> >> On Fri, Apr 3, 2009 at 5:21 AM, Paul Baumgart <address@hidden> wrote:
>> >> > Hi Christian,
>> >> >
>> >> > Thanks, your version looks a lot cleaner.
>> >> >
>> >> > I would like to add one thing, because I think it makes the exclude
>> >> > parameter much more useful:
>> >> >
>> >> > @operation
>> >> > def rsync_project(remotepath, exclude=False, delete=False,
>> >> > extra_opts='',
>> >> > **kwargs):
>> >> >    """
>> >> >    Uploads the current project directory using rsync.
>> >> >    By using rsync, only changes since last upload are actually sent
>> >> > over
>> >> >    the wire, rather than the whole directory like using
>> >> > upload_project.
>> >> >
>> >> >    Requires the rsync command-line utility to be available both on
>> >> > the
>> >> > local
>> >> >    and the remote machine.
>> >> >
>> >> >    Parameters are:
>> >> >        remotepath:         the path on the remote machine to which to
>> >> > rsync
>> >> > the
>> >> >                            current project
>> >> >        exclude (optional): values passed to rsync's --exclude option.
>> >> >                            If the parameter is iterable, each of its
>> >> > elements
>> >> >                            is passed to a separate --exclude
>> >> > argument.
>> >> > If
>> >> > the
>> >> >                            parameter is a string, that string is the
>> >> > value
>> >> >                            passed to the single --exclude argument.
>> >> >                            See the rsync manpage for details on using
>> >> > the
>> >> >                            --exclude option.
>> >> >        delete (optional):  True or False, whether to delete remote
>> >> > files
>> >> > that
>> >> >                            don't exist locally.
>> >> >        extra_opts (optional): Additional command-line options to set
>> >> > for
>> >> > rsync.
>> >> >
>> >> >    The rsync command is built from the options as follows:
>> >> >        rsync [--delete] [--exclude exclude] -pthrvz [extra_opts] \\
>> >> >            <project dir> <fab_user>@<host>:<remotepath>
>> >> >    """
>> >> >    username = ENV.get('fab_user')
>> >> >
>> >> >    if isinstance(exclude, basestring):
>> >> >        exclude = [exclude]
>> >> >
>> >> >    options_map = {
>> >> >        "delete" : '--delete' if delete else '',
>> >> >        "exclude" : exclude and ' '.join(['--exclude "%s"' %
>> >> > opt.replace('"',
>> >> > '\\\\"') for opt in exclude]) or '',
>> >> >        "extra" : extra_opts
>> >> >    }
>> >> >    options = "%(delete)s %(exclude)s -pthrvz %(extra)s" % options_map
>> >> >    cwd = '../' + os.getcwd().split(os.sep)[-1]
>> >> >    userhost = "$(fab_user)@$(fab_host)"
>> >> >    rpath = _lazy_format(remotepath, ENV)
>> >> >
>> >> >    cmd = "rsync %s %s %s:%s" % (options, cwd, userhost, rpath)
>> >> >    local_per_host(cmd, **kwargs)
>> >> >
>> >> > So, instead of taking just a string, it can take either a string or a
>> >> > list,
>> >> > and multiple --exclude options are generated from a list. A string
>> >> > parameter
>> >> > is treated just as it is currently.
>> >> >
>> >> > Paul
>> >> >
>> >> > On Thu, Apr 2, 2009 at 12:38 PM, Christian Vest Hansen
>> >> > <address@hidden> wrote:
>> >> >> Ok. I implemented this change, but I modified it a little bit.
>> >> >>
>> >> >> On Thu, Apr 2, 2009 at 9:17 PM, Christian Vest Hansen
>> >> >> <address@hidden> wrote:
>> >> >>> On Sun, Mar 22, 2009 at 12:51 AM, Paul Baumgart <address@hidden>
>> >> >>> wrote:
>> >> >>>> A different topic this time:
>> >> >>>>
>> >> >>>> The current upload_project() function is a neat feature, but I
>> >> >>>> have
>> >> >>>> two issues with it:
>> >> >>>>
>> >> >>>> 1) It doesn't allow me to exclude certain files/directories (like
>> >> >>>> .git* for example)
>> >> >>>> 2) It uploads the entire project every time, which can be
>> >> >>>> irritating
>> >> >>>> if the change is small but the project is large.
>> >> >>>>
>> >> >>>> So, I made this function, which uses rsync to only upload the
>> >> >>>> differences between the local and remote copies.
>> >> >>>>
>> >> >>>> I would like to get feedback on it, primarily as to whether this
>> >> >>>> would
>> >> >>>> be useful in the mainline Fabric code, and if not, if there is any
>> >> >>>> way
>> >> >>>> to make it be useful.
>> >> >>>>
>> >> >>>> @operation
>> >> >>>> @connects
>> >> >>>> def rsync_project(host, client, env, remotepath, exclude=False,
>> >> >>>> delete=False, extra_opts='', **kwargs):
>> >> >>>>    """
>> >> >>>>    Uploads the current project directory using rsync, so only
>> >> >>>> changes
>> >> >>>> are
>> >> >>>>    uploaded rather than the whole directory like using
>> >> >>>> upload_project.
>> >> >>>>
>> >> >>>>    Requires the rsync command-line utility to be available both on
>> >> >>>> the
>> >> >>>> local
>> >> >>>>    and the remote machine.
>> >> >>>>
>> >> >>>>    Parameters are:
>> >> >>>>        remotepath:         the path on the remote machine to which
>> >> >>>> to
>> >> >>>> rsync the
>> >> >>>>                            current project
>> >> >>>>        exclude (optional): the string passed to rsync's --exclude
>> >> >>>> option.
>> >> >>>>                            See the rsync manpage for details.
>> >> >>>>        delete (optional):  True or False, whether to delete remote
>> >> >>>> files
>> >> >>>> that
>> >> >>>>                            don't exist locally.
>> >> >>>>        extra_opts (optional): Additional command-line options to
>> >> >>>> set
>> >> >>>> for
>> >> >>>> rsync.
>> >> >>>>
>> >> >>>>    The rsync command is built from the options as follows:
>> >> >>>>        rsync [--delete] [--exclude exclude] -pthrvz [extra_opts]
>> >> >>>> <project dir> <fab_user>@<host>:<remotepath>
>> >> >>>>    """
>> >> >>>>
>> >> >>>>    remotepath = _lazy_format(remotepath, env)
>> >> >>>>
>> >> >>>>    username = ENV.get('fab_user')
>> >> >>>>    username = username + '@' if username else username
>> >> >>>>
>> >> >>>>    cwd_name = '../' + os.getcwd().split(os.sep)[-1]
>> >> >>>
>> >> >>> This line right here. Why do you go through all this trouble to
>> >> >>> build
>> >> >>> a relative CWD?
>> >> >>
>> >> >> Elementary, dear Watson. To get a prettier output when the assembled
>> >> >> command is printed to the console.
>> >> >>
>> >> >>>
>> >> >>>
>> >> >>>>
>> >> >>>>    delete_opt = '--delete' if delete else ''
>> >> >>>>
>> >> >>>>    exclude_opt = '--exclude' if exclude else ''
>> >> >>>>    exclude = '"' + exclude.strip('"') + '"' if exclude else ''
>> >> >>>>
>> >> >>>>    return local('rsync %(delete_opt)s %(exclude_opt)s %(exclude)s
>> >> >>>> -pthrvz %(extra_opts)s %(cwd_name)s
>> >> >>>> %(username)s%(host)s:%(remotepath)s'
>> >> >>>>        % locals(), **kwargs) == 0
>> >> >>>>
>> >> >>>>
>> >> >>>> Note that it requires adding a line to the bottom of local():
>> >> >>>>
>> >> >>>> return retcode
>> >> >>>>
>> >> >>>> Regards,
>> >> >>>> Paul
>> >> >>>>
>> >> >>>>
>> >> >>>> _______________________________________________
>> >> >>>> Fab-user mailing list
>> >> >>>> address@hidden
>> >> >>>> http://lists.nongnu.org/mailman/listinfo/fab-user
>> >> >>>>
>> >> >>>
>> >> >>>
>> >> >>>
>> >> >>> --
>> >> >>> Venlig hilsen / Kind regards,
>> >> >>> Christian Vest Hansen.
>> >> >>>
>> >> >>
>> >> >>
>> >> >>
>> >> >> --
>> >> >> Venlig hilsen / Kind regards,
>> >> >> Christian Vest Hansen.
>> >> >>
>> >> >
>> >> >
>> >>
>> >>
>> >>
>> >> --
>> >> Venlig hilsen / Kind regards,
>> >> Christian Vest Hansen.
>> >
>> >
>>
>>
>>
>> --
>> Venlig hilsen / Kind regards,
>> Christian Vest Hansen.
>
>



-- 
Venlig hilsen / Kind regards,
Christian Vest Hansen.