Re: [Duplicity-talk] Caching for pwd and grp operations

[Lluís Batlle i Rossell]

Name resolution is cached by the glibc (even between multiple processes, with
nscd).

Is it that duplicity does not use the usual name resolution mechanisms?

On Thu, Nov 08, 2012 at 10:58:30AM +0100, address@hidden wrote:
> looks like extending grp and pwd and import these instead would be more 
> structured. obviously every change to tarfile must be documented carefully in 
> the source, so it can be reapplied in future versions.
> 
> before doing it, i'd also like to see a comparision of a backup with and w/o 
> the patch.
> 
> ..ede/duply.net
> 
> On 08.11.2012 03:33, Michael Terry wrote:
> > I would strongly prefer solutions that don't actually modify 
> > duplicity.tarfile, because I'd like to drop it completely for the standard 
> > python library versions once we bump the minimum supported python version.
> > -mt
> > 
> > 
> > On 7 November 2012 20:09, Steve Atwell <address@hidden 
> > <mailto:address@hidden>> wrote:
> > 
> >     I'm hitting some nasty performance problems with duplicity and large
> >     /etc/passwd and /etc/group files.  See
> >     https://bugs.launchpad.net/duplicity/+bug/1013446 for more details.
> >     But basically lookups using the pwd and grp modules can be pretty
> >     slow, and duplicity does a lot of them.  Group lookups are
> >     particularly expensive if you have a group with lots of members
> >     because grp parses the comma-separated member list into a Python list
> >     for every lookup.
> > 
> >     Seems like the best way to fix the problem is to cache pwd and grp
> >     lookups in duplicity.  I'm happy to submit a patch that does this.
> > 
> >     Assuming this is a good idea, I've got a question about
> >     duplicity.tarfile.  It looks like this is a lightly modified copy of
> >     Python 2.7's tarfile module, and caching would need to be added here
> >     (as well as duplicity.path).  Would it be fine to replace calls to
> >     grp.* and pwd.* with cached versions?  (E.g., replace calls to
> >     grp.getgrgid with duplicity.cached_ops.getgrgid or something like
> >     that.)  Or would it be better to avoid modifications by writing some
> >     nastier code that replaces the actual functions in the grp and pwd
> >     modules?
> > 
> >     --
> >     Steve Atwell <address@hidden <mailto:address@hidden>>
> > 
> >     _______________________________________________
> >     Duplicity-talk mailing list
> >     address@hidden <mailto:address@hidden>
> >     https://lists.nongnu.org/mailman/listinfo/duplicity-talk
> > 
> > 
> > 
> > 
> > _______________________________________________
> > Duplicity-talk mailing list
> > address@hidden
> > https://lists.nongnu.org/mailman/listinfo/duplicity-talk
> > 
> 
> _______________________________________________
> Duplicity-talk mailing list
> address@hidden
> https://lists.nongnu.org/mailman/listinfo/duplicity-talk