Re: [Duplicity-talk] erroneous incrementals? verify failing? everything else seems okay?

[Kenneth Loafman]

I just ran a 53-volume backup of .jpg files to S3 and verified.  No
problems found.  I'm on 8.04.1 with Boto 1.61, and duplicity 0.5.12.
That should be a close match to your system.

What backup command line did you use?

Are you 32-bit or 64-bit?

...Ken

Josh wrote:
> Its on a Linux system.
> 
> The data is mounted via the root partition (within "/" path)
> 
> ext3 format
> 
> -------------
> This communication (with attachments) is covered by the U.S.C. Sec.
> 2510-252, is confidential, and intended only for the use of the
> individual or entity named above. If the reader of this message is not
> the intended recipient, you are hereby notified that any dissemination,
> distribution, or copying of this communication is strictly prohibited.
> If you receive this communication in error, notify us immediately,
> return the document(s), and destroy all copies of the original message.
> 
> 
> On Tue, Mar 24, 2009 at 12:00 PM, Kenneth Loafman <address@hidden
> <mailto:address@hidden>> wrote:
> 
>     Don't bother finishing... I'm going to have to test some against S3
>     myself.  I just don't see what's going on.
> 
>     What about the other questions?
> 
>     Is the data located on a Windows system or a Linux system?
> 
>     Is it mounted, if so how?
> 
>     I'm guessing its on the Linux system directly from the path, but I just
>     want to verify that.  Do you know what kind of filesystem its on?
> 
>     ...Ken
> 
>     Josh wrote:
>     > here is the scrip I am using for the verify;
>     >
>     > -------------
>     > #!/bin/bash
>     >
>     > # Script to verify data on Amazon S3
>     >
>     > export AWS_ACCESS_KEY_ID=<key>
>     > export AWS_SECRET_ACCESS_KEY=<key>
>     > export PASSPHRASE=<phrase>
>     > GPG_KEY=<key>
>     >
>     > duplicity verify -v9 s3+http://leveleight.gallery
>     > /var/www/web2/g2data/albums/
>     >
>     > # Reset the ENV variables. Don't need them sitting around
>     > export AWS_ACCESS_KEY_ID=
>     > export AWS_SECRET_ACCESS_KEY=
>     > export PASSPHRASE=
>     > -------------
>     >
>     > here are some lines from when I run this script, that may/not be of
>     > use... I still haven't let it run completely as its taking a so
>     long...
>     > but here is what I have
>     >
>     > Main action: verify
>     > Listed
>     >
>     
> s3+http://leveleight.gallery/duplicity-full-signatures.2009-03-23T13:27:46-05:00.sigtar.gpg
>     > Listed
>     >
>     
> s3+http://leveleight.gallery/duplicity-full.2009-03-23T13:27:46-05:00.manifest.gpg
>     > Listed
>     >
>     
> s3+http://leveleight.gallery/duplicity-full.2009-03-23T13:27:46-05:00.vol1.difftar.gpg
>     > Listed
>     >
>     
> s3+http://leveleight.gallery/duplicity-full.2009-03-23T13:27:46-05:00.vol10.difftar.gpg
>     > Listed
>     >
>     
> s3+http://leveleight.gallery/duplicity-full.2009-03-23T13:27:46-05:00.vol100.difftar.gpg
>     > Listed
>     >
>     
> s3+http://leveleight.gallery/duplicity-full.2009-03-23T13:27:46-05:00.vol1000.difftar.gpg
>     > Listed
>     >
>     
> s3+http://leveleight.gallery/duplicity-full.2009-03-23T13:27:46-05:00.vol1001.difftar.gpg
>     > Listed
>     >
>     
> s3+http://leveleight.gallery/duplicity-full.2009-03-23T13:27:46-05:00.vol1002.difftar.gpg
>     > Listed
>     >
>     
> s3+http://leveleight.gallery/duplicity-full.2009-03-23T13:27:46-05:00.vol1003.difftar.gpg
>     > Listed
>     >
>     
> s3+http://leveleight.gallery/duplicity-full.2009-03-23T13:27:46-05:00.vol1004.difftar.gpg
>     > Listed
>     >
>     
> s3+http://leveleight.gallery/duplicity-full.2009-03-23T13:27:46-05:00.vol1005.difftar.gpg
>     > Listed
>     >
>     
> s3+http://leveleight.gallery/duplicity-full.2009-03-23T13:27:46-05:00.vol1006.difftar.gpg
>     > Listed
>     >
>     
> s3+http://leveleight.gallery/duplicity-full.2009-03-23T13:27:46-05:00.vol1007.difftar.gpg
>     > Listed
>     >
>     
> s3+http://leveleight.gallery/duplicity-full.2009-03-23T13:27:46-05:00.vol1008.difftar.gpg
>     > Listed
>     >
>     
> s3+http://leveleight.gallery/duplicity-full.2009-03-23T13:27:46-05:00.vol1009.difftar.gpg
>     > Listed
>     >
>     
> s3+http://leveleight.gallery/duplicity-full.2009-03-23T13:27:46-05:00.vol101.difftar.gpg
>     > Listed
>     >
>     
> s3+http://leveleight.gallery/duplicity-full.2009-03-23T13:27:46-05:00.vol1010.difftar.gpg
>     > Listed
>     >
>     
> s3+http://leveleight.gallery/duplicity-full.2009-03-23T13:27:46-05:00.vol1011.difftar.gpg
>     > Listed
>     >
>     
> s3+http://leveleight.gallery/duplicity-full.2009-03-23T13:27:46-05:00.vol1012.difftar.gpg
>     > Listed
>     >
>     
> s3+http://leveleight.gallery/duplicity-full.2009-03-23T13:27:46-05:00.vol1013.difftar.gpg
>     > Listed
>     >
>     
> s3+http://leveleight.gallery/duplicity-full.2009-03-23T13:27:46-05:00.vol1014.difftar.gpg
>     > Listed
>     >
>     
> s3+http://leveleight.gallery/duplicity-full.2009-03-23T13:27:46-05:00.vol1015.difftar.gpg
>     >
>     > <snip>
>     >
>     > Listed
>     >
>     
> s3+http://leveleight.gallery/duplicity-inc.20090323T182746Z.to.20090324T122905Z.manifest.gpg
>     > Listed
>     >
>     
> s3+http://leveleight.gallery/duplicity-inc.20090323T182746Z.to.20090324T122905Z.vol1.difftar.gpg
>     > Listed
>     >
>     
> s3+http://leveleight.gallery/duplicity-new-signatures.20090323T182746Z.to.20090324T122905Z.sigtar.gpg
>     > 1405 files exist on backend
>     >
>     > <snip>
>     >
>     > Extracting backup chains from list of files:
>     > ['duplicity-full-signatures.2009-03-23T13:27:46-05:00.sigtar.gpg',
>     > 'duplicity-full.2009-03-23T13:27:46-05:00.manifest.gpg',
>     > 'duplicity-full.2009-03-23T13:27:46-05:00.vol1.difftar.gpg',
>     > 'duplicity-full.2009-03-23T13:27:46-05:00.vol10.difftar.gpg',
>     > 'duplicity-full.2009-03-23T13:27:46-05:00.vol100.difftar.gpg',
>     > 'duplicity-full.2009-03-23T13:27:46-05:00.vol1000.difftar.gpg',
>     > 'duplicity-full.2009-03-23T13:27:46-05:00.vol1001.difftar.gpg',
>     > 'duplicity-full.2009-03-23T13:27:46-05:00.vol
>     >
>     > <snip>
>     >
>     > File duplicity-full-signatures.2009-03-23T13:27:46-05:00.sigtar.gpg is
>     > not part of a known set; creating new set
>     > Ignoring file (rejected by backup set)
>     > 'duplicity-full-signatures.2009-03-23T13:27:46-05:00.sigtar.gpg'
>     > File duplicity-full.2009-03-23T13:27:46-05:00.manifest.gpg is not part
>     > of a known set; creating new set
>     > File duplicity-full.2009-03-23T13:27:46-05:00.vol1.difftar.gpg is part
>     > of known set
>     > File duplicity-full.2009-03-23T13:27:46-05:00.vol10.difftar.gpg is
>     part
>     > of known set
>     >
>     > <snip>
>     >
>     > ile duplicity-full.2009-03-23T13:27:46-05:00.vol999.difftar.gpg is
>     part
>     > of known set
>     > File
>     duplicity-inc.20090323T182746Z.to.20090324T122905Z.manifest.gpg is
>     > not part of a known set; creating new set
>     > File
>     duplicity-inc.20090323T182746Z.to.20090324T122905Z.vol1.difftar.gpg
>     > is part of known set
>     > File
>     >
>     duplicity-new-signatures.20090323T182746Z.to.20090324T122905Z.sigtar.gpg
>     > is not part of a known set; creating new set
>     > Ignoring file (rejected by backup set)
>     >
>     'duplicity-new-signatures.20090323T182746Z.to.20090324T122905Z.sigtar.gpg'
>     > Found backup chain [Mon Mar 23 13:27:46 2009]-[Mon Mar 23 13:27:46
>     2009]
>     > Added incremental Backupset (start_time: Mon Mar 23 13:27:46 2009 /
>     > end_time: Tue Mar 24 07:29:05 2009)
>     > Added set Tue Mar 24 07:29:05 2009 to pre-existing chain [Mon Mar 23
>     > 13:27:46 2009]-[Tue Mar 24 07:29:05 2009]
>     > Collection Status
>     > -----------------
>     > Connecting with backend: BotoBackend
>     > Archive dir: None
>     >
>     > Found 0 backup chains without signatures.
>     >
>     > Found a complete backup chain with matching signature chain:
>     > -------------------------
>     > Chain start time: Mon Mar 23 13:27:46 2009
>     > Chain end time: Tue Mar 24 07:29:05 2009
>     > Number of contained backup sets: 2
>     > Total number of contained volumes: 1401
>     >  Type of backup set:                            Time:      Num
>     volumes:
>     >                 Full         Mon Mar 23 13:27:46 2009            
>      1400
>     >          Incremental         Tue Mar 24 07:29:05 2009            
>         1
>     > -------------------------
>     > No orphaned or incomplete backup sets found.
>     > Last full backup date: Mon Mar 23 13:27:46 2009
>     > Using temporary directory /tmp/duplicity-rZ55N_-tempdir
>     > Registering (mktemp) temporary file
>     > /tmp/duplicity-rZ55N_-tempdir/mktemp-jcGOJX-1
>     > Downloading
>     >
>     
> s3+http://leveleight.gallery/duplicity-full.2009-03-23T13:27:46-05:00.manifest.gpg
>     > ===== Begin GnuPG log =====
>     > gpg: encrypted with 2048-bit ELG-E key, ID BLAHLAH, created BLAHLAH
>     > "BLAHLAH (Key for Duplicity) <address@hidden
>     <mailto:address@hidden>
>     > <mailto:address@hidden <mailto:address@hidden>>>"
>     > gpg: Signature made BLAHLAH using DSA key ID BLAHLAH
>     > gpg: Good signature from "BLAHLAH (Key for Duplicity)
>     > <address@hidden <mailto:address@hidden>
>     <mailto:address@hidden <mailto:address@hidden>>>"
>     > ===== End GnuPG log =====
>     > Deleting /tmp/duplicity-rZ55N_-tempdir/mktemp-jcGOJX-1
>     > Forgetting temporary file
>     /tmp/duplicity-rZ55N_-tempdir/mktemp-jcGOJX-1
>     > Registering (mktemp) temporary file
>     > /tmp/duplicity-rZ55N_-tempdir/mktemp-BUaHLW-2
>     > Downloading
>     >
>     
> s3+http://leveleight.gallery/duplicity-full.2009-03-23T13:27:46-05:00.vol1.difftar.gpg
>     > Registering (mktemp) temporary file
>     > /tmp/duplicity-rZ55N_-tempdir/mktemp-YhiFhu-3
>     > Downloading
>     >
>     
> s3+http://leveleight.gallery/duplicity-inc.20090323T182746Z.to.20090324T122905Z.manifest.gpg
>     > ===== Begin GnuPG log =====
>     >
>     > <snip>
>     >
>     > Forgetting temporary file
>     /tmp/duplicity-rZ55N_-tempdir/mktemp-YhiFhu-3
>     > Registering (mktemp) temporary file
>     > /tmp/duplicity-rZ55N_-tempdir/mktemp-LXMvsD-4
>     > Downloading
>     >
>     
> s3+http://leveleight.gallery/duplicity-inc.20090323T182746Z.to.20090324T122905Z.vol1.difftar.gpg
>     > Selecting /var/www/web2/g2data/albums
>     > Difference found: File . has permissions 755, expected 755
>     > Selecting /var/www/web2/g2data/albums/2005
>     > Difference found: New file 2005
>     > Selecting /var/www/web2/g2data/albums/2005/08-11-05_1137.jpg
>     > Difference found: New file 2005/08-11-05_1137.jpg
>     > Selecting /var/www/web2/g2data/albums/2005/08-12-05_1217.jpg
>     >
>     >
>     >
>     >
>     >
>     >
>     >
>     > -------------
>     > This communication (with attachments) is covered by the U.S.C. Sec.
>     > 2510-252, is confidential, and intended only for the use of the
>     > individual or entity named above. If the reader of this message is not
>     > the intended recipient, you are hereby notified that any
>     dissemination,
>     > distribution, or copying of this communication is strictly prohibited.
>     > If you receive this communication in error, notify us immediately,
>     > return the document(s), and destroy all copies of the original
>     message.
>     >
>     >
>     > On Tue, Mar 24, 2009 at 10:16 AM, Kenneth Loafman
>     <address@hidden <mailto:address@hidden>
>     > <mailto:address@hidden <mailto:address@hidden>>> wrote:
>     >
>     >     Josh wrote:
>     >     > I think you misunderstood, I am currently at;
>     >     >
>     >     > Ubuntu version 8.04.2
>     >     > duplicity version 0.5.12
>     >     > boto version 1.6a
>     >
>     >     Yes, I did misunderstand.  Let's start this over...
>     >
>     >     Could you show me the command lines you used?
>     >
>     >     Is the data located on a Windows system or a Linux system?
>     >
>     >     Is it mounted, if so how?
>     >
>     >     It would also help if you could run the verify with a -v9 and
>     let us see
>     >     the first 200 lines and the last 200 lines of the log.  The
>     rest is
>     >     really not needed.
>     >
>     >     ...Thanks,
>     >     ...Ken
>     >
>     >
>     >     _______________________________________________
>     >     Duplicity-talk mailing list
>     >     address@hidden <mailto:address@hidden>
>     <mailto:address@hidden <mailto:address@hidden>>
>     >     http://lists.nongnu.org/mailman/listinfo/duplicity-talk
>     >
>     >
>     >
>     >
>     ------------------------------------------------------------------------
>     >
>     > _______________________________________________
>     > Duplicity-talk mailing list
>     > address@hidden <mailto:address@hidden>
>     > http://lists.nongnu.org/mailman/listinfo/duplicity-talk
> 
> 
> 
>     _______________________________________________
>     Duplicity-talk mailing list
>     address@hidden <mailto:address@hidden>
>     http://lists.nongnu.org/mailman/listinfo/duplicity-talk
> 
> 
> 
> ------------------------------------------------------------------------
> 
> _______________________________________________
> Duplicity-talk mailing list
> address@hidden
> http://lists.nongnu.org/mailman/listinfo/duplicity-talk

signature.asc
Description: OpenPGP digital signature