Re: [Duplicity-talk] Best Duplicity 8 backend for Google Drive?

[zga9uhnq4g]

Thanks Edgar, for spending the time and writing up such a clear description.

I started looking at the gdrive backend code in gitlab last night, and 
finally managed to get it working at 1 AM this morning (before I went to 
bed).  I had similar issues figuring out the Debian packages that 
provide the google modules gdrive needs.  Didn't have time to reply to 
this thread until now.

The only thing I did differently is how I found the folder id.  If you 
right-click on a folder in the Google Drive web page, you can select 
"Get link" which brings up a dialog that allows you to copy the link.  I 
just guessed that the part at the end was the ID, and discovered that it 
worked (after I resolved a few other issues).

My next step is to "replicate" my existing backup to Google Drive, and 
modify my backup script to use Google Drive.
After that I want to try to get the box backend working (my free box 
account has more storage space than my free Google Drive account).

Thanks again for your help.


On 2021-08-17 02:38, edgar.soldin--- via Duplicity-talk 
duplicity-talk-at-nongnu.org |duplicity-talk| wrote:
> ok Peter,
>
> just wasted 2hs on this, because you got me interested :). essentially, the problem is 
> twofold, with the first problem being the "bigger" issue
>
> issue1. duplicity does not provide a client API authentification. it has to be 
> created by every user themselves on https://console.developers.google.com
> issue2. figure out the url to use, as it is not mentioned in the man page
>
> if you managed to do all that _and_ pip3 installed really all (google*) modules 
> duplicity complains about being needed, it works as expected
>
> so for issue1
> 1. login to google drive in your browser
> 2. go to https://console.developers.google.com
> 3. create a project (!), switch to it
> 4. Library -> add Google-Drive-API
> 5. Credentials -> add OAuth-Client
> 6. OAuth Consent screen -> publish (else you'll get 
> https://stackoverflow.com/questions/65756266/error-403-access-denied-the-developer-hasn-t-given-you-access-to-this-app-despi)
> 7. download "client_secret_<client_id>.apps.googleusercontent.com.json" and 
> place it on your backup machine as '~/gdrive_client.json' (used below)
>
> wrt. to issue2
> the url to use looks like
>   gdrive://<client_id>.apps.googleusercontent.com/<subfolder_to place files 
> in>?myDriveFolderID=<a_folder_id>
> e.g.
>   
> gdrive://388783630241-02dsm3ijbnjnjen7266vpa4jv721mt61.apps.googleusercontent.com/test?myDriveFolderID=1v6P4LkVRhoNZOvN0R2VlR7tC55-HmroX
>
> you can get the folder id as described here 
> https://ploi.io/documentation/database/where-do-i-get-google-drive-folder-id
> didn't check if the subfolder is really needed btw.
>
> now you should be set and you can try running a backup like
>   export GOOGLE_CLIENT_SECRET_JSON_FILE=~/gdrive_client.json
>   export GOOGLE_CREDENTIALS_FILE=~/gdrive_creds
>   duplicity-0.8.20/bin/duplicity temp/ 
> gdrive://388783630241-02dsm3ijbnjnjen7266vpa4jv721mt61.apps.googleusercontent.com/test?myDriveFolderID=1v6P4LkVRhoNZOvN0R2VlR7tC55-HmroX
>
> it'll ask you to visit the url and provide a secret and fill '~/gdrive_creds' 
> for subsequent runs.
>
> hope that helps.. ede/duply.net
>
> ps. @Ken - how about we provide a default duplicity client auth with duplicity 
> distro to make life easier for users?
>
>
> On 17.08.2021 05:19, zga9uhnq4g--- via Duplicity-talk wrote:
> > Yes I did read it.  That's the "Note on GDrive Backend" section that I 
> > mentioned I'm having trouble understanding.
> >
> > First, it talks about a "Google My Drive based shared folder" and I want to use 
> > a Google My Drive regular (non-shared) folder (which Ken said should work).
> >
> > Second, even though the "Note on GDrive Backend" section mentions using a regular account (as an alternative to a 
> > "service account"), the example URL in that section as well is the gdrive example in the URL Format section only show 
> > "service account name" and "service account email address".  The last sentence in the "Note on GDrive 
> > Backend" section seems to imply that the OAuth client ID is alternative for the service email address, but doesn't make it 
> > clear how/where to provide that client ID.
> >
> > In addition, the example URLS in the "URL Format" section and the "Note on GDrive 
> > Backend" section look quite  different, so I'm not sure which format to follow.
> >
> >      thanks,
> >      Peter
> >
> > On 2021-08-16 19:41, edgar.soldin--- via Duplicity-talk 
> > duplicity-talk-at-nongnu.org |duplicity-talk| wrote:
> > > not familiar with gdrive, but did you read the man page section
> > > http://duplicity.nongnu.org/vers8/duplicity.1.html#sect27 
> > > <http://duplicity.nongnu.org/vers8/duplicity.1.html#sect27>
> > > ?
> > >
> > > ..ede/duply.net
> > >
> > > On August 17, 2021 12:35:38 AM GMT+02:00, zga9uhnq4g--- via Duplicity-talk 
> > > <duplicity-talk@nongnu.org> wrote:
> > >
> > >      Thanks for the info Ken.
> > >
> > >      A couple follow up questions.
> > >
> > >        * Is there a way to contact the GDrive backend developer with questions, 
> > > or is this list the best way?
> > >        * I'm having trouble understanding the Note on GDrive Backend, 
> > > specifically what the gdrive:// URL should look like.  If my Google Drive 
> > > account is john_doe123@gmail.com and I want the duplicity backup stored in the 
> > > (non-shared) /backup/duplicity folder, can you  (or someone) tell me what the 
> > > gdrive:// URL should look like?
> > >
> > >          thanks again,
> > >          Peter
> > >
> > >
> > >      On 2021-08-15 09:51, Kenneth Loafman kenneth-at-loafman.com 
> > > |duplicity-talk| wrote:
> > > >      Hi Peter,
> > > >
> > > >      All of our backends are written by volunteers, so I'm not the best one to 
> > > > answer.
> > > >
> > > >        * gdocs is 10 years old and some still use it (to be deprecated in 
> > > > 0.9.0).
> > > >        * pydrive replaced gdocs 7 years ago and some still use it (to be 
> > > > deprecated in 0.9.0).
> > > >        * gdrive replaced pydrive 5 months ago.
> > > >
> > > >      As far as I know, all three still work, which is surprising since gdocs is 
> > > > deprecated by Google.  I would suggest using gdrive.
> > > >
> > > >      A service account is for a team while a regular account is a normal Google 
> > > > user.  I don't know any pros or cons.
> > > >
> > > >      No, My Drive folders do not need to be shared, they just need different 
> > > > addressing than shared.
> > > >
> > > >      Sorry, that's all I can think of at the moment,  Hope it helps!
> > > >
> > > >      ...Ken
> > > >
> > > >
> > > >      On Sun, Aug 15, 2021 at 12:23 AM zga9uhnq4g--- via Duplicity-talk 
> > > > <duplicity-talk@nongnu.org <mailto:duplicity-talk@nongnu.org>> wrote:
> > > >
> > > >          Looking at the duplicity 8 man page, it appears that there are several 
> > > > options for storing backups in Google Drive:
> > > >
> > > >            * gdocs://user[:password]@other.host/some_dir
> > > >            * pydrive://<service account’ email 
> > > > address>@developer.gserviceaccount.com/some_dir 
> > > > <http://developer.gserviceaccount.com/some_dir>
> > > >            * gdrive://<service account’ email 
> > > > address>@developer.gserviceaccount.com/some_dir 
> > > > <http://developer.gserviceaccount.com/some_dir>
> > > >
> > > >          Which of them should I expect to be the most reliable and efficient?
> > > >          Searching duplicity-talk, I saw some indication that the gdocs API has been 
> > > > deprecated, which makes me leery of using it.  The man page says "GDrive: is a 
> > > > rewritten PyDrive: backend ...", but I didn't see any mention of gdrive in 
> > > > duplicity-talk, so I'm not sure how widely it is being used.
> > > >
> > > >          I have a couple questions about the Notes on the Pydrive and Gdrive 
> > > > backends in the man page:
> > > >
> > > >            * What are pros and cons of using a service account versus a regular 
> > > > account?
> > > >            * The Note on GDrive Backend only mentions using shared folders for 
> > > > 'My Drive', does that mean I can't store a duplicity backup in a non-shared 
> > > > folder?
> > > >
> > > >              thanks,
> > > >              Peter
> > > >
> > > >          _______________________________________________
> > > >          Duplicity-talk mailing list
> > > >          Duplicity-talk@nongnu.org <mailto:Duplicity-talk@nongnu.org>
> > > >          https://lists.nongnu.org/mailman/listinfo/duplicity-talk 
> > > > <https://lists.nongnu.org/mailman/listinfo/duplicity-talk>
> > >
> > > _______________________________________________
> > > Duplicity-talk mailing list
> > > Duplicity-talk@nongnu.org
> > > https://lists.nongnu.org/mailman/listinfo/duplicity-talk
> >
> > _______________________________________________
> > Duplicity-talk mailing list
> > Duplicity-talk@nongnu.org
> > https://lists.nongnu.org/mailman/listinfo/duplicity-talk
>
> _______________________________________________
> Duplicity-talk mailing list
> Duplicity-talk@nongnu.org
> https://lists.nongnu.org/mailman/listinfo/duplicity-talk