Clone Files to Cloud Storage

From WickyWiki


Links

Install

sudo apt install rclone

Or from the source: 

wget https://rclone.org/install.sh
chmod +x install.sh
sudo ./install.sh

Configure Google Drive Client Access

Configure the account :

Note:

  • Token renewal seems to be managed by rclone.

Configure rclone

Rclone config file

If you have a configuration file 'rclone.conf', you can restore a configuration by placing it in the correct location ~/.config/rclone/rclone.conf 

sudo cp -v rclone.conf ~/.config/rclone/rclone.conf

Rclone config via SSH session

For automatic configuration you will need a regular browser in a desktop session, info on headless configuration:

For example, we are configuring the drive within a PuTTY ssh session from windows. The authentication proces requires a one-time login to your Google account with a regular browser. In order to make this possible we create a SSH tunnel to make the port available outside the PuTTY session. The port that is used by Rclone is '53682'.

What is our IP address? 

ifconfig

Let's say it is '192.168.1.2'. Now create ssh tunnel (login) make http://localhost:53682 available via http://192.168.1.2:53682 : 

ssh -fN -L 192.168.1.2:53682:localhost:53682 user1@192.168.1.2

You will have to login to the ssh session with 'user1'. This tunnel will remain running as a background proces. We can now configure the Google Drive access. 

rclone config
	d                 # delete earlier attempts
	n                 # new
	name> gdrive1
	Storage> drive    # google drive
	client_id>
	client_secret>
	scope> drive            # full access all files
	service_account_file>
	Edit advanced config> n
	Auto config> y          # copy url
	...

The configuration will give you an URL and starts waiting for a response.

  1. Copy the URL, for example: http://localhost:53682/auth?state=...
  2. In the URL replace 'localhost' with the IP address '192.168.1.2' and use it outside the PuTTY session with your browser.
  3. Logon to your Google account and allow access from Rclone.
  4. Google will forward to the location that was provided by Rclone you have to replace 'localhost' with the IP address again. Your browser should now reply with 'Succes!'.
  5. Now go back and continue in the PuTTY session, the configuration will have continued:
	...
	Team drives> n
	Configuration complete.

	is this OK?> y
	quit> q

We don't need the SSH tunnel anymore, stop it like this: 

#PID
ps aux | grep '192.168.1.2:53682' | grep -v 'grep'
#kill
kill -9  $( ps aux | grep '192.168.1.2:53682' | grep -v 'grep' |  awk '{ print $2 }' )

You should now be able to list files on your Google Drive with: 

rclone lsl -v gdrive1:/

To sync from local to Google Drive, name 'gdrive1', folder 'rclone': 

rclone sync -v ~/test4.txt "gdrive1:/rclone/"

To sync from Google Drive to local: 

rclone sync -v  "gdrive1:/rclone/test4.txt" ~/rclone/

Configure Rclone encryption drive

In this configuration we will create an encryption accesspoint 'gdrive2' that accesses files in a folder 'crypt' on a previously configured accesspoint 'gdrive1'. This makes sure only YOU can see the contents of the files, don't loose your encryption password, there is no way to recover it.

Encryption access via Google Drive access: 

rclone config
	n                       # new
	name> gdrive2
	Storage> crypt          # encrypted
	remote> gdrive1:crypt
	filename_encryption> standard   # encrypt the filenames
	directory_name_encryption> true # encrypt directory names
	type password> y 	  # enter encryption password
	salt> n 		      # without salt pass phrase
	advanced config> n
	Configuration complete.

	OK?> y
	quit?> q

You can now decrypt and view files on your Google Drive with: 

rclone lsl -v gdrive2:

To sync from local to Google Drive, name 'gdrive2': 

rclone sync -v ~/Documents "gdrive2:/"

Note:

  • more info on https://rclone.org/crypt/
  • the pass-phrase / salt in this case is more like another password, pick a strong password and you don't need the pass-phrase

Other Rclone commands

Note:

  • Commands seem to be instantly applied, however, server-side it may take some time to complete. For example, when you 'cleanup' and then check the used space with 'about' repeatedly, you may see 'Trashed' going down to 0 in steps.

Used space / free space / trashed: 

rclone about gdrive1:

Clean trash / bin. 

rclone cleanup gdrive1:

Five most recently modified files with size and total number of files: 

rclone tree --sort-modtime --human gdrive1:/ | tail -5

Complete list of Rclone commands

Configure Google Drive Service Account

Note:

  • I didn't test this
  • The service account and the main account do not share access to Google Drive files.
  • You can not use the Webinterface to access the service account Google Drive files. However, a service account is the preferred method for machine-to-machine interaction.
  • To be able to use 'drive-impersonate' you need a "G Suite domain", for this you will need to use a domain that you 'own'. There are various ways to prove you 'own' a domain.

Create a service account :

  • Go to the Developer Console: https://console.developers.google.com/apis/dashboard
  • Login if needed, makes sure you have the correct account
  • Create a project and select it
  • Select "ENABLE APIS AND SERVICES" and enable the "Google Drive API"
  • Use the " Create Credentials" button to create a "Service account key". Pick a "Service account name". "Role" can be empty.
  • Select "Key type JSON" and save the file (rclone.json) for use with rclone config

Configure rclone: 

rclone config
	...
	service_account_file> /home/user1/Scripts/rclone.json
	...

See also

Retrieved from "https://wjv.duckdns.org/mediawiki/index.php?title=Clone_Files_to_Cloud_Storage&oldid=3706"