Clone Files to Cloud Storage: Difference between revisions

From WickyWiki
← Older editNewer edit →
mNo edit summary
Line 31: Line 31:
* Create a project and select it
* Create a project and select it
* Select "ENABLE API'S AND SERVICES" and enable the "Google Drive API"
* Select "ENABLE API'S AND SERVICES" and enable the "Google Drive API"
== Configure rclone ==


Note:
Note:
* Token renewal seems to be managed by rclone.
* Token renewal seems to be managed by rclone.
* For automatic configuration you will need a browser in a desktop session, for info on headless configuration go here:
** https://rclone.org/remote_setup/


<syntaxhighlight lang=bash>
= Configure rclone =
rclone config
n                  # New remote
name> gdrive1      # Any short name
Storage Type> drive # Google Drive
client_id>
client_secret>
scope> drive        # Full access all files, excluding Application Data Folder
root_folder_id>
service_account_file>
Auto config> y      # Your browser will open, login with your gmail account and allow access
Team drives> n
is this OK?> y
</syntaxhighlight>


If you don't have a browser (for example via ssh) take the same steps except for:
== Rclone config file ==


<syntaxhighlight lang=bash>
If you have a configuration file 'rclone.conf', you can restore a configuration by placing it in the correct location ~/.config/rclone/rclone.conf
Auto config> n
</syntaxhighlight>


Then:
<source lang=bash>
# Copy and past the URL to a browser in an other window
sudo cp -v rclone.conf ~/.config/rclone/rclone.conf
# Login to your google account
</source>
# Allow access to rclone
# Copy and paste the returned code back and continue from there with "Team drives>"


Sync from local to Google Drive, name 'gdrive1', folder 'rclone':
== Rclone config via SSH session ==


<syntaxhighlight lang=bash>
For automatic configuration you will need a regular browser in a desktop session, info on headless configuration:
rclone sync -v ~/test4.txt "gdrive1:/rclone/"
* https://rclone.org/remote_setup/
</syntaxhighlight>


Sync from Google Drive to local:
For example, we are configuring the drive within a PuTTY ssh session from windows. The authentication proces requires a one-time login to your Google account with a regular browser. In order to make this possible we create a SSH tunnel to make the port available outside the PuTTY session. The port that is used by Rclone is '53682'.


<syntaxhighlight lang=bash>
What is our IP address?
rclone sync -v  "gdrive1:/rclone/test4.txt" ~/rclone/
</syntaxhighlight>


= Configure Google Drive Service Account =
<source lang=bash>
ifconfig
</source>


Note:
Let's say it is '192.168.1.2'. Now create ssh tunnel (login) make http://localhost:53682 available via http://192.168.1.2:53682 :
* The service account and the main account do not share access to Google Drive files.  
* You can not use the Webinterface to access the service account Google Drive files. However, a service account is the preferred method for machine-to-machine interaction.
* To be able to use 'drive-impersonate' you need a "G Suite domain", for this you will need to use a domain that you 'own'. There are various ways to prove you 'own' a domain.


Create a service account :
<source lang=bash>
* Go to the Developer Console: https://console.developers.google.com/apis/dashboard
ssh -fN -L 192.168.1.2:53682:localhost:53682 user1@192.168.1.2
* Login if needed, makes sure you have the correct account
</source>
* Create a project and select it
* Select "ENABLE APIS AND SERVICES" and enable the "Google Drive API"
* Use the " Create Credentials" button to create a "Service account key". Pick a "Service account name". "Role" can be empty.
* Select "Key type JSON" and save the file (rclone.json) for use with rclone config


== Configure rclone ==
You will have to login to the ssh session with 'user1'. This tunnel will remain running as a background proces. We can now configure the Google Drive access.


<syntaxhighlight lang=bash>
<source lang=bash>
rclone config
rclone config
n                   # New remote
d                # delete earlier attempts
name> gdrive1       # Any short name
n                 # new
Storage Type> drive # Google Drive
name> gdrive1
client_id>  
Storage> drive   # google drive
client_secret>  
client_id>
scope> drive       # Full access all files, excluding Application Data Folder
client_secret>
root_folder_id>  
scope> drive           # full access all files
service_account_file> /home/pi/Scripts/rclone.json
service_account_file>
Edit advanced config?> n
Edit advanced config> n
Configure this as a team drive?> n
Auto config> y          # (1) copy url to a browser
...
</source>
 
The configuration will give you an URL and starts waiting for a response.
 
# Copy the URL, for example: http://localhost:53682/auth?state=...
# In the URL replace 'localhost' with the IP address '192.168.1.2' and use it outside the PuTTY session with your browser.
# Logon to your Google account and allow access from Rclone.
# Google will forward to the location that was provided by Rclone you have to replace 'localhost' with the IP address again. Your browser should now reply with 'Succes!'.
# Now go back and continue in the PuTTY session, the configuration will have continued:
 
<source lang=bash>
...
Team drives> n
Configuration complete.
 
is this OK?> y
is this OK?> y
</syntaxhighlight>
quit> q
</source>


List contents:
We don't need the SSH tunnel anymore, stop it like this:
<source lang=bash>
#PID
ps aux | grep '192.168.1.2:53682' | grep -v 'grep'
#kill
kill -9  $( ps aux | grep '192.168.1.2:53682' | grep -v 'grep' |  awk '{ print $2 }' )
</source>


<syntaxhighlight lang=bash>
You should now  be able to list files on your Google Drive with:
rclone lsf -v "gdrive1:"
<source lang=bash>
</syntaxhighlight>
rclone lsl -v gdrive1:/
</source>


Sync:  
To sync from local to Google Drive, name 'gdrive1', folder 'rclone':
<source lang=bash>
rclone sync -v ~/test4.txt "gdrive1:/rclone/"
</source>


<syntaxhighlight lang=bash>
To sync from Google Drive to local:
rclone sync -v /home/pi/Documents "gdrive1:/rclone/"
<source lang=bash>
</syntaxhighlight>
rclone sync -v "gdrive1:/rclone/test4.txt" ~/rclone/
 
</source>
= Configure rclone encryption drive =


Define an encrypted target, in this example the configured remote 'gdrive1' is used as storage in subfolder 'crypt'.
= Configure Rclone encryption drive =


Note:
In this configuration we will create an encryption accesspoint 'gdrive2' that accesses files in a folder 'crypt' on a previously configured accesspoint 'gdrive1'. This makes sure only YOU can see the contents of the files, don't loose your encryption password, there is no way to recover it.
* more info on https://rclone.org/crypt/
* the pass-phrase / salt in this case is more like another password, pick a strong password and you don't need the pass-phrase


<syntaxhighlight lang=bash>
Encryption access via Google Drive access:
<source lang=bash>
rclone config
rclone config
e/n/d/r/c/s/q> n
n                       # new
name> gdrive2
name> gdrive2
Storage> crypt
Storage> crypt         # encrypted
remote> gdrive1:/crypt
remote> gdrive1:crypt
filename_encryption> 2
filename_encryption> standard  # encrypt the filenames
directory_name_encryption> 1
directory_name_encryption> true # encrypt directory names
type password y/g/n> y
type password> y   # enter encryption password
salt y/g/n> n
salt> n       # without salt pass phrase
advanced config y/n> n
advanced config> n
</syntaxhighlight>
Configuration complete.


Sync from local to Google Drive, name 'gdrive2':
OK?> y
quit?> q
</source>


<syntaxhighlight lang=bash>
You can now decrypt and view files on your Google Drive with:
rclone sync -v /home/pi/Documents "gdrive2:/"
</syntaxhighlight>


List contents (encrypted):
<source lang=bash>
rclone lsl -v gdrive2:
</source>


<syntaxhighlight lang=bash>
To sync from local to Google Drive, name 'gdrive2':
rclone lsf -v "gdrive1:/crypt"
</syntaxhighlight>


List contents (decrypted):
<source lang=bash>
rclone sync -v ~/Documents "gdrive2:/"
</source>


<syntaxhighlight lang=bash>
Note:
rclone lsf -v "gdrive2:"
* more info on https://rclone.org/crypt/
</syntaxhighlight>
* the pass-phrase / salt in this case is more like another password, pick a strong password and you don't need the pass-phrase


= Other commands =
= Other Rclone commands =


Note:
Note:
Line 168: Line 166:
Used space / free space / trashed:
Used space / free space / trashed:


<syntaxhighlight lang=bash>
<source lang=bash>
rclone about gdrive1:
rclone about gdrive1:
</syntaxhighlight>
</source>


Clean trash / bin.  
Clean trash / bin.


<syntaxhighlight lang=bash>
<source lang=bash>
rclone cleanup gdrive1:
rclone cleanup gdrive1:
</syntaxhighlight>
</source>


Five most recently modified files with size and total number of files:
Five most recently modified files with size and total number of files:


<syntaxhighlight lang=bash>
<source lang=bash>
rclone tree --sort-modtime --human gdrive1:/ | tail -5
rclone tree --sort-modtime --human gdrive1:/ | tail -5
</syntaxhighlight>
</source>


List of rclone commands:
[https://rclone.org/commands/ Complete list of Rclone commands]
 
= Configure Google Drive Service Account =


* https://rclone.org/commands/
Note:
* I didn't test this
* The service account and the main account do not share access to Google Drive files.
* You can not use the Webinterface to access the service account Google Drive files. However, a service account is the preferred method for machine-to-machine interaction.
* To be able to use 'drive-impersonate' you need a "G Suite domain", for this you will need to use a domain that you 'own'. There are various ways to prove you 'own' a domain.
 
Create a service account :
* Go to the Developer Console: https://console.developers.google.com/apis/dashboard
* Login if needed, makes sure you have the correct account
* Create a project and select it
* Select "ENABLE APIS AND SERVICES" and enable the "Google Drive API"
* Use the " Create Credentials" button to create a "Service account key". Pick a "Service account name". "Role" can be empty.
* Select "Key type JSON" and save the file (rclone.json) for use with rclone config
 
Configure rclone:
 
<source lang=bash>
rclone config
...
service_account_file> /home/user1/Scripts/rclone.json
...
</source>


= See also =
= See also =


* [https://rclone.org/commands/ List of Rclone commands]
* [[Enlarge disk partition image]]
* [[Enlarge disk partition image]]
* [https://github.com/kaczmarkiewiczp/rcloneExplorer rcloneExplorer (Android)]
* [https://github.com/kaczmarkiewiczp/rcloneExplorer rcloneExplorer (Android)]
Line 195: Line 217:
** Download RcloneBrowser : https://github.com/mmozeiko/RcloneBrowser/releases
** Download RcloneBrowser : https://github.com/mmozeiko/RcloneBrowser/releases
** Download Rclone: https://rclone.org/downloads/
** Download Rclone: https://rclone.org/downloads/
* [[Create and Test an SSH Tunnel]]

Revision as of 10:05, 15 August 2024


Links

Install

sudo apt install rclone

Or from the source: 

wget https://rclone.org/install.sh
chmod +x install.sh
sudo ./install.sh

Configure Google Drive Client Access

Configure the account :

Note:

  • Token renewal seems to be managed by rclone.

Configure rclone

Rclone config file

If you have a configuration file 'rclone.conf', you can restore a configuration by placing it in the correct location ~/.config/rclone/rclone.conf 

sudo cp -v rclone.conf ~/.config/rclone/rclone.conf

Rclone config via SSH session

For automatic configuration you will need a regular browser in a desktop session, info on headless configuration:

For example, we are configuring the drive within a PuTTY ssh session from windows. The authentication proces requires a one-time login to your Google account with a regular browser. In order to make this possible we create a SSH tunnel to make the port available outside the PuTTY session. The port that is used by Rclone is '53682'.

What is our IP address? 

ifconfig

Let's say it is '192.168.1.2'. Now create ssh tunnel (login) make http://localhost:53682 available via http://192.168.1.2:53682 : 

ssh -fN -L 192.168.1.2:53682:localhost:53682 user1@192.168.1.2

You will have to login to the ssh session with 'user1'. This tunnel will remain running as a background proces. We can now configure the Google Drive access. 

rclone config
	d                 # delete earlier attempts
	n                 # new
	name> gdrive1
	Storage> drive    # google drive
	client_id>
	client_secret>
	scope> drive            # full access all files
	service_account_file>
	Edit advanced config> n
	Auto config> y          # (1) copy url to a browser
	...

The configuration will give you an URL and starts waiting for a response.

  1. Copy the URL, for example: http://localhost:53682/auth?state=...
  2. In the URL replace 'localhost' with the IP address '192.168.1.2' and use it outside the PuTTY session with your browser.
  3. Logon to your Google account and allow access from Rclone.
  4. Google will forward to the location that was provided by Rclone you have to replace 'localhost' with the IP address again. Your browser should now reply with 'Succes!'.
  5. Now go back and continue in the PuTTY session, the configuration will have continued:
	...
	Team drives> n
	Configuration complete.

	is this OK?> y
	quit> q

We don't need the SSH tunnel anymore, stop it like this: 

#PID
ps aux | grep '192.168.1.2:53682' | grep -v 'grep'
#kill
kill -9  $( ps aux | grep '192.168.1.2:53682' | grep -v 'grep' |  awk '{ print $2 }' )

You should now be able to list files on your Google Drive with: 

rclone lsl -v gdrive1:/

To sync from local to Google Drive, name 'gdrive1', folder 'rclone': 

rclone sync -v ~/test4.txt "gdrive1:/rclone/"

To sync from Google Drive to local: 

rclone sync -v  "gdrive1:/rclone/test4.txt" ~/rclone/

Configure Rclone encryption drive

In this configuration we will create an encryption accesspoint 'gdrive2' that accesses files in a folder 'crypt' on a previously configured accesspoint 'gdrive1'. This makes sure only YOU can see the contents of the files, don't loose your encryption password, there is no way to recover it.

Encryption access via Google Drive access: 

rclone config
	n                       # new
	name> gdrive2
	Storage> crypt          # encrypted
	remote> gdrive1:crypt
	filename_encryption> standard   # encrypt the filenames
	directory_name_encryption> true # encrypt directory names
	type password> y 	  # enter encryption password
	salt> n 		      # without salt pass phrase
	advanced config> n
	Configuration complete.

	OK?> y
	quit?> q

You can now decrypt and view files on your Google Drive with: 

rclone lsl -v gdrive2:

To sync from local to Google Drive, name 'gdrive2': 

rclone sync -v ~/Documents "gdrive2:/"

Note:

  • more info on https://rclone.org/crypt/
  • the pass-phrase / salt in this case is more like another password, pick a strong password and you don't need the pass-phrase

Other Rclone commands

Note:

  • Commands seem to be instantly applied, however, server-side it may take some time to complete. For example, when you 'cleanup' and then check the used space with 'about' repeatedly, you may see 'Trashed' going down to 0 in steps.

Used space / free space / trashed: 

rclone about gdrive1:

Clean trash / bin. 

rclone cleanup gdrive1:

Five most recently modified files with size and total number of files: 

rclone tree --sort-modtime --human gdrive1:/ | tail -5

Complete list of Rclone commands

Configure Google Drive Service Account

Note:

  • I didn't test this
  • The service account and the main account do not share access to Google Drive files.
  • You can not use the Webinterface to access the service account Google Drive files. However, a service account is the preferred method for machine-to-machine interaction.
  • To be able to use 'drive-impersonate' you need a "G Suite domain", for this you will need to use a domain that you 'own'. There are various ways to prove you 'own' a domain.

Create a service account :

  • Go to the Developer Console: https://console.developers.google.com/apis/dashboard
  • Login if needed, makes sure you have the correct account
  • Create a project and select it
  • Select "ENABLE APIS AND SERVICES" and enable the "Google Drive API"
  • Use the " Create Credentials" button to create a "Service account key". Pick a "Service account name". "Role" can be empty.
  • Select "Key type JSON" and save the file (rclone.json) for use with rclone config

Configure rclone: 

rclone config
	...
	service_account_file> /home/user1/Scripts/rclone.json
	...

See also

Retrieved from "https://wjv.duckdns.org/mediawiki/index.php?title=Clone_Files_to_Cloud_Storage&oldid=3705"