How to migrate cloud provider with rclone

rclone is a command-line program to migrate between a large number of cloud storage providers. It can be used to migrate data from a local environment or other cloud providers to the Exoscale Object Storage.

Installation

rclone is available as a package in most Linux distributions and is also available for Windows and MacOS. Since it is a stand-alone executable it can simply be downloaded and started.

On Windows it is recommended to set the PATH environment variable to point to the folder rclone is located in.

Prerequisites

As prerequisite for the following documentation, you’ll have to:

Create an Object Storage Bucket, either via the portal or the CLI. You can refer to the Object Storage documentation
Create an IAM key, either via the portal or the CLI. You can refer to the IAM documentation

Configuration

Before you can use rclone you will have to configure your accounts first. To do that please enter the following command:

# rclone config

You will be shown the following options:

n) New remote
s) Set configuration password
q) Quit config

Select n to set up your Exoscale Account and enter an account name. (e.g. Exoscale) This name will be used to start the sync.

n) New remote
s) Set configuration password
q) Quit config
n/s/q> n
name> Exoscale

Next, you will be shown a list of supported cloud storage providers. Please select Amazon S3 since the Exoscale Object Storage implements the Amazon S3 standard.

Next, you will be asked for the access key and secret source. Please enter 1:

Choose a number from below, or type in your own value
 1 / Enter AWS credentials in the next step
 \ "false"
 2 / Get AWS credentials from the environment (env vars or IAM)
 \ "true"

Next, enter your access key and secret:

AWS Access Key ID - leave blank for anonymous access or runtime credentials.
access_key_id> EXO...
AWS Secret Access Key (password) - leave blank for anonymous access or runtime credentials.
secret_access_key> ...

Next, you will have to select the region:

 / If using an S3 clone that only understands v2 signatures
12 | eg Ceph/Dreamhost
 | set this and make sure you set the endpoint.
 \ "other-v2-signature"

Select the other-v2-signature option. For the endpoint enter sos-ZONE.exo.io, taking care of substituting ZONE with the proper zone.

Note

You can find all Zones on our website and with the cli: exo zones list

Endpoint for S3 API.
Leave blank if using AWS to use the default endpoint for the region.
Specify if using an S3 clone such as Ceph.
endpoint> sos-at-vie-1.exo.io

For the last four questions please enter the following:

Location constraint: Empty for US Region, Northern Virginia or Pacific Northwest.
Canned ACL: Owner gets FULL_CONTROL. No one else has access rights (default).
Server-Side Encryption: None
Storage class: Default

Then save the configuration with Y.

Note

You will now have to configure the second provider.

Synchronization

Now the cloud providers are configured, so you can synchronize the files. To synchronize from a local folder to the Exoscale Object Storage enter the following command:

rclone sync /LOCAL/FOLDER Exoscale:BUCKET-NAME-HERE/

Instead of Exoscale please enter the name you have given your configuration. To synchronize from a second cloud provider you can use the following command:

rclone sync OtherProvider:SOURCE-DIRECTORY Exoscale:BUCKET-NAME-HERe/