Storage Quick Start
Through this guide you will learn the basics of Exoscale’s Simple Object Storage (SOS).
Object Storage allows you to store, distribute and manage large quantities of files (objects). Examples applications span from simple backup to complex media delivery. You can store almost anything in SOS: assets, backups, email attachments, videos, pictures and anything else you may imagine.
Your data will stay in the zone you initially stored it to, Exoscale will not move date on your behalf. For Example if you first upload data to Simple Object Storage in Geneva, then your data is kept in Switzerland, under Swiss privacy regulation. Whatever the initial location, it is replicated in that same location in at least three physical distinct copies to ensure maximum safety.
All the actions we will perform through the command-line can also be performed through the interface: you can do almost everything in the Exoscale Console as well, although the real power of SOS is in the tools you may script and in the clients you may use in your application.
S3 Compatibility
SOS is S3-compatible: this means it works with most object storage clients and provides library access from a host of languages. You can drop-in from other services and you don’t have to fear to be locked in. Most of the time a simple change of API credentials and endpoint allows to switch to our service.
Here are a few clients and tools that you can use with no friction:
We will use s3cmd
in this guide.
S3 Unsupported Features
Although we strive to maintain full compatibility with the S3 API, we do not support all S3 features. Currently the following functionality cannot be used on Exoscale:
- Object Versioning
- Object Tagging
- Object Lifecycle Management
- Bittorent
- Bucket Policy
- Bucket access logging
- Bucket event notification
- Website
- Encryption should be performed on client side.
Setup
This guide assumes you never used s3cmd. If this isn’t the case, simply take a look at the .s3cfg
example here below, drop in your Exoscale’s API Key and API Secret and you are good to go.
First of all install s3cmd with your favorite package-manager:
- Debian/Ubuntu Linux:
apt-get update && apt-get install s3cmd
- OS X:
brew install s3cmd
Then create a file named .s3cfg
in your home directory and copy/paste the following configuration template:
[default]
host_base = sos-{zone}.exo.io
host_bucket = %(bucket)s.sos-{zone}.exo.io
access_key = $EXO_SOS_KEY
secret_key = $EXO_SOS_SECRET
use_https = True
Replace {zone}
with the zone your bucket belongs to: e.g. for a bucket deployed on the ch-dk-2
zone that would be:
host_base = sos-ch-dk-2.exo.io
host_bucket = %(bucket)s.sos-ch-dk-2.exo.io
Replace $EXO_SOS_KEY
and $EXO_SOS_SECRET
with your API Key and API Secret.
You can choose to use your User Keys or a specific IAM API key.
Your first Bucket
First we need to create a container for all of our objects, which is called a Bucket.
s3cmd mb s3://my-new-bucket
This creates a bucket called ‘my-new-bucket’.
Bucket names are unique across our entire platform. You may encounter an
error creating my-new-bucket
. If this name is already taken you need to
choose another name.
The reason for this is that buckets and their content can be accessed under
https://sos-ch-dk-2.exo.io/my-new-bucket/
or https://my-new-bucket.sos-ch-dk-2.exo.io
.
Buckets become subdomains and prefixed URLs under the sos-ch-dk-2.exo.io
domain, and
hence names have to be unique. This does not mean objects are publicly
accessible: you can define precise access control through IAM Keys or ACLs, and uploaded objects are private by default.
You can check your bucket is here by listing your buckets:
s3cmd ls
Uploading and downloading assets
Now grab a file you have on your computer and upload it to your bucket:
s3cmd put picture.jpg s3://my-new-bucket/picture.jpg
Check your file was uploaded successfully:
s3cmd ls s3://my-new-bucket
You can also verify the presence of your file having a peek in the interface:
select the Storage service from the main menu and you should see your
my-new-bucket
listed here. Select the Bucket and you will find your file!
You can try to upload more files in the Console, with a simple drag and drop. And of course you can get back your files:
s3cmd get s3://my-new-bucket/picture.jpg picture_download.jpg
Advanced Options
Buckets and objects support advanced options: ACL for buckets and objects, CORS for buckets and Metadata for objects.
Those options are available through the API but also through the interface: editing a bucket or an object in the Console opens a drawer with the configurable options.
IAM
It is possible to limit programmatic access to a specific bucket by creating dedicated IAM API Keys and limit them to a single bucket.
ACL
Access Control List (ACL) allows you to define specific permission at the bucket level and single object level, allowing a granular access policy. ACLs are not inherited from parent object.
ACL let you manage access to buckets and objects, as for Read, Write, Read ACP, Write ACP, and Full Control (Read + Write). Your objects may be accessible by their public URL, your buckets writable by your team and so on.
For in-depth details about ACL’s we invite you to read the SOS ACL documentation
CORS
CORS (Cross-Origin Resource Sharing) allows your browser-based applications to perform actions on buckets and objects contained in it. CORS is available only at the bucket level and its configuration is applied to all the objects contained.
The SOS CORS documentation explains it and provides some examples.
Metadata
Each object contained in a bucket can be tagged with key-value pairs. This metadata is presented through HTTP headers on SOS requests and responses.
Have a look on the SOS Metadata documentation for more information and examples.