How to back up with Duplicity to Exoscale Object Storage
Duplicity
Overview
Duplicity backs directories by producing encrypted tar-format volumes and uploading them to a remote or local file server. By using librsync, the incremental archives are space efficient and only record the parts of files that have changed since the last backup. These archives are encrypted and/or signed with GnuPG, to protect them against spying and/or modification by the server.
Provider Details
Setup
Installation tested on Ubuntu 20.04
- Download the latest stable version from Duplicity
-
Install the following packages
apt-get install gettext python3-future python3-fasteners python3-boto3
-
Run the “installer” to configure Duplicity
[root@vm2backup duplicity-0.8.23]# python setup.py install
-
Create an Exoscale SOS Bucket
-
Create specific IAM Keys for this Bucket
By default, all backups are encrypted using GnuPG. We will use ${HOSTNAME} to enable backups of multiple Instances to the same bucket
-
Run the first backup
export AWS_ACCESS_KEY_ID=<YOUR-ACCESS-KEY> export AWS_SECRET_ACCESS_KEY=<YOUR-SECRET-KEY> export PASSPHRASE=SECUREPASSWORD duplicity /var s3://duplicity-backup/${HOSTNAME}/var --s3-endpoint-url=https://sos-de-fra-1.exo.io
-
Run a full backup from the instance
duplicity --exclude /mnt --exclude /tmp --exclude /proc --exclude /run --exclude /sys / s3://duplicity-backup/${HOSTNAME} --s3-endpoint-url=https://sos-de-fra-1.exo.io Local and Remote metadata are synchronized, no sync needed. Last full backup date: none No signatures found, switching to full backup. --------------[ Backup Statistics ]-------------- StartTime 1655462673.30 (Fri Jun 17 10:44:33 2022) EndTime 1655463010.13 (Fri Jun 17 10:50:10 2022) ElapsedTime 336.83 (5 minutes 36.83 seconds) SourceFiles 171645 SourceFileSize 5369568365 (5.00 GB) NewFiles 171645 NewFileSize 5369568365 (5.00 GB) DeletedFiles 0 ChangedFiles 0 ChangedFileSize 0 (0 bytes) ChangedDeltaSize 0 (0 bytes) DeltaEntries 171645 RawDeltaSize 5275730169 (4.91 GB) TotalDestinationSizeChange 2157358512 (2.01 GB) Errors 0
List Files in Backup target
duplicity list-current-files s3://duplicity-backup/${HOSTNAME} --s3-endpoint-url=https://sos-de-fra-1.exo.io
Restore a file from the backup
duplicity restore --file-to-restore var/log/kern.log s3://duplicity-backup/${HOSTNAME} /tmp/kern.log --s3-endpoint-url=https://sos-de-fra-1.exo.io
Remove Backups older than 1 Month
duplicity remove-older-than 1M s3://duplicity-backup/${HOSTNAME} --s3-endpoint-url=https://sos-de-fra-1.exo.io
A simple Backup script which can be adapted to your needs
#!/usr/bin/bash
if [ -f /root/.duplicity.env ]
then
. ./.duplicity.env
else
echo "/root/.duplicity.env not found with needed environment"
exit 1
fi
/usr/local/bin/duplicity cleanup --force s3://${BACKUP_BUCKET_NAME}/${HOSTNAME} --s3-endpoint-url=${S3_ENDPOINT}
/usr/local/bin/duplicity remove-older-than --force ${BACKUP_RETENTION} s3://${BACKUP_BUCKET_NAME}/${HOSTNAME} --s3-endpoint-url=${S3_ENDPOINT}
/usr/local/bin/duplicity --full-if-older-than ${FULL_EVERY} --exclude /mnt --exclude /tmp --exclude /proc --exclude /run --exclude /sys / s3://${BACKUP_BUCKET_NAME}/${HOST
NAME} --s3-endpoint-url=${S3_ENDPOINT}
.duplicity.env File
export AWS_ACCESS_KEY_ID=<YOUR-ACCESS-KEY>
export AWS_SECRET_ACCESS_KEY=<YOUR-SECRET-KEY>
export PASSPHRASE=<YOURVERYSECUREPASSWORD>
export BACKUP_BUCKET_NAME=<YOUR-SOS-BUCKET>
export BACKUP_RETENTION=1M
export S3_ENDPOINT=https://sos-de-fra-1.exo.io
export FULL_EVERY=7D
For this guide we are using a bucket in de-fra-1. You can find all endpoints in our API SOS community guide.