Duplicity

Overview

Duplicity produces encrypted .TAR format volumes and uploads them to a remote or local file server to backs up directories. By using librsync, the incremental archives are space efficient and only record the parts of files that have changed since the last backup. These archives are encrypted with GnuPG to protect against any spying or modification by the server.

Provider Details

Setup

In this example, we test an installation on Ubuntu 20.04.

  1. Download the latest stable version of Duplicity from their website.

  2. Install the following packages:

    apt-get install gettext   python3-future python3-fasteners python3-boto3

  3. Run the installer to configure Duplicity.

    [root@vm2backup duplicity-0.8.23]# python setup.py install

  4. Create an Exoscale SOS bucket.

  5. Create specific IAM Keys for the bucket.

    By default, all backups are encrypted using GnuPG. We will use ${HOSTNAME} to enable backups of multiple instances to the same bucket.

  6. Run the first backup.

    export AWS_ACCESS_KEY_ID=<YOUR-ACCESS-KEY>
export AWS_SECRET_ACCESS_KEY=<YOUR-SECRET-KEY>
export PASSPHRASE=SECUREPASSWORD
duplicity /var s3://duplicity-backup/${HOSTNAME}/var --s3-endpoint-url=https://sos-de-fra-1.exo.io

  7. Run a full backup from the instance.

    duplicity --exclude /mnt --exclude /tmp --exclude /proc --exclude /run --exclude /sys / s3://duplicity-backup/${HOSTNAME} --s3-endpoint-url=https://sos-de-fra-1.exo.io
Local and Remote metadata are synchronized, no sync needed.
Last full backup date: none
No signatures found, switching to full backup.
--------------[ Backup Statistics ]--------------
StartTime 1655462673.30 (Fri Jun 17 10:44:33 2022)
EndTime 1655463010.13 (Fri Jun 17 10:50:10 2022)
ElapsedTime 336.83 (5 minutes 36.83 seconds)
SourceFiles 171645
SourceFileSize 5369568365 (5.00 GB)
NewFiles 171645
NewFileSize 5369568365 (5.00 GB)
DeletedFiles 0
ChangedFiles 0
ChangedFileSize 0 (0 bytes)
ChangedDeltaSize 0 (0 bytes)
DeltaEntries 171645
RawDeltaSize 5275730169 (4.91 GB)
TotalDestinationSizeChange 2157358512 (2.01 GB)
Errors 0

List Files in Backup target

    duplicity list-current-files s3://duplicity-backup/${HOSTNAME} --s3-endpoint-url=https://sos-de-fra-1.exo.io

Restore a file from the backup

    duplicity restore --file-to-restore var/log/kern.log s3://duplicity-backup/${HOSTNAME} /tmp/kern.log --s3-endpoint-url=https://sos-de-fra-1.exo.io

Tips and Tricks

To remove backups that are older than 1 month:

    duplicity remove-older-than 1M s3://duplicity-backup/${HOSTNAME} --s3-endpoint-url=https://sos-de-fra-1.exo.io

Here is simple backup script that you can modify as needed:

    #!/usr/bin/bash
    if [ -f /root/.duplicity.env ]
    then
    . ./.duplicity.env
    else
    echo "/root/.duplicity.env not found with needed environment"
    exit 1
    fi

    /usr/local/bin/duplicity cleanup --force s3://${BACKUP_BUCKET_NAME}/${HOSTNAME} --s3-endpoint-url=${S3_ENDPOINT}
    /usr/local/bin/duplicity remove-older-than --force ${BACKUP_RETENTION} s3://${BACKUP_BUCKET_NAME}/${HOSTNAME} --s3-endpoint-url=${S3_ENDPOINT}
    /usr/local/bin/duplicity --full-if-older-than ${FULL_EVERY} --exclude /mnt --exclude /tmp --exclude /proc --exclude /run --exclude /sys / s3://${BACKUP_BUCKET_NAME}/${HOST
    NAME} --s3-endpoint-url=${S3_ENDPOINT}

    .duplicity.env File
    export AWS_ACCESS_KEY_ID=<YOUR-ACCESS-KEY>
    export AWS_SECRET_ACCESS_KEY=<YOUR-SECRET-KEY>
    export PASSPHRASE=<YOURVERYSECUREPASSWORD>
    export BACKUP_BUCKET_NAME=<YOUR-SOS-BUCKET>
    export BACKUP_RETENTION=1M
    export S3_ENDPOINT=https://sos-de-fra-1.exo.io
    export FULL_EVERY=7D

For this guide, we are using a bucket in DE-FRA-1, or Frankfurt. You can find all zone endpoints in our API SOS guide.

Additional Resources

Duplicity Documentation