Bucket Lifecycle

Bucket lifecycle management in object storage refers to automated rules that manage objects over time. These rules help control storage costs, enforce data retention policies, and maintain storage organization by automatically handling tasks such as deleting old objects, managing versions, and cleaning up incomplete uploads.

Lifecycle on Object Storage

Exoscale Simple Object Storage (SOS) does not yet support native bucket lifecycle management, meaning you can’t configure automatic expiration rules directly in the SOS service. However, we acknowledge customers may need this functionality for operational, cost, or compliance reasons.

To bridge this gap, we’ve created a lightweight tool that emulates lifecycle functionality that you can run client-side and schedule according to your needs.

This tool helps you achieve:

  • Cost Optimization: Automatically removing data that exceeds retention periods to reduce storage costs.
  • Data Governance: Enforcing compliance and retention requirements.
  • Storage Hygiene: Managing versions, removing incomplete uploads, and avoiding clutter in object storage.

Supported Features

  1. Expiration of Objects: Objects created more than X days ago (user-defined) are deleted.
  2. Retention of Non-current Versions: A specified number of non-current versions (versions that are not the most recent) are retained.
  3. Removal of Delete Markers: when all object versions are deleted and only a delete marker remains, dangling delete markers are removed.
  4. Abortion of Multipart Uploads: Incomplete multipart uploads that started before a specified number of days are aborted.

Prerequisites

A container runtime: the tool is package as a container image. We provide a usage example with the Docker CLI, which can be adapted to your runtime environment.

Versioning enabled on your bucket: the tool will error out if versioning is not enabled.

Usage

Configuration

Configuration shares the same format as AWS S3’s bucket lifecycle configuration, but only a subset of AWS’s lifecycle rules are supported.

Lifecycle configuration is specified as a JSON document consisting of one or more rules.

{
    "Rules": [
        {
            "ID": "RULE001",
            "Status": "Enabled",
            "Expiration": {
                "Days": 10,
                "ExpiredObjectDeleteMarker": true
            },
            "NoncurrentVersionExpiration": {
                "NewerNoncurrentVersions": 20
            },
            "AbortIncompleteMultipartUpload": {
                "DaysAfterinitiation": 7
            }
        }
    ]
}

Configuration format

  • Rules: An array containing one or more of rules for bucket lifecycle management.
  • ID: A unique identifier for the rule, such as "RULE001".
  • Status: The state of the rule. Always set it to "Enabled": the presence of a rule in the configuration leads to it being applied. For disabling rules, simply drop them from the configuration.
  • Expiration: Specifies the expiration settings for objects.
    • Days: Objects created 10 days ago or more are deleted.
    • ExpiredObjectDeleteMarker: when all previous versions of an object are deleted and only a delete marker remains, dangling delete markers are purged.
  • NoncurrentVersionExpiration: Defines how non-current versions are handled.
    • NewerNoncurrentVersions: Retains the 20 most recent non-current versions.
    • NoncurrentDays: non-current versions created more than X days ago are deleted. Mutually exclusive with NewerNoncurrentVersions.
  • AbortIncompleteMultipartUpload: Configures the abortion of incomplete multipart uploads.
    • DaysAfterInitiation: Incomplete multipart uploads that started more than 7 days ago are aborted.

Running the tool

Run the following Docker command in your terminal:

docker run \
  -v /path/to/your/config.json:/config.json \
  exoscale/sos-client-bucket-lifecycle  \
  --config /config.json \
  --bucket yourbucket \
  --zone your-zone \
  --access-key EXOSCALE_API_KEY \
  --secret-key EXOSCALE_API_SECRET

Where:

  • /path/to/your/config.json is the actual path to your JSON configuration file.
  • yourbucket is the name of the bucket on which to apply lifecycle rules.
  • your-zone is the Exoscale zone where the bucket is located.
  • EXOSCALE_API_KEY and EXOSCALE_API_SECRET are Exoscale credentials with the necessary IAM permissions to perform relevant operations:
    • get-bucket-versioning
    • list-object-versions
    • list-mulitpart-uploads
    • abort-multipart-upload
    • delete-object

Output Example

Executing lifecycle configuration
[expiration] key: obj1, version 1234 removed
[newer non current versions] key: obj2, version 5678 removed
[non current days] key: obj3, version 9012 removed
[expire delete marker] key: obj4, version 3456 removed
Done

Scheduling

We recommend scheduling this tool to run on a daily basis.

Limitations

  • Filters such as prefixes, object sizes or tags are not supported.
  • All rules that are present in the configuration are considered enabled regardless of their state.

Important Notes

  • Customer Responsibility: This workaround is provided as-is. Users are fully responsible for deploying, configuring, and running the tool. It’s not integrated into the SOS platform, and Exoscale does not operate or monitor it on users’ behalf.
  • Temporary Workaround: This tool is intended as a temporary measure. Exoscale has native bucket lifecycle on the roadmap, and we will deprecate this tool in favor of native functionality once available.
  • Security and Access: Always use appropriate credentials and permissions, especially when running the tool in production environments.
  • Incompatibility with Backup Solutions: Backup solutions that integrate with SOS typically implement retention policies on their own. When that’s the case, performing lifecycle operations on the side risks corrupting backups.