Implement Bucket Lifecycle

Bucket lifecycle management in object storage refers to automated rules that manage objects over time. These rules help control storage costs, enforce data retention policies, and maintain storage organization by automatically handling tasks such as deleting old objects, managing versions, and cleaning up incomplete uploads.

Lifecycle on Object Storage

Exoscale Simple Object Storage (SOS) supports native bucket lifecycle management for expiration of objects.

Bucket Lifecycle enables:

• Cost Optimization: Automatically removing data that exceeds retention periods to reduce storage costs.
• Data Governance: Enforcing compliance and retention requirements.
• Storage Hygiene: Managing versions, removing incomplete uploads, and avoiding clutter in object storage.

Supported Features

1. Expiration of Objects: Objects created more than X days ago (user-defined) are deleted.
2. Retention of Non-current Versions: A specified number of non-current versions (versions that are not the most recent) are retained.
3. Removal of Delete Markers: when all object versions are deleted and only a delete marker remains, dangling delete markers are removed.
4. Abortion of Multipart Uploads: Incomplete multipart uploads that started before a specified number of days are aborted.

Usage

Configuration

Configuration shares the same format as AWS S3’s bucket lifecycle configuration, but only a subset of AWS’s lifecycle rules are supported (storage class transition is currently unsupported).

Lifecycle configuration is specified as a JSON document consisting of one or more rules.

{
    "Rules": [
        {
            "ID": "RULE001",
            "Status": "Enabled",
            "Filter":{
                "Prefix": "logs/"
            },
            "Expiration": {
                "Days": 10
            },
            "NoncurrentVersionExpiration": {
                "NewerNoncurrentVersions": 20
            },
            "AbortIncompleteMultipartUpload": {
                "DaysAfterInitiation": 7
            }
        },
        {
            "ID": "RULE002",
            "Status": "Enabled",
            "Expiration": {
                "ExpiredObjectDeleteMarker": true
            },
            "Filter": {}
        },
        {
            "ID": "RULE003",
            "Status": "Enabled",
            "Filter": {
                "And": {
                    "Prefix": "logs/",
                    "ObjectSizeGreaterThan": 1048576,
                    "ObjectSizeLessThan": 104857600
                }
            },
            "Expiration": {
                "Days": 30
            }
        }
    ]
}

Explanation

• Rules: An array containing one or more of rules for bucket lifecycle management.
• ID: A unique identifier for the rule, such as "RULE001".
• Status: The state of the rule: Enabled or Disabled. Disabled rules have no effect
• Expiration: Specifies the expiration settings for objects.
  • Days: in RULE001, objects created 10 days ago or more are deleted; in RULE003 it’s 30 days
  • ExpiredObjectDeleteMarker: when all previous versions of an object are deleted and only a delete marker remains, dangling delete markers are purged.
• NoncurrentVersionExpiration: Defines how non-current versions are handled.
  • NewerNoncurrentVersions: Retains the 20 most recent non-current versions.
  • NoncurrentDays: non-current versions created more than X days ago are deleted. Mutually exclusive with NewerNoncurrentVersions.
• AbortIncompleteMultipartUpload: Configures the abortion of incomplete multipart uploads.
  • DaysAfterInitiation: Incomplete multipart uploads that started more than 7 days ago are aborted.
• Filter: Every rule requires a Filter field, even if it is empty. Narrows the scope of a rule to a subset of objects. When combining multiple conditions, they should be wrapped in an And block.
  • Prefix: Restricts the rule to objects whose key begins with the given string (e.g. "logs/").
  • ObjectSizeGreaterThan: Restricts the rule to objects strictly larger than the given size in bytes.
  • ObjectSizeLessThan: Restricts the rule to objects strictly smaller than the given size in bytes.

Setting Bucket Lifecycle Configuration

Any tooling that supports AWS S3 API for bucket lifecycle should be fully compatible.

$ aws s3api put-bucket-lifecycle-configuration --bucket yourbucket --lifecycle-configuration file://path/to/your/config.json

$ exo storage bucket lifecycle set sos://yourbucket path/to/your/config.json -z your-zone

Where:

• path/to/your/config.json is the actual path to your JSON configuration file.
• yourbucket is the name of the bucket on which to apply lifecycle rules.
• your-zone is the Exoscale zone where the bucket is located.

Scheduling

Bucket Lifecycle evaluation runs every day at midnight UTC. There might be some delay between evaluating a bucket’s configuration and deleting the respective objects.

Limitations

• Exoscale SOS only supports bucket lifecycle rules for object deletion/expiration.
• AWS S3 requires either a Filter or a Prefix field at the Rule level. Exoscale SOS only supports the Filter field, which is required to be present in all rules (but can be empty).