Auto-Deleting Old S3 Objects: A Complete Guide to S3 Lifecycle Rules

As your S3 bucket accumulates data over time, storage costs quietly compound — logs, temp files, and stale uploads from months ago continue to accrue charges even though they serve no operational purpose. S3 Lifecycle Rules are the native, zero-code solution to automate object expiration and reclaim those costs without manual intervention.

TL;DR

Step Action Method
1 Define a Lifecycle Rule on your bucket AWS Console, CLI, or SDK
2 Set filter scope (entire bucket or prefix/tag) Prefix string or object tag
3 Configure Expiration action at 30 days Days: 30 in rule config
4 S3 marks objects as expired at midnight UTC on day 30 Managed by S3 internally
5 S3 permanently deletes expired objects (usually within 24–48 hrs) Asynchronous S3 garbage collection

How S3 Lifecycle Rules Work: The Architecture

A Lifecycle Rule is a bucket-level policy that instructs S3 to perform automated actions on objects based on their age (days since creation) or date. For deletion, the key action is Expiration. S3 evaluates rules daily and transitions or expires objects that meet the criteria.

There are two distinct scenarios to understand:

  • Non-versioned buckets: S3 permanently deletes the object after the specified number of days.
  • Versioned buckets: S3 inserts a delete marker on the current version. You need a separate NoncurrentVersionExpiration action to purge older versions and an ExpiredObjectDeleteMarker cleanup to remove stale delete markers.
graph TD A["Object Uploaded to S3"] --> B["S3 Records Creation Timestamp"] B --> C["S3 Lifecycle Engine
Evaluates Daily at Midnight UTC"] C --> D{"Object Age >= 30 Days?"} D -- "No" --> E["No Action — Object Retained"] D -- "Yes" --> F{"Bucket Versioning Enabled?"} F -- "No" --> G["Permanent Delete
(No Charge for DELETE Request)"] F -- "Yes" --> H["Insert Delete Marker
on Current Version"] H --> I["Previous Version Becomes
Noncurrent Version"] I --> J{"NoncurrentVersionExpiration
Rule Configured?"} J -- "Yes" --> K["Noncurrent Versions Purged
After Configured Days"] J -- "No" --> L["Noncurrent Versions
Remain in Storage (Incur Cost)"] G --> M["Storage Cost Eliminated"] K --> M
  1. Object Upload: An object lands in S3. Its creation timestamp is recorded — this is the clock S3 uses for age calculations.
  2. Daily Evaluation: S3's internal lifecycle engine scans the bucket daily against all active rules.
  3. Age Check: If the object's age meets or exceeds the configured threshold (30 days), it is marked for expiration.
  4. Non-Versioned Path: The object is permanently deleted. A zero-cost DELETE request is issued by S3 on your behalf.
  5. Versioned Path: A delete marker is added. The previous current version becomes a noncurrent version. A separate NoncurrentVersionExpiration rule must be configured to purge those noncurrent versions.
  6. Cost Impact: Storage charges cease once the object is deleted. Note that S3 does not charge for lifecycle-initiated DELETE requests.
Real-World Analogy: Think of S3 Lifecycle Rules like a self-cleaning refrigerator. You set a rule once: "anything older than 30 days gets thrown out automatically." You don't open the fridge every day to check dates — the appliance handles it. Without this rule, leftovers from months ago silently occupy shelf space (and cost you money) indefinitely.

Implementation: 3 Methods

Method 1: AWS Management Console

  1. Navigate to S3 → Your Bucket → Management → Lifecycle rules → Create lifecycle rule.
  2. Enter a rule name (e.g., delete-objects-after-30-days).
  3. Under Filter type, choose:
    • Apply to all objects for the entire bucket, OR
    • Enter a Prefix (e.g., logs/) to scope the rule to a specific folder.
  4. Under Lifecycle rule actions, check Expire current versions of objects.
  5. Set Days after object creation to 30.
  6. If versioning is enabled, also check Permanently delete noncurrent versions of objects and set a days value.
  7. Review and Create rule.

Method 2: AWS CLI

This is the recommended approach for repeatable, auditable infrastructure. Save the JSON policy to a file and apply it with a single command.

🔽 Click to expand — lifecycle-policy.json (Non-Versioned Bucket) 
{
  "Rules": [
    {
      "ID": "delete-objects-after-30-days",
      "Status": "Enabled",
      "Filter": {
        "Prefix": ""
      },
      "Expiration": {
        "Days": 30
      }
    }
  ]
}

Apply the policy to your bucket:

aws s3api put-bucket-lifecycle-configuration \
  --bucket YOUR_BUCKET_NAME \
  --lifecycle-configuration file://lifecycle-policy.json

Verify the rule was applied:

aws s3api get-bucket-lifecycle-configuration \
  --bucket YOUR_BUCKET_NAME

Method 3: Versioned Bucket (Full Cleanup Policy)

If your bucket has versioning enabled, a simple expiration rule only adds delete markers. You need a comprehensive policy to fully purge all versions and clean up stale markers.

🔽 Click to expand — lifecycle-versioned.json (Versioned Bucket) 
{
  "Rules": [
    {
      "ID": "expire-current-versions",
      "Status": "Enabled",
      "Filter": {
        "Prefix": ""
      },
      "Expiration": {
        "Days": 30
      }
    },
    {
      "ID": "purge-noncurrent-versions",
      "Status": "Enabled",
      "Filter": {
        "Prefix": ""
      },
      "NoncurrentVersionExpiration": {
        "NoncurrentDays": 7
      }
    },
    {
      "ID": "cleanup-expired-delete-markers",
      "Status": "Enabled",
      "Filter": {
        "Prefix": ""
      },
      "Expiration": {
        "ExpiredObjectDeleteMarker": true
      }
    }
  ]
}

Apply it the same way:

aws s3api put-bucket-lifecycle-configuration \
  --bucket YOUR_BUCKET_NAME \
  --lifecycle-configuration file://lifecycle-versioned.json

Scoping Rules: Prefix vs. Tag Filters

Lifecycle rules support granular filtering so you don't have to apply a blanket deletion policy to an entire bucket.

Filter Type Use Case Example
No Filter (empty prefix) Apply to all objects in the bucket "Prefix": ""
Prefix Target a specific "folder" or path "Prefix": "logs/"
Object Tag Target objects by metadata tag {"Key": "temp", "Value": "true"}
Prefix + Tag (AND) Combine both conditions Objects under uploads/ AND tagged temp=true

IAM Permissions

To manage Lifecycle Rules via CLI or SDK, the calling IAM principal needs the following minimum permissions (least privilege):

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "ManageS3LifecycleRules",
      "Effect": "Allow",
      "Action": [
        "s3:GetLifecycleConfiguration",
        "s3:PutLifecycleConfiguration"
      ],
      "Resource": "arn:aws:s3:::YOUR_BUCKET_NAME"
    }
  ]
}

Note: S3 itself performs the actual object deletions internally — no additional IAM permissions are required for the deletion action triggered by the lifecycle rule.

Key Behavioral Notes

  • Timing precision: S3 calculates object age based on the creation date. Expiration is triggered at midnight UTC once the object reaches the configured age. Actual deletion may occur asynchronously within a day or two after the expiration date — you are not charged for storage during this window.
  • Minimum storage duration: Some S3 storage classes (e.g., S3 Glacier Instant Retrieval, S3 Standard-IA) have minimum storage duration charges. If you delete objects before that minimum, you may still be billed for the remainder. Always check the storage class pricing page before setting aggressive expiration windows.
  • No charge for lifecycle DELETE requests: S3 does not charge for DELETE requests initiated by lifecycle rules.
  • Rule limit: Pricing and limits vary; always check the official AWS documentation for the current maximum number of lifecycle rules per bucket.

Wrap-Up & Next Steps

S3 Lifecycle Rules are one of the highest-ROI configurations you can make on any active S3 bucket. A single JSON policy, applied once, eliminates an entire category of manual toil and prevents unbounded storage cost growth.

Recommended next steps:

  • Audit your existing buckets with aws s3api get-bucket-lifecycle-configuration to identify buckets with no rules.
  • Combine Lifecycle Rules with S3 Storage Class transitions (e.g., move to S3 Standard-IA after 30 days, then Glacier after 90 days, then delete after 365 days) for a tiered cost optimization strategy.
  • Enable S3 Storage Lens to get visibility into storage growth trends before and after applying rules.
  • 📖 Official Reference: AWS S3 Object Lifecycle Management Documentation

Glossary

Term Definition
Lifecycle Rule A bucket-level configuration that automates transitions or expiration of objects based on age or date.
Expiration Action A lifecycle action that permanently deletes an object (or adds a delete marker in versioned buckets) after a specified number of days.
Delete Marker A placeholder in a versioned bucket that marks an object as deleted without removing its previous versions from storage.
NoncurrentVersionExpiration A lifecycle action that permanently deletes non-current (older) versions of objects in a versioning-enabled bucket.
Prefix Filter A string used to scope a lifecycle rule to objects whose keys begin with a specific path segment (e.g., logs/).

Comments

Post a Comment

Popular posts from this blog

EC2 No Internet Access in Custom VPC: Attaching an Internet Gateway and Fixing Route Tables

Launching an EC2 instance in a custom VPC only to find it cannot reach the internet is one of the most common networking pitfalls in AWS — and it almost always comes down to two missing pieces: an Internet Gateway (IGW) and a correctly configured Route Table. This post walks you through the exact architecture, the root cause, and the precise steps to fix it. TL;DR Step Action Why It Matters 1 Create and attach an Internet Gateway to your VPC Without an IGW, there is no path between your VPC and the public internet 2 Add a 0.0.0.0/0 route pointing to the IGW in your subnet's Route Table The route table is the traffic director — without this entry, outbound packets have nowhere to go 3 Assign a Public IP or Elastic IP to the EC2 instance The IGW performs NAT only for instances with a public IP address 4 Verify Security Group allows outbound traffic Security Groups are stateful firewalls; outbound rules...
Read more

IAM User vs. IAM Role: Why Your EC2 Instance Should Never Use a User

Choosing between an IAM User and an IAM Role is one of the most consequential security decisions in AWS — get it wrong and you either expose long-lived credentials or block your application entirely. This post cuts through the confusion and gives you a definitive answer for the EC2-to-S3 access pattern. TL;DR — Quick Comparison Dimension IAM User IAM Role Identity type A person or a static service account An assumable identity for AWS services, users, or external principals Credential type Long-lived Access Key ID + Secret Access Key Short-lived, auto-rotating STS tokens (max 12 h by default) Credential rotation Manual — you must rotate keys yourself Automatic — AWS rotates them transparently Stored on EC2? Yes — in ~/.aws/credentials or env vars (risky) No — delivered via the In...
Read more

Lambda Infinite Loop with S3: How to Prevent Recursive Triggers

Writing a processed file back to the same S3 bucket that triggered your Lambda function creates a recursive loop — each output file fires a new invocation, which writes another file, ad infinitum. Left unchecked, this silently drains your AWS budget and exhausts Lambda concurrency limits within minutes. TL;DR Strategy Mechanism Best For Complexity Separate Output Prefix/Bucket S3 event filter on prefix Simple pipelines Low Object Metadata Check Lambda reads S3 object metadata before processing Single-bucket requirement Medium Separate Output Bucket Trigger only on source bucket Clean architectural separation Low DynamoDB Idempotency Guard Track processed ETag/key in DynamoDB Exactly-once guarantees Medium-High Understanding the Recursive Loop Before fixing the problem, you need to understand exactly why it happens. The S3 → Lambda trigger is an event-driven subscription: S3 publishes an ObjectCreat...
Read more