AWS S3 Cost Optimization: Automate Cleanup of Abandoned Buckets

🙋‍♂️ Introduction

Hey folks! I'm Ankit, working as a Kubernetes Engineer at CirrOps and a newly minted AWS Community Builder. I’m passionate about Cloud and Container technologies. But today, I'm switching gears to talk about something equally important - Cost Optimization, more specifically about AWS S3 buckets.

📚 Synopsis

• In most projects, we spin up a large number of AWS S3 buckets while developing an app or testing, or running in production. It's often the go-to solution for media-related applications. But the problem here is, when testing applications or in production environments, we end up creating tons of buckets that eventually get abandoned by users, testers, employees, or applications.
• These buckets not only lead to unnecessary costs but also create a management mess. Deleting those buckets by hand is boring, easy to mess up, and frankly, the last thing anyone wants to do on a Friday evening. So I spent a weekend putting together an event‑driven cleanup workflow that removes or retains buckets automatically for us

🤩 TL;DR (The “30‑Second” Version)

💡High-Level Solution

Imagine every bucket in your account carries a simple tag:

• autoDelete=True or autoDelete=False

A scheduled job or script runs daily and checks each bucket’s last access date. If a bucket hasn’t seen any activity in the last 30 days and its tag is:

• True → Automatically delete the bucket and all of its contents.
• False → Leave the bucket alone.
• Missing or invalid → Send the bucket owner an email with two options:
  1. Keep it (tag it autoDelete=False)
  2. Delete it (confirm deletion)

This approach makes sure that we only remove truly abandoned buckets and gives users one-click control over exceptions.

✅ Prerequisites

• 📍 An AWS account with administrative privileges
• 📍 Basic familiarity with Python and Boto3
• 📍 Understanding of AWS Lambda, EventBridge, SNS, API Gateway, and DynamoDB

📦 List of AWS services

• 🪣 Amazon S3
• 📨 AWS SNS
• ⛅ CloudTrail
• 🚀 DynamoDB
• 🖥️ AWS Lambda
• 🔄 AWS EventBridge
• 🌏 Amazon API Gateway

🎯 Architecture

Let's dive into how this all works together.

A key question you might have is: How can we determine when a bucket was last accessed? There are several approaches, but I personally prefer using a combination of CloudTrail, Lambda, and DynamoDB

Here's the breakdown of the architecture:

⛅ CloudTrail:

• AWS doesn’t expose LastAccessed for a bucket out of the box, but CloudTrail records every object‑level API call.
• I configured a data‑event trail (yes, it costs a little extra) and pointed it at the logging bucket s3EventLoggingStorage
• Each time an object is listed, uploaded, downloaded, or deleted, the event lands in EventBridge and triggers s3EventLogger, which writes:

{
    "BucketName" xyz-terraform: ,
    "EventDateTime": 2024-07-24T00:41:03Z ,
    "EventName": ListObjects ,
    "EventDate": 2024-07-24,
    "EventTime": 00:41:03Z,
    "Status": Active
}

🔄 EventBridge:

EventBridge has two important components in our solution:

• Rule: Created a rule in the event bus with an event pattern that captures every S3 bucket event (except for our CloudTrail logging bucket s3EventLoggingStorage)
  • This rule triggers a Lambda function named s3EventLogger when the pattern matches.
  • The event pattern configuration looks like this:

{
  "source": ["aws.s3"],
  "detail-type": ["AWS API Call via CloudTrail"],
  "detail": {
    "eventSource": ["s3.amazonaws.com"],
    "eventName": ["ListObjects", "ListObjectVersions", "PutObject", "GetObject", "HeadObject", "CopyObject", "GetObjectAcl", "PutObjectAcl", "CreateMultipartUpload", "ListParts", "UploadPart", "CompleteMultipartUpload", "AbortMultipartUpload", "UploadPartCopy", "RestoreObject", "DeleteObject", "DeleteObjects", "GetObjectTorrent", "SelectObjectContent", "PutObjectLockRetention", "PutObjectLockLegalHold", "GetObjectLockRetention", "GetObjectLockLegalHold"],
    "requestParameters": {
      "bucketName": [{
        "anything-but": ["s3EventLoggingStorage"]
      }]
    }
  }
}

• Scheduler: Executes the s3Scanner Lambda function daily at 9:00 AM

🖥️ Lambda

• We have a total of 3 buckets working together:
• 1) s3EventLogger:
  • Primary purpose: Record when any bucket receives any kind of API call
  • Collects event data such as BucketName, EventDate etc..
  • Stores this data in a DynamoDB table called s3DateLogger
• 2) s3Scanner:
  • Triggered every day at 9:00 AM UTC via an EventBridge schedule.
  • Lists all buckets in the account.
  • For each bucket:
    • Fetch the last access date from DynamoDB (if no record exists in DynamoDB, then create an entry and set the date to 15 days in the future as a grace period - useful if you have older buckets)
    • Calculate days since the last access
    • Retrieve the autoDelete tag (if any)
      • Decision logic:
        • autoDelete=True && days ≥ 30 → Delete all object versions, then the bucket
        • autoDelete=False && days ≥ 30 → Skip deletion
    • No valid tag && days ≥ 30 → Publish a notification to SNS
• 3) userHandler:
  • It will be triggered by the API Gateway
  • It either deletes the bucket or adds the autoDelete=False tag based on user choice

📧 SNS:

• SNS sends an email to the user with two links:
  • Keep It: Calls the API Gateway endpoint ?bucket_name={bucket_name}&action=keep
  • Delete It: Calls the API Gateway endpoint ?{bucket_name}&action=delete

🌏 API Gateway:

• Provides endpoints for users to respond to notifications, triggering the userHandler Lambda

🧑‍💻 Source Code

The GitHub repository contains:

• Link: Source code for all three Lambda functions
• Note: This was a weekend project, so the code has room for improvement. If you’re a beginner, try extending it to fit your own environment. 😊

🙌 Conclusion

We explored how to automate S3 cleanup with minimal human intervention and potentially save a bunch of dollars in the process. Give it a try over the weekend or at your convenience. I hope you enjoyed this blog as much as I enjoyed creating it.

And that’s a wrap! 🙌🥂

If you liked my work, please message me on LinkedIn with "Hi + your country name"

• 🙋‍♂️ Ankit Jodhani

📨 Reach me at ankitjodhani1903@gmail.com