Automatically Control CloudWatch Log Group Retention With a Lambda Function

10 minutes ago

5 min read

By default, most AWS services that write to Amazon CloudWatch Logs — AWS Lambda, Amazon ECS, Amazon API Gateway, AWS CodeBuild, and many others — create their log groups with a retention setting of Never expire. Over time, this means your account accumulates hundreds or thousands of log groups full of data nobody reads anymore, and AWS keeps charging you for storing all of it.

In this teratip, I'll show you how to solve this with a small serverless solution: a Lambda function that sets a retention policy on every CloudWatch Log Group in your account. It's triggered two ways: reactively, the moment a new log group is created, and periodically, to catch anything that slipped through (or existed before this was set up). Both triggers are implemented with Amazon EventBridge.

Let's get started!

Step 1 - Create the IAM permissions policy

Before creating the Lambda function, let's create the custom IAM policy it will need on top of its basic execution permissions.

Open the IAM service in the AWS Console.
Under Access management, go to Policies and click Create policy.
Switch to the JSON tab and paste the following policy:


{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "logs:DescribeLogGroups",
        "logs:PutRetentionPolicy"
      ],
      "Resource": "*"
    },
    {
      "Effect": "Allow",
      "Action": [
        "logs:CreateLogStream",
        "logs:PutLogEvents"
      ],
      "Resource": "*"
    }
  ]
}

The first statement is what lets the function list every log group in the account and change its retention. The second statement lets it write its own execution logs.

Click Next, name the policy set-cw-log-group-retention-lambda-policy, add an optional description, and click Create policy.

Step 2 - Create the IAM Role for the Lambda function

In IAM, go to Roles and click Create role.
Under Trusted entity type, choose AWS service, and under Use case select Lambda. Click Next.
Attach two policies: the AWS managed policy AWSLambdaBasicExecutionRole (covers logs:CreateLogGroup for the function's own log group) and the set-cw-log-group-retention-lambda-policy you just created. Click Next.
Name the role set-cw-log-group-retention-role and click Create role.

The trust policy AWS generates for this role looks like this:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "lambda.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

Step 3 - Create the Lambda function

Open the Lambda service and click Create function.
Choose Author from scratch.
Set the function name to set-cw-log-group-retention.
Choose Python 3.14 as the runtime.
Under Permissions, expand Change default execution role, select Use an existing role, and choose set-cw-log-group-retention-role.
Click Create function.

Once the function is created, replace the contents of lambda_function.py in the inline code editor with the following code, then click Deploy:

import boto3
import os

logs = boto3.client('logs')
RETENTION_DAYS = int(os.environ.get("RETENTION_DAYS", "30"))  # 30 is the default value

def lambda_handler(event, context):
    # Reactive execution (a log group was just created)
    if "detail" in event and "requestParameters" in event["detail"]:
        log_group = event["detail"]["requestParameters"]["logGroupName"]
        set_retention(log_group)
        return {"status": "updated", "logGroup": log_group}

    # Periodic execution (check every log group in the account)
    paginator = logs.get_paginator('describe_log_groups')
    for page in paginator.paginate():
        for lg in page["logGroups"]:
            set_retention(lg["logGroupName"])

    return {"status": "completed"}

def set_retention(log_group_name):
    logs.put_retention_policy(
        logGroupName=log_group_name,
        retentionInDays=RETENTION_DAYS
    )
    print(f"Retention set for {log_group_name}")

The same function handles both triggers. When EventBridge invokes it reactively for a single newly-created log group, the event includes a detail.requestParameters.logGroupName field, so the function just updates that one log group. When it's invoked on the periodic schedule, that field isn't present, so it falls through to paginating over describe_log_groups and updating every log group in the account.

Now let's configure it:

Go to the Configuration tab > Environment variables > Edit > Add environment variable, and add RETENTION_DAYS with the number of days you want to keep your logs, for example 30.
The function defaults to 128 MB of memory, 512 MB of ephemeral storage, and a 3 second timeout, which is enough for accounts with a moderate number of log groups. If your account has a very large number of log groups, go to Configuration > General configuration > Edit and increase the Timeout, since the periodic run has to page through all of them in a single invocation.

Step 4 - Test the function manually

Go to the Test tab.
Create a new test event (any name, the default {} template works — this simulates the periodic execution path, since it has no detail.requestParameters).
Click Test.

Check the execution result and, in the Monitor tab, the CloudWatch Logs output — you should see a Retention set for <log group name> line for every log group in the account.

Step 5 - Trigger the function reactively on new log groups

This first EventBridge rule fires the moment a new log group is created anywhere in the account, using the CloudTrail management events that AWS delivers to the default event bus automatically (no need to create a CloudTrail trail).

Open the Amazon EventBridge service.
In the left navigation pane, click Rules, then click Create rule. Use the Advanced builder.
Name it on-log-group-create, choose the default event bus, and set the rule type to Event pattern.
Choose Custom pattern (JSON editor) and paste:

{
  "source": ["aws.logs"],
  "detail-type": ["AWS API Call via CloudTrail"],
  "detail": {
    "eventSource": ["logs.amazonaws.com"],
    "eventName": ["CreateLogGroup"]
  }
}

Under Target, choose AWS service > Lambda function, and select set-cw-log-group-retention.
Click through the remaining steps and Create rule.

Step 6 - Add a periodic check with Amazon EventBridge

The second rule runs on a schedule, as a safety net that re-checks every log group — useful for anything created before this was set up, or for any edge case the reactive rule might miss.

In the left navigation pane, click Schedules, then click Create schedule.
Name it periodic-log-group-retention-check, choose the default event bus.
Change the Schedule pattern to Recurring schedule and Schedule type to Rate-based schedule. Set the rate expression; for example rate(1 day) (adjust the frequency to whatever makes sense for your account).
Set Flexible time window to Off.
Under Select target, choose AWS Lambda, and select set-cw-log-group-retention again.
Click through the remaining steps and Create rule.

Step 7 - Verify everything is working

Create a throwaway log group (or wait for one to be created naturally, e.g. by deploying any Lambda function) and confirm the reactive rule fires within a minute or two. Then open CloudWatch > Log groups and check the Retention column across your account — log groups that previously showed Never expire should now show the value you configured in RETENTION_DAYS.

That's all! From now on, every log group in your account — including the ones created by services you deploy next month — will automatically get a sane retention policy, without anyone having to remember to set it manually.

Ignacio Rubio

Cloud Engineer