How to Use DynamoDB Streams to Trigger a Lambda Function on Data Changes

You have a DynamoDB table and you need to react to every write — inserts, updates, deletes — without polling or building a change-detection layer yourself. DynamoDB Streams with Lambda as an event source is the standard pattern for this, but the wiring has enough moving parts that it's easy to get the IAM wrong, pick the wrong stream view type, or misconfigure the event source mapping and wonder why your function never fires.

TL;DR: DynamoDB Streams + Lambda Event Source Mapping

Step	What You're Doing	Key Decision
1	Enable Streams on the table	Choose StreamViewType (NEW_IMAGE is the most common)
2	Grant Lambda permission to read the stream	Attach AWSLambdaDynamoDBExecutionRole or equivalent custom policy
3	Create the Event Source Mapping	Set StartingPosition, BatchSize, and bisect-on-error behavior
4	Validate end-to-end	Write a test item, check CloudWatch Logs for invocation

How DynamoDB Streams Works

DynamoDB Streams captures a time-ordered sequence of item-level modifications in a table. Each modification is written as a stream record to a shard. The stream is separate from the table itself — enabling it does not affect table read/write capacity. Stream records are retained for 24 hours.

Lambda does not poll the stream directly in the way you'd poll SQS. Instead, the Lambda service maintains an internal poller that reads from the stream shards on your behalf via the Event Source Mapping. Your function is invoked synchronously with a batch of records. If the invocation fails, Lambda retries the entire batch (by default) until it succeeds or the records expire — this is the behavior that makes error handling critical from day one.

graph LR Table["DynamoDB Table
PutItem / UpdateItem / DeleteItem"] --> Stream["DynamoDB Stream
Ordered shard log, 24hr retention"] Stream --> ESM["Event Source Mapping
Lambda-managed poller"] ESM --> Fn["Lambda Function
Batch of stream records"] Fn -->|"Success"| Commit["Batch committed
advance shard iterator"] Fn -->|"Error"| Retry["Retry batch
or bisect if enabled"] Retry -->|"Max retries exceeded"| DLQ["On-Failure Destination
SQS / SNS"]

1. DynamoDB Table — writes (PutItem, UpdateItem, DeleteItem) generate stream records.
2. DynamoDB Stream — ordered, sharded log of item changes, retained 24 hours.
3. Event Source Mapping — Lambda-managed poller; reads shards, batches records, invokes function.
4. Lambda Function — receives a batch of stream records; processes and returns success or throws.
5. On failure — Lambda retries the batch. Configure a DLQ or on-failure destination to avoid infinite retry loops on poison records.

StreamViewType: Pick This Carefully

The stream view type determines what data is included in each stream record. You set this once when enabling the stream; changing it requires disabling and re-enabling the stream, which resets the stream ARN.

StreamViewType	What's in the record	Use when
KEYS_ONLY	Only the key attributes	You only need to know which item changed, not what changed
NEW_IMAGE	Full item after the change	You need the current state to replicate or index
OLD_IMAGE	Full item before the change	Audit trails, undo operations
NEW_AND_OLD_IMAGES	Both before and after	Diff-based processing, change detection

Most teams default to NEW_IMAGE without thinking about it. If your function needs to detect what specifically changed (e.g., only react when a 'status' field transitions from 'pending' to 'active'), you need NEW_AND_OLD_IMAGES — otherwise you're reading a snapshot with no context about what the previous state was.

Step 1: Enable DynamoDB Streams on the Table

If you're creating a new table, you can enable streams in the same call. For an existing table, use update-table. The stream ARN is returned in the response — save it, you'll need it for the event source mapping.

Enable streams on an existing table:

aws dynamodb update-table \
  --table-name MyTable \
  --stream-specification StreamEnabled=true,StreamViewType=NEW_AND_OLD_IMAGES \
  --region us-east-1

Retrieve the stream ARN after enabling:

aws dynamodb describe-table \
  --table-name MyTable \
  --region us-east-1 \
  --query 'Table.LatestStreamArn' \
  --output text

The output looks like: arn:aws:dynamodb:us-east-1:123456789012:table/MyTable/stream/2024-01-15T10:00:00.000. The timestamp suffix is generated by DynamoDB — never construct this ARN manually.

Step 2: IAM — Grant Lambda Permission to Read the Stream

This is where most first-time setups get stuck. Lambda needs permissions on the stream (not the table) to read records. The execution role attached to your Lambda function must include the following actions against the stream ARN.

Minimum required permissions for stream consumption:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "dynamodb:GetRecords",
        "dynamodb:GetShardIterator",
        "dynamodb:DescribeStream",
        "dynamodb:ListStreams"
      ],
      "Resource": "arn:aws:dynamodb:us-east-1:123456789012:table/MyTable/stream/*"
    },
    {
      "Effect": "Allow",
      "Action": [
        "logs:CreateLogGroup",
        "logs:CreateLogStream",
        "logs:PutLogEvents"
      ],
      "Resource": "arn:aws:logs:us-east-1:123456789012:log-group:/aws/lambda/*"
    }
  ]
}

Attach this policy to the Lambda execution role. Alternatively, AWS provides the managed policy AWSLambdaDynamoDBExecutionRole which covers these stream permissions plus CloudWatch Logs — acceptable for non-production use, but the managed policy uses "Resource": "*" for stream actions, which is broader than least privilege.

Attach the managed policy via CLI if you prefer the faster path:

aws iam attach-role-policy \
  --role-name MyLambdaExecutionRole \
  --policy-arn arn:aws:iam::aws:policy/service-role/AWSLambdaDynamoDBExecutionRole

Step 3: Create the Event Source Mapping

The event source mapping is the glue between the stream and your function. Lambda's internal poller uses this configuration to know which stream to read, how to batch records, and where to start reading.

Key parameters to understand before running the command:

• --starting-position: TRIM_HORIZON reads from the oldest available record in the stream; LATEST reads only records written after the mapping is created. For a new integration on an active table, LATEST avoids replaying historical data.
• --batch-size: Number of stream records per Lambda invocation (1–10000). Start conservative — a large batch that causes a timeout will retry the entire batch.
• --bisect-batch-on-function-error: When enabled, Lambda splits a failing batch in half and retries each half separately. This isolates poison records without requiring a DLQ for every failure.
• --destination-config: Where to send records after all retries are exhausted. Use an SQS queue or SNS topic as an on-failure destination.

aws lambda create-event-source-mapping \
  --function-name MyStreamProcessor \
  --event-source-arn arn:aws:dynamodb:us-east-1:123456789012:table/MyTable/stream/2024-01-15T10:00:00.000 \
  --starting-position LATEST \
  --batch-size 100 \
  --bisect-batch-on-function-error \
  --maximum-retry-attempts 3 \
  --destination-config '{"OnFailure":{"Destination":"arn:aws:sqs:us-east-1:123456789012:MyDLQ"}}' \
  --region us-east-1

Verify the mapping was created and is in 'Enabled' state:

aws lambda list-event-source-mappings \
  --function-name MyStreamProcessor \
  --region us-east-1

The State field should show Enabled. If it shows Creating, wait a few seconds and re-run. If it shows Disabled or Problem, check the IAM permissions — that's the most common cause.

stateDiagram-v2 [*] --> Creating : create-event-source-mapping Creating --> Enabled : IAM OK, stream accessible Creating --> Problem : IAM denied or stream ARN invalid Enabled --> Disabling : manual disable or repeated failures Disabling --> Disabled : polling stopped Disabled --> Enabling : manual re-enable Enabling --> Enabled : ready Enabled --> Updating : parameter change Updating --> Enabled : update applied Problem --> Enabled : IAM corrected and mapping re-enabled

1. Enabled — normal operating state; Lambda is polling the stream.
2. Disabling / Disabled — manually triggered or due to repeated failures; polling stops.
3. Enabling — transitional state after creation or re-enable.
4. Problem — Lambda cannot access the stream, typically an IAM or stream ARN issue. Check the StateTransitionReason field in the mapping description.
5. Updating — transitional state when modifying mapping parameters.

Step 4: Write the Lambda Handler

Each invocation receives an event with a Records array. Each record has an eventName (INSERT, MODIFY, REMOVE) and a dynamodb object containing the stream view data.

🔽 Click to expand — Python Lambda handler example

import json

def lambda_handler(event, context):
    for record in event['Records']:
        event_name = record['eventName']  # INSERT, MODIFY, REMOVE
        dynamodb_record = record['dynamodb']

        if event_name == 'MODIFY':
            new_image = dynamodb_record.get('NewImage', {})
            old_image = dynamodb_record.get('OldImage', {})

            # DynamoDB stream records use DynamoDB JSON format
            # e.g., {'status': {'S': 'active'}, 'count': {'N': '5'}}
            new_status = new_image.get('status', {}).get('S')
            old_status = old_image.get('status', {}).get('S')

            if old_status != new_status:
                print(f'Status changed from {old_status} to {new_status}')
                # your business logic here

        elif event_name == 'INSERT':
            new_image = dynamodb_record.get('NewImage', {})
            print(f'New item inserted: {json.dumps(new_image)}')

        elif event_name == 'REMOVE':
            old_image = dynamodb_record.get('OldImage', {})
            print(f'Item deleted: {json.dumps(old_image)}')

    # Return without raising = success; Lambda commits the batch
    return {'statusCode': 200}

One thing that catches people off guard: stream records use DynamoDB JSON format, not plain JSON. The string value 'active' arrives as {"S": "active"}, and a number arrives as {"N": "5"} (as a string). If you're using Python, the boto3.dynamodb.types.TypeDeserializer utility can convert these to native Python types — but for simple cases, accessing the type key directly is fine.

Step 5: Validate End-to-End

Write a test item to the table and confirm the Lambda function was invoked. This catches IAM gaps and misconfigured stream view types before they become production surprises.

Write a test item:

aws dynamodb put-item \
  --table-name MyTable \
  --item '{"pk": {"S": "test-001"}, "status": {"S": "pending"}}' \
  --region us-east-1

Update it to trigger a MODIFY event:

aws dynamodb update-item \
  --table-name MyTable \
  --key '{"pk": {"S": "test-001"}}' \
  --update-expression 'SET #s = :new_status' \
  --expression-attribute-names '{"#s": "status"}' \
  --expression-attribute-values '{":new_status": {"S": "active"}}' \
  --region us-east-1

Check CloudWatch Logs for the function's log group (/aws/lambda/MyStreamProcessor). If you see the invocation log within 10–30 seconds, the pipeline is working. If nothing appears after a minute, check the event source mapping state first — not the function code.

Experience Signal: The Mapping That Looked Fine But Never Fired

A common misdiagnosis pattern: you create the event source mapping, the state shows Enabled, you write items to the table, and the Lambda function never fires. CloudWatch shows zero invocations. You check the function code, the trigger configuration in the console — everything looks correct.

The actual cause, in most cases: the Lambda execution role has permission on the table ARN, not the stream ARN. The IAM policy was written as:

"Resource": "arn:aws:dynamodb:us-east-1:123456789012:table/MyTable"

It should be:

"Resource": "arn:aws:dynamodb:us-east-1:123456789012:table/MyTable/stream/*"

The mapping stays Enabled because Lambda can describe the stream (that check passes), but GetRecords silently fails the authorization check at the shard level. No error surfaces in the function's log group because the function was never invoked. The only place this shows up is in the event source mapping's StateTransitionReason or in CloudTrail as an access denied event on dynamodb:GetRecords.

Check the mapping's state reason directly:

aws lambda get-event-source-mapping \
  --uuid <mapping-uuid> \
  --region us-east-1 \
  --query 'StateTransitionReason'

And verify via CloudTrail if needed:

aws cloudtrail lookup-events \
  --lookup-attributes AttributeKey=EventName,AttributeValue=GetRecords \
  --region us-east-1 \
  --max-results 10

Think of the stream as a separate resource from the table — like a changelog file sitting next to a database. Permissions on the database don't automatically extend to the changelog. The stream ARN is a distinct resource path, and IAM treats it as such.

Depth Signal: Shard Count and Concurrency Behavior

Lambda invokes one concurrent execution per shard in the stream. DynamoDB partitions your table based on throughput and item count, and each partition has one stream shard. This means your Lambda concurrency scales with your table's partition count — not with your batch size or item write rate.

If your table has 10 partitions, Lambda will run up to 10 concurrent executions of your stream processor function, regardless of how many items are written per second. This has two practical implications: first, your function must be safe to run concurrently (avoid shared mutable state); second, if you're hitting Lambda concurrency limits, the bottleneck is shard count, not batch size — increasing batch size won't help, and may actually increase the blast radius of a single failure.

Common Configuration Mistakes

• Wrong starting position on a busy table: Using TRIM_HORIZON on a table with months of stream history will replay every historical change through your function. Use LATEST unless you explicitly need historical replay.
• No bisect-on-error and no retry limit: A single malformed record can block an entire shard indefinitely until the 24-hour stream retention expires. Always set --maximum-retry-attempts and enable --bisect-batch-on-function-error.
• Assuming stream records are in table JSON format: They're in DynamoDB JSON. Plan your deserialization from the start.
• Changing StreamViewType on a live table: Requires disabling and re-enabling the stream, which generates a new stream ARN and breaks any existing event source mappings. Update the mapping to the new ARN after the change.

How to Use DynamoDB Streams: Wrap-Up and Next Steps

The DynamoDB Streams + Lambda pattern is straightforward once the IAM resource path is correct and the event source mapping parameters are set deliberately. The stream view type and error handling configuration are the two decisions that cause the most operational pain if deferred — set them before you go to production, not after your first incident.

For production workloads, also consider:

• Enabling Enhanced Fan-Out is not applicable here (that's Kinesis Data Streams); DynamoDB Streams does not support it natively.
• Using DynamoDB Streams with Kinesis Data Streams as an alternative — you can configure a table to replicate stream data to Kinesis, which gives you longer retention (up to 365 days vs. 24 hours) and Kinesis consumer options.
• Reviewing the official DynamoDB Streams and Lambda integration documentation for current limits and regional availability.

Glossary

Term	Definition
DynamoDB Stream	An ordered, time-limited log of item-level changes in a DynamoDB table, retained for 24 hours.
Event Source Mapping	A Lambda resource that connects a stream or queue to a Lambda function, managing polling and batching.
StreamViewType	Controls which item data (keys only, new image, old image, or both) is included in each stream record.
Starting Position	Determines where in the stream Lambda begins reading: TRIM_HORIZON (oldest) or LATEST (new records only).
Bisect on Error	Event source mapping behavior that splits a failing batch in half to isolate problematic records.

Related Posts

• 📄 DynamoDB Partition Key Design: Why Hot Partitions Kill Performance and How to Fix Them
• 📄 DynamoDB Capacity Modes: Provisioned vs. On-Demand — Which One Saves Money and Avoids Throttling?
• 📄 LSI vs GSI in DynamoDB: Choosing the Right Secondary Index