How to Connect RDS PostgreSQL from Lambda Using Python and Parameter Store

You've got a Python Lambda that needs to query a PostgreSQL RDS instance — straightforward on paper, but the moment you try to import psycopg2, Lambda throws a module-not-found error, and you realize the database password sitting in an environment variable is a security incident waiting to happen. This post walks through building a working psycopg2 Lambda layer, wiring up the VPC networking, and pulling credentials securely from AWS Systems Manager Parameter Store.

TL;DR: Lambda to RDS PostgreSQL Connection

Problem	Solution
`psycopg2` not available in Lambda runtime	Build a Lambda Layer with the compiled binary using Docker
Database password in plaintext env var	Store as SecureString in SSM Parameter Store, fetch at runtime
Lambda cannot reach RDS	Deploy Lambda in the same VPC, configure Security Group rules
Cold start latency from SSM calls	Cache the parameter value in the Lambda execution context

How the Connection Flow Works

Before touching any code, it's worth understanding what actually has to happen for a Lambda function to execute a query against RDS. There are three independent layers that all need to be correct simultaneously — runtime dependencies, network path, and credentials. A failure in any one of them produces a different error, and they don't always fail loudly.

sequenceDiagram participant Inv as Invoker participant Lam as Lambda Function participant SSM as SSM Parameter Store participant RDS as RDS PostgreSQL Inv->>Lam: Invoke Lam->>SSM: GetParameter (db_password) SSM-->>Lam: Decrypted password Lam->>Lam: psycopg2.connect() Lam->>RDS: TCP :5432 (TLS) RDS-->>Lam: Auth OK Lam->>RDS: SQL Query RDS-->>Lam: Result rows Lam-->>Inv: HTTP 200 + JSON

Lambda invocation — the function starts in an execution environment running Amazon Linux 2023.
SSM Parameter fetch — the function calls the SSM API to retrieve the database password. This requires an IAM permission and a network path to the SSM endpoint (VPC endpoint or NAT Gateway).
psycopg2 import — the layer provides the compiled C extension. Without a matching binary, the import fails at this point regardless of network or IAM state.
TCP connection to RDS — Lambda's ENI in the VPC initiates a connection to the RDS endpoint on port 5432. The RDS Security Group must allow inbound traffic from the Lambda Security Group.
PostgreSQL authentication — the credentials fetched from Parameter Store are used to authenticate. A wrong password fails here, not at step 2.

Step 1: Build the psycopg2 Lambda Layer

The Lambda runtime does not include psycopg2. The package has a C extension that must be compiled against the same architecture and OS as the Lambda execution environment. Python 3.11 Lambda functions run on Amazon Linux 2023 (x86_64). Installing psycopg2 on a macOS or Ubuntu dev machine and zipping it will not work — the compiled .so file will be incompatible.

The reliable approach is to compile inside a Docker container that matches the Lambda environment. The public.ecr.aws/lambda/python:3.11 image is the official AWS Lambda base image and is the correct target.

🔽 Click to expand: Build psycopg2 layer with Docker

# Create a working directory
mkdir psycopg2-layer && cd psycopg2-layer

# Build the compiled package inside the Lambda runtime container
docker run --rm \
  --platform linux/amd64 \
  -v "$(pwd):/out" \
  public.ecr.aws/lambda/python:3.11 \
  pip install psycopg2-binary \
    --target /out/python \
    --platform manylinux_2_28_x86_64 \
    --implementation cp \
    --python-version 3.11 \
    --only-binary=:all:

# Package the layer
zip -r psycopg2-layer.zip python/

The --platform manylinux_2_28_x86_64 flag tells pip to fetch a wheel compiled for the manylinux_2_28 ABI, which is compatible with Amazon Linux 2023. The --only-binary=:all: flag prevents pip from falling back to a source build if a matching wheel isn't found — if that happens, you want it to fail loudly here rather than silently produce an incompatible binary.

Publish the layer to your account:

aws lambda publish-layer-version \
  --layer-name psycopg2-python311 \
  --compatible-runtimes python3.11 \
  --compatible-architectures x86_64 \
  --zip-file fileb://psycopg2-layer.zip \
  --region us-east-1

Note the LayerVersionArn from the response — you'll attach it to the function in a later step.

Step 2: Store the Database Password in SSM Parameter Store

Hardcoding the password in an environment variable means it's visible in the Lambda console, in CloudTrail, and in any deployment artifact that captures the function configuration. SSM Parameter Store with the SecureString type encrypts the value at rest using KMS and keeps it out of the function configuration entirely.

aws ssm put-parameter \
  --name "/myapp/prod/db_password" \
  --value "your-actual-db-password" \
  --type SecureString \
  --region us-east-1

Store the non-sensitive connection details — host, port, database name, and username — as separate String parameters or as Lambda environment variables. Only the password needs SecureString treatment.

aws ssm put-parameter \
  --name "/myapp/prod/db_host" \
  --value "mydb.cluster-xxxxxxxxxxxx.us-east-1.rds.amazonaws.com" \
  --type String \
  --region us-east-1

aws ssm put-parameter \
  --name "/myapp/prod/db_user" \
  --value "app_user" \
  --type String \
  --region us-east-1

aws ssm put-parameter \
  --name "/myapp/prod/db_name" \
  --value "appdb" \
  --type String \
  --region us-east-1

Step 3: Configure VPC Networking and Security Groups

Lambda functions are not inside a VPC by default. To reach an RDS instance that isn't publicly accessible (which is the correct production configuration), the Lambda function must be deployed into the same VPC as the RDS instance, or a VPC with routing to it.

Create a dedicated Security Group for the Lambda function:

aws ec2 create-security-group \
  --group-name lambda-rds-sg \
  --description "Security group for Lambda functions accessing RDS" \
  --vpc-id vpc-0123456789abcdef0 \
  --region us-east-1

Add an inbound rule to the RDS Security Group that allows port 5432 traffic from the Lambda Security Group. Replace sg-lambda with the actual Security Group ID returned above, and sg-rds with the Security Group ID attached to your RDS instance:

aws ec2 authorize-security-group-ingress \
  --group-id sg-0rds1234567890abc \
  --protocol tcp \
  --port 5432 \
  --source-group sg-0lambda1234567890a \
  --region us-east-1

Referencing the Lambda Security Group ID as the source (rather than a CIDR block) means the rule tracks the Lambda ENIs automatically as they scale — you don't need to manage IP ranges.

The Lambda function also needs a network path to the SSM API. In a private subnet with no NAT Gateway, deploy a VPC endpoint for SSM:

aws ec2 create-vpc-endpoint \
  --vpc-id vpc-0123456789abcdef0 \
  --service-name com.amazonaws.us-east-1.ssm \
  --vpc-endpoint-type Interface \
  --subnet-ids subnet-0abc123 subnet-0def456 \
  --security-group-ids sg-0lambda1234567890a \
  --private-dns-enabled \
  --region us-east-1

Without either a NAT Gateway or a VPC endpoint for SSM, the boto3 call to fetch the parameter will time out — and the timeout error looks identical to a network misconfiguration on the RDS side, which is a common misdiagnosis.

Step 4: Create the IAM Execution Role

The Lambda execution role needs permissions to write logs, read the SSM parameters, and use the KMS key that encrypts the SecureString parameter. The ssm:GetParameter action requires the KMS kms:Decrypt permission when the parameter type is SecureString.

🔽 Click to expand: IAM policy for Lambda execution role

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AllowCloudWatchLogs",
      "Effect": "Allow",
      "Action": [
        "logs:CreateLogGroup",
        "logs:CreateLogStream",
        "logs:PutLogEvents"
      ],
      "Resource": "arn:aws:logs:us-east-1:123456789012:log-group:/aws/lambda/*"
    },
    {
      "Sid": "AllowSSMParameterRead",
      "Effect": "Allow",
      "Action": "ssm:GetParameter",
      "Resource": [
        "arn:aws:ssm:us-east-1:123456789012:parameter/myapp/prod/db_password",
        "arn:aws:ssm:us-east-1:123456789012:parameter/myapp/prod/db_host",
        "arn:aws:ssm:us-east-1:123456789012:parameter/myapp/prod/db_user",
        "arn:aws:ssm:us-east-1:123456789012:parameter/myapp/prod/db_name"
      ]
    },
    {
      "Sid": "AllowKMSDecrypt",
      "Effect": "Allow",
      "Action": "kms:Decrypt",
      "Resource": "arn:aws:kms:us-east-1:123456789012:key/your-kms-key-id"
    },
    {
      "Sid": "AllowVPCAccess",
      "Effect": "Allow",
      "Action": [
        "ec2:CreateNetworkInterface",
        "ec2:DescribeNetworkInterfaces",
        "ec2:DeleteNetworkInterface"
      ],
      "Resource": "*"
    }
  ]
}

The VPC-related EC2 actions (CreateNetworkInterface, DescribeNetworkInterfaces, DeleteNetworkInterface) are required for Lambda to attach ENIs in your VPC. These actions require "Resource": "*" — the Service Authorization Reference confirms that resource-level restrictions are not supported for these specific actions.

Step 5: Write the Lambda Function

The function fetches credentials from SSM on first invocation and caches them in the module-level scope. Subsequent invocations within the same execution environment reuse the cached values, avoiding an SSM API call on every request. The database connection is also held at module level for connection reuse across warm invocations.

🔽 Click to expand: Full Lambda function code

import os
import json
import boto3
import psycopg2
from psycopg2.extras import RealDictCursor

# Module-level cache — persists across warm invocations
_db_config = None
_connection = None

def get_db_config():
    """Fetch DB config from SSM Parameter Store. Cached after first call."""
    global _db_config
    if _db_config is not None:
        return _db_config

    ssm = boto3.client('ssm', region_name=os.environ['AWS_REGION'])

    def get_param(name, with_decryption=False):
        response = ssm.get_parameter(
            Name=name,
            WithDecryption=with_decryption
        )
        return response['Parameter']['Value']

    _db_config = {
        'host':     get_param('/myapp/prod/db_host'),
        'user':     get_param('/myapp/prod/db_user'),
        'dbname':   get_param('/myapp/prod/db_name'),
        'password': get_param('/myapp/prod/db_password', with_decryption=True),
        'port':     5432,
        'connect_timeout': 5,
        'sslmode':  'require'
    }
    return _db_config


def get_connection():
    """Return a live DB connection, reconnecting if the cached one is closed."""
    global _connection
    try:
        if _connection is None or _connection.closed != 0:
            config = get_db_config()
            _connection = psycopg2.connect(**config)
    except psycopg2.OperationalError as e:
        # Force a fresh connection on next call if this one failed
        _connection = None
        raise e
    return _connection


def lambda_handler(event, context):
    try:
        conn = get_connection()
        with conn.cursor(cursor_factory=RealDictCursor) as cur:
            cur.execute('SELECT id, name FROM users LIMIT 10;')
            rows = cur.fetchall()

        return {
            'statusCode': 200,
            'body': json.dumps([dict(row) for row in rows])
        }
    except psycopg2.OperationalError as e:
        return {
            'statusCode': 503,
            'body': json.dumps({'error': 'Database connection failed', 'detail': str(e)})
        }
    except Exception as e:
        return {
            'statusCode': 500,
            'body': json.dumps({'error': str(e)})
        }

The sslmode: require setting enforces TLS for the connection to RDS. RDS PostgreSQL supports SSL by default, and requiring it prevents the connection from silently falling back to plaintext if the server configuration changes.

One thing worth noting: _connection.closed != 0 checks the psycopg2 connection state. A value of 0 means the connection is open. Any non-zero value indicates it was closed, either by the server or by a prior exception. Without this check, a stale connection from a previous warm invocation will raise an error on the first query rather than reconnecting transparently.

Step 6: Deploy the Lambda Function

Package the function code and create the Lambda function with the VPC configuration and the psycopg2 layer attached:

zip function.zip lambda_function.py

aws lambda create-function \
  --function-name rds-query-function \
  --runtime python3.11 \
  --role arn:aws:iam::123456789012:role/lambda-rds-execution-role \
  --handler lambda_function.lambda_handler \
  --zip-file fileb://function.zip \
  --timeout 30 \
  --memory-size 256 \
  --vpc-config SubnetIds=subnet-0abc123,subnet-0def456,SecurityGroupIds=sg-0lambda1234567890a \
  --layers arn:aws:lambda:us-east-1:123456789012:layer:psycopg2-python311:1 \
  --region us-east-1

Verify the function configuration looks correct before invoking it:

aws lambda get-function-configuration \
  --function-name rds-query-function \
  --region us-east-1 \
  --query '{Runtime:Runtime,VpcConfig:VpcConfig,Layers:Layers}'

Diagnosing Connection Failures — The Misdiagnosis Pattern

Here's a failure pattern that shows up repeatedly: the Lambda function times out after 30 seconds with no useful error message. The instinct is to check the RDS Security Group inbound rules — and they look correct. The subnet routing looks correct. The function is in the right VPC.

The actual cause: the SSM GetParameter call is what's timing out, not the RDS connection. The function never reaches the psycopg2.connect() call. Because the SSM call happens first and the timeout is a generic network timeout, the error surface looks identical to an RDS connectivity problem.

Check CloudWatch Logs for the specific line that times out:

aws logs filter-log-events \
  --log-group-name /aws/lambda/rds-query-function \
  --filter-pattern "ERROR" \
  --region us-east-1

If the log shows the error occurring before any psycopg2 output, the SSM endpoint is unreachable. Verify the VPC endpoint for SSM exists and its Security Group allows HTTPS (port 443) inbound from the Lambda Security Group:

aws ec2 describe-vpc-endpoints \
  --filters "Name=service-name,Values=com.amazonaws.us-east-1.ssm" \
            "Name=vpc-id,Values=vpc-0123456789abcdef0" \
  --query 'VpcEndpoints[*].{State:State,SubnetIds:SubnetIds}' \
  --region us-east-1

The fix is either adding the SSM VPC endpoint (shown in Step 3) or ensuring the private subnet has a NAT Gateway route. Once SSM resolves, the RDS connection errors — if any remain — become visible as distinct psycopg2.OperationalError messages with the actual PostgreSQL error text.

graph TD Start["Lambda times out"] --> Q1{"Does CloudWatch log
show psycopg2 error?"} Q1 -->|No error before timeout| SSMIssue["SSM endpoint unreachable
Check VPC endpoint or NAT"] Q1 -->|psycopg2.OperationalError| Q2{"Error message content?"} Q2 -->|'could not connect'| SGIssue["RDS Security Group
missing inbound rule on 5432"] Q2 -->|'password authentication failed'| CredsIssue["Wrong password in
Parameter Store"] Q2 -->|'SSL connection required'| SSLIssue["sslmode not set
or RDS SSL enforced"]

Wrap-Up and Next Steps for RDS PostgreSQL Lambda Connections

The working setup requires four things to be correct at once: a compiled psycopg2 binary matching the Amazon Linux 2023 runtime, SSM parameters with the right IAM permissions, VPC networking with Security Group rules allowing port 5432, and a network path to the SSM API endpoint. Missing any one of them produces a timeout or import error that doesn't directly identify the missing piece.

From here, consider these operational improvements:

Connection pooling — for high-concurrency workloads, Lambda's per-instance connection model can exhaust RDS connection limits. RDS Proxy sits between Lambda and RDS and pools connections across function instances.
IAM database authentication — instead of a static password in Parameter Store, RDS supports IAM-based authentication tokens generated at runtime. This eliminates the stored credential entirely.
Parameter Store vs. Secrets Manager — Secrets Manager adds automatic rotation for database credentials. If the password rotation requirement exists, Secrets Manager is the appropriate service rather than Parameter Store.

Official references: Lambda VPC configuration, SSM Parameter Store, RDS IAM database authentication.

Glossary

Term	Definition
Lambda Layer	A ZIP archive containing libraries or dependencies that Lambda extracts into the `/opt` directory of the execution environment, making them available to the function code.
SecureString	An SSM Parameter Store parameter type that encrypts the stored value using an AWS KMS key. Requires `WithDecryption=True` and `kms:Decrypt` permission to retrieve the plaintext value.
ENI (Elastic Network Interface)	A virtual network interface that Lambda attaches to your VPC subnet when the function is configured with VPC access. This is what gives the function a private IP address and network path to RDS.
VPC Endpoint (Interface)	A private connection between your VPC and an AWS service API (such as SSM) that routes traffic through the AWS network without requiring a NAT Gateway or internet gateway.
manylinux	A Linux ABI compatibility standard used by Python wheels. `manylinux_2_28` corresponds to glibc 2.28, which is compatible with Amazon Linux 2023.

Search This Blog

SW BBANG