How to Create a Lambda Layer for Shared Python Libraries

You have three Lambda functions, each packaging the same requests library independently. Every deploy zips the same 15 MB, every cold start loads the same bytes, and when you need to pin a new version you touch three functions instead of one. Lambda Layers exist precisely to break this pattern — one versioned artifact, attached to as many functions as you need.

TL;DR: Lambda Layer Creation at a Glance

StepWhat HappensKey Detail
1. Build package locallypip installs into a directory matching the Lambda runtime pathMust be python/ subdirectory
2. Zip the directoryCreates the layer artifactZip must contain python/ at root
3. Publish layer versionLambda stores the artifact and returns a versioned ARNEach publish creates a new immutable version
4. Attach to functionsFunction config references the layer ARNUp to 5 layers per function; combined unzipped size limit applies

How Lambda Layers Work

Before touching the CLI, the directory structure is the part most engineers get wrong on the first attempt. Lambda extracts layer contents into /opt inside the execution environment. For Python runtimes, the interpreter automatically includes /opt/python and /opt/python/lib/pythonX.Y/site-packages in sys.path. If your zip does not place libraries under python/ at the root, the import will fail at runtime with a plain ModuleNotFoundError — no other indication.

graph LR A["Build Host
pip install → python/"] --> B["zip requests_layer.zip"] B --> C["aws lambda publish-layer-version"] C --> D["Layer Version ARN
:layer:requests-layer:1"] D --> E["update-function-configuration
--layers ARN"] E --> F1["Function One
/opt/python/requests"] E --> F2["Function Two
/opt/python/requests"] E --> F3["Function Three
/opt/python/requests"]
  1. Build host — you run pip install targeting a local python/ directory, then zip it.
  2. Layer publish — Lambda stores the zip as an immutable version. The ARN includes the version number.
  3. Function attach — at invocation, Lambda merges the layer contents into /opt before your handler runs.
  4. Runtime path — Python finds requests under /opt/python exactly as if it were in the deployment package.

Layers are region-specific. A layer published in us-east-1 cannot be referenced by a function in eu-west-1. You must publish separately per region if your functions span regions.

Step 1: Build the Python Package Directory

The directory layout must match what Lambda expects under /opt. Use a clean working directory to avoid accidentally bundling your local virtualenv or test dependencies.

mkdir -p layer_build/python
pip install requests \
  --target layer_build/python \
  --platform manylinux2014_x86_64 \
  --implementation cp \
  --python-version 3.12 \
  --only-binary=:all: \
  --upgrade

The --platform and --only-binary flags matter when you are building on macOS or Windows. Lambda runs on Amazon Linux 2, and some packages ship compiled C extensions. Installing without these flags on a non-Linux host can produce binaries that fail silently at import time — the library loads, but crashes on the first native call. If you are on Linux x86_64 and your packages are pure Python, you can omit those flags.

Think of --platform manylinux2014_x86_64 as cross-compiling for the target OS. You are telling pip to fetch the Linux wheel even though your local machine is not Linux.

Step 2: Zip the Layer Artifact

The zip must contain the python/ directory at its root — not a parent directory wrapping it. This is the most common structural mistake.

cd layer_build
zip -r ../requests_layer.zip python/
cd ..

Verify the structure before publishing. A malformed zip wastes a publish operation and leaves you debugging a ModuleNotFoundError in CloudWatch Logs instead of catching it here.

unzip -l requests_layer.zip | head -20

The output should show paths starting with python/requests/, not layer_build/python/requests/ or just requests/.

Step 3: Publish the Layer Version

Publishing uploads the zip and returns a versioned ARN. Each call to publish-layer-version creates a new immutable version — existing versions are never modified.

aws lambda publish-layer-version \
  --layer-name requests-layer \
  --description "requests library for Python 3.12" \
  --zip-file fileb://requests_layer.zip \
  --compatible-runtimes python3.12 \
  --region us-east-1

The response includes the full versioned ARN. Copy it — you need it for the next step.

{
    "LayerArn": "arn:aws:lambda:us-east-1:123456789012:layer:requests-layer",
    "LayerVersionArn": "arn:aws:lambda:us-east-1:123456789012:layer:requests-layer:1",
    "Version": 1,
    ...
}

The LayerVersionArn — not the LayerArn — is what you attach to functions. Attaching the unversioned ARN is not valid.

Step 4: Attach the Layer to Your Lambda Functions

Each function needs an explicit update to reference the layer. If a function already has layers attached, update-function-configuration replaces the entire list — include all layers you want, not just the new one.

aws lambda update-function-configuration \
  --function-name my-function-one \
  --layers arn:aws:lambda:us-east-1:123456789012:layer:requests-layer:1 \
  --region us-east-1

Repeat for each function. If a function already has other layers attached, list all of them in the --layers argument as space-separated ARNs.

aws lambda update-function-configuration \
  --function-name my-function-two \
  --layers arn:aws:lambda:us-east-1:123456789012:layer:requests-layer:1 \
  --region us-east-1

aws lambda update-function-configuration \
  --function-name my-function-three \
  --layers arn:aws:lambda:us-east-1:123456789012:layer:requests-layer:1 \
  --region us-east-1

Step 5: Remove the Library from Individual Function Packages

Attaching the layer does not automatically exclude the library from your function's deployment package. If requests is still bundled in the function zip, it will shadow the layer version — whichever appears first in sys.path wins. Remove it from each function's package and redeploy.

After redeployment, verify the layer is actually resolving the import:

aws lambda invoke \
  --function-name my-function-one \
  --payload '{"test": true}' \
  --region us-east-1 \
  response.json
cat response.json

If you want to explicitly confirm the path at runtime, add a temporary debug line to your handler:

import requests, inspect
print(inspect.getfile(requests))  # should print /opt/python/requests/__init__.py

IAM: Layer Permissions for Cross-Account or Org-Wide Sharing

By default, a layer version is private to the account that published it. Functions in the same account can use it without any additional permission. If you need to share the layer across accounts or an AWS Organization, add a layer permission explicitly.

🔽 Click to expand: cross-account layer permission 
aws lambda add-layer-version-permission \
  --layer-name requests-layer \
  --version-number 1 \
  --statement-id allow-org-use \
  --action lambda:GetLayerVersion \
  --principal "*" \
  --organization-id o-exampleorgid11 \
  --region us-east-1

For same-account usage, no additional policy is required. The function's execution role does not need explicit permission to use a layer in the same account.

The Failure Pattern You Will Hit at Least Once

The symptom: function deploys cleanly, layer is attached, but import requests throws ModuleNotFoundError at runtime. CloudWatch shows nothing useful beyond the exception itself.

The misdiagnosis: most engineers immediately suspect the layer publish failed or the ARN is wrong. Both are easy to verify and usually fine.

The actual cause: the zip was created from the wrong working directory. Running zip -r requests_layer.zip layer_build/python/ instead of cd layer_build && zip -r ../requests_layer.zip python/ produces a zip where paths start with layer_build/python/. Lambda extracts this to /opt/layer_build/python/, which is not on sys.path. The layer version publishes successfully, attaches successfully, and fails silently until the first invocation.

The fix is the unzip -l verification in Step 2 — run it before every publish, not after a failed invocation.

graph TD A["zip created"] --> B{"Root path in zip?"} B -->|"layer_build/python/..."| C["Extracted to /opt/layer_build/python/"] B -->|"python/..."| D["Extracted to /opt/python/"] C --> E["NOT on sys.path
ModuleNotFoundError"] D --> F["On sys.path
import succeeds"]
  1. Wrong zip root — paths start with layer_build/python/, extracted to /opt/layer_build/python/, not on sys.path.
  2. Correct zip root — paths start with python/, extracted to /opt/python/, automatically on sys.path.
  3. The publish and attach steps succeed in both cases — the failure only surfaces at import time.

Updating the Layer When You Need a New Library Version

Layer versions are immutable. To update requests to a newer version, repeat Steps 1–3 to publish a new layer version. The new version gets an incremented version number in its ARN. Then update each function to reference the new ARN.

This is where the operational advantage compounds: updating three functions to point at a new layer version is three update-function-configuration calls with a single changed ARN. Compare that to rebuilding and redeploying three separate function packages.

# Verify which layer version each function currently uses
aws lambda get-function-configuration \
  --function-name my-function-one \
  --region us-east-1 \
  --query 'Layers'

Wrap-Up: Creating Lambda Layers for Shared Python Libraries

The mechanics of creating a Lambda Layer for shared Python libraries reduce to four operations: build into the correct directory structure, zip from the right working directory, publish to get a versioned ARN, and attach that ARN to each function. The directory layout under python/ is the only structural constraint that cannot be recovered from at runtime — verify it with unzip -l before publishing.

For teams managing multiple Lambda functions with shared dependencies, layers eliminate redundant packaging and give you a single place to update library versions. The immutability of layer versions also means you can pin functions to a known-good version while testing a newer one in parallel.

Next steps:

  • Review the AWS Lambda Layers documentation for current size limits and runtime compatibility details.
  • Consider using AWS SAM or the Serverless Framework to manage layer publication and function attachment as infrastructure-as-code.
  • If you are sharing layers across multiple accounts in an organization, review the resource-based policy documentation for layer permissions.

Glossary

TermDefinition
Lambda LayerA zip archive containing libraries or runtime dependencies that Lambda extracts to /opt in the execution environment, shared across multiple functions.
Layer Version ARNThe immutable, versioned identifier for a specific layer publish, in the format arn:aws:lambda:region:account-id:layer:layer-name:version.
manylinux wheelA pre-compiled Python package binary compatible with Amazon Linux 2, used to ensure C-extension libraries work correctly in the Lambda execution environment.
/opt/pythonThe path inside the Lambda execution environment where Python layer contents are extracted and automatically added to sys.path.
Immutable layer versionOnce published, a layer version cannot be modified. Updates require publishing a new version with an incremented version number.

Comments

Post a Comment

Popular posts from this blog

AWS SNS Email Alerts Not Arriving? The Subscription Confirmation Trap Explained

You've wired up an SNS topic, attached an email endpoint, and deployed your alerting pipeline — but your inbox is silent. Nine times out of ten, the culprit is a single missed step: the subscription confirmation email that AWS sends before it will deliver a single message to your endpoint. TL;DR Step Action Status if Skipped 1 Create SNS Topic Topic exists but is inert 2 Subscribe email endpoint to topic Subscription stays in PendingConfirmation 3 Click confirmation link in inbox ❌ No messages ever delivered 4 Publish message to topic ✅ Email delivered only after step 3 Why SNS Requires Subscription Confirmation AWS SNS uses a double opt-in model for email and HTTP/HTTPS endpoints. This is not an arbitrary UX decision — it is a security and anti-spam control. Without it, any AWS account could weaponize SNS to flood arbitrary email addresses with messages. The confirmation handshake proves that ...
Read more

S3 Public Access Denied: Why Your Public Object URL Still Returns 403

S3 Public Access Denied errors catch engineers off guard when the object ACL is already set to public — the real blocker is often a bucket-level or account-level override silently winning. TL;DR: S3 Public Access Denied Fix at a Glance Step What to Do 1 Check account-level Block Public Access — it overrides everything silently 2 Check bucket-level Block Public Access settings and disable the blocking flags 3 Verify the bucket policy does not contain an explicit Deny for public reads 4 Confirm the object ACL is actually set to public-read (or use a bucket policy instead) 5 Test the public URL and validate with a signed URL to isolate the layer Why S3 Public Access Denied Happens Even After Setting an Object Public The direct answer: setting an object ACL to public-read is necessary but not sufficient. AWS evaluates public access through a layered permission model — account-level Block Public A...
Read more

EC2 No Internet Access in Custom VPC: Fix Internet Gateway and Route Table

EC2 no internet access in a custom VPC is one of the most common launch-day failures — the instance is running, but outbound connectivity is completely silent. TL;DR: EC2 No Internet Access — Fix Checklist Step What to Do Why It Matters Complexity 1 Create and attach an Internet Gateway to the VPC Without an IGW, the VPC has no path to the public internet Low 2 Add a default route (0.0.0.0/0) pointing to the IGW in the subnet's route table The IGW exists but traffic has no route to reach it Low 3 Assign a public IP or Elastic IP to the instance Private IPs are not routable on the public internet Low 4 Verify Security Group allows outbound traffic A restrictive outbound rule silently drops all egress Low 5 Verify Network ...
Read more