Why Use AWS Secrets Manager Instead of Hardcoding Credentials?

Storing a database password directly in application code — even in a private repository — is one of the most common and consequential security mistakes in cloud engineering. AWS Secrets Manager exists precisely to eliminate this pattern, and understanding why requires looking at the full lifecycle of a credential: how it's stored, accessed, rotated, and audited.

TL;DR: Secrets Manager vs. Hardcoded Credentials

Concern	Hardcoded Credential	AWS Secrets Manager
Exposure risk	Anyone with repo access sees it	Encrypted at rest (KMS), access via IAM
Rotation	Manual, error-prone, often skipped	Automatic, Lambda-driven, zero-downtime
Audit trail	None	Every access logged in CloudTrail
Blast radius on leak	Credential valid until manually rotated	Rotation limits exposure window
Cross-service reuse	Copy-paste proliferation	Single source of truth via API

Why Secrets Manager Matters: The Real Risk of "Private" Repos

The assumption that a private repository is a safe credential store breaks down across several dimensions. Git history is permanent — a password committed and later removed is still recoverable via git log or git reflog. Repository access is broader than you think: CI/CD pipelines, third-party integrations, contractor accounts, and compromised developer machines all represent exposure vectors. And when that credential needs to rotate — because of a security policy, a suspected breach, or a compliance audit — every service consuming it must be updated simultaneously or face downtime.

Think of a hardcoded credential like a master key taped to the inside of a locked door. The door being locked doesn't protect the key — it just means you have to get inside first.

The deeper problem is operational: hardcoded credentials create a tight coupling between your deployment artifact and a specific secret value. Rotating the password means rebuilding and redeploying. In a microservices environment, that means coordinating deployments across multiple services simultaneously — a coordination problem that teams routinely defer, leaving stale credentials in production indefinitely.

How AWS Secrets Manager Stores and Protects Secrets

Secrets Manager stores secret values encrypted using AWS KMS. By default, it uses the AWS-managed key for Secrets Manager (aws/secretsmanager), but you can specify a customer-managed KMS key for additional control. The secret value is never returned in plaintext from the storage layer — decryption happens in-memory during the GetSecretValue API call, and only to callers with both the Secrets Manager IAM permission and the KMS decrypt permission.

Each secret has a single ARN that remains stable for the lifetime of the secret:

arn:aws:secretsmanager:us-east-1:123456789012:secret:prod/myapp/db-AbCdEf

The six-character suffix (e.g., AbCdEf) is appended by AWS at creation time and is part of the full ARN. This ARN does not change when the secret value is rotated. Rotation creates a new version of the secret, identified by a version ID and staging labels (AWSCURRENT, AWSPENDING, AWSPREVIOUS). The ARN itself is stable across all versions.

graph TD A["Secret ARN
(stable, never changes)"] --> B["Version: v1
Label: AWSPREVIOUS"] A --> C["Version: v2
Label: AWSCURRENT"] A --> D["Version: v3
Label: AWSPENDING (during rotation)"] style A fill:#2d6a9f,color:#fff style C fill:#1e7e34,color:#fff style D fill:#856404,color:#fff

1. Secret ARN — stable identifier, never changes across rotations.
2. Version ID — unique identifier for each secret value; created on every rotation or manual update.
3. Staging labels — AWSCURRENT always points to the active value; GetSecretValue without a version specifier returns this label.
4. AWSPREVIOUS — retained briefly after rotation to allow in-flight connections to drain gracefully.

Automatic Password Rotation: How It Actually Works

Rotation in Secrets Manager is driven by a Lambda function that Secrets Manager invokes on a schedule or on demand. For supported databases (Amazon RDS, Redshift, DocumentDB), AWS provides managed rotation functions you can deploy directly. For other targets, you implement the rotation logic yourself following the documented four-step lifecycle.

sequenceDiagram participant SM as Secrets Manager participant L as Rotation Lambda participant DB as Target Database SM->>L: invoke createSecret L->>SM: store new credential (AWSPENDING) SM->>L: invoke setSecret L->>DB: apply new credential SM->>L: invoke testSecret L->>DB: verify new credential works SM->>L: invoke finishSecret L->>SM: promote AWSPENDING to AWSCURRENT Note over SM: ARN unchanged throughout

1. createSecret — Lambda generates a new credential and stores it under the AWSPENDING staging label.
2. setSecret — Lambda applies the new credential to the target database or service.
3. testSecret — Lambda verifies the new credential works against the target.
4. finishSecret — Secrets Manager moves AWSCURRENT to the new version and AWSPREVIOUS to the old one. The ARN remains unchanged throughout.

Applications that always call GetSecretValue at runtime — rather than caching the value at startup — automatically receive the rotated credential without any redeployment. This is the architectural contract that makes zero-downtime rotation possible.

Rotation doesn't change what your IAM policy points to. It changes what value is returned when you call the API. Your ARN-based policy stays valid across every rotation cycle.

IAM Policy Design for Secrets Manager Access

Access to a secret is controlled by two independent layers: the IAM identity policy (or role) attached to the caller, and optionally a resource-based policy attached to the secret itself. Both must permit the action for access to succeed.

A minimal least-privilege policy for an application that only needs to read a specific secret:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "ReadSpecificSecret",
      "Effect": "Allow",
      "Action": "secretsmanager:GetSecretValue",
      "Resource": "arn:aws:secretsmanager:us-east-1:123456789012:secret:prod/myapp/db-AbCdEf"
    },
    {
      "Sid": "AllowKMSDecrypt",
      "Effect": "Allow",
      "Action": "kms:Decrypt",
      "Resource": "arn:aws:kms:us-east-1:123456789012:key/your-kms-key-id"
    }
  ]
}

Two points worth internalizing here. First, the Resource field uses the full secret ARN including the suffix — this ARN remains valid across all rotations because rotation does not change the ARN. Second, if you use a customer-managed KMS key, the kms:Decrypt permission is required separately; without it, GetSecretValue will fail even if the Secrets Manager permission is correct.

If you want a single policy to cover multiple secrets sharing a name prefix, you can use a wildcard pattern:

"Resource": "arn:aws:secretsmanager:us-east-1:123456789012:secret:prod/myapp/*"

This is a deliberate choice for broader access — not a workaround for ARN instability. Use it only when the policy intent is genuinely to cover multiple secrets.

Diagnosing AccessDenied on GetSecretValue

AccessDenied on GetSecretValue is one of the most common operational issues teams hit when first adopting Secrets Manager. The error message is intentionally opaque for security reasons, so systematic diagnosis matters.

Step 1: Confirm the caller identity and region

Verify which IAM principal is making the call and that it's targeting the correct region. A Lambda function in us-west-2 calling a secret in us-east-1 will fail — secrets are regional resources.

aws sts get-caller-identity

Step 2: Verify the IAM policy covers the correct ARN

Retrieve the exact ARN of the secret, including its suffix, and confirm it matches what's in the IAM policy.

aws secretsmanager describe-secret \
  --secret-id prod/myapp/db \
  --region us-east-1 \
  --query '{ARN:ARN,Name:Name}'

Step 3: Check for a resource-based policy on the secret

A resource-based policy that explicitly denies the caller — or that uses an Allow without including the caller's principal — can block access independently of the identity policy.

aws secretsmanager get-resource-policy \
  --secret-id prod/myapp/db \
  --region us-east-1

Step 4: Check for SCPs or permission boundaries

If the caller is in an AWS Organizations account, a Service Control Policy (SCP) may restrict secretsmanager:GetSecretValue at the account level. SCPs apply regardless of identity policies. Verify with your organization administrator or inspect the effective policies via IAM Access Analyzer.

Step 5: Verify KMS permissions if using a customer-managed key

aws kms get-key-policy \
  --key-id your-kms-key-id \
  --policy-name default \
  --region us-east-1

Confirm the caller's principal appears in the key policy with kms:Decrypt permission. The KMS key policy is a separate authorization boundary from IAM.

Experience Signal: A team migrated their application to a new AWS account and copied the IAM policy verbatim. The policy referenced the secret ARN from the original account. The secret was recreated in the new account with a different suffix. The application received AccessDenied. The symptom looked like a permissions problem; the actual cause was an ARN mismatch — the policy pointed to a non-existent resource in the new account. The fix was running describe-secret in the new account, retrieving the correct ARN, and updating the policy.

Retrieving a Secret at Runtime: Application Integration

The recommended pattern is to call GetSecretValue at application startup or on a cache-miss, rather than at every request. Secrets Manager has API rate limits, and hammering the API per-request is both unnecessary and a source of throttling errors in high-throughput services.

🔽 Click to expand: Python example with caching

import boto3
import json
from functools import lru_cache

@lru_cache(maxsize=1)
def get_db_credentials(secret_name: str, region_name: str) -> dict:
    client = boto3.client('secretsmanager', region_name=region_name)
    response = client.get_secret_value(SecretId=secret_name)
    return json.loads(response['SecretString'])

credentials = get_db_credentials('prod/myapp/db', 'us-east-1')
db_password = credentials['password']

For rotation to be transparent, the cache must have a bounded TTL — not an indefinite lru_cache. In production, use a time-bounded cache (e.g., refresh every 5 minutes) so that rotated credentials are picked up without a full application restart. The AWS SDK for Python (boto3) also supports client-side caching via the aws-secretsmanager-caching library, which handles TTL and refresh automatically.

CLI Reference: Core Secrets Manager Operations

# Create a new secret
aws secretsmanager create-secret \
  --name prod/myapp/db \
  --secret-string '{"username":"admin","password":"changeme"}' \
  --region us-east-1

# Retrieve the current secret value
aws secretsmanager get-secret-value \
  --secret-id prod/myapp/db \
  --region us-east-1

# Rotate a secret immediately (rotation must be configured first)
aws secretsmanager rotate-secret \
  --secret-id prod/myapp/db \
  --region us-east-1

# List all versions of a secret
aws secretsmanager list-secret-version-ids \
  --secret-id prod/myapp/db \
  --region us-east-1

Wrap-Up: Secrets Manager and Credential Hygiene

The case against hardcoding credentials isn't theoretical — it's operational. Every hardcoded password is a rotation event waiting to become an incident. AWS Secrets Manager removes the coupling between your deployment artifact and a specific credential value, enables automatic rotation without redeployment, and provides a complete audit trail via CloudTrail for every GetSecretValue call.

The single most important architectural decision is to always retrieve secrets at runtime via the API, never bake them into environment variables at build time or container image layers. That one change is what makes automatic rotation actually automatic.

For next steps, review the AWS Secrets Manager rotation documentation and the authentication and access control guide.

Glossary

Term	Definition
Secret ARN	The stable Amazon Resource Name for a secret, including a system-generated suffix. Does not change across rotations.
Version ID	A unique identifier for a specific value of a secret. A new version is created on every rotation or manual update.
Staging Label	A human-readable pointer to a secret version. AWSCURRENT always identifies the active value returned by default.
Rotation Function	A Lambda function that implements the four-step rotation lifecycle: createSecret, setSecret, testSecret, finishSecret.
Resource-based Policy	An optional policy attached directly to a secret that controls cross-account or fine-grained access independently of IAM identity policies.

Related Posts

• 📄 AWS Parameter Store vs. Secrets Manager: How to Store API Keys for Free
• 📄 I Leaked My AWS Access Key to GitHub: An Incident Response Playbook
• 📄 Difference Between IAM User and IAM Role: Which One Should Your EC2 Use?