Why Use AWS Secrets Manager Over Hardcoding Credentials?

Every week, thousands of API keys and database passwords are accidentally exposed in public repositories — but the more insidious risk is the assumption that a private repo makes hardcoded secrets safe. This post dismantles that assumption and shows exactly how AWS Secrets Manager eliminates the credential management problem at its root.

TL;DR

DimensionHardcoded CredentialsAWS Secrets Manager
Exposure RiskAny repo clone, log leak, or insider access exposes the secret permanentlySecret never touches source code or environment variables in plaintext
RotationManual — requires code redeploy on every changeAutomatic rotation via Lambda, zero downtime
Audit TrailNone — no visibility into who used the credentialEvery GetSecretValue call logged in AWS CloudTrail
Access ControlWhoever has the codebase has the secretIAM policy-based, least-privilege per service/role
Blast Radius on BreachCredential valid until manually rotated (often never)Rotation window configurable; compromised secret expires quickly
Multi-environment ManagementDifferent secrets require different code branches or config filesSeparate secret ARNs per environment, same application code

The "Private Repo" Fallacy

Let's be precise about the threat model. Storing a database password in a private repository does not mean it is secure. Here is why:

  • Git history is permanent. Even if you delete the secret in a later commit, it remains in git log and any clone made before deletion.
  • Access sprawl. Every developer, CI/CD runner, contractor, and third-party integration tool that clones the repo now holds a copy of that credential.
  • Log leakage. Application errors, stack traces, and debug logs frequently print environment variables or config objects — including secrets — to CloudWatch Logs or external logging platforms.
  • Repo visibility can change. A misconfigured GitHub setting, an org policy change, or a fork can silently make a private repo public.
  • No rotation path. If the credential is baked into code, rotating it requires a code change, a PR, a review cycle, and a deployment — meaning rotation almost never happens in practice.
Analogy: Hardcoding a database password in your source code is like writing your house key's lock combination on the outside of every copy of your house blueprint and distributing those blueprints to your entire team. Even if the blueprints are stored in a locked filing cabinet, every person who ever touched one now knows the combination — and the combination never changes.

How AWS Secrets Manager Works: Architecture Deep Dive

AWS Secrets Manager is a managed service that stores, retrieves, and rotates secrets. Your application never holds the secret statically — it fetches it at runtime via an API call, authenticated by its IAM role.

Runtime Secret Retrieval Flow

sequenceDiagram participant App as "Application
(ECS Task / Lambda)" participant IAM as "AWS IAM
(STS)" participant SM as "AWS Secrets
Manager" participant KMS as "AWS KMS" participant DB as "RDS Database" App->>IAM: Assume IAM Role
(instance/task role) IAM-->>App: Temporary credentials App->>SM: GetSecretValue
(signed with IAM role) SM->>IAM: Validate principal &
resource policy IAM-->>SM: Authorized SM->>KMS: Decrypt secret value KMS-->>SM: Plaintext secret SM-->>App: Secret returned
in API response (in-memory only) App->>DB: Connect using
in-memory credential DB-->>App: Connection established
  1. Application starts on an EC2 instance, ECS task, or Lambda function — no secret is present in the code or environment.
  2. IAM Role authentication: The compute resource uses its attached IAM role to sign the API request to Secrets Manager. No static credentials needed.
  3. Secrets Manager validates the IAM principal against the resource-based policy on the secret and the identity-based policy on the role.
  4. KMS decryption: Secrets Manager uses AWS KMS to decrypt the secret value before returning it. You can use the AWS-managed key or a customer-managed KMS key (CMK) for additional control.
  5. Secret returned in memory: The plaintext secret is returned only in the API response body — it is never written to disk or stored in the environment.
  6. Application connects to the database using the in-memory credential.

Automatic Password Rotation Flow

This is where Secrets Manager provides its most operationally significant value. Rotation is orchestrated by a Lambda function that Secrets Manager invokes on a defined schedule.

sequenceDiagram participant Sched as "Rotation Schedule" participant SM as "AWS Secrets
Manager" participant Lambda as "Rotation Lambda" participant DB as "RDS Database" participant App as "Application" Sched->>SM: Rotation interval reached SM->>Lambda: Invoke — createSecret step Lambda->>SM: Store new password
(AWSPENDING label) SM->>Lambda: Invoke — setSecret step Lambda->>DB: Apply new password
(e.g., ALTER USER) DB-->>Lambda: Password updated SM->>Lambda: Invoke — testSecret step Lambda->>DB: Test login with
AWSPENDING credential DB-->>Lambda: Login successful SM->>Lambda: Invoke — finishSecret step Lambda->>SM: Promote AWSPENDING
to AWSCURRENT Note over SM: Old version labeled
AWSPREVIOUS (briefly valid) App->>SM: GetSecretValue (next call) SM-->>App: Returns new AWSCURRENT
credential automatically
  1. Rotation schedule triggers — either on a defined interval (e.g., every 30 days) or on-demand via the console or CLI.
  2. Secrets Manager invokes the rotation Lambda with a createSecret step. The Lambda generates a new password and stores it as a pending version (AWSPENDING staging label).
  3. setSecret step: The Lambda applies the new password to the actual database (e.g., calls ALTER USER on RDS).
  4. testSecret step: The Lambda verifies the new credential can successfully authenticate to the database.
  5. finishSecret step: Secrets Manager moves the AWSPENDING label to AWSCURRENT and demotes the old version to AWSPREVIOUS. The old password remains briefly valid during this window to prevent connection drops.
  6. Applications fetching the secret after rotation automatically receive the new credential on their next GetSecretValue call — no redeployment required.

Key insight on zero-downtime rotation: For supported RDS databases, AWS provides managed rotation Lambda functions. During the rotation window, both AWSPREVIOUS and AWSCURRENT versions are valid, so existing database connections are not forcibly terminated.

IAM: Least-Privilege Access to Secrets

The security model only holds if access to the secret is tightly scoped. Below is a minimal IAM policy granting a specific application role read-only access to a single secret.

🔽 [Click to expand] IAM Policy — Least-Privilege Secret Access 
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AllowGetSpecificSecret",
      "Effect": "Allow",
      "Action": [
        "secretsmanager:GetSecretValue",
        "secretsmanager:DescribeSecret"
      ],
      "Resource": "arn:aws:secretsmanager:us-east-1:123456789012:secret:prod/myapp/db-password-AbCdEf"
    },
    {
      "Sid": "AllowKMSDecrypt",
      "Effect": "Allow",
      "Action": [
        "kms:Decrypt"
      ],
      "Resource": "arn:aws:kms:us-east-1:123456789012:key/your-cmk-key-id",
      "Condition": {
        "StringEquals": {
          "kms:ViaService": "secretsmanager.us-east-1.amazonaws.com"
        }
      }
    }
  ]
}

Critical notes on this policy:

  • The Resource ARN targets a single specific secret, not *. Secrets Manager appends a 6-character suffix to secret ARNs — use a wildcard suffix (prod/myapp/db-password-*) if you need to match across rotation-generated versions, or pin the full ARN.
  • The kms:ViaService condition ensures the KMS key can only be used when the call originates from Secrets Manager, not directly by the application.
  • The rotation Lambda requires additional permissions: secretsmanager:GetSecretValue, secretsmanager:PutSecretValue, secretsmanager:UpdateSecretVersionStage, and database-specific permissions.

Implementation: Storing and Retrieving a Secret

Step 1: Create a Secret (AWS CLI)

aws secretsmanager create-secret \
  --name "prod/myapp/db-password" \
  --description "Production RDS password for myapp" \
  --secret-string '{"username":"dbadmin","password":"InitialP@ssw0rd!"}' \
  --kms-key-id "arn:aws:kms:us-east-1:123456789012:key/your-cmk-key-id" \
  --region us-east-1

Step 2: Retrieve the Secret at Runtime (Python — boto3)

🔽 [Click to expand] Python — Fetch Secret with Error Handling 
import boto3
import json
from botocore.exceptions import ClientError

def get_secret(secret_name: str, region_name: str = "us-east-1") -> dict:
    """
    Retrieves a secret from AWS Secrets Manager.
    The IAM role attached to the compute resource provides authentication.
    No static credentials are used.
    """
    client = boto3.client(
        service_name="secretsmanager",
        region_name=region_name
    )

    try:
        response = client.get_secret_value(SecretId=secret_name)
    except ClientError as e:
        error_code = e.response["Error"]["Code"]
        # Handle specific Secrets Manager error codes per AWS documentation
        if error_code == "ResourceNotFoundException":
            raise Exception(f"Secret '{secret_name}' not found.")
        elif error_code == "InvalidRequestException":
            raise Exception(f"Invalid request for secret '{secret_name}': {e}")
        elif error_code == "AccessDeniedException":
            raise Exception(f"IAM role lacks permission to access secret '{secret_name}'.")
        else:
            raise e

    # SecretString is returned for text-based secrets
    secret = json.loads(response["SecretString"])
    return secret


# Usage — secret is only in memory, never in code or env vars
if __name__ == "__main__":
    db_credentials = get_secret("prod/myapp/db-password")
    db_host = "mydb.cluster-xyz.us-east-1.rds.amazonaws.com"

    # Connect to database using in-memory credentials
    connection = connect_to_db(
        host=db_host,
        user=db_credentials["username"],
        password=db_credentials["password"]
    )

Step 3: Enable Automatic Rotation (AWS CLI)

For supported Amazon RDS databases, AWS provides a managed rotation Lambda. You can enable rotation with a single CLI command after the Lambda is configured:

aws secretsmanager rotate-secret \
  --secret-id "prod/myapp/db-password" \
  --rotation-lambda-arn "arn:aws:lambda:us-east-1:123456789012:function:SecretsManagerRDSRotation" \
  --rotation-rules "{\"AutomaticallyAfterDays\": 30}" \
  --region us-east-1

Note: The rotation Lambda must be deployed in the same VPC as your RDS instance (or have network access to it) and must have the appropriate IAM permissions to modify database users. AWS provides pre-built rotation Lambda templates for common RDS engines. Refer to the official rotation documentation for engine-specific setup.

Caching: Avoiding Per-Request API Latency

A common concern is latency — calling GetSecretValue on every database query would be unacceptable. The solution is the AWS Secrets Manager Agent or the AWS Secrets Manager caching client (available for Java and Python via the aws-secretsmanager-caching library). The caching client stores the secret in memory and only refreshes it when the TTL expires or a rotation event is detected.

# Install the caching client
pip install aws-secretsmanager-caching
import botocore
import boto3
from aws_secretsmanager_caching import SecretCache, SecretCacheConfig

client = boto3.client("secretsmanager", region_name="us-east-1")
cache_config = SecretCacheConfig()  # Default TTL is 1 hour
cache = SecretCache(config=cache_config, client=client)

# Subsequent calls within TTL are served from in-memory cache
secret_string = cache.get_secret_string("prod/myapp/db-password")

Glossary

TermDefinition
AWSCURRENTThe staging label Secrets Manager assigns to the active, current version of a secret value.
AWSPENDINGThe staging label assigned to a newly generated secret value during the rotation process, before it is promoted to AWSCURRENT.
Customer Managed Key (CMK)A KMS key created and managed by you, providing full control over key policy, rotation, and deletion — used to encrypt secret values in Secrets Manager.
Resource-Based PolicyAn IAM policy attached directly to a Secrets Manager secret that controls which principals (accounts, roles, services) can access it — useful for cross-account access.
Rotation LambdaAn AWS Lambda function that implements the four-step rotation protocol (createSecret, setSecret, testSecret, finishSecret) invoked by Secrets Manager during credential rotation.

Wrap-Up & Next Steps

The operational cost of using Secrets Manager — a few milliseconds of latency on cold start, a small API cost — is negligible compared to the cost of a credential breach or a manual rotation incident at 2 AM. The architecture is straightforward: compute roles fetch secrets at runtime, rotation is automated, and every access is auditable via CloudTrail.

Immediate actions:

  1. Audit your current repositories and CI/CD pipelines for hardcoded credentials using tools like git-secrets or AWS's own Amazon CodeGuru Reviewer.
  2. Migrate existing secrets to Secrets Manager and update application code to use GetSecretValue.
  3. Enable automatic rotation for all RDS and Redshift credentials.
  4. Set up CloudTrail alerts for unexpected GetSecretValue calls from unknown principals.

📖 Official Documentation: AWS Secrets Manager User Guide

Comments

Post a Comment

Popular posts from this blog

EC2 No Internet Access in Custom VPC: Attaching an Internet Gateway and Fixing Route Tables

Launching an EC2 instance in a custom VPC only to find it cannot reach the internet is one of the most common networking pitfalls in AWS — and it almost always comes down to two missing pieces: an Internet Gateway (IGW) and a correctly configured Route Table. This post walks you through the exact architecture, the root cause, and the precise steps to fix it. TL;DR Step Action Why It Matters 1 Create and attach an Internet Gateway to your VPC Without an IGW, there is no path between your VPC and the public internet 2 Add a 0.0.0.0/0 route pointing to the IGW in your subnet's Route Table The route table is the traffic director — without this entry, outbound packets have nowhere to go 3 Assign a Public IP or Elastic IP to the EC2 instance The IGW performs NAT only for instances with a public IP address 4 Verify Security Group allows outbound traffic Security Groups are stateful firewalls; outbound rules...
Read more

IAM User vs. IAM Role: Why Your EC2 Instance Should Never Use a User

Choosing between an IAM User and an IAM Role is one of the most consequential security decisions in AWS — get it wrong and you either expose long-lived credentials or block your application entirely. This post cuts through the confusion and gives you a definitive answer for the EC2-to-S3 access pattern. TL;DR — Quick Comparison Dimension IAM User IAM Role Identity type A person or a static service account An assumable identity for AWS services, users, or external principals Credential type Long-lived Access Key ID + Secret Access Key Short-lived, auto-rotating STS tokens (max 12 h by default) Credential rotation Manual — you must rotate keys yourself Automatic — AWS rotates them transparently Stored on EC2? Yes — in ~/.aws/credentials or env vars (risky) No — delivered via the In...
Read more

Lambda Infinite Loop with S3: How to Prevent Recursive Triggers

Writing a processed file back to the same S3 bucket that triggered your Lambda function creates a recursive loop — each output file fires a new invocation, which writes another file, ad infinitum. Left unchecked, this silently drains your AWS budget and exhausts Lambda concurrency limits within minutes. TL;DR Strategy Mechanism Best For Complexity Separate Output Prefix/Bucket S3 event filter on prefix Simple pipelines Low Object Metadata Check Lambda reads S3 object metadata before processing Single-bucket requirement Medium Separate Output Bucket Trigger only on source bucket Clean architectural separation Low DynamoDB Idempotency Guard Track processed ETag/key in DynamoDB Exactly-once guarantees Medium-High Understanding the Recursive Loop Before fixing the problem, you need to understand exactly why it happens. The S3 → Lambda trigger is an event-driven subscription: S3 publishes an ObjectCreat...
Read more