I Leaked My AWS Access Key to GitHub: An Incident Response Playbook

Accidentally committing an AWS Access Key to a public GitHub repository is one of the most common — and most dangerous — cloud security incidents. Automated bots scan GitHub commits in near real-time, meaning your credentials can be harvested and abused within minutes of exposure.

TL;DR — Immediate Action Checklist

#ActionPriorityTool
1Deactivate the leaked access key immediately🔴 CriticalAWS Console / CLI
2Audit CloudTrail for unauthorized API calls🔴 CriticalCloudTrail / Athena
3Revoke any active sessions spawned by the key🔴 CriticalIAM / STS
4Delete the compromised key permanently🟠 HighIAM Console / CLI
5Rotate all secrets and audit IAM permissions🟠 HighSecrets Manager
6Remove the key from Git history🟡 Mediumgit-filter-repo / BFG
7Implement preventive controls🟡 Mediumgit-secrets / GuardDuty

Understanding the Threat Timeline

Before diving into remediation, understand what you are up against. Credential-harvesting bots monitor GitHub's public event stream via the GitHub API. The window between a commit being pushed and a bot attempting to use the key is measured in seconds to minutes, not hours.

sequenceDiagram participant Dev as "Developer" participant GH as "GitHub (Public Repo)" participant Bot as "Credential Harvesting Bot" participant AWS as "AWS APIs" participant GD as "GuardDuty" participant CT as "CloudTrail" Dev->>GH: git push (with hardcoded AKIA key) Note over GH,Bot: Bots poll GitHub Events API continuously Bot->>GH: Detect AKIA pattern in new commit (seconds later) Bot->>AWS: sts:GetCallerIdentity (reconnaissance) AWS-->>Bot: Returns account ID and ARN Bot->>AWS: iam:ListUsers, ec2:RunInstances, s3:ListBuckets AWS-->>CT: Logs all API calls CT-->>GD: Feeds event stream GD-->>Dev: "UnauthorizedAccess" finding alert Note over Dev: Incident Response begins
  1. Commit Push: Developer accidentally includes ~/.aws/credentials or hardcoded keys in source code and pushes to a public repo.
  2. Bot Detection: Automated scanners polling the GitHub Events API detect the new commit containing key patterns (e.g., AKIA...).
  3. Credential Abuse: The bot attempts API calls — typically probing IAM (GetCallerIdentity), then escalating to EC2, S3, or Lambda to mine crypto or exfiltrate data.
  4. AWS Detection: GuardDuty (if enabled) flags anomalous API calls. CloudTrail records every action.
  5. Incident Response: You execute the remediation playbook outlined below.

Phase 1: Immediate Containment (Do This First)

Step 1 — Deactivate the Key (Not Delete — Yet)

Deactivating first (rather than immediately deleting) preserves the key ID for your CloudTrail forensic audit. Deletion is irreversible and removes the audit anchor.

# Identify the leaked key ID (e.g., AKIAIOSFODNN7EXAMPLE)
aws iam list-access-keys --user-name <IAM_USERNAME>

# Deactivate it immediately
aws iam update-access-key \
  --user-name <IAM_USERNAME> \
  --access-key-id AKIAIOSFODNN7EXAMPLE \
  --status Inactive

If the key belongs to the root account, go directly to the AWS Console → Account Settings → Security Credentials and delete it. Root access keys should never exist in the first place.

Step 2 — Invalidate Active STS Sessions

Even after deactivating the key, any temporary credentials (AssumeRole tokens) already issued using that key remain valid until their natural expiry (up to 12 hours). You must explicitly revoke them.

# Attach an inline deny policy to the IAM user to invalidate all active sessions
# This uses the aws:TokenIssueTime condition key
aws iam put-user-policy \
  --user-name <IAM_USERNAME> \
  --policy-name RevokeActiveSessions \
  --policy-document '{
    "Version": "2012-10-17",
    "Statement": [{
      "Effect": "Deny",
      "Action": "*",
      "Resource": "*",
      "Condition": {
        "DateLessThan": {
          "aws:TokenIssueTime": "2024-01-15T00:00:00Z"
        }
      }
    }]
  }'

Set the aws:TokenIssueTime value to the current UTC timestamp. This denies all requests from tokens issued before this moment, effectively invalidating any active sessions spawned by the compromised key.

Phase 2: Forensic Audit with CloudTrail

Step 3 — Determine the Blast Radius

CloudTrail is your forensic source of truth. Query it to understand exactly what the attacker did — or attempted to do — using the compromised key.

Analogy: CloudTrail is the security camera footage of your AWS account. Even if the attacker is gone, the footage tells you exactly which doors they opened, which files they touched, and what they tried to steal.
🔽 [Click to expand] — CloudTrail Lookup: Find all API calls by the compromised key 
# Look up events by the specific access key ID
# CloudTrail event history covers the last 90 days
aws cloudtrail lookup-events \
  --lookup-attributes AttributeKey=AccessKeyId,AttributeValue=AKIAIOSFODNN7EXAMPLE \
  --start-time 2024-01-01T00:00:00Z \
  --end-time 2024-01-15T23:59:59Z \
  --query 'Events[*].{Time:EventTime,Event:EventName,User:Username,IP:CloudTrailEvent}' \
  --output table

Key API Calls to Investigate

Focus your audit on these high-risk API calls that indicate privilege escalation or data exfiltration:

  • iam:CreateUser, iam:AttachUserPolicy, iam:CreateAccessKey — Attacker creating backdoor accounts
  • iam:AssumeRole — Lateral movement to other roles
  • ec2:RunInstances — Crypto mining infrastructure
  • s3:GetObject, s3:ListBuckets — Data exfiltration
  • lambda:CreateFunction, lambda:InvokeFunction — Malicious function deployment
  • sts:GetCallerIdentity — Initial reconnaissance (almost always the first call)
graph TD A["Attacker Obtains Key"] --> B["Reconnaissance sts:GetCallerIdentity iam:GetUser"] B --> C{"Sufficient Permissions?"} C -- "Yes" --> D["Privilege Escalation iam:CreateUser iam:AttachUserPolicy"] C -- "No" --> E["Lateral Movement sts:AssumeRole"] E --> D D --> F["Persistence iam:CreateAccessKey lambda:CreateFunction"] F --> G["Impact"] G --> G1["Crypto Mining ec2:RunInstances"] G --> G2["Data Exfiltration s3:GetObject"] G --> G3["Ransomware s3:DeleteObject"] style A fill:#ff4444,color:#fff style G1 fill:#ff8800,color:#fff style G2 fill:#ff8800,color:#fff style G3 fill:#ff8800,color:#fff
  1. Reconnaissance: sts:GetCallerIdentity and iam:GetUser to understand the identity and its permissions.
  2. Privilege Escalation: Attempt to create new IAM users, attach admin policies, or assume higher-privileged roles.
  3. Persistence: Create new access keys or backdoor Lambda functions to maintain access even after the original key is rotated.
  4. Impact: Launch EC2 instances for crypto mining, exfiltrate S3 data, or pivot to other services.

Phase 3: Eradication

Step 4 — Delete the Compromised Key

Once your forensic audit is complete, permanently delete the key.

aws iam delete-access-key \
  --user-name <IAM_USERNAME> \
  --access-key-id AKIAIOSFODNN7EXAMPLE

Step 5 — Remediate Attacker-Created Resources

Based on your CloudTrail audit, systematically remove any resources the attacker created:

  • Delete any IAM users, roles, or policies created after the key exposure timestamp.
  • Terminate unauthorized EC2 instances.
  • Delete unauthorized Lambda functions.
  • Audit S3 bucket policies for newly added public access or cross-account grants.
  • Check for new SNS/SQS subscriptions that could be used for data exfiltration.

Step 6 — Issue a New Key and Store It Securely

Never hardcode credentials again. Use AWS Secrets Manager or IAM Roles for compute workloads.

# Generate a new access key
aws iam create-access-key --user-name <IAM_USERNAME>

# Store it in Secrets Manager — never in code or .env files
aws secretsmanager create-secret \
  --name prod/myapp/aws-credentials \
  --description "Rotated IAM credentials post-incident" \
  --secret-string '{"AccessKeyId":"NEWKEYID","SecretAccessKey":"NEWSECRET"}'

Phase 4: Remove the Key from Git History

Deleting the file or adding a new commit does not remove the secret from Git history. Anyone can still access it via git log or by browsing old commits on GitHub.

🔽 [Click to expand] — Purge secrets from Git history using git-filter-repo 
# Install git-filter-repo (preferred over the deprecated git filter-branch)
pip install git-filter-repo

# Remove the specific file containing the secret from ALL history
git filter-repo --path path/to/credentials-file --invert-paths

# Force push all branches to overwrite remote history
# WARNING: This is a destructive operation — coordinate with your team
git push origin --force --all
git push origin --force --tags

# After purging, GitHub's cache may still serve old commits.
# Contact GitHub Support to purge cached views of the exposed data.

Important: After a force push, all collaborators must re-clone the repository. Old local clones still contain the secret in their local history.

Phase 5: Preventive Controls

Remediation without prevention is incomplete. Implement these controls to ensure this never happens again.

graph LR subgraph "Prevention Layer" P1["git-secrets Pre-commit Hook"] P2["GitHub Secret Scanning"] P3["IAM Roles (No Long-lived Keys)"] end subgraph "Detection Layer" D1["GuardDuty Threat Detection"] D2["AWS Config Key Age Rule"] D3["CloudTrail Audit Logging"] end subgraph "Response Layer" R1["Deactivate Key"] R2["Revoke STS Sessions"] R3["Forensic Audit"] R4["Eradicate Resources"] end P1 -->|"Blocks commit"| Dev["Developer Workstation"] P2 -->|"Auto-notifies AWS"| D1 P3 -->|"Eliminates key risk"| Dev D1 -->|"Triggers alert"| R1 D2 -->|"Flags stale keys"| R1 D3 -->|"Provides evidence"| R3 R1 --> R2 --> R3 --> R4 style P1 fill:#2196F3,color:#fff style P2 fill:#2196F3,color:#fff style P3 fill:#2196F3,color:#fff style D1 fill:#FF9800,color:#fff style D2 fill:#FF9800,color:#fff style D3 fill:#FF9800,color:#fff style R1 fill:#F44336,color:#fff style R2 fill:#F44336,color:#fff style R3 fill:#F44336,color:#fff style R4 fill:#F44336,color:#fff

Control 1 — Pre-commit Hooks with git-secrets

# Install git-secrets
brew install git-secrets  # macOS

# Register AWS credential patterns
git secrets --register-aws

# Install hooks into the current repository
git secrets --install

# Now any commit containing AWS key patterns will be blocked
git secrets --scan  # Manual scan of current working tree

Control 2 — Enable AWS GuardDuty

GuardDuty provides continuous threat detection and will alert you to findings such as UnauthorizedAccess:IAMUser/InstanceCredentialExfiltration and anomalous API call patterns from unexpected geographic locations.

# Enable GuardDuty in your account (regional service — run per region)
aws guardduty create-detector --enable --finding-publishing-frequency FIFTEEN_MINUTES

Control 3 — Use IAM Roles Instead of Long-Lived Keys

For EC2, Lambda, ECS, and other AWS compute services, never use access keys. Attach an IAM Role directly to the compute resource. The AWS SDK automatically retrieves short-lived credentials from the instance metadata service (IMDS).

# Attach an IAM role to an EC2 instance (no access keys needed)
aws ec2 associate-iam-instance-profile \
  --instance-id i-0abcdef1234567890 \
  --iam-instance-profile Name=MyAppInstanceProfile

Control 4 — Enable AWS Config Rule for Access Key Age

# Deploy the managed Config rule to flag access keys older than 90 days
aws configservice put-config-rule --config-rule '{
  "ConfigRuleName": "access-keys-rotated",
  "Source": {
    "Owner": "AWS",
    "SourceIdentifier": "ACCESS_KEYS_ROTATED"
  },
  "InputParameters": "{\"maxAccessKeyAge\":\"90\"}"
}'

Control 5 — GitHub Secret Scanning (for Organizations)

GitHub natively integrates with AWS to automatically revoke detected AWS access keys via its Secret Scanning feature. For public repositories, this is enabled by default. For private repositories, it requires a GitHub Advanced Security license. When a key is detected, GitHub notifies AWS, which can automatically quarantine the key.

Glossary

TermDefinition
IAM Access KeyA long-lived credential pair (Access Key ID + Secret Access Key) used to authenticate programmatic AWS API requests.
STS (Security Token Service)AWS service that issues short-lived, temporary security credentials for IAM roles and federated identities.
CloudTrailAWS service that records all API calls made in your account, providing an immutable audit log for forensic investigation.
Blast RadiusThe scope of resources and data potentially accessed or compromised as a result of a security incident.
GuardDutyAWS managed threat detection service that continuously monitors for malicious activity and unauthorized behavior using ML and threat intelligence.

Next Steps

Comments

Post a Comment

Popular posts from this blog

EC2 No Internet Access in Custom VPC: Attaching an Internet Gateway and Fixing Route Tables

Launching an EC2 instance in a custom VPC only to find it cannot reach the internet is one of the most common networking pitfalls in AWS — and it almost always comes down to two missing pieces: an Internet Gateway (IGW) and a correctly configured Route Table. This post walks you through the exact architecture, the root cause, and the precise steps to fix it. TL;DR Step Action Why It Matters 1 Create and attach an Internet Gateway to your VPC Without an IGW, there is no path between your VPC and the public internet 2 Add a 0.0.0.0/0 route pointing to the IGW in your subnet's Route Table The route table is the traffic director — without this entry, outbound packets have nowhere to go 3 Assign a Public IP or Elastic IP to the EC2 instance The IGW performs NAT only for instances with a public IP address 4 Verify Security Group allows outbound traffic Security Groups are stateful firewalls; outbound rules...
Read more

IAM User vs. IAM Role: Why Your EC2 Instance Should Never Use a User

Choosing between an IAM User and an IAM Role is one of the most consequential security decisions in AWS — get it wrong and you either expose long-lived credentials or block your application entirely. This post cuts through the confusion and gives you a definitive answer for the EC2-to-S3 access pattern. TL;DR — Quick Comparison Dimension IAM User IAM Role Identity type A person or a static service account An assumable identity for AWS services, users, or external principals Credential type Long-lived Access Key ID + Secret Access Key Short-lived, auto-rotating STS tokens (max 12 h by default) Credential rotation Manual — you must rotate keys yourself Automatic — AWS rotates them transparently Stored on EC2? Yes — in ~/.aws/credentials or env vars (risky) No — delivered via the In...
Read more

EC2 SSH Connection Timeout: The Exact Security Group Rules You Need to Fix It

You've launched a fresh EC2 instance, fired up your terminal, and run ssh ec2-user@<your-ip> — only to watch the cursor hang until Connection timed out kills your hope. This is one of the most common EC2 onboarding blockers, and it almost always traces back to a misconfigured Security Group inbound rule. TL;DR Check What to Verify Correct Value Protocol Security Group inbound rule protocol TCP Port Destination port in the inbound rule 22 Source Allowed source CIDR or IP Your public IP (e.g., 203.0.113.10/32 ) or 0.0.0.0/0 for testing Instance State EC2 instance status checks 2/2 checks passed Subnet Route Table Route to Internet Gateway (public subnet) 0.0.0.0/0 → igw-xxxxxxxx Public IP / EIP Instance has a reachable public address Public IPv4 or Elastic IP assigned Why "Connection Timed Out" Specifically? A timeout (as opposed to Connection refused ) means your TC...
Read more