How to Copy Files from S3 to EC2 Using AWS CLI: Exact Commands and IAM Setup

Copying a configuration file from S3 to an EC2 instance is one of the most common bootstrap tasks in AWS — yet it fails silently more often than it should, usually because the instance role is missing a single permission or the S3 path is malformed. This guide covers the exact aws s3 cp command, the minimum IAM policy required, and the diagnostic steps that catch the real failure when the obvious fix doesn't work.

TL;DR: Copy a File from S3 to EC2

StepWhat You Do
1. Attach IAM role to EC2Create an instance profile with s3:GetObject on the target bucket/key
2. SSH into the instanceConnect via EC2 Instance Connect or your key pair
3. Run the copy commandaws s3 cp s3://your-bucket/path/to/config.json /etc/myapp/config.json
4. Verifycat /etc/myapp/config.json

How EC2-to-S3 Authentication Works

When you run aws s3 cp on an EC2 instance, the AWS CLI does not use a hardcoded access key. Instead, it queries the Instance Metadata Service (IMDS) at 169.254.169.254 to retrieve temporary credentials issued by the IAM role attached to the instance profile. Those credentials are scoped to exactly the permissions defined in the role's policy. If no role is attached, the CLI has no credentials and the request fails immediately with an authentication error — not a permission error. Understanding this chain matters because the failure mode differs depending on where the chain breaks.

sequenceDiagram participant EC2 as EC2 Instance participant IMDS as IMDS (169.254.169.254) participant STS as AWS STS participant S3 as Amazon S3 EC2->>IMDS: GET /latest/meta-data/iam/security-credentials/role-name IMDS->>STS: Fetch temporary credentials for attached role STS-->>IMDS: AccessKeyId, SecretAccessKey, SessionToken IMDS-->>EC2: Temporary credentials EC2->>S3: GetObject request (signed with temp credentials) S3->>S3: Evaluate IAM policy + Bucket policy S3-->>EC2: Object data streamed to local path
  1. EC2 instance runs aws s3 cp.
  2. AWS CLI queries IMDS to retrieve temporary credentials from the attached IAM role.
  3. S3 evaluates the request against the IAM policy and (if present) the bucket policy.
  4. If both allow the action, the object is streamed to the local destination path.

Step 1: Attach the Correct IAM Role to Your EC2 Instance

Before running any CLI command on the instance, the instance needs an IAM role attached via an instance profile. This is the most common root cause of failure — engineers SSH in and run the copy command before confirming a role is attached at all.

Create a policy document granting the minimum required permission. s3:GetObject is the only action needed to download a single file. s3:ListBucket is required separately if you use wildcards or need to list objects — it is not implied by s3:GetObject.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AllowGetConfigFile",
      "Effect": "Allow",
      "Action": "s3:GetObject",
      "Resource": "arn:aws:s3:::your-bucket-name/path/to/config.json"
    }
  ]
}

If you need to copy an entire prefix (folder), broaden the resource to a prefix wildcard:

"Resource": "arn:aws:s3:::your-bucket-name/path/to/*"

Create the policy, role, and instance profile using the AWS CLI:

# 1. Create the IAM policy
aws iam create-policy \
  --policy-name EC2S3ReadConfigPolicy \
  --policy-document file://s3-read-policy.json \
  --region us-east-1

# 2. Create the IAM role with EC2 as the trusted principal
aws iam create-role \
  --role-name EC2S3ReadRole \
  --assume-role-policy-document '{
    "Version": "2012-10-17",
    "Statement": [{
      "Effect": "Allow",
      "Principal": {"Service": "ec2.amazonaws.com"},
      "Action": "sts:AssumeRole"
    }]
  }' \
  --region us-east-1

# 3. Attach the policy to the role
aws iam attach-role-policy \
  --role-name EC2S3ReadRole \
  --policy-arn arn:aws:iam::123456789012:policy/EC2S3ReadConfigPolicy

# 4. Create the instance profile
aws iam create-instance-profile \
  --instance-profile-name EC2S3ReadProfile

# 5. Add the role to the instance profile
aws iam add-role-to-instance-profile \
  --instance-profile-name EC2S3ReadProfile \
  --role-name EC2S3ReadRole

# 6. Attach the instance profile to a running EC2 instance
aws ec2 associate-iam-instance-profile \
  --instance-id i-0abcdef1234567890 \
  --iam-instance-profile Name=EC2S3ReadProfile \
  --region us-east-1

IAM changes propagate within seconds, but the instance metadata cache can hold stale credentials briefly. If you attach a role to a running instance and immediately test, wait 30–60 seconds before retrying.

Step 2: Run the aws s3 cp Command on the EC2 Instance

SSH into the instance and run the copy command. The destination path must already exist, or you must create it first — aws s3 cp does not create intermediate directories automatically.

# Create the destination directory if it doesn't exist
sudo mkdir -p /etc/myapp

# Copy a single file from S3 to the local filesystem
aws s3 cp s3://your-bucket-name/path/to/config.json /etc/myapp/config.json

To copy an entire prefix recursively:

aws s3 cp s3://your-bucket-name/path/to/ /etc/myapp/ --recursive
Think of the S3 URI as a file path on a remote drive. The bucket name is the drive label, and everything after the first / is the key. There are no real directories — only key prefixes that look like paths.

Useful flags for production use:

FlagPurpose
--region us-east-1Explicitly target the bucket's region; avoids redirect errors when the CLI default region differs
--sseRequest server-side encryption on upload (not applicable for download)
--no-progressSuppress progress output in scripts and user data
--recursiveCopy all objects under a prefix

Step 3: Diagnose When the Copy Command Fails

The error message from aws s3 cp is often misleading. An Access Denied response can mean the IAM policy is missing, the bucket policy explicitly denies the request, the object is in a different account, or the bucket has Block Public Access settings that override a permissive bucket policy. Working through these layers in order is the only reliable approach.

graph TD Start(["aws s3 cp fails"]) Start --> Q1{"Is an IAM role
attached to the instance?"} Q1 -- No --> Fix1["Associate instance profile
via aws ec2 associate-iam-instance-profile"] Q1 -- Yes --> Q2{"Does simulate-principal-policy
return 'allowed'?"} Q2 -- implicitDeny --> Fix2["Add s3:GetObject to
the role's IAM policy"] Q2 -- explicitDeny --> Fix3["Check SCP or permission boundary
for overriding Deny"] Q2 -- allowed --> Q3{"Does bucket policy
contain an explicit Deny?"} Q3 -- Yes --> Fix4["Remove or scope the Deny
in the bucket policy"] Q3 -- No --> Q4{"Is bucket in
another account?"} Q4 -- Yes --> Fix5["Bucket owner must grant
cross-account access in bucket policy"] Q4 -- No --> Fix6["Check VPC endpoint routing
and aws:sourceVpce conditions"]

Diagnostic Step 1: Confirm the instance has a role attached

Before checking policies, confirm the instance actually has credentials. A missing role produces an authentication error, not an access denied error — but both look similar in the CLI output. Query IMDS directly from the instance to verify a role is present.

# From inside the EC2 instance
# IMDSv2 (recommended)
TOKEN=$(curl -s -X PUT "http://169.254.169.254/latest/api/token" \
  -H "X-aws-ec2-metadata-token-ttl-seconds: 21600")
curl -s -H "X-aws-ec2-metadata-token: $TOKEN" \
  http://169.254.169.254/latest/meta-data/iam/security-credentials/

If the response is empty or returns a 404, no role is attached. Return to Step 1 and associate the instance profile.

Diagnostic Step 2: Verify the effective credentials and their scope

Knowing a role is attached doesn't confirm which role or what it allows. The CLI can report the caller identity, which tells you the exact role ARN in use — this catches cases where the wrong profile was attached.

aws sts get-caller-identity

Cross-reference the returned role ARN against the role you created. If they don't match, the wrong instance profile is attached.

Diagnostic Step 3: Simulate the IAM evaluation

IAM Policy Simulator can evaluate whether the attached role's policies allow s3:GetObject on the specific resource — without making a live S3 request. This isolates IAM as the failure layer before you touch bucket policies.

aws iam simulate-principal-policy \
  --policy-source-arn arn:aws:iam::123456789012:role/EC2S3ReadRole \
  --action-names s3:GetObject \
  --resource-arns arn:aws:s3:::your-bucket-name/path/to/config.json

A result of implicitDeny means the IAM policy is missing the permission. A result of explicitDeny means a Deny statement — possibly in an SCP or permission boundary — is overriding the Allow.

Diagnostic Step 4: Check the bucket policy for explicit denies

Even with a valid IAM policy, an explicit Deny in the bucket policy overrides the Allow. This is the layer most engineers skip, and it's the one that causes the most confusion — the IAM simulation passes, but the live request still fails.

aws s3api get-bucket-policy \
  --bucket your-bucket-name \
  --query Policy \
  --output text

Look for any "Effect": "Deny" statements that match your role's ARN or apply to "Principal": "*". A common pattern is a bucket policy that denies all requests not originating from a specific VPC endpoint — if your instance routes S3 traffic over the public internet rather than through a Gateway endpoint, the request is denied at the bucket policy layer regardless of IAM.

An explicit Deny in a bucket policy cannot be overridden by an IAM Allow. The evaluation order is fixed: explicit Deny wins, always.

Diagnostic Step 5: Check Block Public Access settings (cross-account scenarios)

If the bucket belongs to a different AWS account, Block Public Access settings on the bucket owner's account can reject requests even when both the IAM policy and bucket policy appear correct. This only applies to cross-account access patterns.

aws s3api get-bucket-policy-status \
  --bucket your-bucket-name

CLI examples omitted for cross-account Block Public Access verification — this setting is managed by the bucket-owner account and requires their credentials to inspect directly.

Experience Signal: The "Access Denied" That Wasn't an IAM Problem

A common misdiagnosis: the IAM policy looks correct, the policy simulator shows allowed, but aws s3 cp returns An error occurred (403) when calling the GetObject operation: Access Denied. The instinct is to broaden the IAM policy — add s3:*, expand the resource to *. The error persists.

The actual cause: the bucket had a condition in its bucket policy requiring requests to arrive via a specific VPC endpoint (aws:sourceVpce). The EC2 instance was in a public subnet with an internet gateway route to S3, so the request arrived over the public S3 endpoint. The bucket policy's condition evaluated to false, producing an explicit deny that the IAM simulator never saw — because the simulator evaluates IAM policies only, not bucket policies.

The fix was adding an S3 Gateway endpoint to the VPC and updating the instance's route table — not touching IAM at all. The diagnostic that revealed it was aws s3api get-bucket-policy, not the IAM simulator.

Depth Signal: s3:ListBucket and the Misleading 403 vs. 404

When s3:GetObject is allowed but s3:ListBucket is not, attempting to copy a key that doesn't exist returns a 403 Access Denied instead of a 404 Not Found. This is documented S3 behavior: without s3:ListBucket, S3 cannot confirm whether the key exists, so it returns 403 to avoid leaking information about the bucket's contents. Engineers frequently interpret this 403 as a permission problem and spend time adjusting IAM policies when the real issue is a typo in the S3 key path. If you receive a 403 on a key you're certain exists, verify the exact key name first with aws s3 ls s3://your-bucket-name/path/to/ — which requires s3:ListBucket on the bucket ARN (not the object ARN).

# s3:ListBucket must be granted on the bucket ARN, not the object ARN
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AllowGetObject",
      "Effect": "Allow",
      "Action": "s3:GetObject",
      "Resource": "arn:aws:s3:::your-bucket-name/path/to/config.json"
    },
    {
      "Sid": "AllowListBucket",
      "Effect": "Allow",
      "Action": "s3:ListBucket",
      "Resource": "arn:aws:s3:::your-bucket-name"
    }
  ]
}

Wrap-Up: Copying Files from S3 to EC2 Reliably

The aws s3 cp command itself is straightforward. The failure surface is almost entirely in the IAM and bucket policy layers beneath it. The pattern that works in production: attach a least-privilege role before the instance launches (via launch template or Auto Scaling group), verify credentials with aws sts get-caller-identity on first boot, and use aws iam simulate-principal-policy plus aws s3api get-bucket-policy as your two-step diagnostic when access is denied.

For further reading, see the AWS CLI s3 cp reference and the IAM policy evaluation logic documentation.

Glossary

TermDefinition
Instance ProfileThe container that associates an IAM role with an EC2 instance, enabling the instance to retrieve temporary credentials from IMDS.
IMDS (Instance Metadata Service)A link-local HTTP endpoint (169.254.169.254) available inside every EC2 instance that provides instance metadata and temporary IAM credentials.
s3:GetObjectThe IAM action required to download (read) an object from S3. Distinct from s3:ListBucket, which is required to enumerate objects.
Explicit DenyAn IAM or bucket policy statement with "Effect": "Deny" that overrides any Allow, regardless of source.
S3 Gateway EndpointA VPC endpoint type that routes S3 traffic through the AWS private network rather than the public internet, often required by restrictive bucket policies.

Related Posts

Comments

Post a Comment

Popular posts from this blog

EC2 No Internet Access in Custom VPC: Fix Internet Gateway and Route Table

EC2 no internet access in a custom VPC is one of the most common launch-day failures — the instance is running, but outbound connectivity is completely silent. TL;DR: EC2 No Internet Access — Fix Checklist Step What to Do Why It Matters Complexity 1 Create and attach an Internet Gateway to the VPC Without an IGW, the VPC has no path to the public internet Low 2 Add a default route (0.0.0.0/0) pointing to the IGW in the subnet's route table The IGW exists but traffic has no route to reach it Low 3 Assign a public IP or Elastic IP to the instance Private IPs are not routable on the public internet Low 4 Verify Security Group allows outbound traffic A restrictive outbound rule silently drops all egress Low 5 Verify Network ...
Read more

EC2 SSH Connection Timeout: Which Security Group Rules to Check

An EC2 SSH connection timeout is one of the most common blockers when launching a new instance — the fix is almost always a missing or misconfigured inbound rule in your Security Group. TL;DR — Diagnostic Steps at a Glance Step What to Check Common Fix 1 Security Group inbound rules Add SSH (TCP 22) from your IP 2 Subnet route table Ensure 0.0.0.0/0 routes to an Internet Gateway 3 Network ACL rules Allow inbound TCP 22 and ephemeral return ports 4 Instance public IP / Elastic IP Confirm a public IP is assigned 5 Key pair and SSH username Use correct private key and AMI-specific username Why SSH Times Out Instead of Refusing A connection refused error means the TCP handshake reached the host but nothing is listening. A connection timed out error means packets never arrived — or replies never returned. On EC2, that silence is almost always a network-layer block, not an OS-level problem. The Se...
Read more

Difference Between IAM User and IAM Role: Which One Should Your EC2 Use?

Understanding the difference between IAM User and IAM Role is critical — choosing wrong means either hardcoded credentials in your app or a silent security exposure you won't catch until an audit. TL;DR: IAM User vs IAM Role at a Glance Dimension IAM User IAM Role Identity type Permanent, named human or service identity Assumable identity with temporary credentials Credential type Long-lived access keys or password Short-lived STS tokens (auto-rotated) Attached to EC2? No native attachment — keys must be embedded Yes — via Instance Profile, injected automatically Rotation required? Manual — your responsibility Automatic — AWS STS handles it Recommended for EC2 → S3? No Yes — always Cross-account access Requires separate user per account Native — assume role across accounts Audit trail Tied to user identity Tied to role + session name in CloudTrail What IAM User and IAM Role Actual...
Read more