Why CloudWatch Doesn't Show EC2 Memory Usage (And How to Fix It)

You've launched an EC2 instance, opened the CloudWatch console, and found CPU, network, and disk I/O metrics — but no RAM usage. This isn't a bug or an oversight; it's a deliberate architectural boundary that every AWS engineer must understand.

TL;DR

Aspect Default CloudWatch Metrics CloudWatch Agent Metrics
Source AWS Hypervisor layer Guest OS (inside the instance)
Memory (RAM) ❌ Not available ✅ mem_used_percent, mem_available
Disk Space ❌ Not available ✅ disk_used_percent per mount point
CPU Usage ✅ CPUUtilization (hypervisor view) ✅ cpu_usage_user, cpu_usage_system
Installation Required None — automatic CloudWatch Agent on the instance
IAM Permissions None — AWS-managed CloudWatchAgentServerPolicy on instance role
Cost Included in EC2 Custom metrics pricing applies

The Architectural Boundary: Why the Gap Exists

AWS EC2 runs on a virtualization layer called the Nitro Hypervisor (or Xen for older instance types). CloudWatch's default metrics are emitted by this hypervisor — the host-level infrastructure AWS controls. The hypervisor can observe how much CPU time it allocates to your VM, how many bytes traverse the virtual network interface, and how many I/O operations hit the virtual disk. What it cannot see is what happens inside the guest OS — how the Linux kernel distributes RAM across processes, or how full your /var partition is.

Analogy: Think of AWS as a landlord who owns the building. The landlord can measure how much electricity the apartment consumes (hypervisor metrics) but cannot see how many lights you've left on in each room (OS-level memory/disk). To get that detail, you need a sensor inside the apartment — that sensor is the CloudWatch Agent.

This is the core principle: the hypervisor boundary is the visibility boundary. Anything below it (infrastructure) AWS can measure automatically. Anything above it (OS internals) requires an agent running with OS-level access.

Data Flow: Default vs. Agent-Collected Metrics

graph TD subgraph AWS_Infrastructure ["AWS Host Infrastructure"] HV["Nitro Hypervisor"] end subgraph EC2_Instance ["EC2 Instance (Guest OS)"] OS["Linux / Windows OS"] PROC["/proc/meminfo
df, iostat"] AGENT["CloudWatch Agent
(Daemon)"] OS --> PROC PROC --> AGENT end subgraph CloudWatch ["Amazon CloudWatch"] NS1["Namespace: AWS/EC2
(CPUUtilization, NetworkIn)"] NS2["Namespace: CWAgent
(mem_used_percent, disk_used_percent)"] end HV -- "Hypervisor-level metrics
(automatic, free)" --> NS1 AGENT -- "OS-level metrics
(custom, billed)" --> NS2 HV -. "Cannot see inside
guest OS" .-> OS style AWS_Infrastructure fill:#1a1a2e,stroke:#4a90d9,color:#fff style EC2_Instance fill:#16213e,stroke:#4a90d9,color:#fff style CloudWatch fill:#0f3460,stroke:#4a90d9,color:#fff
  1. Nitro Hypervisor continuously monitors the virtual machine's resource consumption from the host perspective and pushes metrics like CPUUtilization and NetworkIn directly to CloudWatch — no configuration needed.
  2. Guest OS (your Linux/Windows instance) holds the ground truth for memory and disk. This data lives in kernel space (/proc/meminfo, df output) and is inaccessible to the hypervisor.
  3. CloudWatch Agent runs as a daemon inside the guest OS, reads OS-level metrics at a configurable interval, and publishes them to CloudWatch under the CWAgent namespace as custom metrics.
  4. CloudWatch stores both streams — default hypervisor metrics and agent-pushed custom metrics — allowing unified dashboards and alarms.

Implementation: Installing and Configuring the CloudWatch Agent

Step 1: Attach the Required IAM Policy

The instance needs permission to write metrics to CloudWatch. Attach the AWS-managed policy CloudWatchAgentServerPolicy to the EC2 instance's IAM role. This follows least-privilege — it grants only cloudwatch:PutMetricData, ec2:DescribeTags, and SSM read access needed by the agent.

# Attach via CLI (replace ROLE_NAME with your instance role)
aws iam attach-role-policy \
  --role-name ROLE_NAME \
  --policy-arn arn:aws:iam::aws:policy/CloudWatchAgentServerPolicy

Step 2: Install the CloudWatch Agent

# Amazon Linux 2 / Amazon Linux 2023
sudo yum install -y amazon-cloudwatch-agent

# Ubuntu / Debian
wget https://s3.amazonaws.com/amazoncloudwatch-agent/ubuntu/amd64/latest/amazon-cloudwatch-agent.deb
sudo dpkg -i amazon-cloudwatch-agent.deb

Step 3: Create the Agent Configuration

The agent is configured via a JSON file. The wizard (amazon-cloudwatch-agent-config-wizard) can generate this interactively, or you can write it directly. Below is a production-ready configuration capturing memory and disk metrics:

🔽 [Click to expand] — /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.json 
{
  "agent": {
    "metrics_collection_interval": 60,
    "run_as_user": "cwagent"
  },
  "metrics": {
    "append_dimensions": {
      "AutoScalingGroupName": "${aws:AutoScalingGroupName}",
      "ImageId": "${aws:ImageId}",
      "InstanceId": "${aws:InstanceId}",
      "InstanceType": "${aws:InstanceType}"
    },
    "metrics_collected": {
      "mem": {
        "measurement": [
          "mem_used_percent",
          "mem_available_percent",
          "mem_used",
          "mem_total"
        ],
        "metrics_collection_interval": 60
      },
      "disk": {
        "measurement": [
          "disk_used_percent",
          "disk_free",
          "disk_used"
        ],
        "metrics_collection_interval": 60,
        "resources": [
          "/",
          "/var",
          "/tmp"
        ]
      },
      "swap": {
        "measurement": [
          "swap_used_percent"
        ]
      }
    }
  }
}

Step 4: Start the Agent

# Load config and start the agent
sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl \
  -a fetch-config \
  -m ec2 \
  -c file:/opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.json \
  -s

# Verify the agent is running
sudo systemctl status amazon-cloudwatch-agent

Step 5: Verify Metrics in CloudWatch

After 1–2 minutes, navigate to CloudWatch → Metrics → CWAgent. You will see metrics dimensioned by InstanceId and (for disk) by path. Create an alarm on mem_used_percent exceeding 85% as a baseline production alert.

Metric Namespace Architecture

graph LR subgraph CW ["Amazon CloudWatch"] subgraph NS1 ["Namespace: AWS/EC2"] M1["CPUUtilization"] M2["NetworkIn / NetworkOut"] M3["DiskReadOps / DiskWriteOps"] end subgraph NS2 ["Namespace: CWAgent"] M4["mem_used_percent"] M5["mem_available_percent"] M6["disk_used_percent
dim: path=/"] M7["swap_used_percent"] end DASH["CloudWatch Dashboard
(Unified View)"] ALARM["CloudWatch Alarms"] end NS1 --> DASH NS2 --> DASH NS2 --> ALARM NS1 --> ALARM style NS1 fill:#1a3a1a,stroke:#4caf50,color:#fff style NS2 fill:#3a1a1a,stroke:#f44336,color:#fff style DASH fill:#1a2a3a,stroke:#4a90d9,color:#fff style ALARM fill:#2a1a3a,stroke:#9c27b0,color:#fff
  1. AWS/EC2 namespace contains all hypervisor-emitted metrics. These are free, automatic, and require no agent.
  2. CWAgent namespace is where the CloudWatch Agent publishes OS-level metrics. This is a custom metrics namespace and is billed per metric per month.
  3. Dimensions on agent metrics include InstanceId, InstanceType, and for disk metrics, the path (mount point) — enabling per-partition alerting.
  4. Both namespaces can be combined in a single CloudWatch Dashboard for a unified operational view.

Setting a Memory Alarm (CLI)

aws cloudwatch put-metric-alarm \
  --alarm-name "EC2-HighMemoryUsage" \
  --alarm-description "Triggers when memory usage exceeds 85%" \
  --metric-name mem_used_percent \
  --namespace CWAgent \
  --statistic Average \
  --period 300 \
  --threshold 85 \
  --comparison-operator GreaterThanThreshold \
  --evaluation-periods 2 \
  --dimensions Name=InstanceId,Value=i-0123456789abcdef0 \
  --alarm-actions arn:aws:sns:us-east-1:123456789012:ops-alerts

Glossary

Term Definition
Nitro Hypervisor AWS's lightweight hypervisor that underpins modern EC2 instances, responsible for virtualizing CPU, memory, and I/O at the host level.
Guest OS The operating system running inside the EC2 virtual machine — Linux or Windows — which has full visibility into its own memory and filesystem state.
CloudWatch Namespace A logical container for CloudWatch metrics (e.g., AWS/EC2, CWAgent) that prevents naming collisions between different services or agents.
Custom Metrics Metrics published to CloudWatch by user-controlled processes (like the CloudWatch Agent), as opposed to metrics emitted automatically by AWS services.
CloudWatchAgentServerPolicy AWS-managed IAM policy granting an EC2 instance the minimum permissions required to publish metrics and retrieve configuration via the CloudWatch Agent.

Next Steps

For fleet-scale deployments, use AWS Systems Manager (SSM) Distributor to install and configure the CloudWatch Agent across hundreds of instances without SSH access. Store your agent configuration in SSM Parameter Store under AmazonCloudWatch-linux and reference it with -c ssm:parameter-name for centralized config management. Official reference: AWS CloudWatch Agent Documentation.

Comments

Post a Comment

Popular posts from this blog

EC2 No Internet Access in Custom VPC: Attaching an Internet Gateway and Fixing Route Tables

Launching an EC2 instance in a custom VPC only to find it cannot reach the internet is one of the most common networking pitfalls in AWS — and it almost always comes down to two missing pieces: an Internet Gateway (IGW) and a correctly configured Route Table. This post walks you through the exact architecture, the root cause, and the precise steps to fix it. TL;DR Step Action Why It Matters 1 Create and attach an Internet Gateway to your VPC Without an IGW, there is no path between your VPC and the public internet 2 Add a 0.0.0.0/0 route pointing to the IGW in your subnet's Route Table The route table is the traffic director — without this entry, outbound packets have nowhere to go 3 Assign a Public IP or Elastic IP to the EC2 instance The IGW performs NAT only for instances with a public IP address 4 Verify Security Group allows outbound traffic Security Groups are stateful firewalls; outbound rules...
Read more

IAM User vs. IAM Role: Why Your EC2 Instance Should Never Use a User

Choosing between an IAM User and an IAM Role is one of the most consequential security decisions in AWS — get it wrong and you either expose long-lived credentials or block your application entirely. This post cuts through the confusion and gives you a definitive answer for the EC2-to-S3 access pattern. TL;DR — Quick Comparison Dimension IAM User IAM Role Identity type A person or a static service account An assumable identity for AWS services, users, or external principals Credential type Long-lived Access Key ID + Secret Access Key Short-lived, auto-rotating STS tokens (max 12 h by default) Credential rotation Manual — you must rotate keys yourself Automatic — AWS rotates them transparently Stored on EC2? Yes — in ~/.aws/credentials or env vars (risky) No — delivered via the In...
Read more

Lambda Infinite Loop with S3: How to Prevent Recursive Triggers

Writing a processed file back to the same S3 bucket that triggered your Lambda function creates a recursive loop — each output file fires a new invocation, which writes another file, ad infinitum. Left unchecked, this silently drains your AWS budget and exhausts Lambda concurrency limits within minutes. TL;DR Strategy Mechanism Best For Complexity Separate Output Prefix/Bucket S3 event filter on prefix Simple pipelines Low Object Metadata Check Lambda reads S3 object metadata before processing Single-bucket requirement Medium Separate Output Bucket Trigger only on source bucket Clean architectural separation Low DynamoDB Idempotency Guard Track processed ETag/key in DynamoDB Exactly-once guarantees Medium-High Understanding the Recursive Loop Before fixing the problem, you need to understand exactly why it happens. The S3 → Lambda trigger is an event-driven subscription: S3 publishes an ObjectCreat...
Read more