Finding Who Deleted an EC2 Instance Using CloudTrail Event History

An EC2 instance disappears without warning — no ticket, no change request, no explanation. Before you can prevent it from happening again, you need to answer one question: who did it? AWS CloudTrail is your forensic audit log, and knowing how to query it efficiently is a non-negotiable skill for any cloud engineer or security responder.

TL;DR

Step Action Where
1 Open CloudTrail Event History AWS Console → CloudTrail → Event History
2 Filter by Event Name: TerminateInstances Event History filter bar
3 Narrow by time range and Resource ID Date picker + Resource ID filter
4 Inspect the event record for userIdentity Event detail JSON
5 (Optional) Query via AWS CLI or Athena for scale Terminal / Athena Console

How CloudTrail Captures EC2 Termination Events

Every AWS API call — including ec2:TerminateInstances — is recorded by CloudTrail as a structured JSON event. CloudTrail Event History retains the last 90 days of management events (control-plane API calls) at no additional charge, per region, per account. For longer retention or cross-account aggregation, you need a Trail writing to S3.

Analogy: Think of CloudTrail as the keycard access log in a secure building. Every door opened (API call made), by every badge holder (IAM principal), at every timestamp, is recorded. When a server goes missing from the server room, you don't guess — you pull the log.

The Data Flow: From API Call to Audit Record

graph LR A["IAM Principal
(User / Role / Service)"] -->|"ec2:TerminateInstances API Call"| B["EC2 API Endpoint"] B -->|"Executes Termination"| C["EC2 Instance
(Shutting Down)"] B -->|"Management Event Recorded"| D["CloudTrail"] D -->|"Queryable for 90 days"| E["Event History
(Console / CLI)"] D -->|"If Trail configured"| F["S3 Bucket
(Long-term Retention)"] F -->|"SQL Query"| G["Amazon Athena"] D -->|"If Trail + EventBridge rule"| H["EventBridge → SNS
(Real-time Alert)"]
  1. IAM Principal (user, role, or service) calls the EC2 TerminateInstances API — via Console, CLI, SDK, or automation.
  2. EC2 API Endpoint receives and authorizes the request, then executes the termination.
  3. CloudTrail intercepts the management event and writes a structured JSON record containing the caller identity, source IP, timestamp, request parameters, and response.
  4. Event History makes the last 90 days of events queryable in the console without any Trail configuration.
  5. If a Trail is configured, the same event is also delivered to an S3 bucket (and optionally CloudWatch Logs or EventBridge) for long-term retention and alerting.

Step-by-Step: Console Investigation

Step 1 — Navigate to CloudTrail Event History

Go to AWS Console → CloudTrail → Event History. Ensure you are in the correct AWS region where the EC2 instance existed. CloudTrail Event History is region-scoped for regional services.

Step 2 — Filter by Event Name

In the filter dropdown, select Event name and type TerminateInstances. This is the exact EC2 API action name recorded by CloudTrail. Set your time range to bracket the suspected deletion window.

Step 3 — Identify the Specific Instance

If multiple terminations appear, add a second filter: select Resource ID and enter the instance ID (e.g., i-0abc1234def567890). This narrows results to events that reference that specific resource.

Step 4 — Inspect the Event Record

Click the matching event row to expand it, then click View event to see the full JSON. The critical fields are:

graph TD ROOT["CloudTrail Event JSON"] --> UI["userIdentity"] ROOT --> ET["eventTime"] ROOT --> SIP["sourceIPAddress"] ROOT --> RP["requestParameters"] ROOT --> RE["responseElements"] UI --> UT["type
(IAMUser / AssumedRole / Root)"] UI --> UA["arn
(Full IAM ARN of caller)"] UI --> USC["sessionContext
(For AssumedRole: reveals base role)"] RP --> IS["instancesSet.items
(List of targeted instance IDs)"] RE --> CS["currentState
(shutting-down / terminated)"] RE --> PS["previousState
(running / stopped)"]
  1. userIdentity.type — Identifies the principal type: IAMUser, AssumedRole, Root, AWSService, etc.
  2. userIdentity.arn — The full ARN of the caller (e.g., arn:aws:iam::123456789012:user/john.doe or arn:aws:sts::123456789012:assumed-role/DevOpsRole/session-name).
  3. userIdentity.principalId — A unique identifier for the principal; useful when the ARN alone is ambiguous for assumed roles.
  4. sourceIPAddress — The IP address from which the call originated. For calls made by AWS services on your behalf, this will be an AWS service DNS name.
  5. eventTime — The exact UTC timestamp of the API call.
  6. requestParameters.instancesSet — Lists the instance IDs that were targeted by the termination request.
🔽 Example CloudTrail Event JSON for TerminateInstances 
{
  "eventVersion": "1.08",
  "userIdentity": {
    "type": "AssumedRole",
    "principalId": "AROAEXAMPLEID:session-name",
    "arn": "arn:aws:sts::123456789012:assumed-role/DevOpsRole/session-name",
    "accountId": "123456789012",
    "sessionContext": {
      "sessionIssuer": {
        "type": "Role",
        "principalId": "AROAEXAMPLEID",
        "arn": "arn:aws:iam::123456789012:role/DevOpsRole",
        "accountId": "123456789012",
        "userName": "DevOpsRole"
      },
      "attributes": {
        "creationDate": "2024-01-15T10:00:00Z",
        "mfaAuthenticated": "false"
      }
    }
  },
  "eventTime": "2024-01-15T14:32:11Z",
  "eventSource": "ec2.amazonaws.com",
  "eventName": "TerminateInstances",
  "awsRegion": "us-east-1",
  "sourceIPAddress": "203.0.113.42",
  "userAgent": "aws-cli/2.13.0",
  "requestParameters": {
    "instancesSet": {
      "items": [
        { "instanceId": "i-0abc1234def567890" }
      ]
    }
  },
  "responseElements": {
    "instancesSet": {
      "items": [
        {
          "instanceId": "i-0abc1234def567890",
          "currentState": { "code": 32, "name": "shutting-down" },
          "previousState": { "code": 16, "name": "running" }
        }
      ]
    }
  },
  "eventID": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
  "readOnly": false,
  "eventType": "AwsApiCall",
  "managementEvent": true,
  "recipientAccountId": "123456789012"
}

Step-by-Step: AWS CLI Investigation

The console is useful for quick lookups, but the AWS CLI gives you scriptable, repeatable forensics. Use aws cloudtrail lookup-events to query Event History programmatically.

🔽 AWS CLI: Look up TerminateInstances events 
# Look up all TerminateInstances events in the last 90 days (region-scoped)
aws cloudtrail lookup-events \
  --lookup-attributes AttributeKey=EventName,AttributeValue=TerminateInstances \
  --start-time "2024-01-15T00:00:00Z" \
  --end-time "2024-01-16T00:00:00Z" \
  --region us-east-1 \
  --query 'Events[*].{Time:EventTime,User:Username,EventId:EventId}' \
  --output table

# Get the full JSON of a specific event by EventId
aws cloudtrail lookup-events \
  --lookup-attributes AttributeKey=EventId,AttributeValue=a1b2c3d4-e5f6-7890-abcd-ef1234567890 \
  --region us-east-1 \
  --query 'Events[0].CloudTrailEvent' \
  --output text | python3 -m json.tool

# Filter by Resource (instance ID) — note: lookup-events supports one attribute at a time
aws cloudtrail lookup-events \
  --lookup-attributes AttributeKey=ResourceName,AttributeValue=i-0abc1234def567890 \
  --region us-east-1 \
  --output json

Important constraint: lookup-events only supports a single --lookup-attributes filter per call. To combine filters (e.g., EventName AND ResourceName), retrieve results by one attribute and post-filter with jq or Python.

🔽 Post-filter with jq: Find termination of a specific instance 
aws cloudtrail lookup-events \
  --lookup-attributes AttributeKey=EventName,AttributeValue=TerminateInstances \
  --region us-east-1 \
  --output json | \
jq '.Events[] | 
  select(
    (.CloudTrailEvent | fromjson | .requestParameters.instancesSet.items[].instanceId) 
    == "i-0abc1234def567890"
  ) | {
    eventTime: .EventTime,
    user: .Username,
    sourceIP: (.CloudTrailEvent | fromjson | .sourceIPAddress),
    userArn: (.CloudTrailEvent | fromjson | .userIdentity.arn)
  }'

Beyond 90 Days: Querying a Trail with Athena

If the deletion occurred more than 90 days ago, or you need to query across multiple accounts/regions, you need a CloudTrail Trail delivering logs to S3, queried via Amazon Athena.

🔽 Athena SQL: Find TerminateInstances for a specific instance 
-- Assumes you have created an Athena table over your CloudTrail S3 bucket
-- using the AWS-provided CloudTrail table DDL or AWS Glue crawler

SELECT
  eventtime,
  useridentity.arn          AS caller_arn,
  useridentity.type         AS caller_type,
  sourceipaddress,
  useragent,
  json_extract_scalar(
    requestparameters,
    '$.instancesSet.items[0].instanceId'
  )                         AS instance_id
FROM cloudtrail_logs
WHERE
  eventsource    = 'ec2.amazonaws.com'
  AND eventname  = 'TerminateInstances'
  AND requestparameters LIKE '%i-0abc1234def567890%'
  AND eventtime BETWEEN '2024-01-01T00:00:00Z' AND '2024-01-31T23:59:59Z'
ORDER BY eventtime DESC;

IAM Permissions Required for This Investigation

Apply least privilege — the investigator only needs read access to CloudTrail, not write or administrative permissions.

🔽 Minimal IAM Policy for CloudTrail Forensics 
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "CloudTrailReadOnly",
      "Effect": "Allow",
      "Action": [
        "cloudtrail:LookupEvents",
        "cloudtrail:GetTrailStatus",
        "cloudtrail:DescribeTrails",
        "cloudtrail:GetEventSelectors"
      ],
      "Resource": "*"
    },
    {
      "Sid": "S3ReadForTrailLogs",
      "Effect": "Allow",
      "Action": [
        "s3:GetObject",
        "s3:ListBucket"
      ],
      "Resource": [
        "arn:aws:s3:::your-cloudtrail-bucket",
        "arn:aws:s3:::your-cloudtrail-bucket/*"
      ]
    },
    {
      "Sid": "AthenaQueryForTrailLogs",
      "Effect": "Allow",
      "Action": [
        "athena:StartQueryExecution",
        "athena:GetQueryExecution",
        "athena:GetQueryResults"
      ],
      "Resource": "arn:aws:athena:us-east-1:123456789012:workgroup/primary"
    }
  ]
}

Preventive Controls: Stop the Next Deletion

Forensics tells you what happened. These controls prevent recurrence:

  • EC2 Termination Protection: Enable DisableApiTermination on critical instances. This requires an explicit API call to disable protection before termination can succeed — creating an additional, auditable step.
  • SCP (Service Control Policy): In AWS Organizations, use an SCP to deny ec2:TerminateInstances for production accounts except via a specific approved role.
  • EventBridge Rule + SNS Alert: Create an EventBridge rule that matches TerminateInstances CloudTrail events and sends an immediate SNS notification to your on-call channel.
  • IAM Permission Boundaries: Restrict which roles can call ec2:TerminateInstances using permission boundaries, limiting blast radius from compromised credentials.

Glossary

Term Definition
CloudTrail Event History A 90-day, region-scoped, no-configuration-required log of management (control-plane) API calls in your AWS account.
Management Event An API call that creates, modifies, or deletes AWS resources (e.g., TerminateInstances, DeleteBucket). Distinct from data events (e.g., S3 GetObject).
userIdentity The CloudTrail JSON field that identifies the IAM principal (user, role, service) that made the API call, including their ARN and session context.
AssumedRole A userIdentity.type value indicating the caller authenticated via sts:AssumeRole. The sessionContext.sessionIssuer field reveals the underlying IAM role.
CloudTrail Trail A configuration that continuously delivers CloudTrail events to an S3 bucket (and optionally CloudWatch Logs), enabling retention beyond 90 days and cross-region/cross-account aggregation.

Next Steps

You now have a repeatable forensic playbook for tracing any destructive API action in AWS. As immediate next steps: (1) verify your account has at least one multi-region Trail writing to S3 for retention beyond 90 days, (2) enable CloudTrail log file integrity validation to ensure logs haven't been tampered with, and (3) consider enabling AWS Config alongside CloudTrail — Config records resource state changes while CloudTrail records API calls, giving you complementary forensic views.

Comments

Post a Comment

Popular posts from this blog

EC2 No Internet Access in Custom VPC: Attaching an Internet Gateway and Fixing Route Tables

Launching an EC2 instance in a custom VPC only to find it cannot reach the internet is one of the most common networking pitfalls in AWS — and it almost always comes down to two missing pieces: an Internet Gateway (IGW) and a correctly configured Route Table. This post walks you through the exact architecture, the root cause, and the precise steps to fix it. TL;DR Step Action Why It Matters 1 Create and attach an Internet Gateway to your VPC Without an IGW, there is no path between your VPC and the public internet 2 Add a 0.0.0.0/0 route pointing to the IGW in your subnet's Route Table The route table is the traffic director — without this entry, outbound packets have nowhere to go 3 Assign a Public IP or Elastic IP to the EC2 instance The IGW performs NAT only for instances with a public IP address 4 Verify Security Group allows outbound traffic Security Groups are stateful firewalls; outbound rules...
Read more

IAM User vs. IAM Role: Why Your EC2 Instance Should Never Use a User

Choosing between an IAM User and an IAM Role is one of the most consequential security decisions in AWS — get it wrong and you either expose long-lived credentials or block your application entirely. This post cuts through the confusion and gives you a definitive answer for the EC2-to-S3 access pattern. TL;DR — Quick Comparison Dimension IAM User IAM Role Identity type A person or a static service account An assumable identity for AWS services, users, or external principals Credential type Long-lived Access Key ID + Secret Access Key Short-lived, auto-rotating STS tokens (max 12 h by default) Credential rotation Manual — you must rotate keys yourself Automatic — AWS rotates them transparently Stored on EC2? Yes — in ~/.aws/credentials or env vars (risky) No — delivered via the In...
Read more

Lambda Infinite Loop with S3: How to Prevent Recursive Triggers

Writing a processed file back to the same S3 bucket that triggered your Lambda function creates a recursive loop — each output file fires a new invocation, which writes another file, ad infinitum. Left unchecked, this silently drains your AWS budget and exhausts Lambda concurrency limits within minutes. TL;DR Strategy Mechanism Best For Complexity Separate Output Prefix/Bucket S3 event filter on prefix Simple pipelines Low Object Metadata Check Lambda reads S3 object metadata before processing Single-bucket requirement Medium Separate Output Bucket Trigger only on source bucket Clean architectural separation Low DynamoDB Idempotency Guard Track processed ETag/key in DynamoDB Exactly-once guarantees Medium-High Understanding the Recursive Loop Before fixing the problem, you need to understand exactly why it happens. The S3 → Lambda trigger is an event-driven subscription: S3 publishes an ObjectCreat...
Read more