Finding Who Deleted an EC2 Instance Using CloudTrail Event History

An EC2 instance disappears without warning, and the on-call engineer's first question is always the same: who ran that termination? Tracing the CloudTrail TerminateInstances event through Event History is the fastest path from mystery to accountability — no third-party tooling required.

TL;DR: CloudTrail TerminateInstances Lookup

Step	Action	What You Get
1	Open CloudTrail → Event History	Pre-indexed, 90-day searchable log
2	Filter by Event name: `TerminateInstances`	All termination events in the region
3	Filter by Resource ID (instance ID)	Scoped to the specific instance
4	Inspect the event record	IAM principal, source IP, time, request parameters

How CloudTrail Captures EC2 Termination Events

Every AWS API call made against EC2 — including TerminateInstances — is recorded as a management event in CloudTrail. The service captures the caller identity, the time of the call, the source IP address, the AWS region, and the full request and response parameters. This data is written to Event History automatically for management events, with a 90-day retention window, at no additional charge. You do not need a trail configured to access Event History, but a trail is required if you need retention beyond 90 days or cross-region aggregation.

The key field for attribution is userIdentity. This JSON block inside the event record tells you whether the call came from an IAM user, an assumed role session (including EC2 instance profiles, Lambda execution roles, or federated identities), the AWS account root user, or an AWS service acting on your behalf. Understanding this distinction matters — a termination triggered by an Auto Scaling scale-in action will show a service principal, not a human user.

sequenceDiagram participant P as IAM Principal participant EC2 as EC2 API participant CT as CloudTrail participant EH as Event History (90d) participant S3 as S3 Trail (optional) P->>EC2: TerminateInstances(i-xxxx) EC2-->>P: Response (terminating) EC2->>CT: Management event emitted CT->>EH: Indexed within ~15 min CT-->>S3: Delivered if trail configured Note over EH: Queryable via Console
or lookup-events API

API Call: A principal calls ec2:TerminateInstances via Console, CLI, or SDK.
CloudTrail ingestion: The management event is captured and indexed within approximately 15 minutes.
Event History: The event becomes queryable in the Console and via the Lookup API for up to 90 days.
S3 Trail (optional): If a trail is configured, the same event is also delivered to S3 for long-term retention and Athena querying.

Step-by-Step: Tracing the TerminateInstances Event

Step 1 — Identify the Instance ID

Before querying CloudTrail, confirm the instance ID of the terminated resource. If the instance is gone from the EC2 console, check your monitoring system, deployment tags, or the EC2 console with the filter set to Terminated instances (they remain visible for a short period after termination).

aws ec2 describe-instances \
  --filters "Name=instance-state-name,Values=terminated" \
  --query "Reservations[*].Instances[*].{ID:InstanceId,State:State.Name,Name:Tags[?Key=='Name']|[0].Value}" \
  --output table \
  --region us-east-1

Operator Rationale: Terminated instances remain in the EC2 API response for a limited time. Capturing the instance ID now prevents a dead end in the CloudTrail query.

Step 2 — Query CloudTrail Event History via Console

Navigate to CloudTrail → Event History in the AWS Console. Set the following filters:

Lookup attribute: Event name
Value: TerminateInstances
Time range: Narrow to the window when the instance disappeared

Once you locate the event, click it to expand the full JSON record. The userIdentity block and requestParameters.instancesSet contain the principal and the targeted instance IDs respectively.

Operator Rationale: Filtering by event name first returns all terminations in the region. Cross-referencing the instance ID inside the event JSON confirms you have the right record when multiple terminations occurred in the same window.

Step 3 — Query via AWS CLI (Recommended for Precision)

The CLI lookup-events command supports filtering by event name and returns structured JSON you can pipe directly into jq for extraction.

aws cloudtrail lookup-events \
  --lookup-attributes AttributeKey=EventName,AttributeValue=TerminateInstances \
  --start-time "2024-01-15T00:00:00Z" \
  --end-time "2024-01-16T00:00:00Z" \
  --region us-east-1 \
  --output json

Operator Rationale: The --start-time and --end-time flags are ISO 8601 UTC. Omitting them returns the most recent events but may miss the target if the termination occurred days ago. Always bound the time range when you know the approximate window.

Step 4 — Extract the Principal and Instance ID

Pipe the output through jq to surface only the fields that matter for attribution:

aws cloudtrail lookup-events \
  --lookup-attributes AttributeKey=EventName,AttributeValue=TerminateInstances \
  --start-time "2024-01-15T00:00:00Z" \
  --end-time "2024-01-16T00:00:00Z" \
  --region us-east-1 \
  --output json | jq '.Events[].CloudTrailEvent | fromjson | {
    eventTime: .eventTime,
    principal: .userIdentity,
    sourceIP: .sourceIPAddress,
    userAgent: .userAgent,
    instances: .requestParameters.instancesSet.items
  }'

Operator Rationale: The CloudTrailEvent field is a JSON string embedded inside the outer JSON response. The fromjson call in jq parses it inline. Without this step, the attribution fields are not directly accessible.

Step 5 — Interpret the userIdentity Block

The userIdentity structure varies by caller type. The table below maps the common types to what you will see in the event record:

Caller Type	userIdentity.type	Key Attribution Field
IAM User	`IAMUser`	`userName`
Assumed Role (human via SSO/federation)	`AssumedRole`	`sessionContext.sessionIssuer.userName` + `arn`
EC2 Instance Profile / Lambda Role	`AssumedRole`	`sessionContext.sessionIssuer.arn` (role ARN)
AWS Service (e.g., Auto Scaling)	`AWSService`	`invokedBy`
Root Account	`Root`	`accountId`

Think of userIdentity as the badge scan log at a building entrance. An IAM user is someone who badged in directly. An assumed role session is a contractor who borrowed a temporary access card issued under a named role — the card shows who issued it, not necessarily who is holding it. The sessionContext is the issuing record.

When the type is AssumedRole, the arn field contains the full session ARN including the role session name — for example, arn:aws:sts::123456789012:assumed-role/DeployRole/jane.doe@example.com. If your organization uses IAM Identity Center, the session name is typically the user's email or username, making attribution straightforward even without IAM user accounts.

Experience Signal: The Auto Scaling Misdiagnosis

A common failure pattern: the CloudTrail event shows userIdentity.type: AssumedRole with an ARN pointing to a role named something like AWSServiceRoleForAutoScaling. The first instinct is to assume a human assumed that role. In practice, this is the Auto Scaling service acting on a scale-in policy — the invokedBy field will read autoscaling.amazonaws.com.

The misdiagnosis leads engineers to audit IAM users and find nothing. The actual fix is to inspect the Auto Scaling group's scaling activity history, which will show the scale-in event that triggered the termination. The CloudTrail event and the Auto Scaling activity log together form the complete picture.

aws autoscaling describe-scaling-activities \
  --auto-scaling-group-name your-asg-name \
  --region us-east-1 \
  --output table

Operator Rationale: Auto Scaling termination events appear in CloudTrail under the service role, not under a human principal. Always check invokedBy before concluding a human was responsible.

Depth Signal: ResourceId Lookup Limitation

The CloudTrail lookup-events API supports filtering by ResourceName (which accepts an instance ID), but this lookup searches the Resources array in the event record. For TerminateInstances, the instance IDs appear in requestParameters.instancesSet.items — not always in the top-level Resources array depending on how the event was structured. If a ResourceName filter returns no results, fall back to filtering by EventName and scanning the requestParameters manually or with jq. Do not assume a null result from a resource-scoped lookup means no termination event exists.

IAM Permissions Required to Query CloudTrail Event History

The principal running these queries needs the following minimum permissions:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "CloudTrailLookupEvents",
      "Effect": "Allow",
      "Action": [
        "cloudtrail:LookupEvents"
      ],
      "Resource": "*"
    }
  ]
}

Note: cloudtrail:LookupEvents does not support resource-level restrictions — "Resource": "*" is required. This is consistent with the AWS Service Authorization Reference for CloudTrail.

When Event History Is Not Enough: Querying a Trail with Athena

Event History covers 90 days and is region-scoped. If the termination occurred beyond that window, or if you need to query across multiple regions or accounts, you need a CloudTrail trail delivering logs to S3, queried via Athena.

🔽 Click to expand: Athena query for TerminateInstances

SELECT
  eventtime,
  useridentity.type AS caller_type,
  useridentity.arn AS caller_arn,
  useridentity.username AS iam_username,
  json_extract_scalar(useridentity, '$.sessionContext.sessionIssuer.userName') AS role_session_issuer,
  sourceipaddress,
  useragent,
  requestparameters
FROM cloudtrail_logs
WHERE eventsource = 'ec2.amazonaws.com'
  AND eventname = 'TerminateInstances'
  AND eventtime BETWEEN '2024-01-01T00:00:00Z' AND '2024-01-31T23:59:59Z'
ORDER BY eventtime DESC;

Operator Rationale: Replace cloudtrail_logs with your actual Athena table name. The table must be created against your trail's S3 prefix using the CloudTrail partition projection or the Athena table creation wizard in the CloudTrail console. Athena queries incur data scan costs — always partition by date to minimize cost.

Wrap-Up: Closing the Loop on CloudTrail TerminateInstances Investigations

Tracing a CloudTrail TerminateInstances event to a specific IAM principal is a three-minute task when you know where to look. Event History gives you 90 days of indexed management events with no trail required. The userIdentity block is the authoritative attribution record — but interpreting it correctly requires distinguishing between IAM users, assumed role sessions, and AWS service principals. For investigations beyond 90 days or spanning multiple accounts, a trail with Athena is the correct tool.

Next steps to harden your environment against unintended terminations:

Enable EC2 termination protection on critical instances (aws ec2 modify-instance-attribute --instance-id i-xxxx --disable-api-termination).
Configure a CloudTrail trail with S3 delivery and log file integrity validation for retention beyond 90 days.
Create a CloudWatch alarm or EventBridge rule on TerminateInstances events to trigger real-time notification.
Review the CloudTrail userIdentity element reference for the full schema of each identity type.

Glossary

Term	Definition
Management Event	A CloudTrail event recording control-plane API operations (create, modify, delete) on AWS resources. Captured by default in Event History.
userIdentity	The JSON block in a CloudTrail event record that identifies the IAM principal that made the API call, including type, ARN, and session context.
AssumedRole	A `userIdentity.type` value indicating the call was made using temporary credentials obtained via `sts:AssumeRole`.
Event History	The CloudTrail console feature providing a searchable, 90-day view of management events in a single AWS region, available without a trail.
Termination Protection	An EC2 instance attribute (`disableApiTermination`) that prevents `TerminateInstances` API calls from succeeding until explicitly disabled.

📄 EC2 Stop vs. Terminate: What Actually Happens to Your EBS Data

Search This Blog

SW BBANG