Lesson 11 of 11 · 13 min · AWS security

KMS encryption and CloudTrail auditing

Covers: AWS CLI, KMS and CloudTrail. Updated 13 Nov 2026. Part of the DevOps Engineer roadmap.

What you will learn

  • Encryption at rest and in transit, and who holds the keys
  • KMS keys, aliases, key policies, grants and rotation
  • Envelope encryption: why a data key is used
  • How AWS services use KMS (S3, EBS, RDS, Secrets Manager)
  • CloudTrail: what it records, how to enable it properly, and how to use it for investigation and alerting

In simple words

Encryption scrambles data so that it is useless without the key. KMS is the service that keeps those keys safe and controls who may use them. CloudTrail is the CCTV of your account: it records who did what and when, so you can investigate afterwards.

Words to know

  • Encryption at rest: protecting data while it is stored.
  • Encryption in transit: protecting data while it travels over the network.
  • KMS: AWS's service for creating and guarding encryption keys.
  • CloudTrail: the log of every action taken in your AWS account.

Encryption basics for cloud workloads

  • In transit: TLS between client and server and between internal services (Networking course).
  • At rest: data on disks and in object stores is encrypted so that a stolen disk, snapshot or leaked bucket copy is useless without the key. In AWS this is usually one checkbox per service, but the choice of key decides who can decrypt and how it is audited.

The key question for any encryption design is: who controls the key? Encryption is only as strong as the control over the key.

AWS KMS

AWS Key Management Service (KMS) creates and controls cryptographic keys. The keys never leave KMS in plain form (they live in hardware security modules); you call the service to encrypt, decrypt or generate data keys, and each call is authorised by policy and logged.

Key types:

  • AWS managed keys (aws/s3, aws/ebs, aws/rds): created automatically per service; free; you cannot change their policy.
  • Customer managed keys (CMKs): you create them, define who may use and administer them, enable rotation, and can disable or schedule deletion. Small monthly fee plus per-request charges. Use for sensitive data, cross-account sharing and audit requirements.
  • Symmetric keys (AES-256-GCM) are the default and what services use; asymmetric keys (RSA/ECC) for signing and external encryption.
  • Multi-region keys for DR and global apps; external key store (XKS) for keys held outside AWS.

Create a key, give it a friendly alias, and inspect it:

KID=$(aws kms create-key --description "demo data key" --query KeyMetadata.KeyId --output text)
aws kms create-alias --alias-name alias/shop-data --target-key-id $KID
aws kms describe-key --key-id alias/shop-data --query 'KeyMetadata.[KeyState,KeySpec,KeyUsage,Origin]' --output text
Enabled   SYMMETRIC_DEFAULT   ENCRYPT_DECRYPT   AWS_KMS

Use aliases (alias/shop-data) in code and configuration rather than key ids, so you can swap the underlying key.

Envelope encryption

KMS encrypts at most 4 KB directly, and calling a remote service for every byte would be slow. Real encryption of large data uses envelope encryption:

  1. Ask KMS to generate a data key under your CMK. KMS returns the data key in two forms: plaintext (use it now) and encrypted under the CMK.
  2. Encrypt your data locally with the plaintext data key (fast), then discard the plaintext key from memory.
  3. Store the encrypted data key next to the encrypted data.
  4. To decrypt, send the encrypted data key to KMS (Decrypt), get the plaintext key back, and decrypt locally.
aws kms generate-data-key --key-id alias/shop-data --key-spec AES_256 \
    --query "[length(Plaintext),length(CiphertextBlob)]" --output text
44    156

(Lengths of the base64-encoded plaintext key, 32 bytes of key material, and of the encrypted copy.) The master key never leaves KMS, each object can have its own data key, and access to decryption is controlled and logged centrally. S3, EBS, RDS and most services do exactly this behind the scenes; SDK libraries (the AWS Encryption SDK) offer it for your own data.

Who can use a key: key policies

Every KMS key has a key policy, the primary access control (IAM policies alone are not enough unless the key policy delegates to IAM, which the default policy does by allowing the account's IAM). Separate roles:

  • Key administrators: may manage the key (rotate, disable, change policy) but not necessarily use it to decrypt.
  • Key users: may use it for Encrypt, Decrypt, GenerateDataKey.
  • Services get access via grants or via conditions such as kms:ViaService (only usable through S3 in region X) and kms:EncryptionContext (extra key-value data that must match, bound to the ciphertext and logged).

To read an S3 object encrypted with a CMK, a principal needs both S3 permission and kms:Decrypt on the key; a missing KMS permission is a classic cause of "AccessDenied" for objects that exist. Cross-account use requires the key policy in the owning account to allow the other account and an IAM policy in the other account.

Rotation, disabling and deletion

  • Automatic rotation (aws kms enable-key-rotation) creates new key material every year (configurable) while old material remains to decrypt old data; the key id stays the same, so nothing in your applications changes. We enabled it and get-key-rotation-status returned True.
  • Disable a key to make data temporarily inaccessible (reversible).
  • Deletion has a mandatory waiting period (7-30 days): schedule-key-deletion returned PendingDeletion with a 7-day window in our test. Once deleted, everything encrypted under it is permanently unreadable: crypto-shredding, but also a disaster if done by mistake. Protect keys with deny policies and alarms on ScheduleKeyDeletion.

Using KMS with services

Service Setting
S3 default bucket encryption aws:kms with a CMK (plus S3 Bucket Keys to cut KMS request costs)
EBS encrypt volumes; set account default encryption and a default key
RDS / Aurora enabled at creation (cannot be turned on later without snapshot copy); snapshots inherit
Secrets Manager / SSM Parameter Store (SecureString) encrypted with KMS keys
CloudWatch Logs, SQS, SNS, DynamoDB, EFS, Lambda environment variables KMS options
CloudTrail logs encrypt the log bucket / trail with a CMK

In Terraform: aws_kms_key, aws_kms_alias, and kms_key_id arguments on resources. Remember the cost side: each API call is billed and has request-rate quotas; high-volume workloads use data-key caching and S3 Bucket Keys.

Good practice: separate keys per environment or data classification; least-privilege key policies; enable rotation; alarm on disable/delete; never store plaintext data keys; enable encryption by default at the account level.

CloudTrail: the audit log

AWS CloudTrail records API calls made in your account (console clicks, CLI, SDK, other services acting for you): who (the principal and, for roles, the session), what (service, action, parameters), when, from where (source IP, user agent), and whether it succeeded or failed (error code). It answers "who deleted that security group?", "which role listed the secrets?", "who disabled the key?" and feeds security tooling.

Event types:

  • Management events (control plane): creating instances, changing IAM, editing security groups. Enabled by default in the last 90 days (viewable under Event history) at no charge.
  • Data events (data plane): S3 object reads/writes (GetObject, PutObject), Lambda invocations, DynamoDB item operations. High volume, off by default, billed; enable selectively for sensitive buckets.
  • Insights events: unusual API activity (spikes in resource provisioning or errors).
  • Network activity events for VPC endpoints (newer feature).

Set it up properly

  1. Create an organisation trail: one trail logging all accounts and all regions into a central S3 bucket in a dedicated logging/security account. Event history alone is not enough: it covers 90 days, and only management events.
  2. Protect the logs: S3 bucket with Block Public Access, versioning, Object Lock or MFA delete, a bucket policy that only allows CloudTrail to write and only security staff to read, SSE-KMS encryption with a CMK, and log file validation (digest files that let you prove logs were not altered).
  3. Send to CloudWatch Logs (and/or EventBridge) for real-time alerting; query history with Amazon Athena over the S3 files, or CloudTrail Lake.
  4. Retention according to compliance (often one year or more), with lifecycle rules to cheaper storage.

Example event (shortened JSON):

{
  "eventTime": "2026-11-12T09:41:07Z",
  "eventSource": "ec2.amazonaws.com",
  "eventName": "AuthorizeSecurityGroupIngress",
  "userIdentity": {"type": "AssumedRole", "arn": "arn:aws:sts::123456789012:assumed-role/dev-admin/asha"},
  "sourceIPAddress": "203.0.113.25",
  "requestParameters": {"groupId": "sg-0abc", "ipPermissions": {"items": [{"fromPort": 22, "ipRanges": {"items": [{"cidrIp": "0.0.0.0/0"}]}}]}},
  "errorCode": null
}

Read it like a detective: Asha, using the dev-admin role from an office IP, opened SSH to the world on security group sg-0abc at 09:41. (The account number and IP are examples.)

Using it for security

  • Investigate: filter by userIdentity, eventName, sourceIPAddress, errorCode (spikes of AccessDenied suggest probing or a broken role). Follow a role session: assumed-role/name/session and the AssumeRole event that created it; for stolen keys, find the accessKeyId and everything it did.
  • Alert (EventBridge rules or CloudWatch metric filters) on high-risk events: root user login, ConsoleLogin without MFA, DeleteTrail/StopLogging, IAM policy changes, AuthorizeSecurityGroupIngress with 0.0.0.0/0, ScheduleKeyDeletion, PutBucketPolicy, access key creation, large GetObject volume. Amazon GuardDuty analyses CloudTrail, VPC Flow Logs and DNS logs for threats (crypto-mining, credential exfiltration) with no setup; AWS Config records resource configuration changes and checks compliance rules; Security Hub aggregates findings.
  • Detect and prevent tampering: attackers try to disable logging first. Use an SCP to deny cloudtrail:StopLogging/DeleteTrail, keep logs in a separate account, and enable log validation.
  • Compliance: CloudTrail is evidence for audits (who accessed what and when).

Audit logs are only useful if someone looks at them: define the alerts, the owners, and the response.

Common mistakes

  • Relying on the 90-day event history instead of a trail to S3.
  • Trails in the same account the attacker controls, with no validation or retention protection.
  • No alerts on root usage, MFA-less logins, trail changes.
  • KMS keys with admin and user rights granted too broadly, or the default key policy left open.
  • Scheduling key deletion without understanding the consequences.
  • Encrypting but exposing: a public bucket with encryption is still public.
  • Forgetting kms:Decrypt for the application role.
  • Storing plaintext data keys or secrets alongside the data.

Try it yourself

  1. Create a customer managed key with an alias, enable rotation and write a key policy separating administrators from users.
  2. Create an S3 bucket with default SSE-KMS using your key, upload an object, and remove kms:Decrypt from a role to observe AccessDenied.
  3. Create an organisation-wide trail (or a single-account trail) to a locked-down bucket with log file validation; make a change in the console and find it in the log.
  4. Write an EventBridge rule that alerts on StopLogging or on root console login.

Check yourself

What is envelope encryption and why is it used?

Data is encrypted locally with a data key, and the data key is itself encrypted by a KMS key. It keeps the master key in KMS, avoids sending bulk data to KMS, and lets each object have its own key.

What happens when you schedule deletion of a KMS key?

After the waiting period (7-30 days) the key is destroyed and everything encrypted under it becomes permanently unreadable; during the period it can be cancelled.

What does CloudTrail record, and where should the logs go?

API calls (who, what, when, where, outcome). Send them from all regions and accounts to a protected S3 bucket in a separate logging account, with encryption, validation and retention.

Why can reading an S3 object fail even when S3 permissions allow it?

If the object is encrypted with a customer managed KMS key, the caller also needs permission to use that key (kms:Decrypt) in the key policy/IAM.

Interview questions

How do you encrypt sensitive data in AWS and control who can decrypt it?

Use customer managed KMS keys with narrowly scoped key policies (separate admins and users), default encryption for S3/EBS/RDS, rotation enabled, conditions like kms:ViaService, and CloudTrail logging of every key use, with TLS for data in transit.

An S3 bucket was made public last night. How do you find out who did it?

Search CloudTrail (organisation trail or Athena/CloudTrail Lake) for PutBucketPolicy, PutBucketAcl or PutPublicAccessBlock events on that bucket, read the userIdentity and sourceIPAddress, then review that principal's other activity and fix the permissions that allowed it.

How do you make sure audit logs can be trusted?

Centralise them in a separate account, restrict write/delete (bucket policy, Object Lock), enable log file validation, encrypt with KMS, deny StopLogging/DeleteTrail with SCPs, and alert on tampering attempts.

Summary

  • KMS holds keys in HSMs; use customer managed keys with alias, key policies, rotation and careful deletion; services use envelope encryption with data keys.
  • Both the service permission and the KMS permission are needed to read encrypted data.
  • CloudTrail records API activity; enable an organisation trail to a protected, validated, encrypted central bucket and alert on high-risk events.
  • Combine with GuardDuty, Config and Security Hub, and make sure someone owns responding to alerts.

Going deeper

See how much of this you can explain

A free mock interview asks you new questions on Cloud platforms and AWS fundamentals for DevOps Engineer and scores every answer, with what to fix.