Monitoring and Logging in AWS

Track, Visualize, and Secure Your Cloud Operations

Modern cloud environments demand visibility and control. AWS offers robust tools to help you monitor applications, infrastructure, and security in real time—ensuring performance, compliance, and rapid issue resolution.

This chapter covers Amazon CloudWatch, AWS CloudTrail, VPC Flow Logs, and Config, with real-world use cases, command-line examples, diagrams, and best practices.

📊 Section 1: Amazon CloudWatch

🔍 What is Amazon CloudWatch?

Amazon CloudWatch is a fully managed monitoring and observability service for AWS cloud resources and applications.

It enables:

• Real-time metrics
• Logs collection & analysis
• Custom dashboards
• Automated alarms & actions
• Anomaly detection with machine learning

🧱 CloudWatch Core Components

Common Use Cases

Example: Create a CloudWatch Alarm for EC2 CPU

AWS Console:

1. Go to CloudWatch → Alarms → Create Alarm
2. Choose EC2 → Select Instance → CPUUtilization
3. Set threshold: > 70% for 5 minutes
4. Add notification via SNS (email, SMS, Lambda)

AWS CLI:

aws cloudwatch put-metric-alarm \
  --alarm-name "HighCPU" \
  --metric-name CPUUtilization \
  --namespace AWS/EC2 \
  --statistic Average \
  --period 300 --threshold 70 \
  --comparison-operator GreaterThanThreshold \
  --dimensions Name=InstanceId,Value=i-1234567890abcdef0 \
  --evaluation-periods 1 \
  --alarm-actions arn:aws:sns:us-east-1:111122223333:MyTopic

aws cloudwatch put-metric-alarm \
  --alarm-name "HighCPU" \
  --metric-name CPUUtilization \
  --namespace AWS/EC2 \
  --statistic Average \
  --period 300 --threshold 70 \
  --comparison-operator GreaterThanThreshold \
  --dimensions Name=InstanceId,Value=i-1234567890abcdef0 \
  --evaluation-periods 1 \
  --alarm-actions arn:aws:sns:us-east-1:111122223333:MyTopic

Dashboards – Centralized Visibility

Features:

• Add graphs, number widgets, text, and log visualizations
• Share across accounts
• Build views like “App Health”, “Billing Overview”, or “Lambda Performance”

📌 Free tier includes 10 custom metrics and 3 dashboards/month.

Advanced CloudWatch Features

• Anomaly Detection: Uses ML to detect unusual metric patterns.
• Metric Math: Combine multiple metrics in formulas.
• Composite Alarms: Combine multiple alarms into logical rules.

Section 2: AWS CloudTrail

🔍 What is AWS CloudTrail?

AWS CloudTrail records every API call made in your AWS account. It tracks who did what, when, and from where.

This is critical for:

• Security audits
• Troubleshooting
• Compliance
• User activity monitoring

🛠️ CloudTrail Key Concepts

Example Use Case

• Who stopped EC2 at 3 AM?
• Who deleted an S3 bucket?
• What IP was used to update a Lambda function?

Answered via CloudTrail logs, stored in S3.

Enable CloudTrail (Console)

1. Go to CloudTrail → Create Trail
2. Name: my-org-trail
3. Storage: Select/Create S3 bucket
4. Choose event types (Management/Data)
5. Optional: Send to CloudWatch Logs
6. Enable

📌 Keep trail enabled in multi-region for complete coverage.

Sample CloudTrail JSON Log Entry

{
  "eventTime": "2025-06-22T02:18:00Z",
  "eventSource": "ec2.amazonaws.com",
  "eventName": "StopInstances",
  "userIdentity": {
    "type": "IAMUser",
    "userName": "admin"
  },
  "sourceIPAddress": "203.0.113.10",
  "requestParameters": {
    "instancesSet": {
      "items": [{"instanceId": "i-0abc1234567890xyz"}]
    }
  }
}

{
  "eventTime": "2025-06-22T02:18:00Z",
  "eventSource": "ec2.amazonaws.com",
  "eventName": "StopInstances",
  "userIdentity": {
    "type": "IAMUser",
    "userName": "admin"
  },
  "sourceIPAddress": "203.0.113.10",
  "requestParameters": {
    "instancesSet": {
      "items": [{"instanceId": "i-0abc1234567890xyz"}]
    }
  }
}

Section 3: VPC Flow Logs

🔍 What Are VPC Flow Logs?

VPC Flow Logs capture IP traffic going to and from network interfaces (ENIs) within your VPC.

Use it to:

• Monitor network activity
• Identify bottlenecks
• Investigate security issues

Create a Flow Log

1. Open VPC Dashboard → Flow Logs
2. Choose VPC/Subnet/ENI
3. Destination: CloudWatch Logs or S3
4. Choose format (default, custom)
5. Create

Sample Log Entry (Simplified):

version account-id interface-id srcaddr dstaddr srcport dstport protocol packets bytes start end action
2 123456789 vpc-eni123 10.0.1.10 10.0.2.12 443 8080 6 4 840 1591130677 1591130737 ACCEPT

version account-id interface-id srcaddr dstaddr srcport dstport protocol packets bytes start end action
2 123456789 vpc-eni123 10.0.1.10 10.0.2.12 443 8080 6 4 840 1591130677 1591130737 ACCEPT

📌 Use tools like Athena or CloudWatch Logs Insights to query large VPC logs.

Section 4: AWS Config

🔍 What is AWS Config?

AWS Config records the configuration state of resources and tracks their changes over time.

Use Cases:

• Compliance auditing
• Drift detection
• Resource history tracking

Example Use Cases:

• Detect if S3 bucket becomes public
• Track who changed a Security Group rule
• Ensure all EC2s are encrypted

✅ Enable AWS Config:

1. Go to AWS Config → Set up
2. Choose resources to track (e.g., all)
3. Store results in S3
4. Create conformance packs to apply policies

✅ AWS Config integrates with AWS Security Hub and CloudWatch Events.

Monitoring Architecture Diagram

                      +--------------------+
                      |    CloudWatch      |
                      |   (Metrics & Alarms)|
                      +--------------------+
                               |
        +----------------------+----------------------+
        |                      |                      |
+---------------+     +----------------+     +----------------+
| CloudTrail     |     | VPC Flow Logs  |     | AWS Config     |
| (API Activity) |     | (Network Logs) |     | (Resource State)|
+---------------+     +----------------+     +----------------+
        ↓                      ↓                      ↓
         → CloudWatch Dashboards ← Centralized Visibility →

                      +--------------------+
                      |    CloudWatch      |
                      |   (Metrics & Alarms)|
                      +--------------------+
                               |
        +----------------------+----------------------+
        |                      |                      |
+---------------+     +----------------+     +----------------+
| CloudTrail     |     | VPC Flow Logs  |     | AWS Config     |
| (API Activity) |     | (Network Logs) |     | (Resource State)|
+---------------+     +----------------+     +----------------+
        ↓                      ↓                      ↓
         → CloudWatch Dashboards ← Centralized Visibility →

Summary