📄 Knowledge Article: Streaming Docker App Logs to AWS CloudWatch from EC2

Summary: This guide covers end-to-end steps for setting up structured logging in a Dockerized Node.js app using Winston, mapping logs to the EC2 host, and streaming them to AWS CloudWatch Logs. It includes all details and road bumps encountered.

🟢 Prerequisites

🚀 Steps

  1. Add Winston to your local backend 
    cd backend
npm install winston --save

    Check and commit updated package.json and package-lock.json.

  2. Create logger.js

    Inside backend directory:

    const winston = require('winston');
const path = require('path');
const fs = require('fs');
const logDir = path.join(__dirname, 'logs');
if (!fs.existsSync(logDir)) {
  fs.mkdirSync(logDir);
}
const logger = winston.createLogger({
  level: 'info',
  format: winston.format.combine(
    winston.format.timestamp(),
    winston.format.json()
  ),
  transports: [
    new winston.transports.File({ filename: path.join(logDir, 'app.log') }),
    new winston.transports.Console()
  ],
});
module.exports = logger;
  3. Update server.js 
    const logger = require('./logger');
logger.info('Server starting...');
logger.error('Error connecting to DB');
// Replace any console.log() with logger.info() or logger.error().
  4. Commit and push changes 
    git add package.json package-lock.json logger.js server.js
git commit -m "Add Winston logger with file and console support"
git push origin main

    Check GitLab CI/CD pipeline → confirm it runs successfully.

  5. Update docker-compose.yml 
    version: "3.8"
services:
  frontend:
    build: ./frontend
    ports:
      - "3000:80"
    depends_on:
      - backend
    restart: always
  backend:
    build: ./backend
    ports:
      - "5000:5000"
    env_file:
      - ./backend/.env
    depends_on:
      - mongo
    restart: always
    volumes:
      - ./backend/logs:/app/logs
  mongo:
    image: mongo
    ports:
      - "27017:27017"
    restart: always

    On EC2 host:

    mkdir -p ~/networth-Tracker/backend/logs

    ⚠️ Road bump: Container restart loop
    Container failed to start due to missing winston. Solved by running npm install winston, committing, pushing, and rebuilding Docker image.

    ⚠️ Road bump: Logs missing on host
    Initially logs existed only inside container. Fixed by adding Docker volume mapping and creating host logs directory.

  6. SSH into EC2 and pull latest code 
    ssh -i your-key.pem ubuntu@<your-ec2-ip>
cd ~/networth-Tracker
git pull origin main
  7. Rebuild and restart Docker containers 
    docker-compose down
docker-compose up --build -d

    Check logs:

    ls -l ~/networth-Tracker/backend/logs/
cat ~/networth-Tracker/backend/logs/app.log

    ✅ You should see log entries.

  8. Install CloudWatch Agent on EC2 
    sudo apt update
sudo apt install -y amazon-cloudwatch-agent
  9. Configure CloudWatch Agent 
    sudo nano /opt/aws/amazon-cloudwatch-agent/bin/config.json

    Paste:

    {
  "agent": {
    "metrics_collection_interval": 60,
    "run_as_user": "root"
  },
  "metrics": {
    "append_dimensions": {
      "InstanceId": "${aws:InstanceId}"
    },
    "metrics_collected": {
      "cpu": {
        "measurement": ["cpu_usage_idle", "cpu_usage_user", "cpu_usage_system"],
        "metrics_collection_interval": 60
      },
      "mem": {
        "measurement": ["mem_used_percent"],
        "metrics_collection_interval": 60
      },
      "disk": {
        "measurement": ["used_percent"],
        "metrics_collection_interval": 60,
        "resources": ["/"]
      }
    }
  },
  "logs": {
    "logs_collected": {
      "files": {
        "collect_list": [
          {
            "file_path": "/home/ubuntu/networth-Tracker/backend/logs/app.log",
            "log_group_name": "networth-tracker-logs",
            "log_stream_name": "{instance_id}"
          }
        ]
      }
    }
  }
}
  10. Start and verify CloudWatch Agent 
    sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a fetch-config -m ec2 -c file:/opt/aws/amazon-cloudwatch-agent/bin/config.json -s
sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a status
  11. Verify logs in AWS Console

    Go to CloudWatch → Logs → Log groups. You should see log streams from your app.

⭐ Summary