AWS ECS with CloudWatch

AWS ECS is a nice environment to run your applications in. But sometimes you want “hot of the press” Docker features, which you can not configure in your task definitions just yet - like the awslogs log driver, which forwards your Docker logs to CloudWatch.

When using the Amazon provided ECS AMIs the setup can be bit complicated for non-us regions, so here’s a simple solution to make it work until the task definitions support the log driver:

Use cloud-init userdata to configure your instances properly:

#cloud-config

write_files:
  - path: /etc/ecs/ecs.config
    content: |
                ECS_CLUSTER=my-cluster
    owner: root:root
  - path: /etc/awslogs/awscli.conf
    content: |
        [plugins]
        cwlogs = cwlogs

        [default]
        region = eu-west-1
        aws_access_key_id = AKIAIOSFODNN7EXAMPLE
        aws_secret_access_key = wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY        
    owner: root:root
  - path: /etc/sysconfig/docker
    content: |
      DAEMON_MAXFILES=1048576
      OPTIONS="--default-ulimit nofile=1024:4096 --log-driver=awslogs --log-opt awslogs-region=eu-west-1 --log-opt awslogs-group=my-cluster"      
  
package_upgrade: true
packages:
  - awslogs

runcmd:
  - service awslogs start
  - chkconfig awslogs on
  - sed -i '/region = us-east-1/c\region = eu-west-1' /etc/awslogs/awscli.conf
  - service awslogs restart
  - service docker restart
  - start ecs

First, I’m configuring my ECS agent to join the right cluster, then I’m writing the awslogs agent configuration.
Here’s the catch I’ve tripped over repeatedly:

when installing the awslogs package, the configuration files region always get’s replaced with us-east-1.

To correct this I’m using sed, replacing the wrong region, and restarting the awslogs agent.

Lastly, the Docker configuration files is overwritten, instructing Docker to forward all logs to CloudWatch, into a log group called my-cluster. This requires a restart of the Docker daemon, followed by a start of the ECS agent.

Done.

Hopefully this workaround won’t be required for too long, because there are two downsides: a) all logs are forwarded to CloudWatch, even those you are not interested in, and b) you can not direct them to per-container log groups.

But for now, it’s good enough - and easy to integrate into tools like terraform :)