Docker on AWS – Part 6 – Console Logging with CloudWatch

In my previous posts, I figured out how to deploy a Docker image containing a Spring Boot application on AWS. Now I’d like to get some basic application logging set up.

Logging with Amazon CloudWatch

I typically set up my applications to write log messages to a file. When using Docker, that isn’t as straightforward. My first thought was to find a way to mount a volume on the server that hosts my Docker container, and write the logs there. I might still pursue that type of solution, however, I found an alternative that I can use right away with minimal effort to set up: Amazon CloudWatch logging.

What is CloudWatch logging?

Amazon CloudWatch does more than just logging, but at this point I’m only interested in the logging feature as it works with ECS. CloudWatch will store individual lines of console output from an ECS Docker instance, and allow you to view them later as individual log entries.

How to set it up to work with Docker on ECS?

Creating a Log Group

The first step is to create a CloudWatch Log Group. The Log Group name will be used when configuring logging in ECS. Start by finding CloudWatch in the list of services in the AWS Management Console.

SBDAWS-PT06-screenshot-001

On the CloudWatch page, choose Logs from the left-hand navigation panel. Then click the “Actions” button and choose “Create log group”.

SBDAWS-PT06-screenshot-002

I created my log group with the name “sbdaws-log-group”. I also set a limit on how long my logs will be kept. I don’t need to keep logs for longer than 30 days.

SBDAWS-PT06-screenshot-003

Enabling CloudWatch Logging in my ECS Task Definition

Enabling this type of logging did not require any code changes or any changes to my Docker image. I’m using the same image as I used in my previous post, Docker on AWS – Part 5 – Spring Boot and AWS SDK for Java.

I’ll be making a change to my existing ECS Task Definition by selecting the definition and clicking the “Create new revision” button.

SBDAWS-PT06-screenshot-004

All the changes will be done within the container defintion.

SBDAWS-PT06-screenshot-005

I’ll keep all the current container settings, but I need to add a few more.

Environment variables for IAM credentials used by logging driver

First, I’ll add two environment variables that will define an IAM user id/key pair. I did this with application code and a properties file in my previous post when I was working with the AWS SDK for Java. Bundling those credentials with the application was an awkward way to do it. I think this approach is cleaner. It also allows the AWS logging driver (covered in the next section) to have access to those IAM credentials.

You should use the same environment variable keys that I use below, but use values for your own IAM user.

SBDAWS-PT06-screenshot-006

Logging driver settings

Next, I’ll scroll down to the “Storage and Logging” section.

Under “Log configuration”, click the log driver dropdown and choose “awslogs”. This should pre-populate several log option property keys below it. For the “awslogs-group” key, set the value to the name of the CloudWatch Log Group that you created earlier. For the “awslogs-region” key, use the region name where this ECS container will be hosted.

There is another key named “awslogs-stream-prefix”. This is optional. I’m not currently using this property, so I won’t comment on it.

SBDAWS-PT06-screenshot-007

When you’ve made these changes, scroll down to the bottom and save your container definition changes.

SBDAWS-PT06-screenshot-008

Then you have to scroll to the bottom one more time to create the new Task Defintion revision.

SBDAWS-PT06-screenshot-009

Now I have a new Task Definition revision. This is “revision 3”.

SBDAWS-PT06-screenshot-010

It isn’t necessary to do this, but I set my older revisions to be inactive so they won’t appear in the list anymore.

SBDAWS-PT06-screenshot-011

Updating the Service definition in my ECS Cluster

I’ve created a new Task Definition revision, but nothing is using that revision yet. I need to go into the settings for my ECS Cluster and change the Service definition so it uses the newest revision of my Task Definition.

I select the existing service, and click the “Update” button.

SBDAWS-PT06-screenshot-012

Now I need to select the new revision of my Task Definition.

SBDAWS-PT06-screenshot-013

Nothing else needs to change when you click through the remaining steps.

How to view the CloudWatch logs?

The log output can be viewed back on the CloudWatch page. First I need to click on my Log Group to access the log streams.

SBDAWS-PT06-screenshot-014

Next, I’ll click on the newest log stream.

SBDAWS-PT06-screenshot-015

And now I see my console log output, where each line that was logged to the console appears as a separate CloudWatch log stream entry.

SBDAWS-PT06-screenshot-016

That’s impressive, given how little effort was needed on my part to set this up.

 

Leave a Reply