docker-compose and load balancing

We’re using docker-compose in production @work and I needed an ad-hoc, lightweight solution to load-balance requests across multiple instances of a specific service, all running on a single host. While docker-compose allows you to scale your services you need to take care of load balancing yourself.

Before sharing my approach, which uses golang, let’s start with the problem description first.

I’ll be using a dummy service, httpd echo, for demonstration purposes.

Given a docker-compose.yml like this:

echo:
  image: nicolai86/http-echo
  ports:
    - 9090:9090

Assuming you need to scale echo, the above docker-compose.yml won’t allow you to scale echo because you defined a port mapping:

$ docker-compose scale echo=2
WARNING: The "echo" service specifies a port on the host. If multiple containers for this service are created on a single host, the port will clash.
Creating and starting 2 ... error

ERROR: for 2  failed to create endpoint dockercomposelbexample_echo_2 on network bridge: Bind for 0.0.0.0:9090 failed: port is already allocated

Removing the port mapping allows you to scale your container but now you need to route the traffic using a reverse proxy.

Available solutions include using a HAProxy / Nginx, with dynamic configuration reloads when your instance pool change. However, these solutions require at least two new, huge-ish docker containers, because of the dependencies required to make this work (did I mention fast access to the internet is a problem?).

Since golang already comes with a ReverseProxy implementation and it compiles to staticly linked binaries, it’s possible to build much smaller proxy containers. As an example, see traefik. However, to dig deeper into the details I decided to roll my own reverse proxy with docker-compose integration.

It works like this:

echo:
  image: nicolai86/http-echo

proxy:
  image: nicolai86/docker-compose-reverse-proxy
  ports:
    - 80:8080
  volumes:
    - "${DOCKER_CERT_PATH}:${DOCKER_CERT_PATH}"
  environment:
    DOCKER_COMPOSE_SERVICE_NAME: echo
    DOCKER_HOST: "${DOCKER_HOST}"
    DOCKER_TLS_VERIFY: "${DOCKER_TLS_VERIFY}"
    DOCKER_CERT_PATH: "${DOCKER_CERT_PATH}"

# docker-compose up -d
Starting dockercomposelbexample_proxy_1
Starting dockercomposelbexample_echo_1

# docker-compose scale echo=10
Creating and starting 2 ... done
Creating and starting 3 ... done
Creating and starting 4 ... done
Creating and starting 5 ... done
Creating and starting 6 ... done
Creating and starting 7 ... done
Creating and starting 8 ... done
Creating and starting 9 ... done
Creating and starting 10 ... done

Now verify that it’s working by checking the X-Internal-Service header:

 # for i in {0..5}; do curl 192.168.99.100 -v 2>&1 | grep X-Int; done
< X-Internal-Service: 172.17.0.15:9090
< X-Internal-Service: 172.17.0.8:9090
< X-Internal-Service: 172.17.0.7:9090
< X-Internal-Service: 172.17.0.8:9090
< X-Internal-Service: 172.17.0.10:9090
< X-Internal-Service: 172.17.0.13:9090

As you can see the requests are handled by different containers! Quick and easy.
The final docker image is ~10mb big:

# docker images | grep docker-compose-reverse-proxy
nicolai86/docker-compose-reverse-proxy   latest              dc50f87279f3        8 hours ago         9.832 MB

For a tiny experiment it really shows the power of go: it’s reasonable small (< 100LOC) and thus easy to understand, replace or extend.

However, I’d still recommend using Traefik if you plan on running this in production.

That’s it for now. Happy Hacking!