Scaleway on terraform: remote-exec provisioners

In this blog post I want to explore two options of using terraform and the remote-exec provisioner with the new Scaleway cloud provider.

using Scaleway

First, signup for Scaleway. Once you have a Scaleway account, export the required credentials to your environment like this:

export SCALEWAY_ACCESS_KEY=<your-access-key> 
export SCALEWAY_ORGANIZATION=<your-organization-id>

You can find out both information easily by using the scw cli; it’ll write the information to the ~/.scwrc file.

Now you can use the scaleway provider like this:

provider "scaleway" {}

resource "scaleway_server" "server" {
  name = "my-server"
  type = "C1"
  image = "eeb73cbf-78a9-4481-9e38-9aaadaf8e0c9" # ubuntu 16.06

You’re now ready to manage your scaleway infrastructure with terraform!

public hosts

By default, the scaleway_server resource will create internal servers only, meaning the servers won’t have a public ip. In order to use remote-exec however, the server must be accessible.

The easiest way to achieve this is by exposing your server using the dynamic_ip_required attribute:

provider "scaleway" {}

resource "scaleway_server" "server" {
  name  = "my-server"
  type  = "C1"
  image = "eeb73cbf-78a9-4481-9e38-9aaadaf8e0c9" # ubuntu 16.06

  dynamic_ip_required = true

  provisioner "remote-exec" {
    inline = "echo hello world"

Now your server will get a public ip assigned and remote-exec will work out of the box!

jump hosts

When you don’t want to expose your servers you can setup a publicly accessible jump host, which then can be used to access your internal servers:

provider "scaleway" {}

resource "scaleway_server" "jump-host" {
  name  = "my-jump-host"
  type  = "C1"
  image = "eeb73cbf-78a9-4481-9e38-9aaadaf8e0c9" # ubuntu 16.06

  dynamic_ip_required = true

resource "scaleway_server" "server" {
  type  = "C1"
  image = "eeb73cbf-78a9-4481-9e38-9aaadaf8e0c9" # ubuntu 16.06

  connection {
    type         = "ssh"
    user         = "root"
    host         = "${self.private_ip}"
    bastion_host = "${scaleway_server.jump-host.public_ip}"
    bastion_user = "root"
    agent        = true

  provisioner "remote-exec" {
    inline = "echo hello world"

this way, only your jump host is publicly accessible and all other servers will remain internal.

That’s it for now. Enjoy Scaleway on terraform :)

Learning Erlang #1

after attending dotScale 2016 I finally got enough motivation to properly start learning Erlang.

From past experience I know that I need a project to play around with while learning a new language. So I decided to work on an outstanding task for traq, my CLI time-tracking tool: a JSON API to use as a remote storage backend.
While rsync (or dropbox, fwiw) would work great as a solid synchronization mechanism I want an API which allows me to write a mobile app for traq someday.

I’m using rebar3 for the project, and I really like that it’s enforcing OTP conventions and best practices. This way I can focus on my project; the only thing I’m really missing is a standalone formatter. erl_tidy produces code I can not (yet) make much sense of, so I stick to my own source formatting; ideally I’d like to not think about this.

If you happen to write something like rebar3 fmt please reach out, would love to test & use a tool which unifies source formatting.

The API is using cowboy as HTTP server and jsx for json en- & decoding. The combination of both modules works good; I’m only missing some minor things:

  • a comprehensive tests-first guide to cowboy. Information is scarce on how to write acceptance/ integration/ unit tests for http handlers. I’ll probably use etest_http for acceptance tests; the official cowboy code base seems to use eunit heavily, but there seems much boilerplate involved; I’ll probably find a minimal example sooner or later.
  • live-reloading of routes in cowboy. sync works great for everything but routes. Probably requires some additional setup.
  • jsx seems to be unable to encode an array of objects:

        {<<"an">>, <<"example">>}

    will produce

    {"an": "example"}

    but not

      {"an": "example"}

    But maybe I’m just overlooking something.

Since I’m very much a beginner at Erlang I will probably learn how to fix all of the above points and get much more fluent with the language in general.

Until then I’ll use the official documentation as much as possible, and Learn you some erlang for specific topics.

All other things aside the current setup works quite well for learning Erlang, which I didn’t expect at first.
Also I’m looking forward to sharing more about my journey into Erlang :)

The five important abilities of systems

Reading The Architecture of Open Source Applications, especially Chapter 4, struck a cord with me:

The authors, Margo Seltzer and Keith Bostic, state, that building systems based on simple components as building blocks lead to systems which are superior to monoliths in the five important -abilities: understandability, extensibility, maintainability, testability, and flexibility.

This made me wonder how to clearly differentiate between these abilities, since many discussions about software architecture evolve around one or more of these abilities.

Also the abilities are rather generic, making them applicable to system design in general, so I consider a strong understanding of them essential to be able to perform well.

Here are some definitions & thoughts:


describes if the code clearly communicates its purpose to readers other then the author.

This is important because code is read more often than written and if your code is hard to understand it will take up more time to read, or worse, it won’t be understood at all; Problem is this is a subjective metric - it depends on the knowledge & background of the reader.

To me, this effectively comes down to:

  • don’t be clever.
  • make implicit assumptions explicit.
  • optimize for readability.


speaks to a system being able to accommodate new functionality or changes over time, without requiring bigger rewrites of existing code.

I think every software engineer has experienced changes cascading through their code base. Yet I think it’s hard to avoid sometimes. When building a system, chances are you don’t know every thing up front, meaning you can always run into this situation.

While I think extensibility can be achieved in parts by building loosely coupled, eventually consistent systems, it’s better to perform technical spikes regularly, to ensure the problem at hand is understood properly, as well as possible integrations into the existing code base.


describes the afford required to keep a system in good shape, e.g. the amount of work required to regularly upgrade your dependencies, fix defects; but also incorporate non functional requirements after a system has launched.

Since I don’t know of a better way to describe this I’d say “how often do you upset your product owner?” - mostly because product owners always want new features, and you can’t deliver them if you’re busy keeping the system up.


measures how easy it is to write test for your system.

Good testability often speaks to an easy understand code base; It’s also extremely important as a safety net when thinking about maintenance. I use tests as an indicator on how easy it is to understand a system. If you’re tests don’t immediately point you to the source of a bug, or require so much setup that it’s hard to even find the testing code, you don’t have this ability.


describes how your system adapts to external changes.

External changes for a system range from slow responses from 3rd parties over occasional timeouts to unavailability.

To me, this mostly concerns operations but is extremely important to get right. If done wrong, your software will be hard to keep up and running.
I’d always argue: if it is worth building a system to begin with, it should always be kept available.

Fortunately, many architectural patterns exist to enable this.

The challenge

I think that these abilities build upon each other; and you always have to find trade offs since you can’t fulfill all of them at once. And trade offs can only be discussed in a specific context, meaning there is no one size fits all solution.

For example, rather generic trade offs:

  • when building a technical spike, it might be okay to write less understandable & maintainable code, since your goal is a proof something is actually doable. When you have to go live with one, however, you should invest time in rebuilding critical systems/ part of a system, which are hard to understand/ maintain.
  • when taking over an existing project on should check if the system reaches the desired level of flexibility & maintainability. Nobody wants to be stuck with a system that fails all the time.
  • when starting a new project, the project expectations should be stated explicitly. E.g. when the project team is expected to change often, writing understandable and testable code should have higher priority, to reduce onboarding time.

But also more specific ones:

  • hard to understand code is hard to test. You are probably missing good abstractions.
  • introducing abstractions just for tests makes your code harder to understand.
  • decoupling two systems, e.g. with a circuit breaker, also make it less easy to understand; simply because there’s more code to read.

If you think it’s simple, then you have misunderstood the problem.
- Bjarne Stroustrup

My key take away is this: software engineering is hard, and it helps to have some guidelines in mind when trying to make the best decision for the moment.
I think these abilities help with that.

docker-compose and load balancing

We’re using docker-compose in production @work and I needed an ad-hoc, lightweight solution to load-balance requests across multiple instances of a specific service, all running on a single host. While docker-compose allows you to scale your services you need to take care of load balancing yourself.

Before sharing my approach, which uses golang, let’s start with the problem description first.

I’ll be using a dummy service, httpd echo, for demonstration purposes.

Given a docker-compose.yml like this:

  image: nicolai86/http-echo
    - 9090:9090

Assuming you need to scale echo, the above docker-compose.yml won’t allow you to scale echo because you defined a port mapping:

$ docker-compose scale echo=2
WARNING: The "echo" service specifies a port on the host. If multiple containers for this service are created on a single host, the port will clash.
Creating and starting 2 ... error

ERROR: for 2  failed to create endpoint dockercomposelbexample_echo_2 on network bridge: Bind for failed: port is already allocated

Removing the port mapping allows you to scale your container but now you need to route the traffic using a reverse proxy.

Available solutions include using a HAProxy / Nginx, with dynamic configuration reloads when your instance pool change. However, these solutions require at least two new, huge-ish docker containers, because of the dependencies required to make this work (did I mention fast access to the internet is a problem?).

Since golang already comes with a ReverseProxy implementation I decided to roll my own reverse proxy with docker-compose integration.

It works like this:

  image: nicolai86/http-echo

  image: nicolai86/docker-compose-reverse-proxy
    - 80:8080
# docker-compose up -d
Starting dockercomposelbexample_proxy_1
Starting dockercomposelbexample_echo_1

# docker-compose scale echo=10
Creating and starting 2 ... done
Creating and starting 3 ... done
Creating and starting 4 ... done
Creating and starting 5 ... done
Creating and starting 6 ... done
Creating and starting 7 ... done
Creating and starting 8 ... done
Creating and starting 9 ... done
Creating and starting 10 ... done

Now verify that it’s working by checking the X-Internal-Service header:

 # for i in {0..5}; do curl -v 2>&1 | grep X-Int; done
< X-Internal-Service:
< X-Internal-Service:
< X-Internal-Service:
< X-Internal-Service:
< X-Internal-Service:
< X-Internal-Service:

As you can see the requests are handled by different containers! Quick and easy.
The final docker image is ~10mb big:

# docker images | grep docker-compose-reverse-proxy
nicolai86/docker-compose-reverse-proxy   latest              dc50f87279f3        8 hours ago         9.832 MB

While I propably wouldn’t use this for a high traffic service this setup works reasonable well since its initial deployment, and since it’s < 100LOC it’s easy to replace and extend.

However, I’d still recommend using Kubernetes or Docker-Swarm in multi-host setups.

That’s it for now. Happy Hacking!