The five important abilities of systems

Reading The Architecture of Open Source Applications, especially Chapter 4, struck a cord with me:

The authors, Margo Seltzer and Keith Bostic, state, that building systems based on simple components as building blocks lead to systems which are superior to monoliths in the five important -abilities: understandability, extensibility, maintainability, testability, and flexibility.

This made me wonder how to clearly differentiate between these abilities, since many discussions about software architecture evolve around one or more of these abilities.

Also the abilities are rather generic, making them applicable to system design in general, so I consider a strong understanding of them essential to be able to perform well.

Here are some definitions & thoughts:

understandability

describes if the code clearly communicates its purpose to readers other then the author.

This is important because code is read more often than written and if your code is hard to understand it will take up more time to read, or worse, it won’t be understood at all; Problem is this is a subjective metric - it depends on the knowledge & background of the reader.

To me, this effectively comes down to:

  • don’t be clever.
  • make implicit assumptions explicit.
  • optimize for readability.

extensibility

speaks to a system being able to accommodate new functionality or changes over time, without requiring bigger rewrites of existing code.

I think every software engineer has experienced changes cascading through their code base. Yet I think it’s hard to avoid sometimes. When building a system, chances are you don’t know every thing up front, meaning you can always run into this situation.

While I think extensibility can be achieved in parts by building loosely coupled, eventually consistent systems, it’s better to perform technical spikes regularly, to ensure the problem at hand is understood properly, as well as possible integrations into the existing code base.

maintainability

describes the afford required to keep a system in good shape, e.g. the amount of work required to regularly upgrade your dependencies, fix defects; but also incorporate non functional requirements after a system has launched.

Since I don’t know of a better way to describe this I’d say “how often do you upset your product owner?” - mostly because product owners always want new features, and you can’t deliver them if you’re busy keeping the system up.

testability

measures how easy it is to write test for your system.

Good testability often speaks to an easy understand code base; It’s also extremely important as a safety net when thinking about maintenance. I use tests as an indicator on how easy it is to understand a system. If you’re tests don’t immediately point you to the source of a bug, or require so much setup that it’s hard to even find the testing code, you don’t have this ability.

flexibility

describes how your system adapts to external changes.

External changes for a system range from slow responses from 3rd parties over occasional timeouts to unavailability.

To me, this mostly concerns operations but is extremely important to get right. If done wrong, your software will be hard to keep up and running.
I’d always argue: if it is worth building a system to begin with, it should always be kept available.

Fortunately, many architectural patterns exist to enable this.

The challenge

I think that these abilities build upon each other; and you always have to find trade offs since you can’t fulfill all of them at once. And trade offs can only be discussed in a specific context, meaning there is no one size fits all solution.

For example, rather generic trade offs:

  • when building a technical spike, it might be okay to write less understandable & maintainable code, since your goal is a proof something is actually doable. When you have to go live with one, however, you should invest time in rebuilding critical systems/ part of a system, which are hard to understand/ maintain.
  • when taking over an existing project on should check if the system reaches the desired level of flexibility & maintainability. Nobody wants to be stuck with a system that fails all the time.
  • when starting a new project, the project expectations should be stated explicitly. E.g. when the project team is expected to change often, writing understandable and testable code should have higher priority, to reduce onboarding time.

But also more specific ones:

  • hard to understand code is hard to test. You are probably missing good abstractions.
  • introducing abstractions just for tests makes your code harder to understand.
  • decoupling two systems, e.g. with a circuit breaker, also make it less easy to understand; simply because there’s more code to read.

If you think it’s simple, then you have misunderstood the problem.
- Bjarne Stroustrup

My key take away is this: software engineering is hard, and it helps to have some guidelines in mind when trying to make the best decision for the moment.
I think these abilities help with that.


docker-compose and load balancing

We’re using docker-compose in production @work and I needed an ad-hoc, lightweight solution to load-balance requests across multiple instances of a specific service, all running on a single host. While docker-compose allows you to scale your services you need to take care of load balancing yourself.

Before sharing my approach, which uses golang, let’s start with the problem description first.

I’ll be using a dummy service, httpd echo, for demonstration purposes.

Given a docker-compose.yml like this:

echo:
  image: nicolai86/http-echo
  ports:
    - 9090:9090

Assuming you need to scale echo, the above docker-compose.yml won’t allow you to scale echo because you defined a port mapping:

$ docker-compose scale echo=2
WARNING: The "echo" service specifies a port on the host. If multiple containers for this service are created on a single host, the port will clash.
Creating and starting 2 ... error

ERROR: for 2  failed to create endpoint dockercomposelbexample_echo_2 on network bridge: Bind for 0.0.0.0:9090 failed: port is already allocated

Removing the port mapping allows you to scale your container but now you need to route the traffic using a reverse proxy.

Available solutions include using a HAProxy / Nginx, with dynamic configuration reloads when your instance pool change. However, these solutions require at least two new, huge-ish docker containers, because of the dependencies required to make this work (did I mention fast access to the internet is a problem?).

Since golang already comes with a ReverseProxy implementation I decided to roll my own reverse proxy with docker-compose integration.

It works like this:

echo:
  image: nicolai86/http-echo

proxy:
  image: nicolai86/docker-compose-reverse-proxy
  ports:
    - 80:8080
  volumes:
    - "${DOCKER_CERT_PATH}:${DOCKER_CERT_PATH}"
  environment:
    DOCKER_COMPOSE_SERVICE_NAME: echo
    DOCKER_HOST: "${DOCKER_HOST}"
    DOCKER_TLS_VERIFY: "${DOCKER_TLS_VERIFY}"
    DOCKER_CERT_PATH: "${DOCKER_CERT_PATH}"
# docker-compose up -d
Starting dockercomposelbexample_proxy_1
Starting dockercomposelbexample_echo_1

# docker-compose scale echo=10
Creating and starting 2 ... done
Creating and starting 3 ... done
Creating and starting 4 ... done
Creating and starting 5 ... done
Creating and starting 6 ... done
Creating and starting 7 ... done
Creating and starting 8 ... done
Creating and starting 9 ... done
Creating and starting 10 ... done

Now verify that it’s working by checking the X-Internal-Service header:

 # for i in {0..5}; do curl 192.168.99.100 -v 2>&1 | grep X-Int; done
< X-Internal-Service: 172.17.0.15:9090
< X-Internal-Service: 172.17.0.8:9090
< X-Internal-Service: 172.17.0.7:9090
< X-Internal-Service: 172.17.0.8:9090
< X-Internal-Service: 172.17.0.10:9090
< X-Internal-Service: 172.17.0.13:9090

As you can see the requests are handled by different containers! Quick and easy.
The final docker image is ~10mb big:

# docker images | grep docker-compose-reverse-proxy
nicolai86/docker-compose-reverse-proxy   latest              dc50f87279f3        8 hours ago         9.832 MB

While I propably wouldn’t use this for a high traffic service this setup works reasonable well since its initial deployment, and since it’s < 100LOC it’s easy to replace and extend.

However, I’d still recommend using Kubernetes or Docker-Swarm in multi-host setups.

That’s it for now. Happy Hacking!


Prototyping with XCode Playgrounds

With the introduction of the Swift programming language Apple also introduced Playgrounds, interactive environments for easy sharing & prototyping.

Today I want to share a tiny example on how to use them to prototype custom UIViews. Specifically a PolarClock. You can download the complete Playground here.

I love playgrounds because they allow you to quickly sketch ideas, and visualize steps, before heading for a fully fledged implementation. Let’s try it out:

First, open a recent XCode (I’m using 7.2) and create a new Playground.

We’ll start by constructing a circle using polar coordinates:

let radius = 50.0
var points = [CGPoint]()
for degree in 0.stride(to: 360.0, by: 10) {
  let theta = (90 + degree) * M_PI / 180.0
  let x = radius * cos(theta)
  let y = radius * sin(theta)
  let p = CGPoint(x: -x, y: y)
  points.append(p)
}

While you can not visualize [CGPoint] with Playgrounds, you can visualize a UIBezierPath:

let bezierPath = UIBezierPath()
bezierPath.moveToPoint(points[0])
for var i = 1; i < points.endIndex; i++ {
  bezierPath.addLineToPoint(points[i])
}
bezierPath

When you click the visualization button you should see a circle:

UIBezierPath visualized

Let’s wrap this in a tiny UIView, called Polar:

import UIKit

public class Polar: UIView {
  public var radius: CGFloat = 20
  public var completeness: CGFloat = 0.1 {
    didSet {
      self.setNeedsDisplay()
    }
  }
  public var strokeWidth: CGFloat = 10
  public var stepSize: CGFloat = 0.25
  public var color: UIColor = UIColor.redColor()

  func path(ps: [CGPoint]) -> UIBezierPath {
    let bezierPath = UIBezierPath()
    bezierPath.lineJoinStyle = .Round
    bezierPath.fillWithBlendMode(.Normal, alpha: 0.0)
    bezierPath.moveToPoint(ps[0])
    for var i = 1; i < ps.endIndex; i++ {
      bezierPath.addLineToPoint(ps[i])
    }
    return bezierPath
  }

  func circle(radius: Double) -> [CGPoint] {
    var points = [CGPoint]()
    let center = CGPoint(x: self.frame.width/2, y: self.frame.height/2)

    for degree in 0.0.stride(to: 360.0*Double(self.completeness), by: Double(self.stepSize)) {
      let theta = (270 + degree) * M_PI / 180.0
      let x = radius * cos(theta)
      let y = radius * sin(theta)
      let p = CGPoint(x: Double(center.x)+x, y: Double(center.y)+y)
      points.append(p)
    }

    return points
  }

  public override func drawRect(rect: CGRect) {
    let ctx = UIGraphicsGetCurrentContext()
    CGContextSetLineWidth(ctx, self.strokeWidth)
    CGContextAddPath(ctx, path(circle(Double(self.radius))).CGPath)
    self.color.set()
    CGContextStrokePath(ctx)
  }
}

The Polar view just placed the above code in separate functions and added some properties, as well as a drawRect method to display the view.

The observing reader might spot the changed theta calculation; this is necessary because the coordinates system for UIViews is different than the preview in the playground. To support a clockwise rotation at the center we start at 270°.

Finally, let’s use the class to build a real PolarClock:

let frame = CGRect(x: 0, y: 0, width: 375, height: 667)
let view = UIView(frame: frame)
XCPShowView("PolarClock", view: view)

let colors = [
  UIColor(red: 23.0/255.0, green: 170.0/255.0, blue: 178.0/255.0, alpha: 1.0),
  UIColor(red: 255.0/255.0, green: 32.0/255.0, blue: 172.0/255.0, alpha: 1.0),
  UIColor(red: 7.0/255.0, green: 241.0/255.0, blue: 255.0/255.0, alpha: 1.0)
]

let clockFrame = CGRect(x: 100, y: 100, width: 100, height: 100)
let clock = (
  Polar(frame: clockFrame),
  Polar(frame: clockFrame),
  Polar(frame: clockFrame)
)
clock.0.backgroundColor = UIColor(red: 0, green: 0, blue: 0, alpha: 0)
clock.0.color = colors[0]
clock.0.radius = 45.0
view.addSubview(clock.0)

clock.1.backgroundColor = UIColor(red: 0, green: 0, blue: 0, alpha: 0)
clock.1.color = colors[1]
clock.1.radius = 35.0
view.addSubview(clock.1)

clock.2.backgroundColor = UIColor(red: 0, green: 0, blue: 0, alpha: 0)
clock.2.color = colors[2]
clock.2.radius = 25.0
view.addSubview(clock.2)

let timer = dispatch_source_create(DISPATCH_SOURCE_TYPE_TIMER, 0, 0, dispatch_get_main_queue())
dispatch_source_set_timer(timer, DISPATCH_TIME_NOW, 1 * NSEC_PER_SEC, 0)
dispatch_source_set_event_handler(timer) {
  let now = NSDate()
  let calendar = NSCalendar.currentCalendar()
  let components = calendar.components([NSCalendarUnit.Hour, NSCalendarUnit.Minute, NSCalendarUnit.Second], fromDate: now)
  let hour = components.hour
  let minutes = components.minute
  let seconds = components.second

  clock.0.completeness = CGFloat(hour) / 24.0
  clock.1.completeness = CGFloat(minutes) / 60.0
  clock.2.completeness = CGFloat(seconds) / 60.0
}
dispatch_resume(timer)

You’ll see something like this, before the playground terminates after some seconds:

UIBezierPath visualized

That’s it! Now it’s just a tiny step to implement the @IBDesignable and use the class inside interface builder:

@IBDesignable
public class Polar: UIView {
  @IBInspectable public var radius: CGFloat = 20
  @IBInspectable public var completeness: CGFloat = 0.1
  @IBInspectable public var strokeWidth: CGFloat = 10
  @IBInspectable public var stepSize: CGFloat = 0.25
  @IBInspectable public var color: UIColor = UIColor.redColor()
  // … omitted
}

Now you can also use your Polar view inside interface builder:

UIBezierPath visualized

I love that you can easily prototype complete views with Playgrounds. The only catch is that only Swift is supported right now, so hopefully you can live without a debugger.

All in all the XCode playgrounds are awesome to share executable code, teach and learn. You should use them probably, too.

Anyway, that’s it for today. Happy hacking!


AWS ECS with CloudWatch

AWS ECS is a nice environment to run your applications in. But sometimes you want “hot of the press” Docker features, which you can not configure in your task definitions just yet - like the awslogs log driver, which forwards your Docker logs to CloudWatch.

When using the Amazon provided ECS AMIs the setup can be bit complicated for non-us regions, so here’s a simple solution to make it work until the task definitions support the log driver:

Use cloud-init userdata to configure your instances properly:

#cloud-config

write_files:
  - path: /etc/ecs/ecs.config
    content: |
        ECS_CLUSTER=my-cluster
    owner: root:root
  - path: /etc/awslogs/awscli.conf
    content: |
        [plugins]
        cwlogs = cwlogs

        [default]
        region = eu-west-1
        aws_access_key_id = AKIAIOSFODNN7EXAMPLE
        aws_secret_access_key = wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
    owner: root:root
  - path: /etc/sysconfig/docker
    content: |
      DAEMON_MAXFILES=1048576
      OPTIONS="--default-ulimit nofile=1024:4096 --log-driver=awslogs --log-opt awslogs-region=eu-west-1 --log-opt awslogs-group=my-cluster"

package_upgrade: true
packages:
  - awslogs

runcmd:
  - service awslogs start
  - chkconfig awslogs on
  - sed -i '/region = us-east-1/c\region = eu-west-1' /etc/awslogs/awscli.conf
  - service awslogs restart
  - service docker restart
  - start ecs

First, I’m configuring my ECS agent to join the right cluster, then I’m writing the awslogs agent configuration.
Here’s the catch I’ve tripped over repeatedly:

when installing the awslogs package, the configuration files region always get’s replaced with us-east-1.

To correct this I’m using sed, replacing the wrong region, and restarting the awslogs agent.

Lastly, the Docker configuration files is overwritten, instructing Docker to forward all logs to CloudWatch, into a log group called my-cluster. This requires a restart of the Docker daemon, followed by a start of the ECS agent.

Done.

Hopefully this workaround won’t be required for too long, because there are two downsides: a) all logs are forwarded to CloudWatch, even those you are not interested in, and b) you can not direct them to per-container log groups.

But for now, it’s good enough - and easy to integrate into tools like terraform :)