While working on umsatz I had to build a tiny currency exchange rates api in go.
While the original API works fine, it clocks in at more than 400 lines of code. Let’s write a shorter version!
In this post we’ll re-build the api, in three steps:
- download, parse & cache the EUR exchange rates history xml, provided by the ECB 1
- add a tiny HTTP JSON API to request rates
- periodically update the cache with new data
This will leave use with a tiny, HTTP currency exchange rates api.
Let’s start by downloading, parsing & caching the EUR exchange xml:
package main
import (
"encoding/xml"
"fmt"
"io"
"net/http"
)
// these structs reflect the eurofxref xml data structure
type envelop struct {
Subject string `xml:"subject"`
Sender string `xml:"Sender>name"`
Cubes []cube `xml:"Cube>Cube"`
}
type cube struct {
Date string `xml:"time,attr"`
Exchanges []exchange `xml:"Cube"`
}
type exchange struct {
Currency string `xml:"currency,attr"`
Rate float32 `xml:"rate,attr"`
}
// EUR is not present because all exchange rates are a reference to the EUR
var desiredCurrencies = map[string]struct{}{
"USD": struct{}{},
"GBP": struct{}{},
}
var eurHistURL = "http://www.ecb.europa.eu/stats/eurofxref/eurofxref-hist-90d.xml"
var exchangeRates = map[string][]exchange{}
func downloadExchangeRates() (io.Reader, error) {
resp, err := http.Get(eurHistURL)
if err != nil {
return nil, err
}
if resp.StatusCode != http.StatusOK {
return nil, fmt.Errorf("HTTP request returned %v", resp.Status)
}
return resp.Body, nil
}
func filterExchangeRates(c *cube) []exchange {
var rates []exchange
for _, ex := range c.Exchanges {
if _, ok := desiredCurrencies[ex.Currency]; ok {
rates = append(rates, ex)
}
}
return rates
}
func updateExchangeRates(data io.Reader) error {
var e envelop
decoder := xml.NewDecoder(data)
if err := decoder.Decode(&e); err != nil {
return err
}
for _, c := range e.Cubes {
if _, ok := exchangeRates[c.Date]; !ok {
exchangeRates[c.Date] = filterExchangeRates(&c)
}
}
return nil
}
func init() {
if reader, err := downloadExchangeRates(); err != nil {
fmt.Printf("Unable to download exchange rates. Is the URL correct?")
} else {
if err := updateExchangeRates(reader); err != nil {
fmt.Printf("Failed to update exchange rates: %v", err)
}
}
}
func main() {
fmt.Println("%v", exchangeRates)
}
There are a few things to note:
- we’re using a
map[string]struct{}to define which currencies we’re interested in.
This adds a little more code since we have to filter the exchange rates, but also cuts down memory usage. - we cache all exchange rates in memory and never update them. Since we’re dealing with historic data only this shouldn’t be a problem.
Next, we add a tiny HTTP wrapper:
// accept strings like /1986-09-03 and /1986-09-03/USD
var routingRegexp = regexp.MustCompile(`/(\d{4}-\d{2}-\d{2})/?([A-Za-z]{3})?`)
func exchangeRatesByCurrency(rates []exchange) map[string]float32 {
var mappedByCurrency = make(map[string]float32)
for _, rate := range rates {
mappedByCurrency[rate.Currency] = rate.Rate
}
return mappedByCurrency
}
func newCurrencyExchangeServer() http.Handler {
r := http.NewServeMux()
r.HandleFunc("/", func(w http.ResponseWriter, req *http.Request) {
if !routingRegexp.MatchString(req.URL.Path) {
w.WriteHeader(http.StatusBadRequest)
return
}
parts := routingRegexp.FindAllStringSubmatch(req.URL.Path, -1)[0]
requestedDate := parts[1]
requestedCurrency := parts[2]
enc := json.NewEncoder(w)
if _, ok := exchangeRates[requestedDate]; !ok {
w.WriteHeader(http.StatusNotFound)
return
}
var exs = exchangeRates[requestedDate]
if requestedCurrency == "" {
enc.Encode(exchangeRatesByCurrency(exs))
} else {
for _, rate := range exs {
if rate.Currency == parts[2] {
enc.Encode(rate)
return
}
}
w.WriteHeader(http.StatusNotFound)
}
})
return http.Handler(r)
}
func main() {
log.Printf("listening on :8080")
log.Fatal(http.ListenAndServe(":8080", newCurrencyExchangeServer()))
}
We can now run the API and it’ll work just fine:
$ curl http://127.0.0.1:8080/2014-12-12
{"GBP":0.7925,"USD":1.245}
$ curl http://127.0.0.1:8080/2014-12-12/USD
{"currency":"USD","rate":1.245}
Adding a period cache updater is quickly done:
func updateExchangeRatesCache() {
if reader, err := downloadExchangeRates(); err != nil {
fmt.Printf("Unable to download exchange rates. Is the URL correct?")
} else {
if err := updateExchangeRates(reader); err != nil {
fmt.Printf("Failed to update exchange rates: %v", err)
}
}
}
func updateExchangeRatesPeriodically() {
for {
time.Sleep(1 * time.Hour)
updateExchangeRatesCache()
}
}
func init() {
updateExchangeRatesCache()
}
func main() {
go updateExchangeRatesPeriodically()
log.Printf("listening on :8080")
log.Fatal(http.ListenAndServe(":8080", newCurrencyExchangeServer()))
}
The API will populate the cache on startup and update it once per hour afterwards.
How much memory does this consume? Let’s check:
func updateExchangeRates(data io.Reader) error {
var e envelop
decoder := xml.NewDecoder(data)
if err := decoder.Decode(&e); err != nil {
return err
}
for _, c := range e.Cubes {
if _, ok := exchangeRates[c.Date]; !ok {
exchangeRates[c.Date] = filterExchangeRates(c)
}
}
runtime.GC()
return nil
}
func printMemoryUsage() {
var memStats runtime.MemStats
runtime.ReadMemStats(&memStats)
fmt.Printf("total memory usage: %2.2f MB\n", float32(memStats.Alloc)/1024./1024.)
}
func main() {
go updateExchangeRatesPeriodically()
printMemoryUsage()
log.Printf("listening on :8080")
log.Fatal(http.ListenAndServe(":8080", newCurrencyExchangeServer()))
}
Note the new call to runtime.GC() which forces a garbage collection. This is important to get a correct memory usage report, otherwise we’d get varying and thus wrong memory usage reports.
Turns out the memory footprint is acceptable, without any optimizations:
- all data since 1999, all currencies: 5.137 MB
- all data since 1999, only USD and GBP: 0.836 MB
- last 90 days, all currencies: 0.211 MB
- last 90 days, only USD and GBP: 0.137 MB
Let’s wrap it up:
In less than 200 lines of code we managed to create a fully functional currency exchange rates api. Compared to the original version we do not cache exchange rates to disk, in favor of keeping everything in memory. This reduces the total lines of code considerably and also removes the need for a separate importer binary.
The API is not perfect, however:
- the data source does not contain data for weekends as well as holidays. For anything production ready we’d want to write a fallback which serves old exchange rates instead of just returning a 404.
However, I’ll leave it for now. You can find the entire source in this gist.
1: If anyone knows a higher precision, open data source for history currency exchange rates I’d love to know. Leave a comment.