Moving to Canada

As mentioned earlier 2017 will be a hectic year - I’m looking forward to immigrating to Canada. I’ll be relocating to Vancouver, British Columbia, to start working at Amazon.

Living abroad and working at one of the big five tech companies (Amazon, Apple, Google, Microsoft, Facebook) have been two of my goals for a long time, and both will be reality end of August. I can’t articulate how excited I am - it’s a huge opportunity to growth for me, and it puts me well outside of my comfort zone. I’ll learn a lot, and hopefully get to know beautiful Canada.

Anyhow, I’m looking forward to sharing more pictures again, some new technical posts, and a great time with my family in Canada! 🇨🇦


Modifying binaries to replace proprietary APIs

Note This is a follow up post on exploring private apis from late May.

Soon I want to use the Things 3 macOS application with my own API. To achieve this goal I have built a working SDK for the things cloud to understand the structure of the communication between client and server. This time I want to modify my Things 3 binary so it actually talks to an API of my choice. Let’s get started.

We know Things 3 talks to cloud.culturedcode.com. So this must be encoded inside the binary somewhere. To find out where, let’s use strings:

Strings looks for ASCII strings in a binary file or standard input.

$ strings /Applications/Things3.app/Contents/MacOS/Things3  | grep "cloud\."

The empty output tells me it’s not part of the main binary, so it must be part of some dependency. Next, I need to find out which dependencies Things 3 has, which can be done using otool:

The otool command displays specified parts of object files or libraries.

We’re specifically interested in shared libraries which come with the binary:

$ otool -L /Applications/Things3.app/Contents/MacOS/Things3 | grep "@"
  @rpath/FoundationAdditions.framework/Versions/A/FoundationAdditions (compatibility version 0.0.0, current version 0.0.0)
  @rpath/CoreJSON.framework/Versions/A/CoreJSON (compatibility version 0.0.0, current version 0.0.0)
  @rpath/KissXML.framework/Versions/A/KissXML (compatibility version 0.0.0, current version 0.0.0)
  @rpath/Base.framework/Versions/A/Base (compatibility version 0.0.0, current version 0.0.0)
  @rpath/ThingsModel.framework/Versions/A/ThingsModel (compatibility version 0.0.0, current version 0.0.0)
  @rpath/SyncronyCocoa.framework/Versions/A/SyncronyCocoa (compatibility version 0.0.0, current version 0.0.0)
  @rpath/ThingsTools.framework/Versions/A/ThingsTools (compatibility version 0.0.0, current version 0.0.0)
  @rpath/QuartzAdditions.framework/Versions/A/QuartzAdditions (compatibility version 0.0.0, current version 0.0.0)
  @executable_path/../Frameworks/TXOnboardingPopUpKit.framework/Versions/A/TXOnboardingPopUpKit (compatibility version 0.0.0, current version 0.0.0)
  @executable_path/../Frameworks/TXVisualDebugKit.framework/Versions/A/TXVisualDebugKit (compatibility version 0.0.0, current version 0.0.0)
  @executable_path/../Frameworks/TXTrialIndicatorKit.framework/Versions/A/TXTrialIndicatorKit (compatibility version 0.0.0, current version 0.0.0)
  @executable_path/../Frameworks/TXPopUpMenuKit.framework/Versions/A/TXPopUpMenuKit (compatibility version 0.0.0, current version 0.0.0)
  @executable_path/../Frameworks/TXLinkDetectorKit.framework/Versions/A/TXLinkDetectorKit (compatibility version 0.0.0, current version 0.0.0)
  @executable_path/../Frameworks/TXListKit.framework/Versions/A/TXListKit (compatibility version 0.0.0, current version 0.0.0)
  @executable_path/../Frameworks/TXToolTipKit.framework/Versions/A/TXToolTipKit (compatibility version 0.0.0, current version 0.0.0)
  @executable_path/../Frameworks/TXCloudIndicatorKit.framework/Versions/A/TXCloudIndicatorKit (compatibility version 0.0.0, current version 0.0.0)
  @executable_path/../Frameworks/TXToolbarKit.framework/Versions/A/TXToolbarKit (compatibility version 0.0.0, current version 0.0.0)
  @executable_path/../Frameworks/TXTrialExpiredKit.framework/Versions/A/TXTrialExpiredKit (compatibility version 0.0.0, current version 0.0.0)
  @executable_path/../Frameworks/TXQuickEntryKit.framework/Versions/A/TXQuickEntryKit (compatibility version 0.0.0, current version 0.0.0)
  @executable_path/../Frameworks/TXDatePickerKit.framework/Versions/A/TXDatePickerKit (compatibility version 0.0.0, current version 0.0.0)
  @rpath/SMStateMachine.framework/Versions/A/SMStateMachine (compatibility version 0.0.0, current version 0.0.0)
  @executable_path/../Frameworks/TXWindowKit.framework/Versions/A/TXWindowKit (compatibility version 0.0.0, current version 0.0.0)
  @executable_path/../Frameworks/HockeySDK.framework/Versions/A/HockeySDK (compatibility version 1.0.0, current version 1.0.0)
  @executable_path/../Frameworks/TXMainWindowKit.framework/Versions/A/TXMainWindowKit (compatibility version 0.0.0, current version 0.0.0)
  @executable_path/../Frameworks/TXAppKit.framework/Versions/A/TXAppKit (compatibility version 0.0.0, current version 0.0.0)
  @executable_path/../Frameworks/TXTagKit.framework/Versions/A/TXTagKit (compatibility version 0.0.0, current version 0.0.0)
  @executable_path/../Frameworks/TXPopoverKit.framework/Versions/A/TXPopoverKit (compatibility version 0.0.0, current version 0.0.0)
  @executable_path/../Frameworks/TXCheckListKit.framework/Versions/A/TXCheckListKit (compatibility version 0.0.0, current version 0.0.0)

Lots of shared libraries, but we know we’re interested in functionality related to the cloud synchronization, so SyncronyCocoa looks like a likely candidate, as strings confirms:

strings /Applications/Things3.app/Contents/Frameworks/SyncronyCocoa.framework/SyncronyCocoa | grep "cloud\."
https://cloud.culturedcode.com/
https://development-dot-thingscloud.appspot.com/

Nice. Next, we need to modify the shared library to use a different domain which, conveniently for development, points to localhost:

cat /etc/hosts | grep cultt
127.0.0.1 cloud.culttcoder.local

Note that it’s important that the domain has the same number of characters as the one we’re trying to replace - if it’s shorter or longer the Things 3 binary will crash on launch.

Now that we have a domain pointing to our local machine we need to patch the SyncronyCocoa.framework. I’ll be using dd to modify the binary. dd in combination with strings is a great tool for making smaller modifications to binary files.

First, we need to find the offset of the string inside the library, using strings:

strings -t d /Applications/Things3.app/Contents/Frameworks/SyncronyCocoa.framework/SyncronyCocoa | grep "cloud\."
36078 https://cloud.culturedcode.com/
36110 https://development-dot-thingscloud.appspot.com/

This tells us that the string https://cloud.culturedcode.com/ is located at offset 36078. Now, we can use dd to change the library at position 36078 so it points to our local domain:

$ printf "https://cloud.culttcoder.local/\x00" > /tmp/api-dns
$ sudo dd if=/tmp/api-dns of=/Applications/Things3.app/Contents/Frameworks/SyncronyCocoa.framework/SyncronyCocoa obs=1 seek=36078 conv=notrunc

Let’s verify the patch was successful:

$ strings -t d /Applications/Things3.app/Contents/Frameworks/SyncronyCocoa.framework/SyncronyCocoa | grep "cloud\."
36078 https://cloud.culttcoder.local/
36110 https://development-dot-thingscloud.appspot.com/

Again, nice. Now we’ve successfully patched the library to talk to our local domain, but Things 3 will crash. As it’s an App Store binary everything is code signed, and our patch invalidated the code signature. Let’s fix that:

$ sudo codesign -f -s - /Applications/Things3.app/Contents/Frameworks/SyncronyCocoa.framework/SyncronyCocoa
/Applications/Things3.app/Contents/Frameworks/SyncronyCocoa.framework/SyncronyCocoa: replacing existing signature

Alright, Things 3 works again, but we don’t have a compatible API server yet. For now let’s create a local setup with a proxy which in turn forwards all requests to the real things cloud API:

For this to work we need a valid SSL certificate. A real deployment can easily obtain one using letsencrypt, but for local development a self signed certificate marked as trusted works just fine:

$ openssl genrsa -out server.key 2048
$ openssl req -new -x509 -sha256 -key server.key -out server.crt -days 3650 -subj '/CN=cloud.culttcoder.local/O=Private/C=DE'
$ sudo security add-trusted-cert -d -r trustRoot -k /Library/Keychains/System.keychain server.crt

Note that the certificate must be valid for the domain I chose before. Next up I’ve setup a tiny HTTPS proxy written in Go:

package main

import (
  "flag"
  "log"
  "net/http"
  "net/http/httputil"
)

func main() {
  listen := flag.String("listen", ":443", "port to listen on")
  flag.Parse()

  director := func(req *http.Request) {
    req.URL.Scheme = "https"
    req.Host = "cloud.culturedcode.com"
    req.URL.Host = "cloud.culturedcode.com"
    req.Header.Set("Connection", "close")

    dump, _ := httputil.DumpRequest(req, true)
    log.Printf("%s\n", string(dump))
  }
  proxy := &httputil.ReverseProxy{Director: director}
  log.Printf("Listening on %s\n", *listen)

  err := http.ListenAndServeTLS(*listen, "server.crt", "server.key", proxy)
  if err != nil {
    log.Fatal("ListenAndServe: ", err)
  }
}

This proxy will forward any request received to https://cloud.culturedcode.com. Now, when we start Things 3 it can talk to the things cloud, just as before, but through our local proxy.

things3 talking to my local thingscloud proy

Soon, it won’t be talking to the things cloud API at all…

That’s it for today, happy hacking!


Moving Forward

Everything changes and nothing stands still. Heraclitus

And 2017 will be quite a hectic year for me because of this. I’m looking forward to sharing more non-technical things in the next few months, too. Mostly pictures I guess, but we’ll see what I’ll find interesting enough to share (:


Exploring private HTTPS apis

Today I want to take a look at how you can explore private HTTPS APIs. I’ll be using @culturedcode Things Cloud as an example: it’s the main engine behind keeping Things for iOS and Things for macOS in sync, and as there is no web version available it’s a little more tricky to take a peek behind the scenes.

First off some requirements: you need to be running macOS for this to work, and you need a Things 3 installation along with a thingscloud account.

Now all traffic between Things and Things cloud is exchanged via HTTPS, which means that all you can see is the DNS name the traffic is going to.

The first step should always be to hope for programming mistakes, like maybe the App is not validating the HTTPS certificate at all, or it’s not using certificate pinning. If this was the case one could use a regular proxy to peek at the traffic, like mitmproxy.

$ pip install "mitmproxy==0.18.2"
$ mitmproxy

Now configure your system to route HTTPS traffic through your local proxy:

sudo networksetup -setsecurewebproxy "Wi-Fi" 127.0.0.1 8080
sudo networksetup -setsecurewebproxystate "Wi-Fi" on

Opening Things however won’t result in any network calls. Luckily culturedcode did a fine job ensuring Things doesn’t just talk to anybody. As we don’t have the real private key to decrypt the traffic we’re done here, right?
Wrong.

One can actually instruct the OS to dynamically load a library at start, using DYLD_INSERT_LIBRARIES, and overwrite the method calls executed when validating SSL certificates. man dyld says:

DYLD_INSERT_LIBRARIES.
This is a colon separated list of dynamic libraries to load before the ones specified in the program. This lets you test new modules of existing dynamic shared libraries that are used in flat-names- pace images by loading a temporary dynamic shared library with just the new modules.

At Blackhot 2012 this approach was first demoed - known as SSL Kill Switch 2. This allows us to disable the SSL verification in place.

Assuming you have XCode installed it’s easy to compile the dylib yourself. Once you’re finished you need to first enable SSL KIll Switch:

$ export DYLD_INSERT_LIBRARIES=$(pwd)/SSLKillSwitch.framework/Versions/A/SSLKillSwitch
2017-05-28 23:44:51.952 sh[59630:4497019] === SSL Kill Switch 2: Fishhook hook enabled.
2017-05-28 23:44:52.033 sh[59632:4497025] === SSL Kill Switch 2: Fishhook hook enabled.
2017-05-28 23:44:52.056 tail[59639:4497037] === SSL Kill Switch 2: Fishhook hook enabled.
2017-05-28 23:44:52.076 sed[59635:4497038] === SSL Kill Switch 2: Fishhook hook enabled.

Now, start Things again:

# /Applications/Things3.app/Contents/MacOS/Things3
2017-05-28 23:47:10.755 Things3[59953:4500129] === SSL Kill Switch 2: Fishhook hook enabled.

When you foreground Things you should see API requests in your proxy:

mitproxy /w API requests

Now all that’s left is to use Things, watch which API calls are being made when, and start inferring how the API actually works. As you can see peeking behind HTTPS APIs used by native application is actually very easy.

The results of me playing around with Things are available on Github: things-cloud-sdk. It’s a basic SDK which allows you to read and write to thingscloud; the most work was actually spent guessing what the cryptic payload names are.

I hope this helps the next time you want to peek behind the scenes of a native app & private HTTPS api. Until then - happy hacking!