Dude, where’s my Kaizen?

From .NET to Ruby to whatever interests me next

RVM CDPATH trouble

CDPATH is one of those things I’ve become accustomed to over the last year and would wan’t a shell without it anymore. In case you’ve never heard of it, it gives you the possiblity to quickly access your favorite directories from anywhere via the cd command.

But as I found out this morning a misconfigured CDPATH can lead to very strange behaviors with RVM. My .zshrc exported CDPATH like this:

1
export CDPATH=~/Coding/Laboratory/

That’s the path to the folder where all my git repositories are located. People more versed with a shell environment probably already see the problem. If you try to install the latest Ruby binaries with rvm install using this configuration, it won’t compile. It aborts with something like this:

1
2
3
4
5
6
/bin/sh: line 0: cd: ext/-test-/array/resize: No such file or directory
/bin/sh: line 0: cd: ext/-test-/add_suffix: No such file or directory
make[1]: *** [ext/-test-/array/resize/all] Error 1
make[1]: *** Waiting for unfinished jobs....
make[1]: *** [ext/-test-/add_suffix/all] Error 1
make: *** [build-ext] Error 2

A bit of google search finally lead me to the solution. . needs to be in the CDPATH as well.

1
export CDPATH=.:~/Coding/Laboratory/

And finally I was able to compile. Good to know, even better to share :-)


Continuous Jasmine

After lots of years I’ve currently the pleasure of working on a Javascript heavy application again. Last time when I worked with Javascript, there was no Backbone, no Underscore, not even a JQuery. Yes, that long ago and it sucked that much for me that I stayed away from the frontend side of things like the devil from water in the following years.

Everybody should be able to get a second chance in life, so when the option came up I thought to myself: “Hey, why not. Let’s give it a try”. Javascript has come a long way since those old times. It’s actually a pretty decent coding environment today. Aestetically it’s still nowhere close to my personal favorites Ruby and FSharp, but it’s at least ok :)

One thing that has also changed since the last time I did Javascript is my willingness to write code that isn’t backed up by some sort of automated testing. It’s virtually non existent. I know I’m a bit of a hardliner with this, but I consider that part of being a professional developer. So I went out to see what I would use to test my Javascript code and I settled with Jasmine. That should come to no surprise since I’m a big fan of Rspec and MSpec as well.

The Jasmine gem

I experimented with several setups to execute the Jasmine specs. Please note that I’ve no problem with a Ruby dependency and honestly, I didn’t bother to search for Node based solutions. I finally settled with the Jasmine gem.

It brings some interesting things to the table. First of all it frees you from managing the SpecRunner.html directly. You don’t have to include spec files manually any more. Secondly and more interesting it comes along with some rake tasks for running the specs in a CI environment. After installing the gem, all you need to do is to run $ jasmine init. This will create the project structure for you, including a rakefile which sets up the rake interface.

$ rake jasmine builds your spec runner file, hosts it inside a Webrick instance and opens a browser showing the specs. That’s nice, but I don’t want to have to reload the browser manually everytime I change a file. That would be tedious.

$ rake jasmine:ci is also interesting. It uses rspec and selenium to fire up the tests, grab the test results from the browser and display them back in the console. It works pretty well for a continuous integration scenario, but it’s also slow as hell. Let’s say the output in the console could also be optimized.

One thing I stumbled over, is a bug in the current version 1.3.1. It relies on yaml, but doesn’t seem to require it. You can work around it partially by manually requiring it in the rakefile, but this doesn’t work for the jasmine:ci task. You can fix this by pinning the version in your Gemfile to 1.3.0 until the problem is fixed.

Gemfile
1
2
3
4
5
6
source :rubygems

group :development, :test do
  gem 'rake'
  gem 'jasmine', '=1.3.0'
end

Jasmine.vim

What is nice, though, it’s that if you’re a vim user, you can use Jasmine.vim which comes with keybindings and vim commands for those rake tasks plus syntax highlighters and lots of useful Jasmine snippets.

I played around with it for an hour and came to the conclusion that although the technical integration is nice, the feedback loop is practically unusable. It reminded me of the moment when I started a test in the first legacy Rails app I encountered and stared into the screen, waiting for something to happen and waiting and waiting. That’s not how a feedback loop is supposed to be.

So I asked @derickbailey, the guy that produces the excellent Watch me code screencasts, what setup he uses in the screencasts. That looked pretty ideal.

Enter guard + livereload

He had vim and a browser window open side by side and whenever he saved a file in vim, the browser window automatically refreshed and rerun his specs. Fast and snappy. That’s how I wanted my feedback loop to be as well.

It’s actually pretty easy to setup. You need guard-livereload and the livereload Chrome extension. Just add guard-livereload to your Gemfile and run $ bundle install.

Gemfile
1
2
3
4
5
6
7
8
source :rubygems

group :development, :test do
  gem 'rake'
  gem 'jasmine', '=1.3.0'
  gem 'rb-fsevent'
  gem 'guard-livereload'
end

rb-fsevent is needed to monitor filesystem changes on OSX which powers my development machine. After that, all you need to do configuration wise is to run guard init livereload, which will add a Guardfile to your project containing the configuration for guard-livereload. You need to tweek it a bit to match the project structure, but that’s easily done.

Guardfile
1
2
3
4
guard 'livereload' do
  watch(%r{public/.+\.(css|js|html)})
  watch(%r{spec/.+\.(css|js|html)})
end

Next up you need to install the browser plugin. The one for Chrome can be found here. Be sure to be logged in with your Google account. I don’t know why, but the extension won’t install without that and gives you a nasty CRX_MAGIC_NUMBER_INVALID error instead. What a lovely error message. When installed you should see a new circle button in your Chrome toolbar.

Now that we’ve got all the pieces together, you can run the development server in one window by running $ rake jasmine and start guard with $ guard start. Once that has been done, you need to open the root url of your development server in the Chrome browser and click the livereload circle button. In the guard console there should now be a notification that a new browser has been connected.

You’re now ready to go and TDD the shit out of your Javascript project …


Oh my Faraday

I’m pretty glad I didn’t write any tests for my client architecture so far. Somehow I had the feeling that most of the code was going to be deleted. And guess what the feeling was right. Last week I replaced the complete middleware stack handling I described in the previous post with the Faraday gem and its friends.

Faraday is an HTTP client lib that provides a common interface over many adapters (such as Net::HTTP) and embraces the concept of Rack middleware when processing the request/response cycle.

Faraday project site

Here’s how it works: If you supply a block to Faraday.new you can use it to configure the stack.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
@connection = Faraday.new(url: base_uri, user_agent: 'social network client', headers: { accept: 'application/json' }) do |builder|
  builder.request   :json
  builder.request   :basic_auth, options[:user], options[:password]
  builder.request   :retry

  builder.response  :json, :content_type => /\b(json|json-home)$/
  builder.response  :mashify

  builder.use       :instrumentation
  builder.adapter   :typhoeus
end

#Like any other HTTP library
response = @connection.get('/users')

Faraday works pretty much like Rack::Builder. You compose the request/response pipeline out of middlewares via the use method. They’re composed from top to botton. request and response are just syntactic sugar for the configuration. builder.request :json is nothing more than builder.use Faraday::Request::Json, though it probably looks cooler the first way. Here’s the full commit that introduced Faraday into the client.

The client is functional with this, but I lost all the caching capabilities. As I found out I couldn’t use Rack::Cache as a middleware. The API’s look similar, but the env hash which gets passed from middleware to middleware isn’t. But in the end this wasn’t a big problem. The faraday-http-cache gem does exactly what I need. And here comes the beauty of the middleware based model: All that needs to be done on the client side is to add a new middleware and you’re done!

1
builder.use :http_cache, :file_store, Dir.pwd + '/tmp'

I cheated a bit though. When I re-ran the client against the server, it didn’t cache. After a bit looking into the code of the faraday-http-cache gem I realized that I had to include public in the Cache-Control directive returned with my responses in order to make it integrate well with the gem.

1
2
3
if stale?(etag: users, public: true)
 render json: users
end

After changing that everything worked as expected. I’m a bit undecided regarding the Cache-Control: public. For all I know this indicates that the response can be cached in shared public caches, which is probably something you often don’t want for your client. private on the other hand indicates that content can be cached in the non-shared private cache of the client, which is in my opinion more appropriate for our scenario and a valid caching scenario. I guess it’s a bug in the faraday-http-cache gem.

The full commit that introduced caching back into the client can be found here. I also removed the Request base class in that commit, because it became unnecessary after the refactoring.

This is probably the end of the heavy client focus for a while. I think I’ve got a pretty good understanding now, how I would start with writing a client for a web API. Sure we haven’t talked about Hypermedia much so far, but we’ll catch up to that.

Next to the table on which I’m writing this post a sheet of paper is waiting to be used for a post. It’s from last October, specifically from the Dev Open Space 2012 in Leipzig. Sergey Shishkin and I used it to prepare a talk about Hypermedia. I won’t surprise you when I reveal that it shows a Hypermedia graph. This is where we continue next time.


HTTP + client side middleware stack == Win

Unbelievable how time flies by. Someone who doesn’t know me personally, could’ve probably come to the conclusion that this little blog series has already halted before it even began in the meantime. Let me assure you that is not the case. I’m still invested in this project. Though I need to admit that things are progressing much slower than I originally had planed or hoped for.

So what is the current state of our journey? And what did I learn so far?

I started by building the most simple Rails based API I could think of. At the moment it’s only capable of listing users and showing an individual user. It doesn’t have any Hypermedia capabilities what so ever. It’s uses the Rails-API for a slimmed down version of the Rails stack, Devise for implementing HTTP Basic Authentication and the faker gem to populate the development database with some test data.

The home document

Then I added a minimal home document that lists the servers resources and how they can be interacted with. I implemented the specification as a mini DSL in Ruby, which is probably over the top, but was fun anyway.

The initial version of the home document for my appGithub
1
2
3
4
5
6
7
8
9
10
11
12
module Api
  HomeDocument = JsonHome::HomeDocument.define do
    resource 'urn:sna:rel:users' do
      href '/users'
    end

    resource 'urn:sna:rel:user' do
      href_template '/users/{user_id}'
      href_vars user_id: 'http://upcoming/documentation/page'
    end
  end
end

So what do you see here? This document lists both resources currently provided by the server app. They both got a unique name which in this case in an URN. One of them (urn:sna:rel:users) has a fixed URL (/users). The other one (urn:sna:rel:user) uses an URI template ala RFC6570. It also lists all parameters that are contained in the template explicitly, together with a link to the documentation of that parameter.

This homedocument will be served at the entry point of the API, which in the case of my sample app is ‘/api’.

routes.rbGithub
1
2
3
4
5
6
7
8
9
SocialNetworkServer::Application.routes.draw do
  devise_for :users, skip: :sessions

  get 'api', to: 'api/home_document#show'

  namespace "api" do
    resources :users, only: [:index, :show]
  end
end

If you query this URI you’ll get the following response

routes.rbGithub
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
$ curl -H 'Accept: application/json-home' -u 'test@test.de:testtest' http://localhost:3000/api

{
   "resources": [
   {
      "urn:sna:rel:users": {
        "href":"/users",
        "hints": {
          "allow":["GET"],
          "representations":["application/json"],
          "accept-ranges":[],
          "accept-put":[],
          "accept-post":[],
          "precondition-req":[],
          "prefer":[]
        }
      }
    },
    {
      "urn:sna:rel:user": {
        "href-template": "/users/{user_id}",
        "href-vars": {
          "user_id":"http://upcoming/documentation/page"
        },
        "hints": {
          "allow":["GET"],
          "representations":["application/json"],
          "accept-ranges":[],
          "accept-put":[],
          "accept-post":[],
          "precondition-req":[],
          "prefer":[]
        }
      }
    }]
}

There’s obviously a lot more in there that we’ve specified in our DSL. That’s because our DSL assumes some defaults when they’re not configured. You can find the complete implementation of it here.

The client

That’s when I stopped working on the server part for a while and started working on a ruby based client. I’ve pointed this out in the intro of this blog series, but it’s worth repeating. Understanding the client side as well as possible and being able to provide guidance, especially in times of transition from a more tranditional HTTP API to a more Hypermedia based approach, is one of my main motivations behind this series. That’s why I’m building the client in parallel to the actual server part.

As usual the Ruby ecosystem was incredibly helpful to get a good start for the client implementation. The commandline client I’ve been building uses Thor for the CLI, Typhoeus as the HTTP library and the uri_template gem for building up URIs from templates. The rest is more or less handrolled at the moment.

The client side gemsGithub
1
2
3
4
5
gem 'typhoeus'
gem 'thor'
gem 'ethon'
gem 'multi_json'
gem 'uri_template'

Some words upfront, before we take a look at the implementation. I didn’t intend to build a browser-like client for the server in this iteration. My goal at this stage is merely understanding how the integration of a home document and the constraint of not building URIs themselves impacts the client. And how we can solve it that the impact on the rest of the client code isn’t that high.

So let’s drill down into the current client design. Let’s start at the API the client code (in our case the thor CLI interacts with). It looks like this.

Our minimal client APIGithub
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
require 'social_network_client/request_dispatcher'
require 'social_network_client/requests'
require 'social_network_client/middlewares'

module SocialNetworkClient

  class Api
    def initialize(base_uri, options)
      @dispatcher = RequestDispatcher.new(base_uri, Middlewares::Stack.for(options))
    end

    def users
      @dispatcher.dispatch(Requests::UsersRequest.new)
    end

    def user(user_id)
      @dispatcher.dispatch(Requests::UserRequest.new(user_id))
    end
  end

end

What is maybe interesting here, is that I took a design approach somewhat different to what I see in the typical client libraries out there. The API class is merely a wrapper around various invokations to a RequestDispatcher instance. The RequestDispatcher is the only place in the code that holds the base URI of the API (or in other words the endpoint where we can find our home document). All other URIs are specified relative to the entry point URI.

The dispatcher is configured with a stack of middlewares. Anyone who has done some Rails/Rack development probably already knows what it does, it’s the same deal. We’re going to take a closer look at this later in the post, because I consider it an interesting realization I had during the development.

What we also see here is that the requests to the server are explicitly modeled. So what do they do?

The UsersRequest classGithub
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
require 'social_network_client/relations'
require 'social_network_client/requests/base'

module SocialNetworkClient
  module Requests

    class UsersRequest < Base
      def prepare(home_document)
        #Relation::Users == 'urn:sna:rel:users'
        @target_uri = home_document.href(Relations::Users)
        self
      end

      def run(http, options)
        http.get(@target_uri, options)
      end
    end

  end
end

A request encapsulates the logic to build up a request from the home document. It’s got two methods that’re invoked by the dispatcher during request execution. The first one is prepare, in which the request class finds the target url for its relation. Remember the only thing the client is supposed to know about the URI scheme is the relation identifier. After obtaining the request it simply requests the target URI. The other request is probably a bit more interesting.

The UserRequest classGithub
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
require 'social_network_client/relations'
require 'social_network_client/requests/base'

module SocialNetworkClient
  module Requests

    class UserRequest < Base
      def initialize(user_id)
        @user_id = user_id
      end

      def identifier
        super + "##{@user_id}"
      end

      def prepare(home_document)
        @target_uri = home_document.href(Relations::User, user_id: @user_id)
        self
      end

      def run(http, options)
        http.get(@target_uri, options)
      end
    end

  end
end

It looks similar except three things. On the one hand it’s parametrized with a user_id. Which is also used as some kind of identfier in the method with that exact name. Even more interesting is the interaction with the home_document in the prepare method. This is the place where the target URI is constructed from the URI template contained in the home document.

As you might have guessed the home_document we’re dealing with here isn’t a deserialized JSON response, it’s a proxy wrapped around it. This class encapsulates the logic of building up the target URIs from the home document received from the server. Its implementation looks like this:

HomeDocumentProxy and the RelationProxyGithub
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
require 'uri_template'

module SocialNetworkClient

  class HomeDocumentProxy
    attr_reader :base_uri

    def initialize(base_uri, home_document)
      @base_uri = base_uri
      @home_document = home_document
    end

    def rel(rel_name)
      relation = @home_document['resources'].detect { |r| r[rel_name] }[rel_name]
      RelationProxy.new(self, relation)
    end

    def href(rel_name, options = nil)
      rel(rel_name).href(options)
    end
  end

  class RelationProxy
    def initialize(document, relation)
      @document = document
      @relation = relation
    end

    def href(options = nil)
      if options
        expand_path expand_template(options)
      else
        expand_path @relation['href']
      end
    end

    def expand_path(path)
      "#{@document.base_uri}#{path}"
    end

    def expand_template(options)
      URITemplate.new(@relation['href-template']).expand(options)
    end
  end

end

With this little piece falling into place, we can finally take a look at how the dispatcher is implemented who orchestrates the interaction between the home document and the requests.

The RequestDispatcherGithub
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
require 'typhoeus'
require 'base64'
require 'json'
require 'forwardable'
require 'social_network_client/home_document_proxy'
require 'social_network_client/failed_api_request'
require 'social_network_client/requests/home_document_request'

module SocialNetworkClient

  class RequestDispatcher
    attr_reader :base_uri

    extend Forwardable
    def_delegators :Typhoeus, :get, :post, :put, :delete

    def initialize(base_uri, middleware)
      @base_uri = base_uri
      @middleware = middleware
    end

    def dispatch(request)
      run request.prepare(home_document)
    end

    def run(request)
      response = @middleware.run(self, request, {})
      raise FailedApiRequest.new(response) unless response.success?
      JSON.parse(response.response_body)
    end

    def home_document
      HomeDocumentProxy.new @base_uri, run(Requests::HomeDocumentRequest.new)
    end

  end
end

The RequestDispatcher is actually pretty forward and (at least on the surface) highly inefficient. At runtime it holds the configured base URI of the API and the reference to the actual HTTP library that is being used. When a request is dispatched, it requests the home document from the server before performing the actual request. Yes, everytime a server request is made the home document is also requested. I can already hear my colleagues from the mobile team staring at me and telling me ‘Hangover’ like: “Yeah, that’s not going to happen!”.

But it’s not bad as it looks. That’s were the middleware stack comes into play.

The middleware stackGithub
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
Dir["#{File.dirname(__FILE__)}/middlewares/*.rb"].sort.each do |path|
  require "social_network_client/middlewares/#{File.basename(path, '.rb')}"
end
require 'social_network_client/cache'

module SocialNetworkClient
  module Middlewares
    class Stack
      def self.for(options)
        Defaults.new(
          Authenticator.new(
            ConditionalGet.new(Delegator.new, Cache.new),
            options[:user],
            options[:password]))
      end
    end
  end
end

It certainly doesn’t look so polished like a Rails middleware stack, but it does the same. It keeps the nitty gritty details of authentication, defaulting accept headers and most importantly caching away from the actual client code (it’s still reachable from the requests though in case it’s needed). I’ll spare you the details of the actual middlewares. You can take a look at them here. We’re only going to take a look at one of them, but before we do, we take a slight detour to the server.

Back to the server app

One part that we as developers often overlook when talking about communication patterns between different involved parties is that certain technologies are optimized for certain ways of communication. In our case that means that the HTTP infrastructure provides all the means to deal with our chatty communication. We just need to use them. You don’t need to build a highly sophisticated chaching solution yourself, HTTP has already a scheme for this. So let’s add caching instructions to our server side.

Adding caching instructions to the server sideGithub
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
module Api

  class HomeDocumentController < ApiController
    def show
      @home_document = HomeDocument
      expires_in 1.hour, public: false
      if stale?(etag: @home_document)
        @home_document
      end
    end
  end

  class UsersController < ApiController

    def index
      users = User.all
      if stale?(etag: users)
        render json: users
      end
    end

    def show
      user = User.find(params[:id])
      if stale?(etag: user)
        render json: user
      end
    end

  end
end

So what did we add here? For the controller that serves the home document we added support for caching via expiration and/or via validation (Conditional GET support). A full discussion about HTTP caching is naturally out of scope for this post. The short story is that every response send by the server will now contain some HTTP caching information that can be used to make the communication more efficient, either by the client itself or some HTTP intermediaries (like caches).

For instance the ‘expires_in’ in the HomeDocumentController tells clients that it’s safe to cache the response in their local private cache for an hour. It’s probably safe to say that the home document doesn’t change frequently (if we leave custom tailored home documents for authenticated users out for the moment). The ‘stale?’ method adds Conditional GET support to a controller.

Every response will now contain an ‘ETag’ header. This is typically some kind of checksum or calculated hash value that can be used to detect a change in the HTTP response body. When supplied with a request via the ‘If-Non-Match’ header, the server can now decide whether he really needs to send the response body back to the client or simply answers with a 304 response code (ala ‘Yo dawg, nothing new for you on my side’), effectively saving bandwidth.

Those two caching techiques together can become a pretty powerful tool to make the chatty nature of a Hypermedia API less burdensome while still retaining its benefits.

Back to the client

So how can we leverage caching on the client side? It would’ve been awesome if the HTTP client library provided this out of the box, but unfortunately most of the ones (including the one I’ve been using for the sample app so far) don’t implement some sort of caching support). That’s where the middleware in the client came handy.

ETag caching in the clientGithub
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
module SocialNetworkClient
  module Middlewares
    class ConditionalGet
      def initialize(inner, cache)
        @inner = inner
        @cache = cache
      end

      def run(http, request, options)
        if entry = cache?(request)
          options[:headers].merge!('If-None-Match' => entry[:etag])
        end

        response = @inner.run(http, request, options)

        if response.modified?
          cache! request, response
        else
          response = entry[:response]
        end

        response
      end

      def cache?(request)
        @cache.resolve(request.identifier)
      end

      def cache!(request, response)
        @cache.store(
          request.identifier,
          etag: response.headers_hash['ETag'],
          response: response)
      end
    end
  end
end

That piece of code should explain why there’s something like an identifier on a request instance. It’s used as a lookup key for caching. Plugging this middleware into the stack will check whether the related resource has been accessed previously. If so it sends the ‘ETag’ with the request. The server either answers with a 304 and no content (then the content from the local cache can be used) or it answers with a new payload and the local cache needs to be updated.

Conclusions

Well, Hypermedia wise we haven’t implemented much yet. There’s some things I learned while implementing the current state though. Most importantly that separating out an HTTP middleware stack from the other communication related code in the client can have a huge impact on the code itself. I really like how the code lays itself out at the moment.

It’s probably a bit overengineered, but I like the direction to which this is going. HTTP provides such a rich set of caching capabilities and also a lot of great debugging tools for the communication. If that is the case, why’re we still building local (even relational) caches that neglect the nature of HTTP?

The interesting question though is why (at least in the Ruby space) does none of the HTTP client libraries implement HTTP caching? Any thoughts on this?

One interesting idea (or so it seems) I would like to pursuit until my next post is whether we can use the Rack infrastructure especially Rack::Cache on the client side. A quick googling suggested that Faraday might give me the integration between Typhoeus and Rack::Cache I’m looking for. If all goes well I should be able to get rid of my naive middleware stack implementation. We’ll see …

See you next time around!


Fighting coupling, let the games begin!

I had almost forgotten what a writers block feels like over the past 1 1/2 years. Unbelievable how hard it sometimes is for me to put thought into words that I feel comfortable with. Or maybe I’m just hypercritical with myself. I don’t know. Whatever the reason, this is my 5th attempt to write this goddamn blogpost and this time I intend to finish it.

Last time we ended up with my realization that a typical ‘RPC over HTTP’ API can have some serious downsides when used for a public facing API, namely tight coupling between the client and server side. Coupling which will likely result in an overall hard to change and hard to evolve distributed system in the long run. The question is, what can we do against that? How can we reach a state were the server and client can evolve independently without stepping on each others toes?

Judging from what I’ve learned so far from the Hypermedia guys, there are solutions to this problem. Some of them may not sound super intuitive at first glance, especially in contrast to the wellknown RPC approach. But be warned, they definitely require changed behavior in clients. Without a change in how clients work with an API the whole idea falls apart. There are certainly different shades of implications for the client, depending on how conform a client wants to be with the new rules, but he has to change nevertheless. If you disagree with that, save your time. You can stop reading this post!

So what is our battleplan? How can we turn our RPC approach into something with less coupling? The short answer is: By systematically reducing the amount of necessary out-of-band information.

I can almost hear you shouting out ‘What is that supposed to mean?’. ‘out-of-band information’ is that kind of information which isn’t present in the API itself, but is necessary to work with the API. It’s a synonym for ‘external documentation’ or in other words nearly every question you answer politely with ‘RTFM’. Coincidently the stuff that’s typically hardcoded in clients: URLs, data structures, parameters, response codes and relationships between API calls. We need to make this information available to a client as part of the API.

In the best possible scenario all a client should need to know is the root url of the API. Everything else should be available via the API itself. Ok, I’m getting ahead of myself here. Let’s start a bit smaller. How could we enhance a pure HTTP RPC API to reduce coupling?

We’re going to start by making larger parts of the documentation available as a dedicated resource at the root URL of the API, a home document.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
GET / HTTP/1.1
Host: example.org
Accept: application/json-home

HTTP/1.1 200 OK
Content-Type: application/json-home
Cache-Control: max-age=3600
Connection: close

{
 "resources": {
   "http://example.org/rel/widgets": {
     "href": "/widgets/"
   },
   "http://example.org/rel/widget": {
     "href-template": "/widgets/{widget_id}",
     "href-vars": {
       "widget_id": "http://example.org/param/widget"
     },
     "hints": {
       "allow": ["GET", "PUT", "DELETE", "PATCH"],
       "representations": ["application/json"],
       "accept-patch": ["application/json-patch"],
       "accept-put": ["application/json"]
       "status": "obsolete"
     }
   }
 }
}

The request for the root URL of our API would return a heavily cachable sitemap for the API, containing detailed information about each resource in the API. If you’re wondering what those keys in the resources JSON object are, they’re link relations as described in RFC5988. In my own words I would describe them as global identifiers for resources. They play an important role in the greater scheme.

We talked briefly about changed rules for a client, this is one of them. If a client wants to relax his coupling to the API, he shouldn’t bind to URLS directly and bind to link relations instead. Using a link relation he is able to obtain the information needed to construct a request from the home document.

Well that sounds like a good place to stop for today. Next time we dive into some code and take a look at how such a home document might feel in real life. If you’re curious, I’ve already implemented the server part for this.


My pain with a non Hypermedia HTTP API

Last post I promised to talk about the WHY I’m interested in hypermedia in more detail. So let’s start right from the beginning.

9 Month ago I joined the API team of XING. If you weren’t aware that XING has an API, no biggie. That’s likely because at the time of writing, we’re still in a ’Closed beta’. Although beta implies a brand new product, the API we’re currently working on is actually a mature product which has been in place for several years now, though not available to the public. Basically if you’re using one of our native mobile apps, you’ve been consuming our API for quite a while now.

You might ask yourself why we’re still in beta then. Well, that’s a totally different story and, forgive me, one I can’t elaborate on.

What matters for this post is that our current API design follows a similar approach to the ones known from Facebook, Twitter or GitHub, which is basically Remote Procedure Call (RPC) over HTTP with extensive out of band documentation. That’s what all the large API providers seem to do.

Pretty much everyone knows how to use such an API. Request in, response out, containing what looks like a serialized version of an object graph. If you don’t understand the API, no problem at all, you can always look up the official documentation of the particular API call. Doesn’t look like a complex problem, right? Well, appearances can be deceiving …

At least I thought it wasn’t complex. You know, I have a special nickname for API development. Having a nickname for something like that is probably already telling a lot. The good thing about it is, that it constantly reminds me of my false assumptions when I came to API development. I call API development my personal Vietnam of Agility.

The approach to building software I mostly used prior to joining my current team was mostly all about starting small, learn and adapt to a better solution. I used to build my software of loosely coupled parts, that made it easy to correct mistakes or false assumptions afterwards and it’s exactly that flexiblity or loose coupling that I currently miss most.

Do you remember the movie The Devil’s Advocate with Keanu Reeves and Al Pacino? If I recall correctly, there’s a scene in that movie where the main actor (played by Reeves) signs a deal with his new employer (played by Pacino) and Pacino has this deceitful smile on his face, as if he knew something his opposite didn’t. If you don’t know the movie, I won’t spoil the ending for you. Let’s just say, both sides weren’t happy with the outcome of the contract in the end, though surely it looked benefitial for both parties when they signed it.

So what has this to do with public-facing RPC APIs over HTTP? As an API provider your public interface is a contract, a contract between you and your clients. What you basically say when you present such an API could be summed up like this

Read this out loud, for a more dramatic effect

I’ll make sure that there won’t be any breaking structural changes. I’m going to ensure that the URI scheme doesn’t change either. If I’ve no other options than to break things, it’s my responsibility to find a way that doesn’t impact you much. You on the other hand, need to figure out how those API calls relate to each other. There’s a wonderful part in the contract (the API documentation) which outlines how to do it. Go and read the f***ing manual! Of course, in order to use the API calls correctly, you have to replicate parts of our business logic for determining which calls can be used in what context.

Wow, that sounds bad that way, doesn’t it? And it doesn’t even sound like it’s all positive for either side. In object-orientation we have a nifty word for the outcome of such a contract, which is tight coupling. Does this ring any alarm bells? Yeah, that’s the trademark of monolithic systems. The kind of system that can only be changed with a lot of cost, effort and pain. I have to admit that this realization took me quite by surprise, but it seems so obvious once you think about it.

And that’s where I think hypermedia can help. It seems to change the game plan between an API provider and its consumers in a way that naturally enforces loose coupling.

Next time we’ll continue this series with a more detailed look on the various forms of coupling in an HTTP based RPC API.


Join me on a journey

In the last couple of months I’ve been reading, listening to and watching everything I could find or buy about REST and Hypermedia. I think I’ve now come to a point where the only logical continuation of this learning process is to try to apply what I’ve learned so far to an actual problem.

It’s time to stop talking theoretically. I really badly want to see a ‘real world’ example of those concepts in play. I want to experience the impact of such a design first hand, both on the server and on the client side as well.

I’ve been carrying this idea with me for several days now and finally decided to give in.

So here’s what I would like to do:

  • I’m going to try myself on designing a Hypermedia API for a social network from scratch. This will include the whole state machine / sitemap design for a small subset of the typical features of a social network. I’m thinking about profiles, contacts and private messages for a start, but that’s not set.

  • I will document the whole process of evaluating existing media types vs. the creation of a new vendor specific one for the API.

  • I’m going to implement the API iteratively, starting with an MVP and then adding more and more features as we go along, documenting every step on the way here on this blog.

  • I will throw in some nasty real world requirements I’ve encountered first hand while working on one of the larger HTTP APIs available in Germany.

  • Last but not least, I want to also explore what Hypermedia means for the client side and this both in a statically and in a dynamically typed language.

You might ask yourself what my motivation behind all this is and I promise you, I will elaborate on that in detail, but not in this post. For the moment let’s just say, I currently tend to think Hypermedia would make parts of my professional life easier. Not sure how much this is going to be in the end, but hey, let’s just find out!

Please note the following: I don’t consider myself an expert on this topic. This is a learning exercise for me as well. I’m very likely to stumble on my way. If I do, I thought my chances for getting up again and continuing this journey would improve a lot with outside feedback and help.

That’s the main reason why I’m doing this out in the open. The source code repository for this can be found here. Feel free to watch it, fork it and/or to contribute. I don’t know how long this all will take me and I can’t guarantee you a regular schedule for updates, but I’m committed to make this work. Hopefully we’ll see this page getting updated frequently with new content.

So, what do you think? Are you in?

-Bjoern

Table of contents

  1. My pain with a non Hypermedia HTTP API
  2. Fighting coupling, let the games begin!
  3. HTTP + client side middleware stack == Win
  4. Oh my Faraday


Machine.Fakes needs your help

If you ask me about my personal highlights in 2011 (technology wise), I would probably name Machine.Fakes as one of those. Writing and using it has been such a tremendous amount of fun and was the source of getting to know lots of interesting people.

As technology is the battlefield of opinionated minds, not every feedback I received was pleasant (especially from the Belgian “what’s wrong with good old unit test” brigade), but overall I’m very pleased with the perception and the feedback for the framework.

As you may have noticed, I recently made a big technology switch. Since the start of the year I spend most of the time working with Ruby on Rails on OSX. Originally I thought Mono would give me a great opportunity to continue using Machine.Fakes, but as it turns out I currently just don’t have the time to do that. Maintaining a tool/framework without actually using it on a daily basis feels just wrong. In retrospective I think I must have anticipated something like this last year, when Alexander Gross and I moved Machine.Fakes into the Machine organization on Github.

The worst future for Machine.Fakes I can imagine, is that it becomes just another abandoned .NET open source framework, like all the dead ones on Codeplex. So this is my call for help:

If you’re using Machine.Fakes and are willing to take over as the active maintainer, please drop me a line. I will assist you as good as I can in order to make this happen!


At the crossroads

512 days or 17 months or 1.5 years. That’s the time between todays post and my last one. Not as long as the Duke Nukem Forever development time, but it still strikes me how much time has gone by since I’ve written something down here.

Today is probably going to be a very personal post and for that matter also non-technical one.

Read on →