re: engineering management
notes on software development, leading teams and changing the way we work

API Design is Interaction Design

For the past few weeks, I have been working with my team on refining the REST API for our Web service. While there will be some new additions in order to meet requirements for applications that will build upon the API, many of the changes are refinements. Early adopters and our own experience has provided a treasure trove of information that has guided the development of the latest version.

From the very beginning, I chose a REST Oriented Architecture (ROA) for the service. To me, this just seemed obvious. I’ve followed the development of XML-RPC and SOAP for years. For awhile, XML-RPC seemed pretty cool and simple to me especially when compared to the mess that is SOAP (which evolved into WS-Death Star). However, in practice I always ended up sending some simple, Plain Ol’ XML over HTTP. For the majority of cases, HTTP and URIs covered the functionality I needed sufficiently.

As REST became a more prevalent meme, I got a copy of RESTful Web Services and read it cover-to-cover. I recommend this book to ANYONE building Web applications. It put what was common practice for me into context and really illustrated what REST and ROA are really about and what tradeoffs need to be made in every application.

One part of the book that really stood out for me were the sections on resource design. Instead of the highly procedural XML-RPC or my own custom-coded XML schemas in my POX/HTTP implementations, REST made me think more about the actual objects, or resources, that I wanted to interact with over the network. Doing this forced me to think harder about naming and behaviors (whether or not they mapped cleanly to GET, PUT, HEAD, DELETE and, of course, POST). For error conditions, there was almost always an HTTP status code that would work for us. And, HTTP headers were incredibly useful in dealing with metadata.

So, on my first pass with our API I attempted to adhere to RESTful principles. After identifying my resources, I spent even more time thinking about representations, the content types and schemas for the data my service would consume and produce. Many of my early assumptions changed quickly as I worked through this process (and again as we started working on the latest version) For instance, I once considered XML a safe, de facto format to support in an API. Then, along came JSON. When, I looked at some of my earlier representations I saw almost an order or magnitude difference in the size of an XML response compared to its JSON counterpart for exactly the same data! Now, in many cases, XML is a requirement. However, if you have the ability to define exactly which representations you will support, think long and hard about when to use XML. Given the proliferation of libraries in almost every conceivable language, JSON is almost as de facto a choice as XML and has even more practical benefits although there are also downsides, like potential security vulnerabilities. As in everything else, one should pick the tools that help solve the problem at hand.

The first versions of our API worked fine. The true test was seeing the number of people who just went to our documentation page and started coding against it. Generally, we received fewer than half a dozen questions via email before they were able to productively use the service. That’s a huge win! We also ate our own dog food by prototyping clients for the service using:

  • curl
  • Ruby (using Merb as our framework)
  • Flash (ActionScript 3 and the Flex framework)
  • Adobe AIR

All the clients worked well, with the minor exception of Flash. Unfortunately, Flash has some limitations with respect to things like:

  • HTTP response codes other than 200
  • PUT and DELETE
  • Custom HTTP headers
  • Using the User-Agent to identify the Flash Player

Surprisingly, Adobe AIR works just fine with our REST API. Here, we had a nicely defined API but needed to make it work with Flash. The thought of either creating a separate API or moving to a least common delimiter approach for everyone was not appealing. However, there was actually a simple solution. If we added a prefix to all our API endpoints, we would know that the client was the Flash Player. In our Web service code, we wrote a proxy that would extract input from the request and then call the REST API directly. When the API code generated a response, the proxy would repackage it appropriately for the player:

  • Always generate a 200 response
  • In the response body, use an XML document to house:
    • The actual HTTP response code
    • Standard and custom headers
    • Response body from the REST API

The beauty of this approach is that it is fairly generic. As we add new endpoints, the proxy does not need to be changed. The Web frameworks router just tries to match everything after the prefix against our existing RESTful routes. If we have a match, then the proxy calls that API. Otherwise, we return an error. Simple. So, how is this interaction design? Well, developers are users, right? For the most part, we try every new API by using curl or these days, rest-client. If the API seems too cumbersome using either of these tools, then we’ve probably done something really wrong. And, we did. There were a few APIs that were a little unwieldy or confusing due to the naming, so we focused on them a lot for the latest version. Deprecation can be a good thing.

As developers, we use tons of APIs from various sources. Some are good, some not so good. Alex Payne, the API Lead at Twitter gave a great presentation the interaction design of APIs. I almost didn’t write this blog post because Alex’s presentation was so spot on. But, I think it is a topic that may not get the attention it deserves so I’ll throw my hat in the ring. In our API, we leverage an internal user authentication service as opposed to rolling our own. This service had an “XML API” that was essentially a Plain Ol’ XML over HTTP endpoint that called into their Java code. Not really RESTful, but generally workable. The problem was that their XML API did not have feature parity with their Java and ActionScript APIs. We were told that this would be addressed with the next version of the API and there would be a new “REST API”. Great!

Well, maybe not so great. The user authentication service was growing to meet the needs of multiple clients and, like us, did not want to rewrite their code to address their non-primary clients. Their approach was to use a proxy approach as well that generated interfaces from their Java class and method definitions. See where this is going yet?

The XML generated from the class and method definitions was approximately an order of magnitude larger than the previous XML. Sure, we got feature parity but good luck trying to figure out how to USE the API. I had to have at least 5 browser tabs open and look through JavaDocs to figure out how to create an account! I know, I know… once we get it done it’s done…or is it? What happens when they change method names or parameter lists? Alex defines “The Humane API” as:

  • Explorable
  • Predictable
  • Consistent

This user authentication API is not humane currently. I suspect that this is rooted in its implementation. It is a Java-based Web service and there are tons of tools and APIs that can turn Java objects into Web services. Unfortunately, the services generated are best consumed by other Java applications and services. My team uses ScrumWorks Pro for our agile planning and task tracking. It too has one of these Java-generated Web service APIs which I just find inscrutable. I’ve seen similar things in the .NET world as well. On the Web, Web service clients can be as simple as curl or as complex as a desktop Windows or Mac application like iTunes. Every programming language/tool has its own idioms, but they all have one thing in common: they likely have one or more libraries for communicating via HTTP.

And that is the interaction design advice I’d like to impart. Recently, there have been a proliferation of projects that help you build applications and services with nothing more than HTTP, Javascript/JSON and general RESTful principles. CloudKit bills itself as an Open Web JSON Appliance. Once you’ve downloaded and installed the Ruby gem, you can expose a resource and immediately interact with it using curl. Awesome! And then there is CouchDB, which is getting a ton of attention and for good reason. Jacob Kaplan-Moss, one of the creators of Django, said this of CouchDB in a blog post called Of the Web:

Let me tell you something: Django may be built for the Web, but CouchDB is built of the Web. I’ve never seen software that so completely embraces the philosophies behind HTTP. CouchDB makes Django look old-school in the same way that Django makes ASP look outdated. Let me try to explain what I’m talking about. You want an API without having to write a line of code? It’s called curl, and it ships with your MacBook. And just look at how simple the APIs are in your favorite language.

If you are building a Web application or service, strive to make it Of the Web.