Writing an API Wrapper in Ruby with TDD

Sooner or later, all developers are required to interact with an API. The most difficult part is always related to reliably testing the code we write, and, as we want to make sure that everything works properly, we continuosly run code that queries the API itself. This process is slow and inefficient, as we can experience network issues and data inconsistencies (the API results may change). Let’s review how we can avoid all of this effort with Ruby.

Our Goal

"Flow is essential: write the tests, run them and see them fail, then write the minimal implementation code to make them pass. Once they all do, refactor if needed."

Our goal is simple: write a small wrapper around the Dribbble API to retrieve information about a user (called ‘player’ in the Dribbble world).
As we will be using Ruby, we will also follow a TDD approach: if you’re not familiar with this technique, Nettuts+ has a good primer on RSpec you can read. In a nutshell, we will write tests before writing our code implementation, making it easier to spot bugs and to achieve a high code quality. Flow is essential: write the tests, run them and see them fail, then write the minimal implementation code to make them pass. Once they all do, refactor if needed.

The API

The Dribbble API is fairly straightforward. At the time of this it supports only GET requests and doesn’t require authentication: an ideal candidate for our tutorial. Moreover, it offers a 60 calls per minute limit, a restriction that perfectly shows why working with APIs require a smart approach.

Key Concepts

This tutorial needs to assume that you have some familiarity with testing concepts: fixtures, mocks, expectations. Testing is an important topic (especially in the Ruby community) and even if you are not a Rubyist, I’d encourage you to dig deeper into the matter and to search for equivalent tools for your everyday language. You may want to read “The RSpec book” by David Chelimsky et al., an excellent primer on Behavior Driven Development.

To summarize here, here are three key concepts you must know:

Mock: also called double, a mock is “an object that stands in for another object in an example”. This means that if we want to test the interaction between an object and another, we can mock the second one. In this tutorial, we will mock the Dribbble API, as to test our code we don’t need the API, itself, but something that behaves like it and exposes the same interface.
Fixture: a dataset that recreates a specific state in the system. A fixture can be used to create the needed data to test a piece of logic.
Expectation: a test example written the from the point of view of the result we want to achieve.

Our Tools

"As a general practice, run tests every time you update them."

WebMock is a Ruby mocking library that is used to mock (or stub) http requests. In other words, it allows you to simulate any HTTP request without actually making one. The primary advantage to this is being able to develop and test against any HTTP service without needing the service itself and without incurring in related issues (like API limits, IP restrictions and such).
VCR is a complementary tool that records any real http request and creates a fixture, a file that contains all the needed data to replicate that request without performing it again. We will configure it to use WebMock to do that. In other words, our tests will interact with the real Dribbble API just once: after that, WebMock will stub all the requests thanks to the data recorded by VCR. We will have a perfect replica of the Dribbble API responses recorded locally. In addition, WebMock will let us test edge cases (like the request timing out) easily and consistently. A wonderful consequence of our setup is that everything will be extremely fast.

As for unit testing, we will be using Minitest. It’s a fast and simple unit testing library that also supports expectations in the RSpec fashion. It offers a smaller feature set, but I find that this actually encourages and pushes you to separate your logic into small, testable methods. Minitest is part of Ruby 1.9, so if you’re using it (I hope so) you don’t need to install it. On Ruby 1.8, it’s only a matter of gem install minitest.

I will be using Ruby 1.9.3: if you don’t, you will probably encounter some issues related to require_relative, but I've included fallback code in a comment right below it. As a general practice, you should run tests every time you update them, even if I won’t be mentioning this step explicitly throughout the tutorial.

Setup

We will use the conventional /lib and /spec folder structure to organize our code. As for the name of our library, we’ll call it Dish, following the Dribbble convention of using basketball related terms.

The Gemfile will contain all our dependencies, albeit they’re quite small.

source :rubygems

gem 'httparty'

group :test do
  gem 'webmock'
  gem 'vcr'
  gem 'turn'
  gem 'rake'
end

Httparty is an easy to use gem to handle HTTP requests; it will be the core of our library. In the test group, we will also add Turn to change the output of our tests to be more descriptive and to support color.

The /lib and /spec folders have a symmetrical structure: for every file contained in the /lib/dish folder, there should be a file inside /spec/dish with the same name and the ‘_spec’ suffix.

Let’s start by creating a /lib/dish.rb file and add the following code:

require "httparty"
Dir[File.dirname(__FILE__) + '/dish/*.rb'].each do |file|
  require file
end

It doesn’t do much: it requires ‘httparty’ and then iterates over every .rb file inside /lib/dish to require it. With this file in place, we will be able to add any functionality inside separate files in /lib/dish and have it automatically loaded just by requiring this single file.

Let’s move to the /spec folder. Here’s the content of the spec_helper.rb file.

#we need the actual library file
require_relative '../lib/dish'
# For Ruby < 1.9.3, use this instead of require_relative
# require(File.expand_path('../../lib/dish', __FILE__))

#dependencies
require 'minitest/autorun'
require 'webmock/minitest'
require 'vcr'
require 'turn'

Turn.config do |c|
 # :outline  - turn's original case/test outline mode [default]
 c.format  = :outline
 # turn on invoke/execute tracing, enable full backtrace
 c.trace   = true
 # use humanized test names (works only with :outline format)
 c.natural = true
end

#VCR config
VCR.config do |c|
  c.cassette_library_dir = 'spec/fixtures/dish_cassettes'
  c.stub_with :webmock
end

There's quite a few things here worth noting, so let’s break it piece by piece:

At first, we require the main lib file for our app, making the code we want to test available to the test suite. The require_relative statement is a Ruby 1.9.3 addition.
We then require all the library dependencies: minitest/autorun includes all the expectations we will be using, webmock/minitest adds the needed bindings between the two libraries, while vcr and turn are pretty self-explanatory.
The Turn config block merely needs to tweak our test output. We will use the outline format, where we can see the description of our specs.
The VCR config blocks tells VCR to store the requests into a fixture folder (note the relative path) and to use WebMock as a stubbing library (VCR supports some other ones).

Last, but not least, the Rakefile that contains some support code:

require 'rake/testtask'

Rake::TestTask.new do |t|
  t.test_files = FileList['spec/lib/dish/*_spec.rb']
  t.verbose = true
end

task :default => :test

The rake/testtask library includes a TestTask class that is useful to set the location of our test files. From now on, to run our specs, we will only type rake from the library root directory.

As a way to test our configuration, let’s add the following code to /lib/dish/player.rb:

module Dish
  class Player
  end
end

Then /spec/lib/dish/player_spec.rb:

require_relative '../../spec_helper'
# For Ruby < 1.9.3, use this instead of require_relative
# require (File.expand_path('./../../../spec_helper', __FILE__))

describe Dish::Player do

  it "must work" do
    "Yay!".must_be_instance_of String
  end

end

Running rake should give you one test passing and no errors. This test is by no means useful for our project, yet it implicitly verifies that our library file structure is in place (the describe block would throw an error if the Dish::Player module was not loaded).

First Specs

To work properly, Dish requires the Httparty modules and the correct base_uri, i.e. the base url of the Dribbble API. Let’s write the relevant tests for these requirements in player_spec.rb:

...
describe Dish::Player do

  describe "default attributes" do

    it "must include httparty methods" do
      Dish::Player.must_include HTTParty
    end

    it "must have the base url set to the Dribble API endpoint" do
      Dish::Player.base_uri.must_equal 'http://api.dribbble.com'
    end

  end

end

As you can see, Minitest expectations are self-explanatory, especially if you are an RSpec user: the biggest difference is wording, where Minitest prefers “must/wont” to “should/should_not”.

Running these tests will show one error and one failure. To have them pass, let’s add our first lines of implementation code to player.rb:

module Dish

  class Player

    include HTTParty

    base_uri 'http://api.dribbble.com'

  end

end

Running rake again should show the two specs passing. Now our Player class has access to all Httparty class methods, like get or post.

Recording our First Request

As we will be working on the Player class, we will need to have API data for a player. The Dribbble API documentation page shows that the endpoint to get data about a specific player is http://api.dribbble.com/players/:id

As in typical Rails fashion, :id is either the id or the username of a specific player. We will be using simplebits, the username of Dan Cederholm, one of the Dribbble founders.

To record the request with VCR, let’s update our player_spec.rb file by adding the following describe block to the spec, right after the first one:

  ...

  describe "GET profile" do

  before do
    VCR.insert_cassette 'player', :record => :new_episodes
  end

  after do
    VCR.eject_cassette
  end

  it "records the fixture" do
    Dish::Player.get('/players/simplebits')
  end

  end

end

After running rake, you can verify that the fixture has been created. From now on, all our tests will be completely network independent.

The before block is used to execute a specific portion of code before every expectation: we use it to add the VCR macro used to record a fixture that we will call ‘player’. This will create a player.yml file under spec/fixtures/dish_cassettes. The :record option is set to record all new requests once and replay them on every subsequent, identical request. As a proof of concept, we can add a spec whose only aim is to record a fixture for simplebits’s profile. The after directive tells VCR to remove the cassette after the tests, making sure that everything is properly isolated. The get method on the Player class is made available, thanks to the inclusion of the Httparty module.

After running rake, you can verify that the fixture has been created. From now on, all our tests will be completely network independent.

Getting the Player Profile

Every Dribbble user has a profile that contains a pretty extensive amount of data. Let’s think about how we would like our library to be when actually used: this is a useful way to flesh out our DSL will work. Here’s what we want to achieve:

simplebits = Dish::Player.new('simplebits')
simplebits.profile
  => #returns a hash with all the data from the API
simplebits.username
  => 'simplebits'
simplebits.id
  => 1
simplebits.shots_count
  => 157

Simple and effective: we want to instantiate a Player by using its username and then get access to its data by calling methods on the instance that map to the attributes returned by the API. We need to be consistent with the API itself.

Let’s tackle one thing at a time and write some tests related to getting the player data from the API. We can modify our "GET profile" block to have:

describe "GET profile" do

  let(:player) { Dish::Player.new }

  before do
    VCR.insert_cassette 'player', :record => :new_episodes
  end

  after do
    VCR.eject_cassette
  end

  it "must have a profile method" do
    player.must_respond_to :profile
  end

  it "must parse the api response from JSON to Hash" do
    player.profile.must_be_instance_of Hash
  end

  it "must perform the request and get the data" do
    player.profile["username"].must_equal 'simplebits'
  end

end

The let directive at the top creates a Dish::Player instance available in the expectations. Next, we want to make sure that our player has got a profile method whose value is a hash representing the data from the API. As a last step, we test a sample key (the username) to make sure that we actually perform the request.

Note that we’re not yet handling how to set the username, as this is a further step. The minimal implementation required is the following:

...
class Player

  include HTTParty

  base_uri 'http://api.dribbble.com'

  def profile
    self.class.get '/players/simplebits'
  end

end
...

A very little amount of code: we’re just wrapping a get call in the profile method. We then pass the hardcoded path to retrieve simplebits’s data, data that we had already stored thanks to VCR.

All our tests should be passing.

Setting the Username

Now that we have a working profile function, we can take care of the username. Here are the relevant specs:

describe "default instance attributes" do

  let(:player) { Dish::Player.new('simplebits') }

  it "must have an id attribute" do
    player.must_respond_to :username
  end

  it "must have the right id" do
    player.username.must_equal 'simplebits'
  end

end

describe "GET profile" do

  let(:player) { Dish::Player.new('simplebits') }

  before do
    VCR.insert_cassette 'base', :record => :new_episodes
  end

  after do
    VCR.eject_cassette
  end

  it "must have a profile method" do
    player.must_respond_to :profile
  end

  it "must parse the api response from JSON to Hash" do
    player.profile.must_be_instance_of Hash
  end

  it "must get the right profile" do
    player.profile["username"].must_equal "simplebits"
  end

end

We’ve added a new describe block to check the username we’re going to add and simply amended the player initialization in the GET profile block to reflect the DSL we want to have. Running the specs now will reveal many errors, as our Player class doesn’t accept arguments when initialized (for now).

Implementation is very straightforward:

...
class Player

  attr_accessor :username

  include HTTParty

  base_uri 'http://api.dribbble.com'

  def initialize(username)
    self.username = username
  end

  def profile
    self.class.get "/players/#{self.username}"
  end

end
...

The initialize method gets a username that gets stored inside the class thanks to the attr_accessor method added above. We then change the profile method to interpolate the username attribute.

We should get all our tests passing once again.

Dynamic Attributes

At a basic level, our lib is in pretty good shape. As profile is a Hash, we could stop here and already use it by passing the key of the attribute we want to get the value for. Our goal, however, is to create an easy to use DSL that has a method for each attribute.

Let’s think about what we need to achieve. Let’s assume we have a player instance and stub how it would work:

player.username
  => 'simplebits'
player.shots_count
  => 157
player.foo_attribute
  => NoMethodError

Let’s translate this into specs and add them to the GET profile block:

...
describe "dynamic attributes" do

  before do
    player.profile
  end

  it "must return the attribute value if present in profile" do
    player.id.must_equal 1
  end

  it "must raise method missing if attribute is not present" do
    lambda { player.foo_attribute }.must_raise NoMethodError
  end

end
...

We already have a spec for username, so we don’t need to add another one. Note a few things:

we explicitly call player.profile in a before block, otherwise it will be nil when we try to get the attribute value.
to test that foo_attribute raises an exception, we need to wrap it in a lambda and check that it raises the expected error.
we test that id equals 1, as we know that that is the expected value (this is a purely data-dependent test).

Implementation-wise, we could define a series of methods to access the profile hash, yet this would create a lot of duplicated logic. Moreover, the would rely on the API result to always have the same keys.

"We will rely on method_missing to handle this cases and ‘generate’ all those methods on the fly."

Instead, we will rely on method_missing to handle this cases and ‘generate’ all those methods on the fly. But what does this mean? Without going into too much metaprogramming, we can simply say that every time we call a method not present on the object, Ruby raises a NoMethodError by using method_missing. By redefining this very method inside a class, we can modify its behaviour.

In our case, we will intercept the method_missing call, verify that the method name that has been called is a key in the profile hash and in case of positive result, return the hash value for that key. If not, we will call super to raise a standard NoMethodError: this is needed to make sure that our library behaves exactly the way any other library would do. In other words, we want to guarantee the least possible surprise.

Let’s add the following code to the Player class:

def method_missing(name, *args, &block)
  if profile.has_key?(name.to_s)
    profile[name.to_s]
  else
    super
  end
end

The code does exactly what described above. If you now run the specs, you should have them all pass. I’d encorage you to add some more to the spec files for some other attribute, like shots_count.

This implementation, however, is not really idiomatic Ruby. It works, but it can be streamlined into a ternary operator, a condensed form of an if-else conditional. It can be rewritten as:

def method_missing(name, *args, &block)
  profile.has_key?(name.to_s) ? profile[name.to_s] : super
end

It’s not just a matter of length, but also a matter of consistency and shared conventions between developers. Browsing source code of Ruby gems and libraries is a good way to get accustomed to these conventions.

Caching

As a final step, we want to make sure that our library is efficient. It should not make any more requests than needed and possibly cache data internally. Once again, let’s think about how we could use it:

player.profile
  => performs the request and returns a Hash
player.profile
  => returns the same hash
player.profile(true)
  => forces the reload of the http request and then returns the hash (with data changes if necessary)

How can we test this? We can by using WebMock to enable and disable network connections to the API endpoint. Even if we’re using VCR fixtures, WebMock can simulate a network Timeout or a different response to the server. In our case, we can test caching by getting the profile once and then disabling the network. By calling player.profile again we should see the same data, while by calling player.profile(true) we should get a Timeout::Error, as the library would try to connect to the (disabled) API endpoint.

Let’s add another block to the player_spec.rb file, right after dynamic attribute generation:

describe "caching" do

  # we use Webmock to disable the network connection after
  # fetching the profile
  before do
    player.profile
    stub_request(:any, /api.dribbble.com/).to_timeout
  end

  it "must cache the profile" do
    player.profile.must_be_instance_of Hash
  end

  it "must refresh the profile if forced" do
    lambda { player.profile(true) }.must_raise Timeout::Error
  end

end

The stub_request method intercepts all calls to the API endpoint and simulates a timeout, raising the expected Timeout::Error. As we did before, we test the presence of this error in a lambda.

Implementation can be tricky, so we’ll split it into two steps. Firstly, let’s move the actual http request to a private method:

...
def profile
  get_profile
end

...

private

def get_profile
  self.class.get("/players/#{self.username}")
end
...

This will not get our specs passing, as we’re not caching the result of get_profile. To do that, let’s change the profile method:

...
def profile
  @profile ||= get_profile
end
...

We will store the result hash into an instance variable. Also note the ||= operator, whose presence makes sure that get_profile is run only if @profile returns a falsy value (like nil).

Next we can add the forced reload directive:

...
def profile(force = false)
  force ? @profile = get_profile : @profile ||= get_profile
end
...

We’re using a ternary again: if force is false, we perform get_profile and cache it, if not, we use the logic written in the previous version of this method (i.e. performing the request only if we don’t have already an hash).

Our specs should be green now and this is also the end of our tutorial.

Wrapping Up

Our purpose in this tutorial was to write a small and efficient library to interact with the Dribbble API; we’ve laid the foundation for this to happen. Most of the logic we’ve written can be abstracted and reused to access all the other endpoints. Minitest, WebMock and VCR have proven to be valuable tools to help us shape our code.

We do, however, need to be aware of a small caveat: VCR can become a double-edged sword, as our tests can become too much data-dependent. If, for any reason, the API we’re building against changes without any visible sign (like a version number), we may risk having our tests perfectly working with a dataset, which is no longer relevant. In that case, removing and recreating the fixture is the best way to make sure that our code still works as expected.

HIGHLIGHTS OF THE DAY