Welcome to the second part of our series on generators and Koa. If you missed it you can read read part 1 here. Before starting with the development process, make sure that you have installed Node.js 0.11.9 or higher.

In this part, we will be creating a dictionary API using Koa.js, and you'll learn about routing, compressing, logging, rate-limiting, and error handling in Koa.js. We will also use Mongo as our datastore and learn briefly about importing data into Mongo and the ease that comes with querying in Koa. Finally, we'll look into debugging Koa apps.

Understanding Koa

Koa has radical changes built under its hood which leverage the generator goodness of ES6. Apart from the change in the control flow, Koa introduces its own custom objects, such as this, this.request, and this.response, which conveniently act as a syntactic-sugar layer built on top of Node's req and res objects, giving you access to various convenience methods and getters/setters. 

Apart from convenience, Koa also cleans up the middleware which, in Express, relied on ugly hacks which often modified core objects. It also provides better stream handling.

Wait, What's a Middleware?

A middleware is a pluggable function that adds or removes a particular piece of functionality by doing some work in the request/response objects in Node.js.

Koa's Middleware

A Koa middleware is essentially a generator function that returns one generator function and accepts another. Usually, an application has a series of middleware that are run for each request. 

Also, a middleware must yield to the next 'downstream' middleware if it is run by an 'upstream middleware'. We will discuss more about this in the error handling section.

Building Middleware

Just one last thing: To add a middleware to your Koa application, we use the koa.use() method and supply the middleware function as the argument. Example: app.use(koa-logger) adds koa-logger to the list of middleware that our application uses.

Building the Application

To start with the dictionary API, we need a working set of definitions. To recreate this real-life scenario, we decided to go with a real dataset. We took the definition dump from Wikipedia and loaded it into Mongo. The set consisted of about 700,000 words as we imported only the English dump. Each record (or document) consists of a word, its type, and its meaning. You can read more about the importing process in the import.txt file in the repository.

To move along the development process, clone the repository and check your progress by switching to different commits. To clone the repo, use the following command:

$ git clone https://github.com/bhanuc/dictapi.git

We can start by creating a base server Koa:

var koa = require('koa');
var app = koa();

app.use(function *(next){
    this.type = 'json';
    this.status = 200;
    this.body = {'Welcome': 'This is a level 2 Hello World Application!!'};
});

if (!module.parent) app.listen(3000);
console.log('Hello World is Running on http://localhost:3000/');

In the first line, we import Koa and save an instance in the app variable. Then we add a single middleware in line 5, which is an anonymous generator function that takes the next variable as a parameter. Here, we set the type and status code of the response, which is also automatically determined, but we can also set those manually. Then finally we set the body of the response. 

Since we have set the body in our first middleware, this will mark the end of each request cycle and no other middleware will be involved. Lastly, we start the server by calling its listen method and pass on the port number as a parameter.

We can start the server by running the script via:

$ npm install koa
$ node --harmony index.js

You can directly reach this stage by moving to commit 6858ae0:

$ git checkout 6858ae0

Adding Routing Capabilities

Routing allows us to redirect different requests to different functions on the basis of request type and URL. For example, we might want to respond to /login differently than signup. This can be done by adding a middleware, which manually checks the URL of the request received and runs corresponding functions. Or, instead of manually writing that middleware, we can use a community-made middleware, also known as a middleware module.

To add routing capability to our application, we will use a community module named koa-router. 

To use koa-router, we will modify the existing code to the code shown below:

var koa = require('koa');
var app = koa();
var router = require('koa-router');
var mount = require('koa-mount');

var handler = function *(next){
    this.type = 'json';
    this.status = 200;
    this.body = {'Welcome': 'This is a level 2 Hello World Application!!'};
};

var APIv1 = new router();
APIv1.get('/all', handler);

app.use(mount('/v1', APIv1.middleware()));
if (!module.parent) app.listen(3000);
console.log('Hello World is Running on http://localhost:3000/');

Here we have imported two modules, where router stores koa-router and mount stores the koa-mount module, allowing us to use the router in our Koa application.

On line 6, we have defined our handler function, which is the same function as before but here we have given it a name. On line 12, we save an instance of the router in APIv1, and on line 13 we register our handler for all the GET requests on route /all. 

So all the requests except when a get request is sent to localhost:3000/all will return "not found". Finally on line 15 , we use mount middleware, which gives a usable generator function that can be fed to app.use().

To directly reach this step or compare your application, execute the following command in the cloned repo:

$ git checkout 8f0d4e8

Before we run our application, now we need to install koa-router and koa-mount using npm. We observe that as the complexity of our application increases, the number of modules/dependencies also increases. 

To keep track of all the information regarding the project and make that data available to npm, we store all the information in package.json including all the dependencies. You can create package.json manually or by using an interactive command line interface which is opened using the $ npm init  command.

{
    "name": "koa-api-dictionary",
    "version": "0.0.1",
    "description": "koa-api-dictionary application",
    "main": "index",
    "author": {
    "name": "Bhanu Pratap Chaudhary",
    "email": "[email protected]"
    },
    "repository": {
    "type": "git",
    "url": "https://github.com/bhanuc/dictapi.git"
    },
    "license": "MIT",
    "engines": {
    "node": ">= 0.11.13"
    }
}

A very minimal package.json file looks like the one above. 

Once package.json is present, you can save the dependency using the following command:

$ npm install <package-name> --save

For example: In this case, we will install the modules using the following command to save the dependencies in package.json.

$ npm install koa-router koa-mount --save

Now you can run the application using $ node --harmony index.js. 

You can read more about package.json here.

Adding Routes for the Dictionary API

We will start by creating two routes for the API, one for getting a single result in a faster query, and a second to get all the matching words (which is slower for the first time). 

To keep things manageable, we will keep all the API functions in a separate folder called api and a file called api.js, and import it later in our main index.js file.

var monk = require('monk');
var wrap = require('co-monk');
var db = monk('localhost/mydb');
var words = wrap(db.get('words'));
/**
* GET all the results.
*/
exports.all = function *(){
    if(this.request.query.word){
        var res = yield words.find({ word : this.request.query.word });
        this.body = res;
    } else {
        this.response.status = 404;
        }
    };
/**
* GET a single result.
*/
exports.single = function *(){
    if(this.request.query.word){
        var res = yield words.findOne({ word : this.request.query.word });
        this.body = res;
    } else {
        this.response.status = 404;
    }
};

Here we are using co-monk, which acts a wrapper around monk, making it very easy for us to query MongoDB using generators in Koa. Here, we import monk and co-monk, and connect to the MongoDB instance on line 3. We call wrap() on collections, to make them generator-friendly. 

Then we add two generator methods named all and single as a property of the exports variable so that they can be imported in other files. In each of the functions, first we check for the query parameter 'word.' If present, we query for the result or else we reply with a 404 error. 

We use the yield keyword to wait for the results as discussed in the first article, which pauses the execution until the result is received. On line 12, we use the find method, which returns all the matching words, which is stored in res and subsequently sent back. On line 23, we use the findOne method available on the collection, which returns the first matching result. 

Assigning These Handlers to Routes

var koa = require('koa');
var app = koa();
var router = require('koa-router');
var mount = require('koa-mount');
var api = require('./api/api.js');

var APIv1 = new router();
APIv1.get('/all', api.all);
APIv1.get('/single', api.single);


app.use(mount('/v1', APIv1.middleware()));
if (!module.parent) app.listen(3000);
console.log('Dictapi is Running on http://localhost:3000/');

Here we import exported methods from api.js and we assign handlers to GET routes /all  /single and we have a fully functional API and application ready.

To run the application, you just need to install the monk and co-monk modules using the command below. Also, ensure you have a running instance of MongoDB in which you have imported the collection present in the git repository using the instructions mentioned in import.txtweird.

$ npm install monk co-monk --save

Now you can run the application using the following command:

$ node --harmony index.js

You can open the browser and open the following URLs to check the functioning of the application. Just replace 'new' with the word you want to query.

• http://localhost:3000/v1/all?word=new
• http://localhost:3000/v1/single?word=new

To directly reach this step or compare your application, execute the following command in the cloned repo:

$ git checkout f1076eb  

Error Handling in Koa

By using cascading middlewares, we can catch errors using the try/catch mechanism, as each middleware can respond while yielding to downstream as well as upstream. So, if we add a Try and Catch middleware in the beginning of the application, it will catch all the errors encountered by the request in the rest of the middleware as it will be the last middleware during upstreaming. Adding the following code on line 10 or before in index.js should work.

app.use(function *(next){
try{
    yield next; //pass on the execution to downstream middlewares
} catch (err) { //executed only when an error occurs & no other middleware responds to the request
this.type = 'json'; //optional here
this.status = err.status || 500;
this.body = { 'error' : 'The application just went bonkers, hopefully NSA has all the logs ;) '};
//delegate the error back to application
this.app.emit('error', err, this);
    }
});

Adding Logging and Rate-Limiting to the Application

Storing logs is an essential part of a modern-day application, as logs are very helpful in debugging and finding out issues in an application. They also store all the activities and thus can be used to find out user activity patterns and interesting other patterns. 

Rate-limiting has also become an essential part of modern-day applications, where it is important to stop spammers and bots from wasting your precious server resources and to stop them from scraping your API.

It is fairly easy to add logging and rate-limiting to our Koa application. We will use two community modules: koa-logger and koa-better-rate-limiting. We need to add the following code to our application:

var logger = require('koa-logger');
var limit = require('koa-better-ratelimit');
//Add the lines below just under error middleware.
app.use(limit({ duration: 1000*60*3 , // 3 min
                max: 10, blacklist: []}));
app.use(logger());

Here we have imported two modules and added them as middleware. The logger will log each request and print in the stdout of the process which can be easily saved in a file. And limit middleware limits the number of requests a given user can request in a given timeframe (here it is maximum ten requests in three minutes). Also you can add a array of IP addresses which will be blacklisted and their request will not be processed.

Do remember to install the modules before using the code using: 

$ npm install koa-logger koa-better-ratelimit --save

Compressing the Traffic

One of the ways to ensure faster delivery is to gzip your response, which is fairly simple in Koa. To compress your traffic in Koa, you can use the koa-compress module. 

Here, options can be an empty object or can be configured as per the requirement.

var compress = require('koa-compress');
var opts =  { 
    filter: function (content_type) { return /text/i.test(content_type) }, // filter requests to be compressed using regex 
    threshold: 2048, //minimum size to compress
    flush: require('zlib').Z_SYNC_FLUSH };
            }
//use the code below to add the middleware to the application
app.use(compress(opts));

You can even turn off compression in a request by adding the following code to a middleware:

this.compress = true;

Don't forget to install compress using npm. 

$ npm install compress --save 

To directly reach this step or compare your application, execute the following command in the cloned repo:

git checkout 8f5b5a6 

Writing Tests

Test should be an essential part of all code, and one should target for maximum test coverage. In this article, we will be writing tests for the routes that are accessible from our application. We will be using supertest and Mocha to create our tests. 

We will be storing our test in test.js in the api folder. In both tests, we first describe our test, giving it a more human readable name. After that, we will pass an anonymous function which describes the correct behavior of the test, and takes a callback which contains the actual test. In each test, we import our application, initiate the server, describe the request type, URL and query, and then set encoding to gzip.  Finally we check for the response if it's correct.

var request = require('supertest');
var api = require('../index.js');

describe('GET all', function(){
  it('should respond with all the words', function(done){
    var app = api;
    request(app.listen())
    .get('/v1/all')
    .query({ word: 'new' })
    .set('Accept-Encoding', 'gzip')    
    .expect('Content-Type', /json/)
    .expect(200)
    .end(done);
  })
})

describe('GET /v1/single', function(){
  it('should respond with a single result', function(done){
    var app = api;

    request(app.listen())
    .get('/v1/single')
    .query({ word: 'new' })
    .set('Accept-Encoding', 'gzip')
    .expect(200)
    .expect('Content-Type', /json/)
    .end(function(err, res){
    if (err) throw err;
    else {
        if (!('_id' in res.body)) return "missing id";
        if (!('word' in res.body)) throw new Error("missing word");
        done();
    }
  });
  })
})

To run our test, we will make a Makefile:

test:
    @NODE_ENV=test ./node_modules/.bin/mocha \
		--require should \
		--reporter nyan \
		--harmony \
		--bail \
		api/test.js

.PHONY: test

Here, we've configured the reporter (nyan cat) and the testing framework (mocha). Note that the import should add --harmony to enable ES6 mode. Finally, we also specify the location of all the tests. A Makefile can be configured for endless testing of your application.

Now to test your app, just use the following command in the main directory of the application. 

$ make test

Just remember to install testing modules (mocha, should, supertest) before testing, using the command below: 

$ npm install mocha should mocha --save-dev 

Running in Production

To run our applications in production, we will use PM2, which is an useful Node process monitor. We should disable the logger app while in production; it can be automated using environment variables.

To install PM2, enter the following command in terminal

$ npm install pm2 -g 

And our app can be launched using the following command:

$ pm2 start index.js --node-args="--harmony" 

Now, even if our application crashes, it will restart automatically and you can sleep soundly. 

Conclusion

Koa is a light and expressive middleware for Node.js that makes the process of writing web applications and APIs more enjoyable. 

It allows you to leverage a multitude of community modules to extend the functionality of your application and simplify all the mundane tasks, making web development a fun activity. 

Please don't hesitate to leave any comments, questions, or other information in the field below.