Rolling Your Own Framework

Building a framework from scratch is not something we specifically set out to do. You’d have to be crazy, right? With the plethora of JavaScript frameworks out there, what possible motivation could we have for rolling our own?

We were originally looking for a framework to build the new content management system for The Daily Mail website. The main objective was to make the editing process much more interactive with all the elements of an article (images, embeds, call-out boxes, and so on) being draggable, modular, and self-managing.

All the frameworks that we could put our hands on were designed for more or less static UI defined by developers. We needed to make an article with both editable text and dynamically rendered UI elements.

Backbone was too low level. It did little more than providing basic objects structure and messaging. We would have to build a lot of abstraction above the Backbone foundation, so we decided that we’d rather build this foundation ourselves.

AngularJS became our framework of choice for building small to medium size browser applications that have relatively static UIs. Unfortunately, AngularJS is very much a black box – it does not expose any convenient API to extend and manipulate the objects that you create with it – directives, controllers, services. Also, while AngularJS provides reactive connections between views and scope expressions, it does not allow defining reactive connections between models, so any application of medium size becomes very similar to a jQuery application with the spaghetti of event listeners and callbacks, with the only difference that instead of event listeners, an angular application has watchers and instead of manipulating DOM you manipulate scopes.

What we always wanted was a framework that would allow;

Developing applications in a declarative way with reactive bindings of models to views.
Creating reactive data bindings between different models in the application to manage data propagation in a declarative rather than in an imperative style.
Inserting validators and translators in these bindings, so we could bind views to data models rather than to view models like in AngularJS.
Precise control over components linked to DOM elements.
Flexibility of views management allowing you to both automatically manipulate DOM changes and to re-render some sections using any templating engine in cases where rendering is more efficient than DOM manipulation.
Ability to dynamically create UIs.
Being able to hook into mechanisms behind data reactivity and to precisely control view updates and data flow.
Being able to extend functionality of components supplied by the framework and to create new components.

We couldn’t find what we needed in existing solutions, so we started developing Milo in parallel with the application that uses it.

Why Milo?

Milo was chosen as the name because of Milo Minderbinder, a war profiteer from Catch 22 by Joseph Heller. Having started from managing mess operations, he expanded them into a profitable trading enterprise that connected everybody with everything, and in that Milo and everybody else "has a share".

Milo the framework has the module binder, that binds DOM elements to components (via special ml-bind attribute), and the module minder that allows establishing live reactive connections between different data sources (Model and Data facet of components are such data sources).

Coincidentally, Milo can be read as an acronym of MaIL Online, and without the unique working environment at the Mail Online, we never would have been able to build it.

Managing Views

Binder

Views in Milo are managed by components, which are basically instances of JavaScript classes, responsible for managing a DOM element. Many frameworks use components as a concept to manage UI elements, but the most obvious one that comes to mind is Ext JS. We had worked extensively with Ext JS (the legacy application we were replacing was built with it), and wanted to avoid what we considered to be two drawbacks of its approach.

The first is that Ext JS does not make it easy for you to manage your markup. The only way to build a UI is to put together nested hierarchies of component configurations. This leads to needlessly complex rendered markup and takes control out of the hands of the developer. We needed a method of creating components inline, in our own, hand-crafted HTML markup. This is where binder comes in.

Binder scans our markup looking for the ml-bind attribute so that it can instantiate components and bind them to the element. The attribute holds information about the components; this can include the component class, facets, and must include the component name.

<div ml-bind=”ComponentClass[facet1, facet2]:componentName”>
  Our milo component
</div>

We’ll talk about facets in a minute, but for now let’s look at how we can take this attribute value and extract the configuration from it using a regular expression.

var bindAttrRegex = /^([^\:\[\]]*)(?:\[([^\:\[\]]*)\])?\:?([^:]*)$/;

var result = value.match(bindAttrRegex);
// result is an array with
// result[0] = ‘ComponentClass[facet1, facet2]:componentName’;
// result[1] = ‘ComponentClass’;
// result[2] = ‘facet1, facet2’;
// result[3] = ‘componentName’;

With that information, all we need to do is iterate over all of the ml-bind attributes, extract these values, and create instances to manage each element.

var bindAttrRegex = /^([^\:\[\]]*)(?:\[([^\:\[\]]*)\])?\:?([^:]*)$/;

function binder(callback) {
    var scope = {};
    
    // we get all of the elements with the ml-bind attribute
    var els = document.querySelectorAll('[ml-bind]');
    Array.prototype.forEach.call(els, function(el) {
        var attrText = el.getAttribute('ml-bind');
        var result = attrText.match(bindAttrRegex);
        
        var className = result[1] || 'Component';
        var facets = result[2].split(',');
        var compName = results[3];
        
        // assuming we have a registry object of all our classes
        var comp = new classRegistry[className](el);
        comp.addFacets(facets);
        comp.name = compName;
        scope[compName] = comp;
        
        // we keep a reference to the component on the element
        el.___milo_component = comp;
    });
    
    callback(scope);
}

binder(function(scope){
    console.log(scope); 
});

So with just a little bit of regex and some DOM traversal, you can create your own mini-framework with custom syntax to suit your particular business logic and context. In very little code, we have setup an architecture that allows for modular, self-managing components, which can be used however you like. We can create convenient and declarative syntax for instantiating and configuring components in our HTML, but unlike angular, we can manage these components however we like.

Responsibility-Driven Design

The second thing we didn’t like about Ext JS was that it has a very steep and rigid class hierarchy, which would have made it difficult to organie our component classes. We tried to write a list of all of the behaviours that any given component within an article might have. For instance, a component could be editable, it could be listening for events, it may be a drop target or be draggable itself. These are just a few of the behaviours needed. A preliminary list we wrote up had about 15 different types of functionality that could be required of any particular component.

Trying to organise these behaviours into some kind of hierarchical structure would have been not only a major headache, but also very limiting should we ever want to change the functionality of any given component class (something we ended up doing a lot). We decided to implement a more flexible object-oriented design pattern.

We had been reading up on Responsibility-Driven Design, which contrary to the more common model of defining the behavior of a class along with the data that it holds, is more concerned with the actions an object is responsible for. This suited us well as we were dealing with a complex and unpredictable data model, and this approach would allow us to leave the implementation of these details to later.

The key thing we took away from RDD was the concept of Roles. A role is a set of related responsibilities. In the case of our project, we identified roles such as editing, dragging, drop zone, selectable, or events among many others. But how do you represent these roles in code? For that, we borrowed from the decorator pattern.

The decorator pattern allows behaviour to be added to an individual object, either statically or dynamically, without affecting the behaviour of other objects from the same class. Now while the run-time manipulation of class behaviour has not been particularly necessary in this project, we were very interested in the type of encapsulation this idea provides. Milo’s implementation is a kind of hybrid involving objects called facets, being attached as properties to the component instance. The facet gets a reference to the component, it’s ‘owner’, and a configuration object, which allows us to customise facets for each component class.

You can think of facets as advanced, configurable mixins that get their own namespace on their owner object and even their own init method, which needs to be overwritten by the facet subclass.

function Facet(owner, config) {
    this.name = this.constructor.name.toLowerCase();
    this.owner = owner;
    this.config = config || {};
    this.init.apply(this, arguments);
}
Facet.prototype.init = function Facet$init() {};

So we can subclass this simple Facet class and create specific facets for each type of behavior that we want. Milo comes prebuilt with a variety of facets, such as the DOM facet, which provides a collection of DOM utilities that operate on the owner component’s element, and the List and Item facets, which work together to create lists of repeating components.

These facets are then brought together by what we called a FacetedObject, which is an abstract class from which all components inherit. The FacetedObject has a class method called createFacetedClass that simply subclasses itself, and attaches all of the facets to a facets property on the class. That way, when the FacetedObject gets instantiated, it has access to all of it’s facet classes, and can iterate them to bootstrap the component.

function FacetedObject(facetsOptions /*, other init args */) {

    facetsOptions = facetsOptions ? _.clone(facetsOptions) : {};

    var thisClass = this.constructor
        , facets = {};

    if (! thisClass.prototype.facets)
        throw new Error('No facets defined');

    _.eachKey(this.facets, instantiateFacet, this, true);

    Object.defineProperties(this, facets);

    if (this.init)
        this.init.apply(this, arguments);

    function instantiateFacet(facetClass, fct) {
        var facetOpts = facetsOptions[fct];
        delete facetsOptions[fct];

        facets[fct] = {
            enumerable: false,
            value: new facetClass(this, facetOpts)
        };
    }
}

FacetedObject.createFacetedClass = function (name, facetsClasses) {
    var FacetedClass = _.createSubclass(this, name, true);

    _.extendProto(FacetedClass, {
        facets: facetsClasses
    });
    return FacetedClass;
};

In Milo, we abstracted a little further by creating a base Component class with a matching createComponentClass class method, but the basic principle is the same. With key behaviors being managed by configurable facets, we can create many different component classes in a declarative style without having to write too much custom code. Here is an example using some of the out-of-the-box facets that come with Milo.

var Panel = Component.createComponentClass(‘Panel’, {
    dom: {
		cls: ‘my-panel’,
		tagName: ‘div’
    },
    events: {
	    messages: {‘click’: onPanelClick}
    },
    drag: {messages: {...},
    drop: {messages: {...},
    container: undefined
});

Here we have created a component class called Panel, that has access to DOM utility methods, will automatically set its CSS class on init, it can listen for DOM events and will setup a click handler on init, it can be dragged around, and also act as a drop target. The last facet there, container ensures that this component sets up it’s own scope, and can, in effect, have child components.

Scope

We had discussed for a while whether or not all components attached to the document should form a flat structure or should form their own tree, where children are only accessible from their parent.

We would have definitely need scopes for some situations, but it could have been handled at implementation level, rather than on a framework level. For example, we have image groups that contain images. It would have been straightforward for these groups to keep a track of their child images without the need for a generic scope.

We finally decided to create a scope tree of components in the document. Having scopes makes many things easier and allows us to have more generic naming of components, but they obviously have to be managed. If you destroy a component, you have to remove it from its parent scope. If you move a component, it must be removed from one and added to another.

The scope is a special hash, or map object, with each of the children contained in the scope as properties of the object. The scope, in Milo, is found on the container facet, which itself has very little functionality. The scope object, however has a variety of methods for manipulating and iterating itself, but to avoid namespace conflicts, all of those methods are named with an underscore at the beginning.

var scope = myComponent.container.scope;

scope._each(function(childComp) {
    // iterate each child component
});

// access a specific component on the scope
var testComp = scope.testComp;

// get the total number of child components
var total = scope._length();

// add a new component ot the scope
scope._add(newComp);

Messaging – Synchronous vs. Asynchronous

We wanted to have loose coupling between components, so we decided to have messaging functionality attached to all components and facets.

The first implementation of the messenger was just a collection of methods that were managing arrays of subscribers. Both the methods and the array were mixed right into the object that implemented messaging.

A simplified version of the first messenger implementation looks something like this:

var messengerMixin =  {
    initMessenger: initMessenger,
    on: on,
    off: off,
    postMessage: postMessage
};


function initMessenger() {
    this._subscribers = {};
}

function on(message, subscriber) {
    var msgSubscribers = this._subscribers[message] =
        this._subscribers[message] || [];

    if (msgSubscribers.indexOf(subscriber) == -1)
        msgSubscribers.push(subscriber);
}

function off(message, subscriber) {
    var msgSubscribers = this._subscribers[message];
    if (msgSubscribers) {
        if (subscriber)
            _.spliceItem(msgSubscribers, subscriber);
        else
            delete this._subscribers[message];
    }
}

function postMessage(message, data) {
    var msgSubscribers = this._subscribers[message];
    if (msgSubscribers)
        msgSubscribers.forEach(function(subscriber) {
            subscriber.call(this, message, data);
        });
}

Any object that used this mix-in can have messages emitted on it (by object itself or by any other code) with postMessage method and subscriptions to this code can be switched on and off with methods that have the same names.

Nowadays, messengers have substantially evolved to allow:

Attaching external sources of messages (DOM messages, window message, data changes, another messenger etc.) – e.g. Events facet uses it to expose DOM events via Milo messenger. This functionality is implemented via a separate class MessageSource and its subclasses.
Defining custom messaging APIs that translate both messages and data of external messages to internal message. E.g. Data facet uses it to translate change and input DOM events to data change events (see Models below). This functionality is implemented via a separate class MessengerAPI and its subclasses.
Pattern subscriptions (using regular expressions). E.g. models (see below) internally use pattern subscriptions to allow deep model change subscriptions.
Defining any context (the value of this in subscriber) as part of the subscription with this syntax:

component.on('stateready',
    { subscriber: func, context: context });

Creating subscription that only dispatched once with the once method
Passing callback as a third parameter in postMessage (we considered variable number of arguments in postMessage, but we wanted a more consistent messaging API than we would have with variable arguments)
etc.

The main design mistake we made while developing messenger was that all messages were dispatched synchronously. Since JavaScript is single-threaded, long sequences of messages with complex operations being carried out would quite easily lock the UI. Changing Milo to make message dispatch asynchronous was easy (all subscribers are called on their own execution blocks using setTimeout(subscriber, 0), changing the rest of the framework and the application was more difficult – while most messages can be dispatched asynchronously, there are many that still have to be dispatched synchronously (many DOM events that have data in them or places where preventDefault is called). By default, messages are now dispatched asynchronously, and there is a way to make them synchronous either when the message is sent:

component.postMessageSync('mymessage', data);

or when subscription is created:

component.onSync('mymessage', function(msg, data) {
    //...
});

Another design decision we made was the way we exposed the methods of messenger on the objects using them. Originally, methods were simply mixed into the object, but we didn’t like that all the methods are exposed and we could not have standalone messengers. So messengers were re-implemented as a separate class based on an abstract class Mixin.

Mixin class allows exposing methods of a class on a host object in such way that when methods are called, the context will still be Mixin rather than the host object.

It proved to be a very convenient mechanism – we can have full control on which methods are exposed and change the names as necessary. It also allowed us to have two messengers on one object, which is used for models.

In general, Milo messenger turned out to be a very solid piece of software that can be used on its own, both in browser and in Node.js. It’s been hardened by usage in our production content management system that has tens of thousands lines of code.

Next Time

In the next article, we’ll be looking at possibly the most useful and complex part of Milo. The Milo models not only allow safe, deep access to properties, but also event subscription to changes at any level.

We’ll also explore our implementation of minder, and how we use connector objects to do one- or two-way binding of data sources.

Note that this article was written by both Jason Green and Evgeny Poberezkin.

HIGHLIGHTS OF THE DAY