Using the Mailgun Store(): A Temporary Mailbox for Your App's Incoming Email

Final product image — What You'll Be Creating

Welcome to a special episode of our Building Your Startup series sponsored by Mailgun. In this series, I'm guiding you through launching a startup from concept to reality using my Meeting Planner app as a real-life example. Every step along the way, we'll release the Meeting Planner code as open-source examples you can learn from. In today's episode, Mailgun stepped in to sponsor a tutorial about how I integrated its message routing and Store() API to handle replies from users.

For example, when people receive meeting requests from others with Meeting Planner, they may just choose to reply and send a note like they would to a typical email thread. However, I wanted to automatically process the reply, add it as a note to the meeting, and then notify the recipient that an update is ready to view. Sounds complicated, but one of Meeting Planner's goals is to reduce the back and forth emails between people about planning and consolidate real-time changes into fewer notifications.

What's Mailgun?

If you're not yet familiar with Mailgun, it's the email automation engine trusted by over 10,000 website and application developers for sending, receiving and tracking emails. By taking advantage of Mailgun's powerful email APIs, developers can spend more time building awesome websites and less time fighting with email servers.

It offers a variety of features for managing email quickly and efficiently for your application:

Mailgun's API supports all of the most popular languages, including PHP, Ruby, Python, C#, and Java, and they offer excellent, well-organized documentation.

It's been a go-to choice for me in many of my consulting activities, startups, and tutorials.

If you'd like to follow today's tutorial and aren't yet familiar with Mailgun, you may want to check out Exploring Mailgun: The Email Engine for Developers or How Geogram built a free group email service using Yii for PHP with MySQL. You may also want to read Store(): a temporary mailbox for all your incoming email for more technical background.

All of the code for Meeting Planner is written in the Yii2 Framework for PHP. If you'd like to learn more about Yii2, check out our parallel series Programming With Yii2 at Envato Tuts+.

If you haven't tried out Meeting Planner yet, try scheduling your first meeting now. It's really coming together and approaching alpha release.

Feedback is welcome. If you have a question or topic suggestion, please post a comment below. You can also reach me on Twitter @reifman.

And thank you, Mailgun, for sponsoring this episode of our startup series!

Keep in mind that, because this is a sponsored tutorial, it may appear ahead of our regular Startup series episodes. So some code may appear further along than in upcoming episodes.

Using Mailgun in Meeting Planner

I've already talked about using Mailgun's SMTP delivery for outbound emails in Meeting Planner, but what about responding to emails?

For example, Meeting Planner's invitations have a concept called notes. Notes are comments or messages shared back and forth between recipients. Currently, they can be posted from the meeting view page. Here's an example:

Mailgun Store - Add a Note to a Meeting via the Web Interface

However, I wanted to make it easy for people to respond to an email about a meeting and have their message easily added as a note to the meeting, and then group that update with other notifications of changes about the meeting. For example, a person might accept places and times from their invitation email but then also reply to the invitation directly from their email application or cloud-based email. The meeting organizer will receive one notification of updates rather than several.

An Overview of How We Will Use Mailgun

Here's a summary of what we're building today:

First, all the Meeting Planner scheduling emails will use a reply-to address to direct everything to a @meetingplanner.io mailbox. Mailgun's assigned to process all the inbound emails via our MX records. With Mailgun routing, we can ask the service to notify our server whenever new messages arrive. Then, in the background, we can process them.

Mailgun parses messages with its expert algorithms so we don't have to. So it's easy for us to determine who replied to a message, which meeting they were replying to, and what note they were writing in their message. Finally, we'll add the note to the meeting and inform the other participant.

Follow along as I walk you through how this is done.

Receiving Inbound Email at Your Domain

Mailgun Store - Customer quote about Mailgun Parsing by Jonathan Novak Uservoice

A while ago, I set up MX records for MeetingPlanner.io to funnel all incoming email directly to Mailgun.

Mailgun Store - Infographic Showing Route of Incoming Email to Mailgun MX

Here's an example of MX records for Mailgun:

Mailgun Store - Example Mailgun MX Records

Using Mailgun's simple routing user interface, I set forwarding up for the jeff and support mailboxes for interested people to reach me or customer support.

Mailgun Store - Routes for Mailbox Exceptions like Support and Jeff

Mailgun's routing features also provide the following capabilities:

Optionally receive raw MIME message
All messages encoded to UTF-8 automatically
Free spam filtering
Easy testing of webhook endpoints
Text-part generation from HTML-only emails
Matching on all email headers (e.g. subject, from:, cc:) and recipients
Ability to chain together multiple filters to provide sophisticated expressions

Once I was ready to integrate inbound routing for meeting scheduling, I decided to use Mailgun's store() API. While Mailgun can deliver parsed emails to your web server in real time, temporary surges or your own service failures might cause you to miss messages. Based on customer feedback, Mailgun decided to build the capacity to store messages in its cloud and let your application process them over a three-day period.

Here's Mailgun describing some possible cases that a cloud store helps resolve:

In some cases, the attachments are large and cause time-outs when we try to POST the data to their servers. In other cases, there is a large volume of inbound email and customers would rather just run a GET request at some interval instead of having to handle multiple POSTs from us. Finally, it serves as redundancy in case their web service goes down and they can't accept our POSTs.

So I configured Mailgun's routing to simply notify the Meeting Planner server whenever it receives a new message. Notice the third configuration below:

Mailgun Store - Notification Route Post URL to Our Server

Whenever wildcard messages come into Meeting Planner, Mailgun will store() them and notify http://meetingplanner.io/mailgun-notification/store that there's a new message and then stop() processing.

Expanding my API usage with Mailgun was again straightforward as their documentation is top notch. It provides support for a variety of languages, e.g. Ruby, Python, PHP, Java, C#, and Go. It offers working examples of implementing Mailgun functionality across its services—whether you're broadcasting, tracking, or routing.

So, using Yii2 and its automated scaffolding code generator, Gii, I created migrations, models, and controllers for a MailgunNotification model.

class m160514_010650_create_mailgun_notification_table extends Migration
{
  public function up()
  {
      $tableOptions = null;
      if ($this->db->driverName === 'mysql') {
          $tableOptions = 'CHARACTER SET utf8 COLLATE utf8_unicode_ci ENGINE=InnoDB';
      }

      $this->createTable('{{%mailgun_notification}}', [
          'id' => Schema::TYPE_PK,
          'url' => Schema::TYPE_STRING.' NOT NULL',
          'status' => Schema::TYPE_SMALLINT.' NOT NULL DEFAULT 0',
          'created_at' => Schema::TYPE_INTEGER . ' NOT NULL',
          'updated_at' => Schema::TYPE_INTEGER . ' NOT NULL',
      ], $tableOptions);
  }

It stores a notification URL from Mailgun which we can fetch from later. For example:

https://api.mailgun.net/v2/domains/meetingplanner.io/messages/eyJw2UsICJrIjogImUyQ5LTk1NmItNGIwOCLWZmZTFjMDU3ZiIsICJzIjogIjNGUiLCAiYyIcyJ9

Here's the code that adds unprocessed notifications with STATUS_PENDING:

public function store($message_url) {
      // store the url from mailgun notification
        $mn = new MailgunNotification();
        $mn->status = MailgunNotification::STATUS_PENDING;
        $temp = str_ireplace('https://api.mailgun.net/v2/','',$message_url);
        $temp = str_ireplace('https://api.mailgun.net/v3/','',$temp);
        $mn->url = $temp;
        $mn->save();
    }

Processing POSTed Notifications

In this scenario, you have to write production code to process and later react to notifications and messages Mailgun notifies you about. Mailgun provides a helpful testing form to test out your server at the bottom of its routing configuration page.

Because production servers are more difficult to debug, it took some time to get this right. So I did run into errors at first.

Mailgun Store - Error as I Develop Mailgun Notification Processing

But then, ultimately, I got incoming notifications working:

Mailgun Store - A Successful Notification Test

Essentially, I parsed the message-url and stored it in the database:

public function actionStore()
    {
      // stores inbound post notification from Mailgun
      if (isset($_POST['message-url'])) {
        MailgunNotification::store($_POST['message-url']);
      }
    }

In this example, imagine that I've responded to a meeting invitation from Tom to remind him to bring a book to our coffee discussion:

Mailgun Store - User scenario as I send an email meeting note

In response, Mailgun notifies us of the message, and a secure URL to access its contents is stored in the MailgunNotification table:

Mailgun Store - MailgunNotification Table

Background Message Processing

Next, I extended Meeting Planner's background processing to fetch new messages from Mailgun and process them. Parsing email with Mailgun is easy as they've done all the work that would otherwise take months (and months... and months... and months).

Essentially, Mailgun takes unstructured inbound email (shown at left) and sends your application parsed, structured data (shown at right):

Mailgun Store - Infographic of Powerful Mailgun API Parsing

Uninformed independent developers spend months coding email parsing on their own, a task that's never really complete. Mailgun manages this for you.

Let's walk through the MailgunNotification::Process method. At first, we look for pending notifications in the MailgunNotification table and invoke my Yiigun.php component which makes a get() request to Mailgun at the URL we received for a notification:

public static function process() {
      $items = MailgunNotification::find()->where(['status'=>MailgunNotification::STATUS_PENDING])->all();
      if (count($items)==0) {
        return false;
      }
      $yg = new Yiigun();
      foreach ($items as $m) {
        $error = false;
        // echo $m->id.'<br />';
        $raw_response = $yg->get($m->url);
        if (is_null($raw_response)) {
          $m->status = MailgunNotification::STATUS_NOT_FOUND;
          $m->update();
          continue;
        }
        $response = $raw_response->http_response_body;

Mailgun returns the detailed data from the message. So we parse the meeting_id from the recipient field, the body text, and the sender email. Again, Mailgun makes this easy:

$stripped_text = \yii\helpers\HtmlPurifier::process($response->{'stripped-text'});
        // parse the meeting id
        if (isset($response->To)) {
          $to_address = $response->To;
        } else {
          $to_address = $response->to;
        }
        $to_address = str_ireplace('@meetingplanner.io','',$to_address);
        $to_address = str_ireplace('mp_','',$to_address);
        $meeting_id = intval($to_address);
        if (!is_numeric($meeting_id)) {
          $error = true;
          $m->status = MailgunNotification::STATUS_INVALID_MEETING_ID;
          $m->update();
          continue;
        }
        // echo 'mid: '.$meeting_id.'<br>';
        // verify meeting id is valid
        if (isset($response->Sender)) {
          $sender = $response->Sender;
        } else {
          $sender = $response->sender;
        }
        // clean sender
        // echo ' pre clean sender: '.$sender.'<br>';
        $sender = \yii\helpers\HtmlPurifier::process($sender);
        // echo 'sender: '.$sender.'<br>';
        $user_id = User::findByEmail($sender);
        if ($user_id===false) {
          $error = true;
          // do nothing
          // to do - reply with do not recognize email address
          $m->status = MailgunNotification::STATUS_UNRECOGNIZED_SENDER;
          $m->update();
          continue;

I especially appreciate that Mailgun provides stripped-text, which takes out the sender's signature and the reply thread. I also use HtmlPurifier to prevent potentially harmful unescaped text from targeting our server.

During this effort, I re-learned that to access a hyphenated property of an object in PHP, you need to surround it with curly brackets:

$stripped_text = \yii\helpers\HtmlPurifier::process($response->{'stripped-text'});

And finally, once we've determined that the meeting and the sender exist and that the latter is an attendee, we add the stripped_text as a note:

  // echo 'check attendee';
          // verify sender is a participant or organizer to this meeting
          $is_attendee = Meeting::isAttendee($meeting_id,$user_id);
          if ($is_attendee) {
            // add meeting note, automatically its logged and meeting touch stamp updated
            MeetingNote::add($meeting_id,$user_id,$stripped_text);
          } else {

The new note will appear on the Meeting event page when the participant returns:

Mailgun Store - Meeting Invitation with Note from Email Added Automatically

Yii supports ActiveRecord events so that when a MeetingNote is added, a MeetingLog entry is automatically created:

public function afterSave($insert,$changedAttributes)
    {
        parent::afterSave($insert,$changedAttributes);
        if ($insert) {
          // if MeetingNote
          MeetingLog::add($this->meeting_id,MeetingLog::ACTION_ADD_NOTE,$this->posted_by,$this->id);
        }
    }

In a future episode, I'll explain how this event will trigger another event which helps Meeting Planner coordinate when it's best to notify meeting attendees that a new note or other change has occurred.

Try sending a Meeting Planner invitation and ask your recipient to reply to the email with a message. You should receive a notification shortly after they've done so and then be able to view it on the event page.

In Closing

I hope you enjoyed this applied look at using Mailgun's routing and store() API. It's a pretty fun, sophisticated API to work with. Mailgun really offers a vast array of email services that are relevant to all kinds of coding and business development. And, impressively, they do a great job at all that they offer.

Certainly this is a sponsored post, but as an experienced user, I offer my authentic recommendation for you to try out the service today. I've been using them for all of my projects.

And, obviously, there are a lot of other scenarios we could build when processing messages. For example, we could let users respond with commands such as "late" and we'll know to text the other person that their colleague is running late.

Please let us know which Mailgun features you'd like to see written about more in the future. You can post them in the comments below or reach me directly on Twitter @reifman.

HIGHLIGHTS OF THE DAY