Building Advanced Email Features With IMAP and PHP

What You'll Be Creating

In this tutorial, I'll walk you through some real world examples of how you can use PHP and IMAP to build new features for managing your email—features that the big email providers haven't built for us.

My interest in this began in 2010 when I wrote Twelve Gmail Ideas to Revolutionize Email (Again), but mostly the ideas I wished for have remained out of reach. For as important as email is, innovation of email as an application has been quite slow.

We’re drowning in email, and managing our inboxes remains a heavy burden. Mail services and clients have done very little to help us with this. Most of the email we receive is sent by machines, not people, and yet we’re the ones who have to individually process all of it. 

Analysis of my own email showed I was receiving email from more than 230 automated senders, far fewer actual people. I was tired of constructing filters in Gmail and filling in a myriad of unsubscribe forms. I wanted to have more control over managing my email and simplifying my life.

Finally, this past year, I decided to build the features I needed. The result is Simplify Email (SE), a small web app you can host yourself which offers a variety of cool new email features all of which you can check out on the project website.

The coolest thing about SE is that it's a platform for reading, analyzing, routing and managing your email—the possibilities abound. Simplify Email is essentially a programmable playground for "hacking" your own email.

I'll walk you through the code of three examples from SE that use PHP, IMAP, and MySQL to work with email:

  1. Checking your inbox and filtering messages
  2. Implementing a Whitelist challenge to unknown senders
  3. Reporting unanswered email

This tutorial will definitely give you a head start at writing IMAP code in PHP. But you can also work directly with the Simplify Email codebase. You can purchase the code for as little as $10, and there is an older open source version (which lacks some of the features we're describing below). Installation guides are provided for typical Linux configurations. I also offer pre-installed images at Digital Ocean for $25 as well as a handheld valet install. SE is written in PHP, in the Yii Framework

Note that you won't be able to access most email servers via your local development machine unless you compile a secure IMAP library for PHP. This is one of the reasons I encourage people to run Simplify Email in droplets at Digital Ocean. There are also a few tricks to getting Google account security to let you in via IMAP.

Working with IMAP

How Simplify Email Works

With SE, you can continue using your email client of choice on both the Web and your mobile devices. You don't have to change any apps or personal habits. SE accesses your email accounts behind the scenes via IMAP; acting as a smart personal assistant, SE pre-processes your email, moving messages to the appropriate places based on everything you've told it.

When a message arrives from a familiar sender, SE moves it to the folder you've specified. When one arrives from an unknown sender for the first time, it moves it to the review folder. 

Every couple of hours (or at a frequency you choose), SE will send you a summary of where it moved your messages and which messages are in review. Note, links for training senders are included for the review folder, making it quite easy to train SE over time.

The Simplify Email Message Digest Shows You What Came In and How It Was Filtered

At any time, you can browse your review folder—you don't have to wait for the digest to arrive. But the advantage of SE is that you no longer have to browse your folders; you can just read your digest to get a view of the email that you've received and train new senders.

1. Checking Your Inbox and Filtering Messages

SE uses several cron tasks to operate in the background on your server. Each is called from DaemonController.php.

The first, processInbox, is called frequently and needs to operate quickly—its job is to screen email and move it out of the Inbox as quickly as possible and into the triage folder, called the filtering folder. 

The second, processFiltering, is more process-intensive and performs deeper operations on email, ultimately moving messages to their final destination.

The ProcessInbox Method

The cron tasks calls processInbox regularly:

The Simplify Email Accounts Configuration Page

For each account, we decrypt your e-mail credentials and then use imap_open to create an IMAP stream to your inbox folder:

Within processInbox, we use PHP library functions imap_search and imap_fetch_overview to retrieve an array of messages:

Then we process the array of messages in the inbox:

Here's an adapted version of publicly available IMAP header parsing code which gathers the additional information that SE needs for a variety of tasks. Basically, it uses imap_rfc822_parse_adrlist to determine recipient information, message-id, subject and timestamps (or sender information when scanning the sent folder):

We create records for the sender and the message envelope within our database:

If the sender is new to us (unknown), we'll send a whitelist challenge email (we'll talk more about whitelist challenges in the next section below):

Next, we determine if the user might have dragged a message from another folder back into the inbox—intending to train it via drag and drop. If so, we set the training for this sender to the inbox. In other words, next time, we'd want to just route messages from this sender to the inbox:

If not, we'll prepare to move the message to the Filtering folder for further processing. First, we may send notifications to the user's phone if there is a sender match or keyword match for notifications (and it's not quiet hours):

If the message was dragged to the inbox, then we'll update our training settings:

The ProcessFiltering Method

The secondary processing method is called processFiltering, also in DaemonController.php. It does the more time-consuming aspects of moving messages to the appropriate folders:

This method opens your email account to search for recent messages and gather data about them. It also uses imap_search, imap_fetch_overview and parseHeader:

The primary processing loop for each message in the filtering folder is quite detailed. First we look at the recipient address, as SE allows people to train folders by recipient address, e.g. messages to the domain go to the veggie folder:

Then we lookup the sender and create a new record in the database (if necessary). If training exists for the sender, we can set the destination folder:

If an untrained (new) sender has verified themselves via a Whitelist challenge (which we'll discuss in the next section below), then we'll route this message to the inbox:

Then, we create a message entry in the database with the envelope information about this message:

If it's from an unknown, unverified sender, we can move the message to the review folder. The review folder contains all the messages from senders we don't recognize.

If the message is from a known sender and we have a destination in mind, we can move it as long as it's not quiet hours (and do not disturb is off):

During quiet hours, messages are primarily kept in the filtering folder.

Every couple of hours, a different process will build the message digest using the message table records to determine which emails were recently received and filtered and how they were routed.

2. Implementing a Whitelist Challenge to Unknown Senders

The goal of the whitelist challenge is to keep any message from an unknown sender, e.g. possibly a marketing bot or spammer, out of your inbox. SE places mail from unknown senders in the review folder. However, if you turn whitelisting on, we send out a challenge email that gives the sender a chance to verify that they're human. If they respond, we'll move the message to your inbox. If the email turns out to be unwanted, you can zap the message from the digest or drag it to any folder you wish to train it to.

The user can turn whitelisting on and off in settings:

Simplify email settings for whitelisting

To implement whitelisting, we send out email challenges whenever mail arrives from a new sender:

ChallengeSender sends a coded link to the user for them to click. We also have some protections to make sure we don't trap ourselves in an email loop with an out of office message:

Then, if the recipient clicks on the coded link, we verify them in the database. The Sender Controller processes these requests and checks their validity:

This tells our processing loops to move this and future messages from this sender to the inbox.

3. Reporting Unanswered Email

Sometimes it helps to see a summary of messages you've sent but not received a reply to. To identify these, Simplify Email monitors messages that have been sent but haven't received a reply.

Every message we receive contains a unique id, called the message_id (part of the IMAP specification). It often looks like this:

Furthermore, when messages are sent in reply to other messages, they have an in_reply_to field which links back to the original message_id.

So, we use a SQL query to find all the received messages that don't have a corresponding reply message referencing their message_id. For this, we use a LEFT OUTER JOIN where there is no in_reply_to id:

We use the $subject_compare mode to differentiate between our sent messages that haven't been answered and our sent replies to a thread that haven't been answered. Here's the unanswered message report in your account:

Unanswered sent emails and unanswered sent replies threads

SE also offers this information as an optional digest, called the unanswered email digest. You can receive it every day, every few days or every week.

We also use similar SQL tabulation with Google Charts to provide reports of how frequently certain people email you:

Reports of inbound email frequency by sender

I'll be writing more about Google Charts for Tuts+ soon

Next Steps

I hope you've found Simplify Email intriguing enough to try your hand at PHP IMAP programming. There are so many cool features you can build without any need for the big email providers to do anything new.

If you have any questions or corrections, please post them in the comments. If you'd like to keep up on my future Tuts+ tutorials and other series, please follow @reifman or visit my author page. You can also contact me here.

Related Links

Here are some additional links that you may find useful:



Related Articles