In April, investor and Netscape founder Marc Andreessen began expanding on the natural 140 character limits of Twitter by publishing his thoughts in sequences of tweets, which some have dubbed tweet storms (entrepreneur Yvo Schaap compiled these on his website).
It didn't take long before BuzzFeed (the listicle upstart that must be stopped) rose up against this practice: Why Twitter’s Newest Tweetstorm™ Trend Must Be Stopped. Personally, I'm probably for anything BuzzFeed is against.
A few services arose to make it easier for mere mortals like ourselves to publish tweet storms but they seemed a bit unreliable and inconsistent. I decided to build the feature myself and I think there's value in doing this with your own app.
In this tutorial, I'll walk you through building a tweet storm feature of your own using the Twitter API. This is a continuation of my series of Twitter API tutorials on Tuts+; you can find a link to all of them on my author page.
Tweet Storm Feature Requirements
First, let's decide what we need our TweetStorm feature to accomplish.
- Create a group of tweets.
- Number them in sequence.
- Publish them in sequential order to Twitter.
- Provide a public web page allowing people to read them together.
- Publish the link to this page in a final tweet.
The results should look something like this:
I'll assume that you're familiar with my earlier Birdcage tutorial and already have code that authenticates your account via OAuth with the Twitter API.
The Database Model
Birdcage uses a Status table for tweets. First, we'll extend this table with a Yii ActiveRecord migration to include fields for the published tweet_id and the tweet storm numerical sequence.
Extending our existing database model is quite easy with Yii's ActiveRecord migrations:
./app/protected/yiic migrate create extend_status_table
Here's the migration code:
<?php
class m141020_182509_extend_status_table_for_groups extends CDbMigration
{
protected $MySqlOptions = 'ENGINE=InnoDB CHARSET=utf8 COLLATE=utf8_unicode_ci';
public $tablePrefix;
public $tableName;
public function before() {
$this->tablePrefix = Yii::app()->getDb()->tablePrefix;
if ($this->tablePrefix <> '')
$this->tableName = $this->tablePrefix.'status';
}
public function safeUp()
{
$this->before();
$this->addColumn($this->tableName,'tweet_id','BIGINT(20) DEFAULT 0');
$this->addColumn($this->tableName,'sequence','TINYINT DEFAULT 0');
$this->addColumn($this->tableName,'error_code','INTEGER DEFAULT 0');
}
public function safeDown()
{
$this->before();
$this->dropColumn($this->tableName,'tweet_id');
$this->dropColumn($this->tableName,'sequence');
$this->dropColumn($this->tableName,'error_code');
}
}
We'll use the tweet_ids to publicly display the entire tweet storm on a web page after publication. The numerical sequence will determine the order of status tweets in our storm.
Next, we need to create a table for a container of status tweets, in essence a container for the tweet storm. I'm going to use the term Group, because we will reuse it for another grouping feature in a follow up tutorial, recurring tweets from within a group. Visit my author page to see when it's out or follow me on Twitter @reifman.
Let's create a new migration to create the Group table:
./app/protected/yiic migrate create create_group_table
The code below builds the schema. Note the foreign key relation to link the tweet storm to a specific Twitter account:
<?php
class m141018_004954_create_group_table extends CDbMigration
{
protected $MySqlOptions = 'ENGINE=InnoDB CHARSET=utf8 COLLATE=utf8_unicode_ci';
public $tablePrefix;
public $tableName;
public function before() {
$this->tablePrefix = Yii::app()->getDb()->tablePrefix;
if ($this->tablePrefix <> '')
$this->tableName = $this->tablePrefix.'group';
}
public function safeUp()
{
$this->before();
$this->createTable($this->tableName, array(
'id' => 'pk',
'account_id'=>'integer default 0',
'name'=>'string default NULL',
'slug'=>'string default NULL',
'group_type'=>'tinyint default 0',
'stage'=>'integer default 0',
'created_at' => 'DATETIME NOT NULL DEFAULT 0',
'modified_at' => 'TIMESTAMP DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP',
), $this->MySqlOptions);
$this->addForeignKey('fk_group_account', $this->tableName, 'account_id', $this->tablePrefix.'account', 'id', 'CASCADE', 'CASCADE');
}
public function safeDown()
{
$this->before();
$this->dropForeignKey('fk_group_account', $this->tableName);
$this->dropTable($this->tableName);
}
}
We'll also build a relational table called GroupStatus which tracks the Status tweets within each Group:
<?php
class m141018_020428_create_group_status_table extends CDbMigration
{
protected $MySqlOptions = 'ENGINE=InnoDB CHARSET=utf8 COLLATE=utf8_unicode_ci';
public $tablePrefix;
public $tableName;
public function before() {
$this->tablePrefix = Yii::app()->getDb()->tablePrefix;
if ($this->tablePrefix <> '')
$this->tableName = $this->tablePrefix.'group_status';
}
public function safeUp()
{
$this->before();
$this->createTable($this->tableName, array(
'id' => 'pk',
'group_id' => 'INTEGER NOT NULL',
'status_id' => 'INTEGER default 0',
), $this->MySqlOptions);
$this->addForeignKey('fk_group_status_group', $this->tableName, 'group_id', $this->tablePrefix.'group', 'id', 'CASCADE', 'CASCADE');
$this->addForeignKey('fk_group_status_status', $this->tableName, 'status_id', $this->tablePrefix.'status', 'id', 'CASCADE', 'CASCADE');
}
public function safeDown()
{
$this->before();
$this->dropForeignKey('fk_group_status_group', $this->tableName);
$this->dropForeignKey('fk_group_status_status', $this->tableName);
$this->dropTable($this->tableName);
}
}
To publicly display the tweet storm on a web page, we actually need another table to cache the HTML code by twitter_id from Twitter's Oembed API method:
<?php
class m141021_203519_create_embed_table extends CDbMigration
{
protected $MySqlOptions = 'ENGINE=InnoDB CHARSET=utf8 COLLATE=utf8_unicode_ci';
public $tablePrefix;
public $tableName;
public function before() {
$this->tablePrefix = Yii::app()->getDb()->tablePrefix;
if ($this->tablePrefix <> '')
$this->tableName = $this->tablePrefix.'embed';
}
public function safeUp()
{
$this->before();
$this->createTable($this->tableName, array(
'id' => 'pk',
'tweet_id' => 'bigint(20) unsigned NOT NULL',
'html' => 'text default null',
'created_at' => 'DATETIME NOT NULL DEFAULT 0',
'modified_at' => 'TIMESTAMP DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP',
), $this->MySqlOptions);
$this->createIndex('tweet_id', $this->tableName , 'tweet_id', false);
}
public function safeDown()
{
$this->before();
$this->dropIndex('tweet_id', $this->tableName);
$this->dropTable($this->tableName);
}
}
Building the Code
The Group Controller and Models
Next, we'll use Yii's scaffolding code generator, Gii, to build the model, controllers and CRUD. On my local environment, I visit http://localhost:8888/twitter/app/gii to access Gii. First, I create the model:
Then, I user the Bootstrap CRUD generator:
We'll also use Gii to create default models for the GroupStatus and Embed tables. They don't need controllers and views.
In the navigation bar view (/app/protected/views/layouts/main.php), I'll add an option in my Compose menu to Group Tweets:
array('label'=>'Compose', 'items'=> array(
array('url'=>array('/status/compose'), 'label'=>'Schedule a tweet'),
array('url'=>array('/group'), 'label'=>'Group tweets'),
array('url'=>array('/status/admin'), 'label'=>'Manage schedule'),
array('url'=>array('/statuslog/admin'), 'label'=>'Review log'),
)),
The Group Tweets Management page looks like this:
Clicking the leftmost icon within each row opens up a group for adding tweets to and sequencing them.
Clicking the add a group menu link will bring up a form that lets you name the tweet storm and choose a Twitter account for it:
We need to extend the default create behavior to complete the model. I'm reusing my Slugify method from Geogram to create a URL-like slug from the name. Add this to Group.php:
/**
* Modifies a string to remove all non ASCII characters and spaces.
*/
public function slugify($text)
{
//sourcecookbook.com/en/recipes/8/function-to-slugify-strings-in-php
// replace non letter or digits by -
$text = preg_replace('~[^\\pL\d]+~u', '-', $text);
// trim
$text = trim($text, '-');
// transliterate
if (function_exists('iconv'))
{
$text = iconv('utf-8', 'us-ascii//TRANSLIT', $text);
}
// lowercase
$text = strtolower($text);
// remove unwanted characters
$text = preg_replace('~[^-\w]+~', '', $text);
if (empty($text))
{
return 'error-generating-slug';
}
return $text;
}
Here's the modified actionCreate in GroupController.php:
public function actionCreate()
{
$model=new Group;
// Uncomment the following line if AJAX validation is needed
// $this->performAjaxValidation($model);
if(isset($_POST['Group']))
{
$model->attributes=$_POST['Group'];
$model->slug=Group::model()->slugify($model->name);
$model->created_at = new CDbExpression('NOW()');
$model->modified_at =new CDbExpression('NOW()');
if($model->save())
$this->redirect(array('view','id'=>$model->id));
}
$this->render('create',array(
'model'=>$model,
));
}
Adding Tweets to the Storm
You then add status tweets for your storm and select the numerical sequence with which they should appear:
We use a derivative form of the status compose form from Birdcage:
<?php
$this->breadcrumbs=array(
'Statuses'=>array('index'),
'Create',
);
/*
$this->menu=array(
array('label'=>'Manage schedule','url'=>array('admin')),
array('label'=>'Review log','url'=>array('statuslog/admin')),
);
*/
?>
<h1>Compose a Tweet for your Group</h1>
<?php
echo $this->renderPartial('_groupform', array('model'=>$model,'maxCount'=>$maxCount)); ?>
Here is the _groupform.php view file:
<?php
$baseUrl = Yii::app()->baseUrl;
$cs = Yii::app()->getClientScript();
$cs->registerScriptFile($baseUrl.'/js/jquery.simplyCountable.js');
$cs->registerScriptFile($baseUrl.'/js/twitter-text.js');
$cs->registerScriptFile($baseUrl.'/js/twitter_count.js');
?>
<?php $form=$this->beginWidget('bootstrap.widgets.TbActiveForm',array(
'id'=>'status-form',
'enableAjaxValidation'=>false,
)); ?>
<?php
if(Yii::app()->user->hasFlash('no_account')
) {
$this->widget('bootstrap.widgets.TbAlert', array(
'alerts'=>array( // configurations per alert type
'no_account'=>array('block'=>true, 'fade'=>true, 'closeText'=>'×'),
),
));
}
?>
<p class="help-block">Fields with <span class="required">*</span> are required.</p>
<?php echo $form->errorSummary($model); ?>
<?php echo $form->hiddenField($model,'account_id',array('value'=>$model->account_id)); ?>
<br />
<?php
echo $form->textAreaRow($model,'tweet_text',array('id'=>'tweet_text','rows'=>6, 'cols'=>50, 'class'=>'span8'));
?>
<p class="right">Remaining: <span id="counter2">0</span></p>
<?php
echo CHtml::activeLabel($model,'sequence',array('label'=>'Sequence:'));
?>
<?php echo $form->dropDownList($model,'sequence', $model->getSequence()); ?>
</div> <!-- end section method -->
<div class="form-actions">
<?php $this->widget('bootstrap.widgets.TbButton', array(
'buttonType'=>'submit',
'type'=>'primary',
'label'=>$model->isNewRecord ? 'Create' : 'Save',
)); ?>
</div>
<?php $this->endWidget(); ?>
<script type="text/javascript" charset="utf-8">
$(document).ready(function()
{
$('#section_schedule').hide();
$('#section_method').hide();
$('#tweet_text').simplyCountable({
counter: '#counter2',
maxCount: <?php echo $maxCount ?>,
countDirection: 'down'
});
});
</script>
Here's the getSequence method the form uses, from Status.php:
public function getSequence() {
$i=1;
$seq = array('default'=>'select sequence below');
while ($i<=99) {
$seq[$i]=$i;
$i++;
}
return $seq;
}
Status tweets within groups are four characters shorter than 140 to allow for the insertion of sequence numbering.
As you add status tweets to your group, the Manage Group page will look something like this:
Future enhancements might include move up/down sequence controls for the status items as well as some auto-numbering when you add new tweets.
Publishing the Storm
While you can just publish the tweets in sequence, it's helpful to manage this in a background process for robustness, in case there are failures midway through.
Since many Twitter API functions require cursoring or paging, I've already built a background process model into my advanced framework, Birdhouse. I'll show you the basics of adding tweet storm publishing to your application.
When you click Publish storm, we create an action to manage this process in the background:
public function publish($group_id) {
// create an action to publish the storm in the background
$gp = Group::model()->findByPK($group_id);
$check_dup = Action::model()->findByAttributes(array('action'=>Action::ACTION_STORM,'item_id'=>$group_id,'status'=>Action::STATUS_ACTIVE));
if (empty($check_dup)) {
$a = new Action;
$a->account_id = $gp->account_id;
$a->action = Action::ACTION_STORM;
$a->status = Action::STATUS_ACTIVE;
$a->item_id = $group_id;
$a->created_at = new CDbExpression('NOW()');
$a->modified_at =new CDbExpression('NOW()');
$a->save();
}
}
Then, the normal cron background tasks manage the Action table and will call Action::model()->publishStorm:
public function publishStorm($action) {
// publish group twitter storm
$results = Group::model()->publishStormItems($action->account_id,$action->item_id);
if ($results) {
// if true, action is complete
$a = Action::model()->findByPk($action->id);
$a->status=self::STATUS_COMPLETE;
$a->save();
}
}
This in turn calls the Group model's publishStormItems:
// publish the group as a twitter storm
public function publishStormItems($account_id,$group_id) {
$error = false;
$account = Account::model()->findByPK($account_id);
// make the connection to Twitter
$twitter = Yii::app()->twitter->getTwitterTokened($account['oauth_token'], $account['oauth_token_secret']);
// get unpublished statuses in specific group
$statuses = Status::model()->in_account($account_id)->stage_zero()->in_specific_group($group_id)->findAll(array('order'=>'sequence ASC'));
$tweet_id = 0;
foreach ($statuses as $status) {
$prefix = $status->sequence.'. ';
// add sequence count as prefix
echo $prefix.$status->tweet_text;lb();
$tweet_id = Status::model()->postTweet($twitter,$status,$prefix);
if ($tweet_id!=0) {
// update stage to published = 1
$ns = Yii::app()->db->createCommand()->update(Yii::app()->getDb()->tablePrefix.'status',array('stage'=>1,'tweet_id'=>$tweet_id),'id=:id', array(':id'=>$status->id));
} else {
$error = true;
}
}
// if finishing up
if (count($statuses)>0 and !$error) {
// publish final tweet with link to the storm
$group = Group::model()->findByPk($group_id);
$status = 'Read or share my tweet storm on '.$group->name.' in its entirety here: '.$_SERVER["SERVER_NAME"].'/storm/'.$group->slug;
if (strlen($status)>120) {
$status = 'Read or share all of my tweet storm: '.$_SERVER["SERVER_NAME"].'/storm/'.$group->slug;
}
$tweet= $twitter->post("statuses/update",array('status'=>$status));
}
// if done, return true
return !$error;
}
The ActiveRecord query we use to find the status tweets in a group that are unpublished is as follows:
$statuses = Status::model()->in_account($account_id)->stage_zero()->in_specific_group($group_id)->findAll(array('order'=>'sequence ASC'));
This query uses scopes which are defined as follows in the Status model:
public function scopes()
{
return array(
// part of a group of tweets
'in_group'=>array(
'condition'=>'status_type='.self::STATUS_TYPE_IN_GROUP,
),
'stage_zero'=>array(
'condition'=>'stage=0',
),
'has_tweet_id'=>array(
'condition'=>'tweet_id<>0',
),
);
}
public function in_specific_group($group_id)
{
$crit = $this->getDbCriteria();
$crit->addCondition("
id IN (
SELECT status_id FROM tw_group_status
WHERE
group_id = :group_id
)
");
$crit->params[':group_id'] = $group_id;
return $this;
}
// custom scopes
public function in_account($account_id=0)
{
$this->getDbCriteria()->mergeWith( array(
'condition'=>'account_id='.$account_id,
));
return $this;
}
As we loop through each status that needs to be tweeted, we add a prefix for the sequence number e.g. "1. My first tweet is ...":
foreach ($statuses as $status) {
$prefix = $status->sequence.'. ';
// add sequence count as prefix
echo $prefix.$status->tweet_text;lb();
$tweet_id = Status::model()->postTweet($twitter,$status,$prefix);
if ($tweet_id!=0) {
// update stage to published = 1
$ns = Yii::app()->db->createCommand()->update(Yii::app()->getDb()->tablePrefix.'status',array('stage'=>1,'tweet_id'=>$tweet_id),'id=:id', array(':id'=>$status->id));
} else {
$error = true;
}
}
When a status is posted, the stage is incremented. Future enhancements could include allowing tweet storms to be reposted multiple times. Currently, we just allow one posting (you're welcome, Buzzfeed).
If all of the tweets have been successfully posted, we post a final tweet with a link to the tweet storm:
// if finishing up
if (count($statuses)>0 and !$error) {
// publish final tweet with link to the storm
$group = Group::model()->findByPk($group_id);
$status = 'Read or share my tweet storm on '.$group->name.' in its entirety here: '.$_SERVER["SERVER_NAME"].'/storm/'.$group->slug;
if (strlen($status)>130) {
$status = 'Read or share all of my tweet storm: '.$_SERVER["SERVER_NAME"].'/storm/'.$group->slug;
}
$tweet= $twitter->post("statuses/update",array('status'=>$status));
}
Here's what a Tweet Storm looks like when published:
Viewing the Storm on the Web
So, while we could display tweet storms publicly on the web in text form like Yvo Schaap, I thought it would be better to use Twitter embeddings that the user can interact with e.g. follow, reply, retweet et al.
Initially, I thought I might be able to use static HTML code and just replace the twitter_id in the view, but Twitter prefers you make an OEmbed call and cache the HTML for each tweet. I created the Embed table above to do this.
So, first, let's create a route in Yii's UrlManager so our app's path can redirect to view tweetstorms by the URL-friendly slug. In /app/protected/config/main.php, add the storm slug redirect below. That will redirect queries to http://yourdomain/storm/my-thoughts-on-twitters-api to the Group controller's lookup action with my-thoughts-on-twitters-api as the parameter:
'urlManager'=>array(
'urlFormat'=>'path',
'showScriptName'=>false,
'caseSensitive'=>false,
'rules'=>array(
'<controller:\w+>/<id:\d+>'=>'<controller>/view',
'<controller:\w+>/<action:\w+>/<id:\d+>'=>'<controller>/<action>',
'<controller:\w+>/<action:\w+>'=>'<controller>/<action>',
'/hybridauth' => '/hybridauth',
'/storm/<slug>' => '/group/lookup/',
'' => 'site/index'
),
),
Using Yii's controller access filters, let's make it so any public visitor can view storms but CRUD actions are only visible to authenticated users. Modify the accessRules in GroupController.php as follows:
public function accessRules()
{
return array(
array('allow', // allow all users to perform 'storm'
'actions'=>array('lookup'),
'users'=>array('*'),
),
array('allow', // allow authenticated user to perform these actions
'actions'=>array('index','view','create','update','delete'),
'users'=>array('@'),
),
array('allow', // allow admin user to perform 'admin' actions
'actions'=>array(''),
'users'=>array('admin'),
),
array('deny', // deny all users
'users'=>array('*'),
),
);
}
When a request arrives for http://yourdomain.com/storm/your-storm-name, it routes to the Group controller lookup action:
public function actionLookup($slug) {
$success = false;
$this->layout = 'storm';
// try look up by slug
$group = Group::model()->findByAttributes(array('slug'=>$slug));
$embeds = Group::model()->fetchEmbeds($group->id);
if ($group !== null) {
$success = true;
$this->render('lookup',array(
'embeds'=>$embeds,
'name'=>$group->name,
));
}
}
The fetchEmbeds method looks first in our database for a cached copy and then outwardly to the Twitter API to fetch the HTML. The fetchEmbeds method builds an array of HTML tweets:
public function fetchEmbeds($group_id) {
$e = new Embed();
$group = Group::model()->findByPk($group_id);
$embed =array();
$statuses = Status::model()->in_account($group->account_id)->has_tweet_id()->in_specific_group($group_id)->findAll(array('order'=>'sequence ASC'));
foreach ($statuses as $status) {
$embed[]=$e->fetch($group->account_id, $status->tweet_id);
}
return $embed;
}
It uses the Embed model's fetch method:
public function fetch($account_id,$tweet_id) {
// is it in embed table
$data = Embed::model()->findByAttributes(array('tweet_id'=>$tweet_id));
if (empty($data)) {
// is there a connection
if (is_null($this->twitter)) {
$account = Account::model()->findByPK($account_id);
// make the connection to Twitter
$this->twitter = Yii::app()->twitter->getTwitterTokened($account['oauth_token'], $account['oauth_token_secret']);
}
$result= $this->twitter->get("statuses/oembed",array('id'=>$tweet_id));
$html = $result->html;
$this->add($tweet_id,$html);
} else {
$html = $data->html;
}
return $html;
}
public function add($tweet_id,$html) {
$e = new Embed();
$e->html = $html;
$e->tweet_id=$tweet_id;
$e->modified_at = new CDbExpression('NOW()');
$e->created_at = new CDbExpression('NOW()');
$e->save();
}
It only initiates the OAuth connection to Twitter if there is at least one tweet_id that needs to be fetched, and it does so only one time for performance reasons.
In Closing
I hope you found this tweet storm tutorial useful. It was definitely a fun feature to build. Please feel free to post corrections, questions or comments below. I do try to keep up with the Tuts+ comment threads. You can also reach me on Twitter @reifman or email me directly.
You can find the initial Twitter API tutorial for Birdcage here, and a link to all of my Twitter API tutorials on my Tuts+ author page as they are published. Birdcage offers a free and open source Github repository for getting started with the basic Twitter API features.
Comments