Advanced Regular Expression Tips and Techniques

Twice a month, we revisit some of our readers’ favorite posts from throughout the history of Nettuts+.

Regular Expressions are the Swiss Army knife for searching through information for certain patterns. They have a wide arsenal of tools, some of which often go undiscovered or underutilized. Today I will show you some advanced tips for working with regular expressions.


Adding Comments

Sometimes, regular expressions can become complex and unreadable. A regular expression you write today may seem too obscure to you tomorrow even though it was your own work. Much like programming in general, it is a good idea to add comments to improve the readability of regular expressions.

For example, here is something we might use to check for US phone numbers.

It can become much more readable with comments and some extra spacing.

Let's put it within a code segment.

The trick is to use the 'x' modifier at the end of the regular expression. It causes the whitespaces in the pattern to be ignored, unless they are escaped (\s). This makes it easy to add comments. Comments start with '#' and end at a newline.


Using Callbacks

In PHP preg_replace_callback() can be used to add callback functionality to regular expression replacements.

Sometimes you need to do multiple replacements. If you call preg_replace() or str_replace() for each pattern, the string will be parsed over and over again.

Let's look at this example, where we have an e-mail template.

Notice that each replacement has something in common. They are always strings enclosed within square brackets. We can catch them all with a single regular expression, and handle the replacements in a callback function.

So here is the better way of doing this with callbacks:

Now the string in $template is only parsed by the regular expression once.


Greedy vs. Ungreedy

Before I start explaining this concept, I would like to show an example first. Let's say we are looking to find anchor tags in an html text:

The result will be as expected:

Let's change the input and add a second anchor tag:

Again, it seems to be fine so far. But don't let this trick you. The only reason it works is because the anchor tags are on separate lines, and by default PCRE matches patterns only one line at a time (more info on: 'm' modifier). If we encounter two anchor tags on the same line, it will no longer work as expected:

This time the pattern matches the first opening tag, and last opening tag, and everything in between as a single match, instead of making two separate matches. This is due to the default behavior being "greedy".

"When greedy, the quantifiers (such as * or +) match as many character as possible."

If you add a question mark after the quantifier (.*?) it becomes "ungreedy":

Now the result is correct. Another way to trigger the ungreedy behavior is to use the U pattern modifier.


Lookahead and Lookbehind Assertions

A lookahead assertion searches for a pattern match that follows the current match. This might be explained easier through an example.

The following pattern first matches for 'foo', and then it checks to see if it is followed by 'bar':

It may not seem very useful, as we could have simply checked for 'foobar' instead. However, it is also possible to use lookaheads for making negative assertions. The following example matches 'foo', only if it is NOT followed by 'bar'.

Lookbehind assertions work similarly, but they look for patterns before the current match. You may use (?< for positive assertions, and (?<! for negative assertions.

The following pattern matches if there is a 'bar' and it is not following 'foo'.


Conditional (If-Then-Else) Patterns

Regular expressions provide the functionality for checking certain conditions. The format is as follows:

The condition can be a number. In which case it refers to a previously captured subpattern.

For example we can use this to check for opening and closing angle brackets:

In the example above, '1' refers to the subpattern (<), which is also optional since it is followed by a question mark. Only if that condition is true, it matches for a closing bracket.

The condition can also be an assertion:


Filtering Patterns

There are various reasons for input filtering when developing web applications. We filter data before inserting it into a database, or outputting it to the browser. Similarly, it is necessary to filter any arbitrary string before including it in a regular expression. PHP provides a function named preg_quote to do the job.

In the following example we use a string that contains a special character (*).

Same thing can be accomplished also by enclosing the string between \Q and \E. Any special character after \Q is ignored until \E.

However, this second method is not 100% safe, as the string itself can contain \E.


Non-capturing Subpatterns

Subpatterns, enclosed by parentheses, get captured into an array so that we can use them later if needed. But there is a way to NOT capture them also.

Let's start with a very simple example:

Now let's make a small change by adding another subpattern (H.*) to the front:

The $matches array was changed, which could cause the script to stop working properly, depending on what we do with those variables in the code. Now we have to find every occurence of the $matches array in the code, and adjust the index number accordingly.

If we are not really interested in the contents of the new subpattern we just added, we can make it 'non-capturing' like this:

By adding '?:' at the beginning of the subpattern, we no longer capture it in the $matches array, so the other array values do not get shifted.


Named Subpatterns

There is another method for preventing pitfalls like in the previous example. We can actually give names to each subpattern, so that we can reference them later on using those names instead of array index numbers. This is the format: (?Ppattern)

We could rewrite the first example in the previous section, like this:

Now we can add another subpattern, without disturbing the existing matches in the $matches array:


Don't Reinvent the Wheel

Perhaps it's most important to know when NOT to use regular expressions. There are many situations where you can find existing utilities than you can use instead.

Parsing [X]HTML

A poster at Stackoverflow has a brilliant explanation on why we should not use regular expressions to parse [X]HTML.

...dear lord help us how can anyone survive this scourge using regex to parse HTML has doomed humanity to an eternity of dread torture and security holes using regex as a tool to process HTML establishes a breach between this world and the dread realm of corrupt entities...

Joking aside, it is a good idea to take some time and figure out what kind of XML or HTML parsers are available, and how they work. For example, PHP offers multiple extensions related to XML (and HTML).

Example: Getting the second link url in an HTML page

Validating Form Input

Again, you can use existing functions to validate user inputs, such as form submissions.

More info: PHP Data Filtering

Other

Here are some other utilities to keep in mind, before using regular expressions:


Thanks so much for reading!

Tags:

Comments

Related Articles