Slashdot Log In
Regular Expression Pocket Reference
Posted by
samzenpus
on Mon Mar 24, 2008 02:26 PM
from the read-all-about-it dept.
from the read-all-about-it dept.
Michael J. Ross writes "When software developers need to manipulate text programmatically — such as finding all substrings within some text that match a particular pattern — the most concise and flexible solution is to use "regular expressions," which are strings of characters and symbols that can look anything but regular. Nonetheless, they can be invaluable for locating text that matches a pattern (the "expression"), and optionally replacing the matched text with new text. Regular expressions have proven so popular that they have been incorporated into most if not all major programming languages and editors, and even at least one Web server. But each one implements regular expressions in its own way — which is reason enough for programmers to appreciate the latest edition of Regular Expression Pocket Reference, by Tony Stubblebine." Read below for the rest of Michael's review.
The second edition of the book was published by O'Reilly Media on 18 July 2007, under the ISBNs 0596514271 and 978-0596514273. On the book's Web page, the publisher makes available the book's table of contents and index, as well as links for providing feedback and any errata. As of this writing, there are no unconfirmed errata (those submitted by readers but not yet checked by the author to see whether they are valid), and no confirmed ones, either. In fact, in my review of the first edition, published in 2004, it was noted that there were no unconfirmed errata, despite the book being out for some time prior to that review. The most likely explanation is that the author — in addition to any technical reviewers — did a thorough job of checking all of the regular expressions in the book, along with the sample code that make use of them. These efforts have paid off with the apparent absence of any errors in this new edition — something unseen in any other technical book with which I am familiar.
Before discussing this particular book, it may be of value to briefly discuss the essential concept of regular expressions, for the benefit of any readers who are not familiar with them. As noted earlier, a regular expression (frequently termed a "regex") is a string of characters intended for matching substrings in a block of text. A regex pattern can match literally, such as the pattern "book" matching both "book" and "bookshelf." A pattern can also use special characters and character combinations — often termed metasymbols and metasequences — such as \w to indicate a single word character (A-Z, a-z, 0-9, or '_'). Thus, the regex "b\w\wk" would match "book," but not "brook."
Here is a simple example to show the use of regexes in code, written in Perl: The statement "$text =~ m/book/;" would find the first instance of the string "book" inside the scalar variable $text, which presumably contains some text. To substitute all instances of the string with the word "publication," you could use the statement "$text =~ s/book/publication/g;" ('g' for globally search) or use "$text =~ s/bo{2}k/publication/g;". In this simplistic example, the second statement makes use of a quantifier, {2}, indicating two of the preceding letter.
These examples employ only one metacharacter (\w) and one quantifier ({2}). The total number of metacharacters, metasymbols, quantifiers, character classes, and assertions (to say nothing of capturing, clustering, and alternation) that are available, in most regex-enabled languages, is tremendous. However, the same cannot be said for the readability of all but the simplest regular expressions — especially lengthy ones not improved by whitespace and comments. As a consequence, when using regexes in their code, many programmers find themselves repeatedly consulting reference materials that do not focus on regular expressions. These resources comprise convoluted Perl books, incomplete tutorials on the Internet, and confusing discussions in technical newsgroups. For too many years, there was no published book providing the details of regexes for the various languages that utilize them, in addition to a clear explanation of how to use regexes wisely.
Fortunately, O'Reilly Media offers two titles in hopes of meeting that need: Mastering Regular Expressions, by Jeffrey Friedl, and Regular Expression Pocket Reference, by Tony Stubblebine. In several respects, the books are related — particularly in that Stubblebine bases his slender monograph upon Friedl's larger and more extensive title, justifiably characterized by Stubblebine as "the definitive work on the subject." In addition, Stubblebine's book follows the structure of Friedl's book, and contains page references to the same. Another major difference is that Regular Expression Pocket Reference is, just as the title indicates, for reference purposes only, and not intended as a tutorial.
At first glance, it is clear that Stubblebine's book packs a great deal of information into its modest 126 pages. That may partly be a result of the terseness of most, if not all, of the regular expression syntax; a metasymbol of more than two characters would be considered long-winded! Yet the high information density is likely also due to the manner in which Stubblebine has distilled the operators and rules, as well as the meaning and usage thereof, down to the bare bones. But this does not imply that the book is bereft of examples. Most of the sections contain at least one, and sometimes several, code fragments that illustrate the regex elements under discussion.
The book begins with a brief introduction to regexes and pattern matching, followed by an even briefer cookbook section, with Perl-style regexes for a dozen commonly-needed tasks, e.g., validating dates. The bulk of the book's material is divided into 11 sections, each one devoted to the usage of regexes within a particular language, application, or library: Perl 5.8, Java,.NET and C#, PHP, Python, Ruby, JavaScript, PCRE, the Apache Web server, the vi programmer's editor, and shell tools.
Each of these sections begins with a brief overview of how regexes fit into the overall language covered in that section. Following this is a subsection listing all of the supported metacharacters, with a summary of their meanings, in tabular format. In most cases, this is followed by a subsection showing the usage of those metacharacters — either in the form of operators or pattern-matching functions, depending upon how regular expressions are used within that language. Next is a subsection providing several examples, which is often the first material that most programmers turn to when trying to quickly figure out how to use one aspect of a language. Each section concludes with a short listing of other resources related to regexes for that particular language.
There are no glaring problems in this book, and I can only assume that all of the regular expressions themselves have been tested by the author and by previous readers. However, there is a minor weakness that should be pointed out, and could be corrected in the next edition. In most of the sections' examples, Stubblebine wisely formats the code so that every left brace ("{") is on the same line as the beginning of the statement that uses that brace, and each closing brace ("}") is lined up directly underneath the first character of the statement. This format saves space and makes it easier to match up the statement with its corresponding close brace. However, in the.NET / C# and PCRE library sections, the open braces consume their own lines, and also are indented inconsistently, as are the close braces, which makes the code less readable, as well as less consistent among the sections.
Some readers may fault the book's sparse index. Admittedly, an inadequate index in any sizable programming book can make it difficult if not impossible to find what one is looking for. As a result, one ends up flipping through the book's pages hoping to luckily spot the desired topic. This is the rather unpleasant method to which a reader must resort when a technical book has no index, or one that is inadequate — which is far too often the case. Stubblebine's index offers only several dozen entries for all the letters of the alphabet, and only two symbols. Some readers might demand that all of the metacharacters and metasequences be listed in the index, so they can be found even faster than otherwise. But given the large number of metacharacters and metasequences, as well as method names, module functions, and everything else relevant, creating an exhaustive index would almost double the size of the book, and be largely redundant with the language-specific sections. Within each language, there is typically a limited enough number of pages that scanning through them to find a particular topic, would not be onerous. On the other hand, some of the index's inclusions and omissions are odd. For instance, two symbols are listed, and yet no others; why bother with those two? Also, a few key concepts are missing, such as grouping and capturing.
Yet aside from these minor blemishes, Regular Expression Pocket Reference is a concise, well-written, and information-rich resource that should be kept on hand by any busy software developer.
Michael J. Ross is a Web developer, writer, and freelance editor.
You can purchase Regular Expression Pocket Reference, Second Edition from amazon.com. Slashdot welcomes readers' book reviews -- to see your own review here, read the book review guidelines, then visit the submission page.
Before discussing this particular book, it may be of value to briefly discuss the essential concept of regular expressions, for the benefit of any readers who are not familiar with them. As noted earlier, a regular expression (frequently termed a "regex") is a string of characters intended for matching substrings in a block of text. A regex pattern can match literally, such as the pattern "book" matching both "book" and "bookshelf." A pattern can also use special characters and character combinations — often termed metasymbols and metasequences — such as \w to indicate a single word character (A-Z, a-z, 0-9, or '_'). Thus, the regex "b\w\wk" would match "book," but not "brook."
Here is a simple example to show the use of regexes in code, written in Perl: The statement "$text =~ m/book/;" would find the first instance of the string "book" inside the scalar variable $text, which presumably contains some text. To substitute all instances of the string with the word "publication," you could use the statement "$text =~ s/book/publication/g;" ('g' for globally search) or use "$text =~ s/bo{2}k/publication/g;". In this simplistic example, the second statement makes use of a quantifier, {2}, indicating two of the preceding letter.
These examples employ only one metacharacter (\w) and one quantifier ({2}). The total number of metacharacters, metasymbols, quantifiers, character classes, and assertions (to say nothing of capturing, clustering, and alternation) that are available, in most regex-enabled languages, is tremendous. However, the same cannot be said for the readability of all but the simplest regular expressions — especially lengthy ones not improved by whitespace and comments. As a consequence, when using regexes in their code, many programmers find themselves repeatedly consulting reference materials that do not focus on regular expressions. These resources comprise convoluted Perl books, incomplete tutorials on the Internet, and confusing discussions in technical newsgroups. For too many years, there was no published book providing the details of regexes for the various languages that utilize them, in addition to a clear explanation of how to use regexes wisely.
Fortunately, O'Reilly Media offers two titles in hopes of meeting that need: Mastering Regular Expressions, by Jeffrey Friedl, and Regular Expression Pocket Reference, by Tony Stubblebine. In several respects, the books are related — particularly in that Stubblebine bases his slender monograph upon Friedl's larger and more extensive title, justifiably characterized by Stubblebine as "the definitive work on the subject." In addition, Stubblebine's book follows the structure of Friedl's book, and contains page references to the same. Another major difference is that Regular Expression Pocket Reference is, just as the title indicates, for reference purposes only, and not intended as a tutorial.
At first glance, it is clear that Stubblebine's book packs a great deal of information into its modest 126 pages. That may partly be a result of the terseness of most, if not all, of the regular expression syntax; a metasymbol of more than two characters would be considered long-winded! Yet the high information density is likely also due to the manner in which Stubblebine has distilled the operators and rules, as well as the meaning and usage thereof, down to the bare bones. But this does not imply that the book is bereft of examples. Most of the sections contain at least one, and sometimes several, code fragments that illustrate the regex elements under discussion.
The book begins with a brief introduction to regexes and pattern matching, followed by an even briefer cookbook section, with Perl-style regexes for a dozen commonly-needed tasks, e.g., validating dates. The bulk of the book's material is divided into 11 sections, each one devoted to the usage of regexes within a particular language, application, or library: Perl 5.8, Java,.NET and C#, PHP, Python, Ruby, JavaScript, PCRE, the Apache Web server, the vi programmer's editor, and shell tools.
Each of these sections begins with a brief overview of how regexes fit into the overall language covered in that section. Following this is a subsection listing all of the supported metacharacters, with a summary of their meanings, in tabular format. In most cases, this is followed by a subsection showing the usage of those metacharacters — either in the form of operators or pattern-matching functions, depending upon how regular expressions are used within that language. Next is a subsection providing several examples, which is often the first material that most programmers turn to when trying to quickly figure out how to use one aspect of a language. Each section concludes with a short listing of other resources related to regexes for that particular language.
There are no glaring problems in this book, and I can only assume that all of the regular expressions themselves have been tested by the author and by previous readers. However, there is a minor weakness that should be pointed out, and could be corrected in the next edition. In most of the sections' examples, Stubblebine wisely formats the code so that every left brace ("{") is on the same line as the beginning of the statement that uses that brace, and each closing brace ("}") is lined up directly underneath the first character of the statement. This format saves space and makes it easier to match up the statement with its corresponding close brace. However, in the.NET / C# and PCRE library sections, the open braces consume their own lines, and also are indented inconsistently, as are the close braces, which makes the code less readable, as well as less consistent among the sections.
Some readers may fault the book's sparse index. Admittedly, an inadequate index in any sizable programming book can make it difficult if not impossible to find what one is looking for. As a result, one ends up flipping through the book's pages hoping to luckily spot the desired topic. This is the rather unpleasant method to which a reader must resort when a technical book has no index, or one that is inadequate — which is far too often the case. Stubblebine's index offers only several dozen entries for all the letters of the alphabet, and only two symbols. Some readers might demand that all of the metacharacters and metasequences be listed in the index, so they can be found even faster than otherwise. But given the large number of metacharacters and metasequences, as well as method names, module functions, and everything else relevant, creating an exhaustive index would almost double the size of the book, and be largely redundant with the language-specific sections. Within each language, there is typically a limited enough number of pages that scanning through them to find a particular topic, would not be onerous. On the other hand, some of the index's inclusions and omissions are odd. For instance, two symbols are listed, and yet no others; why bother with those two? Also, a few key concepts are missing, such as grouping and capturing.
Yet aside from these minor blemishes, Regular Expression Pocket Reference is a concise, well-written, and information-rich resource that should be kept on hand by any busy software developer.
Michael J. Ross is a Web developer, writer, and freelance editor.
You can purchase Regular Expression Pocket Reference, Second Edition from amazon.com. Slashdot welcomes readers' book reviews -- to see your own review here, read the book review guidelines, then visit the submission page.
This discussion has been archived.
No new comments can be posted.
The Fine Print: The following comments are owned by whoever posted them. We are not responsible for them in any way.
Full
Abbreviated
Hidden
Loading... please wait.
General introductions to regex? (Score:2)
Re: (Score:2, Informative)
Re: (Score:2)
Re:General introductions to regex? (Score:5, Informative)
http://www.regular-expressions.info/ [regular-expressions.info]
Parent
Re:General introductions to regex? (Score:5, Informative)
http://laurent.riesterer.free.fr/regexp/ [riesterer.free.fr]
It will color code your regex pattern and the associated matches in the string to be searched so you know what is matching what.
Parent
Re: (Score:3, Funny)
And for the Mac: RegExhibit (Score:5, Informative)
Here's one for the Mac:
http://homepage.mac.com/roger_jolly/software/index.html#regexhibit [mac.com]
Parent
Re: (Score:2, Informative)
Urgh, no. I just had a look at the site, and any site with gems like this right on the front page should definitely be avoided:
Checkin
Re: (Score:3, Funny)
[mini-me:/] luser% grep dead trees
grep: trees: No such file or directory
Re: (Score:2)
Re: (Score:2)
Re:General introductions to regex? (Score:5, Insightful)
Unless you're a hard core mathhead, that's probably not a good place to start with regexes IMHO. That's just going to scare people off from a highly useful tool. One generally does not need to rigorously prove that his regexes are going to work to use them. One does not have to use every feature of a language to make good use of it.
Parent
Re: (Score:2)
Re: (Score:2)
Then you go on and explain wildcards, character classes, and subexpressions and you've covered 95% of what a regular person will use in day to day life, all in the span of about 5 minutes. The hardest part about using regular expressions is usually se
Re: (Score:2)
'course, taking a course in formal language theory is even better (and should be a required course as part of a computing science degree, IMHO).
Re: (Score:2)
Same here.. people here advocating all sorts of wierd stuff like advanced maths theory*, when anyone could work out regular expressions by looking at them for a few minutes. Of course visualisers help for the really complex stuff (which nobody ever uses anyway).
PCRE is actually quite nice - you only have to bother with the setup once. just make a class that you can chuck a regexp string at and reuse it. Depends on the data set I gu
"Regular Expressions for Onion Routing" (Score:2)
is it better than (Score:2)
Re: (Score:2)
Pair up Google with something like Kodos [sourceforge.net] and you are all set. I still struggle with them sometimes, but nothing like before I had the debugger.
Stand back, everyone (Score:5, Funny)
Re:Stand back, everyone (Score:5, Funny)
Parent
useful regular expression (Score:5, Funny)
"Hello, I'm a smart geeky person, please to not beat me up and take my lunch money. I can help you with your math homework"
Sheldon
Re: (Score:3, Funny)
Now you have math problems.
Re:useful regular expression (Score:4, Informative)
For those of you who don't know the reference:
Some people, when confronted with a problem, think "I know, I'll use regular expressions."
Now they have two problems. --Jamie Zawinski, in comp.lang.emacs
Parent
Correction for summary (Score:5, Funny)
A minor correction:
However, there is a minor weakness that should be pointed out, and could be corrected in the next edition. Specifically, the book includes a section on
Re: (Score:2)
ObJWZ (Score:5, Funny)
Because you just can't discuss regular expressions without bringing up this quote [regex.info]:
Some people, when confronted with a problem, think "I know, I'll use regular expressions."
Now they have two problems.
-- Jamie Zawinski, 1997, in alt.religion.emacs
Re: (Score:2)
Re: (Score:2)
Despite being less self-documenting, I have never met a person who prefers to type [[:alnum:]] over \w.
already built in (Score:3, Informative)
There's already a built in regular expression tutorial:
man perlretutRe: (Score:2)
Re: (Score:2)
There's already a built in regular expression tutorial:
man perlretutWhy bother with a book? (Score:2)
Opening the Help file for the app and using its search function is a lot quicker than having to leaf through a book (worse when the book has a ba
Problems (Score:3, Interesting)
Some people, when confronted with a problem, think "I know, I'll use regular expressions." Now they have two problems. --Jamie Zawinski, in comp.lang.emacs
I'll close with a somewhat depressing fact: Regular expression and string processing can be done quickly and efficiently (and was done that way back decades ago, with grep and awk), but is actually done in a horribly inefficient way [swtch.com] in all modern/popular programming language regexp engines.
Re:Problems (Score:5, Interesting)
As for your second complaint... uhh, who cares? Premature optimization is the devil. So if regex's allow you to cleanly implement a simple solution to a problem (and regexes *are* very well suited to certain tasks, even if they do tend to be misused, particularly in languages such as Perl where they're very tightly integrated), it would be foolish to move to another technique based solely on performance concerns without first profiling the code.
'course, the real irony, on the performance front, is that Mr. Zawinski himself said "The heavy use of regexps in Emacs is due almost entirely to performance issues: because of implementation details, Emacs code that uses regexps will almost always run faster than code that uses more traditional control structures." So maybe they aren't so evil or slow after all?
Parent
Perl, regexps (Score:3, Interesting)
"Premature optimization" is a nice slogan - but the regexp performance problems are real, and I have encountered them before (I was extremely surprised to see that the regexp matching is scaling far worse than O(N) as it was clear to me that matching that regexp should be at worst O(N)).
The
Re: (Score:3, Interesting)
That's all well and good, but unless you're parsing extremely large volumes of text, the issues are probably unimportant. Which is, of course, why profiling is so important. Throwing out a perfectly valid solution simply because it is, in theory (or even in practice) slow, is ridiculous if you have other performance problems elsewhere, or if the code is running at a speed that is sufficient for the problem at hand.
Put another
Re: (Score:2)
Regular Expressions & Literate Programming (Score:2)
Re: (Score:2)
I have to get one of these (Score:3, Insightful)
I'd rather stick knitting needles in my eyes than debug a regular expression.
The only cure for that is getting a good reference and having a go at some tutorials until you get good enough to slay the beast. Then you'll be everyone's buddy at the office, because a lot of people feel the same way.
Or you could just stick knitting needles in your eyes and slash your face with a razor and then everyone will leave you alone.
Re: (Score:2)
I'd rather stick knitting needles in my eyes than debug a regular expression.
The only cure for that is getting a good reference and having a go at some tutorials until you get good enough to slay the beast. Then you'll be everyone's buddy at the office, because a lot of people feel the same way.
It's not all that hard if you format them properly with /x first. [perl.com]
:)
If you're using Perl anyway, dunno if others have that feature too. I don't need anything but Perl
Re: (Score:2)
Why do you say that? [ex-parrot.com]
A unanswered question (Score:2)
1) "regh-ex" (hard 'g', like 'ghost')
2) "rej-ex" (soft 'g', like 'gerbil')
3) "re-gex" (hard 'g')
4) "re-jex" (soft 'g')
I use the first one, since those are the two initial syllables of 'regular' and 'expression', but I can see arguments for the others.
Re: (Score:2)
#2 - I don't think of it as the combination of two words, which is probably why I ignore the hard G from 'regular' and replace it with the soft G from 'register'. I do the same with Linux (lin ucks), char (chair), etc.
Although I admit I can't pronounce the word "debacle" correctly for the life of me.
Don't fear - Just download txt2regex (Score:4, Informative)
Therefore I think the best way to learn regular expressions is by example. I highly recommend this small interactive program which will walk you through building regular expressions for a few different languagues. When you think you need a regex for a program, just fire it up and answer the questions.
http://txt2regex.sourceforge.net/ [sourceforge.net]
After a while you won't need txt2regex for simple stuff because you will have hopefully just absorbed the syntax. Once you have mastered the basic regexes which txt2regex can generate you will be able to dive into more advanced topics like capturing groups.
Missing applications (Score:2)
What is the concept of a regular expression? (Score:3, Informative)
A regular expression can be thought of as a program which generates a set of strings - or recognizes a set of strings, which is the same thing. Regular expressions correspond to finite state automatons, so just as a FSA cannot recognize the set of all palindromes, neither can a regular expression. Also languages like perl have extended the capabilities of their regular expression string matchers to include things like backreferences, which cannot be done in a true regular expression, so we tend to use the word 'regexp' nowadays.
Or perhaps I'm just playing the grumpy computer scientist here.
Re: (Score:3, Informative)
Re: (Score:2)