Regular Expressions in Python

What is a Regular Expression?

It’s a string pattern written in a compact syntax, that allows us to quickly check
whether a given string matches or contains a given pattern.

The power of regular expressions is that they can specify patterns, not just
fixed characters. Many examples in this articles can be found on:
Googles Python Course

Basic patterns

a, X, 9
ordinary characters just match themselves exactly.

. ^ $ * + ? { [ ] | ( )
meta-characters with special meanings (see below)

. (a period)
matches any single character except newline ‘n’

w
matches a “word” character: a letter or digit or underbar [a-zA-Z0-9_].
It only matches a single character not a whole word.

W
matches any non-word character.

w+
matches one or more words / characters

b
boundary between word and non-word

s
matches a single whitespace character, space, newline, return, tab, form

S
matches any non-whitespace character.

t, n, r
tab, newline, return

D
matches anything but a digit

d
matches a decimal digit [0-9]

d{1,5}
matches a digit between 1 and 5 in lengths.

{n} d{5}
matches for 5 digits in a row

^
match the start of the string

$
match the of the string end

*
matches 0 or more repetitions

?
matches 0 or 1 characters of whatever precedes it

use . to match a period or to match a slash.

If you are unsure if a character has special meaning, such as ‘@’, you can
put a slash in front of it, @, to make sure it is treated just as a character.

re.findall

The findall() is probably the single most powerful function in the re module
and we will use that function in this script.

In the example below we create a string that have a text with many email
addresses.

We then create a variable (emails) that will contain a list of all the found
email strings.

Lastly, we use a for loop that we can do something with for each email string
that is found.

str = 'purple [email protected], blah monkey [email protected] blah dishwasher'

## Here re.findall() returns a list of all the found email strings
emails = re.findall(r'[w.-]+@[w.-]+', str) ## ['[email protected]', '[email protected]']
  
for email in emails:
    # do something with each found email string
    print email

We can also apply this for files. If you have a file and want to iterate over
the lines of the file, just feed it into findall() and let it return a list of
all the matches in a single step

read() returns the whole text of a file in a single string.

(If you want to read more about file handling in Python, we have written a
‘Cheat Sheet’ that you can find here)

# Open file
f = open('test.txt', 'r')

# Feed the file text into findall(); it returns a list of all the found strings
strings = re.findall(r'some pattern', f.read())

re.search

The re.search() method takes a regular expression pattern and a string and
searches for that pattern within the string.

The syntax is re.search(pattern, string).

where:
pattern
regular expression to be matched.

string
the string which would be searched to match the pattern anywhere in the string.

It searches for first occurrence of RE pattern within string with optional flags.

If the search is successful, search() returns a match object or None otherwise.

Therefore, the search is usually immediately followed by an if-statement to test
if the search succeeded.

It is common to use the ‘r’ at the start of the pattern string, that designates
a python “raw” string which passes through backslashes without change which is
very handy for regular expressions.

This example searches for the pattern ‘word:’ followed by a 3 letter word.

The code match = re.search(pat, str) stores the search result in a variable
named “match”.

Then the if-statement tests the match, if true the search succeeded and
match.group() is the matching text (e.g. ‘word:cat’).

If the match is false, the search did not succeed, and there is no matching text.

str = 'an example word:cat!!'

match = re.search(r'word:www', str)

# If-statement after search() tests if it succeeded
  if match:                      
    print 'found', match.group() ## 'found word:cat'

  else:
    print 'did not find'

As you can see in the example below, I have used the | operator, which search for either pattern I specify.

import re
programming = ["Python", "Perl", "PHP", "C++"]

pat = "^B|^P|i$|H$"

for lang in programming:
    
    if re.search(pat,lang,re.IGNORECASE):
        print lang , "FOUND"
    
    else:
        print lang, "NOT FOUND"

The output of above script will be:

Python FOUND
Perl FOUND
PHP FOUND
C++ NOT FOUND

re.sub

The re.sub() function in the re module can be used to replace substrings.

The syntax for re.sub() is re.sub(pattern,repl,string).

That will replace the matches in string with repl.

In this example, I will replace all occurrences of the re pattern (“cool”)
in string (text) with repl (“good”).

import re
text = "Python for beginner is a very cool website"
pattern = re.sub("cool", "good", text)
print text2

Here is another example (taken from Googles Python class ) which searches for all
the email addresses, and changes them to keep the user (1) but have
yo-yo-dyne.com as the host.

str = 'purple [email protected], blah monkey [email protected] blah dishwasher' 

## re.sub(pat, replacement, str) -- returns new string with all replacements,

## 1 is group(1), 2 group(2) in the replacement

print re.sub(r'([w.-]+)@([w.-]+)', r'[email protected]', str)

## purple [email protected], blah monkey [email protected] blah dishwasher

re.compile

With the re.compile() function we can compile pattern into pattern objects,
which have methods for various operations such as searching for pattern matches
or performing string substitutions.

Let’s see two examples, using the re.compile() function.

The first example checks if the input from the user contains only letters,
spaces or . (no digits)

Any other character is not allowed.

import re

name_check = re.compile(r"[^A-Za-zs.]")

name = raw_input ("Please, enter your name: ")

while name_check.search(name):
    print "Please enter your name correctly!"
    name = raw_input ("Please, enter your name: ")

The second example checks if the input from the user contains only numbers,
parentheses, spaces or hyphen (no letters)

Any other character is not allowed

import re

phone_check = re.compile(r"[^0-9s-()]")

phone = raw_input ("Please, enter your phone: ")

while phone_check.search(phone):
    print "Please enter your phone correctly!"
    phone = raw_input ("Please, enter your phone: ")

The output of above script will be:

Please, enter your phone: s

Please enter your phone correctly!

It will continue to ask until you put in numbers only.

Find Email Domain in Address

Let’s end this article about regular expressions in Python with a neat script I
found on stackoverflow.

@
scan till you see this character

[w.]
a set of characters to potentially match, so w is all alphanumeric characters,
and the trailing period . adds to that set of characters.

+
one or more of the previous set.

Because this regex is matching the period character and every alphanumeric
after an @, it’ll match email domains even in the middle of sentences.

import re

s = 'My name is Conrad, and [email protected] is my email.'

domain = re.search("@[w.]+", s)

print domain.group()

outputs:
@gmail.com

More Reading

https://developers.google.com/edu/python/regular-expressions
http://www.doughellmann.com/PyMOTW/re/
http://www.daniweb.com/