BeautifulSoup Intro

What is BeautifulSoup?

BeautifulSoup is a Python library from www.crummy.com 

What can it do

On their website they write "Beautiful Soup parses anything you give it, and does the tree traversal stuff for you. 

You can tell it to:

"Find all the links"

"Find all the links of class externalLink"

"Find all the links whose urls match "foo.com"

"Find the table heading that's got bold text, then give me that text."" 

BeautifulSoup Example

In this example, we will try and find a link (a tag) in a webpage. 

Before we start, we have to import two modules. (BeutifulSoup and urllib2). 

Urlib2 is used to open the URL we want. 

We will use the soup.findAll method to search through the soup object to match fortext and html tags within the page. 

from BeautifulSoup import BeautifulSoup
import urllib2

url = urllib2.urlopen("http://www.python.org")
content = url.read()
soup = BeautifulSoup(content)
links = soup.findAll("a")

Output

That will print out all the elements in python.org with an "a" tag. 

(The "a" tag defines a hyperlink, which is used to link from one page to another.)

BeautifulSoup Example 2

To make it a bit more useful, we can specify the URL's we want to return.  

from BeautifulSoup import BeautifulSoup
import urllib2
import re

url = urllib2.urlopen("http://www.python.org")
content = url.read()
soup = BeautifulSoup(content)
for a in soup.findAll('a',href=True):
    if re.findall('python', a['href']):
        print "Found the URL:", a['href']

Further Reading

I recommend that you head over to http://www.crummy.com to read more about what you can do with this awesome module.