Getting Started With Mechanize¶ ↑

This guide is meant to get you started using Mechanize. By the end of this guide, you should be able to fetch pages, click links, fill out and submit forms, scrape data, and many other hopefully useful things. This guide really just scratches the surface of what is available, but should be enough information to get you really going!

Let’s Fetch a Page!¶ ↑

First thing is first. Make sure that you’ve required mechanize and that you instantiate a new mechanize object:

require 'rubygems'
require 'mechanize'

agent = Mechanize.new

Now we’ll use the agent we’ve created to fetch a page. Let’s fetch google with our mechanize agent:

page = agent.get('http://google.com/')

What just happened? We told mechanize to go pick up google’s main page. Mechanize stored any cookies that were set, and followed any redirects that google may have sent. The agent gave us back a page that we can use to scrape data, find links to click, or find forms to fill out.

Next, let’s try finding some links to click.

Finding Links¶ ↑

Mechanize returns a page object whenever you get a page, post, or submit a form. When a page is fetched, the agent will parse the page and put a list of links on the page object.

Now that we’ve fetched google’s homepage, let’s try listing all of the links:

page.links.each do |link|
  puts link.text
end

We can list the links, but Mechanize gives a few shortcuts to help us find a link to click on. Let’s say we wanted to click the link whose text is ‘News’. Normally, we would have to do this:

page = agent.page.links.find { |l| l.text == 'News' }.click

But Mechanize gives us a shortcut. Instead we can say this:

page = agent.page.link_with(:text => 'News').click

That shortcut says “find all links with the name ‘News’”. You’re probably thinking “there could be multiple links with that text!”, and you would be correct! If you use the plural form, you can access the list. If you wanted to click on the second news link, you could do this:

agent.page.links_with(:text => 'News')[1].click

We can even find a link with a certain href like so:

page.link_with(:href => '/something')

Or chain them together to find a link with certain text and certain href:

page.link_with(:text => 'News', :href => '/something')

These shortcuts that Mechanize provides are available on any list that you can fetch like frames, iframes, or forms. Now that we know how to find and click links, let’s try something more complicated like filling out a form.

Filling Out Forms¶ ↑

Let’s continue with our google example. Here’s the code we have so far:

require 'rubygems'
require 'mechanize'

agent = Mechanize.new
page = agent.get('http://google.com/')

If we pretty print the page, we can see that there is one form named ‘f’, that has a couple buttons and a few fields:

pp page

Now that we know the name of the form, let’s fetch it off the page:

google_form = page.form('f')

Mechanize lets you access form input fields in a few different ways, but the most convenient is that you can access input fields as accessors on the object. So let’s set the form field named ‘q’ on the form to ‘ruby mechanize’:

google_form.q = 'ruby mechanize'

To make sure that we set the value, let’s pretty print the form, and you should see a line similar to this:

#<Mechanize::Field:0x1403488 @name="q", @value="ruby mechanize">

If you saw that the value of ‘q’ changed, you’re on the right track! Now we can submit the form and ‘press’ the submit button and print the results:

page = agent.submit(google_form, google_form.buttons.first)
pp page

What we just did was equivalent to putting text in the search field and clicking the ‘Google Search’ button. If we had submitted the form without a button, it would be like typing in the text field and hitting the return button.

Let’s take a look at the code all together:

require 'rubygems'
require 'mechanize'

agent = Mechanize.new
page = agent.get('http://google.com/')
google_form = page.form('f')
google_form.q = 'ruby mechanize'
page = agent.submit(google_form)
pp page

Before we go on to screen scraping, let’s take a look at forms a little more in depth. Unless you want to skip ahead!

Advanced Form Techniques¶ ↑

In this section, I want to touch on using the different types in input fields possible with a form. Password and textarea fields can be treated just like text input fields. Select fields are very similar to text fields, but they have many options associated with them. If you select one option, mechanize will de-select the other options (unless it is a multi select!).

For example, let’s select an option on a list:

form.field_with(:name => 'list').options[0].select

Now let’s take a look at checkboxes and radio buttons. To select a checkbox, just check it like this:

form.checkbox_with(:name => 'box').check

Radio buttons are very similar to checkboxes, but they know how to uncheck other radio buttons of the same name. Just check a radio button like you would a checkbox:

form.radiobuttons_with(:name => 'box')[1].check

Mechanize also makes file uploads easy! Just find the file upload field, and tell it what file name you want to upload:

form.file_uploads.first.file_name = "somefile.jpg"

Scraping Data¶ ↑

Mechanize uses nokogiri to parse HTML. What does this mean for you? You can treat a mechanize page like an nokogiri object. After you have used Mechanize to navigate to the page that you need to scrape, then scrape it using nokogiri methods:

agent.get('http://someurl.com/').search("p.posted")

The expression given to Mechanize::Page#search may be a CSS expression or an XPath expression:

agent.get('http://someurl.com/').search(".//p[@class='posted']")

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!