Do-it-yourself Screen Scraping/ Data mining

[Kenneth E.]

Hi all. I am posting this info about this program because I couldn’t find it by searching in the forums search box, so maybe it will help someone who happens to be lost ……where I got stuck.

If you have access to a public website and need to ‘capture’ the data from it to create your database or spreadsheet for your leads (assuming your county doesn’t already offer the information in a spreadsheet format), the keywords you need to learn are “data mining,” “screen scraping”, and “data extraction” (I had never heard of either term to describe capturing info from a website until I went to craigslist and found someone under the ‘services’ section who offered IT services….and explained what I needed. They in turn taught me what it’s called which was the first step! Ha). These are the processes of electronically combing through websites and snatching specific data from them.

You can hire people to do this type of process for you, but if you are a small-time operation like I am, I ended up going with a service called Mozenda (www.mozenda.com). It is a program (you download to your computer) that allows you to customize it and how it clicks through the pages on a website and captures the data you need…on multiple pages. It’s a bit complicated to set up but the customer service people were SUPER helpful. Once it was set up, I pressed ‘go’ and turned it loose for the next few hours. It combed through numerous public records and pulled only the data I told it to. It did all the rest and put the results into spreadsheet format for me, which I then used to create mailers with (mail-merge, anyone?). :)

I recommend it because it worked for me. But, keep in mind the two disadvantages: 1. It can get costly if you have thousands of pages to take data from, and 2. It is a bit complicated to set up.

I understand that a program called Python can do something similar and cheaper….but it may require some programming experience (though you can probably hire some IT person through craigslist to do the programming for you).

Also, I have no affiliation with mozenda at all. Just trying to help others out.

[Joshua Dorkin]

Kenneth Elshoff - Scraping data or content from websites, sometimes dubbed as "Automated Data Collection" is likely a violation of their terms of use.

Ann Bellamy - To continue my point above, just check out the TOS of the site you mentioned and you get the following:

You can certainly see that they would not be fond of scrapers.

I STRONGLY advise against scraping content. It is a fast way to find yourself in court. If a website is going to make their content available for use, they are likely to do so via an API.