MLS Crawler Tool To Summarize

[Sagiv O.]

Do you guys know any tool that crawls the MLS or sites like Zillow/Trulia and extracts information from it to excel or some sort of document with everything summarized?

Will be nice to set up some triggers (show me properties only below $x with y rooms and z ROI)

If not - would you need something like that? If so, how much will you pay for it?

[Ian Ippolito]

Thanks @Hau N., and yes, I do plan to write an article on it in the future. For you and @Sagiv O., here's more information:
1) the tool allows me to  select number of rooms, number of baths, square footage, zip codes to scan etc.

2) it then goes to Zillow to pull from the MLS. @Steve B. is correct that Zillow limits the number of requests you can make daily through their API, which essentially makes it useless for such a tool. So this means you have to do a technique called screen scraping to get by the limit. Unfortunately, this means every time Zillow changes their site the program breaks.  So it is quite a bit of maintenance to keep it running. Also, it means you have to do the requests kind of slowly so that Zillow doesn't kick you off the site. For example, when I scan all the ZIP Codes in my area of Tampa, it takes hours, so typically I run the program overnight and have the report ready for me in the morning. 

3) Some of the MLS properties don't meet the filters. But if they do, then the program looks further because there are still going to be way too many properties at this point. It ties into Trulia to get crime statistics, and then also filters on that. Again the same restrictions as Zillow with screen scraping and keeping it slow.

4) If it passes that test, then it ties into RentoMeter. This one is very important, because at this point there still too many of properties, in most are complete waste of time because they won't meet my yield requirements. So this part of the program does a rough estimate on how much I can expect to earn on the property, and checks that make sure it passes my filter. Ideally, it would be able to do a check on the property itself, but doing that again would overload RentOMeter and cause them to kick the tool off the site. So instead, I have to do it to pass test. At this stage, I just pull the data for the ZIP Code, and then store it and reuse it for every property in the ZIP Code. That greatly reduces the amount of data that has to be pulled from RentOMeter and weeds out 70% of the bad ones which is a huge improvement. Later in the process I do it on the specific property to weed out the rest.

5) The tool applies additional filters.

6) If it survives at this point, then it does a 2nd RentOMeter check on it, but this time on the property itself rather than the ZIP Code. This is a much more accurate check, and can be done at this stage because there are so few properties and it won't overload rent to meter and cause them to kick the tool off the site.

At this point, out of my entire Tampa area with maybe 20 ZIP Codes, I'll end up with a list of 5 or 6 properties that the tool spits out in Excel. Like I said earlier, I start the program before I go to bed and then when I wake up in the morning my report is there. It's like magic and each report says me many man months of time.

Developing the program wasn'tt easy. It took about 3 to 4 months to develop the initial tool and get it to the point where it was usable (not spitting out bad data or missing good properties) and not getting kicked off the sites. And at that point I thought I was done, but that was actually the easy part. The biggest challenge and cost of the tool is keeping it working, because of all the screen scraping. Every time 1 of the sites changes its format even slightly, the tool breaks and has to be reprogrammed. And they are constantly changing their format, which can drive you a little crazy (although it makes your program are very happy since it keeps them gainfully employed).