Washington Data Scraping API - Anyone Else Doing This?

Question

My original plan was to invest solely in Snohomish and King County Washington.  I wanted my custom CRM (built with Podio) to have the ability to auto-populate property data found on the county assessor site.  This drove me to create an API endpoint written in Python that would do this on demand.  I've met others who build code like this and store the data in tables to be queried later, but then you're stuck with older data.

Is anyone else using property scraping solutions on a large or small scale?  As my company grows, I think I will start using a company like ATTOM Data to provide property data so I don't have to continue to coordinate the build and maintenance of scraping tools.  Fellow nerds let me know what you're up to!

Gavin D. · Answer

Yep... I've got bs full of assessor/auditor/recorder data. To keep from having older data, you use "insert on duplicate key update" queries and continually loop over all the sources data..... i created a threaded version of rolling curl in php that keeps about 150 curl threads running async on targets which collects up to about 230 html pages per second.. which pulls down every property large counties like dekalb pretty fast...

Confused though.... What did you create the API endpoint for? and where? isn't your script scraping and writing to Podio API?

If ATTOM has you county that works... Plenty of counties are missing though. custom scrape is the only way to get everything for those....

John Williams · Answer

Is it possible to scrape Zillow?

Trevor Ewen · Answer

@Gavin D.

I don't think I was advocating statewide projects. Specificity is the name of the game for me.

Gavin D. · Answer

@Trevor Ewen  You can not build a process for scraping a whole state....  You have to build individual scrapes for each individual county in a state, with one exception.  When multiple utilize the same software to house, and publish auditor/assessor data, they usually publish the data for all counties with the same layout/search methods... QPublic is a good example.   So you may pick off a group of 10+ counties in a state with one template, however, you still have all of the individually published counties to worry about, not to mention any small counties (low population) that do not publish at all.  To add to the problem of writing multiple scrapes for the collection of the data from html pages, you also have to negotiate the search methods. Meaning they do not publish a list of every property, you must search for properties....some by street address, some by owner name, sometimes only by tax map number.  As well, most counties limit the results returned by the property search on the site. My scrapes take into consideration the max search results, and iterate through alphanumeric character sets...

Lets say the county allows searching by street name and limits search results to 50.  The basic pattern that my search code performs is as follows:  1st search starts with "A" if there are more than 50 results for just street names beginning with the letter "A" (or containing the letter "A" in some cases), another search is performed for "AA", if there are less than 50 results, the links, and then the linked page data is collected, the next search performed is for "AB".   If there is less than 50 results, again, data is collected, and then on to the next search "AC". if 50+ results are found, a new search for "ABA" is performed.My actual process takes into consideration spaces, periods, hyphens (for example St. James Ave.)  as well as street number (when applicable) and  works the same way when it is a property owner name search.  Tax map numbers are usually easier because you can iterate through the alpha numeric characters...but, you gonna have a lot of "no result" results which takes up time.

Next,as the auditor and the assesor are 2 separate offices more often then not, the tax data, and the assesor data are located on 2 or more separate pages.  Luckily one page usually has a link to the other which is simple enough to follow, but that means two (sometimes more) scrape scripts have to be written to consume the published data.Don't get me started about proxies....if you are gonna scrape any substantial amount of data, you better be rolling those page requests through a couple hundred proxy servers, and keep your requests and proxies in line.  Any county that uses .NET and ASP is going to ban an IP address really quick when your script skips one generated cookie. Not to mention cloudlflare scrape mitigation, captchas, and Javascript generated XHR validation post variables.  So get a deathbycaptcha account, and be ready to scrape using multiple languages and headless chrome, selenium is slow, scrapy....learn python, php is fastest but wont solve some problems, and you better know php really really well.To further complicate the process, many counties collect/publish very little information about the property construction.  Some counties assessor's are only required to calculate building count, sqft of liveable space, sqft of lot, or acreage.  Sometimes, they do not even note that building is stick-built as opposed to manufactured housing or a mobile home.  So discerning that a property is a ranch for sure is not possible without data enrichment from some other source.  When this happens you can use zillow api to fill in some blanks (assuming zillow has the data for that county) but you are limited to 1k API calls per day.  In a county with 1 million properties, that scrape is not getting done for a while....  Your other option, scrape realtor.com's site.  If you are looking for complete data, this comes with new set of issues.  Realtors enter most of the data about the property so, you have to account for abbreviations, term variation, and heavy text filtering for a slew of non-noteworthy amenities that "Christa" the Realtor's new assistant entered about the property while trying to be detail oriented like "Front Door", "Broken Toilet", "Roof", and "Located only .4 miles from Dollar Store" --actual examples I had to write regex's for.Now,  what I just laid out is an almost...lol.. almost..worst case scenario.  For the larger counties in a state, you may only run into 1 of those issues. But, I would be willing to bet dollars to dollar store donuts, that I can produce at least one example county per state that requires that level of commitment.  Well, I think maybe NJ may be one that may be entirely on the same platform.   Even NY, you have to contend with each borough in some counties.  Oh.....And not LA county CA...not do-able....well not with out a crap ton of brute force, last time i checked which was a couple years ago, viewing the property data/tax data required a code that is printed on the actual mailed tax bill that is mailed to the property owner.

All in all, scraping a whole state.... not do-doable unless you have a team big enough to account for each scrape....   Case in point, my county, even zillow has bad data.   The only way to gather that type of information is to have people (ie.. consumers or realtors) enter it... just as zillow does.

Here is a hint however, that may point someone looking to dig a little deeper, in the right direction...  National Address Database. (Publically available, opens source DB, for postal use)    Pitfalls.... its really only 23 states, only contains mailable addresses (no lots/acreage), and you better be able to write a solid parsing script and sql queries, and ready to manage 80gb of databases unless you don't mind spending $1k / month for a cloud server with the nuts to handle a single 80GB db.

Trevor Ewen · Answer

@Bryce DeCora

I have worked with several investors building scrapers for specific purchase / research projects. The key thing I have learned is that specificity is important. For instance, it's very hard to build a decent process for 'Single Family Homes in State X', the net is very wide, and the data analysis is just as hard after the scraping.

Focusing on '3 BR, 2 Bath Ranch Homes in A- School Districts' is a much better way to build a representational set. You can then compare them against one another to get a good idea of values in the set.

There are a ton of niche submarkets that do not get their share of analysis. I would say this is the best way to focus your efforts.

Bryce DeCora · Answer

@Gavin D. I had no idea ATTOM was missing a substantial amount of county data!  I guess I would have to manage a team to build scraping solutions for the gaps then.

My local import script calls other endpoints that are hosted on AWS to do different things.  I have one endpoint that scrapes county data and one that takes data and imports it to Podio.  I built endpoints instead of running these locally because I have several cloud applications that want access to this function.  Having an endpoint makes it more accessible as a tool. Hopefully that makes sense.

Washington Data Scraping API - Anyone Else Doing This?

Most Popular Reply