Big Data in Real Estate

Big Data in Real Estate

Title Representative · Lakewood, OH · Member since 2015 · 121 posts · 70 votes

In today's post, I want to use experience I've had in two fields that I really love: databases and real estate. Specifically, with databases I want to go over new techniques using more data and varieties of data as it pertains to real estate. First of all, many companies store data using Excel spreadsheets, flat files, or Quickbooks and there's nothing wrong with these techniques, until there's something wrong with them. These tools suffer to scale very well, which means that as you grow your business you will find it increasingly difficult to manually input sales leads, employees, contractors, accounting, business processes, and so on. However, you have to start somewhere and you can migrate this data to more robust solutions later on. Deciding on what tool to use is important, whether to rent using SaaS or buy is another important decision, but deciding what kind of company you want to be is the most important question to ask yourself.

What kind of data is important to store? Maybe what you think is important today won't be important tomorrow or vice versa. Not all the data you want to know or store are numbers that easily fit into spreadsheets. Are there competitive advantages that can be gained from crunching numbers in real estate? Yes. Does it tell the whole story or the most important part of the story? No. I argue that the numbers, or the "cash flow" is less important than most investors believe it is. With some effort, anyone can collect public available data like: listing prices, sold prices, square feet, year built, rent amount, taxes, etc. What kinds of data are less accessible? How about perspective tenants phone calls in that area? Their jobs? Their criminal records? Their rental history? How do you measure all of that?

What about a home inspection report? I would argue that the condition of the home is far more important than the cash flow. Although, I disagree with the gloom and doom nature of many home inspection reports, i.e. the house is not falling over tomorrow, the condition of an individual home matters far more than the numbers you put into a software program. Mother nature and individual properties do not care about your numbers, nor do your tenants. All of whom, can and will alter those numbers and distort your cash flow.

I just provided some examples of a variety of data that cannot be captured in Relational Databases, like Excel. Pictures, text files, videos have to be stored in NoSQL databases. Quantifying non-structured data is possible using metadata and analysts in order to make analysis easier and more meaningful for decision-makers. Why is any of this important?

You can definitely do business without any of this and many do business without any of these tools. However, you are doing business without capturing a more complete picture of the surrounding landscape. Are you just guessing? For example, what if one company established a tenant call line and they could accept or deny the tenant just based on their voice? What if you could store video for every single property and never show the property to tenants or buyers? What if you could measure air temperature every 10 seconds to minimize heating and cooling costs? What if you could measure water use every 10 seconds and identify tenants that are abusing water? What if you could know the condition of the property without getting off your sofa? Aggregating structured and non-structured data and knowing what to do with it has been the same challenge we faced since the beginning of time. Now, it's just about harnessing our capabilities, tools, and the data to make great decisions.

Data can choke you and drown you if you do not understand it, cause you to pay attention to wrong kind of data, or ignore the most important kinds of data. I think of each of us as investors as ships (some bigger, some smaller) sailing along in a sea of data with all of its variety, volume, and velocity and we have to make decisions along the way. Make the right decisions and you end up on a beautiful island drinking Pina Coladas (I'd prefer Scotch, actually) or make the wrong ones and end up in Davey Jones' locker. Happy Investing! 

4Reply
67 views

Most Popular Reply

Kent, WA · Member since 2015 · 10 posts · 4 votes
10y

If you truly wanted to get into RE big data mining, you could consider natural language classification techniques to see what patterns exist and if there are words/groups of words that predict good deals. 

You could also look at time series data for listing prices to make a statistical prediction on pricing bands which would help you make an optimal winning offer before a price drop causes an influx of competitor buyers. 

There's truly a lot out there in the realm of big data. My day job is in a field where traditional big data solutions (Hadoop on computer clusters) isn't even efficient enough, which has put the focus on resilient data structures (RDD's) and Apache Spark as a foundation for alternative solutions. A bit overkill for what you were mentioning, but it's interesting to think about!

Happy to chat more, and as always feel free to connect!

See this reply in the discussion

17 Replies

Jump to latestLatest
  • Investor · Douglasville, GA · Member since 2014 · 313 posts · 181 votes
    10y

    @Nicholas Varner

    Are you hinting that you are building a NoSQL DB to do some of the things you talked about, or just looking for other ideas that make sense?

    I was just recently introduced to MongoDB where I work, and thought this would be perfect for storing property/area/contact info when I finally get around to doing my direct mail campaign.  It is very scalable, free form, and most importantly; FREE.

    I plan to setup a Linux server in my home office to host the DB, and if all goes well, I might migrate my properties over to it from quickbooks for some serious analysis.  I could put in the age of the mechanicals in each property, along with their life expectancy, to get a better handle on how much I should be putting away each month for capex.  I could track Rentals in the area to see if rents are  on an up or down trend.  I could track wholesaler activity to see which areas are being hit hard (not sure how that might help me,  but it would be interesting).

  • Jon HuberPro Member
    Rental Property Investor · Boca Raton, FL · Member since 2014 · 1k+ posts · 713 votes
    10y

    @Tim Lindstrom You took the question right from me...

    @Nicholas Varner  I'm a Big Data geek myself. My 9-5 is essentially that. I've used some of the data to create my own systems, and they have worked thus far. I am always improving though. A lot of what I get comes from city-data.com. Great info for data geeks, and it's free. 

    We can toss ideas back and forth in this forum.

    One great idea, that I haven't set up yet but am intrigued to... is using RSS feeds from MLS sites to import data into SQL, thus creating ones own db that can be queried.

  • Investor · Boulder, CO · Member since 2015 · 137 posts · 36 votes
    10y

    Hi Nicholas

    I know Big Data (Hadoop) , RDBMS, and REI very well and have professional experience in all.

    You bring up some great points regarding trapping and aggregating data for use in RE. There is a ton of room for IT growth within the RE industry. I know agents who still shun computers and put pen to paper when possible.

    Feel free to connect and we can chat in more detail.

  • Title Representative · Lakewood, OH · Member since 2015 · 121 posts · 70 votes
    10y

    @Tim Lindstrom. Free is big! Cassandra or MongoDB are pretty amazing leaps forward. AWS or Google BigQuery is where I've had more experience, but there's overlap between many of the NoSQL technologies. I would be very interested in building my own NoSQL database to track things like website visits to our page, user-experience on our site, and then building out databases of inspection reports and videos in the markets that I work the most as a realtor. Obviously, possibilities really are endless here. I think dealing with a lot of tenants, there are great applications to reduce time wasted with flakey folks. So many ideas, so little time! 

  • Title Representative · Lakewood, OH · Member since 2015 · 121 posts · 70 votes
    10y

    @Bill Florence I would love to hear your ideas. 

  • Title Representative · Lakewood, OH · Member since 2015 · 121 posts · 70 votes
    10y

    @Jon Huber great idea about regarding MLS data integration. I should reach out to my MLS (NE Ohio) and see what their policy on exporting data. Currently, they limit the amount of exporting one can do for free per month. One thing that is tough is knowing when you are being insane with analysis about stuff that could be considered "cool" but not necessarily practical or important as far as investing is concerned. But part of growing in the field is trying new ideas.

  • Jon HuberPro Member
    Rental Property Investor · Boca Raton, FL · Member since 2014 · 1k+ posts · 713 votes
    10y

    @Nicholas Varner I know most MLS policies do not allow mass export to a spreadsheet, however, I don't believe there would be a limit on an RSS feed. You'd have to confirm. I completely understand what you mean about where to draw the line of practicality. I have definitely created SQL dbs and systems for my own enjoyment that had no practical value outside of my own amusement. But who wouldn't want to use SQL to create an NBA starting lineup optimizer that ranked NBA players by my own personal scoring efficiency algorithm??

  • Buy & Hold Owner · Redlands, CA · Member since 2015 · 5k+ posts · 2k+ votes
    10y

    Guys;  MySQL is easily installed on any PC or Mac :)  In fact, so is the Apache server, PHP and Perl - - my laptop is my development platform for web services.

    As to what data is important or useful is GREATLY dependent upon the intended usage. As I am an investor with MFUs,  the cash flow is paramount to my decisions to walk or proceed.  Inspect data becomes important ONLY if I conclude I want to by, and then it's mostly ignored unless it's a death trap - - you can't get brand new buildings and expect to make money with them.  Most maintenance is easily amortized over time during which the cash flow is still occurring :)

    As a Realtor, significantly different data is need, starting with the details for the MLS entry - - DUH.

    Not to be forgotten, but a major contributor to RE is the NETWORKING contacts, just like here @BP.  That's called a CRM(Customer Relations Management) system with not just simple name, addr, phone, but a chron of who spoke to whom, when and about what.

  • Mike CumbieBusiness Member
    REALTOR® · Brockport, NY · Member since 2015 · 3k+ posts · 4k+ votes
    10y

    Hi @Nicholas Varner,

    I use to jump into a lot of the more "obscure" technology areas, back with MySQL and Redhat running Apache and using PHP for the engine. Then it got to much to keep up with MSSQL and IIS with the new MS server OS's for work, while trying to do my own projects in other areas (the old syntax errors that do not transfer over between things).

    I think it is best if you are going to be "Big Data Mining" to use the same apps that large businesses use (Sure use SQLExpress it's free but the queries can transfer). The reason being is you don't want to be the guy mining the data once you get it going, you are going to want to move onto something else and hire someone to perform those tasks. It is a lot easier to find someone who can plug right in (or to plug into another company and trade data) if you are using the same technology.

    Open source is fine, but if you want to be in the business of technology you are not going to be able to focus on REI when you are trying to make a "shared technology" work within your system. I know SQL won't scale like you are discussing, but there are ways to work with that within the current technologies. Example Have 1 DB full of all sorts of house statistics and another full of water readings (I do know that one DB can handle mind numbing amounts of data but we are talking global domination here). You could have web pages feeding queries to different ones and displaying the results together in one place or coding to combine the data. In the existing model you only need to convince the water department to let you plug in and viola you have the information. You can convince the county tax groups to let you make a connection. With a new DB idea you have to create a whole new system ask them for the data and have a method of importing it.

    Just my 2 cents but love the ideas.

  • Buy & Hold Owner · Redlands, CA · Member since 2015 · 5k+ posts · 2k+ votes
    10y

    @Mike Cumbie good point - - we need yo ask ourselves, "What is my business?" and then as they say "Mind the store".  Every moment spent in technology is a moment that could have been used to look, make, get deals or to improve property or management skills.

  • Kent, WA · Member since 2015 · 10 posts · 4 votes
    10y

    If you truly wanted to get into RE big data mining, you could consider natural language classification techniques to see what patterns exist and if there are words/groups of words that predict good deals. 

    You could also look at time series data for listing prices to make a statistical prediction on pricing bands which would help you make an optimal winning offer before a price drop causes an influx of competitor buyers. 

    There's truly a lot out there in the realm of big data. My day job is in a field where traditional big data solutions (Hadoop on computer clusters) isn't even efficient enough, which has put the focus on resilient data structures (RDD's) and Apache Spark as a foundation for alternative solutions. A bit overkill for what you were mentioning, but it's interesting to think about!

    Happy to chat more, and as always feel free to connect!

  • Title Representative · Lakewood, OH · Member since 2015 · 121 posts · 70 votes
    10y

    @Michael Verkruyse Too big for Hadoop? What the hell are you doing? haha. I like your natural language idea a lot. I was thinking about trying to scrape data from probate court about marriages, births, and deaths for marketing campaigns. 

  • Kent, WA · Member since 2015 · 10 posts · 4 votes
    10y

    Working to help provide search and real-time data mining/analytics on data lakes for large companies. The data isn't necessarily too big for Hadoop, but doing it in an efficient manner requires a better mechanism. Without diving too deep in the weeds, Apache Spark allows the RAM to be leveraged more than Hadoop/MapReduce, which allows real-time operations to be realistic.

    Scraping that data could be an interesting method for marketing!

  • Real Estate Investor · Del Mar, CA · Member since 2016 · 5 posts · 0 votes
    10y

    I love the big data play, but I think the bigger challenge is a mindset over improvement.  People in technology are afraid of big data or Hadoop,  many in real estate are still afraid of Wordpress or html.   

    The bigger play is using machine learning, because just storing a ton of data that somebody has to create some tableau like report isn't going to change a ton.  It just might improve the performance or way things are stored.   Then again half these big data companies are going to be gone in a few years, what happens if you picked the wrong one?   Go to any Hadoop conference from year to year or month to month and you start to notice a trend of some companies just disappeared or wound up acquired and disappeared.  

    That aspect is normal, just look at the BI market years back where the Cognos, Business Objects, Pentaho, JasperReports and so on are were bought up and in some cases just became a far less useful product.     And the big data space is even more vulnerable because most of these companies dont make any money.  Even Hortonworks stock is down to like $10 and they seem to be in the red every quarter.  

    It doesn't matter much if you have a server at home or using a virtual machine, but it sure would stink if all your data and analysis suddenly disappeared because you used some cool new big data and advanced analytics tool in the cloud, but it was their own SaaS cloud and not an Azure or AWS.

    You also have to be careful with the cloud, especially smaller shops and people who are afraid of computers.  There are many many examples of people suddenly getting a thousand dollar cloud bill.   It's easier to spin up and no maintenance costs, but not always cheaper.  Especially when you start processing tons of data and doing real-time analysis.  

    And we all have to worry about getting lost in the weeds because for many people that love technology or data science, many things you find useful or insightful or rather useless. Or even worse it's actually great, but your story and presentation stink so nobody understands it.  There are plenty of statisticians who have great info, but it's far too complicated for the average user to understand or read.  So it becomes worthless.  But it will be interesting to see some changes in RE.

  • Title Representative · Lakewood, OH · Member since 2015 · 121 posts · 70 votes
    10y

    @Bob Pattersonly I enjoyed your response. You bring up some good points about market players and data applications. Industry players are extremely vulnerable with high P/E ratios and growth projections that are optimistic. The industry is still a fledging industry. All you have to do is look at a stock chart of Tableau and that tells you the story. My winners in this space for the future are Microsoft, Amazon, SalesForce, and Google. Facebook could do some interesting things as well. I am not a big fan of Oracle shops. 

    Returning to RE applications. I just read an article the other day that described what the biggest indicator of price appreciation is in real estate. Out of the 5 variables tested, they determined that shadow inventory is the greatest single indicator of whether a market's prices will appreciate or depreciate. To me, how they came to that conclusion and whether or not it can be tested and proven are the types of ways that big data could have applications in real estate. However, whether or not spending the time to acquire, store, structure, and analyze massive amounts of data to determine actionable insights....the jury is still out on that one to me. 

  • Real Estate Investor · Columbia, MD · Member since 2014 · 12 posts · 1 vote
    9y

    Hello all.  I'm trying to find out the better way to construct a database using different data sources.  Not only to construct it but to handle the data processing steps in an automated fashion.  Part of this has to do with real property tax and ownership data.  What are the considerations and methods for taking a list of probate filings and cross-matching it with property records to produce a list of probate cases that show the person passing owned real estate?  I know I need the two data sets, but what are the data processing functions that combine these two data sets?  What is it called and how can it be done?  I don't want to have to do it manually.  thanks for your help.

  • Raleigh, NC · Member since 2018 · 46 posts · 34 votes
    7y

    Hey guys, I see that this post is a few years old.  But can anyone provide me with Lessons Learned on any action taken after this discussion?  I plan to cultivate my own "Big Data" farm to help grow and sustain my Wholesaling business.  I was planning to use Tableau/QlikView to give some visual insights into my data.  I would love to see or hear of any proofs of concept.

Join the conversationCreate a free account to reply, vote on answers and follow this thread.