Investment Statistical Analysis

[Matt Jankowski]

Most advice on BP and all other investor groups focuses on individual experience, anecdotal knowledge, and common wisdom. These things are great, and I have the utmost gratitude for all those who share.

Nonetheless, I have a dream of reaching a higher level of knowledge about the investment decisions I make through applying sophisticated statistics to large data sets. Unfortunately, I'm not sophisticated, nor am I a statistician.

I know I'm surely not the first to think along these lines. Can anyone point me in the right direction for more info on the idea?

Any BP Stats PhD's out there that want to collaborate for fun and profit?

[Jon Holdman]

What you're proposing is exactly what zillow is trying to do. I do believe this is possible, at least theoretically. This is an application of "super crunching". See Super Crunchers, and The End of Theory. The Watson program that won on Jeopardy is another example. Also The Creative Destruction of Medicine. All these are making arguments for the application of statstical analysis of massive datasets in order to find relationships that a person can't easily spot. "The End of Theory" argues its no longer necessary to apply the scientific method by coming up with a hypothesis and then designing experiments to test the hypothesis. Instead you can test the hypothesis by mining existing data for evidence to support or refute the hypothesis. Since I'm pretty sure nobody here has much involvement with massive numbers of data tapes, I'll also throw out that my day job just released a tape analytics product I designed for performing this same sort of analysis on data tapes and drives to detect potential problems before they result in data loss.

The challenge for what you describe, Matt, is data. It should be relatively trivial to look for a relationship between rents and the type of applicances. If you had the data. Even getting rent data is not easy. I don't report actual rent data to anyone. I suspect many other small landlords don't either. You might get some data from from ads, but I've only used a yard sign for the last couple of years. And there's certainly not much information about the kind of appliances in most houses. At best, someone could look at pictures in ads or read text to identify the appliances. So, the challenge is going to be finding data to use for the analysis.

This might lend itself to some sort of crowdsourcing. That is, you have a way for prospective tenants to report information about what they're seeing when they're looking for a place. You would likely have to offer some small compensation. Similarly for ads. You could have some sort of browser that allows people to put in info about units based on what they see in ads. Even then you may have a data quality issue. Ads are inconsistent and now always even correct. And the people doing the data entry may make mistakes.