Most wealthy investors are not purchasing SFH, they are in the commercial space, for those reasons I have mentioned above.  

The person who owned them was super wealthy, like near billionaire status iirc. 

As lower income folks leave, due to housing costs, higher income folks enter the state (retirement folks, CEOs, wealthy foreigners etc.) hence the cost of housing is increasing.

I do have access to private money (wealthy aunt offered to lend money.)

Secrets of the wealthy tell us Debt is tax free. 

And then do the smaller one because the 20% we'd have to put down would be smaller.Background - we're not wealthy!

Another thing you should do, depending on your wealth, is to form some type of corporation or a LLC.  

I can also tell you "The Bankers Code" and "The Wealthy Code" by a personal mentor of mine "George Antone" are two books everyone in this form should read.

-You live where your wealth is being generated, so you can keep a close eye on your investment. 

I could never in a 1000 years think that free and clear property is a bad thing.. the most wealthy of my clients or personal friends and or acquaintances have a majority of their holdings with no debt.