Talk to them and let them know that if they have that many concerns, perhaps it isn't a good fit and you're happy to give them all of their money back. 

@Melissa Dinas We thought about house hacking but it really didn't fit our needs.

Does that fit inline with the quickness for your scaling needs? 

At this point, I'd prefer to grow my investor base from people I know and develop relationships with (which is a slower route- but fits my investing goals).Hope that helps.

Usually a Land Loan is 50% of what exists- and you need a verbal nod from planning that septic, curbs, gutters, utilities, especially water, and the size of what you plan fits the neighborhood or general plan.A construction loan on land is a different thing.

My wife and I have a large growing family and a home that perfectly fits our needs but want to expand our rental portfolio.

They are usually run by picky retired people who love to stick their nose into your business (a generalization... but often fitting!).  

There's no one size fits all when it comes to midterms.

While you won’t get rich doing this it could be a good fit to have supplemental income to purchase your first few deals.

It could be a good fit for you.