My wife and I are expecting as well, and our due date is actually tomorrow!

In my area in Northern Colorado, there are cities that you would not expect to qualify for USDA loans like Windsor and Wellington that you can use the nothing down USDA loan in.Beyond the VA loan to invest in multi-family you are probably looking at creative financing strategies like marketing to find sellers that are willing to do stuff like owner financing, wrap financing, subject to, lease-options/lease-purchases, installment land contracts or the like.

Within 3-4 years I expect them to all be paid off, worth about $3 million and producing about $200k per year in income. 

The customer service level reflects the difference between indifferent salaried people checking boxes and letting automated software do whatever it wants to do, and folks paid for production (like if you went the refinance route, rather than assumption).

Will they expect more down?

Just that learning experience is worth the $100 the first few that come out more than you expect.

Even if you find the right deal and secure the right funding, you might run into property management issues.Your investors (15-20% CoC) have unrealistic expectations.

are these cash buyers expecting calls from strangers like this?

If you have a high enough salary you might be able to buy a small multi- 2 to 4 units, and house hack - living in one unit and then renting out the others. 

That being said you each hired someone to give an opinion, trust in your guy, but don't expect them to trust your guy over their own expert.