I don't think it's risky at all in the current environment, but if interest rates tripled or something then banks very well could become interested in calling loans.

To me, this would have never been the conversation a few years ago when rates were more comfortable for the average investor (likely 5% and below), but it is the reality we are in now!

Then the buyer did some small things that didn’t give me much comfort...like holding the contract and changing terms.

It's going to be less comfortable and more work, but a SFH in a good location is going to appreciate much better than a duplex in a less desirable location. 

It made me feel comfortable right away. 

It's all about your comfort zone and how much attention you can give it. 

I would pay them again in a heartbeat because of the added value of weather protection on the home.I would look into a couple more quotes to give you a broad range, which might make you feel more comfortable in whatever direction you go.  

Today though, we are in a high priced environment with expensive money.  

You'll want to use your skills and some creativity if you want to push yourself farther into the "profitability" side of the scale over "comfortability".  

Cash-flowing SFR's are tough to find in this interest rate environment without value-add rehab work or a large down payment.