House hacking question

[Jesse King]

My wife and I are house hacking and in November we purchased our second property- a half duplex. Yesterday I recieved a call from the owner of the other half. I mentioned to him I would have interest in purchasing his half of the duplex and we discussed some owner financing options which he seemed open minded to! 

This is the first deal I'm seriously considering without using conventional financing. What makes me nervous is the 2 year balloon payment (refinance). I'm a licensed realtor and the neighbor agreed to pay me a 5% commission plus closing costs to help get the deal done.. should I just wait and try to do a conventional loan? If I did the bank would probably look at this only as an investment and would need the 20% down instead of the 5% conventional primary home loan that has worked so well for me in the past...thoughts?