Side by side duplex with two separate deeds!!!

[Steve Salvini]

Current situation:
I recently found a duplex that is side by side that is deeded individually that I would like to buy and live in one side. I currently have an FHA financed duplex that I've been in for about a year and a half and was planning on refinancing out of until I found this. I would need to kill the refinance process and keep my current place in an FHA.

My thought:
Since they are on two separate deeds, I could structure my deal that I buy the investment side for let's say 20 grand since I would have to put 20% down. Then I would buy the side I would live in with a conventional loan with 5% for let's say 80 grand since he is looking for a hundred total. Is this a realistic option? The only drawback would be if the higher side doesn't appraise high enough right (I think it will appraise though)?  

Goal:
Trying to buy the new place with as little down as possible. 

Thanks in advance on any advice on this one.