I want to buy it and build a prefab home 3/2 around 2k sq ft.   

I buy to rehab and turn into rentals so Pinson, Pelham, Bessemer are a great fit for the areas and returns I'm looking for.

My personal preference for a buy and hold would be an already established area (or an area that's still gentrifying, but has already seen significant progress). 

And what type are looking to buy, SF, 2-4 MF, or 5+ MF? 

So even if you had a buy out agreement negotiated up front and execute that buy out agreement, the bank would still require him to be guarantor unless something drastic changed on your side financially.The best scenario here would be to purchase and rehab the property using a hard money lender or bridge lender.  

Where I’m at there are places you can buy with20k down.

(or if they're primary buyers, they don't care about yield, since they're going to live there.)i just went to visit two very distressed properties yesterday that i would buy.... if they were priced 50% lower.  

.: As a contractor and flipper, I dont buy materials from retailers or online.

Financing seems to be tricky with IRA hence feel that its best to buy with self funding fully.

I would suggest you do both of what you suggest - meaning get either a new mortgage (more conservative route) or HELOC on the townhome and use those funds to buy your next investment property.