Mostly I was trying to gauge other's opinions if it was even a smart thing to borrow more from the EIDL loan.  

FHA is typically for borrowers who need a little more wiggle room with DTI (Debt to income Ratios) and Fico scores. 

I also don't borrow for anything that is not an asset.

Any suggestions on the best way to borrow the 20% down?

I would also kind of like my first deal to be all mine, so if I screw up I don't have to answer to anyone else (or risk needing to borrow from Peter to pay Paul).

We would do this for a year or two until we could borrow against our equity to purchase a single-family home.

Start with financing to make sure you have an idea of how much you can borrow. 

Not saying you are that guy, but the bad guys made it difficult for everyone in the future.Your mortgage probably does have a due on sale clause.You might be able to do some financing some other way....like pull from 401k to pay off the land and then owner finance....or get a HELOC on your primary to pay it off, borrow from life insurance policy, something like that. 

@Nicholas I too took advantage of the cares act,to invest in RE.You can take up to 100k if you have been affected by covid,w/o the tax penalty.I was furloughed in March that was my affect.If you are still employed or furloughed you may also be able to borrow and pay yourself back with interest.I love it because it’s pretty quick turn around if a deal comes up.My company changed from an employee owned company to not and we received a lump sum for our esop.I lost 21k during the switch over and decided I would never lose money that I didn’t have my hands on again.I control my investments now and wouldn’t have it any other way.Best of Luck!

Heloc processing is now more complicated and it may impact your borrowing limit if you close and borrow.