For example, a lender might offer you 85% LTC (loan-to-cost)/65% LTV (loan-to-value).Interest rate: What will you be paying to borrow the money?

Many of these loans can be non-recourse or limited recourse, but there will likely be liquidity and net worth requirements of the borrower.

Mostly I was trying to gauge other's opinions if it was even a smart thing to borrow more from the EIDL loan.  

FHA is typically for borrowers who need a little more wiggle room with DTI (Debt to income Ratios) and Fico scores. 

I also don't borrow for anything that is not an asset.

Any suggestions on the best way to borrow the 20% down?

I would also kind of like my first deal to be all mine, so if I screw up I don't have to answer to anyone else (or risk needing to borrow from Peter to pay Paul).

We would do this for a year or two until we could borrow against our equity to purchase a single-family home.

Start with financing to make sure you have an idea of how much you can borrow. 

Not saying you are that guy, but the bad guys made it difficult for everyone in the future.Your mortgage probably does have a due on sale clause.You might be able to do some financing some other way....like pull from 401k to pay off the land and then owner finance....or get a HELOC on your primary to pay it off, borrow from life insurance policy, something like that.