The best rates and terms are going to be with Fannie Mae and most of those loans can be done down to a 620 credit score. 

@Clayton SmithI was able to find a local bank that did an 80% LTV REFI on a 5 year year ARM amortized over 30 years at 5.25% So I will just have to refi again in a few year which I can do with the same bank or go to a conventional Fannie Mae long term loanWorking with the local bank was a breeze compared to conventional lender.

Most conventional loans conform underwriting standards and are backed by Fannie Mae or Freddie Mac.

Here is Fannie Mae's LTV matrix with conventional financing

The deal was financed with a 25% down traditional Fannie Mae/Freddie mac conventional loan.

If you have a lease agreement and deposit then Fannie Mae or Freddie Mac can use that to help your debt to income ratio.Typically 49% debt to income is the highest they allow, but you also need to be able to have a credit score and assets to support it. 

Also FHA has the highest debt ratio allowed at 56.9% and FHA will also count your rents from the other units, helping your debt ratio.Fannie Mae and Freddie Mac will also count income at 1 year showing on a tax return but their debt ratio max. is 50%There are several programs in Non-QM loans, such as 12/24 month bank statement programs. you can also do a Asset depletion program that takes any liquid assets you have, divides it by 60 months to derive your monthly income.

Not sure how familiar you are with them, but portfolio lenders have more flexibility since they keep the loan in-house (no Freddie Mac, no Fannie Mae). 

What's even better if the first one you use FHA (use the 203K to renovate the house on the bank's dime) and then use 5% down Fannie Mae for the second one and then start doing conventional loans until they tell you you can't have any more.  

You can do a Fannie Mae loan X 4 - 1 for each duplex if on separate parcels?