Down payment can be higher than primary, but the good thing is that you won't need that much income to qualify because lenders can you 75% of the market rents for the units of the property.

Both side said that my DTI was too high and I wasn't qualify to borrow.

A good, well-qualified borrower will rarely get into trouble.

Thank you everyone @Ashish Acharya @John Bowens   @Gino Barbaro  @Jacqueline WrightWe'll be staying out of our Traditional and Roth IRAs then for this.And we aren't ready for and aren't qualified for a self directed IRA at this time, so we'll not go that route.The loan out of the 401K is interesting. 

If you have a rent property with good cashflow, odds are you are BETTER qualified for the next mortgage than if you sold the rent property.  

The Qualified Intermediaries I am vetting have provided their exchange agreements.

Does anyone have experience with applying, qualifying, or building with the Guaranteed Rural Rental Housing Program?

The remaining $50,000 qualifies for the tax-free exclusion under Section 121, assuming the homeowner meets the residency requirement.

The last thing that comes to mind would be moving the funds to a rollover IRA and withdrawing them with help from your tax advisor or CPA to qualify for the first-time homebuyer exemption. 

This will hurt the tenant's credit score, it will warn future landlords when they run a credit check, and it will prevent the tenant from qualifying for a personal loan, home loan, etc.