You can consider the medium term route as a possbility as well so you can still occupy it for a few weeks a year.  

Tenants may not occupy residence without utility service except during brief interruptions beyond your control.

Maybe the biggest "risk" might be the fact that you cannot occupy the property at all so technically if you run into hard times and need to move into the property thats not intended - but no real more risk. 

As others have said, this is the best way to get started because you can get much better financing when the property is owner occupied.

You either need to find a way to make and save more money or look into owner occupied investment strategies like a live in flip or house hack.

If you're tax bill is $6-8k (minus the 1031 fees), surely you can make that up by taking multiple swings over a longer period of time.OR pursue some sort of owner occupied strategy with that cash if you can swing a live in flip or house hack.

You can get a normal owner occupied home loan to do this. 

I have a tenant occupied single in Allentown for sale.

Sell it into an S Corp at market rate, that you are the owner of, taking the tax free gains because you owner occupied it. 

If it's in a shared space like a basement that's one thing but if it's in an occupied unit, that is something that may not be allowed in your jurisdiction.