Should I be concerned the bank may have issues with this? 

May be make OZs permanent, let passive losses offset active income, etc. 

Was it for a place to live for work before they found a permanent location or was it used for someone wanting to stay in the area as a 2nd home for the winter months?

Another concern is after the two years, I still wouldn’t be lendable for most properties in my markets due to that lack of income. 

I’ve had concerns about the crime rates so I’m trying to only buy in the better sides of town.

These and 1000 other questions or concerns or issues that to me are potentially of vital importance to your purchase, your investment, your resale and exit strategy for the future.Real Estate is a team sport. 

@Robert EllisAre you leveraging construction-to-permanent loans or private funding?

The concern here is the financing.

This means that your permanent residence is excluded (except if you are doing a house hack in which the part that is rented out is eligible for cost segregation).

The IRS is concerned with the tax return that is reporting the activity of the property.