So, stop where you are, ask for forgiveness (sounds like you already have) and get permits.  

Plus you get depreciation on 4 as well, as well as market appreciation on 4 - which long term is where you can make a lot more money.  

sell LTRs and buy Vacation rentals where you and your family can enjoy as well as make income.  

These factors didn't keep me from buying though, while watching some other small markets in the region where average nightly rents were sometimes higher. 

The market here is slowing down a bit, especially on the retail side where homes are selling for over 500k.On the investment side, we've noticed a significant price decrease in properties in Flood Zone AE due to skyrocketing insurance premiums.

To find a house where the seller is willing to sell it to you Sub2 the existing loan you need to do quite a bit of marketing.

At any point though could increase high enough to where renting isn’t feasible.

I have handled thousands and thousands of maintenance requests over the years and there's only been 2-3 instances where the tenant suffered some damage.

Is market like Texas where they often have soil/foundation issues, or like MN/the Midwest where the asphalt shingles might have hail damage?

I've done it before where I cover the probate expenses for the seller and still make it a successful wholesale.