Seeing the place where you made so many memories with your family trashed, and then going through the emotional turmoil of repairing and remodeling it, knowing it will eventually just get trashed again...are you prepared for that?

I wanted the house and got emotionally attached ... to the neighborhood because I knew the prices starting to go up in this neighborhood quickly.

Best is when they get emotional and start demanding and arguing with judge, oh-yeah, that's when pull out the pop-corn and just back n watch the show, lol. 

Very emotional. 

+95% of agents have only done owner-occupied transactions, which are mostly emotional-based.

Emotions will LIE to you. 

In addition, the owner of the home must not own more than three single-family homes.3) They are unable to provide sufficient documentation, such as the emotional support animal letter, that verifies the ESA is necessary for their health and well-being.4) The tenant provides fraudulent documentation attesting to their need for an emotional support animal (usually a fake emotional support animal letter).5) The landlord can demonstrate that making accommodations for an ESA would impose undue financial burden or logistical burden. 6) The animal is destructive to the property or displays threatening behaviors that could put other tenants or yourself at risk.7) The size or nature of the animal makes it impossible to house safely or humanely.

If you make this a balloon payment in say 5 years, you've helped your cashflow, and you can refinance to pay out the balance of the 2nd mtg.I do have to ask though - you seem quite emotionally invested at this point - what are average house prices in this neighbourhood?

or maybe I was blinded by being emotionally connected to the deal?

If it is worth your time to manage your properties versus not having the learning curve and emotional stress of it, do it.