Seems to be more emotion behind these beliefs than logic...

Long and the short if the persons is telling the truth you must show compassion and give a little extra time, If you've been deceived then don't become emotionally involved with it follow the letter of the law regain you possession and find a new renter.

If you can not separate your emotions from your business I would strongly advise you sell before you get another tenant.

Otherwise, your emotion will play in, as it is now.  

In terms of what you should do with your profits/additional savings, there is a pure math answer and an emotional answer.The pure math answer is to do whatever makes/saves you the most money. 

Take the emotion out of that side of the transaction.

As others have said, duplexes and even 4 unit buildings are often sold on emotion and not the financial performance.

Negotiated an Owner Finance and closed escrow after 2 weeks.Price: $72,500Zillow Est: $111,000Summary:It was an absolute emotional roller coaster, navigating through all the small details that wont be a big deal next time around.

My impression is that those who are against it are basing it on an emotional response.

You sound too emotional to be investing in real estate.