Wrong!

The contractor had done things wrong and he had fired him.

Poor us, we worked so hard, we've tried everything, were transparent, but yet we still don't know where things went wrong and why the building still has not been completed. 

You might appreciate, but most people who do this, have no strategy and also no oversight so things go wrong and they never know.

Is it wrong for me to end his lease considering he has a infant?

Consider all the things that could go wrong and see if the lease addresses them: unauthorized pets or tenants, early termination, security deposit, lease violations, late rent, eviction, lawn maintenance, parking, etc.5.

(I'm currently dealing with a lawsuit for wrongful death on one of my properties - fun right?)

Nothing wrong with LTR's - but what I like most about STR's is you are (usually) operating in better appreciating submarkets. 

I may be wrong but you cannot refi unless the property is seasoned for 12 months when using Fannie, Freddie, Ginnie.

My tenants are decent people and I let them know that I hate these kinds of conversations and if other people are complaining then clearly there is a pattern here that is wrong on their end.