The downfalls to SFH are that it is harder to scale your business as you grow, there is more work in property management, which increases your costs, and the loans get harder to get as you grow. 

This not only effectively increases your hot water supply, but makes it safer.

They were both ARMs that increased to 11%, and he couldn't afford payments.

I wonder if the thought is that by replacing those tenants, my maintenance and repair expenses drop, thus increasing the NOI and value.

If her portion of the rent will increase you may be at risk.

You also have four individuals which increases the number of cars on the property, more visitors, higher likelihood of partying, more inter-personal relationship problems, more wear-and-tear, etc. 

We refinanced our house several years ago to decrease the length of our loan to 15 years which increased our monthly payment a little but has allowed us to make faster gains toward the pay off. 

However, I'm finding it difficult to increase my calls and consistently make 2-3 hours of calls a day while keeping my day job, and I need to be making more calls to get more deals as I find myself spending most of my time with follow up calls. 

I agree with this, but I do think you are better off getting them out as inherited tenants increase risk.

Do I concentrate on paying it down and increasing my credit score?