You also have four individuals which increases the number of cars on the property, more visitors, higher likelihood of partying, more inter-personal relationship problems, more wear-and-tear, etc. 

We refinanced our house several years ago to decrease the length of our loan to 15 years which increased our monthly payment a little but has allowed us to make faster gains toward the pay off. 

However, I'm finding it difficult to increase my calls and consistently make 2-3 hours of calls a day while keeping my day job, and I need to be making more calls to get more deals as I find myself spending most of my time with follow up calls. 

I agree with this, but I do think you are better off getting them out as inherited tenants increase risk.

Do I concentrate on paying it down and increasing my credit score?

My lender says it makes sense to at least refinance even if I wasn’t cashing out to get rid of PMI and have increased tax benefits from the increased interest rate.

But being that it's increased so much it's got us thinking... 

The tenant may withhold the difference between the old rent rate and the new increased rate.

It seems here in the Bay Area, it is starting to soften, DOM increasing, some sellers having to accept lower than listing price.

This change alone stands to gain me ~$150k+ in equity due to the increase in NOI.My concern is that firing this PM may prove complicated due to his status as a tenant.