By paying it off you increase your security. 

I could move into a property that looks like a good deal (either in SF or the Easy Bay) and secure an FHA for 3.5% down by virtue of living in the property as my primary, rent out additional rooms, and essentially function as the property manager.

You'll also want to ensure you have a good property manager to look after the upkeep of the property if you are not there to notice anything deteriorating or which may need attention.California is a sort of beastly state when it comes to taxes and filings.

The one that is leaving wants his security deposit back.

Think of all the things that could go wrong and see if the lease addresses them: unauthorized pets or tenants, early termination, security deposit, lease violations, late rent, eviction, lawn maintenance, parking, etc.5.

If a tenant gets a protection order, I'd change out the locks and probably install a security camera.

Trying to avoid neighborhoods that would require me to hire security for a property, but I don't mind taking some risk.

Would finding a wholesaler to secure deals for me be counterproductive?

Insurance costs will go up when your value goes up, obviously and the bank will likely make you secure a year of premium before refinancing.

Also ensure that you get the security deposits from the seller as well.