You need to get paid the rents so that you can make your payments as well!
Tell him that you expect the rents to be deposited to your account by a certain date to meet your obligations or you will have to find a different company that will. You need to go about this as a business, not a popularity contest so don't worry.
Even if it costs you a bit to change, what's it costing you now? Over $10,000? I call BS on that guy, don't worry about ruining the relationship. Apparently he doesn't care about it. Perhaps he thinks that it's due him for managing.
You don't even have to go in person. I recommend first talking on the phone with several PMs and choose one that you are comfortable with and then put yours to the test. If he doesn't cough up your funds and he's still late after a month, then leave, and you'll already have a backup available.
He probably doesn't even have your money and squandered it (I hope not).
Think about the baby!!!