Divorce courts being closed for so long (specifically, a year of shelter in place....) created a pent up demand for divorce related services, including appraisals. 

A big part of me wants to cash out, pay off my vacation property, and/or use proceeds to fully pay off my primary residence.

Any thoughts on how to proceed?

No court will be on your side, and it will cost you more than you will gain.....

Unless you own a trailer court mobile homes are not a good investment. 

He then proceeded to tell me he can't then apply me for a conventional loan, it must be for an investment property, and can they can potentially call my loan note due if i don't use it as a primary home.

Be prepared to answer that one in court. 

The cash and the seller carry back note ("SCBN") are the "net proceeds" from the sale of the relinquished property. 

You need to reinvest in your portfolio with all proceeds and probably inject additional capital as available (plus having reserves or credit you can tap if a big cap ex comes up).

That means if he stops paying, you take both of them to court.