My husband is a high school mathematics teacher and I’m currently focused on getting my real estate license and starting our investment business.

If you use the money for more investments it will be mathematically better to pay the mortgage over time assuming you still cash flow a bit on the property.

A mathematical formula doesn't make assumptions, it's objective.

As an engineer, a mathematical problem solver, you should be able to understand how this works...and do well at it.Here's you biggest hurdle...it's your greatest advantage...being an engineer.An engineer is trained as a problem solver. 

The property would be worth a great deal more if the rehab was done but I cant prove it mathematically to potential buyer.

But although I don't hear it mentioned as frequently, IRR is still possibly the most heavily-relied-upon calculation used by investing professionals, so it might be good to include.The other nice thing about the mathematical method I'm using is that it can easily accommodate qualitative variables by assigning them a points system (e.g., local economic conditions ranked 1 to 5), which then allows them to be included as part of the analysis alongside the quantitative variables.

However, I'm not sure if using diminishing ROE is a good measure for keeping or selling, mathematically speaking it should always go down in an increasing market which is what you want, you don't want your home value going down.  

And should you decide to exit before your investment horizon ends CAGR calculations become indeterminate (mathematically speaking).

Mathematically speaking is there a best time to sell a rental property?

The worst however, gives you an IRR of -28.47%...ouch.This is a back of the envelope analysis off the top of my dome, so forgive me if there are a few discrepancies, a BS in applied mathematics ain't what it used to be.