Would you suggest refinancing or saving to purchase my second property?

Refinancing to payoff the construction note and also getting cash out.

I had never thought of refinancing it with another lender

We split in May of 2019, I was still living there so to finalize the separation I refinanced her off the loan by paying her equity in July 2019.. 

(I will discuss interest rate buy-down options later.)How much would refinancing cost you in increased debt service?

A huge part if not most of the investors' return will come from the forced appreciation, which will be crystallized when the property is either refinanced or sold.

I want to protect myself and my family from an increased mortgage through refinancing later.

I am a long term buy and holder.I started with 30 year mortgages, but later refinanced into 15 years, and some 10 years when the government adopted the "Home Affordable Refinance Program" (HARP) at very low rates in response to the great financial crisis.

Once you have the adu rented out it might be worth paying the pre-payment penalty and refinancing into a longer loan.

Some lenders will work with any level of investor experience, credit scores as low as 660 and can close in as little as 10 days (there are loan options for 640-660 credit scores- they require 20% down).Another good thing is interest only and 6-24 month loan terms- you can refinance by selling or refinancing to a long term DSCR rental property loan at any time once you complete the rehab.