My primary residence, single family home and a large 7 bedroom duplex 

I'm always conflicted with whether I should leverage my primary residence for the sake of investment properties.

Now sometimes I would get a credit for a new lead but you have to write a detailed explanation every time I would ask for refund and wouldn't always get it rewarded. 

Or refinance for spending cash when it come time to retire.The advantage you have living locally is that you can buy a new primary residence every few years and keep the prior home for a rental.Good luck.

@Trevor Finn Nailed this explanation!

I was told by my lender that this has to be my primary residence for some time.  

My plan is to use my HELOC from my current primary residence as the down payment.

Quote from @Michael Plaks: Quote from @Scott Trench:  Super helpful explanation!

You could 1031 out of the property, purchase a new one or something you might like, "Live" there for two years and convert the property to a primary residence. 

I have a primary house in Santa Maria CA that we are renting out and getting cash flow from it,  thinking of using the equity from that house to buy here in Orange County.