I would consider refinancing or selling it, depending on circumstances. But leverage is key in investing. I actually had the same circumstances 18 months ago. I could either have bought a primary residence all cash or bough 4 investment properties. I chose to buy the investment properties, but in hindsight, I would have done it a little differently. I would have bought my primary residence and used whatever was left for investment properties.
The interest rates and down payment requirements are a lot less on a primary residence. Also, if you live in it for two of the last five years before sale, you can write off $250,000 in capital gains, $500,000 if you're married. So you can buy a house, live in it for two years, rent it for 2 1/2 years and then sell it. So you can have a rotation of three primary residences in addition to your investment portfolio.
The more that you leverage, the more appreciation that you will see. I have a strong investment portfolio, but we're still buying more primary residences to take advantage of it.