There are a couple answers to this, one is math-based and one is not. Mathematically, it should be relatively easy to run your numbers and see what your return-on-investment is. As long as the ROI% on that money is GREATER than the interest rate that you're paying on the debt, then mathematically it makes sense to invest...that's whether you're investing in real estate, the stock market, gold, beanie babies, whatever.
The qualitative answer involves where you and your family are financially. Do you want to save up for a vacation? kid on the way? surgery coming up? Depending on the level of financial impact events that you have on the horizon, you might re-think taking on something somewhat risky like real estate investing while you still have some debt.
All-in-all, you should be "responsibly leveraged." Just about anyone can find someone to give them a loan and get "leveraged," but to be RESPONSIBLY leveraged is different. If you all have a clean financial horizon, I say go invest that money and make minimum payments on your debt. Debt is tool/resource when paid off responsibly and used correctly.