Short Answer: If you are going to invest, save the money for future investing, do not pay down your mortgage.
Long Answer: This is a matter of your risk tolerance. Investing in anything, let alone real estate, has a certain degree of financial risk. Depending on the deal and the circumstances, it may be a lot or very little. If you buy an over-priced property, 100% financed, with personal guarantee with your primary home as collateral, then that is very high risk. If you have an option for a property for 50% of market value and already have people lined up to assign it to before end of your option, then you have very little risk. So it all depends on what sort of deals you are hoping to do.
Assuming you are planning on doing plain vanilla 20-25% down, conventional loan funded purchases of SFR to rent out, then you will need cash and good credit and documented income.
No matter what the "creative real estate" types will tell you, having cash on hand will always make investing in real estate easier. Even if you don't use it, just having it around in case you need it (low appraisal and need to make up difference, price cut for cash purchase, better rates with higher downpayment or paying points, etc) will make your life easier and sometimes will be the only way to make a particular deal work.
Having said that, paying off your primary home is something that is more of an emotional decision than a financial one. In today's low interest environment, your home loan can't be more than 4-6% max. And if it is, you should get started with a refi before dinner time. And assuming you do itemized deductions and take the interest cost tax break, you are looking at 3-5% or so in actual cost of interest for your home loan.
If you can't make at least double to triple that (10-15%) in annualized returns on your real estate holdings, then you need to not be investing in real estate. When the deal is good, you should be able to make double to triple your money in 1-3 years for an annualized return of 30%+. I'm a buy and hold investor and some of the good buys from 2011 have already doubled in value, which means that my original investment of 25% down has returned over ~400% in returns (rough math). (Bought for $115k, $30k down, current value $200k, received 15%+ cash-on-cash returns in rent during that time.)
So if you are truly considering paying down your primary home mortgage as a good alternative to investing in real estate, then you need to hone your investing skills and deal finding skills more before anything else. And I mean that in the most sincere way.
Good real estate deals are very lucrative. And if you aren't finding deals that are obvious slam dunks, don't do it! No one has a gun to your head to buy or invest in anything you don't want to. Don't get frustrated and do something stupid and regret it. It's like you are playing baseball with unlimited balls but 1 strike. Wait for the pitch that you know you can hit.
Good luck!