I have been in the same situation, and like everyone said, decision is very personal. I ended up paying my primary house and to me that's one of the best decisions I have made. It gave me relief and clarity to think about what to do with my money next, and not to worry about mortgage.
I ended up getting a HELOC on the house later and I use that for investments as and when I need it. I would not pay my rental properties with that.
I also decided not to have debts which don't produce income, primary home is generally one of them(unless you are renting out a part of it). The amount I would pay in interest itself on the life of a loan was enough to make me decide that.
I agree, this is not the best financial approach if you just look at the numbers, I would do what make me sleep fine in the night, at the same time making me some money. Depends on how much security you feel in your job too.
I second Mark Poulton, the peace of mind alone is worth it!
Another thing which might help decide is thinking about your final goal, is your priority having financial freedom early on or, to be very wealthy later, I don't see them as same. (But I know we all want both:))