Pay off rentals early OR Pay down Primary house?

[Brett Lee]

I have a duplex that I owe $135,000 on that currently generates me $1000 per month in passive income.  My primary house has a mortgage of $329,000.  I recently sold our old primary house and I have around $200,000.  I am trying to figure out if I should use some of the money to pay off the duplex and use the rest of the money to pay down our primary house.  

I realize there are tax benefits and deductions for rental houses.  However, it would be nice to bring in closer to $2000 per month in passive income.  If I brought that in as passive income I would plan on tax deferring as much as I can from my job (Teacher) to keep my overall taxes down and also get me in a better position for retirement.  I am 37 but really do not want to teach another 30 years... so any advice would be greatly appreciated. 

What are your thoughts?  Good idea?  Terrible idea?  Please let me know your thoughts.