Using 401 K money to pay off my primary mortgage - Good or bad idea ?

[Westin Hudnall]

My story : I'm 30. Fully vested with $55,000 in my 401 K . My primary mortgage has a balance of $35,000 left of my original $182,000 loan I got 5 years ago.I made a lot the past few years during the down economy and was able to pay off my house debt rapidly. I now have a baby ; my wife cut her workload by 75% and I no longer make strong enough income to save enough to pay down my debt any further. 

My goal is to have as much monthly savings as possible to build my buy and hold portfolio.

Does it make sense to pay off my home (P&I $950/month) or to leave it in retirement ?

Why or why not ?

Thanks in advance :)