Should I get an interest only mortgage and invest the principal?

[Paul Spangler]

I was talking with my boss recently about our mortgages, and I wanted to see if anyone else has done this.

I just bought my personal residence and have a 30 year loan for $322k at 4.65%.  He suggested that I get an interest only loan and take what I would be paying towards the principal and invest it in the market.  As long as my investment does better than 4.65%, I'm actually better off than paying down the loan.  I don't know a whole lot about interest only loans, and it sounds like the interest only aspect of it only applies for a portion of the loan repayment period and not the whole thing, so eventually you have to start making larger payments to cover the missed principal payments.  You end up paying the same amount if you let the loan go to term.  

Anyway, has anyone else done this or is there something that I am missing, because an average of 4.65% seems very doable in the market, even with the volatility we have seen lately.