Ok. I think this is kind of an "authoritative" answer. It looks like it's probably not allowed in most cases. I hope John's deduction is allowed but most likely it is not.
http://www.fool.com/personal-finance/taxes/2005/07/01/retirement-loans-is-the-interest-deductible.aspx
Here's the relevant sections.
Qualifying retirement plans
Before we get too far, let's make sure that we're all talking about the same thing. Remember that there are no provisions to borrow against your IRA (either traditional or Roth), SEP, Keogh, or SIMPLE plan. What we're talking about here are qualified retirement plan loans, such as against your pension plan, a 401(k), or a 403(b) plan.
With some very important exceptions, interest paid on a retirement plan loan must be "traced" in order to determine the use of the funds. If the funds are used for personal purposes, then the interest is generally not deductible. But if the proceeds are used for qualified residence, education, business, or investment purposes, and you can trace the proceeds of the loan for such purposes, then you would likely have an interest deduction based upon those provisions of the laws.
Seems simple so far. But as with most tax laws, what seems simple on first blush normally isn't simple at all.
Exceptions for 401(k) and 403(b) plan loans
The law says that interest paid on a loan secured by your account balance is nondeductible if any of the account balance used to secure the loan is attributable to elective deferrals that you made. Well, heck, that's why you signed up for a 401(k) or 403(b) plan -- to elect to defer part of your wages into the plan. So it's easy to see that this exception virtually prohibits an interest deduction for interest that you pay on loans against your 401(k) or 403(b) plan. It's possible that the vested balance in your account is totally from non-elective employer contributions, but extremely unlikely. So generally, interest that you pay on your 401(k) or 403(b) loan is nondeductible -- regardless of how the loan funds were originally used.