Is it worth paying for points?

[Michael King]

I'm finalizing the loan for a 4plex. The loan is 30 years for $240K and the quoted rate is 5.125%. I can buy it down to 5% for $1013, or 4.875% for $1644.

If I buy it down to 5% I save $25 in interest per month. It would take 41-42 payments to recover the $1013 and save $6600 over a 30 year loan without additional payments on the note. 

If I buy it down to 4.875% I save $50 in interest per month, and would take 33 payments to recover the $1644, and save $13200 over the course of a 30 year loan. 

We all like to save money and the answer in my mind seems clear - spend $1644 and buy it down for the $50 a month interest saving, and lowering the payment from $1320 to $1270. But what am I missing? Is there a better use I can make of that $1644? I plan on keeping it for no less than 10 years when my kids hit college. 

Or is this just a really dumb question / first world problem?