Switching lenders during escrow

[Jody Powell]

Hi BP! I am a first time home buyer and have been working with a mortgage broker for the past 3 months or so. I am currently in escrow, and it dawned on me that I should have shopped around for my mortgage rather than just go to the broker suggested by my realtor. So I stopped in at my Credit Union and sure enough they offer me a better percentage rate (4.125 vs 4.375), no points and pay for the appraisal (saving me $545 in closing costs and a quarter of a percentage point). 

I like my broker, but...

Purchase price of $205k. The loan is conventional, 20% down. ($164k loan)

What would you do? Any mortgage brokers or loan officers want to weigh in? 

I feel bad switching but also feel that I need to be financially smart here.