Mortgage issues - First time Buyer

[Nathan Godson]

Hey Everyone, I'm new to this site. I have an interesting situation and i would appreciate some professional insight. I recently just made an offer on my first house on a duplex in Holley, Ny and it got accepted. The purchase price of the home is $92,500. I'm now in the process of obtaining  a mortgage. The problem I'm having is that i am in a first time home buyer program that i signed up for through M&T bank. It's a federal loan program through Federal Loan Home Bank of NY. M&T bank is telling me that an owner occupied duplex they will give me an interest rate of 5.625 with purchasing 0 points and 20% down. The reasoning they gave me was because multi family propertys are more risky and that they increase interest rates for homes under 100k because they don't make as much money. I then called the local credit union and they said rates fluctuated between 5-5.1% today for multi family units if I were to go through them. The problem is that i'm locked into M&T with my first time home buyer program. I get a free 8k towards my mortgage for being in the program. I called Federal Loan Home Bank of NY and they said the only way i could switch lenders is if M&T could not fit me for a suitable loan, meaning i had to not be able to qualify and they would have to be willing to sign off on me. The credit union also stated that they have not received any transfers from the program and they believe it's difficult to do so. My justification to M&T was that my contract with the seller is contingent on me finding a loan at 5% (which the realtor told me it was playing it safe)  and they can’t offer me a suitable loan with a comparable interest rate. So has anyone heard anything about anyone backing out from one lender to go to another with the first time home buyer program? Do multi family and cheaper houses always have higher intrest rate? Any advice is appreciated.             

Side note: no outstanding debt, make 75k a year, 762 credit score. - I feel like they are really screwing me over with what they are offering me. My friends who have way worse jobs and credit scores are getting loans almost a full 1% better then me. Makes no sense. So much for killing myself to get a good credit score.