Can I fire a contractor after he pulled the building permits?

[Frank S.]

I hired a contractor to remodel an old home, a total gut/new floor plan job. We are about half way through, the first round of city inspections should happen soon. However, during the first phase the contractor raised his prices and his original bid was so vague that he adds new cost for specific jobs I assumed was to be included in other jobs he quoted. I have received quotes from other sub-contractors and if I stop using my original contractor I can save about $25,000 or 50% of what he will charge me, not including any "unforeseen cost" if he decides to raise the price a few weeks from now. (even though there should be no unforeseen cost because the entire home was gutted including the underground sewage pipes, everything has been replaced except the roof and exterior walls. If I fire him, what issues will I have because he pulled the city building permit? I was told there are 3 main building inspections: framing/structural, insulation, then final overview after drywall. He should be calling in the first inspection because we are ready for insulation.