I wanted to give a final update on this thread. So I moved out of Providence and after stabilizing the property I decided to sell it as the market was quite hot and I could make a fairly good return. During the selling process, all this issue with the building in the town and tax record came to surface because the appraiser was VERY diligent at his job..
So anyway I had to correct the records. It wasn't as bad as I had thought and a few lessons learned here I hope to help someone else out.
First is that the tax records and building department don't talk to each other. It is actually really easy to get the tax record changed. All I had to do was call the tax department and tell them that the record is wrong and have them come take a look. The tax assessor came the same week I called and did a 10 min walk through to confirm this was just a 4 family and changed it that day. A month later it showed up online too so everything there was all set. If I had done this earlier the appraiser wouldn't have seen any issues as he initially just wanted to see the tax card.
Second is that the building inspection is willing to work with you but there are a lot of red tape... The building inspector was sympathetic to my issue and agreed that it is weird how the city has 5 units while the building is just 4. He came for a walk through pretty quick and confirmed. However, after that he needed me file a permit, provide a floor plan, and get electrical, plumbing, mechanical, and fire department signatures. I drew the floor plan myself with a software and he accepted it. Electrical, plumbing and mechanical signatures were very easy as they understood the issue and signed without coming to the property as they know there were no work done. However, fire department wouldn't sign off until a full fire alarms inspection was done. Being a 4 unit, it is also a different procedure than the regular smoke cert for 3-units or lower. After all that, new CO for 4 unit was finally given.
Through out this entire thing, everyone, including the city and the tax assessor were surprised the appraiser had such issues. They actually told me the appraiser should just go by what he sees because they admit the city records could be wrong.. I am not sure what to make of it but I am glad at the end it is done.