I am often the contrarian. And I will state different jurisdictions have different rules, processes, expectations, etc.
In my area you are required to have a permit to swap a water heater.
I have never pulled a permit for a water heater swap, know no one that has ever pulled a permit for a water heater swap, ever seen or heard about one getting red tagged for no permit, and have never had a plumber recommend getting a permit.
However, if you ask the city if you need a permit for a water heater swap the answer is yes. They will take the permit fee and send an inspector that will provide some assurance that the work was performed correctly.
I pull permits for removal of load bearing walls, addition of bathrooms or bedrooms, addition of footage. I have pulled a permit for removing a fireplace, but probably would not have but I had a significant remodel going with permits and did not want hassle of him making an issue of removing the fireplace (the inspection for fireplace removal consisted of only verifying gas cap was in visible location, he did not even spray it).
I have gotten red tagged once for unpermitted work. The lesson was exactly the opposite of what should be desired. Tenant had some risk items such as extension cord going from outside to inside which invited an inspection. Some of the red tagged items proceeded my ownership. The city made me conform to their exact permit application process which was burdensome and frustrating. An example is on one permit I had plans for electrical, plans for gas, plans for AC condenser. I got comment to submit a single plan. I initially had no clue to the comment so went to permit office. They wanted me to append the 3 plans into a single document. The lady said what if there were 100 documents, my response was there are 3 documents and no where near 100. It took a lot to comply with their permit expectations and was frustrating and time consuming. Prior to inspection I replaced one outlet cover. The inspector wanted me to add 2 screws to anchor the condenser and he did check for leaks at the gas connection (the gas connection that was flagged was decades old and I questioned if was before a permit was required but city stated it was up to me to prove that it was there prior to permit requirement). All that hassle for 2 screws to attach condenser. The lesson to me is jurisdictions can make the permit process to be painful, requiring tribal knowledge. If I had applied for permits when work was done I would have experienced the same permit process issues. It discourages applying for permits.
So my lesson is permit application can be a hassle. Some things do not justify that hassle. Other things such as adding bedrooms, bathrooms, square footage justifying permits.
Good luck