Chances are he hasn’t seen a specific violation from the outside to give you a stop work order. Most townships in PA allow siding & roof replacement, trim replacement, and even window replacement (as long as they are of adequate egress size) without a permit. I have no idea if it’s similar in your township/bureau.
BUT, if all of this is happening on the outside, there’s a good chance the inside is just as, if not more, involved.
I just started working as a Building code inspector in PA and we, believe it or not, have an obligation to stop and ask questions when we see construction. (I’m not too crazy about that part of the job). So this could very well have been the case with your inspector who left his card and asked for you to contact him/her.
If you’re as confident as you say that all the work has been completed that meets/exceeds current code then let him see that and get a proper CO (certificate of occupancy).
What is your approach when it comes time to sell and you list all the upgrades you did to the home? Any realtor worth his or her salt will ask if permits were obtained for those upgrades and if a final inspection and issuance of a CO were obtained. Saying no could cost you a lot more at resale than pulling permits....
My respectfully submitted two cents.
Best of luck to you!