I think it just depends on what things you got used to in your current house vs potential inconveniences in the new smaller one.
I used to live in a 6000sqft house and downsized to a 2000sqft house. This new house has everything I wanted though, the old one was just ridiculous and I didn't need that massive place. For example I didn't need the 20ft ceilings or 3car garage or water fountain in front. I now live in a normal house that still meets my needs and I freed up quite a lot of money by doing so like you are describing.
But if this new house doesn't have a 2nd bathroom and you really need one due to your kids, you will hate that and be kicking yourself later. Or maybe you really want an attached garage and the new one doesn't have it. The list is entirely up to you but if the new place doesn't have the things you want you'll miss your current home later. Also the feeling of security with no mortgage can be tough to let go of for some people. Good luck!