I would check with the code department (or may need to talk to the building dept if there is a separate one). Depending on local zoning and code regulations, you might be able to keep that 3rd unit if you legalize it, or you might have no possible way to have any living space on the third floor, or many in-between options (including the suggested conversion of 3rd floor space to part of 2nd floor unit). Even if you could keep the 3rd unit, the cost to legalize could range from minimal to too-big-to-bother (again, will depend on your exact situation).
Hard to know unless you talk to the folks who enforce the rules. I have given up trying to guess how current nonconforming uses will be treated, I think I have learned to simply ask :)
I was going to list possible factors for officials deeming it a 2 unit or 3 unit allowed building, but the list was unhelpfully long. Even if it could be a 3 unit, don't forget to consider whether the ongoing code, registration and/or inspection requirements are different for 2 units vs. 3 units - usually more units is the way to go, but at least once we have chosen to go with fewer units because of the cost of meeting code requirements at the higher unit count.
In some areas (including CT and NY), the "county records" are up to the tax assessor, but the code enforcement office determines permitted use, and if there is a conflict, code prevails.