Yes. It is possible that he could claim adversarial possession if he has been using your property- I'm not an attorney so please do your own research. This is what a quick google search revealed:
In Michigan, possession must be actual, visible, open, notorious, exclusive, hostile, continuous, and uninterrupted for the statutory period of 15 years.
The laws in VA where I live are written the same way. Our house has a chain link fence running the property line that is actually 3 feet set back on my neighbor's property There are a variety of reasons for that, a huge tree, for one, but the point is the fence has been there for many years, effectively cutting the neighbor off from that portion of their property. That in itself is enough. Our neighbors are not difficult people so it's never really been an issue.
I hope the survey will help you get this settled. Arguing with an attorney is generally a losing battle as they won't have the legal costs that you will. Who knows, maybe they would be unable to prove the 15 year statue. Good luck!