Fence on neighbors property

[Tyco Bronye]

Hi all, thanks in advance to all who reply. I'm in a Minneapolis suburb. I bought a house 5 years ago. When I moved in there were already fences going from the front yard all the way through the backyard.  My neighbor bought his house 7 years ago and fences were also already in place in his yard.  We share one side.  I've only talked to him a handful of times over the years, he seems cool and nice he would usually compliment me on the yard projects I was doing. 

A year ago I was out working on my yard and he stopped by to chat briefly and during this chat he mentioned that his wife was looking into the official property lines and he said, the fence that divides our property in the front yard is actually 2.5 feet on his land at least in this 15 foot long section.  He then said, he doesn't care about what is official and that we should continue to assume the fence is the border.  I didn't think anything of this exchange.  This is a standard 6 foot tall dog eared privacy fence

This 2.5 feet of width had 2 overgrowth trees (my side of the fence) I cut them down because I thought they looked really bad and they were growing into my walkway, They were about 10 feet tall.  My neighbor was really mad and was yelling at me for cutting these down.  He said he lost privacy.  He was kind of hysterical and was mentioning getting a survey and he said that I should pay for half of the survey.  500 total so he said I should pay 250.  He was yelling that the fence is not the official property line.  So the problem is he thinks he owns that 2 feet by 15 foot strip of land that I need to ask him for permission to cut down overgrowth I guess? If that were the case I would much rather he extend the fence into my property versus telling me what I can and can't do on my side of the fence.

I then spent the rest of the day looking into this matter, and found out that it is not a black and white issue of where the original survey said the property lines are (houses built in 1957) there is "adverse possession"

I guess I'm curious in general of people's opinions, and general advice for me.  I have not talked to him but I plan on it soon to tell him my perspective

My opinions right now is, I have no idea if what he is saying is even true.  Why the need for a survey if your confident you know the boundaries?

Of course I'm not going to pay 250 for a survey so he can take my land.  That's like buying a gun for a guy who says he is planning on shooting you.  Since he has the agenda, I'm making no claim on his land I'm fine with things the way they are using the fence as the border, that all money should be spent by him to pursue this if he wants to

 Since we both bought our houses long after those fences were already there I feel I have a reasonable expectation when I buy a house with fences there it's reasonable to assume that's the property I'll be able to use.  Same thing when he bought his house. I feel it's aggressive and over the line on his part to make claims like this, given the fact that we both moved into the our houses as they are.  It would be different of course if I installed the fence without asking him about property lines but that's not the case.  

This seems to be a very common issue when I googled this but they don't say how to remedy the situation.  Is this a small claims court thing? Is it relevant that the fences where already there when he moved in,  aside from the possible claim of adverse possession?  If he just moves the fence what is my recourse? Thanks