Sanity check with this tenet

[Garry Martin]

Hello all,
I am hoping someone can give me an unemotional response to this issue.
I have a tenet in a one family house in upstate ny. In the latest blowup, he called me to ask if I had been on the properity the day before. I told him yes, I was removing some tree limbs that had fallen in the last storm and retrieve a ladder the tenet had borrowed from my stall in the garage. He did not like the fact I was on the properity without calling him first. Keep in mind I never entered the house, just the yard.

He then demanded that I call him before I come on the properity. When I informed him I can come into the yard at any time I choose, he exploded on the phone. I hung up on him several times and he kept calling back. It was not until I told him if he called again, he was out (rental agreement gives me the right to remove the tenet without cause) that he threatened to call the cops if I did it again.

I wanted to give him his 30 day notice right there, but my wife did not want me to decide while I was still so angry. So I am looking for some input from other landlords. I should also say he does pay his rent but has been late by a couple of days for the last 4 months.

As far as I know, I have to give 24 hour notice to enter the unit. Can anyone confirm for me that this does not apply to the yard?
If you were in my shoes, what would you do with him?

Thanks for any input you may have.