New siding, but neighbor is refusing to let contractor go in their yard

[Michael Seeker]

So I've hired a contractor to put new siding up on a MF property. Right next door is a not-for-profit that is a halfway house. My contractor is a pretty reputable and nice guy - he went next door, knocked and asked permission to put his ladder up on their side of the fence because that's the only way to get at the siding.

They said sure, just don't talk or interact with anybody staying there. Well a few hours later he's up on the ladder and they come out and tell him to get off their property or they will call the cops. I've called and left a message for the director to call me back, and the lady I spoke with sounded like she had her panties all in a bunch about it as well.

Can anybody think of a reason why they would care if the contractor puts his ladder up over there?

Assuming they continue to refuse to let us be over there, is there anything we can do to legally do the work? There is absolutely no way to do the work without putting the ladder on their side of the fence - there's not even enough room for scaffolding. Any ideas/suggestions would be great. I'm trying to go the route of a nice conversation, but it sounds like that may not work.