Property Line, Downspouts and Threats from Neighbor

[Abram W.]

My property line ends along the edge of my building. So two of my downspouts flow onto my neighbors property. My neighbor is threatening to physically remove them and take me to court. However, given the property line, I'm not sure what is legally allowed here. I attached a picture for clarification. 

Please advise.