Neighbor Fence needing to be moved.

[Lucas Farrow]

So recently purchased a SF home. Needs alot of renovation work . Lot is a typical Houston style city lot 50x100ft. Our problem is our next door neighbors built a fence <6inches from our home. The previous owners owned the lot directly adjacent to our current renovation home . They sold the vacant lot to a developer and likely just let them put the fence up knowing they were going to sell anyways. 

We are having rehab issues due to the close proximity of the fence. It is built 1ft inside the property line and is not only an eye sore, road block for construction, but may limit resale. See attached image of survey showing fence 1ft inside property line. 

My question is how to best go about removing this fence if A) the owner never responds B) puts up a fuss. Don't know my legal rights in this situation. The fence really has to go. You can grab it easily from an open window.