When we bought our house, both neighbours said the side fences were our responsibility.
My reply was "fine, no problem. I'll replace them with what I want and the good side will be facing us"
And that's how it went, and still is.
One of the idiots later actually fastened some tall bushy type of flowery things to his side of the fence (to keep them upright I guess) and when we replaced the rotting panels the plants came up too.