Yup, having just gone through a fence replacement thing and our city’s building code - of the posts of the fence face your property, it’s your fence. So either this fence was installed improperly and with the posts reversed (probably against code), or the OP is the owner of the shitty fence.

On a side note, the other fence also looks like the posts face in - which means there are two fences on the OPs property - another building code violation where I live.