It’s not so much about the audience perspective as it is the layout of the building. Just like backstage is the area behind the stage, other areas like behind the bar, or the storage areas, green rooms, offices, tech rooms, weird tunnels full of mysterious cables, etc are all collectively ‘back of house’.

Like when you go to a big box store and they see if they’ve got something ‘out the back’, the back of house areas are those generally not seen by the public. The term front of house likely evolved as the opposite of this.