Barbs are considered to be OP with their added HP and extra damage, so yeah, there should be some kind of penalty to armor just to maintain some semblance of game balance between classes. Why would anyone bother with Fighter or any other Fighter subclass if Barbs got all the bonuses and no penalties?