I guess the point is you’re middle aged in regards to your contribution to society. First 15-20 years of your life you pretty much just “take”, while the following 50ish you are expected to chip in. In those terms, 50 sounds about right as being referred to as “middle aged”.