It has to do with the “fashion language” and what each item “is speaking”.

To me, a synthetic jacket speaks “trekking”, a backpack is also “trekking”-ish so it matches in my head. But if you have a tailored wool sports coat with clean silhouette lines, which speaks “city”/“elegant”, and you slap a synthetic backpack on top it clashes to me. I guess it would also wear down the more delicate wool weaves from friction.

Say, I wouldn’t want to wear a backpack with something like this or this

But to me it would be more fitting with something like this or in general when trekking around somewhere.

Again, do what you want - I concede that backpacks are more comfortable. To me it just looks better with certain clothes and worse with others.