It’s been a minute since I read the books, but I think I remember having a similar theory. Maybe it was a missed opportunity by RJ, cuz I remember continuously wondering when the grand reveal would happen that it’s actually all Mat (and canonically, I can’t think of a reason that wouldn’t be as valid as other men’s memories, I don’t think the story arc ever kills that possibility lol).