the magnitude of the rotational inertia […] is roughly the same or maybe a bit higher on larger wheels that are probably heavier

The rotational moment of inertia increases a lot due to the larger diameter J ~ m r^2. Even if the masses were the same, the relatively heavy tyre and the rim are further away from the wheel hub.