Something like the steam deck or the original switch were probably on the upper end of meaningfully “portable” in that sense, and even they can’t really compete with smartphones on that front. But with the currently available chips/batteries/screens, you cannot really get much smaller without starting to limit the games that can be played on them.

There is a whole other conversation to be had about game optimization and the push in large parts of the games industry towards more power intensive games. If the PC/console games space had an incentive to better optimize for lightweight devices, that could change. Especially if something shifted on the smartphone storefront market that created more demand for better less exploitative games there.