While what you said isn’t wrong, it’s not really the main issue. The energy a human body gets from food can be vastly different than what is produced by burning it, and there are further variations per person.

The calorie count on food to my knowledge is based on actual measurements with humans… from one guy doing experiments in the 1800s. And while it’s probably reasonably accurate on average, it’s not really possible to know how much energy a specific person will get from a food from a generalized calorie label. So even if the food itself had no variance, it would be impossible to label the energy intake you will get from it accurately.