I’m guessing the walls are retaining some heat.

Guesstimate your own numbers, but the way my flat is built walls have heat capacity some 150x more than air inside. (reinforced concrete building, walls/floor/ceiling counted as half if shared with other flats) You will need to run a lot of air through, at at least couple C temperature difference, to make a dent in that