I don’t know if offgassing is the reason the water tastes better when cooled overnight. I would do this with an enclosed bottle (no off-gassing possible) and it would taste equally better.

Definitely cooling it is an improvement, I always thought it tasted different due to how our mouth/taste buds responds to the dropping temperature.