It’s about the change being important.

If we stay on a single “time” (say standard work day starts at 4 hours before solar noon), we’ll drift the work time to start/end the work day at some appropriate time.

The issue is that what that time is that most socially useful changes greatly over the year unless you live close to the equator.