Is this on linux? Maybe you need a blacklist udev entry on the HDMI audio device? Could be the hotplug stuff running when you turn the external display on.