Is there an option under Preferences… to turn off notifications entirely?

Maybe turn off, shut down Slack and reboot your Mac. Then turn Notifications back on…?