Pin media player is the setting name to search for.

Settings -> Sound and vibration -> Media -> Pin media player

Maybe theres a way to fix it but it just sucks ass so I turned it off.