If I saw someone wearing a helmet, with cap strapped on somewhere, I’d understand. It’s practical, and it’s pretty clear that your planning to wear it when you remove the helmet, you’re not trying to make some sort of statement. Go for it! (as long as your cap strap is reliable…)