Would it look stupid to place the cap on the handlebars

No.

or clip the cap into the strap of the helmet to carry it while riding?

Yeah kinda, I’m imagining it hanging from your face?

If you have a snap back kind of hat, yeah just attach it anywhere on the bike I’d say, or depending what you’re wearing, to you in some way. Don’t over think it.