My memory is terrible too, but I pretty much only use those phrases when I'm 100% sure I do recall. The rest of the time I feel like it isn't worth speaking up at all.