Sounds like a bug, I had all four hearts before I went to get the flower, the only one I still have is the secret the secret one from the green prince’s area which turns into a thing for your house.