There are some flaws in the common understanding of survival of the fittest, but I will ignore that for now.

Poison is usually energy intensive to make, so there is an investment from the mushroom in being harder to eat. If there is not a strong pressure from predators, or the mushroom adopts a different strategy like making tons of fruiting bodies (mushrooms), it might not be worth the cost to make the poison.

Finally, poison doesn’t automatically win the evolutionary race. We eat tons of stuff that the plant/mushroom intended to be a deterrent or poison (why are peppers spicy?).