If you place a tea bag in a cup of water at 20C in a thermally isolated vacuum chamber, when the chamber pressure is reduced to or below the vapor pressure of water at 20C (about 17 torr, or 1/3 psi), it will begin to boil. The vapor produced will be at 20C and the water in the cup will be 20C and begin to decrease, because of latent heat of vaporization needed for the liquid/gas phase change. The water will continue to boil as long as the pressure is maintained at or below the vapor pressure of water at that temperature. Eventually, the water reaches 0C. Then it will stop boiling and begin to freeze as the latent heat of fusion provides the necessary heat to continue evaporation. When all the water has converted to ice, the vapor pressure is greatly reduced. The ice will sublime (go from solid to gas) still, but as that continues to cool the ice, the vapor pressure also drops. As the temperature drops, sublimation will slow until it is nearly zero. So you would end up with a tea bag encased in ice.

In your example, if you suddenly exposed to the cup and tea bag to the vacuum of space by rapidly venting the air, the water would explosively evaporate, shredding the tea bag. You’d be left with bits of tea leaves, an empty cup, and a lot of very fine ice crystals.