Something in soy can reduce iodine uptake into the thyroid (or utilisation of iodine in thyroid), thus reducing testosterone levels.

I learned this outside of the soyboy-memes, which joke about men who consume soy have low t.