Both washer and dryer can cause shrinkage, but the dryer will cause more. Also natural fabrics will shrink more (by far) than polyesters. To your comment "I'm not gonna handwash. That's just too much." Well no one can blame you for that. But it's still true that the gentler you wash your clothes, the less shrinkage you'll get. It's a balancing act, how much effort you're willing to put in vs how long you want your clothes to last.

Personally I'm a natural fabric addict, there is very little non-natural fabric in my wardrobe. And I also do hardly any hand washing, but not zero. I often use a gentle cycle on my wash machine (top loader, sadly). Any garments that I particularly want to protect go in a garment bag for washing, and do NOT go into the dryer. The bulk of my wardrobe gets dried but on low heat, and pulled before it's quite all the way dry, because a lot of shrinkage happens as the garment goes from barely dry to fully dry. In particular my knits get the low temp dryer, and also any clothes I consider semi-delicate, particularly well loved or barely big enough.