A SAML token verification can be implemented correctly in under 50 lines of code. (Without needing anything beyond a basic crypto library for decryption and signature checks ) then you just have a SAML identity to user account mapping table (so that they can have multiple SAML providers and retain access or switch between different accounts).

But yeah, some shady sites use it to get your name and other information. (Which SAML providers should properly inform you about, as they are the ones packing that data based on what the receiving has registered)