The first ammendment isn’t unlimited there are a fair amount of restrictions. The ones that are allowed follow strict scrutiny, which is basically a two part test. There must be a compelling state interest, AND the law must use the least restive means possible to achieve that interest.

Most these book bans likely don’t meet that standard. They are allowed to exist because they are considered valid until ruled otherwise. To overrule these bans it would be required for someone to have standing that they are materially harmed by the ban. This is how most bans will stay, as proving someone has standing is pretty difficult in this case.