You can believe that people stuck in wheelchairs deserve equal access to public spaces via accommodations like ramps or elevators, and still hate Ted, who happens to be paraplegic, because he is a hateful, racist, fascist MAGA jerk. In this case, your shunning of Ted is not ableist because the reason you treat Ted differently is unrelated to his disability and you are well aware of the fact. He doesn’t get a pass for things unrelated to his disability.

However, if you are aware of someones disability and treat them prejudiciously because of it (or it’s symptoms) then you are the ableist jerk. If you are the one with a disability and people treat you in a prejudicious manner because your disabilities or their symptoms, then they are jerks and not friends.