Because our perception of ninja is based around kabuki theater and not historical accuracy. Kabuki theater stage hands were always dressed in black to blend into the background. When an ninja assassination would happen in a show, the “stage hand” would do the assassination because the audience knows to kinda ignore the stage hands during the production.

IRL ninja would just look like normal people, waiting for the right opportunity to strike at someone.