Rule of thumb I learned and never validated:

Two it starts, with three it can become an issue, with four it might affect the underlying permanent teeth - and there is no difference between thumb or pacifier.