Or the magical withdrawal he must have been going through for at least several decades after Bilbo got it