The implication of the phrase is not that trust is variable depending on the subject’s or object’s physical abilities. Its meaning is more that one doesn’t trust the other person’s motivation, so the trust only goes as far as one’s own ability to affect the other’s figurative position ("throwing” them.)