What ultimately stopps us is the fact that the universe is expanding infintly. In the long run more matter is moving away from us than towards us. And this is true for everywhere in the universe. All atoms will eventually become subatomic particles that are increasingly less likely to interact with one another due to them moving away from one another.