You can do nuclear without irradiating everything just like a nuclear plant doesn’t irradiate everything. The only reason it isn’t done is safety. Rockets fail too often.

The argument that gravity is in anyway a more powerful force than weak force (fission) or strong force (fusion) is wrong. The only thing gravity has is distance. The strong force is 100 trillion trillion trillion times stronger than gravity.