Yes and no. The gravity of the sun will attract the rocket, but there are other things out in space besides the sun.

The problem then is other planets will start whipping the garbage rocket around who knows where. Could even come back around and smash into earth. Same problem with the sun, actually. It’s quite hard to hit something that’s that big when we’re this far away. If you miss even a fraction of a decimal of a degree, the trash rocket will swing around and you’re back to planetary hot potato.

It’s easier to sling the rocket past the south or north pole at a right angle to the solar plane. Up or down it’ll either keep going till it’s another suns problem or it joins the Oort cloud, which is kinda like a giant trash dump for everything that didn’t make it into our solar system when the sun formed.