To get into the sun, we’d probably want to fuel the rockets in space using reaction material mined in space (from the moon or an asteroid). That would more or less eliminate the problem you’re talking about, which is why I kind of skipped over that in my comment. But you’re right; this is one of a million things that makes space travel hard and expensive.

We can get up to any speed with enough time and fuel. The trash rockets would just need to get into a solar orbit, and then burn retrograde for a fairly long while. Or if you add a gravity assist in, this is doable today; the Parker Solar Probe got to (and indeed beyond) that speed, for instance. It’s easier and quicker when there aren’t squishy people aboard (we don’t tend to like acceleration much higher than 9.8m/s², for instance).