Did you forget to unfurl your sails? I did that my first time… It’s definitely slow traveling between islands but “go make a sandwich” slow sounds like something is wrong.

Once you beat the first boss you can start putting engines on your ship, which make it faster and not dependent on wind direction. Also making your ship lighter helps too.