I agree it’s safer to let them pass, but a medical (or personal) emergency does not give you the right to endanger other people on the road by driving fast and/or recklessly. That’s why they paint priority vehicles in bright colours and put flashing lights on them - to make it safer for everyone.

If you have a medical emergency, you call an ambulance. Yes, they will have to drive to you first, but care starts when they arrive. If the emergency isn’t big enough to get an ambulance, there’s no reason to drive fast either.