Federal election times are set by 2 U.S. Code § 7 as 1 day after the 1st Monday in November (of even numbered years). The law is from 1875 and from what I can tell is indeed nominally motivated by the voters’ need to first observe rest day on Sunday and then travel to their polling place. Keeping it and not having a federal holiday coinciding with it is largely aimed at keeping voter turnout low.