The mother’s love protecting Harry caught Voldemort off guard. It’s not like he got to try using the killing curse on Harry several times. Imagine you want to kill a baby, and you have a gun, and literally no reason to conserve ammo. You wouldn’t pull out your knife in the off chance that the baby is bulletproof.