The element with the highest atomic number among all elements except hydrogen, helium, lithium, and beryllium is boron, which has an atomic number of 5.