It is a figure of speech in the sense that voltages are almost always being discussed between two things, and the figure of speech part is that “ground” or “earth” is implied in most, but not all terrestrial cases. Especially in microchips, which like you said have very limited safe voltage ranges, it is very important to give them a stable and consistent reference to ground whenever possible to make sure static voltages don’t develop on the other side that can damage them internally.

Technically, if you’re referring to voltages on an absolute scale (not relative to each other) you’d typically be talking about “electron-volts” which is why powerful electro-magnetic radiation like x-rays, gamma rays, and other cosmic phenomenon are often measured in mega-electron-volts (MeV) for example. This is useful in physics, but just like we don’t use “absolute temperature” in day-to-day life, it is essentially meaningless to us standing down here on the crust of our pitiful little planet made mostly of molten rock covered in conductive water.

So in day-to-day life, we talk about voltages relative to the closest thing we have to a fixed reference point, and the “zero point” we rely on which is mostly a decent reference to the voltage of Earth’s own water table tends to be reasonably consistent and practical for our purposes. Static electricity is what happens when a particular object starts to lose its reference to ground, and large voltages (positive OR negative) can build up RELATIVE to ground, which then sparks when it finally equalizes back to our stable reference point. When large static charges build up in the atmosphere, they too eventually equalize with ground through lightning strikes. This is why “ground” is important to us. High voltages relative to ground can be very dangerous. Even small voltages relative to ground can be damaging to sensitive electronics. There is nothing special about “ground” voltage in particular, in fact it changes constantly with the weather, and solar output too! The important thing about it is that almost everything on this planet is connected to it, either through humidity, water, plants, the moistness of soil and even rock has enough conductivity to maintain a stable connection to Earth’s average voltage. And of course we’re going to use that as a reference and baseline because it’s super helpful for us. We don’t care how many electron-volts the Earth has. We just care that almost everything is connected to it in some meaningful way.