Is there something technically forcing the privilege elevation to install a driver ?

Yes. With few exceptions, drivers need admin permissions to be installed. In part that’s because they need admin permissions to run, and malicious drivers have absolutely been exploited in the past.

Some hardware (e.g. mice, keyboards, storage) don’t need additional drivers to be installed, but that’s because the OS uses generic drivers, or has a whitelisted source (e.g. Windows Update)