Depends a lot. If you are going from 2 ram slots in use to 4 ram slots in use, usually the max clock speeds go down a lot. So the performance will decrease for just about everything you do, whilst the use case for such a setup is very limited.

I have a couple of extra ram sticks to get from 32 to 64gb when I need it. I bought them because I was debugging a rather memory intensive tool. Not only did the tool run in debug mode, which added a lot of overhead. The memory profiler needed to be able to make memory snapshots and analyze them. This just about doubled the memory requirement. So with 32GB I often ran out of memory.

However my Ryzen 5950X does not like 4 sticks of ram one bit. Timings need to be loosened, clocks need to be reduced and even then the system would get unstable every now and again for no reason. So I pulled out the 2 sticks going back to 32GB as soon as the debugging job was done. They are in a drawer in an anti static bag, should I need them. But for day to day 32GB with 2 sticks is a much better experience.