If you are truly starting from scratch, shooting for Raspberry Pi performance isn’t starting small, thats a huge goal. It’s a complex chip built on a fairly modern process node (28 nm for the 4B) using the second-best-established architecture.

The reasonable goal to shoot for would be an 8086-like chip, then perhaps something like a K3-II or early Pentium, then slowly work your way up from there.