Architectural improvements could help, but the big guys can’t even get away from “basic” problems like high temperature sampling. Corporate development is way more conservative than you’d think.

And it’s not getting better. See the “all star” ego teams and Meta, OpenAI and such vs. those that quit.

The Chinese are testing some more interesting things (and actually publishing papers on them), but still pretty conservative all things considered. They drrm content with LLMs as modest coding assistants and document processors, basically.