The training is sophisticated, but inference is unfortunately really a text prediction machine. Technically token prediction, but you get the idea.

For every single token/word. You input your system prompt, context, user input, then the output starts.

The

Feed the entire context back in and add the reply “The” at the end.

The capital

Feed everything in again with “The capital”

The capital of

Feed everything in again…

The capital of Austria

…

It literally works like that, which sounds crazy :)

The only control you as a user can have is the sampling, like temperature, top-k and so on. But that’s just to soften and randomize how deterministic the model is.