Well, what I’m working on is a mod for STALKER Anomaly, and most large models already seem to have good enough awareness of the STALKER games setting. I can imagine it’s a much bigger challenge if you’re making your own game set in your own unique world. I still need to have some minor game information inserted into the prompt, but only like a paragraph detailing some important game mechanics.
Getting longer term interactions to work right is actually what I’ve been working on the last few weeks, implementing a long-term memory for game characters using LLM calls to condense raw events into summaries that can be fed back into future prompts to retain context. The basics of this system was actually already in place created by the original mod author, I just expanded it into a true full on hierarchical memory system with long- and mid-term memories.
But it turns out creating and refining the LLM prompts for memory management is harder than implementing the memory function itself!
lemmeLurk@lemmy.zip 1 week ago
What helped me in my project was a RAG implementation. So instead of trying to have all the information in the prompt, which becomes inconsistent if it becomes too big (especially when working on local llm), I can have quite a big knowledge base in a RAG. Then when being promoted it’s a two step process, first search the RAG for the best match in the knowledge base, then Feed that to the LLM to generate the answers. You could also store noteworthy stuff in the rag on the fly.
Coelacanth@feddit.nu 1 week ago
That’s very interesting and sounds cool - though might be above my paygrade.
What kind of delay does the double prompting incur? I can’t imagine it would be nothing, especially for this use case of calling external LLMs via a proxy.
If anything a RAG for certain game related information might be a nice addition to the current system, rather than replacement. I think for a roleplay and storytelling focus there is a benefit of having a persistent character “life story” of long term memories fed into every prompt, creating character throughlines and potentially even character development.
lemmeLurk@lemmy.zip 1 week ago
I haven’t tried with API yet. With local one it’s super quick though, it adds maybe a couple of seconds at most.
If you have a GPU and want to try it with local one there is a plugin for Godot and unity called “Nobody who” they have implemented the RAG approach out of the box with examples as well. So it wasn’t something I came up with.