I’m testing a few AI-based narrative engines for solo RPG experiences and I’m wondering:
How do you handle world-building coherence over time when the AI generates large parts of the plot on the fly ?
Have any of you tested systems that are actually able to remember past events or characters effectively ?
It has been multiple years, but there was a time when I used to enjoy playing around in this website called “AI Dungeon”. There never really seemed to be a way of getting the AI to remember events beyond the very recent, so you could—either—go along with the nonsense to have a very funny adventure or diligently edit and redo the AI’s prose to get something coherent—which at a certain point just became writing with extra, unnecessary steps. I have no idea how recent models fare, but that is my experience.
I think the knowledge window of AI is going to be a persistent and largely unsolvable problem. If you bake out stuff that the AI makes up, you either lose the flexibility of AI at that point, or try and get the AI to use the baked stuff, but then you get back to the original problem. It would take a new design of AI - which I think is entirely possible, but not likely to happen to serve this niche activity. Perhaps smaller scale custom AI designed for this specific task is the way forward, but that is almost certainly out of the scope of what can be done without megabudgets.
I have a lot of contrarian opinions about this, but the main one is this:
LLMs are very, very good at 1) understanding abstract commands, 2) reasoning at near-human levels about how they might affect the game world and plot, and 3) writing dynamic description that reflects those effects. These are precisely all the things that traditional parser fiction is bad at, so there must be a way to harness that power to deliver a good IF experience. But the answer is neither slop (letting AI write all your text), tweaks (slapping an LLM on top of existing parser systems), nor hallucination (letting the AI invent the whole story). Rather, the fundamental breakthrough will come when we figure out how to harness LLMs to run just the parts of IF that LLMs are good at while keeping human authors firmly in control of the parts of IF where we want human artistry.
I have done enough experimentation with this to be convinced that this is completely possible with existing LLM technology (and will only get more possible in the future). It’s just a matter of figuring out the right parts of your IF system to hand over to an LLM, the right ways to query it, and the right ways to filter the output. I imagine a myriad potential hybrid systems: for example, a conventional parser controls the world model while an LLM runs an overlying social simulation, with opinionated NPCs the player can talk to in full English sentences.
This approach will unlock fundamentally new types of interactive experiences, much as Twine is suited to fundamentally different kinds of interactive experiences compared to Inform. Conversational social dramas may be one such “killer app.”
My other main contrarian opinion is that pretty much nobody will believe this is possible until someone does it, at which point it will seem obvious and inevitable.
LLMs are very, very good at 1) understanding abstract commands, 2) reasoning at near-human levels about how they might affect the game world and plot,
If I may add my own contrarian views, albeit grounded in technical fact (source: I have written patent applications on AI-related inventions, amongst other things), LLMs do not do either of these things. An LLM is simply a pattern matching device, finding the most plausible output for a given input. It sure looks ike clever reasoning sometimes - you can (to a degree) thank the cleverly reasoned data it was trained on. Beyond that, there is the hard technical limitation I mentioned of the context buffer always being finite. This is why the circle will never be squared to give a 100% reliable result for reasoning tasks.
Linear improvements in performance require exponential increases in complexity and resources. We are nearing the limits of that. Furthermore, better models need more data to train on. That is also a finite resource, not least because it gets harder to find pristine training data that does not inadvertently include ‘poisoned’ data which was output by AI instead of humans.
None of which really matters, but this is why I think AI generated content for adventure games is always going to require compromise and limitations of some sort.
AI slop has essentially “poisoned the well” by filling the internet with garbage. I mean AI garbage as opposed to actual human garbage, which was already plentiful. AI can produce garbage far faster than we humans can.
Please, please, I beg of everyone in advance, do not turn this into another thread hundreds of posts long arguing about AI.
Long-term coherence is also required by people who write longer stories or ongoing series, and while there are many different techniques a human can use to track what came before so it can be used to help with the level of continuity of what comes after, many of those techniques rely on a consistent concise summary of some sort that includes just enough details about the “important” things to keep that level high.
eg. some Authors keep detailed notes about characters, the environment those characters exist in, and what has happen to those character.
An LLM is similar in this manor, it uses the (ever increasing?) content supplied via the “prompt” to determine what comes next. And general the larger that supplied content is the more likely the LLM will hallucinate, assuming that the bias of the LLM introduced via its training data remains consistent over the time of that LLM’s usage.
A similar issue exists in the Software Development usage of AI to generate code for the most popular programming languages, in that the larger the code base supplied to the LLM, the more likely the output will contain inconsistencies. One technique being trailed/used with some success is to supply concise specifications of what functionality the code base requires, so the LLM can use that information when “making things up”.
So if a well structured concise summary of what the LLM has outputted can be generate on an ongoing basis, then it can supplied to the LLM as the equivalent of a software development specification, which may help the coherence of the output to remain higher for longer.
(Yay! Another technical contrarian! We have way too many threads about AI slop and not enough debating what else it might be capable of.)
Whether LLMs are really “reasoning” is largely a theory-of-mind question, but that doesn’t mean their output can’t still be useful. Consider the following (schematic) exchange:
PLAYER: Tell Isabella “Wow, I wish I had the confidence to go out dressed like that.”
GAME ENGINE TO LLM: Based on the provided context and state information, assess how this action affects the mood of any characters present and respond with a summary justification and database update in the specified JSON format.
LLM: OK, we know that Isabella tends to be insecure, has been at odds with the player before, and is dressed casually, so in context she would probably interpret this as a subtle insult about her clothes rather than a compliment about her confidence. {“character”:“Isabella”, “mood”:“insulted”}
Does it matter if/how the LLM is actually “reasoning” so long as it still provides useful output? This kind of free-input sentiment analysis is impossible in Inform, but trivial for an LLM.
This is not what the data show. In the early days, the prevailing orthodoxy was that smarter models would always require more parameters and more data, but this has not actually turned out to be true. Benchmark performance and context length for models within the same weight class (e.g., 20B parameters) continues to improve, and there’s a strong industry pressure towards improving small models because they’re cheaper to run. You can now run LLMs locally on your smartphone (Qwen3 4B, Gemma 3n, etc.) that beat the pants off of the original ChatGPT in logic ability (although they usually lack niche subject knowledge).
We may be reaching the limit, but I also stand by my claim that current LLMs are already capable of good IF experiences. It’s an architecture problem rather than an intelligence problem.
Agreed; but the same is true of traditional systems. We need to be asking what new experiences AI can enable rather than dismissing it because it’s not a panacea.
To be clear, I am not having an argument about AI but trying to focus on the pragmatic issues with making adventure games with AI. Pattern matching involves no logic and follows no rules. Call that reasoning if you want, but the ways in which it is not are germane to the challenges of producing consistent and repeatable output based on a set of starting conditions.
Prompting a chatbot to do sentiment analysis seems like a roundabout way to do it. Why wouldn’t you use an algorithm designed for that task, possibly implemented in Inform?
But even if you did do this, there wouldn’t be much point in accepting freeform input and analyzing it only for sentiment. You might as well just implement an INSULT OBJECT command.
The real breakthrough will have to be an advance in world modeling, or NPC modeling. Nothing was stopping anybody from parsing freeform text or analyzing it for sentiment. That’s not the hard part. The problem was, as it remains, what to do with your analysis. That is, what kind of world model do you need, in order to work with richer information than medium-sized dry goods entail? If we’re giving up on using the LLM’s context window itself as the world model, then talking about LLM applications at this stage seems like a hammer looking for a nail.
This comment is so close to getting what I’m talking about, yet so far away.
Sure, you can implement INSULT ISABELLA, or let the player choose from a list of dialogue options with predetermined outcomes, or do procedural sentiment-analysis to see if their input is (for example) more positive or negative. Those are all valid approaches that don’t require an LLM. But they’re a very different experience than what I’m envisioning:
Lady Agatha sits across from you, sipping from her glass of sherry without a hint of pleasure. You can tell she is none too pleased by your plan to marry Lawrence (a commoner? bah!), but if the marriage is to happen, you must have her blessing—or at least, her silence.
> AGATHA, WHAT DO YOU THINK IS IMPORTANT IN A MARRIAGE?
She raises her eyebrows, clearly annoyed at where this is going. “It is important that each party bring something to the table,” she says firmly, gripping her sherry-glass tightly. “An unequal marriage only leads to ruin and disgrace. Like your uncle Peter,” she adds.
> PETER SEEMS TO BE DOING JUST GREAT.
Lady Agatha guffaws. “‘Just great?’ I see Lawrence’s uncouth grammar is rubbing off on you. In any case, if you call Peter living practically penniless in Wendover ‘great,’ I suppose I can’t help you. I heard he sells insurance. Insurance!”
> I MEAN THAT HE SEEMS HAPPY.
Lady Agatha looks down into her glass and doesn’t respond for a long moment. She fidgets with her lace napkin, the heavy rings on her fingers clinking faintly in the silence. “Well…” she says at last, but does not continue.
> AGATHA, ARE YOU HAPPY?
Really think about what it would be like to play a game like this—navigating complex social situations in a battle of wits against autonomous NPCs who react to even the way you phrase your statements. It’s a totally different kind of interactive experience than what can be achieved with conventional parser IF.
Where I agree 100% with Paul is that the central challenge is not getting LLMs to analyze commands (that’s a simple API call), but what to do with that analysis. For the above example, the LLM must obviously handle a fair amount of the world-state and writing beyond simple sentiment analysis—yet without descending into directionless hallucination. Getting the balance right is difficult, but, I think, far from impossible.
I think you’ve been explaining your concept clearly. My point is that if the finer points of your input don’t “stick” in the world model, nothing has been gained over an INSULT command. On the other hand, a game that that could thoroughly model the results of “INSULT ISABELLA’S CLOTHES” would itself be a big advance. Maybe generating a summary to include in the prompt, as mentioned, can serve as something like a world model, but I think the world model isn’t the place to cut corners.
Yeah, that’s the issue I’m not sure can be surmounted.
LLMs, by design, hallucinate. It’s an unavoidable consequence of the architecture. This makes them very bad at maintaining a world model.
This suggests they’d be better at front-end tasks. But the reason parser systems don’t currently support INSULT ISABELLA’S CLOTHES isn’t the parser, it’s the world model required for such a thing. A front-end that can handle AGATHA, WHAT DO YOU THINK IS IMPORTANT IN A MARRIAGE? isn’t helpful without a backend that can meaningfully represent that in the world model.
I’m happy to be proven wrong on this, but fundamentally something needs to act as a world model on the backend (otherwise you just have AI Dungeon and that’s already been done), and I’ve never seen compelling evidence that an LLM can do that.
It seems obvious to me that some kind of LLM will play a role in any kind of advanced language processing going forward. The ability to, through sheer free-association from the context, generate appropriate output, is far beyond what previous technology was able to do.
This is an advance similar to AlphaGo’s advance in board-game playing, taking Go engines from wimpy to superhuman in just 2 years (counting from the start of the project to its victory over Lee Sedol). The difference is that AlphaGo’s activity is intimately tied to the world model, by virtue of the fact that it’s playing a game with a defined state and rules.
I suspect that something of this kind will be necessary to apply LLMs effectively to IF. That is, the world model will need to be part of the LLM’s input, and, importantly, its training. Trying to hack that information into prosumer chatbots via textual prompts, I’m skeptical of.
Not sure if this is stating the obvious, but if you really want a “full” social simulation - one that simulates human characters with the same degree of interiority as a human - then what you’re asking for is an AI that can fully simulate a human’s interiority, i.e. an actual human-level intelligence. I feel like it’s easy to lose sight of that when handwaving about an LLM hypothetically being able to develop a sufficiently complex world model or whatever.
To this I would say, start less ambitious. If someone can get a game to basically do npcs that respond usefully to natural input in a way that’s relevant to the plot, I would consider that a huge victory. But as I think you were insinuating, you could probably pull this off with ChatScript.
I am thinking about it, and I’d like to share my opinion, which is only my opinion and has exclusively to do with what I look for in games.
I think that you’re trying to make God’s piano.
This is a direct reference to a film called The Legend of 1900 (the one with Tim Roth about a pianist born on a ship). I’d provide a direct quote, but I think the film is worth watching, and I wouldn’t want to spoil it. If you don’t want to watch the film, the quote is easily found on IMDB. The point of the quote is, a pianist can make infinite music within 88 keys; that’s his piano to make music on. But give him a piano with infinite keys, and there is no possible way he can make music. That’s not his piano, that’s God’s piano.
So: this is going to be contentious. There will be arguments along the line of “sure he can”, and “ it just seems like he can’t because he’s formatted to think that way”. I accept these. I wish well to people to explore these options. I, however, agree with Tim Roth’s character.
The search for this type of unlimited parser input is not new, as we all know. Galatea is what comes first to mind, and as I understand it Emily Short has delved on this possibility for a while. Some parser systems, commercial I believe, tried to achieve this illusion by pretending to understand more than they actually do (a failure, I believe).
I try to envision playing a game like what you quote, and I go, “nope. I’m out.” That’s way past overwhelming. This is a personal reaction, and I imagine that people who like open worlds, emergent gameplay and stories, and that type of thing, will be all over it. To have no limits like that leaves me floundering and drowning; if I can do and say anything at all, and it all has a weight on how it plays out later… how do I play that? How to I visualise the path to winning the game successfully?
Of course, maybe it’s not about playing or winning. It’s about the experience. A whole new concept. A whole new paradigm. Make a game so rich that no two people will have the same experience. Possibly even that one person, playing twice, will also not have the same experience.
I just wonder… how much of a good thing is that, really? Once you get past the hype. For one thing, how do I replay that to relive something that I really enjoyed? Also, how much do all those things actually change the experience? And if I find an ending which I find conclusive, why would I go back to find any other endings? The most recent game I played which seemed to go wild with the choices (apparently; I didn’t replay) was the sequel to Dreamfall (don’t remember the name; Longest Journey 3). It kept telling me all the time that it was remembering choices, and it gave me the impression that I could have radically different experiences by choosing something else.
Now, I certainly don’t represent every gamer when I say this, but neither do I think I’m alone when I say that I make the choices that I do because I am the person that I am; and with a few exceptions, I’m not all that likely to make different choices on subsequent playthroughs. And if I do, I’ll start feeling uncomfortable, because I’m going ahead with choices I don’t believe in just to see different paths. When the game then puts the responsibility on me for making those choices, if it’s the first playthrough, I can own that responsibility; but after that, it’s increasingly uncomfortable.
I may be getting too far off-track, let me get back a little bit.
A game like the transcript you provide is something I would run away from. But I’m sure there are many who would embrace it thoroughly. So it’s worth pursuing, in the knowledge that it’ll be a hard pass for a certain number of people. Then again, many things are; you can’t please everyone.
The games I personally love, and the ones that touch me, and the ones that give me the greatest fun and joy and which move me, and which remain in my memory and my heart, are all, every single one of them, pianos with 88 keys. None of them have, or pretend to have, infinite keys. I can play music in them; that includes knowing the highest and lowest pitch, and the highest and softest dynamic, I can reach in them.
In game terms, this means knowing the limits of the game; the verbs it accepts and doesn’t; the type of syntax it understands.
You know something? Conversation in games is amazing, because so often those PCs are saying stuff I would never think to say. But of course they are; they know the context so much better than me. And they have a personality of their own, which may make them say stuff I wouldn’t dream of. Galatea’s attempt to work with this has the player be “you” with blank-slate knowledge, and even then I have a hard time making any sort of natural conversation that doesn’t end the same way every time (because when I try to deviate from that, it doesn’t feel like a natural conversation any more, because this is the way I talk and this is the way I pursue a conversation).
I think it’s time to do a TL:DR, or a couple of them.
It’s worth trying, but I suspect it will be polarizing.
The more options you give me, as a player, the more likely it is I’ll never see them (this may or may not be true for others; I often don’t see reviewers mention this, but I can’t imagine I’m alone).
EDIT - I can see such an exchange being possible using a dialog tree. And I rather prefer it like that. That’s just me, though.
Thank you for this extremely thoughtful (and thought-provoking) analysis. I think you have a lot exactly right here, which is why I care so much about LLM-powered IF maintaining authorial intent, and why I don’t expect LLM-powered IF to erase traditional IF any more than the Twine revolution erased parser IF (or, for that matter, parser IF erased traditional fiction).
I would amend this, however, by expanding the “God’s piano” metaphor into a continuum. The lower end is a keychain ornament of a piano that has one button and plays one tune; call this traditional linear fiction. In the middle is an 88-key piano; call this traditional IF. On the far end is an infinite piano; call this AI Dungeon, or another similarly undirected lucid-dreaming-like experience.
Every reader has their own preference on this continuum; but I think this community tends to underestimate the potential appetite for experiences that fall somewhere between “88 keys” and “infinity”. (A microtonal piano with 176 keys, to torture the metaphor.) There are whole communities dedicated to lucid dreaming. AI Dungeon, for all its limitations, is a successful company with tens of thousands of regular users—and they’re actively beta-testing more author-driven experiences. Tens of millions of people play TTRPGs, which is an experience that similarly blends authorial intent with player freedom and natural-language interaction. The current size of the nascent market for LLM-based character-chat/roleplay tools and LLM-driven TTRPG GMing already dwarfs the market for traditional IF.
I’m very sympathetic to people saying “that sort of IF wouldn’t be my cup of tea.” I’m less sympathetic to people saying “that sort of IF is technically impossible” or “nobody would play that sort of IF.”
I’m curious, if the “AI” is supplying the response to the reader’s / player’s input via a LLM, and potentially the audio and visuals via other generative services, and maybe even the code for the project’s “application”. What exactly is the “Author” providing, besides clear & concise specifications for the “AI” to make its “decisions” on?
Because It sounds more like the Author is a systems / business analyst, whose supplying a concise but detailed functional specification, than an “artist” creating a creative work…