This exists! There are Z-machine interpreters for audio devices like Google Assistant. However, the ecosystem and usability for this could still use a lot of work. I’d particularly be interested in a system for playing hands-free while driving.
1 Like
Sounds dangerous 
I would prefer walking.
But great they already exist! I should try it sometime…
1 Like
Triple-A studios will probably perfect chatbot NPCs long before IF does. If nothing else, the big studios can afford to either run their own server side models and hire their own AI specialists or to contract with the Big AI companies whereas IF is, best I can tell, dominated by solo hobbyists. Short of someone with a passion for both AI and IF working on the problem, we’re kind of stuck waiting for FOSS local LLMs to reach that point, and then we still need someone to figure out how to get major IF engines to integrate a local LLM.
That said, all else equal, I’m not sure legitimately Smart NPCs would be that big a revenue booster. Seems unlikely that someone would buy a game where the NPCs have perfect AI dialog but wouldn’t buy the exact same game with NPCs that had traditionally scripted dialog trees.
Though, +1 for If with human narration or even voice acting for NPCs. Also +1 to the option to enter commands via voice and +1 to more console games having keyboard support(Unless I’m mistaken, most, if not all, home consoles since Gen 7 have had USB ports and Bluetooth, there is no reason plugging in a generic USB or bluetooth keyboard shouldn’t work. Wouldn’t want IF where voice commands were the only option though.
3 Likes
I’ve talked to Claude (GenAI) about using LLMs to generate code from a design. It seems to think this is workable with one caveat. It thinks you need two LLMs, one for design and one for code generation, training the latter on a large section of existing IF material.
I’m nearly certain the IF Community would take issue with using the archive for training an LLM. Have we set policies on that?
But I still think without the IF trained LLM there are a ton of interesting use-cases. I think using a small LLM embedded in a game might improve help and hint capabilities and help extract context and intent from a poorly worded command. It would have to have very tight guardrails.
These are interesting times.
I’m not convinced this is even theoretically possible, though. We have all of George R R Martin’s existing A Song of Ice and Fire books, so why hasn’t anyone just had an AI write the upcoming ones to skip the wait? Well, because we want to see GRRM’s ways of answering the open questions that the story has raised, not someone else’s. That takes a sort of creativity that I don’t really think is feasible to delegate to an AI.
In an AI-driven open world game like you describe, imagine that you (as a player) tell the AI you want to investigate how, idk, grain supplies work in this world, something the author hasn’t specified. Now what can happen? Either it hallucinates (“magic grain was invented five thousand years ago by Harkis the White Mage!”) or it stays within the bounds of what the author has specified in a deeply unsatisfying way (“I know nothing about grain supplies in this world, try something else”).
11 Likes
After it’s ingested the old school parser snark of IF Archive, it’s more likely to say
If you love grain supplies so much, why don’t you just marry them?
-Wade
10 Likes
I’m skeptical that using AI to make games responsive to more situations would pan out that well. Fundamentally the limitations of realism in a game are the limitations of its world model. An AI which hallucinates content that has no representation in the world model is not going to be satisfying. That’s AI Dungeon all over again. So maybe limit the AI to actions and conversation about things which actually exist in the game’s world? Ok, how interesting would you find talking to somebody who knew so little?
3 Likes
Fanfic already exists. What would the appeal of fanfic without creative input be? GenAI can be entertaining as a novelty, but that runs out very quickly.
4 Likes
The way I see it, this is a game design problem, not an AI problem. Parser IF is very good at certain things and very bad at others. Hypothetical LLM IF would be very good at different things and very bad at other different things. We’ve learned how to create satisfying IF that leverages the strengths of the parser while avoiding its pitfalls, so why wouldn’t we be able to do the same with LLMs?
The grain-economics example is a good one. To some degree, LLM-IF writers are going to have to accept and design around the fact that some degree of hallucination is a feature, not a bug. Separate playthroughs will never have 100% continuity, but they don’t necessarily need to in order to create the author’s intended experience. There are also ways to improve this: I’ve had some success maintaining consistency of worldbuilding by describing the edges of the defined world schematically, with reference to real-world subjects that are part of the LLM’s knowledge base. E.g., if you ask the same complicated economic question in two different playthroughs, you might not get the same answer, but you will get more similar answers if there’s a line somewhere in the rule chain like “the economic system resembles feudal Bulgaria” (plus a COT loop that catches and corrects out-of-place references to real-world people and events). (This approach works best with large cloud models with a larger knowledge base than what you can run on a consumer GPU.)
My corollary spicy hot-take is that this is going to continue to be a “put up or shut up” situation, with the majority of the IF community not believing this is possible or understanding what I’m talking about, until someone actually releases a working game and proves them wrong.
2 Likes
My broader point is, I think people are getting stuck either imagining (1) free-form hallucination without authorial intent, or (2) LLMs being wedged into traditional parser IF without much benefit, and failing to see the possibility of completely new and fundamentally different forms of IF that use LLMs to deliver a human artistic vision. (And I say this as someone who loves Z-machine games and has long been skeptical of wedging AI into IF when it doesn’t belong.)
2 Likes
I kind of agree with @prevtenet though I understand the actual function of AI is murky with interactive stories.
You don’t want the AI to make up/hallucinate a story with detail it’s going to forget 8 moves later.
You don’t want the AI to generate the prose because it’s boringly the same and based on tropes the AI knows and is trained on.
I have actually experimented with throwing character descriptions at ChatGPT. It’s good at brainstorm fodder but every character description was “book report” ish and way too long. If I tried copy-pasting any of its content it felt enthusiastically fan-fic-ish and not at all my style. Where I found it sort of useful was after asking it to describe several D&D-ish character class tropes, I asked it to come up with a method to break a curse and what each character’s role in the curse-breaking ritual would be. It did a decent job of outlining a plausible method and what each character’s skills might contribute. The output was per usual rather generic and not usable, but it did give me ideas on a structure that part of the plot could take. So I’m all for LLM as a “writing idea buddy” so long as it does not write the actual thing for you. If you ask for a “rogue” character it will write you the most generic rogue you can think of, but if you throw it some curveballs it can make some connections for a writer like “how might this rogue and paladin get along if they dislike each other and have to accomplish this task?”
As far as a live LLM working alongside play of an IF, the only thing I’d probably trust it for would be to fill out unspecified details in the world. If the player examines leaves in a graveyard which aren’t described, it could improvise something instead of “You can’t see any such thing” - however the danger is it will waft rhapsodic with a three paragraph description of graveyard leaves and what they mean to poets, which would put enough emphasis on an unimportant object that it could confuse the player.
4 Likes
The difference being that, in LLM-IF, it matters a lot less which objects are “important.” Why shouldn’t the player be able to pick up a leaf and, in a subsequent scene, throw it at the guard to distract him? In traditional IF, the author would have had to imagine that scenario specifically, or at least schematically by treating “throwing something” as a solution and “leaves” as being part of the class of things that can be thrown. In LLM-IF, the AI has much wider latitude to take any reasonable action and try to find a way to fit it into the scene (or even to take totally unreasonable actions, like acting as if you have an invisibility cloak you don’t actually have, and figure out how the actors in the scene might realistically respond to a PC pretending to have an invisibility cloak).
The fundamental challenge, from both a game-design perspective and a technical one, is “how do we keep the story on track while allowing the player as much freedom as possible?” This is not an easy problem to solve, but I’ve become convinced that it’s possible. Part of the answer is “write stories where some degree of chaotic agency doesn’t ruin the illusion of coherency” (the exact inverse of the classic IF problem, “write stories where some degree of railroading doesn’t ruin the illusion of agency.”) The other part of the answer is to design effective systems for keeping the LLM on track. My attempts have mostly combined four subsystems that are lacking in traditional free-hallucination platforms like Character AI (although all have been done before in some form):
- Feedback loops and effective prompting that constantly try to steer the LLM’s output back towards the world model and story the writer has designed, as well as catch when proposed output is getting too far off-track and rewrite to steer it closer to the intended vision.
- A self-analysis system that exploits LLMs’ skills at summarization and editing to keep track of the most important aspects of the characters and world model and how they have changed (to oversimplify, you ask the LLM “what has changed as a result of this turn, and where should that go in the database?” and then feed that information back into the text-generation loop for the next turn.)
- A separate self-analysis “trigger” system that passes output back through the LLM to identify when key plot-relevant points have happened: the player has been kicked out of the party; someone has died; the cursèd pickle is no longer in the bowl; someone has changed their mind about something; etc.
- A scene-context system that emphasizes different prompting and goals for the LLM based on when and where you are (usually, advancing between scenes is done with triggers from system #3).
This is all still experimental (and expensive to run at scale), but many variations are possible: layered feedbacks, RAG, prompt caching, etc. You could even run entirely separate subsystems behind the scenes to keep track of what offscreen characters are doing from their own perspective.
Conversational game mechanics work very well in this framework. Physical game mechanics (that rely heavily on the particulars and continuity of the world model) are harder to get right, but absolutely magical-feeling when they do work.
I throw this out mainly because I’m convinced that it works in theory, but I don’t know when I’ll have the time and inspiration (let alone compute budget) to actually turn my tech experiments into a full game. So, if someone wants to beat me to the punch, be my guest and I’ll happily play it 
My ideal scenario is that the LLM wouldn’t run the game but can look at the world-state to provide information. If Bob is at the bottom of the stairs and the player asks “where is the knife?” the AI could understand that another character moved through Bob’s location holding it so he could respond “I don’t know where the knife is at. The last time I saw the knife, Duffy was holding it. They went upstairs.” even though that is not a formally scripted response, but could be a major clue about the murder.
1 Like
This previous post might be applicable:
1 Like
This project is quite different from what I’m describing here, but illustrates another interesting angle brought up obliquely in the Consuming vs. producing AI thread. There aren’t many people who enjoy reading AI writing, but there are a lot of people who enjoy doing AI writing, even just for their own personal fun. In a sense, it’s the perfect combination of “I wish I could read a book about ____” and “I want to write a book about ____” but with a fraction of the work. In fact, I would argue that the process of guiding an LLM to write the story you want to read is its own kind of IF experience in the same way that playing a TTRPG is.
For a certain kind of person, it’s a ton of fun (although the potential to be amazed by a twist ending is, obviously, limited). There is a huge (although mostly hidden) market for this kind of experience, as illustrated by the success of services like Character AI and the many AI-writing-tool startups, the vast majority of whose output is never intended for publication.
No need for an LLM for that though. Bob just needs to keep a record of what he’s seen. GOFAI stuff. Could have been done decades ago, and in fact I recall playing around with a tech demo, maybe 15 years ago, that did just that.
2 Likes
My just-warmish take is that when LLMs will be able to spare the players something like:
The man is here, looking at you.
TALK
You seem to want to “talk to somebody”.TALK MAN
You seem to want to “talk to somebody”.TALK TO MAN
Who do you mean, the man who’s just passing by because of a walking daemon or the one ACTUALLY looking at you?THE ONE LOOKING AT ME
I only understood you as far as wanting to look.
it will be well enuf and IF will be tremendously more popular.
You know, I don’t see it very far in the future as both ChatGPT and Siri do this all the time, and they only have to devour Canada’s energy needs every two seconds, to achieve this.
5 Likes
Ha!
There’s also the half litre of water used for every query made to ChatGPT - Sydney Morning Herald article.
-Wade
4 Likes
What I expect to get out of an LLM in a game is basically what I expect to get out a very experienced, very good-at-improv DM in a TTRPG. Follow the outline of the gamebook, roleplay the NPCs through both expected and unexpected situations, and (in a text setting) fill in world details on request. I agree with you (and said earlier) that we’ll see this from a AAA studio long before we see it from indie authors, because getting an LLM to do this well and not go off the rails all the time is hard. You aren’t going to get there just by prompting an off-the-shelf foundation model. You’re going to need to give it some real training and RLHF, and that takes a budget. But if you think the result still won’t be interesting and won’t sell games… well, I think that’s indeed a very unpopular opinion.
2 Likes
Here’s my own unpopular opinion on AI: we should stop stigmatizing IF authors who use LLMs to assist with their prose, and instead encourage correct use. By correct use, I don’t mean just copy-pasting whatever cringe it spits out in response to your first prompt attempt. Correct use means engaging with the LLM similarly to how an author would engage with a hired editor. Over the past couple days, I’ve cleared through writer’s block and made a ton of progress on the storyboard for a game through a process like this:
First, outline the themes and major beats of the game’s plot, and ask the LLM to brainstorm details of a particular room/scene/piece of backstory/etc. Most of its suggestions will be unsalvageably bad, but a few will be rough gems. Pick out the gems, think about how to polish them, and present those refinements to it. Don’t just ignore the bad ones: give back critiques, especially when the same kind of badness pervades a lot of them. The purpose of these critiques is not just to get the LLM to adjust its output; half the time I find myself struck by a great idea while I’m in the middle of criticizing the bad ones.
Once I have the ideas fully formed, write a rough draft of a few paragraphs of prose. Critique my own draft (this can be as simple as “it drags”, and in fact that’s a good prompt even when I don’t think it’s true), feed the draft to the LLM along with the critique, request revisions. Then, same story as in the brainstorming section: most of its edits will be bad, but one or two nice turns of phrase will pop out. Manually incorporate those into my draft, and repeat. Loop until the LLM’s suggestions seem to be consistently for the worse and I don’t want to accept any more, and then ship it.
This is the non-lazy approach to using an LLM to write a game. It’s more work than doing everything manually, but the result is better than I can get through manual effort alone.
5 Likes