This is so interesting that people want this. This is beyond the guy saying “people don’t like making music so we made ai so it’s easier and faster to just prompt it to make music” . Like in my experience the best part of d&d is the dungeon master who knows their friends making a story and details specifically for fun and the fun of the group. And the interactions between players and the dm is where I derived the most enjoyment, automating that sounds insane and peak confusion/self defeating.
I love being a DM but it is no mystery to me. Being a DM is a lot of work. And occasionally, a lot of annoyance. You not only have to facilitate (and sometimes create from scratch) an engaging story that keeps people coming back, you have to wrangle busy schedules and personalities that is often annoying. Lots of times you'll have a bunch of people willing to play, but no one willing to DM. So to me it's not surprising at all, and I experimented with AI in running a game, and quickly concluded it could never be good at it because it's so insanely suggestible and has poor memory.
I tend to agree. I used to run Call of Cthulhu in high school, and there was a lot of preparation involved in running a good scenario. Some people are good at winging it and making stuff up as they go along. I never was. The best experiences for me and my players was always when I'd meticulously designed my own scenario, or used a store-bought scenario and read it thoroughly. Either way, hours of prep.
I would love to have an AI as an assistant Dungeon Master (or game master, or Keeper, or what have you). That is, one person in a group of players maintains the role of a master storyteller, but the AI is ready to fill in details or suggest ways to get the players back on track. This would probably be tedious if you're interacting with the LLM entirely through text, and having to manually keep it up to date with the story. But it could work well if you have a model that understands spoken language listening in on the game and generating cool images and making private suggestions to the game master.
> one person in a group of players maintains the role of a master storyteller, but the AI is ready to fill in details or suggest ways to get the players back on track
As a player and very occasional DM myself, filling in details and trying to get players back on track is where is the fun and challenge. AI could definitely be useful to handle all the paperwork (fight resolution and so on) though
I think the previous posters point was that an LLM DM is boring due to it's generality and lacks the specific quirks of having a human DM with their own viewpoint and understanding of the players, which I agree with. I did some experiments with earlier ChatGPT models for world building and DMing and found it to be so boring, getting it to do anything interesting or giving any kind of push back was so hard. Since they are an averaging of all knowledge / previous creativity you end up with a smooth surface, but part of what makes stories interesting are the rough edges.
However I also agree with you that being a DM is a prohibitive amount of work for someone, say, with kids and a job. It would be awesome to have an LLM as an assistant, maybe feeding in parts of the story and querying it for ideas when you're in a bind. But having it run as a full DM, at least right now, will likely lead to a boring experience.
Yep, my experience using ChatGPT as a helper is very good. It can very quickly cook up a custom monster, give it relevant stats, and create some flavourful actions and attacks to make it unique. Lately I've been trying to hook this up to D&D Beyond, as it's just some fairly simple POST payloads. If I can get to the point where I can describe a vague encounter, and run it in the Beyond encounter tracker with zero effort, that would be amazing, and a lifesaver when the party goes off and does nutty stuff. Beyond that, its story ideas are fairly generic but fine if you really have no time, and it can do a decent job of the occasional flowery monologue if you need it.
Personally I can't stand to run a D&D game as players take it all too seriously.
I will run Toon, Call of Cthulhu or Paranoia any day, the latter without a scenario as I can kill off their all their clones in my very dangerous Alpha Complex and have them rolling on the floor laughing the whole time with a help of a stack of prerolled character sheets and some random encounter tables. (I'd expect an LLM to be able to do the same)
Contrast that to the famous Bloodstone campaign which is the pinnacle of D&D scenarios but can't really be challenging to the players because players have to win over and over again if you're going to use most of the material.
Try a simpler game. I think a lot of people think TTRPG = D&D and the world is worse off for it.
There are numerous tactics that work to face the "not serious" problems you have in a game for "young adults".
For instance, serious players get resentful when somebody not serious comes in late, forgets their character sheet, etc. You could lose either or both of them, but with a little prep you don't have to. In Paranoia it's easy to shove a preroll into their hands, give them a 1-2 minute briefing from "The Computer" and build up tension around this character who mysteriously appears (and if the new player complains that they don't know the rules tell them they're not allowed to know the rules!)
Toon is the epitome of "not serious" and, like the other two games I've mentioned, is a game where you can buy one book and have everything you need, including your first scenario.
That would be my takeaway too. There is room for AI DMs, but they need to get a lot better before they're really fun to play with. I tried a few AI RP just to see how it works and it was pretty boring.
i mean this is on point regarding the effort needed to run a game, but also llms don't really solve any of the problems you listed, so while people want _something_ it doesn't seem like they're crying out for this. also, would love an example of the suggestible ai DM if you have one handy lol
1) "LLM DM" isn't merely mean substituting for human DM in a D&D session. The same capability can be used as a component in a video game, to breathe life into the game world, have it interactively react and evolve along with the player.
EDIT: Take a game like Rimworld, that relies on a scripted RNG dubbed a "storyteller", to decide what random events to hit you with and how hard. It's fun early on, but if you're into role-playing, you'll quickly realize there's no evolving story behind it, just stateless RNG. An LLM DM is exactly what could add that story, make overcoming challenges feel meaningful and allow for player decisions to actually impact the world deeply.
2) There are people like me, who would love to participate in an RPG session, but for various reasons never got invited to those when at school, and now, due to demands of parenthood, can't exactly make time to coordinate with the few people around who are still playing.
There are more, but those are the two that are apparent to me.
> An LLM DM is exactly what could add that story, make overcoming challenges feel meaningful and allow for player decisions to actually impact the world deeply.
No it's not. I don't think you're going to find an LLM with a large enough context window to have a meaningfully involving story spanning multiple sessions.
An LLM isn't going to craft a story element tailored to a character, or more importantly, an individual player. It's not going to understand Sam couldn't make last week's session. An LLM also doesn't really understand the game rules and isn't going to be able to adjudicate house rules based on fun factor.
LLMs can be great tools for gaming but I think their value as a game master is limited. They'll be no better a game master than a MadLibs book.
> I don't think you're going to find an LLM with a large enough context window to have a meaningfully involving story spanning multiple sessions.
First, you don't need much of any context window because you can finetune the LLM. Don't mistake specific engineering choices and tradeoffs and deployment convenience for intrinsic limitations of the technology.
Second, LLMs like Gemini now have context windows of millions of tokens, corresponding to millions of words. Seems like enough for 'multiple sessions'.
> An LLM isn't going to craft a story element tailored to a character, or more importantly, an individual player. It's not going to understand Sam couldn't make last week's session. An LLM also doesn't really understand the game rules and isn't going to be able to adjudicate house rules based on fun factor.
An LLM can do all of that, and you definitely do not know that they can't.
> They'll be no better a game master than a MadLibs book.
They've been better than a Madlibs book since AI Dungeon 1 which was like 6 years ago.
Have you actually used Gemini? I use it a lot for translation, and its context window is more like 150k tokens, rather than the 2M context window they say it has.
Be that as it may, long context window models which are good are not a mirage. By say late 2027, when the LLM providers figure out that they're using the wrong samplers, they will figure out how to get you 2 million output tokens per LLM call which stay coherent.
> I don't think you're going to find an LLM with a large enough context window to have a meaningfully involving story spanning multiple sessions.
Sure you will.
> An LLM isn't going to craft a story element tailored to a character, or more importantly, an individual player.
Sure it is.
> An LLM also doesn't really understand the game rules and isn't going to be able to adjudicate house rules based on fun factor.
Sure it will.
You need to use the tools for their purpose, not for the opposite of it. LLMs have finite context, you need to manage it. LLMs don't have a built-in loop, you need to supply it.
Character stats, names, details about players - those are inputs, and structured ones at that. LLMs shouldn't store them - that's what storage media are for, whether in-memory or a database or a piece of paper. Nor should they manipulate them directly - that's what game systems are for, whether implemented in code or in a rulebook run on a human DM. LLMs are to make decisions - local, intuitive decisions, based on what is in their context. That could be deciding what a character says in a given situation. Or how to continue the story based on worldbuilding database. Or how to update the worldbuilding database based on what it just added to the story. Etc.
> Character stats, names, details about players - those are inputs, and structured ones at that.
Some details about players are structured and can be easily stored and referenced. Some aren't. Consider a character who, through emergent gameplay, develops a slight bias against kobolds; who's going to pick up on that and store it in a database (and at what point)? What if a player extemporaneously gives a monologue about their grief at losing a parent? Will the entire story be stored? Will it be processed into structured chunks to be referenced later? Will the LLM just shove "lost a father" into a database?
Given current limitations I don't see how you design a system that won't forget important details, particularly across many sessions.
> who's going to pick up on that and store it in a database (and at what point)
LLM might, if prompted to look at it, or if there was a defining moment that could invoke such change. It won't pick on a very subtle change, but then most people reading a story wouldn't either - this is more of the kind of stuff fans read into a story when trying to patch potential continuity issues.
> What if a player extemporaneously gives a monologue about their grief at losing a parent? Will the entire story be stored? Will it be processed into structured chunks to be referenced later? Will the LLM just shove "lost a father" into a database?
The scale depends on the design, but I'd say yes, shoving "lost a father" into a database so it pops up in context is a good first step, the next step would be to ensure the entry looks more like "mentioned they continue to grieve after loss of their father <time ago>", followed by a single-sentence summary of their monologue.
Personally, I had some degree of success with configuring LLM (Claude 3.5 Sonnet) for advising on some personal topics across multiple conversations - the system prompt contains notes in <user_background> and <user_goals> tag-delimited blocks, and instructions to monitor the conversation for important information relevant to those notes, and, if found, to adjust those notes accordingly (achieved by having it emit updates in another magic tag, and me manually apply them to the system prompt).
> Given current limitations I don't see how you design a system that won't forget important details, particularly across many sessions.
It's not possible. Fortunately, it's not needed. Humans forget important details all the time, too - but this is fine, because in storytelling, the audience is only aware of the paths you took, not of the countless other possibilities you missed or decided not to take. Same with LLMs (and larger systems using LLMs as components) - as long as they keep track of some details, and don't miss the biggest, most important ones, they'll do the job just fine.
(And if they miss some trivia you actually care about, I can imagine a system in which you could ask about it, and it'll do what the writers and fans always do - retcon the story on the fly.)
I have solved it in my two games by using two systems in my future game:
1. LLM "text" info ,about world, player, text descriptions of world/decisions and so one.
2. Typical D&D stuff with rolls, names, details, decisions that are simple logic.
Rimworld is a great universe where we think about characters' stories, and there's really just like, a couple dozen attributes and straightforward ways they interact with the game.
An LLM context window could easily have 20 times as much interpersonal state, and make it interact in much more unexpected (but plausible) ways. That's going to be a surprising and rewarding gaming experience once someone figures it out.
Context window is arbitrary and can be adjusted on demand. When a new random event needs to be generated, and then fleshed out, the context can contain e.g. list of facts about the story so far, the overall story arc, summary of main characters or plot threads. This can be used by LLM to decide e.g. which faction will attack, and who will be on it, and what their goal will be, etc. After the event concludes, it just becomes another line in story event history. Meanwhile, that can be fed to a differently prompted LLM to evolve the plot arc, update motivations of background characters, etc.
I have a feeling people imagine LLMs as end-to-end DMs that should somehow remember everything and do everything in one inference round. That's not what they are. They're building blocks. They're to be chained and mixed with classical flow control, algorithms, and data storage (as well as the whole interactive game mechanics, in videogame context).
Maybe. But for something like Rimworld++++, you don't need all that sophistication. You need a pile of relevant facts and a clear task to determine something about the game or write some text for the user. Sure, curating and retrieving selectively would probably be even better and more flexible.
>I don't think you're going to find an LLM with a large enough context window to have a meaningfully involving story spanning multiple sessions
You don't need to provide every single previous information to llm, use LLM to summarise previous ones and it gets really compact. It works quite well.
DMing is, in my opinion, very time consuming. Prepping an adventure, familiarizing yourself with the rules, trying to incorporate character backstory in to the narrative, all takes time. As someone that can only play D&D if I am DMing, because none of my friends have the desire or inclination to do so, I would be very happy to pay money for an LLM dungeon master that lets me just play the game without all the prep time. I just don't have 4 extra hours a week like I did in college.
> Like in my experience the best part of d&d is the dungeon master who knows their friends making a story and details specifically for fun and the fun of the group.
Sure.
There are two reason why I can think of someone making a Dungeon Master LLm.
One is that when there is no cake we eat bread.
Don't get me wrong. I'm a DM, and I love playing d&d with my friends. I totally agree with the sentiment you are sharing. But people who are willing, and able to DM for others are not evenly distributed. There is a lot more people who would like to play TTRPGs than people who is willing to step into the DM role.
So in that sense think of this as a substitutions for those games which would not happen otherwise, because they don't have a DM. Or the only person who would DM them is an ass. Or their DM has burnt out.
Is some game better than no game? Sometimes. Depends on how good that game is. And we won't know how good the substitute can be without trying. Heck maybe more people will play, and they will realise how easy is to DM actually.
The other reason is the sheer challenge of it. Dnd has a lot of rules. There are the obvious ones you can read in the book. But there are also un-written rules. Like object-permanence. If a goblin steals a diamond ring from us, and we slay them within minutes they damn well have the diamond ring on their bodies somewhere. If three displacer beasts ambush us and we slay 2 there damn well be 1 more accounted for in some sense.
There are also "story-writing" rules. If we went through hell and high-water to obtain an arrow of dragon slaying after the blacksmith told us about the legend of it, he better not just pull one out of his ass the next time we see him. If the whole lore of goddess X is that they are kind and caring then they should better be at least not cruel when we meet them. These are all hard for an LLM. They are also easy for a human to evaluate. Because we just feel when they are not right. Therefore it is a good challenge to evaluate how good we are at this "making a bucket of sand smart" task.
Consider video game RPGs. They're largely based upon D&D, yet they remove the dungeon master entirely. They are enormously more popular than D&D itself.
Now imagine something in between, where you have a video game, but the NPCs and parts of the story are controlled by an LLM. You can give the players much more freedom, and the creators don't have to write thousands of lines of dialog to account for every possible choice a player can make. The game doesn't have to be "on rails" so much, the LLM can help speed the story along when the player gets confused, and you can have NPCs with much more depth than just several lines of static dialog.
How well this will work in practice remains to be seen. In my experience, ChatGPT itself is a painfully generic DM that relies upon repetitive fantasy tropes. You still need an actual human being to create an interesting story and add depth to the world.
One of the growing trends these days is solitaire RPGs -- where people play alone as both player and Game/Dungeon Master. Typically this isn't done by AI but by rolling on various tables (sometimes called "GM Emulators"). Not everbody has the time or even enjoys the extrovert pressure of a traditional social RPG. A related concept is the "cooperative RPG" which does have multiple players but no DM so everyone can play.
Here's a list of some of the ones available (about 10 years old but it gives you the idea)
I grew up with “choose your own adventure” books which were like a solo adventure: (If you go on the north road turn to page 34 if you follow the river page 57)
Many board games now have a solo mode with an automated player with tables and dice to help randomness.
Obviously there are adventure video games. Some of the new ones often have interesting back stories, really amazing world building and dnd like adventures.
Horizon zero dawn, the Witcher, Star Wars outlaws among many others.
“baldurs gate” really gave me dnd vibes (I played through that one multiplayer).
That style of book led to one type of solo adventure -- the "numbered paragraph" type that is basically a choose your own adventure book except that combat and other such situations are resolved using the RPG rules. Those still exist, but more recently, the goal is to make things more open ended and flexible, at the cost of having the player make decisions on what things mean -- like the player might roll on some tables that say "orcs" and "merchants" and the player needs to invent their own explanation of what that means -- do they encounter orcs attacking merchants? Or orcs who are merchants?
Oh man, I played the shit out of the Lone Wolf books when I was a kid in the eighties and nineties. My brother had the first couple of them, and I still have his old books somewhere with his notes. I should find them for nostalgia's sake (my brother having long since passed).
Isn't this a modern take on the yarn that ended up with students all just bringing a tape recorder to the lecture hall after the lecturer just played a pre-recorded tape as the lecture?
Try doing this while you're driving with the gpt4o advanced audio mode. It's a nice way to kill 40 minutes or so without having to look at a screen. Just tell it you want to play an RPG with it, give it a setting, describe your character and go. It'll mostly be more of a choose your own adventure than a real RPG, but it can sometimes be quite entertaining.
Agreed on DM making the experience special. The cognitive load on a DM is very high though. They have to digest information on the game setting and the monsters, synthesize scenes and react to players' interactions all the while staying true to the narrative. I think LLM's can play a big part in assisting human DMs by either providing low-fidelity ideas for "inspiraton", commonsense based on their training on D&D texts and literature or high fidelity text in some cases to be directly presented to the players.
> I think it's more about the challenge/curiosity of creating the implementation.
Totally this, at least for me, in some circumstances.
TOU's be damned, I've written bots for some online games I've played, not because I want the XP or money or whatever that the bot could do without me working for it, but rather because I found writing the bots fun and engaging.
Before anyone gets in an uproar, I didn't sell them, nor any of the in-game resources gained from them. I was watching them basically all the time, because that's what I was there for - to see my creation work. And I purposely didn't interfere with other "real" players.
It's been written at length here and elsewhere by game devs, but this isn't a thing that anyone would truly want. A purely AI generated or controlled world would have no constraints, and be fundamentally untestable - games aren't really games unless they have constraints. Even the 'purest' sandboxes have some kind of constraint buried within them, and I think you'd find an RPG of this type extremely boring, at least with current technology.
> this isn't a thing that anyone would truly want.
Citation needed.
> A purely AI generated or controlled world would have no constraints
That's a shitty AI then. Make a better one. I can play 2000 Vampire: The Masquerade games with 2000 different groups. They will each be different, but they will also be each distinctly Vampire: The Masquerade ttrpg games. If the AI you are thinking about can't do the same, then think of a better AI.
> at least with current technology.
Well. Who is the group who will make the "next technology"? Should we work on that, or just lay down on the ground and wait for it to fall from the sky? Testing what are the limits of the current technology (as done in the paper we are talking about here) is the way to get there. Or at least to systematically answer the question of where and what are we lacking.
Lol, a citation of what? This is my opinion statement and the rest of my post follows it.
> That's a shitty AI then. Make a better one. I can play 2000 Vampire: The Masquerade games with 2000 different groups. They will each be different, but they will also be each distinctly Vampire: The Masquerade ttrpg games. If the AI you are thinking about can't do the same, then think of a better AI.
Sure, I'll get right on that.
> Well. Who is the group who will make the "next technology"? Should we work on that, or just lay down on the ground and wait for it to fall from the sky? Testing what are the limits of the current technology (as done in the paper we are talking about here) is the way to get there. Or at least to systematically answer the question of where and what are we lacking.
I'm really unsure of what or who you are addressing here but it certainly isn't anything I've written in my post.
Citation that nobody wants what you described? The sentence which I was quoting.
> This is my opinion statement
Your opinon can be that "I don't want this." "nobody wants this" is refers to things outside of your head. Do you see the difference between the two?
> it certainly isn't anything I've written in my post.
Your post is suffused with defeatism. The 3 "no"s it contains are: "nobody wants this", "with current technology this cannot be fun" and an implicit "we can't make the next technology". I believe each of those are wrong, and I'm calling you out on the attitude.
Ok, then what advances are being made in AI technology in this gaming that lends you such confidence? Care to make any citations yourself here? I don't really care what you think of my attitude, nor does it make for productive discussion or good posting.
Happily. I know you are wrong on the "nobody wants this" statement because I want it. With a sweeping generic statement like "nobody wants this" a single example is enough to disprove it. There you have it.
> what advances are being made in AI technology in this gaming that lends you such confidence?
There is a ton of experimenting going on. AI Dungeon and Deep Realms are the two obvious ones. I don't think anyone has found the golden solution yet, but that is also not evidence that no such thing exists.
I don't really buy this. I don't think the tech is quite there, although at this point it might be a matter of clever prompting more than a fundamental limitation depending on the type of game you have in mind. A strong enough DM AI could take a prompt like "I want to play a game with a similar loop to Satisfactory, set in Mordor at the beginning of the second age." And the AI could figure out from there, including devising appropriate constraints.
There's no reason why an AI-driven sandbox cannot have constraints, as well.
Now it's true that, with the current crop of LLMs, a persistent enough player would always be able to break through them. But if it takes conscious and deliberate effort, I think it's reasonable to say that whatever experience the person gets as a result, they were asking for it.
I guarantee you that anyone with knowledge of how these games are run would not want an LLM being a DM.
Its the type of problem which requires a good balance of storytelling, preparation, and tailoring the interactions towards the psychology of the people who are playing (in specific beneficial and entertaining ways).
Its a problem computers won't likely solve anytime soon.
I am an “everything is custom tailored to my players” DM and I use LLMs quite a lot as an assistant. Speeding up prep time is always welcome. Using an LLM directly as a DM? Oh hell no.
Also, I ran a session over Zoom and their AI summary was so useful!
This is nothing new. Bringing ideas from DND into video games is as old as video games themselves. In interviews, John Carmack talks often about how he used to play DND with his friends and then try to take that experience into the games he made. And it goes back even further than that - Rogue, the primogenitor of all roguelike games, was inspired by DND as well. I could find a bunch more examples, but you get the idea.
I largely see it as the digital next step up from something like the Mythic Game Master Emulator, which is popular for solo play and things like trying out new systems.
And there's quite a lot of us that like to play but because of life commitments, getting together with a group on a regular schedule is difficult.
And yet, that's exactly what computer RPGs are all about, one of the most popular video game genre is essentially about automating dungeon masters. It includes early games like Colossal Cave Adventure and Zork, which are chatbots not unlike a very primitive version of ChatGPT.
If you have a good role playing group, with a DM you enjoy playing with, no computer game nor LLM will replace that. But if what you are looking for is the convenience of computer games with the freedom of tabletop RPGs more than the social aspect, then LLMs totally make sense. And even with a social group, it can work as substitute if you don't have a DM (DMing is hard work!).
I can totally see the appeal of using LLMs to _help_ the DM. Not to replace it.
I DM a few games of The Expanse, and using LLMs to plan ahead was a godsend. No, I didn't utilise it to write the story for me - instead I used it to test my planned story and see which way my players might strafe off the road, so I can plan for those. Basically simulated a game using an LLM that acted in place of multiple characters, allowed those to run free (within certain limits, obviously you can't have the LLM players do a dozen actions without the DM having a say), and essentially mapped the potential "off branches", story pathways I didn't plan for initially. This has allowed me to be prepared for the usual dumbass things players might do, such as heading for a strip club in the middle of a total disaster where they (figuratively) have dozens of arrows pointing to the goal of the chapter.
Another interesting aspect of using AI for TTRPGs is to create atmospherics. For Expanse based games, I've bought a number of tile packs and such, to appreciate the artists who put work into it, but I simply don't have the funds to commission a few dozen acrylic matte style scenery images (which I usually put up on my projector, combined with some Hue lights to create the visual atmosphere). With AI, I can even generate them on the fly, should my little gremlins stray off the path. Same for music - AI can incredibly easily generate an atmospheric soundtrack that fits the current scenery, with just a few words, while I can still pay attention to the players.
Last year, I built a DM for myself using the OpenAI api and an elevenLabs voice generator. I asked it set my character in Baldur's Gate, so it could pull upon the huge amount of DnD source material it had been trained on.
A few takeaways:
1. An LLM based DM can give the player essentially infinite richness and description on anything they ask for.
2. There is difficulty in setting the rules for the LLM to follow that match the DnD rulebook. But this is possible to solve for. Also, I found the LLM to be too pliable as a DM. I kept getting my way, or getting my hand held through scenarios. Maybe this is a feature?
3. My conversation quickly began to approach the context window for the LLM and some RAG engineering is very necessary to keep the LLM informed about the key parts of your history.
4. Most importantly, I found that I most enjoy the human connection that I get through DnD and an LLM with a voice doesn't really satisfy that.
> My conversation quickly began to approach the context window for the LLM and some RAG engineering is very necessary to keep the LLM informed about the key parts of your history
Assuming we're talking about GPT-4o, that 128k context window theoretically corresponds to somewhere around 73,000 words. People talk at around 100 words per minute in conversation, so that would be about 730 minutes of context, or about 12 hours. The Gemini models can do up to 2 million tokens of context... which we could extrapolate to 11,400 minutes of context (190 hours), which might be enough?
I would say GPT-4o was only good up to about 64k tokens the last time I really tested large context stuff, so let's call that 6 hours of context. In my experience, Gemini's massive context windows are actually able to retain a lot of information... it's not like there's only 64k usable or something. Google has some kind of secret sauce there.
One could imagine architecting the app to use Gemini's Context Caching[0] to keep response times low, since it wouldn't need to re-process the entire session for every response. The application would just spin up a new context cache in the background every 10 minutes or so and delete the old one, reducing the amount of recent conversation that would have to be re-processed each time to generate a response.
I've just never seen RAG work particularly well... and fitting everything into the context is very nice by comparison.
But, one alternative to RAG would be a form of context compression... you could give the LLM several tools/functions for managing the context. The LLM would be instructed to use these tools to record (and update) the names and information of different characters, places, and items that the players encounter, important events that have occurred during the game, as well as information about who the current players are and what items and abilities those players have, and then the LLM would be provided with this "memory" in the context in place of a complete conversational record. The LLM would then just receive (for example) the most recent 15 or 30 minutes of conversation, in addition to that memory.
> I found the LLM to be too pliable as a DM.
I haven't tried using an LLM as a DM, but in my experience, GPT-4o is happy to hold its ground on things. This isn't like the GPT-3.5 days where it was a total pushover for anything and everything. I believe the big Gemini models are also stronger than the old models used to be in this regard. Maybe you just need a stricter prompt for the LLM that tells it how to behave?
I also think the new trend of "reasoning" models could be very interesting for use cases like this. The model could try to (privately) develop a more cohesive picture of the situation before responding to new developments. You could already do this to some extent by making multiple calls to the LLM, one for the LLM to "think", and then another for the LLM to provide a response that would actually go to the players.
One could also imagine giving the LLM access to other functions that it could call, such as the ability to play music and sound effects from a pre-defined library of sounds, or to roll the dice using an external random number generator.
> 4. Most importantly, I found that I most enjoy the human connection that I get through DnD and an LLM with a voice doesn't really satisfy that.
Sure, maybe it's not something people actually want... who knows. But, I think it looks pretty fun.[1]
One of the harder things with this would be helping the LLM learn when to speak and when to just let the players talk amongst themselves. A simple solution could just be to have a button that the players can press when they want, which will then trigger the LLM to respond to what's been recently said, but it would be cool to just have a natural flow.
The main problem with doing this on chatgpt is that it is _relentlessly_ cheerful and it's almost impossible for it to produce any kind of interpersonal drama or threat or danger.
I tried to get it to run space opera about a diplomatic mission to a newly recontacted planet, full of Game of Thrones-style intrigue and plotting and scheming, and it was like playing with a cheerily optimistic new UN intern that completely believes in the power of compromise and diplomacy to solve all problems.
It was always like: "And so the two of you hashed out your differences over a glass of Dubranian Forblik, and peace reigned forever and ever after."
That's true of how it does role playing games and any kind of fiction at all -- it always wants to tie a neat bow on it at the end of every few paragraphs with a moral and everything.
You also have to ask it to use python to dice roll for anything you want to potentially fail on, or it will always make everything you do succeed wonderfully.
I will say this -- the advanced audio mode for D&D is amazing and it really does act out the characters and it will occasionally slip into different voices or even add sound effects, even though it's not "allowed" to. It will also sometimes copy your voice and act out what it thinks you should do instead of letting you talk, though.
> raised by 3 out of 7 players was the perceived lack of danger for their characters. Participants noted that the gameplay did not present sufficient threats or challenges, which diminished the sense of urgency and excitement typically associated with D&D adventures.
Indeed. But for stuff like this, ChatGPT is overkill. It's better to get a dedicated RP finetune of LLaMA, Qwen, or some other open weights model (you can still run it in the cloud if you don't have hardware to do so locally). There are enough finetunes around by now that you can "dial in" how dark you want it, but for some examples:
https://old.reddit.com/r/LocalLLaMA/ is the place to be, but it's a subreddit with an eclectic population, so apply some caution when browsing it at work.
r/LocalLLaMA has the discussions, but on top of that, I just periodically browse new model lists on Hugging Face. There's a lot of stuff, but most low-effort finetunes tend to focus on small models (since that's much cheaper and faster), so if you only look at 70B+ ones, there's a lot less garbage there.
The old gpt2 engine that used to drive AI Dungeon had the opposite problem, it was obviously trained on a ton of really bad fan fiction and would frequently either shift into filthy erotica or some kind of hyper violent serial killer story, no matter how innocently the story started.
I loved AI Dungeon for generating funny/stupid fanfiction style adventure/action/fantasy/scifi stories. It could keep track of characters and settings, have them return with motivations. And then something unexpected would happen. Provided me with a lot of entertainment.
But my problem with using it as a DM is that you have to police yourself. I could just declare that I did something and it would automatically succeed. There were no rules or antagonism in the system. It was more of a collaborative storytelling tool than a game.
To be fair I haven't used it in years so I dunno how it works nowadays.
Yep, in my game it tried always to "save" player it was impossible for LLM to jail or kill a player to end a game.
I had to explicitly end it by game logic, not by LLM.
I am baffled as to why anyone would want this, or anything like it.
Sure, it's a cool technical demo. But... the point of D&D is a social game played with others. Even if you play a D&D-like video game such as Baldur's Gate 3 (which I'd argue is a fundamentally different experience to playing a tabletop RPG anyway), you're experiencing a world and a story that someone else has crafted.
What's the point of replacing that social interaction, or that connection to another person's creative vision through their art, with an LLM? What value can it ever provide?
I would love a fully generated D&D campaign where I just input some parameters, like a writing prompt, and get a tailored LLM Dungeon Master organizing it. Whether it's the result of someone's creative vision or not is irrelevant to me.
> You know that an LLM is not going to provide any meaningful narrative, or craft a thought provoking story
The title of the paper is "Exploring the POTENTIAL of LLM-based agents..." Right now, yeah, just play BG3. But the context of the discussion is its potential, which is certainly worth discussing imo.
Dear god, it's crucial that adults be capable of creating and maintaining relationships. If it's impossible then, well, that's hell on earth.
Go out and make a new relationship! Do you live in Antarctica? And, if you live somewhere where new relationships can't be made, my god, get out of there! Go, go now! Go somewhere you're allowed to be human!
I've been in a weekly D&D game for over a decade. I don't keep playing it because of the game of D&D (certainly not, I could rant for a long time about the game's myriad shortcomings across many editions). I play it because of the friends I play it with, because of the social connection that exists in and around the game.
To play a tabletop RPG without that isn't the same thing, and to play with LLM is bankrupt of meaning. You'd be better off picking up one of those old-school choose your own adventure books or a video game like BG3. At least those were written by someone with intent, at least there's meaning in the story and setting. Or, finding an online group.
People play games for a variety of reasons. While you enjoy the social aspect, others may find that to be the worst aspect of the game (or just don't have a social circle).
I grew up playing D&D alone in my room using LEGO. It was great and I still have some of the sets I built as dungeons and monsters that I built from reference pictures in the Monster Manual.
Video games are different from solo role playing. It's like trying to substitute shooting hoops in the driveway with NBA '24.
> the point of D&D is a social game played with others
That's one of the points, or a benefit maybe, but its not the WHOLE point. Some people just enjoy RP and fantasy worlds, so a single-player campaign would still be enjoyable (to me, at least, but maybe I'm weird). I def don't see this replacing DMs in real groups any time soon.
I am not interested in this, but an AI DM that allow human players to play would keep the social aspect. Since being DM is a higher threshold of knowledge and time investment, this could unblock a social gathering, rather than substituting it.
Also, if it doesn’t try to keep the story to strict guardrails, the creativity and story creation would come essentially from the human players.
I don’t think those arguments are enough to make me want to use it, but I see the point and possible interesting use.
I've found chatgpt useful for trying out D&D settings that I don't normally get to experience in my usual group. It loses the plot after a while, but it's enough for me to figure out whether I dig the setting enough that I would want to pitch it for a real game.
I'll agree. I've had bad groups and, wow, yeah. I'd rather have just doom scrolled my phone for 4 hours.
So, an AIDnD is wonky now, but it's better than bad DnD (at least with some futzing about that I have done myself).
To really reach here: I think you're placing value on the thing due to the effort involved. That's kind of an old school Marxist way of assigning prices to goods. I'm trying to point out that the value isn't that way for everyone. The 'new' market way of assigning prices is 'whatever someone will pay for it'. And in that very tortured analogy, that means that AIDnD is just as valid if someone likes it enough to do it. The background effort isn't part of the determination of value, necessarily.
Last year, during a period of recovery, I hosted a ChatGPT game with my nieces and newphew. I played the role of being a bride between them and the LLM. I read aloud the text, and translated their actions into queries.
With slight nudges, GPT performed well. It excelled at NPC dialog, location description, and to some extent improvisation. However, eventually the campaign fell apart on session four or five. There was a disagreement between the players over exactly how many maps we had acquired. As you know, answering this question is a needle in a haystack problem and asking the LLM proved to illicit conflicting responses. It ultimately led to me concluding that pausing the campaign was the right choice and accepting that we all still had an amazing time up to that point.
Which leads me back to the questions in the paper:
> We believe that with the above evaluation methods we can try to answer the following
questions:
• How consistent is a LLM Agent in its generative dialogue?
• How well can a LLM Agent keep a user engaged in a narrative plot?
• How creative is a LLM Agent when generating complex story driven narratives?
None of these are exactly relevant to my personal experience running a TTRPG game using ChatGPT and what makes it succeed or fail. What would make such a system an improvement is being able to store and retrieve facts about players (HP, stats, inventory), NPCs(names, location, history), and Places(items, descriptions, inhabitants) in order to solve the needs-haystack problem. Creating a world ontology and being able to CRUD facts against such an ontology would be a game changer(literally).
Despite the ultimate demise of the game, the experiment was a success. I'd highly encourage anyone with a laptop, a handful of dice, and a group of interested friends to give a shot. It's fun, memorable, and a unique way to pass the time and bond.
> What would make such a system an improvement is being able to store and retrieve facts about players (HP, stats, inventory), NPCs(names, location, history), and Places(items, descriptions, inhabitants) in order to solve the needs-haystack problem. Creating a world ontology and being able to CRUD facts against such an ontology would be a game changer(literally).
Curiously, those are the exact things LLMs can't do well, and are not really supposed to. This is all stuff you (or an external system) should keep track of and supply to LLM in context "just in time". This is to say, LLMs are an important component, but alone they won't work. They need their tools :).
In short, LLMs are pure "system 1", the complete solution still needs "system 2", which fortunately is more amenable to classical computing approaches.
Based on my experiences with this. Keeping context relevant is the hardest part. Your human GM will remember that funny remark from the session 3 years ago, but the LLM?
Also for that same module, is it able to integrate it in a broader campaign. I expanded the lore and why the players got here with extra quests, fluff and mystery and they only reached said dungeon at 2nd session only.
Also the way a PC died due to a rat bite that could only be cured by magic (5e of this module is ironically harsher than the 3.5) as he was losing max HP every 24H.
As the players expressed dislike with that character the owning player was willing to go for a new character and I had planned a big drama scene where the players exit in emergency the dungeon and try to rally the nearet village where there's a priestessd and happens that I could match the rolls so he would die in the priestess hands.
But I made the mistake to allow them to spend the night to get their 3rd level which would have allowed the druid to get his cure illnesses spell botching the set up.
Happens that I decided to exploit the fact that all other character the previous days made an offering to Beshaba to curse that character and force rolls during night (and try to find a way to make the new character come into play).
How it unfolded :
- they were sleeping and it was the bard's turn for watch
- the bard was a follower of Tharizdun and was about to sacrifice the dieing character as said character was agonizing
- the new character was a Kara-turian enforcer that was chasing a yokai (he didn't knew) that used other people as disguise, made it happen that the yokai choose the dieing character "by coincidence" appearance
- the yokai being chased opens a portal while hiding to the dungeon
- the new character litteraly pops through a Wild Magic portal that pushed away the bard and tries to bash the old character as the old character runes on his skin glow a last time as he dies and repels everybody
- made the new character pop naked as I wasn't paying attention to what the player was saying when he was listing his inventory
It's that: Spacial awareness, struggling to maintain the personas (and often talking for the player/s), difficulty improvising, pacing issues, lack of initiative, and just an over reliance on prompting to keep going correctly.
I think we just haven't built complex enough architectures to allow this. I have a few ideas boiling that would help facilitate long term context understanding and recall.
Neuro-sama, an AI VTuber streamer, manages to do this.
There are topics brought along multiple streams and manages to remember details of previous streams.
With soemthing like Neuro-sama I wouldn't doubt that Vedal is doing a lot of liftwork behind the scenes to keep sufficient context available to Neuro-sama's LLM.
And with the right kind of engineering, that could be automated too.
I don't think the results would be as good as they are now - Vedal is bringing his own perspective into it - but once you have a proof of concept, you can improve it.
I've been meaning to take a crack at doing this - I'd like to use Whisper to transcribe session recordings and then have an LLM build out a campaign wiki. I enjoy being a DM but note taking and organization is my least favourite part.
Alas, I was desperately keen to read transcripts of actual gameplay and all we got was user surveys?
Was there a structured combat system as sort of this or was it a talking only kind of thing? Who knows?
Trying to get empirical evidence that an LLM can even remotely plausibly work as a DM with no social aspect and (as far as I can tell, reading the paper) no programmed system of stats, rolls, combat etc. seems impressive.
…but was it that? Or just a DnD themed chatGPT conversation? Or a text adventure with an LLM describing each “room”?
Did the LLM actually have agency to make decisions and perform tasks, or was it not an agent at all, just a conversation?
Without a transcript, all you can really say for sure is yeah, I agree; more research is needed.
Using a fine-tuned GPT-3.5 model is definitely a choice, why not pick one of the newer models? The methodology only states they use "ChatGPT" as LLM, but doesn't clarify the model until way later, nor why they picked that model over other existing models.
The methodology also barely explains how they did their baseline experiments with human DMs. Did they do them face-to-face, or via text? Did they have different DMs in the 3 games they used as baseline, or was it the same person? As it stands the research is barely reproducible.
I find this idea fascinating, and I think a lot of very interesting emergent behavior can be added to video games by having something like an LLM act as a Dungeon Master. Rather than having to pre-program a bunch of choices that the player is allowed to pick between, have an AI model evolve the story and just move the pieces on the story board based on what the player actually does. Story driven games these days are still very much railroaded. They are like interactive movies. The player has very limited agency. They can't really pick any choice which the designers haven't thought to implement upfront.
At the same time, I'm personally not super enthusiastic about using LLMs for a pen and paper game. Let's assume for a moment that we have free will, then the most fun aspect of PnP dungeon mastering is the fact that the DM has agency. It adds an (IMO) very fun mini-game where the players try to get away with as much shenanigans as they can, and the DMs job is to try to thwart these shenanigans in the most entertaining ways possible. The keyword is on "try" - the GM has to put entertaining obstacles in front of the players to keep the game fun, but ultimately the players should have an avenue of success. Even if they don't always succeed, they need to succeed enough and when they fail they need to feel that it was fair.
I feel that with an AI DM, this goes out the window. At least with the current crop of AI, I would never feel that I'm facing an agent with it's own free will, rather just a machine. As such, it cannot replace a human DM for the PnP purposes.
This is extremely interesting to me. I understand why so many people are surprised that people would want this, but for whatever reason, I'm very much in the target audience.
I'm a "perpetual DM" in most of my roleplaying groups, having led campaigns spanning years, but very rarely do I get to participate as a player. My kids love me running adventures and one-shots, and while I enjoy it, people don't always recognize how much work it is -- both the prep, as well as the mental effort of being "on" for the entire duration of the adventure. I like afternoon one-shot adventures, but I dislike reading through long sourcebooks. I enjoy crafting puzzles and presenting challenges to the players, but also recognize the mental effort that this takes, and sometimes I just want a copilot to handle the lore and the setting. I already rely on the rules-nerds in my group to act as copilots for the specifics of game mechanics -- I don't have time to read through that many tables and commit them to memory.
Re: datasets, the paper authors used a dataset that they created by scraping Critical Role transcripts -- I doubt they had permission for that. I like contributing to open datasets for training open-source LLMs, and I think it would be really sweet for a playgroup to be able to contribute their play transcripts for such a thing. I would love to see a standard open-source dataset be collected for this, and perhaps even a standardized benchmark for quantifying the performance of particular LLMs in RPG contexts -- rules compliance, needle-in-haystack fact finding, consistency of roleplaying, understanding user intent, challenge creation / resolution, etc.
For those averse to LLM defeating the social aspect of tabletop roleplaying, perhaps a better entry role for an LLM agent is as NPC party companions/followers. Set up the agent with a detailed character sheet and hidden backstory/motivations and set it to listen in on conversations or VTT chat, and interject dialog on a non-spammy basis. Could lead to unusual situational ideas or at least humorous moments.
I've made several DMs for a variety of entertainments. It's as much fun to create them as it is to play a scenario with them. I've experimented with a few methods, such as providing them with an ASCII map that looks like a hack/rogue map, giving them a famous book as the scenario, giving them multiple books and or famous fictional characters and then just letting them loose in a DM/AI guided narrative the user is making decisions.
Of course, I have a "classic" DM that happens to be visiting here on vacation from the dungeons & dragon's multiverse, where his real job is being a DM for visiting tourists. There's a "Shaggy" and his talking dog one can take LSD with when going to Woodstock in '68. There's a StarTrek strategic tactical simulations instructor. I quite like a Underground railroad "conductor" that guides escaped slaves to freedom before the US Civil War.
I think these DM / interactive narrative applications have a huge future in education and training.
I've created two AI LLM D&D-style games. The second one will soon be released as a D&D-style game with Visual Novel elements (AI-generated images_. Here's how I can address some common concerns from comments above:
Lack of Human and Social Interactions: This is a significant drawback when you look it as the social thing, however, they require scheduling. For me I wanted to venture beyond the typical fantasy/sci-fi settings and good bored of those D&D/warhammer world.
LLM Versatility: LLMs can take you on wild (now non-NSFW) adventures. I wanted to play a D&D-style game set in the Venetian Renaissance and GPT delivered with accurate locations, families, and artists. It can do the same with medieval settings from my country, ancient Rome, surrealist landscapes, and whatever your favorite books, movies, or games involve.
Customization: Want a game master who replies in poetry, slang, in style of you favourite author? No problem.
Context: This is a tough one, but there are two approaches:
Retrieval-Augmented Generation (RAG)
Summarization, and then summarization of the summary. I've used this approach in my previous startup and in this game.
Characters: I utilised the randomness of different character styles. You might meet smart/beautiful/evil/good characters with predefined styles, and the LLM quite effectively supports roleplay.
Cons:
Lack of Continuity: The biggest downside is the lack of continuity. Even if I generate a world and scenario, after a few rooms, it doesn't come together like a game would, where previous events feed into current ones, even when I provide again LLM with previous details. I kinda lack big picture design of story. I am working on that
I'm currently running a private beta and looking for co-founders (game/design/coding) in the EU, preferably Poland. Please email me at contact@sentimentscanner.com for more information for beta membership and so one.
The single greatest appeal of Pen and Paper RPGs is human interaction, where obviously the DM has the most important role.
We have a solution for playing RPGs when you have no DM. It is called a "Video game", you can even play it with your friends.
What I have used LLMs for is helping me prepare for certain things, giving ideas for organizations, quests, characters, etc. . They are pretty good at that, although I would avoid relying on them during actual gameplay.
This is a student thesis, which doesn't mean it's bad, but is worth keeping in mind. I see that their control group size was three people (in three different sessions), but I don't see how many people they ran through the AI part of the experiment.
I was wondering the other day whether there were good services for running tabletop games over AI. I don't know about good, because I haven't tried them, but there are certainly a few products like this today. Given that ChatGPT plus some fine-tuning was comparable to a human GM in this experiment, I'd be interested in another experiment that looked at whether these products (which charge more money) offer any benefit over just using ChatGPT. For them to be better than ChatGPT, it seems like they'd almost have to be better than a human GM to be worth the money.
A while ago I was playing with chatgpt and had it create me a text based game. It took some coaxing but eventually it actually created a pretty compelling beginning to a game at least. This wasn't D&D, I had chatgpt just make the rules but the interesting part was creating a compelling story. Maybe it would have fallen apart after a little bit of game play but it was neat. I could see the fun of having completely unique and instantly created modules to run through.
I made something like this to let people explore the lore of my game. It has an AI act like a dungeon master as you play through a story. You can try it for free at:
https://orbsccg.com/adventures
The main game is a strategic collectible card game, but the “adventure mode” is just for fun AI-powered stories.
Yay, can't wait to read this. I spent a lot of time tuning prompts for gpt4 DM sessions, and I had a lot of fun. It has SO much potential, and I was pleasantly surprised!
I am not really considering mainstream commercialization, but I just found it really fun to interact with. We just need a customized model full of fantasy text and a little DM RLHF. It's essentially just a story-telling llm, which obviously will need some work before consistent narratives can be delivered to the user.
Not an AI expert or anything, just wanted to share my excitement about this. I urge you to try it out, if you're into this kind of thing. Tell ChaptGPT it's a DM, explain the rules, give it some context on how deliver the narrative and player choices. You can give some story or character exposition to give it a baseline. Its not too bad.
I don't have any experience with D&D but wanted to learn it to play it with my kids. I used claude to learn the basic rules and then had it help me create the scenario. Then I played as the dm while it simulated being the players and then vice versa.
My experience with D&D and LLM is that is a cheat sheet and a great helper, specially for newbies like me, I'm a Dad of 4 that plays DnD to teach some lessons and to get my kids out of the screens
A good use for this is people wanting to try an RPG, but not having a bunch of people to do that with. A parent wanting to try out this D&D thing so he or she can run it with the kids, for example.
Not having a bunch of people to do that with is solved by going out and creating relationships and communities. This is the basis of democracy, of society.
It's a tremendous loss for our species if we lose the impetus to connect. Do your kids a REAL favor, and build them a community—a SOCIETY—that they can join.
I found it to be a pretty OK first draft character creator, scene described and occasional lore/world builder. Not really anything more. Kinda a DM assistant more than a DM.
I’ve used an llm to help my niece, who is a performer, and has a flair for story, but no rpg experience dm as a human - llm team up. It was very successful
An ideal entry role for LLM in tabletop may be NPC companions/followers in the party that players/DM don't care for roleplaying or find tedious. Set it to listen in on conversations or VTT chat, and interject dialog on a rate-limited basis. Could lead to unusual situational ideas or at least humorous moments.
Yeah, completely automatic DMing just makes it like an open-ended computer game. On the other hand, the AI generated NPC responses are better than I could have come up with quickly, which makes me think co-DMing with an AI partner might be fun, particularly if they get better at remembering the details. That would let the human DM control the flow of the story without worrying that Thug #4 has taken 3hp of damage and is Dazzled. Of course, I most often use Foundry or Roll20 these days and they do a lot of that for you.
I wonder how well it could do for game summaries and recaps?
Like others, I don't really understand why anyone would desire this.
D&D is a social activity, why would I want to remove the most core human component of a social activity. It'd be like someone who enjoys going to a sports bar with friends deciding they'd rather sit next to a vending machine.
Baldur's Gate 3 (and other games of its nature) are narrative experiences crafted by writers to convey a story. I don't play BG3 when I want to play D&D, and when I do want to play something like BG3 I certainly wouldn't be interested in a machine generated amalgamation of other people's ideas with no thought or emotion behind it - that would be fundamentally pointless and a waste of my time - like reading Lorem Ipsum instead of a novel.
This is so interesting that people want this. This is beyond the guy saying “people don’t like making music so we made ai so it’s easier and faster to just prompt it to make music” . Like in my experience the best part of d&d is the dungeon master who knows their friends making a story and details specifically for fun and the fun of the group. And the interactions between players and the dm is where I derived the most enjoyment, automating that sounds insane and peak confusion/self defeating.
> This is so interesting that people want this
I love being a DM but it is no mystery to me. Being a DM is a lot of work. And occasionally, a lot of annoyance. You not only have to facilitate (and sometimes create from scratch) an engaging story that keeps people coming back, you have to wrangle busy schedules and personalities that is often annoying. Lots of times you'll have a bunch of people willing to play, but no one willing to DM. So to me it's not surprising at all, and I experimented with AI in running a game, and quickly concluded it could never be good at it because it's so insanely suggestible and has poor memory.
I tend to agree. I used to run Call of Cthulhu in high school, and there was a lot of preparation involved in running a good scenario. Some people are good at winging it and making stuff up as they go along. I never was. The best experiences for me and my players was always when I'd meticulously designed my own scenario, or used a store-bought scenario and read it thoroughly. Either way, hours of prep.
I would love to have an AI as an assistant Dungeon Master (or game master, or Keeper, or what have you). That is, one person in a group of players maintains the role of a master storyteller, but the AI is ready to fill in details or suggest ways to get the players back on track. This would probably be tedious if you're interacting with the LLM entirely through text, and having to manually keep it up to date with the story. But it could work well if you have a model that understands spoken language listening in on the game and generating cool images and making private suggestions to the game master.
> one person in a group of players maintains the role of a master storyteller, but the AI is ready to fill in details or suggest ways to get the players back on track
As a player and very occasional DM myself, filling in details and trying to get players back on track is where is the fun and challenge. AI could definitely be useful to handle all the paperwork (fight resolution and so on) though
I think the previous posters point was that an LLM DM is boring due to it's generality and lacks the specific quirks of having a human DM with their own viewpoint and understanding of the players, which I agree with. I did some experiments with earlier ChatGPT models for world building and DMing and found it to be so boring, getting it to do anything interesting or giving any kind of push back was so hard. Since they are an averaging of all knowledge / previous creativity you end up with a smooth surface, but part of what makes stories interesting are the rough edges.
However I also agree with you that being a DM is a prohibitive amount of work for someone, say, with kids and a job. It would be awesome to have an LLM as an assistant, maybe feeding in parts of the story and querying it for ideas when you're in a bind. But having it run as a full DM, at least right now, will likely lead to a boring experience.
Yep, my experience using ChatGPT as a helper is very good. It can very quickly cook up a custom monster, give it relevant stats, and create some flavourful actions and attacks to make it unique. Lately I've been trying to hook this up to D&D Beyond, as it's just some fairly simple POST payloads. If I can get to the point where I can describe a vague encounter, and run it in the Beyond encounter tracker with zero effort, that would be amazing, and a lifesaver when the party goes off and does nutty stuff. Beyond that, its story ideas are fairly generic but fine if you really have no time, and it can do a decent job of the occasional flowery monologue if you need it.
If you make progress on the api part, would be awesome to see a write up on HN about it, I'd be really interested in how this develops!
Personally I can't stand to run a D&D game as players take it all too seriously.
I will run Toon, Call of Cthulhu or Paranoia any day, the latter without a scenario as I can kill off their all their clones in my very dangerous Alpha Complex and have them rolling on the floor laughing the whole time with a help of a stack of prerolled character sheets and some random encounter tables. (I'd expect an LLM to be able to do the same)
Contrast that to the famous Bloodstone campaign which is the pinnacle of D&D scenarios but can't really be challenging to the players because players have to win over and over again if you're going to use most of the material.
I DM for early teens and I would kill for a D&D party that took it all too seriously. :)
Try Tales from the Loop -- https://freeleaguepublishing.com/games/tales-from-the-loop-r...
Try a simpler game. I think a lot of people think TTRPG = D&D and the world is worse off for it.
There are numerous tactics that work to face the "not serious" problems you have in a game for "young adults".
For instance, serious players get resentful when somebody not serious comes in late, forgets their character sheet, etc. You could lose either or both of them, but with a little prep you don't have to. In Paranoia it's easy to shove a preroll into their hands, give them a 1-2 minute briefing from "The Computer" and build up tension around this character who mysteriously appears (and if the new player complains that they don't know the rules tell them they're not allowed to know the rules!)
Toon is the epitome of "not serious" and, like the other two games I've mentioned, is a game where you can buy one book and have everything you need, including your first scenario.
I think whatever rules we tried to play, the nob jokes would persist.
That would be my takeaway too. There is room for AI DMs, but they need to get a lot better before they're really fun to play with. I tried a few AI RP just to see how it works and it was pretty boring.
i mean this is on point regarding the effort needed to run a game, but also llms don't really solve any of the problems you listed, so while people want _something_ it doesn't seem like they're crying out for this. also, would love an example of the suggestible ai DM if you have one handy lol
Two reasons:
1) "LLM DM" isn't merely mean substituting for human DM in a D&D session. The same capability can be used as a component in a video game, to breathe life into the game world, have it interactively react and evolve along with the player.
EDIT: Take a game like Rimworld, that relies on a scripted RNG dubbed a "storyteller", to decide what random events to hit you with and how hard. It's fun early on, but if you're into role-playing, you'll quickly realize there's no evolving story behind it, just stateless RNG. An LLM DM is exactly what could add that story, make overcoming challenges feel meaningful and allow for player decisions to actually impact the world deeply.
2) There are people like me, who would love to participate in an RPG session, but for various reasons never got invited to those when at school, and now, due to demands of parenthood, can't exactly make time to coordinate with the few people around who are still playing.
There are more, but those are the two that are apparent to me.
> An LLM DM is exactly what could add that story, make overcoming challenges feel meaningful and allow for player decisions to actually impact the world deeply.
No it's not. I don't think you're going to find an LLM with a large enough context window to have a meaningfully involving story spanning multiple sessions.
An LLM isn't going to craft a story element tailored to a character, or more importantly, an individual player. It's not going to understand Sam couldn't make last week's session. An LLM also doesn't really understand the game rules and isn't going to be able to adjudicate house rules based on fun factor.
LLMs can be great tools for gaming but I think their value as a game master is limited. They'll be no better a game master than a MadLibs book.
> I don't think you're going to find an LLM with a large enough context window to have a meaningfully involving story spanning multiple sessions.
First, you don't need much of any context window because you can finetune the LLM. Don't mistake specific engineering choices and tradeoffs and deployment convenience for intrinsic limitations of the technology.
Second, LLMs like Gemini now have context windows of millions of tokens, corresponding to millions of words. Seems like enough for 'multiple sessions'.
> An LLM isn't going to craft a story element tailored to a character, or more importantly, an individual player. It's not going to understand Sam couldn't make last week's session. An LLM also doesn't really understand the game rules and isn't going to be able to adjudicate house rules based on fun factor.
An LLM can do all of that, and you definitely do not know that they can't.
> They'll be no better a game master than a MadLibs book.
They've been better than a Madlibs book since AI Dungeon 1 which was like 6 years ago.
Have you actually used Gemini? I use it a lot for translation, and its context window is more like 150k tokens, rather than the 2M context window they say it has.
Be that as it may, long context window models which are good are not a mirage. By say late 2027, when the LLM providers figure out that they're using the wrong samplers, they will figure out how to get you 2 million output tokens per LLM call which stay coherent.
> I don't think you're going to find an LLM with a large enough context window to have a meaningfully involving story spanning multiple sessions.
Sure you will.
> An LLM isn't going to craft a story element tailored to a character, or more importantly, an individual player.
Sure it is.
> An LLM also doesn't really understand the game rules and isn't going to be able to adjudicate house rules based on fun factor.
Sure it will.
You need to use the tools for their purpose, not for the opposite of it. LLMs have finite context, you need to manage it. LLMs don't have a built-in loop, you need to supply it.
Character stats, names, details about players - those are inputs, and structured ones at that. LLMs shouldn't store them - that's what storage media are for, whether in-memory or a database or a piece of paper. Nor should they manipulate them directly - that's what game systems are for, whether implemented in code or in a rulebook run on a human DM. LLMs are to make decisions - local, intuitive decisions, based on what is in their context. That could be deciding what a character says in a given situation. Or how to continue the story based on worldbuilding database. Or how to update the worldbuilding database based on what it just added to the story. Etc.
> Character stats, names, details about players - those are inputs, and structured ones at that.
Some details about players are structured and can be easily stored and referenced. Some aren't. Consider a character who, through emergent gameplay, develops a slight bias against kobolds; who's going to pick up on that and store it in a database (and at what point)? What if a player extemporaneously gives a monologue about their grief at losing a parent? Will the entire story be stored? Will it be processed into structured chunks to be referenced later? Will the LLM just shove "lost a father" into a database?
Given current limitations I don't see how you design a system that won't forget important details, particularly across many sessions.
> who's going to pick up on that and store it in a database (and at what point)
LLM might, if prompted to look at it, or if there was a defining moment that could invoke such change. It won't pick on a very subtle change, but then most people reading a story wouldn't either - this is more of the kind of stuff fans read into a story when trying to patch potential continuity issues.
> What if a player extemporaneously gives a monologue about their grief at losing a parent? Will the entire story be stored? Will it be processed into structured chunks to be referenced later? Will the LLM just shove "lost a father" into a database?
The scale depends on the design, but I'd say yes, shoving "lost a father" into a database so it pops up in context is a good first step, the next step would be to ensure the entry looks more like "mentioned they continue to grieve after loss of their father <time ago>", followed by a single-sentence summary of their monologue.
Personally, I had some degree of success with configuring LLM (Claude 3.5 Sonnet) for advising on some personal topics across multiple conversations - the system prompt contains notes in <user_background> and <user_goals> tag-delimited blocks, and instructions to monitor the conversation for important information relevant to those notes, and, if found, to adjust those notes accordingly (achieved by having it emit updates in another magic tag, and me manually apply them to the system prompt).
> Given current limitations I don't see how you design a system that won't forget important details, particularly across many sessions.
It's not possible. Fortunately, it's not needed. Humans forget important details all the time, too - but this is fine, because in storytelling, the audience is only aware of the paths you took, not of the countless other possibilities you missed or decided not to take. Same with LLMs (and larger systems using LLMs as components) - as long as they keep track of some details, and don't miss the biggest, most important ones, they'll do the job just fine.
(And if they miss some trivia you actually care about, I can imagine a system in which you could ask about it, and it'll do what the writers and fans always do - retcon the story on the fly.)
I have solved it in my two games by using two systems in my future game: 1. LLM "text" info ,about world, player, text descriptions of world/decisions and so one. 2. Typical D&D stuff with rolls, names, details, decisions that are simple logic.
His example:
Rimworld is a great universe where we think about characters' stories, and there's really just like, a couple dozen attributes and straightforward ways they interact with the game.
An LLM context window could easily have 20 times as much interpersonal state, and make it interact in much more unexpected (but plausible) ways. That's going to be a surprising and rewarding gaming experience once someone figures it out.
Context window is arbitrary and can be adjusted on demand. When a new random event needs to be generated, and then fleshed out, the context can contain e.g. list of facts about the story so far, the overall story arc, summary of main characters or plot threads. This can be used by LLM to decide e.g. which faction will attack, and who will be on it, and what their goal will be, etc. After the event concludes, it just becomes another line in story event history. Meanwhile, that can be fed to a differently prompted LLM to evolve the plot arc, update motivations of background characters, etc.
I have a feeling people imagine LLMs as end-to-end DMs that should somehow remember everything and do everything in one inference round. That's not what they are. They're building blocks. They're to be chained and mixed with classical flow control, algorithms, and data storage (as well as the whole interactive game mechanics, in videogame context).
Maybe. But for something like Rimworld++++, you don't need all that sophistication. You need a pile of relevant facts and a clear task to determine something about the game or write some text for the user. Sure, curating and retrieving selectively would probably be even better and more flexible.
>I don't think you're going to find an LLM with a large enough context window to have a meaningfully involving story spanning multiple sessions
You don't need to provide every single previous information to llm, use LLM to summarise previous ones and it gets really compact. It works quite well.
DMing is, in my opinion, very time consuming. Prepping an adventure, familiarizing yourself with the rules, trying to incorporate character backstory in to the narrative, all takes time. As someone that can only play D&D if I am DMing, because none of my friends have the desire or inclination to do so, I would be very happy to pay money for an LLM dungeon master that lets me just play the game without all the prep time. I just don't have 4 extra hours a week like I did in college.
> Like in my experience the best part of d&d is the dungeon master who knows their friends making a story and details specifically for fun and the fun of the group.
Sure.
There are two reason why I can think of someone making a Dungeon Master LLm.
One is that when there is no cake we eat bread.
Don't get me wrong. I'm a DM, and I love playing d&d with my friends. I totally agree with the sentiment you are sharing. But people who are willing, and able to DM for others are not evenly distributed. There is a lot more people who would like to play TTRPGs than people who is willing to step into the DM role.
So in that sense think of this as a substitutions for those games which would not happen otherwise, because they don't have a DM. Or the only person who would DM them is an ass. Or their DM has burnt out.
Is some game better than no game? Sometimes. Depends on how good that game is. And we won't know how good the substitute can be without trying. Heck maybe more people will play, and they will realise how easy is to DM actually.
The other reason is the sheer challenge of it. Dnd has a lot of rules. There are the obvious ones you can read in the book. But there are also un-written rules. Like object-permanence. If a goblin steals a diamond ring from us, and we slay them within minutes they damn well have the diamond ring on their bodies somewhere. If three displacer beasts ambush us and we slay 2 there damn well be 1 more accounted for in some sense.
There are also "story-writing" rules. If we went through hell and high-water to obtain an arrow of dragon slaying after the blacksmith told us about the legend of it, he better not just pull one out of his ass the next time we see him. If the whole lore of goddess X is that they are kind and caring then they should better be at least not cruel when we meet them. These are all hard for an LLM. They are also easy for a human to evaluate. Because we just feel when they are not right. Therefore it is a good challenge to evaluate how good we are at this "making a bucket of sand smart" task.
Consider video game RPGs. They're largely based upon D&D, yet they remove the dungeon master entirely. They are enormously more popular than D&D itself.
Now imagine something in between, where you have a video game, but the NPCs and parts of the story are controlled by an LLM. You can give the players much more freedom, and the creators don't have to write thousands of lines of dialog to account for every possible choice a player can make. The game doesn't have to be "on rails" so much, the LLM can help speed the story along when the player gets confused, and you can have NPCs with much more depth than just several lines of static dialog.
How well this will work in practice remains to be seen. In my experience, ChatGPT itself is a painfully generic DM that relies upon repetitive fantasy tropes. You still need an actual human being to create an interesting story and add depth to the world.
One of the growing trends these days is solitaire RPGs -- where people play alone as both player and Game/Dungeon Master. Typically this isn't done by AI but by rolling on various tables (sometimes called "GM Emulators"). Not everbody has the time or even enjoys the extrovert pressure of a traditional social RPG. A related concept is the "cooperative RPG" which does have multiple players but no DM so everyone can play.
Here's a list of some of the ones available (about 10 years old but it gives you the idea)
https://rpggeek.com/geeklist/181957/list-of-solitaire-soloab...
This is interesting.
I grew up with “choose your own adventure” books which were like a solo adventure: (If you go on the north road turn to page 34 if you follow the river page 57)
Many board games now have a solo mode with an automated player with tables and dice to help randomness.
https://boardgamegeek.com/thread/3238902/what-is-the-best-so...
Obviously there are adventure video games. Some of the new ones often have interesting back stories, really amazing world building and dnd like adventures. Horizon zero dawn, the Witcher, Star Wars outlaws among many others. “baldurs gate” really gave me dnd vibes (I played through that one multiplayer).
That style of book led to one type of solo adventure -- the "numbered paragraph" type that is basically a choose your own adventure book except that combat and other such situations are resolved using the RPG rules. Those still exist, but more recently, the goal is to make things more open ended and flexible, at the cost of having the player make decisions on what things mean -- like the player might roll on some tables that say "orcs" and "merchants" and the player needs to invent their own explanation of what that means -- do they encounter orcs attacking merchants? Or orcs who are merchants?
Oh man, I played the shit out of the Lone Wolf books when I was a kid in the eighties and nineties. My brother had the first couple of them, and I still have his old books somewhere with his notes. I should find them for nostalgia's sake (my brother having long since passed).
The modern thing to do would be to automate the game, watch the bots play each other, and make a reaction video for TikTok.
Isn't this a modern take on the yarn that ended up with students all just bringing a tape recorder to the lecture hall after the lecturer just played a pre-recorded tape as the lecture?
That was a scene from the movie "Real Genius"
Try doing this while you're driving with the gpt4o advanced audio mode. It's a nice way to kill 40 minutes or so without having to look at a screen. Just tell it you want to play an RPG with it, give it a setting, describe your character and go. It'll mostly be more of a choose your own adventure than a real RPG, but it can sometimes be quite entertaining.
Agreed on DM making the experience special. The cognitive load on a DM is very high though. They have to digest information on the game setting and the monsters, synthesize scenes and react to players' interactions all the while staying true to the narrative. I think LLM's can play a big part in assisting human DMs by either providing low-fidelity ideas for "inspiraton", commonsense based on their training on D&D texts and literature or high fidelity text in some cases to be directly presented to the players.
I think it's more about the challenge/curiosity of creating the implementation. I'm not sure these AI driven text-RPGs will find long term audiences.
> I think it's more about the challenge/curiosity of creating the implementation.
Totally this, at least for me, in some circumstances.
TOU's be damned, I've written bots for some online games I've played, not because I want the XP or money or whatever that the bot could do without me working for it, but rather because I found writing the bots fun and engaging.
Before anyone gets in an uproar, I didn't sell them, nor any of the in-game resources gained from them. I was watching them basically all the time, because that's what I was there for - to see my creation work. And I purposely didn't interfere with other "real" players.
They are a step on the road to fully responsive rpg worlds though, which don’t have to be text based.
It's been written at length here and elsewhere by game devs, but this isn't a thing that anyone would truly want. A purely AI generated or controlled world would have no constraints, and be fundamentally untestable - games aren't really games unless they have constraints. Even the 'purest' sandboxes have some kind of constraint buried within them, and I think you'd find an RPG of this type extremely boring, at least with current technology.
> this isn't a thing that anyone would truly want.
Citation needed.
> A purely AI generated or controlled world would have no constraints
That's a shitty AI then. Make a better one. I can play 2000 Vampire: The Masquerade games with 2000 different groups. They will each be different, but they will also be each distinctly Vampire: The Masquerade ttrpg games. If the AI you are thinking about can't do the same, then think of a better AI.
> at least with current technology.
Well. Who is the group who will make the "next technology"? Should we work on that, or just lay down on the ground and wait for it to fall from the sky? Testing what are the limits of the current technology (as done in the paper we are talking about here) is the way to get there. Or at least to systematically answer the question of where and what are we lacking.
> Citation needed.
Lol, a citation of what? This is my opinion statement and the rest of my post follows it.
> That's a shitty AI then. Make a better one. I can play 2000 Vampire: The Masquerade games with 2000 different groups. They will each be different, but they will also be each distinctly Vampire: The Masquerade ttrpg games. If the AI you are thinking about can't do the same, then think of a better AI.
Sure, I'll get right on that.
> Well. Who is the group who will make the "next technology"? Should we work on that, or just lay down on the ground and wait for it to fall from the sky? Testing what are the limits of the current technology (as done in the paper we are talking about here) is the way to get there. Or at least to systematically answer the question of where and what are we lacking.
I'm really unsure of what or who you are addressing here but it certainly isn't anything I've written in my post.
> citation of what?
Citation that nobody wants what you described? The sentence which I was quoting.
> This is my opinion statement
Your opinon can be that "I don't want this." "nobody wants this" is refers to things outside of your head. Do you see the difference between the two?
> it certainly isn't anything I've written in my post.
Your post is suffused with defeatism. The 3 "no"s it contains are: "nobody wants this", "with current technology this cannot be fun" and an implicit "we can't make the next technology". I believe each of those are wrong, and I'm calling you out on the attitude.
Ok, then what advances are being made in AI technology in this gaming that lends you such confidence? Care to make any citations yourself here? I don't really care what you think of my attitude, nor does it make for productive discussion or good posting.
> Care to make any citations yourself here?
Happily. I know you are wrong on the "nobody wants this" statement because I want it. With a sweeping generic statement like "nobody wants this" a single example is enough to disprove it. There you have it.
> what advances are being made in AI technology in this gaming that lends you such confidence?
There is a ton of experimenting going on. AI Dungeon and Deep Realms are the two obvious ones. I don't think anyone has found the golden solution yet, but that is also not evidence that no such thing exists.
I don't really buy this. I don't think the tech is quite there, although at this point it might be a matter of clever prompting more than a fundamental limitation depending on the type of game you have in mind. A strong enough DM AI could take a prompt like "I want to play a game with a similar loop to Satisfactory, set in Mordor at the beginning of the second age." And the AI could figure out from there, including devising appropriate constraints.
There's no reason why an AI-driven sandbox cannot have constraints, as well.
Now it's true that, with the current crop of LLMs, a persistent enough player would always be able to break through them. But if it takes conscious and deliberate effort, I think it's reasonable to say that whatever experience the person gets as a result, they were asking for it.
I mean this isn't it fun to generate ancient rome/alice in wonderland/medivial Poland kind of the world?
I guarantee you that anyone with knowledge of how these games are run would not want an LLM being a DM.
Its the type of problem which requires a good balance of storytelling, preparation, and tailoring the interactions towards the psychology of the people who are playing (in specific beneficial and entertaining ways).
Its a problem computers won't likely solve anytime soon.
I am an “everything is custom tailored to my players” DM and I use LLMs quite a lot as an assistant. Speeding up prep time is always welcome. Using an LLM directly as a DM? Oh hell no.
Also, I ran a session over Zoom and their AI summary was so useful!
This is nothing new. Bringing ideas from DND into video games is as old as video games themselves. In interviews, John Carmack talks often about how he used to play DND with his friends and then try to take that experience into the games he made. And it goes back even further than that - Rogue, the primogenitor of all roguelike games, was inspired by DND as well. I could find a bunch more examples, but you get the idea.
I largely see it as the digital next step up from something like the Mythic Game Master Emulator, which is popular for solo play and things like trying out new systems.
And there's quite a lot of us that like to play but because of life commitments, getting together with a group on a regular schedule is difficult.
DMing takes a huge amount of effort and time.
Most likely use of this is that a DM runs it, and overrules/augments its output when necessary/as they see fit.
Some people don't have friends they can play with :sadge:
This. At home it's me and my one kid, and having an additional party as a DM would be priceless.
I was watching a video on YT, heard a cool song, tried to soundhound it, turns out it was an AI generated song, I felt like I was "owned" damn.
If you enjoyed it, does it matter?
Preparing a campaign is an awful lot of work, which at least in my case is why my friend group almost never plays.
>This is so interesting that people want this.
Go outside. There's people for everything.
He said it was interesting, not baffling.
And I said what I said.
And all tautologies are tautological.
At least make an effort to understand what a tautology is.
And yet, that's exactly what computer RPGs are all about, one of the most popular video game genre is essentially about automating dungeon masters. It includes early games like Colossal Cave Adventure and Zork, which are chatbots not unlike a very primitive version of ChatGPT.
If you have a good role playing group, with a DM you enjoy playing with, no computer game nor LLM will replace that. But if what you are looking for is the convenience of computer games with the freedom of tabletop RPGs more than the social aspect, then LLMs totally make sense. And even with a social group, it can work as substitute if you don't have a DM (DMing is hard work!).
they think they want it -- that doesn't mean they will actually like it if they get it
I can totally see the appeal of using LLMs to _help_ the DM. Not to replace it.
I DM a few games of The Expanse, and using LLMs to plan ahead was a godsend. No, I didn't utilise it to write the story for me - instead I used it to test my planned story and see which way my players might strafe off the road, so I can plan for those. Basically simulated a game using an LLM that acted in place of multiple characters, allowed those to run free (within certain limits, obviously you can't have the LLM players do a dozen actions without the DM having a say), and essentially mapped the potential "off branches", story pathways I didn't plan for initially. This has allowed me to be prepared for the usual dumbass things players might do, such as heading for a strip club in the middle of a total disaster where they (figuratively) have dozens of arrows pointing to the goal of the chapter.
Another interesting aspect of using AI for TTRPGs is to create atmospherics. For Expanse based games, I've bought a number of tile packs and such, to appreciate the artists who put work into it, but I simply don't have the funds to commission a few dozen acrylic matte style scenery images (which I usually put up on my projector, combined with some Hue lights to create the visual atmosphere). With AI, I can even generate them on the fly, should my little gremlins stray off the path. Same for music - AI can incredibly easily generate an atmospheric soundtrack that fits the current scenery, with just a few words, while I can still pay attention to the players.
But fully replacing the DM? That's silly.
Just that finding a good DM is hard
Last year, I built a DM for myself using the OpenAI api and an elevenLabs voice generator. I asked it set my character in Baldur's Gate, so it could pull upon the huge amount of DnD source material it had been trained on.
A few takeaways:
1. An LLM based DM can give the player essentially infinite richness and description on anything they ask for.
2. There is difficulty in setting the rules for the LLM to follow that match the DnD rulebook. But this is possible to solve for. Also, I found the LLM to be too pliable as a DM. I kept getting my way, or getting my hand held through scenarios. Maybe this is a feature?
3. My conversation quickly began to approach the context window for the LLM and some RAG engineering is very necessary to keep the LLM informed about the key parts of your history.
4. Most importantly, I found that I most enjoy the human connection that I get through DnD and an LLM with a voice doesn't really satisfy that.
> Also, I found the LLM to be too pliable as a DM. I kept getting my way, or getting my hand held through scenarios. Maybe this is a feature?
LLMs are fine-tuned to be "helpful assistants", so they're basically sycophantic.
This was my experience too. The short context and the optimism bias make chatgpt the wrong solution.
It starts well and then NPCs become inconsistent and the DM basically lets you craft the story by constantly doing a "yes and".
It becomes boring because the stakes feel so low.
> My conversation quickly began to approach the context window for the LLM and some RAG engineering is very necessary to keep the LLM informed about the key parts of your history
Assuming we're talking about GPT-4o, that 128k context window theoretically corresponds to somewhere around 73,000 words. People talk at around 100 words per minute in conversation, so that would be about 730 minutes of context, or about 12 hours. The Gemini models can do up to 2 million tokens of context... which we could extrapolate to 11,400 minutes of context (190 hours), which might be enough?
I would say GPT-4o was only good up to about 64k tokens the last time I really tested large context stuff, so let's call that 6 hours of context. In my experience, Gemini's massive context windows are actually able to retain a lot of information... it's not like there's only 64k usable or something. Google has some kind of secret sauce there.
One could imagine architecting the app to use Gemini's Context Caching[0] to keep response times low, since it wouldn't need to re-process the entire session for every response. The application would just spin up a new context cache in the background every 10 minutes or so and delete the old one, reducing the amount of recent conversation that would have to be re-processed each time to generate a response.
I've just never seen RAG work particularly well... and fitting everything into the context is very nice by comparison.
But, one alternative to RAG would be a form of context compression... you could give the LLM several tools/functions for managing the context. The LLM would be instructed to use these tools to record (and update) the names and information of different characters, places, and items that the players encounter, important events that have occurred during the game, as well as information about who the current players are and what items and abilities those players have, and then the LLM would be provided with this "memory" in the context in place of a complete conversational record. The LLM would then just receive (for example) the most recent 15 or 30 minutes of conversation, in addition to that memory.
> I found the LLM to be too pliable as a DM.
I haven't tried using an LLM as a DM, but in my experience, GPT-4o is happy to hold its ground on things. This isn't like the GPT-3.5 days where it was a total pushover for anything and everything. I believe the big Gemini models are also stronger than the old models used to be in this regard. Maybe you just need a stricter prompt for the LLM that tells it how to behave?
I also think the new trend of "reasoning" models could be very interesting for use cases like this. The model could try to (privately) develop a more cohesive picture of the situation before responding to new developments. You could already do this to some extent by making multiple calls to the LLM, one for the LLM to "think", and then another for the LLM to provide a response that would actually go to the players.
One could also imagine giving the LLM access to other functions that it could call, such as the ability to play music and sound effects from a pre-defined library of sounds, or to roll the dice using an external random number generator.
> 4. Most importantly, I found that I most enjoy the human connection that I get through DnD and an LLM with a voice doesn't really satisfy that.
Sure, maybe it's not something people actually want... who knows. But, I think it looks pretty fun.[1]
One of the harder things with this would be helping the LLM learn when to speak and when to just let the players talk amongst themselves. A simple solution could just be to have a button that the players can press when they want, which will then trigger the LLM to respond to what's been recently said, but it would be cool to just have a natural flow.
[0]: https://ai.google.dev/gemini-api/docs/caching
[1]: https://www.youtube.com/watch?v=9oBdLUEayGI
I did something similar, but tried to get several agents to play a DnD round together. It basically worked, but was insipid.
The main problem with doing this on chatgpt is that it is _relentlessly_ cheerful and it's almost impossible for it to produce any kind of interpersonal drama or threat or danger.
I tried to get it to run space opera about a diplomatic mission to a newly recontacted planet, full of Game of Thrones-style intrigue and plotting and scheming, and it was like playing with a cheerily optimistic new UN intern that completely believes in the power of compromise and diplomacy to solve all problems.
It was always like: "And so the two of you hashed out your differences over a glass of Dubranian Forblik, and peace reigned forever and ever after."
That's true of how it does role playing games and any kind of fiction at all -- it always wants to tie a neat bow on it at the end of every few paragraphs with a moral and everything.
You also have to ask it to use python to dice roll for anything you want to potentially fail on, or it will always make everything you do succeed wonderfully.
I will say this -- the advanced audio mode for D&D is amazing and it really does act out the characters and it will occasionally slip into different voices or even add sound effects, even though it's not "allowed" to. It will also sometimes copy your voice and act out what it thinks you should do instead of letting you talk, though.
The researchers encountered this.
> raised by 3 out of 7 players was the perceived lack of danger for their characters. Participants noted that the gameplay did not present sufficient threats or challenges, which diminished the sense of urgency and excitement typically associated with D&D adventures.
Indeed. But for stuff like this, ChatGPT is overkill. It's better to get a dedicated RP finetune of LLaMA, Qwen, or some other open weights model (you can still run it in the cloud if you don't have hardware to do so locally). There are enough finetunes around by now that you can "dial in" how dark you want it, but for some examples:
https://huggingface.co/jukofyork/Dark-Miqu-70B
https://huggingface.co/SicariusSicariiStuff/Negative_LLAMA_7...
Just curious, how do you keep up to date on these models? Is there a community out there that discusses them?
https://old.reddit.com/r/LocalLLaMA/ is the place to be, but it's a subreddit with an eclectic population, so apply some caution when browsing it at work.
r/LocalLLaMA has the discussions, but on top of that, I just periodically browse new model lists on Hugging Face. There's a lot of stuff, but most low-effort finetunes tend to focus on small models (since that's much cheaper and faster), so if you only look at 70B+ ones, there's a lot less garbage there.
Reminiscent of the Twilight Zone episode where a criminal ends up in the afterlife. Spoilers in link https://en.wikipedia.org/wiki/A_Nice_Place_to_Visit
The old gpt2 engine that used to drive AI Dungeon had the opposite problem, it was obviously trained on a ton of really bad fan fiction and would frequently either shift into filthy erotica or some kind of hyper violent serial killer story, no matter how innocently the story started.
I loved AI Dungeon for generating funny/stupid fanfiction style adventure/action/fantasy/scifi stories. It could keep track of characters and settings, have them return with motivations. And then something unexpected would happen. Provided me with a lot of entertainment.
But my problem with using it as a DM is that you have to police yourself. I could just declare that I did something and it would automatically succeed. There were no rules or antagonism in the system. It was more of a collaborative storytelling tool than a game.
To be fair I haven't used it in years so I dunno how it works nowadays.
Yep, in my game it tried always to "save" player it was impossible for LLM to jail or kill a player to end a game. I had to explicitly end it by game logic, not by LLM.
I am baffled as to why anyone would want this, or anything like it.
Sure, it's a cool technical demo. But... the point of D&D is a social game played with others. Even if you play a D&D-like video game such as Baldur's Gate 3 (which I'd argue is a fundamentally different experience to playing a tabletop RPG anyway), you're experiencing a world and a story that someone else has crafted.
What's the point of replacing that social interaction, or that connection to another person's creative vision through their art, with an LLM? What value can it ever provide?
I would love a fully generated D&D campaign where I just input some parameters, like a writing prompt, and get a tailored LLM Dungeon Master organizing it. Whether it's the result of someone's creative vision or not is irrelevant to me.
I imagine for people like me who do not have anyone nearby willing to play D&D
Right, but why not play BG3? You know that an LLM is not going to provide any meaningful narrative, or craft a thought provoking story.
> You know that an LLM is not going to provide any meaningful narrative, or craft a thought provoking story
The title of the paper is "Exploring the POTENTIAL of LLM-based agents..." Right now, yeah, just play BG3. But the context of the discussion is its potential, which is certainly worth discussing imo.
Find someone! Make someone! Go online!
Dear god, it's crucial that adults be capable of creating and maintaining relationships. If it's impossible then, well, that's hell on earth.
Go out and make a new relationship! Do you live in Antarctica? And, if you live somewhere where new relationships can't be made, my god, get out of there! Go, go now! Go somewhere you're allowed to be human!
I've been in a weekly D&D game for over a decade. I don't keep playing it because of the game of D&D (certainly not, I could rant for a long time about the game's myriad shortcomings across many editions). I play it because of the friends I play it with, because of the social connection that exists in and around the game.
To play a tabletop RPG without that isn't the same thing, and to play with LLM is bankrupt of meaning. You'd be better off picking up one of those old-school choose your own adventure books or a video game like BG3. At least those were written by someone with intent, at least there's meaning in the story and setting. Or, finding an online group.
People play games for a variety of reasons. While you enjoy the social aspect, others may find that to be the worst aspect of the game (or just don't have a social circle).
I grew up playing D&D alone in my room using LEGO. It was great and I still have some of the sets I built as dungeons and monsters that I built from reference pictures in the Monster Manual.
Video games are different from solo role playing. It's like trying to substitute shooting hoops in the driveway with NBA '24.
Would it be fair to say that you also don't appreciate solo rpgs?
> the point of D&D is a social game played with others
That's one of the points, or a benefit maybe, but its not the WHOLE point. Some people just enjoy RP and fantasy worlds, so a single-player campaign would still be enjoyable (to me, at least, but maybe I'm weird). I def don't see this replacing DMs in real groups any time soon.
I am not interested in this, but an AI DM that allow human players to play would keep the social aspect. Since being DM is a higher threshold of knowledge and time investment, this could unblock a social gathering, rather than substituting it.
Also, if it doesn’t try to keep the story to strict guardrails, the creativity and story creation would come essentially from the human players.
I don’t think those arguments are enough to make me want to use it, but I see the point and possible interesting use.
I've found chatgpt useful for trying out D&D settings that I don't normally get to experience in my usual group. It loses the plot after a while, but it's enough for me to figure out whether I dig the setting enough that I would want to pitch it for a real game.
There's a saying in DnD:
"Bad DnD is worse than no DnD"
I'll agree. I've had bad groups and, wow, yeah. I'd rather have just doom scrolled my phone for 4 hours.
So, an AIDnD is wonky now, but it's better than bad DnD (at least with some futzing about that I have done myself).
To really reach here: I think you're placing value on the thing due to the effort involved. That's kind of an old school Marxist way of assigning prices to goods. I'm trying to point out that the value isn't that way for everyone. The 'new' market way of assigning prices is 'whatever someone will pay for it'. And in that very tortured analogy, that means that AIDnD is just as valid if someone likes it enough to do it. The background effort isn't part of the determination of value, necessarily.
Last year, during a period of recovery, I hosted a ChatGPT game with my nieces and newphew. I played the role of being a bride between them and the LLM. I read aloud the text, and translated their actions into queries.
With slight nudges, GPT performed well. It excelled at NPC dialog, location description, and to some extent improvisation. However, eventually the campaign fell apart on session four or five. There was a disagreement between the players over exactly how many maps we had acquired. As you know, answering this question is a needle in a haystack problem and asking the LLM proved to illicit conflicting responses. It ultimately led to me concluding that pausing the campaign was the right choice and accepting that we all still had an amazing time up to that point.
Which leads me back to the questions in the paper: > We believe that with the above evaluation methods we can try to answer the following questions: • How consistent is a LLM Agent in its generative dialogue? • How well can a LLM Agent keep a user engaged in a narrative plot? • How creative is a LLM Agent when generating complex story driven narratives?
None of these are exactly relevant to my personal experience running a TTRPG game using ChatGPT and what makes it succeed or fail. What would make such a system an improvement is being able to store and retrieve facts about players (HP, stats, inventory), NPCs(names, location, history), and Places(items, descriptions, inhabitants) in order to solve the needs-haystack problem. Creating a world ontology and being able to CRUD facts against such an ontology would be a game changer(literally).
Despite the ultimate demise of the game, the experiment was a success. I'd highly encourage anyone with a laptop, a handful of dice, and a group of interested friends to give a shot. It's fun, memorable, and a unique way to pass the time and bond.
> What would make such a system an improvement is being able to store and retrieve facts about players (HP, stats, inventory), NPCs(names, location, history), and Places(items, descriptions, inhabitants) in order to solve the needs-haystack problem. Creating a world ontology and being able to CRUD facts against such an ontology would be a game changer(literally).
Curiously, those are the exact things LLMs can't do well, and are not really supposed to. This is all stuff you (or an external system) should keep track of and supply to LLM in context "just in time". This is to say, LLMs are an important component, but alone they won't work. They need their tools :).
In short, LLMs are pure "system 1", the complete solution still needs "system 2", which fortunately is more amenable to classical computing approaches.
Based on my experiences with this. Keeping context relevant is the hardest part. Your human GM will remember that funny remark from the session 3 years ago, but the LLM?
Also for that same module, is it able to integrate it in a broader campaign. I expanded the lore and why the players got here with extra quests, fluff and mystery and they only reached said dungeon at 2nd session only.
Also the way a PC died due to a rat bite that could only be cured by magic (5e of this module is ironically harsher than the 3.5) as he was losing max HP every 24H. As the players expressed dislike with that character the owning player was willing to go for a new character and I had planned a big drama scene where the players exit in emergency the dungeon and try to rally the nearet village where there's a priestessd and happens that I could match the rolls so he would die in the priestess hands.
But I made the mistake to allow them to spend the night to get their 3rd level which would have allowed the druid to get his cure illnesses spell botching the set up.
Happens that I decided to exploit the fact that all other character the previous days made an offering to Beshaba to curse that character and force rolls during night (and try to find a way to make the new character come into play).
How it unfolded : - they were sleeping and it was the bard's turn for watch
- the bard was a follower of Tharizdun and was about to sacrifice the dieing character as said character was agonizing
- the new character was a Kara-turian enforcer that was chasing a yokai (he didn't knew) that used other people as disguise, made it happen that the yokai choose the dieing character "by coincidence" appearance
- the yokai being chased opens a portal while hiding to the dungeon
- the new character litteraly pops through a Wild Magic portal that pushed away the bard and tries to bash the old character as the old character runes on his skin glow a last time as he dies and repels everybody
- made the new character pop naked as I wasn't paying attention to what the player was saying when he was listing his inventory
- I let them handle the big mess
It's that: Spacial awareness, struggling to maintain the personas (and often talking for the player/s), difficulty improvising, pacing issues, lack of initiative, and just an over reliance on prompting to keep going correctly.
I think we just haven't built complex enough architectures to allow this. I have a few ideas boiling that would help facilitate long term context understanding and recall.
Neuro-sama, an AI VTuber streamer, manages to do this. There are topics brought along multiple streams and manages to remember details of previous streams.
With soemthing like Neuro-sama I wouldn't doubt that Vedal is doing a lot of liftwork behind the scenes to keep sufficient context available to Neuro-sama's LLM.
And with the right kind of engineering, that could be automated too.
I don't think the results would be as good as they are now - Vedal is bringing his own perspective into it - but once you have a proof of concept, you can improve it.
Maybe this is a solution https://www.letta.com/
A context-based retrieval/memory system would be cool, too. Pull in relevant events based on the group's location and NPCs they're with.
I'd love to see some human & machine co-op for DM. That'd be super helpful.
Now this is solid, like an ai enhanced Evernote, but for organization, not generation/creation.
I've been meaning to take a crack at doing this - I'd like to use Whisper to transcribe session recordings and then have an LLM build out a campaign wiki. I enjoy being a DM but note taking and organization is my least favourite part.
I've used LLM's in this capacity, and it's awesome. It quickly becomes a crutch.
Alas, I was desperately keen to read transcripts of actual gameplay and all we got was user surveys?
Was there a structured combat system as sort of this or was it a talking only kind of thing? Who knows?
Trying to get empirical evidence that an LLM can even remotely plausibly work as a DM with no social aspect and (as far as I can tell, reading the paper) no programmed system of stats, rolls, combat etc. seems impressive.
…but was it that? Or just a DnD themed chatGPT conversation? Or a text adventure with an LLM describing each “room”?
Did the LLM actually have agency to make decisions and perform tasks, or was it not an agent at all, just a conversation?
Without a transcript, all you can really say for sure is yeah, I agree; more research is needed.
Using a fine-tuned GPT-3.5 model is definitely a choice, why not pick one of the newer models? The methodology only states they use "ChatGPT" as LLM, but doesn't clarify the model until way later, nor why they picked that model over other existing models.
The methodology also barely explains how they did their baseline experiments with human DMs. Did they do them face-to-face, or via text? Did they have different DMs in the 3 games they used as baseline, or was it the same person? As it stands the research is barely reproducible.
I find this idea fascinating, and I think a lot of very interesting emergent behavior can be added to video games by having something like an LLM act as a Dungeon Master. Rather than having to pre-program a bunch of choices that the player is allowed to pick between, have an AI model evolve the story and just move the pieces on the story board based on what the player actually does. Story driven games these days are still very much railroaded. They are like interactive movies. The player has very limited agency. They can't really pick any choice which the designers haven't thought to implement upfront.
At the same time, I'm personally not super enthusiastic about using LLMs for a pen and paper game. Let's assume for a moment that we have free will, then the most fun aspect of PnP dungeon mastering is the fact that the DM has agency. It adds an (IMO) very fun mini-game where the players try to get away with as much shenanigans as they can, and the DMs job is to try to thwart these shenanigans in the most entertaining ways possible. The keyword is on "try" - the GM has to put entertaining obstacles in front of the players to keep the game fun, but ultimately the players should have an avenue of success. Even if they don't always succeed, they need to succeed enough and when they fail they need to feel that it was fair.
I feel that with an AI DM, this goes out the window. At least with the current crop of AI, I would never feel that I'm facing an agent with it's own free will, rather just a machine. As such, it cannot replace a human DM for the PnP purposes.
This is extremely interesting to me. I understand why so many people are surprised that people would want this, but for whatever reason, I'm very much in the target audience.
I'm a "perpetual DM" in most of my roleplaying groups, having led campaigns spanning years, but very rarely do I get to participate as a player. My kids love me running adventures and one-shots, and while I enjoy it, people don't always recognize how much work it is -- both the prep, as well as the mental effort of being "on" for the entire duration of the adventure. I like afternoon one-shot adventures, but I dislike reading through long sourcebooks. I enjoy crafting puzzles and presenting challenges to the players, but also recognize the mental effort that this takes, and sometimes I just want a copilot to handle the lore and the setting. I already rely on the rules-nerds in my group to act as copilots for the specifics of game mechanics -- I don't have time to read through that many tables and commit them to memory.
Re: datasets, the paper authors used a dataset that they created by scraping Critical Role transcripts -- I doubt they had permission for that. I like contributing to open datasets for training open-source LLMs, and I think it would be really sweet for a playgroup to be able to contribute their play transcripts for such a thing. I would love to see a standard open-source dataset be collected for this, and perhaps even a standardized benchmark for quantifying the performance of particular LLMs in RPG contexts -- rules compliance, needle-in-haystack fact finding, consistency of roleplaying, understanding user intent, challenge creation / resolution, etc.
For those averse to LLM defeating the social aspect of tabletop roleplaying, perhaps a better entry role for an LLM agent is as NPC party companions/followers. Set up the agent with a detailed character sheet and hidden backstory/motivations and set it to listen in on conversations or VTT chat, and interject dialog on a non-spammy basis. Could lead to unusual situational ideas or at least humorous moments.
This is what our DM does. It works really well for NPCs and saves him a shit ton of time and effort.
I've made several DMs for a variety of entertainments. It's as much fun to create them as it is to play a scenario with them. I've experimented with a few methods, such as providing them with an ASCII map that looks like a hack/rogue map, giving them a famous book as the scenario, giving them multiple books and or famous fictional characters and then just letting them loose in a DM/AI guided narrative the user is making decisions.
Of course, I have a "classic" DM that happens to be visiting here on vacation from the dungeons & dragon's multiverse, where his real job is being a DM for visiting tourists. There's a "Shaggy" and his talking dog one can take LSD with when going to Woodstock in '68. There's a StarTrek strategic tactical simulations instructor. I quite like a Underground railroad "conductor" that guides escaped slaves to freedom before the US Civil War.
I think these DM / interactive narrative applications have a huge future in education and training.
I've created two AI LLM D&D-style games. The second one will soon be released as a D&D-style game with Visual Novel elements (AI-generated images_. Here's how I can address some common concerns from comments above:
Cons: I'm currently running a private beta and looking for co-founders (game/design/coding) in the EU, preferably Poland. Please email me at contact@sentimentscanner.com for more information for beta membership and so one.The single greatest appeal of Pen and Paper RPGs is human interaction, where obviously the DM has the most important role.
We have a solution for playing RPGs when you have no DM. It is called a "Video game", you can even play it with your friends.
What I have used LLMs for is helping me prepare for certain things, giving ideas for organizations, quests, characters, etc. . They are pretty good at that, although I would avoid relying on them during actual gameplay.
This is a student thesis, which doesn't mean it's bad, but is worth keeping in mind. I see that their control group size was three people (in three different sessions), but I don't see how many people they ran through the AI part of the experiment.
I was wondering the other day whether there were good services for running tabletop games over AI. I don't know about good, because I haven't tried them, but there are certainly a few products like this today. Given that ChatGPT plus some fine-tuning was comparable to a human GM in this experiment, I'd be interested in another experiment that looked at whether these products (which charge more money) offer any benefit over just using ChatGPT. For them to be better than ChatGPT, it seems like they'd almost have to be better than a human GM to be worth the money.
A while ago I was playing with chatgpt and had it create me a text based game. It took some coaxing but eventually it actually created a pretty compelling beginning to a game at least. This wasn't D&D, I had chatgpt just make the rules but the interesting part was creating a compelling story. Maybe it would have fallen apart after a little bit of game play but it was neat. I could see the fun of having completely unique and instantly created modules to run through.
I made something like this to let people explore the lore of my game. It has an AI act like a dungeon master as you play through a story. You can try it for free at: https://orbsccg.com/adventures
The main game is a strategic collectible card game, but the “adventure mode” is just for fun AI-powered stories.
Yay, can't wait to read this. I spent a lot of time tuning prompts for gpt4 DM sessions, and I had a lot of fun. It has SO much potential, and I was pleasantly surprised!
I am not really considering mainstream commercialization, but I just found it really fun to interact with. We just need a customized model full of fantasy text and a little DM RLHF. It's essentially just a story-telling llm, which obviously will need some work before consistent narratives can be delivered to the user.
Not an AI expert or anything, just wanted to share my excitement about this. I urge you to try it out, if you're into this kind of thing. Tell ChaptGPT it's a DM, explain the rules, give it some context on how deliver the narrative and player choices. You can give some story or character exposition to give it a baseline. Its not too bad.
This is fascinating.
I don't have any experience with D&D but wanted to learn it to play it with my kids. I used claude to learn the basic rules and then had it help me create the scenario. Then I played as the dm while it simulated being the players and then vice versa.
Really nice experience
My experience with D&D and LLM is that is a cheat sheet and a great helper, specially for newbies like me, I'm a Dad of 4 that plays DnD to teach some lessons and to get my kids out of the screens
A good use for this is people wanting to try an RPG, but not having a bunch of people to do that with. A parent wanting to try out this D&D thing so he or she can run it with the kids, for example.
Not having a bunch of people to do that with is solved by going out and creating relationships and communities. This is the basis of democracy, of society.
It's a tremendous loss for our species if we lose the impetus to connect. Do your kids a REAL favor, and build them a community—a SOCIETY—that they can join.
I found it to be a pretty OK first draft character creator, scene described and occasional lore/world builder. Not really anything more. Kinda a DM assistant more than a DM.
I’ve used an llm to help my niece, who is a performer, and has a flair for story, but no rpg experience dm as a human - llm team up. It was very successful
LLDM (Large Language Dungeon Master)
I try to variance of this where I was the DM and the llm played the player characters. It went okay but it was also a bit lame.
An ideal entry role for LLM in tabletop may be NPC companions/followers in the party that players/DM don't care for roleplaying or find tedious. Set it to listen in on conversations or VTT chat, and interject dialog on a rate-limited basis. Could lead to unusual situational ideas or at least humorous moments.
Yeah, completely automatic DMing just makes it like an open-ended computer game. On the other hand, the AI generated NPC responses are better than I could have come up with quickly, which makes me think co-DMing with an AI partner might be fun, particularly if they get better at remembering the details. That would let the human DM control the flow of the story without worrying that Thug #4 has taken 3hp of damage and is Dazzled. Of course, I most often use Foundry or Roll20 these days and they do a lot of that for you.
I wonder how well it could do for game summaries and recaps?
Like others, I don't really understand why anyone would desire this.
D&D is a social activity, why would I want to remove the most core human component of a social activity. It'd be like someone who enjoys going to a sports bar with friends deciding they'd rather sit next to a vending machine.
Do you take issue with Baldur's Gate 3? Video games in general? Why not?
Baldur's Gate 3 (and other games of its nature) are narrative experiences crafted by writers to convey a story. I don't play BG3 when I want to play D&D, and when I do want to play something like BG3 I certainly wouldn't be interested in a machine generated amalgamation of other people's ideas with no thought or emotion behind it - that would be fundamentally pointless and a waste of my time - like reading Lorem Ipsum instead of a novel.
[dead]
So speaking of sex dolls...