Storylet DSL nerdery

[JoshGrams]

Splitting from the MouseQBN topic to avoid derailing it too far…

@videlais

One of the things I have been researching is if it was possible to create a micro-language, like how TinyQBN handles part selection, and pair this with a C++ library for usage with JavaScript, C#, and other languages. In other words, if it is possible to solve that part of the problem space fairly well for most use cases and then create a library that could be used in either Node.js, Unity, or Unreal projects based on a plugin to supply object types and expression parsing.

For parsing simple expressions like this I think the shunting-yard algorithm hits a sweet spot of easy-to-implement but pretty powerful and flexible. Hand-rolling a recursive-descent parser isn’t bad either, but you have to worry about backtracking the parse, and avoiding left-recursion, and you’re hardcoding the precedence levels into your grammar, so it feels a little more finicky to me.

For lexing I usually compile down to a single regular expression with one capturing group per type of token: Moo is a good example of this. It has some error checking and a few bells and whistles but its source code is still only about 650 lines, so it’s not too hard to find the relevant bits? But let me know if you’d like a more minimal example of this approach: I could throw something together.

@videlais

SimpleQBN uses a NoSQL-variant, and MouseQBN uses a per-property encoding where, at least as I can tell, the variables, operators, and values are each a separate property.

I have two favorite data structures for compiling expressions for later evaluation:

• S-expressions with arrays, e.g. ["<", "$fuel", ["*", "$distance", "$milesPerGallon"]]
• Some form of RPN “bytecode” for a stack-based virtual machine a lá Forth, e.g. "$fuel $distance $milesPerGallon * <".split(" ")

Though of course there are further questions, like “do you want to restrict the kinds of allowable conditions to make it easier to graph or analyze the story volume?” and “if you’re restricting them, do you do it in the parser or in the generated data structure?”

@videlais

How availability or openness is determined and when. There are three different ways to do this: after a data operation happens (such as a value change in a global store), after a story event (such as passage transition), or when the author/program asks for it (using a macro or function to ask for all or a limit).

I tend to do it only on request: it’s easy to attach to story events as needed, and it’s not too hard to optimize as necessary by indexing your lookups on common fields. Doing it after a data operation allows some cool things, but a naïve implementation can get slow because data operations are often more common than storylet selection (I think?), and implementing a data/condition dependency graph is extremely cool but not really worth it unless you’re building a whole tool/language around it IMO…

@videlais

TinyQBN uses passage tags, saving the information as part of the internal passage object in SugarCube. As you mentioned, MouseQBN uses an array of objects, and my own SimpleQBN does the same thing for the same reasons, separating the storylet data from passage or other data.

TinyQBN uses passage tags as the initial state: SugarCube really wants passages to be static so it keeps the variable data in an array or dictionary, though I don’t remember how much it exposes that.

As for “separating the storylet data from passage or other data,” one thing I’d really like to see in a dedicated storylet tool is for it to let you view and edit the storylet data both with the text of the individual storylet and as a collection of the metadata without content for a (possibly-dynamically-filtered) collection of storylets.

[videlais (Dan Cox)]

@JoshGrams

For lexing I usually compile down to a single regular expression with one capturing group per type of token: Moo is a good example of this. It has some error checking and a few bells and whistles but its source code is still only about 650 lines, so it’s not too hard to find the relevant bits? But let me know if you’d like a more minimal example of this approach: I could throw something together.

Nice. I have been looking at peggy.js for a similar purpose but haven’t quite committed to a particular project yet.

Ideally, at least for the purpose I am envisioning, the DSL would be as simple as possible. It would also be close to MongoDB in approach, using JSON or easily encoded into JSON for easier serialization. I really like the object-based approach MouseQBN uses for this reason.

@JoshGrams

Though of course there are further questions, like “do you want to restrict the kinds of allowable conditions to make it easier to graph or analyze the story volume?” and “if you’re restricting them, do you do it in the parser or in the generated data structure?”

Yeah. This gets into the problem space of story sifting, which is an entire thing upon itself.

Max Kreminski has done some great work on this problem and developed Felt. Unfortunately, Felt is heavily tied to DataScript, which is itself tied to Clojure.

I’d love to spend time to figure out a general purpose DSL for story sifting/searching for storylets, but probably not any time soon.

@JoshGrams

As for “separating the storylet data from passage or other data,” one thing I’d really like to see in a dedicated storylet tool is for it to let you view and edit the storylet data both with the text of the individual storylet and as a collection of the metadata without content for a (possibly-dynamically-filtered) collection of storylets.

I agree, but part of this issue is the yet-another-authoring-tool problem, yeah? Another specifically aimed at storylets support would be great, but without a dedicated community, it doesn’t survive. (It’s telling, on the other hand, that Harlowe has support for storylets, it is planned/being strongly considered for SugarCube, and exists in Snowman 3.0, even if the story format version itself hasn’t been released yet.)

My thinking has been getting storylet support into tools like Unity and/or Unreal as a next step. That was my thinking with a C++ library, setting up a plugin with data types. I’ve got some dedicated research time coming up, I might look into it more and see what comes of it.

[Hituro (David Donachie)]

@JoshGrams

I tend to do it only on request: it’s easy to attach to story events as needed, and it’s not too hard to optimize as necessary by indexing your lookups on common fields. Doing it after a data operation allows some cool things, but a naïve implementation can get slow because data operations are often more common than storylet selection (I think?), and implementing a data/condition dependency graph is extremely cool but not really worth it unless you’re building a whole tool/language around it IMO…

There’s also so many possible events that you’d have to watch for. It’s not just the larger bits of state change, but every variable in the game would have to be something you watch for changes, down to internal object keys and array values. Unless the system you working with (assuming you don’t have your own containing software) provides events for all these things, its going to be essentially impossible. For example snowman has an event that fires when a state variable changes, but none of the other Twine formats do.

@JoshGrams

TinyQBN uses passage tags as the initial state: SugarCube really wants passages to be static so it keeps the variable data in an array or dictionary, though I don’t remember how much it exposes that.

You can interrogate it with the Story.loopkup() and Story.lookupWith() methods, but not mutate it. So there’s no way, for example, to take a storylet out of the pool if its existence is entirely based on passage data.

What I’m doing with <<storyletscan>> is “fire and forget”. It scans the passages for storylet definitions, shoves them onto the end of the relevant store, and then never looks back at the passages again. @TheMadExile says that passages scans are an expensive operation to be avoided where not needed, and so I assume that the <<storyletscan>> is something to do only once (if you do it at all).

@videlais

Ideally, at least for the purpose I am envisioning, the DSL would be as simple as possible. It would also be close to MongoDB in approach, using JSON or easily encoded into JSON for easier serialization. I really like the object-based approach MouseQBN uses for this reason.

I definitely had the concept of storing your storylets in an external JSON file in mind. Also, I didn’t want to spend all my time on parsing conditions If you don’t mind being tightly bound to SugarCube, there’s no huge reason that conditions couldn’t look like Scripting.evalTwineScript("$variable = 'value'") in the same way as the conditions for the <<if>> macro work. I don’t have a 100% easy to explain reasoning for why I didn’t do that, other than a general hesitation about sticking twinescript into JS objects where it didn’t really belong (and a hesitation over which values of temporary variables to use).

[TheMadExile]

EDIT: I suppose this is technically off-topic for this thread, since I have no DSL nerdery to offer. Mea culpa.

---

I have been invoked!

Storylets coming natively to SugarCube are largely a product of urging by @Chapel . He’s also informed the design, along with a few other interested parties.

They weren’t on my radar for inclusion until Chapel suggested them, because IMO they’re simply an event system with a fruity nomenclature.

The design isn’t entirely settled yet, so anything I mention which is specific to native SugarCube storylets is subject to change.

 

@Hituro

What I’m doing with <<storyletscan>> is “fire and forget”. It scans the passages for storylet definitions, shoves them onto the end of the relevant store, and then never looks back at the passages again.

Assuming that I decide to pull the trigger on the in-passage storylet definitions feature, that’s how SugarCube’s implementation will work, though it does come with some requirements. The current idea is that such passages:

• Are processed only once at startup.
• Must be appropriately tagged. I’m not entirely sure what with though. A storylet tag seems like a natural fit, but it’s in consideration to denote external definition passages (see below).
• Must use the <<storylet>> macro, at the top of the passage, to define the storylet’s details.
• Must not attempt to specify the associated content passage—because that’s the current passage.
• Must contain the actual text content of the storylet—empty storylets are disallowed.

Storylets are definitely going to be able to be defined external to their content passages, within special passages—e.g., by special name (StoryInit) or special tag (storylet). It’s a bit more versatile, as you can use either the macro or the base API to define storylets, and has better separation of concerns, so you’re not repeating yourself.

 

@Hituro

@TheMadExile says that passages scans are an expensive operation to be avoided where not needed, and so I assume that the <<storyletscan>> is something to do only once (if you do it at all).

Yes. Doing full text scans to find in-passage definitions, and parsing them, across the entire regular passage store is not to be done lightly. It’s something best done once during startup.

Further, needing to ignore the definitions themselves, whatever form they take, later during play is also suboptimal.

 

@Hituro

I definitely had the concept of storing your storylets in an external JSON file in mind. Also, I didn’t want to spend all my time on parsing conditions

SugarCube’s native storylets will handle this by using TwineScript conditionals that users are already familiar with—by request, it also allows functions. That said, it does pre-compile each conditional into a callable function for efficiency’s sake.

---

Here’s a really basic overview of the current design of the two definition macros. Again, subject to change.

Macro: <<storylet>>

Registers passages, listed by their names, as a storylet. You can define the storylet’s conditions, priority, and weight via its child tags.

<<storylet [passageList]>>
	[<<cond conditional>>]
	[<<func function>>]
	[<<groups groupList>>]
	[<<unique>>]
	[<<priority expression>>]
	[<<weight expression>>]
<</storylet>>

A storylet’s passage list associates the definition with the given passages.

A storylet’s conditions—defined via its <<cond>>, <<func>>, <<groups>>, or <<unique>> children—are tested whenever storylets are requested to determine if it is available to be chosen. All of its conditions must yield a truthy result for it to be considered available.

A storylet’s priority—defined via its <<priority>> child—is its relative importance compared to other storylets. When checking availability, only storylets with a priority equal to the highest priority seen during that check are yielded.

A storylet’s weight—defined via its <<weight>> child—is its chance of being randomly selected. When calling for a random storylet, a higher weight increases a storylet’s chance to be chosen.

Macro: <<condgroup>>

Condition groups are named lists of conditions that can be added to any storylet by referencing the group’s name. When referenced, a group’s conditions are added to the storylet when checking for availability as though they were its own.

The special group name :all automatically applies to all storylets.

<<condgroup groupName>>
	[<<cond conditional>>]
	[<<func function>>]
	[<<unique>>]
<</condgroup>>

[Hituro (David Donachie)]

@TheMadExile

Further, needing to ignore the definitions themselves, whatever form they take, later during play is also suboptimal.

What I’ve done is make <<storylet>> a macro that returns nothing. But it will still generate a line-break after itself, which is a pain.

And of course, I’ve noted that MQBN is using the exact macro names that you are proposing (storylet, storyletlink, etc.). I did consider pre-emptively naming mine with an “m” prefix to avoid future collision, but I’m assuming that any game using MQBN for storylets is unlikely to be upgraded to a version of SugarCube that uses the same macros, and that once the native macros are in place I’d get more mileage out of extending it than trying to have two systems going at once.

If I’m wrong about the later (i.e. if I prefer my own to the built-in one, or someone else does) it would easy to produce an MQBN2 with prefixed macros … or to agitate for a Config.storyletsEnabled flag to turn off the native implementation entirely.

@TheMadExile

A storylet’s priority—defined via its <<priority>> child—is its relative importance compared to other storylets. When checking availability, only storylets with a priority equal to the highest priority seen during that check are yielded.

One suggestion here, because I ran into the same issue with MQBN, if the user asks for X storylets, and X storylets with the highest available priority don’t exist, I will fill out the returned hand with the next priority down. It sounds like you are thinking about not doing so. It might be worth considering the approach I’m doing, or even having that be something user-definable.

@TheMadExile

Condition groups are named lists of conditions that can be added to any storylet by referencing the group’s name. When referenced, a group’s conditions are added to the storylet when checking for availability as though they were its own.

I think the cond-groups are very clever, allowing you to avoid repeating conditions. I might add them to MQBN if I can think of a syntax I am happy with. With only one universal store of storylets in your version, it’s much harder to switch to some completely different set of storylets at some point in the game, and the cond-groups definitely help with that.

[JoshGrams]

@videlais

I agree, but part of this issue is the yet-another-authoring-tool problem, yeah? Another specifically aimed at storylets support would be great, but without a dedicated community, it doesn’t survive.

I still feel that backwards compatibility would solve most of that. Look how many people have moved to VSCode/Tweego/Twee3LanguageTools, despite the fairly significant setup barrier. And a custom editor could coordinate with a SugarCube storylets extension, so it could add visual editor support for storylets and could pre-parse and separately compile the metadata to avoid the issues of leaving that text in the actual passages. It wouldn’t be as great as a full-custom tool (there are quite a few fundamental things that I dislike about all of the big three Twine story formats), but it could get pretty close in a lot of respects.

If I were to announce, for instance:

Lexiqon is a new Twine-compatible editor: use it in your browser, or install for Win/Mac/Linux/Android/iOS, import and work on your existing Twine stories. Includes spell checking, full-featured search, syntax highlighting for Harlow and SugarCube and Chapbook to help find errors, dynamically grouping passages for more convenient editing, a visual storylet editor/extension for SugarCube, and features to save your changes and share them with collaborators. Join my Patreon at $1/month for the ability to invite all your friends to Google Docs-style collaborative online editing on your projects. Native versions can save to files at a location of your choice. Screenreader and mobile friendly. (**This is not real, sorry**)

Show of hands, how many people would switch to that, like, yesterday? How many more would at least give it a try?

[Hituro (David Donachie)]

@JoshGrams

Show of hands, how many people would switch to that, like, yesterday? How many more would at least give it a try?

You got my vote.

[TheMadExile]

@Hituro

One suggestion here, because I ran into the same issue with MQBN, if the user asks for X storylets, and X storylets with the highest available priority don’t exist, I will fill out the returned hand with the next priority down. It sounds like you are thinking about not doing so.

There are no multi-pull methods in the API at present, so I’m unsure where you got the idea that I favor one approach over another. It’s possible that I have said something against selecting from multiple priorities somewhere, but I don’t recall doing so.

There is an open issue in its repository (see below) to add a pair of multi-pull methods that, while scant on details, does show the intent to allow just that.

As a final thought. Regardless of whether you strictly obey priorities or not, users have to be open to receiving fewer entries than requested, because there simply may not be enough available results based on criteria alone.

---

Add static methods to yield multiple random storylets

Add new multiple random storylet methods Storylet.randomMany() & Storylet.randomManyStorylets().

/**
 * Return random storylet passage names selected by priority (descending).
 *
 * @param count {integer} - How many passage names to yield, subject to availability.
 * @param onlyHighestPriority {boolean} - If `true`, select only from the highest priority.
 * @returns {Array<string>} An array of storylet passage names or an empty array.
 */
Storylet.randomMany(count, onlyHighestPriority)

/**
 * Return random storylets selected by priority (descending).
 *
 * @param count {integer} - How many storylets to yield, subject to availability.
 * @param onlyHighestPriority {boolean} - If `true`, select only from the highest priority.
 * @returns {Array<StoryletDefinition>} An array of storylets or an empty array.
 */
Storylet.randomManyStorylets(count, onlyHighestPriority)

[Hituro (David Donachie)]

@TheMadExile

so I’m unsure where you got the idea that I favor one approach over another

From the phrase “When checking availability, only storylets with a priority equal to the highest priority seen during that check are yielded”, which rather seemed to suggest that you were only going to pull from the highest priority.

The signatures you posted look exactly like what I was suggesting, though, with the user in control of whether they get some lower priority storylets or not.

[TheMadExile]

@Hituro

From the phrase “When checking availability, only storylets with a priority equal to the highest priority seen during that check are yielded”, which rather seemed to suggest that you were only going to pull from the highest priority.

For singular pulls, yes.

No, that wasn’t explicitly spelled out, but nowhere within that capsule description is mention made of doing multiple pulls, while single pulls are.

/shrug

[videlais (Dan Cox)]

@JoshGrams

I still feel that backwards compatibility would solve most of that.

I’m slightly confused. Backward compatibility? Of Twine HTML? Twee?

@JoshGrams

And a custom editor could coordinate with a SugarCube storylets extension, so it could add visual editor support for storylets and could pre-parse and separately compile the metadata to avoid the issues of leaving that text in the actual passages.

I don’t disagree, custom editor options could be amazing in many ways. The central issue, as I understand Twine’s current architecture, is that story formats carry all their CodeMirror functionality with them. This means story formats like Harlowe and Chapbook must package all the CodeMirror menus, syntax highlighting options, and other data in the format.js file. This is loaded by Twine, but also means it is carried over into the published HTML for authors. This increases the build size of all story formats wanting to add passage editing options.

@JoshGrams

Lexiqon is a new Twine-compatible editor…

I mean, the best way to get some of these things into Twine itself is to set a bounty on them. The IFTF could set a financial bounty on certain things or pay someone for weeks or months of work.

Alternatively, on the academia side, undergraduate or graduate students could be paid to work on these things.

We can joke, but, like, trying to make money within the Twine community is not easy.

[Greyelf]

@videlais

…but also means it is carried over into the published HTML for authors. This increases the build size of all story formats wanting to add passage editing options.

The JSON like data in the format.js file contains both:

• a “source” property, that contains the story format’s “template”, which is used to generate the published HTML you mentioned.
• a “setup” property, that contains the “extension” functionality (what you called the CodeMirror functionality) used by the Twine 2.x application.

So unless the Twine & Twee compilers are deliberately appending the “extension” related content to the “template” related content, the build size of a generated Story HTML files should not affected.

And a simple search for ‘CodeMirror’ in the content of a published Harlowe 3.x based Story HTML file will return a “no results” outcome, even though that word can be found in the content of the “setup” property of the Harlowe 3.x format.js file.

[tundish]

@videlais

I’d love to spend time to figure out a general purpose DSL for story sifting/searching for storylets, but probably not any time soon

I’ve been lurking on this discussion, and haven’t chipped in so far since the original post was Twine-specific.

But now might be a useful moment to point to the work I’ve done in this area for my Python framework, Balladeer. There is a simple markup format for dialogue called SpeechMark. Dialogue can be generated programmatically or (relevant to this discussion) selected from a set of TOML files.

You might be reassured to know that both the syntaxes I’m linking to are framework/language agnostic.
So I’d welcome anyone using these resources as reference points or, better still, a basis for a cross-framework Storylet format.

Here’s more detail from the online documentation. Happy to answer any questions.

I don’t intend to be precious I’m quite sure the needs of the Twine community will differ from those I had in mind, so I’m fully prepared for that!

[tundish]

In support of the quest towards a language-agnostic storylet format, I thought I’d link to a new feature I finished coding up today. Documentation online: Balladeer 0.43.0 Conversation trees.

Yes, these are dialogue trees, not storylets or mini-missions. But the format (TOML and SpeechMark) is entirely generic and doesn’t favour any particular language or platform.

[videlais (Dan Cox)]

@videlais

I’d love to spend time to figure out a general purpose DSL for story sifting/searching for storylets, but probably not any time soon.

One of the things I have been exploring more recently is a minimum DSL for story sifting. I came up with replicating JavaScript’s conditional statements and keywords (along with some English shortcuts) in a project named Quis. It isn’t quite at a 1.0 stage yet, but the DSL part has solid testing by itself without an associated project to demonstrate its abilities.

Twine passage metadata

@JoshGrams

[Twine] editor support for storylets and could pre-parse and separately compile the metadata to avoid the issues of leaving that text in the actual passages.

Throughout work on the new Twine 2 JSON format, I was thinking about the metadata issue. Both Twee 3 and Twine 2 JSON formats support metadata properties as key-value pairs for passages. This means it is possible to add any DSL string values, and have a library or parser understand and act on them. (Both YarnSpinner and Dendry have the ability to add arbitrary fields/properties to story units, as two examples of implementations.)

The current issue is lack of direct access to passage metadata in Twine itself. Something I have floated to a couple of people more privately is adding a “Metadata” option to the editor toolbar when working in Twine. This would allow an author to add any key-value pairs they want, and potentially story formats could act on them to enable emergent narrative design patterns (like storylets, microstories, and story sifting approaches).

While I’d like to tackle this myself, I anticipate needing to focus on a new job in early 2024. That written, here are the steps as I see them to get this done:

• Adding storage object and functionality to internal Twine data for keeping track of each passage’s metadata properties.

• Add “Metadata” menu option to passage editing toolbar with associated UI functionality for adding, changing, and removing key-value pairs.

• Updating Twine 2 HTML specification for encoding key-values pairs within the <tw-passagedata> element.

Potentially, this could take different forms. Two approaches I have thought about include the following:

(A) Encoding metadata as JSON string with metadata attribute per passage:

<tw-passagedata
  pid="1"
  name="Start"
  tags="tag1 tag2"
  position="102,99"
  size="100,100"
  metadata="{'name':'John', 'age':30, 'car':null}">
Some content
</tw-passagedata>

(B) Following pattern of <tw-tag> based on HTML <data> element with using name as the name of the passage:

<tw-metadata name="Start">
  <data value="John">name</data>
  <data value="30">age</data>
  <data value="null">car</data>
</tw-metadata>

[Greyelf]

@videlais

(A) Encoding metadata as JSON string with metadata attribute per passage:

From the Other notes section of the MDN Working with JSON page.

JSON requires double quotes to be used around strings and property names. Single quotes are not valid other than surrounding the entire JSON string.

So you might want to correct your example.

[Hituro (David Donachie)]

Wouldn’t

<tw-passagedata
  pid="1"
  name="Start"
  tags="tag1 tag2"
  position="102,99"
  size="100,100"
  name="John"
  age="30"
  car="null">
Some content
</tw-passagedata>

Be more idiomatically correct?

[videlais (Dan Cox)]

@Greyelf

So you might want to correct your example

I’m fine with being wrong on the Internet. Hopefully, my error helps others to find a solution or approach I couldn’t see.

---

@Hituro

Be more idiomatically correct?

Additional attributes is another good solution. To avoid causing problems for HTML5 validation services, and accidentally repeating existing attributes, I could see a data- prefix being used as a variation of what you propose:

<tw-passagedata
 pid="1"
 name="Start"
 tags="tag1 tag2"
 position="102,99"
 size="100,100"
 data-name="John"
 data-age="30"
 data-car="null">
Some content
</tw-passagedata>

[Hituro (David Donachie)]

The use or non-use of data- prefixes in that situation is interesting. They are clearly correct for custom attributes of well-defined HTML tags (i.e. those where the browser already knows which attributes are normal), but <tw-passagedata> is not such a tag. It’s not defined as a custom HTML tag using customElements.define() for example. In that sense, no attribute is more or less “custom” than any other, so age is as valid as name.

On the other hand, there is still a specification for that element as you linked before (in the Twine 2 HTML Output format specification). However that spec just states that the element has: “its metadata stored as its attributes”, which suggests that any metadata should be encoded that way directly. The Twee 3 spec is as close as we have for an official list of such metadata, and it in turn only says that: “the currently supported properties include […] position [and] size”.

My reading, then, is that additional metadata should appear in Twee as:

PassageName [optional tags] { metadata }

And that a translation of that to HTML should encode each key of metadata as an HTML attribute.

So if your metadata is { size: "100,100", position: "100,100", age: 30 } then it should be represented as

<tw-passagedata name="PassageName" position="100,100" size="100,100" age="30">

But if your metadata is { size: "100,100", position: "100,100", character: { name: "John", age: 30 }} then it might be appropriate do to

<tw-passagedata name="PassageName" position="100,100" size="100,100" character="{ name: "John", age: 30 }">

Just my interpretation though

[videlais (Dan Cox)]

@Hituro

My reading, then, is that additional metadata should appear in Twee as:

PassageName [optional tags] { metadata }

Exactly. For example, creating an example story in Twine and exporting as Twee encodes the JSON metadata (position and size) as you mentioned:

:: Untitled Passage {"position":"700,325","size":"100,100"}

@Hituro

And that a translation of that to HTML should encode each key of metadata as an HTML attribute.

Technically, yes, but if we do that approach we run into problems. The central issue are the existing HTML attributes. For example, if a reader were to use the name-value pair of name: Test, the element might accidentally have two name attributes. The same, too, with aria labels, which might confuse screen readers and other accessibility tools.

I do agree with you that we didn’t clarify the language around metadata in passages, though. That’s an area where, as story formats slowly push at the edges of what Twine can do, will need to be revisited at some point.