Storylet DSL nerdery

[JoshGrams]

Splitting from the MouseQBN topic to avoid derailing it too far…

@videlais

One of the things I have been researching is if it was possible to create a micro-language, like how TinyQBN handles part selection, and pair this with a C++ library for usage with JavaScript, C#, and other languages. In other words, if it is possible to solve that part of the problem space fairly well for most use cases and then create a library that could be used in either Node.js, Unity, or Unreal projects based on a plugin to supply object types and expression parsing.

For parsing simple expressions like this I think the shunting-yard algorithm hits a sweet spot of easy-to-implement but pretty powerful and flexible. Hand-rolling a recursive-descent parser isn’t bad either, but you have to worry about backtracking the parse, and avoiding left-recursion, and you’re hardcoding the precedence levels into your grammar, so it feels a little more finicky to me.

For lexing I usually compile down to a single regular expression with one capturing group per type of token: Moo is a good example of this. It has some error checking and a few bells and whistles but its source code is still only about 650 lines, so it’s not too hard to find the relevant bits? But let me know if you’d like a more minimal example of this approach: I could throw something together.

@videlais

SimpleQBN uses a NoSQL-variant, and MouseQBN uses a per-property encoding where, at least as I can tell, the variables, operators, and values are each a separate property.

I have two favorite data structures for compiling expressions for later evaluation:

• S-expressions with arrays, e.g. ["<", "$fuel", ["*", "$distance", "$milesPerGallon"]]
• Some form of RPN “bytecode” for a stack-based virtual machine a lá Forth, e.g. "$fuel $distance $milesPerGallon * <".split(" ")

Though of course there are further questions, like “do you want to restrict the kinds of allowable conditions to make it easier to graph or analyze the story volume?” and “if you’re restricting them, do you do it in the parser or in the generated data structure?”

@videlais

How availability or openness is determined and when. There are three different ways to do this: after a data operation happens (such as a value change in a global store), after a story event (such as passage transition), or when the author/program asks for it (using a macro or function to ask for all or a limit).

I tend to do it only on request: it’s easy to attach to story events as needed, and it’s not too hard to optimize as necessary by indexing your lookups on common fields. Doing it after a data operation allows some cool things, but a naïve implementation can get slow because data operations are often more common than storylet selection (I think?), and implementing a data/condition dependency graph is extremely cool but not really worth it unless you’re building a whole tool/language around it IMO…

@videlais

TinyQBN uses passage tags, saving the information as part of the internal passage object in SugarCube. As you mentioned, MouseQBN uses an array of objects, and my own SimpleQBN does the same thing for the same reasons, separating the storylet data from passage or other data.

TinyQBN uses passage tags as the initial state: SugarCube really wants passages to be static so it keeps the variable data in an array or dictionary, though I don’t remember how much it exposes that.

As for “separating the storylet data from passage or other data,” one thing I’d really like to see in a dedicated storylet tool is for it to let you view and edit the storylet data both with the text of the individual storylet and as a collection of the metadata without content for a (possibly-dynamically-filtered) collection of storylets.

[videlais (Dan Cox)]

@JoshGrams

For lexing I usually compile down to a single regular expression with one capturing group per type of token: Moo is a good example of this. It has some error checking and a few bells and whistles but its source code is still only about 650 lines, so it’s not too hard to find the relevant bits? But let me know if you’d like a more minimal example of this approach: I could throw something together.

Nice. I have been looking at peggy.js for a similar purpose but haven’t quite committed to a particular project yet.

Ideally, at least for the purpose I am envisioning, the DSL would be as simple as possible. It would also be close to MongoDB in approach, using JSON or easily encoded into JSON for easier serialization. I really like the object-based approach MouseQBN uses for this reason.

@JoshGrams

Though of course there are further questions, like “do you want to restrict the kinds of allowable conditions to make it easier to graph or analyze the story volume?” and “if you’re restricting them, do you do it in the parser or in the generated data structure?”

Yeah. This gets into the problem space of story sifting, which is an entire thing upon itself.

Max Kreminski has done some great work on this problem and developed Felt. Unfortunately, Felt is heavily tied to DataScript, which is itself tied to Clojure.

I’d love to spend time to figure out a general purpose DSL for story sifting/searching for storylets, but probably not any time soon.

@JoshGrams

As for “separating the storylet data from passage or other data,” one thing I’d really like to see in a dedicated storylet tool is for it to let you view and edit the storylet data both with the text of the individual storylet and as a collection of the metadata without content for a (possibly-dynamically-filtered) collection of storylets.

I agree, but part of this issue is the yet-another-authoring-tool problem, yeah? Another specifically aimed at storylets support would be great, but without a dedicated community, it doesn’t survive. (It’s telling, on the other hand, that Harlowe has support for storylets, it is planned/being strongly considered for SugarCube, and exists in Snowman 3.0, even if the story format version itself hasn’t been released yet.)

My thinking has been getting storylet support into tools like Unity and/or Unreal as a next step. That was my thinking with a C++ library, setting up a plugin with data types. I’ve got some dedicated research time coming up, I might look into it more and see what comes of it.

[Hituro (David Donachie)]

@JoshGrams

I tend to do it only on request: it’s easy to attach to story events as needed, and it’s not too hard to optimize as necessary by indexing your lookups on common fields. Doing it after a data operation allows some cool things, but a naïve implementation can get slow because data operations are often more common than storylet selection (I think?), and implementing a data/condition dependency graph is extremely cool but not really worth it unless you’re building a whole tool/language around it IMO…

There’s also so many possible events that you’d have to watch for. It’s not just the larger bits of state change, but every variable in the game would have to be something you watch for changes, down to internal object keys and array values. Unless the system you working with (assuming you don’t have your own containing software) provides events for all these things, its going to be essentially impossible. For example snowman has an event that fires when a state variable changes, but none of the other Twine formats do.

@JoshGrams

TinyQBN uses passage tags as the initial state: SugarCube really wants passages to be static so it keeps the variable data in an array or dictionary, though I don’t remember how much it exposes that.

You can interrogate it with the Story.loopkup() and Story.lookupWith() methods, but not mutate it. So there’s no way, for example, to take a storylet out of the pool if its existence is entirely based on passage data.

What I’m doing with <<storyletscan>> is “fire and forget”. It scans the passages for storylet definitions, shoves them onto the end of the relevant store, and then never looks back at the passages again. @TheMadExile says that passages scans are an expensive operation to be avoided where not needed, and so I assume that the <<storyletscan>> is something to do only once (if you do it at all).

@videlais

Ideally, at least for the purpose I am envisioning, the DSL would be as simple as possible. It would also be close to MongoDB in approach, using JSON or easily encoded into JSON for easier serialization. I really like the object-based approach MouseQBN uses for this reason.

I definitely had the concept of storing your storylets in an external JSON file in mind. Also, I didn’t want to spend all my time on parsing conditions If you don’t mind being tightly bound to SugarCube, there’s no huge reason that conditions couldn’t look like Scripting.evalTwineScript("$variable = 'value'") in the same way as the conditions for the <<if>> macro work. I don’t have a 100% easy to explain reasoning for why I didn’t do that, other than a general hesitation about sticking twinescript into JS objects where it didn’t really belong (and a hesitation over which values of temporary variables to use).

[TheMadExile]

EDIT: I suppose this is technically off-topic for this thread, since I have no DSL nerdery to offer. Mea culpa.

---

I have been invoked!

Storylets coming natively to SugarCube are largely a product of urging by @Chapel . He’s also informed the design, along with a few other interested parties.

They weren’t on my radar for inclusion until Chapel suggested them, because IMO they’re simply an event system with a fruity nomenclature.

The design isn’t entirely settled yet, so anything I mention which is specific to native SugarCube storylets is subject to change.

 

@Hituro

What I’m doing with <<storyletscan>> is “fire and forget”. It scans the passages for storylet definitions, shoves them onto the end of the relevant store, and then never looks back at the passages again.

Assuming that I decide to pull the trigger on the in-passage storylet definitions feature, that’s how SugarCube’s implementation will work, though it does come with some requirements. The current idea is that such passages:

• Are processed only once at startup.
• Must be appropriately tagged. I’m not entirely sure what with though. A storylet tag seems like a natural fit, but it’s in consideration to denote external definition passages (see below).
• Must use the <<storylet>> macro, at the top of the passage, to define the storylet’s details.
• Must not attempt to specify the associated content passage—because that’s the current passage.
• Must contain the actual text content of the storylet—empty storylets are disallowed.

Storylets are definitely going to be able to be defined external to their content passages, within special passages—e.g., by special name (StoryInit) or special tag (storylet). It’s a bit more versatile, as you can use either the macro or the base API to define storylets, and has better separation of concerns, so you’re not repeating yourself.

 

@Hituro

@TheMadExile says that passages scans are an expensive operation to be avoided where not needed, and so I assume that the <<storyletscan>> is something to do only once (if you do it at all).

Yes. Doing full text scans to find in-passage definitions, and parsing them, across the entire regular passage store is not to be done lightly. It’s something best done once during startup.

Further, needing to ignore the definitions themselves, whatever form they take, later during play is also suboptimal.

 

@Hituro

I definitely had the concept of storing your storylets in an external JSON file in mind. Also, I didn’t want to spend all my time on parsing conditions

SugarCube’s native storylets will handle this by using TwineScript conditionals that users are already familiar with—by request, it also allows functions. That said, it does pre-compile each conditional into a callable function for efficiency’s sake.

---

Here’s a really basic overview of the current design of the two definition macros. Again, subject to change.

Macro: <<storylet>>

Registers passages, listed by their names, as a storylet. You can define the storylet’s conditions, priority, and weight via its child tags.

<<storylet [passageList]>>
	[<<cond conditional>>]
	[<<func function>>]
	[<<groups groupList>>]
	[<<unique>>]
	[<<priority expression>>]
	[<<weight expression>>]
<</storylet>>

A storylet’s passage list associates the definition with the given passages.

A storylet’s conditions—defined via its <<cond>>, <<func>>, <<groups>>, or <<unique>> children—are tested whenever storylets are requested to determine if it is available to be chosen. All of its conditions must yield a truthy result for it to be considered available.

A storylet’s priority—defined via its <<priority>> child—is its relative importance compared to other storylets. When checking availability, only storylets with a priority equal to the highest priority seen during that check are yielded.

A storylet’s weight—defined via its <<weight>> child—is its chance of being randomly selected. When calling for a random storylet, a higher weight increases a storylet’s chance to be chosen.

Macro: <<condgroup>>

Condition groups are named lists of conditions that can be added to any storylet by referencing the group’s name. When referenced, a group’s conditions are added to the storylet when checking for availability as though they were its own.

The special group name :all automatically applies to all storylets.

<<condgroup groupName>>
	[<<cond conditional>>]
	[<<func function>>]
	[<<unique>>]
<</condgroup>>

[Hituro (David Donachie)]

@TheMadExile

Further, needing to ignore the definitions themselves, whatever form they take, later during play is also suboptimal.

What I’ve done is make <<storylet>> a macro that returns nothing. But it will still generate a line-break after itself, which is a pain.

And of course, I’ve noted that MQBN is using the exact macro names that you are proposing (storylet, storyletlink, etc.). I did consider pre-emptively naming mine with an “m” prefix to avoid future collision, but I’m assuming that any game using MQBN for storylets is unlikely to be upgraded to a version of SugarCube that uses the same macros, and that once the native macros are in place I’d get more mileage out of extending it than trying to have two systems going at once.

If I’m wrong about the later (i.e. if I prefer my own to the built-in one, or someone else does) it would easy to produce an MQBN2 with prefixed macros … or to agitate for a Config.storyletsEnabled flag to turn off the native implementation entirely.

@TheMadExile

A storylet’s priority—defined via its <<priority>> child—is its relative importance compared to other storylets. When checking availability, only storylets with a priority equal to the highest priority seen during that check are yielded.

One suggestion here, because I ran into the same issue with MQBN, if the user asks for X storylets, and X storylets with the highest available priority don’t exist, I will fill out the returned hand with the next priority down. It sounds like you are thinking about not doing so. It might be worth considering the approach I’m doing, or even having that be something user-definable.

@TheMadExile

Condition groups are named lists of conditions that can be added to any storylet by referencing the group’s name. When referenced, a group’s conditions are added to the storylet when checking for availability as though they were its own.

I think the cond-groups are very clever, allowing you to avoid repeating conditions. I might add them to MQBN if I can think of a syntax I am happy with. With only one universal store of storylets in your version, it’s much harder to switch to some completely different set of storylets at some point in the game, and the cond-groups definitely help with that.

[JoshGrams]

@videlais

I agree, but part of this issue is the yet-another-authoring-tool problem, yeah? Another specifically aimed at storylets support would be great, but without a dedicated community, it doesn’t survive.

I still feel that backwards compatibility would solve most of that. Look how many people have moved to VSCode/Tweego/Twee3LanguageTools, despite the fairly significant setup barrier. And a custom editor could coordinate with a SugarCube storylets extension, so it could add visual editor support for storylets and could pre-parse and separately compile the metadata to avoid the issues of leaving that text in the actual passages. It wouldn’t be as great as a full-custom tool (there are quite a few fundamental things that I dislike about all of the big three Twine story formats), but it could get pretty close in a lot of respects.

If I were to announce, for instance:

Lexiqon is a new Twine-compatible editor: use it in your browser, or install for Win/Mac/Linux/Android/iOS, import and work on your existing Twine stories. Includes spell checking, full-featured search, syntax highlighting for Harlow and SugarCube and Chapbook to help find errors, dynamically grouping passages for more convenient editing, a visual storylet editor/extension for SugarCube, and features to save your changes and share them with collaborators. Join my Patreon at $1/month for the ability to invite all your friends to Google Docs-style collaborative online editing on your projects. Native versions can save to files at a location of your choice. Screenreader and mobile friendly. (**This is not real, sorry**)

Show of hands, how many people would switch to that, like, yesterday? How many more would at least give it a try?

[Hituro (David Donachie)]

@JoshGrams

Show of hands, how many people would switch to that, like, yesterday? How many more would at least give it a try?

You got my vote.

[TheMadExile]

@Hituro

One suggestion here, because I ran into the same issue with MQBN, if the user asks for X storylets, and X storylets with the highest available priority don’t exist, I will fill out the returned hand with the next priority down. It sounds like you are thinking about not doing so.

There are no multi-pull methods in the API at present, so I’m unsure where you got the idea that I favor one approach over another. It’s possible that I have said something against selecting from multiple priorities somewhere, but I don’t recall doing so.

There is an open issue in its repository (see below) to add a pair of multi-pull methods that, while scant on details, does show the intent to allow just that.

As a final thought. Regardless of whether you strictly obey priorities or not, users have to be open to receiving fewer entries than requested, because there simply may not be enough available results based on criteria alone.

---

Add static methods to yield multiple random storylets

Add new multiple random storylet methods Storylet.randomMany() & Storylet.randomManyStorylets().

/**
 * Return random storylet passage names selected by priority (descending).
 *
 * @param count {integer} - How many passage names to yield, subject to availability.
 * @param onlyHighestPriority {boolean} - If `true`, select only from the highest priority.
 * @returns {Array<string>} An array of storylet passage names or an empty array.
 */
Storylet.randomMany(count, onlyHighestPriority)

/**
 * Return random storylets selected by priority (descending).
 *
 * @param count {integer} - How many storylets to yield, subject to availability.
 * @param onlyHighestPriority {boolean} - If `true`, select only from the highest priority.
 * @returns {Array<StoryletDefinition>} An array of storylets or an empty array.
 */
Storylet.randomManyStorylets(count, onlyHighestPriority)

[Hituro (David Donachie)]

@TheMadExile

so I’m unsure where you got the idea that I favor one approach over another

From the phrase “When checking availability, only storylets with a priority equal to the highest priority seen during that check are yielded”, which rather seemed to suggest that you were only going to pull from the highest priority.

The signatures you posted look exactly like what I was suggesting, though, with the user in control of whether they get some lower priority storylets or not.

[TheMadExile]

@Hituro

From the phrase “When checking availability, only storylets with a priority equal to the highest priority seen during that check are yielded”, which rather seemed to suggest that you were only going to pull from the highest priority.

For singular pulls, yes.

No, that wasn’t explicitly spelled out, but nowhere within that capsule description is mention made of doing multiple pulls, while single pulls are.

/shrug

[videlais (Dan Cox)]

@JoshGrams

I still feel that backwards compatibility would solve most of that.

I’m slightly confused. Backward compatibility? Of Twine HTML? Twee?

@JoshGrams

And a custom editor could coordinate with a SugarCube storylets extension, so it could add visual editor support for storylets and could pre-parse and separately compile the metadata to avoid the issues of leaving that text in the actual passages.

I don’t disagree, custom editor options could be amazing in many ways. The central issue, as I understand Twine’s current architecture, is that story formats carry all their CodeMirror functionality with them. This means story formats like Harlowe and Chapbook must package all the CodeMirror menus, syntax highlighting options, and other data in the format.js file. This is loaded by Twine, but also means it is carried over into the published HTML for authors. This increases the build size of all story formats wanting to add passage editing options.

@JoshGrams

Lexiqon is a new Twine-compatible editor…

I mean, the best way to get some of these things into Twine itself is to set a bounty on them. The IFTF could set a financial bounty on certain things or pay someone for weeks or months of work.

Alternatively, on the academia side, undergraduate or graduate students could be paid to work on these things.

We can joke, but, like, trying to make money within the Twine community is not easy.