Previewing lyre, a narrative scripting system for quality-based narratives

So for the past several months I’ve been sketching out, and eventually working on, a language and runtime meant to essentially implement StoryNexus-like semantics but in a more flexible way (where “flexible” means “you can run it in more different scenarios”). I’m building this mainly for my own use, but even so it’s useful to do so in the open and share what I’m doing, so this is the first I’m talking publicly about it; a proper public release will come down Soon (ie, this year).

I have a long blog post with proper details. But in short, lyre is…

  • Narrative middleware in the vein of Ink or Yarn, not necessarily a full script-to-game authoring system;
  • Designed to be easy to port to different environments, but currently exists as a compiler, a reference javascript runtime, and a GDScript runtime;
  • Designed for quality-based narratives: storylets as the central type of narrative node, all state and state changes can be made visible to the player, stories can grow organically (ie, you can add more material and make it accessible from existing material without having to explicitly change old material to hook it up; branches can be created from the place they point to);
  • Heavily influenced by StoryNexus, but designed with more elegant semantics and for a broader range of use cases;
  • Aimed at solving different problems from the ones Ink or Yarn are meant to solve
  • Very much an opinionated thing that exists to satisfy the specific shape of my brain, but I think those kinds of tools are often very interesting to people other than their creator
  • Yes, obviously, written without the use of LLMs, while that’s still legal

I really have no idea how useful this will ever be to people who are not me or how much interest there even is in a system like this. But I’m already using lyre to write something relatively substantial.[1] And I think getting some early impressions might be useful in helping shape the language going forward.

Here’s a little sample of what the language looks like; more in the blog post, along with more explanatory comments.

storylet anchor start "Forest Clearing"
    """
    You stand before a crossroads that splits to the left and right. A man
    stands next to the path on the right, wearing a large black velvet cloak.

    { 
    if scarf_color then 
        // You can even nest interpolated text inside other interpolated text.
        `Your { scarf_color } scarf is reassuringly warm.`
    else
        "You are wearing a warm scarf." }
    """
    branch "Examine your scarf"
        """
        What color was it again?
        """
        show ( scarf_color == "" )
        -> select_scarf_color
    
    branch "Talk to the man"
        """
        He looks *trustworthy.*
        """
        req ( social_anxiety < 7 )
        -> conversation

/* ... */

quality cloak_knowledge: boolean "Examined the Cloak"
    """
    You've taken a closer look at the Cloak of Darkness.
    """
    describe_change: () => "You add your observations to your mind palace."

  1. Though expect it to be much shorter than Kinophobia—which to be fair was a six hour game. ↩︎

Sounds fascinating!

I’m the author of a storylet package for Twine/SugarCube MQBN, so I’m always fascinated to see more approaches.

I was going to ask if every storylet must specify the storylets that are available from it, i.e. storylets are always defined top-down, but I see in the final example on the b.log post that from appears to support the opposite, i.e. allowing each storylet to define the game states in which it would be available, and then having the game engine gather those storylets to display.

I’m curious how from gets privileged over show though? What makes one restriction special over the other?

In MQBN you’d specify a storylet available in a room where the Doctor is present as:

all: [
    { type: "tag", tag: "room" },
    { type: "var", var: "$doctor_present", op: "eq", value: true }
]

Or something of that sort, depending on the world model in the game (or { type: "tws", cond: "tags().includes('room') && $doctor_present" } if you prefer to use the TwineScript conditions … as most people do). i.e. there’s just a list of conditions to satisfy that makes the storylet eligible to appear anywhere a pool of storylets is requested.

So here’s an example that’s a bit more straightforward, to explain the exact semantics:

storylet living_room
    branch "Talk to Harry"
        -> talk_to_harry
        show (harry_location = here) 
        req (harry_wants_to_talk ?)

storylet talk_to_harry
    from living_room "Talk to Harry"
        show (harry_wants_to_talk ?)
        req (harry_wants_to_talk ?)

The branch and from declarations here are semantically exactly equivalent.[1] from [somewhere] just specifies what storylet the branch is available from, not a general condition, but you can use a tag (or list of tags) rather than the name of a specific storylet to create a from branch that is available from multiple places; storylet tags are completely static, they’re just a way of defining a fixed group of storylets.

Whether the branch is visible and/or unlocked is down to its show/req conditions, like any branch. The simpler use case is just “I can add stuff to a particular place (storylet) without having to touch the definition of that place”. The more complex use case is “I can make this one storylet available from mulitiple places”. This could be used for example to make games with nesting areas (eg, in FL there are storylets that are available from one specific area, or from any neighborhood in London) or to build something like a topical conversation where you can change the subject at any one of several “hub” nodes.

from specifies strictly which storylets point to this branch while conditions can specify whether the branch appears at all or is available, basically.

Branches in lyre are their own freestanding entities and exist independently from storylets; storylets just point to them by name (and they in turn have “routes” that define how they lead into other storylets). So all of the from/branch hooking up happens at compile time; at runtime the engine looks at the current storylet’s list of branches and filters that down to the ones that are available.

Partly this is organizational (separating “where is this visible” from “under what conditions”) but partly it’s also about helping this scale to very large stories, because it means the runtime doesn’t have to check every of potentially thousands of branches to figure out which ones are available; the subset that could be available is statically determined.

There are various things in lyre that care about the graph of how storylets relate to one another through branches (eg, lyre implements a version of the “perhaps not” button), so this static graph of storylet → branch → storylet relationships is required, although the usual loop in lyre is storylet → branch → result → back to the anchor storylet through a “continue” button.


  1. The lyre compiler will assign either case here different internal names for the branches, and branches created through from will contain metadata about that, but as far as what the runtime cares about and how the game plays, they behave identically. ↩︎

Thanks for the reply, the idea of restricting the pool of storylets whose conditions have to be scanned is something that MQBN does by having multiple storylet stores, so that when you request a storylet (or storylets) you can also specify which store to look in, and thus skip evaluating the conditions of storylets in other stores.

The limitation of that approach is that you can’t query multiple stores at once (you’d have to say "give me all the storylets available in A, and then union that with all the storylets available in B). from sounds more flexible than that, because it allows the author to define all the storylets together and allow the compiler to segment them into pools.

Part of the difference in approach is because MQBN is integrated into Twine, so you can have passages that need a list of storylets, and ones that don’t (i.e. “normal” passages with normal links), so the author needs to actively request/show a list of storylet links when and where they want them rather than leaving it to the engine to decide. That also means that I can’t pre-assume what any given store might be for. One might be for random events, another might just have one NPC’s conversation options, and so on.

It is not obvious to me.

These days I assume heavy AI involvement unless otherwise specified.

Ooh, very neat! Something “like ink but for QBNs” is a thing I’ve been thinking about for a while, and JS and GDScript are exactly the two languages that I’d be most likely to use it in.

The current project I’m working on is going in a very QBN direction, and while I’ve finally figured out a storylet system in Ink that I’m happy with, I have been wondering about looking for something better at some point.

Some thoughts while reading the blog post:

  • The overall mental model seems good, probably?
  • I agree with fixing Ink lists by renaming them to not be called lists.
  • The syntax reads as a bit awkward to me, possibly because it’s a mix of verbose-but-readable-and-easy-to-type keywords on one hand, and concise-but-somewhat-strange uses of special characters on the other.
  • I acknowledge your claim that, because QBNs are more mechanical, there is value in explicitly separating text and code through string delimiters. But I feel that I would be annoyed by needing to use delimiters around every string while writing in a genre that by necessity involves writing a lot of little strings. The markdown-like feel of Ink is something that I really like and would look for in a QBN tool.
  • It seems to me that you want to prefer lots of smaller storylets over a smaller number of larger ones (which I like), but again, the syntax looks to me like it would make actually doing that a bit clunky.
  • I’ve been playing Sunless Sea a lot lately, and am now of the opinion that it is possible to go too far in surfacing mechanics to the player. Many storylets in the Fallen London/StoryNexus style have these awkward “Unlocked when your ‘progress in this particular storyline’ quality is 3 and no more than 3 (you have this)” messages that the player really shouldn’t see. And there are some visible-but-locked options that ride a very weird line between “informing the player about what might be possible in the future” and just straight up showing the player things that it makes no sense for them to know. Not really directly relevant to anything you’ve done, I guess, just something that came to mind when seeing you comment on the StoryNexus model.
  • Any examples of what the JS/GDScript API looks like?
  • Regarding your from, I feel like I’ve seen jokes in the past about how we could make spaghetti code even harder to follow by adding a COMEFROM statement to the already considered-harmful GOTO, so that seems like something to potentially worry about.

Also, there is one specific QBN feature that I’ve been thinking about for my current project: I want the ability to prioritize storylets based on their possible outcomes, e.g. some way of saying “at this point I would prefer a storylet that includes the possibility of earning money,” mainly for balancing reasons. It seems like the design of your system has features that could be made to support this (e.g. if all state changes can be surfaced to the player, then it should also be possible to surface them to the selection algorithm). But it’s not clear to me whether I could easily do something like this in the system as it exists currently?

The syntax is definitely a bit idiosyncratic; lyre uses syntactical ideas that come from Ink, Inform, Yarn, YAML, Javascript/Typescript, Haskell, Rust, Inform, Ruby, and Lua; it’s also informed by StoryNexus’ CMS UI, even though that’s not a language. It’s definitely a bit in flux—syntax is the easiest thing to change as it doesn’t touch the runtime—but the language is complex enough that I don’t think any syntax is going to be immediately obvious. -> could be go, maybe, and I think “effect literals” (using { } instead of the keyword effects) is potentially confusing. But a lot of the syntactical choices are driven by trying to make tokens that get used a lot in rapid succession generally short, and by trying to not overload special characters with too many different senses.

I did experiment with a more markdown-ish syntax (text with “islands of code”) but I discarded it for a few reasons:

  • I want lyre text to be able to contain markdown, html, or GDscript-style bbcode and it’s easier to avoid colliding with the special characters used by all of those if we don’t also have to deal with parsing a “story text is the default” language. I actually would like to leave open the possibility of a dialect that embeds Ink inside lyre storylet body text, so a storylet could contain a scene of dialogue rather than just text.
  • All text in lyre already has to exist inside a named storylet or quality; you can’t just start writing, it wouldn’t mean anything under lyre’s semantics. So the text is already inevitably going to be bracketed by the code somehow, similar to how a Yarn file always has front matter before any text. The """ in lyre is really analogous to the === in Yarn.
  • Lyre has to handle not just the context separation between text and code, but the context separation between declarative code, imperative code, and pure code which all exist as different language contexts. So I felt like if those were all “islands” the syntax was at risk of becoming total soup.
  • I value having some explicitness in what’s going on; storylet, branch, and result could be *, **, and *** but that would obscure that branches and storylets are not the same kind of fundamental entity, etc.

from is funny if you think of it as a comefrom analogue, but really it’s an analogue of the way StoryNexus storylets declare that they exist in a certain location and have certain requirements. Basically lyre replaces “locations” as a special-case concept with storylets that act as hubs that other storylets can arbitrarily attach to. Lyre also wants to have some kind of construct for writing a storylet that’s available from multiple places, and from is that construct.

The JS API looks something like:

// story.json is just the bundle compiled by the lyre compiler, you could also
// load it from a file, fetch it frmo the network, etc.
import story from "./story.json" with { type: "json" };
import LyreRunner from "lyre-js";

const runner = new LyreRunner(story);
let store = {} // Lyre expects the player's qualities to be stored in a hash

// Methods that advance the story return "frames", which are objects that contain
// all the information meant to be displayed to the player.
let frame = runner.begin("start");

// Every method on LyreRunner is a pure function from an outside perspective. This
// helper function takes the quality changes from the frame and the existing
// quality store, and returns a new store with updated state.
store = runner.incorporateChanges(frame.qualityChanges, store);

// (Code to display the frame, get a choice selection from the player)

frame = runner.selectBranch(branchId, frame, store);

The js API as it currently exists is fairly low level and completely stateless; this is so a specific game can have their own implementation for how they want to store player data, and the runtime can work on a server that can scale because it has no internal state (so the same runtime process can serve requests from different players, etc); the story data is taken as immutable. This is why selectBranch needs to have the current frame and quality store passed back to it.

It’s likely that I’ll eventually release an optional “wrapper API” that encapsulates all of the state and is easier to use for a typical use case.

Very interested to follow this! Ink and Inform are my preferred IF tools (or at least the ones I’m most familiar with), so a system that draws on elements of both, and is optimized for QBN, is definitely intriguing to me…

Obviously there are a lot of tradeoffs to be made with respect to the syntax, and it’ll depend on your personal preferences. I just think that maybe my priorities don’t quite align with yours. I mean, I’ve used HTML and bbcode in Ink, and while it’s mildly annoying to escape things at times, it’s a lot less annoying than having to put quotes around every string would be. Also, while I understand that some choices might make life harder for the person writing the parser/compiler, I don’t really care about that as the user.

As far as “what’s the easiest way for the parser to distinguish text from code” is concerned, whenever I think about how I would solve it, I keep coming back to “lines of code should begin with a semicolon.” The semicolon is very easy to type, and lines beginning with a semicolon are basically unheard of in normal text.

The stateless API looks very clean and pleasant to use, IMO! (EDIT: Deleted rest of previous comment, I misread something.)

The proposed syntax would look something like:

;storylet anchor start "Forest Clearing"
You stand before a crossroads that splits to the left and right. A man
stands next to the path on the right, wearing a large black velvet cloak.
{ 
if scarf_color then 
    // You can even nest interpolated text inside other interpolated text.
    `Your { scarf_color } scarf is reassuringly warm.`
else
    "You are wearing a warm scarf." }
;branch "Examine your scarf"
    What color was it again?
    ;show ( scarf_color == "" )
    ;-> select_scarf_color

;branch "Talk to the man"
    He looks *trustworthy.*
    ;req ( social_anxiety < 7 )
    ;-> conversation

;quality cloak_knowledge: boolean "Examined the Cloak"
You've taken a closer look at the Cloak of Darkness.
;describe_change: () => "You add your observations to your mind palace."

obviously this is a matter of taste but I think aesthetically this is not great; there’s a lot of visual noise. it would also imply that you can write things like…

;storylet pet dog
You've pet { dogs_petted } dogs today, which is far too few! You lean down and
give { dog_name } a pet.
; dogs_petted++
{ dog_name } barks happily. Now you've pet { dogs_petted } dogs!

This seems like it should output:

You’ve pet 3 dogs today, which is far too few! You lean down and give Spot a pet.

Spot barks happily. Now you’ve pet 4 dogs!

But, with lyre’s semantics, it really outputs:

You’ve pet 4 dogs today, which is far too few! You lean down and give Spot a pet.
Spot barks happily. Now you’ve pet 4 dogs!

Lyre is generally functional-declarative and doesn’t interpret the contents of a storylet top to bottom as a series of statements that are run in order, so the ability to insert statements in the middle of body text is only ever confusing. That the storylet body is locked inside a “text box” of sorts matches up to what actually is going on under the hood. In Ink and Yarn, lines of text are essentially free-standing “statements” that output the text they contain, which is not at all how lyre works.

Another factor in this is that it makes it harder for the compiler to catch typos. If you miss a semicolon and write:

;storylet start
Forest Clearing

(snip)

branch Examine Your Scarf
;show (scarf_color == "")

You’d get the relatively obscure compiler error “unexpected ;show at [line, column]”, because the parser sees the “show” token in a place where it doesn’t fit any rule (show has no meaning inside a storylet block and you never opened the branch block). Which is hard to track to the missing semicolon at the beginning of the previous line. In fact the missing semicolon could be arbitrarily far back, since there are statements that are valid in both a branch and a storylet.

Meanwhile this:

;branch Examine Your Scarf
show (scarf_color == "")

Compiles, but with a bug that’s only noticeable once you see the branch. Syntax highlighting somewhat catches those cases but neither one creates useful parser-error squiggles (in some hypothetical situation where lyre has an LSP).

It may be possible though to go to a slightly more aesthetic syntax which is something like:

storylet start: Forest Clearing
    You are standing in a forest clearing, etc, you know the drill.
    ---
    // non-textual parts of the storylet

This is more similar to Yarn’s front-matter like syntax.

Without having actually used the language or knowing the intricate details behind it, I was envisioning syntax more like this:

;storylet anchor start: Forest Clearing
You stand before a crossroads that splits to the left and right. A man
stands next to the path on the right, wearing a large black velvet cloak.
;if scarf_color then: 
    // You can even nest interpolated text inside other interpolated text.
    Your { scarf_color } scarf is reassuringly warm.
;else:
    You are wearing a warm scarf.

;branch: Examine your scarf
    What color was it again?
    ;show:  scarf_color == "" 
    -> select_scarf_color

;branch: Talk to the man
    He looks *trustworthy.*
    ;req: social_anxiety < 7 
    -> conversation

;quality cloak_knowledge boolean: Examined the Cloak
You've taken a closer look at the Cloak of Darkness.
;describe_change():
    You add your observations to your mind palace.

The differences being:

  • Using a colon at the end of code lines to avoid even more delimiters, particularly the text associated with start and branch, which seems like an appreciably large percentage of the overall text of the game.
  • I think it’s fine for the -> at the start of the line to stand alone, without also needing a semicolon.
  • The braces around the if seem unnecessary once the if is clearly delineated as code on its own.
  • I added colons after then and else mostly for consistency, so that as the user I don’t have to remember which keywords require a colon at the end and which don’t. However, I am not wedded to this idea.
  • EDIT: also used colons to get rid of the parentheses in show and req, while we’re at it.

I’m confident that changes like this, or something roughly in this ballbark, could be made and supported unambiguously by the parser, although it might be a bit more work. I don’t, in general, think that “making the parser easier to write” is a high priority in my mind, relative to things that improve the user experience. To be clear, I have written my own interpreter for Yarn in the past, so it’s not like I have no experience with trying to write a parser for a language that has this property of mixing code and non-delimited strings.

Regarding the dog-petting example, can you show what it would look like in the current syntax? I agree that it looks unintuitive at first, but it’s not clear to me what use of delimiters would make it less unintuitive.

Regarding errors, “cryptic compilation error related to problems on a previous line” is a common issue in most languages that I’ve worked with (due to e.g. a missing closing delimiter somewhere), so avoiding that is not a hugely compelling argument to me. And “errors caused by forgetting to mark code as code” is an existing problem in Ink that I run into very occasionally, but not nearly enough for it to have ever registered to me as an annoying feature of that language (and I am not shy about calling certain aspects of Ink annoying).

With the comparison to Yarn: firstly, I am also somewhat annoyed by that aspect of Yarn. But more importantly, in Yarn you only need it once per node, not once per branch. But also also, I do agree that eliminating the unnecessary starting """ delimiter, as you suggest at the end, would be a noticeable improvement on its own.

Sorry if I’m being excessively critical. I’m mostly just using this as an opportunity to soapbox about my own preferences, I guess? I’ve thought about this topic (how I would design my own IF language) a lot, and probably just need to pull the trigger on doing it myself some day.

There’s definitely some general syntactical cleanup that can be done, my initial grammar is pretty conservative about being very straightforward but things like obligatory parens around show/req conditions can just go away (there’s no need for a colon there either).

But it’s just not my personal preference to have every keyword prefaced with a ; or have every one need to be a symbol that wouldn’t occur in flowing text (which always has potential to mess up someone’s use specific use case and force a bunch of escaping).

As far as the actual workflow for writing lyre, too, a lot of the time working in StoryNexus I would stub out large segments of story and then fill them in. In lyre this is easily done:

storylet foo
    branch
         -> next // "next" is a keyword that points to the next storylet in the file

storylet bar
    branch
         -> next

storylet baz
    branch
         -> next

storylet qux

quality x
quality z
// etc

Add obligatory this-isn’t-flowing-text symbols and this kind of stubbing out becomes more annoying and gory to do, IMO.

But also, fluidly mixing text and non-text statements implies a semantics that lyre just doesn’t have. Eg, you’d expect to be able to do:

Something something text
;if condition:
    ;value += n
    More text

Lyre would have to have substantially different semantics for this to be meaningful. Right now, the language would semantically cope with having a quality value assignment (effect) inside a string interpolation, but the behavior would not be what anyone expects (it’d interpolate the quality change into the text, not actually perform the change). But the parser actually forbids this, which is, I think, ideal (because it means that doing foo = bar when you mean foo == bar will throw a compiler error in almost all contexts).

This is the point of the dog example; lyre doesn’t conceptualize lines of text as distinct entities.

The way you’d get the intended effect of the dog example in actually-existing lyre is:

storylet pet_dog
"""
You've pet { dogs_petted - 1 } dogs today, which is far too few!
(...) Now you've pet { dogs_petted } dogs!
"""

By design, there’s no way for a value to change halfway through evaluating the body text.

I should also say: making the parsing simpler isn’t just about how hard it is to write, it’s about:

  • Heading off possible parsing bugs or unexpected behavior in edge cases; complicated contextual parsing can get very unintuitive or break in unexpected ways.
  • Making it less likely that future extensions to the language will introduce compiler bugs.
  • Making the parser faster—if body text is cleanly delimited, the tokenizer spends most of its time in a mode where it has a very small set of tokens to recognize.
  • Making it easier for someone to port or modify the compiler for their own needs

All of which does impact user experience rather a lot.

Hi, I’m the creator of Loreline, so I’m always curious to see in which direction other people making writing tools are going, and this discussion makes me want to share some thoughts I have gone through when working on Loreline, which can relate to what is discussed here.

It first looked like this:

Text had to be surrounded with quotes, so that control structures weren’t ambiguous. It was also fairly easy to implement a parser for that.

But then I got annoyed by this obligation to use quotes, though I didn’t want to go through the symbol soup of Ink or the << delimiters of Yarn (as good as they are in what they do regardless), so I tried to see if I could make the parser (or more specifically the lexer) work without quotes. Turns out, a normal person would be able to recognize the difference between prose and a condition or a state change without needing many symbols, so I tried to make the lexer as smart as what we would read naturally.

Some iterations later, the quotes became optional. It already made the lexer much more complicated, because you need to recognize things by pattern, and not just by simple quote delimiters. But the result was worth it, it was already more readable and easier to write!

Now, what about the braces? The text was already naturally using indentation, and it turns out this information alone would be enough to understand the structure, so, let’s make them optional!

Each of these improvements required a lot of work in the lexer, whose job was to differentiate what is prose, what is an actual keyword, etc. and feed that to the parser, but making the writing and reading experience feel more natural was a good drive to push towards this.

These days, the language looks like this:

Trailing conditions on choice options were one of those tricky patterns to get right in the lexer. The syntax coloring is one of a kind as well, but very useful to understand how the text is recognized while typing or reading it. All in all, it’s definitely a complicated route that not everyone would want to take, so it also makes sense to keep things much simpler.

Then, especially when the runtime is able to read those files as-is and parse and execute them on the fly, it becomes a challenge to support such parsing/lexing complexity, as it would take more and more time and effort to port that runtime and parser to each platform, to make sure the diversity of patterns is consistent everywhere. That’s where another technical choice helped a lot: using Haxe for writing the runtime, so that it can be transpiled to a variety of other target languages, without having to rewrite that same runtime over and over again. It’s ok for me to push the runtime and the parser as far as I can in order to make the language easier to use on a daily basis (at least, based on its own opinionated features), given that it is written once and can run the same on all the target platforms (currently: JS, TS, C++, GDScript, PHP, Java, Haxe, C#).

I’ve actually been working on a solution for Loreline to create storylets with a `when` keyword:

Though, unlike Lyre, it is not resolved statically, and you still “insert” additional patterns instead of having them hook up by themselves.

Sorry for hijacking this thread with that, but it just made me want to share those details as I was reading discussions on a subject I’ve spent a lot of time on lately :grinning_face_with_smiling_eyes:

Good luck in your narrative scripting journey, which I understand gives more focus to the storylet structure and breaks away from the more linear tree that other systems may have.

Interesting read!

Very interesting. Personally, i think if you’re making a modern IF system and you still have to put your game text in quotes, you’re doing something wrong and you should have a serious rethink.

Interesting. How did you solve for coping with something like:

beat questioning
    suspect: "No, officer, I don't know anything about that. I just... I think...
    if only she hadn't been there that day!"

Like does the lexer consume that if at the start of the line as if it’s the start of an if-statement and then chokes? Does it backtrack and reanalyze the line, maybe after it runs into that hadn't which can’t be an identifier? Does a line that can’t lex in “code context” automatically become “text context”, meaning that a typo creates a line of text where there otherwise would be code? Is the idea that the lexer just isn’t meant to deal with text that’s been hard-wrapped?

In the example you shared, it’s easy, because the text is surrounded by quotes, so there is no ambiguity, the `if only…` is part of that quoted string.

Here is what happens with other patterns, without quotes:

The last example is recognizing `if only` as a valid condition, whereas `if only she …` is recognized as prose because `only she hadn’t` is not a valid expression that a `if` can consume.

The lexer is holding most of that complexity as it is resolving all those cases so that the parser receives a clean stream of tokens where strings have been properly delimited, quoted or not. That said, the logic is pretty simple. The lexer is matching patterns like “is this the start of a condition?” → yes/no, if not, it checks the next pattern and so on and falls back to prose at the end (I’m simplifying a bit but you get the idea).

It is indeed reading characters for each pattern matching test and backtracking if that doesn’t hold true, but it is done in a way that doesn’t necessarily hurt performance as it’s just reading over and over again the same buffer, which is likely cached by the CPU.

A typo wouldn’t necessarily mean no error that silently fallback to prose, it depends on the cases. The text editor with completion and syntax coloring is very complementary to the writing as it gives you visible hints on how the script is interpreted. From my experience, that’s enough to understand when you mistyped something.

Yeah having thought about this quoting question a lot: lyre doesn’t have a generic concept of “text”, entities have titles and bodies, and both components are optional, both in terms of syntax (it needs to be possible to stub out content without typing in empty quoted text) and in terms of design (it needs to be possible to comfortably write a story where, eg, body text isn’t used at all for branches, or results don’t have titles). Distinguishing between the two inevitably creates a need for explicit delimiters of some kind.

I thought about consolidating them (eg, only bodies exist, and picking out the “title” of something is kicked down to the UI layer to deal with) but qualities, in particular, really want to have both things as distinct entities. Lyre’s fundamental nature as a language for declaring entities rather than writing a sequentially executed set of instructions also kind of points to having a clear separator between the part of something that is text (with interpolations) and the part of something that is non-textual data. Again, it doesn’t make sense in lyre’s semantics to interleave statements that do things inside the expression that makes up the body text. Yarn does the same thing with its separator between the front matter of a node and its body, it’s just that in Yarn there’s a lot more body to the amount of front matter (while lyre treats things like branching as being outside the body).

It is possible of course to get rid of actual quote marks and use something else as a delimiter, which might be more aesthetic:

storylet start::
    It is dark, and you are liable to be eaten by a grue.
    ---
    grue_risk += 1 // Since we last spoke I already simplified the effects syntax

This is a pretty trivial change; the lexer just treats '::\n' as the opening of a body (The double colon differentiates this from a single-line template string which would start with :\n). But the problem is, how do you handle titles? One option is to have the title live inside the body, maybe reifying Markdown convention:

storylet start::
    # Darkness
    It is dark, and you are liable to be eaten by a grue.
    ---

This would then compile to a storylet with a title of “Darkness” and a body of “It is dark, and you are liable to be eaten by a grue.”

The single-line :\n then exists to enable things like unquoted text in interpolated expressions:

storylet start:
    # { if in_darkness then "Darkness" else "Lightness" }
    { if in_darkness then:
        It is dark, and you are liable to be eaten by a grue.
    else:
        Oh, it's really rather bright in here, huh?
    }
    ---

And also nicer syntax for tables, which I don’t remember I’ve shown:

table describe_risk
    with grue_risk
    < 1:
        Entirely safe.
    < 2:
        A little risky.
    // etc

(a table is basically just a function structured as a bunch of pattern matching statements; they’re designed to fill a role similar to quality level/quality change descriptions which are a StoryNexus feature used very extensively in FL)

I also fundamentally don’t think an extremely clever lexer that you could never really re-implement from first principles is a great idea. lyre already uses a simpler, faster, more error-tolerant scannerless tree-sitter parser for syntax highlighting, and once you get to the point where the lexer is backtracking and making determinations like “does this look like a valid line of code or is this body text”, you then tie text editor tooling strictly to the actual language compiler.

I was considering bringing up Loreline as an example of an alternative that was more aligned with my own preferences! Although I think you’ve acknowledged somewhere that it does increase the reliance on syntax highlighting to make it clearer to the user when escaping might be necessary, which I recognize somewhat contradicts the stated goals of Lyre to work with a “regular-ass text editor.”

I respect this opinion, but I disagree with it, in the sense that what matters the most to me is the actual writing and reading experience with the tool, which not only includes the language semantics, but also the editor you are using, the tooling that helps you iterate and so on, and I prefer a thousand time move the complexity to that tooling, rather than making compromises at user level (but I admit what is a compromise or not is highly subjective and is probably not the same for you than it is for me).

And given that I still write only a single codebase in Haxe, I’m not really facing and issue with porting the parser and the runtime to other platforms. It’s something available from the start by design and any evolution to the language lands at the same time everywhere.

For what it’s worth, Loreline syntax is still fairly readable without an editor that provides syntax highlighting. I wouldn’t consider it less readable than a technically easier language to parse.