Idea for fast setup in Inform games (not quite working yet)

Several months ago (in email), there was a discussion about avoiding very slow startup procedures for Inform games. (E.g., Reliques of Tolti-Aph burns bazillions of Glulx CPU cycles in “when play begins” code.)

It would be nice if the I7 compiler could just compile a game file in already-started-up form, but this is difficult for a bunch of reasons.

In unrelated work (also several months ago), I added a Glk facility to store a data chunk in the Blorb file for the game’s use. (github.com/erkyrath/glk-dev/wik … ec-changes) It recently occurred to me that this could be used to solve the startup problem.

Plan: when the game starts up, it tries to open a data chunk with a conventional number. If that chunk is present and nonempty, the game assumes it’s a save file and restores it. The save file contains the game in fully-started-up mode and it can continue from there.

But where does this save data come from? Here’s the clever bit: when the game starts up, if it sees that data chunk exists but is empty, it assumes the author wants the save file written out. So it runs the (slow) startup code, gets itself in position, and then writes out “autorestore.glkdata” as a save file.

If the data chunk doesn’t exist at all, or if the interpreter doesn’t support the Glk data feature, the game ignores all the fancy stuff and just runs the (slow) startup code and gets on with the game.

So the author compiles the game, blorbs it up with a zero-length data chunk, and runs it. There is now an autorestore save file ready to do. The author then blorbs the game up again, replacing the zero-length chunk with this new save data. This is a hassle (and you have to repeat it every time you recompile your game, because you need the save data to be compatible). But it’s easily scriptable.

The nice thing about this plan is that it doesn’t require any updates to the Glulx spec, the Glk spec, or Glulx interpreters. It works entirely in game code. It’s backwards-compatible to old interpreters, too.

So I went to implement a proof-of-concept, and I ran into the snag. (Anybody want to guess? No?)

Glulx save files are Quetzal-based, which means they’re IFF form objects. Blorb files are also IFF form objects. When you embed one IFF form in another, you’re supposed to leave off the inner file’s header. (Because the IFF embedding wrapper is an inner header, basically. This has long been a source of confusion for embedded AIFF files.)

Unfortunately the Glulx interpreter’s restore code gets all confused by this. So my proof-of-concept code fails.

Possible solutions:

(a) Make the interpreter smarter about this. (Requires an interpreter update.)

(b) Shove the save data in there with a double header. (Requires updates to Blorb tools, at least the ones which are trying to be correct about IFF embedding.)

© Prepend a padding byte (or four) to the beginning of the save data. Now it doesn’t look like an IFF file any more, so Blorb tools will shove it in there as raw data.

(d) Make the Glk data-chunk call smarter. That is, we could decide that I screwed up by making that header inaccessible – you’d run into the same problem if the game wanted to read AIFF data out of a Blorb chunk. The fix would be to add an optional flag saying “give me the damn header, if there happens to be one.” (Requires a Glk library update.)

Thoughts? I lean towards © if we want to do this fast and dirty, (d) if there’s time to do it right.

Oop, I missed a possibility:

(e) Make the Glk data-chunk call smarter, but don’t change the API. Instead, change the defined semantics to say “For embedded IFF chunks, the readable file should start at the beginning of the header, not the beginning of the data.” (Requires a Glk library update, but a simpler one, because it doesn’t affect the header file or dispatch.)

I prefer this over (d). It’s technically a backwards-incompatible change to the Glk 0.7.4 spec, but I’m confident that it doesn’t break any existing game files – nobody is using this facility for IFF chunks. (In fact I believe nobody is using it at all, except for my original resstreamtest.ulx unit test.)

I was going to say that it wouldn’t be hard to patch the ulx file itself, but I guess you’d want the stack preserved too.

An aside that I’ve been meaning to ask: could an undo save be stored and used in quetzal form? @restore should work the same as it just needs a result variable and PC - it doesn’t care if the instruction that produced them was @save or @saveundo.

You want to preserve both the stack and any @malloc state that may have been set up.

For what purpose?

Quixe uses a different save format for undo states – it clones some higher-level Javascript data structures rather than serializing everything into a stream. I may wind up doing something similar on iOS, for speed.

Interpreters could treat @saveundo as an easy way to get auto-saving working in most games. This wouldn’t be suitable for switching apps on iOS, but it could work for browser crashes etc.

ZVM also doesn’t serialise the undo state to quetzal, but I might do so to save memory. Memory or speed… which is more pressing?

Um, OK: a Glulx game opened a file in its startup and kept it open, so a game that’s warm-booted via your new way tries to use the file handle it created, but the interpreter now disagrees with the game about whether or not the file handle is open.

Did I win the cupie doll?

Nope, I’ve covered that. The autorestore function calls GGRecoverObjects(), same as the regular player restore function.

The nice thing about interpreter autosave is that the interpreter can handle it as an internal affair. It doesn’t have to match the spec or what any other interpreter does.

I don’t like relying on the undo chain for that, however, because games can fail to @saveundo. (E.g., the ongoing forum thread about handling unwinnability by suppressing @saveundo.) Also, you get the amusing quirk of every session starting out with the line “(Undone.)”

Fizmo already had an autosave feature, although it was intended for a slightly different use case. It generates a save file while the interpreter is paused for input – which does make a difference. (The PC is on @read or @readline; restoring must not store a result code, and must restart the input opcode rather than advancing to the next opcode.) I think we’ve talked about formalizing this variant of Quetzal, but nobody’s written it up. Anyhow, I decided to stick with that model for iOS Fizmo, but decorate the save file with additional (iOS-specific) information about the current screen contents. Also there’s some wiring to go in and make interpreter internals match the Glk state, because this restore path doesn’t have the GGRecoverObjects() call I mentioned in my last post.

Glulx autorestore will require yet more wiring, because some variables that are interpreter internals in GlkFizmo are game internals in Glulx…

Anyhow, autosave is a ball of tigers, as you can see, and I am very reluctant to standardize anything about it. Leave it all as interpreter implementation, and don’t ask questions about the soup.

My proposal in this post avoids the entire soup-pot. (I was very glad when I realized that was possible.) Standard Quetzal, no interpreter hacks, etc. Except for the one snag.

So the problem, if I understand it correctly, is that there is currently no way to make Glk give you an IFF chunk correctly embedded in a blorb, complete with the header you would need to interpret it correctly? That seems like a clear design bug in Glk, and your proposed solution (e) makes the most sense to me. As you mention, the current implementation is already flawed for including AIFF chunks (or other IFF-form chunks, but AIFF is probably the most common) in a blorb; I assume this use case would also be resolved by this change?

I presume you could if you opened the blorb as a file resource and wrote your own IFF parser in Inform… but that’s not as nice as the Glk API just giving you the correct data.

If there are Glk gestalt calls during the startup code, testing for e.g. graphics or sound, wouldn’t the autorestore save file contain the results of those checks on the author’s machine only?

I guess this is a more general question as to how portable saved games really are between different Glk implementations.

You would have to do those checks in a different part of the setup code. Short-circuited setup should not check the interpreter capabilities, or open windows, or anything like that. The feature is intended for setting up tables and such-like.

Making save files portable in this sense should always be possible. You do have to think about the situation and test carefully, if you’re opening extra windows or sound channels.

OK, great. That was my only reservation and I’m glad it’s not an issue.

I lean toward option d, the amended Glk call, mostly because I still haven’t added the Glk data functions from earlier in the year and it’ll be a bit before I can work on Gargoyle again. So from my perspective there’s time to do it right. :slight_smile:

See update in this post: Glk 0.7.4

Right. I’m proposing a change that will fix that. I’ve tested the fix in CheapGlk, but I haven’t posted that change yet – gimme a couple of days.

By the way, half the problem turned out to be that I was generating broken Blorb files. This was a bug in my blorbtool.py script. Or rather, a weakness which was a powerful invitation for the user to make mistakes. (Turns out, even I can’t get the damn AIFF embedding right on the first try. Grrrgghhh.)

Anyhow, I’ve updated the script to catch this sort of stupid mistake. I don’t know if anybody is using this thing, but if you are, grab the updated copy:

eblong.com/zarf/blorb/blorbtool.py

This reminds me of an extension I started writing a while back but never submitted…

[code]Version 1/101121 of Quick Load (for Glulx only) by Jesse McGrew begins here.

Before play begins is a rulebook.

The save snapshot before play begins rule translates into I6 as “QL_SaveSnapshot_R”.
The save snapshot before play begins rule is listed first in the for starting the virtual machine rules.

To save a snapshot for quick load: (- QL_SaveSnapshot(); -).
To decide whether running from a/-- snapshot: (- (0–>6 == QL_Main) -).

The quick load initialise memory rule translates into I6 as “QL_InitialiseMem_R”.
The quick load initialise memory rule is listed instead of the initialise memory rule in the startup rules.

Include (-
[ QL_InitialiseMem_R;
if (0–>6 == QL_Main) { VM_PreInitialise(); rfalse; }
else return (+ initialise memory rule +)();
];

[ QL_Main; @tailcall Main 0; ];

Array ql_buffer -> 4;

[ QL_SaveSnapshot_R;
if (0–>6 == QL_Main) rfalse;
ProcessRulebook((+ before play begins rulebook +));
QL_SaveSnapshot();
];

[ QL_SaveSnapshot res fref i memsize buf checksum;
fref = glk_fileref_create_by_prompt(fileusage_Data + fileusage_BinaryMode, filemode_ReadWrite, 0);
if (fref == 0) jump SFailed;
gg_savestr = glk_stream_open_file(fref, filemode_ReadWrite, GG_SAVESTR_ROCK);
glk_fileref_destroy(fref);
if (gg_savestr == 0) jump SFailed;

! write memory to file
@getmemsize memsize;
glk_put_char_stream(gg_savestr, 0->0);
glk_put_buffer_stream(gg_savestr, 1, memsize - 1);

! change memory size and starting routine
buf = ql_buffer;
buf-->0 = memsize;
glk_stream_set_position(gg_savestr, 3*WORDSIZE, seekmode_Start);	! EXTSTART
glk_put_buffer_stream(gg_savestr, buf, 4);
glk_stream_set_position(gg_savestr, 4*WORDSIZE, seekmode_Start);	! ENDMEM
glk_put_buffer_stream(gg_savestr, buf, 4);

buf-->0 = QL_Main;
glk_stream_set_position(gg_savestr, 6*WORDSIZE, seekmode_Start);	! Start Func
glk_put_buffer_stream(gg_savestr, buf, 4);

! recalculate checksum
glk_stream_set_position(gg_savestr, 0, seekmode_Start);
checksum = 0;
while (i < memsize) {
	QL_ReadWord(gg_savestr, buf);
	if (i ~= 8*WORDSIZE) checksum = checksum + buf-->0;
	i = i + WORDSIZE;
}
buf-->0 = checksum;
glk_stream_set_position(gg_savestr, 8*WORDSIZE, seekmode_Start);	! Checksum
glk_put_buffer_stream(gg_savestr, buf, 4);

glk_stream_close(gg_savestr, 0); ! stream_close
gg_savestr = 0;
rfalse;
.SFailed;
GL__M(##Save, 1);	
rfalse;

];

[ QL_ReadWord str buf len r;
buf–>0 = 0;
len = WORDSIZE;
do {
r = glk_get_buffer_stream(str, buf, len);
if ((~~r) || (r == len)) return;
buf = buf + r;
len = len - r;
} until (~~len);
]; -).

Quick Load ends here.

---- DOCUMENTATION ----

This extension allows us to create snapshots during gameplay. A snapshot is a modified story file that incorporates whatever changes to memory have been made since the game started. This makes it similar to a saved game, but it can be loaded directly without needing a copy of the original story file.

By default, this extension saves a snapshot at the beginning of the game, before the “when play begins” rulebook has run (and before most other initialization has taken place, such as opening Glk windows). The new “before play begins” rulebook will run before the snapshot is taken, so it may be used to perform lengthy initialization that will be baked into the snapshot. The “save snapshot before play begins” rule is responsible for running the rulebook and saving the initial snapshot.

We may also take additional snapshots, using the phrase:

save a snapshot for quick load;

However, we usually only want this to happen on our computer (when we’re preparing a snapshot for distribution), not on the player’s computer when the player has loaded our snapshot. To check whether the game is already running from a snapshot, use the condition:

if running from a snapshot, ...

Section: How Saving Works

The extension uses Glk’s file mechanism to write the contents of memory to a file and update a few header fields. The file that is produced is a Glulx program only (.ulx file), without any Blorb resources. See the section “Republishing existing works of IF”, in the “Releasing” chapter of the Inform 7 manual, for information on how to repackage it as a Blorb if needed.

The interpreter built into the Inform 7 application may not be able to save snapshots. If a save prompt does not appear when the game starts, try compiling the game using the Release feature and running it in a different interpreter.

Section: How Resuming Works

When resuming from a snapshot, nearly all rulebooks will run from the beginning, just the same as when loading the original story file. Only the data in memory will be different: object locations and properties, relations, tables, and global variables. (Contrast this to saved games: restoring from a saved game causes the game to resume exactly where it left off, skipping such rulebooks as “when play begins”.)

The only rules which are not run when loading from a snapshot are the “initialise memory rule” (since memory will already be laid out how we want it in the snapshot) and the entire “before play begins” rulebook.

Therefore, if we need the game to start differently when loading from a snapshot – for example, if we’ve added a new snapshot at the beginning of Part II of the game – we’ll need to test for that explicitly, using “if running from a snapshot” or another condition based on game state.[/code]