Z9/Z11 - proposal for a new z-machine format [an intellectual exercise]

Technical Development Specifications
42

This just came up in the Parchment localStorage thread…

What I implemented for Quixe is that regular full-game saves are sandboxed by the interpreter to a game-specific directory. All other data-saves (partial-memory saves and arbitrary files) are sandboxed by the interpreter to a common directory (shared by all IF games). It is then the game’s responsibility to pick a unique filename.

The fact is that this is a low-use feature. The number of parser games trying to store a preference, achievement, or shared-state save is measured in tens, not thousands. So asking each author to pick a unique filename is not a problem.

(Transcripts? Not sure. Players would probably prefer them to be sandboxed per game. I don’t know if any game hack has been proposed that involves writing a transcript and then reading it back as an arbitrary data file.)

1 Like
43

I strongly dislike games being able to read and write arbitrary filenames. As of standard 1.1. games can already save and restore files with game supplied optional filenames and unfortunately there is a flag that will allow this silently. I will never implement that and I recommend no one else do so either. Relying on cross-platform filename sanitizing should make people very nervous. I’ve done some experimenting with malicious z-code and while file read/write exploits are mostly non-existent (from limited testing), I wouldn’t bet money on that.

If you are going to do something like this, I would mandate a pure ASCII alphanumeric only filename with any extension provided only by the interpreter itself. No paths, no punctuation, no exceptions. Even with those restrictions you need to forbid windows reserved names: LPT1, COM2, CON, NUL, etc. If you allow those at all, even adding an extension isn’t guaranteed to save you.

Tying files to the IFID seems unnecessary. Just have the game suggest a filename. Games in the same series can prompt the user with the default and allow them to change it if they like. To allow games in the same series to share data files, just place saved data (not needed for full save files) in a common directory chosen by the interpreter. zarf seems to have already done this.

Not sure what all can be accomplished this way, but you can definitely fill someone’s storage in a heartbeat with a malicious z-code file and unprompted save data or a transcript. You can also drop nasty surprises using a variety of extensions, not just exe or dll. :slight_smile:

Edit: Even allowing seemingly “harmless” extensions like image or audio formats can be used to take advantage of vulnerabilities in specific applications.

44

This is maybe very useful but when converting the early Zork from MDL to Zil as truthfully as possible I really missed an opcode to get current datetime.

45

I would add to this that if paths are constructed from user names, then crafty users can name themselves CON, AUX, NUL and then cause some havoc. These even can’t be the name of a folder. c:\CON\file.txt is also illegal.

This is perhaps a legacy Windows problem but it could also be a problem in the realm of named pipes on all systems. A two staged attack would be to have a process that infects the system and listens for named pipe file I/O. Then the malicious 2nd stage causes a file name to be generated that is a valid named pipe that causes redirection of the output to the malicious first process.

\\.\pipe\Exploit is a valid file name that redirects to a named pipe.

46

No paths. Never allow paths.

1 Like
47

If this movement takes off and actually gets implemented and adopted, well many of the interpreters are still going to be implemented using Glk and subject to its file restrictions. That means specifying a basename but not an extension. You also can’t control which folder it goes into.

I think the focus should be much more on how to ergonomically read and write to files. There’s no file streams in the Z-Machine. Would it be feasible to add them?

48

I also dislike games reading and writing files “behind my back” so to speak.

What an interpreter can do is not use the requested filename as part of the real filename, just create files like data0003.raw, store the filename in it, and search all the files when looking for a particular one. Alternatively it could store the files in a little database file (e.g. by using the libdb library).

A more radical option I was thinking of trying is to consider the files part of the gamestate, just kept in memory, though saved into savegames. If the game creates a file and later reads it, it will work, but after a restart or restoring a different savegame that data will be gone / be different. Naturally that mucks some things up (achievements etc).

That could even be a choice in a user’s preferences, how much to allow games to access the filesystem: ALLOW or PRETEND or DISALLOW.

1 Like
49

Ultimately all this will become academic whenever it moves from local storage to use the indexed DB functionality. Indexed DB abstracts away a lot of the problems. Its metaphor is like a database. The name of your database, a table in that database, and a row of data in that table. The row of data is your game data. Table and data base name are 100% configurable behind the curtain so to speak. Those are things you wouldn’t let a user give a name to identify. Code creates a well known DB and Table names and the game data goes in the table as a row of data. You can only really clobber someone else’s save data if your DB, Table, and Row are exactly the same. You can luck into someone else’s database and table if you are lucky enough to choose some generic database and table names, probably like db.table, but you won’t overwrite anything yet until your row collides.

50

Agreed. At minimum a “read x bytes from position y in file z” opcode would open up all sorts of possibilities that aren’t available with the existing entire-file-at-once paradigm.

51

I feel like interpreter authors can do a much better job of sandboxing than standard authors. The interpreter authors, for example, are the ones who know whether forbidding / and appending a terp-specific extension like .glkdata are sufficient, or if they’re not even necessary (e.g. in a web browser where there isn’t really a “file system” to speak of).

I think the standard should just allow “persist these bytes somehow, with this identifier” and the interpreter should decide how exactly to implement that (e.g. as a file with that identifier in its name in some sandbox somewhere).

1 Like
52

I think this is the only thing I’d consider support for, and really it’s just a refinement of @save and @restore. In Bocfel, I support these opcodes but confine files to game-specific directories.

For new/refined opcodes, if it were explicitly noted that all games have access to a common “directory” and can overwrite/read each other’s data at will, then at least the contract is clear and nobody’s surprised. And things wouldn’t be defined as “files” so there would be no expectation that the identifier given maps to a filename, or even that a file must exist at all. So sandboxing becomes easier: store in a DB (as noted by others), hash the identifier for a filename, etc. If the filename doesn’t matter, then a lot of the concern goes away.

2 Likes
53

The main use case this doesn’t allow is the trick suggested in the I7 manual, where an internet-connected program writes some data to a file the game can read. But that only works if you’re distributing your game with a custom executable anyway, so you could just use an interpreter that you know stores files in a specific way, and account for that.

54

There’s a broader philosophical question here of “what would a new Standard actually be trying to achieve?” Different proposals in this thread have different ideas of this.

Fundamentally, every new Z-machine feature has to justify its existence by answering the question, “why is this so valuable that it is worth either (1) a massive community effort to get it into as many interpreters as possible, or (2) fragmenting the ecosystem into terps [and games] that support it and those that don’t?” As Z-machines (and Z-machine games) have proliferated, this bar has become increasingly hard to reach.

Some factors make it easier, however:

  • You’re forking the Z-machine for a particular project or platform, where the game is always expected to be shipped with that interpreter (e.g., Vorple).
  • The feature degrades gracefully on non-compliant terps.
  • The feature fills an obvious gap that actual authors actually want filled.
  • The feature is clearly needed in the existing Z-code ecosystem; building a new format or switching to something like Glulx would not achieve the same ends.

The resurgence of retro-IF has changed these considerations dramatically. For a long time, if someone said “I want images, expanded memory, etc.”, the answer was “just use Z6” or “just use Glulx.” But now, there are a lot of games specifically targeting Z3 or Z5 for retro compatibility; a gracefully-degrading Z5 image opcode like arc_image suddenly makes sense.

Returning to the question: what would new standard actually be trying to achieve?

Motivation 1: A “purer”/“more perfect” Z-machine
This is appealing to theorists, but has trouble answering the question “why is this worth fragmenting the ecosystem?” zzo38 tried to do this with their series of “quixotic” terps a number of years ago, and it went down like a lead balloon. At a certain point, aren’t you just creating your own new XYZ-Machine format?

Motivation 2: Support bigger games
This is appealing to authors, but has trouble answering the question “why the Z-machine and not another system?” Today, the Z-machine’s main appeal is retro compatibility, but most retro terps already don’t support the full memory allowance of Z8. Who would a bigger Z-machine actually be helping?

Motivation 3: Improve optimization on retro platforms
This has more reason for existence, but would require a lot of joint effort by terp and compiler developers to be useful. In practice, the changes required to make a significant difference would de facto result in a new incompatible VM format.

Motivation 4: Allow game features that weren’t previously possible (but which would actually be nice to have in retro/Z-code contexts)
IMO, most genuinely viable ideas fall under this motivation. Examples of commonly-discussed pain points that could be addressed:

  • Graphics in Z5/Z8 games. (arc_image seems to have won here.)
  • More granular ways to store/retrieve data from disk.
  • More text-formatting options (e.g., proportional alignment/justification).
  • A socket system for interfacing with out-of-terp software or web APIs.[1]

TL;DR features that improve functionality on retro platforms, but which degrade gracefully on non-compliant terps, are most likely to get traction. Calls for fundamentally redesigning core opcodes and memory models will be a harder sell.

TL;DRR xkcd 927.

  1. This raises many questions, but “what’s the point of internet access if Z-code is mostly a retro thing?” is actually not one of them. “Giving things internet access that were never designed to have internet access” is, like, 40% of the whole retrocomputing scene these days. ↩︎

2 Likes
55

Great points, all. I’m of the opinion the primary reason for changing the Z machine at this point is for better retro experiences, because we’ve already had a newer VM for 20+ years that’s more suitable for anything built in the last 30 years.

  1. A “more perfect Z-machine”. My biggest “issue” is that a lot of large games written in the 90’s were never run on older hardware. You don’t optimize something if you don’t notice it’s slow, and so some great games run unnecessarily slowly on retro hardware. I’d want a Time Machine, or the nerve to ask authors to hand over ancient 30-year-old code so I could optimize it for them.

  2. Support bigger games. Yeah - I’m pretty sure a 128k apple 2 with a hard drive can run a Z8 game of any size, but just barely.

  3. Improve optimization on retro platforms - see #1 - the Z-machine is already fast enough, it’s just that people had no idea their code was slower than necessary walking the entire object and class tree after every command or whatever.

  4. I think the winner here is arc_image. I’m not even sure we need more granular ways to store/retrieve data, but actually implementing the existing v5 save/restore extensions more consistently might be enough. It could open up some hybrid RPG options I suppose, but for a big multi-part adventure, a simple passcode system would probably suffice.

-Dave

2 Likes
56

Perhaps but you can still make the case that Z6 already does images and it is not a community extension

57

Z5 plus an image opcode or two is infinitely easier to implement for interpreter writers than Z6.

4 Likes
58

Perhaps someone will tackle it, Z6, with AI. It looks like a very robust format.

59

Full z6 support, with graphics, is very hard to implement on most 8-bit platforms. Infocom did implement it on Apple II, but games were terribly slow. No one has implemented it for a regular 8-bit platform since then.

ArcImage is a lot more constrained, in ways that makes it a much more viable option for 8-bits.

1 Like
60

This must have been in 1988 then. Z5 and Z6 were in 1987 and 1988 respectively, with Z6 being the finishing touches to Z5. In accordance to The Z-Machine Standards Document

61

I think it was Graham Nelson who said Z6 was designed by committee and they made an absolute mess of it. Even nowadays, many of the interpreters that support Z6 actually just hardcode support for the individual Infocom games that used it, not a general implementation that modern games can use.

arc_image is a much better solution, imo (as a compiler maintainer). Though it may need a different name once non-Arcturus languages use it too.