Z9/Z11 - proposal for a new z-machine format [an intellectual exercise]

Technical Development Specifications
1

I and @fredrik have discussed on an off how to expand the z-machine and I thought it would be fun to open up a brainstorming thread on what to do. I guess it is mostly an intellectual exercise that probably never gonna be implemented (because, you know, Glulx already exists…).

Hera are some potential goals for this new format:

  • it should basically still be a 16-bit system
  • existing intrepreters should need minial changes to support the format
  • it should be able to address more than 512kB
  • remove static data from dynmem
  • minimize waste between routines and between high strings
  • introduce helpful new opcodes
  • allow bigger games but also optimize running smaller games on retro platforms
2 Likes
2

To be able to address more memory.

Introduce a jumptable at the end of the file and let packed addresses be an index to this jumptable.

If every entry in the table is 3 bytes (24 bits) 16 MB is addressable. If one bit is used for designating if it is a string or a routine - 8 MB is addressable. If 4 bytes is used, 4GB/2GB would be addressable.

Waste is 3 (or 4 bytes) per string/routine. Currently 3.5 bytes is wasted on average in z8.

3

The one feature that I’d want from a new Z-machine version would be the ability to have some author-specified amount of state (up to a reasonably small limit) survive a restart, coupled with the ability to specify a different game file to restart from. That would enable arbitrarily large games to be written as a series of smaller modules, and also allow interestingly large games to be written for vintage hardware equipped with only a tape drive.

6 Likes
4

A new opcode that lets you swap values between two variables (and or stack), without use of a third temporary variable.

5

Since this is just academic and one of the requirements is that existing interpreters work, how then?

A “hypervisor”. They solved this problem on the C64 with overlays and bank switching memory. The same idea could be done in software.

Hack into save/restore to save the game state and restore it to a new module. The idea being that player data perseveres across the boundary.

All kinds of things become possible with a system that has infinite scale. There are of course tradeoffs because each modules game logic only knows about itself but that seems acceptable.

6

Just as a historic note, we’ve had a hypothetical Z9 spec and I think then (separately) a hypothetical Z10, so you’re up to Z11. :)

5 Likes
7

Add a memory bank called expanded array memory (xamem) - up to 64 KB which can only be accessed with four new instructions:

xloadb (load a byte from xamem)
xloadw (load a word from xamem)
xstoreb (store a byte in xamem)
xstorew (store a word in xamem)

This new memory bank sits just after readable memory, and contains arrays which have been specifically declared to be placed in this area, and the dictionary. The last part of this area is static. The dictionary is always placed in the static part.

The file contains a word saying where this area starts, a word saying how many bytes it is, and a word saying how many bytes are dynamic.

8

The jumptable Henrik mentions is something I brought up for the z9 proposition back in the day.

I think it would make sense to make each entry 3 bytes, and we can keep track of a value n, where all entries < n are routines, and the rest are strings. This means we can address the full 16 MB, 32 times bigger than a z8 game can be.

1 Like
9

Are these accessible or linkable?

(I guess they are buried deep in old threads archived at if archive.)

10

I looked briefly, but didn’t find a link. Sorry.

11

I think the z9 discussion happened on a mailing list, which was specifically about the Z-machine.

12

Oh yes. Hm. I still have some messages from the mailing list.

I don’t have time to dig through the whole thing right now, but here’s a summary:

Date: Fri, 13 Sep 2002 17:34:52 -0400
From: L. Ross Raszewski <…>
To: z-machine@GMD.DE
Subject: RE: [z-machine] V7, V9, V10

As I recall, things were going great, untill someone said “Let’s make a v10, this being similar to v9, but using the v6 screen model”, at which point, numberous debates erupted about all kinds of new graphics opcodes people wanted (Someone was rather adamant that pallete cycling be added, for example), plunging the proposal into feature-hell. Zarf promptly released Glulx, and the whole thing seemed rather less pressing. (Graham’s v9 proposal was a 32-bit Z-machine, IIRC)

That was written several years after the fact, as you can tell.

2 Likes
13

I found something here in the https://www.ifarchive.org/if-archive/infocom/interpreters/specification/ZMachineArchive-20020808.zip

The subject line is: [z-machine] Proposal for a proposal for Version 9

From ???@??? Mon Jan 18 09:43:49 1999
Return-Path: owner-z-machine@omega.gmd.de
Date: Thu, 14 Jan 1999 01:01:30 +0100 (BST)
From: Graham Nelson <graham@gnelson.demon.co.uk>
Subject: [z-machine] Proposal for a proposal for Version 9
To: Z-Machine Mailing List <z-machine@gmd.de>
Message-Id: <Marcel-1.46-0114000130-d07M+4%@gnelson.demon.co.uk>
Mime-Version: 1.0
Content-Type: TEXT/PLAIN; CHARSET=US-ASCII
X-Organization: none
X-Mailer: ANT RISCOS Marcel [ver 1.46]
Sender: owner-z-machine@omega.gmd.de
Precedence: bulk
X-UIDL: 34642fe02f99443ec31660722a8ce829



The summer and autumn of 1999 mark the twentieth anniversary
of the creation of the Z-machine, when two would-be Infocom
programmers, Joel Berez and Marc Blank, were stranded in Pittsburgh
and passed the time by trying to find a way to squash "Zork" onto
a personal computer.

The following proposal might serve as one way to mark the occasion,
if it finds favour: I suggest that it is now time to create
Version 9.

The proposal I'm making here is at a very informal stage indeed.
Perhaps I might begin with a meta-proposal: a way to conduct
whatever discussions we have about "Version 9", if we choose to
construct one.  Please excuse my rudeness in electing myself
chairman of our committee, so to speak.

  (a) We try above all to agree, and in particular to persuade
      interpreter-writers that a proposed change is worth their
      while.  Conversely, we listen to them if they say that it's
      harder than it looks.
  (b) We make minimal changes, which require minimal work to
      implement, given what they must accomplish.
  (c) We address genuine needs rather than trying to build some
      Platonic ideal of the Z-machine.  In particular, there is
      no need to build Lord Dimwit Flathead the Excessive's
      Version Infinity.
  (d) We consider that other languages besides Inform may one day
      want to compile to the Z-machine.

The two "genuine needs" which I feel are pressing are:

  (1) The limit on story file size (512K to 640K), and above all
      on the size of writeable memory (64K).  Story file
      authors are actually having to go to some lengths to
      get around these limits, and especially the latter one,
      where people are actually writing Inform libraries to
      compress arrays as bitmaps, rotate things in and out of
      lower memory, etc.

  (2) Less pressing, and arguably my own fault, but for reasons
      to do with Inform's typelessness I would like it to be
      possible to distinguish between a number and an object
      at run-time.  (As it is in some other object-oriented
      virtual machines, notably those for Smalltalk and Java.)
      At present the Inform doctrine is that:
          routines, static strings, Z-machine objects and classes
             are "objects" and can be mutually distinguished,
          numbers, characters, arrays, actions, dictionary words,
             etc., etc., are not objects and cannot.
      This is anomalous and unsatisfactory, not just for the
      puritanical reason that it makes the language less "clean"
      than it should be -- though heaven knows, that's true --
      but because it's very hard to write code that can
      satisfactorily print something out in a useful way.

Both of these issues have arisen in work on the new "Infix"
feature for Inform 6.21 (he said mysteriously).

There are clearly other issues which _might_ be addressed,
such as (3) an increase in the number of attributes or (4) an
increase of the maximum number of local and global variables.
I am quite reluctant to go too far down the road of meddling
with everything, though.

Here is my minimal proposal to address (1) and (2), and which has
the fringe benefit of improving Z-machine arithmetic.


1. Version 9 should have a flat 32-bit memory model, thus raising
both limits in (1) above to roughly 1 gigabyte.  All addresses
would be byte addresses, and the formula to convert a packed
routine or string address P to a byte address A would just be
A := P.

2. Everything that is now a 2-byte word should become a 4-byte
word.  In particular, variables and the stack hold 4-byte values.
Minimum stack capacity (in bytes) is doubled to accommodate this.
The opcodes @storew and @loadw work in 4-byte increments, in
that

   @loadw X Y -> Z

would set Z to the 4-byte value at byte address X+4*Y, rather
than the 2-byte value at byte address X+2*Y as at present.
The long operand type holds a 4-byte value, not a 2-byte one.
Some of the Z-machine tables need similar alterations: e.g.
the table of default property values.

Lastly, the header and header extension table are "doubled"
in the sense that what is currently at byte X would in future
be in the byte at 2*X.  What is currently at the 2-byte word
at X, X+1 would in future be a 4-byte word at 2*X, 2*X+1,
2*X+2, 2*X+3.

3. The upper byte of a 4-byte value should be allowed to
contain information about its type, but the Z-machine shouldn't
get involved in whether it does or how it does.  The only
provisions are these:

   (a) the 5 arithmetic opcodes calculate with signed 31-bit
       numbers and always store values with top bit zero.
   (b) when an operand is "used", meaning that the Z-machine
       uses the value to do something rather than simply
       copies it across to a new location (e.g. the stack,
       a variable, an array entry), the following happens:
           if the top bit is set, then the top 8 bits
           are ignored and only the bottom 24 used as the
           value.

For instance, @loadw X Y -> ... "uses" both X and Y.
@storew X Y Z "uses" X and Y but doesn't use Z, which it simply
copies to the new location.

-- 
Graham Nelson | graham@gnelson.demon.co.uk | Oxford, United Kingdom
2 Likes
14

I tmight be nice to smooth over some of the z-machine’s warts instead of adding features, like making the game responsible for re-printing existing input after a timeout instead of the interpreter.

There’s this note in the standard about the object property bytes:
"Bit 6 in the second byte is presently wasted, which is a pity as it could be used to allow up to 128 bytes of property data. But such a change would cause Infocom’s story files to fail (since they set this bit, unlike Inform story files). "

Things like that.

2 Likes
15

I’d argue the only reason for Z11 would be for resource-constrained environments.

  • Unsigned comparison opcodes to make classification easier
  • Dictionary lives in “routine” memory to conserve a bunch of static memory. You might need an opcode to print a dictionary word given address or index, and an opcode to read a dictionary payload byte.

Currently the only thing you can do (IIRC) with a packed string address is print it, so you might as well just not use them at all; then treat everything that isn’t an object index as a routine address (at which point you also don’t need unsigned comparisons). The overhead is two bytes, one for the 0 parameters the routine accepts, and another for the @print_ret opcode.

Another idea I’d considered for V5+ is that instead of always having nine letters per dictionary word, have two dictionaries; one is six letters, and one is nine letters (to avoid wasting two bytes most of the time). The limit case would be another dictionary for three letters, but that’s diminishing returns.

I agree with @sue, a way to allow a certain amount of dynamic memory to survive a restart would allow you to make arbitrarily large games on retro systems by designing careful choke points in the story. The limit case of this in current tech would be to give the player a hexadecimal string or somesuch that is their “password” into the next stage, which would encode a dozen or so bytes of state.

In my own code, I’ve only needed to disambiguate between false, true, an object index, and a routine address. Assuming not more than several hundred objects in the game, nearly 65,000 unique routines is enough to write a pretty complex story. One could argue a story shift of 4 or even 5, while wasteful of padding, could still allow for a very large story on a retro platform that could access that much secondary storage and/or memory.

-Dave

1 Like
16

You can also print it to a location in dynamic memory (using stream 3), so you can technically store static data tables as packed strings in high memory and only unpack them to low memory when needed. I have a (very weird, early-stage) WIP that does this.

(Technically, one could even implement code overlays this way, but this is less useful since code can run from high memory anyway. If for some reason your game needed enormous quantities of small, self-modifying routines, on the other hand…)

17

For Dialog purposes, the main things I want would be better served by a new Standard revision than a new version:

  • A way to print non-BMP characters
  • A way to use images on versions other than 6 (arc_image seems to be going in this direction)
  • An unlimited-range jump, ideally computed

But Dialog tends to run out of string and routine storage faster than it runs out of addressable memory, so that would be the architectural priority.

1 Like
18

Won’t more addressable memory for packed addresses automatically increase storage space for routines and strings?

19

When branching, right?

20

I’m using “addressable” memory to mean the area addressable with direct byte addresses rather than the packed string and routine addresses (“high” memory). Unfortunately the spec doesn’t have a name for it.

Basically, Dialog makes very little use of RAM and byte-addressable ROM. Almost everything gets compiled into routines.

Not necessarily. “Jump unconditionally to this computed absolute packed address” would work fine; then a normal jz or whatever could branch over the jump if needed.

The problem is that Dialog doesn’t make very good use of the Z-machine’s routine structure; it only uses it because routine calls are the only long-range absolute jumps the Z-machine offers. So Dialog generates huge blocks of code, then turns every long-range label in the block into its own routine header, and every long-range jump into “return the packed address you want to jump to”. The whole block is then called from an infinite loop that calls the block, takes the value it returned, calls that, and so on.

Which works okay, and even if Z9 were to become a reality, Dialog would probably still have to work this way—otherwise its games wouldn’t run on old interpreters. But it’s a very messy workaround for a lack of absolute jumps.