Z-Machine 1.2 Proposal

[Dannii]

Thanks for taking my feedback so well btw

Overlapping streams work like they do now. If you don’t want the data sent to multiple streams, don’t turn multiple streams on. It’s perfectly reasonable to allow a game to send text to the screen, and memory, and stream 5 at the same time.

But that’s not how stream 3 works. Probably most authors will want stream 5 to act like stream 3, and not send it to the screen. If you have multiple stream 5s active at once, what happens? I’d suggest making it work like stream 3 (the most recent takes precedence, and older streams don’t get any data), but it should be explicitly said.

It’s probably also worth noting that interpreters probably won’t send the data incrementally, but only when stream 5 is closed. Actually this may need to be made a requirement - a JS stream will need everything at once. Other stream types could perhaps handle it incrementally, but that’s the cost of making it generic.

I’m also happy to remain the contact person for registering gestalts and streams, if you’d rather not.

The input streams thing, yeah, I still don’t know what I’m doing about that. I want the functionality. @read takes text sent to the input stream and puts it into memory (lowercased, but maybe we can fix that). Why is that not good enough? (I’m serious, I need to know what’s needed so I can try to work out what to do).

Because @read then tokenises the data. Because it can’t handle non text data. It could be useful for sending back commands so that the player’s text input doesn’t need to be interfered with, but it’s no good for general data transfer (but writing back from output stream 5/7 would be.)

Surely the game knows which input stream is in use if it keeps track of that internally?

The stream could be empty, reverting to keyboard. The game could be restored and the different streams turned on or off.

Even an interpreter which supports changing the font size needs to be able to tell the game “No, I can’t make the font any smaller.” Interpreters which can’t change the font size at all signal this by never branching (except when the size argument is 0. It’s always possible to set the font to the default size). Given this, I’m not sure we need to have a gestalt.

I personally find branching a pain - I much prefer when it stores to a variable. Maybe most authors feel differently though… A gestalt is needed to know whether the opcode has been implemented or not. The idea for my 1.2 spec was that a very easy to implement @gestalt opcode can be added, after which all other features have gestalt selectors, and though encouraged, could be ignored if not feasible to implement. It’s an approach I’d encourage for your spec too.

There’s a selector for (almost) everything there’s a header flag for.

Not to be too disagreeable, but why? The header flag system is pretty sucky, but duplicating the information sounds messy (what about when people forget to implement one of them etc.)

[Marvin (David Fillmore)]

@Dannii

Overlapping streams work like they do now. If you don’t want the data sent to multiple streams, don’t turn multiple streams on. It’s perfectly reasonable to allow a game to send text to the screen, and memory, and stream 5 at the same time.

But that’s not how stream 3 works. Probably most authors will want stream 5 to act like stream 3, and not send it to the screen. If you have multiple stream 5s active at once, what happens? I’d suggest making it work like stream 3 (the most recent takes precedence, and older streams don’t get any data), but it should be explicitly said.

Of course it’s not. You’re right. I totally knew how this worked when I wrote it up.

Anyway, I still think it’d be useful to be able to send text to the screen and the interpreter at the same time.

@Dannii

It’s probably also worth noting that interpreters probably won’t send the data incrementally, but only when stream 5 is closed. Actually this may need to be made a requirement - a JS stream will need everything at once. Other stream types could perhaps handle it incrementally, but that’s the cost of making it generic.

So we need a way to signify that the interpreter can go ahead and process the data, and (as per someone else’s suggestion) we need a way to say “Finished sending this data” so we can start sending different data. These are probably not the same thing. I need to think about this.

@Dannii

The input streams thing, yeah, I still don’t know what I’m doing about that. I want the functionality. @read takes text sent to the input stream and puts it into memory (lowercased, but maybe we can fix that). Why is that not good enough? (I’m serious, I need to know what’s needed so I can try to work out what to do).

Because @read then tokenises the data. Because it can’t handle non text data. It could be useful for sending back commands so that the player’s text input doesn’t need to be interfered with, but it’s no good for general data transfer (but writing back from output stream 5/7 would be.)

@read can be told not to tokenise the input. I did wonder about non-text data. Is that going to happen a lot? It’s possible (if kinda ugly) to send numbers intended to be read by the game as numbers back using text, but it does require more intelligence on the game’s part. Maybe we just need a completely different way to let the interpreter write information to a table in game memory? Only when the game asks it to, though. A new opcode?

@Dannii

Surely the game knows which input stream is in use if it keeps track of that internally?

The stream could be empty, reverting to keyboard. The game could be restored and the different streams turned on or off.

Okay. Might be good to give the game a way to check this, even if we don’t have a new input stream.

@Dannii

Even an interpreter which supports changing the font size needs to be able to tell the game “No, I can’t make the font any smaller.” Interpreters which can’t change the font size at all signal this by never branching (except when the size argument is 0. It’s always possible to set the font to the default size). Given this, I’m not sure we need to have a gestalt.

I personally find branching a pain - I much prefer when it stores to a variable. Maybe most authors feel differently though… A gestalt is needed to know whether the opcode has been implemented or not. The idea for my 1.2 spec was that a very easy to implement @gestalt opcode can be added, after which all other features have gestalt selectors, and though encouraged, could be ignored if not feasible to implement. It’s an approach I’d encourage for your spec too.

I’m not convinced. I think the opcodes should all be added, but be allowed to do nothing. That’s how the Z-Machine has worked up until now. The features are optional, not the opcodes.

@Dannii

There’s a selector for (almost) everything there’s a header flag for.

Not to be too disagreeable, but why? The header flag system is pretty sucky, but duplicating the information sounds messy (what about when people forget to implement one of them etc.)

What, interpreter writers? If they forget to implement them, they’re doing it wrong. The code for checking a gestalt can be used internally by the interpreter to set the header flags. I feel like asking a game author to start messing around with header flags and gestalt depending on which information he needs about the interpreter (especially when writing a game which requires 1.2 functionality) makes things needlessly complex.

[Marvin (David Fillmore)]

While reading through various parts of the Standard for Research last night, a thought came to me. Can interrupt routines interrupt interrupt routines?

This has been a potential issue since 1.0, with three different ways to have an interrupt routine called, but with the introduction of multiple sounds playing at once in 1.2, conflicts are much more likely.

I suspect that the current answer in most interpreters is ‘yes, interrupts interrupt interrupts’, but I also suspect that the answer should be ‘no, the interpreter queues them up’. Coding a game while worrying about multiple levels on interrupt sounds like a terrible way to spend one’s time.

Thoughts? Should this be made explicit in previous versions of the standard?

[Marvin (David Fillmore)]

@Dannii

I still think it would be wise to document the zoom opcodes.

The Zoom opcodes are currently in the ‘user-defined’ range. If we made them Standard, we’d have to move them. Are they worth making standard? They look to me to be not particularly useful for game writers, more for testing purposes for interpreter writers.

For those who don’t know what the Zoom opcodes do:

[code]start_timer (EXT:128) (no arguments, neither branch nor store)
This makes a note of the time this instruction is used. Normally this
will be the CPU clock time returned by clock().

stop_timer (EXT:129)
This makes a note of the time this instruction is used, storing it
seperately from the time marked by start_timer.

read_timer (EXT:130) (store)
This stores the difference between the start time and end time, as
defined by using the instruction above, in the variable. This time is
in centiseconds.

print_timer (EXT:131)
This displays the difference between the start time and end time as a
decimal number of seconds, to centisecond precision.
[/code]

[Marvin (David Fillmore)]

@Dannii

So far, as far as I’m aware, no attempt to make a 1.2 Standard has been widely agreed on, and no 1.2 Standard exists.

This is unfair - up until now the 1.1 standard was just as much a draft. I’m not even sure if anyone had used 1.1 features yet.

This isn’t actually the case. The 1.1 standard was argued over for years, and eventually the contentious bits were stripped out and it was called final. Admittedly, this was not well announced to the world, so it’s hardly surprising nobody knew it was finished. This is also why I have a problem with the ‘Nobody used 1.1, why bother with 1.2’ argument. Nobody knew 1.1 was finished. I was subscribed to the Z-Machine Mailing List where it was being discussed, and I didn’t know it was done until I asked about it at least a year after it was finalised.

That said, no, 1.1 has been the current standard, agreed upon (or at least not argued about anymore) by everyone who was involved in the discussion at the time, since 2006.

The proposal you put together seems to have not reached any consensus at all. There’s a lot of comments in the thread about it that say ‘I don’t think this should be a gestalt selector’, but they’re still there. The Z-Machine Standard is basically what everyone who cares about it agrees it to be. We have yet to agree to a 1.2 Standard.

@Dannii

You can’t change the meaning of gestalt selectors that are already in use. Parchment shipped with @gestalt a couple of years ago, and its selector 1 is for the standard revision. Having an interpreter revision number is useful - make it 2, or something else.

I might be convinced that it’s worth making a note of these gestalt selectors, but I have a serious problem with ‘I put this non-standard behaviour in a interpreter, so now you have to put it in a standard’. I put Unicode Strings in my interpreter. Then it was removed from the 1.1 Standard. Now it’s back in the 1.2 Standard, but completely changed so that my code is useless. But that’s fine, because it was never standardised. (Admittedly, my interpreter still doesn’t fully work, but it was still a pointless waste of effort on my part)

Now, if it’s in actual games that are out there, that’s more important. It’s still non-Standard behaviour, but we do want to avoid breaking games if we can. Is it?

[DavidK (David Kinder)]

@Marvin

The Zoom opcodes are currently in the ‘user-defined’ range. If we made them Standard, we’d have to move them. Are they worth making standard?

No, I think they should just be ignored. While I have my doubts about how much use any 1.2 Standard will be, I definitely don’t want to see it as a list of random extensions added to random interpreters.

[Dannii]

Marvin, I’m not saying anything should necessarily be included in the final standard, but that when we are taking about anything of effectively unlimited supply, we not reuse anything for a different purpose, even if it was only in a barely considered draft. So this applies to both selectors and streams, though opcodes could be argued to be limited. If we want to reach consensus then you’ll need to provide good arguments why a fairly stable draft implemented both in terps and games should be just dismissed. I do thank you for this work on the standards, but you can’t unilaterally decide to make incompatible changes.

And when I say the depreciated selectors need to be recorded it could just be in a html comment.

Sending data to the screen and terp: the simplest way is to only send directly to the terp and the terp can then send to the screen.

Signalling to process or change type: simplest way for terps is for the author to close the stream and reopen. This is slightly messy for authors, but considering its likely to be done in library functions I wouldn’t worry about authors being put too far out.

Requesting data: could be done by opening and immediately closing a stream. Not ideal, but again, it will be abstracted away by a library.

Any further opcodes will need selectors. Why not include one for font size for consistency’s sake? Zarf always does for Glulx.

[Marvin (David Fillmore)]

@Dannii

Sending data to the screen and terp: the simplest way is to only send directly to the terp and the terp can then send to the screen.

I don’t see the point of this. We don’t want to send everything to the screen that we want to send to the terp. We don’t want to send everything to the terp that we want to send to the screen. We’re using output streams. Multiple output streams can be open at once. Why not use this functionality that’s already in place to decide what gets sent where? It’s the whole point of output streams.

[Dannii]

The screen and transcript stream is like that, but the memory stream is exclusive. Some uses of an interchange stream, like javascript, must also be exclusive. If you want a multi purpose data interchange stream then it should be exclusive. Maybe there could be a second nonexclusive multi purpose stream for other uses, though I suggest that only get planned when people come up with use cases for it…

The alternative is to have to close and then open the screen stream all the time. Hijacking streams like memory and javascript are just so much nicer though…

[Marvin (David Fillmore)]

@Dannii

The screen and transcript stream is like that, but the memory stream is exclusive. Some uses of an interchange stream, like javascript, must also be exclusive. If you want a multi purpose data interchange stream then it should be exclusive. Maybe there could be a second nonexclusive multi purpose stream for other uses, though I suggest that only get planned when people come up with use cases for it…

The alternative is to have to close and then open the screen stream all the time. Hijacking streams like memory and javascript are just so much nicer though…

For javascript output you don’t want to send most of the text to the screen, sure. But this isn’t just for javascript. It’s quite likely that people will end up sending something like room descriptions to both the screen and the interpreter.

You seem to be asking for output stream 5 to stop output to the screen, and then have the interpreter send output back to the screen without using output stream 1. I still say they shouldn’t be exclusive. If you don’t want output to go to a stream, close the stream. For something like javascript, we can have the interpreter store it all up in a buffer until a second identifier is seen that tells it to run the code. That way it can handle the output stream being opened and closed a lot, and even being used for different formats of data in between.

[Marvin (David Fillmore)]

Draft 4: frobnitz.co.uk/zmachine/1.2/draft4.html

Changes in Draft 4:
Removed the extra input stream. Output stream 5 now allows the interpreter to write to a table in memory.
Added a gestalt selector for font_size.
Changed the ‘output stream 5 identifier’ format.
Added the ability for @gestalt to check any input or output stream to see if it is in use.
Clarification for multiple interrupt routines.

[JCZorkmid]

@Marvin

While reading through various parts of the Standard for Research last night, a thought came to me. Can interrupt routines interrupt interrupt routines?

Out of curiosity, has anyone examined what Infocom’s interpreters do? There are a few of these combos we should be able to achieve in the various versions, right?

[Dannii]

Unicode strings: how would everyone feel with restricting them to @print_addr and @print_paddr and not allowing them for @print and @print_ret? The reason is that it would make disassembling considerably more complex. Currently you can just skip ahead until after a stop bit, whereas with unicode strings you would need to process the whole string right then. Perhaps this is only a problem for a subset of interpreters, but for those like my ZVM it’s a huge problem.

Streams:

I thought it would help to present two use cases.

The first is a javascript eval stream. Text is sent to the stream which must be evaluated all at once. It should not be duplicated to the screen. Any data could be returned, including binary data.

The second is a future version of Juhana’s transcript analyser system where the game tags output as library messages or parser errors or other types of things. Probably each of these would need their own identifier. The text should be duplicated on the screen. It could be sent intermittently.

Identifiers: what does sending a sequence of unicode characters mean? In which encoding? Remember you have to send it using the print opcodes I suggest xscii identifiers should be fine. Or numerical identifiers. (Numerical identifers could even be sent through an additional operand to @output_stream, so that the terp can figure out what to do with it immediately, rather than waiting for the first data to be printed to it.)

The two use cases I presented above will need very different behaviour. I don’t think it will be feasible for the interpreter to always understand what each stream is like. This could be because some terps might use a plugin system, while others might have a VM-IO division (similar to Glulx-Glk) so that the VM doesn’t even have to know what each stream is for. So I’d suggest that two different multi use streams be used, one which shares with other streams, and one which doesn’t. The one which shares could transmit the data intermittently, while the exclusive one would not. (And again I’d suggest streams 7 and 8 rather than 5.)

Stream 3 must take precedence over every other stream, because Inform uses it all the time in ways which the author may not be able to predict.

Are there good reasons to only allow one use of stream 5 at a time? If an author was using it for some new purpose but also used it for tagging error messages, I don’t see why the first stream should always be closed - they might find it useful for both streams to get the text. Though actually this makes it tricky, because when there are multiple memory streams only the most recent gets data… if there were multiple sharing stream 5s should they always all get the data? Maybe it would be best for the VM to ask the plugins or IO system to specify exactly how each stream should be handled… thoughts?

Font size: What do you think about specifying that interpreters should take @font_size 1 to mean an increase of 10%, or as close to that as they can get?

[zarf (Andrew Plotkin)]

how would everyone feel with restricting them to @print_addr and @print_paddr and not allowing them for @print and @print_ret?

That makes life easier for your interpreter but harder for the compiler. Possibly this is the right tradeoff, but I’ve just spent a few minutes trawling through the compiler and it’s not obvious how to guarantee this.

The cheap solution is to rig the I6 print/print_ret statement to never use the @print/@print_ret opcodes, when Unicode strings are enabled. (It would instead always generate @print_paddr, plus maybe @new_line @rtrue.) This would cost additional size.

[Dannii]

How much size would it cost? Wouldn’t it just be a word for the address? The text data itself would be the same size I think? Oh, there’d also be padding bytes…

[zarf (Andrew Plotkin)]

I don’t know the opcode encoding size offhand, but yeah, padding bytes – average 4 per @print in a V8 game. Inform code goes through a lot of @print and @print_ret opcodes.

I dunno. It should be possible to scan the output text and decide whether it will generate a Unicode escape. I haven’t even really thought about how that decision would be made.

[Marvin (David Fillmore)]

@Dannii

Identifiers: what does sending a sequence of unicode characters mean? In which encoding? Remember you have to send it using the print opcodes I suggest xscii identifiers should be fine. Or numerical identifiers. (Numerical identifers could even be sent through an additional operand to @output_stream, so that the terp can figure out what to do with it immediately, rather than waiting for the first data to be printed to it.)

It just means ‘you’re not limited to ASCII or Latin-1’.

As you say, the game is printing these characters using normal print opcodes. There are multiple ways to print Unicode characters, and the game has a few different ways to encode the information (most of which aren’t Unicode encoding, but that’s not important). Before the interpreter looks at the string of text to identify it, it has to decode it from ZSCII, no matter which characters are being used.

[Marvin (David Fillmore)]

@Dannii

Stream 3 must take precedence over every other stream, because Inform uses it all the time in ways which the author may not be able to predict.

This is not a problem. I haven’t mentioned this in my 1.2 proposal because the current spec already says

7.1.2.2

Output stream 3 is unusual in that, while it is selected, no text is sent to any other output streams which are selected. (However, they remain selected.)

It will still say that when the new bits are added. That covers stream 5 as well as all the current streams.

[Marvin (David Fillmore)]

[quote=“Dannii”]
Unicode strings: how would everyone feel with restricting them to @print_addr and @print_paddr and not allowing them for @print and @print_ret? The reason is that it would make disassembling considerably more complex. Currently you can just skip ahead until after a stop bit, whereas with unicode strings you would need to process the whole string right then. Perhaps this is only a problem for a subset of interpreters, but for those like my ZVM it’s a huge problem.

Streams:

I thought it would help to present two use cases.

The first is a javascript eval stream. Text is sent to the stream which must be evaluated all at once. It should not be duplicated to the screen. Any data could be returned, including binary data.

The second is a future version of Juhana’s transcript analyser system where the game tags output as library messages or parser errors or other types of things. Probably each of these would need their own identifier. The text should be duplicated on the screen. It could be sent intermittently.

The two use cases I presented above will need very different behaviour. I don’t think it will be feasible for the interpreter to always understand what each stream is like. This could be because some terps might use a plugin system, while others might have a VM-IO division (similar to Glulx-Glk) so that the VM doesn’t even have to know what each stream is for. So I’d suggest that two different multi use streams be used, one which shares with other streams, and one which doesn’t. The one which shares could transmit the data intermittently, while the exclusive one would not. (And again I’d suggest streams 7 and 8 rather than 5.)

I don’t think they do need very different behaviour. The interpreters identifies the format of the data, stores the data in a buffer until it’s ready to be processed, and then processes it.

If you don’t want to share the data, you turn off the other streams. Why is that difficult?

[Marvin (David Fillmore)]

@Dannii

Font size: What do you think about specifying that interpreters should take @font_size 1 to mean an increase of 10%, or as close to that as they can get?

I think we should leave it as vague as possible. The interpreter gets to specify size changes in whatever way is simplest for that interpreter, and the game can check the exact dimension of the new font size and then just cope with it.

This does mean that z5 interpreters would have to signify the difference (in unit sizes) between the different font sizes. Most z5 interpreter currently set the unit size to be the font size, so fonts have a width of 1 and and height of 1. If the interpreter allows font size changes, it would probably have to switch to reporting fonts in pixels.

This is the sort of thing that traditionally header bits are used for. That is, telling the interpreter up front ‘hey I’m gonna do something that means you might want to change your behaviour’. Hmm.