I feel that the proposal would benefit from a “Why improve Z-machine?” section. It should answer questions like:
The most basic feature of a VM for interactive fiction is always going to be the support it has for text input and output. If we’re going to expand the abilities of the Z-Machine, surely adding support for a wider range of languages and characters is a worthwhile goal? If we’re going to support some of Unicode (which we already do), why not all of it?
I also wouldn’t say that easy support of more Unicode characters is anywhere near the main reason for 32-bit IF VMs.
Hmm. That may be worthwhile. As a quick attempt to answer these questions here:
Why not abandon the Z-machine in favor of Glulx?
Zarf is the benevolent dictator for life of the Glulx Standard (which is fine). The Z-Machine Standard, on the other hand, is (beyond the need to be compatible with Infocom’s Z-Machine) rather more of a community effort. Improvements to the Z-Machine have always been by consensus. Improvements to Glulx are by consent of Zarf. I like having both options.
If Glulx is missing something important, why not improve Glulx?
First, see above. Second, Zarf will never agree to add every feature of the Z-Machine to Glulx. Finally, in this anarchistic hobbyist community we have, we all put our effort into those areas we are personally interested in. For me, this is the Z-Machine, not Glulx.
Which interpreters will implement the specified features?
There are already interpreters that use streams 5 and 6, so partly this update is just bringing the Standard into line with what interpreters already do. Aside from that, the gestalt feature will make all the other features optional, so that interpreters can pick which features they want to support. Viola (my interpreter) will, I hope, support everything (except I have no use for streams 5 and 6). I imagine Frotz will be updated to support at least some features. Borg323 seems fairly enthusiastic about adding at least the better Unicode support to jzip.
Frotz seems to be doing fairly well with Unicode implementation. There are ongoing problems getting it to work with NetBSD.
I am also, unrelatedly, the chairperson of the committee that supports the http://inform-fiction.org/zmachine/standards/ site. So once there’s general public agreement on a 1.2 document, I can post that.
It sounds like discussion is still in progress?
Discussion is still in progress, yeah. If it ever comes to an agreement, I’ll put together a full 1.2 Standard document and let you know.
Okay, so my latest idea is: All high surrogates at the end of the main Unicode Translation. Directly after the main Unicode Translation table, 1 byte giving the number of further words (which must be the same as the number of high surrogates at the end of the main table), followed by that many words of low surrogates.
This will save a whole 1 byte on putting the number of high/low surrogate pairs in the header, and keeps all the information together, which I like.
Yes, this seems clean enough, provided a bit in flags 3 is allocated to indicate this table format is used.
It’s taking significantly longer than I expected to put together another draft. I’m rewriting a bit and adding some new features I wasn’t expecting to. I’m still working on it, though (although I’ve been a bit distracted by another project).
I’m also looking at the improved file access feature I proposed, and while I’d really like something like it, I don’t much like the way it’s designed in this proposal. Unless anyone else cares enough about this feature to change my mind, I’m probably going to drop it from the next draft of the proposal.
Any progress on the draft?
I started a new job recently, and I’ve not had much time to work on this. I do hope to get back to it soon, though.
I am currently writing a new z-machine implementation in Rust that I will release publicly when it is a little further along. Not an interpreter, but rather a reusable library that encapsulates as much z-machine behavior as possible. This isn’t my first z-machine implementation. I tend to write one whenever I learn a new language as I find it is a good way to become familiar with a language’s features and idioms. I do like the idea of a 1.2 standard to both extend and clarify the z-machine.
My opinions on the draft are below (I haven’t read all of it in depth, so more opinions may follow later):
Non-BMP unicode - I don’t have an issue with this, although I haven’t looked at it in detail. It may be more difficult for non-utf16 based interpreters to implement (non-Windows OS’s tend to use utf-8, so that’s something to keep in mind).
Sound - Are more channels really needed? Also, I dislike the current effect/music split and multiplying that again by channels seems really ugly. This seems more suited to a hypothetical V9 that could unify the two approaches to playing more sound.
Gestalt opcode - This doesn’t fit the z-machine at all. A header extension table for just the new features would be more z-machine-like. Also, providing a duplicate way to check features for which header flags already exist is redundant and I strongly dislike redundancy - what do you do if they don’t agree through interpreter bug, or otherwise? An example of what this leads to is the transcript bit in the header, plus the output_stream opcode. No thanks.
Interrupt queuing: While this makes sense, my past interpreters never behaved this way, so you may generate games incompatible with existing interpreters. A queue implies memory allocation, so a maximum value should be enforced, with provisions as to what happens when it is exceeded (a missing timeout interrupt could be game-breaking.)
Other features: haven’t read all the details yet, although standardizing print_table seems worthwhile.
Additional comments:
While an update to the standard to add new features is great. I think it would also be useful to more deeply examine some currently undefined behavior as well. I’ve compiled a list of some while working on my interpreters over the years. If anyone thinks a separate thread listing and discussing z-machine UB is worthwhile, I can post what I have.
Also, I think conversation regarding a V9 is worthwhile. For example from the remarks on section 12 (the object table) of the standard: “Bit 6 in the second byte is presently wasted, which is a pity as it could be used to allow up to 128 bytes of property data. But such a change would cause Infocom’s story files to fail (since they set this bit, unlike Inform story files).” This is the perfect candidate for a V9 feature, vs. purely a standard extension.
I think it would be!
It’s been a while since I looked at my own proposal, but I think there’s UTF-8 and UTF-16 used in different areas, just to make things fun for all interpreter writers.
I very much want more sound channels, yes. This is one of the main features I wanted in the spec. I also dislike the current effect/music split, but I lost that argument a long time ago. A hypothetical V9 would probably have a cleaner approach.
The problem with more header extension is it takes up space in the z-code file memory. As for redundancy, yeah, it’s not ideal, but if the header information and the gestalt don’t agree, it’s always an interpreter bug, and it means one of them is wrong. The answer to ‘What if there’s a bug in the interpreter?’ is ‘the game does its best, and also please fix the bug’.
Some good points, which I shall have to think about.
This would be useful, yeah.
While I do want to make a V9 at some point, it is probably going to be a difficult discussion, with differing opinions as to what it should be (or if it should exist). It’s something I hope to tackle one day, but not before 1.2 is done.
That’s true regarding the memory. As for “bugs”, I was thinking more along the lines of interpreter A uses only header flags, B uses only gestalt, C uses both but gets the synchronization wrong somewhere. Compound that with Game A looks only at header flags, B only uses gestalt…I don’t like where that goes.
How about a gestalt only for things which don’t already have mechanisms and leave the others alone? I understand the urge to have a single clean standard interface, but for the Z I think that boat sailed, sunk, and was salvaged years ago. I feel the need to link the required xkcd: xkcd: Standards
I’ll pull together some of the UB I’ve noticed together and put it in another thread.
My idea with the introduction of the gestalt system is that it replaces the header flag system, but the header flags that already exist are still required to be set correctly for backward compatibility. Theoretically, the header flag system would be discouraged for use by newer games, but in practice, if your game doesn’t need the features provided by the 1.2 Standard, it’s safer to avoid using gestalt.
In your examples, interpreter A is fine, but isn’t a Standard 1.2 interpreter, interpreter B doesn’t match any Standard, and should probably be considered buggy, interpreter C is just a buggy 1.2 Standard interpreter.
Game A should work fine on any Standard interpreter, game B will require a 1.2 Standard interpreter (and should check that it’s running on one before using gestalt).
That sounds reasonable. I would not oppose a gestalt opcode if others think it is OK.
I’ve been thinking about the idea of interrupt queueing and combining it with the stuff about nested reads in my undefined behavior thread, and the result is complicated. Consider:
It is definitely legal to call a read instruction from a timeout routine (Border Zone). Either it is legal to return from this timeout interrupt or it is not (currently undefined). This gives us two cases to consider when a timeout routine calls a read (or read_char):
First, in the case the routine is not allowed to return, interrupt queuing implies sound interrupts that fire during the original timeout interrupt or during the seocnd read will never be run. The only real use for nested reads in this case is to end a game via quit, restore, or restart instructions as Border Zone does (or throw? See undefined behavior thread, item #9) , because the lower portion of the call stack becomes permanently inaccessible otherwise.
Second, if the timeout routine is allowed to return and fully nested reads are supported, sound interrupts will be hugely delayed by a nested read or read_char.
This is related to another complication: What happens when a sound interrupt writes to the display during a read? It is entirely possible, even likely, that sound interrupts could be called during a read operation. Do any interpreters handle this case properly, redrawing the input as when returning from a timeout interrrupt? I don’t currently have any data on this. As much as it might be tempting to print “Silence falls.” when a sound file stops playing, I fear the result will be unpredictable without clarification in the standard. Sidenote: Also, what happens if a sound interrupt returns a true value during a read? Does it terminate the read in the manner of a timeout interrupt? That’s a bizarre corner case.
These issues lead me to wonder if I/O instructions other than sound_effect (and read_mouse since it is real-time) should be illegal during sound interrupts. No read/read_char, and no printing of any kind (except if stream 3 is active?). This prevents a lot of complicated interaction between sound and timeout interrupts, and V6 newline interrupts as well, but leaves open the issue of keeping too many simultaneous sound interrupts from causing the loss of a timeout interrupt or newline interrupt, due to a full queue. Edit: Disallowing sound interrupts from doing I/O certainly is in keeping with the spirit of standard section 9.4.4
Edit: Perhaps newline and timeout interrupts can fire immediately and only sound interrupts get queued? That would work as long as sound interrupts can’t trigger the other two types (by printing to the display or reading input).
Section 2.3.2 of the standard: " Formally it has never been specified what the result of an out-of-range calculation should be. The author suggests that the result should be reduced modulo $10000 ."
This wording seems odd. As a suggestion from the author, it lacks conviction.
I noticed in Infocom’s specs that arithmetic overflows are considered an error. The standard makes no mention of them being an error, just reducing the result. I’m not sure if making them a hard error is a good idea though.