Continuing the discussion from Do constant lists still use a lot of memory?:
This, in part, is an unfortunate consequence of compiling down to Inform 6, which puts all user-defined arrays in Z-machine RAM.
Infocom put a lot of theirs in Z-machine ROM, which ZIL supports with the “pure” flag on table definitions.
Is there any way to make a constant array/table in I6, which will get stored in ROM? It seems like it would help quite a lot with making I6 projects fit within the Z-machine limits.
I’d like to find the answer to this too – and whether it can be done with the Library or if compiler hackery is necessary.
This would need a compiler change.
See construct_storyfile_z, in tables.c: static memory (ROM) starts at the grammar table, but arrays are written before then.
How hairy would this change be?
I wrote up the idea a while ago: http://inform7.com/mantis/view.php?id=1354 . I don’t think anyone’s worked through the implementation beyond that writeup.
It’s somewhat hairy. Roughly: (a) add a “static” keyword to the Array directive lexer; (2) update the array handling to keep two array blocks, backpatch them both, and write them both to memory in the appropriate place.
Here’s the problem, though. If I recall Z-code correctly, there’s no opcode to access array data using a “packed” address. This means that even if you put arrays in ROM, they still have to fit inside the 64k addressable area. So you’re not actually avoiding the “running out of RAM” limits, which are actually 64k addressable data limits.
(I recall some discussion of whether you could use @loadw $FFFF x to read values from an array that starts in addressable memory but extends beyond it. But interpreter support was mixed at best. Even if it worked, it’s a pretty brutal hack, and using it for more than a single array would be a hairball.)
Zarf remembers Z-code correctly. (no surprise there)
Since the Z-machine standard does not require interpreters to support arrays that are entirely or to some extent in memory > 64 KB, there is no easy and safe way to add it and expect interpreters to handle it.
New instructions for reading a word (and maybe a byte) from an array which starts at a given packed address would probably be the only reliable solution, but these new instructions should then be included in a new version of the Z-machine standard, either (a) making them available in z5-z8, or (b) introducing them in z9 (which was discussed many years ago but was never completed). And then wait for interpreter support.
If someone was truly memory starved they could use memory streams to write arrays into ram from routines in higher memory.
But while there are definitely improvements to I6 that could be made (no shift, xor, or ternary operators! Using opcodes as expressions), and while we can continue to extend the Z-Machine (my gestalt plan makes this easy), I don’t really think this one would be worth it. The Z-Machine really only serves for nostalgia purposes now. Even the one advantage it had over Glulx, text formatting, is being eroded away.
Are you talking about the same kind of thing mentioned above that Infocom’s ZILCH and now ZILF support, but Inform6 does not?
I’ve toyed with the idea of using a jump with a runtime-computed parameter, to jump into a field of instructions. This would allow random access to the data. For instance, a field of ret instructions with long operands would have an encoding overhead of 50% (three bytes of code to hold two bytes of data).
A raw byte in a packed Z-string requires at best 5.33 bits, at worst 21.33 bits, and on average 17.47 bits per byte (assuming a rectangular distribution of bytes, and no use of abbreviations) which leads to an average overhead of 118% (worst case 167%). If the data is encoded using only lowercase letters, it is possible to bring the overhead down to 13%, but the numbers have to be converted back to bytes using expensive multiplication instructions. A faster option would be to use the lowercase letters plus six uppercase letters, to get 32 code points with an average overhead of 26%.
To combine efficient bit stuffing with random access, we could use a field of print_ret instructions. By adjusting the length of the string literals, we have full control over the tradeoff between random access and encoding overhead. We would have to use a constant bit-length encoding, e.g. lowercase letters only. For instance, with 4-byte literals (plus one byte for the instruction), we can encode six lowercase letters for every five bytes. That would give us random access to a table of 28-bit values, corresponding to an overhead of 43%. With 8-byte literals, we get random access to a table of 56-bit values, and an overhead of 28%.
I agree with the mootness of expanding the Z-Machine anymore, but what I want to know about is what seems to me like something ZILCH and ZILF did/do that Inform6 doesn’t.
Inform doesn’t let you say “this array is a constant and can be placed in static memory”. I assume because there’s so little benefit to doing so now, except for those targeting the Commodore 64 or another machines with only 64kb ram.
The parent discussion was about Inform 7, where there would be value to have fixed size arrays, because flexible lists have a lot of overheads.
I think it’s worthwhile to investigate how to mark an array as constant to benefit those who are trying to cram modern games into older computers.
These discussions come to mind:
Then there’s the renewed interest in PDP-10 emulation spurred by the recent revelation of Infocom’s ZIL source code.
@lft: Could this be done with just a small library for Inform 6, or would it require changes to the compiler? I have a feeling it’s the latter.
@DavidG: Agreed.
To me, playing IF on the Commodore 64 is a better option than on a PC, unless the game gets terribly slow on the C64. For one thing, I don’t have to search for ways to turn off all notifications to get some peace and quiet, and not have a messy background with windows, icons etc. Also, the text is printed at a speed where I can follow what’s happening (probably about 10-15 lines/s at most). On a PC, whenever there’s been a MORE prompt and more text is printed, I need to go back and search for the place where the newly printed text starts.
And considering the great games Infocom released for the C64 and similar computers, and the interest we still see in these computers, I see no reason why they shouldn’t be considered as viable platforms for IF today. The modern-day tools may need some adjusting to produce games that fit these platforms better though.
I assume because there’s so little benefit to doing so now, except for those targeting the Commodore 64 or another machines with only 64kb ram.
The other benefit of moving arrays into static space is smaller save files and smaller undo checkpoints.
This is not completely moot for Glulx. (You recall that the original release of Counterfeit Monkey had some trouble with interpreters that didn’t expect such enormous undo checkpoints.) However, even if the I6 compiler supported the feature, it would take more work to take advantage of it.
It’s an idea with some value, but it’s never looked like it’s had enough value for anyone to sit down and crank it out.
If it doesn’t need to be addressable, you could rewrite the array as a routine that returns the value at a given index, or copies part of the array to a given address, etc. For instance, if you have a block value that’s going to be copied into RAM as soon as it’s used anyway, you could move its initial value into unaddressable high memory as a function that allocates and initializes the value.
There is one other benefit: correctness. Putting constants in ROM means that if you accidentally overwrite them, you’ll get an immediate runtime error instead of a subtle bug down the road.
There are a few other things Inform 6 can’t do that ZIL can, like mix bytes and words when initializing a table (or a long property), and create properties whose length is an odd number of bytes.
I somewhat suspect that could be causing the ongoing trouble with feminine pronouns that I thought I fixed.