Strange behavior at memory cutoff of 32k;

Authoring Inform 6
1

I’m building a game in Inform6 (though not sure if this is related) with PunyInform (same).

I’m planning on releasing as a slightly-trimmed z3 game and the regular z5 game. The z3 version works fine.

The z5 version started responding to “X SOMETHING_NOT_IN_DICT” with “A look in that direction reveals nothing new”, which is the response for looking in a direction. Turning on actions & routine tracing, it seems it thinks the noun is indeed the Directions object.

But: it appears the problem is memory. With some binary-search-debugging, I found that that when I comment out enough text that my “readable memory” dipped below 32k, the behavior went away. Zooming in, with 32764 bytes, it does the expected thing “I don’t know the word ‘ZZZZ’”), but with 32768 bytes, it does the “I think you’re using the Directions object”. It’s not related just to examining, other things like Eat tried to take the directions object.

So, it seems like there’s some tripwire on memory use I’m hitting. I get the memory info on compilation using “inform -s”, and I when it hits 32k or more, the problem starts.

32768 bytes readable memory used (maximum 65536) ←- doesn’t work
32764 bytes readable memory used (maximum 65536) ←- does

Here’s the full output of compilation with -s

In: 17 source code files 15067 syntactic lines
23049 textual lines 693909 characters (ISO 8859-1 Latin1)
Allocated:
2700 symbols 1094473 bytes of memory
Out: Version 5 “Advanced” story file 1.260307 (149K long):
26 classes 327 objects
146 global vars (maximum 233) 2707 variable/array space
116 verbs 777 dictionary entries
253 grammar lines (version 3) 437 grammar tokens (unlimited)
126 actions 31 attributes (maximum 48)
33 common props (maximum 61) 61 individual props (unlimited)
95076 bytes of Z-code 968 unused bytes stripped out (1.0%)
139999 characters used in text 78922 bytes compressed (rate 0.563)
96 abbreviations (maximum 96) 799 routines (unlimited)
12426 instructions of Z-code 7606 sequence points
32768 bytes readable memory used (maximum 65536)
152544 bytes used in Z-machine 109600 bytes free in Z-machine

I’m not sure what layer to begin more debugging here. I’m sure there must be z5 games that use more readable memory, so I very much doubt I’m hitting any kind of zmachine format limit. It happens in the same way in a few different interpreters.

Any advice on where to dig in would be very welcome.

1 Like
2

Just to add: I said the z3 version worked, and that’s well below the 32k cutoff (part of what I #IfDef out from the z3 version is hints and such, so way less text)

3

I am deeply amused, because I fixed this same bug in Dialog last year! The Z-machine’s low-level numerical comparison opcodes are all signed, which means numbers above $8000 (decimal 32,768) are considered negative. Somewhere in the library, memory addresses are being compared with the @jg etc opcodes, or the Inform 6 > etc operators (which compile to @jg etc).

In Dialog’s case, this comparison was to determine whether an entry in a particular table was an object or a memory address, by comparing it against the maximum object number. Addresses above $8000 were treated as negative, and thus interpreted as (illegal) objects instead. I imagine PunyInform is doing something similar, though I don’t know where to look to find it.

3 Likes
4

Negative numbers were a bad idea.

5

@fredrik : may I ping you in as an expert on the Puny internals? Might that be a layer I should investigate?

6

A quick search through puny.h finds that _RoomLike and _ObjectLike both contain a comparison if(p_obj > Directions), but I don’t think either of those should be called with raw memory addresses. PunyInform does supply an UnsignedCompare routine, though, so if you can find which comparison is going haywire, swapping in UnsignedCompare there will fix it. (Or just check if the value is less than 0, in which case it’s not a valid object number; that’s what I did for the Dialog runtime.)

7

It might also be useful to print out a full memory map (-z). It looks like only two bytes of addressable memory have to go above the mark for the bug to show up. What does Inform put at the very end of addressable memory? That’s probably the thing being compared when the bug happens. I suspect dictionary words?

I checked if the address dict_end (the end of the dictionary table) was being compared in a signed manner anywhere, but everywhere it appears, it seems to use UnsignedCompare.

8 
        +=====================+   05e66
Readable|    grammar table    |
memory  + - - - - - - - - - - +   0653a
        |       actions       |
        + - - - - - - - - - - +   06636
        |   parsing routines  |
        + - - - - - - - - - - +   06640
        |     adjectives      |
        +---------------------+   06696
        |     dictionary      |
        +---------------------+   07ee5
        |    static arrays    |
        +=====================+   08048
Above   |       Z-code        |
readable+---------------------+   1effc
memory  |       strings       |
        +---------------------+   26088

It looks like it cuts off in static arrays. I’m starting to read through the puny.h and parser.h files to see if anything jumps out.

9

Now we’re getting somewhere! There aren’t many static arrays in PunyInform, and this one looks the most promising.

10

With the tracing on, when it fails, it thinks it’s acting on the Directions object with chosen_direction=’in’. I’ll read through that part!

11

I can rule out cheap_scenery. That uses arrays and the object it makes it always in scope, so its parsing routines weren’t working, it might find the wrong thing. But: I commented out the inclusion of it (so no scope-object added) and it still gives the strange behavior > 32k.

12

If you edit your copy of puny.h and remove the static modifier from that array, does that fix the problem? That would push it earlier in memory (into RAM, well away from the $8000 mark). If that fixes the problem, that array is the issue.

13

It doesn’t change the behavior. I’ve got a bunch of static arrays; my box-quotes are all defined as such (to reduce dynamic memory). But I don’t see they would be related. Or am I not understanding ZMachine memory?

14

There are a lot of unsigned-comparisons in the puny code. Some of them look very innocent (comparing a variable to the length of an array, etc), some are deep in internals I don’t understand.

I’ll keep reading. Thanks so much for your ideas, @Draconis !

15

Found the DEBUG_XXX constants for the parser and compiled with all of them. Now, “X FOO”, gives this (trimmed)


x foo
Verb#: 15, meta 0.
Grammar address for this verb: 25290
Number of patterns: 1
############ Pattern 0 25291
Action#: 64 Tokens: 1
Token#: 1 Type: 1 (top 0, next 0, bottom 1) data: 0

PHASE 1: Pattern 0 address 25291

Pattern: 25291
TOKEN: 1 wn 2 _parse_pointer 23920
Calling ParseToken: token 1 type 1, data 0
*** Call to UpdateScope for yourself
_PutInScope adding (Directions) (5) to scope. Action = Examine

… added lots of things to scope

*** Updated scope from the Dorter. Found 20 objects.
Calling _ParseNounPhrase(23920);
Entering _ParseNounPhrase, first word is [Unknown]
Testing the direction…
New best result: word count 1, score 300
Testing the object…
Testing Brother Benedict…
Trying to find oughx  tioncg in name (length 4).

^ “oughx  ctioncg” is gibberish; I’m guessing this is showing it’s
already gone wrong?

Testing Brother Hugh…
Trying to find oughx  tioncg in name (length 4).

…

Matched a single object: the in, num words 1, wn 2

^ here’s where “in” come from

A look in that direction reveals nothing new.

^ And there’s the wrong output
16

That definitely looks like the parse_name routine on the Directions object is what’s malfunctioning. Do you use OPTIONAL_FULL_DIRECTIONS?

17

Also, does this bug show up in multiple interpreters? My current suspicion is that your terp’s @scan_table opcode misbehaves when it crosses the $8000 mark. The parse_name in question doesn’t make any direct comparisons of its own, but it does rely on @scan_table doing the right thing for an array that crosses that position in memory.

18

No extended directions for this game, and no ship, either.

Dunno if the probably is in parse_name for directions, though – that garbled thing it’s trying to match (“oughx ctioncg”) makes me think it’s gone off the rails even before it just testing things.

19

I doubt its the interpreter – I’m using Frotz, and get the same behavior on Spatterlight and Gargoyle.

20

This is the bit that I’m looking at:

Calling _ParseNounPhrase(23920);
Entering _ParseNounPhrase, first word is [Unknown]
Testing the direction…
New best result: word count 1, score 300

That says it’s calling parse_name on Directions (“testing the direction”), and the word it’s trying to match is not in the dictionary (“[Unknown]”). That all sounds like it’s working fine to me.

Except then, the Directions object matches one word worth of input (“New best result: word count 1”)! It shouldn’t be doing that. And in the process it corrupts the input buffer and leaves it in a messy state.

The array that’s being consulted here is from line 310 of globals.h. (I think I pointed you to the OPTIONAL_FULL_DIRECTIONS one before; this is the one that matters without OPTIONAL_FULL_DIRECTIONS defined.) If you make that array no longer static in your copy, does that fix the bug? That should push the parsing array down into RAM, well away from $8000.