Heap space exhausted - solution and why

[Pacian]

I’m running into “(Technical trouble: Heap space exhausted. Attempting to recover with UNDO.)” errors in my current work in progress, and I’m struggling to work out what I’ve done wrong.

The errors occur if you use custom verbs or refer to objects that are out of scope.

The errors don’t occur if I’m using the interactive debugger, only if I compile to z5 or z8 and run the file in WinFrotz or Gargoyle.

Looking at this page in the manual, I added (display memory statistics) to the (error $Code entry point) predicate and can see that when things blow up, the last recorded main heap is about half full - regardless of if I increase the heap to maximum in the compiler options, so that all sounds like an infinite loop of some sort.

But I only started encountering this issue after adding a few more objects to the game. And if I delete any random two to four objects, the error goes away.

My game is doing a bunch of stuff that’s a bit strange (messing with undo, moving lots of NPCs around, short-circuiting lots of verbs for particular objects), but I can remove all that behaviour and the error still crops up. But deleting one particular (on every tick) rule seems to stop it.

Except that rule should fail unless the player is in a room with a particular property, and the error crops up in rooms without that property. Deleting the contents of the if/elseif statements in that predicate seem to fix the issue, but why are they eating up infinite memory? Especially when all the conditions to trigger them are false? Why do they stop eating up infinite memory if I delete a couple of unrelated objects?

Can anyone help me work out what I can do to debug this? I’m not sure what code I could post here aside from the entire source code of my work in progress, but here is the (on every tick) rule that seems to be causing the issue. FYI, “current room” is not “adumbral”.

(on every tick)
    (current room $Room)
    (adumbral $Room)
    ~(#Countess is nowhere)
    ~(#Countess is in room $Room)
	(#Countess is in room $Here)
    (par)
    (if)(#Countess is scuttling)(then)
        Something scrabbles in the darkness, drawing slowly closer.
        (now)~(#Countess is scuttling)
    (elseif)($Here = #QuarantineZone)(then)
        (if)(#Countess faces #west)(then)
            (let #Countess go #west)
            (countess crawls)
        (else)
            (now)(#Countess faces #west)
            (countess turns)
        (endif)
    (elseif)($Here = #CoreBay3)(then)
        (if)(#Countess faces #south)(then)
            (let #Countess go #south)
            (countess crawls)
        (else)
            (now)(#Countess faces #south)
            (countess turns)
        (endif)
	(elseif) (first step from $Here to $Room is $Dir) (then)
        (if)(#Countess faces $Dir)(then)
            (let #Countess go $Dir)
            (countess crawls)
        (else)
            (now)(#Countess faces $Dir)
            (countess turns)
        (endif)
	(endif)

Experimenting a bit more, deleting all the (now)(#Countess faces $) lines seems to stop the issue. But why?! How can I debug this?

[Pacian]

I just noticed this similar topic, and took the suggestion there of adding these lines to my intro:

(intro)
	(actions on)
	(trace on)
	(fail)

And my game is also crashing during the (parse object name $Words as $Result $All $Policy) library predicate.

Which I guess is progress, but still leaves me baffled as to what I can do about.

[mathbrush (Brian Rushton)]

Just completely spitballing here: is ‘faces’ a protected word? If you replace it with a different word, does it function better?

Second thought; instead of a predicate that relates a person to directions, have you tried replacing it with four separate predicates, one for each direction?

Neither of these suggestions would explain what’s going wrong, but they might provide a way to stop the bugs.

[cfmoorz2]

nephar fixed my identical issue in the previous post.

turns out i was simply overtaxing the z-machine with per-object variables.
converting these to global variables fixed the problem.

so, for instance, instead of ($ faces $), create a global variable instead like:

(global variable (countess faces $))

this is actually explained ( i missed it as well, or read through it and didn’t realize the significance) in the dialog manual, chapter 9 under ‘memory footprint’.

[Pacian]

I did see that section in the manual, but it didn’t think it would be related to my issue…

But changing all the (#Countess faces ... to (Countess faces ... fixes the issue… for now, at least. So thank you!

The main thing I took away from that section of the Dialog manual is that you should use global variables where you can:

For instance, looking back at the (#troll wields #axe) example from the chapter on dynamic predicates, if the troll and the player are the only characters that wield weapons, it would be much better to use a pair of global variables

But I really do need to keep track of per-object variables in a lot of my cases.

Still, I’ve tried to find a few more single use per-object variables and change them into global variables. Hopefully that will get me over the finish line, but I had been hoping to expand what I already had a bit,
so it would be nice to know what I should do if I run into the limit again. Is it just a case of “You can’t do that in Dialog/the Z Machine”?

EDIT: this does also make me suddenly realise that I haven’t been declaring my global variables, seemingly without issue? I guess I should try and find them all somehow…

[nephar (Timur Ozturk)]

Given your example above, if you simply change all (#Countess faces <direction>) statements to (Countes faces <direction>) then you would be creating as many global per-object flags as there are directions the Countess can face objects in the game. In most cases that would lead to unwanted behaviour when you set one of them, unless you unset the others. Otherwise Countess would be facing several directions at the same time. If you declare it a global variable and you set it to one direction, the old one would be overwritten automatically, since the variable can only hold one value at a time.
The problem with per-object-variables is that when you use it on one object, all other objects in the game get that variable as well even if you never set or reset them for those other objects. This would start bloating memory usage of objects, especially if you have many of them. Global variables on the other hand allocates same amount of memory regardless of the count of objects in your game.

Edit: Oops, I meant

… you would be creating as many global per-object flags as there are directions objects …

above in the first paragraph. Memory footprint won’t be as bad as per-object variables, but still it is a waste of bits in addition to the hassle of setting/resetting multiple flags to make the facing logically consistent.

[Pacian]

Thank you, that does help me understand what’s going on a bit better.

[Pacian]

So this error has continued to dog me…

I’ve removed the obvious per-object variables I can do without, declared and used global variables in most of the places I can think to and also gutted some parts of the standard library (I don’t have any doors or locks, so I removed that code, for example).

The annoying thing is that it seems to trigger on certain vocabulary words, so I’ll only encounter it when I happen to enter the right command.

Right now it’s triggering if I direct an action at the word “time”. There’s only two objects in the game that have “time” as a vocabulary word, and I can refer to them both as long as I don’t use the word time.

It also doesn’t matter if they’re in scope or not. If I enter the command “eat time” in any room, it triggers the heap error.

I’m sure there are a few more variables that I can try and tweak, but it just feels like the error is always there and I’m just waiting to find the next magic phrase to trigger it…

[Pacian]

Now getting this in WinFrotz and Gargoyle. Again, it seems to apply to the vocabulary word “outfit” rather than the actual underlying object, which can be referred to using other vocabulary words.

Turning on actions and trace seem to stop this from happening, and it doesn’t happen in the debugger, so once again wondering how to debug this.

Do I need to scale back my game? It’s so close to being complete.

[nephar (Timur Ozturk)]

I can look at your code if you are willing to share it with me with a DM.

[cfmoorz2]

if you figure it out, please let us know. having had this problem also, i’m curious how it turns out.

[Pacian]

It has still boiled down to an excess of per-object variables, but things that I couldn’t think of any other way to handle, Timur has been able to turn into global variables. I think I just have OOP brain-poisoning.

For example, I had a few different sets of relationships like:

#SomeRoom
(name *) Site of a Disaster
(look *) Rubble blocks the way north.
(from * go #north to #SomeOtherRoom)
(exit * #north is blocked)

#YetAnotherRoom
(name *) Site of a Different Disaster
(look *) Debris blocks the way east.
(from * go #east to #TheLastRoom)
(exit * #east is blocked)

Timur has re-envisioned them as a series of lists of lists like so:

(blocked exits $)       [[#SomeRoom #north] [#YetAnotherRoom #east]

Where previously I would add and remove these properties, they’re now added and removed from the relevant lists.

In another case, Timur has replaced a per-object variable I was using with some behaviour from the standard library.

(This is the gist of the changes, but it undersells the extent to which Timur has provided code to integrate them neatly into what I was already doing, for which I am extremely grateful.)

[nephar (Timur Ozturk)]

I wish we had more games coded in Dialog to build up a healthy dose of know-how accumulation if for nothing else. I still have to ask for complete source code to be sure of locating the cause of strange errors compared to the old-hands of Inform 7 and 6 in this forum, who can feel the culprit just by looking at a code fragment or even at a description.

That being said, I am now fairly confident to say that if one gets heap exhaustion errors or strange addressing errors like mentioned by Pacian four posts above this, and a trace in a compiled z-code binary shows the hiccup happening in the (parse object name ...) rule from the library, then the cause is an overextend of per-object variables probably.

Edit: In the debugger one can get a quick glance at per-objects variables used by invoking the debugger command @dynamic. Example output snippet from that command:

> @dynamic
...
PER-OBJECT VARIABLES
        ($ has parent $)                        
                #player                         #wellhouse
                #door                           #immense
                #cage                           #cobblecrawl
                #bird                           #birdchamber
                #keys                           #wellhouse
                #lamp                           #player
                #food                           #wellhouse
                #bottle                         #wellhouse
 ...
        ($ has relation $)                      
                #player                         #in
                #door                           #in
                #cage                           #in
                #bird                           #in
                #keys                           #in
                #lamp                           #heldby
                #food                           #in
                #bottle                         #in
...
        ($ abbrev $)                            
                #start                          1
                #valley                         0
                #slitroom                       0
                #cobblecrawl                    0
                #debrisroom                     0
                #awkward                        0
                #birdchamber                    0
                #pittop                         0
                #hill                           0
                #wellhouse                      2
...
        ($ was in $)                            
                #dwarf1                         #southside
                #dwarf2                         #eastbank
                #dwarf3                         #misthall
                #dwarf4                         #alike6
                #dwarf5                         #shellroom
                #dwarf6                         #alike13

Please note that any game starts with two per-object variables already in place, ($ has parent $) and ($ has relation $). They are used to track object tree structure and can’t be represented easily or efficiently in any other way.

The ones added by the particular game above are $ abbrev $ and $ was in $. The former tracks how many times the player has seen a specific room’s description, so that every 5th visit to that room the game prints out the verbose description. It has to exist for every single room the game has. Even though that variable doesn’t make sense for other types of objects like items, scenery, directions and relations and wastes memory for all those other object types, per-object variable use here is the most efficient method to operate on.

The second one tracks the previous location the dwarves were in, and it is a prime candidate to convert to one(like a list if lists) or several global variables(six of them for five dwarves and the pirate). It only shows up here as a per-object variable because the author was lazy .

[cfmoorz2]

these dialog “public service announcements” from nephar are always illuminating.

re: lack of dialog games - i have one up now on the ‘spring thing’ site (“zomburbia”). it has it’s moments. people who like old-school parser games seem to like it.

i’ll also be submitting one for parsercomp and have a smaller one that i originally wrote in inform6 but was never happy with (it was my first try at IF and, while it had some good ideas, it was…not good). i’ve redone it using dialog and am much happier with it now. i’ll probably post it to itch eventually when i have time to set up a page.

[Draconis (Daniel Stelzer)]

@nephar

Please note that any game starts with two per-object variables already in place, ($ has parent $) and ($ has relation $). They are used to track object tree structure and can’t be represented easily or efficiently in any other way.

In fact every Dialog program, even ones that don’t use the standard IF library, has the ($ has parent $) per-object variable in it! The object tree is built into the Z-machine at such a fundamental level that this one per-object variable has to be built into the compiler to produce reasonable Z-code.

That’s also why it’s the only variable to have type checking on it. The compiler won’t complain if you set the relation of an object to the number 53, but it will prevent you from setting its parent to that.

[Pacian]

So I’m seeing this again:

But this time the change that triggers it is adding some vocabulary from:

(dict *(lock $)) lock old-fashioned old fashioned deadbolt

to:

(dict *(lock $)) lock old-fashioned old fashioned deadbolt door

If I remove a couple of words from another (dict *()) construct, then the error seems to disappear, but does this seem right? Should I be adding vocabulary manually to objects instead to avoid bumping up against this limit?

Or is this a symptom of something else that’s going wrong?

Edited to add:
I should say that I’ve not added any per-object variables - in fact I’ve cut it down to only two defined by me. But I have added about 50 more objects apparently, so maybe I’m just at a more fundamental limit?

Edited again:
I guess my concern is: I added one vocab word to a class of objects and encountered this error. If I removed a couple of words from a more numerous set of objects, should I be safe for now?

[cfmoorz2]

it may not be causing the issue but i don’t think you actually need the multiquery there. you should be able to just use:

(dict (lock $)) lock old-fashioned old fashioned deadbolt

[Pacian]

You’re right.

And yeah, it makes no difference to the error.

Edited to add:
Also, it looks like I could delete an object instead of a vocab word to fix the issue, or conversely, adding two new objects instead of two new synonyms causes the “heap space exhausted” error to trigger.

I guess it’s interesting that adding synonyms has had this effect, and I might have been a bit less trigger-happy with adding them if I’d realised, but I guess either way it seems that my game is finally at its limit, and if I want to add more objects or synonyms I’ll need to remove some existing ones to make room first.

(I probably could also remove one more of my two remaining per-object variables, as it only ever has one of 3 values, but I’d be a bit reluctant to make a big change like that after all the testing this thing has already had.)

Edited yet again:

OK, I did go back and try and replace that per-object variable with two per-object flags (confirmed by running @dynamic) and that also causes the heap error to trigger, so I guess I really am at some kind of limit for the size of the game. (And as before, increasing the heap value passed to the compiler has no effect.)

[cfmoorz2]

having repeatedly dealt with this problem i can certainly sympathize. i really wish there was a way to compile dialog to something other than just a-machine or z-machine. like glulx maybe? something that would offer the advantages of a stand-alone downloadable file but avoid the frustrating limitations of the z-machine.

[Pacian]

Crap, I just randomly found out that an earlier version of my code that I thought was okay actually returns a heap error for a different vocabulary word. I noticed an issue with the word “switch” (above), but an earlier version blew up for “white”. And the later version blows up for either word.

Looking at my git commits, I added a condition, added a synonym, added an “unlikely” rule, added another condition, noticed the illegal object error above for “turn switch” and fixed it by removing a couple of vocab words… but actually the whole time “x white” would crash the game. Removing two synonyms fixed both issues…

But how do I know there were only two? What about the third word I haven’t tried yet which will still crash the game?!