Under the hood, the compiled game has a big table mapping vocabulary words to the objects they match. I’m guessing this table is eating up too much space which leads to the crash.
Unfortunately there’s no obvious way to fix that except to reduce your game’s vocabulary.
And yes, I would love to see Dialog compile to a 32-bit virtual machine that doesn’t have these limitations. Glulx is the obvious choice, but Linus was also talking about making a separate 32-bit Å-machine alongside the 16-bit one.
I’ve looked at the Dialog source code, and someday I might try to adapt it…but it’s a huge amount of C code with very few comments, so that’s not a task I’m ready to take on until I know I have a lot of free time to burn.
I’ve commented out nine of a certain kind of second-level scenery object, and I’m hoping that’ll give me a bit of safety. I don’t like the idea of being too close to the point where I noticed things breaking, in case there’s more subtle or complex situations where the game still gets pushed above a limit and breaks.
it’s unfortunate - my current project is a straightforward parser game and i’ve gone back to inform (i prefer 6) which is comparatively painful. but if i proceed with dialog i can’t know ahead of time that i won’t wind up overtaxing the z-interpreter and start getting these stupid heap space errors. if i get these with inform i can just compile to glulx and still have the stand-alone file i want.
i start to get frustrated but then remember that no one is getting paid for any of this and that this is simply brilliant people doing amazing things for the love of it.
I’ve been trying to come up with a (relatively) minimal reproducible example of this, with mixed success.
Generating a basic game with a very large number of dictionary words (an object with slightly more than 400 synonyms will do it) gives a heap error when you try to interact with it. However, it gives the same error whatever word you use to refer to it, unlike the example from @Pacian where only certain synonyms trigger the bug. I also can’t reproduce the behaviour where the heap consumption expands - in all my examples, increasing the heap size removes the error (at least until you add more synonyms).
I know that both I and @cfmoorz2 have a lot of on every tick rules, and, for me at least, those rules are often using exhaust loops. I’m not sure if that could have an impact? I do also have a lot of per-object flags, and two per-object variables.
Both heap overflow issues presented in the library (parse object name $Words as $Result $All $Policy) predicate, is that what you are seeing in your attempt?
So this is how that library predicate is implemented:
(parse object name $Words as $Result $All $Policy)
(filter $Words into $Filtered)
(nonempty $Filtered)
(collect $Obj)
%% catch expressions like ‘open the eastern window’
($Filtered = [$Head | $Tail])
(nonempty $Tail)
(parse direction [$Head] $Dir)
(current room $Room)
(determine object $Obj)
*(from $Room go $Dir to object $Obj)
~(direction $Obj)
~(relation $Obj)
(words $Tail sufficiently specify $Obj)
(from words)
*(dict $Obj)
(matching all of $Tail)
(or)
%% this is the normal case
(determine object $Obj)
*($Obj is in scope)
~(direction $Obj)
~(relation $Obj)
(words $Filtered sufficiently specify $Obj)
(from words)
*(dict $Obj)
(or)
*(plural dict $Obj)
(matching all of $Filtered)
(into $Candidates)
(nonempty $Candidates)
(apply policy to $Candidates $Policy $CleanList)
(if)
%% Optimize for the common case.
($CleanList = [$])
(or)
%% E.g. get all wooden.
($All = 1)
(or)
%% A plural word causes all matching objects to be returned.
*($PluralWord is one of $Filtered)
(determine object $PObj)
*($PObj is one of $CleanList)
(from words)
*(plural dict $PObj)
(matching all of [$PluralWord])
(then)
($Result = $CleanList)
(else)
%% Backtrack over each matching object, for disambiguation.
(if) (fungibility enabled) (then)
(strip-fungible $CleanList $UniqueList)
*($Obj is one of $UniqueList)
($Result = [$Obj])
(else)
*($Obj is one of $CleanList)
($Result = [$Obj])
(endif)
(endif)
Notably, this is the main place that the special (determine object $) (from words) (matching all of $) code is used. Since the problem seems to come from having too many dictionary words, I suspect the way that construction is implemented on the back-end is to blame.
Those words have always been words that could potentially require disambiguation.
But they still trigger a crash even in situations where they are unambiguous or even do not refer to any objects in reach of the player.
It isn’t all words that could require disambiguation but a subset - perhaps just one or two words, but it’s hard to really tell how many.
The number of words that trigger an issue seems to increase as the game pushes further past the limit.
Getting further past the limit also triggers illegal object errors instead of heap errors, which I’m not sure how to pin down to a specific predicate.
When I’m not finalising an IFComp entry, maybe I’ll have a crack at reproducing it myself. Perhaps a Python script to generate Dialog objects with synonyms from a randomized pool of words would do it…
A Python script to generate Dialog source code with large numbers of objects/synonyms is exactly what I’m trying at the moment. But the only way I’ve managed to produce heap errors is by giving a single object 400+ synonyms (which exhausts the heap in the query to *(dict $Obj)). Adding large numbers of each-turn rules and per-object variables doesn’t seem to help. And I certainly can’t work out how the parser can crash on specific synonyms for an object which isn’t even in scope …
My Python script made objects from a list of a hundred random words and assigned them a flag based on a coin flip. Then I give all the objects with that flag a synonym.
The tricky part is, which of those words will trigger a crash. I randomly found out that “jail” does it (although in this case only when the objects are in scope…)
I compiled against the standard library and I’ve attached the .z8 file in case the results are non-deterministic somehow…
It’s also very possible not all the parts of this example are necessary to reproduce the issue, but figuring out when it has triggered is hard. Is there an easy way to paste a hundred commands into an interpreter?
Edit: Hmm, or it just that it’s the words which match a lot of objects… Which isn’t quite what I encountered with my WIP…
Edit 2: And increasing the heap size also fixes the issue. OK, I got ahead of myself.
Here’s my current best attempt: for me, this reliably crashes on “x wing” even though there are only three “wing” objects in the game and none are in the starting room… BUT it only crashes if I compile with the maximum heap space size.
Once the IFComp deadline is past and I’ve properly moved into my new apartment and started my teaching for the semester (this week is a bit insane), I’ll see if I can make a version of the compiler that prints diagnostics about the dictionary when in verbose mode. There’s a bunch of tracing stuff in the source but it’s commented out in the current version.
When @Pacian originally described the symptoms, my hypothesis was that dialogc was producing a malformed dictionary entry when the size of the z-code file exceeded a certain threshold, and the invalid data was somehow causing the interpreter to enter an infinite loop allocating heap memory. But on inspection, Dialog z-code files (unlike those generated by Inform) don’t actually attach any additional data to dictionary words at all, so there’s nothing that could even be malformed. Inspecting the binary confirms that the dictionary in explode.z8 contains all of the expected vocabulary words, correctly z-encoded, in the correct order.
I managed to partially automate testing which words cause crashed when used; the first two-thirds of the alphabet is fine, but after a certain point, about one-third to one-half of words cause a stack overflow when used in a noun context (you can type RELEVANCE, but X RELEVANCE crashes the game).
I thought it was the case that none of the vocab defined by the standard library caused a crash, but X VERSION does it.
Partial list of vocab words which cause a crash - many of the ones which come alphabetically after this also do, but after finding the point where words start to cause crashes relatively easily, I had to start testing individual words manually, so I didn’t do the whole dictionary:
fatal error: object 4980 out of range (pc = 0x1cd72)
receipt
fatal error: call stack too deep: 1025 (pc = 0x9ebd)
I had a quick look at explode.dg, but couldn’t see any obvious link between how the words are used in the source and which ones cause crashes (e.g. #Object11 has dictionary words organize and wonder; the former can be used fine but the latter causes a crash).
I guess the next step is to try tracing the execution of the code itself somehow …
I tried it, why not! It does run, and reproduces the error. The log for just the one-move repro is already 100k lines, so I won’t attach the whole thing; but here’s the final stretch before the restore_undo:
The log format is pretty condensed, but it’s roughly <memory address>: <opcode name> <operands> (-> <stores>)? (?<branch>)?
You may notice that the initial section repeats quite a bit. The actual log has several hundred repetitions. Not sure what’s up with that. Maybe it’s indicative?
If you add Dialog tracing to this, I wonder if we can find what happens after the last query is made?
I’m not sure. I recognize some of these routines from the Dialog runtime, but I don’t know what they do, because the only documentation for the runtime is the routine names and parameter lists.
What I was really hoping to track down is where this run starts to differ from one that doesn’t produce the error. I’m not sure when I will have time to try it, but my plan was to start with two non-bugged words of the same length, compare their traces somehow to check what varies only because of a different word being used, then compare again to one of the bugged words and see where the trace starts to diverge. This would probably need some ad-hoc tooling at the least, of course.
(Does TXD work on Dialog-produced z-code, by the way? I think I had gathered that it was supposed to, but I got an error the one time I tried.)
TXD should work with Dialog but it is missing some of the new opcodes in z-machine standard 1.1 around unicode-characters. See this thread for more info. I’ve made an updated version on my GitHub (link in thread).
EDIT: Also, TXD don’t know Dialog grammar format so you have to run it with the switch to skip the decoding of grammar.
Since the Z-machine dictionary is stored sorted alphabetically, my current best guess is that Dialog using some kind of recursion to search through it, which eventually exhausts the heap. That would explain why it’s specifically words at the end of the alphabet causing the problem.
I’m no closer to a solution for this, but in case it helps anyone, here’s the routine R_COLLECT_MATCH_ALL which is called at the end of a (determine object) construction. I’ve used approximately Inform 6 syntax since that’s what I’m used to, but with a couple modifications (e.g. indicating explicitly which local variables are parameters and which aren’t).
[ R_COLLECT_MATCH_ALL input / current keywords oldtop iter word tmp ;
oldtop = top; ! `top` is a global that holds the top of the heap (grows upwards)
R_COLLECT_END(keywords);
l2:
input = R_DEREF(input);
tmp = input & $E000;
if(tmp != $C000) jump l3;
current = input - $4000; ! ref to head element
input = current + 1; ! ref to tail element
current = R_DEREF(current);
if(keywords == nil) jump l1;
iter = keywords;
l4:
iter = iter & nil;
word = memory-->iter;
iter = memory-->(iter+1);
tmp = R_WOULD_UNIFY(word, current);
if(tmp) jump l2;
if(iter ~= nil) jump l4;
l1:
@throw fail; ! `fail` is a global holding a throw destination for a predicate failing
l3:
top = oldtop;
if(input ~= nil) jump l1;
rfalse;
];
