Memory allocation error (z-code) vs blank story tab (Glulx)

Quick background:

  • The “free space” reported by the compiler output refers to the maximum size of the game expressed as “packed memory” – which I don’t want to really get into but is a multiple of the 64K mentioned by DavidK. For Z8, the multiplier is 8x, so 512K.
  • Routines and fixed text can be parked in “high memory” but certain constructs (such as modifiable text, lists and some forms of relation) must go into the “low memory” of the first 64K.
  • Part of the low memory is set aside as a run-time heap for use by those constructs. The amount set aside is 8K by default but can be automatically raised to 16K (the functional maximum) or manually raised using the Use dynamic memory allocation... option that you tried. (This likely had no effect because it had already been increased automatically.) You can use the debugging command >SHOWHEAP to see how large of a heap has been allocated and its current usage.

You must be making heavy use of the heap for it to run out of space on move 2. I have to admit that I’m curious about what you are doing that would cause this. There may be a memory leak affecting your game, but it’s also likely that you could do things differently to reduce the heap memory footprint.

Are you very committed to sticking with the Z-Machine?

I’m not committed to the z-machine, as such, but I’m concerned that I might be doing something that is unnecessarily gobbling up memory and I’m also not super thrilled at having to disable my virus scanner in order to use glulx.

When the game starts, SHOWHEAP reports 8186 of 8208 free. After I open the door, SHOWHEAP still reports 8186 of 8208 free. When I try to go north, the game crashes with the error “*** Run-time problem P49: Memory allocation proved impossible.”

If I remove just one object from the game, there is no crash, and after 57 moves, SHOWHEAP still reports 8186 of 8208 free. If I add a single object definition of any kind, the game crashes on the second move again.

It’s telling you that the memory error occurs somewhere in the small amount of code between the “generate action rule” commencing, and this rule then calling the “standard set going variables” rule.

The “generate action rule” does try to dynamically allocate memory on the memory stack for any action variables needed by the going action before calling the “standard set going variables” rule, and presumably this allocation is what is failing.

I can’t really look into this any further without being able to trace what’s going on in your code precisely, which would require you sending the full source…

I don’t know how to trace what’s going on in my code, but I do know that if I remove one object, SHOWHEAP reports “8186 of 8208 bytes free”, RULES ALL shows a very long list of rules, and there is no crash, even after 57+ moves.

If I add just one object, even a very simple one, this is the result, when I make my second move:

>showheap
8186 of 8208 bytes free.
>rules all
Rules tracing now switched to "all". Type "rules off" to switch it
off again.

[Rule "parse command rule" applies.]
>n
[Rule "declare everything initially unmentioned rule" applies.]
[Rule "generate action rule" applies.]

*** Run-time problem P49: Memory allocation proved impossible.

I’m sure there are lots of things that I’m not doing the best or most efficient way, but I’m very confused by how the addition or removal of one object is causing such a drastic difference between working perfectly and crashing on the second move.

Like Dr. Bates said, at this point it’s not likely anyone will be able to help without source code that reproduces the problem. Usually we ask for the shortest, simplest code that does so, but in this case it may be very hard to reproduce if you remove things.

This is sounding like an Inform 7 bug.

Per drpeterbatesuk’s suggestion stack frame allocation might be where it’s failing, you can try to add:

Include (-

[ StackFrameCreate size new;
	new = I7SFRAME - WORDSIZE*size;
	print "^blockv_stack: ", blockv_stack;
	print "^new: ", new;
	if (new < blockv_stack) { RunTimeProblem(RTP_HEAPERROR); @quit; }
	I7SFRAME = new;
	"^finished!";
];

-) replacing "StackFrameCreate".

and let us know what output results.

EDIT: OK, some interesting things while following up on this. Note that all of the below is for 6M62 and may have changed a bit in 10.1.

First: I think the Use dynamic memory allocation of... option has only one effect, which is to override the default value of 8192 in the declaration of constant DynamicMemoryAllocation.

! Use option:
 Constant DynamicMemoryAllocation = 8192; ! value overridden if 'Use dynamic memory allocation...' provided

8192 bytes is coincidentally also the default size of the Flex heap (which is the automatically-sized heap I mentioned above), but there is no direct connection as I had previously believed (and as seems to be implied in WWI 20.2 Memory limitations based on the description of behavior; perhaps this information is out-of-date?).

Second: The DynamicMemoryAllocation constant has one and only one use in the rest of the code, which is to set the value of constant BLOCKV_STACK_SIZE.

#ifdef TARGET_ZCODE;
Constant BLOCKV_STACK_SIZE = 224;						! note invariant size in Z-machine
#ifnot;													! for TARGET_GLULX
Constant BLOCKV_STACK_SIZE = DynamicMemoryAllocation/4; ! clearer to divide by WORDSIZE?
#endif;

...

Array blockv_stack --> BLOCKV_STACK_SIZE;				! number of words, not bytes -- 448 bytes for Z

The blockv_stack array is used as a stack for the block value subsystem, and it is separate from the Flex heap which is also used extensively by the same system. So, for Glulx (and Glulx only), DynamicMemoryAllocation is used to set the number of entries in the block value stack based on the dictated size in bytes (rounded down to the nearest whole word).

Third: From what I can see, the only place an RTP_HEAPERROR run-time error can be displayed without additional information accompanying it is via StackFrameCreate(). The other two routines from which it can be triggered [which are FlexError() and BlkValueError()] have preceding print statements that should have shown up in your output.

The StackFrameCreate() routine is so simple that it’s hard to imagine a bug. The implication is either that an unexpectedly large size parameter is being fed to it, or (much more likely) that the I7SFRAME pointer is at its lower bound when the call is made. The latter implies a recursion error in your code in which an infinite number of stack frames are being generated. Many things use stack frames (e.g. rules, phrases, block value manipulation), so it’s hard to know what the precise problem is without more information.

If the addition of a new object is the proximate cause, and going north is the trigger, I would look for some interaction between rules governing the going action and any phrases that you’ve defined – such as a rule calling a phrase that directly or indirectly calls that same rule.

EDIT 2: For example, this reproduces your error:

To p1:
	let b1 be "temp1";
	p2.

To p2:
	let b2 be "temp2";
	p1.

Instead of going north:
	p1.

Does any of your logic change its behavior based on the total number of things, or the number of things in a room, or the number of things in player inventory, etc.?

5 Likes

I don’t do anything in my code for going north specifically or going in general, except:

Understand "go to" as going. Understand "go home" as leaving.

I don’t do anything related to numbers of things at all.

I placed your Include code at the top of my code and these are the results:

Without adding an object, I get:

>n

blockv_stack: 32171
new: 32619
finished!

blockv_stack: 32171
new: 32619
finished!

blockv_stack: 32171
new: 32619
finished!

and the game works properly.

If I add one simple object definition, I get:

>n

blockv_stack: 32346
new: -32742
*** Run-time problem P49: Memory allocation proved impossible.

at which point the game crashes.

By the way, I have no problem sharing my full code, if that is allowable (and if it will help).

Much thanks to everyone for trying to help me figure this out!

EDIT - Here is the object that I’m adding for test purposes that makes it crash:

The thingamabob is an object.

but it doesn’t seem to matter what the object actually is. It’s as if I’m at some kind of threshold and one more object tips it over.

You have reached what seems to be a hard-wired Z-machine threshold whereby this error is triggered if the stack frame pointer rises above 32767. That’s roughly where you would expect it to be for a Z-machine game compiled in 10.1.2 with ~70 total objects (each of which adds ~170 bytes to the sub-stack memory).

Compiling for 9.3 ( [Settings] tab → [Language Version] → choose ‘Inform 9.3 (6M62), December 2015’ ) would give you a lot more headroom (~256 total objects) before hitting this limit- each additional object appears to use only ~100 bytes in 9.3 and other (non-object) usage of sub-stack memory is also considerably less than what 10.1.2 appears to consume.

EDITED for a clearer description of the issue in view of Otis’s insight below.

3 Likes

What happens if you include the following (and remove the previous replacement)?

Include (-

[ StackFrameCreate size new;
	new = I7SFRAME - WORDSIZE*size;
	if (UnsignedCompare(new,  blockv_stack) == -1) { RunTimeProblem(RTP_HEAPERROR); @quit; }
	I7SFRAME = new;
];

-) replacing "StackFrameCreate".

It looks like the “top” of the block value stack is above the max signed integer, so the existing test is confused. (Here “top” means the high end from a memory address perspective.)

2 Likes

Ah yes. I see now that it’s more subtle than I thought.

The problem as you point out is that the start in memory of the block value stack array (blockv_stack) has got pushed up in memory so that the end of the stack (where the stack pointer I7SFRAME starts from, moving back from there through the stack array, i.e. downwards in memory) is above the 32767 byte (max signed integer) boundary, so the comparison test intended to check that we’re not about to move the stack frame pointer back beyond the start of the block value stack array misfires, triggering a spurious memory allocation error.

So it’s a longstanding bug in Inform 7 exposed by the memory-hunger of Ver 10, not a limitation of the Z-machine, and your fix should fix it :grinning:

2 Likes

Fascinating! I never would have thought this was the cause from the original error message.

Having built a test project with 70 pebbles to replicate the error, I can confirm that this fix appears to work :smiley:

EDIT: Do you want to log it?

2 Likes

Concur with Bates about logging this as I10 bug: I10 having its “hidden punyinform mode” (that is, compiling with OMIT_UNUSED_ROUTINES, whose incidentally seems has helped isolate the bug) squashing said bug IS important, albeit IMVHO.

Best regards from Italy,
dott. Piergiorgio.

1 Like

I don’t have an account for the new system, so please do submit a report on behalf of the MSC if so inclined.

Note that there is a line in BlkDebugAddress() that uses I7SFRAME in some arithmetic:

! ==== ==== ==== ==== ==== ==== ==== ==== ==== ====
! BlockValues.i6t: Printing Memory Addresses
! ==== ==== ==== ==== ==== ==== ==== ==== ==== ====

[ BlkDebugAddress addr flag d;
	if (flag) { print "###"; return; }

	d = addr - blockv_stack;
	if ((d >= 0) && (d <= WORDSIZE*BLOCKV_STACK_SIZE)) {
		print "s+", (BlkPrintHexadecimal) d;
		d = addr - I7SFRAME;							! <--- HERE
		print "=f"; if (d >= 0) print "+"; print d;
		return;
	}
	
	d = addr - Flex_Heap;
	if ((d >= 0) && (d < MEMORY_HEAP_SIZE + 16)) {
		print "h+", (BlkPrintHexadecimal) d;
		return;
	}

	print (BlkPrintHexadecimal) addr;
];

I think it will still work correctly even if the block value stack’s array partly or fully crosses the maximum positive integer boundary, but it’s worth a double-check.

The additional consumption per object is pretty significant. Did you happen to spot any cause(s)?

No, it was a purely empirical observation of how blockv_stack varied after the addition/removal of an object.

Even more striking is how the baseline memory usage below blockv_stack for an empty project is ~6700 bytes in 9.3 but ~18500 bytes in 10.1

Zed is doing so.

I wanted to give you some information, in case it was helpful.

McAfee is showing that both of the glulx interpreters contain some kind of malware called “Artemis!CBF4903FCEFD”.

They do provide an option to exempt specific files, but that just keeps it from quarantining them; it still won’t allow them to execute. The only way I can run them is if I completely disable virus scanning, which is not a great solution.

My concern is not so much that I have to disable my virus scanner when I’m testing, but that other people who download my game are going to run into the same problem, if they have McAfee (although a Google search only shows my post here, so if anyone has had the same problem, they apparently haven’t posted about it).

By the way, thanks again to everyone who helped with my object limit. It fixed the problem for a while, but I did eventually hit the z-code memory limit and had to switch to glulx anyway.

Most people wanting to play your game will either have their own interpreter set up to their liking already, or will play it online with Parchment (which runs in a web browser and thus bypasses McAfee’s paranoia).

2 Likes

Searching around about this, it seems that “Artemis” is a generic term McAfee uses to indicate “suspicious” files, rather than those containing something specific, for example: https://www.mcafee.com/support/?locale=en-US&articleId=TS100414&page=shell&shell=article-view

As ever with this sort of thing, it’s impossible to know exactly what it is detecting, or what one could do about false positives. There are certainly signs that you are not alone in needing some way to explicitly whitelist files, e.g. McAfee Support Community - How to "White List" False Positives Manually? - McAfee Support Community

As someone who has worked for an AV vendor in the past, I stick with my view that the standards of the industry are not high, and that McAfee has a poor reputation even by these low standards. But that probably isn’t much help to you.

3 Likes