This is a good point, and it’s distinct from my reply to Daniel. It’s two separate problems. Tests (which I recommended) can tell you whether a claim is true. They can’t tell you whether it’s the right thing to say, as in whether it’s the explanation that will build correct intuition in the next person who reads it, versus one that’s technically defensible but pedagogically misleading.
I can’t speak for this project, of course, but in my professional context, this is where people bring in the LLM-as-judge tooling (DeepEval, G-Eval, etc.). The challenge here is that it doesn’t actually resolve the problem; it just relocates it. (I wrote a lot of posts about using DeepEval.) You’re now using a model to judge whether another model’s explanation is sensible, and “sensible” is exactly the kind of judgment that requires already understanding the compiler internals well enough to know what a good explanation looks like!
So, I still lean on tests as the anchor. Once a claim is operationalized as a test, AI isn’t reasoning about it in a vacuum anymore. It has a concrete, checkable artifact to reason from. That’s a narrower job than “is this sensible.” Narrower jobs are exactly where these models are reliable. Explanation-in-the-abstract is where they hallucinate very confidently. Explanation-anchored-to-a-passing-test is where they’re actually useful, at least in my experience. But, to be clear, that’s a narrowing of the problem, not a solution to it. Someone (as in a human) who actually understands the internals still has to be the one who says, “yes, this is sensible.” That judgment doesn’t get cheaper just because its input improved.
This. Absolutely. A passing sample only proves you know what the compiler accepts. A failing sample proves you know the boundary (what it must reject and why), which is a strictly harder claim to substantiate. That’s basically Popper’s falsifiability applied to documentation: an explanation only means something to the extent you can state what would falsify it. As a general rule, anyone can generate happy-path samples. Generating the rejected cases and correctly explaining why they’re rejected is where you find out whether the documentation was ever actually tracking the specification or just describing its surface.
You want verification and falsification. The implausification part is also needed, and that is still largely (and thankfully!) up to humans.
Indeed! I started working on a game recently and found I had to replace a lot of standard actions performed on animate objects with more appropriate rejections. It would be very nice if this made it into the standard library in case one forgets to override something.
Maybe after a few cleanup passes to get rid of the worst slop writing (ex This isn't a special case bolted onto the engine; it falls directly out of how plain queries are compiled.) and in general reworking it to explain better. But if you don’t have the resources to verify the correctness and then continue maintaining the documentation when the compiler is changed, then rejecting it is the safer choice.
I think your intuition is correct here. If a contributor would like potentially-inaccurate-but-voluminous documentation on the internals, they can have a state-of-the-art LLM generate that for them at the time they need it. This means both that they get an updated version of it, and possibly that they use a more capable model to generate it, meaning its quality would be (unverifiably) slightly better.
Persisting LLM output that has had little or no human work in it is meaningless. It’s cheap to recreate. Now, if the prompt used by the author of the PR has had significant work done to it, then that might be worth keeping around, but not the output.
Because the Apple II has more variation than the C64, I have questions. Do the resulting games require an 80-column card? Can they make use of one if it is present? Do they need more than 48K of memory? Can they run on the entire range, or do they need the II+ or later, or the IIe or later? And presumably, they just boot right up from the disk, and don’t need something like ProDOS?
The basic requirement is 64 kB; no ProDOS needed (aambundle makes bootable disks). This means (as I understand it) they need the ][+ or later.
On a ][+ the disks will run in 40-column mode with only capital letters; on a //e, //c, or IIgs, it will run in 80-column mode if an 80-column card is present; on 128k machines, it will have a larger page cache and can support UNDO. All of this is determined at runtime. (It always supports SAVE and RESTORE if there’s enough space on the disk, but only a single slot.)
Depending on the size of the story file, it will produce some combination of:
A 140 kB disk with both the story and the interpreter (and save file support)
A 140kB disk with both the story and the interpreter (but no save file support)
One 140k disk with the interpreter, and another 140k disk with the story
One 800k disk with both the story and the interpreter
All the files that would go on a ProDOS disk, if you want to create your own
It’s impressively thorough—and lest I just come across an AI grognard (which I kind of am), I imagine LLMs were used heavily in making it, but all the elements are both testable and tested, and the author of the pull request clearly has a strong understanding of what’s going on and has fixed several issues as they came up.
For my IFComp entry, it makes two 140k disks (interpreter on one, story on the other) and one 800k disk (all in one), plus the ProDOS files, and I’m going to distribute them all because I don’t know what format actual Apple II fans prefer. I’ve been testing mostly with the 800k disk on an emulated 1 MHz //e with an 80-column card and 128 kB, and while the lack of styling and custom fonts saddens me, the game itself runs great! Even smoother than on the C64.
This makes sense to me, yeah. I’ve invited the author of the pull request to offer their side of the story as well; I don’t want to misrepresent what they’ve done. It’s possible it’s been extensively reviewed and tested and edited before submitting. (I’m just skeptical because I don’t know if anyone has the kind of expertise needed for that.)
No it has not (been extensively reviewed). I spent a limited amount of time getting the documentation to a level of detail I wanted. I certainly did not set up an agentic feedback loop! I did learn a bit about how everything fits together, though.
Now that the IFComp deadline passed, I’ve been putting my nervous energy into Dialog improvements. Lots of little fixes are now in the PR list, if people want to review them! Otherwise, I’ll probably merge them next week.
Highlights:
Colored divs and spans inside (collect words) no longer corrupt the stack on Z-machine
The text immediately after a progress bar inside a styled div is no longer missing its style on Z-machine
Divs and spans inside closures no longer confuse the optimizer with hidden variables (this was supposed to be fixed in 1c/01 but it turns out that fix only affected the debugger, not the compiler)
New CSS syntax for reverse-video mode
The Å-machine backend will give more detail when it adds resource files to the .aastory file, both on success and failure (because I got annoyed not having these details when debugging one of the IFComp games)
I’m also working on adding a bit more styling to the C64 interpreter. Basically the only thing it can do is change the foreground color (not the font, not the background color), so previously:
What do you all think? I don’t have a lot of colors to choose from, but I think the red feels different enough to stand out, for things like tutorial messages—and the “red bold” and “red italic” stand out against the red about as much as the normal “bold” and “italic” stand out against the black.
Is there any way to do custom fonts, ala c64 style? I was looking through my C64 programmers reference because I remembered there was a way to put your own characters in RAM at the cost of a couple of precious kilobytes.
I tried that before, but the hacky way aamrun is created (by including the entire Node runtime in the executable) gets broken by lipo. I’m hoping we can eventually stop distributing aamrun, because it takes up about 90% of the total file size and doesn’t work very well, but until we have more Å-machine interpreters available it seems people need it.
Which is to say, if you have a way to make a fused binary, please go ahead! But I haven’t been able to make it work.
Currently, aambundle (the program that turns an Å-machine file into a C64 disk) has a special font built in, which it uses for any non-ASCII characters. Since an Å-machine file can theoretically have up to 128 non-ASCII characters on it, there’s unfortunately no room for a full reverse-video alphabet.
It would potentially be possible to add a reverse-video alphabet by sacrificing non-ASCII output, but so far that hasn’t seemed like a worthwhile tradeoff.
And speaking of that font, soon, aambundle will warn you if you use a character that’s not supported (with a custom glyph on C64 or a transliteration on Apple II). That way you can know the limits of the font at build time instead of runtime.
That should be straightforward enough; but it’ll be a lot easier for someone who actually has a Mac to implement it, since they can test it properly! So I have no objections, but I’ll leave that for you or Sue; the game of telephone when the aamrun binary broke would have been a lot simpler if the same person was making the builds as testing them.
The bottom line is supposed to be parallel to the top line. Right now I’m not sure how well it accomplishes that. Between color blindness and the limited palette on the C64, though, I’m also not sure I can do much better. Maybe use the pink for bold, orange for italic, and purple for bold italic? Would that be better?
So if people have ideas, please weigh in on this! I’m now going to stop spamming this thread and channel my nervous energy into fixing some more recalcitrant bugs instead. I still can’t figure out why (stop) inside (collect words) behaves differently in the compiler versus the debugger…
This. Absolutely. A passing sample only proves you know what the compiler accepts. A failing sample proves you know the boundary (what it must reject and why), which is a strictly harder claim to substantiate. That’s basically Popper’s falsifiability applied to documentation: an explanation only means something to the extent you can state what would falsify it. As a general rule, anyone can generate happy-path samples. Generating the rejected cases and correctly explaining why they’re rejected is where you find out whether the documentation was ever actually tracking the specification or just describing its surface.
