Over the past year of Dialog development, we’ve built up a huge battery of automated test cases. While they’re still not complete (we don’t test anything on Å-machine, for example), they’re a great reassurance that my new tweaks haven’t broken anything fundamental.
So as I tinker with the Å-machine web interpreter, I’d like to have some automated tests there, too.
The problem is…it’s a web app. The Dialog compiler (and debugger and the Z-machine interpreter we test on) is a command-line program, so I can pipe data in and out and make sure it looks right. But how do you automatically test a web page?
We had the same problem when we built Moondrop Island. I cobbled together a python test runner using selenium.
It’s probably rather ugly code and tailored to our project, but here it is as an example. I recall it being fairly slow though test.py.txt (9.7 KB)
Thanks all! Selenium definitely sounds like the way to go. Since it’ll be running a bunch of tests on the same program, I’ll see if I can code up a little test runner that opens the page, feeds it input line by line from a file, and saves output to another file. Then we can automate the tests the same way we do for the Node version.
FYI that’s what RegTest-HTML already does (though it uses Puppeteer rather than Selenium.) Adapting it for Dialog wouldn’t be too complex, you’d just have to implement the mutation observers. That part of the process would be the same if you started from scratch with Selenium, but using RegTest-HTML would then mean you wouldn’t need to implement the rest of the testing framework. Of course it’s up to you what you want to do.
Using the same tests as for command line was the impetus for me making it.
The difference (a difference anyway) is that RegTest-HTML abstracts input into standard Glk events. That lets you test the game’s responses.
If you want to test the interpreter, ie how it handles key and mouse clicks and menu selections and routes them to the game, then you want Selenium or equivalent.
EDIT-ADD: I should have also said that RegTest-HTML is useful for testing the interpreter VM, that is, the underlying Glulx/Z-machine engine. If there’s a bug there, it will show up as an incorrect game response, which RegTest-HTML can catch.
Where you need Selenium is finding interface bugs, like “the window is scrolled to the wrong point” or “the wrong input element has focus” or “keystrokes are not triggering VM key input events”.
Thankfully, we already* have a test suite** for the VM, using Node.JS; what needs testing at this point is the frontend. So I’m hoping I can set up a test harness that feeds all the input in, one line at a time, and records the innerHTML of the main output window (continuously, in case of screen-clearing effects). Then we can use our normal diff tools to see if anything notable changed in the HTML.
For example, the Node frontend lets us confirm that the actual game is running fine, but it doesn’t do any fancy formatting. Ideally the test harness will give us a file full of this is <span class="aalook-italic" aria-role="emphasis">italic</span>. so we can confirm that part of the frontend works.
(What about the status bar? No idea. But if I can play through a full game and automatically confirm the frontend never crashes during it, ideally on a few different browsers, that’ll be a good start.)
* as of yesterday
** two full-length games with walkthroughs
That’s fair. I use RegTest-HTML to test the essential functionality of Parchment (including save/restore) to make sure I don’t ship a totally broken app (which has happened before…) It can test (fake) keystroke input. But it doesn’t test the styling, scrolling etc.
I found Puppeteer quite easy to use, but I can’t really speak to its advantages/disadvantages compared to Selenium or Playwright etc.
