The last two weeks I’ve been working hard on something new, and it’s now basically finished. Presenting RemGlk-rs: a Rust port of RemGlk!
Almost all Glk features are present, aside from the sound system. This includes some features that until recently I didn’t even realise the original RemGlk was missing, like mouse input. It’s actually half a port of RemGlk, and half a port of my GlkApi from AsyncGlk, because Rust is more similar to Typescript than it is to C.
This has really stretched my knowledge of Rust. First time using mutexes, reference counting, or the FFI system. I’ve focused mostly on correctness so far, I’ll be looking at what I can do for performance and file size in the future. And of course looking at how to get it working in Emglken!
If you want to give it a test you just need to run cargo build in the top level folder. And then, for example, to link Glulxe against it, you would set the following variables (assuming it is in a sibling folder to glulxe):
I’ve now successfully compiled Glulxe against RemGlk-rs. Which is a great first step, but the next steps will be harder…
I got it working by entirely sidestepping libc’s file system code, instead it directly contacts the JS Dialog library. If an interpreter uses the Glk API for all of its file reading and writing (as Glulxe does), then this allows us to not actually use libc. (Note that I’m including glkunix functions like glkunix_stream_open_pathname.)
I haven’t looked in detail at the other interpreters yet, but I don’t think it would be practical to convert everything to Glk functions. Git might be okay, but the others probably not.
I can think of two other options:
Wait for Emscripten’s WasmFS to be ready. I’m not sure how close to ready for our use case it would be.
Replace stdio.h with a new implementation that implements its functions using the Glk API. I don’t know if that would really be feasible, but it might be?
Is it not possible to use the existing (pre-WasmFS) stdio.h emulation implemented mostly in Javascript? It seems emgklen has used that so far, I’m curious if there’s anything about remgkl-rs that’s incompatible with it.
In theory it could be put through Asyncify too, but the FS functions aren’t directly called, so it’s not as simple as other WASM imports. WasmFS is more direct, and being based on syscalls might also involve fewer functions needing to be Asyncified. That’s what I’m hoping at least.
Hugo and Scare also work without changes, that’s 4/6! That leaves only Bocfel and TADS, and I think Bocfel could also be made to work without stdio access to the file system. (I haven’t looked at TADS yet.)
Glkunix already has the glkunix_stream_open_pathname and glkunix_stream_open_pathname_gen functions for opening arbitrary files (that’s how the other interpreters can work without the FS.) But they’re only part of glkunix, and they also don’t have all the file modes that a VM might need (only read or write.)
To implement those functions I had already made this function:
It’s identical to glk_fileref_create_by_name except without filename cleaning. By exporting that function and making some small changes io.cpp, Bocfel is able to start and write savefiles. (It’s currently unable to restore because it thinks the savefile is corrupted, but I’m sure we could fix that.)
@cas What would you think about using this Glk extension in Bocfel?
Is one function enough or would we need more? (This function probably does remove the need for glkunix_fileref_get_filename.)
How would it work in windows?
Can we just say that it should only accept a bare filename?
In order to write autosaves etc, I think we do need some support for paths, so this probably isn’t a viable option.
Would relative (./) paths be okay, and the library would handle the slashes?
Would absolute paths be okay and it would be the interpreter’s responsibility to provide an OS-appropriate path?
Or we could just say that the function is glkunix only (and rename it), and other OSes would have to use the native FS functions like at present. This is what I’m leaning towards now.
Alternatively, rather than a fileref extension, we could just add a glkunix function to directly make a stream, but with more options if ReadWrite or WriteAppend are needed. But I assume it would be useful to be able to call glk_fileref_does_file_exist and glk_fileref_destroy on the files.
It would be great if I can basically bypass stdio entirely! It’s feeling like that might actually be possible, which would mean I wouldn’t need to wait for WasmFS, and it will probably be more efficient too. In essence, if everything uses glk/glkunix functions, then I can make my own syscalls just for Emglken.
(Bocfel did still need Emscripten’s JS FS mode to be enabled. Without it there are errors for the openat syscall. We can probably deal with that later, or just leave it, it’s only about 95kb of unminimised JS.)
I did a quick and dirty test of implementing it in Gargoyle, and made some Bocfel changes that conditionally call it (basically your changes but protected such that it works whether or not the function exists). I assume this will be behind a feature test macro, but for now I defined a NO_STDIO macro for testing purposes. If Glk is being used and NO_STDIO is set, then the new function is used instead of stdio. It seems to work fine in all circumstances. It’s available here.
For Bocfel’s purposes, it has to take arbitrary files, including full paths. This is used for opening files provided by the user (the story file itself plus possible transcripts/command records). As long as interpreters don’t give direct access to this function to games, it won’t matter that it’s arbitrary, since the user has already entrusted the interpreter itself with the filesystem, i.e. the interpreter can already do whatever it wants with standard I/O routines anyway.
Or, rather, I should say, implementations shouldn’t be prevented from allowing arbitrary paths/filenames. Individual implementations would be free to restrict access as/if necessary. With that in mind, I think I’d ultimately have Bocfel default to stdio, even if this function is available, but add a new configuration option to use the new function. So if you’re building Bocfel for a system which needs stdio bypassed, then you tell it, and if the new API function is available, all is good; otherwise, it’s a build failure.
Since this will have to be guarded by a macro, then I don’t particularly see any harm in “allowing” any implementation to provide it if they so desire. Windows Glk, for example, provides the glkunix_fileref_get_filename function, despite its name.
That all sounds reasonable. Great to hear it works!
So what I’m thinking is to define glkunix_fileref_create_by_name_uncleaned and macro GLKUNIX_FILEREF_CREATE_UNCLEANED. The function would allow full paths; relative paths would be based on glkunix_set_base_file the current working directory.
Having it be opt-in for Bocfel is fine of course. If I’m understanding correctly, you’ll make it just check the Bocfel macro NO_STDIO and then assume the function exists, getting a link error if it doesn’t, rather than checking both it and GLKUNIX_FILEREF_CREATE_UNCLEANED?
So for Windows, would it just have to assume the path is correct for the OS? This isn’t theoretical - for Lectrote I’ll have to deal with Windows paths somehow. Except Emscripten still thinks it’s Linux, so I guess I’d have to do something like this.
Just for reference, in Gargoyle, glkunix_stream_open_pathname does not care about glkunix_set_base_file. It ultimately just passes the filename to fopen(), so if it’s relative, it’s relative to the current working directory. glkunix_set_base_file affects the following:
glk_fileref_create_by_name
garglk_add_resource_from_file
Searching for SND and PIC files
It makes sense to me for glkunix_fileref_create_by_name_uncleaned to work just like any “normal” system function, such as fopen, and open relative to the current working directory. However, that’s not a strongly-held conviction, so if you prefer for it to go the other way I’m fine doing it so. It’d just be the only function that operates that way.
I’ll wind up checking both (and I’ll come up with a better macro name than NO_STDIO); basically, something like this:
#if defined(ZTERP_NO_STDIO) && (!defined(ZTERP_GLK) || !defined(GLKUNIX_FILEREF_CREATE_UNCLEANED))
#error No stdio mode only works in a Glk implementation that provides the glkunix_fileref_create_by_name_uncleaned API call
#endif
So it’s at least somewhat friendly…
That’s my gut feeling. As far as I can see, this function is really a stand-in for fopen() and so should act like it. But this is coming firmly from C++/Unix world, so I may well be missing something critical.
Honestly I barely understand exactly what glkunix_set_base_file is meant to do. It wasn’t documented in remglk, but I see it is in cheapglk, which says it also affects glk_fileref_create_by_prompt. I’ll probably have to fix this in RemGlk-rs.
What you’re saying makes sense though. glkunix_fileref_create_by_name_uncleaned should resolve relative paths based on the CWD.
Oh, I’ve realised a complication with glkunix_set_base_file in Parchment, but I think I have a plan for how to make it work.
So in order to get supplementary resources working, I want to map the original download to a predictable path. A game like Trading Punches has two supplementary files. The main URL is at https://unbox.ifarchive.org/2p4260ic2s/trading.hex. I think I’d like to map it and its supplementary resources to these paths (putting the URL excluding the final filename through encodeURIComponent):
Whereas I’d like the CWD to be either /user/ or /user/trading/ (giving each storyfile its own folder could be nice, though I haven’t decided about that yet.)
So what I’m thinking, is that in Parchment glkunix_set_base_file wouldn’t set the CWD, but instead it would set another variable, which would be used by garglk_add_resource_from_file and glkunix_fileref_create_by_name_uncleaned (and any other functions which need to find a supplementary resource file.)
In essence there’d be two “current” directories, one for Glk user files which should be looked for in browser storage, and one for downloaded files. If it tries to load TRADGFX it will know to go download it (though I might cache files for offline play), whereas if a storyfile looks for an achievements .glkdata, then it won’t attempt to download it, it will just say whether or not it exists in /user/. The one thing that wouldn’t work is downloading a .glkdata (or .glksave etc), but that already doesn’t work. People should already know to either construct it as needed, or to put it as a resource chunk in the blorb.
But that’s only in Parchment. Emglken running in Node or Lectrote would make the two current directories the same, as Garglk would. Does this all sound workable?
I’m still trying to work out what makes the most sense. Let’s look at all the functions:
Function
Relative dir
glk_fileref_create_by_name
Glk “current directory”
glk_fileref_create_by_prompt
Glk “current directory”
glkunix_stream_open_pathname
System CWD
glkunix_stream_open_pathname_gen
System CWD
garglk_add_resource_from_file
Storyfile directory
glkunix_fileref_create_by_name_uncleaned
?
It’s essential that glkunix_stream_open_pathname/_gen use the actual system CWD. It’s also essential that garglk_add_resource_from_file uses the same directory as the storyfile, as specified with glkunix_set_base_file.
glk_fileref_create_by_name/_prompt use the Glk “current directory”, which normally is also specified with glkunix_set_base_file, but I might do something funny in the browser for Parchment. For all other interpreters it can be considered the same as the storyfile directory (or the system CWD if glkunix_set_base_file was not called.)
So what should glkunix_fileref_create_by_name_uncleaned be relative to? I can see arguments for both the system CWD and the storyfile directory:
As a generalisation of glkunix_stream_open_pathname (and indeed of all file functions) it should use the system CWD. Anyone calling it will need to take care of their own path joins.
As its main use will be for opening the storyfile or sibling files, it should be relative to the storyfile directory. This would be simpler for the interpreter author, whereas the Glk library author will be better prepared to handle joining paths if they, for example, wanted to use glkunix_fileref_create_by_name_uncleaned to implement glkunix_stream_open_pathname or garglk_add_resource_from_file. Also, joining paths in C can be a bit hacky, whereas Glk libraries are sometimes written in more modern languages with safer path join APIs.
At the moment this is all kind of moot as the interpreter and library authors are the same, but I wanted to think about what might be simplest for another interpreter author wanting to use these functions.
I’m leaning towards the second option, but it doesn’t make too much difference. The only thing would be if you needed access to the system CWD after startup (as glkunix_stream_open_pathname isn’t meant to be used inside of glk_main), but I just can’t think of a scenario in which that would be necessary.
So I tried using glkunix_fileref_create_by_name_uncleaned to implement TADS file functions, and it definitely needs to be relative to the system CWD. But I think the name argument should also be const. Does that sound right @cas? The final signature is
With that function, and a few others, I’ve got TADS working in Emglken again! I didn’t expect to be able to actually get all these interpreters working without a real file system, but most of them have actually been designed with porting in mind, such that their file functions can be mapped to the Glk API instead.
Now the only interpreter left is Bocfel, which @cas has already mostly got working.
Yes, I’m in favor of const char * for the name. I’ll get the Gargoyle branch updated and clean up the Bocfel bit, so it’ll be available in the next release.