Building IF tools with Cosmopolitan Libc; possible game-changer?

Technical Development
1

Since finishing Status Line v3 and publishing builds for multiple platforms, I’ve been thinking a lot about software preservation in general and for the IF development toolchain. I have to note here that “preservation” here refers to “the past decade or so, and into the future.” The retro crowd is not really served by this post. :bowing_man:

Two things in particular have been on my mind:

  1. To make “platform agnostic” builds of things, we need a virtual machine to stand between the OS and z-machine interpreter (for example). I use Pico-8. There was (is?) JVM. Electron tries to provide this with Javascript. We have web browsers, but wither Internet Explorer, Flash, etc? There are engines like Unity, which has seen migration to Godot. And so on… It’s just a never-ending cycle of machines falling in and out of favor.
  2. The alternative is to write in a shared language, like C, and provide massive, complex make files which generate bespoke builds for every platform someone wants to take the time to support.

Write once; run natively anywhere

There is, however, a potential third way. This didn’t really become a viable option (in my opinion) until just last month. But what if we could have the convenience of a virtual machine’s “write once, run anywhere” model with a native application?

If you’re not familiar with Cosmpolitan Libc and the Actually Portable Executable projects, I hope those with some interest/understanding in build tools will take some time to check it out: GitHub - jart/cosmopolitan: build-once run-anywhere c library

The long and the short of it is that v4 of Cosmopolitan Libc came out just last month and this project has gone from “interesting experiment” to “holy crap, we have something useful here.” Justine Tunney is a magician.

She realized that by writing in C and leveraging the shared lineage of modern computers, she could create portable executables which run everywhere. She says,

“I want any software work I’m involved in to stand the test of time with minimal toil. I believe the best chance we have of doing that, is by gluing together the binary interfaces that’ve already achieved a decades-long consensus, and ignoring the APIs. …we can see that few things about systems engineering have changed in the last 40 years at the binary level. Platforms can’t break them without breaking themselves.”

Using her tools, the end result of the build process is a single executable which runs on everything. Copy it to a Mac, it runs. Copy it to Windows, it runs. Copy it to Linux, it runs.

This week I spent time building and using a full, universal IF workflow toolset:

  • I wrote an Inform file in a universal build of vim
    (complete with Inform syntax coloring)
  • I compiled it to z5 using a universal build of inform6
  • I played the game in a universal build of dfrotz (dumb frotz)

I ran this workflow on:

  • Windows in PowerShell.
  • Windows Terminal, WSL2 (Ubuntu).
  • Ubuntu desktop (in VMware)
  • M3 Mac in Terminal.

All tools were built using Justine’s cosmocc, a modified gcc

  • vim was pre-built by Justine (grab it and other tools here)
  • inform6 I built from source (zero problems; worked on the first attempt)
  • dfrotz I built from source (dumb frotz because I’m having trouble getting ncurses working, though others have success with it. Still learning!)
  • (I also added VT100 escape code character display functionality to mojozork; works great, status bar and screen scrolling included)

Built executable sizes are larger than single-platform builds, but maybe not by quite as much as you might expect. As well, I’ve come to believe these may be bloated with debug symbols:

  • vim, with basically everything pre-installed: 18MB
  • inform6: just under 2MB
  • dfrotz: 889KB

With the success of the above builds, I had to try some other things of course.

  • ScottFree built and ran games just fine (with a small source code change to remove deprecated function calls)
  • DialogC built and compiled a z8 file for me without issue. Maybe this can help alleviate a bit of the “which system do you want it for?” question? (at least for 64-bit machines)

Other findings

  • It was a struggle to get the cosmocc tools to work nicely with Windows, both PowerShell and WSL, but I did get it working. macOS and Ubuntu desktop had zero issues for me. Again, this struggle is for building an executable from source with cosmocc. Once built, the final executable delivered to a user does not have nearly problems.
  • The built executables have no file extension. macOS and Linux don’t seem to mind. Windows really wants an extension. So append .exe or .com to the filename to make Windows happy.
  • I had to chmod the executable I copied over to macOS to run it
  • Executables are actually just .zip files at the end of the day. Something like inform6 could (for ease of use for a beginner) be distributed with basic libraries embedded into the executable. Likewise, vim or emacs could be stripped down to the IF-related basics to reduce filesize.
  • I’m still trying to figure out how to build libraries, like ncurses, to add to the cosmocc build workflow. It comes with libc compatible libraries, but adding additional 3rd party stuff requires extra know-how.
  • I have seen screenshots of someone getting Command & Conquer built with these tools, so more complex applications definitely seem possible.
  • There is a branch of Cosmopolitan which targets Windows Vista and Windows 7, in the interest of broadening the reach of such builds. I haven’t tried it.
4 Likes
2

If your platform is the terminal (CLI or TUI apps) then building C programs for that quite straightforward (imho), even when targetting several OSes and/or architectures.

Things only get complex (hideously complex I think) when making GUI programs. I betting the tools you are talking about do not handle such things.

For me the GUI side of things it is very disappointing state of affairs. There are cross-platform toolkits like FLTK, that’s perhaps the lightest one out there, but under the hood it is still a bloated and horrifying spagetti ball-of-mud type of code (do a deep dive on FLTK’s code and you’ll see what I mean). Plus users much prefer the native toolkit, which looks better and works as they are accustomed to.

Anyway, to get back on topic . . . . The binary working on multiple OSes is a very cool hack, but ultimately not a genuinely useful thing (imho).

1 Like
4

This is interesting and very impressive. But there’s still an xkcd-“now we have 15 standards” air to the whole idea.

The project seems to be primarily aimed at universal (well, broad) support over platforms today, not infinite future compatibility. Space, not time.

The fact is that the requirements for executable code change decade-over-decade. “Runs on everything” isn’t everything; it’s a fixed list. According to the docs:

a POSIX-approved polyglot format that runs natively on Linux + Mac + Windows + FreeBSD + OpenBSD + NetBSD + BIOS on AMD64 and ARM64

Right now ARM64 seems like the universal future hardware platform (as the Intel architecture stumbles and falls behind). But if you’ve been around as long as we have, it’s a lot more like “today’s fad”. And that’s not even getting into stuff like signing requirements, which change a lot faster.

(Really, let’s not get into signing requirements. It just leads to shouting.)

Digging into the Cosmopolitan blog posts, I see statements like “Outside phones routers mainframes and cars, the consensus surrounding x86 is so strong, that I’d compare it to the Tower of Babel.” (source) That was posted in 2020 and it already feels out of date, sorry.

(EDIT-ADD: “Tower of Babel” indeed! I just noticed how badly that metaphor lands.)

The reason we stick with “portable C” and “documented VMs” is that they’ve worked for going on forty years. Actually there’s a third approach, which is “emulators for old computers”. Nothing else has worked.

Yes, there’s a constant cycle of porting the interpreters and the emulators (and the Makefiles!) to newer machines. It requires active ongoing work. I don’t see that ever changing.

5 Likes
5

I don’t see a big difference between VM (in this sense) and emulator. They’re both just abstraction layers and only differ in what is running inside.

6

There will always be new incompatible platforms (as long as there are people to make them).

If you want your stuff to run on that future platform your options are:

  1. Port it
  2. Run it in an emulator (which needs ported). This includes everything from a z-machine interpreter, to full blown computer emulation.
2 Likes
7

Just a side note: I recently looked at a web assembly based web technology (i.e. a binary instruction format for a stack-based virtual machine), and I found it quite powerful. The Unity WebGL runtime is also based on this.

2 Likes
8

Speaking of browser stuff, how tricky is it to run old .swf files now that Flash has been abandoned for a few OS/Browser generations? And how well do modern browsers run 20 year old JavaScript? Or outside of the browser, how well does old Java bytecode and old Python code run on a modern JVM and modern Python interpreter respectively?

1 Like
9

Python made not-backward-compatible changes in 2008 with the release of Python 3…and even now, over fifteen years later, and five years after it stopped getting security patches, Python 2 is still preserved in many package managers, for the sake of legacy code. I fully expect to have to keep specifying python3 instead of python in shebangs for the rest of my career, because it’s so important to keep old Python 2 applications around.

2 Likes
10

Flash emulation is a thing: https://ruffle.rs/

Old Javascript has held up pretty well since about 2010. The enormous anchor-weight of old machines running old web browsers has made backwards compatibility a high priority for everybody.

To be concrete: Around 2010, I set up my web site to run my Z-code games with a then-current version of Parchment. I haven’t touched them since. They still run fine.

3 Likes
11

I like the idea of this, but i don’t see “web” as a target in “run anywhere”. If you can’t target web then this isn’t much use over just writing portable C code.

1 Like
12

Thanks for everyone’s feedback. I’m glad for the discussion.
A couple of quick responses.

@andrewj

Things only get complex (hideously complex I think) when making GUI programs.

Yes, but I’ve seen screenshots (and I think there’s a public proof of concept release) of Command & Conquer running using this, so it’s not NOT possible I think and (judging from what I’ve seen in forums) is something they wish to address. So, I’ll be curious to see what v8 of this project can do.

@zarf

The project seems to be primarily aimed at universal (well, broad) support over platforms today, not infinite future compatibility.

With today being as far back as 2003 in the support vector. Of course I concede that if the industry switches over to quantum processors next year, this probably (absolutely?) won’t keep up with that. But if there is a method to minimize the effort required to support the 64-bit era of systems, that doesn’t seem useless to my mind. The build process for the inform6 compiler was

cosmocc -o inform6 inform.c

And that gave me the multi-platform build; no make files or anything else. I mean, that’s pretty dang powerful for just 28 characters. So in regard to the constant cycle of updating emulators, etc, It requires active ongoing work. I don’t see that ever changing. I guess I’m suggesting that something like this might give the community some method to MINIMIZE the work needed, at least to support a (pretty large) subset of machines.

@jkj_yuio
This is portable C, just a new compiler and libraries to teach C new tricks.

2 Likes
13

I’d also add that, recently at least, there are FPGA-based patforms (like the MiSTer and Analogue Pocket), although they’re currently fairly niche.

14

Yeah, I’d also go with WebAssembly, many corporations with much money are backing it, the runtimes have near native speed, it works in the browser…

1 Like
15

the runtimes have near native speed

Probably not fully native speed, but close. I find the performance quite good. You can see it here:
https://interactive-onions.io/MastersLair/

This is a C#/ ASP.NET Blazor App, that runs with WebAssembly.

many corporations with much money are backing it

Mozilla, Microsoft, Apple, Google… Definetely enough market power to define a standard.

1 Like
16

Probably still better than Z machine or Glulx though.

Once WASM has stabilized a bit someone can make a spec for putting IF in WASM (like specifying a metadata section with the IFID and more if needed) and the runtime provided and expected interfaces. I’ll probably do something like this once async ships for the component model.

1 Like
17

All my stuff is in WASM too. I show pictures, play audio and video all in WASM. Bwahaha! Would be up for a spec collaboration sometime.

Here’s a silly game.