Glk extension proposal: images in buffer windows

Glk provides standard ways to check the size of an image (glk_image_get_info), and to draw it either at that size (glk_image_draw) or at a different size (glk_image_draw_scaled). You can also check the size of a graphics window (glk_window_get_size), allowing you to customize your graphics for any type of device or screen.

But there’s one thing missing. There’s no way to check the size of a text buffer window in pixels (glk_window_get_size returns the size in characters, and the size of a character in pixels is entirely up to the library). Which makes it nigh-impossible for the author to choose an appropriate size, either for the image itself, or to have Glk scale it.

One solution is to let the interpreter handle it. This is what I’ve generally done: trusting the interpreter to draw my images at an appropriate size (at the library author’s discretion), like how I trust the interpreter to draw my text with an appropriate line height, appropriate font, and so on. Unfortunately, this does not seem to be a reliable solution.

Dannii has suggested another option: a way to query the size of the screen (or of a text buffer window) in pixels. Then the author can be responsible for deciding exactly how they want it to look on different window sizes. I suggest a different solution, though.

I’m not a graphic designer, and someone writing a mobile Glk library probably knows much more than I do about how images should be scaled on mobile. There’s also one argument to the image-drawing functions that’s currently entirely unused in text buffer windows (it exists to provide coordinates in graphics windows).

I propose that this argument (val2 in the specification) be used to suggest how the image should be scaled. In particular, it can take the values:

  • imagescale_Raw (0): the current behavior. If scale parameters are provided, interpret them as a size in pixels. If no scale parameters are provided, the behavior is unspecified (or could mean “always draw at full size” if we want to specify it fully and change the existing behavior).
  • imagescale_Proportional (1): if scale parameters are provided, interpret them as a fraction of the window width and height (probably as fixed-point values but could also be floats I suppose). If not, scale the image down proportionally so it fits the current window in both dimensions.
  • imagescale_Good (2): use a “good” scaling, whatever that means to the author of the library. We trust library authors to decide what is good typography; this option trusts them also to decide what’s good image placement.

Any thoughts on this? If we go with Dannii’s proposal, these options could also be handled on the game side, with e.g. an Inform extension that measures the size of the window, decides whether it’s mobile or desktop, and applies some design principles based on that. But I think letting the authors of individual Glk libraries have a say based on the devices they’re targetting can give better results overall.

1 Like

Dannii has suggested another option: a way to query the size of the screen (or of a text buffer window) in pixels.

The problem with this approach is that the size of a text window isn’t fixed, and (unlike graphics windows) you can’t redraw images there. So if the player resizes the game window, you wind up with mis-proportioned images in the text pane.

I propose that this argument (val2 in the specification) be used to suggest how the image should be scaled.

This is intuitively appealing. It’s easier for the interpreter to do this work than to try to convey screen layout info into the game and then ask the author to make use of it properly. Also, the interpreter can apply size changes to images displayed in text windows.

Some points:

  • “Good” seems like a trap. I don’t know what the interpreter would do other than “100% proportional”, and I don’t think authors would either. One might give the image some whitespace margin, same as text gets, but I think that would come automatically with Proportional.

  • “a fraction of the window width and height”: Text windows scroll vertically, so perhaps you only want to fit images to the current width? (Allowing vertical scrolling to view the whole image height.) But there’s also a use case for “full-screen” illustrations. Maybe ProportionalSize and ProportionalWidth options?

2 Likes

Perhaps for Proportional, if only one of height and width is provided (the other is zero), it scales to fit that dimension? If both are provided, it scales to fit whichever is smaller. So (100%, 0) gives a full-width image, (0, 100%) gives a full-height image, and (100%, 100%) gives you the biggest size that will fit on the screen without scrolling, whatever that may be.

I’m also not really sure what “Good” would do except that it would be nice to say “you library authors know more about graphic design than me”. It’s like how I wish LaTeX would place figures in typographically-good positions without me having to do it manually.

I agree, I don’t think a vague “good” option will be viable.

What I do think would work is if val2 was specified to mean a percentage max width, with 0 meaning 100%. This would be both backwards and forwards compatible, and would mean that we could also add fitting to window width by default. Max width would mean that we could set for example a margin image to be 25%, but if the window was super wide then the image would not be stretched more than its actual size.

We could encourage but not require that interpreters update the image width if the window is resized.

I would prefer the flexibility of imagescale_Proportional (and then the width and height values from the _scaled call), because, for example, I might want a big illustration to appear as large as possible without requiring scrolling. This could end up being 100% width on mobile (generally portrait orientation), but only 25% width on desktop (generally landscape orientation). If I specify 100% width, it’ll be enormous and require vertical scrolling for desktop players, but if I specify 25% width, it’ll be tiny for mobile players.

You mention not stretching images more than their actual size as a way to avoid this problem, but that’s not necessarily a useful limit: an image could still end up either tiny on a high-density retina display, or enormous on my zoomed-in monitor.

Respecting both the width and height of the window while scaling would account for this automatically, while also letting authors fine-tune the display in other ways. A fleuron between chapters might want to be 100% width but a maximum of 5% height, for example, to avoid obscuring the text on an especially-wide screen.

You’re probably right about Good being very difficult to define. It was mostly a way of punting the problem off to library developers, but a good default could also be established by Inform extension writers, and “scale proportionally to be no wider than the screen and also no taller than the screen” might be that default.

1 Like

What if val2 specified a percentage to be used for both max-height and max-width? This would handle both making sure that large images always fit on the screen, as well as your example of a fleuron: set it to be scaled to 1000px wide by 10px high, and only the width will be constrained by the max-height. (And being able to specify a set 10px high flueron is better than a 5% one.)

Hmm, but a question would be that if you use glk_image_draw_scaled should it try to maintain the aspect ratio or not? For normal scaling operations you’d want to the scaled dimensions to represent the new aspect ratio, which should be maintained after the max-height/width is applied. But for a fleuron you wouldn’t want the aspect ratio to be maintained as it should always be 10px high.

I liked my original idea because it was simple and would be both backwards and forwards compatible (and would let us take care of too-large images by default). For glk_image_draw it still makes sense as the aspect ratio should always be preserved. It’s just more confusing for glk_image_draw_scaled. We could add a flag in val1 to say that the aspect ratio isn’t to be preserved after applying max-height/width, but that’s less backwards compatible. We could add a new function too. Or maybe there aren’t many calls to glk_image_draw_scaled that would matter, and we could just say that it doesn’t preserve aspect ratio?