Glk extension proposal: images in buffer windows

Technical Development Specifications
1

Glk provides standard ways to check the size of an image (glk_image_get_info), and to draw it either at that size (glk_image_draw) or at a different size (glk_image_draw_scaled). You can also check the size of a graphics window (glk_window_get_size), allowing you to customize your graphics for any type of device or screen.

But there’s one thing missing. There’s no way to check the size of a text buffer window in pixels (glk_window_get_size returns the size in characters, and the size of a character in pixels is entirely up to the library). Which makes it nigh-impossible for the author to choose an appropriate size, either for the image itself, or to have Glk scale it.

One solution is to let the interpreter handle it. This is what I’ve generally done: trusting the interpreter to draw my images at an appropriate size (at the library author’s discretion), like how I trust the interpreter to draw my text with an appropriate line height, appropriate font, and so on. Unfortunately, this does not seem to be a reliable solution.

Dannii has suggested another option: a way to query the size of the screen (or of a text buffer window) in pixels. Then the author can be responsible for deciding exactly how they want it to look on different window sizes. I suggest a different solution, though.

I’m not a graphic designer, and someone writing a mobile Glk library probably knows much more than I do about how images should be scaled on mobile. There’s also one argument to the image-drawing functions that’s currently entirely unused in text buffer windows (it exists to provide coordinates in graphics windows).

I propose that this argument (val2 in the specification) be used to suggest how the image should be scaled. In particular, it can take the values:

  • imagescale_Raw (0): the current behavior. If scale parameters are provided, interpret them as a size in pixels. If no scale parameters are provided, the behavior is unspecified (or could mean “always draw at full size” if we want to specify it fully and change the existing behavior).
  • imagescale_Proportional (1): if scale parameters are provided, interpret them as a fraction of the window width and height (probably as fixed-point values but could also be floats I suppose). If not, scale the image down proportionally so it fits the current window in both dimensions.
  • imagescale_Good (2): use a “good” scaling, whatever that means to the author of the library. We trust library authors to decide what is good typography; this option trusts them also to decide what’s good image placement.

Any thoughts on this? If we go with Dannii’s proposal, these options could also be handled on the game side, with e.g. an Inform extension that measures the size of the window, decides whether it’s mobile or desktop, and applies some design principles based on that. But I think letting the authors of individual Glk libraries have a say based on the devices they’re targetting can give better results overall.

2 Likes
2

Dannii has suggested another option: a way to query the size of the screen (or of a text buffer window) in pixels.

The problem with this approach is that the size of a text window isn’t fixed, and (unlike graphics windows) you can’t redraw images there. So if the player resizes the game window, you wind up with mis-proportioned images in the text pane.

I propose that this argument (val2 in the specification) be used to suggest how the image should be scaled.

This is intuitively appealing. It’s easier for the interpreter to do this work than to try to convey screen layout info into the game and then ask the author to make use of it properly. Also, the interpreter can apply size changes to images displayed in text windows.

Some points:

  • “Good” seems like a trap. I don’t know what the interpreter would do other than “100% proportional”, and I don’t think authors would either. One might give the image some whitespace margin, same as text gets, but I think that would come automatically with Proportional.

  • “a fraction of the window width and height”: Text windows scroll vertically, so perhaps you only want to fit images to the current width? (Allowing vertical scrolling to view the whole image height.) But there’s also a use case for “full-screen” illustrations. Maybe ProportionalSize and ProportionalWidth options?

2 Likes
3

Perhaps for Proportional, if only one of height and width is provided (the other is zero), it scales to fit that dimension? If both are provided, it scales to fit whichever is smaller. So (100%, 0) gives a full-width image, (0, 100%) gives a full-height image, and (100%, 100%) gives you the biggest size that will fit on the screen without scrolling, whatever that may be.

I’m also not really sure what “Good” would do except that it would be nice to say “you library authors know more about graphic design than me”. It’s like how I wish LaTeX would place figures in typographically-good positions without me having to do it manually.

4

I agree, I don’t think a vague “good” option will be viable.

What I do think would work is if val2 was specified to mean a percentage max width, with 0 meaning 100%. This would be both backwards and forwards compatible, and would mean that we could also add fitting to window width by default. Max width would mean that we could set for example a margin image to be 25%, but if the window was super wide then the image would not be stretched more than its actual size.

We could encourage but not require that interpreters update the image width if the window is resized.

5

I would prefer the flexibility of imagescale_Proportional (and then the width and height values from the _scaled call), because, for example, I might want a big illustration to appear as large as possible without requiring scrolling. This could end up being 100% width on mobile (generally portrait orientation), but only 25% width on desktop (generally landscape orientation). If I specify 100% width, it’ll be enormous and require vertical scrolling for desktop players, but if I specify 25% width, it’ll be tiny for mobile players.

You mention not stretching images more than their actual size as a way to avoid this problem, but that’s not necessarily a useful limit: an image could still end up either tiny on a high-density retina display, or enormous on my zoomed-in monitor.

Respecting both the width and height of the window while scaling would account for this automatically, while also letting authors fine-tune the display in other ways. A fleuron between chapters might want to be 100% width but a maximum of 5% height, for example, to avoid obscuring the text on an especially-wide screen.

You’re probably right about Good being very difficult to define. It was mostly a way of punting the problem off to library developers, but a good default could also be established by Inform extension writers, and “scale proportionally to be no wider than the screen and also no taller than the screen” might be that default.

1 Like
6

What if val2 specified a percentage to be used for both max-height and max-width? This would handle both making sure that large images always fit on the screen, as well as your example of a fleuron: set it to be scaled to 1000px wide by 10px high, and only the width will be constrained by the max-height. (And being able to specify a set 10px high flueron is better than a 5% one.)

Hmm, but a question would be that if you use glk_image_draw_scaled should it try to maintain the aspect ratio or not? For normal scaling operations you’d want to the scaled dimensions to represent the new aspect ratio, which should be maintained after the max-height/width is applied. But for a fleuron you wouldn’t want the aspect ratio to be maintained as it should always be 10px high.

I liked my original idea because it was simple and would be both backwards and forwards compatible (and would let us take care of too-large images by default). For glk_image_draw it still makes sense as the aspect ratio should always be preserved. It’s just more confusing for glk_image_draw_scaled. We could add a flag in val1 to say that the aspect ratio isn’t to be preserved after applying max-height/width, but that’s less backwards compatible. We could add a new function too. Or maybe there aren’t many calls to glk_image_draw_scaled that would matter, and we could just say that it doesn’t preserve aspect ratio?

7

Coming back to this, since the problem has come up again. How about this?

The upper 16 bits of val2 indicates what to do with the width. The lower 16 bits of val2 indicates what to do with the height. Possible values are:

  • imagescale_[XY]_Absolute ($0000): If a scale parameter is provided, interpret it as an absolute size in pixels (the current behavior). If a scale parameter is not provided, use the full, original dimensions of the image (currently undefined behavior).
  • imagescale_[XY]_Relative ($0001): If a scale parameter is provided, interpret it as a maximum fraction of the size of the current window. If not, use the full current size of the image. The image will be scaled down, preserving the aspect ratio, so that this maximum is not exceeded.
  • imagescale_[XY]_Proportional ($0002): If a scale parameter is provided, interpret it as a fraction of what this dimension would be in order to preserve the current aspect ratio (that is, a fraction of 1 for the height means “calculate the width, then use the height that will preserve the original aspect ratio”; a fraction of 1/2 means “calculate the width, then use the height that will halve the original aspect ratio”). If not, treat it as 1 (that is, preserving the original aspect ratio). If both width and height are proportional, the result is undefined behavior.

“Not provided” means that glk_image_draw was used (not providing a scale parameter for either dimension), or glk_image_draw_scaled was used, and the relevant parameter was set to 0. An image with zero size doesn’t seem useful.

8

Actually, I suppose the more idiomatic option would be to make each of these a flag, rather than passing two enums in one value.

9

I think we should keep the size parameters of glk_image_draw_scaled as referring to the new size of the image, so that the rest of the process won’t even know that the image was ever anything else.

I think there are these constraint options:

  • Whether we want to expand to a max dimension or only constrain if the image would be bigger.

  • Whether to keep the aspect ratio or not

  • Whether to expand/constrain both height and width, or only one or the other. When considering both dimensions, if keeping aspect ratio, then it would fit to the smaller (ex: on a wide screen, 25% would constrain to the height not the width)

    I don’t think it makes much sense to talk about constraining only the height of an image in a buffer window, because it will probably scroll away so quickly. Can you think of a concrete example where you’d want only the height to be constrained? But it can be included too for completeness sake.

val2 could then be a bitmask. The lower 16 bits specify a percentage. The top 16 have these meanings:

bit meaning
1-0 0: constrain both height and width
1: constrain width only
2: constrain height only
2 0: constrain to max
1: expand
3 0: maintain aspect ratio
1: don’t maintain aspect ratio

This also would allow for what I said above, letting val2=0 mean a 100% max, so that we can have fitting by default. (Except that before I just said it would be fitting the max width by default, here it would be both dimensions).

1 Like
10

It comes back to the fleurons again: I want them to cover the full width of the screen, but not take up much vertical space if the screen is wide. That’s the main case I can think of for scaling one but not the other.

11

Yep, so in that case you’d pass $$1101 in the top half of val2 and 100 in the bottom half.

1 Like
12

Passing a percentage in the lower eight bits and a bunch of flags in the upper 24 bits feels right on the Z-machine, but a little clunky in the Glulx world. That’s my main reason for wanting to pass the percentage in the _scaled arguments instead of in val2.

But I do think using a field of flags is more elegant than my original proposal.

13

My suggestion means that the difference between glk_image_draw and glk_image_draw_scaled would remain unchanged, and all the new parameters would be passed in the one existing unused argument. Keeping all the other arguments unchanged is a big plus IMO, at the cost of combining things into a bitmap.

Though I was imagining it being 16 and 16, not 24 and 8. Doesn’t really matter though as neither the flags nor the size as a percentage would need more than a byte each.

Probably should also specify that the width is relative to the window excluding its margins, as margins aren’t accessible to the game so you couldn’t accurately know how wide they are, if they exist at all. So a 100% image would be the same width as a full-justified paragraph.

14

That makes sense to me, re margins. And you’re presumably the one who would end up implementing this, since I’m not the maintainer of any Glk library, so I’m happy to defer to you on those details! But of course zarf, as the maintainer of the spec, gets the final call.

If so many parameters are needed, we could also add a brand new Glk call, but the fact that the existing ones have one parameter unused in text buffer windows (and this isn’t needed in graphics windows, since the game can get and set their dimensions directly) is so tempting…

15

Your problem is 16:9 screens. If a picture is, itself 16/9, then there is no space for text at all underneath it!

Welcome to the “text and picture prolem” :slight_smile:

16

Indeed! The interpreter behavior I’d expect in that case is the same as if you print several screens of text all at once: it stops at the top of the new content and waits for you to scroll manually to the bottom, or hit the “MORE” prompt to do it automatically.

1 Like
17

I guess I should weigh in finally. (I’ve been watching these threads but also distracted with NarraScope and GDC prep.)

Since this is new behavior and not all interpreters will catch up, it would make sense to have a gestalt selector for it. If there’s a gestalt selector, I think we might as well have a new call: glk_image_draw_scaled_ext(). Then we wouldn’t have to cram bits.

I might not have time to write this up this week (again, GDC) but I should be able to do it next week.

Thanks for all the discussion while I was not here!

5 Likes
18

Sounds great!

My last note would be: is there any real benefit to having an image be rendered larger than the current window (so that it’s cut off)? I don’t think there really is; mandating that glk_image_draw should scale it down proportionally to the window’s dimensions seems like it would do The Right Thing with current code, with no more complications needed. I think that’s what web interpreters already do.

(And if you do want the image to render at full size regardless of the window size, well, glk_image_get_size and glk_image_draw_scaled have you covered.)

19

Should the non _scaled function also have an _ext function? I’d think most people would probably want to use these constraints without also scaling the image manually.

And yeah, I think the default without calling _ext should be to fit within the window width.

20

We shouldn’t need four functions, if one of them is just “omit some arguments”.

And yeah, I think the default without calling _ext should be to fit within the window width.

That is, glk_image_draw should scale down to fit the window width if the image’s natural size is too wide. That makes sense to me, yes.