Glk extension proposal: images in buffer windows

The difficulty with glk_image_draw_scaled is that omitting the size arguments will not draw anything. Though I guess glk_image_draw_scaled_ext could be defined differently so that passing 0 for the size uses the original size…

It’s unlikely anyone is depending on the 0=skip drawing function so perhaps it could be redefined. Times like this it would be handy to have a exhaustive record of all the Glk function calls in games on the archive. But that would be very hard to do - we couldn’t just do a static analysis because of the vararg calls.

“Don’t draw anything for size 0” seems like another of those behaviors that’s unlikely to actually be wanted. I think it would be fine to redefine it as “use original size”.

I think I had some reason for it, actually. I don’t remember what it is, but “optionally display an image” feels useful, even though it’s not the only way to do it.

Hey, I bet you thought I’d forgotten about this thread… Here’s my current idea.

We will have a glk_image_draw_scaled_ext() function:

glui32 glk_image_draw_scaled(winid_t win, glui32 image, glsi32 val1, glsi32 val2, glui32 width, glui32 height, glui32 ext);

The new ext argument contains three flags. (I may rename the argument and the flags, but I want to get this written down before I go to bed, so I’m just naming stuff on the fly.)

  • WidthRule (2 bits): can be ORIG, FIXED, or WIDTHRATIO.
  • HeightRule (2 bits): can be ORIG, FIXED, or ASPECTRATIO.
  • MaxWidth (1 bit): boolean.

What do these mean for buffer windows?

  • WidthRule_ORIG: width argument is ignored; displayed width is the image original width.
  • WidthRule_FIXED: displayed width is given by width argument in pixels.
  • WidthRule_WIDTHRATIO: displayed width is the buffer window width multiplied by the width argument, which is interpreted as a 16.16 fixed-point value. (So 0x10000 means 100% of the buffer window width.)
  • HeightRule_ORIG: height argument is ignored; displayed height is the image original height.
  • HeightRule_FIXED: displayed height is given by height argument in pixels.
  • HeightRule_ASPECTRATIO: displayed height is based on the displayed width, maintaining the aspect ratio multiplied by height (as 16.16). That is, if height is 0x10000, the original aspect ratio is maintained. If height is 0x20000, the image is stretched vertically by a factor of 2.

If the MaxWidth flag is true, the image gets a maximum display width of the buffer window window. If it’s too wide, it’s scaled down (strictly proportionally) to fit.

If MaxWidth is false, there is no such limit. (But, to be clear, images must be clipped if they’re too large for a window.)

The current defined behavior of both glk_image_draw() and glk_image_draw_scaled() are MaxWidth=false. But, as discussed above, we would like to change this default to be MaxWidth=true. That’s change in spec behavior but it’s almost certain to be an improvement.

Thus, in the new world, the old glk_image_draw() call becomes a shortcut for:

glk_image_draw_scaled_ext(win, image, val1, val2, _, _, WidthRule_ORIG|HeightRule_ORIG|MaxWidth);

Note that the fifth and sixth arguments are ignored because of ORIG|ORIG.

Similarly, the old glk_image_draw_scaled() call becomes a shortcut for:

glk_image_draw_scaled_ext(win, image, val1, val2, width, height, WidthRule_FIXED|HeightRule_FIXED|MaxWidth);

We keep the rule that if the final width or height is zero, the image is not drawn. (This will happen if the width or height argument is zero, unless we’re in the ORIG rule case.)

As you see, this proposal completely bags the idea of fitting an image into the vertical span of the buffer window. I think there are just too many headaches down that road.

In particular, I would really like the image handling to be implementable in HTML by using pure CSS, with no calculated reference to the window size. I think all the cases in my proposal can be. (I haven’t tested this yet!) But a “max-height” constraint could not be, because the DOM container height is the scroll height, not the visible height.

(Maybe there’s a way to do it with the CSS calc() feature, but it seems like a big added headache even if it is possible.)

Also: remember that the text buffer window can be resized freely by the player. This will automatically resize images if they are defined relative to the window width (or if they hit the MaxWidth constraint). Again, this should work “by magic” (and in an intuitive way) if it’s all handled at the CSS level.

2 Likes

I suppose the extended call could be used in graphics window as well. But it would just be a matter of calculating the desired size and then drawing the image with that size. (The old behavior, specified differently.) No CSS or autoresizing – graphics windows have to be manually redrawn, and this would not change that.

Is it time to consider images in grid windows? Does anybody care about that? That would be a matter of drawing an image on top of the grid text… and then some way of removing the image, since it can’t scroll away… I dunno, maybe that’s too messy to get into.

Having a dynamically updated max height seems very difficult in CSS, but simply setting a max height to the current window height would be simple for GlkOte to do.

I think one of the most useful things you could do is to display an image at say a max of 25% height and 25% width. That doesn’t seem possible with what you’re describing, but would be with a max height option.

Do you know how to implement this in CSS already? It seems quite complex with all the width options too.

Eww no!

But perhaps there could be a basic minimalistic function to draw text in a graphics window?

2 Likes

How about an opacity/alpha parameter?

I’m good with zarf’s proposal! It hits the points I care most about.

It would also be nice to have a way to specify alpha for an image, but that seems orthogonal to this proposal. Still, while we’re messing with the graphics spec, that’s a fair time to introduce some new functionality too!

No, I haven’t done any test implementation. That’s next.

I agree that “draw an image that is guaranteed to fit on the screen” is a meaningful use case. I’m just not sure it’s worth the implementation complexity.

See https://eblong.com/zarf/tmp/glkimagetest/glkimgtest.html.

Remember that the interpreter (the glkapi layer, for Quixe) knows all image sizes. So glkote.js doesn’t have to handle ORIG. This has always been true; both glk_image_draw() and glk_image_draw_scaled() pass explicit image sizes to GlkOte.

So glkote.js only has to be updated for WIDTHRATIO, ASPECTRATIO, and the MaxWidth flag. I think all the interesting combinations are tested in the above page.

For the GlkOte JSON protocol, I am imagining these cases:

width:W, height:H    // the current usage

widthratio:R, height:H   // R is a float; 1.0 means 100% of the window width

width:W, heightratio:R   // R is a float; 1.0 means 1/1 aspect ratio

widthratio:R1, heightratio:R2

Include winmaxwidth:true to give the image a max-width of the window width. winmaxwidth:false is the alternative (overwide image gets clipped). If winmaxwidth is omitted, it defaults to true.

2 Likes

One thing at a time. :)

1 Like

I had no idea there was an aspect-ratio CSS property! That certainly makes this a lot easer.

I like the idea of starting with the JSON protocol and then working back to what Glk function arguments would make sense…

Is this for the flueron use case? Fixed height, with percentage of window width?

If the interpreter is providing all image sizes then I’m not sure what use this has. Can’t it just provide the calculated height?

So this would let you set the width to a percentage of window width, and then also set an aspect ratio? That’s more useful, and would probably be the most common use of these new options.

Could a ratio of 1.0 in all of these be left out?


So, thinking just of the protocol, and assuming the interpreter can do any calculations to image sizes, then at present we can already handle changing the aspect ratio of an image, upsizing it, or downsizing it (like for high DPI images.) What we can’t do is set image sizes relative to the window width. For that I think we have three main use cases:

  1. Fleurons: set a constant height, but percentage width
  2. Set an image to a percentage of width keeping to an aspect ratio (could possibly be longer than the window height)
  3. Set an image to have a max percentage width/height

It’s this third one that has been discussed above, but isn’t supported in Zarf’s proposal here, except for the case of a width of 100%. Ideally also with a max-height.

Adding a property to set the max width as a percentage rather than just winmaxwidth: true would be pretty simple.

Unfortunately I can’t get a max-height to work as well, even if specified as a pixel height (which is what the interpreter could do, calculating the percentage of the current window height.) Unfortunately CSS won’t maintain the aspect ratio when both max-width and max-height are set. Maybe it’s possible with calc(), but for now probably not a viable option.

Yes.

I think so, but it’s tidier to support all combinations of options.

Could a ratio of 1.0 in all of these be left out?

My test had all square images, but the default case will be the image’s natural aspect ratio, not square.

(Hm. I wonder if these ratios should be 16/16 rationals rather than 16.16 fixed-point, to avoid rounding errors for cases like 640x480.)

(Or maybe I should just allow ratio -1 to mean the image’s natural aspect ratio.)

Adding a property to set the max width as a percentage rather than just winmaxwidth: true would be pretty simple.

So like “width=600, max-width=(50% window width)”? I can see that being used, and it’s just as doable on the CSS side.

Unfortunately I can’t get a max-height to work as well

Yeah, I’m pretty sure there’s no good way to do that.

1 Like

Sorry it’s taken so long for me to reply…

I think that while for the Glk API it’s probably simplest for the aspect ratio to be a multiplier of the natural aspect ratio, for the GlkOte protocol it should be the desired end ratio, which can be plugged straight into CSS’s aspect-ratio, which I think can just be a double. The Glk library can easily handle mapping between the two.

If you do add a max-width option, then from my tests it’s actually crucial to not set a CSS height, you have to use aspect-ratio instead.

Lastly, a little bikeshedding: GlkOte protocol properties don’t have to be just letters, we could have width% instead of widthratio and aspect-ratio instead of heightratio.