Mac VoiceOver users wanted for testing (again)

Technical Development Interpreters
1

This is the first part of my planned update of the VoiceOver functionality of Spatterlight.

It only adds one single bare-bones function: the identification of menus. If it thinks the game is showing a menu, it will speak the currently selected line. That’s it! You still have to navigate and select exactly as before.

It supports Infocom InvisiClues-style menus, Beyond Zork menus, Hugo menus and Tads 3 hint menus.

I’m particularly interested in any regressions from previous versions, but also false positives and negatives: menus that are not identified as such and vice versa.

There are still plenty of other problems with VoiceOver in Spatterlight, and I hope to address some of them in the next version. I’m happy for any bug reports or feature requests, whether menu-related or not.

3 Likes
2

This is the second, but hopefully not last, part of the VoiceOver update. This reworks how Spatterlight decides which newly printed text to speak, and adds a custom hyperlinks VoiceOver rotor (requires macOS 10.13 or later.)

Previously, only new text in the currently focused window, typically the main buffer window where command lines are entered, would be read automatically. Now we look through all the windows and try to decide which is the main window. If this has no new text to speak, we look for new text in the other windows. This actually works more reliably than it may sound. Games which separate input and text output into different windows, such as Tads 3 menus, now speak new text automatically when they should.

I’ve also added a custom hyperlinks rotor, activated by the shortcut VO + U (and then left or right arrow until you find it. It won’t be there if the game has no hyperlinks.) It is similar in functionality to opening the built-in item chooser (VO + I) and filtering on “hyperlink”, but hopefully a little more convenient. It tries to sort the links in an intuitive order. Of course, there is not a lot of games with hyperlinks supported yet, but at least The Colder Light and Dead Cities should now be playable using only this rotor.

3

This is the last major part of this VoiceOver overhaul. I guess there might still be some polishing left to do.

Most of this release is code clean up, but many glitches are gone. There should be a noticeable improvement.

Just to recap, back in 0.5.9b Spatterlight borrowed the main speech actions from Zoom:

  • Repat Last Move (shortcut Alt + Command + left arrow)
  • Speak Previous Move (shortcut Alt + Command + up arrow)
  • Speak Next Move (Alt + Command + down arrow)
  • and added Speak Status (Alt + Command + Backspace).

This update adds an alternative way to navigate previous moves: the Command History rotor, which presents the last 50 commands as a list (unless the game has cleared the screen). Selecting a command in the list will select the corresponding text in the scrollback text.

There is also a new Game Windows rotor, which simply lists all buffer and grid windows shown by the game. At the moment it skips any graphics windows for the sake of simplicity, as some games add a lot of empty graphics windows for layout purposes.

All custom rotors are available by pressing VO + U, and can be filtered by typing any text. I’ve also added free text search, VO + F.

Please try this out and let me know what you think! In particular games with menus, links and unusual window layout.

Ping @Jlvp1998, @Morningstar, @blindHunter and anyone else with an interest in accessibility testing on the Mac.

1 Like
4

Wow, this looks really good! My testing so far has been relatively limited, but here are my observations:

  1. I’m particularly enjoying the ability to navigate in-game menus with arrow keys. There was one instance where Voiceover interpreted the first item in an 11-item menu as “item 3 of 13”, but I’m pretty sure that was a game-specific fluke since I haven’t been able to reproduce it anywhere else.
  2. when refocusing on the primary game window, Voiceover always begins reading from the very top of the text scroll back. However, selecting the last entered command via the history rotor is an easy way to jump back down to the point where I left off.
  3. I realize I actually have not dealt with many games that make use of strange or complicated layout features. Are there any titles in particular that I might try experimenting with? I suppose there’s always Bureaucracy, but I think those on-screen forms may be an entirely different can of worms.
  4. This may not necessarily have anything to do with the new update, but after incurring a negative score early on in Zork I, the status window displays 65256 instead of the expected -10.
5

There was one instance where Voiceover interpreted the first item in an 11-item menu as “item 3 of 13”

Ouch, which game?

It seems likely that there will be situations where this function mistakes something else for a menu and breaks the game, so there should probably be a toggle to turn it off.

when refocusing on the primary game window, Voiceover always begins reading from the very top of the text scroll back

Yes, this was the first thing that annoyed me when I tried using VoiceOver with interactive fiction, so it is kind of a major defeat that it still happens.

In Spatterlight 0.6b and earlier, there was an extra “accessibility element” that covered the actual text view and only spoke the text that was visible, that you had to navigate past to get to the real text view behind it, but once you got in there, it would still speak from the very beginning just like now. I thought it was inelegant to have two elements representing the same text (along with a third for the scroll view), and that the “top” element kept getting in the way, so I removed it. Perhaps I should just put it back.

I spent a lot of time implementing a separate input field, which meant that I could set the main text view to “static navigable text”, which (most of the time) does not start reading from the very top. But again, that introduced an extra element that kept getting in the way, and also caused a number of other glitches.

For a long time, the final update of this VoiceOver overhaul was supposed to be the introduction of this separate input field, perhaps released along with a separate version without it, so that testers could compare the two and say which one was the best. But then I myself compared playing a game with an input field to playing without it, and the latter was so much less annoying, so I just ran with that.

Perhaps the input field can still be salvaged. I’ll see if I can compile a version with it for you to try out.

EDIT: I did. Try it out if you are curious. Note that I added another version without the input text field as well, just to make sure that the codebases were the same for easy comparison.

I realize I actually have not dealt with many games that make use of strange or complicated layout features. Are there any titles in particular that I might try experimenting with? I suppose there’s always Bureaucracy

Three games that I tested with were A Colder Light, Dead Cities, and My Angel. All of them Jon Ingold games! The first two use hyperlinks for input, and the third uses input in the status bar (although it has a more screen reader friendly standard mode.) A Colder Light is meant to be played in a web browser, but that does not work very well with VoiceOver, at least it did not when I tried it in Safari.

I actually considered making the Bureaucracy intro form accessible, and now that you mentioned it I’ll have to at least give it a try.

Your fourth point sounds like an easy fix. What is a quick way to get a negative score in Zork?

6
  1. The game that apparently broke the menu system was Winter Wonderland by Laura A. Knauth. All other menu items were announced correctly, only the first one seemed to want to be different.
  2. I ran a few tests both with and without the added text input field, and I do see what you mean when you say it’s kind of annoying to have to deal with an extra element. I personally feel that both versions have their respective merits, but I’d be curious to see what other screen reader users think.
  3. The only one out of those three games I’m familiar with is My Angel, and I’m guessing I must have been in screen reader mode when I played it since I never noticed the unusual layout. I will definitely be taking a look at the other two titles.
  4. To end up with a negative score: open the mailbox, take the leaflet, and throw it at yourself without having previously earned any other points. I’m now wondering if this is also game-specific; I’ve since managed to incur a negative score in at least one other game and have everything work as expected.
7

Thanks! Here is another version that fixes the bugs you mentioned and re-introduces the extra accessibility elements that cover the text views.

8

Thanks, this looks good! I’ve been messing around with Dead Cities and I’m happy to report that the hyperlinks rotor works quite well.
Here’s another observation that may be worth noting. If I navigate to a different application while a game is in progress and then jump back to Spatterlight, Voiceover will only read the text associated with the last move I made. This doesn’t appear to be version-specific–I was running comparisons between 0.6.3b and 0.6.5b when I made the discovery.

9

That is deliberate. Any time the game window gets focus, VoiceOver will say something like: “Spatterlight, Zork I, window, edit text, input at end of text,” followed by the entire game text from the top of the scroll back. And I don’t want it to speak the entire scrollback, so Spatterlight will interrupt VoiceOver by speaking the output of the last move instead.

It is a kludge, but it is the best I can come up with. Of course, we could have it speak the visible text in the scroll view instead of just the last move. Would that be better?

10

On the contrary, I think that if we could get something similar to happen when jumping between windows in the application itself, all our Voiceover scrollback issues would be a thing of the past. When I come back to the primary game window after having looked at something else, oftentimes I just need to be quickly reminded of where I am or what I’ve just done. The last text output does the job perfectly, and if I want to go back further I can either use the search function or the very handy command history rotor :slight_smile:

1 Like
11

What you write makes sense, but is unfortunately tricky to implement without breaking everything, so it will have to wait a little longer.

In the meantime, here is a new version that fixes a crash that would occur if you deleted all old preferences. It also adds VoiceOver support for the Bureaucracy form:

https://github.com/angstsmurf/spatterlight/releases/download/v0.6.6b/Spatterlight.zip

12

Totally understandable, and thank you so much for working through all this with me. I’m looking forward to getting into the meat of Bureaucracy and I will definitely let you know if I find anything else worth mentioning. By the way, I’m glad to see that you found the Applevis community :slight_smile:

13

well, I tried the new update, and, I like it, however, there is no way to save the transcript like with the older version, under the file menu.

14

I think the menu item was just renamed to “Save Scrollback.” Unless there is something else broken.

The renaming was done to avoid confusion with the SCRIPT command that will usually save all output of the main window, while the menu item Save Scrollback/Transcript only saves the scrollback, not any output from before the last time the game cleared the screen.

15

Here is today’s update: https://github.com/angstsmurf/spatterlight/releases/download/v0.6.7b/Spatterlight.zip

It fixes the crash that would occur when stepping through previous moves in the Bureaucracy form. Also fixes the Inform Adventure help menu.

16

if that’s the case, it shows it is dimmed, meaning, it cannot be pressed.

17

if that’s the case, it shows it is dimmed, meaning, it cannot be pressed.

As long as there is a game running with a buffer window, it seems to be working as intended. Of course, that means it will be dimmed out when there is no buffer window, as is often the case when the game is showing a menu, or the Bureaucracy form. I guess things would be less confusing if the game offered to save the contents of the grid window instead in those situations. I’ll look into it.

EDIT: It currently also only works if the buffer window has focus.

18

this might be off topic but, is there a possible way, for someone to make an app for the interactive fiction comunity forum, that way, it could be easier to create a topic and, respond to one. if that were possible, I think it should be for android, mac, ios and, windows, and, lynux, if people use that system.

19

That seems like a prime example of something that should be a new topic, where the people responsible for the forum are more likely to read it.

20

well, it seems the transcript was changed to scrollback, however, unlike with the older version, you are not unable to save the complete one, with the newer version, it only saved the last part, where as with the older version, you can save all of it, the only thing is, with the older version, the menus, are not likely to be navagible.