I am making a game console for interactive fiction - your thoughts?

Technical Development Tools
1

I am interested in making a game console for interactive fiction.

This would be, essentially, an enhanced screenreader for HTML-based Twine games.

This enhanced screenreader for HTML-based interactive fiction would focus on accessibility by utilizing refreshable Braille displays, tactile graphics, and/or self-voicing (for non-RenPy games).

This would be a physical game console, not a virtual console or emulator. I’m currently taking electrical engineering classes and this could be a project I pursue.

What are your thoughts? What would you want from an accessible game console for interactive fiction?

8 Likes
2

I would like to see on an IF console:

  • dark mode
  • ability to run parser games (minimum: z-machine).

It seems to me that you need a lot of hardware/electronics knowledge (which I hope you have), but I might be wrong.

3 Likes
3

Good display support (accessibility, dark themes, font preferences, etc) for Twine has always been tricky. Twine handles its own DOM and CSS. It’s difficult to get in there and improve anything the way we do with parser games.

If you can make it work, that would be a great boon. The ideas could be adapted to a web site as well.

6 Likes
4

I believe @Mewtamer mentioned that only a very small percentage of blind people can read braille. Having a very good built-in/customizable voice library would be key. I would still have a braille system though.

Good sounding speakers for if a player needs to be aware of the room they’re in and a headphone jack for full immersion. Possibly even physical volume sliders/knobs for sound and music. I’m just listing convenience wishes. Not deal breakers, obviously. Get the thing working first! :wink:

I love the idea of a tactile display. Would you be building your own from scratch? What kind of DPI would it have? I know very little about them. Can you animate them? Can you press on them and have a game understand the touched coordinates?

The most basic thing would be to take stock Twine games and have the console read and control them in a convenient manner for blind players. Some games might have to be designed from the get-go to be accessible, is my feeling, but who knows?

I would argue that most people who would benefit from an accessible console are on government assistance so if you can eventually make it affordable, that would be awesome.

2 Likes
5

The bit about tactile graphics sounds intriguing, and I’m curious to know what type of technology you had in mind for displaying dynamic in-game images. A couple of the accessibility tech companies have been experimenting with multi-line braille displays with this eventual goal in mind, but I’m not aware of any devices which currently have the braille real estate or dot resolution to show something as intricate as, say, a piece of hand-drawn cover art. As has already been mentioned, cost is going to be a significant factor as well—unfortunately electronic gadgets like this are not known for being affordable for the average consumer. For example, the Monarch by HumanWare and APH, a 32x10 display which has been generating a fair bit of excitement within the blind and visually impaired community due to its impending launch, is expected to retail somewhere within the ballpark of $18000 … oof!

2 Likes
6

The statistic I’ve heard is that between 1% and 10% of blind individuals are Braille Literate, and my hypothesis is that wide range in estimates comes down to different surveys having different thresholds for literacy. I myself know the braille alphabet and push come to shove, I can touch read uncontracted Braille, but my touch reading speed is downright glacial, to the point I struggle to keep letters in working memory long enough to string into words and words in working memory long enough to string into sentences., so I suspect I’d be counted in the 10%, but not the 1%.

I have previously used a Raspberry Pi 3 to play zcode games with espeakup to read the output from Frotz using just the Pi in a case, earphones, a wireless USB keyboard, and a USB battery pack about the size of the Pi. I never tried running a GUI on my Pi 3, but the general concensus on the Raspberry Vi mailing list seem to be that the Pi 3’s 1 GB of RAM wasn’t enough to run a DE plus a graphical screen reader and have anything leftover for running something like Firefox or LibreOffice, though reports is that the 4GB and 8GB RAM versions of the Pi 4 and the Pi 400(based on the 4GB version of the Pi 4) could handle running a full GUI with a screen reader, though the Pi 400 having a built-in keyboard comes at the lack of a 3.5mm audiojack, requiring the use of a USB sound adaptor, USB earphones/speakers, or bluetooth audio. I haven’t heard anything about accessibility on the Pi 5. I mention this as an example of an off the shelf solution that can already do most of what you’re talking about.

As for Braille displays and tactile displays…

Braille displays start at a few hundred dollars new and 20 and 40 cell displays seem to be the most common resolutions. The highest resolution braille display I know off, which is also the highest resolution tactile graphics display I know of, has 10 lines of32 cells each, or a resolution of 40 rows of pixels by 64 columns… and costs $20k. The Orbit Graffiti, the next highest resolution tactile graphics display I’m aware of, is also about 20 grand, has a 40*60 resolution, and doesn’t adhere to Braille standards. Needless to say, these are way out of the budget of most blind people.

As for DPi, Braille dots are about 1.6 mm in diameter with 2.5mm spacing between adjacent dots in the same cell(I believe that 2.5 is from center to center), and the spacing from a dot to its counterpart in an adjacent cell is about 6.5 mm. so about 10dpi within each cell with a gap about the size of a dot between cells. I believe the Graffiti has a DPI of like 5 or 6 as the specs I remember reading suggested it was around the size of a sheet of US letter printer paper or an iPad.

and to help put 4064 into perspective, the TI-83 has a resolution of 6496 and the original Gameboy a resolution of 160*144, or more than twice the total pixels and 8 times the total pixels.

As I understand it, the high prices regarding Braille and tactile displays are a combination of a market too small to drive economies of scale(blind people are a niche market to begin with, Braille readers are a niche within that niche, the tech would need to improve several orders of magnitude to gain any mainstream appeal), lots of tiny, independant moving parts(at a bare minimum, each cell requires 6 pegs 1.6mm in diameter that need to shift between two positions on demand, and most modern cells use 8-dot cells), and possibly the need to have a human assembly line worker insert all of those pins when assembling the devices.

Of course, if you can figure out a way to build a grid of a few thousand tiny pins that can independently move up and down several times a second and get the cost under a hundred bucks per unit, or even a grand a unit, well you’d probably make a lot of blind people quite happy.

12 Likes
7

Found this research paper on tactile displays and it covers the Graffiti device and others and dives into a new solution. It’s way over my pay grade and about 250 pages of academic mumbo jumbo, plus another 50 pages of citations n’ shit.

→ https://discovery.ucl.ac.uk/id/eprint/10176514/14/Bhatnagar_10176514_thesis_revision_sigs_removed.pdf

Just for those who might be interested in the technology.

1 Like
8

Screenreaders like Orca are written in Python. I can imagine making a desktop application (as Python is difficult to run online, even with WebAssembly) that enables this display support for Twine.

1 Like
9

If I understand correctly - are you saying the most useful/interesting part of a hypothetical interactive fiction console would be the tactile graphic display?

an off the shelf solution … can already do most of what you’re talking about

I have been researching tactile graphics because I do see this as well. As multiple comments have described, they are not easy to create.

2 Likes
10

I seems like the most straight forward strategy is the most expensive. Micro solenoids/actuators can get pricey. I wonder if there would be a way for rows of camshafts or something to raise and lower pins. If there’s a will, there’s a way!

I remember looking at the guts of an old mechanical calculator once. Those things were engineering marvels.

11

I imagine a talk with friends:

“I have a game console.”
“X box? PS? Switch?”
“Nah, a interactive fiction console with accessability.”

That would be great!

5 Likes
12

On an orthogonal subject, I remember listening to an Article in Choice Magazine Listening several years ago, possibly a full decade at this point. It was about electrostatic vibration, the phenomenon by which, if you cover a conductor with a thin insulator then electrify the conductor and run your finger over the insulator, the current alters the apparent texture of the surface. The article mentioned that someone had found that the apparent texture varied with both the frequency and amplitude of the current and had managed to integrate the effect into a touchscreen. Sadly, I have no idea what ever came of that project, no idea what issue of CML it was in or whwere CML sourced the article from originally(CML is an audio publication targetted at the blind, but their articles are sourced from other publications), and the prototype was called the Tesla Touch, so all one gets from a google search is stuff about the touchscreens in Tesla electric vehicles. Also, I understand one limitation is that the effect only works so long as one drags their finger across the surface.

Though, doing a little googling to try and find out if there are materials that expand/contract when exposed to light, I found the following article:

The article is about artificial muscles, but imagine integrating the light sensitive polymers mentioned in the article into a 3-d printed array of pins and putting said grid atop a oled screen. Then imagine making the pin grid sufficiently transparent and finding a version of the photocontracting polymer that is activated by infrared light so it could be combined with an RGB OLEd display… Not that integration with a visual display is important to a blind person, but again, there aren’t enough blind people to drive economies of scale, so a tacto-visual display that both blind and sighted people would use could drive prices down more quickly than a tactile-only display that only appeals to the blind. Audiobooks are fairly easy to find largely because the tech used for mainstream distribution of music works just as well for audiobooks for the most part* and there are probably more sighted people who would rather listen to someone else read a book than read it themselves than there are blind people. and many of the devices one would want speech on already have speakers, so adding speech is purely a software problem.

*Biggest caveat I can think of is that audiobooks can take a lot of discs to distribute as CDs that will play in a bog standard CD Player(e.g. the Narnia Books average 4 CDs each and are some of the shortest titles I own as physical audio books, the first volume of The Years of Lyndon Johnson is 34 discs). Also means a single audiobook might take up more SSD space than a band’s entire discography, but with modern drive capacities, this is mostly irrelevant even with a lossless format(best I can tell, largest audiobook I have on my computer is the fan recorded audiobook of Ward by Wildbo at 13 Gigabytes in mp3, but that’s also a 2 million+ word behemoth that would probably be broken into a dozen or more volumes if it was ever published in print, the largest single volume audio book is the first volume of the Years of Lyndon Johnson at 6.1 GB in flac).

Though, a 3-d printed grid that uses diamagnetic materials for the pins and a grid of tiny electromagnets built into an IC wafer is another option, though that might be some of what the solenoid solution is looking at.

Also, while Braille displays only have two pin positions, I think it’s worth noting the Graffiti had five height levels for each pixel.

Also, as it turns out, a standard extra wide page of Braille is 25 lines of 40 cells, or half the standard character resolution of the Linux console on modern machines(which is 25 lines of 80 characters), though Braille can fit a bit more in due to its many contractions(Standard, 6-dot Braille has 63 possible characters not counting white space. 26 of these correspond to the letters of the English alphabet and a few are common punctuation, but most of the other half are glyphs representing common 2- and 3-letter combinations, such as all six dots representing for. most letters of the alphabet also represent a common word beginning with that letter, there are several glyphs that sever purely to say “this mark followed by a certain letter represents a certain word, and nearly a hundred abbreviations exclusive to braille… sadly, while I once knew the entirety of the standard grade 2 contractions for American English Braille, I’ve forgotten most of them(another reason I probably count in the 10% but probably not the 1%), though I remember that brl is the abbreviation for Braille, a fact that is reflected in places such as the Linux package for braille support in the console being called brltty. Oh, and the standard wide page format for Braille is 11*11.5” while the narrower page format uses US letter size, at least in the US, no idea about embossed on paper braille in countries where A-series paper is the most widely used paper sizing scheme.

3 Likes
13

Braille displays and tactile graphics displays are EXPENSIVE. My 40 character braille display had to be replaced last year. I could only afford a 20. Tactile graphics displays can easily cost $10K.

Self-voicing isn’t as bad. There are several open source versions.

iPhones and iPads have pretty good support for most of your ideas. Maybe an app for those?

Jeff

14

Oh, I encountered one of those back in undergrad! I don’t remember what happened to the technology either, though.

1 Like
15

My opinion as a blind IF player who is fluent in both braille and screenreaderese: access to tactile graphics is one of the only advantages that playing on a physical console would confer over using a computer with a screenreader and standard 40-cell braille display. I’d also like to mention that being able to access a physical block of text and note where all the clickable links are located would certainly make me more willing to play choice-based titles in general; one of the reasons why I personally tend to avoid games written in Twine is that scrolling through a website and clicking on every piece of text I encounter in search of links to advance the story is just not my idea of fun.

6 Likes
16

As a result, I maintain a very well-commented CSS-stylesheet for Harlowe, explicitly to give authors a starting point in modifying these things. As a result I’ve had a lot of conversations with people over the years about it. One of these days I’m going to add SugarCube support.

2 Likes
17

@Morningstar

Which IF game have you enjoyed the most, notably for the way it addressed blind accessibility?

18

I will start out by mentioning that I primarily play parser-based IF, which means that a large part of any game’s accessibility is going to be dependent on the screenreader compatibility of its interpreter. That aside, most text-based IF is accessible right out of the box and does not need to be modified for visually-impaired players. The exception to this generalization would be games that require players to look at ASCII graphics or physical feelies to solve puzzles, in which case an author might implement a screenreader-friendly mode with alternative, non-visual solutions. Prince Quisborne from last year’s IF Comp is a perfect example, and I can personally attest that @Johnnywz00 has done a wonderful job adapting the game for screenreader users.
To answer your question more directly, some of the best accessibility handling I’ve encountered in all my years of playing IF can be seen in the Eldrum games by Act None Studios. They are text-based RPG’s with built-in CYOA mechanics, designed for mobile devices and I think somewhat outside the scope of discussion for this forum; yet somehow the first in the series, Untold, managed to make its way to IFDB, so I suppose it would be considered IFfy enough. All in-game elements, including the turn-by-turn combat system, have been optimized for screenreader compatibility based off feedback from blind and visually impaired beta testers using a range of test devices.

5 Likes
19

@Morningstar

I appreciate your insight. I figured parsers are inherently a great medium for blind accessibility, but it’s useful to hear that consideration must still be given to the puzzles and gameplay. I’m really curious about Eldrum - Untold now. Making menu/RPG systems work well for the blind is quite the accomplishment. I’ll have to watch and listen to some screen-reader gameplay (if it exists). I believe a choice-based IF system can work well for blind users, but the underlying system has to be designed from the ground up with that in mind. I see a transcript system that can be toggled where the game hides it’s visual interface (allowing shortcut keys to execute choices and commands) and reads out lines of text while keeping the world mechanics going behind the scenes. Flip the switch and the transcript is hidden and the visual system is presented again. I don’t know why, but my mind is drawn to a 4 direction, 2 button system (classic NES controller style) for navigating a blind accessible RPG choice IF game.

I know there are blind accessible game jams in general each year, but I wonder if a short blind accessible IF game jam would be an interesting challenge. I’d love to see examples of choice-based entries and how story prose can be made more blind friendly. The cool thing about a short game jam is that each game might only be 15 - 30 minutes long so it would not be such a test of patience for reviewing experimental systems and prose delivery.

Does anyone know if there is already an ongoing short IF game jam with a blind accessible theme? Just curious.

1 Like
20

@enderverse

I see there are very small micro-stepper motors available for robotics. They are much cheaper than I expected. Shaving the screw shaft and applying a simple cam would give you practically infinite increments of pin height. The pins themselves would be a bit bulkier, but it really depends on what you want to achieve with the tactile display. For me, my brain goes into a mini-map application where the pins raise and represent walls in a dungeon. You basically feel your surroundings and possible directions to navigate. I believe this would aid in creating a mental map in more granular detail for blind people. It might be a cool experience for sighted gamers too so your console might appeal beyond accessibility into novelty… larger audience.

I like the idea of animated pin patterns too.

Anyway, the nice thing about using stepper motors and shafts is that the shaft can rest in a solid mount and the device would be pretty durable. Slightly larger rounded pins equals less delicate parts too. It’s a Fisher-Price tactile display… but don’t toss it down the stairs. :wink:

I use CNC machines so I’m kind of putting two and two together. Let me know your thoughts! :slight_smile:

Edit: You could also use soft cornered square pins to really give a solid shape to grid-based images. The walls that are raised wouldn’t feel like a bumpy representation, but a smooth, straight representation. Like square pixels, basically. Food for thought.

Update: → Micro Stepper Motors