- Fellowship
- Open Signal New Media Fellow
- Medium
- Research & prototypes
- Collaborator
- Genia Penksik
- Focus
- Real-time captioning, sign language in 360°
Virtual Worlds Made Accessible Beyond Sound
Solving the design challenges of real-time captioning and sign language in 360° environments.
As an Open Signal New Media Fellow I built social VR experiences around sign language and AI generated captioning, so that a conversation in a virtual space can be followed without sound. I presented the work as a virtual talk.
This project works out what live captioning has to become inside a 360° environment, and across XR more generally: augmented, virtual and mixed reality. On a flat screen a caption sits at the bottom of the frame, which is how video conferencing tools like Zoom and Google Meet deliver theirs. In a headset there is no bottom of the frame. A caption has to know where the viewer is looking and where the sound is coming from, and it has to stay readable while both keep moving. Some of those problems we expected. Others only turned up once we were in the headset.
I started this when meetings moved online. I designed it with Genia Penksik, a software developer and engineer in the UK. I am a Deaf visual sound artist in the US with a background in UX design. We set out to share the design and technical solutions publicly, so that other developers can build the same accessibility into their own applications.
