publications | Ruei-Che Chang

2024

WorldScribe: Towards Context-Aware Live Visual Descriptions

Ruei-Che Chang, Yuxuan Liu, and Anhong Guo

In Proceedings of the 37th Annual ACM Symposium on User Interface Software and Technology (UIST ’24), Pittsburgh, PA, USA, 2024

🏆 Best Paper Award (Top 1%)

Abs HTML PDF Video 30s Preview

Automated live visual descriptions can aid blind people in understanding their surroundings with autonomy and independence. However, providing descriptions that are rich, contextual, and just-in-time has been a long-standing challenge in accessibility. In this work, we develop WorldScribe, a system that generates automated live real-world visual descriptions that are customizable and adaptive to users’ contexts: (i) WorldScribe’s descriptions are tailored to users’ intents and prioritized based on semantic relevance. (ii) WorldScribe is adaptive to visual contexts, e.g., providing consecutively succinct descriptions for dynamic scenes, while presenting longer and detailed ones for stable settings. (iii) WorldScribe is adaptive to sound contexts, e.g., increasing volume in noisy environments, or pausing when conversations start. Powered by a suite of vision, language, and sound recognition models, WorldScribe introduces a description generation pipeline that balances the tradeoffs between their richness and latency to support real-time use. The design of WorldScribe is informed by prior work on providing visual descriptions and a formative study with blind participants. Our user study and subsequent pipeline evaluation show that WorldScribe can provide real-time and fairly accurate visual descriptions to facilitate environment understanding that is adaptive and customized to users’ contexts. Finally, we discuss the implications and further steps toward making live visual descriptions more context-aware and humanized.
EditScribe: Non-Visual Image Editing with Natural Language Verification Loops

Ruei-Che Chang, Yuxuan Liu, Lotus Zhang, and Anhong Guo

In Proceedings of the 26th International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS ’24), St. John’s, Newfoundland and Labrador, Canada, 2024

Abs PDF Video

Image editing is an iterative process that requires precise visual evaluation and manipulation for the output to match the editing intent. However, current image editing tools do not provide accessible interaction nor sufficient feedback for blind and low vision individuals to achieve this level of control. To address this, we developed EditScribe, a prototype system that makes image editing accessible using natural language verification loops powered by large multimodal models. Using EditScribe, the user first comprehends the image content through initial general and object descriptions, then specifies edit actions using open-ended natural language prompts. EditScribe performs the image edit, and provides four types of verification feedback for the user to verify the performed edit, including a summary of visual changes, AI judgement, and updated general and object descriptions. The user can ask follow-up questions to clarify and probe into the edits or verification feedback, before performing another edit. In a study with ten blind or low-vision users, we found that EditScribe supported participants to perform and verify image edit actions non-visually. We observed different prompting strategies from participants, and their perceptions on the various types of verification feedback. Finally, we discuss the implications of leveraging natural language verification loops to make visual authoring non-visually accessible.
Audio Description Customization

Rosiana Natalie, Ruei-Che Chang, Smitha Sheshadri, Anhong Guo, and Kotaro Hara

In Proceedings of the 26th International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS ’24), St. John’s, NL, Canada, 2024

Abs PDF Video

Blind and low-vision (BLV) people use audio descriptions (ADs) to access videos. However, current ADs are unalterable by end users, thus are incapable of supporting BLV individuals’ potentially diverse needs and preferences. This research investigates if customizing AD could improve how BLV individuals consume videos. We conducted an interview study (Study 1) with fifteen BLV participants, which revealed desires for customizing properties like length, emphasis, speed, voice, format, tone, and language. At the same time, concerns like interruptions and increased interaction load due to customization emerged. To examine AD customization’s effectiveness and tradeoffs, we designed CustomAD, a prototype that enables BLV users to customize AD content and presentation. An evaluation study (Study 2) with twelve BLV participants showed using CustomAD significantly enhanced BLV people’s video understanding, immersion, and information navigation efficiency. Our work illustrates the importance of AD customization and offers a design that enhances video accessibility for BLV individuals.
SoundShift: Exploring Sound Manipulations for Accessible Mixed-Reality Awareness

Ruei-Che Chang, Chia-Sheng Hung, Bing-Yu Chen, Dhruv Jain, and Anhong Guo

In Proceedings of the 2024 ACM Designing Interactive Systems Conference (DIS ’24), IT University of Copenhagen, Denmark, 2024

Abs PDF Video

Mixed-reality (MR) soundscapes blend real-world sound with virtual audio from hearing devices, presenting intricate auditory information that is hard to discern and differentiate. This is particularly challenging for blind or visually impaired individuals, who rely on sounds and descriptions in their everyday lives. To understand how complex audio information is consumed, we analyzed online forum posts within the blind community, identifying prevailing challenges, needs, and desired solutions. We synthesized the results and propose SoundShift for increasing MR sound awareness, which includes six sound manipulations: Transparency Shift, Envelope Shift, Position Shift, Style Shift, Time Shift, and Sound Append. To evaluate the effectiveness of SoundShift, we conducted a user study with 18 blind participants across three simulated MR scenarios, where participants identified specific sounds within intricate soundscapes. We found that SoundShift increased MR sound awareness and minimized cognitive load. Finally, we developed three real-world example applications to demonstrate the practicality of SoundShift.
ImageExplorer Deployment: Understanding Text-Based and Touch-Based Image Exploration in the Wild

Andi Xu, Minyu Cai, Dier Hou, Ruei-Che Chang, and Anhong Guo

In Proceedings of the 21st International Web for All Conference (W4A ’24), , 2024

Abs HTML PDF Video

Blind and visually-impaired (BVI) users often rely on alt-texts to understand images. AI-generated alt-texts can be scalable and efficient but may lack details and are prone to errors. Multi-layered touch interfaces, on the other hand, can provide rich details and spatial information, but may take longer to explore and cause higher mental load. To understand how BVI users leverage these two methods, we deployed ImageExplorer, an iOS app on the Apple App Store that provides multi-layered image information via both text-based and touch-based interfaces with customizable levels of granularity. Across 12 months, 371 users uploaded 651 images and explored 694 times. Their activities were logged to help us understand how BVI users consume image captions in the wild. This work informs a holistic understanding of BVI users’ image exploration behavior and influential factors. We provide design implications for future models of image captioning and visual access tools.
AdvCAPTCHA: Creating Usable and Secure Audio CAPTCHA with Adversarial Machine Learning

Hao-Ping Hank Lee, Wei-Lun Kao, Hung-Jui Wang, Ruei-Che Chang, Yi-Hao Peng, Fu-Yin Cherng, and Shang-Tse Chen

(USEC’24), San Diego, CA, USA, 2024

Abs PDF

Audio CAPTCHA (Completely Automated Public Turing test to tell Computers and Humans Apart) is an accessible alternative to the traditional CAPTCHA for people with visual impairments. However, the literature has found that audio CAPTCHA suffers from both lower usability and security than its visual counterpart. In this paper, we propose AdvCAPTCHA, a novel audio CAPTCHA generated by using adversarial machine learning techniques. By conducting studies with people with and without visual impairments, we show that AdvCAPTCHA can outperform the \textitstatus quo audio CAPTCHA in security but not usability. We demonstrate AdvCAPTCHA’s feasibility of providing detection of malicious attacks. We also present an evaluation metric, \textitthresholding, to quantify the trade-off between usability and security for audio CAPTCHA design. Finally, we discuss approaches to the real-world adoption of AdvCAPTCHA.

2023

Understanding (Non-)Visual Needs for the Design of Laser-Cut Models

Ruei-Che Chang, Seraphina Yong, Fang-Ying Liao, Chih-An Tsao, and Bing-Yu Chen

In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (CHI ’23), Hamburg, Germany, 2023

Abs PDF Video

Laser-cutting is a promising fabrication method that empowers makers, including blind or visually-impaired (BVI) creators, to create technologies that fit their needs. Existing work on laser-cut accessibility has facilitated easier assembly as a workaround for existing models. However, laser-cut models are still not designed to accommodate the needs of BVI users. Integrating BVI needs can enrich the greater maker community by enabling cross-group discourse on laser-cut making. To investigate how laser-cut model design can be more accessible overall, we study laser-cut assembly as a process deeply intertwined with the fundamental design of laser-cut models. We present a study with seven sighted and seven BVI participants to compare their usage of laser-cut model affordances during assembly. Data for the BVI participants in this study originate from a previous work [13]. We identify assembly cues common or unique to sighted and BVI users, and discuss implications to improve general accessibility in laser-cut design.

2022

OmniScribe: Authoring Immersive Audio Descriptions for 360° Videos

Ruei-Che Chang, Chao-Hsien Ting, Chia-Sheng Hung, Wan-Chen Lee, Liang-Jin Chen, Yu-Tzu Chao, Bing-Yu Chen, and Anhong Guo

In Proceedings of the 35th Annual ACM Symposium on User Interface Software and Technology (UIST ’22), Bend, OR, USA, 2022

Abs HTML PDF Video

Blind people typically access videos via audio descriptions (AD) crafted by sighted describers who comprehend, select, and describe crucial visual content in the videos. 360° video is an emerging storytelling medium that enables immersive experiences that people may not possibly reach in everyday life. However, the omnidirectional nature of 360° videos makes it challenging for describers to perceive the holistic visual content and interpret spatial information that is essential to create immersive ADs for blind people. Through a formative study with a professional describer, we identified key challenges in describing 360° videos and iteratively designed OmniScribe, a system that supports the authoring of immersive ADs for 360° videos. OmniScribe uses AI-generated content-awareness overlays for describers to better grasp 360° video content. Furthermore, OmniScribe enables describers to author spatial AD and immersive labels for blind users to consume the videos immersively with our mobile prototype. In a study with 11 professional and novice describers, we demonstrated the value of OmniScribe in the authoring workflow; and a study with 8 blind participants revealed the promise of immersive AD over standard AD for 360° videos. Finally, we discuss the implications of promoting 360° video accessibility.
Puppeteer: Exploring Intuitive Hand Gestures and Upper-Body Postures for Manipulating Human Avatar Actions

Ching-Wen Hung, Ruei-Che Chang, Hong-Sheng Chen, Chung Han Liang, Liwei Chan, and Bing-Yu Chen

In Proceedings of the 28th ACM Symposium on Virtual Reality Software and Technology (VRST ’22), Tsukuba, Japan, 2022

Abs PDF Video

Body-controlled avatars provide a more intuitive method to real-time control virtual avatars but require larger environment space and more user effort. In contrast, hand-controlled avatars give more dexterous and fewer fatigue manipulations within a close-range space for avatar control but provide fewer sensory cues than the body-based method. This paper investigates the differences between the two manipulations and explores the possibility of a combination. We first performed a formative study to understand when and how users prefer manipulating hands and bodies to represent avatars’ actions in current popular video games. Based on the top video games survey, we decided to represent human avatars’ motions. Besides, we found that players used their bodies to represent avatar actions but changed to using hands when they were too unrealistic and exaggerated to mimic by bodies (e.g., flying in the sky, rolling over quickly). Hand gestures also provide an alternative to lower-body motions when players want to sit during gaming and do not want extensive effort to move their avatars. Hence, we focused on the design of hand gestures and upper-body postures. We present Puppeteer, an input prototype system that allows players directly control their avatars through intuitive hand gestures and upper-body postures. We selected 17 avatar actions discovered in the formative study and conducted a gesture elicitation study to invite 12 participants to design best representing hand gestures and upper-body postures for each action. Then we implemented a prototype system using the MediaPipe framework to detect keypoints and a self-trained model to recognize 17 hand gestures and 17 upper-body postures. Finally, three applications demonstrate the interactions enabled by Puppeteer.

2021

Daedalus in the Dark: Designing for Non-Visual Accessible Construction of Laser-Cut Architecture

Ruei-Che Chang, Chih-An Tsao, Fang-Ying Liao, Seraphina Yong, Tom Yeh, and Bing-Yu Chen

In The 34th Annual ACM Symposium on User Interface Software and Technology (UIST ’21), Virtual Event, USA, 2021

Abs PDF Video

Design tools and research regarding laser-cut architectures have been widely explored in the past decade. However, such discussion has mostly revolved around technical and structural design questions instead of another essential element of laser-cut models — assembly — a process that relies heavily on components’ visual affordance, therefore less accessible to blind or low vision (BLV) people. To narrow the gap in this area, we co-designed with 7 BLV people to examine their assembly experience with different laser-cut architectures. From their feedback, we proposed several design heuristics and guidelines for Daedalus, a generative design tool that can produce tactile aids for laser-cut assembly given a few high-level manual inputs. We validate the proposed aids in a user study with 8 new BLV participants. Our results revealed that BLV users can manage laser-cut assembly more efficiently with Daedalus. Going forth from this design iteration, we discuss implications for future research on accessible laser-cut assembly.
AccessibleCircuits: Adaptive Add-On Circuit Components for People with Blindness or Low Vision

Ruei-Che Chang, Wen-Ping Wang, Chi-Huan Chiang, Te-Yen Wu, Zheer Xu, Justin Luo, Bing-Yu Chen, and Xing-Dong Yang

In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (CHI ’21), Yokohama, Japan, 2021

Abs PDF Video

In this paper, we propose the designs for low cost and 3D-printable add-on components to adapt existing breadboards, circuit components and electronics tools for blind or low vision (BLV) users. Through an initial user study, we identified several barriers to entry for beginners with BLV in electronics and circuit prototyping. These barriers guided the design and development of our add-on components. We focused on developing adaptations that provide additional information about the specific component pins and breadboard holes, modify tools to make them easier to use for users with BLV, and expand non-visual feedback (e.g., audio, tactile) for tasks that require vision. Through a second user study, we demonstrated that our adaptations can effectively overcome the accessibility barriers in breadboard circuit prototyping for users with BLV.

2020

Glissade: Generating Balance Shifting Feedback to Facilitate Auxiliary Digital Pen Input

Kai-Chieh Huang, Chen-Kuo Sun, Da-Yuan Huang, Yu-Chun Chen, Ruei-Che Chang, Shuo-wen Hsu, Chih-Yun Yang, and Bing-Yu Chen

In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (CHI ’20), Honolulu, HI, USA, 2020

🏅 Best Paper Honorable Mention Award

Abs PDF Video

This paper introduces Glissade, a digital pen that generates balance shifting feedback by changing the weight distribution of the pen. A pulley system shifts a brass mass inside the pen to change the pen’s center of mass and moment of inertia. When the mass is stationary, the pen delivers a constant yet natural sensation of weight, which can be used to convey a status. The pen can also generate a variety of haptic clues by actuating the mass according to the tilt or rotation of the pen, two commonly-used auxiliary pen input channels. Glissade demonstrates new possibilities that balance shifting feedback can bring to digital pen interactions. We validated the usability of this feedback by determining the recognizability of six balance patterns – a mix of static and dynamic patterns chosen based on our design considerations – in two controlled experiments. The results show that, on average, the participants could distinguish between the patterns with a 94.25% accuracy. At the end, we demonstrate a set of novel interactions enabled by Glissade and discuss the directions for future research.
Exploring the Design Space of User-System Communication for Smart-home Routine Assistants

Yi-Shyuan Chiang, Ruei-Che Chang, Yi-Lin Chuang, Shih-Ya Chou, Hao-Ping Lee, I-Ju Lin, Jian-Hua Jiang Chen, and Yung-Ju Chang

In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (CHI ’20), Honolulu, HI, USA, 2020

Abs PDF Video

AI-enabled smart-home agents that automate household routines are increasingly viable, but the design space of how and what such systems should communicate with their users remains underexplored. Through a user-enactment study, we identified various interpretations of and feelings toward such a system’s confidence in its automated acts. That confidence and their own mental models influenced what and how the participants wanted the system to communicate, as well as how they would assess, diagnose, and subsequently improve it. Automated acts resulted from false predictions were not generally considered improper, provided that they were perceived as reasonable or potentially useful. The participants’ improvement strategies were of four general types, all of which will be discussed. Factors affecting their preferred levels of involvement in automated acts and their interest in system confidence were also identified. We conclude by making practical design recommendations for the user-system communication design spaces of smart-home routine assistants.
TanGo: Exploring Expressive Tangible Interactions on Head-Mounted Displays

Ruei-Che Chang, Chi-Huan Chiang, Shuo-wen Hsu, Chih-Yun Yang, Da-Yuan Huang, and Bing-Yu Chen

In Proceedings of the 2020 ACM Symposium on Spatial User Interaction (SUI ’20), Virtual Event, Canada, 2020

Abs PDF Video

We present TanGo, an always-available input modality on VR headset, which can be complementary to current VR accessories. TanGO is an active mechanical structure symmetrically equipped on Head-Mounted Display, enabling 3-dimensional bimanual sliding input with each degree of freedom furnished a brake system driven by micro servo generating totally 6 passive resistive force profiles. TanGo is an all-in-one structure that possess rich input and output while keeping compactness with the trade-offs between size, weight and usability. Users can actively gesture like pushing, shearing, or squeezing with specific output provided while allowing hands to rest in stationary experiences. TanGo also renders users flexibility to switch seamlessly between virtual and real usage in Augmented Reality without additional efforts and instruments. We demonstrate three applications to show the interaction space of TanGo and then discuss its limitation and show future possibilities based on preliminary user feedback.

2019

Masque: Exploring Lateral Skin Stretch Feedback on the Face with Head-Mounted Displays

Chi Wang, Da-Yuan Huang, Shuo-wen Hsu, Chu-En Hou, Yeu-Luen Chiu, Ruei-Che Chang, Jo-Yu Lo, and Bing-Yu Chen

In Proceedings of the 32nd Annual ACM Symposium on User Interface Software and Technology (UIST ’19), New Orleans, LA, USA, 2019

Abs PDF Video

We propose integrating an array of skin stretch modules with an head-mounted display (HMD) to provide two-dimensional skin stretch feedback on the user’s face. Skin stretch has been found effective to induce the perception of force (e.g. weight or inertia) and to enable directional haptic cues. However, its potential as an HMD output for virtual reality (VR) remains to be exploited. Our explorative study firstly investigated the design of shear tactors. Based on our results, Masque has been implemented as an HMD prototype actuating six shear tactors positioned on the HMD’s face interface. A comfort study was conducted to ensure that skin stretches generated by Masque are acceptable to all participants. The following two perception-based studies examined the minimum changes in skin stretch distance and stretch angles that are detectable by participants. The results help us to design haptic profiles as well as our prototype applications. Finally, the user evaluation indicates that participants welcomed Masque and regarded skin stretch feedback as a worthwhile addition to HMD output.