The Design of a Smartphone Camera Lens-based Application to Simplify the Lives of Those With Visual or Cognitive Impairments 

Jocelyn L. Gilbertson

Cognitive Psychology, Iowa State University | HCI/PSYCH 521: Cognitive Psychology of Human-Computer Interaction, Dr. Stephen Gilbert

December 5th, 2023


Introduction

“Almost 20 million Americans, or 8 percent of the U.S. population, have visual impairments. Visual impairments, including blindness, are one of the leading causes of loss of independence among people aged 65 and older.”

Many people are affected by visual impairments which include variation of blindness or other visual impairments that make viewing and reading text in various real-life scenarios difficult. There have been various research studies done on how to assist those with visual or cognitive impairments with things such as using a smartphone or by using other types of interfaces to assist with daily life and activities. However, there has not been a study done on how a camera lens-based application and associated interface could assist those with cognitive or visual impairments, how they assist with the interaction of the world around them, mainly with written or typographical elements. 

This study aims to examine how a smart phone application can assist people with cognitive disabilities or visual impairments to see and read text in various real-world scenarios, and in real time. I will focus on designing a camera lens-based application specifically designed for people with cognitive disabilities or visual impairments. I will review research related to the improvement of technological aids and applications as well as interfaces, for those with visual or cognitive disabilities. This will be done in order to suggest new research of a tool that combines numerous aspects of each study to improve the daily life of those living with cognitive or visual impairments and make interacting with typographical elements in daily life easier and more accessible. The proposed technology will utilize a person's camera lens on their smartphone, along with an application-based interface that works toward improving the independence level and to assist with the completion of various text-based acts of assistance. The technology needed to create this type of tool exists, and would be able to be built and designed with tools and development processes.

Literature

In a study conducted by M. Braun, M. Wölfel, G. Renner and C. Menschik, researchers examined interfaces and how accessible they were to those with various disabilities. This research primarily focused on interfaces and accessibility for people with disabilities (they refer to them as PWD’s). The researchers primary question was “to what extent the class of ‘natural’ user interfaces-including touch, voice and touchless are accessible to people with intellectual and possibly motor disabilities.” They focused on three main types of interfaces commonly used by the PWD community: touch interfaces, voice-commanded interfaces, and touchless interfaces. For each type, the researchers detailed the extent at which each of these interfaces are currently used by PWD, how they are used, and the experience users commonly have while interacting with each type of control.These interfaces have been adjusted for accessibility. An important note the researchers touched on was that accessibility settings that can be adjusted by the user are difficult for PWD to adjust without the help of a non-PWD user, which is counterintuitive when trying to design accessible user interfaces. “Participants were recruited from three different institutions in Germany. The majority of the participants (76.9%) lived in residential groups in facilities. Individual, less restricted participants (23.1%) lived in outpatient residential groups (also part of facilities). They observed a total of 91 participants using one of the three interfaces. The age of the participants ranged from 31 to 79 years.” To gather data, they used a Questionnaire for Observing Communicative Skills Revision (OCS-R), as this is a style of questionnaire specifically for caregivers PWD’s. It is “a structured diagnostic observation instrument for assessing communicative abilities and forms of expression of children, adolescents, and adults with limitations in their communicative abilities or their communicative development.” For the questionnaire, they asked questions regarding interaction, perception, and motor skills of the participants. Additionally, they used observational studies with a variety of tasks using the three types of control. Through the observational studies, they observed participants attempt to complete tasks with touch, voice, or touchless interfaces. The researchers highlighted that this data cannot be comprehensive because of its nature, and can be applied to individual cases. The results show that the three interface types are accessible to some people with disabilities, but the majority faces major problems while using them. In order to make current interface types usable for people with any level of knowledge or ability, accessibility or universal design decisions can be made in the development process to further the usability of products. 

In a study done by Janebäck, J., & Jonsson, E., researchers examined the development and prototype testing of a smartphone application specifically for those with disabilities. This research primarily focused on a user study and prototype of a smartphone application interface for an application that is to promote independence in everyday life for the target group, which was a group with cognitive impairments. The research question of this project was: “What should be considered when designing a UI for people with cognitive disability? (Jäneback, vi).” The researchers held user-testing, based on a prototype developed in Figma. Once developed and tested, the researchers found new information that pointed to what worked better on the smartphone application for the target group. For the interviews, the researchers held semi-structured interviews and group interviews. The participants were 15-20 people who were recruited; the participants were measured at different levels of function, different ages (approximately 18-60) and were noted to have different levels of cognitive ability. Some individual questions were as follows: “What are your thoughts about the fact that there is an app coming up? Do you use apps on your phone frequently? Is there any app you find difficult to use? What makes it difficult? Is there anything else you want to mention about the coming application? (Jäneback, 27).” Some group interview questions were as follows: “What is the first thing you think about when we say "Grunden’s new application" ? Name one function you would like the application to have. What is of highest importance regarding the application? When would you like to use the application? (in what situation)? Name one good and one bad application (Jäneback, 27).” As a result of interview findings, prior knowledge of accessibility, and WCAG standards, the researchers developed custom prototypes that met the specific functions of the needs identified by their research. The main functionality of the application was for users to easily find activities that cater to their interests and to make an independent and active lifestyle seem more appealing and achievable to the user group. The researchers also kept in mind common usability standards, including to “Use a contrast ratio for text and pictures of at least 4.5:1 to assure accessibility; Have a minimum touch target of at least 9x9 mm to facilitate interaction for people with impaired motor skills; Use standard icons and symbols to help with memory and recognition since people tend to link a specific symbol to an action of function (Jäneback, 63).” 

Future Directions

Currently, research in products similar to what I'm proposing does exist, but does not function in the ways I am suggesting. Some applications function similarly to what I am proposing, but in separate interfaces. Therefore, I am proposing a study similar to Janebäck’s (2022), in which the product is designed, developed, and subsequently tested in an observational environment with cross-sectional studies and semi-structured user interviews. This would further research in the area of designing accessible products for those with cognitive or visual impairments that make interacting with text or typographical elements difficult. Some of the topics that would be studied would be the development of an accessible interface, testing of the interface, testing of the scanning portion of the application, and ensuring that the application is overall usable and functional for those with various disabilities or visual impairments.

With this application, those with cognitive or visual impairments could hold their phone to scan a text or typographical element. Once scanned, the application would provide various solutions for communicating the content to the user. The application could ‘translate’ the text to be a different font, make it a more accessible visible size, read the text aloud to the user, or another choice of communication chosen by the user that meets their specific needs. This application could then work with existing technology to help users in other ways, as desired. For example, the application could use the text it identified to provide things like verbalization of words or sentences, or provide definitions of words and provide similes to better understand the text or typographical element. Additionally, this app could sync with applications, such as the user's native or chosen map application (ex. Apple maps), allowing the application to recognize locations. This would allow the users to bookmark frequented places, or create a library of frequented locations or frequently scanned text, making it easier and faster for the user to utilize the application.

When I began this research project, I had the goal to come up with completely new  technology or a technological advancement that could significantly assist those with visual or cognitive impairments while interacting with the world around them regarding typographical and text-based interactions. Although applications exist with the function to scan and read text aloud, there are many limitations. Handwriting, script-style fonts, or otherwise difficult to read typographical elements are unlikely to be scanned and utilized due to the limitations of current software and technological advancements. Upon examining other applications built with similar functions, I identified gaps that these existing applications had. By identifying these gaps, I was able to identify places where a new application could be developed and build upon the foundations that already exist from research and existing applications. 

I believe that by developing this application, there is a high likelihood that daily life for those with cognitive, visual, or other impairments could be more accessible and allow those who utilize it to have more independent lives, or less reliant on help from others around them. By developing this application on the findings of what would be found in the semi-structured user interviews, cross-sectional studies of the use of the application, and prototype testing, this app would be developed with the key users in mind. Independence and self-reliance is highly important in the hierarchy of self-esteem needs, and I believe this application could assist those with varying needs achieve a level of independence when interacting with the world and typographical elements around them.

References

Janebäck, J., & Jonsson, E. (2022). Designing an activity application for people with cognitive disabilities What should be considered when designing a UI for people with cognitive disabilities? https://odr.chalmers.se/server/api/core/bitstreams/7af19127-f340-406a-b03f-cad8736ef7c6/content

M. Braun, M. Wölfel, G. Renner and C. Menschik, "Accessibility of Different Natural User Interfaces for People with Intellectual Disabilities," 2020 International Conference on Cyberworlds (CW), Caen, France, 2020, pp. 211-218, doi: 10.1109/CW49994.2020.00041.