Ru Zarin's Interactive Multi-touch Applications for the Cognitively Disabled: Social stories, talking symbols, a piano, and more!

 

Trollskogen Final Demo from Ru Zarin on Vimeo.

The above video is a demonstration of Ru Zarin's masters degree project at Umea Universitet.

This project examined ways to use multi-touch technologies to support communication and learning among children with Down syndrome, autism, and related diagnoses within the age group of 5-8. I especially liked the animated social story application.

"Trollskogen is a communicative framework designed to enhance communication among people with cognitive disabilities. The forest is split up into interactive modules that provide a fun and engaging learning environment while helping improve on certain aspects of speech, reading/writing and symbol based languages. This framework has been deployed on a custom multi-touch table prototype built at the Interactive institute Umeå, enabling the children to interact with their fingers in a more natural, intuitive way rather than a traditional keyboard/mouse setup."

RELATED READING
RELATED VIDEOS OF THE PROJECT DEVELOPMENT

Ru collaborated with the illustrator Linda Arvidsson for the characters and objects found in the troll forest theme.  

I especially liked the Troll Forest Cabin, which is shown in the social stories section of the video.

Cross posted on the Interactive Multimedia Technology blog.

Establishing Joint Attention with Students with Autism Spectrum Disorders using the HP TouchSmart PC and NUI Suite Snowflake Software

I've been using my HP TouchSmart PC at work with students with disabilities. I'm experimenting with the NUI Suite SnowFlake on my TouchSmart.

The software utilizes a plug-in from NextWindow that allows for more than one touch on the screen at a time. I found that interacting with the Particles application delighted students with severe autism.

I found that many of the activities included in the Snowlake software provided opportunities for the students to establish joint attention. I also noticed an increase in the number of vocalizations and/or verbalizations among the students. Of course, this was NOT a scientific study.

NUI Suite Snowflake Multi-Touch Applications (Demonstrated on a touch-table, but the effect is similar on the HP TouchSmart.)

Note: The Particle application starts at 3:00 - the demo is set to show how it could be used as a social activity in a bar, but it is clear to see the possibilities for use with students.

Update: Here is the SnowFlake application running on a TouchSmart:


More about the HP TouchSmart PC in education:
Interacting and Communicating with HP TouchSmart Notes: Photos, Video, Audio, and More


Joint Attention:

Definition of Joint Attention from UConn:
"Joint Attention is the process of sharing one’s experience of observing an object or event, by following gaze or pointing gestures. It is critical for social development, language acquisition, cognitive development…"

Establishing joint attention is an important step in the development of social interaction skills among young people who have autism spectrum disorders.

Joint Attention Study Has Implication for Understanding Autism Science Daily, 9/29/07
Asperger-Advice: Joint Attention
Autism Games: Joint Attention and Reciprocity
Why is joint attention a pivotal skill in autism?
Tony Charman

Philos Trans R Soc Lond B Biol Sci. 2003 February 28; 358(1430): 315–324. doi: 10.1098/rstb.2002.1199.

AT&T and Universal Design

"AT&T Unveils Universal Design Approach to Help Developers Meet the Wireless Needs of Customers with Disabilities"

This is good news!

From the AT&T website:

"It is our goal that the concept of 'design for all' is not viewed as a constraint but as a catalyst for innovation across the industry," said Carlton Hill, vice president of Marketing for AT&T's mobile unit. "We believe that, by making our methodology on Universal Design available for all to see, we can show the importance and value of creating wireless products and services that are usable and beneficial to as many people as possible. The end result will be more choices for more consumers."

"Integration of Universal Design into business practice is fundamental to our mission of equitable access to wireless technologies for people of all ages and abilities," said Jim Mueller, project director, User-Centered Research, Rehabilitation Engineering Research Center for Wireless Technologies at Georgia Tech. "We're glad that AT&T shares our commitment and is making its Universal Design methodology publicly available."

Examples of accessible products/services:

* Mobile Speak is a powerful, full-fledged screen-reader with an easy-to-learn command structure, intuitive speech feedback in several languages and Braille support, which can be used with or without speech. Unlike other screen-readers for mobile phones, Mobile Speak automatically detects information that the blind user should know, just as a user with sight would easily find highlighted items or key areas of the screen at a glance.
* Mobile Magnifier is a flexible, full-screen magnification application that supports low- and high-resolution screens and can be used with or without speech feedback. Magnification software is compatible with a wide variety of mobile devices.


AT&T's Universal Design Policy
AT&T's Universal Design Principles (pdf)is a good resource about Universal Design and mobile/hand held technology, complete with pictures, example scenarios, and links to additional information.

Google Lab's Search for the Visually Impaired

Just a brief post:

Google Labs' search feature for people with visual impairments is a good idea. It has been around since 2006. The search results are displayed to ensure that the top results are websites that work with screen readers and keyboard navigation.

Overview of Accessible Solutions from Google

Google Lab's Accessible Search FAQ

Google Lab's Accessible Users Group

CITEd's Multimedia Instruction of Social Skills Resources

I recently came across a great resource for integrating multimedia technology to support social skills development. The Multimedia Instruction of Social Skills website was developed by the CITEd research center, and provides links to research, tips and strategies to use in the classroom, and a link to a variety of online resources and products.

Multimedia instruction can be used for teaching and learning activities that touch on a variety of topics. Multimedia tools can play an important part in universal design for learning (UDL) and efforts towards RTI (Response to Intervention).

Here is a list of some "quick links" from the CITEd website:

1. Universal Design for Learning in a Digital Multimedia Environment
2. Learning with Multimedia Agents
3. Learning History with Multimedia Materials
4. Learning Mathematics with Virtual Manipulatives
5. Learning with Computer Games and Simulations
6. Learning to Read with Multimedia Materials
7. Learning a Second Language with Multimedia Materials
8. Using Multimedia Tools to Help Students Learn Science
9. Multimedia Instruction for Students Who Are Deaf
10. Multimedia Instruction of Social Skills
11. Multimedia Geography Instruction
12. Multi-User Virtual Environments for Education

FYI:

CITEd is the Center for Implementing Technology in Education.

"CITEd supports leadership at state and local education agencies to integrate instructional technology for all students to achieve high educational standards. CITEd provides this support through identification of best practices, innovative online technical assistance tools, professional development, and communities of practice."

"CITEd is funded by the U.S. Department of Education, Office of Special Education Programs and is a cooperative effort of the American Institutes for Research (AIR), the Center for Applied Special Technology (CAST), and the Education Development Center (EDC)."

Technology and Alternative Cognitive Assessment?

I've been struggling to provide alternative assessments to use when I work with students who have previously tested within the severe to profound range of cognitive ability. Often these young people have more than one disability and it is very difficult to identify "hidden" cognitive strengths or areas of emerging skills.

Although some of students do not have the skills necessary to point to items on a test stimulus card, or understand the directions required by traditional cognitive test batteries, some of them they CAN point and "click" on a screen, using a mouse or an AT device.

I know of several students who test within the severe range of cognitive ability who who can navigate websites to find entertainment, music, learning activities, and special interests, often without assistance. These skills are important, but they are not included in standarized cognitive or adaptive behavior assessments.

Some people are working hard to figure out a "better way".

Organization SoftTouch, Inc. provided a workshop for educators of students who have severe disabilities addressing this concern. The workshop,
"Show Me What You Know — Alternate Assessment for Students with Autism, Cognitive and Physical Disabilities", was presented at a the Conference 2006: Technology & Persons with Disabilities.

The people at CAST- Center for Applied Special Technology, known for getting the word out about Universal Design for Learning (UDL), are also concerned about the need for accessible assessments.

I'm interested in alternative assessments and how cognitive evaluation procedures can embody the principles of universal design and access. School psychologists need more tech in our toolboxes if we are to provide meaningful assessments. This technology needs to embody the principals of "universal usability", a topic that I'll address more in-depth in a future blog.

We also need to look at current research in related fields, such as cogntive science. Here is an example:

According to the article in the Charlotte Observer (April 12, 2007), IBM volunteered to work with researchers from the Center for Behavioral Neuroscience to develop computer games designed to test the reasoning , learning, and memory skills of primates. The computer games are installed on touch-screen monitors embeeded in primate habitats in several zoos, including Zoo Atlanta.

I haven't done much reading in this area, but from what I can see, the research with primates has the potential to help develop better methods of assessing cognitive functioning of people who have strengths and abilities that can not be assessed through traditional psychological and cognitive tests.

Interesting Resources:

"The Alliance for Technology Access (ATA) is the national network of community-based Resource Centers, Developers, Vendors, and Associates dedicated to providing information and support services to children and adults with disabilities, and increasing their use of standard, assistive, and information technologies."

Julie Martin-Malivel's website Dr. Martin-Malivel conducts comparative cognition research in human and non-human primates.

"Chimpanzee Facial Expressions are Helping Researchers Understand Human Emotional Communication" This is being conducted by the Yerkes National Primate Research Center Cognitive Testing Facility. Food for thought.

"Universal Usability"
Universal Usability principles ensure that web sites and web-based applications are accessible. I think that these principles should be considered when planning cognitive and related assessments for people who have low incidence and/or multiple disabilities. (Leave a comment- I want to know what people think about this!)

Novint Falcon 3-D Haptic Game Controller - This device looks like it will have great potential for young people with visual and mobility impairments.

Link to post about Universal Design and Access for games and multimedia applications.

If you are interested in Universal Design for Learning and the use of learning and edutainment games with students, take a look at the post about game accessibility on the Interactive Multimedia Technology blog.