Nitasha Mathayas, PH.D.

GRASP

The Project

GRASP (GestuRe Augmented Simulations for supporting exPlanations) is an NSF-funded collaboration between the University of Illinois at Urbana-Champaign and Concord Consortium. The goal of this project is to understand the role that gestures play in reasoning about critical concepts in science.

My role

  • gesture research
  • simulation design
  • protocol design
  • data collection
  • data analysis
  • publications
  • Proposal Development

This project inspired my dissertation research. In this project, I led key design improvements for 1 of 3 AR gesture-sensing science simulations in collaboration with Concord Consortium LLC, while supporting the team on other simulations. I led mixed-methods analysis of over 200+ student interactions to develop foundational AR simulation design principles that contributed to my dissertation research.

Simulation Design

The goal of the project was to integrate gesture-based interactions into middle school science simulations, making abstract concepts like molecular mechanisms tangible and accessible. In one compelling example shown in the video, students explore thermal conduction by forming a fist to represent a molecule and wiggling it to simulate high-temperature molecular movement. This isn’t just a clever teaching technique—it’s rooted in embodied cognition theory, which recognizes that our minds and bodies are intrinsically connected, with human understanding fundamentally shaped by our physical experiences. By transforming invisible phenomena into physical actions, students develop deeper, more intuitive scientific understanding.

Building on this embodied approach, I collaborated with the research and software team on a design-based research project (similar to prototyping and testing cycles in industry settings) that transformed three key simulations: gas pressure, thermal conduction, and seasons. These were meticulously redesigned with integrated gesture-based interactions and implemented across a diverse sample of over 200 middle school students throughout the United States. The iterative design process allowed us to refine both the physical interactions and instructional scaffolding, creating more intuitive and effective learning experiences.

  • Screenshot of gas-pressure simulation
  • Screenshot of seasons simulation
  • Screenshot of heat transfer simulation
  • Students using the seasons simulation
  • Student bumpping molecules while using heat transfer simulation

Research Questions

In the following study, I am demonstrating part of the analysis I conducted for my dissertation research. Here I investigated what and how students learnt from using one gesture-augmented simulation that depicted heat transfer at the molecular level.

  1. From a quantitative perspective, to what extent did students explanations take on more scientific characteristics?
  2. From a qualitative perspective, in what ways did students’ explanations take on more scientific characteristics?

Data Collection

During the 2016-2017 school year, I interviewed 21 middle school students as they used the gesture augmented simulation depicting heat transfer (thermal conduction specifically)

This year the reserach team and myself tested the third iteration of the simulation and were collecting data for this third cycle.

We video recorded students as they used the simulation and explained how heat transferred from the immersed end to the tip of the spoon.

Each interview took about 30 minutes and was video recorded and then transcribed for analysis. The laptop screen showing the simulation was simultaneiously recorded and matched to video for further analysis.

Mixed Methods Data Analysis

Creating explanation scores

  • Interviews transcribed verbatim
  • Data Analysis Software: Excel
  • Iterative coding process (Miles & Huberman, 2014)
    • Scientific explanation coding scheme developed internally with feedback from research team and scientists
    • Score range of 0-6
    • All interviews were coded independently by myself and one external researcher. We achieved 91% interrator agreement independently then discussed and resolved differences leading to complete agreement
  • Due to low number of participants and not meeting assumptions of parametric tests, I conducted the Wilcoxon’s signed ranks test to compare pre and post simulation scores
  • The scores were calculated across important sections within student use of the interview and can be traced to see how students’ scored changed over time of interview

Multiple Case Analysis

  • I then selected three student cases for in-depth analysis on their interaction with the simulation
  • Appling methods of grounded theory, I explored what dimensions of sensemaking (cognitive, social, material etc.) influenced student learning and in what ways
    • Through single case study and then cross-case study, I refined my understanding of these dimensions of sensemaking and identified which simulation design features and interactions facilitated student’s learning

Findings

Quantitative insights

  • Wilcoxon’s signed ranks test show significant improvement in the characteristics of students’ explanation (T = 1.5, n = 21, p < 0.001)
  • Analysis across when specific features of the simulation were introduced to students shows that their scores improved as each feature was introduced.
    • More specifically when selecting for codes that targeted causal-mechanisms, the magnification model feature led to significant growth in student scores, suggesting x% improvement from introducing this new feature into the simulation

Patterns

Visually looking across each student’s scores , patterns emerged across their score charts which allowed me to categorize them under three personas: A: the students with relatively steady scire growth, B: the students with high scores before using the gesture augmented s1imulation, and C: the student with no overall score gain.

The case studies targeted one representive student from each group. Each case can be also be called a learner persona. I then went in depth and conducted a microgenetic analysis of their every movement in the video to understand how these patterns of learning emerged.

Key features of each learner persona

Summary of case study analysis

My cross case studies revealed a fascinating pattern in how students grasp complex scientific concepts. For students to fully understand and explain scientific mechanisms, they needed to progress through several key stages:

  • Foundation of Understanding: Students needed prior knowledge of basic representations
  • Physical-Conceptual Connection: Linking gestures to concepts (like using a fist to represent a molecule)
  • Immersive Interaction: Physically engaging with the simulation environment
  • Explanatory Mindset: Approaching the simulation as something to understand, not just manipulate
  • Recognition of Knowledge Gaps: Awareness of what they didn’t yet understand

The Result? When these conditions aligned, students could perceive and articulate the underlying causal mechanisms at work.

Conclusions

Physical Actions Transform Thinking: Through gesturing, students’ explanations evolved from simply attributing effects to “heat” to understanding the deeper concept of “molecular movement”

Movement Creates Understanding: When students used control-gestures, they naturally shifted into an explanatory mindset

<< TEA

-Back-

ELASTICS>>