Publications

Investigating children’s embodied learning in mixed-reality environments, where they collaboratively simulate scientific processes, requires analyzing complex multimodal data to interpret their learning and coordination behaviors. Learning scientists have developed Interaction Analysis (IA) methodologies for analyzing such data, but this requires researchers to watch hours of videos to extract and interpret students’ learning patterns. Our study aims to simplify researchers’ tasks, using Machine Learning and Multimodal Learning Analytics to support the IA processes. Our study combines machine learning algorithms and multimodal analyses to support and streamline researcher efforts in developing a comprehensive understanding of students’ scientific engagement through their movements, gaze, and affective responses in a simulated scenario. To facilitate an effective researcher-AI partnership, we present an initial case study to determine the feasibility of visually representing students’ states, actions, gaze, affect, and movement on a timeline. Our case study focuses on a specific science scenario where students learn about photosynthesis. The timeline allows us to investigate the alignment of critical learning moments identified by multimodal and interaction analysis, and uncover insights into students’ temporal learning progressions.

Algorithmic bias in educational environments has garnered increasing scrutiny, with numerous studies highlighting its significant impacts. This research contributes to the field by investigating algorithmic biases, i.e., selection, label, and data biases in the assessment of students’ affective states through video analysis in two educational settings: (1) an open-ended science learning environment and (2) an embodied learning context, involving 41 and 12 students, respectively. Utilizing the advanced High-speed emotion recognition library (HSEmotion) and Multi-task Cascaded Convolutional Networks (MTCNN), and contrasting these with the commercially available iMotions platform, our study delves into biases in these systems. We incorporate real student data to better represent classroom demographics. Our findings not only corroborate the existence of algorithmic bias in detecting student emotions but also highlight successful bias mitigation strategies. The research advances the development of equitable educational technologies and supports the emotional well-being of students by demonstrating that targeted interventions can effectively diminish biases.