Data-driven tools, such as AI-Assistants, are becoming more common in everyday life. One end-user context where this is apparent is games. Many tools have been developed with the goal of helping players perform better in-game. However, these tools are rarely designed or evaluated from the perspective of learning, even though improving one’s gameplay performance is a form of learning. As a result, there is currently little general knowledge of how players use their data to learn and master gameplay and how data-driven tools can be designed to support that process. In the case of domains such as esports, improving computational support for players who do not belong to established communities may improve diversity. Games, additionally, represent complex, dynamic situations where players must make decisions without all of the information available to them. The findings of work conducted in this context may generalize to similar, non-game-related, domains, such as disaster response.
In my Ph.D. work, I aimed to expand our knowledge of how players learn from data. In doing so, I have contributed to the development of better support tools for games, which could make complex gameplay, known to have real-world benefits for players including enhanced fine motor and problem-solving skills, more accessible to more players. I conducted research on how players learn and master gameplay, what they require from data-driven tools meant to aid them in those processes, and the extent to which existing computational support meets players’ needs. I conducted this research in a user-centric, mixed methods manner, combining qualitative approaches such as interviews, observations, and think-aloud with quantitative, data-driven, analyses and statistics. In future work, I hope to expand this work beyond the domain of games, explore the intersection between computational support and learning more broadly, and find opportunities to design tools to support users in complex, dynamic situations.
Please go here to read the final dissertation document
Dissertation Overview:
Thrust 1: Self-Regulated Learning in Complex Gameplay
The first thrust of my dissertation research examines SRL in the context of complex games, specifically esports, and aims to answer “How do players engage Self-Regulated Learning skills in the context of learning and improving and play?” I explored this question through two studies. The first, an interview study, asked esports players about their goals, practice routines, and experiences trying to improve at their respective games. It followed this with a hypothetical design exercise asking players to design a fictional computational tool that could help them improve at their game. The results revealed four activities and four challenges that players face when trying to improve. The second study replicated Kitsantas and Zimmerman’s 2002 study and looked at how SRL skills were executed by League of Legends players at various skill levels. The results revealed that there were statistically significant differences in the forethought phase, but not in the performance or self-reflection phases, suggesting that players across skill levels were executing SRL skills evenly in these phases. Based on these results, we conclude that the data visualizations available to players during these phases encourage SRL skill execution, regardless of skill level.
Relevant Publications and Presentations:
Kleinman, E., Shergadwala, M., Seif El-Nasr, M. (2022). Kills, Deaths, and (Computational) Assists: Identifying Opportunities for Computational Support in Esport Learning. 2022 CHI Conference on Human Factors in Computing Systems.
Kleinman, E., Gayle, C., & Seif El-Nasr, M. (2021). “Because I’m Bad at the Game!” A Microanalytical Study of Self-Regulated Learning in League of Legends. Frontiers in Psychology, Educational Psychology. (To Appear)
Kleinman, E., Shergadwala, M., Seif El-Nasr, M. (2021) “Tell me Why I Keep Dying”: A Qualitative Study of User Requirements for AI Assistants for Esports. Esports Conference, University of California, Irvine.
Kleinman, E., Habibi, R., Powell, G. B., Reeves, B., Prather, J., & Seif El-Nasr, M. (2024, May). “Backseat Gaming” A Study of Co-Regulated Learning within a Collegiate Male Esports Community. In Proceedings of the CHI Conference on Human Factors in Computing Systems (pp. 1-14).
Thrust 2: Computational Support for Self-Regulated Learning in Complex Games
Thus, the second thrust of this work sought to answer “How do data-driven tools support self-regulated learning skills in complex games?” I explored this question, again, through two studies, this time both survey studies. First, I worked with collaborators to conduct a systematic review of existing support tools for esport games. This review revealed nine intervention types offered by the tools. I then conducted two surveys. The first gauged players’ preferences in terms of when during the gameplay experience they would want to interact with each intervention type. They could choose from before, during, or after play, corresponding to the three phases of SRL discussed in Zimmerman’s model. The results reveal usage patterns and preferences from players regarding how existing features support SRL skills across the three phases for complex games and opportunities for better support. The second survey, which is ongoing work, presents players with the four activities and challenges discovered in the first interview study, and asks them to mark how useful each intervention type is in addressing them. The results of this work will reveal the extent to which existing data-driven features support common practices for learning and improvement in the context of SRL.
Relevant Publications and Presentations:
Erica Kleinman, Reza Habibi, Yichen Yao, Christian Gayle, and Magy Seif El-Nasr. 2022. “A Time and Phase for Everything” – Towards A Self-Regulated Learning Perspective on Computational Support for Esports. Proc. ACM Hum.-Comput. Interact. 6, CHI PLAY, Article 218 (October 2022), 27 pages. https://doi.org/10.1145/3549481
Kleinman, E., Seif El-Nasr, M.. (2021). Using Data to “Git Gud”: A Push for a Player-Centric approach to the Use of Data in Esports, In EHPHCI: Esports and High-Performance HCI Workshop, 2021 CHI Conference on Human Factors in Computing Systems.
Thrust 3: Meaning Making in Process Visualizations
Reviewing gameplay, and gameplay data, post-play, during the self-reflection stage, is one of the most common executions of SRL in the context of improving at a complex game. Further, being able to understand the causal relationships between in-game actions is recognized, by many players, as critical to successful improvement over time. Based on this understanding, I present the argument that process visualizations, those that visualize data in a granular, action by action manner, can enhance self-regulated learning during the self-reflection phase. Thus, this third thrust explores the question “How do players of complex games extract meaningful insights from visualizations of process?” This question was explored through two studies. The first was a qualitative study that examined how players of the esport game DotA 2 made meaning from spatio-temporal visualizations of others’ gameplay data. The results revealed a preliminary interaction taxonomy for the domain as well as a process model. The second study built on the results of the first, but looked instead at a sequential-process visualization. The results revealed two methods for sense making using sequential-process visualizations in the context of complex games.
Relevant Publications and Presentations:
Kleinman, E., Preetham, N., Teng, Z., Bryant, A., & Seif El-Nasr, M. (2021). ” What Happened Here!?” A Taxonomy for User Interaction with Spatio-Temporal Game Data Visualization. Proceedings of the ACM on Human-Computer Interaction, 5(CHI PLAY), 1-27.
Kleinman, E., Villareale, J., Shergadwala, M., Teng, Z., Bryant, A., Zhu, J., Seif El-Nasr, M. (2022). Towards an Understanding of how Players Makeeaning from Post-Play Process Visualizations. ICEC 2022.
Thrust 4: Learning and Self-Reflection through Process Visualizations
The final question I sought to explore was “How do process visualizations of one’s own and others’ gameplay data impact self-reflection and learning?” This question was explored through two studies. The first, a mixed methods study published at CHI 2023, explored the impact of others’ process data on adaptation and quality of reflection. The results revealed that comparison with peers had a significant impact on one’s willingness to adapt their strategy. The second study, currently under review, looked at learning in League of Legends, measured by improvement in performance in a simple task, and the impact of reflecting on a process visualization compared to a traditional, aggregate visualization. Results found that reflecting on the process visualization significantly improved player performance and significantly influenced process-oriented reflection. From these results, I discussed the impact of accuracy in the reflection process, breakdowns, and potential directions for future work to better expand our understanding of the use and potential of process visualizations in games.
Relevant Publications and Presentations:
Erica Kleinman, Jennifer Villareale, Murtuza N. Shergadwala, Zhaoqing Teng, Andy Bryant, Jichen Zhu, and Magy Seif El-Nasr. 2023. “What else can I do?” Examining the Impact of Community Data on Adaptation and Quality of Reflection in an Educational Game. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems
Kleinman, E., Xu, J., Pfau, J., & Seif El-Nasr, M. (2024). ” Trust the Process”: An Exploratory Study of Process Visualizations for Self-Reflection in League of Legends. Proceedings of the ACM on Human-Computer Interaction, 8(CHI PLAY), 1-28.