This paper describes the design and evaluation of Zoo U, a novel computer game to assess children’s social skills development. Zoo U is an innovative product that combines theory-driven content and customized game mechanics. The game-like play creates the opportunity for stealth assessment, in which dynamic evidence of social skills is collected in real time and players’ choices during gameplay provide the needed data. To ensure the development of an engaging and valid game, we utilized an iterative data-driven validation process in which the game was created, tested, revised based on student performance and feedback, and retested until game play was statistically matched to independent ratings of social skills. We first investigated whether the data collected through extensive logging of student actions provided information that could be used to improve the assessment. We found that detailed game logs of socially relevant player behavior combined with external measures of player social skills provided an efficient vector to incrementally improve the accuracy of the embedded assessments. Next, we investigated whether the game performance correlated with teachers’ assessments of students’ social skills competencies. An evaluation of the final game showed (a) significant correlations between in-game social skills assessments and independently obtained standard psychological assessments of the same students and (b) high levels of engagement and likeability for students. These findings support the use of the interactive and engaging computer game format for the stealth assessment of children’s social skills. The created innovative design methodologies should prove useful in the design and improvement of computer games in education. 1. Introduction Social skills comprise a group of behaviors and knowledge that help children create and maintain friendships and navigate a multitude of situations involving other people. A child’s social skills competence can strongly influence his or her sense of well being and adjustment [1, 2]. The importance of social skills and peer relationships increases through the elementary school years and into adolescence, with peers becoming key providers of support, advice, companionship, and affirmation [3]. Children with strong social skills and the relationships built on these skills tend to have more positive emotional, behavioral, and academic functioning. Positive peer relationships also function as a protective factor against negative outcomes in the face of stressful life events such as poverty [2, 4, 5]. In contrast, a
References
[1]
J. Kupersmidt and M. DeRosier, “How peer problems lead to negative outcomes: an integrative mediational model,” in Children’s Peer Relations: From Development to Intervention, pp. 119–138, American Psychological Association, 2004.
[2]
J. Parker, K. Rubin, S. Erath, J. Wojslawowicz, and A. Buskirk, “Peer relationships, child development, and adjustment: a developmental psychopathology perspective,” Developmental Psychopathology, vol. 1, pp. 419–493, 2006.
[3]
W. Furman and D. Buhrmester, “Age and sex differences in perceptions of networks of personal relationships,” Child Development, vol. 63, no. 1, pp. 103–115, 1992.
[4]
A. E. Kazdin, “Effectiveness of psychotherapy with children and adolescents,” Journal of Consulting and Clinical Psychology, vol. 59, no. 6, pp. 785–798, 1991.
[5]
S. Luthar, Resilience and Vulnerability: Adaptation in the Context of Childhood Adversities, Cambridge University Press, New York, NY, USA, 2003.
[6]
M. Boivin and S. Hymel, “Peer experiences and social self-perceptions: a sequential model,” Developmental Psychology, vol. 33, no. 1, pp. 135–145, 1997.
[7]
S. A. Erath, K. S. Flanagan, and K. L. Bierman, “Social anxiety and peer relations in early adolescence: behavioral and cognitive factors,” Journal of Abnormal Child Psychology, vol. 35, no. 3, pp. 405–416, 2007.
[8]
J. V. Carney, “Bullied to death: perceptions of peer abuse and suicidal behaviour during adolescence,” School Psychology International, vol. 21, no. 2, pp. 213–223, 2000.
[9]
M. Brendgen, F. Vitaro, and W. M. Bukowski, “Affiliation with delinquent friends: contributions of parents, self-esteem, delinquent behavior, and rejection by peers,” Journal of Early Adolescence, vol. 18, no. 3, pp. 244–265, 1998.
[10]
M. E. Feinberg, T. A. Ridenour, and M. T. Greenberg, “Aggregating indices of risk and protection for adolescent behavior problems: the Communities That Care Youth Survey,” Journal of Adolescent Health, vol. 40, no. 6, pp. 506–513, 2007.
[11]
J. D. Hawkins, R. F. Catalano, and J. Y. Miller, “Risk and protective factors for alcohol and other drug problems in adolescence and early adulthood: implications for substance abuse prevention,” Psychological Bulletin, vol. 112, no. 1, pp. 64–105, 1992.
[12]
C. Spooner, “Causes and correlates of adolescent drug abuse and implications for treatment,” Drug and Alcohol Review, vol. 18, no. 4, pp. 453–475, 1999.
[13]
C. B. Fleming, K. P. Haggerty, R. F. Catalano, T. W. Harachi, J. J. Mazza, and D. H. Gruman, “Do social and behavioral characteristics targeted by preventive interventions predict standardized test scores and grades?” Journal of School Health, vol. 75, no. 9, pp. 342–349, 2005.
[14]
L. J. Woodward and D. M. Fergusson, “Childhood peer relationship problems and later risks of educational under-achievement and unemployment,” Journal of Child Psychology and Psychiatry and Allied Disciplines, vol. 41, no. 2, pp. 191–201, 2000.
[15]
J. D. Coie, K. A. Dodge, R. Terry, and V. Wright, “The role of aggression in peer relations: an analysis of aggression episodes in boys' play groups,” Child development, vol. 62, no. 4, pp. 812–826, 1991.
[16]
M. E. DeRosier, J. B. Kupersmidt, and C. J. Patterson, “Children's academic and behavioral adjustment as a function of the chronicity and proximity of peer rejection,” Child Development, vol. 65, no. 6, pp. 1799–1813, 1994.
[17]
M. Greenberg, C. Domitrovich, and B. Bumbarger, “Prevention of mental disorders in school-aged children: current state of the field,” Prevention & Treatment, vol. 4, no. 1, pp. 1–52, 2001.
[18]
K. W. Merrell, “Assessment of children’s social skills: recent developments, best practices, and new directions,” Exceptionality, vol. 9, no. 1-2, pp. 3–18, 2001.
[19]
K. W. Merrell, Behavioral, Social, and Emotional Assessment of Children and Adolescents, Laurence Erlbaum Associates, Mahwah, NJ, USA, 1999.
[20]
K. W. Merrell and G. A. Gimpel, Social Skills of Children and Adolescents: Conceptualization, Assessment, Treatment, Laurence Erlbaum Associates, Mahwah, NJ, USA, 1998.
[21]
V. J. Shute and F. Ke, “Games, learning, and assessment,” in Assessment in Game-Based Learning: Foundations, Innovations, and Perspectives, D. Ifenthaler, D. Eseryel, and X. Ge, Eds., Springer, New York, NY, USA, 2012.
[22]
Y. Cheong, A. Jhala, B. Bae, and R. M. Young, “Automatically generating summaries from game logs,” in Proceedings of the 4th Artificial Intelligence and Interactive Digital Entertainment International Conference (AIIDE '08), pp. 167–172, 2008.
[23]
J. Rowe and J. Lester, “Modeling user knowledge with dynamic bayesian networks in interactive narrative environments,” in Proceedings of the 6th Annual AI and Interactive Digital Entertainment Conference, pp. 57–62, Palo Alto, Calif, USA, 2010.
[24]
A. Tveit and G. B. Tveit, “Game usage mining: information gathering for knowledge discovery in massive multiplayer games,” in Proceedings of the International Conference on Internet Computing (IC ’02), Session on Web Mining, 2002.
[25]
V. J. Shute and D. Zapata-Rivera, “Educational assessment using intelligent systems,” Educational Testing Service Report RR-08-68, ETS, Princeton, NJ, USA, 2008.
[26]
M. E. DeRosier and S. H. Mercer, “Improving student behavior: the effectiveness of a school-based character education program,” Journal of Research and Character Education, vol. 5, pp. 131–148, 2007.
[27]
F. Gresham and S. Elliott, Social Skills Rating System, American Guidance Service, Circle Pines, Minn, USA, 1990.
[28]
Conduct Problems Prevention Research Group (CPPRG), Social Competence Scale (Teacher Version), 1990, http://www.fasttrackproject.org/.
[29]
M. E. DeRosier, “Using computer-based social tasks to assess students’ social skills: findings from the Zoo U pilot evaluation,” Final Report for the U.S. Department of Education, Washington, DC, USA, 2011.
