There is a consensus that serious games have a significant potential as a tool for instruction. However, their effectiveness in terms of learning outcomes is still understudied mainly due to the complexity involved in assessing intangible measures. A systematic approach—based on established principles and guidelines—is necessary to enhance the design of serious games, and many studies lack a rigorous assessment. An important aspect in the evaluation of serious games, like other educational tools, is user performance assessment. This is an important area of exploration because serious games are intended to evaluate the learning progress as well as the outcomes. This also emphasizes the importance of providing appropriate feedback to the player. Moreover, performance assessment enables adaptivity and personalization to meet individual needs in various aspects, such as learning styles, information provision rates, feedback, and so forth. This paper first reviews related literature regarding the educational effectiveness of serious games. It then discusses how to assess the learning impact of serious games and methods for competence and skill assessment. Finally, it suggests two major directions for future research: characterization of the player’s activity and better integration of assessment in games. 1. Introduction Serious games are designed to have an impact on the target audience, which is beyond the pure entertainment aspect [1, 2]. One of the most important application domains is in the field of education given the acknowledged potential of serious games to meet the current need for educational enhancement [3, 4]. In this field, the purpose of a serious game is twofold: (i) to be fun and entertaining, and (ii) to be educational. A serious game is thus designed both to be attractive and appealing to a broad target audience, similar to commercial games, and to meet specific educational goals as well. Therefore, assessment of a serious game must consider both aspects of fun/enjoyment and educational impact. In addition to considering fun and engagement, thus, serious games’ assessment presents additional unique challenges, because learning is the primary goal. Therefore, there is also a need to explore how to evaluate the learning outcomes to identify which serious games are most suited for a given goal or domain, and how to design more effective serious games (e.g., what mechanics are most suited for a given pedagogical goal, etc.). In this sense, the evaluation of serious games should also cover player performance assessment. Performance assessment
References
[1]
J. P. Gee, What Video Games Have to Teach Us about Learning and Literacy, Palgrave MacMillan, New York, NY, USA, 2007.
[2]
F. L. Greitzer, O. A. Kuchar, and K. Huston, “Cognitive science implications for enhancing training effectiveness in a serious gaming context,” ACM Journal on Educational Resources in Computing, vol. 7, no. 3, article 2, 2007.
[3]
F. De Grove, P. Mechant, and J. Van Looy, “Uncharted waters? Exploring experts' opinions on the opportunities and limitations of serious games for foreign language learning,” in Proceedings of the 3rd International Conference on Fun and Games, pp. 107–115, Leuven, Belgium, September 2010.
[4]
R. van Eck, “Digital game-based learning: it's not just the digital natives who are restless,” EDUCAUSE Review, vol. 41, no. 2, pp. 16–30, 2006.
[5]
J. Cannon-Bowers, “The state of gaming and simulation,” in Proceedings of the Training Conference and Expo, Orlando, Fla, USA, March 2006.
[6]
V. Shute, M. Ventura, M. Bauer, and D. Zapata-Rivera, “Melding the power of serious games and embedded assessment to monitor and foster learning: flow and grow,” in Serious Games: Mechanisms and Effects, U. Ritterfeld, M. Cody, and P. Vorderer, Eds., pp. 295–321, Routledge, Taylor and Francis, Mahwah, NJ, USA, 2009.
[7]
J. Gosen and J. Washbush, “A review of scholarship on assessing experiential learning effectiveness,” Simulation & Gaming, vol. 35, no. 2, pp. 270–293, 2004.
[8]
R. T. Hays, “The effectiveness of instructional games: a literature review and discussion,” Tech. Rep. 2005-004, Naval Air Warfare Center, Training Systems Division, 2005.
[9]
A. A. Kulik, “School mathematics and science programs benefit from instructional technology,” United States National Science Foundation (NSF), National Center for Science and Engineering Statistics (NCSES), InfroBrief NSF-03-301, November 2002, http://www.nsf.gov/statistics/infbrief/nsf03301/.
[10]
R. Blunt, “Do serious games work? Results from three studies,” eLearn Magazine, vol. 2009, no. 12, 2009.
[11]
B. Bergeron, Developing Serious Games, Thomson Delmar Learning, Hingham, Mass, USA, 2006.
[12]
M. Prensky, Don't Bother Me Mom—I'm Learning!, Paragon House, 2006.
[13]
S. Livingston, G. Fennessey, J. Coleman, K. Edwards, and S. Kidder, “The Hopkins games program: final report on seven years of research,” Report No. 155, Johns Hopkins University, Center for Social Organization of Schools, Baltimore, Md, USA, 1973.
[14]
J. Chin, R. Dukes, and W. Gamson, “Assessment in simulation and gaming: a review of the last 40 years,” Simulation & Gaming, vol. 40, no. 4, pp. 553–568, 2009.
[15]
T. M. Connolly, E. A. Boyle, E. MacArthur, T. Hainey, and J. M. Boyle, “A systematic literature review of the empirical evidence on computer games and serious games,” Computers and Education, vol. 59, no. 2, pp. 661–686, 2012.
[16]
P. M. Kato, S. W. Cole, A. S. Bradlyn, and B. H. Pollock, “A video game improves behavioral outcomes in adolescents and young adults with cancer: a randomized trial,” Pediatrics, vol. 122, no. 2, pp. e305–e317, 2008.
[17]
S. W. Cole, D. J. Yoo, and B. Knutson, “Interactivity and reward-related neural activation during a serious videogame,” PLoS ONE, vol. 7, no. 3, Article ID e33909, 2012.
[18]
S. D. Dandeneau and M. W. Baldwin, “The inhibition of socially rejecting information among people with high versus low self-esteem: the role of attentional bias and the effects of bias reduction training,” Journal of Social and Clinical Psychology, vol. 23, no. 4, pp. 584–602, 2004.
[19]
F. Bellotti, R. Berta, A. De Gloria, A. D'Ursi, and V. Fiore, “A serious game model for cultural heritage,” Journal on Computing and Cultural Heritage, vol. 5, no. 4, pp. 1–27, 2012.
[20]
F. Bellotti, R. Berta, and A. De Gloria, “Designing effective serious games: opportunities and challenges for research,” International Journal of Emerging Technologies in Learning, vol. 5, pp. 22–35, 2010.
[21]
G. Bente and J. Breuer, “Making the implicit explicit: embedded measurement in serious games,” in Serious Games: Mechanisms and Effects, U. Ritterfield, M. J. Cody, and P. Vorderer, Eds., pp. 322–343, Routledge, New York, NY, USA, 2009.
[22]
D. Michael and S. Chen, “Proof of learning: assessment in serious games,” October 2005, http://www.gamasutra.com/view/feature/2433/proof_of_learning_assessment_in_.php.
[23]
J. Enfield, R. D. Myers, M. Lara, and T. W. Frick, “Innovation diffusion: assessment of strategies within the diffusion simulation game,” Simulation & Gaming, vol. 43, no. 2, pp. 188–214, 2012.
[24]
C. Boston, “The concept of formative assessment,” Practical Assessment, Research & Evaluation, vol. 8, no. 9, 2002.
[25]
P. Moreno-Ger, D. Burgos, and J. Torrente, “Digital games in eLearning environments: current uses and emerging trends,” Simulation & Gaming, vol. 40, no. 5, pp. 669–687, 2009.
[26]
C. Sebastian, A. Anantachai, J. H. Byun, and J. Lenox, “Assessing what players learned in serious games: in-situ data collection, information trails, and quantitative analysis,” in Proceedings of the 10th International Conference on Computer Games: AI, Animation, Mobile, Educational and Serious Games, pp. 10–19, 2007.
[27]
K. Becker and J. R. Parker, The Guide to Computer Simulations and Games, John Wiley & Sons, Indianapolis, Ind, USA, 2011.
[28]
P. Dugard and J. Todman, “Analysis of pre-test-post-test control group designs in educational research,” Educational Psychology, vol. 15, no. 2, pp. 181–198, 1995.
[29]
National Center for Technology Innovation (NCTI), “Experimental Study Design,” 2012, http://www.nationaltechcenter.org/index.php/products/at-research-matters/experimental-study-design/.
[30]
L. Allen, M. Seeney, L. Boyle, and F. Hancock, “The implementation of team based assessment in serious games,” in Proceedings of the 1st Conference in Games and Virtual Worlds for Serious Applications (VS-GAMES '09), pp. 28–35, Coventry, UK, March 2009.
[31]
K. Corti, Game-Based Learning: A Serious Business Application, PIXELearning, Coventry, UK, 2006.
[32]
J. H. Brockmyer, C. M. Fox, K. A. Curtiss, E. McBroom, K. M. Burkhart, and J. N. Pidruzny, “The development of the Game Engagement Questionnaire: a measure of engagement in video game-playing,” Journal of Experimental Social Psychology, vol. 45, no. 4, pp. 624–634, 2009.
[33]
W. A. IJsselsteijn, W. van de Hoogen, C. Klimmt et al., “Measuring the experience of digital game enjoyment,” in Proceedings of Measuring Behavior, pp. 88–89, Maastricht, The Netherlands, August 2008.
[34]
M. Csikszentmihalyi, Flow: The Psychology of Optimal Experience, Harper & Row, New York, NY, USA, 1990.
[35]
B. Cowley, D. Charles, M. Black, and R. Hickey, “Toward an understanding of flow in video games,” Computers in Entertainment, vol. 6, no. 2, pp. 1–28, 2008.
[36]
P. Sweetser and P. Wyeth, “GameFlow: a model for evaluating player enjoyment in games,” ACM Computers in Entertainment, vol. 3, no. 3, pp. 1–24, 2005.
[37]
F. L. Fu, R. C. Su, and S. C. Yu, “EGameFlow: a scale to measure learners' enjoyment of e-learning games,” Computers and Education, vol. 52, no. 1, pp. 101–112, 2009.
[38]
S. H. Janicke and A. Ellis, “Psychological and physiological differences between the 3D and 2D gaming experience,” in Proceedings of the 3D Entertainment Summit, Hollywood, Calif, USA, September, 2011.
[39]
H. F. Jelinek, K. August, H. Imam, A. H. Khandoker, A. Koenig, and R. Riener, “Heart rate asymmetry and emotional response to robot assist task challenges in stroke patients,” in Proceedings of the Computing in Cardiology Conference, Hangzhou, China, September 2011.
[40]
J. M. Kivikangas, G. Chanel, B. Cowley et al., “A review of the use of psychophysiological methods in game research,” Journal of Gaming & Virtual Worlds, vol. 3, no. 3, pp. 181–199, 2011.
[41]
L. E. Nacke, Affective ludology: scientific measurement of user experience in interactive entertainment [Ph.D. thesis], Blekinge Institute of Technology, Karlskrona, Sweden, 2009.
[42]
A. Plotnikov, N. Stakheika, A. De Gloria et al., “Exploiting real-time EEG analysis for assessing flow in games,” in Workshop: “Game Based Learning for 21st Century Transferable Skills”, at iCalt 2012, Rome, Italy, June 2012.
[43]
L. E. Nacke, “Physiological game interaction and psychophysiological evaluation in research and industry,” Gamasutra Article, June 2011, http://www.gamasutra.com/blogs/LennartNacke/20110628/7867/Physiological_Game_Interaction_and_Psychophysiological_Evaluation_in_Research_and_Industry.php.
[44]
M. Salminen and N. Ravaja, “Oscillatory brain responses evoked by video game events: the case of super monkey ball 2,” Cyberpsychology & Behavior, vol. 10, no. 3, pp. 330–338, 2007.
[45]
C. Loh, “Designing online games assessment as information trails,” in Games and Simulations in Online Learning: Research and Development Frameworks, D. Gibson, C. Aldrich, and M. Prensky, Eds., pp. 323–348, Information Science Publishing, Hershey, Pa, USA, 2007.
[46]
G. N. Yannakakis and J. Hallam, “Evolving opponents for interesting interactive computer games,” in Proceedings of the International Conference on Computer Games: Artificial Intelligence, Design and Education, 2004.
[47]
T. W. Malone, “Toward a theory of intrinsically motivating instruction,” Cognitive Science, vol. 5, no. 4, pp. 333–369, 1981.
[48]
H. Iida, N. Takeshita, and J. Yoshimura, “A metric for entertainment of boardgames: its implication for evolution of chess variants,” in Proceeding of: Entertainment Computing: Technologies and Applications, IFIP First International Workshop on Entertainment Computing (IWEC '02), R. Nakatsu and J. Hoshino, Eds., pp. 65–72, Kluwer Academic, Boston, Mass, USA, 2003.
[49]
F. Bellotti, R. Berta, A. De Gloria, and L. Primavera, “Enhancing the educational value of video games,” Computers in Entertainment, vol. 7, no. 2, pp. 23–41, 2009.
[50]
M. Hassenzahl and R. Wessler, “Capturing design space from a user perspective: the repertory grid technique revisited,” International Journal of Human-Computer Interaction, vol. 12, no. 3-4, pp. 441–459, 2000.
[51]
N. Zoanetti, “Software for online testing and quizzes,” 2011, http://www.assessmentfocus.com/online-testing.php.
[52]
A. Flynn, F. Concannon, and M. Campbell, “An evaluation of undergraduate students' online assessment performances,” Advanced Technology for Learning, vol. 3, no. 1, pp. 15–51, 2006.
[53]
C. Hewson, “Can online course-based assessment methods be fair and equitable? Relationships between students' preferences and performance within online and offline assessments,” Journal of Computer Assisted Learning, vol. 28, no. 5, pp. 488–498, 2001.
[54]
E. Guzmán, R. Conejo, and J. L. Pérez-de-la-Cruz, “Improving student performance using self-assessment tests,” IEEE Intelligent Systems, vol. 22, no. 4, pp. 46–52, 2007.
[55]
J. Hattie and D. Masters, “asTTle—Assessment Tools for Teaching and Learning,” HEFCE JISC, 2006, http://www.jisc.ac.uk/media/documents/projects/asttle_casestudy.pdf.
[56]
J. Hattie, G. Brown, P. Keegan et al., “Validation evidence of asTTle reading assessment results: norms and criteria,” Asttle Tech. Rep. 22, University of Auckland/Ministry of Education, November 2003.
[57]
J. Hattie, “Large-scale assessment of student competencies,” in Symposium: Working in Today's World of Testing and Measurement: Required Knowledge and Skills (Joint ITC/CPTA Symposium); the 26th International Congress of Applied Psychology, Athens, Greece, July 2006.
[58]
Questionmark Corporation, “Questionmark Perception Measure Knowledge, Skills and Attitudes Securely for Certification, Regulatory Compliance and successful Learning Outcomes,” 2012.
[59]
J. Bull and D. Stephens, “The use of question mark software for formative and summative assessment in two universities,” Innovations in Education and Teaching International, vol. 36, no. 2, pp. 128–135, 1999.
[60]
G. M. Velan, R. K. Kumar, M. Dziegielewski, and D. Wakefield, “Web-based self-assessments in pathology with Questionmark Perception,” Pathology, vol. 34, no. 3, pp. 282–284, 2002.
[61]
HEFCE JISC, “Case study 5: making the most of a computer-assisted assessment system University of Manchester,” 2010, http://www.jisc.ac.uk/media/documents/programmes/elearning/digiassess_makingthemost.pdf.
[62]
M. Wood, Human Computer Collaborative Assessment—Access by Computer (ABC)—University of Manchester, HEFCE JISC, 2009.
S. Jordan and T. Mitchell, “e-Assessment for learning? The potential of short-answer free-text questions with tailored feedback,” British Journal of Educational Technology, vol. 40, no. 2, pp. 371–385, 2009.
[65]
F. Noorbehbahani and A. A. Kardan, “The automatic assessment of free text answers using a modified BLEU algorithm,” Computers & Education, vol. 56, no. 2, pp. 337–345, 2011.
[66]
I. D. Beatty and W. J. Gerace, “Technology-enhanced formative assessment: a research-based pedagogy for teaching science with classroom response technology,” Journal of Science Education and Technology, vol. 18, no. 2, pp. 146–162, 2009.
[67]
C. Fies and J. Marshall, “Classroom response systems: a review of the literature,” Journal of Science Education and Technology, vol. 15, no. 1, pp. 101–109, 2006.
[68]
C. Fies and J. Marshall, “The C3 framework: evaluating classroom response system interactions in university classrooms,” Journal of Science Education and Technology, vol. 17, no. 5, pp. 483–499, 2008.
[69]
IMS Global Learning Consortium, “IMS Question & Test Interoperability Specification (QTI),” 2012, http://www.imsglobal.org/question/.
[70]
C. Smythe and P. Roberts, An Overview of the IMS Question & Test Interoperability Specification, Computer Aided Assessment, Leicestershire, UK, 2000.
[71]
Codility Ltd., “Codility: WE TEST CODERS,” 2009, http://codility.com/.
[72]
About IKM, “Overview,” 2011, http://www.ikmnet.com/about/overview.cfm.
[73]
M. M. Clarke, G. F. Madaus, C. L. Horn, and M. A. Ramos, “Retrospective on educational testing and assessment in the 20th century,” Journal of Curriculum Studies, vol. 32, no. 2, pp. 159–181, 2000.
[74]
L. B. Resnick and D. P. Resnick, Assessing the Thinking Curriculum: New Tools for Educational Reform, Learning Research and Development Center: University of Pittsburgh and Carnegie Mellon University, Pittsburgh, Pa, USA, 1989.
[75]
M. Lipman, “Some thoughts on the formation of reflective education,” in Teaching-Thinking Skills: Theory and Practice, J. B. Baron and R. J. Sternberg, Eds., pp. 151–161, W. H. Freeman, New York, NY, USA, 1987.
[76]
C. Tribune, “Standardized testing will limit students' future,” April 2010, http://articles.chicagotribune.com/2010-04-21/news/chi-100421shafer_briefs_1_standardized-test-scores-teacher-and-principal-evaluations.
[77]
Fairtest, “What's Wrong with Standardized Tests?” May 2012, http://www.fairtest.org/facts/whatwron.htm.
[78]
E. J. Short, M. Noeder, S. Gorovoy, M. J. Manos, and B. Lewis, “The importance of play in both the assessment and treatment of young children,” in An Evidence-Based Approach to Play in Intervention and Prevention: Integrating Developmental and Clinical Science, S. Russ and L. Niec, Eds., Guilford, London, UK.
[79]
A. S. Kaugars and S. W. Russ, “Assessing preschool children's pretend play: preliminary validation of the affect in play scale-preschool version,” Early Education and Development, vol. 20, no. 5, pp. 733–755, 2009.
[80]
F. Bellotti, R. Berta, A. De Gloria, and L. Primavera, “Adaptive experience engine for serious games,” IEEE Transactions on Computational Intelligence and AI in Games, vol. 1, no. 4, pp. 264–280, 2009.
[81]
C. H. Tan, K. C. Tan, and A. Tay, “Dynamic game difficulty scaling using adaptive behavior-based AI,” IEEE Transactions on Computational Intelligence and AI in Games, vol. 3, no. 4, pp. 289–301, 2011.
[82]
L. Doucet and V. Srinivasany, “Designing entertaining educational games using procedural rhetoric: a case study,” in Proceedings of the 5th ACM SIGGRAPH Symposium on Video Games, pp. 5–10, Los Angeles, Calif, USA, July 2010.
[83]
J. Froschauer, I. Seidel, M. G？rtner, H. Berger, and D. Merkl, “Design and evaluation of a serious game for immersive cultural training,” in Proceedings of the 16th International Conference on Virtual Systems and Multimedia (VSMM '10), pp. 253–260, IEEE CS Press, Seoul, Republic of Korea, October 2010.
[84]
H. Kelly, K. Howell, E. Glinert et al., “How to build serious games,” Communications of the ACM, vol. 50, no. 7, pp. 44–49, 2007.
[85]
J. Swarz, A. Ousley, A. Magro et al., “CancerSpace: a simulation-based game for improving cancer-screening rates,” IEEE Computer Graphics and Applications, vol. 30, no. 1, pp. 90–94, 2010.
[86]
M. A. Zielke, M. J. Evans, F. Dufour et al., “Serious games for immersive cultural training: creating a living world,” IEEE Computer Graphics and Applications, vol. 29, no. 2, pp. 49–60, 2009.
[87]
M. King and R. Newman, “Evaluating business simulation software: approach, tools and pedagogy,” On the Horizon, vol. 17, no. 4, pp. 368–377, 2009.
[88]
A. J. Stainton, J. E. Johnson, and E. P. Borodzicz, “Educational validity of business gaming simulation: a research methodology framework,” Simulation & Gaming, vol. 41, no. 5, pp. 705–723, 2010.
[89]
U. Ritterfeld, M. Cody, and P. Vorderer, Eds., Serious Games: Mechanisms and Effects, Routledge, New York, NY, USA, 2009.
[90]
F. Bellotti, R. Berta, A. De Gloria, and L. Primavera, “A task annotation model for SandBox Serious Games,” in Proceedings of IEEE Symposium on Computational Intelligence and Games (CIG '09), pp. 233–240, Milano, Italy, September 2009.
