There is a growing debate in the literature regarding the tradeoffs between lab and field evaluation of mobile devices. This paper presents a comparison of field-based and lab-based experiments to evaluate user experience of personalised mobile devices at large sports events. A lab experiment is recommended when the testing focus is on the user interface and application-oriented usability related issues. However, the results suggest that a field experiment is more suitable for investigating a wider range of factors affecting the overall acceptability of the designed mobile service. Such factors include the system function and effects of actual usage contexts aspects. Where open and relaxed communication is important (e.g., where participant groups are naturally reticent to communicate), this is more readily promoted by the use of a field study. 1. Introduction Usability analysis of systems involving stationary computers has grown to be an established discipline within human-computer interaction. Established concepts, methodologies, and approaches in HCI are being challenged by the increasing focus on mobile applications. Real-world ethnographic studies have received relatively little attention within the HCI literature, and little specific effort has been spent on delivering solid design methodologies for mobile applications [1]. Researchers and practitioners have been encouraged to investigate further the criteria, methods, and data collection techniques for usability evaluation of mobile applications [2]. Lab-based experiments and field-based experiments are the methods most discussed in relation to evaluating a mobile application [2–4]. There has been considerable debate over whether interactions with mobile systems should be investigated in the field or in the more traditional laboratory environment. There seems to be an implicit assumption that the usability of a mobile application can only be properly evaluated in the field, for example, Kjeldskov and Stage [5]. Some argue that it is important that mobile applications are tested in realistic settings, since testing in a conventional usability lab is unlikely to find all problems that would occur in real mobile usage (e.g., [2, 6, 7]). For example, Christensen et al. [6] presented a study of how ethnographic fieldwork can be used to study children’s mobility patterns via mobile phones. Authors consider that field studies make it possible to carry out analysis that can broaden and deepen understanding of peoples’ everyday life. However, some authors have highlighted how ethnographic field
References
[1]
D. Raptis, N. Tselios, and N. Avouris, “Context-based design of mobile applications for museums: a survey of existing practices,” in Proceedings of the 7th International Conference on Human Computer Interaction with Mobile Devices & Services, 2005.
[2]
C. P. J. M. Van Elzakker, I. Delikostidis, and P. J. M. Van Oosterom, “Field-based usability evaluation methodology for mobile geo-applications,” Cartographic Journal, vol. 45, no. 2, pp. 139–149, 2008.
[3]
J. Kjeldskov, M. B. Skov, B. S. Als, and R. T. H？egh, “Is it worth the hassle? Exploring the added value of evaluating the usability of context-aware mobile systems in the Field,” in Proceedings of the 6th International Mobile Conference (HCI '04), Springer, 2004.
[4]
X. Sun, D. Golightly, J. Cranwelly, B. Bedwell, and S. Sharples, “Participant experiences of mobile device-based diary studies,” International Journal of Mobile Human Computer Interaction. In press.
[5]
J. Kjeldskov and J. Stage, “New techniques for usability evaluation of mobile systems,” International Journal of Human Computer Studies, vol. 60, no. 5-6, pp. 599–620, 2004.
[6]
P. Christensen, M. Romero, M. Thomas, A. S. Nielsen, and H. Harder, “Children, mobility, and space: using GPS and mobile phone technologies in ethnographic research,” Journal of Mixed Methods Research, vol. 5, no. 3, pp. 227–246, 2011.
[7]
E. G. Coleman, “Ethnographic approaches to digital media,” Annual Review of Anthropology, vol. 39, pp. 487–505, 2010.
[8]
L. Baillie, “Future Telecommunication: exploring actual use,” in Proceedings of the International Conference on Human-Computer Interaction, IOS press, 2003.
[9]
M. Esbj？rnsson, B. Brown, O. Juhlin, D. Normark, M. ？stergren, and E. Laurier, “Watching the cars go round and round: designing for active spectating,” in Proceedings of the Conference on Human Factors in Computing Systems (CHI '06), pp. 1221–1224, New York, NY, USA, April 2006.
[10]
X. Sun, “User requirements of personalized mobile applications at large sporting events,” in Proceedings of the IADIS Multi Conference on Computer Science and Information Systems(MCCSIS '10), Freiburg, Germany, 2010.
[11]
H. B. L. Duh, G. C. B. Tan, and V. H. H. Chen, “Usability evaluation for mobile device: a comparison of laboratory and field tests,” in Proceedings of the 8th International Conference on Human-Computer Interaction with Mobile Devices and Services (MobileHCI '06), pp. 181–186, September 2006.
[12]
A. Pirhonen, S. Brewster, and C. Holguin, “Gestural and audio metaphors as a means of control for mobile devices,” in Proceedings of the Conference on Human Factors in Computing Systems (CHI '02), pp. 291–298, New York, NY, USA, April 2002.
[13]
R. Graham and C. Carter, “Comparison of speech input and manual control of in-car devices while on-the-move,” in Proceedings of the 2nd Workshop on Human Computer Interaction with Mobile Devices (HCI '1999), Edinburgh, UK, 1999.
[14]
J. Lai, K. Cheng, P. Green, and O. Tsimhoni, “On the road and on the web? Comprehension of synthetic and human speech while driving,” in Proceedings of the Conference on Human Factors in Computing Systems (CHI '01), pp. 206–212, New York, NY, USA, April 2001.
[15]
B. Mobasher, “Data mining for personalization,” in The Adaptive Web: Methods and Strategies of Web Personalization, P. Brusilovsky, A. Kobsa, and W. Nejdl, Eds., pp. 1–46, Springer, Berlin, Germany, 2007.
[16]
D. Wu, I. Im, M. Tremaine, K. Instone, and M. Turoff, “A framework for classifying personalization schemes used on e-commerce websites,” in Proceedings of the 36th Hawaii International Conference on Systems Sciences (HICSS '03), p. 222b, Maui, Hawaii, USA, 2003.
[17]
L. Norros, E. Kaasinen, J. Plomp, and P. Rama, Human-Technology Interaction Research and Design. VTT Roadmap, VTT Research Notes 2220, Espoo, Finland, 2003.
[18]
E. Frias-Martinez, S. Y. Chen, and X. Liu, “Evaluation of a personalized digital library based on cognitive styles: adaptivity versus adaptability,” International Journal of Information Management, vol. 29, no. 1, pp. 48–56, 2009.
[19]
J. Dewey, Art as Experience, Perigee, New York, NY, USA, 1980.
[20]
J. Rasmussen, “Human factors in a dynamic information society: where are we heading?” Ergonomics, vol. 43, no. 7, pp. 869–879, 2000.
[21]
L. Arhippainen and M. T？hti, “Empirical Evaluation of User Experience in Two Adaptive Mobile Application Prototypes,” in Proceedings of the 2nd International Conference on Mobile and Ubiquitous Multimedia, Lule？, Sweden, 2003.
[22]
M. Hassenzahl and N. Tractinsky, “User experience—a research agenda,” Behaviour and Information Technology, vol. 25, no. 2, pp. 91–97, 2006.
[23]
S. Siegel and N. J. Castellan, Nonparametric Statistics For the Behavioral Sciences, McGraw-Hill, New York, NY, USA, 1988.
[24]
E. Law, V. Roto, A. P. O. S. Vermeeren, J. Kort, and M. Hassenzahl, “Towards a shared definition of user experience,” in Proceedings of the 28th Annual Conference on Human Factors in Computing Systems (CHI '08), pp. 2395–2398, Florence, Italy, April 2008.
[25]
E. L. C. Law and P. van Schaik, “Modelling user experience—an agenda for research and practice,” Interacting with Computers, vol. 22, no. 5, pp. 313–322, 2010.
[26]
C. H. Lin, P. J. Sher, and H. Y. Shih, “Past progress and future directions in conceptualizing customer perceived value,” International Journal of Service Industry Management, vol. 16, no. 4, pp. 318–336, 2005.
[27]
X. Sun and A. May, “Mobile personalisation at large sports events—user experience and mobile device personalisation,” in Human-Computer Interaction 2007, vol. 11, Springer, Berlin, Germany, 2007.
[28]
X. Sun and A. May, “The role of spatial contextual factors in mobile personalization at large sports events,” Personal and Ubiquitous Computing, vol. 13, no. 4, pp. 293–302, 2009.
[29]
J. Hackos and J. Redish, User and Task Analysis For Interface Design, John Wiley & Sons, New York, NY, USA, 1998.
[30]
A. Nilsson, U. Nuldén, and D. Olsson, “Spectator information support: exploring the context of distributed events,” in Proceedings of the International ACM SIGGROUP Conference on Supporting Group Work, 2004.
[31]
B. Oertel, K. Steinmuller, and M. Kuom, “Mobile multimedia services for tourism,” in Information and Communication Technologies in Tourism 2002, K. W. Wober, A. J. Frew, and M. Hitz, Eds., pp. 265–274, Springer, New York, NY, USA, 2002.
[32]
D. D. Salvucci, “Predicting the effects of in-car interfaces on driver behavior using a cognitive architecture,” in Proceedings of the Conference on Human Factors in Computing Systems (CHI '01), pp. 120–127, New York, NY, USA, April 2001.
[33]
A. Kaikkonen, A. Kek？l？inen, M. Cankar, T. Kallio, and A. Kankainen, “Usability testing of mobile applications: a comparison between laboratory and field testing,” Journal of Usability Studies, vol. 1, pp. 4–16, 2005.
