To avoid use errors when handling medical equipment, it is important to develop products with a high degree of usability. This can be achieved by performing usability evaluations in the product development process to detect and mitigate potential usability problems. A commonly used method is cognitive walkthrough (CW), but this method shows three weaknesses: poor high-level perspective, insufficient categorisation of detected usability problems, and difficulties in overviewing the analytical results. This paper presents a further development of CW with the aim of overcoming its weaknesses. The new method is called enhanced cognitive walkthrough (ECW). ECW is a proactive analytical method for analysis of potential usability problems. The ECW method has been employed to evaluate user interface designs of medical equipment such as home-care ventilators, infusion pumps, dialysis machines, and insulin pumps. The method has proved capable of identifying several potential use problems in designs. 1. Introduction In the development of user interfaces, it is important to consider the need for these to be simple and safe to handle for the user group in the intended context. This is especially true of safety-critical technical equipment such as medical equipment, where a possibility of harm to patients can arise from erroneous use of the devices [1–4]. Several studies have shown that there is a clear connection between problems of usability and human error; for example, Obradovich and Woods [5], Lin et al. [6], and the FDA [7] have also referred to this problem. Liljegren [2] has shown in a doctoral thesis that medical personnel rank “difficulty of making mistakes” as the most important aspect of good usability for medical equipment. An important step in the development of usable technology is to try in advance to identify and evaluate the occasions, in the interaction between user and product, when there is a possibility of errors arising [8]. To identify the problems that can give rise to errors in handling a product, evaluations are normally made of the product’s user interface with realistic tasks, that is, in a usability evaluation. Jaspers [9] presents an overview of methods used in medical technology usability evaluations. The usability evaluation of user interfaces can proceed according to two different approaches: empirical and analytical [3, 10]. Empirical evaluation involves studies of users who interact with the user interface by carrying out different tasks, which is done in what are known as usability tests [11]. Usability tests have been employed to
References
[1]
D. R. Kaufman, V. L. Patel, C. Hilliman et al., “Usability in the real world: assessing medical information technologies in patients' homes,” Journal of Biomedical Informatics, vol. 36, no. 1-2, pp. 45–60, 2003.
[2]
E. Liljegren, Increasing the Usability of Medical Technology Methodological Considerations for Evaluation, Department of Product and Production Development, Chalmers University of Technology, G？teborg, Sweden, 2004.
[3]
Y. Liu, Usability Evaluation Pf Interface Design for Medical Devices—Exploring Methodologies for Evaluation Prototypes, Department of Product and Production Development, Chalmers University of Technology, G？teborg, Sweden, 2004.
[4]
J. Zhang, V. L. Patel, T. R. Johnson, and E. H. Shortliffe, “A cognitive taxonomy of medical errors,” Journal of Biomedical Informatics, vol. 37, no. 3, pp. 193–204, 2004.
[5]
J. H. Obradovich and D. D. Woods, “Users as designers: how people cope with poor HCI design in computer-based medical devices,” Human Factors, vol. 38, no. 4, pp. 574–592, 1996.
[6]
L. Lin, R. Isla, K. Doniz, H. Harkness, K. J. Vicente, and D. J. Doyle, “Applying human factors to the design of medical equipment: patient-controlled analgesia,” Journal of Clinical Monitoring and Computing, vol. 14, no. 4, pp. 253–263, 1998.
[7]
U.S. Food and Drug Administration, Postmarket information—Device surveillance and reporting processes. 09/29/2011, http://www.fda.gov/medicaldevices/deviceregulationandguidance/humanfactors/ucm124851.htm.
[8]
J. Zhang, T. R. Johnson, V. L. Patel, D. L. Paige, and T. Kubose, “Using usability heuristics to evaluate patient safety of medical devices,” Journal of Biomedical Informatics, vol. 36, no. 1-2, pp. 23–30, 2003.
[9]
M. W. M. Jaspers, “A comparison of usability methods for testing interactive health technologies: methodological aspects and empirical evidence,” International Journal of Medical Informatics, vol. 78, no. 5, pp. 340–353, 2009.
[10]
H. R. Hartson, T. S. Andre, and R. C. Williges, “Criteria for evaluating usability evaluation methods,” International Journal of Human-Computer Interaction, vol. 13, no. 4, pp. 373–410, 2001.
[11]
J. Nielsen, Usability Engineering, Academic Press, Boston, Mass, USA, 1993.
[12]
K. Garmer, E. Liljegren, A. Osvalder, and S. Dahlman, “Application of usability testing to the development of medical equipment. Usability testing of a frequently used infusion pump and a new user interface for an infusion pump developed with a human factors approach,” International Journal of Industrial Ergonomics, vol. 29, no. 3, pp. 145–159, 2002.
[13]
A. W. Kushniruk and V. L. Patel, “Cognitive and usability engineering methods for the evaluation of clinical information systems,” Journal of Biomedical Informatics, vol. 37, no. 1, pp. 56–76, 2004.
[14]
J. Nielsen and R. L. Mack, Eds., Usability Inspection Methods, Wiley, New York, NY, USA, 1994.
[15]
M. J. Graham, T. K. Kubose, D. Jordan, J. Zhang, T. R. Johnson, and V. L. Patel, “Heuristic evaluation of infusion pumps: implications for patient safety in Intensive Care Units,” International Journal of Medical Informatics, vol. 73, no. 11-12, pp. 771–779, 2004.
[16]
C. Lewis, P. Polson, C. Wharton, and J. Rieman, Testing a Walkthrough Methodology for Theory-Based Design of Walk-Up-and-Use Interfaces, ACM, Seattle, Wash, USA, 1990.
[17]
P. G. Polson, C. Lewis, J. Rieman, and C. Wharton, “Cognitive walkthroughs: a method for theory-based evaluation of user interfaces,” International Journal of Man-Machine Studies, vol. 36, no. 5, pp. 741–773, 1992.
[18]
C. Lewis and C. Wharton, “Cognitive walkthrough,” in Handbook of Human-Computer Interaction, M. Helander, T. K. Landauer, and P. Prabhu, Eds., pp. 717–732, Elsevier Science BV, New York, NY, USA, 1997.
[19]
A. W. Kushniruk, D. R. Kaufman, V. L. Patel, Y. Lévesque, and P. Lottin, “Assessment of a computerized patient record system: a cognitive approach to evaluating medical technology,” M.D. Computing, vol. 13, no. 5, pp. 406–415, 1996.
[20]
J. Horsky, D. R. Kaufman, M. I. Oppenheim, and V. L. Patel, “A framework for analyzing the cognitive complexity of computer-assisted clinical ordering,” Journal of Biomedical Informatics, vol. 36, no. 1-2, pp. 4–22, 2003.
[21]
E. Liljegren and A. Osvalder, “Cognitive engineering methods as usability evaluation tools for medical equipment,” International Journal of Industrial Ergonomics, vol. 34, no. 1, pp. 49–62, 2004.
[22]
Y. Liu, A. Osvalder, and S. Dahlman, “Exploring user background settings in cognitive walkthrough evaluation of medical prototype interfaces: a case study,” International Journal of Industrial Ergonomics, vol. 35, no. 4, pp. 379–390, 2005.
[23]
H. Ryu, “Collective web usability analysis: cognitive and activity walkthroughs,” International Journal of Web Engineering and Technology, vol. 4, no. 3, pp. 286–312, 2008.
[24]
C. Wharton, J. Bradford, R. Jeffries, and M. Franzke, “Applying cognitive walkthroughs to more complex user interfaces: experiences, issues, and recommendations,” in Proceedings of the ACM Conference on Human Factors in Computing Systems (CHI '92), pp. 381–388, May 1992.
[25]
A. Holzinger, “Usability engineering methods for software developers,” Communications of the ACM, vol. 48, no. 1, pp. 71–74, 2005.
[26]
A. Chapanis, “Some reflections on progress,” in Proceedings of the Human Factors Society 20th Meeting, Santa Monica, Calif, USA, 1985.
[27]
K. T. Ulrich and S. D. Eppinger, Product Design and Development, McGraw-Hill, Boston, Mass, USA, 2004.
[28]
N. A. Stanton, “Hierarchical task analysis: developments, applications, and extensions,” Applied Ergonomics, vol. 37, no. 1, pp. 55–79, 2006.
[29]
B. Kirwan and L. K. Ainsworth, Guide to Task Analysis, Taylor & Francis, London, UK, 1992.
[30]
D. E. Embrey, “SHERPA: a systematic human error reduction and prediction approach,” in Proceedings of the International Topical Meeting on Advances in Human Factors in Nuclear Power System, American Nuclear Society, Knoxville, Tenn, USA, 1986.
[31]
C. Baber and N. A. Stanton, “Task analysis for error identification: a methodology for designing error-tolerant consumer products,” Ergonomics, vol. 37, no. 11, pp. 1923–1941, 1994.
[32]
J. May and P. Barnard, “The case for supportive evaluation during design,” Interacting with Computers, vol. 7, no. 2, pp. 115–143, 1995.
[33]
C. Wharton, J. Rieman, C. Lewis, and P. G. Polson, “The cognitive walkthrough method: a practitioners's guide,” in Usability Inspection Methods, J. Nielsen and R. L. Mack, Eds., pp. 106–139, John Wiley & Sons, New York, NY, USA, 1994.
[34]
L. O. Blig？rd and S. Wass, Analysis and Development of User Interface for Home Care Airflow Generators, Product and Production Develepment, Chalmers Universtity of Technology, G？teborg, Sweden, 2002.
[35]
Y. Liu, Usability Evaluation of Medical Technology: Investigatingthe Effect of User Background and Users Expertise, Chalmers University of Technology, G？teborg, Sweden, 2009.
[36]
National Patient Safety Agency, 2008, A risk matrix for risk managers, http://www.nrls.npsa.nhs.uk/EasySiteWeb/getresource.axd?AssetID=60149&...
[37]
E. W. Israelski and W. H. Muto, Human Factors Risk Management as a Way to Improve Medical Device Safety: A Case Study of the Therac-25 Radiation Therapy System in Using Human Factors to Improve Patient Safety, J. W. Gosbee and L. L. Gosbee, Eds., Joint Commission Resources, Oakbrook Terrace, Ill, USA, 2005.
[38]
L. O. Blig？rd and A. L. Osvalder, “An analytical approach for predicting and identifying use error and usability problem,” vol. 4799 of Lecture Notes in Computer Science, pp. 427–440, Springer, Berlin, Germany, 2007.
[39]
L.-O. Blig？rd and A.-L. Osvalder, “CCPE-methodology for a combined evaluation of cognitive and physical ergonomics in the interaction between human and machine,” Human Factors and Ergonomics in Manufacturing & Service Industries, vol. 19, no. 6, 2012.
[40]
R. Jeffries, J. R. Miller, C. Wharton, and K. M. Uyeda, User Interface Evaluation in the Real World: A Comparison of Four Techniques, ACM, New Orleans, La, USA, 1991.
[41]
Z. Ruttkay and R. Op Den Akker, “Affordances and cognitive walkthrough for analyzing human-virtual human interaction,” in Verbal and Nonverbal Features of Human-Human and Human-Machine Interaction, Lecture Notes in Computer Science (Including Subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), pp. 90–106, Springer, Berlin, Germany, 2008.
[42]
P. Koutsabasis, T. Spyrou, and J. Darzentas, “Evaluating usability evaluation methods: criteria, method and a case study,” in Human-Computer Interaction. Interaction Design and Usability, Lecture Notes in Computer Science (Including Subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), pp. 569–578, Springer, Berlin, Germany, 2007.
[43]
T. Hollingsed and D. G. Novick, “Usability inspection methods after 15 years of research and practice,” in Proceedings of the 25th ACM International Conference on Design of Communication (SIGDOC '07), pp. 249–255, October 2007.
[44]
J. R. Miller and R. Jeffries, “Interface-usability evaluation: science of trade-offs,” IEEE Software, vol. 9, no. 5, pp. 97–98, 1992.
[45]
L. O. Blig？rd, Prediction of Medical Device Usability Problems and Use Errors—An Improved Analytical Methodical Approach, Chalmers University of Technology, G？teborg, Sweden, 2007.
[46]
L. O. Blig？rd, S. Wass, E. Liljegren, and A. L. Osvalder, “Using a human factors engineering process to develop new user interfaces for home car air-flow generators,” in Proceedings of the 35th Annual Congress of the Nordic Ergonomics Society Conference, Reykjavik, Iceland, 2003.
[47]
A. Gross and J. Foufas, Developing a Human Factors Engineering Process for Medical Device Design. Methodological Considerations, Product and Production Development, Chalmers University of Technology, G？teborg, Sweden, 2003.
[48]
E. Liljegren, L. O. Blig？rd, and A. L. Osvalder, “Developing user-friendly interfaces for medical devices,” in Proceedings of the 8th IFAC Symposium on Automated Systems Based on Human Skill, G？teborg, Sweden, 2003.
[49]
L. O. Blig？rd and A. L. Osvalder, Using Enhanced Cognitive Walkthrough as a Usability Evaluation Method for Medical Equipment, IEA, Maastricht, The Netherlands, 2006.
[50]
L. O. Blig？rd, M. Eriksson, and A. L. Osvalder, “Cassette versus conventional bloodlines—analytical evaluation of three dialysis machines,” 2006, Internal Report Gambro Lundia AB, Classified.
[51]
L. O. Blig？rd, M. Eriksson, and A. L. Osvalder, “Work physiology evaluation of new machine prototype II,” 2006, Internal Report Gambro Lundia AB, Classified.
