In this research, we study the relationship between mental workload and facial temperature of aircraft participants during a simulated takeoff flight. We conducted experiments to comprehend the correlation between work and facial temperature within the flight simulator. The experiment involved a group of 10 participants who played the role of pilots in a simulated A-320 flight. Six different flying scenarios were designed to simulate normal and emergency situations on airplane takeoff that would occur in different levels of mental workload for the participants. The measurements were workload assessment, face temperatures, and heart rate monitoring. Throughout the experiments, we collected a total of 120 instances of takeoffs, together with over 10 hours of time-series data including heart rate, workload, and face thermal images and temperatures. Comparative analysis of EEG data and thermal image types, revealed intriguing findings. The results indicate a notable inverse relationship between workload and facial muscle temperatures, as well as facial landmark points. The results of this study contribute to a deeper understanding of the physiological effects of workload, as well as practical implications for aviation safety and performance.
References
[1]
Devos, H., Gustafson, K.M., Ahmadnezhad, P., Liao, K., Mahnken, J.D., Brooks, W.M., Burns, J.M., et al. (2020) Psychometric Properties of NASA-TLX and Index of Cognitive Activity as Measures of Cognitive Workload in Older Adults. Brain Sciences, 10, Article 994. https://doi.org/10.3390/brainsci10120994
[2]
Hananingrum, P., Athqia, A.A. and Wahyudiono, Y.D.A. (2022) Relationship between Age, Gender, Job Placement, and Social Relationships with the Mental Workload of Managers. The Indonesian Journal of Occupational Safety and Health, 11, 377-389. https://doi.org/10.20473/ijosh.v11i3.2022.377-389
[3]
Vidulich, M.A. and Tsang, P.S. (2012) Mental Workload and Situation Awareness. In: Salvendy, G., Ed., Handbook of Human Factors and Ergonomics, Wiley, New York, 243-273. https://doi.org/10.1002/9781118131350.ch8
[4]
Shah, M. and Peikari, H.R. (2016) Electronic Prescribing Usability: Reduction of Mental Workload and Prescribing Errors among Community Physicians. Telemedicine and E-Health, 22, 36-44. https://doi.org/10.1089/tmj.2014.0246
[5]
Zahednezhad, H., Shokrollahi, N., Gheshlagh, R.G. and Afshar, P.F. (2021) Does Heavy Mental Workload Affect Moral Sensitivity among Critical Care Unit Nursing Professionals? A Cross-Sectional Study. BMC Nursing, 20, Article No. 140. https://doi.org/10.1186/s12912-021-00662-8
[6]
Ganesh, K., Snekhalatha, U. and Krishnan, P.T. (2021) Deep Learning Techniques for Automated Detection of Autism Spectrum Disorder Based on Thermal Imaging. Proceedings of the Institution of Mechanical Engineers, Part H: Journal of Engineering in Medicine, 235, 1113-1127. https://doi.org/10.1177/09544119211024778
[7]
Stemberger, J., Allison, R.S. and Schnell, T. (2010) Thermal Imaging as a Way to Classify Cognitive Workload. 2010 Canadian Conference on Computer and Robot Vision, Ontario, 31 May-2 June 2010, 231-238. https://doi.org/10.1109/CRV.2010.37
[8]
Kosonogov, V., Zorzi, L.D., Honoré, J., Martínez-Velázquez, E.S., Nandrino, J., Martínez-Selva, J.M., Sequeira, H., et al. (2017) Facial Thermal Variations: A New Marker of Emotional Arousal. PLOS ONE, 12, e0183592. https://doi.org/10.1371/journal.pone.0183592
[9]
Pollina, D.A., Dollins, A.B., Senter, S.M., Brown, T.E., Pavlidis, I., Levine, J.A., Ryan, A.H., et al. (2006) Facial Skin Surface Temperature Changes during a “Concealed Information” Test. Annals of Biomedical Engineering, 34, 1182-1189. https://doi.org/10.1007/s10439-006-9143-3
[10]
Wang, Z., Horng, G., Hsu, T., Chen, C. and Jong, G. (2020) A Novel Facial Thermal Feature Extraction Method for Non-Contact Healthcare System. IEEE Access, 8, 86545-86553. https://doi.org/10.1109/ACCESS.2020.2992908
[11]
Liu, X., Wang, Y. and Luan, J. (2021) Facial Paralysis Detection in Infrared Thermal Images Using Asymmetry Analysis of Temperature and Texture Features. Diagnostics, 11, Article 2309. https://doi.org/10.3390/diagnostics11122309
[12]
Antoine, M., Abdessalem, H.B. and Frasson, C. (2022) Cognitive Workload Assessment of Aircraft Pilots. Journal of Behavioral and Brain Science, 12, 474-484. https://doi.org/10.4236/jbbs.2022.1210027
[13]
Liu, Y., Ayaz, H. and Shewokis, P.A. (2017) Multisubject “Learning” for Mental Workload Classification Using Concurrent EEG, fNIRS, and Physiological Measures. Frontiers in Human Neuroscience, 11, Article 277607. https://doi.org/10.3389/fnhum.2017.00389
[14]
Hogervorst, M.A., Brouwer, A. and Erp, J.V. (2014) Combining and Comparing EEG, Peripheral Physiology and Eye-Related Measures for the Assessment of Mental Workload. Frontiers in Neuroscience, 8, Article 322. https://doi.org/10.3389/fnins.2014.00322
[15]
Rebsamen, B., Kwok, K. and Penney, T.B. (2011) Evaluation of Cognitive Workload from EEG during a Mental Arithmetic Task. Proceedings of the Human Factors and Ergonomics Society Annual Meeting, 55, 1342-1345. https://doi.org/10.1177/1071181311551279
[16]
Rojas, R.F., Debie, E., Fidock, J., Barlow, M., Kasmarik, K., Anavatti, S.G., Abbass, H.A., et al. (2020) Electroencephalographic Workload Indicators during Teleoperation of an Unmanned Aerial Vehicle Shepherding a Swarm of Unmanned Ground Vehicles in Contested Environments. Frontiers in Neuroscience, 14, Article 40. https://doi.org/10.3389/fnins.2020.00040
[17]
Jafari, M., Zaeri, F., Jafari, A., Najafabadi, A.T.P., Al-Qaisi, S. and Hassanza-deh-Rangi, N. (2020) Assessment and Monitoring of Mental Workload in Subway Train Operations Using Physiological, Subjective, and Performance Measures. Human Factors and Ergonomics in Manufacturing & Service Industries, 30, 165-175. https://doi.org/10.1002/hfm.20831
[18]
Woods, B. and Byrne, A. (2018) The Effect of Multitasking on the Communication Skill and Clinical Skills of Medical Students. BMC Medical Education, 18, Article No. 76. https://doi.org/10.1186/s12909-018-1183-5
[19]
Lim, W.L., Sourina, O., Liu, Y. and Wang, L. (2015) EEG-Based Mental Workload Recognition Related to Multitasking. 2015 10th International Conference on Information, Communications and Signal Processing (ICICS), Singapore, 2-4 December 2015, 1-4. https://doi.org/10.1109/ICICS.2015.7459834
[20]
Han, P., Xu, L. and Lv, X. (2020) Optimised Analysis of Community Medical App User Experience under Cognitive Load Theory. E3S Web of Conferences, 179, Article ID: 02063. https://doi.org/10.1051/e3sconf/202017902063
[21]
Stone, R. and Wei, C. (2011) Exploring the Linkage between Facial Expression and Mental Workload for Arithmetic Tasks. Proceedings of the Human Factors and Ergonomics Society Annual Meeting, 55, 616-619. https://doi.org/10.1177/1071181311551126
[22]
Deng, Z., Chen, Y., Qiang, Y., Xu, Z. and Ye, X. (2023) An Experimental Study on Web Interface Design Optimization Based on User Cognitive Load. Frontiers in Artificial Intelligence and Applications, 365, 506-518. https://doi.org/10.3233/FAIA220744
[23]
Hernàndez-Sabaté, A., Vidalón, J.E.Y., Folch, P., Piera, M.À. and Gil, D. (2022) Recognition of the Mental Workloads of Pilots in the Cockpit Using EEG Signals. Applied Sciences, 12, Article 2298. https://doi.org/10.3390/app12052298
[24]
Wang, X., Li, D., Menassa, C.C. and Kamat, V.R. (2019) Can Infrared Facial Thermography Disclose Mental Workload in Indoor Thermal Environments? Proceedings of the 1st ACM International Workshop on Urban Building Energy Sensing, Controls, Big Data Analysis, and Visualization, New York, NY, 10 November 2019, 87-96. https://doi.org/10.1145/3363459.3363528
[25]
Hassoumi, A., Peysakhovich, V., Coz, A., Hurter, C. and Causse, M. (2022) Thermal Imaging of the Face: Mental Workload Detection in Flight Simulator. In: Ayaz, H. and Asgher, U. Eds., Neuroergonomics and Cognitive Engineering, Springer, Cham. https://doi.org/10.54941/ahfe1001822
[26]
Cacioppo, J.T., Tassinary, L.G. and Berntson, G. (2007) Handbook of Psychophysiology. 3rd Edition, Cambridge University Press, Cambridge.
[27]
Critchley, H. and Harrison, N.A. (2013) Visceral Influences on Brain and Behavior. Neuron, 77, 624-638. https://doi.org/10.1016/j.neuron.2013.02.008
[28]
Füstös, J., Gramann, K., Herbert, B.M. and Pollatos, O. (2012) On the Embodiment of Emotion Regulation: Interoceptive Awareness Facilitates Reappraisal. Social Cognitive and Affective Neuroscience, 8, 911-917. https://doi.org/10.1093/scan/nss089
[29]
Sokolov, A.A., Zeidman, P., Erb, M., Ryvlin, P., Friston, K.J. and Pavlova, M. (2018) Structural and Effective Brain Connectivity Underlying Biological Motion Detection. Proceedings of the National Academy of Sciences, 115, E12034-E12042. https://doi.org/10.1073/pnas.1812859115
[30]
Acharya, U.R., Joseph, K.P., Kannathal, N., Lim, C. and Suri, J.S. (2006) Heart Rate Variability: A Review. Medical and Biological Engineering and Computing, 44, 1031-1051. https://doi.org/10.1007/s11517-006-0119-0
[31]
Valenti, V.E. (2015) The Recent Use of Heart Rate Variability for Research. Journal of Human Growth and Development, 25, 137. https://doi.org/10.7322/jhgd.102991
[32]
Sztajzel, J. (2004) Heart Rate Variability: A Noninvasive Electrocardiographic Method to Measure the Autonomic Nervous System. Swiss Medical Weekly, 134, 514-522.
[33]
McCraty, R. and Shaffer, F. (2015) Heart Rate Variability: New Perspectives on Physiological Mechanisms, Assessment of Self-Regulatory Capacity, and Health Risk. Global Advances in Health and Medicine, 4, 46-61. https://doi.org/10.7453/gahmj.2014.073
[34]
Assaf, A.H., Abdessalem, H.B. and Frasson, C. (2023) Detection and Recuperation of Mental Fatigue. Journal of Behavioral and Brain Science, 13, 15-31. https://doi.org/10.4236/jbbs.2023.132002
[35]
Wang, X., Chen, J. and You, W. (2017) Feature Weight Driven Interactive Mutual Information Modeling for Heterogeneous Bio-Signal Fusion to Estimate Mental Workload. Sensors, 17, Article 2315. https://doi.org/10.3390/s17102315
[36]
Lecoutre, L., Lini, S., Bey, C., Lebour, Q. and Favier, P.A. (2015) Evaluating EEG Measures as a Workload Assessment in an Operational Video Game Setup. Proceedings of the 2nd International Conference on Physiological Computing Systems (PhyCS 2015), Angers, Loire Valley, 11-13 February 2015, 112-117.
[37]
Lowe, D.J., James, S.A., Lloyd, A. and Clegg, G. (2016) Feasibility of EEG to Monitor Cognitive Performance during Venous Cannulation: EEG Distracted Intravenous Access (e-diva). BMJ Simulation and Technology Enhanced Learning, 2, 68-72. https://doi.org/10.1136/bmjstel-2015-000082
[38]
Masaki, A., Nagumo, K., Lamsal, B., Oiwa, K. and Nozawa, A. (2020) Anomaly Detection in Facial Skin Temperature Using Variational Autoencoder. Artificial Life and Robotics, 26, 122-128. https://doi.org/10.1007/s10015-020-00634-2
[39]
Merla, A. and Romani, G.L. (2007) Thermal Signatures of Emotional Arousal: A Functional Infrared Imaging Study. 2007 29th Annual International Conference of the IEEE Engineering in Medicine and Biology Society, Lyon, 22-26 August 2007, 247-249. https://doi.org/10.1109/IEMBS.2007.4352270
[40]
Gane, L., Power, S., Kushki, A. and Chau, T. (2011) Thermal Imaging of the Periorbital Regions during the Presentation of an Auditory Startle Stimulus. PLOS ONE, 6, e27268. https://doi.org/10.1371/journal.pone.0027268
[41]
Rantala, M.J., Moore, F.R., Skrinda, I., Krama, T., Kivleniece, I., Kecko, S. and Krams, I. (2012) Evidence for the Stress-Linked Immunocompetence Handicap Hypothesis in Humans. Nature Communications, 3, Article No. 694. https://doi.org/10.1038/ncomms1696
[42]
Nitschke, J.P., Sunahara, C.S., Carr, E.W., Winkielman, P., Pruessner, J.C. and Bartz, J.A. (2020) Stressed Connections: Cortisol Levels Following Acute Psychosocial Stress Disrupt Affiliative Mimicry in Humans. Proceedings of the Royal Society B: Biological Sciences, 287, Article ID: 20192941. https://doi.org/10.1098/rspb.2019.2941
[43]
Westbrook, K.E., Nessel, T.A., Hohman, M.H. and Varacallo, M. (2024) Anatomy, Head and Neck: Facial Muscles. StatPearls, Treasure Island. https://www.ncbi.nlm.nih.gov/pubmed/29630261
[44]
Bhattacharjee, S. (2018) Procerus Sign: Mechanism, Clinical Usefulness, and Controversies. Annals of Indian Academy of Neurology, 21, 164-165. https://doi.org/10.4103/aian.AIAN_408_17
[45]
Tong, J., Lopez, M.J. and Patel, B.C. (2024) Anatomy, Head and Neck: Eye Orbicularis Oculi Muscle. StatPearls, Treasure Island. https://www.ncbi.nlm.nih.gov/pubmed/28722936
[46]
Bloom, J., Lopez, M.J. and Rayi, A. (2024) Anatomy, Head and Neck: Eye Levator Labii Superioris Muscle. StatPearls, Treasure Island. https://www.ncbi.nlm.nih.gov/pubmed/31082075
[47]
Sobiesk, J.L. and Munakomi, S. (2024) Anatomy, Head and Neck, Nasal Cavity. StatPearls, Treasure Island. https://www.ncbi.nlm.nih.gov/pubmed/31334952
[48]
Skochdopole, A., Bay, C., Grome, L., Vorstenbosch, J., Yu, J.Z., Winocour, S., Reece, E.M., et al. (2023) Current Surgical Outcomes of Nasal Tip Grafts in Rhinoplasty: A Systematic Review. Plastic & Reconstructive Surgery, 152, 603e-616e. https://doi.org/10.1097/PRS.0000000000010257
[49]
Jain, P. and Rathee, M. (2024) Anatomy, Head and Neck, Orbicularis Oris Muscle. StatPearls, Treasure Island. https://www.ncbi.nlm.nih.gov/pubmed/31424753
[50]
Nguyen, J.D. and Duong, H. (2024) Anatomy, Head and Neck: Face. StatPearls, Treasure Island. https://www.ncbi.nlm.nih.gov/pubmed/31855374
[51]
Germann, A.M. and Al Khalili, Y. (2024) Anatomy, Head and Neck, Risorius Muscle. StatPearls, Treasure Island. https://www.ncbi.nlm.nih.gov/pubmed/31082043
[52]
Rathee, M. and Jain, P. (2024) Anatomy, Head and Neck: Buccinator Muscle. StatPearls, Treasure Island. https://www.ncbi.nlm.nih.gov/pubmed/31536284
[53]
Hoerter, J.E. and Patel, B.C. (2024) Anatomy, Head and Neck, Platysma. StatPearls, Treasure Island. https://www.ncbi.nlm.nih.gov/pubmed/31424878
[54]
Corcoran, N.M. and Goldman, E.M. (2024) Anatomy, Head and Neck, Masseter Muscle. StatPearls, Treasure Island. https://www.ncbi.nlm.nih.gov/pubmed/30969691
[55]
Kim, J., Jeong, H., Cho, J., Pak, C.J., Oh, T.S., Hong, J.P., Yoo, J., et al. (2022) Numerical Approach to Facial Palsy Using a Novel Registration Method with 3D Facial Landmark. Sensors, 22, Article 6636. https://doi.org/10.3390/s22176636
[56]
Reineg: Muscles of the Face, Colorful Anatomy Info Poster. https://stock.adobe.com/ca/images/muscles-of-the-face-colorful-anatomy-info-poster/309366859
