Traditionally only speech communicates emotions via mobile phone. However, in daily communication the sense of touch mediates emotional information during conversation. The present aim was to study if tactile stimulation affects emotional ratings of speech when measured with scales of pleasantness, arousal, approachability, and dominance. In the Experiment 1 participants rated speech-only and speech-tactile stimuli. The tactile signal mimicked the amplitude changes of the speech. In the Experiment 2 the aim was to study whether the way the tactile signal was produced affected the ratings. The tactile signal either mimicked the amplitude changes of the speech sample in question, or the amplitude changes of another speech sample. Also, concurrent static vibration was included. The results showed that the speech-tactile stimuli were rated as more arousing and dominant than the speech-only stimuli. The speech-only stimuli were rated as more approachable than the speech-tactile stimuli, but only in the Experiment 1. Variations in tactile stimulation also affected the ratings. When the tactile stimulation was static vibration the speech-tactile stimuli were rated as more arousing than when the concurrent tactile stimulation was mimicking speech samples. The results suggest that tactile stimulation offers new ways of modulating and enriching the interpretation of speech. 1. Introduction In daily communication we acquire emotion-related information via several senses. Recently the investigation of the utilization of the sense of touch in technology contexts (e.g. cars, tablets, mobile phones, etc.) has been very active. Current mobile phones use routinely direct tactile manipulation for operation. Increasingly phones also utilize tactile feedback. Tactile feedback in mobile phones can be used and is aimed at mediating basically same type of information that is mediated in human-human communication. As mobile phones are still used also for conversations a natural question to come in mind is how speech and tactile information might function together. Human touch system is designed so that via it we can get cognitive, social, and emotional information. From early studies by Harlow [1–3] we learned that tactile information is closely related to the human emotional system. People use the sense of touch when they aim, for instance, to communicate affection or to get someone’s attention in a socially acceptable manner [4]. It has also been shown that people are capable of sending and receiving emotion-related information (e.g., expressions of anger or love) through
References
[1]
H. F. Harlow, “The nature of love,” American Psychologist, vol. 13, no. 2, pp. 673–685, 1958.
[2]
H. F. Harlow, Learning to Love, Albion, San Francisco, Calif, USA, 1971.
[3]
H. F. Harlow, R. O. Dodsworth, and M. K. Harlow, “Total social isolation in monkeys,” Proceedings of the National Academy of Sciences of the United States of America, vol. 54, no. 1, pp. 90–97, 1965.
[4]
S. E. Jones and A. E. Yarbrough, “A naturalistic study of the meanings of touch,” Communication Monographs, vol. 52, pp. 19–56, 1985.
[5]
M. J. Hertenstein, D. Keltner, B. App, B. A. Bulleit, and A. R. Jaskolka, “Touch communicates distinct emotions,” Emotion, vol. 6, no. 3, pp. 528–533, 2006.
[6]
J. Smith and K. MacLean, “Communicating emotion through a haptic link: design space and methodology,” International Journal of Human Computer Studies, vol. 65, no. 4, pp. 376–387, 2007.
[7]
J. N. Bailenson, N. Yee, S. Brave, D. Merget, and D. Koslow, “Virtual interpersonal touch: expressing and recognizing emotions through haptic devices,” Human-Computer Interaction, vol. 22, no. 3, pp. 325–353, 2007.
[8]
M. Ilves and V. Surakka, “Emotions, anthropomorphism of speech synthesis, and psychophysiology,” in Emotions in the Human Voice. Volume III: Culture and Perception, K. Izdebski, Ed., pp. 137–152, Plural Publishing, San Diego, Calif, USA, 2009.
[9]
M. Ilves, V. Surakka, and T. Vanhala, “The effects of emotionally worded synthesized speech on the ratings of emotions and voice quality,” in Proceedings of the Affective Computing and Intelligent Interfaces (ACII ’11), pp. 588–598, October 2011.
[10]
A. Chang, S. O'Modhrain, R. Jacob, E. Gunther, and H. Ishii, “ComTouch: design of a vibrotactile communication device,” in Proceedings of the 4th Conference on Designing Interactive Systems: Processes, Practices, Methods, and Techniques (DIS '02), pp. 312–320, June 2002.
[11]
M. S. Beauchamp, “See me, hear me, touch me: multisensory integration in lateral occipital-temporal cortex,” Current Opinion in Neurobiology, vol. 15, no. 2, pp. 145–153, 2005.
[12]
J. Driver and T. Noesselt, “Multisensory interplay reveals crossmodal influences on “Sensory-Specific” brain regions, neural responses, and judgments,” Neuron, vol. 57, no. 1, pp. 11–23, 2008.
[13]
J. J. Foxe, I. A. Morocz, M. M. Murray, B. A. Higgins, D. C. Javitt, and C. E. Schroeder, “Multisensory auditory-somatosensory interactions in early cortical processing revealed by high-density electrical mapping,” Cognitive Brain Research, vol. 10, no. 1-2, pp. 77–83, 2000.
[14]
J. J. Foxe, G. R. Wylie, A. Martinez et al., “Auditory-somatosensory multisensory processing in auditory association cortex: an fMRI study,” Journal of Neurophysiology, vol. 88, no. 1, pp. 540–543, 2002.
[15]
J. P. Bresciani, M. O. Ernst, K. Drewing, G. Bouyer, V. Maury, and A. Kheddar, “Feeling what you hear: auditory signals can modulate tactile tap perception,” Experimental Brain Research, vol. 162, no. 2, pp. 172–180, 2005.
[16]
T. Ro, J. Hsu, N. E. Yasar, L. Caitlin Elmore, and M. S. Beauchamp, “Sound enhances touch perception,” Experimental Brain Research, vol. 195, no. 1, pp. 135–143, 2009.
[17]
J. Vroomen and M. Keetels, “Perception of intersensory synchrony: a tutorial review,” Attention, Perception, and Psychophysics, vol. 72, no. 4, pp. 871–884, 2010.
[18]
H. Gillmeister and M. Eimer, “Tactile enhancement of auditory detection and perceived loudness,” Brain Research, vol. 1160, no. 30, pp. 58–68, 2007.
[19]
E. C. Wilson, C. M. Reed, and L. D. Braida, “Integration of auditory and vibrotactile stimuli: effects of phase and stimulus-onset asynchrony,” Journal of the Acoustical Society of America, vol. 126, no. 4, pp. 1960–1974, 2009.
[20]
M. Sato, C. Cavé, L. Ménard, and A. Brasseur, “Auditory-tactile speech perception in congenitally blind and sighted adults,” Neuropsychologia, vol. 48, no. 12, pp. 3683–3686, 2010.
[21]
M. M. Bradley, “Measuring emotion: the self-assessment manikin and the semantic differential,” Journal of Behavior Therapy and Experimental Psychiatry, vol. 25, no. 1, pp. 49–59, 1994.
[22]
M. M. Bradley and P. J. Lang, “Affective reactions to acoustic stimuli,” Psychophysiology, vol. 37, no. 2, pp. 204–215, 2000.
[23]
P. J. Lang, M. K. Greenwald, M. M. Bradley, and A. O. Hamm, “Looking at pictures: affective, facial, visceral, and behavioral reactions,” Psychophysiology, vol. 30, no. 3, pp. 261–273, 1993.
[24]
H. Schlosberg, “Three dimensions of emotion,” Psychological Review, vol. 61, no. 2, pp. 81–88, 1954.
[25]
J. Anttonen and V. Surakka, “Emotions and heart rate while sitting on a chair,” in Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '05), pp. 491–499, April 2005.
[26]
R. J. Davidson, “Cerebral asymmetry, emotion and affective style,” in Brain Asymmetry, R. J. Davidson and K. Hugdahl, Eds., pp. 361–387, The MIT Press, Cambridge, Mass, USA, 1995.
[27]
V. Surakka, Contagion and modulation of human emotions [Ph.D. thesis], Department of Psychology, Acta Universitatis Tamperensis, 1998.
[28]
K. Salminen, V. Surakka, J. Lylykangas et al., “Emotional and behavioral responses to haptic stimulation,” in Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '08), pp. 1555–1562, April 2008.
[29]
K. Salminen, V. Surakka, J. Rantala, J. Lylykangas, P. Laitinen, and R. Raisamo, “Emotional responses to haptic stimuli in laboratory versus travelling by bus contexts,” in Proceedings of the Affective Computing and Intelligent Interaction (ACII '09), pp. 1–7, September 2009.
[30]
J. Rantala, R. Raisamo, J. Lylykangas et al., “The role of gesture types and spatial feedback in haptic communication,” IEEE Transactions on Haptics, vol. 4, no. 4, pp. 295–306, 2011.
