A multimodal analysis of the sequential organization of verbal and nonverbal interaction

The paper addresses the study of the formal features of conversation from both a theoretical and an empirical perspective. In particular, following the framework and terminology of Sacks, Schlegoff and Jefferson (1974), turn-taking and sequencing are designated ‘formal’ features of interaction. Furthermore, the study adopts and fits into the framework of the theoretical modelling of human–computer interaction (HCI), as outlined in Hunyadi (2011). After reviewing the theoretical background of the multimodal analysis of conversation, a communicative act-based pragmatic annotation system will be outlined which is to contribute to a better understanding of the sequential organization of interaction. Finally, a case study on the sequential organization of nonverbal cues for turn regulation as well as their alignment with verbal content will be described.