Development of iconic gesture-speech integration in preschool children (EcoGest) (Completed)

2017–2021

Empirical and computational study of iconic gesture-speech integration and its development in preschool children

This project funded by the DFG (German Research Foundation) aims to provide a detailed account of the development of iconic gesturing and its integration with speech in different communicative genres. We will study pre-school children at 4 to 5 years of age to investigate their speech-accompanying iconic gesture use and to develop a computational cognitive model of their development. We apply qualitative and quantitative methods to study children's speech-gesture behavior and to evaluate our findings with computational cognitive modeling in terms of the following aspects: (1) forms of iconic gesturing, (2) positions of gestures, (3) semantic coordination of speech and gesture, (4) packaging of information.

Project Description

Gesture and speech form an integrated communicative system. This integration has been investigated for the purpose of communication as well as learning. Recent research attempting to explain and model this integration focus on how gesture influences speech, and how gesture — as a special form of action — forces the speaker to incorporate sensorimotor information into the representation. In these cognitive approaches, the fact that gesture is employed purposefully in various communicative genres and thus interacts with speech for different discursive purposes is disregarded. Yet types of communicative genres have been shown to bear different cognitive and communicative requirements on the speaker and hearer: For example, a narration requires different structural knowledge and verbal skills from a speaker than an explanation.

While for competent speakers, the appearance of gestures in different communicative genres might occur fluent, for children learning to cope with the variety of discursive contexts, the role of gesture for thinking and communicating might be better observable: One possibility is that for particular genres, gestures are required resources for depicting strategies facilitating mutual understanding, and children discover them with increasing experience in interaction and with growing cognitive and linguistic competencies. Another — not exclusive — possibility is that gesture plays an important role on the way to master increasingly changing communicative requirements, which becomes visible when children learn to produce different communicative genres.

Thus, in addition to sensorimotor information that is recruited for the act of speaking, gesture might be an integral resource of speaking in relation to different communicative purposes defined by genre. The goal of this project is to provide a detailed account of the development of iconic gesturing and its integration with speech, following three main research questions:

How does iconic gesture use vary in different communicative genres?
What patterns of gestural iconic forms can be identified and how do they interact with semantic content packaged in speech and gesture?
How do these forms change across preschool age (4 to 5 years)?

In association with: XPRAG.de

Stimuli of our Study: Ecogest_Stimuli.pdf

Team

EcoGest Team

Principal Investigators

Scientific Assistants

Student Assistants

Claudia Jeronski
Moritz Pottkämper
Bettina Wagner

News

Poster presentation at the GESPIN 2020, Stockholm, Sweden
Poster presentation at the EuroCogSci-2019, Bochum, Germany
Poster and oral presentations at the GESPIN-2019, Paderborn, Germany
Poster presentation at the LingCologne2019, Cologne, Germany
Talk at the The 12th International Symposium on Iconicity in Language and Literature, Lund, Sweden
Poster presentation at the KogWis 2018, Darmstadt
Recent press report about EcoGest with the title "Children's Gestures, and What They Mean" was published on 12.04.2018 at the Uni.news magazine.
EcoGest KickOff-Meeting was held on 12.2.-13.2.2018 at CITEC, Bielefeld University.

Publications

Abramov O., Kern F., Koutalidis S., Mertens U., Rohlfing K. and Kopp S. (2021). The Relation Between Cognitive Abilities and the Distribution of Semantic Features Across Speech and Gesture in 4-year-olds. Cognitive Science Journal, Vol. 45, Issue 7. DOI
Mertens U. and Rohlfing K. (2021). Progressive Reduction of Iconic Gestures Contributes to School-Aged Children's Increased Word Production. Frontiers in Psychology, 12, 651725. DOI
Koutalidis S., Mertens U., Abramov O., Kopp S., Rohlfing K., Kern F. (2020). Multimodal marking of focus in German preschoolers' utterances with the focus-particles also, only, and even/still. In Proc. of the 7th GESPIN — Gesture and Speech in Interaction Conference. Stockholm: KTH Royal Institute of Technology.
Crawshaw, C. E., Kern, F., Mertens, U., & Rohlfing, K. J. (2020). Children's Narrative Elaboration After Reading a Storybook Versus Viewing a Video. Frontiers in Psychology, 11. DOI
Mertens, U., Abramov, O., Kern, F., Kopp, S., Nemeth, A., & Rohlfing, K. J. (2019). Children's viewpoint: Iconic co-speech gestures and their relation to linguistic structure across two communicative genres. Proceedings of the 6th Gesture and Speech in Interaction Conference, 62–67. DOI
Mertens, U., Abramov, O., Nemeth, A., Kern, F., Kopp, S., & Rohlfing, K. J. (2019). Children's ability of spatial thinking is related to observer viewpoints in iconic co-speech gestures. Proceedings of the 27th European Symposium on Artificial Neural Networks, Computational Intelligence and Machine Learning (ESANN).
Abramov O., Mertens U., Németh A., Kern F., Rohlfing K., Kopp S. (2019). The relation between individual differences in speech-gesture behavior of 4-year-olds across three different experimental tasks. EuroCogSci-2019, Bochum, Germany.
Abramov O., Mertens U., Németh A., Kern F., Rohlfing K., Kopp S. (2019). Semantic coordination of speech and gesture in young children. GESPIN-2019, Paderborn, Germany.
Mertens U., Kern F., Kopp S., Abramov O., Németh A. and Rohlfing K. (2019). Children's viewpoint in gesture and their relation to linguistic structure. In Proc.: LingCologne-2019, Cologne.
Abramov O., Kopp S., Kern F., Rohlfing K., Mertens U. and Németh A. (2019). Iconicity in child reports: Does gesture-speech integration of preschoolers really differ from adults? In Proc.: ILL-12, Lund, Sweden.
Abramov O., Kopp S., Németh A., Kern F., Mertens U. and Rohlfing K. (2018). Towards a computational model of child gesture-speech production: How information is spread across modalities in pre-school children. KogWis 2018, Darmstadt.
Kern, F. (2018). How hand movements gain meaning in embodied interaction. In: D. Favareau (eds.): Co-operative Engagements in Intertwined Semiosis. Essays in the Honour of Charles Goodwin. Tarku University.
Kern, F. & K. Rohlfing (2016). Children's gestures in two decontextualized speech contexts. Paper presented at the ISGS, Paris, July, 18–22.
Bergmann, K., Kahl, S. & Kopp, S. (2013). Modeling the semantic coordination of speech and gesture under cognitive and linguistic constraints. In R. Aylett, B. Krenn, C. Pelachaud, H. Shimodaira (Eds.), IVA 2013.
Kern, F. & Rohlfing, K. J. (2015). Contextual relations of iconic gestures. Paper presented at the 37. Jahrestagung der DGfS, Leipzig, 4.–6. März.
Lücking, A., Bergmann, K., Hahn, F., Kopp, S. & Rieser, H. (2013). Data-based analysis of speech and gesture: the Bielefeld Speech and Gesture Alignment corpus (SaGA) and its applications. Journal on Multimodal User Interfaces.
Kern, F. (2009). Zum Gebrauch von Gestik und Stimme in kindlichen Erzählungen. Habilitationsvortrag, Potsdam.
Rohlfing, K. J. & Kopp, S. (2007). Meaning in the timing? The emergence of pointing patterns. In Proceedings of the Symposium on Language and Robots (pp. 79–82). Aveiro, Portugal.

Previous project

Next project