Not logged in.

Contribution Details

Type Journal Article
Scope Discipline-based scholarship
Title Gesture retrieval and its application to the study of multimodal communication
Organization Unit
Authors
  • Mahnaz Parian-Scherb
  • Peter Uhrig
  • Luca Rossetto
  • Stephane Dupont
  • Heiko Schuldt
Item Subtype Original Work
Refereed Yes
Status Published electronically before print/final form (Epub ahead of print)
Language
  • English
Journal Title International journal on digital libraries
Publisher Springer
Geographical Reach international
ISSN 1432-5012
Page Range Epub ahead of print
Date 2023
Abstract Text Comprehending communication is dependent on analyzing the different modalities of conversation, including audio, visual, and others. This is a natural process for humans, but in digital libraries, where preservation and dissemination of digital information are crucial, it is a complex task. A rich conversational model, encompassing all modalities and their co-occurrences, is required to effectively analyze and interact with digital information. Currently, the analysis of co-speech gestures in videos is done through manual annotation by linguistic experts based on textual searches. However, this approach is limited and does not fully utilize the visual modality of gestures. This paper proposes a visual gesture retrieval method using a deep learning architecture to extend current research in this area. The method is based on body keypoints and uses an attention mechanism to focus on specific groups. Experiments were conducted on a subset of the NewsScape dataset, which presents challenges such as multiple people, camera perspective changes, and occlusions. A user study was conducted to assess the usability of the results, establishing a baseline for future gesture retrieval methods in real-world video collections. The results of the experiment demonstrate the high potential of the proposed method in multimodal communication research and highlight the significance of visual gesture retrieval in enhancing interaction with video content. The integration of visual similarity search for gestures in the open-source multimedia retrieval stack, vitrivr, can greatly contribute to the field of computational linguistics. This research advances the understanding of the role of the visual modality in co-speech gestures and highlights the need for further development in this area.
Free access at DOI
Digital Object Identifier 10.1007/s00799-023-00367-0
Other Identification Number merlin-id:23893
PDF File Download from ZORA
Export BibTeX
EP3 XML (ZORA)