Criticità dei modelli computazionali nell’interpretazione testo-immagine nei meme d’odio: considerazioni translinguistiche, pragmatiche e culturali

English title:
Criticalities of computational models for the interpretation of text-image interplays in hateful memes: translanguaging, pragmatic and cultural considerations

Maria Grazia Sindoni is Professor of English Linguistics and Translation at the University of Messina (Italy). Her main research interests include multimodal critical discourse analysis, translanguaging, language and media ideologies, mainly in digital scenarios. She is currently working on how sociosemiotic and computational conceptualizations of multimodality can be bridged theoretically and empirically. 

Abstract

This presentation will illustrate some unresolved issues with Artificial Intelligence (AI)-driven automatic interpretation of the interplay between text and image in hateful memes. It will show that the meaning of memes is not produced by one semiotic resource (such as language) in isolation, but rather by the mutual text-image interplay, which is also influenced by pragmatic context, culture, and linguistic repertoires of those interpreting the memes’ meaning making. Through a comparison of 1) ‘in vitro’ hate memes purposefully created for the "Hateful Memes Challenge" by the Facebook AI research group (2020) and 2) user-generated memes in English, Italian, Spanish, and German, the current limitations in AI-driven automatic text-image interpretation will be discussed. In the conclusions, it will be argued that multimodal critical discourse studies and sociosemiotics, combined with  translanguaging theories, can help understand (and thus mitigate) the mechanisms of hate production in memes. Although memes appear to be basic texts, they are particularly elusive for the currently used unimodal automatic monitoring systems.