KInIT researchers have introduced MultiSocial, a benchmark dataset designed to evaluate the ability of detection methods to identify AI-generated texts in the social media environment – where messages are shorter, more informal, and linguistically diverse.
While previous studies have focused mostly on English and long-form content like essays or news articles, MultiSocial fills a key research gap. It includes over 470,000 texts in 22 languages across 5 social media platforms (e.g., Twitter, Telegram, WhatsApp, Gab, and Discord), combining human-written posts with content generated by 7 multilingual large language models (LLMs).
In total, 17 detection methods were benchmarked, both in zero-shot and fine-tuned settings. The results show that fine-tuned detectors can successfully adapt to social media data and consistently outperform zero-shot models. The dataset also highlights how the choice of platform and language used for training affects detection performance. For instance, models fine-tuned on Telegram data showed the best cross-platform generalisation, while Discord-trained detectors performed well on their native platform but struggled to transfer.
The research further found that larger foundational models tend to perform better, but even smaller models – such as mDeBERTa – can outperform large zero-shot detectors when fine-tuned properly.
MultiSocial enables realistic, large-scale testing of detection tools and supports ongoing efforts to build more reliable and multilingual safeguards against AI-generated information manipulation online.
Link to Zenodo: https://zenodo.org/records/16792911