In an era where digital communication plays an increasingly vital role, content moderators have become key figures in ensuring the quality and appropriateness of online discussions on platforms. Researchers from the University of Vienna explored the perspectives of these content moderators in their study. Between June and September 2022, a total of 18 interviews were conducted with individuals from Austria and Germany. During these interviews, participants provided insights into the comments they "moderate" and the extent to which technological tools, including Artificial Intelligence (AI), are employed in their work. The findings suggest the "significant role content moderators play in shaping public discourse", as highlighted by study author Andrea Stockinger.

Anyone who has not personally encountered online hate speech has, at the very least, heard of it. Communication researchers are observing how online discussions are increasingly being conducted in an uncivil manner. To counteract this development effectively and create an environment that promotes a civil culture of discussion, there is a growing reliance on content moderation techniques. These include the activity of professional content moderators who screen forums and social media platforms.

Their tasks involve reviewing comments, deleting problematic ones, and intervening in hostile discussions to facilitate an unhindered flow of conversation. Laws defining restrictions on free speech and community guidelines set by companies help them distinguish between civilized and uncivilized online speech. The sheer volume of user comments presents a significant challenge for content moderators to keep up with their responsibilities. Consequently, many of them employ semi-automated systems (e.g., based on AI) in the form of word filters, blocklists, and other tools, such as detecting pornographic content or copyrighted material. While these technologies assist in this extensive work, they have limited capacity to comprehend contexts, human cultures, and power dynamics, so at this point in time, they are less effective in responding to subtle hate comments.

In conversation with content moderators

Previous research focuses predominantly on the learning capabilities of AI, rather than on content moderators who bear the responsibility for determining which content and comments remain on the internet. To address this gap, Andrea Stockinger, Svenja Schäfer, and Sophie Lecheler from the University of Vienna conducted a research project involving interviews with 18 content moderators from Austria and Germany. Both countries have similar legal regulations governing comment sections on online platforms. A portion of the sample was employed as social media managers, while the other part managed the comment forums for their respective organizations. They worked for news organizations or political parties.

The interviews were conducted via video call between June and September 2022. The participants were asked about their professional profile as content moderators, about moderation strategies of their organization (e.g., community guidelines), as well as about their use of technological tools and AI when reviewing comments.

Clear cases vs. gray areas

In both countries, content moderators are increasingly dealing with problematic comments. "While some comments clearly violate laws or community guidelines and are therefore deleted (some automatically), the majority of comments fall into a gray area that is not clearly regulated. This is where automated solutions reach their limits as well", reports study author Andrea Stockinger. These gray-area comments range from satire, sarcasm, and memes to creative paraphrases of insults. In such cases, moderators often rely on their intuition to determine whether a gray area comment negatively impacts a discussion, considering the company's values, the target audience, and their personal sensibilities as criteria for moderation.

How are problematic comments being addressed?

Building upon the findings, Stockinger explains: "The way this gray area is moderated varies significantly among different organizations, platforms, and even individual moderators." The majority use word filters, blocking lists, and AI tools frequently to proactively identify aggressive and inappropriate comments. In cases that are less clear-cut, they often resort to asking questions to grasp the context of the comment. When users make false statements, some moderators attempt to correct them, while others opt to remove these comments entirely. The approach to interacting with users also differs: some moderators consider dialogue with users important, while others remain distant. They expressed a desire for enhanced technological tools to perform their duties more effectively, particularly when dealing with clear-cut, problematic cases. Nonetheless, they remain skeptical about AI decisions in gray area cases.

Artificial Intelligence as a helpful addition – but far from a replacement

The extent to which AI is used for comment screening varies across organization sizes, moderator workloads, and even platforms. For example, hate speech generates much more traffic on Facebook compared to Instagram. On a personal level, many moderators see themselves as spokespeople for their organization and are responsible for engaging with their community. It is difficult for an AI to do this work authentically.

"The results of the study highlight the crucial role of content moderators in shaping public discourse", summarizes Stockinger, "they emphasize the need for clearer and more nuanced regulations and provide practical implications for (semi-)automated content moderation strategies."