Detectors for online hate speech can be easily duped by humans, study shows

September 14, 2018, Aalto University
detectors that screen out online hate speech can be easily duped by humans, study shows
How Google Perspective rates a comment otherwise deemed toxic after some inserted typos and a little love. Credit: Aalto University

Hateful text and comments are an ever-increasing problem in online environments, yet addressing the rampant issue relies on being able to identify toxic content. A new study by the Aalto University Secure Systems research group has discovered weaknesses in many machine learning detectors currently used to recognize and keep hate speech at bay.

Many popular social media and online platforms use hate speech detectors that a team of researchers led by Professor N. Asokan have now shown to be brittle and easy to deceive. Bad grammar and awkward spelling—intentional or not—might make toxic social media comments harder for AI detectors to spot.

The team put seven state-of-the-art hate speech detectors to the test. All of them failed.

Modern natural language processing techniques (NLP) can classify based on individual characters, words or sentences. When faced with textual data that differs from that used in their training, they begin to fumble.

"We inserted typos, changed word boundaries or added neutral words to the original . Removing spaces between words was the most powerful attack, and a combination of these methods was effective even against Google's comment-ranking system Perspective," says Tommi Gröndahl, doctoral student at Aalto University.

Google Perspective ranks the 'toxicity' of comments using text analysis methods. In 2017, researchers from the University of Washington showed that Google Perspective can be fooled by introducing simple typos. Gröndahl and his colleagues have now found that Perspective has since become resilient to simple typos yet can still be fooled by other modifications such as removing spaces or adding innocuous words like 'love."

Here’s how Google Perspective toxicity rating reacts to typos and a little love thrown in an otherwise hateful sentence. Credit: Aalto University

A sentence like "I hate you" slipped through the sieve and became non-hateful when modified into "Ihateyou love."

The researchers note that in different contexts the same utterance can be regarded either as hateful or merely offensive. Hate is subjective and context-specific, which renders techniques insufficient as stand-alone solutions.

The researchers recommend that more attention be paid to the quality of data sets used to train machine learning models—rather than refining the model design. The results indicate that character-based detection could be a viable way to improve current applications.

The study was carried out in collaboration with researchers from University of Padua in Italy. The results will be presented at the ACM AISec workshop in October.

The study is part of an ongoing project called "Deception Detection via Text Analysis in the Secure Systems" at Aalto University.

Explore further: Hate speech from women is judged harsher than that from men

More information: All You Need is "Love": Evading Hate-speech Detection. arxiv.org/abs/1808.09115

Related Stories

Hate speech from women is judged harsher than that from men

July 31, 2018

Women who make hateful remarks on social media are likely to be judged more severely than men who make the same comments. This is also true for reactions to hate speech (counter speech) which when made by women are less accepted ...

Using multi-task learning for low-latency speech translation

August 13, 2018

Researchers from the Karlsruhe Institute of Technology (KIT), in Germany, have recently applied multi-task machine learning to low-latency neural speech translation. Their study, which was pre-published on ArXiv, addresses ...

Recommended for you

Team breaks world record for fast, accurate AI training

November 7, 2018

Researchers at Hong Kong Baptist University (HKBU) have partnered with a team from Tencent Machine Learning to create a new technique for training artificial intelligence (AI) machines faster than ever before while maintaining ...

0 comments

Please sign in to add a comment. Registration is free, and takes less than a minute. Read more

Click here to reset your password.
Sign in to get notified via email when new comments are made.