How Do AI Detectors Work?

How Do AI Detectors Work?

AI detectors, also known as AI writing detectors or content detectors, are tools that can identify whether a piece of text was written by a human or an artificial intelligence (AI) solution such as ChatGPT.

Typically, an AI detector will use machine learning algorithms trained on human-written and AI-generated text to identify patterns in natural language and assess whether a piece of text was written by a human or not.

This involves breaking the input text down into segments, giving each sentence a score, and then using these scores to provide an overall percentage score to denote how much of the text it believes is AI-generated.

To assess content, the algorithms use techniques like natural language processing to assess the text’s originality. Two of the most important measures used are perplexity and burstiness.

Perplexity vs. Burstiness

Perplexity is a measurement of how unpredictable or random a piece of text is. large language models (LLMs) and AI content creation tools are designed to create low-perplexity texts, which are more predictable but make more sense to the reader.

AI detectors measure perplexity because low perplexity scores indicate that an automated solution wrote a piece of text, whereas a high perplexity score suggests that it’s been written by a human user with more inconsistencies in language choice.

On the other hand, burstiness is used to assess the variance in sentence structure and length. Texts with limited variation in sentence structure and length are referred to as having low burstiness, while texts with more variation between those two variables have high burstiness.

AI-generated text has less burstiness, producing sentences of an average length with a more consistent structure. In comparison, human writers will use sentences of different lengths, with less overall consistency.

Why are AI Detectors Important?

AI detectors are important tools for helping enterprises, academic institutions, and other entities that need to verify that a piece of text is human-written.

For example, academics can use these tools to help check that students are writing their own essays, while marketers can use them to ensure paid-for content has been written from scratch, and recruiters can deploy them to check that candidates’ applications are genuine.

These tools are becoming increasingly useful now that the adoption of AI has increased. In the education sector alone, Turnitin found that out of 65 million student papers reviewed, 3.3% of all papers (over 2 million) had been flagged as containing 80% or more AI-written text.

Similarly, while it’s difficult to tell the exact prevalence of AI-generated content online, misinformation tracking service NewsGuard has found that 50 online news sites are almost entirely written by artificial intelligence software.

The data points outlined above highlight that AI-generated content is prevalent, and while this isn’t necessarily a bad thing in certain circumstances, there are other areas where this can’t be overlooked, such as education.

Are AI Detectors Reliable?

The reliability of AI detectors depends on a number of factors. While some solution providers like Originality AI and Turnitin claim to have 99% and 98% accuracy when detecting content created by AI, the reliability of a detector can vary based on a number of factors.

One of the biggest issues is that users can instruct solutions like ChatGPT, that’s designed to bypass content scanners. In addition, some individuals will also create mixed text, using AI to create text and then editing it or vice versa in an attempt to avoid detection.

At the same time, many solutions are prone to false positive results and can suggest that human-written articles were actually written by an AI program instead. This can be extremely harmful if an organization or institution takes action against an individual based on this assessment.

As a result, AI detectors are best used as a tool to help highlight content that warrants further investigation. That being said, these solutions are continually improving over time, so there is the potential that their accuracy will increase in the foreseeable future.

What are Some Examples of AI Detectors?

Today, there are a wide range of AI detectors available on the market. GPTZero, Copyleaks, Scribbr, Undetectable AI, and Turnitin all provide solutions for detecting AI-generated content. The overall accuracy of each varies.

These tools are also designed for different use cases. For example, GPTZero is trained to analyze student writing and academic prose to identify content created with ChatGPT or GPT-4.

Other tools like Copyleaks offer a more hybrid content checker approach supporting education and enterprise institutions with native plagiarism detection capabilities.

How Should I Respond to AI-Generated Content?

AI-generated content isn’t necessarily bad in itself, so it’s on each organization and institution to assess whether the use of AI in a certain context (and to a certain extent) indicates misconduct or dishonesty.

For example, if a student uses ChatGPT to write an essay and pass it off as their own work, then this is a sign of misconduct and an intent to mislead the marker.

Ultimately, if a user has an AI detection or plagiarism score of less than 15%, then this is generally considered acceptable for the piece to be considered an original work.