A Statistical Defence for AI Detection

Technology:

In January, we released GPTZero’s first AI detection model publicly for everyone. The demand was deafening — with seven million views and half million users in the first week, GPTZero was called Hero of the Week on UK radio, internationally covered, in Japan, France, Australia and over thirty countries, and even landed on the front page of the New York Times.

The thesis was simple — build a model that is efficient and effective, and make it accessible to every person who needs it. To do so, the original GPTZero model applied a ‘statistical approach’, leveraging academic research in natural language processing to convert written words to numbers for analysis.

Today, the first principles from GPTZero’s original detection model is still being applied widely. These methods are efficient — leveraging numerical analysis instead of intensive deep text analysis, they are the least computationally expensive of AI detection methods; additionally, they are actually the main applications behind dozens of other AI detector apps including ZeroGPT, Copyleaks, Originality, and Writer AI. They are still effective — remaining as one of the core ‘nodes’ of the modern GPTZero detection model, alongside our novel text search and deep learning detection methods.

The statistical layer of GPTZero’s AI detection model is based on a combination of two statistical indicators — perplexity and burstiness. Together, they form the first layer behind GPTZero’s AI detection.

Image 1: Edward explaining 'Perplexity' and 'Burstiness' to Anderson Cooper

What is Perplexity and Burstiness?

You can interpret the perplexity as a measure of the 'entropy' in a sentence, equivalent to a measure of how likely an AI model would have chosen the exact same set of words as found in the document. We’ve trained our model to identify when the input text looks very similar to something written by an AI model. For example, the sentence, “Hi there, I am an AI _” would most likely be continued by an AI model with the word “assistant”, which would have low perplexity. On the other hand, if the next word that followed was “potato”, then that sentence would have much higher perplexity, and also a greater likelihood of being written by a human. Over the course of hundreds of words, these probabilities compound to give us a clear picture of the origin of this document. There isn’t an absolute scale for perplexity, but generally, a perplexity above 85 is more likely than not from a human source.

Burstiness, on the other hand, is a measure of how much writing patterns and text perplexities vary over the entire document. As humans, we have a tendency to vary our writing patterns. Philosophically, our short-term memory activates, and dissuades us from writing same (or similar) things twice. Conversely, language models have a significant ‘AI-print’ where they write with a very consistent level of AI-likeness.

While a person could easily write an AI-like sentence by accident, people tend to vary their sentence construction and diction a lot throughout a document. On the other hand, models formulaically use the same rule to choose the next word in the sentence, leading to low burstiness.

Compared to other statistical methods for AI detection, burstiness is a unique to the GPTZero detector, allowing our model to evaluate long-term-context and perform better with additional text.