Inside Our New ‘Compare with AI Text’ Tool

Perhaps the most challenging aspect of designing the AI detection user experience is effectively communicating detection results. When something is flagged as “AI-written”, the next question is always: why? 

Our detection model outputs a numerical confidence score for each predicted class (mainly 'AI', 'human', or 'mixed'). The reasons "why" text is classified as AI or human are significantly more difficult to uncover.

First, our model is capable of learning complex relationships between phrases in a document, which it uses to distinguish between AI and human-generated text. However, these inner workings do not necessarily map onto easily understood concepts we generally associate with AI writing ("triteness", for example).

Also, AI-generated text and human text are not always dissimilar, nor are they strictly mutually exclusive labels. For example, type the following into ChatGPT: "Recite the first paragraph of the Declaration of Independence," and watch ChatGPT output the unaltered, famous document. Should the founding document of the United States now be classified as AI-generated? 

Making AI Detection Visible 

Despite the inherent difficulty in doing so, it is our responsibility to help users understand the "why" of AI-detection as best we can, and over the last year, we have released a suite of tools to achieve that goal. For example, our AI Vocabulary feature highlights words and phrases that are more likely to appear in LLM-generated text than human text.  Also, we recently added a natural language explanation (NLE) feature to our core detection model. NLE helps describe, in plain English, the similarities the scanned text has with AI-text in our training set.

GPTZero's latest endeavor in making AI-detection more human-understandable is the new feature: Compare with AI Text (known internally as LLM Parrot), a tool that facilitates comparing a user's maybe-AI-text to known AI text retrieved directly from OpenAI's API. Both texts are displayed side-by-side, and similar words or phrases are highlighted. This presents users with a tailored result, highlighting similarities beyond known AI phrases (like "play a significant role in shaping") while taking into account the type of content being analyzed.

"Compare with AI Text" tool for an essay about the Gulf Stream's impact on Europe's climate.

To use "Compare with AI Text", you can either manually compose a prompt or automatically generate a prompt based on your input text by selecting "Generate Automatically". This works by generating a prompt (of reasonable length and specificity) that could have plausibly produced the scanned text. 

For example, if the inputted text is an essay about the construction of the CN tower, then an apt candidate prompt might read "Write a 500-word essay about the construction of the CN tower". That prompt is then sent to ChatGPT, and the response from ChatGPT is rendered as it is downloaded from OpenAI's servers.

Why This Feature?

Before this feature, some users were already trying to do a version of manual comparison: copying their text into ChatGPT and prompting it to generate a similar output before going line-by-line to see overlaps. That process is painstakingly slow and manual, plus it’s easy to miss partial matches. Compare with AI text automates that workflow while also flagging near-matches that would be hard to otherwise spot. 

Once the entire response has been downloaded, similar pairs of text segments are highlighted, and a number is inserted beside the matched segment to denote the relationship. For example, in the screenshotted text, "the eastern coast of the United States" appears in both the explicitly ChatGPT-generated text, and in the inputted text box (which also happens to be generated by ChatGPT in this case).

While the example above contains many exact string (text) matches, we employ a fuzzy-string matching algorithm to compare the OpenAI text to the user's originally inputted text, and so can identify near-similar string pairs as well. The algorithm is a modified version of a Levenshtein distance algorithm that looks for pairs of text segments for which a small number of edits (i.e., word-replacements) would be required to make those segments identical.

Using the feature is simple: initiate an Advanced Scan (or switch to the Advanced Scan view using the right sidebar), click the Compare with AI Text, type a prompt (or generate automatically), and hit "Generate AI Text". After the ChatGPT response finishes, the matches will appear.

What These Matches Actually Mean 

This feature is a big step forward in helping GPTZero users visualize AI-likeness. It exists independently from our core AI-detection model, serving as a separate, external indication of AI usage. That means a document that is detected as "likely AI" by our core model may not necessarily contain matches with ChatGPT (or any other LLMs). 

Conversely, a document classified by GPTZero as "human-written" could contain several matches with an AI document (recall the "Recite the first paragraph of the Declaration of Independence" prompt from above that yields text that is, for all intents and purposes, "human"). Furthermore, LLMs operate probabilistically, generating slightly different documents and match results on every usage. 

That said, empirically, we have noticed AI-written input text generally yields significantly more matches than human text. This article, for example, yielded zero Compare with AI Text matches for the following prompt (generated automatically):

"Write a detailed product description for a new AI-detection tool feature that compares user-submitted text with AI-generated text to identify similarities and enhance user understanding of AI content classification."

By streamlining the comparison between user text and plausible AI text, we expand our holistic interpretability suite, giving users an unprecedented level of transparency and confidence in our AI detection results.