How We're Making Advanced Scan Highlighting Even More Accurate
A major update helps to make AI results even easier to interpret.
Since the launch of Advanced Scan in late 2023, GPTZero has been leading the way when it comes to the interpretability of AI detection scores.
Over recent years, we have expanded our offering to include AI Vocabulary, which highlights phrases that are overused by LLMs, as well as Natural Language Explanations, which explain an AI prediction through higher-level features such as tone and perspective.
Today, we introduce a major update to Advanced Scan that makes highlighting even more accurate in longer documents, and makes highlighting in one section less likely to affect highlighting elsewhere. These improvements are now available to all users.
High-Level Technical Changes
To understand the improvements made to Advanced Scan, it’s important to understand how our detector handles long documents. Our detector processes a document in multiple chunks. This is why scanning a document one paragraph at a time can yield different results than scanning a full document: the context available to the detector is fundamentally different. To derive a final AI score, predictions across chunks are aggregated, which is what is shown as a single score. The figures below illustrate these concepts.
Advanced Scan previously used the final document-level score as the prediction to explain, rather than the per-chunk scores. The result: given a limited compute budget, the impact of a sentence had to be considered with respect to all other sentences in the document, rather than with respect to its immediately surrounding context. In that setup, sentence impact was shallow and essentially drowned out by the myriad other sentences in the document.
We reduce the effect of irrelevant global sentence relationships by moving towards a local, chunk-level analysis for Advanced Scan. Specifically, each chunk of a long document is defined to be several hundred words (tokens to be exact). By considering local sentence interactions constrained within the scope of a chunk, more variations of a sentence (the crux of Advanced Scan) can be analyzed, without leading to combinatorial explosion. This affords much higher fidelity highlighting on longer documents, without significantly increasing the amount of time a scan takes.
Additionally, the impact of sentence editing on other sections of a long document should now be reduced. Previously, editing a sentence in a conclusion paragraph could potentially impact the highlighting in the introduction. This is no longer the case since global sentence interactions are now excluded from analysis. Improvements to Advanced Scan stability for common document editing cases, such as drafting essays with our Chrome Extension in Google Docs, are ongoing, and we strive to make the user experience of such a powerful interpretability tool even greater.
Case Studies of Improved Highlighting
Below are some examples of this improvement on several documents.
Example 1 (Website Claude Example Text)
Example 2 (Mixed Text)
Example 3 (Human Text)
Looking Ahead
The unprecedented level of transparency offered by the new version of Advanced Scan is a step-change in empowering users to understand what aspects of their writing influence AI detection results the most. In other words, this update makes it simpler than ever to see exactly what is behind an AI detection result.
For example, highly confident AI predictions that rely on only a few sentences can be an indication that the result on a given document might not be stable, particularly if the removal of these sentences flips the prediction. By making these patterns more visible, GPTZero continues to drive innovation in the machine learning interpretability space and surpasses other AI detectors as being both the most accurate and most interpretable solution.
