AI Detection

GPTZero records 0 false positives on essays by ESL students or students with disabilities

Being falsely accused of using AI can be damaging, especially for students developing their writing skills. GPTZero 5.0 achieved 0 FP across the 10,402 student essays in PERSUADE 2.0 and multiple learner corpora.

Anna Gao

4 min read
GPTZero records 0 false positives on essays by ESL students or students with disabilities

Early research and social media posts have claimed that AI detectors are biased against English language learners. Educators have also wondered if detectors incorrectly flag students with disabilities, as developing writers tend to use simpler vocabulary and follow more predictable patterns.

These concerns are reasonable, and the cost of a false positive can be high. For students working hard to learn writing in English, being wrongly accused of using AI can dampen their enthusiasm and confidence.

At GPTZero, we've put a lot of effort into eliminating bias against developing writers. GPTZero 5.0 is the latest result of that work, so we tested it on a wider range of student writing.

The Data

We evaluated GPTZero 5.0b on multiple sets of human-written English text.

PERSUADE 2.0. PERSUADE 2.0 is a collection of argumentative essays by US students in grades 6 through 12, written for state and national writing assessments between 2010 and 2020, with demographic information. We evaluated every group the dataset reports: English language learner status, disability status, economic disadvantage, gender, race/ethnicity, and grade level. All essays came from PERSUADE 2.0's official held-out test split, which GPTZero was not trained on.

Learner corpora, following Turnitin's protocol. In 2023, Turnitin published a public evaluation of its AI Writing Indicator on human-written texts from three datasets (ASAP, ICNALE and PELIC) on documents of at least 300 words, broken down by L1 writers (English as a first language) and L2 writers (English as an additional language). GPTZero did the same, but on a wider corpus. We ran GPTZero 5.0 on ASAP 2.0, ICNALE, and PELIC, and two extra learner corpora: ELLIPSE and CELL, using all 300+ word texts in each dataset.

ChatGPT-Detector-Bias Dataset. Liang et all released a benchmark from non-native English writers; it includes 91 TOEFL essays, 70 college admission essays, and 88 essays written by U.S. 8th-grade students. We ran GPTZero 5.0 on all of these essays.

Transparent Evaluation Results

PERSUADE 2.0: 0 false positives in every group

Across all 10,402 essays, GPTZero 5.0 classified every text as human-written. This includes the 914 essays by English language learners, the 1,172 by students with disabilities, and every other group in the dataset.

Students with disabilities are those with an Individualized Education Program (IEP) or a Section 504 plan, which support students with learning disabilities or health conditions. Essays with a missing attribute are labeled Unknown, and groups with fewer than 100 essays are omitted from the breakdown.

Comparison with Turnitin and Pangram

Turnitin and Pangram have each published false positive rates for L1 and L2 writers on ASAP, ICNALE, and PELIC texts of 300+ words. Neither publishes per-dataset breakdowns, so we compare aggregate rates.

Following Turnitin's public evaluation protocol, we evaluate human-written English texts of at least 300 words and report aggregate False Positive Rates for L1 and L2 writers. L1 writers use English as their first language; L2 writers use English as an additional language. GPTZero results aggregate evaluation results from ASAP 2.0, ICNALE, and PELIC. Published competitor results come from Pangram's blog comparison and Turnitin's public evaluation; both publish aggregate rates without per-dataset breakdowns.

GPTZero 5.0 achieved 0 FP of 5,527 L2 texts and 0 of 4,097 L1 texts. Turnitin reports 1.4% for L2 writers and 1.3% for L1 writers, and Pangram reports 0.02% and 0.00%. 

More Learner Corpora Breakdown

As we ran GPTZero 5.0 on more learner corpora outside the Turnitin comparison, a more detailed breakdown per dataset is necessary for transparency and to make GPTZero’s perfect result on these datasets clearer. 

The full breakdown here of learner corpora reports every GPTZero cohort of at least 300 words from ASAP 2.0, ICNALE, PELIC, CELL, and ELLIPSE. GPTZero 5.0 achieved zero false positives across all datasets, for both L1 and L2 writer groups, which is a strong indicator towards eliminating bias. 

L1 writers use English as their first language, while L2 writers use English as an additional language. Unknown status means the dataset does not specify either group.

ChatGPT-Detector-Bias : 0 False Positives on all essays

Liang et al. (2023) concluded that commercial AI detectors exhibit bias against non-native English writers. However, this is not true for GPTZero. We evaluated GPTZero 5.0 on all of the 3 student essay datasets in this benchmark and achieved 0 false positives on all essays, regardless of writer groups. 

What This Means For Educators

Fairness is a baseline that an AI detector should meet before it enters the classroom. It has guided GPTZero from inception, and GPTZero is dedicated to building a de-biased AI detector. GPTZero 5.0 is a great milestone; however, it is not a finish line. We will continue testing for potential biases in writing from the students who are at the highest risk of being misjudged, keep sharing what we find, and keep raising the bar of fairness and accuracy with every new model release.