Why AI detectors disagree, and what they actually measure

No detector identifies an author. Each one measures something that tends to go with machine writing.

The first two families respond to fluent, low-surprise prose sustained over many sentences. They don’t look for a word list or for em dashes. That’s why one edited sentence rarely changes a verdict, and why an AI rewrite of a whole human paragraph often does.

Detectors need enough text to separate a pattern from chance, and the vendors say so in their own documentation:

• Pangram’s model card sets a minimum input of 50 words.
• Turnitin asks for “at least 300 words of prose text in a long-form writing format”.
• Copyleaks requires at least 350 characters, about 60 words, and suggests testing around 350 words.
• Watermark checks need length too. The original watermark paper (Kirchenbauer et al.) ran its main tests on 200-token passages, roughly 150 words.

An early study found that both people and automatic detectors get better as excerpts get longer, and that even multi-sentence excerpts fooled expert raters over 30% of the time (Ippolito et al., ACL 2020). A short social post or a product description can fall below most of these floors.

The best-known result is Liang et al. (2023): seven detectors flagged essays by non-native English writers as AI at an average rate of 61.22%. That bias belonged mainly to perplexity-based tools, which mistake plain vocabulary for machine text. Retrained commercial classifiers report much lower rates on these groups, though those figures are mostly the vendors’ own.

Other patterns hold across studies:

• Formulaic and concise writing is flagged more, because it’s predictable. Constrained genres such as recipes, legal boilerplate and product specs are hard cases.
• Light AI polish on human writing gets flagged. In APT-Eval (Saha and Feizi, 2025), “extremely minor” polishing by the small Llama-2-7B model got 32.31% of human texts flagged by ZeroGPT, 42.56% by Pangram and 64.71% by GPTZero, all early-2025 versions.
• Base rates multiply small errors. When Vanderbilt disabled Turnitin’s AI detector in August 2023, it noted that a 1% false-positive rate across its 75,000 yearly submissions would mean about 750 papers wrongly flagged.

The strongest current classifiers are much better than the 2023 tools. OpenAI withdrew its own classifier in July 2023 after it caught only 26% of AI text. In an Epoch AI test reported in July 2026, Pangram, GPTZero and Originality.ai missed at most 0.7% of plainly prompted AI text, and Pangram and GPTZero flagged none of 495 human passages.

The same test shows the limit. When models were given five samples of an author’s writing and asked to imitate it, Pangram missed 10%, GPTZero 11% and Originality.ai 18%; in scientific writing the misses rose to 24% to 29%. Going further, fine-tuning a model on an author’s complete works cut detection from 97% to 3% in a 2025 study by Chakrabarty and colleagues. And readers who use AI heavily themselves did better than most tools: a majority vote of five such readers misclassified 1 of 300 articles in Russell, Karpinska and Iyyer (ACL 2025).

Detection of plain, unedited assistant output is close to solved. Detection of writing as people actually produce it, edited, mixed and imitated, isn’t.

• Our two-detector example is one 197-word sample on one day. We’ve left both tools unnamed because one result was read from an API score whose scale we had to infer, and a single sample says nothing about either tool’s general accuracy.
• Most vendor accuracy figures are measured by the vendors on data they chose. Independent tests of the newest production models are scarce.
• Detectors and models both change every few months. Dates matter on every number here.

Pick five pieces a person on your team wrote before 2022, each over 300 words, and run them through the detector you use. Write down the scores. That’s your false-positive baseline for that tool on your kind of writing, and it’s the number to remember the next time a score looks alarming.