How Do AI Detectors Work? The Science Behind Perplexity & Bias [2026]

By Imran Khan (AI Tech Safar)

In July 2023, someone ran the United States Constitution through an AI detector. It came back flagged as machine-written. Not a modern AI-assisted essay, not a ChatGPT draft — the actual 1787 Constitution, written by men in wigs with quill pens. That single, slightly absurd result tells you more about how AI detectors actually work than most explainer articles do: they aren't reading for meaning. They're measuring how *predictable* your word choices are, and it turns out 18th-century legal drafting is, statistically, pretty predictable.

If you've ever had a detector flag something you wrote yourself, or you're trying to figure out whether these tools are worth trusting at all, here's what's actually happening under the hood — and where it breaks down.

Table of Contents

• Quick Summary & Key Takeaways
• The Two Core Signals: Perplexity and Burstiness
• Why Detectors Get It Wrong — Often
• Watermarking: The Other Approach
• How the Major Tools Actually Compare
• Frequently Asked Questions (FAQs)

Quick Summary & Key Takeaways

• Detectors don't "recognize" AI text — they measure statistical patterns like word predictability (perplexity) and sentence-length variation (burstiness), then guess based on how human-like those patterns look.
• Low predictability = more likely flagged human; high predictability = more likely flagged AI — which is why the famously formulaic U.S. Constitution got flagged as AI-generated in 2023.
• Non-native English writers get falsely flagged at dramatically higher rates. A Stanford study found detectors misclassified 61.22% of TOEFL essays — all genuinely human-written — as AI-generated, versus near-zero false positives on native-speaker essays.
• A newer detection approach, watermarking, works completely differently — it doesn't guess from patterns at all, it looks for an invisible statistical signature deliberately built into the AI's output.
• No detector is reliable enough to be the only evidence in a real accusation — every major tool, including ones claiming under 1% false positive rates, has documented cases of flagging genuine human writing.

The Two Core Signals: Perplexity and Burstiness

Most AI detectors — GPTZero, the original Turnitin model, Originality.ai's earlier versions — are built around two statistical measurements, not comprehension.

Perplexity measures how "surprised" a language model would be by your word choices. Language models are trained on massive amounts of human text, so they build up a sense of which words commonly follow which other words. When AI generates text, it tends to pick the statistically likely next word most of the time — that's literally how the underlying technology works. Human writing is messier. We use unusual phrasing, personal quirks, and the occasional weird word choice that a model would rarely predict. Low perplexity (predictable, "smooth" text) reads as more AI-like to a detector. High perplexity (surprising, varied text) reads as more human.

Burstiness measures variation in sentence length and structure across a piece of writing. Human writers naturally alternate — a short punchy sentence, then a longer one that winds through a few clauses, then another short one. AI-generated text has historically tended toward more uniform sentence lengths and rhythms. High burstiness (lots of variation) reads as human; low burstiness (consistent, even rhythm) reads as AI.

Put together, a detector is essentially asking two questions: is this suspiciously predictable, and is this suspiciously uniform? If both answers are yes, it flags the text.

Why Detectors Get It Wrong — Often

This is where the method's core weakness shows up, and it's not a minor edge case. A team of Stanford researchers led by James Zou tested seven widely-used AI detectors against 91 real TOEFL essays — English-proficiency test essays written entirely by non-native English speakers, with zero AI involvement. The detectors flagged 61.22% of them as AI-generated. Eighteen of the ninety-one essays were unanimously flagged as AI by all seven detectors. Eighty-nine of the ninety-one were flagged by at least one.

The mechanism explains why: non-native speakers, especially those writing carefully for a proficiency exam, tend to use more standard grammar and more common vocabulary — precisely the low-perplexity pattern detectors associate with AI. Native English speakers in the same study were flagged at a much lower rate, often under 10%. It's the same reason the Constitution got flagged — formal, formulaic writing scores as "predictable" whether a human or a machine produced it.

This isn't a solved problem, but it also isn't a frozen one. Newer detectors have taken very different approaches to it. Turnitin's own later data claimed a sub-1% false positive rate on larger, longer-text datasets — though that specific detector wasn't part of the original Stanford test. GPTZero shipped an updated model afterward and, per third-party testing, still showed roughly 7.7% false positives on the same TOEFL benchmark. A newer entrant called Pangram reported a 0% false positive rate on that identical TOEFL set, along with near-zero rates across two additional ESL essay datasets — a notably different result from the tools the Stanford study covered. The pattern across the industry since 2023 has been consistent: detectors that specifically re-trained to correct for this bias closed most of the gap; detectors that didn't kept producing it.

💡 AI Tech Safar Insight
The detail that gets lost in most coverage of this issue is that "AI detector accuracy" isn't one number — it's at least two very different numbers depending on who's being tested. A detector can genuinely advertise under 1% false positives on native-English, moderate-length text and still misclassify second-language writers at ten times that rate, because the two populations produce statistically different prose. If you're evaluating a detector — for a classroom policy, a publishing workflow, or your own writing — the number that matters isn't the marketing headline accuracy rate. It's whether that rate was tested against writing that looks like the writing you're actually going to run through it.

Watermarking: The Other Approach

Perplexity-based detection guesses after the fact, using statistics. Watermarking works differently: it's built into the AI model itself at the moment text is generated. Google's SynthID, for example, subtly biases which "equally good" word choices the model makes at each step, creating a statistical pattern that's invisible to a human reader but detectable by a matching algorithm — provided you have access to check for it. In theory, watermarking sidesteps the whole non-native-speaker bias problem entirely, because it's not judging your writing style at all; it's checking for a signature that only exists if a specific AI system generated the text.

The catch is adoption and durability. Watermarking only works if the AI provider actually implements it, if the output isn't edited enough to break the pattern, and if whoever wants to check for it has access to the matching detection system — which isn't always public. It's a promising direction, but as of 2026 it's nowhere near a universal standard across every AI tool people use to generate text.

How the Major Tools Actually Compare

Tool	Core Method	Known ESL False-Positive Behavior
Original 7 detectors (2023 Stanford test)	Perplexity / burstiness	61.22% average false positive rate on TOEFL essays
GPTZero (post-update)	Perplexity / burstiness, retrained	~7.7% on the same TOEFL benchmark, per third-party testing
Turnitin	Proprietary classifier	Claimed under 1% on internal large-scale testing; not part of the original Stanford study
Pangram	Classifier trained to correct for ESL bias	0% reported on the same TOEFL set and two additional ESL datasets
Watermark-based (e.g. SynthID)	Embedded generation signature	Not style-dependent, but requires provider adoption and detector access

Frequently Asked Questions (FAQs)

Q1: Can AI detectors be trusted for academic or professional accusations?
Not on their own. Given documented false-positive rates as high as 61% for specific populations of genuinely human writers, most universities and legal guidance now recommend detector results be treated as one input for a conversation, not standalone proof of misconduct.

Q2: Why did the U.S. Constitution get flagged as AI-generated?
Because perplexity-based detectors flag statistically predictable, formulaic writing — regardless of when it was written or by whom. Formal legal and historical documents often score as "predictable" in the same way AI text does.

Q3: Do AI detectors get better over time?
Some do, meaningfully. Detectors that specifically retrained on non-native English datasets after the 2023 Stanford findings — like the update behind Pangram's reported results — showed dramatic improvements. Others updated their marketing more than their models.

Q4: What's the difference between perplexity-based detection and watermarking?
Perplexity-based detection guesses after the fact by analyzing statistical patterns in the text. Watermarking is built into the AI model at generation time, embedding a detectable signature regardless of writing style — but only if the specific AI tool used supports it.

Q5: Are AI detectors more likely to flag certain kinds of human writing?
Yes. Beyond non-native English speakers, research and detector vendors have separately noted elevated false-positive patterns in heavily-edited writing, formal business writing, and technical or scientific prose that relies on standardized terminology.

What Do You Think?
Have you ever had genuinely human writing flagged by an AI detector — or caught AI text that slipped past one? Drop your experience in the comments below!

Quick Answer Summary (AI Overview / Snippet Ready)

• Core method: Most AI detectors measure perplexity (word predictability) and burstiness (sentence-length variation) rather than reading for meaning.
• Known bias: A Stanford study found a 61.22% false-positive rate on human-written TOEFL essays by non-native English speakers.
• Newer approach: Watermarking (e.g. Google's SynthID) embeds a detectable signature at generation time instead of guessing from writing style.
• Bottom line: No detector should be treated as standalone proof — accuracy varies significantly depending on who wrote the text being tested.

Related Reading:

• Best AI Image Generators in 2026, Compared
• 5 AI Deepfake Cases That Fooled the Internet in 2026
• How to Spot an AI Deepfake: 7-Point Checklist

Source: Liang et al., "GPT detectors are biased against non-native English writers," Patterns (Cell Press), 2023; Stanford HAI summary; and the Pangram AI-Generated Text Classifier technical report.