Perplexity-based AI code detectors answer a simple question: how likely is this sequence of tokens to have been written by a human? The lower the perplexity, the more predictable the code — exactly what you get when a language model samples from its own distribution. That simple idea, married to burstiness scoring and structural fingerprinting, now catches AI-generated code from ChatGPT, Copilot, Claude, and Gemini with F1 scores north of 0.90 in controlled benchmarks.
But the real story is in the engineering. A raw perplexity threshold alone would bury you in false positives — those carefully commented, cleanly styled submissions from your best students. The detectors that actually work in classrooms and code-review pipelines combine multiple independent signals, and they calibrate on institution-specific baselines. Here’s how every layer operates.
The Mathematics of Perplexity in Code
Given a tokenized source-code sequence w1, w2, …, wN, perplexity is the exponentiated average negative log-likelihood of the sequence under a specific language model:
PP(W) = exp( - (1/N) * Σᵢ₌₁ᴺ log P(wᵢ | w₁, …, wᵢ₋₁) )If a Java method begins public static int, a model trained on millions of human-written functions will assign a high conditional probability to add or calculate and a low probability to wombat. A whole file whose tokens constantly fall in the high-probability bucket — i.e., the model finds it maximally predictable — will have a perplexity close to 1–3 on the model’s tokenizer. Human code, even from a skilled developer, vibrates between 5 and 30 in many studies, because we introduce genuine variability: personal naming conventions, imperfect comments, half-refactored codepaths, and the occasional offbeat idiom.
In practice, most detectors use a base model like GPT-2’s Byte-Pair Encoding tokenizer or a CodeBERT-family model fine-tuned on natural code. The choice of underlying model matters enormously. A model trained predominantly on clean open-source libraries will flag neat student code as “unusually predictable” unless you inject a domain-specific calibration offset. That’s why the raw log probabilities are rarely the final judgment — they feed into a statistical test or a logistic regression that accounts for baseline human perplexity in the same course or company repository.
Why AI-Generated Code Has Low Perplexity
Large language models generate code by sampling from a probability distribution shaped during training. Even with temperature above zero, the model’s output tokens overwhelmingly remain in the high-density region of its own manifold. The result: sequences that look “smoothed” — no genuinely surprising jump to a variable name the model hasn’t seen before, no function signature that blatantly ignores the typical ordering of parameters.
A 2023 study by Stanford’s AI Lab examined 12,000 Python and Java submissions across three semesters. ChatGPT-4-generated code exhibited a median token-level perplexity of 2.4 (IQR 1.9–3.1) when evaluated by a CodeBERT-based estimator. Human submissions from the same assignments had a median of 8.7 (IQR 5.2–14.3). The distribution overlap was about 7% — meaning a naive perplexity cutoff of 3.0 would flag 93% of AI code but also 7% of legitimate student code. In a class of 300 students, that’s 21 spurious accusations. Clearly, the threshold alone is dangerous.
The solution is to treat perplexity as one input among several. When combined with burstiness (next section) and AST fingerprinting, the false-positive rate in the same Stanford dataset fell to 1.8% while still detecting 91% of AI-generated code. That triaging model is what production detectors like Codequiry’s AI code detector have optimized for — not raw perplexity alone.
Burstiness and Other Statistical Signals
Burstiness measures the variance of perplexity across contiguous chunks of code. Humans write in bursts — a few lines of tightly structured logic, then a comment block with high syntactic entropy, then a copy-pasted snippet from Stack Overflow that spikes the local perplexity. LLM output, in contrast, keeps a remarkably stable perplexity trace, because the model has no off-days, no distractions, no sudden bouts of creativity.
To quantify burstiness, a detector slices the token sequence into windows of 50–100 tokens and computes the coefficient of variation of window-level perplexity scores. In the Edinburgh Language and Code Lab’s 2024 benchmark of six AI code detectors, burstiness added an average of 9.4 points to an F1 score when paired with perplexity alone.
Other signals include:
- Token-rank slope. For each position, record the model’s rank of the actual token. Human code shows occasional low-rank surprises; AI code stays in the top-5 predicted tokens nearly continuously.
- Repetition pattern. LLMs tend to repeat identical sub-tree structures within a file more often than humans do, because they lack the meta-reasoning that says “this repeated block should be factored.”
- Comment-to-code ratio. GPT-4-turbo and Claude 3, when asked to provide “well-documented” code, often sprinkle comments that mirror the function name almost verbatim — a pattern Codility and other platforms scored as 3× more common in AI submissions.
“We could spot ChatGPT code in under two seconds just from the comment style. It would write // Calculate the sum of the array above a loop that already screamed ‘summation’ to a human reader. That’s a tic you don’t unsee.” — Markus Lindström, TA Coordinator, KTH Royal Institute of Technology, on detecting AI-generated Python assignments.
AST Fingerprinting Catches the Refactoring Gap
Perplexity and burstiness work on flat token streams, but code has structure. A detector that also parses the source into an Abstract Syntax Tree (AST) can extract fingerprints that survive light plagiarism-hiding tactics — variable renaming, loop-to-list-comprehension swaps, and reordering of independent statements. When Copilot or ChatGPT generates two similar solutions to the same prompt, the resulting ASTs often share a common skeletal shape: the same control-flow nesting, the same declaration ordering, and even the same shape of exception handling.
Codequiry computes a normalized AST hash for each submission and compares it against a corpus of known AI-generated patterns, as well as against peer submissions. A submission with extremely low perplexity and a high AST similarity to known ChatGPT output triples the confidence signal. In a 2024 blind test across 2,000 student assignments in Java, this combined signal reduced false positives from 2.1% (perplexity alone) to 0.4%, while still flagging 87% of assignments suspected to be fully AI-generated.
The AST signal also helps distinguish “AI-assisted” from “fully AI-generated.” A student who writes most of the logic but uses Copilot to generate a helper function will show a spike in AI-similar AST patterns only within that function’s subtree, not across the entire file. Segmenting by function versus whole-file similarity gives instructors a nuanced view — and a better conversation starter than a binary flag.
Real-World Performance and Ground Truth
A proper assessment needs a benchmark where ground truth is known. The Edinburgh Code-AI-2024 dataset contains 8,000 files — 4,000 written by CS undergraduates under exam conditions, 4,000 generated by GPT-4, GPT-3.5, Claude 2, and Gemini Pro across identical prompts. Every detector was evaluated on recall, precision, and false-positive rate at file level.
| Detector | Methodology | F1 Score | False-Positive Rate | Can distinct assistant? |
|---|---|---|---|---|
| Perplexity-only (GPT-2 tokenizer) | Log-likelihood threshold at 3.0 | 0.79 | 6.8% | No |
| Perplexity + burstiness | Linear discriminant on log PP & CV of window PP | 0.88 | 3.1% | No |
| Codequiry AI Detector | Perplexity + burstiness + AST fingerprinting + web-source cross-check | 0.92 | 1.8% | Yes — attribute to ChatGPT, Copilot, Claude, or Gemini |
| Commercial detector A (text-based) | Zero-shot GPT-2 fine-tune (ignores code structure) | 0.71 | 14.2% | No |
Numbers from the Edinburgh benchmark, averaged across Java, Python, and C++ subsets. Commercial detector A was a widely advertised tool that treated code as prose; its high false-positive rate stemmed from flagging boilerplate and well-documented student code equally. The lesson: code detection demands code-aware models. Using a generic text AI detector on programming assignments yields a coin-flip accuracy for Python and worse for C++.
Codequiry’s web-source cross-check provides an additional hard signal. Many students who prompt an LLM for code will also paste snippets from the model’s training corpus — GitHub, tutorial sites, Stack Overflow. The plagiarism checker for code runs each segment’s fingerprint against 20+ million public repositories. If a submission shows both low perplexity and a 78% web match to a Copilot-generated repo, the evidence stack becomes hard to dispute. This multi-layered approach is what moves a detector from “interesting research tool” to something a university’s academic integrity board can rely on.
Deploying AI Detection in CS Courses Without a Witch Hunt
The best detector in the world is useless if it turns a department into an adversarial courtroom. Faculty at UC San Diego and the University of Toronto report that the single most important design decision was not to hide the detection. Students are told in syllabus and in lecture: “AI detection is active, it runs on every submission, and it looks at statistical fingerprints plus code structure.” That transparency alone reduces misuse by roughly 30%, according to a controlled rollout at UCSD in Fall 2023, because the perceived risk of getting caught shifts the cost-benefit calculation far earlier than any actual sanction.
On the technical side, the detector should produce a confidence score and a heatmap — not a yes/no verdict. Codequiry’s AI code detector highlights specific code regions with the highest AI probability, so a TA can open the file and immediately see whether a single helper function is suspect or the entire 200-line module. During office hours, that heatmap becomes a teaching tool: “This function block has a 94% AI probability — walk me through it.” Many students who initially denied using ChatGPT later confessed they’d plugged in a sub-problem they couldn’t solve, which sparked a discussion about acceptable AI boundaries in the course.
For instructors who already use a code plagiarism checker, the integration is straightforward. Codequiry’s dashboard runs peer-similarity, web-source detection, and AI detection in a single pass, returning a combined similarity report. TA workloads drop because they no longer need to run MOSS and a separate AI detector and then manually cross-reference the results. And because the system covers 36+ languages, the same workflow serves the intro Python course as well as the senior C++ systems lab.
Where All Detectors Still Break — and Where They’re Headed
No detector handles heavily obfuscated, line-by-line rewritten code perfectly. If a student takes GPT-4 output and manually renames variables, reorders independent statements, swaps for-loop forms, and injects natural-looking comments, the perplexity signal drifts back toward human territory. AST fingerprinting helps — the skeletal structure often survives line-level edits — but a motivated attacker with strong coding skills can push the detection probability below 50%.
The next generation of detectors will need to look at temporal patterns. Did the student’s first commit push 400 lines in one shot with no revision history, followed by a series of tiny comment-only edits an hour later? In a Spring 2024 pilot at Reykjavik University, commit-trace features boosted AI detection recall by 11% for such batch submissions. The approach is privacy-safe — git timestamps and diff statistics, not keystroke logging — and it meshes naturally with the GitHub Classroom and GitLab workflows already used in many CS programs.
Watermarking, while promising, remains brittle. OpenAI’s deterministic token-sampling proposal has not been deployed in production APIs, and it fails trivially if the student retypes the code. Until watermarking is broadly adopted, the practical path is statistical detection plus good assignment design — crafting problems that require architecture diagrams, intermediary design artifacts, and in-class explanations. Every layer reduces the attack surface a little more.
Frequently Asked Questions
What exactly is perplexity in AI code detection?
Perplexity measures how well a language model predicts the next token in a sequence. Low perplexity means the model is not surprised — typical of AI-generated output. High perplexity means more unpredictability, which is common in human-written code. Detectors set a threshold, but effective systems combine it with other signals like burstiness and AST analysis.
Can a low-perplexity score alone prove a student used ChatGPT?
No. Some human-written code is naturally clean and predictable, especially boilerplate or well-documented functions. Using perplexity alone leads to false-positive rates around 6–7% in benchmarks. A responsible detector treats it as one input among several and never issues a final judgment without human review of the heatmap.
What is the actual false-positive rate for the best AI code detectors?
In the Edinburgh Code-AI-2024 benchmark, the top-performing multi-signal detector (combining perplexity, burstiness, AST fingerprints, and web-source checking) achieved a 1.8% false-positive rate at the file level. That rate must be interpreted in the context of a specific student population; it will shift slightly depending on the course’s baseline code style. The detector should always be calibrated on a sample of human-only submissions from the same cohort.
Does Codequiry differentiate between fully AI-generated and AI-assisted code?
Yes. By segmenting the AST, Codequiry’s AI detector can flag specific functions as likely AI-written while recognizing that the rest of the file is human-originated. Instructors see a per-function AI probability, not a single flag for the whole assignment, which leads to fairer conversations with students.
To test Codequiry on your own course or codebase, run a batch of submissions through the combined code plagiarism checker and AI-detection pipeline — the detection report includes a heatmap, similarity scores, and web-source matches in one view.