The simplest question a CS instructor can ask about a suspicious piece of code is also the hardest to answer with certainty: did a student write this, or did a language model?
AI code detection sits at the intersection of natural language processing, software engineering, and forensic statistics. The tools don’t look for telltale comments like “Generated by ChatGPT.” They pull apart the statistical structure of the code itself—how predictable it is token by token, how its rarity spikes and settles, and what large language models leave behind that human fingers almost never do.
What you’re about to read is a walk through the two core signals: perplexity and burstiness. I’ll use Python and C++ snippets to show both how these numbers work and where they break down.
What Makes AI-Generated Code Statistically Different
The core insight is that LLMs like GPT-4o, Claude 3.5 Sonnet, and Gemini 2.0 Flash are token predictors trained on a nearly frozen distribution of billions of source files. When they generate a function, they sample one token at a time, each step pushing toward the highest-probability next token given the context. The result is code that lands cleanly inside the distribution the model absorbed.
Human programmers, even experienced ones, don’t code like that. We pause. We backspace. We make idiosyncratic choices—sometimes a while loop, sometimes a for, sometimes a recursion. The distribution of our typing is lumpier and more uneven, and the probabilities we assign to tokens in our head don’t align with any single checkpoint of any single model.
That statistical difference is measurable with two values: how surprised a model would be by the code (perplexity), and how lumpy the sequence of those surprises is (burstiness). Measure both on a given block of code, and you get a surprisingly strong signal.
Perplexity as a Measure of Surprise in Code
Perplexity is the exponentiated average negative log-likelihood of a sequence under a model. In plain English: if you take a language model trained on a giant corpus of human-written code and feed it a snippet, how “surprised” is it by each token?
For AI-generated code, the answer is usually “not very surprised.” The generator and the detector share similar training data distributions, so the tokens the generator picks tend to land at or near the top of the detector model’s probability ranking. The result: low perplexity.
Consider a short Python function to compute Fibonacci numbers:
def fibonacci(n):
a, b = 0, 1
for _ in range(n):
a, b = b, a + b
return a
When a GPT-based detector evaluates this snippet, it sees an extremely canonical solution. Every token after the function signature is the highest-probability choice: the tuple assignment, the loop over range(n), the two-variable swap. The log probabilities barely dip. Perplexity here will be well under 10—often in the single digits for pure GPT-4 output.
A student who writes that exact same code by memory? Still low perplexity—this is exactly where naive detection fails. I’ll return to that edge case shortly.
Burstiness and the Rhythms of Human Coding
Perplexity alone is not enough, because highly idiomatic human code can also have low perplexity. The missing piece is the distribution of those token-level surprisals over the length of the sequence—a measure called burstiness.
Burstiness captures the degree to which rare, surprising events cluster together. Human-written text—and code—tends to be highly bursty. A developer might write three lines of clean, predictable list comprehension, then drop an obscure itertools.pairwise call and a manual __slots__ assignment in the next. The model, in contrast, produces a smooth sequence of moderate-to-high-probability tokens with no sudden cliffs.
In practice, burstiness is computed by sliding a window over the per-token log-probability sequence, measuring variance or calculating the fraction of tokens that lie below a rarity threshold. AI-generated code consistently shows low burstiness scores. Human code shows the opposite—even in simple homework assignments.
Here’s a synthetic example in C++ that would register as non-bursty (smooth, machine-like):
int sum_vector(const std::vector& vec) {
int total = 0;
for (int val : vec) {
total += val;
}
return total;
}
A student who writes this for a CS 101 assignment almost never produces it in a straight, linear flow. They might add an include they didn’t need, then remove it. Write the loop with an index, then refactor to range-based for. Copy-paste the function header from an earlier assignment. That messiness is invisible in the final text but encoded in the burstiness statistic because an impartial detector sees the final token sequence and finds the density of rare tokens—perhaps that stray return 0; they deleted and re-added—spiking in a few spots.
Token-Level Fingerprinting of Large Language Models
Beyond perplexity and burstiness, a newer generation of detectors profiles which tokens the model favors in specific syntactic positions. Each LLM has a distinct token distribution at the decoder layer: GPT-4 Turbo favors trailing docstring quotes differently than Claude 3, and Gemini Pro tends to use parentheses around return expressions more often in certain languages.
A detector can build an explicit fingerprint by feeding thousands of known AI-generated samples through the detector model, recording the top-k probability vectors at each token position in a syntax tree, and then scoring a candidate snippet by how closely its tokens align with that vector. This is not the same as measuring overall perplexity—it’s looking for model-specific tics.
For example, consider a Java method signature. GPT-4o tends to generate:
public static int factorial(int n) {while Claude 3.5 Sonnet often produces:
public static int factorial(int n) {
Note the opening brace placement. A token-position fingerprint detector would weight that specific byte pair heavily when distinguishing between models, and a snippet that matches the brace style of a known LLM in 90% of cases accumulates a high model-likelihood score.
When Codequiry’s engine runs AI detection on a batch of student submissions, it combines a general perplexity-burstiness score with a multi-model fingerprint layer. This stacking significantly cuts down on the false positives that a perplexity-only check would produce for idiomatic human code.
Combining Signals for a Stronger Verdict
Any single statistic—perplexity, burstiness, fingerprint alignment—can mislead. A student who memorized the “canonical” solution can produce low-perplexity, low-burstiness output. An AI-generated snippet with a deliberate prompt injection (“add a nonsensical comment”) can briefly spike burstiness.
The standard response in the detection literature is to stack scores and look for convergence. A submission that hits all of the following is overwhelmingly likely to be AI-generated:
- Perplexity below a threshold (often < 15 on a code-specific GPT detector model)
- Burstiness in the bottom 10th percentile relative to a corpus of genuine student work
- A token-fingerprint score matching GPT-4o or Claude with > 0.85 confidence
The thresholds are tuned per language, per assignment type, and sometimes per institution. A Python function in a beginner course has a different baseline distribution than a C++ template metaprogram in a senior-level systems class. Effective detectors let instructors set language-specific sensitivity and show raw scores alongside the binary “AI / not AI” flag.
“When we started stacking perplexity with burstiness and a lightweight AST comparison against known peer submissions, our false positive rate dropped from 11% to under 3% on a labeled validation set of 1,200 assignments.” — internal benchmarking note from the Codequiry AI detection team
Where AI Detection Breaks Down (and How to Handle It)
No statistical detector is foolproof. The easiest failure mode is the well-prompted human-imitating model. A student who prompts ChatGPT with “Write this function in the style of a novice CS student, with a few minor syntax quirks and a small bug” can produce output that mimics human burstiness. Detection then relies on token fingerprinting, which itself struggles when the model is heavily fine-tuned or when the prompt explicitly reshapes the output distribution.
Another common pitfall is AI-assisted code. A student types out 80% of a function, encounters a logic snag, and asks Copilot to complete the remaining 20%. The final file will have mixed statistics—low burstiness only in the last few lines—and a global perplexity score may stay above the threshold. Detection will flag it as ambiguous at best.
This is why a AI code detector that only returns a single score is insufficient. Instructors need line-level or block-level attribution, and they need that attribution combined with similarity checks. If a section of a file is simultaneously flagged as AI-generated and a strong web-match exists for that block on GitHub, the case becomes much clearer. Detecting AI-generated code works best as one part of a broader integrity workflow—not as a standalone silver bullet.
What Professors Should Look For in an AI Code Detection Tool
If you’re evaluating a plagiarism checker for code that also claims AI detection, here’s the realist’s checklist:
- Transparent per-file scores, not just verdicts. You need the raw perplexity and burstiness numbers, not a green/red dot.
- Language-model specific fingerprinting. Any tool that doesn’t tell you which model it thinks generated the code (GPT-4, Claude, Gemini) is probably running a one-size-fits-all detector that collapses under varied assignments.
- Integration with peer-similarity and web-source checks. AI detection alone has too many edge cases. A platform that combines AI detection with a source code plagiarism checker across the course roster and the open web gives you a web of evidence rather than one fragile signal.
- Language-specific baselines. Python and C++ have different background perplexities. A good tool normalizes by language and by the difficulty of the assignment.
Codequiry’s AI detection module gives you a stacked score: a general perplexity-burstiness check, a multi-model fingerprint match, and an automatic cross-reference with the code plagiarism checker that scans peer submissions and web sources. In the dashboard, you see a single report that flags submissions scoring high on AI-generated confidence and shows whether the flagged block also appears elsewhere—so you’re never making an accusation on a single number alone.
Frequently Asked Questions
How exactly does an AI code detector use perplexity?It runs the code through a detector language model trained on human-written code and calculates the average negative log probability of each token. Low perplexity means the model found the sequence highly predictable, which is a common property of AI-generated text. Human code tends to show higher perplexity because it deviates from the most-probable token path more frequently.
What does burstiness measure in code?Burstiness measures whether rare, low-probability tokens cluster together or are spread evenly. Human coders produce bursty patterns—sudden unpredictable choices—while language models produce a smooth, consistent probability distribution. Low burstiness alongside low perplexity is a strong indicator of machine generation.
Can a student beat AI detection by editing the generated code?Superficial edits like renaming variables or reformatting do almost nothing—perplexity and burstiness operate on the underlying token sequence, not formatting. Deeper refactors that change the distribution can reduce the confidence score, but combining AI detection with peer-similarity checking closes the loophole: if the refactored code still matches a known AI pattern and shows no similarity to other students, the statistical shadow remains.
Does Codequiry’s AI detector work with languages other than Python?Yes. The detector models are trained on multi-language corpora covering Python, Java, C++, JavaScript, TypeScript, and several other common languages. Perplexity baselines and fingerprints are calibrated per language, so the scoring stays accurate whether you’re grading a Python intro course or a senior-level C++ elective.
Ready to see how these signals play out on your own assignments? Try the AI code detector built into Codequiry—you’ll get per-file scores, model attribution, and a cross-reference with plagiarized sources, all in one report.