Last spring I spent a long evening reviewing 63 take-home submissions for a Java course I run. The assignment was an in-memory key-value store with a TTL eviction policy. Similarity reports caught two pairs of obvious copy-paste. But the uncomfortable part was five files that had no peer overlap, no suspicious comments, and a rhythm I could not quite place. So I ran their method bodies through a small code language model and looked at token-level perplexity. Three of those five had scores far enough from the cohort baseline that I pulled them into follow-up conversations.
Token-level perplexity is not a verdict. It is a statistical signature that can help an AI code detector flag probable model output, especially when paired with similarity and structural checks. This is how it works, where it breaks, and how we use the thresholds in bootcamp grading.
How Perplexity Works on Source Code
A language model assigns a probability to each token given the tokens before it. If the model sees the sequence Cursor cursor = new Cursor(...) and predicts cursor with 0.91 probability, that token is not surprising. If a human writes var row = map.get(k).orElse(null) and the model assigns the orElse call a probability of 0.03, that token is surprising. Perplexity takes the average of those surprised reactions across the whole sequence and turns it into a single number. Low perplexity means the model found the sequence very predictable. High perplexity means the model kept getting surprised.
The useful fact for AI code detection is that GPT-4o, Claude 3.5 Sonnet, and Copilot all generate from similar autoregressive machinery. Their output tends to fall along high-probability paths for the same family of models. A detection tool does not need to use the exact same model that generated the code. A reasonably close code language model, including a small open model like StarCoder2 15B, can often spot the characteristic smoothness.
Perplexity is a score of surprise, not a verdict.
Under the hood, the calculation looks close to this:
def mean_negative_log_prob(tokens, scorer):
total = 0.0
for i, token in enumerate(tokens):
# scorer returns log P(token | tokens[:i])
total += scorer.log_prob(tokens[:i], token)
return total / len(tokens)
You exponentiate that mean to get perplexity. In practice, most code detectors keep the raw mean negative log-likelihood or an entropy-like transform because the raw number behaves more linearly for threshold tuning.
Why Raw Perplexity Is Not Enough by Itself
If you stop at a single perplexity number, you will chase false positives. Comments, boilerplate, and license headers are all highly predictable. A Java file with 40 lines of Javadoc and getters will look artificially smooth. Code copied from Stack Overflow also tends to have low perplexity because it is canonical, widely seen, and exactly what a model expects. That is why any reasonable workflow strips comments, imports, and generated accessors before scoring the meaningful method bodies.
Strong developers also write low-perplexity code in small doses. A very experienced Java programmer who writes terse, idiomatic code can produce sequences that a model finds completely ordinary. That does not mean the developer used an LLM. It means the style overlaps with the distribution the model learned. I have seen this enough times that I never treat a low score alone as evidence.
A few things to watch before trusting a raw score:
- Generated accessors, constructors, and Javadoc headers drag the score down without any meaningful signal.
- Code copied from the web can look model-like because it is already canonical.
- Very experienced developers sometimes write with the same regularity as an LLM.
Burstiness and the Shape of Human Code
Human code is bursty. A developer might write a clever one-line ternary, then a six-line error handler that nobody else would structure the same way, then a one-word comment, then a blank line. The probability of the next token jumps around. LLM output, especially from ChatGPT and Copilot, tends to be smoother. The model stays inside a narrow band of likely sequences, so the per-token losses have lower variance even when the mean is similar.
That variance is burstiness. A detector can measure it by looking at the standard deviation of the token-level losses or by tracking runs of consecutive low-surprise tokens. Human code often has short runs of very ordinary tokens followed by a single jarring choice. LLM code keeps the surprise fairly level for longer stretches. When both mean perplexity is low and burstiness is low, the file looks much more like generated output.
I have also seen the reverse. A student who writes heavily self-conscious, overly defensive code can produce a high burstiness signal that looks human. The shape matters, but it can be imitated.
Perplexity Thresholds for Detecting AI Code
For the Java method bodies in our program, using a smaller open code model to compute mean negative log-likelihood, we settled on loose review bands. The numbers are from our own cohort, so treat them as a starting point, not a universal rule. The first term I used these numbers I forgot to strip Javadoc comments before tokenizing, and the comment tokens pushed several clearly human files into the flag range. That evening's review queue was longer than it needed to be.
| Submission type | Typical mean negative log-likelihood | Review action |
|---|---|---|
| Human Java with natural comments | 0.48 to 0.72 | No flag |
| Human compact competitive style | 0.31 to 0.47 | Review, conversation |
| GPT-4o generated Java | 0.22 to 0.34 | High suspicion if no web match |
| Copilot with strong user style | 0.28 to 0.44 | Ambiguous, stack with similarity |
I want to repeat that these bands come from a few hundred method-level inspections. We have not tested them across thousands of institutions or against every model release. A new model version can shift the distribution in a week.
The useful move is to look at the distribution across a whole cohort. If a file sits three standard deviations below the class mean on mean negative log-likelihood and the burstiness is also low, that file deserves a closer read. The threshold is less about a magic number and more about locating outliers in the cohort.
False Positives Are Real, Especially for Terse Coders
One of the five files I flagged last spring turned out to be written by a student from a competitive programming background. The code was compact, had almost no comments, and used variable names like tmp and idx. The perplexity score looked bad. But when I sat down with her, she walked me through the eviction logic from memory and refactored it live. Her style simply overlapped with the model's expected style. I cleared the flag. That conversation mattered more than the number.
False positives happen more often on short files. A 40-token method does not give a detector enough statistical surface to make a stable call. The same is true for code that is mostly imports, annotations, and framework scaffolding. If the detector flags a short Spring controller, I read it manually before assuming anything.
The unpleasant false negative is harder to catch. A student or developer can generate a first draft with an LLM, then edit it heavily. The edits add human noise. The mean loss number drifts upward, but the structure may still be generated underneath. That is why a single perplexity threshold misses a meaningful share of AI-assisted code. I treat the score as a sorting tool, not a classifier with a clean boundary.
Stacking Perplexity with Peer and Web Similarity
The strongest way I have found to use perplexity is alongside a code plagiarism checker. A file with low perplexity but no peer match and no web match tells one story. A file with low perplexity and a 92% match to a GitHub repository tells a much clearer story. The two signals together change how I allocate review time.
Codequiry's approach follows that same logic. Its AI detection report does not rely on a single statistical marker. It combines token-level scoring with structural fingerprinting, peer-similarity checking, and web-source tracing. When I review a batch, the score breakdown separates peer similarity, web similarity, and AI generation so I can see which signal drove the flag. That matters because a low AI score with strong web matches usually means copied code, not generated code.
From a workflow view, I care less about the exact perplexity number and more about which students appear at the top of the review queue because several independent signals agreed. That is the practical difference between using raw model probabilities and using a purpose-built detector.
What I Tell Students About Perplexity Scores
I show students the score distribution in the first week of the term. It makes the review process less mysterious and more honest. I explain that a low perplexity score is not an accusation, and I tell them that if a flag leads to a conversation, they will get a chance to explain their code. That part has reduced the anxiety around automated review more than any policy document.
It has also reduced the number of students who think they can paste from ChatGPT without being noticed. Once they see that the detector is looking at token probability patterns, not just phrasing, the assignment feels riskier to game. Some students still try. Most realize the detection surface is larger than they expected.
Frequently Asked Questions
Can professors reliably detect ChatGPT code with perplexity alone?
No. Perplexity alone produces both false positives and false negatives. It works best as one signal in a combined detector that also checks peer similarity, web sources, and structural features.
What is a typical perplexity threshold for AI code?
There is no universal threshold. For Java method bodies scored with a small code model, a mean negative log-likelihood below about 0.34 is suspicious in our cohort, but the number shifts by language, model, and file size.
Does GitHub Copilot code have distinguishable perplexity?
Sometimes. Copilot often mirrors the user's surrounding style, so its output can be harder to flag than raw ChatGPT output. When the surrounding repo is idiomatic, Copilot additions blend in and require structural or similarity checks.
How does Codequiry compare to raw perplexity tools?
Raw perplexity gives you a probability score. Codequiry combines that kind of statistical signal with fingerprinting, peer comparison, and web-source tracing, then presents the results in a review queue. You can run one cohort through the AI code detector and see how the flags distribute before you commit to a policy.