How AI Code Detectors Score ChatGPT, Claude, and Copilot

AI code detectors score a file on three signals: perplexity (how predictable each token is under a reference model), burstiness (how much that predictability varies across the file), and structural regularity (how uniform the shape of the code is). Detectors built for essays lean on the first two. Code-native detectors lean on the third. That split explains most of the disagreements you get when you run the same submission through two different tools.

I teach a 14-week backend bootcamp and I maintain two open-source Python libraries, which means I read a lot of student submissions and a lot of pull requests from strangers. Last spring I got tired of arguing with colleagues about whether a file "looked generated," so I built a small scoring pipeline and ran 900 generated files through it: 300 from ChatGPT, 300 from Claude, 300 from Copilot autocomplete. I added 240 human-written files as a control. All of it Python and Java, drawn from my own courses and from a handful of public repos where I could get contributor consent.

Before anyone quotes the percentages back at me: this is a few hundred files, not a benchmark suite. We haven't tested past that scale. Treat the numbers as directional and the mechanics as the actual point.

What perplexity actually measures on source code

Perplexity is the exponentiated mean negative log-likelihood per token, measured against some reference language model. Lower means the text was more predictable to that model. On prose, generated text tends to land at lower perplexity than human text because an LLM drifts toward the most probable phrasing available.

Here's the roughly correct way to get per-token numbers out of a code model, in this case StarCoder2-3B from 2024:

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

tok = AutoTokenizer.from_pretrained("bigcode/starcoder2-3b")
model = AutoModelForCausalLM.from_pretrained("bigcode/starcoder2-3b")

def token_logprobs(path):
    src = open(path).read()
    ids = tok(src, return_tensors="pt").input_ids
    with torch.no_grad():
        logits = model(ids).logits
    lp = torch.log_softmax(logits, dim=-1)
    return lp[0, :-1].gather(-1, ids[0, 1:].unsqueeze(-1)).squeeze(-1)

Those values are in nats. On my corpus, fully generated Python files averaged about half a nat lower per line than human files, which sounds like a clean separation until you look at the overlap. Roughly a third of the human files sat inside the generated range entirely. Beginner code does this constantly: heavy boilerplate drags perplexity down, then a variable name like xq or foo2 yanks it back up, and the file averages out to something indistinguishable from generated code.

Burstiness is weaker on code than on essays

Burstiness is the variance of per-token surprise. Humans spike. You write twelve boring lines of a loop, then one line with a regex that nobody on the team can fully parse, then three lines of comment explaining why. Generated code is flatter, and the flattery is measurable.

On its own, burstiness flagged 71% of my fully generated files and 29% of the human ones. That is a terrible standalone detector and it's the reason people who only ever tested essay detectors get frustrated with code. Prose has sentences of wildly different lengths. Code doesn't. Almost every line in a Python function is between 15 and 60 characters, and the natural variance is small enough that the signal gets buried in formatting noise.

Where AI code detection actually gets traction

Structural regularity did the heavy lifting. Not the shape of the tokens, the shape of the program. Generated files are evenly clean in a way that people who debugged their own code at 2am never quite manage. The tells I keep seeing, in rough order of reliability:

  • A docstring on every function, including three-line getters that don't need one
  • Type hints applied uniformly and correctly throughout the entire file
  • Every I/O call wrapped in try/except Exception with a generic log message
  • No commented-out code, no stray print() statements, no TODO markers
  • Identifiers like result, data, value, temp repeated across unrelated functions
  • Comment density that stays constant from the first function to the last

That last one is the strongest single feature I've found. Fully generated files in my set averaged one comment every 4.1 lines. Human files averaged one every 11.6 lines, and the density swung between blocks, sometimes 1-in-2 and sometimes 1-in-40. Uniformity is the signal, not volume.

Compare two functions that do the same thing:

def get_user(user_id: int) -> dict:
    """Retrieve a user record by ID."""
    try:
        user = db.fetch(user_id)
        return {"id": user.id, "name": user.name}
    except Exception as e:
        logger.error(f"Failed to fetch user: {e}")
        return {}
def get_user(uid):
    # ticket 4412 - fetch() returns None for soft-deleted rows, handle that
    u = db.fetch(uid)
    if u is None:
        return None
    d = {"id": u.id, "name": u.name}
    # print("DEBUG", d)  # leave for now
    return d

The second one came from a real PR and the commented-out debug line is what gives it away as human. Nobody generating code leaves that behind. It's the residue of a person who was confused for twenty minutes.

The tell isn't that generated code is clean. It's that it's evenly clean.
Codequiry AI code detection report with average AI score, highest file score and a risk distribution
AI code detection: probability scores per file, flagging submissions likely written by ChatGPT, Copilot, Claude or Gemini.

The per-model numbers, including the awkward one

Source Mean logprob (nats) Burstiness (σ) Comment density Flagged at 0.7
Human (240 files) −2.41 1.87 1 per 11.6 lines 9%
ChatGPT (300) −2.92 0.94 1 per 4.1 lines 84%
Claude (300) −2.88 1.02 1 per 3.6 lines 81%
Copilot (300) −2.63 1.41 1 per 7.8 lines 52%

Copilot is the hard case and it's the one most courses refuse to talk about. Inline autocomplete produces hybrid files: a human types the structure, the model fills in six lines of a dictionary comprehension, the human edits two of them and moves on. The file is genuinely co-authored. You can't score it as generated or human without picking a threshold that's really a policy decision about what you'll accept.

Claude ran slightly hotter on comment density than ChatGPT in my set, mostly docstrings, and slightly more variable in burstiness. That's a weak signal on a small sample. I wouldn't build a Claude-vs-ChatGPT classifier on it and I'd be suspicious of anyone selling one.

Where the signals disagree, and why

Three false-positive patterns show up over and over, and if you're running any of this in a classroom you'll meet all three in the first semester.

Non-native English speakers get flagged on text detectors. A comment written in slightly unusual English has high perplexity under a model trained on GitHub prose, and a text-based detector reads that as "human, but oddly so" or, worse, as adversarial rewriting. This is a well-documented problem with essay detectors and it transfers directly to code submitted by international students.

Templated codebases look generated because they are generated. A Django model file, a Spring controller, a Rails scaffold, all of these have been machine-produced since well before LLMs existed. JHipster was doing this in 2013. A confident AI score on a file that a student scaffolded is a nonsense result, and it's a common one in web development courses.

Refactoring destroys token-level signals but not structural ones. Rename every variable, reorder the functions, convert a for loop to a comprehension, and perplexity jitters randomly while the structural features stay put. That asymmetry is the reason code-native detectors are worth their cost. A AI code detector that only reads tokens will miss a student who ran the generated output through a formatter, and a source code plagiarism checker that only matches text will miss the same student copying a peer.

Codequiry insights score breakdown separating peer similarity, web similarity and AI generation, with match sources
The score breakdown: peer similarity, web similarity and AI probability reported separately, with where the matches came from.

Stacking the signals instead of picking one

The practical conclusion from all this is boring and correct: no single signal survives contact with a real cohort. What works is running AI scoring, peer similarity, and web-origin matching over the same submission in one pass and reading the three together. A file with a high AI score and zero peer matches is a different conversation from a file with a high AI score and a 94% match to a GitHub repo from 2019.

This is where the tooling gap is widest. Most detectors I've evaluated do exactly one of those three things. Codequiry does all three against the same upload: token and AST similarity across peer submissions, web results traced back to repositories and Stack Overflow threads, and a per-file AI score. For a bootcamp that's the difference between exporting three reports and reconciling them in a spreadsheet, which is what I was doing before. The code plagiarism checker side and the AI side share the same review queue, so a file that scores 0.81 on AI and 12% on peer similarity sorts next to a file that scores the reverse, and you can decide which one to open first.

Codequiry per-file AI analysis showing AI versus human probability for each file with written indicators
Drilling into one submission: per-file AI and human probabilities, each with the stylistic indicators behind the score.

If you're currently on MOSS or JPlag and only need peer similarity, that's a defensible choice and the Codequiry vs MOSS comparison lays out where the tradeoffs land. If you need to say anything at all about AI authorship, those tools won't help and never claimed they would.

Using the score as triage, not as a verdict

At the threshold I'd actually defend, my pipeline flagged 9% of human files. Per file, that's tolerable for a review queue and completely unacceptable as the basis for an academic integrity charge. The move is to treat the score as a sort order, not a finding, then read the file and ask the student about one specific thing in it.

Concretely: open the flagged file, find the try/except block, and ask why catching bare Exception was the right call there. A student who wrote it will have a reason, usually a bad one, sometimes a good one. A student who pasted it will describe what the code does rather than why it's shaped that way. That conversation takes ninety seconds and it's more reliable than any threshold I've tuned.

The same logic applies to hiring take-homes, which is where I see the most overreach right now. A 0.82 AI score on a take-home is a reason to schedule a follow-up, not a reason to reject. Engineering managers who treat it as a rejection filter are going to lose good candidates from the templated-code false positives above, and they'll never know it happened.

Frequently asked questions

Can a detector tell if Copilot helped with a file?

Partially, and only as a probability. Mixed-authorship files score in the middle because half the tokens came from a person. Expect detection rates around 50% for autocomplete-assisted code versus 80%+ for fully prompted generation, based on my sample.

Does renaming variables defeat AI code detection?

It defeats token-level perplexity and burstiness almost completely. It doesn't touch structural features like comment density uniformity, docstring coverage, or AST shape, which is why detectors that score on structure hold up and pure text detectors don't.

What's a realistic false positive rate for AI code detection?

On my corpus, about 9% of human files crossed a 0.7 threshold, with beginner code and templated framework code making up most of that. Any vendor quoting a false positive rate below 2% is describing a threshold you wouldn't want to use.

If you want to see how the three signals stack on your own submissions, you can run a batch through the Codequiry code plagiarism checker and compare the AI, peer, and web columns side by side before you decide what your threshold should be.