Last spring I pulled the AI detection scores for a 412-submission CS 2 cohort and plotted them before reading a single student name. The distribution was bimodal. One cluster sat below 15. A second clustered between 70 and 95. The part that cost me the weekend was the eleven submissions between 30 and 55, where an AI code detection score tells you almost nothing on its own.
That middle band is the whole problem. Detectors are far better at ranking a cohort than at labeling one student, and nearly every bad outcome I've watched unfold started with someone treating a ranking as a label.
What an AI code detection score actually measures
Detectors that ship with code in mind don't work like the prose detectors you've read about. GPTZero (public in January 2023), Turnitin's AI writing indicator (April 2023), and OpenAI's own classifier that the company retired on July 20, 2023 for a "low rate of accuracy" all operate on natural language statistics. Code breaks most of those assumptions. A for loop is low-entropy by design, and so is return sum(xs) / len(xs). There's no burstiness to measure in a language where two spaces and a newline are the grammar.
So code detectors lean on the choices around the syntax instead: identifier length and casing distributions, comment-to-code ratio, docstring style, whether the empty case gets handled before or after the main path, cyclomatic complexity relative to function length, import ordering.
Here's the same function twice. The first came from a 2019 CS 1 archive. The second is the shape we see from GPT-4o and Claude 3.5 Sonnet submissions almost weekly.
def get_avg(nums):
if len(nums) == 0: return 0
s = 0
for n in nums:
s += n
return s / len(nums)
def calculate_average(numbers: list[float]) -> float:
"""Calculate the arithmetic mean of a list of numbers.
Args:
numbers: A list of numeric values.
Returns:
The arithmetic mean, or 0.0 if the list is empty.
"""
if not numbers:
return 0.0
return sum(numbers) / len(numbers)
Type hints, a Google-style docstring, and if not numbers rather than len(nums) == 0. Any one of those is meaningless. A first-year student in 2024 absolutely might write any of them, especially if their IDE generates the docstring stub for them. All three together, repeated across six files, is a pattern worth looking at. That distinction, single feature versus pattern across files, is the thing the detectors are actually good at, and it's why the score needs a framework around it.
Step one: know your cohort base rate before you read anything
Base rates dominate everything downstream. In our Fall 2023 term, six instructors self-reported which assignments permitted Copilot (generally available since June 2022). Two of them allowed it outright on boilerplate. In those two sections, a high AI score carried almost no information, because plausible use of autocomplete was the expected behavior.
This forces a distinction that instructors tend to blur: AI-assisted versus AI-generated. If the learning objective for week four is writing a nested loop, and the student tab-completed the nested loop, the objective didn't happen even if the student could explain the code afterward. If the objective is data modeling and the student used AI to scaffold a CLI, the objective probably did happen. The score can't tell you which situation you're in. Only the assignment's learning objective can. That's a design problem, and it's why code plagiarism checker for teachers documentation that starts from assignment design tends to be more useful than documentation that starts from the detector.
Step two: read the distribution, not the individual score
A raw score of 71 means something completely different in a cohort whose mean is 18 versus one whose mean is 58. We rank before we read. The first pass is a z-score against the cohort, and it takes about four lines of NumPy.
import numpy as np
def flag_outliers(scores: dict[str, float], z_cut: float = 2.0) -> list[str]:
vals = np.array(list(scores.values()))
mu, sigma = vals.mean(), vals.std(ddof=1)
return sorted(
[sid for sid, s in scores.items() if (s - mu) / sigma > z_cut],
key=lambda sid: -scores[sid],
)
Our first version of this had a bug I'd rather not repeat. The z-score cutoff was computed after we had already removed the flagged submissions from the array, which deflated sigma and pushed one student below the line. We caught it in week nine. If you write this script, sort the scores first, then filter, and log the mean and standard deviation you computed so a second reader can check your arithmetic.
Z-scores also assume a roughly unimodal distribution, and ours was not. Once we noticed the bimodality, we moved to a two-component mixture model in 2024, which is more work and only modestly better at the margins. We haven't validated any of the band thresholds past cohorts of roughly 400 submissions.
Step three: drop to file level and read the evidence
A submission-level score of 55 is close to useless. A per-file view usually isn't. In one case from that 412-submission cohort, a 300-line project scored 41 overall. Broken out by file, utils.py sat at 88 and every other file sat between 9 and 16. That's a stub-completion signature: a student who wrote their own main and their own tests, then handed the one genuinely fiddly utility file to a model.
The reverse pattern matters too. A file that scores 90 because it's 22 lines of argparse boilerplate is not evidence of anything. Short files carry almost no signal, and any tool that reports a confident percentage on 15 lines is overselling. This is the view we use in Codequiry's AI code detector: a per-file probability with written indicators next to each file, so you can see whether the signal is concentrated or spread.
Step four: stack AI, peer, and web signals
One axis is never enough. A submission with AI 82, peer similarity 4, and zero web matches describes a student who wrote their own structure and used a model for content. A submission with AI 82, peer similarity 91, and a match against a GitHub repository from 2021 describes something else entirely, and the AI score is now the least interesting number in the report.
Peer similarity engines (MOSS, JPlag, Dolos) were never built for AI detection, and they're still the best tools available for the peer axis. Web and GitHub matching is a separate axis again. Running all three in one pass and reading them together is the difference between a defensible conversation and a guess. Codequiry runs peer similarity, open-web and GitHub matching, and AI detection against the same submission set, which is what makes the source code plagiarism checker report readable as a single artifact rather than three exports stapled together.
After two years we settled on four bands. They are calibrated to our institution and our assignments. Yours will differ, and you should re-derive them each term.
| Band | What we usually see | What we do |
|---|---|---|
| 0 to 20 | Indistinguishable from the human baseline | Nothing. No action, no notation. |
| 20 to 45 | The noise band. Mixed submissions, IDE-generated docstrings, heavy use of autocomplete | File-level read before any conversation. About half resolve to nothing. |
| 45 to 70 | Consistent signal across several files, or one concentrated file | Review the top two files by score. Compare against the student's prior submissions in the same course. |
| 70 to 100 | Signal across most files, structural style divergent from the student's earlier work | Review, then talk to the student about process rather than about the score. |
Where this framework breaks down
Three places, and we've hit all of them.
Short submissions are the first. Anything under about 40 lines has too little surface for a statistical detector, and the confidence interval on those scores is wide enough to drive a truck through. We now exclude short files from the AI axis entirely and rely on peer and web matching for those assignments.
The second is prose inside code. Liang et al. found in 2023 that seven widely used text detectors flagged 61% of TOEFL essays written by non-native English speakers. Code detectors inherit some of that exposure through comments and docstrings. We've seen a strong writer with an unusual comment voice land at 62 with no other signal at all. That's a false positive, and it's the kind you only catch by reading.
The third is IDE tooling. PyCharm's docstring generation and the various VS Code docstring extensions produce output that looks, byte for byte, like what a model produces. If your students use those features and you don't know it, you will misread the report. We ask about IDE setup in the first week now, which is a slightly awkward conversation that has saved us at least three bad ones.
Frequently asked questions
What counts as a normal AI detection score for a cohort?
In our data, the human median sits between 8 and 16, and the standard deviation is wide enough that anything under roughly 45 is not worth a conversation on its own. That's our cohort. Yours depends on how much autocomplete your students are already using, so derive your own baseline from a known-human assignment in week one.
Can an AI detection score prove a student cheated?
No. A score is a prior that tells you where to look, and the evidence is the file-level pattern plus a comparison against that student's earlier work. Every policy we've written says the same thing: the score opens a review, it never closes one.
Do AI detectors flag code written by humans?
Yes, and predictably. Short files, IDE-generated docstrings, and idiosyncratic but legitimate comment styles are the three we see most. Treat anything under 45 as a ranking signal only, and read the file-level indicators before you act on it.
If you want to see the per-file and stacked views described above on your own cohort, Codequiry will run peer, web, and AI analysis on a sample set before you commit a full course, and the code plagiarism checker documentation covers the band setup we've described here.