Detecting AI-generated code in student submissions is not a single-tool decision; it's a layered grading problem. The most reliable signal comes from combining statistical measures like token-level perplexity and burstiness with peer similarity and web-source checks, then requiring students to submit process evidence. In a fall 2024 review of 143 CS1 Python assignments at a mid-sized public university, this approach caught 31 of 38 AI-assisted or fully AI-generated submissions (81.6% recall), while a standalone AI code detector caught 24 and instructor intuition caught 19.
This post outlines what those layers look like in a real grading workflow, where they break down, and how to adjust assignment design so AI-generated code leaves a trail. The data comes from a semester-long audit I ran across three sections of an introductory programming course, plus follow-up interviews with the four TAs who graded them.
Why Traditional Code Similarity Misses AI-Generated Submissions
MOSS, JPlag, and similar tools are excellent at finding copy-paste plagiarism. They tokenize source code, compare fingerprints, and detect near-duplicates across a peer corpus. But AI-generated code often has no peer match. If two students prompt ChatGPT independently with the same assignment statement, they may receive syntactically different solutions — different variable names, different loop structures, occasionally different algorithms. The code is "original" in the sense that it does not match any other student file or public GitHub snippet.
That does not mean the students did their own work. It means the evidence lives somewhere else: in the statistical properties of the code itself.
Key insight: A plagiarism detector answers "is this code similar to something else?" An AI detector answers "was this code likely generated by a language model?" These are different questions, and reliable grading requires both.
The Statistical Fingerprint of LLM-Written Code
Most LLM detection for source code works by analyzing token sequences. A language model produces tokens by sampling from a probability distribution at each step. That process leaves two measurable traces:
- Perplexity — how "surprised" a model is by the next token. LLM-generated code tends to have lower perplexity when measured by a similar model because the sequence is exactly what the model would predict. Human-written code is more idiosyncratic; it contains more surprising transitions.
- Burstiness — the degree to which rare tokens cluster together. Human programmers often use simple constructs for most lines, then drop in a dense block of unusual library calls or an edge-case workaround. LLM output is more uniform; it maintains a steady, confident style throughout.
These measures work better on code than on prose because programming languages have stricter grammar. The difference between a human loop and a model loop is not just word choice — it shows up in identifier entropy, comment density, and the variability of structural patterns.
Consider two Python solutions to the same simple problem: check whether any permutation of a string is a palindrome.
A typical first-year student wrote this:
def has_palindrome_permutation(s):
counts = {}
for c in s:
counts[c] = counts.get(c, 0) + 1
odd = 0
for v in counts.values():
if v % 2 == 1:
odd += 1
if odd > 1:
return False
return True
ChatGPT produced this for the same prompt:
# Check if any permutation of the string is a palindrome
# Approach: count character frequencies; at most one odd count allowed
def can_form_palindrome(input_string):
char_frequency = {}
for char in input_string:
char_frequency[char] = char_frequency.get(char, 0) + 1
odd_count = 0
for count in char_frequency.values():
if count % 2 != 0:
odd_count += 1
if odd_count > 1:
return False
return True
The differences are subtle but measurable. The AI version has explanatory comments restating the obvious, a more verbose function name, and a slightly more structured variable naming scheme. Individually, none of that proves anything. Across 200 tokens, the token-level perplexity of the second snippet is about 31% lower than the first when scored by a model trained on Python AST node sequences from prior student cohorts.
Layering Detection Signal in a Grading Workflow
No single detector, including the best commercial tools, should be used to make a final academic-integrity decision on its own. The workflow that worked best in our fall 2024 audit had four ordered layers:
- Peer similarity check — run all submissions through a code plagiarism checker that supports token, AST, and fingerprint comparison. This catches the blatant copy-paste cases immediately.
- Web-source match — check against Stack Overflow, GitHub, and public tutorial code. Students often copy from the same web pages; a web-source match can confirm copied code even when AI detection is ambiguous.
- AI-specific probability score — run suspicious or all submissions through an AI code detector that reports a probability plus the specific token regions that triggered the score.
- Instructor or TA review with process evidence — look at the student's revision history, comments, and required process artifacts. Flagged cases are always reviewed by a human before any penalty is assigned.
Here is how the four major tools compare across those layers, based on our semester audit and vendor documentation:
| Tool | Peer Similarity | Web-Source Match | AI Code Detection | Refactoring Resistance | API / CI Support |
|---|---|---|---|---|---|
| MOSS | Strong, token-based | Limited | None | Moderate | No official API |
| JPlag | Strong, AST-based | None | None | Good | Local/self-hosted |
| Turnitin | Weak for code | Text-oriented | Text AI only | Poor for code | Limited |
| Codequiry | Strong, token + AST + fingerprint | GitHub, Stack Overflow, open web | Built-in LLM detection | Strong — survives renaming/reformatting | Web dashboard + REST API |
MOSS is still a fine free option for peer similarity, but it has no AI detection. JPlag handles refactoring better than MOSS but still misses AI-generated code entirely. Turnitin's code support is thin. Codequiry's value is that both similarity and AI detection sit in the same report, so a TA does not have to juggle three tools to grade one assignment.
What the Benchmark Numbers Showed
I collected 143 submissions from three sections of CS1. Students completed the assignment in Python under an honor code that allowed consulting documentation and instructor-provided examples but prohibited generating code with AI tools. Before grading, I labeled submissions using a combination of student interviews, revision history, and known Copilot usage logs from VS Code telemetry where students had opted in.
Of the 143 submissions, 38 were labeled as AI-assisted or fully AI-generated. The recall and precision of each detection method looked like this:
| Detection Method | Recall (caught / 38) | Precision (correct flags / all flags) | False Positives |
|---|---|---|---|
| Instructor intuition alone | 19 / 38 (50.0%) | 86.4% | 3 of 22 flags |
| Peer similarity only | 12 / 38 (31.6%) | 92.3% | 1 of 13 flags |
| Standalone AI detector only | 24 / 38 (63.2%) | 77.4% | 7 of 31 flags |
| Layered workflow (all four layers) | 31 / 38 (81.6%) | 93.9% | 2 of 33 flags |
Precision improved because the layered workflow required agreement between at least two independent signal types — for example, an AI probability above 0.75 plus a peer similarity match to another AI-generated variant, or an AI probability above 0.85 plus a web-source match to a known tutorial prompt. The two false positives in the layered workflow were both students with unusual coding styles: one who had learned Python from algorithmic trading tutorials and another who wrote aggressively comment-dense code as a personal habit.
That is the realistic tradeoff. Even a good detector produces false positives. The correct institutional response is not to lower the threshold until all flags are noise; it is to treat the detector as a decision-support tool and require a human review step.
Assignment and Rubric Design That Reduces AI Substitutability
The other half of detection is making AI-assisted submissions harder to produce convincingly in the first place. An assignment that asks for a single `.py` file with no process evidence is nearly impossible to police. Add these three elements and the detection problem shrinks dramatically:
1. Require iterative process artifacts
Ask students to submit a short design log, a Git repository with at least three commits, or a screenshot of their IDE's local history. AI-generated code often appears in a single commit with no earlier scaffolding. A student who writes code, even badly, leaves a trail of abandoned attempts and incremental fixes. A student who pastes ChatGPT output has one clean, linear commit. Rubrics can award 5–10% for evidence of iteration, which gives honest students a measurable advantage over prompt-based submission.
2. Add an in-class component
No detection tool can replace sitting down with a student and asking them to extend their own code. In our audit, nine students who were flagged by the layered workflow were asked in a 10-minute oral to add a simple input validation check to their palindrome function. Seven could not do it without looking up basic syntax. Two could, and those two were eventually cleared because they could explain their design choices coherently. This is not a punishment; it is a formative assessment under a different condition.
3. Design for traceability in the prompt itself
Prompts that ask for a single well-known algorithm ("check if a