Eleven names came up on the first scan. The department chair had them printed out and a Friday deadline for deciding which ones went to the honor court. I was the outside consultant, which on that campus meant I could say uncomfortable things without losing tenure.
I told her to throw the printout away and start over. Not because the scan was wrong. Because a score isn't a decision, and eleven names is what a department gets when it treats suspected AI-generated code in student submissions as a verdict.
I've sat on both sides of this. As a startup CTO I had to verify that contractor code was actually theirs before I signed a six-figure invoice. Now I get called in when a department's numbers spike and nobody knows what happens next. The process below is what I hand them. Five steps, and the first three exist mainly to stop you from sanctioning the wrong student.
Step one: build a cohort baseline before you look at any name
The most common mistake is running a check and reading it top to bottom. Peer similarity is a relative number. It only means something against the distribution it came out of.
Here's what blows up baselines: you hand out starter code. If 90 students begin from the same Node.java or the same graph.py, they share hundreds of tokens before anyone types a line of their own. I've seen CS1 cohorts sit at a 68% median peer similarity for that reason alone. In that class, an 85% match is unremarkable. A 97% match is worth twenty minutes of your time.
Run the whole cohort, open the distribution, and only then sort. Outlier ranking does that faster than reading rows.
One operational thing nobody warns you about. The cohort is a moving target. In fall 2023 a course we were supporting ran its first pass on a Monday. The late-add window closed that Wednesday and nobody re-ran the checks. Twenty-two submissions got scored against a group of sixty-one when the real cohort was seventy-four, and the top match in that batch turned out to be two students who had both copied the same instructor example from week two. Three hours of a TA's week, gone. Put the re-run on the calendar at the same time you build the assignment.
Step two: know which engine produced the number
Three different scores, three different error models. Faculty conflate them constantly, and that's where most of the trouble starts.
Peer similarity is essentially deterministic. When two files match at 91%, you can open both and read the overlap. The engine can be wrong about whether that overlap counts as misconduct. It can't be wrong about the overlap existing. Adjudication is a human judgment applied to evidence you can see.
Web similarity is evidence too, with one sharp edge. A student who lifted code from a GitHub repo and a student who copied from a tutorial they cited in a comment look identical to the scanner. Attribution has to be in your rubric before the scan runs, or you'll punish the good habit.
AI generation is probabilistic and always will be. There's no source file to compare against. The text-detector tricks (perplexity, burstiness) translate poorly to code because code is low entropy by construction. A function that loops over a list looks statistically like every other function that loops over a list. What you get from an AI code detector is a per-file probability plus a list of written indicators. Not a finding.
A similarity score is a lead. An AI score is a prior. Neither one is a finding.
Which is why I want them on one screen. Three dashboards means three interpretations by three different people, none of whom talked to each other.
Step three: read the file for artifacts a human leaves behind
This is old-fashioned and it still works. Below is the top of a submission I looked at last year. CS1, Python, second assignment. The student had been in the course for five weeks.
from typing import List
def calculate_average(numbers: List[float]) -> float:
"""
Calculate the average of a list of numbers.
Args:
numbers: A list of numeric values.
Returns:
The arithmetic mean as a float.
"""
if not isinstance(numbers, list):
raise TypeError("Input must be a list")
if len(numbers) == 0:
return 0.0
total = 0.0
for num in numbers:
if not isinstance(num, (int, float)):
raise ValueError("Every element must be numeric")
total += num
return total / len(numbers)
Count the tells. Full type hints five weeks into a course that hasn't mentioned the typing module. A docstring that restates the function name and then documents arguments nobody asked about. Type guards raising TypeError and ValueError on inputs the spec never discussed. An accumulator loop where a real beginner writes sum(numbers) / len(numbers) and moves on. Every branch handled, including the empty list, which is exactly the case beginners forget.
The human version of this function usually crashes on an empty list. It might not have a docstring at all. It probably has a leftover print() from debugging, a variable named total2, or a comment that says "not sure why this works."
I'll be straight about the limits. The tells have gotten quieter. The cheerful "Here's how you can do that:" comments that showed up in 2023 submissions are mostly gone, and the models stopped bolting a README onto every answer around the same time. Artifacts are supporting evidence now, not proof. A clean-looking file is neutral, not exculpatory.
Step four: the ten-minute code walkthrough
Nothing in this process is more reliable, and it costs nothing.
Sit down with the student and their own file open. Don't ask what the code does. Ask why. Why is the isinstance check there? What happens if the caller passes a tuple? Which line would you delete first if I told you to simplify this?
Students who shared code with a friend usually survive this. They read the file, they remember the assignment, and they can stumble through the logic because they at least wrote part of it. Students who pasted a model's output tend to go quiet exactly at the defensive guards they never asked for. They can describe what the function does. They can't tell you why it's shaped that way.
The process detail matters as much as the technique. Announce at the start of term that walkthroughs happen, and run them on a random sample, not only on the flagged. A walkthrough that only happens to accused students is an interrogation, and students will treat it that way, correctly.
Step five: escalate on convergence, never on one score
My rule is two independent signals minimum before anything enters a formal process. Peer overlap plus a failed walkthrough. A web match plus uncommitted history on a project that required commits. An AI score above your published threshold, plus artifacts, plus a student who can't explain a line.
Publish the threshold before the semester starts. Not because students will read it carefully, but because it forces your department to agree on a number in September instead of arguing about one in April. The number can be conservative. A conservative threshold everyone honors beats a sharp one that gets challenged and quietly withdrawn.
Keep the evidence file. Diff, scores, walkthrough notes, date. Every case I've watched fall apart collapsed on documentation, not on detection.
Where AI code detection tools actually help
None of the above is an argument against scanners. It's an argument against reading one number as a verdict. Get all three signals in a single pass and the first four steps take an afternoon instead of a week.
That's the case for Codequiry in a department: peer similarity, web and GitHub matches, and AI generation scored per file in one run across 65 languages, with a dashboard a TA can actually operate at 11pm. The engines handle the parts machines are good at, including token and AST comparison that survives renaming and reformatting, cohort-wide outlier ranking, and tracing a snippet back to a GitHub repo or a Stack Overflow answer. The judgment stays with the person who knows the assignment.
If you're weighing Codequiry versus MOSS, the practical difference is that MOSS has been free since 1994 for good reasons and still emails you a URL to an HTML page with no AI scoring, no web corpus, and no interface your adjunct wants to learn. Same story with JPlag and Dolos: solid research lineage, thin operations. That tradeoff used to be obvious. It isn't anymore.
Run your first check on a course you already know well and compare what the scanner flags against who you'd have flagged on instinct. Do that once before you run it on anything that matters. You'll learn more about a tool's behavior in one pass than from any vendor accuracy chart, and you'll stop being surprised by the scores.
Most departments I've worked with end up with a written five-step process, a published threshold, and a TA who can run a code plagiarism checker for teachers without escalating anything on their own authority. That's the whole bar. Everything past it is paperwork nobody reads.
Start with one cohort and a code plagiarism check on an assignment from last term. The scores will surprise you less than the process will.