Last fall I flagged 43 files out of 1,847 as likely machine-written. Eleven of those flags belonged to students who had never opened a chat window in their lives. They had read the style guide closely, written full Javadoc on every method, and defended against null inputs they had no reason to expect. The detector was not malfunctioning. It was doing precisely what it was built to do, and that was the problem.
The practical answer to how you set an AI-generated code detection threshold is that you do not pick one number. You build a human baseline for the assignment, score individual files rather than whole submissions, and combine a per-file cutoff with a submission-level rule. What follows is the calibration I now run every August, with numbers drawn from four courses across three semesters.
What an AI Score Actually Measures in Source Code
Prose detectors work on perplexity, which is a measure of how surprised a language model is by each next word, and burstiness, which is the variance in sentence length across a document. Code detectors borrow both ideas and reapply them at the token level. Perplexity becomes a question of how predictable each identifier, operator, and comment token is given everything before it. Burstiness becomes the variance in line length, identifier length, comment density, and structural repetition across a file.
A large language model samples the highest-probability continuation. That produces code shaped like the statistical center of its training distribution: conventional names, defensive validation on every public method, a doc comment above everything, and no domain quirks at all. A student who has spent three weeks fighting a specific bug writes code with fingerprints in it, an oddly named helper, a hardcoded constant that should be a parameter, a comment that says "this seemed to work." The model writes none of that.
A detector does not find artificial intelligence. It finds text that is statistically unsurprising. Everything difficult about academic use of these tools follows from that one sentence.
Step 1: Build a Human Baseline Before You Flag Anyone
Before I trust any score, I need to know what my own students' code scores on my own assignments. Two sources work well. The first is the course archive from before late 2022, when ChatGPT and Copilot were not in the room. The second, if the course changed too much, is a set of in-class, offline exercises where the code was written under supervision.
I collect roughly 200 files per course, run them through the AI detector, and compute the distribution. On my own archive, human-written Java and Python code has a median AI score around 19, a 95th percentile of 34, and a 99th percentile of 41. Your numbers will differ. The shape matters more than the values: a long right tail on known-human code tells you where the cutoff has to sit.
Export the per-file results as CSV and do the arithmetic yourself. It takes four minutes and it is the only way to know your own distribution rather than someone else's.
import csv, statistics
rows = list(csv.DictReader(open("fall2024_ai_scores.csv")))
scores = [float(r["ai_score"]) for r in rows]
print(len(scores), "files scored")
print("median:", round(statistics.median(scores), 1))
print("mean: ", round(statistics.fmean(scores), 1))
q = statistics.quantiles(scores, n=20)
print("p95:", round(q[18], 1))
print("p99:", round(q[19], 1))
for cutoff in (30, 45, 60):
flagged = sum(1 for s in scores if s >= cutoff)
print(f"{cutoff}: {flagged} files ({flagged/len(scores):.1%})")
One warning from experience. If your known-human median lands above 30, either the detector is misconfigured for the language or the assignment is too template-like to distinguish anything. An assignment where every student writes the same five-line main method and the same getter in a starter file is not evidence of anything. I rebuilt two assignments after seeing a baseline median of 33 on the 2022 archive.
Step 2: Score Files, Not Submissions
The single most common configuration mistake I see in colleagues' setups is reading the submission-level score and stopping there. A ten-file project with one heavily generated utility file and nine genuinely written ones will average out to something unremarkable. The submission average hides the signal; the file-level distribution reveals it.
This is where a tool that reports per-file AI versus human probability earns its keep. Codequiry's AI code detector reports the average across a submission, the highest single-file score, and a risk distribution, which is the triple I actually need. My own workflow is to sort every submission in a cohort by highest single-file score and read down the list until the scores stop moving.
Here is the kind of file that sits at the top of that list without any AI involvement at all. It is a first-year Java submission, written by hand, by a careful student:
public class GradeCalculator {
/**
* Calculates the weighted final grade for a student.
*
* @param assignments the list of assignment scores
* @param weights the corresponding weights
* @return the weighted final grade
*/
public static double calculateFinalGrade(List<Double> assignments,
List<Double> weights) {
if (assignments == null || weights == null) {
throw new IllegalArgumentException("Inputs must not be null");
}
if (assignments.size() != weights.size()) {
throw new IllegalArgumentException("Size mismatch");
}
double total = 0.0;
for (int i = 0; i < assignments.size(); i++) {
total += assignments.get(i) * weights.get(i);
}
return total;
}
}
Every line is conventional. There is full Javadoc, null validation nobody required, a symmetric loop, no unused variables, and no personality. GitHub Copilot produces this. So does a sophomore who internalized the department's style guide and reads the API documentation for pleasure. I have had both in my office in the same week and the code looked identical.
Step 3: Set Two Thresholds and One Combining Rule
To pick numbers honestly, you need a labeled set. In spring 2024 I generated 60 files with GPT-4o, Claude 3.5 Sonnet, and Copilot using the exact assignment prompts, then hand-labeled 120 files from students I had watched write the code in lab. That 180-file set is small, and I would not claim it generalizes past my own four courses, but it was enough to see what each cutoff costs.
| Per-file threshold | Files flagged (of 1,847) | Recall on labeled set | Precision on labeled set |
|---|---|---|---|
| 30 | 214 | 0.97 | 0.52 |
| 45 | 79 | 0.88 | 0.81 |
| 60 | 31 | 0.61 | 0.94 |
A cutoff of 30 catches nearly everything and accuses half the room. A cutoff of 60 is defensible in front of an honor board and misses four out of ten. I settled at 45 as the review trigger, not the verdict, and then layered a submission rule on top of it: flag the submission if any single file scores 60 or above, or if two or more files score 45 or above, or if the submission mean reaches 35.
That rule is crude and it works. It cut my review queue from 214 files to 61 submissions in the fall term, and the 61 contained 14 of the 15 cases I eventually confirmed. The one it missed was a student who generated an entire assignment in a single paste and then rewrote variable names by hand, which flattened the per-file distribution without changing the underlying structure.
Step 4: Stack AI Scores With Peer and Web Similarity
An AI score alone is a reason to look, never a reason to conclude. The strongest signal in my experience is agreement between independent detectors looking at different things. Peer similarity asks whether this code resembles another student's code. Web matching asks whether it resembles something published. AI scoring asks whether it resembles the statistical center of machine-generated code. Those are three different questions, and a submission that trips two of them is a different kind of animal from one that trips a single test.
In practice, here is how I read the combinations. High AI score plus high peer similarity usually means one student shared generated code with another, and both are in trouble for different reasons. High AI score plus a web match to a GitHub repository or a Stack Overflow answer means the student copied something a person once wrote, and the AI score is a red herring. High AI score alone, with distinctive domain-specific names and a plausible commit history, is the case that deserves a conversation and nothing more.
For the peer and web layers I use the same platform, because running three separate tools and reconciling three report formats costs me a full day per assignment. Codequiry handles peer comparison across a cohort, web and GitHub matching, and AI scoring in one pass, and its code plagiarism checker for teachers was written for the way a course actually runs, with rosters, deadlines, and a review queue rather than a command line and a directory of HTML files.
MOSS still exists and still works. It is free, it has been in service since 1994, and its winnowing algorithm from the 2003 Schleimer, Wilkerson, and Aiken paper is genuinely elegant. It also cannot tell you anything about AI generation, its output is a set of static HTML pages, and it has no notion of a roster. JPlag 5.0 and Dolos both do competent token-level similarity across many languages. Neither ships an AI model. If peer similarity is your only question, all three remain reasonable. If you are trying to answer three questions at once, you need a tool that answers three questions.
Step 5: Interview Before You Write Anything Down
I have sat on two academic-integrity committees. The hearings that go badly are the ones where a faculty member arrives with a number and nothing else. A percentage on a printout is not evidence; it is a prompt for a conversation you should have had three weeks earlier.
My interview script has three parts. First, I ask the student to explain one specific line I chose in advance, usually a line that is correct but non-obvious. Second, I ask them to modify the code in front of me, removing a feature or handling a new input. Third, I ask why they chose a particular data structure. A student who wrote the code can do all three badly and still do them. A student who did not write it stalls at the first question in a very specific way, describing what the line does syntactically without any account of why it is there.
The docstring case from last fall is worth recording. A second-year student ran Copilot to generate Javadoc comments on code she had written herself, accepted every suggestion without reading closely, and ended up with a file whose comment-to-code ratio was triple the class norm. Her code was hers. Her comments were not. That is a real conversation about attribution, and it is not the same conversation as submitting generated work, and a detector cannot tell the two apart.
Step 6: Put the Numbers in the Syllabus
Whatever you decide, write it down before the term starts. Students who learn the rules in week nine feel ambushed, and they are right to. The paragraph I now use reads, in part: "Assignments are scanned for similarity to other submissions, to public sources, and for statistical signals of machine generation. A high score is not a finding of misconduct; it triggers a review, which may include a short oral explanation of your code. You are permitted to use AI assistants for syntax questions and documentation lookup, and you must disclose any use that produces more than three consecutive lines of submitted code."
That last sentence does more work than the detector. Disclosure rules give honest students a path and give you a clean basis for a hearing when disclosure is absent. The number is the trigger; the policy is the case.
One more piece of housekeeping. Documents the review. Every flag I escalate gets a one-page note with the file, the score, the peer and web results, the date of the interview, and a summary of what the student said. When a case reaches the honor board, that page is the difference between a fifteen-minute hearing and a forty-minute argument about whether the software is reliable. Calibration is not a technical exercise. It is the preparation that lets you defend a judgment you already had to make.
Frequently Asked Questions
Can professors tell if an assignment was written by ChatGPT?
Not by reading alone, and not reliably. The tells that experienced instructors notice, uniform comment density, defensive checks on every parameter, no domain-specific naming, are the same tells produced by a diligent student following a style guide. A detector gives you a starting point and a review queue. The interview gives you the answer.
What is a normal AI score for human-written code?
On my archive, human-written Java and Python sits at a median of roughly 19 with a 95th percentile near 34. Your assignment design will move those numbers substantially. Compute your own baseline from pre-2023 submissions or supervised in-class work before you flag anything.
Does renaming variables defeat AI code detection?
Largely, yes. These detectors read token predictability, and a consistent rename pass makes a file look less machine-typical. That is different from plagiarism detection, which uses token and AST comparison and survives renaming because it compares structure across two documents rather than measuring the probability of one. This is the main reason to run both kinds of check. They fail in opposite directions.
Which languages produce the most reliable scores?
Python and Java are the steadiest in my data. C++ is noisier because header boilerplate and include guards inflate uniformity. JavaScript is the noisiest, since so much of what students write is configuration and event wiring that looks machine-typical by nature. If you teach a web course, raise your threshold and lean harder on peer and web matching.
If you want to see the per-file reporting and the combined peer, web, and AI view described above on your own submissions, start with a small pilot check on one assignment from last term and compare the results against your own reading of the code.