Set your review threshold from the distribution of a clean cohort, not from a number someone posted on a forum. Pull the percentiles first, flag the top few percent for human review, and treat anything above 85% peer similarity as a claim that needs an explanation rather than a verdict. The number decides who gets looked at. The evidence decides what happens next.
In spring 2022 I sat in on an honor council hearing at a mid-sized state university. A student had been accused on a 71% MOSS match on a Java assignment. The professor could not explain what 71% measured. The student's advisor asked whether 71 was high. Nobody in the room could answer, because nobody had ever calibrated the number against that course.
The student was exonerated on a technicality about submission timestamps. He may well have copied. The point is that the tool worked and the process failed. That hearing cost the department three weeks and one very annoyed associate dean.
What does a code similarity percentage actually measure?
A similarity score is a ratio of matched structure, not a probability of misconduct. Two students who independently write a bubble sort will score 85% or higher because there is only one way to write a bubble sort. A shared starter file pushes the whole cohort's baseline up by 30 points. Neither of those is cheating.
A similarity score is not a finding. It is a queue position.
That distinction is the whole game. Tools like MOSS, JPlag 5.x, and Dolos have done the mechanical part well for years. What they have never done is tell you which of the 240 submissions you should open first, or what to say when you do.
Step 1: Separate the three numbers before you look at any of them
You are measuring three different things and they need three different thresholds. Peer similarity asks whether a student copied from another student in the same cohort. Web similarity asks whether they pulled from GitHub, Stack Overflow, or a tutorial. AI generation asks whether a language model produced the file. A single blended score destroys all of that information.
This is the main reason I stopped defaulting to MOSS for anything I have to defend later. MOSS gives you one number per pair and a diff page, and it is genuinely excellent at that. But it has no concept of the open web as a source and no concept of AI authorship, so you end up stitching three unrelated reports into one narrative by hand.
A code plagiarism checker that reports peer, web, and AI scores as separate axes lets you say something precise in a hearing: "Your peer score was 94. Your web score was 2. Your AI score was 0.11. We are here about the first number only." That sentence is worth more than any report you can print.
Step 2: Lock your normalization settings and never change them mid-term
Every similarity engine has a normalization layer sitting between the raw file and the comparison. Comments stripped or not. Whitespace collapsed or not. Identifiers normalized or not. String and numeric literals masked or not. Each toggle moves scores by 10 to 25 points on typical undergraduate code.
Pick a configuration in week one of the term and write it into the syllabus. If you change it in week nine, every comparison you made in week three becomes indefensible.
Here is what the toggles actually cost you. Take this working method:
public static int sumEven(int[] nums) {
int total = 0;
for (int i = 0; i < nums.length; i++) {
if (nums[i] % 2 == 0) {
total += nums[i];
}
}
return total;
}Now the version a student hands in after fifteen minutes with a rename tool:
public static int evenTotal(int[] values) {
int acc = 0;
for (int idx = 0; idx < values.length; idx++) {
if (values[idx] % 2 == 0) {
acc += values[idx];
}
}
return acc;
}With identifier normalization off, that pair scores in the low 80s on most token engines, which is enough to flag but easy to argue about. With identifier normalization on, it is a 100% match. Renaming variables is not a technique. It is a typo with extra steps.
Structural refactoring is the real escape hatch, and it is worth being honest about it. Rewrite the loop as a stream:
public static int evenTotal(int[] values) {
return Arrays.stream(values).filter(v -> v % 2 == 0).sum();
}That drops below 40% on both JPlag 5.x and Dolos in my spot checks, and I do not think any tool should pretend otherwise. A genuine structural rewrite looks like a genuine structural rewrite. What you can catch is the student who rewrote the loop but kept the original's dead code, kept its odd edge-case guard, kept its exact comment spacing. Those survivors are the tell.
Mask literals if you have hardcoded test values in the assignment. Students copy the professor's test cases, and those get flagged as inter-student matches constantly. Masking string and numeric constants cuts that noise by roughly a third in my experience.
Step 3: Run a calibration pass on a cohort you trust
Before you accuse anyone, you need to know what normal looks like in this specific course. Two ways to get there.
The cheap way is to run the checker against last term's submissions where you already know the outcome. The better way, if you have no history, is to run the current cohort and look at the shape of the distribution before you look at the top match. Read the median first. Read the 90th percentile second. Only then scroll to the top of the list.
Numbers from a 240-student Intro to Java course I worked with last year:
| Statistic | Peer similarity | Web similarity |
|---|---|---|
| Median | 11% | 0% |
| 90th percentile | 34% | 6% |
| 95th percentile | 41% | 18% |
| Highest match | 94% | 100% |
A median of 11% tells you the starter code was light and the assignment had real design freedom. Now look at a 30-person senior seminar with a 200-line provided skeleton. Median peer similarity there was 62%. The same 85% threshold means completely different things in those two rooms.
Step 4: Set thresholds relative to the cohort, not against an absolute bar
Absolute thresholds are a trap. Cohort-relative thresholds travel. Here is the tiering I use, and it is deliberately conservative at the bottom so you do not waste a weekend on false positives.
| Tier | Peer score | What it usually means | Action |
|---|---|---|---|
| Certain review | 95 to 100 | Identical file or a shared private repo | Pull the diff today |
| Likely review | 80 to 94 | Copy with cosmetic edits, or a shared solution skeleton | Human review within 48 hours |
| Triage | 60 to 79 | Common idioms, shared starter, or a real match | Compare against the cohort median first |
| Ignore unless | Below cohort p95 | Normal overlap | Skip |
The triage band is where careers get damaged by carelessness. If your cohort median is 11%, a 68% pair is worth twenty minutes of your time. If your cohort median is 62%, a 68% pair is background noise and you should not open it.
This is the piece a raw MOSS run cannot give you, and it is the piece that decides whether your semester is calm or a mess of appeals. A tool that ranks submissions by how far they sit outside the cohort's own distribution is doing the calibration math for you. Codequiry's smart review queue works this way, and it is the single feature that changed how I run these checks.
Step 5: Build the evidence packet while the check is still fresh
Do this on the day you run the check, not in November when the hearing is scheduled. Evidence decays. Students delete repos, submissions get resubmitted, and your memory of why a pair looked suspicious is gone in six weeks.
For each flagged pair, capture four things and put them in one PDF.
- The synced side-by-side diff. Not the raw files. The diff, with the matched regions highlighted line by line. A committee can read a diff. They cannot read two 300-line Java files and reach a conclusion.
- The match metadata. Token or node counts, matched regions, and the score under each engine configuration you ran.
- Both submission timestamps, to the minute. This is what saved the student in that 2022 hearing, and it is what will save you from accusing the wrong person next time. If you can see which file landed first, you can usually tell who copied from whom.
- The web match source, if any. The actual URL, the repository name, the date the repo was created. "It looked like it came from GitHub" is not evidence.
If you are running this at scale, pull the results through the API into your own store rather than screenshotting a web page. Something like this, roughly:
import requests, statistics
BASE = "https://api.codequiry.com/v1"
HDRS = {"Authorization": f"Bearer {API_KEY}"}
check = requests.get(f"{BASE}/checks/{check_id}/results",
headers=HDRS, timeout=30).json()
peer = [m["peer_score"] for m in check["matches"] if m["peer_score"] is not None]
print("n =", len(peer))
print("median =", round(statistics.median(peer), 1))
print("p90 =", round(statistics.quantiles(peer, n=10)[8], 1))
print("p95 =", round(statistics.quantiles(peer, n=20)[18], 1))The endpoint paths and field names above come from the docs and drift between versions, so check them before you paste this into anything. The shape of the response is not going to surprise you. Run the same percentile math for web and AI scores separately, and store all three per submission.
Step 6: Treat AI detection as a separate axis with its own threshold
An AI score is per file, not per submission, and averaging it across a ten-file project hides everything interesting. A student who wrote nine files and generated the tenth will land at 0.12 on the submission average and 0.94 on the file that mattered.
Always read the highest single file score, then look at the written indicators for that file. What pushes a file's score up is real: uniform comment density on code that does not need comments, defensive null checks on parameters the assignment guarantees are non-null, a docstring on every method including the private helpers, imports the code never uses, and a suspiciously tidy handling of an edge case that the starter never hinted at.
What pushes it up falsely is more common than vendors admit. Lombok-generated boilerplate. Textbook-standard getters and setters. Files written in the style of the professor's own reference solution, which is exactly how a good student learns to write. Files that went through an IDE formatter before submission.
Set your AI threshold at whatever your false positive tolerance is, and then never use an AI score as the sole basis for an accusation. Pair it with something human: an interview question about the file, a request to explain a specific design decision, a comparison against the student's earlier work. An AI code detector tells you where to look. It does not tell you what to conclude.
Step 7: Write the policy down before the hearing
Everything above is worth nothing if the student finds out about your thresholds for the first time in an email accusing them of misconduct. Put four things in the syllabus in week one: the tool you use, the three scores you collect, the tier that triggers a human review, and the appeals path.
Two sentences is enough. "Submitted code is checked against peer submissions, public repositories, and AI generation. Peer similarity above 80% triggers instructor review. You will be contacted before any finding is recorded." That is a policy a committee can read and a student can understand.
One detail nobody warns you about: the registrar's roster export usually arrives with a UTF-8 byte order mark on the first line, and half your student IDs come through with a leading invisible character. I lost two weeks to that in 2023 and the threshold policy slipped to week three. Sanitize the CSV before you import anything.
Most institutions run a code plagiarism checker for teachers without ever writing the sentence above, and then wonder why every hearing turns into a debate about the tool. The tool is the least interesting part of the conversation. The calibration and the paper trail are the whole thing.
If you want a fair look at how the engines actually differ under the settings I described, the Codequiry vs MOSS comparison is the honest place to start.
Frequently Asked Questions
What peer similarity score counts as code plagiarism?
There is no universal number, which is why you calibrate against your own cohort. In practice, 95% or above is almost always a real match, 80 to 94% warrants a human review, and anything below your cohort's 95th percentile is normal overlap. A single absolute threshold applied across courses and assignments will generate false positives in small or heavily scaffolded courses.
Does renaming variables defeat code plagiarism detection?
No. Token-based and AST-based engines normalize identifiers before comparison, so a rename-only edit scores 100%. What does move a score down meaningfully is a genuine structural rewrite, such as replacing an explicit loop with a stream pipeline or a recursive helper. Even then, surviving markers like dead code, unusual edge-case handling, and comment placement usually keep the pair on the review list.
Should an AI detection score be used as evidence on its own?
No. AI detection is probabilistic and it produces false positives on boilerplate, generated getters and setters, and code written in a textbook style. Use the score to pick which files deserve a conversation, then confirm with an interview or a request to explain a specific design decision. Treat the score as a lead, not a conclusion.
How large does a cohort need to be for relative thresholds to work?
Thirty submissions is enough to get a usable median and a rough 90th percentile. Below twenty, percentiles get noisy enough that I would fall back to reviewing every pair above 70% by hand. Above a hundred, the distribution stabilizes and the tuning becomes mostly about masking the starter code.
If you want to see the review queue and the evidence packet on your own submissions before you commit to a policy, run a code plagiarism check on one assignment from last term and look at the median first.