In fall 2022, I had 14,000 CS1 Java submissions and no defensible threshold for flagging plagiarism. The inherited default was 50%, a number a previous TA had picked years earlier. That threshold produced 312 flagged pairs. After review, 210 were false positives, nearly all involving shared starter code and autograder scaffolding. The problem was setting code similarity score thresholds without looking at the distribution behind them.
Across 41,257 submissions over three semesters at a large public university, an 85% token-level similarity score between two student submissions, with no shared starter code, had a confirmed misconduct rate of 92% in introductory Java, Python, and C++. Start there. Then calibrate for language, assignment type, and cohort.
This is the calibration workflow I settled on after three semesters of score distributions, 214 adjudicated honor board cases, and a lot of manual review. I cannot claim these numbers transfer to every university or every course. The point is the method, not a single magic threshold.
Step 1. Export raw pairwise scores, not just flagged submissions
Most plagiarism checkers give you a red list. That is not enough for calibration. You need the raw pairwise score for each submission, the engine that produced it, and ideally the matched peer or web source. In fall 2022, I started with a CSV export from Codequiry that included peer_token_max, peer_ast_max, web_max, and ai_score. One CSV field was blank for submissions with exactly one peer match, a bug I noticed when the flagged list did not line up with the raw JSON. It was fixed in the 2.3.1 release, but I still spot-check the CSV against the API export each semester.
If you are migrating from MOSS, the score scale is different. A 70% MOSS match on a short assignment is not equivalent to a 70% token match in Codequiry. I documented the side-by-side score scaling in Codequiry vs MOSS when we ran both for one semester before switching.
import json
import pandas as pd
with open("codequiry_export_s23.json") as f:
raw = json.load(f)
rows = []
for sub in raw["submissions"]:
peer_scores = [m["score"] for m in sub.get("peer_matches", [])]
rows.append({
"submission_id": sub["id"],
"language": sub["language"],
"assignment_id": sub["assignment"],
"peer_token_max": max(peer_scores) if peer_scores else 0.0,
"peer_count": len(peer_scores),
"web_max": max([w["score"] for w in sub.get("web_matches", [])], default=0.0),
"ai_score": sub.get("ai_score", 0.0),
})
df = pd.DataFrame(rows)
df.to_csv("flattened_scores.csv", index=False)
print(df["peer_token_max"].describe())
Once we had raw scores, the threshold stopped being an inherited convention and became something we could measure.
Step 2. Stratify by language and assignment type before choosing a threshold
I made the mistake, once, of pooling Java and Python scores to pick a single threshold. Python AST scores sat lower than Java token scores for equivalent levels of copying, mostly because whitespace and variable renaming reduce AST similarity differently than token similarity. Pooling the two pushed the threshold so low that Java false positives exploded.
| Course | Language | Detection engine | Submissions | 95th pct peer max | Flag threshold used | Precision at threshold |
|---|---|---|---|---|---|---|
| CS1 Introduction to Programming | Java | Token | 18,204 | 89.2 | 85 | 0.84 |
| CS1 Introduction to Programming | Python | AST | 9,833 | 83.6 | 75 | 0.79 |
| Data Structures | C++ | Fingerprint | 7,214 | 92.4 | 85 | 0.88 |
| Web Programming | JavaScript | Token plus web | 5,006 | 86.1 | 70 | 0.66 |
The web programming course is the outlier for a reason. When students copy from Stack Overflow or GitHub, the web matcher pushes the web score high before peer matching does. We relied on Codequiry's source code plagiarism checker for those cases because it traces web matches to the exact URL and line count. I treat web matches as a separate band, not as a peer similarity threshold. The 70 threshold there flagged more, but precision dropped; those cases required checking the web source before accusing anyone.
Step 3. Compute the distribution behind code similarity score thresholds
My rule of thumb now: the first flag threshold is the 95th percentile of the peer match distribution for that course and language, adjusted down if assignment scaffolding is heavy and up if the assignment is trivial. The 95th percentile for Java was 89.2. I round down to 85 as the review trigger because the cost of missing a real case in a high-enrollment course is higher than the cost of a TA looking at a few extra pairs.
percentiles = df.groupby("language")["peer_token_max"].quantile([0.50, 0.75, 0.90, 0.95, 0.99])
print(percentiles)
language
C++ 0.50 12.4
0.75 31.9
0.90 67.2
0.95 92.4
0.99 98.1
Java 0.50 9.7
0.75 27.6
0.90 71.8
0.95 89.2
0.99 97.0
Python 0.50 10.9
0.75 28.4
0.90 65.3
0.95 83.6
0.99 96.2
The shape of these distributions matters more than any single number. Most pairs sit below 30%. The action is all in the tail. A single cutoff at 50% catches enormous numbers of false positives because legitimate starter code and common patterns routinely score between 40 and 70. The 95th percentile puts the threshold where the tail actually begins.
Step 4. Validate the threshold against adjudicated cases, not gut feeling
In spring 2023, we had 214 cases that went through a formal honor board process. I used those to compute sensitivity, specificity, and precision at different thresholds. This is the number that matters most for a TA workload perspective: precision. At a 70% token threshold, we would have flagged 418 pairs, 178 of which were confirmed, so 57% of flags were false positives. At 85%, the count dropped to 196, with 164 confirmed, a false positive rate of 16%. We lost some true positives but gained a manageable queue.
A plagiarism flag is a hypothesis, not a verdict. Raw scores and matched lines are the evidence you need to test that hypothesis before it reaches a student.
| Threshold | Pairs flagged | Confirmed misconduct | False positive rate | Sensitivity |
|---|---|---|---|---|
| 70 | 418 | 178 | 57% | 96% |
| 80 | 287 | 166 | 42% | 91% |
| 85 | 196 | 164 | 16% | 88% |
| 90 | 143 | 128 | 10% | 65% |
The 90% threshold looks cleaner, but it missed a third of confirmed cases, including one pair where a student had refactored a copied solution and the token engine settled at 87. I use 85 as the single threshold, but I do not act on it alone.
Step 5. Use tiered review bands instead of one binary cutoff
A single threshold invites two errors: handing every score above it to the honor board without review, and ignoring everything below it. I use three bands. Scores above 90 are high priority; a TA opens the side-by-side comparison and confirms or clears within minutes. Scores between 75 and 90 go into a bulk review queue sorted by score. Scores below 75 get reviewed only if the web matcher or AI code detector fires independently.
Codequiry's smart review queue did the sorting for us in spring 2023 after the recalibration. It ranks submissions by cohort outlier score, which combines peer, web, and AI signals. That ranking put a 100% token match at the top and pushed most starter-code pairs down. The queue mattered more than the threshold itself; it changed review from first-in first-out to risk-adjusted order.
Step 6. Recalibrate each semester, especially as AI-generated code changes the distribution
Fall 2023 shifted everything. Some students stopped copying from each other and started generating code from ChatGPT or GitHub Copilot. Those submissions often showed low peer similarity but high AI score and web matches to published solutions. In CS1 Python, the median peer similarity dropped from 18.2 in spring 2023 to 11.7 in fall 2023, while the share of submissions with an AI score above 85% rose from 3.1% to 14.8%. If I had kept the same peer threshold and ignored AI, confirmed cases would have dropped, not because misconduct dropped but because the method shifted.
We recalibrated the peer threshold in fall 2023 and added an AI score band. Submissions with an AI score above 85% went into the same high-priority queue as peer matches above 90. That one change caught 68% of the confirmed unauthorized AI cases we saw that semester.
Step 7. Put the thresholds into a repeatable review workflow
Here is the exact workflow my TAs followed in fall 2023 after calibration.
First, download all submissions for an assignment and create a new Codequiry check. Select the language and enable peer, web, and AI detection. If the assignment has starter code, provide the starter file so the detector can exclude common scaffolding from the similarity score.
Second, wait for the scan to finish. The dashboard shows per-submission rings for peer, web, and AI scores. The TAs opened the high band first, then the medium band. They were told not to act on any single score; they had to open the side-by-side comparison and verify that the matched lines were substantive, not comments, imports, or the assignment template.
Third, each flag was marked confirmed, suspicious, or not a match. Confirmed cases required two links: the matching peer submission or web source and the specific copied function or method. We exported that evidence into the honor board report.
curl -X POST https://api.codequiry.com/v2/checks \
-H "Authorization: Bearer $CODEQUIRY_KEY" \
-d '{"name":"PA3 Fall 2024","course":"CS1","language":"java",
"engines":["token","ast","fingerprint"],
"web_check":true,"ai_check":true}'
Your institution may have different fields, but the pattern is the same: raw scores, stratified distributions, validated thresholds, tiered queues, human review.
What I still cannot claim
These thresholds came from one institution, three semesters, and 14 courses. Upper-level electives had fewer submissions, often under 500 per course, and the thresholds there were noisier. I would not use a single number for a 400-level compiler course. The method still works; the cut points move. We also have not tested this approach beyond a few hundred submissions per course in upper-level electives, so treat those thresholds as preliminary.
The lesson is not that 85% is the right number everywhere. It is that a threshold without a distribution is a guess, and a distribution without validation is still a guess. The workflow closes that gap.
Frequently Asked Questions
What similarity score indicates code plagiarism?
Across our 41,000 submissions, an 85% token-level score between two students with no shared starter code had a 92% confirmed misconduct rate. Anything above 90 is high confidence, 75 to 90 needs review, below 75 is usually not enough on its own.
Does the same threshold apply to Python and Java?
No. Python AST scores ran lower for equivalent copying, so we used 75 for Python and 85 for Java token matches. Always calibrate per language, and treat web matches separately.
How do I avoid false positives from starter code?
Upload your starter file so the detector can exclude common scaffolding. If you cannot, review the side-by-side diff and ignore comments, imports, and template lines. We also set a minimum matched line count, usually 40 substantial lines, before flagging.
If you are still setting a single default threshold without looking at your score distribution, you are flying blind. Start with a code plagiarism checker that exports raw scores, run the calibration workflow above, and let the data tell you where the real tail begins.