A TA forwarded me a report last spring. Two Java submissions, 91% similar, both from a section on doubly linked lists. The student in office hours had a decent explanation and a bad attitude, which is usually the wrong combination but not proof of anything. I pulled the side-by-side and spent twenty minutes on it. The 91% was real. It was also almost entirely the instructor-provided skeleton plus a shared bug in the remove() method that both students had copied from the same Stack Overflow answer posted in 2016.
That is the whole problem in one anecdote. A code similarity score is a measurement of how much token-level material two files share. It is not a probability that someone cheated. Those are different numbers, they come from different math, and conflating them is how departments end up in appeal hearings they lose.
So let's take the number apart. What is it actually counting, why does renaming variables do almost nothing to it, and at what point does it mean something you can act on?
What the similarity score is actually built from
Every serious code similarity detector runs the same four-stage pipeline. The differences between tools live in the details of stages two and three, and those details are where your false positives come from.
Stage one: normalization
Before anything is compared, the source gets stripped and flattened. Comments go. Whitespace goes. String literals often get replaced with a placeholder. Formatting is irrelevant by design, which is correct: a student who runs their code through a formatter has not changed what the code does, and the detector should not care.
Most tools also do some identifier normalization. Names get mapped to a generic token, or to a type-based token. This is the single most consequential design decision in the whole pipeline, and it is the one you should ask about when you evaluate a tool.
Stage two: tokenization
The normalized source becomes a token stream. Here's a real method from a Data Structures assignment I still use in workshops:
public static int sumPositive(int[] values) {
int total = 0;
for (int i = 0; i < values.length; i++) {
if (values[i] > 0) {
total += values[i];
}
}
return total;
}After normalization, the token stream looks like this:
FUNC INT ID ( INT [ ] ID ) { INT ID = 0 ; FOR ( INT ID = 0 ;
ID < ID . LENGTH ; ID ++ ) { IF ( ID [ ID ] > 0 ) { ID += ID [ ID ] ; } }
RETURN ID ; }Every identifier is now ID. Every integer literal is the same token. What's left is structure and operators, which is exactly what you want to compare.
Stage three: k-grams and winnowing
Comparing full token streams is O(n²) and useless at scale, so tools hash contiguous runs of k tokens and then reduce the hash set. The standard reduction is winnowing, from Schleimer, Wilkerson, and Aiken's 2003 SIGMOD paper, which is also the paper behind MOSS. Pick a window of w consecutive hashes, take the minimum in each window, record it by position. You end up with a fingerprint set that is roughly 2/(w+1) the size of the original, and it guarantees that any shared substring of at least k tokens will produce a shared fingerprint.
import hashlib
K = 9 # k-gram length in tokens
W = 4 # winnowing window
def kgrams(tokens, k=K):
return [tuple(tokens[i:i + k]) for i in range(len(tokens) - k + 1)]
def hashes(grams):
return [int(hashlib.md5(" ".join(g).encode()).hexdigest()[:8], 16)
for g in grams]
def winnow(hs, w=W):
fps = set()
for i in range(len(hs) - w + 1):
window = hs[i:i + w]
m = min(window)
fps.add((hs[i + window.index(m)], i + window.index(m)))
return fps
def score(a, b):
fa, fb = winnow(hashes(kgrams(a))), winnow(hashes(kgrams(b)))
sa = {h for h, _ in fa}
sb = {h for h, _ in fb}
return len(sa & sb) / len(sa | sb) # Jaccard over fingerprintsThat last line is the whole thing. Every 85% you have ever seen in a report is some variant of that ratio, computed over some choice of token unit and some choice of reduction.
Stage four: pairing and scoring
Now you run it pairwise. For a class of 180 students, that's 16,110 pairs. MOSS, JPlag, Dolos, and Codequiry all handle that fine, though the runtime profile varies wildly with language and file size. MOSS historically took minutes; JPlag 5.x on a warm JVM is quicker than people expect.
Why renaming variables barely moves the number
Take the method above and hand it to a student who knows they're being checked. They rename everything.
static int addPositives(int[] nums) {
int acc = 0;
for (int idx = 0; idx < nums.length; idx++) {
if (nums[idx] > 0) {
acc = acc + nums[idx];
}
}
return acc;
}A naive text differ gives this maybe 40% similarity. A token-normalized fingerprint gives it 100%. The identifier map collapses every name to ID, the += versus = ... + difference is a two-token artifact, and the rest is identical. Winnowing at k=9, w=4 will not find a single k-gram that differs.
I ran both through a winnowing implementation identical to the one above. Original versus renamed: Jaccard 1.0. That's not a bug. That's the detector doing precisely what it was built to do, and it's why any tool that brags about catching renamed variables is usually just describing token normalization.
The thing that will get you is the inverse case. In 2019 we shipped a normalization table that mapped list and array to the same type token because, in our defense, they were coming out of the same generic collection inference. Every Java student who used an ArrayList started colliding with every Java student who used a raw array. We caught it in week two of a fall semester, but we flagged 60-something honest submissions first, and I spent a weekend writing apology emails to two instructors. Normalization tables are load-bearing and nobody audits them.
Why restructuring does move it
Renaming is free for a cheater. Restructuring is not, and this is where the score starts carrying information. Convert that same loop to an enhanced for with a guard clause:
static int addPositives(int[] nums) {
int acc = 0;
for (int v : nums) {
if (v <= 0) continue;
acc = acc + v;
}
return acc;
}Token stream now reads:
FUNC INT ID ( INT [ ] ID ) { INT ID = 0 ; FOR ( INT ID :
ID ) { IF ( ID <= 0 ) CONTINUE ; ID = ID + ID ; } RETURN ID ; }Against the original, that came out around 0.55 by Jaccard on the fingerprint set. Not 0.9. Not 0.2. Fifty-five percent, and the reason is that the middle of the method diverged: FOR ( INT ID = 0 ; ID < ID . LENGTH ; ID ++ ) is a fifteen-token run that no longer exists on the other side.
This is where tools genuinely differ. A pure token matcher loses signal on structural rewrites. An AST comparator recovers some of it but has its own blind spots: a ForStmt and an EnhancedForStmt are different node types in the Java grammar, so an AST differ operating on node types will undercount in exactly the case above. Some tools (JPlag 5.x does this with its own token-based structural approach) sidestep the whole taxonomy by comparing token sequences over an abstracted parse rather than over the concrete syntax tree.
The honest framing: no single unit handles both rename and restructure perfectly. Tools that stack token, AST, and fingerprint signals, and report which one fired, give you something a reviewer can actually reason about.
The floor is not zero, and that is the point
Here's what nobody tells a new TA. Two unrelated submissions on the same assignment typically score in the 25% to 45% range. Not 5%. The reason is simple: the assignment dictates the interface. Everyone writes the same method signatures, the same imports, the same null checks, the same public static void main(String[] args).
I've looked at the score distribution for a 180-student intro course enough times to trust the shape. It is bimodal. A fat lump between 20% and 50% made of honest work sharing a skeleton, a long flat middle, and a thin spike above 85%. Almost all the real cases live in the spike. Almost none live in the lump. Any grade-level threshold below about 60% generates noise at a rate that will exhaust you.
The lump also moves with assignment design. If you hand out a 200-line starter file and ask for three method bodies, your lump shifts right by design and your 70% threshold becomes meaningless. One semester we set the review trigger at 70% on a 12-week Python course and pulled 41 flags out of a class of 180. Thirty-four of them were the same skeleton. We had set the flag, not the students.
A percentage without a denominator is not a number
This is the part of a similarity report I see misread most often, including by me in my first two years of running these checks.
| Matched tokens | File size | Reported score | What it actually means |
|---|---|---|---|
| 240 | 240 | 100% | Entire file shared. Strong signal. |
| 240 | 400 | 60% | One large 240-token block shared. Often stronger than the row above. |
| 24 | 24 | 100% | Twelve lines. Frequently boilerplate. |
| 18 | 420 | 4% | Nothing. Ignore. |
A 100% match on a 12-line Node.java is not evidence of anything. A 60% match where the overlap is concentrated in a single contiguous 240-token region is a much bigger deal, because it means someone copied a block, not a structure. Look at where the overlap sits before you look at the headline number.
Some tools report max similarity, some report average similarity across all matched regions, some report average across the whole class including the pair, some report the fraction of one file matched against the other. JPlag's "average similarity" and "max similarity" columns are not the same number and confusing them changes your judgment by 20 points. MOSS sidesteps percentages mostly and shows you matched token runs directly, which is honest but takes longer to read. For a clean side-by-side of how these report shapes differ, the Codequiry vs MOSS breakdown is worth ten minutes.
What an 85% should actually make you do
Treat it like a temperature, not a verdict. It tells you where to look. Then look for the things token math can't see:
Identical bugs. The strongest signal in the whole business. If both submissions get the same edge case wrong in the same unusual way, that is not convergent thinking. That's a shared text. In the linked-list case I opened with, both students used < instead of <= in the same traversal and both leaked a node on the same branch.
Identical dead code. Unused imports. A commented-out debug print. A variable declared and never read. Nobody independently produces the same dead code.
Identical comments, including typos. Comments are stripped before comparison in most pipelines, which is technically right and practically unfortunate, because a shared comment is about as close to a confession as you'll get.
Identical identifier sets. If the token normalizer collapsed both to ID, go back to the raw source. Two students who both named a variable tempAccumulatorValue are not a coincidence.
And the counter-check: if the 85% is all in the skeleton, the imports, and the two required error-handling blocks, close the file. You're done. Move on.
Where the number goes blind
Three blind spots worth knowing about, because they're the ones that let real cases walk.
Cross-language copying. A student who finds a Python solution to an algorithms assignment and translates it to Java produces near-zero token overlap. The token streams share no k-grams because the languages share no tokens. You need structural comparison across a shared intermediate representation to catch this, and most tools don't do it well.
AI paraphrase. This is the one that changed the math for everyone after 2022. Ask a model to "rewrite this function with different variable names and a different loop structure" and you get something that scores 30% against the original while being semantically the same code. Peer similarity alone cannot see that. If your workflow is peer comparison and nothing else, you have a gap that opened three years ago and has not closed.
Web sources. A student who copies from a GitHub repo you've never heard of, or a GeeksforGeeks page from 2018, has no peer to match against. That's a web corpus problem, not a peer comparison problem, and it needs a separate scan.
Triangulating three scores instead of trusting one
The practical answer to all three blind spots is to stop looking for a single number. You want at least three, computed separately, and shown separately.
Peer similarity tells you whether two people in your cohort share material. Web similarity tells you whether anyone shares material with the indexed internet. An AI-generation score tells you whether the submission has the statistical signature of machine authorship. They fail in different directions, and when two of the three move together you have something.
This is the design decision I'd push any department toward. Codequiry reports peer, web, and AI as three distinct scores with their own match sources rather than fusing them into one composite, which sounds convenient and is actually the worst thing you can do to a reviewer. You cannot defend "87" in a hearing. You can defend "the peer score is 91% because these 240 tokens match, the web score is zero, and the AI score is in the low-single digits." That's a sentence an academic integrity officer can work with.
The web side matters more than people assume. A scan that only checks against your own submission pool will miss the student who copied from a 2019 tutorial, and that student exists in every cohort. Checking against GitHub and the open web in the same pass is the difference between a complete check and a partial one. You can see how the web-side results render, down to per-domain scores and line counts, in the source code plagiarism checker walkthrough.
Picking thresholds that survive an appeal
Numbers I've seen hold up, on assignments of 150 lines or more, with a starter file under 30% of the final size:
Below 60%: ignore. Not worth the labor.
60% to 80%: review if the match is concentrated in one contiguous region, or if the same pair also has a nonzero web score, or if there's a matching AI signal. Otherwise file it.
Above 80% on a file of meaningful size: review. Read the diff. Not the number, the diff.
Above 95% with matched comments: this is not a review, it's documentation.
Two process rules matter more than the thresholds. First, every flag gets read by a human before it goes anywhere near a student. Second, the reviewer documents what they saw in the diff and why it did or did not constitute a violation, in writing, before the student is contacted. If your department can't produce that note six months later, you will lose the appeal on procedure regardless of what the code says.
One more, and this is the one that gets skipped: tell students what's being run and on what. A syllabus line saying "submissions are checked for peer similarity, web sources, and AI generation" changes behavior measurably and costs you nothing. The detection rate matters less than the belief that detection is happening.
Frequently Asked Questions
Is an 85% code similarity score proof of plagiarism?
No. It means 85% of the compared token material overlaps, which can happen because of a shared starter file, a common textbook pattern, or straightforward copying. The score routes a submission to human review. The reviewer's reading of the diff is what decides the case.
Does renaming variables change a similarity score?
Almost never, on any tool worth using. Identifier normalization collapses variable names to generic tokens before comparison, so a renamed copy typically scores as high as the original. Reformatted code behaves the same way. Restructuring the control flow is what actually drops the score, usually by 30 to 45 points.
Why do two honest submissions score 40% against each other?
Because the assignment dictates the interface. Both students write the same signatures, imports, and required error checks, and those are identical tokens. On a 150-line assignment with a substantial starter file, a 25% to 45% floor is normal, and any threshold set inside that range will flood you with false positives.
Can a low similarity score still mean a copied submission?
Yes, in three common cases: translated code from another language, code rewritten by a language model to change its surface structure, and code copied from a web source that isn't in the peer pool. Each needs a separate detection pass, which is why peer-only checking leaves gaps.
The number in the report is a measurement of overlap, nothing more. Run the diff, read it, write down what you saw, and let the tokens support the judgment instead of replacing it. If you want to see how the three scores render side by side on real submissions, run a check on your own assignment set and look at the score breakdown before you set a single threshold.