In spring 2004, a shell script wrapping diff compared 612 submissions from an introductory Java course. That's 186,966 pairs. The review queue it produced was unusable: roughly nine of every ten flagged pairs turned out to be two students sharing the same starter file. The tool wasn't broken. It was comparing lines, and lines are the least stable thing in a program.
Rerun the same archive with modern tooling and the queue drops to something a single TA can read in an afternoon. Code similarity detection has changed its unit of comparison twice since the early 1990s, from lines to token sequences to syntax trees, and each change cut the review load by roughly an order of magnitude. What follows is the arc, with the numbers I can defend and the caveats I can't avoid.
What line-based similarity could never see
The first generation of automated checks borrowed Unix diff and its longest-common-subsequence core, wrapped in a script that normalized whitespace with -w and ignored blank lines with -B. The output is a set of line spans appearing in both files. It's fast, it's transparent, and it fails on the first thing students learn to change.
# Submission A
total = 0
for value in scores:
total = total + value
mean = total / len(scores)
# Submission B
acc = 0
for s in scores:
acc += s
avg = acc / len(scores)These two files share zero identical lines. A diff-based checker reports no match at all, while any reader will tell you it's the same program. The metric was measuring typography, not authorship. That's the whole failure mode of the era in one example, and it's why the 1990s produced so much tooling instead of so much casework.
The token turn, and why winnowing scaled
Michael Wise's Greedy String Tiling (1993) proposed matching maximal runs of common tokens rather than common lines. JPlag, built by Lutz Prechelt's group at Karlsruhe and described in the 2000 paper "Finding Plagiarisms among a Set of Programs with JPlag," applied that idea to submissions at course scale. The normalization step did most of the work: lex the source, drop comments and layout, map identifiers and literals to placeholders, then compare token strings with a minimum match length (historically around 9 tokens, tunable, and the setting people get wrong most often).
MOSS took a different route to a similar place. It was used in Berkeley courses from the mid-1990s and formally described in the 2003 SIGMOD paper on winnowing by Schleimer, Wilkerson, and Aiken. Winnowing hashes every window of k consecutive tokens and keeps only the minimum hash per window, which bounds the fingerprint set and makes the comparison cheap even for a few thousand submissions. The published guarantee is what matters for tool builders: it selects the same fingerprints regardless of where in the file the matching text sits.
| Era | Representative tools | Unit of comparison | Invariant to | Blind spot |
|---|---|---|---|---|
| 1974 to 1993 | Unix diff, sdiff | Lines, via longest common subsequence | Whitespace, blank lines | Renaming, reformatting, reordering |
| 1993 to 2000 | Greedy String Tiling, JPlag | Normalized token strings | Comments, layout, identifier names | Control-flow rewrites, helper extraction |
| 1994 to 2003 | MOSS winnowing | Hashed k-gram fingerprints | The above, plus scale and noise | Same tokens in a different structure |
| 2007 to 2014 | Deckard (2007), GumTree (2014) | AST subtrees, tree edit scripts | Loop form, statement order, extraction | Parser coverage per language, runtime cost |
| 2022 onward | Dolos (Tree-sitter), Codequiry | AST plus token fingerprints | Refactoring within a language | Freshly generated code, a different problem |
Why trees survived the refactoring arms race
Token matching has a predictable ceiling, and students who have been caught once find it quickly. Convert a for to a while, hoist a condition into a boolean, pull the body into a helper: the token sequence changes enough that a k-gram matcher scores the pair well below any sensible threshold, while the underlying structure is untouched.
// Version 1
int max = 0;
for (int i = 0; i < arr.length; i++) {
if (arr[i] > max) max = arr[i];
}
// Version 2
int best = arr[0];
for (int x : arr) {
best = Math.max(best, x);
}An AST comparison normalizes both bodies into ordered node types (declaration, loop, conditional, comparison, assignment) and aligns the sequences. The token overlap here is modest; the structural overlap is close to total. Deckard's 2007 characteristic vectors and GumTree's 2014 tree edit scripts are the two ideas most modern engines inherit from, and the practical result is a matcher that survives renaming, reformatting, and most of the mechanical refactoring students actually perform.
It is not magic. GumTree-style edit distances are expensive at cohort scale, which is why production engines run fingerprints first to shortlist candidate pairs and only then do the structural comparison. Pairwise AST diffing across 84,000 pairs without a prefilter is a good way to spend a weekend watching a CPU spin.
A re-analysis of one archived cohort
I went back to a 2019 archive from a second-year data structures course: 412 submissions, 84,636 pairs, eleven cases that had been formally adjudicated. I ran three configurations over the same files, all at default settings, and counted flagged pairs and review load. The methodology is simple and the limits are real, so read the caveats before the numbers.
| Method | Pairs flagged | Adjudicated cases surfaced (of 11) | Manual review load |
|---|---|---|---|
| Line diff, whitespace normalized | 2,140 | 6 | Overwhelming, mostly shared boilerplate |
| Token k-gram fingerprints | 388 | 10 | About 40 pairs needed a real read |
| AST plus token, defaults | 121 | 11 | 9 pairs needed a real read |
Two caveats. First, the eleven adjudicated cases came from pairs someone had already flagged, so the true positive count is an undercount of whatever the earlier tools missed, and small n means those columns are directional at best. Second, this is one course at one institution, not a benchmark; I've seen token matching outperform AST matching on assignments with heavy numeric boilerplate, where the structure is trivially similar across all solutions. The pattern I'd defend is the ratio, not the absolute counts: each generation of comparison reduces the pairs a human has to read by roughly a factor of three to ten.
Where the arc leaves us, and the axis it didn't cover
Everything above assumes copied code is a matching problem. It is: two artifacts share a lineage, and you're recovering the lineage. Generated code breaks that assumption, because there is no source document to match against. A submission produced from a prompt has no peer, no repository, and no web page behind it. The AST is clean because the code is genuinely original.
So the field split. Deterministic copying is handled by similarity engines of the kind described above; probabilistic generation is handled by statistical signals over the file itself (token predictability, structural regularity, comment-to-code ratios, the specific idioms each model family overproduces). Neither signal substitutes for the other, and a reviewer looking at one column while ignoring the other will misjudge the case. This is the reason Codequiry runs peer similarity, web and GitHub source tracing, and an AI code detector inside a single check rather than selling them as three products.
Practical summary for anyone buying or building this
If you're choosing tooling, the questions that separate engines are narrower than the marketing suggests. Ask what the unit of comparison is and whether the engine prefilter candidates before structural comparison. Ask how it handles shared starter files, since that's where most false positives come from and where an instructor's time goes. Ask whether the thresholds are visible and adjustable, because a default tuned for one course will not fit another.
- Line-based comparison belongs in a museum. If a vendor's documentation talks about "matching text," it's a text tool wearing a code tool's clothes.
- Token fingerprints are still the right prefilter at cohort scale; treat them as a shortlisting mechanism rather than a verdict.
- AST comparison earns its cost on refactored submissions, which is precisely the population that survives every cheaper method.
- Web and repository tracing is a separate corpus from peer comparison and catches a different population of submissions entirely.
- Match scores are evidence for a conversation, not proof of misconduct. Across the cohort above, the nine AST-flagged pairs that needed a read produced four honor-code cases and five cases of students who had worked together on a shared test harness.
For a fair comparison of the older options, the Codequiry versus MOSS writeup covers where a command-line matcher still wins (scriptability, cost) and where a maintained engine with a source code plagiarism checker interface and per-pair reports saves the reviewer's afternoon. Both are honest positions. Most departments that moved off MOSS did so for workflow reasons, not detection reasons.
Frequently asked questions
Does AST-based detection catch deliberately refactored code?
Up to a point. Mechanical refactoring (renaming, loop form changes, helper extraction, reordering independent statements) is largely handled. Genuine reimplementation from a different design, or a rewrite that changes the data structures, will fall below any threshold you'd want to run, and that's the correct behavior.
Why do course starter files create so many matches?
Because hundreds of submissions legitimately contain the same code. The fix is structural, not algorithmic: exclude the provided files from comparison, or set a per-file baseline before scoring. An engine that can't exclude known-shared code will bury you in pairs, which is exactly what the 2004 queue did.
Is token fingerprinting obsolete now that AST comparison exists?
No. Winnowing-style fingerprints are how you avoid doing 84,636 structural comparisons. The pipeline that works is fingerprint to shortlist, AST to confirm, human to adjudicate.
The fastest way to see the difference between eras is to run your own archive through a tool that reports both: a code plagiarism checker that separates peer similarity, web matches, and generation signals will show you in one report which of your 2,140 pairs were ever worth reading.