The first plagiarism check I ever ran was a Python script wrapped around difflib, in the fall of 2011, over 187 submissions of a linked-list assignment in CS2. It flagged nothing. Fourteen pairs in that cohort had clearly been written together. I found them by reading, badly, over two weekends.
Twelve years later I ran the same archive through a modern pipeline and the same fourteen pairs surfaced in ninety seconds, along with two I had missed entirely and one I had wrongly accused. That gap is the whole story of code similarity detection in miniature, and it is worth understanding generation by generation, because most of the tooling arguments I hear in department meetings are really arguments between 1995 and 2024.
So I did an actual experiment. I dug 1,914 submissions out of six semesters of my own courses, planted 40 known copies with deliberate disguises, and ran four generations of matching technology against the set. What follows is the method, the numbers, and an honest account of where each approach breaks.
The method, and why I seeded my own ground truth
Real plagiarism cases are inconvenient research subjects. You never know the true positive rate because you never learn about the copies you missed. So I built ground truth by hand.
The archive covers two courses: CS2 in Java (nine assignments, 2019 through 2024) and a data structures course in Python (seven assignments, same window). Total 1,914 files, median length 118 lines, longest 640. I took 40 clean submissions, produced one disguised duplicate of each, and removed the originals from the pool so every planted pair was the only instance of that solution in the set. Then I labeled the 80 files as known-positive and everything else as presumed independent (I have not audited every remaining pair by hand, so the "false positive" column below means unflagged files that got flagged, not confirmed innocence).
The disguises were chosen to mirror what students actually do when they are trying not to get caught, which is also what students do when they are legitimately collaborating and trying not to look suspicious. Five transformations, eight pairs each: identifier renaming, reformatting with comments stripped, a for loop rewritten as a while, a 30-line method split into three helpers, and a switch converted to an if/else if chain. These map roughly onto what the clone-detection literature calls Type-1 through Type-3 clones, which I'll come back to.
1974 to 1993, when a diff was the whole idea
Unix diff shipped in 1974 and it computes a longest common subsequence over lines. For source code it is close to the worst possible representation: rename one variable and every line containing it is "different," reformat and every line moves. The first purpose-built tools understood this. Dick Grune's SIM, circulating at VU Amsterdam from the late 1980s, tokenized programs before comparing them and used dynamic programming to align token sequences, which is a genuine conceptual leap: compare the structure, not the text.
SIM also introduced the thing that would define the next thirty years of argument, which is a similarity threshold you have to tune. Set it low and you drown in flags. Set it high and the interesting cases, the ones where a student changed just enough, slip through.
import difflib, itertools, pathlib
subs = {p.name: p.read_text() for p in pathlib.Path("subs").glob("*.java")}
for a, b in itertools.combinations(sorted(subs), 2):
r = difflib.SequenceMatcher(None, subs[a], subs[b]).ratio()
if r > 0.85:
print(f"{r:.2f} {a} {b}")That is essentially my 2011 script, and it is essentially SIM's ancestor with a worse algorithm. Against the seeded set at a 0.85 threshold it caught 11 of 40. It caught the renamed-identifier pairs well enough (5 of 8) because renaming short variables rarely moves the character-level ratio much. It caught reformatted pairs worse (4 of 8). It caught nothing that had been restructured.
The fingerprint era, MOSS, and the winnowing algorithm
Alex Aiken released MOSS at UC Berkeley in 1994 and it is still running. The algorithm underneath was published later, in the 2003 SIGMOD paper by Schleimer, Wilkerson, and Aiken on winnowing, and it is worth stating plainly because it explains both why MOSS works and where it fails.
You tokenize the source with a language-specific lexer, so whitespace and comments vanish and identifiers can be normalized. You take every contiguous k-token sequence (k is typically around 25 to 40 depending on language). You hash each one. Then you winnow: from each window of hashes, keep the minimum, which yields a sparse but guaranteed-dense fingerprint set. Two documents that share a long run of tokens will share fingerprints no matter how the surrounding code was rearranged, because the guarantee is that any match of length at least k will produce a shared fingerprint.
The winnowing guarantee is the reason MOSS survived three decades of refactoring tricks: any copied run of tokens longer than k cannot hide, no matter what the student does to the rest of the file.
Against my seeded set, MOSS at default settings caught 29 of 40. Perfect on renaming and reformatting, as you would expect. Weaker on structure change: 5 of 8 for loop rewrites, 4 of 8 for method extraction, 4 of 8 for the switch conversion. When you split a method in three, no contiguous 30-token run survives intact across the boundary, and winnowing is fundamentally about contiguous runs.
Two operational notes from eleven years of running it. The result pages are ephemeral, gone within a couple of weeks, so export the HTML the same day or lose the evidence. And the submission mechanism was designed for a batch job in 1994 and still behaves like one, which is fine until you need to explain a flag to a student's advisor in a meeting that starts in ten minutes.
Token sequences, greedy string tiling, and why JPlag held on
JPlag arrived from the University of Karlsruhe in 1996, from Prechelt, Malpohl, and Philippsen, and it also works on token sequences, but with greedy string tiling rather than fingerprints. GST (Michael Wise, 1993) finds maximal common substrings and marks them, then repeats on the unmatched remainder, which means it can produce several disjoint matched regions between the same two files instead of one aggregate score. In practice this is a much better fit for the question a human actually asks, which is "show me the parts that match," not "give me a number."
JPlag 4.0, released in early 2023, rebuilt the pipeline and added tree-based comparison alongside the classic token mode, plus a proper HTML report format. On my seeded set, JPlag 4.0 in token mode caught 33 of 40. Method extraction went 6 of 8. It is the tool I would still recommend to a department with no budget and one technically comfortable person to run it.
The flag that bites everyone is --min-token-length, which defaults to 9. Leave it alone and a two-line helper function that got extracted and renamed will never be reported, because it is under the floor. I have watched a colleague argue for a full honor code hearing on a case that a 6-token threshold would have reframed as "these two students share a utility function and nothing else," which is a different conversation. Our department now standardizes on 6 with a documented rationale, and we say so in the syllabus.
Syntax trees enter, and clone research crosses over
The idea that source should be compared as a tree rather than a string did not come from the plagiarism community. It came from clone detection, the industrial cousin of this work. Ira Baxter's work on Design Maintenance Systems in the mid-1990s used abstract syntax trees for diffing, and by 2007 the field had a serious benchmark: Bellon and colleagues ran six clone detectors over four open-source systems and found, uncomfortably, that the tools barely agreed on what a clone was. Each detector's output overlapped the others' by a small fraction.
That finding shaped everything after it. Roy and Cordy's 2007 survey formalized the Type-1 through Type-4 taxonomy that my seeded transformations map onto, and it reframed the problem: you are not looking for identical text, you are looking for a distance metric over semantics, and any single metric is a proxy.
An AST matcher (think GumTree-style tree edit distance, or the tree mode in JPlag 4.x, or Dolos, which the Dodona team at KU Leuven first released publicly in 2022) normalizes the tree before comparing it. Renaming vanishes because leaves are compared by type. Reformatting vanishes because formatting is not in the tree. Method extraction is caught because you compare subtrees, and a subtree that got hoisted into a helper is still a subtree. On my set the AST matcher caught 36 of 40, including 7 of 8 method-extraction pairs.
What it still misses is the control-flow rewrite. Converting a switch to an if chain changes the tree's shape at a level that tree edit distance charges heavily for. Only 6 of 8 of those pairs surfaced, and the two that did not were the ones where the student also reordered the cases.
What code similarity detection actually bought us, generation by generation
Here are the numbers from the re-run, broken out by disguise type. Every cell is out of 8 planted pairs.
| Disguise applied | Line diff (0.85) | MOSS (winnowing) | JPlag 4.0 (token/GST) | AST matcher |
|---|---|---|---|---|
| Renamed identifiers | 5 | 8 | 8 | 8 |
| Reformatted, comments stripped | 4 | 8 | 8 | 8 |
| Loop rewritten as while | 1 | 5 | 6 | 7 |
| Method split into helpers | 1 | 4 | 6 | 7 |
| Switch converted to if-chain | 0 | 4 | 5 | 6 |
| Total found (of 40) | 11 | 29 | 33 | 36 |
And the column I care about as much as recall, because it is the one that eats my evenings:
| Method | Independent pairs flagged | Average review time per flag | Total review load, this cohort |
|---|---|---|---|
| Line diff | 22 | 3 min | ~1 h 40 m |
| MOSS | 5 | 4 min | ~2 h 15 m |
| JPlag 4.0 | 6 | 5 min | ~3 h 15 m |
| AST matcher | 4 | 7 min | ~4 h 40 m |
Read those two tables together and you get the real tradeoff nobody puts in the tool documentation. Better matching produces fewer flags but each flag is harder to adjudicate, because the matcher is now catching structural similarity that requires a human to distinguish from "two students both wrote the obvious merge sort." The AST matcher's flags are the most accurate and the most expensive, at seven minutes each. My afternoon is the budget line that never appears in the comparison chart.
Four and a half of those seven minutes are the side-by-side diff. The other two and a half are the part that tooling never solved until recently: figuring out whether the submission timestamps make the copying direction plausible, checking whether the two students were ever in the same lab section, and opening last week's assignment to see whether they were already converging. That is a database problem disguised as a similarity problem. My spreadsheet has fourteen columns and half of them are not about the code at all.
2022 to now, when the source of the code changed
Everything above is about copied code and the assumption underneath it is that a copy has a human author who started somewhere. GitHub Copilot went generally available in June 2022. ChatGPT shipped on November 30 of that year. Stack Overflow's 2024 developer survey put AI tool use at 62 percent of respondents, up from 44 percent the year before. For a grading pipeline, the relevant number is not 62. It is that the 2023 and 2024 submissions in my archive, the two most recent semesters, contain a class of file I have no peer for. There is no second submission to match against.
Similarity detection is structurally blind to this. Winnowing, GST, and tree edit distance all answer "is this like something else in my set?" When the answer is "no, it is like something in a training corpus," every one of those algorithms returns zero and the assignment comes back clean.
A similarity score of 4 percent and an AI score of 91 percent frequently describe the same file. A tool that reports only the first number is not lying to you, it is answering a different question than the one you asked.
The signals are different in kind, too. Detection on the generation side looks at token-level predictability (perplexity, which for code means how surprised a language model is by each token given the ones before it), at variance across a document rather than within a document, and at stylistic tells: uniform comment density, defensive checks that no student would write, documentation strings on private methods, consistent Optional usage in a course that has not taught Optional. None of that is similarity. You cannot get there by comparing files to each other.
This is the point where I stopped running two tools and started running one. Codequiry does the peer and web comparison with token, AST, and fingerprinting passes that survive renaming and reformatting, which is what the fourth row of my table above measures, and it runs an AI-generation pass over the same upload so the report has both numbers next to each other. The workflow change is smaller than it sounds and bigger than it looks: one CSV, four score columns, sorted by max rather than by peer alone. When the peer score is 8 and the AI score is 87, I know which conversation I am having before I open the file.
Why one detector stopped being enough
The honest summary of the tool comparison landscape in 2025 looks like this.
| Tool | Origin | Core method | AI-generation scoring | Web and GitHub corpus |
|---|---|---|---|---|
| SIM | VU Amsterdam, late 1980s | Token-level dynamic programming | No | No |
| MOSS | UC Berkeley, 1994 | k-gram fingerprints plus winnowing | No | No |
| JPlag | Karlsruhe, 1996; 4.x in 2023 | Greedy string tiling, tree mode added | No | No |
| Dolos | KU Leuven, 2022 | Tree and token fingerprinting | No | No |
| Turnitin | 1997; AI writing flag 2023 | Text corpus matching | Text only, not source code | Yes, for prose |
| Codequiry | Commercial, API and dashboard | Token, AST, and fingerprinting combined | Yes, per file and per submission | Yes, GitHub and open web |
Nothing in that table is a criticism of MOSS or JPlag. They are free, they are maintained by people who care, and MOSS in particular is a small miracle of longevity. But the problem they were built to solve was scoped in 1994 to a closed set of student files, and three of the four things I now need to check (AI generation, code lifted from GitHub, code lifted from a tutorial site) fall outside that scope. If you want to see how the two approaches differ on the same submission set, there is a side-by-side comparison of Codequiry and MOSS that walks through it.
What has not changed in thirty years
The threshold argument. Every tool since SIM has asked the instructor to pick a number, and every instructor picks it in a hurry and then defends it in a hearing. My department now writes the number into the syllabus with a one-paragraph explanation of what it means, which has cut the number of "my score was 41 percent, is that plagiarism" emails to roughly zero.
The review bottleneck. A detector has never once decided a case. It ranks, and a human adjudicates, and the human's time per flag has gone up, not down, as the matchers got better.
And the false accusation risk, which is the one I lose sleep over. My seeded run produced four flags against presumably independent pairs even with the best matcher. Those are the files where two students both wrote the same obvious solution to a well-specified problem, which is not misconduct, it is a badly designed assignment. If your assignment has one correct shape, your detector is measuring your assignment.
What I would tell a new TA running their first check this semester: run the peer pass and the AI pass on the same upload, export everything to a spreadsheet the same day, sort by maximum score across all columns rather than peer score alone, and budget seven minutes per flag rather than three. Start with an established code plagiarism checker that gives you peer, web, and AI numbers in one report, because stitching three tools together is how you end up in 2011 again, reading 187 files by hand and hoping you are thorough. If you are setting up a course from scratch, the instructor workflow walkthrough covers the submission portal and the first-check setup in about the time it takes to drink a coffee.
The last thirty years of this field have been a long argument about representation, from lines to tokens to trees, and the argument was worth having because each move caught something real. The next argument is not about representation at all. It is about provenance, and it started in November 2022. Run the AI detection pass alongside the peer check and you will see what I mean in your own archive.