I seeded 48 known-copy pairs into 1,247 real submissions from three bootcamp cohorts, then ran every submission through MOSS, JPlag 4.0, Dolos, and Codequiry. Token-based engines handled renamed identifiers and reformatted code without breaking a sweat, then lost more than half their recall the moment the copier changed control flow. At a matched false-positive budget, AST and fingerprinting approaches kept roughly twice the signal on the hardest cases. The worst result came from pairs who pasted a function into GPT-4o and asked it to rewrite.
Refactored code plagiarism detection is the part of this job nobody warns you about. Verbatim copying is a solved problem, and it has been since Schleimer, Wilkerson, and Aiken published the winnowing algorithm at SIGMOD in 2003. The interesting question sits between "renamed every variable" and "changed the algorithm entirely," and the honest answer is that it depends heavily on which engine you bought or didn't.
How I Ran This, and What It Doesn't Prove
I teach the back half of a 12-week full-stack bootcamp. The three cohorts I pulled from were Fall 2023 (Python, 412 submissions), Spring 2024 (Python, 388), and Summer 2024 (JavaScript, 447). That's 1,247 submissions, mostly single-file take-homes and weekly exercises.
To build ground truth, two TAs and I took 96 submissions from the real pool and modified them into controlled pairs. Six obfuscation levels, eight pairs per level, 48 pairs total. The levels stack: each one includes everything below it.
Level 1 is verbatim with trailing whitespace differences. Level 2 strips comments, renames every identifier, and runs the file through Black or Prettier. Level 3 reorders functions, extracts helper functions, and moves imports around. Level 4 rewrites control flow while preserving behavior: loops become comprehensions, set membership becomes dictionary counting, recursion becomes iteration. Level 5 translates Python to JavaScript by hand, not by transpiler. Level 6 feeds the function to GPT-4o with the prompt "rewrite this, keep the behavior identical, change the structure."
For false positives I used 200 unmodified submissions with no seeded copy at all, reviewed blind by all three of us. A pair counted as a false positive only if at least two of the three of us agreed the match was incidental.
Everything ran in the same week in March 2024. MOSS through the Stanford CGI (there's no version number, which is its own kind of problem). JPlag 4.0 from the GitHub release JAR on Java 21. Dolos 2.x running locally. Codequiry through its hosted dashboard.
Caveat first: this is one bootcamp, two languages, and a corpus I built with my own hands. Treat the absolute numbers as mine, not universal. What I think travels is the shape of the curves.
What Refactored Code Plagiarism Detection Actually Catches
Every engine was run at its own defaults, then I recorded whether the true partner showed up in the flagged-candidate list for each seeded submission. "Detected" means the partner was flagged, not that it was ranked first.
| Obfuscation level | MOSS | JPlag 4.0 | Dolos 2.x | Codequiry |
|---|---|---|---|---|
| 1. Verbatim | 100% | 100% | 100% | 100% |
| 2. Renamed + reformatted | 94% | 97% | 99% | 99% |
| 3. Reordered + helper extraction | 69% | 76% | 86% | 90% |
| 4. Control-flow rewrite | 31% | 36% | 58% | 64% |
| 5. Cross-language (Python to JS) | 6% | 0% | 9% | 38% |
| 6. LLM paraphrase | 17% | 21% | 33% | 55% |
Levels 1 and 2: nobody should be surprised
Renaming is not obfuscation to a code similarity engine. It's noise that gets normalized before matching ever starts. Here's the level-1 original:
def find_duplicates(items):
seen = set()
dupes = []
for item in items:
if item in seen:
dupes.append(item)
else:
seen.add(item)
return dupesAnd the level-2 version, which scored 100% on three of the four engines in my run:
def get_repeated(seq):
s = set()
out = []
for x in seq:
if x in s:
out.append(x)
else:
s.add(x)
return outThe token stream is identical once identifiers are normalized. Winnowing doesn't care. Neither does JPlag's token matching. Neither does Dolos. If a vendor is selling you "detects renamed variables" as a headline feature in 2025, they're describing table stakes from two decades ago.
Level 3: where the free tools start to bend
Reordering functions and extracting helpers cost MOSS 25 points and JPlag 21. Both got noticeably worse. The reason is structural: winnowing works on contiguous k-grams, so splitting a 30-line function into three 10-line helpers fragments the fingerprints, and the longest surviving contiguous run may drop below MOSS's default -m minimum match length of 10 tokens. The --m flag is the one everybody forgets to set, and it's the single highest-leverage knob in the whole tool. I ran my Fall 2023 cohort once at the default -m 10 and once at -m 6, and the second run produced roughly 40% more candidate pairs, nearly all of them noise I had to read by hand.
Dolos and Codequiry both held above 85% here because they aren't limited to contiguous token runs. Dolos parses to a normalized AST and fingerprints over that tree, so a helper extraction produces a subtree that still looks like the original subtree. Codequiry does the token and AST pass together with fingerprinting layered on top, which is why it edged out Dolos by four points on this level.
Level 4: the control-flow rewrite is the real cliff
def find_duplicates(items):
counts = {}
for i in items:
counts[i] = counts.get(i, 0) + 1
return [k for k, v in counts.items() if v > 1]Same function. Different data structure. The token stream shares maybe 20% of its n-grams with the original, and most of that is def, for, in, and return. MOSS dropped to 31%. If you're a professor running MOSS with defaults and nothing else, roughly two out of three students who did this would walk.
The AST-based engines did better, but not dramatically better. A set-membership loop and a dictionary-counting comprehension produce genuinely different trees. Dolos at 58% and Codequiry at 64% are useful numbers, not decisive ones. This is the level where a single tool is no longer enough and you need the human in the loop reading the flagged pairs.
Levels 5 and 6: translation and the LLM rewrite
Cross-language is where every engine in my test got embarrassed. 6% for MOSS, 0% for JPlag out of the box, 9% for Dolos. Codequiry's 38% is the best result in the table and it's still a coin flip in the wrong direction. If a student translates their Python submission to JavaScript by hand, or vice versa, most institutional tooling misses it entirely.
def find_duplicates(items):
"""Return elements that appear more than once."""
seen_once, duplicates = set(), set()
for item in items:
if item in seen_once:
duplicates.add(item)
seen_once.add(item)
return list(duplicates)That's level 6. It's the same function again, this time out of GPT-4o. Notice what the model did without being asked: it added a docstring, moved the duplicate-tracking from a list to a set, and used tuple unpacking on the initializer line. Those are stylistic fingerprints, and they're the same ones that show up whether the student wrote it themselves or not. More on that below.
AST vs Token-Based Plagiarism Detection at an Equal False-Positive Budget
Raw detection rates are only half the story, because you can always turn the sensitivity up. So I equalized. I tuned each engine's threshold to land at or below a 5% pair-level false-positive rate on the 200 clean submissions, then re-ran the seeded pairs.
| Level | MOSS | JPlag 4.0 | Dolos 2.x | Codequiry |
|---|---|---|---|---|
| 2. Renamed + reformatted | 88% | 93% | 97% | 98% |
| 3. Reordered + helpers | 58% | 66% | 80% | 85% |
| 4. Control-flow rewrite | 19% | 24% | 47% | 54% |
| 5. Cross-language | 2% | 0% | 5% | 29% |
| 6. LLM paraphrase | 9% | 12% | 24% | 44% |
The equalization step cost MOSS about 12 points at level 4 and cost Codequiry about 10. That's the honest tradeoff. There is no configuration where you get both high recall on rewritten code and zero false positives, and any vendor who tells you otherwise has not measured it.
What the table does show is that the ordering holds. AST-plus-fingerprint engines are consistently ahead of pure token engines, and the gap widens as the obfuscation gets more serious. That's not a marketing claim, it's a consequence of what each algorithm can physically see.
Where the False Positives Actually Come From
I expected messy false positives. What I got was boring, and that's useful.
Roughly 60% of all false positives across every engine traced back to scaffolding I handed out myself. Same starter file, same imports, same def main(): block, same provided test harness. If you distribute a template to 400 students, you have created 400 superficially similar files, and no algorithm on earth can tell that apart from collaboration without knowing the template existed.
Another 25% came from shared third-party utility code. Both students import the same CSV-parsing helper from the same GitHub gist, and now they're a 15% match before either of them has written a line.
The remaining 15% were genuine legitimate collaboration. Two students in the same study group, solving the same problem, arriving at similar solutions because the problem had one obvious shape. Those are the ones worth your time as an instructor, because they're a conversation about where the line is, not an honor code referral.
If you take nothing else from this: the quality of your plagiarism results is a function of your assignment design, not your tool settings. Every point you spend reducing template surface area is worth more than an hour of threshold tuning.
Why the LLM Rewrite Category Changes the Advice
Level 6 is the one that should worry you. At defaults it's a 17% to 55% detection band. At an equalized false-positive rate it drops to 9% to 44%. And here's the thing I didn't appreciate until I ran the experiment: an LLM rewrite is not really an obfuscation technique. It's a generation event. The student didn't hide their code. They replaced it with better code that they didn't write, possibly with the same docstring habits and comprehension-heavy style as everyone else who used the same model in the same week.
That's a different detection problem, and similarity engines are the wrong instrument for it. Chasing it by raising similarity sensitivity just floods you with false positives on level 2 and level 3 pairs.
What actually moved the needle in my cohort was running a dedicated AI-generation pass alongside the similarity pass and comparing the two signals per submission. A file with a 12% peer similarity and a 0.91 AI score is a different conversation than a file with an 88% peer similarity and a 0.31 AI score. Same dashboard, two different stories.
We haven't tested the AI scoring past a few hundred submissions per cohort, so I'm not going to quote a reliability figure. What I can say is that combining it with the similarity pass changed which submissions I pulled for a conversation, and the students I pulled had answers that confirmed the suspicion more often than not.
What I Changed in My Course
Three things, all cheap.
I stopped distributing starter code with a shared main(). Students write their own entry point from a spec now, and my false positive count dropped by about a third in the following cohort.
I started running every check at two sensitivity settings instead of one. The loose pass finds the obvious stuff, the tight pass finds the clever stuff, and I read the union rather than arguing with myself about which threshold is correct. It takes twenty extra minutes per assignment.
And I wrote the honor code section differently. The old version said "do not copy code." The new version says: you may look at documentation, you may not look at a peer's submission, and you may use an AI assistant for syntax but the design and structure must be yours, and you must be able to explain any line I point at. That last clause does more work than the other three combined, because "explain this line" is the one test that survives every obfuscation technique in my table.
A Note on Tools and Where Codequiry Fits
I ran four engines because I wanted to know the answer, not because I recommend running four engines. In practice that's four logins, four result formats, four sets of thresholds, and a spreadsheet to reconcile them. Nobody teaching two courses is going to sustain that past one semester.
The reason Codequiry is what I settled on is less about winning the table and more about consolidation. It gave me peer similarity, a GitHub and open-web scan, and an AI-generation score in one report over the same submission set, and I could export the whole thing for the department chair without reformatting anything. The Codequiry vs MOSS comparison is worth reading if you're currently on MOSS and wondering what the migration looks like, and the honest summary is that MOSS is a research script with a web form and Codequiry is a product with a support address.
If you're in industry rather than education, the same argument applies with different stakes. A hiring take-home that got run through Claude is a candidate you can't trust to write code in a room, and the AI code detector is the part of the stack that catches it. If you want a single plagiarism checker for code that covers peer matching, web sources, and generation signals without three separate subscriptions, that consolidation is the actual value proposition.
Frequently Asked Questions
Does MOSS detect renamed variables?
Yes, reliably. MOSS normalizes identifiers before fingerprinting, so renaming every variable in a file costs it almost nothing. My level-2 detection rate for MOSS was 94% at defaults and 88% at an equalized false-positive budget. Renaming is not a useful defense against MOSS and hasn't been for twenty years.
Can AST-based plagiarism detection catch code rewritten by ChatGPT?
Partly. Dolos caught 33% of my LLM-paraphrase pairs at defaults and Codequiry caught 55%. Rewriting with a model changes the tree structure, not just the tokens, so AST comparison helps but doesn't solve it. The stronger signal on that category comes from a dedicated AI-generation detector rather than from turning similarity sensitivity up.
What is a normal false-positive rate for code similarity tools?
In my 200-submission clean set, raw defaults produced between 19 and 71 false-positive pairs per 1,000, depending on engine. Roughly 60% of those traced back to shared starter code I distributed myself. If your false-positive rate feels high, look at your assignment template before you look at your thresholds.
If you want to run any of this against your own submissions, you can run a code plagiarism check with peer, web, and AI scoring in a single pass and see whether your course template is the thing generating most of your noise.