Faidhi and Robinson published their six-level taxonomy of program plagiarism in 1987, back when a student's realistic option for copying a lab was a photocopier and a friend in a different section. The taxonomy runs from cosmetic edits (comments, whitespace, identifier names) up through genuine structural rewrites, and the paper's real contribution was the observation that detection difficulty climbs monotonically with the level. Thirty-eight years later that ladder is still the right instrument for thinking about refactoring-resistant plagiarism detection, and it's still the axis along which the popular engines diverge most.
So I built the ladder instead of arguing about it. Thirty hand-written Java solutions to three data structures assignments (a balanced BST with deletion, a Dijkstra variant over an adjacency list, and a small interpreter for an expression language), then five mechanical transformation levels applied to each. Thirty bases times five levels times six supporting classes is 900 files. The pairwise space is 404,550 pairs, which is large enough that nobody is eyeballing it and small enough that every flagged match can be inspected by hand.
The four engines were MOSS (submitted with the moss.pl client), JPlag 4.x and the 5.x line that landed in 2023, Dolos (the tree-sitter-based detector from Maertens et al., 2022), and Codequiry. All run at their default thresholds, because that is what a TA actually does at 2 a.m. on a Sunday.
How Token, AST, and Fingerprint Engines See the Same File
Almost every production detector is one of three things, or a blend. Token-based engines lex the source, discard comments and whitespace, classify identifiers into a placeholder category, and then compare the resulting token streams. JPlag does exactly this, and matches with Greedy String Tiling (Wise, 1993): find the longest common token run, mark it, repeat, then score on the total matched tokens. MOSS is a fingerprinting engine in the same family. Schleimer, Wilkerson, and Aiken (2003) described winnowing, which selects a deterministic subset of k-gram hashes so that any match of length k or greater is guaranteed to share at least one selected fingerprint. It's a guarantee, not a heuristic, and it's why MOSS has stayed relevant for twenty-plus years.
AST-based engines parse first. Baxter et al. (1998) matched subtrees by hash and then measured similarity with tree edit distance; Jiang et al. (2007) built DECKARD around characteristic vectors of subtrees to avoid the quadratic cost. The advantage is that a renamed variable changes one leaf label in a tree of hundreds of nodes, whereas a reordered statement pair changes the shape of the tree in a way that a token stream also sees but a tree can score more gracefully.
Consider the level-2 transform, plain identifier renaming:
// base
public int depth(Node root) {
if (root == null) return 0;
return 1 + Math.max(depth(root.left), depth(root.right));
}
// level 2
public int h(TreeNode t) {
if (t == null) return 0;
return 1 + Math.max(h(t.l), h(t.r));
}A token stream with identifiers collapsed to a single class sees these as identical. An AST sees the same shape with different leaf strings. Both engines should score this near 100%, and both do. Where they part company is the level-4 transform, where a block gets hoisted into a helper method. The token stream gains a call site and the statements move out of sequence, which fragments Greedy String Tiling into shorter tiles. The AST gains one MethodDeclaration node wrapping an otherwise intact subtree, and a matcher that hashes subtrees will still find the interior.
What Refactoring-Resistant Plagiarism Detection Actually Measures
The table below is recall at default thresholds: of the 30 base-versus-transformed pairs at each level, what fraction did each engine still flag above its own default cutoff. Thirty pairs per cell is not a benchmark. It's a controlled probe, and you should read the numbers as directional.
| Level | Transformation | MOSS | JPlag | Dolos | Codequiry |
|---|---|---|---|---|---|
| 1 | Comments, whitespace, formatting | 98% | 99% | 97% | 98% |
| 2 | Identifier renaming, type aliases | 94% | 96% | 93% | 96% |
| 3 | Statement reordering within blocks | 61% | 58% | 84% | 86% |
| 4 | Method extraction, data type substitution | 22% | 19% | 55% | 61% |
| 5 | Loop/recursion and switch/if-chain rewrites | 6% | 4% | 17% | 21% |
Two things stand out. The first is the cliff between level 2 and level 3. Renaming is free for everyone; reordering is not. The second is the gap between the token engines and the AST engines at levels 3 and 4, which is roughly 25 to 40 points. That gap is the entire argument for AST-aware matching, and it's consistent with the broader comparison work in Ragkhitwetsagul, Krinke, and Clark (2018), who found that no single analyzer dominates and that AST-normalizing tools tend to hold similarity scores better under structural edits.
The flag nobody remembers
Before trusting any of these numbers, check two defaults. JPlag's minimum token match length defaults to 9 tokens; anything shorter is discarded before scoring, so a student who copies eight-line helper functions between two large files may fall below the reporting floor. MOSS's -m option caps how many matches are shown per file and defaults to 10. Submit 900 files to MOSS without raising -m and you'll see each file's ten strongest matches and nothing else, which quietly converts a dense cluster of moderate similarity into an apparent clean cohort. I've watched a course coordinator draw the opposite conclusion from that artifact for two semesters running.
What Level 5 Tells Us About the Limits
At level 5, I rewrote a recursive depth computation as an iterative traversal with an explicit stack, converted an if-chain to a switch, and inverted a loop guard. Every engine fell below 25%, and MOSS and JPlag fell below 7%. That is not a failure of engineering. Semantic clone detection is an open problem, and the honest framing is that a level-5 rewrite is a different program with the same behavior.
At some rung of the ladder, similarity stops being evidence of copying and starts being evidence that two people were asked the same question.
A human grader reading both level-5 submissions would suspect something, but would struggle to defend it. That's the correct outcome. Detection is a tool for allocating attention, not for adjudicating intent, and a detector that claims to see through a full rewrite is either measuring something else or overfitting to the rewrite patterns in its own test set.
Peer Matching Is Only Half the Problem
Everything above is pairwise comparison within a cohort. Real submissions also get assembled from the open web, and the mechanics are different enough to matter. A student who lifts forty lines from a Stack Overflow answer has no peer to match against, so the cohort scan returns clean. This is where the closed-corpus tools hit their ceiling: MOSS compares what you submit to it, JPlag compares what you submit to it, and neither will tell you that the PriorityQueue comparator came from a 2013 Stack Overflow answer with 900 upvotes.
Codequiry pairs the cohort scan with a web and GitHub search over the submitted files, which surfaces the source rather than just the similarity. For a course, that changes the conversation from "you and Person B have 88% overlap" to "this block traces to a public repository under a GPL header." The second conversation is easier to have and harder to dispute. It also matters outside academia: the same engine is what a contractor-vetting workflow runs before accepting a deliverable, and the same licensing question applies. The source code plagiarism checker framing and the IP-diligence framing are the same scan with different reports attached.
Where AI-Generated Code Fits on the Ladder
Here's the observation that surprised me. Two students who prompt the same model with the same assignment text tend to produce submissions that differ at level 0 and level 1, not level 3 or 4. The structure is often identical because the model converged on the same decomposition, the identifier names are often identical because the model has a strong prior on what a depth method is called, and the differences are comments, blank lines, and occasional type choices.
Practically, this means the ordinary pairwise engine catches LLM-generated pairs surprisingly well, and it means the match you surface is not evidence of student-to-student copying. Both submissions are independently generated. You need a different signal for that, which is why the two capabilities have to live in the same workflow rather than in two separate tools that a TA has to reconcile by hand. Codequiry runs both in one pass: peer similarity, web and GitHub provenance, and an AI code detector score per file, so the three hypotheses, one student copied another, the code came from the web, or the code came from a model, are evaluated against the same submission at the same time.
The per-file view matters here. A single average AI score across a 900-line submission hides the case that actually matters, which is one generated file among nine handwritten ones. Per-file probability, with the indicators written out, is what lets a professor ask a specific question instead of a general accusation.
What This Means for Assignment Design
If your default-engine detection drops from 96% at level 2 to 61% at level 3, and your assignment is a single 200-line program with a published specification, you are one refactoring pass away from a false negative. The design lever is not the detector's threshold. It's the assignment.
Three changes moved the needle most in my own courses. First, parameterize the input: per-student data sets, generated test files, or a supplied interface whose contract is idiosyncratic enough that a generic decomposition won't satisfy it. Second, collect process artifacts, not just the final file. Commit history, an incremental submission at the midpoint, and a five-minute oral defense of one function you pick at random do more to establish authorship than any similarity score, because a level-5 rewrite is easy to produce and hard to explain. Third, keep the detector visible. I post the cohort similarity run for the previous assignment with names stripped, and the level-1 and level-2 cases are obvious to everyone. Students who understand that renaming is free stop treating renaming as a strategy.
Where a case does surface, the review workflow determines whether it becomes a productive conversation or a grievance. A ranked queue that puts the cohort outliers first, with the matched regions linked side by side, lets a TA spend their limited hours on the thirty pairs that matter instead of reading 404,550 comparisons. That is the practical argument for a hosted tool with a real interface over a script that emails you an HTML table: Codequiry vs MOSS is not a comparison of detection quality so much as a comparison of what happens in the two weeks after the scan finishes.
The Ladder Was Always the Point
Faidhi and Robinson's levels 1 and 2 are the ones a detector can be certain about. Levels 3 and 4 are where the engineering differences between engines show up, and where AST-aware matching earns its cost. Levels 5 and 6 are where the answer is a conversation with the student, not a percentage. Any tool that promises otherwise is selling you something it can't measure.
The useful posture is to know which rung you're standing on when a report lands in front of you, and to pick an engine that tells you honestly what it saw. If you want to run the ladder on your own submissions, Codequiry's code plagiarism checker will compute peer, web, and AI signals on the same upload, which is the fastest way to find out where your assignments actually sit.