Comparing plagiarism detection algorithms side by side reveals that not all similarity scores are created equal. Across 15,000 real CS1 Java submissions from three universities—when we hand‑verified 500 high‑similarity pairs—token‑based winnowing, AST hashing, and multi‑layer fingerprinting produced strikingly different recall‑precision curves. Understanding those curves is what separates a manageable inbox of real cases from a flood of false alarms.
The data we’ll walk through comes from a controlled study we conducted during the 2024 academic year. We used token‑based winnowing (the algorithm behind MOSS), AST hash comparison (similar to the technique in JPlag and Dolos), and multi‑factor fingerprinting (the approach Codequiry layers on top of both token and AST signals). The goal wasn’t to crown a winner, but to understand where each technique shines, where it breaks, and what that means for professors, TAs, and academic integrity boards who need reliable evidence—not just a percentage.
How Token‑Based Winnowing, AST Hashing, and Fingerprinting Actually Work
Before the numbers, a quick concrete grounding. Suppose two students submit the following identical‑logic Java method, but one has renamed variables and reordered statements.
// Original submission
public static int sumPositives(int[] arr) {
int total = 0;
for (int i = 0; i < arr.length; i++) {
if (arr[i] > 0) {
total += arr[i];
}
}
return total;
}
// Suspect submission with renaming and reordering
public static int addNonNegatives(int[] nums) {
int sum = 0;
for (int j = 0; j < nums.length; j++) {
if (nums[j] > 0) sum += nums[j];
}
return sum;
}A token‑based winnowing tool first strips code down to a sequence of tokens—keywords, identifiers, operators, literals—and ignores whitespace and comments.
PUBLIC STATIC INT ID INT [ ] ID LBRACE INT ID = 0 SEMI FOR LPAREN INT ID = 0 SEMI ID LT ID DOT LENGTH SEMI ID PP RPAREN LBRACE IF LPAREN ID [ ID ] GT 0 RPAREN ID PLUSEQ ID [ ID ] SEMI RBRACE RETURN ID SEMI RBRACEThen it slides a window of size k (MOSS uses k‑grams; winnowing picks a subset of hashes from the sequence) to generate fingerprints. This works astonishingly well for copy‑paste with cosmetic changes. But tokenization loses the structural relationships—the AST tree is flattened into a linear bag of tokens. That means a method with the same logic but different loop structures (a while loop, a stream operation, or even just inverting an if condition) may produce token sequences that barely overlap.
AST hashing goes deeper. The code is parsed into an abstract syntax tree, and then either the full tree structure or subtrees are hashed. JPlag, for instance, computes similarity by comparing optimized AST fingerprints. In the example above, even after variable renaming, the tree structures match—a ForStatement node containing a Block with an IfStatement and an assignment. This makes AST‑based detectors more refactoring‑resistant. However, they can also flag structurally similar template code—every main method in a CS1 class may share the same skeleton, leading to high false‑positive rates unless post‑processing filters are used.
Multi‑factor fingerprinting bridges the gap. Codequiry tokenizes the submission, parses the AST, and then extracts a set of layered fingerprints: token‑level n‑grams, statement‑level structural hashes, and control‑flow‑graph signatures. Rather than relying on a single similarity score, the engine cross‑references these signals and surfaces only matches that satisfy multiple constraints. A submission that merely shares the same control‑flow skeleton without token overlap drops out; a heavily refactored copy that shares the same statement‑level bone structure still triggers.
The 15,000‑Submission Experiment Setup
We gathered submissions from three large public universities, all using the same CS1 Java curriculum (eight assignments, from a “Hello World” program to a simple text‑based game). That yielded 15,182 individual .java files across 1,897 students. We ran all three algorithms—token‑based winnowing (MOSS‑style, using the Stanford implementation with k=5), AST hashing (a reimplementation of the JPlag‑style algorithm, tuning the minimum match length parameter), and Codequiry’s production multi‑factor fingerprinting—on the full pairwise matrix for each assignment.
From the pairs flagged with a similarity score above 30% (the common default threshold), we randomly sampled 500 across the three detectors, stratified by score bands, and gave them to a team of three experienced TAs for blind manual review. The TAs classified each pair as “plagiarized” (evidence of direct copying or unacceptable collaboration), “independent but similar” (template code, same textbook examples, or convergent design), or “uncertain.” Only pairs where two out of three reviewers agreed on “plagiarized” were treated as ground‑truth positives; the rest became negatives for our analysis. This gave us 213 confirmed plagiarized pairs and 287 confirmed non‑plagiarized pairs.
Where Each Algorithm Turns Blurry: The Precision‑Recall Tradeoffs
The table below summarizes the raw performance at the default 30% threshold. We then varied the threshold to draw precision‑recall curves.
| Algorithm | Precision at 30% | Recall at 30% | False Positive Rate |
|---|---|---|---|
| Token‑based winnowing | 0.58 | 0.91 | 0.42 |
| AST hashing | 0.49 | 0.88 | 0.51 |
| Multi‑factor fingerprinting (Codequiry) | 0.87 | 0.84 | 0.13 |
Token‑based winnowing caught the most real cases—unsurprising, since most student plagiarism in CS1 is fairly lazy copy‑paste. But nearly half its flags were false alarms, primarily from shared boilerplate (assignment skeletons, identical public static void main(String[] args) methods, or repeated textbook‑provided helper functions). For a TA reviewing 100 flagged pairs, that means 42 hours wasted on innocent “matches.”
AST hashing suffered even worse from template noise. In one assignment where all students built a simple address‑book class with addContact, removeContact, and listContacts methods, AST hashing flagged 73% of the class pairs above 30%—almost entirely due to identical method‑level structure imposed by the spec. Its recall held up because structural matches still found refactored plagiarism, but the false positive rate made the output nearly unusable without manual triage.
Codequiry’s multi‑factor fingerprinting traded a few percentage points of recall for a dramatic precision boost. Because it required corroboration across token and AST layers, it automatically filtered out many structurally identical but independently written submissions. The 0.84 recall figure still captured refactored copies that token‑based winnowing might have missed—for instance, cases where students translated the same logic from a for loop to a while loop and changed all variable names, a pattern we observed in 27 of the 213 confirmed plagiarism cases.
The key insight: No single algorithm is universally best. Winnowing excels at verbatim or lightly cosmetically altered code; AST methods survive refactoring but drown in structural noise; layered fingerprinting is the only approach that keeps precision high enough for real‑world academic workflows.
Why “Refactoring‑Resistant” Matters More Than Ever
CS students are increasingly aware that running a code plagiarism checker is part of routine grading. A 2023 survey at a large Midwestern university found that 41% of students who admitted to copying code said they performed at least some refactoring—variable renaming, loop conversion, method extraction—specifically to beat automated detectors. Token‑based winnowing’s weakness here is well‑documented: simply extracting a block of code into a helper method drastically changes the token sequence, and MOSS‑style tools can miss it entirely if the k‑gram overlap falls below the selection threshold.
Our data confirms this. Among the 213 confirmed plagiarized pairs, we identified 34 cases where the student had applied non‑trivial refactoring (more than just renaming). Token‑based winnowing flagged only 19 of those above the 30% threshold; AST hashing caught 31; multi‑factor fingerprinting caught 32. The two missed by fingerprinting were extreme—one pair had a complete structural rewrite using entirely different algorithms, something even a human reviewer might have struggled to classify as plagiarism without access to version history. But the critical detail is that AST hashing flagged 68 false positive pairs among those 34, while fingerprinting only generated 4. A code plagiarism checker for teachers that forces TAs to wade through 68 innocent but structurally similar pairs to find 31 real cases will quickly lose adoption.
False Positives Aren’t Just Annoying—They Undermine Trust
When a detector cries wolf too often, two things happen. First, reviewing staff start ignoring flags below a certain threshold, which means genuine cases with cleverly obfuscated code slip through. Second, students who are falsely accused suffer unnecessary stress, and the integrity process loses credibility. In our study, we tracked the “boy who cried wolf” effect: after reviewing 150 false positives from AST hashing, one TA admitted to skipping any pair that looked like “yet another address‑book match,” inadvertently dismissing a real plagiarized pair buried in the noise.
That’s why any code plagiarism checker deployed at scale needs to optimize for precision as much as recall. The engineering challenge isn’t just detecting similarity—it’s presenting the signal in a way that respects the reviewer’s time. Codequiry’s report interface, for example, groups matches by confidence tier, highlights the overlapping regions side‑by‑side, and shows exactly which fingerprint layers triggered, so a TA can decide in seconds whether to escalate.
What This Means for CS Departments Choosing a Production Tool
Many departments default to MOSS because it’s free and well‑known. That’s understandable, but our results suggest that relying solely on token‑based winnowing gives you a binary choice: accept a high false‑positive rate or raise the threshold and miss refactored copies. A Codequiry vs MOSS comparison becomes less about “which tool finds more plagiarism” and more about “which tool’s flagged set is small enough and accurate enough that a TA can actually inspect every match.” That’s the metric that changes departmental workflow.
For institutions that also want to detect code plagiarism from online sources—Stack Overflow snippets, GitHub Gists, tutorial code—the algorithm choice matters even more. Web‑sourced code often arrives with heavy adaptation. A student who copies a complete Stack Overflow answer but migrates it from Python to Java fundamentally changes the token stream. AST hashing can still catch the structural skeleton if the logic is preserved, while token‑based detectors flounder. Multi‑layer approaches that check web corpora alongside peer‑to‑peer submissions catch both the lazy copy‑paste and the cross‑language port.
Frequently Asked Questions
What algorithm does MOSS use, and why does it sometimes miss refactored code?
MOSS uses winnowing, a technique that selects fingerprints from a sliding window of token k‑grams. It’s excellent at catching verbatim or near‑verbatim code but can miss copies where the token sequence has been disrupted by reordering statements, extracting methods, or swapping loop constructs—because those changes radically alter which fingerprints are selected.
How can I reduce false positives when checking student code for plagiarism?
Use a tool that layers multiple detection techniques—token matching, AST comparison, and control‑flow fingerprinting—and that allows you to exclude template boilerplate through instructor‑provided base files. Combining signals reduces noise from structurally similar but independently written code while still catching refactored copies.
Is there a threshold that works universally for code similarity scores?
No. The appropriate threshold depends on the assignment complexity, the language, and the detection algorithm. Our study found that a single 30% threshold yielded precision as low as 0.49 with AST methods, but 0.87 with multi‑factor fingerprinting. Tools that provide context‑aware tiered scoring (rather than one flat percentage) let reviewers calibrate per assignment.
Can code plagiarism detectors catch cross‑language plagiarism?
Traditional token‑based and AST‑based detectors operate within a single language. However, structural fingerprinting techniques can sometimes identify similar logic across languages, especially if the control‑flow structures and algorithm patterns are preserved. For reliable cross‑language detection, specialized tools that extract language‑agnostic features are required—and this is an active research area.
Ready to see what the difference between 49% precision and 87% precision looks like on your own course submissions? Run a plagiarism check on your next assignment batch with Codequiry.