If you've ever stared at a MOSS report where dozens of flagged pairs turned out to be innocent starter-code reuse, you know the frustration. You need a code plagiarism detector accurate enough to find true copying without burying TAs in false alarms. To answer that question with real numbers, we benchmarked four major code plagiarism checkers against a dataset of 2,400 student Java submissions with known ground truth. Codequiry caught 94% of plagiarized pairs at a 0.3% false positive rate, while MOSS achieved 87% recall with 2.1% false positives, and JPlag landed at 81% recall with 0.9% false positives. The 20-point F1 gap came down to differences in how each tool handles refactoring and starter-code filtering.
Designing a Benchmark for Code Plagiarism Checkers
We began with a corpus from a large public university’s CS1 course in Java. The dataset contained 2,400 single-file submissions for a linked-list manipulation assignment. The course used a common starter file with method stubs and a helper class; all students received this base. The grading team had manually reviewed every submission over three semesters, yielding 168 confirmed pairwise plagiarism cases—some verbatim copy-paste, others aggressively refactored (variable renaming, loop reordering, extracted helper methods). We also recorded 57 borderline pairs that the honor committee ultimately deemed not plagiarized, plus thousands of innocent pairs where similarity came only from the starter code.
We built a ground-truth validation set by randomly selecting 500 pairs: 100 known positives, 100 known borderline, and 300 clean negatives sampled across semesters. Each tool received exactly the same directory of anonymized student files, with identical submission metadata. We ran:
- MOSS (September 2023 release, Stanford), via the standard mossnet script with Java language flag and base-file exclusion for the starter code.
- JPlag (v4.1.0) with default Java similarity thresholds, excluding the known base file from analysis.
- Turnitin (Feedback Studio with code similarity, institution’s existing license), using its default similarity report generation.
- Codequiry (May 2024 database snapshot) with Java detection profiles, automatic starter-code suppression, and web plagiarism scanning enabled.
Key to a fair benchmark: All tools were configured to exclude the same starter code. Without this step, innocent base-file similarity drowned out true plagiarism signals in MOSS and JPlag, inflating false positives by up to 8×.
For each tool we calculated precision, recall, F1 score, and false positive rate (FPR) on the 500-pair validation set, treating the borderline pairs as negatives (i.e., only confirmed plagiarism counted as positive). This conservative design lets a tool’s sensitivity shine only if it avoids overcalling.
The Numbers: F1 Scores Across Checkers
After processing, we collected similarity scores and applied each vendor’s recommended threshold for “likely plagiarized.” The table below shows the core results on the validation set.
| Tool | Precision | Recall | F1 Score | False Positive Rate |
|---|---|---|---|---|
| MOSS | 80.2% | 87.0% | 83.5% | 2.1% |
| JPlag | 93.1% | 81.0% | 86.7% | 0.9% |
| Turnitin | 66.7% | 72.0% | 69.2% | 4.3% |
| Codequiry | 95.9% | 94.0% | 95.0% | 0.3% |
The F1 score difference between Codequiry and MOSS was 11.5 points, and nearly 26 points over Turnitin. That gap traces directly to each tool’s AST and token-handling choices, not to tuning. MOSS’s false positives clustered around pairs that implemented the same algorithm with similar variable names but were written independently; its winnowing algorithm is notoriously sensitive to matching tokens, especially in smaller codebases. JPlag’s token-based comparison (with subsequent string-tiling) was far more precise but missed heavily refactored copies where methods were inlined or control flow altered. Codequiry’s multi‑model ensemble—combining AST fingerprinting, control-flow graph hashing, and web plausibility cross-checks—pushed recall higher without sacrificing precision.
Resistance to Common Refactoring Obfuscations
Among the 100 known positives, 31 had been obfuscated beyond simple variable renaming. Techniques included:
- Reordering independent statements inside a method
- Extracting a block into a private helper and calling it once
- Changing iterative loops to equivalent while-loops
- Flipping conditional branches and negating logic
MOSS still detected 26 of them (84%), JPlag found 21 (68%), Turnitin flagged 16 (52%), and Codequiry caught 29 (94%). When Codequiry missed a refactored case, it was usually a student who had genuinely rewritten the assignment in a functionally equivalent but structurally different way—only web plagiarism detection caught those when the code matched a GitHub repository.
Automating the Benchmark Pipeline
Running one-off comparisons works, but a FAANG engineering manager once told me, “If I can’t reproduce your numbers next week, I don’t believe them.” A repeatable benchmark lets you track tool drift and calibrate thresholds for your own student population. Here’s a minimal Python skeleton that submits to MOSS, parses results, and compares against a ground-truth CSV. You can extend it for Codequiry’s API or JPlag’s CLI.
import subprocess, csv, re
from collections import defaultdict
def run_moss(base_file, submission_dir):
cmd = f"perl mossnet -l java -b {base_file} {submission_dir}/*.java"
output = subprocess.check_output(cmd, shell=True).decode()
# Parse URL from output; assume we saved MOSS report HTML
return "http://moss.stanford.edu/results/..." # simplified
def parse_moss_from_html(html_path):
pairs = {}
# Extract pairs and percentages: e.g., "98% submission1.java submission2.java"
with open(html_path) as f:
for line in f:
m = re.search(r'(\d+)%\s+(\S+)\s+(\S+)', line)
if m:
pct = int(m.group(1))/100.0
f1 = m.group(2).split('/')[-1]
f2 = m.group(3).split('/')[-1]
pairs[(f1,f2)] = pct
return pairs
def evaluate(predicted_pairs, ground_truth_path, threshold=0.6):
true_positives = set()
with open(ground_truth_path) as f:
for f1,f2,label in csv.reader(f):
if label == '1':
true_positives.add((f1,f2))
flagged = {pair for pair,score in predicted_pairs.items() if score >= threshold}
tp = flagged & true_positives
fp = flagged - true_positives
fn = true_positives - flagged
precision = len(tp)/len(flagged) if flagged else 0
recall = len(tp)/len(true_positives) if true_positives else 0
return precision, recall, tp, fp, fn
if __name__ == "__main__":
predicted = parse_moss_from_html("moss_report.html")
precision, recall, tp, fp, fn = evaluate(predicted, "ground_truth.csv", threshold=0.6)
print(f"Precision: {precision:.3f}, Recall: {recall:.3f}, FP pairs: {len(fp)}")
Running this weekly on fresh assignment batches—or when a new checker version drops—keeps your integrity workflow honest. Codequiry’s batch API and webhooks allow fully automated benchmarks, so TAs never wait for pairwise results.
Why AST Fingerprinting and Web Cross-Checks Matter
MOSS and JPlag rely primarily on token-level comparisons. Token-based approaches are fast but brittle: they treat a renamed variable or a reordered loop as an entirely new token sequence. MOSS’s winnowing partially compensates by selecting fingerprints from the entire document, but it still misses deeply refactored code. AST fingerprinting, which Codequiry employs, hashes the abstract structure of the code—the parent-child relationships in the parse tree. That structure survives identifier renaming, comment deletion, and even some statement reordering. In our benchmark, AST fingerprints found 6 additional refactored positives that MOSS and JPlag both missed.
Web cross-checks add another layer. When Codequiry detects high internal similarity between two submissions, it simultaneously searches a database of public GitHub repositories, Stack Overflow answers, and tutorial sites. If both student submissions are nearly identical to the same web source, the tool lowers the plagiarism score because the similarity may stem from independent web copying—a less severe offense in many honor codes—rather than peer-to-peer collusion. This triage logic reduced false positives by an additional 0.5 percentage points in our test, flagging 3 innocent pairs that MOSS called plagiarized because both students had independently copied from a popular GeeksforGeeks snippet.
Where the Benchmark Breaks Down
No single dataset can represent every institution. Our corpus came from one university’s intro Java course with a single assignment structure. The results may shift when you add Python, C++, or multi-file projects. JPlag’s performance, for instance, improved on C++ assignments in a separate pilot because its token set normalized more aggressively, while MOSS’s relative recall dropped on object-oriented C++ due to false positives from template-heavy boilerplate. We also excluded borderline cases from the positive set, which may undersell a tool’s ability to flag suspicious work for human review.
Benchmarking your own checker directly with your student population is indispensable. If your department requires >95% recall with zero false positives, you’ll likely need to combine automated detection with a trained TA triage step, regardless of tool.
To run your own benchmark, start with a semester’s worth of anonymized submissions and a manually verified list of plagiarized pairs. Use a code plagiarism checker like Codequiry alongside MOSS or JPlag on the same inputs, then compute F1 scores as demonstrated. The difference will tell you which tool’s signal aligns best with your faculty’s judgment.
Frequently Asked Questions
How accurate is MOSS for Java assignments?In our benchmark of 2,400 Java submissions, MOSS achieved 87% recall and 80% precision with a false positive rate of 2.1%. Without base-file exclusion, false positives can exceed 15% because of starter code overlaps.
Can JPlag detect refactored plagiarism?JPlag found 68% of heavily refactored copies, missing cases where loops were reordered or methods inlined. Its precision is high (93%), making it reliable for confirming copy-paste but less sensitive to structural obfuscation.
What’s the best code plagiarism detector for CS1 courses?Our data shows that a multi-model checker using AST fingerprinting and web cross-referencing—such as Codequiry—yields the highest F1 (95.0%). In CS1 courses where students share starter code and lookup web snippets, this combination dramatically reduces false positives while catching refactored plagiarism.
How can I benchmark plagiarism checkers on my own assignments?Collect anonymized submissions, manually label known plagiarized pairs, run each tool with identical base-file exclusions, and compute precision and recall at your chosen threshold. Automate with the Python script above and repeat each semester.