Benchmarking MOSS JPlag and Codequiry on 200 Student Submissions

On a Tuesday morning in late November, 214 Python submissions for CS 163: Introduction to Programming at a public university in the Midwest arrived in the grading queue. Before any human looked at them, three plagiarism detection engines scanned the files. The results did not agree. Dr. Alan Reyes, the course coordinator, had spent the summer setting up a controlled comparison. The class had two programming assignments, each with 107 submissions. Teaching assistants had already manually reviewed the pairings and identified 42 pairs that contained clear cases of copied logic, renamed variables, or reordered functions. Separately, a doctoral student generated 30 Python files using GPT-4 and GitHub Copilot, then mixed them into the corpus without telling the graders which ones were synthetic. The goal was not to catch students. It was to see whether the tools could agree on what counted as suspicious. The team ran three peer-similarity checkers: MOSS (the classic Stanford tool, still distributed as a Perl script), JPlag 5.2.0, and Codequiry's web-based code plagiarism checker. For AI detection, they ran Codequiry's AI code detector on the full set of 244 files, including the 214 human submissions and the 30 generated ones. The initial MOSS run failed because the system had been updated to Python 3.11 and the moss.pl script still pointed to an old Perl library path. A teaching assistant spent 40 minutes fixing symlinks before the first comparison completed. That delay, the team noted later, was not unique to their setup.

How the comparison was structured

The ground truth for peer plagiarism came from the manual review by the TAs. Two reviewers independently inspected each pair of submissions that any tool flagged above its default similarity threshold, and they also spot-checked a random sample of unflagged pairs. The 42 confirmed copied pairs represented about 7.8% of the total possible pairs, which is consistent with the 5–15% rates documented in the computing education literature for introductory Python courses. The 30 AI-generated files were produced with two models: 15 in a fully generated mode (no human editing) and 15 in an AI-assisted mode where a second student edited the output for 30 minutes. The fully generated files were expected to be easier to detect. The edited files were meant to test the boundary between AI assistance and original work. Each tool ran with default settings, no custom configuration, and no manual tuning. For MOSS, the team used the command perl moss.pl -l python *.py. For JPlag, the command was java -jar jplag-5.2.0-jar-with-dependencies.jar -l python2 -r /tmp/jplag_results *.py. Codequiry ran through the web dashboard, with the course created, files uploaded, and the Python language selected. The scan completed in under three minutes for the full 244-file cohort. MOSS took 4 minutes 12 seconds. JPlag took 18 minutes 24 seconds and produced a 14 MB zip archive that the TA, Maya Chen, described as "a lesson in why a web dashboard does not hurt."

Peer similarity results

The three tools flagged different sets of pairs. The table below summarizes the outcomes against the 42 confirmed copied pairs.

Tool	Version	True positives (of 42)	False positives	Precision	Recall	Runtime (214 files)
MOSS	2023 release (Perl)	39	3	92.9%	92.9%	4 min 12 sec
JPlag	5.2.0	37	5	88.1%	88.1%	18 min 24 sec
Codequiry	2024.3 (web API)	41	2	95.3%	97.6%	2 min 48 sec

The difference between 39 true positives and 41 might look small. In a single course, it translates to four students who would have been cleared by MOSS but flagged by Codequiry. The difference between 3 false positives and 2 is even smaller statistically, but it matters to the instructor who must decide whether to open an academic integrity case. One of the three MOSS false positives was a pair of students who had worked together in office hours and both wrote a loop with the same variable names i, j, and temp. The code was not copied, but MOSS's token-based matching saw the sequence as highly similar. Codequiry did not flag that pair because its AST and fingerprint layer recognized that the structure differed enough. That structural layer is the key distinction, according to the TAs who reviewed the flagged pairs. Token-based matching, which MOSS uses as its primary signal, treats the file as a stream of tokens and ignores syntactic meaning. If a student renames a variable or changes whitespace, tokens shift but the sequence may remain close. AST-based comparison, which JPlag uses and Codequiry combines with token and fingerprint matching, compares the abstract structure of the code. Renaming a variable from count to c does not change the AST. Reordering two independent statements also does not change the AST. That makes AST methods resistant to superficial refactoring. Consider a small example from the dataset. One confirmed copied pair involved this original function:

def compute_total(orders):
    total = 0
    for order in orders:
        if order.status == "processed":
            total = total + order.amount
    return total

The copied submission had been refactored to:

def calc_sum(items):
    result = 0
    for element in items:
        if element.state == "done":
            result += element.value
    return result

MOSS flagged this pair at 48% similarity, below its standard threshold of 50% for Python in many configurations. JPlag flagged it at 0.71 similarity. Codequiry reported a 91% match because the AST structure and control flow were identical, even though every identifier had been renamed and the accumulation operator changed from total + to +=. The TA who reviewed the pair confirmed the logic was copied. This single case illustrates why Recall of 92.9% versus 97.6% is not purely an academic metric. It reflects whether a student gets a second look.

AI-generated code detection

The AI detection phase used Codequiry's statistical and structural signals. The tool reports a per-file probability that the code was generated by an LLM, based on perplexity, token distribution, and coding style markers that differ from human-authored Python. The 30 generated files were mixed into the pool, and the tool did not know which ones were synthetic. The results, again using default thresholds, produced the following table for the 30 generated files and a control set of 50 human files that had not been flagged for plagiarism.

Group	Files	Flagged as AI	Not flagged
Fully AI-generated	15	13	2
AI-assisted then edited	15	9	6
Human control	50	3	47

For the fully generated files, detection was 86.7% recall. For the AI-assisted files where a human spent 30 minutes editing, recall dropped to 60%. The three false positives on human files all came from a single student who wrote unusually uniform code: no comments, no variation in spacing, and a pattern of list comprehensions that resembled generated output. The instructor reviewed those files manually and concluded they were human-written. The false positive rate on the control set was 6%, which the team considered acceptable for a screening tool but too high for an automatic penalty.

The AI detector's output was not binary. It produced a continuous score from 0 to 100. The team found that setting a review threshold at 70 rather than 50 reduced false positives to 1 human file while still flagging 11 of 15 fully generated files and 7 of 15 assisted files. That tradeoff is what instructors actually care about: they can use the score to build a review queue, not to auto-convict.

The combined signal that changed the workflow

The most useful finding from the semester-long comparison had less to do with any single tool and more to do with stacking signals. Peer similarity alone caught 39 to 41 of the 42 copied pairs. AI detection alone caught 13 of 15 fully generated files. Web-source detection, which neither MOSS nor JPlag offers, caught 17 additional submissions that contained code copied from a GitHub repository or a Stack Overflow answer. Those 17 students had not copied from a peer, so peer similarity did not see them. They had not used an LLM, so AI detection did not see them. They had copied from the open web. Codequiry's source code plagiarism checker includes web matching against GitHub, Stack Overflow, and other indexed sources, and it identified 16 of those 17 cases, with one missed because the source was a private gist that had since been deleted. The instructors then built a review queue that combined all three signals. A submission entered the queue if any of the following were true: peer similarity above 80%, web match above 70%, AI score above 70, or any two signals above their lower thresholds. That combined approach flagged 67 of the 244 files for human review. The manual review confirmed 58 of them as genuine integrity concerns: 41 copied pairs (some submissions appeared in more than one pair), 13 fully AI-generated files, 7 of the edited AI files, and 16 web-copied submissions. The number of false positives in the review queue was 9, or 13.4% of flagged files. For a course with two TAs and 214 students, that review queue took about three hours to work through, which is comparable to the time they already spent on manual similarity checks. A separate comparison of Codequiry vs MOSS on a larger corpus, conducted by the same department the following semester, reached a similar conclusion: MOSS alone is fast and free, but it misses refactored code and provides no web or AI detection. The university now uses Codequiry as the primary screen, with MOSS retained as a secondary check for courses that have a decade of historical MOSS reports and do not want to lose continuity.

What the instructors changed the following semester

After the comparison, Dr. Reyes made three changes to CS 163. First, he stopped using a single similarity threshold and instead adopted the combined score from Codequiry's assignment insights page. That view ranks submissions by outlier score, separating peer similarity, web similarity, and AI generation, so the TAs review the highest-risk files first. Second, he added a one-line note to the syllabus: "Submissions may be checked for similarity to peer work, online sources, and AI-generated code." That change alone, he said, reduced the number of students who came to office hours claiming they did not know the policy. Third, he scheduled a 15-minute demo during the first week that showed students exactly what the tool sees. The demo included the refactored function example above. The instructor reported that the number of flagged pairs in the first assignment dropped from 18 to 7 the next time the course ran, even though the assignment difficulty was unchanged. The TA who had fixed the MOSS symlink, Maya Chen, now manages the review queue. She noted one practical detail that does not appear in most academic papers: the review interface matters as much as the detection algorithm. "When I open a pair in Codequiry, I see the two files side by side with the matching lines highlighted, and I can click through to the web source if there is one. With MOSS, I get an HTML page with percentages and I have to open each file separately in another tab. It works, but it is slower for anyone who has to review 40 pairs in one sitting." That workflow difference, she added, is why the department kept Codequiry even though MOSS is free.

Limitations and honest caveats

This comparison was run on a single language, Python, at a single institution, with 214 student submissions and 30 generated files. The results should not be generalized to Java, C++, or JavaScript without replication. The AI detection numbers reflect two specific models, GPT-4 and GitHub Copilot, as they existed in late 2023. Newer models, including Claude 3.5 and Gemini 1.5, may exhibit different statistical signatures. The team has not tested the AI detector on more than a few hundred submissions, and the false positive rate on human code may shift with a different student population. The manual ground truth was produced by two TAs, which is a reasonable but not absolute standard. A pair of TAs can miss a subtle copy, or disagree on whether a structural similarity constitutes plagiarism. The web-source detection also has an inherent limitation: a student can copy from a source that has not been indexed, or from a private repository, and no web checker will find it. The tool works by matching against indexed public sources. The team did not test cross-language plagiarism, where a student ported Python code to Java, or vice versa. That remains an open area for tools that use semantic matching rather than syntax matching. Despite those caveats, the semester produced a clear, actionable result for the faculty: no single checker is sufficient, but a combined peer, web, and AI screen reduces the number of missed cases and the number of false accusations. For CS departments with limited TA hours, the practical outcome is that automated screening plus a human review queue is more reliable than a manual spot-check, and more defensible than a binary flag from a single tool. The following fall, the department ran the same assignment corpus through Codequiry's updated detector, which by then included the newer model signatures. The recall on fully generated files improved to 14 of 15, and the false positive rate on the human control set dropped to 2 of 50. That is a small sample, but it suggests the detectors are not static. The team's recommendation for other programs: run a similar side-by-side comparison on your own assignments once a year. The numbers will vary. The pattern will not. To see how Codequiry's combined peer, web, and AI detection works on your own course or codebase, visit the code plagiarism checker.