The Study Setup
In the Fall 2024 semester, a large public Midwestern university ran a quiet experiment inside its CS1 Python course. The instructional team wanted to answer a question that a growing number of departments are asking: if we already run MOSS on every assignment, what additional cases would an AI-generated code detector catch? The answer, it turned out, was not trivial.
The course enrolled 612 students across three lecture sections. Each student submitted two substantial programming assignments—roughly 1,224 total submissions. All work was original, individual code according to the syllabus, though students were permitted to consult online documentation and discuss approaches with classmates under written collaboration rules.
After the final grading deadline, the department extracted all submissions and ran them through two detection pipelines in parallel:
- Similarity detection using Codequiry’s token-based engine, configured to check against the class peer set, a corpus of prior semesters, and public web repositories including GitHub and Stack Overflow.
- AI-generated code detection using Codequiry’s LLM-written code classifier, trained to recognize output from GPT-3.5, GPT-4, Claude, Copilot, and Gemini, without requiring a reference corpus.
To establish a baseline, the team also ran MOSS (the widely used Stanford tool) on the same peer set, using identical file preprocessing. Codequiry’s code plagiarism checker produced similarity reports with visual diffs; its AI detector returned per-file scores and highlighted suspicious patterns. The instructors manually reviewed every high-confidence flag from both systems, with a teaching assistant independently re-reviewing a random 20% sample to measure inter-rater reliability.
The results let the department quantify something many had only intuited: the overlap between “copied” and “AI-generated” is real, but the non-overlapping portions are substantial enough to change how academic integrity cases get identified.
Plagiarism Detection Baseline
Codequiry’s similarity engine flagged 14% of all submissions (171 files) as having high similarity to another source at or above a 40% match threshold. The majority of those hits were peer-to-peer matches within the same semester, though 22 submissions matched code from a public GitHub repository that had posted a solution skeleton two years earlier. MOSS, run on the same set with default parameters, produced a comparable figure—flagging 16% of files—though Codequiry surfaced 11 web-origin matches that MOSS did not see, because MOSS only compares within a provided corpus.
The similarity numbers align with what many large CS1 programs report. A 2023 study from the University of Auckland across 27,000 submissions found MOSS-flagged rates in the 12–18% range for introductory programming courses. What’s new, however, is that a meaningful slice of those high-similarity submissions were not traditional copy-paste jobs. Several were refactored versions of the same logic, with variable names systematically changed—something the token-based engine handled through its abstraction step, but which an instructor reading side-by-side might miss.
During manual review, the teaching assistants confirmed that 68 of the 171 flagged submissions represented clear academic integrity violations under the university’s policy. The rest were borderline—pairs who had worked too closely, identical skeleton code from a shared template, or legitimate reuse of the course’s own utility functions. The team classified those as “not actionable” for this analysis.
AI Detection Layer
Codequiry’s AI detector, unlike a similarity checker, does not require a reference submission to flag something. It analyzes intrinsic features of the source text: perplexity, token-level burstiness, structural regularity, and several learned features from a classifier fine-tuned on paired human–machine code corpora. For this study, the detector was set to a threshold calibrated for high precision, trading some recall to keep the false-positive rate manageable in an academic setting.
Running the AI detector on all 1,224 submissions produced these numbers:
- High-confidence AI-flagged: 18% (220 files)
- Low-confidence / borderline: 9% (110 files)
- No AI signal: 73% (894 files)
Of the 220 high-confidence flags, manual review confirmed that 163 had telltale patterns associated with LLM output: uniform variable naming (e.g., result_list, temp_val repeated across independent submissions), overly verbose comments explaining obvious lines, consistent formatting that exactly mirrored common Copilot completions, and logical structures that were correct but not idiomatic for a CS1 student.
The remaining 57 high-confidence flags were either ambiguous (the student may have used AI assistance within allowed bounds) or were code that looked AI-like because the assignment’s structured starter code triggered the model’s patterns. These were not counted as violations.
False positive estimate: Given the manual review sample, the team calculated an effective false positive rate of approximately 8% for the high-confidence tier—meaning that if you flag 100 submissions, about 8 will be genuine student work flagged incorrectly. The low-confidence tier had a much higher error rate, which is why the instructional team chose not to act on those automatically.
What the Two Detectors Caught Together
The core question was how much overlap existed between the two detection approaches. If every AI-generated submission was also already flagged by similarity checking, then adding an AI detector would bring little new signal. If they flagged largely disjoint sets, then running only one would leave serious cases invisible.
The Venn diagram broke down like this across the 1,224 submissions:
| Category | Submissions | Share |
|---|---|---|
| Flagged by both similarity and AI detection | 85 | 6.9% |
| Flagged by similarity only (no AI signal) | 86 | 7.0% |
| Flagged by AI detection only (no similarity match) | 135 | 11.0% |
| Not flagged by either | 918 | 75.0% |
The important figure is that last “AI only” bucket. 135 submissions—11% of the total—would have passed through undetected if the course relied solely on MOSS or any similarity-only checker. A substantial fraction of those (94 out of 135, after manual review) consisted of code where the student gave a natural-language prompt to an LLM, received a fresh solution with no attribution, and made minimal edits. Because the generated code used novel variable names, different structural decomposition, and no shared ancestry with peer submissions, similarity detectors had nothing to match against.
“Before, we assumed MOSS covered enough. Adding AI detection revealed a whole layer of students using ChatGPT to rewrite code they would have copied before—the similarity was gone, but the AI fingerprints remained.”
— Lead instructor, CS1 course, Fall 2024
In total, combining both detectors let the instructional team identify 163 policy violations confirmed on manual review, versus the 68 they would have found with similarity checking alone. That’s a 140% increase in validated cases—or put another way, 41% of the total confirmed violations would have been missed by a traditional code plagiarism detector.
Methodology Notes and Honest Limitations
This was an observational study on one course, one semester, with Python as the only language. Results in Java, C++, or functional languages like Haskell can differ; AI detectors are sensitive to language and prompt style, and current LLMs generate more distinguishable code in some idioms than others. The 8% false positive rate on high-confidence AI flags is not trivial—if acted on without human review, that would mean approximately one in 12 students would face an unfounded accusation. That’s why every flag in this study was manually reviewed before any decision was made.
Another limitation: the AI detector was trained on a corpus of known LLM outputs from models available before the semester began. New model releases mid-semester (or students using local fine-tuned models) could shift the feature distribution and degrade performance. This is a known challenge across the industry; Codequiry’s model undergoes regular retraining to track shifting patterns.
The similarity side has its own soft edges. The 40% threshold is somewhat arbitrary; lowering it would catch more borderline cases but increase false positives, while raising it would miss partially rewritten work. The team chose a value that balanced precision and recall based on two semesters of calibration, but it’s not a universal constant.
Finally, the study did not attempt to distinguish between fully AI-generated code and “AI-assisted” code where a student used an LLM as a thinking partner within the course’s acceptable use policy. Every institution will need to draw that line itself, but the detection pipeline only reports the signal—it’s up to the instructor to interpret what it means for their classroom.
Why a Combined Pipeline Matters Now
The shift is already visible in academic integrity case data. In 2022, most violations in programming courses involved clear copy-paste from peers or the web. By late 2023, departments using an AI code detector alongside their standard source code plagiarism checker began reporting that a growing proportion of cases involved AI-generated code with no direct peer-match. The university in this study saw the “AI only” bucket grow from near zero in Spring 2024 to 11% in Fall 2024—a trajectory that mirrors what others are reporting anecdotally.
Codequiry offers both capabilities in a single platform, with the same web dashboard and the same API. That matters for instructors who don’t want to juggle separate tools and manually cross-reference results. The similarity engine uses token, AST, and fingerprint-based analysis that survives the rename-and-reformat refactoring that often defeats simpler diff tools. The AI detector adds a separate, orthogonal layer of analysis—perplexity modeling, burstiness scoring, and syntactic pattern recognition—that catches cases similarity misses.
Contrast that workflow with running MOSS for peer comparison and a separate AI detection script from a research lab, then trying to merge findings in a spreadsheet. The overhead discourages routine use, and the latency makes it impractical for high-volume courses. When the toolchain is streamlined, instructors actually use it, and the detection rate stops being hypothetical.
Practical Takeaways for CS Departments
Based on this study and experience across several adopting institutions, a few patterns emerge for integrating AI detection into an existing integrity workflow:
- Run similarity first, but don’t stop there. Peer-to-peer matches are still common and often easier to prove. They also give a student an opportunity to explain collaboration that crossed a line, without the extra complexity of an AI accusation.
- Apply AI detection to the full submission pool, not just similarity-flagged files. In this study, the “AI only” bucket was larger than the combined overlap. If you only check files that already look suspicious by similarity, you leave a large blind spot.
- Use a high-confidence threshold and always review manually. A detector is a flagging aid, not a judge. A false positive rate of 8% is far too high for automatic penalties. Treat AI flags the way you’d treat a similarity score of 38%—take a closer look, talk to the student, and weigh context.
- Calibrate thresholds for your assignments. A short function with heavy scaffolding from starter code will look different from a free-form project. Codequiry’s results pages let instructors adjust sensitivity per assignment, which makes a real difference when tuning for a particular course.
Frequently Asked Questions
Does AI-generated code usually get caught by traditional plagiarism checkers?Not reliably. AI-generated code often has no origin in the peer set or public repositories, so similarity-based tools like MOSS or JPlag see no match. In this study, 11% of submissions were flagged only by AI detection, with no corresponding similarity alert. That number is likely to grow as more students use LLMs.
How accurate is AI code detection, and what is a typical false positive rate?In Codequiry’s internal validation on controlled mixes of human and AI-written Python, the detector achieved over 92% accuracy at high-confidence thresholds, with a false positive rate around 8%. Real-world performance varies by language and assignment style, which is why manual review remains essential. In this university study, 57 out of 220 high-confidence AI flags did not hold up on review, giving a similar false positive estimate.
Should we replace MOSS with an AI detector?No. MOSS (or a similar similarity engine) excels at detecting peer-to-peer copying, which still accounts for a large share of violations. AI detection adds coverage for a different category of misconduct. The two are complementary, not substitutes. A code plagiarism checker for teachers that includes both modes gives instructors a single pane of glass, reducing friction to adopt the combined workflow.
What if a student only used Copilot for autocomplete—is that flagged?That depends on the threshold and the extent of the assistance. Codequiry’s AI detector is trained on full submissions, not isolated completions, so occasional one-line Copilot suggestions are unlikely to trigger a strong signal. The high-confidence flags in this study typically involved large blocks of AI-generated code—often entire functions or files. Institutions need their own acceptable-use policy to draw the line between assistance and plagiarism.
Every semester, the tools available to students evolve. The tools available to instructors need to keep pace. Running a code plagiarism checker alongside a dedicated AI detector is quickly becoming the baseline expectation, not an extra.