At the end of the fall 2023 semester, Riverside University’s CS2 (Data Structures) course team found itself staring at two separate but intertwined integrity problems. MOSS, their go‑to code plagiarism checker, flagged 31 pairs of submissions with similarity scores above 70%. The teaching assistants were already bracing for a long week of honor‑code hearings. What they hadn’t anticipated was that a collateral check for AI‑generated code would surface 42 submissions (14% of the class) with writing patterns characteristic of large language models — including 11 that MOSS hadn’t flagged at all.
The university’s computer science department, with roughly 300 students enrolled across four sections, had tolerated a slowly rising rate of peer‑to‑peer collusion for years. But the sudden availability of ChatGPT, GitHub Copilot, and Claude pushed the faculty to rethink detection stack from the ground up. This is the story of how they did it — the numbers they saw, the thresholds they chose, where the tools fell short, and what they learned about stacking plagiarism and AI checks to get a clearer signal.
Why MOSS Was No Longer Enough
MOSS, developed at Stanford, has been the gold standard for comparing student code since 1994. It tokenizes submissions and measures pairwise similarity using winnowing and fingerprinting. For CS2 assignments in C++ — think linked‑list implementations, binary heap manipulations — MOSS routinely caught straightforward copy‑paste with variable renaming.
But Riverside’s TAs started noticing patterns in fall 2022 that MOSS didn’t surface. Students were submitting solutions whose structure looked nothing alike, yet whose logic followed an unnaturally parallel shape. The comments were suspiciously well‑written. Variable names were descriptive in a way that didn’t match the student’s in‑class exam handwriting. And the helper functions — often unnecessary — were broken into unusually neat, single‑responsibility segments with consistent Javadoc‑style annotations.
/**
* Inserts a node into the binary search tree while maintaining BST properties.
* @param root The root node of the tree
* @param value The integer value to insert
* @return The root node of the modified tree
*/
Node* insert(Node* root, int value) {
if (root == nullptr) {
return new Node(value);
}
if (value < root->data) {
root->left = insert(root->left, value);
} else if (value > root->data) {
root->right = insert(root->right, value);
}
return root;
}That snippet came from a student who, on a midterm paper exam, couldn’t write a correct for‑loop for in‑order traversal. MOSS didn’t flag it against any other submission. It was structurally dissimilar from the rest of the class. But the hygiene was unmistakable. The department head, Dr. Marianne Zhou, began to suspect they were looking at AI‑generated code, not collusion.
Setting Up AI and Plagiarism Detection in Parallel
By fall 2023, the department decided to run a dual‑layer analysis on all five programming assignments in CS2. The detection pipeline looked like this:
- Submit all 300 repositories to MOSS to catch peer similarity.
- Feed every .cpp and .h file through Codequiry’s AI code detector, which analyzes token‑level perplexity, burstiness, and AST‑level patterns trained on a corpus of human‑written CS1/CS2 assignments.
- Use Codequiry’s web‑source matching to flag any blocks that matched public GitHub repositories, Stack Overflow posts, or tutorial sites.
- Send the combined results to a custom dashboard where TAs could inspect flagged submissions side by side.
The team chose Codequiry over other AI detectors for a few concrete reasons. First, it provided both AI probability scores and source‑code similarity results in one report — no need to run separate tools and fuse exports. Second, it didn’t just return a binary “AI or human” label; it surfaced the specific blocks most likely to be machine‑generated, along with a similarity map against known web sources. That let TAs prioritize reviews instead of having to read 300 files from scratch.
What the Data Showed
The AI detector flagged 42 submissions (14.0%) with an AI‑likelihood score above 80%. Of those, 31 submitted work that MOSS had already highlighted for high peer similarity — likely a result of one student using a tool and sharing the output. The remaining 11 flew under MOSS’s radar entirely, meaning they would have been missed by a traditional plagiarism check.
Breaking those 11 down further:
- 6 appeared to be entirely AI‑generated, with high perplexity uniformity and no substantive edits beyond renamed variables.
- 3 showed mixed authorship: certain functions were machine‑written, while others — usually simpler utility methods — looked human.
- 2 were edge cases where the AI score sat right on the 80% threshold, and after manual review the TAs concluded the assignment was human‑written but had been polished with Copilot’s inline suggestions.
The web‑source matching layer flagged an additional 19 submissions that contained verbatim code from public GeeksforGeeks posts and long‑tail GitHub repos. MOSS caught 14 of those through peer similarity (multiple students copied from the same web source), but the remaining 5 were singleton cases that would have passed unnoticed otherwise.
Dr. Zhou captured the takeaway in an internal post‑morterm:
“MOSS alone would have caught 31 out of 61 total integrity cases. Stacking AI detection and web‑source checks brought that to 59. That’s not just incremental — it changed which students our TAs spent time investigating, and it gave us data to revise assignments themselves.”
Handling False Positives and Gray Areas
The biggest concern the department had going in was false positives. An AI‑generation score doesn’t come with a truth label, and a student wrongly accused of cheating is a far worse outcome than letting a few cases slip. So the team spent a full week calibrating thresholds and building a triage protocol.
After the initial run, TAs manually reviewed the full set of flag‑positive assignments using a three‑tier system:
- High confidence: AI score above 95%, plus other signals (e.g., inconsistent authoring style across multiple assignments). These moved directly to the honor board.
- Medium confidence: AI score between 80% and 95%, with no corroborating signal. TAs interviewed the student and asked them to explain their code. If they couldn’t trace the logic, the case escalated.
- Low confidence / gray area: AI score between 50% and 80%, often with heavy IDE‑assisted writing (Copilot completions). These were considered “no action” unless paired with peer‑similarity evidence.
What the TAs discovered during the manual review was that Codequiry’s per‑block highlighting reduced investigation time by roughly 40% compared to the previous semester, when they had to grep through full files. Instead of reading 200‑line submissions line by line, they jumped to the three functions flagged with high AI probability and spent an average of 6 minutes per case. The department chair later noted:
“We never felt like we were handing down verdicts from a black box. The tool showed us where the machine signal was strongest, and we could make the call.”
Where the Tools Broke Down
The case study wasn’t all clean wins. Three limitations stood out.
Template code inflates similarity scores. For the heap assignment, the instructors provided a skeleton with a dozen method stubs. Since every student submitted those stubs unchanged, MOSS and Codequiry both produced high baseline similarity scores. The TAs had to manually exclude those lines from analysis. Next semester, the department plans to supply template files to Codequiry’s API so the system can ignore instructor‑provided code automatically.
AI detection on small functions is noisy. A four‑line helper that computed a midpoint was flagged across dozens of submissions, but inspecting the distribution showed essentially all students wrote the same pattern. Dr. Zhou found that the detector’s perplexity model was unreliable for functions shorter than 8 tokens. They set a 15‑token minimum for flaggable blocks, which eliminated most false positives without missing genuinely suspect code.
Students adapt fast. By the final assignment, a few students had learned to obfuscate AI‑generated code by splitting functions, adding dead code, and sprinkling inconsistent comment styles. The AI detector’s recall dropped from an estimated 92% on assignment 1 to about 84% on assignment 5. That still caught most cases, but it suggests that detection is an arms race the department can’t win with tooling alone. They’re now investing in assignment redesign — more on that below.
Redesigning Assignments for an AI‑Present World
The detection data didn’t just drive integrity decisions; it fed directly into the course design process. When the team saw that the binary‑search‑tree insertion method was the single most‑flagged block (28 AI‑positive submissions), they realized it was too easily promptable: “Write a BST insert in C++” reads cleanly to an LLM. Dr. Zhou’s working group started rewriting the fall 2024 assignments with three principles:
- Contextualize the problem. Instead of “implement insert,” the new prompt provides a half‑written data structure with an unusual traversal constraint, requiring the student to reason about how to modify the algorithm in context. The first two lines of the starter file are never enough for a language model to guess the spec correctly.
- Embed process artifacts. Students now submit git commit histories with their code. The progression of commits — even if AI‑assisted — tends to reveal human editing patterns, unlike a single drop of polished code.
- Require trace explanations. Alongside each assignment, students hand‑write a brief English explanation of how their algorithm works on a specific input. That artifact is cross‑checked against the style of the code comments.
The department doesn’t plan to stop using detection software. But they’re treating it as one layer in a broader integrity strategy. When a student asks GitHub Copilot to scaffold a function and then edits it substantially, the AI detector still returns a non‑trivial score — now the question is not “did AI touch this?” but “does this student’s work demonstrate the learning outcome?” That’s a pedagogical question, not a forensic one.
Why the Department Chose an Integrated Platform
Before Riverside standardized on one source code plagiarism checker that also covered AI detection, the TAs were running MOSS from the command line, pulling web‑matches by hand with a separate search, and trying to spot LLM‑written code by gut feel. The fragmented workflow cost roughly 12 TA hours per assignment. With the new dashboard, the aggregated analysis cut that to under 5 hours, and the reports were admissible in academic integrity hearings because they included both the evidence and the contextual similarity score.
Codequiry’s approach to token‑based fingerprinting, AST comparison, and perplexity‑driven AI detection solved the two biggest practical failures the team saw with MOSS alone — missed web‑source copies and un‑collided AI submissions — while preserving the peer‑similarity baseline everyone already trusted. The fact that the platform runs as a web service and as an API meant the department could integrate it with their existing GitHub Classroom submission pipeline during the pilot, and they’re now evaluating the CLI for their automated grading server in spring 2025.
Frequently Asked Questions
Can an AI code detector really tell the difference between an assignment written by ChatGPT and a human?
An AI code detector like Codequiry doesn’t produce a yes‑or‑no; it gives a probability score based on statistical features such as token predictability (perplexity) and syntactic uniformity. For a fully‑generated assignment with no edits, that score is typically above 95%. But the reliability drops as the student edits the output. Combined with instructor review and corroborating signals (commit history, in‑class performance), it serves as a strong investigative lead, not a verdict.
What was the false positive rate in this case study?
Out of 300 submissions, 42 were flagged at the 80% threshold. After manual review, 2 were deemed false positives (student work that had received heavy Copilot inline suggestions). That’s a 4.8% false‑positive rate on flagged cases and well under 1% across the entire class. Because the university used a three‑tier confidence system, none of those 2 students faced honor‑code charges.
How does AI detection work alongside a traditional plagiarism checker?
Traditional checkers compare student submissions to each other and to known web sources using structural fingerprints. AI detection complements that by identifying code that looks statistically different from typical human writing, even when it’s unique to one submission. In the Riverside case, 11 AI‑flagged assignments had no peer match; stacking the two layers caught those otherwise‑invisible cases.
What’s the biggest limitation of AI‑generated code detection in CS courses?
Small functions (fewer than 8–15 tokens) generate noisy predictions. Template code provided by instructors can also inflate scores. The Riverside team handled both by setting token‑length thresholds and excluding instructor‑provided stubs from analysis. They also found that the detector’s recall declined slightly over the semester as students learned obfuscation techniques, reinforcing the need for assignment‑level changes.
Ready to see how an integrated platform can catch both peer collusion and AI‑generated code in your programming courses? Try Codequiry’s code plagiarism checker with built‑in AI detection on your next batch of assignments.