In the fall of 2024, Purdue University Northwest's computer science department tripled the number of confirmed AI-generated code violations in its core data structures course. The jump came from a simple change: integrating a dedicated AI code detector alongside the token‑based plagiarism checker they had used for years. Across 150 students and eight programming assignments, the teaching team surfaced 63 substantiated AI‑written submissions — a 34% increase over the 47 suspected cases they had the previous semester, most of which could never be proved.
The key wasn't just better detection; it was combining similarity checking with AI‑specific statistical models and web‑source lookups so that no single signal had to carry the burden of proof alone.
The Detection Gap at a Regional University
Like many mid‑sized public universities, Purdue Northwest teaches the introductory CS sequence on a tight budget. CS 20100 (Data Structures and Algorithms) is a twice‑yearly course with three lab sections and a single faculty coordinator — Dr. Elaine Marks — supported by two graduate TAs. For nearly a decade, the plagiarism detection workflow was straightforward: after each due date, a TA ran all submissions through Stanford's MOSS (Measure of Software Similarity), reviewed the highest‑similarity pairs, and brought clear cases of peer‑to‑peer copying to the academic integrity board.
That workflow worked well for the copy‑paste collusion it was designed to catch. But in late 2022, Marks started seeing something new. Students were turning in code that looked clean and idiomatic, often with comments and error handling that first‑year students rarely include. The MOSS reports showed no meaningful matches against other submissions or against public GitHub repositories the department had indexed. Yet the code felt off — mechanically fluent but lacking the mistakes and personal touches Marks had learned to expect.
“We could smell it,” Marks told me in February 2025. “But MOSS doesn't have a nose. We had no tool‑based evidence, so we couldn't file cases. Students learned that if they said ‘I swear I wrote it,’ we had to back off.”
The situation crystallized in Spring 2024, when a suspiciously perfect recursive backtracking solution appeared in three different sections within the same hour. Each submission had different variable names — one used currentIdx, another pos, a third pointer — but the algorithmic structure was identical down to the ordering of edge‑case checks. MOSS gave the pair a similarity score of 12%. “That’s noise level,” Marks said. “We were flying blind.”
Why MOSS Wasn't Enough — and What Needed to Change
MOSS excels at finding large blocks of copied code between student submissions because it normalizes whitespace, variable names, and some control structures before comparing token fingerprints. That strength is also its blind spot for AI‑generated code. When a large language model produces original code for each student — even when prompted similarly — the resulting token sequences often vary enough to fall below MOSS's typical thresholds. And because MOSS is fundamentally a peer‑to‑peer and peer‑to‑canonical‑source tool, it has no facility for analyzing the statistical texture of a single file to determine whether a human wrote it.
During summer 2024, Marks and her TAs conducted a small experiment. They fed 40 known‑human assignments from Spring 2022 (before ChatGPT was widely available) and 40 submissions from Spring 2024 that they strongly suspected were AI‑generated into three different source code plagiarism checker pipelines:
- MOSS with peer‑to‑peer only: flagged 6 of the 40 AI submissions (15%) as having high similarity to another student's work, but none of those matches were structurally related to the AI; they were coincidental.
- JPlag 4.1 (token + AST): flagged 9 of the 40 (22.5%) as similar to some public code, but again, most hits were irrelevant, and the tool provided no confidence metrics on whether individual files were AI‑authored.
- Codequiry’s combined pipeline (similarity + web + AI detection): flagged 38 of the 40 AI submissions (95%) with an “AI‑DETECTED” label and a probability score, while correctly classing all 40 known‑human submissions as human. The two missed cases were heavily refactored adaptations where the student had interleaved original code with AI‑generated snippets — a scenario the detector flagged as “mixed.”
The numbers convinced the department to run a full pilot in Fall 2024. They chose Codequiry because it offered token‑fingerprint similarity, AST‑based comparison, web‑source matching against Stack Overflow and public GitHub, and a dedicated AI detection model all inside a single dashboard. Crucially, the AI detector didn't just look for perplexity and burstiness in natural‑language strings — it operated on the token streams and AST patterns of the source code itself, trained on a corpus of millions of human‑written and LLM‑generated files across Java, Python, C++, and JavaScript.
“I didn't want my TAs bouncing between four different tools and trying to synthesize reports by hand. Codequiry let us run one scan and see peer matches, web matches, and AI predictions layered on top of each other.”
Implementing AI Detection Alongside Plagiarism Checking
The integration was script‑driven. Purdue Northwest uses Canvas for LMS and collects submissions as zip archives. After each deadline, a TA exported the archive and pushed it to Codequiry’s REST API via a Python script that created a new assignment, uploaded all submissions, and triggered a scan. The scan returned a structured JSON report — similarity pairs, web source URLs, and per‑file AI probability scores — that the script then flattened into a CSV for the TAs’ review queue.
Concurrently, the team retained their MOSS workflow for a few weeks to compare outputs. The dual‑run comparison confirmed what the summer experiment suggested: MOSS caught peer‑to‑peer copying well, but the AI‑specific detections had virtually no overlap with MOSS’s high‑similarity pairs.
Within the Codequiry web dashboard, TAs saw a list of submissions sorted by a composite risk score that combined similarity percentage, web‑match count, and AI probability. Each flagged submission could be expanded to show:
- The top matching peer submissions (with side‑by‑side diff)
- Any matching web sources — including direct Stack Overflow excerpts and GitHub gist clones
- The AI detection score (0–100) and a breakdown of the top structural signals contributing to the score
- A snippet overlay highlighting the specific blocks most indicative of machine origin
What the Numbers Revealed
Over the 15‑week semester, CS 20100 students submitted 1,200 assignment files (150 students × 8 assignments). The detection breakdown looked like this:
| Detection Category | Fall 2023 (MOSS only) | Fall 2024 (Codequiry combined) |
|---|---|---|
| Confirmed peer‑to‑peer plagiarism | 23 | 27 |
| Confirmed web‑source copies (Stack Overflow, GitHub) | 0 (no tool support) | 12 |
| Confirmed AI‑generated code | 47 suspected, 12 upheld | 63 upheld |
| Total upheld academic integrity cases | 35 | 102 |
The 34% increase in confirmed AI cases (from the 47 suspected to 63 upheld) is the headline figure, but Marks emphasized that the true value came from the actionable evidence layer. “In the past, if we suspected AI, we had to schedule an office‑hours interrogation where we’d ask the student to explain their own recursion logic. It was exhausting and adversarial. Now we have a report that shows the exact lines flagged as machine‑generated, plus any matching web sources. That changes the nature of the conversation from accusation to fact‑finding.”
A deeper dive into the AI‑detected submissions revealed some patterns that instructors should watch for:
- Pressure‑drop assignments: AI use spiked on the final project (28% of submissions flagged) and the graph‑traversal lab (22%), both due near the end of the semester when students were crunched.
- Evening submission times: 74% of AI‑flagged submissions were turned in between 10 PM and 3 AM, compared with 41% of human‑flagged submissions.
- Comment style uniformity: AI‑generated code overwhelmingly used complete‑sentence comments with active voice (“We create a temporary buffer to hold…”), lacking the typical first‑year shorthand (“// buffer for temp vals”). The detector’s comment‑pattern signal was a strong secondary indicator.
Correlating AI Flagging with Web‑Source Matches
One of the most powerful workflows for the TAs was layering the AI prediction with the web‑source findings. In 18 of the 63 upheld AI cases, Codequiry’s web‑source checker also found a near‑verbatim Stack Overflow or GitHub snippet that the student had fed into a large language model. The student then asked the LLM to tweak variable names and add comments — creating code that was structurally identical to the web source but textually different enough to evade a pure fingerprint check.
For example, a suspicious Java method for balancing an AVL tree traced back to a popular GeeksforGeeks example. The student had run it through ChatGPT with the prompt “rewrite this code but change all variable names and add detailed comments.” The resulting submission passed a manual skim and showed only 22% similarity to the original source in a token‑based diff. But the AI code detector scored the submission at 94% probability, and the web‑source match confirmed provenance. The case was open‑and‑shut.
This combination — web match plus AI detection — creates a provenance chain: the web source shows where the logic came from, and the AI score shows that an LLM was used to disguise it. Neither signal alone would have been conclusive at most honor boards; together, they left little room for dispute.
Student Reactions and Policy Shifts
The department notified students on day one that all code submissions would be scanned by both plagiarism and AI‑detection software. A week‑one lecture on academic integrity now includes a slide showing anonymised flagged snippets and the exact signals the tool uses. Marks reported that the visible threat of detection changed behavior early: “By assignment three, the rate of AI‑flagged submissions dropped from 18% to 9%. Students who might have tested the boundaries after the first assignment realized we could see it.”
Still, the department made a conscious decision not to chase every AI signal. The detector thresholds were tuned so that only submissions with scores above 85% and at least one corroborating signal (web match, unusual submission patterns, or peer‑similarity overlap) were flagged for formal review. This reduced false positives — in post‑semester analysis of a known‑human control set, the false‑positive rate at that threshold was less than 1.5%.
Marks emphasized that the tool is used as a triage layer, not a judge: “Every flagged case still gets human review. We look at the code, we look at the student’s history, we talk to them. The report just gives us a place to start.”
Lessons for Other CS Departments
Other institutions considering a similar rollout can draw a few practical lessons from Purdue Northwest’s experience:
- Don’t rely on a single signal. Pure AI detectors generate false positives and miss cleverly hybridized code. Combining similarity, web, and AI checks gives you corroborating evidence that holds up in front of a committee.
- Automate the scan into your existing pipeline. The REST API integration meant TAs spent 10 minutes per assignment on scanning instead of hours hand‑reviewing suspect pairs. The CSV export plugged directly into their grading rubric.
- Use the results as a teaching tool. By showing students what gets flagged and why, you actively reduce cheating before it happens. The week‑one lecture became one of Marks’s most effective deterrents.
- Set realistic thresholds. If you flag every submission with a 50% AI score, you’ll drown in false positives and burn instructor hours. Purdue Northwest found the sweet spot above 85% with corroboration.
- Keep MOSS (or an equivalent) for peer‑to‑peer baselines, but don’t expect it to solve the AI problem. The tools serve different functions, and a combined pipeline is more efficient than running them in isolation.
For departments looking to adopt a unified tool that does all of this without stitching together five open‑source projects, a code plagiarism checker with integrated AI detection offers the fastest path. The API‑first design makes it straightforward to embed into existing LMS autograders, and the layered report means TAs aren’t left guessing which signal to trust.
Frequently Asked Questions
Can MOSS detect AI-generated code?
No. MOSS detects similarities between code submissions by comparing token fingerprints, but it has no mechanism to evaluate whether a single file was written by a human or an LLM. AI code written independently for each student rarely triggers MOSS because the token sequences vary too much.
What is a realistic false-positive rate for AI code detection?
In controlled tests with known‑human submissions, Codequiry’s AI detector returned false positives below 2% at Purdue Northwest’s 85‑threshold setting. However, any AI detector should be treated as a flagging tool, not a final verdict — human review remains essential.
How does combining AI detection with web-source checking improve academic integrity cases?
When AI‑generated code is based on copying a public snippet and then masking it, the web match provides the origin and the AI score explains the disguise. Together they create a chain of evidence that is harder for a student to deny than a single metric alone.
Is it possible to detect AI‑assisted code versus fully AI‑generated code?
Current detectors can identify blocks of code that are statistically likely to be machine‑generated, but distinguishing “student wrote the skeleton, AI filled in the methods” from “AI wrote everything” is still a research problem. Tools like Codequiry flag mixed submissions with a lower confidence score, alerting reviewers to possible partial use.
For any CS department trying to close the detection gap between human copying and machine‑generated submissions, a multi‑signal approach is the clearest path forward. Purdue Northwest’s experience shows that adding AI detection to an existing plagiarism workflow can surface a third more violations while giving instructors the documentation they need to act. Learn how an integrated AI code detector and plagiarism checker can work in your courses.