In 1994, Alex Aiken’s MOSS paper appeared at the SIGCSE technical symposium and gave programming instructors something they’d never had before: automated, pairwise token-based similarity scoring that could surface suspicious student submissions without anyone having to diff source files by hand. State University of Plains, a public R2 with about 800 CS enrollments, adopted MOSS within a year of its public release. For almost three decades, the department’s workflow was stable — run MOSS nightly on the week’s submissions, manually inspect pairs above the 70%-similarity threshold, and let a code plagiarism checker handle the rest. Faculty felt confident they were catching the straight copy-pasters, the variable-renamers, and the control-flow-rearrangers. Then the 2022 fall semester arrived and broke that confidence completely.
This is not a disaster story. No honor-code meltdown, no mass expulsion. It’s the quieter, more instructive arc of how a single CS department incrementally rebuilt its academic-integrity tooling to answer a question that MOSS was never designed to ask: was this code written by a human at all?
The MOSS Era (1994–2022)
For years, the department treated MOSS like a noisy but indispensable instrument. They knew it couldn’t catch everything. Students who inserted dummy loops, split functions into tiny stubs, or moved logic across files often slipped below the similarity threshold. But the tool kept the most brazen copying in check. TAs spent perhaps four hours per course each week reviewing flagged pairs, and the honest students at least believed there was a system watching.
In 2019, one instructor experimented with JPlag’s AST-based mode on a data-structures course, hoping to catch refactoring tricks that MOSS missed. The experiment added another 90 minutes to grading time and surfaced a handful of extra cases, but the instructor ultimately dropped it — the effort-to-yield ratio felt poor when the department was already understaffed on TA hours. A few faculty members watched the Dolos project with interest but didn’t adopt it. The prevailing view was that MOSS was enough, and the real problem was the copy-paste culture, not something radically new.
When the pandemic pushed everything online, cheating anxiety spiked, but the forms of misconduct remained legible to similarity-based tools. Students shared solutions over Discord, posted on Chegg, and repurposed GitHub repositories. MOSS caught many of those cases because the underlying code was shared in a traceable way — tokens matched, even if comments were stripped. The department also began using a source code plagiarism checker that could search public GitHub and Stack Overflow, catching web-sourced plagiarism that MOSS alone would miss. By 2021, the department felt reasonably fortified.
The First AI-Generated Assignments
In late September 2022, a TA grading a junior-level algorithms course noticed something odd. A student’s solution to a dynamic programming problem (longest common subsequence in Python) included a @functools.lru_cache decorator — a pattern that had never appeared in any lecture, lab, or textbook example for that course. The recursive structure was unnervingly clean, with inline comments that read like a textbook but didn’t match the student’s previous writing style. A manual MOSS run against the course corpus returned a 23% similarity to the next-closest submission — well below the department’s usual threshold.
Over the following week, similar anomalies surfaced in three other courses. In an operating-systems project that required students to implement a basic shell in C, one submission contained a sigaction setup with a meticulously correct sa_sigaction handler, including a #define _POSIX_C_SOURCE guard that none of the other 47 students had used. The code compiled flawlessly on the first attempt, something that happened in perhaps 5% of submissions historically. The TA who first flagged it said the style felt “professional, but not in a student way — it looked like polished documentation examples glued together.”
By mid-October, the chair of the undergraduate committee estimated that roughly 15% of submissions across the department might involve some form of AI assistance, based on manual review of suspicious cases. But manual review alone couldn’t scale, and MOSS was useless against code that had no peer-sourced doppelgänger to compare against.
The fundamental challenge had flipped. Similarity detection assumes two submissions share a common ancestor; AI-generated code is often unique in its surface structure even when it fulfills the same specification. The department’s trust in token-based scoring had been built on the assumption that cheating meant copying, not composing from scratch with a language model.
Statistical Anomalies: Perplexity and Burstiness
The instructors started reading everything they could find on AI-text detection, but code presented a different kind of signal. Tools like GPTZero and Originality.ai focused on natural-language perplexity and burstiness — metrics that measure how predictable a sequence of words is and how much that predictability varies through the text. Human writing tends to have irregular burstiness, with sharp spikes in unexpected word choices; LLM output tends toward a more uniform, high-probability distribution. But does that translate cleanly to code?
Dr. Marisol Vance, the algorithms instructor, ran an informal experiment. She took the 12 most suspicious assignments from her course and computed token-level perplexity using a small GPT-2 model fine-tuned on Python. The AI-suspected submissions showed significantly lower mean perplexity (16.3 vs. 22.7 for human controls) and notably narrower variance. But there was a problem: the students who were strongest at writing clean, idiomatic Python also produced lower perplexity scores. A false-positive risk that could cost an earnest student a referral to the honor committee was unacceptable.
Vance then tried a complementary signal: conditional probability dispersion across function boundaries. Human programmers tend to vary their coding style between helper functions — one might be terse and imperative, another verbose with error handling. LLM-generated code, she found, sustained a more even stylistic profile across an entire file. By combining token perplexity and inter-function variance into a simple logistic regression, she could separate AI-suspected from human submissions with about 79% accuracy on a small validation set. But that still meant one in five human students would be wrongly flagged in the worst-case interpretation, and the model’s confidence was too low to act on without additional evidence.
“Perplexity alone is a terrible gatekeeper,” Vance told the department’s curriculum committee. “But if you combine it with something else — especially something orthogonal like structural similarity to known sources — you start to get a picture you can actually work with.”
Layering Signals: Similarity + AI + Web Checks
By spring 2023, the department decided to move from single-detector methods to a layered approach. The core idea: no single metric was trustworthy, but when multiple independent signals converged on the same submission, the case for AI involvement became much stronger. The stack they built looked like this:
- Peer similarity (MOSS/JPlag) — would still catch copy-paste plagiarism, the old enemy.
- Web-source matching — a check against GitHub, Stack Overflow, and public tutorial sites, catching boilerplate and cloned snippets.
- AI-generated code detection — a statistical model tuned for source code, not prose, trained to spot the structural fingerprints of LLM-written output.
The department evaluated several platforms that promised to detect AI-written code. Most early entrants in 2022–23 were natural-language detectors hastily repackaged for code, and their accuracy on Python and Java was poor — F1 scores below 0.6 in internal testing. A few newer tools, including the detector built into Codequiry, showed more consistent performance because they modeled code-specific features: syntactic structure stability, comment-to-code style alignment, and the kind of exhaustive error-handling that students rarely write but LLMs love to include. Crucially, them offering a AI code detector that didn't require sending code off to a black-box API — submissions stayed self-contained within the institution’s own workflow, a detail the university’s data-privacy officer insisted on.
What made the biggest difference, though, wasn’t the AI detector in isolation. It was the ability to view AI-detection scores alongside peer-similarity percentages and web-source matches in a single report. When a submission scored high on AI likelihood and also flagged for web-copied code (the classic pattern of an LLM assembling snippets from public tutorials), the combined weight was enough to justify a closer review. Submissions that only triggered one signal — perhaps a student whose clean style produced low perplexity — could be routed for a lighter look instead of an automatic flag.
The deployment was not a smooth curve. The first semester of layered detection (Fall 2023) generated 2.7 times more flagged submissions than MOSS alone ever had. TAs burned hours investigating false positives, mostly students who wrote unnaturally uniform code because they had memorized design patterns from textbooks. The committee responded by introducing a tiered review system: an automated combination score under 60 meant no flag; 60–80 triggered a TA check focused on clarifying questions rather than accusation; above 80 triggered a faculty-led review with the student invited to explain their work. The process added about 12 hours of combined TA time per course per semester, but faculty felt it was manageable — and it was certainly more manageable than prosecuting a full-blown honor-code crisis after grades were already posted.
Where the Numbers Landed
By Spring 2024, with three semesters of data behind them, the department had a clearer picture. Across all undergraduate courses, the combined detector stack flagged roughly 31% of submissions as having high AI-generation probability (either alone or combined with peer/web similarity). After manual review, around 78% of those high-confidence flags were confirmed as either fully AI-generated or heavily AI-assisted beyond the department’s acceptable-use policy. About 3% of human-written submissions were incorrectly flagged as high-confidence AI — though all were resolved without formal charges once the student demonstrated their process. The remaining 19% sat in a gray zone where the evidence was strong but not dispositive; most of those students received a warning and a note about proper attribution of AI assistance.
One important finding: the false-positive rate was markedly lower for assignments in C and Java compared to Python. The department’s theory is that Python’s standard library and idiomatic patterns are so extensively represented in LLM training data that the line between “Pythonic human” and “Pythonic LLM” is inherently blurrier. Instructors in the Python-heavy intro courses adjusted by setting slightly higher thresholds for AI detection and relying more on in-class follow-up exercises to validate authorship.
Interestingly, the rate of old-fashioned peer-to-peer plagiarism also declined — from an average of 8% per course to 4%. Faculty suspect this was partly because students realized the detection system had become broader and more sophisticated, and partly because some would-be copiers had simply shifted to AI tools instead of copying classmates. That didn’t eliminate the problem; it just relocated it to a different signal path.
What the Instructors Actually Look For Now
Vance described the new rhythm in a note to her colleagues: “I don’t open MOSS first anymore. I open the combined report, sort by highest AI score, and look for submissions that also have at least one web match. That’s my high-priority queue. Then I look at the remaining high-AI submissions, and finally the high-similarity leftovers.”
The markers that instructors told us they’ve learned to recognize independently (often before the detector even flags them) include:
- Consistent, explanatory inline comments that don’t match any student’s previous comment style — the telltale “explain the obvious” habit of LLMs.
- Overly defensive error-checking for edge cases that were never mentioned in the assignment specification.
- Use of language features or libraries that appear in official documentation but haven’t been taught in class —
functools.partialin a course that only coveredmapandfilter, for instance. - A flat stylistic fingerprint across an entire file: no shortcuts, no trace of debugging detritus, no unused imports left behind.
// AI-generated example from an Operating Systems shell assignment
// Students never write sigaction this cleanly.
#define _POSIX_C_SOURCE 200809L
#include <signal.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h>
void sigchld_handler(int sig) {
int saved_errno = errno;
pid_t pid;
while ((pid = waitpid(-1, NULL, WNOHANG)) > 0) {
printf("Child process %d terminated.\n", pid);
}
errno = saved_errno;
}
The difference from human student code isn't just correctness — it's the absence of struggle. A typical student's sigaction implementation would contain commented-out attempts, a few unnecessary includes, and a handler that forgets to save and restore errno. What the LLM produces is defensive, idiomatic, and unnaturally complete.
Reflections on a Thirty-Year Arc
It’s worth stepping back to appreciate how strange this moment is for CS educators. For decades, the central challenge of code-integrity detection was identifying when one submission was derived from another. The underlying assumption — that cheating meant copying — held so firmly that the entire tooling ecosystem was built around it. MOSS, JPlag, Dolos, and every commercial derivative work on the principle that suspicious code looks like other code. AI-generated code inverts that: it looks like nothing you’ve ever seen, yet it bears an eerie resemblance to a Platonic ideal of the solution.
The department’s experience suggests that the way forward isn’t to replace similarity detection with AI detection, nor to treat them as independent pipelines. The signal multiplies when they’re combined. A submission that is simultaneously AI-suspicious and similar to a GitHub repository carries different weight than one flagged on only one dimension. The tool that lets you see both at once — and ideally adds web-origin checks as a third axis — is more useful than any single-detector point solution.
“We spent twenty-eight years optimizing for one kind of academic dishonesty, and then the floor shifted,” said the department chair. “The students who’d have been copy-pasting three years ago are now prompting a model. That’s not a moral shift — it’s a workflow shift. Our detection has to reflect that.”
For the 2024–25 academic year, the department plans to extend its layered pipeline further, integrating the detect code plagiarism and AI-detection checks directly into its autograding scripts via API calls. The idea is to give TAs a pre-sorted queue every Monday morning, with the most suspicious submissions already highlighted and ready for human judgment. They’re also piloting a mandatory in-class “code explanation” exercise for any submission flagged above a certain threshold — not as punishment, but as a lightweight verification step that turns out to be a useful pedagogical tool on its own.
The big lesson is that no single detector, score, or statistic is trustworthy enough to accuse a student. But a constellation of independent signals, reviewed by humans who know what student code normally looks like, can catch the vast majority of AI-generated submissions without burning the honest students in the process. State University of Plains didn’t invent that idea — but after three semesters of trial and error, they’ve shown that it can be implemented at scale in a real CS department with a modest TA budget and a willingness to rethink thirty-year-old assumptions.
Frequently Asked Questions
Can MOSS detect AI-generated code?
No. MOSS compares submissions against one another using token-based similarity. Since AI-generated code is often structurally unique and doesn’t share a common ancestor with a peer submission, it rarely triggers MOSS’s similarity thresholds. AI detection requires a different set of statistical and structural methods that MOSS was never designed for.
What’s a realistic false-positive rate for AI code detectors?
It varies by language and detector. In the department’s experience, a layered approach that combines AI detection with peer similarity and web-source matching can hold false positives to around 3% — meaning about 3 in 100 human-written submissions may be incorrectly flagged. Most of those are resolved through a brief manual review process rather than formal charges.
Why is AI-generated Python harder to detect than Java or C?
Python’s standard idioms are so heavily represented in LLM training data that the boundary between “Pythonic human” and “Pythonic LLM” is naturally blurrier. C and Java tend to have more varied stylistic signatures among students, making the contrast with LLM-generated code easier to spot.
Is it better to use a single tool or multiple tools?
Multiple independent signals are far more reliable than any one metric alone. The department’s most effective workflow combines peer-similarity scoring, web-source matching, and AI-specific code detection. When several orthogonal signals converge on the same submission, the case for misconduct is much stronger than when a single score suggests suspicion.
Start building a layered detection workflow for your own courses with the tools that combine AI detection, peer similarity, and web origin tracking in one place — try the AI code detector that State University of Plains eventually made part of its nightly pipeline.