Last fall, our CS2: Data Structures course served 230 students across five programming assignments in Java. For years, we relied on MOSS to catch outright copying — renamed variables, rearranged loops, tweaked whitespace. But by mid‑semester, a handful of submissions felt off in a way MOSS wasn't equipped to flag: no close peer matches, not even suspiciously low similarity scores. The code was simply too perfect, too un‑student‑like to be written at 2 a.m. We decided to run an experiment and layer an AI‑generated code detector across the very same submissions, side by side with MOSS. Here is what we found, what surprised us, and where we think tooling needs to go next.
Why combine MOSS and an AI code detector?
MOSS (Measure of Software Similarity) has been the gold standard in CS education for decades. It excels at finding code that shares the same origin — two students who copied from each other, or one student who took a skeleton and passed it around. It does this by tokenizing source files, building fingerprints from n‑grams, and detecting suspiciously long overlapping sequences. That approach is remarkably resistant to simple refactoring, but it has a blind spot: identical assignments generated independently by an LLM from the same prompt, with neither student collaborating, produce structurally different enough code that MOSS often gives them a pass.
Meanwhile, an AI code detector works on a different axis entirely. Instead of hunting for shared‑ancestor copy‑paste, it models the stylistic entropy of the code itself — the statistical fingerprint of how an LLM chooses tokens. That means it can flag submissions that look original relative to the rest of the class but whose token‑by‑token probability distribution screams GPT‑4 or Claude Sonnet. The combination, we hypothesized, would catch two very different kinds of dishonesty without forcing us to become full‑time forensic investigators.
Setting up the detection pipeline
We didn't want to change our existing workflow radically. MOSS was already integrated through a simple script that collected zipped directories per assignment and uploaded them via the moss‑script command‑line tool. To add AI detection, we used the Codequiry REST API, which accepts source files and returns an AI‑authorship probability score alongside a detailed report. Our TA wrote a short Python wrapper that ran after the MOSS batch each week:
import requests, os
CODEQUIRY_API_KEY = os.environ["CODEQUIRY_KEY"]
ASSIGNMENTS = ["a1", "a2", "a3", "a4", "a5"]
for a in ASSIGNMENTS:
with open(f"submissions/{a}/student42.java", "rb") as f:
r = requests.post(
"https://api.codequiry.com/v1/ai/detect",
headers={"apikey": CODEQUIRY_API_KEY},
files={"file": f}
)
result = r.json()
if result["confidence"] > 0.85:
print(f"High AI confidence for {a}/student42.java: {result['confidence']:.2f}")
We ran every single submission through the detector — not only flagged near‑duplicates from MOSS. The per‑call cost was modest, and we wanted the full picture. Total runtime for the largest assignment (180 submissions, average ~280 lines of Java) took under four minutes on a single machine. No student data left our campus network beyond the source files themselves, which was a non‑negotiable requirement from our IT security office.
What the AI detector found that MOSS didn't
Across all five assignments, MOSS flagged 62 unique pairs or groups with similarity above its default threshold of 30%. After manual review by two TAs, 41 of those groups confirmed as likely collaborative plagiarism — a fairly typical rate for a large introductory‑intermediate course.
The AI detector, with a threshold of 0.85 confidence (on a 0‑to‑1 scale), flagged 34 submissions that MOSS had not associated with any flagged pair. Those 34 came from 28 distinct students. That is, 12% of the class had at least one assignment whose code looked AI‑generated but was not a near‑duplicate of any peer. On the surface, they were all unique, all passed our automated test suites, and all followed the assignment specification faithfully.
"I saw a Longest Common Subsequence solution that used a beautifully clean bottom‑up DP, with inline comments like 'We allocate an (m+1)×(n+1) table because …' — the kind of exposition a struggling sophomore wouldn't spontaneously produce. MOSS gave it a 0% peer similarity. The AI detector tagged it 98%." — Dr. Sarah Chen, CS2 lead instructor
We manually inspected each of the 34 high‑confidence flags. We looked for telltale signs: improbable variable‑name consistency (e.g., dpTable followed by perfectly idiomatic camelCase in a student who previously used table and my_array), comments that read like textbook paragraphs, and structural choices that matched known LLM output patterns — such as always extracting a helper method for the recursive backtracking step even when a single‑method solution would have been more natural. We ended up classifying 26 of the 34 as very likely AI‑generated by a tool like ChatGPT, with another 4 ambiguous and only 4 probably false positives. That brought our detected cheating rate from roughly 18% (MOSS alone) to just over 29% — a substantial uplift.
The false positive problem with AI detection
We were most worried about falsely accusing a student who happened to write unusually polished code. To test this, we fed the detector a random sample of 50 submissions from earlier semesters (all pre‑ChatGPT, written when GPT‑3 was the state of the art and rarely used by undergrads). The detector's 0.85‑confidence threshold produced a false positive rate of 4% in that historical sample. That's not zero, and it means that at scale, a purely automated pipeline would incorrectly flag a handful of honest students. Our process of layering manual review on top before taking any academic‑integrity action kept us safe. We never inform a student solely based on an AI confidence score; it's always one piece of a larger picture that includes in‑office code‑walkthroughs and discussion.
The detector gave noticeably higher confidence on problems that had clear, well‑known algorithmic solutions — dynamic programming, tree traversals, Dijkstra's — because the LLM tends to generate near‑boilerplate implementations of those. For open‑ended assignments (e.g., "build a simple text adventure game"), the scores were far more scattered and less reliable. We suspect this reflects the training distribution: the LLM has seen thousands of Dijkstra implementations, but text adventure logic is all over the place.
Where MOSS still shines
Despite all the excitement around AI detection, MOSS remains indispensable for a reason many instructors overlook: it catches collaborative copying that no AI‑authorship model can see. Two students who work together to flesh out a solution, debug side‑by‑side, and then each submit a slightly refactored version will produce code that carries the same fingerprint of shared intellectual history. MOSS's token‑based n‑gram fingerprinting is exquisitely tuned to that scenario.
In our experiment, seven of the MOSS‑flagged pairs contained zero AI detection hits. Those were clear cases of peer‑to‑peer copying — sometimes with variable renames, sometimes with method extraction — that would have sailed past a pure AI detector. This isn't a weakness; it's a reminder that code similarity and AI authorship are orthogonal signals. The strongest detection posture stacks both.
Moreover, MOSS takes zero configuration. We drop files in, get results back. The AI detector required us to handle API keys, think about thresholds, and build that small Python wrapper. For a time‑strapped adjunct teaching two courses and serving on a curriculum committee, the friction matters. Tools like the Codequiry web dashboard (which we later explored) help close that gap by offering a single interface that runs both checks at once.
Why a unified tool could be better
Running two separate pipelines — MOSS for similarity, Codequiry for AI detection — gave us excellent coverage but introduced operational toil. We had to maintain two sets of credentials, two result formats, and two review workflows. In discussions after the semester, the TAs unanimously preferred the idea of a single platform that accepted a batch of submissions, ran both checks, and presented a combined report. That's precisely what the code plagiarism checker inside Codequiry does: it uses the same token‑plus‑AST‑fingerprinting approach as MOSS (augmented with web‑source and GitHub checks) and layers the AI detection on top. For our department's upcoming spring offering, we plan to pilot that unified flow and collapse our toolchain.
A combined tool also solves the provenance problem we kept stumbling into: when a submission was flagged by both MOSS and the AI detector, we could quickly rule out the "two students independently asked ChatGPT the same question" hypothesis because the similarity scores were too high. Conversely, an AI‑only flag with zero similarity strongly indicated an LLM‑assisted but otherwise isolated submission. Having both signals in the same UI made these differential diagnoses far faster.
Lessons for CS instructors
If you are thinking about adding AI detection to your existing code‑integrity workflow, a few findings from our semester may help:
- Threshold tuning matters. The 0.85 confidence threshold we used was a deliberate trade‑off: we wanted to catch the obvious cases and accept a modest false positive rate, knowing we would manually review. If you plan to auto‑warn without human review, you would want a much higher threshold (and even then, tread carefully).
- Don't abandon similarity checks. The AI detector does not replace MOSS, JPlag, or Dolos. It adds a new dimension. The worst‑case cheater in a post‑LLM world uses ChatGPT to generate a base, then edits it with a friend to evade both detectors. You need both lenses.
- Historical baselines help. Running the detector on a batch of known‑honest, pre‑LLM code gives your team a feel for the natural level of false positives and prevents over‑reliance on a single number.
- Code walkthroughs remain the gold standard. When a student can explain every line and why they chose that approach, even a 99%‑confidence AI flag becomes much less alarming. The detector is a triage tool, not a judge.
Web‑source checks added yet another layer. A small handful of submissions that MOSS and AI both missed turned up when we ran them through the web‑matching component of the plagiarism checker for code. They contained verbatim snippets from popular GeeksforGeeks articles — something MOSS won't flag unless a peer also copied the same snippet. Including the web in the detection net closed the loop.
Frequently Asked Questions
Can MOSS detect AI-generated code?
No, MOSS is designed to find code that shares ancestry — either two students copied from each other or one copied from a common source. AI-generated code written independently by different students can look structurally different even when prompted identically, so MOSS often fails to flag it.
How accurate is an AI code detector on student Java assignments?
In our controlled test on pre‑LLM Java submissions at the 0.85 confidence threshold, the detector produced about 4% false positives. Accuracy varies by assignment type: highly algorithmic problems (sorting, DP) yield stronger signals than open‑ended projects.
Should I replace MOSS with an AI-focused code checker?
Not yet. MOSS remains the best tool for finding peer‑to‑peer copying. The strongest approach we found was using both side by side, or switching to a platform like Codequiry that combines similarity, web‑source, and AI detection in a single report.
What's the difference between code similarity and AI authorship detection?
Code similarity detection compares submissions to each other and to known sources to find shared structure. AI authorship detection analyzes the probability distribution of tokens within a single file to determine if it was written by a human or an LLM. They measure completely different things, and neither alone catches everything.
If your department is curious about piloting a combined workflow, the Codequiry AI code detector with its batch‑check API and built‑in similarity engine removes much of the integration toil we experienced — and you can try it on a sample of this semester's submissions without disturbing your current process.