What 4,200 Submissions Revealed About AI-Written Code

In the spring 2024 term, a mid-sized public university ran 4,213 submissions from its CS 2 course through three integrity engines at once: peer-to-peer similarity, web source matching, and AI-generation scoring. Just under 26 percent of submissions crossed the department's 65 percent peer threshold. After manual review, 7.4 percent held up as unauthorized collaboration. The distance between those two numbers is the story of the past three semesters here.

The department asked that the university not be named while a revised academic integrity policy works its way through the faculty senate. The course, the counts, and the people are real. The letterhead is not.

Three Detectors, Three Kinds of Evidence

CS 2 is Data Structures and Algorithms, 527 students across 11 sections, eight programming assignments, roughly 78 percent Java and 22 percent Python depending on section. Until 2022 the course used exactly one tool: MOSS, the Measure Of Software Similarity system Alex Aiken released in 1994 and which still runs as a Perl script posting to a Stanford server. One faculty member knew how to invoke it. Results arrived as a static HTML page of match percentages, got skimmed, and were filed.

"We were making policy decisions off a number nobody had calibrated," said Priya Raghavan, an associate professor who took over as undergraduate program director in 2022. That year the course generated 14 honor code referrals. Two students appealed and won, on the grounds that the similarity evidence had never been explained to them and that at least part of the match was starter code the instructor had distributed.

The committee that formed afterward made a decision that shaped everything since: they would treat similarity, web provenance, and AI generation as three distinct forms of evidence with three distinct thresholds, rather than collapsing them into a single misconduct score.

Why the First Pilot Flagged Every Submission in One Lab

The Fall 2023 pilot did not go smoothly. The binary search tree lab ships with a 140-line skeleton file that students extend. Nobody marked that file as shared material before the first comparison run, so the peer engine returned a 100 percent match for all 611 submissions in the lab. The TA who ran it, Marcus Oyelaran, spent a weekend confirming what he already suspected.

"I opened the first twelve reports and every single one was the same file at 100 percent. That's the moment you learn that a similarity tool tells you two things are similar, not that a person did something wrong."

The fix was unglamorous. The department now maintains a per-assignment baseline set: starter code, provided test harnesses, the `import` block, and the standard `public static void main(String[] args)` scaffolding that appears in every Java submission regardless of who typed it. Anything in the baseline is excluded before scoring. Raghavan estimates that baseline exclusion removed roughly 60 percent of the raw match volume in introductory assignments.

The piloted workflow, with three engines reporting into one view, took a semester longer to reach production than planned because the university's registrar wanted a data retention review before any vendor touched student names. The department ran the 2023 pilot on de-identified submissions and went live in January 2024.

Codequiry AI code detection report with average AI score, highest file score and a risk distribution
AI code detection: probability scores per file, flagging submissions likely written by ChatGPT, Copilot, Claude or Gemini.

What Token and AST Matching Caught That Text Comparison Missed

The clearest example from the term involves two submissions in the same section that a plain text diff would have scored at 61 percent. Rename the identifiers, collapse the braces, and the two functions look like different code. Here is submission 4471:

public static int depth(Node root) {
    if (root == null) return 0;
    int left = depth(root.left);
    int right = depth(root.right);
    return Math.max(left, right) + 1;
}

And here is submission 2338:

public static int treeHeight(Node n) {
    if (n == null) {
        return 0;
    }
    int lh = treeHeight(n.left);
    int rh = treeHeight(n.right);
    return (lh > rh ? lh : rh) + 1;
}

The token stream differs. The variable names differ. The control flow is identical: null guard, two recursive calls, ternary or Math.max selection, add one, return. Codequiry's token and AST comparison scored the pair at 94 percent. That number mattered less, in the end, than the fact that both students independently described the same office-hours explanation to Oyelaran. The case closed as shared study, not copying, and went into the file as a false positive with a documented reason.

"That's the outcome I want more of," Raghavan said. "A tool that forces a conversation and then lets the conversation settle it."

The department's threshold work reflects the same logic. JPlag 4.2, which they still run as a second opinion on Java-only assignments, uses a token-based comparison with good visualizations but no notion of what a starter file is, so its raw scores run higher on scaffolded labs. Dolos, the open-source tool that parses with tree-sitter and produces a browsable report, gave them a cleaner structural view than MOSS but still required the TA to read every flagged pair. None of these are wrong tools; they are instruments that answer a narrower question than the one an integrity committee asks.

Side-by-side code comparison in Codequiry showing a 91% match between two student submissions
Side-by-side comparison: Codequiry lines up matching code between two submissions, with confirmed and false-positive review labels.

AI-Written Code Rarely Looks Like Copied Code

The finding that surprised the committee most was the overlap analysis. Of the 506 submissions with at least one file scoring above 85 percent on AI-generation probability, only 111, about 22 percent, also crossed the 65 percent peer threshold. The other 78 percent had no twin in the cohort. They were not copies of anything the peer engine could see.

The 111 that did overlap told a different story. When Oyelaran traced those clusters, he found students who had shared prompts and outputs: one person's session with a model, pasted into a group chat, pasted into four submissions with minor edits. A peer engine sees that as collusion. It isn't quite. It's a distribution problem wearing collusion's clothes.

"Copying means one student took something. Prompt-sharing means the assignment itself is now trivially transferable. Those need different responses, and for a while we were giving both the same one."

That distinction pushed a course-level change rather than a discipline-level one. Four assignments in the sequence were rewritten in summer 2024 so that the specification depends on data generated by the student's own earlier submission, which makes a shared answer useless. In the first term after the rewrite, the AI-flagged-cluster rate on those four assignments dropped from 9.1 percent to 2.3 percent. The solo-flag rate did not move much.

The Threshold Argument the Department Could Not Avoid

Where to draw the line is the part of this work that no vendor solves. The department's approach was to set thresholds low enough to catch cases, accept that most flags will be false positives, and staff the review capacity to handle the volume. That is a staffing decision disguised as a technical one.

Signal Above threshold Share of 4,213 Confirmed after review Precision
Peer similarity ≥ 65% 1,094 26.0% 312 28.5%
Web match ≥ 30 consecutive tokens 379 9.0% 141 37.2%
AI score ≥ 85% on any file 506 12.0% 91 admitted 18.0%

The AI column needs a footnote and Raghavan is careful about it. The 18 percent precision figure comes from 60 interviews selected from the 506 flagged submissions, plus written follow-up with the remaining students. It counts only students who admitted that a model wrote substantial portions of a submitted file. Students who said they used a model to understand a compilation error, or to generate test cases they then discarded, were logged as assisted and not sanctioned. That line is the department's, not the tool's.

"Every number in that AI column is a human judgment stacked on top of a probability," she said. "Anyone quoting a raw AI score in a disciplinary letter is going to lose the appeal."

The department's own guidance, circulated to TAs in February 2024, is blunt about this: a high AI score is a reason to open a conversation, never a finding. In practice, TAs prepare for those conversations by reading the submission's comments and commit history alongside the score. Genuine student code in this course tends to carry debugging scars, commented-out attempts, and inconsistent indentation between sessions. Files that arrive whole, with tidy docstrings citing time complexity in a format the course never taught, read differently to a human being than to a classifier.

What Three Semesters of AI Code Detection Changed

Fall 2024 brought the policy rewrite. The old zero-tolerance framing, which produced 14 referrals in a 527-student course, was replaced with a three-tier disclosure model: no AI use, AI for explanation and debugging, AI for generated content that must be cited. Students declare a tier per assignment in the submission portal. A declaration that contradicts the detection signal triggers a meeting, not a hearing.

Referrals fell to six. Confirmed misconduct fell by less than the referral count suggests, because a chunk of what used to be referrals became meetings. Raghavan does not claim the new policy reduced cheating. She claims it produced a more accurate record of what students actually did, which is a different and more defensible goal.

The technical side now runs on a single check per assignment rather than three separate passes. The department uses Codequiry's code plagiarism checker for the peer and web passes together, since it compares against the cohort, the open web, and GitHub in one run, and stacks the AI code detector output alongside. The instructor-facing reports matter here for a boring reason: a teaching assistant who has to reconcile three HTML dumps from three tools will stop reconciling them by week six. Oyelaran's rule is that any workflow requiring more than one browser tab per case does not survive midterms.

Codequiry web results tracing copied code to GitHub repositories and other web sources with per-domain scores
Web results: every domain a submission matched, scored per source, from GitHub repos to tutorial sites.

Web matching turned out to be the least controversial signal and the most straightforwardly actionable. The single most-copied source in the term was a public GitHub repository from a different university's 2019 offering of the same course, complete with the same lab names. Thirty-one submissions matched it above the 30-token threshold. That is not an AI problem and never was, and it is the kind of thing a cohort-only comparison will never see.

Where the Detection Still Breaks Down

Three failure modes showed up repeatedly and none of them have clean fixes.

The first is small cohorts. Sections of 18 students in the summer term generate too few pairwise comparisons for a threshold to mean anything, and the baseline is nearly a third of the code in each submission. The department now pools summer sections into a single comparison set and accepts that the resulting numbers are weaker.

The second is legitimate collaboration the tool cannot distinguish from copying. Study groups, tutoring center sessions, and the department's own peer-led instruction sessions produce structurally similar solutions to a well-specified problem. Raghavan's guidance to TAs is to treat a match as a prompt for a question about process, not an accusation. Half the time the answer is "we worked on it together in the tutoring center," which is permitted and unverifiable.

The third is language coverage. The course added a Rust-based systems module in spring 2025, and the department had to submit a custom parser definition before the tools could read it at all. That took two weeks and a conversation with support. Raghavan notes, dryly, that the assignment most likely to be AI-generated in a new language is the one nobody can scan yet.

What the department has now that it did not have in 2022 is a defensible account of what its numbers mean. Every flag in the system carries a reviewer, a reason, and an outcome, including the roughly 70 percent of peer flags that resolve as false positives. That archive is what lets Raghavan answer a faculty senate question with evidence instead of an anecdote.

For programs starting this work, the sequence that worked here is unremarkable and worth copying: pick one course, exclude the baseline first, run all three signal types on the same corpus so the overlap is visible, and budget TA hours for review before you buy anything. The detection is the easy part. The code plagiarism checker will tell you which two files resemble each other; someone still has to read them both and decide what that means.