Harlow State University's computer science department logged 4,120 submissions in the 2024-25 academic year across 14 courses and nine instructors. The first full-cohort AI generation scan, run in October 2024, put 214 submissions above 0.50 on two independent detectors and another 174 above 0.50 on only one. The integrity committee opened 41 cases. Nineteen ended in a finding.
Those four numbers are the short version. What follows is how the pipeline that produced them was assembled, what the distributions looked like, and where the process turned out to be noticeably weaker than the tooling implied.
What Harlow State Was Trying to Solve
Harlow State is a mid-sized public institution: 19,400 undergraduates, a CS1 sequence that grew from 640 enrolled students in 2019 to 1,180 in fall 2024, and a teaching staff that is roughly 60% adjunct and graduate teaching assistant. The department had used MOSS for pairwise peer similarity since 2016. That tool answered one question well, which was whether two students in the same section had turned in overlapping code.
It did not answer two other questions the committee kept running into. The first was source provenance: code lifted from a GitHub repository or a six-year-old Stack Overflow answer passes a peer check cleanly because no classmate wrote it. The second was authorship. Submissions started arriving that were internally consistent, matched nobody in the cohort, traced to no web source, and still read nothing like the student's own work from three weeks earlier.
The associate chair summarized the gap in the committee's March 2025 minutes, and the line has stuck with me:
"We were not short of suspicion. We were short of a way to make suspicion comparable across sections."
That is the actual problem in most departments. Not detection. Comparability.
The Detection Stack They Assembled
By spring 2025 the department was running three layers on every submission, plus a fourth layer that was human. The table below is close to the configuration sheet they gave me, with contract details trimmed.
| Layer | Tool | Version / contract | What it returns |
|---|---|---|---|
| Peer similarity | MOSS (fall 2024 baseline) | Stanford MOSS, 2024 build | Pairwise percentage, HTML results page, no API |
| Peer, web, and AI | Codequiry | Enterprise, REST API plus dashboard | Per-submission peer score, web score, AI score |
| AI cross-check | General text detector on comment-stripped source | 2025 commercial model | Per-file probability |
| Adjudication | Department rubric | Revised July 2025 | Committee notes, outcome, sanction |
The third layer is the one that generated the most argument internally. General-purpose text detectors were trained on prose, not code, and comments are the only prose in a source file. Stripping them first was a compromise: it removed the detector's easiest signal and left it reasoning about a token stream it was never really built for. I would treat that layer as a tiebreaker and nothing more.
The Codequiry layer handled the code-native side. Its AI code detector scores each file on a continuous scale rather than emitting a binary verdict, which matters for the same reason a p-value matters: you need to see where a score sits relative to a distribution before you act on it.
Calibrating Against a Known-Human Control Set
Before the department used a single AI score in a conversation with a student, they built a control set. The archivist pulled 312 submissions from 2017 through 2021, all of them predating public access to ChatGPT and GitHub Copilot's general availability, all from courses still being taught. These files are human-written by construction. Running the detector across them gives you the false positive floor for coursework of that type.
| Course level | n | Median AI score | 90th percentile | Share at or above 0.50 |
|---|---|---|---|---|
| CS1 (intro) | 104 | 0.11 | 0.34 | 4.8% |
| CS2 | 98 | 0.14 | 0.41 | 7.1% |
| Upper division | 72 | 0.19 | 0.52 | 12.5% |
| Graduate | 38 | 0.24 | 0.58 | 18.4% |
The gradient is the finding. Median score climbs with course level, and so does the right tail. Graduate-level code is more idiomatic, more uniformly structured, more likely to use the same handful of design patterns everybody uses, and therefore more statistically indistinguishable from generated code. A flat 0.50 threshold applied across all four levels produces a false positive rate that ranges from roughly one in twenty to roughly one in five.
The control set is 312 files from one institution, and I would not carry the exact percentages anywhere else. But the shape, a monotonic increase in score with code sophistication, has reproduced in every cohort the department has re-run since. That part is worth planning around.
Where the Two Detectors Disagreed
Across the full 4,120 submissions, the code-native detector and the comment-stripped text detector agreed on the broad strokes and diverged at the edges. Cohen's kappa on the 0.50 cutoff was 0.58, which is moderate agreement and nowhere near enough to treat either score as ground truth.
| Category | n | Share of corpus |
|---|---|---|
| Both at or above 0.50 | 214 | 5.2% |
| Code-native detector only | 174 | 4.2% |
| Text detector only | 96 | 2.3% |
| Neither | 3,636 | 88.2% |
The 214 in the top row were where the committee spent its time. Of the 41 cases opened, 34 came from that bucket. The 174 code-native-only cases tended to be short procedural assignments with almost no comments, which gives the text detector nothing to chew on. The 96 text-detector-only cases were mostly heavy-comment files where the student's own writing style happened to read formulaic.
Harlow State encoded the triage in a small function that the LMS gradebook import calls. It is not sophisticated, but writing the policy as code forced the committee to argue about specific numbers instead of adjectives:
def review_tier(ai_score, peer_max, web_max):
"""Returns the escalation tier for one submission.
Thresholds are per course level; these are the CS1 values.
"""
if ai_score >= 0.85 or peer_max >= 0.90 or web_max >= 0.80:
return "committee_review"
if ai_score >= 0.55 or peer_max >= 0.65:
return "instructor_notice"
return "no_action"
The gap between the two thresholds, 0.55 for a notice and 0.85 for committee review, is deliberate. It creates a band where the instructor is told and nothing else happens. Roughly 340 submissions a year landed in that band, and the overwhelming majority of them resolved as a conversation about citation practice.
Peer Similarity Did Not Go Away
The department ran 137 submitted pairs above 0.90 peer similarity across the year. Sixty-one of those came from just two courses, both of which use the same three textbook exercises every term. That is a curriculum problem wearing a detection costume.
What surprised the committee was not the volume but the triage cost. MOSS has matched against structure for years, and JPlag 4.x does the same thing with a cleaner interface; neither is fooled by variable renaming or reformatting. Token normalization and AST comparison have been table stakes since the early 2000s. The bottleneck was never the matching. It was that 400 submissions produce 79,800 pairs, and an HTML page sorted by percentage does not tell a TA which forty of those actually matter.
Sorting by cohort outlier score instead of raw pairwise percentage compressed the review list to 34 items for the largest CS1 section. If you are still triaging by raw percentage, a code plagiarism checker that ranks by how unusual a submission is within its cohort will save you more hours than any improvement in matching accuracy.
The Escalation Policy That Made It Work
Three tiers, each with a named owner and a clock. Instructor notice within five business days of the scan. Committee review within ten. Student contact only after a human has read the code, and the message never contains the score. Students are told their submission has been referred for a conversation, which is accurate and does not invite an argument about model calibration with someone who has not read the calibration study.
Scores also never appear in a grade. The committee was unanimous on this and I think they were right. A 0.62 from a detector is an invitation to look, not a fact about authorship, and the moment it enters a gradebook it becomes something the student has to disprove.
The other thing that worked was a rubric with worked examples rather than a policy paragraph. "AI-assisted" got defined as autocomplete on boilerplate, docstring generation, and syntax questions. "AI-generated" got defined as any submitted logic the student did not author, regardless of how much editing followed. Both categories shipped with two annotated code samples each. Case volume in the notice tier dropped by about a third the semester after the examples went out, which the committee attributes to clarity and I am inclined to agree with.
Instructor Variance Was Bigger Than the Tooling
Six instructors taught the same CS2 course in spring 2025, using identical assignments, identical thresholds, and identical rubrics. Here is what they did with the notices they received.
| Instructor | Sections | Submissions | Share escalated further |
|---|---|---|---|
| A | 2 | 118 | 21.2% |
| B | 2 | 124 | 18.5% |
| C | 1 | 61 | 9.8% |
| D | 2 | 119 | 6.7% |
| E | 1 | 58 | 3.4% |
| F | 2 | 122 | 2.5% |
An eight-fold spread in escalation rate, on the same scores, in the same course, in the same semester. The detector variance across the whole year does not come close to that. Anyone building an integrity program should expect human variance to dominate the system, and should budget effort accordingly: calibration sessions, shared adjudication notes, a second reader on any committee referral. The thresholds are the cheap part.
Two instructors in that table had discretionary grading authority and no formal integrity training. That is not a criticism of them. It is a description of the job.
What They Would Do Differently
The first scan in September 2024 included 41 resubmissions that had already been processed under an earlier check ID. The AI score column double-counted them, which inflated the CS1 section average by about three points before a TA noticed the duplication in the export. Nothing about the detection was wrong; the pipeline was. If you are building this, deduplicate on student ID plus assignment ID before you compute any cohort statistic.
The second change is more structural. Harlow State calibrated once, on a pooled control set, and applied the result to every course. Running a per-course control set of 60 to 80 archived files would have caught the CS1 versus graduate divergence eighteen months earlier and produced better thresholds for both ends of the curriculum.
The third is language. The syllabus was rewritten in July 2025 to separate AI-assisted from AI-generated with two concrete examples each, and to state plainly that detectors score files rather than people. That sentence did more for the department's relationship with its students than any report the committee produced.
The tooling piece is genuinely the simplest part to get right. A platform that returns peer, web, and AI signals from one submission event, with a review queue that ranks what to look at first, removes most of the operational friction. Harlow State's committee ran a Codequiry vs MOSS comparison as part of its spring 2025 brief, and the deciding factor was not detection quality. It was that one system returned all three scores through an API the registrar's office could actually call.
If your department is at an earlier stage, the sequence that worked here is worth copying: build a control set before you build a policy, pick thresholds per course level rather than globally, and treat every score as a logging decision until a person has read the file. You can detect code plagiarism and AI generation in the same scan without turning the scan into the verdict.
Start with a small pilot cohort and a control set you trust, then run a similarity check on submissions you already know the answer for. The false positive data will teach you more in an afternoon than a semester of thresholds guessed from the vendor's documentation.