Stacking AI Detection and Plagiarism Checks in One Grading Pass

Two detectors answer two different questions. A similarity engine asks whether a submission resembles another submission, a public repository, or a tutorial. An AI code detector asks whether the file reads as machine-written. Running both over the same corpus, in the same pass, and putting the outputs on one page is the practice I want to describe here. The department that let me watch it work cut its integrity review meetings roughly in half, and, more interesting to me, changed what those meetings were about.

The setting was a mid-sized public university I'll call Ridgeline State, about 18,400 undergraduates, and a CS2 course, CS 210 Data Structures, taught in Java to 412 students in the fall of 2024. Eleven teaching assistants handled the grading load. I was there as an outside reviewer because the department chair had asked me to sit on two of its academic-integrity hearings the previous spring. At the second hearing I said something unhelpful: that the department was asking a text-similarity tool to answer a question about authorship. The chair remembered the comment and invited me back to watch them try to fix it.

The two questions the old workflow kept conflating

For years Ridgeline ran what most of us run. MOSS, Alex Aiken's pairwise matcher, first released in 1994 and still maintained, for submission-to-submission similarity. Turnitin on the written design documents that accompanied two of the seven assignments. Neither tool was the problem. The problem was the gap between them.

MOSS answers "does this resemble that?" with a scaled score on a 1-to-1000 range, and it answers it well. But a file can be 100 percent original and still not be the student's work. Turnitin reads prose, not Java. Between the two of them sat the question the honor board actually cared about in the spring of 2024: nine referrals, four of which turned out to be students who had asked a model to explain the assignment, understood the explanation, and then discovered they could not reproduce the code from memory. That is a policy failure before it is a detection failure, and no scanner fixes it.

The chair's summary in May was blunt: "We were catching the students who copied from each other and missing the ones who copied from something that doesn't have a name."

Writing the disclosure policy before touching the pipeline

The department spent most of June on the policy and about a week on the tooling. That ratio was deliberate, and I think it is the reason the fall semester went as smoothly as it did. GitHub Copilot had been free for verified students since 2022. ChatGPT was on every laptop by the fall of 2023. What the department lacked was not awareness but a boundary.

They settled on three disclosure bands, printed in the syllabus and repeated in the header comment of every assignment file.

Band 1 prohibits generative tools entirely, for assignments 1, 3, 5, and 7. Band 2 permits a model to explain an error, a concept, or a compiler message, provided the student writes every line of submitted code. Band 3 permits generated code, on assignment 9 only, with the prompt and the tool disclosed alongside the submission.

The disclosure header is worth showing because it is the artifact that made the whole arrangement work in practice:

// AI USE: band 2
// Tool: ChatGPT (GPT-4o), Sept 18
// Prompt: "why does my iterator throw ConcurrentModificationException"
// What I wrote: the remove() method and the whole test class

A student who fills that in honestly has done nothing wrong under band 2. A student who leaves it out on a band 1 assignment has now created a policy question that is separable from a technical one, which is exactly where a review meeting wants to be. The header also gave the TAs something to check without having to make a judgment call about style.

What one check actually produces

Ridgeline's replacement for the three-tool dance was a single Codequiry check per assignment, configured once and reused. Each submission came back with three numbers rather than one: peer similarity against the 412-student cohort, web similarity against GitHub and indexed sources, and a per-file AI-generation score.

What made the peer and web side usable at this scale is that the engine is not doing text comparison. It tokenizes, builds an abstract syntax tree, and fingerprints structure, so a student who renames every variable, converts a for loop to a while loop, and reindents the file still gets caught. That property matters more in CS2 than in almost any other course, because CS2 is where students learn to refactor, and where a plagiarism tool that only reads surface text quietly stops working. The same structural approach carries over to how the tool traces a match to its source, whether that source is a classmate or a public repository.

I want to be fair to the alternatives here. JPlag, now on version 5.x, is excellent and free, and Dolos out of TU Braunschweig has a clean visual interface and good language coverage. Both are pairwise or multi-file similarity tools. Neither ships an AI-generation score, and both require you to build the cohort comparison workflow yourself. The department had been running a MOSS pipeline through a homegrown script since 2019, and the TA time spent on that script was the hidden cost nobody had itemized.

The first assignment that made the case for stacking

Assignment 6 is a graph traversal exercise: build an adjacency list, run BFS and DFS, and answer four reachability questions. It is the assignment where students historically struggle, and the one where the previous year's MOSS run had produced a 62-entry match list that the TAs spent nine hours triaging.

Here is what the combined pass produced across the term, with submission counts rounded to the nearest unit.

Assignment	Submissions	Peer match above 55%	Web match, 40+ tokens	Any file at AI score 80+
1, warm-up	408	31	12	58
3, linked list	401	26	9	34
5, hash table	396	44	17	39
6, graph traversal	394	52	23	61
9, extension	287	18	6	102

Assignment 1 is the one that surprises people. The AI column is high because the assignment ships with a provided starter file, and a starter file is boilerplate: imports, a class skeleton, a handful of accessors. Models are extremely good at producing boilerplate, so boilerplate scores like a model. The TAs learned within a week to exclude provided files from the AI score and to look only at files the student authored.

Assignment 9 is high for the opposite and more honest reason. It is the band 3 assignment. Students were permitted to generate code there, and many did, and the score is doing its job rather than finding misconduct.

Reading an AI score on a file a student wrote by hand

The most useful hour of the whole semester was the one where a TA named Priya walked me through the file that had scored 94. It was a GradeBook class, roughly 70 lines, and the student was a second-degree student in her thirties who had spent nine years as a business analyst before switching to CS. She wrote like this:

public final class GradeBook {

    private final List<Student> students;

    public GradeBook(List<Student> students) {
        this.students = Objects.requireNonNull(students, "students must not be null");
    }

    public OptionalDouble average() {
        return students.stream()
                .mapToDouble(Student::finalGrade)
                .average();
    }
}

And here is a typical CS2 submission that scored 22:

public class GradeBook {
    ArrayList<Student> list = new ArrayList<Student>();
    public double average() {
        double sum = 0;
        for (int i = 0; i < list.size(); i++) {
            sum = sum + list.get(i).finalGrade();
        }
        return sum / list.size();
    }
}

The second version throws an exception on an empty roster and leaks its internal list, and it is unmistakably human. The first version is defensive, uses a stream, names its exception message, and marks the class final. It is also exactly what a careful adult with professional habits writes after reading the Java documentation, which is what she had done, and which she demonstrated in office hours by rewriting the average() method from scratch while Priya watched.

This is the part of AI detection that gets discussed least and matters most. The signals detectors weigh, things like comment density, identifier uniformity, idiomatic use of standard library patterns, and complete error handling, are correlated with experience as well as with generation. A strong student can look like a model. So can a file of pure accessors. So can a provided interface stub.

The TAs adopted three rules. Files under about 40 lines were treated as inconclusive and not surfaced to students at all. Scores were read as a distribution over a file, never as a verdict on a person. And any file above 80 went to a human before a name went into a report. I have watched enough of these to say plainly that a per-file probability is a triage instrument. It is not evidence, and a department that treats it as evidence will eventually accuse the wrong student, probably one with an unusual educational background and good habits.

What the review meeting looked like after the change

The fall review queue for assignment 6 held 34 entries, down from 62, and it was ordered by a combination of signals rather than alphabetically by student surname. The top of the list was a submission with 97 percent peer similarity, a 61 percent web match to a GitHub repository last updated in 2021, and an AI score of 12. That is a student who found a repository and adapted it. The flags agreed with each other, which is the easiest kind of case.

The middle of the list was harder and more interesting. Fourteen submissions had a high AI score and no peer match at all. Those fourteen became a single policy conversation rather than fourteen individual ones, because the pattern suggested a shared cause rather than fourteen independent decisions. It turned out that a review session run by the campus tutoring center had been demonstrating the assignment with a model, and several students had reproduced the demonstration closely. Nobody had plagiarized anybody. The tutoring center changed its format in week 8.

Across the term the numbers came out modest, which I consider a sign the process was working rather than a sign it was weak. Forty-six flags, nineteen student conversations, six referrals to the honor board, four findings of a policy violation, all four involving an undisclosed band 1 violation rather than copied code. The two hearings I sat in on took thirty-five minutes each. The previous spring's took ninety.

Where the workflow still breaks

Inline completions. This is the hole nobody has closed. A student typing in VS Code with Copilot enabled accepts three-line suggestions all afternoon, and a per-file AI score may never move, because three lines here and four lines there is not what a file-level model is measuring. The disclosure header is the only control the department has for that, and disclosure depends on the student believing it matters. I don't have a good answer for this and neither does anyone else I've asked.

Group work is the second gap. Peer similarity within an assigned team is expected and meaningless, and the TAs had to remember to exclude teammates from the cohort comparison for the two partnered assignments. That is a configuration step, and configuration steps get forgotten in week 11.

The tooling itself had a bad fortnight. The command-line client at version 2.3.1 was reporting Java package declarations as common three-line matches, which inflated the web score on 22 submissions during assignment 3 until a patch landed. The TAs caught it because a student who had clearly written her own code appeared with a 44 percent web match to a repository she had never seen. A tool that never produces a strange result is a tool nobody is checking.

What another department should copy, and what to skip

Copy the policy work first. The three bands took four faculty meetings and a syllabus rewrite, and every downstream decision got easier because of them. Copy the disclosure header. It costs one line of student effort and it converts a technical suspicion into a policy question, which is where a faculty committee actually has authority.

Copy the single pass. Running similarity, web, and AI checks in one batch per assignment, rather than a week apart in three different tools, is what let the TAs read the three numbers together. The code plagiarism checker side of the house and the AI side are the same dashboard, and the value is in the proximity, not in either score alone.

Skip the temptation to publish a threshold. Ridgeline never told students that 80 means anything, because it doesn't, and a published cutoff is an invitation to write to 79. Skip, also, the idea that a detector can carry an integrity policy on its own. We haven't tested this workflow past a few hundred submissions per run, and I'd want three more semesters before claiming the referral rate holds at a larger program.

The thing I keep coming back to is the meeting in May, when the chair said the department had been catching the wrong students. What changed by December was not the accuracy of any single number. It was that the peer score, the web score, and the AI score arrived together, and the faculty could finally see which of the three questions each case was actually asking. If you want to see what that combined view looks like on your own cohort, the team runs sample checks through the Codequiry platform with a five-submission guided demo.