How a Course Coordinator Automates Weekly Code Integrity Checks

A weekly integrity pass over a 600-student course takes roughly 90 minutes once it is scripted: one run for peer similarity, one for web sources, one for AI generation, then a single triage table sorted by combined evidence. The automation is the easy half. The hard half is deciding, before you look at any names, what each score is allowed to mean.

What follows is the workflow I built and ran across nine sections of an intro CS course in spring 2024 (n = 1,140 submissions over 11 assignments), with the numbers that came out of it and the parts that did not work the first time.

What the three signals actually measure

Most integrity tooling returns a number and stops. That number means something different depending on which comparison produced it, and conflating them is where most department disputes start. Write the mapping down before you automate anything.

Signal Compared against Typical false positive What the score supports
Peer similarity Every other submission in the same cohort, token and AST level Shared skeleton code, common idioms, a single distinctive but obvious approach taught in lecture Two students' files derive from a common ancestor
Web similarity GitHub repositories, Stack Overflow, tutorial sites, documentation Copying the recommended idiom from the course's own posted examples The submission contains text that exists elsewhere on the public web
AI generation A statistical model of human-written versus machine-written code Non-native English comments, heavily templated code, small files with too little signal The file's surface statistics resemble LLM output

Only the first of those three produces evidence you can put in front of a student without an argument. The other two produce leads.

Step 1. Freeze the assignment spec, then strip it

Similarity scoring is only fair if every submission is compared on the part the student actually wrote. That means your preprocessing script has to know three things: which files were handed out, which files the build system generates, and which lines are boilerplate that every IDE inserts.

We learned this the expensive way. The 2023 version of our prep script did not strip the default IDE file banner, and for one assignment every submission matched every other submission on the same eleven-line header. The similarity report showed 100% pairs between students who had never met. It took a TA four hours to figure out why.

import re, pathlib

PROVIDED  = {"Main.java", "ConsoleHelper.java", "StudentTests.java"}
GENERATED = re.compile(r"(Test|_test)$")
BANNER    = re.compile(r"/\*\*.*?\*/", re.S)

def normalize(path: pathlib.Path) -> str | None:
    if path.name in PROVIDED or GENERATED.search(path.stem):
        return None                      # not student work, exclude entirely
    text = path.read_text(encoding="utf-8", errors="replace")
    text = BANNER.sub("", text, count=1) # strip the IDE header, once
    lines = [ln.rstrip() for ln in text.splitlines()]
    lines = [ln for ln in lines if ln.strip()]
    return "\n".join(lines)

Run this over a known-clean pair of submissions and eyeball the output before you run it over a cohort. If the normalized files still contain a comment block that five students happened to copy from the lecture slides, exclude it by line range rather than by regex.

Step 2. Build the corpus before the semester starts

The peer corpus is not "all student code in the world." It is a specific decision you make in week one, and it determines what your reports can and cannot show.

Three corpora matter. The current cohort (this semester's submissions for this assignment) is the one you must have, always. Prior semesters of the same assignment are valuable but only if the assignment is stable; if you changed the spec, last year's solutions are a weak match source and will generate noise. The public web corpus is handled separately in step 4 because its matches need different interpretation.

If you are running an open-source tool, this is where the limits show up. MOSS has been the default for two decades, and its -m flag (maximum matches per file) will quietly truncate results on large cohorts if you leave it at the default: a 600-submission run can bury a real pair under hundreds of common-idiom hits. JPlag handles this better with its minimum token match setting, but you still own the corpus decision. Dolos gives you nicer visualizations and the same corpus problem. The tradeoff between these tools is covered in more depth in Codequiry vs MOSS if you are choosing between them.

Step 3. Run peer similarity against the cohort, not the internet

Once the corpus and preprocessing are fixed, the run itself should be a single call. We drive everything from a script rather than the dashboard because the same script has to run on a schedule during grading week and again when a student appeals in November.

import requests, pathlib

BASE = "https://api.codequiry.com/v1"
HEAD = {"Authorization": f"Bearer {API_KEY}"}

def submit_check(course_id: str, name: str, files: list[pathlib.Path]) -> str:
    payload = [("files", (p.name, p.read_bytes(), "text/plain")) for p in files]
    r = requests.post(
        f"{BASE}/courses/{course_id}/checks",
        headers=HEAD,
        data={"name": name, "language": "java", "engines": "peer,web,ai"},
        files=payload,
        timeout=180,
    )
    r.raise_for_status()
    return r.json()["check_id"]

Keep the key on a machine that is not a shared TA laptop, and check the endpoint path against your own account page before the first run; the API versioning changed once between the 2023 and 2024 academic years and a stale path fails silently in some HTTP clients.

Codequiry API keys page with a masked key, signed webhook configuration and API resources
The API surface: an account key, signed webhooks for finished checks, and docs for wiring scans into CI.

The output that matters is not the highest percentage. It is the distribution of pairwise scores across the cohort. In our CS1 cohort, unrelated submissions peaked between 28% and 35% on most assignments, because students all receive the same skeleton class and everyone writes the same three or four loop idioms. That is the floor. Anything below it is noise.

Set your review threshold from the cohort's own distribution, not from a number someone posted on a mailing list. A 40% threshold on an assignment with a shared skeleton can produce two hundred pairs that mean nothing.

Step 4. Read web matches by domain, not by score

A 90% match against a GitHub repository and a 90% match against the course's own posted solution are the same number and completely different events. The score tells you the text is identical. The domain tells you whether that is a problem.

Match source Default action
Course materials, lecture repo, provided examples Suppress. Add to the exclusion list permanently.
Stack Overflow or official documentation Check for attribution and whether the copied segment is the core of the solution. A cited idiom is usually fine; an entire method pasted from an answer is not, depending on your syllabus.
Tutorial sites and homework-help aggregators Review. These are usually the exact assignment, reposted.
Personal or public GitHub repos with the same assignment name Review first. This is the highest-yield category in most cohorts.
Codequiry web results tracing a submission to a Stack Overflow question with line and token counts
Tracing code to its source: a submission matched to a Stack Overflow answer, down to lines and tokens.

We found that tutorial-site matches clustered hard by assignment. Two of our eleven assignments accounted for 78% of all web hits in the semester, both because they had been indexed by homework-help sites years earlier under a generic name. Renaming the assignment file and changing the sample output did not help; changing the actual problem did.

Step 5. Treat AI scores as a distribution

This is the signal people most want to read as a verdict, and it is the one that punishes that reading hardest. A per-file AI score is a likelihood under a model, not a measurement of authorship. The useful question is not "is this file 73% AI?" but "how unusual is 73% in this cohort, on this assignment, given what we know about the false positive rate?"

To answer that you need a baseline. Ours came from the 2019 archive of the same course (n = 430 submissions, all written before any of these models existed). Mean file score was 12, 95th percentile was 31. That number is our floor. Anything below it is inside the range that human students produce.

Mean file score Share of spring 2024 cohort Default action
0 to 30 71% No action. This is the human baseline band.
31 to 55 18% Flag for review only if another signal is also elevated.
56 to 80 8% Review the file directly. Check comments, naming, and error handling.
81 to 100 3% Interview. Bring the file to the conversation.
Codequiry AI code detection report with average AI score, highest file score and a risk distribution
AI code detection: probability scores per file, flagging submissions likely written by ChatGPT, Copilot, Claude or Gemini.

The distribution itself is informative. Our 81-to-100 band grew from 0.4% of the cohort in spring 2023 to 3% in spring 2024, which is roughly what you would expect from a tool going from novelty to default for a subset of students. It is not a scandal number. It is a number that tells you the interview load you are signing up for.

Score the files individually, not the submission as a whole. A multi-file project with one 90% file and four 10% files is a different situation from five files at 60%, and a submission-level average flattens that distinction away. Codequiry reports the per-file breakdown alongside the submission average, which is why we run AI-written code detection through it rather than through a text-oriented checker; a general-purpose text detector has no idea what a Java method signature looks like and returns confident nonsense on short files.

Step 6. Collapse three reports into one triage queue

Three dashboards is two too many. The TA on duty needs one list, sorted by how much evidence the submission carries, with the underlying signals visible on the row.

import pandas as pd

df = pd.read_csv("signals.csv")     # one row per submission
df["evidence"] = (
    0.45 * df.peer_max                      # highest pair score in cohort
  + 0.25 * df.ai_mean                       # mean per-file AI score
  + 0.30 * (df.web_max.clip(upper=90) / 90 * 100)
)
queue = df[df.evidence > 55].sort_values("evidence", ascending=False)
print(queue.head(20)[["student_id", "peer_max", "ai_mean", "web_max", "evidence"]])

The weights encode what we learned about precision on our own data, and they are not universal. Peer similarity carries the most weight because it has the highest confirmed precision: in our manual review of 52 pairs scoring above 70% with at least eight identical non-trivial lines, 50 were confirmed as copied, 96%, though the sample is small enough that I would not quote that number outside this course. AI scores carry less weight because the false positive rate in the 31-to-55 band is still materially higher than we would like.

Codequiry smart review queue ranking submissions by cohort outlier score, topped by a 100% match
The smart review queue: cohort outliers ranked by priority, so a TA reviews the riskiest five, not all fifty.

What the queue changes is throughput. A TA working a 40-submission review list sorted by combined evidence resolves the real cases in the first twelve rows and spends the remaining time dismissing noise, instead of the reverse.

Step 7. Interview six questions, not one

"Explain this code to me" is a bad question. It is answerable from having read the file five minutes earlier, and it produces either a fluent recitation or a panicked silence, neither of which discriminates well. We use a fixed set, asked in the same order, so that the answers are comparable across students:

  1. Walk me through what happens when this method receives an empty list.
  2. Change one line so it handles a negative input. Talk while you do it.
  3. Why did you choose this data structure over the alternative?
  4. There is a bug on line 40. Find it without running the code.
  5. What would you have to change to make this work for a different file format?
  6. Which part of this submission took you the longest, and what did you try first?

Question six does most of the work. Students who wrote the code produce specific, slightly boring answers about a bug they hit on Tuesday. Students who did not produce generalities. This is not proof, and we do not treat it as proof; it is one input among several, and it is documented as such.

Bring the synced diff into the room. Seeing the two files side by side and being asked to point at the line they wrote is a different exercise from being asked to describe their process in the abstract.

Side-by-side code comparison in Codequiry showing a 91% match between two student submissions
Side-by-side comparison: Codequiry lines up matching code between two submissions, with confirmed and false-positive review labels.

Step 8. Recalibrate every semester

Every threshold above is a statement about a specific cohort, on a specific assignment, at a specific point in time. Change the assignment and the distribution moves. Change the model version behind the AI detector and the distribution moves again. We re-run the baseline against the archived pre-2022 corpus each August, before the fall semester starts, and we log the shift.

The cost of running the pipeline, once scripted, was about 40 minutes of human time per weekly pass and the rest unattended. The first semester cost considerably more, mostly in the preprocessing step and in arguing about thresholds after the first batch of reports went out with the IDE banner problem still in it.

If you take one thing from this: the scanner is not the integrity process. The scanner produces a ranked list of files to look at. Everything that matters happens after that, in a conversation with a student who either can explain their code or cannot.

For departments moving from ad-hoc MOSS runs to a repeatable weekly pass, the code plagiarism checker handles peer, web and AI signals in one run so that step 6 is a sort rather than a manual merge.

Frequently Asked Questions

What is a reasonable similarity threshold for flagging student code?

It depends on how much skeleton code you hand out. Measure the score distribution of unrelated submissions on one assignment first; in our courses the noise floor sat between 28% and 35%. Set the review threshold above the point where the distribution flattens, not at a fixed number.

Can students defeat peer similarity by renaming variables and reformatting?

Not against token and AST-based comparison, which is what most modern tools use. Renaming a variable changes the token, but the structural fingerprint survives. Surface text matching fails here; that is the main practical argument for structural tools.

How accurate is AI code detection on student assignments?

It depends heavily on file length and language. Our false positive rate on files under 40 lines was high enough that we stopped scoring them entirely and reviewed by hand. On larger files, with a cohort-specific baseline, the high-score band was reliable enough to justify an interview and nothing more.

Do I need the API, or can I use the dashboard?

Most courses do not need the API. The dashboard covers the full workflow for a single course and a single assignment; the API matters when you are running weekly passes across multiple sections, or when the same pipeline feeds a code plagiarism report that has to be reproducible months later during an appeal.

If you want to see the triage queue described in step 6 against your own submissions, you can check the pricing tiers or run a free check on a past assignment.