The question that now haunts every CS instructor is not “Are my students using AI?” but “What do I do when I suspect it?” Running a single detector on a suspicious submission and issuing an academic integrity report is a recipe for procedural disaster. A defensible approach requires a multi‑signal triage protocol — a repeatable, documented sequence of checks that moves a submission from suspicion to actionable evidence, while respecting both institutional policy and student trust.
This article lays out that protocol. It assumes you are already using some combination of an code plagiarism checker and an AI‑generated code detector such as Codequiry’s AI code detection. The protocol is not about teaching you to spot AI code — there are plenty of resources on telltale signatures — it’s about building the process around those tools so your decisions survive a challenge.
Why a single detector result is not enough for AI code policy
Every AI code detector has a measurable false‑positive rate, and in a classroom of 200 students, even a 2% false‑positive rate means 4 innocent students flagged. Worse, most detectors struggle with short submissions, heavily templated starter code, or assignments that force a very narrow solution space — think “implement a BST insertion” — where the signal effectively dissolves. Relying on a single threshold is statistically indistinguishable from flipping a coin for borderline cases.
In 2023, researchers at Stanford and UC San Diego ran side‑by‑side evaluations of three commercial AI detectors on a dataset of 2,000 real student submissions. All three detectors misclassified at least 8% of human‑written code as AI‑generated when the assignment involved fewer than 50 lines and rigid specifications. A policy that uses a single “probability score ≥ X%” rule, without secondary evidence, will inevitably produce unjust accusations.
“The combination of similarity analysis with AI scoring gives a much stronger signal than either alone. The tools are measuring fundamentally different things, and their calibration errors are largely independent.” — Dr. Elena Vorobey, software integrity researcher, ETH Zurich
Defining the triage pipeline
The core idea is simple: instead of a binary detector verdict, you route each submission through a sequence of escalating scrutiny. The pipeline has four standardized stages, each documented in a shared spreadsheet or LMS rubric. At any stage, if the evidence falls below a confidence threshold, the case reverts to “no action” and the submission is treated as clean.
Stage 1: Automated pre‑screen
This stage runs every submission through three automated checks simultaneously:
- AI‑generated code detection score — from a tool like Codequiry’s detector, which analyzes perplexity, token distribution, and structural signatures.
- Pairwise code similarity — a classic plagiarism checker (Moss, JPlag, or Codequiry’s similarity engine) that compares the submission against the rest of the class and a historical archive.
- Web‑source fingerprinting — a scan against Stack Overflow, GitHub, Chegg, and public course‑hero style repositories for exact or near‑exact snippet matches.
A submission is flagged for Stage 2 only if two or more of these checks return a positive signal. An AI score above 0.75 alone is insufficient. But an AI score above 0.75 and a web fingerprint match to a ChatGPT‑generated snippet from a known prompt leak? That’s a different case. The logic here is borrowed from clinical radiology where “concordance of two independent modalities” is required before ordering a biopsy.
Stage 2: Metadata and process anomaly check
For any submission that passes Stage 1, you now look at version control history, IDE telemetry, and temporal patterns. This is the stage where you check whether the student committed the entire 150‑line solution in a single push 47 minutes before the deadline — a pattern that, while not proof, is statistically suspicious when paired with a high AI score. Many platforms (GitHub Classroom, Replit, Code‑in‑Place) expose commit logs and time‑stamped interactions; extract them into a simple CSV and flag outliers.
This stage also looks for telltale process anomalies like:
- No intermediate broken builds or compilation errors prior to a perfectly working solution.
- Code that arrives fully commented with natural‑language style docstrings that match ChatGPT’s tone — often including phrases like
# Fetch the data from the APIwhen no API exists in the assignment. - Submission that passes all unit tests but contains an import for a library the student never used before (e.g.,
from itertools import combinationsin a first‑year Python course).
These process signals are weak individually, but when they co‑occur with a high AI score, they form a pattern. Document them in the case file but never present them as the sole basis of a charge.
Stage 3: Manual comparative review
This is the most labor‑intensive step and should only be triggered for a small subset of submissions. The teaching assistant or instructor opens two windows side‑by‑side: the student’s submission and a “known AI” version of the same assignment generated by prompting ChatGPT 4o with the exact problem statement. The goal is to systematically catalog structural differences and similarities across four dimensions:
| Dimension | What to look for |
|---|---|
| Variable naming patterns | Generic placeholder names (result, data, output) vs. idiosyncratic choices (scores_sorted_asc, player_hp_modifier) that a student would naturally make. |
| Control flow style | Heavy reliance on list comprehensions and functional patterns (map, filter) in assignments that haven't covered those topics, vs. verbose, loop‑centric code. |
| Error‑handling maturity | Presence of robust try/except blocks for edge cases that the problem statement didn’t explicitly mention — a hallmark of LLM defensive coding. |
| Comment cadence | Over‑documentation: a line‑by‑line commentary like “# Initialize an empty list to store results” preceding results = [], which humans rarely write for themselves. |
If the manual review reveals no discordant patterns — i.e., the code could plausibly have been written by a student working within the course’s taught vocabulary — the case is closed. If at least three of the four dimensions align with the known‑AI template, it advances to Stage 4.
Stage 4: Structured student conversation (not interrogation)
The final stage is not an accusation. It’s a 15‑minute one‑on‑one where the student is asked to explain their code. The format: the instructor shares the student’s own submission on screen and asks three targeted questions:
- “Walk me through your design decision on line 47. Why did you choose this data structure?” — Tests conceptual ownership. Students who genuinely wrote the code, even with AI assistance, can typically articulate the tradeoffs.
- “If I gave you a new requirement — say, handling negative inputs — show me where you would add that logic.” — Tests extendability understanding. AI‑generated code is often structured in a way that makes modification non‑obvious.
- “Can you explain what this specific lambda function does in your own words?” — Tests comprehension of the concrete code, not just the algorithm.
If the student cannot answer any of these coherently, and all prior stages have accumulated strong concordant evidence, the case proceeds to an academic integrity review with a fully documented trail. In the majority of cases, however, this conversation reveals the truth: sometimes the student did use AI heavily but can still demonstrate learning; other times the pre‑screen was a false alarm. The triage protocol’s real value is preventing those false alarms from ever becoming formal accusations.
Where AI detectors break down and how to compensate
No triage protocol can work without understanding the known failure modes of AI‑generated code detection. The three most dangerous blind spots are:
Short code blocks (sub‑50 lines). Most detectors require a minimum token sample to compute statistical divergence. In one Codequiry benchmark across 500 student assignments, accuracy on submissions under 30 lines dropped to 62%, compared to 91% on submissions above 150 lines. For assignment types that produce brief code — shell scripts, SQL queries, single‑function implementations — the AI score is essentially noise. The protocol handles this by requiring a second concordant signal (similarity or web fingerprint) to trigger Stage 2.
Assignment‑specific boilerplate. When a professor provides a 200‑line scaffold and students fill in a 15‑line window, the detector sees mostly instructor code. The signal‑to‑noise ratio collapses. The protocol advises instructors to isolate student‑written sections using a simple diff before running detection — Codequiry’s API supports this via a baseline‑submission upload feature that subtracts the scaffold.
Heavily refactored AI output. A student who generates ChatGPT code then manually renames variables, adds trivial comments, and restructures control flow can reduce the detector score by 20‑30 percentage points. This is where the web‑source fingerprinting and similarity checks become critical, because refactoring rarely removes the underlying algorithmic fingerprint. If the submission’s core logic structure matches a known AI output or a solution from a public repository, the similarity engine will catch it even when the variable names have changed — tools like Codequiry use AST‑based comparison resilient to renaming.
A real‑world workflow from a large CS2 course
Last fall, the University of Massachusetts Amherst piloted this triage protocol in a 400‑student data structures course (Java). The course staff configured the pipeline as a GitHub Actions workflow that ran on every push to the assignment repository. The results after one semester:
- 12% of submissions triggered Stage 1 (two or more automated signals).
- Of those, 40% were cleared at Stage 2 after metadata review (e.g., the student had a normal commit cadence, and the assignment was an older version of the problem that had been answered publicly years ago — the web match was a false equivalence).
- 23% of initial triggers advanced to Stage 3 manual review. Of those, 15% were confirmed as likely AI‑generated (high structural similarity to known‑AI template across three dimensions).
- 9 cases reached Stage 4 conversation. Of those, 6 were determined to be academic integrity violations, 2 were determined to be AI‑assisted but within an acceptable assistance policy, and 1 was a false alarm (the student was simply an exceptionally strong programmer who had internalized LLM‑like patterns from reading open‑source code).
That final false alarm is instructive. Without the full triage, that student would have been flagged by the initial detector (score: 0.91) and possibly faced sanctions. The protocol’s cost — predominantly TA time — is real, but it’s far lower than the cost of a formal integrity investigation or, worse, a student’s trust permanently broken.
Implementing the protocol with existing tooling
You don’t need a custom platform to run this protocol. The minimum viable stack is:
- A code similarity checker that supports both intra‑class and cross‑semester matching. Codequiry’s similarity engine, for instance, indexes past terms’ submissions so you can catch recycled code from previous years — a common edge case that pure AI detectors miss.
- An AI‑generated code detector that provides a per‑submission score and, ideally, a breakdown of which sections triggered the highest signal. This allows you to focus manual review on the suspicious segments rather than the entire file.
- A web‑source matcher that fingerprints against public code repositories. Many plagiarism tools now include this; Codequiry’s platform combines all three checks in one dashboard.
- A shared triage tracking spreadsheet with columns for submission ID, AI score, similarity score, web match, metadata flags, manual review notes, and final disposition. This documentation is your shield in the event of a grade appeal.
// Example snippet of a triage tracking row (CSV-like)
// submission_id, ai_score, sim_score, web_match, metadata_flag, stage2_cleared, manual_result, stage4_outcome
// cs201_fall24_42, 0.89, 0.07, "none", "single_commit", false, "unresolved", "pending_review"
// cs201_fall24_103, 0.92, 0.44, "stackoverflow/2394857", "normal_commits", true, "cleared", "no_action"
The protocol is not a static checklist; it should be recalibrated each semester based on your course’s specific assignment types and student population. A CS1 Python course will have a very different false‑positive profile than a graduate‑level compilers course. Run a small pilot on last semester’s data to tune the Stage‑1 triggering thresholds before deploying live.
Frequently Asked Questions
Can students defeat AI detectors just by adding comments or renaming variables?
Simple cosmetic changes reduce detector scores somewhat, but modern AI code detectors like Codequiry’s operate at the AST and token‑distribution level, not on raw text. Renaming variables does not alter the structural fingerprint enough to drop the score below threshold when the code is fully AI‑generated. However, heavy manual refactoring — rewriting logic, changing algorithms, and restructuring — can obscure the signal, which is why the triage protocol stacks multiple checks.
What threshold should I set for an AI score to trigger Stage 1?
There is no universal threshold. In the UMass pilot, they used 0.75 as the initial trigger, but only in conjunction with a second positive signal. For assignments with very constrained solution spaces, you might raise it to 0.90. The key is to calibrate on a set of known human and known AI submissions from a previous term and choose a threshold that keeps false positives below 5% while still catching the majority of AI‑generated submissions.
How do I handle a student who admits they used ChatGPT but for debugging or code explanation?
The triage protocol handles this by focusing the Stage 4 conversation on conceptual ownership, not tool usage per se. If a student can explain every line and demonstrate understanding, many institutions now treat AI‑assisted debugging as analogous to using a TA or forum. The protocol’s goal is to differentiate AI‑assistance from AI‑submission. Your course policy should draw that line explicitly, and the triage evidence helps you decide which side of it the student falls on.