In the fall of 2023 I graded 214 submissions for the fourth assignment in CS1, and 61 percent of them called a helper function that appeared nowhere in my lecture notes, my slides, or the textbook. The function worked. That was the problem. If you want to catch AI-written code in student submissions, the short answer is that no detector score will do it alone, and the durable fix is assignment design: build the assignment so that the work leaves a trail a human being can read.
I have taught introductory programming for fifteen years and sat on two academic-integrity committees. What follows is not a policy argument. It is a description of what I now put in the handout, what I ask students to submit, and what I actually do on the Thursday night when the grading is finished and forty flags are sitting in a review queue.
Why a detector score will not settle a case in front of an honor board
Every AI code detector I have used, including the good ones, returns a probability. Probabilities are useful for sorting a queue and close to useless in a hearing. The committee asks a different question: what did this student actually do between the handout being posted and the file being uploaded? A number cannot answer that. A commit log sometimes can.
The second issue is base rate. Introductory assignments have very few correct shapes. A student who writes clean, idiomatic, well-commented Python for a linked-list exercise looks statistically unusual, because most novices write ragged code with a stray print on line 40. That tidiness is precisely what the detector was trained to notice. I once saw a 0.91 AI score on a submission written by a sophomore who had spent the previous summer at a paid bootcamp where an instructor drilled her on comment style. Nothing about the case was exotic. The score was correct in a statistical sense and wrong in an administrative one, and it cost the committee two meetings and the student three weeks of dread before it was resolved.
So I treat AI scores the way I treat a smoke alarm. Useful. Loud. Not a fire inspection.
Requiring artifacts of process costs students almost nothing
The cheapest change I made, and the one students complained about least, was asking for evidence of how the code came to exist. Not an essay. A short, structured note attached to the submission: the tools you used, the prompts you sent if any, the part you got stuck on, and the one design decision you would defend. Three sentences to a paragraph. Graded for specificity, not for honesty in the confessional sense, because a vague note is its own signal.
Alongside that, the repository has to show history. GitHub Classroom gives me commit timestamps for free, and I have learned more from timing than from any score. A student who writes 900 lines in a single commit at 2:14 a.m. the night before the deadline has told me something. So has the student with eleven commits across nine days, including one at 3:40 p.m. titled "broke the refill logic, reverting." That second student is doing the work, and the trail says so before I have read a line of code.
I am no longer looking for proof that a student cheated. I am looking for evidence that a student worked, and the absence of that evidence is far easier to defend in a hearing than the presence of a suspicious number.
Design around a visible intermediate state
This is the part that changed my outcomes most, and it costs one extra due date. Ask for something that exists before the finished program exists. A failing test. A hand-drawn state diagram photographed on a phone. A trace of three executions with inputs I choose. Whatever it is, it has to be specific enough that producing it requires having thought about the problem.
In CS2 I now require the test file five days before the implementation. Students get a stub and a rubric line that says a third test must fail on a plausible wrong implementation.
# test_rate_limiter.py, due five days before rate_limiter.py
import pytest
from rate_limiter import Bucket
def test_bucket_drains_and_refuses():
b = Bucket(capacity=5, refill_per_second=1)
for _ in range(5):
assert b.take() is True
assert b.take() is False
# TODO: add one test that passes on a correct token bucket and fails
# on the naive "reset the counter every whole second" version.
That TODO is the whole assignment compressed into a sentence. A student who cannot write the test has not understood the problem, and I find out on a Tuesday rather than during finals week. Students who can write it have usually already done the thinking that the implementation requires.
Change the problem so the generic answer is wrong
Language models are extremely good at the modal version of a problem. They are bad at your version, if your version has one deliberate irregularity in it. I put that irregularity in the starter file and forbid students from touching the signature.
# rate_limiter.py, starter file. Do not modify __init__ or take().
class Bucket:
def __init__(self, capacity, refill_per_second):
# Argument order matters here: capacity first, refill second.
self.capacity = capacity
self.refill = refill_per_second
self.tokens = capacity
def take(self, n=1):
raise NotImplementedError
Reversing two constructor arguments is a trivial change and it breaks most pasted solutions immediately, because the pasted code calls Bucket(refill, capacity) or reaches for a library that does not match the contract. What I care about is not that the code fails. It is that the failure produces a traceback the student then has to explain to me. Any student who can explain the traceback has learned something. Any student who cannot has a question to answer.
Personalization does the same job with less cleverness. Seed the input data with the student's university ID, or give each student a distinct log file to parse. The plagiarism-resistant assignment is usually the one where the correct output differs per student.
Make the grading conversation possible
I grade a random 20 percent of submissions with a five-minute code walk, worth 8 percent of the assignment grade. The student opens their own file and I point at a random line and ask why it is there. It takes a morning for 180 students with two TAs, and it has done more to change student behavior than any policy document I have written. Students who understand their code enjoy it. Students who do not tend to become very quiet.
Two rules keep this humane. First, the walk cannot lower a grade for style, only for inability to explain. Second, I never schedule a walk because of a detector flag. I schedule walks randomly, so the walk is a normal part of the course and not an accusation. The moment a code walk becomes a punishment, students stop volunteering for office hours.
Where detection tools still earn their keep
None of the above replaces scanning. It changes what the scanning is for. I run the whole cohort through a code plagiarism checker and an AI code detector after the late deadline closes, and I use the results to decide where to spend my attention, not to decide who is guilty.
The reason I consolidated onto Codequiry rather than running MOSS and a separate AI tool is that the three signals belong on one screen. Peer similarity catches the pair who split an assignment. Web matching catches the student who lifted a repository from GitHub or a Stack Overflow answer with the original variable names still attached. AI scoring catches the file that nobody in the room wrote. MOSS is free and I still respect it, but it has no web index and no AI component, and its output is a URL full of HTML that a first-year TA has to squint at. JPlag 5.x is well maintained and fast, and Dolos 2.x produces the prettiest reports of the three, but neither addresses generated code.
What I have found most usable is the per-file breakdown. An average AI score for a submission is nearly meaningless when one file is a 0.2 and another is a 0.87. The indicators printed next to each file give me something to quote in a conversation with a student, which is worth more than the number itself.
A worked example, with the numbers
Spring 2023, CS2, token bucket rate limiter, 189 submissions. The handout was one paragraph and a link to the Wikipedia article. Results: 38 percent of submissions scored in the high range for AI generation, six cases went to the honor board, two were upheld, and I spent most of April reading file diffs.
Fall 2024, same assignment, same course level, 203 submissions. Five changes: the failing test due five days early, the three-sentence process note, the reversed constructor arguments in the starter file, commit history as a submission requirement, and five-minute walks for a random fifth of the class. High-range AI scores fell to 11 percent, and one case went to the board.
I want to be careful about that comparison. It is not a controlled study, the cohorts differ, and the fall 2024 group had heard from upperclassmen that CS2 had gotten strange about due dates. What I can say with confidence is that the conversations I had with students changed character. Instead of arguing about a score, we talked about a test they had written or a commit they had made. That is a much better Thursday night.
What this costs, honestly
Grading time went from about four minutes per submission to about eleven. TA training takes a two-hour session at the start of term, and I have to be in the room for the first dozen walks because TAs are nervous about conducting them and students are nervous about attending. The process notes have to be read, which sounds trivial and is not, because a third of them are two words long and need a response.
One thing failed outright. In spring 2024 I tried running the walks for the whole class rather than a sample, with three TAs and 180 students. We abandoned it in week six. The scheduling alone consumed a TA's entire allocation, and the walks started feeling like interrogations because we were doing eight a day. The sample works. The full rollout does not, at least not at my institution's staffing levels.
Frequently asked questions
Can professors tell if an assignment was written by ChatGPT?
Not by reading it, not reliably, and certainly not alone. Reading catches careless use, such as a comment explaining a concept the student has never mentioned in section. The polished cases need artifacts, commit history, and a short conversation. A detector gives you a ranked list to start from.
Should I ban AI tools outright in my course?
I do not, because a rule I cannot enforce teaches students that rules are theater, and I would rather have a disclosed transcript than a hidden one. What I require instead is a specific process note. Students who used a tool and can explain the output do fine. Students who used a tool and cannot explain it have a problem regardless of what my syllabus says.
What is the cheapest change I can make this week?
Move one required artifact ahead of the implementation. A failing test, a diagram, a trace. It costs you a single extra due date and it converts a grading problem into a teaching moment five days earlier, when the student can still act on it.
The other half of the equation is having a scanner you trust to sort the queue once the deadline passes. You can see how the checker built for teachers handles a full cohort on a single course dashboard, which is roughly what I look at on a Thursday night.