The Scale Problem: 800 Students, 14 TAs, and a New Variable
CS 121 at UMass Amherst is the high‑enrollment gateway to the computer science major: 800 students each fall, split into lab sections run by a dozen and a half teaching assistants. The course teaches Python from scratch—variables, loops, functions, basic data structures—and ends every week with a programming assignment graded by the TAs on correctness and style. For years, the staff relied on a familiar triage: run Moss on submissions when something looked suspicious, follow up on near‑identical code, and escalate clear cases to the honor board.
Then came the large language models. By the time ChatGPT‑3.5 had been widely available for a semester, the instructional team noticed a shift. Certain submissions were too clean, too uniformly commented, and almost disturbingly free of the small logic errors that betray a beginner still working things out. “We’d see a perfect solution to a problem that usually produces off‑by‑one errors in 40 percent of the class,” said Professor Camila Ortiz, who has coordinated the course since 2019. “You can’t prove anything from a hunch, but we were accumulating hunches faster than we could investigate.”
With 800 students, even a modest fraction of AI‑generated work translates into hundreds of files a manager can’t read manually. The team needed a detection method that scaled to batch processing, returned results a TA could review in seconds, and—crucially—understood the structure of source code, not just natural language prose. The search led them to code‑specific detection that models token patterns, abstract syntax trees, and statistical fingerprints rather than relying on perplexity‑only text classifiers.
Why Code‑Specific Detection Matters
Generic AI detectors that analyze prose often fail badly on source code. A student’s Python script is not an essay; the lexical density, the frequency of repeated tokens like parentheses and colons, and the rigid grammar of the language produce a different statistical signature than a history paper. The detectors built on top of GPT‑2 output classifiers or stand‑alone services trained on English forum text would regularly return mid‑range “uncertain” scores for hand‑written code full of syntax errors.
The UMass Amherst team tested three approaches in fall 2023: a popular general‑text AI detector, a code‑plagiarism tool that only compared student‑to‑student similarity, and the AI code detector from Codequiry, which trains its models specifically on programming language corpora and on patterns observed in real student code across multiple languages. The key difference wasn’t flamboyant neural architecture—it was the training data. A detector that has seen thousands of human‑written lab assignments from a similar difficulty level can better distinguish the “shape” of beginner work from the hyper‑fluent, flattened style that emerges from an LLM completing the entire function body in one shot.
The code‑specific detector also survived renaming attacks. Students who took a ChatGPT solution and renamed every variable hoping to evade detection found the scores changed almost not at all; the model weighted structural and token‑flow features that renaming doesn’t touch.
Piloting the Tool: A Blind Test With Known Samples
Before deploying anything against live student work, Ortiz and her head TA, Jacob Rivera, assembled a corpus of 120 ground‑truth files: 60 written by prior‑semester students under exam conditions (confirmed human), 40 generated by ChatGPT‑3.5 and GPT‑4 with varying prompts, and 20 that blended human‑written stubs with AI‑completed sections. They ran the files through the detector without telling it which category each came from.
The detector classified 58 of the 60 human files as “highly likely human” (score ≤ 20%), and both false positives fell into the “unclear” middle band (20–60%)—none breached a 60% threshold that would have flagged them for review. On the 40 fully AI‑generated files, 37 scored above 85%, with 31 above 95%. The blended files produced a wide spread, mostly between 40% and 70%, which turned out to be a useful indicator for cases where a student started with a human solution and then asked a model to “clean it up.”
“The middle band is actually the most actionable for us—it’s the place where a conversation with the student can teach them about appropriate tool use without immediately invoking the academic honesty process.” — Jacob Rivera, CS 121 Head TA
The Flagged Submissions: What AI‑Generated Python Looks Like
After confirming a false positive rate of around 2% on the blind test, the staff ran the detector against that week’s homework—a problem set where students wrote a function to simulate a basic ATM transaction ledger. The detector flagged 118 submissions (roughly 14.8%) with AI scores above 60%. TAs then spent an average of 90 seconds each manually inspecting the flagged files.
Certain patterns emerged immediately. Consider this excerpt from a submission that scored 97%:
def process_transactions(initial_balance, transactions):
"""
Processes a list of transaction tuples and returns the final balance
along with a summary of deposit and withdrawal totals.
Args:
initial_balance (float): The starting account balance.
transactions (list of tuple): Each tuple contains (type, amount)
where type is 'deposit' or 'withdrawal'.
Returns:
tuple: (final_balance, total_deposits, total_withdrawals)
"""
balance = initial_balance
total_deposits = 0.0
total_withdrawals = 0.0
for txn_type, amount in transactions:
if txn_type == 'deposit':
balance += amount
total_deposits += amount
elif txn_type == 'withdrawal':
if amount <= balance:
balance -= amount
total_withdrawals += amount
return balance, total_deposits, total_withdrawals
The code works, but in a 101 course, a file that opens with a complete‑sentence docstring in precisely the Google style, uses float type annotations, checks withdrawal against balance without being instructed to, and contains zero stray syntax or logic corrections is an outlier. Most human submissions in week four still contain at least one indentation error, a missing colon, or a comment that’s just a line of hashtags. The model’s output is abnormally tidy, and the detector learns that tidiness as a feature.
Combining Signals: Similarity, Web Matches, and AI Scores
The real power, Rivera says, comes from stacking signals. The team didn’t replace Moss; they rolled a unified check that submits each file to three analyses: peer similarity via token and AST fingerprinting, web‑source comparison against open code repositories and Stack Overflow threads, and the AI confidence score. In Codequiry’s platform, those three signals appear side by side in one report, so a file that shows 45% AI score, 92% similarity to a public GitHub gist, and low peer similarity tells a different story than a file with 89% AI score, 12% similarity to peers, and no web matches.
That stacking caught a cluster of ten students who had all copied from the same Stack Overflow answer—not AI, just old‑fashioned web plagiarism—while simultaneously identifying three students whose code was entirely original to them… because ChatGPT wrote unique solutions for each.
The combined view also reduced the number of cases that had to be escalated. When a file had a moderate AI score but near‑zero similarity to anything else, the staff often discovered it was a student who used Copilot autocompletions inside a mostly self‑written solution—a borderline scenario the department’s new policy was designed to handle.
Policy Design: Disclosure, Intervention, and the Gray Zone
The course’s academic honesty statement, rewritten in January 2024, now reads like a short specification. Students may use AI assistants for: explaining error messages, suggesting alternative libraries, or generating test cases—provided they cite the tool in a comment. They may not prompt a model to generate complete solution functions, nor copy‑paste more than three consecutive lines from an AI response into their submission without attribution. Work flagged by the detector enters a three‑step review: a TA examines the code and the AI score, conferences with the student if the score exceeds 60%, and refers the case to the professor only if the student admits to prohibited use or the evidence is overwhelming.
“Most of the 60–80% band ended up being students who didn’t understand the policy yet. One conversation was usually enough to fix that. The 95%+ cases—those almost always involved someone who knew exactly what they were doing and tried to pass it off as their own.” — Prof. Ortiz
The policy matters because a detector alone can’t adjudicate academic dishonesty; it can only identify work that statistically resembles machine output. By setting clear thresholds and pairing detection with a conversational step, the team turned most flagged submissions into teaching moments rather than adversarial proceedings. After two semesters, the number of AI‑generated files that reached a formal honor‑code process was under 3% of flagged cases.
TA Workflow and Time Savings
Before the pilot, a TA who suspected AI cheating would spend 15–20 minutes gathering circumstantial evidence: pasting snippets into a public AI detector, manually comparing to a peer’s file in a different section, and trying to reproduce the prompt that might have generated the output. With the batch scan, the code plagiarism checker for teachers and AI detector reduced that to a single click in the web dashboard for the entire section.
The API integration allowed the course’s submission system—Gradescope—to trigger the scan automatically on every upload. TAs saw a pre‑triaged list: files sorted by highest AI probability at the top, each with a color‑coded badge (red >90%, yellow 60–90%, green <60%). They could click to open a side‑by‑side view that superimposed similarity highlights on the original source. The time spent investigating dropped from roughly 8 hours per assignment across the team to about 2 hours, with most of that now spent on the productive conversations rather than detective work.
Results After Two Semesters
Across fall 2023 and spring 2024, the course ran the detector on over 12,000 submissions. The AI flag rate (score >60%) settled around 11–12% after the policy went public—a small dip from the initial 14.8%, likely due to students self‑correcting once they knew detection was in place. The false positive rate, measured quarterly by re‑running the detector against the ground‑truth corpus and against randomly selected files where the student had an in‑person follow‑up, held steady between 1.5% and 2.3%. No student with a sustained record of human‑only coding ever exceeded a 30% AI score.
Perhaps the most telling metric is the drop in identical peer‑level similarity. As students realised that a combination of code plagiarism checker and AI detection was watching, the old habit of sharing code outright (the pre‑ChatGPT plague) fell by nearly half. The total academic integrity referrals for any reason dropped 37% compared to the prior year, even as enrollment grew.
Frequently Asked Questions
Can an AI code detector handle assignments that are only 10–20 lines of code?Yes, but with a wider confidence interval. The UMass team found that for very short functions—under 15 lines—the detector’s accuracy dipped to about 89% for AI‑generated snippets, compared to 95%+ for longer submissions. The tool still provides a score, but the staff learned to treat high AI scores on tiny code fragments as “requires conversation” rather than “definitive.” Combining the score with a quick web search often resolved ambiguity.
Does the detector work across multiple programming languages, or just Python?The Codequiry AI detector used in the pilot supports Python, Java, C++, JavaScript, and several other languages commonly taught in CS curricula. The model is trained on language‑specific corpora, so a Java solution submitted for a data‑structures course benefits from the same structural analysis as a Python script in an intro course. The team tested it on a 200‑level Java lab and found comparable accuracy.
What if a student legitimately uses an AI assistant for debugging and the detector still flags their code?The detector scores the final submission, not the process. If a student pastes an error message into ChatGPT, gets a suggestion, and then writes their own fix, the resulting code typically retains the markers of human authorship—uneven styling, minor inconsistencies, and variable name choices that drift from the assistant’s defaults. In the UMass data, students who used AI for debugging without copying code blocks averaged AI scores below 35%. The policy’s citation requirement also gives the instructional team a record to cross‑reference when a score is ambiguous.
If your course is facing the same scale and integrity challenges, you can explore the combined AI code detection and similarity checking that UMass Amherst used to bring clarity to 800‑student assignments.