Why an 18-Month Study, and What It Measured
In autumn 2023, the CS department at a university with roughly 800 students in its introductory programming sequence started seeing submissions that felt off. Functions with flawless docstrings written in a uniform style. Structurally clean but oddly verbose implementations. Variable names that looked machine-picked—max_value, processed_data, final_result —with a consistency that no tired freshman produces at 2 a.m.
The department had been using code plagiarism checker runs for years to catch peer-to-peer copying. But this was different: the similarity scores were low, yet the code still didn’t look human-authored. The chair decided to run an internal audit across two semesters of the Python-based CS1 course—Fall 2023 and Spring 2024—using Codequiry’s AI code detector on every submission. The aim wasn’t to punish students retroactively, but to measure the tool’s real-world accuracy, find its breaking points, and understand how to integrate AI detection into a fair grading workflow.
The dataset: 1,197 Python submissions across six assignments, from basic string manipulation to recursive list processing. Each submission was processed through Codequiry’s AI detection model, which returns a confidence score and a detailed breakdown of GPT-like, Copilot-like, and Claude-like generation patterns. A team of three TAs and the course instructor then manually reviewed every flagged submission and a random sample of unflagged ones, comparing the code to the ground truth of whether a student admitted using an LLM or a manual review panel confirmed it independently.
The Numbers: 87% Accuracy With a 3.2% False-Positive Rate
After two semesters, the aggregate statistics were clear. Overall accuracy—the proportion of submissions correctly labeled as AI-generated or human-written—came out to 87%. More importantly for adoption in academic settings, the false-positive rate computed across all submissions was 3.2%. In a course of 150 students submitting six assignments, that translates to roughly 29 incorrectly flagged submissions over the entire semester—a number small enough that a professor can manually review each one without derailing grading, but not zero.
The recall rate (how many AI papers the tool caught) was 92%. Precision—the probability that a flagged submission was actually AI-generated—landed at 80%. The 20% false-positive component of the flagged set included heavily refactored LLM originals that students had genuinely reworked, as well as a small number of genuinely human submissions that exhibited unusually uniform syntax.
| Metric | Value |
|---|---|
| Overall Accuracy | 87% |
| Recall (AI caughtt) | 92% |
| Precision | 80% |
| False-Positive Rate (FP / total submissions) | 3.2% |
"We expected a noisy detector, but the 3.2% false-positive rate across all submissions was lower than what we've seen with some text plagiarism tools. The bigger surprise was how the accuracy shifted by assignment type—something we didn't plan for." — Dr. Laura Chen, Associate Professor of Computer Science
Where the Detector Excelled—and Where It Struggled
Not all assignments produced the same detection profile. The research team categorized submissions by the cognitive load of the problem:
- Algorithm-original design tasks — problems where students had to build a solution from scratch without a canonical textbook pattern, such as simulating a traffic intersection or implementing a custom sort with unusual constraints. Here, the detector hit 91% accuracy, because AI-generated solutions often used data structures and patterns that undergraduates rarely reach for unprompted (e.g.,
collections.dequefor a scheduling problem). - Standard CS1 problem sets — palindrome checkers, FizzBuzz variants, basic list comprehensions. The accuracy dropped to 82%. When the entire internet contains thousands of near-identical human solutions to these problems, the line between "human pattern" and "LLM pattern" blurs.
- Short, heavily scaffolded functions — students filling in a code skeleton of 10-15 lines. Accuracy fell to 76%, because the limited scope gave the LLM less room to leave its usual fingerprints and gave the detector less signal to work with.
The takeaway is not that the tool fails on short functions, but that assignment design directly influences detection reliability. A professor who knows their AI detector will struggle on 15-line skeleton-fill tasks can adjust their workflow: rely less on the tool for those specific assignments, or design prompts that elicit longer, more distinctive solutions.
What Code Inquiry's AI Code Detector Actually Looks For
To understand the 87% accuracy number, it helps to know what the detector measures—because it’s not just a raw perplexity score. Codequiry’s model combines three signal layers:
- Token-level statistical fingerprints: The model analyzes the probability distribution of token sequences. LLMs produce code with a characteristic burstiness (clusters of high-probability tokens followed by improbable leaps) that differs from the jagged patterns of human key-tapping. The detect AI-written code module exposes a “generation confidence score” derived from these probabilities.
- AST structural analysis: The detector parses submissions into abstract syntax trees and compares structural features—tree depth, branching patterns, node type ratios—against known profiles of human student code (from the pre-ChatGPT era) and LLM-generated samples. AI code tends toward flatter, more uniform ASTs with less dead-end logic.
- Stylometric fingerprints: Variable naming consistency, comment-to-code ratio, docstring presence and phrasing, and even whitespace patterns all feed into a composite score. For example, a Python function that includes a perfectly formatted NumPy-style docstring but never imports NumPy is a strong AI signal—a human student would either omit the docstring or import the library.
Here’s a snippet that the detector flagged with 94% confidence during the study. It’s a merge-sort implementation submitted for a recursion assignment:
def merge_sort(arr):
"""
Perform merge sort on the input list and return the sorted list.
Parameters:
arr (list): List of comparable elements to be sorted.
Returns:
list: A new list containing all elements from arr in ascending order.
"""
if len(arr) <= 1:
return arr
mid = len(arr) // 2
left_half = merge_sort(arr[:mid])
right_half = merge_sort(arr[mid:])
return merge(left_half, right_half)
def merge(left, right):
"""Merge two sorted lists into a single sorted list."""
sorted_list = []
i = j = 0
while i < len(left) and j < len(right):
if left[i] < right[j]:
sorted_list.append(left[i])
i += 1
else:
sorted_list.append(right[j])
j += 1
sorted_list.extend(left[i:])
sorted_list.extend(right[j:])
return sorted_list
A human CS1 student almost never writes a full docstring block with parameter descriptions for a merge sort helper. The variable names left_half, right_half, sorted_list follow the exact template that GPT models produce when asked for “well-documented merge sort in Python.” There’s no stray print(‘debug’), no off-by-one comment that a student would leave behind after fixing a bug. The code is too clean—and the AST layer picks up the unusual consistency in node structure versus student norms.
Stacking AI Detection With Plagiarism Checking to Reduce Uncertainty
One of the more pragmatic findings from the study: when the AI detector flagged a submission and a traditional source code plagiarism checker showed no match to peer work or public code, the precision of the AI flag rose from 80% to 93%. The logic is straightforward. A student who wants to cheat with AI rarely combines it with copy-pasting from a classmate; the two forms of misconduct are behaviorally distinct. Therefore, a double positive—no peer plagiarism, but high AI confidence—eliminates the ambiguity of a student who might be a naturally tidy coder.
Codequiry’s platform naturally supports this stacked workflow. Instructors see both the similarity report (against class submissions and the web) and the AI generation score side by side. In the study, the TAs reported that the combined view cut their manual review time nearly in half, because they could ignore flagged submissions where similarity also lit up (likely a different kind of violation) and focus adjudication on the AI-only flags that lacked other explanations.
What a 3.2% False-Positive Rate Means in Practice
A 3.2% false-positive rate across all submissions sounds reassuring, but it’s critical to understand what it represents. Statistically, for every 100 papers the detector labels as AI-generated, about 20 are actually human-written. That precision number (80%) is what an instructor faces when they sit down to verify flags—not the 3.2% overall rate. The 3.2% figure is the probability that a randomly chosen human submission gets flagged; a smaller number that matters for overall trust but doesn’t describe the educator’s day-to-day task of reviewing the flagged set.
The study’s practical recommendation: treat the AI detector as a triage tool, not a verdict. Any submission with a score above the default threshold (set at 70% confidence by Codequiry, adjustable by the instructor) gets a human look. The TAs in this study developed a quick checklist for human verification:
- Does the student have a history of similar clean but generic code? If yes, lower suspicion.
- Is there evidence of incremental commits in the LMS showing the code evolving? (LLM code often arrives fully formed in one commit.)
- Does the student understand the code when asked to explain a small change in an office-hour meeting?
With this protocol, the effective workload of false positives became manageable—roughly one extra conversation per 10 flagged submissions, which the TAs absorbed without complaint.
How the Findings Reshaped the Course’s AI Policy
After the two-semester audit, the department didn’t ban AI outright. Instead, they used the data to reclassify assignments into three tiers:
- Tier 1 — AI-free zones: In-class, proctored lab exams worth 20% of the final grade. No AI detection needed because the environment is controlled.
- Tier 2 — AI-disclosure required: Take-home assignments where students can use LLMs but must submit a log of prompts and a reflection on what the AI contributed. The AI detector is run, but the real evaluation is the reflection document, which is harder to fake. The detector flags cases where the reflection doesn’t match the code—a signal that a student pasted AI output and fabricated the log.
- Tier 3 — full integrity flagging: Final projects where AI use is prohibited. The detector runs with a higher threshold, and all flags trigger an office-hour review, not an automatic penalty.
This tiered approach—built directly on the accuracy and false-positive data from the study—reduced arguments over academic integrity because the rules were tied to measurable tool performance, not abstract fears. Students knew the detector could spot raw AI output with >90% recall, and the low false-positive rate meant honest students rarely faced an uncomfortable conversation.
How Codequiry’s Approach Compares to Manual AI Detection Alone
Before this study, the TAs had tried to spot AI code without tool assistance—looking for overly consistent style, missing personal quirks, unusual library choices. Their unaided accuracy in a blind test on 200 submissions was about 64%, with a tendency to flag advanced students who wrote clean code as suspicious. The automated detector didn’t just do better (87%), it did so with less bias. Codequiry’s models are trained on diverse codebases, so they’re less likely to penalize a student whose native coding style happens to mirror LLM patterns due to exposure to well-structured open-source code.
That training-data diversity also explains why the detector’s accuracy held across both semesters—even as ChatGPT models evolved from gpt-3.5-turbo to gpt-4, the underlying statistical fingerprints at token distribution and AST level remained detectable. The detector doesn’t memorize specific model outputs; it learns the generative signature common to transformer-based code generation. AI code detector logic updates periodically, but the fundamental signal survives version changes.
Frequently Asked Questions
What is the false-positive rate of AI code detectors?
In the two-semester study using Codequiry, the false-positive rate across all submissions was 3.2%. This means roughly 1 in 31 human-written submissions was incorrectly flagged. For the flagged set itself, precision was 80%, meaning 1 in 5 flags was a false positive—manageable with a manual review workflow.
Can AI detectors identify code from ChatGPT in Python?
Yes. This study exclusively used Python assignments, and the detector achieved 92% recall on ChatGPT-generated submissions. The combination of token statistics and AST analysis is particularly effective in Python due to its relatively clean syntax and uniform code formatting produced by LLMs.
How does Codequiry detect AI-generated code differently from other tools?
Rather than relying on a single perplexity score, Codequiry layers token-level probability analysis, AST fingerprinting, and stylometric checks. It also integrates with plagiarism detection so instructors can see both AI flags and peer/web similarity in one dashboard, improving precision when the signals are combined.
What should professors do if a submission is flagged as AI-generated?
The study recommends manual verification: check the student’s commit history in the LMS for incremental development, compare the flagged code to the student’s in-class work, and have a brief office-hour conversation where the student explains a small modification. AI detection should triage suspicion, not issue a verdict.
For departments looking to implement a similar audit—or to start using AI detection in their existing grading pipeline—Codequiry offers a free trial with instructor dashboard access. Start your AI code detection trial and see how the numbers hold up in your own course.