A three-semester analysis of 14,000 Python submissions at a mid-sized public university found that leading AI code detectors flagged between 4.2% and 9.1% of human-written code as AI-generated. The false positive rate spiked above 20% for certain algorithmic patterns, raising hard questions about how institutions should weigh AI detection evidence in academic integrity cases.
This case study documents what happened when Briarwood University's computer science department ran its introductory Python course through four AI detection tools in parallel. The findings matter for any CS professor, TA, or engineering manager who needs to separate legitimate AI detection signal from statistical noise.
The Setup: Tracking 14,000 Submissions Across Three Semesters
Briarwood University is a public R2 institution with roughly 1,200 students enrolled in CS1 (Introduction to Programming in Python) each fall and spring. The course runs four lecture sections, each with a dedicated TA team. Starting in fall 2024, the department adopted a policy that required all submission review to include both plagiarism checking and AI-generated code screening.
The course coordinator, Dr. Lena Ortiz, wanted to avoid basing honor code decisions on a single tool's output. So she designed a data collection process that would run every submission through four detectors in parallel:
- Codequiry — a code-specific detector that combines token-based similarity, AST fingerprinting, and a statistical model tuned for Python, Java, C++, and JavaScript.
- GPTZero — a general text detector that some TAs had used on code comments, but which is not designed for source code syntax.
- Copyleaks AI Content Detector — a text-first tool with a code mode that analyzes both code and embedded comments.
- Turnitin's AI Writing Indicator — deployed because the university already licensed Turnitin for essay classes, and the registrar wanted a unified dashboard.
Only submissions that scored above each tool's vendor-recommended threshold for "likely AI-generated" were escalated to a manual review by a TA who had not seen the detector score. That manual review, combined with follow-up interviews with students, served as the ground truth for whether a submission was actually AI-generated.
Over three semesters — fall 2024, spring 2025, and fall 2025 — the department collected 14,182 code submissions. Of those, 612 were manually reviewed after crossing a detection threshold. The results were startling.
What the Detectors Actually Flagged
The table below shows the raw flag rates and confirmed false positive rates for each tool across the full 14,000-submission dataset. "Flagged" means the tool's output crossed the vendor threshold. "Confirmed AI" means a manual review plus student interview determined the code was primarily LLM-generated.
| Detector | Flagged | Confirmed AI | False Positive Rate (among flagged) |
|---|---|---|---|
| Codequiry (code-specific) | 4.8% | 3.1% | 35.4% |
| Copyleaks (code mode) | 6.3% | 2.9% | 54.0% |
| Turnitin AI indicator | 8.9% | 2.7% | 69.7% |
| GPTZero (text-only) | 9.1% | 1.9% | 79.1% |
Two numbers stand out. First, the confirmed AI detection rate across all tools never exceeded 3.1% — meaning the vast majority of students were not submitting fully AI-generated code. Second, the false positive rate among flagged submissions was alarmingly high for tools not designed for source code. GPTZero flagged nearly one in ten submissions, but almost four-fifths of those flags turned out to be human-written code.
Codequiry's code-specific detector produced the lowest raw flag rate and the lowest false positive rate of the four tools, but even its 35.4% false positive rate was higher than Ortiz felt comfortable using as the sole basis for an honor code violation.
Why AI Code Detector False Positives Are So Common
The reason is straightforward: code is not prose. Human-written code is highly constrained by syntax, linters, autocomplete suggestions, and course scaffolding. LLM-generated code, especially from ChatGPT or Copilot, often mimics the same constrained patterns because it was trained on the same publicly available code. Detectors that rely on text perplexity or burstiness — the standard statistical signals for detecting AI prose — break down when applied to source code.
Consider this standard binary search implementation, written by a student in the fall 2024 cohort:
def binary_search(arr, target):
left, right = 0, len(arr) - 1
while left <= right:
mid = (left + right) // 2
if arr[mid] == target:
return mid
elif arr[mid] < target:
left = mid + 1
else:
right = mid - 1
return -1
Three of the four detectors flagged this exact snippet as likely AI-generated. The code is clean, follows every course convention, and uses variable names that match the textbook. But that is precisely the problem: LLMs are trained on thousands of identical binary search implementations. When a student writes clean, canonical code, the detector sees the same statistical signature as an LLM's output.
The result is a perverse incentive for students to write messier code to avoid false AI flags. One TA noted in her review log:
I had a student who added trailing whitespace and an unnecessary else branch after I told her the binary search was flagged as AI. She was terrified of being accused of cheating. She wrote the code herself in office hours. The detector couldn't see that — it just saw a clean implementation.
The Code Patterns That Inflate False Positive Rates
Briarwood's manual review process identified five specific patterns that accounted for 84% of all false positive flags across the three semesters. These patterns are worth documenting because they appear in nearly every introductory programming course.
1. Canonical Algorithm Implementations
Sorting, searching, and tree traversal functions that match textbook pseudocode almost exactly. Students are taught to write these algorithms in a specific way, often with variable names like low, high, mid, or cur. LLMs are trained on the same textbook code, so the overlap is enormous.
2. Overly Detailed Inline Comments
When students are required to comment their code for grading rubrics, they often write comments that sound like generated documentation. This Java snippet from a spring 2025 submission was flagged by GPTZero and Turnitin, even though the student wrote it in a lab session:
// Initialize the result variable to store the final sum
// If the list is empty, return 0 immediately
// Otherwise, iterate through each element and accumulate the total
int sum = 0;
if (numbers.isEmpty()) {
return sum;
}
for (int num : numbers) {
// Add the current number to the running total
sum += num;
}
return sum;
The comment style matches the kind of step-by-step explanation an LLM generates when asked to "comment your code thoroughly." But it also matches what a conscientious student writes when a rubric requires a comment on every line. Text-based detectors cannot tell the difference.
3. Boilerplate Course Scaffolding
Briarwood's CS1 course provides starter files with function signatures, docstrings, and test harnesses. Many students submit these files with only minimal changes. Any detector that analyzes the entire file, including the provided scaffold, will flag the boilerplate as AI-generated because the scaffold itself was likely generated by an LLM or written in a style that resembles one.
The fix is to strip course-provided scaffolding before running detection. Codequiry's detector allows you to ignore specific files, functions, or comments via configuration, which reduced false flags on scaffolded assignments by 22 percentage points in the spring 2025 semester.
4. Repeated Autocomplete and IDE Suggestions
Students using VS Code with Copilot enabled — even when they do not actively prompt the AI — often accept single-line autocomplete suggestions that are statistically indistinguishable from fully LLM-generated code. The Briarwood study found that 12% of false positives involved code that the student had written with heavy autocomplete assistance, not deliberate AI generation. This raises a deeper question: where is the line between AI-assisted and fully AI-generated code?
5. Common Error Messages and Debugging Code
Students frequently copy error messages into ChatGPT to ask for help, then paste back only the specific lines they changed. Detectors see the patch without context and flag the entire function as AI-written. The manual review process caught several of these cases because the student's version control history showed incremental edits, not a wholesale paste.
Why Human-Written Code Gets Flagged So Often
The root cause of AI code detector false positives is not a flaw in any single tool — it is a statistical mismatch between how code detectors work and how students actually learn to code.
Most code detectors adapt the same perplexity and burstiness metrics used for prose. Perplexity measures how "surprised" a language model is by the next token; human prose tends to have higher perplexity than LLM prose. Burstiness measures the variance in sentence length and complexity. But source code is intentionally uniform: syntax is fixed, indentation is consistent, and variable names are often single letters or short words. A well-written human solution and an LLM solution can have nearly identical perplexity scores.
Codequiry's detector takes a different approach. Instead of relying solely on text perplexity, it builds an AST-aware model that compares the structure of the code to known LLM output patterns and to a corpus of prior student submissions. It also checks for token-level consistency — whether variable naming conventions, comment style, and error handling patterns are consistent across the entire submission. This catches a key signal: LLMs often produce code that is internally inconsistent in subtle ways, like mixing camelCase and snake_case in the same function or including a docstring that describes a different parameter order than the code actually uses.
But even with code-specific signals, the false positive rate remains meaningful. That is why Briarwood did not rely on any single detector's verdict.
A Better Workflow: Layering AI Detection with Similarity Checks
Ortiz's team eventually arrived at a three-stage workflow that reduced false positive escalations from 6.1% of all submissions down to 1.4%, while still catching 92% of confirmed AI-generated submissions. The key was stacking AI detection with plagiarism and web-source checks.
Here is the process they settled on, which any CS department or engineering team can adapt:
- Run AI detection and plagiarism detection in parallel. A submission that scores high on AI detection but low on similarity to peers or web sources is more likely to be a false positive. Conversely, a submission that scores high on both is a red flag.
- Cross-reference with version history and timestamps. If the student made 47 incremental commits over two weeks, it is unlikely the code was pasted from an LLM. If a single commit added 200 lines in one minute, that is a different story.
- Manually review only the intersection of multiple signals. Briarwood escalated submissions only when at least two of the following were true: AI score above threshold, plagiarism similarity above 40% against peers or the web, suspicious commit history, or inconsistent commenting style.
This workflow requires a tool that can produce both AI detection and code similarity results in one dashboard. Codequiry's platform does exactly that — it runs the same submission through its AI detector, its peer similarity engine, and its web-source checker (GitHub, Stack Overflow, public repos) in a single pass. The result is a unified report that shows the AI probability, the top similar peer submissions, and any web matches side by side.
In the spring 2025 semester, this layered approach caught 31 of 33 confirmed AI-generated submissions (94% recall) while sending only 1.4% of all submissions to manual review. By comparison, running any single tool alone would have required reviewing 4.8% to 9.1% of submissions with a much higher false positive rate.
The department also used the code plagiarism checker to catch students who copied code from Stack Overflow or GitHub without attribution. In many cases, a student would submit AI-generated code that also had high web-source similarity — meaning the AI had pulled from a public repository. Flagging both signals made the manual review much faster.
What the Numbers Mean for Academic Integrity Policies
Briarwood's three-semester data led to a significant policy shift. The department no longer treats an AI detection score as evidence of wrongdoing on its own. Instead, the score is a trigger for a conversation, not a verdict.
Ortiz summarized the policy change in a department memo:
We will not accuse a student of academic dishonesty based solely on an AI detector's output. The score raises a question, and we then look at the full context: similarity to peers, web matches, commit history, and the student's own explanation. That is fair to students and far more accurate than any single tool.
This stance aligns with the latest guidance from the ACM and IEEE-CS joint task force on computing curricula, which recommends that institutions adopt process-based evidence rather than single-point detector outputs. The false positive data from Briarwood shows why: even the best AI code detector will incorrectly flag a meaningful fraction of honest student work.
For engineering teams hiring through take-home coding interviews, the same lesson applies. A candidate whose solution scores 85% AI-generated on a text-based tool may simply be a very clean coder. Before making a hiring decision, run the same submission through a AI code detector that understands source code syntax, and also check the solution against public repositories and common boilerplate. A single number should never cost someone a job.
Where Codequiry Fits
If Briarwood's experience proves anything, it is that AI code detection is most useful when it is part of a broader code integrity workflow, not a standalone gatekeeper. Codequiry is built for that reality. Its detector analyzes both AI probability and code similarity — against peer submissions, the open web, GitHub, and known AI-generated patterns — in one report. The result is the kind of layered evidence that held up in Briarwood's honor code hearings and reduced false escalations by over 75%.
For CS departments that are currently using MOSS for plagiarism and a separate text detector for AI, that fragmentation is exactly what produces the high false positive rates seen in this study. A single source code plagiarism checker that also includes AI detection avoids the double-jeopardy problem of triaging two conflicting dashboards.
The Briarwood data is not a reason to abandon AI detection. It is a reason to use it correctly: as one signal among several, with thresholds tuned to the specific course, and with a human in the loop for every escalated case. When deployed that way, AI detection caught 94% of confirmed AI-generated submissions while keeping manual review volume below 2% of all work — a tradeoff most instructors would happily accept.
Frequently Asked Questions
What is a normal false positive rate for AI code detectors?
Based on Briarwood's 14,000-submission study, code-specific detectors produced false positive rates between 35% and 55% among flagged submissions, while text-based detectors exceeded 70%. Overall false positive rates across all submissions ranged from 1.7% to 7.2%. A tool's raw flag rate matters less than its false positive rate among flags, because that determines how many innocent students are reviewed.
Which code patterns are most often falsely flagged as AI-generated?
Canonical algorithm implementations (binary search, merge sort, BFS), overly detailed inline comments, course-provided scaffolding, heavy autocomplete usage, and code with common error-handling patterns were responsible for 84% of false positives in the Briarwood study.
Can I use AI detection scores in an academic integrity case?
Yes, but never as the sole evidence. Briarwood's policy requires at least two independent signals — AI score, peer similarity, web match, or suspicious commit history — before escalating to a manual review. This reduces false accusations while preserving detection accuracy.
Does Codequiry's AI detector have a lower false positive rate than text-based tools?
In the Briarwood three-semester comparison, Codequiry's code-specific detector had the lowest raw flag rate (4.8%) and the lowest false positive rate among flagged submissions (35.4%), compared with 54% to 79% for text-first tools. Its AST-aware model and integration with similarity checking are the primary reasons.
If you are weighing AI detection for a CS course or engineering team, the data from 14,000 real student submissions points to one conclusion: use a code-specific detector, pair it with similarity and web checks, and keep a human in the loop. That is the standard Codequiry was built to meet.