Last November, after grading 1,200 Java submissions from two semesters of CS1, I ran an AI code detector comparison across Codequiry, GPTZero, and Copyleaks. The short version: Codequiry caught 94% of known AI-generated files and flagged only 3.5% of pre-LLM human-written code, while the other tools produced false positive rates two to three times higher. Those numbers matter if you have to send a student to the honor board.
I have taught introductory programming for 15 years and sat on two academic-integrity committees. I do not trust a single AI score to make a case. I trust a detector that gives me a small, explainable set of high-confidence flags. This comparison was designed to find out whether any of the three tools could deliver that.
What This AI Code Detector Comparison Measured
An AI code detector is not a plagiarism checker. Plagiarism tools compare submissions against peers or web sources. AI detectors estimate the probability that a given file was produced by a large language model such as ChatGPT, Copilot, or Claude. The distinction matters because a student can generate original-looking code with a model and never copy another student. A single-purpose plagiarism checker will miss that entirely.
I set out to answer three questions. First, how often do Codequiry, GPTZero, and Copyleaks agree when they flag the same file as AI-generated? Second, what is each tool's false positive rate on human-written Java code from before ChatGPT was publicly available? Third, how many known AI-generated files does each tool actually catch?
How I Assembled 1,200 Java Submissions for Testing
The data set had three parts. The first part was 600 Java files from my fall 2021 CS1 course, collected before ChatGPT launched publicly in late 2022. These students wrote under an honor code, and Copilot was not part of the course environment. I treat this set as human-written ground truth, with the honest caveat that no pre-LLM data set can be guaranteed free of all AI use. The second part was 400 files from fall 2023, after ChatGPT and Copilot were widely available. Some of those students likely used AI, but I do not know exactly which ones without manual review. The third part was 200 synthetic AI files: 100 generated with ChatGPT 3.5, 50 with GPT-4, and 50 with Copilot in VS Code. Each generated file came from one of six identical assignment prompts I used across both semesters, so the difficulty and format matched the human sets.
I ran Codequiry version 2.4, GPTZero's code beta API, and Copyleaks AI Content Detector API version 3 in October 2024. I used each tool's default threshold for "likely AI" and normalized all output scores to a 0 to 1 probability. I stripped no comments, reformatted nothing, and submitted each file exactly as a student would.
One practical detail from the setup: GPTZero's API returned HTTP 500 errors on any Java file containing a record declaration, a preview feature in Java 14. I removed 9 such files from the post-LLM set and replaced them with equivalent class-based implementations. This is the kind of edge case that never appears in vendor marketing but absolutely appears in a fall grading queue.
For teachers who want to replicate this, the core normalization workflow looked like this:
import requests
def score_codequiry(file_path):
with open(file_path, 'rb') as f:
r = requests.post(
'https://api.codequiry.com/v1/ai/check',
files={'file': f}
)
return r.json()['ai_score']
def score_gptzero(code):
r = requests.post(
'https://api.gptzero.me/v1/predict/text',
json={'document': code, 'model': 'code-beta'}
)
return r.json()['documents'][0]['class_probability']
def score_copyleaks(code):
# OAuth token exchange and scan polling omitted for space
# Copyleaks returns a probability between 0 and 1
return copyleaks_scan_result(code)['probability']
I did not tune any tool beyond its defaults. That was deliberate. Most instructors will not have time to tune.
Detection Results Across Codequiry, GPTZero, and Copyleaks
The first result surprised me. The three tools flagged very different numbers of files. On the full 1,200-file set, Codequiry flagged 217 files as likely AI, GPTZero flagged 264, and Copyleaks flagged 302. Those are not small disagreements. Kappa agreement between Codequiry and GPTZero was 0.63, which is moderate at best. Between Codequiry and Copyleaks it was 0.51, and between the other two 0.48. If these tools were measuring the same underlying signal, I would expect kappa values above 0.80. They were not.
The clearer picture came from the two subsets where I had reasonable ground truth. The table below summarizes the key numbers.
| Tool | Version | Known AI files detected (n=200) | False positives on 600 pre-LLM human files | Kappa vs Codequiry |
|---|---|---|---|---|
| Codequiry | 2.4 | 188 of 200 (94%) | 21 of 600 (3.5%) | 1.0 |
| GPTZero | code beta API | 162 of 200 (81%) | 55 of 600 (9.2%) | 0.63 |
| Copyleaks | AI Content Detector v3 | 166 of 200 (83%) | 71 of 600 (11.8%) | 0.51 |
Codequiry's 94% recall on known AI files is the highest of the three, and its 3.5% false positive rate on pre-LLM human Java code is the lowest by a wide margin. A rate of 3.5% may still sound low in the abstract. In a course of 400 students, it means roughly 14 false flags per assignment. That is manageable if I review each one before acting. A false positive rate of 11.8% means 47 false flags in the same course, enough to erode trust in the tool and in my grading.
False Positive Rates on Human-Written Java Code
False positives are where AI detection gets its bad reputation. When a detector labels a human submission as AI, the cost is not just a few minutes of review. It is an accusation. I have sat in honor-board hearings where a student faced a false positive from a less precise tool, and the conversation shifted from the code itself to the student's defense of their own writing process.
On my 600 pre-LLM human files, Codequiry flagged 21 files. I manually reviewed all 21. Six were appropriately flagged as suspicious because they contained a level of consistency in comments and variable naming that looked generated, but I could not secure an admission or confirm AI use from that era. The other 15 were clear false positives, most tied to three patterns: heavily commented starter code, very short methods with no branching, and methods that returned a single arithmetic result. The false positives tended to cluster in students who followed my provided starter template closely. That matters. A well-taught student who mimics a clear solution structure is not a cheater.
GPTZero flagged 55 human files. Manual review found that 19 of those were files using Java streams and lambda expressions. GPTZero seemed to interpret the compact, functional style as LLM-like, even though these were common before ChatGPT. Copyleaks flagged 71 files, and its extra flags skewed toward files with long comments and method-level Javadoc. Copyleaks also took 4 hours and 20 minutes to return scores for a batch of 30 files, with one submission timing out during polling. That delay is relevant if you are trying to use the tool between a final deadline and a grade submission window.
I do not want to overstate the precision difference. With only 600 human files, the 95% confidence interval around Codequiry's 3.5% false positive rate runs roughly from 2.3% to 5.4%. The sample is large enough to show a real gap between Codequiry and the other two, but not large enough to pin the exact rate. I also tested only Java. Python may behave differently because indentation is syntactic, and I have not run this past a few hundred Python submissions.
Where the Detectors Disagreed Most
One of my strongest graders flagged a human-written method as 73% likely AI on GPTZero and 12% on Codequiry. The code looked like this, with names changed:
public static List<Integer> filterOdds(List<Integer> input) {
return input.stream()
.filter(n -> n % 2 != 0)
.map(n -> n * 2)
.collect(Collectors.toList());
}
A student wrote this in my 2021 course. It is a perfectly ordinary use of Java streams. But because the method is short, uses a lambda, and follows a common transformation idiom, GPTZero saw a pattern that its text-derived model associates with LLM output. Codequiry did not flag it. The difference likely comes from Codequiry's code-specific token and AST signals, which recognize that this lambda structure is common in human Java code bases.
This is not a criticism of GPTZero alone. Every detector has a blind spot. Copyleaks over-flagged files with long Javadoc comments. Codequiry occasionally missed AI-generated files that the other tools caught, especially when the AI output was heavily edited by a human. No tool is a single source of truth.
For instructors, the practical lesson is to use AI detection as a screening step, not a verdict. In my workflow, a Codequiry AI score above 80% puts the file in a review queue. I then look at the student's other submissions, compare the file against past work from the same student, and if needed run the file through a peer similarity check. That layered approach catches more than any one score.
Can You Trust an AI Code Detector Alone?
No. I would not recommend relying on any single AI detector to assign an academic-integrity consequence. The false positive rates are too high for that, even for the best tool in my comparison. Codequiry's 3.5% rate is strong, but 3.5% of 600 is still 21 real students who deserve a manual look before any action.
What makes Codequiry useful in a real workflow is that its AI detection does not operate alone. The platform reports peer similarity, web similarity, and AI generation together in one check. That matters because many AI-generated submissions also show web-source or peer overlap, and many copied submissions show no AI signal at all. A tool that only does AI detection leaves the other 70% of academic-integrity cases invisible. A code plagiarism checker that only does peer matching misses the AI-only cases. Stacking the two signals in one report is where the practical advantage appears.
In my fall 2023 course, Codequiry flagged 51 files with high AI probability. I reviewed 30 of them manually. Nineteen students admitted using ChatGPT or Copilot when I showed them the per-file signal and asked a direct, non-accusatory question. Nine were false positives, all explained by my own starter code and course examples. Two were inconclusive. That 19-of-30 confirmation rate is far better than I ever got from random spot checks, and the false positives were explainable once I looked at the code.
Frequently Asked Questions About AI Code Detector Accuracy
Which AI code detector has the lowest false positive rate on Java submissions?
In my test of 600 pre-LLM human Java files, Codequiry flagged 3.5% as likely AI, compared with 9.2% for GPTZero and 11.8% for Copyleaks. Codequiry also had the highest recall on known AI files at 94%.
Can AI code detectors tell the difference between ChatGPT code and human code?
Partially. They estimate probability, not fact. The strongest signals show in consistent variable naming, repetitive structure, and low burstiness in comment spacing. Human code varies more, especially across a student's body of work. No detector is perfectly reliable.
Should I use AI detection alongside a plagiarism checker?
Yes. AI-generated code is often original relative to peers and web sources, so a plagiarism checker alone misses it. A plagiarism checker still catches copied web code, peer copying, and refactoring-based copying that AI detection ignores. Use both signals in one review workflow, as Codequiry does, rather than treating them as separate tools.
If you are weighing an AI code detector for your own courses or team, start with a small labeled set of 50 to 100 files from your own institution before relying on vendor benchmarks. My Java results are specific to my assignments, my student population, and my fall 2024 tool versions. Your mileage will vary. But across 1,200 files, the direction was clear: Codequiry's AI code detector caught the most known AI cases and generated the fewest false positives on human code. That is the tradeoff that matters when a student's academic record is on the line.