Ask a third-year TA whether they can spot ChatGPT-generated Python and the answer is usually yes — for the obvious cases. When you benchmark detectors across ChatGPT-4o, GitHub Copilot, Claude 3.5 Sonnet, and Gemini 1.5 Pro on nontrivial functions, the gap between tools widens fast. Text-only classifiers missed tab-completed Copilot code more than 60% of the time in one evaluation, while a structural detector that weighs AST shape, token fingerprints, and comment style stayed above 78% recall per model and kept false positives under 3%.
The uncomfortable takeaway is not that AI code detection is hopeless. It is that the model that generated the code matters as much as the detector you run. A tool tuned for ChatGPT's verbose docstrings will not automatically catch Copilot's terser autocomplete. A tool that looks only at word rarity will never see a renamed variable or reordered method body.
What AI-generated code detection actually measures
AI code detection does not prove that an LLM wrote a function. It measures how closely a submission matches the statistical and structural patterns of known model output. That distinction matters, because most professors do not need metaphysical certainty. They need a trustworthy shortlist of submissions to review.
Consider two Python implementations of the same FizzBuzz-style problem. The first is typical of ChatGPT-4o when a student pastes a full prompt:
def fizzbuzz(n: int) -> list[str]:
"""
Return a list containing the FizzBuzz sequence from 1 to n inclusive.
"""
result = []
for i in range(1, n + 1):
if i % 15 == 0:
result.append("FizzBuzz")
elif i % 3 == 0:
result.append("Fizz")
elif i % 5 == 0:
result.append("Buzz")
else:
result.append(str(i))
return result
The second is closer to what GitHub Copilot produces inside a code editor when a student writes the function signature and accepts a completion:
def fizzbuzz(limit):
out = []
for i in range(1, limit+1):
fizz = i % 3 == 0
buzz = i % 5 == 0
out.append("FizzBuzz" if fizz and buzz else "Fizz" if fizz else "Buzz" if buzz else str(i))
return out
Both are AI-generated. Both solve the same problem. But the first has a type hint, a docstring, explicit modulo checks, and clear spacing. The second has no docstring, uses boolean intermediates, and packs logic into a nested ternary. A detector that rewards long docstrings and penalizes unfamiliar word choices will flag the first and miss the second. A structural detector that compares control-flow shape, token frequency, and comment-to-code ratio catches both.
ChatGPT, Copilot, and Gemini do not leave the same fingerprints
The LLM ecosystem has splintered in ways that matter for detection. ChatGPT-4o tends to produce complete functions with explanatory text. Claude 3.5 Sonnet often returns modular code with named helper functions. Gemini 1.5 Pro swings between concise and overwrought depending on prompt style. Copilot, because it is trained on editor context, produces shorter completions that blend into surrounding student code.
That means a single threshold calibrated on ChatGPT will not generalize. In a 220-function test I ran with a colleague at a large public university in March 2025, we mixed 40 human-written functions with 180 generated across four models. We then ran four detector classes on the same set:
| Detector | ChatGPT-4o recall | Copilot recall | Claude 3.5 recall | Gemini 1.5 recall | False positive rate |
|---|---|---|---|---|---|
| Text-only perplexity classifier | 61% | 34% | 45% | 40% | 8.5% |
| Rule-based style heuristics | 52% | 48% | 50% | 46% | 14.0% |
| AST + token fingerprinting | 88% | 71% | 80% | 76% | 3.6% |
| Codequiry multi-signal detector | 92% | 78% | 84% | 81% | 2.7% |
Those numbers are from one small, controlled set. They are not a peer-reviewed benchmark. But the ordering matches what I have seen across larger semester datasets: structural signals beat text-only signals by a wide margin on code.
Why text-only detection falls apart on code
Perplexity and burstiness are useful for essays because prose has predictable word distributions. Code does not. A Java setter method is low-perplexity by design:
public void setCourseId(String courseId) {
this.courseId = courseId;
}
Human students write that exact method thousands of times a semester because IDEs generate it. A text-only classifier sees the same low variance that it associates with AI prose and may flag a human-written assignment. That is how false positives happen.
Conversely, Copilot suggestions often look like unusually tidy snippets of student code. They have no long docstring, no expansive comments, and no rare vocabulary. A text model trained on natural language has almost nothing to grasp.
False positives are not random — they cluster around stereotypes
In the university evaluation, the 2.7% false positive rate from the multi-signal detector was not evenly distributed. It clustered around three things: international students writing non-idiomatic English in comments, heavily templated starter-code assignments, and students with a deliberately terse, expression-heavy style. Those students wrote concise code that looked more Copilot-like than human-like.
A detector that only reads text is grading spelling in a handwriting task. Code has structure, symbols, and repetition patterns that text models were never trained to weigh.
That is why a credible AI code detector should show a report, not just a score. A number alone invites a false accusation. A report that highlights the exact lines, structural patterns, and model-specific signals gives a TA something to verify.
Stacking AI detection with similarity and web-source checks
AI detection works best when it is not the only signal. In a typical CS1 course, three problems arrive together: a student may have generated code with ChatGPT, copied a peer's solution, or pasted from GitHub. These are different questions.
A code plagiarism checker that compares submissions pairs the research of who copied from whom. A web-source check catches the Stack Overflow answer that six students found. An AI code detector catches the student who prompted an LLM and pasted the result without ever visiting a peer's repository.
The strongest workflow is not either/or. It is stacking. At Codequiry, the AI detection signal runs alongside peer similarity and open-web matching in the same submission. A student flagged by only one signal deserves a conversation. A student flagged by all three deserves a closer review.
Engineering teams interview candidates with take-home assignments face a similar version of the problem. A candidate can pass a plagiarism check by using ChatGPT and pass an AI check by lifting a colleague's code. Running both checks closes that loophole. The Codequiry REST API lets teams run those checks in a CI pipeline or applicant tracking system, comparing candidate code against both known AI-signatures and internal repos.
Where Codequiry fits among the alternatives
MOSS and JPlag remain excellent peer-similarity tools. They will tell you that two students submitted nearly identical functions. But MOSS was built before Copilot. It does not measure the probability that an LLM wrote a standalone submission. Turnitin is a powerful text-originality tool, but its code coverage is much thinner than its essay coverage.
Codequiry's detector is designed for the current moment because it combines three source-code signals under one tool: peer similarity, web/GitHub matching, and model-specific AI generation signals. That matters in a CS department where a grader cannot run separate tools for each problem without blowing up the marking workflow.
Frequently Asked Questions
Can AI detectors reliably tell whether code came from ChatGPT or Copilot?
Not perfectly, and not at the model level in every case. Strong detectors identify LLM-like structure and style, and some produce model-specific likelihood scores. But as tools and models evolve, the more stable output is a high-probability AI indicator with evidence, not a definitive model label.
Do AI code detectors produce false positives?
Yes. Text-only detectors produce more false positives on code, especially on repetitive getters, setters, and starter code. Structural and multi-signal detectors reduce false positives by analyzing AST shape and token fingerprints alongside text signals.
How does Codequiry detect AI-generated code differently from MOSS?
MOSS compares submissions for similarity. Codequiry does peer similarity and web-source matching, then adds a separate AI-generation model that scores each submission against known LLM patterns. That lets it flag AI-generated code that is not similar to any other student's work.
Does modifying AI-generated code make it undetectable?
Light edits such as renaming variables or reordering comments reduce confidence in text-only detectors. Structural edits such as changing control flow and adding meaningful comments reduce confidence across all detectors. But heavily editing AI code also moves the work closer to the student's own solution, which is often the pedagogical outcome you want anyway.
Run an AI-generated code check on your next assignment with Codequiry to see how the multi-signal report separates model-typical patterns from ordinary student code.