In the spring of 2024 a hiring manager at a 400-person fintech asked me to audit their take-home pipeline. Fourteen months of data, 1,180 backend submissions, six engineers doing review on top of their normal work. They had a suspicion and no evidence: candidates were pasting the prompt into ChatGPT, getting a working solution in ninety seconds, and submitting it as a forty-minute effort.
What they needed wasn't a verdict on any single candidate. It was a workflow that produced a defensible signal at scale, with a paper trail, and without burning six senior engineers on manual code archaeology. That's the problem this piece addresses, and it's the same problem a CS professor faces with a 300-student assignment.
Short version: AI code detection on take-home submissions is real, it works better than most skeptics claim, and it is far weaker as standalone evidence than most vendors imply. The useful configuration is a two-pass pipeline where detection scores triage, similarity checks corroborate, and a human makes the call.
What AI code detection actually measures
Text detectors like GPTZero (2023) and Turnitin's AI writing indicator (launched April 2023) lean on two statistical properties of generated prose. Perplexity measures how surprised a language model is by the next token; machine text tends to be low-perplexity because the model picks probable continuations. Burstiness measures variance in sentence length and structure; human writing swings between short and long, machine writing hums along at a steady register.
Both signals transfer poorly to source code. Code is already low-entropy by design. Identifiers come from a small shared vocabulary, control flow follows a handful of canonical shapes, and a for-loop written by a 12-year veteran looks structurally identical to a for-loop written by Claude. If you compute perplexity over a Java file, you get a number, but that number is mostly telling you that Java is predictable.
What works instead is a composite of structural and stylistic markers computed per file and then aggregated. Vendors differ on the exact feature set, but the categories are stable:
| Signal family | What it looks for | Strength |
|---|---|---|
| Lexical entropy | Identifier diversity, comment-to-code ratio, vocabulary spread | Moderate; degrades on short files |
| Structural idiom density | Use of stdlib helpers where a hand-rolled loop would be more natural | Strong on junior-level prompts |
| Documentation shape | Uniform block comments on every function, including trivial ones | Strong, easy to game |
| Error handling posture | Defensive validation for inputs that a human wouldn't bother guarding | Strong |
| Formatting consistency | Perfect adherence to a style guide with no drift across 200 lines | Moderate |
The documentation and error-handling markers are the two that surprise people. Here's the difference in practice. A human writing a mean function under time pressure produces something like this:
def mean(xs):
return sum(xs) / len(xs)
An LLM asked for the same thing, with no constraint on verbosity, tends to produce this:
def calculate_average(numbers: list[float]) -> float:
"""
Calculate the average of a list of numbers.
Args:
numbers (list[float]): A list of numeric values.
Returns:
float: The arithmetic mean of the input values.
Raises:
ValueError: If the input list is empty.
"""
if not isinstance(numbers, list):
raise TypeError("Input must be a list")
if len(numbers) == 0:
raise ValueError("Cannot calculate average of an empty list")
total = 0
for num in numbers:
if not isinstance(num, (int, float)):
raise TypeError("All elements must be numeric")
total += num
return total / len(numbers)
Note what's happening. The second version checks types that a Python function doesn't need to check, manually accumulates a sum it could delegate to sum(), and documents all of it in a docstring style lifted straight from a Google developer-education page. None of those choices is wrong. Together they are a fingerprint.
A single one of these in a 40-line submission means nothing. That's why per-file scoring is misleading and per-submission aggregation matters. Most of the tools I've worked with will report a per-file probability and then a submission-level rollup, and it's the rollup you should read first.
The base rate problem in a 200-submission pipeline
This is the part that gets skipped in vendor decks, and it's the part that determines whether your process survives an appeal.
Suppose your detector has 90% sensitivity and 90% specificity. Both numbers sound strong. Now run it against 200 submissions where you believe, from prior evidence, that 10% are fully machine-generated.
| True AI rate | Flagged (of 200) | Actually AI | Precision |
|---|---|---|---|
| 10% | 26 | 18 | 0.50 |
| 20% | 36 | 36 | 0.69 |
| 40% | 52 | 72 | 0.86 |
At a 10% base rate, half the flags are wrong. This isn't a knock on any particular tool; it's arithmetic, and it applies with equal force to a plagiarism detector, a medical screening test, and a spam filter. The practical consequence is that a flag is a reason to look, not a reason to reject, and your written policy needs to say so before the first candidate asks.
The good news is that you can move the base rate. If you require a short design write-up alongside the code, or an annotated diff explaining three decisions, or a fifteen-minute walkthrough of the submitted solution, the population of submissions that survive all four filters is much smaller and much more suspicious. The base rate in that residual population is closer to 60% than 10%, and the detector suddenly looks like a precision instrument.
Stacking detection with similarity and web checks
An AI score on its own is an opinion about style. It gets much stronger when it agrees with something independent.
Peer similarity catches the candidate who got the solution from a friend or a public repo. Web and GitHub matching catches the one who adapted a Stack Overflow answer wholesale. AI generation scores catch the one who typed the prompt themselves and never shared the result with anyone. These are three different failure modes with three different signatures, and a submission that trips two of them is not twice as suspicious, it's a different category of evidence.
If you're running the checks manually rather than through an API, you can stack a AI code detector alongside a conventional code plagiarism checker and reconcile the outputs by hand. At any real volume that reconciliation is the bottleneck, which is why I usually push teams toward a single platform that reports peer, web, and AI scores on the same submission record. That's the workflow Codequiry was built around, and it's the one place where I think the tooling category has genuinely improved since the MOSS era. MOSS is still fine for intra-class similarity, and I'll defend it for that, but it has no opinion about whether a file was machine-authored and no interface beyond a set of HTML result pages.
Building a two-pass review workflow
Here's the process I recommended to the fintech, and the one I'd recommend to a course coordinator with a 300-student assignment. It has two passes because the economics of human attention demand it.
Pass one, automated. Run every submission through similarity, web, and AI detection on ingest. Persist the raw scores, not just the verdict. You want to be able to say "this file scored 0.87 on March 3rd with engine version X" eighteen months later, because that's what due process looks like. Set the routing threshold low, around 0.5 for the AI score, because pass two is cheap and false negatives are worse than false positives here.
Pass two, human. An engineer or TA opens only the routed submissions and answers three questions in writing. Does the code solve the problem in a way that matches the candidate's or student's stated approach? Are there structural fingerprints that a competent human wouldn't produce under time pressure? Does the submission look like the same person wrote the rest of it?
That third question is the one people forget. If a candidate submitted three take-home problems over two weeks and two of them are terse and idiomatic while the third is uniformly documented with Google-style docstrings and defensive type checks, the inconsistency is the evidence. Style, like most things, is more informative as a within-person comparison than as an absolute measurement.
The most reliable signal in a take-home pipeline isn't the detection score. It's the score disagreeing with the candidate's own earlier work.
Where the scores fall apart
Short submissions. Below roughly 80 to 100 lines, the per-file statistical features don't have enough tokens to stabilize, and most engines will fall back to a wide confidence interval or a neutral score. A LeetCode-style 30-line function is essentially undetectable this way, which matters because a lot of take-homes are exactly that shape.
Boilerplate-heavy tasks. If the prompt asks for a Spring Boot controller or a React component, the human answer and the machine answer converge, because both are following the same framework conventions. Spring Boot with its annotation-driven setup produces near-identical source whether a person or a model wrote it. Framework-shaped code is a bad substrate for this kind of analysis.
Non-English comments and identifiers. Detection quality drops for submissions written with non-Latin comments, partly because the training distribution skews English and partly because the tokenizer handles it differently. I don't have a clean number for the size of the drop, and I'd be skeptical of anyone who quotes one without publishing their test set.
And a config detail that bit us: the minimum token threshold, which in several engines defaults to something around 25 tokens per file, silently skips tiny files with no warning in the output. Our first pass on the fintech data reported clean results for 41 submissions that had never actually been analyzed. We caught it because the file counts in the report didn't sum to the submission counts. Check that arithmetic before you trust a clean run.
A note on policy, not tooling
Whatever you deploy, the written policy matters more than the engine. State explicitly whether AI assistance is permitted, and if it is, whether it's permitted for scaffolding, autocomplete, or full solution generation. Copilot has been generally available since June 2022; a candidate in 2025 who used it for a docstring is not the same case as one who asked Claude to write the whole thing. If your policy doesn't distinguish those, your reviewers will invent their own distinction, and it will vary by reviewer.
Then give people a path to explain themselves. A fifteen-minute call resolves most ambiguous flags, and the ones it doesn't resolve are exactly the ones you want on record.
Frequently Asked Questions
Can AI code detectors tell ChatGPT code from Copilot code from human code?
Not reliably at the source level. ChatGPT, Claude, Gemini, and Copilot share training data and stylistic priors, so a per-model attribution is closer to a guess than a measurement. What the tools detect is machine authorship as a category, and even that should be read as a probability band rather than a point estimate.
What's a reasonable false positive rate to accept on take-home submissions?
Accept a high false positive rate at the triage stage and a low one at the decision stage. Routing 25% of submissions to a human review that takes eight minutes each is a good trade. Rejecting a candidate on a score alone is not, and at typical base rates the error rate on that decision is somewhere between 15% and 50%.
Does requiring a live coding session make detection unnecessary?
No, but it changes what detection is for. If you already run a live session, use detection as a consistency check: a candidate whose live code looks nothing like their take-home is a signal worth interrogating. The take-home becomes the artifact and the live session becomes the stylistic control.
If you want to see how the three score families look on a single submission record, the AI code detector page has a live walkthrough of the per-file and submission-level views.