Last March I ran a hiring loop for a senior backend role at a Series B company in Austin. Sixty-one candidates, 52 submissions, one open req. The take-home was a rate limiter in Python with a small test suite, roughly three hours of work for someone who knows what they're doing.
Twenty-nine of those 52 submissions carried type hints on every function signature and a Google-style docstring on every method. That alone doesn't disqualify anyone. It's the first thing I look at now, because the base rate moved. In 2019 I saw maybe one submission in ten built like that.
The reliable way to detect AI-generated code in take-home interviews is not a single detector score. It's three cheap signals stacked: an AI-generation score, a web and GitHub match, and a short live conversation about the code. Miss any one of them and you'll either wave through the wrong people or torch a good candidate's afternoon.
What AI-written take-home code actually looks like
Detectors score syntax. So do I, but with my eyes, and the patterns are consistent enough that I can usually call it before the scan finishes.
def calculate_average(numbers: list[float]) -> float:
"""Calculate the mean of a list of numbers.
Args:
numbers: A list of numeric values.
Returns:
The arithmetic mean of the input values.
Raises:
ValueError: If the input list is empty.
"""
if not numbers:
raise ValueError("The input list cannot be empty.")
total = 0.0
for number in numbers:
total += number
return total / len(numbers)
Look at the ratio. Twenty lines of function and twelve lines of documentation for something a second-year student writes as return sum(numbers) / len(numbers). The manual accumulation loop instead of sum() is another tell. LLMs love an explicit loop with a typed accumulator. Humans who know Python reach for the builtin.
Other markers show up in the same submissions: defensive validation for inputs that can't occur, custom exception classes nobody asked for, a README that explains what a rate limiter is before explaining the design decisions. Vibe code is over-explained and under-decided.
Why a single AI score is not a decision
The mechanics matter here, because they explain the failure modes. AI detectors measure token-level predictability, roughly the median per-token log-likelihood (perplexity) and the variance of that likelihood across a file (burstiness). Machine output is smooth: statistically flat, uniformly plausible, no spikes. Human code spikes constantly, because people write a clever line, get bored, then dump four lines of copy-pasted boilerplate.
That model punishes a specific kind of human. The candidate from a CS program that drilled JavaDoc discipline into everyone scores high. The candidate writing English as a second language, who comments more heavily to compensate, scores high. Last year one candidate scored 71 on AI and turned out to have pasted his own code from a personal repo, which is a different problem entirely.
We've now run this on roughly 214 submissions across three clients. That's enough to find the failure modes and not nearly enough to quote a false positive rate. Anyone who hands you an exact accuracy number is selling something, including the vendor.
So the score is a triage tool. It decides who gets read carefully. It doesn't decide who gets rejected.
A workflow to detect AI-generated code in take-home interviews
Here's the pipeline I've settled on, after two iterations that were worse. It runs on every submission automatically, and it produces a queue rather than a verdict.
Signal one is the AI generation score, computed per file rather than per submission, since candidates often paste one AI-written helper into otherwise human work. Per-file scoring is where Codequiry's AI code detector earns its place in this stack, because most tools give you one number for the whole repo and the average hides exactly the file you care about.
Signal two is a web and GitHub match. This catches the candidate who found a blog post from 2021 and retyped it, which reads as human to an AI detector and is still fraud in a take-home.
Signal three is peer similarity against the rest of the cohort, which is cheap to add and catches the far more common case of two candidates sharing answers in a group chat. MOSS has done this since 1994 and JPlag since the early 2000s. A code plagiarism checker that also does AI scoring saves you from running a second tool and a second export.
Signal three matters more than people expect. In the March loop, two submissions scored 94 percent similar to each other and within normal human range on AI. Neither candidate had any idea the other existed, or so they claimed. Turned out both had found the same YouTube tutorial.
Wiring the scan into the hiring pipeline
Manual uploads don't survive contact with a real hiring cycle. If a recruiter has to remember to export a zip and drop it into a dashboard, it will happen for the first six candidates and stop happening in week three. Automation or nothing.
Submissions land in the ATS. A webhook fires. The scanner pulls the archive, runs the three engines, and posts scores back to the candidate record. Codequiry exposes both a REST API and a CLI for exactly this, plus signed webhooks so you can verify the callback actually came from the scan. Set the callback to land on a staging endpoint first. You'll want to eyeball the JSON before it touches your ATS.
One mistake worth confessing: our first version had no idempotency key, so a webhook retry duplicated one candidate's row three times and he showed up in the shortlist at three different score tiers. Took a full day to figure out why the funnel counts didn't add up. Learned it the boring way.
# triage rule, run after the scan completes
queue = []
for row in scan_results:
risk = 0
if row["ai_score"] >= 80: risk += 2
if row["web_similarity"] >= 40: risk += 2
if row["peer_similarity"] >= 60: risk += 1
if risk >= 3:
queue.append(row)
queue.sort(key=lambda r: r["risk"], reverse=True)The weights aren't sacred. They reflect what I've seen: AI scores and web matches are strong evidence, peer similarity is weak on its own because two people genuinely can write the same 30-line solution to a FizzBuzz-class problem. Adjust the numbers to your own cohort, then write down why you picked them. You'll need that note the first time HR asks.
Picking a threshold you can defend to a candidate
Threshold selection is the part everyone skips, and it's the part that determines whether this process is defensible in a conversation with the person on the other side of the table. Here's where I've landed.
| AI score band | What we do |
|---|---|
| 85 and above | Read every file. Run the follow-up call. Score alone never rejects. |
| 60 to 84 | Second reviewer reads the code. Escalate only if a web match or peer match lands too. |
| Below 60 | Normal review. Peer and web signals still apply. |
Notice the 85-plus band still doesn't reject. That's deliberate. Any single automated score that gates employment decisions is a lawsuit with a UI, and if you're hiring in New York City, Local Law 144 covers automated employment decision tools. Whether a code scanner counts as one is a question for your lawyer, not your engineering manager. Ask before you ship the gate.
Cost matters too, because I've had to argue this budget line at three companies. Per-seat tooling for a hiring pipeline that runs 50 to 200 submissions a year is usually cheaper than one bad senior hire at fully loaded cost, which is somewhere between $150K and $250K depending on market. Codequiry pricing lands in the range where a hiring manager can approve it without a procurement fight, which sounds like a small thing until you're three weeks from closing a req.
The ten-minute call does more than the scanner
The scanner tells you where to look. A conversation tells you whether the code is theirs.
One candidate scored 74 on AI for a clean, well-documented implementation. Four out of ten code reviewers would have flagged him. I asked him to walk me through why he chose a sliding window over a token bucket. He started drawing on the whiteboard before I finished the question, argued himself into a different approach mid-explanation, and landed on the tradeoff faster than I would have. He'd used Copilot inline the whole way, understood every line, and shipped better code than he would have alone. We hired him. He's still there.
Contrast that with a candidate who scored 91. I asked why the implementation used a lock instead of a queue, a decision that mattered because both patterns were visible in the file. He said it was for thread safety. I asked which threads. Silence for eleven seconds. That one wasn't close.
Two questions do most of the work. "Walk me through this function line by line, including why you didn't do it the other way." And: "What did you get wrong the first time and how did you find it?" Anyone who wrote the code has a war story about it. Anyone who generated it has a summary.
Where this falls apart
Pair take-homes break every signal in the stack. If you let candidates use AI, telling the detector to flag it is incoherent, and you've built a policy that punishes the honest half of your applicant pool.
The harder truth is that banning AI in a take-home in 2025 may be the wrong call regardless of what your scanner says. The job changed. Junior engineers at every serious shop now write code alongside Copilot, Claude, and Cursor, and the skill you care about is directing those tools and knowing when the output is wrong. If your take-home forbids the tools, you're screening for a job that doesn't exist anymore.
Which is why I've mostly shifted toward open-book take-homes with a live review stage, and use scanning as an integrity check rather than a gate. The pipeline catches the handful of candidates who farmed the assignment out to a friend or a model and can't explain it afterward. That's the actual threat, and it's a small fraction of the pile. The rest is just people writing code the way code gets written now.
For teams running this at class scale rather than hiring scale, the mechanics are the same and the numbers change. A 400-student cohort needs the cohort-level sorting more than the individual case, which is what Codequiry solutions is built around. The triage rule above survives the translation with a higher weight on peer similarity.
Frequently Asked Questions
Can an AI detector tell the difference between Copilot-assisted code and fully generated code?
Not reliably, and anyone claiming otherwise is overselling. Inline Copilot use produces mixed signals because the human is still making structural decisions. The score goes up, but usually not to the 85-plus band, and the code holds up under questioning. That's the case where the follow-up call matters most.
How accurate is AI-generated code detection on take-home submissions?
It varies by language, task type, and how much the candidate edited the output. Short files with heavy documentation score highest and false-positive most. My practical advice is to tune your threshold on 50 to 100 submissions you've already reviewed by hand, then treat the score as a sorting key rather than a verdict.
Should I tell candidates they'll be scanned?
Yes. Put it in the take-home instructions in one sentence. Candidates who plan to cheat will cheat anyway, and the honest ones appreciate knowing the rules. It also keeps the process defensible if a rejection is ever questioned months later.
If you want to run this on your next hiring loop, start with a single cohort, hard-code the threshold, and review every flagged submission by hand before you trust the queue. The AI code detection report that falls out of a Codequiry AI scan gives you per-file scores and the indicators behind them, which is what you need for the phone call.