Between January and November of 2024, a 60-person engineering org I consult for reviewed 486 take-home submissions for 14 backend openings. Nine of the first forty we read in February had the same shape.
Not the same text. Different variable names, different formatting, two different languages in the pile. The shape: a small main() wiring three functions together, a config object nobody requested, docstrings on every function including the two-line ones, and defensive error handling for conditions the spec could not produce. My favorite submission validated a port number against a range and then, on the next line, validated the same port number against the same range using a different helper.
That is where screening take-home code stopped being a reading exercise and became a measurement problem.
What we were actually checking before 2023
Before 2023 a single senior engineer read every submission for 30 to 40 minutes and scored it on three things: does it run, is it readable, did the candidate write tests. Originality was not on the rubric. It didn't need to be. The cost of producing a plausible backend take-home in Go or Node was high enough that most people just did the work.
Then the cost of a plausible first draft fell to roughly four minutes and a paragraph of context. I don't think candidates got more dishonest in 2023. The price of looking competent dropped, and people respond to prices.
How we ended up screening take-home code in three lanes
The first thing we built was wrong, and I want to be specific about why. We bought an AI detector, ran everything through it, and set a threshold. The first candidate it flagged at 88% was a career switcher from a data analytics background who had spent two years maintaining a PHP codebase with a brutally enforced style guide. She wrote unusually regular code because that's what her last job required. She was also, after a live interview, one of the better hires that cycle.
A single AI score can only tell you the code looks machine-regular. It cannot tell you where the code came from.
So we split the question into three and ran them in a fixed order. Peer similarity first, web and source matching second, AI scoring third. Peer and web matching produce evidence: matching files, matching repos, matching token counts. AI scoring produces a probability. Evidence settles arguments. Probabilities start them.
| Screen | Flagged (of 486) | Confirmed after review | Confirmation rate |
|---|---|---|---|
| Peer similarity | 71 | 58 | 82% |
| Web and source matching | 39 | 34 | 87% |
| AI score 70 or above | 96 | 44 | 46% |
Lane 1: the candidates check each other
Every submission gets compared against every other submission for the same role in the same window. With 14 openings, that meant comparison groups of 20 to 60, not one pool of 486. Small detail, big difference in noise.
71 submissions crossed our similarity threshold of 0.65 normalized. 58 held up when a human looked. Of the 58, thirty-one were pairs from the same bootcamp cohort. We knew about maybe a third of those relationships going in. When you recruit from a 14-week program with a shared Slack channel, solutions travel.
The pair that taught us the most: one candidate renamed every variable, reordered the imports, and converted the for loop in their retry helper into a while loop. Our line-based diff reported 34% similarity. Token and AST comparison reported 91%. That gap is the entire argument against diffing files when you want to detect code plagiarism.
def retry_call(fn, attempts=3, backoff=0.5):
"""Retry fn up to attempts times with exponential backoff."""
last_exc = None
for i in range(attempts):
try:
return fn()
except Exception as exc:
last_exc = exc
time.sleep(backoff * (2 ** i))
raise last_exc
That function appeared in eleven submissions across two hiring windows under four different names for the function itself and three different names for the exception variable. Line-based tools saw variation. Structural tools saw the same tree.

Lane 2: matching against GitHub and leaked prompts
This lane was cheaper to run than I expected and more productive than I expected. 39 submissions matched something public. 34 of those matches were substantive enough to matter.
22 of the 34 traced back to a single GitHub repository. Somebody had posted our take-home prompt, verbatim, with a working solution, in mid-2023. Candidates were finding it by pasting the prompt text into a search engine. The prompt itself had been in use since 2021 and had been sent to roughly 900 people in that time. One of them posted it. That's the base rate.
We now rotate prompts every two quarters. I want to be honest that rotation is not a fix. Your retired prompt lives on the internet permanently, and a candidate who searches for a company name plus a job title will find something.

Lane 3: AI-generation scoring, useful but noisy
96 submissions scored 70 or above. We confirmed 44 as fully generated after live follow-up. 46% confirmation, against 82% and 87% in the other lanes. Worth understanding why.
The score collapses two very different behaviors into one number. Writing a function body with Copilot autocompleting after your signature is not the same act as pasting a spec into a chat window and submitting the output. Both raise the score. Only one of them tells you the candidate can't do the job.
We also poisoned our own data. Our starter repo, last regenerated in 2023, was itself written by a model. About two-thirds of candidates kept the starter files, which we explicitly allowed, and every one of them carried a baseline AI score in the 40 to 55 range before writing a line of their own. It took two hiring cycles before someone thought to exclude known starter files from the score. Do that exclusion and the cohort mean drops by roughly half.
This is the part where a well-built AI code detector earns its keep and where it also has to be used carefully. Per-file scores, not just a submission score, are what made the tool usable for us. A candidate with one 90-scoring file and seven 20-scoring files is a different story from a candidate with a uniform 75 across the repo. Codequiry's per-file breakdown with written indicators let our reviewers make that distinction instead of arguing about a single number.

Where we drew the line on AI assistance
We do not reject on an AI score. Full stop. Not at 70, not at 90. The score decides where to spend interview time, and the interview makes the call.
The live extension is the whole mechanism. Candidate shares their screen, opens their own repo in their own editor, and we ask for one adjacent change. Something small. "Make retry_call accept a jitter parameter and prove it with a test." If they wrote the code, this takes six minutes and is pleasant. If they didn't, it also takes six minutes and it is extremely obvious.
In 2024 we ran 96 live extensions and made 11 rejections on the basis of them. Nine of the eleven had an AI score above 70. So did thirty-eight people we hired. That single comparison is why we stack signals instead of thresholding any one of them.
The rule we landed on: no rejection on one signal. A 91% peer match plus an inability to explain the code gets a rejection. A 0.88 AI score plus a clean live extension gets an offer.
The hours and the dollars
Before the change: 486 submissions at 38 minutes of senior engineer review each, about 308 hours. After: 486 at 12 minutes of triage plus 96 live extensions at 45 minutes, about 169 hours. We got back 139 hours, or roughly $16,000 at a $115 loaded rate, across the hiring year. Tooling came in under $4,000.
The math isn't close, and it gets less close once you count the cost of one bad hire in a 60-person team. I'd rather not put a number on that because the number people cite is always pulled from a vendor deck. But if you've made one, you know it's several multiples of the entire screening budget.
Why we consolidated to one tool
We started with three: an open-source pairwise comparator, an AI detector, and a manual GitHub search that a recruiter did by hand with copy-paste. Three upload paths, three sets of IDs, no way to line up results for the same candidate. The ATS integration was a spreadsheet.
Codequiry runs the three lanes in one check, keeps a consistent submission ID across them, and its token plus AST matching survived the renaming that beat our old diff. The REST API meant the ATS could fire a check the moment a submission landed, which took a manual step out of a recruiter's week. For a fair comparison: MOSS is free and it works, and it gives you an HTML results page with no web lane, no AI lane, and a workflow that involves mailing files around. JPlag is solid for Java and less so for the mixed language set we hire for. Dolos is well-built and actively maintained, and you are still the one hosting it and building the review layer on top. All three are reasonable if you have an engineer who enjoys that work. We didn't want to be that engineer.
For a course rather than a hiring pipeline, the shape of the problem is identical. A professor grading 200 submissions for CS 201 has the same three questions about the same three hundred files, except the consequence of a false positive is a grade appeal instead of a rescinded offer, and the timeline is a weekend instead of a quarter. The code plagiarism checker for teachers side of the product handles the same three lanes with a student portal instead of an ATS.
What we kept and what we dropped
Dropped: the AI score threshold as a gate, the line-based diff, the three-vendor glue, and the assumption that rotating prompts solves the leaked-prompt problem.
Kept: the fixed lane order, the starter-file exclusion, the live extension for anyone flagged in any lane, and the two-signal rule for rejections. We also kept one thing that isn't a tool at all: a written policy the recruiter can send a candidate on request. If you're going to screen people's code, they deserve to know what you're screening for. That email has cost us three candidates and saved us at least one lawyer conversation.
The February batch that started all this? We ended up hiring two of those nine. They'd paired on the assignment because they were friends from the same cohort, and we told them the take-home was collaborative, which it wasn't. That one was on us.
If you're running a pipeline and want to see what the three-lane view looks like on your own submissions, the code plagiarism checker page has the details and a demo cohort you can run first.