A Hiring Manager's Audit of AI-Generated Code in Take-Home Tests

An audit of 1,284 take-home coding assignments from 23 companies between July 2024 and March 2025 found that 31.2% were flagged as likely AI-generated at high confidence. Senior engineers manually reviewing those flags confirmed 279 as fully or substantially LLM-written, with a false positive rate of 4.5% among high-confidence flags. The screening used Codequiry's AI code detector paired with its source similarity check.

This is not a theoretical exercise. A 400-person fintech asked the question every engineering manager eventually faces: when a remote candidate submits a clean, well-commented solution in 40 minutes, did they actually write it? The following report covers the audit methodology, the signals that separated AI-generated submissions from human ones, the false positives we encountered, and what we changed in the hiring loop afterward.

How Often Do Candidates Use AI on Take-Home Coding Tests?

Across 1,284 submissions collected from 47 hiring pipelines, we ran every solution through Codequiry's AI-written code detector at a high-confidence threshold of 0.90 on a 0–1 scale. The dataset broke down by language as follows:

• Python — 57%
• JavaScript/TypeScript — 26%
• Java — 11%
• Go — 4%
• Rust — 2%

At that threshold, 401 submissions (31.2%) were flagged as high-confidence AI-generated. We manually reviewed all 401 flags with two senior engineers who had no access to the AI confidence scores. The adjudication process was simple: each reviewer independently labeled the code as fully AI-generated, substantially AI-assisted, ambiguous, or human. Where reviewers disagreed, a third engineer broke the tie.

The final classification of the 401 high-confidence flags:

• Fully AI-generated: 214 (53.4%)
• Substantially AI-assisted: 65 (16.2%)
• Ambiguous: 104 (25.9%)
• Human false positives: 18 (4.5%)

Put another way, 69.6% of high-confidence flags were confirmed as AI-generated or AI-assisted. The false positive rate among high-confidence flags was 4.5%, or 1.4% of all submissions. That is a meaningful number. If a hiring team automatically rejects every high-confidence flag, they will turn away roughly one qualified candidate for every twenty-two flags.

Signals That Separate LLM-Written Code from Human Code

We logged 14 candidate signals for each flagged submission. The table below shows the prevalence of the six most discriminative signals among the 279 confirmed AI-generated/AI-assisted submissions compared with a random sample of 220 unflagged human submissions from the same hiring pipelines.

The pattern is not that AI-generated code is too simple or too complex. It is that LLM output tends to be uniformly clean. Human code is messier in ways that reflect a real development process: abandoned half-fixes, inconsistent formatting, a skipped edge case that was noticed later.

The candidates who used AI weren't the ones who failed the assignment. They were the ones whose clean, perfectly commented solutions took 25 minutes instead of four hours.

A Representative Flagged Submission

Here is a minimal Python function from a flagged submission. The candidate claimed it took 90 minutes to complete. It passed all tests and included this docstring:

def process_payment(transaction_id: str, amount_cents: int, currency: str) -> dict:
    """
    Process a payment transaction and return the normalized result.

    Args:
        transaction_id: The unique identifier for the transaction.
        amount_cents: The payment amount in cents.
        currency: The ISO 4217 currency code.

    Returns:
        dict: A dictionary containing the normalized payment result.
    """
    if amount_cents <= 0:
        raise ValueError("amount_cents must be positive")
    if currency not in SUPPORTED_CURRENCIES:
        raise ValueError(f"Unsupported currency: {currency}")
    normalized_amount = round(amount_cents / 100, 2)
    result = {
        "transaction_id": transaction_id,
        "amount": normalized_amount,
        "currency": currency.upper(),
        "status": "pending",
    }
    return result

On its own, this function is unremarkable. The AI detector did not flag it because of the docstring alone. It flagged it because of a combination of signals: the same docstring pattern repeated across all 11 functions in the submission, the consistent use of `f"..."` interpolation only where a human would have mixed styles, the absence of any debugging artifact, and the fact that the candidate's variable naming had no drift across 350 lines. The same candidate's JavaScript functions used the identical comment structure and argument validation order.

By contrast, a confirmed human submission that solved the same take-home problem looked like this:

def pay(tx, amt, cur):
    # TODO: handle refunds later
    if amt <= 0:
        return None
    if cur not in ['USD', 'EUR']:
        print("bad currency", cur)
        return None
    norm = amt / 100  # cents to dollars
    return {'tx': tx, 'amt': norm, 'cur': cur}

The human version uses a shorter function name, a TODO comment that remains in the final submission, a quick `print` debugging line, and no type hints. This is not better code. It is just human. The statistical signature of the two blocks is very different, and a detector trained on millions of human and LLM-generated code samples can learn that difference.

Reducing False Positives Without Missing Real Cases

The initial 0.90 threshold was chosen to cast a wide net. But a 4.5% false positive rate among high-confidence flags is too high if the hiring team treats the score as a binary pass/fail. We tested three alternative rules on the same 1,284 submissions:

The last row deserves explanation. When we combined the AI detection score with Codequiry's code plagiarism checker, the source similarity check caught 10 additional submissions that were not high-confidence AI but were directly copied from public GitHub repositories or Stack Overflow answers. The combination also raised overall recall because some candidates pasted AI-generated code from online sources rather than generating it themselves.

The practical recommendation from the audit was to treat the AI score as a triage signal, not a verdict. The hiring team now uses a score of 0.90 or higher to require a live code walkthrough. For scores above 0.95 with at least two discriminative signals, the candidate is asked a second, proctored coding task. Nobody is rejected solely on the detector's output.

Why Traditional Plagiarism Checkers Miss the Biggest Share of AI Code

MOSS, JPlag, and Dolos are built for a different problem. They compare submissions against each other or against a corpus of known online code. A candidate who uses GitHub Copilot or ChatGPT to generate a unique solution from a prompt may produce code that matches no existing submission exactly. The similarity score stays low. But the code is still not the candidate's own work.

The table below summarizes the separation of concerns:

This matters for hiring pipelines because take-home tests have become the easiest place for AI-assisted cheating to slip through. Unlike a live coding exercise, a take-home test gives the candidate time to prompt, regenerate, refactor, and clean up the output. A similarity checker alone cannot see that process. A statistically trained AI detector can flag it, but only if the hiring team is willing to use the score as a reason to dig deeper rather than as a final judgment.

What We Changed