The submission arrived at 2:14 a.m. Four commits, forty-seven minutes apart, all green on the provided test suite. The README had a table of contents. The code had docstrings on private helper functions.
I've been on both sides of this. I teach the backend module at a 14-week bootcamp and I maintain two open-source libraries that get a steady trickle of outside pull requests. When we started seeing take-homes that looked like this in late 2023, my first instinct was that I was pattern-matching on style rather than substance. So we stopped guessing and instrumented the thing.
What follows is our working audit. It is not a product pitch and it is not a magic threshold. It's a research note from a small program with an honest methodology section, because the single biggest mistake in take-home AI detection is treating a score as a verdict.
How we set up the audit
Between Fall 2023 and Fall 2024 we collected 412 submissions of the same take-home: a small REST service over a supplied CSV, roughly six hours of expected work, no framework restrictions. Three cohorts. Same rubric, same reviewers, same provided test suite.
Each submission went through four independent signals:
Peer similarity (Codequiry, run at the default 70% threshold across each cohort), web and GitHub source matching, a per-file AI generation score, and a small stylometric model two of us trained on 180 known-human and 140 known-AI samples from previous cohorts. Then, for 63 submissions that at least one signal flagged, we ran a twelve-minute live extension interview: share your screen, add one endpoint to your own code, talk me through the design decision behind the CSV parser.
Nineteen of those 63 candidates could not extend or explain their own submission. That's our ground truth proxy. It's an imperfect one, and I'll get to why.
The signals, ranked by what they actually predicted
Here's the table nobody puts in the marketing deck. "Interview failure" means the candidate could not add a straightforward endpoint or explain a function they had committed under their own name.
| Signal (threshold) | Flagged in the 63 | Interview failures caught | Predictive value |
|---|---|---|---|
| Peer similarity > 70% | 14 | 11 | 79% |
| Two of three signals agree | 22 | 14 | 64% |
| Stylometric model > 0.70 | 18 | 10 | 56% |
| Web match > 15% of tokens | 29 | 12 | 41% |
| AI generation score > 60 | 41 | 16 | 39% |
Read that bottom row carefully, because it's the one that should shape your policy. The AI detector flagged 41 of our 63, made 16 correct calls, and produced 25 false alarms. It also missed three candidates who fell apart at the interview. A 39% predictive value is not useless. It's a triage tool. If you're using it as an expulsion letter, you are going to hurt people.
Peer similarity did best, and the reason is boring: two candidates who cheated off each other produce near-identical code, and near-identical code is easy to detect. Two candidates who both asked Claude for a CSV parser produce code that differs in every identifier and every comment. Detecting shared origin is a solved-ish problem. Detecting non-human origin is a different problem, and it's much harder.
Web matching deserves a note. Our 41% number is misleading if you read it as "web matching is bad." It's measuring the wrong thing. Most of the 29 web-flagged submissions were candidates who had found a Stack Overflow answer for CSV edge cases and dropped it in with the original comments intact. That's a code provenance problem, and Codequiry's web view traced 22 of them to specific repository files or answer pages, line and token counts included. It just isn't an AI-authorship problem, and conflating the two is where a lot of academic integrity programs go sideways.
Where the AI detector quietly breaks down
We did one experiment that changed how I talk about these tools.
We took 40 submissions our AI detector had scored above 70 and re-ran them through a two-line prompt: "Simplify this code. Remove redundant comments and type hints." Then we re-scored. The average AI score dropped 23 points. Nothing about the code's origin had changed. Only the surface texture had.
An AI detector is mostly reading style, and style is the cheapest thing to change. That's not a reason to abandon detection. It's a reason to stop treating the score as an authorship fact.
The same effect shows up in the other direction. We noticed, anecdotally, that candidates whose first language wasn't English sometimes scored higher because they leaned on a model as a syntax and idiom helper rather than a solution generator. The code reads a bit generic and a bit over-documented, which is exactly what the detector is trained to notice. We don't have the sample size to quantify this, and I won't pretend otherwise. But it lines up with the Liang et al. 2023 finding that GPT detectors flagged 61% of TOEFL essays as machine-written, and it's the reason our escalation rule requires two agreeing signals rather than one.
The tells, in order of how often I actually see them
Before any tool runs, a manual read gets you surprisingly far. These are the patterns that showed up again and again in the 19 interview failures, and notably not in the 44 clean interviews.
The first is a comment that explains what the code does instead of why. Human developers under time pressure write comments like # the vendor sends dates as strings, don't trust this. Models write # Parse the CSV file into a list of dictionaries. The comment is redundant with the function name, and it's there because the model was trained to produce documentation, not insight.
The second is defensive scaffolding nobody asked for. Guards for empty input on a function the test suite already guarantees is non-empty. A custom exception class hierarchy for a six-hour exercise. Type hints on every parameter, including an Optional[int] that's never None.
The third is LeetCode shape. A solution that mirrors the canonical interview answer rather than the shape of your actual problem. Here's a real pair, same problem, one from a clean interview and one from a failed one:
# Clean interview submission
def top_k(rows, k):
counts = Counter(r["region"] for r in rows)
return [name for name, _ in counts.most_common(k)]
# Submission that could not be extended live
from typing import List
from collections import defaultdict
def top_k_frequent(items: List[str], k: int) -> List[str]:
"""
Return the k most frequent items in the given list.
Args:
items: A list of string items.
k: The number of most frequent items to return.
Returns:
A list containing the k most frequent items.
"""
if not items or k <= 0:
return []
counts = defaultdict(int)
for item in items:
counts[item] += 1
sorted_items = sorted(counts.items(), key=lambda pair: pair[1], reverse=True)
return [item for item, _ in sorted_items[:k]]
Look at what the second version did. It threw away the domain. The first one knows the rows have a "region" key. The second one is a generic function that would be correct in any codebase, which is precisely the problem: it was retrieved from a distribution of codebases, not from your assignment.
Those three tells are soft. They're the reason I open the file before I open the dashboard. They are also why I don't trust anyone, myself included, who claims they can spot AI code from a screenshot.
A six-point audit checklist for take-home code
This is the version we now run on every cohort. It's ordered by cost, cheapest first.
- Run the prompt-leak test before you ship the assignment. Paste your own assignment text into three current models and see if you get a working submission in one shot. If you do, your assignment is measuring prompt writing. Ours was for two of the three exercises we used in 2023. We rewrote both to require a decision the model can't make for you, like "choose the pagination strategy and justify it against the provided latency trace."
- Look at the commit graph, not the diff. Human work is lumpy. Four commits at forty-seven-minute intervals with no fixups and no dead code is a shape I have never produced in my life. Check whether the first commit is already complete and passing.
- Check the environment fingerprints. Lockfile present but doesn't match the course toolchain. Editor config absent. Dependencies pinned to versions released last month for a problem your cohort was given in September. A
.gitignorethat includes folders your course never generates. - Read the comments for voice. What versus why, as above. Count the ratio of docstring lines to logic lines. Anything above roughly 1:3 on a six-hour exercise is worth a second look, and it is not proof of anything.
- Run peer, web, and AI detection in one pass and escalate on agreement. This is the step where tooling choice matters, and it's where a single-purpose checker falls short. Codequiry's pitch is that it runs peer similarity, open web and GitHub matching, and AI generation scoring in the same check, and after a year of stitching three separate tools together I can tell you the integration is the part that saves you time. A two-of-three escalation rule is only practical if the three results land on one screen.
- Interview from their own code, for twelve minutes, screen shared. Not a quiz. "Add a delete endpoint." If they wrote it, this is trivial and takes eight minutes. If they didn't, it takes four minutes to find out and nobody has to accuse anyone of anything.
The order matters. Steps 1 and 2 cost you nothing and catch a surprising share of cases. Step 5 costs real money. Step 6 costs your calendar and is the only step that produces something you can put in front of a review board.
Drawing the line between assisted and generated
Here's where I've shifted most in eighteen months. The binary question, did a model touch this code, is unanswerable with confidence and increasingly meaningless as a policy. Every candidate I hired this year used Copilot or Cursor. I use them.
A workable line, and one you can write down:
- Disclosure is required and unpenalized. Tell us what you used and for what. Autocomplete is fine. Syntax lookup is fine. "Explain why my RSpec mock isn't stubbing" is fine.
- Generation is penalized. Asking a model to produce the solution, in whole or in part, and submitting it as your work, is the violation. Not the tool. The substitution.
- Comprehension is the test. If you can extend it live, debug it, and explain the tradeoff, the origin story stops mattering to me as an instructor.
Write that into the assignment page, not the honor code PDF nobody opens. We put it in a three-line box at the top of the README with the estimated time. Our disclosure rate went from near zero to 71% of submissions in one cohort. Nothing else we did had that effect.
Honest limitations
Our sample is 412 submissions from one program, one take-home, and three cohorts of a fairly homogeneous applicant pool. The ground truth is a twelve-minute interview, which produces false negatives: some people freeze under observation regardless of authorship. We probably undercounted. We also can't separate "used AI to generate" from "used AI to translate my thoughts into Python" in the interview data, because we never asked, and by the time we wanted to the cohort had graduated.
And we haven't tested this past a few hundred submissions. If you teach a 900-person intro course, your base rate is different and your escalation rule should be more conservative than ours. Detection precision collapses when the prior collapses.
What actually changed in our process
We redesigned the assignments, added the disclosure box, and stopped treating detector scores as findings. We also kept the tooling, because a triage layer that narrows 412 submissions to 63 is worth paying for, and because the alternative is doing peer comparison by hand. If you're evaluating options, the honest comparison is this: MOSS is free and has been the standard for two decades, but you submit by email and get HTML back, and it does peer comparison only. JPlag is self-hosted, well maintained, and excellent at what it does, which is also peer comparison only. Dolos is the nicest modern interface of the open-source group. Turnitin covers prose extremely well and code not at all. A code plagiarism checker that also scores AI generation and traces web sources in the same run is a different category of tool, and it's the category that matches the actual problem in 2025.
If you want to see what the three-signal view looks like on your own submissions, the AI code detector handles the generation side and the audit checklist above is yours to steal. We stole most of it from people who were doing this longer than we were.
Frequently Asked Questions
Can professors tell if a take-home assignment was written by ChatGPT?
Not from reading it alone, not reliably. In our 63-candidate interview set, the AI score predicted failure 39% of the time, which is much better than a coin flip and nowhere near a verdict. Style-based tells (over-documentation, generic naming, LeetCode shape) raise suspicion. They don't establish authorship.
What is the most reliable way to detect AI-generated code in a take-home?
Stack two independent signals and follow with a live extension interview. In our data, requiring two of three detection signals to agree lifted predictive value from 39% to 64%, and the interview caught the rest. No single score beat the combination.
How accurate are AI code detectors on source code?
Better than on prose in one specific way: code has structure, so token and AST comparison can catch shared lineage with high confidence. Worse in another: a two-line "simplify this" prompt dropped our average AI score by 23 points without changing anything real. Treat the number as a ranking, not a probability.
Is using ChatGPT on a take-home assignment cheating?
That's a policy question, not a technical one, and the answer has to be written down before the assignment goes out. Our rule is disclosure required, solution generation penalized, comprehension is the real test. Whatever line you draw, put it on the assignment page where students actually see it.
If you're running a cohort this term and want the peer, web, and AI signals in one report instead of three browser tabs, start with a Codequiry check and see what your current submissions look like.