Line 112 of a React take-home read // TODO: add error handling. Line 113 was a try/catch with a retry and an exponential backoff. The student had written 40 lines of JavaScript the week before and could not tell me what AbortController did when I asked him about it in office hours.
That was March 2024, and it's when I stopped trusting my gut on AI-generated code and started counting instead. I teach at a 14-week full-stack bootcamp and I maintain a small open-source CLI on the side, so most of my week is spent reading code someone else wrote and guessing where it came from. This post is about the guessing, and what happened when I replaced my gut with a spreadsheet and a detector that runs at the file level.
Six cohorts, 4,800 submissions, three scoring passes
Between January 2022 and November 2024 I kept every take-home submission from six cohorts. Two hundred students per cohort, four assignments each, 4,800 submissions of record. The assignments were a Python data-cleaning script, a React dashboard against a mock API, a Node service with tests, and a Java CLI (we switched to Java in cohort four because two of our hiring partners asked for it).
Each submission went through three passes. First, Codequiry's AI code detector, run per file instead of per submission, because a suspicious function inside an otherwise human repo is the common case and a submission-level score hides it. Second, a text-based detector on the README and any long comments. Third, two TAs blind-reviewing a 600-submission sample without seeing any scores, so I had something to compare the machines against.

We ended up with ground truth on 214 submissions. Some students admitted it in the post-course survey. Some left evidence in git. One left a chat.openai.com share link in a block comment at the top of server.js, which is the closest thing to a signed confession I've seen in ten years of teaching. One housekeeping note before the numbers: our grading harness posts to the API at submission time, and in February the webhook retries double-counted about 40 results. We deduped by submission hash by hand. If you build your own pipeline, store the hash before you store anything else.
A detector score is a smell, not a verdict. Two independent signals is a finding. One signal is a conversation.
Signal one: the comments stop sounding like the student
Comments are the cheapest thing to check and the thing nearly everyone skips. In our confirmed-AI files, comment and docstring lines averaged 18.4% of total lines. In confirmed-human files, 6.1%. That gap held across all four assignments and all three languages, and it widened after mid-2023.
The reason isn't that models comment more. It's that they comment differently. A bootcamp student in week 11 writes # todo: fix this before demo day or nothing at all. A model writes a full Args/Returns docstring for a six-line helper. Compare these two, both real, both from the same assignment:
def load_users(path: str) -> list[dict]:
"""Load user records from a JSON file.
Args:
path: Path to the JSON file.
Returns:
A list of user dictionaries.
"""
try:
with open(path, "r", encoding="utf-8") as handle:
data = json.load(handle)
return [user for user in data if user.get("active")]
except FileNotFoundError:
logger.error("User file not found: %s", path)
raise
except json.JSONDecodeError as exc:
logger.error("Malformed JSON in %s: %s", path, exc)
raise
def load_users(path):
# TODO ask dan about the encoding thing, broke on his laptop
f = open(path)
data = json.load(f)
f.close()
out = []
for u in data:
if u["active"]: out.append(u)
return out
The second one is worse code. The file handle leaks, the one-line if is a formatting crime, and the comment names a human being. It is also unmistakably a person, and the first one is unmistakably not. That's the whole signal, and you can read it in about four seconds once you know to look.
Signal two: formatting stops drifting
Human code accumulates damage. There's a function indented with six spaces because someone pasted it out of a Stack Overflow answer in 2021 and never ran a formatter. There's a stray double blank line, a mix of single and double quotes in the same file, a commented-out console.log someone meant to delete. None of that survives an LLM, because the model has effectively been trained on the prettier output of ten thousand style guides.
I measured this one crudely: within-file variance in indentation, quote style, and line length. Confirmed-human Node submissions had a mean of 3.2 distinct formatting irregularities per file. Confirmed-AI files had 0.4, and most of that was trailing whitespace from our own template. A perfectly formatted file is not proof of anything. A 300-line file with zero formatting drift from a student who has never once run black is worth a second look.
Signal three: the API surface doesn't match the syllabus
Models reach for the modern version of everything. Python match statements, httpx instead of requests, structuredClone instead of a JSON round-trip, Array.prototype.at() used with total confidence in a file where the student writes for (var i = 0; ...) two functions later.
const snapshot = structuredClone(state); // shipped in Chrome 98
const lastOrder = orders.at(-1); // shipped in 2022
I want to be careful here, because this signal alone is weak. Good students read release notes. We had a student in cohort five who genuinely used .at(-1) everywhere because she'd seen it on MDN that week, and her code was 100% hers. The signal isn't "new syntax showed up." It's a mismatch between the sophistication of the API choice and the sophistication of everything around it. A file that uses structuredClone and also has three nested callbacks and a global variable named temp2 is telling you something.
Signal four: everything is defended, including things nobody attacks
This is the one that catches the most submissions for us. LLM-written code defends against failure modes the assignment cannot produce. Every fetch has abort handling and a timeout. Every function has a try/except that logs and re-raises. Input validation on a function that only ever receives a hardcoded string. I once graded a 60-line script that read a local CSV and found exponential backoff with jitter, and the student could not explain what "jitter" was for.
The reverse is also true, and it's useful. Human novice code fails in boring, predictable ways: an unhandled None, an off-by-one, a fetch with no error path at all. When I see a file with zero unguarded edges in an assignment that is basically a minefield, I stop reading the code and start asking about the process.
Signal five: the tests were written by someone who never had the bug
Take-home tests are diagnostic artifacts. Human students write tests for the thing that broke on them at 1am, name it test_it_works_now, and leave the rest uncovered. LLM-generated suites cover the impossible path (test_load_users_with_invalid_permissions on a function that reads from ./data/users.json), use full-sentence test names, and pass on the first run.
First-run passes are the tell I trust most, because our assignment design makes them rare. We plant a deliberate gotcha in every take-home: a date field in local time, an ID that's a string in one endpoint and a number in another. Human submissions fail that test at least once. AI-assisted submissions often hit it perfectly, and about a third of those explain it in the README with a sentence that reads exactly like the explanation a model would give.
Git history supports the same read. One repo had fourteen commits, and the first one was 900 lines of finished, formatted, tested code at 11:47pm on a Sunday.
Where the scores fall apart
Now the honest part. Of the 486 confirmed-human submissions in our blind sample, 33 (6.8%) scored above 0.6 AI probability on at least one file. That's a false positive rate high enough to ruin someone's week if you act on a single score. When I required two files above 0.6 plus one corroborating signal from the list above, it dropped to 10 submissions (2.1%). That's not zero, and I don't expect it to be.
The failure modes are consistent. Files under 40 lines are pure noise; there isn't enough material for anything to be right. Java is the worst language for this by a mile, because half of a novice Java file is public static void main, getters, setters, and imports, and boilerplate looks generated no matter who generated it. Students who use Copilot inline and then edit heavily land in the 0.35 to 0.55 band, which is exactly where I have no idea what to do with the number. And students writing in their second or third language produce identifiers and comments that read stiff to an automated scorer, which is a real and unfair source of false positives.

None of this means the tools don't work. It means a score is a triage device. The thing I actually want from a detector is a ranked queue of files worth reading, and that's a much lower bar than proof.
Peer similarity catches something AI detection can't
The case that finally pushed us to run a peer check alongside the AI check: two students in cohort five submitted code that scored 0.38 and 0.41 on AI probability, both below any threshold I'd act on. Against each other, they matched at 97%. They had sat next to each other, typed the same prompt into the same model, and pasted the same output after renaming a few variables to match the assignment spec.
That's the argument for running peer similarity, web checks, and AI detection in one pass rather than three. AI detection answers "did a model write this." Peer similarity answers "is this the same code as someone else in the room," which catches the shared-prompt case that AI detection misses entirely. Web matching answers a third question, whether the code came off GitHub or a tutorial, which is a different conversation with the student and usually a much shorter one.

Most of the older tools only do one of these. MOSS gives you a percentage and an HTML page and nothing else, which is fine and has been fine since 1994, but it has no opinion about what an LLM looks like. If you're curious how the comparison shakes out, there's a longer writeup on how Codequiry compares to MOSS. The short version is that a single report with peer, web, and per-file AI scores is worth more than three reports I have to reconcile myself.
What we changed in the take-homes
Detection is downstream of assignment design. If your take-home is a task a model has seen ten thousand times, you will spend your semester doing forensics instead of teaching. We redesigned around one idea: make the task depend on something the model cannot know.
We ship a small internal SDK with real documentation and one deliberate quirk. billingsdk.charge() takes amount_cents as an integer, not dollars, and it throws a LedgerMismatch if you retry within 900ms. The SDK is ours. It is not in any training corpus. A student who writes the whole take-home by hand reads the docs and gets it. A student who prompts their way through it watches the model hallucinate a dollar-based amount parameter with total confidence, and then either reads the docs and fixes it or doesn't.
Three other changes, all cheap. Students submit a short NOTES.md explaining one design decision and one thing they'd do differently; it's easy to write and hard to fake convincingly. Every submission gets a five-minute oral defense with a TA asking about two specific functions, which is where the March 2024 student fell apart. And every submission goes through Codequiry at submit time from our grading script, so I see a ranked list before I open a single file. If you're setting this up for a course, the code plagiarism checker for teachers walkthrough covers the roster and submission side of it.

I don't love the oral defense. It costs TA hours we don't have, and it puts a shy second-language student at a disadvantage that has nothing to do with their code being original. We kept it because the alternative was trusting a number, and I trust the number less.
Frequently Asked Questions
Can you actually tell if a take-home was written by ChatGPT?
On files longer than about 100 lines, with two independent signals, usually yes. On a 40-line function, no, and anyone claiming otherwise is selling you something. I catch roughly 8 in 10 of the fully generated submissions I've confirmed by other means, and I let a fair number through.
What AI score should trigger a conversation with a student?
We use 0.6 or higher on two or more files in the same submission, plus one of the five signals above. One file above 0.9 is worth opening, not accusing. Anything below 0.5 I look at only if a peer match or a web match already put the submission on my list.
Does using GitHub Copilot count as cheating?
That's a policy question, not a detection one, and your syllabus needs to answer it before the semester starts. Our rule is that inline completion is fine if you can defend the line, and pasting a prompt's full output is not. We state that in writing on day one, which has done more to reduce incidents than any detector did.
Is there a way to make a take-home AI-resistant without surveillance?
Private APIs, deliberate quirks, and short oral defenses. Anything that makes the task depend on information outside the model's training data does more work than a detector ever will. You can run your first check on Codequiry to see the peer, web, and AI scores side by side on a sample cohort, which is the fastest way to understand what your current take-home is actually measuring.