A pull request arrives with 62 lines of new TypeScript, a tidy changelog entry, and a useDebounce hook that handles every edge case in the ticket. Nothing is misspelled. Nothing is commented out. It merges in eleven minutes. That's the shape of the problem: the detector has to make a call from a diff of a few hundred tokens, and most of the statistical machinery behind AI detection was built for essays, not for code.
Detecting AI-generated code in pull requests is a different job from detecting it in a term paper. Essays are long, unconstrained, and full of stylistic fingerprints. Diffs are short, constrained by a compiler, and full of text that a model produces exactly the way a template does. Here's what the score is computed from, where it goes quiet, and how we wired it into a merge gate without turning every PR into a courtroom.
What the score is actually computed from
Two families of detectors show up in vendor documentation, and they measure different things.
The first is reference-model scoring. You push the code through a code-trained language model and average the negative log-likelihood of every token. That average is a per-token perplexity. Humans write code the model finds surprising: an oddly shaped regex, a workaround for a floating point comparison, three lines that exist because a downstream system sends a string where it should send an integer. LLM output reads as probable because it is probable. It came out of a model that has seen ten thousand calculateTotal methods.
The second is feature classification. No language model, just a classifier over things you can count: comment-to-code ratio, identifier entropy, distribution of AST node types, docstring uniformity, defensive null checks on every parameter. Cheaper to run, easier to explain in a review comment, and it fails in different places.
/**
* Calculates the total price of the given list of items.
*
* @param items the list of items
* @return the total price
* @throws IllegalArgumentException if items is null
*/
public BigDecimal calculateTotal(List<Item> items) {
if (items == null) {
throw new IllegalArgumentException("Items list cannot be null");
}
return items.stream()
.map(Item::getPrice)
.reduce(BigDecimal.ZERO, BigDecimal::add);
}
Read that as a reviewer. The Javadoc restates the signature. The null check guards a parameter nobody in the codebase passes as null. The stream pipeline is the version that appears in tutorials, with BigDecimal::add as a method reference rather than the lambda a person writes at 4pm on a Thursday. Any one of those is nothing. All four together, in a file where the neighboring methods carry no Javadoc at all, is a signal.
What the score is not: a probability that a human didn't write it. It's a probability that this text resembles the model's output distribution. Those are different statements, and the gap between them is where most of the arguments happen.

Why short diffs weaken the statistical signal
Every detector that aggregates per-token scores has a minimum viable window, and it's larger than people assume. A 60-line Java diff runs about 500 tokens. Strip the context lines and the diff markup and maybe 350 of those are new code. At that size the confidence interval on a mean log-likelihood is wide enough to swallow most of the real difference between human and machine.
Variance is the bigger problem. Human code has spiky perplexity. A developer writes nine boring functions and one horrible date-parsing function with a hardcoded array of month abbreviations and a comment that says // I'm sorry. The whole-file variance is high. LLM output is flat, function after function, at roughly the same per-token likelihood. That flatness is easy to measure across 900 lines and essentially unmeasurable in a 40-line diff.
Concrete case from last quarter. A 34-line diff scored 0.71 on our gate, which put it in the review band. The same author's PR the following week, 900 lines of Kotlin, scored 0.08. What actually happened: Copilot wrote the DTO mapper, the developer hand-wrote everything else and rewrote about half the mapper before committing. The small diff was the only place the tool had enough evidence to say anything, and it was right, narrowly.
Practical rule we landed on: nothing under roughly 1,500 new tokens gets an automatic block. Below that the score goes into the PR comment as context and a person decides. We haven't tested this past a few hundred PRs, so treat the number as a starting point, not a calibration constant.
Cheap signals that survive a small diff
Feature checks don't care about window size. They're local, they're explainable in one sentence, and they cost almost nothing to run on every push. The cheapest one we run is a grep over the added lines.
git diff --unified=0 origin/main...HEAD \
| grep '^+' \
| grep -nEi "certainly|here('| i)s the (updated|complete)|as an ai|hope this helps|feel free to" \
&& echo "model courtesy text in diff" && exit 1
That check has caught three real incidents in eighteen months, all of them someone pasting a chat response straight into a source file. It has never fired on a human writing a comment that happened to begin with "Here's".
The rest of the local set is equally unglamorous. Imports that nothing resolves against, like Jackson annotations landing in a module whose build file has no Jackson dependency. A new file where every public method is wrapped in a try/catch that rethrows RuntimeException, uniformly, including the getters. Placeholder identifiers that survived implementation: yourMethodName, exampleInput, sampleData. Completion density, meaning a diff with no dead code, no debug prints, no commented-out first attempts, and no leftover TODO. Real code, in my experience, is a little embarrassed of itself.
None of these prove anything. But each one gives a reviewer a sentence to act on, which is worth more than a float between 0.6 and 0.7.
Generated boilerplate is not generated logic
The false positive rate on scaffolding is why a lot of teams tear these gates out in month two. Spring Initializr output, protoc stubs, Prisma migrations, OpenAPI clients, create-react-app trees, Rails scaffolds. All of it is template-produced, uniform, and structurally identical to LLM output, because both come from deterministic generation over a common pattern. Detectors cannot separate them and shouldn't be asked to.
Count the boring human code too. Getters, setters, DTOs, toString() overrides, test fixtures with eleven alphabetical fields. A 300-line DTO file will score as AI in most tools and there is nothing in that file worth arguing about.
# .ai-gate-ignore
generated/**
**/proto/**
src/main/java/**/dto/**
src/main/resources/db/migration/**
*.min.js
Two rules for that file. It lives in the repo and gets reviewed like any other config change, and the gate reports what it skipped on every run. An ignore list that grows quietly is precisely how an audit trail ends up with a hole in the one directory where the contractor worked.
Where the AI-generated code check sits in the pipeline
Three placements. Only one of them works.
Pre-commit hook: we tried it. The detector call added 19 seconds to every commit, and by the end of week one half the team had --no-verify in muscle memory. That isn't a tooling failure, it's a design failure. Anything that slow belongs in CI where nobody is waiting on it interactively.
Pull request job: the right place. The diff already exists, the reviewer is already engaged, and the result lands where the decision gets made.
Nightly full-repo sweep: worth having as a drift check, particularly after a merge freeze, but it catches things far too late to matter for an individual PR.
name: code-provenance
on: pull_request
jobs:
ai-gate:
runs-on: ubuntu-latest
timeout-minutes: 5
steps:
- uses: actions/checkout@v4
with:
fetch-depth: 0
- name: Collect diff
run: git diff origin/${{ github.base_ref }}...HEAD -- '*.py' '*.java' '*.kt' '*.ts' > diff.patch
- name: Score the diff
env:
CODEQUIRY_KEY: ${{ secrets.CODEQUIRY_KEY }}
run: |
curl -sS -X POST "$CODEQUIRY_API/analyze" \
-H "Authorization: Bearer $CODEQUIRY_KEY" \
-F "[email protected]" \
-o report.json
jq -e '.ai_score | numbers' report.json
- uses: actions/upload-artifact@v4
with:
name: ai-detection-report
path: report.json
retention-days: 400
The base URL and the key both come off the API keys page in the dashboard, which is where you also register the webhook if you want results pushed into a ticketing system instead of read from an artifact.

One warning, from experience. Our first version compared the score with jq -e '.ai_score > 0.65' and moved on. One response returned the field as a string, jq evaluated the comparison as false, and every PR passed for nine days before anyone noticed the gate had gone quiet. Now the step asserts the field is a number and fails closed when it isn't. A quality gate you cannot see failing is worse than no gate at all.

Thresholds, labels, and who makes the call
The tool should never close a pull request. A merge gate that rejects on a probabilistic score teaches people to route around it, and the routing is always cheaper than the appeal. What works instead is three bands with different levels of friction.
| Score band | Gate behavior | Who acts |
|---|---|---|
| below 0.35 | passes, no comment | nobody |
| 0.35 to 0.65 | passes with a PR comment listing the three highest-scoring files | reviewer reads it as context |
| above 0.65 | merge blocked by the ai-review-required check | code owner clears the label with a one-line justification |
Write the decision into the pull request itself, not into a Slack thread. When a security reviewer or an auditor asks why a given file merged, the answer needs to live in the same place as the code. We keep a provenance: line in the PR body, added by the reviewer, with one of three values and a short justification.
In our repos the top band catches roughly 4% of PRs. Reviewers clear about seventy percent of those after one question ("walk me through what this function does"), usually because the developer used Copilot for part of it and can explain every line. The other thirty percent become a real conversation, and maybe a fifth of those end with the code rewritten. That ratio is fine by me. The goal isn't catching people, it's making sure nobody ships something they can't defend.
Running similarity, web, and AI checks in one pass
Three failure modes look similar in a review and need different evidence.
A contractor pastes a module from a previous employer. No model involved anywhere. The signal is web-source and cross-submission similarity, and it's the one carrying real legal weight.
A junior leans on Claude for a service layer and doesn't fully understand what came back. AI signal only, plus one reviewer question about a specific branch.
A developer finds the exact answer on Stack Overflow and adapts it. Web signal, with a line and token count pointing at the question with 47 upvotes and the accepted answer.
Running three tools with three dashboards meant three ignore lists and three places a result could be quietly dropped. We consolidated. Codequiry scores peer, web, and AI on the same submission and produces one report, and for our purposes that consolidation was the deciding factor rather than any single score's accuracy. If you're coming off MOSS, the honest comparison is that MOSS is free, emits a per-run HTML file, and carries no AI signal at all; the Codequiry vs MOSS comparison covers where the two workflows actually diverge.

What the report has to say for the audit trail
A score without a record is a rumor. For every gate run we persist: base and head commit SHA, detector version and model ID, aggregate score, per-file scores, byte and token counts, the list of skipped paths, and the reviewer's decision with a name and a timestamp.
Model IDs drift. A 0.62 from the reference model we ran in March is not the same 0.62 as the one from October, and if you can't tell which produced which, your historical thresholds are decoration. We store the version string in the artifact and in the PR comment footer.
Retention sits at 400 days, aligned to our audit cycle, as a CI artifact with a copy pushed into the ticketing system. That's longer than anyone wants to pay for and shorter than a lawyer might ask for. Pick a number, write it down, and keep it in the same document as the threshold policy so the two can't drift apart.
Where this breaks
Inline autocomplete. A developer typing their own function with Copilot suggesting tokens is not AI-generated code under any definition worth writing a policy around, and no detector will separate it from hand-written work. That's a handbook question, not a scanner question.
Rewritten output. Ask a model for a function, then rewrite it, rename everything, add your own error handling. The score drops, sometimes hard. Most of the time that's the correct outcome, because what merged is code the developer understands well enough to modify.
Tool disagreement. We pushed the same 800-line file through three detectors and got 0.12, 0.44, and 0.68. Averaging those produces a number with no meaning. Pick one, calibrate it against your own repositories, publish the calibration. Our recipe: 50 PRs merged before June 2022 (a clean pre-Copilot-GA baseline) and 50 files known to be generated, score both sets, then place the threshold where false positives cost less than misses.
Locale. Non-English comments and docstrings shift feature-based scores, because identifier and comment entropy look different in languages the reference model saw less of during training. If some of your developers write comments in Vietnamese or Portuguese, sample their known-human PRs before you commit to a threshold.
The gate isn't there to catch anyone. It's there so that when someone asks why a particular function is in production, there's a record of a person who read it first. If you want to see what the combined report looks like on a real submission, the AI code detector page has a sample run, and you can point a code plagiarism checker at the same files to see the peer and web signals sitting next to the AI score.