I've had an AI code detection job in our merge request pipeline for about four months. It flags one or two suspicious files a week across 14 active repos. It also generated 41 false positives in the first two weeks because I trusted a vendor default threshold. That second number is the one worth walking through, because the pipeline is easy and the policy tuning is where most teams quietly give up.
This is how I wired AI code detection into GitLab CI so it runs on every merge request, fails the build on hard evidence, and leaves enough of an audit trail that we can defend a rejected PR without getting into a shouting match with a contractor.
The merge request is the cheapest place to catch AI-generated code
Once something merges to main, it starts accumulating dependencies, tests, and refactors. The cost of removing a generated component two weeks later is a multiple of the cost of catching it in review. Our group has a simple rule: no AI detection, no green merges. It didn't come from a compliance mandate. It came from an incident in April where a contractor shipped a Python utility that looked competent in review, had no tests, and contained a 94% AI probability on the exact file the reviewer skimmed because the diff was long.
The merge request is the last place an attentive human is actually looking at the code.
If you don't put the AI check there, you're spending real money to find LLM output after it's already in main.
GitLab makes this easy. A job in a merge request pipeline runs before the branch can be merged, but only after the code has been written and pushed. That's the right moment. You aren't slowing a developer's local loop, and you aren't after the fact. You get signal at the one point where a policy gate has teeth.
Pick an AI code detection tool that fits GitLab CI
I trialed three approaches before settling on Codequiry. Two of the tools I looked at were text detectors wearing code-colored hats. They cared about English prose perplexity, not token sequences in a Python file. A third did source-level analysis but required a self-hosted model server that our infrastructure team refused to deploy without another six months of security review. I needed something that ran from a thin Linux container in CI, returned JSON, and didn't force me to maintain a separate server.
Codequiry's AI code detector worked for us because the same API key covers AI detection and source similarity. We didn't want two vendors in the gate. The Python client installs from PyPI, and the CLI accepts a path plus an engine flag. For GitLab CI, the setup is just an environment variable and a script block. The key lives in a masked CI variable, and the report lands as an artifact so the reviewer can click through the score instead of trusting a red X.
The .gitlab-ci.yml that calls the scanner
We run the scan as its own stage, before build and test. That keeps the feedback loop short and prevents a failing AI gate from wasting build minutes. The job is restricted to merge request events, which means direct pushes to main from maintainers bypass it. That's a hole we accept for emergency fixes, but it's a policy hole, not a technical one.
ai-scan:
stage: verify
image: python:3.12-slim
rules:
- if: '$CI_PIPELINE_SOURCE == "merge_request_event"'
variables:
CODEQUIRY_API_KEY: $CODEQUIRY_API_KEY
script:
- pip install --quiet codequiry-python==1.4.2
- codequiry scan --path . --engine ai --output ai-report.json --format json
- python .gitlab/check_ai_score.py ai-report.json 0.68
artifacts:
paths:
- ai-report.json
expire_in: 7 days
The first version of this job had a timing bug I mentioned earlier. The scanner is asynchronous for larger repos, and my initial script polled the status endpoint every 15 seconds. For our legacy Django monorepo, the scan took just under three minutes. The job's 240-second default timeout expired on the third attempt, and the MR slid through with a green checkmark because GitLab read the timeout as a pipeline failure, not a policy failure. That's not a bug in Codequiry. It's the API being honest about async work. I switched to a 600-second job timeout and a CLI status poller with a 20-second interval. Read the async section of whichever tool you pick before you wire the gate.
Reading an AI score report without losing trust
The report Codequiry returns is not a single number. It has a cohort average, a highest file score, and per-file probabilities with line-level indicators. That matters because a repo-wide average of 0.43 can hide one file sitting at 0.91. We don't fail on the average. We fail on the files.
{
"average_ai_score": 0.48,
"highest_ai_score": 0.93,
"files": [
{
"path": "src/services/reconciliation.py",
"ai_score": 0.93,
"lines_flagged": 42
}
]
}
I set a hard fail threshold at 0.68 after two months of tuning. Below 0.60 I almost never care. Between 0.60 and 0.68, I generate a review comment but don't block the merge. The reason for that band is simple: a developer who legitimately writes a lot of similar Django view code can trip a 0.55 on boilerplate, and we weren't going to make them rework a pull request over that. The threshold is a business decision, not a vendor setting.
One thing I always tell new reviewers: look at the diff, not just the score. If the flagged lines are untouched legacy code, the file score is a historical artifact. I've started running a quick script that intersects flagged file paths with `git diff origin/main...HEAD --name-only`. If a file isn't in the diff, I drop it from the fail list. That single change eliminated most of the remaining false positives.
Failing the build is the easy part; the audit trail is the hard part
Our policy gate script is deliberately small. It reads the JSON report, applies two thresholds, and exits nonzero only when a file crosses the hard fail line. Everything between soft review and hard fail gets printed as a warning and attached to the MR discussion via the GitLab API.
import json, sys
with open(sys.argv[1]) as f:
report = json.load(f)
hard_fail = float(sys.argv[2]) if len(sys.argv) > 2 else 0.68
soft_review = hard_fail - 0.10
failed = []
review = []
for file in report.get("files", []):
score = file.get("ai_score", 0.0)
if score >= hard_fail:
failed.append(file)
elif score >= soft_review:
review.append(file)
for file in failed:
print(f"FAIL: {file['path']} ai_score={file['ai_score']:.3f}")
for file in review:
print(f"REVIEW: {file['path']} ai_score={file['ai_score']:.3f}")
sys.exit(1 if failed else 0)
The report artifact is stored for seven days, which is long enough to answer a contractor's dispute without keeping indefinitely. We also send a signed webhook to a Slack channel with the highest-scoring file path and the MR link. That webhook is the part most teams skip. The CI job fails, someone clicks merge anyway with admin rights, and three weeks later nobody can find the evidence. An audit trail with a short retention window is more useful than a permanent pile of JSON nobody reads.
Tuning thresholds when every repo has a different baseline
We have four repo types under the same GitLab group, and a single 0.68 threshold did not survive contact with all four. The table below is where we landed after two months of adjusting weekly. The idea is not scientific. We haven't tested this past a few hundred submissions, and the numbers drift as model versions change.
| Repo type | Hard fail | Soft review | Notes |
|---|---|---|---|
| Python services | 0.68 | 0.58 | Greenfield code, fewer boilerplate files |
| Legacy Java monolith | 0.75 | 0.65 | Generated getters and DTOs push baseline up |
| Infrastructure IaC | 0.60 | 0.50 | Terraform and Ansible YAML trip high on templated blocks |
| Contractor repos | 0.65 | 0.55 | Stricter because external authorship is higher risk |
If you're starting fresh, I'd begin at 0.65 and expect a false positive rate near 8 to 11 percent for the first couple of runs. Watch the files that land in soft review. That band tells you more about your own codebase's baseline than any vendor documentation will. Lower the hard fail threshold only after you've looked at a few weeks of soft review files and decided none of them deserved a block.
What changes when you scan student submissions instead of production code
I consulted on a data structures course at a state university last spring, and the problem there was different. A student's AI-generated selection sort submission is often a complete file, not a partial diff. There is no legacy code to mask. The instructor wanted a hard pass/fail per assignment, not a merge request gate. Codequiry handles that case too, but the workflow flips from CI event to batch scan.
In that setting, the useful signal comes from stacking AI detection with peer similarity and web-source checks. A student who pastes a Stack Overflow answer and then edits the variable names will sometimes show a lower AI probability than a student who asked ChatGPT for a clean-room solution. The AI score alone misses the copied-work problem. That's why I keep saying Codequiry should be thought of as a code plagiarism checker with an AI layer, not a one-trick LLM detector. MOSS remains good for peer similarity in a course, and JPlag catches some refactoring, but neither gives you a web-source match or an AI probability in the same report.
The university ran the same Codequiry scan across 280 submissions after the midterm. It flagged 14 high-risk files with AI scores above 0.80, and another 9 with strong web or peer matches. The instructor reviewed all 23 manually. Three were false positives from students using the same course skeleton. That's a 13 percent false discovery rate, which is acceptable if a human is the final arbiter. No one was reported to the honor board on the basis of a score alone.
Build the gate, then defend it
Developers will initially see an AI detection gate as an accusation. The fastest way to get the policy reversed is to make it block a legitimate pull request without a clear explanation. That's why the soft review band and the audit trail matter more than the fail threshold. Our message to engineering is simple: the scanner produces evidence for review, not a verdict. The verdict belongs to the person who reads the diff.
Once the gate has been running for a month, pull the numbers. We saw our highest-scoring files cluster around three patterns: generated CRUD endpoints, autogenerated tests that asserted nothing, and a surprising number of regex-heavy parsing utilities. That last one became a training issue, not a policy issue. A developer thought GitHub Copilot was a faster way to write a date parser. It was, until the parser failed on leap years. The gate didn't just catch LLM output. It caught the places where we had stopped paying attention.
If you're setting up your first AI code detection job in GitLab CI, start with a small repo, a loose threshold, and a week of soft review. Don't wire the block step until you've read a few dozen reports yourself. After that, treat the bot like any other CI check: deterministic, observable, and bound by a policy a human can override with a clear reason.
Wiring this took an afternoon. Tuning it took two months. The alternative was realizing in a post-incident review that nobody could tell human code from model output. I'd rather have the gate. You can try the AI code detector with your own merge request pipeline and see where your baseline lands.