Finley Systems, a logistics SaaS company with about 90 engineers, runs a core team of 55 full-time developers and regularly augments its capacity with remote contractors. In early 2024, as the firm pushed to ship a new route-optimization module, the engineering leadership realized they were absorbing code from 12 different external contributors without any structured way to check if that code was original or properly licensed. Their manual review stopped at a quick glance for obvious bugs — no one was tracing provenance, checking for open-source license conflicts, or looking for telltale signs of copy-paste from Stack Overflow. So the team built an automated integrity gate that now screens every contractor pull request before it can land in the monorepo.
The pipeline leans heavily on Codequiry’s code plagiarism checker, layered with static analysis and a lightweight internal similarity comparison. The result is a system that catches everything from a rip of a GPL-licensed authentication module to an entire service class generated by ChatGPT and pasted without editing. Here’s how they chose the tools, what the first month of data looked like, and the hard-won lessons about false positives and threshold tuning.
The Challenge of Scaling Contractor Code Reviews
For years, Finley Systems had code review practices that were thorough for full-time employees but almost non-existent for external contributors. Contractors were typically hired for tightly scoped deliverables — a new microservice endpoint, a dashboard widget, a data pipeline — and their pull requests were reviewed by a single senior developer who looked for obvious logic errors and style violations. There was no formal step to check whether the submitted code matched publicly available code online or had been generated by an AI assistant.
That gap caught up with them when an internal audit of a six-month-old contractor submission uncovered that a 187‑line controller had been lifted verbatim from a popular GitHub repository under the AGPL license. The snippet had made it through code review unnoticed. While the legal exposure was manageable, the engineering director, Marcus Okonkwo, saw it as a systemic risk: “We were just lucky that the AGPL code hadn’t spread further. Next time it could be proprietary code from a competitor or a batch of AI-generated nonsense that passes a smoke test but fails subtly in production.”
Marcus and his team decided they needed an automated, repeatable way to verify the originality of every external code contribution before human review even began. The goal wasn’t to replace manual review — it was to give reviewers a concise, evidence-backed signal on code provenance so they could use their time on logic, design, and security rather than detective work.
Building an Automated Integrity Gate
Tool Selection: Code Plagiarism Detection and Beyond
The team evaluated several options. They knew about MOSS (Measure of Software Similarity) from academic contexts, but MOSS is designed to compare a batch of student submissions against one another — it does not scan the open web for matches and has no native AI-detection capability. For scanning against online sources, they needed something that could index massive amounts of public code (GitHub, Stack Overflow, tutorial sites), handle multiple languages (their stack is primarily Java and Python), and return actionable match reports.
They settled on Codequiry for the web‑plagiarism and AI‑detection layers because it provides a single API that could be dropped into their CI pipeline and returned similarity scores along with source URLs for matches. For open-source license auditing, they retained FOSSA — they were already using it for dependency compliance — and configured it to scan incoming contractor code snippets for license conflicts at the same time.
Additionally, to catch a contractor copying from Finley’s own internal closed-source code, they built a small harness that ran MOSS (using the Stanford server with a custom script) on every new PR’s changed files against a snapshot of the internal monorepo’s main branch. This was a safety net for the edge case where a contractor might have had prior access to proprietary code or a full-time employee accidentally leaked a snippet on a public forum that was then reused.
Integrating Codequiry API into CI/CD
The core of the automation sits in GitHub Actions. Every pull request from any of the contractor‑managed repositories triggers a workflow called Code Originality Check before the normal test suite runs. The workflow checks out the branch, identifies new and modified files, and sends them to the Codequiry API for analysis.
Here’s a simplified version of the GitHub Actions workflow they use:
name: Code Originality Check
on:
pull_request:
branches: [main]
jobs:
integrity-gate:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
with:
fetch-depth: 0
- name: Set up Python 3.9
uses: actions/setup-python@v4
with:
python-version: '3.9'
- name: Install dependencies
run: pip install requests
- name: Run Codequiry originality scan
env:
CODEQUIRY_API_KEY: ${{ secrets.CODEQUIRY_API_KEY }}
run: python scripts/codequiry_check.py
The Python wrapper script extracts the diff, filters out test files and generated code (to reduce noise), concatenates the changed source files within language‑specific chunks, and calls the Codequiry API endpoint for each chunk. For a typical PR of 200–400 lines, the check completes in under 30 seconds. The API response includes a list of matched web sources, similarity percentages, and — when AI detection is requested — a probability score for each chunk indicating AI‑generated content.
import os
import requests
import sys
def get_files_changed():
# Run git diff to get list of changed .java or .py files
# (simplified for illustration)
pass
def scan_file_chunk(file_content, filename):
headers = {"Authorization": f"Bearer {os.environ['CODEQUIRY_API_KEY']}"}
payload = {
"code": file_content,
"language": "java" if filename.endswith(".java") else "python",
"check_ai": True
}
r = requests.post("https://api.codequiry.com/v2/check", json=payload, headers=headers)
return r.json()
results = []
for fname, content in get_files_changed().items():
try:
result = scan_file_chunk(content, fname)
if result["similarity_score"] > 0.35 or result["ai_probability"] > 0.6:
results.append((fname, result))
except Exception as e:
print(f"Error scanning {fname}: {e}", file=sys.stderr)
if results:
print("Potential issues found:")
for fname, r in results:
print(f" {fname}: similarity {r['similarity_score']}, AI prob {r['ai_probability']}")
sys.exit(1)
else:
print("All clear.")
The workflow fails if any file exceeds a configured threshold (initially 0.35 for overall similarity and 0.6 for AI probability), automatically blocking the merge. A summary comment is posted to the PR with links to the matched sources so the reviewer can inspect them immediately.
Layering Web-Source and AI Detection
One early decision the team debated was whether to run AI detection on every file or only on files that already triggered a high similarity score. They chose to run both scans simultaneously because they observed that AI‑generated code often had no direct web matches — it was novel, not plagiarized from a single source — yet still posed a quality and IP ownership risk.
By layering the two checks, they could also cross‑reference findings. A file with a high AI probability and a high similarity to a known online source that itself was AI‑generated reinforced the signal. Conversely, a suspiciously high AI score on a file with zero web matches required closer manual review, often revealing telltale ChatGPT‑style comments (// Step 1: fetch data from API) and unusually verbose variable names that a human developer would shorten.
Marcus’s team configured Codequiry’s detection engine to compare against a broad index that includes GitHub public repositories, Stack Overflow snippets, tutorial sites, and well‑known AI‑generated code datasets. They didn’t have to maintain that index themselves — the API automatically updates its corpus — so the ongoing maintenance burden was essentially zero.
Real-World Results in the First Month
The integrity gate went live in March 2024. For the next 30 days, the team tracked every contractor pull request that passed through the pipeline. The numbers were eye-opening even for a team that expected to find some issues.
- Total contractor PRs processed: 121
- PRs flagged for high external code similarity (≥ 0.35): 14 (11.6%)
- Confirmed copy-paste from Stack Overflow or public GitHub repos: 6 (included one 200‑line AGPL‑licensed authentication module)
- PRs flagged for AI‑generated content (≥ 0.60 probability): 9 (7.4%)
- PRs where a human reviewer agreed it was likely AI‑written: 4 (in one case, the code still contained the placeholder comment Generated by ChatGPT)
The 14 similarity flags also included three trivial matches—small utility functions like a base64 encoder that had near-identical implementations everywhere—and five borderline cases that were discussed with the contractor and ultimately accepted after adding proper attribution.
Marcus was particularly struck by the AI detection numbers. “I did not expect to see straight-up ChatGPT output with zero modification on 3% of our incoming contractor code. That’s not a hypothetical — it’s live production code we almost shipped.” Those submissions were rejected, and the contractors involved were given a clear policy reminder. No project timelines were blown; the gate caught the code before it ever reached integration testing.
False Positives and Threshold Tuning
No automated system is free of false positives. In the first week, the team discovered that the similarity threshold of 0.35 was too aggressive — it flagged several instances of boilerplate Spring Boot configuration classes that any Java project would contain. They raised the similarity trigger to 0.40 for files under 50 lines and kept 0.35 for larger blocks, reasoning that a short boilerplate match is less concerning than a 200‑line segment.
For AI detection, the initial threshold of 0.60 caught some pieces the reviewers eventually deemed human‑written but heavily templated (e.g., a CRUD service class generated from an IntelliJ template). They introduced a quick‑approval path: if the AI score was between 0.60 and 0.75 and the similarity score to web sources was under 0.20, the reviewer could manually dismiss the flag without escalation. This cut the false‑positive rate in half while retaining the ability to catch unambiguous AI‑generated blocks.
Lessons Learned and Ongoing Adjustments
Six months in, the Finley team considers the gate a permanent part of their CI infrastructure, not a temporary experiment. They’ve distilled a few principles from the experience.
Combine signals, don’t rely on one. AI detection alone can be noisy — especially for boilerplate code — but pairing it with web‑source similarity and a lightweight policy check (e.g., scanning comments for AI‑artifact phrases like “generated by”) produces far fewer false alarms while catching real issues.
Thresholds must adapt to file size and language. A 90% similarity match on a 15‑line utility function is often innocuous; the same match on a 300‑line business logic module is not. They now use a sliding scale based on lines of code, defined in a simple JSON configuration file that reviewers can adjust without touching pipeline code.
Contractors need a clear policy, not just a gate. After the first month’s findings, Finley updated its contractor agreements to explicitly state that code must be original, that AI assistants used during development must be disclosed, and that repeated flagged submissions may result in contract termination. Combined with the automated checks, this has shifted behavior: contractor PRs flagged for plagiarism and AI dropped by over 60% in the following quarter.
The team is now exploring deeper integration, such as automatically extracting matched snippet URLs and injecting them directly into sonar‑style review dashboards, and expanding the MOSS‑based internal comparison to catch subtle code‑clone refactoring across PRs. For now, the combination of Codequiry’s API checks and the lightweight workflow strikes the right balance between catching real problems and respecting developer velocity.
Frequently Asked Questions
Can automated checks catch code copied from Stack Overflow?
Yes. Codequiry’s web‑source comparison indexes millions of Stack Overflow answers and public GitHub repositories. When a contractor copies a block and pastes it unchanged, the system flags the match and provides a direct URL to the source, making it easy for a reviewer to verify.
Does Codequiry detect AI-generated code from ChatGPT, Copilot, or Claude?
It does. The AI detection model identifies statistical patterns characteristic of large language model output — things like consistent low‑perplexity text, uniform token distribution, and certain syntactic artifacts. It returns a probability score that teams can use as a triage signal, with the understanding that no detector is perfect and human review remains important.
How do you avoid flagging common utility functions that look the same everywhere?
By tuning similarity thresholds based on file size and, where possible, language. Short files (under 50 lines) are treated with a higher threshold to reduce noise from boilerplate, while larger files use a more sensitive scale. Teams can also whitelist specific file patterns (e.g., config classes) from the check.
Is this workflow only useful for contractor code?
No. The same pipeline can be extended to full-time employee contributions, open-source inbound contributions, and even legacy codebase audits. The core idea — automated provenance checking at the pull request level — applies anywhere code ownership and license clarity matter, including enterprise monorepos and public open-source projects.