Open source license scanning began as a legal review exercise, not an engineering one. It is now a pipeline stage, with SPDX identifiers, SBOMs, and policy-as-code gates deciding whether a build can go to production. If you are responsible for this in 2024, you are not really building a scanner. You are building a way to turn license evidence into an approval decision, and that decision needs to survive an audit.
I run DevSecOps for a mid-size fintech. We scan every dependency, every Docker layer, and every vendored file before code reaches production. I have sat through one external audit where the auditor asked for our license conclusions, not our license declarations, and we spent two days explaining the difference between the two.
Open source license scanning before SPDX was mostly grep and hope
In the early 2000s, compliance meant reading COPYING files and searching for the phrase "GNU General Public License" inside a tarball. The first scanner I remember using professionally was FOSSology 1.0, which came out of HP around 2007. It walked a source tree, matched license headers, and emitted a browser report. It was better than grep, but it was slow. Its detection rate fell hard once someone reformatted a header or changed a few words.
A typical pre-SPDX check looked like this:
grep -R "GNU General Public License" --include="*.c" --include="*.java" .That catches a verbatim header. It does not catch the same file with the header stripped, translated, or moved to an unusual location.
SPDX gave us a shared vocabulary, not a verdict
SPDX 1.0 arrived in 2011, pushed by the Linux Foundation and a working group of legal and engineering people from Wind River, TI, HP, and others. The core idea was small but important: a fixed list of license identifiers such as MIT, Apache-2.0, and GPL-2.0-only. A scanner could now say "Apache-2.0" instead of a paragraph of legalese, and a policy engine could act on that string.
{
"spdxVersion": "SPDX-2.3",
"packages": [
{
"name": "json-sanitizer",
"versionInfo": "1.2.3",
"licenseDeclared": "MIT",
"licenseConcluded": "MIT"
}
]
}The distinction between licenseDeclared and licenseConcluded is the single most misunderstood part of SPDX. Declared is what the package says about itself. Concluded is what a human or tool determined after looking at the actual files. In practice, most scans leave licenseConcluded as NOASSERTION because they only look at manifest metadata.
SPDX is a vocabulary, not a verdict. The scanner proposes. The compliance policy disposes.
The dependency graph era changed what we scan
By 2017, GitHub had a dependency graph, npm and Maven had license metadata, and scanning moved from files to package manifests. That was a real improvement for speed. It also introduced a false sense of coverage. A package can declare MIT in its pom.xml and still have a vendored file copied out of a GPL-3.0 project with the header removed.
We ran into that with a Python service in 2021. The repository declared MIT at the root. A contractor had copied two utility functions from a GPL-3.0 CLI project and pasted them into a utility module. No license header. No attribution. The scanner saw the declared root license and passed it. An auditor caught it later because the function signatures matched the upstream project precisely.
That gap is why license scanning and source similarity checking have to live in the same pipeline. A code plagiarism checker that compares token sequences and AST structure against public GitHub catches the copied function even when the license header is gone. The license scanner handles the declared metadata. The similarity tool handles the provenance.
Policy-as-code and the long tail of edge cases
Today, the real problem is not scanning. It is turning scan results into a decision without blocking every build or letting everything pass. In CI, we run a policy gate that maps SPDX identifiers to allow, warn, and fail buckets.
# .license-policy.yml
fail-on:
- GPL-3.0-only
- AGPL-3.0-only
- SSPL-1.0
warn-on:
- LGPL-2.1-only
- MPL-2.0
allow:
- MIT
- Apache-2.0
- BSD-2-Clause
- BSD-3-Clause
- ISC
- Python-2.0Policy-as-code has its own failure modes. A flag named include-dev-dependencies is the one everyone forgets. We shipped our first gate with that set to false by accident. For six weeks, the scanner ignored dev dependencies. Three GPL-3.0 jars sat in test fixtures unreviewed. The issue was not the scanner. It was the pipeline quietly narrowing its own scope.
That is why I treat license gates as evidence producers, not decision makers. The gate can block on a clear copyleft dependency. It cannot tell you whether a copied function in a vendored file came from Stack Overflow under CC BY-SA or from a private project.
Where scanners still fail and what to do about it
The three gaps I see in production are stripped headers, generated code, and package metadata drift. Stripped headers are caught by similarity matching. Generated code is a newer problem. Copilot, Codex, and Gemini can emit functions that reproduce license-covered snippets without any license text. The original license may be in a training corpus, but the output carries no metadata. We treat AI output as a provenance risk and run an AI code detector alongside license and similarity scans. The combination catches the ambiguous middle: code that is not clearly human-written and not clearly flagged by a license scanner.
Metadata drift is simpler. A package can have an old license in its manifest and new files under a different license. Tools like ClearlyDefined and OpenChain have done real work here, but the data is only as good as the last curation pass. We have not tested our current pipeline past a few hundred unique service repos, so I would not call the problem solved.
Codequiry is useful in this stack because it returns web source matches, peer similarity, and AI-written code scores in one report. When a stripped header is involved, the web match is usually the evidence that matters. It traces a copied file back to the exact GitHub repo or Stack Overflow question, with per-domain scores and line counts. That is forensic evidence, not a checkbox.
A practical way to line these up
If you are in a mid-size team and you do not want to buy every scanner, start with a manifest-level scanner for speed. Layer on a source-level scan for files that contain no package metadata. Then add a web and peer similarity check for copied code. If your team uses Copilot or Codex, add an AI detector to the same review queue. That is more tooling than most teams want, but it beats finding out during an audit.
Codequiry fits the last two layers in a single API and dashboard. The source code plagiarism checker is the piece that catches the stripped-header problem. For a university or bootcamp, the same workflow applies to student submissions, where license provenance and copied source overlap heavily.
License scanning is a chain of custody problem
The useful lesson from this history is not which tool won. It is that license scanning evolved from finding license text to proving where code came from. SPDX standardized the label. SBOMs standardized the inventory. Policy gates standardized the decision. None of them solved the provenance problem by themselves.
If you are setting this up now, do not let declared metadata be the only signal. Add a similarity scan against public sources. Add an AI detector if your team generates code. Keep the policy gate simple, and set include-dev-dependencies to true on day one.
When you are ready to stack license scanning with source and AI provenance checks, run a code similarity check on your riskiest repos first.