In March 2003, the SCO Group filed a $5 billion suit against IBM, alleging that large parts of the Linux kernel had been lifted from Unix System V. The claim never survived a courtroom. The question underneath it did, and that question is what open source license detection has spent two decades learning to answer: not what a file says about its license, but where the code inside it came from.
The distance between those two questions is the whole history of this tooling. A scanner that reads a header is reading a claim. A scanner that reads structure is reading evidence. The industry spent a long time building the first kind and calling it the second.
The compliance problem began as a reading problem
Version 1 of the GNU General Public License shipped in February 1989. Version 2 arrived in June 1991 and was adopted by the Linux kernel in 1992, when Linus Torvalds wrote his now-famous note inviting commercial use under its terms. The mechanism of compliance in that design was a block of prose at the top of every source file, plus a LICENSE file at the root of the tree.
Richard Stallman's assumption was that the notice would travel with the code. In practice notices do not travel. Minifiers strip them. Code generators omit them. Developers consolidating a directory delete them because they look like clutter. Anyone copying forty useful lines out of a repository into a different project leaves them behind entirely.
/*
* Copyright (C) 2004 Example Corp.
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
* the Free Software Foundation; either version 2 of the License, or
* (at your option) any later version.
*/Delete the comment and the obligation does not disappear. That has been the settled legal position since 2008. Detecting that the comment was deleted is a different problem from proving it, and for most of the 1990s no product on the market did either one.
Jacobsen, BusyBox, and the decade enforcement got teeth
The legal foundation moved in August 2008, when the Federal Circuit decided Jacobsen v. Katzer. Before that ruling, open source licenses were widely treated as contracts: breach got you damages if you could prove them, which in practice meant almost nothing. After it, an open source license was a condition of copyright, and violating the condition was infringement, with statutory damages and injunctions on the table.
Enforcement followed quickly. The Software Freedom Conservancy, then operating as part of the Software Freedom Law Center, filed a series of BusyBox suits beginning in December 2007 against Monsoon Multimedia, and later against Xterasys, High-Gain Antennas, Verizon, and eventually Best Buy, Samsung, and Westinghouse. The settlements were structurally similar: publish the source, appoint a compliance officer, notify downstream recipients. In Germany, Harald Welte's gpl-violations.org project had been obtaining injunctions since 2004, including against Fortinet in 2005.
Bradley Kuhn, who worked those cases and later founded the Software Freedom Conservancy, has argued for two decades that the objective was never damages. It was getting the source published. The lawsuits were a means of forcing a disclosure that the license already required.
The enforcement wave made one thing obvious to general counsel: a company could be liable for code it did not know it had. That is a purchasing problem before it is a legal one, and it is where the first commercial scanners found their buyers.
Grepping for the wrong string
Black Duck Software was founded in 2002 in Burlington, Massachusetts, and its early products did roughly what the name suggests. The company's knowledge base catalogued open source components by filename, checksum, and license text. If a build tree contained a known archive, Black Duck could name it.
What that approach could not do was read a file. A single C source file with no header, or a JavaScript bundle produced by a build tool, carried no filename signal at all. The first generation of scanners was excellent at inventory and nearly blind to ancestry.
Hewlett-Packard's FOSSology, released as an open source project in 2007 and later moved to the Linux Foundation, made the split explicit in its architecture. Separate agents handled full license texts, short license fragments, and Debian copyright files. The reason for two text agents is the reason the field stayed hard for a decade: full-text matching has high precision and low recall, and fragment matching has the reverse. A short excerpt of the Apache 2.0 notice appears inside dozens of unrelated third-party headers, so fragment matchers generate false positives unless someone tunes minimum match lengths by hand. FOSSology's maintainers spent years on that tuning.
The same maintenance burden hit every scanner that followed. The SPDX License List passed 600 identifiers in 2023, and keeping a rule set synchronized with it is its own job. scancode-toolkit's detection rule set now runs past two thousand entries, and its release notes routinely include license-text corrections that shift match counts by a percentage point or two between versions.
The plagiarism community had already solved this
While legal departments were reading headers, a computer science professor at Berkeley had built something structurally better. Alex Aiken's MOSS system went into service in 1994 to compare student programming assignments. Its underlying algorithm was published nine years later, in a 2003 SIGMOD paper by Saul Schleimer, David Wilkerson, and Aiken titled "Winnowing: Local Algorithms for Document Fingerprinting."
The idea is compact. Tokenize the source, discard whitespace and comments, break the token stream into overlapping k-grams, hash each k-gram, then select the minimum hash in each window of consecutive k-grams. The surviving set of hashes is a fingerprint that survives renaming, reformatting, comment removal, and moderate reordering.
import hashlib
def kgram_hashes(tokens, k=5):
grams = (" ".join(tokens[i:i + k]) for i in range(len(tokens) - k + 1))
return [int(hashlib.md5(g.encode()).hexdigest()[:8], 16) for g in grams]
def winnow(hashes, window=4):
# Sketch, not the reference implementation.
out, last = [], None
for i in range(len(hashes) - window + 1):
m = min(hashes[i:i + window])
if m != last:
out.append(m)
last = m
return outThat property, surviving cosmetic change, is exactly what license detection needed and what text matching lacked. Parallel work in Europe landed on related answers. Guido Malpohl and Lutz Prechelt's jPlag, developed at the University of Karlsruhe from 1996 and published in 2000, used greedy string tiling on token sequences. CCFinder, from a Japanese team in 2002, used suffix trees to find cloned blocks across large codebases.
Two fields with a shared algorithmic ancestor stayed in separate products for fifteen years. License scanners optimized for reading manifests and license texts at industrial scale. Plagiarism detectors optimized for pairwise comparison inside a fixed corpus. Neither vendor had a reason to cross the aisle, because neither customer was asking for the other product.
SPDX made licenses machine-readable, not verifiable
The SPDX project launched in 2010, published version 1.0 in August 2011, reached 2.0 in 2015, became ISO/IEC 5962:2021, and shipped 3.0 in April 2024. Before SPDX, every scanner had its own identifier vocabulary, and reconciling two vendors' reports meant a human mapping table. SPDX ended that by giving the industry a shared dictionary.
It did not end the underlying problem. An SPDX identifier is a declaration. A package can faithfully declare MIT in its manifest and still ship a GPL-licensed file in a vendored subdirectory, and nothing in the SPDX layer will notice. The declaration is portable; the evidence is not.
| Era | Dominant method | Representative tools | Reads well | Blind spot |
|---|---|---|---|---|
| 1989 to 2002 | Manual review of headers and LICENSE files | Legal checklists, grep | Clearly labeled copyleft notices | Copied code with the notice removed |
| 2002 to 2010 | String and regex matching against license texts | Black Duck Protex, FOSSology (2007) | Full texts pasted into files | Paraphrased, minified, or partial notices |
| 2010 to 2019 | Curated identifiers plus component fingerprints | SPDX, scancode-toolkit (2015), Licensee, FOSSA | Declared licenses at repository scale | Undeclared reuse and generated code |
| 2019 to 2025 | SBOM-driven dependency scanning plus code similarity | CycloneDX, SPDX 3.0, CI-native scanners | Transitive dependency inventory | Code reproduced by language models |
SBOMs turned inventory into a legal requirement
Executive Order 14028, signed in May 2021, put software bills of materials on the federal procurement path. CISA published its minimum SBOM elements in September 2023. The EU Cyber Resilience Act entered into force in December 2024 with its own phased supply chain obligations. CycloneDX was standardized as ECMA-424 and SPDX reached 3.0, so for the first time there is a plausible answer to "what is in this build."
What SBOMs describe is declared components. They do not describe code ancestry. A regulated manufacturer can now produce a machine-readable inventory of eleven hundred dependencies and still be unable to say whether the same twenty lines of parsing logic appear in a competitor's product, a contractor's handoff, or a student's senior project. The inventory problem and the provenance problem were never the same problem, and the standardization decade solved only the first.
Copilot reopened the question SCO asked
GitHub Copilot entered technical preview in June 2021 and general availability in June 2022. Almost immediately, the code provenance question that had been dormant since the BusyBox settlements came back in a form no existing scanner could handle.
GitHub published its own measurement: roughly one percent of suggestions, in files where public code matches were enabled, contained snippets of about 150 characters or more that matched public code. The mitigation was a filter that blocks such suggestions by default. That threshold is the interesting number. A hundred and fifty characters is about two lines of tightly written Java, or three lines of a C macro, or a single dense Python lambda chain. Code below it passes.
Legal exposure moved in parallel. Doe v. GitHub was filed in the Northern District of California in November 2022. In July 2024, Judge Jon Tigar dismissed most of the claims but allowed a DMCA Section 1202(b) theory about removal of copyright management information, along with some contract claims, to proceed. The case is not a ruling on whether generated code infringes. It is a ruling that the question is worth litigating, which is enough to change procurement behavior.
The filter also only covers suggestions the model emits. It does nothing about a developer pasting a function out of a chat window, and it says nothing about the license status of near-verbatim output below the character threshold. A scanner reading the resulting file sees a file with no header and no manifest entry.
What the detection stack actually looks like in 2025
Three tool categories now operate in parallel, and they answer different questions. Conflating them is the most common mistake in compliance reviews.
Dependency scanners (Syft, Trivy, the Snyk and FOSSA agents, Black Duck's current generation) walk manifests and lockfiles to produce the bill of materials. They are precise about what was declared and installed, and silent about what was typed. License text scanners (scancode-toolkit, FOSSology, Licensee) identify declared or detectable licenses per file with high accuracy on full texts. Similarity and AI detectors (MOSS, jPlag, Dolos, and Codequiry) compare code against other code, which is the only one of the three that speaks to ancestry.
That last category is where the two historical threads finally meet. Codequiry's source code plagiarism checker combines the token and AST-level fingerprinting that came out of the winnowing lineage with a corpus that includes peer submissions, public GitHub repositories, and indexed web sources, plus a separate model for AI-generated code. For an engineering team auditing a contractor handoff, that combination catches the case no SBOM will ever catch: a module that imports nothing unusual, declares nothing, and was reproduced almost exactly from a repository three years ago.
The comparison to the academic tools is instructive. MOSS remains free and effective at pairwise comparison inside a submitted set, but it is a batch service with a plain-text report and no web-source or AI dimension. Codequiry's side-by-side against MOSS lays out where the workflows diverge, and the practical difference for an enterprise is a REST API and a CLI that can be called from a release gate rather than a results link emailed to a grading account.
Multi-language coverage matters more here than it does in grading. A single commercial codebase in 2025 routinely mixes Java services, TypeScript front ends, Python data pipelines, and shell provisioning scripts, and a license obligation attaches to the copied fragment in whichever of those it happens to live. Codequiry's detection stack covers 65 parsed languages, which is roughly the range you need before a scan stops producing silent gaps.
What the record does not show
Honesty requires flagging the soft numbers. No published study has measured how much code in commercial repositories is attributable to language model output, and the estimates that circulate are extrapolations from memorization benchmarks that disagree by an order of magnitude. GitHub's one percent figure was measured by GitHub, on GitHub's completion data, and has not been independently replicated at the same scale.
Vendor-reported detection accuracy is similarly unaudited. Similarity scanners quote precision and recall on benchmark corpora that they did not always build, and the corpora are rarely public. Nobody has published a false positive rate for license detection across a representative sample of real repositories, which means every "detected license" field in an SBOM carries an unquantified error bar.
There is also a structural limit that no amount of engineering removes. Similarity detection compares against a corpus. If the source a developer copied from was a private repository, an internal wiki, or a paid tutorial, no scanner on the market will surface the match, because the match target does not exist in any index a vendor can crawl.
Three decades, one direction
The arc is consistent even where the numbers are not. Detection started with text a human wrote to declare intent. It moved to structure that survives when the declaration is stripped. It moved again to inventory that regulators can read. Each step widened the corpus being compared and narrowed the gap between what a file claims and what it contains.
The next step is almost certainly finer granularity. Per-file attribution is already too coarse for a codebase assembled from generated snippets, vendored libraries, and human edits in the same commit. Function-level provenance with a confidence score is the obvious direction, and the tooling to produce it exists in pieces.
A compliance lead at a European industrial supplier, who asked not to be named because a vendor negotiation is in progress, described the current state without much optimism. "We can tell you every component in the build," she said. "We cannot tell you who wrote it." Three decades of tooling have made the first sentence easy. The second one is still the honest answer to SCO's question.
For teams that need the second sentence answered inside a single workflow, a code plagiarism checker that runs peer, web, and AI analysis in one pass is the closest thing currently shipping.