Count the tokens, not the lines. That's the shortest version of how modern code similarity detection works, and it's also where most teams draw the wrong conclusion about their scanners. Run git diff on two submissions and you get a line-by-line text comparison that treats int count and int total as unrelated lines even when they mean exactly the same thing. Structural detectors ignore that surface and compare the shape underneath. What follows is a walk through what token normalization and AST normalization actually do, where each one earns its keep, and how to slot both into a real pipeline.
Why line diffing fails at code similarity detection
Line-level diffing is what git diff, MOSS's base comparison, and most naive compare-two-files scripts do first. It's cheap, fast, and for finding edits in a document, it's excellent. For finding plagiarism, it's close to useless on its own.
The issue is that a copier's edits are surface edits. They rename variables and functions, reorder methods, swap a for loop for a while, strip comments, and reformat the file with black or prettier. Every one of those produces a diff that looks substantial on screen while changing nothing about the logic. If your detector only sees lines, you see a 45% text match and move on. Two independently written solutions to fizzbuzz also produce a 20 to 30% text match on boilerplate alone, which is the other half of the false-positive problem.
Line diffing's failure mode is symmetric. It under-reports copied code and it over-reports similar-looking code.
What token normalization strips
A token-based detector begins by lexing the source. Lexing is language-aware. It doesn't see characters and whitespace, it sees tokens: identifiers, keywords, literals, operators, punctuation. public static void main(String[] args) in Java becomes a stream that looks roughly like KW_PUBLIC KW_STATIC KW_VOID IDENT LPAREN TYPE_IDENT LBRACKET RBRACKET IDENT RPAREN LBRACE.
Normalization then throws away what a copier can trivially change. Variable names collapse to placeholders. String and numeric literals become LITERAL. Comments and whitespace vanish. Formatting is irrelevant because the lexer never emitted it in the first place.
Two snippets in normalized form can be compared with a k-gram fingerprint: a sliding window of N tokens, hashed. The Winnowing algorithm from Schleimer, Wilkerson, and Aiken (SIGMOD 2003) is the canonical formulation. Hash every k-gram, then within each window of size w, keep only the minimum hash. The kept hashes become the document's fingerprint. Two documents sharing a fingerprint share a passage. The method survives insertion, deletion, and reordering in ways a line diff does not.
The insight behind fingerprinting isn't that it finds identical strings. It's that it finds identical strings after the copier has changed the surface properties a human reader would notice.
A 2021 survey by Alzahrani and colleagues on source code plagiarism reviewed more than forty tools and found that tokenization plus normalization was the most common successful technique across the board. Nothing exotic here. It's the baseline for a reason.
What token normalization gets you, in practice: from catching roughly 40% of transformed copies with line diffing to something closer to 85 or 90%, depending heavily on the language and how aggressive the rewrites were. Python with heavy list-comprehension chaining and type hints is harder to normalize cleanly than Java.
# Original submitted code
def sum_evens(values):
total = 0
for v in values:
if v % 2 == 0:
total += v
return total
# Token-normalized form the detector sees
FUNC IDENT LPAREN IDENT RPAREN LBRACE
IDENT ASSIGN LITERAL SEMI
FOR IDENT IN IDENT LBRACE
IF IDENT PERCENT LITERAL EQEQ LITERAL LBRACE
IDENT PLUSEQ IDENT SEMI RBRACE RBRACE
RETURN IDENT SEMI RBRACE
Where AST normalization goes further
Abstract syntax trees go one level deeper. A parser turns the token stream into a tree that encodes precedence, nesting, and scope. Two expressions that are lexically identical can produce different trees. Two expressions that are lexically different can produce identical trees.
Consider two versions of the same computation.
# Version A
total = 0
for n in nums:
if n % 2 == 0:
total += n
# Version B
total = sum(n for n in nums if n % 2 == 0)
Token normalization flags these as different. AST normalization sees them as close, because the semantic structure overlaps even though the surface doesn't. A good detector handles both cases. Exact token matches score high, and structural matches that share subgraphs also score above threshold.
The classic approach is subtree hashing. Hash each AST node together with the hashes of its children, producing a canonical fingerprint per subtree. Then compare multisets of subtree hashes. If two submissions share a large number of distinct subtree fingerprints, they share structure. This is the same idea behind Merkle trees and the Deckard system from Jiang et al. (ICSE 2007). Subtree overlap is a strong signal for code that was copied and then restructured.
Where it breaks: DSLs, obscure languages, and macro-heavy code. If no good parser exists for the language, you can't build an AST. This is one reason the supported-language list actually matters when you pick a tool. Codequiry approves about 65 languages and offers a custom parser path for the rest, which is one of the few genuinely differentiating specs in this space.
Layering token and AST matching
In practice you don't pick one method. You stack them and read the differences between them.
The pipeline that works best for both academic submissions and vendor code has three passes. First, lex and normalize into a token stream. Second, parse and compute subtree fingerprints. Third, compare both against a peer corpus and against a web and GitHub corpus. The three passes produce different scores, and the gap between them is the useful signal. If token similarity is 95% and AST similarity is 30%, you're looking at boilerplate-heavy code where two independent authors converged. If both scores sit above 80%, you're looking at either a copy or a shared upstream source.
Running this in a pipeline instead of on a hard drive
Most writing about plagiarism detection lives in the academic world. A professor bulk-uploads a course's submissions, runs MOSS, reads a report. Batch workflow, semester cadence.
That model breaks for enterprises. A fintech repository receives code from full-time employees, contractors, vendors, and open source dependencies. You can't bulk-upload a live repo the way you can a course submission. You need detection at the gate, not at end of quarter.
Three integration patterns I've actually used:
- Pre-merge CI step. On every PR, extract the changed files and run them against a reference corpus of past submissions, vendor drops, and web sources. Fail the check when a score crosses a policy threshold. We use 75% for peer similarity and 60% for web.
- Nightly repo-wide scan. The whole codebase gets re-scanned on a schedule. This catches slow accumulation a diff-level check misses, like ten PRs each adding 100 lines from the same Stack Overflow answer.
- Vendor onboarding scan. Any third-party code drop gets scanned before it enters main. This is the enterprise equivalent of an academic integrity run for a contractor.
For the CI step, it's a REST call or a CLI invocation. On our side it's a shell step in Jenkins that calls the API, parses the JSON, and exits non-zero when the score is above the threshold. Fifteen lines of bash. The interesting failure wasn't the scanning, it was the policy. We set the peer threshold to 85% initially and got zero hits for an entire quarter. Turned out the vendor drops were running through prettier before they reached us, which is a normal formatting step but also a token-shuffling step at the tool level. Once we lowered the threshold and added a raw-source submission path, we found two upstream blocks matching public GitHub repos verbatim.
The academic translation of the same pipeline
Everything above reads the same way in a university course. A professor's peer similarity check is the same token plus AST comparison run against the class cohort. A web check is the same scan against GitHub, Stack Overflow, and question-answer sites. The main difference is scheduling. Academic runs are per-assignment, not per-push.
If you're teaching, the design implication is that the assignment itself determines how much the detector can do. Prompts that require a specific data structure, a fixed output format, and a specific input class give the detector more signal. Prompts that ask for a main method, a Scanner loop, and a class shell will always produce baseline similarity noise regardless of how good the tool is. Token and AST normalization are the wrong lever to pull if an assignment is generating 40% baseline matches everywhere.
Where the scores still lie
Honest caveats, because the tools have limits.
Token and AST normalization both miss genuine independent convergence. Two students who learned from the same course notes will write similar code without ever having seen each other's work. A 70% structural match doesn't prove anything. It flags a review.
Very short submissions are hard. A five-line function has room for maybe ten meaningful k-grams, which is within the noise floor of the algorithm. Below fifteen lines, no tool I've used gives you a reliable number.
AI-generated code sits awkwardly in all of this. A submission produced by Claude or Copilot often has a distinct style, but it produces clean, idiomatic tokens and a shallow, regular AST. It doesn't match another submission because no other submission exists. Peer similarity detection is the wrong tool for the job. That's a separate signal, and one an AI code detector can supply alongside peer and web matching, which is one reason I moved our stack over from a pure similarity scanner.
What to actually configure
If I were setting this up from scratch today:
- Threshold at 60% for peer similarity, 75% for web similarity, and manually review anything in the 40 to 60 band.
- Normalize identifiers and literals, but preserve comments during the first pass so reviewers can see original intent.
- Keep k-gram size at the low end (5 to 8) for short assignments and higher (10 to 15) for production code where boilerplate is common.
- Scan the raw source before any automated formatter runs. This is the single highest-impact workflow decision and most teams get it wrong the first time.
- Log every scan result. The audit trail is what converts a similarity score from a hint into evidence.
If you want a single place to see peer, web, and AI scores on the same submission, that's most of what a source code plagiarism checker is for. Codequiry is the one I've ended up on after cycling through MOSS and JPlag at two jobs, mainly because the REST API and CLI fit a CI pipeline and because peer, web, and AI scores land in the same report instead of three separate ones.
The fastest path if you're setting this up for a course or a team: run one historical batch through the code plagiarism checker before you commit to thresholds, and let the corpus tell you what your baseline actually looks like.