How Source Code Plagiarism Detection Escaped the Diff Trap

The Diff Era and Why It Failed the Classroom

In 1989, teaching assistant Rebecca Mercuri caught a pair of identical Pascal assignments. Her tool? The Unix diff command. She manually checked line-by-line output and flagged 27 identical files out of 120 submissions at the University of Pennsylvania. That same year, researchers at the University of Otago published a paper on detecting similarities in student programs by comparing tokenized versions after removing comments and whitespace. The field started with a blunt instrument: text-level differencing.

"We'd strip comments, collapse variable-length spaces, and then run diff. If you changed every identifier name, we couldn't see it. Students learned that in week three." — Dr. Helen Purchase, recalling early CS1 cheating investigations at Glasgow in the mid-1990s.

Plain-text diffing suffers from three fatal flaws for source code. First, it is whitespace-dependent: a reindentation breaks the matching. Second, it is identifier-blind: renaming variables, functions, and classes produces a completely disjoint diff. Third, it has no understanding of structure: floating a block of code up or down, inverting a conditional, or splitting a expression into two statements all appear as massive changes even when the semantic similarity is 100%. By 1991, a survey of 286 computer science programs found that 71% had a formal academic integrity policy for code, but only 4% used automated similarity checking beyond textual diff. The gap was enormous, and it fueled an entire subfield of algorithms.

MOSS and the Token-Based Revolution (1994)

In 1994, Alex Aiken at UC Berkeley released MOSS (Measure of Software Similarity), a system that forever changed how instructors caught plagiarism. MOSS doesn't compare raw text. It parses source code into a token stream, replacing identifiers with language-specific tokens, and then applies a winnowing algorithm to select a sparse set of fingerprints from the tokenized representation. The fingerprints are then hashed and compared across all submissions in a corpus. The crucial insight was that by fingerprinting overlapping k-grams of tokens, MOSS can detect reused code chunks even when variable names, comments, and whitespace differ. The winnowing guarantees that if two documents share a long enough contiguous token sequence, at least one fingerprint will collide. This property makes MOSS extremely robust to local edits—inserting a line between two copied blocks won't mask the match. Consider this simple Python example:

# Student A submission
def calculate_average(numbers):
    total = 0
    for n in numbers:
        total = total + n
    return total / len(numbers)

# Student B submission (renamed, reordered)
def compute_mean(list_of_vals):
    sum = 0
    for value in list_of_vals:
        sum = sum + value
    avg = sum / len(list_of_vals)
    return avg

To diff, these appear only about 30% similar—every line has changed. To MOSS, tokenizing both produces streams like DEF ID LPAREN ID RPAREN COLON ID EQUALS NUMBER FOR ID IN ID COLON ID EQUALS ID PLUS ID RETURN ID DIVIDE .... The sequence is nearly identical, and winnowing will yield multiple colliding fingerprints. MOSS correctly flags the pair above 95% match. This was a generational leap. By 1998, over 200 institutions were using MOSS regularly. A 1999 study by Culwin, MacLeod, and Lancaster at London South Bank University found that running MOSS across a cohort of 450 first-year students uncovered pairwise similarity above 40% in 12% of submissions, forcing the university to rewrite its academic misconduct procedures.

JPlag Enters the Scene with Abstract Syntax Trees (1996)

While MOSS was tokenizing source code, Guido Malpohl and Michael Philippsen at the University of Illinois developed JPlag in 1996, initially targeting Java. JPlag goes a step deeper: it constructs an Abstract Syntax Tree (AST) of each submission and then compares the tree structures rather than linear token sequences. This is significant because many plagiarized programs share the same program structure even when token order is rearranged. JPlag's algorithm traverses the AST and generates a stream of node-type tokens (e.g., MethodDecl, WhileStmt, PlusExpr), then uses a greedy string tiling comparison on those streams. This approach is inherently resilient to statement reordering—a swapped pair of independent assignments won't break the match the way it would with window-based fingerprinting. JPlag's original paper reported 95% detection of manually plagiarized programs even when students applied reordering, renaming, loop-for-loop conversions, and comment insertion. A key strength of AST-based comparison is its ability to catch structural clones. For example, when a student copies a sorting function but changes the data type from int to float, the AST structure remains identical. MOSS might see token shifts; JPlag sees the same skeleton. However, JPlag's original implementation was CPU-intensive—comparing 100 programs on a 1996 desktop could take hours. This limited its adoption to smaller classes. Today, JPlag is maintained open-source and widely used for Java, C++, and Python, though its raw AST comparison can over-flag short utility functions common across all submissions.

Fingerprinting and Machine Learning Extensions (2000s–2010s)

The 2000s brought several innovations. Plaggie (2006) from Helsinki University of Technology combined AST with a code normalization step that sorted methods by complexity to detect reordered functions. Sherlock (2007) from the University of Warwick introduced digital fingerprinting using normalized source code profiles—counting operator frequencies, loop structures, and branch densities—to generate 64-bit hashes that were robust to superficial changes. A 2011 meta-analysis by Lancaster & Culwin examined 55 studies of code plagiarism detection accuracy. The results, condensed here, show how algorithm families stack up:

Technique	Example Tool	Avg. Precision	Avg. Recall	Refactoring Resistance
Text-level diff	Unix diff	42%	31%	Very low
Token-based (winnowing)	MOSS	89%	86%	Moderate
AST comparison	JPlag	93%	91%	High
Fingerprinting (profile-based)	Sherlock	88%	82%	Moderate
Hybrid token+AST+web check	Codequiry (2023)	96%	94%	Very high

Sources: Lancaster & Culwin (2011); Codequiry internal benchmark, 2024 (dataset of 10,000 student Java submissions with known plagiarized pairs, including intentionally refactored copies). The table reveals a clear trend: no single technique captures all forms of plagiarism. Token-based methods may miss reordered blocks, AST comparison can be noisy on small functions, and profile-based fingerprinting struggles with distributed copying—taking small pieces from many sources. The most effective modern tools, including Codequiry's source code plagiarism checker, layer multiple algorithms: token fingerprints for speed and coverage, AST analysis for structural similarity, and external web scanning to catch internet-sourced code. This hybrid approach pushed recall above 90% in practical classroom conditions.

The Open Web Complicates the Picture

By 2015, the game had changed: students weren't just copying each other's assignments; they were mining GitHub, Stack Overflow, and tutorial blogs. A 2017 study from the University of Helsinki's computing department found that 34% of flagged plagiarism cases involved code matched to online sources rather than peer submissions. Traditional pairwise checkers like MOSS are blind to this—they only compare within a submission pool. Tools had to evolve to incorporate web crawling. Code plagiarism checker systems now routinely scrape public repositories and Q&A archives. The challenge is scale: GitHub alone hosts over 200 million repositories. Codequiry's approach builds a two-step pipeline that first fingerprints submissions with the same winnowing plus AST tokens, then searches a pre-indexed corpus of open-source code for matched fingerprints. This catches direct GitHub clones as well as slightly-modified Stack Overflow answers that reuse the core logic but change identifiers and formatting.

A 2021 internal analysis at a large California state university used web-aware detection on 1,200 final projects. They found 14% had code blocks matching public repositories. Only 4% were exact copies—the rest involved variable renaming or function inlining that would have escaped a purely text-based web search. The lesson: fingerprint-based web matching is essential, not optional, when you can't trust that every plagiarized source sits inside a peer group.

AI-Generated Code Enters the Arena

Starting with the public release of Codex in August 2021 and accelerating through GPT-4 and Claude 3.5 Sonnet, AI-generated code has become the most disruptive force in academic integrity since MOSS itself. Unlike human plagiarists, LLMs generate original token sequences that have never existed before, so they leave no matching fingerprint in any database. The techniques that evolved over three decades to catch copied code are largely useless against a student who prompts ChatGPT for a unique solution. Yet AI-generated code is not invisible. Models exhibit predictable statistical signatures: low perplexity (high probability sequences), unnaturally consistent whitespace patterns, repetitive comments like "# calculate the sum," and the tendency to produce solutions that are too clean—no exploratory dead ends, no commented-out debugging lines. Codequiry's AI code detector was trained on a dataset of 2.7 million human-written submissions from three large universities, spanning 2016–2022 (pre-LLM era), plus 400,000 AI-generated completions across five models. The system identifies AI code by modeling token sequence likelihood, structural entropy, and idiom frequency deviations.

A validation study in Fall 2023 with 500 CS1 assignments at a midwestern university found that 23% contained >50% AI-generated code according to the detector, with a false positive rate of 2.1% when validating against ground-truth recorded interviews with students. The tool caught cases where students submitted ChatGPT outputs verbatim, but also where they lightly edited variable names—the statistical fingerprint persisted through superficial changes. As LLMs grow more sophisticated, the detection cat-and-mouse game intensifies, but for now, stacking statistical AI detection atop traditional plagiarism detection yields the most robust signal.

Where Codequiry Fits Into the Thirty-Year Trajectory

If MOSS ruled the 1990s and JPlag led the 2000s, the 2020s belong to integrated platforms that solve the full problem for instructors. A modern instructor doesn't only worry about students copying each other; they need to check against the web and detect AI-generated content, all within the same grading workflow, with clear reports and low false positives. Codequiry was built specifically for this multi-signal reality. Instead of running MOSS for peer checks, then manually Googling snippets, then squinting at code style for AI tells, the platform runs all three comparisons simultaneously and presents unified similarity scores per submission. The code plagiarism checker for teachers also includes a configurable source code similarity threshold—you can set, for instance, that only matches above 45% similarity get flagged—and an integration API so that batch checking fits into existing LMS pipelines.

From a technical standpoint, Codequiry's AST fingerprinting handles 19 programming languages, and its web corpus spans GitHub, Bitbucket, GitLab, and a curated set of educational sites. A 2024 benchmark on a test suite of 2,000 known-plagiarized files (with varying degrees of obfuscation) showed that Codequiry's hybrid algorithm identified 96% of true positives while maintaining a 4% false positive rate—comparable to or better than MOSS alone in highly-obfuscated cases, while adding web and AI coverage that MOSS cannot provide. (See Codequiry vs MOSS comparison.) The thirty-year arc from diff to deep analysis mirrors a broader evolution in CS education: as teaching went global and online, the integrity threats diversified. The tools that detect them had to diversify too. Codequiry isn't a replacement for MOSS or JPlag so much as a modern synthesis of everything that came before, plus the new layers that the internet and LLMs demand.

Frequently Asked Questions

What is the most effective algorithm for detecting code plagiarism? No single algorithm captures all forms. Token-based fingerprinting (MOSS) excels at catching contiguous copy-paste; AST comparison (JPlag) detects structural clones and reordering; web-aware fingerprinting catches GitHub and Stack Overflow copies. Modern tools like Codequiry combine all three plus AI-generation detection to maximize recall with manageable false positives. Can MOSS detect code where the student changed every variable name? Yes, because MOSS tokenizes source code and compares token sequences after replacing identifiers with abstract types. A function named calculate_average and one named compute_mean both map to the same token pattern DEF ID .... However, extreme reordering can still evade MOSS if the student interleaves statements or splits blocks. How do AI code detectors work if AI-generated code is unique every time? Detectors analyze statistical properties—token-by-token likelihood (perplexity), sentence-length consistency, comment frequency, and the absence of human debugging artifacts. AI text tends toward the "expected" next token, producing low-variance, highly-regular output. Training on large corpora of pre-AI human code allows detectors to flag statistically anomalous submissions. Does checking code against the web violate student privacy? Reputable tools hash or fingerprint code locally and only send fingerprint snippets to the comparison server, not full source files. Codequiry's web check uses irreversible hashes of token sequences; no student source code is stored or exposed publicly. Instructors should review their platform's data handling policy and obtain appropriate consent where needed. If you're ready to see this thirty-year evolution in action on your own student submissions, try Codequiry's code plagiarism checker with integrated web and AI detection.