The Long Road to Refactoring-Resistant Code Plagiarism Detection

Refactoring-resistant plagiarism detection is about matching program structure, not surface text. The technique that works in 2025 combines token normalization, AST comparison, and fingerprinting, so a student can rename every variable, swap loops, and reformat the file and the detector still sees the same skeleton. I'll walk through how we got here, then show you a concrete workflow you can run yourself, whether you build a small checker in Python or use a dedicated tool like Codequiry.

Why line diff fails spectacularly

In the early 1990s, the obvious way to check if two student submissions were copied was to run diff on them. If the line-by-line diff showed a high percentage of matching lines, you had a case. This worked for copy-paste plagiarism where a student literally submitted the same file with a new header comment. It fell apart the moment someone changed variable names, added blank lines, or moved a function from the bottom to the top. I used diff on a pair of Python assignments back in 2019 and got a 12% similarity on two files that were obviously the same code, just with every identifier renamed and comments stripped. That was a wake-up call.

The first real improvement was token-based comparison. Instead of comparing raw lines, tools like MOSS (released in 1994) and JPlag (1996) broke the source into lexical tokens: identifiers, keywords, operators, string literals. They normalized whitespace, comments, and sometimes identifier names. So public static void main and int main became token sequences that could be aligned. MOSS uses a winnowing algorithm to select fingerprints from token windows, which is why it scales to thousands of submissions. JPlag originally used token-sequence alignment with dynamic programming. Both were massive steps forward, but they still had a blind spot: reordering independent statements, swapping the order of functions, or changing a for loop into a while loop often broke the token sequence enough to lower similarity below a suspicious threshold.

The refactoring problem, defined

Students who plagiarize rarely submit an exact copy. They run through a mental checklist that looks a lot like an automated refactoring pass in an IDE: rename every local variable, change function names, invert an if/else condition, replace a for loop with a list comprehension, reorder two independent statements. They also do things like change the order of methods inside a class or rename a file. Long before GitHub Copilot existed, this was the standard way to disguise copied code in CS1 and CS2. The detector's job is to see through all that noise and still recognize the underlying algorithmic skeleton.

Structural similarity is the strongest signal of plagiarism because students rarely change control flow when they copy. They rename, reformat, reorder, but the skeleton remains.

That insight drove the next generation of detectors toward AST comparison. An abstract syntax tree represents the grammatical structure of the code independently of formatting, comments, and identifier names. Two pieces of code that parse into the same AST shape, after normalizing names, are almost certainly doing the same thing. Python's ast module makes this easy to play with. Java has javac with -Xprint or libraries like Eclipse JDT. C and C++ have Clang's AST dump. The idea is the same: build a normalized AST dump, then compute a similarity score using tree edit distance or a hashed fingerprint.

Fingerprints that survive reordering

AST comparison handles renaming and reformatting, but it can still be tripped up by reordering functions or statements that don't affect the overall structure but do change the serialized AST string. That's where fingerprinting comes in. Instead of comparing whole-file hashes, you split the normalized token stream or AST into overlapping n-grams, hash each one, and keep a subset of those hashes, usually every k-th hash or the smallest ones. Two files that share many of these fingerprints are likely derived from the same source, even if entire blocks have moved around. MOSS uses winnowing to pick fingerprints from token windows. The technique is well documented, and a student can implement a simple version in 50 lines of Python.

Building a simple refactoring-resistant checker in Python: 5 steps

If you want to understand how these detectors actually work, build a minimal one. It won't replace a production tool, but it will teach you the tradeoffs. I've used this exercise in my bootcamp and with TAs who wanted to stop relying on black-box similarity scores.

1. Normalize the token stream. Strip comments, whitespace, and docstrings. Normalize all identifiers to a single placeholder like IDENT. Keep keywords and operators as-is because they define the control flow. Python's tokenize module does the heavy lifting.
2. Build a structural fingerprint. Use the normalized token list to generate n-gram hashes (I use n=5 for Python). Keep the 20% smallest hashes, or use winnowing. This is the part that survives reordering, because moving a block of code still leaves many local n-grams intact.
3. Optionally compute an AST dump. Normalize all Name nodes to X, function names to F, class names to C. Dump the AST to a string and hash it. This catches deeper structural similarity that token n-grams might miss, like changing a for to a while but keeping the same nesting.
4. Score pairs. Compare tokens with Jaccard similarity or edit distance. Compare fingerprints with set intersection. Compare AST hashes with equality or near-equality. A combined score above 0.75 on any two of these signals is worth reviewing.
5. Set a review threshold. I usually start at 0.7 for token Jaccard, 0.3 for fingerprint overlap, and exact match for normalized AST dump (or 0.9 using tree edit distance). Anything above gets a manual look.

import tokenize, io, ast, hashlib

def normalize_tokens(code):
    tokens = []
    for tok in tokenize.generate_tokens(io.StringIO(code).readline):
        if tok.type in (tokenize.COMMENT, tokenize.NL, tokenize.NEWLINE,
                        tokenize.INDENT, tokenize.DEDENT):
            continue
        if tok.type == tokenize.NAME:
            tokens.append('IDENT')
        elif tok.type == tokenize.STRING:
            tokens.append('STRING')  # strip docstrings and literal text
        else:
            tokens.append(tok.string)
    return tokens

def ngram_fingerprints(tokens, n=5, keep_ratio=0.2):
    hashes = []
    for i in range(len(tokens) - n + 1):
        window = ' '.join(tokens[i:i+n])
        h = hashlib.md5(window.encode()).hexdigest()
        hashes.append(h)
    if not hashes:
        return set()
    # keep the smallest hashes (winnowing would be more robust)
    threshold = sorted(hashes)[int(len(hashes) * keep_ratio)]
    return {h for h in hashes if h <= threshold}

def normalized_ast_dump(code):
    class Renamer(ast.NodeTransformer):
        def visit_Name(self, node):
            return ast.copy_location(ast.Name(id='X', ctx=node.ctx), node)
        def visit_FunctionDef(self, node):
            node.name = 'F'
            self.generic_visit(node)
            return node
        def visit_ClassDef(self, node):
            node.name = 'C'
            self.generic_visit(node)
            return node
    tree = ast.parse(code)
    tree = Renamer().visit(tree)
    return ast.dump(tree, include_attributes=False)

def jaccard(set_a, set_b):
    if not set_a and not set_b:
        return 1.0
    return len(set_a & set_b) / len(set_a | set_b)

def score_pair(code1, code2):
    t1, t2 = normalize_tokens(code1), normalize_tokens(code2)
    f1, f2 = ngram_fingerprints(t1), ngram_fingerprints(t2)
    a1, a2 = normalized_ast_dump(code1), normalized_ast_dump(code2)
    token_sim = jaccard(set(t1), set(t2))
    fingerprint_sim = jaccard(f1, f2)
    ast_sim = 1.0 if a1 == a2 else 0.0
    return {'token_jaccard': token_sim,
            'fingerprint_jaccard': fingerprint_sim,
            'ast_match': ast_sim}

One small gotcha I ran into: I originally didn't strip string literals, and two students who both copied the assignment prompt into a module docstring got flagged as 60% similar even though their actual code was different. Now I always normalize all strings to STRING before hashing.

This simple detector is fine for a bootcamp cohort of 40 people. I haven't tested it on a class larger than a hundred or so submissions, so if you're running a 500-student intro course, you'll want something more robust, or you'll spend all weekend reviewing false positives. That's where a dedicated code plagiarism checker pays for itself.

Running Codequiry for refactoring-resistant detection: 4 steps

Codequiry implements the three layers I just described, token, AST, and fingerprinting, but it also checks against the open web and GitHub, and it has a separate AI code detector for ChatGPT, Copilot, Claude, and Gemini. For a CS professor who doesn't want to maintain Python scripts, the workflow is simpler.

1. Create a course and check. In the Codequiry dashboard, create a course for the assignment, pick the language, and upload the student submissions. You can also upload a baseline file if the assignment provided starter code.
2. Run the peer and web scan. Codequiry compares every pair of submissions using token, AST, and fingerprint matching. It also searches public GitHub and web sources for matching code. The peer similarity score reflects structural similarity, so a refactored copy still scores high.
3. Review the smart queue. The dashboard ranks submissions by cohort outlier score, so a student whose code is structurally similar to three others but whose style is wildly different jumps to the top. You can click into any pair to see a side-by-side diff with matched regions highlighted.
4. Set actionable thresholds. For a CS1 course, I flag peer similarity above 0.75 for manual review. For web matches, anything above 0.50 is worth a look, especially if the source is a public GitHub repo from a previous semester. For AI detection, Codequiry gives an AI score per file; I review anything above 0.80 as likely AI-generated.

The key advantage over a hand-rolled checker is that Codequiry combines peer comparison, web-source matching, and AI detection in one report. If a student copied from Stack Overflow and then asked ChatGPT to refactor the code to avoid peer matching, the three signals together catch it. No single detector is perfect, but stacking them reduces the number of cases that slip through.

Why MOSS and JPlag still matter, and where Codequiry fits

MOSS (Winnowing, token-based) is free and runs on Stanford servers. JPlag (token and AST, open source) has a helpful CLI for batch jobs. Dolos (open source, AST-based, nice web UI) is popular in Europe. They're all solid tools, but they're primarily peer-to-peer detectors. They don't search the web by default, and they don't detect AI-generated code. If a student copies from a GitHub repo from another university that isn't in your peer set, MOSS won't find it. Codequiry explicitly checks web and GitHub sources, and its comparison with MOSS shows where the two approaches diverge. For a professor who wants one dashboard instead of three separate tools, Codequiry's consolidated report is the main draw.

I've used Codequiry for two semesters of a Python bootcamp and for a take-home hiring test at a startup I advise. The false positive rate is low enough that I can review flagged submissions in about 15 minutes per cohort of 30. That's a big deal when you're also teaching, grading, and maintaining an open source library in your spare time.

Frequently Asked Questions

What is refactoring-resistant plagiarism detection?

It's a set of techniques that compare the structural and algorithmic content of source code rather than surface text, so it catches copies even after variable renames, formatting changes, and minor refactoring like loop reordering.

Can AST comparison detect plagiarism between different programming languages?

No. AST comparison is language-specific because each language has its own grammar. Cross-language plagiarism detection is a separate, much harder problem. Codequiry supports 65+ languages by using language-specific parsers, but it does not claim to match Java code to Python code semantically.

How do Codequiry's peer, web, and AI scores work together?

Each submission gets three separate scores: peer similarity (against other students), web similarity (against GitHub and the open internet), and AI generation likelihood. You can set independent thresholds for each. A high peer score with a low AI score suggests traditional copying; a high AI score with low peer and web scores suggests the student generated the code from scratch using an LLM.

Is refactoring-resistant detection foolproof?

No. A determined student who rewrites the algorithm from scratch, changes the data structures, and modifies the control flow substantially will lower similarity below detection thresholds. But that level of effort is often indistinguishable from actually learning the material, which is the goal. The detector is a time-saver for obvious and semi-obvious cases, not an automated honor court.

If you're still using line diff or manual code review for a large intro course, it's time to step up. Grab a code plagiarism checker that understands structure, not just text, and you'll catch the refactored copies that used to sail through.