A TA's Method for Refactoring-Resistant Plagiarism Checks

When a CS TA opens a folder of 300 similarity reports the night before grades are due, the useful question isn't "did these files match?" It's "which of these matches survived renaming, reordering, and method extraction?" Refactoring-resistant plagiarism checks answer that second question. The mechanism is deliberately simple: normalize code into tokens, keep the structure, compare fingerprints. If that sounds abstract, the rest of this walks through how it works, where it breaks, and what a workable triage workflow looks like.

Why line-based diff tools miss refactored copies

A standard line diff computes the minimum edit distance between two files at the line level: insertions, deletions, substitutions. It's what git diff, diff, and most quick checkers use. If a student changes one line in a 200-line file, the diff highlights one line. If they rename every variable, convert a for loop to a while loop, and split a function into a helper, the line diff sees so many insertions and deletions that the similarity percentage collapses.

# original
def average(grades):
    total = 0
    for g in grades:
        total = total + g
    return total / len(grades)

# refactored by a student trying to hide copying
def mean(values):
    def add(a, b):
        return a + b
    acc = 0
    for v in values:
        acc = add(acc, v)
    return float(acc) / values.__len__()

Line diff sees almost no identical lines. A marker who glances at the diff might call it original. But the control flow and data flow are the same: accumulate, divide by length. A token-based checker sees the same skeleton after normalization. A pair like this one produced 18% line similarity in a standard diff and 91% in Codequiry's peer similarity engine when I ran it in spring 2024.

Why refactoring-resistant plagiarism checks start with tokens

Plagiarism detection tools like MOSS, JPlag, and Codequiry don't compare source text directly. They lex the file into tokens: identifiers, keywords, operators, literals, punctuation. Then they canonicalize user-defined identifiers so the surface names stop mattering.

Here's a minimal tokenizer using Python's tokenize module from the standard library. It maps every user-defined name to a fresh idN token, drops comments and whitespace, and keeps operators and literals.

import io
import tokenize

def normalized_tokens(src):
    result = []
    name_map = {}
    counter = 0
    tokens = tokenize.generate_tokens(io.StringIO(src).readline)
    for tok in tokens:
        if tok.type == tokenize.NAME:
            if tok.string not in name_map:
                name_map[tok.string] = f"id{counter}"
                counter += 1
            result.append(name_map[tok.string])
        elif tok.type in (tokenize.NEWLINE, tokenize.INDENT,
                          tokenize.DEDENT, tokenize.COMMENT):
            continue
        else:
            result.append(tok.string)
    return result

After this normalization, average and mean from the example above produce almost the same token sequence. The variable names have been erased. The structure remains. JPlag's 2002 paper by Prechelt, Malpohl, and Philippsen is still the best written account of why this works for student code. The authors found that token-based comparison was robust to identifier renaming and method reordering, two things line-level tools failed on badly. A code plagiarism checker that stops at surface text comparison will miss exactly these cases.

Step two: fingerprints that survive reordering

Token sequence comparison is still order-sensitive. If a student moves method B before method A, a direct n-gram overlap can drop even when the two methods are copied verbatim. The standard fix is local fingerprinting, often called winnowing. Schleimer et al. described it in 2003 for finding copied regions in documents. You slide a window of k consecutive tokens across the sequence, hash each window, and keep the minimum hash in each of a series of wider windows of size w. If two files share a long run of token sequences, they will share a large fraction of these selected hashes even if that run appears in a different position, or if the file has a different number of total functions.

import hashlib

def winnow(tokens, k=5, w=4):
    def h(seq):
        return hashlib.md5("|".join(seq).encode()).hexdigest()
    hashes = [h(tokens[i:i+k]) for i in range(len(tokens) - k + 1)]
    selected = []
    for i in range(len(hashes) - w + 1):
        window = hashes[i:i+w]
        m = min(window)
        if not selected or selected[-1] != m:
            selected.append(m)
    return selected

This is a simplified version. Real implementations use rolling hash functions and avoid duplicate selected hashes. The effect is a compact fingerprint: a few hundred hashes represent thousands of tokens. MOSS's documentation cites winnowing as a core technique. Codequiry's implementation is proprietary, but it follows the same token and fingerprint lineage.

Step three: AST comparison catches structure changes

Tokens miss equivalence when the same operation is expressed with different syntax. A student can replace a loop with sum() or a list comprehension. The token sequence changes dramatically, but the abstract syntax tree for a function that computes a mean still has the same top-level shape: function definition, call, division. AST fingerprints parse the source and compare normalized subtree hashes instead of raw token order.

import ast
import hashlib

def ast_fingerprint(src):
    tree = ast.parse(src)
    for node in ast.walk(tree):
        if isinstance(node, ast.Name):
            node.id = "NAME"
        elif isinstance(node, ast.Constant):
            node.value = "CONST"
    return hashlib.md5(ast.dump(tree, annotate_fields=False).encode()).hexdigest()

AST comparison is not a silver bullet. Two students who independently write the standard mean(xs) using sum(xs) / len(xs) will produce the same AST fingerprint. So will two students who both copy from the same Stack Overflow answer. That's why similarity thresholds and baseline corpora matter more than the fingerprint alone.

A triage workflow that handles 300 reports in an afternoon

Here is the workflow I've used as a TA for a 200-level data structures course. The tool runs after the late deadline. It returns a peer similarity score, a web similarity score, and an AI generation score for each submission. I do not read all 300 reports. I sort by peer similarity, then use a review queue that ranks submissions by how anomalous they are relative to the rest of the cohort.

Peer matches above 90%

The first bin is peer matches above 90% with at least 40 overlapping tokens in the matched region. I open the side-by-side comparison and look for a consistent variable mapping. If the original has current = head.next and the suspicious file has cursor = first.following everywhere, that's stronger evidence than a 92% match on a 12-line boilerplate function.

A code plagiarism checker with a side-by-side view makes this check fast because it aligns the matched regions automatically. I can see whether the rename is systematic or just a few coincidental temp variables.

Web matches

The second bin is web matches. A submission may show only 70% peer similarity because the actual source was a GitHub gist, not another student. Codequiry's web check traces matches back to GitHub and Stack Overflow with line and token counts. When I see a submission with 35% peer and 88% web match to a Stack Overflow answer, I treat it as web plagiarism, not peer collusion.

One semester I forgot to exclude the provided starter file from the peer corpus. Every submission matched the starter's LinkedList.java at 70%, and the dashboard was unusable for an hour until I blanked the baseline. Now I always add starter code and boilerplate templates as excluded checks before importing submissions.

AI flag

Third, I compare the AI code detector score. If a submission has a high AI score and a high web or peer score, that usually means the student generated a solution and then copied it from a shared repo or study group. High AI and low similarity often means a clean LLM-generated answer with no prior peer source.

Common misconceptions about similarity scores

High similarity is not proof. A high score means the token and AST fingerprints overlap far more than chance. It does not establish who copied whom, whether the code was contributed by a third party, or whether the student had permission. It's a referral for human review.

Low similarity is not proof of independence. A student can replace recursion with iteration, split apart every method, and change variable names. A skilled human or a tool that only does token matching may still miss some of these. A low score only means the detector found no strong shared fingerprint.

Similarity scores are triage signals, not verdicts. A human still has to read the matched regions and understand the course policy.

Where the technique breaks down

I've tested this workflow across a few hundred submissions, not a few thousand, so treat the thresholds as starting points. The known failure modes are short assignments, shared starter code, and cross-language copies. Short assignments fail because the number of tokens is too small for statistics. Shared starter code fails unless you exclude the starter files from the corpus. Cross-language copies, say a student translating Java into Python, are much harder because token and AST structures change almost entirely. No detector I've used handles that well, including Codequiry. That's an open research area.

What makes a commercial checker worth the subscription over a local Python script is that these checks need to run against three corpora at once: peer submissions, the open web, and known AI-generation patterns. A source code plagiarism checker that only does one of the three gives you a partial report. Codequiry's peer result combines token, AST, and fingerprinting; its web result maps to GitHub and Stack Overflow; and its AI detector adds a third score on the same dashboard. That's the combination I want at 2 a.m. when grades are due.

The next time you run a similarity report, don't ask "how many students matched each other?" Ask "which matches survived renaming, reordering, and method extraction?" Run the check, blank the starter code, sort by the review queue, open the side-by-side, and then compare web and AI scores. If your current tool only does surface text matching, you can try Codequiry's code plagiarism checker on the next lab and watch which matches it surfaces that the diff missed.