What 2,312 CS1 Python Submissions Revealed About Code Copied

Here is what I can say after running a Python code plagiarism audit across 2,312 CS1 submissions from three bootcamp cohorts. Peer similarity scoring plus starter-code exclusion plus manual diff review caught 41 solid cases that a normal text diff would have missed. The whole workflow fits into a Friday afternoon if your export is clean.

I teach Python at a coding bootcamp that runs four cohorts a year. I also maintain a small open source CLI tool, so I spend too much time looking at other people's code. In fall 2024 I noticed a capstone submission that looked off. The variable names were a little too neat, the comments a little too generic, and two students who supposedly never worked together had the same unusual error handling block. I started digging. By January I had pulled 2,312 files from three cohorts and run them through the same pipeline.

Why Python plagiarism hides from a normal diff

Text plagiarism detectors tokenize prose. Code plagiarism is different. A student can rename every variable, reorder functions, replace a for loop with a list comprehension, and still keep the same underlying structure. Here is the kind of thing I mean.

# Student A
def average(nums):
    total = 0
    for n in nums:
        total += n
    return total / len(nums)

# Student B
def mean(values):
    s = 0
    for v in values:
        s += v
    return s / len(values)

A plain text diff flags almost none of that. A code plagiarism checker that uses token-based comparison and abstract syntax trees will flag the same loop shape, the same accumulator pattern, and the same return expression. Codequiry also fingerprints control flow, so even reordered helper functions do not drop the score the way they drop a line diff.

What I used for the 2,312-file audit

I pulled submissions from Canvas, flattened them with Python 3.11 and pandas 2.2.0, and ran the scans through the Codequiry dashboard as of January 2025. The fall capstone used a Flask 3.0 starter skeleton, which became a separate problem I will get to in step 2. I also ran MOSS locally on one assignment as a sanity check. MOSS agreed with the top 12 pairwise matches, but it did not show web sources, and I did not want to manage three separate result sets.

Step 1 Export every submission and flatten the directory

Canvas gives you a pile of zipped folders with inconsistent names. I wrote a short script to unpack each archive and rename files with the student identifier. This part is not glamorous, but a clean audit starts with a clean archive.

import pathlib
import shutil
import zipfile

archives = pathlib.Path("canvas_export").rglob("*.zip")
out = pathlib.Path("flat_submissions")

for archive in archives:
    student_id = archive.stem.split("_")[0]
    with zipfile.ZipFile(archive) as zf:
        for name in zf.namelist():
            if name.endswith((".py", ".ipynb")):
                dest = out / student_id / name.replace("/", "_")
                dest.parent.mkdir(parents=True, exist_ok=True)
                with zf.open(name) as src, open(dest, "wb") as dst:
                    shutil.copyfileobj(src, dst)

I also dropped .ipynb files from the first run because notebook metadata creates false pairs. Convert them to .py first with nbconvert if you need them. That single step removed 11 near-identical pairs that were just shared Colab notebook output.

Step 2 Exclude starter code before you trust any score

This is the step I got wrong first. The fall cohort capstone included a Flask skeleton with route setup, error handlers, and a database connect function. I uploaded the submissions without marking the starter template. The checker matched every student against the same 80-line skeleton, and my top risk list looked like half the class had copied from one source. They had, but the source was me. After I uploaded the starter code as a template and excluded it, the cohort peer similarity distribution dropped sharply.

Starter code exclusion is not optional. If you skip it, your review queue becomes a list of students who followed your own instructions. If your platform does not let you upload a template, at least add the starter file hash to the exclusion list before you look at scores.

Step 3 Run peer similarity and web source checks together

Create a new check in the dashboard, name it for the course and assignment, select Python under detection language, and upload the flattened folder. I keep peer matching and web matching on for every assignment. AI detection gets its own pass later, because mixing all three at once makes it harder to argue about individual scores with a student.

I use Codequiry for this because it runs peer matching and web source matching in the same check. Most tools do one or the other. Web matching changed the conversation immediately. A student could have low peer similarity because they copied only a function from a GitHub repo. The web check surfaced 14 cases where the same Stack Overflow answer string appeared in three different assignments. Most were fine, students using common idioms, but five included the exact comment from the answer and an unused import.

You do not need a source code plagiarism checker with a CI pipeline to catch most cases. The dashboard plus a CSV export is enough for a single TA. The pipeline version matters only when you scan every assignment every week.

Step 4 Read the distribution, not the individual score

After the scan finishes, I export the peer similarity report and filter out scores below 40. Most of my policy decisions became easier once I saw the distribution.

Peer similarity band	Share of 2,312 submissions	What I did
0 to 39	72.3%	No review unless web or AI signal was high
40 to 59	17.8%	Reviewed clusters of three or more
60 to 79	6.7%	Reviewed every pair
80 to 100	3.2%	Opened every file and wrote a finding

Those bands are specific to my course. They are not universal. In a first Python course, two students independently solving the same warmup exercise often land in the 40s. A 55 on a linked list homework can be natural similarity. An 85 on a capstone almost never is. Thresholds have to be contextual.

I also used a rough DataFrame filter for the review queue. The column names change slightly between exports, so check your file first. The idea is to catch any submission with at least one strong signal.

import pandas as pd

df = pd.read_csv("codequiry_export.csv")
flagged = df[
    (df["peer_score"] >= 60)
    | (df["web_score"] >= 25)
    | (df["ai_score"] >= 80)
]
flagged.sort_values("peer_score", ascending=False).to_csv("review_queue.csv", index=False)

I have not tested these exact numbers past my own cohorts, so treat them as local calibration, not a universal law.

Step 5 Open the side-by-side diff and look for four tells

A high score is a hypothesis. The manual diff is where you confirm it. When I open a pair, I am not looking for identical lines. I am looking for details the students would not both produce independently.

• Comment typos that match exactly, including trailing spaces
• Dead variables assigned but never used in the same places
• The same unusual print statements or debugging leftovers
• Error handling with the same message string, maybe misspelled

One pair in the spring cohort had the same comment reading # handle empty lsit above a try block. The typo, the comment placement, and the order of the except clauses matched. The peer similarity score was 94. No question remained.

Step 6 Stack the AI score as a separate signal

AI-generated Python is a different problem. It does not always look like peer similarity, because no other student may have the same code. The students who use ChatGPT for a warmup exercise tend to produce low peer scores and puzzlingly clean code. I ran the AI code detector on the same flattened submissions after the plagiarism pass.

What I learned is that AI score alone is not enough. I saw AI scores in the 90s on legitimate code that used literate variable names and flat structure. I also saw AI scores below 50 on code that later turned out to be fully generated but then hand-edited by the student. The useful pattern was stacking AI score with peer and web evidence. A submission with a 19 peer score, a 12 web score, and an 88 AI score got a closer look. One score in isolation did not.

I do not report an AI score to a student as proof. I use it as a reason to ask a technical question in a follow-up. That has worked better than any automated verdict.

What changed in how I review student work

I stopped treating a plagiarism check as a one-time event. After the audit, I moved the workflow into the course. Each assignment now gets scanned 48 hours after the deadline, and I use the smart review queue to rank submissions by outlier score. That saves the TA hours each week. The rule is simple: if the peer score passes 70, I do not decide until a human opens the diff. If the web source shows a GitHub repo, I include the URL in the academic integrity report. That paper trail matters.

I also changed the assignments. Not every copied answer is a cheating crisis. Some of my old warmup exercises were so identical that two independent students would produce a 45 peer score without sharing anything. I now add an individual reflection file to each submission, and I require students to sign a two-line comment explaining one design decision. That marker gives the detector something varied to work with and makes the review conversation easier.

For faculty who want a reusable classroom workflow, the code plagiarism checker for teachers setup handles the student roster, assignment grouping, and due dates without you writing scripts. I did not need that at first, but once the audit became routine it saved more time than the export script did.

Frequently Asked Questions

What peer similarity score indicates copied Python code?

In my CS1 course, 80 or higher almost always meant copied code. Between 60 and 79 I reviewed every pair. Scores below 40 rarely got a second look unless web or AI evidence was unusual. Context matters.

Can a Python plagiarism checker catch renamed variables and reordered functions?

Yes, if it compares token sequences and abstract syntax trees rather than raw text. Renaming variables and moving functions may lower the score but not erase the structural fingerprint. A plain text diff will miss many of these cases.

How do you exclude starter code from a plagiarism scan?

Upload the starter files as a template in the same language as the assignment. The checker marks those as shared input and removes them from peer calculations. I forgot this once and the fall cohort looked like a mass copying event.

If you want to run this on your next assignment, start with a code plagiarism checker that gives peer, web, and AI scores in one dashboard.