Last fall, the instructional team behind CS 1200 — an introductory Python course at a large public university in the Southeast — ran its semester-end integrity audit the way it had for years. They uploaded 1,247 submissions across four assignments to MOSS, skimmed the top similarity clusters, and followed up on the most egregious cases. The results felt familiar: a handful of near-identical pairs, a few students who’d clearly shared work, and the usual uncomfortable conversations. What they didn’t realize was how much they were missing.
Then one TA decided to cross-check a random sample against Chegg using manual searches. Within an hour, she had found verbatim matches for two entire functions that had scored high marks — code that MOSS had flagged with a similarity score of just 14% because the students had renamed variables and restructured control flow. Another TA started pasting suspiciously clean submissions into a GPT detector. By the end of the week, the team had identified 47 submissions with strong signals of either web plagiarism or AI generation that had flown completely under the peer-only radar.
The following semester, the course adopted Codequiry — a platform that runs token-based comparison, AST fingerprinting, web-source checking, and AI detection in a single pass. The results reshaped how they thought about code integrity in a high-enrollment setting.
What a single-tool workflow misses
CS 1200 — Python for non-majors — typically enrolls between 380 and 450 students, split across lecture sections and two dozen lab sections overseen by graduate TAs. Assignments are short: a few functions per week, auto-graded on correctness with unit tests written by the professor. Like many courses of this scale, the integrity policy relied almost entirely on MOSS, the Stanford-authored similarity checker that compares submissions pairwise using winnowed fingerprints.
MOSS is fast, free, and remarkably good at catching copy-paste collusion — but it has well-known blind spots. It only checks against this semester’s submissions. It doesn’t search the web. It doesn’t look at Chegg, GitHub, Stack Overflow, or tutorial sites. And it has no concept of AI-authored code. In a course where solutions proliferate online, peer-only detection is a partial answer at best.
“We were basically applying a burglary alarm to the front door while leaving every window wide open,” the course’s head TA told me. “MOSS caught the kids who copy-pasted each other’s code without changing anything. It had no prayer against a student who grabbed a solution from a public Gist and then refactored it just enough.”
MOSS caught the obvious copy-paste cases. It had no prayer against a student who grabbed a solution from a public Gist and then refactored it just enough.
What the web check surfaced
The team’s first shift was adding an explicit web-source plagiarism check. The assignment that triggered the discovery was a tic-tac-toe board validator: students wrote a function to accept a 3×3 list of lists and return whether the board had a winner. A Chegg search for “Python tic-tac-toe board validator” returned three posted solutions, two of which were nearly verbatim what students had submitted.
On Codequiry’s similarity report for that assignment, the web-match column lit up immediately. The platform’s crawler indexes a wide range of online sources — public GitHub repos, Pastebin, answer sites, tutorial platforms — and surfaces side-by-side comparisons. The report showed that 71 submissions (roughly 18% of the class) had unusually high similarity to known online sources, with 41 of those flagged as “high confidence” copy-paste after a TA review.
Even more valuable than the flag count was the breakdown of which sources students were pulling from. The top hits weren’t random GitHub repos; they clustered heavily on a Chegg post from 2019, a Codementor tutorial, and a Stack Overflow answer with 47 upvotes. That intelligence let the teaching team redesign the assignment for the next offering — not by making it harder, but by varying the input dimensions and requiring a specific edge-case handling that none of the public solutions accounted for.
Adding the AI detection dimension
By spring semester, the team had a new problem: submissions that were grammatically perfect, used sophisticated language model idioms, and yet felt oddly sterile. These weren’t web copies; they were almost certainly LLM-generated.
Codequiry’s AI detection module, released after the explosion of ChatGPT and GitHub Copilot, analyzes submissions across multiple statistical signals. For code, it looks at token distribution, comment-to-code ratio, variable naming entropy, and structural patterns that differ between human-written and machine-generated code. An LLM like Claude or GPT-4 tends to produce unnervingly consistent spacing, predictable comment phrasing (lots of “Helper function to calculate…”), and error-handling patterns that feel templated even when they’re syntactically correct.
# Flagged as likely AI-generated: overly explained, unnaturally consistent
def calculate_winner(board):
"""
Helper function to determine the winner of the tic-tac-toe game.
This function checks all possible winning combinations on a 3x3 board
and returns the symbol of the winning player, or None if no winner exists.
"""
# Check rows for a win
for row in board:
if row[0] == row[1] == row[2] and row[0] is not None:
return row[0]
# Check columns for a win
for col in range(3):
if board[0][col] == board[1][col] == board[2][col] and board[0][col] is not None:
return board[0][col]
# Check diagonals for a win
if board[0][0] == board[1][1] == board[2][2] and board[0][0] is not None:
return board[0][0]
if board[0][2] == board[1][1] == board[2][0] and board[0][2] is not None:
return board[0][2]
return NoneThe code above was flagged by Codequiry’s model. The student who submitted it — a sophomore with a B-minus average — had never written a comment that long in any previous assignment. The variable names board, row, col exactly matched the assignment spec, as did the function signature, but the internal logic was unnervingly “textbook” in a way that human first-draft code rarely is. The AI score: 94.7%.
Across the semester, 12.3% of submissions scored above the 80% AI-confidence threshold on at least one assignment. That didn’t mean all were definitively AI-written — the teaching team manually reviewed every high-confidence flag and escalated only clear-cut cases — but the pattern was unmistakable. Notably, more than half of the AI-flagged submissions showed no significant peer similarity and no web matches. They were fresh, unique, and synthetic in a way that none of the existing tools would have caught.
Token fingerprinting versus surface reformatting
One persistent challenge in large courses is that students know enough about how similarity checkers work to attempt evasion. They rename variables from i to idx, swap for loops for while equivalents, and add or remove whitespace. Basic text diffing and even string-based plagiarism checkers get thrown by these changes.
Codequiry’s core comparison engine operates at the token and AST level. It lexes each submission into a token stream, normalizes identifiers, and then compares token sequences using n-gram overlaps. For C-family languages and Python, it also extracts AST fingerprints — hashes of normalized tree substructures — so that semantically identical code with different formatting or variable names still generates a high similarity score.
A concrete illustration: consider two student submissions for a function that computes the factorial of n. One writes a straightforward iterative loop:
def factorial(n):
result = 1
for i in range(1, n+1):
result = result * i
return resultThe other changes variable names and inverts the loop direction:
def get_fact(x):
ans = 1
for num in range(x, 0, -1):
ans *= num
return ansTo a text diff, these are markedly different. Codequiry’s token-based analysis, however, strips identifiers and normalizes loop patterns; the resulting token sequences share enough high-weight n-grams to produce a similarity score in the 78–85% range. Pair that with the fact that both submissions came from the same lab section and the resemblance is impossible to ignore. A source code plagiarism checker that works purely on character-level text matching would likely miss this.
How the instructional workflow actually runs
In practice, the course’s workflow looks like this:
- Submission deadline. Students submit via the existing LMS (Canvas, with Gradescope for auto-grading).
- Batch export. The head TA exports all .py files per assignment and uploads them through Codequiry’s web dashboard or via the REST API for automated pipelines.
- Run analysis. Codequiry processes the batch — roughly 400 files — in under two minutes, simultaneously generating peer similarity scores, web-source matches, and AI-generation probabilities.
- Triage. Two TAs spend about 90 minutes reviewing the flagged results. High-confidence web matches and AI flags get side-by-side comparisons; moderate peer-similarity clusters are spot-checked.
- Escalate. Clear-cut cases (verbatim Chegg code, AI-generated submissions with corroborating metadata like static comment patterns) go to the professor for the formal academic integrity process. Borderline cases receive a warning and the offer of an oral code walkthrough.
“The dashboard literally cut our integrity-review time by 60%,” the head TA said. “Instead of blindly spot-checking, we’re looking at a ranked report and spending our time on the 10% that matter. That’s huge when you’re a TA who also has 20 hours of grading and office hours.”
Where MOSS, JPlag, and general-purpose tools fall short
Course teams often default to MOSS because it’s free and familiar, and JPlag is popular in Europe for Java-focused courses. Both are solid peer-to-peer fingerprinting engines, but neither scans the web, and neither detects AI-generated code. Turnitin, which many universities already license for essay plagiarism, has a “code similarity” mode that performs basic text matching and is widely regarded as inadequate for source code — it doesn’t parse programming-language structure, so variable renaming and whitespace changes break its matching entirely.
The comparison table that drove CS 1200’s switch looked roughly like this:
| Capability | MOSS | JPlag | Turnitin | Codequiry |
|---|---|---|---|---|
| Peer code similarity | Yes | Yes | Partial | Yes (token + AST) |
| Web-source matching | No | No | No | Yes |
| AI-generated code detection | No | No | No | Yes |
| Survives renaming/refactoring | Moderate | Good | Poor | Good |
| Web dashboard + reports | No | CLI only | Yes | Yes |
| API for pipeline integration | No | Partial | No | Yes |
For a team already drowning in logistics, having a single code plagiarism checker that covered peer, web, and AI detection — with a clean web UI for TAs and a documented API for potential CI integration — was the deciding factor. Read more about how it compares to MOSS specifically on our Codequiry vs MOSS page.
The redesign effect
The most interesting long-term outcome wasn’t the number of catches; it was how the detection data informed assignment design. After two semesters of data, the professor could see exactly which assignments attracted the most web plagiarism (the tic-tac-toe validator topped the list) and which ones triggered the most AI flags (anything that could be phrased as a self-contained, spec-driven function — string formatters, palindrome checkers, simple recursive traversals).
The team now designs assignments with these lessons in mind. Instead of “Write a function to check if a string is a palindrome,” students get a partially implemented class with a specific internal representation and must fill in three method stubs whose behavior is tightly coupled to that representation. The spec isn’t easily pasteable into ChatGPT, and even if a student tries, the LLM’s output rarely respects the imposed constraints. The result: AI-flag rates dropped from 12% to 4.5% without making the course harder.
Codequiry’s continued scanning also helps close the loop. When the team experiments with a new assignment format, they run the previous semester’s submissions for comparison and get a clear signal on whether the design change reduced external copying.
Frequently Asked Questions
How does Codequiry detect AI-generated Python code?
It analyzes token probability distributions, comment consistency, variable-naming entropy, and structural patterns that distinguish machine-generated code from human-written work. The model is tuned on a large corpus of both human and LLM-authored code across multiple languages, and it provides a confidence score for each submission so instructors can focus manual review where it matters most. See the AI code detector for details.
Can Codequiry find code copied from Chegg or Stack Overflow?
Yes. Its web-check crawler indexes public code repositories, answer sites, tutorial platforms, and paste sites. When a submission matches a known online source, the report shows side-by-side comparisons with the original URL, similarity percentage, and highlighted overlap regions.
What's the false positive rate like for the AI detection module?
In the CS 1200 deployment described here, the teaching team reviewed every AI flag above 80% confidence and found that roughly 85% of those cases were, after closer inspection, either definitively AI-generated or contained large AI-authored blocks. The remaining 15% were ambiguous — typically students who wrote unusually clean, well-commented code that the model found suspicious but the TA could reasonably attribute to strong human work.
Can I integrate Codequiry with our existing LMS or autograder pipeline?
Yes. Codequiry provides a REST API that lets you submit batches programmatically and retrieve results. Several large courses use it to trigger a plagiarism and AI scan automatically after the Gradescope autograder finishes, so TAs see a unified view without manual file wrangling.
For course teams trying to move beyond single-dimensional integrity checks, Codequiry’s code plagiarism checker combines web, peer, and AI detection in a workflow that won’t swamp your TAs.