A TA's Script for Sorting 300 Code Similarity Reports by Office Hours

In October 2023, Maya Chen, head TA for UC San Diego's CSE 11, a 412-student introduction to Java, faced the weekly assignment 3 similarity report. Codequiry had generated 312 code similarity reports with at least one peer match above the course's 40 percent review threshold. Office hours began at 2 p.m. By 12:50 p.m., she had reduced the queue to 14 files worth opening. The method was not a better plagiarism detector. It was a better sorting order.

The core workflow is a three-pass triage. Rank submissions by Codequiry's cohort outlier score, not raw similarity percentage. Review only the top 10 to 15 percent by risk. Then open side-by-side diffs only when the automated score is supported by a suspicious pattern.

Most TAs do the opposite. They sort the report alphabetically or by submission time and click through every flag. That is how a 300-report queue consumes an entire weekend. Chen's approach treats similarity as a ranking problem.

"Codequiry's peer score is not a verdict," Chen said. "It is a sorting mechanism. The actual determination still happens in the diff viewer."

Prioritize Code Similarity Reports by Outlier Score, Not Raw Percentage

Raw match percentage is noisy. Two students who submit the exact same 20-line helper method often show a 100 percent match, but if both copied from the same lecture example, that is not misconduct. A student who renames every variable and reorders methods may only reach 38 percent while still copying the structure. Codequiry's code plagiarism checker for teachers addresses this with a cohort outlier score that weights token, AST, and fingerprint similarity against the distribution of all other submissions.

At UC San Diego, the course policy sets a review threshold of 40 percent peer similarity. But Chen does not start at 40. She starts at the top of the outlier queue.

1. Open the assignment insights page and switch the sort order from "Highest similarity" to "Review first."
2. Ignore every flag below 40 percent unless the outlier score is above 85. The outlier score is the stronger signal for weekly labs.
3. Open the top 10 percent of the queue in the evidence review. For 300 submissions, that is 30 files.
4. Tag each as Clear, Needs Review, or Escalate. Do not read line by line yet.
5. Run the Python script below against the exported CSV to regroup by student pairs, because one student who copied from two different classmates will otherwise appear twice in the queue.

One setting is easy to miss. When creating the check, the "Include web sources" box is not selected by default in the course template Chen inherited. Without it, the peer queue can look clean while Stack Overflow and GitHub copies do not appear. Chen toggles it on before every weekly lab.

Export the Report and Regroup by Student Pairs

The dashboard is good for a quick scan. The CSV export is better for repeatable triage. Chen exports the assignment insights CSV and runs a small script on Python 3.11 with pandas 2.1.4. The goal is to collapse duplicate flags and rank authors by the highest outlier signal across all their matches.

import pandas as pd

# Export from Codequiry assignment insights as CSV.
# Columns: submission_id, author, peer_score, web_score, ai_score, matched_to, outlier_score
df = pd.read_csv("assignment3_similarity_report.csv", parse_dates=["submitted_at"])

# Keep only pairs above the 40 percent peer threshold.
above_threshold = df[df["peer_score"] >= 0.40]

# Rank by outlier score first, then peer score.
ranked = above_threshold.sort_values(["outlier_score", "peer_score"], ascending=[False, False])

# Group by author to catch students copied from multiple peers.
grouped = ranked.groupby("author").agg(
    flags=("matched_to", "count"),
    max_peer=("peer_score", "max"),
    max_outlier=("outlier_score", "max"),
).sort_values("max_outlier", ascending=False)

print(grouped.head(30).to_string())

The grouped table collapses duplicate flags. In one week, a student appeared as a 94 percent match to one peer and a 61 percent match to another. The raw queue showed two rows; the script showed one author with two flags, which moved that student into the Escalate tag before office hours.

Read the Side-by-Side Diff Only When Three Signals Agree

The second pass is where most false positives die. For each of the remaining 14 files in Chen's queue, she opens the side-by-side diff and looks for three signals in order. A high outlier score alone is not enough. A high peer percentage alone is not enough. The evidence review has to connect the two.

1. Token pattern: Are the same unusual variable names, string literals, or error messages preserved across submissions?
2. AST structure: Are the method boundaries, loop nesting, and conditional branches identical even after renaming?
3. Web source: Does a domain match to Stack Overflow, GitHub, or a tutorial explain the overlap before peer copying is assumed?

Only after two of those three signals agree does Chen open the linked submissions in the evidence review. If the peer score is high but the token pattern looks like standard textbook code, she marks Clear. If the AST structure matches but the web match points to a single tutorial that both students cited, she still reviews but lowers the severity.

"The 40 percent number is arbitrary," said Carlos Mendez, a head TA at Georgia Tech who ran a similar workflow for a 200-student data structures course. "MOSS gives you raw similarity, but for weekly labs, the distribution matters more. A student who sits two standard deviations above the class on AST overlap gets reviewed even if the raw percentage is only 35."

Mendez still runs MOSS on final projects because it is free and familiar to honor committees. For weekly labs, he uses Codequiry because the web-source check saves a separate plagiarism search and the side-by-side diff is already labeled with match sources. That matches Chen's experience at UCSD, where the detect code plagiarism workflow has to produce evidence a professor can act on within a day, not a raw similarity score that requires an hour of manual interpretation per file.

Handle Small Sections Differently

Not every TA agrees on the outlier threshold. Priya Raman, a TA at the University of Washington, found Codequiry's outlier queue less useful for smaller sections, where a 12-student class often has no meaningful distribution. In her section, a pair at 78 percent peer similarity and 0 web matches was still reviewed manually because the sample size was too small to trust the outlier math.

For sections under 25 students, Raman sorts by peer score first and treats any match above 65 percent as reviewable. She also opens the web-source panel on every flagged pair, because in small classes the most common false positive is two students independently finding the same Stack Overflow answer. The workflow changes, but the principle stays the same: use the automated score to reduce the number of files a human must open, and use the diff to make the final call.

What the Two Hours Actually Look Like

Chen's Friday triage follows a tight schedule. The first 20 minutes are spent checking the assignment settings, toggling the web-source box, and exporting the CSV. The next 30 minutes run the Python script and sort the top 30 authors into Clear, Needs Review, and Escalate tags. The next hour is the second pass: for the 14 files tagged Needs Review or Escalate, she opens the evidence review and reads the side-by-side diff. The final 10 minutes are for writing up three to five concise honor code referrals with the specific token and AST evidence linked.

The output is not a perfect academic judgment. It is a defensible shortlist. Some semesters the rollout slips because a new TA forgets the web-source toggle. Some weeks the outlier queue is dominated by a single heavily copied lab, and the second pass takes longer. But the core pattern has held across three offerings of the course.

Frequently Asked Questions

What is the fastest way to triage a large code similarity report?

Start with Codequiry's "Review first" queue, which uses cohort outlier scoring rather than raw similarity. For a 300-submission assignment, TAs often close the review queue after inspecting the top 10 to 15 percent.

Does a high peer similarity score always mean plagiarism?

No. Lecture starter code, standard library calls, and short helper methods can produce high matches without misconduct. TAs should combine the outlier score, the side-by-side diff, and the web-source matches before deciding.

Can this workflow be automated?

Yes. Export the Codequiry report as CSV and use a short Python script to rank students by outlier score and group duplicate flags by author. The script above works with pandas 2.1.4.

For TAs and instructors who want the outlier queue and web-source matching without maintaining a MOSS server, Codequiry's code plagiarism checker offers the same two-click setup Chen uses each week.