In a 400-student Data Structures course last fall, the teaching team at a large public university spent roughly 12 hours per assignment manually sifting through MOSS output, cross-referencing GitHub repositories, and composing awkward emails to suspected plagiarists. By the third project, they had missed two clear cases of ChatGPT-generated code because MOSS flagged nothing — the code was structurally unique, semantically hollow. That's the moment a TAs' Slack channel lit up with the same question: can we just automate this entire pipeline?
The answer is yes. With Codequiry's REST API and a few Python scripts, you can pull submissions directly from Canvas, run both plagiarism and AI-generated code detection, and push annotated results back into your grading workflow. This article is the guide we would have wanted six months ago — complete with working code, real thresholds, and the sharp edges you'll actually hit.
Why Not Just Run MOSS and Move On?
MOSS (Measure of Software Similarity) remains the default tool in most CS departments. It's battle-tested, fast, and free. But the landscape has shifted under it. Modern student plagiarism isn't just about copying from a neighbor — it's web-sourced snippet reassembly (Stack Overflow, GitHub gists, tutorial repos) and LLM-generated code that shares no fingerprint with a known peer submission. MOSS sees token-level overlap, so it misses web-sourced code unless you seed a separate baseline corpus, and it's completely blind to AI-authored functions.
Codequiry addresses this by running three parallel analysis engines on every submission:
- Peer similarity — a combination of token-based fingerprinting, AST comparison across 20+ languages, and an n-gram matching layer that catches refactored clones (renamed variables, reordered blocks, inlined functions). False positive rates sit below 3% when tuned appropriately for class size.
- Web source matching — a continuously crawled index of GitHub, Stack Overflow, tutorial sites, and paste bins. Codequiry's engine breaks a submission into logical units and checks each against the index, returning exact URLs when a match is found. This is the part that catches the student who "borrowed" a red-black tree implementation from a blog post.
- AI-generated code detection — a model trained on the stylistic signatures of LLM output (GPT-4, GPT-4o, Claude 3.5, Gemini, Copilot, Code Llama). The detector analyzes perplexity patterns in token sequences, comment-to-code ratio anomalies, consistent lack of idiomatic mistakes, and structural repetitiveness that differs from human writing. In a 2024 internal evaluation across 12 CS courses, the model achieved 91% recall with a false positive rate of 4%, outperforming text-based detectors that stumble on code.
When you wire all three engines into an automated pipeline, the combination catches significantly more than any single pass. In that Data Structures course we mentioned, adding AI detection surfaced 8 cases out of 400 submissions that peer similarity alone missed — all confirmed by instructor review.
What You'll Need to Build This
We're going to build a Python pipeline that orchestrates the full scanning workflow. Assumptions:
- You have instructor-level access to a Canvas course with an active assignment (any language — Codequiry supports Python, Java, C++, C, JavaScript, and 15+ others).
- You have a Codequiry account with API access (the education plan is common for departments).
- You're comfortable running Python 3.10+ and installing
requests. - You can generate a Canvas API token from Account → Settings → Approved Integrations.
The entire pipeline will live in a single Python script. Here's the high-level flow:
- Poll Canvas for all submissions to a specified assignment.
- Download each student's code files.
- Upload the batch to Codequiry, initiating a similarity scan that includes web-source and AI analysis.
- Poll the scan until it completes, then retrieve per-file results.
- For any submission exceeding configurable thresholds, post a comment to the Canvas submission notifying the grader.
Step 1 — Fetching Submissions From Canvas
Canvas exposes a robust API. We'll use the submissions endpoint with include[]=submission_history to grab the correct file attachments even for multiple attempts. The script below pulls every student's latest submission file(s) into a local folder organised by user ID.
import requests
import os
import json
import time
from pathlib import Path
CANVAS_BASE = "https://your-institution.instructure.com"
CANVAS_TOKEN = "your-token-here"
COURSE_ID = 12345
ASSIGNMENT_ID = 67890
HEADERS = {"Authorization": f"Bearer {CANVAS_TOKEN}"}
DOWNLOAD_DIR = Path("./submissions")
DOWNLOAD_DIR.mkdir(exist_ok=True)
def fetch_submissions():
url = f"{CANVAS_BASE}/api/v1/courses/{COURSE_ID}/assignments/{ASSIGNMENT_ID}/submissions"
params = {"include[]": "submission_history", "per_page": 100}
subs = []
while url:
resp = requests.get(url, headers=HEADERS, params=params)
resp.raise_for_status()
subs.extend(resp.json())
url = resp.links.get('next', {}).get('url')
params = None # subsequent requests use the 'next' URL
return subs
def download_attachments(submission):
user_id = submission['user_id']
# Get the latest attempt with attachments
history = submission.get('submission_history', [])
if not history:
return []
latest = history[-1]
attachments = latest.get('attachments', [])
downloaded = []
for att in attachments:
file_url = att['url']
file_name = att.get('filename', f'file_{att["id"]}')
user_dir = DOWNLOAD_DIR / str(user_id)
user_dir.mkdir(exist_ok=True)
file_path = user_dir / file_name
resp = requests.get(file_url, headers=HEADERS)
resp.raise_for_status()
with open(file_path, 'wb') as f:
f.write(resp.content)
downloaded.append(file_path)
return downloaded
After running this, you'll have a directory tree with one subfolder per student. This code only grabs attachments submitted through Canvas's file uploader; if you're using GitHub Classroom or external tools, you'll need a different retrieval step, but the scanning logic remains identical.
Step 2 — Uploading to Codequiry and Running the Scan
Codequiry's API organises work into assignments. You create an assignment once, then upload all submissions against it. The API key is found in your account dashboard. We'll use the /v2/assignment/create endpoint to set up the scan context, then upload each folder as a separate submission.
CODEQUIRY_API_KEY = "ck_live_..."
CODEQUIRY_BASE = "https://api.codequiry.com/v2"
HEADERS_CQ = {
"apikey": CODEQUIRY_API_KEY,
"Content-Type": "application/json"
}
def create_assignment(name, language="python"):
payload = {
"name": name,
"language": language,
"check_web": True,
"check_ai": True # enable AI detection
}
resp = requests.post(f"{CODEQUIRY_BASE}/assignment/create",
headers=HEADERS_CQ, json=payload)
resp.raise_for_status()
return resp.json()["assignment_id"]
def upload_submission(assign_id, user_id, files):
# Codequiry accepts a zip or individual files; using zip is simpler
import zipfile
zip_path = DOWNLOAD_DIR / f"{user_id}.zip"
with zipfile.ZipFile(zip_path, 'w') as zf:
for f in files:
zf.write(f, arcname=f.name)
with open(zip_path, 'rb') as f:
files_payload = {'file': (str(zip_path), f, 'application/zip')}
data = {'assignment_id': assign_id, 'submission_id': str(user_id)}
# Use multipart upload; the endpoint for submissions
resp = requests.post(f"{CODEQUIRY_BASE}/submission/upload",
headers={"apikey": CODEQUIRY_API_KEY},
files=files_payload, data=data)
resp.raise_for_status()
os.remove(zip_path)
return resp.json()
After all uploads are complete, you start the scan:
def start_scan(assign_id):
resp = requests.post(f"{CODEQUIRY_BASE}/assignment/start/{assign_id}",
headers=HEADERS_CQ)
resp.raise_for_status()
return resp.json()["job_id"]
def poll_scan(job_id, interval=10):
while True:
resp = requests.get(f"{CODEQUIRY_BASE}/job/status/{job_id}",
headers=HEADERS_CQ)
status = resp.json()
if status["state"] == "completed":
return status
elif status["state"] == "failed":
raise RuntimeError("Scan job failed")
time.sleep(interval)
Depending on class size and code length, a scan typically completes in 2–10 minutes. The polling loop respects the API rate limit (60 requests/minute on standard plans).
Step 3 — Retrieving and Interpreting the Results
Once the scan finishes, Codequiry returns per-submission metrics. The key fields:
- overall_similarity — the highest peer match score for that submission, 0–100.
- web_match_count — number of distinct code blocks traced back to online sources, with URLs.
- ai_confidence — a score from 0.0 to 1.0 indicating the model's confidence that the submission is AI-generated.
We'll set thresholds informed by institutional data: overall_similarity ≥ 60, web_match_count ≥ 2, or ai_confidence ≥ 0.85. These are deliberately conservative to keep false positives low.
def fetch_results(assign_id):
resp = requests.get(f"{CODEQUIRY_BASE}/assignment/results/{assign_id}",
headers=HEADERS_CQ)
return resp.json()
def flag_submission(result, sim_thresh=60, web_thresh=2, ai_thresh=0.85):
flags = []
if result["overall_similarity"] >= sim_thresh:
flags.append(f"peer similarity {result['overall_similarity']}%")
if result["web_match_count"] >= web_thresh:
flags.append(f"{result['web_match_count']} web sources")
if result["ai_confidence"] >= ai_thresh:
flags.append(f"AI confidence {result['ai_confidence']:.2f}")
return flags
Codequiry also returns matched snippets and their line ranges. You can embed these directly in the grader comment to save TAs from manual comparison work.
Step 4 — Posting Annotations Back to Canvas
Canvas submissions support a text_comment via the PUT submissions endpoint. We'll post a concise note that contains the flags and a link to the full Codequiry report (each scan gets a web dashboard link).
def post_canvas_comment(user_id, comment_text):
url = f"{CANVAS_BASE}/api/v1/courses/{COURSE_ID}/assignments/{ASSIGNMENT_ID}/submissions/{user_id}"
payload = {
"comment": {
"text_comment": comment_text
}
}
resp = requests.put(url, headers=HEADERS, json=payload)
resp.raise_for_status()
Here's the full main loop that ties everything together:
def main():
print("Fetching submissions from Canvas...")
subs = fetch_submissions()
print(f"Found {len(subs)} submissions")
# Create Codequiry assignment
assign_id = create_assignment("Assignment3_Spring2025", "java")
# Upload each student's code
for sub in subs:
user_id = sub['user_id']
files = download_attachments(sub)
if not files:
continue
print(f"Uploading {user_id} ({len(files)} files)")
upload_submission(assign_id, user_id, files)
print("Starting scan...")
job_id = start_scan(assign_id)
print(f"Job {job_id} running, waiting...")
poll_scan(job_id)
print("Fetching results...")
results = fetch_results(assign_id)
# Flag and comment
for r in results["submissions"]:
uid = r["submission_id"]
flags = flag_submission(r)
if flags:
comment = f"Codequiry flagged: {', '.join(flags)}. Full report: https://app.codequiry.com/report/{assign_id}/{uid}"
print(f"Flagging {uid}: {flags}")
post_canvas_comment(uid, comment)
print("Done.")
if __name__ == "__main__":
main()
Tuning Thresholds for Your Course
The numbers above are starting points, not gospel. In an intro Python course where students write nearly identical 15-line functions, a 60% similarity threshold will fire on every other submission — you'll want to push that higher (75%+) and perhaps add a minimum character count. In a senior-level algorithms course where implementations can legitimately converge on the same optimal solution, you might lower the peer threshold but raise the web-match bar because originality matters more. Run a dry semester with manual review to calibrate before fully automating comments.
For AI detection, the 0.85 confidence threshold catches around 80% of fully AI-generated submissions while falsely flagging about 4% of human-written code. If your institution's policy treats AI as a potential academic integrity case only when combined with other evidence, consider stacking signals — only flag when ai_confidence ≥ 0.85 and overall_similarity < 30% (to avoid double-counting peer-plagiarism cases where the AI detector might also fire on oddly similar boilerplate).
Scheduling Recurring Runs
This script is designed to be run once per assignment after the due date. Set up a cron job (or a scheduled GitHub Action) that executes it at midnight the day after the deadline. Many departments also run a pre-deadline scan a day before the due date, minus the commenting step, to give instructors a heads-up about emerging patterns. Just skip the post_canvas_comment call and dump flagged user IDs to a CSV for a quick manual spot-check.
A workflow we've seen work well: the TA runs the pipeline 24 hours before the deadline, exports the 10 highest-scoring submissions by similarity/AI confidence, and does a 30-minute rapid review. They then post an anonymous announcement in Canvas: "We've noticed a pattern of suspicious submissions. If you've used external sources without attribution, now is the time to revise." It's remarkable how many resubmissions follow.
Handling False Positives and Academic Policy
No automated system replaces human judgment. The pipeline is a triage layer, not a judge. When a student's submission is flagged, the TA should still open the Codequiry report, inspect the side-by-side diff or AI analysis, and make the final call. We recommend including a neutral language template in the Canvas comment: "Your submission has been flagged by our automated integrity check for [reason]. This does not constitute an accusation but requires a manual review. If you have questions, please attend office hours."
Document the process in your syllabus — students respond better when they know exactly what tools are in use and what the escalation path looks like. Institutions that adopt code scanning tools like Codequiry often report a drop in plagiarism cases after the first semester, not because the tool catches everyone, but because the presence of systematic detection shifts norms.
Frequently Asked Questions
Does Codequiry's AI detection work across all programming languages?
Yes. The model operates on token sequences, not language grammar, so it performs consistently on Python, Java, C++, JavaScript, and others. Calibration varies slightly — Python and JavaScript show the highest accuracy due to abundant training data, but all major languages are supported.
Can I run this pipeline with GitHub Classroom instead of Canvas uploads?
Absolutely. Swap the Canvas API calls for GitHub's REST API to clone repositories after the deadline. The Codequiry upload and scanning steps remain unchanged; just point the download logic at your cloned repos.
What if two students legitimately pair-programmed and submitted similar code?
Pair programming is allowed in many courses. You can handle this by having students list partners in a canvas comment or in-file header. Then exclude those pairs from peer similarity flagging by passing exclude_pairs in the Codequiry assignment creation payload. The web and AI checks still apply.
How much does this cost for a department?
Codequiry's education pricing scales by number of students and scans per term. Most CS departments find that the time savings from automated checks alone recoup the cost within a semester — one university reported saving 160 TA-hours per large-enrollment course.
---