OpenCoderRank: Self-Hosted Coding Assessments

• OpenCoderRank is a self-hosted assessment platform that leverages LLMs for generating and managing time-bound coding and MCQ tests with strict proctoring constraints.
• The system integrates a Flask back-end, SQLite database, and dynamic front-end to deliver randomized questions, enforce time limits, and display live leaderboards.
• It ensures assessment integrity by disabling copy-paste, enforcing full-screen mode, and automating submissions when time expires.

OpenCoderRank is an end-to-end, self-hosted platform for delivering time-bound coding and multiple-choice assessments that balances the twin needs of rapid question creation, aided by LLMs, and assessment integrity, by enforcing proctoring constraints. It is presented as a bridge between problem setters and problem solvers: problem setters can craft or generate assessment content and self-host assessments, while problem solvers can prepare for time constraints and unfamiliar problems in a setting intended to simulate technical assessments. The platform is positioned as a no-cost and customizable solution for technical assessments in resource-constrained environments (Dutta et al., 8 Sep 2025).

1. Architectural decomposition

OpenCoderRank’s architecture divides naturally into four major components, all wired together via a Flask back-end and a lightweight SQLite database (Dutta et al., 8 Sep 2025). Internally, problem setters interact with the “question-generation” and “question-management” GUIs, while problem solvers only see the “assessment-delivery” screens. At the conclusion of any assessment, the scoring engine writes results back to SQLite and displays a live leaderboard.

Component	Elements	Role
Question-Generation Module	Prompt-templating layer, LLM call, parser	Packages user parameters into a single LLM prompt, receives a JSON list of question definitions, validates and stores metadata in the SQLite “questions” table
Assessment-Delivery Module	Front-end served by Flask, per-question timer widget, UI enforcement, random sequencer	Delivers questions, enforces time limits and UI restrictions, and shuffles questions at the start of each assessment session
Scoring Engine	Automated judge, test-harness runner, score calculator, ranking module	Scores MCQs and code submissions, measures pass/fail and may record execution time, computes total score, and orders solvers
Self-Hosting Framework	Two-step setup, optional deployment, open-source release	Runs the entire stack locally and supports deployment to public cloud or tunneling services

The system-level design combines assessment authoring, timed delivery, automated evaluation, and local deployment into a single stack. A plausible implication is that the platform is intended not merely as a question bank, but as a complete assessment environment with both authoring-side and solver-side interfaces.

2. LLM-mediated question generation

OpenCoderRank delegates the heavy lifting of crafting new coding and MCQ problems to LLMs (Dutta et al., 8 Sep 2025). The setter fills in a short form specifying num_questions, topic, level (Easy/Medium/Hard), question_type (mcq/python/sql/java), and output language. The system concatenates these values into a single prompt:

“Generate {num_questions} questions of type {question_type} on topic {topic} at {level} difficulty in {language}. Each question must be a valid JSON object with fields: id (6-digit via generate_id()), challenge_id, title, description, points (5/10/15), time_limit_seconds (60/120/300), plus either options/correct_answer_index (for MCQ), starter_code + test_cases (for Python/Java), or schema + expected_query_output (for SQL).”

This prompt is sent to the LLM API. The LLM’s reply is post-processed into a Python list of dicts, each matching a fixed schema. The paper gives the following representative record:

$q \in Q$5

The platform then inserts these records into the questions table. The core generator is summarized in the paper as:

$q \in Q$6

The key technical point is that question generation is constrained by an explicit JSON schema and parameterized prompt template rather than by unconstrained free-form text generation. This suggests that OpenCoderRank treats LLM output as structured content to be validated and persisted, not as content to be displayed directly without post-processing.

3. Assessment delivery under hard time constraints

When a solver begins an assessment, the server retrieves N random question IDs from the “questions” table, serves them one-by-one or all-at-once in a React-style front-end, and for each question starts a JavaScript timer initialized to question.time_limit_seconds (Dutta et al., 8 Sep 2025). The interface disables paste into the answer editor, whether a textarea or code editor widget, and listens for “time up” events to auto-submit whatever is in the editor as the final answer.

The assessment-delivery module, as described in the architecture, includes a front-end in HTML/CSS/JavaScript served by Flask, a per-question timer widget that counts down from the question’s specified time_limit_seconds, UI enforcement of full-screen mode and disabled copy-paste for the code-entry or answer field, and a random sequencer that shuffles questions at the start of each assessment session.

In this way, each question imposes a hard time limit. The session cannot be extended, and the UI actively prevents switching tabs or copying/pasting. The paper states that this simulates an in-person whiteboard or IDE-in-exam setting. A plausible implication is that temporal control is treated as part of the assessment specification itself, not merely as a scheduling option.

4. Scoring semantics and ranking rules

The scoring engine provides an automated judge for MCQs and for Python, Java, and SQL; it executes submitted code against a suite of test cases, measures pass/fail on each test, and may record execution time per test case (Dutta et al., 8 Sep 2025). The score calculator converts test results, execution times, and question weights into a per-solver total score, and the ranking module orders solvers by descending score, with cumulative runtime used as a tiebreaker.

Although the paper does not publish a fully fleshed-out mathematical model, it states the implied scoring rules as follows. Let $Q$ be the set of questions in an assessment, and for each $q \in Q$ let $p_q$ be its point value and $T_q$ its time limit. For an MCQ, define $pass_q = 1$ if answer_index_submitted = correct_answer_index, else $0$. For a coding question, let

$$pass_q = \frac{\# \text{ of test cases passed}}{\# \text{ of total test cases}}.$$

The solver’s raw score $S$ is

$S = \sum_{q \in Q} p_q \cdot pass_q$

If the solver exceeds $T_q$, a penalty term can be applied, though this is not explicitly formalized in the paper. The example given is

$q \in Q$0

where $q \in Q$1 is a small fixed penalty coefficient.

The ranking is simply all participants sorted descending by $q \in Q$2 or $q \in Q$3, with runtime used as a tiebreaker:

$q \in Q$4

A common misunderstanding would be to treat execution time as the primary optimization target. The description instead places execution time mainly in two roles: it may be recorded per test case, and cumulative runtime is used as a tiebreaker.

5. Integrity safeguards and anti-cheating design

To ensure that LLM assistance or external copying do not trivialize assessments, OpenCoderRank enforces full-screen browser mode at all times, with switching out ending the session; complete disablement of copy and paste in the answer fields; random sequencing of questions per session; and automatic time-out and submission, with no manual overrides (Dutta et al., 8 Sep 2025).

These measures are presented as guardrails rather than as a formally evaluated security model. The paper states that together they make it significantly harder for a solver to “Google” or paste LLM-generated code mid-assessment. This formulation is important: the claim is about making such behavior significantly harder, not about proving impossibility. A plausible implication is that OpenCoderRank adopts practical proctoring constraints aimed at preserving assessment integrity in the LLM era, rather than attempting exhaustive prevention of all external assistance.

The integrity model is therefore closely tied to the delivery model. Full-screen enforcement, disabled copy-paste, random sequencing, and automatic submission are not ancillary features; they are integrated with the timed session workflow and are part of the platform’s central rationale.

6. Implementation, deployment, and empirical status

The implementation stack is specified as follows: backend in Python 3 plus Flask; database in SQLite with zero-admin configuration; and frontend in vanilla HTML/CSS/JS using a code-editor component for Python, Java, and SQL (Dutta et al., 8 Sep 2025). For deployment, pip install -r requirements.txt and python app.py start a local server. The paper also states that the platform can be deployed to any PaaS, including Heroku and Render.com, or exposed via ngrok for remote participants.

Problem setters have full control over which languages are enabled (mcq, python, sql, java), points per question, time limits, hints/remarks, and the LLM prompt template, which is open-source and can be tweaked for difficulty or style. Because the entire codebase is open-source and uses only free dependencies, the paper states that labs or startups in resource-constrained environments can run it at zero cost.

The empirical status of the system is explicitly limited. The paper primarily introduces the design and prototype of OpenCoderRank and does not include a formal user study or quantitative A/B experiment. Instead, it motivates two research questions: how to improve pedagogy in resource-limited settings (RQ1) and how to preserve integrity in the LLM era (RQ2). It then demonstrates through feature descriptions and example workflows that OpenCoderRank addresses both. Future work is expected to measure instructor/developer setup time relative to commercial alternatives, student preparedness and performance on unseen problems, and resistance to LLM-based cheating, such as the incidence of code sharing or paste attempts.

This evaluation profile matters for interpretation. The platform is documented as a coherent integration of LLM-assisted question generation, timed assessment delivery, automated scoring, and proctoring constraints, but not as a system already validated through formal comparative experiments.