--- title: Candidate Filtering for Bangla Code Generation url: https://www.emergentmind.com/topics/candidate-filtering-cf type: topic --- # Candidate Filtering for Bangla Code Generation Candidate Filtering (CF) in the context of Bangla-to-code generation refers to systematic procedures for selecting, refining, and validating outputs generated by large language models (LLMs) to ensure strict functional correctness under defined unit tests. While not always explicitly named as “Candidate Filtering,” these selection strategies underpin the most successful models in the BLP-2025 shared task and recent Bangla code-generation frameworks, forming a critical component between LLM inference and final program selection. ## 1. Formal Definition and Motivation Candidate Filtering is defined as the process by which multiple code samples (candidates), produced by a generative model given a natural language instruction (in Bangla), are systematically evaluated and pruned to retain only those meeting predefined standards, typically passing all specified unit tests. In the BLP-2025 task, the formal requirement is that “a Python function implementation […] must satisfy every assert in the test_list” attached to each prompt [2512.19122], [2511.16787], [2511.07382]. Motivation for CF arises from the inherent stochasticity of LLM outputs and the lack of type signatures or behavioral annotation in the Bangla-language prompt: multiple generation passes and subsequent validation shrink the solution space to the semantically correct subset, especially in low-resource and cross-lingual settings where direct generation accuracy is low. ## 2. System Architectures Employing Candidate Filtering Recent high-performing systems for Bangla code generation employ multi-stage agent architectures for effective candidate pruning: - **Two-Agent Pipelines**: NALA_MAINZ [2511.16787] deploys a primary code-generation agent ("G_θ"), which samples candidate solutions, and a secondary debugger agent ("D_ϕ"), which selectively reruns tests and proposes minimal code edits only if initial candidates fail. Only those candidates that pass all asserts after Stage 2 are retained—constituting the output of the CF mechanism. - **Feedback Loops**: Retriv [2511.07382] and BanglaForge [2512.19122] frameworks institute iterative, test-driven refinement. After an initial greedy or low-temperature sample, failures are diagnosed from stack traces and assertion feedback; the model is re-prompted with this feedback over several passes (typically 2–5), producing a new candidate at each step. The process concludes with only the first candidate that passes all test cases being selected. In these designs, the candidate pool is implicitly filtered at each iteration by executable correctness under the provided tests, with additional filtering based on format constraints and error diagnostics. ## 3. Evaluation Metrics and Pass@K Computation The effectiveness of Candidate Filtering is operationalized via the Pass@K suite of metrics, which measure the probability that at least one of K independently sampled candidates passes all unit tests. The precise definitions, as implemented on BLP-2025 and MBPP-Bangla [2509.09101], [2511.07382], [2512.19122], [2511.16787], are as follows: \[ \mathrm{Pass@K} = 1 - \frac{\binom{N_\text{fail}}{K}}{\binom{N}{K}} \] where $N$ is the number of samples drawn, and $N_\text{fail}$ is the number that fail all tests. For $K = 1$ (the leaderboard metric), this simplifies to: \[ \mathrm{Pass@1} = 1 - \frac{N_\text{fail}}{N} \] The pipeline for CF consists of: 1. Generate $N$ code completions per problem. 2. For each, run all original test cases. 3. Discard any sample failing any assertion. 4. For Pass@1, output is “correct” if any remaining sample exists. This filter—based strictly on exact unit-test passage—underpins all result reporting and system tuning. ## 4. Comparative Results and Impact of Candidate Filtering The impact of robust CF is evident in experimental results: | System | Pass@1 (Dev) | Pass@1 (Test) | Notes | |--------------------------------------|--------------|---------------|------------------------------------------------------------| | NALA_MAINZ GPT-5 (Stage 1: gen only) | 64.6% | — | No debugging/filtering | | NALA_MAINZ GPT-5 (Stage 2: +debug) | — | 95.4% | Test-driven CF raises accuracy by +30.8 pp | | Retriv (feedback-guided) | — | 93.4% | Three feedback-guided CF passes | | BanglaForge (dual-model loop) | — | 84.0% | Coder + Reviewer, retrieval-augmented, 5 self-refinement | | Baseline models (single-pass) | ≤69% | ≤40% | High failure rates without CF | | TigerCoder (MBPP-Bangla, Pass@1) | — | 82.0% | Strict Pass@1 with multi-PL references, multi-candidate evaluation | *CF thus delivers 25–40 percentage points gain in test accuracy over single-shot generation without validation, firmly establishing it as a bottleneck technology for LRL code generation* [2509.09101], [2511.16787], [2511.07382], [2512.19122]. ## 5. Methodological Elements and Best Practices Candidate Filtering incorporates several methodological best practices as deployed in state-of-the-art systems: - **Immediate rejection of samples with any syntax or assertion failure** [2511.16787], [2511.07382]. - **Diagnosis-driven revision:** Error traces and failing tests are explicitly included in subsequent prompts to maximize the informativeness of feedback for correcting the candidate [2511.07382], [2512.19122]. - **Iterative minimization:** Only a restricted number of regeneration passes (typically ≤3) are allowed, with early exit upon success (i.e., as soon as $p$ passes all tests, no further candidates are sampled) [2511.07382], [2512.19122]. - **Isolation of code evaluation:** All filtering is conducted in sandboxed environments using identical runtime versions to ensure reproducibility and comparability across systems [2509.09101]. From ablation studies: - Omitting the debugger agent in two-stage pipelines leads to 10–30 points Pass@1 reduction [2511.16787]. - Removing execution-feedback-based iterative filtering in dual-model setups yields up to −25 points penalty [2512.19122]. - Large gains result from combining test- and trace-based filtering with controlled prompt translations and bilingual retrieval augmentation [2512.19122], [2511.07382]. ## 6. Challenges, Limitations, and Future Directions Despite its efficacy, Candidate Filtering faces limitations: - **Reliance on Test Suite Quality:** CF’s ceiling is determined by the completeness and representativeness of the test suite. Hidden/brittle cases or unconstrained instructions may cause overfitting and “specification leakage” [2511.16787]. - **Dependence on proprietary agents:** The best-performing pipelines use closed LLM APIs (e.g., GPT-5), restricting reproducibility [2511.16787]. - **Finite resource constraints:** Iterative CF incurs significant computational overhead, particularly for large K or multiple feedback passes [2511.07382]. Directions for improvement include: - Human-curated translation and denser retrieval for better cross-lingual alignment [2512.19122]. - Systematic expansion of test suite coverage to minimize behavioral underspecification [2511.16787]. - Adaptive or learned self-refinement (e.g., learned stopping, reward shaping) [2512.19122]. ## 7. Context within Bangla Code Generation and Benchmarking Candidate Filtering is central to the construction and evaluation protocols of major Bangla code-generation benchmarks in 2025, including MBPP-Bangla [2509.09101], BLP-2025 Task 2 [2512.19122], [2511.16787], [2511.07382], and mHumanEval-Bangla [2512.19122]. Its core role is to ensure that solution selection is based not on surface features or probabilistic majority, but on strict, test-driven semantic correctness, thus enabling fair model comparison and analysis of cross-lingual reasoning bottlenecks. The methodology codified in these shared tasks and benchmarks is likely to influence best practices in multi-modal and low-resource code generation beyond Bangla. Source: https://www.emergentmind.com/topics/candidate-filtering-cf