LLM-FS-Agent: Transparent Feature Selection
- LLM-FS-Agent is a deliberative, role-based feature selection framework that employs four specialized agents to provide transparent and auditable decision-making.
- The system combines semantic reasoning with quantitative metadata to enhance efficiency and interpretability in high-dimensional cybersecurity contexts.
- Empirical validation on IoT intrusion detection shows significant training time reduction and reliable explanation chains compared to traditional methods.
LLM-FS-Agent is a deliberative, role-based LLM architecture for transparent feature selection under high-dimensional data, introduced to address the interpretability and auditability deficits of both traditional feature-selection methods and single-LLM selectors. In the reported formulation, feature ranking is produced through a structured debate among four specialized agents—Initiator, Refiner, Challenger, and Judge—so that each inclusion or exclusion decision is accompanied by a textual rationale rather than only a scalar score. The system is evaluated in cybersecurity, specifically IoT intrusion detection on CIC-DIAD 2024, where it is compared with LLM-Select and PCA across multiple LLM backbones, feature subset sizes, and downstream classifiers (Bal-Ghaoui et al., 7 Oct 2025).
1. Concept and problem setting
LLM-FS-Agent is defined around feature selection in high-dimensional data, where large numbers of variables can slow learning pipelines, reduce interpretability, and increase overfitting risk. The underlying motivation is the curse of dimensionality: as feature count grows, computational cost rises and model generalization can deteriorate. Within that setting, the method is explicitly motivated by three objectives: preserving interpretability by keeping original semantic features, improving efficiency by reducing downstream training and inference cost, and improving predictive performance by removing noisy, redundant, or weakly relevant variables (Bal-Ghaoui et al., 7 Oct 2025).
The architecture is also positioned against the usual families of feature selection—filter, wrapper, and embedded methods. Those methods are described as often operating like black boxes: they can rank or eliminate features, but usually do not provide rich, human-readable explanations for why a feature should be retained or discarded. The paper treats this as particularly consequential in cybersecurity, where analysts care whether a selected variable is genuinely meaningful or merely spuriously correlated.
Within LLM-based feature selection, the paper distinguishes text-based feature selection, which relies on feature names and task descriptions, from data-driven feature selection, which provides samples or statistics. It argues that prior LLM-based methods, especially single-LLM approaches such as LLM-Select, often lack structured reasoning, do not expose a transparent deliberation trace, can be inconsistent across backbones or prompts, offer limited justification for inclusion or exclusion, and do not explicitly model critique around redundancy, weak signals, collinearity, or adversarial misuse (Bal-Ghaoui et al., 7 Oct 2025).
A central implication is that LLM-FS-Agent is not proposed merely as another ranking mechanism. It is proposed as a feature-selection workflow that is meant to be transparent, auditable, and robust in addition to being competitive in downstream performance.
2. Deliberative role-based architecture
The core design is a sequential, four-role deliberation in which each feature is assessed through proposal, refinement, critique, and adjudication. Rather than asking one LLM for a direct ranking, the system decomposes the decision process into specialized roles (Bal-Ghaoui et al., 7 Oct 2025).
| Role | Function |
|---|---|
| Initiator | First-pass semantic relevance assessment |
| Refiner | Adds supporting arguments and quantitative metadata |
| Challenger | Critiques weaknesses, redundancy, bias, and misleading interpretations |
| Judge | Synthesizes prior arguments and assigns final importance score |
The Initiator Agent performs the initial semantic assessment of each feature relative to the task. Its operative question is whether a feature name appears relevant given the prediction objective. It produces a preliminary relevance judgment and an initial argument.
The Refiner Agent strengthens that argument by adding supporting context. The paper states that this stage may include metadata such as the mean and standard deviation of the feature–target correlation, so the architecture is not purely semantic. The Refiner is the stage where limited statistical context is explicitly injected into the reasoning.
The Challenger Agent acts as a critic in a peer-review-like stage. It examines prior reasoning for weaknesses and attempts to surface redundancy, bias, false positives, and possible adversarial or misleading interpretations. This is one of the paper’s most distinctive mechanisms, because critique is treated as intrinsic to feature selection rather than as an after-the-fact explanation layer.
The Judge Agent is the final arbiter. It synthesizes the supportive and adversarial arguments and assigns the final importance score. The paper defines that score as a weighted combination of refined and challenged scores:
The extracted text does not report numerical values for and , so the precise weighting policy remains unspecified.
Operationally, the interaction pattern is sequential: the input consists of feature names and a textual task description; the Initiator produces initial analyses; the Refiner elaborates them; the Challenger critiques them; and the Judge produces the final score and rationale. The paper repeatedly characterizes this as a structured debate rather than simple score aggregation (Bal-Ghaoui et al., 7 Oct 2025).
3. End-to-end methodology and evaluation pipeline
The reported experimental pipeline begins with the CIC-DIAD 2024 IoT intrusion detection dataset. The preprocessing steps are explicit: the workflow starts from 84 features, focuses on three classes—Benign, Brute Force, and Mirai—cleans the data, removes highly collinear features with a Pearson correlation threshold greater than $0.9$, standardizes remaining numerical features with StandardScaler to mean $0$ and standard deviation $1$, and applies random undersampling to balance the classes. After preprocessing, the class counts are Benign: 3,619, Mirai: 5,170, and BruteForce: 3,619, for a total of 12,408 samples (Bal-Ghaoui et al., 7 Oct 2025).
The multi-agent system then receives two inputs: feature names and a textual description of the prediction task. The paper states that exact prompts for all four agents are provided in figures, but the extracted text does not reproduce them. What is recoverable is their functional partition: semantic relevance assessment for the Initiator, strengthened analysis plus quantitative metadata for the Refiner, critical review for the Challenger, and final arbitration for the Judge.
For each feature, the system records an initial argument, a refined argument with supporting evidence, a critique, and a final verdict with score. Features are then ranked by their final scores, and the ranked list is truncated into six subset sizes:
Each top- subset is evaluated with four downstream classifiers: Random Forest, XGBoost, Support Vector Classifier, and Logistic Regression. The primary performance metrics are Accuracy and AUC, while training time and inference time are additionally measured for XGBoost. The comparison is performed against LLM-Select and PCA, using the same LLM backbone, the same subset sizes, and the same downstream metrics (Bal-Ghaoui et al., 7 Oct 2025).
The reported LLM backbones, all locally deployed through Ollama, are Llama 3.2 (2B), Gemma_3 (4B), Qwen (4B), Phi-3 Mini (3B), and Mistral (4B). The paper also contains a noted inconsistency: one discussion passage says that experiments used a single LLM architecture across all agent roles, specifically Llama3.2, whereas the reported tables compare several backbones overall. The most plausible reading given the reported tables is that each run used a single backbone consistently across roles, but different runs used different backbones.
Several implementation details are absent from the extracted text: there is no explicit pseudocode, no multi-round debate beyond the single Initiator → Refiner → Challenger → Judge sequence, no exact scoring scale for all agents, no extractable full prompt text, no explicit train/validation/test split protocol, and no classifier hyperparameters. The methodology is therefore architecturally clear but partially under-specified (Bal-Ghaoui et al., 7 Oct 2025).
4. Transparency, interpretability, and domain reasoning
Transparency is the stated centerpiece of LLM-FS-Agent. For every feature, the system preserves what the paper calls a record of the feature selection decision-making process. That record includes the Initiator’s rationale, the Refiner’s supporting explanation, the Challenger’s counter-argument, and the Judge’s final explanation and score. Feature selection is therefore represented as a textual argument chain rather than merely a ranked numeric list (Bal-Ghaoui et al., 7 Oct 2025).
This design is contrasted with two baselines. Traditional methods such as PCA provide no semantic rationale, because they project data into latent dimensions rather than preserving interpretable original variables. Single-agent LLM selectors provide some reasoning, but the paper argues that they lack an internal critique stage and therefore do not expose a peer-review-like deliberation trace. LLM-FS-Agent’s main interpretability claim is not that it discovers wholly different features in every case, but that it makes the decision process inspectable and therefore more auditable.
The paper’s qualitative discussion of Src Port and Dst Port illustrates this interpretability claim. These features receive sharply different scores from different backbones: Qwen assigns 1.0 to both, Gemma assigns 0.0 to both, Phi3-mini gives intermediate values such as 0.65 and 0.45, and Mistral gives 0.1 and 0.0. The paper explains why these features are contentious in intrusion detection: source and destination ports may be weak indicators because they can be spoofed, or may correlate with benign traffic patterns, making them unreliable as standalone attack indicators. The Judge’s reported reasoning is described as reflecting a security-aware assessment of this vulnerability to spoofing and contextual unreliability (Bal-Ghaoui et al., 7 Oct 2025).
A plausible implication is that the Challenger stage is especially valuable in domains where apparently relevant metadata may also be easy to manipulate. The paper itself frames this as improvement in auditability, because one can inspect not only the final ranking but also the reasons a feature survived criticism.
5. Empirical findings
The main empirical claim is deliberately qualified: LLM-FS-Agent is superior or comparable to baselines overall, but it does not uniformly dominate LLM-Select across all backbones and subset sizes. The paper identifies as the strongest practical subset size and describes it as near-optimal. It also states that at LLM-FS-Agent is “the only method to consistently achieve the highest results across multiple classifier and metric combinations” (Bal-Ghaoui et al., 7 Oct 2025).
On the reported 0 comparison against LLM-Select, the average gains are small. Average AUC improves from 0.9751 / 0.9652 for LLM-Select XGBoost / Random Forest to 0.9738 / 0.9640 for LLM-FS-Agent XGBoost / Random Forest, with an overall reported 1. Average accuracy moves from 0.8902 / 0.8764 to 0.8881 / 0.8769, with 2. The paper’s interpretation is that average gains are small, but deliberation seems to have a regularizing effect when the single-agent baseline is unstable or suboptimal. It highlights especially noticeable gains with Qwen and Gemma, and slight underperformance with already strong single-agent cases such as Mistral and Phi3-mini.
Across subset sizes, the reported narrative is similarly nuanced. LLM-Select is competitive at 3, and outperforms at 4 in some cases. LLM-FS-Agent shows a clearer advantage at 5. The paper also states that LLM-based feature selection is stronger than PCA overall, and that XGBoost and Random Forest are the strongest downstream models.
The strongest statistically supported result concerns efficiency rather than mean predictive quality. For XGBoost, average training time drops from 0.205 s with LLM-Select to 0.111 s with LLM-FS-Agent, corresponding to an average 1.87× train speedup and a reduction of about 46%. The abstract states that this reduction is statistically significant, with a mean difference of 0.094 s and 6 for XGBoost training time (Bal-Ghaoui et al., 7 Oct 2025).
| Metric | Mean difference | Interpretation |
|---|---|---|
| AUC | +0.0010 | Not statistically significant |
| Accuracy | +0.0017 | Not statistically significant |
| Training Time | -0.094 s | Statistically significant, large effect |
| Inference Time | -0.0001 s | Not statistically significant |
The accompanying statistical table reports Cohen’s 7 for training time, interpreted as a large effect size, while AUC, accuracy, and inference-time differences are all labeled small and not statistically significant. The paper therefore presents transparency and efficiency as the clearest practical advantages, while treating predictive superiority as conditional rather than uniform.
6. Limitations, related directions, and naming ambiguity
The paper explicitly acknowledges several limitations. First, experiments used the same LLM architecture across all roles within a run, which may introduce model-specific biases. Second, although downstream training-time savings are significant, the multi-agent deliberation itself introduces additional orchestration and prompting overhead, and total end-to-end wall-clock cost of feature selection is not quantified. Third, improvements in AUC and accuracy are positive on average but not statistically significant. Fourth, the extracted methodology leaves important details unspecified, including exact prompt text, exact Judge weights 8 and 9, full selected feature lists, and detailed hyperparameter settings (Bal-Ghaoui et al., 7 Oct 2025).
The ablation picture is also limited. The paper varies subset size, backbone, and downstream classifier, but does not report formal ablations for removing the Challenger or Refiner, changing the number of agents, adding multiple debate rounds, varying prompt variants, changing Judge weights, or testing seed sensitivity. As a result, robustness is argued more through cross-backbone and cross-classifier consistency than through architectural isolation studies.
The future directions named by the authors are threefold: using different LLMs for different roles, tool-use integration so that agents such as the Refiner or Challenger can perform simple statistical tests, and dynamic adjustment of agent weights 0 based on feature ambiguity or complexity. This suggests a move toward more hybrid, tool-augmented systems.
In the broader literature represented by related agent architectures, LLM-FS-Agent remains comparatively compact. FS-Researcher treats the file system as a durable external memory and shared coordination medium for long-horizon research, using a dual-agent division between Context Builder and Report Writer (Zhu et al., 2 Feb 2026). FIRMHIVE uses a runtime-grown Tree of Agents with delegation as an executable primitive for large-scale firmware analysis (Zhang et al., 23 Nov 2025). LLM-as-Code argues that program control flow should remain deterministic and code-owned, with the LLM invoked only at scoped reasoning sites (Qi et al., 14 Jun 2026). These works operate on longer-horizon execution, persistent workspaces, or runtime control boundaries than the single-sequence four-role debate of LLM-FS-Agent. A plausible implication is that future feature-selection systems could combine LLM-FS-Agent’s explicit critique-and-judgment structure with stronger runtime control or tool-grounded execution.
The acronym itself is not entirely unique. An unrelated paper uses LLM-FS-Agent to denote a training-free LLM agent for few-shot classification and segmentation, built around GPT-4, GPT-4Vision, and SAM (Meng et al., 2023). In the present feature-selection context, however, LLM-FS-Agent denotes the deliberative architecture for transparent feature ranking introduced in 2025 (Bal-Ghaoui et al., 7 Oct 2025).
Taken as a whole, LLM-FS-Agent is best understood as a debate-based feature-selection framework whose main contribution is not a universal gain in predictive metrics, but the transformation of feature ranking into an auditable, role-specialized reasoning process. On the reported IoT intrusion-detection task, that process yields competitive to superior results around 20 features, and its most statistically secure advantage is a significant reduction in XGBoost training time alongside a substantially richer explanation trace (Bal-Ghaoui et al., 7 Oct 2025).
