DevFinder: Benchmark & Dev Search

• DevFinder is a dual-purpose concept that defines a smart-home benchmark for command-to-device mapping and a hypothetical developer search system for code retrieval.
• As a smart-home benchmark, it uses human-annotated command-device pairs and the Device Relevance Score to quantify LLM/SLM performance in realistic multi-device environments.
• In developer support, DevFinder generalizes techniques like replacement-class retrieval and similar-code search by combining candidate generation with explicit structural evidence.

Searching arXiv for the cited works and related DevFinder references. Tool call: arxiv_search("(Huang et al., 10 Jul 2025) HomeLLaMA DevFinder") DevFinder denotes, in the cited literature, two distinct but related constructs. In "Towards Privacy-Preserving and Personalized Smart Homes via Tailored Small LLMs," it is a smart-home benchmark introduced to quantitatively evaluate the plan quality of LLM/SLM-based smart home assistants through command-to-device-set prediction (Huang et al., 10 Jul 2025). In interpretive discussions built around prior software-engineering systems, DevFinder is also used as the name of a hypothetical development-search and recommendation system that generalizes techniques from replacement-class retrieval, similar-code search, fix-locus recommendation, and source-file-set search (Samak et al., 2021, Kashyap et al., 2017, Mobilio et al., 2019, Ishio et al., 2017). The term therefore spans both a concrete benchmark for smart-home assistants and a broader conceptual template for developer-support tooling.

1. Terminological scope and problem framing

Within the smart-home literature, DevFinder is defined narrowly and operationally. It is a test benchmark, released as a dataset, of smart-home commands with human-annotated device labels. Its purpose is to test whether a model can correctly understand a natural language command and select the appropriate set of smart devices that should be involved in the resulting action plan. The benchmark was designed to fill what is described as an evaluation gap: existing smart-home work often shows examples or user satisfaction, but lacks a standard, device-level quantitative benchmark that can directly measure how well a model maps commands to relevant devices in realistic multi-device smart homes, especially for on-device small LLMs that must balance performance and privacy (Huang et al., 10 Jul 2025).

In the software-engineering material, DevFinder is not presented as a standalone published system with a single canonical implementation. Rather, it appears as a hypothetical name for a broader family of development-search and recommendation tools. In that usage, the central problem is no longer command-to-device mapping but artifact retrieval under strong compatibility constraints: replacement classes rather than merely similar classes, similar source code rather than token overlap, original components rather than isolated files, and fix locations rather than anomalous traces alone. The shared emphasis is that search quality depends on structured compatibility signals, not only on lexical resemblance (Samak et al., 2021, Kashyap et al., 2017).

2. DevFinder as a smart-home benchmark

The benchmark centers on a single core capability that underpins many smart-home behaviors: given a natural language command, determine which devices in a realistic home setup are relevant. The task can therefore be summarized as command to relevant device set prediction. The commands cover environmental or climate control, atmosphere adjustment, power or energy management, lighting, security, and other common home scenarios. They are not restricted to simple commands such as direct actuation of a named device; the benchmark includes under-specified natural commands that require contextual interpretation, such as party setups or comfort control, where multiple devices may be jointly relevant (Huang et al., 10 Jul 2025).

DevFinder is built on top of the public IFTTT smart home dataset. From that dataset, the authors select 100 test commands that have human-annotated device labels and cover a wide range of scenarios. For these 100 commands, human annotators label the set of devices that should be activated or used if the home is equipped with a comprehensive set of commercial smart devices. This yields a ground-truth device set for each command. Evaluation assumes a comprehensive smart home setup with commonly used smart devices; the augmented training setup earlier in the same work contains 39 devices in total, and a compatible comprehensive set is assumed for DevFinder evaluation. The benchmark itself is the held-out test set: 100 real, human-labelled examples not used during training, whereas training uses a separate augmented dataset of 14K command-device pairs synthesized and labeled with GPT-4 (Huang et al., 10 Jul 2025).

Each benchmark entry conceptually contains a natural language command $s_i$ and a ground-truth device set $G_l$. The paper does not publish a formal schema, but the structure is explicitly described as an input command paired with a labeled subset of the comprehensive device inventory. Because the device universe is rich rather than minimal, the task is nontrivial: a model must select an appropriate subset from many possible devices rather than merely identify a single directly named appliance.

3. Evaluation methodology and empirical role

DevFinder uses Device Relevance Score, or DRS, as its primary quantitative metric. Given a ground-truth device set $G_l$ and a model-predicted device set $G_r$, the score is defined as

$DRS = \frac{|G_l \cap G_r| - |G_r - G_l|}{|G_r|}.$

Here, $|G_l \cap G_r|$ counts correctly predicted relevant devices, and $|G_r - G_l|$ counts extra predicted devices that are not in the ground truth. The score is normalized to lie in $[-1,1]$. Predicting only correct devices with no extras yields $DRS = 1$, whereas predicting many wrong devices and few correct ones yields a low or negative value. The benchmark reports average DRS over the 100 commands (Huang et al., 10 Jul 2025).

The benchmark is used to compare HomeLLaMA against cloud LLM systems and other on-device systems. The evaluated models include HomeGPT, Sasha, and SAGE on the cloud side, and TT-Gemma, TT-Phi-2, and HomeLLaMA on the on-device side. For fair comparison of local device identification, the cloud-assisted module of HomeLLaMA is disabled when DRS is computed. Stochastic models are evaluated at temperatures $t=0.1$ and $G_l$0 to study robustness and the creativity-versus-hallucination trade-off. Results are reported by model type, precision setting such as FP16 versus INT8, and temperature (Huang et al., 10 Jul 2025).

The reported trends are central to the paper’s empirical claims. Cloud LLM assistants have the best DRS. HomeLLaMA achieves DRS values comparable to GPT-3.5 and close to GPT-4 in some scenarios despite running locally, and clearly outperforms TT-Gemma and TT-Phi-2 across scenarios and quantization settings. The paper attributes this to the tailored 14K command-device training set synthesized with GPT-4 and to the structured inference paradigm consisting of two-step device identification and home-specific matching. DevFinder is therefore the principal objective benchmark supporting the claim that a tailored on-device SLM can approach cloud LLM performance while preserving user privacy (Huang et al., 10 Jul 2025).

4. Personalization, privacy, and benchmark limitations

DevFinder is not, by itself, a personalization benchmark. Its labels are generic device sets appropriate for each command, independent of a particular user’s taste. In the smart-home system described around it, personalization is downstream: the assistant must first identify all candidate devices that could be relevant, and user-profile learning then refines how those devices should be configured, such as brightness or temperature. DevFinder commands are also used as initial prompts in user studies, where plans, profile summaries, and personalized re-plans are subsequently evaluated by human ratings (Huang et al., 10 Jul 2025).

Its role in privacy evaluation is more direct. The benchmark commands serve as the representative in-home commands in activity-monitoring experiments involving PrivShield. In that setting, each real command is paraphrased and mixed with $G_l$1 adversarial commands generated by the local SLM, and all $G_l$2 commands are sent as a batch to a cloud LLM. Attackers then try to identify the real command across multiple rounds. The work measures attack success rate on this query set, so DevFinder underlies both utility evaluation through DRS and privacy evaluation through activity-monitoring success. The overall effective success rate is written as

$G_l$3

where $G_l$4 is the attack success rate when PrivShield is used and $G_l$5 is the fraction of user queries that actually go through the cloud (Huang et al., 10 Jul 2025).

Several limitations are explicit or directly inferable from the benchmark design. DevFinder currently contains 100 test commands, which is sufficient for comparison on a well-defined device-mapping task but small relative to large NLP benchmarks. It is derived from IFTTT and a comprehensive but finite device inventory, so it may not cover highly specialized devices, unusual routines, or multimodal inputs. It is built and evaluated in English. It also evaluates which devices should be used, not how they should be configured, and thus does not directly measure fine-grained personalization, multi-turn dialogue coherence, or safety against dangerous or inappropriate actions. This suggests that future benchmarks could combine device relevance with multi-turn personalization and safety or privacy tasks, but that extension is not part of DevFinder as presently defined (Huang et al., 10 Jul 2025).

5. DevFinder as a hypothetical development-search and recommendation system

In the software-engineering discussions, DevFinder is presented as a possible generalization of several existing techniques rather than as a single deployed framework. One lineage comes from "Searching for Replacement Classes," where the underlying problem is: given a query class $G_l$6 and a large search corpus $G_l$7, return a ranked subset of classes that can act as replacement classes for $G_l$8. The relevant design pattern is a two-stage pipeline: embedding-based candidate retrieval followed by type-aware analysis. The latter constructs a type similarity matrix $G_l$9, a field mapping $G_l$0, and a method mapping $G_l$1, then ranks candidates by the number and quality of matched methods. In that interpretation, DevFinder would inherit the principle that search should target replacement or fit, not similarity alone, and that explicit mappings are first-class outputs rather than by-products (Samak et al., 2021).

A second lineage comes from "Source Forager," which treats code search as comparison across multiple feature-classes rather than a single representation. The described feature-classes include type-operation coupling, skeleton tree, decorated skeleton tree, weighted natural-language terms, CFG $G_l$2-subgraphs, library-call features, type signature, local types, constants, and comments. Similarity is computed per feature-class and then aggregated by a normalized weighted average,

$G_l$3

The proposed relevance of this design to DevFinder is that a developer-support search system should use a modular, extensible, multi-feature similarity architecture with either dynamic feature selection or learned feature weights (Kashyap et al., 2017).

A third lineage comes from "Source File Set Search for Clone-and-Own Reuse Analysis," where the query is a set of source files rather than a single file or class. There the search engine represents files with token trigrams, uses Jaccard similarity, and scales retrieval through $G_l$4-bit minwise hashing with $G_l$5 and $G_l$6. At the component level it defines

$G_l$7

and

$G_l$8

then filters redundant results via a dominance relation over components. In a DevFinder interpretation, these mechanisms support large-ecosystem retrieval of original or related components from a directory-sized query rather than a snippet-level query (Ishio et al., 2017).

A fourth lineage comes from "FILO: FIx-LOcus Recommendation for Problems Caused by Android Framework Upgrade." There, a DevFinder-like tool is described as using one failing execution on an upgraded OS and one working execution on the old OS, identifying Suspicious Invocation Blocks, building a failure call tree, and ranking app methods by suspiciousness. The suspiciousness score is

$G_l$9

with $G_r$0 and $G_r$1 in the implementation. The associated symptomatic anomalous events function as explanatory evidence, so the search result is not merely a ranked method list but a ranked list tied to abnormal API or callback sequences (Mobilio et al., 2019).

6. Shared architectural themes and conceptual significance

Across these usages, DevFinder is associated with a recurrent architectural pattern: broad candidate generation followed by structurally constrained refinement. In smart homes, that pattern appears as local SLM inference complemented, when needed, by cloud assistance and privacy shielding. In replacement-class retrieval it appears as embedding-based pruning followed by type-aware optimization. In similar-code search it appears as a weighted combination of heterogeneous feature-classes. In source-file-set search it appears as approximate signature filtering followed by exact Jaccard and component-level aggregation. In fix-locus recommendation it appears as trace differencing followed by call-tree-based ranking (Huang et al., 10 Jul 2025, Samak et al., 2021, Kashyap et al., 2017, Ishio et al., 2017, Mobilio et al., 2019).

A second commonality is that DevFinder is consistently tied to explainability through intermediate structure. Smart-home DevFinder exposes device-level correctness through human-annotated ground-truth sets and DRS. ClassFinder-style DevFinder exposes field and method mappings. Source-file-set search exposes per-file similarity matrices and component dominance. FILO-style DevFinder exposes Suspicious Invocation Blocks and their reachable methods. This suggests that, in the cited material, DevFinder is less a single algorithm than an organizing label for search systems that couple retrieval with explicit evidence about why a result is relevant.

A third commonality is evaluation under realistic operational constraints. The smart-home benchmark exists because qualitative examples and user satisfaction alone were judged insufficient. Replacement-class search is evaluated on approximately 600,000 Java classes. Source Forager is evaluated with Mean Average Precision under large distractor sets. Source-file-set search is evaluated on 75 cloned components against 10 million files in Debian packages. FILO is evaluated on 12 real Android apps with upgrade-induced compatibility problems. The plausible implication is that DevFinder, whether interpreted narrowly as the smart-home benchmark or broadly as a developer-support paradigm, is defined by measurable task formulations, held-out or ground-truthed scenarios, and output structures intended to be directly actionable for expert users.