--- title: Detect-Classify-Compare Framework url: https://www.emergentmind.com/topics/detect-classify-compare-framework type: topic --- # Detect-Classify-Compare Framework The detect-classify-compare framework denotes a family of staged analytical workflows in which candidate instances are first isolated or flagged, then assigned labels or routed through an auxiliary classifier, and finally compared through fingerprints, discriminative descriptors, or explanation-oriented profiles. In the supplied literature, this pattern appears in visually comparing binary classifiers through learning from disagreement, in static source-code vulnerability analysis with MARFCAT, and in CLIP-based visual classification with comparative descriptors; the sources show that the ordering and semantics of the three stages are domain dependent rather than canonical [2201.07849] [1010.2511] [2411.05357]. ## 1. Scope and domain-specific meanings Across the cited work, “detect,” “classify,” and “compare” do not denote a single fixed algorithm. Instead, the same stage labels are instantiated with different objects: disagreement regions between classifiers, distances to vulnerability fingerprints, or pairwise discriminative text cues among visually similar classes. The sources therefore support reading the framework as a modular pattern rather than as a standardized protocol. | Context | Detect | Classify / Compare | |---|---|---| | Learning-from-disagreement | Identify instances where two classifiers disagree after score-based thresholding | Train a discriminator on disagreement instances; compare models via SHAP and feature profiling | | MARFCAT | Flag a file when the minimal distance to stored fingerprints falls below a threshold | Assign CVE/CWE type by nearest fingerprint; compare via distance or similarity to class centroids | | Comparative descriptors for CLIP | Comparative descriptor module is specified for incorporation into a pipeline | Classify by averaging image-text similarities over descriptor sets; compare a target class against nearest semantic neighbors | A recurrent structural property is that comparison is not merely an evaluation afterthought. In the LFD setting, comparison is itself the main analytical target. In MARFCAT, comparison is operationalized as fingerprint matching. In comparative descriptors, comparison is encoded directly into the text prompts by conditioning on semantically nearest classes. ## 2. Learning-from-disagreement as a detect-classify-compare pipeline The LFD framework starts from two pre-trained binary classifiers, $f_1 : X \to [0,1]$ and $f_2 : X \to [0,1]$, each returning a real-valued score for input $x \in X$. The detection step is score-based. For a user-selected cutoff $\alpha$, thresholds $\tau_1,\tau_2$ are defined as the $\alpha$-quantiles of the score lists $S_1=\{f_1(x):x\in X\}$ and $S_2=\{f_2(x):x\in X\}$. Binary capture indicators are then defined by $y_1(x)=I[f_1(x)\ge \tau_1]$ and $y_2(x)=I[f_2(x)\ge \tau_2]$. The disagreement set is $$ D_{\text{disagree}}=\{x\in X \mid y_1(x)\ne y_2(x)\}. $$ It is split into $D^-=\{x\in X \mid y_1(x)=1,y_2(x)=0\}$, captured by $f_1$ only, and $D^+=\{x\in X \mid y_1(x)=0,y_2(x)=1\}$, captured by $f_2$ only [2201.07849]. The classification step trains a discriminator on disagreement instances. Each $x \in D_{\text{disagree}}$ is mapped to a meta-feature vector $z(x)\in \mathbb{R}^M$, where the meta-features are user-defined probes of the two base models’ behaviors. The paper uses a SHAP-compatible binary classifier, specifically XGBoost, to approximate $g(z;\theta)\approx P[\text{label}=1\mid z]$, with label $1$ indicating membership in $D^+$. The training objective is empirical cross-entropy: $$ L(\theta)= -\sum_{x\in D_{\text{disagree}}} \left[\ell(x)\log g(z(x);\theta)+(1-\ell(x))\log(1-g(z(x);\theta))\right], $$ where $\ell(x)=I[x\in D^+]$. The meta-features are domain-specific. In the merchant case study, they include time-series counts and rates over a fixed historical window, such as `nonzero_numApprTrans`, `nonzero_amtDeclTrans`, `mean_avgAmtAppr`, `mean_rateTxnAppr`, and `mean_numApprTrans`, with 70 meta-features in total for the merchant dataset. In the GNN case, graph-based merchant affinities include `n_connection` and `entropy_neighbors`. In the Avazu CTR case, each original categorical feature yields four probes: `n_impressions_f`, `n_clicks_f`, `ctr_f`, and `n_active_hours_f`, producing 168 meta-features from 42 original features. The comparison step is then built on the trained discriminator rather than directly on the base models. This design allows comparative interpretation even when the compared classifiers’ training features are not available. A plausible implication is that LFD treats disagreement regions as a compressed representation of model difference: only the captured-only cells are retained, and the comparison is delegated to a meta-feature space chosen to probe hypothesized behavioral distinctions. ## 3. Comparative interpretation, feature profiling, and empirical behavior in LFD LFD interprets the discriminator with Tree-SHAP. Because $g$ is a tree ensemble, each disagreement instance receives SHAP values $\phi_j(x)$ satisfying $$ g(z)=\phi_0+\sum_{j=1}^M \phi_j, $$ where $\phi_0=E_z[g(z)]$ is the expected base-score. The standard global importance measure is $$ \text{Magnitude}_j=\frac{1}{N}\sum_{x\in D}|\phi_j(x)|. $$ The framework then adds four complementary metrics: Consistency, Contrast, Correlation, and Overall. Consistency is based on the inverse of a bin-weighted Shannon-entropy over the empirical distribution of feature values within SHAP-value bins. Contrast is the Jensen-Shannon divergence between feature-value histograms on $\phi_j(x)>0$ and $\phi_j(x)\le 0$. Correlation is the absolute Pearson correlation between $z_j(x)$ and $\phi_j(x)$. Overall is the linear combination $$ \text{Overall}_j = 2\cdot \text{Magnitude}_j + \text{Consistency}_j + 2\cdot \text{Contrast}_j + \text{Corr}_j. $$ All five metrics are exposed for user-driven ordering and comparison [2201.07849]. The visual analytics system provides two coordinated views. The Disagreement Distribution View overlays the captured sets $A^+=\{y_1=1\}$ and $B^+=\{y_2=1\}$, shows the two non-overlap disagreement cells $A^+\setminus B^+$ and $B^+\setminus A^+$, and allows users to sweep $\alpha$ from $0\%$ to $100\%$ while dynamically recomputing cell sizes. The Feature View offers an “Overview + Details” design with two vertical lists of area-plots, one for the TP discriminator and one for the FP discriminator; features may be ordered by any of Magnitude, Consistency, Contrast, Corr, or Overall. On demand, a bubble-plot shows a 2D histogram of SHAP versus $z_j$, and a transfer-function widget remaps feature-value to color. The framework is evaluated in two industrial case studies. In merchant category verification, the dataset contains approximately 3.8 million merchants and 2.5 years of daily transaction logs, with the true-category binary label “restaurant” versus “non-restaurant.” Base models include Tree, CNN, RNN, and GNN. The discriminator achieves AUC $>0.99$ on the TP side and AUC $>0.98$ on the FP side. Reported findings include `nonzero_numApprTrans` high $\Rightarrow$ RNN correctly captures active merchants where Tree misses, and `neighbor_entropy` high $\Rightarrow$ GNN captures merchants with diverse affinities better than RNN. In CTR prediction on Avazu, the dataset contains 40 million ad-impressions over 10 days and 21 anonymized categorical fields. Day 1 is used to build historical meta-features, Days 2–8 for training, Day 9 for test/comparison, and Day 10 is held out for ensembling. Base models are LR using FTRL-proximal and Fi-GNN. The discriminator AUC is $0.993$ on Day 9 TP and $0.989$ on FP. The top TP-side meta-features by Overall are `ctr_site_id`, `ctr_site_app_id`, and `ctr_c14`; high `ctr_site_id` indicates that GNN catches click-instances better than LR. The same paper uses the top-15 meta-features, ranked by each metric, in Feature-Weighted Linear Stacking of LR+GNN. On Day 10, all FWLS ensembles beat the individual base models. The finding reported in the paper is that ranking by Overall yields the strongest Day 9 AUC, while Contrast slightly wins Day 10. The ablation study further reports that, with the 5 highest-Overall features alone, AUC is approximately $0.95$, whereas random 5 features give AUC approximately $0.62 \pm 0.12$. This suggests that the compare stage in LFD is not limited to explanation; it can also guide feature-wise ensembling and meta-feature selection. ## 4. MARFCAT: fingerprint comparison, vulnerability detection, and weakness classification MARFCAT realizes a detect-classify-compare framework for static source-code vulnerability analysis using the open-source MARF framework and the MARFCAT application. Its Compare stage is centered on a transformation $\Phi$ that maps raw source code into a fixed-length numerical fingerprint. Two parallel pipelines are provided. In the signal-processing pipeline, source text or binary is treated as a 1-D signal of overlapping n-grams, with experiments using $n=2$. After optional preprocessing such as silence removal or band-pass filtering, the short-time Fourier transform or direct FFT is computed, and the magnitude spectrum is aggregated into a $d$-dimensional feature vector $$ \mathbf{x}=\Phi_{\mathrm{signal}}(\text{code})=[f_1,f_2,\dots,f_d]^T. $$ In the NLP pipeline, code is viewed as a sequence of tokens or characters, unigrams or n-grams are extracted, and empirical counts or smoothed probability vectors are recorded [1010.2511]. Training files annotated with CVE or CWE labels are used to build a fingerprint store by clustering feature vectors by class. In the preliminary release, one centroid per CVE/CWE is computed: $$ \mu_c=\frac{1}{N_c}\sum_{i:y_i=c}\Phi(\text{code}_i). $$ A test file $\mathbf{x}$ is compared against a stored fingerprint $\mu_c$ by distance or similarity measures including Euclidean distance, $$ D_2(\mathbf{x},\mu_c)=\|\mathbf{x}-\mu_c\|_2, $$ cosine similarity converted to a distance, $$ D_{\cos}(\mathbf{x},\mu_c)=1-\frac{\mathbf{x}\cdot \mu_c}{\|\mathbf{x}\|\|\mu_c\|}, $$ as well as Chebyshev, Minkowski, and Diff distances. The Detection stage uses the minimum distance over all trained classes: $$ D_{\min}(\mathbf{x})=\min_c D(\mathbf{x},\mu_c). $$ If $D_{\min}(\mathbf{x})$ falls below a predefined threshold $\tau$, the file is flagged for further analysis. The paper notes that MARFCAT’s current reference implementation uses nearest-centroid and k-NN classifiers, but that the same detection stage could be backed by any supervised model; a linear SVM hinge-loss formulation is given as an example. The Classification stage assigns a suspicious file to a specific vulnerability or weakness type. In the nearest-centroid scheme, $$ \hat c(\mathbf{x})=\arg\min_c D(\mathbf{x},\mu_c). $$ A probabilistic alternative using multinomial logistic regression or a neural-network softmax layer is also specified. The evaluation uses CVE-selected vulnerable/fixed pairs for C/C++, including Wireshark 1.2.0 versus 1.2.9 and Chrome 5.0.375.54 versus 5.0.375.70, CVE-selected Java with Tomcat 5.5.13 versus 5.5.29, and non-CVE cases Dovecot 2.0β6 and Pebble 2.5-M2. Metrics are Precision, Recall, and F1. For the SATE2010 top configuration with the signal pipeline and first guess, reported results are approximately 92.7% precision and 100% recall for Wireshark CVEs, approximately 83.7% precision and approximately 93% recall for Tomcat CVEs, and approximately 90.9% precision and 100% recall for Chrome CVEs. For Dovecot 2.0β6 and Pebble 2.5-M2, the paper reports no false negatives on non-CVE data under tuned thresholds. The unigram-plus-add-delta NLP pipeline yields slightly lower precision but remains competitive, with Wireshark CVEs at 83.3%. A common misunderstanding would be to assume that “compare” must occur after detection and classification. MARFCAT shows the opposite ordering explicitly: the fingerprint comparison machinery is foundational, and detection and classification are derived from it. ## 5. Comparative descriptors in CLIP-based visual classification In the CLIP setting, the relevant contribution is a comparative descriptor module suitable for direct incorporation into a detect-classify-compare pipeline. A comparative descriptor is defined as a short text phrase that highlights features by which a target class $t$ can be distinguished from one of its semantically nearest neighbors $s$. Instead of describing a class in isolation, the method asks an LLM for cues specific to the pair $(t,s)$, for example: “What are useful visual features for distinguishing a $\{t\}$ from a $\{s\}$ in a photo?” The stated purpose is to focus on subtle inter-class differences, reduce ambiguity, and avoid overly generic or spurious attributes [2411.05357]. Let $C$ be the set of candidate classes. For each class $c\in C$, the standard CLIP-style prompt is $f_c :=$ “A photo of a $c$,” with text embedding $e_c := \phi_T(f_c)\in \mathbb{R}^d$. Semantic proximity is measured entirely in CLIP’s text space by cosine similarity: $$ \mathrm{sim}_{\mathrm{text}}(i,j)=\cos(e_i,e_j) =\frac{e_i\cdot e_j}{\|e_i\|\|e_j\|}. $$ For each target class $t$, the method selects the top-$n$ nearest neighbors $S_t=\text{top-}n\{|C\setminus\{t\}| \text{ ranked by } \mathrm{sim}_{\mathrm{text}}(t,\cdot)\}$. For each pair $(t,s)$ with $s\in S_t$, an LLM generates a short phrase $d_{t,s}$, whose CLIP text embedding is $c_{t,s}:=\phi_T(d_{t,s})\in \mathbb{R}^d$. The full descriptor set is $D_t^{\mathrm{comp}}=\{c_{t,s}:s\in S_t\}$. Classification averages similarities between the image embedding $\phi_I(x)$ and the comparative descriptor embeddings: $$ \mathrm{score}_{\mathrm{comp}}(t\mid x)= \frac{1}{|D_t^{\mathrm{comp}}|} \sum_{c\in D_t^{\mathrm{comp}}} s(\phi_I(x),c), $$ where $s(u,v)=u\cdot v$ or cosine similarity if both vectors are L2-normalized. The baseline zero-shot classifier predicts $\tilde t_0=\arg\max_{t\in C}\cos(\phi_I(x),e_t)$, while the comparative-descriptor classifier predicts $\tilde t=\arg\max_{t\in C}\mathrm{score}_{\mathrm{comp}}(t\mid x)$. The paper also allows fusion of the original prompt with the comparative descriptors by setting $D'_t=\{e_t\}\cup D_t$ and averaging over the union. A filtering process removes descriptors that are irrelevant to CLIP’s vision-language space. For class $t$, a small set of $M$ representative images is gathered, and the mean image feature is $$ \bar v_t=\frac{1}{M}\sum_{m=1}^M \phi_I(x_{t,m}). $$ The lower-bound threshold is $$ LB_t=\min\{\cos(\bar v_t,e_t),\tau\}, $$ with $\tau\approx 0.3$. Each raw descriptor embedding $c$ is scored by $$ \mathrm{score}_t(c)=\cos(\bar v_t,c), $$ descriptors with $\mathrm{score}_t(c)