Context State Object Recognition

• Context State Object recognition is a joint classification approach that predicts an object and its state by fusing visual features with external language knowledge.
• It employs a dual-headed ResNet architecture combined with semantic scores from Concept-Net to re-score predictions via a fusion and refinement mechanism.
• Empirical results on cooking imagery demonstrate that integrating language priors improves accuracy in both object and state recognition tasks.

Searching arXiv for the target paper and closely related object/state recognition work. Joint object–state recognition is a visual classification problem in which an image is assigned both an object label and a state label, under the premise that the two are statistically and semantically coupled. In cooking imagery, the coupling is especially strong: object identity constrains plausible states, and state cues can sharpen object recognition. “Joint Object and State Recognition using Language Knowledge” formalizes this dependence by combining a two-headed convolutional architecture with semantic relatedness derived from Concept-Net, so that visual confidences and language-based object–state likelihoods jointly determine the final predictions (Jelodar et al., 2019).

1. Problem setting and formalization

The method assumes a single RGB image $x$ containing one cooking object in an unknown state. The prediction target is a pair of labels: an object label $o \in O=\{o_1,\dots,o_{N_{\mathrm{obj}}}\}$ and a state label $s \in S=\{s_1,\dots,s_{N_{\mathrm{state}}}\}$. The formulation is explicitly joint, because object and state are treated as correlated rather than independent variables. The paper therefore introduces the marginal posteriors $P(o \mid x)$ and $P(s \mid x)$ together with the joint posterior $P(o,s \mid x)$, and seeks estimates $\hat P(o \mid x)$ and $\hat P(s \mid x)$ obtained by combining visual evidence with language knowledge (Jelodar et al., 2019).

This framing departs from a purely appearance-based classifier in two ways. First, it assumes that some object–state combinations are semantically more likely than others, such as “creamy butter” relative to “creamy potato.” Second, it permits the final decision to be derived either from the joint posterior by selecting $(\hat o,\hat s)$ through an argmax over object–state pairs, or from independent marginal decisions. The core methodological claim is that object and state should be recognized together because each serves as context for the other (Jelodar et al., 2019).

2. Visual backbone and dual classification heads

The visual component begins with standard preprocessing. Each image is resized to $224 \times 224$, zero-centered, and normalized by Imagenet means. Standard data augmentation through random crops and flips is used to balance classes. The base network is ResNet-50 up through the global average-pooling layer, with all layers except the final block frozen to Imagenet-pretrained weights (Jelodar et al., 2019).

From the shared feature vector $o \in O=\{o_1,\dots,o_{N_{\mathrm{obj}}}\}$0, the network branches into two parallel fully connected plus softmax heads. One head predicts object confidences,

$o \in O=\{o_1,\dots,o_{N_{\mathrm{obj}}}\}$1

and the other predicts state confidences,

$o \in O=\{o_1,\dots,o_{N_{\mathrm{obj}}}\}$2

The two heads are trained jointly with cross-entropy losses $o \in O=\{o_1,\dots,o_{N_{\mathrm{obj}}}\}$3 and $o \in O=\{o_1,\dots,o_{N_{\mathrm{obj}}}\}$4, summed as

$o \in O=\{o_1,\dots,o_{N_{\mathrm{obj}}}\}$5

The resulting architecture is therefore a shared ResNet feature extractor with a double-classification layer, optimized so that object and state supervision act on the same representation (Jelodar et al., 2019).

The significance of this design is architectural economy rather than explicit structured prediction inside the CNN itself. Correlation between object and state is not imposed by a joint decoder at the visual stage; instead, the network produces two marginals that are subsequently re-scored using language knowledge.

3. Concept-Net as a semantic prior over object–state pairs

The language component is built from Concept-Net, described as a large graph whose nodes are words or phrases and whose edges carry weighted assertions such as RelatedTo and UsedFor. The method extracts from the RelatedTo relation a numerical relatedness score $o \in O=\{o_1,\dots,o_{N_{\mathrm{obj}}}\}$6 for each object–state pair. Google N-gram frequencies can also be used instead, with the same procedure (Jelodar et al., 2019).

For each pair $o \in O=\{o_1,\dots,o_{N_{\mathrm{obj}}}\}$7, the semantic score is defined as

$o \in O=\{o_1,\dots,o_{N_{\mathrm{obj}}}\}$8

where $o \in O=\{o_1,\dots,o_{N_{\mathrm{obj}}}\}$9 and $s \in S=\{s_1,\dots,s_{N_{\mathrm{state}}}\}$0 contain synonyms such as singular and plural lexical variants. The scores are then normalized so that

$s \in S=\{s_1,\dots,s_{N_{\mathrm{state}}}\}$1

The paper interprets $s \in S=\{s_1,\dots,s_{N_{\mathrm{state}}}\}$2 as a joint prior, or semantic likelihood, that an object and a state co-occur in natural language (Jelodar et al., 2019).

This semantic layer functions as an external source of structured knowledge. It does not inspect the image; instead, it encodes linguistic regularities about which states are compatible with which objects. A plausible implication is that the system uses language statistics as a surrogate for commonsense constraints that are difficult to learn robustly from a limited visual dataset alone.

4. Fusion of visual confidences and language knowledge

The fusion stage combines the CNN outputs with the semantic prior through conditional tables derived from $s \in S=\{s_1,\dots,s_{N_{\mathrm{state}}}\}$3. The method first computes

$s \in S=\{s_1,\dots,s_{N_{\mathrm{state}}}\}$4

The ResNet priors are denoted

$s \in S=\{s_1,\dots,s_{N_{\mathrm{state}}}\}$5

The language-augmented marginals are then obtained by Bayes-style fusion:

$s \in S=\{s_1,\dots,s_{N_{\mathrm{state}}}\}$6

$s \in S=\{s_1,\dots,s_{N_{\mathrm{state}}}\}$7

The intuition given in the paper is that each object hypothesis is re-scored by summing over all possible states, weighted by both the CNN state confidence and the semantic relatedness between that state and the object; the state update is symmetric (Jelodar et al., 2019).

The pipeline then performs a second fusion step. For objects, it constructs

$s \in S=\{s_1,\dots,s_{N_{\mathrm{state}}}\}$8

and for states,

$s \in S=\{s_1,\dots,s_{N_{\mathrm{state}}}\}$9

These are stacked into $P(o \mid x)$0 and passed to two small three-layer MLPs, one for object correction and one for state correction, each trained with cross-entropy to output $P(o \mid x)$1 or $P(o \mid x)$2 (Jelodar et al., 2019).

A further safeguard is introduced because the language graph can itself be misleading. The paper gives the example that “grated butter” may be overrepresented in Concept-Net. To address this, a refinement-selector gate is trained as a small CNN-based binary classifier that predicts, per image, whether the raw CNN head or the MLP output is more trustworthy. At test time, the gate selects between the two confidence sets on a per-sample basis (Jelodar et al., 2019). This mechanism directly addresses a common misconception about language priors: they are not treated as uniformly reliable, but as potentially corrective and potentially error-inducing.

5. Dataset, protocol, and empirical results

The experiments are conducted on a cooking-object dataset of approximately 9,500 images, covering 15 object classes and 11 states. Two “mixed/other” states are dropped, leaving $P(o \mid x)$3 and $P(o \mid x)$4. The split is 70% train, 15% validation, and 15% test, with flips and crops used to balance both object and state classes. Evaluation uses average per-class accuracy, defined as the average of recall over classes (Jelodar et al., 2019).

The reported test results show a consistent improvement from adding language knowledge on top of the two-head ResNet. The gains are larger for state recognition than for object recognition, but both tasks improve.

Model	State acc.	Object acc.
ResNet (two-head only)	79.4%	74.1%
ResNet+Concept-Net (SVM fusion)	79.7%	74.2%
ResNet+Google N-gram + MLP	80.1%	74.2%
ResNet+Concept-Net + MLP	80.4%	74.3%
ResNet+Concept-Net + MLP + Refinement	80.9%	75.0%

The paper states that incorporating a language prior always improves over the raw ResNet baseline. The best single-shot pipeline without refinement yields a $P(o \mid x)$5 gain in state accuracy and a $P(o \mid x)$6 gain in object accuracy. The refinement stage adds a further $P(o \mid x)$7 for states and $P(o \mid x)$8 for objects, reaching final scores of $P(o \mid x)$9 and $P(s \mid x)$0 respectively (Jelodar et al., 2019).

A qualitative example illustrates how the semantic prior changes the decision process: the raw CNN confuses “diced strawberry” with “tomato paste,” while Concept-Net re-scoring pushes the joint likelihood toward “tomato paste” (Jelodar et al., 2019). This example is important because it shows that the method is not merely smoothing logits; it actively biases prediction toward semantically coherent object–state pairs.

6. Position within object-state and context-aware recognition

The 2019 formulation is narrowly defined: one cooking object, one image, and a single object label plus a single state label. Later work broadened the object-state problem in ways that expose the scope of this earlier model. “Learning Multiple Object States from Actions via LLMs” formulates object state recognition as a multi-label classification task because multiple states can coexist simultaneously, and it accumulates past state descriptions so that the current state depends on temporal context (Tateno et al., 2024). Relative to that later formulation, the 2019 pipeline addresses semantic compatibility between object and state, but not coexistence of multiple states or long-range temporal accumulation.

The method also differs from context-aware recognition approaches that model visual context rather than lexical co-occurrence. Context-LGM introduces correlated latent variables for object and context and conditions the contextual posterior on the object representation (Liu et al., 2021). CATNet studies how contextual modulation depends on the amount, quality, and temporal dynamics of scene context in object recognition (Zhang et al., 2019). Spatial Memory Network models instance-level context and object–object relationships through a sequential memory update process (Chen et al., 2017). ContextHOI uses a dedicated context branch to learn informative spatial context for human–object interaction recognition without manual background labels (Jia et al., 2024). A plausible implication is that the 2019 paper occupies an intermediate methodological position: it is more structured than a pure two-head visual classifier, yet less expressive than later temporal, latent-variable, or spatial-memory formulations of context.

Its principal conceptual contribution is therefore specific and delimited. It shows that object recognition and state recognition can be improved by treating them as a joint prediction problem and by introducing a language knowledge graph as a semantic prior (Jelodar et al., 2019). Its limitations are equally clear from the setup: the task is restricted to cooking imagery, the dataset contains a fixed set of 15 objects and 9 retained states, the semantic signal is sourced from external language statistics rather than learned end-to-end from multimodal corpora, and the language graph can encode spurious associations that must be filtered by a refinement gate (Jelodar et al., 2019). Within those boundaries, the paper provides a concrete early example of how contextual language knowledge can improve fine-grained visual state recognition in robotic cooking scenarios.