Standard Glass Filling Estimation

• Standard glass filling is a task that estimates a cup's content level from a single RGB image, addressing issues like transparency and occlusion.
• The approach involves 4-way classification (0%, 50%, 90%, unknown) using a ResNet-18 backbone and adversarial pretraining to mitigate reliance on spurious features.
• Empirical results show that adversarial training followed by fine-tuning improves robustness, achieving up to 88% accuracy across diverse container shapes.

Standard glass filling denotes the estimation of the level of content in a cup or drinking glass from a single RGB image. In the formulation studied in "Improving filling level classification with adversarial training" (Modas et al., 2021), the task is posed as a 4-way classification problem over fill levels $y \in \{0\%, 50\%, 90\%, \text{unknown}\}$. The problem is technically nontrivial because transparencies in both the container and its content, vessel-shape variation, and partial occlusions by the holder’s hand make single-image cues ambiguous, while the availability of only a few thousand labeled examples encourages overfitting to dataset-specific artefacts rather than physically meaningful volumetric cues.

1. Problem formulation and sources of ambiguity

The task is defined on an input image $x \in [0,1]^{H \times W \times 3}$, with a classifier $f_\theta(x) = y$. The labels correspond to $0\%$, $50\%$, $90\%$, and an additional "unknown" class. The "unknown" label is used for non-transparent cases, reflecting the fact that visibility of the content itself is sometimes insufficient for deterministic inference (Modas et al., 2021).

Several factors make the problem deceptively hard. The containers include both stemmed and stemless geometries, as well as straight-walled and tapered forms. The walls may be transparent, translucent, or opaque, and the content may itself be transparent, as in water, or opaque, as in rice or pasta. Frequent hand occlusion further corrupts the visual evidence. In this setting, a standard CNN often "cheats" by exploiting spurious correlations such as cup color or shadow rather than learning the true physical cues that correlate with fill level.

Adversarial examples are used to motivate the study design. Minimal perturbations $\delta$ that flip the classifier’s decision expose the brittleness of standard optimization and suggest that conventional training emphasizes non-semantic, non-robust features. The central hypothesis is therefore that adversarially trained source models may transfer better to a small-data, high-ambiguity target task than conventionally trained source models.

2. Dataset construction and shape-generalization protocol

The empirical study uses the CORSMAL Containers Manipulation dataset, denoted C-CCM. It comprises approximately $10{,}269$ manually verified, Mask-R-CNN–cropped frames from 8 container types: 4 cups and 4 stemmed glasses, acquired under varied backgrounds and illumination conditions (Modas et al., 2021). The containers include transparent, translucent, and opaque walls; the content may be transparent or opaque.

Each frame is annotated with fill level in $\{0\%, 50\%, 90\%\}$, together with an "unknown" class for non-transparent cases. Occlusion and container transparency flags are also provided. These annotations are important because they separate geometric ambiguity from visibility ambiguity and enable evaluation under nonuniform observation conditions.

The evaluation emphasizes shape generalization rather than only in-distribution accuracy. Three train/test protocols are defined:

• $S_1$: train on all but champagne flute; test on flute.
• $x \in [0,1]^{H \times W \times 3}$0: swap beer cup vs. wine glass between train/test.
• $x \in [0,1]^{H \times W \times 3}$1: train on all stemmed vessels; test on stemless cups.

This protocol makes the task less about memorizing container identities and more about transferring fill-level reasoning across unseen vessel geometries. A plausible implication is that the reported performance is intended to measure robustness to cross-shape shift rather than only recognition of familiar exemplars.

3. Transfer learning and adversarial objective

The method is structured around a source domain $x \in [0,1]^{H \times W \times 3}$2, with approximately $x \in [0,1]^{H \times W \times 3}$3 million images, and a target domain $x \in [0,1]^{H \times W \times 3}$4, with approximately $x \in [0,1]^{H \times W \times 3}$5 thousand images (Modas et al., 2021). The classifier is decomposed as

$x \in [0,1]^{H \times W \times 3}$6

where $x \in [0,1]^{H \times W \times 3}$7 is ResNet-18 up to the global pooling layer and $x \in [0,1]^{H \times W \times 3}$8 is the final fully connected layer.

Adversarial training on the source domain is implemented via $x \in [0,1]^{H \times W \times 3}$9-PGD with 10 steps. For each $f_\theta(x) = y$0, the perturbation is defined by

$f_\theta(x) = y$1

and the adversarial risk is

$f_\theta(x) = y$2

The standard target-domain classification risk is

$f_\theta(x) = y$3

with cross-entropy loss $f_\theta(x) = y$4.

Although one could in principle optimize

$f_\theta(x) = y$5

the implementation is decoupled. First, the model is pre-trained on the source domain using either standard training or adversarial training. Second, the first $f_\theta(x) = y$6 ResNet blocks are frozen and the remaining layers are fine-tuned on the target domain using only $f_\theta(x) = y$7. This decoupling is operationally important because it separates robustness acquisition on a large generic source dataset from specialization on the small target dataset.

4. Network architecture, regimes, and hyperparameters

The backbone is ResNet-18 with 4 blocks plus pooling, and the classifier head is a fully connected layer producing 4 outputs (Modas et al., 2021). Optimization uses SGD with cross-entropy loss for 30 epochs and linear learning-rate decay. For training on the target domain only, the initial learning rate is $f_\theta(x) = y$8; for transfer learning, it is $f_\theta(x) = y$9. Batch sampling is class-balanced by inverse frequencies.

Six regimes are evaluated:

• ST: standard training on $0\%$0 only.
• AT: adversarial training on $0\%$1 only with $0\%$2-PGD and $0\%$3.
• ST→FT: standard training on $0\%$4, then freezing the first $0\%$5 layers and fine-tuning on $0\%$6.
• ST→AFT: standard training on $0\%$7, then freezing the first $0\%$8 layers and adversarial fine-tuning on $0\%$9.
• AT→FT: adversarial training on $50\%$0, then freezing the first $50\%$1 layers and fine-tuning on $50\%$2.
• AT→AFT: adversarial training on $50\%$3, then freezing the first $50\%$4 layers and adversarial fine-tuning on $50\%$5.

Ablation results identify the freezing depth as a critical variable. Fixing the first block only, $50\%$6, maximized generalization, whereas freezing more than one block steadily degraded performance. Source-domain perturbation radius $50\%$7 varied by split: $50\%$8 for $50\%$9, $90\%$0 for $90\%$1, and $90\%$2 for $90\%$3. By contrast, $90\%$4 was fixed at $90\%$5 for all target adversarial steps, because higher $90\%$6 prevented convergence.

5. Empirical performance and error characteristics

The reported test accuracies are averaged over the three container types in each split (Modas et al., 2021). For $90\%$7, the accuracies are: ST $90\%$8, AT $90\%$9, ST→FT $\delta$0, ST→AFT $\delta$1, AT→FT $\delta$2, and AT→AFT $\delta$3. For $\delta$4, they are: ST $\delta$5, AT $\delta$6, ST→FT $\delta$7, ST→AFT $\delta$8, AT→FT $\delta$9, and AT→AFT $10{,}269$0. For $10{,}269$1, they are: ST $10{,}269$2, AT $10{,}269$3, ST→FT $10{,}269$4, ST→AFT $10{,}269$5, AT→FT $10{,}269$6, and AT→AFT $10{,}269$7.

Across all three splits, AT→FT is the strongest regime. The reported margins over ST→FT are $10{,}269$8 percentage points in $10{,}269$9, $\{0\%, 50\%, 90\%\}$0 percentage points in $\{0\%, 50\%, 90\%\}$1, and $\{0\%, 50\%, 90\%\}$2 percentage points in $\{0\%, 50\%, 90\%\}$3. This establishes that adversarial training in the source domain, followed by standard fine-tuning in the target domain, is consistently more effective than both target-only training and standard source pretraining.

The results also show that adversarial training on the target domain alone is not beneficial in this setting. Standard AT on $\{0\%, 50\%, 90\%\}$4 alone hurts performance, and adversarial fine-tuning often underfits unless the source backbone is already robust. This is an important correction to any blanket assumption that adversarial optimization uniformly improves downstream accuracy on small, specialized vision datasets.

No per-category confusion matrices were released. However, the reported error pattern indicates that most errors occur between $\{0\%, 50\%, 90\%\}$5 and $\{0\%, 50\%, 90\%\}$6 on complex shapes, particularly under tapered-versus-straight wall ambiguity. This suggests that the dominant residual difficulty lies in discriminating intermediate from high fill levels when geometric perspective and wall shape distort direct visual estimation.

6. Interpretation, practical baseline, and significance

The interpretation offered for the performance gains is that adversarial training on the large-scale source domain forces the ResNet-18 backbone to discard non-robust, texture-biased features and focus on more semantically faithful cues such as shape edges and consistent shading (Modas et al., 2021). When such a backbone is fine-tuned on the small C-CCM dataset, these robust features generalize better to unseen vessel geometries and occlusion patterns, thereby mitigating overfitting to backgrounds, lighting, and container textures.

For practical deployment, the recommended procedure is explicit. One should start from an ImageNet-robust ResNet-18 trained with 10-step $\{0\%, 50\%, 90\%\}$7-PGD and $\{0\%, 50\%, 90\%\}$8–$\{0\%, 50\%, 90\%\}$9; freeze the first ResNet block when fine-tuning on a small fill-level dataset; fine-tune with standard cross-entropy for 30 epochs using an initial learning rate of $S_1$0, linear decay, and class-balanced sampling; and use $S_1$1 for any adversarial fine-tuning on the target domain, while prioritizing standard fine-tuning if convergence issues arise.

Within the scope of the reported experiments, this AT→FT recipe yielded test accuracies up to $S_1$2 on held-out vessel types and was described as a solid baseline for robot vision systems that estimate liquid levels from single RGB frames. More broadly, the study positions standard glass filling as a compact but informative benchmark for transfer learning under small data, shape shift, transparency-induced ambiguity, and vulnerability to spurious visual correlations.