Determine the contributions to limited residual-embedding effectiveness in military imagery

Determine the individual contributions of underrepresentation in VLM training data and limited environmental variation, and characterize the connection between these factors, to explain their impact on concept purity and the effectiveness of residual embeddings for military vehicle imagery.

Background

The proposed error slice discovery method relies on residual embeddings formed by subtracting class-level mean embeddings. The authors argue that this approach requires concept purity: image embeddings should separate the main object from environmental context, while sufficient environmental diversity is needed for class averaging to remove object-related information without preserving spurious environmental correlations.

The experiments suggest that residual embeddings are less effective for military vehicle data than for dog-breed data, potentially because military vehicles are underrepresented in VLM training data and because the military images contain less variation in their surroundings. The paper does not establish the separate or interacting effects of these factors, leaving their relationship and relative importance unresolved.