Domain-Agnostic Video Discriminator

• Domain-agnostic video discriminators (DVDs) are neural architectures that mitigate dataset-specific biases by focusing on invariant spatial and temporal features.
• They employ strategies such as class-conditioned adversarial confusion, spatial-temporal decomposition, and graph-based instance mixing for robust feature learning.
• In applications like robotic reward learning, face anti-spoofing, and GAN-based video synthesis, DVDs have demonstrated state-of-the-art generalization and performance.

A Domain-agnostic Video Discriminator (DVD) is a neural architecture or training framework explicitly designed to ignore domain- and dataset-specific biases in video processing tasks. The term “domain-agnostic” in this context refers to the ability of the DVD to either: (i) learn features invariant to extraneous domains in classification, recognition, or anti-spoofing contexts; (ii) provide generalizable reward or similarity functions for cross-domain imitation or reinforcement learning; or (iii) disentangle spatial and temporal evidence in generative adversarial settings without encoding dataset-specific priors. Several distinct approaches and architectures—most prominently in video reward learning, face anti-spoofing, domain adaptation, and adversarial video synthesis—fall into this category, unified by their explicit mechanism for suppressing spurious correlations stemming from extraneous domains.

1. Motivations and General Frameworks

Domain-agnostic video discriminators serve to overcome failures of generalization caused by dataset shifts prevalent in video data, including shifts in background, lighting, camera, scene layout, or data modality. DVD approaches seek to build detectors, reward functions, or classifiers whose internals and outputs are robust when transferred to novel domains not seen in training. This is critical for applications such as open-world robotic reward learning, face anti-spoofing, generative video synthesis, and video domain adaptation, where naively trained models overfit to domain-specific artifacts and fail to generalize (Chen et al., 2021, Saha et al., 2019, Clark et al., 2019, Luo et al., 2020).

DVD architectures typically employ one or more of the following strategies:

• Explicit class-conditioned adversarial confusion to enforce domain invariance,
• Decomposing video evidence into spatial and temporal streams to avoid dataset-specific priors,
• Cross-domain bipartite graph modeling with conditional adversarial terms,
• Training discriminators on cross-task or cross-domain similarity rather than absolute class labels.

2. DVD in Robotic Reward Learning

In the context of learning reward functions for generalist robots, the Domain-agnostic Video Discriminator (DVD) is defined as a similarity-based video classifier trained to determine whether two videos depict the same task, regardless of domain (e.g., human demonstration vs. robot trial). Given a collection of both human “in-the-wild” videos and limited robot demonstrations, the DVD learns a function $D_\phi(v_1, v_2) \in [0,1]$ indicating whether $v_1$ and $v_2$ share the same functional task.

Network and Training

• Shared video encoder: a fixed 3D-CNN backbone, pretrained on a broad dataset (e.g. Something-Something V2).
• Similarity network: a two-layer MLP or cosine-similarity head, receiving encodings for a video pair.
• Objective: binary cross-entropy loss over positive (same-task) and negative (different-task) video pairs:

$$\mathcal{L}(\phi) = -\mathbb{E}_{(v_1,v_2,y)}[y\log D_\phi(v_1,v_2) + (1-y)\log(1-D_\phi(v_1,v_2))]$$

• Balanced sampling ensures cross-domain (human/robot) and cross-task coverage.
• Data augmentation (random rotations, crops) enhances invariance.

Zero-Shot Generalization and Deployment

At test time, the DVD produces a reward for robotic control by scoring similarity between a user-supplied human demo $v_{\mathrm{demo}}$ and predicted robot trajectories, using a video prediction model (e.g., SV2P) for model-based planning. This method yields strong zero-shot generalization both to new environments and completely unseen tasks, outperforming supervised baselines and yielding state-of-the-art transferability (Chen et al., 2021).

3. DVD for Face Anti-Spoofing and Domain-Invariant Feature Learning

In anti-spoofing systems, domain-agnostic video discriminators enforce invariance to spurious variability (e.g., backgrounds, lighting) that confound standard classifiers. The dominant approach combines a deep backbone (ResNet-50 for image frames) with temporal modeling (LSTM over frame features), whose output feature $H$ is shared between live/spoof classifiers and a class-conditional domain discriminator.

Core Mechanisms

• Class-conditional domain discriminator: After two shared fully connected layers, samples are separated by true class label (live/spoof), each handled by a dedicated head producing domain classification logits (for $D$ source domains).
• Adversarial training: A Gradient Reversal Layer (GRL) multiplies domain discrimination gradients by $-\lambda_{GRL}$, causing the feature extractor ($\theta_e, \theta_r$) to maximize domain confusion while the domain heads ($\theta_f, \theta_l, \theta_s$) minimize domain loss.
• Objective:
  • Main classification: 2-way cross-entropy on live/spoof.
  • Domain adversarial: cross-entropy over domains, per class.
  • Final energy is a balanced sum of both losses.

This structure enforces that features discriminative for live/spoof are simultaneously indistinguishable with respect to extraneous domains. Empirical results show superior generalization of live/spoof decision boundaries to new domains lacking in the training data (Saha et al., 2019).

4. DVD in Adversarial Video Generation (DVD-GAN)

In generative adversarial video modeling, a domain-agnostic discriminator is implemented via parallel spatial and temporal discriminators, each lacking hardwired flow or background priors and focused on generic consistency. The Dual Video Discriminator GAN (DVD-GAN) demonstrates this design:

Architecture

• Spatial discriminator $D_s$: receives $k$ randomly sampled frames, applies 2D convolutional (ResNet-based) stacks, outputs per-frame real/fake logits—enforcing high-fidelity still frames.
• Temporal discriminator $D_t$: processes the full video after 2×2 average pooling, uses 3D convs (early layers) then 2D convs, yields a single clip-level logit—enforcing temporal coherence.
• Both discriminators are trained together; no dataset-specific priors (e.g., flow estimation) are involved.

Losses

• Hinge-GAN objective for both $D_s$ and $D_t$, plus standard generator adversarial loss:

$$\begin{align*} L_{D_s} &= \mathbb{E}_{x\sim data} \sum_{i=1}^k \rho(1 - D_s(F_i(x))) + \mathbb{E}_{z} \sum_{i=1}^k \rho(1 + D_s(F_i(G(z)))) \ L_{D_t} &= \mathbb{E}_{x\sim data} \rho(1 - D_t(\phi(x))) + \mathbb{E}_{z} \rho(1 + D_t(\phi(G(z)))) \ L_G &= -\mathbb{E}_z \left[ \sum_{i=1}^k D_s(F_i(G(z))) + D_t(\phi(G(z))) \right] \end{align*}$$

• Training is performed on large-scale datasets (Kinetics-600, UCF-101) with heavy regularization and spectral normalization.

Generalization

This discriminator design is directly portable to new datasets, video types, and frame-lengths without architectural or hyperparameter retuning, establishing state-of-the-art video synthesis and prediction performance (e.g., IS = 32.97 on UCF-101, FVD = 69.2 on Kinetics-600) (Clark et al., 2019).

5. Graph-based Domain-Agnostic Video Prediction and Domain Adaptation

The Adversarial Bipartite Graph (ABG) learning framework applies a domain-agnostic principle at the level of cross-domain instance mixing:

Graph Construction and Message Passing

• A bipartite graph connects all source-domain frames to all target-domain frames with edge weights derived from a learned metric network.
• Message passing aggregates embeddings across domains at both frame and (optionally) video level, feeding the aggregated representation through further neural layers.

Conditional Adversarial Learning

• Beyond standard adversarial domain confusion, class labels (true/predicted) are embedded and concatenated to node features, and a conditional discriminator seeks to tell source from target.
• Alternating minimization is used: feature aggregator and classifier parameters are trained to minimize a sum of cross-entropy losses plus (negated) adversarial term; the domain discriminator is trained to maximize this term.

Semi-supervised Extension

• Video-level graphs and edge-supervision terms permit partially labeled target domains.
• Supervision via additional binary cross-entropy terms over edges whose source/target labels match.
• All graph reasoning is kept active at test time, enforcing symmetric domain-mixing and eliminating exposure bias.

This graph-based DVD approach yields significant boosts in domain adaptation accuracy, by up to 40% over standard adversarial domain adaptation, due to direct, instance-level cross-domain feature aggregation and conditional alignment (Luo et al., 2020).

6. Benchmarks, Analysis, and Practical Considerations

DVD frameworks have been validated on large and diverse video benchmarks:

Application Area	DVD Variant	Domain Types	Proxy Tasks/Evaluations
Robotic reward learning	Similarity-based DVD	Human/Robot	Task and env. zero-shot generalization; VMPC
Face anti-spoofing	Class-cond. domain disc.	Camera, lighting	Live/spoof accuracy on unseen datasets
GAN Video Synthesis	Dual spatio-temporal disc.	Action, domain	FVD, IS on Kinetics, UCF, BAIR
Domain adaptation	Bipartite graph w/ cond. DA	Source/Target	Recognition accuracy after UDA

Notable observations and limitations:

• DVD reward learning requires at least a seed set of robot samples per task for domain alignment (Chen et al., 2021).
• Domain-agnostic design greatly improves transfer to novel scenes or tasks with no retraining (Saha et al., 2019, Clark et al., 2019).
• Massive batch sizes and spectral normalization stabilize adversarial video synthesis (Clark et al., 2019).
• Graph-based DVD generalizes better by aligning class-conditional rather than marginal distributions (Luo et al., 2020).

Failures are observed for fine-grained or highly dexterous distinctions exceeding the capacity of fixed backbones, and for radical viewpoint mismatches between domains.

7. Connections and Extensions

DVD approaches unify several trends:

• Adversarial training for feature invariance,
• Metric learning via similarity discriminators,
• Graph-structured cross-domain instance propagation,
• Perceptual metrics for visual planning and control.

A plausible implication is that future DVD models will combine instance-level graph mixing, temporal architectures, and adversarial conditioning to push transfer capabilities closer to true open-domain generalization, especially as large-scale human action datasets and unsupervised video corpora continue to grow. Extending DVD-based reward learning to end-to-end policy learning and integrating with advanced video transformers remains an open research direction.