VideoFDB: Multifaceted Video Research Suite

• VideoFDB is a versatile research label applied to multiple video analysis tasks, including surveillance bird detection, full-duplex audiovisual conversations, and partial copy detection.
• In the flying bird detection context, the FBD-SV-2024 benchmark offers 483 clips with spatio-temporal annotations to tackle challenges like small, low-contrast, and dynamic targets.
• VideoFDB also underpins a KNN-based copy detection framework and an AV2AV conversational benchmark, advancing precise temporal localization and robust multimodal evaluation.

Searching arXiv for papers using the term "VideoFDB" to ground the article in the relevant literature. VideoFDB is a reused label in recent arXiv literature rather than a single standardized resource. It denotes a purpose-built benchmark for flying bird detection in surveillance videos, embodied by the FBD-SV-2024 dataset (Sun et al., 2024); the first benchmark for full-duplex audio-visual-to-audio-visual conversational agents (Mazumdar et al., 28 May 2026); and a video feature database framework for fast partial video copy detection (Tan et al., 2021). A video deflickering report also uses the phrase “the first end-to-end VideoFDB framework” for the combination of the DeViD dataset and VDFP (Zhou et al., 20 May 2026). The shared label therefore spans distinct problem formulations, supervision regimes, and evaluation protocols.

1. Terminological scope

In the cited literature, “VideoFDB” is attached to multiple technically unrelated artifacts. The following usages are explicit in the source papers (Sun et al., 2024, Mazumdar et al., 28 May 2026, Tan et al., 2021).

Usage	Artifact	Domain
VideoFDB / FBD-SV-2024	Dataset and benchmark	Flying bird detection in surveillance video
VideoFDB	Benchmark	Full-duplex AV2AV conversational agents
VideoFDB	Framework	Partial video copy detection using KNN and a global feature database

The shared acronym does not come with a common ontology, task definition, or metric family across these papers. This suggests that “VideoFDB” currently functions as a homonymous research label rather than a unified benchmark lineage. For technical reading, disambiguation by paper title and arXiv identifier is therefore necessary.

2. VideoFDB as FBD-SV-2024 for flying bird detection

In "FBD-SV-2024: Flying Bird Object Detection Dataset in Surveillance Video" (Sun et al., 2024), VideoFDB is embodied by a dataset comprising 483 short video clips, captured at $1{,}280\times720$ resolution and 25 fps, yielding a total of 28,694 extracted frames. Among these frames, 23,833 contain one or more flying bird instances, for an overall object count of 28,366; the remaining 4,861 frames serve as “negative” examples. Objects are localized with tight axis-aligned bounding boxes defined by $(x_{\min}, y_{\min}, x_{\max}, y_{\max})$, and the dataset uses a single class “bird” with WordNet ID n01503061. Each annotation includes two auxiliary fields: Difficulty $$\in \{0:\text{easy}, 1:\text{moderate}, 2:\text{difficult}, 3:\text{hard}\}$$, assigned by human judgment, and Track ID to link instances across frames. Labels are provided in both image-centric $(\texttt{bird\_i\_j.jpg})$ and video-centric $(\texttt{bird\_i/j.jpg})$ folder structures to support static and spatio-temporal detection pipelines.

The dataset was collected from outdoor fixed surveillance cameras deployed at various real-world sites, specifically airfields, substations, and farmland, during December 2023–May 2024. Manual screening isolated 483 segments likely to contain flying birds, with variation in background complexity, lighting, and occlusion. Frame extraction was automated, the annotation tool was LabelImg, and quality control used three-round cross-checking: initial bounding-box placement, a verification pass to catch omissions or gross errors, and fine-tuning of box edges and attribute fields. Each round was handled by a different annotator. No quantitative inter-annotator agreement such as $\kappa$ was reported, but the paper states that the cross-round design ensures high consistency.

The benchmark emphasizes three difficulties. First, about $36.7\%$ of bird instances are rated difficulty 2 or 3 and are barely distinguishable in a single frame, so human observers rely on motion cues across consecutive frames. Second, the scale distribution is heavily skewed toward small objects: $94\%$ of instances measure between $10\times10$ and $70\times70$ pixels in the full 720p frame, with $(x_{\min}, y_{\min}, x_{\max}, y_{\max})$0 below $(x_{\min}, y_{\min}, x_{\max}, y_{\max})$1, $(x_{\min}, y_{\min}, x_{\max}, y_{\max})$2 in $(x_{\min}, y_{\min}, x_{\max}, y_{\max})$3–$(x_{\min}, y_{\min}, x_{\max}, y_{\max})$4, and only $(x_{\min}, y_{\min}, x_{\max}, y_{\max})$5 above $(x_{\min}, y_{\min}, x_{\max}, y_{\max})$6. Third, birds are non-rigid in flight, with wing-flap cycles, rapid motion, and intermittent visibility caused by occlusions from poles and foliage.

A suite of modern object and video object detectors was retrained from scratch on the training split of 400 clips and 23,979 frames, and evaluated on the held-out test split of 83 clips and 4,715 frames. The benchmark uses

$(x_{\min}, y_{\min}, x_{\max}, y_{\max})$7

Pascal VOC 2007

$(x_{\min}, y_{\min}, x_{\max}, y_{\max})$8

and

$(x_{\min}, y_{\min}, x_{\max}, y_{\max})$9

Detector	AP$$\in \{0:\text{easy}, 1:\text{moderate}, 2:\text{difficult}, 3:\text{hard}\}$$0
YOLOv5l	0.558
YOLOv6l	0.585
YOLOXl	0.620
YOLOv8l	0.584
YOLOv9e	0.577
YOLOv10l	0.550
SSD	0.599
FGFA	0.198
SELSA	0.400
Temporal RoI Align	0.371
FBOD-BMI	0.692
FBOD-SV	0.719

Performance degrades sharply on small objects. YOLOXl’s $$\in \{0:\text{easy}, 1:\text{moderate}, 2:\text{difficult}, 3:\text{hard}\}$$1 is 0.280, whereas FBOD-SV reaches 0.313 by aggregating multi-frame features before extraction. Confusion-matrix analysis shows that static detectors such as YOLOXl and SSD suffer from many missed detections, while FBOD-SV lowers misses at the expense of increased false positives, specifically 398 on the test set. Within this usage, VideoFDB is therefore a surveillance-video benchmark centered on spatio-temporal detection of small, low-contrast, shape-varying targets.

3. VideoFDB as a benchmark for full-duplex AV2AV conversational agents

In "VideoFDB: Evaluating Full-Duplex Vision-Speech Capabilities in Conversational Agents" (Mazumdar et al., 28 May 2026), VideoFDB is defined as the first benchmark to evaluate full-duplex audio-visual-to-audio-visual conversational agents. The benchmark contains 237 dyadic video-call snippets, each centered on one annotated nonverbal event, drawn from natural two-person English video-conferencing sessions recorded locally at 720 p/30 fps video and 24 kHz audio. Clip length has median 46 s with IQR 34–61 s, and the event window has median 2.5 s with IQR 2–5 s. The 11 conversational dynamics are Pause Handling (13 clips), Gaze Avoidance with Pause (18), Nonverbal Interruption (8), Adaptor Handling (14), Face Emotion Display (50), Nonverbal Backchanneling (34), Laughter (30), Emotion Matching (13), Verbal Backchanneling (17), Verbal Interruption (24), and Turn-taking (16).

The benchmark introduces a taxonomy that separates Perception from Generation behaviors. Perception evaluates whether an agent correctly interprets a user’s nonverbal cue and decides when or whether to speak; its rubric axes are Fluency, Conversational Flow, and Semantic Grounding. Generation, used only for agents emitting video, evaluates Fluency, Dyadic Affect Match, and Nonverbal Cue Appropriateness. Examples given in the benchmark include gaze aversion during user thinking, where the agent perception task is to stay silent in-window, and hand-raise, where the perception task is to yield the floor within approximately 1.5 s.

Evaluation is rubric-based and LM-as-judge. For each clip, the user audio and video are streamed to the agent in real time; the resulting agent audio and video are recorded; transcripts with timestamps and automatically generated AV captions are assembled into a “judge payload”; and gpt-4o returns axis-wise scores from 0 to 5 with brief reasoning. The overall score is

$$\in \{0:\text{easy}, 1:\text{moderate}, 2:\text{difficult}, 3:\text{hard}\}$$2

where $$\in \{0:\text{easy}, 1:\text{moderate}, 2:\text{difficult}, 3:\text{hard}\}$$3 is the number of active axes. Timing is captured by TOR-Alignment. With

$$\in \{0:\text{easy}, 1:\text{moderate}, 2:\text{difficult}, 3:\text{hard}\}$$4

the benchmark defines

$$\in \{0:\text{easy}, 1:\text{moderate}, 2:\text{difficult}, 3:\text{hard}\}$$5

and reports it alongside median reaction latency.

The evaluated systems include closed-source AV2A models such as Gemini 2.5 Flash Native, Gemini 3.1 Flash Live, OpenAI Realtime mini, and OpenAI Realtime; open-source AV2A models such as MiniCPM-o 4.5, MiniOmni2, and VITA 1.5; audio-only counterparts; and cascaded speech-to-avatar systems. In key perception results, the human reference reaches Fluency 4.16, Flow 4.20, Grounding 4.24, Overall 4.20, and 90% TOR-Alignment at 1400 ms. MiniCPM-o 4.5 reaches the highest AV2A overall score at 3.40 with 73% TOR-Alignment at 720 ms, while Gemini 2.5 reaches Overall 3.17 with 72% at 3160 ms, OpenAI realtime-mini reaches 2.73 with 66% at 5320 ms, and MiniOmni2 reaches 1.19 with 64% at 3080 ms. No AV2A agent improves over its audio-only run on all perception axes; adding video often reduces TOR-Alignment by 0–5 percentage points and increases latency.

The paper identifies two systematic failure modes. “Captioning collapse” occurs when models treat user video as an image-captioning prompt rather than dialogue context; MiniOmni2 produces “Frame caption: …” in 87% of responses, and VITA1.5 combines 17% captioning with 74% generic “I don’t have vision” disclaimers. “Visual-stream ignorance” is exemplified by gpt-realtime-mini producing near-identical AV2A and audio-only transcripts. For cascaded speech-to-avatar systems, the paper states that avatar layers only generate motion after speech tokens are produced and therefore cannot insert listener cues such as nods or smiles during the user’s turn; reported latency of 2.8–3.5 s further precludes realtime nonverbal coordination. In this usage, VideoFDB is an interaction benchmark for streaming audiovisual grounding and timing-sensitive conversational behavior.

4. VideoFDB as a partial video copy detection framework

In "A Fast Partial Video Copy Detection Using KNN and Global Feature Database" (Tan et al., 2021), VideoFDB denotes a framework for partial video copy detection built around a global feature database. The system has three main stages: feature extraction, global database indexing, and query-time detection. Offline, it extracts one frame per second from every reference video, computes a CNN feature for each frame, optionally passes the sequence through a Transformer encoder to inject temporal context, and stores all normalized frame features in a global KNN-searchable database indexed by $$\in \{0:\text{easy}, 1:\text{moderate}, 2:\text{difficult}, 3:\text{hard}\}$$6. Frame descriptors evaluated include VGG16 pool5 with multiple pooling variants and ResNet-29 + RMAC, with ResNet-29 + RMAC outperforming VGG16 in the reported tests. Each feature vector $$\in \{0:\text{easy}, 1:\text{moderate}, 2:\text{difficult}, 3:\text{hard}\}$$7 is $$\in \{0:\text{easy}, 1:\text{moderate}, 2:\text{difficult}, 3:\text{hard}\}$$8-normalized: $$\in \{0:\text{easy}, 1:\text{moderate}, 2:\text{difficult}, 3:\text{hard}\}$$9

The global database is defined as

$(\texttt{bird\_i\_j.jpg})$0

where $(\texttt{bird\_i\_j.jpg})$1 is the feature of frame $(\texttt{bird\_i\_j.jpg})$2 from reference video $(\texttt{bird\_i\_j.jpg})$3. Because all vectors are $(\texttt{bird\_i\_j.jpg})$4-normalized, cosine distance

$(\texttt{bird\_i\_j.jpg})$5

is equivalent, up to monotonic transform, to Euclidean distance $(\texttt{bird\_i\_j.jpg})$6. The paper therefore notes that approximate-NN indexes such as FAISS Flat-CPU, Flat-GPU, and IVF-Flat-GPU can be used.

At query time, for each query frame feature $(\texttt{bird\_i\_j.jpg})$7, the system performs a top-$(\texttt{bird\_i\_j.jpg})$8 KNN search in $(\texttt{bird\_i\_j.jpg})$9, accumulates scores per reference video,

$(\texttt{bird\_i/j.jpg})$0

and keeps the top-$(\texttt{bird\_i/j.jpg})$1 candidates. For each candidate, it builds a sparse similarity matrix $(\texttt{bird\_i/j.jpg})$2 with $(\texttt{bird\_i/j.jpg})$3, retaining only entries among the top-$(\texttt{bird\_i/j.jpg})$4 matches with similarity above $(\texttt{bird\_i/j.jpg})$5. A modified temporal network then seeks the best diagonal path under step-size and diagonal-band constraints, producing $(\texttt{bird\_i/j.jpg})$6 and a normalized similarity

$(\texttt{bird\_i/j.jpg})$7

Experimental evaluation uses VCDB core, with 528 reference videos and 9 236 annotated copy-segments, under a segment-level detection protocol in which a detected segment is true positive if it overlaps ground truth in at least one frame. The reported segment-$(\texttt{bird\_i/j.jpg})$8 scores are 0.8613 without the Transformer and 0.8764 with the Transformer, compared with 0.8025 for MMTA, 0.6740 for LAMV, 0.6503 for CNN (ResNet-29 RMAC without KNN), and 0.6440 for MYLiu. Ablation results include approximately 0.901 $(\texttt{bird\_i/j.jpg})$9 for the reconstructed sparse similarity matrix plus modified temporal network on a 500-query subset, 0.8613 versus 0.7657 for ResNet-29 RMAC versus VGG-16-MP, and $\kappa$0 and $\kappa$1 $\kappa$2 gains for 2-head and 8-head Transformers. In runtime terms, KNN search per query frame for $\kappa$3 and a database of approximately 100k vectors takes about 20 ms with brute-force matrix multiplication and sorting, 2.55 ms with FAISS Flat-CPU, 1.48 ms with FAISS Flat-GPU, and 1.82 ms with FAISS IVF256-Flat-GPU. Here, VideoFDB is not a benchmark dataset but a retrieval-and-localization architecture built around a searchable global feature store.

5. Associated usage in video deflickering

The VDFP report (Zhou et al., 20 May 2026) is centered on "Video Deflickering with Flicker-banding Priors" rather than on a benchmark explicitly titled VideoFDB, but its conclusion states: “Together they establish the first end-to-end VideoFDB framework.” In that report, the relevant components are the DeViD dataset and the VDFP model. DeViD contains 108 clips, approximately 45 minutes of footage, with 60 clips borrowed from existing VSR benchmarks and 48 newly captured scenes on an LED-matrix projection rig. The capture protocol fixes a smartphone on a stand in front of a calibrated LED screen, varies positions, replays original clean content onscreen, and carefully synchronizes start and stop. Alignment uses border cropping and stretching, dynamic time-warping via grayscale-difference peaks, and Lab-space histogram matching.

VDFP itself combines a Degradation Field Modeling Based on Rolling Shutter Mechanism, a spatial-temporal continuous prior perception module, and a diffusion-based restoration pipeline. The degradation model defines a stripe-orthogonal coordinate, a time-dependent phase shift, and a dual-layer banding fusion with base and thick layers. The continuous prior perception module is a 3D extension of Swin-UNet that takes $\kappa$4 and predicts $\kappa$5. It is trained with Flicker-Aware Mean Squared Error,

$\kappa$6

where the target is the simulated luminance-drop amplitude map.

The restoration architecture uses a two-stage pipeline: pre-train the CPP Swin-UNet on simulated DFM data, then freeze a pre-trained text-to-video diffusion backbone, the STAR model, and append a slim “Video-ControlNet” finetuned to accept the predicted confidence maps. Condition injection enlarges the diffusion U-Net input from four channels to five with zero initialization,

$\kappa$7

which the report states guarantees no disruption of the frozen generative priors at the start of finetuning. Training uses AdamW with the same weight decay as STAR, a learning rate of $\kappa$8, 16-frame clips at $\kappa$9, batch size 1 clip per GPU on two NVIDIA A6000 GPUs, and finetuning until validation metrics plateau at approximately 50K iterations. On DeViD, VDFP is reported to reach SSIM 0.6974, LPIPS 0.2958, MUSIQ 48.89, BRISQUE 36.45, DISTS 0.1378, DOVER 0.6484, and $36.7\%$0, improving over the best non-VDFP baseline DLoRAL. This usage suggests a broader and less canonical application of the VideoFDB label in video restoration.

6. Comparison, misconceptions, and research directions

A common misconception would be to treat VideoFDB as a single dataset or benchmark. In the cited literature, it is not. One usage centers on object instances and bounding boxes in surveillance video (Sun et al., 2024); another on dyadic clips, rubric axes, and TOR-Alignment for streaming conversational behavior (Mazumdar et al., 28 May 2026); a third on frame descriptors, KNN retrieval, and diagonal-path localization for copy detection (Tan et al., 2021). The VDFP report adds an associated use tied to restoration priors and deflickering evaluation (Zhou et al., 20 May 2026). The differences are not cosmetic: they alter the unit of annotation, the temporal abstraction, the role of video context, and the definition of success.

The methodological contrasts are equally sharp. In FBD-SV-2024, temporal aggregation helps recover small, low-contrast targets that are often inconspicuous in single frames. In the AV2AV benchmark, full-duplex timing and nonverbal coordination are central, and adding video does not automatically improve perception scores or TOR-Alignment. In partial copy detection, the decisive component is a KNN-searchable global feature database followed by constrained temporal localization. In deflickering, the key design is a physics-informed degradation prior coupled to a zero-initialized conditional diffusion finetuning. The shared name therefore masks substantial heterogeneity in model class, data regime, and evaluation semantics.

The research directions proposed by the respective papers also diverge. For flying bird detection, the recommendations include motion-guided warping at early layers, dynamic label assignment, anchor-free heads, lightweight optical flow or motion attention modules, and semi-supervised or self-training approaches leveraging negative frames (Sun et al., 2024). For full-duplex AV2AV evaluation, the stated next steps are end-to-end AV2AV models that jointly synthesize speech and nonverbal motion with sub-second temporal alignment, increased video sampling rates, improved multimodal fusion, broader benchmark coverage, and safety and misuse guardrails (Mazumdar et al., 28 May 2026). For partial copy detection, future work includes learned indexing such as product-quantization and OPQ, end-to-end training of the Transformer plus temporal localization, and more robust spatio-temporal features such as 3D-CNNs and cross-frame attention (Tan et al., 2021). For deflickering, the limitations note a current focus on LED-matrix displays and propose extension to LCD/OLED panels, global-shutter cameras, and real-time on-device deployment (Zhou et al., 20 May 2026).

Taken together, these works show that VideoFDB is best understood as a multivalent term in contemporary video research. Its meanings range from surveillance-video detection benchmark, to full-duplex conversational evaluation suite, to feature-database retrieval framework, with an additional associated use in restoration. Accurate interpretation therefore depends on domain context and citation rather than on the label alone.