DensSiam: Dense Siamese Tracker

• DensSiam is an end-to-end Siamese visual tracker that uses dense connectivity to enhance information flow and mitigate vanishing gradients.
• It integrates a Self-Attention module in the target branch to capture non-local semantic context, boosting robustness against appearance changes.
• Empirical evaluations on VOT benchmarks demonstrate enhanced accuracy and robustness while maintaining real-time performance at 60 FPS.

Searching arXiv for the cited DensSiam papers and closely related context. DensSiam is an end-to-end Siamese visual tracker for single-object tracking that combines dense connectivity with a Self-Attention module in order to address two limitations identified in prior Siamese trackers: the difficulty of making Siamese backbones deeper while preserving generalization, accuracy, speed, and parameter efficiency, and the locality of conventional convolutional features, which makes the learned appearance model less robust to substantial appearance changes such as blur, occlusion, clutter, or deformation (Abdelpakey et al., 2018). In the formulation used by the model, an exemplar patch from the first frame and a larger search region from later frames are mapped into feature spaces, and a similarity-learning function produces a response map whose maximum indicates the target location.

1. Problem formulation and motivation

DensSiam is situated in the fully convolutional Siamese-tracking paradigm. In this setting, the tracker receives an exemplar or target patch from the first frame and a larger search image from subsequent frames. A Siamese network processes both inputs, and a correlation-based similarity stage produces a response map over candidate positions. The method treats SiamFC and related designs as strong baselines because they are fast and fully convolutional, yet it argues that conventional Siamese trackers remain constrained by local receptive-field processing and by the difficulty of scaling backbone depth without increasing the burden of shared parameters or aggravating optimization issues such as vanishing gradients (Abdelpakey et al., 2018).

The paper’s central motivation is therefore twofold. First, convolutional Siamese trackers typically compute similarity from local neighborhoods, so the resulting appearance model is largely local and may not capture broader semantic context or long-range dependencies. Second, deeper Siamese architectures are difficult to construct in a way that simultaneously preserves real-time speed, high accuracy, generalization across domains, and parameter efficiency. DensSiam is proposed as a joint response to both issues: dense connectivity is used to improve information flow, feature reuse, and gradient propagation in a deeper Siamese backbone, while Self-Attention is used to produce a non-local target representation that better captures semantic context.

This design perspective places robustness to appearance variation at the center of the method. The paper explicitly motivates the use of non-local features for handling motion blur, occlusion, clutter, deformation, and illumination changes. A plausible implication is that DensSiam treats the target template not merely as a local descriptor but as a context-enriched representation intended to remain stable under visual perturbation.

2. Network organization and branch asymmetry

The DensSiam architecture contains two branches. The target branch processes the exemplar image from the first frame, and the search branch processes the search region in subsequent frames. The two branches share the same network architecture and “share the same network parameters across channels extracted from the same dense layers,” except that the target branch includes a Self-Attention module and the search branch does not (Abdelpakey et al., 2018). Structurally, the model is therefore almost symmetric, but the target side is augmented with non-local modeling.

For the target branch, the paper summarizes the architecture as

$$\text{input} \rightarrow \text{ConvNet} \rightarrow \text{DenseBlock}_1 \rightarrow \text{TransitionLayer} \rightarrow \text{DenseBlock}_2 \rightarrow \text{TransitionLayer} \rightarrow \text{DenseBlock}_3 \rightarrow \text{DenseBlock}_4 \rightarrow \text{SelfAttention}.$$

The search branch has the same sequence but omits the final Self-Attention stage. The exemplar input size is $127 \times 127$, and the search input size is $255 \times 255$.

Stage	Operation	Output
Initial convolution	$7 \times 7$ conv, stride 2	$(8, 61, 61, 72)$
Dense Block 1	$[1 \times 1 \text{ conv}] \times 2$, $[3 \times 3 \text{ conv}] \times 2$	$(8, 61, 61, 144)$
Transition Layer 1	$1 \times 1$ conv + average pool	$(8, 30, 30, 36)$
Dense Block 2	$127 \times 127$0, $127 \times 127$1	$127 \times 127$2
Transition Layer 2	$127 \times 127$3 conv + average pool	$127 \times 127$4
Dense Block 3	$127 \times 127$5, $127 \times 127$6	$127 \times 127$7
Dense Block 4	$127 \times 127$8	$127 \times 127$9
Self-Attention	$255 \times 255$0	$255 \times 255$1

The correlation layer receives the Self-Attention-enhanced target representation and the search representation, and computes the similarity function between the target patch with non-local features and candidate patches in the search image. Because the search image is larger than the exemplar, multiple candidate positions are evaluated in one forward pass, following the fully convolutional Siamese-tracking pattern.

3. Dense connectivity and non-local target modeling

Dense connectivity is the core structural principle of DensSiam. Rather than using purely sequential propagation,

$255 \times 255$2

the model adopts DenseNet-style concatenative connectivity within each dense block:

$255 \times 255$3

where $255 \times 255$4 is the input to the block, $255 \times 255$5 is the output of the $255 \times 255$6 layer, $255 \times 255$7 is a composite transformation of batch normalization, ReLU, and a $255 \times 255$8 convolution, and $255 \times 255$9 denotes concatenation of all preceding feature maps (Abdelpakey et al., 2018). The dense blocks use BN, ReLU, pooling, and convolution layers, and the repeated $7 \times 7$0 and $7 \times 7$1 convolutions shown in the architecture table constitute a bottleneck-style dense construction.

The paper argues that this dense connectivity preserves useful information from all earlier layers, improves information flow, increases generalization capability, leverages feature reuse, and helps tackle vanishing gradients. The reported channel growth across dense blocks—such as from 72 to 144 in Dense Block 1, from 36 to 180 in Dense Block 2, and from 36 to 252 in Dense Block 3—is consistent with the idea that each new layer contributes additional feature maps by concatenation rather than by summation. This suggests that the tracker’s descriptor at correlation time retains low-level detail, mid-level texture and shape cues, and higher-level semantic information simultaneously.

Between dense blocks, DensSiam inserts transition layers consisting of a convolutional layer, average pooling, and dropout. These layers reduce spatial size and channel count for efficiency, and they also align feature-map sizes because the network does not use padding. The paper explicitly states that DensSiam avoids padding because it is fully convolutional and padding would violate that property as used in the design. Since feature maps therefore shrink through convolutions, the transition layers are needed to match dimensions for downstream processing. Dropout in dense blocks and transition layers is also described as reducing overfitting, especially to negative classes.

Self-Attention is the second key component. It appears only in the target branch, after the final dense block. Its purpose is to convert the target representation from a purely local convolutional descriptor into a non-local one. The input is $7 \times 7$2, and in the implemented architecture the module operates on a tensor of size $7 \times 7$3. The module forms three $7 \times 7$4 projections,

$7 \times 7$5

computes affinities

$7 \times 7$6

normalizes them into attention weights

$7 \times 7$7

and produces a weighted non-local aggregation

$7 \times 7$8

The manuscript’s notation is described as imperfectly typeset, but the intended meaning is a softmax-like affinity normalization followed by weighted aggregation over all positions. Although the paper does not use explicit query-key-value terminology, the formulation maps naturally onto that interpretation. The significance for tracking is that the target embedding can incorporate information from all target positions rather than only from local neighborhoods.

4. Training objective, optimization, and runtime behavior

DensSiam is trained offline from scratch on the ILSVRC15 video object detection dataset, described as 1.3 million labeled frames in 4000 sequences (Abdelpakey et al., 2018). The optimization settings are: 100 epochs, 53,200 sampled pairs per epoch, SGD with momentum $7 \times 7$9, mini-batch size 8, and a learning rate annealed geometrically each epoch from $(8, 61, 61, 72)$0 to $(8, 61, 61, 72)$1. The implementation environment is TensorFlow 1.8 on a Xeon E5 2.20 GHz CPU and a Titan XP GPU.

The training loss is logistic. For an individual target-candidate score,

$(8, 61, 61, 72)$2

where $(8, 61, 61, 72)$3 is the score for a target-candidate pair and $(8, 61, 61, 72)$4 is the ground-truth label. For the full score map, the loss is averaged over all positions:

$(8, 61, 61, 72)$5

The paper states that logistic loss is used “for both dense block and Self-Attention model to calculate the weights using SGD,” indicating that the entire architecture, including the attention module, is trained end-to-end through the tracking objective.

At inference time, the tracker extracts the target patch from the first frame, passes it through the target branch to form the Self-Attention-enhanced exemplar embedding, crops a search region around the prior target location in each new frame, processes that region through the search branch, correlates the two features, and selects the location with the highest response as the target position. Scale is handled by a three-scale search:

$(8, 61, 61, 72)$6

with linear interpolation factor $(8, 61, 61, 72)$7. The reported speed is 60 FPS, which the paper treats as real-time performance.

The design rationale for runtime efficiency is explicit. DensSiam remains fully convolutional, uses dense feature reuse rather than relearning redundant filters, compresses features with transition layers, inserts Self-Attention only in the target branch, and performs feed-forward evaluation during online tracking after offline training. The paper does not report an explicit parameter count, but it claims that dense connectivity reduces the burden of shared parameters relative to deeper conventional Siamese architectures.

5. Empirical evaluation on OTB and VOT

The reported evaluation covers OTB2013 and OTB2015 as validation benchmarks, and VOT2015, VOT2016, and VOT2017 as testing benchmarks (Abdelpakey et al., 2018). For the VOT datasets, the paper uses Accuracy ($(8, 61, 61, 72)$8), Robustness ($(8, 61, 61, 72)$9), and Expected Average Overlap (EAO). Lower robustness is better because it reflects fewer failures.

On VOT2015, DensSiam reports $[1 \times 1 \text{ conv}] \times 2$0, $[1 \times 1 \text{ conv}] \times 2$1, $[1 \times 1 \text{ conv}] \times 2$2, and $[1 \times 1 \text{ conv}] \times 2$3. The paper compares these results against MDNet, DeepSRDCF, EBT, SRDCF, BACF, EAST, Staple, and SiamFC. According to the provided comparison, DensSiam improves over SiamFC from $[1 \times 1 \text{ conv}] \times 2$4 to $[1 \times 1 \text{ conv}] \times 2$5 in accuracy, from $[1 \times 1 \text{ conv}] \times 2$6 to $[1 \times 1 \text{ conv}] \times 2$7 in robustness, and from $[1 \times 1 \text{ conv}] \times 2$8 to $[1 \times 1 \text{ conv}] \times 2$9 in EAO, while remaining real-time though slower than SiamFC’s 86 FPS.

On VOT2016, DensSiam reports $[3 \times 3 \text{ conv}] \times 2$0, $[3 \times 3 \text{ conv}] \times 2$1, $[3 \times 3 \text{ conv}] \times 2$2, and $[3 \times 3 \text{ conv}] \times 2$3, compared against ECOhc, Staple, STAPLE+, SiamRN, and GCF. In the reported table, this is the best EAO and the best robustness, with $[3 \times 3 \text{ conv}] \times 2$4 exceeding ECOhc’s $[3 \times 3 \text{ conv}] \times 2$5 and robustness $[3 \times 3 \text{ conv}] \times 2$6 improving on ECOhc’s $[3 \times 3 \text{ conv}] \times 2$7, Staple’s $[3 \times 3 \text{ conv}] \times 2$8, and SiamRN’s $[3 \times 3 \text{ conv}] \times 2$9.

On VOT2017, DensSiam reports $(8, 61, 61, 144)$0, $(8, 61, 61, 144)$1, $(8, 61, 61, 144)$2, and $(8, 61, 61, 144)$3, compared against SiamDCF, ECOhc, CSRDCF++, SiamFC, SAPKLTF, Staple, and ASMS. In the reported comparison, DensSiam again has the strongest result in the table, with the best accuracy, best robustness, and best EAO. Relative to SiamFC, the reported gains are $(8, 61, 61, 144)$4 versus $(8, 61, 61, 144)$5 in accuracy, $(8, 61, 61, 144)$6 versus $(8, 61, 61, 144)$7 in robustness, and $(8, 61, 61, 144)$8 versus $(8, 61, 61, 144)$9 in EAO.

For OTB2013 and OTB2015, the paper states that experiments were performed and claims strong performance, but the provided text does not include the actual OTB result tables or plots. Consequently, the available evidence in the present record is numerical for the VOT benchmarks and qualitative for OTB.

A notable feature of the evaluation is the absence of explicit ablation tables isolating dense connectivity and Self-Attention. The text argues conceptually that dense connectivity improves generalization, feature reuse, and gradient flow, and that Self-Attention improves non-local semantic representation and robustness, but it does not provide a dedicated numerical decomposition of dense-only versus dense-plus-attention variants. Any separation of the contributions is therefore architectural and interpretive rather than experimentally isolated within the reported material.

6. Scope, limitations, and naming ambiguity

The paper explicitly claims three contributions: a new end-to-end densely Siamese architecture for tracking, a Self-Attention-based response representation that captures semantic information about the target object, and better training dynamics and generalization through improved feature propagation, feature reuse, and mitigation of vanishing gradients (Abdelpakey et al., 2018). In practical terms, the distinguishing characteristics relative to earlier Siamese trackers are the densely connected backbone, non-local target modeling via Self-Attention, a more expressive target representation before correlation, and real-time operation at 60 FPS.

Several limitations are either stated indirectly or follow from the reported design. DensSiam does not include explicit online adaptation beyond the standard search and scale updates, so it primarily relies on an offline-learned similarity function and a fixed target embedding from the first frame. Self-Attention is applied only on the target branch, which is computationally efficient but leaves the search branch local. The paper does not provide explicit ablations, parameter counts, memory-cost analysis, failure-case visualizations, or attribute-wise breakdowns in the provided text. Scale handling remains a standard three-scale search rather than a more specialized scale-estimation module, and the benchmark scope is limited to short-term tracking rather than long-term tracking, re-detection, or disappearance-and-reappearance scenarios.

A common source of confusion is nomenclature. “DensSiam” in the 2018 tracking literature denotes the end-to-end densely Siamese network with Self-Attention for object tracking (Abdelpakey et al., 2018). It should be distinguished from “DenseSiam,” a 2022 self-supervised or unsupervised learning framework for dense prediction tasks such as semantic segmentation, object detection, and instance segmentation, where the central ideas are pixel consistency and region consistency rather than Siamese visual tracking (Zhang et al., 2022). The similarity of names obscures the fact that the two works address different problem classes, use different objectives, and belong to different methodological lineages.

Within the object-tracking context, DensSiam’s principal significance lies in its attempt to combine DenseNet-style feature reuse with non-local target modeling inside a fully convolutional Siamese tracker. The reported empirical message is that a denser backbone and a Self-Attention-enhanced exemplar can improve accuracy and robustness on VOT benchmarks while preserving real-time throughput.