PlanarTrack: Benchmark for Planar Tracking

Updated 4 July 2026

PlanarTrack is a benchmark family for planar object tracking that utilizes four-corner annotations to estimate homography transformations.
It evolved from a 2023 dataset with 1,000 videos to expanded versions with over 733K frames, enabling both short- and long-term evaluations.
Benchmark evaluations via metrics like p@5 and p@15 highlight challenges such as occlusion, motion blur, and dynamic planar surfaces in real-world scenes.

Searching arXiv for the benchmark and the 2026 tracking paper to ground the article in current papers. I found the benchmark lineage and the later evaluation paper in the provided arXiv records, including the original benchmark (Liu et al., 2023), the expanded benchmark (Jiao et al., 27 Oct 2025), and the 2026 study that revisits PlanarTrack annotations and reports new state of the art (Serych et al., 23 Feb 2026). PlanarTrack is a benchmark family for planar object tracking, a task in which the target is a planar region and the output is its geometric state as a quadrilateral or homography rather than an axis-aligned box. In the original benchmark, introduced in 2023, the objective was to provide a large-scale, realistic testbed for tracking planar objects in unconstrained scenes, with 1,000 videos, more than 490K images, and dense four-corner annotation. A later benchmark paper with the same name expanded this design to 1,150 sequences and over 733K frames, explicitly adding long-term evaluation and preserving the principle that each sequence contains a unique target (Liu et al., 2023, Jiao et al., 27 Oct 2025).

1. Task definition and geometric scope

PlanarTrack is built around planar object tracking rather than generic object tracking. Generic tracking predicts an axis-aligned bounding box, whereas planar tracking estimates a 2D transformation, typically a homography, between the initial target and the current frame, and localizes the target by four corner points. In the formulation used by later work on the benchmark, given video frames $(I_t)_{t=0}^T$ and target control points $X_0$ in frame $I_0$ , the goal is to estimate homographies $H_t$ such that

$X_t = \mathcal{W}({H}_t, X_0),$

where $\mathcal{W}(\cdot,\cdot)$ denotes warping by the homography; the pose is treated as an 8 degrees-of-freedom homography transformation, with $H_t \in \mathbb{R}^{3\times 3}$ and one degree fixed by scale (Serych et al., 23 Feb 2026).

This task definition determines both annotation and evaluation. Each target is represented by four control points forming a quadrilateral. The original benchmark explicitly motivates this representation by noting that four points are the least number of points needed to determine the planar transformation, and all subsequent analysis on PlanarTrack treats the four corners as the canonical observable state. A plausible implication is that PlanarTrack is not merely a dataset for localizing rigid objects; it is a benchmark for image-to-image planar registration under severe nuisance factors.

2. Benchmark construction and version history

The 2023 benchmark was introduced as a large-scale challenging dataset dedicated to planar object tracking. It contains 1,000 videos, 1,000 unique planar targets, and more than 490K densely annotated frames. All videos were collected in complex unconstrained scenarios from the wild, and each target appears in only one video. The authors recorded the videos themselves using smartphones, initially collecting over 2,500 videos and then manually curating them down to 1,000 usable sequences after expert inspection and removal of unsuitable parts (Liu et al., 2023).

A later benchmark paper, also named PlanarTrack, enlarges and restructures the dataset. It reports 1,150 sequences, over 733K densely annotated frames, and 1,150 unique targets, organized as 1,000 short-term sequences and 150 long-term sequences. It also formalizes a train/test split of 805 training sequences and 345 test sequences, with the long-term component explicitly designed to include frequent target enter/leave events (Jiao et al., 27 Oct 2025).

Benchmark version	Scale	Distinctive structure
2023 PlanarTrack	1,000 videos, 490K frames, 1,000 unique targets	700/300 split; short-term emphasis
2025 PlanarTrack	1,150 sequences, 733K frames, 1,150 unique targets	1,000 short-term + 150 long-term; 805/345 split

Both versions emphasize realism through scenario and target diversity. The 2023 benchmark lists targets such as box, poster, picture, board, logo, door, mirror, book, traffic sign, tile, wall, screen, and table, captured in locations including shopping mall, street, library, restaurant, supermarket, playground, park, museum, apartment, hall, and classroom (Liu et al., 2023). The later benchmark broadens this into 21 target classes and 19 scenarios, explicitly including mirrors, screens, and transparent plates, which are atypical but operationally important planar targets because their geometry remains planar while their appearance may be unstable (Jiao et al., 27 Oct 2025).

3. Annotation model and evaluation protocol

PlanarTrack uses four-corner annotation as its fundamental representation. In the 2023 benchmark, each frame is manually labeled with four corner points if all four corners or all four edges are clearly visible; otherwise, if corners and edges are unavailable because of occlusion, out-of-view, or severe blur, the frame is assigned an absent flag. Annotation follows a multi-stage verification protocol: a sequence is first annotated by a volunteer, then checked by two experts, and returned for refinement if unanimous agreement is not reached (Liu et al., 2023).

The later benchmark retains the same geometric representation but strengthens annotation quality control. It uses a custom MATLAB annotation tool with zoom-in support, trains annotators on difficult cases, and applies multi-round verification until unanimous approval. It also reports an explicit annotation-quality study based on 10,920 re-annotated frames: the mean alignment error between original GT and refined GT is 5.71 pixels, 10.71% of annotations exceed 15 pixels, and 75.98% fall below 5 pixels (Jiao et al., 27 Oct 2025).

Evaluation protocols changed across the benchmark lineage. The 2023 version follows POT-style metrics, using PRE, the percentage of frames where alignment error between predicted and ground-truth corner points is within a threshold, and SUC, the percentage of successful frames in which the discrepancy between estimated and real homography is smaller than or equal to a threshold. The later benchmark explicitly de-emphasizes SUC, arguing that it depends strongly on target location in the image, and instead reports planar precision at fixed pixel thresholds, especially $P@5$ and $P@15$ (Liu et al., 2023, Jiao et al., 27 Oct 2025).

A later study that uses PlanarTrack as its main benchmark makes the alignment criterion explicit. It defines the alignment error between the estimated homography $H$ and ground truth $X_0$ 0 on control points $X_0$ 1 as

$X_0$ 2

with the intended meaning being the RMSE of the four warped control-point displacements. From this, $X_0$ 3 is the fraction of frames whose alignment error is below 5 pixels, and $X_0$ 4 is the fraction below 15 pixels (Serych et al., 23 Feb 2026).

4. Challenge taxonomy and empirical difficulty

The original benchmark defines eight challenge factors: occlusion, motion blur, rotation, scale variation, perspective distortion, out-of-view, low resolution, and background clutter. Perspective distortion is the most common challenge, while low resolution and motion blur are highlighted as especially difficult. The authors further show that PlanarTrack targets are generally smaller and move faster than those in POT-210 and POT-280, and that adjacent-frame overlap is lower, all of which increase difficulty (Liu et al., 2023).

The later benchmark preserves most of this structure but adds light interactive surface as an explicit attribute, covering mirrors, transparent plates, and screens. Although one sentence says “eight challenging factors,” the paper in fact lists nine: OCC, MB, ROT, SV, PD, OV, LR, BC, and LIS. It also states that 1,135 out of 1,150 sequences contain multiple challenge factors simultaneously, making multi-factor stress conditions the rule rather than the exception (Jiao et al., 27 Oct 2025).

The benchmark’s empirical role is to expose degradation that earlier planar benchmarks obscure. On the 2023 test set, WOFT achieves PRE $X_0$ 5 and SUC $X_0$ 6, HDN achieves $X_0$ 7, and GIFT achieves $X_0$ 8, while classical direct methods such as IC and ESM are markedly weaker. The paper emphasizes the large drop from POT-210 to PlanarTrack; for WOFT, the score decreases from $X_0$ 9 on POT-210 to $I_0$ 0 on PlanarTrack $I_0$ 1 (Liu et al., 2023).

The 2025 benchmark reaches the same conclusion with a larger and partly long-term protocol. On the full test set, WOFT reports $I_0$ 2 and $I_0$ 3, HDN $I_0$ 4, and GIFT $I_0$ 5. The long-term split is substantially harder: WOFT drops from $I_0$ 6 to $I_0$ 7 in $I_0$ 8 between the short-term and long-term test subsets, and HDN drops from $I_0$ 9 to $H_t$ 0. The paper identifies low resolution and light interactive surfaces as particularly damaging conditions, with LIS introduced precisely because planar geometry can remain stable while appearance changes due to reflection, refraction, or display content (Jiao et al., 27 Oct 2025).

A recurring misconception is that rigid planar objects should be easy for generic trackers. PlanarTrack argues against this directly through PlanarTrack $H_t$ 1, a derived benchmark that converts four-corner annotations into axis-aligned boxes and evaluates generic trackers. In both the 2023 and 2025 benchmark lines, generic trackers degrade substantially relative to standard box-tracking datasets, suggesting that planar targets are difficult even when the task is relaxed from homography-level tracking to box localization (Liu et al., 2023, Jiao et al., 27 Oct 2025).

5. Role in later tracker development and benchmark revision

PlanarTrack became a primary stress test for long-term planar tracking in later methodological work. The 2026 paper on SAM-H and WOFTSAM describes PlanarTrack as the main challenging benchmark for long-term planar tracking and emphasizes that, unlike older datasets dominated by textured rigid surfaces, it contains difficult cases such as extreme scale changes, distractors, reflections, transparent objects, dynamic appearance changes, and unconventional targets such as virtual planar regions (Serych et al., 23 Feb 2026).

That paper also illustrates how benchmark composition affects method rankings. On PlanarTrack $H_t$ 2, it reports WOFT at $H_t$ 3 on $H_t$ 4, HVC-Net at $H_t$ 5, WOFTSAM at $H_t$ 6, and SAM-H at $H_t$ 7. The notable result is that SAM-H, a segmentation-boundary-based homography tracker, exceeds WOFTSAM and all previous correspondence-based trackers on raw PlanarTrack aggregate scores. The explanation given is benchmark-specific: many PlanarTrack targets are mirrors, transparent surfaces, dynamic displays, or virtual planar regions, so appearance-based matching can lock onto the wrong planar content, whereas a segmentation-driven boundary tracker remains aligned to the annotated surface itself (Serych et al., 23 Feb 2026).

The same paper also turns PlanarTrack into a case study in annotation sensitivity. It argues that original initial-frame annotations are not precise enough for evaluating high-precision trackers with $H_t$ 8, and therefore re-annotates all PlanarTrack $H_t$ 9 initial frames at single-pixel or subpixel precision. The mean alignment error between original and re-annotated initial frames is reported as only 1.9 pixels, yet the effect on $X_t = \mathcal{W}({H}_t, X_0),$ 0 is substantial for correspondence-based methods, because they align current frames to the initial frame and propagate initialization error throughout the sequence. Under the revised initialization, WOFT changes to $X_t = \mathcal{W}({H}_t, X_0),$ 1, WOFTSAM to $X_t = \mathcal{W}({H}_t, X_0),$ 2, and SAM-H to $X_t = \mathcal{W}({H}_t, X_0),$ 3 on $X_t = \mathcal{W}({H}_t, X_0),$ 4 (Serych et al., 23 Feb 2026).

This revision clarifies a methodological point that is easy to miss: benchmark difficulty in PlanarTrack is not determined only by scene conditions but also by the precision of the initial quadrilateral. A plausible implication is that comparisons at strict thresholds, especially $X_t = \mathcal{W}({H}_t, X_0),$ 5, measure both tracker quality and initialization fidelity.

6. Significance, limitations, and open directions

PlanarTrack’s main significance is that it recasts planar tracking as a large-scale benchmark problem rather than a small benchmark afterthought. The 2023 paper frames this explicitly as a response to the deep-learning era, arguing that prior planar benchmarks were too small, too lab-like, too repetitive in target identity, and too weakly challenging to support modern methods. The 2025 extension strengthens this position by adding long-term sequences, larger scale, denser annotation, and a target distribution that includes visually unstable but geometrically planar objects (Liu et al., 2023, Jiao et al., 27 Oct 2025).

The benchmark also exposes two objective controversies. The first concerns evaluation: the later benchmark rejects SUC as its main metric because homography-error thresholds depend strongly on image position, while the 2026 re-annotation study shows that $X_t = \mathcal{W}({H}_t, X_0),$ 6 can be distorted by initialization noise. The second concerns what should count as a strong planar tracker: PlanarTrack contains cases where segmentation-driven geometry outperforms appearance-driven correspondence, which complicates the older assumption that precise homography fitting from correspondences is the dominant solution class (Jiao et al., 27 Oct 2025, Serych et al., 23 Feb 2026).

The original benchmark had an acknowledged limitation: its videos were relatively short and therefore not well suited to long-term tracking. The expanded benchmark addresses this directly with 150 long-term sequences, but the resulting evaluations show that long-term planar tracking remains weak, especially under disappearance/reappearance, low resolution, and light-interactive surfaces. The future directions stated across the benchmark line are correspondingly concrete: better handling of low resolution, better handling of motion blur or light-interactive appearance change, stronger temporal modeling, improved re-detection, and, in the later benchmark, possible integration of multi-modal cues such as depth or inertial data (Liu et al., 2023, Jiao et al., 27 Oct 2025).

In that sense, PlanarTrack functions not only as a dataset but also as a boundary object for the subfield. It formalizes planar tracking as four-corner or homography estimation in unconstrained scenes, provides a benchmark structure that reveals failure modes hidden by earlier datasets, and has already forced methodological revisions in both tracking design and evaluation practice.