CAMERA-Q: Coordinated Multi-Camera Tracking

• CAMERA-Q is a coordinated multi-camera system for active multi-object tracking that transforms per-camera 2D detections into a shared world coordinate frame.
• It formulates camera control as a centralized, partially observable multi-agent Markov decision process with a discrete 27-way action space over translation, rotation, and zoom.
• Using reward shaping and a factorized Double DQN, CAMERA-Q improves target coverage by nearly 9% compared to static camera baselines in simulated environments.

CAMERA-Q is a coordinate-aligned multi-camera collaboration system for Active Multi-Object Tracking (AMOT) that formulates camera control as a centralized partially observable multi-agent Markov decision process and learns a value-based policy to maximize the fraction of mobile targets covered by at least one camera at any time (Fang et al., 2022). Its defining idea is to transform per-camera detections into a shared world coordinate frame before decision-making, so that the controller reasons over global target and camera geometry rather than over disjoint image-plane observations. In the reported formulation, each camera is treated as an agent, but control remains centralized: the controller aggregates all camera observations and outputs all per-camera actions jointly.

1. Task model and camera-control scope

CAMERA-Q is defined for AMOT, a setting in which multiple cameras are controlled collaboratively so as to maximize coverage of moving targets in a shared visual field (Fang et al., 2022). The system is explicitly modeled as a centralized partially observable multi-agent Markov decision process. At discrete time $t$, camera $i$ receives an observation $o_{i,t}$, the controller forms the joint observation $o_t=\langle o_{1,t},\dots,o_{n,t}\rangle$, and then outputs per-camera actions $a_{i,t}$.

A common misconception is that CAMERA-Q is a decentralized multi-agent system. The formulation states the opposite: the controller obtains all cameras’ observations and issues all actions, so this is centralized control, not decentralized execution. This distinction matters because the method depends on joint state construction, shared reward structure, and centralized value estimation.

The action space is discrete and factorized into three independent dimensions: translation along the court boundary, yaw rotation, and zoom. For each camera, translation is chosen from clockwise, stay, or anticlockwise; rotation from left, stay, or right; and zoom from in, stay, or out. Each agent’s joint action is therefore $a=[a_m,a_r,a_z]$ with $3\times3\times3=27$ combinations. In the reported environment, the translation step is 100 world units along the boundary, the rotation step is 10 degrees, and zoom changes by 10% multiplicative increments (Fang et al., 2022).

The camera model used in the environment is fully calibrated. Each camera produces a $640\times480$ RGB frame with a $90^\circ$ horizontal field of view and is positioned on the court boundary at a height of 500 world units. Intrinsics $K$ and extrinsics $i$0 are assumed known at every step. This calibrated geometry is not peripheral; it is the mechanism that makes coordinate alignment possible.

2. Coordinate alignment and observation construction

CAMERA-Q constructs a compact, world-aligned observation in three stages (Fang et al., 2022). First, a lightweight detector, YOLOv4-Tiny, processes each image $i$1 and outputs 2D bounding boxes $i$2 and confidences for visible targets. Second, it applies a pinhole projection model specialized to the ground plane. With a world point $i$3 and image pixel $i$4,

$i$5

Because all targets live on the ground plane $i$6, the mapping reduces to

$i$7

where $i$8 are the first two columns of $i$9. Letting $o_{i,t}$0, the inverse mapping is

$o_{i,t}$1

Third, CAMERA-Q uses the bottom-center of each detection as a footpoint heuristic,

$o_{i,t}$2

and estimates the world coordinate as

$o_{i,t}$3

This produces per-camera estimates of absolute 2D ground-plane positions for detected targets. Because $o_{i,t}$4 are known from the simulator, detections from all cameras can be aligned into a shared world frame. The reported mean Euclidean error of this estimate is 29.6 world units with standard deviation 8.04 on a $o_{i,t}$5 field, which is described as small relative to field size (Fang et al., 2022).

For each camera $o_{i,t}$6, the observation $o_{i,t}$7 concatenates detected target coordinates in the world plane, camera posture $o_{i,t}$8, yaw angle $o_{i,t}$9, zoom level $o_t=\langle o_{1,t},\dots,o_{n,t}\rangle$0, and $o_t=\langle o_{1,t},\dots,o_{n,t}\rangle$1 distances to other cameras. All agents’ partial observations are then shared and concatenated, together with per-agent one-hot identifiers and last actions, into the joint observation processed by the Q-network. This means that the learning problem is not image-based end to end; it is geometry-based after a detection-and-alignment front end.

3. Coverage metric and reward design

The global objective is target coverage. A target $o_t=\langle o_{1,t},\dots,o_{n,t}\rangle$2 is considered covered by camera $o_t=\langle o_{1,t},\dots,o_{n,t}\rangle$3 at time $o_t=\langle o_{1,t},\dots,o_{n,t}\rangle$4 if its ground-truth bounding box area $o_t=\langle o_{1,t},\dots,o_{n,t}\rangle$5, normalized by frame area $o_t=\langle o_{1,t},\dots,o_{n,t}\rangle$6, exceeds a minimum scale $o_t=\langle o_{1,t},\dots,o_{n,t}\rangle$7:

$o_t=\langle o_{1,t},\dots,o_{n,t}\rangle$8

The instantaneous global coverage is

$o_t=\langle o_{1,t},\dots,o_{n,t}\rangle$9

and the episode score is the time average

$a_{i,t}$0

This same coverage quantity is used as the team reward term (Fang et al., 2022).

The per-agent reward combines a shared team component and an individual shaping component:

$a_{i,t}$1

with $a_{i,t}$2, and

$a_{i,t}$3

The individual term is

$a_{i,t}$4

with $a_{i,t}$5, $a_{i,t}$6, and $a_{i,t}$7.

The four shaping terms encode distinct control preferences. The bounding-box size reward encourages sufficiently large visible targets and is clipped to $a_{i,t}$8:

$a_{i,t}$9

The visibility-share term discourages redundant overlap across cameras by dividing credit among cameras that simultaneously observe the same target:

$a=[a_m,a_r,a_z]$0

where $a=[a_m,a_r,a_z]$1 is the number of cameras seeing target $a=[a_m,a_r,a_z]$2. The direction-centering term keeps targets near the center of the field of view:

$a=[a_m,a_r,a_z]$3

with $a=[a_m,a_r,a_z]$4. The position-dispersion term discourages camera clustering:

$a=[a_m,a_r,a_z]$5

with $a=[a_m,a_r,a_z]$6 world units.

The ablation evidence is important because it clarifies what CAMERA-Q is not. It is not simply “global coverage plus Q-learning,” and it is not simply “coordinate alignment plus joint observation.” Reported training after 200k steps shows that removing visibility share, direction, box size, or position dispersion lowers coverage, and that both decentralized training without the team reward and team-only reward without individual shaping are inferior to the full design (Fang et al., 2022). This suggests that the method depends on a specific interaction between global coordination and local shaping.

4. Value-based learning architecture

CAMERA-Q is implemented as a centralized, value-based multi-agent learner built on Double DQN (Fang et al., 2022). All agents share a single Q-network that takes the joint observation and a target camera identifier and outputs per-camera action values. Instead of producing a flat 27-way action distribution, the network factorizes the action space into three heads, one each for translation, rotation, and zoom:

• $a=[a_m,a_r,a_z]$7 for translation,
• $a=[a_m,a_r,a_z]$8 for rotation,
• $a=[a_m,a_r,a_z]$9 for zoom.

The joint action is composed by per-head argmax:

$3\times3\times3=27$0

and similarly for $3\times3\times3=27$1 and $3\times3\times3=27$2. The selected action is then $3\times3\times3=27$3. The motivation given is that factorization accelerates learning over the 27-way joint action by exploiting conditional independence across control axes.

The Double DQN target uses an online network $3\times3\times3=27$4 and a target network $3\times3\times3=27$5. For a transition $3\times3\times3=27$6, the next greedy per-head actions are selected with $3\times3\times3=27$7, and evaluation is performed with $3\times3\times3=27$8:

$3\times3\times3=27$9

The loss is the squared temporal-difference error,

$640\times480$0

and the canonical Double DQN update is summarized as

$640\times480$1

The architecture contains a per-camera encoder with two fully connected layers, concatenation with a one-hot agent code and the agent’s last action, a joint feature formed by concatenating all camera features, and optionally a GRU layer to capture temporal dependencies. The joint feature then feeds fully connected layers followed by the three action heads. Action selection is $640\times480$2-greedy per axis.

The reported hyperparameters are learning rate $640\times480$3, discount factor $640\times480$4, $640\times480$5 annealing from 1.0 to 0.1 over 50,000 steps, batch size 32 episodes, episode length 100 steps, target-network update every 100 episodes, and total training horizon 500k steps, approximately 5k episodes (Fang et al., 2022).

5. Experimental environment and empirical behavior

Training and evaluation are carried out in a virtual but credible 3D environment named “Soccer Court” (Fang et al., 2022). The ground plane is $640\times480$6 world units, with 100 units equal to 1 meter. Six cameras start evenly spaced on the boundary at height 500. Twenty-two human-like targets, organized as two teams, move on the ground plane with random waypoint walking at constant speed; a new destination is chosen when the current one is reached or after 15 seconds. Rendering is based on Unreal Engine and UnrealCV. Object masks provide ground-truth boxes for reward computation and evaluation, and occlusions occur naturally.

The environment is used to test both the full system and several ablations. A concise summary of the reported coverage results is below.

Method	Coverage
Baseline	63.0 ± 4.5%
CAMERA-Q	71.9 ± 5.8%
Ours+	72.1 ± 5.0%
Ours−	66.9 ± 5.8%

The baseline uses six fixed cameras evenly spaced on the border, all pointing toward the field center. CAMERA-Q improves coverage by 8.9% absolute over this baseline. Ours+ replaces the detector with ground-truth boxes and reaches 72.1 ± 5.0%, which indicates that detector noise is not the main bottleneck. Ours− removes coordinate alignment and uses raw boxes as input, dropping to 66.9 ± 5.8%. The reported interpretation is that inverse projection to world coordinates adds approximately 5% coverage and improves convergence (Fang et al., 2022).

The reward ablations are equally informative. After 200k training steps, the full model achieves 63.6 ± 5.4%, while removing visibility share yields 61.5 ± 5.3%, removing direction yields 55.5 ± 5.3%, removing box size yields 57.8 ± 5.4%, removing position dispersion yields 61.0 ± 6.9%, decentralized training with only individual terms yields 61.6 ± 5.5%, and team-only reward yields 55.7 ± 4.9% (Fang et al., 2022). These results support the claim that global and local shaping terms are both necessary.

6. Limitations, interpretation, and relation to adjacent work

CAMERA-Q relies on accurate intrinsics, accurate extrinsics, and the assumption that targets live on the plane $640\times480$7 (Fang et al., 2022). Errors arise with truncation, calibration drift, or off-plane targets. The reported discussion suggests that robustness could be improved with multi-point foot detection, temporal filtering, or learned homography refinement. The control space is also discretized, so translation, rotation, and zoom are adjusted in coarse steps rather than continuously. A stated limitation is that communication latency is ignored; observations and actions are synchronized at every step, and there are no bandwidth constraints in the simulator. For larger systems, the authors note that central concatenation scales linearly in the number of cameras but may require attention or graph-based encoders for dozens of cameras or targets.

A common misunderstanding is to treat CAMERA-Q as a generic multi-camera tracking framework. Its primary objective is coverage, not cross-camera identity persistence. A plausible implication is that it occupies a distinct point within a broader design space of camera-network control. “Intelligent Querying for Target Tracking in Camera Networks using Deep Q-Learning with n-Step Bootstrapping” formulates target tracking as an MDP that decides which camera to query next, with the goal of reducing the number of queried frames rather than moving or zooming cameras (Sharma et al., 2020). “Toward Global Sensing Quality Maximization: A Configuration Optimization Scheme for Camera Networks” instead optimizes camera configurations using a sensing-quality quantity derived from the number of pixels occupied by a unit-length object in the image, with reported improvements on AprilTag detection tasks (Zhang et al., 2022). CAMERA-Q differs from both by coupling active camera motion with coverage-based reward shaping and by making coordinate alignment the center of its state representation.

The method’s central insight is therefore geometric rather than merely reinforcement-learning based: local detections are first lifted into a shared world frame, and only then are collaboration and control learned. In the reported experiments, that design choice accounts for a material portion of the final gain, and it is the main reason CAMERA-Q is presented as a system for global, simultaneous optimization of multiple active cameras rather than as a collection of independently acting viewpoint controllers (Fang et al., 2022).