DARIL: Autoregressive Imitation in Surgical Planning

• The paper demonstrates that DARIL leverages dual-task objectives to jointly optimize current action recognition and next action prediction using a GPT-2 decoder for improved surgical planning.
• DARIL combines a BiLSTM encoder with a GPT-2 decoder to process 1024-dimensional Swin features from a 20-frame context, enabling effective prediction of instrument-verb-target triplets.
• Empirical evaluations on the CholecT50 benchmark show robust performance with a current action mAP of 34.6% and smooth degradation in planning over extended time horizons.

Searching arXiv for DARIL and adjacent autoregressive imitation-learning papers. Dual-task Autoregressive Imitation Learning (DARIL) is an imitation-learning framework introduced for surgical action planning from video, where future behavior is modeled as causal prediction of surgical action triplets from recent visual context. In the cited formulation, DARIL predicts instrument-verb-target triplets on the CholecT50 benchmark using a dual-task objective that combines current action recognition and next action prediction, augmented by next embedding prediction and surgical phase recognition (Boels et al., 7 Jul 2025). The method is “autoregressive” in the sense that future action generation is treated as causal prediction with a GPT-2 decoder, and “dual-task” because the training objective jointly optimizes current and next action objectives (Boels et al., 7 Jul 2025). Within the paper’s broader comparison, DARIL serves as the principal imitation-learning baseline against several reinforcement-learning alternatives and achieves the strongest reported performance on expert-annotated surgical action prediction (Boels et al., 7 Jul 2025).

1. Definition and task formulation

DARIL is defined in the source paper as Dual-task Autoregressive Imitation Learning for future surgical action prediction from video (Boels et al., 7 Jul 2025). The core planning problem is to map an observed frame history,

$\{f_1, f_2, \dots, f_t\},$

to a future action sequence,

$\{a_{t+1}, a_{t+2}, \dots, a_{t+H}\}.$

Each action is an instrument-verb-target triplet,

$a_i = (I_i, V_i, T_i),$

drawn from predefined vocabularies, and the paper states that the dataset contains 100 distinct action classes total (Boels et al., 7 Jul 2025).

A key property of the benchmark is that each frame can contain multiple simultaneous actions, specifically 0–3 actions per frame, because multiple instruments may act simultaneously (Boels et al., 7 Jul 2025). The paper nevertheless describes DARIL’s supervised action losses as cross-entropy losses for direct prediction of the 100 action classes (Boels et al., 7 Jul 2025). This creates an important interpretive tension in the formulation: the benchmark is effectively sparse and multi-label at the frame level, whereas the printed loss description is class-centric. The paper does not resolve this explicitly. A plausible implication is that DARIL’s practical label representation or evaluation interface contains implementation details not fully documented in the manuscript.

The paper states that DARIL uses four supervised targets: current action recognition, next action prediction, next embedding prediction, and surgical phase recognition (Boels et al., 7 Jul 2025). The “dual-task” designation refers specifically to the joint optimization of

$p(a_t \mid f_{t-w+1:t})$

and

$p(a_{t+1} \mid f_{t-w+1:t}),$

while the embedding and phase terms function as auxiliary supervision (Boels et al., 7 Jul 2025).

Autoregression is motivated by the temporal dependence of surgical activity. The paper explicitly gives the next-step conditional

$$p(a_{t+1}|f_{t-w+1:t}) = \text{GPT-2}(\text{FrameEmb}(f_{t-w+1:t}))$$

with context window $w=20$ (Boels et al., 7 Jul 2025). The manuscript does not print a full multi-step factorization for planning horizons $H>1$. This suggests that the intended autoregressive mechanism is causal rollout through the GPT-2 decoder, but the exact conditioning structure for later predicted steps is not fully specified.

2. Model architecture and representational structure

DARIL processes 1024-dimensional Swin transformer features through a two-part temporal architecture: a BiLSTM encoder for temporal current action recognition and a GPT-2 decoder for causal future action generation (Boels et al., 7 Jul 2025). The input context window is explicitly fixed at $w=20$ (Boels et al., 7 Jul 2025).

The BiLSTM branch is described as responsible for current recognition, while the GPT-2 branch is used for future prediction (Boels et al., 7 Jul 2025). The paper further states that DARIL has separate prediction heads for the combined <instrument, verb, and target> class and surgical phase components (Boels et al., 7 Jul 2025). In the reported formulation, the triplet is predicted as a joint 100-class output, rather than by composing separate instrument, verb, and target predictions (Boels et al., 7 Jul 2025).

At training time, the model consumes the recent frame context $f_{t-w+1:t}$, represented by the 1024-dimensional Swin features, and produces four outputs: current action class $\{a_{t+1}, a_{t+2}, \dots, a_{t+H}\}.$0, next action class $\{a_{t+1}, a_{t+2}, \dots, a_{t+H}\}.$1, next embedding $\{a_{t+1}, a_{t+2}, \dots, a_{t+H}\}.$2, and phase prediction (Boels et al., 7 Jul 2025). At inference time, the same recent context is used for current recognition and future planning at horizons including 1s, 2s, 3s, 5s, 10s, 20s (Boels et al., 7 Jul 2025).

Several architectural details remain unspecified in the source. The paper does not provide the exact Swin variant, whether the backbone is frozen or fine-tuned, the BiLSTM hidden size, the GPT-2 depth or attention configuration, the decoding rule for long-horizon planning, or the precise rollout mechanics (Boels et al., 7 Jul 2025). This matters because DARIL is presented as an autoregressive model, but the manuscript only fully prints the one-step next-action conditional. A plausible implication is that the model’s “autoregressive” designation is architecturally justified by causal decoding, while operational reproducibility of long-horizon rollout depends on omitted engineering choices.

3. Learning objective and supervision signals

The paper gives a combined objective with four components: $\{a_{t+1}, a_{t+2}, \dots, a_{t+H}\}.$3 in a malformed printed form, which is described as likely intending the sum of the four losses (Boels et al., 7 Jul 2025). The exact weighting coefficients are not reported.

The current action recognition loss is printed as

$\{a_{t+1}, a_{t+2}, \dots, a_{t+H}\}.$4

and the next action loss as

$\{a_{t+1}, a_{t+2}, \dots, a_{t+H}\}.$5

with the paper stating that both are cross-entropy losses for direct prediction of the 100 action classes (Boels et al., 7 Jul 2025). The embedding regression term is

$\{a_{t+1}, a_{t+2}, \dots, a_{t+H}\}.$6

and $\{a_{t+1}, a_{t+2}, \dots, a_{t+H}\}.$7 is identified as a phase recognition loss, though no explicit formula is given (Boels et al., 7 Jul 2025).

This design gives DARIL a multi-objective structure beyond its named dual-task core. The current and next triplet losses train present-state recognition and immediate future anticipation, while the embedding and phase losses plausibly regularize temporal representation learning. That interpretation is consistent with the paper’s description, though the manuscript does not provide a decomposition of their relative contributions.

The paper also does not explicitly state whether teacher forcing is used in the GPT-2 branch, whether multi-step rollout is trained directly, whether scheduled sampling is used, or whether predicted actions are recursively fed back during training (Boels et al., 7 Jul 2025). Since long-horizon planning is evaluated, these omissions are significant. A plausible implication is that DARIL’s reported planning behavior may rely on implementation choices not recoverable from the paper alone.

4. Dataset, benchmark protocol, and evaluation

DARIL is evaluated on CholecT50, described as 50 laparoscopic cholecystectomy videos with frame-level annotations and 100 distinct triplet classes (Boels et al., 7 Jul 2025). The paper uses a standard split protocol with test videos

$\{a_{t+1}, a_{t+2}, \dots, a_{t+H}\}.$8

training on the remaining 40 videos (Boels et al., 7 Jul 2025). The reported frame counts are 78,968 frames for training and 21,895 frames for test, sampled at 1 FPS (Boels et al., 7 Jul 2025).

The context window is 20 frames, so each example uses the recent 20-step visual history represented as Swin features (Boels et al., 7 Jul 2025). Because the data are sampled at 1 FPS, a plausible implication is that the context corresponds to approximately 20 seconds of history, and that a 10-second planning horizon corresponds to 10 future frames. The paper strongly implies this mapping, although it does not state it explicitly.

Evaluation centers on mAP over action triplets and their components, using benchmark IVT metrics (Boels et al., 7 Jul 2025). The paper reports both current recognition and future planning performance at multiple horizons. It also reports component-wise scores for Instrument (I), Verb (V), Target (T), Instrument-Verb (IV), Instrument-Target (IT), and full Instrument-Verb-Target (IVT) (Boels et al., 7 Jul 2025).

The most relevant protocol distinctions are summarized below.

Element	Reported value
Dataset	CholecT50
Training videos	40
Test videos	$\{a_{t+1}, a_{t+2}, \dots, a_{t+H}\}.$9
Sampling rate	1 FPS
Input features	1024-dimensional Swin transformer features
Context window	$a_i = (I_i, V_i, T_i),$0
Planning horizons	1s, 2s, 3s, 5s, 10s, 20s
Main metric	IVT mAP

The paper describes planning curves with 95% confidence intervals but does not print the interval values (Boels et al., 7 Jul 2025). It also does not formalize the mAP equation, deferring instead to standard IVT benchmark conventions.

5. Empirical performance and comparison with reinforcement learning

DARIL is the strongest method reported in the paper’s main IL-versus-RL comparison (Boels et al., 7 Jul 2025). Its headline results are 34.6% current action triplet recognition mAP and 33.6% next frame prediction mAP, with planning degrading smoothly to 29.2% at 10-second horizons (Boels et al., 7 Jul 2025). The paper also reports 31.2% at 5 seconds (Boels et al., 7 Jul 2025).

The primary comparison table reports the following IVT mAP values:

Method	Current	1s	5s	10s
DARIL (Ours)	34.6	33.6	31.2	29.2
DARIL + IRL	33.1	32.1	29.6	28.1
DARIL + Direct Video RL	33.2	22.6	19.3	15.9
Latent World Model + RL	33.1	14.0	9.1	3.1

The paper highlights DARIL’s smooth planning degradation and states that the decline from 1s to 10s is a 13.1% relative decrease (Boels et al., 7 Jul 2025). By contrast, the world-model RL baseline drops to 3.1% at 10s, and direct video RL reaches 15.9% (Boels et al., 7 Jul 2025).

Component-wise, DARIL achieves 91.4 current and 88.2 next for Instrument, 69.4 and 68.1 for Verb, 52.7 and 52.5 for Target, 42.9 and 38.8 for IV, 43.5 and 43.6 for IT, and 34.6 and 33.6 for IVT (Boels et al., 7 Jul 2025). This pattern indicates that instrument recognition is substantially easier than full triplet prediction, while the target dimension is weaker and more variable.

The paper’s conceptual conclusion is not that reinforcement learning is universally inferior, but that distribution matching on expert-annotated test sets systematically favors IL over potentially valid RL policies that differ from training demonstrations (Boels et al., 7 Jul 2025). In the paper’s interpretation, DARIL is well aligned with both the supervision source and the evaluation metric: teleoperated surgery provides expert demonstrations, and the benchmark rewards similarity to expert annotations (Boels et al., 7 Jul 2025). This challenges stronger assumptions about RL superiority in sequential decision making within this particular benchmark regime.

6. Position within autoregressive imitation learning and reported limitations

DARIL sits within a broader line of work that treats autoregressive generation as a sequential imitation problem rather than only a one-step supervised problem. “Autoregressive Knowledge Distillation through Imitation Learning” frames autoregressive generation as finite-horizon imitation learning, where prefixes are states and next tokens are actions, and emphasizes training on student-generated prefixes to reduce exposure bias (Lin et al., 2020). “SequenceMatch: Imitation Learning for Autoregressive Sequence Modelling with Backtracking” similarly recasts autoregressive sequence modeling as occupancy-measure matching over trajectories and introduces a corrective backspace action to mitigate compounding error (Cundy et al., 2023). These works are not DARIL papers, but they clarify the technical lineage of DARIL’s autoregressive imitation viewpoint.

By contrast, “Imitation Learning from Observations: An Autoregressive Mixture of Experts Approach” studies state-only imitation with latent controls and a switching autoregressive policy, but it is explicitly characterized as adjacent to DARIL rather than a direct predecessor, because it does not use a dual-task architecture and instead relies on known dynamics inversion and a structured latent-mode model (Wang et al., 2024). This comparison is useful because it distinguishes DARIL’s video-based dual-task predictive setting from other forms of autoregressive imitation learning.

The DARIL paper itself identifies several limitations. First, planning performance declines with horizon, which is consistent with autoregressive compounding error (Boels et al., 7 Jul 2025). Second, as an imitation learner, DARIL cannot exceed demonstrator behavior and is fundamentally tied to the expert data distribution (Boels et al., 7 Jul 2025). Third, the method is evaluated only on CholecT50, so the evidence base is single-dataset (Boels et al., 7 Jul 2025). Fourth, the paper explicitly notes possible overfitting risk due to the limited number of videos (Boels et al., 7 Jul 2025). Fifth, the manuscript does not clearly explain how multiple simultaneous actions per frame are represented in a model described through cross-entropy over 100 action classes (Boels et al., 7 Jul 2025).

Reproducibility is also constrained by missing implementation details. The paper specifies the dataset split, sampling rate, feature dimension, context window, temporal modules, and the four loss components, but does not report optimizer, learning rate, batch size, number of epochs, loss weights, exact GPT-2 configuration, exact BiLSTM configuration, rollout decoding method, or multi-label handling (Boels et al., 7 Jul 2025). This suggests that DARIL is conceptually well defined but not fully reproducible from the paper alone.

For DARIL-oriented research more broadly, the clearest practical lesson is that performance in sequential prediction can depend as much on alignment between expert demonstrations and evaluation protocol as on the nominal distinction between imitation learning and reinforcement learning. The paper’s empirical result is therefore both methodological and epistemic: on expert-annotated surgical planning benchmarks, a dual-task autoregressive imitation model can outperform RL variants not necessarily because RL is categorically weaker, but because the benchmark operationalizes success as expert-consistent trajectory prediction (Boels et al., 7 Jul 2025).