---
title: 'DARIL: Autoregressive Imitation in Surgical Planning'
url: https://www.emergentmind.com/topics/dual-task-autoregressive-imitation-learning-daril
type: topic
---

# DARIL: Autoregressive Imitation in Surgical Planning

Searching arXiv for DARIL and adjacent autoregressive imitation-learning papers.
Dual-task Autoregressive Imitation Learning (DARIL) is an imitation-learning framework introduced for surgical action planning from video, where future behavior is modeled as causal prediction of surgical action triplets from recent visual context. In the cited formulation, DARIL predicts instrument-verb-target triplets on the CholecT50 benchmark using a dual-task objective that combines current action recognition and next action prediction, augmented by next embedding prediction and surgical phase recognition [2507.05011]. The method is “autoregressive” in the sense that future action generation is treated as causal prediction with a GPT-2 decoder, and “dual-task” because the training objective jointly optimizes current and next action objectives [2507.05011]. Within the paper’s broader comparison, DARIL serves as the principal imitation-learning baseline against several reinforcement-learning alternatives and achieves the strongest reported performance on expert-annotated surgical action prediction [2507.05011].

## 1. Definition and task formulation

DARIL is defined in the source paper as **Dual-task Autoregressive Imitation Learning** for **future surgical action prediction from video** [2507.05011]. The core planning problem is to map an observed frame history,
\[
\{f_1, f_2, \dots, f_t\},
\]
to a future action sequence,
\[
\{a_{t+1}, a_{t+2}, \dots, a_{t+H}\}.
\]
Each action is an **instrument-verb-target triplet**,
\[
a_i = (I_i, V_i, T_i),
\]
drawn from predefined vocabularies, and the paper states that the dataset contains **100 distinct action classes total** [2507.05011].

A key property of the benchmark is that **each frame can contain multiple simultaneous actions**, specifically **0–3 actions per frame**, because multiple instruments may act simultaneously [2507.05011]. The paper nevertheless describes DARIL’s supervised action losses as cross-entropy losses for direct prediction of the **100 action classes** [2507.05011]. This creates an important interpretive tension in the formulation: the benchmark is effectively sparse and multi-label at the frame level, whereas the printed loss description is class-centric. The paper does not resolve this explicitly. A plausible implication is that DARIL’s practical label representation or evaluation interface contains implementation details not fully documented in the manuscript.

The paper states that DARIL uses four supervised targets: **current action recognition**, **next action prediction**, **next embedding prediction**, and **surgical phase recognition** [2507.05011]. The “dual-task” designation refers specifically to the joint optimization of
\[
p(a_t \mid f_{t-w+1:t})
\]
and
\[
p(a_{t+1} \mid f_{t-w+1:t}),
\]
while the embedding and phase terms function as auxiliary supervision [2507.05011].

Autoregression is motivated by the temporal dependence of surgical activity. The paper explicitly gives the next-step conditional
\[
p(a_{t+1}|f_{t-w+1:t}) = \text{GPT-2}(\text{FrameEmb}(f_{t-w+1:t}))
\]
with context window \(w=20\) [2507.05011]. The manuscript does not print a full multi-step factorization for planning horizons \(H>1\). This suggests that the intended autoregressive mechanism is causal rollout through the GPT-2 decoder, but the exact conditioning structure for later predicted steps is not fully specified.

## 2. Model architecture and representational structure

DARIL processes **1024-dimensional Swin transformer features** through a two-part temporal architecture: a **BiLSTM encoder for temporal current action recognition** and a **GPT-2 decoder for causal future action generation** [2507.05011]. The input context window is explicitly fixed at **\(w=20\)** [2507.05011].

The BiLSTM branch is described as responsible for current recognition, while the GPT-2 branch is used for future prediction [2507.05011]. The paper further states that DARIL has **separate prediction heads for the combined <instrument, verb, and target> class and surgical phase components** [2507.05011]. In the reported formulation, the triplet is predicted as a **joint 100-class output**, rather than by composing separate instrument, verb, and target predictions [2507.05011].

At training time, the model consumes the recent frame context \(f_{t-w+1:t}\), represented by the 1024-dimensional Swin features, and produces four outputs: current action class \(a_t\), next action class \(a_{t+1}\), next embedding \(\hat e_{t+1}\), and phase prediction [2507.05011]. At inference time, the same recent context is used for current recognition and future planning at horizons including **1s, 2s, 3s, 5s, 10s, 20s** [2507.05011].

Several architectural details remain unspecified in the source. The paper does not provide the exact Swin variant, whether the backbone is frozen or fine-tuned, the BiLSTM hidden size, the GPT-2 depth or attention configuration, the decoding rule for long-horizon planning, or the precise rollout mechanics [2507.05011]. This matters because DARIL is presented as an autoregressive model, but the manuscript only fully prints the one-step next-action conditional. A plausible implication is that the model’s “autoregressive” designation is architecturally justified by causal decoding, while operational reproducibility of long-horizon rollout depends on omitted engineering choices.

## 3. Learning objective and supervision signals

The paper gives a combined objective with four components:
\[
\mathcal{L} = \mathcal{L}_{\text{current} + \mathcal{L}_{\text{next} + \mathcal{L}_{\text{embed} + \mathcal{L}_{\text{phase}
\]
in a malformed printed form, which is described as likely intending the sum of the four losses [2507.05011]. The exact weighting coefficients are not reported.

The current action recognition loss is printed as
\[
\mathcal{L}_{\text{current} = -\sum_{t} \log p(a_t|f_{t-w+1:t}),
\]
and the next action loss as
\[
\mathcal{L}_{\text{next} = -\sum_{t} \log p(a_{t+1}|f_{t-w+1:t}),
\]
with the paper stating that both are **cross-entropy losses for direct prediction of the 100 action classes** [2507.05011]. The embedding regression term is
\[
\mathcal{L}_{\text{embed} = \sum_{t} ||e_{t+1} - \hat{e}_{t+1}||^2,
\]
and \(\mathcal{L}_{\text{phase}}\) is identified as a phase recognition loss, though no explicit formula is given [2507.05011].

This design gives DARIL a multi-objective structure beyond its named dual-task core. The current and next triplet losses train present-state recognition and immediate future anticipation, while the embedding and phase losses plausibly regularize temporal representation learning. That interpretation is consistent with the paper’s description, though the manuscript does not provide a decomposition of their relative contributions.

The paper also does not explicitly state whether teacher forcing is used in the GPT-2 branch, whether multi-step rollout is trained directly, whether scheduled sampling is used, or whether predicted actions are recursively fed back during training [2507.05011]. Since long-horizon planning is evaluated, these omissions are significant. A plausible implication is that DARIL’s reported planning behavior may rely on implementation choices not recoverable from the paper alone.

## 4. Dataset, benchmark protocol, and evaluation

DARIL is evaluated on **CholecT50**, described as **50 laparoscopic cholecystectomy videos** with frame-level annotations and **100 distinct triplet classes** [2507.05011]. The paper uses a standard split protocol with test videos
\[
[2,6,14,23,25,50,51,66,79,111],
\]
training on the remaining **40 videos** [2507.05011]. The reported frame counts are **78,968 frames** for training and **21,895 frames** for test, sampled at **1 FPS** [2507.05011].

The context window is **20 frames**, so each example uses the recent 20-step visual history represented as Swin features [2507.05011]. Because the data are sampled at 1 FPS, a plausible implication is that the context corresponds to approximately 20 seconds of history, and that a 10-second planning horizon corresponds to 10 future frames. The paper strongly implies this mapping, although it does not state it explicitly.

Evaluation centers on **mAP** over action triplets and their components, using benchmark IVT metrics [2507.05011]. The paper reports both **current** recognition and future planning performance at multiple horizons. It also reports component-wise scores for Instrument (I), Verb (V), Target (T), Instrument-Verb (IV), Instrument-Target (IT), and full Instrument-Verb-Target (IVT) [2507.05011].

The most relevant protocol distinctions are summarized below.

| Element | Reported value |
|---|---|
| Dataset | CholecT50 |
| Training videos | 40 |
| Test videos | \([2,6,14,23,25,50,51,66,79,111]\) |
| Sampling rate | 1 FPS |
| Input features | 1024-dimensional Swin transformer features |
| Context window | \(w=20\) |
| Planning horizons | 1s, 2s, 3s, 5s, 10s, 20s |
| Main metric | IVT mAP |

The paper describes planning curves with **95% confidence intervals** but does not print the interval values [2507.05011]. It also does not formalize the mAP equation, deferring instead to standard IVT benchmark conventions.

## 5. Empirical performance and comparison with reinforcement learning

DARIL is the strongest method reported in the paper’s main IL-versus-RL comparison [2507.05011]. Its headline results are **34.6%** current action triplet recognition mAP and **33.6%** next frame prediction mAP, with planning degrading smoothly to **29.2%** at 10-second horizons [2507.05011]. The paper also reports **31.2%** at 5 seconds [2507.05011].

The primary comparison table reports the following IVT mAP values:

| Method | Current | 1s | 5s | 10s |
|---|---:|---:|---:|---:|
| DARIL (Ours) | 34.6 | 33.6 | 31.2 | 29.2 |
| DARIL + IRL | 33.1 | 32.1 | 29.6 | 28.1 |
| DARIL + Direct Video RL | 33.2 | 22.6 | 19.3 | 15.9 |
| Latent World Model + RL | 33.1 | 14.0 | 9.1 | 3.1 |

The paper highlights DARIL’s **smooth planning degradation** and states that the decline from 1s to 10s is a **13.1% relative decrease** [2507.05011]. By contrast, the world-model RL baseline drops to **3.1%** at 10s, and direct video RL reaches **15.9%** [2507.05011].

Component-wise, DARIL achieves **91.4** current and **88.2** next for Instrument, **69.4** and **68.1** for Verb, **52.7** and **52.5** for Target, **42.9** and **38.8** for IV, **43.5** and **43.6** for IT, and **34.6** and **33.6** for IVT [2507.05011]. This pattern indicates that instrument recognition is substantially easier than full triplet prediction, while the target dimension is weaker and more variable.

The paper’s conceptual conclusion is not that reinforcement learning is universally inferior, but that **distribution matching on expert-annotated test sets systematically favors IL over potentially valid RL policies that differ from training demonstrations** [2507.05011]. In the paper’s interpretation, DARIL is well aligned with both the supervision source and the evaluation metric: teleoperated surgery provides expert demonstrations, and the benchmark rewards similarity to expert annotations [2507.05011]. This challenges stronger assumptions about RL superiority in sequential decision making within this particular benchmark regime.

## 6. Position within autoregressive imitation learning and reported limitations

DARIL sits within a broader line of work that treats autoregressive generation as a sequential imitation problem rather than only a one-step supervised problem. “Autoregressive Knowledge Distillation through Imitation Learning” frames autoregressive generation as finite-horizon imitation learning, where prefixes are states and next tokens are actions, and emphasizes training on student-generated prefixes to reduce exposure bias [2009.07253]. “SequenceMatch: Imitation Learning for Autoregressive Sequence Modelling with Backtracking” similarly recasts autoregressive sequence modeling as occupancy-measure matching over trajectories and introduces a corrective backspace action to mitigate compounding error [2306.05426]. These works are not DARIL papers, but they clarify the technical lineage of DARIL’s autoregressive imitation viewpoint.

By contrast, “Imitation Learning from Observations: An Autoregressive Mixture of Experts Approach” studies state-only imitation with latent controls and a switching autoregressive policy, but it is explicitly characterized as adjacent to DARIL rather than a direct predecessor, because it does not use a dual-task architecture and instead relies on known dynamics inversion and a structured latent-mode model [2411.08232]. This comparison is useful because it distinguishes DARIL’s video-based dual-task predictive setting from other forms of autoregressive imitation learning.

The DARIL paper itself identifies several limitations. First, planning performance declines with horizon, which is consistent with autoregressive compounding error [2507.05011]. Second, as an imitation learner, DARIL cannot exceed demonstrator behavior and is fundamentally tied to the expert data distribution [2507.05011]. Third, the method is evaluated only on **CholecT50**, so the evidence base is single-dataset [2507.05011]. Fourth, the paper explicitly notes possible overfitting risk due to the limited number of videos [2507.05011]. Fifth, the manuscript does not clearly explain how multiple simultaneous actions per frame are represented in a model described through cross-entropy over **100 action classes** [2507.05011].

Reproducibility is also constrained by missing implementation details. The paper specifies the dataset split, sampling rate, feature dimension, context window, temporal modules, and the four loss components, but does not report optimizer, learning rate, batch size, number of epochs, loss weights, exact GPT-2 configuration, exact BiLSTM configuration, rollout decoding method, or multi-label handling [2507.05011]. This suggests that DARIL is conceptually well defined but not fully reproducible from the paper alone.

For DARIL-oriented research more broadly, the clearest practical lesson is that performance in sequential prediction can depend as much on alignment between expert demonstrations and evaluation protocol as on the nominal distinction between imitation learning and reinforcement learning. The paper’s empirical result is therefore both methodological and epistemic: on expert-annotated surgical planning benchmarks, a dual-task autoregressive imitation model can outperform RL variants not necessarily because RL is categorically weaker, but because the benchmark operationalizes success as expert-consistent trajectory prediction [2507.05011].

Source: https://www.emergentmind.com/topics/dual-task-autoregressive-imitation-learning-daril