LingUNet: Language-Conditioned Goal Prediction

• The paper presents LingUNet, a network that integrates linguistic and visual inputs via dynamically synthesized 1×1 convolutions to predict spatial goal maps.
• LingUNet employs an encoder-decoder architecture with language-conditioned filters, decoupling high-level goal localization from low-level action planning.
• Empirical results on the LANI benchmark show that removing language conditioning drastically degrades performance, underscoring its essential role.

LingUNet is a fully differentiable, language-conditioned image generation network explicitly designed for visual goal prediction in instruction-following tasks. It computes spatial probability maps of goal locations within panoramic first-person observations, conditioned on free-form natural language instructions. LingUNet forms the central component in a model decomposition approach to 3D embodied instruction execution, isolating language-grounded goal localization from low-level action planning, with the former trained by supervised learning and the latter by reinforcement learning (Misra et al., 2018).

1. Architectural Overview

LingUNet operates in an encoder-decoder (“U-Net”) mode, fusing linguistic and visual information through a sequence of language-conditioned convolutional operations. Inputs consist of a panoramic RGB image, $\bar I$, aggregating six 60° camera views ($128 \times 768$), and a tokenized instruction $x = \langle x_1,\ldots,x_n \rangle$.

The model maps $\bar{I}$ to low-level visual feature maps via a compact convolutional neural network (CNN) incorporating explicit positional encodings. These feature maps are recursively processed through $m$ “down” convolutional layers, producing a hierarchy of spatial features $F_1, \ldots, F_m$. Linguistic context is represented by encoding $x$ with a single-layer LSTM ($d=256$), yielding a terminal state $h$ partitioned into $m$ contiguous segments ($128 \times 768$0). Each $128 \times 768$1 is projected and reshaped into a text-conditioned $128 \times 768$2 convolutional kernel $128 \times 768$3, used to modulate the corresponding visual map $128 \times 768$4 by feature-wise convolution, producing $128 \times 768$5. A mirrored sequence of $128 \times 768$6 “up” deconvolutional blocks, interleaved with skip-connections, reconstructs the spatial goal distribution.

The output, $128 \times 768$7, parameterizes a softmax over spatial locations (plus a dedicated “out-of-sight” channel) yielding the predicted goal map $128 \times 768$8.

2. Language-Conditioned Modulation and Dynamic Filtering

The central innovation in LingUNet is its explicit language-conditioning at each spatial scale by means of dynamically synthesized $128 \times 768$9 convolutional kernels. Specifically, instruction encoding proceeds as: $x = \langle x_1,\ldots,x_n \rangle$0 The vector $x = \langle x_1,\ldots,x_n \rangle$1 of dimension $x = \langle x_1,\ldots,x_n \rangle$2 is divided into $x = \langle x_1,\ldots,x_n \rangle$3 subvectors $x = \langle x_1,\ldots,x_n \rangle$4. Each $x = \langle x_1,\ldots,x_n \rangle$5 generates a kernel

$x = \langle x_1,\ldots,x_n \rangle$6

reshaped to match the shape required for per-channel $x = \langle x_1,\ldots,x_n \rangle$7 convolutions for each encoder layer $x = \langle x_1,\ldots,x_n \rangle$8. The kernel $x = \langle x_1,\ldots,x_n \rangle$9 is then convolved with the feature map $\bar{I}$0: $\bar{I}$1 This process enables the instruction to selectively control feature transformation across network depth, analogous in spirit to dynamic filter networks or FiLM, but using full text-dependent parameterization of the $\bar{I}$2 filters at every layer (Misra et al., 2018).

3. Image Generation and Heatmap Prediction

During the encoding phase, each convolutional block applies the sequence: Conv $\bar{I}$3 Leaky-ReLU $\bar{I}$4 instance normalization. The decoding (deconvolutional) blocks apply Deconv $\bar{I}$5 Leaky-ReLU $\bar{I}$6 instance norm, except at the final output. Dropout ($\bar{I}$7) is applied to $\bar{I}$8.

The final unnormalized output $\bar{I}$9 is passed (with bias $m$0) through a softmax across all spatial locations $m$1 (plus one “out-of-sight” channel) to produce the probability distribution: $m$2 Supervision is provided by a cross-entropy loss against the gold goal pixel (or “out-of-sight” flag): $m$3 where $m$4 at the ground-truth location.

4. Integration with Decomposed Instruction Execution

In downstream instruction-following tasks, LingUNet supplies a spatial goal map $m$5, supporting modularity between high-level goal inference and low-level action selection. Action generation is realized as a contextual bandit with policy gradient: at each step $m$6, the current view and predicted goal are projected to a binary mask $m$7 (flattened), concatenated with an “out-of-sight” flag $m$8, and processed by a single-layer LSTM ($m$9). Action $F_1, \ldots, F_m$0 is selected via softmax over an affine transformation of the LSTM output and a time embedding.

The reward structure comprises task-completion success ($F_1, \ldots, F_m$1), collision/incorrect stop ($F_1, \ldots, F_m$2), per-step penalty ($F_1, \ldots, F_m$3), and potential-based shaping nudging the agent toward the goal; entropy regularization ($F_1, \ldots, F_m$4) stabilizes learning. Notably, LingUNet and the control policy are trained entirely separately: goal prediction via supervised learning and action via reinforcement learning. This decomposition maintains low variance for goal localization while enabling exploration in action generation (Misra et al., 2018).

5. Concrete Hyper-parameters

LingUNet is instantiated as follows:

Component	Specification	Value
Panorama $F_1, \ldots, F_m$5	6 cameras × $F_1, \ldots, F_m$6	$F_1, \ldots, F_m$7
CNN$F_1, \ldots, F_m$8 (preprocessing)	Conv: 128 filters $F_1, \ldots, F_m$9, stride 4, pad 3; ReLU <br> Conv: 64 filters $x$0, stride 1, pad 1; ReLU	—
Positional channels ($x$1)	Appended as input	—
Encoder depth ($x$2)	Number of “down”/“up” Conv/Deconv layers	4
Each “down” Conv$x$3	32 filters, $x$4, stride 2; leaky-ReLU; instance-norm	—
Each “up” Deconv$x$5	32 filters, $x$6, stride 2; leaky-ReLU; instance-norm; dropout ($x$7 on $x$8)	—
LSTM$x$9/$d=256$0	Single-layer, hidden size	256
Embeddings $d=256$1, $d=256$2	Dimensionality	32
Final goal-map	Matches $d=256$3’s spatial dims after one up-step ($d=256$4) plus 1 out-of-sight

6. Empirical Results and Ablation Analysis

Empirical evaluation on the LANI navigation benchmark quantifies the impact of architectural choices via ablations. Performance is measured by stop-distance (SD, lower is better) and task-completion (TC, higher is better):

Method	SD	TC
Full LINGUNet (“OA”)	8.65	35.7%
w/o action-RNN	9.21	31.3%
w/o language conditioning	10.65	23.0%
joint (end-to-end) training	11.54	21.8%
oracle goals supplied	2.13	94.6%

Eliminating language-conditioned filters caused the largest degradation, confirming the necessity of text-dependent modulations. Training goal and control modules jointly (vs. separate decomposition) significantly impairs both goal accuracy and downstream behavior, indicating the effectiveness of decoupled training. Providing oracle (ground-truth) goals enables near-perfect navigation, establishing that goal prediction is the principal bottleneck in the pipeline (Misra et al., 2018).

7. Significance and Scope

LingUNet operationalizes a general approach for goal prediction in visually and linguistically grounded tasks requiring interpretable spatial inference. Its design supports direct visualization and manipulation of goal heatmaps, facilitating integration with modular planning or control architectures. The method proceeds entirely from demonstration data, without reliance on external resources. LingUNet provides a reference framework for evaluating and isolating language-to-goal mapping in embodied instruction following, as exemplified with the LANI and CHAI benchmarks (Misra et al., 2018). Its modularity, language-to-image fusion mechanism, and empirical results establish LingUNet as a canonical architecture for interpretable, instruction-conditioned visual goal prediction.