---
title: System-Level Instructions in NLP
url: https://www.emergentmind.com/topics/system-level-instructions
type: topic
---

# System-Level Instructions in NLP

to=arxiv_search ,最新高清无码专区  天天中彩票双色球ીjson
{"query":"2203.08512 ConTinTin Continual Learning from Task Instructions", "max_results": 5}
to=arxiv_search 开号网址json
{"query":"2212.03813 Robustness of Learning from Task Instructions", "max_results": 5}
to=arxiv_search  彩神争霸能json
{"query":"continual learning from task instructions NLP arXiv", "max_results": 10}
ConTinTin is a continual-learning paradigm for natural language processing in which a model acquires new tasks from **textual instructions** rather than from large task-specific labeled datasets alone. It is defined around a sequential stream of previously unseen tasks, each described by an instruction and accompanied by a small number of exemplars embedded in that instruction. The learner is required to infer a new task’s output behavior from its instruction, transfer knowledge from earlier tasks to later ones, and retain or even improve performance on earlier tasks while keeping model capacity fixed. The formulation was introduced as the first explicit study of continual learning from task instructions in NLP, together with an instruction-centric method, **InstructionSpeak**, and an evaluation protocol based on transfer distance [2203.08512].

## 1. Problem formulation

ConTinTin begins from the observation that mainstream supervised NLP usually assumes a predefined static task and supervision primarily in the form of labeled examples. In contrast, the paradigm assumes that a learner can exploit two richer supervisory sources: prior experience across tasks and natural-language instructions. The model is first initialized on a small ordered set of training tasks,
\[
S = [s_1, s_2, \cdots, s_k],
\]
where each \(s_i\) has an instruction \(d_{s_i}\) and labeled examples
\[
\{(x_{s_i}^j, y_{s_i}^j)\}_{j=1}^n.
\]
The paper explicitly argues that \(k\) should stay small, because otherwise the objective of reducing annotation burden is undermined [2203.08512].

After initialization, the learner enters an evolution stage over an unseen task stream
\[
U = [u_1, u_2, \cdots, u_i, \cdots].
\]
Each unseen task \(u_i\) has a textual description \(d_{u_i}\) and an evaluation set
\[
\{(x_{u_i}^j, y_{u_i}^j)\}_{j=1}^n.
\]
If the learner after processing tasks \([u_1,\dots,u_i]\) is denoted \(M_i\), then for each new task \(u_i\), \(M_i\) must generate outputs for inputs \(x_{u_i}^j\) based on the instruction \(d_{u_i}\). The idealized target is zero-example learning from instruction, although in practice the instructions contain a small number of embedded examples and some tasks also include negative examples [2203.08512].

The paradigm is defined by three goals. First, the learner must generate the expected outputs of a new task by learning from its instruction. Second, it should exhibit **forward transfer**, meaning that knowledge acquired from upstream tasks helps downstream task acquisition. Third, it should exhibit **backward transfer**, meaning that performance on earlier tasks is retained or even improved after learning later tasks. Fixed model capacity is part of the setting, so these gains are not supposed to come from adding memory modules or increasing parameter count [2203.08512].

## 2. Relation to neighboring paradigms

ConTinTin is intentionally positioned against several adjacent learning settings. It is not standard multitask learning, because tasks are not jointly trained in batch; they arrive sequentially and must be incorporated online. It is not ordinary instruction tuning, because instruction following is evaluated under a continual stream with explicit transfer and retention requirements. It is not meta-learning in the usual episodic sense, because the objective is not rapid adaptation from support sets under a stationary task distribution, but continual evolution under fixed capacity. It also differs from classical task-continual learning in NLP because supervision for new tasks comes mainly from **instruction text** rather than explicit labeled training sets [2203.08512].

This distinction matters methodologically. In ConTinTin, instructions are not merely prompts appended to an otherwise conventional training regime. They are the central supervision objects through which new tasks are specified, transferred, and later revisited. A plausible implication is that the quality, specificity, and variability of instructions become central determinants of performance. Later work on robustness of learning from task instructions studied manipulated, paraphrased, and concise instructions, and reported that models are often robust to many perturbations when demonstrations are included, whereas robustness is much weaker without demonstrations [2212.03813].

The formulation therefore shifts the emphasis of continual NLP from example replay alone to instruction-conditioned task acquisition. Earlier-task competence is not only something to be protected against forgetting; it is something that can, in principle, be refreshed from the earlier task descriptions themselves. This suggests a different view of task memory: instructions act simultaneously as task specifications and as compressed task memories [2203.08512].

## 3. Evaluation protocol and transfer metrics

A major contribution of ConTinTin is its evaluation protocol. Rather than reporting only a final average score, it defines transfer metrics as functions of **transfer distance** \(i\), the number of tasks between two model states. The underlying task metric in the experiments is ROUGE-L, so all transfer scores are expressed in ROUGE-L units [2203.08512].

For forward transfer, the paper defines \(\overrightarrow{g}_i\) as the expected gain on a task when that task is learned after \(i\) additional upstream tasks. For a target task \(t\), one samples a random position \(k\), constructs one chain where \(t\) is learned after \(k-1\) tasks and another where it is learned after \(k+i-1\) tasks, and computes
\[
\overrightarrow{g}_{i,t}^j = M_{k+i}(t) - M_k(t).
\]
Averaging over \(m\) random samplings gives
\[
\overrightarrow{g}_{i,t}=\frac{1}{m}\sum_{j=1}^m{\overrightarrow{g}_{i,t}^j},
\]
and averaging over all unseen tasks gives
\[
\overrightarrow{g}_i=\frac{1}{|U|}\sum_{t\in U}{\overrightarrow{g}_{i,t}}.
\]
Positive and increasing \(\overrightarrow{g}_i\) indicates that learning more upstream tasks helps future task acquisition [2203.08512].

Backward transfer is defined symmetrically. For a target task \(t\), the model first learns \(t\) at position \(k\), then learns \(i\) subsequent tasks, and the gain from later downstream experience is
\[
\overleftarrow{g}_{i,t}^j = M_{k+i}(t) - M_k(t).
\]
Averaging gives
\[
\overleftarrow{g}_{i,t}=\frac{1}{m}\sum_{j=1}^m{\overleftarrow{g}_{i,t}^j}, \qquad
\overleftarrow{g}_i=\frac{1}{|U|}\sum_{t\in U}{\overleftarrow{g}_{i,t}}.
\]
If \(\overleftarrow{g}_i > 0\), later tasks improve earlier ones; if it is negative, forgetting dominates. This metric design makes retention and improvement observable across different temporal separations rather than only at the end of training [2203.08512].

## 4. InstructionSpeak

InstructionSpeak is the method proposed for the ConTinTin setting. It is built on **BART-base** and treats every task as text-to-text generation. Its central claim is that task instructions should not merely be prepended as prompts; they should be exploited more deeply to improve both forward and backward transfer [2203.08512].

The encoder input concatenates the input instance with instruction metadata and positive examples from the instruction:
\[
\textcolor{purple}{[Input] input string} \ 
\textcolor{orange}{[Title] title string [Prompt] prompt string [Definition] definition string [Avoid] things to avoid string [Caution] caution string} \ 
\textcolor{blue}{[POS1] [Input] input string [Output] output string [Explanation] explanation string \cdots [POSn] [Input] input string [Output] output string [Explanation] explanation string}.
\]
The input instance is placed first to avoid truncation in long encoder sequences. During pretraining on \(S\), the full template is used; during continual learning on \(U\), the positive-example block is dropped because tuning examples are taken directly from the task instruction [2203.08512].

InstructionSpeak contains two strategies.

The first is **Negative Training**. The paper argues that “unfavorable clues” in instructions, such as “things to avoid” and explicit negative examples, can still be useful supervision. A direct objective that maximizes the probability of gold outputs while minimizing the probability of negative outputs was found harmful, because negative outputs often share many tokens with correct outputs and direct suppression can interfere with decoding. The adopted strategy is therefore to pretrain on negative outputs as if they were positive targets, then finetune on the true positive outputs. The intended effect is primarily on forward transfer: negative outputs can still teach rough output manifolds, formatting constraints, and semantic proximity to desired answers [2203.08512].

The second is **History Training**. Instead of replaying stored labeled examples, the method revisits prior **instructions**. When learning the \(i\)-th unseen task \(u_i\), the model first batches the instructions of earlier tasks \([u_1,\dots,u_{i-2}]\) and tunes on them with a smaller learning rate. Because instructions are short but supervision-rich—containing title, definition, prompt, caution, avoidance rules, and examples—this acts as a compact rehearsal mechanism. The intended effect is mainly on backward transfer by refreshing the mapping from old instructions to old task behavior [2203.08512].

The full training pipeline is specified as six steps: pretrain on \(S\) to get \(M^*\); use \(M^*\) to predict on \(S\) and collect the negative-example set \(S^{-}\); pretrain on \(S^{-}\) and finetune on \(S\) to get the starting model \(M\); for each unseen task \(u_i\), tune on instructions of earlier tasks \([u_1,\cdots,u_{i-2}]\); tune on negative examples of \(u_i\) if available; and then tune on positive examples of \(u_i\). There is no separate replay buffer of raw examples, no auxiliary contrastive objective, and no explicit ranking loss [2203.08512].

## 5. Benchmark and empirical results

The benchmark is derived from **Natural-Instructions**, reorganized for continual learning. The source resource contains **61 tasks** across six categories: 13 question generation, 16 answer generation, 12 classification, 8 incorrect answer generation, 10 minimal modification, and 2 verification. For ConTinTin, the main experiments sample \(k=5\) tasks as the initialization set \(S\) and use the remaining tasks as the unseen stream \(U\). Task order is not fixed; multiple random splits and orders are generated and results are averaged, reflecting the intrinsic order sensitivity of continual learning [2203.08512].

The baselines are **Seq-finetune**, **LAMOL**, and **Multi-task**. Seq-finetune pretrains BART on \(S\) and then sequentially finetunes on \(U\) without anti-forgetting mechanisms. LAMOL is adapted from prior task-continual learning by generating pseudo-examples for previous tasks before learning a new one. Multi-task jointly trains on all tasks in \(U\) after pretraining on \(S\) and serves as an upper bound that removes the sequential constraint. The paper deliberately excludes methods that add memory modules or adapters because ConTinTin values fixed capacity [2203.08512].

Optimization uses pretrained BART-base from HuggingFace with \(m=10\) Monte Carlo samplings for transfer metrics, \(k=5\), max input length 1024, learning rate \(5\times10^{-5}\), and 3 epochs for most phases. History Training uses learning rate \(5\times10^{-6}\) and one epoch. Batch size is 5 on \(S\) and 2 during continual learning on \(U\). For evaluation on unseen tasks, 1k labeled examples per task are randomly selected [2203.08512].

On the random-stream ConTinTin benchmark, the forward-transfer scores are as follows.

| Method | \(\overrightarrow{g}_1\) | \(\overrightarrow{g}_{10}\) | \(\overrightarrow{g}_{20}\) | \(\overrightarrow{g}_{30}\) | \(\overrightarrow{g}_{40}\) |
|---|---:|---:|---:|---:|---:|
| Seq-finetune | 1.44 | 3.28 | -3.74 | 2.9 | -0.36 |
| LAMOL | -1.34 | 1.41 | 3.31 | -5.40 | -0.03 |
| InstructionSpeak | 2.16 | 5.06 | 2.29 | 4.07 | 4.39 |

The backward-transfer scores are:

| Method | \(\overleftarrow{g}_1\) | \(\overleftarrow{g}_{10}\) | \(\overleftarrow{g}_{20}\) | \(\overleftarrow{g}_{30}\) | \(\overleftarrow{g}_{40}\) |
|---|---:|---:|---:|---:|---:|
| Seq-finetune | 1.57 | 0.04 | -0.19 | -6.48 | -9.46 |
| LAMOL | 2.67 | 2.21 | 9.42 | 6.33 | 7.21 |
| InstructionSpeak | 1.44 | 5.21 | 7.33 | 14.99 | 12.31 |

InstructionSpeak is therefore the only method that remains consistently positive or near-positive across long forward distances while also being strongly positive across long backward distances. The paper emphasizes the broader trend that both transfer measures generally grow with transfer distance, suggesting accumulation of useful knowledge rather than pure interference [2203.08512].

The ablations support the division of labor between the two strategies. Without Negative Training, forward transfer drops sharply: \(\overrightarrow{g}_1\) becomes \(-2.89\), and all forward-transfer values weaken, while backward transfer remains moderately positive. Without History Training, forward transfer remains relatively reasonable, but backward transfer degrades substantially, with \(\overleftarrow{g}_{10}=-2.78\), \(\overleftarrow{g}_{20}=-0.83\), \(\overleftarrow{g}_{30}=1.35\), and \(\overleftarrow{g}_{40}=3.49\). This matches the intended interpretation that negative-output learning mainly helps new-task understanding, while instruction rehearsal mainly combats forgetting [2203.08512].

On the standard Natural-Instructions split, the paper reports category-wise ROUGE-L. Its reimplementation of the original setting reaches mean 39.35. InstructionSpeak without continual learning reaches 41.82. Under forward-transfer evaluation, InstructionSpeak reaches 42.93 mean, including 70.96 on classification and 90.41 on minimal modification. Under backward-transfer evaluation, the mean drops to 32.66, largely because classification degrades to 7.51, which the paper attributes to dilution of category-specific supervision such as label-space regularities [2203.08512].

## 6. Analyses, limitations, and significance

The paper reports several practical analyses. Increasing the size of the initialization set \(k\) consistently improves forward transfer, supporting the interpretation that more initial tasks better teach the model how to use instructions. Transferability also varies by category: question generation, answer generation, and classification tend to benefit more from transfer, whereas minimal modification and verification often benefit less. The proposed explanation is a combination of category frequency in the stream and cross-task similarity [2203.08512].

The qualitative analysis emphasizes that instructions are heterogeneous supervision objects. A task description may include a title, prompt, definition, cautionary notes, “things to avoid,” and positive and negative examples with explanations. One example in the paper shows that a discouraged answer can still be semantically informative even when it violates evaluation formatting rules. This helps motivate the negative-output pretraining strategy. At the same time, the paper notes a concrete failure mode: directly minimizing the probability of negative outputs harms learning because these outputs often share lexical content with correct answers [2203.08512].

Several limitations are explicit. The benchmark is small, with only 61 tasks and at most \(k \le 20\) possible initialization tasks in the original version. Negative examples are scarce and are often improvised for initialization tasks by treating model errors as negative outputs. The method still relies on some positive exemplars inside instructions and does not establish strong pure zero-shot performance from instruction text alone. Evaluation is ROUGE-L-based even for heterogeneous tasks, which may not fully capture task success. Although model capacity is fixed, computational cost still grows with stream length because prior instructions are repeatedly revisited [2203.08512].

The broader conceptual significance is that ConTinTin reframes continual NLP as **continual instruction-conditioned task acquisition**. Instructions become persistent supervision objects: they specify new tasks, carry positive and negative constraints, and can be revisited later as compact memories of previous tasks. This suggests a different architecture for continual learning in language systems—one in which transfer and retention are mediated not only by parameter updates or replay buffers, but by the structure and reuse of human-authored task descriptions themselves [2203.08512].

Source: https://www.emergentmind.com/topics/system-level-instructions