Output Embedding as a Valid Embedding

• Output embeddings are vector representations that map discrete outputs to a continuous space while preserving key semantic, predictive, and decision-theoretic structures.
• They are constructed through methods like tied embeddings in language models, nCCA in multimodal tasks, and calibrated surrogate losses in structured prediction.
• Empirical studies show that valid output embeddings enhance model performance by reducing perplexity and improving parameter efficiency and semantic coherence.

An output embedding is a vector representation—often realized as the topmost weight matrix in a neural network model—that acts as a mapping from a discrete output space (such as vocabulary, labels, or structured predictions) to a continuous space, with the goal of capturing semantic, predictive, or decision-theoretic structure. Research across modern language modeling, computer vision, and surrogate loss design now systematically investigates when and why such output embeddings constitute "valid embeddings" for the original target space, often in the sense of preserving statistical or semantic information, maintaining consistency properties, and supporting downstream tasks.

1. Definitions and Canonical Examples

An embedding is a mapping $\phi: \mathcal{A} \rightarrow \mathbb{R}^d$ from a discrete set $\mathcal{A}$ (words, classes, reports) into a continuous space, such that salient properties—e.g., statistical roles, relationships, or semantics—are preserved. In the context of output embeddings, the mapping is typically realized as a learned matrix or parameter set whose rows correspond to output entities.

• Language Modeling (LM): The output embedding $V \in \mathbb{R}^{C \times H}$ is the weight matrix mapping RNN/LSTM hidden states $h_t$ to vocabulary scores, via logits $\ell_t = V h_t$. Each row $V_k$ represents the embedding for word $k$, and the softmax probability is $$p(o_t|h_t) = \frac{\exp(V_{o_t}^\top h_t)}{\sum_x \exp(V_x^\top h_t)}$$ (Press et al., 2016).
• Multimodal Learning: In visual question answering, joint output embeddings map both visual and linguistic information into a shared space; e.g., mean box pooling yields image features, which are projected along with candidate textual answers into a space where similarity encodes the match quality (Mokarian et al., 2016).
• Surrogate Losses: For structured prediction or classification, output/decision embeddings are used to embed discrete reports into $\mathbb{R}^d$ for the design of convex surrogate losses, enabling consistent statistical learning (Finocchiaro et al., 2022).

2. Theoretical Guarantees of Output Embeddings

The validity of an output embedding is domain-specific but often formalized through a preservation of relationships or consistency properties. In neural language modeling, the output embedding $V$ is proven to act as a distributional embedding: words with similar conditional distributions $p(\cdot|h)$ converge to similar rows in $V$. Theoretically, this arises from the cross-entropy training objective, which encourages embeddings of semantically or functionally similar outputs to be geometrically proximal (Press et al., 2016).

In convex surrogate design, a surrogate loss $L: \mathbb{R}^d \to \mathbb{R}^{+Y}$ is said to embed a discrete loss $\ell: R \to \mathbb{R}^{+Y}$ if there exists an injective map $\phi: S \to \mathbb{R}^d$ from a representative set $S$ such that:

• $L(\phi(s)) = \ell(s)$ for all $s \in S$,
• The mapping preserves minimizers over the output simplex, providing calibration and enabling linear regret bounds between surrogate and original decision losses (Finocchiaro et al., 2022).

3. Practical Construction and Training

Construction of valid output embeddings depends on the application domain and optimization objective.

• LLMs:
  • Untied Case: The output embedding $V$ is updated at every step for every possible output.
  • Tied Embeddings: Setting input and output embeddings equal ($U=V$) leads to more uniform updates and enforces sharing, which empirically improves generalization and model compactness without degrading intrinsic embedding quality (Press et al., 2016).
  • Regularization: When over-regularization occurs, introducing a projection matrix $P$ before softmax with mild $\ell_2$ weight decay ameliorates the issue.
• Multimodal Embeddings (Visual Madlibs):
  • Both images (via mean box pooling) and textual answers (via mean/sum-pooled word2vec vectors) are projected via normalized CCA (nCCA) into a joint space. Cosine similarity is used during inference and training to enforce alignment between correct image-answer pairs and separation from negatives (Mokarian et al., 2016).
• Surrogate Loss Embeddings:
  • Polyhedral surrogates: Given a discrete loss, construct a convex polyhedral surrogate whose minimizers correspond to those of the original loss; the embedding map's existence and calibration are guaranteed when the Bayes risks match (Finocchiaro et al., 2022).

Table 1. Core Mechanisms in Output Embedding Construction

Domain	Embedding Form	Validity Criterion
Language Modeling	$V \in \mathbb{R}^{C\times H}$	Distributional similarity
Multimodal (VQA)	Joint nCCA projections	Cosine similarity for correct pairs
Surrogate Losses	$\phi$ into $\mathbb{R}^d$	Embedding of minimizers and calibration

4. Empirical Outcomes and Calibration

Empirical studies demonstrate the superiority or consistency of output embeddings as valid vector representations:

• In LLMs, tying embeddings reduces perplexity, increases parameter efficiency (up to ~52% reduction for three-way tying in translation), and maintains or exceeds word similarity task performance compared to input embeddings. For example, Penn Treebank LSTMs with tied embeddings reach lower test perplexity than untied models (Press et al., 2016).
• In multimodal settings (Visual Madlibs), output embeddings aligned via nCCA and mean box pooling yield a +5.9 percentage point improvement on "easy" and +1.4 on "hard" tasks over global-image CCA baselines. The joint space enhances discriminability and semantic coherence (Mokarian et al., 2016).
• In polyhedral surrogate design, embedding is necessary and sufficient for calibration (i.e., consistency) and ensures linear regret transfer. Examples include binary hinge loss for 0–1 classification and abstain surrogates; polyhedral constructions yield matched Bayes risks that guarantee valid embedding (Finocchiaro et al., 2022).

5. Limitations and Invalid Embeddings

Output embeddings are not universally valid. Notably:

• Some surrogates, such as the logistic loss for binary classification or certain multiclass SVM top-$k$ surrogates, are not polyhedral and do not embed the desired loss; their Bayes risks differ. Consistency may still hold under weaker theoretical frameworks, but embedding-based calibration fails (Finocchiaro et al., 2022).
• In small LLMs without additional regularization, simple weight tying can cause over-regularization, affecting expressiveness. Introducing a regularized projection improves matters (Press et al., 2016).

6. Applications in Structural and Semantic Domains

Output embeddings support a range of advanced applications:

• Semantic Similarity: In LLMs, output embeddings are usable in downstream similarity tasks, often matching or surpassing the quality of input embeddings.
• Knowledge Graphs and Ontologies: Geometric output embeddings for description logics (e.g., EL++) produce vector-space interpretations that act as certified models of logical TBoxes, thus supporting model-theoretic reasoning and downstream prediction (such as protein–protein interactions) (Kulmanov et al., 2019).
• Structured Prediction: Embedding frameworks enable the principled design of surrogate losses for structured outputs (rankings, abstain, top-$k$) that maintain statistical consistency under surrogate minimization (Finocchiaro et al., 2022).

7. Summary and Significance

Output embeddings function as valid embeddings when the encoding preserves essential properties of the original output space: distributional similarity, semantic coherence, or minimizer correspondence in the context of surrogate losses. Theoretical frameworks and empirical work establish the conditions under which such embeddings maintain calibration, discriminability, and optimality transfer, directly impacting the performance and reliability of modern machine learning systems (Press et al., 2016, Mokarian et al., 2016, Finocchiaro et al., 2022).