Semantic-Based Pseudo Interaction Construction

• Semantic-based pseudo interaction construction is a method that transforms natural-language feedback into a revised interaction sequence, merging long-term behavior with immediate intent.
• It employs a dual alignment masking module and a Constructor LLM to filter out outlier items and rewrite user histories for enhanced recommendation accuracy.
• This approach bridges sequential and conversational recommendation systems, offering actionable insights for improving real-time adjustment in diverse domains.

Semantic-based pseudo interaction construction denotes a class of methods that convert semantically interpretable signals into synthetic or rewritten interaction structures that can be consumed by a downstream model. In its most explicit recent formulation, the term refers to sequential recommendation with conversational feedback, where natural-language preference statements are transformed into a pseudo-interaction sequence that rewrites a user’s historical item sequence so that it reflects both long-term behavior and real-time intent (Wang et al., 11 Sep 2025). More broadly, closely related constructions appear in weakly supervised segmentation, cross-modal alignment, semantic correspondence, clustering, and semantic question answering, where an intermediate semantic representation is not treated as an incidental by-product but as the decisive supervision interface between weak evidence and a final prediction model (Li et al., 2021).

1. Conceptual definition and problem setting

In sequential recommendation, the base setting is standard next-item prediction. Given a user $u \in \mathcal{U}$ and historical interaction sequence

$$\mathcal{I}(u) = \{v^{(u)}_1, v^{(u)}_2, \dots, v^{(u)}_{N_u}\},$$

a sequential recommender predicts the next item. CESRec augments this process by first generating a recommendation, then collecting conversational feedback, and then refining the original sequence into a pseudo-interaction sequence (Wang et al., 11 Sep 2025). Operationally, a pseudo interaction is an item inserted or substituted into the historical sequence based on the semantics of conversational feedback, so that the modified sequence better reflects current intent. The refined sequence is written as

$$I_{\text{pseudo}(u)} = \text{Constructor}(I'(u), \text{feedback}),$$

where $I'(u)$ is the masked historical sequence after outlier removal and $\text{feedback}$ is the user’s natural-language response (Wang et al., 11 Sep 2025).

The motivation is that sequential recommendation systems and conversational recommendation systems exhibit complementary strengths and opposite weaknesses. Sequential methods model long-term preference from historical behavior, usually through collaborative filtering and sequential patterns, but do not adapt well to real-time intent shift. Conversational systems can capture immediate preference through dialogue, but often ignore historical interaction sequences (Wang et al., 11 Sep 2025). Semantic-based pseudo interaction construction is introduced precisely to bridge this gap by translating conversational semantics into a sequence-level signal that ordinary sequential recommenders can consume.

A key property of this formulation is that the updated user sequence is not merely appended with text. Rather, it is a transformed item sequence that is passed back into the recommender: $$v^{(u)}_{N_u+1} = \text{SRS}(I_{\text{pseudo}(u)}).$$ This makes the method model-agnostic in the sense described by CESRec: after sequence refinement, the downstream recommender can remain a standard model such as SASRec or an LLM-based sequential recommender such as LLaRA (Wang et al., 11 Sep 2025).

This suggests a general definition across domains: semantic-based pseudo interaction construction uses semantic evidence to build an intermediate surrogate structure—sequence items, pseudo-labels, pseudo-masks, classifiers, correspondences, or interaction options—that is then consumed by a downstream learner. In several related papers, the exact phrase “pseudo interaction” is not used, but the construction principle is structurally similar (Zhuang et al., 2017, Zou et al., 2020, Ma et al., 13 Oct 2025).

2. CESRec architecture and sequence rewriting mechanism

CESRec organizes semantic-based pseudo interaction construction as a five-stage pipeline (Wang et al., 11 Sep 2025). First, a standard sequential recommender runs on the original sequence and produces an initial recommendation. Second, the user provides natural-language feedback about that recommendation. Third, a dual-alignment masking module computes semantically and collaboratively aligned item representations and masks historical outliers. Fourth, a Constructor LLM consumes the masked sequence and the feedback text and outputs a pseudo-interaction sequence. Fifth, the original sequential recommender is run again on that pseudo sequence to produce the final recommendation.

The feedback signal is produced after the initial recommendation: $\text{feedback} = \text{User-Interaction}(v^{(u)}_{\text{rec}, Attr_{\text{target})}.$ The paper’s illustrative example is that, if the system recommends Avatar but the preference is for Christopher Nolan films, the user may say, “I don't like film directed by James Cameron; I prefer Christopher Nolan” (Wang et al., 11 Sep 2025). That utterance is then used to semantically alter the interaction history.

The central design choice is sequence rewriting rather than architectural redesign of the recommender. CESRec states that the pseudo interactions are not arbitrary synthetic clicks; they are semantically plausible item substitutions meant to rewrite historical evidence in a way that reconciles long-term preference with current conversational intent (Wang et al., 11 Sep 2025). In the MovieLens case study, the system masks an outlier action/animation movie and, after feedback “I don't like comedy; I prefer horror,” replaces “Cops and Robbersons” with “Carnosaur 2” inside the historical sequence, preserving sequence length and order while changing item identity (Wang et al., 11 Sep 2025).

The paper repeatedly uses the word “replace,” but does not formalize a deterministic replace-or-append rule. It also does not specify reordering, timestamp modification, or explicit item reweighting. The most faithful interpretation is therefore masked-item-aware sequence rewriting, largely through semantic substitution (Wang et al., 11 Sep 2025).

A concise representation of the end-to-end update is:

$I(u)' = \text{Dual-Alignment}(I(u)),$

$$I_{\text{pseudo}(u)} = \text{Constructor}(I'(u), \text{feedback}),$$

$$v^{(u)}_{N_u+1} = \text{SRS}(I_{\text{pseudo}(u)}).$$

This formulation matters because it converts language into an object that legacy sequential recommenders already know how to process: a revised item sequence rather than an auxiliary text feature (Wang et al., 11 Sep 2025).

3. Semantic-collaborative alignment and pseudo-sequence generation

The pseudo-sequence construction depends on a prior denoising stage. CESRec identifies “outlier items” as historical interactions that deviate from the user’s core preference, such as accidental clicks or transient interests, and masks them before sequence construction (Wang et al., 11 Sep 2025). Each item $$\mathcal{I}(u) = \{v^{(u)}_1, v^{(u)}_2, \dots, v^{(u)}_{N_u}\},$$0 is associated with content information $$\mathcal{I}(u) = \{v^{(u)}_1, v^{(u)}_2, \dots, v^{(u)}_{N_u}\},$$1, such as the item title. An LLM-based extractor produces a semantic embedding: $$\mathcal{I}(u) = \{v^{(u)}_1, v^{(u)}_2, \dots, v^{(u)}_{N_u}\},$$2 Because semantic similarity alone may not preserve collaborative patterns important in recommendation, CESRec introduces an adapter that projects the LLM semantic embedding into the recommender’s collaborative embedding space: $$\mathcal{I}(u) = \{v^{(u)}_1, v^{(u)}_2, \dots, v^{(u)}_{N_u}\},$$3 and the appendix specifies the adapter as an MLP: $$\mathcal{I}(u) = \{v^{(u)}_1, v^{(u)}_2, \dots, v^{(u)}_{N_u}\},$$4 It is trained by aligning the projected semantic representation to the recommender’s collaborative embedding $$\mathcal{I}(u) = \{v^{(u)}_1, v^{(u)}_2, \dots, v^{(u)}_{N_u}\},$$5 using

$$\mathcal{I}(u) = \{v^{(u)}_1, v^{(u)}_2, \dots, v^{(u)}_{N_u}\},$$6

The paper characterizes this as “dual alignment” because the hybrid embedding combines semantic meaning from item text with collaborative structure from the sequential recommender space (Wang et al., 11 Sep 2025).

Outlier detection then aggregates the hybrid embeddings of all items in the sequence to form a user representation: $$\mathcal{I}(u) = \{v^{(u)}_1, v^{(u)}_2, \dots, v^{(u)}_{N_u}\},$$7 where $$\mathcal{I}(u) = \{v^{(u)}_1, v^{(u)}_2, \dots, v^{(u)}_{N_u}\},$$8 is mean pooling. Each item is scored by cosine similarity to that user representation: $$\mathcal{I}(u) = \{v^{(u)}_1, v^{(u)}_2, \dots, v^{(u)}_{N_u}\},$$9 Items with the lowest similarity are treated as outliers, and the top $$I_{\text{pseudo}(u)} = \text{Constructor}(I'(u), \text{feedback}),$$0 such items are masked, producing

$$I_{\text{pseudo}(u)} = \text{Constructor}(I'(u), \text{feedback}),$$1

In the main setup, one item is masked, although additional analysis varies the number of masked items (Wang et al., 11 Sep 2025).

After masking, CESRec performs semantic pseudo sequence construction through a module called the Constructor, implemented as a fine-tuned LLM. The Constructor takes the masked historical sequence and the feedback text and outputs a new sequence $$I_{\text{pseudo}(u)} = \text{Constructor}(I'(u), \text{feedback}),$$2 (Wang et al., 11 Sep 2025). The paper does not provide an explicit closed-form matching score, retrieval index, candidate generation mechanism, or threshold for replacement. Instead, the Constructor is treated as a sequence generator conditioned on history and feedback. The training objective is an autoregressive sequence prediction loss: $$I_{\text{pseudo}(u)} = \text{Constructor}(I'(u), \text{feedback}),$$3

Because no naturally labeled pseudo-sequence dataset exists, CESRec synthesizes supervision. It randomly samples items from the interaction sequence as “Outlier Items,” uses the target item as the ground-truth signal, and derives feedback from the transition between the outlier item and the target item (Wang et al., 11 Sep 2025). This means the Constructor is trained to generate a revised sequence that better matches a target preference shift.

The paper does not present a single unified end-to-end CESRec loss. Rather, it gives the main components separately: $$I_{\text{pseudo}(u)} = \text{Constructor}(I'(u), \text{feedback}),$$4 for the Constructor, $$I_{\text{pseudo}(u)} = \text{Constructor}(I'(u), \text{feedback}),$$5 for the adapter, and the backbone sequential recommender’s own standard training objective (Wang et al., 11 Sep 2025).

4. Empirical behavior, iterative refinement, and practical constraints

CESRec reports that adding semantic-based pseudo interaction construction to SASRec improves performance on all three datasets discussed in the paper. On Video Games, SASRec improves from HR@5 $$I_{\text{pseudo}(u)} = \text{Constructor}(I'(u), \text{feedback}),$$6 and NDCG@5 $$I_{\text{pseudo}(u)} = \text{Constructor}(I'(u), \text{feedback}),$$7 to $$I_{\text{pseudo}(u)} = \text{Constructor}(I'(u), \text{feedback}),$$8 and $$I_{\text{pseudo}(u)} = \text{Constructor}(I'(u), \text{feedback}),$$9 with CESRec-LLaMA3. On MovieLens, HR@5 rises from $I'(u)$0 to $I'(u)$1, and on Toys from $I'(u)$2 to $I'(u)$3 (Wang et al., 11 Sep 2025). The paper attributes this to pseudo-interaction sequences that incorporate current feedback and help the recommender capture real-time preference.

The ablation “CESRec w/o c.” removes the Constructor and keeps only dual alignment masking. On Video Games, this yields HR@5 $I'(u)$4, below full CESRec at $I'(u)$5, though still above SASRec at $I'(u)$6. Similar patterns hold on MovieLens ($I'(u)$7 vs. $I'(u)$8) and Toys ($I'(u)$9 vs. $\text{feedback}$0) (Wang et al., 11 Sep 2025). This isolates pseudo interaction construction itself as a substantial part of the gain. The variant “CESRec w/o d.a.” keeps pseudo construction but removes dual alignment masking, and it performs below full CESRec, indicating that masking improves pseudo-sequence quality by reducing semantic bias from outliers (Wang et al., 11 Sep 2025).

The paper also studies iterative conversational feedback. As the number of feedback rounds increases, HR and NDCG steadily improve across datasets (Wang et al., 11 Sep 2025). This indicates that pseudo interaction construction can function not only as a one-shot rewrite, but as an iterative sequence refinement mechanism.

Implementation details further constrain the method’s practical form. CESRec mentions LLaMA-2-7B and LLaMA-3-8B as Constructor backbones and uses LoRA for fine-tuning. For user feedback simulation in experiments, it uses GPT-4o-mini as a user simulator. Outlier masking is reported as fast, around $\text{feedback}$1–$\text{feedback}$2 seconds, whereas pseudo sequence construction dominates runtime, roughly $\text{feedback}$3–$\text{feedback}$4 seconds depending on the LLM (Wang et al., 11 Sep 2025).

Several limitations are stated explicitly. The method depends on the availability of natural-language conversational feedback. If feedback is vague or ambiguous, pseudo sequence construction may become inaccurate. CESRec also requires item textual content such as titles to compute semantic embeddings, so it is not purely ID-based. Training pseudo sequences relies on synthetic supervision by sampling outlier items and using target items as preferred replacements, which may not perfectly reflect real conversational transitions. Finally, masking too many items can hurt performance by shortening usable history and discarding useful evidence (Wang et al., 11 Sep 2025).

5. Related formulations in segmentation, vision, and representation learning

The same construction principle appears in other areas, although the terminology often differs. In weakly supervised semantic segmentation, “Pseudo-mask Matters” argues that pseudo-masks are not merely an intermediate artifact but the decisive supervision interface between image-level semantics and dense segmentation learning (Li et al., 2021). Its PMM framework redesigns both pseudo-mask construction from class activation maps and training under noisy pseudo-mask supervision, introducing Coefficient of Variation Smoothing, Proportional Pseudo-mask Generation, Pretended Under-Fitting, and Cyclic Pseudo-mask (Li et al., 2021). Here the pseudo object is not an item sequence but a dense semantic mask; nevertheless, the common pattern is that semantic evidence is transformed into an intermediate supervision structure that a downstream model consumes.

PseudoSeg follows a similar logic for semi-supervised semantic segmentation, but with a different construction mechanism. It generates calibrated soft pseudo labels by fusing decoder predictions with self-attention Grad-CAM maps, then applies them in a one-stage weak-to-strong consistency framework (Zou et al., 2020). The paper explicitly states that it does not use the phrase “semantic-based pseudo interaction construction,” yet its pseudo labels are constructed from segmentation predictions, localization-driven semantic priors, class presence information, and self-attention propagation over hypercolumn features (Zou et al., 2020). This suggests a broader methodological family in which pseudo supervision is made semantically richer by combining multiple structured signals before it is reused for training.

In unsupervised domain adaptation for segmentation, the notion of semantic interaction is moved from labels to model branches. “Unsupervised Domain Adaptation with Implicit Pseudo Supervision for Semantic Segmentation” uses a tri-learning architecture in which every two branches produce pseudo labels to train the third one, and aligns those pseudo labels based on probability-distribution similarity (Xu et al., 2022). DIDA extends this idea by coupling semantic-level and instance-level consistency and regenerating pseudo labels through scattering and gathering between class space and an instance bank (Yao et al., 2023). SeCo pushes the supervision unit from pixels to semantic connectivities, using SAM-based aggregation and a connectivity classifier to retain or relabel region-level pseudo labels (Zhao et al., 2023). Across these papers, the downstream learner is trained not from raw predictions alone but from semantically structured pseudo objects—agreement-filtered pixels, regenerated semantic labels, or connectivity masks.

A different but closely analogous construction appears in interaction recognition. “Towards Context-aware Interaction Recognition” does not use the term “pseudo interaction,” but explicitly constructs a context-dependent interaction classifier from subject and object semantics encoded with word2vec: $\text{feedback}$5 This allows the model to generate a classifier for unseen subject-object contexts through semantic transfer (Zhuang et al., 2017). A plausible implication is that semantic-based pseudo interaction construction need not always synthesize new data instances; it can also synthesize the decision function applied to an interaction.

In cross-modal alignment, PICO reweights feature-level image-text interaction using pseudo semantic probabilities derived from matched pairs and refined through iterative style-prototype construction (Ma et al., 13 Oct 2025). The local interaction

$\text{feedback}$6

turns pseudo semantic reliability into a gate on patch-word coupling (Ma et al., 13 Oct 2025). Again, the pseudo object is neither a sequence nor a mask, but a reliability-weighted interaction channel.

6. Broader interpretation, adjacent paradigms, and recurring design patterns

Beyond recommendation and segmentation, semantic-based pseudo interaction construction also appears as a strategy for building surrogate correspondences, clarifications, or bridges where direct supervision is unavailable. DIY-SC constructs pseudo semantic correspondences across images using nearest-neighbor matching, 3D-aware chaining, relaxed cyclic consistency, and spherical prototype filtering, then trains an adapter on those pseudo correspondences (Dünkel et al., 5 Jun 2025). SPICE constructs prototype-centered and reliability-filtered pseudo labels for clustering, combining prototype pseudo-labeling and reliable pseudo-labeling in a staged training procedure (Niu et al., 2021). IQA, in semantic question answering, does not generate pseudo labels in the segmentation sense, but it formalizes an interaction-option space over candidate semantic parses and supports simulated interaction by accepting or rejecting options according to a gold query (Zafar et al., 2020). This offers a formal route to pseudo interaction trajectories in dialogue-based semantic parsing.

Two further papers sharpen the idea of semantic bridging. “Semantic Bridging Domains: Pseudo-Source as Test-Time Connector” treats a pseudo-source subset not as a replacement for the source domain but as a semantic bridge, rectifying its features with universal semantics from a pretrained model before aligning the remaining target data (Yang et al., 4 Mar 2026). “Data-free Universal Adversarial Perturbation with Pseudo-semantic Prior” recursively extracts pseudo-semantic priors from the evolving perturbation itself, creating semantic samples by crop-and-resize and reweighting them by KL-derived hardness (Lee et al., 28 Feb 2025). In both cases, semantics are used to construct an intermediate object that mediates learning where original supervision is missing.

Several recurring design patterns are visible across these formulations.

Pattern	Instantiation	Example paper
Semantic rewriting	Conversational feedback rewrites a historical item sequence	(Wang et al., 11 Sep 2025)
Semantic structuring	Weak predictions are converted into masks, connectivities, or calibrated soft labels	(Li et al., 2021, Zou et al., 2020, Zhao et al., 2023)
Cross-view interaction	One branch, view, or space supervises another through agreement or calibration	(Xu et al., 2022, Yao et al., 2023)
Semantic transfer	Semantics generate classifiers, correspondences, or interaction weights for unseen cases	(Zhuang et al., 2017, Dünkel et al., 5 Jun 2025, Ma et al., 13 Oct 2025)

A plausible synthesis is that semantic-based pseudo interaction construction is best understood not as a single algorithmic template but as a design principle: when direct supervision is unavailable, weak, noisy, or mismatched to the downstream model, semantic evidence can be reorganized into a surrogate interaction object that is structurally closer to the learner’s native input space. In CESRec, that object is a pseudo-interaction sequence (Wang et al., 11 Sep 2025). In PMM, it is a pseudo-mask that serves as the supervision bridge between image-level labels and dense segmentation (Li et al., 2021). In PICO, it is a semantic probability that reweights cross-modal feature interaction (Ma et al., 13 Oct 2025). In IQA, it is an interaction option whose acceptance or rejection prunes a semantic query space (Zafar et al., 2020).

A common misconception is to treat these pseudo constructions as simple post-processing. The cited papers consistently present them as central modeling components rather than auxiliary heuristics. PMM states that pseudo-masks are the decisive supervision interface in weakly supervised segmentation (Li et al., 2021). CESRec shows that removing the Constructor materially reduces gains even when denoising remains (Wang et al., 11 Sep 2025). PICO’s ablations identify weighting of interaction as the dominant factor in performance change (Ma et al., 13 Oct 2025). The broader significance is therefore methodological: the quality, semantics, and update rule of the pseudo interaction often determine how effectively weak signals can be converted into usable supervision for a downstream system.