MemExplorer: Meme & Memory System Exploration

• MemExplorer is a dual-purpose framework that extracts meme context evidence and optimizes heterogeneous memory systems for agentic inference.
• It employs a multimodal approach with models like KME, MAT, and MA-LSTM to align visual, textual, and contextual knowledge in meme analysis.
• For memory co-design, MemExplorer uses multi-objective Bayesian optimization to balance throughput and energy efficiency in NPUs.

MemExplorer encompasses two distinct but thematically related research streams: the first concerns evidence mining for social-media memes via knowledge-enriched multimodal context modeling, while the second addresses the automated exploration and co-design of heterogeneous memory subsystems for next-generation agentic inference NPUs. Both are foundational efforts at the intersection of context-sensitive interpretation—semantic in the former, architectural in the latter.

1. Overview and Definition

The term "MemExplorer" refers to two primary research developments. In the context of computational linguistics and multimodal reasoning, MemExplorer often serves as a shorthand for advanced systems that mine, interpret, and align explanatory context for internet memes by bridging visual, textual, and world-knowledge gaps. This is exemplified by the MEMEX framework and its benchmark system MIME, detailed in MEMEX: Detecting Explanatory Evidence for Memes via Knowledge-Enriched Contextualization (Sharma et al., 2023).

A parallel, infrastructure-centric usage arises in systems architecture, wherein MemExplorer designates a memory system synthesizer and optimizer for heterogeneous NPU-based (Neural Processing Unit) inference clusters, as formally introduced in MemExplorer: Navigating the Heterogeneous Memory Design Space for Agentic Inference NPUs (Wu et al., 17 Apr 2026). This tool embodies automated, constraint-aware design space exploration for compute-memory co-design across numerous memory technologies.

2. MEMEX Task and the Meme Context Corpus

The MEMEX task is formally defined as sentence-level evidence detection for meme explanations. Given a meme $M = (I, T)$, where $I$ is the raw image and $T$ the OCR-extracted overlaid text, and a context document $C = \{c_1, \ldots, c_n\}$, the objective is to select a subset $E \subseteq C$ such that $E$ succinctly explains the meme's background. The learning objective models $P(\vec{y}|M, C) = \prod_{i=1}^n P(y_i|M, c_i)$, with forced binary decisions $\hat{y}_i=1$ if $P(y_i|M, c_i) > 0.5$.

This task demands modeling of cross-modal alignment between the multifaceted content of memes and factual textual sentences, often under high abstraction and minimal explicit cues.

The Meme Context Corpus (MCC) underpins empirical work on this task, comprising 3,400 meme–context pairs curated via web search and Reddit crawling, with context primarily sourced from Wikipedia and supplementary web resources. Annotation guidelines emphasize minimal self-contained evidence units and inter-annotator agreement (Cohen's $\kappa$ up to 0.72). The dataset is segmented 80%/10%/10% for training, validation, and test (Sharma et al., 2023).

3. MIME Model: Architecture and Methodology

MIME (MultImodal Meme Explainer) represents a layered, knowledge-enriched multimodal solution structured as follows:

(a) Knowledge-Enriched Meme Encoder (KME): The model extracts a joint representation $I$0 from the image and OCR text using MMBT, infuses commonsense via GCN-derived ConceptNet subgraph embeddings ($I$1), and combines them with learned gated fusion (parameterized by $I$2).

(b) Meme-Aware Transformer (MAT): The context sentences are BERT-encoded; meme-conditioned multi-head attention is effected by broadcasting $I$3 into key/value transformations with dynamic gating ($I$4).

(c) Meme-Aware LSTM (MA-LSTM): Processes the sentence representations sequentially, with each gate modulated by $I$5, ensuring persistent meme influence at all steps.

(d) Output and Loss: The fused representations $I$6 are scored via MLP and sigmoid, yielding binary evidence predictions. Binary cross-entropy loss is used for optimization.

Training employs Adam optimizer ($I$7, $I$8, $I$9), a learning rate of $T$0, batch size of 16, dropout $T$1, and mixed-precision for memory efficiency.

4. Evaluation, Results, and Analysis

MIME and baselines (BERT, ViT, VisualBERT, MMBT, CLIP, BAN) are evaluated using F1, accuracy, macro-precision/recall, and exact match (E-M) over test splits. MIME achieves F1 of $T$2 (up $T$3 pts over best multimodal baseline), macro precision $T$4, recall $T$5, and exact match $T$6. Ablation reveals that the removal or downgrading of KME, MAT, or MA-LSTM submodules each reduce F1 by $T$7 pts, underscoring their functional necessity.

Qualitative assessment highlights typical failure modes: missing predictions due to abstractness or insufficient evidence granularity (12.5%), over-inclusion of peripheral sentences (14%), and lexical bias-induced errors (14%). Salient challenges involve modeling irony or sarcasm, addressing sparse textual cues, and mitigating spurious lexical overlaps.

5. MemExplorer for Heterogeneous Memory Co-Design

In the domain of accelerator architectures for agentic LLM inference, MemExplorer denotes a synthesizer and optimizer for memory system and NPU co-design. It is architected to accommodate diverse memory technologies (SRAM, HBM variants, LPDDR, GDDR, high-bandwidth flash) via a unified abstraction: each memory type is characterized by capacity ($T$8), peak bandwidth ($T$9), fixed latency ($C = \{c_1, \ldots, c_n\}$0), static background power ($C = \{c_1, \ldots, c_n\}$1), and per-bit energy ($C = \{c_1, \ldots, c_n\}$2). Key constraints include physical shoreline (Eq. 1), effective double-buffered bandwidth (Eq. 2), and parametric power consumption (Eq. 3).

A system configuration $C = \{c_1, \ldots, c_n\}$3 encompasses compute parameters (e.g., PE array size, VLEN), on-/off-chip memory layer/stacks per tech, quantization precisions, and software scheduling priorities.

6. Optimization Formulation and Search Algorithms

MemExplorer formalizes design space exploration (DSE) as a bi-objective optimization: $C = \{c_1, \ldots, c_n\}$4 subject to constraints on total power, physical attach region, bandwidth, and analytic model specifics.

The search algorithm is multi-objective Bayesian optimization (MOBO) with Expected Hypervolume Improvement (EHVI) acquisition, employing independent GP surrogates for the two objectives. Initialization is by $C = \{c_1, \ldots, c_n\}$5 Sobol seedings, followed by iterative candidate evaluation and GP updating up to $C = \{c_1, \ldots, c_n\}$6. Comparative results show superior Pareto convergence versus NSGA-II or random search (Wu et al., 17 Apr 2026).

7. Workload Characterization and Empirical Results

MemExplorer evaluation spans representative agentic inference tasks, including BFCL-WebSearch (Qwen3-32B) and OSWorld-LibreOffice (Llama3-70B), dissected into prefill (activation-bound) and decode (KV cache-bound) phases. Key system baselines include a fixed-memory PLENA NPU and commercial H100/A100 GPUs.

Empirical findings (700 W TDP) are summarized as:

• Prefill-optimized design achieves $C = \{c_1, \ldots, c_n\}$7 throughput and $C = \{c_1, \ldots, c_n\}$8 energy efficiency over baseline NPU; $C = \{c_1, \ldots, c_n\}$9 vs. H100.
• Decode-optimized solution attains $E \subseteq C$0 throughput, $E \subseteq C$1 Token/J vs. baseline ($E \subseteq C$2 vs. H100).
• System-level impact: TTFT reduced by 40%, throughput up to $E \subseteq C$3, Token/J up to $E \subseteq C$4 on multi-device setups.
• Case studies (e.g., LLaDA-8B, Qwen3.5-397B) report Token/J gains of $E \subseteq C$5 (diffusion) and up to $E \subseteq C$6 (large MoE prefill).

The aggregate Pareto frontier reveals a monotonic increase in Token/J and throughput with higher permissible device power and memory hierarchy rearrangement.

---

MemExplorer, in both semantic alignment and hardware optimization instantiations, epitomizes the utility of principled, data/model-driven exploration in resolving challenges at the boundary of context inference—be it extracting the latent “backstory” of memes through multimodal model design (Sharma et al., 2023), or navigating the combinatorial design space of memory systems to efficiently serve agentic LLM workloads under stringent physical constraints (Wu et al., 17 Apr 2026).