PosterHarness: Turning Scientific Poster Generation into an Auditable Instruction-Following Benchmark

Abstract: Text-rich image models can now design poster-scale layouts, but we lack ways to measure whether they honor scientific communication contracts: legible labels, prescribed aspect ratios, and -- above all -- abstaining from fabricated scientific figures. We present POSTERHARNESS, an auditable harness reframing poster generation as measurable instruction-following tasks, with a pilot benchmark and failure taxonomy. POSTERHARNESS uses a placeholder-first contract to separate two jobs models otherwise conflate. The model performs visual-summary design: typography, reading path, color, and background -- but never draws data-bearing figures. Every figure region must be an empty labeled placeholder; a deterministic compositor inserts real source-paper figures at detected coordinates. This makes properties measurable: placeholder count and ID accuracy, blankness, aspect-ratio compliance, abstention from synthesized graphics, public-text hygiene, and source-figure provenance -- with failures logged as explicit rejections, not hidden in plausible-looking output. We instantiate the harness on 12 papers (6 HEP, 6 AI/ML-adjacent) and report three findings. (i) A counterfactual probe shows the placeholder contract drives VLM-counted synthesized figures from 34 to 0 across three papers. (ii) A failure taxonomy identifies blocking contracts: placeholder geometry, placeholder QA, template critic, and public text. (iii) Comparison with Paper2Poster shows a trade-off: PosterHarness yields higher-resolution artifacts, lower white-canvas fraction, and stronger VLM visual preference; the deterministic baseline retains slightly more PosterQuiz-style information and runs faster. We report this as regime characterization, not a superiority claim. All artifacts, prompts, manifests, and audit scripts are released as a reusable evaluation component.

• The paper introduces a novel benchmark that separates visual layout design from deterministic figure insertion to ensure auditable scientific poster generation.
• It employs a multi-stage pipeline integrating LLM/VLM planning, explicit placeholder contracts, and QA-driven regeneration to verify integrity and provenance.
• Results demonstrate enhanced visual quality and zero hallucinated figures, albeit with increased latency and cost due to rigorous audits.

PosterHarness: An Auditable Benchmark for Instruction-Following in Scientific Poster Generation

Motivation and Problem Statement

PosterHarness addresses the persistent challenge in automating scientific poster generation: ensuring the faithfulness and auditable provenance of scientific figures within visually sophisticated, text-rich layouts. Traditional approaches—either deterministic, template-based renderers or fully generative image models—have clear limitations. Deterministic systems such as Paper2Poster ("Paper2Poster: Towards Multimodal Poster Automation from Scientific Papers" (Pang et al., 27 May 2025)) offer strict figure provenance but suffer from visually stilted results constrained by PPTX-style grid layouts. Conversely, diffusion or image-generation models can produce visually compelling posters, but they often fabricate scientific figures, violating critical contracts of verifiability and numerical integrity required in scientific communication. Even with advances in multimodal models such as GPT-4o ("Preliminary Explorations with GPT-4o(mni) Native Image Generation" (Cao et al., 6 May 2025)), free-form image synthesis cannot be trusted to represent authentic scientific data.

PosterHarness reframes scientific poster generation as a multi-stage, measurable instruction-following problem, where compliance with a battery of explicit contracts can be independently audited. The system's central architectural decision is to separate visual-summary design from scientific-figure rendering. Image models handle only global composition, typography, and non-data-bearing art; all data-carrying figure regions are rendered as labeled, blank placeholders, replaced deterministically with source-figure assets under strict geometry and aspect-ratio constraints. With this separation, PosterHarness makes measurable several properties previously conflated in end-to-end generative approaches: placeholder count and accuracy, blankness, aspect-ratio adherence, absence of hallucinated scientific plots, public-text hygiene, and explicit, logged failure propagation.

Figure 1: Overview of the PosterHarness pipeline, showing separation of image-model-based layout design and deterministic source-figure insertion.

System Architecture and Pipeline

PosterHarness employs a multi-stage workflow integrating LLM/VLM planning, declarative domain profiles, explicit placeholder contracts, QA-driven regeneration, and deterministic replacement. The pipeline is as follows:

• Paper Ingestion: Uses automated PDF/PPTX parsing to extract text units and all available figure assets, associating them with the paper’s structure.
• Domain Classification: An LLM classifier routes each paper input through profile selection (e.g., HEP, CS/ML, Bio), with configuration files defining layout spines, figure rules, and forbidden decorative elements.
• Planning: The system emits structured specifications—section layout, storyboard, copy-decks, figure-to-placeholder assignments—via LLMs operating under constraints from the selected domain profile and information density presets.
• Template Generation: A state-of-the-art image model is prompted with four-layered constraints, including the explicit placeholder contract, to generate a high-resolution visual summary template. Placeholders are strictly labeled and must remain blank of any scientific content.
• Critic-Guided QA and Regeneration: A VLM-based template critic audits each candidate template across dimensions such as visual quality, artistry, information density, and strict placeholder compliance. Failed generations trigger guided regeneration with structured, prompt-appended critique until acceptance or explicit failure.
• Geometry Analysis and Insertion: Detected placeholders are verified for accuracy (IoU, aspect ratio), then filled with source-figure assets via deterministic rendering. Containment and geometric fidelity are strictly enforced.
• Final QA and Micro-Repair: Minor public-facing defects (e.g., text mistakes) are patched locally; any geometry or placeholder contract failures result only in regeneration or rejection.
• Output and Audit Trails: All artifacts, intermediate prompts, specifications, figures, QA results, and logs are saved for full run-level transparency and later audit.

  Figure 2: Example showing left: labeled placeholder template, right: final poster with deterministic source-figure insertion.

  

  Figure 3: Illustration of localized defect repair: header typo correction via targeted image edit without altering layout geometry.

Evaluation Methodology

PosterHarness is instantiated on a cost-aware, cross-domain benchmark of 12 curated scientific papers (6 HEP, 6 non-HEP AI/ML/bio). Metrics capture the essential instruction-following properties:

• Source-Figure Provenance: A deterministic proxy test and containment audit confirm that all accepted scientific figures are directly sourced, with zero model-synthesized plots.
• Visual and Information Quality: A single VLM produces blinded pairwise preference scores (visual quality, overall impression) and supports PosterQuiz-style information recall tests for paper content read directly from rendered posters.
• Run Audits and Failure Taxonomy: Manifests log successful runs, explicit failures, retries, and failure types (e.g., placeholder geometry rejection, QA failures).
• Prompt Ablations: Control probes isolate the effect of the placeholder contract: model generations with and without explicit placeholder instructions are compared for figure hallucination rates.

  Figure 4: Effect of placeholder prompt: left, direct image model baseline with many synthesized plots and no provenance; right, explicit placeholder contract preserving compositional slots for auditable insertion.

  

  Figure 5: Side-by-side visual comparison: Paper2Poster (deterministic, grid-like, information-dense) versus PosterHarness (editorial design, rich color/composition, source-figure preservation).

Results and Numerical Evidence

Key findings include:

• Perfect Provenance Compliance: On all 12 benchmark posters, accepted scientific-figure regions are exclusively composed of source assets after deterministic replacement.
• Visual Quality Regime Shift: PosterHarness outputs have an average native resolution of $25.17$ Mpx (vs. Paper2Poster’s $2.86$ Mpx), a much lower white-canvas fraction ($0.199$ vs $0.677$), and higher color entropy (mean $3.15$ bits vs $2.15$), demonstrating an editorial-quality design regime rarely achievable with grid-based deterministic systems.
• Instruction-Following Impact: In a three-paper prompt ablation, the explicit placeholder contract reduces VLM-counted hallucinated scientific figures from $34$ (no placeholder) to $0$. All placeholder IDs, blankness, and aspect labels are correctly rendered only under the full placeholder contract.
• Trade-offs: PosterHarness achieves higher visual preference scores in VLM-based audits (20/24 pairwise wins, mean score $4.42$ vs $3.38$), but Paper2Poster preserves slightly higher PosterQuiz-style information recall ($2.86$0 vs $2.86$1). Generation cost and latency are higher for PosterHarness ($2.86$2 min vs $2.86$3 min), primarily due to QA-induced retries and strict failure propagation.
• Failure Signals: Audit logs attribute most failures to placeholder-geometry and blankness rejections—exactly the contracts this architecture is designed to expose.

Theoretical and Practical Implications

PosterHarness substantiates that separating composition from source-data insertion enables more transparent, auditable, and visually compelling poster generation without sacrificing the core tenet of scientific figure provenance. The results advocate for the incorporation of hard architectural contracts, prompt engineering, and multi-stage VLM-driven audits in downstream visual document synthesis tasks.

Declarative, YAML-based domain profiles generalize the framework, allowing community-driven adaptation of domain-specific poster conventions. The capability to plug in higher-density presets (as in HEP) or to extend profiles to new scientific domains (bio, chem, astro) suggests scalability and broad applicability.

The system shifts the benchmarking paradigm for multimodal instruction-following from aesthetic-only or agent-evaluation tasks (“did you generate a good-looking poster?”) toward engineering assertions of compliance (“did you follow every measurable contract for auditably trustworthy scientific communication?”). Importantly, critical properties (e.g., source-figure inclusion, placeholder accuracy, aspect compliance) can be checked deterministically, not merely via subjective model or human scoring.

Future Directions

Expanding PosterHarness to population-scale benchmarks (50-100+ papers) is needed to generalize claims. Further integration with native vector formats (PDF/SVG/HTML) and richer, discipline-specific domain profiles is essential for downstream human editability and print-ready outputs. Human expert evaluations (preregistered, blinded studies) will be necessary to complement single-VLM pilot ratings, especially to quantify the value of dense-mode and domain-adaptive profiles.

Technically, future work should ablate each prompt and audit component, compare performance with critic-disabled or one-shot settings, and extend the system to act as a visual reading interface for scientific search platforms and publishers—where transparent provenance and audit artifacts are critical for user trust.

Conclusion

PosterHarness demonstrates that an explicit, auditable placeholder-first pipeline enables scalable, instruction-following scientific poster generation with strict provenance guarantees and measurable compliance on key visual and data contracts. The paradigm provides not only a more trustworthy approach for AI-assisted scientific summarization but also a reusable benchmark for structured evaluation of multimodal instruction-following in large foundation models. Open-sourcing the entire system—including prompts, domain profiles, artifacts, and evaluation code—provides a reproducible basis for continued research on trustworthy AI in scientific communication.