PKBench: Dual Benchmark for Text & Animation

• PKBench is a dual-purpose benchmark framework that evaluates both top‑k text analytics and generative cartoon production using authentic datasets.
• For text analytics, it extends T²K² to assess DBMS performance on tweet datasets, employing TF-IDF and Okapi BM25 for detailed query response measurements.
• For cartoon production, it simulates artist workflows with sparse human-drawn keyframes to quantify animation coherence and visual fidelity using reference-free metrics.

PKBench refers to two distinct, technical benchmarks designed for different domains: one for top‑k keyword and document processing in text analytics, and another for cartoon production evaluation using human-drawn keyframes. Both are constructed to simulate real-world challenges in their respective fields, leveraging authentic data and representative workloads to advance performance assessment and guide future development.

1. Definitions and Scope

PKBench denotes:

• The multidimensional evolution of T${}^2$K${}^2$ (i.e., T${}^2$K${}^2$D${}^2$) for evaluating top‑k keyword and document extraction in text mining workloads (Truica et al., 2018).
• A dedicated benchmark for generative cartoon production, evaluating animation coherence and visual fidelity on real human-drawn sketches (Li et al., 14 Aug 2025).

Each benchmark explicitly addresses unmet requirements in its field by offering authentic evaluation datasets and protocol-driven metric reporting.

2. PKBench in Top‑k Text Analytics

PKBench as T${}^2$K${}^2$D${}^2$ is constructed to support rigorous evaluation of top‑k keyword/document extraction algorithms and database system implementations:

• Schema: The original normalized schema organizes authors, documents, gender, geo-location, words, and vocabulary frequency links. The T${}^2$K${}^2$D${}^2$ extension adopts a star schema, with a central Document_Fact table parameterized over word, time, author, and location dimensions.
• Workload Model: Benchmarks feature a real tweet dataset (up to 2.5 million tweets) processed at configurable scale factors. Queries Q1–Q4 are defined by progressive constraints—on gender (c₁), date (c₂), location (c₃), and keywords (c₄)—and are formalized in relational algebra notation.
• Weighting Algorithms: Term scoring is directly implemented by TF-IDF and Okapi BM25 formulas:

$$\mathrm{TF}(t, d) = K + (1-K)\cdot\frac{f_{t,d}}{\max_{t'\in d} f_{t',d}}$$

$\mathrm{IDF}(t, D) = 1 + \log \frac{N}{n}$

$$\mathrm{TFIDF}(t, d, D) = \mathrm{TF}(t,d)\cdot\mathrm{IDF}(t,D)$$

$$\mathrm{Okapi}(t, d, D) = \frac{\mathrm{TFIDF}(t, d, D) \cdot (k_1 + 1)}{\mathrm{TF}(t,d) + k_1 \cdot (1-b + b \cdot ||d||/\mathbb{E}_{d'\in D}||d'||)}$$

Parameters $K$ (for TF), $k_1$ and $b$ (for BM25), control term frequency and document length normalization.

• Implementations: PKBench is deployed on relational systems (Oracle, PostgreSQL) and document-oriented systems (MongoDB using aggregation pipelines or MapReduce).
• Performance Metrics: The principal metric is query response time over warm runs, with scale-out experiments, average/standard deviation reporting, and system-optimized sharding.

3. PKBench for Cartoon Production

The second PKBench, introduced with ToonComposer (Li et al., 14 Aug 2025), is designed to quantitatively evaluate generative cartoon production in realistic settings:

• Dataset Composition: Consists of 30 cartoon scenes, each sample containing:
  • A colored reference frame setting appearance/style.
  • A textual scene prompt.
  • Two human-drawn keyframe sketches executed by professional artists (typically the start/end frames).
• Workflow Simulation: Samples mimic artist workflow—sparse, high-quality sketches instead of dense, per-frame annotations. The benchmark encapsulates style diversity, sketch variation, and authentic imprecisions, making it a challenging and realistic testbed for post-keyframing generation methods.
• Metric Design: PKBench employs reference-free metrics (derived from VBench) including:
  • Aesthetic Quality (A.Q.): Perceptual fidelity of generated frames.
  • Motion Consistency (M.C.): Temporal smoothness of animation between sparse keyframes.
  • Subject Consistency (S.C.) & Background Consistency (B.C.): Stability of foreground objects and background elements.
  • These metrics sample both qualitative patches and aggregate quantitative measures. Human studies provided additional preference/efficiency scoring.

4. Comparative Evaluation Frameworks

The design of both PKBench variants is aimed at benchmarking not just algorithmic accuracy, but also the real-world system and workflow efficiency:

PKBench Domain	Primary Function	Key Metrics
Text Analytics	Top‑k extraction in databases	Query time, scalability
Cartoon Production	Generative animation with sparse input	Quality, consistency, efficiency

In the text analytics benchmark, axis of comparison include DBMS type (Oracle, PostgreSQL, MongoDB), query complexity, and scale factor. In cartoon production, comparison is made against prior methods for inbetweening and colorization, with paired user studies and objective metrics promoting grounded evaluation.

5. Technical Significance and Genericity

Both PKBench methodologies are instances of domain-specific benchmarking that emphasize genericity, relevance, and scalability:

• Genericity: PKBench for text enables top‑k processing on any textual corpus with parameterized queries and modular schemas. PKBench for cartoon production captures representative artistic workflows across varied scenes and styles.
• Design Principles: Benchmarks are structured according to widely accepted benchmarking principles—relevance, portability, simplicity, and scalability (per Jim Gray (Truica et al., 2018)). Constraints and metrics can be adjusted for system-specific needs.
• Domain Utility: PKBench benchmarks inform system optimization for text mining, DBMS selection for analytics, and guide development of generative models in animation pipelines. The use of authentic, labor-intensive datasets (tweets and professional sketches) elevates their utility as rigorous, real-world standards.

6. Limitations and Evaluation Considerations

PKBench’s reliance on reference-free evaluation in cartoon production is a deliberate consequence of the absence of ground truth for each human sketch sample. A plausible implication is that metrics—and visual patch comparisons—must be interpreted in the context of artistic intention, not absolute pixel agreement. In the text analytics domain, tradeoffs are compounded by implementation differences (e.g., aggregation pipelines vs. MapReduce in NoSQL, star-schema vs. normalized schema in SQL).

Common misconceptions may include conflating PKBench with synthetic or fully automated benchmarks; however, both variants use real, manually curated datasets and focus on replicating the data and workflow nuances encountered in professional practice.

7. Practical Application and Future Directions

PKBench frameworks serve as prototypical standards in their fields, enabling:

• Side-by-side analysis of algorithmic and system response to structured workloads.
• Systematic assessment of computational tradeoffs, efficiency bottlenecks, and workflow fidelity.
• Benchmark-based troubleshooting and performance tuning in data engineering and creative production environments.

This suggests that future directions may extend PKBench datasets to additional modalities (e.g., comics, informal documents, other animation genres) and incorporate new, domain-driven metrics, thereby sustaining relevance as data processing, generative modeling, and professional workflows evolve.