Common Prototype Proposal (CPP)

• CPP is a community-infrastructure initiative that offers an operational platform for natural, scholarly conversational search.
• It aims to counter biased datasets by generating authentic interaction logs through real-use scholarly tasks like literature discovery and conference planning.
• The proposal leverages personal research graphs and privacy-aware designs to support cumulative evaluation and collaborative research challenges.

Searching arXiv for the primary paper and directly related scholarly-conversational infrastructure references. arXiv search query: (Balog et al., 2020) Common Prototype Proposal (CPP) denotes, in the conversational-search literature, a proposal for a shared, operational research substrate around which a community can converge. In the formulation associated with the “Common Conversational Community Prototype,” the CPP is instantiated as a “Scholarly Conversational Assistant”: a system intended to be simultaneously “a useful search tool,” “a means to create datasets for further academic research,” and “a platform for running evaluation challenges by groups across the community” (Balog et al., 2020). It was advanced from discussions at Dagstuhl Seminar 19461 on Conversational Search and is framed as a response to a structural problem in the field: too few natural conversational search corpora, too much dependence on biased datasets derived from imagined needs or constrained collection settings, and too little shared infrastructure for cumulative evaluation (Balog et al., 2020).

1. Definition, scope, and terminological placement

The CPP is not presented as an abstract reference architecture only. It is explicitly envisioned as an operational platform around which the conversational search community can contribute components, gather interactions, and run evaluations (Balog et al., 2020). Its defining feature is commonality at the level of infrastructure: a single system should support real use, generate reusable data, and anchor shared tasks. In this sense, the CPP is a community-infrastructure proposal rather than an algorithmic contribution.

The proposal is conceptual and infrastructural rather than mathematically specified. The paper includes no formal equations, objective functions, notation, scoring formulas, or LaTeX mathematical definitions; the main technical anchor is the system’s reliance on a “personal research graph (PKG)” together with the triad of goals just noted (Balog et al., 2020). This distinguishes the CPP from prototype-based or optimization-centric uses of the same acronym in other domains.

The acronym itself is polysemous across the literature. In unrelated contexts, “CPP” has denoted the C preprocessor in software product lines (Baum et al., 2019), coverage path planning for micro air vehicles (Kuang et al., 8 May 2025), a consistency-driven prototype-prompting framework for generalized intent discovery (Wei et al., 10 Jun 2025), and a prototype common readout board for LHCb upgrades (Langouët, 2024). Within conversational search, however, the relevant meaning is the Common Prototype Proposal centered on a shared scholarly conversational system.

2. Motivation and choice of scholarly search as the initial domain

The proposal begins from a methodological critique. The field has “only few natural conversational search corpora,” while many existing corpora are derived from “imagined needs,” Wizard-of-Oz settings, logs, or forum crawls. The consequence is that “significant research effort” is shaped by biased data and inherited metrics rather than by a common platform built around meaningful and impactful problems (Balog et al., 2020). The CPP is intended to reverse that ordering: first build the platform around real needs, then derive data and evaluation from actual use.

A second motivation is practical. The academic research community repeatedly performs scholarly search and recommendation tasks that are suitable for conversational support but currently yield little reusable research value. Concrete examples include selecting program committee members, finding interest profiles in EasyChair, and identifying relevant sessions in conference apps such as Whova. The proposal argues that the cumulative time spent on such tasks may exceed the cost of building a shared prototype that both helps users and produces research artifacts (Balog et al., 2020).

The choice of scholarly search is itself a design decision. The paper justifies it on four grounds: the data involved is “generally considered less private”; the users are researchers themselves, forming an “active knowledgeable participant base”; the domain supports a broad range of unmet information needs; and it allows academic groups to avoid direct competition with commercial providers (Balog et al., 2020). This contrasts with domains such as movies, music, news, and podcasts, which may have larger audiences but raise stronger privacy concerns and commercial overlap.

Privacy is treated as a “critical constraint.” The proposal cites prior living-lab experience as evidence that privacy can materially limit deployment, which is why scholarly data—papers, public talks, committee memberships, and related activities—was seen as a comparatively tractable starting point (Balog et al., 2020). The domain choice is therefore not merely topical; it is part of the operational feasibility argument for a common prototype.

3. Intended users, recurring tasks, and information needs

The first users are the conversational search research community itself: researchers, conference attendees, organizers, and potentially reviewers or committee chairs (Balog et al., 2020). The system is meant to support recurring scholarly tasks through natural-language interaction rather than through isolated query-response retrieval.

The proposal gives three representative information needs. The first is “What should I read?”, which concerns discovering research in a new area of interest. The second is “Help me plan my attendance”, which concerns choosing sessions to attend and people to talk to at a conference; the paper further notes that organizers could use such information to optimize room allocations. The third is “Whom should I invite?”, which concerns identifying PCs, SPCs, session chairs, invited speakers, and similar roles (Balog et al., 2020).

These examples are significant because they extend beyond ad hoc retrieval. The envisioned assistant covers recommendation, planning, matching, and task completion. Conversational interaction is positioned as essential because the underlying needs are multi-step, underspecified, and personalized (Balog et al., 2020). A conversational interface can iteratively refine intent, ask clarifying questions, and integrate personal context in ways that conventional scholarly search interfaces often do not.

The proposal also expects the system to reveal new use cases over time. By logging all interactions, the platform could identify classes of information needs “that have potential for study” (Balog et al., 2020). This suggests that the CPP is not only a response to known tasks but also an instrument for discovering the empirical structure of scholarly conversational search itself.

4. Architectural substrate and interaction workflow

The assistant is to operate on a personal research graph, drawing on Balog and Kenter’s PKG agenda. More precisely, it uses “the portion of the PKG that the user wants to share with the system” (Balog et al., 2020). This is a central design decision: the common prototype is not merely a generic scholarly graph query system, but a conversational layer over partially shared personal and public scholarly knowledge.

The PKG may include authorship information linked to a public citation graph, conference committee membership and awards, talks given in public settings, conference and session attendance, and—in the private portion—annotations of papers and notes on talks (Balog et al., 2020). This combination of public and personal signals is meant to support needs that are personalized without requiring full disclosure of private scholarly activity.

This suggests an architecture with a conversational interface, logging and feedback collection, retrieval and recommendation over a public-plus-personal research graph, and privacy-aware storage and control over which parts of the PKG are shared. The paper points to Macaw as “an extensible conversational information seeking platform” supporting multiple interfaces and modalities, and to TIRA Integrated Research Architecture as “a modularized platform for shared tasks,” which together indicate a plausible implementation strategy for front-end interaction and evaluation infrastructure (Balog et al., 2020).

The interaction workflow is explicit. Users conduct conversations around realistic scholarly needs; the system logs all interactions; and after each conversation users may evaluate the system through a questionnaire, optionally with free-text feedback, and may “possibly leave comments behind for individual system utterances” (Balog et al., 2020). The consequence is that the platform is instrumented to produce rich supervision signals: naturally occurring interaction traces, whole-dialogue ratings, free-text assessments, and potentially utterance-level commentary.

The interface may also support voice input and speech recognizers. The paper notes this possibility while also identifying associated risks, especially privacy concerns and recognition difficulty for proper names and technical vocabulary (Balog et al., 2020). The resulting architecture is therefore multimodal in ambition but privacy-sensitive in execution.

5. Data production, benchmarks, and community evaluation

A central premise of the CPP is that one operational system can generate multiple classes of academic resources. The paper repeatedly frames these resources as “tools, data, and evaluation approaches” (Balog et al., 2020). More concretely, the prototype is expected to generate realistic conversational logs grounded in real information needs, corpora of dialogues and conversational search sessions, post-conversation questionnaires, optional free-text feedback, possibly utterance-level comments, samples of supported information needs, and reusable benchmarks and performance measures derived from shared tasks (Balog et al., 2020).

The proposal also names candidate knowledge sources for the scholarly layer: ORKG, Semantic Scholar, ACM DL, dblp, ACL Anthology, OpenReview, Google Scholar, Citeseer, Arnetminer, and conference attendance apps such as Whova (Balog et al., 2020). These are presented as candidate raw materials rather than as a completed benchmark.

As a benchmarking instrument, the role of the prototype is explicit. One of its purposes is to serve as “a platform for running evaluation challenges by groups across the community” (Balog et al., 2020). The paper aligns this ambition with traditions from IR and NLP, invoking Cranfield/TREC-style evaluation and CoNLL-style shared tasks as models for how common corpora and performance measures become reusable benchmarks.

The evaluation roadmap extends beyond static benchmarking. The proposal explicitly mentions living labs and evaluation-as-a-service. Shared tasks may be developed “in the form of living labs,” allowing early systems to be deployed in practice while generating realistic evaluation data; later, the same skeleton could support a workshop shared task or a challenge at TREC or CLEF (Balog et al., 2020). The CPP is therefore envisioned as a common experimental substrate: common APIs, common data, shared settings, and direct comparability across systems.

A recurring technical tension concerns offline data release. The paper notes that offline datasets are possible, but warns that deletion-rights and private-data removal issues become difficult once data has been distributed (Balog et al., 2020). This makes the living-lab and evaluation-as-a-service framing especially important, since it offers a route to realistic evaluation without assuming unrestricted redistribution of private interaction data.

6. Governance, limitations, and incremental development path

The proposal does not specify a formal governance model, steering committee, or institutional charter. It nonetheless identifies several operational requirements. Some research team must “own the decisions” about who gets value from the work and oversee development, and a live service requires coordination “of how live experiments are planned and executed” (Balog et al., 2020). Governance is thus underdeveloped institutionally but recognized as indispensable practically.

Sustainability is treated as a major constraint. Academic systems are often deployed on fragile, non-redundant servers; the proposed service would likely need cloud sponsorship or a host institution with “significant cluster resources”; and the hosting decision should account for “long-term commitment” (Balog et al., 2020). Because contributors will change over time, the paper emphasizes modularization, clear interfaces, and documentation. Stability and reproducibility become especially important if the platform hosts online challenges with participant-submitted code.

The paper’s discussion of limitations is unusually extensive. It identifies risks in privacy and data retention, especially for logged interactions and voice input; in the distinction between opinion and fact in indexing, since signals such as tweets about papers are not factual in the same way as citation or authorship metadata; in speech recognition, because scholarly dialogue contains many proper names and technical terms; in PKG implementation, which must support both cloud-side and client-side storage while preserving user control; in low usage volume, which could make the platform uninformative unless incentives such as payment or gamification are introduced; and in the sheer implementation and operational burden of building, scaling, and maintaining the system (Balog et al., 2020).

There is also a competitive limitation. Numerous systems for scholarly publications and conference data already exist, including dblp, Semantic Scholar, ACM DL, Google Scholar, ACL Anthology, OpenReview, arXiv, Athena, Citeseer, Arnetminer, and ArXivDigest. For the prototype to succeed, it must “either be more useful than these, or potentially integrate with them” (Balog et al., 2020). The CPP is therefore not justified by novelty of access to scholarly metadata alone; its claim rests on conversational support, shared evaluation, and community instrumentation.

The implementation plan is deliberately incremental. The authors write that “the project is ambitious, but we think it can be grown incrementally” (Balog et al., 2020). Two entry paths are proposed. One is implementation-first: recruit graduate students, start coding, check the system into GitHub, and build on existing infrastructure. The other is evidence-first: begin by collecting evidence that the community actually wants the system, for example through sample dialogues or information needs (Balog et al., 2020). From there, the envisioned trajectory is early prototype release, community use and interaction logging, workshop or lab shared tasks, possible TREC/CLEF-scale challenges, and eventually living-lab-style continuous evaluation.

Taken together, the CPP defines a community infrastructure agenda for conversational search. Its central claim is that progress requires a shared, real-use scholarly assistant rather than another isolated dataset or one-off system. The proposed Scholarly Conversational Assistant is meant to serve practical scholarly work, expose realistic conversational behavior, generate reusable resources, and align evaluation across groups. Its significance lies less in algorithmic novelty than in the proposition that a field can organize itself around a single operational prototype and thereby make realistic information needs, interaction traces, and cumulative benchmarking co-evolve (Balog et al., 2020).