Systematic Multivocal Literature Review

• Systematic MLR is a methodical review technique that synthesizes evidence from both peer-reviewed and gray literature to create a comprehensive evidence map.
• It follows a rigorous, multi-stage protocol including planning, search strategy, study selection, and integrated synthesis to ensure reliability and transparency.
• By merging academic insights with real-world industry data, MLRs bridge the gap between theory and practice, guiding informed decision-making and innovation.

A Systematic Multivocal Literature Review (MLR) is a protocol-based secondary study that synthesizes evidence from both peer-reviewed ("formal") literature and non–peer-reviewed ("gray") literature such as blogs, technical reports, white papers, and practitioner slide decks. The objective is to obtain a holistic, evidence-based map that rigorously integrates the “state-of-the-art” with the “state-of-the-practice.” MLRs are distinguished from classical Systematic Literature Reviews (SLRs) by their deliberate and methodologically explicit inclusion of gray literature, providing an augmented evidence base that captures both academic rigor and industrial insight (Garousi et al., 2017, Kamei et al., 2021).

1. Formal Definition and Rationale

In the context of software engineering and applied research domains, a Systematic Multivocal Literature Review (MLR) is defined as a secondary research method that seeks to answer a set of predefined research questions $R$ by collecting, evaluating, and synthesizing evidence units $E$ from both traditional peer-reviewed sources ($T$) and nontraditional or non–peer-reviewed ("gray") sources ($G$). The evidence corpus $E$ is thus constructed as $E = E_T \cup E_G$ with $|G| > 0$, and both components are subjected to comparable scrutiny during planning, inclusion/exclusion, quality appraisal, extraction, and synthesis (Kamei et al., 2021).

The rationale for MLR is outlined by Garousi et al.: MLRs are indicated when a research topic is complex, under-represented in formal literature, or characterized by rapid industry innovation where key findings reside in gray channels. Typical motivations include bridging gaps between academic research and practitioner experience, validating or challenging academic theories with real-world data, and synthesizing best practices in domains with extensive practitioner-generated guidance (Garousi et al., 2017, Tarhan et al., 2019).

2. Multistage Methodological Framework

A Systematic MLR follows a rigorously documented, multi-phase protocol that parallels classical SLR practice with additional GL-specific adaptations (Garousi et al., 2017, Wang et al., 2022, Uulu et al., 22 Jul 2025):

1. Planning and Protocol Registration:
   • Motivate the use of MLR via a GL-relevance checklist.
   • Define research questions using frameworks such as GQM (Goal-Question-Metric) or PICOC.
   • Specify clear inclusion and exclusion criteria applicable to both $T$ and $G$ sources.
2. Search Strategy:
   • Develop separate, Boolean-enhanced search strings for formal databases (e.g., Scopus, IEEE Xplore) and web/GL sources (e.g., Google Search, arXiv).
   • Use piloting and snowball keyword refinement to maximize recall and precision (Wang et al., 2022).
   • Employ theoretical saturation or effort-bound stopping for GL searching (e.g., first $N$ Google hits or until new codes cease emerging) (Garousi et al., 2017).
3. Study Selection and Quality Assessment:
   • Apply dual (or triple) independent reviewer voting for both $E$0 and $E$1 sources, resolving disagreements by consensus or majority rule (Garousi et al., 2017, Tarhan et al., 2019, Qasse et al., 3 Apr 2025).
   • For gray literature, implement a quality checklist covering: producer authority, methodology clarity, objectivity, novelty/date, referenced position, outlet type, and impact (e.g., backlinks, social metrics) (Garousi et al., 2017, Qasse et al., 3 Apr 2025).
   • Establish a scoring schema (e.g., 1/0.5/0 per criterion) and set source retention thresholds (e.g., ≥ 0.5 normalized score).
4. Data Extraction and Coding:
   • Use traceable extraction templates mapping each data item to the RQ it serves.
   • Distinguish coding frameworks for $E$2 and $E$3 when warranted, but support unified thematic structures for synthesis.
   • Employ open, axial, and selective coding borrowed from grounded theory where relevant (especially for complex or practice-driven domains) (Tarhan et al., 2019, Uulu et al., 22 Jul 2025).
5. Synthesis and Analysis:
   • Integrate $E$4 and $E$5 streams using side-by-side tables, comparative frequency analyses, and narrative synthesis.
   • Quantify GL contribution via metrics such as GL Contribution Ratio:

   $E$6

   and Source Balance Index:

   $E$7

   (Kamei et al., 2021).

6. Reporting:

   • Distinguish $E$8- vs $E$9-driven findings.
   • Organize by contribution type, research type, best practices, models, success factors, etc.
   • Provide method transparency (PRISMA diagrams, extraction tables), describe validity threats, and encourage data sharing.

3. Typology of Sources and Quality Appraisal

A Systematic MLR incorporates a broad spectrum of evidence sources. Types of gray literature identified empirically as most frequent and impactful include blog posts, slide presentations, project/software descriptions, whitepapers, technical reports, webinars, and guideline documents. Table 1 below presents the distribution observed in a tertiary study across nine MLRs (Kamei et al., 2021):

GL Type	Relative Frequency (%)	Typical Contribution Roles
Blog post	30.7	Recommendation, Opinion, Explanation
Slide presentation	11.7	Recommendation, Solution, Explanation
Project description	10.9	Solution, Tool, Recommendation
Whitepaper	6.5	Explanation, Recommendation
Technical report	6.5	Explanation, Classification

GL quality is assessed via multi-dimensional checklists, scoring for author credibility, clarity of aims and methods, evidence of supporting data, currency, referenced position, and outlet type (with "Tier 1" reserved for reports, white papers, theses, and "Tier 3" for opinion blogs) (Garousi et al., 2017, Qasse et al., 3 Apr 2025).

4. Taxonomies, Frameworks, and Synthesis Techniques

MLRs in computing fields frequently establish conceptual taxonomies (e.g., monitored aspects in ML systems, types of software test automation best practices, upgrade mechanisms for smart contracts), which are derived via thematic coding and iteratively refined classification schemas (Naveed et al., 17 Sep 2025, Wang et al., 2022, Qasse et al., 3 Apr 2025):

• Component Taxonomy: e.g., for ML monitoring: Data, Model Behavior, Operations & Infrastructure, Responsible ML, Business (Naveed et al., 17 Sep 2025).
• Techniques/Tools Taxonomy: e.g., statistical, distance-based, learning-based anomaly detection, performance/robustness metrics (Naveed et al., 17 Sep 2025).
• Best Practices Taxonomy: e.g., in test automation: strategy development, resource allocation, tool selection, script quality, knowledge transfer, measurement, SUT design (Wang et al., 2022).
• Success Factors: e.g., in CGSD: time-to-market acceleration, robust support structure, security management, integrated process governance (Akbar et al., 2022).

The synthesis approach includes:

• Quantitative counting/frequency analyses of $T$0 and $T$1 contributions.
• Qualitative thematic/narrative synthesis, highlighting convergence, divergence, and exclusive GL contributions.
• Reciprocal analysis to juxtapose industry vs. academic “voices” by coding category (Uulu et al., 22 Jul 2025).
• Formal grading of empirical support (e.g., flagging which practices have been robustly evaluated) (Wang et al., 2022).
• Conflict analysis to catalog disagreement between $T$2 and $T$3 sources and surface open research gaps.

5. Empirical Impact and Domain-Specific Insights

MLRs have demonstrated that GL supplies uniquely practice-driven, context-sensitive, and timely evidence often absent in formal channels. Empirical results frequently show that substantial proportions of novel techniques, patterns, or best practices are derived exclusively or predominantly from gray literature—e.g., in one review, 90.5% of Android architectural guidelines were found only in practitioner blogs, and several risk types in DevOps were identified solely in white papers and web sources (Kamei et al., 2021).

Practitioner evidence is often used for solution proposals, actionable recommendations, tool and platform selection heuristics, and detailed technical explanations, while formal literature dominates conceptual models, rigorous comparative studies, and controlled empirical work. Domains characterized by rapid technology turnover (e.g., ML monitoring, cloud-based development, MLOps) or where codified knowledge lags practice are especially reliant on multivocal synthesis (Naveed et al., 17 Sep 2025, Eken et al., 2024).

6. Limitations, Challenges, and Recommendations

Common challenges in executing MLRs arise from discoverability (GL’s dispersed, non-indexed nature), variable metadata quality, GL quality heterogeneity, and traceability of extraction. Strategies to mitigate these include:

• Digital traceability schemes linking each GL artifact to its RQ and coded output.
• Dual/triple independent review and formal consensus for selection/quality.
• Publication and archiving of full source lists, extraction spreadsheets, and coding schemas to ensure transparency and replicability (Tarhan et al., 2019, Garousi et al., 2017).
• Explicit reporting of limitations and sensitivity to missing GL or paywalled reports.

Guideline recommendations consistently emphasize: upfront justification for GL inclusion, pre-registered protocols, systematic and transparent quality filtering for GL, separate synthesis and analysis by source type, and community-accessible dissemination of extraction and synthesis artifacts.

7. Emerging Trends and Methodological Innovations

Recent advances include the integration of LLM-based assistants for initial search and filtering in MLR workflows, with quality metrics such as Positive Percent Agreement (PPA) to ensure model-mediated decisions remain within human-level reliability bounds (Matalonga et al., 16 Sep 2025). Reciprocal analysis frameworks and layered QA workstreams (e.g., LLM-first-pass plus manual calibration) are gaining traction for dual-stream narrative synthesis, particularly in domains where evidence volume or heterogeneity is unmanageable with human labor alone (Uulu et al., 22 Jul 2025, Matalonga et al., 16 Sep 2025).

The trajectory of MLR research highlights increasing discipline in evidence base balance, harmonization of taxonomies and terminologies, and systematic benchmarking of contributions from both peer-reviewed and practitioner-driven sources.

---

References:

• (Garousi et al., 2017) Guidelines for including grey literature and conducting multivocal literature reviews in software engineering
• (Kamei et al., 2021) What Evidence We Would Miss If We Do Not Use Grey Literature?
• (Wang et al., 2022) Improving Test Automation Maturity: a Multivocal Literature Review
• (Eken et al., 2024) A Multivocal Review of MLOps Practices, Challenges and Open Issues
• (Qasse et al., 3 Apr 2025) The Myth of Immutability: A Multivocal Review on Smart Contract Upgradeability
• (Uulu et al., 22 Jul 2025) AI for Better UX in Computer-Aided Engineering: Is Academia Catching Up with Industry Demands? A Multivocal Literature Review
• (Matalonga et al., 16 Sep 2025) Accelerating Discovery: Rapid Literature Screening with LLMs
• (Naveed et al., 17 Sep 2025) Monitoring Machine Learning Systems: A Multivocal Literature Review
• (Akbar et al., 2022) Successful Management of Cloud Based Global Software Development Projects: A Multivocal Study
• (Tarhan et al., 2019) Maturity assessment and maturity models in healthcare: A multivocal literature review