MAST Framework for AI System Evaluation

• MAST is a framework that defines nine standards for evaluating AI and ML systems, emphasizing data provenance, uncertainty quantification, and logical explanation.
• It employs a four-point Likert scale and adjustable weighting to quantitatively benchmark system trustworthiness and credibility.
• Applied in identity verification and text summarization, MAST guides both prescriptive design and post-hoc evaluation to enhance operational relevance.

The Multisource AI Scorecard Table (MAST) is a tradecraft-derived evaluation and design tool for AI and ML systems, originating in the intelligence and defense communities to ensure the trustworthiness, transparency, and operational relevance of AI-enabled decision support. MAST formalizes criteria from Intelligence Community Directive (ICD) 203, distilling analytic tradecraft into nine concrete standards covering every stage of the data and model lifecycle—from data provenance and uncertainty quantification to logical explanation, accuracy, and visualization. The framework is implemented as a unified scorecard, supporting both system design and quantitative evaluation by developers, operators, and stakeholders in sensitive domains (Blasch et al., 2021, Salehi et al., 2023, Cohen et al., 2024).

1. Foundation and Structure

MAST was established by the ODNI/DHS Analytic Exchange Program with direct grounding in the nine standards of ICD 203, a doctrine adopted across the US intelligence community for analytic rigor. The nine criteria are:

1. Sourcing: Quality and credibility of sources, data, and methodology.
2. Uncertainty: Expression and explanation of uncertainties in judgments.
3. Distinguishing: Delineation between factual inputs and model/analyst assumptions.
4. Analysis of Alternatives (AoA): Assessment of alternative scenarios or hypotheses.
5. Customer Relevance: Tailoring outputs to user-specific needs and context.
6. Logical Argumentation: Clarity of evidence combination and inferential chains.
7. Consistency: Explanation of changes or consistency in outputs over time.
8. Accuracy: Presentation of quantitative performance/likelihoods/metrics.
9. Visualization: Effective use of visuals to clarify or augment analytics.

Each criterion is defined with reference to both human analytic standards and the unique requirements of ML systems, spanning data sourcing, model logic, and deployment/relevance phases (Blasch et al., 2021, Salehi et al., 2023, Cohen et al., 2024).

2. Scoring Methodology and Scorecard Design

The MAST scorecard operationalizes these criteria as a table or checklist. For each system or major component, rows represent the nine criteria. Columns capture: (i) the standard, (ii) paraphrased evaluation question, (iii) system feature(s) mapping to the criterion, and (iv) responsible development stage.

Scoring is performed on a four-point Likert scale, with scores per criterion m_i ∈ {1=Poor, 2=Fair, 3=Good, 4=Excellent}. The aggregate MAST score is:

$$\mathrm{MAST}_{\text{total}} = \sum_{i=1}^{9} w_i m_i,$$

where w_i are importance weights (uniform by default). This yields a possible range 9–36. For finer granularity, a normalized score is obtained by

$$\bar{S} = \frac{\sum_{i=1}^9 w_i m_i}{\sum_{i=1}^9 w_i}.$$

Developers can adjust weights for domain priorities—e.g., higher for accuracy in safety-critical applications. The scorecard functions both as a quantitative benchmark and as a completeness checklist in system design (Blasch et al., 2021, Cohen et al., 2024). Internal consistency (Cronbach’s α) reaches 0.91 in empirical studies (Salehi et al., 2023).

3. Application and Use Cases

MAST has been applied to a wide spectrum of AI-DSS prototypes, including identity verification (Facewise), document summarization (READIT), and sensor fusion, across government, security, and analytical contexts.

• Facewise (ID Verification): The high-MAST variant incorporated data-source exposition, confidence metrics, and analytic explanations mapped directly to each criterion. The low-MAST version reduced or omitted these features. Evaluator studies showed significantly higher MAST scores and perceived trust for the high-MAST variant, though not necessarily improved user task accuracy (Salehi et al., 2023).
• READIT (Text Summarization): The high-MAST prototype provided model/data sheets, uncertainty/cosine similarity scoring, and several interactive visualizations. Again, high-MAST scoring correlated with higher perceived trust, credibility, and reduced risk, but not with increased report accuracy.
• Other Domains: MAST has informed iterative design of security reporting, healthcare decision aids, and strategic forensic platforms, and is explicitly adapted via PADTHAI-MM as an iterative design methodology for trustworthy AI in intelligence and defense (Cohen et al., 2024).

Use-case summary tables map each feature to relevant MAST criteria and capture rationale and achieved scores (see (Blasch et al., 2021) Table 2–4 for exemplars and methodology).

4. Practical Guidance for Design and Evaluation

MAST is intended for use in two main modes:

• Prescriptive Design Checklist: During system development, map candidate features to MAST criteria to identify gaps in source transparency, argument logic, or user-facing explanations early in the process (Cohen et al., 2024). This enables teams to manage trade-offs in complexity, computational cost, and user cognitive load.
• Post-hoc Evaluation: After deployment or prototyping, have expert evaluators rate each MAST criterion against actual system features. High scores indicate strong alignment with analytic tradecraft and are empirically associated with elevated end-user trust and reduced perceived risk (Salehi et al., 2023).

Empirical results consistently show that higher MAST scores predict higher reported trust and benefit, as well as lower perceived risk and higher message credibility, though often without improvement in objective performance metrics (Salehi et al., 2023, Cohen et al., 2024). This suggests that MAST is primarily a perception- and trust-oriented tool, not a direct measure of downstream task effectiveness.

5. Validation, Impact, and Limitations

Validation has been conducted on multiple cohorts, including subject matter experts in transportation security and defense intelligence. Studies demonstrate that high-MAST designs reliably yield stronger trust, higher message credibility, and lower perceived risk, with R² values ranging from 0.30 to 0.74 across trust- and risk-related outcome regressions (Salehi et al., 2023). Principal components analysis shows that MAST scores are the major predictor of the main “positive trust/benefit perception” component in user evaluations.

Nevertheless, the framework does not guarantee improved user-AI joint performance or accuracy. Certain criteria, such as Customer Relevance, can be insensitive to increases in provenance or explanation, depending on domain expectations and task structure. Another limitation is that MAST’s scoring is moderately subjective and can be domain-specific; further research is needed into more objective rubrics and domain-adapted criterion mappings (Blasch et al., 2021, Salehi et al., 2023, Cohen et al., 2024).

6. Best Practices, Recommendations, and Future Directions

MAST’s practical deployment should be coupled with explicit mapping of planned features to criteria, iterative evaluation (A/B testing, prototyping, and user studies), and—where needed—domain-specific weighting or adaptation of the nine standards (Cohen et al., 2024). Common recommendations include:

• Early integration of MAST in development to inform feature prioritization.
• Employing MAST alongside routine usability and performance testing, to ensure that increases in perceived trust also support objective utility.
• Extending or customizing MAST criteria for high-risk domains (e.g., privacy or robustness in medical AI).
• Developing automated or semi-automated tools for real-time or continuous MAST evaluation, to reduce scoring subjectivity and reviewer burden.
• MAST-aligned documentation (datasheets, model cards) to fulfill Sourcing, Uncertainty, and Distinguishing criteria.

Areas for future research include dynamic and context-sensitive scoring, integration with fairness and robustness frameworks, expansion to ensemble and multi-agent systems, and establishing MAST as a certification metric for procurement or regulatory compliance (Blasch et al., 2021, Cohen et al., 2024).

---

References

• Multisource AI Scorecard Table for System Evaluation (Blasch et al., 2021)
• Evaluating Trustworthiness of AI-Enabled Decision Support Systems: Validation of the Multisource AI Scorecard Table (MAST) (Salehi et al., 2023)
• PADTHAI-MM: Principles-based Approach for Designing Trustworthy, Human-centered AI using MAST Methodology (Cohen et al., 2024)