ECT-QA: Assessing Subjective QA Quality

• ECT-QA is a framework for evaluating QA responses by measuring six subjective dimensions such as clarity, specificity, and tone in communication.
• It benchmarks models using annotated earnings call transcripts and employs weighted F1 metrics to capture pragmatic and factual quality.
• The methodology is transferable beyond finance, highlighting the importance of assessing soft misinformation in varied high-stakes domains.

ECT-QA refers to a family of methodologies and benchmarks for fine-grained evaluation of Quality Assessment (QA) tasks, with "ECT" standing for a domain-specific or feature-specific evaluation focus. In the context of recent research, particularly "SubjECTive-QA: Measuring Subjectivity in Earnings Call Transcripts' QA Through Six-Dimensional Feature Analysis" (Pardawala et al., 2024), ECT-QA defines a rigorous, multidimensional framework for evaluating not only factual correctness but also the pragmatic, subjective qualities of QA responses in high-stakes, formal communication scenarios such as earnings call transcripts.

1. Conceptual Foundations and Motivation

ECT-QA emerges from the realization that conventional QA accuracy metrics cannot fully capture the characteristics that make an answer useful, transparent, or trustworthy in real-world settings—especially where answers may be factually correct yet uninformative, evasive, or ambiguously worded. In earnings calls and analogous domains, subjective answer quality is crucial because official communication often leverages framing, tone, and specificity as strategic devices.

Conventional fact-checking frameworks focus on identifying explicit factual inaccuracies. However, ECT-QA is constructed around "soft misinformation"—cases where answers, while factually true, fail to deliver clarity, specificity, or relevance, thus potentially misleading stakeholders through omission, vagueness, or obfuscation rather than outright falsehood (Pardawala et al., 2024).

2. Data Construction, Annotation Protocol, and Subjectivity Dimensions

The SubjECTive-QA benchmark (Pardawala et al., 2024) operationalizes ECT-QA by assembling a dataset of 2,747 long-form QA pairs extracted from 120 NYSE-listed company earnings call transcripts spanning 2007–2021. Each QA exchange is manually annotated along six orthogonal subjective quality dimensions:

1. Assertive: Certainty and confidence expressed about company outcomes.
2. Cautious: Use of risk-averse or hedged statements.
3. Optimistic: Positive outlook about future outcomes.
4. Specific: Provision of concrete, technical, or detailed responses.
5. Clear: Transparency and lack of convolution.
6. Relevant: Topical alignment and answer completeness with respect to the question.

Each QA is labeled on a 3-point ordinal scale for each dimension (2: positive demonstration, 1: neutral, 0: negative demonstration), with final labels determined via majority vote among three independent annotators. The overall annotation agreement rate is 48.94% full agreement, 45.18% partial, and 5.88% disagreement, with highest annotator consensus for "Clear" and "Relevant," and lower agreement for more subjective dimensions like "Assertive," "Cautious," and "Specific."

3. Benchmarking Models, Evaluation Metrics, and Empirical Results

ECT-QA is formalized algorithmically as a 3-class sequence classification problem for each feature. Evaluation employs weighted F1, with experimental baselines including BERT base (uncased), FinBERT-tone, RoBERTa-base, and strong LLMs (Llama-3-70b-Chat, Mixtral, GPT-4o).

Key empirical findings from (Pardawala et al., 2024):

• Easier Features: "Clear" (BERT base, 80.93% F1) and "Relevant" (Llama-3-70b-Chat, 82.75% F1) are more objective, with small performance variance across models (~2.17%).
• Harder Features: "Assertive" (RoBERTa-base, 49.10% F1), "Specific" (FinBERT best, score not numerically stated), "Cautious" (BERT base, 60.66% F1), and "Optimistic" (RoBERTa-base, 62.69% F1) demonstrate higher subjectivity and lower interrater reliability, with a broader performance delta (~10.01%).
• RoBERTa-base achieves the highest average weighted F1 (63.95%) across all six dimensions.
• On out-of-domain (White House press briefings/gaggles, 65 QA pairs): "Clear" (BERT base, 0.8415), "Assertive" (RoBERTa-base, 0.6947), "Specific" (FinBERT-tone, 0.6992), mean F1 across features: 65.97%. This demonstrates the transferability of the subjective QA evaluation framework beyond finance to political domains.

Model performance supports the claim that PLMs (including LLMs) can learn to recognize subjective answer features, but with varying generalizability and feature-specific strengths.

4. Analytical Methods, Metrics, and Feature Independence

The evaluation relies on weighted F1 and cross-entropy loss for classification. The paper presents a Pearson correlation analysis across features, reporting values in $[-0.08, 0.39]$, confirming that the six chosen features are largely independent and should be modeled separately.

Metrics summary:

Feature	Best Model	Weighted F1 (%)
Clear	BERT base	80.93
Relevant	Llama-3-70b-Chat	82.75
Assertive	RoBERTa-base	49.10
Cautious	BERT base	60.66
Optimistic	RoBERTa-base	62.69
Specific	FinBERT-tone	(best, n/a)

Agreement and feature independence statistics contextualize the limits of subjectivity measurement and support the validity of reporting per-dimension scores.

5. Limitations and Dataset Scope

ECT-QA as instantiated in SubjECTive-QA has several acknowledged constraints:

• Coverage is limited to U.S. NYSE companies (2007–2021).
• Annotator pool is relatively small and possibly homogeneous.
• Transcripts are text-only; prosodic features observable in speech (intonation, pitch, etc.) are absent.
• Subjectivity introduces higher uncertainty at the segment level; while annotation is rigorous, some features inherently resist complete objectification.
• The dataset is available under the CC BY 4.0 license, facilitating reproducibility and extension.

6. Generalization and Application Beyond Financial QA

ECT-QA's multidimensional framework for subjective answer evaluation is not limited to finance. Out-of-domain evaluations demonstrate its relevance to domains such as political communication, press briefings, public policy, and any formal setting where answer framing, tone, and hedging alter the communicative utility of information. The capability to detect "soft misinformation"—that is, answers that are technically valid yet pragmatically misleading—is generalizable to misinformation detection and regulatory transparency assessment.

The approach also underlines the importance of expanding evaluation targets in QA from answer correctness to holistic answer quality incorporating clarity, relevance, confidence, detail, hedging, and positive/negative framing.

7. Practical and Methodological Significance

ECT-QA, through the lens of SubjECTive-QA (Pardawala et al., 2024), operationalizes the shift in QA research from binary factual evaluation to comprehensive quality assessment that models human communicative expectations in critical domains. The framework supports nuanced benchmarking of PLMs and LLMs on their ability to provide not just accurate, but also high-quality and socially transparent responses. The methodological structure—feature selection, multi-rater ordinal annotation, per-feature F1 scoring, and transfer testing—serves as a foundation for broader subjective QA evaluations in both academic research and high-stakes industry applications.