EMS Performance Audit Reports

• Performance Audit Reports are systematic evaluations that integrate structured data processing with deep learning NER to assess EMS clinical protocol adherence.
• The modular system architecture combines structured field analysis and unstructured narrative processing to generate detailed compliance metrics and visual dashboards.
• Advanced evaluation metrics, with F1 scores around 0.98, demonstrate the effectiveness of automated audit systems in reducing manual review time and improving quality assurance.

A performance audit report in Emergency Medical Services (EMS) systematically evaluates adherence to clinical protocols, documents individual and system-level performance, and identifies deficiencies in care delivery. Traditionally reliant on manual chart review, recent advancements leverage automated systems that integrate structured data processing with deep learning-based Named Entity Recognition (NER) to generate comprehensive, high-resolution audit outputs. Key innovations target efficiency, scalability, and analytic reproducibility across tens of thousands of ambulance incident records (Han et al., 2020).

1. System Architecture and Workflow

Automated performance audit reporting in EMS is organized into modular subsystems that process both structured fields (e.g., chief complaint, vital signs) and unstructured free-text clinical narrative (e.g., paramedic notes). The core architecture comprises:

• Data Ingestion Layer: Aggregates structured clinical fields alongside unstructured free-text reports containing narratives of findings and interventions.
• NER Pipeline: Applies text preprocessing (lower-casing, punctuation stripping, whitespace tokenization) before sequence tagging using a deep neural BiLSTM-CRF model.
• Audit-Logic Engine: Assigns clinical scenarios using dictionary-based logic (e.g., detecting acute coronary syndrome if “chest pain” present and $\mathrm{SBP} \geq 90$ mmHg), matches scenarios to protocol-defined required actions, and compares "performed" versus "expected" actions.
• Report Generator & Database: Collates binary flags at the case level (each protocol action), aggregates provider-level and system-wide metrics, and stores results in an SQL-style database for tabular and dashboard visualization.

The linear workflow for each incident involves:

1. Evaluating structured fields for Boolean triggers.
2. NER extraction of actions/entities from free text.
3. Consolidation of "performed actions."
4. Application of protocol rules to yield a vector of pass/fail (binary) audit flags.
5. Statistical postprocessing for visualization and quality improvement (Han et al., 2020).

2. Named Entity Recognition Model: Formalism and Data Labeling

The central component for unstructured data processing is a BiLSTM-CRF model, parameterized as follows:

• Embedding and Sequence Modeling:

For tokens $w_1,\ldots,w_n$, embeddings $e_i \in \mathbb{R}^d$ are learned, followed by bidirectional LSTM encoding ($\overrightarrow{h_i}$, $\overleftarrow{h_i}$), producing $$h_i = [\overrightarrow{h_i};\overleftarrow{h_i}] \in \mathbb{R}^{2H}$$.

• Tag Prediction & CRF Decoding:

Emission probabilities $P_i \in \mathbb{R}^K$ for $K$ IOB2 tags (entity boundaries by type); CRF computes

$$\mathrm{score}(X, y) = \sum_{i=1}^n (A_{y_{i-1}, y_i} + P_i[y_i])$$

and

$$\log p(y|X) = \mathrm{score}(X, y) - \log \sum_{y'} \exp(\mathrm{score}(X, y'))$$

Labeling proceeds via a weakly supervised pipeline:

• Construction of synonym lexicons per entity (single/multi-token).
• Fuzzy string matching (length $\geq5$) and exact match (shorter tokens) for auto annotation of 44,211 narratives in IOB2 format.
• Manual correction (2.5% dev/test splits) by clinicians to produce gold-standard labels.

Recognized entities span 17 types (e.g., ECG, Stroke Assessment, Aspirin, Adrenaline) across clinical procedures, findings, and medications (Han et al., 2020).

3. Protocol Audit Logic and Report Generation

Audit logic determines, for each scenario $S$ (e.g., ACS, Stroke) and each action $A$ (e.g., Aspirin administration), which cases are eligible and whether required actions were performed/documented:

• $\mathrm{eligibility\_flag}(S, c) = 1$ if case $c$ meets scenario import criteria (structured/unstructured trigger)
• $\mathrm{performed\_flag}(A, c) = 1$ if action extracted from text by NER
• $$\mathrm{pass\_flag}(A, c) = \mathrm{eligibility\_flag}(S, c) \wedge \mathrm{performed\_flag}(A, c)$$

These flags are assembled into:

• Case-level tables: List all actions for case $c$ with pass/fail.
• Provider-level metrics:

$$P_p(A) = \frac{\sum_{c \in \mathrm{cases}(p)} \mathrm{pass\_flag}(A, c)}{\sum_{c \in \mathrm{cases}(p)} \mathrm{eligibility\_flag}(S, c)} \times 100\%$$

• System-level summaries: Bar charts of adherence by action/scenario, monthly compliance time series.

A standard report comprises a summary ("4,200 ACS cases met eligibility; aspirin was documented in 98%") and action-level tables as shown below.

Action	Eligible cases	Adherence (%)
12-lead ECG	4,200	98.0
Sublingual Nitroglycerin	2,900	95.0

Provider dashboards display adherence rankings and case drill-downs (Han et al., 2020).

4. Evaluation Metrics and Model Performance

System evaluation employs token-level classification metrics (weighted over non-O classes) and entity-level MUC-5/SemEval’13 metrics. Core formulae:

$$P = \frac{TP}{TP+FP}, \quad R = \frac{TP}{TP+FN}, \quad F_1 = 2\,\frac{PR}{P+R}$$

On held-out data (2.5% test set):

• Entity type matching: F₁ ≈ 0.981 (BiLSTM-CRF), BERT-base ≈ 0.981, ClinicalBERT ≈ 0.982
• Strict span + type: BiLSTM-CRF F₁ ≈ 0.976

Model efficiency:

• BiLSTM-CRF: 3.8M parameters, 40MB, mean inference time ≈ 7.5ms/sentence (CPU)
• BERT-base: 109.5M, 274MB, ≈ 16.2ms
• ClinicalBERT: 108.3M, 286MB, ≈ 25.6ms

BiLSTM-CRF is selected for audit deployment based on superior compactness and inference speed (1–2 orders of magnitude faster than BERT-based alternatives) (Han et al., 2020).

5. Impact on Clinical Audit Efficiency and Quality

Manual EMS chart review incurs ≈2 minutes per case (e.g., 10,000 cases = 333 hours auditor labor). The automated system processes tens of thousands of records in minutes with $<10$ ms per record.

Qualitative improvements include:

• Enabling near-complete audit coverage, not limited sampled subsets.
• Reducing auditor fatigue and inter-rater variability.
• Accelerating loops for remediation and retraining.

Integration is achieved through:

• A Flask-based web application (NEREMSR) delivering immediate NER highlights.
• Back-end batch jobs that update SQL audit tables and drive key performance indicator dashboards nightly.
• Planned future deployment embedded in live ambulance-reporting for real-time alerts (Han et al., 2020).

6. Context, Limitations, and Future Directions

By combining a weakly supervised, deep-learning NER with deterministic audit logic, the described system transforms unstructured paramedic narratives into structured compliance datasets without human bottlenecks. Limitations are not explicitly detailed in the data, but the scope is currently constrained to entity types for which robust lexicons and protocol logic exist. Future plans include full embedding into real-time EMS workflows and extension of audit entities and logic as underlying clinical protocols evolve (Han et al., 2020). This suggests scalability across clinical domains where protocol adherence must be robustly assessed from heterogeneous record sources.