Accessibility Scout: AI-Driven Auditing

• Accessibility Scout is an AI-driven system that tailors accessibility assessments by integrating detailed user models with web-based photo analysis.
• It employs large language models and semantic segmentation to decompose tasks and generate personalized barrier lists surpassing generic ADA audits.
• User feedback loops and dynamic model updates ensure continually refined, context-aware evaluations that are both scalable and cost-efficient.

Accessibility Scout is an AI-driven system for generating personalized accessibility assessments of built environments, leveraging LLMs to adapt evaluation results to the unique mobility requirements, preferences, and environmental interests of individual users. Unlike traditional checklist-based approaches, such as those grounded in Americans with Disabilities Act (ADA) compliance auditing, Accessibility Scout scales via web-based photo analysis and collaborative human-AI model refinement, yielding nuanced, context-aware assessments that evolve with user feedback (Huang et al., 31 Jul 2025).

1. System Architecture and Workflow

Accessibility Scout employs a hybrid user- and machine-learning model comprising several key components:

1. User Model Construction and Update:
   • A structured user model (JSON) encodes the individual's mobility limitations, reach, device use, sensitivity, preferences, and salient value systems.
   • Model initialization via freeform natural language input, user-driven annotated concern feedback, and incremental refinement from ongoing system interactions.
   • Updates occur in real-time as users supply clarifications or new accessibility-related attributes.
2. Visual Environment Assessment:
   • Input: Scene photograph and user-stated intent.
   • Semantic segmentation (e.g., Semantic-SAM) detects environmental objects and regions.
   • The LLM decomposes intent into canonical tasks and subtasks following hierarchical task modeling best practices (e.g., “wash hands” → “enter”, “approach sink”, “find soap”, etc.).
3. Personalized Concern Generation:
   • For each scene-task pair, system prompts LLM with environment and user model, requesting possible barriers relevant to that individual's profile.
   • Output: Custom “concern” objects (with textual rationale and image mask) indicating potential impediments (e.g., unreachable dispenser, narrow doorway, excessive volume/noise).
4. Feedback Integration:
   • Users interactively validate, edit, or supplement concern set.
   • Modifications are collapsed via semantic similarity clustering (utilizing Sentence-BERT cosine similarity, threshold 0.7).
   • All feedback recalibrates the user model, refining future predictions for both the individual and similar demographic groups.
5. Scalability and Cost:
   • Average processing cost is $0.021/image (ChatGPT-4o, as evaluated); throughput approaches 3,553 images/minute via parallel LLM requests.
   • Entire assessment pipeline orchestrates human-AI collaboration for both model construction and concern validation.

2. Personalization, Human-in-the-Loop Adaptation, and Concern Diversity

Accessibility Scout directly models personalization through dynamic user model updating based on both explicit self-description and iterative human feedback. Algorithms for concern generation combine user modeling, scene segmentation, intent decomposition, and environment context:

• The LLM-generated concern lists are filtered and merged using text embedding similarity, producing clusters of semantically equivalent or closely related concerns.
• Wasserstein distance quantifies divergence in concern distributions between different user models.
• Personalized output proves both quantitatively and qualitatively distinct: users with differing mobility levels or sensitivities receive barrier lists tailored to their context, not reducible to generic standards.

A critical feature is transparent explainability: users can inspect not only what is flagged but why—and override or supplement any concern as desired, supporting trust and continual adaptation.

3. Evaluation: Technical Studies and User-Centered Assessment

Accessibility Scout was evaluated through three complementary studies:

• Formative Study: Six regular wheelchair users revealed that existing static or ADA-based audit tools insufficiently address individual needs, often deterring exploration of novel spaces. Simple LLM personalization markedly improves concern relevance.
• Technical Evaluation: On 500 diverse images spanning U.S. environments with 11 user models, system hallucination rates were low (6.63%). Concern clustering demonstrated both ADA-like and beyond-ADA discrimination, tackling environmental noise, seating patterns, and visual clutter in addition to codified physical criteria.
• User Study: Ten individuals rated personalized concerns significantly more useful than generic ones (mean 4.68/7 vs 3.35/7; p<.001). Content of flagging (i.e., what is detected) outweighed text personalization, but user trust was enhanced via collaborative feedback loops. Demand for both overview and detailed modes surfaced, as did needs for group-based as well as individual-based profile adaptation.

4. Comparison to Traditional Accessibility Audit Methods

Traditional ADA/Checklist Audit	Accessibility Scout (LLM-based)
Generic, one-size-fits-all	Individualized (mobility, sensor, preference)
Binary threshold (pass/fail)	Nuanced, multi-level concern gradation
Labor-intensive/manual	Automated, scalable (internet-scale images)
Visible static features only	Reasoning over context, task, inferred issues
Static, non-adaptive	Dynamic, updated by user feedback

Accessibility Scout demonstrates the ability to “extend beyond traditional ADA considerations,” flagging barriers such as noise, visual distraction, and task context critical for real-world independent access.

5. Technical Algorithms and Quantitative Methods

• LLM-based Task Decomposition:
  • $$\text{LLM}(\text{Image Description}, \text{User Intent}) \to \{ \text{Task}_i: [\text{subtasks, primitive actions, locations}] \}$$
• Concern Clustering:
  • Cosine similarity in embedding space: $$\text{sim}(x, y) = \frac{\vec{x} \cdot \vec{y}}{\|\vec{x}\| \|\vec{y}\|}$$
• Distributional Analysis:
  • Wasserstein distance for concern cluster comparison: $$W(p, q) = \inf_{\gamma \in \Gamma(p, q)} \int |x-y| d\gamma(x, y)$$
• Scalability: LLM requests parallelized, supporting thousands of environments per minute on commodity hardware.
• Cost efficiency: $0.021$ USD per image at current commercial API pricing.

6. Implications, Limitations, and Future Directions

The Accessibility Scout system enables scalable, personalized, human-centric accessibility auditing—a substantial advancement over static checklist paradigms. Its flexibility supports individual, group, and dynamic adaptation, integrating both physical and “soft” barriers (e.g., environmental noise). A plausible implication is that large-scale accessibility databases could be updated dynamically for entire cities or venues based on crowd-sourced user models, supporting research, policy, and advocacy at unprecedented specificity.

Key limitations include:

• Exclusively visual input (for now); future work could include multimodal data (floor plans, occupancy, temporality).
• Possibility of LLM hallucination, mostly mitigated via human review and explainability.
• User model currently boxed as a static JSON; future directions include vector databases, multi-modal fusion, and continuous retrieval-augmented generation.
• Needs for contextual and group/family-level modeling remain.
• Ultimate accuracy and acceptability depend on continual stakeholder engagement and careful privacy engineering.

7. Conclusion

Accessibility Scout demonstrates that LLMs, when governed by collaborative human-AI feedback and robust user modeling, can yield accessibility assessments tailored to individual needs at scale. This approach provides nuanced, adaptable, and highly contextual flagging of barriers, departing from generic static standards and advancing accessibility as a dynamic, lived-process. Scalability, cost-efficiency, and user-centricity establish the system as a future direction for physical accessibility evaluation and automated environmental design (Huang et al., 31 Jul 2025).