TSG Mentor: Intelligent Guidance Assistant

• TSG Mentor is an AI-powered interactive assistant that transforms unstructured troubleshooting and academic guides into executable workflows through advanced parsing and decision support.
• It utilizes a modular pipeline architecture integrating data ingestion, semantic retrieval, and LLM-based synthesis to deliver context-sensitive instructions and real-time quality checks.
• Its implementation streamlines incident troubleshooting and educational inquiry by reducing manual review burdens and ensuring accuracy via user feedback loops and systematic evaluations.

A TSG Mentor is an intelligent, interactive assistant designed to guide users—whether on-call engineers or university students—through complex procedural knowledge captured in Troubleshooting Guides (TSGs) or university-related informational resources. TSG Mentor systems combine advanced information ingestion, natural language understanding, workflow parsing, and interactive decision support to improve the utility, accuracy, and accessibility of structured and unstructured guidance documents across domains (Shetty et al., 2022, Rahman et al., 6 Nov 2025).

1. Definition and Functional Scope

A TSG Mentor functions as an automated advisor capable of parsing, synthesizing, and delivering step-wise, context-sensitive guidance. In the engineering domain, it supports on-call engineers (OCEs) by transforming unstructured TSGs into executable workflows, highlighting potential quality issues, and delivering context-aware hints and safety checks (Shetty et al., 2022). In academic settings, a TSG Mentor may operate as an AI-powered chatbot, ingesting and semantically indexing sources such as course lists, university policies, and forum data to support student queries with precise, context-grounded responses (Rahman et al., 6 Nov 2025).

The operational roles of a TSG Mentor include:

• Automatic parsing and structuring of multifaceted procedural guides.
• Real-time validation and quality issue flagging (e.g., broken links, outdated steps).
• Support for human-in-the-loop verification, parameter suggestion, and error mitigation.
• Interactive feedback incorporation and continuous adaptation to evolving knowledge bases.

2. System Architecture and Data Processing Pipeline

TSG Mentor systems integrate modular data ingestion, information retrieval, component classification, and LLM or workflow synthesis engines. System design typically follows a pipeline architecture, enabling both efficiency and incremental updating.

Data Ingestion

For university guidance, ingestion modules support CSVs (e.g., course schedules), university web content, and community forums. Key stages include:

• Schema definition and mapping to relational tables (e.g., SQLite).
• Incremental update processing driven by timestamps and hash-based change detection.
• Document chunking (RecursiveCharacterTextSplitter) and summarization for long-form content.
• Semantic embedding generation (e.g., all-MiniLM-L6-v2) for vector store population (Chroma DB).
• Persistent mapping between relational data and vector store entries for hybrid retrieval.

In incident troubleshooting, TSG Mentor workflows parse markdown-based TSGs into typed statements via component identification (meta-learning classifiers) and further extract actionable parameters or conditions via program synthesis (Shetty et al., 2022).

Efficiency

Incremental ingestion reduces computational complexity from O(N) to O(ΔN), where ΔN is the set of new or updated records. For a typical university setup, full ingestion may require ≈369 seconds, while incremental updates complete in ≈107 seconds. Embedding initialization incurs negligible warm-start costs (≈4.4 seconds) (Rahman et al., 6 Nov 2025).

3. Retrieval, Parsing, and Hybrid Fusion Techniques

Document Retrieval and Ranking

TSG Mentor systems employ hybrid retrieval to maximize answer relevance:

• BM25-based lexical search scores, with standard parameterization ($k_1\approx1.5$, $b\approx0.75$).
• Semantic nearest-neighbor retrieval using sentence embeddings (e.g., 384-dim all-MiniLM-L6-v2 vectors; retrieval via cosine or L2 similarity in FAISS/Chroma DB).
• Score normalization and fusion:

$$\text{Combined}(D, Q) = \lambda \cdot \mathrm{BM25}_n(D, Q) + (1-\lambda)\cdot \mathrm{Sim}_n(D, Q)$$

where $\lambda$ is a tunable blending factor.

• De-duplication ensures the fusion considers only unique, high-scoring documents.

Component Identification and Workflow Synthesis

In troubleshooting contexts, each TSG statement is classified as a component (e.g., Kusto query, PowerShell command) via a meta-learned SiameseNet architecture. Each component is parsed into constituents by program synthesis—typically Programming-by-Example (PBE) with operators such as Regex.Find and deterministic, human-editable output (Shetty et al., 2022). Handling for natural-language conditionals and multiline parameters is included.

Evaluation

• Component identification: accuracy ≈ 0.89, macro-F1 = 0.87.
• Component parsing: precision = 0.94, recall = 0.91.
• Coverage: 21.2% of TSG lines automatable and correctly parsed across most-used guides.

4. LLM Integration, Prompting, and Interactive Support

University-oriented TSG Mentor instances leverage LLMs (e.g., LLaMA-3.3-70B), integrating retrieval-augmented generation:

• Prompt templates specifying the mentor persona emphasizing factuality, citation, and stylistic constraints (e.g., ≤150 words, professional/empathetic tone).
• Structured context input: system prompt, retrieved snippets with provenance, user query, and dialog history (sliding window mechanism).
• Hyperparameters: temperature = 0.2, max tokens = 512, top-p = 0.9; window pruning prioritizes context documents and latest exchanges.

Mentoring Interface

For troubleshooting automation, the synthesized workflow is rendered as executable notebook cells, with affordances for form-based parameter editing, inline preview/logs, and safety checks (e.g., escalation guardrails, dry-run mode). TSG Mentor interaction includes accept/reject/edit actions at each procedural step, fostering collaborative human-AI troubleshooting (Shetty et al., 2022).

5. Quality Monitoring, Feedback Loops, and Governance

Quality assurance in TSG Mentor systems is anchored by automated detection of common guide defects, as empirically observed in real-world TSG corpora:

Quality Issue	Frequency (%)
Completeness	32.2
Broken Link	13.3
Correctness	11.2
Readability	10.3
User Experience (UX)	10.1
Empty	7.2
Up-to-dateness	6.5
Relevance	4.0
Other	5.2

TSG Mentor frameworks prompt authors or OCEs for corrective action upon detection of these issues (Shetty et al., 2022). A multi-tiered feedback loop captures user edits and action logs, informing incremental retraining of the parsing engine and flagging new or ambiguous guide components. Step-level user feedback (e.g., thumbs-up/down) is aggregated and presented on dashboards for continuous quality improvement and auditing.

Policy enforcement and safety modes (e.g., approval gating for sensitive operations, mandatory dry-runs) are standard.

6. Evaluation Metrics and Empirical Results

• University chatbot responses are evaluated using BERTScore (precision-max, 0.831 achieved) and METEOR (0.809), indicating strong semantic and paraphrastic alignment with reference answers (Rahman et al., 6 Nov 2025).
• TSG automation is measured through component identification accuracy (0.89), parsing precision (0.94), recall (0.91), and overall automation coverage (≈21% of lines fully automatable) (Shetty et al., 2022).
• For interactive guidance workflows, the mentor achieves high-confidence automation (over 90% accuracy for identified and parsed steps) and systematically reduces the manual review burden.
• System efficiency metrics include ingestion latency, update speed, retrieval/inference latency, and resource usage, monitored continuously.

7. Design, Deployment, and Extension Guidelines

Deployment of TSG Mentor solutions at scale follows a phased roadmap:

1. Discovery and Data Collection: Catalog relevant structured and unstructured sources; estimate data volume and update rates.
2. Pipeline Construction and Testing: Implement data schema, ingestion logic, chunking, embedding, and vector store modules; validate updates in a staging environment.
3. Retrieval Engine and LLM Integration: Integrate BM25 and Chroma DB, refine the hybrid query API, deploy the LLM, and develop user interfaces.
4. Evaluation and Tuning: Benchmark on realistic queries, adjust fusion weights, prompt phrasing, and context management.
5. User Feedback/Pilot: Pilot with limited cohorts, collect usability and satisfaction analytics, and iterate on knowledge base and system parameters.
6. Production Roll-out and Monitoring: Implement high-availability, automate ingestion/model updates, monitor key metrics, and schedule periodic content and system reviews (Rahman et al., 6 Nov 2025).

Continuous extendibility is enabled by onboarding new component types via minimal labeled examples and supporting community-driven PBE-spec libraries. Integration with ticketing and incident management systems is recommended for enterprise troubleshooting deployments.

---

The TSG Mentor framework, implemented across both engineering and academic guidance domains, synthesizes high-accuracy parsing, hybrid information retrieval, and interactive mentoring to elevate procedural knowledge utility, minimize errors, and enable scalable, adaptive support (Shetty et al., 2022, Rahman et al., 6 Nov 2025).