ChatIYP: NL Interface for Internet Yellow Pages

• ChatIYP is a natural-language interface to the Internet Yellow Pages, enabling plain-English queries over a unified routing and infrastructure graph.
• It uses a three-stage retrieval-augmented generation pipeline combining symbolic Cypher translation, vector-based search, and LLM reranking for context.
• The system returns both a natural-language answer and the executable query for transparency, though it struggles with complex multi-hop queries.

Searching arXiv for the cited paper and closely related work to ground the article in current literature. ChatIYP is a natural-language interface to the Internet Yellow Pages (IYP), a graph-based knowledge base that aggregates Internet routing and infrastructure information from multiple heterogeneous sources into a unified representation. Its stated purpose is to let users ask plain-English questions about Internet infrastructure and routing without requiring prior knowledge of Cypher or the exact IYP schema. The system is presented as a domain-specific Retrieval-Augmented Generation (RAG) pipeline that retrieves graph and textual context from IYP, generates an executable Cypher query, executes that query against the underlying graph, and returns both a natural-language answer and the corresponding query for transparency (Andritsoudis et al., 23 Sep 2025).

1. IYP as the underlying knowledge base

IYP aggregates Internet routing and infrastructure information from multiple heterogeneous sources, including BGP routing tables and WHOIS-like records, into a unified graph-based knowledge base (Andritsoudis et al., 23 Sep 2025). In that graph, Internet entities such as Autonomous Systems (ASes), IP prefixes or blocks, and countries are represented as nodes, while their inter-entity relations are represented as edges. This graph organization supports expressive multi-hop questions over interconnected Internet entities.

The practical obstacle addressed by ChatIYP is that IYP is not directly accessible to non-experts. Querying it requires at least three forms of expertise: Cypher syntax, knowledge of the graph schema, and familiarity with the exact labels and properties used in IYP. The poster illustrates this barrier with the question “What is the percentage of Japan’s population in AS2497?”, which must be written as:

MATCH (:AS {asn:2497})-[p:POPULATION]-(:Country {country_code:'JP'}) RETURN p.percent

The significance of ChatIYP lies in turning this schema-bound graph querying problem into a natural-language interaction problem. In methodological terms, this shifts access to IYP from direct logical-form authoring toward a grounded, domain-specific retrieval-and-generation workflow. This suggests that the main challenge is not generic question answering, but faithful mediation between natural language and a graph knowledge base whose utility depends on schema fidelity and executable graph traversal (Andritsoudis et al., 23 Sep 2025).

2. Retrieval architecture and schema grounding

ChatIYP is organized as a compact three-stage pipeline: user query, retrieval, and generation (Andritsoudis et al., 23 Sep 2025). A user submits a natural-language question through a web interface; the system then performs retrieval using three complementary mechanisms; finally, the generation stage produces both the user-facing answer and a refined Cypher query.

The retrieval stage is the core architectural contribution. Rather than depending on a single retriever, ChatIYP combines graph-query generation, vector-based semantic retrieval, and reranking. The first component, the TextToCypherRetriever, uses a LLM to map the natural-language question into a Cypher query. The authors describe the prompt chain as “fine-tuned on IYP query patterns.” The generated Cypher is executed against the Neo4j graph database backing IYP, and the result is returned as a structured graph response. This is the symbolic retrieval path and is the mechanism that most directly exploits the graph schema.

The second component, the VectorContextRetriever, serves as a fallback or context-expansion mechanism when structured querying fails or returns sparse results. It retrieves textual context from nearby graph nodes by dense-vector similarity over node descriptions. According to the poster, this is useful for vague questions, incomplete terminology, and cases in which graph structure alone does not recover the intended meaning.

The third component, the LLMReranker, reranks candidate contexts returned by the prior stages before they are passed to generation. The rationale given is robustness: symbolic retrieval contributes precision and schema-aware access, semantic retrieval improves recall and ambiguity tolerance, and reranking improves context selection quality before answer synthesis (Andritsoudis et al., 23 Sep 2025).

The paper does not provide pseudocode, retrieval scoring equations, or a separately formalized schema-linking module of the sort often discussed in text-to-SQL systems. Schema awareness is instead handled implicitly through prompt engineering for text-to-Cypher translation and through tight coupling to Neo4j and the IYP schema. A reasonable reading is that ChatIYP operationalizes schema grounding through domain-adapted prompting rather than through an explicitly exposed symbolic linker.

3. Generation, implementation, and transparency

Once retrieval has produced graph or textual context, the generation stage passes the original user input together with the retrieved context to an LLM that generates both a natural-language response and a refined Cypher query (Andritsoudis et al., 23 Sep 2025). In the implementation used for experiments, the backbone model is GPT-3.5-Turbo. This stage therefore performs two distinct functions: answer synthesis and query refinement.

A distinctive design choice is that ChatIYP returns the answer together with the underlying query. This “answer plus underlying query” presentation is framed as a transparency mechanism, allowing technically sophisticated users to inspect what was actually executed against the graph. Within the scope of the poster, this inspectability is one of the principal usability features of the system.

Implementation-wise, the system is built with LlamaIndex and Neo4j (Andritsoudis et al., 23 Sep 2025). LlamaIndex is used because it supports both symbolic and semantic retrieval over graph-structured data and integrates with Neo4j, while Neo4j stores and queries the IYP graph itself. The poster also mentions a web interface, though front-end details are not elaborated.

The paper explicitly does not describe multi-agent decomposition, iterative self-correction loops, tool-calling traces, or explicit chain-of-thought procedures beyond the retrieval-plus-generation pipeline. In that sense, ChatIYP is comparatively narrow and task-specific: it is not positioned as a general conversational agent, but as a graph-grounded interface specialized for IYP querying (Andritsoudis et al., 23 Sep 2025).

4. Evaluation design and metric selection

ChatIYP is evaluated on CypherEval, described as a benchmark containing more than 300 natural-language questions over IYP, each paired with a gold Cypher query and labeled by difficulty as Easy, Medium, or Hard (Andritsoudis et al., 23 Sep 2025). The examples span both general and technical domains, which makes the benchmark suitable for a system whose task is natural-language access to a graph knowledge base rather than free-form document QA.

The evaluation protocol is answer-centric rather than exact-query-match-centric. A validation model executes the gold Cypher query on the IYP graph and then prompts GPT-3.5 to generate a reference natural-language answer. ChatIYP’s output is then compared against these references using BLEU, ROUGE, BERTScore, and G-Eval (Andritsoudis et al., 23 Sep 2025). This design reflects the fact that multiple distinct Cypher queries can be semantically equivalent, while the deployed system’s utility depends on answer correctness rather than exact string identity of the logical form.

The roles of the metrics are stated directly in the poster. BLEU measures n-gram precision between response and reference. ROUGE emphasizes n-gram or subsequence recall. BERTScore uses contextual embeddings from a pretrained LLM to assess semantic similarity. G-Eval uses GPT-4 as a judge to evaluate factuality, relevance, and informativeness (Andritsoudis et al., 23 Sep 2025).

A central methodological conclusion is that G-Eval is better suited than overlap-based metrics for IYP-querying agents. The authors argue that BLEU is brittle because semantically correct paraphrases may be heavily penalized, ROUGE correlates poorly with factual correctness, and BERTScore exhibits a ceiling effect because many IYP answers have narrow linguistic variability. By contrast, G-Eval shows a bimodal score distribution, which the authors interpret as clearer separation between good and bad responses and better alignment with human judgment; accordingly, they adopt G-Eval as the primary metric (Andritsoudis et al., 23 Sep 2025).

This emphasis on grounding and evaluator choice is consistent with broader IR research that treats retrieval-augmented systems as hybrids requiring evaluation beyond classic lexical overlap, especially when answers are synthesized rather than merely ranked (Huang et al., 2024).

5. Empirical behavior, strengths, and failure modes

The main empirical finding is that ChatIYP performs solidly on simple prompts but degrades as query complexity increases (Andritsoudis et al., 23 Sep 2025). On easy prompts, over half of responses score above 75% under G-Eval. Performance drops for more complex prompts, especially hard questions that require multi-hop reasoning. The paper interprets this trend as consistent with the structure of IYP itself: harder questions impose more demanding graph traversals, stronger schema fidelity requirements, and more reliable compositional reasoning across multiple relations.

The authors also report that there is no consistent performance gap between general and technical prompts. Their interpretation is that structural complexity matters more than domain-specific vocabulary. In other words, “technicality” in wording is not the principal challenge; graph reasoning complexity is.

The figures described in the poster reinforce these findings. One boxplot compares metric distributions and supports the claim that G-Eval offers better discrimination than BLEU, ROUGE, or BERTScore. Another boxplot presents G-Eval by difficulty category and supports the trend that easy queries are handled relatively well whereas medium and especially hard queries are more error-prone. Multi-hop reasoning is specifically identified as a major source of difficulty (Andritsoudis et al., 23 Sep 2025).

Several limitations are explicit or strongly implied. ChatIYP is not robust on hard queries, particularly those involving complex graph traversals or multi-step reasoning. The retrieval and generation process can fail when symbolic translation into Cypher is wrong, incomplete, or underspecified. Sparse symbolic retrieval is a practical issue, which is why the vector-based fallback exists; however, semantic context cannot fully substitute for accurate executable graph logic. The use of GPT-3.5-Turbo may also impose model-capacity limits affecting schema fidelity, compositional reasoning, and query correctness. In addition, the poster omits many implementation details, including prompt design, exact reranking method, embedding model choice, context-window management, fallback thresholds, and detailed error analysis (Andritsoudis et al., 23 Sep 2025).

Likely failure modes identified or implied by the paper include incorrect schema grounding, malformed or semantically incorrect Cypher generation, insufficient retrieval for ambiguous questions, and degraded behavior on multi-hop or compositional tasks. The poster’s discussion of evaluation further suggests another failure mode at the assessment layer: traditional text metrics may treat an incorrect answer as acceptable or penalize a correct paraphrase, motivating the strong preference for G-Eval (Andritsoudis et al., 23 Sep 2025).

6. Position within conversational retrieval research

ChatIYP belongs to a broader class of systems that combine retrieval, generation, and interaction rather than relying on a monolithic parametric model. Survey-style work on ChatGPT in information retrieval characterizes this design space as a shift from rigid, supervised, task-specific pipelines toward generative, instruction-following, retrieval-grounded assistants, while also emphasizing persistent issues of hallucination, source attribution, interpretability, and controllability (Huang et al., 2024). ChatIYP fits this pattern closely, but in a domain-specific form: instead of generic document retrieval, it is grounded in a graph knowledge base and a graph query language.

Its narrowness is also defining. ChatIYP does not attempt to replace retrieval with end-to-end generation; rather, it treats retrieval and executable graph access as the substrate of trustworthy answering. The system’s return of both answer and generated Cypher query places provenance and inspectability at the interface layer, which aligns with the broader IR literature’s emphasis on exposing evidence and supporting verification (Huang et al., 2024).

A useful adjacent comparison is the chat-based retrieval paradigm explored in image retrieval. ChatIR treats retrieval as a progressive clarification process in which the system asks follow-up questions, appends each question-answer pair to dialogue history, and updates retrieval after every round, yielding gains over single-shot retrieval (Levy et al., 2023). ChatIYP, by contrast, is not presented as a multi-turn clarification system: it operates as a retrieval-plus-generation pipeline over IYP and does not include an adaptive questioning policy (Andritsoudis et al., 23 Sep 2025). This suggests a possible extension rather than an existing feature. A plausible implication is that future IYP interfaces could benefit from candidate-aware clarification, especially when symbolic translation is ambiguous or when a query underspecifies the intended graph traversal.

The future directions stated or directly motivated by the poster are comparatively focused: improve performance on hard queries, especially multi-hop reasoning, and refine retrieval and generation quality (Andritsoudis et al., 23 Sep 2025). The text specifically motivates better text-to-Cypher prompting, stronger schema-aware reasoning, improved reranking, and possibly more explicit reasoning over graph paths. Within the larger conversational IR landscape, ChatIYP is therefore best understood not as a general chatbot, but as a specialized graph-grounded access layer for Internet routing knowledge whose principal research contributions are its hybrid retrieval design and its argument that LLM-as-a-judge evaluation is more appropriate than overlap-based metrics for this class of systems (Andritsoudis et al., 23 Sep 2025).