Ans-Chat: Unsupervised Answer Identification

• Ans-Chat is an unsupervised method for answer identification in organizational group chat that clusters question–candidate pairs using Kernel Density Estimation and a conservative seeding strategy.
• It models recurrent discussion patterns by emphasizing structural features like message timing, mention behavior, and interaction geometry over pure textual similarity.
• Empirical results on Slack channels show that combining textual and structural features substantially improves answer retrieval performance in noisy, asynchronous chat environments.

Searching arXiv for the primary paper and closely related chat/answer-identification work to ground the article. Ans-Chat is an unsupervised method for answer identification in organizational group chat, introduced for settings such as Slack where multiple conversations unfold concurrently, replies may arrive asynchronously, and useful answers are difficult to locate either in real time or retrospectively. Rather than generating responses, it operates over an existing chat stream: given messages already identified as questions, it predicts which subsequent messages are actual answers. Its central idea is to model recurrent discussion patterns in a channel by clustering question–candidate-answer pairs into answer and non-answer classes with Kernel Density Estimation (KDE) and a conservative seeding strategy, thereby avoiding channel-specific manual tagging and reducing dependence on supervised training data (Tepper et al., 2020).

1. Problem setting and communicative context

Ans-Chat addresses organizational group chat rather than forums, email, or one-to-one messaging. In this environment, multiple discussions are intertwined, collaborators participate asynchronously across locations and time zones, and the “always-on” nature of the medium creates information overload. A message that is close to a question in time may belong to another discussion entirely, so answer identification is not reducible to simple response detection (Tepper et al., 2020).

The method is motivated by practical work needs. The paper explicitly frames answer identification as useful for alerts when a question gets answered, conversation summarization, catch-up support after absence, and Q&A database construction for future lookup and bots. It also argues that Slack threads do not fully solve the problem because threads are not widely used, require extra effort to open and manage, can hide content and discourage participation, and a reply may fit more than one thread (Tepper et al., 2020).

The paper distinguishes organizational group chat from discussion forums in several ways. Forums are described as more segregated, structurally clearer, and linguistically better formed, whereas group chat is brief, noisy, context-dependent, and densely interleaved. This makes forum-oriented answer-finding methods a poor fit. A plausible implication is that successful answer identification in chat must rely less on well-formed discourse structure and more on local interaction patterns.

2. Formalization and overall pipeline

Ans-Chat assumes that question messages have already been identified by external tools, specifically a combination of eAssistant and Watson Natural Language Classifier. For each detected question, it collects subsequent messages within a fixed window of $w=10$ as candidate answers, then predicts which question–message pairs are true answer relations (Tepper et al., 2020).

The clustered unit is not an individual message in isolation but a pair

$x = (\text{question}, \text{potential answer}),$

with the full instance set denoted by $X$. Each pair is assigned to one of two clusters, the answer cluster $A$ or the non-answer cluster $N$, with

$X = A \cup N.$

A question may have multiple answers in $A$ (Tepper et al., 2020).

The generic clustering formulation begins from a heuristic initializer $H$: $$A = \{x \in X \mid H(x) = \text{True}\}, \qquad N = X / A.$$ After learning cluster models $M_A$ and $x = (\text{question}, \text{potential answer}),$0, pairs are reassigned by comparing model scores: $x = (\text{question}, \text{potential answer}),$1 Ans-Chat instantiates this framework with KDE-based cluster modeling and conservative seed sets that remain fixed during iteration (Tepper et al., 2020).

Operationally, the pipeline is: question identification, candidate generation, feature extraction, seed initialization, iterative KDE-based clustering, and output of predicted answer messages for each question. Clustering runs for up to $x = (\text{question}, \text{potential answer}),$2 iterations or until fewer than 10 pairs switch clusters (Tepper et al., 2020).

3. KDE-based clustering and conservative seeding

The paper’s main methodological choice is KDE as the cluster model. Rather than representing a cluster by a centroid, KDE uses all examples in the cluster and estimates a smooth density: $x = (\text{question}, \text{potential answer}),$3 where $x = (\text{question}, \text{potential answer}),$4 is a cluster, $x = (\text{question}, \text{potential answer}),$5 is the kernel, and $x = (\text{question}, \text{potential answer}),$6 is the smoothing bandwidth. In Ans-Chat, these density estimates serve as the answer and non-answer models, $x = (\text{question}, \text{potential answer}),$7 and $x = (\text{question}, \text{potential answer}),$8 (Tepper et al., 2020).

The final reassignment rule applies only to unseeded pairs: $x = (\text{question}, \text{potential answer}),$9 This conservative variant is intended to improve robustness by preventing high-confidence seed examples from bouncing between clusters (Tepper et al., 2020).

The answer seed cluster $X$0 contains pairs where the asker mentioned the answerer, or where the asker later acknowledged the answerer with expressions such as “ok” or “yes.” The non-answer seed cluster $X$1 contains pairs where question and candidate answer come from the same author, or where the time between them is improbable for an answer. The paper defines improbable time experimentally as less than one second or more than 10 hours (Tepper et al., 2020).

This design encodes a specific view of organizational chat: answerhood is partly a structural phenomenon. The method is therefore not primarily a semantic QA model, but a clustering system over conversational regularities such as mention behavior, plausible reply lag, and asker follow-up.

4. Features and representational choices

Ans-Chat uses both textual and structural features for each question–candidate pair. The paper emphasizes that rich text analysis performs poorly on sparse chat language, so the main textual signal highlighted in the core KDE comparisons is Jaccard similarity between question and potential answer, computed on stemmed tokens after stop-word removal. Cosine similarity is also mentioned in the broader feature inventory, but the key experiments emphasize Jaccard (Tepper et al., 2020).

The selected structural features are: distance in number of messages between the question and the potential answer, distance in number of messages between the potential answer and the questioner’s next message, and whether the questioner mentions the replier in the question message. The larger feature inventory additionally includes time difference in seconds, mention patterns, asker acknowledgment such as “okay,” “thanks,” or thumbs up, answerer activity in the window, previous correspondence activity, text length, and text types identified by eAssistant such as promise, request, and actionable statements (Tepper et al., 2020).

The paper also experimented with a larger feature set together with SVD dimensionality reduction. That larger representation reduced performance, which the authors attribute to complexity and noise. This suggests that, in this task, a small number of structurally interpretable features may be more effective than a broader but less discriminative representation (Tepper et al., 2020).

The implementation uses Python with sklearn for KDE and k-means, nltk for text processing and language/translation models, and gensim for word2vec. For KDE, the bandwidth is selected over 10 bandwidth points using 3-fold cross validation (Tepper et al., 2020).

5. Evaluation and empirical profile

The evaluation uses two Slack channels from work groups in the authors’ organization, spanning one year and over 20,000 total messages. Channel 1 contains 8,345 messages and 1,105 questions, reported as 13%. Channel 2 contains 15,052 messages and 2,854 questions, reported as 5%, though the paper notes this percentage is internally inconsistent as printed (Tepper et al., 2020).

For each channel, 1,000 messages were tagged, yielding about 300 identified questions and their answers per channel. Each message was annotated by three taggers. Fleiss’ kappa was 0.741 on channel 1 and 0.758 on channel 2, and majority vote was used for evaluation (Tepper et al., 2020).

Against the unsupervised discussion-forum baseline of Deepak and Visweswariah, KDE with text only already improved F-score. On channel 1, the best forum-oriented baseline reached $X$2, whereas KDE with Jaccard text reached $X$3. On channel 2, the best forum-oriented baseline reached $X$4, whereas KDE with Jaccard text reached $X$5 (Tepper et al., 2020).

Conservative clustering further improved results. With text only, regular KDE clustering achieved $X$6, $X$7, $X$8 on channel 1 and $X$9, $A$0, $A$1 on channel 2. Conservative KDE clustering raised this to $A$2, $A$3, $A$4 on channel 1 and $A$5, $A$6, $A$7 on channel 2 (Tepper et al., 2020).

The strongest results came from combining text and structure under the full Ans-Chat setting. Text only gave $A$8 and $A$9. Structure only gave $N$0 and $N$1. Text plus structure yielded the best scores: $N$2, $N$3, $N$4 on one channel and $N$5, $N$6, $N$7 on the other, for an average around $N$8 (Tepper et al., 2020).

These results support the paper’s main empirical claim: structural conversational signals matter much more than pure text in organizational group chat. The marked jump from text-only to text-plus-structure, especially on channel 2, indicates that answer relations are better captured by local interaction geometry than by lexical overlap alone.

6. Significance, limitations, and relation to later answer-oriented chat systems

Ans-Chat is significant as an early answer-identification system tailored to the realities of organizational group chat rather than forums or standalone QA. Its main conceptual move is to treat answerhood as an emergent clustering problem over conversational structure. This differs from retrieval-style answer ranking systems that score question–answer semantic compatibility directly, such as the deep similarity neural network approach that embeds questions and answers, then selects the answer with the highest learned similarity score (Minaee et al., 2017).

The paper also has clear limitations. It depends on an upstream question-identification step, so false positives and false negatives in question detection can degrade downstream answer identification. It uses a fixed local window of $N$9, which can miss long-range answers. The tagged dataset is small, with 1,000 tagged messages and about 300 questions-and-answers per channel, and the paper explicitly notes that this constrained parameter exploration and validation. It also does not provide a detailed per-case error analysis beyond discussion of question-identification mistakes (Tepper et al., 2020).

Later research suggests several ways the broader “answer-oriented chat” design space expanded beyond Ans-Chat’s original scope. Human-in-the-loop advisory systems for live-chat operators learned to recommend questions to ask, resolution content, and useful information from structured conversation state rather than clustering answer relations in raw chat (Aviv et al., 2021). Explainable answer-assessment platforms such as AERA Chat combined answer scoring with natural-language rationales, visualization, follow-up chat, and annotation workflows, making the assessment decision inspectable and revisable (Li et al., 2024). Sensor-grounded QA systems such as SensorChat moved answer generation into non-textual domains by decomposing natural-language questions, querying long-term multimodal sensor histories, and assembling grounded answers for both quantitative and qualitative questions (Yu et al., 5 Feb 2025).

This suggests a broader historical interpretation. Ans-Chat occupies an early position in a lineage of systems centered on locating, ranking, explaining, or assembling answers inside interactive environments. Its distinct contribution within that lineage is narrow but durable: it showed that in multi-party organizational chat, answer identification can be modeled effectively through unsupervised estimation of channel-specific interaction patterns, and that structural features such as mentions, timing, and asker follow-up can dominate lexical similarity for this task (Tepper et al., 2020).