---
title: 'GraphCS: LLM Benchmark for Community Search'
url: https://www.emergentmind.com/topics/graphcs
type: topic
---

# GraphCS: LLM Benchmark for Community Search

GraphCS is a benchmark for evaluating Large Language Models on community-search tasks in graphs, introduced in conjunction with the CS-Agent framework for LLM-based community search [2508.09549]. It formalizes community search as the problem of producing, from a graph-query pair \((G,q)\), a connected subgraph \(H\) containing the query node and satisfying a specified cohesiveness criterion, with performance assessed against a ground-truth community \(C^*\). Within the reported formulation, GraphCS currently supports single-node queries, uses two synthetic graph families with graded difficulty, and evaluates four classic community-search metrics under multiple prompting regimes. The benchmark is presented as the first benchmark for evaluating LLMs on community-search tasks and, more broadly, as an initial bridge between LLMs and graph analysis [2508.09549].

## 1. Formal problem setting

GraphCS defines an undirected graph as \(G=(V,E)\), where \(V\) is the set of \(n\) vertices and \(E\subseteq V\times V\) is the edge set [2508.09549]. A community query \(q\in Q\) requests a subgraph \(H=(V_H,E_H)\subseteq G\) satisfying specified cohesiveness constraints, and the target community is denoted \(C^*\). In the current version of the benchmark, the supported query type is the single-node query \(q=v\), which asks for a community \(H\) containing vertex \(v\) [2508.09549].

The objective is constrained in two ways. First, \(H\) must satisfy connectivity: it is connected and \(q\in V_H\). Second, it must satisfy cohesiveness: it maximizes or meets a pre-specified metric, such as \(k\)-core or \(k\)-truss [2508.09549]. The benchmark therefore treats community search as a graph-to-subgraph mapping problem,
$$
\Phi_{\rm LLM}:(G,q)\longmapsto H,
$$
with evaluation based on the distance between \(\Phi_{\rm LLM}(G,q)\) and the gold community \(C^*\) [2508.09549].

This formulation is notable because it does not cast the task as unrestricted graph description or code generation. Instead, it requires a structurally valid subgraph aligned with a formal community notion. A plausible implication is that GraphCS probes whether an LLM can operationalize graph-theoretic constraints rather than merely recognize community-related language.

## 2. Datasets and graph construction

GraphCS comprises two synthetic datasets, each partitioned into three difficulty levels, \(\#\)Easy, \(\#\)Medium, and \(\#\)Hard, by node count [2508.09549]. The two datasets are PSG and LFR. No additional normalization or pruning is applied; all graphs are verbalized for LLM input via a “Node List + Adjacency List” textual format [2508.09549].

| Dataset | Size | Difficulty statistics |
|---|---:|---|
| PSG | 6 240 graphs | Easy: \(|V|\in[5,10]\), mean \(|E|\approx18.6\), mean \(|V_H|\approx6.1\); Medium: \(|V|\in[11,25]\), \(|E|\approx56.1\), \(|V_H|\approx8.4\); Hard: \(|V|\in[26,35]\), \(|E|\approx127.7\), \(|V_H|\approx10.2\) |
| LFR | 6 120 graphs | Easy: \(|V|\in[10,20]\), \(|E|\approx23.2\), \(|V_H|\approx5.5\); Medium: \(|V|\in[21,30]\), \(|E|\approx75.0\), \(|V_H|\approx9.8\); Hard: \(|V|\in[31,40]\), \(|E|\approx122.4\), \(|V_H|\approx11.9\) |

The PSG dataset, or Probabilistic Structural Graph dataset, is generated by assigning edges with probability \(p\), decomposed into two values: \(p_{\rm dense}=0.8\) within a randomly chosen core subset \(\mathcal S_{\rm dense}\), and \(p_{\rm sparse}=0.2\) elsewhere [2508.09549]. This produces planted dense communities of varying size. The construction is explicitly controlled and exposes the model to relatively clean structural signals.

The LFR dataset uses the Lancichinetti–Fortunato–Radicchi generator with degree-distribution exponent \(\tau_1=1.8\), community-size exponent \(\tau_2=1.2\), and mixing parameter \(\mu=0.1\) [2508.09549]. These settings yield realistic, power-law graphs with planted community structure. Relative to PSG, this introduces graph instances whose topology is intended to be closer to common benchmarks in graph mining.

The use of textual graph verbalization is central to GraphCS. Because all inputs are serialized as “Node List + Adjacency List,” the benchmark tests LLM behavior under a purely language-mediated representation of graph structure rather than direct tensor or combinatorial input.

## 3. Task taxonomy and benchmark protocol

GraphCS evaluates four classic cohesiveness metrics, divided into vertex-centric and edge-centric categories [2508.09549].

| Category | Task | Definition |
|---|---|---|
| Vertex-centric | \(k\)-Core | Largest subgraph in which every vertex has degree \(\ge k\) |
| Vertex-centric | \(k\)-Clique | A set of \(k\) vertices each pair of which shares an edge |
| Edge-centric | \(k\)-Truss | Largest subgraph where every edge lies in at least \(k-2\) triangles |
| Edge-centric | \(k\)-Edge-Connected Component (\(k\)-ECC) | Subgraph that remains connected after any \(k-1\) edge removals |

For each graph and each metric, the benchmark issues a single-node query for a random seed \(v\) and asks the model to return the subgraph \(H\) satisfying that metric [2508.09549]. This protocol standardizes the interaction across graph families and cohesiveness notions, allowing direct comparison of performance across different structural regimes.

Every LLM is tested under three prompting paradigms: ZERO-SHOT, FEW-SHOT with one in-context example, and ZERO-SHOT CHAIN-OF-THOUGHT (0-CoT) [2508.09549]. The reported models are ChatGPT (GPT-3.5-turbo), Gemini-2.0, Llama3-8B, and Mixtral-8x7B [2508.09549]. The benchmark is therefore not limited to a single architectural family or deployment style; it includes both closed-source and open-source systems.

The taxonomy is important because the four tasks do not stress the same combinatorial primitives. Vertex-centric tasks can often be validated through degree or adjacency inspection, whereas edge-centric tasks require reasoning over triangle support or edge connectivity. This distinction becomes consequential in the reported results.

## 4. Evaluation criteria

The primary quantitative metric in GraphCS is the F1-score between the predicted community \(H\) and the ground truth \(C^*\) [2508.09549]. Precision and Recall are defined as
$$
\mathrm{Precision}=\frac{|V_H\cap V_{C^*}|}{|V_H|}, \qquad
\mathrm{Recall}=\frac{|V_H\cap V_{C^*}|}{|V_{C^*}|},
$$
and the benchmark uses
$$
\mathrm{F1}=\frac{2\,\mathrm{Precision}\times\mathrm{Recall}}{\mathrm{Precision}+\mathrm{Recall}}.
$$

This choice evaluates overlap at the vertex-set level rather than requiring exact subgraph isomorphism or a task-specific structural penalty. The emphasis is therefore on whether the returned node set matches the target community.

In the CS framework associated with GraphCS, the Validator agent additionally emits a normalized score,
$$
\mathrm{Score}\in[0,5],
$$
to quantify structural soundness and compliance with the cohesiveness metric [2508.09549]. This score is not presented as a replacement for F1; rather, it functions as an internal quality signal within the agentic procedure.

A plausible implication is that GraphCS separates two evaluation layers: benchmark-level agreement with gold communities through F1, and framework-level assessment of candidate validity through the Validator’s normalized score.

## 5. Empirical performance patterns

GraphCS reports F1-scores, in percent, for all four LLMs on both datasets, across the four tasks and three difficulty levels, under ZERO-SHOT, FEW-SHOT, and 0-CoT prompting [2508.09549]. The concise summary given for ChatGPT identifies several benchmark-level regularities.

First, FEW-SHOT is the best prompting regime. On PSG \(k\)-Core, the reported ChatGPT F1-scores are Easy 83.8, Medium 77.0, and Hard 79.2 [2508.09549]. ZERO-SHOT is second, while 0-CoT is often the worst; one reported example is PSG \(k\)-ECC Hard at only 3.2% under 0-CoT [2508.09549].

Second, vertex-centric tasks, namely \(k\)-Core and \(k\)-Clique, yield higher F1 than edge-centric tasks, namely \(k\)-Truss and \(k\)-ECC [2508.09549]. This indicates a marked asymmetry in how current LLMs handle different forms of graph cohesiveness.

Third, performance degrades significantly as graph size or community complexity increases, and it also degrades on realistic LFR instances [2508.09549]. The LFR setting therefore appears to be more demanding than PSG, despite both containing planted community structure.

Fourth, GraphCS reports that LLMs can sometimes exceed \( \mathrm{F1}=90\% \) on easy vertex-centric tasks under FEW-SHOT prompting [2508.09549]. At the same time, the benchmark shows that this success does not transfer uniformly across tasks or difficulty levels.

The benchmark also identifies a specific failure mode termed output bias. Under ZERO-SHOT or 0-CoT prompting, models frequently stray from returning vertex sets and instead propose specific code snippets or irrelevant solutions [2508.09549]. This output bias can affect over 50% of cases with 0-CoT prompting [2508.09549]. In this setting, the error is not merely suboptimal community identification; it is a mismatch between the required output type and the generated response.

## 6. Interpretation, limitations, and relation to CS-Agent

GraphCS presents LLMs as having preliminary community-search ability but also substantial structural brittleness [2508.09549]. The paper’s interpretation is that LLMs struggle on edge-centric metrics and larger graphs, suggesting that they rely more on pattern-matching than true combinatorial reasoning [2508.09549]. This is an interpretive claim made within the benchmark analysis rather than a theorem established by the experiments.

Prompt sensitivity is another central conclusion. FEW-SHOT grounding drastically reduces bias and improves stability, whereas chain-of-thought reasoning in the 0-CoT setting is described as counterproductive [2508.09549]. This directly challenges the common assumption that additional verbal reasoning traces necessarily improve performance on structured graph tasks. Within GraphCS, the opposite pattern is reported.

The benchmark also records a model-family difference: ChatGPT and Gemini outperform Llama3 and Mixtral, indicating that model scale and pretraining data matter [2508.09549]. Because all models are evaluated on the same benchmark protocol, this comparison functions as an empirical observation about current LLM capability under a fixed text-based graph interface.

GraphCS is also the setting in which CS-Agent is introduced [2508.09549]. CS-Agent is a dual-agent collaborative framework in which two LLMs act as Solver and Validator; through iterative feedback and refinement, it dynamically refines initial results without fine-tuning or additional training, and after the multi-round dialogue a Decider module selects the optimal community [2508.09549]. Extensive experiments are reported to show that CS-Agent significantly improves the quality and stability of identified communities compared to baseline methods [2508.09549]. In this context, GraphCS serves not only as a benchmark but also as the evaluation environment for agentic mitigation of the benchmark’s observed failure modes.

Several limitations are explicit. GraphCS currently supports only single-node queries, with future versions planned to add multi-node or keyword-based queries [2508.09549]. Its datasets are synthetic, even when designed to be realistic, and its graph input modality is restricted to textual serialization. These design choices delimit the present scope of the benchmark. At the same time, they make the benchmark controlled, reproducible, and directly comparable across prompting paradigms and model families. This suggests that GraphCS is best understood as an initial experimental substrate for community-search evaluation in LLMs rather than as an exhaustive account of graph reasoning performance.

Source: https://www.emergentmind.com/topics/graphcs