GraphCS: LLM Benchmark for Community Search

• GraphCS is a benchmark for evaluating LLMs on community search tasks by mapping graph-query pairs to connected subgraphs that meet specific cohesiveness criteria.
• It uses two synthetic datasets, PSG and LFR, with graded difficulty levels and a textual graph format to assess both vertex-centric and edge-centric metrics.
• The benchmark evaluates performance across prompting paradigms, revealing insights into LLMs' structural reasoning, output bias, and sensitivity to task complexity.

GraphCS is a benchmark for evaluating LLMs on community-search tasks in graphs, introduced in conjunction with the CS-Agent framework for LLM-based community search (Hua et al., 13 Aug 2025). It formalizes community search as the problem of producing, from a graph-query pair $(G,q)$, a connected subgraph $H$ containing the query node and satisfying a specified cohesiveness criterion, with performance assessed against a ground-truth community $C^*$. Within the reported formulation, GraphCS currently supports single-node queries, uses two synthetic graph families with graded difficulty, and evaluates four classic community-search metrics under multiple prompting regimes. The benchmark is presented as the first benchmark for evaluating LLMs on community-search tasks and, more broadly, as an initial bridge between LLMs and graph analysis (Hua et al., 13 Aug 2025).

1. Formal problem setting

GraphCS defines an undirected graph as $G=(V,E)$, where $V$ is the set of $n$ vertices and $E\subseteq V\times V$ is the edge set (Hua et al., 13 Aug 2025). A community query $q\in Q$ requests a subgraph $H=(V_H,E_H)\subseteq G$ satisfying specified cohesiveness constraints, and the target community is denoted $C^*$. In the current version of the benchmark, the supported query type is the single-node query $H$0, which asks for a community $H$1 containing vertex $H$2 (Hua et al., 13 Aug 2025).

The objective is constrained in two ways. First, $H$3 must satisfy connectivity: it is connected and $H$4. Second, it must satisfy cohesiveness: it maximizes or meets a pre-specified metric, such as $H$5-core or $H$6-truss (Hua et al., 13 Aug 2025). The benchmark therefore treats community search as a graph-to-subgraph mapping problem,

$H$7

with evaluation based on the distance between $H$8 and the gold community $H$9 (Hua et al., 13 Aug 2025).

This formulation is notable because it does not cast the task as unrestricted graph description or code generation. Instead, it requires a structurally valid subgraph aligned with a formal community notion. A plausible implication is that GraphCS probes whether an LLM can operationalize graph-theoretic constraints rather than merely recognize community-related language.

2. Datasets and graph construction

GraphCS comprises two synthetic datasets, each partitioned into three difficulty levels, $C^*$0Easy, $C^*$1Medium, and $C^*$2Hard, by node count (Hua et al., 13 Aug 2025). The two datasets are PSG and LFR. No additional normalization or pruning is applied; all graphs are verbalized for LLM input via a “Node List + Adjacency List” textual format (Hua et al., 13 Aug 2025).

Dataset	Size	Difficulty statistics
PSG	6 240 graphs	Easy: $C^*$3, mean $C^*$4, mean $C^*$5; Medium: $C^*$6, $C^*$7, $C^*$8; Hard: $C^*$9, $G=(V,E)$0, $G=(V,E)$1
LFR	6 120 graphs	Easy: $G=(V,E)$2, $G=(V,E)$3, $G=(V,E)$4; Medium: $G=(V,E)$5, $G=(V,E)$6, $G=(V,E)$7; Hard: $G=(V,E)$8, $G=(V,E)$9, $V$0

The PSG dataset, or Probabilistic Structural Graph dataset, is generated by assigning edges with probability $V$1, decomposed into two values: $V$2 within a randomly chosen core subset $V$3, and $V$4 elsewhere (Hua et al., 13 Aug 2025). This produces planted dense communities of varying size. The construction is explicitly controlled and exposes the model to relatively clean structural signals.

The LFR dataset uses the Lancichinetti–Fortunato–Radicchi generator with degree-distribution exponent $V$5, community-size exponent $V$6, and mixing parameter $V$7 (Hua et al., 13 Aug 2025). These settings yield realistic, power-law graphs with planted community structure. Relative to PSG, this introduces graph instances whose topology is intended to be closer to common benchmarks in graph mining.

The use of textual graph verbalization is central to GraphCS. Because all inputs are serialized as “Node List + Adjacency List,” the benchmark tests LLM behavior under a purely language-mediated representation of graph structure rather than direct tensor or combinatorial input.

3. Task taxonomy and benchmark protocol

GraphCS evaluates four classic cohesiveness metrics, divided into vertex-centric and edge-centric categories (Hua et al., 13 Aug 2025).

Category	Task	Definition
Vertex-centric	$V$8-Core	Largest subgraph in which every vertex has degree $V$9
Vertex-centric	$n$0-Clique	A set of $n$1 vertices each pair of which shares an edge
Edge-centric	$n$2-Truss	Largest subgraph where every edge lies in at least $n$3 triangles
Edge-centric	$n$4-Edge-Connected Component ($n$5-ECC)	Subgraph that remains connected after any $n$6 edge removals

For each graph and each metric, the benchmark issues a single-node query for a random seed $n$7 and asks the model to return the subgraph $n$8 satisfying that metric (Hua et al., 13 Aug 2025). This protocol standardizes the interaction across graph families and cohesiveness notions, allowing direct comparison of performance across different structural regimes.

Every LLM is tested under three prompting paradigms: ZERO-SHOT, FEW-SHOT with one in-context example, and ZERO-SHOT CHAIN-OF-THOUGHT (0-CoT) (Hua et al., 13 Aug 2025). The reported models are ChatGPT (GPT-3.5-turbo), Gemini-2.0, Llama3-8B, and Mixtral-8x7B (Hua et al., 13 Aug 2025). The benchmark is therefore not limited to a single architectural family or deployment style; it includes both closed-source and open-source systems.

The taxonomy is important because the four tasks do not stress the same combinatorial primitives. Vertex-centric tasks can often be validated through degree or adjacency inspection, whereas edge-centric tasks require reasoning over triangle support or edge connectivity. This distinction becomes consequential in the reported results.

4. Evaluation criteria

The primary quantitative metric in GraphCS is the F1-score between the predicted community $n$9 and the ground truth $E\subseteq V\times V$0 (Hua et al., 13 Aug 2025). Precision and Recall are defined as

$E\subseteq V\times V$1

and the benchmark uses

$E\subseteq V\times V$2

This choice evaluates overlap at the vertex-set level rather than requiring exact subgraph isomorphism or a task-specific structural penalty. The emphasis is therefore on whether the returned node set matches the target community.

In the CS framework associated with GraphCS, the Validator agent additionally emits a normalized score,

$E\subseteq V\times V$3

to quantify structural soundness and compliance with the cohesiveness metric (Hua et al., 13 Aug 2025). This score is not presented as a replacement for F1; rather, it functions as an internal quality signal within the agentic procedure.

A plausible implication is that GraphCS separates two evaluation layers: benchmark-level agreement with gold communities through F1, and framework-level assessment of candidate validity through the Validator’s normalized score.

5. Empirical performance patterns

GraphCS reports F1-scores, in percent, for all four LLMs on both datasets, across the four tasks and three difficulty levels, under ZERO-SHOT, FEW-SHOT, and 0-CoT prompting (Hua et al., 13 Aug 2025). The concise summary given for ChatGPT identifies several benchmark-level regularities.

First, FEW-SHOT is the best prompting regime. On PSG $E\subseteq V\times V$4-Core, the reported ChatGPT F1-scores are Easy 83.8, Medium 77.0, and Hard 79.2 (Hua et al., 13 Aug 2025). ZERO-SHOT is second, while 0-CoT is often the worst; one reported example is PSG $E\subseteq V\times V$5-ECC Hard at only 3.2% under 0-CoT (Hua et al., 13 Aug 2025).

Second, vertex-centric tasks, namely $E\subseteq V\times V$6-Core and $E\subseteq V\times V$7-Clique, yield higher F1 than edge-centric tasks, namely $E\subseteq V\times V$8-Truss and $E\subseteq V\times V$9-ECC (Hua et al., 13 Aug 2025). This indicates a marked asymmetry in how current LLMs handle different forms of graph cohesiveness.

Third, performance degrades significantly as graph size or community complexity increases, and it also degrades on realistic LFR instances (Hua et al., 13 Aug 2025). The LFR setting therefore appears to be more demanding than PSG, despite both containing planted community structure.

Fourth, GraphCS reports that LLMs can sometimes exceed $q\in Q$0 on easy vertex-centric tasks under FEW-SHOT prompting (Hua et al., 13 Aug 2025). At the same time, the benchmark shows that this success does not transfer uniformly across tasks or difficulty levels.

The benchmark also identifies a specific failure mode termed output bias. Under ZERO-SHOT or 0-CoT prompting, models frequently stray from returning vertex sets and instead propose specific code snippets or irrelevant solutions (Hua et al., 13 Aug 2025). This output bias can affect over 50% of cases with 0-CoT prompting (Hua et al., 13 Aug 2025). In this setting, the error is not merely suboptimal community identification; it is a mismatch between the required output type and the generated response.

6. Interpretation, limitations, and relation to CS-Agent

GraphCS presents LLMs as having preliminary community-search ability but also substantial structural brittleness (Hua et al., 13 Aug 2025). The paper’s interpretation is that LLMs struggle on edge-centric metrics and larger graphs, suggesting that they rely more on pattern-matching than true combinatorial reasoning (Hua et al., 13 Aug 2025). This is an interpretive claim made within the benchmark analysis rather than a theorem established by the experiments.

Prompt sensitivity is another central conclusion. FEW-SHOT grounding drastically reduces bias and improves stability, whereas chain-of-thought reasoning in the 0-CoT setting is described as counterproductive (Hua et al., 13 Aug 2025). This directly challenges the common assumption that additional verbal reasoning traces necessarily improve performance on structured graph tasks. Within GraphCS, the opposite pattern is reported.

The benchmark also records a model-family difference: ChatGPT and Gemini outperform Llama3 and Mixtral, indicating that model scale and pretraining data matter (Hua et al., 13 Aug 2025). Because all models are evaluated on the same benchmark protocol, this comparison functions as an empirical observation about current LLM capability under a fixed text-based graph interface.

GraphCS is also the setting in which CS-Agent is introduced (Hua et al., 13 Aug 2025). CS-Agent is a dual-agent collaborative framework in which two LLMs act as Solver and Validator; through iterative feedback and refinement, it dynamically refines initial results without fine-tuning or additional training, and after the multi-round dialogue a Decider module selects the optimal community (Hua et al., 13 Aug 2025). Extensive experiments are reported to show that CS-Agent significantly improves the quality and stability of identified communities compared to baseline methods (Hua et al., 13 Aug 2025). In this context, GraphCS serves not only as a benchmark but also as the evaluation environment for agentic mitigation of the benchmark’s observed failure modes.

Several limitations are explicit. GraphCS currently supports only single-node queries, with future versions planned to add multi-node or keyword-based queries (Hua et al., 13 Aug 2025). Its datasets are synthetic, even when designed to be realistic, and its graph input modality is restricted to textual serialization. These design choices delimit the present scope of the benchmark. At the same time, they make the benchmark controlled, reproducible, and directly comparable across prompting paradigms and model families. This suggests that GraphCS is best understood as an initial experimental substrate for community-search evaluation in LLMs rather than as an exhaustive account of graph reasoning performance.