Forest-Chat: Interactive Forest Change

• Forest-Chat is an LLM-driven agent architecture that enables interactive analysis of forest changes using bi-temporal satellite imagery.
• It integrates supervised multi-task learning and zero-shot change detection to perform pixel-level segmentation, semantic captioning, and quantitative assessments.
• Designed for transparent, efficient forest monitoring, it uses point-prompt mechanisms and LLM orchestration to refine change interpretation and guide analysis.

Forest-Chat is an LLM-driven agent architecture designed for interactive forest change analysis using bi-temporal satellite imagery. The system combines deep vision-LLMs with LLM orchestration to enable natural language control over diverse remote sensing image change interpretation (RSICI) tasks, including pixel-level change detection, semantic captioning, object counting, deforestation percentage estimation, and interactive change reasoning (Brock et al., 21 Jan 2026, Brock et al., 8 Jan 2026). Forest-Chat represents a significant advance in accessible, explainable, and efficient forest monitoring workflows, supporting fine-grained user interaction through both supervised and zero-shot perception modules.

1. System Architecture and Data Flow

Forest-Chat consists of three core components orchestrated in a two-layer agent architecture: a multi-level change interpretation (MCI) backbone for supervised dual-task learning, a zero-shot change detection module (AnyChange, SAM-based), and an LLM-based orchestration layer.

• Multi-Level Change Interpretation (MCI): The MCI core employs a Siamese SegFormer encoder (MiT-B1 by default) on bi-temporal image pairs. Three Bi-temporal Iterative Interaction (BI3) layers fuse multi-scale visual features, which are split into two task-specific heads: a segmentation decoder for dense binary change masks and a Transformer-based decoder for free-form change captions. Both tasks share the encoder and are optimized jointly.
• Zero-Shot Change Detection (AnyChange/SAM): This module uses the Segment Anything Model (SAM) with a ViT_h backbone to generate mask embeddings for each temporal image, which are matched via pairwise cosine similarity. Higher dissimilarity implies change; regions with change confidence above threshold τ are marked as changed. The module supports interactive point-prompt refinement, allowing users to guide segmentation by clicking on target regions.
• LLM-Based Orchestration: An LLM (e.g., ChatGPT-4o-mini) interprets user input, determines the required analysis steps, invokes vision-language "tools" via structured Python APIs, collates outputs (masks, captions, statistics), and synthesizes succinct responses accessible to both technical and non-technical users. The LLM employs few-shot prompts and iterates over multiple conversational turns, refining results in response to follow-up queries.

The following table summarizes the key system modules:

Module	Backbone / Method	Purpose
MCI	Siamese SegFormer + BI3	Supervised mask/caption generation
Zero-Shot	SAM (ViT_h), latent matching	Promptable, training-free CDC
LLM	ChatGPT-4o-mini	Tool orchestration, synthesis

2. Model Components and Mathematical Formulations

• Zero-Shot Bi-Temporal Latent Matching: For each mask proposal $m_i$ at time $t_1$ with embedding $e_i$ and $m_j$ at $t_2$ with $e_j$, the method computes:

$$\text{sim}(e_i, e_j) = \frac{e_i \cdot e_j}{\|e_i\|\|e_j\|}$$

Change confidence is proportional to $1 - \text{sim}$. Regions exceeding threshold $\tau$ are classified as changed.

• Supervised Multi-Task Loss: Let $\mathcal{L}_{\text{det}}$ be the pixel-wise cross-entropy for change mask prediction, and $\mathcal{L}_{\text{cap}}$ be the sequence cross-entropy for caption generation. To prevent task domination, loss balancing is achieved via gradient detach:

$$\mathcal{L}_{\text{total}} = \mathcal{L}_{\text{det}} + \mathrm{detach}(\mathcal{L}_{\text{cap}}) \quad \text{and} \quad \mathcal{L}_{\text{total}} = \mathcal{L}_{\text{cap}} + \mathrm{detach}(\mathcal{L}_{\text{det}})$$

• Object Counting and Deforestation Percentage:

$$D = \frac{\sum_{i \in \text{change}} A_i}{\sum_{j \in \text{forest}} A_j} \times 100\%$$

where $A_k$ is the pixel area of region $k$. Object counting is performed by extracting connected components from the binary change mask.

• Evaluation Metrics:
  • Mean Intersection-over-Union (mIoU):

  $$\mathrm{mIoU} = \frac{1}{C} \sum_{c=1}^C \frac{TP_c}{TP_c + FP_c + FN_c}$$ - BLEU-4: Assesses 1–4-gram precision between generated and reference captions, using brevity penalty as in Papineni et al. (2002).

3. Datasets and Annotation Protocol

Forest-Chat is evaluated primarily on the Forest-Change dataset, which includes:

• 334 bi-temporal satellite image pairs (cropped to $256 \times 256$ pixels, original GSD $\sim 30$ m/pixel).
• Per-pixel binary change masks (1 = forest loss, 0 = no-change). Foreground (change) area is highly imbalanced, with $>50\%$ of samples exhibiting $<5\%$ change.
• Five captions per pair: one expert-authored human description focused on spatial, quantitative, and qualitative aspects; four rule-based captions generated from mask statistics such as percent loss and patch spatial distribution.
• Data split: 270 train, 31 validation, 33 test.

Annotation proceeds in two stages: human experts first provide semantic descriptions, and rule-based scripts generate metric-driven captions. This results in a bimodal distribution of caption lengths and a diverse caption corpus grounded in both domain expertise and automated pattern mining (Brock et al., 8 Jan 2026, Brock et al., 21 Jan 2026).

4. Interactive Analysis and User Interface

Forest-Chat provides interactive capabilities foundational for transparency and user agency:

• Point-Prompt Mechanism: Users can click on perceived change objects in either input image. The system passes the $(x, y)$ coordinates to SAM, generating a local mask proposal and matching embeddings bi-temporally for refined change segmentation.
• Feedback Loop: After each LLM-controlled "tool" execution, the outputs (e.g., mask arrays, patch statistics) are inspected for adequacy. The LLM may issue additional prompts (such as new points) or switch between zero-shot and supervised branches based on the analysis needs.
• This interactive refinement accommodates both exploratory and hypothesis-driven workflows, streamlining mask precision and semantic output quality.

Use cases include generating overlays of forest change, extracting quantitative statistics (patch count, loss percentage), and providing customized semantic interpretations at sub-region and timescale granularity.

5. Empirical Performance

Forest-Chat demonstrates competitive and in certain domains state-of-the-art results on both change detection and captioning benchmarks.

Change Detection and Captioning Metrics:

Dataset	Model	mIoU	BLEU-4	CIDEr-D
LEVIR-MCI-Trees	FC-Supervised	88.13	34.41	48.69
	FC-Zero-shot	47.32	--	--
Forest-Change	FC-Supervised	67.10	40.17	38.79
	FC-Zero-shot	59.51	--	--

Standard ablation studies show that larger SegFormer backbones modestly improve both mIoU and BLEU-4 scores. On Forest-Change, MiT-B0 yields mIoU 65.86, BLEU-4 34.56; MiT-B2 attains mIoU 68.01, BLEU-4 43.23. Loss balancing via simple detach strategy outperforms dynamic uncertainty or gradient-surgery methods (Brock et al., 21 Jan 2026).

Qualitatively, Forest-Chat accurately localizes both small, fragmented deforestation events and large cohesive clearings, correctly mapping spatial patterns and change types in both mask and caption outputs.

6. Implementation Details

• Framework Components: PyTorch is used for the vision-LLMs (SegFormer, custom heads); HuggingFace Transformers for the decoder and LLM interface; FastAPI or Gradio for web deployment.
• Tool APIs: Analysis tools (mask overlay, statistics, connected component labeling) are implemented as callable Python modules.
• Training: The system employs Adam optimization (LR = $10^{-4}$), with early stopping on plateau of combined mIoU and BLEU-4 on validation.
• Resources: Example inference notebooks, preprocessing scripts, and prompt templates are available at the provided repository.

Dependencies include torch, torchvision, transformers, numpy, scikit-image, and shapely for spatial statistics (Brock et al., 8 Jan 2026).

7. Impact, Limitations, and Extensions

Forest-Chat provides a unified conversational interface for expert and non-expert users, automating complex RSICI workflows tailored to forest monitoring. The dual backbone (supervised and zero-shot) ensures immediate prototyping, while joint mask/captioning tasks promote interpretability.

Limitations include challenges with highly fragmented, small-scale change regions; inference overhead in AnyChange (SAM, ViT_h); and a reliance on rule-based or mask-derived captions for semantic diversity. The LLM orchestration layer may occasionally generate invalid code, suggesting future work in tool indexing and robust API referencing.

Potential extensions are manifold: adaptation to other environmental monitoring tasks (e.g., coastal erosion, glacier retreat, urban sprawl, wetland loss, agriculture) is feasible by retraining the MCI module and updating captioning vocabularies. Incorporation of multisensor (SAR, multispectral) inputs and saturation of geography-aware captions would further strengthen system generalization. Scaling to global deployment with distributed agents and richer temporal sequences, as well as tight integration with external databases (e.g., Global Forest Watch API), represent concrete research directions (Brock et al., 21 Jan 2026, Brock et al., 8 Jan 2026).

---

References:

• "Forest-Chat: Adapting Vision-Language Agents for Interactive Forest Change Analysis" (Brock et al., 21 Jan 2026)
• "Vision-Language Agents for Interactive Forest Change Analysis" (Brock et al., 8 Jan 2026)