DRScaffold: Boosting Dense-Scene Reasoning in Lightweight Vision Language Models

Abstract: Lightweight vision-LLMs perform competitively on standard benchmarks yet fail systematically in dense-scene reasoning, where multiple objects, attributes, and relations must be jointly grounded and resolved through multi-step inference. Such capability is critical for real-world applications where models must reliably interpret cluttered environments. Yet existing training signals provide no explicit grounding between reasoning steps and the underlying visual entities and relations, leaving lightweight models free to generate fluent but visually unanchored reasoning chains. To address this gap, we first introduce DRBench, a benchmark of 14,573 questions across 2,943 images, organized into five task categories spanning three progressive reasoning layers. Building on DRBench, we propose DRScaffold, a supervised fine-tuning framework that decomposes the supervision target into four causally ordered stages, enforcing grounded reasoning without architectural modification. Experiments on three lightweight VLMs demonstrate substantial gains on DRBench while preserving or improving performance on general-purpose benchmarks. Notably, Qwen2.5-VL-3B trained with DRScaffold surpasses the frozen Qwen2.5-VL-32B on DRBench, demonstrating that structured supervision can substitute for a significant portion of model scale in dense-scene reasoning. Our code and models are available at https://github.com/irene-shi/DRScaffold .

• The paper demonstrates that a structured four-stage schema (entity grounding, relation modeling, stepwise reasoning, and answer generation) significantly improves dense-scene reasoning in lightweight VLMs.
• The methodology uses staged gradient masking and mixed fine-tuning on the DRBench benchmark, yielding performance gains up to +27.6 percentage points over baselines.
• The results validate that explicit intermediate supervision anchors visual and textual cues, enabling robust performance in complex scenarios such as indoor scenes and urban driving.

DRScaffold: Boosting Dense-Scene Reasoning in Lightweight Vision LLMs

Motivation and Problem Definition

Lightweight Vision-LLMs (VLMs), despite competitive performance on standard multimodal benchmarks, exhibit systematic failure in dense-scene reasoning tasks, where accurate interpretation requires grounding multiple entities, modeling complex relations, and performing multi-step inference. The lack of explicit supervision over intermediate steps leaves lightweight models prone to generating superficially plausible yet visually unanchored reasoning chains. This undermines reliability in downstream applications, especially in domains such as domestic robotics and autonomous driving, where dense-scene understanding is critical. Existing training protocols—outcome-based and flat chain-of-thought (CoT) distillation—are fundamentally limited, as they fail to enforce entity-relation-grounded reasoning.

Benchmark: DRBench

To address these deficiencies, the paper introduces DRBench, a comprehensive dense-scene reasoning benchmark comprising 14,573 questions over 2,943 images, with a structured annotation format encompassing key entities, scene graphs, and staged reasoning chains (Figure 1).

Figure 1: DRBench sample annotation with image, question, final answer, and structured intermediate stages (entity, scene graph, reasoning chain).

DRBench images are sourced from Hypersim (static indoor scenes) and Cityscapes (dynamic urban outdoor scenes), covering high compositional complexity and diverse spatial distributions. The dataset stresses three primary dense-scene failure modes—relation misinterpretation, attribute misbinding, and hallucinated dependencies—through strict visual grounding requirements and adversarial question design (Figure 2, Figure 3).

Figure 2: Entity and relation distribution in DRBench, differentiating indoor and outdoor domains.

Figure 3: Distribution and taxonomy of question categories; each illustrated by representative samples.

DRBench employs a multi-stage validation pipeline combining automated filtering, LLM-based annotation, and human verification, ensuring that each question is decisively vision-dependent and structurally consistent (Figure 4).

Figure 4: Quality-control pipeline, highlighting filtering phases with illustrative failure cases.

A unique feature is the False Premise Rejection category, probing hallucination robustness by embedding visually absent referents in complex queries, making hallucination strictly harder to detect.

Methodology: DRScaffold Framework

DRScaffold is a supervised fine-tuning framework designed for lightweight VLMs, imposing a fixed four-stage schema on the reasoning process: entity grounding, relation modeling, stepwise reasoning, and answer generation (Figure 5).

Figure 5: DRScaffold schema, with each output stage conditionally grounded on preceding stages.

Training employs staged gradient masking—progressively unlocking fields for loss computation across four phases—forcing models to internalize grounding before answer synthesis. Mixed fine-tuning (DRBench + LLaVA-Instruct-150K) is used in the final phase to mitigate catastrophic forgetting while promoting generalization.

Experimental Evaluation

DRScaffold was applied to three lightweight VLMs (InternVL2.5-2B, Qwen2.5-VL-3B, Phi4-multimodal) spanning 2–6B parameter scales. Extensive evaluation was performed on DRBench, as well as general benchmarks (MMMU, MMStar, RealWorldQA, HallusionBench, BLINK).

Key quantitative results:

• DRScaffold yields dominant improvements on DRBench: Qwen2.5-VL-3B + DRScaffold achieves 62.6 combined accuracy (+27.6pp), surpassing the frozen Qwen2.5-VL-32B, demonstrating that structured supervision can compensate for a significant portion of model scale in dense-scene tasks.
• Gains on HallusionBench indicate enhanced entity-attribute grounding and improved hallucination robustness (see Table of main results in the paper).
• Phi4-multimodal + DRScaffold achieves the highest average benchmark score despite modest DRBench improvement due to stronger pre-training priors.

Qualitative comparisons show that baseline lightweight models frequently misidentify or hallucinate visual evidence in complex scenes, while DRScaffold-tuned counterparts consistently anchor reasoning steps to validated entities and relations, yielding correct answers (Figure 6).

Figure 6: Qualitative comparison of base and DRScaffold-enhanced models; correct and incorrect answers highlighted.

Correlation analysis negates output length as a confounding factor, with DRScaffold-trained models achieving both higher accuracy and shorter outputs (Figure 7).

Figure 7: Scatter plot of output length vs. accuracy for different training methods.

Stage-wise ablation studies and diagnostic metrics (Scene Graph F1, Reasoning Validity) confirm monotonic accuracy gains and validate that causal ordering of stages is necessary; removal or randomization of stages strongly degrades performance (Figure 8, Figure 9).

Figure 8: Stage-wise diagnosis—category accuracy, scene graph F1, and reasoning validity improvements.

Figure 9: Visualization of model outputs at each stage of training; chain-of-thought and answer correctness evolve with schema enforcement.

Implications and Future Directions

The results strongly support the claim that structured intermediate supervision, causally ordering entity, relation, and reasoning fields, induces dense-scene reasoning capabilities in lightweight VLMs without architectural modifications. Practically, this enables deployment of low-latency, resource-efficient models in domains requiring robust scene understanding. Theoretical implications include substantiation of the hypothesis that scale can be partially replaced by structure, provided supervision targets are cognitively aligned with task demands. The staged gradient masking strategy induces inductive biases that prevent shortcut learning and encourage factually anchored rationalization.

Limitations include coverage restricted to indoor and urban driving scenes, diminishing returns on models with already strong dense-scene priors, and increased training time proportional to stage count. Future work should explore dynamic schema adaptation, broader scene distributions, and hybrid methods combining architectural and supervision innovations.

Conclusion

DRScaffold delivers a principled solution to dense-scene reasoning in lightweight VLMs by enforcing compositional grounding through a four-stage schema and staged supervision. Extensive empirical validation demonstrates substantial accuracy gains and improved robustness to hallucination, with the surprising result that small models can outperform larger frozen baselines when given structured supervision. This work offers a blueprint for scalable multimodal reasoning and underlines the importance of explicit intermediate supervision in bridging the gap between visual fluency and factual grounding in practical multimodal systems.