VEHME: Satellite & Handwritten Math Benchmarks

• VEHME is a dual-purpose initiative combining a high-resolution satellite dataset for Middle Eastern vehicle detection with a vision-language model for assessing handwritten math expressions.
• The satellite dataset features 4,282 valid image tiles and over 113,000 annotated vehicles, rigorously curated to overcome geographic bias in overhead imagery.
• The vision-language model leverages spatial prompting and reinforcement learning to deliver human-interpretable grading with state-of-the-art accuracy on handwritten mathematics.

The acronym VEHME refers to two distinct, high-profile research efforts: the Vehicles in the Middle East (VME) dataset and benchmark for satellite vehicle detection, and VEHME, a Vision-LLM for Evaluating Handwritten Mathematics Expressions. Both embody advances in dataset design and multimodal machine learning, serving as benchmarks and architectures for specific, under-explored domains. This entry details their technical foundations, construction methodologies, benchmark results, and broader research implications.

1. Vehicles in the Middle East (VME): Satellite Detection Dataset

The Vehicles in the Middle East (VME) dataset directly addresses the geographic bias prevalent in overhead imagery benchmarks by providing a high-resolution satellite vehicle detection corpus focused on Middle Eastern contexts. Unlike established datasets, which are dominated by Western, East Asian, or globally blended scenes, VME consists exclusively of image tiles from 54 cities spanning 12 Middle Eastern countries. Imagery is sourced from Maxar's SecureWatch platform, with natural-color pan-sharpened data at 30–50 cm ground sampling distance, off-nadir angles less than 30°, and low cloud cover. Images are temporally recent, primarily within a few years of 2024.

Tile generation starts from 2,714 large satellite snapshots. Random 512 × 512 pixel crops undergo filtering to eliminate empty or irrelevant regions, yielding 4,282 valid tiles. Final ground-truth comprises 113,737 oriented bounding boxes (OBBs), consisting of 101,564 cars (≈89.3%), 6,846 trucks (≈6.0%), and 5,327 buses (≈4.7%). Each OBB is labeled by a hybrid system: an AI-assisted crowdsourcing platform (Co-one) performs initial annotation, targeting 95% accuracy via 6,000+ annotators, followed by cross-validation and error correction. Every vehicle is bounded by four oriented corners, with corresponding class identifiers, and derived horizontal bounding boxes (HBBs) for interoperability with standard COCO-style protocols (Al-Emadi et al., 28 May 2025).

2. Dataset Statistics, Composition, and Benchmark Integration

By MS-COCO criteria, all annotated cars are tiny (area < 1,024 px²), while trucks and buses are mostly small, with negligible medium-sized outliers. Tiles are split at 5/8 train, 1/8 validation, and 2/8 test ratios. The table below summarizes split statistics:

Category	Train (#imgs / #ann.)	Val (#imgs / #ann.)	Test (#imgs / #ann.)
Car	2,449 / 63,051	768 / 12,055	1,066 / 26,458
Bus	1,574 / 3,079	513 / 1,004	821 / 1,702
Truck	1,041 / 4,140	347 / 1,004	694 / 1,702
All	2,505 / 70,270	536 / 13,733	1,241 / 29,734

To enable global benchmarking, VME was incorporated into the Car Detection in Satellite Imagery (CDSI) dataset, which unites six overhead image benchmarks (VME, xView22, DOTA-v2.0, VEDAI, DIOR, FAIR1M-2.0). Only small cars (HBB area < 400 px²) are retained as "car;" other small objects (< 400 px²) are merged as a consolidated hard-negative "other" class. Labels are standardized to HBB+MS-COCO format. Final CDSI size is 23,250 images, 896,760 car instances, and 185,619 other-small instances; a CDSI* variant excludes VME to measure its cross-region transfer impact (Al-Emadi et al., 28 May 2025).

3. Detection Architectures, Experimental Protocols, and Evaluation Metrics

Two state-of-the-art object detectors were evaluated. TOOD (Task-Aligned One-Stage Detector) offers a classic, strong one-stage paradigm; DINO (DETR-style with Swin-L backbone and denoising) leverages transformer-based architecture for robust object–relation modeling. All models adopt the SAHI (Slicing Aided Hyper Inference) framework with fine-tuning, full inference, and patch overlap–optimized for recall on tiny objects. Training protocols vary: TOOD uses SGD (batch size 16, 24 epochs, learning rate modifications at epochs 9, 16, 22); DINO (Swin-L) employs AdamW (batch size 2, 36 epochs, decays at epochs 27, 33). Standard augmentations are applied. Detection is evaluated by IoU and mean Average Precision (mAP) at relevant thresholds:

• $$\mathrm{IoU} = \frac{\mathrm{area}(B_\mathrm{pred}\cap B_\mathrm{gt})}{\mathrm{area}(B_\mathrm{pred}\cup B_\mathrm{gt})}$$
• $$\mathrm{mAP} = \frac{1}{|\mathcal{T}|}\sum_{t\in\mathcal{T}} \mathrm{AP}(t)$$

where AP(t) denotes area under the precision-recall curve for threshold t (Al-Emadi et al., 28 May 2025).

4. Empirical Results and Cross-Region Generalization

On the VME test set, TOOD achieves mAP50 = 45.1% (car: 30.3%, bus: 27.9%, truck: 33.4%), and DINO achieves mAP50 = 50.6% (car: 32.8%, bus: 29.8%, truck: 35.9%). DINO outperforms TOOD by +7.2% mAP50 overall, underscoring both the dataset's difficulty and the efficacy of transformer backbones.

When models are trained on existing Western or global datasets and evaluated on VME, performance degrades significantly (TOOD mAP50 ranges: xView 38.2%, DOTA 53.3%, VEDAI 5.1%, DIOR 33.5%, FAIR1M 15.8%). DINO's mAP50 is modestly better but remains low, especially on VEDAI and DIOR. The CDSI* (no VME) variant achieves mAP50 = 62.3% on its own test set but exhibits pronounced underperformance on Middle Eastern imagery (Al-Emadi et al., 28 May 2025).

A notable finding is that models trained on VME alone generalize moderately well to global datasets (DINO mAP50 on CDSI test: 58.8%), confirming the benefit of including regionally diverse data. Full CDSI training (with VME) further boosts cross-region performance (TOOD mAP50: 82.3% on VME, 73.8% on CDSI test; DINO: 86.8% on VME, 74.5% on CDSI test). The inclusion of VME yields measurable mAP50 gains compared to CDSI* (+6.0% for TOOD, +4.3% for DINO).

Error analysis indicates that major detection failures are due to imperfect localization (IoU < 0.50) and background false positives; class confusion is a smaller component. False negatives are rare in Middle Eastern test sets with well-tuned models.

5. Limitations, Biases, and Future Directions (VME/CDSI)

Several dataset and benchmark limitations are identified:

• Urban-rural imbalance: most VME tiles are urban, with limited rural/desert representation.
• Seasonal and weather artifacts: haze and dust in some images introduce unavoidable blur.
• Temporal diversity is constrained; most source imagery is recent and excludes extreme seasonal conditions.

Recommended future research directions include the integration of more rural and peri-urban environments, use of multi-spectral or SAR modalities, pan-temporal sequences for vehicle tracking, and domain adaptation methods to mitigate remaining geographic biases (Al-Emadi et al., 28 May 2025).

6. VEHME: Vision-LLM for Handwritten Math Assessment

VEHME (Vision-LLM for Evaluating Handwritten Mathematics Expressions) is a specialized vision-LLM aimed at automated, interpretable grading of open-form handwritten mathematics. VEHME adopts an encoder-decoder VLM backbone (Qwen2.5-VL-7B-Instruct), augmented with two critical modules: an Expression-Aware Visual Prompting Module (EVPM) for robust spatial attention, and a two-phase training pipeline blending supervised fine-tuning and reinforcement learning using Group Relative Policy Optimization (GRPO) (Nguyen et al., 26 Oct 2025).

Given a tuple $(q, r, s)$ of problem image, reference answer (in LaTeX-like math tokens plus text), and student solution image, the vision encoder produces visual tokens incorporating both raw pixels and EVPM-predicted expression bounding boxes. These tokens guide the language decoder to generate autoregressive output comprising a chain-of-thought, a correctness label, and, when necessary, an error localization string.

EVPM is a lightweight, YOLOv11-based oriented bounding-box predictor, trained on 10,000 synthetically rendered multi-line math canvases. The EVPM training loss combines $\ell_1$ box regression and binary cross-entropy for objectness. Inference yields $K$ rotated boxes per image, each localizing a math line or chunk, providing explicit spatial prompts to the VLM (Nguyen et al., 26 Oct 2025).

Supervised fine-tuning utilizes labels distilled from a 32B math-reasoning LLM (QwQ-32B), formatted as > ...<correctness>...<localization>. The SFT loss is standard sequence cross-entropy. Reinforcement learning proceeds with GRPO, sampling groups of completions, standardizing advantages, and optimizing a composite reward $r(\hat{y})$ integrating correctness, localization, length, cosine similarity, and repetitiveness.

Empirical evaluation on the AIHub and FERMAT handwritten math benchmarks demonstrates VEHME's state-of-the-art open-source performance: AIHub error detection 73.01% accuracy (49.22% F1), localization 61.13% accuracy (58.18% F1); FERMAT error detection 62.61% accuracy (29.81% F1), localization 31.90% accuracy (44.36% F1). VEHME outperforms other open-source baselines by 20–25 points and approaches proprietary models (GPT-4o, Gemini-2.x). A hallmark is that predictions produce human-interpretable reasoning traces with precise spatial localization, leveraging bounding box cues to attend to lines regardless of arbitrary handwriting layout (Nguyen et al., 26 Oct 2025).

7. Practical Implications and Extensions

VEHME's architecture obviates the need for brittle OCR and template-based methods, rendering it robust to messy, real-world student submissions. Its parameter-efficient design (7B backbone with LoRA adapters) enables deployment on moderate hardware or distillation for e-learning platforms, providing real-time feedback, stepwise student guidance, and aggregate analytics for instructors.

This suggests that the spatial prompting and multimodal, interpretable reasoning framework of VEHME could underpin similar auto-grading advances across STEM domains (e.g., physics derivations, chemical equations) and facilitate scaling to millions of submissions, provided sufficient synthetic data and expert distillation pipelines are available.

---

Both VEHME in the context of remote-sensing and mathematical handwriting grading represent benchmarks advancing their respective domains by focusing on under-represented modalities and contexts, leveraging rigorous annotation and innovative machine learning architectures to close longstanding gaps in generalization, interpretability, and real-world performance (Al-Emadi et al., 28 May 2025, Nguyen et al., 26 Oct 2025).