EdgeCrafter: Compact ViTs for Edge Dense Prediction

• EdgeCrafter is a unified compact Vision Transformer framework designed for dense prediction tasks on resource-constrained edge devices, integrating a compact ViT backbone with task-specific distillation.
• It employs a detection-specialized teacher model and a lightweight multi-scale feature generator to enhance performance across object detection, instance segmentation, and human pose estimation.
• Empirical results on COCO demonstrate that EdgeCrafter achieves competitive AP scores with lower GFLOPs, outperforming traditional lightweight CNN systems in efficiency and accuracy.

EdgeCrafter is a unified compact Vision Transformer (ViT) framework for edge dense prediction, where “edge” denotes deployment on resource-constrained edge devices rather than low-level edge extraction. Introduced around ECDet, an object detector built from a distilled compact backbone and an edge-friendly encoder–decoder design, the framework is extended to instance segmentation through ECInsSeg and to human pose estimation through ECPose. Its motivating claim is that the gap between compact ViTs and lightweight CNN systems is largely due to insufficient task specific representation learning in small scale ViTs, rather than an inherent mismatch between ViTs and edge dense prediction (Liu et al., 19 Mar 2026).

1. Problem setting and framework scope

EdgeCrafter addresses dense prediction under strict computation and memory limits. The paper situates the problem in the practical dominance of CNN-based architectures such as YOLO for lightweight object detection, instance segmentation, and pose estimation, while compact ViTs often fail to realize a comparable accuracy–efficiency tradeoff even with large scale pretraining. EdgeCrafter is presented as a counterexample to the view that compact ViTs are intrinsically unsuitable for such workloads: it combines a compact ViT backbone, task-specialized distillation, and an edge-aware prediction stack, then evaluates the result on COCO detection, instance segmentation, and human pose estimation (Liu et al., 19 Mar 2026).

The framework is organized around three task instantiations. ECDet is the detection model and serves as the architectural core. ECInsSeg reuses the distilled backbone and encoder, adding a lightweight mask head on top of the feature map $F^{(8)}$. ECPose replaces the detection head with a set-based pose head using one instance token and $K$ keypoint tokens per query. In all three cases, the downstream models are trained on COCO annotations only.

A central organizational distinction in the paper is between the student backbone, denoted ECViT, and the teacher models, denoted ECTeacher. The teacher is not simply a frozen generic foundation model; it is first adapted to detection, and only then used for feature distillation. This design choice is foundational to the rest of the framework.

2. ECViT backbone and edge-friendly prediction stack

The ECViT backbone departs from standard ViT patch embedding. Instead of a single $16\times16$ convolution with stride $16$, EdgeCrafter uses a convolutional stem composed of four $3\times3$ convolution layers with stride $2$. The stated purpose is to preserve fine spatial detail while maintaining progressive receptive-field growth; the ablation reported for ECDet-M further shows that ConvStem with dilation $1$ raises $AP_S/AP_M/AP_L$ by $+2.2/0.9/-0.1$ at $+2.4$ GFLOPs in the “Patch Embedding” study (Liu et al., 19 Mar 2026).

The transformer body is single-stage. The model family is expressed through variants $K$0, $K$1, $K$2, and $K$3, which are used in ECDet-S, ECDet-M, ECDet-L, and ECDet-X, respectively. Across these scales, the embedding dimension increases from $K$4 to $K$5, the number of heads from $K$6 to $K$7, and the FFN expansion ratio from $K$8 to $K$9. At $16\times16$0 resolution, the associated teacher assignments are ECTeacher-S for the smallest model and ECTeacher-B for the larger three scales.

The multi-scale feature generator is explicitly designed without a heavy FPN. Let $16\times16$1 be the last two block tokens. These are fused by

$16\times16$2

then reshaped to spatial resolution $16\times16$3. Multi-scale maps are produced by bilinear resampling $16\times16$4 and a $16\times16$5 convolution $16\times16$6: $16\times16$7

The encoder–decoder then follows a DETR-like decomposition. The encoder uses AIFI on $16\times16$8 to enlarge context and CCFF to fuse features from $16\times16$9. The decoder performs set prediction with 4 layers of self-attention, deformable cross-attention, and FFN over 300 learned object queries $16$0. The paper’s reported decoder configuration is 4 layers, 300 queries, and FFN $16$1 (Liu et al., 19 Mar 2026).

For detection, the total loss is

$16$2

Here $16$3 is a Varifocal-style focal loss for classification, $16$4 and $16$5 are standard box-coordinate regression terms, and $16$6 and $16$7 are decoupled distillation focal loss and fine-grained localization loss.

3. Detection-specialized teacher construction and feature distillation

Task-specialized distillation is the core methodological claim of EdgeCrafter. The paper first converts a large DINOv3 ViT into a detection-specialized teacher by attaching the ECDet head to DINOv3-$16$8 and fine-tuning on COCO detection. This yields ECTeacher-S and ECTeacher-B. The teacher is therefore aligned with the downstream geometry of dense prediction before the student distillation phase begins (Liu et al., 19 Mar 2026).

The distillation target is the student’s last-block representation. Let $16$9 denote the student’s last-layer tokens and $3\times3$0 the teacher’s last two block outputs. A linear adapter $3\times3$1 aligns the channel dimensions, and the feature loss is

$3\times3$2

The reported distillation setup uses ImageNet1K plus COCO images for 50 epochs, the LARS optimizer, warmup plus cosine decay, and batch size $3\times3$3.

Several ablations clarify how the distillation procedure behaves. Adding one register token improves alignment; removing it drops AP from $3\times3$4 to $3\times3$5. Matching student capacity to teacher capacity is important: smaller students are paired with DINOv3-S, larger students with DINOv3-B, while DINOv3-L as teacher degrades the ECDet-M ablation result to $3\times3$6 AP. Adapting the teacher to detection before distillation adds $3\times3$7 AP, and using IN1K+COCO rather than IN1K only adds $3\times3$8 AP in the same study.

Feature alignment depth is also studied directly on ECViT-T+. Aligning one student layer to one teacher layer yields $3\times3$9 AP; aligning one student layer to the teacher’s last two layers yields $2$0 AP; aligning one student layer to three teacher layers reaches $2$1 AP on T+ but is reported as less stable on larger students; aligning two student layers to two teacher layers yields $2$2 AP. The adopted default is therefore single-student-layer alignment to the teacher’s last two layers. A plausible implication is that EdgeCrafter treats distillation not as generic representation transfer but as a controlled interface between compact student capacity and task-adapted teacher semantics.

4. Task-specific model variants

ECDet is the canonical instantiation. It reuses the distilled ECViT backbone together with the lightweight multi-scale generator, the RT-DETR-style encoder, and the 300-query decoder. The backbone and encoder are reused across tasks, which gives the framework a unified structure rather than three independent model families (Liu et al., 19 Mar 2026).

ECInsSeg adds a lightweight mask head on $2$3. Mask logits are formed as a dot product between a query MLP and a pixel MLP. The training objective augments the detection loss: $2$4 This design keeps the task-specific addition relatively small while preserving the set-prediction formulation inherited from ECDet.

ECPose replaces the detection head with a set-based pose head containing one instance token and $2$5 keypoint tokens per query. Its losses are given as

$2$6

The formulation is notable because the pose model is not framed as a heatmap regressor; it remains structurally consistent with the set-prediction design used for detection.

The training pipeline is three-stage. First, DINOv3-$2$7 weights are adapted into ECTeacher models by fine-tuning with the ECDet head on COCO detection using AdamW and a learning rate schedule from $2$8 over 50 epochs. Second, ECViT variants $2$9 are distilled for 50 epochs using IN1K+COCO, crop $1$0, flip, color augmentation, mixup, LARS, and one register token. Third, the downstream task models are trained using the distilled ECViT plus encoder, with task-specific heads, COCO annotations only, and training durations of 50–90 epochs with AdamW, a $1$1 learning rate, LARS-scaled backbone learning rate, and Mosaic/Mixup.

5. Empirical performance on COCO

All reported metrics are on COCO val2017, and latency is measured on NVIDIA T4 with batch size $1$2, FP16, and TensorRT v10.6 (Liu et al., 19 Mar 2026).

Task	EdgeCrafter result	Comparison reported in the same evaluation
Object detection	ECDet-S: 51.7 AP, 10M params, 26 GFLOPs, 5.4 ms	Higher AP than YOLOv10-S (46.3), D-FINE-S (48.5), and DEIMv2-S (50.9)
Human pose estimation	ECPose-X: 74.8 AP, 50.6M params, 172.2 GFLOPs, 14.3 ms	Higher AP than YOLO26-Pose-X* (71.6, O365) and DETRPose-X* (73.3, O365)
Instance segmentation	ECInsSeg-S: 43.0 AP, 10.3M params, 33.1 GFLOPs, 7.0 ms	Comparable to RF-DETR-Seg-S* at 43.1 AP with substantially fewer parameters

For detection, the scale progression is monotonic: ECDet-S achieves $1$3 AP, ECDet-M $1$4, ECDet-L $1$5, and ECDet-X $1$6. The corresponding AP$1$7 values are $1$8, $1$9, $AP_S/AP_M/AP_L$0, and $AP_S/AP_M/AP_L$1, while AP$AP_S/AP_M/AP_L$2 reaches $AP_S/AP_M/AP_L$3, $AP_S/AP_M/AP_L$4, $AP_S/AP_M/AP_L$5, and $AP_S/AP_M/AP_L$6. The smallest model already exceeds the lightweight baselines listed in the paper, and the larger models continue that trend.

For pose estimation, ECPose-S records $AP_S/AP_M/AP_L$7 AP, AP$AP_S/AP_M/AP_L$8, AP$AP_S/AP_M/AP_L$9, $+2.2/0.9/-0.1$0, and $+2.2/0.9/-0.1$1. At the large end, ECPose-X reaches $+2.2/0.9/-0.1$2 AP, AP$+2.2/0.9/-0.1$3, AP$+2.2/0.9/-0.1$4, $+2.2/0.9/-0.1$5, and $+2.2/0.9/-0.1$6. The comparison emphasized in the abstract is that ECPose-X exceeds YOLO26Pose-X despite the latter’s reliance on extensive Objects365 pretraining.

For instance segmentation, ECInsSeg-S reaches $+2.2/0.9/-0.1$7 AP, AP$+2.2/0.9/-0.1$8, AP$+2.2/0.9/-0.1$9, $+2.4$0, $+2.4$1, and $+2.4$2. ECInsSeg-X reaches $+2.4$3 AP with AP$+2.4$4 and AP$+2.4$5. The paper characterizes ECInsSeg as comparable to RF-DETR while using substantially fewer parameters.

6. Ablations, design rationale, and interpretation

The ablation studies define the framework’s design rationale more sharply than the headline metrics alone. On the optimization side, LARS is reported as $+2.4$6 AP relative to AdamW in the distillation setting. On feature aggregation, the mean of the last two layers gives the best trade-off at $+2.4$7M parameters, $+2.4$8 GFLOPs, $+2.4$9 ms latency, and $K$00 AP; concatenation or STA, as used in DEIMv2, provides no clear gain once overhead is counted (Liu et al., 19 Mar 2026).

The patch-embedding stem dilation study shows that moderate receptive field is preferred. With dilation $K$01, AP is $K$02; with $K$03, AP falls to $K$04; with $K$05, AP is $K$06 with a larger drop on small and medium objects. This is consistent with the stated motivation for the four-layer convolutional stem: preserving fine spatial detail without overly diffuse receptive-field expansion.

A common misconception directly addressed by the paper is that compact ViTs are inherently a poor fit for edge dense prediction. EdgeCrafter’s authors reject that interpretation and instead attribute the usual shortfall to inadequate task-specific representation learning in small-scale ViTs. The empirical package supporting that claim is not just the final COCO numbers but the combined evidence from teacher task adaptation, capacity matching, register tokens, and feature alignment depth. This suggests that the framework’s main contribution lies in making compact ViTs task-specialized before downstream deployment, rather than merely shrinking a generic ViT.

The framework’s scope is also specific. Despite the name, EdgeCrafter is not a system for low-level edge maps, 3D edge reconstruction, or geometric edge fitting. It is a compact ViT family for dense prediction on edge devices, instantiated for detection, segmentation, and pose. Code availability is listed at https://intellindust-ai-lab.github.io/projects/EdgeCrafter/, reinforcing its presentation as a deployable research framework rather than a single-task benchmark model (Liu et al., 19 Mar 2026).