Automated Segmentation and Tracking of Group Housed Pigs Using Foundation Models

Abstract: Foundation models (FM) are reshaping computer vision by reducing reliance on task-specific supervised learning and leveraging general visual representations learned at scale. In precision livestock farming, most pipelines remain dominated by supervised learning models that require extensive labeled data, repeated retraining, and farm-specific tuning. This study presents an FM-centered workflow for automated monitoring of group-housed nursery pigs, in which pretrained vision-language FM serve as general visual backbones and farm-specific adaptation is achieved through modular post-processing. Grounding-DINO was first applied to 1,418 annotated images to establish a baseline detection performance. While detection accuracy was high under daytime conditions, performance degraded under night-vision and heavy occlusion, motivating the integration of temporal tracking logic. Building on these detections, short-term video segmentation with Grounded-SAM2 was evaluated on 550 one-minute video clips; after post-processing, over 80% of 4,927 active tracks were fully correct, with most remaining errors arising from inaccurate masks or duplicated labels. To support identity consistency over an extended time, we further developed a long-term tracking pipeline integrating initialization, tracking, matching, mask refinement, re-identification, and post-hoc quality control. This system was evaluated on a continuous 132-minute video and maintained stable identities throughout. On 132 uniformly sampled ground-truth frames, the system achieved a mean region similarity (J) of 0.83, contour accuracy (F) of 0.92, J&F of 0.87, MOTA of 0.99, and MOTP of 90.7%, with no identity switches. Overall, this work demonstrates how FM prior knowledge can be combined with lightweight, task-specific logic to enable scalable, label-efficient, and long-duration monitoring in pig production.

• The paper presents a novel FM-driven pipeline that eliminates manual annotation and retraining for pig segmentation and tracking.
• The methodology integrates Grounding-DINO and Grounded-SAM2 with modular post-processing to address challenges like occlusion, stacking, and low-light conditions.
• Quantitative evaluations show robust performance with metrics such as J=0.83, F=0.92, and over 80% correct tracking across long-duration videos.

Foundation Model-Based Pig Video Segmentation and Tracking: Technical Summary and Implications

Motivation and Conceptual Framework

Precision livestock farming increasingly relies on visual monitoring systems for group-housed pig management, with conventional pipelines dominated by supervised learning models (SLMs) and persistent limitations in scalability, annotation cost, and adaptability. The paper "Automated Segmentation and Tracking of Group Housed Pigs Using Foundation Models" (2604.03426) proposes an alternative paradigm utilizing vision-language foundation models (FM) to eliminate dependencies on supervised model training, manual data annotation, and retraining, enabling scalable, label-efficient, and long-duration pig monitoring.

The FM-centered workflow leverages Grounding-DINO and Grounded-SAM2 for prompt-based detection, segmentation, and tracking. The pipeline is modular, allowing farm-specific adaptation via post-processing strategies that enhance performance under occlusion, stacking, and low-light conditions. The conceptual shift is illustrated below.

Figure 1: Conceptual comparison and the FM-based pipeline, showing text-prompted detection, mask generation, and mask propagation across video frames.

Experimental Setting and Data Acquisition

Data were collected from a nursery farm using high-resolution cameras arranged to maximize coverage of multiple pens. Pigs were individually marked and videos acquired in RGB during the day and IR-grayscale at night, covering 120 pigs across 0.97 TB of footage.

Figure 2: Schematic and actual setup for camera placement relative to nursery pens.

Three distinct tasks structured the evaluation: (1) static image pig detection; (2) short-term video segmentation; (3) long-term video segmentation with identity and mask consistency.

Detection Performance

Grounding-DINO was applied to 1,418 annotated images. Model thresholds were systematically tuned, and detection criteria adhered to strict bounding box IoU. At the optimal threshold, true positives, false positives, and false negatives were explicitly quantified. Detection yielded strong daytime (RGB) performance, diminished recall and precision under night vision, and degraded results with occlusion or stacking.

Figure 3: Plot of F1 score, recall, and precision across thresholds; hexbin histogram of precision/recall; post-evaluation breakdown (TP, FP, FN).

Detection performance is strongly influenced by environmental factors, notably illumination and pig separation. False positives often stem from model detection of animals outside pen boundaries or partial-body visibility not present in ground truth, underscoring the discord between universal FM perception and dataset-specific annotation policies.

Figure 4: Visual detection examples across daytime/nighttime and pig activity/stacking; illustrations of false positive scenarios.

Short-Term Video Segmentation and Tracking

Short-term segmentation was performed on 550 video clips, yielding 4,927 auto-annotated videos and 2.96 million frames. Grounded-SAM2 masks were generated for each detected pig; tracking errors were manually categorized by an expert observer, and modular post-processing corrected systematic issues.

Common challenges identified include: (1) detection outside the target pen, suppressed by spatial filtering; (2) duplicated mask labels, resolved with mask-level NMS; (3) ID switches and lost tracks, mitigated via model substitution (SAM2-Large), bidirectional propagation, and anchor frame re-initialization.

Figure 5: Short-term video segmentation examples including pigs outside the pen, stacking, incorrect masks, duplicated IDs, ID switches, and lost tracks.

Tracking reliability remained >80% correct tracks, with remaining errors predominantly arising during periods of high animal activity or occlusion.

Figure 6: Distribution of stacked/unstacked pigs and hourly activity patterns; error rates and categories for tracking within active pig tracks.

Bidirectional propagation and anchor frame correction, as illustrated below, effectively suppressed ID switches and stabilized segmentation.

Figure 7: Diagram showing error correction via GroundingDINO anchor frame and bidirectional SAM2 tracking.

Long-Term Video Segmentation and Modular Pipeline

The extended pipeline comprised six modules: initializer, tracker, matcher, mask refinement, re-identification, and post-segmentation QC—designed to maintain identity and mask consistency over multi-hour durations.

Figure 8: Modular workflow integrating initialization, mask propagation, matching, mask refinement, ReID, and quality control.

Quantitative evaluation on 132 uniformly sampled frames from a 2-hour video reported region similarity ($\mathcal{J} = 0.83$), contour accuracy ($\mathcal{F} = 0.92$), combined $\mathcal{J}{content}\mathcal{F} = 0.87$, MOTA = 0.99, MOTP = 90.7%, and no identity switches. Segmentation accuracy remained robust across individual pigs, with slight drops correlating with animal inactivity and stacking.

Figure 9: Time-series of mean $J$, $F$, and $J{content}F$ scores; per-individual pig $J$ scores.

Qualitative results demonstrate identity maintenance and mask propagation via modular association across video clips. The initializer selects optimal anchor frames; tracker ensures temporal continuity; matcher aligns identities across segments.

Figure 10: Extended segmentation showing anchor initialization, mask propagation, and Matcher module results across varied pig activity states.

Mask refinement modules ensure masks remain within pen boundaries and maintain spatial coherence, preventing erroneous jumps and duplicate IDs during occlusion or clustering.

Figure 11: Mask refinement filters suppress drift outside pen and large spatial jumps, ensuring blob consistency across frames.

Error Analysis and Limitations

Principal error sources include heavy occlusion and stacking, particularly during inactivity or feeding, and degraded night-time performance owing to grayscale IR imaging. Rule-based filtering and modular post-processing alleviate—but do not fully eliminate—tracking failures in these scenarios. Model interpretation occasionally diverges from dataset labeling conventions, especially concerning partial and outside-pen animal detections.

Theoretical and Practical Implications

This FM-driven, zero-shot monitoring pipeline demonstrates that contemporary foundation models, when augmented with lightweight domain-specific post-processing, achieve segmentation and tracking performance comparable to fully supervised pipelines, with superior scalability and diminished annotation requirements. The modular workflow is robust under challenging livestock environments.

This approach obviates retraining for each deployment, enabling rapid scalability across farms. It also supports efficient evaluation by limiting annotation effort to post-hoc review of error-flagged frames, accommodating constraints in farm-side computational infrastructure. Surveillance, behavior analysis, and longitudinal phenotyping in swine production now become practically scalable, facilitating downstream integration of FM-accelerated annotation engines and synthetic data generation, and opening a path toward minimally supervised digital phenotyping.

The adaptability of FMs is noteworthy; extension to other livestock species or domains (e.g., sheep, cattle, poultry) is straightforward given prompt-based architectures. However, resource demands and edge inference limitations require further development of lightweight FM variants (e.g., MobileSAM, Track-Anything) and knowledge distillation for practical deployment.

Prospects for Future AI Development

The modular FM pipeline sets a precedent for fully unsupervised livestock monitoring. Opportunities for improvement remain in robust occlusion handling, integration of adaptive ReID modules, and resilience to adverse imaging conditions. Multimodal models (e.g., SAM3, Sa2VA) may further enhance semantic grounding and temporal consistency. Expanded cross-site evaluation and domain-adaptive methods are warranted to ensure external validity in commercial farm settings.

Conclusion

The research establishes that foundation models alone suffice for reliable, scalable, and label-free pig segmentation and tracking across long-duration videos, achieving strong quantitative and qualitative results without supervised retraining or extensive annotation. FM-centered approaches facilitate rapid deployment and high accuracy, serving as a viable path toward broader autonomous precision livestock farming, while exposing technical limitations related to occlusion, stacking, and domain shift that remain active areas for future research.