UniCrop: Universal Crop Data Pipeline

• UniCrop is a universal crop yield data pipeline that automates data acquisition, cleaning, harmonisation, and feature engineering from diverse environmental sources.
• The pipeline integrates over 200 variables from satellites, meteorological, soil, and topographic datasets using a configuration-driven modular architecture.
• Validation on rice yield data shows robust performance, highlighting UniCrop’s scalability and adaptability across various crops and regions.

UniCrop is a universal and reusable data pipeline for crop yield prediction that automates the acquisition, cleaning, harmonisation, and engineering of multi-source environmental data for any given location, crop type, and temporal window (Khidirova et al., 4 Jan 2026). It is designed to address a recurrent limitation in crop-yield modelling: existing approaches remain crop- or region-specific and require data engineering efforts, which constrains scalability, reproducibility, and operational deployment. In the reported implementation, UniCrop retrieves, harmonises, and engineers over 200 environmental variables from Sentinel-1/2, MODIS, ERA5-Land, NASA POWER, SoilGrids, and SRTM, then reduces them to a compact, analysis-ready feature set through a structured feature-reduction workflow with minimum redundancy maximum relevance (mRMR) (Khidirova et al., 4 Jan 2026).

1. Scope and problem formulation

UniCrop is defined primarily as a data-engineering framework rather than as a single predictive model. Its central claim is that the preparation of consistent and harmonised multi-source data is the primary bottleneck in operational crop yield modelling, and that this bottleneck can be addressed by decoupling data specification from implementation (Khidirova et al., 4 Jan 2026). The framework is intended to support any crop, region, and time frame through simple configuration updates rather than modifications to core pipeline logic.

The motivating data regime is explicitly heterogeneous. Accurate crop yield prediction is described as relying on diverse data streams, including satellite, meteorological, soil, and topographic information. UniCrop formalises this heterogeneity into a universal pipeline in which the user specifies a field table and a feature-mapping file, while the system generates the data-fetching, harmonisation, and feature-engineering workflow automatically.

A common misunderstanding is to treat UniCrop as synonymous with the downstream regression models used in its validation. The reported study instead uses four baseline machine learning models to validate the quality of the engineered feature set. This distinction matters: the framework’s primary contribution is reusable, configuration-driven data preparation, while model fitting is a downstream stage built on top of the engineered features.

2. Modular pipeline architecture

UniCrop is organised as five modular stages, each driven purely by a human-readable configuration file, specifically unicrop_feature_mapping.csv plus region- or crop-specific parameters (Khidirova et al., 4 Jan 2026). The user provides two principal inputs: a table of fields containing latitude, longitude, planting and harvest dates, and crop type; and a feature-mapping CSV listing each desired environmental variable with key name, API parameter, source dataset, and derivation notes. From these inputs, the pipeline builds a fetch plan enumerating every field × variable × date request.

The second stage is data acquisition. Multi-source data are retrieved in parallel via Google Earth Engine (GEE) and public APIs. The listed sources are Sentinel-1 SAR, Sentinel-2 optical, MODIS, ERA5-Land reanalysis, NASA POWER agroclimate, SoilGrids, and SRTM topography.

The third stage is harmonisation and cleaning. Raw outputs are temporally aligned to the crop’s defined window, spatially aggregated or buffered around each field polygon, standardised into ISO dates and consistent column names with numeric casting and float precision, deduplicated while favoring more complete records, and documented through provenance metadata stored in unicrop_columns_manifest.csv. Missing values are intentionally carried through until fold-wise imputation to avoid leakage.

The fourth and fifth stages are feature engineering and feature reduction. Agro-climatic and phenological indicators are computed automatically, and the resulting feature space is reduced to 15 features by default through near-zero-variance filtering, correlation pruning, a family-preservation heuristic, and iterative mRMR selection. The modularity of these stages is central to the framework’s transferability: the only manual step is specifying what to fetch, not how to engineer it.

3. Integrated data sources and harmonisation logic

UniCrop ingests and pre-processes a broad set of environmental data families (Khidirova et al., 4 Jan 2026). Sentinel-2 MSI contributes all surface reflectance bands and derived vegetation indices including NDVI, EVI, SAVI, NDRE, and CI$_{\text{red-edge}}$, with cloud masking using SCL. Sentinel-1 GRD contributes VV and VH backscatter, the VV/VH ratio, Radar Vegetation Index (RVI), and texture measures. MODIS products include MOD13Q1, MOD15A2, MOD16A2, and MOD17A2, yielding 16-day NDVI/EVI, fPAR, LAI, evapotranspiration (ET), and GPP. ERA5-Land provides hourly temperature, precipitation, soil moisture, and radiation, which are converted to daily aggregates. NASA POWER supplies T2M, T2M$_{\min}$, T2M$_{\max}$, RH2M, VPD, DTR, wind speed, and solar radiation. SoilGrids contributes 0–30 cm texture fractions, soil organic carbon (SOC), pH, bulk density, and N content, while SRTM contributes elevation, slope, aspect, and hillshade.

The harmonisation procedure standardises these heterogeneous sources into a common analytical frame. All layers are re-projected to a common CRS; time series are merged on (field, date); dates are represented in ISO-8601 format; naming is made consistent; and numeric values are cleaned into uniform numeric types. Missingness is preserved until fold-wise imputation. This design allows optical, SAR, meteorological, pedological, and topographic variables to be combined without prematurely collapsing source-specific uncertainty or introducing cross-fold leakage.

The architecture implies a strict separation between source heterogeneity and model-ready representation. A plausible implication is that UniCrop is intended not merely to aggregate variables, but to impose a stable, provenance-aware schema over datasets with distinct spatiotemporal resolutions and delivery mechanisms.

4. Feature engineering and reduction

Feature engineering in UniCrop includes agro-climatic and phenological indicators such as Growing Degree Days, chill nights, seasonal amplitude, SAR texture metrics, and cross-terms (Khidirova et al., 4 Jan 2026). The examples given are Growing Degree Days, defined as $\mathrm{GDD}_{\text{base}\,10}$; chill nights as the count of days with $T_{\min}<15^\circ\mathrm{C}$; seasonal amplitude as $\max\mathrm{NDVI} - \min\mathrm{NDVI}$; SAR texture metrics such as the variance of VV/VH; and cross-terms including clay × radiation and elevation × temperature.

The feature-reduction workflow is staged. It begins from $\sim 160$–$200$ harmonised and engineered variables, while the abstract describes automatic retrieval and engineering of over 200 environmental variables. Near-zero-variance features are removed first. Pairs with Pearson $r\ge 0.98$ are then pruned, keeping the feature with higher mutual information with yield. A family-preservation heuristic ensures that at least one candidate remains from each data family: meteorology, vegetation, SAR, soil, and topography. Finally, mRMR is applied until $K=15$ features have been selected.

The mRMR formulation is stated explicitly. For target $_{\min}$0, candidate feature $_{\min}$1, and already selected set $_{\min}$2, relevance is

$_{\min}$3

Redundancy with the current selected set is

$_{\min}$4

Two criteria are given. The difference form is

$_{\min}$5

and the quotient form is

$_{\min}$6

where $_{\min}$7 is a small constant to avoid division by zero. Features are added iteratively until $_{\min}$8. All mutual information calculations and Pearson and Spearman correlations are performed inside each cross-validation fold to prevent leakage. This fold-wise execution is methodologically consequential because it preserves the distinction between training-time feature selection and evaluation-time generalisation.

5. Modelling protocol and empirical validation

UniCrop was validated on a rice yield dataset comprising 557 field observations, using only the selected 15 features (Khidirova et al., 4 Jan 2026). The training protocol employs 5-fold shuffle cross-validation at the field level, with all preprocessing nested inside each training fold. The preprocessing pipeline comprises fold-wise imputation, scaling, feature filtering, and feature selection.

The imputation strategy is source-family-specific. Meteorological variables use an iterative multivariate imputer; vegetation indices use K-NN imputation with local district/season grouping; soil and topography use median imputation; and extremes are winsorised at 1%. Scaling is performed with a robust scaler based on median and IQR. The baseline models are LightGBM, Random Forest, SVR with RBF kernel, and ElasticNet with $_{\min}$9–$_{\max}$0 regularised linear regression.

A constrained ensemble is fit on out-of-fold predictions. Let $_{\max}$1 denote the out-of-fold predictions from model $_{\max}$2. The ensemble weights $_{\max}$3 are obtained by solving

$_{\max}$4

via Sequential Least Squares Programming (SLSQP). Performance is reported with RMSE and the coefficient of determination $_{\max}$5:

$_{\max}$6

$_{\max}$7

The best single-model result is obtained by LightGBM, with RMSE $_{\max}$8 kg/ha and $_{\max}$9. The constrained ensemble further improves performance to RMSE $\mathrm{GDD}_{\text{base}\,10}$0 kg/ha and $\mathrm{GDD}_{\text{base}\,10}$1. These results are reported as cross-validated results for the rice case study. They should therefore be interpreted as evidence that the compact feature set retains predictive signal under nested preprocessing, rather than as a claim that a particular regressor defines UniCrop itself.

6. Adaptability, implementation, and relation to harmonised crop data infrastructures

UniCrop is explicitly configuration-driven (Khidirova et al., 4 Jan 2026). Because all data sources, variables, temporal windows, and spatial bounds are defined in a small set of configuration files rather than hard-coded in scripts, the framework can be repurposed for any crop type, region or administrative unit, and planting or harvest dates or multi-season studies. The stated examples of crop type are maize, wheat, and soybean. Adaptation requires editing the field metadata table, unicrop_feature_mapping.csv, and optionally config.py for directory paths or GEE asset IDs. No changes to core pipeline logic are required.

The implementation stack is Python-based and uses pandas, geopandas, scikit-learn, LightGBM, SVR in scikit-learn, SHAP for interpretability, and the GEE Python API. The codebase is structured into config.py and paths.py for global settings and directories; data_acquisition/ for source retrieval via GEE or REST API; harmonisation/ for spatial and temporal alignment and cleaning; feature_engineering/ for GDD, texture, and cross-terms; feature_selection/ for near-zero-variance filtering, collinearity filtering, the family heuristic, and mRMR; modeling/ for the CV loop, imputation, scaling, training, ensemble fitting, and metrics; and interpretability/ for SHAP analysis. The GitHub repository contains a fully open example on a public maize yield dataset, sample field tables, unicrop_feature_mapping.csv, scripts and notebooks, and a Dockerfile and environment YAML for reproducibility.

The deployment sequence is also specified: clone the repository and install dependencies with conda env create -f environment.yml; prepare a CSV of fields with latitude, longitude, dates, and yields if available; edit unicrop_feature_mapping.csv; run python run_pipeline.py --stage all; launch train_models.py; and review results and SHAP outputs in the output/ directory. This procedural explicitness is part of the framework’s operational orientation.

In the broader ecosystem of agricultural data infrastructures, UniCrop is adjacent to large-scale harmonised crop-parcel resources such as EuroCrops (Schneider et al., 2023). EuroCrops contains geo-referenced polygons of agricultural croplands from 16 countries of the European Union, translated and mapped into the Hierarchical Crop and Agriculture Taxonomy (HCAT), and released as harmonised vector data for downstream applications including crop yield estimation (Schneider et al., 2023). A plausible implication is that EuroCrops-style parcel harmonisation and UniCrop-style environmental feature engineering are complementary: the former standardises field geometry and crop labels, while the latter standardises multi-source environmental covariates for yield modelling. This distinction helps separate two often conflated tasks in agricultural ML pipelines—label harmonisation and environmental feature construction—even when both are prerequisites for scalable regional or cross-regional analysis.