---
title: 'RF-LGBM: Ensemble for Repurchase Prediction'
url: https://www.emergentmind.com/topics/random-forest-with-lightgbm-rf-lgbm
type: topic
---

# RF-LGBM: Ensemble for Repurchase Prediction

Random Forest with LightGBM (RF-LGBM) is an ensemble learning framework specifically developed to predict customer repurchase behavior in community e-commerce platforms, integrating Random Forest (RF) and LightGBM models via a soft-voting strategy. It includes novel advances in feature engineering, sample balancing, and hyperparameter optimization, yielding improved classification performance on highly imbalanced behavioral data [2109.00724].

## 1. Composition and Architecture of RF-LGBM

RF-LGBM is composed of two base learners:

- **Random Forest (RF)**: Trained on binary labels $\{y_i\}_{i=1}^n$, RF generates $M$ bootstrapped samples from the training set and constructs a binary decision tree $h_m(x)$ for each. At each split, RF selects a random subset of features and minimizes an impurity criterion (e.g., Gini). The class probability estimate is
  $$
  P_{\text{RF}}(y=1|x) = \frac{1}{M} \sum_{m=1}^M h_m(x)
  $$
  where $h_m(x)\in\{0,1\}$.

- **LightGBM**: A gradient-boosted ensemble, fitting $T$ trees sequentially via optimization of a regularized loss. The raw score is $f(x)=\sum_{t=1}^T f_t(x)$, with probability output
  $$
  P_{\text{LGBM}}(y=1|x) = \sigma(f(x)) = \frac{1}{1 + e^{-f(x)}}
  $$
  Training uses second-order Taylor expansion of $L$, as well as GOSS (Gradient-based One-Side Sampling) and EFB (Exclusive Feature Bundling) for computational efficiency.

The two outputs are combined by a weighted average, yielding an ensemble probability:
$$
P_\text{ensemble}(y=1\mid x) = \omega P_\text{RF}(y=1\mid x) + (1-\omega) P_\text{LGBM}(y=1\mid x)
$$
with $\omega = 0.5$ (equal weights) in reported experiments.

Final classification is by thresholding at $\tau=0.5$: label 1 if $P_\text{ensemble}\geq 0.5$, else 0 [2109.00724].

## 2. Enhanced Feature Engineering: Improved RFM Model

RF-LGBM leverages an improved RFM (Recency, Frequency, Monetary) model expanded to five indicators for fine-grained description of purchase behavior:

- $R$ (Recency): $T_\text{last\_time} - T_\text{p\_last\_time}$, time since last purchase.
- $F$ (Frequency): Count of purchases within reference window.
- $M$ (Monetary): $\sum_{i=1}^F M_i$, total purchase value.
- $S$ (Span): $T_\text{p\_last\_time} - T_\text{p\_first\_time}$, customer-product relationship duration.
- $T$ (Average Inter-purchase Interval): $S / F$.

The conceptual rationale is that lower $R$ and $T$ denote more recent or habitual buying, while higher $F$, $M$, $S$ indicate loyalty and high-value customers.

## 3. Class Imbalance Handling: SMOTE-ENN Pipeline

The native prevalence of positive (repurchase) instances is approximately 9%. To address this, RF-LGBM applies a two-stage balancing technique:

1. **SMOTE (Synthetic Minority Over-sampling Technique)**: For each minority sample $X_i$, $k=5$ nearest neighbors $\{X_{ij}\}$ are selected, and synthetic samples $Y_j$ generated by
   $$
   Y_j = X_i + \mathrm{rand}(0,1) \cdot (X_{ij} - X_i)
   $$
2. **ENN (Edited Nearest Neighbor)**: Cleansing stage that applies a 5-NN classifier to the augmented data; samples with disagreement between observed and predicted class are removed.

Following SMOTE-ENN, the class distribution is approximately balanced: $\sim 52$k positives to $\sim 52$k negatives (1:1).

## 4. Hyperparameter Optimization via TPE

Hyperparameter search is automated via the Tree-structured Parzen Estimator (TPE). For each learner, the search space includes:

- RF: $n_\text{estimators}\sim$ UniformInteger[50,500], $max\_depth\sim$ UniformInteger[5,30], $criterion\in\{$gini, entropy$\}$, $max\_features\sim$ UniformInteger[1, \#features].
- Objective: maximize F$_1$ on a held-out validation set.

TPE models $p(x|y)$ by densities $l(x)$ (good trials, $y<y^*$) and $g(x)$ (bad, $y\geq y^*$) and maximizes expected improvement:
$$
EI_{y^*}(x) \propto \left(\gamma + \frac{g(x)}{l(x)}(1-\gamma)\right)^{-1}
$$
where $\gamma=p(y<y^*)$.

Empirical comparison shows TPE tuning completes in 31.25 s versus $\sim 150$ s (random search) and $\sim 200$ s (grid), with more than 450% speed-up and higher F$_1$.

## 5. Predictive Performance and Experimental Comparison

The following summarizes performance (mean test set metrics, 10 random splits):

| Model        | Accuracy | Recall | F$_1$  | Source             |
|--------------|----------|--------|--------|--------------------|
| RF-LGBM      | 0.871    | 0.952  | 0.859  | [2109.00724]       |
| RF           | 0.862    | 0.911  | 0.853  | [2109.00724]       |
| LightGBM     | 0.858    | 0.951  | 0.842  | [2109.00724]       |
| XGBoost      | --       | --     | ~0.83  | [2109.00724]       |
| CNN-LSTM     | 0.856    | 0.839  | 0.847  | H. Xiaoli et al.   |
| LSTM         | 0.802    | 0.842  | 0.822  | H. Xiaoli et al.   |
| Teacher-Student | 0.9198 | --     | --     | Shen et al.        |

RF-LGBM outperforms both constituent single models and published CNN-LSTM and LSTM baselines in terms of F$_1$ metric. This suggests that probabilistic soft-voting between random forests and gradient-boosted trees, when rigorously optimized and balanced, produces robust results in imbalanced structured behavioral data domains.

## 6. Methodological Significance and Application Context

By combining: (1) an improved five-feature RFM profile, (2) SMOTE-ENN balancing, (3) TPE-driven hyperparameter tuning, and (4) soft-voting ensemble of RF and LightGBM, RF-LGBM delivers a repurchase predictor with both improved F$_1$ and over 4$\times$ reduction in training time compared to traditional search strategies. The methodology is particularly suited to community e-commerce, characterized by volatile customer loyalty and severe class imbalance [2109.00724]. A plausible implication is that this ensemble strategy can generalize to other domains exhibiting similar heterogeneity in behavioral data and label imbalance.

Source: https://www.emergentmind.com/topics/random-forest-with-lightgbm-rf-lgbm