Coupled Input-Forget Gate (CIFG) in Recurrent Networks

• The CIFG cell couples the input and forget gates, reducing parameters by 25% compared to LSTMs while preserving essential sequence modeling capabilities.
• Scale-invariant extensions leverage ReLU and max-normalization to stabilize gradients and enable effective training with plain SGD in federated environments.
• Empirical results show SI-CIFG achieves faster convergence, improved accuracy, and better privacy-utility trade-offs, making it suitable for cross-device learning under differential privacy constraints.

The Coupled Input–Forget Gate (CIFG) is a memory-efficient recurrent cell variant, structurally derived from the LSTM, distinguished by the coupling of input and forget gates and more recently extended with scale-invariant nonlinearities for improved stability and efficiency in federated and differentially private learning. Characterized by reduced parameter count—25% fewer than LSTM—CIFG cells retain the salient gating and memory mechanisms requisite for sequence modeling, while mitigating both computational and memory overhead. Scale-invariant extensions further optimize CIFG for high-variance, cross-device federated settings, enabling effective training with non-adaptive optimizers such as plain SGD on-device without loss of utility or convergence stability (Ro et al., 2024).

1. Standard CIFG Cell: Architecture and Formulation

CIFG cells implement a shared gating strategy between the input gate ($i_t$) and forget gate ($f_t$), imposing the relationship $i_t = 1 - f_t$. At each timestep $t$, given input $x_t \in \mathbb{R}^d$ and preceding hidden state $h_{t-1} \in \mathbb{R}^h$, the following calculations are performed:

$$f_t = \sigma(W_f x_t + U_f h_{t-1} + b_f), \qquad i_t = 1 - f_t$$

$o_t = \sigma(W_o x_t + U_o h_{t-1} + b_o)$

with $\sigma$ denoting elementwise sigmoid, $W_*$, $f_t$0, $f_t$1 the learned parameters for each gate or update. Cell state and output are evaluated as:

$f_t$2

The coupling of gates yields a parameter reduction and computational savings, while maintaining the nonlinear sequence processing capabilities typical of LSTMs.

2. Scale-Invariant Extensions: SI-CIFG Cell

Standard CIFG exhibits sensitivity to input scale and parameter magnitudes, often causing instability during training under SGD. The scale-invariant CIFG (SI-CIFG) substitutes conventional nonlinearities with scale-invariant surrogates composed of ReLU and max-normalization operations:

• Max-normalization:

$f_t$3

• SI-sigmoid:

$f_t$4

• SI-tanh:

$f_t$5

These replacements yield gate and cell updates:

$f_t$6

$f_t$7

$f_t$8

$f_t$9

Gradient magnitudes become invariant to rescalings, directly addressing gradient explosion/vanishing and enabling robust, higher learning rates in federated settings (Ro et al., 2024).

3. Training Dynamics in Cross-Device Federated Learning

In federated experiments—where client optimizers are restricted to plain SGD and severe computational/memory constraints apply—SI-CIFG demonstrates accelerated and more stable convergence relative to standard CIFG. On the Stack Overflow dataset, SI-CIFG (19M parameters) attains test perplexity 33.6 within 100 rounds; baseline CIFG (19M) converges to ~35.5 after 3000 rounds. In production settings (live keyboard), SI-CIFG remains stable (>2000 rounds), achieving test perplexity $i_t = 1 - f_t$0 compared to $i_t = 1 - f_t$1 for CIFG, alongside enhanced next-token accuracy (Ro et al., 2024).

4. Privacy-Utility Trade-Offs under Differential Privacy Constraints

With DP-FTRL server aggregation and a fixed z-CDP privacy budget ($i_t = 1 - f_t$2, $i_t = 1 - f_t$3-DP $i_t = 1 - f_t$4 for reasonable $i_t = 1 - f_t$5), SI-CIFG (6M params) yields superior utility metrics at a fixed privacy loss compared to base CIFG:

Model	Perplexity	In-vocab Accuracy
SI-CIFG	$i_t = 1 - f_t$6	$i_t = 1 - f_t$7
CIFG	$i_t = 1 - f_t$8	$i_t = 1 - f_t$9

Training conditions: noise multiplier $t$0, clipping norm $t$1, 6500 clients per round. On Federated Adam (non-private), SI-CIFG (19M) outperforms CIFG in both convergence speed and final Stack Overflow test accuracy (33.6% vs. 33.1%) (Ro et al., 2024).

5. Implementation and Deployment Recommendations

For practical adoption, all CIFG nonlinearities and cell updates are wrapped in a scale-invariant ReLU+max-norm pipeline. Max-normalization and other SI operations should be performed with full float precision prior to any quantization on resource-limited devices. SI-CIFG remains compatible with plain SGD (client, no moment storage) and both adaptive (FedAdam) and non-adaptive (SGD+momentum) server-side optimizers. The SI principle also extends to Transformer architectures (row-norm replaces SoftMax), feed-forward layers, and further enhances convergence gains when deployed at scale (Ro et al., 2024). SI-CIFG models tolerate higher client learning rates (e.g. 0.5–2.0) and exhibit reduced sensitivity to server optimizer hyperparameters, simplifying deployment and diminishing hyperparameter search costs.

6. Extensions and Research Context

The SI-CIFG methodology generalizes to other architectures, notably scale-invariant Transformer blocks and attention mechanisms, where similar ReLU+row-norm operations can replace SoftMax. Empirical evidence substantiates that such extensions maintain or enhance convergence and stability under federated and differentially private constraints. A plausible implication is that scale-invariance may serve as a general architectural principle for robust aggregation under statistical heterogeneity and privacy constraints. The short proof of scale-invariance for SI activations directly clarifies avoidance of recurrent gradient instability, interacting closely with themes from foundational work (e.g., “On the difficulty of training RNNs”—Pascanu et al. 2013) on high-variance client updates.

7. Summary and Implications

Coupling input and forget gates while enforcing scale-invariant nonlinearities in recurrent networks yields robust, memory-efficient sequence models with demonstrable utility, stability, and privacy advantages in federated learning settings. SI-CIFG achieves faster convergence, superior accuracy, and improved privacy-utility trade-offs, particularly under challenging cross-device scenarios using memory-constrained optimizers and strict privacy budgets. This suggests scale-invariant gating and activation mechanisms are instrumental for enabling scalable, private, and efficient distributed model training (Ro et al., 2024).