WeChat-YATT: Scalable RLHF and UI Automation

• WeChat-YATT is a dual-framework that addresses scalable RLHF training and industrial UI automation testing using modular design and a hybrid ML/LLM approach.
• It leverages distributed parallel controllers, dynamic GPU placement, and asynchronous micro-batching to optimize resource allocation and reduce latency.
• The framework achieves significant performance gains with up to 59% faster RLHF processing and cost-effective UI testing, enhancing robustness on production platforms.

WeChat-YATT (Yet Another Transformer Trainer in WeChat) refers to two distinct, production-scale frameworks that have been independently developed for (1) scalable and balanced Reinforcement Learning from Human Feedback (RLHF) training of large models (Wu et al., 11 Aug 2025), and (2) industrial UI automation testing using a Retrieval-Augmented Generation (RAG) hybrid with LLMs and classical machine learning (Feng et al., 2024). In both domains, WeChat-YATT achieves practical breakthroughs in efficiency, cost, and robustness for large-scale deployments on the WeChat product platform.

1. RLHF Training Framework: System Architecture and Components

WeChat-YATT for RLHF training is architected for simplicity, scalability, and efficient orchestration of complex multimodal feedback workflows operating at Tencent production scale. The system decomposes the RLHF pipeline into disjoint modular processes across the following main components:

• Actor: policy model responsible for generating candidate responses.
• Sampler: rollout engine (e.g., vLLM/SGlang) tasked with auto-regressive generation of outputs.
• Generative Reward Model (GenRM): assigns token-level rewards, including support for chain-of-thought evaluation.
• Critic & Reference Policy: computes value estimates and log-probabilities for PPO-style updates.
• GDataset: sharded, distributed data store enabling efficient prompt/result access.
• Parallel Controllers: one per SPMD rank, responsible for orchestrating resource allocation, stage transitions, and fine-grained scheduling.
• RPC Fabric: enables exactly-once remote procedure invocation with high throughput.

Two execution modes are provided. In the partial-colocated mode, generative and reward stages run asynchronously on GPU subsets, streaming rollouts to the reward model without global synchronization. In fully-colocated mode, models time-share the full GPU pool in a lock-step, swap-in/swap-out manner. This design enables asynchronous micro-batching and flexible matching of compute loads.

The PPO RLHF training pipeline, executed per iteration, includes generation (Actor→Sampler), reward assignment (Sampler→GenRM), preparation (Critic & Reference Policy), and PPO update (Actor+Critic).

2. Parallel Controller Programming Model and Load Distribution

The system eschews traditional centralized controller architectures in favor of a distributed, parallel-controller abstraction instantiated per SPMD rank. Each controller maintains local state, manages a resource group (GPUs + CPU RAM slice), and exposes core operations: wake_up, pull_data (sharded GDataset read), run_stage, collect_results (all-gather/reduce aggregation), offload/onload (model movement between CPU/GPU), and terminate.

Workload balancing is analytically modeled. Given $M$ tasks over $P$ controllers, the expected per-controller workload is $E[W_i] = M/P$ with variance scaling as $Var[W_i] = \sigma^2/P$, so for large $P$, maximum imbalance decays as $O(1/\sqrt{P})$. PPO stage communication uses a single all-reduce of size $O(\log P)$ and $O(1)$ per-micro-batch RPCs. By avoiding centralized “master” resources and hot communication paths, bottlenecks—both CPU and network-side—are minimized.

3. Dynamic Placement Schema and Resource Adaptivity

WeChat-YATT introduces an adaptive, performance-driven GPU placement schema between Sampler (rollout) and GenRM (reward). The placement optimization is formalized as minimizing observed average generation time $f(x)$ over assignment $x$ GPUs to Sampler, $P$0 to GenRM, subject to full GPU utilization:

$P$1

A ternary-search algorithm is invoked at periodic intervals to re-estimate the optimal $P$2. The run-time adaptation loop measures GPU utilization and per-stage latency, invokes search, reallocates GPUs, and remaps process groups dynamically. The initial split uses model size; ongoing repartitioning tracks changes in sequence length distribution and output rejection rates.

4. RLHF Experimental Results and Production Deployment

Testing on clusters of 8×NVIDIA H20-96GB GPUs equipped with CUDA 12.4, PyTorch 2.6, Megatron-core 0.12.2, and SGLang 0.4.6/vLLM, models include Qwen2.5-3B-Instruct (Actor, balanced), Qwen2.5-Math-1.5B (Actor, unbalanced), and GenRM variants up to 72B. Evaluated on GSM8K, WeChat-YATT is compared to VeRL (state-of-art RLHF baseline) across several conditions:

Condition	VeRL Step Time	YATT (full-colocated)	Speedup	Notes
1.5B ↔ 3B Balanced	$P$3	$P$4	≈59.2% faster	Uniform stage speedup
1.5B / 72B Unbalanced	$P$5	$P$6	≈20% faster	Reward stage affected by model swaps
Partial-col. w/ 10–40% rejections	—	16.2–24.1% faster	—	Asynchronous, lower latency

Dynamic placement further reduces average generation time $P$7 below static size-weighted splits across all tested sequence regimes.

In production, WeChat-YATT integrates with Tencent's large training clusters, leverages elastic checkpointing, and supports features like in-app Q&A, math assistants, and multimodal (image + text) RLHF. Observed improvements include a 2× RLHF throughput gain, a >30% reduction in 95th percentile reward latency, and substantially fewer out-of-memory/interruption events due to process-level isolation.

5. UI Automation Testing Architecture and Methodology

A distinct WeChat-YATT framework (termed CAT in (Feng et al., 2024)) addresses cost-effective, high-coverage regression testing of the WeChat UI at industrial scale. The system adopts a hybrid two-phase architecture blending Retrieval-Augmented Generation with transformer-based ML and fallback LLM optimization.

• RAG Module: Selects 1 (rarely more) representative example from 37,971 historical labeled UI tasks via cosine similarity in a T5-encoder space, combined with normalized usage frequency:

$P$8

The chosen example and developer instructions are assembled with the new user-specified task as a succinct LLM prompt.

• ML-Based UI Element Matcher: For each decomposed UI action step (parsed from the LLM output as bracketed primitives), the ML matcher encodes both target label and candidate UI XML node using a shared, fine-tuned T5 encoder. Cosine similarity $P$9 is computed for each:

$E[W_i] = M/P$0

If $E[W_i] = M/P$1 (empirically tuned, e.g. $E[W_i] = M/P$2), the corresponding UI element $E[W_i] = M/P$3 is selected.

• LLM Optimizer: When no candidate exceeds threshold, the LLM (e.g., GPT-4) is prompted with a pruned view-hierarchy and label, returning a corrected target. Only ∼10% of steps require this expensive fallback.

Empirically, this hybrid covers 90% of previously unseen WeChat UI test specifications at a cost of $0.34 per test, surfacing 141 defects over 2,010 tasks (Dec 2023–Jun 2024).

6. Comparative Empirical Evaluation and Integration

On 2,010 unseen UI testing tasks, WeChat-YATT achieves:

• Automation coverage: 90% (matching the LLM-only AdbGPT baseline)
• Average LLM cost: $E[W_i] = M/P$41.07/test for AdbGPT; total $1,467 saved)
• Average test time: 2.65 min, including device/LLM latency
• Bug detection: 141 unique defects surfaced

Ablation highlights the robustness of the hybrid design:

Variant	Coverage	LLM Cost
0-shot RAG	50%	$0.34
5-shot RAG	66%	$0.85
ML-only (no optimizer)	52%	negligible
LLaMA70B backbone	71%	$0.00
AdbGPT (LLM only)	90%	$1.07
Seq2Act (ML only)	35%	none

The framework is fully integrated into WeChat's CI pipeline, automatically executing on developer-provided test descriptions. Manual effort is reduced by ~60%, with a dashboard surfacing failures for further review.

Key engineering challenges include handling ADB/network flakiness and view-hierarchy token pruning to constrain LLM prompt lengths. Critical parameter tuning (confidence threshold τ) balances cost versus reliability. Lessons highlight the importance of prompt minimalism, single high-quality retrievals, and ML/LLM workload separation.

7. Engineering Lessons and Best Practices

For RLHF training, modular process isolation (Sampler, GenRM, Actor), distributed SPMD controller logic, statistics-driven batch reordering, low-cost dynamic placement, and process-level isolation underpin scalability and maintainability. Robust, exactly-once RPC and asynchronous checkpointing address preemption and reliability.

For UI automation, a simple yet robust hybridization of deterministic ML with LLM fallback achieves coverage/cost advantages unattainable by either approach in isolation. The quality of the retrieved example is more critical than increasing the number of context shots. Prompt minimalism and selective fallback usage are essential for both performance and cost control.

Both frameworks demonstrate that targeted architectural choices—parallel control, adaptive placement, prompt-efficient RAG, and fallback optimization—can yield scalable, cost-effective solutions for RLHF and automation at the scale of real-world production deployments (Wu et al., 11 Aug 2025, Feng et al., 2024).