FlexLLM: Custom LLM Inference & Deployment

• FlexLLM is a family of techniques that enables dynamic customization of LLM inference parameters, enhancing security and efficiency.
• It integrates moving target defenses for black-box API security with co-serving systems for simultaneous inference and parameter-efficient finetuning on shared GPUs.
• The framework also features a composable HLS library and advanced memory management, addressing hardware bottlenecks and enabling rapid domain-adapted solutions.

FlexLLM denotes a family of techniques and frameworks enabling flexible, highly efficient customization of LLM inference and deployment. Across recent literature, FlexLLM specifically refers to: (1) moving target defenses for black-box LLM API security by dynamically reconfiguring decoding hyperparameters and system prompts; (2) a co-serving runtime for simultaneous inference and parameter-efficient finetuning (PEFT) at token-level granularity on shared GPUs; and (3) a composable high-level synthesis (HLS) library for the rapid hardware specialization of LLM inference pipelines with multi-stage accelerator architectures. Each instantiation of FlexLLM targets core bottlenecks of LLM production deployment: security against adversarial attacks, hardware utilization and throughput, and seamless algorithm–system–hardware coupling for domain-adapted solutions.

1. Moving Target Defense in Black-Box LLM APIs

FlexLLM introduces a robust defense paradigm for LLMs deployed via black-box APIs, especially against jailbreak attacks that coerce models into producing disallowed content despite system prompt controls (Chen et al., 2024). In this setting, model weights and internal states are inaccessible; only decoding hyperparameters—temperature ($T$), top-P ($p$), top-K ($K$), and maximum output length—plus system prompts, can be manipulated. FlexLLM formalizes the set of decoding hyperparameter configurations as $H = \{h_1, \ldots, h_{|H|}\}$, where each $h$ is a tuple $(T, p, K, \text{max\_tokens})$.

Defense proceeds in two phases:

• Static Optimization: A bi-criteria objective identifies a "best safe" configuration $h^*$ as

$$h^* = \arg\min_{h\in H}\Bigl(\mathcal{L}_{\text{attack}}(h) + \lambda\,\mathcal{L}_{\text{quality}}(h)\Bigr),$$

where $\mathcal{L}_{\text{attack}}$ is the average jailbreak success rate over adversarial prompts, $\mathcal{L}_{\text{quality}}$ captures benign perplexity or user ratings, and $\lambda$ trades off reliability versus utility.

• Dynamic Moving Target Defense (MTD): At inference time, FlexLLM samples time-varying pairs $(h^{(t)}, s^{(t)})$—that is, per-query decoding hyperparameters and system prompts—from dynamically maintained pools vetted for rejection of known adversarial attacks. The attack surface thus shifts continuously, thwarting transferability and adaptation by query-based adversaries.

Empirical results demonstrate substantial attack success rate (ASR) reductions: e.g., on Dolphin-llama2-7B, FlexLLM–MTD yields mean ASR of 15% (0% for DeepInception), compared to 33% baseline and 65% for Retokenization static defense. Similar robust reductions are observed on Vicuna-7B (ASR from 9%→3%) and Llama2-7B-chat (3%→1%), with minimal latency or quality penalty (e.g., $<$2% perplexity increase; online overhead 5–10 ms per query). These outcomes are statistically significant at $p<0.01$ (Chen et al., 2024).

2. Co-Serving LLM Inference and Parameter-Efficient Finetuning

FlexLLM implements the first system that fuses LLM inference and parameter-efficient finetuning (PEFT) at the token level within shared GPU environments (Oliaro et al., 2024). Traditional serving stacks isolate inference and finetuning—leading to resource fragmentation. FlexLLM exploits the observation that inference workloads are bandwidth-bound, while PEFT workloads are compute-bound, and both use a shared backbone model.

Its system architecture includes:

• PEFT-as-a-Service Interface: Both inference (prompt→next-token) and fine-tuning (sequence→loss) requests are serviced by a unified API.
• Co-Serving Iterations: Each iteration comprises pre-fill (for key/value cache setup), decode (one token generation using fused backbone and PEFT bypass modules), and token-level fine-tuning (mini-batch forward/backward over a window of tokens).
• Kernel Fusion and Memory Optimization: By leveraging static compilation, dependent parallelization, and graph pruning, FlexLLM minimizes redundant activation storage and launch overhead. Graph pruning eliminates activation dependencies unrelated to trainable PEFT weights, with memory savings up to 80% (activation footprint reduction by 7–8×).
• Hybrid Token Scheduler: A runtime scheduler dynamically interleaves inference and fine-tuning tokens in each iteration according to latency SLOs. The formal objective is:

$$\max \ U = c_i + \alpha s \quad \text{subject to} \quad L(c_i, s) \leq L_{\text{SLO}},$$

where $c_i$ is the number of inference tokens, $s$ is the number of fine-tuning tokens, $L$ is the estimated latency.

Comprehensive benchmarks on LLaMA-3.1-8B, Qwen-2.5-14B, and Qwen-2.5-32B exhibit up to a 4.8× boost in fine-tuning throughput under heavy workload, and 2.5–6.8× under lighter loads, with $>$76% of peak fine-tuning efficiency preserved even at maximum inference demand. Inference SLOs (20 req/s) are met without visible impact to user latency (Oliaro et al., 2024).

3. Hardware Specialization: HLS Library and Accelerator Design

FlexLLM provides a composable HLS library for the rapid development of customized accelerator pipelines for LLM inference (Zhang et al., 22 Jan 2026). This library—implemented in C++/TAPA HLS—exposes fine-grained architectural parameters, enabling designers to tune prefill and decode stages independently.

Key architectural degrees of freedom:

• Temporal Reuse vs. Spatial Dataflow: Temporal reuse shares engines across layers and tokens for high resource efficiency, while spatial dataflow instantiates per-kernel modules for low-latency streaming.
• Stage-Customized Design: Prefill (compute-bound) adopts a hybrid array for arithmetic intensity, while decode (memory-bound, autoregressive) relies on temporally reused, intra-token-parallel INT4 engines.
• Parallelism and Quantization: Module parameters include token_parallelism (TP), block_parallelism (BP), weight_parallelism (WP), and head_parallelism. Quantization is configurable down to INT1, per-tensor/channel/token, with outlier handling leveraging learned rotations and Fast Hadamard Transform (FHT) blocks. Experimental ablation on WikiText-2 for Llama-3.2 1B shows INT4/INT8 quantization delivers 12.68 perplexity (vs. 8.94 for FP16; SpinQuant INT4 baseline at 13.30).
• Composability: Modules (LinearLayer, NonLinearLayer, Quantizer, Dequantizer) are instantiated via streaming interfaces, allowing arbitrary pipeline composition for rapid hardware specialization.

4. Advanced Memory and Context Management

FlexLLM’s HLS framework integrates a Hierarchical Memory Transformer (HMT) plug-in, enabling efficient long-context LLM inference by mitigating the $\mathcal{O}(L^2)$ cost of key/value cache prefill. HMT employs segment-wise compression, memory-attention over a rolling queue of summary embeddings, and cross-attention augmentation.

Key workflow:

1. Prompts are partitioned into segments; partial summaries are computed via half-segment and topic-token reduction.
2. Cross-attention retrieves relevant content from $N$ most recent memory blocks.
3. The model recombines full segments, contextually retrieved summaries, and recent tokens to produce new memory embeddings.

On AMD U280 FPGA, HMT reduces prefill latency by 23.23× and extends the viable context window by 64×, with negligible resource overhead ($<$7.5% CLB, $<$2% DSP) and end-to-end latency impact ($<$0.6%) (Zhang et al., 22 Jan 2026).

5. Empirical Evaluation and Comparative Results

FlexLLM Application	Platform/Model	Key Experimental Metrics
MTD Defense (LLM API)	Dolphin-llama2-7B	ASR 15% (vs. 33% baseline, 65% Retokenization); $<$2% PPL increase
Co-Serving System (GPU)	LLaMA-3.1-8B, etc.	Up to 80% memory savings; 1.9–4.8× finetuning speedup; SLO-compliant latency
HLS Accelerator (FPGA/ASIC projection)	Llama-3.2 1B on U280/V80	1.29–4.71× speedup vs. A100/BF16; up to 6.27× energy efficiency; HMT 23× faster

Statistical significance (p < 0.01) is reported for all major ASR reductions in MTD defense, and fine-tuning metrics remain within 76% of theoretical maxima even at peak inference load. The source code for the co-serving system is publicly available (https://github.com/flexflow/FlexFlow/).

6. Limitations, Extensibility, and Future Directions

• For black-box moving target defense, scalability is currently limited by the combinatorial grid search over the hyperparameter space $|H|$ and controlled pool expansion for prompts. There is no formal regret or convergence bound for adaptive adversaries, though expected ASR reduction can be bounded by re-weighted sampling over "safe" configurations.
• The co-serving runtime and compilation framework are primarily designed for models with frozen backbones and trainable small PEFT modules; extension to large-scale, multi-modal, or non-autoregressive architectures is noted as future work.
• HLS hardware support is functionally modular and easily extensible (≈10K LoC for the library; 1K LoC for complete model integration); projected platforms include current and next-generation AMD FPGAs.
• Cross-system composability is a core property: FlexLLM defenses can be stacked with SafeDecoding and post-generation filters; kernel and scheduling strategies can be ported across inference, finetuning, and quantized low-precision deployment scenarios.

A plausible implication is that the architectural and methodological flexibility instantiated in current FlexLLM systems and libraries provides a template for unifying security, efficiency, and hardware realization in a rapidly evolving LLM deployment landscape.