---
title: ProfilingAgent for Hardware-Aware Model Compression
url: https://www.emergentmind.com/topics/profilingagent
type: topic
---

# ProfilingAgent for Hardware-Aware Model Compression

ProfilingAgent is a profiling-guided, multi-agent system that uses large language models as decision-making components to automate hardware-aware pruning and quantization of vision foundation models. It integrates static metrics such as MACs and parameter counts with dynamic signals such as per-layer latency and memory, and uses those signals to generate architecture-specific compression strategies rather than uniform, architecture-agnostic heuristics. The system was evaluated on ImageNet-1K, CIFAR-10, and CIFAR-100 with ResNet-101, ViT-B/16, Swin-B, and DeiT-B/16, where pruning maintained competitive or improved accuracy and quantization achieved up to 74% memory savings with consistent inference speedups of up to 1.74 times faster [2509.05584].

## 1. Problem setting and conceptual basis

ProfilingAgent addresses a specific limitation of conventional model compression pipelines. Modern foundation models, including CNNs such as ResNet-101 and transformer-based vision models such as ViT-B/16, Swin-B, and DeiT-B/16, have tens to hundreds of millions of parameters and require billions of MACs per inference. Traditional compression methods such as structured pruning and post-training quantization are often applied with uniform or static heuristics, for example L1/L2-norm pruning or uniform PTQ, and therefore ignore architectural and runtime heterogeneity. Different layers contribute differently to latency, memory usage, and accuracy sensitivity, and those contributions depend on the deployment substrate rather than on parameter count alone [2509.05584].

The defining claim of ProfilingAgent is that profilers already expose rich per-layer metrics, but those signals are rarely integrated into automated compression workflows. ProfilingAgent closes that loop by elevating profiling to a first-class input to agentic reasoning. The system reasons jointly over layer names and types, MACs and parameter counts, per-layer latency and memory, and overall model performance metrics, then converts those observations into machine-executable pruning and quantization plans. This places profiling not after optimization as a diagnostic artifact, but inside optimization as the primary basis for decision making.

This also shifts the optimization target from uniform sparsification to a hardware-aware trade-off. In the iterative pruning component, model comparison follows a hierarchical objective: first maximize accuracy, then minimize parameter count, and finally minimize latency. A plausible implication is that ProfilingAgent is designed less as a pure compression engine than as a profile-guided selector over a three-way Pareto surface of accuracy, size, and runtime.

## 2. Multi-agent architecture

ProfilingAgent is organized into three principal components: **Profiling**, **Optimization**, and **Iterative Pruning**. Within these components, the system includes the Acquisition Agent, Input Shape Resolver Agent, ProfilingAgent, Analysis Agent, Pruning Agent, Quantization Agent, Evaluation Agent, and Iterative Pruning Agent [2509.05584].

| Component | Agents | Function |
|---|---|---|
| Profiling | Acquisition, Input Shape Resolver, ProfilingAgent, Analysis | Retrieve model, resolve input shape, collect metrics, propose strategies |
| Optimization | Pruning, Quantization, Evaluation | Apply compression and benchmark original and modified models |
| Iterative Pruning | Iterative Pruning Agent | Refine pruning across multiple rounds using prior analysis and evaluation |

The **Acquisition Agent** retrieves pretrained models and processors from Hugging Face and configures the `AutoImageProcessor` to match the model’s preprocessing requirements. The **Input Shape Resolver Agent** uses GPT-4o to determine the expected input tensor shape and returns a JSON object with fields `channels`, `height`, `width`, and `sequence_length`. The **ProfilingAgent** collects static metrics with `ptflops` and dynamic metrics with PyTorch Profiler plus manual per-submodule timing. The **Analysis Agent** then reads the profiling report and asks GPT-4o for structured pruning and quantization recommendations in JSON form.

The optimization side is deliberately split. The **Pruning Agent** applies regex-based recommendations using a `DependencyGraph` so that structural pruning remains dependency-safe. The **Quantization Agent** applies post-training dynamic quantization using PyTorch’s `quantize_dynamic`. The **Evaluation Agent** measures top-1 accuracy, average inference latency, parameter count, and memory usage. Finally, the **Iterative Pruning Agent** closes the loop by loading prior analysis and evaluation logs, querying the LLM for a new pruning configuration, applying it, and updating the best model according to the hierarchical objective.

The architecture is notable for separating profiling, reasoning, execution, and evaluation into distinct agent roles. This separation is explicit in the workflow rather than merely conceptual, and it gives ProfilingAgent the character of a modular optimization stack rather than a single monolithic compressor.

## 3. Profiling pipeline and LLM-guided analysis

ProfilingAgent’s profiling stage combines static and dynamic measurement. Static metrics are obtained with `ptflops`, which computes MACs and parameter counts. Dynamic metrics are obtained with **PyTorch Profiler** on CPU and GPU together with manual per-layer timing hooks, yielding per-layer latency, memory usage, tensor shapes, operator-level breakdowns, number of calls, and snapshots of overall inference latency and peak memory. The profiling experiments were run on the NCSA Delta cluster using AMD EPYC 7763 64-core CPUs, NVIDIA A100-SXM4-40GB GPUs, 251 GB RAM, CUDA 11.8, and PyTorch 2.6.0 [2509.05584].

The Input Shape Resolver Agent removes a common source of friction in model profiling. For the model `google/vit-base-patch16-224`, the system prompt asks for `channels`, `height`, `width`, and `sequence_length`, and GPT-4o returns the required JSON specification. This automated shape discovery is important because profiling is only meaningful when the model is exercised under the correct preprocessing and input assumptions.

The Analysis Agent is the system’s main reasoning layer. It constructs a prompt that includes layer names and types, MACs and parameter counts per layer or block, latency and memory statistics, overall model performance metrics, and deployment constraints, and then requests structured recommendations. The returned JSON has separate pruning and quantization sections with fields such as `layer`, `pruning_type`, `pruning_ratio`, `quantization_type`, `dtype`, and `justification`. A representative recommendation for a ResNet model uses a regex such as `encoder\.stages\d+\.layer\.\d+\.convolution`, specifies `"structured"` pruning at ratio `0.2`, and justifies it as reducing `Conv2d` parameters and MACs to improve latency [2509.05584].

This analysis stage is not described as a free-form assistant. It is a constrained planner over structured profiler outputs. The system prompt explicitly binds the LLM to architecture-aware reasoning over measured bottlenecks, which is why ProfilingAgent differs from compression methods that merely ask an LLM to “prune the model” without a profiling substrate.

## 4. Optimization mechanisms: structured pruning, dynamic quantization, and iterative refinement

ProfilingAgent supports two optimization mechanisms. The first is **structured pruning**, defined as removing entire structural units so that tensor shapes remain dense and compatible with standard kernels. For CNNs such as ResNet-101, this means output-channel or filter pruning in `Conv2d` layers. For transformer-based models such as ViT-B/16, Swin-B, and DeiT-B/16, it includes pruning rows of `Linear` layers and pruning attention heads from multi-head self-attention. The Pruning Agent reads regex layer patterns, pruning types, and pruning ratios from the Analysis Agent, constructs a `DependencyGraph`, and applies dependency-safe structural edits to the baseline model [2509.05584].

The second mechanism is **post-training dynamic quantization**. The Quantization Agent applies PyTorch’s `torch.quantization.quantize_dynamic` to `nn.Linear` layers, using either `qint8` or `float16`. In the reported experiments, the LLM consistently recommends full dynamic QInt8 quantization of all linear layers, expressed as a JSON policy such as `"layer": "all"`, `"quantization_type": "dynamic"`, and `"dtype": "qint8"`. Pruning and quantization are described as decoupled but composable: pruning modifies the graph structure, while quantization reduces precision on the original or pruned graph.

The most distinctive mechanism is the **Iterative Pruning Agent**. Its loop begins with an initial pruned model \(M_0\), evaluates its latency and accuracy, and stores it as the current best model \(M_{\text{best}}\). For each subsequent iteration \(i\), the agent loads prior profiling data \(P\), previous analysis \(A_{i-1}\), and previous evaluation \(E_{i-1}\), queries the LLM for a new pruning configuration \(R_i\), applies that configuration to produce \(M_i\), evaluates \((\mathrm{lat}_i, \mathrm{acc}_i)\), and updates the best model according to the lexicographic criterion
\[
\max \mathrm{Acc}(M') \quad \text{then} \quad \min \text{Params}(M') \quad \text{then} \quad \min \mathrm{Latency}(M').
\]

Comparative experiments with GPT-4o and GPT-4-Turbo highlight that reasoning quality materially changes pruning behavior. With GPT-4o, ProfilingAgent reports \(\Delta\)Acc of \(-1\%\) on DeiT-B/16, \(+1\%\) on ResNet-101, \(+2\%\) on ViT-B/16, and \(-1\%\) on Swin-B, whereas GPT-4-Turbo produces more aggressive pruning with \(-4\%\) on DeiT-B/16 and \(-14\%\) on ResNet-101. The paper interprets this as evidence that iterative pruning quality depends on the LLM’s ability to reason over profiler outputs rather than merely emit pruning ratios [2509.05584].

## 5. Empirical evaluation

ProfilingAgent was evaluated on ImageNet-1K, Imagenette, CIFAR-10, and CIFAR-100 using ResNet-101, ViT-B/16, Swin-B, and DeiT-B/16. The pruning baselines were L1-norm structured pruning, L2-norm structured pruning, and random structured pruning at fixed global ratios of 1%, 10%, and 20%. The quantization baseline was ONNX Runtime dynamic PTQ on MatMul and Gemm nodes. The reported evaluation metrics include top-1 accuracy, parameter count, memory footprint, average inference latency, and relative speedup [2509.05584].

On **Imagenette pruning**, ProfilingAgent reports \(\Delta\)Acc of \(-1\%\) with 86.6M → 80.9M parameters for DeiT-B/16, \(+1\%\) with 44.5M → 43.0M for ResNet-101, \(-1\%\) with 87.8M → 81.1M for Swin-B, and \(+2\%\) with 86.6M → 78.1M for ViT-B/16. By contrast, at 20% global pruning, L1/L2 baselines cause severe drops on ResNet-101, including \(-17\%\) accuracy. The paper’s interpretation is that ProfilingAgent achieves comparable or better parameter reduction with much smaller accuracy loss and, in some cases, improved generalization on smaller datasets.

On **CIFAR-10 and CIFAR-100 pruning without post-pruning fine-tuning**, the reported results remain small in magnitude but consistent across architectures. ResNet-101 moves from 84.10 to 84.20 on CIFAR-10 with 48.81M → 47.30M parameters and from 83.30 to 78.40 on CIFAR-100 with 42.71M → 41.20M. ViT-B/16 moves from 98.50 to 97.90 on CIFAR-10 with 87.12M → 81.45M and from 90.00 to 88.10 on CIFAR-100 with 85.88M → 83.04M. Swin-Base and DeiT-B/16 show similar behavior, typically with modest memory reduction and sub-1% degradation on several settings.

On **ImageNet-1K quantization**, the most prominent gains come from dynamic quantization rather than pruning. For ViT-B/16, ONNX reports 74.1% memory reduction and 0.3869 → 0.2179 s average inference, whereas ProfilingAgent reports 74.2% memory reduction, 0.2320 → 0.1316 s, a 1.66× speed gain, and \(\Delta\)Acc of 0%. For Swin-Base, ProfilingAgent reports 74.7% memory reduction, 0.2442 → 0.1556 s, a 1.46× speed gain, and \(\Delta\)Acc of 0%. For DeiT-B/16, it reports 74.2% memory reduction, 0.2312 → 0.1317 s, a 1.65× speed gain, and \(\Delta\)Acc of \(-1\%\). ResNet-101 shows smaller memory gains, 3.4%, with 0.1211 → 0.1199 s and \(\Delta\)Acc of 0%.

The separate **ImageNet-1K pruning** results clarify the system’s trade-off profile. ViT-B/16 goes from 80.3 to 79.5 with 3.3% memory and parameter reduction and 1.02× speedup; Swin-Base goes from 85.3 to 84.7 with 1.9% reduction and 1.02× speedup; DeiT-B/16 goes from 81.0 to 80.2 with 2.0% reduction and 0.98× speedup. ResNet-101 shows 82.4 → 79.7 with 2.9% reduction and 0.97× slowdown. The paper notes that slight slowdowns on ResNet-101 can occur due to channel misalignment effects after structured pruning, which underscores that pruning does not automatically translate into faster kernels on real hardware.

## 6. Position in the literature and limitations

ProfilingAgent sits at the intersection of model compression, profiling-guided systems optimization, and agentic LLM systems. In adjacent work, profiling-centered agents or profiling layers serve different functions: unified provenance for agentic workflows and downstream reliability analysis in PROV-AGENT [2508.02866], causal graph tracing for post-hoc root-cause localization in deployed multi-agent systems in AgentTrace [2603.14688], self-evolving capability profiles for adaptive routing in FlyRoute [2605.22057], behavioral trajectory monitoring for computer-use safety in ProjGuard [2605.13631], and dynamic profiling for tabular workflow synthesis in ProfiliTable [2605.12376]. This suggests that “ProfilingAgent” names not only a specific compression system but also a broader design pattern in which profiling is promoted from passive observability to active control.

Within that landscape, the distinctive contribution of ProfilingAgent is to use profiling traces as the direct substrate for architecture-specific pruning and quantization of vision models. Its contributions are a profiling-guided multi-agent architecture, LLM-driven structured pruning and dynamic quantization, an Iterative Pruning Agent with empirical feedback, and empirical validation across multiple backbones and datasets. The system therefore treats profiling as both measurement and policy input, rather than as a retrospective diagnostic.

The paper also identifies several limitations. Collecting extensive per-layer profiling adds overhead. The system relies on proprietary LLMs, especially GPT-4o, which raises reproducibility and availability concerns. Profiling results are hardware-specific, so strategies learned on A100/H200 GPUs and EPYC/Intel CPUs may not transfer directly without re-profiling. The reported methods are entirely post-training and do not integrate pruning-aware training or quantization-aware training. Pruning and quantization are applied sequentially and evaluated mostly separately, so joint optimization remains limited. A plausible implication is that ProfilingAgent is best understood as a strong proof of concept for profiling-guided agentic optimization rather than a final account of end-to-end hardware–model co-design.

The broader significance of the system lies in its inversion of the usual relationship between profiling and optimization. Rather than hand-designing compression heuristics and then using profilers to validate them, ProfilingAgent uses profilers to generate the heuristics themselves. That formulation is likely to remain relevant even if the specific agent stack, backbone models, or compression operators change.

Source: https://www.emergentmind.com/topics/profilingagent