Multi-Granularity Instruction Set

• Multi-Granularity Instruction Sets are architectural paradigms that structure instructions at varying semantic scales to enable fine control and adaptive resource reuse.
• They employ a methodology of constructing a base instruction set with domain-specific extensions, evaluated using metrics like the Reusability and Extra Cost Factors.
• These systems enhance heterogeneous and parallel computing by supporting dynamic adaptation, efficient scheduling, and scalability in applications from ASIPs to multimodal AI models.

A multi-granularity instruction set is an architectural and algorithmic approach that enables fine-grained control, adaptability, and efficiency in complex computing and machine learning systems by structuring instructions and their implementations at different semantic or functional scales. This paradigm encompasses approaches ranging from the design of Application-Specific Instruction-set Processors (ASIPs) to advanced multimodal LLMs and domain-adapted computing architectures, addressing challenges such as resource reuse, extensibility, heterogeneous workloads, and complex instruction following. Below, the key concepts, methodologies, and research directions underpinning multi-granularity instruction sets are surveyed, with reference to prominent studies and concrete technical practices drawn from the literature.

1. Formalization and Metrics for Multi-Granularity Instruction Set Design

A multi-granularity instruction set is often formalized through the selection of a base set of reusable instructions shared across a set of target applications, with the remainder of the instruction implementation tailored as application- or domain-specific extensions. The methodology typically involves:

• Base Instruction Set Construction: Derived as the intersection of the instruction sets required by the target application set (either intra-domain or inter-domain).
• Instruction Set Extension: Additional (‘extra cost’) instructions are selectively integrated to support requirements absent from the base set.

The effectiveness and implications of such partitioning are quantitatively assessed using metrics introduced in (Ragel et al., 2014):

$$\text{Reusability Factor} = \frac{(\# \text{instructions in Base ISA}) \times 100}{\# \text{instructions in mASIP}}$$

$$\text{Extra Cost Factor} = \frac{(\# \text{instructions for a given app.} - \# \text{instructions in Base ISA}) \times 100}{\# \text{instructions in mASIP}}$$

These formulas enable rigorous evaluation of design reusability and marginal cost for extending support to additional applications or domains, providing actionable signals when integrating applications of varying heterogeneity.

2. Domain-Aware Granularity and Instruction Set Reuse

The effective granularity of instruction selection is heavily domain dependent. Empirical analysis (Ragel et al., 2014) across ARM-Thumb and PISA ISAs using benchmarks (MiBench, MediaBench, SPEC2006) demonstrates:

• Intra-domain Applications: Share a high proportion of instructions, yielding a Reusability Factor of ca. 59% (ARM-Thumb), 49% (PISA) and a relatively low Extra Cost Factor (~24% ARM-Thumb).
• Inter-domain Applications: Exhibit substantially fewer shared instructions, leading to a lower Reusability Factor (~28% ARM-Thumb) and a much higher Extra Cost Factor (~67% ARM-Thumb).

This dichotomy is robust across architectures, confirming that intra-domain ASIPs can be highly efficient, while multi-domain support incurs a significant increase in instruction complexity, area, and NRE cost. The implication is that effective multi-granularity instruction sets must account for the semantic cohesion of the application set, which directly determines the tractability of finding a large, reusable base.

3. Multi-Granularity in Heterogeneous and Parallel Systems

In heterogeneous computing scenarios, multi-granularity manifests as the concurrent support for multiple levels of algorithmic or system parallelism:

• HPVM (Srivastava et al., 2016): Utilizes a hierarchical dataflow graph to structure parallel program semantics at multiple levels—task, coarse-grain data, fine-grain data, and pipelined parallelism. Instructions are encoded as LLVM intrinsics, abstracting the representation of node granularity, data replication (all-to-all, one-to-one), flexible scheduling, and tiling—essential for performance portability across CPUs, GPUs, and SIMD accelerators.
• Instruction Composability (Yang et al., 28 Jun 2024): The Composable Instruction Set (CIS) architecture defines composition at both the spatial level (decomposing computations into resource-centric instructions) and the temporal level (explicit looping and synchronization operators). This enables hardware platforms to schedule parallel loops and stream-processing tasks with minimized control overhead and near-optimal PE utilization.
• Resource-centric Programming: Each processing resource operates under its own local finite-state controller, and additions (e.g., new functional units) involve only extending relevant instruction subsets, enabling heterogeneous architectures to be naturally constructed and extended for domain-specific accelerators.

4. Dynamic and Adaptive Multi-Granularity Implementations

Advanced architectures employ on-the-fly adaptation of instruction set granularity and dynamic resource tailoring:

• Dynamically Reconfigurable ISAs (Papaphilippou et al., 2022): Fast-reconfigurable FPGAs embedded in CPU cores allow the active instruction set to shift according to workload—groups of instructions are loaded as needed, with reconfiguration hidden behind software abstraction layers. An instruction disambiguator (acting as an L0 cache) determines if FPGA-based instructions are loaded, fetching bitstreams on demand and drastically reducing silicon area without hardening infrequently used instructions.
• Multi-Width Instructions (Chen et al., 2022): Microprocessors with support for compressed (16-bit) and full-width (32-bit) instructions enable fine-grained control over memory footprint and performance. Instruction decoding logic utilizes opcode bit signatures to switch modes, while clock period adaptation (based on per-instruction delay) further tunes execution efficiency via dynamic phase shifting mechanisms.

5. Multi-Granularity Instruction Sets in Multimodal and LLM Systems

Instruction sets that operate at multiple semantic or structural levels have been extended into the design of multimodal LLMs (LMMs):

• Multi-Granularity Visual Flows (Zhao et al., 25 Jun 2024): In MG-LLaVA, images are simultaneously processed at low resolution (global scene context) and high resolution (local fine details) using separate visual encoders. Features are fused via a Conv-Gate network, and object-level tokens (from bounding-box detectors and RoI Align) capture instance-level semantics. The modular hierarchy aligns with instruction-level granularity control in vision tasks.
• Multi-Granularity Segmentation and Captioning (Zhou et al., 20 Sep 2024): MGLMM generates segmentation masks and captions at panoptic, instance, and fine-grained sub-part levels, controlled via extended token sets ([SEG], <p>, </p>) and a unified data format. The automation of hierarchical segmentation and textual alignment supports instruction-driven adjustment of output granularity across tasks.
• Complex Instruction Following (Huang et al., 17 Feb 2025): The MuSC framework introduces coarse-grained (constraint-level) and fine-grained (token-level) self-contrast in LLM training by generating positive/negative instruction–response pairs through constraint decomposition and dynamic token-aware preference optimization. This design substantially improves LLM fidelity on complex, multi-constraint instructions.

6. Extensions Beyond Classical Architectures: Quantum and Tool-Augmented Systems

Multi-granularity principles extend into unconventional architectures:

• Quantum Processing Units (Britt et al., 2017): The debate between RISC-like and CISC-like QPU ISAs centers on the tension between fixed-width packetization for pipelining (imposing limits on addressable qubits) and domain-specific compound instructions (e.g., quantum error correction, quantum Fourier transform). The hybridization of instruction set strategies is necessary for future scalability and classical-quantum integration.
• Tool-Augmented LLMs (Wu et al., 23 Sep 2024): MGToolBench and ToolPlanner introduce instruction granularity at the category, tool, and API levels. The RL framework incorporates reward stratification (Pass, Match), path planning for solution path generation, and tree-based reasoning with multi-turn tool calls. Experimental results confirm markedly improved adherence to user intent and task success rates in variable-instruction granularity scenarios.

7. Implications, Trade-offs, and Research Outlook

Adopting a multi-granularity instruction set introduces several key system design trade-offs:

• Efficiency vs. Flexibility: In domain-constrained ASIPs, high instruction reusability reduces design and runtime cost, while highly heterogeneous environments may require overhead for dynamic composition or runtime extension.
• Compiler and Scheduling Complexity: Resource-centric, composable ISAs demand sophisticated scheduling and code generation algorithms, often requiring constraint programming or advanced instruction schedulers.
• Program Size and Abstraction Overhead: Replicating loop structures and granularity control across multiple resources may increase program size, but, as demonstrated, absolute code counts remain tractable for the data streaming and kernel-dense workloads targeted (Yang et al., 28 Jun 2024).
• Interoperability: Architecture independence of metrics and formalizations (Ragel et al., 2014, Srivastava et al., 2016), unified annotation and formatting schemes (Zhou et al., 20 Sep 2024), and abstraction layers for quantum-classical hybrid systems (Britt et al., 2017) are paramount to ensure broad applicability and future extensibility.

Multi-granularity instruction sets have become foundational to both classical and emerging computational paradigms by facilitating principled resource reuse, supporting hierarchical control, and enabling scalable, efficient design in diverse architecture and application scenarios. Their continued evolution will underpin advances in specialized hardware, large-scale AI, and complex hybrid machine reasoning systems.