LLM-Rosetta: A Hub-and-Spoke Intermediate Representation for Cross-Provider LLM API Translation

Abstract: The rapid proliferation of LLM providers--each exposing proprietary API formats--has created a fragmented ecosystem where applications become tightly coupled to individual vendors. Switching or bridging providers requires $O(N^2)$ bilateral adapters, impeding portability and multi-provider architectures. We observe that despite substantial syntactic divergence, the major LLM APIs share a common semantic core: the practical challenge is the combinatorial surface of syntactic variations, not deep semantic incompatibility. Based on this finding, we present LLM-Rosetta, an open-source translation framework built on a hub-and-spoke Intermediate Representation (IR) that captures the shared semantic core--messages, content parts, tool calls, reasoning traces, and generation controls--in a 9-type content model and 10-type stream event schema. A modular Ops-composition converter architecture enables each API standard to be added independently. LLM-Rosetta supports bidirectional conversion (provider-to-IR-to-provider) for both request and response payloads, including chunk-level streaming with stateful context management. We implement converters for four API standards (OpenAI Chat Completions, OpenAI Responses, Anthropic Messages, and Google GenAI), covering the vast majority of commercial providers. Empirical evaluation demonstrates lossless round-trip fidelity, correct streaming behavior, and sub-100 microsecond conversion overhead--competitive with LiteLLM's single-pass approach while providing bidirectionality and provider neutrality. LLM-Rosetta passes the Open Responses compliance suite and is deployed in production at Argonne National Laboratory. Code is available at https://github.com/Oaklight/LLM-rosetta.

• The paper presents a provider-neutral intermediate representation that reduces cross-provider LLM API translation complexity from O(N²) to O(N).
• It details a modular hub-and-spoke converter architecture that ensures bidirectional, lossless translation and robust streaming with sub-100 μs overhead.
• Empirical evaluations demonstrate accurate semantic fidelity, scalable integration, and successful production deployment in real-world LLM gateways.

LLM-Rosetta: A Hub-and-Spoke Intermediate Representation for Cross-Provider LLM API Translation

Introduction

The fragmentation of cloud-based LLM APIs, each with their proprietary schemas and payload conventions, has resulted in a combinatorial burden for interoperability across providers. The $O(N^2)$ growth in required bilateral adapters impedes portability and prevents practical multi-provider architectures in research and production environments. "LLM-Rosetta: A Hub-and-Spoke Intermediate Representation for Cross-Provider LLM API Translation" (2604.09360) presents a provider-neutral, semantically-rich Intermediate Representation (IR) and a modular conversion architecture that reduces the integration challenge to $O(N)$, aligning with established compiler and data interchange precedents.

Problem Framework and Existing Limitations

While syntactic divergence is substantial—including differences in message unit granularity, content typing, tool invocation mechanisms, and streaming event schemas—LLM APIs share a semantic foundation regarding role-tagged messages, content fragments, tool calls, and generation controls. Prior approaches such as LangChain or LiteLLM have only partially addressed the abstraction challenge: LangChain embeds adaptation logic at the application level, obscuring low-level differences but not resolving them; LiteLLM and similar SDK-level proxies adopt a single-provider schema as a lingua franca, losing features and failing to provide bidirectional or lossless conversion.

LLM-Rosetta's principled approach contrasts with these by focusing on pure API payload translation. Rather than privileging any single provider's conventions or relying on informal, brittle mapping code, it establishes a hub-and-spoke architecture around a carefully designed IR capable of semantic round-trips.

Intermediate Representation Design

The IR is engineered for semantic completeness, provider neutrality, bidirectional fidelity, incremental extensibility, and streaming compatibility.

Figure 1: Overview of the IR schema, highlighting the modular decomposition of content, messages, tools, and configuration, with containment relationships via typed arrows.

LLM-Rosetta's IR specifies:

• 9 content part types: Text, image, audio, files, tool calls, tool results, reasoning traces, refusals, and citations—each with provider-agnostic structure and optional metadata for round-trip integrity.
• Role-tagged messages: Explicit System, User, Assistant, and Tool roles, each aggregating content parts with timestamping, IDs, and streaming state.
• Typed tool definition structures: Including function schema, configurable selection logic, and tool call configuration.
• Rich generation configuration: Capturing all relevant sampling/decoding params (temperature, top-p, max tokens, etc.), response format, streaming toggles, and reasoning controls.
• Request/response wrappers: Modular encapsulation for payloads and completion statistics.

Streaming is modeled via a schema of ten event types, supporting both chunk-level delta events and higher-level content lifecycle notifications. This standardizes disparate SSE mechanisms while allowing converter-specific ordering and buffering policies.

Hub-and-Spoke Converter Architecture

To maximize extensibility and maintainability, LLM-Rosetta decomposes provider-specific adaptation logic into four orthogonal modules: ContentOps, MessageOps, ToolOps, and ConfigOps. Each provider integration becomes a composition of these reusable domains, rather than a monolithic, fragile implementation.

Figure 2: LLM-Rosetta’s hub-and-spoke design for multi-provider interoperability. Each converter translates bidirectionally between a proprietary format and the central IR, enabling $O(N)$–scalable cross-provider translation.

Separation of concerns ensures:

• ContentOps isolate low-level quirks (e.g., base64 encoding, nesting conventions);
• MessageOps handle role mapping and conversational structuring;
• ToolOps capture tool definition and invocation subtleties;
• ConfigOps manage provider-specific parameters, including nonstandard settings.

Testability is enhanced by the ability to independently validate these modules (over 1300 unit and integration tests), and the architecture accommodates rapid adaptation to emerging API standards or provider extensions.

Implementation and Streaming Semantics

LLM-Rosetta is implemented in Python using TypedDict for static type assurance and discriminated unions. The library provides primary entry points for semantic translation in both directions—request and response, batch and streaming—with a ConversionContext for fidelity guarantees (lossless "preserve" mode vs. provider-neutral "strip" mode).

Stream translation handles the canonicalization of disparate provider event schemas into the IR's ten event types, enforcing correct ordering, payload accumulation, and finish notification consistency. The streaming infrastructure tracks context for block indices, tool call buffers, and deferred events, ensuring real-time, robust transformation between arbitrary provider pairs.

Empirical Evaluation

Round-Trip and Cross-Provider Fidelity

Unit and integration tests demonstrate lossless round-trips across all supported providers in preserve mode—structural equality is maintained modulo insignificant JSON representation details. The suite also confirms correct streaming event order and content assembly across edge cases and complex multi-turn, tool-rich conversations.

Cross-provider translation tests validate semantic equivalence: although provider-specific metadata (such as Anthropic's caching or Google's grounding metadata) cannot always be projected, all core conversational, reasoning, and tool call semantics are preserved. The limitations of cross-provider fidelity are quarantined to features with zero expressible analog in the target schema.

Performance

Conversion overhead is consistently below 80 μs per round-trip for complex payloads and below 25 μs for simple interactions (median latencies). These figures are on par with or outperform SDK-level proxies despite the richer, bidirectional translation and superior fidelity. Streaming scenarios incur no measurable per-chunk latency beyond these figures, preserving practical throughput in low-latency deployments.

Deployment

Deployment in Argonne National Laboratory's production LLM gateway (Argo-Proxy) eliminated thousands of lines of ad hoc translation code, enabling unified support for evolving and heterogeneous provider APIs with a single, maintainable converter stack. The deployment cycle surfaced edge cases, leading to maturity in context management and event ordering logic.

Limitations and Potential Extensions

Coverage is currently restricted to four major API standards, which subsumes the majority commercial providers but omits non-chat modalities and potential future divergences (e.g., embeddings, batch operations). Backward compatibility and conformance testing for all provider permutations remains a challenge, given the pace of schema evolution in the LLM API landscape.

The IR’s union-of-features design, together with the provider_extensions escape hatch, is expected to scale to most future wire protocol additions, though genuinely novel modalities (e.g., fully agentic workflow traces, real-time control for non-text media) may eventually require substantial IR restructuring.

Implications and Future Perspectives

LLM-Rosetta’s main theoretical implication is that the LLM API landscape's divergence is predominantly syntactic, not semantic. A well-engineered IR, grounded in the semantic intersection of provider schemas, renders $O(N)$ hub-and-spoke translation both practical and robust. This reframes interoperability from an intractable maintenance liability to a tractable extension problem, enabling experimentation, model ensembling, and vendor-agnostic deployment with minimal incremental cost.

Practically, LLM-Rosetta facilitates the construction of gateways, orchestration layers, and evaluation harnesses that are decoupled from API churn. Future developments should target integration of non-chat modalities and formalization of cross-provider conformance suites. The approach will also inform protocol negotiation for agent mesh frameworks and emerging model context protocols.

Conclusion

LLM-Rosetta (2604.09360) establishes that the core semantics of LLM communication—across OpenAI, Anthropic, Google, and OpenAI Responses APIs—can be faithfully and efficiently captured in a provider-neutral IR, supporting <100 μs translation for full payloads, robust streaming, and lossless round-trips. Its modular converter architecture, rigorous evaluation, and production use cases position it as a canonical format translation layer for the evolving LLM ecosystem.