- The paper introduces a cryptographically-rooted, attestation-based design that leverages modern TEEs to secure Kubernetes workloads.
- The paper outlines a modular architecture featuring Confidential VMs, Certificate Distribution Service, and a raTLS mesh to enforce secure intra-cluster communications.
- The paper demonstrates compatibility with managed Kubernetes services by isolating trust to hardware vendors and measured software, mitigating risks from untrusted control planes.
C8s: A Confidential Kubernetes Architecture
Overview and Motivation
The C8s architecture formalizes a cryptographically-rooted, attestation-based approach for delivering confidentiality and integrity guarantees for Kubernetes workloads, suitable for deployment on untrusted, third-party infrastructure including all major managed Kubernetes offerings (EKS, GKE, AKS). The architecture is predicated on leveraging modern Trusted Execution Environments (TEEs), namely AMD SEV-SNP, Intel TDX, and NVIDIA GPU Confidential Computing, to establish a trust boundary around Confidential VMs (CVMs) that is measurable, attestable, and provable to independent verifiers.
C8s departs markedly from both contractual assurances (e.g., compliance certifications, licensing) and runtime access controls by rendering such mechanisms unnecessary for workload confidentiality within the defined trust model. Instead, C8s shifts the trust anchor to hardware vendors and the attested TEE codebase, excluding cloud providers, node OS, and Kubernetes control planes from the boundary of trust.
Threat Model, Assets, and Adversary Scope
C8s explicitly protects:
- Workload inputs/outputs (e.g., inference queries, analytics, financial records)
- Proprietary artifacts (e.g., model weights, datasets, cryptographic material)
- Intermediate computational state
The trust boundary encompasses only TEEs, hardware vendors (AMD, Intel, NVIDIA), the attested code, and the Certificate Distribution Service (CDS)—it does not include cloud operators, control plane, storage, or Kubernetes administrators. The model assumes physical hardware hosts as honest for non-invasive attacks, but excludes physical access attacks as shown feasible by recent works (e.g., TEE.Fail, Battering RAM, BadRAM).
C8s addresses attacks such as hypervisor/host-side memory snooping, host-initiated tampering/snapshots, code substitution, mitm, unauthorized container scheduling, and key exfiltration from privileged hosts. It intentionally does not claim to mitigate side channels, DoS, application-layer vulnerabilities, application-layer extraction, or physical probing.
Architecture and Component Composition
The architecture is centered around establishing a verifiable, cryptographically enforced trust boundary—defined strictly by hardware measurement, TEE context, and an attestation chain verifiable by third parties. The design is modular and applies a repeatable, hierarchical set of controls ("secure the runtime, measure, bind, verify, secure egress") to each component intended to be trusted.
Core architectural components:
- CVM node image: Attested VM image supporting hardware-mediated encryption, measurement, and (on GPUs) combined CPU/GPU attestation.
- Certificate Distribution Service (CDS): The exclusive attestation verifier and certificate authority inside the trust domain, responsible for verifying measurements, issuing (per-pod/node) raTLS mesh certificates, manifest signing, and attestation-gated secret brokering.
- Attestation service: In-TEE DaemonSet/per-pod service generating measurement-linked reports for CDS appraisal.
- NRI image policy enforcer: Node Resource Interface hook enforcing container allow-list policies at container launch, independent of the control plane.
- Kettle (attested build system): A TEE-based reproducible image build pipeline, producing attestable outputs.
- raTLS mesh: An overlay encrypting all pod-to-pod traffic, using ephemeral, per-pod credentials issued by CDS post attestation.
- Ingress router(s): Attested CVM-based network ingress, terminating external TLS in-TEE, supporting both pass-through and OHTTP/multi-recipient encrypted forwarding.
- Secrets Manager Proxy: Confidential broker allowing fine-grained attestation-gated access to secrets managers.
C8s is deployment-flexible—supporting node-level and pod-level CVM boundaries. Node-level isolation preserves standard kernel-based separation across shared hardware, whereas pod-level boundaries provide maximal tenant isolation via per-pod hardware context.
Attestation, Certificate Distribution, and Key Brokering
The CDS is the definitive verifier and root of trust for all in-cluster identities. Each pod (or node) presents attestation reports tied to its runtime's ephemeral public key. The CDS validates measurement digests, endorsement chains, and policy compliance before certificate issuance. Key release is strictly contingent on attestation; keys are brokered using hybrid modes supporting both direct (plaintext-in-flight, non-persistent in CDS) and wrapped (only ciphertext transiting CDS, preferencing crypto-agility via HPKE/AGE) schemes. The CDS never holds application secrets at rest, minimizing its blast radius.
Short-lived certificates with automated rotation eliminate long-lived credential risk, with certificate renewal tightly bound to re-attestation. This approach obviates the need for explicit revocation channels; measurement or TCB floor changes revoke access at certificate renewal.
Network Encryption and Secure Routing
All intra-cluster traffic is enforced via a pod-level raTLS overlay mesh anchored with per-pod credentials. Identity is tied directly to TEE attestation, closing gaps in traditional mTLS designs which rely on CA issuance semantics only. This ensures that only attested, policy-compliant pods can participate in the mesh, fully mitigating mitm and network compromise by untrusted infrastructure.
Ingress routing is similarly secured: an attested CVM-based ingress router terminates public TLS endpoints, exposes attestation metadata to clients, and routes internal traffic over the mesh. An Encrypted Ingress Router mode enables full opaque routing using recipient stanzas and multi-recipient encryption—payloads never decrypted until reaching an attested destination pod.
Managed Kubernetes Compatibility and Control Plane Minimization
A core architectural achievement is the practical accommodation for untrusted, unmeasured Kubernetes control planes typical of managed offerings (EKS, GKE, AKS). All confidentiality guarantees are enforced solely by attested components (CVMs, CDS, NRI enforcer, raTLS mesh). The control plane’s reach is limited to liveness and orchestration; a compromised/malicious control plane cannot exfiltrate data, inject unauthorized code, or intercept traffic without detection and policy enforcement failure.
Self-hosted clusters can, optionally, bring the control plane within the CVM trust boundary, but this is not required for practical confidentiality.
Operational Deployment, Trade-offs, and Limitations
Per-pod CVMs offer hard multi-tenancy at hardware level, fitting high-compliance and multi-tenant workloads, but incur higher resource overhead and VM cold-start latency (especially when integrating with managed cloud VMs). Node-level CVMs support resource efficiency and GPU sharing, but only provide kernel-level isolation between pods.
Operational complexity includes deploying the CDS, integrating Kettle pipeline for image builds, and managing per-boundary attestation orchestration. Importantly, existing application code and deployment pipelines need minimal modification; Kubernetes workloads inherit C8s guarantees transparently.
GPU Confidential Computing is only available on compatible hardware (post-NVIDIA H100); systems lacking such support remain vulnerable at the VRAM layer. Side channels, runtime vulnerabilities, and physical attacks on silicon are explicitly not addressed.
Theoretical and Practical Implications; Future Work
C8s delivers a unique and practical architecture for high-assurance confidential computing in cloud-native clusters without the infeasible requirement to trust or modify cloud-managed Kubernetes control planes. It enables secure AI/ML inference, fine-tuned model use, and privacy-compliant analytics in untrusted environments, with strong cryptographic and hardware-rooted guarantees.
From a theoretical standpoint, C8s embodies a separation of concerns critical to future cloud infrastructure: operational liveness and orchestration can be delegated to untrusted entities without compromising the confidentiality or integrity of workload computation and data.
Open directions include:
- Attested build pipeline publication detailing build reproducibility properties
- Finer-grained mesh termination for pod-level boundaries
- Scale-out mechanisms for per-workload measured policy enforcement and efficient key rotation
- Extension of allow-list policies to consistency-transparent logs
- Further reduction in attestation freshness windows under hardware throughput constraints
Conclusion
C8s demonstrates that attestation-rooted, hardware-enforced confidentiality is attainable for generalized Kubernetes workloads, with robust compatibility across major managed Kubernetes services and deployment models. By confining trust to hardware and specifically measured software artifacts inside TEEs, and by rigorously excluding the control plane and supporting infrastructure from the trust domain, C8s provides a foundation for high-assurance, confidential cloud workloads suitable for sensitive AI, analytics, and data processing tasks, with extensible design and well-founded cryptographic semantics.
