Kube-virt - k8s in k8s

From UVOO Tech Wiki
Revision as of 15:11, 2 August 2026 by Busk (talk | contribs) (Created page with "One of the most popular and powerful use cases for KubeVirt, often referred to as **Kubernetes-in-Kubernetes** or **Tenant Clusters**. In this architecture, your underlying b...")
(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to navigation Jump to search

One of the most popular and powerful use cases for KubeVirt, often referred to as Kubernetes-in-Kubernetes or Tenant Clusters.

In this architecture, your underlying bare-metal Kubernetes cluster acts as the infrastructure management layer (the control plane for hardware), and KubeVirt provisions VMs that join together to form separate, isolated tenant Kubernetes clusters.

Why Do This? (Key Benefits)

  • Multi-Tenancy & Hard Isolation: Containers share the host kernel. If you need strict security boundaries between teams or environments (e.g., Prod vs. Dev), VMs provide hardware-level virtualization isolation.
  • Infrastructure-as-Code (GitOps): You can manage the lifecycle of entire target Kubernetes clusters declaratively using standard kubectl commands or GitOps pipelines (ArgoCD/Flux).
  • Cluster API (CAPI) Integration: Using the Cluster API Provider KubeVirt (CAPK), you can declare a target Kubernetes cluster in YAML, and CAPI + KubeVirt will automatically spin up the VMs, run cloud-init to install Kubernetes on them, and join them into a fully functional tenant cluster.
  • Testing & Sandbox Environments: It allows developers to spin up ephemeral, full-featured Kubernetes clusters (with custom kernel modules or CNI configurations) without needing real physical hardware.

How the Architecture Looks

┌────────────────────────────────────────────────────────────────────────┐
│                      BARE-METAL KUBERNETES CLUSTER                     │
│  (Management Cluster: Physical Servers + Storage + KubeVirt Operator)  │
│                                                                        │
│  ┌──────────────────────────────────────────────────────────────────┐  │
│  │                    TENANT KUBERNETES CLUSTER                     │  │
│  │                                                                  │  │
│  │   ┌────────────────────┐              ┌────────────────────┐     │  │
│  │   │ KubeVirt VM 1      │              │ KubeVirt VM 2      │     │  │
│  │   │ (Tenant Control)   │              │ (Tenant Worker)    │     │  │
│  │   │                    │              │                    │     │  │
│  │   │ ┌────────────────┐ │   Network    │ ┌────────────────┐ │     │  │
│  │   │ │ Tenant K8s API │ ◄────────────────►│ Tenant Pods    │ │     │  │
│  │   │ └────────────────┘ │ (Overlay/VLAN) │ └────────────────┘ │     │  │
│  │   └────────────────────┘              └────────────────────┘     │  │
│  └──────────────────────────────────────────────────────────────────┘  │
└────────────────────────────────────────────────────────────────────────┘

Typical Workflow

  1. Deploy Bare-Metal K8s: You set up physical servers with a Kubernetes distribution (e.g., Talos, RKE2, OpenShift, or upstream K8s).
  2. Install KubeVirt: Deploy the KubeVirt operator on the bare-metal cluster.
  3. Use Cluster API (CAPK): Apply a manifest defining a tenant cluster (e.g., 1 Control Plane VM, 3 Worker VMs).
  4. Bootstrapping via cloud-init: KubeVirt boots the VMs, and cloud-init runs scripts (like kubeadm join or k3s) to wire the nodes into the new nested cluster.

Things to Consider

  • Nested Virtualization / Overhead: Ensure hardware-assisted virtualization (/dev/kvm) is available on your bare-metal nodes so KubeVirt runs near native speed.
  • Networking: You’ll need a CNI or plugin like Multus or OVN-Kubernetes on the bare-metal cluster to assign dedicated IPs or VLANs to the VM nodes so the tenant cluster can route traffic cleanly.
Retrieved from "https://tech.uvoo.io/index.php?title=Kube-virt_-_k8s_in_k8s&oldid=5735"