diff --git a/docs/02-General/04-flavors-of-metalstack.md b/docs/02-General/04-flavors-of-metalstack.md index 18c0671f..de0bd7d5 100644 --- a/docs/02-General/04-flavors-of-metalstack.md +++ b/docs/02-General/04-flavors-of-metalstack.md @@ -22,12 +22,12 @@ If you want more features, keep reading. [Gardener](https://gardener.cloud/) is an open-source managed Kubernetes service. It provides a good "batteries-included" developer experience and should be your first choice for a Kubernetes-as-a-service solution. -Gardener is vendor agnostic and can be used with a wide selection of infrastructure providers. One big advantage are its containerized control planes. These allow for control planes to not require three machines for each managed cluster, called `Shoot`. This makes operating many smaller clusters more economical, compared to bare-metal control planes. +Gardener is vendor agnostic and can be used with a wide selection of infrastructure providers. One big advantage are its containerized control planes. These allow for control planes to not require three machines for each managed cluster, called `Shoot`. This makes operating many smaller clusters more economical, compared to bare-metal control planes. -We provide support to run Gardener on metal-stack via [Gardener extensions](../05-Concepts/04-Kubernetes/01-gardener.md). This integration is production-hardened, well documented, used by many organizations in production and build on top of the open-source project [Gardener](https://gardener.cloud/). +We provide support to run Gardener on metal-stack via [Gardener extensions](../05-Concepts/04-Kubernetes/02-gardener.md). This integration is production-hardened, well documented, used by many organizations in production and build on top of the open-source project [Gardener](https://gardener.cloud/). ## Cluster API -Our [Cluster API integration](../05-Concepts/04-Kubernetes/02-cluster-api.md) is a more verbose approach to provide Kubernetes clusters with metal-stack. Our implementation is still in early development. It is based on the [Cluster API](https://cluster-api.sigs.k8s.io/) project. +Our [Cluster API integration](../05-Concepts/04-Kubernetes/03-cluster-api.md) is a more verbose approach to provide Kubernetes clusters with metal-stack. Our implementation is still in early development. It is based on the [Cluster API](https://cluster-api.sigs.k8s.io/) project. -Configuring Cluster API is very verbose. It requires additional tooling to provide a developer experience. Cluster API will give you building blocks to build a Kubernetes-as-a-service platform on top of it, but no more. We do not recommend you use Cluster API, unless you already have a large platform engineering team, that is very experienced in bare-metal K8s operations and they agree that your specific requirements cannot be modelled with Gardener. In any other case, you will have more success with Gardener. \ No newline at end of file +Configuring Cluster API is very verbose. It requires additional tooling to provide a developer experience. Cluster API will give you building blocks to build a Kubernetes-as-a-service platform on top of it, but no more. We do not recommend you use Cluster API, unless you already have a large platform engineering team, that is very experienced in bare-metal K8s operations and they agree that your specific requirements cannot be modelled with Gardener. In any other case, you will have more success with Gardener. diff --git a/docs/04-For Operators/03-Deployment/01_guide.md b/docs/04-For Operators/03-Deployment/01_guide.md index 75bb9125..94710e3a 100644 --- a/docs/04-For Operators/03-Deployment/01_guide.md +++ b/docs/04-For Operators/03-Deployment/01_guide.md @@ -8,61 +8,80 @@ sidebar_position: 1 metal-stack is a Metal-as-a-Service (MaaS) platform that turns on-premises bare-metal servers into an elastic, self-managed cloud infrastructure. At its core, metal-stack consists of two components: -- **Control Plane** — The central management layer (APIs, databases, scheduling) +- **Control Plane** — The central management layer (APIs, databases, messaging, scheduling) - **Partitions** — The data center infrastructure (servers, leaf switches, BMCs) These two components alone provide a fully functional **Bare-Metal as a Service (MaaS)** platform. You can allocate machines, manage networks, configure firewalls, and operate servers via REST/gRPC APIs and the `metalctl` CLI — without any Kubernetes cluster lifecycle management. ## Kubernetes Cluster Lifecycle Management -If you need to provision **Kubernetes clusters** on top of your bare-metal infrastructure, metal-stack integrates with KCLM solutions like [Gardener](../../05-Concepts/04-Kubernetes/01-gardener.md) or [Cluster-API](../../05-Concepts/04-Kubernetes/02-cluster-api.md). These are **optional layers** built on top of the MaaS foundation — not prerequisites. +If you need to provision **Kubernetes clusters** on top of your bare-metal infrastructure, metal-stack integrates with Kubernetes Cluster Lifecycle Management (KCLM) solutions like [Gardener](../../05-Concepts/04-Kubernetes/02-gardener.md) or [Cluster API](../../05-Concepts/04-Kubernetes/03-cluster-api.md). These are **optional layers** built on top of the MaaS foundation — not prerequisites. :::tip -You can use metal-stack as a pure MaaS platform without any KCLM integration. The KCLM layer is only needed if you want to automate Kubernetes cluster provisioning on your bare-metal infrastructure. +You can use metal-stack as a pure MaaS platform without any KCLM integration. The KCLM layer is only needed if you want to automate Kubernetes cluster provisioning on your bare-metal infrastructure. See the [KCLM concepts overview](../../05-Concepts/04-Kubernetes/01-kclm.md) for a comparison of the available options. ::: -### Deployment Approach +## Deployment Approach We bootstrap the [metal control-plane](../../05-Concepts/01-architecture.mdx#metal-control-plane) as well as our [partitions](../../05-Concepts/01-architecture.mdx#partitions) with [Ansible](https://www.ansible.com/) through CI. The deployment has multiple phases, resulting in a Git repository containing Ansible playbooks, inventory files and CI/CD workflows. -1. **[Control Plane](./03_control-plane.mdx)** — Deploy the metal-stack API, databases, and ingress on a bootstrap Kubernetes cluster. This alone gives you a functional MaaS platform. -2. **[Partition](./04_partition.md)** — Configure the network fabric (leaf switches, management spine, DHCP, PXE) and connect your bare-metal servers to the control plane. -3. **[KCLM with Gardener](./05_kclm.md)** — Add Gardener to manage Kubernetes cluster lifecycle on your bare-metal infrastructure. +1. **[Bootstrap Infrastructure](./02_bootstrap-infrastructure.md)** — Decide where the initial Kubernetes cluster(s) that host metal-stack and, optionally, your KCLM run. This is a planning phase, not an Ansible deployment. +2. **[Control Plane](./03_control-plane.mdx)** — Deploy the metal-stack APIs, databases and ingress onto the initial Kubernetes cluster. This alone gives you a functional MaaS platform. +3. **[Partition](./04_partition.md)** — Bootstrap the out-of-band network, configure the network fabric (leaf, spine and exit switches, DHCP, PXE) and connect your bare-metal servers to the control plane. +4. **[KCLM with Gardener](./05_gardener.md)** — Add Gardener to manage the Kubernetes cluster lifecycle on your bare-metal infrastructure. Alternatively, see [Cluster API](./06_cluster-api.md). + +Two cross-cutting topics complete the picture once the platform runs: [GPU Workers](./07-gpu-workers.md) and [Offline Resilience](./08_offline-resilience.md). By the end of this guide, your deployment repository will look something like this: ```text my-metal-stack-deployment/ +├── ansible.cfg ├── deploy_metal_control_plane.yaml # Control plane -├── deploy_partition.yaml # Partition -├── deploy_gardener.yaml # KCLM -├── inventory/ -│ ├── inventory.yaml -│ ├── group_vars/ # Variables per host group -│ └── host_vars/ # Variables per host -├── roles/ # Custom Ansible roles -├── files/ +├── deploy_gardener.yaml # KCLM (optional) +├── deploy_mgmt_servers.yaml # Partition: management servers +├── deploy_mgmt_switches.yaml # Partition: management switches +├── deploy_spines_exits.yaml # Partition: spine and exit switches +├── deploy_leaves.yaml # Partition: leaf switches +├── inventories/ +│ ├── control-plane.yaml # inventory for the Kubernetes deployments +│ ├── partition.yaml # inventory for the partition hosts +│ ├── group_vars/ # variables per host group +│ └── host_vars/ # variables per host +├── roles/ # custom Ansible roles +├── files/ # certificates, keys, static assets └── .github/workflows/ # CI/CD pipelines ``` The different phases of the deployment guide show which files to add and how they fit into this structure. :::tip -You can use the [mini-lab](https://github.com/metal-stack/mini-lab) as a template project for your own deployment. It uses the same approach as described in this guide. +The [mini-lab](https://github.com/metal-stack/mini-lab) is the fastest way to see a complete, runnable parametrization of the very same Ansible roles. It brings up a virtualized metal-stack — including an optional Gardener or Kamaji landscape — on a single machine. + +Treat it as a **reference for how the roles are wired together, not as a production blueprint.** The mini-lab intentionally cuts corners that are unacceptable in production: it runs on a single [kind](https://kind.sigs.k8s.io/) cluster, ships unencrypted secrets, fakes the `kube-system/shoot-info` ConfigMap and patches load balancer statuses by hand. Wherever this guide references the mini-lab, it does so to point at a concrete example — never as a recommendation. ::: -We recommend using the same Ansible roles that we use to deploy metal-stack in our own environments. They are available in the [metal-roles](https://github.com/metal-stack/metal-roles) repository. +We recommend using the same Ansible roles that we use to deploy metal-stack in our own environments. They are available in the [metal-roles](https://github.com/metal-stack/metal-roles) repository, split into [`control-plane`](https://github.com/metal-stack/metal-roles/tree/master/control-plane), [`partition`](https://github.com/metal-stack/metal-roles/tree/master/partition) and [`common`](https://github.com/metal-stack/metal-roles/tree/master/common) roles. Every role carries a `README.md` that documents its variables — these READMEs are the authoritative variable reference, while this guide focuses on the red line through the deployment. -To simplify dependency management, we provide a dedicated [deployment base image](https://github.com/metal-stack/metal-deployment-base) on GitHub Container Registry. This Docker image bundles all required tools and libraries, keeping your deployment environment lean and easy to maintain. +To simplify dependency management, we provide a dedicated [deployment base image](https://github.com/metal-stack/metal-deployment-base) on GitHub Container Registry. This container image bundles all required tools and libraries (Ansible, `kubectl`, `helm`, `cosign`, `metalctl`, …), keeping your deployment environment lean, reproducible and easy to maintain — locally and in CI. -The remainder of this guide assumes you are using our Ansible roles and the deployment base image. Make sure you have [Docker](https://docs.docker.com/get-started/get-docker/) installed. Every deployment environment is unique, so this guide describes how we set up metal-stack in our own infrastructure. You will need to adapt the steps to match your specific requirements. +## What You Need to Know -:::warning -You will likely need to learn to write Ansible playbooks if you want to be able to deploy the metal-stack as presented in this documentation. However, even when starting without any knowledge of Ansible it should be possible to follow these docs. In case you need further explanations regarding Ansible please refer to [docs.ansible.com](https://docs.ansible.com/). -::: +This guide assumes you are comfortable with the following. It does not teach them: + +- **Kubernetes** — You will operate metal-stack on Kubernetes and troubleshoot it with `kubectl`. +- **Ansible** — All deployments are Ansible playbooks and roles. Refer to [docs.ansible.com](https://docs.ansible.com/) if you are new to it; you do not need to be an expert to follow along, but you will write playbooks. +- **Data center networking** — BGP, VRFs, EVPN/VXLAN and DHCP/PXE. The [networking concepts](../../05-Concepts/03-Network/01-theory.md) chapter explains how metal-stack uses them. +- **Your KCLM of choice** — If you plan to deploy [Gardener](./05_gardener.md) or [Cluster API](./06_cluster-api.md), you should already be familiar with its own concepts and terminology. + +Every deployment environment is unique. This guide describes how we set up metal-stack in our own infrastructure — you will need to adapt the steps to match your hardware, network and compliance requirements. Make sure you have [Docker](https://docs.docker.com/get-started/get-docker/) (or a compatible container runtime) installed to run the deployment base image. :::info -If you do not want to use Ansible for deployment, you need to come up with a deployment mechanism by yourself. You can still learn from our [metal-roles](https://github.com/metal-stack/metal-roles) repository and [Helm charts](https://github.com/metal-stack/helm-charts/) — but be aware that the Ansible roles tie everything together (variable management, dependency ordering, environment-specific configurations), so building an equivalent from scratch will require significant effort. +If you do not want to use Ansible for deployment, you need to come up with a deployment mechanism by yourself. You can still learn from our [metal-roles](https://github.com/metal-stack/metal-roles) repository and [Helm charts](https://github.com/metal-stack/helm-charts/) — but be aware that the Ansible roles tie everything together (variable management, dependency ordering, environment-specific configuration), so building an equivalent from scratch will require significant effort. +::: + +:::tip +Stuck? The [troubleshooting guide](../06-troubleshoot.md) collects the issues we run into most often, and the [community page](/community) tells you how to reach us. Feedback on this guide is explicitly welcome — it is how we make the deployment easier for the next operator. ::: diff --git a/docs/04-For Operators/03-Deployment/02_bootstrap-infrastructure.md b/docs/04-For Operators/03-Deployment/02_bootstrap-infrastructure.md index 74f26ad7..2f81293a 100644 --- a/docs/04-For Operators/03-Deployment/02_bootstrap-infrastructure.md +++ b/docs/04-For Operators/03-Deployment/02_bootstrap-infrastructure.md @@ -21,56 +21,57 @@ If you only need **Bare-Metal as a Service** without KCLM, you need at least **o Your control plane Kubernetes cluster can run anywhere — on a hyperscaler, in your own data center, or on [metalstack.cloud](https://metalstack.cloud). A managed cluster removes the operational burden of running Kubernetes yourself and can even strengthen fail-safe operation. Learn more about the [rationale for this approach](../../05-Concepts/01-architecture.mdx#target-deployment-platforms) and find concrete hosting suggestions below. ::: -## KCLM Solutions +## Requirements for the Initial Cluster -metal-stack supports three Kubernetes Cluster Lifecycle Management solutions, each with different maturity levels and capabilities. -See the [Kubernetes Concepts Section](../../05-Concepts/04-Kubernetes/01-gardener.md) for a detailed comparison. +Whichever hosting option you pick, the initial cluster has to satisfy three properties: -### Gardener +- **Reachable from the partitions** — Every partition must reach the metal-api over HTTPS as well as gRPC (`50051`), NSQ (`4150`) and metal-console (`5222`). These layer-4 services need explicit exposure; see [Control Plane](./03_control-plane.mdx). +- **Persistent storage** — The control plane databases hold your masterdata and IP address management. Provide a storage class with persistent volumes and enable the [backup-restore-sidecar](./03_control-plane.mdx#setting-up-the-backup-restore-sidecar). +- **Independent lifecycle** — The initial cluster must not run on the machines that metal-stack manages, otherwise you create a circular dependency you cannot recover from. -[Gardener](../../05-Concepts/04-Kubernetes/01-gardener.md) is the **recommended** path for Kubernetes cluster lifecycle management. It is battle-tested in production for over seven years at financial-sector customers and bundles several day-2 capabilities natively (DNS, backup, audit). Gardener manages entire clusters as Kubernetes-native resources with a strong separation between platform operators and end-users. +## Choosing a KCLM Solution -:::tip -Gardener is the recommended solution for production environments. See the [Gardener concept section](../../05-Concepts/04-Kubernetes/01-gardener.md) for terminology and architecture details. -::: - -### Cluster-API - -[Cluster-API](../../05-Concepts/04-Kubernetes/02-cluster-api.md) is a CNCF project maintained by a Kubernetes SIG that provides declarative cluster management through a management cluster. The metal-stack provider (CAPMS) is **under development** and not yet production-ready. - -:::warning -Cluster-API with metal-stack is in development and not advised for production use. Please use Gardener for production workloads. -::: +metal-stack integrates with two Kubernetes Cluster Lifecycle Management solutions — Gardener and Cluster API (the latter optionally combined with Kamaji as control-plane provider). They differ substantially in maturity, day-2 capabilities and operational model. -#### Kamaji +The [KCLM concepts chapter](../../05-Concepts/04-Kubernetes/01-kclm.md#two-approaches-one-infrastructure) contains the full comparison and decision matrix. The short version: -[Kamaji](../../05-Concepts/04-Kubernetes/02-cluster-api.md#kamaji) allows a similar control plane hosting model as Gardener, where the control plane runs on dedicated infrastructure separate from worker nodes. -Kamaji therefore uses ClusterAPI to support different infrastructure- and control-plane providers. -However, Kamaji integrations with metal-stack **have not been evaluated in production-grade scenarios** by metal-stack. +| Solution | Status | Guide | +| ------------------------------------------------------------------------------------------------- | ---------------------------------------------------------- | ------------------------------------------------------------------------------ | +| [Gardener](../../05-Concepts/04-Kubernetes/02-gardener.md) | **Recommended**, production-proven for 7+ years | [Gardener deployment](./05_gardener.md) | +| [Cluster API (CAPMS)](../../05-Concepts/04-Kubernetes/03-cluster-api.md) | Beta, under active development | [Cluster API deployment](./06_cluster-api.md) | +| [Cluster API + Kamaji](../../05-Concepts/04-Kubernetes/03-cluster-api.md#kamaji-with-metal-stack) | Not evaluated in production-grade scenarios by metal-stack | [Cluster API deployment](./06_cluster-api.md#kamaji-as-control-plane-provider) | :::warning -Kamaji with metal-stack is in development and not advised for production use. Please use Gardener for production workloads. +Only Gardener is recommended for production workloads today. Cluster API with metal-stack is in beta, and the Kamaji integration has not been evaluated in production-grade scenarios by metal-stack. Your KCLM choice determines the required cluster topology described below, so make it before you deploy anything. ::: ## Deployment Options -There are three supported approaches for hosting the initial cluster: +There are three approaches for hosting the initial cluster. All three are viable; they trade operational simplicity against autonomy. + +| | Option 1: Shared | Option 2: Dedicated | Option 3: Autonomous | +| ------------------- | --------------------------- | ------------------------------- | ------------------------ | +| Clusters | 1 | 2 | 2 (nested) | +| Operational effort | Lowest | Low | Highest | +| Failure isolation | Weak | Good | Good | +| Digital sovereignty | Depends on provider | Depends on provider | Full | +| Recommended for | Evaluation, small platforms | **Most production deployments** | Sovereignty requirements | ### Option 1: Shared Initial Cluster -It is possible to use a **single initial cluster** for both metal-stack and the KCLM solution. This approach is technically feasible but **not recommended** for production environments. Sharing a single cluster mixes platform infrastructure with lifecycle management, which can complicate operational boundaries and failure isolation. +It is possible to use a **single initial cluster** for both metal-stack and the KCLM solution. This approach is technically feasible but **not recommended** for production environments. Sharing a single cluster mixes platform infrastructure with lifecycle management, which complicates operational boundaries and failure isolation — a Gardener upgrade and a metal-api upgrade then share one blast radius. ### Option 2: Dedicated Initial Clusters We recommend using **dedicated initial clusters** for metal-stack and the KCLM solution — one cluster for the metal-stack control plane and a separate cluster for the KCLM. -This approach provides clearer operational boundaries, better isolation and simplified failure boundaries. +This approach provides clearer operational boundaries, better isolation and simplified failure boundaries. In Gardener terminology the second cluster becomes the _runtime cluster_ (and typically the first Seed, the "soil"); see the [Gardener deployment guide](./05_gardener.md#architecture-overview). ### Option 3: Autonomous Control Plane For self-hosted deployments, metal-stack can be set up with an [Autonomous Control Plane](/community/MEP-18-autonomous-control-plane) cluster. This approach is the best choice for organizations that require full digital sovereignty and autonomy over their entire infrastructure stack. -The autonomous control-plane cluster serves as a minimal control plane whose sole purpose is to host the production control plane cluster (the "Matryoshka principle"). +The autonomous control-plane cluster serves as a minimal cluster whose sole purpose is to host the production control plane cluster (the "Matryoshka principle"). This brings several advantages like failure isolation, separate operational responsibility, minimal resource requirements and full control and ownership. The only requirement from metal-stack is that your partitions can establish network connections to the metal control-plane. @@ -82,13 +83,17 @@ The only requirement from metal-stack is that your partitions can establish netw For the shared and dedicated cluster approaches, the initial cluster can be hosted anywhere — a hyperscaler, metalstack.cloud, or any other managed Kubernetes provider. Some common options: - **metalstack.cloud** — A Kubernetes cluster can be created via [UI](https://metalstack.cloud/de/documentation/UserManual#creating-a-cluster), CLI, or [Terraform](https://github.com/metal-stack-cloud/terraform-provider-metal). -- **GCP/GKE** — A GCP account is required. The Ansible [gcp-auth role](https://github.com/metal-stack/ansible-common/tree/master/roles/gcp-auth) can be used for authentication, and the [gcp-create role](https://github.com/metal-stack/ansible-common/tree/master/roles/gcp-create) for creating a GKE cluster. - - Suggested defaults: `gcp_machine_type`: e2-standard-8, `gcp_autoscaling_min_nodes`: 1, `gcp_autoscaling_max_nodes`: 3 +- **GCP/GKE** — A GCP account is required. The Ansible [gcp-auth role](https://github.com/metal-stack/ansible-common/tree/master/roles/gcp-auth) can be used for authentication, and the [gcp-create role](https://github.com/metal-stack/ansible-common/tree/master/roles/gcp-create) for creating a GKE cluster. Use `metal_control_plane_host_provider: gcp` in this case. + - Suggested defaults: `gcp_machine_type`: `e2-standard-8`, `gcp_autoscaling_min_nodes`: 1, `gcp_autoscaling_max_nodes`: 3 + +:::info +`metal_control_plane_host_provider` is only evaluated by the Gardener roles, which use it to discover the runtime cluster's node, pod and service CIDRs. It is asserted to be either `metal` or `gcp`. Any other Kubernetes distribution works as well, but you then have to provide the `kube-system/shoot-info` ConfigMap yourself — see the [Gardener prerequisites](./05_gardener.md#prerequisites). +::: ### For Option 3: Autonomous Control Plane with k3s For the autonomous control plane approach, [MEP-18](/community/MEP-18-autonomous-control-plane) proposes using [k3s](https://k3s.io/) as the initial cluster. This is because KCLM solutions are not yet able to create an initial cluster themselves (though this may change with implementations like [GEP-28](https://github.com/gardener/enhancements/blob/main/geps/0028-self-hosted-shoot-clusters/README.md) for Gardener). -The k3s nodes can be either bare metal machines or virtual machines. For a minimal setup, a single node with 8–16 cores, 64GB RAM, and two NVMe drives of 1TB is a good starting point. For high availability, a clustered k3s configuration across multiple nodes is recommended, with ETCD replication and backup-restore mechanisms configured for metal-stack and KCLM components. +The k3s nodes can be either bare metal machines or virtual machines. For a minimal setup, a single node with 8–16 cores, 64 GB RAM, and two NVMe drives of 1 TB is a good starting point. For high availability, a clustered k3s configuration across multiple nodes is recommended, with etcd replication and backup-restore mechanisms configured for metal-stack and KCLM components. See the [Autonomous Control Plane](/community/MEP-18-autonomous-control-plane) proposal for detailed architecture, failure scenarios, and implementation guidance. diff --git a/docs/04-For Operators/03-Deployment/03_control-plane.mdx b/docs/04-For Operators/03-Deployment/03_control-plane.mdx index 2b6fee10..81b542a2 100644 --- a/docs/04-For Operators/03-Deployment/03_control-plane.mdx +++ b/docs/04-For Operators/03-Deployment/03_control-plane.mdx @@ -14,6 +14,20 @@ While it is theoretically possible to deploy metal-stack without Kubernetes (see The control plane deployment described here requires an initial Kubernetes cluster as bootstrap infrastructure, which is described in [Bootstrap Infrastructure](./02_bootstrap-infrastructure.md). We use the [metal-stack control-plane Ansible roles](https://github.com/metal-stack/metal-roles/tree/master/control-plane) to deploy all control-plane components. +Before you try to deploy the control-plane, you should have had a look at the [architecture](../../05-Concepts/01-architecture.mdx) concepts section, which explains what each of the components you are about to deploy actually does. + +:::info Layer-4 exposure +Besides the HTTP(S) ingress, the control plane also has to expose three **layer-4** services so that a [partition](./04_partition.md) can reach them. Plan this together with your ingress controller — for example through [TCP/UDP service exposure of ingress-nginx](https://kubernetes.github.io/ingress-nginx/user-guide/exposing-tcp-udp-services/) or a dedicated `Service` of type `LoadBalancer`: + +| Port | Protocol | Service | Purpose | +| ----- | -------- | --------------- | -------------------------------- | +| 4150 | TCP | `nsqd` | NSQ daemon (TLS) | +| 5222 | TCP | `metal-console` | Machine console forwarding (SSH) | +| 50051 | TCP | `metal-api` | metal-api gRPC API (protobuf) | + +The [mini-lab](https://github.com/metal-stack/mini-lab) shows a minimal variant of this exposure. +::: + Create the deployment folder and enter it — this is the root of your deployment repository as outlined in the [Deployment Guide](./01_guide.md): ```bash @@ -45,8 +59,7 @@ By the end of this section, your repository will contain the following files and │      │   └── release_vector.yaml │      └── control-plane │        ├── common.yaml -│         └── metal.yml -├── generate_role_requirements.yaml +│         └── metal.yaml └── roles └── ingress-controller └── tasks @@ -110,7 +123,19 @@ control-plane: We do this since we are deploying to Kubernetes and do not need to SSH-connect to any hosts for the deployment (which is what Ansible typically does). This inventory is also necessary to pick up the variables inside `inventories/group_vars/control-plane` during the deployment. -Most of the properties in there are up to taste, but make sure you enable the [Jinja2 native environment](https://jinja.palletsprojects.com/en/2.11.x/nativetypes/) as this is needed for some of our roles in certain cases. +## Ansible Configuration + +Most Ansible settings are up to taste, but you **must** enable the [Jinja2 native environment](https://jinja.palletsprojects.com/en/stable/nativetypes/). Several of our roles rely on Jinja2 returning native Python types (lists, dicts, booleans) instead of strings. A minimal `ansible.cfg` looks like this: + +```ini +[defaults] +jinja2_native = True +inventory = inventories/control-plane.yaml +roles_path = ~/.ansible/roles +force_color = True +``` + +The same effect can be achieved with the `ANSIBLE_JINJA2_NATIVE=True` environment variable, which is what the CI examples in this guide use. ## Control Plane Playbook @@ -139,12 +164,18 @@ Basically, this playbook does the following: - Include all the modules, filter plugins, etc. of [ansible-common](https://github.com/metal-stack/ansible-common) into the play - Deploys an ingress-controller into your cluster - Deploys the metal-stack by - - Running preparation tasks - - Deploying NSQ - - Deploying the rethinkdb database for the metal-api (wrapped in a backup-restore-sidecar), - - Deploying the postgres database for go-ipam (wrapped in a backup-restore-sidecar) - - Deploying the postgres database for the masterdata-api (wrapped in a backup-restore-sidecar) - - Applying the metal control plane helm chart + - Running preparation tasks (namespace, image pull secrets, common resources) + - Deploying [NSQ](https://nsq.io/) — the message queue that connects metal-api and metal-core + - Deploying the RethinkDB database for the metal-api (wrapped in a backup-restore-sidecar) + - Deploying the PostgreSQL database for [go-ipam](https://github.com/metal-stack/go-ipam) (wrapped in a backup-restore-sidecar) + - Deploying the PostgreSQL database for the masterdata-api (wrapped in a backup-restore-sidecar) + - Applying the metal control plane Helm chart + +The role order matters: `prepare` creates the namespace all other roles deploy into, and the `metal` role expects the message queue and all three databases to be present. + +:::tip +The control-plane roles are additive. Once the basic deployment works, you can extend the same playbook with optional roles such as `monitoring`, `logging`, `auditing-timescaledb` (audit backend), `headscale` (firewall VPN), `zitadel`/`zitadel-db` (identity provider) or `isolated-clusters`. See the [control-plane role overview](https://github.com/metal-stack/metal-roles/tree/master/control-plane) for the complete list, and the [mini-lab playbook](https://github.com/metal-stack/mini-lab/blob/master/deploy_control_plane.yaml) for an example that enables most of them. +::: ## Setup an ingress-controller @@ -169,13 +200,25 @@ The [ansible-common](https://github.com/metal-stack/ansible-common) repository c ## Deployment Parametrization -Now you can parametrize the referenced roles to fit your environment. The role parametrization can be looked up in the role documentation on [metal-roles/control-plane](https://github.com/metal-stack/metal-roles/tree/master/control-plane). You should not need to define a lot of variables for the beginning as most values are reasonably defaulted. You can start with the following content for `group_vars/control-plane/common.yaml`: +Now you can parametrize the referenced roles to fit your environment. The role parametrization can be looked up in the role documentation on [metal-roles/control-plane](https://github.com/metal-stack/metal-roles/tree/master/control-plane). You should not need to define a lot of variables for the beginning as most values are reasonably defaulted. You can start with the following content for `inventories/group_vars/control-plane/common.yaml`: ```yaml --- -metal_control_plane_ingress_dns: # if you do not have a DNS entry, you could also start with .nip.io +# The DNS domain under which all control-plane services are exposed via ingress. +# If you do not have a DNS entry yet, you can start with .nip.io. +metal_control_plane_ingress_dns: + +# Optional but recommended: a name for this stage/landscape. It is used for +# prefixing and is inherited by the Gardener roles as the Garden name. +metal_control_plane_stage_name: demo ``` +:::warning Secrets +Any variable holding a credential — HMAC keys, database passwords, backup credentials, OIDC client secrets — belongs into an [Ansible Vault](https://docs.ansible.com/ansible/latest/vault_guide/index.html) encrypted file, never into a plain group_vars file. The examples in this guide reference vaulted variables (for example `metal_api_admin_key`) instead of inlining values. + +metal-stack ships with the default HMAC key `change-me`. Replace it with strong random values before the control plane becomes reachable from anywhere but your laptop. +::: + ## Providing Certificates We have several components in our stack that communicate over encrypted gRPC just like Kubernetes components do. @@ -370,7 +413,7 @@ Also define the following configurations for `cfssl`: Running the `roll_certs.sh` bash script without any arguments should generate you the required certificates. -Now Provide the paths to these certificates in `group_vars/control-plane/metal.yaml`: +Now provide the paths to these certificates in `inventories/group_vars/control-plane/metal.yaml`: ```yaml --- @@ -431,11 +474,75 @@ docker run --rm -it \ If you are having issues regarding the deployment take a look at the [troubleshoot document](../06-troubleshoot.md). Please give feedback such that we can make the deployment of the metal-stack easier for you and for others! ::: +## GitHub Action + +You can also automate the control plane deployment through a GitHub Action workflow. Here is an example `.github/workflows/deploy-control-plane.yaml`: + +```yaml +--- +name: Deploy control plane + +on: + workflow_dispatch: + inputs: + deploy-control-plane: + description: 'Which control-plane target to deploy' + required: true + type: choice + options: + - metal-stack + +permissions: + contents: read + +env: + ANSIBLE_INVENTORY: inventories/control-plane.yaml + ANSIBLE_FORCE_COLOR: "1" + ANSIBLE_JINJA2_NATIVE: "True" + + # Update these with your actual values + CLUSTER_ID: + DEFAULT_PROJECT_ID: + + KUBECONFIG: /tmp/.kubeconfig + +jobs: + metal-stack: + name: Deploy metal-stack control plane + if: ${{ inputs.deploy-control-plane == 'metal-stack' }} + + runs-on: ubuntu-latest + container: ghcr.io/metal-stack/metal-deployment-base:v0.22.18 + + steps: + - name: Checkout + uses: actions/checkout@v7 + + - run: | + metal ctx add demo --api-token ${METALSTACKCLOUD_API_TOKEN} --default-project ${DEFAULT_PROJECT_ID} --activate + metal cluster kubeconfig ${CLUSTER_ID} + + printf '%s' "${ANSIBLE_VAULT_PASSWORD}" > ${ANSIBLE_VAULT_PASSWORD_FILE} + + ansible localhost -m metalstack.base.metal_stack_release_vector + ansible-playbook deploy_metal_control_plane.yaml --diff + env: + METALSTACKCLOUD_API_TOKEN: ${{ secrets.METALSTACKCLOUD_API_TOKEN }} + ANSIBLE_VAULT_PASSWORD: ${{ secrets.ANSIBLE_VAULT_PASSWORD }} + ANSIBLE_VAULT_PASSWORD_FILE: .vault.txt +``` + +:::tip +The example uses GitHub Actions with `ubuntu-latest` runners (not self-hosted) and a runtime cluster hosted on [metal-stack cloud](https://metalstack.cloud/en). +As mentioned in [Bootstrap Infrastructure](./02_bootstrap-infrastructure.md), any Kubernetes cluster can serve as the runtime — you can adapt the workflow to use self-hosted runners or a different cluster provider as needed. +The `metal` CLI is used to fetch the kubeconfig from the metal-stack API. Update the `CLUSTER_ID` and `DEFAULT_PROJECT_ID` environment variables with your actual values. +::: + ## Providing Images After the deployment has finished, you should consider deploying some masterdata entities into your metal-api. For example, you can add your first machine sizes and operating system images. You can do this by further parametrizing the [metal role](https://github.com/metal-stack/metal-roles/tree/master/control-plane/roles/metal). We will just add an operating system for demonstration purposes. Add the following variable to your `inventories/group_vars/control-plane/common.yaml`: -``` +```yaml metal_api_images: - id: firewall-ubuntu-3.0 name: Firewall 3 Ubuntu @@ -457,17 +564,19 @@ metal_api_images: - machine ``` -See the [metal-images reference](https://metal-stack.io/docs/references/metal-images/#supported-images) for currently supported images. +See the [metal-images reference](../../08-References/Deployment/metal-images/metal-images.md) for currently supported images and the [operating systems](../02-operating-systems.md) chapter for the image contract if you want to build your own. Then, re-run the deployment to apply your changes. Our playbooks are idempotent. :::info -Image versions should be regularly checked for updates. +`metal_api_images` is more than a convenience: the Gardener roles default both the `CloudProfile` machine images and the `provider-metal` extension configuration to this very list. Keeping it as the single source of truth is what makes CloudProfile and extension agree — see the [Gardener deployment guide](./05_gardener.md#how-the-pieces-connect). + +Image versions should be checked for updates regularly; expired images can no longer be used for new allocations. ::: ## Setting up metalctl -You can now verify the existence of the operating system images in the metal-api using our CLI client called [metalctl](https://github.com/metal-stack/metalctl). The configuration for `metalctl` should look like this: +You can now verify the existence of the operating system images in the metal-api using our CLI client called [metalctl](https://github.com/metal-stack/metalctl). See the [metalctl reference](../../08-References/Clients/metalctl/metalctl.md) for the full command set. The configuration for `metalctl` should look like this: ```yaml # ~/.metalctl/config.yaml @@ -492,12 +601,16 @@ Issue the following command: ```bash $ metalctl image ls -ID NAME DESCRIPTION FEATURES EXPIRATION STATUS -ubuntu-19.10.20200331 Ubuntu 19.10 20200331 Ubuntu 19.10 20200331 machine 89d 23h preview +ID NAME DESCRIPTION FEATURES EXPIRATION STATUS +debian-13.0 Debian 13 Debian 13 Latest Release machine 89d 23h supported +firewall-ubuntu-3.0 Firewall 3 Ubuntu Firewall 3 Ubuntu Latest firewall 89d 23h supported +ubuntu-26.4 Ubuntu 26.04 Ubuntu 26.04 Latest Release machine 89d 23h supported ``` The basic principles of how the metal control plane can be deployed should now be clear. It is now up to you to move the deployment execution into your CI and add things like certificates for the ingress-controller and NSQ. +At this point you have a working **MaaS platform**: you can create projects and networks, register machines and allocate them through the metal-api. What is still missing is a [partition](./04_partition.md) with actual hardware — without it, there are no machines to allocate. + ## Setting Up the backup-restore-sidecar The backup-restore-sidecar can come in very handy when you want to add another layer of security to the metal-stack databases in your Kubernetes cluster. The sidecar takes backups of the metal databases in small time intervals and stores them in a blobstore of a cloud provider. For each database that will be backed up, a lifecycle rule is established. The backup mechanism is deactivated by default and must be activated by the operator. This way your metal-stack setup can even survive the deletion of your Kubernetes control plane cluster (including all volumes getting lost). After re-deploying metal-stack to another Kubernetes clusters, the databases come up with the latest backup data in a matter of seconds. @@ -513,6 +626,8 @@ metal-stack currently supports two authentication methods: - user authentication through [OpenID Connect](https://openid.net/developers/how-connect-works/) (OIDC) - [HMAC](https://en.wikipedia.org/wiki/HMAC) auth, typically used for access by technical users (because we do not have service account tokens at the time being) +There are three HMAC authentication types — `Metal-View`, `Metal-Edit` and `Metal-Admin` — which map to the corresponding roles. Partitions authenticate with the view and edit keys (see [Partition Connection](./04_partition.md#partition-connection)), and Gardener shoots use an admin or edit key (see [Gardener](./05_gardener.md#shootsyaml--shoot-clusters-and-shooted-seeds)). For the authorization model behind these roles, see [User Management](../../05-Concepts/02-user-management.md). + If you decided to use OIDC, you can parametrize the [metal role](https://github.com/metal-stack/metal-roles/tree/master/control-plane/roles/metal) for this by defining the variable `metal_masterdata_api_tenants` with the following configuration: ```yaml diff --git a/docs/04-For Operators/03-Deployment/04_partition.md b/docs/04-For Operators/03-Deployment/04_partition.md index 0e4e0236..794f2486 100644 --- a/docs/04-For Operators/03-Deployment/04_partition.md +++ b/docs/04-For Operators/03-Deployment/04_partition.md @@ -10,58 +10,65 @@ A partition is the data center infrastructure layer — the physical servers, sw This section continues from the [Control Plane](./03_control-plane.mdx) deployment and covers how to deploy the required infrastructure services and how to connect your partition to the control plane, using the [metal-stack partition Ansible roles](https://github.com/metal-stack/metal-roles/tree/master/partition). It is assumed that all cabling is done. +Before you try to deploy the partition, you should have had a look at the [architecture](../../05-Concepts/01-architecture.mdx) and [networking](../../05-Concepts/03-Network/01-theory.md) concepts sections — the partition deployment is the point where the network theory becomes concrete configuration, and the roles will not make sense without it. + +:::info +Unlike the control plane, a partition deployment touches **physical hardware you have to bootstrap by hand first**. This chapter therefore has two halves: the manual bootstrap of the out-of-band network (firewalls, management servers, management switches) and the automated Ansible deployment of everything else. Budget most of your time for the first half — it is the part that differs the most between environments. +::: During this section, our repository will grow to look something like the following: ```text . -├── deploy_mgmt_servers.yaml # management server services -├── deploy_mgmt_switches.yaml # mgmtleaves and mgmtspines SONiC -├── deploy_spines_exits.yaml # spine/exit switch SONiC -├── deploy_leaves.yaml # leaf switch SONiC + metal-core -├── inventory -│ ├── inventory.yaml # all partition host groups -│ ├── host_vars +├── deploy_mgmt_servers.yaml # management server services +├── deploy_mgmt_switches.yaml # mgmtleaves and mgmtspines SONiC +├── deploy_spines_exits.yaml # spine/exit switch SONiC +├── deploy_leaves.yaml # leaf switch SONiC + metal-core +├── inventories +│ ├── partition.yaml # all partition host groups +│ ├── host_vars/ │ └── group_vars │ ├── all/ -│ │ ├── release_vector.yaml -│ │ └── metal-stack/ -│ │ ├── partition.yaml # partition → control plane connection -│ │ └── control-plane.yaml # vault-encrypted: API keys, BMC passwords +│ │ └── release_vector.yaml # unchanged, shared with the control plane +│ ├── partition/ +│ │ ├── common.yaml # partition → control plane connection +│ │ └── secrets.yaml # vault-encrypted: HMAC keys, BMC passwords │ ├── mgmtservers/ │ │ ├── common.yaml +│ │ └── metal-bmc.yaml │ ├── mgmtspines/ -│ │ ├── common.yaml -│ │ ├── metal-bmc.yaml +│ │ └── common.yaml │ ├── mgmtleaves/ -│ │ ├── common.yaml -│ ├── exits/ -│ │ ├── common.yaml -│ │ └── sonic.yaml +│ │ └── common.yaml │ ├── spines/ │ │ ├── common.yaml │ │ └── sonic.yaml -│ ├── leaves/ +│ ├── exits/ │ │ ├── common.yaml -│ │ ├── sonic.yaml -│ │ └── metal-core.yaml -│ └── partition/ -│ └── common.yaml +│ │ └── sonic.yaml +│ └── leaves/ +│ ├── common.yaml +│ ├── sonic.yaml +│ └── metal-core.yaml └── .github/ └── workflows/ - └── deploy-partition.yaml # CI/CD for partition + └── deploy-partition.yaml # CI/CD for partition ``` +:::tip +Because the partition plays target real hosts over SSH, they need their own inventory (`inventories/partition.yaml`) next to the `localhost`-only `inventories/control-plane.yaml` from the [Control Plane](./03_control-plane.mdx#inventory) guide. The `inventories/group_vars/all/` directory is shared by both, which is why the release vector only has to be declared once. +::: + ## Out-Of-Band-Network -The first step of deploying a partition is to deploy the Out-Of-Band-Network. +The first step of deploying a partition is to deploy the Out-Of-Band-Network (OOB network). It provides a dedicated management network that remains accessible even when the production network is down or unconfigured. It is the foundation that enables remote bootstrapping, monitoring, and maintenance of all partition hardware — from leaf and spine switches to bare-metal servers via their BMC/IPMI interfaces. -The [partition networking](https://metal-stack.io/docs/next/networking) is designed to be secure, fully routable via BGP, scalable, resilient, deployable through CI/CD, and selectively accessible from the internet. +The [partition networking](../../05-Concepts/03-Network/01-theory.md) is designed to be secure, fully routable via BGP, scalable, resilient, deployable through CI/CD, and selectively accessible from the internet. To deploy a partition and its networking stack remotely and in a nearly automatic manner, **some components must be initially bootstrapped manually**: - the management firewalls, management servers, management spines and management leaves need to be configured -- a CI/CD-Runner of your choice needs to be installed to run the automated deployments. +- a CI/CD runner of your choice needs to be installed to run the automated deployments The result should look something like the following image, but could vary for different deployments. @@ -95,7 +102,7 @@ The firewalls (EdgeRouters) must fulfill the following requirements: Management servers are the main bootstrapping components of the Out-Of-Band-Network and serve as jump hosts for all partition components. Once they are installed, every other component can be deployed automatically. For our high-availability setup, there are again two of them, each connected to their management-firewall. -Bootstrapping the management servers requires remote IPMI access and a way to perform an unattended OS installation with an Ansible user and SSH keys pre-configured. The exact approach depends on your hardware, existing infrastructure, and preferred automation tools. Below are two common examples, but any solultion is fine. +Bootstrapping the management servers requires remote IPMI access and a way to perform an unattended OS installation with an Ansible user and SSH keys pre-configured. The exact approach depends on your hardware, existing infrastructure, and preferred automation tools. Below are two common examples, but any solution is fine. **Preconfigured ISO with preseed** — generate an ISO with a preseed file that installs an OS and an Ansible user, then attach it via the BMC's virtual media function. @@ -115,9 +122,13 @@ The [mgmt-server role](https://github.com/metal-stack/metal-roles/tree/master/pa The runner on the management server needs: - An SSH key pair for Ansible authentication -— The public key in each switch's `authorized_keys` (this can be achieved via the `ZTP` scripts) +- The public key in each switch's `authorized_keys` (this can be achieved via the `ZTP` scripts) - CI/CD secrets or any kind of access to the Ansible Vault password and private key +:::warning Chicken-and-egg +The CI runner cannot deploy itself. Bootstrap it either manually or from a hosted runner, and only then hand over to the self-hosted runner for all subsequent partition deployments. The same applies to the management firewalls and the base OS of the management servers. +::: + ### Spine & Exit Management Switches (Management Spines) Management spines connect the management interfaces of all spine and exit switches to the management servers to enable out-of-band management. @@ -145,15 +156,15 @@ After the initial bootstrapping, the management interfaces of the leaves continu In larger deployments, a dedicated set of out-of-band switches (mgmtoobs) may be used to isolate BMC/IPMI traffic from the management network. These switches connect directly to server BMCs and provide a separate L2 domain for IPMI traffic, keeping it isolated from management server and switch management interfaces. They are deployed through the same SONiC automation as other partition switches. ### Leaves and Spines (Production Network) -We have now reached the point where a lot of the configuration happens automatically via the official Ansible roles. -After the initial install via ONIE and ZTP, metal-core will take over the configuration of the leaves and spines. -The general role of the leaves and spines is explained in the [CLOS](https://metal-stack.io/docs/next/networking#clos) concepts section. +We have now reached the point where a lot of the configuration happens automatically via the official Ansible roles. +After the initial install via ONIE and ZTP, the `sonic-config` role establishes the base configuration, and `metal-core` takes over the dynamic, per-machine configuration of the leaves. -They also add the last piece of the puzzle for the bare-metal provisioning: -As the workers are directly connected to the leaves, meaning PXE boot requests need to be handled by them. -This is achieved by relaying DHCP requests from those interfaces to the management server (via the management-spine) so workers receive IP addresses for PXE boot. -After that, they are able to pull the image for the automated install and setup. +The general role of leaves and spines is explained in the [CLOS](../../05-Concepts/03-Network/01-theory.md#clos) concepts section, and the routing details in [BGP](../../05-Concepts/03-Network/01-theory.md#bgp) and [EVPN](../../05-Concepts/03-Network/01-theory.md#evpn). + +They also add the last piece of the puzzle for bare-metal provisioning: the workers are directly connected to the leaves, so their PXE boot requests have to be handled there. +This is achieved by relaying DHCP requests from those interfaces to the management server (via the management spine), so workers receive IP addresses for PXE boot. +After that, they are able to pull the [metal-hammer](https://github.com/metal-stack/metal-hammer) discovery image and the operating system image for the automated install and setup. The full sequence is documented as the [machine provisioning sequence](../../05-Concepts/01-architecture.mdx#machine-provisioning-sequence). --- @@ -164,20 +175,21 @@ The next step is to configure the Ansible inventory and playbooks that define yo The playbooks and directory structure shown in this document represent a **reference implementation** — one way to organize your deployment. The metal-stack partition roles are designed to be flexible, and you are free to organize playbooks, group hosts differently, or run services on different machines as long as the following architectural constraints are met: -- **PXE boot requires DHCP in the same Layer-2 domain** as unprovisioned servers. The PXE VLAN (`vlan4000`) must reach all bare metal servers that need provisioning. In our reference setup, the management server runs the DHCP server, and leaf switches run a DHCP relay (configured via the `sonic-config` role) that forwards requests from the production network back to the management server. You may place the DHCP server and relay wherever your network topology allows, as long as the L2 domain is preserved. See the [networking documentation](../../05-Concepts/03-Network/01-theory.md#pxe-boot-mode) for the full PXE/DHCP theory. +- **PXE boot requires DHCP in the same Layer-2 domain** as unprovisioned servers. The PXE VLAN (`vlan4000` by default, configurable via `metal_core_pxe_vlan_id` and `dhcp_relay_interface`) must reach all bare metal servers that need provisioning. In our reference setup, the management server runs the DHCP server, and the switches run a DHCP relay that forwards requests from the production network back to the management server. You may place the DHCP server and relay wherever your network topology allows, as long as the L2 domain is preserved. See the [networking documentation](../../05-Concepts/03-Network/01-theory.md#pxe-boot-mode) for the full PXE/DHCP theory. - **`metal-core` must run on leaf switches** to dynamically configure them from the metal-api. -- **Pixiecore must be reachable** by servers during PXE boot (TFTP/HTTP). -- **`metal-bmc` must run on a host with BMC/IPMI network access** to manage bare-metal servers. -- **Web-Server with switch images must be reachable** by switches during install -- **Image cache must be reachable** by machines during provisioning +- **Pixiecore must be reachable** by servers during PXE boot (TFTP/HTTP) — it serves the metal-hammer kernel and initrd. +- **`metal-bmc` must run on a host with BMC/IPMI network access** to manage bare-metal servers (power state, boot order, machine discovery). +- **A web server holding the switch images must be reachable** by switches during ONIE install and ZTP. +- **The image cache must be reachable** by machines during provisioning — it serves the operating system images and is also the key to [offline resilience](./08_offline-resilience.md). You can split these services across multiple playbooks, combine them into fewer playbooks, or run them on different hosts — the roles are independent and can be mixed and matched. The key is ensuring the services are deployed and the network dependencies are satisfied. ### Host Inventory -Add your networking infrastructure to the inventory and adapt the host names and group structure to match your physical topology. For production switches, use disjoint groups (e.g. odd/even) to enable rolling deployments without fabric disruption. +Add your networking infrastructure to `inventories/partition.yaml` and adapt the host names and group structure to match your physical topology. For production switches, use disjoint groups (e.g. odd/even) to enable rolling deployments without fabric disruption — every leaf pair, spine pair and exit pair should be split across the two groups so that one half of the fabric always stays up. ```yaml +--- partition: children: mgmtservers: @@ -207,12 +219,15 @@ partition: ### Partition Connection -The `inventory/group_vars/all/metal-stack/partition.yaml` file connects the partition to the control plane: +The `inventories/group_vars/partition/common.yaml` file connects the partition to the control plane. `metal_partition_id` must match a partition that exists in the metal-api — create it with `metalctl partition create` (or via the `metal` role) before the first deployment: ```yaml +--- metal_region: metal_partition_id: +metal_partition_timezone: Europe/Berlin +# REST API of the metal-api metal_partition_metal_api_protocol: https metal_partition_metal_api_addr: metal_partition_metal_api_port: 443 @@ -220,23 +235,42 @@ metal_partition_metal_api_basepath: /metal/ metal_partition_metal_api_hmac_edit_key: "{{ metal_control_plane_api_edit_key }}" metal_partition_metal_api_hmac_view_key: "{{ metal_control_plane_api_view_key }}" +# gRPC endpoint of the metal-api, consumed by metal-core and metal-hammer. +# The certificates are the ones generated in the control plane guide. metal_partition_metal_api_grpc_address: "{{ metal_partition_mgmt_gateway }}:50051" metal_partition_metal_api_grpc_ca_cert: "{{ lookup('file', 'certs/ca.pem') }}" metal_partition_metal_api_grpc_client_cert: "{{ lookup('file', 'certs/metal-api-grpc/client.pem') }}" metal_partition_metal_api_grpc_client_key: "{{ lookup('file', 'certs/metal-api-grpc/client-key.pem') }}" -metal_partition_mgmt_gateway: 172.17.0.1 +# Gateway through which partition components reach the control plane +metal_partition_mgmt_gateway: ``` -The HMAC keys and other sensitive values should be stored in an Ansible vault. +:::warning +The HMAC keys must match the ones configured for the control plane, and the gRPC client certificate must be signed by the same CA as the metal-api server certificate — both come from the [Providing Certificates](./03_control-plane.mdx#providing-certificates) step. All of these values are secrets and belong into an Ansible Vault encrypted file. +::: + +For the full list of shared partition variables, see the [metal-roles/partition README](https://github.com/metal-stack/metal-roles/tree/master/partition). -### Leaf and Spine Variables +### Leaf, Spine and Management Server Variables -Switch variables define SONiC configuration, `metal-core` settings, and monitoring exporters. Management server variables define BMC access, DHCP configuration, and image cache settings. +Switch variables define the SONiC configuration, `metal-core` settings, and monitoring exporters. Management server variables define BMC access, DHCP configuration, and image cache settings. -The `sonic-config` role handles FRR routing configuration, DHCP relay setup on switches, and generates the complete `config_db.json` for SONiC. For complex configurations such as port breakouts, VRFs, EVPN underlay, interconnects, and extended CACL rules, refer to the [sonic-config role](https://github.com/metal-stack/metal-roles/tree/master/partition/roles/sonic-config) documentation and the `sonic-config` role defaults. +The `sonic-config` role handles FRR routing configuration, the DHCP relay setup on switches, and generates the complete `config_db.json` for SONiC. For complex configurations such as port breakouts, VRFs, EVPN underlay, interconnects, and extended control-plane ACL (CACL) rules, refer to the [sonic-config role](https://github.com/metal-stack/metal-roles/tree/master/partition/roles/sonic-config) documentation and its role defaults. -For the full variable reference, see the [metal-roles/partition](https://github.com/metal-stack/metal-roles/tree/master/partition) documentation. +Each role documents its variables in its own README. The ones you will touch most often: + +| Role | Purpose | README | +| --------------------- | ----------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | +| `mgmt-server` | Base configuration of the management server (BGP, DNS, SSH) | [mgmt-server](https://github.com/metal-stack/metal-roles/tree/master/partition/roles/mgmt-server) | +| `mgmt-firewall` | Reference configuration for the management firewalls | [mgmt-firewall](https://github.com/metal-stack/metal-roles/tree/master/partition/roles/mgmt-firewall) | +| `sonic-config` | SONiC `config_db.json` and FRR for all switches | [sonic-config](https://github.com/metal-stack/metal-roles/tree/master/partition/roles/sonic-config) | +| `metal-core` | Dynamic leaf configuration driven by the metal-api | [metal-core](https://github.com/metal-stack/metal-roles/tree/master/partition/roles/metal-core) | +| `metal-bmc` | BMC/IPMI access and machine discovery | [metal-bmc](https://github.com/metal-stack/metal-roles/tree/master/partition/roles/metal-bmc) | +| `dhcp` / `dhcp-relay` | DHCP server and relay for PXE boot | [dhcp](https://github.com/metal-stack/metal-roles/tree/master/partition/roles/dhcp), [dhcp-relay](https://github.com/metal-stack/metal-roles/tree/master/partition/roles/dhcp-relay) | +| `pixiecore` | Serves metal-hammer kernel and initrd via PXE | [pixiecore](https://github.com/metal-stack/metal-roles/tree/master/partition/roles/pixiecore) | +| `image-cache` | Local mirror for OS images, kernels and metal-hammer | [image-cache](https://github.com/metal-stack/metal-roles/tree/master/partition/roles/image-cache) | +| `ztp` | Zero Touch Provisioning payloads for SONiC switches | [ztp](https://github.com/metal-stack/metal-roles/tree/master/partition/roles/ztp) | ## Ansible Playbooks @@ -244,7 +278,7 @@ The following sections show a reference playbook structure. Each service is depl ### Management Server (`deploy_mgmt_servers.yaml`) -The management servers run the CI runners, image caches, DHCP server, ZTP, `metal-bmc`, and Pixiecore and optional services like Tailscale. This is the central bootstrap host — once deployed, it enables automated provisioning of all other partition components. +The management servers run the CI runners, image cache, DHCP server, ZTP web server, `metal-bmc` and Pixiecore, plus optional services such as [Tailscale](../../05-Concepts/03-Network/03-tailscale.md). This is the central bootstrap host — once deployed, it enables automated provisioning of all other partition components. ```yaml --- @@ -252,8 +286,10 @@ The management servers run the CI runners, image caches, DHCP server, ZTP, `meta hosts: mgmtservers roles: - name: ansible-common + tags: always - name: metal-roles/common/roles/defaults - name: metal-roles/partition/roles/mgmt-server + - name: metal-roles/partition/roles/lvm - name: metal-roles/partition/roles/dhcp - name: metal-roles/partition/roles/ztp - name: metal-roles/partition/roles/metal-bmc @@ -262,9 +298,13 @@ The management servers run the CI runners, image caches, DHCP server, ZTP, `meta - name: artis3n.tailscale ``` +:::info +The `lvm` role prepares the local volumes that the image cache uses. `artis3n.tailscale` is a third-party Galaxy role and only needed if you expose services via Tailscale; drop it otherwise. +::: + ### Management Switches (`deploy_mgmt_switches.yaml`) -Deploys SONiC on management leaves and management spines. These switches form the out-of-band management network and relay DHCP requests from production switches and worker servers to the management server. The `sonic-config` is used to configure FRR routing, BGP peering, and the DHCP relay agent on these switches. +Deploys the SONiC configuration on management leaves and management spines. These switches form the out-of-band management network and relay DHCP requests from production switches and worker servers to the management server. The `sonic-config` role configures FRR routing, BGP peering, and the DHCP relay agent on these switches. ```yaml --- @@ -272,31 +312,38 @@ Deploys SONiC on management leaves and management spines. These switches form th hosts: mgmtleaves,mgmtspines roles: - name: ansible-common + tags: always - name: metal-roles/common/roles/defaults - name: metal-roles/partition/roles/sonic-config ``` ### Production Spines and Exits (`deploy_spines_exits.yaml`) -Deploys SONiC on spine and exit switches in disjoint groups (e.g. odd/even) to avoid disrupting the entire fabric at once. The `sonic-config` is used to configure FRR routing, BGP underlay, and the DHCP relay agent on exit switches to forward PXE requests to the management server. +Deploys the SONiC configuration on spine and exit switches in disjoint groups (e.g. odd/even) to avoid disrupting the entire fabric at once. The `sonic-config` role configures FRR routing, the BGP underlay, and the DHCP relay agent that forwards PXE requests to the management server. ```yaml --- - name: deploy odd spines and exits hosts: odd_switches + any_errors_fatal: true roles: - name: ansible-common + tags: always - name: metal-roles/common/roles/defaults - name: metal-roles/partition/roles/sonic-config - name: deploy even spines and exits hosts: even_switches + any_errors_fatal: true roles: - name: ansible-common + tags: always - name: metal-roles/common/roles/defaults - name: metal-roles/partition/roles/sonic-config ``` +`any_errors_fatal: true` aborts the play as soon as one switch fails, so a broken configuration is not rolled out across the whole group. + The inventory defines the disjoint groups: ```yaml @@ -315,7 +362,7 @@ Deploy the odd group first, verify the fabric is stable, then deploy the even gr ### Production Leaves (`deploy_leaves.yaml`) -Deploys SONiC configuration and `metal-core` on leaf switches. `metal-core` dynamically configures the leaf from the metal-api and relays DHCP requests from the metal-hammer (discovery image) during PXE provisioning. +Deploys the SONiC configuration and `metal-core` on leaf switches. `metal-core` receives switch configuration from the metal-api via gRPC and reconciles the ports of allocated machines onto the leaf, and it relays the DHCP requests of the metal-hammer (discovery image) during PXE provisioning. Again, the same disjoint groups approach is used. Deploy one group, verify stability, then proceed with the next. @@ -323,17 +370,29 @@ Again, the same disjoint groups approach is used. Deploy one group, verify stabi --- - name: deploy even leaves hosts: even_leaves + any_errors_fatal: true roles: - name: ansible-common + tags: always - name: metal-roles/common/roles/defaults - name: metal-roles/partition/roles/sonic-config - name: metal-roles/partition/roles/metal-core -# - name: deploy odd leaves -# hosts: odd_leaves -# ... +- name: deploy odd leaves + hosts: odd_leaves + any_errors_fatal: true + roles: + - name: ansible-common + tags: always + - name: metal-roles/common/roles/defaults + - name: metal-roles/partition/roles/sonic-config + - name: metal-roles/partition/roles/metal-core ``` +:::warning +`metal-core` registers the switch with the metal-api on first start. Verify with `metalctl switch ls` that every leaf shows up and reports its ports before you allocate the first machine — an unregistered leaf cannot be configured for a machine allocation. +::: + ## CI/CD Workflow Add a CI/CD workflow to your repository (`.github/workflows/deploy-partition.yaml`). The example below uses GitHub Actions, but any runner works — the pattern is the same. @@ -359,7 +418,7 @@ on: - leaves env: - ANSIBLE_INVENTORY: inventory/inventory.yaml + ANSIBLE_INVENTORY: inventories/partition.yaml ANSIBLE_FORCE_COLOR: "1" ANSIBLE_JINJA2_NATIVE: "True" @@ -370,12 +429,17 @@ jobs: runs-on: self-hosted container: ghcr.io/metal-stack/metal-deployment-base:v0.22.18 steps: - - uses: actions/checkout@v7 - - run: rm -rf /github/home/.ansible/roles/* + - name: Checkout + uses: actions/checkout@v7 + + - name: Clean ansible dir + run: rm -rf /github/home/.ansible + - run: | printf '%s' "${ANSIBLE_VAULT_PASSWORD}" > ${ANSIBLE_VAULT_PASSWORD_FILE} printf '%s' "${ANSIBLE_PRIVATE_KEY}" > ${ANSIBLE_PRIVATE_KEY_FILE} chmod 400 ${ANSIBLE_PRIVATE_KEY_FILE} + ansible localhost -m metalstack.base.metal_stack_release_vector ansible-playbook deploy_mgmt_servers.yaml --diff env: @@ -383,11 +447,14 @@ jobs: ANSIBLE_VAULT_PASSWORD_FILE: .vault.txt ANSIBLE_PRIVATE_KEY: ${{ secrets.ANSIBLE_PRIVATE_KEY }} ANSIBLE_PRIVATE_KEY_FILE: .ssh-key + + - name: Clean ansible dir after deploy + run: rm -rf /github/home/.ansible ``` Copy this job block for each remaining phase, adjusting the job name, `if` condition, and playbook name. -Each phase is independent and can be deployed separately via workflow dispatch. +Each phase is independent and can be deployed separately via workflow dispatch. Note the `rm -rf /github/home/.ansible/*` step: it clears previously downloaded role artifacts so that `metal_stack_release_vector` always fetches the roles matching the pinned release instead of reusing a stale cache. ## Deployment Order @@ -412,15 +479,36 @@ The recommended deployment sequence respects service dependencies. Each phase ca 4. Production Leaves → deploy_leaves.yaml ├── SONiC (via sonic-config role) └── metal-core +``` + +## Verifying the Partition + +Once all four phases have run, verify from your workstation against the metal-api: +```bash +# The partition must be known to the metal-api +metalctl partition ls + +# Every leaf, spine and exit switch registered by metal-core +metalctl switch ls + +# Machines discovered by metal-bmc and reported by metal-hammer. +# Freshly discovered machines appear in state "Waiting" / "Available". +metalctl machine ls ``` +If machines never appear, the PXE path is the usual culprit: check that the DHCP relay forwards from the PXE VLAN to the management server, that Pixiecore is reachable, and that the image cache serves the metal-hammer artifacts. The [troubleshooting guide](../06-troubleshoot.md) walks through the individual steps; `metalctl machine console ` gives you the machine's console via metal-console. + ## Updating Components To update metal-stack components: -1. **Update the release version** in `inventory/group_vars/all/release_vector.yaml`. Do not skip versions. -2. **Commit and push** the change to your deployment repository -3. **Trigger the deployment** via GitHub Actions workflow dispatch -4. The `metal_stack_release_vector` Ansible module fetches the latest compatible component versions from the OCI registry -5. Wait for the pipeline to finish +1. **Update the release version** in `inventories/group_vars/all/release_vector.yaml`. Do not skip minor versions — consult the release notes for the target version first. +2. **Commit and push** the change to your deployment repository. +3. **Trigger the deployment** via workflow dispatch — control plane first, then the partition. +4. The `metal_stack_release_vector` Ansible module fetches the component versions and the matching Ansible roles from the OCI registry and verifies their signatures. +5. Wait for the pipeline to finish and verify as described above. + +:::warning +The control plane and its partitions must stay within the same metal-stack release. Always roll out the control plane first, then the partitions — never the other way around. For switch OS upgrades there is a dedicated `sonic-upgrade` role; see [Maintenance](../04-maintenance.md). +::: diff --git a/docs/04-For Operators/03-Deployment/05_gardener.md b/docs/04-For Operators/03-Deployment/05_gardener.md new file mode 100644 index 00000000..436b87ee --- /dev/null +++ b/docs/04-For Operators/03-Deployment/05_gardener.md @@ -0,0 +1,698 @@ +--- +slug: /deployment/gardener +title: Gardener (KCLM) +sidebar_position: 5 +--- + +# Deploying Gardener with metal-stack + +This guide shows how to deploy [Gardener](https://gardener.cloud/) on top of your metal-stack infrastructure using the [`gardener-*` Ansible roles](https://github.com/metal-stack/metal-roles/tree/master/control-plane) from the [metal-roles](https://github.com/metal-stack/metal-roles) repository. Gardener with metal-stack turns your bare-metal servers into a Kubernetes-as-a-Service platform where teams can self-serve clusters. + +This guide assumes you are already familiar with Gardener's core concepts — [Garden](https://gardener.cloud/docs/gardener/concepts/operator/), [Seed](https://gardener.cloud/docs/gardener/concepts/gardenlet/), [Shoot](https://gardener.cloud/docs/gardener/concepts/apiserver/) and [CloudProfile](https://gardener.cloud/docs/gardener/concepts/apiserver/#cloudprofiles) — and have already completed the [Control Plane](./03_control-plane.mdx) and [Partition](./04_partition.md) deployment guides. For an overview of how Gardener integrates with metal-stack at a conceptual level, see the [Gardener concepts guide](../../05-Concepts/04-Kubernetes/02-gardener.md); for the reasoning behind choosing Gardener over Cluster API, see the [KCLM comparison](../../05-Concepts/04-Kubernetes/01-kclm.md#two-approaches-one-infrastructure). + +:::warning +A partition with registered, allocatable machines is a hard prerequisite. Gardener will happily accept a `Shoot`, but the `provider-metal` extension can only reconcile it if the metal-api can allocate machines and firewalls of the requested sizes and images in the requested partition. Verify with `metalctl machine ls` before you start. +::: + +The upstream [Gardener landscape setup guide](https://gardener.cloud/docs/gardener/deployment/setup_gardener/) describes the same building blocks — operator, `Garden`, extensions, `CloudProfile`, DNS secrets, `Gardenlet`, `ManagedSeed` — but expects you to render and apply the manifests yourself. The `gardener-*` roles are a thin, opinionated automation layer over exactly those building blocks, pre-wired for metal-stack: each role owns one resource and pulls all container images and Helm chart references from the metal-stack [release vector](./03_control-plane.mdx#releases-and-ansible-role-dependencies). + +:::tip +The [mini-lab](https://github.com/metal-stack/mini-lab) contains a working, minimal Gardener deployment (`deploy_gardener.yaml` plus `inventories/group_vars`) that uses the very same roles. It is the fastest way to see a complete, runnable parametrization. + +It is **not** a production reference: it runs on a single kind cluster, fakes the `kube-system/shoot-info` ConfigMap, patches the Istio load balancer status by hand, deploys MinIO and PowerDNS as in-cluster stand-ins for object storage and DNS, and skips managed seeds entirely. Use it to understand _how_ the roles interact, not _what_ to configure. +::: + +## Repository structure after this section + +The following files are added to the repository structure from the previous sections: + +```text +. +├── deploy_gardener.yaml # Gardener deployment playbook +├── inventories +│ ├── control-plane.yaml # unchanged, reused for Gardener +│ └── group_vars +│ ├── all/ +│ │ └── release_vector.yaml # unchanged +│ └── control-plane/ +│ ├── common.yaml # updated: host provider, stage name +│ └── gardener/ +│ ├── operator.yaml # operator + virtual garden + backup + dns +│ ├── extensions.yaml # provider/OS/CNI/shoot extensions +│ ├── cloud_profile.yaml # CloudProfile +│ ├── projects.yaml # Projects +│ ├── gardenlet.yaml # first (unmanaged) Seed +│ ├── shoots.yaml # Shoots / shooted seeds +│ ├── managed_seeds.yaml # optional: ManagedSeeds +│ └── secrets.yaml # vault-encrypted credentials +└── .github/ + └── workflows/ + └── deploy-control-plane.yaml # updated: adds a gardener job +``` + +## Architecture Overview + +The metal-roles deploy Gardener in the **virtual Garden** pattern described upstream: the `gardener-operator` runs on your _runtime cluster_ and reconciles a `Garden` resource, which spins up a nodeless _virtual Garden_ cluster hosting the Gardener API (`Shoot`, `Seed`, `Project`, `CloudProfile`, …). The gardenlet also runs on the runtime cluster and registers it as the first, unmanaged `Seed` (a "soil"). + +```mermaid +graph TD + subgraph RT["Runtime cluster (= soil / first Seed)"] + OP["gardener-operator"] + EXTRES["Extension resources"] + GLPOD["gardenlet pods"] + end + subgraph VG["Virtual Garden (nodeless)"] + API["kube-apiserver + gardener-apiserver"] + CP["CloudProfile metal"] + PR["Projects"] + GLRES["Gardenlet resource"] + SEEDA["Seed seed-a"] + end + subgraph SEED["Shooted seed seed-a (bare metal)"] + USH["End-user shoot control planes"] + end + OP -->|reconciles Garden| API + OP -->|registers| EXTRES + EXTRES -->|ControllerRegistrations| API + GLRES -->|deploys| GLPOD + GLPOD -->|registers soil as Seed| API + CP --> SEEDA + PR --> SEEDA + SEEDA -->|ManagedSeed| SEED + SEED --> USH +``` + +The soil is reserved for _infrastructure_ shoots. Those shoots are turned into Gardener-managed Seeds via `ManagedSeed`, and end-user shoot control planes are hosted there. This is the recommended upstream pattern and the one the roles are built for. + +**Order matters.** Every role except `gardener-operator` and `gardener-extensions` obtains a kubeconfig for the virtual Garden through the `virtual_garden_kubeconfig` module, which only works once the operator has created the `Garden` and `gardener-virtual-garden-access` has deployed the token-requestor secret: + +| Role | Applies to | Deploys | Requires | +| -------------------------------- | --------------- | ---------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------- | +| `gardener-operator` | runtime cluster | `garden` namespace, backup + DNS provider secrets, operator Helm chart, `Garden` resource | Runtime cluster; cert-manager only if the dashboard is enabled | +| `gardener-extensions` | runtime cluster | one `operator.gardener.cloud/v1alpha1` `Extension` per enabled extension | `gardener-operator` | +| `gardener-virtual-garden-access` | runtime cluster | `ManagedResource` + token-requestor secret that yields a rotating virtual-Garden kubeconfig, plus optional `ClusterRoleBinding`s for OIDC auth | `Garden` reconciled | +| `gardener-cloud-profile` | virtual Garden | `CloudProfile` named `metal` | virtual-garden-access | +| `gardener-projects` | virtual Garden | `Project` resources | virtual-garden-access | +| `gardener-gardenlet` | virtual Garden | internal/default domain secrets, backup secret, `Gardenlet` resource (first Seed) | virtual-garden-access, extensions | +| `gardener-shoots` | virtual Garden | provider `Secret` + `CredentialsBinding` + `Shoot` per entry | `CloudProfile`, `Project`, a ready `Seed` | +| `gardener-managed-seeds` | virtual Garden | backup secret + `ManagedSeed` per entry | a reconciled shooted seed | + +Two optional roles are not part of the minimal setup but are worth knowing about: `gardener-monitoring-certs` (wildcard monitoring certificates for soil and seeds, requires reachable seed API servers) and `gardener-logging` (ships seed logs to the central metal-stack Loki, requires the `logging` role). + +## Prerequisites + +Before deploying Gardener, ensure the following is in place. This mirrors the [upstream prerequisites](https://gardener.cloud/docs/gardener/deployment/setup_gardener/#prerequisites), with the metal-stack specifics added: + +- **A runtime cluster** — An existing Kubernetes cluster hosting the Gardener control plane. It can be the same cluster as your metal-stack control plane, but a dedicated cluster is recommended for production. See [Bootstrap Infrastructure](./02_bootstrap-infrastructure.md). +- **An ingress controller** — The roles assume `ingress-nginx` (the `Garden` and `Gardenlet` templates hardcode `ingress.controller.kind: nginx`). The virtual Garden API server itself is exposed through the Istio gateway that the operator deploys. +- **A DNS zone plus credentials** — Gardener needs to create records for the virtual Garden API server, the shoot internal and default domains, and seed ingress. Supported provider types come from the deployed DNS extensions (e.g. `google-clouddns` via `provider-gcp`, `powerdns` via the `dns-powerdns` extension). +- **A backup bucket** — S3-compatible or GCP object storage for the virtual Garden etcd and for every Seed. `gardener-operator` **asserts** that `gardener_operator_backup_infrastructure.provider` is either `gcp` or `S3`. +- **cert-manager with a DNS-solving `ClusterIssuer`** — Only required if you let the operator deploy the Gardener dashboard; the role then requests a wildcard certificate for the ingress domain. +- **ACME account** — The `shoot-cert-service` extension is enabled by default and requires `gardener_extension_shoot_cert_service_issuer_email` to be set (asserted). +- **metal-stack API credentials** — An admin/edit HMAC key for the metal-api, handed to shoots via the provider secret. +- **Cluster networking facts** — `gardener-operator` derives the runtime cluster's node, pod and service CIDRs. For `metal_control_plane_host_provider: metal` it reads them from the `kube-system/shoot-info` ConfigMap (present in every Gardener-managed shoot); for `gcp` it queries `gcloud` and additionally needs `gcp_cluster_name` and `gcp_region`. + +:::warning +`metal_control_plane_host_provider` has **no default** and is asserted to be `metal` or `gcp`. If your runtime cluster is neither a metal-stack shoot nor a GKE cluster, you must provide a `kube-system/shoot-info` ConfigMap with `nodeNetwork`, `podNetwork` and `serviceNetwork` yourself — this is exactly what the mini-lab does in its playbook's `pre_tasks`. +::: + +## Step 1: Add the Playbook + +Create `deploy_gardener.yaml` in your repository root. It reuses the `control-plane` inventory from the [Control Plane](./03_control-plane.mdx#inventory) guide (a single `localhost` entry, because everything is applied to Kubernetes) and chains the `gardener-*` roles in the order shown above: + +```yaml +--- +- name: Deploy Gardener + hosts: control-plane + connection: local + gather_facts: false + roles: + - name: ansible-common + tags: always + - name: metal-roles/control-plane/roles/gardener-operator + - name: metal-roles/control-plane/roles/gardener-extensions + - name: metal-roles/control-plane/roles/gardener-virtual-garden-access + - name: metal-roles/control-plane/roles/gardener-cloud-profile + - name: metal-roles/control-plane/roles/gardener-projects + - name: metal-roles/control-plane/roles/gardener-gardenlet + # add once the first seed is ready: + # - name: metal-roles/control-plane/roles/gardener-shoots + # - name: metal-roles/control-plane/roles/gardener-managed-seeds +``` + +You do not need to list `metal-roles/common/roles/defaults` or the `gardener-defaults` role explicitly — each `gardener-*` role pulls them in through its `meta/main.yml` dependencies. Including `ansible-common` is what makes the custom modules (`setup_yaml`, `virtual_garden_kubeconfig`, `discovery_api_k8s`) and filters (`machine_images_for_cloud_profile`, `shoot_admin_kubeconfig`) available. + +:::info +Deploy in two passes on a green field: first everything up to `gardener-gardenlet`, verify that the `Garden` reports `RuntimeComponentsHealthy` and `VirtualComponentsHealthy` and that the `Seed` becomes `GardenletReady`, then enable `gardener-shoots`. `gardener-shoots` waits for each `Shoot` to report `lastOperation.state: Succeeded` (36 retries, 10 s apart by default) and fails if no Seed can host it. +::: + +To inspect the landscape afterwards, obtain a kubeconfig for the virtual Garden the same way the roles do: + +```yaml +post_tasks: + - name: Get kubeconfig for virtual garden access + virtual_garden_kubeconfig: + garden_name: "{{ gardener_defaults_garden_name }}" + + - name: Write it out for manual inspection + ansible.builtin.copy: + content: "{{ virtual_garden_kubeconfig }}" + dest: .virtual-garden-kubeconfig + mode: "0600" +``` + +## Step 2: Configure Group Variables + +All Gardener configuration lives under `inventories/group_vars/control-plane/gardener/`. Each file maps to a role. Container image names, image tags and Helm chart references are resolved from the release vector, so you normally only set the variables shown here. Sensitive values (HMAC keys, service account JSONs, ACME private keys) belong in an Ansible vault file. + +Two shared variables must be set in `inventories/group_vars/control-plane/common.yaml`: + +```yaml +# Names the Gardener landscape. gardener_defaults_garden_name defaults to this +# value, so it also becomes the Garden resource name and the metalControlPlanes +# key in the CloudProfile. You already set this in the control plane guide. +metal_control_plane_stage_name: demo + +# Mandatory and asserted: "metal" or "gcp". Determines how the runtime cluster +# CIDRs are discovered and is rendered as the Seed's provider type. +metal_control_plane_host_provider: metal +``` + +:::tip +`gardener_defaults_garden_name` defaults to `{{ metal_control_plane_stage_name }}` and is inherited by every role (`gardener_operator_garden_name`, `gardener_cloud_profile_garden_name`, …). Keep the default unless you have a reason to diverge — the gardenlet name must match the Seed name across upgrades. +::: + +### operator.yaml — Operator, virtual Garden, backup and DNS + +This role creates the `garden` namespace, installs the operator Helm chart and applies the `Garden` resource. Backup and DNS configuration belong to the same role because both are referenced by the `Garden`. + +```yaml +# --- Virtual Garden ------------------------------------------------------- +# Domain under which the virtual Garden kube-apiserver is exposed through Istio. +gardener_operator_virtual_garden_public_dns: gardener-kube-apiserver.{{ metal_control_plane_ingress_dns }} + +# Domain for runtime-cluster ingresses (monitoring, dashboard). MANDATORY, no default. +gardener_operator_ingress_dns_domain: k8s. + +# Storage class for the virtual Garden etcd volumes (20Gi main, 10Gi events). +# If unset, the cluster's default storage class is used. +gardener_operator_virtual_garden_etcd_storage_class: csi-lvm + +# Renders spec.runtimeCluster.provider.region in the Garden resource. +gardener_operator_runtime_cluster_provider: local + +# Multi-replica etcd + control plane for the virtual Garden. +gardener_operator_high_availability_control_plane: true + +# --- etcd backup (asserted: provider must be "gcp" or "S3") --------------- +gardener_operator_backup_infrastructure: + provider: S3 + bucket: my-garden-backup-bucket + region: europe-west3 + +gardener_operator_backup_infrastructure_secret: + endpoint: "{{ garden_backup_endpoint | b64encode }}" + accessKeyID: "{{ garden_backup_access_key | b64encode }}" + secretAccessKey: "{{ garden_backup_secret_key | b64encode }}" + +# --- DNS providers -------------------------------------------------------- +gardener_operator_dns_providers: + - name: powerdns + type: powerdns + secretData: + apiKey: "{{ powerdns_api_key | b64encode }}" + server: "{{ powerdns_server | b64encode }}" + +# --- Dashboard (optional) ------------------------------------------------- +gardener_operator_dashboard_enabled: false +``` + +#### Notes on the operator configuration + +- `gardener_operator_backup_infrastructure_secret` is applied verbatim as the `data:` of the `virtual-garden-etcd-main-backup-secret`, so all values must already be base64-encoded. The same applies to each DNS provider's `secretData`. Consult the [Gardener etcd backup secret examples](https://gardener.cloud/docs/gardener/deployment/setup_gardener/#garden) for the expected keys per provider. +- `provider: S3` requires the `backup-s3` extension, `provider: gcp` the `provider-gcp` extension (see below) — the extension is what reconciles the `BackupBucket` in the runtime cluster. +- The `Garden` template only renders `spec.runtimeCluster.ingress` and `spec.virtualCluster.dns` **if `gardener_operator_dns_providers` is non-empty**, and always uses `gardener_operator_dns_providers[0].type` as the provider for both. Put your primary provider first. +- The virtual Garden service CIDR is fixed to `100.64.0.0/13` and the maintenance window to `220000+0100`–`230000+0100` by the template. Make sure `100.64.0.0/13` does not overlap with your runtime cluster or partition networks. +- Enabling the dashboard additionally requires `gardener_operator_wildcard_ingress_certificate_cluster_issuer` plus a cert-manager `ClusterIssuer`; the role then blocks until the wildcard certificate secret exists (up to 10 minutes). For OIDC login set `gardener_operator_dashboard_oidc_issuer_url`, `..._client_id`, `..._client_id_public` and `..._client_secret`. +- If automatic DNS creation is not available, create the A record for `gardener_operator_virtual_garden_public_dns` manually, pointing at the external address of the `istio-ingressgateway` service that the operator deploys. For purely local setups the role offers `gardener_operator_expose_virtual_garden_through_ingress_nginx: true` as an alternative — that is what the mini-lab uses, and it is not meant for production. + +### extensions.yaml — Provider and shoot extensions + +This role applies one `operator.gardener.cloud/v1alpha1` `Extension` resource per enabled extension into the **runtime** cluster. The operator then translates each into a `ControllerDeployment` and `ControllerRegistration` in the virtual Garden, exactly as described in the [upstream extension registration docs](https://gardener.cloud/docs/gardener/extensions/registration/). + +Every extension follows the same pattern: `gardener_extension__enabled` toggles it, and an assert makes sure the corresponding Helm chart reference resolved from the release vector. These are **enabled by default**: `provider-metal`, `provider-gcp`, `os-metal`, `networking-calico`, `networking-cilium`, `shoot-cert-service`, `shoot-dns-service`. All others default to `false`. + +```yaml +# --- Infrastructure provider (the essential one) -------------------------- +gardener_extension_provider_metal_enabled: true + +# Machine images offered to Shoot workers. Defaults to metal_api_images, so it +# stays in sync with the images you registered in the metal-api. +gardener_extension_provider_metal_machine_images: "{{ metal_api_images | default([]) }}" + +# Shoot etcd: storage class and backup cadence +gardener_extension_provider_metal_etcd_storage_class_name: csi-lvm +gardener_extension_provider_metal_etcd_backup_schedule: "0 */2 * * *" +gardener_extension_provider_metal_etcd_delta_snapshot_period: "5m" + +# Defaults injected by the admission controller when a Shoot omits CIDRs +gardener_extension_provider_metal_admission_default_pods_cidr: 10.248.64.0/18 +gardener_extension_provider_metal_admission_default_services_cidr: 10.248.192.0/18 + +# --- Operating system ----------------------------------------------------- +# One OperatingSystemConfig type is registered per entry. +# Default: [ubuntu, debian, nvidia]. Trim it to what your CloudProfile offers. +gardener_extension_os_metal_types: + - ubuntu + - debian + +# --- CNI: keep only what you actually offer in the CloudProfile ----------- +gardener_extension_networking_cilium_enabled: true +gardener_extension_networking_calico_enabled: false + +# --- Shoot services ------------------------------------------------------- +# shoot-cert-service is enabled by default and its issuer email is asserted. +gardener_extension_shoot_cert_service_issuer_email: support@example.com +gardener_extension_shoot_cert_service_issuer_private_key: "{{ acme_account_private_key }}" + +# DNS for shoot API servers and shoot-owned records +gardener_extension_dns_powerdns_enabled: true + +# --- Backup provider matching gardener_operator_backup_infrastructure ----- +gardener_extension_backup_s3_enabled: true + +# --- Optional add-ons ----------------------------------------------------- +gardener_extension_csi_driver_lvm_enabled: true +gardener_extension_acl_enabled: false +gardener_extension_audit_enabled: false + +# Not needed unless your runtime cluster is GKE +gardener_extension_provider_gcp_enabled: false +``` + +#### Notes on the extensions + +- If you do not run on GCP, disable `provider-gcp` explicitly — it is on by default and would otherwise register an unusable `DNSRecord/google-clouddns` and `BackupBucket/gcp` handler. +- `provider-metal` embeds an `imageVectorOverwrite` pinning the metal-stack components deployed into shoots (`metal-ccm`, `firewall-controller-manager`, `machine-controller-manager-provider-metal`, `csi-lvm-*`, `droptailer`, `node-init`) to the versions from your release vector. This is why shoot components stay consistent across the fleet. +- `os-metal` also accepts `nvidia` for GPU worker groups — see [GPU Workers](./07-gpu-workers.md). +- Some variables were renamed and the role **fails hard** if you still use the old names (e.g. `gardener_cert_management_issuer_email` → `gardener_extension_shoot_cert_service_issuer_email`). The failure message names the replacement. +- The `duros` extension defaults still point at a pre-release chart; do not enable it in production. + +For the complete list of extension variables see the [`gardener-extensions` role README](https://github.com/metal-stack/metal-roles/tree/master/control-plane/roles/gardener-extensions). + +### cloud_profile.yaml — Defining your metal-stack infrastructure + +The CloudProfile is a Gardener resource that describes your metal-stack infrastructure capabilities: available Kubernetes versions, machine types, regions, and zones. The `gardener-cloud-profile` role renders this from your group vars into a `CloudProfile` Kubernetes resource. + +```yaml +# URL of the metal-api — critical for Gardener to provision infrastructure +gardener_cloud_profile_metal_api_url: https://api. + +# Firewall images (auto-derived from machine images if not specified) +gardener_cloud_profile_firewall_images_from_machine_images: true +gardener_cloud_profile_firewall_images: + - firewall-ubuntu-3.0 + +# Firewall controller versions +gardener_cloud_profile_firewall_controller_versions: + - version: v2.5.0 + url: https://images.metal-stack.io/firewall-controller/v2.5.0/firewall-controller + classification: supported + +# Available Kubernetes versions for Shoot clusters +gardener_cloud_profile_kubernetes: + versions: + - version: 1.33.13 + - version: 1.34.9 + - version: 1.35.6 + +# Available machine types +gardener_cloud_profile_machine_types: + - name: c1-medium-x86 + cpu: "8" + gpu: "0" + memory: 128Gi + usable: true + storage: + class: standard + type: default + size: 960G + +# Available regions and zones +gardener_cloud_profile_regions: + - name: "{{ metal_region }}" + zones: + - name: demo-rack + +# Partition configuration +gardener_cloud_profile_partitions: + demo-rack: + default-machine-types: + firewall: + - c1-medium-x86 +``` + +#### Notes on the CloudProfile + +- `gardener_cloud_profile_kubernetes` and `gardener_cloud_profile_regions` are the only **asserted** variables of this role. The resulting resource is always named `metal` with `spec.type: metal`; `gardener_cloud_profile_stage_name` becomes the key under `providerConfig.metalControlPlanes`, which is what shoots reference to reach your metal-api. +- `gardener_cloud_profile_metal_api_url` is derived automatically: `https://api.{{ metal_control_plane_gateway_dns }}` when `metal_api_httproute_enabled` is true, otherwise `https://api.{{ metal_control_plane_ingress_dns }}`. Only override it if your metal-api lives elsewhere. +- `gardener_cloud_profile_machine_images` defaults to `metal_api_images`, so CloudProfile and `provider-metal` always agree. With `gardener_cloud_profile_firewall_images_from_machine_images: true` (default) every image carrying the `firewall` feature is additionally offered as a firewall image — the explicit `gardener_cloud_profile_firewall_images` list is then only needed for extras. +- The role maps images to Gardener `machineImages` via `gardener_cloud_profile_os_cri_mapping`, which by default only covers `ubuntu` and `debian`. Add an entry for any further OS (for example `nvidia`) or its versions will be dropped from the CloudProfile. +- Use `gardener_cloud_profile_os_compatibility_mapping` to express kubelet/OS-version constraints, and Gardener's `classification` plus `expirationDate` fields inside `gardener_cloud_profile_kubernetes.versions` to steer deprecation and auto-updates. +- `zones` map to metal-stack partitions. `gardener_cloud_profile_partitions..default-machine-types.firewall` restricts the firewall sizes selectable in that partition; an optional `network-isolation` key enables [isolated clusters](../../05-Concepts/04-Kubernetes/06-isolated-clusters.md). +- By default the role waits until the `CloudProfile` CRD is served by the virtual Garden before applying, which is what makes an initial bootstrap succeed on the first run. + +### projects.yaml — Team isolation + +Gardener `Project`s isolate teams and give them a namespace in the virtual Garden. You need at least one project to create shoots. + +```yaml +gardener_project_defaults: + namespace: garden + owner: admin + protected_toleration: true + members: [] + +gardener_projects: + - name: infrastructure # holds the shooted seeds + description: Infrastructure clusters + - name: prod + owner: alice@example.com + members: + - kind: User + name: bob@example.com + role: admin + roles: [admin] +``` + +With `protected_toleration: true` (the default) the project may schedule shoots onto Seeds tainted with `seed.gardener.cloud/protected`. Keep this on for the project that holds your shooted seeds and consider turning it off for end-user projects, so their shoots never land on the soil. + +:::info +The first project uses `namespace: garden`, which already exists. Additional projects get their own namespace derived from the project name. +::: + +### gardenlet.yaml — Registering the first Seed + +The gardenlet registers your runtime cluster as the first, unmanaged Seed ("soil"). The role also deploys the `internal-domain` and `default-domain` secrets that Gardener uses for shoot DNS records — these are what the upstream guide calls the [DNS setup for internal and external domains](https://gardener.cloud/docs/gardener/deployment/setup_gardener/#dns-setup-for-internal--external-domains). + +```yaml +# All three are MANDATORY and asserted. +gardener_gardenlet_default_dns_domain: k8s. +gardener_gardenlet_default_dns_provider: powerdns +gardener_gardenlet_default_dns_credentials: + apiKey: "{{ powerdns_api_key | b64encode }}" + server: "{{ powerdns_server | b64encode }}" + +gardener_gardenlets: + - name: "{{ gardener_defaults_garden_name }}" + + # Required: every Seed needs its own backup configuration. + backup_infrastructure: + provider: S3 + region: europe-west3 + bucket: my-seed-backup-bucket + backup_infrastructure_secret: + endpoint: "{{ seed_backup_endpoint | b64encode }}" + accessKeyID: "{{ seed_backup_access_key | b64encode }}" + secretAccessKey: "{{ seed_backup_secret_key | b64encode }}" + + # Ingress domain of this Seed; falls back to the default DNS domain. + dns_domain: "soil.{{ gardener_defaults_garden_name }}." + + # Keep the soil invisible to the scheduler so only tolerating shoots + # (your shooted seeds) land here. + visible: false + taints: + - seed.gardener.cloud/protected + + additional_labels: + cluster.metal-stack.io/partition: demo-rack + +# Gardenlet defaults applied to all seeds +# shoot_reconcile_in_maintenance_only: true — only reconcile shoots during their maintenance window +# shoot_respect_sync_period_overwrite: true — respect the syncPeriod on Shoot specs +gardener_gardenlet_defaults: + shoot_reconcile_in_maintenance_only: true + shoot_respect_sync_period_overwrite: true +``` + +#### Notes on the gardenlet + +- The gardenlet name should equal the Seed name and must remain stable — renaming it creates a new Seed instead of adopting the existing one. +- Pod and service CIDRs default to the runtime cluster's CIDRs, read from the `Garden` resource. Override them per gardenlet with `pods:` / `services:` if they would overlap with shoot networks. +- `taints: [seed.gardener.cloud/protected]` is the default. Combined with `visible: false` this reserves the soil for infrastructure shoots, matching the [upstream recommendation](https://gardener.cloud/docs/gardener/deployment/setup_gardener/#gardenlet). +- When the gardenlet must run in a **different** cluster than the operator, set `kubeconfigSecretRef` to a manually created kubeconfig secret plus `garden_client_connection.gardenClusterAddress: https://` — see [Deploy Gardenlet via Operator](https://gardener.cloud/docs/gardener/deployment/deploy_gardenlet_via_operator/#remote-clusters). +- `shoot_reconcile_in_maintenance_only: true` and `shoot_respect_sync_period_overwrite: true` are the defaults, so shoots only reconcile inside their maintenance window and honour the ignore annotation. Adjust `shoot_concurrent_syncs` (default `20`) for large seeds. + +### shoots.yaml — Shoot clusters and shooted seeds + +Each entry produces a provider `Secret`, a `CredentialsBinding` and a `Shoot`. To scale the landscape you create "shooted seeds" here first — infrastructure shoots that are turned into Seeds by the `gardener-managed-seeds` role. + +```yaml +# HMAC key that shoots use against the metal-api. Base64-encoded into the +# per-shoot provider secret. +gardener_shoot_default_metal_api_hmac: "{{ metal_api_admin_key }}" + +# The role waits for each shoot to reach lastOperation.state == Succeeded. +gardener_shoot_rollout_wait_enabled: true +gardener_shoot_rollout_wait_retries: 36 +gardener_shoot_rollout_wait_delay: 10 + +gardener_shoots: + - name: seed-a + seed_name: "{{ gardener_defaults_garden_name }}" # scheduled onto the soil + project_id: + purpose: infrastructure + region: "{{ metal_region }}" + partition: demo-rack + networks: + - internet + - + k8s_version: "1.34.9" + networking_type: cilium + networking_pod_cidr: 10.240.0.0/13 + networking_service_cidr: 10.248.0.0/18 + worker_groups: + - worker_count: 3 + worker_size: c1-medium-x86 + worker_cri: containerd + worker_max_surge: 1 + worker_max_unavailable: 0 + worker_image: + name: debian + version: "12.0" + firewall_size: c1-medium-x86 + firewall_image: firewall-ubuntu-3.0 + high_availability_control_plane: node + csi_driver_lvm_extension: + enabled: true + default_storage_class: csi-lvm + + # Makes this shoot eligible to become a Seed and sizes its API server. + managed_seed: + tolerations: + - key: seed.gardener.cloud/protected + api_server: + replicas: 3 + autoscaler: + min_replicas: 1 + max_replicas: 5 +``` + +#### Notes on shoot configuration + +- `project_id` is the metal-stack project UUID. It is written into the `cluster.metal-stack.io/project` annotation and the `InfrastructureConfig`, and determines which metal-stack project the machines, IPs and firewalls are allocated in. +- `networks` must list the metal-stack network IDs the firewall attaches to. `internet` is required for external reachability; add the partition's private network. +- `worker_size`, `firewall_size` and `firewall_image` must exist in the CloudProfile (and, for the firewall, in the partition's `firewallTypes`). +- `high_availability_control_plane` accepts `node` or `zone`. With a single partition per zone, use `node`. +- `namespace` defaults to `garden`; set it to the project namespace when the shoot belongs to a non-default project. +- Use `credentials_binding_name` to reference an existing binding instead of letting the role create one. The old `secret_binding_name` key is rejected by an assert. +- Optional per-shoot keys include `audit_policy` (rendered into a ConfigMap and wired into the kube-apiserver), `structured_auth_config`, `audit_extension_splunk`, `machine_creation_timeout`, `storage_class_name` and `control_plane_feature_gates`. + +:::warning +`gardener_shoot_rollout_wait_enabled` makes the playbook block for up to six minutes per shoot by default. For landscapes with many shoots, either raise the retries or disable the wait and monitor reconciliation separately. +::: + +### managed_seeds.yaml — Turning shooted seeds into Seeds + +A `ManagedSeed` installs a gardenlet into an existing shoot and registers it as a Seed. This is how the landscape scales: end-user shoot control planes then run on these Gardener-managed seeds instead of on the soil. The `name` **must** match the shoot name defined in `shoots.yaml`. + +```yaml +# Mandatory and asserted — used for the seed's internal DNS domain. +gardener_managed_seed_default_dns_domain: k8s. +gardener_managed_seed_default_dns_provider: powerdns + +gardener_managed_seed_defaults: + visible: true + excess_capacity_reservation: true + external_traffic_policy: Local + +gardener_managed_seeds: + - name: seed-a # must match a shoot from shoots.yaml + region: "{{ metal_region }}" + pod_cidr: 10.240.0.0/13 # must match the shoot's networking + service_cidr: 10.248.0.0/18 + ingress_domain: ingress.seed-a.k8s. + logging_enabled: false + backup_infrastructure: + provider: S3 + region: europe-west3 + bucket: my-seed-a-backup-bucket + backup_infrastructure_secret: + endpoint: "{{ seed_backup_endpoint | b64encode }}" + accessKeyID: "{{ seed_backup_access_key | b64encode }}" + secretAccessKey: "{{ seed_backup_secret_key | b64encode }}" +``` + +#### Notes on managed seeds + +- `pod_cidr` and `service_cidr` must be the CIDRs of the underlying shoot, and they must not overlap with any shoot hosted on that seed. Plan your CIDR ranges before creating seeds. +- `backup_infrastructure_secret` is applied unconditionally, so it must be provided for every managed seed even though the commented example in the role omits it. +- `visible: true` (default) makes the seed eligible for end-user shoots, unlike the soil. `excess_capacity_reservation` keeps spare capacity so new shoot control planes schedule quickly. +- Unlike the soil, the gardenlet bootstraps itself here via a bootstrap token and inherits configuration from the parent gardenlet (`mergeWithParent`). + +## Step 3: Run the Deployment + +Run the playbook exactly like the control plane deployment, using the deployment base image so that the release vector and roles are fetched and verified: + +```bash +export KUBECONFIG= + +docker run --rm -it \ + -v $(pwd):/workdir \ + --workdir /workdir \ + -e KUBECONFIG="${KUBECONFIG}" \ + -e K8S_AUTH_KUBECONFIG="${KUBECONFIG}" \ + -e ANSIBLE_INVENTORY=inventories/control-plane.yaml \ + -e ANSIBLE_JINJA2_NATIVE=True \ + ghcr.io/metal-stack/metal-deployment-base:${METAL_VERSION} \ + /bin/bash -ce \ + "ansible -m metalstack.base.metal_stack_release_vector localhost + ansible-playbook deploy_gardener.yaml" +``` + +The roles are idempotent: if a role fails, fix the configuration and re-run — only missing changes are applied. The mandatory-variable asserts fail fast with `not all mandatory variables given, check role documentation`, so most misconfigurations surface before anything is applied. + +### Verifying the landscape + +Against the **runtime** cluster: + +```bash +kubectl get garden +# NAME K8S VERSION GARDENER VERSION LAST OPERATION RUNTIME VIRTUAL API SERVER OBSERVABILITY +# demo 1.34.9 v1.125.0 Succeeded True True True True + +kubectl get extensions.operator.gardener.cloud +``` + +Against the **virtual Garden** (using the kubeconfig from the playbook's `post_tasks`): + +```bash +export KUBECONFIG=.virtual-garden-kubeconfig +kubectl get cloudprofile metal +kubectl get seeds +kubectl get shoots -A +``` + +Wait for the `Garden` conditions `RuntimeComponentsHealthy` and `VirtualComponentsHealthy`, then for the `Seed` condition `GardenletReady`, before enabling the shoot roles. + +### CI/CD Pipeline + +The Gardener deployment is part of the same workflow as the control plane deployment. Add a `gardener` job to your `.github/workflows/deploy-control-plane.yaml`: + +```yaml +--- +name: Deploy control plane + +on: + workflow_dispatch: + inputs: + deploy-control-plane: + description: "Which control-plane target to deploy" + required: true + type: choice + options: + - metal-stack + - gardener + +permissions: + contents: read + +env: + ANSIBLE_INVENTORY: inventories/control-plane.yaml + ANSIBLE_FORCE_COLOR: "1" + ANSIBLE_JINJA2_NATIVE: "True" + + # Update these with your actual values + CLUSTER_ID: + DEFAULT_PROJECT_ID: + + KUBECONFIG: /tmp/.kubeconfig + +jobs: + gardener: + name: Deploy Gardener + if: ${{ inputs.deploy-control-plane == 'gardener' }} + + runs-on: ubuntu-latest + container: ghcr.io/metal-stack/metal-deployment-base:v0.22.18 + + steps: + - name: Checkout + uses: actions/checkout@v7 + + - run: | + metal ctx add demo --api-token ${METALSTACKCLOUD_API_TOKEN} --default-project ${DEFAULT_PROJECT_ID} --activate + metal cluster kubeconfig ${CLUSTER_ID} + + printf '%s' "${ANSIBLE_VAULT_PASSWORD}" > ${ANSIBLE_VAULT_PASSWORD_FILE} + + ansible localhost -m metalstack.base.metal_stack_release_vector + ansible-playbook deploy_gardener.yaml --diff + env: + METALSTACKCLOUD_API_TOKEN: ${{ secrets.METALSTACKCLOUD_API_TOKEN }} + ANSIBLE_VAULT_PASSWORD: ${{ secrets.ANSIBLE_VAULT_PASSWORD }} + ANSIBLE_VAULT_PASSWORD_FILE: .vault.txt +``` + +The job follows the same pattern as the [control-plane workflow](./03_control-plane.mdx#github-action) — only the playbook differs. If you host Gardener on a **dedicated** runtime cluster (the recommended [Option 2](./02_bootstrap-infrastructure.md#option-2-dedicated-initial-clusters)), point `CLUSTER_ID` at that cluster instead of the metal-stack control plane cluster. + +## How the Pieces Connect + +Understanding the data flow between metal-stack and Gardener helps troubleshoot issues: + +```mermaid +graph LR + MAI["metal_api_images"] -->|default| CP["CloudProfile metal"] + MAI -->|default| PE["provider-metal Extension"] + CP -->|referenced by| SH["Shoot"] + HMAC["gardener_shoot_default_metal_api_hmac"] -->|provider Secret + CredentialsBinding| SH + SH -->|reconciled by| PE + PE -->|endpoint from CloudProfile| MA["metal-api"] + MA -->|machines, networks, IPs, firewalls| PART["Partition"] +``` + +| Connection | Configuration Variable | Purpose | +| ----------------- | ----------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------- | +| metal-api URL | `gardener_cloud_profile_metal_api_url` | Endpoint the provider extension calls to provision resources | +| Control plane key | `gardener_cloud_profile_stage_name` | Key under `metalControlPlanes` that shoots resolve to the metal-api | +| Machine images | `metal_api_images` → `gardener_cloud_profile_machine_images` / `gardener_extension_provider_metal_machine_images` | Keeps CloudProfile and extension in sync | +| HMAC secret | `gardener_shoot_default_metal_api_hmac` | Shoot-to-metal-api authentication via the per-shoot provider secret | +| Machine types | `gardener_cloud_profile_machine_types` | Machine sizes selectable for workers and firewalls | +| Regions/zones | `gardener_cloud_profile_regions` | Region and zone (= partition) placement | +| Partition config | `gardener_cloud_profile_partitions` | Firewall types and network isolation per partition | +| Shoot networks | `gardener_shoots[].networks` | metal-stack network IDs the firewall attaches to | +| Shoot partition | `gardener_shoots[].partition` | Which metal-stack partition hosts the workers | + +## Troubleshooting + +| Symptom | Likely cause | +| ------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `not all mandatory variables given, check role documentation` | An asserted variable is unset. Check `metal_control_plane_host_provider`, `gardener_operator_backup_infrastructure.provider`, `gardener_operator_ingress_dns_domain`, `gardener_cloud_profile_kubernetes`, `gardener_cloud_profile_regions`, the three `gardener_gardenlet_default_dns_*` variables, `gardener_managed_seed_default_dns_domain` and `gardener_extension_shoot_cert_service_issuer_email`. | +| The role fails with `the variable ... was renamed to ...` | You are using a deprecated variable name; rename it as instructed and remove the old one. | +| `virtual_garden_kubeconfig` times out (120 retries) | The `Garden` is not healthy yet, or `gardener-virtual-garden-access` has not run. Check `kubectl get garden` and the operator logs in the `garden` namespace. | +| `KeyError: 'nodeNetwork'` in the operator role | The runtime cluster has no `kube-system/shoot-info` ConfigMap. Either set `metal_control_plane_host_provider: gcp` or provide the ConfigMap yourself. | +| etcd of the virtual Garden does not reconcile | Add the `druid.gardener.cloud/etcd-druid` finalizer on the `ETCD` resource manually, as noted in the role README. | +| Shoots stay pending | No Seed tolerates them. Check the Seed taints, `visible` setting, and the project's `protected_toleration`. | + +For general deployment issues, see the [troubleshooting guide](../06-troubleshoot.md). diff --git a/docs/04-For Operators/03-Deployment/05_kclm.md b/docs/04-For Operators/03-Deployment/05_kclm.md deleted file mode 100644 index 372fc154..00000000 --- a/docs/04-For Operators/03-Deployment/05_kclm.md +++ /dev/null @@ -1,198 +0,0 @@ ---- -slug: /deployment/kclm -title: Kubernetes Cluster Lifecycle Management (KCLM) -sidebar_position: 5 ---- - -# Kubernetes Cluster Lifecycle Management (KCLM) - -This guide covers the deployment of a Kubernetes Cluster Lifecycle Management (KCLM) solution to use metal-stack as a cloud provider. metal-stack supports three KCLM solutions: - -## KCLM Solutions Overview - -### Gardener - -[Gardener](../../05-Concepts/04-Kubernetes/01-gardener.md) is the **recommended** KCLM solution for metal-stack. It is battle-tested in production for over seven years at financial-sector customers and bundles more day-2 capabilities natively (DNS, backup, audit). Gardener manages entire clusters as Kubernetes-native resources with a strong separation between platform operators and end-users. - -:::tip -We recommend using a **dedicated cluster** for Gardener, separate from the metal-stack initial cluster. While it is technically possible to deploy both metal-stack and Gardener on the same initial cluster, dedicated clusters provide better isolation, clearer operational boundaries, and align with production best practices for critical infrastructure. For guidance on setting up the initial cluster, see the [Bootstrap Infrastructure](./02_bootstrap-infrastructure.md) documentation. -::: - -For more details on Gardener terminology, architecture, operational model, failure domains, and operational features, see the [Gardener concept doc](../../05-Concepts/04-Kubernetes/01-gardener.md). - -#### Deployment Summary - -Gardener can be deployed with the `gardener-*` [Ansible roles](https://github.com/metal-stack/metal-roles/tree/master/control-plane/roles). - -The following data center infrastructure dependencies are treated as given and must be available before deploying Gardener: - -- **DNS** — For cluster domain resolution -- **NTP** — Time synchronization across all nodes -- **ACME** — Certificate authority (for shoot certificates via `shoot-cert-service`) -- **S3-compatible object storage** — For etcd backups with `gardener-extension-backup-s3` -- **Git-Hosting with CI/CD** — You must set up your own Git repository and CI/CD pipeline to manage cluster deployments (see [Fleet Management and GitOps](#fleet-management-and-gitops) below). - -The following dependencies are introduced: - -- CNI: Calico or Cilium -- MetalLB for exposing the Kubernetes API servers of the clusters - -In summary, this results in the following cluster hierarchy: - -- **Garden cluster** — The Gardener control plane (Gardener API server, controller manager, scheduler, admission controller) deployed on a dedicated cluster. -- **Seed** — A cluster running the `gardenlet` agent, connected to the Gardener control plane. Seeds are deployed inside the metal-stack partition and orchestrate cluster provisioning within that site. -- **Shoot** — Every fully provisioned and managed Kubernetes cluster. Shoot control planes run as pods in the Seed namespace, while worker nodes are provisioned as bare-metal machines on metal-stack infrastructure. - -:::tip -We are officially supported by [Gardener dashboard](https://github.com/gardener/dashboard). The dashboard helps you manage Shoots, Seeds, and Projects through a web UI. -::: - -#### Core Controllers - -The Gardener platform consists of the following core controllers, all deployed via the Ansible roles: - -| Component | Responsibility | -| ------------------------------------------- | ---------------------------------------------------------------------------------------- | -| `gardener-operator` | Deploys Gardener components, gardenlets, and extensions; manages platform updates | -| `gardener-apiserver` | Extends the kube-apiserver with Gardener-specific resources (Shoot, Seed, Project, etc.) | -| `gardener-scheduler` | Decides where clusters are placed across the Gardener landscape (Seeds) | -| `gardener-controller-manager` | Reconciles common Gardener resources (projects, controller installations, etc.) | -| `gardenlet` | Agent running on each Seed; orchestrates provisioning of new clusters within that Seed | -| `gardener-resource-manager` | Runs inside Shoots; reconciles desired resources and checks their health | -| `etcd-druid` | etcd cluster operator with built-in backup-restore functionality | -| `machine-controller-manager` | Manages worker node lifecycle (rolling updates, health recreation, scaling) | -| `machine-controller-manager-provider-metal` | Integrates metal-stack machine provisioning API with Gardener's MCM | - -#### Gardener Extensions - -Gardener's extensibility model allows provider-specific reconcilers to be deployed during cluster provisioning. The `gardener-extensions` [Ansible role](https://github.com/metal-stack/metal-roles/tree/master/control-plane/roles/gardener-extensions) deploys the following extensions into the Gardener runtime cluster: - -| Extension | Purpose | -| -------------------------------------- | -------------------------------------------------------------------------------------------------------- | -| `gardener-extension-provider-metal` | IaaS integration — reconciles Infrastructure, ControlPlane, and Worker resources via the metal-stack API | -| `os-metal-extension` | Translates Gardener's generic `OperatingSystemConfig` into cloud-init/ignition userdata | -| `gardener-extension-networking-calico` | Calico CNI extension | -| `gardener-extension-networking-cilium` | Cilium CNI extension as an alternative to Calico | -| `gardener-extension-dns-powerdns` | DNS management via PowerDNS | -| `shoot-dns-service` | DNS service for Shoot clusters | -| `gardener-extension-backup-s3` | etcd backup to S3-compatible object storage | -| `gardener-extension-audit` | Audit logging webhook | -| `gardener-extension-acl` | Access control list management | -| `shoot-cert-service` | Certificate management with Let's Encrypt (supports Shoot-level issuers) | -| `gardener-extension-csi-driver-lvm` | LVM-based CSI driver for local storage | -| `gardener-extension-ontap` | NetApp ONTAP CSI driver | - -Most extensions are enabled/disabled via Ansible variables (e.g., `gardener_extension_provider_metal_enabled`). Key configuration variables for the metal provider include: - -- `gardener_extension_provider_metal_etcd_storage_class_name` — Storage class for Shoot etcds -- `gardener_extension_provider_metal_etcd_backup_schedule` — etcd backup schedule -- `gardener_extension_provider_metal_machine_images` — Machine images (typically matches CloudProfile) -- `gardener_extension_provider_metal_admission_default_pods_cidr` — Default pod CIDR for Shoots -- `gardener_extension_provider_metal_admission_default_services_cidr` — Default services CIDR for Shoots - -For the full variable reference, see the [gardener-extensions README](https://github.com/metal-stack/metal-roles/tree/master/control-plane/roles/gardener-extensions). - -#### Fleet Management and GitOps - -You must set up your own Git repository and CI/CD pipeline to manage cluster deployments. This gives you peer review, audit trails, and rollback capabilities. - -**What you need to build:** - -1. **Git repository** — Store the following as YAML manifests: - - `cloudprofiles/` — CloudProfile definitions (whitelisted regions, machine types, OS images, Kubernetes versions) - - `seeds/` — Seed configurations per data center - - `projects//shoots/` — Per-project Shoot manifests - - `extensions/` — Helm charts for Gardener extensions -2. **CI/CD pipeline** — Deploy manifests from Git to the Gardener API (Virtual Garden). This pipeline is your primary interface for fleet-wide changes. -3. **Branching strategy** — Use separate branches or environments (staging → production) to validate changes before rolling them out fleet-wide. - -**Operational capabilities provided by Gardener:** - -Once your GitOps pipeline is in place, Gardener provides the following day-2 operational features: - -- **CloudProfile validation** — Administrators define allowed regions, machine types, operating systems, and Kubernetes versions. Shoot specs are validated against the CloudProfile before being stored in the Virtual Garden's ETCD. -- **Multi-stage environments** — End-users can label clusters as `evaluation` or `development` to test upcoming Kubernetes versions and auto-upgrades before rolling them out to `production` clusters. -- **Maintenance time windows** — Configurable per Shoot; all day-2 operations (Kubernetes patch updates, machine image updates) are carried out within these windows. -- **Emergency patching** — Administrators can apply fleet-wide changes via image vector overwrites in the Gardener deployment Git repository. Changes must be validated in a dedicated staging environment first. -- **Accidental deletion protection** — Shoot deletion is guarded by specific annotations. ETCD backup retention timeouts are configurable, allowing cluster restoration after accidental deletion. - -### Cluster-API - -[Cluster-API](../../05-Concepts/04-Kubernetes/02-cluster-api.md) is a CNCF project maintained by a Kubernetes SIG that provides declarative cluster management through a management cluster. The metal-stack provider (CAPMS) is **under development** and not yet production-ready. - -The [cluster-api-provider-metal-stack (CAPMS)](https://github.com/metal-stack/cluster-api-provider-metal-stack/) infrastructure provider translates CAPI resources into metal-stack API calls for machine, firewall, and IP allocation. CAPMS is tested against the Kubeadm Bootstrap Provider (CABPK) and uses the Add-on Provider for Helm (CAAPH) for installing CNIs like Calico and the metal-ccm. - -:::warning -Cluster-API with metal-stack is in development and not advised for production use. Please use Gardener for production workloads. We are actively looking for exchange and adopters — if you are interested in using Cluster-API with metal-stack, please [join our community](/community) to help shape future integration efforts. -::: - -For more details on Cluster-API concepts, architecture, operational model, and control plane hosting, see the [Cluster-API concept section](../../05-Concepts/04-Kubernetes/02-cluster-api.md). - -Unlike Gardener, which provides a complete Kubernetes-as-a-Service platform with integrated day-2 operations (DNS, backup, certificate rotation, audit), Cluster-API is a declarative cluster management framework. Operators must assemble their own day-2 tooling — CNI, CCM, DNS, backup, and certificate management — and manage them through GitOps workflows. - -#### Deployment - -Cluster-API with metal-stack is deployed through the [cluster-api-provider-metal-stack (CAPMS)](https://github.com/metal-stack/cluster-api-provider-metal-stack/) infrastructure provider. The [CAPMS reference documentation](../../08-References/Kubernetes/cluster-api-provider-metal-stack/cluster-api-provider-metal-stack.md) covers the deployment in detail. - -**Deployment flow** - -1. **Prepare management cluster** — A Kubernetes cluster to host CAPI and CAPMS providers and cluster state -2. **Install CAPMS** — Deploy the CAPMS provider into the management cluster -3. **Configure `clusterctl`** — Register the metal-stack provider URL and set environment variables (API credentials, project, partition, machine images and sizes, cluster name, Kubernetes version) -4. **Generate and apply cluster manifest** — Use `clusterctl generate cluster` to produce a YAML with `Cluster`, `MetalStackCluster`, `KubeadmControlPlane`, `MachineDeployment`, and `MetalStackMachine` resources, then apply it -5. **Deploy add-ons** — Install CNI (Calico) and `metal-ccm` via `ClusterResourceSet` and CAAPH -6. **Retrieve kubeconfig** — Access the provisioned cluster - -**Network integration** - -Network integration for Cluster-API is currently more manual compared to Gardener. Node networks must be created manually via `metalctl` and provided as environment variables. IP addresses for the control plane also need to be allocated in advance through `metalctl`. Firewall rules are currently static and can be applied to firewall nodes; no automatic firewall controller is in place yet. Automatic network resource allocation is on the roadmap for CAPMS. - -For service exposure, CAPMS uses KubeVIP in BGP mode to allocate and announce public IPs, similar to the MetalLB-based approach in Gardener. - -**Air-gapped environments** - -For air-gapped deployments, follow the [Cluster API Operator air-gapped environment guide](https://cluster-api-operator.sigs.k8s.io/topics/configuration/air-gapped-environtment). All required images must be mirrored to an OCI registry reachable from the management cluster. - -**Fleet management and GitOps** - -You must set up your own Git repository and GitOps operator to manage cluster deployments. - -**What you need to build:** - -1. **Git repository** — Store cluster manifests generated via `clusterctl generate cluster `. Each cluster gets its own set of YAML files containing `Cluster`, `MetalStackCluster`, `KubeadmControlPlane`, `MachineDeployment`, and `MetalStackMachine` resources. -2. **GitOps operator** — Deploy ArgoCD or FluxCD to watch your Git repository and apply manifests to the management cluster, ensuring drift-free declarative delivery. -3. **Per-cluster CI/CD** — Essential components (CNI, CCM) are rolled out on a per-cluster basis. Changes to `MachineTemplate` or `ClusterResourceSet` are staged through the Git repository with standard approval processes. - -**Platform capabilities:** - -- **Cluster migration** — `clusterctl move` enables moving workload cluster resources between management clusters, pausing controllers during the move to prevent worker node loss. - -### Kamaji - -[Kamaji](https://kamaji.clastix.io/) is a Control Plane Manager for Kubernetes that runs control planes as pods within a management cluster, reducing operational overhead and costs. It supports multi-tenancy, high availability, and integrates with Cluster API as a `ControlPlaneProvider`. - -Kamaji allows a similar control plane hosting model as Gardener, where the control plane runs on dedicated infrastructure separate from worker nodes. - -:::warning -Kamaji integrations with metal-stack have not been evaluated in production-grade scenarios. We are actively looking for exchange and adopters — if you are interested in using Kamaji with metal-stack, please [join our community](/community) to help shape future integration efforts. -::: - -#### Kamaji with metal-stack - -Kamaji acts as a `ControlPlaneProvider` with Cluster API, while CAPMS acts as the `InfrastructureProvider`. This setup manages **tenant clusters** on metal-stack infrastructure, combining Kamaji's control plane management with metal-stack's bare-metal provisioning. - -Like Cluster-API, Kamaji is a framework rather than a complete platform — operators must assemble their own day-2 tooling (CNI, CCM, DNS, backup, certificate management) and manage them through GitOps workflows. - -**Deployment** - -1. **Prepare management cluster** — A Kubernetes cluster to host Kamaji and CAPMS providers -2. **Install Kamaji and CAPMS** — Deploy both providers into the management cluster -3. **Create a control plane VIP** — MetalLB assigns a virtual IP for the tenant API server -4. **Generate and apply tenant cluster manifest** — Use `clusterctl generate cluster` to produce a YAML with `Cluster`, `MetalStackCluster`, `KubeadmControlPlane`, `MachineDeployment`, and `MetalStackMachine` resources, then apply it -5. **Deploy add-ons** — Install CNI (Calico) and `metal-ccm` into the tenant cluster - -A working showcase is available in the [`capi-lab`](https://github.com/metal-stack/cluster-api-provider-metal-stack/blob/main/DEVELOPMENT.md#running-the-kamaji-flavor) setup, which extends the `mini-lab` with a Kamaji flavor. See our [blog post](/blog/2026/04-kamaji) for a detailed walkthrough of the architecture and setup. - -**Fleet management and GitOps** - -Since Kamaji with metal-stack uses Cluster-API under the hood, fleet management follows the same pattern as Cluster-API. Tenant cluster manifests are generated via `clusterctl`, stored in Git, and deployed through your CI/CD pipeline. diff --git a/docs/04-For Operators/03-Deployment/06-gpu-workers.md b/docs/04-For Operators/03-Deployment/06-gpu-workers.md deleted file mode 100644 index 3a6938ce..00000000 --- a/docs/04-For Operators/03-Deployment/06-gpu-workers.md +++ /dev/null @@ -1,64 +0,0 @@ ---- -slug: /deployment/gpu-workers -title: GPU Workers -sidebar_position: 6 ---- - -# GPU Workers - -For workloads which require the assistance of GPUs, support for GPUs in bare metal servers was added to metal-stack.io v0.18.0. - -## GPU Operator installation - -With the nvidia image a worker has basic GPU support. This means that the required kernel driver, the containerd shim and the required containerd configuration are already installed and configured. - -To enable `Pods` that require GPU support to be scheduled on a worker node with a GPU, a `gpu-operator' must be installed. -This has to be done by the cluster owner after the cluster is up and running. - -The simplest way to install this operator is as follows: - -```bash -helm repo add nvidia https://helm.ngc.nvidia.com/nvidia -helm repo update - -kubectl create ns gpu-operator -kubectl label --overwrite ns gpu-operator pod-security.kubernetes.io/enforce=privileged - -helm install --wait \ - --generate-name \ - --namespace gpu-operator \ - --create-namespace \ - nvidia/gpu-operator \ - --set driver.enabled=false \ - --set toolkit.enabled=true -``` - -After that `kubectl describe node` must show the gpu in the capacity like so: - -```plain -... -Capacity: - cpu: 64 - ephemeral-storage: 100205640Ki - hugepages-1Gi: 0 - hugepages-2Mi: 0 - memory: 263802860Ki - nvidia.com/gpu: 1 - pods: 510 -... -``` - -With this basic installation, the worker node is ready to process GPU workloads. - -:::warning -However, there is a caveat - only one 'Pod' can access the GPU. If this is all you need, no additional configuration is required. -On the other hand, if you are planning to deploy multiple applications that require GPU support, and there are not that many GPUs available, you will need to configure the `gpu-operator` to allow the GPU to be shared between multiple `Pods`. -::: - -There are several approaches to sharing GPUs, please consult the official Nvidia documentation for further reference. - -- [https://developer.nvidia.com/blog/improving-gpu-utilization-in-kubernetes](https://developer.nvidia.com/blog/improving-gpu-utilization-in-kubernetes) -- [https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/gpu-operator-mig.html](https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/gpu-operator-mig.html) -- [https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/gpu-sharing.html](https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/gpu-sharing.html) - -With this, happy AI processing. diff --git a/docs/04-For Operators/03-Deployment/06_cluster-api.md b/docs/04-For Operators/03-Deployment/06_cluster-api.md new file mode 100644 index 00000000..b8318c45 --- /dev/null +++ b/docs/04-For Operators/03-Deployment/06_cluster-api.md @@ -0,0 +1,100 @@ +--- +slug: /deployment/cluster-api +title: Cluster API (KCLM) +sidebar_position: 6 +--- + +# Cluster API for Kubernetes Cluster Lifecycle Management + +This section covers deploying [Cluster API](../../05-Concepts/04-Kubernetes/03-cluster-api.md) on top of your existing metal-stack infrastructure. Cluster API with metal-stack is a declarative cluster management framework — you declare the desired state of your clusters and the Cluster API controllers reconcile the actual state. + +This guide does **not** use Ansible roles. Unlike Gardener, which is deployed via the `gardener-*` [metal-roles](https://github.com/metal-stack/metal-roles/tree/master/control-plane), Cluster API with metal-stack is set up directly through `clusterctl` and the [cluster-api-provider-metal-stack (CAPMS)](https://github.com/metal-stack/cluster-api-provider-metal-stack/) infrastructure provider. Consequently, the [Control Plane](./03_control-plane.mdx) and [Partition](./04_partition.md) guides still apply unchanged — only this KCLM layer differs. + +:::warning[Beta] +Cluster API with metal-stack is in beta and not yet recommended for production workloads. Please use [Gardener](./05_gardener.md) for production deployments. We are actively looking for exchange and adopters — if you are interested in using Cluster API with metal-stack, please [join our community](/community) to help shape future integration efforts. +::: + +:::info +Because Cluster API is a framework rather than a platform, this page is deliberately shorter than the [Gardener guide](./05_gardener.md): there is far less to configure, but substantially more day-2 tooling to build yourself. The [Cluster API concepts page](../../05-Concepts/04-Kubernetes/03-cluster-api.md#what-to-build-yourself) lists exactly what that entails, and the [KCLM decision matrix](../../05-Concepts/04-Kubernetes/01-kclm.md#decision-matrix) puts it into perspective. +::: + +## Prerequisites + +Before deploying Cluster API, ensure the following infrastructure is in place: + +- **Management cluster** — A Kubernetes cluster to host the Cluster API controllers and the desired cluster state. This can be any Kubernetes distribution (kind, k3s, a Gardener Shoot, or any managed Kubernetes). It must have network access to your metal-stack infrastructure. +- **metal-stack infrastructure** — A running metal-stack installation with at least one partition with available machines and operating system images (see [metal-images](https://github.com/metal-stack/metal-images) for pre-built ones). +- **CLI tools** — `metalctl` for communicating with the metal-stack API ([installation](https://github.com/metal-stack/metalctl)) and `clusterctl` for initializing providers and generating cluster manifests. +- **External dependencies** — DNS, NTP, ACME (optional), S3-compatible storage for backups, and Git-Hosting with CI/CD for GitOps-driven deployment. + +## Deployment + +Cluster API with metal-stack is deployed through the [cluster-api-provider-metal-stack (CAPMS)](https://github.com/metal-stack/cluster-api-provider-metal-stack/) infrastructure provider. The [CAPMS reference documentation](../../08-References/Kubernetes/cluster-api-provider-metal-stack/cluster-api-provider-metal-stack.md) covers the deployment in detail. + +**Deployment flow** + +1. **Prepare management cluster** — A Kubernetes cluster to host CAPI and CAPMS providers and cluster state +2. **Install CAPMS** — Deploy the CAPMS provider into the management cluster +3. **Configure `clusterctl`** — Register the metal-stack provider URL and set environment variables (API credentials, project, partition, machine images and sizes, cluster name, Kubernetes version) +4. **Allocate resources** — Create node networks, firewalls, and control plane IPs via `metalctl` +5. **Generate and apply cluster manifest** — Use `clusterctl generate cluster` to produce a YAML with `Cluster`, `MetalStackCluster`, `KubeadmControlPlane`, `MachineDeployment`, and `MetalStackMachine` resources, then apply it +6. **Deploy add-ons** — Install CNI (Calico) and `metal-ccm` via `ClusterResourceSet` and CAAPH +7. **Retrieve kubeconfig** — Access the provisioned cluster + +**Network integration** + +Network integration for Cluster API is more manual than for Gardener, and this is the part that surprises most operators coming from the Gardener path: + +- Node networks must be created up front via `metalctl network allocate` and handed to `clusterctl` as environment variables. +- Control plane IPs must be acquired in advance with `metalctl network ip create`. +- Firewall rules are static — there is no [firewall-controller-manager](../../05-Concepts/04-Kubernetes/05-firewall-controller-manager.md) integration yet, so no `ClusterWideNetworkPolicy` reconciliation. +- Service exposure uses KubeVIP in BGP mode instead of MetalLB. + +Automatic network resource allocation is on the CAPMS roadmap. Until then, plan these resources as part of your GitOps repository so they do not drift. + +**Air-gapped environments** + +For air-gapped deployments, follow the [Cluster API Operator air-gapped environment guide](https://cluster-api-operator.sigs.k8s.io/topics/configuration/air-gapped-environtment). All required images must be mirrored to an OCI registry reachable from the management cluster. On the metal-stack side, combine this with [Offline Resilience](./08_offline-resilience.md) so that machine provisioning itself does not depend on the internet. + +## Kamaji as Control Plane Provider + +[Kamaji](https://kamaji.clastix.io/) is a Control Plane Manager for Kubernetes that runs tenant control planes as pods inside the management cluster instead of on dedicated worker nodes. It plugs into Cluster API as a `ControlPlaneProvider`, while CAPMS remains the `InfrastructureProvider`. Conceptually this gets you close to Gardener's control plane hosting model — see [Cluster API + Kamaji](../../05-Concepts/04-Kubernetes/01-kclm.md#cluster-api--kamaji-the-middle-ground) for the trade-offs. + +:::warning +Kamaji integrations with metal-stack **have not been evaluated in production-grade scenarios** by metal-stack. We are actively looking for exchange and adopters — if you are interested in using Kamaji with metal-stack, please [join our community](/community) to help shape future integration efforts. +::: + +**Deployment** + +1. **Prepare management cluster** — A Kubernetes cluster to host Kamaji and CAPMS providers +2. **Install Kamaji and CAPMS** — Deploy both providers into the management cluster +3. **Create a control plane VIP** — MetalLB assigns a virtual IP for the tenant API server +4. **Generate and apply the tenant cluster manifest** — Use `clusterctl generate cluster --flavor kamaji-tenant`, then apply the result +5. **Deploy add-ons** — Install CNI (Calico) and `metal-ccm` into the tenant cluster + +A working showcase is available in the [`capi-lab`](https://github.com/metal-stack/cluster-api-provider-metal-stack/blob/main/DEVELOPMENT.md#running-the-kamaji-flavor) setup, which extends the `mini-lab` with a Kamaji flavor. See our [blog post](/blog/2026/04-kamaji) for a detailed walkthrough of the architecture and setup. + +Because Kamaji sits underneath Cluster API, everything in the following section applies unchanged. + +## Fleet Management and GitOps + +Cluster API ships no fleet management of its own — you set up the Git repository and GitOps operator yourself: + +1. **Git repository** — Store the manifests generated via `clusterctl generate cluster `. Each cluster gets its own set of YAML files containing `Cluster`, `MetalStackCluster`, `KubeadmControlPlane`, `MachineDeployment` and `MetalStackMachine` resources, plus the pre-allocated networks and IPs. +2. **GitOps operator** — Deploy Argo CD or Flux to watch the repository and apply manifests to the management cluster, ensuring drift-free declarative delivery. +3. **Per-cluster CI/CD** — Essential components (CNI, CCM) are rolled out per cluster. Changes to a `MetalStackMachineTemplate` or `ClusterResourceSet` are staged through the repository with the usual approval process — they are **not** rolled out fleet-wide automatically as they would be with Gardener. + +Use multi-stage environments (staging → production) and validate manifests in CI to keep the risk of platform changes manageable. + +For the day-2 capabilities this model does and does not give you — cluster migration via `clusterctl move`, emergency patching, certificate rotation, audit configuration — see [Cluster API concepts](../../05-Concepts/04-Kubernetes/03-cluster-api.md#fleet-management-and-gitops). + +## Available Flavors + +CAPMS provides different [cluster template flavors](https://cluster-api.sigs.k8s.io/cluster-api/commands/generate-cluster.html#flavors) for `clusterctl generate cluster`: + +| Flavor | Description | K8s Compatibility | +| --------------- | ------------------------------------------------------------------ | ----------------- | +| _(default)_ | Expects the user to deploy a CNI and a CCM manually | >= v1.33 | +| `calico` | Installs Calico CNI + metal-ccm via `ClusterResourceSet` and CAAPH | >= v1.33 | +| `pre-v1.33` | Same as default but for Kubernetes versions < v1.33 | < v1.33 | +| `kamaji-tenant` | Kamaji tenant cluster template (requires Kamaji installed) | >= v1.33 | diff --git a/docs/04-For Operators/03-Deployment/07-gpu-workers.md b/docs/04-For Operators/03-Deployment/07-gpu-workers.md new file mode 100644 index 00000000..0e6a2c9c --- /dev/null +++ b/docs/04-For Operators/03-Deployment/07-gpu-workers.md @@ -0,0 +1,80 @@ +--- +slug: /deployment/gpu-workers +title: GPU Workers +sidebar_position: 7 +--- + +# GPU Workers + +For workloads that require the assistance of GPUs, support for GPUs in bare metal servers was added in metal-stack v0.18.0. + +## Prerequisites + +Offering GPU worker nodes is a cooperation between the operator and the cluster owner. As an **operator**, you have to make GPU machines and the matching operating system available: + +1. **Register machines with GPUs** — The machine sizes must be known to the metal-api and be reported with their GPUs. Check with `metalctl size ls` and `metalctl machine ls`. +2. **Provide an `nvidia` image** — Add one of the `debian-nvidia` images from [metal-images](../../08-References/Deployment/metal-images/metal-images.md) to `metal_api_images` as described in [Providing Images](./03_control-plane.mdx#providing-images). +3. **Offer it through your KCLM** — With Gardener, keep `nvidia` in `gardener_extension_os_metal_types` (it is part of the default) _and_ add an `nvidia` entry to `gardener_cloud_profile_os_cri_mapping`. The mapping only covers `ubuntu` and `debian` by default, so without that entry the image is silently dropped from the `CloudProfile` and the size cannot be selected. See [extensions.yaml](./05_gardener.md#extensionsyaml--provider-and-shoot-extensions) and [cloud_profile.yaml](./05_gardener.md#cloud_profileyaml--defining-your-metal-stack-infrastructure). + +Everything below is then done by the **cluster owner** inside their own cluster. + +## GPU Operator installation + +With the `nvidia` image a worker has basic GPU support. This means that the required kernel driver, the containerd shim and the required containerd configuration are already installed and configured. + +To enable `Pods` that require GPU support to be scheduled on a worker node with a GPU, a `gpu-operator` must be installed. +This has to be done by the cluster owner after the cluster is up and running. + +The simplest way to install this operator is as follows: + +```bash +helm repo add nvidia https://helm.ngc.nvidia.com/nvidia +helm repo update + +kubectl create ns gpu-operator +kubectl label --overwrite ns gpu-operator pod-security.kubernetes.io/enforce=privileged + +helm install --wait \ + --generate-name \ + --namespace gpu-operator \ + --create-namespace \ + nvidia/gpu-operator \ + --set driver.enabled=false \ + --set toolkit.enabled=true +``` + +Note `driver.enabled=false`: the NVIDIA kernel driver already ships with the `nvidia` machine image, so the operator must not attempt to install its own. + +After that, `kubectl describe node` must show the GPU in the node capacity like so: + +```plain +... +Capacity: + cpu: 64 + ephemeral-storage: 100205640Ki + hugepages-1Gi: 0 + hugepages-2Mi: 0 + memory: 263802860Ki + nvidia.com/gpu: 1 + pods: 510 +... +``` + +With this basic installation, the worker node is ready to process GPU workloads. + +:::warning +There is a caveat: with this basic configuration only **one** `Pod` can access the GPU. If that is all you need, no additional configuration is required. +If you plan to deploy multiple applications that require GPU support and there are not that many GPUs available, you have to configure the `gpu-operator` so that a GPU can be shared between multiple `Pods`. +::: + +There are several approaches to sharing GPUs — time-slicing, MPS and MIG. Please consult the official NVIDIA documentation for further reference: + +- [Improving GPU utilization in Kubernetes](https://developer.nvidia.com/blog/improving-gpu-utilization-in-kubernetes) +- [GPU Operator with MIG](https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/gpu-operator-mig.html) +- [GPU sharing](https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/gpu-sharing.html) + +:::info +GPU machines are only offered in the partitions where the corresponding hardware is racked. Model this through your `CloudProfile` regions and zones (zones map to metal-stack partitions), so that end-users can only request GPU worker groups where GPUs actually exist. See [cloud_profile.yaml](./05_gardener.md#cloud_profileyaml--defining-your-metal-stack-infrastructure). +::: + +With this, happy AI processing. diff --git a/docs/04-For Operators/03-Deployment/07_offline-resilience.md b/docs/04-For Operators/03-Deployment/07_offline-resilience.md deleted file mode 100644 index 6dcef5ab..00000000 --- a/docs/04-For Operators/03-Deployment/07_offline-resilience.md +++ /dev/null @@ -1,32 +0,0 @@ ---- -slug: /deployment/offline-resilience -title: Offline Resilience -sidebar_position: 7 ---- - -# Offline Resilience - -It is possible to use metal-stack without any external network dependencies by integrating your own DNS and NTP configuration into the stack. This feature is great for workloads requiring strong independence and reliability. Even in case of an internet connection failure, your infrastructure remains operational. Existing machines do not encounter any downtime as well as new machines can be provisioned. All you need to have in place is a DNS and NTP server configured and accessible for metal-stack. - -NTP servers need to be configured on the pixiecore and the metal-hammer microservices. This can be achieved by providing a list of NTP servers with the following Ansible variable through metal-roles: - -```yaml -pixiecore_metal_hammer_ntp_servers: [] -``` - -In the background, the pixiecore is taking the NTP servers and passing it via the `MetalConfig` to the metal-hammer. When booting bare-metal servers, the metal-hammer needs to configure NTP servers. It recognises the ones from the `MetalConfig` and configures itself accordingly. -If no NTP servers are passed along, the following standard servers are used: - -- 0.de.pool.ntp.org -- 1.de.pool.ntp.org -- 2.de.pool.ntp.org - -Moreover, machine and firewall images need to be configured with your custom DNS and NTP servers. The customisation can be made via the fields `ntp_servers` an `dns_servers` and specifying a list of servers in the creation request for the machine or firewall. - -Within a partition default values for DNS and NTP servers can be configured. They are applied to all machines and firewalls within this partition, but can be replaced by specifying different ones inside the machine allocation request. - -Thus, for creating a partition as well as a machine or a firewall, the flags `dnsservers` and `ntpservers` can be provided within the `metalctl` command. - -In order to be fully offline resilient, make sure to check out `metal-image-cache-sync`. This component provides copies of `metal-images`, `metal-kernel` and `metal-hammer`. - -This feature is related to [MEP14](/community/MEP-14-independence-from-external-sources). diff --git a/docs/04-For Operators/03-Deployment/08_offline-resilience.md b/docs/04-For Operators/03-Deployment/08_offline-resilience.md new file mode 100644 index 00000000..42b09088 --- /dev/null +++ b/docs/04-For Operators/03-Deployment/08_offline-resilience.md @@ -0,0 +1,52 @@ +--- +slug: /deployment/offline-resilience +title: Offline Resilience +sidebar_position: 8 +--- + +# Offline Resilience + +It is possible to operate metal-stack without any external network dependencies by integrating your own DNS and NTP configuration into the stack. This is essential for workloads requiring strong independence and reliability: even if the internet uplink fails, your infrastructure remains operational — existing machines suffer no downtime, and new machines can still be provisioned. All you need in place is a DNS and an NTP server reachable from within the partition. + +Offline resilience rests on three independent pillars. Configure all three; each one on its own leaves a gap. + +## 1. Time — NTP for Provisioning + +NTP servers need to be configured for [pixiecore](https://github.com/metal-stack/metal-roles/tree/master/partition/roles/pixiecore) and [metal-hammer](https://github.com/metal-stack/metal-hammer). This is achieved by providing a list of NTP servers through the following Ansible variable of the `pixiecore` role, deployed as part of the [partition](./04_partition.md#management-server-deploy_mgmt_serversyaml): + +```yaml +pixiecore_metal_hammer_ntp_servers: + - ntp1.your-datacenter.example + - ntp2.your-datacenter.example +``` + +Under the hood, pixiecore takes the NTP servers and passes them via the `MetalConfig` to the metal-hammer. When booting bare-metal servers, the metal-hammer picks up the servers from the `MetalConfig` and configures itself accordingly. +If no NTP servers are passed along, the following public defaults are used — which is precisely what you want to avoid in an offline-resilient setup: + +- `0.de.pool.ntp.org` +- `1.de.pool.ntp.org` +- `2.de.pool.ntp.org` + +## 2. Time and Name Resolution — DNS and NTP for Machines + +Beyond provisioning, the running machine and firewall images need to be configured with your custom DNS and NTP servers. This is done via the `dnsservers` and `ntpservers` fields of the machine or firewall allocation request. + +Defaults can be configured **per partition** and are then applied to all machines and firewalls in that partition, unless the allocation request overrides them. Both `metalctl partition create` and `metalctl machine create` / `metalctl firewall create` accept the corresponding `--dnsservers` and `--ntpservers` flags. + +Setting the partition defaults is the recommended approach: it keeps every machine consistent without requiring every consumer — including your [KCLM](../../05-Concepts/04-Kubernetes/01-kclm.md) — to know about it. + +## 3. Artifacts — The Image Cache + +Machine provisioning pulls the operating system image, the kernel and the metal-hammer from the [global image store](https://images.metal-stack.io). To become independent of it, deploy the [`image-cache` role](https://github.com/metal-stack/metal-roles/tree/master/partition/roles/image-cache) on your management servers. It runs [metal-image-cache-sync](https://github.com/metal-stack/metal-image-cache-sync), which mirrors the images configured in the metal-api — `metal-images`, `metal-kernel` and `metal-hammer` — into the local file system, and falls back to the global image store only on cache misses. + +:::warning +For a genuinely offline-capable partition, the fallback to the internet must never be needed. Verify after every image update that the cache holds all images referenced by `metal_api_images` **before** you cut the uplink, and give the cache enough disk space to hold every image version still in use. +::: + +## Kubernetes Workloads + +The three pillars above make **metal-stack** offline resilient. Kubernetes clusters on top have their own dependency on container registries. For clusters that must be fully decoupled from the internet, combine this chapter with [Isolated Kubernetes Clusters](../../05-Concepts/04-Kubernetes/06-isolated-clusters.md), which mirrors all strictly required container images into a private registry and enforces the network restrictions with `ClusterWideNetworkPolicy` resources. + +## Background + +This feature set is based on [MEP-14 — Independence from External Sources](/community/MEP-14-independence-from-external-sources). diff --git a/docs/05-Concepts/04-Kubernetes/01-gardener.md b/docs/05-Concepts/04-Kubernetes/01-gardener.md deleted file mode 100644 index 535eb82b..00000000 --- a/docs/05-Concepts/04-Kubernetes/01-gardener.md +++ /dev/null @@ -1,78 +0,0 @@ ---- -slug: /gardener -title: Gardener -sidebar_position: 1 ---- - -# Gardener - -[Gardener](https://gardener.cloud/) is an open source project for orchestrated Kubernetes cluster provisioning. It supports many different cloud providers, metal-stack being one of them. Using the Gardener project, metal-stack can act as a machine provider for Kubernetes worker nodes. - -The idea behind the Gardener project is to start with a dedicated set of Kubernetes clusters (this can be a single cluster, too), which are used to host Kubernetes control planes for new Kubernetes clusters. The new Kubernetes control planes reside in dedicated namespaces of the initial clusters ("Kubernetes in Kubernetes" or "underlay / overlay Kubernetes"). For suggestions on how to set up the initial cluster, see the [Bootstrap Infrastructure](../../04-For%20Operators/03-Deployment/02_bootstrap-infrastructure.md) section in the Deployment Guide. - -Gardener's architecture is designed for multi-tenant environments, with a strong distinction between the operator and the end users. In Gardener, Kubernetes control planes for different tenants may reside in the same operator cluster. This approach makes it very suitable for being used with bare metal because it allows taking full advantage of the server resources. Another implication is that end users do not have access to their control plane components, such as the kube-apiserver or the ETCD. These are managed by the operator and in case of metal-stack even physically divided from the end user's workload. - -Gardener allocates machines from a cloud provider and automatically deploys a kubelet to those nodes, which then joins the appropriate control plane. Operators can also nest clusters so that newly provisioned clusters can be used to spin up more clusters, leading to nearly infinite scalability (also known as "kubeception" model). - -## Terminology - -We would like to explain the most important Gardener terms. The terminology used in the Gardener project has many similarities to the architecture of Kubernetes. Additional information can also be found in the [official glossary](https://github.com/gardener/documentation/blob/master/website/documentation/glossary/_index.md). - -### Garden Cluster - -The Garden Cluster is a Kubernetes cluster that runs the Gardener Control Plane. - -The control plane components introduce dedicated Kubernetes API resources for provisioning new Kubernetes clusters with the Gardener. It also takes care of the validation for many of those Gardener API resources and also reconciling some of them. The components are the following: - -- Gardener API Server -- Gardener Controller Manager -- Gardener Scheduler -- Gardener Admission Controller - -The control plane components can be deployed in the Garden Cluster through the Gardener Operator. - -The Garden cluster can also be used as [seed](#seeds-and-soils) cluster. - -### Virtual Garden - -A recommended way to deploy the Gardener is running a "virtual cluster" inside the Garden cluster. It is basically a Kubernetes control plane without any worker nodes, providing the Kubernetes API in an own ETCD. Its purpose is to store all Gardener resources (such that they reside inside a dedicated ETCD) and provide an individual update lifecycle from the Garden Cluster. End users can have access to own project namespaces in the virtual garden, too. - -The virtual garden consists of the following components: - -- garden kube-apiserver -- etcd -- kube-controller-manager - -More details about the virtual garden can be found in the description of [`gardener-operator`](https://github.com/gardener/gardener/blob/master/docs/concepts/operator.md). - -### Seeds and Soils - -A seed cluster is a cluster in which an agent component called the `Gardenlet` is running. The gardenlet is connected to the Gardener Control Plane and is responsible for orchestrating the provisioning of new clusters inside the seed cluster. The control plane components for the new clusters run as pods in the seed cluster. - -A seed cluster can also be called a soil if the Gardenlet has been manually deployed by the operator and not by the Gardener. Clusters created on the soil can be turned into seed clusters by the operator using a Gardener resource called `ManagedSeed`. This resource causes Gardener to automatically deploy the Gardenlet to the new cluster, such that the resulting cluster is not called a soil. - -### Shoot - -Every Kubernetes cluster that is fully provisioned and managed by Gardener is called a `Shoot` cluster. It consists of the shoot control plane running on the seed cluster and worker nodes running the actual workload. - -## Gardener Integration Components - -During the provisioning flow of a cluster, Gardener emits resources that are expected to be reconciled by controllers of a cloud provider. This section briefly describes the controllers implemented by metal-stack to allow the creation of a Kubernetes cluster on metal-stack infrastructure. - -If you want to learn how to deploy metal-stack with Gardener, please check out the corresponding [deployment-guide section](../../04-For%20Operators/03-Deployment/05_kclm.md). - -### gardener-extension-provider-metal - -The [gardener-extension-provider-metal](https://github.com/metal-stack/gardener-extension-provider-metal) contains of a set of webhooks and controllers for reconciling cloud provider specific resources of `type: Metal`, which created by Gardener during the cluster provisioning flow. - -Primarily, its purpose is to reconcile `Infrastructure`, `ControlPlane`, and `Worker` resources. - -The project also introduces an own API (`ProviderConfiguration` resources) and consists of an admission-controller to validate them. This admission controller should be deployed in the Gardener control plane cluster. - -### os-metal-extension - -Due to the reason metal-stack initially used ignition to provision operating system images (today, cloud-init is supported as well) there is an implementation of a controller that translates the generic `OperatingSystemConfig` format of Gardener into ignition userdata. It can be found on Github in the [os-metal-extension](https://github.com/metal-stack/os-metal-extension) repository. - -### machine-controller-manager-provider-metal - -Worker nodes are managed through Gardener's [machine-controller-manager](https://github.com/gardener/machine-controller-manager) (MCM). The MCM allows out-of-tree provider implementation via sidecar, which is what we implemented in the [machine-controller-manager-provider-metal](https://github.com/metal-stack/machine-controller-manager-provider-metal) repository. diff --git a/docs/05-Concepts/04-Kubernetes/01-kclm.md b/docs/05-Concepts/04-Kubernetes/01-kclm.md new file mode 100644 index 00000000..5edec6de --- /dev/null +++ b/docs/05-Concepts/04-Kubernetes/01-kclm.md @@ -0,0 +1,365 @@ +--- +slug: /kubernetes +title: Kubernetes Cluster Lifecycle Management +sidebar_position: 1 +--- + +# Kubernetes Cluster Lifecycle Management + +Kubernetes Cluster Lifecycle Management (KCLM) is the foundation of metal-stack's **Kubernetes as a Service** solution — enabling organizations to provision, operate, and decommission production-grade Kubernetes clusters on bare metal with the same ease and reliability as hyperscaler managed services, while retaining full control over the underlying infrastructure. + +## Why Kubernetes Cluster Lifecycle Management? + +Running Kubernetes on bare metal delivers unmatched performance, cost efficiency, and compliance advantages — but managing clusters at scale introduces significant operational complexity. Without automated lifecycle management, organizations face: + +- **Manual provisioning** that is slow, error-prone, and inconsistent +- **Drift and configuration divergence** across clusters ("snowflake clusters") +- **Limited scalability** — human operators cannot manage hundreds or thousands of clusters +- **Compliance risks** in regulated environments where audit trails, isolation, and controlled change processes are mandatory +- **Operational burden** on platform teams who must manage Kubernetes internals rather than enabling developer self-service + +KCLM solves these challenges by automating the entire cluster lifecycle — from design and bootstrap through scaling, upgrades, and decommissioning — while enforcing consistency, reproducibility, and separation of duties across multi-tenant environments. + +## What KCLM Automates + +metal-stack's KCLM solution covers the complete cluster lifecycle: + +```mermaid +flowchart LR + subgraph "Cluster Lifecycle" + A["Design & Template\nVersion-controlled YAML specs"] --> B["Bootstrap & Configure\nControl planes, workers, CNI, CSI"] + B --> C["Scale\nManual & auto-scaling of worker groups"] + C --> D["Upgrade\nRolling K8s version & patch updates"] + D --> E["Operate\nHealth checks, node replacement"] + E --> F["Hibernate\nScale down for non-24/7 clusters"] + F --> G["Decommission\nGraceful deletion with cleanup"] + end + + style A fill:#e1f5fe + style B fill:#e1f5fe + style C fill:#fff3e0 + style D fill:#fff3e0 + style E fill:#e8f5e9 + style F fill:#f3e5f5 + style G fill:#ffebee +``` + +| Phase | Capabilities | +| ----------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| **Design & Template** | Version-controlled cluster specifications constrained by administrator-defined profiles (allowed regions, machine types, Kubernetes versions) | +| **Bootstrap & Configuration** | Automated provisioning of control planes, worker nodes, CNI, CSI, and cloud controller — all reconciled to desired state | +| **Scaling** | Manual and automatic scaling of worker groups with configurable strategies (max surge, max unavailable, drain timeouts) | +| **Upgrades** | Rolling Kubernetes version upgrades, in-place Kubelet patch updates, and machine image updates within configurable maintenance windows (Gardener); spec-driven rolling updates (Cluster API) | +| **Node Replacement** | Automatic health-based remediation with configurable timeouts; workload shifting to replacement nodes | +| **Decommissioning** | Graceful cluster deletion with finalizer-based resource cleanup to prevent orphaned infrastructure | +| **Migration & Restore** | Control plane migration across failure domains; etcd backup and automatic recovery (Gardener); clusterctl-based cluster migration between management clusters (Cluster API) | +| **Hibernation** | Infrastructure scale-down for non-24/7 clusters to minimize resource waste (Gardener only) | + +## Kubernetes as a Service + +KCLM transforms bare metal infrastructure into a **self-service Kubernetes platform** — but the degree of self-service varies significantly between the two approaches: + +- **Gardener** provides a full self-service API where end-users (developers, team leads, project owners) can create clusters, manage node worker groups, configure maintenance windows, and enable/disable auto-upgrades — all limited only to whitelisted machine types and regions defined by platform administrators. Access is via standard `kubectl` with OIDC-based authorization and project-scoped permissions. +- **Cluster API** does not provide end-user self-service natively. Administrators manage the management cluster and provision clusters through GitOps workflows. If self-service is desired, a custom API layer must be built on top of Cluster API. + +Platform administrators, meanwhile, focus on **providing the platform** — managing seed clusters (Gardener) or the management cluster (Cluster API), whitelisting machine types, delivering Kubernetes version updates, and ensuring fleet-wide consistency through GitOps-driven processes. This separation of responsibilities mirrors how hyperscalers operate: administrators manage the infrastructure and platform components; end-users consume the Kubernetes API and focus on their workloads. + +## Two Approaches, One Infrastructure + +metal-stack provides KCLM through two integration paths, both consuming the same metal-stack API for bare-metal node provisioning: + +| Aspect | [Gardener](./02-gardener.md) | [Cluster API](./03-cluster-api.md) | +| ------------------------------- | ---------------------------------------------------------------------- | --------------------------------------------------------------------------------------- | +| **Status** | Recommended, production-ready | Beta, under active development | +| **Governance** | NeoNephos Foundation | CNCF (Kubernetes SIG) | +| **Experience** | 7+ years in financial-sector production | CNCF project, metal-stack integration in development | +| **Day-2 capabilities** | Native (DNS, backup, audit, certificate rotation, maintenance windows) | Assembled through GitOps and add-on providers | +| **Control plane hosting** | Dedicated namespaces on Seed clusters (physically isolated) | On worker nodes (CABPK) or dedicated (Kamaji, integration unevaluated) | +| **Operational model** | End-user self-service via Virtual Garden API | Administrator-managed GitOps workflows | +| **Complexity** | More opinionated, higher initial setup | Less opinionated, lower initial setup but higher ongoing administration | +| **Maintenance windows** | Built-in, per-cluster configurable | Continuous reconciliation, no built-in windows | +| **Control plane resource cost** | Shared on Seed clusters (efficient) | 3 dedicated nodes per cluster (wasteful) or shared via Kamaji (integration unevaluated) | +| **Use case** | Production fleets, regulated environments, multi-tenant platforms | Simple deployments, teams willing to build day-2 operations | + +Both converge on vanilla Kubernetes and metal-stack infrastructure, ensuring replaceability and vendor independence. + +## Platform vs Framework: What Each Solution Does For You + +The fundamental difference between Gardener and Cluster API is philosophical: **Gardener is a complete platform** (like AWS EKS or GKE), while **Cluster API is a framework** (like building your own managed Kubernetes service). This distinction drives everything else — administration burden, resource efficiency, day-2 capabilities, and operational complexity. + +### Gardener: The Complete Platform + +Gardener delivers a **turnkey Kubernetes as a Service platform** out of the box. When you deploy Gardener, you get: + +- **Built-in day-2 operations**: DNS management, etcd backup & restore, certificate rotation, audit logging, access control lists, hibernation, shoot migration, VPN — all native, no assembly required +- **Multi-tenant self-service**: End-users create and manage clusters through a Kubernetes-native API with project isolation, quotas, and role-based access control +- **Maintenance windows**: Per-cluster configurable windows for automatic updates (Kubernetes versions, machine images) — no manual coordination +- **Version skew enforcement**: Static validation rules in the API server that cannot be circumvented, preventing incompatible Kubernetes version combinations +- **Physical isolation**: Each Shoot's control plane runs in a dedicated namespace on a Seed cluster, physically separated from end-user workloads — critical for compliance requirements +- **Fleet-wide consistency**: All components (CNI, CSI, CCM, DNS, backup) are rolled out fleet-wide with every platform release, integration-tested across the compatibility matrix + +**What you build and maintain**: A Garden cluster (Gardener control plane), Seed clusters (one per data center site), and the Virtual Garden. This is significant infrastructure — typically 3-5 nodes for the Garden cluster and one Seed cluster per site. But once running, day-2 operations are largely automated. + +**Administration burden**: High initial setup (deploying Garden, Seeds, extensions), but **low ongoing administration**. Once the platform is running, cluster provisioning, upgrades, backups, certificate rotation, and decommissioning are all automated. Platform administrators focus on platform updates and extension management, not individual cluster operations. + +**Resource efficiency**: Control planes run as pods on Seed clusters — multiple Shoot control planes share Seed infrastructure. This is efficient for fleets but means control planes are not physically isolated from other workloads on the same Seed (though they are in dedicated namespaces). + +### Cluster API + CABPK: The Bare Framework + +Cluster API with the Kubeadm Bootstrap Provider (CABPK) is a **cluster provisioning framework** — it creates Kubernetes clusters but leaves everything else to you. + +**What you get out of the box**: + +- Cluster creation, scaling, and deletion via declarative manifests +- Machine provisioning on metal-stack infrastructure +- Add-on installation via ClusterResourceSet + CAAPH (Helm) + +**What you must build yourself**: + +- DNS management (no built-in DNS service) +- etcd backup & restore (no built-in operator) +- Certificate rotation (manual processes or custom tooling) +- Audit logging (configure via kubeadmConfigSpec, no centralized management) +- Maintenance windows (none — CAPI reconciles continuously) +- Version skew enforcement (not strictly enforced — risk management is your responsibility) +- Hibernation (no built-in capability) +- Cluster migration (no built-in capability) +- Multi-tenant self-service (no built-in project isolation, quotas, or user API) +- Access control lists (no built-in firewall controller) + +**Resource waste**: With CABPK, each cluster's control plane runs on dedicated worker nodes. For high availability, you need **3 control plane nodes per cluster** — nodes that exist solely to run kube-apiserver, etcd, controller-manager, and scheduler. These nodes cannot run user workloads. For a fleet of 10 clusters, that's 30 nodes wasted on control planes alone. This is a significant cost multiplier. + +**Administration burden**: Low initial setup (just the management cluster), but **very high ongoing administration**. Every day-2 task — certificate rotation, backup management, DNS configuration, audit policy management, version upgrades — must be built, tested, and maintained by your team. Each cluster upgrade requires manual coordination. There are no maintenance windows, no version skew enforcement, and no automated failure recovery beyond basic MachineHealthCheck. + +**Philosophy**: CAPI+CABPK is for teams that want maximum flexibility and are willing to invest significant engineering effort to build their own managed Kubernetes platform. It's the difference between buying AWS EKS and building your own EKS on EC2. + +### Cluster API + Kamaji: The Middle Ground + +Kamaji acts as a Control Plane Manager for Cluster API, running tenant control planes as pods within the management cluster rather than on dedicated worker nodes. + +**What Kamaji adds over CABPK**: + +- **Resource efficiency**: Multiple tenant control planes share management cluster infrastructure — no dedicated control plane nodes per cluster. A single management cluster can host dozens of tenant control planes +- **Physical isolation**: Control planes run in dedicated namespaces, separate from worker node workloads +- **Multi-tenant capability**: Built-in support for multiple tenant clusters on shared infrastructure + +**What you still must build yourself** (same as CABPK): + +- DNS, backup, certificate rotation, audit logging, maintenance windows, version skew enforcement, hibernation, cluster migration, multi-tenant self-service, ACLs + +**Resource efficiency**: Dramatically better than CABPK. Instead of 3 dedicated nodes per cluster for control planes, you share management cluster capacity across all tenants. For a fleet of 10 clusters, you might need only 3-5 management cluster nodes total. + +**Administration burden**: Still high ongoing administration — Kamaji solves the control plane hosting problem but not the day-2 operations problem. You still assemble and maintain all day-2 tooling through GitOps workflows. + +**Status**: Kamaji integrations with metal-stack have not been evaluated in production-grade scenarios by metal-stack. Kamaji itself is used in production elsewhere; it is the metal-stack integration that lacks our validation. It is a promising approach for resource efficiency but carries higher risk for production workloads. + +### Decision Matrix + +| Aspect | Gardener | CAPI + CABPK | CAPI + Kamaji | +| ----------------------------------- | ----------------------------------------------------------------- | ---------------------------------------------------------- | --------------------------------------------------------- | +| **Philosophy** | Complete platform (buy) | Framework (build) | Framework + shared control planes | +| **Day-2 operations** | All built-in, automated | Build everything yourself | Build everything yourself | +| **Control plane nodes per cluster** | Shared on Seeds (efficient) | 3 dedicated nodes (wasteful) | Shared on management cluster (efficient) | +| **Resource waste for 10 clusters** | Minimal | ~30 nodes wasted | Minimal | +| **Multi-tenant self-service** | Built-in (projects, quotas, API) | Build yourself | Build yourself | +| **Maintenance windows** | Built-in, per-cluster | None | None | +| **Version skew enforcement** | Enforced by API server | Not enforced | Not enforced | +| **Certificate rotation** | Automated (8-hour windows) | Manual/custom | Manual/custom | +| **etcd backup & restore** | Built-in (etcd-druid) | Build yourself | Build yourself | +| **Hibernation** | Built-in | Not available | Not available | +| **Shoot/Cluster migration** | Built-in | Not available | Not available | +| **Initial setup complexity** | High (Garden + Seeds) | Low (management cluster) | Low (management cluster) | +| **Ongoing administration** | Low (platform updates) | Very high (build & maintain everything) | Very high (build & maintain everything) | +| **Production readiness** | 7+ years, 10,000+ clusters | Beta, metal-stack integration in development | Not evaluated with metal-stack | +| **Best for** | Production fleets, regulated environments, multi-tenant platforms | Simple deployments, teams with strong Kubernetes expertise | Resource-constrained fleets, teams willing to accept risk | + +:::tip[Recommendation] +For production workloads, regulated environments, or any scenario where you need multi-tenant self-service with minimal ongoing administration, **Gardener is the clear choice**. The platform approach eliminates the need to build and maintain day-2 operations tooling, enforces version skew policies, and provides physical isolation between control planes and workloads. + +Cluster API with CABPK is only recommended for simple, single-cluster deployments where you accept the resource waste of dedicated control plane nodes and the ongoing administration burden of building your own day-2 operations. Cluster API with Kamaji offers better resource efficiency but carries higher risk, as we have not evaluated that integration in production-grade scenarios. +::: + +## Core Concepts + +These concepts apply to both Gardener and Cluster-API approaches. + +```mermaid +flowchart TD + subgraph "Git Repository" + G["Version-controlled YAML manifests\n(Cluster specs, CloudProfile, extensions)"] + end + + subgraph "CI/CD Pipeline" + P["Peer review → Audit trail → Apply"] + end + + subgraph "KCLM Controllers" + R["Reconciliation Loop\nCompare desired vs actual state"] + D["Drift Detection\nAutomatic remediation"] + end + + subgraph "Infrastructure" + K["Kubernetes Clusters\nVanilla K8s on bare metal"] + M["metal-stack API\nMachine, network, firewall allocation"] + end + + G --> P + P --> R + R --> D + D --> K + K -.->|provisions via| M + M -->|reports status| R + + style G fill:#e3f2fd + style P fill:#fff3e0 + style R fill:#e8f5e9 + style D fill:#fce4ec + style K fill:#f3e5f5 + style M fill:#e0f2f1 +``` + +### Domain Abstraction + +KCLM separates lifecycle management concerns from adjacent domains through well-defined contracts and extension points. Bare-metal provisioning, networking, storage, and Kubernetes distribution are integrated via provider extensions that implement generic interfaces — allowing each domain to evolve independently without requiring changes to the lifecycle orchestrator. This contract-based approach ensures that KCLM components do not depend on any specific infrastructure provider, and multiple domains can run in parallel. + +### Declarative State & GitOps + +The intended cluster state is defined declaratively in version-controlled YAML manifests stored in Git repositories, serving as the single source of truth. Changes are applied by updating specifications in the repository, with CI/CD pipelines enforcing peer review, audit trails, and rollback capability. This GitOps-driven workflow prevents configuration drift and ensures every cluster is reproducible from its manifest. + +### Reconciliation & Drift Detection + +KCLM follows an orchestration-driven model where controllers run continuous reconciliation loops, comparing the current state against the desired state defined in version-controlled specs. Drift is detected at an early stage and remediated automatically without manual intervention. In emergency situations, operators can pause reconciliation to prevent unintended changes. When controllers cannot self-heal, monitoring integrations alert operators. + +```mermaid +flowchart LR + subgraph "Desired State" + GIT["Git repository\nYAML manifests"] + SPEC["Kubernetes CRDs\nCluster specs, profiles"] + end + + subgraph "Reconciliation Loop" + WATCH["Controllers watch\nKubernetes API"] + COMPARE["Compare desired vs\nactual state"] + REMEDIATE["Remediate drift\nCreate/Update/Delete"] + end + + subgraph "Actual State" + K8S["Kubernetes clusters\nRunning workloads"] + INFRA["Bare metal machines\nNetworks, firewalls"] + end + + GIT -->|apply| SPEC + SPEC --> WATCH + WATCH --> COMPARE + COMPARE -->|drift detected| REMEDIATE + REMEDIATE --> K8S + REMEDIATE --> INFRA + K8S -->|API reports| WATCH + INFRA -->|API reports| WATCH + + PAUSE["⏸ Pause reconciliation\nEmergency override"] -.->|can interrupt| COMPARE + + style GIT fill:#e3f2fd + style SPEC fill:#e3f2fd + style WATCH fill:#c8e6c9 + style COMPARE fill:#fff9c4 + style REMEDIATE fill:#f8bbd0 + style K8S fill:#f3e5f5 + style INFRA fill:#e0f2f1 + style PAUSE fill:#ffebee +``` + +### Replaceability & Vendor Independence + +KCLM produces clusters built on **vanilla Kubernetes** — no forks or patches. As long as a replacement lifecycle management tool supports vanilla Kubernetes, migration requires no action on the cluster side. The contract-based integration model means that replacing the KCLM orchestrator or the infrastructure provider does not force changes to the Kubernetes distribution or the workloads running on it. + +## High Availability & Failure Domains + +KCLM supports multiple control plane topologies for on-prem failure domains. The available options differ between Gardener and Cluster API: + +| Topology | Gardener | Cluster API | +| ----------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| **Single-site HA** | Multiple control plane nodes across machines in the same Seed with etcd spread for quorum. Default production-confirmed choice. | Multiple control plane Machines within a single partition with etcd replicas on separate Machines. Natively supported. | +| **Multi-rack** | Shoot control plane nodes spread across multiple racks within one Seed with etcd spread across racks. Rack-level failure isolation via MachineDeployment topology spread constraints. | Multi-failure-domain topology distributes control plane Machines across multiple zones or regions with etcd spread accordingly. Natively supported through ClusterClass topology definitions. | +| **Multi-site** | Shoot control planes replicated across Seeds corresponding to different sites or data centers. MachineDeployments use zone constraints to distribute workers across regions. Higher latency for cross-seed communication requires multi-seed configuration. | Control plane Machines deployed across multiple CAPI management clusters or across widely separated MetalPools with cross-site etcd replication. Requires additional operator effort for cross-site networking. | +| **Dedicated isolation** | A Shoot gets its own dedicated Seed cluster with no shared control plane with other tenants. Highest compliance level for critical infrastructure at highest resource cost. | A dedicated Cluster with its own isolated MetalPool and exclusive use of MetalPools. Same isolation level as Gardener's dedicated Seed for critical infrastructure. | + +Worker nodes are automatically distributed across racks using a rack-spreading algorithm, and well-known Kubernetes topology labels (`machine.metal-stack.io/rack`, `machine.metal-stack.io/chassis`, `topology.kubernetes.io/region`, `topology.kubernetes.io/zone`) are provided on nodes — enabling end-users to configure Pod topology spread and anti-affinity rules. With MEP-19 (metal-stack Enhancement Proposal 19), routing across data center partitions will also be supported, allowing worker nodes to reside in separate metal-stack partitions while maintaining a single Kubernetes cluster — provided the partitions are geographically close enough for stable low-latency connectivity. + +Failure scenarios are engineered for automation: + +- **Worker failures** — Automatic health-based replacement within configurable health timeouts (default monitored by node-problem-detector in Gardener; MachineHealthCheck in Cluster API) +- **HA control plane** — Orchestrated three-node etcd clusters spread across the cluster topology with pod disruption budgets and topology spread constraints +- **etcd backups** — Scheduled full and incremental snapshots (typically three-minute deltas) enable automatic recovery from data loss +- **Accidental deletion protection** — Configurable backup retention allows emergency access to cluster resources before cleanup +- **BGP Anycast** — Spreads traffic across partitions and automatically routes around unreachable nodes within seconds + +:::note[Management plane availability] +The KCLM management layer is designed so that its absence does not impact cluster availability. Workloads continue running and end-users retain Kubernetes API access even when the management plane is unavailable. Outages of the Gardener cluster or the management cluster only cause cluster provisioning to become unavailable. +::: + +## Upgrades, Rollback, and Change Management + +**Upgrade strategies** — For minor Kubernetes version upgrades, Gardener rolls worker groups according to a configurable rollout strategy (max surge, max unavailable, drain timeouts). For patch updates, Gardener applies in-place Kubelet upgrades within a jittered 5-minute window, preventing vanishing of route announcements for more than one node at a time. Cluster API orchestrates a one-by-one worker node roll triggered by spec updates. + +**Blue-green updates** — End-users can achieve zero-downtime upgrades through two approaches: (1) using multiple clusters with BGP Anycast to spread workloads across clusters, or (2) using worker groups or machine pools with different Kubernetes kubelet versions and OS versions, combined with Kubernetes node taints and tolerations for traffic routing. + +**Rollback** — Kubernetes versions are not allowed to be rolled back. End-users are required to test Kubernetes upgrades in a staging cluster first. Tools exist to test for deprecated APIs before running the actual upgrade. + +**Downtime expectations** — With Gardener's HA control plane feature enabled, no downtime is expected during regular version or Kubernetes upgrades. In-place Kubelet upgrades happen within a jittered 5-minute window. MetalLB follows a one-by-one rolling update strategy so that route announcements only occur node-by-node until the speaker daemon set pod reports readiness again. Cluster API triggers an orchestrated worker roll, updating control planes and kubelets one-by-one. + +## Audit & Traceability + +Kubernetes API audit policies are configurable per cluster, with logs forwarded to external sinks (e.g., Splunk, S3) for auditable change tracking: + +- **Gardener** — The `gardener-extension-audit` extension allows shoot owners or operators to configure buffered forwarders to audit sinks. Audit policies for the kube-apiserver are configured via standard Kubernetes Policy manifests, with each cluster having its own set of policies. The same extension can be configured for the Garden cluster's apiserver. +- **Cluster API** — Audit configuration can be passed to the kube-apiserver via the `kubeadmConfigSpec` of the `KubeadmControlPlane` resource before cluster creation. The management cluster's kube-apiserver audit also needs to be configured separately. Cluster API does not provide a centralized audit management toolset and relies on cloud-native standards to be set up by the operator. + +## Day-2 Operations, HA, and Advanced Topics + +The following topics differ significantly between the two solutions and are therefore covered on the individual solution pages: + +| Topic | Gardener | Cluster API | +| -------------------- | ---------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------- | +| Day-2 operations | [Operational features](./02-gardener.md#operational-features) — built-in | [What to build yourself](./03-cluster-api.md#what-to-build-yourself) — assembled through GitOps | +| HA & failure domains | [Control plane topologies](./02-gardener.md#control-plane-topologies) | [Control plane topologies](./03-cluster-api.md#control-plane-topologies) | +| Add-on lifecycle | [Operational features](./02-gardener.md#operational-features) — fleet-wide | [Fleet management](./03-cluster-api.md#fleet-management-and-gitops) — per cluster via `ClusterResourceSet` | +| Version policy | [Version skew policy](./02-gardener.md#version-skew-policy) — enforced by the API server | [Operational model](./03-cluster-api.md#operational-model) — risk management via approval gates | +| Upgrades & rollback | [Upgrade & rollback](./02-gardener.md#upgrade--rollback) | [Upgrade & rollback](./03-cluster-api.md#upgrade--rollback) | +| Audit & traceability | [Audit & traceability](./02-gardener.md#audit--traceability) | [Fleet management](./03-cluster-api.md#fleet-management-and-gitops) | + +## Network Integration + +KCLM interacts with network automation systems through the cloud provider contract. During cluster creation, private node networks are automatically allocated, firewall rules are generated from cluster policies, and public IPs for LoadBalancer services are dynamically assigned. The [metal-ccm](./04-cloud-controller-manager.md) (cloud controller manager) bridges Kubernetes services with the underlying bare-metal networking stack, supporting MetalLB in BGP mode for IP address announcement. CNIs leverage the native routing infrastructure and auto-detect MTU requirements. + +## Bootstrap & Air-Gapped Environments + +KCLM bootstraps clusters using OCI-registry-pulled images and bootstrap tokens that work once for node joining. All cluster components — including the node agent, kubelet, CNI, and CSI — are pulled from registries that must be reachable from the target environment. This model supports air-gapped deployments where all required images are pre-pulled into a local registry. Clusters only reach ready state when all worker nodes have joined, VPN is established, and all managed resources are healthy. + +## What Makes metal-stack KCLM Great + +The combination of metal-stack's bare-metal infrastructure with KCLM delivers unique advantages: + +### Physical Isolation for Compliance + +Kubernetes control planes run in dedicated namespaces or seed clusters, physically separated from end-user workloads on metal-stack partitions. Operator-managed components are inaccessible to cluster owners — critical for compliance requirements. + +### Fleet-Wide Consistency + +Every cluster is provisioned from version-controlled YAML manifests with thorough integration testing across the compatibility matrix. No exceptions, no snowflake clusters — all features implemented as Kubernetes controllers and carried out fleet-wide. + +### Automatic Failure Recovery + +Controllers continuously reconcile desired state against actual state. Failed nodes are automatically detected and replaced within configurable health timeouts. Workloads shift seamlessly to replacement nodes without manual intervention. + +### Topology Awareness + +metal-stack distributes worker nodes across racks using a rack-spreading algorithm and provides well-known Kubernetes topology labels — enabling end-users to configure Pod topology spread and anti-affinity rules for high availability. + +### Zero Downtime for Management Outages + +The KCLM management layer is designed so that its absence does not impact cluster availability. Workloads continue running and end-users retain Kubernetes API access even when the management plane is unavailable. + +### Proven at Scale + +metal-stack with Gardener operates environments with **280 Kubernetes clusters** across **5 data centers** and **1,800 physical servers** (including 200 servers in a metro environment across disjunct locations). The largest observed clusters hold 64 nodes; metal-stack itself supports up to 1,024 nodes per cluster. Proven Gardener installations elsewhere manage **10,000+ clusters** — demonstrating that consistent, automated lifecycle management is the key to scaling bare-metal Kubernetes fleets. For Cluster API, the integration test environment covers a management cluster with three worker nodes, with the biggest test clusters including 8 cluster nodes. diff --git a/docs/05-Concepts/04-Kubernetes/02-cluster-api.md b/docs/05-Concepts/04-Kubernetes/02-cluster-api.md deleted file mode 100644 index 43840d61..00000000 --- a/docs/05-Concepts/04-Kubernetes/02-cluster-api.md +++ /dev/null @@ -1,18 +0,0 @@ ---- -slug: /cluster-api -title: Cluster API -sidebar_position: 2 ---- - -# Cluster API - -[Cluster API](https://cluster-api.sigs.k8s.io/) is a Kubernetes project that aims to simplify the management of Kubernetes clusters. It provides a declarative way to create, configure, and manage clusters using Kubernetes-style APIs. - -We provide the [Cluster API provider for metal-stack (CAPMS)](https://github.com/metal-stack/cluster-api-provider-metal-stack) infrastructure provider that allows the declaration of Kubernetes clusters. - -:::warning "Early Development Stage" -This project is currently under heavy development and is not advised to be used in production any time soon. -Please use our stack on top of [Gardener](./01-gardener.md) for production workloads. -::: - -See the [cluster-api-provider-metal-stack documentation](../../08-References/Kubernetes/cluster-api-provider-metal-stack/cluster-api-provider-metal-stack.md) for more in-depth information. diff --git a/docs/05-Concepts/04-Kubernetes/02-gardener.md b/docs/05-Concepts/04-Kubernetes/02-gardener.md new file mode 100644 index 00000000..8dec1118 --- /dev/null +++ b/docs/05-Concepts/04-Kubernetes/02-gardener.md @@ -0,0 +1,236 @@ +--- +slug: /gardener +title: Gardener +sidebar_position: 2 +--- + +# Gardener + +[Gardener](https://gardener.cloud/) is an open source project for orchestrated Kubernetes cluster provisioning governed by the NeoNephos Foundation. It supports many different cloud providers, metal-stack being one of them. Using the Gardener project, metal-stack can act as a machine provider for Kubernetes worker nodes. + +Gardener is the **recommended** Kubernetes Cluster Lifecycle Management (KCLM) solution for metal-stack. It is battle-tested in production for over seven years at financial-sector customers and bundles more day-2 capabilities natively (DNS, backup, audit). Gardener manages entire clusters as Kubernetes-native resources with a strong separation between platform operators and end-users. + +This page describes **what** Gardener with metal-stack does and why. For **how** to deploy it, see the [Gardener deployment guide](../04-For%20Operators/03-Deployment/05_gardener.md); for the comparison with the alternative, see the [KCLM overview](./01-kclm.md#two-approaches-one-infrastructure). + +## Outcomes + +Gardener targets three primary outcomes for bare-metal Kubernetes operations: + +- **Automation:** Gardener reconciliation loops across Shoot, Seed, and extension CRDs automate design, bootstrap, scaling, upgrades, and decommission end-to-end. The machine-controller-manager handles node lifecycle (create, replace, drain) automatically. Certificate rotation, ETCD backup, and add-on installation are built-in extensions. +- **Reproducibility:** Every Gardener Shoot cluster is a Git-versioned YAML manifest with a CloudProfile constraining versions, machine types, and regions. Infrastructure definitions, extension Helm charts, and seed configurations are version-controlled. GitOps operators deploy those manifests via CI/CD pipelines enabling peer review, audit trails, and rollback. +- **Risk reduction:** Gardener isolates each Shoot's control plane in its own namespace with dedicated ETCD, physically separated from end-user workloads on metal-stack partitions. Operator-managed control plane components (kube-apiserver, ETCD) are inaccessible to Shoot owners. etcd-druid provides continuous backup and automatic recovery. Admission controllers, health probes, and readiness gates reject misconfigurations early. + +## Architecture + +Gardener uses a hierarchical cluster model — often called the "kubeception" model — where Kubernetes clusters host other Kubernetes clusters. This architecture provides physical isolation between the Kubernetes control plane and end-user workloads, which is critical for compliance in regulated environments. + +### Cluster Hierarchy + +The diagram below shows the full deployment architecture — from the bootstrap infrastructure hosting the Garden Cluster (which can run on metal-stack cloud, any cloud provider, or on-prem), through the metal control plane, down to the metal-stack partition where Seeds provision Shoot clusters with their workloads. The firewall-controller in the partition integrates with the metal-stack firewall for Shoot network isolation. + +![Gardener cluster hierarchy](gardener-cluster-hierarchy.png) + +### Core Components + +| Component | Responsibility | +| ------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| **Garden cluster** | The top-level cluster that runs the Gardener control plane (API server, controller manager, scheduler, admission controller). Deployed via the `gardener-operator`. | +| **Virtual Garden** | A recommended deployment pattern where Gardener runs inside a virtual cluster on the Garden cluster. This provides a dedicated ETCD for Gardener resources and an independent update lifecycle from the Garden cluster itself. End users get project namespaces in the virtual garden. | +| **Seed cluster** | A cluster where a `gardenlet` agent runs. The gardenlet connects to the Gardener control plane and orchestrates provisioning of new clusters within that Seed. Typically one Seed per data-center site. A Seed that has been manually deployed (not by Gardener) is called a **soil**. | +| **Shoot cluster** | Every fully provisioned and managed Kubernetes cluster. The Shoot's control plane (kube-apiserver, etcd, controller-manager, scheduler) runs as pods in a dedicated namespace on a Seed, while worker nodes run on bare-metal machines provisioned via the metal-stack API. | + +### Core Controllers + +| Controller | Purpose | +| ----------------------------- | ---------------------------------------------------------------------------------------- | +| `gardener-operator` | Deploys Gardener components, gardenlets, and extensions; manages platform updates | +| `gardener-apiserver` | Extends the kube-apiserver with Gardener-specific resources (Shoot, Seed, Project, etc.) | +| `gardener-scheduler` | Decides where clusters are placed across the Gardener landscape (Seeds) | +| `gardener-controller-manager` | Reconciles common Gardener resources (projects, controller installations, etc.) | +| `gardenlet` | Agent running on each Seed; orchestrates provisioning of new clusters within that Seed | +| `gardener-resource-manager` | Runs inside Shoots; reconciles desired resources and checks their health | +| `etcd-druid` | etcd cluster operator with built-in backup-restore functionality | +| `machine-controller-manager` | Manages worker node lifecycle (rolling updates, health recreation, scaling) | + +### Provider Extensions + +| Extension | Purpose | +| --------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `gardener-extension-provider-metal` | Integrates metal-stack API with Gardener (machines, networks, firewalls, IPs). See also [Cloud Controller Manager](./04-cloud-controller-manager.md) and [Firewall Controller Manager](./05-firewall-controller-manager.md) for integration details. | +| `gardener-extension-audit` | Configures buffered forwarders to audit sinks (e.g., Splunk, S3) | +| `gardener-extension-networking-calico` | Provides Calico CNI in shoot clusters | +| `gardener-extension-networking-cilium` | Provides Cilium CNI in shoot clusters | +| `gardener-extension-shoot-dns-service` | Provides DNS records for the Kubernetes API server; allows shoot owners to create DNS records within their project domain | +| `gardener-extension-shoot-cert-service` | Provides certificates for services exposed in shoot clusters | + +### Integration Extensions + +| Extension | Purpose | +| ------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------- | +| `os-metal-extension` | Translates Gardener's generic `OperatingSystemConfig` format into cloud-init (or ignition) userdata for metal-stack nodes | +| `machine-controller-manager-provider-metal` | Integrates metal-stack machine provisioning API with Gardener's MCM as an out-of-tree sidecar | + +For a complete architecture overview with diagrams, see the [Gardener documentation](https://gardener.cloud/docs/getting-started/architecture/). + +## Gardener Integration Components + +During the provisioning flow of a cluster, Gardener emits resources that are expected to be reconciled by controllers of a cloud provider. metal-stack implements the following integration components: + +```mermaid +sequenceDiagram + participant G as Gardener
Shoot Controller + participant EXT as gardener-extension-
provider-metal + participant MCM as machine-controller-
manager-provider-metal + participant MS as metal-stack API + participant NODE as Bare Metal Node + + G->>EXT: Creates Infrastructure
ControlPlane Worker resources + EXT->>MS: Allocate machines, networks,
firewalls, IPs + MS-->>EXT: Returns machine IDs, IPs + EXT->>MCM: Creates Worker resources + MCM->>MS: Provision worker machines + MS-->>MCM: Machines ready + MCM->>MS: Generate cloud-init userdata + MCM->>NODE: Bootstrap with cloud-init + NODE-->>MCM: Node joined + MCM->>G: Node ready + G->>G: Reconcile Shoot status +``` + +### gardener-extension-provider-metal + +The [gardener-extension-provider-metal](https://github.com/metal-stack/gardener-extension-provider-metal) contains a set of webhooks and controllers for reconciling cloud provider specific resources of `type: Metal`, which are created by Gardener during the cluster provisioning flow. + +Primarily, its purpose is to reconcile `Infrastructure`, `ControlPlane`, and `Worker` resources. The project also introduces its own API (`ProviderConfiguration` resources) and consists of an admission controller to validate them. + +### os-metal-extension + +The [os-metal-extension](https://github.com/metal-stack/os-metal-extension) translates Gardener's generic `OperatingSystemConfig` format into cloud-init (or ignition) userdata for metal-stack nodes. It enables the operating system bootstrap process for worker nodes. + +### machine-controller-manager-provider-metal + +Worker nodes are managed through Gardener's [machine-controller-manager](https://github.com/gardener/machine-controller-manager) (MCM). The MCM allows out-of-tree provider implementation via sidecar, which is implemented in the [machine-controller-manager-provider-metal](https://github.com/metal-stack/machine-controller-manager-provider-metal) repository. + +The MCM handles the full worker node lifecycle: creating new machines, draining and replacing unhealthy nodes, and performing rolling updates across worker groups. It prevents meltdown scenarios through configurable backoffs and watches Kubernetes node conditions continuously — certain conditions can be configured as critical, triggering health timeouts that cause automatic node replacement. The MCM also supports custom controllers that contribute to node conditions, enabling advanced health monitoring beyond the default `node-problem-detector`. + +## Operational Model + +Gardener differentiates between **end-users** (shoot owners) and **platform administrators** (seed owners), similar to how hyperscalers offer Kubernetes as a Service: + +| Role | Responsibilities | +| ------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | +| **End-Users** | Create/manage clusters via API (limited to whitelisted machine types), manage worker groups, trigger cluster updates, configure maintenance windows and auto-updates | +| **Administrators** | Set up Seed clusters in data centers, whitelist machine types, provide Kubernetes versions and OS images, define lifecycle policies, fleet-wide GitOps-driven operations | + +End-users access the Virtual Garden through a feature-rich Kubernetes API with OIDC-based authorization. Resources like `ShootQuota` and custom webhooks can further restrict what end-users can do and ensure that they operate only within their specific, operational boundaries. As the API of Gardener is provided by Kubernetes itself, it is also an option to further narrow down the end-user scope by hiding the Gardener API behind a custom API. Platform administrators manage the entire platform through GitOps-driven processes with approval workflows. + +Gardener clearly defines responsibility boundaries — administrators manage the control plane lifecycle, CNI, CSI drivers, and other platform components, while end-users focus on their workloads. Components on the worker nodes like the Kubelet or CNI/CSI pods are continuously monitored and reconciled to the desired state, which makes it hard for end-users to misconfigure essential cluster components. The `node-problem-detector` is deployed by default to contribute node conditions that the machine-controller-manager watches for health timeouts. Custom controllers can also contribute node conditions for advanced health monitoring. Features like auto-updates for Kubernetes versions or machine images can be enabled and disabled by end-users individually and are carried out by Gardener in configurable maintenance time windows. + +## Failure Domains + +From a Kubernetes Cluster Lifecycle Management perspective, managing multiple failure domains is essential — allowing clusters to be provided in different geographical locations and enabling end-users to distribute workloads across different failure domains. To make topology information available in the resulting Kubernetes clusters, metal-stack's KCLM integrations automatically provide well-known Kubernetes topology labels on the node resources: + +- `machine.metal-stack.io/chassis` +- `machine.metal-stack.io/rack` +- `topology.kubernetes.io/region` +- `topology.kubernetes.io/zone` + +While metal-stack distributes cluster worker nodes across racks automatically using a rack-spreading algorithm, information like region and zone comes from the end-user's requirements. With this information, Kubernetes features like Topology Spread and PodAntiAffinity can be configured easily by end-users. With MEP-19 (metal-stack Enhancement Proposal 19), routing across data center partitions will also be supported, allowing worker nodes to reside in separate metal-stack partitions while maintaining a single Kubernetes cluster — provided the partitions are geographically close enough for stable low-latency connectivity. + +The KCLM is designed so that every failure domain can function without the KCLM itself — meaning the absence of the KCLM does not impact the availability of the Kubernetes clusters or the network traffic. Outages of the Gardener cluster only cause cluster provisioning to become unavailable, without interrupting workloads or preventing end-users from interacting with the Kubernetes API of their clusters. + +## Control Plane Hosting + +In Gardener, the Kubernetes control planes for end-user clusters reside in so-called "seed clusters". In the metal-stack case, the seed clusters run on bare metal servers and are physically divided from the end-user worker nodes. Seed clusters can run at geographically different locations from the worker nodes. A secure connection between the Kubernetes control plane and components running on the worker nodes is established using VPN. It is also possible with Gardener to migrate Kubernetes control planes across seed clusters if necessary (shoot migration). + +There is a clear distinction between administrators, who are responsible for the Kubernetes control plane to run, and end-users, which only consume the provided Kubernetes API. This hosting model is mandatory in Gardener and suits well for pairing with bare metal in compliance-sensitive environments. + +## Control Plane Topologies + +Gardener supports multiple control plane topologies for on-prem failure domains: + +| Topology | Description | Use Case | +| ------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------- | +| **Single-seed HA** | Multiple control plane nodes (kube-apiserver, controller-manager, scheduler) across machines in the same Seed with etcd spread across machines for quorum. Default production-confirmed choice. | Single-site deployments, standard production | +| **Multi-rack** | Shoot control plane nodes spread across multiple racks within one Seed with etcd spread across racks. Rack-level failure isolation via MachineDeployment topology spread constraints. | Rack-level failure isolation within a single data center | +| **Multi-site** | Shoot control planes replicated across Seeds corresponding to different sites or data centers. MachineDeployments use zone constraints to distribute workers across regions. Higher latency for cross-seed communication requires multi-seed configuration. | Disaster recovery across geographically separated sites | +| **Dedicated Seed** | A Shoot gets its own dedicated Seed cluster with no shared control plane with other tenants. Highest compliance level for critical infrastructure at highest resource cost. | Strictest compliance requirements for critical infrastructure | + +All topologies are natively supported. Multi-site requires additional multi-seed configuration but no special extensions. + +## Upgrade & Rollback + +**Minor version upgrades** — Gardener rolls worker groups according to a configurable rollout strategy (drain timeouts, serial stateful set eviction, max surge and max unavailable nodes). For patch updates, Gardener applies in-place Kubelet upgrades within a jittered 5-minute window, preventing vanishing of route announcements for more than one node at a time. + +**Blue-green updates** — End-users can achieve zero-downtime upgrades through two approaches: (1) using multiple clusters with BGP Anycast to spread workloads across clusters, or (2) using worker groups with different Kubernetes kubelet versions and OS versions, combined with Kubernetes node taints and tolerations for traffic routing. + +**Rollback** — Kubernetes versions are not allowed to be rolled back. End-users are required to test Kubernetes upgrades in a staging cluster first. Tools exist to test for deprecated APIs before running the actual upgrade. + +**Downtime expectations** — With HA control plane enabled, no downtime is expected during regular version or Kubernetes upgrades. When upgrading MetalLB in a shoot cluster, it follows a one-by-one rolling update strategy so that route announcements only occur node-by-node until the speaker daemon set pod reports readiness again. + +## Domain Model + +The KCLM integrates with adjacent domains based on contracts. This way, the domains can be used without requiring direct upstream contributions to the KCLM but rather extend it and ideally run multiple domains in parallel. Gardener does not depend in any form on metal-stack components or APIs nor does it depend on other adjacent domains. metal-stack has implemented the integration in the form of extensions: + +- **Bare metal machine provisioning** — Abstraction through `CloudProfile`, which is referenced in a Shoot resource. Provider extensions can then implement the contracts for provisioning a cluster. For metal-stack this is the `gardener-extension-provider-metal`. +- **Network** — Abstraction through Kubernetes CNI, provider and network extensions. Gardener itself is unaware of the concrete network infrastructure that it provisions clusters on. Network extensions allow setting up CNIs like Cilium, Calico or others while provider extensions can set up infrastructure resources dynamically during provisioning. +- **Storage** — Abstraction through Kubernetes CSI, provider and storage extensions. Identical to network. +- **Kubernetes Distribution** — Gardener creates a Shoot cluster with vanilla Kubernetes components from the upstream Kubernetes project. No other Kubernetes distributions are supported. + +Gardener produces clusters built on vanilla Kubernetes that passes 100% CNCF Kubernetes conformance. It provisions and manages upstream Kubernetes distributions without modification — no fork or patched Kubernetes is used. As long as a replacement KCLM also supports vanilla Kubernetes, no action is required. + +## Operational Features + +Gardener provides several day-2 operational capabilities out of the box: + +- **Shoot migration** — Control planes can be migrated across Seed clusters for geographic relocation +- **VPN** — Secure connection between control plane components on Seeds and worker nodes +- **Scaling** — Manual and automatic scaling of worker groups with configurable strategies (max surge, max unavailable, drain timeouts); Cluster Autoscaler automatically adjusts worker groups based on requested pod resources +- **Hibernation** — Scale down infrastructure by deleting worker nodes and scaling down control plane components; can be triggered manually or by schedule +- **Maintenance time windows** — End-user configurable windows for automatic updates (Kubernetes versions, machine images) +- **Certificate rotation** — Automated certificate management for cluster components and shoot services; certificates are rotated in short time frames (8 hours). For certain components (e.g., CA of the Kubernetes API server), rotation is a two-phase process: first the new CA is added (giving users time to reconfigure clients), then the old CA is removed in a completion phase +- **etcd backup & restore** — Continuous etcd backups with automatic recovery via etcd-druid (typically configured with three-minute incremental deltas); configurable backup retention allows emergency access to cluster resources after accidental shoot deletion, which is further protected by special labels that prevent accidental shoot deletion. Shoot deletion is safeguarded by specific annotations — during deletion, Gardener attempts to clean up resources gracefully by triggering finalizers of controllers deployed into the shoot cluster, preventing orphaned external resources +- **DNS management** — Automatic DNS record creation for shoot API servers and cluster-internal services +- **Audit logging** — Configurable per-cluster audit policies with forwarding to external sinks +- **Access control lists** — Network-level access control for shoot clusters +- **Emergency reconciliation pause** — Operators can use Kubernetes annotations on seed resources to prevent shoot cluster reconciliation in emergency situations +- **Image vector overwrites** — Administrators can perform emergency patching fleet-wide by configuring image vector overwrites in the Gardener deployment Git repository, validated in staging environments before rollout. Emergency patches for OS images can also be offered through the `CloudProfile` and automatically applied within configured maintenance time windows for clusters with auto-upgrade enabled +- **Component upgrades** — Essential cluster components (CNI, CSI, CCM, DNS, audit) are rolled out with every new Gardener version fleet-wide, ensuring consistency across the entire landscape. End-users focus on their workloads without needing to manage Kubernetes internals + +## Bootstrap & Air-Gapped Environments + +Gardener uses a built-in bootstrap mechanism based on the `gardener-node-agent`. The kubelet and gardener-node-agent are both pulled from an OCI registry just like all other components running in the final cluster. This OCI registry must be reachable from within the (air-gapped) environment and must contain all images of the Gardener release's compatibility matrix. + +The kubelet is joined using a bootstrap token that is generated during the shoot provisioning workflow and only works once from the node to join the control plane. After a creation timeout in the machine-controller-manager (by default 20 minutes) the token gets recreated with another attempt to join the worker node with a new machine. + +Shoot clusters only reach ready state when all worker nodes have successfully joined, shoot VPN is running, and all managed resources (includes CNI, CSI, etc.) are healthy and running. + +## External Dependencies + +The following data center infrastructure dependencies are treated as given and must be available before deploying Gardener: + +- **DNS** — For cluster domain resolution (PowerDNS or external) +- **NTP** — Time synchronization across all nodes +- **ACME** — Certificate authority (for shoot certificates via shoot-cert-service) +- **S3-compatible object storage** — For etcd backups (gardener-extension-backup-s3) +- **Git-Hosting with CI/CD** — For GitOps-driven deployment of manifests +- **MetalLB** — For exposing Kubernetes API servers of Shoot clusters (layer 2 load balancing) + +## Scalability + +With the described operational model it is possible to scale massively. metal-stack operates an environment with **280 Kubernetes clusters** across **5 data centers** and **1,800 physical servers**. Proven Gardener installations manage **10,000+ clusters** — demonstrating that consistent, automated lifecycle management is the key to scaling bare-metal Kubernetes fleets. The key behind this high degree of scalability is that all clusters are provided consistently, providing a high level of integration testing and avoiding custom provisioning solutions. + +End-users can test minor Kubernetes upgrades first in clusters labeled `evaluation` or `development` before rolling out to `production`-labeled clusters. For evaluation clusters, auto-upgrades for Kubernetes and OS images can be enabled to stage updates, reduce manual effort, and gain security patches more quickly. Deprecation of Kubernetes versions can be configured in the `CloudProfile`, and end-users can pin worker group Kubernetes versions as long as the version skew policy is adhered. The Cluster Autoscaler automatically adjusts worker groups based on requested pod resources. + +## Version Skew Policy + +Gardener enforces Kubernetes version skew policies through static validation rules in the `gardener-apiserver` that cannot be circumvented. Worker group versions can be pinned as long as the skew policy is respected. Minor version upgrades are applied one-by-one following the [Gardener version skew policy](https://gardener.cloud/docs/gardener/deployment/version_skew_policy/). For details on supported Kubernetes versions and upgrade sequencing, see the [Gardener deployment documentation](https://gardener.cloud/docs/). + +## Audit & Traceability + +Kubernetes API audit policies are configurable per cluster, with logs forwarded to external sinks (e.g., Splunk, S3) for auditable change tracking: + +- **Shoot clusters** — The `gardener-extension-audit` extension allows shoot owners or operators to configure buffered forwarders to audit sinks. Audit policies for the kube-apiserver are configured via standard Kubernetes Policy manifests, with each cluster having its own set of policies. +- **Garden cluster** — The same audit extension can be configured for the gardener-apiserver and virtual kube-apiserver. + +For configuration examples, see the [Gardener deployment guide](../04-For%20Operators/03-Deployment/05_gardener.md). diff --git a/docs/05-Concepts/04-Kubernetes/03-cloud-controller-manager.md b/docs/05-Concepts/04-Kubernetes/03-cloud-controller-manager.md deleted file mode 100644 index ef82b964..00000000 --- a/docs/05-Concepts/04-Kubernetes/03-cloud-controller-manager.md +++ /dev/null @@ -1,11 +0,0 @@ ---- -slug: /metal-cloud-controller-manager -title: Cloud Controller Manager -sidebar_position: 3 ---- - -# metal Cloud Controller Manager - -CCM stands for [cloud-controller-manager](https://kubernetes.io/docs/concepts/architecture/cloud-controller/) and is the bridge between Kubernetes and a cloud-provider. - -We implemented the [cloud provider interface](https://github.com/kubernetes/cloud-provider/blob/master/cloud.go) in the [metal-ccm](https://github.com/metal-stack/metal-ccm) repository. With the help of the cloud-controller-controller we provide metal-stack-specific properties for Kubernetes clusters, e.g. load balancer configuration through MetalLB or node properties. diff --git a/docs/05-Concepts/04-Kubernetes/03-cluster-api.md b/docs/05-Concepts/04-Kubernetes/03-cluster-api.md new file mode 100644 index 00000000..a9570ede --- /dev/null +++ b/docs/05-Concepts/04-Kubernetes/03-cluster-api.md @@ -0,0 +1,224 @@ +--- +slug: /cluster-api +title: Cluster API +sidebar_position: 3 +--- + +# Cluster API + +[Cluster API](https://cluster-api.sigs.k8s.io/) is a Kubernetes project that aims to simplify the management of Kubernetes clusters. It provides a declarative way to create, configure, and manage clusters using Kubernetes-style APIs. + +We provide the [Cluster API provider for metal-stack (CAPMS)](https://github.com/metal-stack/cluster-api-provider-metal-stack) infrastructure provider that allows the declaration of Kubernetes clusters. + +:::warning[Beta] +Cluster API with metal-stack is in beta and not yet recommended for production workloads. Please use [Gardener](./02-gardener.md) for production deployments. We are actively looking for exchange and adopters — if you are interested in using Cluster API with metal-stack, please [join our community](/community) to help shape future integration efforts. +::: + +For deployment instructions, see the [Cluster API deployment guide](../04-For%20Operators/03-Deployment/06_cluster-api.md). + +## Architecture + +Cluster API (CAPI) is a CNCF project maintained by a Kubernetes SIG that manages clusters through a management cluster holding `Cluster`, `MachineDeployment`, and `KubeadmControlPlane` resources. A metal-stack-specific CAPI infrastructure provider translates CAPI resources into metal-stack API calls. Control plane nodes are created as Machines on metal-stack infrastructure, and node bootstrapping uses kubeadm (or alternative bootstrap providers). Add-on lifecycle is managed through `ClusterResourceSet` objects. + +```mermaid +graph TB + subgraph "Management Cluster" + CAPI["Cluster API Controllers
Cluster, MachineDeployment,
KubeadmControlPlane"] + CAPMS["CAPMS Provider
Infrastructure controllers"] + CABPK["CABPK
Bootstrap provider"] + CAAPH["CAAPH
Add-on Provider for Helm"] + end + + subgraph "Workload Cluster" + CP["KubeadmControlPlane
Control plane nodes"] + MD["MachineDeployment
Worker nodes"] + CRS["ClusterResourceSet
CNI + CCM add-ons"] + end + + subgraph "metal-stack Infrastructure" + MS["metal-stack API
Machines, networks, firewalls"] + FW["Firewall nodes"] + end + + CAPI --> CAPMS + CAPI --> CABPK + CAPI --> CAAPH + CAPMS --> MS + CABPK --> CP + CAPMS --> MD + CAAPH --> CRS + MS -.->|provisions| CP + MS -.->|provisions| MD + MS -.->|provisions| FW + + style CAPI fill:#bbdefb + style CAPMS fill:#c8e6c9 + style CABPK fill:#fff9c4 + style CAAPH fill:#f8bbd0 + style MS fill:#e0f2f1 +``` + +### Core Components + +| Component | Responsibility | +| --------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| **Management Cluster** | A Kubernetes cluster that hosts the Cluster API controllers and the desired cluster state (as custom resources). It is the central control plane from which new workload clusters are declared and reconciled. | +| **Workload Cluster** | A Kubernetes cluster whose lifecycle is managed by the Management Cluster via CAPI resources. Its control plane and worker nodes are provisioned according to the declarative spec. | +| **Infrastructure Provider** | A set of controllers that translate CAPI's generic infrastructure resources (Cluster, Machine) into provider-specific resources. metal-stack is an officially listed infrastructure provider for Cluster API. The provider implementation is called cluster-api-provider-metal-stack (CAPMS). | +| **Bootstrap Provider** | Generates bootstrap data (e.g., cloud-init or ignition userdata) for joining new nodes. CAPMS is tested against the Kubeadm Bootstrap Provider (CABPK). | +| **ClusterResourceSet** | A CAPI feature for applying add-on resources (CNI, CCM, etc.) to newly created workload clusters automatically. CAPMS uses this with the Add-on Provider for Helm (CAAPH) for its calico flavor, to install calico as CNI and the [metal-ccm](./04-cloud-controller-manager.md). | +| **ClusterClass** | Defines reusable templates with parameterized variable schemas for tenant customization, enabling standardized cluster templates across the fleet. | +| **MachineHealthCheck** | Checks machine healthiness and takes remediation steps — unhealthy machines are automatically replaced by adding and deleting them on the infrastructure provider side, with safeguards in place (e.g., for not losing etcd quorum). | + +For a complete architecture overview with diagrams, see the [Cluster API documentation](https://cluster-api.sigs.k8s.io/user/concepts#concepts). + +## CAPMS CRDs + +CAPMS implements the CAPI infrastructure provider contract for bare metal via metal-stack. The following CRDs are provided by CAPMS: + +| CRD | Purpose | +| ------------------------------ | ---------------------------------------------------------------------------------- | +| `MetalStackCluster` | Infrastructure cluster resource — allocates a control plane virtual IP (VIP) | +| `MetalStackMachine` | Bridges CAPI infrastructure machines to metal-stack machines (bare metal servers) | +| `MetalStackMachineTemplate` | Defines reusable machine specs (image, size, etc.) for MetalStackMachine resources | +| `MetalStackFirewallDeployment` | Declares firewall deployments protecting a cluster's network perimeter | +| `MetalStackFirewallTemplate` | Provides the configuration template for deployed firewalls | + +## Operational Model + +The operational model for Cluster API is less opinionated than the one described for [Gardener](./02-gardener.md). We recommend administrators to fully take care of the management cluster and control the declarative state of the entire infrastructure using GitOps-driven workflows. For this scenario, administrators provide end-users with workload clusters and end-users do not gain access to the management clusters. This approach is very controlled and does not allow end-users to provide clusters in a self-service fashion. If such a behavior is desired, we recommend developing another API layer on top of Cluster API that solves this specific purpose. + +In comparison to the Gardener approach, the Cluster API model is much less complex and more versatile, allowing for more individual configurations at the cost of operational overhead, time and scalability. With the kubeadm controller there is no physical isolation between the Kubernetes control plane and the end-user. Provisioned clusters should be configured with proper RBAC permissions for end-users such that unintended misconfiguration or meltdown is prevented. + +Changes need to be coordinated individually per cluster. Cluster API itself does not provide a maintenance time window and reconciles continuously instead. Unlike Gardener, the Kubernetes version skew policy is not strictly enforced in Cluster API — risk management falls within the scope of platform administration and deployment processes with approval gates. + +## Control Plane Hosting + +In the case of Cluster API with bootstrap provider Kubeadm (CABPK), the Kubernetes control planes reside on the worker nodes in the same cluster and are not fully isolated from the end-users. This hierarchy is a mandatory setup in Cluster API when using the Kubeadm provider. + +There are other providers from the CABPK ecosystem (e.g. [Kamaji](https://kamaji.clastix.io/)), which allow similar hosting models as the one described for Gardener, where the control plane runs on dedicated infrastructure separate from worker nodes. However, those integrations have not been evaluated in production-grade scenarios — at least from our side. + +### Control Plane Topologies + +Cluster API supports multiple control plane topologies for on-prem failure domains: + +| Topology | Description | Use Case | +| ------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------- | +| **Single-site HA** | Multiple control plane Machines within a single partition with etcd replicas on separate Machines. Natively supported. | Single-site deployments, standard production | +| **Multi-failure-domain** | Control plane Machines distributed across multiple zones or regions with etcd spread accordingly. Natively supported through ClusterClass topology definitions. | Rack/zone-level failure isolation | +| **Multi-site** | Control plane Machines deployed across multiple CAPI management clusters or across widely separated MetalPools with cross-site etcd replication. Requires additional operator effort for cross-site networking. | Disaster recovery across geographically separated sites | +| **Dedicated isolation** | A dedicated Cluster with its own isolated MetalPool and exclusive use of MetalPools. Same isolation level as Gardener's dedicated Seed for critical infrastructure. | Strictest compliance requirements for critical infrastructure | + +All topologies are natively supported. Multi-site requires additional multi-pool configuration and cross-site networking setup. + +### Kamaji with metal-stack + +[Kamaji](https://kamaji.clastix.io/) is a Control Plane Manager for Kubernetes that runs control planes as pods within a management cluster, reducing operational overhead and costs. It supports multi-tenancy, high availability, and integrates with Cluster API as a `ControlPlaneProvider`. + +Kamaji allows a similar control plane hosting model as Gardener, where the control plane runs on dedicated infrastructure separate from worker nodes. + +:::warning +Kamaji integrations with metal-stack have not been evaluated in production-grade scenarios. We are actively looking for exchange and adopters — if you are interested in using Kamaji with metal-stack, please [join our community](/community) to help shape future integration efforts. +::: + +Kamaji acts as a `ControlPlaneProvider` with Cluster API, while CAPMS acts as the `InfrastructureProvider`. This setup manages **tenant clusters** on metal-stack infrastructure, combining Kamaji's control plane management with metal-stack's bare-metal provisioning. + +Like Cluster API itself, Kamaji is a framework rather than a complete platform — operators must assemble their own day-2 tooling (CNI, CCM, DNS, backup, certificate management) and manage it through GitOps workflows. What Kamaji changes is _where the control plane runs_, not _how much you have to build_. + +A working showcase is available in the [`capi-lab`](https://github.com/metal-stack/cluster-api-provider-metal-stack/blob/main/DEVELOPMENT.md#running-the-kamaji-flavor) setup, which extends the `mini-lab` with a Kamaji flavor. See our [blog post](/blog/2026/04-kamaji) for a detailed walkthrough of the architecture and setup, and the [Cluster API deployment guide](../04-For%20Operators/03-Deployment/06_cluster-api.md#kamaji-as-control-plane-provider) for the deployment steps. + +## Domain Model + +CAPI does not depend in any form on metal-stack components or APIs. metal-stack has implemented the integration in the form of the CAPMS infrastructure provider. The different domains are abstracted/integrated as follows: + +- **Bare metal machine provisioning** — Abstraction through Cluster resource, which holds references to a specific control plane and infrastructure provider, for which dedicated CRDs exist. For the metal-stack integration, the dedicated resources are provided by the cluster-api-provider-metal-stack. +- **Network** — Abstraction through Kubernetes CNI, infrastructure provider and ClusterResourceSet. It is unaware of the concrete network infrastructure and the infrastructure provider can set up resources dynamically if necessary. +- **Storage** — Abstraction through Kubernetes CSI and ClusterResourceSet. Identical to network with the exception that the infrastructure provider does not really participate in providing CSI for the workload cluster. +- **Kubernetes Distribution** — CAPMS currently integrates with CABPK for installing vanilla Kubernetes via kubeadm. This relies on vanilla Kubernetes; other distributions have not been explored. + +CAPI produces clusters built on vanilla Kubernetes. As long as a replacement KCLM also supports vanilla Kubernetes, no action is required. + +## Outcomes + +- **Automation:** CAPI reconciles Cluster, KubeadmControlPlane, and MachineDeployment CRDs from a management cluster. The metal-stack CAPI provider allocates machines, firewalls, and IPs declaratively via manifests. Scaling, upgrading, deletion, and add-on installation (via ClusterResourceSet + Helm) follow the same reconciler loop. +- **Reproducibility:** CAPI stores all resources (Cluster, ClusterClass, MachineTemplate, MetalPool) as Git-versioned YAML. ClusterClass defines reusable templates with parameterized variable schemas for tenant customization. GitOps operators (ArgoCD/FluxCD) ensure drift-free declarative delivery. +- **Risk reduction:** CAPI isolates each managed Cluster in its own namespace within the management cluster. The reconciler continuously compares actual infrastructure against desired spec. Failed nodes trigger automatic MachineHealthCheck remediation. Each managed cluster is independent, so automation errors affect only the target cluster. All operations are Git-auditable through versioned manifests. + +## Network Integration + +Network integration for Cluster API is currently more manual compared to Gardener. Node networks must be created manually via `metalctl` and provided as environment variables. IP addresses for the control plane also need to be allocated in advance through `metalctl`. Firewall rules are currently static and can be applied to firewall nodes; no automatic firewall controller is in place yet. Automatic network resource allocation is on the roadmap for CAPMS. + +For service exposure, CAPMS uses KubeVIP in BGP mode to allocate and announce public IPs, similar to the MetalLB-based approach in Gardener. + +## Air-Gapped Environments + +For air-gapped deployments, follow the [Cluster API Operator air-gapped environment guide](https://cluster-api-operator.sigs.k8s.io/topics/configuration/air-gapped-environtment). All required images must be mirrored to an OCI registry reachable from the management cluster. + +## Fleet Management and GitOps + +You must set up your own Git repository and GitOps operator to manage cluster deployments. + +**What you need to build:** + +1. **Git repository** — Store cluster manifests generated via `clusterctl generate cluster `. Each cluster gets its own set of YAML files containing `Cluster`, `MetalStackCluster`, `KubeadmControlPlane`, `MachineDeployment`, and `MetalStackMachine` resources. +2. **GitOps operator** — Deploy ArgoCD or FluxCD to watch your Git repository and apply manifests to the management cluster, ensuring drift-free declarative delivery. +3. **Per-cluster CI/CD** — Essential components (CNI, CCM) are rolled out on a per-cluster basis. Changes to `MachineTemplate` or `ClusterResourceSet` are staged through the Git repository with standard approval processes. + +**Platform capabilities:** + +- **Cluster migration** — `clusterctl move` enables moving workload cluster resources between management clusters, pausing controllers during the move to prevent worker node loss +- **Emergency patching** — Achieved through editing resources in the management cluster, e.g., update of machine OS image in the `MachineTemplate` or update of `ClusterResourceSet`. Unlike Gardener, this change is not rolled out fleet-wide automatically and should be staged through the Git repository with standard approval processes +- **Certificate rotation** — No direct workflow is described for certificate rotation at the landscape level; this is to be defined by platform administrators using manual/custom processes based on standard tooling (e.g., `kubeadm` certificate renewal) +- **Deletion protection** — Cluster API offers no equivalent to Gardener's deletion-confirmation annotations. Protection has to come from the GitOps layer (branch protection, approval gates) and from Kubernetes finalizers +- **Cluster Autoscaler** — Automatic scaling of worker groups based on requested pod resources, configured per `MachineDeployment` + +**Audit configuration** — Audit configuration can be passed to the kube-apiserver via the `kubeadmConfigSpec` of the `KubeadmControlPlane` resource before cluster creation. Each cluster can have its own audit policy. The management cluster's kube-apiserver audit also needs to be configured separately. Cluster API does not provide a centralized audit management toolset and relies on cloud-native standards to be set up by the operator. + +## External Dependencies + +The following data center infrastructure dependencies are treated as given and must be available before deploying Cluster API with metal-stack: + +- **DNS** — For cluster domain resolution +- **NTP** — Time synchronization across all nodes +- **ACME** — Certificate authority +- **S3-compatible storage** — For backups +- **Git-Hosting with CI/CD** — For GitOps-driven deployment of manifests + +## Scale & Testing + +For Cluster API, the integration test environment covers a management cluster with three worker nodes. The biggest test clusters have included 8 cluster nodes. As both CAPI and Gardener share the same metal-stack control plane, from the metal-stack perspective the [numbers mentioned for Gardener](./02-gardener.md#scalability) (280 clusters, 5 data centers, 1,800 physical servers) also apply here. Validations in CAPI are not as thoroughly implemented as in Gardener, but all usual management workflows for metal-stack clusters (creation, move, and deletion) are integration tested within a matrix of Kubernetes version, CNI, and OS version. + +## Fleet Operations + +Unlike Gardener, Cluster API does not provide built-in fleet-wide operations. All operational changes must be applied through GitOps-driven workflows with approval processes. The risk of platform updates can be reduced by utilizing multi-stage environments (staging → production). Cluster manifests should be validated through CI/CD pipelines before deployment to prevent misconfigurations and manage divergence across the fleet. + +End-users can test Kubernetes upgrades first in clusters labeled `evaluation` or `development` before rolling out to `production`-labeled clusters. For evaluation clusters, auto-upgrades for machine specs can be enabled to stage updates and reduce manual effort. + +## Upgrade & Rollback + +**Minor version upgrades** — Control plane upgrades are triggered by spec updates in the `Cluster` resource, which should be managed in Git using GitOps-driven processes for deployment and approval. Cluster API orchestrates a one-by-one worker node roll. There are no maintenance time windows — CAPI reconciles continuously. + +**Blue-green updates** — End-users can achieve zero-downtime upgrades through two approaches: (1) using multiple clusters with BGP Anycast to spread workloads across clusters, or (2) using worker groups or machine pools with different Kubernetes kubelet versions and OS versions, combined with Kubernetes node taints and tolerations for traffic routing. + +**Rollback** — Kubernetes versions are not allowed to be rolled back. End-users are required to test Kubernetes upgrades in a staging cluster first. Tools exist to test for deprecated APIs before running the actual upgrade. + +**Downtime expectations** — Every Kubernetes or machine update triggers an orchestrated worker roll, updating the Kubernetes control planes and kubelets of the worker nodes one-by-one. Unlike Gardener, there is no jittered upgrade window to prevent simultaneous route announcement vanishing. + +## MEP-19 — Cross-Partition Clusters + +With metal-stack Enhancement Proposal 19 (MEP-19), routing across data center partitions will be supported, allowing worker nodes to reside in separate metal-stack partitions while maintaining a single Kubernetes cluster. This requires partitions to be geographically close enough for stable low-latency connectivity. This enhancement will enable Cluster API to provision clusters with workers spread across partitions, similar to the multi-failure-domain topology. + +## What to Build Yourself + +Unlike Gardener, Cluster API with metal-stack requires you to assemble your own day-2 operations tooling: + +- **DNS management** — No built-in DNS service; configure via external DNS providers +- **etcd backup & restore** — No built-in operator; deploy etcd-druid or similar tooling +- **Certificate rotation** — No direct workflow; use manual/custom processes based on `kubeadm` certificate renewal +- **Audit logging** — Configure via `kubeadmConfigSpec` per cluster; no centralized management +- **Maintenance windows** — None built-in; CAPI reconciles continuously +- **Version skew enforcement** — Not strictly enforced; risk management via approval gates +- **Hibernation** — Not available +- **Cluster migration** — Available via `clusterctl move` between management clusters +- **Multi-tenant self-service** — Not available; build custom API layer on top of Cluster API +- **Access control lists** — No built-in firewall controller; firewall rules are currently static diff --git a/docs/05-Concepts/04-Kubernetes/04-cloud-controller-manager.md b/docs/05-Concepts/04-Kubernetes/04-cloud-controller-manager.md new file mode 100644 index 00000000..945353ab --- /dev/null +++ b/docs/05-Concepts/04-Kubernetes/04-cloud-controller-manager.md @@ -0,0 +1,42 @@ +--- +slug: /metal-cloud-controller-manager +title: Cloud Controller Manager +sidebar_position: 4 +--- + +# metal Cloud Controller Manager + +The [cloud-controller-manager](https://kubernetes.io/docs/concepts/architecture/cloud-controller/) (CCM) is the bridge between Kubernetes and a cloud provider. metal-stack provides its own implementation, [metal-ccm](https://github.com/metal-stack/metal-ccm), which implements the [Kubernetes cloud provider interface](https://github.com/kubernetes/cloud-provider/blob/master/cloud.go). + +## Purpose + +metal-ccm provides metal-stack-specific integration for Kubernetes clusters, primarily: + +- **Load balancer configuration** — Exposes `Service` resources of type `LoadBalancer` by allocating and announcing public IP addresses via MetalLB +- **Node property management** — Annotates nodes with infrastructure-specific labels (rack, region, zone, chassis) for topology-aware scheduling + +## Load Balancer + +metal-ccm integrates with [MetalLB](https://metallb.universe.tf/) in BGP mode to expose Kubernetes `Service` resources of type `LoadBalancer`. When a user creates a LoadBalancer service, metal-ccm: + +1. Allocates a public IP address from the metal-stack infrastructure pool +2. Configures MetalLB to announce that IP via BGP to the network topology + +This provides layer-4 load balancing without requiring dedicated hardware load balancers. + +## Node Labels + +metal-ccm annotates every worker node with well-known Kubernetes topology labels that reflect the underlying bare-metal infrastructure: + +- `machine.metal-stack.io/rack` — The rack where the physical server resides +- `machine.metal-stack.io/chassis` — The chassis identifier +- `topology.kubernetes.io/region` — The geographic region +- `topology.kubernetes.io/zone` — The availability zone or rack + +These labels enable end-users to configure Pod topology spread constraints and anti-affinity rules, ensuring high availability of workloads across failure domains. + +## Deployment + +For Gardener deployments, metal-ccm is deployed as part of the Shoot cluster provisioning flow via the `gardener-extension-provider-metal`. For Cluster API deployments, it is installed through `ClusterResourceSet` objects alongside the CNI (Calico). + +For detailed build and deployment instructions, see the [metal-ccm reference guide](../../08-References/Kubernetes/metal-ccm/metal-ccm.md). diff --git a/docs/05-Concepts/04-Kubernetes/04-firewall-controller-manager.md b/docs/05-Concepts/04-Kubernetes/04-firewall-controller-manager.md deleted file mode 100644 index 5c11ac57..00000000 --- a/docs/05-Concepts/04-Kubernetes/04-firewall-controller-manager.md +++ /dev/null @@ -1,15 +0,0 @@ ---- -slug: /firewall-controller-manager -title: Firewall Controller Manager -sidebar_position: 4 ---- - -# Firewall Controller Manager - -To make the firewalls created with metal-stack easily configurable through Kubernetes resources, we add our [firewall-controller](https://github.com/metal-stack/firewall-controller) to the firewall image. The controller watches special CRDs, enabling users to manage: - -- nftables rules -- Intrusion-detection with [suricata](https://suricata.io/) -- network metric collection - -Please check out the [guide](../../08-References/Kubernetes/firewall-controller-manager/firewall-controller-manager.md) on how to use it. diff --git a/docs/05-Concepts/04-Kubernetes/05-firewall-controller-manager.md b/docs/05-Concepts/04-Kubernetes/05-firewall-controller-manager.md new file mode 100644 index 00000000..ab365140 --- /dev/null +++ b/docs/05-Concepts/04-Kubernetes/05-firewall-controller-manager.md @@ -0,0 +1,45 @@ +--- +slug: /firewall-controller-manager +title: Firewall Controller Manager +sidebar_position: 5 +--- + +# Firewall Controller Manager + +The firewall-controller-manager (FCM) is a collection of controllers responsible for managing the lifecycle of firewalls in a [Gardener](https://gardener.cloud/) shoot cluster for the metal-stack provider. + +The FCM is typically deployed into the shoot namespace of a seed cluster by the [gardener-extension-provider-metal](https://github.com/metal-stack/gardener-extension-provider-metal/). + +The design of the FCM is inspired by Gardener's [machine-controller-manager](https://github.com/gardener/machine-controller-manager) and Kubernetes' built-in resources `Deployment`, `ReplicaSet` and `Pod`. + +## Architecture + +The FCM introduces the following [CRDs](https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/): + +| Custom Resource | Description | +| -------------------- | --------------------------------------------------------------------------------------------------------------------------------- | +| `FirewallDeployment` | Contains the spec template of a `Firewall` resource, similar to a `Deployment`. Implements update strategies like rolling update. | +| `FirewallSet` | Similar to `ReplicaSet`. Typically owned by a `FirewallDeployment`. Attempts to run the defined number of `Firewall` replicas. | +| `Firewall` | Similar to a `Pod`. Has a 1:1 relationship to a firewall in the metal-stack API. | +| `FirewallMonitor` | Deployed into the user's shoot cluster. Useful for monitoring the firewall or triggering user-initiated actions. | + +### Controllers + +| Controller | Responsibility | +| ------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `FirewallDeploymentController` | Manages the lifecycle of `FirewallSet`s. Syncs the `Firewall` template spec and triggers a `FirewallSet` roll when significant changes are made. Supports `RollingUpdate` and `Recreate` strategies. Also deploys a service account for the firewall-controller. | +| `FirewallSetController` | Creates and deletes `Firewall` objects according to the spec and replica count. Reports `Firewall` status. | +| `FirewallController` | Creates and deletes the physical firewall machine via the [metal-api](https://github.com/metal-stack/metal-api). | + +## User Actions + +The FCM exposes user-facing capabilities through annotations on `FirewallMonitor` and `Firewall` resources: + +- **FirewallSet rolling** — Users can trigger a rolling update of the current firewall set by annotating a `FirewallMonitor`. This is useful when firewall configuration changes need to be applied. +- **Systemd service restart** — Users can restart systemd services on a firewall node by annotating a `FirewallMonitor`. The firewall-controller enforces a whitelist of allowed services. Operators can override this whitelist by annotating the `Firewall` resource directly. + +## Deployment + +The FCM is deployed into the shoot namespace of a seed cluster as part of the Gardener extension provisioning flow. + +For detailed configuration and development instructions, see the [firewall-controller-manager reference guide](../../08-References/Kubernetes/firewall-controller-manager/firewall-controller-manager.md). diff --git a/docs/05-Concepts/04-Kubernetes/05-isolated-clusters.md b/docs/05-Concepts/04-Kubernetes/06-isolated-clusters.md similarity index 99% rename from docs/05-Concepts/04-Kubernetes/05-isolated-clusters.md rename to docs/05-Concepts/04-Kubernetes/06-isolated-clusters.md index 3d45c6c5..3469970d 100644 --- a/docs/05-Concepts/04-Kubernetes/05-isolated-clusters.md +++ b/docs/05-Concepts/04-Kubernetes/06-isolated-clusters.md @@ -1,7 +1,7 @@ --- slug: /isolated-kubernetes-clusters title: Isolated Clusters -sidebar_position: 5 +sidebar_position: 6 --- # Isolated Kubernetes Clusters diff --git a/docs/05-Concepts/04-Kubernetes/07-storage.md b/docs/05-Concepts/04-Kubernetes/07-storage.md index 9573f003..9d60e437 100644 --- a/docs/05-Concepts/04-Kubernetes/07-storage.md +++ b/docs/05-Concepts/04-Kubernetes/07-storage.md @@ -18,7 +18,7 @@ In the meantime, we have started to integrate third-party solutions into our met - Built-in multi-tenant capabilities - Configurable compression and replication factors -We are maintaining an open source integration for running LightOS in our [Gardener](./01-gardener.md) cluster provisioning. You can enable it through the controller registration of the [gardener-extension-provider-metal](https://github.com/metal-stack/gardener-extension-provider-metal). +We are maintaining an open source integration for running LightOS in our [Gardener](./02-gardener.md) cluster provisioning. You can enable it through the controller registration of the [gardener-extension-provider-metal](https://github.com/metal-stack/gardener-extension-provider-metal). With the integration in place, the extension-provider deploys a [duros-controller](https://github.com/metal-stack/duros-controller) along with a Duros Storage CRD into the seed's shoot namespace. The duros-controller takes care of creating projects and managing credentials at the Lightbits Duros API. It also provides storage classes as configured in the extension-provider's controller registration to the customer's shoot cluster such that users can start consuming the Lightbits storage immediately. @@ -35,7 +35,7 @@ metadata: name: csi-pvc spec: accessModes: - - ReadWriteOnce + - ReadWriteOnce resources: requests: storage: 100Mi diff --git a/docs/05-Concepts/04-Kubernetes/gardener-cluster-hierarchy.drawio.svg b/docs/05-Concepts/04-Kubernetes/gardener-cluster-hierarchy.drawio.svg new file mode 100644 index 00000000..48d46387 --- /dev/null +++ b/docs/05-Concepts/04-Kubernetes/gardener-cluster-hierarchy.drawio.svg @@ -0,0 +1,1270 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+
+
+ Garden Cluster (soil) +
+
+
+ + + Garden Cluster (soil) + + + + + + + + + + + + + + +
+
+
+ Virtual Garden +
+
+
+ + + Virtual Garden + + + + + + + + + + + + + + +
+
+
+ Seed (shooted seed) +
+
+
+ + + Seed (shooted seed) + + + + + + + + + + + + + + +
+
+
+ Shoot 1 +
+
+
+ + + Shoot 1 + + + + + + + + + + + + + + + + + + +
+
+
+ firewall +
+
+
+ + + firewall + + + + + + + + + + + + + + + +
+
+
+ firewall-controller +
+
+
+ + + firewall-controller + + + + + + + + + + + + +
+
+
+ Backup Buckets +
+
+
+ + + Backup Buck... + + + + + + + + + + + + + + + + + + + +
+
+
+ gardenctl +
+
+
+ + + gardenctl + + + + + + + + + + + + + + + + + + + + + + + + +
+
+
+ bootstrap infrastructure +
+
+
+ + + bootstrap infrastructure + + + + + + + + + + + +
+
+
+ metalstack.cloud, any cloud provider or on-prem +
+
+
+ + + metalstack.cloud, any cloud provider or on-prem + + + + + + + + + + + +
+
+
+ s3 compatible +
+ storage +
+
+
+
+ + + s3 compatible... + + + + + + + + + + + + +
+
+
+ metal-stack partition +
+
+
+ + + metal-stack partition + + + + + + + + + + + +
+
+
+ on-prem +
+
+
+ + + on-prem + + + + + + + + + + + + +
+
+
+ + Gardener Deployment + +
+
+
+ + + Gardener Deployment + + + + + + + + + + + + + + + +
+
+
+ kubectl +
+
+
+ + + kubectl + + + + + + + + + + + + +
+
+
+ + manages + +
+
+
+ + + manages + + + + + + + + + + + + + + + + + + + + + + + +
+
+
+ user +
+
+
+ + + user + + + + + + + + + + + + + + +
+
+
+ tenant-api +
+
+
+ + + tenant-api + + + + + + + + + + + metal-console + + + + + + + + + + +
+
+
+ ipam +
+
+
+ + + ipam + + + + + + + + + + + + + + + +
+
+
+ metal-apiserver +
+
+
+ + + metal-apiserver + + + + + + + + + + + + + + +
+
+
+ metal control plane +
+
+
+ + + metal control plane + + + + + + + + + + + + + + + + + + + + + +
+
+
+ + garden-kube-apiserver + +
+
+
+ + + garden-kube-apiserver + + + + + + + + + + + +
+
+
+ metal-api +
+
+
+ + + metal-api + + + + + + + + + + + + + + + + + + + + + + + +
+
+
+ Shoot 2 +
+
+
+ + + Shoot 2 + + + + + + + + + + + + + + + + + + + + + + + + +
+
+
+ + Shoot 1 + +
+
+
+ + + Shoot 1 + + + + + + + + + + + + + + + + + +
+
+
+ + manages + +
+
+
+ + + manages + + + + + + + + + + + + +
+
+
+ + manages + +
+
+
+ + + manages + + + + + + + + + + + + +
+
+
+ etcd +
+
+
+ + + etcd + + + + + + + + + + + + + + + + + + + +
+
+
+ garden +
+
+
+ + + garden + + + + + + + + + + + +
+
+
+ seed-admission-controller +
+
+
+ + + seed-admission-controller + + + + + + + + + + + + + + + +
+
+
+ gardenlet +
+
+
+ + + gardenlet + + + + + + + + + + + +
+
+
+ resource-manager +
+
+
+ + + resource-manager + + + + + + + + + + + + + + + + + + +
+
+
+ provider-metal +
+
+
+ + + provider-metal + + + + + + + + + + + +
+
+
+ provider-gcp +
+
+
+ + + provider-gcp + + + + + + + + + + + +
+
+
+ dns-external +
+
+
+ + + dns-external + + + + + + + + + + + + + + + +
+
+
+ os-metal +
+
+
+ + + os-metal + + + + + + + + + + + + + + + + + + + + + + + + + +
+
+
+ + Shoot 2 + +
+
+
+ + + Shoot 2 + + + + + + + + + + + +
+
+
+ + extension + +
+ + providers + +
+
+
+
+ + + extension... + + + + + + + + + + + +
+
+
+ shoot +
+ + control planes + +
+
+
+
+ + + shoot... + + + + + + + + + + + +
+
+
+ Workload +
+
+
+ + + Workload + + + + + + + + + + + + + + + + +
+
+
+ kube-apiserver +
+
+
+ + + kube-apiserver + + + + + + + + + + + + + + +
+
+
+ Workload +
+
+
+ + + Workload + + + + + + + + + + + + + + + + +
+
+
+ Workload +
+
+
+ + + Workload + + + + + + + + + + + + + + + + +
+
+
+ Workload +
+
+
+ + + Workload + + + + + + + + + + + + + + + + + + + + + + + + +
+
+
+ apiserver +
+
+
+ + + apiserver + + + + + + + + + + + +
+
+
+ controller-manager +
+
+
+ + + controller-manager + + + + + + + + + + + +
+
+
+ scheduler +
+
+
+ + + scheduler + + + + + + + + + + + +
+
+
+ admission-controller +
+
+
+ + + admission-controller + + + + + + + + + + + + + + + + + + + + Text is not SVG - cannot display + + + + diff --git a/docs/05-Concepts/04-Kubernetes/gardener-cluster-hierarchy.png b/docs/05-Concepts/04-Kubernetes/gardener-cluster-hierarchy.png new file mode 100644 index 00000000..3ecd9133 Binary files /dev/null and b/docs/05-Concepts/04-Kubernetes/gardener-cluster-hierarchy.png differ diff --git a/docs/06-For CISOs/Security/04-communication-matrix.md b/docs/06-For CISOs/Security/04-communication-matrix.md index b911b146..1abf73fb 100644 --- a/docs/06-For CISOs/Security/04-communication-matrix.md +++ b/docs/06-For CISOs/Security/04-communication-matrix.md @@ -116,7 +116,7 @@ Please note that every [networking setup](../../05-Concepts/03-Network/01-theory | VLAN | Switches, Firewalls | Layer 2 traffic segmentation. | | VXLAN | Switches, Firewalls | Encapsulate Layer 2 frames in Layer 3 packets for network virtualization. | | EVPN | Switches, Firewalls | Overlay network technology for scalable and flexible network architectures. | -| VPN | Firewalls | Management access [without open SSH ports](/community/MEP-9-no-open-ports-to-the-data-center). | +| VPN | Firewalls | Management access [without open SSH ports](/community/MEP-9-no-open-ports-to-the-data-center). | | BGP | Multiple | Routing protocol for dynamic routing and network management. | | SSH | Management Server, Switches | Secure shell access for management and configuration. | | LLDP | Switches, Machines | Link Layer Discovery Protocol for network device discovery. | @@ -124,7 +124,7 @@ Please note that every [networking setup](../../05-Concepts/03-Network/01-theory ## With Gardener -When using metal-stack in [conjunction with Gardener](../../05-Concepts/04-Kubernetes/01-gardener.md), the following communication is required by metal-stack components. +When using metal-stack in [conjunction with Gardener](../../05-Concepts/04-Kubernetes/02-gardener.md), the following communication is required by metal-stack components. :::info The following table might not be displayed in completeness. Scroll to the right to see all entries. @@ -154,7 +154,7 @@ The following table might not be displayed in completeness. Scroll to the right ## With Cluster API -By using the [Cluster API provider for metal-stack](../../05-Concepts/04-Kubernetes/02-cluster-api.md), the following communictations are required by metal-stack components. +By using the [Cluster API provider for metal-stack](../../05-Concepts/04-Kubernetes/03-cluster-api.md), the following communications are required by metal-stack components. :::info The following table might not be displayed in completeness. Scroll to the right to see all entries.