From 1c847ecbc71cac46b3b86351257acbf8d3ede110 Mon Sep 17 00:00:00 2001 From: Matthias Hartmann Date: Sat, 1 Aug 2026 02:42:19 +0200 Subject: [PATCH 1/2] feat: concepts guide overhaul --- .../03-Deployment/01_guide.md | 2 +- .../03-Deployment/03_control-plane.mdx | 63 + .../03-Deployment/04_partition.md | 3 +- .../03-Deployment/05_gardener.md | 695 +++++++++ .../04-For Operators/03-Deployment/05_kclm.md | 198 --- .../03-Deployment/06_cluster-api.md | 111 ++ .../{06-gpu-workers.md => 07-gpu-workers.md} | 2 +- ...resilience.md => 08_offline-resilience.md} | 2 +- docs/05-Concepts/04-Kubernetes/01-gardener.md | 78 - docs/05-Concepts/04-Kubernetes/01-kclm.md | 379 +++++ .../04-Kubernetes/02-cluster-api.md | 18 - docs/05-Concepts/04-Kubernetes/02-gardener.md | 242 ++++ .../03-cloud-controller-manager.md | 11 - .../04-Kubernetes/03-cluster-api.md | 243 ++++ .../04-cloud-controller-manager.md | 49 + .../04-firewall-controller-manager.md | 15 - .../05-firewall-controller-manager.md | 54 + ...ed-clusters.md => 06-isolated-clusters.md} | 2 +- .../gardener-cluster-hierarchy.drawio.svg | 1270 +++++++++++++++++ .../gardener-cluster-hierarchy.png | Bin 0 -> 301951 bytes 20 files changed, 3112 insertions(+), 325 deletions(-) create mode 100644 docs/04-For Operators/03-Deployment/05_gardener.md delete mode 100644 docs/04-For Operators/03-Deployment/05_kclm.md create mode 100644 docs/04-For Operators/03-Deployment/06_cluster-api.md rename docs/04-For Operators/03-Deployment/{06-gpu-workers.md => 07-gpu-workers.md} (99%) rename docs/04-For Operators/03-Deployment/{07_offline-resilience.md => 08_offline-resilience.md} (99%) delete mode 100644 docs/05-Concepts/04-Kubernetes/01-gardener.md create mode 100644 docs/05-Concepts/04-Kubernetes/01-kclm.md delete mode 100644 docs/05-Concepts/04-Kubernetes/02-cluster-api.md create mode 100644 docs/05-Concepts/04-Kubernetes/02-gardener.md delete mode 100644 docs/05-Concepts/04-Kubernetes/03-cloud-controller-manager.md create mode 100644 docs/05-Concepts/04-Kubernetes/03-cluster-api.md create mode 100644 docs/05-Concepts/04-Kubernetes/04-cloud-controller-manager.md delete mode 100644 docs/05-Concepts/04-Kubernetes/04-firewall-controller-manager.md create mode 100644 docs/05-Concepts/04-Kubernetes/05-firewall-controller-manager.md rename docs/05-Concepts/04-Kubernetes/{05-isolated-clusters.md => 06-isolated-clusters.md} (99%) create mode 100644 docs/05-Concepts/04-Kubernetes/gardener-cluster-hierarchy.drawio.svg create mode 100644 docs/05-Concepts/04-Kubernetes/gardener-cluster-hierarchy.png diff --git a/docs/04-For Operators/03-Deployment/01_guide.md b/docs/04-For Operators/03-Deployment/01_guide.md index 75bb9125..35ab72da 100644 --- a/docs/04-For Operators/03-Deployment/01_guide.md +++ b/docs/04-For Operators/03-Deployment/01_guide.md @@ -29,7 +29,7 @@ The deployment has multiple phases, resulting in a Git repository containing Ans 1. **[Control Plane](./03_control-plane.mdx)** — Deploy the metal-stack API, databases, and ingress on a bootstrap Kubernetes cluster. This alone gives you a functional MaaS platform. 2. **[Partition](./04_partition.md)** — Configure the network fabric (leaf switches, management spine, DHCP, PXE) and connect your bare-metal servers to the control plane. -3. **[KCLM with Gardener](./05_kclm.md)** — Add Gardener to manage Kubernetes cluster lifecycle on your bare-metal infrastructure. +3. **[KCLM with Gardener](./05_gardener.md)** — Add Gardener to manage Kubernetes cluster lifecycle on your bare-metal infrastructure. By the end of this guide, your deployment repository will look something like this: diff --git a/docs/04-For Operators/03-Deployment/03_control-plane.mdx b/docs/04-For Operators/03-Deployment/03_control-plane.mdx index 2b6fee10..b2d702fd 100644 --- a/docs/04-For Operators/03-Deployment/03_control-plane.mdx +++ b/docs/04-For Operators/03-Deployment/03_control-plane.mdx @@ -14,6 +14,8 @@ While it is theoretically possible to deploy metal-stack without Kubernetes (see The control plane deployment described here requires an initial Kubernetes cluster as bootstrap infrastructure, which is described in [Bootstrap Infrastructure](./02_bootstrap-infrastructure.md). We use the [metal-stack control-plane Ansible roles](https://github.com/metal-stack/metal-roles/tree/master/control-plane) to deploy all control-plane components. +Before you try to deploy the control-plane, you should have had a look at the [architecture](../../05-Concepts/01-architecture.mdx) concepts section. + Create the deployment folder and enter it — this is the root of your deployment repository as outlined in the [Deployment Guide](./01_guide.md): ```bash @@ -431,6 +433,67 @@ docker run --rm -it \ If you are having issues regarding the deployment take a look at the [troubleshoot document](../06-troubleshoot.md). Please give feedback such that we can make the deployment of the metal-stack easier for you and for others! ::: +## GitHub Action + +You can also automate the control plane deployment through a GitHub Action workflow. Here is an example `.github/workflows/deploy-control-plane.yaml`: + +```yaml +--- +name: Deploy control plane + +on: + workflow_dispatch: + inputs: + deploy-control-plane: + description: 'Which control-plane target to deploy' + required: true + type: choice + options: + - metal-stack + +env: + ANSIBLE_INVENTORY: inventory/inventory.yaml + ANSIBLE_FORCE_COLOR: "1" + ANSIBLE_JINJA2_NATIVE: "True" + + # Update these with your actual values + CLUSTER_ID: + DEFAULT_PROJECT_ID: + + KUBECONFIG: /tmp/.kubeconfig + +jobs: + metal-stack: + name: Deploy metal-stack control plane + if: ${{ inputs.deploy-control-plane == 'metal-stack' }} + + runs-on: ubuntu-latest + container: ghcr.io/metal-stack/metal-deployment-base:v0.9.2 + + steps: + - name: Checkout + uses: actions/checkout@v7 + + - run: | + metal ctx add demo --api-token ${METALSTACKCLOUD_API_TOKEN} --default-project ${DEFAULT_PROJECT_ID} --activate + metal cluster kubeconfig ${CLUSTER_ID} + + echo ${ANSIBLE_VAULT_PASSWORD} > ${ANSIBLE_VAULT_PASSWORD_FILE} + + ansible localhost -m metalstack.base.metal_stack_release_vector + ansible-playbook deploy_metal_stack.yaml + env: + METALSTACKCLOUD_API_TOKEN: ${{ secrets.METALSTACKCLOUD_API_TOKEN }} + ANSIBLE_VAULT_PASSWORD: ${{ secrets.ANSIBLE_VAULT_PASSWORD }} + ANSIBLE_VAULT_PASSWORD_FILE: .vault.txt +``` + +:::tip +The example uses GitHub Actions with `ubuntu-latest` runners (not self-hosted) and a runtime cluster hosted on [metal-stack cloud](https://metalstack.cloud/en). +As mentioned in [Bootstrap Infrastructure](./02_bootstrap-infrastructure.md), any Kubernetes cluster can serve as the runtime — you can adapt the workflow to use self-hosted runners or a different cluster provider as needed. +The `metal` CLI is used to fetch the kubeconfig from the metal-stack API. Update the `CLUSTER_ID` and `DEFAULT_PROJECT_ID` environment variables with your actual values. +::: + ## Providing Images After the deployment has finished, you should consider deploying some masterdata entities into your metal-api. For example, you can add your first machine sizes and operating system images. You can do this by further parametrizing the [metal role](https://github.com/metal-stack/metal-roles/tree/master/control-plane/roles/metal). We will just add an operating system for demonstration purposes. Add the following variable to your `inventories/group_vars/control-plane/common.yaml`: diff --git a/docs/04-For Operators/03-Deployment/04_partition.md b/docs/04-For Operators/03-Deployment/04_partition.md index 0e4e0236..5fc640e4 100644 --- a/docs/04-For Operators/03-Deployment/04_partition.md +++ b/docs/04-For Operators/03-Deployment/04_partition.md @@ -10,6 +10,7 @@ A partition is the data center infrastructure layer — the physical servers, sw This section continues from the [Control Plane](./03_control-plane.mdx) deployment and covers how to deploy the required infrastructure services and how to connect your partition to the control plane, using the [metal-stack partition Ansible roles](https://github.com/metal-stack/metal-roles/tree/master/partition). It is assumed that all cabling is done. +Before you try to deploy the partition, you should have had a look at the [architecture](../../05-Concepts/01-architecture.mdx) and [networking](../../05-Concepts/03-Network/) concepts sections. During this section, our repository will grow to look something like the following: @@ -145,7 +146,7 @@ After the initial bootstrapping, the management interfaces of the leaves continu In larger deployments, a dedicated set of out-of-band switches (mgmtoobs) may be used to isolate BMC/IPMI traffic from the management network. These switches connect directly to server BMCs and provide a separate L2 domain for IPMI traffic, keeping it isolated from management server and switch management interfaces. They are deployed through the same SONiC automation as other partition switches. ### Leaves and Spines (Production Network) -We have now reached the point where a lot of the configuration happens automatically via the official Ansible roles. +We have now reached the point where a lot of the configuration happens automatically via the official Ansible roles. After the initial install via ONIE and ZTP, metal-core will take over the configuration of the leaves and spines. The general role of the leaves and spines is explained in the [CLOS](https://metal-stack.io/docs/next/networking#clos) concepts section. diff --git a/docs/04-For Operators/03-Deployment/05_gardener.md b/docs/04-For Operators/03-Deployment/05_gardener.md new file mode 100644 index 00000000..b49cee33 --- /dev/null +++ b/docs/04-For Operators/03-Deployment/05_gardener.md @@ -0,0 +1,695 @@ +--- +slug: /deployment/gardener +title: Gardener (KCLM) +sidebar_position: 5 +--- + +# Deploying Gardener with metal-stack + +This guide shows how to deploy [Gardener](https://gardener.cloud/) on top of your metal-stack infrastructure using the [`gardener-*` Ansible roles](https://github.com/metal-stack/metal-roles/tree/master/control-plane) from the [metal-roles](https://github.com/metal-stack/metal-roles) repository. Gardener with metal-stack turns your bare-metal servers into a Kubernetes-as-a-Service platform where teams can self-serve clusters. + +This guide assumes you are already familiar with Gardener's core concepts — [Garden](https://gardener.cloud/docs/gardener/concepts/operator/), [Seed](https://gardener.cloud/docs/gardener/concepts/gardenlet/), [Shoot](https://gardener.cloud/docs/gardener/concepts/apiserver/) and [CloudProfile](https://gardener.cloud/docs/gardener/concepts/apiserver/#cloudprofiles) — and have already completed the [Control Plane](./03_control-plane.mdx) and [Partition](./04_partition.md) deployment guides. For an overview of how Gardener integrates with metal-stack at a conceptual level, see the [Gardener concepts guide](../../05-Concepts/04-Kubernetes/02-gardener.md). + +The upstream [Gardener landscape setup guide](https://gardener.cloud/docs/gardener/deployment/setup_gardener/) describes the same building blocks — operator, `Garden`, extensions, `CloudProfile`, DNS secrets, `Gardenlet`, `ManagedSeed` — but expects you to render and apply the manifests yourself. The `gardener-*` roles are a thin, opinionated automation layer over exactly those building blocks, pre-wired for metal-stack: each role owns one resource and pulls all container images and Helm chart references from the metal-stack [release vector](./03_control-plane.mdx#releases-and-ansible-role-dependencies). + +:::tip +The [mini-lab](https://github.com/metal-stack/mini-lab) contains a working, minimal Gardener deployment (`deploy_gardener.yaml` plus `inventories/group_vars`) that uses the very same roles. It is a **development environment**, not a production reference, but it is the fastest way to see a complete, runnable parametrization. +::: + +## Repository structure after this section + +The following files are added to the repository structure from the previous sections: + +```text +. +├── deploy_gardener.yaml # Gardener deployment playbook +├── inventories +│ ├── control-plane.yaml # unchanged, reused for Gardener +│ └── group_vars +│ ├── all/ +│ │ └── release_vector.yaml # unchanged +│ └── control-plane/ +│ ├── common.yaml # updated: host provider, stage name +│ └── gardener/ +│ ├── operator.yaml # operator + virtual garden + backup + dns +│ ├── extensions.yaml # provider/OS/CNI/shoot extensions +│ ├── cloud_profile.yaml # CloudProfile +│ ├── projects.yaml # Projects +│ ├── gardenlet.yaml # first (unmanaged) Seed +│ ├── shoots.yaml # Shoots / shooted seeds +│ ├── managed_seeds.yaml # optional: ManagedSeeds +│ └── secrets.yaml # vault-encrypted credentials +└── .github/ + └── workflows/ + └── deploy-control-plane.yaml # updated: adds a gardener job +``` + +## Architecture Overview + +The metal-roles deploy Gardener in the **virtual Garden** pattern described upstream: the `gardener-operator` runs on your *runtime cluster* and reconciles a `Garden` resource, which spins up a nodeless *virtual Garden* cluster hosting the Gardener API (`Shoot`, `Seed`, `Project`, `CloudProfile`, …). The gardenlet also runs on the runtime cluster and registers it as the first, unmanaged `Seed` (a "soil"). + +```mermaid +graph TD + subgraph RT["Runtime cluster (= soil / first Seed)"] + OP["gardener-operator"] + EXTRES["Extension resources"] + GLPOD["gardenlet pods"] + end + subgraph VG["Virtual Garden (nodeless)"] + API["kube-apiserver + gardener-apiserver"] + CP["CloudProfile metal"] + PR["Projects"] + GLRES["Gardenlet resource"] + SEEDA["Seed seed-a"] + end + subgraph SEED["Shooted seed seed-a (bare metal)"] + USH["End-user shoot control planes"] + end + OP -->|reconciles Garden| API + OP -->|registers| EXTRES + EXTRES -->|ControllerRegistrations| API + GLRES -->|deploys| GLPOD + GLPOD -->|registers soil as Seed| API + CP --> SEEDA + PR --> SEEDA + SEEDA -->|ManagedSeed| SEED + SEED --> USH +``` + +The soil is reserved for *infrastructure* shoots. Those shoots are turned into Gardener-managed Seeds via `ManagedSeed`, and end-user shoot control planes are hosted there. This is the recommended upstream pattern and the one the roles are built for. + +**Order matters.** Every role except `gardener-operator` and `gardener-extensions` obtains a kubeconfig for the virtual Garden through the `virtual_garden_kubeconfig` module, which only works once the operator has created the `Garden` and `gardener-virtual-garden-access` has deployed the token-requestor secret: + +| Role | Applies to | Deploys | Requires | +| ---- | ---------- | ------- | -------- | +| `gardener-operator` | runtime cluster | `garden` namespace, backup + DNS provider secrets, operator Helm chart, `Garden` resource | Runtime cluster; cert-manager only if the dashboard is enabled | +| `gardener-extensions` | runtime cluster | one `operator.gardener.cloud/v1alpha1` `Extension` per enabled extension | `gardener-operator` | +| `gardener-virtual-garden-access` | both | `ManagedResource` + token-requestor secret that yields a rotating virtual-Garden kubeconfig | `Garden` reconciled | +| `gardener-cloud-profile` | virtual Garden | `CloudProfile` named `metal` | virtual-garden-access | +| `gardener-projects` | virtual Garden | `Project` resources | virtual-garden-access | +| `gardener-gardenlet` | virtual Garden | internal/default domain secrets, backup secret, `Gardenlet` resource (first Seed) | virtual-garden-access, extensions | +| `gardener-shoots` | virtual Garden | provider `Secret` + `CredentialsBinding` + `Shoot` per entry | `CloudProfile`, `Project`, a ready `Seed` | +| `gardener-managed-seeds` | virtual Garden | backup secret + `ManagedSeed` per entry | a reconciled shooted seed | + +Two optional roles are not part of the minimal setup but are worth knowing about: `gardener-monitoring-certs` (wildcard monitoring certificates for soil and seeds, requires reachable seed API servers) and `gardener-logging` (ships seed logs to the central metal-stack Loki, requires the `logging` role). + +## Prerequisites + +Before deploying Gardener, ensure the following is in place. This mirrors the [upstream prerequisites](https://gardener.cloud/docs/gardener/deployment/setup_gardener/#prerequisites), with the metal-stack specifics added: + +- **A runtime cluster** — An existing Kubernetes cluster hosting the Gardener control plane. It can be the same cluster as your metal-stack control plane, but a dedicated cluster is recommended for production. See [Bootstrap Infrastructure](./02_bootstrap-infrastructure.md). +- **An ingress controller** — The roles assume `ingress-nginx` (the `Garden` and `Gardenlet` templates hardcode `ingress.controller.kind: nginx`). The virtual Garden API server itself is exposed through the Istio gateway that the operator deploys. +- **A DNS zone plus credentials** — Gardener needs to create records for the virtual Garden API server, the shoot internal and default domains, and seed ingress. Supported provider types come from the deployed DNS extensions (e.g. `google-clouddns` via `provider-gcp`, `powerdns` via the `dns-powerdns` extension). +- **A backup bucket** — S3-compatible or GCP object storage for the virtual Garden etcd and for every Seed. `gardener-operator` **asserts** that `gardener_operator_backup_infrastructure.provider` is either `gcp` or `S3`. +- **cert-manager with a DNS-solving `ClusterIssuer`** — Only required if you let the operator deploy the Gardener dashboard; the role then requests a wildcard certificate for the ingress domain. +- **ACME account** — The `shoot-cert-service` extension is enabled by default and requires `gardener_extension_shoot_cert_service_issuer_email` to be set (asserted). +- **metal-stack API credentials** — An admin/edit HMAC key for the metal-api, handed to shoots via the provider secret. +- **Cluster networking facts** — `gardener-operator` derives the runtime cluster's node, pod and service CIDRs. For `metal_control_plane_host_provider: metal` it reads them from the `kube-system/shoot-info` ConfigMap (present in every Gardener-managed shoot); for `gcp` it queries `gcloud` and additionally needs `gcp_cluster_name` and `gcp_region`. + +:::warning +`metal_control_plane_host_provider` has **no default** and is asserted to be `metal` or `gcp`. If your runtime cluster is neither a metal-stack shoot nor a GKE cluster, you must provide a `kube-system/shoot-info` ConfigMap with `nodeNetwork`, `podNetwork` and `serviceNetwork` yourself — this is exactly what the mini-lab does in its playbook's `pre_tasks`. +::: + +## Step 1: Add the Playbook + +Create `deploy_gardener.yaml` in your repository root. It reuses the `control-plane` inventory from the [Control Plane](./03_control-plane.mdx#inventory) guide (a single `localhost` entry, because everything is applied to Kubernetes) and chains the `gardener-*` roles in the order shown above: + +```yaml +--- +- name: Deploy Gardener + hosts: control-plane + connection: local + gather_facts: false + roles: + - name: ansible-common + tags: always + - name: metal-roles/control-plane/roles/gardener-operator + - name: metal-roles/control-plane/roles/gardener-extensions + - name: metal-roles/control-plane/roles/gardener-virtual-garden-access + - name: metal-roles/control-plane/roles/gardener-cloud-profile + - name: metal-roles/control-plane/roles/gardener-projects + - name: metal-roles/control-plane/roles/gardener-gardenlet + # add once the first seed is ready: + # - name: metal-roles/control-plane/roles/gardener-shoots + # - name: metal-roles/control-plane/roles/gardener-managed-seeds +``` + +You do not need to list `metal-roles/common/roles/defaults` or the `gardener-defaults` role explicitly — each `gardener-*` role pulls them in through its `meta/main.yml` dependencies. Including `ansible-common` is what makes the custom modules (`setup_yaml`, `virtual_garden_kubeconfig`, `discovery_api_k8s`) and filters (`machine_images_for_cloud_profile`, `shoot_admin_kubeconfig`) available. + +:::info +Deploy in two passes on a green field: first everything up to `gardener-gardenlet`, verify that the `Garden` reports `RuntimeComponentsHealthy` and `VirtualComponentsHealthy` and that the `Seed` becomes `GardenletReady`, then enable `gardener-shoots`. `gardener-shoots` waits for each `Shoot` to report `lastOperation.state: Succeeded` (36 retries, 10 s apart by default) and fails if no Seed can host it. +::: + +To inspect the landscape afterwards, obtain a kubeconfig for the virtual Garden the same way the roles do: + +```yaml + post_tasks: + - name: Get kubeconfig for virtual garden access + virtual_garden_kubeconfig: + garden_name: "{{ gardener_defaults_garden_name }}" + + - name: Write it out for manual inspection + ansible.builtin.copy: + content: "{{ virtual_garden_kubeconfig }}" + dest: .virtual-garden-kubeconfig + mode: "0600" +``` + +## Step 2: Configure Group Variables + +All Gardener configuration lives under `inventories/group_vars/control-plane/gardener/`. Each file maps to a role. Container image names, image tags and Helm chart references are resolved from the release vector, so you normally only set the variables shown here. Sensitive values (HMAC keys, service account JSONs, ACME private keys) belong in an Ansible vault file. + +Two shared variables must be set in `inventories/group_vars/control-plane/common.yaml`: + +```yaml +# Names the Gardener landscape; also becomes the Garden resource name and the +# metalControlPlanes key in the CloudProfile. Defaults to metal_control_plane_stage_name. +metal_control_plane_stage_name: demo + +# Mandatory and asserted: "metal" or "gcp". Determines how the runtime cluster +# CIDRs are discovered and becomes the Seed's provider type. +metal_control_plane_host_provider: metal +``` + +:::tip +`gardener_defaults_garden_name` defaults to `{{ metal_control_plane_stage_name }}` and is inherited by every role (`gardener_operator_garden_name`, `gardener_cloud_profile_garden_name`, …). Keep the default unless you have a reason to diverge — the gardenlet name must match the Seed name across upgrades. +::: + +### operator.yaml — Operator, virtual Garden, backup and DNS + +This role creates the `garden` namespace, installs the operator Helm chart and applies the `Garden` resource. Backup and DNS configuration belong to the same role because both are referenced by the `Garden`. + +```yaml +# --- Virtual Garden ------------------------------------------------------- +# Domain under which the virtual Garden kube-apiserver is exposed through Istio. +gardener_operator_virtual_garden_public_dns: gardener-kube-apiserver.{{ metal_control_plane_ingress_dns }} + +# Domain for runtime-cluster ingresses (monitoring, dashboard). MANDATORY, no default. +gardener_operator_ingress_dns_domain: k8s. + +# Storage class for the virtual Garden etcd volumes (20Gi main, 10Gi events). +gardener_operator_virtual_garden_etcd_storage_class: csi-lvm + +# Renders spec.runtimeCluster.provider.region in the Garden resource. +gardener_operator_runtime_cluster_provider: local + +# Multi-replica etcd + control plane for the virtual Garden. +gardener_operator_high_availability_control_plane: true + +# --- etcd backup (asserted: provider must be "gcp" or "S3") --------------- +gardener_operator_backup_infrastructure: + provider: S3 + bucket: my-garden-backup-bucket + region: europe-west3 + +gardener_operator_backup_infrastructure_secret: + endpoint: "{{ garden_backup_endpoint | b64encode }}" + accessKeyID: "{{ garden_backup_access_key | b64encode }}" + secretAccessKey: "{{ garden_backup_secret_key | b64encode }}" + +# --- DNS providers -------------------------------------------------------- +gardener_operator_dns_providers: + - name: powerdns + type: powerdns + secretData: + apiKey: "{{ powerdns_api_key | b64encode }}" + server: "{{ powerdns_server | b64encode }}" + +# --- Dashboard (optional) ------------------------------------------------- +gardener_operator_dashboard_enabled: false +``` + +#### Notes on the operator configuration + +- `gardener_operator_backup_infrastructure_secret` is applied verbatim as the `data:` of the `virtual-garden-etcd-main-backup-secret`, so all values must already be base64-encoded. The same applies to each DNS provider's `secretData`. Consult the [Gardener etcd backup secret examples](https://gardener.cloud/docs/gardener/deployment/setup_gardener/#garden) for the expected keys per provider. +- `provider: S3` requires the `backup-s3` extension, `provider: gcp` the `provider-gcp` extension (see below) — the extension is what reconciles the `BackupBucket` in the runtime cluster. +- The `Garden` template only renders `spec.runtimeCluster.ingress` and `spec.virtualCluster.dns` **if `gardener_operator_dns_providers` is non-empty**, and always uses `gardener_operator_dns_providers[0].type` as the provider for both. Put your primary provider first. +- The virtual Garden service CIDR is fixed to `100.64.0.0/13` and the maintenance window to `220000+0100`–`230000+0100` by the template. Make sure `100.64.0.0/13` does not overlap with your runtime cluster or partition networks. +- Enabling the dashboard additionally requires `gardener_operator_wildcard_ingress_certificate_cluster_issuer` plus a cert-manager `ClusterIssuer`; the role then blocks until the wildcard certificate secret exists (up to 10 minutes). For OIDC login set `gardener_operator_dashboard_oidc_issuer_url`, `..._client_id`, `..._client_id_public` and `..._client_secret`. +- If automatic DNS creation is not available, create the A record for `gardener_operator_virtual_garden_public_dns` manually, pointing at the external address of the `istio-ingressgateway` service in the `virtual-garden-istio-ingress` namespace. + +### extensions.yaml — Provider and shoot extensions + +This role applies one `operator.gardener.cloud/v1alpha1` `Extension` resource per enabled extension into the **runtime** cluster. The operator then translates each into a `ControllerDeployment` and `ControllerRegistration` in the virtual Garden, exactly as described in the [upstream extension registration docs](https://gardener.cloud/docs/gardener/extensions/registration/). + +Every extension follows the same pattern: `gardener_extension__enabled` toggles it, and an assert makes sure the corresponding Helm chart reference resolved from the release vector. These are **enabled by default**: `provider-metal`, `provider-gcp`, `os-metal`, `networking-calico`, `networking-cilium`, `shoot-cert-service`, `shoot-dns-service`. All others default to `false`. + +```yaml +# --- Infrastructure provider (the essential one) -------------------------- +gardener_extension_provider_metal_enabled: true + +# Machine images offered to Shoot workers. Defaults to metal_api_images, so it +# stays in sync with the images you registered in the metal-api. +gardener_extension_provider_metal_machine_images: "{{ metal_api_images | default([]) }}" + +# Shoot etcd: storage class and backup cadence +gardener_extension_provider_metal_etcd_storage_class_name: csi-lvm +gardener_extension_provider_metal_etcd_backup_schedule: "0 */2 * * *" +gardener_extension_provider_metal_etcd_delta_snapshot_period: "5m" + +# Defaults injected by the admission controller when a Shoot omits CIDRs +gardener_extension_provider_metal_admission_default_pods_cidr: 10.248.64.0/18 +gardener_extension_provider_metal_admission_default_services_cidr: 10.248.192.0/18 + +# --- Operating system ----------------------------------------------------- +# One OperatingSystemConfig resource is registered per type. +gardener_extension_os_metal_types: + - ubuntu + - debian + +# --- CNI: keep only what you actually offer in the CloudProfile ----------- +gardener_extension_networking_cilium_enabled: true +gardener_extension_networking_calico_enabled: false + +# --- Shoot services ------------------------------------------------------- +# shoot-cert-service is enabled by default and its issuer email is asserted. +gardener_extension_shoot_cert_service_issuer_email: support@example.com +gardener_extension_shoot_cert_service_issuer_private_key: "{{ acme_account_private_key }}" + +# DNS for shoot API servers and shoot-owned records +gardener_extension_dns_powerdns_enabled: true + +# --- Backup provider matching gardener_operator_backup_infrastructure ----- +gardener_extension_backup_s3_enabled: true + +# --- Optional add-ons ----------------------------------------------------- +gardener_extension_csi_driver_lvm_enabled: true +gardener_extension_acl_enabled: false +gardener_extension_audit_enabled: false + +# Not needed unless your runtime cluster is GKE +gardener_extension_provider_gcp_enabled: false +``` + +#### Notes on the extensions + +- If you do not run on GCP, disable `provider-gcp` explicitly — it is on by default and would otherwise register an unusable `DNSRecord/google-clouddns` and `BackupBucket/gcp` handler. +- `provider-metal` embeds an `imageVectorOverwrite` pinning the metal-stack components deployed into shoots (`metal-ccm`, `firewall-controller-manager`, `machine-controller-manager-provider-metal`, `csi-lvm-*`, `droptailer`, `node-init`) to the versions from your release vector. This is why shoot components stay consistent across the fleet. +- `os-metal` also accepts `nvidia` for GPU worker groups — see [GPU Workers](./07-gpu-workers.md). +- Some variables were renamed and the role **fails hard** if you still use the old names (e.g. `gardener_cert_management_issuer_email` → `gardener_extension_shoot_cert_service_issuer_email`). The failure message names the replacement. +- The `duros` extension defaults still point at a pre-release chart; do not enable it in production. + +For the complete list of extension variables see the [`gardener-extensions` role README](https://github.com/metal-stack/metal-roles/tree/master/control-plane/roles/gardener-extensions). + +### cloud_profile.yaml — Defining your metal-stack infrastructure + +The CloudProfile is a Gardener resource that describes your metal-stack infrastructure capabilities: available Kubernetes versions, machine types, regions, and zones. The `gardener-cloud-profile` role renders this from your group vars into a `CloudProfile` Kubernetes resource. + +```yaml +# URL of the metal-api — critical for Gardener to provision infrastructure +gardener_cloud_profile_metal_api_url: https://api. + +# Firewall images (auto-derived from machine images if not specified) +gardener_cloud_profile_firewall_images_from_machine_images: true +gardener_cloud_profile_firewall_images: + - firewall-ubuntu-3.0 + +# Firewall controller versions +gardener_cloud_profile_firewall_controller_versions: + - version: v2.5.0 + url: https://images.metal-stack.io/firewall-controller/v2.5.0/firewall-controller + classification: supported + +# Available Kubernetes versions for Shoot clusters +gardener_cloud_profile_kubernetes: + versions: + - version: 1.33.13 + - version: 1.34.9 + - version: 1.35.6 + +# Available machine types +gardener_cloud_profile_machine_types: + - name: c1-medium-x86 + cpu: "8" + gpu: "0" + memory: 128Gi + usable: true + storage: + class: standard + type: default + size: 960G + +# Available regions and zones +gardener_cloud_profile_regions: + - name: "{{ metal_region }}" + zones: + - name: demo-rack + +# Partition configuration +gardener_cloud_profile_partitions: + demo-rack: + default-machine-types: + firewall: + - c1-medium-x86 +``` + +#### Notes on the CloudProfile + +- `gardener_cloud_profile_kubernetes` and `gardener_cloud_profile_regions` are the only **asserted** variables of this role. The resulting resource is always named `metal` with `spec.type: metal`; `gardener_cloud_profile_stage_name` becomes the key under `providerConfig.metalControlPlanes`, which is what shoots reference to reach your metal-api. +- `gardener_cloud_profile_metal_api_url` is derived automatically: `https://api.{{ metal_control_plane_gateway_dns }}` when `metal_api_httproute_enabled` is true, otherwise `https://api.{{ metal_control_plane_ingress_dns }}`. Only override it if your metal-api lives elsewhere. +- `gardener_cloud_profile_machine_images` defaults to `metal_api_images`, so CloudProfile and `provider-metal` always agree. With `gardener_cloud_profile_firewall_images_from_machine_images: true` (default) every image carrying the `firewall` feature is additionally offered as a firewall image — the explicit `gardener_cloud_profile_firewall_images` list is then only needed for extras. +- The role maps images to Gardener `machineImages` via `gardener_cloud_profile_os_cri_mapping`, which by default only covers `ubuntu` and `debian`. Add an entry for any further OS (for example `nvidia`) or its versions will be dropped from the CloudProfile. +- Use `gardener_cloud_profile_os_compatibility_mapping` to express kubelet/OS-version constraints, and Gardener's `classification` plus `expirationDate` fields inside `gardener_cloud_profile_kubernetes.versions` to steer deprecation and auto-updates. +- `zones` map to metal-stack partitions. `gardener_cloud_profile_partitions..default-machine-types.firewall` restricts the firewall sizes selectable in that partition; an optional `network-isolation` key enables [isolated clusters](../../05-Concepts/04-Kubernetes/06-isolated-clusters.md). +- By default the role waits until the `CloudProfile` CRD is served by the virtual Garden before applying, which is what makes an initial bootstrap succeed on the first run. + +### projects.yaml — Team isolation + +Gardener `Project`s isolate teams and give them a namespace in the virtual Garden. You need at least one project to create shoots. + +```yaml +gardener_project_defaults: + namespace: garden + owner: admin + protected_toleration: true + members: [] + +gardener_projects: + - name: infrastructure # holds the shooted seeds + description: Infrastructure clusters + - name: prod + owner: alice@example.com + members: + - kind: User + name: bob@example.com + role: admin + roles: [admin] +``` + +With `protected_toleration: true` (the default) the project may schedule shoots onto Seeds tainted with `seed.gardener.cloud/protected`. Keep this on for the project that holds your shooted seeds and consider turning it off for end-user projects, so their shoots never land on the soil. + +:::info +The first project uses `namespace: garden`, which already exists. Additional projects get their own namespace derived from the project name. +::: + +### gardenlet.yaml — Registering the first Seed + +The gardenlet registers your runtime cluster as the first, unmanaged Seed ("soil"). The role also deploys the `internal-domain` and `default-domain` secrets that Gardener uses for shoot DNS records — these are what the upstream guide calls the [DNS setup for internal and external domains](https://gardener.cloud/docs/gardener/deployment/setup_gardener/#dns-setup-for-internal--external-domains). + +```yaml +# All three are MANDATORY and asserted. +gardener_gardenlet_default_dns_domain: k8s. +gardener_gardenlet_default_dns_provider: powerdns +gardener_gardenlet_default_dns_credentials: + apiKey: "{{ powerdns_api_key | b64encode }}" + server: "{{ powerdns_server | b64encode }}" + +gardener_gardenlets: + - name: "{{ gardener_defaults_garden_name }}" + + # Required: every Seed needs its own backup configuration. + backup_infrastructure: + provider: S3 + region: europe-west3 + bucket: my-seed-backup-bucket + backup_infrastructure_secret: + endpoint: "{{ seed_backup_endpoint | b64encode }}" + accessKeyID: "{{ seed_backup_access_key | b64encode }}" + secretAccessKey: "{{ seed_backup_secret_key | b64encode }}" + + # Ingress domain of this Seed; falls back to the default DNS domain. + dns_domain: "soil.{{ gardener_defaults_garden_name }}." + + # Keep the soil invisible to the scheduler so only tolerating shoots + # (your shooted seeds) land here. + visible: false + taints: + - seed.gardener.cloud/protected + + additional_labels: + cluster.metal-stack.io/partition: demo-rack + +# Gardenlet defaults applied to all seeds +# shoot_reconcile_in_maintenance_only: true — only reconcile shoots during their maintenance window +# shoot_respect_sync_period_overwrite: true — respect the syncPeriod on Shoot specs +gardener_gardenlet_defaults: + shoot_reconcile_in_maintenance_only: true + shoot_respect_sync_period_overwrite: true +``` + +#### Notes on the gardenlet + +- The gardenlet name should equal the Seed name and must remain stable — renaming it creates a new Seed instead of adopting the existing one. +- Pod and service CIDRs default to the runtime cluster's CIDRs, read from the `Garden` resource. Override them per gardenlet with `pods:` / `services:` if they would overlap with shoot networks. +- `taints: [seed.gardener.cloud/protected]` is the default. Combined with `visible: false` this reserves the soil for infrastructure shoots, matching the [upstream recommendation](https://gardener.cloud/docs/gardener/deployment/setup_gardener/#gardenlet). +- When the gardenlet must run in a **different** cluster than the operator, set `kubeconfigSecretRef` to a manually created kubeconfig secret plus `garden_client_connection.gardenClusterAddress: https://` — see [Deploy Gardenlet via Operator](https://gardener.cloud/docs/gardener/deployment/deploy_gardenlet_via_operator/#remote-clusters). +- `shoot_reconcile_in_maintenance_only: true` and `shoot_respect_sync_period_overwrite: true` are the defaults, so shoots only reconcile inside their maintenance window and honour the ignore annotation. Adjust `shoot_concurrent_syncs` (default `20`) for large seeds. + +### shoots.yaml — Shoot clusters and shooted seeds + +Each entry produces a provider `Secret`, a `CredentialsBinding` and a `Shoot`. To scale the landscape you create "shooted seeds" here first — infrastructure shoots that are turned into Seeds by the `gardener-managed-seeds` role. + +```yaml +# HMAC key that shoots use against the metal-api. Base64-encoded into the +# per-shoot provider secret. +gardener_shoot_default_metal_api_hmac: "{{ metal_api_admin_key }}" + +# The role waits for each shoot to reach lastOperation.state == Succeeded. +gardener_shoot_rollout_wait_enabled: true +gardener_shoot_rollout_wait_retries: 36 +gardener_shoot_rollout_wait_delay: 10 + +gardener_shoots: + - name: seed-a + seed_name: "{{ gardener_defaults_garden_name }}" # scheduled onto the soil + project_id: + purpose: infrastructure + region: "{{ metal_region }}" + partition: demo-rack + networks: + - internet + - + k8s_version: "1.34.9" + networking_type: cilium + networking_pod_cidr: 10.240.0.0/13 + networking_service_cidr: 10.248.0.0/18 + worker_groups: + - worker_count: 3 + worker_size: c1-medium-x86 + worker_cri: containerd + worker_max_surge: 1 + worker_max_unavailable: 0 + worker_image: + name: debian + version: "12.0" + firewall_size: c1-medium-x86 + firewall_image: firewall-ubuntu-3.0 + high_availability_control_plane: node + csi_driver_lvm_extension: + enabled: true + default_storage_class: csi-lvm + + # Makes this shoot eligible to become a Seed and sizes its API server. + managed_seed: + tolerations: + - key: seed.gardener.cloud/protected + api_server: + replicas: 3 + autoscaler: + min_replicas: 1 + max_replicas: 5 +``` + +#### Notes on shoot configuration + +- `project_id` is the metal-stack project UUID. It is written into the `cluster.metal-stack.io/project` annotation and the `InfrastructureConfig`, and determines which metal-stack project the machines, IPs and firewalls are allocated in. +- `networks` must list the metal-stack network IDs the firewall attaches to. `internet` is required for external reachability; add the partition's private network. +- `worker_size`, `firewall_size` and `firewall_image` must exist in the CloudProfile (and, for the firewall, in the partition's `firewallTypes`). +- `high_availability_control_plane` accepts `node` or `zone`. With a single partition per zone, use `node`. +- `namespace` defaults to `garden`; set it to the project namespace when the shoot belongs to a non-default project. +- Use `credentials_binding_name` to reference an existing binding instead of letting the role create one. The old `secret_binding_name` key is rejected by an assert. +- Optional per-shoot keys include `audit_policy` (rendered into a ConfigMap and wired into the kube-apiserver), `structured_auth_config`, `audit_extension_splunk`, `machine_creation_timeout`, `storage_class_name` and `control_plane_feature_gates`. + +:::warning +`gardener_shoot_rollout_wait_enabled` makes the playbook block for up to six minutes per shoot by default. For landscapes with many shoots, either raise the retries or disable the wait and monitor reconciliation separately. +::: + +### managed_seeds.yaml — Turning shooted seeds into Seeds + +A `ManagedSeed` installs a gardenlet into an existing shoot and registers it as a Seed. This is how the landscape scales: end-user shoot control planes then run on these Gardener-managed seeds instead of on the soil. The `name` **must** match the shoot name defined in `shoots.yaml`. + +```yaml +# Mandatory and asserted — used for the seed's internal DNS domain. +gardener_managed_seed_default_dns_domain: k8s. +gardener_managed_seed_default_dns_provider: powerdns + +gardener_managed_seed_defaults: + visible: true + excess_capacity_reservation: true + external_traffic_policy: Local + +gardener_managed_seeds: + - name: seed-a # must match a shoot from shoots.yaml + region: "{{ metal_region }}" + pod_cidr: 10.240.0.0/13 # must match the shoot's networking + service_cidr: 10.248.0.0/18 + ingress_domain: ingress.seed-a.k8s. + logging_enabled: false + backup_infrastructure: + provider: S3 + region: europe-west3 + bucket: my-seed-a-backup-bucket + backup_infrastructure_secret: + endpoint: "{{ seed_backup_endpoint | b64encode }}" + accessKeyID: "{{ seed_backup_access_key | b64encode }}" + secretAccessKey: "{{ seed_backup_secret_key | b64encode }}" +``` + +#### Notes on managed seeds + +- `pod_cidr` and `service_cidr` must be the CIDRs of the underlying shoot, and they must not overlap with any shoot hosted on that seed. Plan your CIDR ranges before creating seeds. +- `backup_infrastructure_secret` is applied unconditionally, so it must be provided for every managed seed even though the commented example in the role omits it. +- `visible: true` (default) makes the seed eligible for end-user shoots, unlike the soil. `excess_capacity_reservation` keeps spare capacity so new shoot control planes schedule quickly. +- Unlike the soil, the gardenlet bootstraps itself here via a bootstrap token and inherits configuration from the parent gardenlet (`mergeWithParent`). + +## Step 3: Run the Deployment + +Run the playbook exactly like the control plane deployment, using the deployment base image so that the release vector and roles are fetched and verified: + +```bash +export KUBECONFIG= + +docker run --rm -it \ + -v $(pwd):/workdir \ + --workdir /workdir \ + -e KUBECONFIG="${KUBECONFIG}" \ + -e K8S_AUTH_KUBECONFIG="${KUBECONFIG}" \ + -e ANSIBLE_INVENTORY=inventories/control-plane.yaml \ + -e ANSIBLE_JINJA2_NATIVE=True \ + ghcr.io/metal-stack/metal-deployment-base:${METAL_VERSION} \ + /bin/bash -ce \ + "ansible -m metalstack.base.metal_stack_release_vector localhost + ansible-playbook deploy_gardener.yaml" +``` + +The roles are idempotent: if a role fails, fix the configuration and re-run — only missing changes are applied. The mandatory-variable asserts fail fast with `not all mandatory variables given, check role documentation`, so most misconfigurations surface before anything is applied. + +### Verifying the landscape + +Against the **runtime** cluster: + +```bash +kubectl get garden +# NAME K8S VERSION GARDENER VERSION LAST OPERATION RUNTIME VIRTUAL API SERVER OBSERVABILITY +# demo 1.34.9 v1.125.0 Succeeded True True True True + +kubectl get extensions.operator.gardener.cloud +``` + +Against the **virtual Garden** (using the kubeconfig from the playbook's `post_tasks`): + +```bash +export KUBECONFIG=.virtual-garden-kubeconfig +kubectl get cloudprofile metal +kubectl get seeds +kubectl get shoots -A +``` + +Wait for the `Garden` conditions `RuntimeComponentsHealthy` and `VirtualComponentsHealthy`, then for the `Seed` condition `GardenletReady`, before enabling the shoot roles. + +### CI/CD Pipeline + +The Gardener deployment is part of the same workflow as the control plane deployment. Add a `gardener` job to your `.github/workflows/deploy-control-plane.yaml`: + +```yaml +--- +name: Deploy control plane + +on: + workflow_dispatch: + inputs: + deploy-control-plane: + description: 'Which control-plane target to deploy' + required: true + type: choice + options: + - metal-stack + - gardener + +env: + ANSIBLE_INVENTORY: inventories/control-plane.yaml + ANSIBLE_FORCE_COLOR: "1" + ANSIBLE_JINJA2_NATIVE: "True" + + # Update these with your actual values + CLUSTER_ID: + DEFAULT_PROJECT_ID: + + KUBECONFIG: /tmp/.kubeconfig + +jobs: + gardener: + name: Deploy Gardener + if: ${{ inputs.deploy-control-plane == 'gardener' }} + + runs-on: ubuntu-latest + container: ghcr.io/metal-stack/metal-deployment-base:v0.9.2 + + steps: + - name: Checkout + uses: actions/checkout@v7 + + - run: | + metal ctx add demo --api-token ${METALSTACKCLOUD_API_TOKEN} --default-project ${DEFAULT_PROJECT_ID} --activate + metal cluster kubeconfig ${CLUSTER_ID} + + echo ${ANSIBLE_VAULT_PASSWORD} > ${ANSIBLE_VAULT_PASSWORD_FILE} + + ansible localhost -m metalstack.base.metal_stack_release_vector + ansible-playbook deploy_gardener.yaml + env: + METALSTACKCLOUD_API_TOKEN: ${{ secrets.METALSTACKCLOUD_API_TOKEN }} + ANSIBLE_VAULT_PASSWORD: ${{ secrets.ANSIBLE_VAULT_PASSWORD }} + ANSIBLE_VAULT_PASSWORD_FILE: .vault.txt +``` + +:::tip +The example uses GitHub Actions with `ubuntu-latest` runners (not self-hosted) and a runtime cluster hosted on [metal-stack cloud](https://metalstack.cloud/en). As mentioned in [Bootstrap Infrastructure](./02_bootstrap-infrastructure.md), any Kubernetes cluster can serve as the runtime — you can adapt the workflow to use self-hosted runners or a different cluster provider as needed. The `metal` CLI is used to fetch the kubeconfig from the metal-stack API. Update the `CLUSTER_ID` and `DEFAULT_PROJECT_ID` environment variables with your actual values. +::: + +## How the Pieces Connect + +Understanding the data flow between metal-stack and Gardener helps troubleshoot issues: + +```mermaid +graph LR + MAI["metal_api_images"] -->|default| CP["CloudProfile metal"] + MAI -->|default| PE["provider-metal Extension"] + CP -->|referenced by| SH["Shoot"] + HMAC["gardener_shoot_default_metal_api_hmac"] -->|provider Secret + CredentialsBinding| SH + SH -->|reconciled by| PE + PE -->|endpoint from CloudProfile| MA["metal-api"] + MA -->|machines, networks, IPs, firewalls| PART["Partition"] +``` + +| Connection | Configuration Variable | Purpose | +| ---------- | ---------------------- | ------- | +| metal-api URL | `gardener_cloud_profile_metal_api_url` | Endpoint the provider extension calls to provision resources | +| Control plane key | `gardener_cloud_profile_stage_name` | Key under `metalControlPlanes` that shoots resolve to the metal-api | +| Machine images | `metal_api_images` → `gardener_cloud_profile_machine_images` / `gardener_extension_provider_metal_machine_images` | Keeps CloudProfile and extension in sync | +| HMAC secret | `gardener_shoot_default_metal_api_hmac` | Shoot-to-metal-api authentication via the per-shoot provider secret | +| Machine types | `gardener_cloud_profile_machine_types` | Machine sizes selectable for workers and firewalls | +| Regions/zones | `gardener_cloud_profile_regions` | Region and zone (= partition) placement | +| Partition config | `gardener_cloud_profile_partitions` | Firewall types and network isolation per partition | +| Shoot networks | `gardener_shoots[].networks` | metal-stack network IDs the firewall attaches to | +| Shoot partition | `gardener_shoots[].partition` | Which metal-stack partition hosts the workers | + +## Troubleshooting + +| Symptom | Likely cause | +| ------- | ------------ | +| `not all mandatory variables given, check role documentation` | An asserted variable is unset. Check `metal_control_plane_host_provider`, `gardener_operator_backup_infrastructure.provider`, `gardener_operator_ingress_dns_domain`, `gardener_cloud_profile_kubernetes`, `gardener_cloud_profile_regions`, the three `gardener_gardenlet_default_dns_*` variables, `gardener_managed_seed_default_dns_domain` and `gardener_extension_shoot_cert_service_issuer_email`. | +| The role fails with `the variable ... was renamed to ...` | You are using a deprecated variable name; rename it as instructed and remove the old one. | +| `virtual_garden_kubeconfig` times out (120 retries) | The `Garden` is not healthy yet, or `gardener-virtual-garden-access` has not run. Check `kubectl get garden` and the operator logs in the `garden` namespace. | +| `KeyError: 'nodeNetwork'` in the operator role | The runtime cluster has no `kube-system/shoot-info` ConfigMap. Either set `metal_control_plane_host_provider: gcp` or provide the ConfigMap yourself. | +| etcd of the virtual Garden does not reconcile | Add the `druid.gardener.cloud/etcd-druid` finalizer on the `ETCD` resource manually, as noted in the role README. | +| Shoots stay pending | No Seed tolerates them. Check the Seed taints, `visible` setting, and the project's `protected_toleration`. | + +For general deployment issues, see the [troubleshooting guide](../06-troubleshoot.md). + +## Next Steps + +- **[GPU Workers](./07-gpu-workers.md)** — Offer GPU worker groups in shoots +- **[Offline Resilience](./08_offline-resilience.md)** — Operating the landscape without upstream connectivity +- **[Gardener Concepts](../../05-Concepts/04-Kubernetes/02-gardener.md)** — Architecture, operational model and failure domains +- **[Upstream Gardener docs](https://gardener.cloud/docs/)** — API reference and component documentation diff --git a/docs/04-For Operators/03-Deployment/05_kclm.md b/docs/04-For Operators/03-Deployment/05_kclm.md deleted file mode 100644 index 372fc154..00000000 --- a/docs/04-For Operators/03-Deployment/05_kclm.md +++ /dev/null @@ -1,198 +0,0 @@ ---- -slug: /deployment/kclm -title: Kubernetes Cluster Lifecycle Management (KCLM) -sidebar_position: 5 ---- - -# Kubernetes Cluster Lifecycle Management (KCLM) - -This guide covers the deployment of a Kubernetes Cluster Lifecycle Management (KCLM) solution to use metal-stack as a cloud provider. metal-stack supports three KCLM solutions: - -## KCLM Solutions Overview - -### Gardener - -[Gardener](../../05-Concepts/04-Kubernetes/01-gardener.md) is the **recommended** KCLM solution for metal-stack. It is battle-tested in production for over seven years at financial-sector customers and bundles more day-2 capabilities natively (DNS, backup, audit). Gardener manages entire clusters as Kubernetes-native resources with a strong separation between platform operators and end-users. - -:::tip -We recommend using a **dedicated cluster** for Gardener, separate from the metal-stack initial cluster. While it is technically possible to deploy both metal-stack and Gardener on the same initial cluster, dedicated clusters provide better isolation, clearer operational boundaries, and align with production best practices for critical infrastructure. For guidance on setting up the initial cluster, see the [Bootstrap Infrastructure](./02_bootstrap-infrastructure.md) documentation. -::: - -For more details on Gardener terminology, architecture, operational model, failure domains, and operational features, see the [Gardener concept doc](../../05-Concepts/04-Kubernetes/01-gardener.md). - -#### Deployment Summary - -Gardener can be deployed with the `gardener-*` [Ansible roles](https://github.com/metal-stack/metal-roles/tree/master/control-plane/roles). - -The following data center infrastructure dependencies are treated as given and must be available before deploying Gardener: - -- **DNS** — For cluster domain resolution -- **NTP** — Time synchronization across all nodes -- **ACME** — Certificate authority (for shoot certificates via `shoot-cert-service`) -- **S3-compatible object storage** — For etcd backups with `gardener-extension-backup-s3` -- **Git-Hosting with CI/CD** — You must set up your own Git repository and CI/CD pipeline to manage cluster deployments (see [Fleet Management and GitOps](#fleet-management-and-gitops) below). - -The following dependencies are introduced: - -- CNI: Calico or Cilium -- MetalLB for exposing the Kubernetes API servers of the clusters - -In summary, this results in the following cluster hierarchy: - -- **Garden cluster** — The Gardener control plane (Gardener API server, controller manager, scheduler, admission controller) deployed on a dedicated cluster. -- **Seed** — A cluster running the `gardenlet` agent, connected to the Gardener control plane. Seeds are deployed inside the metal-stack partition and orchestrate cluster provisioning within that site. -- **Shoot** — Every fully provisioned and managed Kubernetes cluster. Shoot control planes run as pods in the Seed namespace, while worker nodes are provisioned as bare-metal machines on metal-stack infrastructure. - -:::tip -We are officially supported by [Gardener dashboard](https://github.com/gardener/dashboard). The dashboard helps you manage Shoots, Seeds, and Projects through a web UI. -::: - -#### Core Controllers - -The Gardener platform consists of the following core controllers, all deployed via the Ansible roles: - -| Component | Responsibility | -| ------------------------------------------- | ---------------------------------------------------------------------------------------- | -| `gardener-operator` | Deploys Gardener components, gardenlets, and extensions; manages platform updates | -| `gardener-apiserver` | Extends the kube-apiserver with Gardener-specific resources (Shoot, Seed, Project, etc.) | -| `gardener-scheduler` | Decides where clusters are placed across the Gardener landscape (Seeds) | -| `gardener-controller-manager` | Reconciles common Gardener resources (projects, controller installations, etc.) | -| `gardenlet` | Agent running on each Seed; orchestrates provisioning of new clusters within that Seed | -| `gardener-resource-manager` | Runs inside Shoots; reconciles desired resources and checks their health | -| `etcd-druid` | etcd cluster operator with built-in backup-restore functionality | -| `machine-controller-manager` | Manages worker node lifecycle (rolling updates, health recreation, scaling) | -| `machine-controller-manager-provider-metal` | Integrates metal-stack machine provisioning API with Gardener's MCM | - -#### Gardener Extensions - -Gardener's extensibility model allows provider-specific reconcilers to be deployed during cluster provisioning. The `gardener-extensions` [Ansible role](https://github.com/metal-stack/metal-roles/tree/master/control-plane/roles/gardener-extensions) deploys the following extensions into the Gardener runtime cluster: - -| Extension | Purpose | -| -------------------------------------- | -------------------------------------------------------------------------------------------------------- | -| `gardener-extension-provider-metal` | IaaS integration — reconciles Infrastructure, ControlPlane, and Worker resources via the metal-stack API | -| `os-metal-extension` | Translates Gardener's generic `OperatingSystemConfig` into cloud-init/ignition userdata | -| `gardener-extension-networking-calico` | Calico CNI extension | -| `gardener-extension-networking-cilium` | Cilium CNI extension as an alternative to Calico | -| `gardener-extension-dns-powerdns` | DNS management via PowerDNS | -| `shoot-dns-service` | DNS service for Shoot clusters | -| `gardener-extension-backup-s3` | etcd backup to S3-compatible object storage | -| `gardener-extension-audit` | Audit logging webhook | -| `gardener-extension-acl` | Access control list management | -| `shoot-cert-service` | Certificate management with Let's Encrypt (supports Shoot-level issuers) | -| `gardener-extension-csi-driver-lvm` | LVM-based CSI driver for local storage | -| `gardener-extension-ontap` | NetApp ONTAP CSI driver | - -Most extensions are enabled/disabled via Ansible variables (e.g., `gardener_extension_provider_metal_enabled`). Key configuration variables for the metal provider include: - -- `gardener_extension_provider_metal_etcd_storage_class_name` — Storage class for Shoot etcds -- `gardener_extension_provider_metal_etcd_backup_schedule` — etcd backup schedule -- `gardener_extension_provider_metal_machine_images` — Machine images (typically matches CloudProfile) -- `gardener_extension_provider_metal_admission_default_pods_cidr` — Default pod CIDR for Shoots -- `gardener_extension_provider_metal_admission_default_services_cidr` — Default services CIDR for Shoots - -For the full variable reference, see the [gardener-extensions README](https://github.com/metal-stack/metal-roles/tree/master/control-plane/roles/gardener-extensions). - -#### Fleet Management and GitOps - -You must set up your own Git repository and CI/CD pipeline to manage cluster deployments. This gives you peer review, audit trails, and rollback capabilities. - -**What you need to build:** - -1. **Git repository** — Store the following as YAML manifests: - - `cloudprofiles/` — CloudProfile definitions (whitelisted regions, machine types, OS images, Kubernetes versions) - - `seeds/` — Seed configurations per data center - - `projects//shoots/` — Per-project Shoot manifests - - `extensions/` — Helm charts for Gardener extensions -2. **CI/CD pipeline** — Deploy manifests from Git to the Gardener API (Virtual Garden). This pipeline is your primary interface for fleet-wide changes. -3. **Branching strategy** — Use separate branches or environments (staging → production) to validate changes before rolling them out fleet-wide. - -**Operational capabilities provided by Gardener:** - -Once your GitOps pipeline is in place, Gardener provides the following day-2 operational features: - -- **CloudProfile validation** — Administrators define allowed regions, machine types, operating systems, and Kubernetes versions. Shoot specs are validated against the CloudProfile before being stored in the Virtual Garden's ETCD. -- **Multi-stage environments** — End-users can label clusters as `evaluation` or `development` to test upcoming Kubernetes versions and auto-upgrades before rolling them out to `production` clusters. -- **Maintenance time windows** — Configurable per Shoot; all day-2 operations (Kubernetes patch updates, machine image updates) are carried out within these windows. -- **Emergency patching** — Administrators can apply fleet-wide changes via image vector overwrites in the Gardener deployment Git repository. Changes must be validated in a dedicated staging environment first. -- **Accidental deletion protection** — Shoot deletion is guarded by specific annotations. ETCD backup retention timeouts are configurable, allowing cluster restoration after accidental deletion. - -### Cluster-API - -[Cluster-API](../../05-Concepts/04-Kubernetes/02-cluster-api.md) is a CNCF project maintained by a Kubernetes SIG that provides declarative cluster management through a management cluster. The metal-stack provider (CAPMS) is **under development** and not yet production-ready. - -The [cluster-api-provider-metal-stack (CAPMS)](https://github.com/metal-stack/cluster-api-provider-metal-stack/) infrastructure provider translates CAPI resources into metal-stack API calls for machine, firewall, and IP allocation. CAPMS is tested against the Kubeadm Bootstrap Provider (CABPK) and uses the Add-on Provider for Helm (CAAPH) for installing CNIs like Calico and the metal-ccm. - -:::warning -Cluster-API with metal-stack is in development and not advised for production use. Please use Gardener for production workloads. We are actively looking for exchange and adopters — if you are interested in using Cluster-API with metal-stack, please [join our community](/community) to help shape future integration efforts. -::: - -For more details on Cluster-API concepts, architecture, operational model, and control plane hosting, see the [Cluster-API concept section](../../05-Concepts/04-Kubernetes/02-cluster-api.md). - -Unlike Gardener, which provides a complete Kubernetes-as-a-Service platform with integrated day-2 operations (DNS, backup, certificate rotation, audit), Cluster-API is a declarative cluster management framework. Operators must assemble their own day-2 tooling — CNI, CCM, DNS, backup, and certificate management — and manage them through GitOps workflows. - -#### Deployment - -Cluster-API with metal-stack is deployed through the [cluster-api-provider-metal-stack (CAPMS)](https://github.com/metal-stack/cluster-api-provider-metal-stack/) infrastructure provider. The [CAPMS reference documentation](../../08-References/Kubernetes/cluster-api-provider-metal-stack/cluster-api-provider-metal-stack.md) covers the deployment in detail. - -**Deployment flow** - -1. **Prepare management cluster** — A Kubernetes cluster to host CAPI and CAPMS providers and cluster state -2. **Install CAPMS** — Deploy the CAPMS provider into the management cluster -3. **Configure `clusterctl`** — Register the metal-stack provider URL and set environment variables (API credentials, project, partition, machine images and sizes, cluster name, Kubernetes version) -4. **Generate and apply cluster manifest** — Use `clusterctl generate cluster` to produce a YAML with `Cluster`, `MetalStackCluster`, `KubeadmControlPlane`, `MachineDeployment`, and `MetalStackMachine` resources, then apply it -5. **Deploy add-ons** — Install CNI (Calico) and `metal-ccm` via `ClusterResourceSet` and CAAPH -6. **Retrieve kubeconfig** — Access the provisioned cluster - -**Network integration** - -Network integration for Cluster-API is currently more manual compared to Gardener. Node networks must be created manually via `metalctl` and provided as environment variables. IP addresses for the control plane also need to be allocated in advance through `metalctl`. Firewall rules are currently static and can be applied to firewall nodes; no automatic firewall controller is in place yet. Automatic network resource allocation is on the roadmap for CAPMS. - -For service exposure, CAPMS uses KubeVIP in BGP mode to allocate and announce public IPs, similar to the MetalLB-based approach in Gardener. - -**Air-gapped environments** - -For air-gapped deployments, follow the [Cluster API Operator air-gapped environment guide](https://cluster-api-operator.sigs.k8s.io/topics/configuration/air-gapped-environtment). All required images must be mirrored to an OCI registry reachable from the management cluster. - -**Fleet management and GitOps** - -You must set up your own Git repository and GitOps operator to manage cluster deployments. - -**What you need to build:** - -1. **Git repository** — Store cluster manifests generated via `clusterctl generate cluster `. Each cluster gets its own set of YAML files containing `Cluster`, `MetalStackCluster`, `KubeadmControlPlane`, `MachineDeployment`, and `MetalStackMachine` resources. -2. **GitOps operator** — Deploy ArgoCD or FluxCD to watch your Git repository and apply manifests to the management cluster, ensuring drift-free declarative delivery. -3. **Per-cluster CI/CD** — Essential components (CNI, CCM) are rolled out on a per-cluster basis. Changes to `MachineTemplate` or `ClusterResourceSet` are staged through the Git repository with standard approval processes. - -**Platform capabilities:** - -- **Cluster migration** — `clusterctl move` enables moving workload cluster resources between management clusters, pausing controllers during the move to prevent worker node loss. - -### Kamaji - -[Kamaji](https://kamaji.clastix.io/) is a Control Plane Manager for Kubernetes that runs control planes as pods within a management cluster, reducing operational overhead and costs. It supports multi-tenancy, high availability, and integrates with Cluster API as a `ControlPlaneProvider`. - -Kamaji allows a similar control plane hosting model as Gardener, where the control plane runs on dedicated infrastructure separate from worker nodes. - -:::warning -Kamaji integrations with metal-stack have not been evaluated in production-grade scenarios. We are actively looking for exchange and adopters — if you are interested in using Kamaji with metal-stack, please [join our community](/community) to help shape future integration efforts. -::: - -#### Kamaji with metal-stack - -Kamaji acts as a `ControlPlaneProvider` with Cluster API, while CAPMS acts as the `InfrastructureProvider`. This setup manages **tenant clusters** on metal-stack infrastructure, combining Kamaji's control plane management with metal-stack's bare-metal provisioning. - -Like Cluster-API, Kamaji is a framework rather than a complete platform — operators must assemble their own day-2 tooling (CNI, CCM, DNS, backup, certificate management) and manage them through GitOps workflows. - -**Deployment** - -1. **Prepare management cluster** — A Kubernetes cluster to host Kamaji and CAPMS providers -2. **Install Kamaji and CAPMS** — Deploy both providers into the management cluster -3. **Create a control plane VIP** — MetalLB assigns a virtual IP for the tenant API server -4. **Generate and apply tenant cluster manifest** — Use `clusterctl generate cluster` to produce a YAML with `Cluster`, `MetalStackCluster`, `KubeadmControlPlane`, `MachineDeployment`, and `MetalStackMachine` resources, then apply it -5. **Deploy add-ons** — Install CNI (Calico) and `metal-ccm` into the tenant cluster - -A working showcase is available in the [`capi-lab`](https://github.com/metal-stack/cluster-api-provider-metal-stack/blob/main/DEVELOPMENT.md#running-the-kamaji-flavor) setup, which extends the `mini-lab` with a Kamaji flavor. See our [blog post](/blog/2026/04-kamaji) for a detailed walkthrough of the architecture and setup. - -**Fleet management and GitOps** - -Since Kamaji with metal-stack uses Cluster-API under the hood, fleet management follows the same pattern as Cluster-API. Tenant cluster manifests are generated via `clusterctl`, stored in Git, and deployed through your CI/CD pipeline. diff --git a/docs/04-For Operators/03-Deployment/06_cluster-api.md b/docs/04-For Operators/03-Deployment/06_cluster-api.md new file mode 100644 index 00000000..495a3a4b --- /dev/null +++ b/docs/04-For Operators/03-Deployment/06_cluster-api.md @@ -0,0 +1,111 @@ +--- +slug: /deployment/cluster-api +title: Cluster API (KCLM) +sidebar_position: 6 +--- + +# Cluster API for Kubernetes Cluster Lifecycle Management + +This section covers deploying [Cluster API](../../05-Concepts/04-Kubernetes/03-cluster-api.md) on top of your existing metal-stack infrastructure. Cluster API with metal-stack is a declarative cluster management framework — you declare the desired state of your clusters and the Cluster API controllers reconcile the actual state. + +This guide does **not** use Ansible roles. Unlike Gardener, which is deployed via the `gardener-*` metal-roles, Cluster API with metal-stack is set up directly through `clusterctl` and the [cluster-api-provider-metal-stack (CAPMS)](https://github.com/metal-stack/cluster-api-provider-metal-stack/) infrastructure provider. + +:::warning[Beta] +Cluster API with metal-stack is in beta and not yet recommended for production workloads. Please use [Gardener](./05_gardener.md) for production deployments. We are actively looking for exchange and adopters — if you are interested in using Cluster API with metal-stack, please [join our community](/community) to help shape future integration efforts. +::: + +## Prerequisites + +Before deploying Cluster API, ensure the following infrastructure is in place: + +- **Management cluster** — A Kubernetes cluster to host the Cluster API controllers and the desired cluster state. This can be any Kubernetes distribution (kind, k3s, a Gardener Shoot, or any managed Kubernetes). It must have network access to your metal-stack infrastructure. +- **metal-stack infrastructure** — A running metal-stack installation with at least one partition with available machines and operating system images (see [metal-images](https://github.com/metal-stack/metal-images) for pre-built ones). +- **CLI tools** — `metalctl` for communicating with the metal-stack API ([installation](https://github.com/metal-stack/metalctl)) and `clusterctl` for initializing providers and generating cluster manifests. +- **External dependencies** — DNS, NTP, ACME (optional), S3-compatible storage for backups, and Git-Hosting with CI/CD for GitOps-driven deployment. + +## Deployment + +Cluster API with metal-stack is deployed through the [cluster-api-provider-metal-stack (CAPMS)](https://github.com/metal-stack/cluster-api-provider-metal-stack/) infrastructure provider. The [CAPMS reference documentation](../../08-References/Kubernetes/cluster-api-provider-metal-stack/cluster-api-provider-metal-stack.md) covers the deployment in detail. + +**Deployment flow** + +1. **Prepare management cluster** — A Kubernetes cluster to host CAPI and CAPMS providers and cluster state +2. **Install CAPMS** — Deploy the CAPMS provider into the management cluster +3. **Configure `clusterctl`** — Register the metal-stack provider URL and set environment variables (API credentials, project, partition, machine images and sizes, cluster name, Kubernetes version) +4. **Allocate resources** — Create node networks, firewalls, and control plane IPs via `metalctl` +5. **Generate and apply cluster manifest** — Use `clusterctl generate cluster` to produce a YAML with `Cluster`, `MetalStackCluster`, `KubeadmControlPlane`, `MachineDeployment`, and `MetalStackMachine` resources, then apply it +6. **Deploy add-ons** — Install CNI (Calico) and `metal-ccm` via `ClusterResourceSet` and CAAPH +7. **Retrieve kubeconfig** — Access the provisioned cluster + +**Network integration** + +Network integration for Cluster API is currently more manual compared to Gardener. Node networks must be created manually via `metalctl` and provided as environment variables. IP addresses for the control plane also need to be allocated in advance through `metalctl`. Firewall rules are currently static and can be applied to firewall nodes; no automatic firewall controller is in place yet. Automatic network resource allocation is on the roadmap for CAPMS. + +For service exposure, CAPMS uses KubeVIP in BGP mode to allocate and announce public IPs, similar to the MetalLB-based approach in Gardener. + +**Air-gapped environments** + +For air-gapped deployments, follow the [Cluster API Operator air-gapped environment guide](https://cluster-api-operator.sigs.k8s.io/topics/configuration/air-gapped-environtment). All required images must be mirrored to an OCI registry reachable from the management cluster. + +## Kamaji as Control Plane Provider + +[Kamaji](https://kamaji.clastix.io/) is a Control Plane Manager for Kubernetes that runs control planes as pods within a management cluster, reducing operational overhead and costs. It integrates with Cluster API as a `ControlPlaneProvider`. + +:::warning +Kamaji integrations with metal-stack have not been evaluated in production-grade scenarios. This is a lab-only showcase. +::: + +Kamaji acts as a `ControlPlaneProvider` with Cluster API, while CAPMS acts as the `InfrastructureProvider`. This setup manages **tenant clusters** on metal-stack infrastructure, combining Kamaji's control plane management with metal-stack's bare-metal provisioning. + +**Deployment** + +1. **Prepare management cluster** — A Kubernetes cluster to host Kamaji and CAPMS providers +2. **Install Kamaji and CAPMS** — Deploy both providers into the management cluster +3. **Create a control plane VIP** — MetalLB assigns a virtual IP for the tenant API server +4. **Generate and apply tenant cluster manifest** — Use `clusterctl generate cluster` to produce a YAML with `Cluster`, `MetalStackCluster`, `KubeadmControlPlane`, `MachineDeployment`, and `MetalStackMachine` resources, then apply it +5. **Deploy add-ons** — Install CNI (Calico) and `metal-ccm` into the tenant cluster + +A working showcase is available in the [`capi-lab`](https://github.com/metal-stack/cluster-api-provider-metal-stack/blob/main/DEVELOPMENT.md#running-the-kamaji-flavor) setup, which extends the `mini-lab` with a Kamaji flavor. See our [blog post](/blog/2026/04-kamaji) for a detailed walkthrough of the architecture and setup. + +**Fleet management and GitOps** + +Since Kamaji with metal-stack uses Cluster API under the hood, fleet management follows the same pattern as Cluster API. Tenant cluster manifests are generated via `clusterctl`, stored in Git, and deployed through your CI/CD pipeline. + +## Fleet Management and GitOps + +You must set up your own Git repository and GitOps operator to manage cluster deployments. + +**What you need to build:** + +1. **Git repository** — Store cluster manifests generated via `clusterctl generate cluster `. Each cluster gets its own set of YAML files containing `Cluster`, `MetalStackCluster`, `KubeadmControlPlane`, `MachineDeployment`, and `MetalStackMachine` resources. +2. **GitOps operator** — Deploy ArgoCD or FluxCD to watch your Git repository and apply manifests to the management cluster, ensuring drift-free declarative delivery. +3. **Per-cluster CI/CD** — Essential components (CNI, CCM) are rolled out on a per-cluster basis. Changes to `MachineTemplate` or `ClusterResourceSet` are staged through the Git repository with standard approval processes. + +**Platform capabilities:** + +- **Cluster migration** — `clusterctl move` enables moving workload cluster resources between management clusters, pausing controllers during the move to prevent worker node loss +- **Emergency patching** — Achieved through editing resources in the management cluster, e.g., update of machine OS image in the `MachineTemplate` or update of `ClusterResourceSet`. Unlike Gardener, this change is not rolled out fleet-wide automatically and should be staged through the Git repository with standard approval processes +- **Certificate rotation** — No direct workflow is described for certificate rotation at the landscape level; this is to be defined by platform administrators using manual/custom processes based on standard tooling (e.g., `kubeadm` certificate renewal) +- **Audit configuration** — Audit configuration can be passed to the kube-apiserver via the `kubeadmConfigSpec` of the `KubeadmControlPlane` resource before cluster creation. Each cluster can have its own audit policy. The management cluster's kube-apiserver audit also needs to be configured separately. Cluster API does not provide a centralized audit management toolset. + +## Available Flavors + +CAPMS provides different [cluster template flavors](https://cluster-api.sigs.k8s.io/cluster-api/commands/generate-cluster.html#flavors) for `clusterctl generate cluster`: + +| Flavor | Description | K8s Compatibility | +|--------|-------------|-------------------| +| *(default)* | Expects the user to deploy a CNI and a CCM manually | >= v1.33 | +| `calico` | Installs Calico CNI + metal-ccm via `ClusterResourceSet` and CAAPH | >= v1.33 | +| `pre-v1.33` | Same as default but for Kubernetes versions < v1.33 | < v1.33 | +| `kamaji-tenant` | Kamaji tenant cluster template (requires Kamaji installed) | >= v1.33 | + +## Next Steps + +- **[KCLM Overview](./05_kclm.md)** — Introduction to Kubernetes Cluster Lifecycle Management with metal-stack +- **[Gardener Deployment Guide](./05_gardener.md)** — metal-stack's recommended, production-ready KCLM solution +- **[Cluster API Concepts](../../05-Concepts/04-Kubernetes/03-cluster-api.md)** — Architecture, operational model, and control plane hosting +- **[CAPMS Reference](../../08-References/Kubernetes/cluster-api-provider-metal-stack/cluster-api-provider-metal-stack.md)** — Full CAPMS documentation +- **[CAPMS Development Guide](../../08-References/Kubernetes/cluster-api-provider-metal-stack/DEVELOPMENT.md)** — Local development with capi-lab +- **[Kamaji Blog Post](/blog/2026/04-kamaji)** — Architecture and setup walkthrough + + diff --git a/docs/04-For Operators/03-Deployment/06-gpu-workers.md b/docs/04-For Operators/03-Deployment/07-gpu-workers.md similarity index 99% rename from docs/04-For Operators/03-Deployment/06-gpu-workers.md rename to docs/04-For Operators/03-Deployment/07-gpu-workers.md index 3a6938ce..2658274f 100644 --- a/docs/04-For Operators/03-Deployment/06-gpu-workers.md +++ b/docs/04-For Operators/03-Deployment/07-gpu-workers.md @@ -1,7 +1,7 @@ --- slug: /deployment/gpu-workers title: GPU Workers -sidebar_position: 6 +sidebar_position: 7 --- # GPU Workers diff --git a/docs/04-For Operators/03-Deployment/07_offline-resilience.md b/docs/04-For Operators/03-Deployment/08_offline-resilience.md similarity index 99% rename from docs/04-For Operators/03-Deployment/07_offline-resilience.md rename to docs/04-For Operators/03-Deployment/08_offline-resilience.md index 6dcef5ab..40b5a440 100644 --- a/docs/04-For Operators/03-Deployment/07_offline-resilience.md +++ b/docs/04-For Operators/03-Deployment/08_offline-resilience.md @@ -1,7 +1,7 @@ --- slug: /deployment/offline-resilience title: Offline Resilience -sidebar_position: 7 +sidebar_position: 8 --- # Offline Resilience diff --git a/docs/05-Concepts/04-Kubernetes/01-gardener.md b/docs/05-Concepts/04-Kubernetes/01-gardener.md deleted file mode 100644 index 535eb82b..00000000 --- a/docs/05-Concepts/04-Kubernetes/01-gardener.md +++ /dev/null @@ -1,78 +0,0 @@ ---- -slug: /gardener -title: Gardener -sidebar_position: 1 ---- - -# Gardener - -[Gardener](https://gardener.cloud/) is an open source project for orchestrated Kubernetes cluster provisioning. It supports many different cloud providers, metal-stack being one of them. Using the Gardener project, metal-stack can act as a machine provider for Kubernetes worker nodes. - -The idea behind the Gardener project is to start with a dedicated set of Kubernetes clusters (this can be a single cluster, too), which are used to host Kubernetes control planes for new Kubernetes clusters. The new Kubernetes control planes reside in dedicated namespaces of the initial clusters ("Kubernetes in Kubernetes" or "underlay / overlay Kubernetes"). For suggestions on how to set up the initial cluster, see the [Bootstrap Infrastructure](../../04-For%20Operators/03-Deployment/02_bootstrap-infrastructure.md) section in the Deployment Guide. - -Gardener's architecture is designed for multi-tenant environments, with a strong distinction between the operator and the end users. In Gardener, Kubernetes control planes for different tenants may reside in the same operator cluster. This approach makes it very suitable for being used with bare metal because it allows taking full advantage of the server resources. Another implication is that end users do not have access to their control plane components, such as the kube-apiserver or the ETCD. These are managed by the operator and in case of metal-stack even physically divided from the end user's workload. - -Gardener allocates machines from a cloud provider and automatically deploys a kubelet to those nodes, which then joins the appropriate control plane. Operators can also nest clusters so that newly provisioned clusters can be used to spin up more clusters, leading to nearly infinite scalability (also known as "kubeception" model). - -## Terminology - -We would like to explain the most important Gardener terms. The terminology used in the Gardener project has many similarities to the architecture of Kubernetes. Additional information can also be found in the [official glossary](https://github.com/gardener/documentation/blob/master/website/documentation/glossary/_index.md). - -### Garden Cluster - -The Garden Cluster is a Kubernetes cluster that runs the Gardener Control Plane. - -The control plane components introduce dedicated Kubernetes API resources for provisioning new Kubernetes clusters with the Gardener. It also takes care of the validation for many of those Gardener API resources and also reconciling some of them. The components are the following: - -- Gardener API Server -- Gardener Controller Manager -- Gardener Scheduler -- Gardener Admission Controller - -The control plane components can be deployed in the Garden Cluster through the Gardener Operator. - -The Garden cluster can also be used as [seed](#seeds-and-soils) cluster. - -### Virtual Garden - -A recommended way to deploy the Gardener is running a "virtual cluster" inside the Garden cluster. It is basically a Kubernetes control plane without any worker nodes, providing the Kubernetes API in an own ETCD. Its purpose is to store all Gardener resources (such that they reside inside a dedicated ETCD) and provide an individual update lifecycle from the Garden Cluster. End users can have access to own project namespaces in the virtual garden, too. - -The virtual garden consists of the following components: - -- garden kube-apiserver -- etcd -- kube-controller-manager - -More details about the virtual garden can be found in the description of [`gardener-operator`](https://github.com/gardener/gardener/blob/master/docs/concepts/operator.md). - -### Seeds and Soils - -A seed cluster is a cluster in which an agent component called the `Gardenlet` is running. The gardenlet is connected to the Gardener Control Plane and is responsible for orchestrating the provisioning of new clusters inside the seed cluster. The control plane components for the new clusters run as pods in the seed cluster. - -A seed cluster can also be called a soil if the Gardenlet has been manually deployed by the operator and not by the Gardener. Clusters created on the soil can be turned into seed clusters by the operator using a Gardener resource called `ManagedSeed`. This resource causes Gardener to automatically deploy the Gardenlet to the new cluster, such that the resulting cluster is not called a soil. - -### Shoot - -Every Kubernetes cluster that is fully provisioned and managed by Gardener is called a `Shoot` cluster. It consists of the shoot control plane running on the seed cluster and worker nodes running the actual workload. - -## Gardener Integration Components - -During the provisioning flow of a cluster, Gardener emits resources that are expected to be reconciled by controllers of a cloud provider. This section briefly describes the controllers implemented by metal-stack to allow the creation of a Kubernetes cluster on metal-stack infrastructure. - -If you want to learn how to deploy metal-stack with Gardener, please check out the corresponding [deployment-guide section](../../04-For%20Operators/03-Deployment/05_kclm.md). - -### gardener-extension-provider-metal - -The [gardener-extension-provider-metal](https://github.com/metal-stack/gardener-extension-provider-metal) contains of a set of webhooks and controllers for reconciling cloud provider specific resources of `type: Metal`, which created by Gardener during the cluster provisioning flow. - -Primarily, its purpose is to reconcile `Infrastructure`, `ControlPlane`, and `Worker` resources. - -The project also introduces an own API (`ProviderConfiguration` resources) and consists of an admission-controller to validate them. This admission controller should be deployed in the Gardener control plane cluster. - -### os-metal-extension - -Due to the reason metal-stack initially used ignition to provision operating system images (today, cloud-init is supported as well) there is an implementation of a controller that translates the generic `OperatingSystemConfig` format of Gardener into ignition userdata. It can be found on Github in the [os-metal-extension](https://github.com/metal-stack/os-metal-extension) repository. - -### machine-controller-manager-provider-metal - -Worker nodes are managed through Gardener's [machine-controller-manager](https://github.com/gardener/machine-controller-manager) (MCM). The MCM allows out-of-tree provider implementation via sidecar, which is what we implemented in the [machine-controller-manager-provider-metal](https://github.com/metal-stack/machine-controller-manager-provider-metal) repository. diff --git a/docs/05-Concepts/04-Kubernetes/01-kclm.md b/docs/05-Concepts/04-Kubernetes/01-kclm.md new file mode 100644 index 00000000..df0d8ae5 --- /dev/null +++ b/docs/05-Concepts/04-Kubernetes/01-kclm.md @@ -0,0 +1,379 @@ +--- +slug: /kubernetes +title: Kubernetes Cluster Lifecycle Management +sidebar_position: 1 +--- + +# Kubernetes Cluster Lifecycle Management + +Kubernetes Cluster Lifecycle Management (KCLM) is the foundation of metal-stack's **Kubernetes as a Service** solution — enabling organizations to provision, operate, and decommission production-grade Kubernetes clusters on bare metal with the same ease and reliability as hyperscaler managed services, while retaining full control over the underlying infrastructure. + +## Why Kubernetes Cluster Lifecycle Management? + +Running Kubernetes on bare metal delivers unmatched performance, cost efficiency, and compliance advantages — but managing clusters at scale introduces significant operational complexity. Without automated lifecycle management, organizations face: + +- **Manual provisioning** that is slow, error-prone, and inconsistent +- **Drift and configuration divergence** across clusters ("snowflake clusters") +- **Limited scalability** — human operators cannot manage hundreds or thousands of clusters +- **Compliance risks** in regulated environments where audit trails, isolation, and controlled change processes are mandatory +- **Operational burden** on platform teams who must manage Kubernetes internals rather than enabling developer self-service + +KCLM solves these challenges by automating the entire cluster lifecycle — from design and bootstrap through scaling, upgrades, and decommissioning — while enforcing consistency, reproducibility, and separation of duties across multi-tenant environments. + +## What KCLM Automates + +metal-stack's KCLM solution covers the complete cluster lifecycle: + +```mermaid +flowchart LR + subgraph "Cluster Lifecycle" + A["Design & Template\nVersion-controlled YAML specs"] --> B["Bootstrap & Configure\nControl planes, workers, CNI, CSI"] + B --> C["Scale\nManual & auto-scaling of worker groups"] + C --> D["Upgrade\nRolling K8s version & patch updates"] + D --> E["Operate\nHealth checks, node replacement"] + E --> F["Hibernate\nScale down for non-24/7 clusters"] + F --> G["Decommission\nGraceful deletion with cleanup"] + end + + style A fill:#e1f5fe + style B fill:#e1f5fe + style C fill:#fff3e0 + style D fill:#fff3e0 + style E fill:#e8f5e9 + style F fill:#f3e5f5 + style G fill:#ffebee +``` + +| Phase | Capabilities | +| ----------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| **Design & Template** | Version-controlled cluster specifications constrained by administrator-defined profiles (allowed regions, machine types, Kubernetes versions) | +| **Bootstrap & Configuration** | Automated provisioning of control planes, worker nodes, CNI, CSI, and cloud controller — all reconciled to desired state | +| **Scaling** | Manual and automatic scaling of worker groups with configurable strategies (max surge, max unavailable, drain timeouts) | +| **Upgrades** | Rolling Kubernetes version upgrades, in-place Kubelet patch updates, and machine image updates within configurable maintenance windows (Gardener); spec-driven rolling updates (Cluster API) | +| **Node Replacement** | Automatic health-based remediation with configurable timeouts; workload shifting to replacement nodes | +| **Decommissioning** | Graceful cluster deletion with finalizer-based resource cleanup to prevent orphaned infrastructure | +| **Migration & Restore** | Control plane migration across failure domains; etcd backup and automatic recovery (Gardener); clusterctl-based cluster migration between management clusters (Cluster API) | +| **Hibernation** | Infrastructure scale-down for non-24/7 clusters to minimize resource waste (Gardener only) | + +## Kubernetes as a Service + +KCLM transforms bare metal infrastructure into a **self-service Kubernetes platform** — but the degree of self-service varies significantly between the two approaches: + +- **Gardener** provides a full self-service API where end-users (developers, team leads, project owners) can create clusters, manage node worker groups, configure maintenance windows, and enable/disable auto-upgrades — all limited only to whitelisted machine types and regions defined by platform administrators. Access is via standard `kubectl` with OIDC-based authorization and project-scoped permissions. +- **Cluster API** does not provide end-user self-service natively. Administrators manage the management cluster and provision clusters through GitOps workflows. If self-service is desired, a custom API layer must be built on top of Cluster API. + +Platform administrators, meanwhile, focus on **providing the platform** — managing seed clusters (Gardener) or the management cluster (Cluster API), whitelisting machine types, delivering Kubernetes version updates, and ensuring fleet-wide consistency through GitOps-driven processes. This separation of responsibilities mirrors how hyperscalers operate: administrators manage the infrastructure and platform components; end-users consume the Kubernetes API and focus on their workloads. + +## Two Approaches, One Infrastructure + +metal-stack provides KCLM through two integration paths, both consuming the same metal-stack API for bare-metal node provisioning: + +| Aspect | [Gardener](./02-gardener.md) | [Cluster API](./03-cluster-api.md) | +| ------------------------------- | ---------------------------------------------------------------------- | --------------------------------------------------------------------------- | +| **Status** | Recommended, production-ready | Beta, under active development | +| **Governance** | NeoNephos Foundation | CNCF (Kubernetes SIG) | +| **Experience** | 7+ years in financial-sector production | CNCF project, metal-stack integration in development | +| **Day-2 capabilities** | Native (DNS, backup, audit, certificate rotation, maintenance windows) | Assembled through GitOps and add-on providers | +| **Control plane hosting** | Dedicated namespaces on Seed clusters (physically isolated) | On worker nodes (CABPK) or dedicated (Kamaji, unevaluated) | +| **Operational model** | End-user self-service via Virtual Garden API | Administrator-managed GitOps workflows | +| **Complexity** | More opinionated, higher initial setup | Less opinionated, lower initial setup but higher ongoing administration | +| **Maintenance windows** | Built-in, per-cluster configurable | Continuous reconciliation, no built-in windows | +| **Control plane resource cost** | Shared on Seed clusters (efficient) | 3 dedicated nodes per cluster (wasteful) or shared via Kamaji (unevaluated) | +| **Use case** | Production fleets, regulated environments, multi-tenant platforms | Simple deployments, teams willing to build day-2 operations | + +Both converge on vanilla Kubernetes and metal-stack infrastructure, ensuring replaceability and vendor independence. + +## Platform vs Framework: What Each Solution Does For You + +The fundamental difference between Gardener and Cluster API is philosophical: **Gardener is a complete platform** (like AWS EKS or GKE), while **Cluster API is a framework** (like building your own managed Kubernetes service). This distinction drives everything else — administration burden, resource efficiency, day-2 capabilities, and operational complexity. + +### Gardener: The Complete Platform + +Gardener delivers a **turnkey Kubernetes as a Service platform** out of the box. When you deploy Gardener, you get: + +- **Built-in day-2 operations**: DNS management, etcd backup & restore, certificate rotation, audit logging, access control lists, hibernation, shoot migration, VPN — all native, no assembly required +- **Multi-tenant self-service**: End-users create and manage clusters through a Kubernetes-native API with project isolation, quotas, and role-based access control +- **Maintenance windows**: Per-cluster configurable windows for automatic updates (Kubernetes versions, machine images) — no manual coordination +- **Version skew enforcement**: Static validation rules in the API server that cannot be circumvented, preventing incompatible Kubernetes version combinations +- **Physical isolation**: Each Shoot's control plane runs in a dedicated namespace on a Seed cluster, physically separated from end-user workloads — critical for compliance requirements +- **Fleet-wide consistency**: All components (CNI, CSI, CCM, DNS, backup) are rolled out fleet-wide with every platform release, integration-tested across the compatibility matrix + +**What you build and maintain**: A Garden cluster (Gardener control plane), Seed clusters (one per data center site), and the Virtual Garden. This is significant infrastructure — typically 3-5 nodes for the Garden cluster and one Seed cluster per site. But once running, day-2 operations are largely automated. + +**Administration burden**: High initial setup (deploying Garden, Seeds, extensions), but **low ongoing administration**. Once the platform is running, cluster provisioning, upgrades, backups, certificate rotation, and decommissioning are all automated. Platform administrators focus on platform updates and extension management, not individual cluster operations. + +**Resource efficiency**: Control planes run as pods on Seed clusters — multiple Shoot control planes share Seed infrastructure. This is efficient for fleets but means control planes are not physically isolated from other workloads on the same Seed (though they are in dedicated namespaces). + +### Cluster API + CABPK: The Bare Framework + +Cluster API with the Kubeadm Bootstrap Provider (CABPK) is a **cluster provisioning framework** — it creates Kubernetes clusters but leaves everything else to you. + +**What you get out of the box**: + +- Cluster creation, scaling, and deletion via declarative manifests +- Machine provisioning on metal-stack infrastructure +- Add-on installation via ClusterResourceSet + CAAPH (Helm) + +**What you must build yourself**: + +- DNS management (no built-in DNS service) +- etcd backup & restore (no built-in operator) +- Certificate rotation (manual processes or custom tooling) +- Audit logging (configure via kubeadmConfigSpec, no centralized management) +- Maintenance windows (none — CAPI reconciles continuously) +- Version skew enforcement (not strictly enforced — risk management is your responsibility) +- Hibernation (no built-in capability) +- Cluster migration (no built-in capability) +- Multi-tenant self-service (no built-in project isolation, quotas, or user API) +- Access control lists (no built-in firewall controller) + +**Resource waste**: With CABPK, each cluster's control plane runs on dedicated worker nodes. For high availability, you need **3 control plane nodes per cluster** — nodes that exist solely to run kube-apiserver, etcd, controller-manager, and scheduler. These nodes cannot run user workloads. For a fleet of 10 clusters, that's 30 nodes wasted on control planes alone. This is a significant cost multiplier. + +**Administration burden**: Low initial setup (just the management cluster), but **very high ongoing administration**. Every day-2 task — certificate rotation, backup management, DNS configuration, audit policy management, version upgrades — must be built, tested, and maintained by your team. Each cluster upgrade requires manual coordination. There are no maintenance windows, no version skew enforcement, and no automated failure recovery beyond basic MachineHealthCheck. + +**Philosophy**: CAPI+CABPK is for teams that want maximum flexibility and are willing to invest significant engineering effort to build their own managed Kubernetes platform. It's the difference between buying AWS EKS and building your own EKS on EC2. + +### Cluster API + Kamaji: The Middle Ground + +Kamaji acts as a Control Plane Manager for Cluster API, running tenant control planes as pods within the management cluster rather than on dedicated worker nodes. + +**What Kamaji adds over CABPK**: + +- **Resource efficiency**: Multiple tenant control planes share management cluster infrastructure — no dedicated control plane nodes per cluster. A single management cluster can host dozens of tenant control planes +- **Physical isolation**: Control planes run in dedicated namespaces, separate from worker node workloads +- **Multi-tenant capability**: Built-in support for multiple tenant clusters on shared infrastructure + +**What you still must build yourself** (same as CABPK): + +- DNS, backup, certificate rotation, audit logging, maintenance windows, version skew enforcement, hibernation, cluster migration, multi-tenant self-service, ACLs + +**Resource efficiency**: Dramatically better than CABPK. Instead of 3 dedicated nodes per cluster for control planes, you share management cluster capacity across all tenants. For a fleet of 10 clusters, you might need only 3-5 management cluster nodes total. + +**Administration burden**: Still high ongoing administration — Kamaji solves the control plane hosting problem but not the day-2 operations problem. You still assemble and maintain all day-2 tooling through GitOps workflows. + +**Status**: Kamaji with metal-stack has not been evaluated in production-grade scenarios. It is a promising approach for resource efficiency but carries higher risk for production workloads. + +### Decision Matrix + +| Aspect | Gardener | CAPI + CABPK | CAPI + Kamaji | +| ----------------------------------- | ----------------------------------------------------------------- | ---------------------------------------------------------- | --------------------------------------------------------- | +| **Philosophy** | Complete platform (buy) | Framework (build) | Framework + shared control planes | +| **Day-2 operations** | All built-in, automated | Build everything yourself | Build everything yourself | +| **Control plane nodes per cluster** | Shared on Seeds (efficient) | 3 dedicated nodes (wasteful) | Shared on management cluster (efficient) | +| **Resource waste for 10 clusters** | Minimal | ~30 nodes wasted | Minimal | +| **Multi-tenant self-service** | Built-in (projects, quotas, API) | Build yourself | Build yourself | +| **Maintenance windows** | Built-in, per-cluster | None | None | +| **Version skew enforcement** | Enforced by API server | Not enforced | Not enforced | +| **Certificate rotation** | Automated (8-hour windows) | Manual/custom | Manual/custom | +| **etcd backup & restore** | Built-in (etcd-druid) | Build yourself | Build yourself | +| **Hibernation** | Built-in | Not available | Not available | +| **Shoot/Cluster migration** | Built-in | Not available | Not available | +| **Initial setup complexity** | High (Garden + Seeds) | Low (management cluster) | Low (management cluster) | +| **Ongoing administration** | Low (platform updates) | Very high (build & maintain everything) | Very high (build & maintain everything) | +| **Production readiness** | 7+ years, 10,000+ clusters | Beta, metal-stack integration in development | Not evaluated with metal-stack | +| **Best for** | Production fleets, regulated environments, multi-tenant platforms | Simple deployments, teams with strong Kubernetes expertise | Resource-constrained fleets, teams willing to accept risk | + +:::tip[Recommendation] +For production workloads, regulated environments, or any scenario where you need multi-tenant self-service with minimal ongoing administration, **Gardener is the clear choice**. The platform approach eliminates the need to build and maintain day-2 operations tooling, enforces version skew policies, and provides physical isolation between control planes and workloads. + +Cluster API with CABPK is only recommended for simple, single-cluster deployments where you accept the resource waste of dedicated control plane nodes and the ongoing administration burden of building your own day-2 operations. Cluster API with Kamaji offers better resource efficiency but carries higher risk as an unevaluated integration. +::: + +## Core Concepts + +These concepts apply to both Gardener and Cluster-API approaches. + +```mermaid +flowchart TD + subgraph "Git Repository" + G["Version-controlled YAML manifests\n(Cluster specs, CloudProfile, extensions)"] + end + + subgraph "CI/CD Pipeline" + P["Peer review → Audit trail → Apply"] + end + + subgraph "KCLM Controllers" + R["Reconciliation Loop\nCompare desired vs actual state"] + D["Drift Detection\nAutomatic remediation"] + end + + subgraph "Infrastructure" + K["Kubernetes Clusters\nVanilla K8s on bare metal"] + M["metal-stack API\nMachine, network, firewall allocation"] + end + + G --> P + P --> R + R --> D + D --> K + K -.->|provisions via| M + M -->|reports status| R + + style G fill:#e3f2fd + style P fill:#fff3e0 + style R fill:#e8f5e9 + style D fill:#fce4ec + style K fill:#f3e5f5 + style M fill:#e0f2f1 +``` + +### Domain Abstraction + +KCLM separates lifecycle management concerns from adjacent domains through well-defined contracts and extension points. Bare-metal provisioning, networking, storage, and Kubernetes distribution are integrated via provider extensions that implement generic interfaces — allowing each domain to evolve independently without requiring changes to the lifecycle orchestrator. This contract-based approach ensures that KCLM components do not depend on any specific infrastructure provider, and multiple domains can run in parallel. + +### Declarative State & GitOps + +The intended cluster state is defined declaratively in version-controlled YAML manifests stored in Git repositories, serving as the single source of truth. Changes are applied by updating specifications in the repository, with CI/CD pipelines enforcing peer review, audit trails, and rollback capability. This GitOps-driven workflow prevents configuration drift and ensures every cluster is reproducible from its manifest. + +### Reconciliation & Drift Detection + +KCLM follows an orchestration-driven model where controllers run continuous reconciliation loops, comparing the current state against the desired state defined in version-controlled specs. Drift is detected at an early stage and remediated automatically without manual intervention. In emergency situations, operators can pause reconciliation to prevent unintended changes. When controllers cannot self-heal, monitoring integrations alert operators. + +```mermaid +flowchart LR + subgraph "Desired State" + GIT["Git repository\nYAML manifests"] + SPEC["Kubernetes CRDs\nCluster specs, profiles"] + end + + subgraph "Reconciliation Loop" + WATCH["Controllers watch\nKubernetes API"] + COMPARE["Compare desired vs\nactual state"] + REMEDIATE["Remediate drift\nCreate/Update/Delete"] + end + + subgraph "Actual State" + K8S["Kubernetes clusters\nRunning workloads"] + INFRA["Bare metal machines\nNetworks, firewalls"] + end + + GIT -->|apply| SPEC + SPEC --> WATCH + WATCH --> COMPARE + COMPARE -->|drift detected| REMEDIATE + REMEDIATE --> K8S + REMEDIATE --> INFRA + K8S -->|API reports| WATCH + INFRA -->|API reports| WATCH + + PAUSE["⏸ Pause reconciliation\nEmergency override"] -.->|can interrupt| COMPARE + + style GIT fill:#e3f2fd + style SPEC fill:#e3f2fd + style WATCH fill:#c8e6c9 + style COMPARE fill:#fff9c4 + style REMEDIATE fill:#f8bbd0 + style K8S fill:#f3e5f5 + style INFRA fill:#e0f2f1 + style PAUSE fill:#ffebee +``` + +### Replaceability & Vendor Independence + +KCLM produces clusters built on **vanilla Kubernetes** — no forks or patches. As long as a replacement lifecycle management tool supports vanilla Kubernetes, migration requires no action on the cluster side. The contract-based integration model means that replacing the KCLM orchestrator or the infrastructure provider does not force changes to the Kubernetes distribution or the workloads running on it. + +## High Availability & Failure Domains + +KCLM supports multiple control plane topologies for on-prem failure domains. The available options differ between Gardener and Cluster API: + +| Topology | Gardener | Cluster API | +| ----------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| **Single-site HA** | Multiple control plane nodes across machines in the same Seed with etcd spread for quorum. Default production-confirmed choice. | Multiple control plane Machines within a single partition with etcd replicas on separate Machines. Natively supported. | +| **Multi-rack** | Shoot control plane nodes spread across multiple racks within one Seed with etcd spread across racks. Rack-level failure isolation via MachineDeployment topology spread constraints. | Multi-failure-domain topology distributes control plane Machines across multiple zones or regions with etcd spread accordingly. Natively supported through ClusterClass topology definitions. | +| **Multi-site** | Shoot control planes replicated across Seeds corresponding to different sites or data centers. MachineDeployments use zone constraints to distribute workers across regions. Higher latency for cross-seed communication requires multi-seed configuration. | Control plane Machines deployed across multiple CAPI management clusters or across widely separated MetalPools with cross-site etcd replication. Requires additional operator effort for cross-site networking. | +| **Dedicated isolation** | A Shoot gets its own dedicated Seed cluster with no shared control plane with other tenants. Highest compliance level for critical infrastructure at highest resource cost. | A dedicated Cluster with its own isolated MetalPool and exclusive use of MetalPools. Same isolation level as Gardener's dedicated Seed for critical infrastructure. | + +Worker nodes are automatically distributed across racks using a rack-spreading algorithm, and well-known Kubernetes topology labels (`machine.metal-stack.io/rack`, `machine.metal-stack.io/chassis`, `topology.kubernetes.io/region`, `topology.kubernetes.io/zone`) are provided on nodes — enabling end-users to configure Pod topology spread and anti-affinity rules. With MEP-19 (metal-stack Enhancement Proposal 19), routing across data center partitions will also be supported, allowing worker nodes to reside in separate metal-stack partitions while maintaining a single Kubernetes cluster — provided the partitions are geographically close enough for stable low-latency connectivity. + +Failure scenarios are engineered for automation: + +- **Worker failures** — Automatic health-based replacement within configurable health timeouts (default monitored by node-problem-detector in Gardener; MachineHealthCheck in Cluster API) +- **HA control plane** — Orchestrated three-node etcd clusters spread across the cluster topology with pod disruption budgets and topology spread constraints +- **etcd backups** — Scheduled full and incremental snapshots (typically three-minute deltas) enable automatic recovery from data loss +- **Accidental deletion protection** — Configurable backup retention allows emergency access to cluster resources before cleanup +- **BGP Anycast** — Spreads traffic across partitions and automatically routes around unreachable nodes within seconds + +:::note[Management plane availability] +The KCLM management layer is designed so that its absence does not impact cluster availability. Workloads continue running and end-users retain Kubernetes API access even when the management plane is unavailable. Outages of the Gardener cluster or the management cluster only cause cluster provisioning to become unavailable. +::: + +## Upgrades, Rollback, and Change Management + +**Upgrade strategies** — For minor Kubernetes version upgrades, Gardener rolls worker groups according to a configurable rollout strategy (max surge, max unavailable, drain timeouts). For patch updates, Gardener applies in-place Kubelet upgrades within a jittered 5-minute window, preventing vanishing of route announcements for more than one node at a time. Cluster API orchestrates a one-by-one worker node roll triggered by spec updates. + +**Blue-green updates** — End-users can achieve zero-downtime upgrades through two approaches: (1) using multiple clusters with BGP Anycast to spread workloads across clusters, or (2) using worker groups or machine pools with different Kubernetes kubelet versions and OS versions, combined with Kubernetes node taints and tolerations for traffic routing. + +**Rollback** — Kubernetes versions are not allowed to be rolled back. End-users are required to test Kubernetes upgrades in a staging cluster first. Tools exist to test for deprecated APIs before running the actual upgrade. + +**Downtime expectations** — With Gardener's HA control plane feature enabled, no downtime is expected during regular version or Kubernetes upgrades. In-place Kubelet upgrades happen within a jittered 5-minute window. MetalLB follows a one-by-one rolling update strategy so that route announcements only occur node-by-node until the speaker daemon set pod reports readiness again. Cluster API triggers an orchestrated worker roll, updating control planes and kubelets one-by-one. + +## Audit & Traceability + +Kubernetes API audit policies are configurable per cluster, with logs forwarded to external sinks (e.g., Splunk, S3) for auditable change tracking: + +- **Gardener** — The `gardener-extension-audit` extension allows shoot owners or operators to configure buffered forwarders to audit sinks. Audit policies for the kube-apiserver are configured via standard Kubernetes Policy manifests, with each cluster having its own set of policies. The same extension can be configured for the Garden cluster's apiserver. +- **Cluster API** — Audit configuration can be passed to the kube-apiserver via the `kubeadmConfigSpec` of the `KubeadmControlPlane` resource before cluster creation. The management cluster's kube-apiserver audit also needs to be configured separately. Cluster API does not provide a centralized audit management toolset and relies on cloud-native standards to be set up by the operator. + +## Day-2 Operations, HA, and Advanced Topics + +The following topics are covered in detail on the individual solution pages, as their implementation differs significantly between Gardener and Cluster API: + +- **[Day-2 Operations](./02-gardener.md#operational-features)** — Configuration changes, certificate rotation, component upgrades, emergency patching, and audit & traceability (Gardener: built-in; Cluster API: assembled through GitOps) +- **[High Availability & Failure Domains](./02-gardener.md#failure-domains)** — Control plane topologies (single-site HA, multi-rack, multi-site, dedicated isolation), worker node distribution, rack-spreading, and failure recovery automation +- **[Add-on Lifecycle Management](./02-gardener.md#operational-features)** — CNI, CSI, CCM, DNS, and audit component management with compatibility matrices and fleet-wide consistency (Gardener: fleet-wide; Cluster API: per-cluster via ClusterResourceSet) +- **[Version Policy & Safe Transitions](./02-gardener.md#version-skew-policy)** — Kubernetes version skew enforcement, staged upgrades, and emergency patching workflows (Gardener: enforced by API server; Cluster API: risk management via approval gates) +- **[Blue-Green Update Strategy](./01-kclm.md#what-kclm-automates)** — Zero-downtime upgrade patterns using BGP Anycast or worker groups with taints and tolerations +- **[Cluster API Deep Dive](./03-cluster-api.md)** — Lifecycle coverage, add-on management, compatibility matrix, and platform capabilities + +:::note[MEP-19 — Cross-Partition Clusters] +With metal-stack Enhancement Proposal 19 (MEP-19), routing across data center partitions will be supported, allowing worker nodes to reside in separate metal-stack partitions while maintaining a single Kubernetes cluster. This requires partitions to be geographically close enough for stable low-latency connectivity. +::: + +:::note[Solution-Specific Details] +HA topologies, add-on management, and version policies differ significantly between Gardener and Cluster API. See the individual solution pages for implementation-specific details. +::: + +## Network Integration + +KCLM interacts with network automation systems through the cloud provider contract. During cluster creation, private node networks are automatically allocated, firewall rules are generated from cluster policies, and public IPs for LoadBalancer services are dynamically assigned. The [metal-ccm](./04-cloud-controller-manager.md) (cloud controller manager) bridges Kubernetes services with the underlying bare-metal networking stack, supporting MetalLB in BGP mode for IP address announcement. CNIs leverage the native routing infrastructure and auto-detect MTU requirements. + +## Bootstrap & Air-Gapped Environments + +KCLM bootstraps clusters using OCI-registry-pulled images and bootstrap tokens that work once for node joining. All cluster components — including the node agent, kubelet, CNI, and CSI — are pulled from registries that must be reachable from the target environment. This model supports air-gapped deployments where all required images are pre-pulled into a local registry. Clusters only reach ready state when all worker nodes have joined, VPN is established, and all managed resources are healthy. + +## What Makes metal-stack KCLM Great + +The combination of metal-stack's bare-metal infrastructure with KCLM delivers unique advantages: + +### Physical Isolation for Compliance + +Kubernetes control planes run in dedicated namespaces or seed clusters, physically separated from end-user workloads on metal-stack partitions. Operator-managed components are inaccessible to cluster owners — critical for compliance requirements. + +### Fleet-Wide Consistency + +Every cluster is provisioned from version-controlled YAML manifests with thorough integration testing across the compatibility matrix. No exceptions, no snowflake clusters — all features implemented as Kubernetes controllers and carried out fleet-wide. + +### Automatic Failure Recovery + +Controllers continuously reconcile desired state against actual state. Failed nodes are automatically detected and replaced within configurable health timeouts. Workloads shift seamlessly to replacement nodes without manual intervention. + +### Topology Awareness + +metal-stack distributes worker nodes across racks using a rack-spreading algorithm and provides well-known Kubernetes topology labels — enabling end-users to configure Pod topology spread and anti-affinity rules for high availability. + +### Zero Downtime for Management Outages + +The KCLM management layer is designed so that its absence does not impact cluster availability. Workloads continue running and end-users retain Kubernetes API access even when the management plane is unavailable. + +### Proven at Scale + +metal-stack with Gardener operates environments with **200+ Kubernetes clusters** across **5 data centers** and **1,800 physical servers** (including 200 servers in a metro environment across disjunct locations). Maximum nodes per cluster: 64 (up to 1,024 in metal-stack OnPrem). Proven Gardener installations manage **10,000+ clusters** — demonstrating that consistent, automated lifecycle management is the key to scaling bare-metal Kubernetes fleets. For Cluster API, the integration test environment covers a management cluster with three worker nodes, with the biggest test clusters including 8 cluster nodes. + +## Next Steps + +- **[Gardener](./02-gardener.md)** — Detailed concept documentation for metal-stack's recommended KCLM solution +- **[Cluster API](./03-cluster-api.md)** — Concept documentation for the CNCF-based alternative +- **[Cloud Controller Manager](./04-cloud-controller-manager.md)** — How metal-ccm bridges Kubernetes and bare-metal networking +- **[Firewall Controller Manager](./05-firewall-controller-manager.md)** — Firewall rule management via Kubernetes CRDs +- **[Deployment Guide](../04-For%20Operators/03-Deployment/01_guide.md)** — Deployment guide diff --git a/docs/05-Concepts/04-Kubernetes/02-cluster-api.md b/docs/05-Concepts/04-Kubernetes/02-cluster-api.md deleted file mode 100644 index 43840d61..00000000 --- a/docs/05-Concepts/04-Kubernetes/02-cluster-api.md +++ /dev/null @@ -1,18 +0,0 @@ ---- -slug: /cluster-api -title: Cluster API -sidebar_position: 2 ---- - -# Cluster API - -[Cluster API](https://cluster-api.sigs.k8s.io/) is a Kubernetes project that aims to simplify the management of Kubernetes clusters. It provides a declarative way to create, configure, and manage clusters using Kubernetes-style APIs. - -We provide the [Cluster API provider for metal-stack (CAPMS)](https://github.com/metal-stack/cluster-api-provider-metal-stack) infrastructure provider that allows the declaration of Kubernetes clusters. - -:::warning "Early Development Stage" -This project is currently under heavy development and is not advised to be used in production any time soon. -Please use our stack on top of [Gardener](./01-gardener.md) for production workloads. -::: - -See the [cluster-api-provider-metal-stack documentation](../../08-References/Kubernetes/cluster-api-provider-metal-stack/cluster-api-provider-metal-stack.md) for more in-depth information. diff --git a/docs/05-Concepts/04-Kubernetes/02-gardener.md b/docs/05-Concepts/04-Kubernetes/02-gardener.md new file mode 100644 index 00000000..151bf77d --- /dev/null +++ b/docs/05-Concepts/04-Kubernetes/02-gardener.md @@ -0,0 +1,242 @@ +--- +slug: /gardener +title: Gardener +sidebar_position: 2 +--- + +# Gardener + +[Gardener](https://gardener.cloud/) is an open source project for orchestrated Kubernetes cluster provisioning governed by the NeoNephos Foundation. It supports many different cloud providers, metal-stack being one of them. Using the Gardener project, metal-stack can act as a machine provider for Kubernetes worker nodes. + +Gardener is the **recommended** Kubernetes Cluster Lifecycle Management (KCLM) solution for metal-stack. It is battle-tested in production for over seven years at financial-sector customers and bundles more day-2 capabilities natively (DNS, backup, audit). Gardener manages entire clusters as Kubernetes-native resources with a strong separation between platform operators and end-users. + +For deployment instructions, see the [Gardener deployment guide](../04-For%20Operators/03-Deployment/05_gardener.md). + +## Outcomes + +Gardener targets three primary outcomes for bare-metal Kubernetes operations: + +- **Automation:** Gardener reconciliation loops across Shoot, Seed, and extension CRDs automate design, bootstrap, scaling, upgrades, and decommission end-to-end. The machine-controller-manager handles node lifecycle (create, replace, drain) automatically. Certificate rotation, ETCD backup, and add-on installation are built-in extensions. +- **Reproducibility:** Every Gardener Shoot cluster is a Git-versioned YAML manifest with a CloudProfile constraining versions, machine types, and regions. Infrastructure definitions, extension Helm charts, and seed configurations are version-controlled. GitOps operators deploy those manifests via CI/CD pipelines enabling peer review, audit trails, and rollback. +- **Risk reduction:** Gardener isolates each Shoot's control plane in its own namespace with dedicated ETCD, physically separated from end-user workloads on metal-stack partitions. Operator-managed control plane components (kube-apiserver, ETCD) are inaccessible to Shoot owners. etcd-druid provides continuous backup and automatic recovery. Admission controllers, health probes, and readiness gates reject misconfigurations early. + +## Architecture + +Gardener uses a hierarchical cluster model — often called the "kubeception" model — where Kubernetes clusters host other Kubernetes clusters. This architecture provides physical isolation between the Kubernetes control plane and end-user workloads, which is critical for compliance in regulated environments. + +### Cluster Hierarchy + +The diagram below shows the full deployment architecture — from the bootstrap infrastructure hosting the Garden Cluster (which can run on metal-stack cloud, any cloud provider, or on-prem), through the metal control plane, down to the metal-stack partition where Seeds provision Shoot clusters with their workloads. The firewall-controller in the partition integrates with the metal-stack firewall for Shoot network isolation. + +![Gardener cluster hierarchy](gardener-cluster-hierarchy.png) + +### Core Components + +| Component | Responsibility | +|-----------|---------------| +| **Garden cluster** | The top-level cluster that runs the Gardener control plane (API server, controller manager, scheduler, admission controller). Deployed via the `gardener-operator`. | +| **Virtual Garden** | A recommended deployment pattern where Gardener runs inside a virtual cluster on the Garden cluster. This provides a dedicated ETCD for Gardener resources and an independent update lifecycle from the Garden cluster itself. End users get project namespaces in the virtual garden. | +| **Seed cluster** | A cluster where a `gardenlet` agent runs. The gardenlet connects to the Gardener control plane and orchestrates provisioning of new clusters within that Seed. Typically one Seed per data-center site. A Seed that has been manually deployed (not by Gardener) is called a **soil**. | +| **Shoot cluster** | Every fully provisioned and managed Kubernetes cluster. The Shoot's control plane (kube-apiserver, etcd, controller-manager, scheduler) runs as pods in a dedicated namespace on a Seed, while worker nodes run on bare-metal machines provisioned via the metal-stack API. | + +### Core Controllers + +| Controller | Purpose | +|------------|---------| +| `gardener-operator` | Deploys Gardener components, gardenlets, and extensions; manages platform updates | +| `gardener-apiserver` | Extends the kube-apiserver with Gardener-specific resources (Shoot, Seed, Project, etc.) | +| `gardener-scheduler` | Decides where clusters are placed across the Gardener landscape (Seeds) | +| `gardener-controller-manager` | Reconciles common Gardener resources (projects, controller installations, etc.) | +| `gardenlet` | Agent running on each Seed; orchestrates provisioning of new clusters within that Seed | +| `gardener-resource-manager` | Runs inside Shoots; reconciles desired resources and checks their health | +| `etcd-druid` | etcd cluster operator with built-in backup-restore functionality | +| `machine-controller-manager` | Manages worker node lifecycle (rolling updates, health recreation, scaling) | + +### Provider Extensions + +| Extension | Purpose | +|-----------|---------| +| `gardener-extension-provider-metal` | Integrates metal-stack API with Gardener (machines, networks, firewalls, IPs). See also [Cloud Controller Manager](./04-cloud-controller-manager.md) and [Firewall Controller Manager](./05-firewall-controller-manager.md) for integration details. | +| `gardener-extension-audit` | Configures buffered forwarders to audit sinks (e.g., Splunk, S3) | +| `gardener-extension-networking-calico` | Provides Calico CNI in shoot clusters | +| `gardener-extension-networking-cilium` | Provides Cilium CNI in shoot clusters | +| `gardener-extension-shoot-dns-service` | Provides DNS records for the Kubernetes API server; allows shoot owners to create DNS records within their project domain | +| `gardener-extension-shoot-cert-service` | Provides certificates for services exposed in shoot clusters | + +### Integration Extensions + +| Extension | Purpose | +|-----------|---------| +| `os-metal-extension` | Translates Gardener's generic `OperatingSystemConfig` format into cloud-init (or ignition) userdata for metal-stack nodes | +| `machine-controller-manager-provider-metal` | Integrates metal-stack machine provisioning API with Gardener's MCM as an out-of-tree sidecar | + +For a complete architecture overview with diagrams, see the [Gardener documentation](https://gardener.cloud/docs/getting-started/architecture/). + +## Gardener Integration Components + +During the provisioning flow of a cluster, Gardener emits resources that are expected to be reconciled by controllers of a cloud provider. metal-stack implements the following integration components: + +```mermaid +sequenceDiagram + participant G as Gardener
Shoot Controller + participant EXT as gardener-extension-
provider-metal + participant MCM as machine-controller-
manager-provider-metal + participant MS as metal-stack API + participant NODE as Bare Metal Node + + G->>EXT: Creates Infrastructure
ControlPlane Worker resources + EXT->>MS: Allocate machines, networks,
firewalls, IPs + MS-->>EXT: Returns machine IDs, IPs + EXT->>MCM: Creates Worker resources + MCM->>MS: Provision worker machines + MS-->>MCM: Machines ready + MCM->>MS: Generate cloud-init userdata + MCM->>NODE: Bootstrap with cloud-init + NODE-->>MCM: Node joined + MCM->>G: Node ready + G->>G: Reconcile Shoot status +``` + +### gardener-extension-provider-metal + +The [gardener-extension-provider-metal](https://github.com/metal-stack/gardener-extension-provider-metal) contains a set of webhooks and controllers for reconciling cloud provider specific resources of `type: Metal`, which are created by Gardener during the cluster provisioning flow. + +Primarily, its purpose is to reconcile `Infrastructure`, `ControlPlane`, and `Worker` resources. The project also introduces its own API (`ProviderConfiguration` resources) and consists of an admission controller to validate them. + +### os-metal-extension + +The [os-metal-extension](https://github.com/metal-stack/os-metal-extension) translates Gardener's generic `OperatingSystemConfig` format into cloud-init (or ignition) userdata for metal-stack nodes. It enables the operating system bootstrap process for worker nodes. + +### machine-controller-manager-provider-metal + +Worker nodes are managed through Gardener's [machine-controller-manager](https://github.com/gardener/machine-controller-manager) (MCM). The MCM allows out-of-tree provider implementation via sidecar, which is implemented in the [machine-controller-manager-provider-metal](https://github.com/metal-stack/machine-controller-manager-provider-metal) repository. + +The MCM handles the full worker node lifecycle: creating new machines, draining and replacing unhealthy nodes, and performing rolling updates across worker groups. It prevents meltdown scenarios through configurable backoffs and watches Kubernetes node conditions continuously — certain conditions can be configured as critical, triggering health timeouts that cause automatic node replacement. The MCM also supports custom controllers that contribute to node conditions, enabling advanced health monitoring beyond the default `node-problem-detector`. + +## Operational Model + +Gardener differentiates between **end-users** (shoot owners) and **platform administrators** (seed owners), similar to how hyperscalers offer Kubernetes as a Service: + +| Role | Responsibilities | +|------|-----------------| +| **End-Users** | Create/manage clusters via API (limited to whitelisted machine types), manage worker groups, trigger cluster updates, configure maintenance windows and auto-updates | +| **Administrators** | Set up Seed clusters in data centers, whitelist machine types, provide Kubernetes versions and OS images, define lifecycle policies, fleet-wide GitOps-driven operations | + +End-users access the Virtual Garden through a feature-rich Kubernetes API with OIDC-based authorization. Resources like `ShootQuota` and custom webhooks can further restrict what end-users can do and ensure that they operate only within their specific, operational boundaries. As the API of Gardener is provided by Kubernetes itself, it is also an option to further narrow down the end-user scope by hiding the Gardener API behind a custom API. Platform administrators manage the entire platform through GitOps-driven processes with approval workflows. + +Gardener clearly defines responsibility boundaries — administrators manage the control plane lifecycle, CNI, CSI drivers, and other platform components, while end-users focus on their workloads. Components on the worker nodes like the Kubelet or CNI/CSI pods are continuously monitored and reconciled to the desired state, which makes it hard for end-users to misconfigure essential cluster components. The `node-problem-detector` is deployed by default to contribute node conditions that the machine-controller-manager watches for health timeouts. Custom controllers can also contribute node conditions for advanced health monitoring. Features like auto-updates for Kubernetes versions or machine images can be enabled and disabled by end-users individually and are carried out by Gardener in configurable maintenance time windows. + +## Failure Domains + +From a Kubernetes Cluster Lifecycle Management perspective, managing multiple failure domains is essential — allowing clusters to be provided in different geographical locations and enabling end-users to distribute workloads across different failure domains. To make topology information available in the resulting Kubernetes clusters, metal-stack's KCLM integrations automatically provide well-known Kubernetes topology labels on the node resources: + +- `machine.metal-stack.io/chassis` +- `machine.metal-stack.io/rack` +- `topology.kubernetes.io/region` +- `topology.kubernetes.io/zone` + +While metal-stack distributes cluster worker nodes across racks automatically using a rack-spreading algorithm, information like region and zone comes from the end-user's requirements. With this information, Kubernetes features like Topology Spread and PodAntiAffinity can be configured easily by end-users. With MEP-19 (metal-stack Enhancement Proposal 19), routing across data center partitions will also be supported, allowing worker nodes to reside in separate metal-stack partitions while maintaining a single Kubernetes cluster — provided the partitions are geographically close enough for stable low-latency connectivity. + +The KCLM is designed so that every failure domain can function without the KCLM itself — meaning the absence of the KCLM does not impact the availability of the Kubernetes clusters or the network traffic. Outages of the Gardener cluster only cause cluster provisioning to become unavailable, without interrupting workloads or preventing end-users from interacting with the Kubernetes API of their clusters. With MEP-19, worker nodes can also be spread across separate data center partitions while maintaining a single Kubernetes cluster, provided the partitions are geographically close enough for stable low-latency connectivity. + +## Control Plane Hosting + +In Gardener, the Kubernetes control planes for end-user clusters reside in so-called "seed clusters". In the metal-stack case, the seed clusters run on bare metal servers and are physically divided from the end-user worker nodes. Seed clusters can run at geographically different locations from the worker nodes. A secure connection between the Kubernetes control plane and components running on the worker nodes is established using VPN. It is also possible with Gardener to migrate Kubernetes control planes across seed clusters if necessary (shoot migration). + +There is a clear distinction between administrators, who are responsible for the Kubernetes control plane to run, and end-users, which only consume the provided Kubernetes API. This hosting model is mandatory in Gardener and suits well for pairing with bare metal in compliance-sensitive environments. + +## Control Plane Topologies + +Gardener supports multiple control plane topologies for on-prem failure domains: + +| Topology | Description | Use Case | +|----------|-------------|----------| +| **Single-seed HA** | Multiple control plane nodes (kube-apiserver, controller-manager, scheduler) across machines in the same Seed with etcd spread across machines for quorum. Default production-confirmed choice. | Single-site deployments, standard production | +| **Multi-rack** | Shoot control plane nodes spread across multiple racks within one Seed with etcd spread across racks. Rack-level failure isolation via MachineDeployment topology spread constraints. | Rack-level failure isolation within a single data center | +| **Multi-site** | Shoot control planes replicated across Seeds corresponding to different sites or data centers. MachineDeployments use zone constraints to distribute workers across regions. Higher latency for cross-seed communication requires multi-seed configuration. | Disaster recovery across geographically separated sites | +| **Dedicated Seed** | A Shoot gets its own dedicated Seed cluster with no shared control plane with other tenants. Highest compliance level for critical infrastructure at highest resource cost. | Strictest compliance requirements for critical infrastructure | + +All topologies are natively supported. Multi-site requires additional multi-seed configuration but no special extensions. + +## Upgrade & Rollback + +**Minor version upgrades** — Gardener rolls worker groups according to a configurable rollout strategy (drain timeouts, serial stateful set eviction, max surge and max unavailable nodes). For patch updates, Gardener applies in-place Kubelet upgrades within a jittered 5-minute window, preventing vanishing of route announcements for more than one node at a time. + +**Blue-green updates** — End-users can achieve zero-downtime upgrades through two approaches: (1) using multiple clusters with BGP Anycast to spread workloads across clusters, or (2) using worker groups with different Kubernetes kubelet versions and OS versions, combined with Kubernetes node taints and tolerations for traffic routing. + +**Rollback** — Kubernetes versions are not allowed to be rolled back. End-users are required to test Kubernetes upgrades in a staging cluster first. Tools exist to test for deprecated APIs before running the actual upgrade. + +**Downtime expectations** — With HA control plane enabled, no downtime is expected during regular version or Kubernetes upgrades. When upgrading MetalLB in a shoot cluster, it follows a one-by-one rolling update strategy so that route announcements only occur node-by-node until the speaker daemon set pod reports readiness again. + +## Domain Model + +The KCLM integrates with adjacent domains based on contracts. This way, the domains can be used without requiring direct upstream contributions to the KCLM but rather extend it and ideally run multiple domains in parallel. Gardener does not depend in any form on metal-stack components or APIs nor does it depend on other adjacent domains. metal-stack has implemented the integration in the form of extensions: + +- **Bare metal machine provisioning** — Abstraction through `CloudProfile`, which is referenced in a Shoot resource. Provider extensions can then implement the contracts for provisioning a cluster. For metal-stack this is the `gardener-extension-provider-metal`. +- **Network** — Abstraction through Kubernetes CNI, provider and network extensions. Gardener itself is unaware of the concrete network infrastructure that it provisions clusters on. Network extensions allow setting up CNIs like Cilium, Calico or others while provider extensions can set up infrastructure resources dynamically during provisioning. +- **Storage** — Abstraction through Kubernetes CSI, provider and storage extensions. Identical to network. +- **Kubernetes Distribution** — Gardener creates a Shoot cluster with vanilla Kubernetes components from the upstream Kubernetes project. No other Kubernetes distributions are supported. + +Gardener produces clusters built on vanilla Kubernetes that passes 100% CNCF Kubernetes conformance. It provisions and manages upstream Kubernetes distributions without modification — no fork or patched Kubernetes is used. As long as a replacement KCLM also supports vanilla Kubernetes, no action is required. + +## Operational Features + +Gardener provides several day-2 operational capabilities out of the box: + +- **Shoot migration** — Control planes can be migrated across Seed clusters for geographic relocation +- **VPN** — Secure connection between control plane components on Seeds and worker nodes +- **Scaling** — Manual and automatic scaling of worker groups with configurable strategies (max surge, max unavailable, drain timeouts); Cluster Autoscaler automatically adjusts worker groups based on requested pod resources +- **Hibernation** — Scale down infrastructure by deleting worker nodes and scaling down control plane components; can be triggered manually or by schedule +- **Maintenance time windows** — End-user configurable windows for automatic updates (Kubernetes versions, machine images) +- **Certificate rotation** — Automated certificate management for cluster components and shoot services; certificates are rotated in short time frames (8 hours). For certain components (e.g., CA of the Kubernetes API server), rotation is a two-phase process: first the new CA is added (giving users time to reconfigure clients), then the old CA is removed in a completion phase +- **etcd backup & restore** — Continuous etcd backups with automatic recovery via etcd-druid (typically configured with three-minute incremental deltas); configurable backup retention allows emergency access to cluster resources after accidental shoot deletion, which is further protected by special labels that prevent accidental shoot deletion. Shoot deletion is safeguarded by specific annotations — during deletion, Gardener attempts to clean up resources gracefully by triggering finalizers of controllers deployed into the shoot cluster, preventing orphaned external resources +- **DNS management** — Automatic DNS record creation for shoot API servers and cluster-internal services +- **Audit logging** — Configurable per-cluster audit policies with forwarding to external sinks +- **Access control lists** — Network-level access control for shoot clusters +- **Emergency reconciliation pause** — Operators can use Kubernetes annotations on seed resources to prevent shoot cluster reconciliation in emergency situations +- **Image vector overwrites** — Administrators can perform emergency patching fleet-wide by configuring image vector overwrites in the Gardener deployment Git repository, validated in staging environments before rollout. Emergency patches for OS images can also be offered through the `CloudProfile` and automatically applied within configured maintenance time windows for clusters with auto-upgrade enabled +- **Component upgrades** — Essential cluster components (CNI, CSI, CCM, DNS, audit) are rolled out with every new Gardener version fleet-wide, ensuring consistency across the entire landscape. End-users focus on their workloads without needing to manage Kubernetes internals + +## Bootstrap & Air-Gapped Environments + +Gardener uses a built-in bootstrap mechanism based on the `gardener-node-agent`. The kubelet and gardener-node-agent are both pulled from an OCI registry just like all other components running in the final cluster. This OCI registry must be reachable from within the (air-gapped) environment and must contain all images of the Gardener release's compatibility matrix. + +The kubelet is joined using a bootstrap token that is generated during the shoot provisioning workflow and only works once from the node to join the control plane. After a creation timeout in the machine-controller-manager (by default 20 minutes) the token gets recreated with another attempt to join the worker node with a new machine. + +Shoot clusters only reach ready state when all worker nodes have successfully joined, shoot VPN is running, and all managed resources (includes CNI, CSI, etc.) are healthy and running. + +## External Dependencies + +The following data center infrastructure dependencies are treated as given and must be available before deploying Gardener: + +- **DNS** — For cluster domain resolution (PowerDNS or external) +- **NTP** — Time synchronization across all nodes +- **ACME** — Certificate authority (for shoot certificates via shoot-cert-service) +- **S3-compatible object storage** — For etcd backups (gardener-extension-backup-s3) +- **Git-Hosting with CI/CD** — For GitOps-driven deployment of manifests +- **MetalLB** — For exposing Kubernetes API servers of Shoot clusters (layer 2 load balancing) + +## Scalability + +With the described operational model it is possible to scale massively. metal-stack operates an environment with **280 Kubernetes clusters** across **5 data centers** and **1,800 physical servers**. Proven Gardener installations manage **10,000+ clusters** — demonstrating that consistent, automated lifecycle management is the key to scaling bare-metal Kubernetes fleets. The key behind this high degree of scalability is that all clusters are provided consistently, providing a high level of integration testing and avoiding custom provisioning solutions. + +End-users can test minor Kubernetes upgrades first in clusters labeled `evaluation` or `development` before rolling out to `production`-labeled clusters. For evaluation clusters, auto-upgrades for Kubernetes and OS images can be enabled to stage updates, reduce manual effort, and gain security patches more quickly. Deprecation of Kubernetes versions can be configured in the `CloudProfile`, and end-users can pin worker group Kubernetes versions as long as the version skew policy is adhered. The Cluster Autoscaler automatically adjusts worker groups based on requested pod resources. + +## Version Skew Policy + +Gardener enforces Kubernetes version skew policies through static validation rules in the `gardener-apiserver` that cannot be circumvented. Worker group versions can be pinned as long as the skew policy is respected. Minor version upgrades are applied one-by-one following the [Gardener version skew policy](https://gardener.cloud/docs/gardener/deployment/version_skew_policy/). For details on supported Kubernetes versions and upgrade sequencing, see the [Gardener deployment documentation](https://gardener.cloud/docs/). + +## Audit & Traceability + +Kubernetes API audit policies are configurable per cluster, with logs forwarded to external sinks (e.g., Splunk, S3) for auditable change tracking: + +- **Shoot clusters** — The `gardener-extension-audit` extension allows shoot owners or operators to configure buffered forwarders to audit sinks. Audit policies for the kube-apiserver are configured via standard Kubernetes Policy manifests, with each cluster having its own set of policies. +- **Garden cluster** — The same audit extension can be configured for the gardener-apiserver and virtual kube-apiserver. + +For configuration examples, see the [Gardener deployment guide](../04-For%20Operators/03-Deployment/05_gardener.md). + +## Next Steps + +- **[KCLM Overview](./01-kclm.md)** — Introduction to Kubernetes Cluster Lifecycle Management with metal-stack +- **[Gardener Deployment Guide](../04-For%20Operators/03-Deployment/05_gardener.md)** — Step-by-step deployment instructions +- **[Gardener Documentation](https://gardener.cloud/docs/)** — Official Gardener documentation and API reference diff --git a/docs/05-Concepts/04-Kubernetes/03-cloud-controller-manager.md b/docs/05-Concepts/04-Kubernetes/03-cloud-controller-manager.md deleted file mode 100644 index ef82b964..00000000 --- a/docs/05-Concepts/04-Kubernetes/03-cloud-controller-manager.md +++ /dev/null @@ -1,11 +0,0 @@ ---- -slug: /metal-cloud-controller-manager -title: Cloud Controller Manager -sidebar_position: 3 ---- - -# metal Cloud Controller Manager - -CCM stands for [cloud-controller-manager](https://kubernetes.io/docs/concepts/architecture/cloud-controller/) and is the bridge between Kubernetes and a cloud-provider. - -We implemented the [cloud provider interface](https://github.com/kubernetes/cloud-provider/blob/master/cloud.go) in the [metal-ccm](https://github.com/metal-stack/metal-ccm) repository. With the help of the cloud-controller-controller we provide metal-stack-specific properties for Kubernetes clusters, e.g. load balancer configuration through MetalLB or node properties. diff --git a/docs/05-Concepts/04-Kubernetes/03-cluster-api.md b/docs/05-Concepts/04-Kubernetes/03-cluster-api.md new file mode 100644 index 00000000..dd6631e3 --- /dev/null +++ b/docs/05-Concepts/04-Kubernetes/03-cluster-api.md @@ -0,0 +1,243 @@ +--- +slug: /cluster-api +title: Cluster API +sidebar_position: 3 +--- + +# Cluster API + +[Cluster API](https://cluster-api.sigs.k8s.io/) is a Kubernetes project that aims to simplify the management of Kubernetes clusters. It provides a declarative way to create, configure, and manage clusters using Kubernetes-style APIs. + +We provide the [Cluster API provider for metal-stack (CAPMS)](https://github.com/metal-stack/cluster-api-provider-metal-stack) infrastructure provider that allows the declaration of Kubernetes clusters. + +:::warning[Beta] +Cluster API with metal-stack is in beta and not yet recommended for production workloads. Please use [Gardener](./02-gardener.md) for production deployments. We are actively looking for exchange and adopters — if you are interested in using Cluster API with metal-stack, please [join our community](/community) to help shape future integration efforts. +::: + +For deployment instructions, see the [KCLM deployment guide](../04-For%20Operators/03-Deployment/05_kclm.md). + +## Architecture + +Cluster API (CAPI) is a CNCF project maintained by a Kubernetes SIG that manages clusters through a management cluster holding `Cluster`, `MachineDeployment`, and `KubeadmControlPlane` resources. A metal-stack-specific CAPI infrastructure provider translates CAPI resources into metal-stack API calls. Control plane nodes are created as Machines on metal-stack infrastructure, and node bootstrapping uses kubeadm (or alternative bootstrap providers). Add-on lifecycle is managed through `ClusterResourceSet` objects. + +```mermaid +graph TB + subgraph "Management Cluster" + CAPI["Cluster API Controllers
Cluster, MachineDeployment,
KubeadmControlPlane"] + CAPMS["CAPMS Provider
Infrastructure controllers"] + CABPK["CABPK
Bootstrap provider"] + CAAPH["CAAPH
Add-on Provider for Helm"] + end + + subgraph "Workload Cluster" + CP["KubeadmControlPlane
Control plane nodes"] + MD["MachineDeployment
Worker nodes"] + CRS["ClusterResourceSet
CNI + CCM add-ons"] + end + + subgraph "metal-stack Infrastructure" + MS["metal-stack API
Machines, networks, firewalls"] + FW["Firewall nodes"] + end + + CAPI --> CAPMS + CAPI --> CABPK + CAPI --> CAAPH + CAPMS --> MS + CABPK --> CP + CAPMS --> MD + CAAPH --> CRS + MS -.->|provisions| CP + MS -.->|provisions| MD + MS -.->|provisions| FW + + style CAPI fill:#bbdefb + style CAPMS fill:#c8e6c9 + style CABPK fill:#fff9c4 + style CAAPH fill:#f8bbd0 + style MS fill:#e0f2f1 +``` + +### Core Components + +| Component | Responsibility | +|-----------|---------------| +| **Management Cluster** | A Kubernetes cluster that hosts the Cluster API controllers and the desired cluster state (as custom resources). It is the central control plane from which new workload clusters are declared and reconciled. | +| **Workload Cluster** | A Kubernetes cluster whose lifecycle is managed by the Management Cluster via CAPI resources. Its control plane and worker nodes are provisioned according to the declarative spec. | +| **Infrastructure Provider** | A set of controllers that translate CAPI's generic infrastructure resources (Cluster, Machine) into provider-specific resources. metal-stack is an officially listed infrastructure provider for Cluster API. The provider implementation is called cluster-api-provider-metal-stack (CAPMS). | +| **Bootstrap Provider** | Generates bootstrap data (e.g., cloud-init or ignition userdata) for joining new nodes. CAPMS is tested against the Kubeadm Bootstrap Provider (CABPK). | +| **ClusterResourceSet** | A CAPI feature for applying add-on resources (CNI, CCM, etc.) to newly created workload clusters automatically. CAPMS uses this with the Add-on Provider for Helm (CAAPH) for its calico flavor, to install calico as CNI and the [metal-ccm](./04-cloud-controller-manager.md). | +| **ClusterClass** | Defines reusable templates with parameterized variable schemas for tenant customization, enabling standardized cluster templates across the fleet. | +| **MachineHealthCheck** | Checks machine healthiness and takes remediation steps — unhealthy machines are automatically replaced by adding and deleting them on the infrastructure provider side, with safeguards in place (e.g., for not losing etcd quorum). | + +For a complete architecture overview with diagrams, see the [Cluster API documentation](https://cluster-api.sigs.k8s.io/user/concepts#concepts). + +## CAPMS CRDs + +CAPMS implements the CAPI infrastructure provider contract for bare metal via metal-stack. The following CRDs are provided by CAPMS: + +| CRD | Purpose | +|-----|---------| +| `MetalStackCluster` | Infrastructure cluster resource — allocates a control plane virtual IP (VIP) | +| `MetalStackMachine` | Bridges CAPI infrastructure machines to metal-stack machines (bare metal servers) | +| `MetalStackMachineTemplate` | Defines reusable machine specs (image, size, etc.) for MetalStackMachine resources | +| `MetalStackFirewallDeployment` | Declares firewall deployments protecting a cluster's network perimeter | +| `MetalStackFirewallTemplate` | Provides the configuration template for deployed firewalls | + +## Operational Model + +The operational model for Cluster API is less opinionated than the one described for [Gardener](./02-gardener.md). We recommend administrators to fully take care of the management cluster and control the declarative state of the entire infrastructure using GitOps-driven workflows. For this scenario, administrators provide end-users with workload clusters and end-users do not gain access to the management clusters. This approach is very controlled and does not allow end-users to provide clusters in a self-service fashion. If such a behavior is desired, we recommend developing another API layer on top of Cluster API that solves this specific purpose. + +In comparison to the Gardener approach, the Cluster API model is much less complex and more versatile, allowing for more individual configurations at the cost of operational overhead, time and scalability. With the kubeadm controller there is no physical isolation between the Kubernetes control plane and the end-user. Provisioned clusters should be configured with proper RBAC permissions for end-users such that unintended misconfiguration or meltdown is prevented. + +Changes need to be coordinated individually per cluster. Cluster API itself does not provide a maintenance time window and reconciles continuously instead. Unlike Gardener, the Kubernetes version skew policy is not strictly enforced in Cluster API — risk management falls within the scope of platform administration and deployment processes with approval gates. + +## Control Plane Hosting + +In the case of Cluster API with bootstrap provider Kubeadm (CABPK), the Kubernetes control planes reside on the worker nodes in the same cluster and are not fully isolated from the end-users. This hierarchy is a mandatory setup in Cluster API when using the Kubeadm provider. + +There are other providers from the CABPK ecosystem (e.g. [Kamaji](https://kamaji.clastix.io/)), which allow similar hosting models as the one described for Gardener, where the control plane runs on dedicated infrastructure separate from worker nodes. However, those integrations have not been evaluated in production-grade scenarios — at least from our side. + +### Control Plane Topologies + +Cluster API supports multiple control plane topologies for on-prem failure domains: + +| Topology | Description | Use Case | +|----------|-------------|----------| +| **Single-site HA** | Multiple control plane Machines within a single partition with etcd replicas on separate Machines. Natively supported. | Single-site deployments, standard production | +| **Multi-failure-domain** | Control plane Machines distributed across multiple zones or regions with etcd spread accordingly. Natively supported through ClusterClass topology definitions. | Rack/zone-level failure isolation | +| **Multi-site** | Control plane Machines deployed across multiple CAPI management clusters or across widely separated MetalPools with cross-site etcd replication. Requires additional operator effort for cross-site networking. | Disaster recovery across geographically separated sites | +| **Dedicated isolation** | A dedicated Cluster with its own isolated MetalPool and exclusive use of MetalPools. Same isolation level as Gardener's dedicated Seed for critical infrastructure. | Strictest compliance requirements for critical infrastructure | + +All topologies are natively supported. Multi-site requires additional multi-pool configuration and cross-site networking setup. + +### Kamaji with metal-stack + +[Kamaji](https://kamaji.clastix.io/) is a Control Plane Manager for Kubernetes that runs control planes as pods within a management cluster, reducing operational overhead and costs. It supports multi-tenancy, high availability, and integrates with Cluster API as a `ControlPlaneProvider`. + +Kamaji allows a similar control plane hosting model as Gardener, where the control plane runs on dedicated infrastructure separate from worker nodes. + +:::warning +Kamaji integrations with metal-stack have not been evaluated in production-grade scenarios. We are actively looking for exchange and adopters — if you are interested in using Kamaji with metal-stack, please [join our community](/community) to help shape future integration efforts. +::: + +Kamaji acts as a `ControlPlaneProvider` with Cluster API, while CAPMS acts as the `InfrastructureProvider`. This setup manages **tenant clusters** on metal-stack infrastructure, combining Kamaji's control plane management with metal-stack's bare-metal provisioning. + +Like Cluster-API, Kamaji is a framework rather than a complete platform — operators must assemble their own day-2 tooling (CNI, CCM, DNS, backup, certificate management) and manage them through GitOps workflows. + +**Deployment** + +1. **Prepare management cluster** — A Kubernetes cluster to host Kamaji and CAPMS providers +2. **Install Kamaji and CAPMS** — Deploy both providers into the management cluster +3. **Create a control plane VIP** — MetalLB assigns a virtual IP for the tenant API server +4. **Generate and apply tenant cluster manifest** — Use `clusterctl generate cluster` to produce a YAML with `Cluster`, `MetalStackCluster`, `KubeadmControlPlane`, `MachineDeployment`, and `MetalStackMachine` resources, then apply it +5. **Deploy add-ons** — Install CNI (Calico) and `metal-ccm` into the tenant cluster + +A working showcase is available in the [`capi-lab`](https://github.com/metal-stack/cluster-api-provider-metal-stack/blob/main/DEVELOPMENT.md#running-the-kamaji-flavor) setup, which extends the `mini-lab` with a Kamaji flavor. See our [blog post](/blog/2026/04-kamaji) for a detailed walkthrough of the architecture and setup. + +**Fleet management and GitOps** + +Since Kamaji with metal-stack uses Cluster-API under the hood, fleet management follows the same pattern as Cluster API. Tenant cluster manifests are generated via `clusterctl`, stored in Git, and deployed through your CI/CD pipeline. + +## Domain Model + +CAPI does not depend in any form on metal-stack components or APIs. metal-stack has implemented the integration in the form of the CAPMS infrastructure provider. The different domains are abstracted/integrated as follows: + +- **Bare metal machine provisioning** — Abstraction through Cluster resource, which holds references to a specific control plane and infrastructure provider, for which dedicated CRDs exist. For the metal-stack integration, the dedicated resources are provided by the cluster-api-provider-metal-stack. +- **Network** — Abstraction through Kubernetes CNI, infrastructure provider and ClusterResourceSet. It is unaware of the concrete network infrastructure and the infrastructure provider can set up resources dynamically if necessary. +- **Storage** — Abstraction through Kubernetes CSI and ClusterResourceSet. Identical to network with the exception that the infrastructure provider does not really participate in providing CSI for the workload cluster. +- **Kubernetes Distribution** — CAPMS currently integrates with CABPK for installing vanilla Kubernetes via kubeadm. This relies on vanilla Kubernetes; other distributions have not been explored. + +CAPI produces clusters built on vanilla Kubernetes. As long as a replacement KCLM also supports vanilla Kubernetes, no action is required. + +## Outcomes + +- **Automation:** CAPI reconciles Cluster, KubeadmControlPlane, and MachineDeployment CRDs from a management cluster. The metal-stack CAPI provider allocates machines, firewalls, and IPs declaratively via manifests. Scaling, upgrading, deletion, and add-on installation (via ClusterResourceSet + Helm) follow the same reconciler loop. +- **Reproducibility:** CAPI stores all resources (Cluster, ClusterClass, MachineTemplate, MetalPool) as Git-versioned YAML. ClusterClass defines reusable templates with parameterized variable schemas for tenant customization. GitOps operators (ArgoCD/FluxCD) ensure drift-free declarative delivery. +- **Risk reduction:** CAPI isolates each managed Cluster in its own namespace within the management cluster. The reconciler continuously compares actual infrastructure against desired spec. Failed nodes trigger automatic MachineHealthCheck remediation. Each managed cluster is independent, so automation errors affect only the target cluster. All operations are Git-auditable through versioned manifests. + +## Network Integration + +Network integration for Cluster API is currently more manual compared to Gardener. Node networks must be created manually via `metalctl` and provided as environment variables. IP addresses for the control plane also need to be allocated in advance through `metalctl`. Firewall rules are currently static and can be applied to firewall nodes; no automatic firewall controller is in place yet. Automatic network resource allocation is on the roadmap for CAPMS. + +For service exposure, CAPMS uses KubeVIP in BGP mode to allocate and announce public IPs, similar to the MetalLB-based approach in Gardener. + +## Air-Gapped Environments + +For air-gapped deployments, follow the [Cluster API Operator air-gapped environment guide](https://cluster-api-operator.sigs.k8s.io/topics/configuration/air-gapped-environtment). All required images must be mirrored to an OCI registry reachable from the management cluster. + +## Fleet Management and GitOps + +You must set up your own Git repository and GitOps operator to manage cluster deployments. + +**What you need to build:** + +1. **Git repository** — Store cluster manifests generated via `clusterctl generate cluster `. Each cluster gets its own set of YAML files containing `Cluster`, `MetalStackCluster`, `KubeadmControlPlane`, `MachineDeployment`, and `MetalStackMachine` resources. +2. **GitOps operator** — Deploy ArgoCD or FluxCD to watch your Git repository and apply manifests to the management cluster, ensuring drift-free declarative delivery. +3. **Per-cluster CI/CD** — Essential components (CNI, CCM) are rolled out on a per-cluster basis. Changes to `MachineTemplate` or `ClusterResourceSet` are staged through the Git repository with standard approval processes. + +**Platform capabilities:** + +- **Cluster migration** — `clusterctl move` enables moving workload cluster resources between management clusters, pausing controllers during the move to prevent worker node loss +- **Emergency patching** — Achieved through editing resources in the management cluster, e.g., update of machine OS image in the `MachineTemplate` or update of `ClusterResourceSet`. Unlike Gardener, this change is not rolled out fleet-wide automatically and should be staged through the Git repository with standard approval processes +- **Certificate rotation** — No direct workflow is described for certificate rotation at the landscape level; this is to be defined by platform administrators using manual/custom processes based on standard tooling (e.g., `kubeadm` certificate renewal) +- **Shoot deletion protection** — Special labels safeguard against accidental shoot deletion; configurable backup retention allows emergency access to cluster resources before cleanup +- **Cluster Autoscaler** — Automatic scaling of worker groups based on requested pod resources, configured per `MachineDeployment` + +**Audit configuration** — Audit configuration can be passed to the kube-apiserver via the `kubeadmConfigSpec` of the `KubeadmControlPlane` resource before cluster creation. Each cluster can have its own audit policy. The management cluster's kube-apiserver audit also needs to be configured separately. Cluster API does not provide a centralized audit management toolset and relies on cloud-native standards to be set up by the operator. + +## External Dependencies + +The following data center infrastructure dependencies are treated as given and must be available before deploying Cluster API with metal-stack: + +- **DNS** — For cluster domain resolution +- **NTP** — Time synchronization across all nodes +- **ACME** — Certificate authority +- **S3-compatible storage** — For backups +- **Git-Hosting with CI/CD** — For GitOps-driven deployment of manifests + +## Scale & Testing + +For Cluster API, the integration test environment covers a management cluster with three worker nodes. The biggest test clusters have included 8 cluster nodes. As both CAPI and Gardener share the same metal-stack control plane, from the metal-stack perspective it is guaranteed to work with the numbers mentioned for Gardener (200+ clusters, 5 data centers, 1,800 physical servers). Validations in CAPI are not as thoroughly implemented as in Gardener, but all usual management workflows for metal-stack clusters (creation, move, and deletion) are integration tested within a matrix of Kubernetes version, CNI, and OS version. + +## Fleet Operations + +Unlike Gardener, Cluster API does not provide built-in fleet-wide operations. All operational changes must be applied through GitOps-driven workflows with approval processes. The risk of platform updates can be reduced by utilizing multi-stage environments (staging → production). Cluster manifests should be validated through CI/CD pipelines before deployment to prevent misconfigurations and manage divergence across the fleet. + +End-users can test Kubernetes upgrades first in clusters labeled `evaluation` or `development` before rolling out to `production`-labeled clusters. For evaluation clusters, auto-upgrades for machine specs can be enabled to stage updates and reduce manual effort. + +## Upgrade & Rollback + +**Minor version upgrades** — Control plane upgrades are triggered by spec updates in the `Cluster` resource, which should be managed in Git using GitOps-driven processes for deployment and approval. Cluster API orchestrates a one-by-one worker node roll. There are no maintenance time windows — CAPI reconciles continuously. + +**Blue-green updates** — End-users can achieve zero-downtime upgrades through two approaches: (1) using multiple clusters with BGP Anycast to spread workloads across clusters, or (2) using worker groups or machine pools with different Kubernetes kubelet versions and OS versions, combined with Kubernetes node taints and tolerations for traffic routing. + +**Rollback** — Kubernetes versions are not allowed to be rolled back. End-users are required to test Kubernetes upgrades in a staging cluster first. Tools exist to test for deprecated APIs before running the actual upgrade. + +**Downtime expectations** — Every Kubernetes or machine update triggers an orchestrated worker roll, updating the Kubernetes control planes and kubelets of the worker nodes one-by-one. Unlike Gardener, there is no jittered upgrade window to prevent simultaneous route announcement vanishing. + +## MEP-19 — Cross-Partition Clusters + +With metal-stack Enhancement Proposal 19 (MEP-19), routing across data center partitions will be supported, allowing worker nodes to reside in separate metal-stack partitions while maintaining a single Kubernetes cluster. This requires partitions to be geographically close enough for stable low-latency connectivity. This enhancement will enable Cluster API to provision clusters with workers spread across partitions, similar to the multi-failure-domain topology. + +## What to Build Yourself + +Unlike Gardener, Cluster API with metal-stack requires you to assemble your own day-2 operations tooling: + +- **DNS management** — No built-in DNS service; configure via external DNS providers +- **etcd backup & restore** — No built-in operator; deploy etcd-druid or similar tooling +- **Certificate rotation** — No direct workflow; use manual/custom processes based on `kubeadm` certificate renewal +- **Audit logging** — Configure via `kubeadmConfigSpec` per cluster; no centralized management +- **Maintenance windows** — None built-in; CAPI reconciles continuously +- **Version skew enforcement** — Not strictly enforced; risk management via approval gates +- **Hibernation** — Not available +- **Cluster migration** — Available via `clusterctl move` between management clusters +- **Multi-tenant self-service** — Not available; build custom API layer on top of Cluster API +- **Access control lists** — No built-in firewall controller; firewall rules are currently static + +## Next Steps + +- **[KCLM Overview](./01-kclm.md)** — Introduction to Kubernetes Cluster Lifecycle Management with metal-stack +- **[Gardener](./02-gardener.md)** — metal-stack's recommended, production-ready KCLM solution +- **[KCLM Deployment Guide](../04-For%20Operators/03-Deployment/05_kclm.md)** — Step-by-step deployment instructions +- **[Cluster API Documentation](https://cluster-api.sigs.k8s.io/)** — Official Cluster API documentation diff --git a/docs/05-Concepts/04-Kubernetes/04-cloud-controller-manager.md b/docs/05-Concepts/04-Kubernetes/04-cloud-controller-manager.md new file mode 100644 index 00000000..8fc4cee6 --- /dev/null +++ b/docs/05-Concepts/04-Kubernetes/04-cloud-controller-manager.md @@ -0,0 +1,49 @@ +--- +slug: /metal-cloud-controller-manager +title: Cloud Controller Manager +sidebar_position: 4 +--- + +# metal Cloud Controller Manager + +The [cloud-controller-manager](https://kubernetes.io/docs/concepts/architecture/cloud-controller/) (CCM) is the bridge between Kubernetes and a cloud provider. metal-stack provides its own implementation, [metal-ccm](https://github.com/metal-stack/metal-ccm), which implements the [Kubernetes cloud provider interface](https://github.com/kubernetes/cloud-provider/blob/master/cloud.go). + +## Purpose + +metal-ccm provides metal-stack-specific integration for Kubernetes clusters, primarily: + +- **Load balancer configuration** — Exposes `Service` resources of type `LoadBalancer` by allocating and announcing public IP addresses via MetalLB +- **Node property management** — Annotates nodes with infrastructure-specific labels (rack, region, zone, chassis) for topology-aware scheduling + +## Load Balancer + +metal-ccm integrates with [MetalLB](https://metallb.universe.tf/) in BGP mode to expose Kubernetes `Service` resources of type `LoadBalancer`. When a user creates a LoadBalancer service, metal-ccm: + +1. Allocates a public IP address from the metal-stack infrastructure pool +2. Configures MetalLB to announce that IP via BGP to the network topology + +This provides layer-4 load balancing without requiring dedicated hardware load balancers. + +## Node Labels + +metal-ccm annotates every worker node with well-known Kubernetes topology labels that reflect the underlying bare-metal infrastructure: + +- `machine.metal-stack.io/rack` — The rack where the physical server resides +- `machine.metal-stack.io/chassis` — The chassis identifier +- `topology.kubernetes.io/region` — The geographic region +- `topology.kubernetes.io/zone` — The availability zone or rack + +These labels enable end-users to configure Pod topology spread constraints and anti-affinity rules, ensuring high availability of workloads across failure domains. + +## Deployment + +For Gardener deployments, metal-ccm is deployed as part of the Shoot cluster provisioning flow via the `gardener-extension-provider-metal`. For Cluster API deployments, it is installed through `ClusterResourceSet` objects alongside the CNI (Calico). + +For detailed build and deployment instructions, see the [metal-ccm reference guide](../../08-References/Kubernetes/metal-ccm/metal-ccm.md). + +## Next Steps + +- **[KCLM Overview](./01-kclm.md)** — Introduction to Kubernetes Cluster Lifecycle Management +- **[Gardener](./02-gardener.md)** — Gardener integration +- **[Cluster API](./03-cluster-api.md)** — Cluster API integration +- **[metal-ccm Reference](../../08-References/Kubernetes/metal-ccm/metal-ccm.md)** — Build and deployment instructions diff --git a/docs/05-Concepts/04-Kubernetes/04-firewall-controller-manager.md b/docs/05-Concepts/04-Kubernetes/04-firewall-controller-manager.md deleted file mode 100644 index 5c11ac57..00000000 --- a/docs/05-Concepts/04-Kubernetes/04-firewall-controller-manager.md +++ /dev/null @@ -1,15 +0,0 @@ ---- -slug: /firewall-controller-manager -title: Firewall Controller Manager -sidebar_position: 4 ---- - -# Firewall Controller Manager - -To make the firewalls created with metal-stack easily configurable through Kubernetes resources, we add our [firewall-controller](https://github.com/metal-stack/firewall-controller) to the firewall image. The controller watches special CRDs, enabling users to manage: - -- nftables rules -- Intrusion-detection with [suricata](https://suricata.io/) -- network metric collection - -Please check out the [guide](../../08-References/Kubernetes/firewall-controller-manager/firewall-controller-manager.md) on how to use it. diff --git a/docs/05-Concepts/04-Kubernetes/05-firewall-controller-manager.md b/docs/05-Concepts/04-Kubernetes/05-firewall-controller-manager.md new file mode 100644 index 00000000..c3a8cac0 --- /dev/null +++ b/docs/05-Concepts/04-Kubernetes/05-firewall-controller-manager.md @@ -0,0 +1,54 @@ +--- +slug: /firewall-controller-manager +title: Firewall Controller Manager +sidebar_position: 5 +--- + +# Firewall Controller Manager + +The firewall-controller-manager (FCM) is a collection of controllers responsible for managing the lifecycle of firewalls in a [Gardener](https://gardener.cloud/) shoot cluster for the metal-stack provider. + +The FCM is typically deployed into the shoot namespace of a seed cluster by the [gardener-extension-provider-metal](https://github.com/metal-stack/gardener-extension-provider-metal/). + +The design of the FCM is inspired by Gardener's [machine-controller-manager](https://github.com/gardener/machine-controller-manager) and Kubernetes' built-in resources `Deployment`, `ReplicaSet` and `Pod`. + +## Architecture + +The FCM introduces the following [CRDs](https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/): + +| Custom Resource | Description | +|----------------|-------------| +| `FirewallDeployment` | Contains the spec template of a `Firewall` resource, similar to a `Deployment`. Implements update strategies like rolling update. | +| `FirewallSet` | Similar to `ReplicaSet`. Typically owned by a `FirewallDeployment`. Attempts to run the defined number of `Firewall` replicas. | +| `Firewall` | Similar to a `Pod`. Has a 1:1 relationship to a firewall in the metal-stack API. | +| `FirewallMonitor` | Deployed into the user's shoot cluster. Useful for monitoring the firewall or triggering user-initiated actions. | + +### Controllers + +| Controller | Responsibility | +|------------|---------------| +| `FirewallDeploymentController` | Manages the lifecycle of `FirewallSet`s. Syncs the `Firewall` template spec and triggers a `FirewallSet` roll when significant changes are made. Supports `RollingUpdate` and `Recreate` strategies. Also deploys a service account for the firewall-controller. | +| `FirewallSetController` | Creates and deletes `Firewall` objects according to the spec and replica count. Reports `Firewall` status. | +| `FirewallController` | Creates and deletes the physical firewall machine via the [metal-api](https://github.com/metal-stack/metal-api). | + +## User Actions + +The FCM exposes user-facing capabilities through annotations on `FirewallMonitor` and `Firewall` resources: + +- **FirewallSet rolling** — Users can trigger a rolling update of the current firewall set by annotating a `FirewallMonitor`. This is useful when firewall configuration changes need to be applied. +- **Systemd service restart** — Users can restart systemd services on a firewall node by annotating a `FirewallMonitor`. The firewall-controller enforces a whitelist of allowed services. Operators can override this whitelist by annotating the `Firewall` resource directly. + +## Deployment + +The FCM is deployed into the shoot namespace of a seed cluster as part of the Gardener extension provisioning flow. + +For detailed configuration and development instructions, see the [firewall-controller-manager reference guide](../../08-References/Kubernetes/firewall-controller-manager/firewall-controller-manager.md). + +For detailed configuration and development instructions, see the [firewall-controller-manager reference guide](../../08-References/Kubernetes/firewall-controller-manager/firewall-controller-manager.md). + +## Next Steps + +- **[KCLM Overview](./01-kclm.md)** — Introduction to Kubernetes Cluster Lifecycle Management +- **[Gardener](./02-gardener.md)** — Gardener integration +- **[Cluster API](./03-cluster-api.md)** — Cluster API integration +- **[Firewall Controller Manager Reference](../../08-References/Kubernetes/firewall-controller-manager/firewall-controller-manager.md)** — Detailed configuration and development instructions diff --git a/docs/05-Concepts/04-Kubernetes/05-isolated-clusters.md b/docs/05-Concepts/04-Kubernetes/06-isolated-clusters.md similarity index 99% rename from docs/05-Concepts/04-Kubernetes/05-isolated-clusters.md rename to docs/05-Concepts/04-Kubernetes/06-isolated-clusters.md index 3d45c6c5..3469970d 100644 --- a/docs/05-Concepts/04-Kubernetes/05-isolated-clusters.md +++ b/docs/05-Concepts/04-Kubernetes/06-isolated-clusters.md @@ -1,7 +1,7 @@ --- slug: /isolated-kubernetes-clusters title: Isolated Clusters -sidebar_position: 5 +sidebar_position: 6 --- # Isolated Kubernetes Clusters diff --git a/docs/05-Concepts/04-Kubernetes/gardener-cluster-hierarchy.drawio.svg b/docs/05-Concepts/04-Kubernetes/gardener-cluster-hierarchy.drawio.svg new file mode 100644 index 00000000..48d46387 --- /dev/null +++ b/docs/05-Concepts/04-Kubernetes/gardener-cluster-hierarchy.drawio.svg @@ -0,0 +1,1270 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+
+
+ Garden Cluster (soil) +
+
+
+ + + Garden Cluster (soil) + + + + + + + + + + + + + + +
+
+
+ Virtual Garden +
+
+
+ + + Virtual Garden + + + + + + + + + + + + + + +
+
+
+ Seed (shooted seed) +
+
+
+ + + Seed (shooted seed) + + + + + + + + + + + + + + +
+
+
+ Shoot 1 +
+
+
+ + + Shoot 1 + + + + + + + + + + + + + + + + + + +
+
+
+ firewall +
+
+
+ + + firewall + + + + + + + + + + + + + + + +
+
+
+ firewall-controller +
+
+
+ + + firewall-controller + + + + + + + + + + + + +
+
+
+ Backup Buckets +
+
+
+ + + Backup Buck... + + + + + + + + + + + + + + + + + + + +
+
+
+ gardenctl +
+
+
+ + + gardenctl + + + + + + + + + + + + + + + + + + + + + + + + +
+
+
+ bootstrap infrastructure +
+
+
+ + + bootstrap infrastructure + + + + + + + + + + + +
+
+
+ metalstack.cloud, any cloud provider or on-prem +
+
+
+ + + metalstack.cloud, any cloud provider or on-prem + + + + + + + + + + + +
+
+
+ s3 compatible +
+ storage +
+
+
+
+ + + s3 compatible... + + + + + + + + + + + + +
+
+
+ metal-stack partition +
+
+
+ + + metal-stack partition + + + + + + + + + + + +
+
+
+ on-prem +
+
+
+ + + on-prem + + + + + + + + + + + + +
+
+
+ + Gardener Deployment + +
+
+
+ + + Gardener Deployment + + + + + + + + + + + + + + + +
+
+
+ kubectl +
+
+
+ + + kubectl + + + + + + + + + + + + +
+
+
+ + manages + +
+
+
+ + + manages + + + + + + + + + + + + + + + + + + + + + + + +
+
+
+ user +
+
+
+ + + user + + + + + + + + + + + + + + +
+
+
+ tenant-api +
+
+
+ + + tenant-api + + + + + + + + + + + metal-console + + + + + + + + + + +
+
+
+ ipam +
+
+
+ + + ipam + + + + + + + + + + + + + + + +
+
+
+ metal-apiserver +
+
+
+ + + metal-apiserver + + + + + + + + + + + + + + +
+
+
+ metal control plane +
+
+
+ + + metal control plane + + + + + + + + + + + + + + + + + + + + + +
+
+
+ + garden-kube-apiserver + +
+
+
+ + + garden-kube-apiserver + + + + + + + + + + + +
+
+
+ metal-api +
+
+
+ + + metal-api + + + + + + + + + + + + + + + + + + + + + + + +
+
+
+ Shoot 2 +
+
+
+ + + Shoot 2 + + + + + + + + + + + + + + + + + + + + + + + + +
+
+
+ + Shoot 1 + +
+
+
+ + + Shoot 1 + + + + + + + + + + + + + + + + + +
+
+
+ + manages + +
+
+
+ + + manages + + + + + + + + + + + + +
+
+
+ + manages + +
+
+
+ + + manages + + + + + + + + + + + + +
+
+
+ etcd +
+
+
+ + + etcd + + + + + + + + + + + + + + + + + + + +
+
+
+ garden +
+
+
+ + + garden + + + + + + + + + + + +
+
+
+ seed-admission-controller +
+
+
+ + + seed-admission-controller + + + + + + + + + + + + + + + +
+
+
+ gardenlet +
+
+
+ + + gardenlet + + + + + + + + + + + +
+
+
+ resource-manager +
+
+
+ + + resource-manager + + + + + + + + + + + + + + + + + + +
+
+
+ provider-metal +
+
+
+ + + provider-metal + + + + + + + + + + + +
+
+
+ provider-gcp +
+
+
+ + + provider-gcp + + + + + + + + + + + +
+
+
+ dns-external +
+
+
+ + + dns-external + + + + + + + + + + + + + + + +
+
+
+ os-metal +
+
+
+ + + os-metal + + + + + + + + + + + + + + + + + + + + + + + + + +
+
+
+ + Shoot 2 + +
+
+
+ + + Shoot 2 + + + + + + + + + + + +
+
+
+ + extension + +
+ + providers + +
+
+
+
+ + + extension... + + + + + + + + + + + +
+
+
+ shoot +
+ + control planes + +
+
+
+
+ + + shoot... + + + + + + + + + + + +
+
+
+ Workload +
+
+
+ + + Workload + + + + + + + + + + + + + + + + +
+
+
+ kube-apiserver +
+
+
+ + + kube-apiserver + + + + + + + + + + + + + + +
+
+
+ Workload +
+
+
+ + + Workload + + + + + + + + + + + + + + + + +
+
+
+ Workload +
+
+
+ + + Workload + + + + + + + + + + + + + + + + +
+
+
+ Workload +
+
+
+ + + Workload + + + + + + + + + + + + + + + + + + + + + + + + +
+
+
+ apiserver +
+
+
+ + + apiserver + + + + + + + + + + + +
+
+
+ controller-manager +
+
+
+ + + controller-manager + + + + + + + + + + + +
+
+
+ scheduler +
+
+
+ + + scheduler + + + + + + + + + + + +
+
+
+ admission-controller +
+
+
+ + + admission-controller + + + + + + + + + + + + + + + + + + + + Text is not SVG - cannot display + + + + diff --git a/docs/05-Concepts/04-Kubernetes/gardener-cluster-hierarchy.png b/docs/05-Concepts/04-Kubernetes/gardener-cluster-hierarchy.png new file mode 100644 index 0000000000000000000000000000000000000000..3ecd9133df99d7a7c34f868c7605fe25b51567d2 GIT binary patch literal 301951 zcmeFa>8`6-)-9O2C6!87x-U@wL;CKO@(2U&q>@T>#@GfM_xK;i-ZXA-gN@A_+*hc7 z^+@$bHRb~L_HIr@btE79i3?)DOH*MIv(Q1ZWi5wu@8=@*`( ze}b|KUH|0n@F86NBB;OcJlC+>dud=bhvTEcRR!bPIpHwQ*t@%Re$K%4*MG0j(Z|-x zHg`Au zw^1#IM`yR!p&yhzSGeMvom-w<95@H=Do*q3*`d#ZEX8?>W3>06wfu|6|0m@t>n%yU zlw;A)FMR7CzxGucr}e&=${%U2Dpqs#UES@akYs+~oZL&%p5n`XwI+1{mU0?|$$E5f zJvu!Fns6*gc3XU^;Lr1Bop>aDI`<{sCk0d>ll-$1Z{rib|AiM??1zqr+wH?~Rigi> zvE&qQ)(_t(4f+q??Bc7tWCNs5PwR7-J${Z8H+lFBw67mOuUFyY$8GFwUguW;^x^7c zeM|b7aF~#BG-x96>&GU2%tdx^-;`45>AF7QYBpP*9r|^!l(>SKuH)T0Io$!9;&Q%L zFEm}4Xsz+BP+s~5_$D<$_82^Xy;iT3r@Doi`LFHM$-f7RIHy4Jug~ENz*ohBOa4+Q ztF-99cghqe$}$F1&gbD}%TkAL|AUkhxCaU? z!ySwGWeTA}#Q8mUx!w-qbC`tj9^WtHB#C#pU`t=fuD9`7p!$`*$~L5J=+m`He87e7 z`H*KKFX4^CKbdtYl}8CF#URub!pUC*DgVf~&R72zUcMpd<&{C(;um*WT+VJ6FXKYD zJ>)z!zV?Kbu)0?T^IpArJMU^nQfeyAY_;=3qa7qmy_)s+PSe;0+^Q{S{jhOe?WWO6 z*Dmg?ab50Om*w5?SNwLf(*33pd8rwSqNO^SIW{>fb1GJ3RE$uL7qji&ZQD4<5zUcU zwIc6kszz#!C84iscbvn!v#DV3n&Mz&+)PDGO;te*5NFx|phz_Qz(1 zV}d!JRM5}eP`}f+`knqFac2lY&`&oFPZ(C0`ov$gx9MVxBuJl%E2SBEf*B4%CtB-T zT^Uugl`7X4yX|?Y#*vvByu(i_eSs_6xp;zoTl0& z%o5Hsa2{}a=V+`|E5uks8mpEtd8iVbQ@;?fMVhra9#bLD!`q-qjI?wBL`!x(vL2UvhTY6*T)T2N{r@)}e8I z#rXJ^I`d&{=+@bpTN!W&%xgYyN%mRb(xLv-KIVowWu_Vc2R>}lgB-yhBi1L6zI==Y zvPYlP7x4h(2%O-z{jHEjTB=vFvbb#44<_s z^$R73Xd+CkeQ6fNM?w2lBB0JLH5m&7?m@-(%vs3DY;(JkGsA-}nu#3^^RP zIl>#+4rEu@F@5&f955%GYgT|;`i^u0?15{T9DD)u0A6h3nu?`P2sa$Y2OUEGW7zYU z@Na{cIOs+M|6(rS4dMaJ0k$onACu1IdNf{e7`Ox9+qd3$!(^Pdk?XdvZtmz)fJ_sn zld+jVSE7l&p*!H8gPl&Qm@DQWk!?Y4Hfs}YqfPXc8R(0~rO~)>o)6ApvVc7UKV!%X z^Xsd`O{8ZU8_5B7A7Z?bhTkL);wi>iBvZ6wGt2;ARbY=WSLu(zF%jArLuVKd*}jGO z1LM#ibinL+oiZH5&dtmx&b~ng>^N{kxT=D?={R_i za7x@l`a)mo2J--JGaLAUj8zGk&@lc@mFyDpC+@yeUziW(BHp2&9P{uGkaNc70nRnE z4Ph48#F#>G1jYiqG0vyH`)WS69Mg=~tFWUC!VGb~!Tf~{eZe=e`Q!WySOK>BW0C_f z7ZJ9|ZV6M+E%Ec6^bc;yHt=WA{n!SMG``@cT=v8h5pYJonQt`+gP1cL8?G~P9pPb< zG90Nk>73?BV0qeh34EDdKaC5&kEVc#MnLcQJ4B;~1w_f!>Ok zC$}}sk!%xf!B-2Ij{^Q8+%P+w5MMF>L;5A2VEW11#~AW9VFUCx+9-LH^a0*x#|@H) zn4beop&#N_h6}>5x&aRxJle;3;4oYR%tIe!Z|q#+3F2PFY0PFb_&YYY+y+hZ5yZbt z#_T-sHn711-(=f}4++;4bASU#-^AJQbC4az46Km-M;eWzlG~+IB|aqig4@xT4ZYCk z4d0)o&=d2kY>t+0lJ6t_U59>{?1`_Ljpe=$9O{YS4#Esiu@L4=oN|LKz#%qtp;Z~TFrVZlY~HNB z!f(^xh%=xgl2c(5=FLNFh`602jG2&^$uMZ%9P*#}9Keq_p8L^U2NchdFCZC_95DyT zHwE95UlOd`kAsH{8VAmU%*aldPXbO6yAfXdD#l5^igEphd;S>2;RrGYuj~+mtMH42{V`&1#1^XO9W*&1FG0M4w6T z=wARHV0#W^M8{bSK{%mtz@{-r=$QFD8V|+v{%N4g|j()@9aU&U{<1v?}jgZ}|O zF%Ix==0gq!@W?nF=NKg}DJG=;XdI9i#TOLIldi~TfM>^a9dJB@Et8%peqpw-VmS*q z0OwHr3)`aDF=BBu{4C<0+}7Y<$OllI49*yn{#l#|y|Xxjc$@4Wyv<^D#0RjuhDHfX`9{@ff8|1NwAqt-Zydp-Pz)q0A61O5IqwC=F+1P*;vH|#EkH$!I0Dnoq z^n|cTvJ)|XiV^Sdg9*#YR0FyrKg|3exC8P~DPGiI1K`-mApXR-H4$x##35wYBzx!p zF&62U@dI*gEw?55Ok4(A#F&9q#wW~wlB~c_#hhT9ww(Sy0^R3&1#XWRex)1lR}S zKmlBae!M*ACSCy_vG@!=8#=`G@Y%@4XdHwonnNB>f)5cdAXh1LOtA-xJ4K2qz~zW@ zFfQniVq)S^maAf%Ii|n|41Wg2#iTRL6z09M$z zSWcMdhItN67=@mPXTm&-dl(+IDyli86JVdk445~=31UEwd@E#$T$%Dt!a2+55Fe1d zGm6s*t8D&^XJB{4B_hTRUIe~jo4`Ju%PlGP8^gb`d~w3VH{4mQNBlwhRZZ9(wd;Y` zkq1C0m^+JYF%I}R%!zy_^S3l+a17)L{-->S_?+Yi{4*}WIJ64%0X-vUx-&k`^@5lQ z_``9Q7ZiA4+yq@B24OLWLHQN6N&b=MNW4TiB|D;PC1Ywf6nDR8>9`@89 zlMf=H6g@@EEk(Vj>Knd>=-lZZp?3(ck+=KBl2qU3*<{t?-)}I>rsqN zHpKEo;t|B_!~vA2u(8wkm_N>^oC`9~<1^9?)dN{vVnthUF~y_Q9!I{D#d_$Eum?S{ z9GBso$q2q2$KVH$vmr*q+$qO{J;S%na_pOkPcROVxRb>JWDCSy#Gxz?WAPaLwLyA> zk0ih9Q;vpsi)@W#l;^}b_m{N*idkVd6mvu@?gH0Q04^uKCqG9VL2)^=b&M7H1nDTpJKE7T`VRhqokPF# z`dl897rcnL0=^aYf$@>uqg~XMa?W7(U8VRNF;)hO|g5<>EvTF z#6Z9U%l~i==1N>ZybOG@{0?nQV8>*?tR4XTJEA2V@*EVt+E8WGWr6p|D}2xf*Mn1$ zgOQKJcpwwR!N{R8Ud)+tLZbp1^5k3c@iE_-^C!hI<6*#LROH|!%e-b;)Cb`Yz;R?_ z71$@?nK+r+MgC2{QI{UET$H#UpWz!A?n)U#9+X1@8|w{fu^G$riQnM&z=Jni@(P9-;s)|NBs=gp z(<8;{q${$2@>^PoIdChhcX7EanJ)+K5WlfH5y=(yO}3NUG)H~r-?Kh47cmsc2k|S* zVJR1cEkNFdU!~7vZxsKG$*-}TfW}V!Fu#HGfy>vL82LQ(Oa2@4+#eF?hjBXW1ILhC z@P*w|jg;*FUfMnRdgRy4&!c|L`T|D4VaRVFKk||JH7r+Vbv2E{G5EVYHzG^}GmsDQ zH`Q|JUIMi#5oe*MMllEJ9pj+7C-dPHyTXsLcsak<0KWxZAdaTkuH;>+4dWQqaR|#9 z)og)%7H2U$Q2n3$0&2ymt5F@Ca$@2|wN+b^&Lh+>S)B*ApUa`J6L3At!zh2xZ5(Z5 z>@+tvXMu7%$cuiLJXwr^+BC&wo{tu~W7vkxcxb~B`#S6ieZ!VWS7bjNd?Efvu`&D& z!#mYBNnVtf1Iy%NqRyG=kH${AqOq}iHqgb6Y{agR+$`k7m?zaAfP0F|DK=nuqvmRCuGaSorr-cE@VEWtedfz3+FQ3Q`^7`&V@~*HV90RUj*h@ z+xR_=nN819svA=5?IY%fOh~V=`#kSQ4H9;T@gfeU7$L7KvG0k$3coov2t!nt0`_dG zr(q1lA!w8Cd!cTO93-D7`PyiD1~=#OVH^cpnZcHjW8~bqA+98h>}AD0zAU|Z!eT%0X?Y#} zW6KT8+rMw?wtmdbdaRuiv6PNh)0Ow@>)f~UM)4ol|R_i^cqiiqdWPQS4|8b7P?SIe5`LI5e)<;76kBdOxt=rtkd)&}}x5%?Nf&D!8h`dWyMOUjIiC$33-farJ?c{YT9*~>r>@#77Z*O*`7}Hw7njzjmljVP z_NSV+hqdY;kU!19@od9i%)q|@MWtS@?0g4{0xH47^6Z~W{q@h(|B2z>bM0?D0r1(r zrGMd7cS2MARx3pGph&S?JN>9gT8&ro`p@y><(l9J`yQ>awvBvufQMDspJyC-dT|)y zcmB~63|~Bd(fHuSMefV^j<0>q{6E#oYn5UV@B=XZR4Z?3&F@&b%r&bY*4DB|Xg)+_ z9Qz?+$Fo3XZ4YM8R|+b4H2VtI__=0Zs}v)B(C&!L z9S|>8;{K`r{NFoV7evx$5vG+`5UXE#^U|gt(NG@yDGs245>(ZXo@P9@>64|_bRMptp)IF5od z#CxZEerXi}Z*k!5V*2|1)r0jZ!~<^>C9)>pKZN!8tYALBFV^0lm^uhK&LEw+XZ!Qm=lKwxbq}C$N z#y4+IP2drZPr3FFeQHABo3(G`aLS(#b}aDU>P15PKy@%I};f0y2wV|b%?fr@R6?%%F`$L^n4d$fL!smzQ=;^l00nxmC!J7(_^Cx|m56yY^ zZHRX@oG9k{jWW$Lf2%`pS-!bp`T7W5|B|2?3T{Y$c(nrilL+IZZa>s|UfdczwfHFn zc*V*mHc;xqm)YSD0gYHMvSJEk(IYPiUp)X{7EK?cgWAu0_Pr()S?GV-4G=}MfE~Je z3)agIUr@PN$&-Lj3Vc=Vmvp{z&jCGJ`I=AUS;Fu4J>SK_W3*o_28oy-84{)5)dRdz2(|h{Jtu*}uP)yhW;44d`mLM)?^bxmgx33TRiVMIij!uljf# z=65>~MEZ)n&X4MVFDtA6*bD>(=jXEi{c`<6qC%wvSxFZ4#w%HrI?O>?=z@gvd&kN~ z`A~KI^`RDLv)2m!)|sE)K2&f1!4koT!rdzYe4iSAml4;>mt4Fv;?isXFKNV$M)6jz z0&+s>iGDAvO7s1POyCD%@m1#!#S5RUJMfavY3`27{v0mMiEt%`YtKw zAt(LvzSGalNo%F2`A-KUMadC&Qz!s=EjpH;-lCCrNhv;jOddZmP=llY(CT;9_ZUXh z%8JiB1}bUke@Oo{#JJ^4IM=tPhOy0)z7UF3?}) z=s(xR7O8VdXxvvf_R4#B?=t?M>STY<8x-mNM+fMqTvd6>Uw8U_=l4mlJoo-q zbKSq3Tzt5A$>p^EWH*0&QEaKckJj{4R>js{0r=;=(X{gZ-rV1d;P(*~wL%AvhFBWs zPeR4#r53nv?QYKr%BSDFZG6FME7Tk~cTd%w;sAFz#aFx_`GiB@Q2gKi?}yVGFL$OY z!2{iXhRsK7{w^FIUxG_~S`?H}530()RB3&@dr9#Aw7i;&kME;~@2}E+%!R1o5BCM~ zIXvF7`R9P8`K|f*Q&hAH=}maWH;B3)zWg2d`Mp{$arUDZ>;BjWH3U(3U-Pfj@P^FK%u^|2qrVP% z%3{C+dOoTDFBt;xA6{H7Gzt|wT5DO_dX)Fi_5bCwI}h?^O#gRA^)OpdUXs6!^0!(1 zAGN~2mh*pw-k<0E8{ZmDu`bjaB46RESg@62xuEkeVgbHd`NqyfNLiMfP3h};~)S1zcam+?UvD7{!A5% z`_=Vf8{c=R@jb+H`M~l>?-NWuCj1BztT^s&*#77c ze2ZURLT~-;ygmUP=Wp~X?$?2QCdz*Xl*(wN2u|L1zIXsu37|(%q106<_9qeI=QYHe z%M$NAf8JZg_Ev8X?L4Tl^kDDRn8WPnD6x!;ABseDM`rYoKG8u2UEW+}i+Jg7DlHc( zzj*&Z2D=MIm;7V*pk~EZM68GX3JY&y+Dgkk-}fGCZPmi_T?J#kk`$dy9|mrMM?mNe z{h(;}B*|w3FUb4x#_gBuzpCYft$)$`SD)8xAr;D}TkuRBd+d~vSBbs<^a`W5BK?P@ zVnU@}gw;j24eoQd`5p1U_QIxG?W@Q;kN?gG7q7C&!>paVM?LNQ5QIH`n`_{Mu`xdf zj6OKhPj2fE)Mc5U{k(%kQIY*@0=1G8K7~tvemcN?L&faF>nrjR{qAR=eoqYdeRvjW zMhVWm@6sF)8=UcY7fVEb?|OtExxT4fCj6^`-Ji7K$y*`yBNYGWb|;5DwmVV6soWo? zTvzvF2|w0o5@-`2fA#63>MN2%f4IHbDWE@;xophlrXB2BDSM-{tjU285dXk3dFkM-kbj;t!8bf4LH%~iu{qC0(kAI)zW zHvCX;_$0-8e(xx>(id`Do?PU3eIaf3^AEm9{H8t>3IAzt7cP%O16+?NI zLf*bPu{)O~u5JzQ!lWPXh8KOtNtWxOTWb7$>%G1vpSl9)Ue>XnD2*x~rYqHe%x&%A z+;}AqInS0_ZXX-n@Ox*yi2Q*Q)$z()99ZN1l@m!N$+0d9z34|ymT!)M5$W=DcwPoZ zZ?Nel^Rc9*d_|NbN0$jPk7JU|wse##n%g?a)1%7Gc0J!VNNB9-H?66hoSy7i{4#Ov zsq2e#J5v4HY!hEUN#??-93b104#C`>)-|)`%ggb~OryT#dyjn)>0+ z$MUdj`~f|dt*VKt^;haxIoY~g(^^>{lhR`OGm<+a^&rpBb|z$|CoLE5x?QPQb~-QT z`Z|WpQ8)M?kFE1dRjmxoi9g7OV^5k$7{q&x9uogxv_rhlt6%9!H9dAcADekH@jZ;R zYs+5r(vAWIJKe;|;^uB}%A|G{i)Q`k1_AAa!nE-^!jr9aB>r%X z0`B0n?Xjt}Kb-^Fi;~L;H&L9Jy||;xpY;Q2hZ0`k7ab=$CM;o_P}|9Li_;?M{0cK@ z9RC_%5+#nMwj10wT`rA#u`_A8x)OUsMGS6=G@hO}s$Si50k*DSJo3gu=t)U+bT*Kkj**TwV>UbDqFQsd= znYAZ3tsAC>wDM>1W)U9(!F2TDRoX2I{FiMXJJwpZ`QX;8*?wr*b>T4dVDuMB2hNzR zOXP1h6|tqdZ7JzSM$5La*~!bGA@7d$jQ*Rs4vQ3(fg7>jstV(B+j{|3yA6?UYNeswd-B@i|G9V>^lq zPYEqqs!zsoH%NMNW37wI09dOH*)g^#89#c&U&AW3(Q z?`SezSLz$D;PdN7gfLUywEfnuCpjJ5r)}wHKWc0?t=KSn6(wygJU383*=O9(=VC1C z8h4OqfuqK=?OL+!o1?w>KF-iDzUu)-QNTA_j-O5pX?E=Pm%g;YG(P%1-|TlrdZKxQ zoiZ7y-Ps`)wdxmhu5|?MHsn3@f3s)iMA=oGR5vS)g`ufjPg*L+i6R6>M~MdkjPFIllDgzu65dSaO)bp9W{26-;aA>YrOmaLmUEn3M&YQDP1!mi@Y#Oil!ewB62wlCHlaNNmt0{|gh_#Y(^?;X9ZAD0fd9t>|y z(QF%nG)8R2rcp3XtuvDY?AfNCyIpA#Ek9ut!U+?pAzq>_SVHPWy_@w)CT09k>uxa5 z&ai$TNY|sXnd_T@U$~ND?8ed4SB8p}HmyU8yA959tTd&;)_9q={rzz~@y(wbmM3Fa5anGT9M{<*mnXD#UEE(##vhTk4d`%@2Wpy32HkyM5r)5>AH(V&b>!(4?jdzmhwqC}sf+_t5 zF-y;XX-veU(VokF!FWOVD37)i&$JEPj*}0Adbs<{yoWPbI^a$%K2GIZ&*!R(+6HgA zZK?ik1l`N$oj>(Qu_rgYkJhJ$>7t?FjnIot%AXsm)b!(J)|d{D%TqjaS`%}d*tea; z1#WChU90P;ddqiapGJT2P4pzUU8^lQ%UMl9lry^;ee9!o84mqzyT4J^Nf3&QaP!f> z8mb!9ZtD5y8tQC{cT!8-dhUy^uE~*O05y}3NOti)Vo|eVO;Mx_g7#t$?7R)C`-P+E!XTB8g_#-^!dkcjG z!-eg51KaONHeP%aY|e-zE9&Sd5!a2;scgUZH~BR#*XGi>VT-n|pW4{0jzztSAtioo zY~5A+z07N!idMB>`iD(Ji>|dT+05wXA22%K=52qy=)0>`=*-@Gzc}&Y<3KdJTm5ix z4%@S#49|yHuk5-(mP(8FaTblNnce+Cv)f6}{bVRty0^^5b}-Aj8=RcAG;hvAvZVJ} zR+#5AyOt9tov7s{zT{bnMMuQw#c zz0cm3a&jl!^tqwk8C4BVsq0F5Iy#HJ!aebG9~omVLq_`Aj3+s)|Tdc&6F@^i!Bd>eCa zO)YJJDr9RDf0zN-bGsa_5D3b~NOJvKeTL#O52#*S{xfGQ5 z@{kto;a@Y9EY6a=3H9^(kLZo$QZpo~ioE`7aq@j|pE}~p>!I64-&l<$H%R*)LRwj| z*V4uDwRk_=+TC?k(q((y1}{_^?Sm>0U(|v-(E{&1cmjM@s)wCstVavAq6*PrZ1hFR zTTlJ;C@W1<8j>miD>!&8KIWi{Uoj<=yFJa+79}(pFBs6fFnp za$qO`-ZGu4Cs}G;7VF7HGN<;Y5As~z+jsx!yqqF5Cp6xBpN&X)zTHfMi+(bk#k4B7 zB`J~I#nG1>|1^xJ&ds-ss2-d!&@WnuYZck8YW=1iE>m+D*qa-#D^9E1QIE4kn;TaY znLqYxM3Z|^A$2Ir{lW^ z-IwjzxQ(G{*EYHjY2xk^>%HA8B|gWbo> zk|MN?t)I2oHRditD48$uc)oUen|YkJkNc$Es>Zh5?N=pt$ZOrvnfsV=##=`ZBuzb< zmfpH!WRV;TlQ2q;ezT=)5gxb4wl%x2%s$yV!{Y0Rc^SE#tOuLu;Kqt7Pt-78*H=j- zZRHG6xKKgH7JZ2O<79dPqZyT~X023p(mT%N=2|$QVgXwBsInvu^u8Ws^Ktx<-KVq% zq|?1lhLvO1vKDjQJP18c>5FpI4bsb4;i&=hEWEGJk&nYzB|Sv#V%OGT=ba=Aybver zqyA~>*W+&73kk~(c8MrSv#=oK1v&~Q!Jyyawe?XD=ciS3FI|TH%~!qM5a_&U7AJQq zlh%DSzE7mu{p_fNe!U*8h_%NNB`8-1+m~>)_ui%9V$W$?x9@Fjyv7?xBumO9ZNjqNSwzNz=~n=^<0Q(_MEKZ1i9#NJ2szFXxV_n42En z+vbLIj@tEpx1~Wm8F1>czfjY(Z)jXqeOVloW>BfQ@~&yhFWW}_y=-ss8Lzaz>nH#& zW2?ganqyThTbNGkE&HvISu*(1)Q6zF#5(Sh_9@wHP-)y=$8~GklOIkm$9K;5zF*Vf z`v&p&tOw`#xIJr%e727I;dsTsSrwQ;v$#e|W+*c+RrT(#Plmca@;m(=#xq{i1;4)S z3}ofxuEXZFYG98OR?T{>MJS7j)IJP$r*vG4Z3Jp^N9^ip=ual=VIAacPWQ^%U#rqQ zzK&zQiCNp{mITgd`w*}4p0j1Nr#hYC(I58KD%Ia_BUO-;%8X0z)9EOkaf2wd8C^>T zU)NLAb$5o`vrofx=>jfVu(o~OY1~`f0?GrG;-{Fa4IY%K&a3PKslBfECwefDn`+n# zh6hO+9;%u@HrKwiaww-fXVcDcKjE^LyiknZ%F$4t7ODPU=6#LsR2uI5-MS%R8;Aa4 zqhcrU#17R|cKgG_rW;%@C{o+}bfa(AJp?1NzkbYD(2KH7TY)IeICng729r~l)j(o+ z-QiK@mF_?vNUFYHsp*2IN700ZYNP@YigxF?yigdUImh=SyBj;H(UVl5S-mb@DQRjA z1k0U3IKS8v%k5U$r}d@L-pHr4(OQT7-ifaaDm~e8rAueogYn3W_6sE5jV-oC_M@e5 z1ygO;lyxCqD%~M!EpE~~+UIkR(>mCcYt=gsXCbzlnJTwfN~Rr9`8=H6$&?;--9UA3 zfEtyw@Zc#smoU)wd+eIlsdfkR#D3BAdwf@~p0qzT=1Am7>SdOPYg2Zu;|&qxtImF? zQ~5ipAOam8g!rB`?$S%lrN@nAyLxVfB^MhUNyZ;(sW#3WsXH4E?_Am%oOnOwC1rDU zBza%;!C|b+0z1x8A7#AK*Xc1kG@jRjHd2 z*QlT93pr2>zQ#$H(S_v<3gXnGwvF3%Ah*Fo?<5Gwge%+Cd>4a@7p~t)tFfnb16-r; zV#V3IxUnK4V8zXnSGm(w(-c=Nv{c+I6BcVbcg;SGXUe^0OZGrz_s2^6sP$%DCA@(u zj~7=@(bI%Qphld*N|owaH>z-G@g;y!e2kCdy{9iU#kDgH)o|-l>BM6_JV{dPXsd^n z((2nU6V(5&gpYT1cs9G6Ini%Xm!e}gnD-p1rLM68{k@l1V9}*hF|&F=!Dp2L>>}9@pPL(SHT6{Vm)Etc1p&OGF^|QT`1#yRy4dKNG;Nw@ zb2>SrLTKLfqZ=3nFMu zK(d}UOu8G}J8jfZk2}d>;`sf8GqE>aYXxp&cVNpI*r%Ge7=>1&)vhl)$Dp;}BYE7~ z$NKERFTEGnKL=X7iYgt0SdpQjCuJ2~M-wZq*^vCs;44aRp!J$Zw>??brn)gcPFkCe zU%6s`ws(0WyEYRRlY!y&k)#9%M9i&`yd9zr*y?t7C)aUDI3%dT%k4w9e(!JqgDl&} zyKd}||BjA{)9ngD(Cm4gaf_-*HN=-Jg$YH3c#n4~?>*i>*RhOOvQ1l_hvUJ-h@0+_ zJL)0bVrz37De$mW*hiEb*O@MKHEt9~9`eVFWn(Kw#h=D!A1jPni0l=u<$lrMKD@*- zpCzr%6-6cZiFS1E1>4@ssupSO`4*uWN~lbceZ4ZsT)(*8N-wnG&xpkhsVT|D{igab4tGcW(RO5iObHh3&Sp_EE) zpKGn|fFI0WlZ-b;!9Gc;s;re^^N^ey{lk*OHObYTgHOuAyq=A)-&bux(3Yd|3(kvd zPf2tc_oU_boY-(#{L5K4R8eA_>3BAHYYyJ(hrClZT+FiJ_K+^gaPh-|?9UpL`^=Q> znspwW;z?OZdX1~}9pfK6s)yFCjo28xmJFrUi*ycNOAIB`qQ*(Q#&P3x)N35JTRX*V zRUR9zf`2~5+vogT?+n_Z+f$w;M-oB~#21ZEXG^;&R(C|5!6E3agyZ!PY>vlDKuL}( z$n%YAd)eFqb$~xV#<5U|vUk$vRH^q4gQ?bN!Jt|p)?TRDOtGc)b({o^gQ8%^x)!{g z{1^ptRG-}K#Z=fFHFYJ2H=U1%a_z$JMsdYDa$RlIzF&pPk=Raste4Q|33jGI~#KUFf{z@`E$J1oY$pzd#xJIA&*DyB;N_B@YEe)ULm@5pjL+ShXG( zAX~lq`m}wPB>h_Fu(Y5tC4}9!c4dc(;0F!JVI`PP?b@tkb|T5KwyVMRXl(pi%u)UD z-Pz4BUUbzU=^S?`JX39+x4V8NHpUX}BFk4-m8SA_ImRYq%=}(CY)095a6TH?#|lYk zUE$kO_3m&pe{d2;kx_(aujjWY2j@myLqXo3CD?@wla+eIW!gBsk3njkE#p#e*lmW# zS!#E5T8SpFlMpq>*vW6(721pUEV>-_`vyPnR~PVft!`WBclAyIuGbOFX#|_OVsmq) zdD?NCW?wj5s{NOb84TEynSb}>PW%?7fV0bW-Eu;yZfef?6+pRe@pV{T%lO z+EOiR&vrtF49pdyzzgcfv?J_p=lAy;^%#sRb7s{+iaR~E%xxr1`t>D^_om=h!}}=z z79(?3FFQr!#Aq?<9pa$n-?#RDoc0Var3wn?#3o#8}^bs@{2tp_dSO|6@SEeAooWvE?9@^(D493DCNF430`mSXt4MiuAv z=XmmTra)c-`%?m1WbenrmAd|_?YK*A*VH{~aktsoPO+G*sjT_|?z&yy<3u!8V-)n{ zi58>@lJqu*EPrOSHPf+2ZAGXzH`c+hha1z5QP6nIQc?2@lCZ1Js>dFWx@6F5Ut^SW zD#VAgpb zgETrqzuB?%y|Z83G2(6|G?w#&6^xeL(VoPov4d4BC%>2T+k3nid%MX`mE{_%E^g?H z)^4n{YI2d0okE)P`*fR1FXG2&0Nk^$OX5O~Q5_BJb7hIFWfY(1^W@g-2KM&m)k-(@ zmX9_a;;XS1Hg9JgqNrQ$tYEoHC);7U*vH(^xUBWLh5e(a$|6Ao7Y!1n3g)g1yj6GK zY70l)?WBAXKNv3iCx=zzbVNU)b%Im8QLLpWq`I$AeY@fo2)CF&Z|}B; zLO{|u^t14g&ipGD_`Q!1Fzh^vTeVlxd8DSx<cyg%;0ejD~c3X+Dfbj{;{8l?R!Nt?N*X(u}-H@;@kMYh~~l& z^)PK3_XS!S^D$bi5F4f=>q6PXq(7W={BD;Q4zlZKO~kr6F~7zx@rLS^`q13r7LQVw zI?~KMXx&M?+q5TqK6k_I3cd?cd5kK-DO7K4SO@-Mpnzw4 z-TCT8?M}9#gZSC1G(WDW!0H|Iu1e-Gxey>9*V(ar>Z5JkPDSL`)|T$@MX)C- z=G$tg$8+n3KknsH!IFhy#zMgvi!so6w1tzGC#`VQ??wHdK7DU_H{4GK4{1c^qPkeI z6_M#Q#I1jEdBIHcl&x&Ls8XIU2ym4qs(dp?m6I3TKTf1h=hOY>GOEK}q)sc1N1JoH zq0ANwdgN*dg+4~lWxm>98@zVJ%qE9$=2vlNij#MXPb;c6@nU0W(1{bhOpo$5me!s7 z#tcr07D^uA^>{C%f%~b5a(DhsjFuaNA{awLS}_{lT?4uF$Fq-g_L0tZY9}bsmyfNQ zt~8XmLi?DKslMV$ztBi@MO%77TiB~|N69+vmKbSiR-TI4;9tX4somILdBe+U8Uzi$ zGgMYBdC;vsED?S6J|gAsC?NaRvEeTRUws+bCw0k>+>Z#1sNI9I?@io&%W-D5Qq^c( z2s$;alElh(&*EYuX{b{jBQ8CtS}b+PTUnO}^C4aH^63gGRkE}?x5cTE9a%gqcZQRy zc0z9Mzioz^FN%MB$`V-;Fvxv+XyEs(&;NQ)ETQ`0Z5ra2`{eM=Z#
ipevRd0vO z_545f4No=Ph+q_2=sR z=j!}28hovLJy+*HSLf$(nf_dz|6HB_T%F%~uFik1&VR1XN4)r4o&Q{&j~eN7b^dd8 z{&RKyb9Mf6bw2W>=j#0D>ip;G{O9WY=j#0D>in{X`dpo#-()WCtUg!ge_m&@dallY zuFik1&Swioid8hV=j#0D>ip;G{O9U?TAT1(ouA)>#lo}a>ip;G{O9WY=j#0O$)M-z z{O9WY=j#0D>ip;G{C}O*`6B;~7xE?T#ryNW+DtAeHLlBD>$1EX{)*pjR=VFbB0L%vilU`DnK?E&D|0GVWK;~i(P=T; z?%lSHa~#neiB&7YtAp_}18Xb^eO0^T9NwKx1$);N2P5NVD)jUX9wNqzo~^Noc5Dv6 zeRC{bh$Z@CGs7{#98W6f=WeLq>0A9yf04K|@E(YMx@maAu)5SI{;IuA7h^75=~Ho~ zG$XubI2?pdwAQt{GOA{H^L2f(+n$$d9GRKHJN%^57r3&W3!dk_t@*`asi!rAU+fl9 z(!@(nL}z^N3`nl+?RhbN=vQxXt|CWEy_Jz(aD56nYzCta9y^_gy=Ws^cZtcu+L?(J zig(EMVrE*_8*BF-z07JP-FlHn>JJJdq-Dw-o9FI8da_8nH12zy`)Y@yp*Po}9*m)jr0+TQr8d z$oKIgW{jtb@!jdwT1@AWe{(V){W&7$gE`|k-ou0I2y6M+Rogc+%z13yaE{Q6?5wY+ zcPrB{Uai_2iK!KJA`@?!#b=CJ!g&VH15WQ8jkSt*`(%(aSd>9+Lb#~@f1{?zO8a<~@_F3T4q5ji8 z=7u@pv9|y?!0U2U4{`*5j98yM`tmUr$R2%CU&I5DBXEM>k`LWxeX{Wg*G8o40&A+1 zkY1o$^kdxmBaYf8JrOoV(jjz1IDrhs8+gE{-{{9Dj6**@#&t6@512wbg-zhQyP4b4 z9qkd`Ab+%jv0ITtHd2hyz!)c_Gkn&n)GzD^x*@FNby?~J$1zu$6XEzSf}8MSR%Tze z5A584UnUbPBbj)2=#gxb2R^7@a31LsI916ufkp5o^onygn3DutIlwh1fCG8h{~dBd zn`Y70^T4Vz#L%P68bUeT&_ps z1&4t<@VyP}ZkUYoHgete)y*A!3Xo~SbTT#*=n8KEnk3L2@Xx_cCsoW9bCAfkAUB(} ziMGiepa%ne(YQ1k7tZs+IZPI?XW(ZHd0~EimAHxYOk*QC!0tngH`4H%>RHcAkMx)2JAR+L%6DfyXiQ1k#I`fLbijx)D7kV+-5fL0~xClFri`mn=08Q z=1<&xr@k;B%tgFIKRM>5wMpkRM;bHv53*CH7s3v425g0Rknlqo zAWRXTV~p@Wk;G$M%)X0xlOM-8wF>lB#5}pJVUA>*XbZktz%Gcmrj-VkmL()M_)GdLZ3H$f0jZ|%&)RJTDnQTkNA^A-yGQ-><_j= zm?Pa8I4)uS6c@p!$HW7W^Dbo=pjZT$&8!S@PP9RMK>gTcPkH}$I^U=G1pS$0GZ-tl z5b-awsXMO4*vU?y1GEoZlRYpWZ@~8fJNOR%tHd1HI%H`iG+we5T#Fdlr#6_ZO=dI9 z2O{R(kS#%9zzd5Jm|Y7v&iIpj6WXuBo=MgMd=8B<#S6sfd+LYDp7@&CSnlh_&L)s~9KwD#rC2@(JX-s;oW86tO()knx8A3`3uG1vZY~9^)5ar|>%x;sL@b=0?6w zU7ay5ia!&?=UE|d$d$e`V51(|*ueiY`Be~m5zdI82}h7Icx8teT!mjG?2i$9Ber1i zFpGu=F#qO* z!-&_AgOPur7%^ftfN@jo41daeZY~S5A^J>uNB;uw0NZmQBRbCFI>HH!12&C0LdVSK z(Re7PC;yWBKGG%OlID+V{3?E>| z{3po@{8Y>dwn;u&H4r0HObb0TE+-r^yX5-$`~|XqYKz$e*){P2*$ngV;7P>8G-hCi zWG5otz_=!`Uy4atj6^j9^hNm+#!azQgg(bUeHI{Nig$4R2C|Ug!|*-hUy8|qclZ_9 zK8=y&1=)Lrj94CkK1mi7clrXbha8z?g*XAV2gHE_xD5SxdCpC|0zP8#8GJT$i0k3A zk&Dqd2vanNJe~v}B3?kQQs|gs4;FWd6jOl95$9lB&>zLb#G@=%#W-_Jfe#q|42p|M zXP7IvjQl@wB*m%Zi;-L8ID|~HA}&YXoVNk2uyL`RFwYJ19GWl+JrB=>c^3CDJmS@r z8S$9|?6a5w^JX|f49Jmhg)EUPQ{G89XZalEsT9XioK9F}^JhE*yCW_UF>df8@D1Ap z_UT-_bB=r@{2R*`Cp>(^oyB^@AEaN^gxyiQ9(Wyj0Ca-6v)C5nfRDqR$agY-OJfGd zK%U@#%HxR7Nq)dT;}VQRt3V&nGjgUo)T z^yt8c!q&-0fQzXw)Ww*;@)1iACW(tFPXq2@PyI3ZAd;;InUdd{0Pj>2LVU<_u?gfz ze8$F(F{AFr{DyfaABiy{uO`1hz7+M2F~zVR@iEyD%M*!55U&#lP@clZPTynxIG=JZ z$UKkFNHmCu`xhy9{NBHjN!`{iQ|cH2;1Nyp2b0^%K*Ps z%$ei{ZUeWnyk|o84t&DL6K05`=Z6HxFn@46)iny8ELon*YHR||WBGV~4*8o!gnr;3 zz~#jE!yV=!3Xc1NP`UmIIN#DR$2}oqSA&7zlV^`5(@~T!{;amw``~-=U2O z?3nDA)dPTkN3?`Po`XVP*x(R#S>QdwHWSPTZE!s}6*(CBIE)7}K^%-68so*BDJL{4 zkReaLB_AL2ojHF}95WsUOh!cxPO{8vmPLIK{s1^38>_%R3D3mI)GqRG`i;8uh~=Wh z{rC*uz;IW}5b~fL64+R8P>ao2o=^M+zX#sirN}*49N@#gSU-dvk~g?Lb6D)-Q%#iG z#&6V8pYA%;$_GJV*%GgF37pTDTF~*OQ3j@?1|!T z$Q(F^EiyT026&V5-U-FeRCj>Qftzz5z^!-SdT;^5J>>*5s+~?KUnGtJe~|wtyWddE z2adtqfj{PVDaT}S3}J&}c{Vq)d*sy=f5Hz_otpU{78_8Ef@^aBj`?6rkSoiP$nFby z!!JT#Xb1CSdZU^+IFI?#4cR5~666&OGsF$#cSv^Nai&L#(@9rk|Kzu{5_8~IR`23+ zSu$S^+#!Brbs~~0?3-*Sw`q?0%)e)SWG-STk`Llnmcvpm2wQ->3%^RA$=)db8IxaQ zIRTBG`eA+p=L46oH8Jve=$HIA=7~*6p&!QSun!zVZowCJPc>4q|9ffoYmJpQ|t;q#^UAtUIY9Vc!4;YV!M)esWyybRL3DKXH>HV_F0_8@Idu{@(ZXHqpn7E zaLS2^7u8m6Nji^EzhreD*nTdD!cM^TEDxjnJ-2bRjj_|**qjB*?I17uUGiix25Qq3 zmw7%~=#F659h2?%6;WJF*eGLUOZ^4`ZHGe*o?&E~nUl;gRk|)RwG<$@2XS*r}3FMtc-Xlbw() z6L%s8rnr##kg{&ZVlJG^bWd#qFE|%AjoKhEL4FaKV{PO26fa_hP0v!Q8&d4;Bj$!o zNUyN_Jnu&h5_X62A`Ye)A+Ia3?}@((zd1GtLsXXn_H3%BVGP6}Xp`=Hp>B*EB%dew z+Gu(PH|O$U90gmM!IqF?_v|IRxRG#mxik(ph5DyQ2ACZZjT4+&<~R0FqdEalisI? zclAGS*gtRBKX2GSZ`i|mK5y95qLk+idjtk$lK#A5|GZ)UykY;mVNZ#{^M*YY98nBl zMU&?ZdnD=48}=xSJ#W~fu<*QL&u*MPZ`eO?*gtRB=Y>}k`JOlIsSu5uw$B^(&l~n^ zLFMy?{qu%B-2kGSEYBPEw3w1DdU)Qjf8MZv-mrh(uz%jLf8MZP?~9w`|AhVTgj#VE zzPwv67F){w)o<0;i-iK$x2^9k=k+PSfB&%kU9!GGdUBB9DRpccOHbA(r+Bk27f9^m zJ?)D35^lF24!Ols@*d=mhTnG1D<7ZVg{#$;E>43mS&t5`M|<$4C-;D3r}(m8tyeg} zQS(@mF$y#KzCvSQpOZUX&BpY&WCKs^bLT->GX%KH1_@w{$diY9H572l-L7|3b6KePpGBFe$CS*4?Qt^*amB2NzY$ori zFxUz@XKPj-G{x5&{DY5bhUHTBPnYm)We-6wx{58)?KR-%!6|V8EB-cWBe6?Qc=|Ovyiue~?8U+D zJR-smHub8T_crxNdcUr<*SDGEJ$}KB7}0+1vA3$gIB{Az^z{keAV0*I;}7f``;|5; z|3ca7}M&v2POUy_I%a!uj}*2S|Kk$N$=eq|F5IB2I=R6iq=^(AMf3*_IC6!_F3P;WDX0!l1!K;eRA-LA`<#rCF3BXWFLs6=))&N{cVk%&IM6y+}&;gAe{ouU+tVN#ITQ>=>kFD_B&+%~OOdqT&=SVm>#HQG{ z`jhtl?F_8K@gLosfJMQd!%4ncd$q*JC`I_GA!V~Lum=@>(%GN&YOc4n*|ej1=(*c6 zv-qPg<~r>0VlE||%Kc%=b#*_c_k~-dNuW)9{MDzAs;@{6{o(dzr-1%Y=CU!Ho023u zTYQ{#>{e$Ij%iPVwXJP6dpZ>NdJmS%cAuwBeD1z;J}a{Cv;(}BM@_}~uq)pc-_T`= zc6J}+=TT^f!=Z@O5_T;8@!_<&g z{w&@s;^&Uqx&L_XsO>&?)PC-${oGNz|J+ggxuf=TN9~(>e(tFK+)=yLaGyJBKX=rA z?x_9TQJac!rtRo5)%T=wXTZ*;22n5+CJ`+cB>lOa<^ZV+##=ZM`unS7EKh@zIf~C zfuyNN)6!daj4YC4VG>5^(Qmetttv_Fv2D%nE3-dC>5vBe+IxqWCIn!=Kq0*};k@&v87$?(9C{3R`YCm_>KJJqCDcNjPzqY-O>(;a<+bU3| zGnf?!)z8`9_iK7(Y!Bk`Sr5+faeLMj`D`8a!}01IHHS8Uc)UhRW>Ty#RrT(#Plmca z@;m(=#xq{i1;4)S3}ofxuEXZFYT#{LZK`I;bbVgIXef(`)IJP$r*vG4Z3Fvebi}Tn zhW=!-9@g!(WKQ?W+Fz^EJid-&zG+oj_PHg=_N?urw61t;FnMdZ8tti0XL$67y|qg9 z_uEJnB&9Oro;zwkchoLX_uNtYoee*C)PC-$z0oyk+BD7PbaM97iFwnH&mFZ7Seeio z$=hLDKeoEv-N|*_5e^Be@N(M}&a;Rr?;Q?qRF>`IT{m_%R-YUbr`yFoN6nts8Mm;^ zPp%=plsaf26cOUt8hlrIb&vPYb=iw0+qC6*I37%lxal6bqaI?Rh}`BjQsCjGmLGk# zah>TxSK~%;SzkPU#Wj;$Ytv%o3Rg*?x7G>Q0+)?|vqxN%0?dOi#&mFbjG8l04b4TrdSMT_2`Bdr8YMnV% zEH-n+=H^QCwBt6-zHqoy`!D{Q{9c&>lKfWb0^vvZC*WF@MAF7v~qH$uh81)Wu(DLtFdp}Nl1`0bWD=3xugd0A0 z)PC-${oGMI9Ky)3m*;aw?dOi#&mFa2OQmoz_}6fc!N!)4{!NUW+G#AMzI<%ebfuxh z6?)_Ldw+F_7aK!Yw51ocg}o|wl&sTkiP+~58-M1fmhGz4ZtSnT;bk=qf`;E2Dyx<} z=vE~Sw?6V)38f$0hzm_xrqR;3jtzeq`0C5ZKB-H7M=w{4cYkc17gAeR$lnQpce$*608K?41X= zV^z8L1r-pv2p2>}L_k3i=djb03ktg=J3D>nq?|-$OglS$r=Ao96hT2nP$?g90YRD- z3#eQvieLe0_i~jY2+~o?1w=)K@Bi+S$teR!nHlxWGtZnk$*ya?>uqbT_qT~cwLQir z#;Fzh+9nHZT*AgL?Ck}OnrYQ@*w(vrtISK^3tTVF({AFW>-AscPGBc4^*lR)pMS3n zJNcT$8+N_7r=((8_rF~GHn3N5vI%jiB)rs}8!sKHKEVV4 zQ!G^^v0Trki)v9|DnMk6mmtt}oWxiH!X^y7lcebA@d|C}ENI@;jOG?H%Y!!IRsF)ts0d%ed;NH($?cJ%WJVwLbocrUb*cNDIL0qO#P zW31I-CD;wV2LzRXkAPzkEMwjE0Y0murwK|uCdde2AYOu+;#~purFX{t0hS5~09^pS zmC+%h8q_oc_#t2y<1pY?I3Iu#oCiP|K{)^iBe)}gWT@rjh$aEyAo$*x#W@A^9e|#R z0NN343m~rsWt#Lu824uZ8mE3F08Pe(|2hEkLz@uv2hW^Hg>|^kVmV4z)=8~21~@mF z0l);naYc3K)4p+T#I>@Je(nEb79+w^E?B$0qQ5>MInJbqgPiC!KJTUSjXF~pGnL|8 zq+X~;2AOoEp0GMn1#1$34#gu?W&Et+XAM86Mq7feF+JJhSlOCy&vS!ncg{+ACh_Xkj0nViXl&JjyngiG;GKNg*pfUm{L5^5J!3Z1< z_%Oj|04JJ^DS&e{aF&ok|4z;iYoH$;kOY7&>Xibr1I#-Cp#V_?OmSUbU`!D>iJ&(G zl|gMPf~P=sA@ek539tq5jdzsL#{>XjntT|t0EyAyMKmTMX9Q;sGa1)M-D}7{LAU7o05v9LdqM*P0iaIc z*vXg=)0kiYErTAzn1Y@m_$kH&U}YF9i}VP=Y4D5$wbFr-4zwfA_rm|hITPI!rZI)K zmoX-$`e95EqcGratQGo{0Kt$04fO7a63}y^8v%M+E=M`&=teF*>_Ttba$=nUbVeIH zA*(h{bPLclMvX^mrbc;QsT!lHH7((Jlv&AG>r4rw3Q`EEsiD;|O2ij$Ie@VuWRs39 z*E9}BcXFVk>43O|L_?!N`l~)jbQsbJY&^9tFj!G&3`iw@)_}MIPMpxDiGn@BGr-CM z;{#NWA(80m)O1T!@}jSr8<{Opc)9GL1}p~P8BkfsAHfZ67;^*27XW>dqQT5yy8s-BGlfK zz);Y4XeX_2y<{)(PS6vHFxm3~>0=s)cqcqRFzW;fBrqm)8{V1p^*BL4HN8vyLB0y! z8&lH&3F1EFYXG87Fh%kUG~heFBX}L@p0NgMr~bxFCZL~_^J$cj-#}0_{XSZ|19S>q zOV=S^K!Y=rF4Lj9TE99pTdslK2>MCyL;g!`ltNIB`!JdG%(&2tpl{F zHmV}H&g6UOB0_{wF9MebtAG)VQK=ylNo$k$MqV#jN5V$}rB)l{tb#-yH8AuvDl?`>IyoOB2i_XIU-GC4Aq{L~1W;PdJB7EY z-JhOcv)+XL2FjZ(G479J$dXcpPN3k>KCn)B7e|B{kYw8RdlG83W_^y)IC=#}Bc2g> zKSA?e31d2rc*D>(>ok^RJrPMSf`b?q` z4oqXw(1a|gK-4EdI%}hiA`{J{o&M}O+FtAXskSHNA9?QjbCMSf`5=u6uYua$lpWyK zNh9me2|EWb+>(KZt;yaHR%G4x31Gf8JU7lKgr+7Nka2iCT0a3`3eO7~2U>=(i9iD; zG}%nrbuR`8L_EW^XI-2Nbb~y3!}Weqh|I%d?Pt@zQPA)fvSO-HAnhs|+O}34Wzz8$ zJXuB(>os^6#<|?&TzDRkCHM+e;K+gX#P9IP@Vlm2V9^j+mIMi&wigx?ayupuyfy*? z588t~#-s)3Gz$%1EtgL9i@sDpHy0n{UY3G2HA-s!NGg`Er^fJVbgeNiik<<1_VPBi zm$7=fc^mG~EUGxOKG7}uQKF}jZAR)^?5v~s77*oD^VZ=)p~_UGv5c`I!AhBa!t!_5 zq7b94;+61^oO(rI7Vu7w1AljX@30W2B z8EVIW{5GjiG``s;gsdbzJS8WDtPCTfn6wGJXd0^&ndx!@dB^yI1^j1d6U(wTvB4g} zn~>MZ33M4n4$*aEhoS+`GSFqEkyA>X+9^_b zM!~4$sDGF%)TjVilL{~zU+DS373zh(j2Tp`+hL@q%?zs3y;3bjn|;P^6fZ1K(=So^ z&G>lZR50=^uM4&;jan*JwAyNXw;S)*OI@_^%zLO#R5{n8Dok1jOeM)fxqPV}Q@fR3 zMQ!5yyxQxj4IEIXFStohO;zUGgww0#US4#Jqs#tPGBlN%m8%mTTWNR+|}uV&z)gmE1NKa}8v15eXb2#Ao#kP(n1Ew~TdoAulq z0oRK6s~S|FtOE%t%yR3AvZ9!>7kLk{$1AZ)rdlF6;H+a@6gZcjHdf(SNY;!s zX^GmHhJ+QN0BQCMHhEaGYAFDxWX(3R)t)C%iP$rVZXm{c#<)Hb%2W3heN2w$kQDPl zaWfU1KR`Q>3lTxE1GiUmOSQHDiI}MNaFZzMN~CI{?7}3#ErxrFAQj0oO7u`7gvh;P zA5sEDa1JB%*W7-DxB(D?HDWmhL!>@I1z!bVt^zO(8D9lpQfP{C0b2a*Dgbj8fVm35 zTm@jR0x*$2L%6vLz(iUCVJHi~YZZVA|9lmIDHOX%HBzk8gMY529Wm)iDRr78fZtu9 zG%=-NDP3zp%8KF_qyQC?Bc#Xy)@{HC*Cu|n2@y({;yhLf^DLIENJxdT4vakTBOer->m{LF{M)L zA-#*VB&Hi^5((+M5aa+;C+G)>VmHLGkPW1aK4rb z@6qHRQhEfA)8jgt))kVtE^d!kl7bObrV) z^qoW#k&qosdQzN@$wrDPArp`pk~8XGsfTAotnH;Z5{nP;J7gRaxC*7$;1AI61H_;T z;!Qg(z|h}`CN5V2nCL_F_bLDr%NdZTSOs8edK6Qu(3ccb4mBN3V}_>k5;AlK#yl4N z6);{g-e^G_=?aW@(kC=-k*^_rGWL!#BZiiEkd44pDuc0#yqKn=Y4VzZP9R;3X|6<# zA>D&18`u)c7Zm^^)h5uPA1Eh;g$gt=uPwx*#c?zpOEw01H_dK@N0AEjhqgeD>;f$` z4C5NwR4Oelq+Am%>L49uM~a3dqVg2jNe zK%)kmqAlcty~$Q_AJ_-zYSQ;r_$f5~51B!~kxs|`u)qU-1RFu}#6(0aVxT^yd+0y) zrL&rRkWWBzXTlrF7okO_xGomdU>e$r_rvqxI5A99{8)lR_n-x?G_6kGtpYH^6&h=1 z8?FK{wME46d7#^%uV}F_7H86;A8lbS76xj54Ed$B2#Uo|n)<_nUY`8Vf(8By`QDI2 z_|VW|fgZ2W7$83#OBxdJndrL&Eh~ZFCXnu>dd+TFfzlMV4EI1B6Jr=fV ze!M=W8M-I#f$z0@jB{9ig7+JT(^#IN`-#xOv|w0)UcfO`TEao&mgWH@{aVnu6q9Ms z3cI2EpQPKdplZZG2h)8eJU4ukW~+xK2AWQXJ-}y(Ev&}4r==1!?ne}VAP&)de&}y) zVQRgrJvVZrXb)PPiT1-nDnnObX(7!o(DTrOHd<<=#YWoF3xO7AX-j#ieehg#O-*OP z4#vBy0L)bY<|+X5l>}f~7~729fWDY>_Sp4gPQy=e*=#`X8PuZk^qf7T&ya7V33~(A zImPVyyMR#ZhdUK>-Ll57H0e;wxlt2T)Qa3F^jc@6orJl5?UxK@|Fv`In{}`$?Kf(B zyT+X~4vv*N6>RL&mUhrd>s`LF2sULRA*W$ePMfi&VgtgaUIJwbUW;Ng5lx}l{3T>)=Ic{J#cLQ#y8(7WVs(;@b z*mxYTxAhBuZ5zL=MXeqaUS_sRDF~G&%y`p`ZVrre5KmHv5vDnbxH&PhH?& z#67iUX$-}ZiqY1=D|z>1Q0_yM&E@|ZNAsmRaJtWPWMiluX2o7AT^5}l0|mEem!`8$ zTWY(lToZR)r^lS_zDi#d)Vhl0UGzvDCI4=#151?9lg#vg_SWmJMc&qi#sx~;AmQ*9a& zy)v=QVztg#4T$R|WC0DEJS9^T=-xkl=RvzK=UJ|YD(jeV`8Ju(b1{Zxp6a}1efF=h zu5fB2>FT!a)ot6f@rE)pl+{2+VRhT~>b7ld+xP0W?bU7DwEcg2tK{moZF2+r>b7lj z+x6r6lD72Z|sCQ}nR=_kbP1thf zUIX((0!D$Io>Ev4FncSdjo52)isnaa=*4l8ye5GEW+Wak81z>0JIgT$@a7FXG z4Y-fr=CHYE+(z#k0-;TREU%MmGme^0s$1PN#)0OPpkhw1X*Fq|e?IUf&9k|fTO=i!re~-M6%0#GHWMNWHtf*6nOr#bhZaMVEE+ zYRzMFCYz?!C$rq^)<@VG_l!B2K9L27Iew?4!D2HcWLg@`&o+`I7`o@Vl)kBdUus#7 zVrf3t*QV;%V*|P^-Kfl0hO}--H}kCdDJcQt5rkr{m&1&{{;Sp3YATr0ucWwWe82=n zMWv$ek=bO~n;+v(jjGZa79+ z3SmejQdNxBxJXcD=_#hq&1a;jKg+V?x$%he4c*?X8W-$ihUFHRrB5k#t*kP1XQ=E) z!pnEu8HE#_-mJUelLR-%@!GBE>L%%>6H%n_9Ph>Ds{>zXVIX0v9A6A^4Ievb7>}}! zBcokEoRkCoJl@S0AK8*owGr=&X@-sD3ML+P(bYpyFV_j$m|VZ4Dm~uquvP{|K|-5P zGvJ15k&810nf2wOsO)2HW=h$r63bLw*w}>+O4wzQWS3K0+YHAIGZLOiw-#m=%z_{T%MF$)xHo3(gzz805i3rxBZQsm{#@Nq_9xIsrwtMkd6oXRQo zIk^gkY`Uf{ENhuQBQOwIAKNO=$tjmN>{GaWq8J`jqY@cV_GWY~_ z>3xaeXh5yWl=7{->_uBbknty`pCRH+b8@$!N_FUZUs(vwvtxO_kJ9#BJJL;Ys?Sxf z6iuq_lMAR%mf^U3qL9hYM^zn=AP( zmI&%`n(sP!2DE(*zi^32citgb;^Cwc8^;p`Hp${477?w7t)9}7{FMk(h+0CdQ&e*W zg*BRQ>Pdq#SE$bWt7Uo6#{>FkkZ28T9)s(YC4VN^%Zyb|Bi0$S{;_HgyV|*O8Rca4 z-k3aLNA?C-YipreW?C7)oCxz8-&eb$-eYczdyM)VV;kD-$#6&wh}`o2L2Fl7EHkgR zlOkT2MIj|QGZrI^x5{=WVm5U}Rb~CT@~{?U25LynBz-xNR||38ULNvnBGs)4qNB-_ zQ4nYeDFb)n2cnSmd%NL*FjtFNsx~n#2_Q52Y=ugx*D!GzW{Y%M;vB06QmsnW!DeP_ z3zhgnrjc1TrrIr(LgW^zwX{Du^ad*fi7hH=wU$*|Wm`BS#uwP7Oz~km5Gir8wLAsXtMo`oBk~e}Yc*o1eNRW>9q(#wMOR6#!%L%!%==Zl> zp=DQY2j>fXxmRsiYngZ|B!ztgUq0buopDPdj?LW5dm&dvF9cd?k$1W5Qa+b!waeWV z-Z)sUqoJ%(f6ftVtNC0nJg0>4yaRLCPez~%#mS)(t2T9>cht)n6rr+r8YQnb22@|e zS!C_W7FuqJrRju8@}m8KXO?w)Ypf_TokYyeF>J=Vq}S9zwjh+d>&AE*>Sr-0Cc)JQ zQPCOo&(@V97al}2%Pd=XJ}J~BkJArVoXclwa!|Ba@`{r1%r=)$>&3p3D?p5@Wvj=R z0gF~*)9qk1udp?|#u8$ATM0)4vtrL%h&A|RObDi1s@>~Y;hmiUn|BzuQqAh2oDL`X zh9xr)!xnc|@(N6d&GYSEOBQ?c&V;8ofE}NfUA0k&L_?~h*q`N9D3OL>@A;Q%VP^m} z?T10hDp%bNR*9-1XloQ$UE)W9wwx&PQQUSNHic@i{oI} zuh?A~=(tWU2LO++l2w+rgxbuApw#9wOeq#0dhB7pWMdQXt+dxLw3#!I=E|rnCUk>A zHd)OtlTkEIrN_GM{c0Urf(=ze!^lLLm>!ceSi^e;;;vDQU1;;QIk6T~3{zr^b*Wtt zi;~d5v$lN9FxFL7-owV~(tOIVM6(K1McI?a=Azirx0WhNSkt*kHy7=srA!|VZQGvj z1{zCRQR-v@(GGf~EQcc;QHdIMHfxwgb9_|ltj}MyDf3(=91%jP$RJdnW1znlwTe5G zg3sLJxthn`WN`Zoo9orWy<(D4SxM+xn!&}E&%=-HoKT#Lb-O6*H_y$=oG4(yx~mB5 zJh7>Gcl`D6c@@H|+KXH=oDza_sfbi=@spu!^o%vY=Q?Vnmu~o&!oWW}r$iMuTVw|o zwkP*gv4kCOu-Hm+nw{vWDAks`9UNA~B2(Z8erYZu}WKy`HgJRZls8KpznkyBHSd z*>cZYi-(1FlN;nE_y{#(WaeEFopyowT$ z8%j!8$CbK2TH{N-a>bsl)Rx*;Nf64; zq0nvTBq5y&L28)NoMH!WgR5pxk=K`8D#kiI)RPX*kz$=HauKJ?-ou zSafjRvA|u|36^ zh>Qezahv1mybgIj+N!y|)dgpr#?0(cY2oP<*$P^BHkvCIK|NV){nTDTZO80Vur9Xx znO40k@$qg;6)lZmc|2gVi~THmGS7IV#5}>kpd0PQIZJRj&dsv+yen7kWJ>IuCtabD zq&rY(Q2cf0DtbKA@MY9qpz1616t)oQMG7|7pGZU&>mGM*6ilfDr_U?ID{N|ZByjTB zf?g49)PTO`i||G4Z%b;uO! z>a~2OOt8{zmFtC2oH|Yi$(c_L+W3g%Cd$O+z^p@Jc$dfzI_*rh$unrjrIvcw94Mnk zF4G71Wn9g$ypT&IBeq;2HOnrKJYB(3MJUZOWtnGY6LZN_2k~T8kRd~2*4O3&lU6h} zk{TQirRXpe_pv$aoFp-djCb~>BHkFbQ21)0d<72A} z(afk?s1|rOB~@p0y;h~oTfI4Q%0s$KtG9ut!bX?~TS(G)<9U~&6f0vDeb4Bd$< zSQk@qeY~U9@JglGtY8sr)mB1^+Sx)Wr3~5}?+LTj!tCrw>2WN*@4Bnlqh)(4J%}>? zp+Xy@jMs4MqG232=qHD4y2|_cj>1;+F(Dt5cnDs8kP+5Gr$vzuS$+akyXMW@# zDGQnTP8FB4xfffXnw{c&+0!fc#g3H8rW~A~WqWKc+g8|&(uu37%utC)Rad!#2VLO` zsuN6mhd1hxEBQef`jdBJ+B<{IGyzQy0c^w{+93l(uLE%3>FB~n*7r$5Tc>4YDT<{h`@;_gnX$Ci|W z!@^|~)nd}F*uAn;u+`>rKL1jms8SFHDl0@n-tf?EZF<;D$T>HZ-6^&cXR+a#T6++d zO%-YMhBHtL6oztNwJ=>NC@`^wj?>Sy#yQ?0TD=nMR%4B&ktKO(HCgO>(+euow^i8i zyqcIRWkR(%b>MF|dw!oomAPQl!RF#tZ*aEDER8tTdPXd>UZEY1h`te?tjejjWTiT2 zD;^(Thcn<$$dUA@7>?3dU+iUCiT%upuodYo;ZnGd_t1j{T z)n2L(jV4*f&PY)ZlA(5*^OS5YWGtZ83zS*1$Hb^u9?vDzrP@73=#ETqAsM%{$G%8* zP7Jqfp#mSZbJbvZ&`4ClkLZ&wUzERgrhU>jWArmlJt#c6Qd;S z9Cb57Eb4Mqi=?#^RSHlRGAOdaI2T(N*a=RRMS0}+CQ73^bPqdb3#nc>>2ggh$8D^wN9BQa!(@zlu`=?9YfPr? zT^NmIHl`+HjL?o|g~1>yc^QW=pKbA;vc;L>8lEsWUw69}x9w2vqS(SnL;^6cMl70L z%S0Rz*_KMVeDbWD#T44Za!ak43xK1gzbx| ziLO}gs_lB$%X8gGTWqx#@Zee&csh<_dugPZ^MWu^N*zfkwXF?tR8iAbuV_mC zSzY1)dEJJ@h2&0rkk3I}yycRth#B|D$7IoYP^I`|w2uru$Wfk`Z1@2^@G6OLNnwi_iYdF}RG{5VGGiYHc44TP@ghHMM>2krDR&1`Y`mc~N^TYf zexli$fO^$Exgrd>Tr}RQ16HfQAl%80Ro*enWzEWI9i6A@%1c{HJ24z%-qa$~%i_TncSNe$K=Vg*REu#nHO zJ&)H7XE{)|b?4<|Bp0;!JEgwcU*kFhx35%*hB{VD%o5@}@tt$1UIp#^!qRyjL8lBVnPyHaqS)1aj2slev>hWT2g! zDa7I93z=qIr1W>y0uMCavZC@#0q6(J_gFVPpU z7^veh5=V_TD8Pap7Di-;N_&334>6ScWu7ffn#H&eavac~Vi`#sF~snpQ7~K=d5hO8 zvrLyo&oGvv**B(KBYz?SF3n3;C&Tt<`&!Jv!%uB{>#AQabAITdMJ))CsT5_9$#Zx( zHJeic1BY7h2Fo}{Y^Z#up2Nh79S$odOD643XQc)>E9)KDYx$n$YiCAYxs^!sBcII5 zF@c>PAI?xZ$XA9?mF<2`9N$LxF-9rx=!%j922Jgk5yUU`kJV;D%zF78Xhz6} zNBWk4nh+};mi4t|p;hF4QF%Uz{Qm;UhwhyjUcUBr;9`lxYLL9+TzM{(VSPd=F&7zN zOE89u`C_K#R0dgZ)R~bIavXPvkfQMCY`l#8L>m8xaFw70pgMFJp6F*OP(fiCE>8N% zs>dFqlqE4M`H#2 zl=GI$u!_>0YWLZfAKM>fYoeD;dBR~&D%nMbFgz67IW_N<6?L&3W7d>t*J>p0bvJ#v zp}=L?u_wvM)vWo%GcLie&-S6BosecuKxFwq=tssSrhwQf|Rx z_RVtjIP!qLV6f52WpcCl2+M}%hfAXMQGkIkaGcy8iBbdc?`*r52?oI?@PrD1gpaaF zPWYEv2vf1c#had~9EGe(*M&XktQ?lPxk@6IlvGcy&9ub|O|kiz8e1-)=-g5%mlo2Z zE$C%t)oKCpV?QE$9T{Fpsa*wIUVZTFd9N?G!u^_+zRFq0Jzrn}NhR0`f3+Jb*F*C~ zXVvSUtHU)z3RP=b({B#dsteV4eKwS2<>-)Ulra4f6vV7BpJ^bYv*=isDb$(%3fIV4s^+1Db z$Qj$L$B#*7tJj@{iu1dbklMp@K#=HQ^Qi-x@2t*cdm@K9 zX>PGjbmuhxmx&CKMNx5NrXWLa%|UNPE7pv8VdE{`xe&r74kkVx|k1ceec4&I45aFsUTn_jb?qx@_jTBaQR0FYAIaIW@7h2_7yKN~`kdw1< zb2ULOjOQXy9nhak{aADf-^>g=Dc%)o#Zq3k4+F|q!P*O9oaSY=w2V85}e2v6Z76A zS7y{m7Wd_{!(^a4Z*8j{U%$mbPX?hUyZ#cj1|;@aGQ~Pv3-&HS!*lkn_b98M~y~~p0{4GwuE+^trx3kr)aI=O*>Hp z*D!fT|4IFnv)dQtoHQQ{mb2}Uo5>4Ji>rZ|4KCV>OUgobxxeO_bFpfoQ;C#^9kv-_ zlkjNDOkgZ3aVh7*thDiPcu@RxS3LGmR87POlI)9=_()lnYMF2#;aq65ypN09;8Ew4 zybOwFWtp#asQXW0YaiN~~j>1yx&j&nCzFC(n37wskGArIUts>!jCKUD#$si8GVw%)J09ryNpMpl65ZX$imY3f__^x zCbL>wyY-2=VIeWESD7)H&fUPm?>JM&u6QT-@%Q0a` z1|e>$;|tfTY%=P~)AJ5_Of`tQi7}pks3LL}`VHR8*sIyuP)jZQlu=!7CI%r1S<6Me ziaY|JQBm43nd(Kumf%8~hX>wmwHgIvKPIZt)L^v%m5I$%`ADBee!3L%A#U;w0Uc7H zcuiN^e3Euoi{TN2H+o_|gQXaX)k_)Y(sF;Y(}LiUb5`-FQ`c8a6T>%0~?W>GAs&ISaKfg%$92WK{G>o zDpy3RPWJ;FUap-M85Qfkg-SkKDHrP*Teh4IEcQX%0bA4)ki}-EnH}MYgO-YvpKC?R zvw5a1C)G|OgHT*T7$H*fCYYw zJp??q?!^`TMYcOD4D$JW##Je0m~coAS?a~5GoDV}i&MHWtn$v0xM$iuCXpWCO03UP z#E*jU2Gj3QBu}yEE!rJrr5zpH{UeJvEc<%BI@akJow)A#SMcU4739ObDK%=Nr7=sr zMGI-e!EB^Y(L*vCPX?4=&}wUOIc4mDud6HOiSMG;C|@bY!wpRF@GdsMro%07FsU|b zzS`0iV>Dw3t!Ph-k>(lFpe(zh%n;KR)*zJ1v{wn8Lit8_q-E_(NZ`qZ;9PND2qx>5 z4E*Pa#j|up1y=8bM*VdS}_uQD4ARihy>wRq$WF`o=QAWS6a|s;SwUtm<*9SLM5(snEPvLejS${itZoy{$kz$#t0#WzFAT+^4yq-EaW zrF9{=sT47ln0eim2MoWSm= z-i_mUBKByN{>P}gPlV61RvCQ^@()!=$4{LagsR;Yy`ng(pWLlysH(1!gFp}k zK^0ZNGh+=5Z^!*m6#)QsWcqO)RqRH!@d9>tgZx0y^#oO3r|Q&zfQy3>=n*&{BnYkY zf1nEJI{l7krpoLRjsx{|#L%-ks0uid4Ny5971ZgPtQ8(NRTI~!0XPrs09t_+bO}(A zG-?gL!}H-8sJc4nIaDzn)$;NFcn7K~&WtD3$hCG1gQmnxwIhfZi43ZpqdI!EfGY8* z>JE|)o{4A$ct0X*P!)BmrVb(l-Ydck#;B$aA_nmC0+APzjxZ5dsj58ghi3#_S|fCT zdI8D>s0|>~;yR#hpl%sa9jKj31V}fDv8yyq3Ns-Gt29j-1FJO6Rhp(orqC!_WzzSg z*Bu}&jGDpK~))gk;W}ifrzjH z{X8Pd3?>T4pbrt%Cm}=Md#xJLh)CU_f$0>mB+s6wn5s`>&7^rs@ zB4fzVs~~w3O+rw}e!%_^Swd?U()0K{1A7A6kWQSS_9A^w_J(YM70(5`L4-5yy@`sA z=k!`Y1~CZYMDD z{U6uRXi4}#^#|+Fo(Hw1a9_KA3@9IDTVTh~PSh^A7g2d6D=nUZOZhJ(&oWEJ1$^qS$jpdW0xJq*KGR5Ce8W^VP|AXmn4- zfuGQ0EKxp52Rkr@4jl*EFvfx=Os5+<9mi;~hpEfxenj+pxaF-ahiQ1KN(e{5@GoC3hWig zLkW#gM0y*#U4h=#e09evO>>o|Nj4jDq1$Z4V!HgxWJe+26q6wqguf6OBevCIQ1U&9 zbj8}qr-&vp@N+-`gZ~0sY}4X3I!>eT5g`kHh9-ZKM%lx8Hu7gc{sI9^yC236epbzK zfOCle20sR4fW`^MrzGiUl%4#5?HAI&?AlqXqcv z4AJnA4}naMiK++UZz2i5IH}1wbh?&@!DrNtmPliOF-3VL%GnTgPM3Z7=QQT*L;yy< zgZyQZ%W;H=#6&8kJcvfWCaPWrc^;w~D#+6y$3?jjiaExbJ%g?%-EZasU>n95H@F7$ zyl%&Y0ulF+{~?+l**}UAkbBXe0d=4YxCe4z8hKAK{A|20jptRGrcOTuxfM2_WCC#{ z=&q0lE&f9sqw6Y#a*MF#T6_l`mUQ5E=u4C8nXyM~Ejn51&x>D+Etq7Doqnq zPmB%daBORUE+#T55R8xV z)?Pg{Hz5SUZX_$AvArQ-6?=3%SST#6uSys+v)1Ces%;x&!rtaIro1N;_SlR^Q{Yo| z^fpC~rpsb`{;B$xrDEoD*TwW{e^4rK{e?AiOKYvJT#j#pbhpJ~oiSU^=yAJ~S=Qqg z5XsytEzn??jay37Z9e=I-|z3jrR2PH2E;JSwu~Bht?ZXYoMtvbX%$HdI)_BuJ3vYD1BPmC; z!MYAN;gE$F%$YRLFmaM*@l|XSnII(f+iU^|$$t<C@qEjRj}!8h z>W-JXGnInc1YwzLbtL=`4UB(#9ksr;RnIk1yMuc06<*$Gwb^GJHqYb_-iz)!v80w6 zi)Heqhgylgo_3{dGd9CG4qO_|6=@3i|8wW%(giU zK{RcO1BJegF=UR67fsF9U;hHnM&sGyMgAd zVaNbu|3aS1aPB6>*q{5Hm*xsujJRWwb=KXqY0P&c*UTxU?vw{N{jJ{UOB9-Bficpj ztoQTAW{pa*NUQ^{Qv#>$$XtzwKWV?1|C^<1yAKn@GOUNXq1As8>@ z=Ib88;Mz0bxzlw`qi5XPX^aYURIBszUD_;33g(3h@&=Vj<9U||)U3h5P&>V{s;JFe z-CIPx0{zozFGlquwtoxK`7_s)Kw#>c43#=!E^(KKw_qf>yW1h+_;% zwKIy!W@?g6-RPOlz?4iP1ya;i)08DNc{K%8{%aOX9U_5giICaFiI6F`E3fu(WHFRC z5z;sr??wK5E%FMJw_Cj2xm*$F-5h6P@B>~HN ze^X18O~hhZ9`#>0QMPW_zS83Evdoyqei#|Y?CINGFzJ(XOD7_)c0mkNj7SUEKE$>29gEOa#?WCSX|GU;YI@7 zoZXq03^G>K0UJpwOs6wf?;DwpZmCpUlPfkV-EKu~t}*5k>VTmx6_l4U-Y3>*1!u%e z5Ka3mngAG1%5%;WzpPG*b=jbISeO+Dz1`@>>8>?r0yj$6x-G`c@zltEag!GqZc`H% z#&6FzZK1Cmpt*{S;jupd#KlVezTN>nonmpB^H_;S}o@8D(@Qu7F2H6X7 ztM#0~dNTu7muGS6!C~g#UuYKb@4bP|GUohp|nPykXbdKG~MB_yZ$lEr^{|+C#HJz zWp#j@+8;P(iwNO{3Jyfip}3H`V1E$&h-`|N67M!l@>Y$7RNu#z3W*WrfVDdxgv@ zb=XE5pf3CJ6%LcmR0^%tuCzL)R{l(RM4Mn&8qWLFxGXliYf3{}T(&wYAW zE8P|&Z#kty|4Gf}O6wJ!|CDCaeAB7nsdoQ`4W>B}V_HlsXLc^1o7v>XrpvKBf4j^| zZfX;elL?(=koDSL$s!2ICG2D zM#*Tq94oB_TH@VrYQjDV=2wM%vOQ(P2)m7OuG>>GNm7wvXY zvVnGVYQf(IOg@-v5*H;O%r##BRLKYPO{cslbJ;!3S1!ssm|HDh`sPe%ynm|AnUkFa z8^Q$IEat4N>FSviWmEdVVsp%xeLHRb%u_dx`EwepCO3?m`QA~kqrDlU@~X2j&GS` z@$bv5=dgou*{X_33UZM)+Ys8z>bAOOT#K`FPE>?3n`ugcS&_6`DL%2fCkrw*QkNRi zIAye!b>b=}Y!wqWK`6~S#zld1WgGQwUQkhvR>o~!0X-?%VlDmZ^2Rk&JgHUR!D zl|-)DwK9~znnVZzrJ+nv26v3I#9E2NFiOy}C{3%NbT^PtR0(wc;XqtdHdc~~m1jnb%@ff1oj9Jfm3}9A##KoS^$+9HDGE zA@Nw0p9VGprJyy;8sY9nROUL;#dBeN$qL5k5STlyO;Dy5Z34s&%17fkE6PmMeFrF2 zt(EdeNq^jz%2{VHo`IsM)(D@0ai1alkPRpnE5`o_?SyNFdr=KpL5Lx(gt~*qJdPzK z5q^UFUZHZ-GJ`%q*>W#%O@ttzaZJAx!hqV45PM+@l~>jKg3tlbHCj8M}(7k#&XVQPXuDhV`pl`xtBd82K%5WzM zO`_>-=w=`nfO#7cQiw|Llh08DYJtjsLoWk|34I1#2&|BSjvzgymxazaNFN&ZfyzY# z%|O4`tkLuUwnf8lLcg(&h#1!F5RNC52?PBJG%2nPgc^``I7cFM0PqdE-Uf;k`Vfc! zASj@tfeeEEB5cYSIMiwl7*V>_3;mTO zED73Mu|vO-k2s1F`U&{oupmJ{6RuN3oeXu<0{M$xLU5AKBmD_I=e0w}+3{?oZypcc6zg-&XfSfmomt{?Og(+w?}Xw>QcQxlNRRg*`*bXmCq!%lF`gDLCaoFRf0BQ)$Kfi)NnylE zgdZzdwRmt0Bm~L709gcHLPMWvF(mmFq@P(FUn9%}#c_~vt$&P|h;XuA3t}d+>kMK; ziof)+2ec~23_kY)K|#IvxdaR0b2w_Cm(onIu*D5B= z@Xtt(tzyC;+laTVS{?(~GRUt3Xfz|^kc~0WUIIj>;uLJ>H8{jh_EF%me4k&({Vq_YiLMf$S$rU zYM30@2kjX8Y2t4n(?yL9~~{Ijc?8W*78GmHw}{`cww99I}Me* ziU|YaMsGv3hlZdI4@))VV#l!Kz|%or*4-9 zzM`KK)D3(J4S9!tGy7LD@JIAhKaS!qYqG&l4b$sn-6kfR9U z_$coNL>}}tX$C9E-RKx>?iJGE-_m*Hey=gC5sfdNC{Y?BO#l|$=2+x}!yGG*>;}v;A#I49JQcjQj zPYr2{IRrh|hhx#Nggt5j$4Ieo81h1W4Lq0C8238%3bJR8dn3N4{Ar@z)$auvL0c2*`qlX0`zp z4_z&x{SADU-j~jN0(ob`DFfLI*>MoM6W2pdp3t`F8=ib5)<$|9c@wfBdhVTU3Vz42 zTHn*PAul)? zXK!w5o0_vnp}+-8aWb%o!St<>1m z_EN2`HCvAQ*-&fEY);pV$5<|AN<5HxD{PWQsnShvt(m!rdsWF9ifw6mx8-CEU@^cg zTD-i_iG9)LzO;V?CHDfBqpd1AgMEe_9iW~qA)isr;*~&Y8hG(9tPk52HjPZz zI(C{Y!0GMv6Cm;i++P1ek|yhd=%v^*i>bEEV8Vg0!R44x6_=z$^p`a*LR(gzzpmh7 z5BC+VwrsO7cp!Q#(}bA&IR3uJe2(T?as!9C)9q!ufoBO~^2Oa?6Rx&w7Qo(+x?`Ot zz)qt{^-lw<{=6X9m9)-F&=Ka2p?_E%;iO!veN{(5O{~PJU8aP<9M+W5%cdi&Mj&Kr zxfgeXO;<;_ndO_VH`s1ODBSh$Mx5GC?b$%PsjVK@VxwlY8YW`OOZ2QOv9xGX!#H!J z%xJjH3IJ0P(K?FBrdE!9<8`OYn84WNyVIyOGS!{kLb+7zZ$eC(jn|8&TeHdc;_?Zn zO&Kz@@j4W6Lyu*uV9AO~*O-afbaXC-E))Pw4O0DsDR+BQOSSvFp7k*2{&j0t7>07K zR<3Bh!N$v2ata064E;XRS7bN<-7@QpPAN62(PMaL(^7A7&zPm{H;%J0Syb{5G?Bh4 z_{%5KHvz%-I3|FUO<(XWCjWg(W0gsP_MyF=)S0T(<1H;6~kMe zv^DT6Vkg%sht*;U;A!uk42#yJzuLFON>1tI8papK34l%m7)ghYUh5ZL^{VY&g~iW_ z;by--ZOaeu=zZ%al&7-e&G&xtj&ru&`nS>9uOz&$Vb7Mf-#gD9_u0f7g;RFh>ZsRx zfB1>>dk`!0t!yd)2j@o$_k)gU8<8ebv*SKl<4nPm8#2{_&3s7k<-H z-1U|tHoNHK_w6x#+~h=0bKB4C^z^q+z2W4LoIK&=(7ENDT-+=F;XSsHE_=gK|90sC zE8OUvt}V|x^0FsC{iajgyT4;|;iFsKBW!-t{2m9s=BQU~zZ?C&-N$|}9{BS^!)NWZ z{buy>f!n_8z-!;7eM7%L?%wRk?&&|>KK93>jT7#9?dCU$<^?k1NB2J^^V3tRdrZza z>OgMuo|mrwvvXc^&-{TGU#Q-+#yp;GxY-`Z&2N7E$8hH-cZemE^Lrh8?v)4btAB$B zFm6*UY<|;)yZAnO`i;NHz1nx}p6jk}9`J_ErSBbn;&ppn`-!}H60U7tVYj!v#@c)E z_qTjUIp(lu|MtB>wQ>FF?|bCIa`ziQ=iavava8+W+x#v6)qCoPAA9w#r@GXq0$=~o zCBqw!f9sj?GdlLos{0#lHM#tQ@BHED*VilO-uk82?R)Pxb3fjC&(a?sx&41CjmmE> zJEZ*R9hW}3r;^Ry6Z}f(h%X(qX3g8sct88 zGgAGTXP#O3`rW5G8;>7!ap#=1opaCeUUp^auATo-t~}t^TjrMIUgbEN+R%7UX*>T# z;_NkN?7Q~;KRDxz-|$;pfA)#@{rUbgj(y;UGx;)I;r;1*Pk2xA>a)&~e%HSHC&!$7 zCjZH&zuUOYdEimMJiKx1g|BUYAHBo7o~*3dXU&7x&f6}CoyqNfGaAlz(Kjx< z@S5bU7hL|#t*?IgrmuDGz52+5#t)wJ(e2;i`_m(LTzcCB?}_@3_|T=})Qhz5NG+Kfc}F`r;W^<#*rxfh*s0(RqiT zx!;vfeyi})ZN7Xh|DY}$yQO#d;&qycKfcSy><8PA{3LVk{I#dtc=W%WE**1FcK=g; z_J!}i_u-w}m2Y{v_YE7rx$O3{-@Wzkes#gxd)6NIw>Mrgc=V3j&N%ex2mgjf$Q1JV z?>|<5nmg{?bAJ8n|8(xQ+Yg_)eejzj-}`~@{{D{ZPTNmCc4xO49c;doo zCw}&}hZmmu!>`r`b2nYwz5o2dZw_&t)qC*2oom;w6>s})qwuDE58CCwFFWKf51{ux zvGX=tB=`BiO{L)z*Sel~-xI&Ez4oF1{py?SAO7J-Kk}~sJn`ns{`S;uhdvfL`rGdl z4to3Uhrj-jTfQUCta;;Ezr6VyANs*Z|D*rVj>AVU{o`MoxBu6@@$Wx+>qXbx{H=m` z*gaogd-wcsgGRo#6xR;u<-Eak6!-lWpBUo zTfa>0a^q)HU(dewrt{@5efT@`*KU5mvw!XGdqnN2%b)$*zFR%~#?5cL`Ml^EA3=}p zbN*H5-1n!ae|Ld6c<83Rx2|{(z9e5L+n?w8uetBbxrc80@*V&7R_8Y3hcEio;{{J^ZtnkCJNNk4Uiel1liNOX{{LM3 zwd0>Y?}S61d_ey3FHW{ydZqg{XI=lqeb;{BiERs?`mbmIeE-|`wy!<(&-cFSrn^75 zg>N+@|Z7VXO5IRmtOOYU9Y|7qu5k77 zd*`_K&fWPf{;!{Vhqw(%>6s_z<)EmFkzV4QbYQO4nf4%2`{zJ!~nfP+tdG-#!ea`_`_nlkaa>u0y|K%MY!ZzTVTKo;I~$IZjfeYbTN7rk-I zwOgJ0rnMij+|$k?z?`o z@Y8L-uwC`LJFUrH`N2zu@&WamnakD7|CYLMhqKmfF|xe>0oQfyUw^l8>hHg~-CwN- zJ-PO?C*66^$ItoU>8GFm&@+GgOZXkW&ph_=AO196RaJG%o40w}f7}#3_#J1Td)RfQ zPg{?Huw8iOVchTkvely(pZaF;mzN**>{g9KFMa&#eQwdbi%=!0os-$=^P7&(FWG#r~(>jMjbk?x8DEUz=-RfBWqp zx^(o!x!dlsW2wFC{Wt9JH|N%Gxa~uCwJ+J{l4DN1_nMOqzVgZ|AG`X5eLis5pYHnP z!9P2;bKFmFnZN$@ht~*iI{5z6274S)-e!lxe|Hdsdd&yE*X-|i*fW>>nmOu;@Bc^h z*WW*LhsO6m{p}-;I_5wRbo%G-uney`;)dhzxaPI5-|2|Yywx@Hk$bN`;g7o=Dx7@! zb|3Oz?>ULv=bi5qZnvZz2{c%zb70%;J9s5mwjY(+gaXE zbLU3S*!4X}eB_;12$%2Q>Xpto$C<8s`auV(JL*!fYYRo&2AkSAXNd-<}*j z!gHSV;0L$*+MnKe%1K*qhT9yIIpVy~BU^m>OONdwc*AZ-yz2Dmj;F8L`=Q--eEk0F zA3gB0JMX;nHFXT(!cXEKyz)o8#akb_?_}i{+1taX*e&n)#Vt=ga{E4B8RO;a*&*!P zJr_TA=g;^2e)jv@eRFLb!DBAJ4t)?c9z(ss6!HzTrUY$IcfIxUhBq z-SFe~{1QyXRrmk+;>Wi8%9&@LxwUY>o4=?2=Ach~7orsSzcs@5fAK4r)Hh$b-$_Te zf8@C2+U@RkU$FM_%j&^fzFm6wti!g--`ctV!Y8A@_}!ltwwBjy_M?N}^ZK*jw7AI* ze9hds`+sTgOFsOQkI$TbcW>M7{G+=(@{vdWbFXu{TW!7O+Bd%I(Y-F2yE*yBPaLCO zXm?-nJJ+Fq9lm|{>gM|-zw*u_-@X6Zqk{FLPx|5=KNv)}miOIv-@Si!+(A2h+5JiO zYge3l^b@w~@RPv7;&(c`xI*xt8)4eG^TJ7BkuUNpYy;@|He>@}?2 zb?f%K+;#1h7ugQjiXCpd{bx&8yGcjy%)Ryc1GnC*e*bl+#mfs1+`w)fJNw4aTiNpDj-+PRyd=-EFmb{JlpPzVhsk zKJYrVkl*3(-4A+Ts~Z(*moxr!>%~`G*}nCn?kUG#xBJ)b+>HN^-S7GBNtIuIHGTc& z7wq}9g@$9Y{iBt`w>YFX!|x}){e~^B-R-6epSW<> z*Y5E1HypBKAt%)ixbw)H9@_E#_%{!KJ6`9a!et-%^DYnU@Qu5^b=CHtxSxOYH<#~w z|BfgAWRGpGzh&zSieK9*{aEOor(bi(Ltj4cy>okg`cF^peCrAF-N)bIj9up5{K>$y z;yUg-uPg8KZ?=2ia_vv=y77o_UUA<(Z`RZgT#obZ_}u3{7r$q>`rn@Z!(RIjiWfck^S51d!amB6|2;O`dH4U2Kf7Hf_1-UEdhj2= zv4@bmiM=;@s@Agd(-SYIEgWy#sy}^0>myWE7 zTix)L}Q5MTf7Umm?Y{Mg~+;qd$`ZhP-ZTfZZ6+m7XD{`%+67tVGzZr%U! z`Tfdi-!VJf`tF}(Kl|F#*Ieaz%l1FJ>ow>9d8*g_wvT=A?i+>c5BSraAK&V(_C+5a zJqZ1B@19rAee72U9enNO$*-n1d-ayv{^D5WEpK%-cexGj$=;Foynony{F;3)fAWe? z*#ETm4iEqNiZ||)`u?N8zIb+P`S{ar9zXfO_OIQ3i!*<9Q7!%K8xHvE4e4S0-p|k7 zf8kl-yXL-fbhrUc`pEl=Kilqd80e=D`P(0Ubvn08;DYnJPoI6=w~pUwzf<{eG4FbNC`<(ln`#kUS2Rxrl_F8+cImfuhHGX61 zp6^V)dIX2yqgt%O5PEdg7w)cLq;cbu@ds#Hu9MXQ$#}*ObQ0#E^jPf|bs3^^_7`(N z-XH-_vY!j&Nga%hAolDq8)qV0H1oOYqIqKc;oKF1i7;p~4Hv_m*v*=ZyDs^6|3p5g zQdKk0xt9%3*lpOx#$ZY_SnRd5nIrn#smW?rcC3WF59aH6 z7m(!&i`hyy>y~RHc<~pV$ew{K?E%Pv-yC<7D}HVPSzt@r&!L!qHHlgI)kw~lr&$j zI9w4Vc>KOd`Uou+O--SeO~$5cO1_+*pU*W$a~T^$A03jiNPU6dm!XQW!mH(+l2i{p|&S{n&{mRV7&k?k9_w*TK1W>~|Oy{FmaB zbgz4t1}qDAX=RtkO7X) z11`~txNS5c_0Z+6Dp4{9ot9gYlct-UVwMc|lQ;N%_yVpD_3`q2wN?wfpBBD3wgsUP ze|yx8f4;?|@wNeDIRJ%0v)1C)ZbV*d!Y8ki2H_bT z%f&Ohri}E{s|8IJ`h&^ zuwxjUFZ^%TWyWjt*v;6r9!4XOKxz3_?8@jIJ&P1@66<6MU10A6# z{`$wEOiz6NjYOf~4h+fv>NNufxG&+r2IvtZ1A6WL_H@|um1?PbFByIfV{TUtqR-A3 zerJxF6HxmueZb9T{3FfOj7_!?Mgdo<4YkEr{K;<%9X61D@Drt9`8Ag}V_kWrRWcr8?cVYJB zyTkTpmY349JUq0hHF5ta+W!uFE0El|oLM00V39)iS4K*K#_Zq*CP#8tJ_Jw-oHT@t zw<}m{DR0L?a+4kzl0E+W=l)gEF0nwRPT7(pv%=rK1pBxl~K5(G%{GDgn{Bn^4ZVH{@XuUQ-3^R`}eZ)gg@HU;!NUC#(dKJm$d$OA^jen z1_N9V+)S?TBY~VGB&k^c|G6$7R%R`@V$=d8N|LQH98weX&wmCM)U(GELHx20*3Ni` zeA<7_rhi|638VK*%h=fLR|?tle_Z%4>j*ef5I7CqlzgB3&i`GGe>qFoaPQw6Vl$-0 z;cvZ^#LDdc-=__3s^sCOG*J;a0wHT8k0-GH{aBVhTzIFiElt)^%OUtfD4ct3j+N}Jl#z&ZXoc)^FJF2H7103MEui!0Bh zLhm3?I^p8$Str&a_p_%@3D~mt!sz|Lrjts3hx_lx1~xZ+Hx zI{pGFweczZv**pt&F9?l^zZwsx?|*TM7;$A^BcZ?UhHM8P$2uyRfXYUt*hiwjx`&@ z89GSw1?jGIgA}`ysygSElV9Rgr)s7gWkEEr`qk((j7vyJc=<7P*8n}>=K6XYsICnv z|BwTa?`xx?pufd}S;;ejvzHF?$7#QRb__%1+c-KZ$O*Xf?HJiaSxOHNk8^0EUejlb z@1rK~$b^@d2hbKWjoGU@XcP$lu~&uRD^rD&SPjSao{Ce3;xc?0W=sdtY5Mg&uxc<8 zm~?8hEsHovNlCYD?6`^O`nc^k!d}ic5kQNkcU__>Bv1$x-X(|9C}xX)>*gKAF%*3U z8$iN&=y;Hxr9RUlX41@ULeER^YD(PR51^VQF7FcLnO}j@Rj6PqRqrWsJQC z@Dtz0P{}rx8&pe1l3W9aYE83DTL3*xt|x8$>C>m%bG4Sc3qaU?2jTd}aciWl09bR! zhRwGp{enkZHsb9rr31vYz|g3b^I-W49K%d^XY^6(Gs)`N!!A6XGe(^j_ogEZ%QzRe^PM>K z^-N0>wlV4hp+0*~8{y`;v03U2ccqIwPCJu~I#6<;rc8?WcGoO|c0k0Y_% z8+l>sqFHUS{8Z&*0Yi~&5jqW?rS;U;Bu>t)0`!y63f+$r&2COYer>I~g;?~_2%Dw# z4+f#Eq2Bg)Ib24Xac8>oAn7_l;xBUIhGafb{uFq43?) zLw*7W17q2{urr*`AHSW9X0F1UXPNmjY$J=;Pya zQ(n&eJMNY0*Tu!f5U2o2@*^IngVdM2ymLNhP6zW9Y23E$7Ii-fno>m{xgbSm8~}fE z3V0^!6`i>IG}=w|%Qxq{wT(`Pb5TsJtYbdzg<@-Q(y9_}^-wz!s10tYM8F7!@$ZBoTp=b3Wl*qe+EKLT@4Bw$9vm4^M#QAIno8$S)wb<1Hq9%%-(=Rvmn ztWXI}HL`x$hXM!e>lh2to0l6gjiV-OAe?;WKMsK3ku92^=oShwW+fvdyYEL~2@Juf zq^(ufyv)h(?;shJl#smsJC=b2A`H)(+&Y-TRRXG>u$%ol5NBp8?T6HX^RbS~#viGz zt&Ie~_~3@NfUQ>ubOhyNJ;cWdqR@RGDc=#l@&{cG-}e)o6qJ=ODkXFW4GUMG+|kR) z&dwddcAOJ;uZEVH0j;_|cy)cc2`*(FA*wx5W&KMqYgUhlV&3leg=Nu(INx3`!f_#w zwRkhiGPsKV46qG}X{|0C3ngf7+05|D9to07;gle6hK#R+;747r2CAc10nsLPA%gsBysu6v27xW858OG7 z2+x9ibm%$M)ma<+4tF3MxbdVeps|q%)?!Bd&%Q>QUXwxEN7^09aWy?FR%RR~e0%bl zbvvy#dcBT7)M#LmhrU)}v6i7Bz3afxrIt@;*Wa^Ee?ETl(kdfSoG2U95Oy2_|6qWA z7?N0S<_KmYf}Zz28#Zu7#(1pOwjna}0y6B>&K7d|^~7O}EqR-mr!RpAzVO|Dk{+2n z7LdWnCYyNQc^H%Pl*uh=;QVr+Q@{obWQ(W zktC>?8d~s)R?^YR54!*=pCq}E85!HisqOR^j^Qxbe^#g#g-I4GSd}x z?&I}WbLrh)o%e!#TpPWKJe^2Z=t66IV~5c`#n_oPKeVKrcc4h5 z!uGJ0`$MxiwBJG0t|jU+$B(nR1Jz;W?yR?Sj>5R4q1vdp^<7u&gQmL+z!#`I8+yuC z_Z6CgE^AeS;qc-?7@ENU-LDcSD~5;*sSSnPl%HWs#{773s17RY>EyyT9+}41P4yMH zL(~$2FQ__$upOi*JF~yUA;9@Ft3I>Fi=RVM{1Oj3SFpS$HO!m7S7lcrgnzIlp7&w< zO1i$ajmZ0=rWb51?sl=4ggy-a?ZkkBfxh9MQm}?9*4h1+ZRt7j{+l9{wn1`u4R_YH zH$~{8VH%O7LX3eFDqVPw)$-@-ZS|SW^US1nLxqbVhga*zEjUtXjk>upE(h`L4yr{FX4k_dMu2h^*B zS>kkfO>Fh#q(m4xZG3sMx(YTDwXTP*(;8q68^T2G%=y8x&e1VM`fJkR*H<@hhuzJ_ zkcLpGZhSZGl?N!s=oSS9lc)?vAHC$6jDY{Ib0=(usK5)pGfABAkIygH2bm5Okp*Tf z-#HhreeQL;Jou3V;WBA|+?aLq`?s|GY8s!DMZ5ivBMa)e%CVPq0+8?tA^t-3CHzc2Qi1Imv(PrbL6&UdFvoH@}q+pBQ*ZMfIQ zR6E7E^{CZ4fs;XlYq+EZf!~h9Tp6P2JMpeZK|&fs_Azt6zus&vAK$AQ7TonAmWd`w znO-Vf_bV(^ElWrE^hKin)wsCz@k&=X4SM4SQe-}M1XR3Tn$C+;qF=K@Pa-DtzQ^o1 zay2no5F$^{q*?nFG4;>o_dhO~WizO_`;(sxChNt@kz3MziwRd^SO;IJfhGo4JQMH> zrl=_>L?e>o5mNAV?WxGM2|NRzt*>mVLzz|v+D~Z&(nYU+xP!cIQw*H+Tg1;`t?fX# zoZ>Bihi^NxaH-BBuOQSk4FW5RiF6!B1__ouOIuakkfHELQ5jLKkn0_yxFw=(s7|#M z2CWp9cnC6NaA2MY7C(g4dlvbRx=x(T9Fcqa*|Lv0?CstnZD~V2qKZHDaE0UU6b%Hi z6S*Q&Z&QU|3~?A6K6}teSw58;BXqqKI3|*)i|i0Yi(Efq(*WTL>z}m@JD?AFoGlA^ z%9gH!O+xx~1=Gc8Zw3RkEu@?}-MJkeBRH@2XPtGVDEAlyPG7_&J59#Nc!r6ilB2JSk49)kAuo>T+OTNG`tEbka%J~?BnT_Vl*WUAWkfDSsZ>QS1po#>-_MU%?`6gLBMAh4Hzb6%tJQ ze&8nnHo@Re9+I}LQr!qXP<3${DF9^*TD~7p_=u9AE zwF~jmT1kbj7vbyVN~nv{Cm&0|l}>+Er%YIRp@C2Pk=1ztW-`L;5noU^ZVe$F4 zBDcdrf9Pj(^KaMYf_#r(KA!VE?jjQ7tA5!4lP`{A{4l5N{^RpfKSBu#OvahowggcI z(*M)x;9~_O@`#?2$%S3SC+z?KY50g&2=FhQwWk|sp+WV5y`7lqsU2jXDD*m-&xaPHn1ZBU z36tc74TL5zzKNvnG!hgNf)_HNko!^iYi8hl*hEk~^6~#vtT16tim0)Z0gH)uW^bu5 zT2G_jFeFLDsof)ui|0fA?VQvh(w&X0xgz}V)I%HAPdQ`%1N)fhlPc$ zfsX6NCp778egPNEgzFwsuX@XcZyayRlvmCOJOn_CE@Nh9<^@l|yI)Cc#%JJopUKcJ z=hUHlYt2%#1@FsiB zS>D_D*kKb@KUBOa{No{WEFGky zCjJ=;23mJaz$M=Fm8fv&bj0%VGSRUK>0HFIBvCgwwJ%BV5zbbFNIh*Nc_(a@m?gir zJ~h?l`?MG3AuPy3it=XC4LmpE4YZBcx?i5VJ`28=hrg_a?;yx%Xz}_HtRE&rZ6>m6 zP^Xi9MD!Rp{T=x^7KP%l3Jn=*Be(v68Jf1I0m8!4BJR}wKT$woH%CjI=`%GET z9C0{rD?=v2m__%RlVQ+}}H_ z4l~me!)N}jw|yySaLrMPt8{|8(eZ&TcmQxPpa}H@LEf#0crBNUZqZMkp}>3nuXlZl8?=hk1Yh`%v=s?(&yay~wn)NLVa`<`k#H)(A{4jkn7`Ab=L= zTn>;4MyG)Br3T4~Am;1N90Fk2PqHutCiqwFHy8VaHO9l~&WF_g+oWE%%ZmUKp^>;g ztmSOrhxy?{*CjwNgyUGuH4VwKXh+W6|}_E$kMjl zUhFj3Ec*IRPXbek+-RxQCmOo>QK{NQkuN?r#9|K=iyYtqQ@0H#((AO$6U($c zzgFZNYX>G76IN58i5F<@4f^^Jwtq;H+Pwg-LyCkK<^00Js+WgKk zce**e3pzAeSop zkEb-PC3Mlq{s7=W#>5XQAF^dG{_*3-TdrrNI}#s;EIVq9vWJ4|Vb$ozc-gPoPV^@- zo<9rTY*8tbD^PAzZ9?C>Vtwq0Phgq(G`0sFA-%9!JrpQ`k! zstS-zrZkdgs`s^25%qH6d3geF;v|Pns3Zd0|6@=1q>6jbC5yn5-6+J?)zv-GJ8Wy9 z^N+DRUXgHSLOq%pOc(g6JRe2}`bEcFz4b+xmzPfNf9@_x0N~P?Qjf8rQp_PpvSI?D z$!?Iv2W97hPQEUDf(wN%dG}yy*Hjq~?>x|U^|sTi`NUk;(6<6^7h$h8Vv(6SdqN`Z7hv0fZ>wQ=0ogfsoEr&y8A+o z&reTd=qV^EdAd)5D0o(_TK%SA((eQu*cPPwZx5WI9-7D+Kn&DSI;YS! zCmiCyvb6c}lt@>q;JBxuDzB5Qp@C^;Mgez`Es=8R#nAiB%V#SwY@w6SnaX3N6S&bx zWJak8=ohVJ2KcVsk|EGjl@Nv0f=LAxjnfx$8GCiC!C2!Q|1_V452(CT3`&BAHZ|0M z%Q|p0)8;>e(uG=si@;O})vyAQEy8NL^uyP-k5XihLIG-qx7Y?*cCmE_cp-^fYEul& zlgCxg$19)6N%ZMdOX;l_F%J65zv8wV0UP>_>#5edSmF^)cc_jQa#rlp^A*fxQx1Ak z07X??14=Do)E~9armF&!r7Gn4sG@3wX21k(O$p{VWf{!~`Wz}KfF1u%c%@pyWvE>q z*`86X`ll<9rvf($d2exO#nV=?GW`4iN)poW1b9JO-26pS2;`CadiL*`#<=THr%Pq} zLtU|Y^n)*97?kei`~LCTVX8Pmn&wZAi zHwiV5*xW_Jrui6T(u%1#m{WiQTteKKdbp{{63uDfyh^qF;HEiQLC$A%Dqj-@z5!j>{fH0(@4bpA1^&Gjw*aQFHgQ*9Qg@x<-5h!R1icm4RVJD~TpSgl2Vi zqO;$KTt(cC2s+*y`m*oHcZDeM-t-)~;>+}0q$>tflSaq}ExlTKrq)pK5l||<75eoy ztSnR^@v6>5=lD+wOag83QVqN)1E{f9pSD2`sm2h8bK6{I>D#p?Lp>3Pi=Ld5qqdlg z%mwdu#=mP09ygsXtv9g^js%o(Fi6mLR{)fzfCh2qbqX^l+%gZ%%}Xw#u=W5ma_n7f zqCLtACe>UzSy^opZabTZI1(Npt@gUZ37yg3T7N?N5fe@oX;kS<^ZB0-L737z?=y&$ zbeu4st0Ik>FIp|?S)xyWKA zxnN8mzD+#Y4oFmt>3f3y6&BiMB>6zy`EGT6{%{6o*%QeA;5Nek1MhMOaV|v-vn#V*>gNv5}_2wihtZ8jr7SkD+u~Qi3av5k^HEQC@gh% zdZ@zU43rEGV$WeJTIROgdh??~%&nDMfcEWM?}uTf%lv%PR;<-vhsgi&%_K59Rl3_# zWl%aaC`P=JF4b1u{$OhxH}oF=CKdCEs{OC`>f4A|r>%@9NnLpCD^WLIdY&(nxEr`t zlR9rdf|O?TtXZ)8y}L%o7QtXOX3p6^dOTnXpc~)_Spvwgym08pt({B}KZ;`%;-<)% zP6Wcvv%Oh14QGVB#5UA>z#MmU)u7rQw*0nw5kGVRgn|Zy$`pzU0KZDaWo$u%-l$$c zz)B*PTj>m5o}oub7qH0ny@gi%g;ea|WRHi1I|$xv<=8k6Ey)tH==q1(IqPi zwPPGwY(1Haw0q8BWbKg9ZvH^Iv4}J+l1Qx z473q30xnh)0h3QF8hw!Fv=|I8RQ<%_Xr88B;jp-@<+-0=CcUwr6OK#tGs#XzK+WX? z#cub8shilP?UBcL`Wx|Y4>&E*z&*NBe&*#JYSizU(u2DKMYjmW`A@uGo-EJLD&!Zt zrv2PVUBtl~*v!4++muBGkBZb%GyHsrDv58$(-WHZ%V zvr-WAClzv0@y^kSmvK<`4ZI{73=E0Jk^f9y*=Bj5X>aSfo+1pFp4+p z{4(C6|M^_daz#dL9e-A~kmG=={n{C#IF%E7TIjNzHrdn>si7h_n30ArY+=`3&aUpt z#iY|R@bKYe23-lnUq0M!E+3FwXD&*Tytb5!kvxr7RDu&HGseiA`kyAXf%Rjkhie+6 z7DRZEqRZvAX8q_#h=hRXl2Nany78BgeD1U-+kZv#D3Zs5f`TDNL)SITM6~=7iWF#_ zvX9zsM`X0R|BScWgrQ?o*H8UF_lG{Odd7E!Q6!T1k8S-Go#%})Y3j0lhS@Q){7KcyV0?yl|TT-Uu2oFLo7h?g}3|i-3(az|{Hd+|#VN6$e=( zdOR1Rj+xA3JW!Y?PNoTMMLhu`I@<=z>E_I|A;78D1fRnf+$H~;rHAJF12`(=eDpE! z_CPb;KM^i1pMo{;(S5g`9C8IgIFsZ@HI{c(MzWV-MLuuIA1BEJrXguK(F~9{@sFAk z8fxt)SU}BehYdHMxLNp2SwC3#IimRM*St#&z$dlPNEpknpn59@&l(S42DW(WxYzJ@ z9B{W;9b>av3vYIafL!D8eIyuqloU((Tsr*4<4EC0I{8}l;7o%th^sqvk}U4Nn;FEl zv}ZKQ=4^)!Nv?n>6?R3$erRUPa*d^Aocsk=u!eX8rKD7<#iRT>PI4U#e93$Sp@0Rz z{584+mY=MsYr-MiSiOoQvYUb959AAo-Ud34af1Ske@!X&|8;VCA|IuTU0hsv=DInh z491`G$}tSFJ|++l+HzvAve>G0l6b{b<{)37*uvjwb^g8hI@=@kj9qo>?(-f|{kO5Z zqvnplGxwdO!LY;GoagAoN~i`v)$)|PIn`U;-HczXjVTZfl5g@vr4Q%s#ZpsVNY=Yw zSKb5JeDM47kec!t;>M$mu7|k@0+vnF>DjQ%#Qr`A4gL~YKLZMD0V#qtN%fozC*NSq zz;cRaeR7lRfUK?Z_$h`dE6e!=?R&2^m>C)6vmb*#vMKG3I-V}cFB1Y+Owdg2$pkVa zXIX}G1PhABwudqufH6y*Q4i~-)+BPDKf1h34f z_1QH|asL~0XTu-;1xn$;13YN5u#hGwlm?07@odB+%V)9sf&S0M|H}*e>`8ggoz)`} z3A%G{*w*>%P(P-{81WR0NgtI7#_sw*dioZ^WE2$b7$hX=BRNT^a}u&-kB8t*rqzv~ zJN#YI|C_C<H+4!5x`8VFvivb_@nsOKeHShI?>EmN* zV~%N~i!2ca^2f7&t#bN;s~CQD5Np7R`s?R})2X~a7z+pp{K!cvFcujC zo>@+kefQ3BEqmkynj)glb(*AZB(kg1-FH!w`yK68kF=EUh_){9PaI>wp`@qEq)Byg zAy7YF@dVX$>z4EqJ;|dl5s)?L;|9}db?0UbWaP4J9 zY}Q67c~z3ag>nde$C@9oSy}5LNlp{so4>mWm+LMpWQ{bG6BR$eCXs=?g$#5~9MDh^ zuaI$DZ_7mHg$MuiH-DIm7<=8Sd$u&?yM8{(p^?NM>ayaJljrQ%ywLxrqNz@nN&p#h zwAK;C#SB8Kz1VO%$NBY1gT0?fLhX>E*+T>ofFswi8!WB*<$fBj3;d!$FwQU&zysX3 ztR6?X=baDjYxMCse%*F>d%!=G%sFZc(Iu}8?2;A!Ns<@FO57Ymwg2u`XX|Ti^R?EU zpZ;t9{YhMl)pV7DjVQY)zo_^(%*5a#VH+#v>qc_)6RL;;bBb1cO2}-KHx;vfhVzgK zgBqicuX6aoeEj#qA*~AnF%=}aW^LWYMJixrOgbXtwV0TUWacX6!q5*VioRG4rA`G> z$@XpW3xvxQxY6Hm5k^ov%q{w)*6;3tn&=G36Fc+K?10~(Vc+_9j12Xnk@6bnU!k7{0ErB-FIMz5VLycr_g2^~PC3 z==UC5&gAogQ!=e(4y|U_nil<@hyze~*#4@w%{c&wIC~~TwiVV%PAV8tYXAd!eP>|i z5+B%(2a9DoLUkZr{E#eIasihqU9OqdwihD+6%-fQ!vxCfQLYX!4w>hKxi~YLBbfq{ zL&6dX-Z6kKzG?QyGc0@u!*%esg@m|7L<2#nghplnes|7xT80K;)cx#Kbbj8sDB4>OzB|q0n$>q$vZsOB9}A=GW?tf(IFG zfC|_11l?#43msWTm0<*)x+%5xwu^NgT_FBPd#HMym962yJZHDng4x#Trqkx9#y)yn zz7EnYy)O6b--pxV@@aDe3PnoZdh9zUJSD|&Im{GMJ&?8P561Foym^4o?xBH?wOs3s zjvpCOh*+=0uLa$gpV}_#I4M_=52^X4w^vFV;O%@Xsg_`<;~PNXCe%RgJqIjGZ94E> zlYnK4Z6P{CfWg$6RAc$GwiZlGzJG9I38Qri#+O@x#b^Bi5V&7yNJUS;e54la&2P|i za+s@BZPp$J$~RB!F5BW1bXcy!2-pmGpbcP0H;cejeuL*`@%y#2-k;l3`8yz-%|J$v z%`q9x-T{!hKI*gC7wQG4{UBtWi<)Re<5HXfrX2AMPw=>ye)663V;E4wn1v0yh+#VOi45 zzX{Auc|x)H(Ek_3#oj#fDzpODkSF@^%%nle$Z(4M+Y9J_ z(>OF$sMrCwTn(nE)1amYYJy)}h&cM%-HctpNFY6E5WS<4Pp5B&X%Iz_laniSye6DJjd8rBDdFP}>v9Nnt!-U_xBY654t6xW1b8FBI-NR*_wU~;VT<-&w6xC7F zq8K_$qReNMU{PX8RKe0|IRh!wcMza%-e9yE2|^QVZzYuP7;Mahyx7|iF;d>$y$R?* z24;lI!dM;?p|(rj7qrGciv7FaEIR{9e3pI)e{>h>4wE~~9LdxeGd_I3Pt{*)75b{< z<U+>57zOPbX5_N$)q)JqF$4sJQ&P_>k;uH_BQhimGmzy|LDVNk_wY0ql|2=- zT5M9UGa04uqasTS_z5}>IOyo;IKyuFaWcJ6L_*zxy}3#i6YS#RQl#U48~`^Q@z9H4 z@wz)#P3F@7=JiKVG6}Q+;(FY#Pr8sb{6p`TTbD*VF|`JFrpx|}u<9i-BQ)7CHug!G zYnA0rQn#WTOI2Af;_-w!et$q~i zk)PAo|KK%?i;G*#_`!8#vFz7jJSyQ3p9LpEM~`$8>%IF*XLleXR`T<=u}4@h^Yb}k z!BBwDyf9h8J}5V)vi5hj4_G)Vi>+}`@Drw^N_YTZsJx+}fdwOI&<)ca&Ac6}9dqAL@)W$~ zpyp#0>1M6X&@33H9)vD~j;OehL*bKtr?m0)T>IN*F*nkb5%G0{hEVTfV!h*A!xUrw zT<1BI3OcqW&u|^Fh6A=>pe-e3}~y zeYMfX3X4PJ=2--sU9vVx4hjq`P$F6D6Ojohurry47=tI8A$D&<>rJCxtun{BjoIuI|C^4sKUbSNrK z;V7t=HUXb~x6hubKe>wCa;d<)4PDX>rpn?6WDSGq)iA52|o_i0la-8LM zLIMW#57f*W>u2J33B&h!eP+Jl?Xb>TPB^ixrU!rx$UojNg(k!^iIV zr;fKg{E2v0tW=htX zc6PnsETQ6BnQBYAXD_mtOvb0{2@l)vT_oe$DI51H<8@&;NbjW_e<{dm3@$7UefQYa zjI<7jqg9|%-nFyR6#4GkWZaS7jaVjQBj#m%nYb_5v|uxS8kaH6SCBI45zjX_m8s_k zzdU4G`J1QP#`I4f3G#1xhs(>n_IIFRU2XT zOm@r`bg@F;8{H$u8|G781#R`PI%+LFWJo<{q=t@D=OQPne<$c@vt^etH7VRjx1`d) zrxTLoVP?U@97vg{_8S8h*z26=Zmvxf)pTpq_yvOvmob+|!Vd3>g6 z0p>)gYl-N;iFLYw~1l+E+ zlut#>jcMctOwy%PE5CMDP=49XxV5lKClUaSZ}s(uTiQ2ds{1s6x=MLL?)BNO(Uxpan2>SeZ>9rM zHjhDv(*yg){Vn;JFS2KOkEIQ7NQdYoinmw?I$(i~%ql*y2qIeG!Sx zeiymD!Sbr7ljNI&RHd}m%=zzWVgx4Q>RoJNzdANscdKTV@B_v_iH1{~ksbX)Kd#Lx zX4Ivm!fq*q3zXHWj0_#Y#AJ4sY)Xh-kGeB&hBJl2^-3sbx6`k?jJ99D{0PH&0~ZXf z7ee0Y>scsC;WF&>OGa85B=XiYWI5XkSut-VMn(pIzrTs$g^g>}AccNkmSHSu_TW<@ zs+!&sJb$P7ba8HP^2=FWzGBY6pRl)HyEMkyGgu;ca{A#YBzK;F`ack!H@yS0Vt+M_`foH?R#CAG{=qlRXc zPE)DUhr@Kbk!%s85i71UGnx3#jEaBZ_syx4c6}99=myj+`*MSS(0Iqzk1yCo?MOG+ zuuGHpv%Vfqj>C>Aj{Wn@*a=b;SHClM@}7usJ`3qOh8iuTmNiH?(t9qFpC-T0BIWfj z(|?%;{EygL!Cp=z(V8!bVdFi(_9=Rr$^JCeTXITdJt$$_`1S>jx>Jg@v zd}Hu~IuBa*A-@)_-#+>>E&edk*WFKRF-nX~*2OVIYl^pQn*Y*Cv)TN_?leUHx99Gq zA@P2Om9;l>>Wpo63eETvL)Yg5tK(S$LsLy=_I_`yG2Kh{VkR>Z$$csGDJMG;(0^{0 zrkF1}Sj1TFN7ryRcH~*WF;d4%3!ai?m9GBg=GU`psW%qzMMW%38pZ)BXP=3LkqiD_T%&}w9$LnOx zuUcq^XLsI*y|TZYUHDWyNSC8zmLl^)TVXTtLVtb)Eo_@c0?nWyS24G_znNaa;mT9BD4?NPK_lkHD>dBn`T>}w<$F>i-Ld{bBB_2*Z~ zYPV%&x)5o0yZ!dMp~3rNT!p0tBEcpf-GlksK*kSWwk!CR%2dHCjMVQ=`aNpVg|v)o zmTbK{A1T#qgIO=FZ`Wtz5^-5#gj-<5-f>k`r`h24*PpjE(AL#%4$j+iRjzTaUk=PF zU-1(&hrs~byLd*A*@p=TtUmCHrHyAgI$TLcGv&{22-viv(q$kkmOpP2{V!Eo@XV(f z^SwKMc{wFq8x5G47OIp8-~MI@pz4vFE>f0JZ>n#$y}L=zpDfnl%DzX!!K}rwOy?>~ zgWDJ51DF-gsx*@T`OhpIf9~osqd|(kOp)O&)vwNzuyU?n?i!wgnewnQV2N0)?#-5< zEcx70q#@c~H6-!a&*5`gdv^HSsG04&{uV1ULEf`!w|o(7|JH zeG@S{Eab!zL2%l%sD5j1NImIP;_f^deL1b8!)7&i7KlisSxuARof97)ktF?42c~q-kg+~>JT|uQz+x6k}f%&uGe0$Cgbn*3=9-KECDg@2xiDzB67yx zL*IgwYh63i=l%Zn0_0OKp(!rC-r`KSp&9+WX)zLXu0!QA7h6xMB8Q?^_%$GJz2u!P zbXVgCI=0B06WX!5pd|X?@r$y{dj$Q5%h&8ogUeN{Qm)ZD*wg=fSvrKTu4_D9PS z@{V0c@o75M+MI)X4E7W5={=P#2FHuJStFS2?<=Q`gXJU zuSL5W-;0~FVi7T$;jfSl6H>3l*}t_EbA`!_qSe_w?WZ+C zfOqkB@*XSk!qQP;4B>a!aNZb2w+@GFo=) z-MHQ^eE*ZIKT}?#0_kmZ%6She_uFd$=NKx(Sj?C1>nC$tCa*N>4w~-v>!#IIOZlzP z(IVQh9u*4827V%cB5w0BlP`$teRZ_gk%-5gKcegz35hE%v8A$fTwjl5q~}m3&B^`% znGCh88HGTJ+l7|nd;`{xbdTD4R#PQq+M)8sgC8+7_~kt^o!h0hlYdZ8WkEV!woK}b47y@?ZEp7x!`)iUMxfpL?KLQWh%OI%dT+pB za{xIF%>gniq`ATVCw}oeM9?Zkd|zk%eUIi{ZwfL~Ws7EAJ=2OpLN|Ds+??wuqvyUx z{m#DSN?Egt*z`BYr)gb!3Hx}YU!;bn%7hNp8ZT1ay+SMEGfWp6g^FHja(=fv)UIxl zvs|cb#n!IA&<#U*CgDzNJ5J93^Q-00kZ&NmQD5JKH`3*!dEm5vwp?@{wPM|N8P7L# zxWW`JR(i!()MCD!FK1~atUVA}qRGUAgnhI|Dakn|wr6g-mB{gzZ>la9VD`fbzFJ7? zB#@?&BVsP*6f*PmdgVY+Y4mKeaaEl9^94>eCgGC>=W^N+AA8)u6tT{QCcJ_Uq~H_@ zEox5s$?%Z3o@^=RS-8!$GMu)aT0M{~0?S|{W(Z$ewmxT&EUXt_w}Sq{CuscV+F-~H zj;?Izdn2QiYoD%c*o*y$m5Y(z#hE&7&T`XYrTv7TUm!a};YRJ(4{&B(GSvjg< z5k6a|wwKE|NIjNB@QT4}Qhc71S2J1tf!#zc4l|3nZ?T$d4IC#z6f1haZ+GT%m-imE zYP;UN^rH}8Jw))V8@e^mcj-LAsDSFy2pxZ(x=p`sF!7@|A6({8@=#$)KO2nrUhsYI z^bYpBXy?$6D(~96mSVh;xSt-Gi>ccE_IJ-lUb}vi-QnBfwvQi=nA4jbRThaB5x=iq zefnJ&AKHdTEZVuseO}Hwf)BO;fPNI>U=%7gHzV@FmvUqaP3!k)3gK|mC)kK`=+DR{M9x* zMp|1D_tzfxg`y@%WgB9Lwi8(p79(aF(M)eNdA=ZQ=X@T zx<4Ba?+q7P?mklj*X6yqsVE9_5PrkYI?Bmko_wZ%mi%h$#xr`Qs`_}0aYOymcpX(^ z8)kX!H{^a^pI*MZWu!jIIyjp{B@a_ffHs{7qzQfJ`_9x!*xnuai(ZH=1n+XaE9AQG z(!NZ)#vx>j`i5Dr(-pk6M@SQlq?7?tU7<*UJj!~`VbON2SN1ynyQhF?8i#oTbga!R z0157PAjxfq)GM_%lAx+>u6UXW7nbIJJAO--Cfe%EhXLE-CT5czKi>Qp8f6p28hCMd`#ZTBm!EYWaNnF30WO+8KZO*+|*IE3<2dbPS`UaK{O`HB4bjTzn> zJyMvnsPZT450egAj#~+9Uk4L?Vvc`Q&SiE}_t#nB7sn6#n*v9$1CF*{g7|Ouhg(jQ z>Dhto6U`mKIgB1G)HC;9wR=$Q(9*#DAi6p!_(9|li5VkHSRcnPcixWwGQD=LGR|?z zT1}C!{%n8u1P0}UBzn}TVB@)4t7?uiK=LN3v5zFs8B3nvrA#V_scYZUu1yWvX?VA} z;Od9!$OxK!R?f&;39jiVD~tb#Mc@>-Qx_WZma@G=(mi<_Lqmc#e-crjOdme=ty}mP zblSR&;B}I>;xhjN0V(@2h8htXMXY{PN+%LTvE%O;cSG4k`|&yWq*|WdphmRm8u~e+ z<(_Ju{Hz<)`li9RGh*TIT`RX_)C-NQQMH*nq#;7PBVt>PHi`C;*KM!rmAED0U2vq(1yF z8;e%HVV7 zr7fz9gskOSr`CL}gRUlptCVAFc-a)6)4Iw0>W~JBd))Zrwi1Av8)Y(OKhb7O9ytO# zqR`L{0`_5kn>a<#Y{OL|BA3f$N&Bk53j!wXHXVc`l~E+a1|qojuEPf zVD)B|tU+Mb$`9{iRfnO_X07++jtrc77 zoTdp0a;)>ZQuf!EoJbrL{UlIS7`j>NhU#c4bwARz0l{*5RNoJcajgV50z!CTTn-CZ z$p3T*c^JKcOqa9SapR;m30solS%v&iTXtM&_1^Y2UWFY>+&oMEf~*}_#UZLeP;Cq6 zmwleR^uAfwS>m~Ln1 zWf(K^{ZKi_8Olq<%Tb0SF5#Lm1O*o%&@4c(B$3>*xMq&t=cf%%-*(T97C4JjxmR(n zc2ps$loM!u>H=}V4XdldvsglhDA(8|h`H5`L)(yo_-uj6^0H{aLFFoas$u#>sH zD!eFly4Z3)$Db(F8UGnkCOKT3e17JpPLvZZ}f5q&ULPiUAB2bhv8if^e!P`i~Y+VRGS=WFzELO z(_BiTe?d^sHf$i;HIhj+gO^7w4g|--f9i3SVyFeaf;B`;h5nh1e{o+7 z!Z9R`A=B%pcR0opx$kEF^F$-aqLo0q^a|#UqZK#B&3*es%pFTe)R<~P((;KSJ(nyF6@RSdRK1mec;8j$L5##oHUI+{TEjfMaC=S-lQIRs&|%xz1+ zlaAg7lC&M%(sA?$y!sG~Tsv>e~@R8MBCV@3& zxwUv57(s+xcqPo7Wz2lZ=Cz8qdhA@2zdl65ZTjKVT7}mAG$hnEfNp^NTxadC@bsvl z(&yGpDq2~bL)x!5ae9)}Y}(BhyohVepB))>Y0D9@f^4+s7-g)G)ydtzof*tXTD9O_ zW!tV(hit-wkBzVW2b~>hPw0Snr{S1u4NzpAAG!+(6F3;AaE||E4v#9$utUBm>P-B*4dHw#X z*3*M6n#gN(<_SNXK_1%9%DZ7@i{<9Y>j{7R^BTc{o};T651UJ?_a_c*5Rd&qyn1ky zO8O6&&3fwL4>xqVTPce+?qQ4d%N&}`ruCTmra4w)CFsOHl+gm8M7p)WJeoC8a~E+X zUu?!u`vmRPtJPN7UQtRK@#GXL96|Mn{!29n_1E~OT0}(sF-#hT^2JSNAOou!sT__*o$4D{mv(;=ix9SGBdUfje?2@Dug69(rNu?={rrs=s6X--eETBt3wZF26rsnO@8`X5P z<9AeIR}u}qAD$`-P@SSeCv$GKNkhpx#?eWjKNb1hJu>3oK*>2wU>QJ4&q!%b>FYw4{JdT8|;)lXX8?QJEl`bJB~Zqab_SH zyAn|pm-4CUYKeK3ce5Rtuj)ByxHpP|;>PFO)L`WP_o(afn1qn@inDjXOQt$21=kB0 z7ACXDOmpb-Xp~gEFt%aGp6~v^!2rXJQYY5Ii8xJJSc;PO(eNa|G2Rc-d9CkTXJ5m- zYJa1>LJ)gR@D7bgA|4|iGu3bts-LlvF4kB>yz*S9EAV6G#{>VfXvG7GN@R{X`^@l6>5>j@hhG@sj_7uO-XKU(L#Lq3uhrmkjvET*&C#(Tcy(Ji zCaqdX#khZk7@u#<>7G%`|&^Ax=3X zC$ez%qjI)n4Y!fUrG(RBDOigwso)JGby|SgL|&C_le6{*4|imZKep~mRmRHd)h00Y zxw)Q&)QUb1;ym{I3XRv|3#q})HA?%SL89*JgGE`5Mbq}rbWIop=aX$Cr(tZ(;$95C zY-OJ}kEdrK5cbS|qaI+@7pIX(FtI8}q`NK7tP;)BBw1B~0IQ>FDm-dZltNy0J6zyN z*qSvA>SllOPEqE#s4!*qMug5%C3FqJTQAB`VbN|6D{5c5zJ?q3{_iX8A&@owJD%9; zs$tpPW#;u1`M3x1!)qaKG>94faP{VB+jlyiJ<2_Rj8sb`+!N5AbO!VXW5iQI11Tjz z6cvo&nII_(s1j&#JjuAW9?T2HW$vmlnJmBoDQnfT*!9GX zj-pCN58`Hj%T(9?TDR<8_fU=2;JFGiJNs4XFHA%`Iz^)c?JK=_F{T}T(9x)COtGCRy56hXdV*$ej%I{b zDc^GjWi7@9XZRSN(@!Q2c2X-Vg-Vxi~E-I?ucyeaHJ-))RXW8hRh80z}eSbabowolI#Z)do z;{LbYw-(ncYnqNw1z7h!Xs&Fw;;{)5ZucRvaaVWNje_+%!ev!y_m|611pPLUtJEXj z>NB}oY||#n7Q~rnx~E0)DaP%qTI&wJ)eMowmaQg*Z=s<=lF5$O?{bh|FGW zX<;GY0xTToSIO_m-nb75tL=5x<+}_WM0nt=G>P2ItmHkDtcHC&nRiah!&CkGSM(VXNnLyF+$7GQsCgSaLNQ9t)7S_;y*(@3XRm@T zG=vu*eN(=KHP`B_VsXHoy z0b1e#Nin>bIRG-qyxZ-$on}?6B0s`vB>lD7GMt?AZY`*Z?Hn8bNg{P&;Aa|V>tQuueJ|I;L^ln4_h((q zP_!kJVr2}5EzILN$^*1}!VNbe%aI>ofg<7ls#Dt9&e#epDtXvOGP|SxM7yhyW$t~( z?~6lTIoqmcl>p+;pNl_GYjippXBCsVfC`GVV3NALf>Ae>Js4;7OW3kalRe$H-z^sk zgl@(s*`VtsYY3<`=aGns(bL$9eYCenUOVq!&lf66x}^(h=#Z1gM71${@oOyF zDU?oEK1*WDS?vD)d}#ru$19X2(eJkB9{3U_jlfaq1o8wol3qBuxw^_ZlAt2d0N#>;&(h(W!&*-!~OfB4`I~}<%VA;4V*=;!fTm^ zqTq-1&0N)zYU zhdHs#HS`j0=jAGEnRz)^!rUn^>Mo2<>Q6OSJu`T!e%FF#22x|n&Fb5UoT8B+Z%=OIX zEV`sEblRorCGPcgXy;y)8&=LMDcQ!q5qQ}2WuODRpQQC;a$_BS6VZd8H4_&5s4jjT3d+ymQ{4K6^pT~0+l>9@pAJ+h3-In$W&RGl9SNow5Fl=2$Gj!Y_*BJ z1_ie#P}9*(y%)9M48*$F-0X)v9_F3bPU3P1JGj3=!JKxiVP##MsXwa#T!7t_O`EB5 zgFR}t8mEW5*hd0Zlgx|Tiyz8QtL^LrUaudtVZ>M6zNb3n#9PZd3Q|bkc3&1+kt4EI3iKQ^T)RS zfIX1x$k{FA<8TJ_?A$<*+USMa9{=r1vj(h-x2KC$bJJLiBbB3Z)q(Z)1fD2vXntOG ztKR+Llc;~C(qI~2OYn171}!l2hfWs#SZ!k>*g5+%YQXR0_F7N+>EZrz9tw(0C5rm6 z%WJcWW6F7DQdlKfH-I*@jy_ql`v>n~gY;{mcpB#Iqrwkg2iO||ZIE0(wG7FUwvhHF zwUV~b8`5D18|GSQhA;Jb4{LXQq%D3JUgJ(-Z$B)PnSa!3FIBTEp>A~wZ#VOpvgQ*z zfll{wHH8qF@Vj3?6poDxP*QX;qIGv-fb}y@>saY#ME}fx(sNwDF93Aya(*7xCN3CX-w!W6cd9iueZ0( z(!m&M(^6emjI#a)A4OfmZg?_kljq*;A%!6+w#;6JY(5+H<2tw7d6MK-+_7TT$vbD6wgGBO_C(Gp0ud5o7#NKej74OC~3BK2BZBkxi){D zl+Wfz${Y~o^~x;{9eh1K`vyl1r) zA!bfEM}&IYk24xpH50_(Zkey?0SKJY(`u885JYrA_Q$62#+zwy$^MRyrSP~YK5S=7 zDr6mvtimFABXU=$G#;B3q;Nfv^*bNdi&P+O;Bx$^2HMOITaOPOwTH7dlUBF3Wr1mf zMk8A3xJx~N5LA+xj;4a=L#Vc{20bB)1UxQ}X2xTB0cP`irKMH-%V{HC9%~(hqAl(% zDylX2UBR|r&aRf>A%QfnpI6qQY`@i2-1_65Ohxfi+SNBF$8vKH<1_--!*0?y#dmS= z@cMJUE$Wy~A95j>H6K%(8n3nrH)x@x!!1|sG-jGkW5#09#U_ddD;ajW3M`Kr1XFAg z4MlW2-K$?L(l<^o!&@yiTrps69L!%S?kQ>>NFC35f9hpQd!D~7eU~sA&5xT zZfmFHOIs<%kk~NX0#1UMXTABeBmVB@CLhQYy%qv4LAVd2j-E=b#ewyZO0g9&O zrEom-`%nG#(&2zFobNZoE_R>D!C^3SGYY_&me;wRb!U%0{D+sd3~KU46rnn8-G z^S%FuGL5f`MpSL7$VlJC8PDl#OFzaijV|5MB?(OYIE~SFSDFs@HxOKZVOZgxHFx>P zXyv?^43_`J0sz=Z36^=%z`&kfWu~m52t*%#KRZrG09iayZ8|*5ZqZ0`npPteuUu?@ zBavBJ$_2!tV<4O9D^1|LmLJoJ0?!qD)}${_J{t=5kh!5pBHRVjW6~-~A1pT1Vy+zl zNvA^+r%ev(u$G{lpxNd2XgknqJ0Xzoal?rOQhsn(J?b{M}ms_IYdN3;a7v0i*srKR8yzhgv`wdGHPW>BTm z9*bHdxHCDrNoGSWL5Z^$27ae@q|^IKV~KLAOlxP|widPW*iHm|lOs#$A2~ z?yNkQwE%vY3{=67+}sLpkwWLIF^%6~o^A_fkBHT_7^Uo%SVXaKAb6k?+iY04gA_$HQX-b=bsaQE45mGDXF~=L#gB_15Y8B z_lpqq27B{hlmdeZn)j#I2nzEr1XAxlL#L4Ub3=U=@>j__mcpVnmsCUOc;5(Dc-3M4 z+jDuqyBLc|f`HYM{B+e9rMtt|P+**zNyfJ{HUER_Ih6&H?R)O%5Y3hZyT?QE%d=5) zvw5F(vw`KpncxFaWwVoa!A|v=b!5c_tZgbI7T+$(S$YRBB23{{X1~#zcF8V}NQrMog zvVO}R-gVwTI+$j=Li)E8S*QxHZxc7zn*lH{C|f;CP=JzLq$R+xlp(^ogX>c!%$q6G z4IaG}l(HCV7|PF?Z6ajoJ5RT&Y<TV6bxQsBzD7x4|`bQte@q?nM5FbMA_VH1y5LwRb9SAg^0s5=tG z70$-m+Vy9-8&)9-{FRGAH179K7We0eDwL#rF?U9?WVaeSKKv$o{CTJM-sNLHvKR?7 z_ql&i67#9j#(*U@oD=L)vTZNE?p*ZYUgn(;fg=`+Reec`kdW?)M9wSx@)@&&>kQAr z8LArN@rtjRpR+Lr(Ur6OCGiH6?LI_*#t=m8cAf~9WcY*-F;FnmsBqqg!bq|2YF*y0FCbzla9L69fm z1^4$ZeqoD0zxZ|k4`x9CSgVFsFesA3Hx_X7D~+m{&M%PMdh)epuZqxDuJEHf@mDq{V`_K1H^5 zXd!!K4eyDFnpp1*WA`2%j^FI|mPq2NT?i$?cvhROTK$Wa&2PCti3!o9r=L)uYRMhi z7Q7OjwRg;Mg~?NIBrM(O(iRW3D5RDmGEOJ*8+NNrQA7D2IW}hibNl|&Cks_Ky5ch# ze_8t{_uxPJ{V@MvE43CP1wbJs4GdD=4{M!wf(`{kz0WBcp4|K|xDuV6oyQ(rh;a56 zMnn?x>a{V{uSiwBb>Ph-+WP4*`@({#G+fg;ixo;1CgRV*P?3dH4|=NQ(1eHe8ZxPW zCP&N~^guiPP4g~11+IfGokz#!$sq-I*gR))LCUOR`(4zsHXGeT=aK&5oSia#t`CJQ zWv86JpW*rLhq-Lmf1zP zdPUqaOZ0@GKB))qZaNL6D)bMOtT+uO!URvb5`FSB@YH{GAmB-KN#e&TBEG!EmoWRD zIte(~6`|`E`$%xcO-lNlCq6n-jo&$7--JKO%=15owFB#>(e!LGm%UT@<~diQlhs82 z@CkJ4u|iSt8+;vB)1h|*gE~jqeMV<#j=`Ky-p=4T__zH47Uj3Pq0L_s8FXBTRT$d% zm~tW_7_86l2ReA^%>Qgf`f|>WjTHG$Un+DN6fE8Qboe*_J9imQ8V{@%h~9&(E*aZ$ z3&tftw~7QaQ{R6N^Xy&S2T^U2if|((l}`FuHllpMhR@X%;S3y=|DU}dTzWl0Rq#_J zets1(zKkZZ8Y-8mFr%*}K6YPre$-&Y8_g*ZZs=ym% zRHlm-)Z``trF7erjQs!E2l#`Gguh{k{I3A^Y%&PIsJwZRJ;VL=|L%w4pR?78w~?d} zGulc_&kEI}fSSPiUq{npdA>540kTdBV*IPz7H`cB6a@AE$v>!g`O*XmobK~+JqaYi zIS-Du$3gIY1 zyzEr1Hou+Ihd;I^3Y?Pcg52zTG?K8b=NQuGQw2L|f?H^;7jxK~!m3dFyK;NcVpm>% zHP&dln!?p!bZ^soZF;oc)6;L;a=W4ndScyPyz6}Rp$7>XWH0T1d^7R--|N%!5vst% zkZm3{WzasJ7M59hQWt>jO`kGH*$3$*MHutS%kPZuZH~d-mhY}=cg~%MS%y6k@A!KS zU`9(SXU+KYIO*YmmgiwbNZe38Cx`!CIO5Bs_E-s8k$-YUDpFrSl3`wBedupRJcTgy z6e*bgp$KbFAG?OEoJ=pZuqE*<+K{jt0MUfadfwI&@2ISIGGoo`pP1S0{;8*a{{3&CZ~LqkN*+8F+KBt4x3@uH9;LunXpVIWNVaaOK&r_S z)2`*`5cbTc{~2|j81UeqsNQ8j&;A6pMK!;?pG&2~zVhE~t;zJK-wUrqz)QdU_X2~3 zHP!R5i9<4@KfrFKwbZ54D4$(PW?dmT-a368LD*!SISQ5MrDSdjUVV z&rUw^4&0|mk}~H76Nic-{p0D5mIa{_?~&y|vMn+gFoZ#xphc1{X%81k_eC2?Awa;|U*-WVZr%I|E88EdmU=7JWJLa{JE2YS5JmxoB zVNUfc8(Nqo?)g=Njzp%eZwW=lQh9YPy{Cq9Y=34j|Jsb1-o-qjmGic~VwlQhX-U_$ zcD3OT?t#I!!`mf&+fWBy_8m#Y9oGf7WUq18R9T*oepZE%A&;fo?rVqTmbW%@!=vnU z%Pc}m4)SjlR#iO47k)gyeE4J0wu zvAj?x^-LSS?9D8nNTHNLRteiL!aDoEGM|34Fl?w(K6V|)>5Y2hbe!P%q`pjW!c@hL8wZZE~W*8c+s}1Synn* zqL97mAlns;05VCf%l%&0?U9$RON~<3zHe!+VTlo=r$Oid%+sphO`}5bE>mSg13o_A z<12u-Azb&x1UW*67~g1mMwIW;XvNk&~FV=yu^9U=QuR-q+cqd%{_oLgcBcx+%rvg<+fZY;%-Uu_h+Z4 zdYPY_+FNdSnVYp*Sss^M9_m|v8(tsXK=zo=cN&hiPDO$Y{F~*4en8nsWu(EQjfy0~ zpC9o_s5jr&vOqEso&26(Ob%V2*w5vb;vv4FfH9S(kd@RryBS0p;&5qRI6JYavp<=; zSJ`I0*hq;oDV=Mw{jMSz;UL+xcRtm6Hod+>^l4R|1<8_Ijjv`_3HQphRP*g14U1xI zBS`pKTjR+(mV1Ycm-XFtzr71Y>>u0|YP+>ubRnlc@s}Q7n?!VEUpVE2(a)g`{-N75 zAiRK3&iC+%&MhfD`kHrV6Cx3_6NgH4o?$|mHIws~ueUz05A(RQ+2$>gHfwpO6PEc9 zV)xZFsjSN{&NZ8cm^zm-Ryk+HAn6+^ER=GjEo(Qs@BJ+9idj^|V_{=`GYztWYE#|B zYwN~!UNsheSCyJ{!Q-hedaj@#fq?=pVgOIZza`=pBxKa7xLpgP1I@qe6QE5}93(Td zxSVuiZ@4%guPkwUxbt1^7CuZkEE8rnpLCfvy52CEbp@5!H# zYq@+S->^88>V7MxP-R3)%I|l4vYwl--hd2T%_O<>NOe=8^aIpI2+QQ4hP72fIt8$fL#KYHZ%!?-9qa7M( zg>W~)hC=L;67Npm@=d?EdGWirElYMsRct);0{W%enMYwy3cXL!uU5ILC*|Dge_Hq5 zr<@%?4pY8&v*PCtLsG-$<+&ou?B(ZnYBXkcIqV?O>-9{-R1ES?RG!i?0skbEkv(TA zNmQ$t@Bj78lmM(2P#5-BfgC_ZQhFWZaHS^@%(wC~;pf;jhIAV(#+HR|RDf-sT7byW`brVj!o z^+AeC)Xl28mjlbk5lszKhWvxv*wj#Cir)nVIJHW?3{Xo>hQV{is-g<%g19%Eg8i|Q z8*#9U4al+}r=}Ogty0tc$T~$pO^Da9CVQR(MUhlf%~{|Fxw+0c8`cey5)N8wmPs4! z0tF1ULi$)DZE&GjtcK;Va?c()PW*HyBjiID;~cmSg%=}wk<@s8zZQMn{UMwbxrKJN zwnM|vfUtie-Of*fl#+>Kb#k-X&@Lb3pU*~Rl z|4BKPP{lyg!=UNauX5}3J|>1a^K#W0(LLS@bB@YZnNmF-n3H9q>#0z8+SvxOZvuh8 zNIvyLN3WU=7uo$TD!pw9jR^ANMN_h&KZ3{-mpYDiUP zlN(#Z?2t0ENlFt5Uv&Qv)_$hdhjuYNJq#ql18V%=$`{CKxQ~uj=woAQe$zrq<+Sm{q{YGDbCNyt z`Esi&%t_4tyy&+Idoh`Y1$DZ`O@~WK_q;w}OIh6bxEoR=u=BcayfaOKez}DFRRf`w>Qu}yv!<&+W$kdyTUF)( zd&Z57V4)>N)5ZeTDZ2#i=@s=ZM#=7;&BoUj{=+u!_kDMzvupI551V`1yhXngS&DW| zdIr5HeUSjWqyRUYof3zKLbQ!b*>2^|ep*s=rE<;clv#^pTH+8KW0 zv8?RJs;9G5v3i3@V+awCd^03@Cm4^_`{`_$zFNS`o@p@zzcx8@^Cp_AQqHkR>+Xu! z;OfY4b$00b$~J}?(+9iiSZEdGrABF2gzITc70(a6M#L!r;M$97=Zn4~(`iLVBcr;& z4^1j;$w?BQ0hm{YMHU90AqfeF9dZ;jG`uwIjOFg`pqp#QN_ceNGrK+aNKIK5(<#cw zAS5y@P1=>ek#~Z^v%jTV7GTu+?Ku)K1)IXJ4sBIwqC?lSfW4XD~8M}jh@hhq8<9zGp0vTFjnzz z!%yya(&!QR!{)a9112>)3psUt;r_IP(A*0=Gr_&g*VyCD(!6*ls1D5aDdhx9ghutZKO0D~b+;l{s-$a*xcN~W# z_Lae;F-W6?G5I19QMaGoZ0QC;XcZcHnr5$8*KhK!M6C#{7H>9(T<5%vUE6x#namTp z1aT{Z|HjfaD0PP()Ma$MojhLkastJO-ijjwwx2}kzw58H=yV2_Wr|AqGj}DjnRh@z zoic-VsnTj}$D$I=mR>D5Q2rHmyI^xc!nWF~lje4b${?GC0{G1^fTF&7E#vR3T5U1_ z>Un?BT5YXWOQn?%Xh;YzHWSju-E?(yZzqj8EZ!R}_pzrVgcWiSv_sA~>5DtltW@#VF%B3gYxID0g2 z9JLwj{yc7|zIbl4#3JCodsR9bbuIDG6FwrrL-RFZ5H8-_pk}X+UNq+4|b(*7<5ly*HCUQJ~d%vOd=>kuEj8d2HV~^ zV{Yib>Lb6=M)u@z=vKq$F41&&ocmb5Df|FC?DqRE8^Z9K)~X59Du_yL6=+S(knm&c zs2LPplnWND1vVXj8YcUpU6|(ZDBkK!V#lz>%u^m~m{?I*q)87ibCt#-Q!;#&SIT$I zzBXmm92ek!c5K;0bp~SS(zq+PE;&EZ-N3Ytv!-pWdHeYKR3w}k7MQXZ>C+`u2dBZZ z7EbDkh7szYVGVw7dD?NA>~9aUbE(EBW;KrfWU@wiaq+FRzv}VkVYWkxqTsAKF?T9Y zR>+stt(j`M+{JHhXY>1kuPdb6J2&j<| zIprJzSwfKvN*`11Zk;RFyQRdC*CXC{<2Fjq{DpvG5B-^!_9tt@H{()@*h~3z`vnh$BB&{=mM~wUI-YEmd9977;E2eEQ(~D`EVcNFkU9+ ziAAIz69~M(zk`z^^MXB*I8N}h@QvK<(9C_b);ddZx4YXZz$j?LKG4wPvtIYVRVkjf zecQ*fa*X(nA-2ceN*?OGaS&smjFo`KFcedLtZ<%_^#-dY;4rpYO z%16t_ft>P2Eh0CCQTGs`d)6)!Wz(x(+PEGl;o)Op(U&R~-+EnuPdxWWpbf4l*X5Jp zvfC}ZJh)3;kiq)(0{-p-zkv5j&86ORI8M_&bB(h>qsccfiegxf@7CzgT#32^X20@$ z^&WY@U}^`O;gC+XSd?NGh3Q%c?#a&9;GN3BOnK>T8$!zFQ01}jJBB%6SEczuE!lts zUqAdgEVZEdY{&>x+^}4iF4o}PXjh0Tke65%5PoVT<<2G)aS0=d*M5~5@jkcbQ`Fw` z9EvNrGFP2M-5eaxxcoDrfr(}3Cm(2a-@Sy;Hd$O{_^G~pByF1sCqJ=hh zOE(9Es*B2YbZ~dh%hT4(X$ROf-5wu_8|)<9^mk|!vIKBvaBGeNn@?|xJ=JdiaE=>JZ&Xg%uEtmOq&VC)lgwaA!^9GU(IGZa{C?Bi{3)ms;X`aVla0`Q|}nB zHV5arlXznuFt-bYC=EXOL|zI!5_9XGp@`FvD=J@ zsW?h|Jj-8Lfb>1kn6$(^=&3KdZyNx8q|1n6G^fqHL;^j_r|4DZbInX*m4cviy(d(o z=?cPS2P5J;$F1&Npo`szC#~U^eo3IDrA5;4W9Julx45(xO47roS@qTLrO;cuJsS{y zhJ}jTy7J)d_SgwAF><7|;aJOkH(#dz%5<@j)iAjnzt-}j|ICBmC3Du(Zc^!ld`ofU z#ach~gaGJl=Rf@LlYust>md~IH2YfDOd0Zp(-SZV%`s^($9kYQUaV7Ub0z4NV>dqg z;qNsBD7D-zgop+4k5^awAmfv<)~0rxk)!<8v6_I*iw!WdL?-X7eKx=T&s4n#z_C8?Yqz6UdvLJuZW zihbdaLUm_Ms#$*8^=1bu)i+aws#d*IMe3RMuH~7oLpIVa0^A=-qk$dhVRq}xy{NjVxG*NL9IIO$m$ifEAFu zUOlDy->vVxXhsv;A1=nS3wv|Ydm2SaO7}k%rY@B-4;=1c;kc-H!lDm<*~%LAU7?i| z9)5FN7-)RLmD9MpNS2-uWk6TWiViP5DokPry;AR;P0yMQ$n<}~a4YE3j*?1n%^My* zO)6Cdn-<@lR6DB^x!Om08z7}LPMEz5r+|~D_%Gm&;^_!g?>;@- zPk~7rL-&1_W}YD`gg6@G8wp1cj4;!BLdu^;ayBueoR6H8pf%+gVhqgW_ zseG~+ulLhbIF``uLhIMa77%vvO;(uMjB*n6+6?$+P)hZ=7#w6p29AKsF*r(}(96Hf zUlfyi5pzoJgX~d@1Yf{o$9QzrGqlNXg*qAN;A$Mdg zr0ol*HFxgGae^awDXHNFhbJQUv;bQhG)VQ>gP$n{>A4> zVxZ4oyHS5P2-8d6OVH3sm|3^scG%(fN`o4l3z%D1E1u2npZdRk>VsNZ_M= zm3*dj$h0bF>Nw{lE^%;#{X`&nNsWR&sg>BYGvTz@A1P(Cd=`z)CC)@;rAL^N5?|G? z^Zevo*-;*d*b)mDXWp1nH#8@8sGl?&mhmthOL@a$|xmc8Rrt0?Grr#<5G|E zWs^*4$(>%xyzdTvPepiZ*AD(7?Px6b2`r-!1ubT*C$!P6KFI)qG6ux9q2R5W#iCv8 z-UTfix5A!53GdthTi1oJ+M;N&&H2#emk~xOl%9caD(%GtSZ=I|roMXvMa&Gb55gA| z6BIoa!Jb3>!=js=SGX4m&pe(ON$ue$8t*tfe2XFUW$FZ<-&@X2i9Ci%T{VH;23@CF zV}Ra(`|RJ%#dWw{QrcY7Qgt5rcn-;Z+f7awREvPXu&rI|UavobINeR|&(apIF;_JH zdZXoW+D?7@L>gvuOV%v^04w{yd<;Ldtibaj`!RiHY1y!%`a8)R6Hpy8IY6lZwE@=$ zQmFcH{>Hp}NGQ^;<9L)P>@CgkwNa~q^zp?Js>jbB8X1o83i+S29cfg{X}w}7zY9(x zPRE!8J|9_S6b2NvNr=F|;+ zM$T#03YVI~8$8y?u3i0Q%Vb~11Uwql*2mx;kcJr;NM|$I5WMskccdYul^W(PZnHm6yqr@WZ<8~cZD@c?x328^7tzvJ;z>W#Hd*v&W6$v#Csno`FEQ8s z@N(DFvff683zhu75_oA%N!M6PQlDI2rP2izyZ)aaYEjyZEDPRb3IFFR=tJkddlxPb z{rrgI@wH?$n@D&kh(e2s$>jJz&fg0gY%Ef0YzL5temfFZnUfM^rmLbfFCl#~`ufEF z;;zv5#nlUi8a3tD+VJUdsoyT$ni%~4Ho9+pBIa}@+nX*W#%DBxrrDS0ko+}X53rq| z881x&tUt)n*2@p)^-ZRg><$m`=#sSVNKkK{z0s#Blj6H&>Eu&+<&3=28H6QO&mZxQ z&RFD>_K978A0yP=w42uQh5Q$2kUpC_!~DM8Xmjb;7&o%@hrk8k#$!`*j!MNWz}WDA z8>X*_yexezk%<+D9%l43i0%+Q^Mjc|A@fHnS^^FOTH>w7PHu4<;qfn{zR8n6a@{`P3g{0(euxE404p%ha>Y4kjhp)>@d8_QN2SXg;Z8 z<0oC}+d3TXXPgYftd};0jdLMeuluPDE!(ke%6QJ4Q>|n>!#EbOOD)M*@4glQ4T}91 zyyUE+PH4;j zN~rzNX$|;LgYqX}JbMeTtxh-}O~ONQxO@Y9jg$4p+9h#S$oEW{czBIDcW6WZYiz`# z+g?a5M;ll3gp4zDe=K!>bK*q|Q{A!_FC>cdO=IK{vTItHt0kFYU0Y6Q5_Qv0w?7xF z>waMnU0$5_yUXo<1Q<*mN{8*0^!v)*v=belpw3)6`9=)tiL!ruUwaDj9 zUn-k6o=44PWutn(eWOG^)4dNl!!f{+endZfo@TErWaG}Ge*AeFZn$u4)#y?BSn*BG zRKJ5_0EXiKm$ojU{lB!eXaa2g%KCmf{qtKJ-C<%wU*{(9w%}#9W}G5F1hv|N z;3I-VJVwGg9{TO46DCN&tPLF-ILHfKHgJBvFz#IYg+1GBvx2(OYsZ|hWA#Y$X$X5r zxhE1mzS?9`#|MG0ku;o~1Hl=4qd2KLmi=zW7~vL$ggbdIQpJD!JUUACw63APvJ!i! zqP6{X;gi{&U$x0nppe<^&24#pnv{I)(Gq<_L4nm(VP$i2_Tf^CDk~16*<7+Q^%y5Z zg4>_1@%yV5cf+V0DcP&T<-%AR#ZW&{?@CLLdxaEM%UF|g6+I`%ozt(_kL`WL69MDo zeaC;Q_pD>`_m5lrW+^IpWq1~NaOQqhCBEHe{u!3Q(#7~=n_u*&#|s)1rJpf0-$NzS zxYQqPr*ciE75`XQO&Wcy7>2l&(5wBC#&px+vOkixT&^E{oI8%&l_oTsFEKY8bqJDE zG#}X=3-$5=-nR=VyM+lW>RXk3*AH82zOnSR1e_4sk7bm~NCx429s&l{D&ssK01(l| z2OHLM`RF#`6n+RFEy=tE-q=wqAt5$p(F*esY+MN63_al)$Yc959>i?vq@bflT&uKm z7JvXmcxzH#@#4eR*zjR3UK1NqBp^)f0c~+7(lRKq|BtP=42z@Lx`q?nf=dVv39cF3 zodhSi2iM>>xVt5|ySoJl?(XjH?hapb%lXcEuJ^K$s6y9nVGnH^~Jk{dLtyjOqRr@^ecjZQnMtO z)p^(WSz#zI9%w8q`c{C@LgcJ{eI|XH!tO^^Djd(<^X{-I(uLHQ2~G@1_QBzCBiruV zPW49vhXG~ovcjsaBh5Vf1)#jy!3ZPjGJDrBF)=aFQOmi(z{Nsa?N$zD67(iY0LstLF9uM}^kvqHaZ!H#X$7w5RD-9kIb4L!>LlGL*2x=2I&(a*l7xQ$-gSo(}|aV8>l~DCt2DybPNM{X_U5|u>pHtj_u!pW^XE# z%|&U|SG1mQYA5_MEETJb9(19677P@fpXRNvPTxTjN=kj-HekrTVE;90mn*PS@5`^j zM*Nw9w)wkfyi2q}rFBkMQ!K5+Q)T9cId97h~LM5!5DO06>v2OB#Vn@&L?RxSTK ziNn>rA~^Nd;4r#^nmfK7v0%Buk%3p{W_amtNS#f!E#3PwVr(1C3SumTJj}-n(#6Ej ztSmp^n(+qR@9!w=w;vOzxszOwkAK=58}xd=7W7_Y`AQbA#pVNZn^TrH)nZW{u?%qm z8@&`+5>}G)(66YIh4+=E*|CVI^m>>PMjlBFZd5HxbQR{gtt#xUWNf>aVp2eE*rH&_J>J%nZBGFb}<#8&lFIn8KIP3#d53M zk2>=ff#MnG3zU-ap=ba?t0gk5Xw?Kz#o`6dNYPe*^o9!I{@z|+YW^t{PP=%CY$=5H z=^aE4uz{HrKHb5L3>eWjIPCX3o-C9Ic824!`lB3zB5@d$l7(nZp@)G3Y`#pHkJse# zCHausRzOw^fVd#|17lMP&N;xRRz#xYg@VJh*tjXxYoraPxU&0bfWc!SEqMb3q$qrq>3j{eiv7?3C zF=m7IB-w`;5drx^QW;_D(``eQ436~lC?o00FvE+r#o;mtz# zoTtefqfA0m7jeTN{IQ&^7Fy*N&F7Nfi=P;xg*`GvMjbuxlx5Pmh74Vqgm`V?QTAT* z<~tZS#xKs0u#J4JZ{_lHMDE9;M2jf(PEXs;wg`BvF-1VWW;>tx8;McQJ=mXxt{T#s zqrlsAsd>_=ysBXAAT?*sc@)#R8^w6uxm6aOh9cvU)*2fq2uyzX)T^HjyxMKx81&IYI+lhr)d_0-2)JhkX2WSE(koI(X zBd%|;u3`*?RN9H@NgU9`VwnLl-`A>)czj9XbHR^2ma>NWIH#`Uof)&YM==6s5xLnTLhGf2$|w5k z#^V4xAU$s^I(m$)pZpIs9#i??GWN`7*D16nQ{Vk%LnNc%aDg}L3DPS3xE@$qGJ!&Y zvhTUm69xG9kvL<^eA6(01qP7tAl;Er_v!VGUlG9%+393S&L!{z=|~qtD}#LEuDEfj zR#PGgJ}JifHw?h0TIF*HS?oxrz!#q^Lnmo@u91AQ&fL4^1S18o0`lKPEzdUq^_36` z0uu6K(;caAOxau9R&1888{Z*eaARtueum3~M$7W4xUS_EE#Qs8I$v#UNBs({8PY`|#D@zp!%6(`bzI#&T|(k+$|K|w$9);23j z`6e~p>YW!I$4#KO$4p=%G(lf&w~u5fn^E?|?cX58z0cC^(sFjOV0jF7fC&FMTj zhM77)mmsz5MB8t9y~V+PYyi1jZz2&ee0ah9$rlR+oUozrC@2&ed&1f^!k~fYn8Kq+`YG80 zCN`c0p931^^)bE(1-<9^{6abF7J0fVtC`vWjWJNyK-wM+PnrTmf@U)%XSY+b@A~;J zC*pwnivc)fozehYqdbw0k-0LoLRT1l48rrkDKhGTg zt~1iKX>aVM|4}TL^%;Qi-D6km^+lVW9#G`t7+&Za(ub;!03RrNoQtpp9?)_UVEVQmJkKw7!MBA?PauCHxc~#!-*hUa9!FIBV~?Ce}lB`*pXnd6+MxvTYxsHtL;vROVS0A{3NS%ZLxQaH-{ z(Wr((9Fvq1@t9!qR%k9%#DZIYjW;bo10k6&FY9m6s-)78Rm^73-k@%-O zh8{%^?%_=Foz}VA-PW}ESSBw1%&@D=)&8JTsYa)A@ojL#hb!cnJ-hL!xeOt*>vqJj zxpaZ1E$;X1<#F+#i`zoQkh{%3vbu%__275eXEJDV;a}fa=60;f0@ebnKjlAYmVXxz zGOSiquBPr8N%}5N7?B!+HPF0Ocr-8_Ej zatrx0HYtH#;U`i0jxIw1m^Fd0C3f}BAl=;+A*kp8LBN;^uz>4hnPBLRSgNxkJH{ZX z_caMTV5?5MAM1?J2$rG^@JsI;Ni-G_bUI#X(RLQZfg3T@%?NVZ9)2%e9J?bIfjYxj z#$P5zN3FhX$x^8(IeB4HzD4|MM_JX?DlYkaCJC`Xx+7C1&5IaBcEAZb$`IRFGD4r3lCcylpo$JxgXD4rs>EbHn3-IJKA+Vatnq% zJz%id7}CHB8yu#|PxXo{O$F{r5mewA^XF~ zcAH4H|KMdhIb)e0tGUw4d}`cXnwB9-cbp;WcJ-&jeZz@bJ>}*X0C2nLrBXN_;;iN& z3|LigY)>%l)1+vDznC}QZ6}>4NWl%W{+Rz_4KD)(w4Zf`SI-^|4{+40pjYSvA4YA$ zC3iSt5w&^22{`nc>>d4*V)B#{^dHlAcV$GJ{cH65){BLMGR^nK(>7gfHo8k_si_lr zt`zK5@uBvHl4`f^xW+BX~LA&1Q5=9sK=*=jKl3A zb)dfHp$JeD#UfJm-Q9UBy+yayX{BF?K z(P!8>^Za^~Tda6}0wy^g<%Y*(oU9}is56{zH{++6BgPH-lgJ?=VrTIUhRbh8vuZvX zZ6PBKjsPB$3(ZfCYg3QvY5>`KU4%?HbDC_?SEt}(X@xKLfKsi+lhihqLcZrM#8V52 zXDdJF`}4gk&gAX|d6_9xZ|hGhZ&*;5m zZCBSuKO9$f?b>4s4mQAE?PGtW5Pn){Hy@!$DKCfHeKpYiNhgOzt-uI)w@na`(sMc9 zP8P^pV`wv@{}?Lpmaj=ry{+4??iUNyGkyL!D*5h{coZ4@>dnWsB_HM0#Wh=eU0O0u z(x%9DE@5u%gw%5wB*NBZ*B2=k9vA*iSYFUs+eSijh zMhjNyvMUupu!Kf2JesFdFalUi_8P+ApysC0cHcwhw+C`4U;VMslV`Fw7iUn|L zwb=|$8+!(>+k?XvXG*(T_3p==ZlL&&mfULmh_Tvxg#(m?t23g?8qb3*0jWVZz_)HU z$V|%f)kqjG6ImB?4S?4^!^z|m4yZ`PP#e$!!Z2^wyu7@elZ}+>`t-jku&hO6r?DZ- z`*V)LR4$5nS`gZ-#D9xO2KwD#u;HTzj4O(nvl^37lS)}(Z3^eNs8Tl`!AB$1+*$M( z*-VVLYK$}730@+Q81CMA6~)QF8pVgGsa2<$tZqgsyiz2_e3&Jo$OEx;PwsYO+J~E9Wf9KTf z>Rx#dNNxOVx?M*+U(~Jm>?2_M_La=zaDNG|zr?FI5p5ZG~Q=K6D$WslAa z_AE*P;m|Rm_I?WSn+jZ6X8x|q@zI8i^ZFs_TL(tPtyV~9(50cj z`7cq4l~#1faYO1Y=Tly1zW(fjj(%Toaj1dRy)cSy=s+ai@a5U%7eHAE z(!p5(dKyj-W-JSoOErwT`@J0ss%lRWp`Cv#RtZ$2P39GC9chj{eE~d2>K9jGM9>C(H$$TJFWMFnQw3k&Av$xCgz#>3S>NhYNjmc=tP4%EQY2XkJRG z#eLznt}Ddjds*YSdY5b4tR97KwtCAm(RBHmgbnOEK$Y*i^Rp?xplyzB#cvgrpXKkL zI!PVB^&wT=M*ZSe0C?|3I5zPWwqEWsr?{Xj6+2yPXE^IIXVVB+?4w?zmL=7<+{sAODFP6+TEMgGK>l`zC!zn(c8m8?&1{z!-z^MD6>*&b0~Q?}$FP?J@>5lCRyf^upS^DszJ z+T|~tnx`)4n(eN?nUF0Lw!L$PJa}-{*2)#eWz}m((yF$So31pGxbAo)x?LZ^0jff- z#1oh_8liNF6#(O~Hq_&K%t+&r!bC_xa8gjAU6zQJlT@ucOUl43ItIXPMFysEs(w+$^~MUq0}oK>J|ZfPhqs{Xr~CR?@+=KI2V_ zotoF)z|_w6kC0za_!Iw6Ov7@FR$o5BC24HY{Q?tYsh+<-FtzcAR5AWCe%Pg}?cMR9 zLMvJRw?H9x#+W!2RX$M%2- z(%IRi@YG3(O5P6#g?ji|`NgAHE=p9grtr}3p5gSXJbA@wpw)G^C);F_vPEUBOUuxT z>y6aalvY_`tO{b0Y$n9PXeA_{(=M^ud@j|4dACuqte4ewqL3~iKYP0<8Ka(1w;j&VSx@A4Ld2X#cFxVT%q&zvyx6Qec$=z zCsQUMm7;cs#lWt~jw|e5XY=I%O&j+_XHJ0+=VdZD6PhnUlJL|$ED^b-!gQrH3QPiX zzpEA$HN6lPpSs`|W$URTfP`vWDl(8hUJwa@G_0iFWMV8!zvoV+Td3BbGE*>Gs5LiA znzqojH=WS|++m{P+g?7L zR_63q`c+|53*DovO^{1&j7+=7)|$+zUg9RXt)D4ljpoM*Eg7#6-O`=g3&qJE_<-@z&!+}X_dGOumE!Fz* zt=UK1i&|l3(Q4-7Umnf|$?a zcl|<`gX8!nl%08bTaG4k>x}sQ#6Y_97vZ~DLW(=)*LXdun~%!)zik&1)`D^et5#6N zZ3JgHEVVc(G7{oMlkV?(p?)7OwmUa22_11{=iTc@;7kR9DEbXbf-}n@J|W~k`#mZ~ zC4J?6fG_(9MU9j>#nq~w>u0sa{akhUb3qvT<44BykvF`CWQ+u!ftjlxkyZ$R1O|#A}W4;hf^IPu*`RY1l;jrwTM*;Fw+wY_K zZfv*OFGv4Xk}Sk7@|!Bryvb<8TeM`<{-WeS0N%)uYH)$iVF8RsVkqs6h_(=0!zz_NLQovzj*hJMvQl%Jc)&W=chUA(-^- z-5%UpnwLYQHI@<0i)&Vlmu{o5KKPEu%Rl%2E&T-YOMgmdP)o7);j&%irD%-)8R~M| zUrOmN@*e6V{l8F?M~gj-{|*z@34eFzivIs%O(qR>0eitSo1r~@ALTomHI%cd6#!++KqlXax!wV)Y3!zr}^#-?`khm<|6Weq!JyD=CAr;WB>WW*Q6nG=u{*-=dtyH zIJyx_*p*nO8=;db&&Rjti+7(WDFbVN%*D1mZ7(d+KcjA43{mu`(dYr||1Mc(hw|!R zI&RJJWI4t@03hE7=EE6QsKh7rNg?n%5s0A>45by`VcI%6#!VS-8UN_gkVr=MM&Bao z$SI1`zxxy%)9M<1WI6|kC`Ykf|76Y~$djS5v3Ax2iOedyd$AK|M( z^;rS3jY-jrr#7bi!B6qa;2T92YF|~9m)w2tegu?_h~NSpJ`0rk{?()El=fX9N_GD4m3!0+Af3pWSdb;?Y6ZbSCZ&J`@Us-$XC7W192H9C13-1_R1P^z?F z;^r*V3?&r}=B~Mn;$4!M@RQzx?Gn0%)UoqT?IheFeth51ou z>n$lBjCT0UucPPi@$Q|#PG=ZK(t+>0uZfmN45d;(u|eLuuI=pKqI6#vT*RH}iUe2i zz+RW1D7ogf5BRoQ9d3-Y(PAh1ys^3{9R1Hw- zo~6=LbW*83nkVmg8Av!Sey+OO8efD@la`Hmbor>0JrPr6*GAGN8EBqRs9*8xcA&3R ztEJy!v5{nf#b^!RzU>!urMr`zfuJ)91UUAM{i|q)xk$Pbw2Qfud0`@E(<2Yd`os?Y zY%zt+v4jnXkQ)>WPnUI%floYsG+smluv^U$48)iRBY4@1{SyoNHh2-1E()qxnqITROD671n+!rHi*p3HQ(`&+zyV)8#K zgLt%NRpnilF%nffVobrIP^*0)EG7=(Izl}Vq54$DTE=3Pw^8aHs*8Dgfi94UR0h5o z$#l+7Je2a>7Pux%3pGh;(!o*BCu{leA+_Nng0NwEiD$>OULl9xm#xrmHZzplNZXU* zI8dKC^S>mKUayXRH)PqN8`)kmbF5!4Vt=H50MOUnuOeH2X@mgLBr!ySL@R@AjeW}Z z`YqKV>n&r(9#B|m7TjH6AMfDc27A7`>hpX6gDy*T)wV+stK~xG~Ng14n(U<&z znh+NraMqj1Fn69vAy4*2P@YVnM?;KQyTtX{wdP`uA+2f9w!sL4^y1uapm=tm>x=&O zpM>N(Av=IX{FfpQaU2Haw`m;@!0-9@2K&Vns&)_7+R`AEHwFZT)*vEVlY+#J%;2sV zJyFTW1~EaC=L>ytm|3mlQb9#`&COw^nmxTnLxfdGh5kgU^rg>@nNe=`?!JQlz?sw+mXnttz z!a+h`F*f2C#e`2N2`4$Q5Td8r6ld(05bykv85-(7?g5_PO3?fqVbT5gupa+t$+xx9 zWryA;Zu-E4fr}?9;bG@w<2YkE-D6QI{T|@w_wqpHQwS5gK@q}9J0WDPXrSV$U3y&O zjX^5}4s$7IRteh)st7I$YNT>GMRd4;^V%g?Y0pF&zI-RgS3^&P2ahV!lLMI6Wov^8 zjk#?y7ZwFO$)b&5?btAog^7Fh4cLnLkzf@IydaXw$V@wLm{P5=yHJJ-Ij!S<4IBwvrL0dth0;}X4!mUC z49FJ-CzdzjWAE0f<|IvhcCw~zj6262b*l+jOC5%Zjf=-+J} z;JbMPhl_40Y;644qkAOlAR|VS;4x`E7QIyL_@bT?2Z{kT7!Rq(0iI29xTlg{hcx6# z;=dt+NGRW3{i}5chnn!`PcZGE;<<*{hAUWA%ZJHoAy2L5!S}+HlJvGbxMa|KQ-~Go zh_2nJFeE`v@3~{j@$L@hh^ad`C0W(Gi^ILj?Jm;{MBR|C`eF3fC6E~s0W|gY%{Rc7 z;*GbwGrLRlZ!ph+LBv^voD3mv-!K7R&lxR+gN>BO-rfMbPLOj2^Y!L))Z=-~fMh}D zY=}LrVG(0IcYL~9jv-bI?ZNvxmpVu;+`owzs9wneZy?G|-5Oj=FSbzVZcWfG8dkal zecN>(27~+rB1Nn5%z{i=2EEPTzIJvHQ|eqhSgPmbe~d8r)6H9pFl5)3E(wl$1C^(3 zzz5ryj0K}uj&-0hh#<4WGr)s2Vw;N5e$nlRhfgRI$RF4iO6+GNOU@lTI|+N0%mohD zfmN+%zD_|<`T;iAn%$)!i~olE_xr^ER|8S(8{d3G)JR4RiliXY2_XQ{bIzAbwL|ZH zo+IWY*_PEOAXCN(ZQt{FTlBbrZx8 zC>X>UcZ?VaBoJp%#Mi{!ED;o)6tU%kp6j?Hin@> zIq2Vwla-NEhEy}?`s54Wt4r^0B8~=b30E0dRS0|G+xpS_Uk-dFyKsWLyN^zh=?5|A|XpRtrnEXc&tT!C$i?Lyk zTm~y}7y!Er^Z#uYcP3V6wrJatO3NF(cfu~y$fz{eo1!SJ9LaMX&XbDo4#A{u|KCn= zf=E8kFjvcL+M;23dwpIf!8NolXn>5%QTh`WoHDTcXk(<>@ z>*jU=+{^YI75^LH4=a&$!^U0N?E9ZfHjqn-z zx6`F&+oX4QR)NmvmtU1m5}X?+22yS-qIzaKO)#e9jb!?`!|+(Q1HuV)Uz@C&?6!Ut z?W6nFQ+KRcl_%9&RK8R>5_-yEYT|oTDUV~I3&JePOl1D(VKsHBgA_IAkjA|ee_lX2bu~a1i;9M0R zmTHCY6-Hrd{VIaYow3yD@e&#)Yvs}X=c=txzfiwtBtXHXeSS;c3dNspmC?=&?$+Pv zdS=*{!ehe%^nN$@)`DoyZ!YpE7I+J|6RF#_wjTX1ssUi(-tIoH`w(0eb`s}dxd>M; z@T$M5)D?^^-xDAIzEXvj8X2vKluaABw(6gZjaDysMlU-w!1JvY3V8CmBq$Z*WvTl` zR$ve~V)@U~*VnV;h=Pta8$w1;Wj9VMI-nSKe3>vh56T+p={3V{^vMZYBg{OQIeEsu zmh>F1gHF8i`rwxIEGW^yU4^2mTnv2BI}L2>?i```0L0Ns6Z$jHf2Q9vn@654 z&)y)xb%{ve;A*=+ea zY~ul0U`DN2(~nR>zWLTaBYu3e%*~qTB?x^}I@4)Cnab(@nS8HH)bDuYkyyZR7(y$S zM!hbFzeecawPE!DW#jf0I}z|9JAd6B4xQ zLY%e?xikEv9~9o4c!<23@a*4#1uT+C9f(KNcpZI>RcUw?Zd%MUWnYn%Jlnblq{_n& zfspMs_6Jk9Cnt3iN&W*MFMYW8{!#ehP1pf?D6odDH~hQ}ut*z317UazbPf$f4a`ws zkIj$dkr;5m;)hozA-ZlW;3|*}62w1e^a1);?oNk?i%D#G_C9%*`2odZDC_Sqt|*&j zF2uy?lwDn2IX5NVN3UxM@pUah?=F4>#R7ts7)S5$lmdNTNK1iur;hJ4?;ii@X?Lg5KnT{+b5<;g79?AR{fD zW31QtBJM#gpD*UfWmrT5Cdmf+?eXg+B|TM)&!zrn(psU;x!)2JWAti4&@xxWIi~)2 zhi9`n*LK$Wzca|MgxcF-UpYG1fy36|zs;Q7j=P^7D1k=GL$dE2&6o_sK_HC`PWH;c z+xx>%d7!oB@IT+L4B)rGPMi}8RR0>qPxbO`_)phC_mq@TzDj;(heZuus)Nkyw>7Ke zolioTEty&WT~vSFY%dEiZrh0xG3uQCPx}8EqQ9>|+=2nTA}c3v2tA&OTeJT9w?61? zNSELU_XO{dlD6|62b1&V0;yDo-G|Gg1SQxznQ2Ubw z(kMKW%$g@j34J>Emt~#F`V=x& zth&)vyCN;0;r^tyxw9kD;on40twbvpMedf~X$UZKjQ1x|x&TE8#S}XA-?CfsE@$WG z*%f*#Xd&1`(OZV_PJB`iVR&&tfHAu2VDeqi?OBhdzIH>Ag9c9`@F0>k97e;!XLCkb zB3YtW3f3qnjvrX|=~QuvC&#AR5^+>Pgd$e<2Q7Bp5T3 z?^%WF@#%@O-ns+E%{KtsjgZyclmEltTbldp3gy$f!sj-0BH?@4v^^>^(KgvwCZfO| z)Jj*RRQE!-#ZI3~8&cW>@ud{-EFGI%^)#4l;aRs~JEz&fnJe9yC zS!n*D!b6%F>51TV+JR)U9hZ!`+1!vRYIGg2OrTwRLBCu98kRjJKLx+~7aM}}|XY}wbliRx8RDiXOE6{Hs zzxP#pndrLt11xueWz%(RD<(6FBOnc|A9j75bb?TRdw#)Cvma9wRU}sB5H`=OGYA!f zf;@C=nR_4itznhP_0(#L;<8HaNwlBv>Fq=;N;PIozjEdX@1tZzM@@k?*K*Z^TG~u! z{F<~-6uHYI` zz*rb#*-}BIuhH)4KKu0gI&&xO#HjJ<+BnN-WLSK)ZITfwI5aeOxoL$N1{q(+?qE8I zTA4ng*!Q%bhNrVwRU5C#`5iO4bg~ZjvlGqKwpm#drJv1Pz~DTB$#@hl;iWiVfvTCd zn*deTbG{ltCmOEVHkuzkkWb#6b_^MxEPL$TU!jUyEIO6NZiP8tnU49wPlPe)@ha-c z6^aNHghu=n4I(QiyQ^b7jqzIFJ%vEH%ey?cR7lUs&?tEj2jcpbogXTx0q8zZ)NG;rsn8@bqx4ykS-H z_9%_hEjZO>kE2QXMay|;pGBm^w8q!fiG;|KXui&Xtm$?>WkC9*^H| z9rU^*#>9|tGvhI?9uD6K*mxpPXaCp-IG1rboyJ>lwRd)6{)+|l zD|%an1_k9{P)J8z?gxD~5SQB<|Fm0FhE3COb>NW~82H6>H2sgnyxRz%s|ohwqSRyujlX?q+gT@e6kcUL8*Q{wR8Kw&^6 zSze|F%RrxRY{zlsNp_CSWIPaH%DD*5yyJAc#|I=Um41%aO(`ZT`F4wBFAf4F& zum~1P^Lz7XYPn$lMqXk5>N#`Ptv;dc`c+%It9!+9U(s)o<3PP>%EuSec^@tbkXV6t z+F9L8<~8}VeQ+S5NUbD?g7rdpI|CPiL;m zVfWp;4?BoiI&NwR`<&AcM`bR;@AW3}a_!^Rm}})yl;E}URa=eumoIQ4q9Ws5md#{Z z9(VR@duv~z5in)RPwCC)E5A-P*p5gmmi5=&k#JOMufCOM@@CNjW_31JeH7379L(225p5LYEbGBf?w=RqzfmyQe zA82xCESB!GZl51G&VML2EfLd@6suP8Kb{Yudy%2986>kFkx?DT7X9(@4vwS^q{RX# zE*arYUaePGSE9@$trNKD=;CSa_g?W_H}Ye3){rsE>7GP7)JNAxXKoe?zdvPI#87Fs zx3_PaJw{;0P8O@`Z2`5qENXxa<~9DoQCL{PRma@`!qMvuTOX7S!dtpLJKu*ksBdJ% zO{V)nn^|9pa6v7}CXt*W3|=h3XH4G+#X+f;>^BGX{S)w(x=GSMF1O5L@yfqSQ(ieY z$Ekd_4Eik$LpAwbRH-P27&D34VtxEi|2Kxj*y}P3VWxR8PsiUEh$&O^;?+KPZxM+|;m2O~72O4gW7=91sAqM3Q9hL{T`l{H-@p0{VN(9QoI^vKOJIJq^3W+oc8N~c5Y-AuWDU%xOJ3KCMfPKOo-kt^ev@nT)r*=7{a z)Pp4{8TfoX#3%H4=>d^VRa<)@QP@5UdIZm7JavQvs%SWYNpHpY8Ai`LCzdvdRELk!Ju)$D2d*$j(lk=v^q?XYQ-T5O=aUHzknl|}O??de5!qFYLl z@5${c#?$Y|H-|40AHGLv-eA>6@wBw;t>NPkidj1+-#>LUCHzFi5Xw}dLjw%{=1C}f zt;CH!)Hfjd2yKH<&k_FAcg-LU-_j0kn%5w77-EQ8o#Jjh-hJPnuSxlXEE0T+3wY+F zxAI%wU875|CMo6sJpaX3mxBaQ&s5F+cn-l@2M1sss)({Lv;_DhGY)#%N=EUx-g;6` zo*q|WKBpm9sh>{WV!1*HIpO|; zb!oP8O9rq2qUfEFmj`zQprpgRD=erxs2dAta&g2(tp=hKVphxLO!IGewNv8fK3_8q zWmAQwE6V2r#~<8DEODNHLT#?NRu~LmC$d?lvPM9Htm5u^#@4 zhlcjcfV1A@sN8y|7smZn(M%Xe+7Q~t3K0XncB8qI3Qy>p+Bc9BrG%#Ekso?|q=>Rb zR_H%d28WuY$xH;aaA>&l9@~y>ud!CUELbd=cW^(SEf9&uASmt@jkKVr%-f>_QUa{^ z*Sj?R*z{(?Vq)@HoMDsfvM249*o74v&N5V zB-_5cEK}@a+<=GAS#3jMp?~Z}G@J!xQTJGrbex=RdJ$mi9FC9uG4=e{whiAWlB27r z(|+tya9y6Q$O3W-);xyhcv{}OO*iZFO(vn4j;)7F-X`H7VxR&=b98(5)LkG?5eDQj zW*{SJ%sBcKw&7GB3BcbQ9ThR3p`M9W8-SF#u6Y!5<+3=dtv5uov|DgQ?#@Nt!L^6r zjNs;{OPbIbYAtn>NXuvBl4fqcgEF2+k3@{iC zjI^z9c3j)N13y3W+O2oKWof>(j{3lMzj<7W$K`m;Sg24SoW%B$OZJW=$9mxGy)h9h ze&w~Jy!#v7$wHCv%SP>1m>JfVfc35rE};7awaW|lV-qZ3C5xr_$Ie8|@HyN#K^1o< z#s%JPca%g3QM}SL8$g`a0mVLMEx^je?QFfv*~zKS8ik_EL&`r#BJP6^ z%)4>12*MJ}mFA`Q>{jU(*G51y)IR>5?e^dppctu2aZdc$9Zpb$Lcpo;vg(6!d3CTc zBQu=HA|dAau3dtmY121QWRd0aU`A%8$)(n-y54qYA9xdI^s*1OfQ!wnIBmRB+~GQD z%wo`P)(C0~C*b1DlT66e!}ozjnU28c$n4j}w`_U(&1*i(2<*Xp(_;*idmp4zxlPiC z-$KId0)gM8(MW2^&GFI#`03srU2qhTdMoJmat2yM3LC=#%O>RwjY|EXv&V}OE;;wk zK(t_#ed&VNH3;m@ZpHw+F!_6ctuN3+kDHHp*t)n0X#rQh2B3E6rZezv4p6AMItZ!Ne!TenDIQNtX*(%u``0PYv4d?%{u(&)%{F$fw4 zoZP=s1V^0`0;e7W=UH=|&6YT%bvEGlr%ePh3@t?bQ4h0cJqFxCVZUS|^H(B1_eQ`6 z-qFa)>haR;67g!SiSRdNi2GU-!Wq!o-{k}(tCJ6QBdI(oKpVQeTStI{6H2!W&HXbF zE6c>ut6-9Z;+>T2Z*C*bce&^Z3wI7-W=3}dqID}D$i%`W9k}8BING6((cd~c-tCQP znkFR;-9eR0(t=$21VJE^?e#y|P8YrKOFfq}I9g5u4}ifTBfrH?1n;PwfBDbt0a)4`R2}c@Pq+SfMd#FK z&$Tj;IrL}d8pUG802ugRbl7WV9mQ1QQ>YYHlvd;TQt@e|g$J&oX%aD(3Feje)@%(@v zh>RD+_Dc33J=EypNnn_EZIW< zU`jPNFq_tQdpE&7_bV%5u!K&;rNFCwx9=8H_Tn)^BN>;pL74C@DLQY4{f?c$j2@h1$xea3XAw8Xv47j-iG zD@IxJg8cDZ2^9Ok3kTAR3mw3ZE@gF)9lMG17WL;i>{;_dIU4Te0GrdTAz>FmxMGTP z;bcf4C#e-{y)?|MM)jU8f=9=XZzVuq{cDcBB+0GB1Tox9X3H~%*RJ1tYD>d&+q0UL zLNz8}hkTOT0%G{tq69ejG)BOhh_NZbkc4Em$fR^0E|c@yCezi9~JNXu+{v08s3 zNn8Z4Q1I34#oIrI$ zk|Mv+qFdew^n%cl1b_0$wApIS8?wi2rj+BX$>WPr8F`@9cL-dv@6MpJ2D{7Mu~}dP z?M&rzq(?qT2bUOjt5mW8xAVGYy^23ZHVJ}g#M z%lT?52+?1*yFi0aJD}2kXs6i_#Qi#D+0t*82YY+JJT;s&BetD)r^y=lA~eLPA0q z%{w&g?6VcX3Xqj|B~5#s2onr;yAgQE$bYebUD?Az#iDi%kF>#WUl!`D^*Zk^cBT&B zf>I*Dar|SMB0tVM8tWRl1PO6pACcG*qt{mO;-?tl*s}9MNoPaZ;u?$T(w~}j{i?a$ z1pZdL;Ia*MVmrKC}$HVx7$-5t{1ozfy8-Q6G{ z4bmW8(w!0_-Q6A1`7J!>+cg|$yzg2w#vEfzu#J$*W}YKH$o~0SCpv$1%f+na zS71Y+X%~6QZ1$@<62d#HS)|I~!{T3QG7uU=F0N@1)dXJ4cb?SNvr(y+In9p`x0B$` z3)mH?S7Zsyg5+;S#N|+sE1xN32=-D>b_Ae=8FU5~{*Y%vpo+$Xu?CPOjb2OB9Po?E zVpfc_23Jmk8k%ju32r^eNmuUML zEjdb{XSJ24ZY$~Qe@kSuPPRW@ZjS_7xgGF-%fl#7IX2j>J^T4C(%T#AqzhY4AeBoz z{MAbN+<~NY)?FA=%Lk3RkC{rLV!i7>W79AqB*a}v;hO*a;*JC_mfNH`v=fD?Nk7J` z?yPpvb)dp>iZT0rER`WBXnp|?;8#XH;XlXrJSqHMm#=(ugY)(EEjEoN{!k~VO$JtC z4d^Xqe+A`;?ozF--r&K+uiU?3zCKDo_3Typ;O0<@Tc(X}6bdro-+$*!@WxVis&X6M zlK@={8$b?BzY#Y&9m!6!Y=C8jJsy_}`RAYp87(&uUfIZf_|X;)YMH!!g@@5xsUEQQ z8`V8yw)OF$s2_z?!Zk(&Byk2}kF%K5xPE?lI3`Nk=|-nAS|(A7>#Hlo>!^N7^lk7J zR9q0}Kq2;>W50?_y+pM{F;`Ns78``P4Ft^2%mgo4{bO+IZW1$P)3}hvqAS*l@7}D5 zsm}9jQ~y2Utr*c>t-#uS*TrFfdIWs{qnt+B6VHt*Dz@k^1Z7O%)5yr8L{NC9t_fCV zWDcitj?ut>!wSsvaBy(QvS(g6|4^a+;p%uLl2kab;CG=?K1Z&=Bg( z43;tR8K~$GmSmsG6Co1AcUa~3tvIJ6K#|22i8$&<5Xd$A!z^W zw?RKg`zPP8%fLA4d`RbV3k z*dhIDVAtr3!A%Ly$3Xi0-Aw<-yxNBpT4 z=0S{9Ks(v1R{OTc?(fz3inZK-l<1A6;-o!5qzr^hD#EUw4py^%L*8JEw3&o$?svPH zN)uX-!Nj+d5d`eJr|93Jy)dxP@sw=9K|2(2+1aQH3<$vVr=d;cb^Gz>;r3`MnwlST zB1es7qOe@g{UiVz#vVlS$rd7&Vx5V5?O4XSgn_a&BBbu2-r;-VuP zCw=PQu~A^t)& zyizJQw#jI+F`pst;#&i z=0bYjD|QAS(W)% zp2d_ZqU6I|v}19->E8$MNfk}*Z`$q)-+gtmRsvSEmIwO#{|5WHmCkuKw@|*0RIK?3 zx%!qge$H*l3&{Uv8b?3(cN+PL+6Slz8xCfws6*jR51TyaE&qO?4Jr7*r_7fM1a)85 zK=K*4h{}EFB3I3@{P@Z zDoGMN7*3%NRAWrCe>np4tT6L$o*wsqh2npM-<5#85BxW;u&KlcowbKNO z`$DkjWl}aD3xJ>H^;j)glq%fgq}E8{X!&Etj$AJ`Q=O8pPwr!^FTL+DWJpp?Hw)h_ zfm3q0vs1WYS(@&$nFMp?xsA_i;rzpL>Q^netTz1jXG+hRdD3V*$LbFtYY(UA>u%mh zKJ;}6qh*S#3`Y_lz?y;wMOic=?$0crK0W@fy&8x4 zH%^>FU;J3A((LuF=eKaENnK~~bg-UIF{ihHwsz_tpcPtafH1L*PzK%MVq#*3 zar|4~>BA(+EMEkenqE?1@AnnjszRL}VqQDNq(o(>I(IEJ^H5~5%e z8?(4J+9BNEe*@;9_CnZ;((am;RWsw`8jTC3YdlOYtl8Pwz!-s!$MfOZtaiKu;`5!Gx#%>H zwI5f!AxF?H2!n3=_c!N-d~3;{-3J%lz^AYt;wkjFNPg%@K{C*&|FjXD4*-1_0iK)f zY&FnF;L`YiYpPTo11(E%csNdS9#T+eX70;2k{WkrTd)`<{pm|EoA3<=fs@#TFzPI4 zZxS^_i&$z-D0e%X%H`PqGj(*c>+eBxI)VVcw9NVS*D{$4J%XLOF;EqJB|S(aH46Tk z3Q2d}Eer#86KKr0I~yAtR=>XqZj(pxe9D!GA$x1;YQxW!Ci^8Txv9B1BF>kvkp(;NF!cLuwY4&#g7kSEXwIv z$)zyNG;lNxso>Z}Nhbf=I)USI*lx30RXC>m?^@3bc;EK@ea9nkr*XdrM{?|)w}4l| zZcJxm-@<=EP>$F^sZX>$xUDO~Em6knnoEzw`i(Ek+qfEw{fq->l|4 zwgR6CPn4JR$5&@#a?SjYclKofN72%6HRya+3rg6VtaK28TmyoFP;!VlKXuXWod|O6 zK4qKxXitDuSp)KdZif452>Xrw>H2$>;iq^7aN<9{&GcJt%Wcqf`<(w@-~DuyJ>W*M zl4qu{ZFWkrB#y>4WX2E;O96a@2`U)$`@OQ({anfTPiPPxj=|w!xku1mTZ3yz_WCt2 zPX{$7-IG!K$?Zo{ovdqedVV6e!n|EZHso>2Q zN+}*-Px30j!OP2w$Q*IW{ZE7$zW`Ad8C6kcN$*q{4<>=x?UC9M=t9(4I3gJ#k zjYtTV>^{%4k&tYhaJ$!GV4qJxG<)sJ=BC)F{Z766^(QB1PEtMt@_|>ZmXpdCpe1Rj zR=XtG)vhoMxJBo`TYi$xhgDB4!v_eIXxCTnYNc^GDs$!#^SX{Vd>A1^cV+<(>;R0) z$Z`sBFp|KVtKy!G2nopyL?!6S(s^nC|Ic$LDZ@NIJsn;DQ`Tz~#k;9gY5YFLJt+Qc z@sUT@v1_ISVDc%Ui|It?O#Dd;P%6Mj_u|rbw)U?){o*pt=bLU>FfMK zouq&U@Cq%8s-xO`%;xIq>J_+?kzh^TR3{29W3d$JcmGnB;dEk~f5Cu%QnWXcF)VcB z0%Hzy&jaX8t|Sz87)5iV9}$N)Mt5LOLd+~PfDa|zXjP_EEla!5pO+NZ%qyU)a&SU* z5`s<#9F{O{yKEy=DxSE)iJ1Kb|}#8eRTu>Uw&5;-_a;#loTt z?dPOP&H75j3CYT+HK#Eyxg9lEnEgtv(AT7!)c;{M^A0~<5wT-^7qn8T!VhHI0Jy$< z06-?sT(voNv*VsBzyXxxcwf|jAH4+7pDQ-`Q{BMX;cTlIA}wpru0c zElo}V3!_fMx_bR5lWCB`vphlY2nOrBS$QZZx4>RJ3@t+fPt<%R*Tcbq5t z9mV)skE|F?9TXZ*0LG|X1d=P40 z_2(;G*YoPKeB`v4^hD?o z)5p$^jx#W=;aXSVB`4#p-y^q9eaDfNogL2%`BT34vYJUrkr}AM7dl|Ax3j=yw)yAEr z64_2Sl_uvC_gbq0_v_Qzk4vC&H0=&Jn605(nR#2kR|R^UV_+T=0Ai9PC-dQCjZ~>l zu;7fD768OONLUtZWMkupdv5@Wo~;UY5`q^~Uk7-y!VgjfFjY{w5fxsK53guAgrkW0 zA$uP=p3f5>i}7DA`z?-guEYBRzFu;Ft^;CtC_>$XUeXnqEICmUqd$T6z7xsmHsMDQ0E*IU7&fE6OL2)(vRJ20f ziwJmuGo9BX|4jfIg(pp?!QmvS&uBrr*0$hQRBLxG6e85sIlAHt+*EC--{$ zGsAM=fXAnQJ8wVyq76r0%`Hx@B>t?a!u! zxu~)7eytMj{L~>X7lHKna$$gOoK<6yz~xKABl9rN+7tj><$iLmWbvN%kuSbhYxVTF z1o^UlMbq2sDV^n#NS6zH;GtF$6_+VbNq;(pA)rBo=lKejIfmbZsY}P@!5l6&CT6A- zbduruyJ18mq!Dns@vr|3gWzsowAbL{RtaD%ws7c6vaWm#r|}KrTfo)Rpuc9W__)%+ z*`JccUj`GE{6AT#j~3>%T}$QQ;J`WteL6iRCgw_#O(9bVzVYnT`6qfE@9}YD&Wx#Nd7U~+mp{vLOWv%57EDib z-1K(i9X-8tE3j0l(r@=wrAcifnI05dziMiO_Qh6W+6cPEcV+=IFaxgd=3&|HHq`>* zdkr@FGRpunz{)EiTY5e^mX``wfS|s|I>Q^mVg9N=L!H(UAwlQcwMc(sdRCyc6s52XAiK;b2#pnOAvxdc5 zfqM1yX0GBwp;Es>KwNHJpt2{oyEj?F zL9gVFka{Om&|ss{)&nP2ST>UJ7r>rC|>$8;B@uG$3Op>)^8>-Nkw{9xac z;*dBhK_$bOLNuDIw>QH%aFZ*b@2FD>Vval!< z<%&xtcqrNJq$xq(@$6@NP4mEuo3F*`Dp0=3d8NXACN-S?!sRDCA|Z`XM~*&{ABKMf z&YSN|IF@*>XuZkYaS2(2w7hYA$FA*rue)W!&*Eqe!tZhfL$MKNgZh~No!Fpm17fOC)uTj{IG$!OT=1lfJ;AG*8A`vpg$)hV-NIQE^Kh z?$_8Z`Th`8=7%2U+Yhp3@vAzzEgaqN-;LzrD(KkmTuV7L1M8$$X1WCC*@whGRr-7= zKhv3h%cim6#4tK<^u;qc*M_~`Q;(brYM1oYuu=$CdhzT?dJtM7DNk}rmg^TzipeBVuv{Xgv+Cgudf6@n3+bS&q)e=lf8y2TP-zD{1U?RXN3U$m?L4o zSUn*KAj{PJIq%zFFPn?-)I*gKP5NE{To$T-!G4$*@^dE198}ORr<0&9v0|tL1m{7R zp=%_t7I<#pO4XzWI)jVYDJ>5YsKy*FIQ66P_- zm}3}u-Bl3@y2g+z*Fu^1jkRlFTSnaw6~xTXF*gRGR?NWbLxhgtG9kvk*>knoBu5qe zu0W{#2A)T@ji`4HNfAmCAt!S-s5~GaE8V0z!qsm0r)5E&n-TsG7u4To^Npi>bRWxB zA?CByh17?*JJ-soxph;!6GdBS=mo~Mf{!`gtUw`DJ&3p~L(fdR%Jqrer|q4Ot6;0I zE9s*Up`uxvdm=hOvDP8SXU8lUR~}(Ywf1|w-mQ(@mM@$>0}cnW9OkLd8@76mQ4DE< zY0}8*^;HRBZ`4S|HqjiM1oZ3%^H^=|&<+e{UUIv$p=vn>M&7b11%9hRj$gUIpU0w4 zq9M)$09GVKIef(io~JF_dS}Pssi<3d4}1Cr#iT#3q8+GFk8JNjihXQ@)H zxS>9ukVqE3hjJ2(F0jt~Az`DboJ##cw(vYro%#6)a^&$g03nt}z=CMSu~KGx9wb~& zX1a}rzy02fl(*JDFo;i+5R?!$^qLPM8+-wet4V+p_Cd}QQ8dpV4{j7-41@re3ii8X z5^yCp(WoIP;Gw!(h9;&yrdSCMa+$e&6(>w25le;|q~&Wh5JFRA+BUB8DxO|r6u6h! z%Q-Vuyu`E=$z6Ufo?c8BBhXb(BVX|Yo2^oM9(b#*{r1}VsfaHyOB$AwuN5JOY#5dl z)Y5s2#8iNz5iF|3cj_z+SW~rX|e4S8fX$02MG4U!Z z3vrqT#R1tq<}x`UUn!AF`7u2|UtgZA$~5h z+VI{s`^EyxMaBvvX*v#B=#{SXj2=SQ257JJ_c%W9tJ6X)Zc7+~u>~7b;{2 zHsYM6t4D?Gp<&V;1tn&9J#fOLpB7{M-lZCF;L?85{`?;-picxwj_1Vx^v={Uey4Qz z$t4xzl_CZ5{v9oRx!F@CxHn$7W+1C2V&d%FtB}{&DHaMvsY#-oV>VU=67FK3c-u|G z6X9W@F;)^C=X0;49|GnE35PN9xT}u)2K+84Ruqsa8tTJJZ^67xg%~7Y6LGx#0$N^O zS{Y=VYmaMgIkVe+U(S`FXHZW(3+MMrK#R(dRVdxHE ztL~8R{pmMd%NFUlElI$J5{k*}2@g`=a$)ZwX$~@QUJhb^?kmPO zxR>}TSpQ-Ryex$dBnTIrw3olxd%o>G;6PU^Dk8Fd3e8Ulp}78N+eYx=tIQr^MI#_a zz1E1NKscYEQ+Eo|QOgyV3i*hrg@`g9Kog5m<#m6q7I&7@BjgXfAO~qh-#_m{;W1I< zuuSwsx&=^6;6Rf)B3d~5qU6lus~ruc#^DYsbAbg65yd!-avGUY_bK~wAN4N>8f~!J z#`0sq3UaPD+nO1c+Na?@-W|@AV>nYqS?I5WEd9KlM@eo?^%G_Q0N0x|g>Mv>#i;4} z7>B@Vo;1h@mKO)VFp8N<_)R?X`MLjH{o{XFKxq`teZ{EMPMq$JdaBf0yntJbDOFF5 z%%2r=6nb<`Y2XE{T}k5X;6+5+A&5|_8xbbngxeW}p9(`sNv;x}&)WdvP~&=?b~+`L zLH)1hZ*v)n#*hRdvCw!H0~2#)C8jYA z`gZBij#tv}?*Yd~XsC?g+Z5p2-r{B%RLzkvGPV!WeDc)v+#OnjX@n9#u^^ zL*Sj>w_*CeTl zr%{G4CFaStoGeDt14vHd-9kvr<8!4{um|YZGap)hhdzgVWR#(BU6&uNMlcn2(5s^x zsz2@#_)b|mW5|Vhf2Dc#Krj^^N*7|qa|>ADHz|i?4mwQG*o4&K;kMpIkwMB)%+T{g z0;B$mR+SM@P05>Lf*MP&YXPSV4ca-D6mH|*rVzlP1d~BREdA&>h-3;7Qk(#s*yO=l zVeJlPnt?)e!W@o?{sbmlzu>ML&GCRw7_As3(He1G)vD#vII4x)IX4U!=i3B~Q#NS|IheLeiC_L1smirGdzh zngT@Wi9cVwgQspvWw&9eD=B>Yb0URhNJYQHD2au?3$9gMFSRnzZr zl`N_NNw=c7AaPHc$*0K-!N=wBj`?CeuaAo8U@d~jeBB=Y3l%k2&~lgZ9QTXVB!)NG zj*97tpyRjLpn3dX7+pgIihH;}=!0kE5Lt`%!2;#-<|b8Iyl;+B4j2b;tf2M;ietpQ z8}gR-PxNo=QuzM<{G38j0vR|U^-E$}=x=H7tb}(fYsU#{1WWE*hsVOOy-b1pWckt& z|E7kmN_?pGTapY*#uP55g05d%*V)ur7Gg_s%TY5cJhh0G>)z*@&e?tkxfn-=#CDFn z@l)m5=R!vR^BD0+fUn^sPHP7l>_oXagq%b)>0-t#!^P`rC5R6^%d?&!OhxGs&_drx zr?3rKJ|V+WU+L>NxcPJ0<$(#gy1UtSi}?ueav~h1xE>;E$(C5~z*b zjR#Ops#_Kvjyk%U&(m7nP9JI>Gd7E1Y8 z`P1@<-1CM*RQt@#;2uaPYvZQzFTK5XDW@qf)O-7IBvRomlG2o?Pn7W-6Gf1~>ubx@ z{!33>`inF4*B-+9bYzBz|3wo40lvNLI4br)xo&RNVPloWF@s`vnLGv;YN#rt_b@%--k4~Au6et!PR zZc&*S=%;(B;E9?~dI(lHGWfk(@@GoV5b>u`>j8IAmIjIeu&B{uJBI54gIgp;FZsfp zXvECD)&$fB^MDU!T%|gpC!N`Sskd4rD<5Q)KtfB(2&N+Rats|dwP9d-H7oxz1)&{{ z3{MZ;9suD%%83$W!Ro;p#kXe{7n8s9`un<>;M zkN*1Ejc7MrOsL`?J-PKZLu+3F;-mc_D)MzWI$A2~m~LZZYgFiT-(?_*rP993RKZ*NXK>m)aJ6@fStj8sy*`UY+55%jR`F+UT6UNB)AU8hzn*_S65d| z0$5_<17@3{te&18Y*v-S&S+$n*|4TQ)0!-yTq=h_!1UdEtXvbo6!!zT0EZzCs#v-3 zOLF?w+FEu3$Rp4rRBM4rQKxuF05eNKR_gOKNKkhdr}C%xf^C;oPjtRzoTwtU!_nz0-hK}mVrEY|13;= z{O_Nm;(cGGAnXV|#Cl*pHRv=A3jmomM1B6eS{V#S(K2a7>EVjn&a|WlngI)A7QGIS zP&yDY$*j(vvc(KDI^aBTKfX0ahl8V^i3bZ4@9nRjT2B=Y{6^%+TU!LT%DR`vg83g7 zMR)IKRN|k#MQ#rvA61o9*bQZ7`_aZ=UB5{kp;tW}TZ&wh%KSScl!(s>;f$~3c0V#-B+%osHa z?Zx$IIZi#kgNu5Ix^OB|uQoirm(Gjeet%l-5`0}Yc$vy1d`-5YWL?$COJr3}uwY*8 z<~o-;Z&ZoU&r&w3U7{ypTc2u}fXU}nH2nVdYMoc^``%fr?4#7BV}q8**5@RF7Rol* zVHOxb3?7r;EDX(Y8n!#zRyg0c18gX|rZ5`R8o^@DZ%H}8VEr{w>`O%H z(8$P1xpW>p!0xM8}+G21^nuHZ52GeAMDWeG0pF+gA!uyKenNjTlFG=4pF9oSD zqn=ATjm$P4Cs9HPiIFKWD<#G;If1423N6vGiz_%1p*M&FFJT8GdBko+xo;uuMx|K1 zQL9zLh;HiJg-^h19xC4sqt1=DJUT*3s^TD)c3G1t1lq_n%y2Kw&7dimM++thG46m} zdszncfnRGOUIQh@0zV1=xM}p&Mt@MskC0(^NUk!FE#CKxd`LRGqln5p2x{2amNiw_ zQr*|YbXw_wZ+6f(FHtE`-Hoz($=W^fcZPGw^$4hEXn+wN$JE_mq!9z9 zNFL5$1(=aXgoT7CFscQMR8y+t_?#y{tuzX#^XF|UM|@|q*-auBDk+Vm3WIHu`;Cnq zF*N?XpDd~L?a1?uL{H897pE8P5mPYM#u5FQUDhp0)X$og z)gl^4jlDSdd;8<3I%gIw9ESXUs>?53(Kyhfr+%4R^0vSf2PO5ZEQEF)qS4S@mm zuY>|&jrDqAUZ|ZIPM# z5()iExBu@Z4TV3a6CZY!c+HPej0%IJh()=xkaT4nVSi@g%LhUD@TR zl$HAAKp?C$FxjFpRuc44MoRUp%?)Xf&Oi-o@dXK1UU3>0_i7{Nh~C3PuB)GK1Wx*y ztdmCDG3V!H+o=z1J$M|##KtwkIiVx7_*z_8nl^*Z z_xC!VV)+2wK%!IHkQ8oQ?-yE<0)P1bumEU8A?G(wYuAEysuOAHM6-R)lu`PnX+)5+ zz#ZE0p(Bb_GM|?>h|Pwri1c39&GTagUkw4JHYj&hzrccVui|kX>DW zH|Nqtr%XhEO)&l0c$Ce_3Zsok=+vRgbea*8*9kfAFL`BzEm>5uaK@}-613YK7POiQ zM~^M8RD>ZC$XITz;y3lY>i2?ldEDuZp0fOQ*ZzzFFqJB=#!aJBaNJaL4mc`0@s?kF zJNGQ8ZB{MRloWApbVCUq>=%q@*ec6lM|(yLcn`{Ae%nG2TAD*W?Y-NNdbL^bp`lWw z3D1LPeM%5|V1opaGp#Z!@!=qOLx+47-0dlx1?MESrYgxAxj|FY?jNdPpC1-EWSqwb z>DA}{Lio>-gFwwawxGk5`U~Pr2>>+?8z(OZlJx5+cT;VdyYkpgC5E-0^m@Qk# z_4P8{byz2lrecSR_dAEJ{0|>^!Uvw4d|;+C=En-DZdtr>oY>?yd$fZzI!m9Qe3~{f zT3Fg6Tn=HY3Pta_iS7-mThUByaFrBq{k@o7LtNu%HLgITpPE16>1s9VhPp;*=WzF_ zX{+8fWOwK0zRhv@FVV8k`l&RsX*A?)Y+YPj>?t?sT%EGnST2?Hkr-u0@+nNG9Gg=q zq(w@308-Y04}qb1wIRj_SArB8^hp_sQ2?jJBl`@Zlx)vBSTy1mTv${O z}~$fhOgp;WVDLNGH{JAW`{^fA5$PQ8w=rK zR60{MePypF7!^_u8Pv5y2VeP;IBwe&P<3OJYKV>wy0i|5HDkJTdQTN8T#J28A99yeR|!+85pYMdjxkQAu#Nayc<#nXEQ9@Ki(!;>nwA0I0>S?;=u?&q&b<@s;{16vy<*4ap_ot**$0jkPI6B~M%^ z?~rxSTBYHekrk7CbAOsvC{>?*lpJ~C6&^w59IaTG-VyM5;AlEs$t_j5g$S$p!(J8# zVpn;=8_MU%*DtY{SiIU78)Pjh%SpQAFT10|o=rxPp+!+f#)qmZo=a93wO`j4Z7CB4 zr$9=+Q+}r%8*PupWO|3K`DgWOM!n zEw{6|g4)FD$=cP{AiwOmAB(f;&^uPCapg4;*U{*B_jO05A%m$!r*!Au5|fgzs?HH! zv$sm~OZktgS?s(A`Mp%+Q}T;%%^_%x-)X|jWf->uOGQllalT6GOOnzMkE>|Tv{jCOw~(S;Cn{)4cpJH$(wpdS zlC5`;E z4znxn8czNU;RQ)?>+F@ven2#eD`qR%YREIW>1RA#ZK(IVi`9)fc=b*GBK{QBuuP1V zB#L&=&(M_bQKFFK$^~83rJIf~E<nrVD?Rv{zx*bhUNheKA#u)|JdKD5s3JjA ze@?_8a4Sd^AuP;YJumCB-Cm6`tqjF=M3EVinFN!(gImzb!&E_}XW~s5$;lh(`Drvt z5EN#4}xAor)0;K-ePj$Tti)v$Q!sF`}ptur~V8x|^a zZ)<0VCW*PvRw>Npd^~yOA#rORBb}R>BDkx0!NNSb`p7S>B-Sme-n-HucrkOh9@qM1 ze|@N7LG!>B!pQsc74db<;T|T|3nrmhR{$VdZ3c;Tqe$B}Ite|F5bHVVIeL<>ijhHw z@-Q1y5?7xLZQ8EtUa!W#mL`92SsF%@mg{j&Z@DxlI2{`UDdpWK)S*2SSFe#MX(Vl= zr@@BABA+fP;OtmoQ5kH@X#5@(MVC*!89J> z-8fWO9G1#V7!TylFH2QblBgg>&oeyaXz2_&|B{RRb(AF&^TzRFR>Ne?;F4sW9Fg>O zfk5=jyh!YqRq&6AXRg8|6$o z1=S5ddKrn7E}>dtaa=x!@n*J~RYjw-L3bQBPQOAL`?6)7jLYph^I_wmMq{sRTuZ1}u{ z8QG`lqWV?wFnT6I7U>==9V`x)nsR-tyXwBVJI)SOHKUjTqdBdkWC-s_iR+-FraL$a zox9U6i7v0OL=yTH=AGm-M-Z^PhQ5tPJ2CSzWbT-7Jh$ykYC{k@mOm!VClS;++OxF&c2ou(0hsR29y%3B-{j~Q{4nd2w zRS1R+!!@xXM8@v;1vz3f@qUR_bX580qgf7V%)qNIwTv^boK~%{6asis{@jlj7EQqn z%PInii}JzsK^8p59>u?m*s9OnW48{jW--S-Hh1xe?qv#Fa(28R_ha>2w=_IFn9e&#N07Wi1;TtCEKraGUqjpoYW*59{BGYCsjI`@tMRU~W8i~r7)uPd zo~&n|wW=sj)_Q5g=kJav8!QL6PL12dBgcV)tC`;=@#$Kkk-80Q&*lY^5W0D(0RXVZ zbKIF}^$K{p-RQDyAx2n{$NGy{sc-w&>S?Z`EgfsS2zmowqbDRmlXBWj)FmGM@W2gX zO*+OG%E^4-&A;Tx6{o+6>U=y{4E@vX#TetnT}_CJS+|lqec!zAQQH90zoA z<&|`BtJPZRoa@>>+_-o=l~wB5A!RJ>4jRtt>_qeHiDzRuB1Wvesk3&Y$?xCtAKd?h zJ}M^6m{>TJ<(Aubh`jSOuI9|e5#N8;xVO%je>NW~_5xo6_-J-b`4dx9RMXrpH<2_K zCw_iDa%gI@m9A_EnHjpGe3-dGuFiD6@TQ=3lJ*7|Kmh9~NW+P5|| zasA~R98qGC92z!KGMVSWM*Z}W5}o53T`M~x0-B%NO`C1v7Z6N%t;ljL9BaIkYKw*Be)k-`kie7r_!xki-GBp3GiT%RWO2KT90sR(3 zh3<1{uwSC9<6)5kui06+yT{_5!kSt{)Z6Y%4Lp&-ejU2C;}>L1zg*w$Hd7vnGR^#; zz%<+64K+`(_55S1yu|5|-9t}+U3GnIN+~kr>Un#~jxg(N(1Q|#p)1Y=nZWdV__IZR znGx|wi#9fS{=pR`Mb$j{&BW)+p9{a*m!3z8%^Z$3h_{@a6DTlyrQwC8=?_)jx9$@M zxI3E+v4rE9cfLfG@(w)lsJc#>_W6TAi}o7fB*|fc@)vb!`m3>{0Z)bP%Wv0ySviSe z!Y);wh>FDoq6EBom%%J^za;slfA4HC>8ctXuvR4)-roN>S}QmIGE?@qRz1`1MEK6D zA5Tn(xUU>e6fwki49pS0IDZQ9czlNElc>upmt^2N!QTEdciso7W4!w!6K`L|#kEYXyp~31ae(w44X>u4v1~p8T zSN%QKShpMrLxkdU`oX=%9ToqvHd$nvX$2L4w1R&bhZg0G=9-GRxezsWD;2y`a&E`b z3?9O%R5^VJ!`&H*rz>1YMRHhPecN#+?aY>Kn7wVdO?mThv{X1YYEq2{gDn`}a1Ous z?kIL~P|N-k*h&6-peL%)E4*`4Sp72bRxH}fb-q-WTT7ue3!hCcr=W@9KP*7=^0Gk$ za&RZ?BMvl@0-mG*o%%BnziGE=3PKn2`L3PG=Q&pLg6UOtX=@ot*i}s3!T7l><;h>< zZ~ems(=%A`qvsyj0&woI1sfS}8C=N+>xCMCv~V_hNbJT{i;eU7XP#rMX~ z9SJ}!GmWu4{Mr?_0F0punUTL z!Th-!KSh#~?F{UGT0i*lpw(h6_~l zCLvIlK;h$)1NX-90tf+8u&#wlO4_6Oh`LMV>{4#Y25T=lRO9G37BQ z0MU?nX2{3tw3!!{tjgqULTfTRZ-Q;>UvFhbAn9u@f_u38BX6M#clsC(V(Ok%`F5M`K^xHlM4$ z%G*lk^Uy?UFE4*n0x;J|;KJmu1}3T};3>_S)Zh>iG3bZ6BICUNX1my=k;|k}*_91A zL<$m@mzQ$W!N)1KEhkn$&|QsFy29&v7Muzez_afH06oGjTP(uR;1Oty#J=(7*z3d8 zCxJ6F?|%MCVTs}E>(05o0_JSeqM+a*9}>19jAf%Kl=us$mlxxy0q2Rzuz3SuYrSnJ z-lLtFs?*m!AdUG~iH3#P)zyUvMy1+V%dX_7G{0juPd3Npj)p0mIr767%zPO=7t$m;i8XX)`lK-HVS*q<@l1JiS3KpV==4SKiYXG{u* zM2az9oM7Qpvfex|)rGr6+I1D;@#*+_|91{TC8e< z$~CdObX?11pX?YmZi`kl=yjXwuwig4->hUxm!PXh;bCBW>m(1QslL5Ds_oE46g+l- z@Ox^fJ!vl4n41B2&kpS ze+zgzf1!oHp^TnfU!qqdv+bR$4Vn*huA{K=T?I??;^0|ZD^SW90h0@(_G;DQcJKsD z&VW4!F9~8SonEOnr?@vx^8d)(MiPLn=Pc=)44~sJ0N14+43bCYb{xK>q@-Z|#)z4# zwXKx{)+_|8m4bfHEHtfS$VAVcI1c)LB*jxA*Xy(EEL71ch{kv9T4u zp2k$rxt;H{4P2X(z&QSszQy+$ZyZO`W06(DFX>DWFvT6_*9P zDp4u=iWQfLA65w_fG9we=4K4cbjBU#fM&Wo9Ky?mb3jgfFRBHOrVSO4&#>$99yZcJ zNB$E;o;2#$!B9NGoQ`#gk|<@AKnvA4#dmkyxuYqbtkm0iZwON>>>aRqAiwqwyn~!r zsFb(=nqKwc!|a@!n#$}IP9c|Vfw2sZZ<(-<+1S+7<%RZ-LiWyjpdx?DYuQZbPLY4h z9(cGq$)^+|z)1bOCL&0I<${49Z4l7d2kQ|kR~dW)OjYGKxc-M}6b+REzGk#1OzRyU zxv$WCxR@%_-yi1&3hw_$)ma8q)pcuI5oFVBQlz`3yQI5Yx}`yS(@1wpr*tFTEhP=o z-Q8W^;(6caob&x>vtrIQ=NMz&%exE?+Pc zU%|v-uS|WCO6`Ab23{ZenB5mGVqfFale@;tUF%tuxcA-C-9pkpRpjn$^HLd)Rc6@h z-QvFf6iGegKaJV{k;?iaqU|zzmZnbI6c{>Lex(<0H2|_DmZ5ju_X}mm&K6TUng`V@ zvMv=X)p1uvnuN{S`%R+)=AMudN*f()d50iMjEvDBapao%^xF*lyV?vlKv{O+hi zVW||-9B!9^Tif2C_cQU*I3@3<;VZDFjQCb^E8!4kS{tpN?YI6ZJUI0=4ub_N*wSzn z3UC$DE@J+l%WqmkF=ZRj3{W-hlil5trLK{Dq&9P6++E)wq;Pv5v0GeGQV)dE@UVXQj<$> zMb!%5*7J~b-#ihZUqYsU1DfSjhJjnjhyz-8lG4jti?`o z_o1UP?r#*>A12yyj&cV8GoNjUIz0dJGGO#wlwJK(w{*WON{ctWQ2(puewF^=YNH?m zz1`NuP(`D_{N|E~nlquYg|0P?9mS=|bv2k<;C6!{(hqa+Ym_(lVB1MEAwHAA+Hc`* zz4~k`8+g>D%ru;ACxP4VTL)pSQ|OPq4Maj|FlIJQv2H&ry7uL;UIx3WpHovtzh_!p z$D)8-+34S@2Drq%m@f|tj9ZD~!Wmo)R=qRLPBvmdatVK3a_q#R8~(#!+2%Vlyxbn$ z{{6OBuNs?^k(`~KBvgB&UwD@91-5UU;JYCfp8(PJ=$>8)v&4)0xRk2R&5Qu2U7Jhi zH>b9ng{}R^6_bOsLYDyq+aCSa9luXchN#ae)e3Xjf98(doo3T!Zwt^WB%0%*la*^W)rbF@DEY)a$9!oOr;9vWhXqvYpualPrWc+(- z?hadTVztO0_TnVCX#oCDt|uGJj#gj(m({05bX@_Rt=PZ7wO6sAgrX}SV#*|h$r)Qe zww7pmNQ2enc}PDgTxJJ|&8F<6gT1}7c!VOs{sr5|@)@-dgq6Owx>?#9>=ZKtk{E|g zosm7bGlBo}jV!`IEgH9x=)=p%gap{hn4gkn(teKOIG16^-CBLZ`{q#$Xj?T!(vsdn zb?pmcWioG0KB6P(%2?_n5!10V8w&N?dxe3U@`DH(t!s`O?BjD#7?^~-57)!_Yx>82 zenQ82uh|k7CqEg*w(_Hz)lwhd*D0M$Rd;o$t+l3)i5twamqQ8rui z?0GL>4kuH+Wcb#Cf$V~{_a`;y8oum!FiXn*uG`515-ZCEm<<%;X(O6<)cYTnhb9Iwm0n>3e%FqZ`!z_-^S zu$?^gSzwLkkkQ%v`m=wrQe>=I4jnNH#&H%3fiH2P@q!5%zN z_jT8P6Ol_W+cIOa+sjkdB``w?N>yWNpiIbp?WRX6Ong{Z$l<1-`O1k35YM(c{zpOf z@9T+H2bH$|Rm%lfSccbbe}YWg)S^-QE;Z9;DnO$O^K6g!x*R%3xjN`tn?{|#NkK$S0 zsD45{rJe$sDO!3js4sSWw+PbwJA&ijk!Qe3qKf&$Z`%K)dy-YUGbO|eR>c&nebYTV zx!0o+^8ekp%s=-n$rk~fV=0`f!V2;VF^+sdEo6W0RGEv>TgkeA8Z9BHSj3WyYIlh- zEr%lwmYeK`W1tk-9<){*9JwMdDs&o_X89xWCcahOMdrF$d6V3I1TRLG7=)|y5J-T3Sf zn2}qF^zN!sPj~!k*OGaN$d4-F2Fv6^plx>!F)dXZ`z`yw%g<{Rmk;(S7_VhkRuHli zViXb)k`YoVk6%aIsq6#Sv8V-s&rySdlmkY~(X0%1K0YH?>_Aeo8`L*qQq+ICuv*64 z`yr-Zj-dx`gC%T$wmh%09dpzqoMOhfh_Yu)>eD=BtkL`LN`Fz69$ne-^6&VrYxCVO zhc^i}6ZUg&yA^W9CtEFCsEM@T4MOE}ZzI zn}|?2$=sdJJ2 z?@*W>1DOXwS}_c7`^@^vo=cckV;?%T;P6k>ySlU@7{{j}rOFwmFz+3DtcnM5OCh(h z=aT-98d`u~XQ0~q&m+al5vi};vM*uGX`_=+YfEuqg)jLNML4_!G zPs0+&!7&;9>(@^fU~Pcx7DtZsFU28w^ijYG-oELomh&#_h_j@F$#@)CSF9~TRq`L~ z+C=0LUP}!ev>aD<2~5v9d4~qgxmsOw|`L; zYV_$>Aq5g505Xt_nu?m7#E+mAteo3P(OAhauq`|=N0zva7^R21zvP;m&+ZkI(}|`a z%xR0qj;DTugNiRXbjONIY9)FCo+Wy}%w)TVUwxaPYTC6g>2;K-d1yj_Nrmg+*yx)s z%~#CF;E{W)B6bc#GFh>NhNSF#6>apM*9L1PcGL6aCAZ;_G8e8F@2S&H44PCb62Tis z@x_Dq0ljQQM-Pu9Qt2|g7fxu7Eoy@UZWlSUxnUS|S5-|=C}SUvIkD?tJ3q=?d(DhC9&62<%ij&hitsbk-Z;Tvfjr)9n3gQ?N48a$7RHgv6{k zMUPUA{OQ~maXfE%-mMHtx^$NO5mMGs^8&U?A`bLmHbYU!!wI4be4j~BfbfXvDx#C= z{7OMx&BV`l^jjT>zn%aa;0GnOg|!gC{UBxCaYqLMY|oU;{U=-|Js^mw@%MpxO^I=^ zm^hf9ScsR52 zXAu}2z7S$KYMUQiM(Px-Ueir%1kH-F*Z8ArG&n4G_ua||;-UFIgsxcy)fkpb4ADm} zC6pN5wOd&~LkS>k*Q}wKCHJ&Ftw>n_rh~EHNh0Yo4eP|p5n#UV(Z#r`NMF}3(uQey1!iZmT zEn-r72`pzgH)Y=cl_(KAe>dl-H_3|VgeGQPvQ%j1w9$d6)Czk{dqu={S&>vM3px48 zYBOXt{KMA$a-m50Z;aMVL&R(g%|ds7S;^9Ko8w6AW0oxgTdT5}@|Voy8dZ>(OtSSt zzp{y-2hG~6qSft86*kaBv0;&hL!7>4vsVVzbTbD#&7^Y#@P?sH)`| zz38&6z!a~=b=Q`P`8L8|2NLA9wH7Z8IOH|V+(RqD8JRf=WEa-u|^G3w5=I`xU~RxBPEY;Lz5nGu7MCiq_W{x|1rBHGma23d`x)e zN{tYWQ#kTtAj_8!X-j>2&25})H;low5wk2Tpg`f%n?Hcg*pG_b-RX<7nBd-F&KZz+ zDs|hP^r(|UruxP5IHi-=jG1whaGFg0FcMJHU_V7#uEC$v_;z3^izY9t*XAT-V32>2 znF_7Z$!^4M*5mWpNnmoeG;ow4WVx72!nj8Q-N`RJp}ge7ON4-8NRG9+B9y0lW5%=?-K)_EDv*Hk zpc1qnhEn>*C|8dSZ$ z%!buRU%5(+7M3bQ#My4%P>T>^+iVeHcP;77&l~5RyR1ZSsgTdzbP2IMXgEj%wG=Yp zEYSfhmERRI%v@>lxNy)|APj~&@A3H>Ic21?Mb7uf4$Bm8C; zU4=R^uCY-8XUr@$H{Lk&+?7|TswT2r zI<{m<^h@%gxoi=d=rb&fBwZPBCK^xvjUuwd-4w&{D8&Wpv+wCKhYVo zI|=Qo{K+hGbjOTqb^t+@I*w)2A0t9eI7E#4afz0fcyX`>OYKDpL!h>f8jkuB=@8;Oc7A=2%s3bNK8=Md0d(Iwj7uG;?AZ$-W>mr{Eejh3+)R%;k&Kb_^qs=%m!FYoy7;(D9!ppjNaCCI3*58qRN z{xNVJOi~xlOhmObSdZ;xjcv}4GK8Z3vAx1kXsSE}t~n>guvZo`^jb)w^V*``6R?bN z7xAT3vjEvBaT2e5)LG~! zbT>s8CA{mj3p6==7y74(egFMn>0ko^MKm##RoV%t!~ITcXJxnbVfm+`mYCuw8{hPq zZve`e`inXEKOS$Yw`x-J2W5TdvNYZuG@$$r z0bJk!j8c=x>oJVWvRhDH*^;U6s(&I&JxH#d`yI^R1Gjh!a^}=}y`vd_tr&Oa<}c+8 zm{8?V=*osnY(H{+Wb*pUG$#lqEo6c3%D0G)(>5=@j2if&9>fMvlrT+cU&2)k!KrTq zt@ynznwi$}dVJJu!pK09fQSZIp2lT@WWQzqn6c^Z`$pPW8b67cF66zREq;8f;i5fn z@}bHg7qgJ4(wX7-JiAoTgtVc+4xIu&G_t-oQ=r>*#kky zULBBB!T4>fieLtqA@6{cKt(kkPf+qzq@ky(j7;K*IQBbrv1`Gs68Uj~MdjR*ibXAn zRp;+OW<73QDY9s2jca6l_cXmQw(N$HFJC!@<#0rhg%i*x9*X)tM$*1W*f$9g-1Hw{ zLeOjJ+*y9;1j%?y43%oRdj_NcGm2?qExf1ey$ln;n} z^v~6NpCR3&BbrYw5|hD-Q3W&EaL!OLYpUcv<3P<$_3Z}Z>I!2Pw7Z{nK5~HYkw^mxYurz?tQK8N6rc_#ZEWRwWkr_ z{Qk`hzA2NFqxLf9%5h7|NQnb2SB3XYG=anIgFm|gv>@cF#G+USFP*qAI*qC)Lo;)X zjH!uMuR3Aw0`p3i696H17dp69))WxAA8 z&3jecueL7UT1SJ?+IRP=`Hr3tX`U2MpRdqPFqm@y+Y+Y(4hV)l$OII=FKlX$R78E)5 zZ}nDQ<}BhXvd>jI=^*A1W&J>WlxiXUPa2g(I`qkY4nyw)fTy)!v) zi#$li-_SZ83SKGxc!@Vnn8m`qnRD3*`}RV#W2;C0yf-;FN(w=GH508PmgG%`f8nRx zS%^^#fA#-d302~gHz#k#pMWB~)#fL3jVb#qN^^JEo-$U7F>4h|HG3K7&%6iKFNQ&t>37q?T)+MV#}U%#Q);?0vPB6&8| zvizL%J59s7LmD1TL+AYKRvpV`YXRPUjL-jMMPy+P=htI484ql?^=oOy0i-`n4td8w zo*|GRi65^T zzjU4KOUSZXYN*XqDw2XaDw}T^3~6Yu@FqEXyEfaXYBn@pEgo@W)=s$N$X(>}u&f4{ z@J=|T%P`R#MAD)uKzoi%+yKg&iHy924sC#t82BOeH)>jvE|2@Z71M2Ur;Dsr6A;g1 z6!J)jq-hvvFYwNeEy4#wT?LV?pQvX41T6?GbP&1+OV)SH8()1A3?S4g)LcZExH|e7 zK5O}a6s(YG!KXUcRo3=*af@;~`yn3}ERKb{jG2K2lPZgmbdB-rhUF?Ntflfvuef0# zASzkA67H5l>MaLubVfK3hqnQP z7hz+9ySDy!3WJavCBr5b9ByiPV}0wl* zp?H^sH-Lb8UDCW&9AQRNf?JZm8Izz8KLkWv_UlG`XnDKgRhD-C?yZbl@6;^AOUTLq zcn3A~RAzk_vMdbY1IKQ6zTYc^B^Uw1G4zRFetOf)s?*K3E6AQ@r=PG+sfh*?sTk6l z;Qv9j^9uS=l2Y3lM5eMf$` zO}7n-VB)-YC=9c`!cG^ZHABK_?q3zY$@YeeZ`{xHN$muV&CK;()IDyBq zw~8OUjANQU)%g^6l|04?RX-@qcHS}D2HLxMGQ7*gb5HP=`Wy@_F4E(jTH&}nzdz2@ zA#>y~9FdIvib7Zdi&o7b7d9DH5a!O7ycBY_8^3+4nKxaKD=`_iN5g2MjrDLMo*`@% z%HE!Lw1&7WaV#kNOgmI%MH~(6apmdedoMpDYvanzOQIrmlyJCrAB+K8EhYnoC8=2{ULw{C5@?iy z-uhCEEbDmG0HOr%?ilt?5xN~ig=rl*u+h0}2($m|&S~qXvLZJrZj#{Z~1ORlWS)>9E)RuTo zA>Lx6T}h=g7Tn$3a8jfII}z?4@)2;QCQxNrPsR^VU8qG#V`VJgf#ov-Q+V(Qda8W# zC)(^z{aQs<5fUB6Pp(d|ha6NKx;oy|NWpsV)K_tTh*^z4p;aVpo6lbJprDm75i)a} zd=)4fnEk=6VPjGTXyaA&^YBenB~V!s0*O?Uz2;%%x-uYfkZ7*KeJM`g`_GA=bd+%4 zcDjc+%n}hg_p|T>dxTAXTS!N~gljqarC2b7WAM<v$rn6ps7N1G)V;2xWSokkeugg-+P z20(Er-v-!V!PKpZgWst)29n8g@8XHcIe9X(F!<6sar?C(^5}zhWSUT1#NZ4uD`bDR zc*%#!teo`K0hyFz2=XWeZ(7oqgOo^D>6hvE`MV?|zHqssMAw(2Vt)p%j74;!bH)um zq@-42lv1=N9t8jipRf6^u960#VJXzykpoJq?OmyNTRBm59C+%_30*8gUWLRGvroAS zo<56%bzmFJc{r^qcS#LC7|Du4utRq+!gH8mWEtaSl4bnTi^f+|4ttv@xXpLKAfqO1 zD-d%J1JKhRvJra`#tqIKx*qaKP``L>2w2ffRDDZ9`@`Swf3 zt@<;VxT#{>^NWIh^jD3tYzol=`@=(~rRh&R>-jT(tqxvsT+Kr%J2+fjljif{+F_`u zRoiaP1QpLrvz#myIbM5-*x|D!5B<$b*ajE04|>=(UT=JLc7{dFHpww-T`^R|OI+-K zHNeISdwnSl%2c0`c6hPvhAGh#1&$_-y()k0Bpwp4s$KH$jfcNf(kh1WD{)H2NK(W7 z(fK)bW47+jHeI8IT?OBnY{)eS%0C!<_hO9h8!ng5)(lgXy!cVdl@2V^ycwCqj&MQ| zS$=mK+opN$xx_}F*wdKm66Ql7K7tz*oC<$te-!2VIiI&1g?clYvMwj36<@s+GQJ2^z* zZf92DzRL+AT?3cvJTIm$Y4$@(m1E%KWq7`tx9JYdJq)}nna)8khQ`drWslFK&GEvB0i}&6&>G|oZ{Eb#c{P!(o3CQ&pqhK?C z1m5a0PV8q+o5xCwpGrdCc(1O?{_(eMG^FLvEpn3q;_M>rw^=^A#aWlvi(*=+sY%-q zTv1zUsFU4>g++twAA>*|4)%G|BiA;&G~>3x8N)J9#5cmY-W4%>u>jhOJP=RlkbxF= zSwF#AD^WC*FJqzFP7!Pp4%A?iWxw8q2LgKe_afAiqoz5(=3If|60U)RvJ8;`YKMxZ zgLOWaZf^2fI9?U|5d z!)O^+m@PP^_h!(BF@luzI%=wGO)1tos*?ZJo~-63uy+gr+Z%_h>ExwSEm~yQF`Q*J zv{}cmff6&33nn#He(W$pI@Yf1-L!bZobJ8Fi&QA^*^QWzpHtDPMz)dtQhY{$>YF~<(P3R20o9d{ zRD1jF)1b|{aGbgP4IgWHFDGD9KikWCc*V#)cJOAJMn!rqg(2#PXZ-cmqPXI_pI?Wn zwF{dvW!4|VK%|_s8v0m zPSwA)fCOYMvmThHhWu*Z`we}#BQQT+3uSx?s7tk zy*U|mhdF~>6LHc-v1hV3RUBuXA&&I-A3bk(f~IOM)M2137Z_pCNOJt;;ULyemYdc$ z`FGaaB_4gl75=W}_YO6;V}@Vcf#kGoGEwF&i_s|JH=Ppow$E<&ZBI8+;mEg6Z`{7{ zB~Xc)APaM47V5Qnu{zzTGy%tVVv8Jp`&i{+R>%OAY*JgN5i-f zv^X%OSuLL=^xzSrdEZiupJr*fl$!0sOvfx6>tPHv@Tw{0#g-d}hO!>4Cl0F|D{Zab zt=7pLpS3npEdwAG+zk@x8+k@sk(=fhxi|GK@+>zaxfM6wvPj=QDw(fL7$47b zxy=}>+WcO|p%SWvr@Rg>?(w*ab^3m!+v=V(SQQd=kk22&Jjv-cY7#*qL#$6ojqA;@ znkIaW+A;MvSLgze;ghS68bjRuDmZ-AC#L}u!CNF}?McoNXq!YIj(l?Is7?BIgVi5r-qU$_WEUeSTgSZ34k7iMyVg7$AxGyeB{)y;W=?(65;0IVdZd zsRGH{`IRv&vx}LWBS~tX(K00WdR0Q!3M4{g;{$e-$p;g=^`uU|#+o;n;v^s4e|Jhz z%bX~e#2XcYN@j9mM`-R-1r)YKfE{W)>Mf-uXfe1ko96eFnuyf>j~>B*0$KrCPm7yu z@}R`%50q3zoW5XxAo43jSXOkfHjQ>cja6;bFVO?MEDsvS_wyqZWy8NGO; zy&NLW(umw$0w$ScYdB!&D6&>1OwlzQ43Tioy)(Hz3(bZzq%=~jXW5!VyQ*jN zEU+!qv(vfd&A)8kD<7yniw_Ir$R+TMqhn?$?6~@B^r|f0iymyK$X0un&pSA>-4}BM zaUr=KjSA$wc8Wihl_Je@+-huEd0~nAJe|)yk?!!tqTS)*uWFOy(?(~MdM&;^hC26g z?u{dyvvyQ2M@C;?-+Gz2qlvsB>np5DV`4fT1Yr-|R{I2wKKp>KvlJx3DEr& z*?TYJN~{>O>){tIhN)w!2i@ClD`?k6F^t;c>zZzw5n)bC-??36a1>33%3*-91yN(+#6HE0-gw9%Ak$@; zlBbL>+_cH+Jv}501|(kWUj4kBIxi^v_SxjVX?R9uyq!dLmdV#Jy=pK}4`@r`1@uAJ zN|x5cYQ*s1KNsP6apgxcBStAP_s2hu@>^3csaIqb_4K~g-`i0h*(^6}+Io0zh^XpE zP>2}i^f+dG)=0_M*tye=+6m9YP2?V~S@dv3D3$ecf?c{k{t1-cfYO~-nMr)-(T}08 zJX}k<*3kaul>?3#%aZV6nQFBAt>UO@s4-H;UDYAH)L7`D>)IKG-_E~2>yr`#Y<2oI zD!%l{%1O9_JBC7Ix5A>6 zej&Y$?&@`Q!TM5l2WN;*(7#~a>%>Ad#ZrHz4beAvoLOD8NgqfkQP0bGA4SR=h5lZq zQWhwmwwiSRz?yaT%q=mQk2b%KL~*L|m^vZFoggs33^qBWHL^YxiTrN`b+~Ub$h4hh zVY%8`p4Q4A_3}L@I0~N5B)D&0PTiMVzvE3xtD$oSYVwC_`Pz)GW~Iahtdzp<-ACxb z$j{%E7E8s|O4uKKed7HTJoWNrSiD7^VvO;A5E>ey6H)UTl=$xs*QO65yOkTNkS}}0 zIQ{Bx86d$$24HaSqUYk_7RjOZiMK!10)Q=hC_!Dpnv#W2UmkA__myf9)_CZhgbOm| zfyQ0&jP@Ydldb#7gQADwB{M*}5?oPmS9W0wT|ox4=7RT7qe)K)i`u&0o99QA6#B~o zr&!g{pW42`VkS~)3x(@3F592Yi@xhZTUVTM@~fmpR}u55zQ#3+IA^OFDdX0P%;2+V z5qNmwESQvRUAU=Rw{$~J=gyreR*((}pU1ScOrmMFeyJ1D^PjGS2p>Jq-&nREJr3V( zMd@*eLSraie{I(~ut6;{+?~p{cSkvyZm!?65nYgcu#>NKuYuhc$fAEV$#!_G-@t#{ z_Esg^IDpPmAZV6OwKX33OyWF~8jF|{aEua?2#OZLGw19BJ97?__}3<%`ll$H!Ait5 zLPjn6I^$r`nf$Dsp>NPof(_+uC#2f#)DR%+X2T_h;s$jGI$N5GEtBkJjurZ4v&!w3 z1bJVI{;87#LJ77P_hn2$`O-bd4$43P zPDC+h=u?@>jnS7*icAn%$}eGWc!Bd;s>@S6#o`@vu4AW^rS@Ng-C=s^Z*xRu+SxWS zZMf9+5hY+#i3TL9Vm)ul#wUg zSr2vu^>13K7~Da{gq%!?Xu74C2xz^`6=ya(j?8*~t+%P3FEk#H-RDz^CZdK<_a?m4 zs!%Ks{0#foT5E^b*IjuiyGD*ytb7+^ zEI)!y+-A@FR=js#@E#}?d3KFVgI?3d_pNOGK3P2@Rj4rl4Uq^Wo<>S!lfM?szq)G? z#u_{V((=Es36V*))ufrQXp5GLD1YMWA~MMCFwKMy6DM-};U)vk{vgBovtpvQtQ+wX z+i*g+k{gGkeP^cGf&=XKEduP|R==iMS9V%HFYJGqZq@N~D;Sh7eUQO{C?5aIg?3(J zmx&nTeiX5LU&jw#ph=;+0Xz3#K^zAtz~RDtY@uI;6Xl_X5sp&U1kyjOnR>)Y+r#9q zuJI6&+i83%mB67e+oR{M0x1r#lk<15^RDGPr-#3;l5^QF1mpxNPa&KwV|KLKneeQ5 z<>xIly!4N_k*M}FH@GwP5v)Mj4|6wFB~|70*}eLku)>~3Jg$N|1JT*2OC83?l4uvR%3`oel} zqS;vVusFj*^kyws)@Sf`$wQ%exCt_~D>$o9RU&Gdb6nJSEJ95t`V-;`l`x#AE z*N(GGopdOvxvv0{;dGgbdWmj%)2xGW)Uh3f7b==ia$J2-J&o=76rv9E!P4W6Y3pkR zcK8ibG2&L(To6!@{}6{I6T0Bw8g;yLB<{&`S=`wIpgEWXavw-y8)0eJcq zrg>B>9ye82DP3N35{I9o){H{Av&}#?K&tXVotT>!upy5>Q~wKfZO< ziW#?yuz(m9TLwLV0vFI5VdaJCpWQc+S3COAtgXDbFW`5DOD^6uis8%e@+c&;Fk2<; z)CplX>6x`T*4`~y6iTbu@=1{|Leb7^2H3z+0Kdu%8F4zPrqnBnms+ZX+g!F(>u*I#k1m&cC$)y+N@=0$ve10#Q4KP8@qdeEe+6DMgK?DW*nj zSZ71_bz7J)GXoPe_bWmtR=jbgw7}Tpax`VPag&$mvc5$k=&)1)IdqsCGOw^gctLY6!+Z)$!b!Se=1m!SgqW+woFseugti}m-jNA} z$6Bh336rwj4ca_)_BX1V{XWz+YZJ{>xgeSG>c;)IQufM(iHg+nrr&Zf*p-sH+##Y6 zi2b1Nf4X&_Nwc2cJh==qy`GPu`KB@c$W48hS1*6+KVr?tAS#L^{S~FEKu^R0mQs)2a`Y>ts4gos!AbTd_iSoLy z1i^R_ap7gD-1fmNbvaiKeJHE^yOsC|9#c)~!OnJB@{Uh5u7G}v4!qn^V!PM)244Ms zRdwRXM^OYU%ndzUTiglj;G^5Sjk$gN9>}oJ7P>96ql2qC!F$AhoYWUiQihVYlu3c5 zFb>SOHxb}#Kek+-W{c>`=w23Gjd}RLDo}rLz@>x6m!eH{=%K)$7d7<$%;Nsi8Wz{f z7mcTMTIA2F%NphCYjs9L17rWT>vB0)^zj6T`0(xY?qNOK70ZfB`xb$Wg>A;cxH{sV zm+GV!(-rpGpe+M#KZ?L*nZ2nl`BPUEe)uVA7}8DF14!3kb8E^@7${@->#_G=(MrcU zs(JPf(A_l((fImtkWo8bs6hDBuf)dZV4Nn0Z%rlnjiFu%TyUt3@D$O#&^2`BnI*5i zq?zcPINxF`xyEMmjo=8RK>amP`9ng+hDWB(>kCd_6PL}bGG}Kf+X!wBBK4-bH@Y&h zxtSs`m!+y1EbGgh+Mezf>?`q~wstvPnTA{Fy=@hh^Br`u->D7JK~d6yl6^nI*i=65^R!(7fN z9(K8|g5R6FIRItZ8tTp^9z`IlVVj)6=zeYW#S@$3#eNIT%_G@>GpD~m3S~{FG3?a# zvhB2dO0?rAj`OLd$^OdM+|#0Yo!jYS21GN&07A`p3Zq%UEJ(r$QS3q?<@zj2;D1Zs zv(W6Br6N{9-FvbWH~pOIEUN=!*-!R&$+z(x_GKkG4J$HS;(8J!(awk8@Y9qhXi6Sx zqL|cyr$*8tn?VHEsgqWo#aBccN-lYwyN zpZyYym)6SMwi0l60k^P2}3=t1dGx#Gnzleq2UjQv;Lm zRt6$zYONWQ?MX`q%5L2z`y%wN@q)A{Qa*#l9DDoZ*1ORJg}s302*I9!jI1!rOg2P5 zHESs2D=L{puj9xFn<9nzJSTzOm19zj>z0OrCDDF#BLl}n5wGi z)D`;c$kS>^^I?{z>;sHh!{SPbx2{e@sHBglU0To^^Z_P9Q+u_6FMKEPpb-%D58-PM zh;-vvd%*{+dMIcxF6{%J{Us0AkNZv4wRRmR6V>`h5efOND|{ow6RC8Vd)uBs<=HlK zvBN82j9>081}LR@&L#8;-oe3~=jpx>+afYpiriYRu_H$=9*!4kwbG>8SNB-XWjQ?u zBOCy-j$~A>UXdYgb0~0=3}o%D0H-5!dT2tmt4%iPJika_i#u(kPlF${90jTOq zSZ&8e_s2Mbee4!%%;GXKe9Sxdidj>`SCM$zbBnybsTr{KvfeMAz~#_Cu5$ct6h*pJ zb_m_;6yPmPbz@3&Mq{6E3{1B$5lA1Onc`v}iz)E`mShNl*ihp)kCJ#9xLI!Di}MRz z_F7}KTneh*0RP~s(%Y1>rsPna_eVd0$sZr5WHaok1e5QVL+$#c*&h1~i%q!{k79zF_ zHdXIk0hWl>Jd}e1Q&>se0N|fqu~oLIn3!%!Q%t324Er)qAc~ke+dQKa5R_w6DSWnE zYN{RWMiEG3fHnhMnE?mGfQBZggC@X1`2q$2yaQ<#c0(8^?Tdg{w($5*qy-+PWq0G^`^U=%F7XxZh5P2+!rfraG(`0d*@ z3r9IXhri7z-;Fj&=Up!>%%u`cnaeCZ_Un>E`#Rl$#%c~=+<+kNwY{4Zq~A)x4vS{( zMOtgB94a>W8P(eR=i&S#tg^VyyEngkd-qUv)Jjwv;70(D;~kt@6PIdmK8V!S-Q5y3 zrK+mx{qIA-n*wn7+{#d~^Fx9KlvYWfBS`_IK%WlUG3tM>o;bHe|L({-(;! zV8c?(UmlZ`WH$yefHo-cLsDJZa-B^dva(2~+cu7XOq>(IJ5h6V;KYMdP4RmL{9otO ze{sFTL_mW}Zsc6w+bi<*#R#12IMeo;yK6kOL zdF1mT;QQal%>;>Lu2|{sTZ+BQUoj;w0O%O^{eYg2&9V!N{tL$XN5Gx|^F2!D9Dn|} zn3&jprOxdz2Ql%^cl5t^DeR4Bt{afx$f^`zC=uYK&@yDh%>jdICWq}mjr9`xSs)PauXGBK&PZ+ z9DY>IgeCK!k4paiY8U?v3y2H34j^{6uN|$l(5`r|H|6VyTBut0!})c}$IGk@@Bj#! zP$Aa5Z;gET8zVz{WKLRd2xOC)MXeUEdC~~~kF|GVpc!2$Bc-!W= z;B!wvoAru9ONgKu*U^-=le{GEpAhmRnx&%nNFJ2fG8nVi#xSsRA zHD6aY(BVFuixhgOe)Z4YXmFddKsZ2|AU#EY0v>1A0USU^Xm>n1nVhHN(BgA>M2rV% zetUd>DWFC4NYCWR($rD@KIJk?@P_JEhg)y81?#>0VuJX4YDRq;U}JAv!WHKsuyE+HWhh z$Td?y^hrD!GcB>gTHsU}X-{v4J97{gTMWp{3u?_qei%ZRtU~~vpk%&`zAUf1@=jPj zh-_gJ8WXy6_vI@lZzqFYT&8HhGRoYvMHz$4Pl#ID99|sa}#<>pJZ&-lAPV|LEaHoxkTJLu#x6}$%q0TfjT2-RT) z_71>!nUtQMekH-RG-F%t-x-M=tY{{sYA|aC#-Kp+i>$&084!8^O$*^}fC7T5e`h;& z5&@U16l2_yH^beWbjfzE#q$~|fKUy!My1`sVlcwJKn4S$D7k=iwrQymqxM}{Mm{}aEX2Vo(fA+LktJWf-u9g>AWaHjLKTCk&s zm?QK%zBf~u?sk5MSKA}sZr-W-=E-&(h6>Nvj(+g@JQ-1gl- zh=r-21w$QV2n$qO_hzU$;1#f{S*n+)P}~D13$5bxg+&Iy`Y!@(k=EB*ejFW9uK>aA zza>hgRN>BkZ@zw4)=gjt?iSUZp^za@&eV2Wm_Vmd+LJEmn`u6tE}bQpDkDzIX4+Fy zyt*Fd30>5&ohoFgu&kwK{pv30e^Tvqpn%JAOR(&Hn0t7&3u<?4KkJIoE^B#&L)vk@Ug&^aH}(PrL<}9 z5pMf1N^;?$n=QC@j$)R8j6);n*=(wIt?p+Pl-Z%d9Ox_TNi^%XL!^iq7YNh*2*BK4 zy#3|84JS43ZJhT4tc>gKe`^7U|CG!AO7Yr^KweXV5q?1rrc$aElLoasjD!>hs+TLp z2;jac=hnNQoO+5A4YUjHSKjyBn!h_9B1_&!qyz1nr!b)`e+aQ@Cih*~v;KwWu;+-2 zj7wJW4bE@7)o95m48?=&fV<*pqP>FO=I=|Kmn+Dhn>OpU1n~G8 zmToA67)$7C=A~96iubrA-P{$&4r>w-xZ*1hgcO+RR#P(RSto88{SJUw+m~|(uJo$T z=kKk3`*c<9xx5()l7RMGuTmz&f1^E!+6+&?<8+T8pT<4b3@@+RWYMO)DP<$J6O3%z1F?6EkIf ze~h=)d@(v7dDq7LCv1PPb_mln1gQ&m=&T_C|Km&~vEM@Obyf=3=h56>M(6HrT9?y7 zXR^aIR1OwLwNJ>B@=N?*N!-3d8lD*FMR%VH?K7159rWJ{Lf0!e^uer1{R&54XXg|M~oxcZs0Z2 zajQaFlO4zYc$&4dcVU@kwWUAvejYP)CnJUkGap}iC>EUpN)GUGej9KJ0NbwgGXN9O zOZ!+YCpied=CN#QvSP4; zkTxcPE|6yU`1GVa6iWdI?!2F#GVo%u&MA{XFAii*IlYodGpR1t;l+p4JfPEJ=FKs% zfr~UB&Sk!3iR^J+xPK)nwc&45D)4F}8eisQ)fR~9QP(}T2_E1kzSFED96Z_%1E!*w-cG&lT<2wak-Z zO6mZb>g`PANa@8P3gjpdeg6sWecvGpmdsBB`-x$N!tDKWU$D7pD1x&~b}@YCAf}4w z4HJRJukS;bGgdlxGX0aiP2At_qZ9hFA%`ebVa-6kTDn3cZ7pKIV$n6sHYzD7zb$h1 zm=Wz;Y@zqf&DXmk^?ikFwhJ`yaAlZ9XecO>Qq3}{&x}Astgu2uLzQjUpUIE8W?s19 zNprRY!W%4bG}-9~97Ug6gMuU%0bM@D%*@O!uB0O@_f{JfW3S8v$W+en3aSzcEftiV zDYuAf6-w-5wZZ|vET2aAtd?7F%Tw9{KR;qR$PeE?qJPbja!XTe@=2BG6o_S-H)BmK zXw&s!<@3I;$Gx7j))-nC)ab9$?aYl#pi?W9O=2v0Y&cnAY0K;1N~{jJ4;hvITr)5F{Dn5yEwG*Jc&>tK z_oZ2;D=w)u;pDp_o9CZcZTHeW)x@H6Pg*yz;nWPp=g9bTO);w31Sr@ zNO5EB;Vk@((0mnirEwFn3;zB2eFchZbuJos=c#VeHP#E2zu#pnf>IfHy`P;$@PES0 zd`GR`Lg=HIjP!T|t+(4qSKTk9cTG(Ga25 zBUKp12zs6AvCXu;3FlIhK1ka3{^p<9<|bbWViv1^yAbiC3RAyd)Zxu=p?7T<5o{lMB9$N4ZIITqTe=|$K*G`6EhLS=VbmeU-ANmw(d9Utxmi#3O8cTccBwg|)yp4m6>=DznUwtZD=M6*hj z2tt09-=kH}d!X{75J+#83V%A)QGUpzEM2zi*ueM|7^PvAP&Th(<$gtnB7|=K^$Q^W z+h-+Wc`alO;KSXv^Cqyu+y=%`g!^U7+O_Y_3qqU{!zdK<7@wcjm?1w)s3~f>%|k4;mffC)lHSa)KdYsimA}@z z(JLaZij_+L8gdPw7Ni*kfcMBtGI^U>c-t>&FgblBYjxh-vb%LI+#iaOcl42_057N` zRW~r6`-5UzsDKXK$kIuY8X>T%7Yh$ug z)GFeZWBkpX!K2l)^f}XOS9Bi72MlAsLFqM5@Ik@ z%3XGtp|K&1Mv)ZAu{lMrRZi~BeJ;zc9$%)^3jPI-UgQ#Y9G_Ly5qrw_cj@3=zj?z{ za*xSjkd+|HDQ#kjloa0Ez9qYBzM$7fZz=4wfY*uWvjoVpr?T^vt-+wl=Uai-m*>}C z6GtVISt>5{nYGEZNo8{Uy!ZKPbsAH{T69Nq2L@^`?vF;62&wDX&fkInT%X?Cf&c(%+gb!PVis}d+qUXX7XODg z(;*#{?L$&f@Kqi(j;Q{TDgJ8HjC-M^pqgTjh`8l^{zHV(TO01l0zLCG|IJUCi)m{= z&D%ZGoi(rt91Vs{&TkgATTq1i^{F(Jf{=caID><6HpB5iPeZbS=b?<5VvKcymGt!{ z_Yx+EujYgv`j0=HDW<=G-t%OBOI)$&22%ZV)0~WZ$Lr}xz4MyN@vF8)YL~M>$I^>D zg}@5`Y9vqRvT^|B0~<;3C{~t73}(4*_Ou9@NZgrviE)AD>Zs$e`Cuo(Y^RwgtwAdv zyY-@61Mg{^(Km^Ae<7pxA*`?8;-5QLlFS*Br3na+gx=OqB%WUBZcJ{!@RXg!4-J}&JX1&}aK@;96-BaE_e zHF{T9plvF4zUJdvmlamPS_$Y2%;qS<;9mB8!ku=5m4kaxx4cH5J%ABVq+6&WhdrRV zDV)5EKDiyfe7ygRMK3pj49fgN_U9;tV?t?Rp-pAbhXC$Q?~l_dx4KHv z!5x0kN5b$+%~4$;o~G#2xaUQ&;CinlBIo|vX&9+)FlBCk4zZUM&Ri1aw=*n5$ z>2%)SLi@fFxB>h2E#VR#I6rtKc6aM{xUk+HU6;TAnaRjMpK=;?XJv7jckwniNBBfo zUwX8Na+tNYQmkF!DOLfBI6BPnrRJmm{YON!*fiy{io-nm$N7>uY>x8MwVKsPU_+^O zP=0p3G_w|0@oyeMk5gjq3j}r_*Rt^Ur_yU_ROvSHg*RPYL=gmoMm5 z+;Fdo?r1D&Js*r8dU8J_?Pzx?Zun)s$=X1h%>(uBV%@-l&H7uZFhyvhl9d#rUOAgu z)wc}dY1j4FK8LmuD&I;B#VkSjhlQ>z2_$T3m$C&4*!p2Jx|uEXAQ~RZ)Srw{Wv*i^ zAB0ay)zZUrcp684hX9ReR0)kr0}|gSt6ZyJWX~@{)r!R{achTR7o9?}xT4c+2_y!a z!mF*;W#Oo{OAX=?=u}t?dc>;5Yp`v?DE3mEq79*QmKX0G$G1%tT~o79s&H1=oQ2;r zxqSoikzj<0do2eycwvtg>~z}-CjJg48ylMxOYGs6u1#Jl^&54)nl`%5Ei*}Ds2C%Y z)iLyOGx@*Dd`kQFt;$8whd>~T_X~J{*JmrhtP`gsc{jP-I;m=MKLQUCHWPoQ#M!xT zTQ#;ikddGehvf>1I$W^~QER6LWiTE*SSUj+%MvYbvaM zo0ta!)rHqJl6TD`{7=gaP9MUSAw}vqStpuUYTDmQ2Pgrw+t)v78nJs}ehD~8%j16` z?QBn>6?^sgahJluB@(|KOf2Sw(}((7%FC{>pl`i_VR1j{lOj1^?2}nL-}o-vW=_$L z6XVRR8_!HI2akLfN0@fKO;$T|(Y`FWie+fn0%Pl|+7LJPPnq~^H$i+d%*rQaoqE@X zz{iUdUv@b#=O)iofW6sh93d_tCJ(YT3fj1ux{`c9P~~!+k`zyWWRK z1$}NmJpu7hvn!-N)#ajq4CbChCR2ai`EhgOT1z_5^W0bJjYP8TU>v{bJNr&oR;_u5 zXM(M3yRIvSviIN7Mken@5blP3YF2=t=0T{Q#RcFZz6c0%2dpV{6WbTz4`=JeE7~_Z zZ2jbCVq)UxxVK4Sg$q9v6p=qkXLCb)Yj|MX+xtarqgj@7%*1DtL3kqef^?P_YDP!P z92DNu0Tb6$#F@e6^pjX2M!iUz0RK??FFPpv1cH|S-H{pU%V$#elj}hpG@pU`JPUgLahP*+w>O&47e)uY1uPt8^?so6#Y(}`;)^9Sz)#SR$HA+I-6fhb z^?%cHnEGAly0%v5t61wgRMXWTjh-wdK;jpq2s=tswI-`5*&#A8kSP|_MgC!MUXVoh zA^G!{Zwli@Hx1BUJ+&0BVJ|5yj93>3CyG@ zzjLyM$%Pb|Vn=jz?{|>HH4I~^Jjx`z*Xhf)?!*zid{CfxAEl}AbhRm9`BFOh-oYz} z{`_NEQX))UF?fq;979oXMRGKq?KlycX!8n#l<*_Aswgh{gV))IIVW-w*b?u`6~le( zBTyG%i*5Q#jkl8G)cEN-Ut-q@cvZtwnKW$PT^f|@HEBTQy-xL(=cak0NlAUhR|cPn zHo!@9YUM)n>rcuFJ-C(OH`Ay|2&jhi%DVsbs}MlIe>nS%na1&*OoayrSG#+7aI>ah zX(Ww7RtD*7;4P27J8*b-fTor=7N&3Qi&Q&s?_lvBoNX5pd|qxic^Czs(jI#GQu%(= zF3&>G81<6%T8(dW-+ntO0{GTXyFl?-Wc8V_-moA1k2%lH;Dm<0Qn~o;-F3lN-loZ) zidd5`5E~k*QcxkzQSaLcVURI_^3K(cQ?2qR@rQD$`?T+czN{Iz3)XT@f|;_&YY!;$ z@85s4F6-Q#(1u2YsqqDNu!Q@p-vVMdGd?|s9SiGu%vn*=4GeOQZ?4l#AvWC@gnL#* za(UcDW6IX1QJFlcf1#p0 zsadwZ1+z^$IS00xSYDA&JNvbVSHLik`#?cqXwyL`ltVbYT!_EUP zIqz?tLDnpTtU|;;4j|uduE^|L2vMSdrV^=c+HPrYgXLs4*w$fc6ktGbsmQaoI5!Y} zps^KorBS&;x*|wDJ?Ws20I=$7v1nAtDH$Up2K#`PgceA{J;#v2{XtsE$m(AxH z`SxnI|F$dS1|4qOqV*v!q%Lvh5l8l@A|?xU*`rj}zEq_G$>oAIIaOehGKY}0)^h#Z zvr!NI4VwWG@9IoX<84)eV+tGz>&dg*gnsI^^+sD+7;>Kvg4`FDF#eh{c>oGvP~YdvaT0rsZGbj4&_^d$ zs(Nj%dwS@!(M$Meu9G*UmTc-J4p!l)S62hiz0OIUb7@(4=Icl|=VFVd?fiT;Rd!q}&%x8W@ z@kzgJ;L18-ekkx85X#t{f4`3Uel12mC|bY+{(x?N$!oT>wJ|#K2uPGR=bv5m)jR?0121W z&2E}+95$iEGkvk7?)%5(Q#X&tx`YnC6hGzyVZf1p#RwOF5)M|*;+taykF_ptmE_A0 zk||6W76C#H+`JJ8p0}vn5l1i<^e$7%Hz^fPB}4r-*Rdm@9sHwAvnJP|M5O>c?i)#l z9AvOiD`ob=(biCxBAY&PJ9+J{+2!^-eAr`~$t(Q!5t<1{wIpKq0XuUjT34#)aP{45Xg}Ma zh^=PjpR;Vx*75TGmsA}iNOY+E!+|_o#SQKX+1kHsV4H);gxj=ro4cn7ND7gL7WVc( z)#Ya(8Iaft?M))X{$>Y;M_Dj5=QN? z=N#5DC;Tp$=`XFLhK=h5tP)QH8$E8!kuj2kf#%47J)rm|CnhWbj1(HIWj(I@+ar6M z+kNK#eaVXHQ2*wwZA3X|nx7QfdGuisW;BDO8`mB*;UlExok4=s0US#1%n6sDiJhOP z^{04~ZOEO!MWj;Vrr+?TIwLAF_}T21D6|8x-UXl~KNjJDJtu5GD&k_Z4WPSIINOC4iO<)>*Htg!2vJagK$8lMk1UGal(fCw1 zraf%gJRR>ieRf9eepNg<;tTp%R$Y}i79MpUpu0B})cWZ2QhBVbM!wz_DnqZd+8|_c zn(!5KI19ODA6_3g7OGjG_l#}5xtT7ntb`rlF_nfS2GbT(qY{IQ5i5W<%M|71k?ju0 zh0&*ASBrUJ*`{L+910mt{7Kc(FL?=1a~DtzQO9lF%MqjbLNbI=Q}S7^GFZ0e*yfi> z_eBLeMnnp=8nup`M(hn2Vxw*W?quQ3Ltsg*7Jsd1z;MIn{>#O2SXqqhN4no@fQ-=j zgMs&T7v&jPRt*BeB?O;(%QEM-mcjrz04!jB^*n1YmchaSQ&*oSwLQZ7D1(12dyCjj z-uG@N>T<5$sygF^xA1xFA(u~%yiwmHmGCfhLa#gT%m;cxs_Hj9xzw4nk&){+KQgSyoMb9gs zG?pAg0V6fsf^_(t6M!5Om?Q@NXg-UQ)7KjRmpuE+A!K%P@b_E;)U4ceRXe|IszWdz ztJuXdyfOzy_LTCMFJVP~7Wb>hdy5gCjGugp_OE(94L`dmdxfol$ zA9N9lBZo_@d{ND>1MRvS{B+$HGkn@dR2gFI^vg(Vx@3lcxzXC8yzE$UOOBY7gHw<7 zR;SjVzujO@Ml+E<;}L8)wh?*Js9Tw!T%d(J)6u50&@US=Bmq&gD7WWs_7(3h)mi9G zD?OByiTS|-!*GcM$--@BY!9(sc#OBfS!K}OW(+7v#i|8$&t1%jDVJcERf(}EI)lA? z${Rv`0V@^b@4rcROkmAaQ>B9G&8`yG#d{Zy%UYsp zF=mE8i%e=r3toS0gC8YKx?CtTg3_l>BPMn(**hxJli;)n>7di$#_z9{=6B`z4zk@xnms-VsFC7t^;&n40 z164{79Qk5B0Fh4P2G#p{gzUwe(LlhBP{vD{$A#By7W6|dgS}LN+-05fF%PDxFG#}! z9wT7+E|rx{E)0Aadgf5Hw|*N%J&n!=RIZ*~e=iQB|N-H@m%2bA(|r61tzSa)*9TUD3k<+dMggNB;1(w3677_P55x zKryv*W4ap*hh?ML=sTXxP~FCPn?1)29WxtDsKoZ=qzsj(2&KXN5LONe1Xm4l4ZWCQ zJHCxOB{P~1wv!-~LGzyla6L9S9Olp8I2>-N_fKQN^!Yr;59Cs5#War}7|cM~@IFxT zx(~xQxLt8DYRJ1^yI>Qh{=4yiPX~+*aPR;Bf2~krXd>TW98`Zu{Lo;FZz|sHODbRe zJGnoq(>)m^@vfAqgv2lL-dCR@L#g#jwcj+c{hJa=z@XDBr&$U`oINHmB^iM@DT$=n ziAl^kikJ*Ks}lx9n1Z$hNL;-mfPIiAZQhada+qWnAFn7x*N z?==bpVr#3*xZSNPQly6|aw53nxj2#Tq`Z35AK){O5kZGZp^!}sUZ^ryF-#prfZfd4 zFS+dvK?wH0Hs|wy@Gnz|^of--ysuvxUMV8ih~xW0UW18|B0tPtv+QAHVTx8Jb3fB$ zuL*s?60UxJu?knF6XM69?I~35kW!6ajDgBT0A^rMwAl2<`Mxu%wK_hI^#1(z$!$Ek z$3|cF-?JPYs>q5Fd!uwWePD7=1KNCnT^=_B%cL-pzevHwx{YD$ZPwACeFc-pQaQrZ z%2W_n>sot%a>Fl93X3kar`%}X+brcpxE%MDY%dlYl=H zx4;A%J?L5w^ItSGMV9~oN`*Fdg6WZGP z#L^{=RF&SIxkToO_DUO&W6sAkXsG^kND(@6=e@8JWyiPdx)7ijGvlfV&#Dm<@_OkZ zuZwTF6n0{h#HjsHnW9q=G50qz(~Cg&>3dtYE)U8-RIvd5f9YcWnH; zZiV_t|A0AP?;61x)9F)khYo^-kf8U~zCc4hvbUh}CJH;`H=C^iLC)Yrw5ui)$ zDkfPD-LCDOMk4vOfQ-amWWlln5onB_wbWNq?}#JLvO^{{IO^}0(LqpezbN*;k)(#u zXf~!@hniZ#2|&3jk?#Cq^AW)*(Nk{@CI+cygw~4K2myakv{V|zlw&Gz8hK(66Jvo6 ztH+x@V@KaNEgNj{Z2%ZB<*-Uv1x#bvBuqYDUliYl`RQD)mL_z3<}bMF z$gZ1iy2J~=I9j3k&BCHGk@(|jR9;{3xS4tfB1O*0EhccKeNG_LipCd~K+r^Cemvdy zWw`f1J#J7-PRJB8$>(`rewWGbc?u-xuloQLXg*odxa;VaO-krl6#UHiD-N^NB)?%t zAMPTzs3AQ4)y5uO($?IG#%1p%(PBKD>f(LIN{d7unAbq9eN`vDMy0O9@ofnt^4s=a zSoF|Wbh$LPhzNY1*e?vFDKyKBGBt%wG0% zc$LTfSxK41Huw{ev|qrL8`Q)OT3mz`EZfbMs$H~K1VYB3LbSrf`XIMA;u0jIb&b+6 zut`SZA5E8ORAmEh%VE+E&Gvw!GWD~nu~4FN(Ox8YSCp)E9u$i)$_9US&HN_?yoS{c zNL(##aun1O|JN3;pmu>93ht}h+%C>jm%X#^Lq4GfX9oWm9LEYu3Lb386TB>p#G zOgi8yGPej>UEi`s!pnRx(r(+$b)EkzLIT&MW+01iFlC5UBPc|io7L0l?=gAga1&Ae z+o7&!8aTFvrGY?$ra&zIn^M5n>Q&cVcP;0DeTP)k=U)K&_cDg$iI48MF<%vek%orB z@ZlR!kMb8}*nwL`&n~G8v5>Gp0aHbna?L3baKpK28EuA^n6MrY4}1j%GM@u%0Q<4y zhD;V38tfg+&tNTfc_DuJ!wdX78@jb2E?#yqWvmo5>ke%)c~5u^`mRWSC|KcGNg12V ze0nAHT>xyK07R<|M^tsMi7HT7kv#X&cw%D(CeqHf0%wn_o!)5h%acX*h<3W0CxiE{m(M(?r z_X{f1SASFVR5%X?*2SV#9W5qSvv;JA(e6BEKwZjFk#qRARWei3fcp_u_jz?&T z!D;OOe$1PJl|kU&n=<{MH&u-fj!PG@hd@aX0hyOV%>rRZe_*6>g&o-@XM80L}o(_~1q6nRPf}%Ma8D zQA>MHVP79^X^1$j14({z{}?8}-+pp(d>gB*nA@aG&#Rr56f`;Q3E18WX};0xp>i;+!mju%Lp%8-LjW6HbT2CmeM-I9 zqbuVEqWCoy7+u8|IXn);`|?Gy@v0j&f~ngam!R&cb>;lQTRaGYG?(AWw%=$ppnPz5GIuxFe*vKC(42v+h4TJ0*cdm->G=0>1vQrbB(*>+1p? zYZ7ewq|xSZ&1U0}giCxz+wI&4h&P~&b(;ZE;H0;&q9hqClnxn7&KCC!%#M!S|5o{X zAKK)}T{NSHz@yPUa`nlC9#ifCEH;zTr~2cK&-$KSfvyoN!Xu!sTgzwGe>kn5=>MiO zv$7S>eM-kq&$g;6WO5i5!d~(;O_v|EF)^)*VFI07TC!FnVhn4*uIjEC+j5!*uq16Q zXIM|xyQ%)o_{z!W^Ny{3YYirTgB4(-RVS=4ksJIKZ^qEORab(D& zI_$`;k*A;cWD~KrSsQB?*B>&vf->_wMGm$yNZKS_H-|q0U2sLjOn)ERXVrKc+I-Y# zVpIzrRA&B45uCtynqA9RIB~aCEi(I_@bAmc{os5j7T5eVpOKqCVm*l3S|9EF0AN6_cBd8{u4J);M7y0;uH zsNuu);~sXlLK}B(n%?J{t=p`7lI^920XU{%|VbLSa(Wzs}sgG;I;pnmav`|+p$E{p% zrM<5w0k1(LrUEt?Oj$o>=#l-en?VmB3R>X8ANJJ{Hha}yD>MX zHc)Gc$7xq@58OmDsqQZxz#jejWs~A{8^+2XyHwy=q<*BRBb!vQ{=)u~AR%j`yVluX%5{<!t7Cb}?jFF6CTJBSuFP5s`-SC2( zk|sjt z02mfqxARpifJXdb6@QKPekSR!U93bxkg3z^Y~@J!OvLA&4N!#32qf45Oq|)p#l-}M z@{p1#p@7#S=8D(jVrN_FxrgAYk{%=+>bP7lRyLG@t5>kYK|k1u?Z5% zgFXpjK2m(9Z4E91BS&l~vl7yRKBrLZC9#@@A0sl4xrbPn7ahJi?m64J>5`bpO7G3x zBbR-`%Z7NuB*NQI(#|_h+1N(m6?*B=I(4@4mg`O2m~Z(WruTk2 z5BB~Nd8jdCj|#yR^!M-+JtaMwl1tlW%weSXGGT0y)|L-Br1E(GvL+!L$a)2C zA*{#rLDz6EM>HuXm~ni;aSgjQ@;4^i&!RdtW_ixnRe9zfYd}YjC{d|9T>N7a`=b*t zD=cOT$Y-^^AJ*JRBJenooE#mQ!&8{=r-CK{GT&^uLDzaGfIKo9OJ#>21L9CUj24c` zz+71U^%ff*pSuH!1ltrai}w$9cl!Xof)%bujQRJe zaGA5(Zh>df*IDp)Ku`#jE2WnpSRrGCcC^LJx;&HjqVr<2VIE3m%~WcO4Q2Q->t}~8 z!gP+8*FC4;inYb@h>0eT3-j&(|M>|W&*kGNAqjZ}j?@*G^aI!!47yy`0ONx+ z3RN{caGTjHl9h!8t@HJIBp4vS2T;3&u(*eZ!{&>FhhU~cz#t*uvzSU{)4+sHSdXW1 zPE5qxs^}yNZ&qCBbko!xTaN4T!*mSXg6pEd^bYb*fQCN}qTLg#=JbgcuYG9t> zIifz3ShWHm#;K2>l%Z6FZ#z!*+eZVb~C9+x&1(>Hsf8Tbf6v&`XWC<>8)~kC#*8x;V zqc+#GT2W!_6+l{Ze=cY8+jvTv&2P+x z$Kj4mXHeNry|cOb34K7$Fi$elym|1|?Q7Ssr5TGbo*KeA7K>p7CyQ~jU(b>aL8!_= zktdnQk+xR=@eqBkYw@{!Uf2_uy!2T@WeNnz-w8mj7CYu+h6J8>7**W+Y|njStbn*& zF&>+xQuMN~`0SGjIob`^+1*K17t%yLOrwe2y$5U7biJ+(}8zup2m{K}rm~p-7P^^g3 z_N*pflkdm7Fwp$@e06(^F=Jb%R)#5HyIlXthA7I!x5nO7tQ#NJOMwU&X~7~?o**E% zx&e&74z!Es8WV$??N;li_;?bKKrubs5H?GV0a00enHuy4sQ9M@pP+VTQ&gWq;;6Ev zdns+UwgbHW#KX0_jx7VE_oA&Z6$4lngN1iog)IKB)6FHSbx67$t_eU^nk4A`jG>nu zEaQ4}FE8Ld@9guiuD9MU?bHoLrQZpH(=4LC^_eM`W_8_NMD4sR3c$xo6LKp9m}*O1 z6W980o>y3m2AAQLI3$e%CR!}OfKS8t_pE`2OYdwqq|Y}>%1-Y#d`U?y7$BKImJHq@ z6+~PKn0bj|m|uVN%~_w!mY`$ma9^W>$x;wg40gvW#<9?3Z8sb1y}!S!q>c24HW1BfAiYYLjRa`QmjQ4#DT$2k0+D0$fEDe0&6Kqi+JrSc4AeDA>ee z>u)d*Ibv66N-Zj8u9uU-k%N<;cNx3ey*rrUIxoBe1k)@ysFllq2Pf{2C!}rFiEDOG zOeo?HQXeV8`=Cv!Jd~er?wtGz+Umkmw#U8lY8cOGb2IKh(aGwU22wLIVm&l>=L%@<4e~4f45A8YQ=+`DN1eTe|N~0}aEVq++Z_-NX zpd~N~{PxhQXJeqCpn95>clvH(Ly5yVMgiQ)VyibD5$|{U7murn)5Q=IW9HI2iwTkb z&tJBHWi=%LCg^p`bTQcop~L6P+-t#HQ@eruKCZ)IO_RBDx=|!J63Y$WYSNH57Yk-=F z*YSJ>-u~Iy*Sq66x}8G#w9^_sLIVIU-3HXgI>mSaDI2`o9Bv*S2~U#%fqzT$UqRx>fhj`;)5dy=&H5 zbB#6U9OJ{Nix-tdJb3KdA2Za5TKcK5@g-6q4puXH@Q~9R`Djw}twZL~k#oDWCi5E5 zOc|jrH@ha+e{sJMj^<)~3?Ky`V9z_{u7R#gH<;;>^}brCYQ1_KHCgU3p*Mn1&9-fZ zg+mJRgg?9@PJbQQ^6CwRjwcc}G9od0S(fy1V`pGMhYkzYc9WMVGST6#-fB9)A{+GF!S+z z4)^nNL2syMN-VIOJWbxWqCfH_w}^a3r)3{si4UD)vWFyQvY!cbKSm^hQc(b;D;)(7 zj@M5+do52a&3s>%56~EVj*vV)uL)a&IR+SB#84P5g3rlpvz!J_zy<|7l{x_Q=Dz?0 z%QkQoCqQ0z8mnren8SBx4&=;!L_|cpok5kFbwog>>_&Vrce`}t?&yem^PU=UIF3p# z0ii=+g@*zpn*W(u4u!sw2^h-4cG}qZppHBguq=&_vEFKpRNS@NNh|iuoP;YCTr#6Y z2ns|?0{&mEQr@Mmv>{6;K4yw=H)!PPIDve+s1 zricOxl@v?AjAc8R-K)*Wn?g*>>)R<|qWL|8Vx<$D1-O-B_c!4WISCT5zNIy6qW71C z7#yE#A{aOrwv`%bL{v~R6d}+WRRssyFF~Hf7XatTPYuf%qin2lu}c0+yG^u4V>Nd2 zG=q5zD|8T;^ZCt{O4*k$ocIRr=oj0Dl3pOC(hft#;!i15ebr~b#%}2UYMJ!xD>=p# zk7}i`W+CJ|jv=`X^06P*Y07XxX0neCQQ~$cb*U%L$Qk6S>FPnG^vXMV;s$+z78QGd zVKcrw_RcT69uB@ZQom04NtSLERUgQ(|c3Y#&6*Ua?v zt0x<+ySiPtampZv)rNYz!)WGZecvgAnI#s{l0DTZ&QKwVpmHju`5Y7XIjQJvzq{-f~(u~Bj+6_(aLiRp6}_QB;(I+wD7Q_sMkwJ?XycY8KPHI+qP zt-Z!&`|#H;dwh&~7>ms<8!b9fCdSAB`JGIpPQG%R;L0Hqswhu0c#P_ZA6`VFUrHKG z4s4W(eHrWfIKgS{{4$o+FvUxCpBdqi@5OZjeA7WdzraDCiRk_ab{S7R0w_j4S9`D7 zR7<*yn_0M&PPz|>9QfaB#aB?)^eH8w2w^t#{F8`23v?RgmXIEeP946+eG{dXnLFQ(!nNns;rV!Cqt7` zy(x#P2{8OmM!V;2aE}Z-$nsSsycHWhMgnkzgK+sZ&ePQ%e?c-E2zrkCbou;=|6y4@ z!r9JbHES!*d9ybCv;P^Yj+o$i?P-f__Ir0*Dw(pXZ8(siVuQKe-*0b+OnZsNF3+yu zo)H?ypNprmd@xE7qtD4Ab2EB8QN12O6lLRV$jRZWapj;H;mz>uE~Pe`;GWc^+rqaP zPpsm$2_mq_;QDyQ5O~~rYd`!&JM5TQCf!L{HDZQ&_YNZPC@&*8b`2ZfdGF)UFiNeV zbAI;1NBBa*&?^iCcwCxf(tna}j z$ZVCQJ)>c}LX-1)YMGfM@m6LZ4AFgWgweG0k!>{i@IAmPTQK^m z5%MzjxvXyJE|MkhjHL zGwqB1yjmIeO>V>^s4N#Voc2=4hm*ZEAqu4?>c5#!u19!Y;6G=rj8t{`L^XHG0_IFP znVxW^@PtJwZurLE9=PVh^$cjn3+zELc3@A0q~Vil=$H3x2G}^3>3UAY8mGxZ%5=nd zMNrcVP~I(>k(bxhh`oFv(?xwTepZVq2=o>F;gR0cp;*1>3wKs!693A6u&grKfYL`B z+0ZsB{&^NK$h)LfHRlC^|A(RQgYisT>jntcO`&?@0`QV=sDWw}N+srqX$u=Uo&w%T zpOppciscrb%(yK=_CFrSemq^bv!GFu7%}6}=3p3!JnC+{P(n0c`efDGF7#kHvM=#8 z2oNAJa;!tpfm$WoL(Yqbj)|NyJ)O)^n6!~*Bet_uv+UUNdD2kLDR}Iuy9I}Wt6y5P z_84wR&`=A@`25}vDEY<@k__}!Ihpi+LEZ%eDl=Pu?Y+f=xS^6`&{4i6?y&bVmltzG zq5ebvABEJeC@$8`1P;Z$)yRIJVTUY^AY?Fi2AZZ-Q#W?{vSw0*YlKfmGP zT`VqRSdL}5LV<`^8#^xSI|#wFH%cM#tjnU-L@f`1FB6$$uvH|^aL|6Yj`@TyfC86H zzV%LJ<_5M~@`xOr6ki_9fM~u*v^h%NpmU`*(Qm0D$6+;*$RKn`@+c{cWE#2$+Fi68 zEusf4x2Hg1QXbY=RE2Ls&md`oPD5@2_=3)vk>@S#9%jWg(P8uln z64-~K91N7!*jDlkGSw`kPqx(qO*x`ufs!*R5+iF0Hc9=V5@zq8+WbS7wUl7b>wteM z)no+(`)yRoKe^adHO}fE2#%^8*nT|yGM`2krL^03lJ`+QkOTfit{n-sZ@Tjs|K||* zITsPacqA;9TMzt#vf!dDF08eO?8+Q?aI}$7nEyaSOOj&l%D$C6##d2y$34C-c-lEF zsm_B5kF?~qUzA=yyMH+u>#>PGz3Xx3X>wC5%n=xLY++QrsJY=(ZWevjucw`f*?HjI z9V@?RUHa3z_F5u%z2F(=+d{v|`<~pO#8+RhdumFA3PW-#(m0NAZw4iedt#qh?YwA~ zBrj1$2tGl5KG19~@9}7j zGz+KkXjMx`9fl|w0@zyqXjL25lSpMb zj53_30TV=eI%NvE0hX9tzjNozBws4!EP~rS5ex@jG_bhaL%%PlOp)>yeJHW;mU3ah@MH6+) zM<2)9$*}8_C_S8&Fdp6Zkz4A85zyvC73(tPIDoc@$70wA~D9+;r;_ym`E(Fuv zn(}gCUxVD;lMVekD~-g*`!!8;6NDyFrnChz>orz8%p977N#0F%aixkq0VA?(otrOD zmw$V@)~TN615fTv@i??1OK$vMNlgzl1jI;OT%1en+@e|0rVdXaz=x|R3ym)Rp^P^t%}_8Nih8=pns zBpIvMS?9m8+%9g9GL6Z+sg(g=*^^C@t-w(MTGIM zXFqwtOTF&U#O8^TCQVA|x>9`H+w#2JLO-bgl7&z=r$Lk|hT45$7p%zh=St$xofs_s zHWB>eg#L7EzdYWZG;IDQW_vEwR&$?+n3)mkm+^d za=+#LU}LndsV=a}x>z3YbY9+na>gy4O=qhqv90;UcS0K>t3&a{$*Git3zr8t@JfM= zf#gidw2=7UpI{i)Nvu$szb31Pf@Qcq=GhF;^g&-y3AjA*S$rg>Tb@8iTkfo4?RHul zHcgynWG^?=L@s{Tgb@{m%Qb@Gi)$I@oypv#aRBHK@gwo$zv?n%K2|Y}fp8-6Sl77B z2|jOmkUSvO71Ap?aDUc{> zjJjoo>>&z=YA#Kc5Gv6f46i+GN5AZYxlzzqXC|{^IIjcn%7{aM#}uyYjQ`aFdIe$+ zw&9yhwW~44MGk#~fr1z%G$DR*gc8P~9%^E1z9a3x0GM!8<3PnIPwJw*J38;~iXlNt zhmum*=4RiUn_m@jXG`GTTRul|1klP&jK*Bfl7d4XCM%N+>^Kr6_6|I7cQC5I*fDuI z|2;S(JUpC-#dTMW%l4Ax+nz+c>ia5)3sAj~$6*hQefdUkI@=O{8TtEyTcGpKR12%9 z5XY)or5#R=;EQrf7TY(tD)lC;`D%mYg_jN@XIID2jDwrQapZ9QkHl1ZRU{1t z{a;Y`#3Sa4d6RUSl?*z#ye<^FpUhMi3()Z_>blF>zuz6TP)**Im5&KuEOw;VFsF0)C6M%9jF+^K0p9^!xTjz9jGkpSiB3^rdl!)`ZDNm8%pGphtM+|}0a z@PHQD;dWWM&HaOjfZrwIa5U$z=(>?1xTS^DM`+*s{ACK;w%QKlrR8Kvy=#;EE~BP~ zk>BGn0GWU*A&=iRY3ZX?5Iv^^5m+a#(CS;_p1b3ID&c*}AlFpjYV&6H+^byR2MSJD z@_C~hP2!R1%^WF@rS*kv0}b1n+4(yz+6&=eXrUIIgRM9gQ}7h zW+a?#44$^dB(k}5ar2(#NJ%5*eilz~CdISzZj4I!PKoi5`PHDWJy9EI;|9upu1k4I49IpJZ6WIfPX*?%K$8l6)@E%-6$DN_PO2`KR5 zv!Cj5f^Z-q2pf|~f{m1Z2+MgcfIM6Lf^ZPY2dDvIX|$?CQ$gWB{)TaN4LcnDX_PH| z{+Boi4shBCFL>p9zvEpv@@4Qr{QmvBZ(~EBqxM{!Y6yxv#)LZVfP6S+zuWuPylgr< z>^=D9CQ_C9`tU;lnwgndtILJeu)~Me@M&BouI%AtGy0nv+CzpWW9>DLwEFnKx@MIQ zA_pF#Toz}n;m1QqxsW;T7(-0QQ50Oju z`@!^~@`Cb9qu9fql8G0%5ZgV0Vn zC(GvFU6BU#2>rxoJ5EQsOntOp)%P+bQPUWIq*|z#Ac=eYwV$APVE~EZF_Zi2wCzmf zW&yuMKdZuWC4`nq1(l08+3oYB;u9ib%+7g~RXf?md-L_*Sd07;NlS_mm_N~bGIQ?F zjLBTCEJ!F{k4K3<;fRLa-Yt}eH!24cH zCMg=oEoG@MU#9eZR$j|#RttPzo~GfkMgd)6gqL$xRIsA$oOX?&_$hmoC`Jkm$t;UX zWFczVsPP8QfmlBW+$`or-t3wBG^+mZ`P=JYkpS0n*Qj8Lth3hXq82+BnnlxrIU&#e zasL!&93Hhy60@`0x}lb*Je46NP0rTPUX!TbAqnH~2U?18{kUFMJ=Mss$PSpR2$SzK zJQ{h-Kps^9YJ_sNTfQf3qG6RlT5dxuJaHxUY&3vYp2p6!yjOY*Nb84|)qdHy$aKi=jx{90ebj%^ zZ6|n5VxMuw2%|x%Y0^YRl@BEa=Zz^MLG0`dJ@u(|jtMyd&l7#@Cgk6Jf~EY8I{`|b zhNo+OU;xE|3s0kM1J#Fv>_1FSbPhB-FklMr1@H3j8~7LiB>r3#yy)nC<$p($se>C< z76CqCzE!i-?v0&T9je22I{4*ywV~1JZjBO$zw$^bBlekoJ9aDc3e&5n^FHMN!)fKH zUBf}D6GFhpvJC*RD?GlLTE{PrM7e97cJ@1YT()=<1gz+vq*r>2>X1gtF}U823aySp zp4NY5b}WbCyhau_&?Yajp=^WWecQFc3&zdd=wQA7-Vt?LZAK+yaDTlZo(Kz4~Ti}KX}fRrGN=-0dx;4OmeUa+3j&_b?qw^O5F4A+z)5~a3;&? z=#~L|8lB}LH8cLbP>V%~>p{Q_n)g3!#YQU2*dmf=&`>D*6z*VXuUGjopl7AJwEfQE z@M|E9S{pX&gVg`ZG?nIR&*TiG?@6+LL#_oVS7$U2YfCCabvTioLGkhVeV1E6IqT|{ zCA~VQkIImIh&*`kH#@LkaMnlq+#x?~E$K8@0m#TsRP%GdF#qL%5$}NIs?AYvFWnm) zSS+AgA5}F(JpNZDPOSM<@4lp^eY$|yh@|yBp?XD;!lt)L*XzRnNokI zsB!2jPabNO*rsa55^!#-rhO4TpBGDDusa-2MXpS%f}4Ed3WtTK7B1)t4WR#U_WB#b zOaE7R+>dc*H#YjK+R2?)-?h-hJE+@RJN&cz8M1)i5$J}bE3~GC<}9(|fxJZ#U+jRV z$M@80b&$|16-ucj2tp|L0AI;_ktX}h$m{E z*D?jD{}NTH8d^6xGTw8n3%%YPzq#8m<0Sc3iYN^-1fq@_M0bUDS_n`BID`R=B1Y1u z9@MGgtWV3d@+-z0D`VAyY({dOVz%87gMBuTYs0i(qd%<8i7`OzH}U1YZ~SGx10H>v zYaO#gOZEE(;dwV(84)6Og7TXdlCMaaB>w8~9%y*SXb=AQv9I4~o&V1Rdf!W-D3J8Q z5bis*KFn|8ou@KKzJ845{y)203H&a&MYFU}AS}V|USkA#E1rv3YSHlfKM@e|dy<^^{Jlfyp?krUBj7>pp}B|T;T2=dzZ^P5?DBX6 zv$eDRC1V4r`0fq%ZReZA+mx%L!Bx;8X{WavD(>KsCDXK?QneAQJ2=&zsLL zPSF&WL2xZ>g=Bp$*B?B{W#fORQ;Cri0hlvk3?c^7Jg_Pjlu7YD zepE(;q7XpSx!K-gLjTU)N+01i9L<&O~F=orXeGO1! zzqD_DfF8H`x6fL1l8ZoW5bwj-jgD(&82+lXV@dMoeR?zlNRk)0`DMFEfpe0uCCHIQV)Kka;(%(-li4L@}8HbP}{wcffDT)ZgIti<&^)!1h4EYp9ja`AeMZNZIlUpdA}BNvHg@iHn>cM zQbIx$W41ypnv#B|4a1b{37?&QItpZw2|?@Rb+c5V3J&R+6fBa$2y5^{pa&Y#2^+sE~e~O5caGAUbdcDIRo&zs7!Zr#_0zW38@MvN;P23|In~v%JlfVn+2^SFC5<7x? z!An5!k5HViR^CI5=MnN*WXC5|JW=CLR|Fsy3#XaQShxI^{G|BG$$|_TgJNVnEQ;AL z7eUpZ^n%$qAmkfafRvRlAt=*aMn-tFjyMvrpwaozwq1>+p@+rgAj>YkgPyv*fvf(l z2`$}n>Aq|%jCH;%ty@lyPWj>Tq_ORM*gKPIaItlqu>i=s>$ zb-C3|;BagUujxdMj8Apx&K)-|C*~i9&pBGnVZ2}ffavz@ge(PkbKLIy7JhW4YRHN2 zAzdjOD*E%2_(y1 z?(A~Jf`kAUn3*ieQ1cBh3OfcZnJ%C5G`(*(G8z;c)obq!DrL(LV6j7+p_st_iNId8?FQ5}ft00`@SukcP} zKzNUu4iy;biIIJ0f||`7FFKoaAef!?wc>sw3?vz6B9{Z@+2s-M{;d@$v-uoq!J1nx z5$lwWILK^l_ey!=)9B-4XU=FmleoZe*#6U>VJ64mr{Jzw+&*4ylMSWefs z>S=Z!DOvEj4{8dF#dv}}_N&HG-W`aC4=u@wh__Kca-j6ovB>ji#9Fjv17pc}?5-Cr zl59w!@bHI(r-AYV(fciPgpXf}vz`)++zXJ_TY0M5SY?AqFnoAF5xWqqqJ}9FBYAZb z<$4Q*e-eR1J^vpfFsOo!dU)6A3=^wB>*+}CjK8t}quC4d?#z$?e!<~U?u@&h;c|z6 z)n)7tZYVc`b<#g&%(+;@20fmz@?M;%J=N-=EQ*iC)AS^NEZAl#2uS(cPVz}n|>rzz7iN>YpmbGCw_ z$S-BYynUU4E+~JC4V*9wL6NLZ+NX49 z^TDv$z73v}#%)eJ91xGa!SELGU#Hd~FHyA7U@*#c)%W&uX%Fm+DQ;&86vP{F*Vt2V z8pNXO;sM4f>drv7BX3L+(m1O&*S&Hk%z&)$VxZuWP@e@|e;xrSQIMfy;a+}&S(3ha z*l$%>5vT48@CXl=fK!9F{sR`nJ}D}Nngvh8!<`3r$Bk2)C=|rM*|IS!9jxa6CU8_6 z9~Y<$K42UxA3BHIwE`lC*Gf`&XiCIN)TmMG)xf;OwRVjKUNrs%6H~v!$Z|8Z%omit zIEsge*DLAMJRn|D0*XsPKS6%ICt@@dV>kPd76sm9m)@oFH5}ncnaWg_Y_ckI2jWMF zpCEDVAVHgD${}c-#ay}*#__Z^Y|cWKx~$5ka%3xt6I$nX3P_J&AyG5 z6R~`klC)f}qzCf-uNv_LGZvSok5=b6IJ0$_dVKC+&q%S)!c|@y5=vK9tA(%io0F-? z=*_@mB6+mCfdeds@D`c!z{}#^zn2rE+Z%fSGK4M^lv9gD!^;vZIS!I=0>}K>uyu$p;;&|*NH=zlYDWbks&YlMQxD{)`gb+k`L2L+4Iki%o^%}+AA$k`!?T!}$IxrwGOdt}p6RYEQ+ z2*ue`RX}gh(xzQIl+nF`NGmz~qO$0<9xY1&UdA8aCXYD(O02Pw~TJg!%gpn z9x`*T5LOu2JY&Hnwd3 zM(pc-W`;3J)8NC8I>#?sO+Id4qsqi^qbqI}TGpIKz6jB*GzPXVtU_{G33MC-tquRp z6a>r4z$#?_#oO8yhV5nz+mCpuQbp5&P9+E9XM?${G6CowSFgon7!OX}Knse*5sybB zQx2ub7(w2?{yd^U2g>J2Wfk6ee3CeE%DoE46=dmA19Ek#|Eh?^NRYrZ7Bw6Rqy>I5 zUE+4F>KYCaEgzI4-UBDBpJPp0e|4&~rkJKfKqgh(5u4mK$da^Z!Upess6M)>0?2rH z0UH&kQDui7ex&WkZ>%Jx1l($vw-l=`YQaK3(8KTTgIR-DMV8XkcglE+3WCsLZD(84 z3sOP+4Mp+F+N?QsMiNNLLm(jWUJFP?0G>7y6VvzYnBgEH0{BaBXe1E=L!}V7pnn(elCC@ zs)+h4G8(*Kd7zwv;+i-&n0_M7tt-WAr%>$vM^qLss5s8Iw=OPgD=F< z*2b^Upp=`5!<%(2!}5t&Hl>IENVEZBBLzJuMi#UCLYgD67H+h(DW=+} ziyR$}Dc>i+5bN~0=36tFm%YIkVvF`>7n0j!(06Np6YZAd&eN-7@6zx=i%zDqsC5V1 zU?g#MIq$po&f3I4pazYJ7p2e)q^rDu0o7y+@CUcYW{F9((@aQ5o$yaTok?ys0~-fI zHQ`w(rk$4L%SITn-MTEMwDZM77l2mu^z?YuXpGz1+G+s}dF<9`O~il2$|dzXz}baY z(;v8sXP$g8Xwxc(9TtB{{6R2N5$7UKGTD7uTva6+O!;4aCtUGm}}cCpO)2-S&hfV7yjNDJ^A9Xy)3d_Rc#gU`Rmaat=^n?99WH4a(O z6egzT32;FbMKt5O4Bb$KVU1-_d+NhGAB z3A*^(0zdnV%IsLzL=qO-FUvtUNxU2^uK`$l`AodMW*y3dW`Qi=`w#mt`q2> zDk_h%dK6f)q!Cuk=0S~V1;if0%{96|p8@Bm=Z(+00rJ@zwujdO-dh#E`+h$6rrsY* z1Q=z9)efE}`?A>P+Rfr`l|6^_Z~7*7-9Lo1emj4M?@^&V#wBT;_(!T9FDaXXz#A}3gvvcvPGfxA>?LW3a}EO*5@ zI@N^xyc^^(CvzD2BJl0Y(%>4VfL`0J6wC8?Ke}mq~{%&*#Nlr&ZOuplAk{j;@h$PbFs>358r70U# z=FAKSxd})dG59#aBf4kDI2=r z4*qs(I;aUAF|?=(*@Ezpe6T`M3W*-QSIn%~i?-|cg6V8ue2|lm+0+W3Y1637ys=4uhcHO%8a z3cI5}6bT=~MEI==*mJe6oUA3G%yn&bsm1-7K18G0==p(FHnsco%@6eTZ?2H54W0{9 zXUDg)b6`KKpkr%L^qcPHxhKC&<&NLjymmLJ0}W-z%S{G)nKlG52{IE95lsQ^AgDcl zZ(c$%*Sn6NGJ}aM5Lo$}9>L=}cMjwKP>R4{V*q;}Eb??37G*W6%Vv-06)!c^VRSM! zaS^j7-+nfpM+Rb$C>hFlAu^pvc+hW9lcJHPg*fVCMlR_3l{-yY(ThY%O~C0Pk}BcX zTrg&}u6pq7_*Mrp{MMiM7YBe+RXY-+P_Ern3hqYDz#u0G^f$i*8l6cQk)g1WC4|8w z%*csiM|a3Ts6>Ob$U;oSf+(bS@3~6GCywu*Wz|TUrQsx2w8qYkxMTh-*KUw(7}aEp zu)4f%<6+{80?BDU!7wsa+qV*h5KQiCRR|7k8_3BKwftW#z;eez;~Q6*r5hMXMgY7=}%WG8#ZV(4E~74kECQ!TB1F0#`cyy4 zwyRz(el=Gb75WYTk=~J0__17BRF$M}O4c)t`NM;W3E{SXSBRu|24!N%UZH(QV`?Q} z?)x6IlqBFjLGrnr7t4cDaLoO6l}+wW!ynDRsHP@|g6)8I(Rb+^*O?E$wYq*tA(CQ&={FUX#{AD`|J&P9K7WvuCA?Z4A7Z}R z^?lyvKf&5TxlMQ^k9<+OR;393e{>_wU;B1L&2`#Q<`q8&jRy&zo*`(?_;>iLv`VS|lmzmsBx{xOn@hCaEE@~<@*rl>@liHyJ@0~0saxT|8JrE3=u@IdD+kT z1nzkI)BFsj)6_D@|2x{}$BJMa)?`Ltz?bqMKQc-F zbBOqSUjG|nK(g;oT;fb~Gri0npsV}OnE&_NVm45Cz%)xPF;kHvFd_j4Yag8gVa^&d zgGva*F`|P!LiG6{U`f#Ww)wxulh2X=kb$CQT3uC^qzVqtO-oV}X$uHmX=Qx)`p0hR z3o!zQV*AucE<)QE5?=4!Z7;+6Rl<3~&nmw?3`CFrCxZcXrWXD>w_cY3d5=tv*&#*P zcIb9!nGcPHOF%m@*)RV%lIKK_!0A}}GMVjv(X62Z?{iQ@`oE}@GW?w6x>Gx>qF9(% zmS{gE>WBbb)cjf;)OzREJ+6DM^$~I4&Hg`=KoW~eQt)GlNz3cxd%i38q;98SwOW5y zu36!+%keQ)enY7&*wpu__1Vc;gb?$N0mWb{u;a`CE>-E$Ow)_+MuAq>;1p@Wpd^%k zD1lo)zHTf#&#(DcO)A+y;AGn^Od1)4HfWadx61N?`me8@h0PUDTJ%33K&<(B2?3$# z`8%kA7=R1INv*TR@szi&)2c|O=YaOIc-9o&^%vR5oy(WK_1Vo~-zhVAI!;}3OJJjF;5kOD)wGSV^9AAac<)2R58f z_r;K_E7vy(u~Ye{PP!kBtk%&m0~Gmiazdv&t5Wg>T80T}$P*Yh9Ero%6g-e{t03XAhHJ}LCU7E7|IVO7hh`&&5uXe^%mg=>naU=sLmg$s$bb@*ddB%ENGv>3 z&SS1XOnU7pm9W&xJ)YJM_v%weic;0o)enjTW;K4+GNlzBnDs7??9(oLpW8m8^7tnQn`KLWG0`V2NUi(X? z>$@Ku{e!wbT6Ug0ugU>KKmMDVRSuqfI)uZ5Hlh2A6h_bOY-B})^(|Uan)LwM1I~`E z*6dUt!Z<#g3|QIp$+0&VCfXxkTyHSM8(1eBS_?U)* ze#J(&>Cc5nzdcR`B!$9jTkXb%r=q_${<*hJ^(vvVzAo~>L2jMA)f4Th*BzGP+I1@~|(gg~Kq2 zameEwRcjNMnQhKj3RO hUXPxCHE7LdkBWB*IlwnL%tVA`pM5K}`ZziscnaO^wL| zgV4p}=y-J92t|}zmW2oRs#gA4^l#03Lnep`L5LWirnJoIUoO_JX)y2UjJYY z@ZBgxw%KrnDS|0uEl@0zO{qd!m80DAsk=YMLo*~3;TGPlDq)WheuW@=9HyV(sy0QY zVeb#7v{#gz+xdW#{o;esxi(aVElbjDcP#QtX;Z$3)Hx|<0JoD7~D-3Oz=I9)1dM$8k?+p<>?($78f#7P@;Hq{RM>IvU~+9YwL zVW(T-?(Asd*?54;T-)AsV2;afYFfsWlhpb1q}I{Z$xLAQZeCuGU%3uO6A?~V89MzJ zXE41B&L-&}p%G$zY3@o=5eUnL6^L(zUx9^#yNi78^I;{^+c;P8xJf7vz|&* zUtzZ^n*{4xGBf)7Em*F8$YEsh*kL{3Ea*A1C}9lpE94C#DEt__&fsd5LImc1qMv6{ z!^7NthMi<%s3Mjx|Wv@6j>JhbTwN|ez-v_VV zd2B|)Q|~9#qr8hJGKXY{=2-(c`<`8ROYE)R;bidF;O(oMVg6`cEYbE;_d587=;0|t zp=oD}X*-@U*N~7f;lU(K?GW12|IvLjWT3_AjSd*u5oH3L$?SHyJI3ZAgKF|l(p{9R?ugd47y;v zAI?J$#?zzULSWJJ0ne?l#tdQ7X2(+n8jUC{_^(8HjgJ~R3p@t_GuCUIJa)M4cuOS) z*j-Q_@(#o|;2>RbG(2@%eh4M>i!t{-zuK5&%9ud=@hj*5LAB)O!T?WYqE@V0RN^Wk z`pxJTO`HmD>;M`}TzjI37#kifPK(QHM2wlM!9fHY9EuVyErBkjlH>{U5!NyMT~XV? zvc8}?2@3iPw1cxNxDCC{a+7WR0&&$lO4#o?@@VL+*EiFDapGU9XOuknzVP%Tcl)8t zX*JlkTrW^WN{5+)2ZI+aTv!rwx@s_9-Yn+bS=42xI^vynXrDHp+-g`zS}ax?(8HdJ z;m=zoP{|Gm^^bHFgDwP@jRF_>wL0SbJ9At!$vnJQzj_wCMA zI)%n#FHdie5z4F zkpPXHHwY3wTV99^U)cU=4ngn8UDO9lfsaFq(xJkU0e*!7@x(|^ikbXEfqa}nlM~M) z$>@0uV**QpRSF~9FG7j<9{3I@ zbHQ<}URz&&CJwb;EDL(hc8EV;l=;<(s**)^`4-*d#TEx%K>1UTCGa;s5^b+qCX4yDU2`fz5F!-fYNsjeH0Cs;!6$6vkq zAQu=b|0^wtjzd5|0DT@TP4wpSGD2f>bt)1ICQ5?h9s#iZ|AKw3g9EDPy67QwfTzdZ z{J&ZNoL2wf3#xI6_U(htKBGo@_bmxnQ$Evk(}f<*_PjQK9Q1t-JnG@jQwBFq6Bh;T zRUim7lD+TN^^4-gjez9+NPh9H}yz#%;pLI-yCJ z%Q=$nkE(TL1maa4EXYi0>^)l^h-u1VG!I^5ODXVsue1A^q4JZnmpmZ9gWr4sFepgW9KV!MROe%CtMD9c*SBr{eq#AV498JMSDQps^h7W;J>O;n2A69CSZy7uoDAqoWuYH3N_|G6ym=3 z9_xjVk8inE-(3vDwX(TLXUkT?_0qaK zGsX&?cI(}{!?7BApOfv1RZ8@9OX9bO5v5+8;UQoAJ44O?ueUjymQ*VHW4xtiI*2Y=ujuAkR3IL z#)cvPNf6Oic*aJwGFnT9Rv2j87Fqkl zG_bRFm{)|J7cj*x!OznxVeB|^mrf{krKcCMlH6#GRcGKe%B1AGQ4^?XULCBNctFCt zr>s2L^3QDHaAzj}a6R(L;FU$vrxMXJ31DvW%xkC1m#rpirLUMSttZDSx=`b>)0{N< z6^yn!{u_t*XBM(DMY00cyvJorzV_ypSi5j%vvO1A|&VMLuk zb#*tGrF_S^4@WL7Sl?Pmfb8Rtgvya(AET9^ZV6Zy3~HN-Uteoj$d+y1?r|J?uK>G0 zv=`V_dC#hHfdeI>P-ox5QzPZ++Fz|(DOUAmzAExZ!3Vn>{aCHuK)=8s9`kDVhQKa{ z+z;IOxLKssZo#;K4YKgni<{*4E(Pn`weT^djcg6_S2XQhVbL~|;l|5*pL~@`)bm+E zU#nJSn>q*mqG6p1Ra)G%H$!1dtp(Fd2V<6IpH7G2Iyj7J8+WOGo^QW6>4&$&g)J+_ zACdartrE&Kh6znFbTBC}`{l0+6$Fwk)vw>0IUjk8X;r>T`!&_AxFIlwQQ>CI20^5l zeWHqI=s2`8I-fU*Aaz2F9@?r3qQ8}Vy7q>cE*xK`WG>*t!Fz?tDf6J$Tu1jtgN);{ zUB;u9=b7r(bq|doZxdvwuFgCk@8mc6jux%udk%d9p}Td4 z-JzAvzE8{alcOBQ(l0-`J#K*N3Rkaon>%{V_`bvF*7wG?C!0QHsMdx{)w)XsjpLak zkTSPUOSyk6JXQ{e+i!cXf`uWJ`r^T*m#J~k$>1rbraLCzugUifWw+hfVMAeqm%4wq zr};Ax)DJ3{JaV(az@aC>mjIaFvHrO#=GyhA_yB;`0g;3OhlB1%smG!in_)8 zP3mxAYehSBpaUKATu7>qy0GV0q{dIiWgK6+!I^ik^GTA+_^k}D(X9F-=N)rG;?2N% zEbrDR={gpP_M!yC?)%Mbtucj!iLyso*{cOCT=~>O@lC8%t@7@V2tFDUt8K;z;d=&Z z-`R08_YL)Cy3jJ9IfhcMA3Pwf_S!QkFEEuNng7t?t<9vAu{d6I?@*;g`21Z#4+2gL zwW_Z!v%Yy3x-BWnU02eFIOjM-Cp6y-G+nqAvE_#+ap=oBmDHutD35(kKP4!xG8Sh` z??o((`CM$ihSJQvni~FFoEgDB)>0hWKrivW_pm70uTLUMgmi#vDD6q?ES^OPW2 z^5UO3B!j!{tlQ|0Un?${?t)zaytXs@={p!!lXVC8I|(tGCU)4B^*wb_lU;(1`IhPC z>35&UziUzCk@f+O8l%Tj`H|#VAEp|icAgCV$206;ugD>wof}S=Zi8d+%3PTGa$6_! zWksU2xU%hSovF|e=O3OkYqO;rs64u@xb6|diQ_JUauzbCZ#CE%?bm5cJIu@R{O}1H zMh4AT3h(JWfi@uLD0@&T3MdaFmlqJpQ6O_P>rPYEa2A(Cem>M zyDoI_l>=`R`xu*l&FWjy;lU2QfsEL8%L^C$F=|fVZ(f2z)dNq@df@B37Q|z;8X9l_wKDB@Z9H1(#T{(3 z&nmMPScjM=+;VP?h9H9P^cN23gicI}Mmm}6l&r5@=%j5sC_8RmtK}4CVHm#3)jgOW zcXv&;@*#+N&e3WKxPcM+!aSjp`S^66A9 zmE!qktq1S&co!)+**|cADhQSQHp$gLz_IrDZHR zl6V`tec6pjb**IavFyA2pfHT$liyi}_6lf6Z|{^mQL6HtAjt4ZoD#z(rD zq|Mj>IGzTCdFJzz3iX(uy&uDigP@a?CG_)3>%Jb-R$R}BzuzZd&)tRVT0LRRkUv+y zwBGlT|6AY2hiit_j9iTC(xO6#shfe%c~q^;sHl$!%J`>=f(5f|Cbjdryvkp4-2sQI z2ENW+P4=eNd_RyJJ=z$sBpmopBnnVIQpVFfiRtP270qYmP3+MRS z5-4F*N)3xr#vMadfB(z#gfibh$KyrZm5ueFX8GdsI`Vf@r)9#rMVkT z=>Dn`yFk)aif7!j*Ma5*Q+BU)B3Yc&EXm04XHFP5F>Vb9t z)x{8hN*%+baF0j3yR%t*D;`W3aN8w=)WOwj+mAj$>l0|>1HZoe8;zqwK6NVH7sA<`bVhKfYGvqS>*9R z{&o})^x)dmy#Z#P$;d(cZXt2%94-`hrGp0h%4TP1OGVc~0DRS)5_#X8TuSlRw37n? zi!;qL^Zn^fRJPCS((gSUXr9curC(=%&Wzoo_L_OQHX!m@+k+(9#}ARzg7XOy9=28R z@pR6zNV|L)v&aZ@J9l4v(y3*skgjqvGEtIACd1W^ZqPu;aj{g!3@osL5jV^+c|4*~ zCX5`k5`c?!#YhoEsg4(yB<@)RIiv6pwe9Vc80Sro6B><6-c$ zo#B*ol}{fZCz<=!^;o}_L-!`|#_0k$T=#1EGbg8EUgTUX(;_CF)XCT~UbasAPoJMn z3%@(?HSYTK7VYQ;eX+2hR$#chj5VV&2mg^$qdZ6a?go$;Lp_<2rDI@tUu?ShI@3yKM*^7;Hf@W+S{=@>x62KAH|mjEJay1So-_+SQAqarMvi_ zMrQw+$+{XYJml2gvgR(*>{}ewimF~x0E3_L>it+>U*AX9<^NfZ8v#eOOs_Wt<4|pY zNK@z3{CwPEH$hO>=Kw;s5UleYfE1G(kEI^}=*!5u%VQ5=ON(fa38wE}B-qkOoLA4y zO{fw>y9w#&2UVVWbm91z$J}z@m=94Frx?#IDJGQwCw3TJErT2!j*lc?PV>|?!=gHd z=Hs)vUuSkzR#mB#mzP6%So&uBbM!A#>;V{#NZq6L#xH;Yw+lpKxb|C5d=Q)iaz6bc zj<>>S5vvYoVi;}QPsUxUT2gh9I<|gY1#k|bk=R?8b-=LYd{VJbXYoMIIxk+dXtc0q zmTJL85zZVykP!l$27MFUwetri7BE@zo_2umMv+Ak8$6|N;UkIi?b~c6Z{JR3sJ$TG zT@G1*{#szFQsHIC{F<7R$Bhk}K|WlFuTQ z;`~l{qrcziV|tK=EDR`s2D^XL7o~zf`?ZTd`N9ujhaAM=xi%E0yxK5x*GhZ%;P?Ug zcbW_(f6~1#5ozS%s8!02EZ4EHM3S$JF|Lgpa}UE_@>+W~*Il9fgjO>HpCSJ@{sKS3 zwBxrD8fp>yMqxzW2OZ$qN_2Eb6b0DHg2w)@#=l%J@XU$@W+<1m`YoO+8dw*r7; zl3`WaqhNfVtQnu(BV+^L8R^v)EK-DHCVy1Dr!-m=V&4AvY6YP^j*Cb|@bSrnQRHct zu)t=!TU*9e-gex%*w;YMNrz?|NoIp6Zo~xf)j`S`x^gS8hi6Y+^ADr#kv(_F$joTn zoTgPPb#7ZctROVr^uqqZy1TbLuk^JDu7p$vwxdg0i&f#RK{_-PG|#Oy3rVuXc%vUv z_3T0GI{>mrw1gy@ZnQYcp*+wx5U`x}$U=5}D#me*OiBbb{a`jLhv`SXt9pv72$?Q- z*WHH_+}_7x{9Qk#mjqXA=;N(B4HxQ~d$qLA2I@#gpJ}7Oh&?rYTGh4mTBX#F^|Q9L z)iwG(R>g2$y?SNraDuq;5EfyDjKa_3Xry7kkJVHfMT)C!#O7Qt>G4yX;fpyE?#`}j z9&HW69>s;16BO{&sU%jgEh@e8J9>r8?@1J~i6{f?&_qbsEwCL*xaw49XpSK)%B2tX?>Wm5Y@vcU`sy&c6^ zT1V^&FAAl2Fkf6j3>u{2>QBAA9+dk$TjI)>h532Haj%0bLh#TFk^5j_?4wMTZB|qH z0n8e%)Sd6<+ckVyaTffs!eh`3`_C9*z*zoj>vM4ocjEHqdPZv^5UHb8+tz#^okzXAIfrA!77zPRm zcU8jaGTIU)Md&4*QzRn3e{LGcQUHfa0B$T1fU+EcgaQ8YY%mq))BD;0-*e)RnD>Zu z!rL?%_SKdS~1-TNz9!|<+YbFV%2_Bf%mzPTHKq1|yU zoYttHUT?}`kKy^aIp^EQG~HOK{i(B z?QJ0gYYB=}9>wL7Xf4~nDTVt)vkI&?x#);G`cWjxj(>`yLum7srVku{wXyR1Ip3T1 zbE+wH3ml2>PnkGNMm9@EMiw9isEN5>y@wLOxa)SONTLKx$^|{PmoIv+O<)6x*;|V) z@t^_k0RI?UbWg?r$Xf^<8JF6_0hsp7{MKU#cF0x~1>o|9l%Dh(OvR%=axweGN|I6e zv(jn#%7V4WH(>b>AT>}C8A+Zj<(6v@tYiP&mfhkab)bTRLJA?=Znh)dILwD9dA}Wj zK%_vAHa+(@d&8LI&2DC>onQD?;bIxGEKrAft%8tr92Rv1 z_V<>YHK!L?~IbqSsp`SfqML?u=gAB?c6yc^LE za+H*mjC6F)pw*%cHjS<4fv1VG9v(WN0;1gW2xm3zGZ)g8LsGZxW6R_`uH~c zfeb!y4_okg{}7{<5vMxRJV43($nmZ7luiO|&nocXuH@*M1q<^75;BGhp!B4nIq<1= zPPlEeq0lAZJmay#lm4E#K|BdS+%^WXim4Er%04jl-(KxkD24?Mal=C*e}UK#Lxj&F z|0FMhf}2;?b3eyn`Z$|ERuU#Xb1;C50Ssl5)~O8j&!vc;JZu*XFpuwCOS214wFEQH zEYxQo>@Fb;AZ;^tz_-6{f(szD(Qo4d5Q2RS1vo%%@*MW5egG~k6!7FIku;*>bi{u* z@=1%Q;{YI84%VRvGF=~I^1nIQ1pt#oU2?M1W>tJ1pWqu$^%EfP0!)JWKS5hziqr3M zx?sdL0IU-k5Wsf(Rcm{Awtjz1hwP)#uAZh&bXPDoy}GA2bz$Wg1SRTwu1_EB+ zl1RrLoxjhWa{xYiaWD#|aHt=vQFkBV4yg*{dhGI7!0dR zN;SX7*(rn5&*KDmMfZrNq~KvpI&~MU-YJo73PKf&DUbiVnrI+Cewk-Xpmd6#C{u

TG{4$yL80%UIJZ~iRL$_RCwPGxXT8N3cP!7#>c z7HQy!@?g?~V1u7E4>$PM|A z?t++MgScf}M|c0l7inIYIfD6*bI|jJ;0&{+#$%~o^ZSpq=zd431x#u*VmE~6qSY`k zmi}rCi(XKw@#go82KGd(7i@p>{AD3?1se0sAAOCS0%$F2``G{WfNBg#o7vV5OFHQk z>5SO+@uvz%2K{N{B#!vo)D2FEEG7dB1CSOu@I9sLUyEXud|)Rr8`U~=#^ZOjSuxHP z-P`tAA*`UFdNy=QJQ1f{b?8jp!(P@LL`{q~{JYRvFC02`!~lR9lC zCcPykK;OowKKP!)_;(ed%a&pXVuN?kcDYOrn!&;?@Qg{7EE}23!RtV_o_e z4GaEJB#3VG<)@#WPo8a4^07aKw5^{5-hOXyd0qokW+po%A7i^JE4m({NR&r0)dtpg zF<4u_dx#w9rW;lqGpF(c*zXa%>>Xep0Pg}P@&r5kWQ6`6aG6`d1->T@hCEDJbP;hLuGy+ok1 zj(N`Bcq`A{t#JQ-BuxU7r|sf>jkFy1=^CU#&fd+-AvT^24D0mbHPFQX?Q5`l#USJr|SC|W%IKiJ+Z AOaK4? literal 0 HcmV?d00001 From 93357e3bd8e6229fb45814ec0907630ea262b505 Mon Sep 17 00:00:00 2001 From: Matthias Hartmann Date: Sat, 1 Aug 2026 02:51:15 +0200 Subject: [PATCH 2/2] review: throughline --- docs/02-General/04-flavors-of-metalstack.md | 8 +- .../03-Deployment/01_guide.md | 65 ++++-- .../02_bootstrap-infrastructure.md | 61 ++--- .../03-Deployment/03_control-plane.mdx | 98 ++++++-- .../03-Deployment/04_partition.md | 215 ++++++++++++------ .../03-Deployment/05_gardener.md | 137 +++++------ .../03-Deployment/06_cluster-api.md | 69 +++--- .../03-Deployment/07-gpu-workers.md | 36 ++- .../03-Deployment/08_offline-resilience.md | 46 ++-- docs/05-Concepts/04-Kubernetes/01-kclm.md | 64 ++---- docs/05-Concepts/04-Kubernetes/02-gardener.md | 84 ++++--- .../04-Kubernetes/03-cluster-api.md | 79 +++---- .../04-cloud-controller-manager.md | 7 - .../05-firewall-controller-manager.md | 27 +-- docs/05-Concepts/04-Kubernetes/07-storage.md | 4 +- .../Security/04-communication-matrix.md | 6 +- 16 files changed, 571 insertions(+), 435 deletions(-) diff --git a/docs/02-General/04-flavors-of-metalstack.md b/docs/02-General/04-flavors-of-metalstack.md index 18c0671f..de0bd7d5 100644 --- a/docs/02-General/04-flavors-of-metalstack.md +++ b/docs/02-General/04-flavors-of-metalstack.md @@ -22,12 +22,12 @@ If you want more features, keep reading. [Gardener](https://gardener.cloud/) is an open-source managed Kubernetes service. It provides a good "batteries-included" developer experience and should be your first choice for a Kubernetes-as-a-service solution. -Gardener is vendor agnostic and can be used with a wide selection of infrastructure providers. One big advantage are its containerized control planes. These allow for control planes to not require three machines for each managed cluster, called `Shoot`. This makes operating many smaller clusters more economical, compared to bare-metal control planes. +Gardener is vendor agnostic and can be used with a wide selection of infrastructure providers. One big advantage are its containerized control planes. These allow for control planes to not require three machines for each managed cluster, called `Shoot`. This makes operating many smaller clusters more economical, compared to bare-metal control planes. -We provide support to run Gardener on metal-stack via [Gardener extensions](../05-Concepts/04-Kubernetes/01-gardener.md). This integration is production-hardened, well documented, used by many organizations in production and build on top of the open-source project [Gardener](https://gardener.cloud/). +We provide support to run Gardener on metal-stack via [Gardener extensions](../05-Concepts/04-Kubernetes/02-gardener.md). This integration is production-hardened, well documented, used by many organizations in production and build on top of the open-source project [Gardener](https://gardener.cloud/). ## Cluster API -Our [Cluster API integration](../05-Concepts/04-Kubernetes/02-cluster-api.md) is a more verbose approach to provide Kubernetes clusters with metal-stack. Our implementation is still in early development. It is based on the [Cluster API](https://cluster-api.sigs.k8s.io/) project. +Our [Cluster API integration](../05-Concepts/04-Kubernetes/03-cluster-api.md) is a more verbose approach to provide Kubernetes clusters with metal-stack. Our implementation is still in early development. It is based on the [Cluster API](https://cluster-api.sigs.k8s.io/) project. -Configuring Cluster API is very verbose. It requires additional tooling to provide a developer experience. Cluster API will give you building blocks to build a Kubernetes-as-a-service platform on top of it, but no more. We do not recommend you use Cluster API, unless you already have a large platform engineering team, that is very experienced in bare-metal K8s operations and they agree that your specific requirements cannot be modelled with Gardener. In any other case, you will have more success with Gardener. \ No newline at end of file +Configuring Cluster API is very verbose. It requires additional tooling to provide a developer experience. Cluster API will give you building blocks to build a Kubernetes-as-a-service platform on top of it, but no more. We do not recommend you use Cluster API, unless you already have a large platform engineering team, that is very experienced in bare-metal K8s operations and they agree that your specific requirements cannot be modelled with Gardener. In any other case, you will have more success with Gardener. diff --git a/docs/04-For Operators/03-Deployment/01_guide.md b/docs/04-For Operators/03-Deployment/01_guide.md index 35ab72da..94710e3a 100644 --- a/docs/04-For Operators/03-Deployment/01_guide.md +++ b/docs/04-For Operators/03-Deployment/01_guide.md @@ -8,61 +8,80 @@ sidebar_position: 1 metal-stack is a Metal-as-a-Service (MaaS) platform that turns on-premises bare-metal servers into an elastic, self-managed cloud infrastructure. At its core, metal-stack consists of two components: -- **Control Plane** — The central management layer (APIs, databases, scheduling) +- **Control Plane** — The central management layer (APIs, databases, messaging, scheduling) - **Partitions** — The data center infrastructure (servers, leaf switches, BMCs) These two components alone provide a fully functional **Bare-Metal as a Service (MaaS)** platform. You can allocate machines, manage networks, configure firewalls, and operate servers via REST/gRPC APIs and the `metalctl` CLI — without any Kubernetes cluster lifecycle management. ## Kubernetes Cluster Lifecycle Management -If you need to provision **Kubernetes clusters** on top of your bare-metal infrastructure, metal-stack integrates with KCLM solutions like [Gardener](../../05-Concepts/04-Kubernetes/01-gardener.md) or [Cluster-API](../../05-Concepts/04-Kubernetes/02-cluster-api.md). These are **optional layers** built on top of the MaaS foundation — not prerequisites. +If you need to provision **Kubernetes clusters** on top of your bare-metal infrastructure, metal-stack integrates with Kubernetes Cluster Lifecycle Management (KCLM) solutions like [Gardener](../../05-Concepts/04-Kubernetes/02-gardener.md) or [Cluster API](../../05-Concepts/04-Kubernetes/03-cluster-api.md). These are **optional layers** built on top of the MaaS foundation — not prerequisites. :::tip -You can use metal-stack as a pure MaaS platform without any KCLM integration. The KCLM layer is only needed if you want to automate Kubernetes cluster provisioning on your bare-metal infrastructure. +You can use metal-stack as a pure MaaS platform without any KCLM integration. The KCLM layer is only needed if you want to automate Kubernetes cluster provisioning on your bare-metal infrastructure. See the [KCLM concepts overview](../../05-Concepts/04-Kubernetes/01-kclm.md) for a comparison of the available options. ::: -### Deployment Approach +## Deployment Approach We bootstrap the [metal control-plane](../../05-Concepts/01-architecture.mdx#metal-control-plane) as well as our [partitions](../../05-Concepts/01-architecture.mdx#partitions) with [Ansible](https://www.ansible.com/) through CI. The deployment has multiple phases, resulting in a Git repository containing Ansible playbooks, inventory files and CI/CD workflows. -1. **[Control Plane](./03_control-plane.mdx)** — Deploy the metal-stack API, databases, and ingress on a bootstrap Kubernetes cluster. This alone gives you a functional MaaS platform. -2. **[Partition](./04_partition.md)** — Configure the network fabric (leaf switches, management spine, DHCP, PXE) and connect your bare-metal servers to the control plane. -3. **[KCLM with Gardener](./05_gardener.md)** — Add Gardener to manage Kubernetes cluster lifecycle on your bare-metal infrastructure. +1. **[Bootstrap Infrastructure](./02_bootstrap-infrastructure.md)** — Decide where the initial Kubernetes cluster(s) that host metal-stack and, optionally, your KCLM run. This is a planning phase, not an Ansible deployment. +2. **[Control Plane](./03_control-plane.mdx)** — Deploy the metal-stack APIs, databases and ingress onto the initial Kubernetes cluster. This alone gives you a functional MaaS platform. +3. **[Partition](./04_partition.md)** — Bootstrap the out-of-band network, configure the network fabric (leaf, spine and exit switches, DHCP, PXE) and connect your bare-metal servers to the control plane. +4. **[KCLM with Gardener](./05_gardener.md)** — Add Gardener to manage the Kubernetes cluster lifecycle on your bare-metal infrastructure. Alternatively, see [Cluster API](./06_cluster-api.md). + +Two cross-cutting topics complete the picture once the platform runs: [GPU Workers](./07-gpu-workers.md) and [Offline Resilience](./08_offline-resilience.md). By the end of this guide, your deployment repository will look something like this: ```text my-metal-stack-deployment/ +├── ansible.cfg ├── deploy_metal_control_plane.yaml # Control plane -├── deploy_partition.yaml # Partition -├── deploy_gardener.yaml # KCLM -├── inventory/ -│ ├── inventory.yaml -│ ├── group_vars/ # Variables per host group -│ └── host_vars/ # Variables per host -├── roles/ # Custom Ansible roles -├── files/ +├── deploy_gardener.yaml # KCLM (optional) +├── deploy_mgmt_servers.yaml # Partition: management servers +├── deploy_mgmt_switches.yaml # Partition: management switches +├── deploy_spines_exits.yaml # Partition: spine and exit switches +├── deploy_leaves.yaml # Partition: leaf switches +├── inventories/ +│ ├── control-plane.yaml # inventory for the Kubernetes deployments +│ ├── partition.yaml # inventory for the partition hosts +│ ├── group_vars/ # variables per host group +│ └── host_vars/ # variables per host +├── roles/ # custom Ansible roles +├── files/ # certificates, keys, static assets └── .github/workflows/ # CI/CD pipelines ``` The different phases of the deployment guide show which files to add and how they fit into this structure. :::tip -You can use the [mini-lab](https://github.com/metal-stack/mini-lab) as a template project for your own deployment. It uses the same approach as described in this guide. +The [mini-lab](https://github.com/metal-stack/mini-lab) is the fastest way to see a complete, runnable parametrization of the very same Ansible roles. It brings up a virtualized metal-stack — including an optional Gardener or Kamaji landscape — on a single machine. + +Treat it as a **reference for how the roles are wired together, not as a production blueprint.** The mini-lab intentionally cuts corners that are unacceptable in production: it runs on a single [kind](https://kind.sigs.k8s.io/) cluster, ships unencrypted secrets, fakes the `kube-system/shoot-info` ConfigMap and patches load balancer statuses by hand. Wherever this guide references the mini-lab, it does so to point at a concrete example — never as a recommendation. ::: -We recommend using the same Ansible roles that we use to deploy metal-stack in our own environments. They are available in the [metal-roles](https://github.com/metal-stack/metal-roles) repository. +We recommend using the same Ansible roles that we use to deploy metal-stack in our own environments. They are available in the [metal-roles](https://github.com/metal-stack/metal-roles) repository, split into [`control-plane`](https://github.com/metal-stack/metal-roles/tree/master/control-plane), [`partition`](https://github.com/metal-stack/metal-roles/tree/master/partition) and [`common`](https://github.com/metal-stack/metal-roles/tree/master/common) roles. Every role carries a `README.md` that documents its variables — these READMEs are the authoritative variable reference, while this guide focuses on the red line through the deployment. -To simplify dependency management, we provide a dedicated [deployment base image](https://github.com/metal-stack/metal-deployment-base) on GitHub Container Registry. This Docker image bundles all required tools and libraries, keeping your deployment environment lean and easy to maintain. +To simplify dependency management, we provide a dedicated [deployment base image](https://github.com/metal-stack/metal-deployment-base) on GitHub Container Registry. This container image bundles all required tools and libraries (Ansible, `kubectl`, `helm`, `cosign`, `metalctl`, …), keeping your deployment environment lean, reproducible and easy to maintain — locally and in CI. -The remainder of this guide assumes you are using our Ansible roles and the deployment base image. Make sure you have [Docker](https://docs.docker.com/get-started/get-docker/) installed. Every deployment environment is unique, so this guide describes how we set up metal-stack in our own infrastructure. You will need to adapt the steps to match your specific requirements. +## What You Need to Know -:::warning -You will likely need to learn to write Ansible playbooks if you want to be able to deploy the metal-stack as presented in this documentation. However, even when starting without any knowledge of Ansible it should be possible to follow these docs. In case you need further explanations regarding Ansible please refer to [docs.ansible.com](https://docs.ansible.com/). -::: +This guide assumes you are comfortable with the following. It does not teach them: + +- **Kubernetes** — You will operate metal-stack on Kubernetes and troubleshoot it with `kubectl`. +- **Ansible** — All deployments are Ansible playbooks and roles. Refer to [docs.ansible.com](https://docs.ansible.com/) if you are new to it; you do not need to be an expert to follow along, but you will write playbooks. +- **Data center networking** — BGP, VRFs, EVPN/VXLAN and DHCP/PXE. The [networking concepts](../../05-Concepts/03-Network/01-theory.md) chapter explains how metal-stack uses them. +- **Your KCLM of choice** — If you plan to deploy [Gardener](./05_gardener.md) or [Cluster API](./06_cluster-api.md), you should already be familiar with its own concepts and terminology. + +Every deployment environment is unique. This guide describes how we set up metal-stack in our own infrastructure — you will need to adapt the steps to match your hardware, network and compliance requirements. Make sure you have [Docker](https://docs.docker.com/get-started/get-docker/) (or a compatible container runtime) installed to run the deployment base image. :::info -If you do not want to use Ansible for deployment, you need to come up with a deployment mechanism by yourself. You can still learn from our [metal-roles](https://github.com/metal-stack/metal-roles) repository and [Helm charts](https://github.com/metal-stack/helm-charts/) — but be aware that the Ansible roles tie everything together (variable management, dependency ordering, environment-specific configurations), so building an equivalent from scratch will require significant effort. +If you do not want to use Ansible for deployment, you need to come up with a deployment mechanism by yourself. You can still learn from our [metal-roles](https://github.com/metal-stack/metal-roles) repository and [Helm charts](https://github.com/metal-stack/helm-charts/) — but be aware that the Ansible roles tie everything together (variable management, dependency ordering, environment-specific configuration), so building an equivalent from scratch will require significant effort. +::: + +:::tip +Stuck? The [troubleshooting guide](../06-troubleshoot.md) collects the issues we run into most often, and the [community page](/community) tells you how to reach us. Feedback on this guide is explicitly welcome — it is how we make the deployment easier for the next operator. ::: diff --git a/docs/04-For Operators/03-Deployment/02_bootstrap-infrastructure.md b/docs/04-For Operators/03-Deployment/02_bootstrap-infrastructure.md index 74f26ad7..2f81293a 100644 --- a/docs/04-For Operators/03-Deployment/02_bootstrap-infrastructure.md +++ b/docs/04-For Operators/03-Deployment/02_bootstrap-infrastructure.md @@ -21,56 +21,57 @@ If you only need **Bare-Metal as a Service** without KCLM, you need at least **o Your control plane Kubernetes cluster can run anywhere — on a hyperscaler, in your own data center, or on [metalstack.cloud](https://metalstack.cloud). A managed cluster removes the operational burden of running Kubernetes yourself and can even strengthen fail-safe operation. Learn more about the [rationale for this approach](../../05-Concepts/01-architecture.mdx#target-deployment-platforms) and find concrete hosting suggestions below. ::: -## KCLM Solutions +## Requirements for the Initial Cluster -metal-stack supports three Kubernetes Cluster Lifecycle Management solutions, each with different maturity levels and capabilities. -See the [Kubernetes Concepts Section](../../05-Concepts/04-Kubernetes/01-gardener.md) for a detailed comparison. +Whichever hosting option you pick, the initial cluster has to satisfy three properties: -### Gardener +- **Reachable from the partitions** — Every partition must reach the metal-api over HTTPS as well as gRPC (`50051`), NSQ (`4150`) and metal-console (`5222`). These layer-4 services need explicit exposure; see [Control Plane](./03_control-plane.mdx). +- **Persistent storage** — The control plane databases hold your masterdata and IP address management. Provide a storage class with persistent volumes and enable the [backup-restore-sidecar](./03_control-plane.mdx#setting-up-the-backup-restore-sidecar). +- **Independent lifecycle** — The initial cluster must not run on the machines that metal-stack manages, otherwise you create a circular dependency you cannot recover from. -[Gardener](../../05-Concepts/04-Kubernetes/01-gardener.md) is the **recommended** path for Kubernetes cluster lifecycle management. It is battle-tested in production for over seven years at financial-sector customers and bundles several day-2 capabilities natively (DNS, backup, audit). Gardener manages entire clusters as Kubernetes-native resources with a strong separation between platform operators and end-users. +## Choosing a KCLM Solution -:::tip -Gardener is the recommended solution for production environments. See the [Gardener concept section](../../05-Concepts/04-Kubernetes/01-gardener.md) for terminology and architecture details. -::: - -### Cluster-API - -[Cluster-API](../../05-Concepts/04-Kubernetes/02-cluster-api.md) is a CNCF project maintained by a Kubernetes SIG that provides declarative cluster management through a management cluster. The metal-stack provider (CAPMS) is **under development** and not yet production-ready. - -:::warning -Cluster-API with metal-stack is in development and not advised for production use. Please use Gardener for production workloads. -::: +metal-stack integrates with two Kubernetes Cluster Lifecycle Management solutions — Gardener and Cluster API (the latter optionally combined with Kamaji as control-plane provider). They differ substantially in maturity, day-2 capabilities and operational model. -#### Kamaji +The [KCLM concepts chapter](../../05-Concepts/04-Kubernetes/01-kclm.md#two-approaches-one-infrastructure) contains the full comparison and decision matrix. The short version: -[Kamaji](../../05-Concepts/04-Kubernetes/02-cluster-api.md#kamaji) allows a similar control plane hosting model as Gardener, where the control plane runs on dedicated infrastructure separate from worker nodes. -Kamaji therefore uses ClusterAPI to support different infrastructure- and control-plane providers. -However, Kamaji integrations with metal-stack **have not been evaluated in production-grade scenarios** by metal-stack. +| Solution | Status | Guide | +| ------------------------------------------------------------------------------------------------- | ---------------------------------------------------------- | ------------------------------------------------------------------------------ | +| [Gardener](../../05-Concepts/04-Kubernetes/02-gardener.md) | **Recommended**, production-proven for 7+ years | [Gardener deployment](./05_gardener.md) | +| [Cluster API (CAPMS)](../../05-Concepts/04-Kubernetes/03-cluster-api.md) | Beta, under active development | [Cluster API deployment](./06_cluster-api.md) | +| [Cluster API + Kamaji](../../05-Concepts/04-Kubernetes/03-cluster-api.md#kamaji-with-metal-stack) | Not evaluated in production-grade scenarios by metal-stack | [Cluster API deployment](./06_cluster-api.md#kamaji-as-control-plane-provider) | :::warning -Kamaji with metal-stack is in development and not advised for production use. Please use Gardener for production workloads. +Only Gardener is recommended for production workloads today. Cluster API with metal-stack is in beta, and the Kamaji integration has not been evaluated in production-grade scenarios by metal-stack. Your KCLM choice determines the required cluster topology described below, so make it before you deploy anything. ::: ## Deployment Options -There are three supported approaches for hosting the initial cluster: +There are three approaches for hosting the initial cluster. All three are viable; they trade operational simplicity against autonomy. + +| | Option 1: Shared | Option 2: Dedicated | Option 3: Autonomous | +| ------------------- | --------------------------- | ------------------------------- | ------------------------ | +| Clusters | 1 | 2 | 2 (nested) | +| Operational effort | Lowest | Low | Highest | +| Failure isolation | Weak | Good | Good | +| Digital sovereignty | Depends on provider | Depends on provider | Full | +| Recommended for | Evaluation, small platforms | **Most production deployments** | Sovereignty requirements | ### Option 1: Shared Initial Cluster -It is possible to use a **single initial cluster** for both metal-stack and the KCLM solution. This approach is technically feasible but **not recommended** for production environments. Sharing a single cluster mixes platform infrastructure with lifecycle management, which can complicate operational boundaries and failure isolation. +It is possible to use a **single initial cluster** for both metal-stack and the KCLM solution. This approach is technically feasible but **not recommended** for production environments. Sharing a single cluster mixes platform infrastructure with lifecycle management, which complicates operational boundaries and failure isolation — a Gardener upgrade and a metal-api upgrade then share one blast radius. ### Option 2: Dedicated Initial Clusters We recommend using **dedicated initial clusters** for metal-stack and the KCLM solution — one cluster for the metal-stack control plane and a separate cluster for the KCLM. -This approach provides clearer operational boundaries, better isolation and simplified failure boundaries. +This approach provides clearer operational boundaries, better isolation and simplified failure boundaries. In Gardener terminology the second cluster becomes the _runtime cluster_ (and typically the first Seed, the "soil"); see the [Gardener deployment guide](./05_gardener.md#architecture-overview). ### Option 3: Autonomous Control Plane For self-hosted deployments, metal-stack can be set up with an [Autonomous Control Plane](/community/MEP-18-autonomous-control-plane) cluster. This approach is the best choice for organizations that require full digital sovereignty and autonomy over their entire infrastructure stack. -The autonomous control-plane cluster serves as a minimal control plane whose sole purpose is to host the production control plane cluster (the "Matryoshka principle"). +The autonomous control-plane cluster serves as a minimal cluster whose sole purpose is to host the production control plane cluster (the "Matryoshka principle"). This brings several advantages like failure isolation, separate operational responsibility, minimal resource requirements and full control and ownership. The only requirement from metal-stack is that your partitions can establish network connections to the metal control-plane. @@ -82,13 +83,17 @@ The only requirement from metal-stack is that your partitions can establish netw For the shared and dedicated cluster approaches, the initial cluster can be hosted anywhere — a hyperscaler, metalstack.cloud, or any other managed Kubernetes provider. Some common options: - **metalstack.cloud** — A Kubernetes cluster can be created via [UI](https://metalstack.cloud/de/documentation/UserManual#creating-a-cluster), CLI, or [Terraform](https://github.com/metal-stack-cloud/terraform-provider-metal). -- **GCP/GKE** — A GCP account is required. The Ansible [gcp-auth role](https://github.com/metal-stack/ansible-common/tree/master/roles/gcp-auth) can be used for authentication, and the [gcp-create role](https://github.com/metal-stack/ansible-common/tree/master/roles/gcp-create) for creating a GKE cluster. - - Suggested defaults: `gcp_machine_type`: e2-standard-8, `gcp_autoscaling_min_nodes`: 1, `gcp_autoscaling_max_nodes`: 3 +- **GCP/GKE** — A GCP account is required. The Ansible [gcp-auth role](https://github.com/metal-stack/ansible-common/tree/master/roles/gcp-auth) can be used for authentication, and the [gcp-create role](https://github.com/metal-stack/ansible-common/tree/master/roles/gcp-create) for creating a GKE cluster. Use `metal_control_plane_host_provider: gcp` in this case. + - Suggested defaults: `gcp_machine_type`: `e2-standard-8`, `gcp_autoscaling_min_nodes`: 1, `gcp_autoscaling_max_nodes`: 3 + +:::info +`metal_control_plane_host_provider` is only evaluated by the Gardener roles, which use it to discover the runtime cluster's node, pod and service CIDRs. It is asserted to be either `metal` or `gcp`. Any other Kubernetes distribution works as well, but you then have to provide the `kube-system/shoot-info` ConfigMap yourself — see the [Gardener prerequisites](./05_gardener.md#prerequisites). +::: ### For Option 3: Autonomous Control Plane with k3s For the autonomous control plane approach, [MEP-18](/community/MEP-18-autonomous-control-plane) proposes using [k3s](https://k3s.io/) as the initial cluster. This is because KCLM solutions are not yet able to create an initial cluster themselves (though this may change with implementations like [GEP-28](https://github.com/gardener/enhancements/blob/main/geps/0028-self-hosted-shoot-clusters/README.md) for Gardener). -The k3s nodes can be either bare metal machines or virtual machines. For a minimal setup, a single node with 8–16 cores, 64GB RAM, and two NVMe drives of 1TB is a good starting point. For high availability, a clustered k3s configuration across multiple nodes is recommended, with ETCD replication and backup-restore mechanisms configured for metal-stack and KCLM components. +The k3s nodes can be either bare metal machines or virtual machines. For a minimal setup, a single node with 8–16 cores, 64 GB RAM, and two NVMe drives of 1 TB is a good starting point. For high availability, a clustered k3s configuration across multiple nodes is recommended, with etcd replication and backup-restore mechanisms configured for metal-stack and KCLM components. See the [Autonomous Control Plane](/community/MEP-18-autonomous-control-plane) proposal for detailed architecture, failure scenarios, and implementation guidance. diff --git a/docs/04-For Operators/03-Deployment/03_control-plane.mdx b/docs/04-For Operators/03-Deployment/03_control-plane.mdx index b2d702fd..81b542a2 100644 --- a/docs/04-For Operators/03-Deployment/03_control-plane.mdx +++ b/docs/04-For Operators/03-Deployment/03_control-plane.mdx @@ -14,7 +14,19 @@ While it is theoretically possible to deploy metal-stack without Kubernetes (see The control plane deployment described here requires an initial Kubernetes cluster as bootstrap infrastructure, which is described in [Bootstrap Infrastructure](./02_bootstrap-infrastructure.md). We use the [metal-stack control-plane Ansible roles](https://github.com/metal-stack/metal-roles/tree/master/control-plane) to deploy all control-plane components. -Before you try to deploy the control-plane, you should have had a look at the [architecture](../../05-Concepts/01-architecture.mdx) concepts section. +Before you try to deploy the control-plane, you should have had a look at the [architecture](../../05-Concepts/01-architecture.mdx) concepts section, which explains what each of the components you are about to deploy actually does. + +:::info Layer-4 exposure +Besides the HTTP(S) ingress, the control plane also has to expose three **layer-4** services so that a [partition](./04_partition.md) can reach them. Plan this together with your ingress controller — for example through [TCP/UDP service exposure of ingress-nginx](https://kubernetes.github.io/ingress-nginx/user-guide/exposing-tcp-udp-services/) or a dedicated `Service` of type `LoadBalancer`: + +| Port | Protocol | Service | Purpose | +| ----- | -------- | --------------- | -------------------------------- | +| 4150 | TCP | `nsqd` | NSQ daemon (TLS) | +| 5222 | TCP | `metal-console` | Machine console forwarding (SSH) | +| 50051 | TCP | `metal-api` | metal-api gRPC API (protobuf) | + +The [mini-lab](https://github.com/metal-stack/mini-lab) shows a minimal variant of this exposure. +::: Create the deployment folder and enter it — this is the root of your deployment repository as outlined in the [Deployment Guide](./01_guide.md): @@ -47,8 +59,7 @@ By the end of this section, your repository will contain the following files and │      │   └── release_vector.yaml │      └── control-plane │        ├── common.yaml -│         └── metal.yml -├── generate_role_requirements.yaml +│         └── metal.yaml └── roles └── ingress-controller └── tasks @@ -112,7 +123,19 @@ control-plane: We do this since we are deploying to Kubernetes and do not need to SSH-connect to any hosts for the deployment (which is what Ansible typically does). This inventory is also necessary to pick up the variables inside `inventories/group_vars/control-plane` during the deployment. -Most of the properties in there are up to taste, but make sure you enable the [Jinja2 native environment](https://jinja.palletsprojects.com/en/2.11.x/nativetypes/) as this is needed for some of our roles in certain cases. +## Ansible Configuration + +Most Ansible settings are up to taste, but you **must** enable the [Jinja2 native environment](https://jinja.palletsprojects.com/en/stable/nativetypes/). Several of our roles rely on Jinja2 returning native Python types (lists, dicts, booleans) instead of strings. A minimal `ansible.cfg` looks like this: + +```ini +[defaults] +jinja2_native = True +inventory = inventories/control-plane.yaml +roles_path = ~/.ansible/roles +force_color = True +``` + +The same effect can be achieved with the `ANSIBLE_JINJA2_NATIVE=True` environment variable, which is what the CI examples in this guide use. ## Control Plane Playbook @@ -141,12 +164,18 @@ Basically, this playbook does the following: - Include all the modules, filter plugins, etc. of [ansible-common](https://github.com/metal-stack/ansible-common) into the play - Deploys an ingress-controller into your cluster - Deploys the metal-stack by - - Running preparation tasks - - Deploying NSQ - - Deploying the rethinkdb database for the metal-api (wrapped in a backup-restore-sidecar), - - Deploying the postgres database for go-ipam (wrapped in a backup-restore-sidecar) - - Deploying the postgres database for the masterdata-api (wrapped in a backup-restore-sidecar) - - Applying the metal control plane helm chart + - Running preparation tasks (namespace, image pull secrets, common resources) + - Deploying [NSQ](https://nsq.io/) — the message queue that connects metal-api and metal-core + - Deploying the RethinkDB database for the metal-api (wrapped in a backup-restore-sidecar) + - Deploying the PostgreSQL database for [go-ipam](https://github.com/metal-stack/go-ipam) (wrapped in a backup-restore-sidecar) + - Deploying the PostgreSQL database for the masterdata-api (wrapped in a backup-restore-sidecar) + - Applying the metal control plane Helm chart + +The role order matters: `prepare` creates the namespace all other roles deploy into, and the `metal` role expects the message queue and all three databases to be present. + +:::tip +The control-plane roles are additive. Once the basic deployment works, you can extend the same playbook with optional roles such as `monitoring`, `logging`, `auditing-timescaledb` (audit backend), `headscale` (firewall VPN), `zitadel`/`zitadel-db` (identity provider) or `isolated-clusters`. See the [control-plane role overview](https://github.com/metal-stack/metal-roles/tree/master/control-plane) for the complete list, and the [mini-lab playbook](https://github.com/metal-stack/mini-lab/blob/master/deploy_control_plane.yaml) for an example that enables most of them. +::: ## Setup an ingress-controller @@ -171,13 +200,25 @@ The [ansible-common](https://github.com/metal-stack/ansible-common) repository c ## Deployment Parametrization -Now you can parametrize the referenced roles to fit your environment. The role parametrization can be looked up in the role documentation on [metal-roles/control-plane](https://github.com/metal-stack/metal-roles/tree/master/control-plane). You should not need to define a lot of variables for the beginning as most values are reasonably defaulted. You can start with the following content for `group_vars/control-plane/common.yaml`: +Now you can parametrize the referenced roles to fit your environment. The role parametrization can be looked up in the role documentation on [metal-roles/control-plane](https://github.com/metal-stack/metal-roles/tree/master/control-plane). You should not need to define a lot of variables for the beginning as most values are reasonably defaulted. You can start with the following content for `inventories/group_vars/control-plane/common.yaml`: ```yaml --- -metal_control_plane_ingress_dns: # if you do not have a DNS entry, you could also start with .nip.io +# The DNS domain under which all control-plane services are exposed via ingress. +# If you do not have a DNS entry yet, you can start with .nip.io. +metal_control_plane_ingress_dns: + +# Optional but recommended: a name for this stage/landscape. It is used for +# prefixing and is inherited by the Gardener roles as the Garden name. +metal_control_plane_stage_name: demo ``` +:::warning Secrets +Any variable holding a credential — HMAC keys, database passwords, backup credentials, OIDC client secrets — belongs into an [Ansible Vault](https://docs.ansible.com/ansible/latest/vault_guide/index.html) encrypted file, never into a plain group_vars file. The examples in this guide reference vaulted variables (for example `metal_api_admin_key`) instead of inlining values. + +metal-stack ships with the default HMAC key `change-me`. Replace it with strong random values before the control plane becomes reachable from anywhere but your laptop. +::: + ## Providing Certificates We have several components in our stack that communicate over encrypted gRPC just like Kubernetes components do. @@ -372,7 +413,7 @@ Also define the following configurations for `cfssl`: Running the `roll_certs.sh` bash script without any arguments should generate you the required certificates. -Now Provide the paths to these certificates in `group_vars/control-plane/metal.yaml`: +Now provide the paths to these certificates in `inventories/group_vars/control-plane/metal.yaml`: ```yaml --- @@ -451,8 +492,11 @@ on: options: - metal-stack +permissions: + contents: read + env: - ANSIBLE_INVENTORY: inventory/inventory.yaml + ANSIBLE_INVENTORY: inventories/control-plane.yaml ANSIBLE_FORCE_COLOR: "1" ANSIBLE_JINJA2_NATIVE: "True" @@ -468,7 +512,7 @@ jobs: if: ${{ inputs.deploy-control-plane == 'metal-stack' }} runs-on: ubuntu-latest - container: ghcr.io/metal-stack/metal-deployment-base:v0.9.2 + container: ghcr.io/metal-stack/metal-deployment-base:v0.22.18 steps: - name: Checkout @@ -478,10 +522,10 @@ jobs: metal ctx add demo --api-token ${METALSTACKCLOUD_API_TOKEN} --default-project ${DEFAULT_PROJECT_ID} --activate metal cluster kubeconfig ${CLUSTER_ID} - echo ${ANSIBLE_VAULT_PASSWORD} > ${ANSIBLE_VAULT_PASSWORD_FILE} + printf '%s' "${ANSIBLE_VAULT_PASSWORD}" > ${ANSIBLE_VAULT_PASSWORD_FILE} ansible localhost -m metalstack.base.metal_stack_release_vector - ansible-playbook deploy_metal_stack.yaml + ansible-playbook deploy_metal_control_plane.yaml --diff env: METALSTACKCLOUD_API_TOKEN: ${{ secrets.METALSTACKCLOUD_API_TOKEN }} ANSIBLE_VAULT_PASSWORD: ${{ secrets.ANSIBLE_VAULT_PASSWORD }} @@ -498,7 +542,7 @@ The `metal` CLI is used to fetch the kubeconfig from the metal-stack API. Update After the deployment has finished, you should consider deploying some masterdata entities into your metal-api. For example, you can add your first machine sizes and operating system images. You can do this by further parametrizing the [metal role](https://github.com/metal-stack/metal-roles/tree/master/control-plane/roles/metal). We will just add an operating system for demonstration purposes. Add the following variable to your `inventories/group_vars/control-plane/common.yaml`: -``` +```yaml metal_api_images: - id: firewall-ubuntu-3.0 name: Firewall 3 Ubuntu @@ -520,17 +564,19 @@ metal_api_images: - machine ``` -See the [metal-images reference](https://metal-stack.io/docs/references/metal-images/#supported-images) for currently supported images. +See the [metal-images reference](../../08-References/Deployment/metal-images/metal-images.md) for currently supported images and the [operating systems](../02-operating-systems.md) chapter for the image contract if you want to build your own. Then, re-run the deployment to apply your changes. Our playbooks are idempotent. :::info -Image versions should be regularly checked for updates. +`metal_api_images` is more than a convenience: the Gardener roles default both the `CloudProfile` machine images and the `provider-metal` extension configuration to this very list. Keeping it as the single source of truth is what makes CloudProfile and extension agree — see the [Gardener deployment guide](./05_gardener.md#how-the-pieces-connect). + +Image versions should be checked for updates regularly; expired images can no longer be used for new allocations. ::: ## Setting up metalctl -You can now verify the existence of the operating system images in the metal-api using our CLI client called [metalctl](https://github.com/metal-stack/metalctl). The configuration for `metalctl` should look like this: +You can now verify the existence of the operating system images in the metal-api using our CLI client called [metalctl](https://github.com/metal-stack/metalctl). See the [metalctl reference](../../08-References/Clients/metalctl/metalctl.md) for the full command set. The configuration for `metalctl` should look like this: ```yaml # ~/.metalctl/config.yaml @@ -555,12 +601,16 @@ Issue the following command: ```bash $ metalctl image ls -ID NAME DESCRIPTION FEATURES EXPIRATION STATUS -ubuntu-19.10.20200331 Ubuntu 19.10 20200331 Ubuntu 19.10 20200331 machine 89d 23h preview +ID NAME DESCRIPTION FEATURES EXPIRATION STATUS +debian-13.0 Debian 13 Debian 13 Latest Release machine 89d 23h supported +firewall-ubuntu-3.0 Firewall 3 Ubuntu Firewall 3 Ubuntu Latest firewall 89d 23h supported +ubuntu-26.4 Ubuntu 26.04 Ubuntu 26.04 Latest Release machine 89d 23h supported ``` The basic principles of how the metal control plane can be deployed should now be clear. It is now up to you to move the deployment execution into your CI and add things like certificates for the ingress-controller and NSQ. +At this point you have a working **MaaS platform**: you can create projects and networks, register machines and allocate them through the metal-api. What is still missing is a [partition](./04_partition.md) with actual hardware — without it, there are no machines to allocate. + ## Setting Up the backup-restore-sidecar The backup-restore-sidecar can come in very handy when you want to add another layer of security to the metal-stack databases in your Kubernetes cluster. The sidecar takes backups of the metal databases in small time intervals and stores them in a blobstore of a cloud provider. For each database that will be backed up, a lifecycle rule is established. The backup mechanism is deactivated by default and must be activated by the operator. This way your metal-stack setup can even survive the deletion of your Kubernetes control plane cluster (including all volumes getting lost). After re-deploying metal-stack to another Kubernetes clusters, the databases come up with the latest backup data in a matter of seconds. @@ -576,6 +626,8 @@ metal-stack currently supports two authentication methods: - user authentication through [OpenID Connect](https://openid.net/developers/how-connect-works/) (OIDC) - [HMAC](https://en.wikipedia.org/wiki/HMAC) auth, typically used for access by technical users (because we do not have service account tokens at the time being) +There are three HMAC authentication types — `Metal-View`, `Metal-Edit` and `Metal-Admin` — which map to the corresponding roles. Partitions authenticate with the view and edit keys (see [Partition Connection](./04_partition.md#partition-connection)), and Gardener shoots use an admin or edit key (see [Gardener](./05_gardener.md#shootsyaml--shoot-clusters-and-shooted-seeds)). For the authorization model behind these roles, see [User Management](../../05-Concepts/02-user-management.md). + If you decided to use OIDC, you can parametrize the [metal role](https://github.com/metal-stack/metal-roles/tree/master/control-plane/roles/metal) for this by defining the variable `metal_masterdata_api_tenants` with the following configuration: ```yaml diff --git a/docs/04-For Operators/03-Deployment/04_partition.md b/docs/04-For Operators/03-Deployment/04_partition.md index 5fc640e4..794f2486 100644 --- a/docs/04-For Operators/03-Deployment/04_partition.md +++ b/docs/04-For Operators/03-Deployment/04_partition.md @@ -10,59 +10,65 @@ A partition is the data center infrastructure layer — the physical servers, sw This section continues from the [Control Plane](./03_control-plane.mdx) deployment and covers how to deploy the required infrastructure services and how to connect your partition to the control plane, using the [metal-stack partition Ansible roles](https://github.com/metal-stack/metal-roles/tree/master/partition). It is assumed that all cabling is done. -Before you try to deploy the partition, you should have had a look at the [architecture](../../05-Concepts/01-architecture.mdx) and [networking](../../05-Concepts/03-Network/) concepts sections. +Before you try to deploy the partition, you should have had a look at the [architecture](../../05-Concepts/01-architecture.mdx) and [networking](../../05-Concepts/03-Network/01-theory.md) concepts sections — the partition deployment is the point where the network theory becomes concrete configuration, and the roles will not make sense without it. + +:::info +Unlike the control plane, a partition deployment touches **physical hardware you have to bootstrap by hand first**. This chapter therefore has two halves: the manual bootstrap of the out-of-band network (firewalls, management servers, management switches) and the automated Ansible deployment of everything else. Budget most of your time for the first half — it is the part that differs the most between environments. +::: During this section, our repository will grow to look something like the following: ```text . -├── deploy_mgmt_servers.yaml # management server services -├── deploy_mgmt_switches.yaml # mgmtleaves and mgmtspines SONiC -├── deploy_spines_exits.yaml # spine/exit switch SONiC -├── deploy_leaves.yaml # leaf switch SONiC + metal-core -├── inventory -│ ├── inventory.yaml # all partition host groups -│ ├── host_vars +├── deploy_mgmt_servers.yaml # management server services +├── deploy_mgmt_switches.yaml # mgmtleaves and mgmtspines SONiC +├── deploy_spines_exits.yaml # spine/exit switch SONiC +├── deploy_leaves.yaml # leaf switch SONiC + metal-core +├── inventories +│ ├── partition.yaml # all partition host groups +│ ├── host_vars/ │ └── group_vars │ ├── all/ -│ │ ├── release_vector.yaml -│ │ └── metal-stack/ -│ │ ├── partition.yaml # partition → control plane connection -│ │ └── control-plane.yaml # vault-encrypted: API keys, BMC passwords +│ │ └── release_vector.yaml # unchanged, shared with the control plane +│ ├── partition/ +│ │ ├── common.yaml # partition → control plane connection +│ │ └── secrets.yaml # vault-encrypted: HMAC keys, BMC passwords │ ├── mgmtservers/ │ │ ├── common.yaml +│ │ └── metal-bmc.yaml │ ├── mgmtspines/ -│ │ ├── common.yaml -│ │ ├── metal-bmc.yaml +│ │ └── common.yaml │ ├── mgmtleaves/ -│ │ ├── common.yaml -│ ├── exits/ -│ │ ├── common.yaml -│ │ └── sonic.yaml +│ │ └── common.yaml │ ├── spines/ │ │ ├── common.yaml │ │ └── sonic.yaml -│ ├── leaves/ +│ ├── exits/ │ │ ├── common.yaml -│ │ ├── sonic.yaml -│ │ └── metal-core.yaml -│ └── partition/ -│ └── common.yaml +│ │ └── sonic.yaml +│ └── leaves/ +│ ├── common.yaml +│ ├── sonic.yaml +│ └── metal-core.yaml └── .github/ └── workflows/ - └── deploy-partition.yaml # CI/CD for partition + └── deploy-partition.yaml # CI/CD for partition ``` +:::tip +Because the partition plays target real hosts over SSH, they need their own inventory (`inventories/partition.yaml`) next to the `localhost`-only `inventories/control-plane.yaml` from the [Control Plane](./03_control-plane.mdx#inventory) guide. The `inventories/group_vars/all/` directory is shared by both, which is why the release vector only has to be declared once. +::: + ## Out-Of-Band-Network -The first step of deploying a partition is to deploy the Out-Of-Band-Network. +The first step of deploying a partition is to deploy the Out-Of-Band-Network (OOB network). It provides a dedicated management network that remains accessible even when the production network is down or unconfigured. It is the foundation that enables remote bootstrapping, monitoring, and maintenance of all partition hardware — from leaf and spine switches to bare-metal servers via their BMC/IPMI interfaces. -The [partition networking](https://metal-stack.io/docs/next/networking) is designed to be secure, fully routable via BGP, scalable, resilient, deployable through CI/CD, and selectively accessible from the internet. +The [partition networking](../../05-Concepts/03-Network/01-theory.md) is designed to be secure, fully routable via BGP, scalable, resilient, deployable through CI/CD, and selectively accessible from the internet. To deploy a partition and its networking stack remotely and in a nearly automatic manner, **some components must be initially bootstrapped manually**: - the management firewalls, management servers, management spines and management leaves need to be configured -- a CI/CD-Runner of your choice needs to be installed to run the automated deployments. +- a CI/CD runner of your choice needs to be installed to run the automated deployments The result should look something like the following image, but could vary for different deployments. @@ -96,7 +102,7 @@ The firewalls (EdgeRouters) must fulfill the following requirements: Management servers are the main bootstrapping components of the Out-Of-Band-Network and serve as jump hosts for all partition components. Once they are installed, every other component can be deployed automatically. For our high-availability setup, there are again two of them, each connected to their management-firewall. -Bootstrapping the management servers requires remote IPMI access and a way to perform an unattended OS installation with an Ansible user and SSH keys pre-configured. The exact approach depends on your hardware, existing infrastructure, and preferred automation tools. Below are two common examples, but any solultion is fine. +Bootstrapping the management servers requires remote IPMI access and a way to perform an unattended OS installation with an Ansible user and SSH keys pre-configured. The exact approach depends on your hardware, existing infrastructure, and preferred automation tools. Below are two common examples, but any solution is fine. **Preconfigured ISO with preseed** — generate an ISO with a preseed file that installs an OS and an Ansible user, then attach it via the BMC's virtual media function. @@ -116,9 +122,13 @@ The [mgmt-server role](https://github.com/metal-stack/metal-roles/tree/master/pa The runner on the management server needs: - An SSH key pair for Ansible authentication -— The public key in each switch's `authorized_keys` (this can be achieved via the `ZTP` scripts) +- The public key in each switch's `authorized_keys` (this can be achieved via the `ZTP` scripts) - CI/CD secrets or any kind of access to the Ansible Vault password and private key +:::warning Chicken-and-egg +The CI runner cannot deploy itself. Bootstrap it either manually or from a hosted runner, and only then hand over to the self-hosted runner for all subsequent partition deployments. The same applies to the management firewalls and the base OS of the management servers. +::: + ### Spine & Exit Management Switches (Management Spines) Management spines connect the management interfaces of all spine and exit switches to the management servers to enable out-of-band management. @@ -146,15 +156,15 @@ After the initial bootstrapping, the management interfaces of the leaves continu In larger deployments, a dedicated set of out-of-band switches (mgmtoobs) may be used to isolate BMC/IPMI traffic from the management network. These switches connect directly to server BMCs and provide a separate L2 domain for IPMI traffic, keeping it isolated from management server and switch management interfaces. They are deployed through the same SONiC automation as other partition switches. ### Leaves and Spines (Production Network) + We have now reached the point where a lot of the configuration happens automatically via the official Ansible roles. -After the initial install via ONIE and ZTP, metal-core will take over the configuration of the leaves and spines. +After the initial install via ONIE and ZTP, the `sonic-config` role establishes the base configuration, and `metal-core` takes over the dynamic, per-machine configuration of the leaves. -The general role of the leaves and spines is explained in the [CLOS](https://metal-stack.io/docs/next/networking#clos) concepts section. +The general role of leaves and spines is explained in the [CLOS](../../05-Concepts/03-Network/01-theory.md#clos) concepts section, and the routing details in [BGP](../../05-Concepts/03-Network/01-theory.md#bgp) and [EVPN](../../05-Concepts/03-Network/01-theory.md#evpn). -They also add the last piece of the puzzle for the bare-metal provisioning: -As the workers are directly connected to the leaves, meaning PXE boot requests need to be handled by them. -This is achieved by relaying DHCP requests from those interfaces to the management server (via the management-spine) so workers receive IP addresses for PXE boot. -After that, they are able to pull the image for the automated install and setup. +They also add the last piece of the puzzle for bare-metal provisioning: the workers are directly connected to the leaves, so their PXE boot requests have to be handled there. +This is achieved by relaying DHCP requests from those interfaces to the management server (via the management spine), so workers receive IP addresses for PXE boot. +After that, they are able to pull the [metal-hammer](https://github.com/metal-stack/metal-hammer) discovery image and the operating system image for the automated install and setup. The full sequence is documented as the [machine provisioning sequence](../../05-Concepts/01-architecture.mdx#machine-provisioning-sequence). --- @@ -165,20 +175,21 @@ The next step is to configure the Ansible inventory and playbooks that define yo The playbooks and directory structure shown in this document represent a **reference implementation** — one way to organize your deployment. The metal-stack partition roles are designed to be flexible, and you are free to organize playbooks, group hosts differently, or run services on different machines as long as the following architectural constraints are met: -- **PXE boot requires DHCP in the same Layer-2 domain** as unprovisioned servers. The PXE VLAN (`vlan4000`) must reach all bare metal servers that need provisioning. In our reference setup, the management server runs the DHCP server, and leaf switches run a DHCP relay (configured via the `sonic-config` role) that forwards requests from the production network back to the management server. You may place the DHCP server and relay wherever your network topology allows, as long as the L2 domain is preserved. See the [networking documentation](../../05-Concepts/03-Network/01-theory.md#pxe-boot-mode) for the full PXE/DHCP theory. +- **PXE boot requires DHCP in the same Layer-2 domain** as unprovisioned servers. The PXE VLAN (`vlan4000` by default, configurable via `metal_core_pxe_vlan_id` and `dhcp_relay_interface`) must reach all bare metal servers that need provisioning. In our reference setup, the management server runs the DHCP server, and the switches run a DHCP relay that forwards requests from the production network back to the management server. You may place the DHCP server and relay wherever your network topology allows, as long as the L2 domain is preserved. See the [networking documentation](../../05-Concepts/03-Network/01-theory.md#pxe-boot-mode) for the full PXE/DHCP theory. - **`metal-core` must run on leaf switches** to dynamically configure them from the metal-api. -- **Pixiecore must be reachable** by servers during PXE boot (TFTP/HTTP). -- **`metal-bmc` must run on a host with BMC/IPMI network access** to manage bare-metal servers. -- **Web-Server with switch images must be reachable** by switches during install -- **Image cache must be reachable** by machines during provisioning +- **Pixiecore must be reachable** by servers during PXE boot (TFTP/HTTP) — it serves the metal-hammer kernel and initrd. +- **`metal-bmc` must run on a host with BMC/IPMI network access** to manage bare-metal servers (power state, boot order, machine discovery). +- **A web server holding the switch images must be reachable** by switches during ONIE install and ZTP. +- **The image cache must be reachable** by machines during provisioning — it serves the operating system images and is also the key to [offline resilience](./08_offline-resilience.md). You can split these services across multiple playbooks, combine them into fewer playbooks, or run them on different hosts — the roles are independent and can be mixed and matched. The key is ensuring the services are deployed and the network dependencies are satisfied. ### Host Inventory -Add your networking infrastructure to the inventory and adapt the host names and group structure to match your physical topology. For production switches, use disjoint groups (e.g. odd/even) to enable rolling deployments without fabric disruption. +Add your networking infrastructure to `inventories/partition.yaml` and adapt the host names and group structure to match your physical topology. For production switches, use disjoint groups (e.g. odd/even) to enable rolling deployments without fabric disruption — every leaf pair, spine pair and exit pair should be split across the two groups so that one half of the fabric always stays up. ```yaml +--- partition: children: mgmtservers: @@ -208,12 +219,15 @@ partition: ### Partition Connection -The `inventory/group_vars/all/metal-stack/partition.yaml` file connects the partition to the control plane: +The `inventories/group_vars/partition/common.yaml` file connects the partition to the control plane. `metal_partition_id` must match a partition that exists in the metal-api — create it with `metalctl partition create` (or via the `metal` role) before the first deployment: ```yaml +--- metal_region: metal_partition_id: +metal_partition_timezone: Europe/Berlin +# REST API of the metal-api metal_partition_metal_api_protocol: https metal_partition_metal_api_addr: metal_partition_metal_api_port: 443 @@ -221,23 +235,42 @@ metal_partition_metal_api_basepath: /metal/ metal_partition_metal_api_hmac_edit_key: "{{ metal_control_plane_api_edit_key }}" metal_partition_metal_api_hmac_view_key: "{{ metal_control_plane_api_view_key }}" +# gRPC endpoint of the metal-api, consumed by metal-core and metal-hammer. +# The certificates are the ones generated in the control plane guide. metal_partition_metal_api_grpc_address: "{{ metal_partition_mgmt_gateway }}:50051" metal_partition_metal_api_grpc_ca_cert: "{{ lookup('file', 'certs/ca.pem') }}" metal_partition_metal_api_grpc_client_cert: "{{ lookup('file', 'certs/metal-api-grpc/client.pem') }}" metal_partition_metal_api_grpc_client_key: "{{ lookup('file', 'certs/metal-api-grpc/client-key.pem') }}" -metal_partition_mgmt_gateway: 172.17.0.1 +# Gateway through which partition components reach the control plane +metal_partition_mgmt_gateway: ``` -The HMAC keys and other sensitive values should be stored in an Ansible vault. +:::warning +The HMAC keys must match the ones configured for the control plane, and the gRPC client certificate must be signed by the same CA as the metal-api server certificate — both come from the [Providing Certificates](./03_control-plane.mdx#providing-certificates) step. All of these values are secrets and belong into an Ansible Vault encrypted file. +::: + +For the full list of shared partition variables, see the [metal-roles/partition README](https://github.com/metal-stack/metal-roles/tree/master/partition). -### Leaf and Spine Variables +### Leaf, Spine and Management Server Variables -Switch variables define SONiC configuration, `metal-core` settings, and monitoring exporters. Management server variables define BMC access, DHCP configuration, and image cache settings. +Switch variables define the SONiC configuration, `metal-core` settings, and monitoring exporters. Management server variables define BMC access, DHCP configuration, and image cache settings. -The `sonic-config` role handles FRR routing configuration, DHCP relay setup on switches, and generates the complete `config_db.json` for SONiC. For complex configurations such as port breakouts, VRFs, EVPN underlay, interconnects, and extended CACL rules, refer to the [sonic-config role](https://github.com/metal-stack/metal-roles/tree/master/partition/roles/sonic-config) documentation and the `sonic-config` role defaults. +The `sonic-config` role handles FRR routing configuration, the DHCP relay setup on switches, and generates the complete `config_db.json` for SONiC. For complex configurations such as port breakouts, VRFs, EVPN underlay, interconnects, and extended control-plane ACL (CACL) rules, refer to the [sonic-config role](https://github.com/metal-stack/metal-roles/tree/master/partition/roles/sonic-config) documentation and its role defaults. -For the full variable reference, see the [metal-roles/partition](https://github.com/metal-stack/metal-roles/tree/master/partition) documentation. +Each role documents its variables in its own README. The ones you will touch most often: + +| Role | Purpose | README | +| --------------------- | ----------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | +| `mgmt-server` | Base configuration of the management server (BGP, DNS, SSH) | [mgmt-server](https://github.com/metal-stack/metal-roles/tree/master/partition/roles/mgmt-server) | +| `mgmt-firewall` | Reference configuration for the management firewalls | [mgmt-firewall](https://github.com/metal-stack/metal-roles/tree/master/partition/roles/mgmt-firewall) | +| `sonic-config` | SONiC `config_db.json` and FRR for all switches | [sonic-config](https://github.com/metal-stack/metal-roles/tree/master/partition/roles/sonic-config) | +| `metal-core` | Dynamic leaf configuration driven by the metal-api | [metal-core](https://github.com/metal-stack/metal-roles/tree/master/partition/roles/metal-core) | +| `metal-bmc` | BMC/IPMI access and machine discovery | [metal-bmc](https://github.com/metal-stack/metal-roles/tree/master/partition/roles/metal-bmc) | +| `dhcp` / `dhcp-relay` | DHCP server and relay for PXE boot | [dhcp](https://github.com/metal-stack/metal-roles/tree/master/partition/roles/dhcp), [dhcp-relay](https://github.com/metal-stack/metal-roles/tree/master/partition/roles/dhcp-relay) | +| `pixiecore` | Serves metal-hammer kernel and initrd via PXE | [pixiecore](https://github.com/metal-stack/metal-roles/tree/master/partition/roles/pixiecore) | +| `image-cache` | Local mirror for OS images, kernels and metal-hammer | [image-cache](https://github.com/metal-stack/metal-roles/tree/master/partition/roles/image-cache) | +| `ztp` | Zero Touch Provisioning payloads for SONiC switches | [ztp](https://github.com/metal-stack/metal-roles/tree/master/partition/roles/ztp) | ## Ansible Playbooks @@ -245,7 +278,7 @@ The following sections show a reference playbook structure. Each service is depl ### Management Server (`deploy_mgmt_servers.yaml`) -The management servers run the CI runners, image caches, DHCP server, ZTP, `metal-bmc`, and Pixiecore and optional services like Tailscale. This is the central bootstrap host — once deployed, it enables automated provisioning of all other partition components. +The management servers run the CI runners, image cache, DHCP server, ZTP web server, `metal-bmc` and Pixiecore, plus optional services such as [Tailscale](../../05-Concepts/03-Network/03-tailscale.md). This is the central bootstrap host — once deployed, it enables automated provisioning of all other partition components. ```yaml --- @@ -253,8 +286,10 @@ The management servers run the CI runners, image caches, DHCP server, ZTP, `meta hosts: mgmtservers roles: - name: ansible-common + tags: always - name: metal-roles/common/roles/defaults - name: metal-roles/partition/roles/mgmt-server + - name: metal-roles/partition/roles/lvm - name: metal-roles/partition/roles/dhcp - name: metal-roles/partition/roles/ztp - name: metal-roles/partition/roles/metal-bmc @@ -263,9 +298,13 @@ The management servers run the CI runners, image caches, DHCP server, ZTP, `meta - name: artis3n.tailscale ``` +:::info +The `lvm` role prepares the local volumes that the image cache uses. `artis3n.tailscale` is a third-party Galaxy role and only needed if you expose services via Tailscale; drop it otherwise. +::: + ### Management Switches (`deploy_mgmt_switches.yaml`) -Deploys SONiC on management leaves and management spines. These switches form the out-of-band management network and relay DHCP requests from production switches and worker servers to the management server. The `sonic-config` is used to configure FRR routing, BGP peering, and the DHCP relay agent on these switches. +Deploys the SONiC configuration on management leaves and management spines. These switches form the out-of-band management network and relay DHCP requests from production switches and worker servers to the management server. The `sonic-config` role configures FRR routing, BGP peering, and the DHCP relay agent on these switches. ```yaml --- @@ -273,31 +312,38 @@ Deploys SONiC on management leaves and management spines. These switches form th hosts: mgmtleaves,mgmtspines roles: - name: ansible-common + tags: always - name: metal-roles/common/roles/defaults - name: metal-roles/partition/roles/sonic-config ``` ### Production Spines and Exits (`deploy_spines_exits.yaml`) -Deploys SONiC on spine and exit switches in disjoint groups (e.g. odd/even) to avoid disrupting the entire fabric at once. The `sonic-config` is used to configure FRR routing, BGP underlay, and the DHCP relay agent on exit switches to forward PXE requests to the management server. +Deploys the SONiC configuration on spine and exit switches in disjoint groups (e.g. odd/even) to avoid disrupting the entire fabric at once. The `sonic-config` role configures FRR routing, the BGP underlay, and the DHCP relay agent that forwards PXE requests to the management server. ```yaml --- - name: deploy odd spines and exits hosts: odd_switches + any_errors_fatal: true roles: - name: ansible-common + tags: always - name: metal-roles/common/roles/defaults - name: metal-roles/partition/roles/sonic-config - name: deploy even spines and exits hosts: even_switches + any_errors_fatal: true roles: - name: ansible-common + tags: always - name: metal-roles/common/roles/defaults - name: metal-roles/partition/roles/sonic-config ``` +`any_errors_fatal: true` aborts the play as soon as one switch fails, so a broken configuration is not rolled out across the whole group. + The inventory defines the disjoint groups: ```yaml @@ -316,7 +362,7 @@ Deploy the odd group first, verify the fabric is stable, then deploy the even gr ### Production Leaves (`deploy_leaves.yaml`) -Deploys SONiC configuration and `metal-core` on leaf switches. `metal-core` dynamically configures the leaf from the metal-api and relays DHCP requests from the metal-hammer (discovery image) during PXE provisioning. +Deploys the SONiC configuration and `metal-core` on leaf switches. `metal-core` receives switch configuration from the metal-api via gRPC and reconciles the ports of allocated machines onto the leaf, and it relays the DHCP requests of the metal-hammer (discovery image) during PXE provisioning. Again, the same disjoint groups approach is used. Deploy one group, verify stability, then proceed with the next. @@ -324,17 +370,29 @@ Again, the same disjoint groups approach is used. Deploy one group, verify stabi --- - name: deploy even leaves hosts: even_leaves + any_errors_fatal: true roles: - name: ansible-common + tags: always - name: metal-roles/common/roles/defaults - name: metal-roles/partition/roles/sonic-config - name: metal-roles/partition/roles/metal-core -# - name: deploy odd leaves -# hosts: odd_leaves -# ... +- name: deploy odd leaves + hosts: odd_leaves + any_errors_fatal: true + roles: + - name: ansible-common + tags: always + - name: metal-roles/common/roles/defaults + - name: metal-roles/partition/roles/sonic-config + - name: metal-roles/partition/roles/metal-core ``` +:::warning +`metal-core` registers the switch with the metal-api on first start. Verify with `metalctl switch ls` that every leaf shows up and reports its ports before you allocate the first machine — an unregistered leaf cannot be configured for a machine allocation. +::: + ## CI/CD Workflow Add a CI/CD workflow to your repository (`.github/workflows/deploy-partition.yaml`). The example below uses GitHub Actions, but any runner works — the pattern is the same. @@ -360,7 +418,7 @@ on: - leaves env: - ANSIBLE_INVENTORY: inventory/inventory.yaml + ANSIBLE_INVENTORY: inventories/partition.yaml ANSIBLE_FORCE_COLOR: "1" ANSIBLE_JINJA2_NATIVE: "True" @@ -371,12 +429,17 @@ jobs: runs-on: self-hosted container: ghcr.io/metal-stack/metal-deployment-base:v0.22.18 steps: - - uses: actions/checkout@v7 - - run: rm -rf /github/home/.ansible/roles/* + - name: Checkout + uses: actions/checkout@v7 + + - name: Clean ansible dir + run: rm -rf /github/home/.ansible + - run: | printf '%s' "${ANSIBLE_VAULT_PASSWORD}" > ${ANSIBLE_VAULT_PASSWORD_FILE} printf '%s' "${ANSIBLE_PRIVATE_KEY}" > ${ANSIBLE_PRIVATE_KEY_FILE} chmod 400 ${ANSIBLE_PRIVATE_KEY_FILE} + ansible localhost -m metalstack.base.metal_stack_release_vector ansible-playbook deploy_mgmt_servers.yaml --diff env: @@ -384,11 +447,14 @@ jobs: ANSIBLE_VAULT_PASSWORD_FILE: .vault.txt ANSIBLE_PRIVATE_KEY: ${{ secrets.ANSIBLE_PRIVATE_KEY }} ANSIBLE_PRIVATE_KEY_FILE: .ssh-key + + - name: Clean ansible dir after deploy + run: rm -rf /github/home/.ansible ``` Copy this job block for each remaining phase, adjusting the job name, `if` condition, and playbook name. -Each phase is independent and can be deployed separately via workflow dispatch. +Each phase is independent and can be deployed separately via workflow dispatch. Note the `rm -rf /github/home/.ansible/*` step: it clears previously downloaded role artifacts so that `metal_stack_release_vector` always fetches the roles matching the pinned release instead of reusing a stale cache. ## Deployment Order @@ -413,15 +479,36 @@ The recommended deployment sequence respects service dependencies. Each phase ca 4. Production Leaves → deploy_leaves.yaml ├── SONiC (via sonic-config role) └── metal-core +``` + +## Verifying the Partition + +Once all four phases have run, verify from your workstation against the metal-api: +```bash +# The partition must be known to the metal-api +metalctl partition ls + +# Every leaf, spine and exit switch registered by metal-core +metalctl switch ls + +# Machines discovered by metal-bmc and reported by metal-hammer. +# Freshly discovered machines appear in state "Waiting" / "Available". +metalctl machine ls ``` +If machines never appear, the PXE path is the usual culprit: check that the DHCP relay forwards from the PXE VLAN to the management server, that Pixiecore is reachable, and that the image cache serves the metal-hammer artifacts. The [troubleshooting guide](../06-troubleshoot.md) walks through the individual steps; `metalctl machine console ` gives you the machine's console via metal-console. + ## Updating Components To update metal-stack components: -1. **Update the release version** in `inventory/group_vars/all/release_vector.yaml`. Do not skip versions. -2. **Commit and push** the change to your deployment repository -3. **Trigger the deployment** via GitHub Actions workflow dispatch -4. The `metal_stack_release_vector` Ansible module fetches the latest compatible component versions from the OCI registry -5. Wait for the pipeline to finish +1. **Update the release version** in `inventories/group_vars/all/release_vector.yaml`. Do not skip minor versions — consult the release notes for the target version first. +2. **Commit and push** the change to your deployment repository. +3. **Trigger the deployment** via workflow dispatch — control plane first, then the partition. +4. The `metal_stack_release_vector` Ansible module fetches the component versions and the matching Ansible roles from the OCI registry and verifies their signatures. +5. Wait for the pipeline to finish and verify as described above. + +:::warning +The control plane and its partitions must stay within the same metal-stack release. Always roll out the control plane first, then the partitions — never the other way around. For switch OS upgrades there is a dedicated `sonic-upgrade` role; see [Maintenance](../04-maintenance.md). +::: diff --git a/docs/04-For Operators/03-Deployment/05_gardener.md b/docs/04-For Operators/03-Deployment/05_gardener.md index b49cee33..436b87ee 100644 --- a/docs/04-For Operators/03-Deployment/05_gardener.md +++ b/docs/04-For Operators/03-Deployment/05_gardener.md @@ -8,12 +8,18 @@ sidebar_position: 5 This guide shows how to deploy [Gardener](https://gardener.cloud/) on top of your metal-stack infrastructure using the [`gardener-*` Ansible roles](https://github.com/metal-stack/metal-roles/tree/master/control-plane) from the [metal-roles](https://github.com/metal-stack/metal-roles) repository. Gardener with metal-stack turns your bare-metal servers into a Kubernetes-as-a-Service platform where teams can self-serve clusters. -This guide assumes you are already familiar with Gardener's core concepts — [Garden](https://gardener.cloud/docs/gardener/concepts/operator/), [Seed](https://gardener.cloud/docs/gardener/concepts/gardenlet/), [Shoot](https://gardener.cloud/docs/gardener/concepts/apiserver/) and [CloudProfile](https://gardener.cloud/docs/gardener/concepts/apiserver/#cloudprofiles) — and have already completed the [Control Plane](./03_control-plane.mdx) and [Partition](./04_partition.md) deployment guides. For an overview of how Gardener integrates with metal-stack at a conceptual level, see the [Gardener concepts guide](../../05-Concepts/04-Kubernetes/02-gardener.md). +This guide assumes you are already familiar with Gardener's core concepts — [Garden](https://gardener.cloud/docs/gardener/concepts/operator/), [Seed](https://gardener.cloud/docs/gardener/concepts/gardenlet/), [Shoot](https://gardener.cloud/docs/gardener/concepts/apiserver/) and [CloudProfile](https://gardener.cloud/docs/gardener/concepts/apiserver/#cloudprofiles) — and have already completed the [Control Plane](./03_control-plane.mdx) and [Partition](./04_partition.md) deployment guides. For an overview of how Gardener integrates with metal-stack at a conceptual level, see the [Gardener concepts guide](../../05-Concepts/04-Kubernetes/02-gardener.md); for the reasoning behind choosing Gardener over Cluster API, see the [KCLM comparison](../../05-Concepts/04-Kubernetes/01-kclm.md#two-approaches-one-infrastructure). + +:::warning +A partition with registered, allocatable machines is a hard prerequisite. Gardener will happily accept a `Shoot`, but the `provider-metal` extension can only reconcile it if the metal-api can allocate machines and firewalls of the requested sizes and images in the requested partition. Verify with `metalctl machine ls` before you start. +::: The upstream [Gardener landscape setup guide](https://gardener.cloud/docs/gardener/deployment/setup_gardener/) describes the same building blocks — operator, `Garden`, extensions, `CloudProfile`, DNS secrets, `Gardenlet`, `ManagedSeed` — but expects you to render and apply the manifests yourself. The `gardener-*` roles are a thin, opinionated automation layer over exactly those building blocks, pre-wired for metal-stack: each role owns one resource and pulls all container images and Helm chart references from the metal-stack [release vector](./03_control-plane.mdx#releases-and-ansible-role-dependencies). :::tip -The [mini-lab](https://github.com/metal-stack/mini-lab) contains a working, minimal Gardener deployment (`deploy_gardener.yaml` plus `inventories/group_vars`) that uses the very same roles. It is a **development environment**, not a production reference, but it is the fastest way to see a complete, runnable parametrization. +The [mini-lab](https://github.com/metal-stack/mini-lab) contains a working, minimal Gardener deployment (`deploy_gardener.yaml` plus `inventories/group_vars`) that uses the very same roles. It is the fastest way to see a complete, runnable parametrization. + +It is **not** a production reference: it runs on a single kind cluster, fakes the `kube-system/shoot-info` ConfigMap, patches the Istio load balancer status by hand, deploys MinIO and PowerDNS as in-cluster stand-ins for object storage and DNS, and skips managed seeds entirely. Use it to understand _how_ the roles interact, not _what_ to configure. ::: ## Repository structure after this section @@ -46,7 +52,7 @@ The following files are added to the repository structure from the previous sect ## Architecture Overview -The metal-roles deploy Gardener in the **virtual Garden** pattern described upstream: the `gardener-operator` runs on your *runtime cluster* and reconciles a `Garden` resource, which spins up a nodeless *virtual Garden* cluster hosting the Gardener API (`Shoot`, `Seed`, `Project`, `CloudProfile`, …). The gardenlet also runs on the runtime cluster and registers it as the first, unmanaged `Seed` (a "soil"). +The metal-roles deploy Gardener in the **virtual Garden** pattern described upstream: the `gardener-operator` runs on your _runtime cluster_ and reconciles a `Garden` resource, which spins up a nodeless _virtual Garden_ cluster hosting the Gardener API (`Shoot`, `Seed`, `Project`, `CloudProfile`, …). The gardenlet also runs on the runtime cluster and registers it as the first, unmanaged `Seed` (a "soil"). ```mermaid graph TD @@ -76,20 +82,20 @@ graph TD SEED --> USH ``` -The soil is reserved for *infrastructure* shoots. Those shoots are turned into Gardener-managed Seeds via `ManagedSeed`, and end-user shoot control planes are hosted there. This is the recommended upstream pattern and the one the roles are built for. +The soil is reserved for _infrastructure_ shoots. Those shoots are turned into Gardener-managed Seeds via `ManagedSeed`, and end-user shoot control planes are hosted there. This is the recommended upstream pattern and the one the roles are built for. **Order matters.** Every role except `gardener-operator` and `gardener-extensions` obtains a kubeconfig for the virtual Garden through the `virtual_garden_kubeconfig` module, which only works once the operator has created the `Garden` and `gardener-virtual-garden-access` has deployed the token-requestor secret: -| Role | Applies to | Deploys | Requires | -| ---- | ---------- | ------- | -------- | -| `gardener-operator` | runtime cluster | `garden` namespace, backup + DNS provider secrets, operator Helm chart, `Garden` resource | Runtime cluster; cert-manager only if the dashboard is enabled | -| `gardener-extensions` | runtime cluster | one `operator.gardener.cloud/v1alpha1` `Extension` per enabled extension | `gardener-operator` | -| `gardener-virtual-garden-access` | both | `ManagedResource` + token-requestor secret that yields a rotating virtual-Garden kubeconfig | `Garden` reconciled | -| `gardener-cloud-profile` | virtual Garden | `CloudProfile` named `metal` | virtual-garden-access | -| `gardener-projects` | virtual Garden | `Project` resources | virtual-garden-access | -| `gardener-gardenlet` | virtual Garden | internal/default domain secrets, backup secret, `Gardenlet` resource (first Seed) | virtual-garden-access, extensions | -| `gardener-shoots` | virtual Garden | provider `Secret` + `CredentialsBinding` + `Shoot` per entry | `CloudProfile`, `Project`, a ready `Seed` | -| `gardener-managed-seeds` | virtual Garden | backup secret + `ManagedSeed` per entry | a reconciled shooted seed | +| Role | Applies to | Deploys | Requires | +| -------------------------------- | --------------- | ---------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------- | +| `gardener-operator` | runtime cluster | `garden` namespace, backup + DNS provider secrets, operator Helm chart, `Garden` resource | Runtime cluster; cert-manager only if the dashboard is enabled | +| `gardener-extensions` | runtime cluster | one `operator.gardener.cloud/v1alpha1` `Extension` per enabled extension | `gardener-operator` | +| `gardener-virtual-garden-access` | runtime cluster | `ManagedResource` + token-requestor secret that yields a rotating virtual-Garden kubeconfig, plus optional `ClusterRoleBinding`s for OIDC auth | `Garden` reconciled | +| `gardener-cloud-profile` | virtual Garden | `CloudProfile` named `metal` | virtual-garden-access | +| `gardener-projects` | virtual Garden | `Project` resources | virtual-garden-access | +| `gardener-gardenlet` | virtual Garden | internal/default domain secrets, backup secret, `Gardenlet` resource (first Seed) | virtual-garden-access, extensions | +| `gardener-shoots` | virtual Garden | provider `Secret` + `CredentialsBinding` + `Shoot` per entry | `CloudProfile`, `Project`, a ready `Seed` | +| `gardener-managed-seeds` | virtual Garden | backup secret + `ManagedSeed` per entry | a reconciled shooted seed | Two optional roles are not part of the minimal setup but are worth knowing about: `gardener-monitoring-certs` (wildcard monitoring certificates for soil and seeds, requires reachable seed API servers) and `gardener-logging` (ships seed logs to the central metal-stack Loki, requires the `logging` role). @@ -143,16 +149,16 @@ Deploy in two passes on a green field: first everything up to `gardener-gardenle To inspect the landscape afterwards, obtain a kubeconfig for the virtual Garden the same way the roles do: ```yaml - post_tasks: - - name: Get kubeconfig for virtual garden access - virtual_garden_kubeconfig: - garden_name: "{{ gardener_defaults_garden_name }}" - - - name: Write it out for manual inspection - ansible.builtin.copy: - content: "{{ virtual_garden_kubeconfig }}" - dest: .virtual-garden-kubeconfig - mode: "0600" +post_tasks: + - name: Get kubeconfig for virtual garden access + virtual_garden_kubeconfig: + garden_name: "{{ gardener_defaults_garden_name }}" + + - name: Write it out for manual inspection + ansible.builtin.copy: + content: "{{ virtual_garden_kubeconfig }}" + dest: .virtual-garden-kubeconfig + mode: "0600" ``` ## Step 2: Configure Group Variables @@ -162,12 +168,13 @@ All Gardener configuration lives under `inventories/group_vars/control-plane/gar Two shared variables must be set in `inventories/group_vars/control-plane/common.yaml`: ```yaml -# Names the Gardener landscape; also becomes the Garden resource name and the -# metalControlPlanes key in the CloudProfile. Defaults to metal_control_plane_stage_name. +# Names the Gardener landscape. gardener_defaults_garden_name defaults to this +# value, so it also becomes the Garden resource name and the metalControlPlanes +# key in the CloudProfile. You already set this in the control plane guide. metal_control_plane_stage_name: demo # Mandatory and asserted: "metal" or "gcp". Determines how the runtime cluster -# CIDRs are discovered and becomes the Seed's provider type. +# CIDRs are discovered and is rendered as the Seed's provider type. metal_control_plane_host_provider: metal ``` @@ -188,6 +195,7 @@ gardener_operator_virtual_garden_public_dns: gardener-kube-apiserver.{{ metal_co gardener_operator_ingress_dns_domain: k8s. # Storage class for the virtual Garden etcd volumes (20Gi main, 10Gi events). +# If unset, the cluster's default storage class is used. gardener_operator_virtual_garden_etcd_storage_class: csi-lvm # Renders spec.runtimeCluster.provider.region in the Garden resource. @@ -226,7 +234,7 @@ gardener_operator_dashboard_enabled: false - The `Garden` template only renders `spec.runtimeCluster.ingress` and `spec.virtualCluster.dns` **if `gardener_operator_dns_providers` is non-empty**, and always uses `gardener_operator_dns_providers[0].type` as the provider for both. Put your primary provider first. - The virtual Garden service CIDR is fixed to `100.64.0.0/13` and the maintenance window to `220000+0100`–`230000+0100` by the template. Make sure `100.64.0.0/13` does not overlap with your runtime cluster or partition networks. - Enabling the dashboard additionally requires `gardener_operator_wildcard_ingress_certificate_cluster_issuer` plus a cert-manager `ClusterIssuer`; the role then blocks until the wildcard certificate secret exists (up to 10 minutes). For OIDC login set `gardener_operator_dashboard_oidc_issuer_url`, `..._client_id`, `..._client_id_public` and `..._client_secret`. -- If automatic DNS creation is not available, create the A record for `gardener_operator_virtual_garden_public_dns` manually, pointing at the external address of the `istio-ingressgateway` service in the `virtual-garden-istio-ingress` namespace. +- If automatic DNS creation is not available, create the A record for `gardener_operator_virtual_garden_public_dns` manually, pointing at the external address of the `istio-ingressgateway` service that the operator deploys. For purely local setups the role offers `gardener_operator_expose_virtual_garden_through_ingress_nginx: true` as an alternative — that is what the mini-lab uses, and it is not meant for production. ### extensions.yaml — Provider and shoot extensions @@ -252,7 +260,8 @@ gardener_extension_provider_metal_admission_default_pods_cidr: 10.248.64.0/18 gardener_extension_provider_metal_admission_default_services_cidr: 10.248.192.0/18 # --- Operating system ----------------------------------------------------- -# One OperatingSystemConfig resource is registered per type. +# One OperatingSystemConfig type is registered per entry. +# Default: [ubuntu, debian, nvidia]. Trim it to what your CloudProfile offers. gardener_extension_os_metal_types: - ubuntu - debian @@ -365,7 +374,7 @@ gardener_project_defaults: members: [] gardener_projects: - - name: infrastructure # holds the shooted seeds + - name: infrastructure # holds the shooted seeds description: Infrastructure clusters - name: prod owner: alice@example.com @@ -451,7 +460,7 @@ gardener_shoot_rollout_wait_delay: 10 gardener_shoots: - name: seed-a - seed_name: "{{ gardener_defaults_garden_name }}" # scheduled onto the soil + seed_name: "{{ gardener_defaults_garden_name }}" # scheduled onto the soil project_id: purpose: infrastructure region: "{{ metal_region }}" @@ -519,9 +528,9 @@ gardener_managed_seed_defaults: external_traffic_policy: Local gardener_managed_seeds: - - name: seed-a # must match a shoot from shoots.yaml + - name: seed-a # must match a shoot from shoots.yaml region: "{{ metal_region }}" - pod_cidr: 10.240.0.0/13 # must match the shoot's networking + pod_cidr: 10.240.0.0/13 # must match the shoot's networking service_cidr: 10.248.0.0/18 ingress_domain: ingress.seed-a.k8s. logging_enabled: false @@ -599,12 +608,15 @@ on: workflow_dispatch: inputs: deploy-control-plane: - description: 'Which control-plane target to deploy' + description: "Which control-plane target to deploy" required: true type: choice options: - - metal-stack - - gardener + - metal-stack + - gardener + +permissions: + contents: read env: ANSIBLE_INVENTORY: inventories/control-plane.yaml @@ -623,7 +635,7 @@ jobs: if: ${{ inputs.deploy-control-plane == 'gardener' }} runs-on: ubuntu-latest - container: ghcr.io/metal-stack/metal-deployment-base:v0.9.2 + container: ghcr.io/metal-stack/metal-deployment-base:v0.22.18 steps: - name: Checkout @@ -633,19 +645,17 @@ jobs: metal ctx add demo --api-token ${METALSTACKCLOUD_API_TOKEN} --default-project ${DEFAULT_PROJECT_ID} --activate metal cluster kubeconfig ${CLUSTER_ID} - echo ${ANSIBLE_VAULT_PASSWORD} > ${ANSIBLE_VAULT_PASSWORD_FILE} + printf '%s' "${ANSIBLE_VAULT_PASSWORD}" > ${ANSIBLE_VAULT_PASSWORD_FILE} ansible localhost -m metalstack.base.metal_stack_release_vector - ansible-playbook deploy_gardener.yaml + ansible-playbook deploy_gardener.yaml --diff env: METALSTACKCLOUD_API_TOKEN: ${{ secrets.METALSTACKCLOUD_API_TOKEN }} ANSIBLE_VAULT_PASSWORD: ${{ secrets.ANSIBLE_VAULT_PASSWORD }} ANSIBLE_VAULT_PASSWORD_FILE: .vault.txt ``` -:::tip -The example uses GitHub Actions with `ubuntu-latest` runners (not self-hosted) and a runtime cluster hosted on [metal-stack cloud](https://metalstack.cloud/en). As mentioned in [Bootstrap Infrastructure](./02_bootstrap-infrastructure.md), any Kubernetes cluster can serve as the runtime — you can adapt the workflow to use self-hosted runners or a different cluster provider as needed. The `metal` CLI is used to fetch the kubeconfig from the metal-stack API. Update the `CLUSTER_ID` and `DEFAULT_PROJECT_ID` environment variables with your actual values. -::: +The job follows the same pattern as the [control-plane workflow](./03_control-plane.mdx#github-action) — only the playbook differs. If you host Gardener on a **dedicated** runtime cluster (the recommended [Option 2](./02_bootstrap-infrastructure.md#option-2-dedicated-initial-clusters)), point `CLUSTER_ID` at that cluster instead of the metal-stack control plane cluster. ## How the Pieces Connect @@ -662,34 +672,27 @@ graph LR MA -->|machines, networks, IPs, firewalls| PART["Partition"] ``` -| Connection | Configuration Variable | Purpose | -| ---------- | ---------------------- | ------- | -| metal-api URL | `gardener_cloud_profile_metal_api_url` | Endpoint the provider extension calls to provision resources | -| Control plane key | `gardener_cloud_profile_stage_name` | Key under `metalControlPlanes` that shoots resolve to the metal-api | -| Machine images | `metal_api_images` → `gardener_cloud_profile_machine_images` / `gardener_extension_provider_metal_machine_images` | Keeps CloudProfile and extension in sync | -| HMAC secret | `gardener_shoot_default_metal_api_hmac` | Shoot-to-metal-api authentication via the per-shoot provider secret | -| Machine types | `gardener_cloud_profile_machine_types` | Machine sizes selectable for workers and firewalls | -| Regions/zones | `gardener_cloud_profile_regions` | Region and zone (= partition) placement | -| Partition config | `gardener_cloud_profile_partitions` | Firewall types and network isolation per partition | -| Shoot networks | `gardener_shoots[].networks` | metal-stack network IDs the firewall attaches to | -| Shoot partition | `gardener_shoots[].partition` | Which metal-stack partition hosts the workers | +| Connection | Configuration Variable | Purpose | +| ----------------- | ----------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------- | +| metal-api URL | `gardener_cloud_profile_metal_api_url` | Endpoint the provider extension calls to provision resources | +| Control plane key | `gardener_cloud_profile_stage_name` | Key under `metalControlPlanes` that shoots resolve to the metal-api | +| Machine images | `metal_api_images` → `gardener_cloud_profile_machine_images` / `gardener_extension_provider_metal_machine_images` | Keeps CloudProfile and extension in sync | +| HMAC secret | `gardener_shoot_default_metal_api_hmac` | Shoot-to-metal-api authentication via the per-shoot provider secret | +| Machine types | `gardener_cloud_profile_machine_types` | Machine sizes selectable for workers and firewalls | +| Regions/zones | `gardener_cloud_profile_regions` | Region and zone (= partition) placement | +| Partition config | `gardener_cloud_profile_partitions` | Firewall types and network isolation per partition | +| Shoot networks | `gardener_shoots[].networks` | metal-stack network IDs the firewall attaches to | +| Shoot partition | `gardener_shoots[].partition` | Which metal-stack partition hosts the workers | ## Troubleshooting -| Symptom | Likely cause | -| ------- | ------------ | +| Symptom | Likely cause | +| ------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `not all mandatory variables given, check role documentation` | An asserted variable is unset. Check `metal_control_plane_host_provider`, `gardener_operator_backup_infrastructure.provider`, `gardener_operator_ingress_dns_domain`, `gardener_cloud_profile_kubernetes`, `gardener_cloud_profile_regions`, the three `gardener_gardenlet_default_dns_*` variables, `gardener_managed_seed_default_dns_domain` and `gardener_extension_shoot_cert_service_issuer_email`. | -| The role fails with `the variable ... was renamed to ...` | You are using a deprecated variable name; rename it as instructed and remove the old one. | -| `virtual_garden_kubeconfig` times out (120 retries) | The `Garden` is not healthy yet, or `gardener-virtual-garden-access` has not run. Check `kubectl get garden` and the operator logs in the `garden` namespace. | -| `KeyError: 'nodeNetwork'` in the operator role | The runtime cluster has no `kube-system/shoot-info` ConfigMap. Either set `metal_control_plane_host_provider: gcp` or provide the ConfigMap yourself. | -| etcd of the virtual Garden does not reconcile | Add the `druid.gardener.cloud/etcd-druid` finalizer on the `ETCD` resource manually, as noted in the role README. | -| Shoots stay pending | No Seed tolerates them. Check the Seed taints, `visible` setting, and the project's `protected_toleration`. | +| The role fails with `the variable ... was renamed to ...` | You are using a deprecated variable name; rename it as instructed and remove the old one. | +| `virtual_garden_kubeconfig` times out (120 retries) | The `Garden` is not healthy yet, or `gardener-virtual-garden-access` has not run. Check `kubectl get garden` and the operator logs in the `garden` namespace. | +| `KeyError: 'nodeNetwork'` in the operator role | The runtime cluster has no `kube-system/shoot-info` ConfigMap. Either set `metal_control_plane_host_provider: gcp` or provide the ConfigMap yourself. | +| etcd of the virtual Garden does not reconcile | Add the `druid.gardener.cloud/etcd-druid` finalizer on the `ETCD` resource manually, as noted in the role README. | +| Shoots stay pending | No Seed tolerates them. Check the Seed taints, `visible` setting, and the project's `protected_toleration`. | For general deployment issues, see the [troubleshooting guide](../06-troubleshoot.md). - -## Next Steps - -- **[GPU Workers](./07-gpu-workers.md)** — Offer GPU worker groups in shoots -- **[Offline Resilience](./08_offline-resilience.md)** — Operating the landscape without upstream connectivity -- **[Gardener Concepts](../../05-Concepts/04-Kubernetes/02-gardener.md)** — Architecture, operational model and failure domains -- **[Upstream Gardener docs](https://gardener.cloud/docs/)** — API reference and component documentation diff --git a/docs/04-For Operators/03-Deployment/06_cluster-api.md b/docs/04-For Operators/03-Deployment/06_cluster-api.md index 495a3a4b..b8318c45 100644 --- a/docs/04-For Operators/03-Deployment/06_cluster-api.md +++ b/docs/04-For Operators/03-Deployment/06_cluster-api.md @@ -8,12 +8,16 @@ sidebar_position: 6 This section covers deploying [Cluster API](../../05-Concepts/04-Kubernetes/03-cluster-api.md) on top of your existing metal-stack infrastructure. Cluster API with metal-stack is a declarative cluster management framework — you declare the desired state of your clusters and the Cluster API controllers reconcile the actual state. -This guide does **not** use Ansible roles. Unlike Gardener, which is deployed via the `gardener-*` metal-roles, Cluster API with metal-stack is set up directly through `clusterctl` and the [cluster-api-provider-metal-stack (CAPMS)](https://github.com/metal-stack/cluster-api-provider-metal-stack/) infrastructure provider. +This guide does **not** use Ansible roles. Unlike Gardener, which is deployed via the `gardener-*` [metal-roles](https://github.com/metal-stack/metal-roles/tree/master/control-plane), Cluster API with metal-stack is set up directly through `clusterctl` and the [cluster-api-provider-metal-stack (CAPMS)](https://github.com/metal-stack/cluster-api-provider-metal-stack/) infrastructure provider. Consequently, the [Control Plane](./03_control-plane.mdx) and [Partition](./04_partition.md) guides still apply unchanged — only this KCLM layer differs. :::warning[Beta] Cluster API with metal-stack is in beta and not yet recommended for production workloads. Please use [Gardener](./05_gardener.md) for production deployments. We are actively looking for exchange and adopters — if you are interested in using Cluster API with metal-stack, please [join our community](/community) to help shape future integration efforts. ::: +:::info +Because Cluster API is a framework rather than a platform, this page is deliberately shorter than the [Gardener guide](./05_gardener.md): there is far less to configure, but substantially more day-2 tooling to build yourself. The [Cluster API concepts page](../../05-Concepts/04-Kubernetes/03-cluster-api.md#what-to-build-yourself) lists exactly what that entails, and the [KCLM decision matrix](../../05-Concepts/04-Kubernetes/01-kclm.md#decision-matrix) puts it into perspective. +::: + ## Prerequisites Before deploying Cluster API, ensure the following infrastructure is in place: @@ -39,73 +43,58 @@ Cluster API with metal-stack is deployed through the [cluster-api-provider-metal **Network integration** -Network integration for Cluster API is currently more manual compared to Gardener. Node networks must be created manually via `metalctl` and provided as environment variables. IP addresses for the control plane also need to be allocated in advance through `metalctl`. Firewall rules are currently static and can be applied to firewall nodes; no automatic firewall controller is in place yet. Automatic network resource allocation is on the roadmap for CAPMS. +Network integration for Cluster API is more manual than for Gardener, and this is the part that surprises most operators coming from the Gardener path: + +- Node networks must be created up front via `metalctl network allocate` and handed to `clusterctl` as environment variables. +- Control plane IPs must be acquired in advance with `metalctl network ip create`. +- Firewall rules are static — there is no [firewall-controller-manager](../../05-Concepts/04-Kubernetes/05-firewall-controller-manager.md) integration yet, so no `ClusterWideNetworkPolicy` reconciliation. +- Service exposure uses KubeVIP in BGP mode instead of MetalLB. -For service exposure, CAPMS uses KubeVIP in BGP mode to allocate and announce public IPs, similar to the MetalLB-based approach in Gardener. +Automatic network resource allocation is on the CAPMS roadmap. Until then, plan these resources as part of your GitOps repository so they do not drift. **Air-gapped environments** -For air-gapped deployments, follow the [Cluster API Operator air-gapped environment guide](https://cluster-api-operator.sigs.k8s.io/topics/configuration/air-gapped-environtment). All required images must be mirrored to an OCI registry reachable from the management cluster. +For air-gapped deployments, follow the [Cluster API Operator air-gapped environment guide](https://cluster-api-operator.sigs.k8s.io/topics/configuration/air-gapped-environtment). All required images must be mirrored to an OCI registry reachable from the management cluster. On the metal-stack side, combine this with [Offline Resilience](./08_offline-resilience.md) so that machine provisioning itself does not depend on the internet. ## Kamaji as Control Plane Provider -[Kamaji](https://kamaji.clastix.io/) is a Control Plane Manager for Kubernetes that runs control planes as pods within a management cluster, reducing operational overhead and costs. It integrates with Cluster API as a `ControlPlaneProvider`. +[Kamaji](https://kamaji.clastix.io/) is a Control Plane Manager for Kubernetes that runs tenant control planes as pods inside the management cluster instead of on dedicated worker nodes. It plugs into Cluster API as a `ControlPlaneProvider`, while CAPMS remains the `InfrastructureProvider`. Conceptually this gets you close to Gardener's control plane hosting model — see [Cluster API + Kamaji](../../05-Concepts/04-Kubernetes/01-kclm.md#cluster-api--kamaji-the-middle-ground) for the trade-offs. :::warning -Kamaji integrations with metal-stack have not been evaluated in production-grade scenarios. This is a lab-only showcase. +Kamaji integrations with metal-stack **have not been evaluated in production-grade scenarios** by metal-stack. We are actively looking for exchange and adopters — if you are interested in using Kamaji with metal-stack, please [join our community](/community) to help shape future integration efforts. ::: -Kamaji acts as a `ControlPlaneProvider` with Cluster API, while CAPMS acts as the `InfrastructureProvider`. This setup manages **tenant clusters** on metal-stack infrastructure, combining Kamaji's control plane management with metal-stack's bare-metal provisioning. - **Deployment** 1. **Prepare management cluster** — A Kubernetes cluster to host Kamaji and CAPMS providers 2. **Install Kamaji and CAPMS** — Deploy both providers into the management cluster 3. **Create a control plane VIP** — MetalLB assigns a virtual IP for the tenant API server -4. **Generate and apply tenant cluster manifest** — Use `clusterctl generate cluster` to produce a YAML with `Cluster`, `MetalStackCluster`, `KubeadmControlPlane`, `MachineDeployment`, and `MetalStackMachine` resources, then apply it +4. **Generate and apply the tenant cluster manifest** — Use `clusterctl generate cluster --flavor kamaji-tenant`, then apply the result 5. **Deploy add-ons** — Install CNI (Calico) and `metal-ccm` into the tenant cluster A working showcase is available in the [`capi-lab`](https://github.com/metal-stack/cluster-api-provider-metal-stack/blob/main/DEVELOPMENT.md#running-the-kamaji-flavor) setup, which extends the `mini-lab` with a Kamaji flavor. See our [blog post](/blog/2026/04-kamaji) for a detailed walkthrough of the architecture and setup. -**Fleet management and GitOps** - -Since Kamaji with metal-stack uses Cluster API under the hood, fleet management follows the same pattern as Cluster API. Tenant cluster manifests are generated via `clusterctl`, stored in Git, and deployed through your CI/CD pipeline. +Because Kamaji sits underneath Cluster API, everything in the following section applies unchanged. ## Fleet Management and GitOps -You must set up your own Git repository and GitOps operator to manage cluster deployments. +Cluster API ships no fleet management of its own — you set up the Git repository and GitOps operator yourself: -**What you need to build:** +1. **Git repository** — Store the manifests generated via `clusterctl generate cluster `. Each cluster gets its own set of YAML files containing `Cluster`, `MetalStackCluster`, `KubeadmControlPlane`, `MachineDeployment` and `MetalStackMachine` resources, plus the pre-allocated networks and IPs. +2. **GitOps operator** — Deploy Argo CD or Flux to watch the repository and apply manifests to the management cluster, ensuring drift-free declarative delivery. +3. **Per-cluster CI/CD** — Essential components (CNI, CCM) are rolled out per cluster. Changes to a `MetalStackMachineTemplate` or `ClusterResourceSet` are staged through the repository with the usual approval process — they are **not** rolled out fleet-wide automatically as they would be with Gardener. -1. **Git repository** — Store cluster manifests generated via `clusterctl generate cluster `. Each cluster gets its own set of YAML files containing `Cluster`, `MetalStackCluster`, `KubeadmControlPlane`, `MachineDeployment`, and `MetalStackMachine` resources. -2. **GitOps operator** — Deploy ArgoCD or FluxCD to watch your Git repository and apply manifests to the management cluster, ensuring drift-free declarative delivery. -3. **Per-cluster CI/CD** — Essential components (CNI, CCM) are rolled out on a per-cluster basis. Changes to `MachineTemplate` or `ClusterResourceSet` are staged through the Git repository with standard approval processes. +Use multi-stage environments (staging → production) and validate manifests in CI to keep the risk of platform changes manageable. -**Platform capabilities:** - -- **Cluster migration** — `clusterctl move` enables moving workload cluster resources between management clusters, pausing controllers during the move to prevent worker node loss -- **Emergency patching** — Achieved through editing resources in the management cluster, e.g., update of machine OS image in the `MachineTemplate` or update of `ClusterResourceSet`. Unlike Gardener, this change is not rolled out fleet-wide automatically and should be staged through the Git repository with standard approval processes -- **Certificate rotation** — No direct workflow is described for certificate rotation at the landscape level; this is to be defined by platform administrators using manual/custom processes based on standard tooling (e.g., `kubeadm` certificate renewal) -- **Audit configuration** — Audit configuration can be passed to the kube-apiserver via the `kubeadmConfigSpec` of the `KubeadmControlPlane` resource before cluster creation. Each cluster can have its own audit policy. The management cluster's kube-apiserver audit also needs to be configured separately. Cluster API does not provide a centralized audit management toolset. +For the day-2 capabilities this model does and does not give you — cluster migration via `clusterctl move`, emergency patching, certificate rotation, audit configuration — see [Cluster API concepts](../../05-Concepts/04-Kubernetes/03-cluster-api.md#fleet-management-and-gitops). ## Available Flavors CAPMS provides different [cluster template flavors](https://cluster-api.sigs.k8s.io/cluster-api/commands/generate-cluster.html#flavors) for `clusterctl generate cluster`: -| Flavor | Description | K8s Compatibility | -|--------|-------------|-------------------| -| *(default)* | Expects the user to deploy a CNI and a CCM manually | >= v1.33 | -| `calico` | Installs Calico CNI + metal-ccm via `ClusterResourceSet` and CAAPH | >= v1.33 | -| `pre-v1.33` | Same as default but for Kubernetes versions < v1.33 | < v1.33 | -| `kamaji-tenant` | Kamaji tenant cluster template (requires Kamaji installed) | >= v1.33 | - -## Next Steps - -- **[KCLM Overview](./05_kclm.md)** — Introduction to Kubernetes Cluster Lifecycle Management with metal-stack -- **[Gardener Deployment Guide](./05_gardener.md)** — metal-stack's recommended, production-ready KCLM solution -- **[Cluster API Concepts](../../05-Concepts/04-Kubernetes/03-cluster-api.md)** — Architecture, operational model, and control plane hosting -- **[CAPMS Reference](../../08-References/Kubernetes/cluster-api-provider-metal-stack/cluster-api-provider-metal-stack.md)** — Full CAPMS documentation -- **[CAPMS Development Guide](../../08-References/Kubernetes/cluster-api-provider-metal-stack/DEVELOPMENT.md)** — Local development with capi-lab -- **[Kamaji Blog Post](/blog/2026/04-kamaji)** — Architecture and setup walkthrough - - +| Flavor | Description | K8s Compatibility | +| --------------- | ------------------------------------------------------------------ | ----------------- | +| _(default)_ | Expects the user to deploy a CNI and a CCM manually | >= v1.33 | +| `calico` | Installs Calico CNI + metal-ccm via `ClusterResourceSet` and CAAPH | >= v1.33 | +| `pre-v1.33` | Same as default but for Kubernetes versions < v1.33 | < v1.33 | +| `kamaji-tenant` | Kamaji tenant cluster template (requires Kamaji installed) | >= v1.33 | diff --git a/docs/04-For Operators/03-Deployment/07-gpu-workers.md b/docs/04-For Operators/03-Deployment/07-gpu-workers.md index 2658274f..0e6a2c9c 100644 --- a/docs/04-For Operators/03-Deployment/07-gpu-workers.md +++ b/docs/04-For Operators/03-Deployment/07-gpu-workers.md @@ -6,13 +6,23 @@ sidebar_position: 7 # GPU Workers -For workloads which require the assistance of GPUs, support for GPUs in bare metal servers was added to metal-stack.io v0.18.0. +For workloads that require the assistance of GPUs, support for GPUs in bare metal servers was added in metal-stack v0.18.0. + +## Prerequisites + +Offering GPU worker nodes is a cooperation between the operator and the cluster owner. As an **operator**, you have to make GPU machines and the matching operating system available: + +1. **Register machines with GPUs** — The machine sizes must be known to the metal-api and be reported with their GPUs. Check with `metalctl size ls` and `metalctl machine ls`. +2. **Provide an `nvidia` image** — Add one of the `debian-nvidia` images from [metal-images](../../08-References/Deployment/metal-images/metal-images.md) to `metal_api_images` as described in [Providing Images](./03_control-plane.mdx#providing-images). +3. **Offer it through your KCLM** — With Gardener, keep `nvidia` in `gardener_extension_os_metal_types` (it is part of the default) _and_ add an `nvidia` entry to `gardener_cloud_profile_os_cri_mapping`. The mapping only covers `ubuntu` and `debian` by default, so without that entry the image is silently dropped from the `CloudProfile` and the size cannot be selected. See [extensions.yaml](./05_gardener.md#extensionsyaml--provider-and-shoot-extensions) and [cloud_profile.yaml](./05_gardener.md#cloud_profileyaml--defining-your-metal-stack-infrastructure). + +Everything below is then done by the **cluster owner** inside their own cluster. ## GPU Operator installation -With the nvidia image a worker has basic GPU support. This means that the required kernel driver, the containerd shim and the required containerd configuration are already installed and configured. +With the `nvidia` image a worker has basic GPU support. This means that the required kernel driver, the containerd shim and the required containerd configuration are already installed and configured. -To enable `Pods` that require GPU support to be scheduled on a worker node with a GPU, a `gpu-operator' must be installed. +To enable `Pods` that require GPU support to be scheduled on a worker node with a GPU, a `gpu-operator` must be installed. This has to be done by the cluster owner after the cluster is up and running. The simplest way to install this operator is as follows: @@ -33,7 +43,9 @@ helm install --wait \ --set toolkit.enabled=true ``` -After that `kubectl describe node` must show the gpu in the capacity like so: +Note `driver.enabled=false`: the NVIDIA kernel driver already ships with the `nvidia` machine image, so the operator must not attempt to install its own. + +After that, `kubectl describe node` must show the GPU in the node capacity like so: ```plain ... @@ -51,14 +63,18 @@ Capacity: With this basic installation, the worker node is ready to process GPU workloads. :::warning -However, there is a caveat - only one 'Pod' can access the GPU. If this is all you need, no additional configuration is required. -On the other hand, if you are planning to deploy multiple applications that require GPU support, and there are not that many GPUs available, you will need to configure the `gpu-operator` to allow the GPU to be shared between multiple `Pods`. +There is a caveat: with this basic configuration only **one** `Pod` can access the GPU. If that is all you need, no additional configuration is required. +If you plan to deploy multiple applications that require GPU support and there are not that many GPUs available, you have to configure the `gpu-operator` so that a GPU can be shared between multiple `Pods`. ::: -There are several approaches to sharing GPUs, please consult the official Nvidia documentation for further reference. +There are several approaches to sharing GPUs — time-slicing, MPS and MIG. Please consult the official NVIDIA documentation for further reference: + +- [Improving GPU utilization in Kubernetes](https://developer.nvidia.com/blog/improving-gpu-utilization-in-kubernetes) +- [GPU Operator with MIG](https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/gpu-operator-mig.html) +- [GPU sharing](https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/gpu-sharing.html) -- [https://developer.nvidia.com/blog/improving-gpu-utilization-in-kubernetes](https://developer.nvidia.com/blog/improving-gpu-utilization-in-kubernetes) -- [https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/gpu-operator-mig.html](https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/gpu-operator-mig.html) -- [https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/gpu-sharing.html](https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/gpu-sharing.html) +:::info +GPU machines are only offered in the partitions where the corresponding hardware is racked. Model this through your `CloudProfile` regions and zones (zones map to metal-stack partitions), so that end-users can only request GPU worker groups where GPUs actually exist. See [cloud_profile.yaml](./05_gardener.md#cloud_profileyaml--defining-your-metal-stack-infrastructure). +::: With this, happy AI processing. diff --git a/docs/04-For Operators/03-Deployment/08_offline-resilience.md b/docs/04-For Operators/03-Deployment/08_offline-resilience.md index 40b5a440..42b09088 100644 --- a/docs/04-For Operators/03-Deployment/08_offline-resilience.md +++ b/docs/04-For Operators/03-Deployment/08_offline-resilience.md @@ -6,27 +6,47 @@ sidebar_position: 8 # Offline Resilience -It is possible to use metal-stack without any external network dependencies by integrating your own DNS and NTP configuration into the stack. This feature is great for workloads requiring strong independence and reliability. Even in case of an internet connection failure, your infrastructure remains operational. Existing machines do not encounter any downtime as well as new machines can be provisioned. All you need to have in place is a DNS and NTP server configured and accessible for metal-stack. +It is possible to operate metal-stack without any external network dependencies by integrating your own DNS and NTP configuration into the stack. This is essential for workloads requiring strong independence and reliability: even if the internet uplink fails, your infrastructure remains operational — existing machines suffer no downtime, and new machines can still be provisioned. All you need in place is a DNS and an NTP server reachable from within the partition. -NTP servers need to be configured on the pixiecore and the metal-hammer microservices. This can be achieved by providing a list of NTP servers with the following Ansible variable through metal-roles: +Offline resilience rests on three independent pillars. Configure all three; each one on its own leaves a gap. + +## 1. Time — NTP for Provisioning + +NTP servers need to be configured for [pixiecore](https://github.com/metal-stack/metal-roles/tree/master/partition/roles/pixiecore) and [metal-hammer](https://github.com/metal-stack/metal-hammer). This is achieved by providing a list of NTP servers through the following Ansible variable of the `pixiecore` role, deployed as part of the [partition](./04_partition.md#management-server-deploy_mgmt_serversyaml): ```yaml -pixiecore_metal_hammer_ntp_servers: [] +pixiecore_metal_hammer_ntp_servers: + - ntp1.your-datacenter.example + - ntp2.your-datacenter.example ``` -In the background, the pixiecore is taking the NTP servers and passing it via the `MetalConfig` to the metal-hammer. When booting bare-metal servers, the metal-hammer needs to configure NTP servers. It recognises the ones from the `MetalConfig` and configures itself accordingly. -If no NTP servers are passed along, the following standard servers are used: +Under the hood, pixiecore takes the NTP servers and passes them via the `MetalConfig` to the metal-hammer. When booting bare-metal servers, the metal-hammer picks up the servers from the `MetalConfig` and configures itself accordingly. +If no NTP servers are passed along, the following public defaults are used — which is precisely what you want to avoid in an offline-resilient setup: + +- `0.de.pool.ntp.org` +- `1.de.pool.ntp.org` +- `2.de.pool.ntp.org` + +## 2. Time and Name Resolution — DNS and NTP for Machines + +Beyond provisioning, the running machine and firewall images need to be configured with your custom DNS and NTP servers. This is done via the `dnsservers` and `ntpservers` fields of the machine or firewall allocation request. + +Defaults can be configured **per partition** and are then applied to all machines and firewalls in that partition, unless the allocation request overrides them. Both `metalctl partition create` and `metalctl machine create` / `metalctl firewall create` accept the corresponding `--dnsservers` and `--ntpservers` flags. + +Setting the partition defaults is the recommended approach: it keeps every machine consistent without requiring every consumer — including your [KCLM](../../05-Concepts/04-Kubernetes/01-kclm.md) — to know about it. + +## 3. Artifacts — The Image Cache -- 0.de.pool.ntp.org -- 1.de.pool.ntp.org -- 2.de.pool.ntp.org +Machine provisioning pulls the operating system image, the kernel and the metal-hammer from the [global image store](https://images.metal-stack.io). To become independent of it, deploy the [`image-cache` role](https://github.com/metal-stack/metal-roles/tree/master/partition/roles/image-cache) on your management servers. It runs [metal-image-cache-sync](https://github.com/metal-stack/metal-image-cache-sync), which mirrors the images configured in the metal-api — `metal-images`, `metal-kernel` and `metal-hammer` — into the local file system, and falls back to the global image store only on cache misses. -Moreover, machine and firewall images need to be configured with your custom DNS and NTP servers. The customisation can be made via the fields `ntp_servers` an `dns_servers` and specifying a list of servers in the creation request for the machine or firewall. +:::warning +For a genuinely offline-capable partition, the fallback to the internet must never be needed. Verify after every image update that the cache holds all images referenced by `metal_api_images` **before** you cut the uplink, and give the cache enough disk space to hold every image version still in use. +::: -Within a partition default values for DNS and NTP servers can be configured. They are applied to all machines and firewalls within this partition, but can be replaced by specifying different ones inside the machine allocation request. +## Kubernetes Workloads -Thus, for creating a partition as well as a machine or a firewall, the flags `dnsservers` and `ntpservers` can be provided within the `metalctl` command. +The three pillars above make **metal-stack** offline resilient. Kubernetes clusters on top have their own dependency on container registries. For clusters that must be fully decoupled from the internet, combine this chapter with [Isolated Kubernetes Clusters](../../05-Concepts/04-Kubernetes/06-isolated-clusters.md), which mirrors all strictly required container images into a private registry and enforces the network restrictions with `ClusterWideNetworkPolicy` resources. -In order to be fully offline resilient, make sure to check out `metal-image-cache-sync`. This component provides copies of `metal-images`, `metal-kernel` and `metal-hammer`. +## Background -This feature is related to [MEP14](/community/MEP-14-independence-from-external-sources). +This feature set is based on [MEP-14 — Independence from External Sources](/community/MEP-14-independence-from-external-sources). diff --git a/docs/05-Concepts/04-Kubernetes/01-kclm.md b/docs/05-Concepts/04-Kubernetes/01-kclm.md index df0d8ae5..5edec6de 100644 --- a/docs/05-Concepts/04-Kubernetes/01-kclm.md +++ b/docs/05-Concepts/04-Kubernetes/01-kclm.md @@ -68,18 +68,18 @@ Platform administrators, meanwhile, focus on **providing the platform** — mana metal-stack provides KCLM through two integration paths, both consuming the same metal-stack API for bare-metal node provisioning: -| Aspect | [Gardener](./02-gardener.md) | [Cluster API](./03-cluster-api.md) | -| ------------------------------- | ---------------------------------------------------------------------- | --------------------------------------------------------------------------- | -| **Status** | Recommended, production-ready | Beta, under active development | -| **Governance** | NeoNephos Foundation | CNCF (Kubernetes SIG) | -| **Experience** | 7+ years in financial-sector production | CNCF project, metal-stack integration in development | -| **Day-2 capabilities** | Native (DNS, backup, audit, certificate rotation, maintenance windows) | Assembled through GitOps and add-on providers | -| **Control plane hosting** | Dedicated namespaces on Seed clusters (physically isolated) | On worker nodes (CABPK) or dedicated (Kamaji, unevaluated) | -| **Operational model** | End-user self-service via Virtual Garden API | Administrator-managed GitOps workflows | -| **Complexity** | More opinionated, higher initial setup | Less opinionated, lower initial setup but higher ongoing administration | -| **Maintenance windows** | Built-in, per-cluster configurable | Continuous reconciliation, no built-in windows | -| **Control plane resource cost** | Shared on Seed clusters (efficient) | 3 dedicated nodes per cluster (wasteful) or shared via Kamaji (unevaluated) | -| **Use case** | Production fleets, regulated environments, multi-tenant platforms | Simple deployments, teams willing to build day-2 operations | +| Aspect | [Gardener](./02-gardener.md) | [Cluster API](./03-cluster-api.md) | +| ------------------------------- | ---------------------------------------------------------------------- | --------------------------------------------------------------------------------------- | +| **Status** | Recommended, production-ready | Beta, under active development | +| **Governance** | NeoNephos Foundation | CNCF (Kubernetes SIG) | +| **Experience** | 7+ years in financial-sector production | CNCF project, metal-stack integration in development | +| **Day-2 capabilities** | Native (DNS, backup, audit, certificate rotation, maintenance windows) | Assembled through GitOps and add-on providers | +| **Control plane hosting** | Dedicated namespaces on Seed clusters (physically isolated) | On worker nodes (CABPK) or dedicated (Kamaji, integration unevaluated) | +| **Operational model** | End-user self-service via Virtual Garden API | Administrator-managed GitOps workflows | +| **Complexity** | More opinionated, higher initial setup | Less opinionated, lower initial setup but higher ongoing administration | +| **Maintenance windows** | Built-in, per-cluster configurable | Continuous reconciliation, no built-in windows | +| **Control plane resource cost** | Shared on Seed clusters (efficient) | 3 dedicated nodes per cluster (wasteful) or shared via Kamaji (integration unevaluated) | +| **Use case** | Production fleets, regulated environments, multi-tenant platforms | Simple deployments, teams willing to build day-2 operations | Both converge on vanilla Kubernetes and metal-stack infrastructure, ensuring replaceability and vendor independence. @@ -151,7 +151,7 @@ Kamaji acts as a Control Plane Manager for Cluster API, running tenant control p **Administration burden**: Still high ongoing administration — Kamaji solves the control plane hosting problem but not the day-2 operations problem. You still assemble and maintain all day-2 tooling through GitOps workflows. -**Status**: Kamaji with metal-stack has not been evaluated in production-grade scenarios. It is a promising approach for resource efficiency but carries higher risk for production workloads. +**Status**: Kamaji integrations with metal-stack have not been evaluated in production-grade scenarios by metal-stack. Kamaji itself is used in production elsewhere; it is the metal-stack integration that lacks our validation. It is a promising approach for resource efficiency but carries higher risk for production workloads. ### Decision Matrix @@ -176,7 +176,7 @@ Kamaji acts as a Control Plane Manager for Cluster API, running tenant control p :::tip[Recommendation] For production workloads, regulated environments, or any scenario where you need multi-tenant self-service with minimal ongoing administration, **Gardener is the clear choice**. The platform approach eliminates the need to build and maintain day-2 operations tooling, enforces version skew policies, and provides physical isolation between control planes and workloads. -Cluster API with CABPK is only recommended for simple, single-cluster deployments where you accept the resource waste of dedicated control plane nodes and the ongoing administration burden of building your own day-2 operations. Cluster API with Kamaji offers better resource efficiency but carries higher risk as an unevaluated integration. +Cluster API with CABPK is only recommended for simple, single-cluster deployments where you accept the resource waste of dedicated control plane nodes and the ongoing administration burden of building your own day-2 operations. Cluster API with Kamaji offers better resource efficiency but carries higher risk, as we have not evaluated that integration in production-grade scenarios. ::: ## Core Concepts @@ -282,7 +282,7 @@ KCLM supports multiple control plane topologies for on-prem failure domains. The | **Single-site HA** | Multiple control plane nodes across machines in the same Seed with etcd spread for quorum. Default production-confirmed choice. | Multiple control plane Machines within a single partition with etcd replicas on separate Machines. Natively supported. | | **Multi-rack** | Shoot control plane nodes spread across multiple racks within one Seed with etcd spread across racks. Rack-level failure isolation via MachineDeployment topology spread constraints. | Multi-failure-domain topology distributes control plane Machines across multiple zones or regions with etcd spread accordingly. Natively supported through ClusterClass topology definitions. | | **Multi-site** | Shoot control planes replicated across Seeds corresponding to different sites or data centers. MachineDeployments use zone constraints to distribute workers across regions. Higher latency for cross-seed communication requires multi-seed configuration. | Control plane Machines deployed across multiple CAPI management clusters or across widely separated MetalPools with cross-site etcd replication. Requires additional operator effort for cross-site networking. | -| **Dedicated isolation** | A Shoot gets its own dedicated Seed cluster with no shared control plane with other tenants. Highest compliance level for critical infrastructure at highest resource cost. | A dedicated Cluster with its own isolated MetalPool and exclusive use of MetalPools. Same isolation level as Gardener's dedicated Seed for critical infrastructure. | +| **Dedicated isolation** | A Shoot gets its own dedicated Seed cluster with no shared control plane with other tenants. Highest compliance level for critical infrastructure at highest resource cost. | A dedicated Cluster with its own isolated MetalPool and exclusive use of MetalPools. Same isolation level as Gardener's dedicated Seed for critical infrastructure. | Worker nodes are automatically distributed across racks using a rack-spreading algorithm, and well-known Kubernetes topology labels (`machine.metal-stack.io/rack`, `machine.metal-stack.io/chassis`, `topology.kubernetes.io/region`, `topology.kubernetes.io/zone`) are provided on nodes — enabling end-users to configure Pod topology spread and anti-affinity rules. With MEP-19 (metal-stack Enhancement Proposal 19), routing across data center partitions will also be supported, allowing worker nodes to reside in separate metal-stack partitions while maintaining a single Kubernetes cluster — provided the partitions are geographically close enough for stable low-latency connectivity. @@ -317,22 +317,16 @@ Kubernetes API audit policies are configurable per cluster, with logs forwarded ## Day-2 Operations, HA, and Advanced Topics -The following topics are covered in detail on the individual solution pages, as their implementation differs significantly between Gardener and Cluster API: +The following topics differ significantly between the two solutions and are therefore covered on the individual solution pages: -- **[Day-2 Operations](./02-gardener.md#operational-features)** — Configuration changes, certificate rotation, component upgrades, emergency patching, and audit & traceability (Gardener: built-in; Cluster API: assembled through GitOps) -- **[High Availability & Failure Domains](./02-gardener.md#failure-domains)** — Control plane topologies (single-site HA, multi-rack, multi-site, dedicated isolation), worker node distribution, rack-spreading, and failure recovery automation -- **[Add-on Lifecycle Management](./02-gardener.md#operational-features)** — CNI, CSI, CCM, DNS, and audit component management with compatibility matrices and fleet-wide consistency (Gardener: fleet-wide; Cluster API: per-cluster via ClusterResourceSet) -- **[Version Policy & Safe Transitions](./02-gardener.md#version-skew-policy)** — Kubernetes version skew enforcement, staged upgrades, and emergency patching workflows (Gardener: enforced by API server; Cluster API: risk management via approval gates) -- **[Blue-Green Update Strategy](./01-kclm.md#what-kclm-automates)** — Zero-downtime upgrade patterns using BGP Anycast or worker groups with taints and tolerations -- **[Cluster API Deep Dive](./03-cluster-api.md)** — Lifecycle coverage, add-on management, compatibility matrix, and platform capabilities - -:::note[MEP-19 — Cross-Partition Clusters] -With metal-stack Enhancement Proposal 19 (MEP-19), routing across data center partitions will be supported, allowing worker nodes to reside in separate metal-stack partitions while maintaining a single Kubernetes cluster. This requires partitions to be geographically close enough for stable low-latency connectivity. -::: - -:::note[Solution-Specific Details] -HA topologies, add-on management, and version policies differ significantly between Gardener and Cluster API. See the individual solution pages for implementation-specific details. -::: +| Topic | Gardener | Cluster API | +| -------------------- | ---------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------- | +| Day-2 operations | [Operational features](./02-gardener.md#operational-features) — built-in | [What to build yourself](./03-cluster-api.md#what-to-build-yourself) — assembled through GitOps | +| HA & failure domains | [Control plane topologies](./02-gardener.md#control-plane-topologies) | [Control plane topologies](./03-cluster-api.md#control-plane-topologies) | +| Add-on lifecycle | [Operational features](./02-gardener.md#operational-features) — fleet-wide | [Fleet management](./03-cluster-api.md#fleet-management-and-gitops) — per cluster via `ClusterResourceSet` | +| Version policy | [Version skew policy](./02-gardener.md#version-skew-policy) — enforced by the API server | [Operational model](./03-cluster-api.md#operational-model) — risk management via approval gates | +| Upgrades & rollback | [Upgrade & rollback](./02-gardener.md#upgrade--rollback) | [Upgrade & rollback](./03-cluster-api.md#upgrade--rollback) | +| Audit & traceability | [Audit & traceability](./02-gardener.md#audit--traceability) | [Fleet management](./03-cluster-api.md#fleet-management-and-gitops) | ## Network Integration @@ -368,12 +362,4 @@ The KCLM management layer is designed so that its absence does not impact cluste ### Proven at Scale -metal-stack with Gardener operates environments with **200+ Kubernetes clusters** across **5 data centers** and **1,800 physical servers** (including 200 servers in a metro environment across disjunct locations). Maximum nodes per cluster: 64 (up to 1,024 in metal-stack OnPrem). Proven Gardener installations manage **10,000+ clusters** — demonstrating that consistent, automated lifecycle management is the key to scaling bare-metal Kubernetes fleets. For Cluster API, the integration test environment covers a management cluster with three worker nodes, with the biggest test clusters including 8 cluster nodes. - -## Next Steps - -- **[Gardener](./02-gardener.md)** — Detailed concept documentation for metal-stack's recommended KCLM solution -- **[Cluster API](./03-cluster-api.md)** — Concept documentation for the CNCF-based alternative -- **[Cloud Controller Manager](./04-cloud-controller-manager.md)** — How metal-ccm bridges Kubernetes and bare-metal networking -- **[Firewall Controller Manager](./05-firewall-controller-manager.md)** — Firewall rule management via Kubernetes CRDs -- **[Deployment Guide](../04-For%20Operators/03-Deployment/01_guide.md)** — Deployment guide +metal-stack with Gardener operates environments with **280 Kubernetes clusters** across **5 data centers** and **1,800 physical servers** (including 200 servers in a metro environment across disjunct locations). The largest observed clusters hold 64 nodes; metal-stack itself supports up to 1,024 nodes per cluster. Proven Gardener installations elsewhere manage **10,000+ clusters** — demonstrating that consistent, automated lifecycle management is the key to scaling bare-metal Kubernetes fleets. For Cluster API, the integration test environment covers a management cluster with three worker nodes, with the biggest test clusters including 8 cluster nodes. diff --git a/docs/05-Concepts/04-Kubernetes/02-gardener.md b/docs/05-Concepts/04-Kubernetes/02-gardener.md index 151bf77d..8dec1118 100644 --- a/docs/05-Concepts/04-Kubernetes/02-gardener.md +++ b/docs/05-Concepts/04-Kubernetes/02-gardener.md @@ -10,7 +10,7 @@ sidebar_position: 2 Gardener is the **recommended** Kubernetes Cluster Lifecycle Management (KCLM) solution for metal-stack. It is battle-tested in production for over seven years at financial-sector customers and bundles more day-2 capabilities natively (DNS, backup, audit). Gardener manages entire clusters as Kubernetes-native resources with a strong separation between platform operators and end-users. -For deployment instructions, see the [Gardener deployment guide](../04-For%20Operators/03-Deployment/05_gardener.md). +This page describes **what** Gardener with metal-stack does and why. For **how** to deploy it, see the [Gardener deployment guide](../04-For%20Operators/03-Deployment/05_gardener.md); for the comparison with the alternative, see the [KCLM overview](./01-kclm.md#two-approaches-one-infrastructure). ## Outcomes @@ -32,43 +32,43 @@ The diagram below shows the full deployment architecture — from the bootstrap ### Core Components -| Component | Responsibility | -|-----------|---------------| -| **Garden cluster** | The top-level cluster that runs the Gardener control plane (API server, controller manager, scheduler, admission controller). Deployed via the `gardener-operator`. | +| Component | Responsibility | +| ------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| **Garden cluster** | The top-level cluster that runs the Gardener control plane (API server, controller manager, scheduler, admission controller). Deployed via the `gardener-operator`. | | **Virtual Garden** | A recommended deployment pattern where Gardener runs inside a virtual cluster on the Garden cluster. This provides a dedicated ETCD for Gardener resources and an independent update lifecycle from the Garden cluster itself. End users get project namespaces in the virtual garden. | -| **Seed cluster** | A cluster where a `gardenlet` agent runs. The gardenlet connects to the Gardener control plane and orchestrates provisioning of new clusters within that Seed. Typically one Seed per data-center site. A Seed that has been manually deployed (not by Gardener) is called a **soil**. | -| **Shoot cluster** | Every fully provisioned and managed Kubernetes cluster. The Shoot's control plane (kube-apiserver, etcd, controller-manager, scheduler) runs as pods in a dedicated namespace on a Seed, while worker nodes run on bare-metal machines provisioned via the metal-stack API. | +| **Seed cluster** | A cluster where a `gardenlet` agent runs. The gardenlet connects to the Gardener control plane and orchestrates provisioning of new clusters within that Seed. Typically one Seed per data-center site. A Seed that has been manually deployed (not by Gardener) is called a **soil**. | +| **Shoot cluster** | Every fully provisioned and managed Kubernetes cluster. The Shoot's control plane (kube-apiserver, etcd, controller-manager, scheduler) runs as pods in a dedicated namespace on a Seed, while worker nodes run on bare-metal machines provisioned via the metal-stack API. | ### Core Controllers -| Controller | Purpose | -|------------|---------| -| `gardener-operator` | Deploys Gardener components, gardenlets, and extensions; manages platform updates | -| `gardener-apiserver` | Extends the kube-apiserver with Gardener-specific resources (Shoot, Seed, Project, etc.) | -| `gardener-scheduler` | Decides where clusters are placed across the Gardener landscape (Seeds) | -| `gardener-controller-manager` | Reconciles common Gardener resources (projects, controller installations, etc.) | -| `gardenlet` | Agent running on each Seed; orchestrates provisioning of new clusters within that Seed | -| `gardener-resource-manager` | Runs inside Shoots; reconciles desired resources and checks their health | -| `etcd-druid` | etcd cluster operator with built-in backup-restore functionality | -| `machine-controller-manager` | Manages worker node lifecycle (rolling updates, health recreation, scaling) | +| Controller | Purpose | +| ----------------------------- | ---------------------------------------------------------------------------------------- | +| `gardener-operator` | Deploys Gardener components, gardenlets, and extensions; manages platform updates | +| `gardener-apiserver` | Extends the kube-apiserver with Gardener-specific resources (Shoot, Seed, Project, etc.) | +| `gardener-scheduler` | Decides where clusters are placed across the Gardener landscape (Seeds) | +| `gardener-controller-manager` | Reconciles common Gardener resources (projects, controller installations, etc.) | +| `gardenlet` | Agent running on each Seed; orchestrates provisioning of new clusters within that Seed | +| `gardener-resource-manager` | Runs inside Shoots; reconciles desired resources and checks their health | +| `etcd-druid` | etcd cluster operator with built-in backup-restore functionality | +| `machine-controller-manager` | Manages worker node lifecycle (rolling updates, health recreation, scaling) | ### Provider Extensions -| Extension | Purpose | -|-----------|---------| -| `gardener-extension-provider-metal` | Integrates metal-stack API with Gardener (machines, networks, firewalls, IPs). See also [Cloud Controller Manager](./04-cloud-controller-manager.md) and [Firewall Controller Manager](./05-firewall-controller-manager.md) for integration details. | -| `gardener-extension-audit` | Configures buffered forwarders to audit sinks (e.g., Splunk, S3) | -| `gardener-extension-networking-calico` | Provides Calico CNI in shoot clusters | -| `gardener-extension-networking-cilium` | Provides Cilium CNI in shoot clusters | -| `gardener-extension-shoot-dns-service` | Provides DNS records for the Kubernetes API server; allows shoot owners to create DNS records within their project domain | -| `gardener-extension-shoot-cert-service` | Provides certificates for services exposed in shoot clusters | +| Extension | Purpose | +| --------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `gardener-extension-provider-metal` | Integrates metal-stack API with Gardener (machines, networks, firewalls, IPs). See also [Cloud Controller Manager](./04-cloud-controller-manager.md) and [Firewall Controller Manager](./05-firewall-controller-manager.md) for integration details. | +| `gardener-extension-audit` | Configures buffered forwarders to audit sinks (e.g., Splunk, S3) | +| `gardener-extension-networking-calico` | Provides Calico CNI in shoot clusters | +| `gardener-extension-networking-cilium` | Provides Cilium CNI in shoot clusters | +| `gardener-extension-shoot-dns-service` | Provides DNS records for the Kubernetes API server; allows shoot owners to create DNS records within their project domain | +| `gardener-extension-shoot-cert-service` | Provides certificates for services exposed in shoot clusters | ### Integration Extensions -| Extension | Purpose | -|-----------|---------| -| `os-metal-extension` | Translates Gardener's generic `OperatingSystemConfig` format into cloud-init (or ignition) userdata for metal-stack nodes | -| `machine-controller-manager-provider-metal` | Integrates metal-stack machine provisioning API with Gardener's MCM as an out-of-tree sidecar | +| Extension | Purpose | +| ------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------- | +| `os-metal-extension` | Translates Gardener's generic `OperatingSystemConfig` format into cloud-init (or ignition) userdata for metal-stack nodes | +| `machine-controller-manager-provider-metal` | Integrates metal-stack machine provisioning API with Gardener's MCM as an out-of-tree sidecar | For a complete architecture overview with diagrams, see the [Gardener documentation](https://gardener.cloud/docs/getting-started/architecture/). @@ -83,7 +83,7 @@ sequenceDiagram participant MCM as machine-controller-
manager-provider-metal participant MS as metal-stack API participant NODE as Bare Metal Node - + G->>EXT: Creates Infrastructure
ControlPlane Worker resources EXT->>MS: Allocate machines, networks,
firewalls, IPs MS-->>EXT: Returns machine IDs, IPs @@ -117,9 +117,9 @@ The MCM handles the full worker node lifecycle: creating new machines, draining Gardener differentiates between **end-users** (shoot owners) and **platform administrators** (seed owners), similar to how hyperscalers offer Kubernetes as a Service: -| Role | Responsibilities | -|------|-----------------| -| **End-Users** | Create/manage clusters via API (limited to whitelisted machine types), manage worker groups, trigger cluster updates, configure maintenance windows and auto-updates | +| Role | Responsibilities | +| ------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | +| **End-Users** | Create/manage clusters via API (limited to whitelisted machine types), manage worker groups, trigger cluster updates, configure maintenance windows and auto-updates | | **Administrators** | Set up Seed clusters in data centers, whitelist machine types, provide Kubernetes versions and OS images, define lifecycle policies, fleet-wide GitOps-driven operations | End-users access the Virtual Garden through a feature-rich Kubernetes API with OIDC-based authorization. Resources like `ShootQuota` and custom webhooks can further restrict what end-users can do and ensure that they operate only within their specific, operational boundaries. As the API of Gardener is provided by Kubernetes itself, it is also an option to further narrow down the end-user scope by hiding the Gardener API behind a custom API. Platform administrators manage the entire platform through GitOps-driven processes with approval workflows. @@ -137,7 +137,7 @@ From a Kubernetes Cluster Lifecycle Management perspective, managing multiple fa While metal-stack distributes cluster worker nodes across racks automatically using a rack-spreading algorithm, information like region and zone comes from the end-user's requirements. With this information, Kubernetes features like Topology Spread and PodAntiAffinity can be configured easily by end-users. With MEP-19 (metal-stack Enhancement Proposal 19), routing across data center partitions will also be supported, allowing worker nodes to reside in separate metal-stack partitions while maintaining a single Kubernetes cluster — provided the partitions are geographically close enough for stable low-latency connectivity. -The KCLM is designed so that every failure domain can function without the KCLM itself — meaning the absence of the KCLM does not impact the availability of the Kubernetes clusters or the network traffic. Outages of the Gardener cluster only cause cluster provisioning to become unavailable, without interrupting workloads or preventing end-users from interacting with the Kubernetes API of their clusters. With MEP-19, worker nodes can also be spread across separate data center partitions while maintaining a single Kubernetes cluster, provided the partitions are geographically close enough for stable low-latency connectivity. +The KCLM is designed so that every failure domain can function without the KCLM itself — meaning the absence of the KCLM does not impact the availability of the Kubernetes clusters or the network traffic. Outages of the Gardener cluster only cause cluster provisioning to become unavailable, without interrupting workloads or preventing end-users from interacting with the Kubernetes API of their clusters. ## Control Plane Hosting @@ -149,12 +149,12 @@ There is a clear distinction between administrators, who are responsible for the Gardener supports multiple control plane topologies for on-prem failure domains: -| Topology | Description | Use Case | -|----------|-------------|----------| -| **Single-seed HA** | Multiple control plane nodes (kube-apiserver, controller-manager, scheduler) across machines in the same Seed with etcd spread across machines for quorum. Default production-confirmed choice. | Single-site deployments, standard production | -| **Multi-rack** | Shoot control plane nodes spread across multiple racks within one Seed with etcd spread across racks. Rack-level failure isolation via MachineDeployment topology spread constraints. | Rack-level failure isolation within a single data center | -| **Multi-site** | Shoot control planes replicated across Seeds corresponding to different sites or data centers. MachineDeployments use zone constraints to distribute workers across regions. Higher latency for cross-seed communication requires multi-seed configuration. | Disaster recovery across geographically separated sites | -| **Dedicated Seed** | A Shoot gets its own dedicated Seed cluster with no shared control plane with other tenants. Highest compliance level for critical infrastructure at highest resource cost. | Strictest compliance requirements for critical infrastructure | +| Topology | Description | Use Case | +| ------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------- | +| **Single-seed HA** | Multiple control plane nodes (kube-apiserver, controller-manager, scheduler) across machines in the same Seed with etcd spread across machines for quorum. Default production-confirmed choice. | Single-site deployments, standard production | +| **Multi-rack** | Shoot control plane nodes spread across multiple racks within one Seed with etcd spread across racks. Rack-level failure isolation via MachineDeployment topology spread constraints. | Rack-level failure isolation within a single data center | +| **Multi-site** | Shoot control planes replicated across Seeds corresponding to different sites or data centers. MachineDeployments use zone constraints to distribute workers across regions. Higher latency for cross-seed communication requires multi-seed configuration. | Disaster recovery across geographically separated sites | +| **Dedicated Seed** | A Shoot gets its own dedicated Seed cluster with no shared control plane with other tenants. Highest compliance level for critical infrastructure at highest resource cost. | Strictest compliance requirements for critical infrastructure | All topologies are natively supported. Multi-site requires additional multi-seed configuration but no special extensions. @@ -234,9 +234,3 @@ Kubernetes API audit policies are configurable per cluster, with logs forwarded - **Garden cluster** — The same audit extension can be configured for the gardener-apiserver and virtual kube-apiserver. For configuration examples, see the [Gardener deployment guide](../04-For%20Operators/03-Deployment/05_gardener.md). - -## Next Steps - -- **[KCLM Overview](./01-kclm.md)** — Introduction to Kubernetes Cluster Lifecycle Management with metal-stack -- **[Gardener Deployment Guide](../04-For%20Operators/03-Deployment/05_gardener.md)** — Step-by-step deployment instructions -- **[Gardener Documentation](https://gardener.cloud/docs/)** — Official Gardener documentation and API reference diff --git a/docs/05-Concepts/04-Kubernetes/03-cluster-api.md b/docs/05-Concepts/04-Kubernetes/03-cluster-api.md index dd6631e3..a9570ede 100644 --- a/docs/05-Concepts/04-Kubernetes/03-cluster-api.md +++ b/docs/05-Concepts/04-Kubernetes/03-cluster-api.md @@ -14,7 +14,7 @@ We provide the [Cluster API provider for metal-stack (CAPMS)](https://github.com Cluster API with metal-stack is in beta and not yet recommended for production workloads. Please use [Gardener](./02-gardener.md) for production deployments. We are actively looking for exchange and adopters — if you are interested in using Cluster API with metal-stack, please [join our community](/community) to help shape future integration efforts. ::: -For deployment instructions, see the [KCLM deployment guide](../04-For%20Operators/03-Deployment/05_kclm.md). +For deployment instructions, see the [Cluster API deployment guide](../04-For%20Operators/03-Deployment/06_cluster-api.md). ## Architecture @@ -28,18 +28,18 @@ graph TB CABPK["CABPK
Bootstrap provider"] CAAPH["CAAPH
Add-on Provider for Helm"] end - + subgraph "Workload Cluster" CP["KubeadmControlPlane
Control plane nodes"] MD["MachineDeployment
Worker nodes"] CRS["ClusterResourceSet
CNI + CCM add-ons"] end - + subgraph "metal-stack Infrastructure" MS["metal-stack API
Machines, networks, firewalls"] FW["Firewall nodes"] end - + CAPI --> CAPMS CAPI --> CABPK CAPI --> CAAPH @@ -50,7 +50,7 @@ graph TB MS -.->|provisions| CP MS -.->|provisions| MD MS -.->|provisions| FW - + style CAPI fill:#bbdefb style CAPMS fill:#c8e6c9 style CABPK fill:#fff9c4 @@ -60,15 +60,15 @@ graph TB ### Core Components -| Component | Responsibility | -|-----------|---------------| -| **Management Cluster** | A Kubernetes cluster that hosts the Cluster API controllers and the desired cluster state (as custom resources). It is the central control plane from which new workload clusters are declared and reconciled. | -| **Workload Cluster** | A Kubernetes cluster whose lifecycle is managed by the Management Cluster via CAPI resources. Its control plane and worker nodes are provisioned according to the declarative spec. | +| Component | Responsibility | +| --------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| **Management Cluster** | A Kubernetes cluster that hosts the Cluster API controllers and the desired cluster state (as custom resources). It is the central control plane from which new workload clusters are declared and reconciled. | +| **Workload Cluster** | A Kubernetes cluster whose lifecycle is managed by the Management Cluster via CAPI resources. Its control plane and worker nodes are provisioned according to the declarative spec. | | **Infrastructure Provider** | A set of controllers that translate CAPI's generic infrastructure resources (Cluster, Machine) into provider-specific resources. metal-stack is an officially listed infrastructure provider for Cluster API. The provider implementation is called cluster-api-provider-metal-stack (CAPMS). | -| **Bootstrap Provider** | Generates bootstrap data (e.g., cloud-init or ignition userdata) for joining new nodes. CAPMS is tested against the Kubeadm Bootstrap Provider (CABPK). | -| **ClusterResourceSet** | A CAPI feature for applying add-on resources (CNI, CCM, etc.) to newly created workload clusters automatically. CAPMS uses this with the Add-on Provider for Helm (CAAPH) for its calico flavor, to install calico as CNI and the [metal-ccm](./04-cloud-controller-manager.md). | -| **ClusterClass** | Defines reusable templates with parameterized variable schemas for tenant customization, enabling standardized cluster templates across the fleet. | -| **MachineHealthCheck** | Checks machine healthiness and takes remediation steps — unhealthy machines are automatically replaced by adding and deleting them on the infrastructure provider side, with safeguards in place (e.g., for not losing etcd quorum). | +| **Bootstrap Provider** | Generates bootstrap data (e.g., cloud-init or ignition userdata) for joining new nodes. CAPMS is tested against the Kubeadm Bootstrap Provider (CABPK). | +| **ClusterResourceSet** | A CAPI feature for applying add-on resources (CNI, CCM, etc.) to newly created workload clusters automatically. CAPMS uses this with the Add-on Provider for Helm (CAAPH) for its calico flavor, to install calico as CNI and the [metal-ccm](./04-cloud-controller-manager.md). | +| **ClusterClass** | Defines reusable templates with parameterized variable schemas for tenant customization, enabling standardized cluster templates across the fleet. | +| **MachineHealthCheck** | Checks machine healthiness and takes remediation steps — unhealthy machines are automatically replaced by adding and deleting them on the infrastructure provider side, with safeguards in place (e.g., for not losing etcd quorum). | For a complete architecture overview with diagrams, see the [Cluster API documentation](https://cluster-api.sigs.k8s.io/user/concepts#concepts). @@ -76,13 +76,13 @@ For a complete architecture overview with diagrams, see the [Cluster API documen CAPMS implements the CAPI infrastructure provider contract for bare metal via metal-stack. The following CRDs are provided by CAPMS: -| CRD | Purpose | -|-----|---------| -| `MetalStackCluster` | Infrastructure cluster resource — allocates a control plane virtual IP (VIP) | -| `MetalStackMachine` | Bridges CAPI infrastructure machines to metal-stack machines (bare metal servers) | -| `MetalStackMachineTemplate` | Defines reusable machine specs (image, size, etc.) for MetalStackMachine resources | -| `MetalStackFirewallDeployment` | Declares firewall deployments protecting a cluster's network perimeter | -| `MetalStackFirewallTemplate` | Provides the configuration template for deployed firewalls | +| CRD | Purpose | +| ------------------------------ | ---------------------------------------------------------------------------------- | +| `MetalStackCluster` | Infrastructure cluster resource — allocates a control plane virtual IP (VIP) | +| `MetalStackMachine` | Bridges CAPI infrastructure machines to metal-stack machines (bare metal servers) | +| `MetalStackMachineTemplate` | Defines reusable machine specs (image, size, etc.) for MetalStackMachine resources | +| `MetalStackFirewallDeployment` | Declares firewall deployments protecting a cluster's network perimeter | +| `MetalStackFirewallTemplate` | Provides the configuration template for deployed firewalls | ## Operational Model @@ -102,12 +102,12 @@ There are other providers from the CABPK ecosystem (e.g. [Kamaji](https://kamaji Cluster API supports multiple control plane topologies for on-prem failure domains: -| Topology | Description | Use Case | -|----------|-------------|----------| -| **Single-site HA** | Multiple control plane Machines within a single partition with etcd replicas on separate Machines. Natively supported. | Single-site deployments, standard production | -| **Multi-failure-domain** | Control plane Machines distributed across multiple zones or regions with etcd spread accordingly. Natively supported through ClusterClass topology definitions. | Rack/zone-level failure isolation | -| **Multi-site** | Control plane Machines deployed across multiple CAPI management clusters or across widely separated MetalPools with cross-site etcd replication. Requires additional operator effort for cross-site networking. | Disaster recovery across geographically separated sites | -| **Dedicated isolation** | A dedicated Cluster with its own isolated MetalPool and exclusive use of MetalPools. Same isolation level as Gardener's dedicated Seed for critical infrastructure. | Strictest compliance requirements for critical infrastructure | +| Topology | Description | Use Case | +| ------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------- | +| **Single-site HA** | Multiple control plane Machines within a single partition with etcd replicas on separate Machines. Natively supported. | Single-site deployments, standard production | +| **Multi-failure-domain** | Control plane Machines distributed across multiple zones or regions with etcd spread accordingly. Natively supported through ClusterClass topology definitions. | Rack/zone-level failure isolation | +| **Multi-site** | Control plane Machines deployed across multiple CAPI management clusters or across widely separated MetalPools with cross-site etcd replication. Requires additional operator effort for cross-site networking. | Disaster recovery across geographically separated sites | +| **Dedicated isolation** | A dedicated Cluster with its own isolated MetalPool and exclusive use of MetalPools. Same isolation level as Gardener's dedicated Seed for critical infrastructure. | Strictest compliance requirements for critical infrastructure | All topologies are natively supported. Multi-site requires additional multi-pool configuration and cross-site networking setup. @@ -123,21 +123,9 @@ Kamaji integrations with metal-stack have not been evaluated in production-grade Kamaji acts as a `ControlPlaneProvider` with Cluster API, while CAPMS acts as the `InfrastructureProvider`. This setup manages **tenant clusters** on metal-stack infrastructure, combining Kamaji's control plane management with metal-stack's bare-metal provisioning. -Like Cluster-API, Kamaji is a framework rather than a complete platform — operators must assemble their own day-2 tooling (CNI, CCM, DNS, backup, certificate management) and manage them through GitOps workflows. - -**Deployment** +Like Cluster API itself, Kamaji is a framework rather than a complete platform — operators must assemble their own day-2 tooling (CNI, CCM, DNS, backup, certificate management) and manage it through GitOps workflows. What Kamaji changes is _where the control plane runs_, not _how much you have to build_. -1. **Prepare management cluster** — A Kubernetes cluster to host Kamaji and CAPMS providers -2. **Install Kamaji and CAPMS** — Deploy both providers into the management cluster -3. **Create a control plane VIP** — MetalLB assigns a virtual IP for the tenant API server -4. **Generate and apply tenant cluster manifest** — Use `clusterctl generate cluster` to produce a YAML with `Cluster`, `MetalStackCluster`, `KubeadmControlPlane`, `MachineDeployment`, and `MetalStackMachine` resources, then apply it -5. **Deploy add-ons** — Install CNI (Calico) and `metal-ccm` into the tenant cluster - -A working showcase is available in the [`capi-lab`](https://github.com/metal-stack/cluster-api-provider-metal-stack/blob/main/DEVELOPMENT.md#running-the-kamaji-flavor) setup, which extends the `mini-lab` with a Kamaji flavor. See our [blog post](/blog/2026/04-kamaji) for a detailed walkthrough of the architecture and setup. - -**Fleet management and GitOps** - -Since Kamaji with metal-stack uses Cluster-API under the hood, fleet management follows the same pattern as Cluster API. Tenant cluster manifests are generated via `clusterctl`, stored in Git, and deployed through your CI/CD pipeline. +A working showcase is available in the [`capi-lab`](https://github.com/metal-stack/cluster-api-provider-metal-stack/blob/main/DEVELOPMENT.md#running-the-kamaji-flavor) setup, which extends the `mini-lab` with a Kamaji flavor. See our [blog post](/blog/2026/04-kamaji) for a detailed walkthrough of the architecture and setup, and the [Cluster API deployment guide](../04-For%20Operators/03-Deployment/06_cluster-api.md#kamaji-as-control-plane-provider) for the deployment steps. ## Domain Model @@ -181,7 +169,7 @@ You must set up your own Git repository and GitOps operator to manage cluster de - **Cluster migration** — `clusterctl move` enables moving workload cluster resources between management clusters, pausing controllers during the move to prevent worker node loss - **Emergency patching** — Achieved through editing resources in the management cluster, e.g., update of machine OS image in the `MachineTemplate` or update of `ClusterResourceSet`. Unlike Gardener, this change is not rolled out fleet-wide automatically and should be staged through the Git repository with standard approval processes - **Certificate rotation** — No direct workflow is described for certificate rotation at the landscape level; this is to be defined by platform administrators using manual/custom processes based on standard tooling (e.g., `kubeadm` certificate renewal) -- **Shoot deletion protection** — Special labels safeguard against accidental shoot deletion; configurable backup retention allows emergency access to cluster resources before cleanup +- **Deletion protection** — Cluster API offers no equivalent to Gardener's deletion-confirmation annotations. Protection has to come from the GitOps layer (branch protection, approval gates) and from Kubernetes finalizers - **Cluster Autoscaler** — Automatic scaling of worker groups based on requested pod resources, configured per `MachineDeployment` **Audit configuration** — Audit configuration can be passed to the kube-apiserver via the `kubeadmConfigSpec` of the `KubeadmControlPlane` resource before cluster creation. Each cluster can have its own audit policy. The management cluster's kube-apiserver audit also needs to be configured separately. Cluster API does not provide a centralized audit management toolset and relies on cloud-native standards to be set up by the operator. @@ -198,7 +186,7 @@ The following data center infrastructure dependencies are treated as given and m ## Scale & Testing -For Cluster API, the integration test environment covers a management cluster with three worker nodes. The biggest test clusters have included 8 cluster nodes. As both CAPI and Gardener share the same metal-stack control plane, from the metal-stack perspective it is guaranteed to work with the numbers mentioned for Gardener (200+ clusters, 5 data centers, 1,800 physical servers). Validations in CAPI are not as thoroughly implemented as in Gardener, but all usual management workflows for metal-stack clusters (creation, move, and deletion) are integration tested within a matrix of Kubernetes version, CNI, and OS version. +For Cluster API, the integration test environment covers a management cluster with three worker nodes. The biggest test clusters have included 8 cluster nodes. As both CAPI and Gardener share the same metal-stack control plane, from the metal-stack perspective the [numbers mentioned for Gardener](./02-gardener.md#scalability) (280 clusters, 5 data centers, 1,800 physical servers) also apply here. Validations in CAPI are not as thoroughly implemented as in Gardener, but all usual management workflows for metal-stack clusters (creation, move, and deletion) are integration tested within a matrix of Kubernetes version, CNI, and OS version. ## Fleet Operations @@ -234,10 +222,3 @@ Unlike Gardener, Cluster API with metal-stack requires you to assemble your own - **Cluster migration** — Available via `clusterctl move` between management clusters - **Multi-tenant self-service** — Not available; build custom API layer on top of Cluster API - **Access control lists** — No built-in firewall controller; firewall rules are currently static - -## Next Steps - -- **[KCLM Overview](./01-kclm.md)** — Introduction to Kubernetes Cluster Lifecycle Management with metal-stack -- **[Gardener](./02-gardener.md)** — metal-stack's recommended, production-ready KCLM solution -- **[KCLM Deployment Guide](../04-For%20Operators/03-Deployment/05_kclm.md)** — Step-by-step deployment instructions -- **[Cluster API Documentation](https://cluster-api.sigs.k8s.io/)** — Official Cluster API documentation diff --git a/docs/05-Concepts/04-Kubernetes/04-cloud-controller-manager.md b/docs/05-Concepts/04-Kubernetes/04-cloud-controller-manager.md index 8fc4cee6..945353ab 100644 --- a/docs/05-Concepts/04-Kubernetes/04-cloud-controller-manager.md +++ b/docs/05-Concepts/04-Kubernetes/04-cloud-controller-manager.md @@ -40,10 +40,3 @@ These labels enable end-users to configure Pod topology spread constraints and a For Gardener deployments, metal-ccm is deployed as part of the Shoot cluster provisioning flow via the `gardener-extension-provider-metal`. For Cluster API deployments, it is installed through `ClusterResourceSet` objects alongside the CNI (Calico). For detailed build and deployment instructions, see the [metal-ccm reference guide](../../08-References/Kubernetes/metal-ccm/metal-ccm.md). - -## Next Steps - -- **[KCLM Overview](./01-kclm.md)** — Introduction to Kubernetes Cluster Lifecycle Management -- **[Gardener](./02-gardener.md)** — Gardener integration -- **[Cluster API](./03-cluster-api.md)** — Cluster API integration -- **[metal-ccm Reference](../../08-References/Kubernetes/metal-ccm/metal-ccm.md)** — Build and deployment instructions diff --git a/docs/05-Concepts/04-Kubernetes/05-firewall-controller-manager.md b/docs/05-Concepts/04-Kubernetes/05-firewall-controller-manager.md index c3a8cac0..ab365140 100644 --- a/docs/05-Concepts/04-Kubernetes/05-firewall-controller-manager.md +++ b/docs/05-Concepts/04-Kubernetes/05-firewall-controller-manager.md @@ -16,20 +16,20 @@ The design of the FCM is inspired by Gardener's [machine-controller-manager](htt The FCM introduces the following [CRDs](https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/): -| Custom Resource | Description | -|----------------|-------------| +| Custom Resource | Description | +| -------------------- | --------------------------------------------------------------------------------------------------------------------------------- | | `FirewallDeployment` | Contains the spec template of a `Firewall` resource, similar to a `Deployment`. Implements update strategies like rolling update. | -| `FirewallSet` | Similar to `ReplicaSet`. Typically owned by a `FirewallDeployment`. Attempts to run the defined number of `Firewall` replicas. | -| `Firewall` | Similar to a `Pod`. Has a 1:1 relationship to a firewall in the metal-stack API. | -| `FirewallMonitor` | Deployed into the user's shoot cluster. Useful for monitoring the firewall or triggering user-initiated actions. | +| `FirewallSet` | Similar to `ReplicaSet`. Typically owned by a `FirewallDeployment`. Attempts to run the defined number of `Firewall` replicas. | +| `Firewall` | Similar to a `Pod`. Has a 1:1 relationship to a firewall in the metal-stack API. | +| `FirewallMonitor` | Deployed into the user's shoot cluster. Useful for monitoring the firewall or triggering user-initiated actions. | ### Controllers -| Controller | Responsibility | -|------------|---------------| +| Controller | Responsibility | +| ------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `FirewallDeploymentController` | Manages the lifecycle of `FirewallSet`s. Syncs the `Firewall` template spec and triggers a `FirewallSet` roll when significant changes are made. Supports `RollingUpdate` and `Recreate` strategies. Also deploys a service account for the firewall-controller. | -| `FirewallSetController` | Creates and deletes `Firewall` objects according to the spec and replica count. Reports `Firewall` status. | -| `FirewallController` | Creates and deletes the physical firewall machine via the [metal-api](https://github.com/metal-stack/metal-api). | +| `FirewallSetController` | Creates and deletes `Firewall` objects according to the spec and replica count. Reports `Firewall` status. | +| `FirewallController` | Creates and deletes the physical firewall machine via the [metal-api](https://github.com/metal-stack/metal-api). | ## User Actions @@ -43,12 +43,3 @@ The FCM exposes user-facing capabilities through annotations on `FirewallMonitor The FCM is deployed into the shoot namespace of a seed cluster as part of the Gardener extension provisioning flow. For detailed configuration and development instructions, see the [firewall-controller-manager reference guide](../../08-References/Kubernetes/firewall-controller-manager/firewall-controller-manager.md). - -For detailed configuration and development instructions, see the [firewall-controller-manager reference guide](../../08-References/Kubernetes/firewall-controller-manager/firewall-controller-manager.md). - -## Next Steps - -- **[KCLM Overview](./01-kclm.md)** — Introduction to Kubernetes Cluster Lifecycle Management -- **[Gardener](./02-gardener.md)** — Gardener integration -- **[Cluster API](./03-cluster-api.md)** — Cluster API integration -- **[Firewall Controller Manager Reference](../../08-References/Kubernetes/firewall-controller-manager/firewall-controller-manager.md)** — Detailed configuration and development instructions diff --git a/docs/05-Concepts/04-Kubernetes/07-storage.md b/docs/05-Concepts/04-Kubernetes/07-storage.md index 9573f003..9d60e437 100644 --- a/docs/05-Concepts/04-Kubernetes/07-storage.md +++ b/docs/05-Concepts/04-Kubernetes/07-storage.md @@ -18,7 +18,7 @@ In the meantime, we have started to integrate third-party solutions into our met - Built-in multi-tenant capabilities - Configurable compression and replication factors -We are maintaining an open source integration for running LightOS in our [Gardener](./01-gardener.md) cluster provisioning. You can enable it through the controller registration of the [gardener-extension-provider-metal](https://github.com/metal-stack/gardener-extension-provider-metal). +We are maintaining an open source integration for running LightOS in our [Gardener](./02-gardener.md) cluster provisioning. You can enable it through the controller registration of the [gardener-extension-provider-metal](https://github.com/metal-stack/gardener-extension-provider-metal). With the integration in place, the extension-provider deploys a [duros-controller](https://github.com/metal-stack/duros-controller) along with a Duros Storage CRD into the seed's shoot namespace. The duros-controller takes care of creating projects and managing credentials at the Lightbits Duros API. It also provides storage classes as configured in the extension-provider's controller registration to the customer's shoot cluster such that users can start consuming the Lightbits storage immediately. @@ -35,7 +35,7 @@ metadata: name: csi-pvc spec: accessModes: - - ReadWriteOnce + - ReadWriteOnce resources: requests: storage: 100Mi diff --git a/docs/06-For CISOs/Security/04-communication-matrix.md b/docs/06-For CISOs/Security/04-communication-matrix.md index b911b146..1abf73fb 100644 --- a/docs/06-For CISOs/Security/04-communication-matrix.md +++ b/docs/06-For CISOs/Security/04-communication-matrix.md @@ -116,7 +116,7 @@ Please note that every [networking setup](../../05-Concepts/03-Network/01-theory | VLAN | Switches, Firewalls | Layer 2 traffic segmentation. | | VXLAN | Switches, Firewalls | Encapsulate Layer 2 frames in Layer 3 packets for network virtualization. | | EVPN | Switches, Firewalls | Overlay network technology for scalable and flexible network architectures. | -| VPN | Firewalls | Management access [without open SSH ports](/community/MEP-9-no-open-ports-to-the-data-center). | +| VPN | Firewalls | Management access [without open SSH ports](/community/MEP-9-no-open-ports-to-the-data-center). | | BGP | Multiple | Routing protocol for dynamic routing and network management. | | SSH | Management Server, Switches | Secure shell access for management and configuration. | | LLDP | Switches, Machines | Link Layer Discovery Protocol for network device discovery. | @@ -124,7 +124,7 @@ Please note that every [networking setup](../../05-Concepts/03-Network/01-theory ## With Gardener -When using metal-stack in [conjunction with Gardener](../../05-Concepts/04-Kubernetes/01-gardener.md), the following communication is required by metal-stack components. +When using metal-stack in [conjunction with Gardener](../../05-Concepts/04-Kubernetes/02-gardener.md), the following communication is required by metal-stack components. :::info The following table might not be displayed in completeness. Scroll to the right to see all entries. @@ -154,7 +154,7 @@ The following table might not be displayed in completeness. Scroll to the right ## With Cluster API -By using the [Cluster API provider for metal-stack](../../05-Concepts/04-Kubernetes/02-cluster-api.md), the following communictations are required by metal-stack components. +By using the [Cluster API provider for metal-stack](../../05-Concepts/04-Kubernetes/03-cluster-api.md), the following communications are required by metal-stack components. :::info The following table might not be displayed in completeness. Scroll to the right to see all entries.