# Exoscale Documentation — Containers (SKS) > Scalable Kubernetes Service (SKS): fully managed Kubernetes with a production-ready control plane, integrated with Exoscale infrastructure. --- url: https://community.exoscale.com/product/compute/containers/ --- # Containers (SKS) - [Quick Start - Containers](https://community.exoscale.com/product/compute/containers/quick-start/index.md) - Get started quickly with a guided setup to achieve your first successful use of the service, including the required steps and configuration. - [Overview - Containers](https://community.exoscale.com/product/compute/containers/overview/index.md) - Terminology and key features at a glance, learn the core concepts, capabilities, and options of this service before you dive deeper. - [How-To - Containers](https://community.exoscale.com/product/compute/containers/how-to/index.md) - Step-by-step guides to configure the service and use specific features, with clear instructions for common tasks and problem solving. - [Service Boundaries - Containers](https://community.exoscale.com/product/compute/containers/service-boundaries/index.md) - Service boundaries: quotas, limits, and the guaranteed service levels (SLA) for this product, including key constraints to plan and operate reliably. - [Reference - Containers](https://community.exoscale.com/product/compute/containers/reference/index.md) - Reference documentation for API, CLI, and tools, browse comprehensive details, parameters, commands, and integration notes. - [Recipes - Containers](https://community.exoscale.com/product/compute/containers/recipes/index.md) - Informational, concise guides showing how to integrate with external tools and solve specific scenarios. --- url: https://community.exoscale.com/product/compute/containers/quick-start/ --- # Quick Start - Containers Exoscale's Scalable Kubernetes Service (SKS) offers a robust managed Kubernetes solution that includes: Managed Kubernetes Control Planes : Seamlessly handles the management tasks of your control planes. Dynamic Nodepool Attachment : Easily attach and manage dynamic NodePools as your needs evolve. Control Plane Access Management Facilities : Provides security and efficient access controls for your control planes. Comprehensive API Support : Full API access to customize and integrate with your existing systems. With SKS, you can effortlessly create and manage Kubernetes clusters throughout any Exoscale zone, ensuring streamlined and efficient lifecycle management for your Kubernetes environments. > [!TIP] > New to Kubernetes? Learn the basic concepts of Kubernetes and container > orchestration with our free [Certified Container Engineer (CCE) Course][1] in > the [Exoscale Academy][2]. [1]: https://academy.exoscale.com/courses/course-v1:Exoscale+CCE300+R2025/about [2]: https://www.exoscale.com/academy/ ## Prerequisites To interact with Exoscale SKS and the resulting Kubernetes cluster, the following items are required: * An active [Exoscale account][exo-account] * The [Exoscale CLI tool][exo-cli] - version 1.24 or above * The `kubectl` binary (see note below) * Basic knowledge of Kubernetes clusters and YAML manifest files ## Creating a Cluster from the CLI Creating a cluster can be done with the `exo compute sks create` command, which creates a control plane. By default, the CNI Calico is deployed on the cluster. The Exoscale Cloud Controller Manager (CCM) is automatically wired for interaction with Exoscale IaaS. Because Kubernetes is a clustered application, the control plane and the nodes need to communicate. Useful ingress ports to have open in your security group before starting: * 30000 to 32767 TCP from all sources for NodePort and LoadBalancer services * 10250 TCP with the security group as a source To create a new group like the above, you can use the CLI: ```bash exo compute security-group create sks-security-group ``` ```bash exo compute security-group rule add sks-security-group \ --description "NodePort services" \ --protocol tcp \ --network 0.0.0.0/0 \ --port 30000-32767 ``` ```bash exo compute security-group rule add sks-security-group \ --description "SKS kubelet" \ --protocol tcp \ --port 10250 \ --security-group sks-security-group ``` Depending on which CNI plugin you want to use, you will also need to open additional ports. If using **Calico as CNI plugin** (default), you need to open: * 4789 UDP with the security group as a source for VXLAN communication between nodes ```bash exo compute security-group rule add sks-security-group \ --description "Calico traffic" \ --protocol udp \ --port 4789 \ --security-group sks-security-group ``` Then you can initiate the cluster: ```bash exo compute sks create my-test-cluster \ --zone de-fra-1 \ --service-level pro \ --nodepool-name my-test-nodepool \ --nodepool-size 3 \ --nodepool-security-group sks-security-group ``` If using **Cilium as CNI plugin**, you need to open: * 8472 UDP with the security group as a source for VXLAN communication between nodes * 4240 TCP with the security group as a source for network connectivity health API (health-checks) * PING (ICMP type 8 & code 0) with the security group as a source for health checks ```bash exo compute security-group rule add sks-security-group \ --description "Cilium (healthcheck)" \ --protocol icmp \ --icmp-type 8 \ --icmp-code 0 \ --security-group sks-security-group ``` ```bash exo compute security-group rule add sks-security-group \ --description "Cilium (vxlan)" \ --protocol udp \ --port 8472 \ --security-group sks-security-group ``` ```bash exo compute security-group rule add sks-security-group \ --description "Cilium (healthcheck)" \ --protocol tcp \ --port 4240 \ --security-group sks-security-group ``` Then, you can initiate the cluster, specifying Cilium as CNI plugin using the `--cni` flag: ```bash exo compute sks create my-test-cluster \ --zone de-fra-1 \ --cni cilium \ --service-level pro \ --nodepool-name my-test-nodepool \ --nodepool-size 3 \ --nodepool-security-group sks-security-group ``` This `exo compute sks create` command starts a fully dedicated Kubernetes cluster in your organization in the **Frankfurt** zone with the service level **pro** and with **3** nodes. We strongly recommend you to have at least two workers on your SKS clusters. Some components cannot be upgraded safely on single node clusters. The Exoscale orchestrator then creates a new [instance pool][ipoolapidoc] with 3 instances on your behalf. By default, the instances sizes are Medium instances with 50 GB disks attached. These default values can be changed at creation time with additional `exo compute sks create` command flags. Run the command without parameters to get the full list: ```bash ✔ Creating SKS cluster "my-test-cluster"... 18s ✔ Adding Nodepool "my-test-nodepool"... 33s ┼───────────────────┼──────────────────────────────────────────────────────────┼ │ SKS CLUSTER │ │ ┼───────────────────┼──────────────────────────────────────────────────────────┼ │ ID │ 3c761780-6536-41b2-8950-551c4c5bf80a │ │ Name │ my-test-cluster │ │ Description │ │ │ Zone │ de-fra-1 │ │ Creation Date │ 2026-05-18 11:38:02 +0000 UTC │ │ Auto-upgrade │ false │ │ Enable kube-proxy │ true │ │ Endpoint │ 3c761780-6536-41b2-8950-551c4c5bf80a.sks-de-fra-1.exo.io │ │ Version │ 1.36.0 │ │ Service Level │ pro │ │ CNI │ calico │ │ Add-Ons │ exoscale-cloud-controller │ │ │ metrics-server │ │ Audit Enabled │ false │ │ Feature Gates │ │ │ State │ running │ │ Labels │ n/a │ │ Nodepools │ b16f3ade-72da-44a5-b8be-a8777761dec1 | my-test-nodepool │ ┼───────────────────┼──────────────────────────────────────────────────────────┼ ``` You could also add the `--auto-upgrade` option to the command in order to enable automatic upgrades of your control plane components to the latest Kubernetes patch version available. ## Managing Nodepools You can create, delete, update, or scale SKS nodepools at any time. The `--nodepool` options on `exo compute sks create` are optional. You also add nodepools to an existing cluster by using the `exo compute sks nodepool add` command. Each nodepool can have different characteristics - such as Security Groups, Private Networks, instance type, disk size, Anti-Affinity Groups and so on - depending on the applications you want to run on them. You can retrieve the list of all the available commands and options with the `--help` flag in the CLI, for example: * `exo compute sks --help` * `exo compute sks nodepool --help` * `exo compute sks nodepool add --help` ## Identifying your Cluster and Interacting with Kubernetes First, identify your newly created cluster with the command: ``` exo compute sks list ┼──────────────────────────────────────┼─────────────────┼──────────┼ │ ID │ NAME │ ZONE │ ┼──────────────────────────────────────┼─────────────────┼──────────┼ │ 3c761780-6536-41b2-8950-551c4c5bf80a │ my-test-cluster │ de-fra-1 │ ┼──────────────────────────────────────┼─────────────────┼──────────┼ ``` Then list the full details: ```bash exo compute sks show my-test-cluster --zone de-fra-1 ``` ### Kubeconfig You can then generate a `kubeconfig` file for your cluster: ```bash exo compute sks kubeconfig my-test-cluster kube-admin \ --zone de-fra-1 \ --group system:masters > my-test-cluster.kubeconfig ``` The `kube-admin` parameter is the name of your user in the kubeconfig file and in the certificate. You can configure fine-grained permissions by creating an user in Kubernetes and then reference it in your kubeconfig. You can optionally pass a TTL (in seconds) for the validity period of your kubeconfig's certificate by using the `--ttl` option. By default, a TTL of 30 days is used. The `group` parameter is a list of the Kubernetes groups associated with the generated certificate. This command returns a kubeconfig file. You can use it by passing it to `kubectl` using the `--kubeconfig` flag to send queries to your cluster. > [!NOTE] > Installing kubectl > > From now on, the interaction with Kubernetes itself must be done with > Kubernetes tooling. `kubectl` (short for Kubernetes Control) is the [official > command-line tool][install-kubectl] for Kubernetes. It is compatible with > Linux, macOS or Windows. You can list your Kubernetes workers nodes with: ```bash kubectl --kubeconfig my-test-cluster.kubeconfig get node ``` Output: ```bash NAME STATUS ROLES AGE VERSION pool-ad590-fizsv Ready 14m v1.36.0 pool-ad590-iyqua Ready 14m v1.36.0 pool-ad590-tjxga Ready 14m v1.36.0 ``` Your Scalable Kubernetes Service cluster is now fully ready for production. [lbls]: https://community.exoscale.com/product/compute/instances/how-to/labels/index.md [exo-account]: https://portal.exoscale.com/register [exo-cli]: https://community.exoscale.com/tools/command-line-interface/index.md [ipoolapidoc]: https://community.exoscale.com/reference/api/compute/instance-pool/#scale-instance-pool [install-kubectl]: https://kubernetes.io/docs/tasks/tools/ --- url: https://community.exoscale.com/product/compute/containers/overview/ --- # Overview - Containers Exoscale's Scalable Kubernetes Service (SKS) is a fully managed Kubernetes offering designed to simplify the deployment, scaling, and management of containerized applications. SKS provides a production-ready control plane, seamless integration with Exoscale infrastructure, and supports the full Kubernetes ecosystem. ## Terminology Cluster : A virtual entity encapsulating a Kubernetes control plane and one or more nodepools. Control Plane : The set of components that manage the lifecycle of a Kubernetes cluster, including the API server, scheduler, and controller manager. InstancePool : A group of similar Compute instances whose lifecycle is managed by the scheduler, created upon a set of user-specified instance properties (such as size, template, security groups, et cetera). NodePool : A group of compute instances managed by SKS, serving as worker nodes in your Kubernetes cluster. Node : A compute instance within a NodePool that runs Kubernetes workloads. ## Features Scalable Kubernetes Service has an expansive feature set: Managed Control Plane : SKS delivers a fully managed, highly available Kubernetes control plane, eliminating the need for manual setup and maintenance. Dynamic NodePools : Easily scale your workloads by adding or removing NodePools, which are groups of compute instances managed by SKS. Full Cluster Lifecycle Management : Create, upgrade, and delete clusters on demand. SKS supports automatic upgrades to the latest Kubernetes patch versions. Seamless Integration with Exoscale Services : Leverage Exoscale's Network Load Balancer, Block Storage via CSI driver, and other services directly within your SKS clusters. Customizable NodePools : Use any Exoscale instance type within your NodePools, including CPU-optimized, memory-optimized, and GPU instances. High Availability (Pro Plan) : The [Pro Cluster Plan](https://www.exoscale.com/pricing/#sks) offers a resilient, highly available control plane to ensure your workloads remain operational and plus automatic scalability of control plane resources. CNCF Certified : SKS is certified by the Cloud Native Computing Foundation, ensuring compatibility and support for standard Kubernetes APIs. ## Architecture Following is a breakdown of the overall SKS architecture. ![](sks-diagram.png "Overall SKS Architecture Diagram") ## Pricing Tiers Exoscale offers SKS in two tiers: Starter and Pro, each tailored to different use cases | Feature | STARTER | PRO | | ------------------------ | --------------------- | -------------------------- | | Usage | Development, testing, proof of concepts | Production workloads requiring high availability | | API | Yes | Yes | | CLI | Yes | Yes | | Terraform Support | Yes | Yes | | Control Plane Scaling | No | Yes | | Auto-Upgrade | Opt-in | Opt-in | | High Availability | No | Yes | | Backup of etcd | No | Min. Daily | | SLA | No | 99.95% | | Price | Free | [See Pricing](https://www.exoscale.com/pricing/#sks) | > [!NOTE] > Starter clusters without attached NodePools will be deleted after 30 days of > inactivity. ## Control Plane Exoscale runs the Kubernetes control plane for you – including the API Server, Controller Manager, Scheduler, and Konnectivity Server – ensuring high availability (in Pro plans), security, and minimal operational overhead. You don't manage or access the control plane directly. You deploy several components when you create a cluster. Some run on the Exoscale side, and some run on your cluster. ### Control Plane Components (Exoscale-managed) - The Kubernetes [API Server](https://kubernetes.io/docs/reference/command-line-tools-reference/kube-apiserver/). - The Kubernetes [Controller Manager](https://kubernetes.io/docs/reference/command-line-tools-reference/kube-controller-manager/). - The Kubernetes [Scheduler](https://kubernetes.io/docs/reference/command-line-tools-reference/kube-scheduler/). - The [Konnectivity Server](https://github.com/kubernetes-sigs/apiserver-network-proxy), which is used to provide a TCP level proxy for the control plane to cluster communication. - The [Exoscale Cloud Controller Manager](https://github.com/exoscale/exoscale-cloud-controller-manager) also comes as a default. ### In-Cluster Components Below are the components enabled by default. For some of them, you can choose to either enable or disable them. > [!WARNING] > When a component is enabled it will be managed by Exoscale. Therefore you > should **not** override or modify their related manifests. Failing to do so > can freeze your cluster preventing any further updates. #### [Kube Proxy](https://kubernetes.io/docs/reference/command-line-tools-reference/kube-proxy/) Forward the traffic to pods (using the `iptables` mode). Can be enabled/disabled? : Yes, at cluster creation time. Unless explicitly enabled from the API, it is disabled when Cilium CNI is selected. #### [CoreDNS](https://coredns.io/) Manage the cluster DNS. The CoreDNS ConfigMap (named `coredns` in the `kube-system` namespace) is never updated by Exoscale after the first deployment so you can override it if needed. Can be enabled/disabled? : No. #### [Konnectivity agent](https://github.com/kubernetes-sigs/apiserver-network-proxy) Can be enabled/disabled? : No. #### Container Network Interface (CNI) plugin Manage the cluster network. Can be enabled/disabled? : Yes, you can choose between [Calico](https://projectcalico.docs.tigera.io/about/about-calico) (default) and [Cilium](https://docs.cilium.io/en/stable/overview/intro/) at cluster creation time. You can also create a cluster _without_ CNI plugin. #### [Metrics server](https://github.com/kubernetes-sigs/metrics-server) Gather pods and nodes metrics. Can be used to auto-scale your deployments. Can be enabled/disabled? : Yes, at cluster creation time. ### Additional Features - Rapid cluster provisioning (≈2 minutes) - Automated or manual Kubernetes upgrades - Private networking and NLB support - CLI, API, Terraform, Pulumi, and web UI integration - OIDC integration for secure RBAC access ## NodePools A NodePool is a logical group of Kubernetes workers. When you create a NodePool, you specify: * the characteristics for the workers (instance disk size, instance type, firewall or anti-affinity rules, ...) * the number of workers The worker instances (or virtual machines) will be provisioned and will join the cluster automatically. A cluster can have multiple NodePools, each with its own characteristics depending of what you need. NodePools are backed by [Instance Pools](https://community.exoscale.com/product/compute/instances/how-to/instance-pools/index.md). When a NodePool is created, a new instance pool named `nodepool--` is also created. > [!NOTE] > You **cannot interact** with this Instance Pool directly. It will be managed > completely by Exoscale depending on the NodePool state. If you want to scale your NodePool, or evict machines from it, you need to target the NodePool, not the Instance Pool. > [!IMPORTANT] > A NodePool with an anti-affinity group can not contain more than 8 nodes. ### Exoscale Cloud Controller Manager The [Exoscale Cloud Controller Manager](https://github.com/exoscale/exoscale-cloud-controller-manager) performs several actions on the cluster: - Validates nodes. When a node joins the cluster, the controller will enable it and approve the Kubelet server certificate signing request. - Manages load balancers. The controller will automatically provision and manage Exoscale [Network Load balancers](https://www.exoscale.com/syslog/network-load-balancer-release/) based on Kubernetes services of the type `LoadBalancer` for a highly available load balancing setup for your cluster. When the Exoscale Cloud Controller is enabled, an [IAM](https://community.exoscale.com/product/security/iam/index.md) key is automatically provisioned on your account. You can find more information about this Cloud Controller IAM key (and how to rotate it) in the [SKS Certificates and Keys](https://community.exoscale.com/product/compute/containers/how-to/certificates-and-keys/index.md) section. > [!CAUTION] > If your SKS cluster is deployed with managed CCM enabled, you should not > install it yourself. Otherwise your own CCM setup may conflict with the > managed one and cause unexpected behavior. ## Service Level and Support Exoscale's Scalable Kubernetes Service (SKS) offers a fully managed control plane with defined service levels and support scopes. ### Control Plane SLA In the Pro service tier, the following Kubernetes control plane components are covered by a 99.95% Service Level Agreement (SLA): * `etcd` * `kube-apiserver` * `kube-scheduler` * `kube-controller-manager` * Exoscale Cloud Controller Manager (CCM) This ensures high availability and reliability for the core components managing your cluster's state and operations. ### Node-Level SLA While the control plane is managed by Exoscale, the worker nodes within your SKS cluster are your responsibility. Each node is subject to Exoscale's standard compute SLA of 99.95%, ensuring consistent performance and uptime for your workloads. ### In-Cluster Components Exoscale deploys several essential components within your SKS clusters to facilitate networking, service discovery, and storage integration: * CNI plugins: `Calico` or `Cilium` * `CoreDNS` * `Konnectivity` * `kube-proxy` * `metrics-server` * Container Storage Interface (CSI) driver These components are not covered by the SKS SLA due to the shared responsibility model. However, Exoscale provides best-effort support, including upgrade tools and operational guidance. In certain scenarios, support may request temporary access to your cluster's kubeconfig to assist with troubleshooting. ### Best Practices for Stability To ensure optimal performance and supportability of your SKS cluster: Resource Management : Set appropriate resource requests and limits for all workloads. This practice helps prevent resource contention and ensures fair scheduling across the cluster. Monitoring Node Health : Regularly monitor the health status of your nodes. If nodes are reported as “NotReady” or if core components exhibit unexpected behavior, verifying resource configurations can aid in diagnosis. Following these best practices helps maintain cluster stability and simplifies troubleshooting, enabling faster and more effective support when needed. --- url: https://community.exoscale.com/product/compute/containers/how-to/ --- # How-To - Containers Step-by-step guides to configure the service and use specific features, with clear instructions for common tasks and problem solving. - [Configure SKS Egress Filtering](https://community.exoscale.com/product/compute/containers/how-to/sks-egress-security-groups/index.md) - Secure SKS outbound traffic with egress filtering by allowing only required DNS, NTP, image pull, control plane, and API rules. - [Manage SKS Lifecycle](https://community.exoscale.com/product/compute/containers/how-to/lifecycle-management/index.md) - Manage the SKS lifecycle with supported versions, sequential upgrades, nodepool rotation, managed addons, backups, and plan changes. - [Use Gateway API on SKS](https://community.exoscale.com/product/compute/containers/how-to/gateway-api/index.md) - Use Gateway API on Exoscale SKS with Traefik to define entry points, advanced HTTP routing, and Kubernetes traffic management. - [Autoscale SKS Node Pools](https://community.exoscale.com/product/compute/containers/how-to/autoscaling/index.md) - Autoscale Exoscale SKS node pools with Cluster Autoscaler to add or remove nodes based on workload demand and scheduling needs. - [Configure CPU Manager Policies with Karpenter](https://community.exoscale.com/product/compute/containers/how-to/cpu-manager-policies/index.md) - Configure Kubernetes CPU manager policies on Exoscale SKS nodes provisioned by Karpenter for predictable CPU allocation and workload isolation. - [Configure OCI registries pull-through cache](https://community.exoscale.com/product/compute/containers/how-to/container-registry-mirroring/index.md) - Reduce egress costs and strengthen your OCI supply-chain - [Enable GPU Support in SKS Nodes](https://community.exoscale.com/product/compute/containers/how-to/gpu-sks-nodes/index.md) - Enable GPU support on Exoscale SKS nodes to run ML, analytics, and accelerated workloads with NVIDIA device plugins and GPU pods. - [Use Karpenter on SKS](https://community.exoscale.com/product/compute/containers/how-to/karpenter/index.md) - Use Karpenter on Exoscale SKS to automate node provisioning, fine-tune scaling, and match workloads with the right compute resources. - [Use Load Balancers and Ingress](https://community.exoscale.com/product/compute/containers/how-to/loadbalancer-ingress/index.md) - Expose services on Exoscale SKS with network load balancers and Traefik ingress for external access, routing, and traffic distribution. - [Isolate Critical Components](https://community.exoscale.com/product/compute/containers/how-to/isolate-critical-components/index.md) - Isolate critical SKS components on dedicated nodes with taints, tolerations, and node pools to improve cluster stability and security. - [Manage Certificates and Keys](https://community.exoscale.com/product/compute/containers/how-to/certificates-and-keys/index.md) - Manage SKS certificate authorities and addon credentials, retrieve public certs, and rotate CCM, CSI, Karpenter credentials or the operators CA. - [Use Feature Gates on SKS](https://community.exoscale.com/product/compute/containers/how-to/feature-gates/index.md) - Manage Kubernetes Feature Gates on Exoscale SKS by enabling or disabling supported features for clusters and Karpenter nodes. - [Use Kubernetes Audit](https://community.exoscale.com/product/compute/containers/how-to/kubernetes-audit/index.md) - Use Kubernetes Audit on Exoscale SKS to send API server audit events to a secure webhook endpoint for monitoring and analysis. - [Use OpenID Connect](https://community.exoscale.com/product/compute/containers/how-to/openid-connect/index.md) - Use OpenID Connect on Exoscale SKS to enable external authentication and integrate cluster access with your identity provider. - [Use Labels on SKS](https://community.exoscale.com/product/compute/containers/how-to/labels/index.md) - Use labels on Exoscale SKS clusters, nodepools, and Karpenter resources to organize workloads and control scheduling behavior. - [Set Instance Prefixes](https://community.exoscale.com/product/compute/containers/how-to/managed-instance-prefixes/index.md) - Set custom instance prefixes for Exoscale SKS nodepools to control how managed compute instance names are generated and displayed. - [Use Kubernetes Taints](https://community.exoscale.com/product/compute/containers/how-to/kubernetes-taints/index.md) - Use Kubernetes taints on Exoscale SKS nodepools to control pod scheduling and require matching tolerations for selected workloads. - [Use a Custom CNI](https://community.exoscale.com/product/compute/containers/how-to/custom-cni-setup/index.md) - Use a custom CNI on Exoscale SKS by configuring Pod CIDR, API access, DNS, network interfaces, and service proxy behavior. - [Disable kube-proxy](https://community.exoscale.com/product/compute/containers/how-to/kube-proxy-removal/index.md) - Disable kube-proxy on Exoscale SKS when using a custom CNI with proxy replacement, configured at cluster creation via CLI or Terraform. - [Use User Namespaces](https://community.exoscale.com/product/compute/containers/how-to/user-namespaces/index.md) - Use user namespaces on Exoscale SKS to improve pod isolation by mapping container root to unprivileged host user IDs for better security. - [Mount SOS in SKS](https://community.exoscale.com/product/compute/containers/how-to/sks-mountpoint-s3/index.md) - Mount Exoscale SOS buckets in SKS with the Mountpoint S3 CSI driver for read-heavy workloads, shared volumes, and prefix isolation. - [Restrict ingress traffic with IP allow-lists](https://community.exoscale.com/product/compute/containers/how-to/restrict-ingress-traffic-ip-allow-lists/index.md) - Secure SKS inbound traffic with ingress filtering by allowing only required IP entry and NLB - [Use GPU MIG Partitioning](https://community.exoscale.com/product/compute/containers/how-to/gpu-mig-partitioning/index.md) - Partition Exoscale MIG-capable GPUs into isolated instances with dedicated compute and memory for efficient multi-workload execution on SKS --- url: https://community.exoscale.com/product/compute/containers/how-to/sks-egress-security-groups/ --- # Configure SKS Egress Filtering If you want to harden your networking policies you may consider implementing egress security groups. This will drop all outbound traffic from your SKS cluster by default, and you will need to explicitly allow traffic for your workloads. > [!WARNING] > This setup is for advanced users only. Make sure you understand the > implications of modifying egress traffic before proceeding. Egress filtering > makes debugging more complex. ## SKS mandatory rules Before adding any egress rules, you need to define the rules listed in the table below. | Name | Protocol | Network | Port | Flow | Security Group | | ----------------------- | ---------- | ----------- | ------------- | -------- | --------------------------- | | NTP service | UDP | 0.0.0.0/0 | 123 | egress | | | DNS service (TCP) | TCP | 0.0.0.0/0 | 53 | egress | | | DNS service (UDP) | UDP | 0.0.0.0/0 | 53 | egress | | | Container images pull | TCP | 0.0.0.0/0 | 443 | egress | | | SKS control plane | TCP | 0.0.0.0/0 | 30000-32767 | egress | | | SKS inter-node | TCP | | 1-65535 | egress | sks-egress-security-group | | SKS API | TCP | | 1-65535 | egress | public-sks-apiservers | You can provision those rules from the CLI with the following commands: ```bash # create the security group exo compute security-group create sks-egress-security-group # allow NTP service exo compute security-group rule add sks-egress-security-group \ --description "NTP service" \ --protocol udp \ --network 0.0.0.0/0 \ --flow egress \ --port 123 # allow DNS resolution for TCP protocol exo compute security-group rule add sks-egress-security-group \ --description "DNS service (TCP)" \ --protocol tcp \ --network 0.0.0.0/0 \ --flow egress \ --port 53 # allow DNS resolution for UDP protocol exo compute security-group rule add sks-egress-security-group \ --description "DNS service (UDP)" \ --protocol udp \ --network 0.0.0.0/0 \ --flow egress \ --port 53 # allow Container images pull exo compute security-group rule add sks-egress-security-group \ --description "Container images pull" \ --protocol tcp \ --network 0.0.0.0/0 \ --flow egress \ --port 443 # allow SKS control plane communication exo compute security-group rule add sks-egress-security-group \ --description "SKS control plane" \ --protocol tcp \ --network 0.0.0.0/0 \ --flow egress \ --port 30000-32767 # allow SKS inter-node communication exo compute security-group rule add sks-egress-security-group \ --description "SKS inter-node" \ --protocol tcp \ --port 1-65535 \ --flow egress \ --security-group sks-egress-security-group # allow SKS API communication exo compute security-group rule add sks-egress-security-group \ --description "SKS API" \ --protocol tcp \ --port 1-65535 \ --flow egress \ --security-group public-sks-apiservers ``` ## See Also * A breakdown of [Exoscale Security Groups](https://community.exoscale.com/product/networking/security-group/index.md) --- url: https://community.exoscale.com/product/compute/containers/how-to/lifecycle-management/ --- # Manage SKS Lifecycle ## Supported Versions Exoscale SKS supports the latest Kubernetes **patch releases** (e.g., 1.30.X). If your cluster is facing any issue Exoscale support may request to upgrade to a supported version. **Minor versions** are supported for approximately 14 months after their release. Minor release support is aligned on the upstream [project version support lifecycle](https://kubernetes.io/releases/patch-releases/#support-period). ## Upgrades ### Upgrade Paths The following upgrade paths are supported: * From a minor release to next supported minor **and** latest patch release * From a patch release to the latest patch release NodePools always get the latest template matching the running minor release while following the latest patch release. For example, if you are running an older 1.30.1 release and 1.30.2 is available but you have not performed the upgrade of the cluster, cycling your NodePool or creating new nodes within it will provision the new nodes with the 1.30.2 template. Lifecycle of components running inside a customer cluster (Calico, CoreDNS, Konnectivity, Kubeproxy) is fully managed by Exoscale. These components can be upgraded as follows: * Minor releases without breaking change and CVE fix releases: at any time * Major to minor release with breaking changes: only at cluster minor or patch release upgrade (triggered by customer) ### Auto-upgrades You can enable auto-upgrades of the control plane components to the latest patch release available on your current Kubernetes minor release by setting the `auto-upgrade` parameter to `true` on your cluster. For example, you can either set the `--auto-upgrade` flag when you create a cluster using the CLI (with `exo compute sks create`), or enable auto-upgrade on an existing cluster with `exo compute sks update`. Auto-upgrades work exactly like upgrades triggered manually by you. ### Release schedule The design goal for SKS is to make versions available in a 2-week timeframe after their upstream release. However, this timeframe can vary depending on the importance or any breaking changes associated with the release. ### Upgrading a cluster To upgrade a cluster: 1. Upgrade the SKS control plane 2. Cycle your NodePools so the nodes are running the latest version > [!IMPORTANT] > Upgrades must be performed sequentially through each minor version. For example, when you need to upgrade e.g. from 1.27.5 to 1.30.2 you will need to perform these 2 steps for every minor release (1.28 and 1.29): 1. Upgrade the SKS control plane to 1.28.<latest version> 2. Cycle your NodePools so the nodes are running version 1.28.<latest version> 3. Upgrade the SKS control plane to 1.30.2 4. Cycle your NodePools so the nodes are running version 1.30.2 **Finding out your SKS control plane version** To list the current running version: ```bash exo compute sks show --zone de-fra-1 my-test-cluster ``` Output: ```bash ┼───────────────┼──────────────────────────────────────────────────────────────────┼ │ SKS CLUSTER │ │ ┼───────────────┼──────────────────────────────────────────────────────────────────┼ │ ID │ aaea400a-b9d3-458d-adc7-e1f0b2a2ed51 │ │ Name │ my-test-cluster │ │ Description │ my-test-cluster │ │ Zone │ de-fra-1 │ │ Creation Date │ 2024-07-10 09:38:33 +0000 UTC │ │ Endpoint │ https://aaea400a-b9d3-458d-adc7-e1f0b2a2ed51.sks-de-fra-1.exo.io │ │ Version │ 1.29.6 │ │ Service Level │ pro │ │ CNI │ calico │ │ Add-Ons │ exoscale-cloud-controller │ │ State │ running │ │ NodePools │ 2bc04c15-5996-46b2-acfb-1ac24aead8e2 | my-NodePool │ ┼───────────────┼──────────────────────────────────────────────────────────────────┼ ``` Here the running version is 1.29.6 - to list available versions: **List available versions** ```bash exo compute sks versions ``` Output: ```bash ┼─────────┼ │ VERSION │ ┼─────────┼ │ 1.30.2 │ │ 1.29.6 │ │ 1.28.11 │ ┼─────────┼ ``` The version 1.30.2 is the latest available. **Trigger the upgrade of the cluster control plane** ```bash exo compute sks upgrade my-test-cluster --zone de-fra-1 1.30.2 ``` Output: ```bash ✔ Upgrading SKS cluster "my-test-cluster"... 1m57s ┼───────────────┼──────────────────────────────────────────────────────────────────┼ │ SKS CLUSTER │ │ ┼───────────────┼──────────────────────────────────────────────────────────────────┼ │ ID │ aaea400a-b9d3-458d-adc7-e1f0b2a2ed51 │ │ Name │ my-test-cluster │ │ Description │ my-test-cluster │ │ Zone │ de-fra-1 │ │ Creation Date │ 2024-07-10 09:38:33 +0000 UTC │ │ Endpoint │ https://aaea400a-b9d3-458d-adc7-e1f0b2a2ed51.sks-de-fra-1.exo.io │ │ Version │ 1.30.2 │ │ Service Level │ pro │ │ CNI │ calico │ │ Add-Ons │ exoscale-cloud-controller │ │ State │ running │ │ NodePools │ 2bc04c15-5996-46b2-acfb-1ac24aead8e2 | my-NodePool │ ┼───────────────┼──────────────────────────────────────────────────────────────────┼ ``` However, the NodePool attached still has nodes running in the previous version: ```bash kubectl --kubeconfig my-test-cluster.kubeconfig get node ``` Output: ```bash NAME STATUS ROLES AGE VERSION pool-ad590-fizsv Ready 14m v1.29.6 pool-ad590-iyqua Ready 14m v1.29.6 pool-ad590-tjxga Ready 14m v1.29.6 ``` ### NodePool Upgrade There are several ways to do a NodePool upgrade: 1. Scale the NodePool and drain any old nodes 2. Drain and destroy nodes one by one 3. Create a new NodePool **Scale the NodePool and drain old nodes** The first solution is to scale your NodePool and then drain nodes from it. For example, your could scale your NodePool of 3 nodes to 6 instances: ```bash exo compute sks NodePool scale --zone de-fra-1 my-test-cluster my-NodePool 6 ``` (Note that we double the number of nodes in this guide for demonstration purposes, but you can upgrade your nodes in smaller batches if desired.) When the new nodes are available, you can use `kubectl drain ` on your old nodes to remove pods. After the nodes are drained, you can execute the command `exo compute sks NodePool evict --zone de-fra-1 my-test-cluster my-NodePool `. You can provide several nodes as an argument to the command. #### Drain and destroy nodes one by one Another solution is to drain and destroy nodes one by one: * drain one node using `kubectl drain ` * remove it the node your cluster with `kubectl delete node ` * destroy the node with `exo compute instance delete ` * repeat as needed A new instance will be automatically created to replace one that was destroyed. #### Create a new NodePool Another method is to create a new NodePool and then drain all nodes from the old NodePool. Then you can delete the old NodePool using `exo compute sks NodePool delete --zone de-fra-1 my-test-cluster my-NodePool`. > [!TIP] > Are you interested in general rollout and update topics with Kubernetes? Take > a look at the __free__ [Certified Container Engineer Course][1] in our online > academy. [1]: https://academy.exoscale.com/courses/course-v1:Exoscale+CCE300+R2025/about ## Enabling new functionality: managed addons Managed addons refers to a set of bespoke optional (but recommended) components and configuration deployed and managed by Exoscale onto the SKS cluster. - **Exoscale Cloud Controller Manager CCM** - Lets the cluster interact with the Exoscale Platform and enables kubelet to be properly validated to join the cluster (allowing kubectl logs/exec) - **Exoscale Container Storage Interface CSI** - Provides the necessary setup to attach BlockStorage volumes to an SKS cluster - **Metrics Server** - Records metrics on the cluster Addons can be specified upon cluster creation or when updating the cluster later on. > [!IMPORTANT] > When enabling CCM on a running cluster, make sure to migrate all workloads to > new NodePools and destroy the previous ones. See the previous sections > [Drain and destroy nodes one by one](#drain-and-destroy-nodes-one-by-one) and > [Create a new NodePool](#create-a-new-nodepool) for information on how to > achieve this. ## Backup Strategies While Exoscale SKS doesn't include built-in backup features, you can implement backups using the following tools: - [UrBackup](https://community.exoscale.com/product/storage/object-storage/recipes/urbackup/index.md): Back up data to Exoscale Object Storage. - [Veeam Backup](https://community.exoscale.com/product/storage/object-storage/recipes/veeam/index.md) : Utilize Veeam with Exoscale Object Storage, including object lock for immutable backups. These solutions provide flexibility in securing your data according to your specific needs. ## Upgrading from Starter to Pro Plan Exoscale SKS offers [two plans](https://community.exoscale.com/product/compute/containers/overview/index.md#pricing-tiers): - Starter: Ideal for everyday testing; free of charge. - Pro: Suitable for production workloads requiring high availability and support; billed per hour. Any Starter cluster can be upgraded to Pro. However a Pro cluster can't be downgraded. --- url: https://community.exoscale.com/product/compute/containers/how-to/gateway-api/ --- # Use Gateway API on SKS The [Gateway API](https://gateway-api.sigs.k8s.io/) is the successor to Kubernetes Ingress, providing a more expressive, extensible, and role-oriented approach to managing traffic routing. It introduces resources like `Gateway` and `HTTPRoute` that offer finer control over how traffic enters and flows through your cluster. ## Key Benefits - **Role-oriented**: Separates concerns between infrastructure providers, cluster operators, and application developers - **Expressive**: Native support for header-based routing, traffic splitting, and request manipulation - **Extensible**: Custom resources can extend functionality without modifying core APIs - **Portable**: Standard API works across multiple implementations (Traefik, Cilium, Envoy, Istio, Kong, and more) ## Prerequisites - An Exoscale SKS cluster with kubectl access - [Helm](https://helm.sh/docs/intro/install/) installed - Basic Kubernetes knowledge If you don't have access to an SKS cluster, follow the [Quick Start Guide](https://community.exoscale.com/product/compute/containers/quick-start/index.md). ## Install Gateway API CRDs First, install the Gateway API Custom Resource Definitions (CRDs): ```bash kubectl apply -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.2.1/standard-install.yaml ``` > [!NOTE] > Check the [Gateway API releases](https://github.com/kubernetes-sigs/gateway-api/releases) for the latest version. ## Install Traefik with Gateway API Provider Add the Traefik Helm repository and install Traefik with the Gateway API provider enabled: ```bash helm repo add traefik https://traefik.github.io/charts helm repo update helm install traefik traefik/traefik -n traefik --create-namespace \ --skip-crds \ --set providers.kubernetesGateway.enabled=true \ --set providers.kubernetesCRD.enabled=false \ --set providers.kubernetesIngress.enabled=false ``` > [!NOTE] > The `--skip-crds` flag prevents the Helm chart from reinstalling Gateway API CRDs that were > already applied in the previous step. The Traefik Helm chart bundles Gateway API CRDs and would > otherwise conflict with the manually installed ones. > > This configuration enables Gateway API support while disabling traditional Ingress. > If you need both Gateway API and Ingress support, omit the `kubernetesCRD` and > `kubernetesIngress` flags, or set them to `true`. This installation creates a `LoadBalancer` service, which automatically provisions an Exoscale Network Load Balancer through the CCM. Verify the installation: ```bash kubectl get pods -n traefik -l app.kubernetes.io/name=traefik kubectl get svc -n traefik traefik ``` > [!NOTE] > The commands above assume Traefik is installed in the `traefik` namespace, as shown in the Helm command. ## How Traffic Flows on Exoscale When you create the Traefik Service of type `LoadBalancer`, Exoscale provisions a Network Load Balancer (NLB) through the CCM. Traffic then follows this path: - Client -> Exoscale NLB (Service `LoadBalancer`) - NLB -> NodePort on cluster nodes - NodePort -> Traefik pod - Traefik -> `Gateway` -> `HTTPRoute` -> Service -> Pods This is why you must allow NodePort traffic in the node security group. ## Configure Security Group for NodePort Traffic The Network Load Balancer routes traffic to NodePorts on your cluster nodes. You need to allow this traffic through a security group. Create a security group and add a rule for the NodePort range: ```bash exo compute security-group create traefik-nodeports exo compute security-group rule add traefik-nodeports \ --protocol tcp \ --port 30000-32767 \ --network 0.0.0.0/0 ``` Attach the security group to your nodepool. You can do this via the Portal or CLI: ```bash exo compute sks nodepool update \ --security-group traefik-nodeports \ --security-group ``` > [!IMPORTANT] > The `--security-group` flag replaces all security groups on the nodepool. Make sure to include any existing security groups in the command to avoid removing them. ## Create Gateway API Resources Gateway API separates concerns across different roles: - **`GatewayClass`**: Defined by the infrastructure provider (e.g., Traefik). Typically set up once per cluster. - **`Gateway`**: Managed by platform or cluster operators. Defines entry points, ports, and TLS configuration. - **`HTTPRoute`**: Owned by application developers. Defines routing rules and can be bundled alongside applications in Helm charts. ### GatewayClass The `GatewayClass` defines which controller handles Gateways. Create a file named `gatewayclass.yaml`: ```yaml apiVersion: gateway.networking.k8s.io/v1 kind: GatewayClass metadata: name: traefik spec: controllerName: traefik.io/gateway-controller ``` Apply it: ```bash kubectl apply -f gatewayclass.yaml ``` ### Gateway The `Gateway` resource defines the entry point for traffic. Create a file named `gateway.yaml`: ```yaml apiVersion: gateway.networking.k8s.io/v1 kind: Gateway metadata: name: traefik-gateway spec: gatewayClassName: traefik listeners: - name: http protocol: HTTP port: 8000 ``` > [!NOTE] > Traefik uses port 8000 for its `web` entryPoint by default. The external LoadBalancer maps port 80 to this internal port, so the Gateway listener must match the Traefik entryPoint port (8000 unless you override Helm values). If you set `ports.web.port=80`, update the Gateway listener to `port: 80`. Apply it: ```bash kubectl apply -f gateway.yaml ``` ### Example Application Deploy a simple application to test the setup. Create a file named `app.yaml`: ```yaml apiVersion: apps/v1 kind: Deployment metadata: name: hello labels: app: hello spec: replicas: 3 selector: matchLabels: app: hello template: metadata: labels: app: hello spec: containers: - name: hello image: nginxdemos/hello:plain-text ports: - containerPort: 80 --- apiVersion: v1 kind: Service metadata: name: hello spec: selector: app: hello ports: - port: 80 targetPort: 80 ``` Apply it: ```bash kubectl apply -f app.yaml ``` ### HTTPRoute The `HTTPRoute` defines how traffic is routed to your services. It only becomes active when it is attached to a `Gateway` via `parentRefs`. Create a file named `httproute.yaml`: ```yaml apiVersion: gateway.networking.k8s.io/v1 kind: HTTPRoute metadata: name: hello-route spec: parentRefs: - name: traefik-gateway rules: - matches: - path: type: PathPrefix value: / backendRefs: - name: hello port: 80 ``` Apply it: ```bash kubectl apply -f httproute.yaml ``` ## Verify the Setup Check that the Gateway is ready: ```bash kubectl get gateway traefik-gateway ``` Get the external IP of the LoadBalancer: ```bash kubectl get svc -n traefik traefik -o jsonpath='{.status.loadBalancer.ingress[0].ip}' ``` Test the route: ```bash curl http:/// ``` You should see the nginx hello response. ## Gateway API vs Ingress Gateway API is a good fit when you need advanced routing, multi-team ownership boundaries, or a consistent policy model across teams and clusters. Ingress remains a good choice for simple routing and legacy setups. > [!NOTE] > Gateway API is still evolving, and feature support varies by controller. Always check your controller's Gateway API support matrix. > [!TIP] > You can run Ingress and Gateway API side by side in Traefik by enabling both providers. This can help with gradual migration, but increases operational complexity. > [!IMPORTANT] > Gateway API does not natively support Traefik's `TLSOption` CRD. If you need to customize TLS settings (such as minimum TLS version, cipher suites, or mTLS policies), use Ingress with `IngressRoute` instead, where `TLSOption` references work as expected. | Feature | Ingress | Gateway API | | --------- | --------- | ------------- | | Role separation | Single resource | GatewayClass, Gateway, HTTPRoute | | Header routing | Annotation-dependent | Native support | | Traffic splitting | Not supported | Native support | | Request/response manipulation | Controller-specific | Standardized filters | | Multi-tenancy | Limited | Built-in support | | Extensibility | Annotations | Custom resources | **When to use Ingress**: Simple routing needs, existing configurations, controller-specific features. **When to use Gateway API**: Complex routing, multi-team environments, portability requirements, new deployments. ## Cleanup When deleting resources, follow this order to avoid orphaned cloud resources: > [!CAUTION] > Always delete the LoadBalancer service before deleting the nodepool or cluster. Otherwise, the NLB may not be automatically cleaned up. ```bash # Delete Gateway API resources kubectl delete httproute hello-route kubectl delete gateway traefik-gateway kubectl delete gatewayclass traefik # Delete the application kubectl delete -f app.yaml # Uninstall Traefik (this deletes the LoadBalancer service) helm uninstall traefik # Delete Gateway API CRDs kubectl delete -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.2.1/standard-install.yaml # Remove security group (after cluster deletion) exo compute security-group delete traefik-nodeports ``` ## Other Gateway API Controllers While this guide uses Traefik, the Gateway API is supported by many controllers: - **[Cilium Gateway API](https://docs.cilium.io/en/stable/network/servicemesh/gateway-api/gateway-api/)**: eBPF-based networking with Gateway API support - **[Envoy Gateway](https://gateway.envoyproxy.io/)**: Managed Envoy proxy for Kubernetes - **[Istio](https://istio.io/latest/docs/tasks/traffic-management/ingress/gateway-api/)**: Service mesh with Gateway API integration - **[Kong](https://docs.konghq.com/kubernetes-ingress-controller/latest/concepts/gateway-api/)**: API gateway with Gateway API support The concepts in this guide apply to other controllers with minor configuration differences. ## See Also * For traditional load balancing and ingress, see [SKS Load Balancer and Ingress Controller](https://community.exoscale.com/product/compute/containers/how-to/loadbalancer-ingress/index.md) * [Official Gateway API documentation](https://gateway-api.sigs.k8s.io/) * [Traefik Gateway API documentation](https://doc.traefik.io/traefik/providers/kubernetes-gateway/) * [Gateway API implementations](https://gateway-api.sigs.k8s.io/implementations/) * For details on SKS service levels and shared responsibility, see [SKS Overview](https://community.exoscale.com/product/compute/containers/overview/index.md) --- url: https://community.exoscale.com/product/compute/containers/how-to/autoscaling/ --- # Autoscale SKS Node Pools One of the primary features of running your applications on Kubernetes is the ability to scale your cluster without your intervention based on the current workload. In Kubernetes, one can scale: * the pods themselves vertically by raising and lowering resource requests using the Vertical Pod Autoscaler * the pods horizontally by raising and lowering the number of pods in a deployment using the Horizontal Pod Autoscaler * the number of nodes by resizing the nodepool based on either node utilization or pod deployment requirements using the cluster Autoscaler First, we will describe the cluster Autoscaler option. ## Prerequisites As a prerequisite for the following documentation, you need: - An Exoscale SKS cluster on the Pro plan - Access to your cluster via `kubectl` - Basic Linux knowledge If you do not have access to an SKS cluster, follow the [Quick Start Guide](https://community.exoscale.com/product/compute/instances/quick-start/index.md). ## Deploying the Cluster Autoscaler In the [Kubernetes Autoscaler](https://github.com/kubernetes/autoscaler/) repository, you can find the Exoscale provider in the [./cluster-autoscaler/cloudprovider/exoscale](https://github.com/kubernetes/autoscaler/tree/master/cluster-autoscaler/cloudprovider/exoscale) folder. As described in the README for that repository, you need to create a secret with an appropriate API key as well as the zone of your cluster: ```bash export EXOSCALE_API_KEY="EXOxxxxxxxxxxxxxxxxxxxxxxxx" export EXOSCALE_API_SECRET="xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx" export EXOSCALE_ZONE="ch-gva-2" kubectl -n kube-system create secret generic exoscale-api-credentials \ --from-literal=api-key="$EXOSCALE_API_KEY" \ --from-literal=api-secret="$EXOSCALE_API_SECRET" \ --from-literal=api-zone="$EXOSCALE_ZONE" ``` This API key **MUST** be authorized to perform the following API operations: ```bash evict-sks-nodepool-members get-instance get-instance-pool get-quota list-sks-clusters scale-sks-nodepool ``` Then you can get the [deployment manifests](https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/cloudprovider/exoscale/examples/cluster-autoscaler.yaml). Make sure to modify the deployment `image` to match the kubernetes version you are currently using. Then deploy the Autoscaler: ```bash kubectl apply -f cluster-autoscaler.yaml ``` > [!NOTE] > When testing, adjust the commented timeouts towards the end of the file to > see scale-down happening within a minute or so, instead of the slower > behavior you might want in a production deployment. ### Multiple nodepools In case your cluster has multiple nodepools, you might want to tell the cluster Autoscaler which nodepool should be scaled up or down. Otherwise, a random nodepool will be scaled. Create a `ConfigMap` with your nodepools **Instance Pool** ID (**not** the SKS nodepool ID itself): ```yaml apiVersion: v1 kind: ConfigMap metadata: name: cluster-autoscaler-priority-expander namespace: kube-system data: priorities: |- 50: - 00719c6e-1d06-4053-afea-8926c3431ef7 ``` As well as adding an argument to the autoscaler deployment: ```yaml - --expander=priority ``` The cluster Autoscaler will then target this prioritized nodepool instead. ## Putting it to the Test To test everything is working as it should, we deploy a `DaemonSet` that is very busy, but still fit in our current deployment of two nodes: ```yaml apiVersion: apps/v1 kind: Deployment metadata: name: stress namespace: default spec: replicas: 2 selector: matchLabels: run: stress template: metadata: labels: run: stress spec: affinity: podAntiAffinity: requiredDuringSchedulingIgnoredDuringExecution: - labelSelector: matchExpressions: - key: run operator: In values: - stress topologyKey: kubernetes.io/hostname containers: - image: nixery.dev/stress:latest name: stress command: - stress - --cpu - "1" resources: limits: cpu: 300m memory: 30Mi requests: cpu: 150m memory: 15Mi ``` Note how we use a podAntiAffinity to ensure these pods cannot share a node. This reflects real requirements, but the node filling up with pods would result in a similar behavior. After this is deployed, the CPU limit will be consumed in each pod and the Kubernetes metrics confirm this: ```bash kubectl top pods NAME CPU(cores) MEMORY(bytes) stress-d77bdc8db-22dwb 292m 0Mi stress-d77bdc8db-pd4xn 291m 0Mi ``` We now use the Horizontal Pod Autoscaler to react on these internal metrics and increase this deployments replicas to 11: ```yaml apiVersion: autoscaling/v2 kind: HorizontalPodAutoscaler metadata: name: stress namespace: default spec: minReplicas: 1 maxReplicas: 11 metrics: - resource: name: cpu target: averageUtilization: 50 type: Utilization type: Resource scaleTargetRef: apiVersion: apps/v1 kind: Deployment name: stress ``` We can see new pods being spawned and new nodes being added after a short time, as the pods can not be scheduled onto an existing node: ```bash NAME STATUS ROLES AGE VERSION pool-0f4cd-fquea Ready 3d6h v1.29.6 pool-0f4cd-llrsh NotReady 16s v1.29.6 pool-0f4cd-vhend NotReady 15s v1.29.6 pool-0f4cd-ygwhj Ready 3d6h v1.29.6 pool-0f4cd-zknko NotReady 15s v1.29.6 [...] ``` You can look into the cluster Autoscaler logs to see it taking action: ```bash kubectl -n kube-system logs deployment/cluster-autoscaler ``` Output: ```bash [...] I0324 19:50:12.521062 1 scale_up.go:675] Scale-up: setting group 0f4cd2ad-2825-4ae7-aaa2-1fd0f8e0af19 size to 5 I0324 19:50:12.530709 1 log.go:32] exoscale-provider: scaling SKS Nodepool afe7aa21-b4f6-409b-a70b-6fc31d6fada1 to size 5 [...] ``` When the new nodes are ready, the pods will start up on them. When you remove the test deployment, superfluous nodes will be removed after a grace period. ## Tips and Tricks * You can add `--v=4` to the list of arguments to get more information in the logs about why the cluster Autoscaler makes a certain decision. * Certain pods can prevent the Autoscaler from removing a node. See the [CA FAQ](https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/FAQ.md#what-types-of-pods-can-prevent-ca-from-removing-a-node) for more information. * You can also [annotate certain nodes](https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/FAQ.md#how-can-i-prevent-cluster-autoscaler-from-scaling-down-a-particular-node), so they will not be touched by the CA. * Longhorn Users: Note that the cluster Autoscaler is not fully supported. See the Issues section of the [Longhorn repository](https://github.com/longhorn/longhorn/issues/2203). > [!TIP] > Are you interested in kubernetes storage topics like Volumes, Persistent > Volumes and Longhorn? Take a look at the __free__ [Certified Container > Engineer Course][1] in our online academy. [1]: https://academy.exoscale.com/courses/course-v1:Exoscale+CCE300+R2025/about --- url: https://community.exoscale.com/product/compute/containers/how-to/cpu-manager-policies/ --- # Configure CPU Manager Policies with Karpenter Kubernetes CPU manager policies control how CPUs are allocated to containers running on a node. With Karpenter on Exoscale SKS, you can configure the kubelet CPU manager through the `ExoscaleNodeClass` resource and apply the configuration to all nodes provisioned from that node class. ## Prerequisites As a prerequisite for the following documentation, you need: - An Exoscale SKS cluster on the Pro plan with the Karpenter addon enabled. - Access to your cluster with `kubectl`. - A Karpenter `NodePool` and an `ExoscaleNodeClass` configured for Karpenter. - Basic Kubernetes and YAML knowledge. If you do not have access to an SKS cluster, follow the [Quick Start Guide](https://community.exoscale.com/product/compute/instances/quick-start/index.md). ## Understanding CPU manager policies The Kubernetes CPU manager allocates CPUs to containers that request CPU resources. It can improve CPU affinity and reduce contention for workloads that are sensitive to latency or require predictable CPU performance. Kubernetes supports the following CPU manager policies: - `none`: the default policy. CPU resources are managed by the standard scheduler and container runtime mechanisms. - `static`: allows eligible containers with an integer CPU request to receive exclusive CPUs. Other containers continue to use the shared CPU pool. For more information about CPU manager policies, CPU requests, and exclusive CPUs, see the [official Kubernetes documentation](https://kubernetes.io/docs/tasks/administer-cluster/cpu-management-policies/). > [!IMPORTANT] > The `static` policy is useful only when workloads request CPU resources correctly. A container must request an integer number of CPUs to be eligible for exclusive CPU allocation. ## Configure CPU manager on Karpenter nodes CPU manager settings are configured in the `spec.kubelet` section of an `ExoscaleNodeClass`. The settings are applied when Karpenter provisions a node from that node class. The following example enables the `static` policy, reserves CPU for Kubernetes and system components, and changes the CPU manager reconciliation period: ```yaml apiVersion: karpenter.exoscale.com/v1 kind: ExoscaleNodeClass metadata: name: cpu-static spec: imageTemplateSelector: {} kubelet: cpuManagerPolicy: static cpuManagerReconcilePeriod: 10s kubeReserved: cpu: "200m" systemReserved: cpu: "100m" ``` `cpuManagerPolicy` accepts `none` or `static`. The default is `none`. The `static` policy requires a non-zero CPU reservation in either `kubeReserved.cpu` or `systemReserved.cpu`; the `ExoscaleNodeClass` validation rejects the resource if this requirement is not met. `cpuManagerReconcilePeriod` controls how often the kubelet reconciles CPU assignments. Its default value is `10s`, so it can usually be omitted. ### Configure static policy options The `static` policy can be fine-tuned with `cpuManagerPolicyOptions`. Karpenter supports the following options: - `full-pcpus-only` - `distribute-cpus-across-numa` - `prefer-align-cpus-by-uncorecache` - `strict-cpu-reservation` - `align-by-socket` - `distribute-cpus-across-cores` For example: ```yaml apiVersion: karpenter.exoscale.com/v1 kind: ExoscaleNodeClass metadata: name: cpu-static spec: imageTemplateSelector: {} kubelet: cpuManagerPolicy: static cpuManagerPolicyOptions: - full-pcpus-only - distribute-cpus-across-numa kubeReserved: cpu: "200m" systemReserved: cpu: "100m" ``` These options are valid only when `cpuManagerPolicy` is set to `static`. The Kubernetes version running on the node must support the selected options. Refer to the [Kubernetes CPU manager documentation](https://kubernetes.io/docs/tasks/administer-cluster/cpu-management-policies/) for their behavior and compatibility. ## Deploy the configuration Apply the `ExoscaleNodeClass` before applying or updating a `NodePool` that references it: ```bash kubectl apply -f exoscale-nodeclass.yaml kubectl apply -f nodepool.yaml ``` A `NodePool` must reference the node class through `nodeClassRef`: ```yaml apiVersion: karpenter.sh/v1 kind: NodePool metadata: name: cpu-optimized spec: template: spec: nodeClassRef: group: karpenter.exoscale.com kind: ExoscaleNodeClass name: cpu-static requirements: - key: node.kubernetes.io/instance-type operator: In values: - standard.medium - standard.large ``` New nodes provisioned for this `NodePool` use the configured CPU manager policy. Existing nodes are not reconfigured in place. When the CPU manager configuration changes, Karpenter detects the drift and replaces affected nodes when disruption policies allow it. ## Verify the configuration Check that the node class and node pool are ready: ```bash kubectl get exoscalenodeclass cpu-static kubectl get nodepool cpu-optimized kubectl get nodeclaim -o wide ``` After a node has been provisioned, inspect its kubelet configuration: ```bash kubectl get node -o yaml ``` You can also check the kubelet logs on the node to confirm that the CPU manager started with the expected policy. The exact log output depends on the SKS node image and Kubernetes version. To test exclusive CPU allocation, deploy a pod with an integer CPU request and limit: ```yaml apiVersion: v1 kind: Pod metadata: name: cpu-manager-test spec: restartPolicy: Never containers: - name: test image: registry.k8s.io/pause:3.10 resources: requests: cpu: "1" limits: cpu: "1" ``` The pod must be scheduled on a node using the `static` policy. Pods with fractional CPU requests, such as `500m`, use the shared CPU pool and are not assigned exclusive CPUs. > [!WARNING] > Changing CPU manager settings affects node allocation and will cause Karpenter to replace nodes. Plan the change around your workload disruption budgets and verify that enough capacity remains available during the replacement. --- url: https://community.exoscale.com/product/compute/containers/how-to/container-registry-mirroring/ --- # Configure OCI registries pull-through cache Pulling images directly from public registries works, but it is rarely the best fit in production. Routing container image pulls through a private mirror helps you reduce egress costs, control which images are allowed in your cluster, and avoid hitting rate limits imposed by upstream registries. With Karpenter on Exoscale SKS, mirror endpoints and pull credentials are configured through the `ExoscaleNodeClass` resource. ## Prerequisites As a prerequisite for the following documentation, you need: - An Exoscale SKS cluster on the Pro plan with the Karpenter addon enabled. - Access to your cluster with `kubectl`. - A Karpenter `NodePool` and an `ExoscaleNodeClass` configured for Karpenter. - A reachable OCI-compliant registry to use as a mirror. - Basic Kubernetes and YAML knowledge. If you do not have access to an SKS cluster, follow the [Quick Start Guide](https://community.exoscale.com/product/compute/instances/quick-start/index.md). ## How registry mirroring works The `ExoscaleNodeClass` exposes a structured `spec.containerRegistry` field. Karpenter renders it into the SKS node agent configuration, which configures container runtime registry mirrors. When it resolves an image reference, it first queries the configured mirror endpoints. If the mirror returns a manifest, it pulls the image from there. Otherwise, the request falls back to the upstream registry. Two kinds of configuration are supported: - `mirrors`: per-registry mirror endpoints, with optional TLS material to authenticate the mirror. - `credentials`: per-registry pull credentials (basic auth, registry auth token, or identity token) used when contacting a private registry. Both `mirrors` and `credentials` reference Kubernetes Secrets that must live in the `kube-system` namespace. Karpenter validates the secrets and the referenced keys at reconciliation time. If a secret is missing or malformed, the `ExoscaleNodeClass` reports a `ContainerRegistrySecretsResolved=False` condition with a precise reason (for example `SecretMissing`, `InvalidPEM`, `TLSSecretInconsistent`). > [!IMPORTANT] > `spec.containerRegistry` takes priority over `spec.userData`, always configure mirroring in a single location, preferably `spec.containerRegistry`. ## Configure a mirror The following example routes pulls for `docker.io` and `ghcr.io` to a private mirror served over HTTPS at `https://mirror.example.com`. The mirror exposes a public CA bundled by the cluster, so no TLS secret is required. ```yaml apiVersion: karpenter.exoscale.com/v1 kind: ExoscaleNodeClass metadata: name: with-mirror spec: imageTemplateSelector: {} containerRegistry: mirrors: - registry: docker.io endpoints: - url: https://mirror.example.com - registry: ghcr.io endpoints: - url: https://mirror.example.com ``` Each `mirrors` entry accepts one to ten `endpoints`. When several endpoints are listed, the container runtimes tries them in order and falls back to the next one on failure. This is useful to expose a primary mirror and a backup mirror in a different region or zone. ### Use a private mirror with a custom CA When the mirror serves a certificate signed by a private CA, the CA bundle must be mounted on every Karpenter node. Create a Secret in `kube-system` containing the CA, then reference it from the mirror endpoint: ```bash kubectl -n kube-system create secret generic mirror-ca \ --from-file=ca.crt=./mirror-ca.pem ``` ```yaml apiVersion: karpenter.exoscale.com/v1 kind: ExoscaleNodeClass metadata: name: with-mirror spec: imageTemplateSelector: {} containerRegistry: mirrors: - registry: docker.io endpoints: - url: https://mirror.example.com tlsSecretRef: name: mirror-ca ``` The referenced Secret must live in `kube-system` and contain at least `ca.crt` as a PEM-encoded certificate. Mutual TLS is supported by adding `tls.crt` and `tls.key` to the same Secret: both keys must be set together, otherwise reconciliation fails with `TLSSecretInconsistent`. When the mirror preserves the upstream path (`https://mirror.example.com/v2/...`), you can leave `overridePath` unset. If the mirror exposes a flattened path layout (for example mapping multiple registries under a single namespace), set `overridePath: true` on the endpoint. > [!WARNING] > Avoid `skipVerify: true` in production. It disables certificate validation against the mirror and should only be used for ephemeral or test setups. ## Configure pull credentials To pull from a private registry, configure a credential entry. Three authentication schemes are supported, and exactly one of `basic`, `auth`, or `identityToken` must be set per credential: ```bash kubectl -n kube-system create secret generic registry-pull \ --from-literal=username='robot' \ --from-literal=password='secret-token' ``` ```yaml apiVersion: karpenter.exoscale.com/v1 kind: ExoscaleNodeClass metadata: name: with-mirror spec: imageTemplateSelector: {} containerRegistry: credentials: - registry: reg.example.com basic: usernameSecretRef: name: registry-pull key: username passwordSecretRef: name: registry-pull key: password ``` The `auth` scheme uses a single key (typically the pre-encoded `user:password` string used by the Docker registry `WWW-Authenticate` flow): ```yaml credentials: - registry: reg.example.com auth: authSecretRef: name: registry-pull key: auth ``` The `identityToken` scheme is intended for short-lived OIDC-based tokens: ```yaml credentials: - registry: my-oidc-registry.example.com identityToken: identityTokenSecretRef: name: ecr-identity-token key: identitytoken ``` Referenced Secrets must exist in `kube-system` and contain the requested key. The reconciler returns a `SecretMissing` or `SecretKeyMissing` reason when the lookup fails. ## Deploy the configuration Apply the `ExoscaleNodeClass` before applying or updating a `NodePool` that references it: ```bash kubectl apply -f exoscale-nodeclass.yaml kubectl apply -f nodepool.yaml ``` A `NodePool` must reference the node class through `nodeClassRef`: ```yaml apiVersion: karpenter.sh/v1 kind: NodePool metadata: name: mirrored spec: template: spec: nodeClassRef: group: karpenter.exoscale.com kind: ExoscaleNodeClass name: with-mirror requirements: - key: node.kubernetes.io/instance-type operator: In values: - standard.medium - standard.large ``` New nodes provisioned for this `NodePool` start with the configured mirrors and credentials. Existing nodes are not reconfigured in place. When the mirror configuration or any referenced Secret changes, Karpenter detects the drift through the `ContainerRegistrySecretsResolved` condition and replaces affected nodes when disruption policies allow it. ## Verify the configuration Check that the node class reports ready: ```bash kubectl get exoscalenodeclass with-mirror \ -o jsonpath='{.status.conditions[?(@.type=="ContainerRegistrySecretsResolved")]}' ``` > [!WARNING] > Changing the registry configuration affects node allocation and will cause Karpenter to replace nodes. Plan the change around your workload disruption budgets and verify that enough capacity remains available during the replacement. --- url: https://community.exoscale.com/product/compute/containers/how-to/gpu-sks-nodes/ --- # Enable GPU Support in SKS Nodes Exoscale SKS allows you to run GPU-accelerated workloads, such as Machine Learning (ML), data analytics, and video transcoding, on your cluster. In this documentation, we will guide you through the steps to enable GPU support in Exoscale SKS nodes. ## Prerequisites As a prerequisite for the following documentation, you need: - An Exoscale SKS cluster on the Pro plan. - An organization with at least one GPU instance type authorized. - Access to your cluster via `kubectl`. - Basic Linux knowledge. If you do not have access to an SKS cluster, follow the [Quick Start Guide](https://community.exoscale.com/product/compute/instances/quick-start/index.md). SKS relies on nvidia-open package. The following Exoscale compute GPU profiles are supported by SKS through this driver: - gpu3 (A40) - gpua30 - gpu3080ti - gpua5000 - gpurtx6000pro > [!NOTE] > Starting with Kubernetes 1.33.0, gpu2 profile is not compatible anymore with SKS as it relies on deprecated nvidia proprietary driver. ## Enabling GPU Support in SKS > [!IMPORTANT] > If you are running nodes with Kubernetes version 1.31.12, 1.32.8, 1.33.4, or > 1.34.0 or above, this section is **not relevant** anymore, and you can skip > it. On those versions of SKS node OS, the device plugin is **already > installed** as a system service directly communicating with the Kubelet > process. If you previously deployed the NVIDIA Device Plugin DaemonSet, you > can remove it once you upgraded all GPU nodes of your cluster to such > versions. To use GPUs in Kubernetes, the [NVIDIA Device Plugin][1] is required. The NVIDIA Device Plugin is a [DaemonSet][2] that automatically enumerates the number of GPUs on each node of the cluster and allows Pods to run on GPUs. To enable GPU support in Exoscale SKS nodes, you need to deploy the following DaemonSet: ```shell kubectl create -f https://raw.githubusercontent.com/NVIDIA/k8s-device-plugin/main/deployments/static/nvidia-device-plugin.yml ``` > [!NOTE] > This is a simple static DaemonSet meant to demonstrate the basic features of > the `nvidia-device-plugin`. [1]: https://github.com/NVIDIA/k8s-device-plugin/ [2]: https://kubernetes.io/docs/concepts/workloads/controllers/daemonset/ ## Running and Testing GPU Jobs Once nodes with GPU are ready, NVIDIA GPUs can be requested by a container using the `nvidia.com/gpu` resource type: ```bash cat < [!CAUTION] > Make sure to keep that role untouched – do not modify or delete it – as it is > essential for Karpenter's functionality. Karpenter-provisioned nodes are **not backed by an Instance Pool** and therefore **cannot be used as backends for Network Load Balancers (NLBs)**. NLBs can only target Instance Pools, this means you must keep at least one classic SKS Nodepool for any workloads that need to receive traffic through an NLB (e.g. `LoadBalancer`-type Services or Ingress/Gateway controllers). See [SKS Load Balancer and Ingress Controller](https://community.exoscale.com/product/compute/containers/how-to/loadbalancer-ingress/index.md) for details. Karpenter further requires certain CRDs to be installed in your cluster. When you enable the Karpenter addon, these CRDs are also automatically installed. ```bash kubectl get crds | grep karpenter exoscalenodeclasses.karpenter.exoscale.com 2025-11-04T10:13:23Z nodeclaims.karpenter.sh 2025-11-04T10:13:26Z nodeoverlays.karpenter.sh 2025-11-04T10:13:26Z nodepools.karpenter.sh 2025-11-04T10:13:27Z ``` ## Configuration If you haven't previously set up any nodes or configured the [cluster-autoscaler](https://github.com/kubernetes/autoscaler) before, all your SKS-backed pods will be in a Pending state on your cluster; this is expected, as no `Node` is available to schedule them. In case you add Karpenter to an existing cluster where the cluster-autoscaler is used, they won't conflict on node ownership. However, if pods require some more nodes to be scheduled and both cluster-autoscaler nodepool and Karpenter nodepool can satisfy the request, both controllers may try to provision new nodes. This may lead to overprovisioning of nodes in your cluster. Existing pods won't be rescheduled to Karpenter nodes if they are already scheduled on cluster-autoscaler nodes. ```bash ❯ kubectl get pod -A NAMESPACE NAME READY STATUS RESTARTS AGE kube-system calico-kube-controllers-59556d9b4c-dgjdj 0/1 Pending 0 7m2s kube-system coredns-7d78fc9666-9rrqf 0/1 Pending 0 6m58s kube-system coredns-7d78fc9666-kgclq 0/1 Pending 0 6m58s kube-system konnectivity-agent-5b79d466d4-99jtg 0/1 Pending 0 6m57s kube-system konnectivity-agent-5b79d466d4-stbq5 0/1 Pending 0 6m57s kube-system metrics-server-6654fc5ff6-cjs46 0/1 Pending 0 6m55s ``` The main advantage of Karpenter is the possibility to have a fine-grained control over the node provisioning, permitting to have the best fitting nodes for your workloads. Karpenter relies on 2 Kubernetes CRDs to automatically scale the cluster's Nodes: - `ExoscaleNodeClass`: Exoscale SKS-specific implementation details for Karpenter - `NodePool`: Karpenter generic nodepool definition ### ExoscaleNodeClass ```yaml apiVersion: karpenter.exoscale.com/v1 kind: ExoscaleNodeClass metadata: name: standard spec: imageTemplateSelector: # version: Kubernetes version (semver format like "1.34.1") # If omitted (or if you use imageTemplateSelector: {}), the control plane's # current Kubernetes version will be auto-detected at runtime version: "1.34.1" # variant: Template variant (optional, defaults to "standard") # Options: "standard" for regular workloads, "nvidia" for GPU-enabled nodes variant: "standard" # Disk size in GB (default: 50) diskSize: 100 # Security groups (optional) # List the security group IDs or names to attach to instances # Ensure to set a correct security group allowing necessary traffic # like CNI or ingress securityGroupSelectorTerms: - id: "" - name: " 92s v1.34.1 k-main-x65sj Ready 107s v1.34.1 ❯ kubectl get nodepool NAME NODECLASS NODES READY AGE standard-main standard 2 True 3m38s ❯ kubectl get nodeclaim NAME TYPE CAPACITY ZONE NODE READY AGE standard-main-5hw57 standard.small on-demand ch-gva-2 k-main-5hw57 True 3m19s standard-main-x65sj standard.small on-demand ch-gva-2 k-main-x65sj True 3m19s ``` We see our pods scheduled on the newly created nodes: ```bash ❯ kubectl get pod -A -o wide NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES kube-system calico-kube-controllers-59556d9b4c-dgjdj 1/1 Running 0 3h6m 192.168.86.193 k-main-x65sj kube-system calico-node-87lkx 1/1 Running 0 6m2s k-main-x65sj kube-system calico-node-wz6bx 1/1 Running 0 5m47s k-main-5hw57 kube-system coredns-7d78fc9666-9rrqf 1/1 Running 0 3h6m 192.168.86.195 k-main-x65sj kube-system coredns-7d78fc9666-kgclq 1/1 Running 0 3h6m 192.168.115.130 k-main-5hw57 kube-system konnectivity-agent-5b79d466d4-99jtg 1/1 Running 0 3h6m 192.168.86.194 k-main-x65sj kube-system konnectivity-agent-5b79d466d4-stbq5 1/1 Running 0 3h6m 192.168.115.129 k-main-5hw57 kube-system kube-proxy-cb59l 1/1 Running 0 6m2s k-main-x65sj kube-system kube-proxy-xrbnm 1/1 Running 0 5m47s k-main-5hw57 kube-system metrics-server-6654fc5ff6-cjs46 1/1 Running 0 3h6m 192.168.86.196 k-main-x65sj ``` ## Cleanup If Karpenter is enabled on your SKS cluster and you want to disable it, or remove your SKS cluster, make sure to first delete all Karpenter `NodePools`. Our API will prevent you from disabling Karpenter or deleting the cluster if any `NodePool` is still present. ```bash kubectl delete nodepools.karpenter.sh --all ``` ## FAQ ### What happens if I delete a Karpenter compute instance outside of Kubernetes? Karpenter will automatically detect the missing node and will provision a new one to replace it when needed. Please note that any workloads running on the deleted node will be rescheduled to other available nodes . You may experience a brief disruption in your services after abruptly removing the node. ### What does Karpenter do when downscaling a nodepool? When Karpenter decides to downscale a nodepool, it first cordons the node to prevent new pods from being scheduled on it. Then, it evicts all pods running on the node, respecting Pod Disruption Budgets (PDBs) if they are defined. Once the node is drained, Karpenter terminates the compute instance associated with the node. This process helps to ensure that the cluster remains stable and that workloads are not disrupted more than necessary. You can prevent Karpenter from evicting certain pods by setting the `karpenter.sh/do-not-evict` annotation to `true` on the Pod spec or on the Node spec. ### What happens if I remove a Karpenter `NodeClaim` object directly? The NodeClaim is tied to the node. At Exoscale we decided to be conservative and we evict pods running on the node. After 15 minutes, Karpenter will terminate the compute instance associated with the node if it has not successfully finished to evict all pods. Removing this object is not recommended, you should only do it for testing purposes or in case of emergency. --- url: https://community.exoscale.com/product/compute/containers/how-to/loadbalancer-ingress/ --- # Use Load Balancers and Ingress Kubernetes spans an internal network across all nodes automatically. Your pods and containers are reachable through a private IP address inside the cluster. To route traffic from the outside, you need in most cases a load balancer and optionally an ingress. > [!TIP] > For new deployments, use [Gateway API with Traefik](https://community.exoscale.com/product/compute/containers/how-to/gateway-api/index.md), which is the recommended, modern approach. This guide focuses on the classic LoadBalancer + Ingress setup. ## Prerequisites As a prerequisite for the following documentation, you need: - An Exoscale SKS cluster - Access to your cluster via kubectl - Basic Linux knowledge If you don't have access to a SKS cluster, follow the [Quick Start Guide](https://community.exoscale.com/product/compute/containers/quick-start/index.md). > [!NOTE] > For Network Load Balancers (or NLBs) to be created automatically, you need > the [Exoscale Cloud Controller > Manager](https://github.com/exoscale/exoscale-cloud-controller-manager) (or > CCM) add-on installed in your cluster. By default, the CCM should come > automatically installed. In the CCM documentation, you can also find the > [reference on service > annotations](https://github.com/exoscale/exoscale-cloud-controller-manager/blob/master/docs/service-loadbalancer.md#annotations) > which you might need for your service or ingress configuration. ## Exposing a Single Service with an Exoscale Load Balancer To expose a single pod or a deployment, you need to create a service of type `LoadBalancer`. The following example creates a deployment named `hello`, which consists of 3 hello-world pods/containers. This could be your web application: ```bash kubectl create deployment hello --image=nginxdemos/hello:plain-text --replicas=3 ``` As there is no service yet defined, you can only access the webpage of the containers via the respective internal IP inside the cluster. Use `kubectl get pods -o wide`, if you want to take a look at the created pods. To allow access from outside the cluster, you can expose the deployment with a load balancer: ```bash kubectl expose deployment hello --port=80 --target-port=80 --name=hello-service --type=LoadBalancer ``` This will create a service named `hello-service`, exposing the deployment `hello`. `--port` specifies the port on where the LoadBalancer should listen, which should be port 80 in most cases. `--target-port` specifies the target port that the container exposes or provides. As we specified the flag `--type=LoadBalancer`, Kubernetes will talk over the CCM to Exoscale, and create a Network Load Balancer (or NLB). To get its IP address, you can use `kubectl get svc`: ```bash kubectl get svc -n default ``` Output: ```bash NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE hello-service LoadBalancer 10.102.248.197 194.182.170.158 80:30665/TCP 2m3s kubernetes ClusterIP 10.96.0.1 443/TCP 52m ``` You can also see your load balancer in the Portal under `Compute` in the main navigation under the Load Balancers section, or use the CLI with the `exo compute load-balancer` and `exo compute load-balancer show` commands. In this case, the web application would be now reachable via `194.182.170.158`. The service `LoadBalancer` will automatically route and balance traffic to your pods. > [!NOTE] > Deleting a cluster will not always delete the Network Load Balancers created > by a Kubernetes service. Make sure to check your load balancers in the Portal > or CLI after deleting a SKS cluster, and delete the load balancers if > necessary. When the annotation `exoscale-loadbalancer-external` is set to > true (see the example below), the load balancer will never be automatically > deleted. You can also specify this setup as a manifest, where you can define additional annotations for the `LoadBalancer` service: ```yaml apiVersion: apps/v1 kind: Deployment metadata: labels: app: hello name: hello spec: replicas: 3 selector: matchLabels: app: hello template: metadata: labels: app: hello spec: containers: - image: nginxdemos/hello:plain-text name: hello --- apiVersion: v1 kind: Service metadata: labels: app: hello name: hello-service annotations: # Uncomment if you want to use an already existing Exoscale LoadBalancer #service.beta.kubernetes.io/exoscale-loadbalancer-id: "09191de9-513b-4270-a44c-5aad8354bb47" #service.beta.kubernetes.io/exoscale-loadbalancer-external: "true" # When having multiple Nodepools attached to your SKS Cluster, # you need to specify then the ID of the underlying Instance Pool the NLB should forward traffic to #service.beta.kubernetes.io/exoscale-loadbalancer-service-instancepool-id: "F0D7A23E-14B8-4A6E-A134-1BFD0DF9A068" spec: ports: - port: 80 protocol: TCP targetPort: 80 selector: app: hello type: LoadBalancer ``` You can find all annotations for the `LoadBalancer` service in the [Exoscale CCM](https://github.com/exoscale/exoscale-cloud-controller-manager/blob/master/docs/service-loadbalancer.md#annotations) documentation. > [!NOTE] > A Network Load Balancer can only route to one instance pool. If you have > multiple nodepools attached to an SKS cluster, define which instance pool > (underlying one of the attached nodepools) to route to via an annotation. > Also, when a Kubernetes service points to a scheduled pod outside of its > attached instance pool, it can still be reached, as Kubernetes automatically > handles internal routing. However, this extra hop might result in less > efficient traffic balancing. > [!IMPORTANT] > NLB backends are currently only available through classic SKS > Nodepool. Nodes provisioned by > [Karpenter](https://community.exoscale.com/product/compute/containers/how-to/karpenter/index.md) are > **not** eligible and therefore **cannot serve as NLB > backends**. If you use Karpenter, make sure your `LoadBalancer`-type Services > and Ingress/Gateway controllers are scheduled on a classic SKS Nodepool (for > example, using node selectors or affinities). ![](routing-1.png "Routing with a LoadBalancer Service") When you have multiple services or websites, you can either create additional Network Load Balancers, or use an ingress in addition to the LB-service as explained in the next chapter. ## Using an Ingress to Expose Multiple Websites A Network Load Balancer only provides layer 4 (TCP) routing and cannot distinguish between different hostnames or website paths. To distinguish between the two, we need to enable layer 7 routing, we can point the NLB to a reverse proxy (or ingress) which is deployed on an arbitrary amount of nodes. For defining routes, you create ingress resources. However, they have no effect without an **ingress controller**, which is a deployed reverse proxy. The next section explains how to install the Traefik Ingress Controller. > [!NOTE] > There are multiple solutions for ingress controllers available. Check the > [Kubernetes > documentation](https://kubernetes.io/docs/concepts/services-networking/ingress-controllers/) > for more information. The next figure shows how the routing works with a combination of Network Load Balancer, Ingress Controller and deployments for different websites: ![](routing-2.png "Routing using an Ingress") > [!NOTE] > Are you looking for general information on ingress and ingress configuration? Take a look at the __free__ [Certified Container Engineer Course](https://academy.exoscale.com/courses/course-v1:Exoscale+CCE300+R2025/about) in our online academy. ### Deploying Traefik Ingress Controller [Traefik](https://traefik.io/traefik/) is a modern cloud-native ingress controller that supports both traditional Kubernetes Ingress resources and the newer Gateway API. Install it using Helm: ```bash helm repo add traefik https://traefik.github.io/charts helm repo update helm install traefik traefik/traefik --namespace traefik --create-namespace ``` This creates a Deployment running the Traefik proxy and a `LoadBalancer` service that provisions an Exoscale Network Load Balancer through the CCM. > [!NOTE] > You can customize the installation with Helm values. For example, to run > Traefik as a DaemonSet on all nodes, use `--set deployment.kind=DaemonSet`. > See the [Traefik Helm chart documentation](https://github.com/traefik/traefik-helm-chart) > for all available options. Wait until the Traefik pods have started: ```bash kubectl get pods -n traefik -l app.kubernetes.io/name=traefik --watch ``` It should show *Running* after a while: ```bash traefik-7d9f4b8c5-xk2mn 1/1 Running 0 45s ``` Use ctrl+c on your keyboard to cancel the `--watch` command. You are now ready to deploy ingress configurations. To get the IP of the `LoadBalancer`, you can use `kubectl get svc -n traefik traefik` or check your load balancer information using the Portal or CLI. ### Creating Ingress Rules You can now use an ingress rule to connect your applications with the Load Balancer. Below is an example ingress rule. Save it to a file and apply it with `kubectl apply -f rules.yaml`: ```yaml apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: ingress-myapp spec: ingressClassName: traefik rules: - http: paths: - path: /app1 pathType: Prefix backend: service: name: hello port: number: 80 - path: /app2 pathType: Prefix backend: service: name: other-service port: number: 80 ``` The example will route the path `/app1` and `/app2` to different backend services. Note that the ingress must point to an internal ClusterIP service. For running this example, you can take the example deployment in the first section, and change `type: LoadBalancer` to `type: ClusterIP`. The address `http://YOURLOADBALANCERIP/app1` will then route to the `hello` deployment. You can also specify different hosts, either in the same ingress ruleset or via different manifests as in the following example: ```yaml apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: ingress-myapp spec: ingressClassName: traefik rules: - host: "example.com" http: paths: - path: / pathType: Prefix backend: service: name: hello port: number: 80 --- apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: ingress-otherapp spec: ingressClassName: traefik rules: - host: "example-otherapp.com" http: paths: - path: / pathType: Prefix backend: service: name: other-service port: number: 80 ``` Use `kubectl get ingress` to list the rulesets that have been created. This command will also list the external IP address of the `LoadBalancer`. The next figure shows how the objects or services of the whole configuration are interrelated: ![](routing-2-config.png "Link of configurations using Ingress") ## See Also * For modern Kubernetes-native routing, see [Gateway API with Traefik](https://community.exoscale.com/product/compute/containers/how-to/gateway-api/index.md) * [Traefik documentation](https://doc.traefik.io/traefik/) * [Traefik Kubernetes Ingress documentation](https://doc.traefik.io/traefik/providers/kubernetes-ingress/) * The Exoscale CCM documentation on [integrating with Network Load Balancers](https://github.com/exoscale/exoscale-cloud-controller-manager/blob/master/docs/service-loadbalancer.md) * A breakdown of [Exoscale Network Load Balancers](https://community.exoscale.com/product/networking/nlb/index.md) * The Kubernetes documentation for [LoadBalancer](https://kubernetes.io/docs/concepts/services-networking/service/#loadbalancer) * The Kubernetes documentation for [Kubernetes Ingress](https://kubernetes.io/docs/concepts/services-networking/ingress/) --- url: https://community.exoscale.com/product/compute/containers/how-to/isolate-critical-components/ --- # Isolate Critical Components In Kubernetes clusters you have components more critical than others. These components should be isolated from user workloads to ensure the stability and security of your cluster. Here you will learn how to master this isolation using Kubernetes and Exoscale features. We will focus on SKS critical components but the same principles can be applied to your own critical workloads (ie: your monitoring stack, your logging stack, etc...). ## Prerequisites - Knowledge of SKS - how to create and/or update an SKS Cluster ## Exoscale Critical Components Isolation Overview To isolate critical components in your SKS cluster, you need to configure the appropriate Kubernetes resources. This typically involves: - Creating dedicated node pools for critical components. - Using Kubernetes taints and tolerations to ensure that critical components are scheduled on the appropriate nodes. Exoscale critical components includes: - Your choice of CNI plugin (calico or cilium) - [CoreDNS](https://coredns.io/) - [Metrics Server](https://github.com/kubernetes-sigs/metrics-server) Our managed addons are using Kubernetes `tolerations` and `affinities` to ensure they are scheduled on dedicated nodes. Here is an extract of the pod spec we apply to our critical managed addons: ```yaml spec: affinity: nodeAffinity: preferredDuringSchedulingIgnoredDuringExecution: - weight: 100 preference: matchExpressions: - key: exoscale.com/node-role operator: In values: - system tolerations: - operator: "Exists" key: CriticalAddonsOnly ``` With this configuration our critical components will be scheduled (if possible) on nodes labeled with `exoscale.com/node-role=system` and tainted with `CriticalAddonsOnly=true:NoSchedule`. Note that we are not using `nodeSelector` here, which permits our critical pods to run on any SKS cluster node if no dedicated node is available. This is important to ensure the availability of these critical components, as this node isolation is for more advanced users. ## Configuration In the example below, we will create a dedicated node pool for critical components and configure taints and tolerations accordingly, in the context of an SKS cluster. First ensure you have an anti-affinity group created to spread your node pools across different hosts: ```bash exo compute anti-affinity-group create my-cluster-sks-critical-workloads ``` Now you can choose either to use `SKS nodepools` or `Karpenter nodepools` ### Using SKS nodepools If you choose to use SKS nodepools, you can create a new node pool with the appropriate labels and taints: ```bash exo compute sks nodepool add my-sks-cluster critical-nodepool \ --size 2 \ --label exoscale.com/node-role=system \ --taint CriticalAddonsOnly=true:NoSchedule \ --instance-type small \ --disk-size 20 \ --anti-affinity-group my-cluster-sks-critical-workloads ``` ### Using Karpenter nodepools If you choose to use Karpenter nodepools, you can create a new node pool with the appropriate labels and taints using the following manifest: ```yaml apiVersion: karpenter.sh/v1 kind: NodePool metadata: name: exoscale-critical-pods spec: disruption: budgets: - nodes: 50% consolidateAfter: 2m consolidationPolicy: WhenEmptyOrUnderutilized limits: cpu: 2 memory: 4Gi template: spec: expireAfter: 720h nodeClassRef: group: karpenter.exoscale.com kind: ExoscaleNodeClass name: standard requirements: - key: node.kubernetes.io/instance-type operator: In values: - standard.medium taints: - effect: NoSchedule key: CriticalAddonsOnly value: "true" ``` ### Validate setup Now we have our nodes ready in our SKS cluster: ```bash ❯ kubectl get node NAME STATUS ROLES AGE VERSION pool-a2437-hctjt Ready 53s v1.34.1 pool-a2437-weezc Ready 47s v1.34.1 ``` You can verify that the nodes have the correct labels and taints: ```bash ❯ kubectl get node pool-a2437-hctjt -o yaml apiVersion: v1 kind: Node metadata: ... labels: ... exoscale.com/node-role: system ... name: pool-a2437-hctjt resourceVersion: "3457021066" uid: afeeec78-9aac-458c-993d-838eee510d96 spec: ... taints: - effect: NoSchedule key: CriticalAddonsOnly value: "true" ... ``` You can see that our pods are correctly scheduled on our dedicated nodes: ```bash ❯ kubectl get pod -A -o wide NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES kube-system calico-kube-controllers-59556d9b4c-zhqfr 1/1 Running 0 7m57s 192.168.218.117 pool-a2437-hctjt kube-system calico-node-sksc9 1/1 Running 0 3m29s pool-a2437-hctjt kube-system calico-node-vdlcj 1/1 Running 0 3m23s pool-a2437-weezc kube-system coredns-7d78fc9666-9sk8d 1/1 Running 0 7m51s 192.168.218.130 pool-a2437-hctjt kube-system coredns-7d78fc9666-jlcsg 1/1 Running 0 7m51s 192.168.153.66 pool-a2437-weezc kube-system konnectivity-agent-8468bff8f8-c4dmm 1/1 Running 0 7m50s 192.168.153.65 pool-a2437-weezc kube-system konnectivity-agent-8468bff8f8-l6x8k 1/1 Running 0 7m50s 192.168.218.129 pool-a2437-hctjt kube-system kube-proxy-kpkzt 1/1 Running 0 3m29s pool-a2437-hctjt kube-system kube-proxy-lj22f 1/1 Running 0 3m23s pool-a2437-weezc kube-system metrics-server-6654fc5ff6-qn4dk 1/1 Running 0 7m48s 192.168.218.131 pool-a2437-hctjt ``` With this setup, your SKS critical components are now isolated on dedicated nodes, enhancing the security of your Kubernetes cluster. ## Conclusion Isolating critical components in your SKS cluster is essential for maintaining the stability and security of your Kubernetes environment. By leveraging dedicated node pools, taints, and tolerations, you can ensure that these vital components are protected from potential disruptions caused by user workloads. This approach not only enhances the reliability of your cluster but also provides a robust framework for managing critical workloads effectively. This isolation strategy can be extended to other critical workloads within your cluster, allowing you to maintain high availability and performance across your Kubernetes infrastructure. You may consider creating dedicated nodepools for your CI/CD runners, your monitoring stack, your logging stack, ArgoCD/FluxCD, etc... --- url: https://community.exoscale.com/product/compute/containers/how-to/certificates-and-keys/ --- # Manage Certificates and Keys ## Certificates Authorities Several certificates authorities are generated when you create a cluster. You can retrieve the public certificate for some of these authorities by using the `exo compute sks authority-cert ` command in the CLI. The authorities are: - `kubelet`: the authority generating certificates for the Kubelet daemon running on the workers. - `aggregation`: the authority used by the aggregation layer. - `control-plane`: the authority of the cluster control plane. - `operators`: the authority signing the X.509 client certificates embedded in the kubeconfig files we generate for you, with `exo compute sks kubeconfig` through the `generate-sks-cluster-kubeconfig` API call. The API server trusts it as a client CA (`enable-operators-ca`, enabled by default), which is what makes those kubeconfigs work. It can be rotated, see [Rotate the operators CA](#rotate-the-operators-ca). - `etcd`: the authority securing the communications with the etcd datastore backing your control plane. It is entirely internal to the managed control plane and never exposed. Only `kubelet`, `aggregation` and `control-plane` can be retrieved with `authority-cert`. ## Addon credentials The Exoscale Cloud Controller Manager (CCM), the Exoscale Container Storage Interface (CSI) and the Karpenter addons each need a set of credentials to communicate with the platform. An IAM role and api key are created automatically per-addon and per-cluster on your account when you create or update a cluster with addons selected. The lifecycle of these credentials is managed by Exoscale. They are deleted automatically once the cluster is deleted. Credentials managed by Exoscale SKS all have following naming scheme: `sks--` eg. `sks-ccm-859ece7e-f4ec-4eab-a77f-d06a1cfc08fd` You can list them on your account with: ```bash exo iam api-key list ``` ## Rotation SKS lets you rotate two different things, and they have very different consequences: - **The addon credentials** (CCM, CSI, Karpenter): the IAM keys Exoscale uses on your behalf. Rotating them is transparent for your users. - **The operators CA**: the authority signing the client certificates in your kubeconfig files. Rotating it revokes `kubectl` access for everyone, see [Rotate the operators CA](#rotate-the-operators-ca). ### Why rotate the addon credentials Rotation replaces a credential with a freshly generated one and invalidates the previous one. You can use this feature when: - **The credential was deleted or altered by mistake.** Deleting the `sks-ccm-*` API key or its IAM role breaks the addon: no `LoadBalancer` service is reconciled anymore, and nodes TLS bootstrapping fails. Similarly, no volume is provisioned through the CSI, or no node is created by Karpenter. Rotating recreates the role and the key and re-injects them into the cluster. - **You suspect a leak or a compromise.** A key that appeared in a log, a screenshot, a support ticket or a git repository should be considered public. Rotating revokes it immediately. - **Your security policy requires periodic renewal.** Exoscale does not expire these credentials automatically, so a scheduled rotation is yours to trigger, for example as part of a quarterly review or an audit requirement. - **Someone with access to the credentials left the team.** Rotating closes the door without touching the rest of the cluster. Rotating an addon credential only concerns the credentials Exoscale manages for your cluster. It does not touch your own API keys, your kubeconfig, or your workloads. ### What happens during an addon credential rotation Each rotation is an asynchronous operation on the cluster: 1. A new IAM role and API key are generated. 2. They are pushed to the addon running in your cluster (for ccm and csi) or in our managed control plane fleet platform (for Karpenter). 3. The previous credential is revoked. The cluster moves to a transient state while this happens (`rotating-ccm-credentials`, `rotating-csi-credentials` or `rotating-karpenter-credentials`) and returns to `running` once the operation is over. Check it with: ```bash exo compute sks show --zone de-fra-1 my-cluster ``` > [!IMPORTANT] > The addon is briefly unable to talk to the Exoscale API while the new credentials propagate. This is not disruptive for running workloads, but a reconciliation triggered exactly at that moment, a volume attachment or a node provisioning, may be retried a few seconds later. Rotate outside of a scaling event or a deployment window if you want to avoid the retry. ### Rotate the CCM credentials The Exoscale Cloud Controller Manager manages `LoadBalancer` services (Network Load Balancers), node TLS bootstrapping and Node metadata. Rotate its credentials with: ```bash exo compute sks rotate-ccm-credentials --zone de-fra-1 my-cluster ``` ### Rotate the CSI credentials The Exoscale Container Storage Interface provisions and attaches Block Storage volumes to your nodes. Rotate its credentials with: ```bash exo compute sks rotate-csi-credentials --zone de-fra-1 my-cluster ``` ### Rotate the Karpenter credentials [Karpenter](https://community.exoscale.com/product/compute/containers/how-to/karpenter/index.md) creates and deletes compute instances on your behalf, so it holds the most privileged addon role. Rotate its credentials with: ```bash exo compute sks rotate-karpenter-credentials --zone de-fra-1 my-cluster ``` > [!NOTE] > Rotation is only relevant while the Karpenter addon is enabled on the cluster. It has no effect on the `NodePool` and `ExoscaleNodeClass` resources you deployed, nor on the Nodes already provisioned. ### Rotate the operators CA Unlike the three operations above, no IAM credential is involved here. This replaces the certificate authority that signs the client certificates in the kubeconfig files generated for your cluster, and updates the trust store of the API server accordingly. ```bash exo compute sks rotate-operators-ca --zone de-fra-1 my-cluster ``` > [!WARNING] > **Every kubeconfig previously generated for this cluster stops working immediately.** Their client certificates were signed by the old CA, which the API server no longer trusts. Anyone who needs `kubectl` access, including your CI/CD pipelines and any tool holding a stored kubeconfig, must generate a new one: > > ```bash > exo compute sks kubeconfig --zone de-fra-1 my-cluster admin -g system:masters > $HOME/.kube/my-cluster.config > ``` This is the way to **revoke cluster access before previous certificates expire**. Kubeconfigs are issued with a TTL, 30 days by default on the portal, and there is no per-certificate revocation: rotating the CA is what invalidates them all at once. Use it when: - A kubeconfig leaked, was committed to a repository, or sat on a lost or compromised machine. - An administrator left the team and held a long-lived kubeconfig. - Your security policy mandates a periodic CA renewal. Plan the rotation with the people and pipelines that need access, so new kubeconfigs are issued right after. --- url: https://community.exoscale.com/product/compute/containers/how-to/feature-gates/ --- # Use Feature Gates on SKS Kubernetes can be extended with Feature Gates allowing you to control specific Kubernetes features with Exoscale SKS. Exoscale SKS supports Feature Gates for both Control Plane (SKS Cluster) and Data Plane (SKS NodePool). Feature Gates support in Exoscale SKS allows to: * enable Feature Gates which are disabled by default; * disable previously enabled Feature Gates. Feature Gates can be enabled or disabled at any time during the lifecycle of a SKS Cluster, i.e. when creating or updating a SKS Cluster. Changes to the Feature Gate list at the SKS Cluster level are transitively propagated to SKS Nodepool. However, SKS Nodepool must be [upgraded][nodepool-upgrade] after enabling or disabling Feature Gates for the change to be applied. Feature Gates are supported by Exoscale SKS for all Kubernetes releases above and including 1.33.7, 1.34.3 and 1.35.0. Each Kubernetes version comes with a list of [supported Feature Gates](#supported-feature-gates) which are a subset of the Kubernetes Feature Gates. Only Feature Gates supported for a given Kubernetes version can be enabled. At the API level, the [`feature_gates` field][feature-gates-field] is an array of strings. The full list of Feature Gates must be given when modified. The full list of Feature Gates given to the API endpoint specifies the list of **enabled** Feature Gates. Any [supported Feature Gate](#supported-feature-gates) not listed by the API endpoint should be considered disabled. [feature-gates-field]: https://community.exoscale.com/reference/api/sks/cluster/#create-sks-cluster ## Prerequisites * A running Exoscale SKS Cluster (Control Plane) * A running Exoscale SKS NodePool for any Feature Gates applying to the Data Plane (Kubelet) ## Enabling Feature Gates We can enable Feature Gates when we create or update a SKS Cluster. For example, to enable the `ImageVolume` Feature Gates when creating a SKS Cluster, we can use the following command via exo CLI: ```shell ➜ exo compute sks create cluster-001 --feature-gates ImageVolume ✔ Creating SKS cluster "cluster-001"... 1m9s ┼───────────────────┼────────────────────────────────────────────────────────────┼ │ SKS CLUSTER │ │ ┼───────────────────┼────────────────────────────────────────────────────────────┼ │ ID │ ca0317bd-7ea9-4ffa-8720-89ee947761e2 │ │ Name │ cluster-001 │ │ Description │ │ │ Zone │ ch-gva-2 │ │ Creation Date │ 2026-01-19 15:50:39 +0000 UTC │ │ Auto-upgrade │ false │ │ Enable kube-proxy │ true │ │ Endpoint │ ca0317bd-7ea9-4ffa-8720-89ee947761e2.ppsks-ch-gva-2.exo.io │ │ Version │ 1.35.0 │ │ Service Level │ pro │ │ CNI │ calico │ │ Add-Ons │ exoscale-cloud-controller │ │ │ metrics-server │ │ Audit Enabled │ false │ │ Feature Gates │ ImageVolume │ │ State │ running │ │ Labels │ n/a │ │ Nodepools │ n/a │ ┼───────────────────┼────────────────────────────────────────────────────────────┼ ``` Respectively, to also enable the `MemoryQoS` Feature Gate for the same example cluster, we can update the cluster via the exo CLI command: ```shell ➜ exo compute sks update cluster-001 --feature-gates "ImageVolume,MemoryQoS" ✔ Updating SKS cluster "cluster-001"... 48s ┼───────────────────┼────────────────────────────────────────────────────────────┼ │ SKS CLUSTER │ │ ┼───────────────────┼────────────────────────────────────────────────────────────┼ │ ID │ ca0317bd-7ea9-4ffa-8720-89ee947761e2 │ │ Name │ cluster-001 │ │ Description │ │ │ Zone │ ch-gva-2 │ │ Creation Date │ 2026-01-19 15:50:39 +0000 UTC │ │ Auto-upgrade │ false │ │ Enable kube-proxy │ true │ │ Endpoint │ ca0317bd-7ea9-4ffa-8720-89ee947761e2.ppsks-ch-gva-2.exo.io │ │ Version │ 1.35.0 │ │ Service Level │ pro │ │ CNI │ calico │ │ Add-Ons │ exoscale-cloud-controller │ │ │ metrics-server │ │ Audit Enabled │ false │ │ Feature Gates │ ImageVolume │ │ │ MemoryQoS │ │ State │ running │ │ Labels │ n/a │ │ Nodepools │ n/a │ ┼───────────────────┼────────────────────────────────────────────────────────────┼ ``` > [!NOTE] > The full list of Feature Gates must be specified when modified via the API > endpoint hence why we include the previously enabled `ImageVolume` Feature > Gate. Only specifying `MemoryQoS` would have enabled the `MemoryQoS` Feature > Gate while the `ImageVolume` Feature Gate would have been disabled. > [!TIP] > The `--feature-gates` option from exo CLI uses a comma-separated list of > strings to define a list of Feature Gates. ## Disabling Feature Gates We can disable previously enabled Feature Gates at any time by updating the given SKS Cluster. Since Feature Gates support for SKS Cluster comes as a toggling list, disabling Feature Gates from an SKS Cluster means we have to update the given cluster with the cluster's Feature Gate list minus the Feature Gates that we want to disable. As an example, let's use the cluster definition from the section above: ```shell ➜ exo compute sks get cluster-001 ┼───────────────────┼────────────────────────────────────────────────────────────┼ │ SKS CLUSTER │ │ ┼───────────────────┼────────────────────────────────────────────────────────────┼ │ ID │ ca0317bd-7ea9-4ffa-8720-89ee947761e2 │ │ Name │ cluster-001 │ │ Description │ │ │ Zone │ ch-gva-2 │ │ Creation Date │ 2026-01-19 15:50:39 +0000 UTC │ │ Auto-upgrade │ false │ │ Enable kube-proxy │ true │ │ Endpoint │ ca0317bd-7ea9-4ffa-8720-89ee947761e2.ppsks-ch-gva-2.exo.io │ │ Version │ 1.35.0 │ │ Service Level │ pro │ │ CNI │ calico │ │ Add-Ons │ exoscale-cloud-controller │ │ │ metrics-server │ │ Audit Enabled │ false │ │ Feature Gates │ ImageVolume │ │ │ MemoryQoS │ │ State │ running │ │ Labels │ n/a │ │ Nodepools │ n/a │ ┼───────────────────┼────────────────────────────────────────────────────────────┼ ``` Following this example, to remove the `MemoryQoS` Feature Gate from cluster-001 we can use the following exo CLI command: ```shell ➜ exo compute sks update cluster-001 --feature-gates "ImageVolume" ✔ Updating SKS cluster "cluster-001"... 45s ┼───────────────────┼────────────────────────────────────────────────────────────┼ │ SKS CLUSTER │ │ ┼───────────────────┼────────────────────────────────────────────────────────────┼ │ ID │ ca0317bd-7ea9-4ffa-8720-89ee947761e2 │ │ Name │ cluster-001 │ │ Description │ │ │ Zone │ ch-gva-2 │ │ Creation Date │ 2026-01-19 15:50:39 +0000 UTC │ │ Auto-upgrade │ false │ │ Enable kube-proxy │ true │ │ Endpoint │ ca0317bd-7ea9-4ffa-8720-89ee947761e2.ppsks-ch-gva-2.exo.io │ │ Version │ 1.35.0 │ │ Service Level │ pro │ │ CNI │ calico │ │ Add-Ons │ exoscale-cloud-controller │ │ │ metrics-server │ │ Audit Enabled │ false │ │ Feature Gates │ ImageVolume │ │ State │ running │ │ Labels │ n/a │ │ Nodepools │ n/a │ ┼───────────────────┼────────────────────────────────────────────────────────────┼ ``` To reset the Feature Gate list from a given SKS Cluster, i.e. to disable all Feature Gates previously enabled, we update the SKS Cluster with an empty Feature Gate list. Continuing with the example above, we disable all Feature Gates for the targeted SKS Cluster via the exo CLI command: ```shell ➜ exo compute sks update cluster-001 --feature-gates "" ✔ Updating SKS cluster "cluster-001"... 45s ┼───────────────────┼────────────────────────────────────────────────────────────┼ │ SKS CLUSTER │ │ ┼───────────────────┼────────────────────────────────────────────────────────────┼ │ ID │ ca0317bd-7ea9-4ffa-8720-89ee947761e2 │ │ Name │ cluster-001 │ │ Description │ │ │ Zone │ ch-gva-2 │ │ Creation Date │ 2026-01-19 15:50:39 +0000 UTC │ │ Auto-upgrade │ false │ │ Enable kube-proxy │ true │ │ Endpoint │ ca0317bd-7ea9-4ffa-8720-89ee947761e2.ppsks-ch-gva-2.exo.io │ │ Version │ 1.35.0 │ │ Service Level │ pro │ │ CNI │ calico │ │ Add-Ons │ exoscale-cloud-controller │ │ │ metrics-server │ │ Audit Enabled │ false │ │ Feature Gates │ │ │ State │ running │ │ Labels │ n/a │ │ Nodepools │ n/a │ ┼───────────────────┼────────────────────────────────────────────────────────────┼ ``` ## Supported Feature Gates Exoscale SKS allows a subset of all available Kubernetes Feature Gates. Mainly, those are Alpha-staged Feature Gates which are disabled by default. Beta-staged Feature Gates are enabled by default. More details are available in the [Kubernetes feature stages documentation][feature-stages]. Below you can find the available Feature Gates for Exoscale SKS per Kubernetes version. For more details about each Feature Gate refer to the [Feature Gates reference documentation][feature-reference]. [feature-stages]: https://kubernetes.io/docs/reference/command-line-tools-reference/feature-gates/#feature-stages [feature-reference]: https://kubernetes.io/docs/reference/command-line-tools-reference/feature-gates/#feature-gates ### Kubernetes v1.37 List of supported Feature Gates: - `SELinuxMountReadWriteOncePod` - `UnknownVersionInteroperabilityProxy` - `JobManagedBy` - `ImageVolumeWithDigest` - `KubeletEnsureSecretPulledImages` - `WinOverlay` - `NativeHistograms` - `DRAPartitionableDevices` - `PodGroupPreemptionPolicy` - `PodDeletionCost` - `InterPodAffinityHostnameFastPath` - `UserNamespacesHostNetworkSupport` - `KubeProxyNFTablesLocalhostNodePorts` - `ServiceAccountTokenNodeBindingValidation` - `KubeletTracing` - `ImageMaximumGCAge` - `NodeInclusionPolicyInPodTopologySpread` - `RestartAllContainersOnContainerExits` - `DRAAdminAccess` - `EmptyDirVolumeMode` - `LocalStorageCapacityIsolationFSQuotaMonitoring` - `DRADeviceTaints` - `EventedPLEG` - `ServiceAccountTokenJTI` - `PodSchedulingReadiness` - `NFTablesProxyMode` - `RemoteRequestHeaderUID` - `ManifestBasedAdmissionControlConfig` - `HPAConfigurableTolerance` - `PodObservedGenerationTracking` - `MatchLabelKeysInPodAffinity` - `DRAPartitionableDevicesType` - `CRDValidationRatcheting` - `NominatedNodeNameForExpectation` - `AtomicWriteVolumeUserFields` - `TopologyManagerPolicyOptions` - `ExternalServiceAccountTokenSigner` - `VolumeLimitScaling` - `StorageVersionMigrator` - `StatefulSetRecreateStrategy` - `MutablePVNodeAffinity` - `InPlacePodVerticalScalingSchedulerPreemption` - `ConsistentListFromCacheSkipTimeoutFallback` - `NodeLifecycleConditions` - `MutablePodResourcesForSuspendedJobs` - `AllowUnsafeMalformedObjectDeletion` - `SELinuxMount` - `VolumeBindMountOptions` - `PersistentVolumeClaimUnusedSinceTime` - `DRADerivedAttributes` - `PodCertificateRequest` - `RecursiveReadOnlyMounts` - `InPlacePodLevelResourcesVerticalScaling` - `MatchLabelKeysInPodTopologySpread` - `WatchList` - `KubeletCgroupDriverFromCRI` - `MemoryQoS` - `ReduceDefaultCrashLoopBackOffDecay` - `StorageVersionHash` - `SELinuxChangePolicy` - `DisableAllocatorDualWrite` - `DRAResourcePoolStatus` - `GracefulNodeShutdown` - `TopologyAwareWorkloadScheduling` - `NodeSwap` - `RecoverVolumeExpansionFailure` - `ClusterTrustBundleProjection` - `CustomResourceFieldSelectors` - `ControllerManagerReleaseLeaderElectionLockOnExit` - `DRAConsumableCapacity` - `RuntimeClassInImageCriApi` - `H2CContainerProbe` - `GenericWorkload` - `ContainerRestartRules` - `PodsAPI` - `KubeletAllocatedPodsEndpoint` - `LoggingAlphaOptions` - `WindowsGracefulNodeShutdown` - `VolumeAttributesClass` - `ComponentStatusz` - `DisableNodeKubeProxyVersion` - `WindowsHostNetwork` - `KubeletSeparateDiskGC` - `NodeDeclaredFeatures` - `DRADeviceTaintRules` - `APIResponseCompression` - `ServiceAccountTokenPodNodeInfo` - `KubeletCrashLoopBackOffMax` - `ComponentFlagz` - `ServiceAccountTokenNodeBinding` - `KubeletServiceAccountTokenForCredentialProviders` - `PodLevelResourceManagers` - `MaxUnavailableStatefulSet` - `KubeletFineGrainedAuthz` - `ShardedListAndWatch` - `OpenAPIEnums` - `RelaxedEnvironmentVariableValidation` - `CPUManagerPolicyOptions` - `PodLevelResources` - `SchedulerPreQueueingHints` - `ImageVolume` - `DRAResourceClaimDeviceStatus` - `APIServerIdentity` - `DRANodeAllocatableResources` - `ContainerStopSignals` - `DRAOptionalNodeOperations` - `CompositePodGroup` - `ClusterTrustBundle` - `PodAndContainerStatsFromCRI` - `InPlacePodVerticalScaling` - `WinDSR` - `KubeletPodResourcesGet` - `InPlacePodVerticalScalingExclusiveCPUs` - `StorageCapacityScoring` - `PortForwardWebsockets` - `CloudControllerManagerWebhook` - `HPAScaleToZero` - `CloudControllerManagerWatchBasedRoutesReconciliation` - `DRADeviceBindingConditions` - `ContextualLogging` - `GracefulNodeShutdownBasedOnPodPriority` - `KubeletInUserNamespace` - `NodeLogQuery` - `MultiCIDRServiceAllocator` - `GRPCContainerProbeTLS` - `InPlacePodVerticalScalingMemoryBackedVolumes` - `CPUManagerPolicyAlphaOptions` - `CoordinatedLeaderElection` - `DRAExtendedResource` - `ConstrainedImpersonation` - `CustomCPUCFSQuotaPeriod` - `ProcMountType` - `ResourceHealthStatus` - `HostnameOverride` - `UserNamespacesSupport` - `EvictionRequestAPI` - `TaintTolerationComparisonOperators` - `ClearingNominatedNodeNameAfterBinding` - `WindowsCPUAndMemoryAffinity` - `RotateKubeletServerCertificate` - `PodReadyToStartContainersCondition` - `MutatingAdmissionPolicy` - `TranslateStreamCloseWebsocketRequests` - `MutableCSINodeAllocatableCount` - `SchedulerAsyncPreemption` - `RelaxedServiceNameValidation` - `APIServingWithRoutine` - `PodLogsQuerySplitStreams` - `DeploymentReplicaSetTerminatingReplicas` - `DRAListTypeAttributes` - `APIServerWebhookAuthenticationToken` - `KubeletPSI` - `MutableSchedulingDirectivesForSuspendedJobs` - `SupplementalGroupsPolicy` - `InPlacePodVerticalScalingExclusiveMemory` - `DynamicResourceAllocation` - `CSIVolumeHealth` - `DefaultPodSysctls` - `CBORServingAndStorage` - `TopologyManagerPolicyAlphaOptions` - `QOSReserved` - `CrossNamespaceVolumeDataSource` - `StorageVersionAPI` - `PreferSameTrafficDistribution` - `DRADeviceCompatibilityGroups` - `EnvFiles` - `DRAPrioritizedList` - `StrictIPCIDRValidation` - `DRAWorkloadResourceClaims` - `PodTopologyLabelsAdmission` - `CRIListStreaming` - `WorkloadWithJob` - `ContainerCheckpoint` - `KubeletPodResourcesDynamicResources` ### Kubernetes v1.36 List of supported Feature Gates: - `SELinuxMountReadWriteOncePod` - `UnknownVersionInteroperabilityProxy` - `JobManagedBy` - `ImageVolumeWithDigest` - `KubeletEnsureSecretPulledImages` - `AnonymousAuthConfigurableEndpoints` - `WinOverlay` - `NativeHistograms` - `DRAPartitionableDevices` - `PodDeletionCost` - `SidecarContainers` - `UserNamespacesHostNetworkSupport` - `ServiceAccountTokenNodeBindingValidation` - `KubeletTracing` - `ImageMaximumGCAge` - `NodeInclusionPolicyInPodTopologySpread` - `RestartAllContainersOnContainerExits` - `DRAAdminAccess` - `LocalStorageCapacityIsolationFSQuotaMonitoring` - `DRADeviceTaints` - `PodLifecycleSleepAction` - `EventedPLEG` - `ServiceAccountTokenJTI` - `WorkloadAwarePreemption` - `PodSchedulingReadiness` - `NFTablesProxyMode` - `RemoteRequestHeaderUID` - `ManifestBasedAdmissionControlConfig` - `AnyVolumeDataSource` - `HPAConfigurableTolerance` - `PodObservedGenerationTracking` - `MatchLabelKeysInPodAffinity` - `CRDValidationRatcheting` - `NominatedNodeNameForExpectation` - `TopologyManagerPolicyOptions` - `ExternalServiceAccountTokenSigner` - `VolumeLimitScaling` - `StorageVersionMigrator` - `MutablePVNodeAffinity` - `LogarithmicScaleDown` - `GangScheduling` - `JobPodReplacementPolicy` - `APIServerTracing` - `MutablePodResourcesForSuspendedJobs` - `AllowUnsafeMalformedObjectDeletion` - `SELinuxMount` - `PersistentVolumeClaimUnusedSinceTime` - `PodCertificateRequest` - `RecursiveReadOnlyMounts` - `InPlacePodLevelResourcesVerticalScaling` - `MatchLabelKeysInPodTopologySpread` - `WatchList` - `AuthorizeNodeWithSelectors` - `KubeletCgroupDriverFromCRI` - `MemoryQoS` - `ReduceDefaultCrashLoopBackOffDecay` - `StorageVersionHash` - `SELinuxChangePolicy` - `DisableAllocatorDualWrite` - `DRAResourcePoolStatus` - `GracefulNodeShutdown` - `TopologyAwareWorkloadScheduling` - `NodeSwap` - `RecoverVolumeExpansionFailure` - `ClusterTrustBundleProjection` - `CustomResourceFieldSelectors` - `ControllerManagerReleaseLeaderElectionLockOnExit` - `DRAConsumableCapacity` - `GenericWorkload` - `ContainerRestartRules` - `PodsAPI` - `LoggingAlphaOptions` - `JobBackoffLimitPerIndex` - `WindowsGracefulNodeShutdown` - `CPUCFSQuotaPeriod` - `VolumeAttributesClass` - `ComponentStatusz` - `DisableNodeKubeProxyVersion` - `WindowsHostNetwork` - `KubeletSeparateDiskGC` - `NodeDeclaredFeatures` - `DRADeviceTaintRules` - `APIResponseCompression` - `ServiceAccountTokenPodNodeInfo` - `KubeletCrashLoopBackOffMax` - `ComponentFlagz` - `ServiceAccountTokenNodeBinding` - `KubeletServiceAccountTokenForCredentialProviders` - `PodLevelResourceManagers` - `MaxUnavailableStatefulSet` - `KubeletFineGrainedAuthz` - `ShardedListAndWatch` - `OpenAPIEnums` - `RelaxedEnvironmentVariableValidation` - `CPUManagerPolicyOptions` - `RetryGenerateName` - `ConsistentListFromCache` - `AuthorizeWithSelectors` - `PodLevelResources` - `ImageVolume` - `DRAResourceClaimDeviceStatus` - `APIServerIdentity` - `DRANodeAllocatableResources` - `ContainerStopSignals` - `ClusterTrustBundle` - `PodAndContainerStatsFromCRI` - `InPlacePodVerticalScaling` - `WinDSR` - `KubeletPodResourcesGet` - `InPlacePodVerticalScalingExclusiveCPUs` - `StorageCapacityScoring` - `PortForwardWebsockets` - `CloudControllerManagerWebhook` - `HPAScaleToZero` - `JobSuccessPolicy` - `CloudControllerManagerWatchBasedRoutesReconciliation` - `DRADeviceBindingConditions` - `ContextualLogging` - `GracefulNodeShutdownBasedOnPodPriority` - `KubeletInUserNamespace` - `NodeLogQuery` - `MultiCIDRServiceAllocator` - `CPUManagerPolicyAlphaOptions` - `CoordinatedLeaderElection` - `DRAExtendedResource` - `ConstrainedImpersonation` - `ProcMountType` - `ResourceHealthStatus` - `HostnameOverride` - `UserNamespacesSupport` - `TaintTolerationComparisonOperators` - `ClearingNominatedNodeNameAfterBinding` - `WindowsCPUAndMemoryAffinity` - `RotateKubeletServerCertificate` - `PodReadyToStartContainersCondition` - `MutatingAdmissionPolicy` - `TranslateStreamCloseWebsocketRequests` - `MutableCSINodeAllocatableCount` - `SchedulerAsyncPreemption` - `RelaxedServiceNameValidation` - `APIServingWithRoutine` - `PodLogsQuerySplitStreams` - `DeploymentReplicaSetTerminatingReplicas` - `DRAListTypeAttributes` - `KubeletPSI` - `RelaxedDNSSearchValidation` - `MutableSchedulingDirectivesForSuspendedJobs` - `SupplementalGroupsPolicy` - `InPlacePodVerticalScalingExclusiveMemory` - `DynamicResourceAllocation` - `CSIVolumeHealth` - `CBORServingAndStorage` - `TopologyManagerPolicyAlphaOptions` - `QOSReserved` - `CrossNamespaceVolumeDataSource` - `StructuredAuthenticationConfiguration` - `StorageVersionAPI` - `PodLifecycleSleepActionAllowZero` - `PreferSameTrafficDistribution` - `EnvFiles` - `DRAPrioritizedList` - `StrictIPCIDRValidation` - `DRAWorkloadResourceClaims` - `PodTopologyLabelsAdmission` - `RuntimeClassInImageCriAPI` - `CRIListStreaming` - `WorkloadWithJob` - `ContainerCheckpoint` - `KubeletPodResourcesDynamicResources` ### Kubernetes v1.35 List of supported Feature Gates: - `SELinuxMountReadWriteOncePod` - `UnknownVersionInteroperabilityProxy` - `JobManagedBy` - `KubeletEnsureSecretPulledImages` - `AnonymousAuthConfigurableEndpoints` - `WinOverlay` - `DRAPartitionableDevices` - `PodDeletionCost` - `SidecarContainers` - `UserNamespacesHostNetworkSupport` - `ServiceAccountTokenNodeBindingValidation` - `KubeletTracing` - `ImageMaximumGCAge` - `NodeInclusionPolicyInPodTopologySpread` - `HonorPVReclaimPolicy` - `RestartAllContainersOnContainerExits` - `DRAAdminAccess` - `LocalStorageCapacityIsolationFSQuotaMonitoring` - `DRADeviceTaints` - `PodLifecycleSleepAction` - `EventedPLEG` - `ServiceAccountTokenJTI` - `PodSchedulingReadiness` - `NFTablesProxyMode` - `RemoteRequestHeaderUID` - `AnyVolumeDataSource` - `HPAConfigurableTolerance` - `PodObservedGenerationTracking` - `MatchLabelKeysInPodAffinity` - `CRDValidationRatcheting` - `NominatedNodeNameForExpectation` - `TopologyManagerPolicyOptions` - `ExternalServiceAccountTokenSigner` - `VolumeLimitScaling` - `StorageVersionMigrator` - `MutablePVNodeAffinity` - `LogarithmicScaleDown` - `GangScheduling` - `JobPodReplacementPolicy` - `APIServerTracing` - `MutablePodResourcesForSuspendedJobs` - `AllowUnsafeMalformedObjectDeletion` - `SELinuxMount` - `InTreePluginPortworxUnregister` - `PodCertificateRequest` - `RecursiveReadOnlyMounts` - `InPlacePodLevelResourcesVerticalScaling` - `MatchLabelKeysInPodTopologySpread` - `WatchList` - `AuthorizeNodeWithSelectors` - `KubeletCgroupDriverFromCRI` - `MemoryQoS` - `ReduceDefaultCrashLoopBackOffDecay` - `StorageVersionHash` - `SELinuxChangePolicy` - `DisableAllocatorDualWrite` - `GracefulNodeShutdown` - `ServiceTrafficDistribution` - `NodeSwap` - `RecoverVolumeExpansionFailure` - `ClusterTrustBundleProjection` - `CustomResourceFieldSelectors` - `DRAConsumableCapacity` - `GenericWorkload` - `ContainerRestartRules` - `LoggingAlphaOptions` - `JobBackoffLimitPerIndex` - `WindowsGracefulNodeShutdown` - `CPUCFSQuotaPeriod` - `VolumeAttributesClass` - `ComponentStatusz` - `DisableNodeKubeProxyVersion` - `WindowsHostNetwork` - `KubeletSeparateDiskGC` - `NodeDeclaredFeatures` - `DRADeviceTaintRules` - `APIResponseCompression` - `ServiceAccountTokenPodNodeInfo` - `KubeletCrashLoopBackOffMax` - `ComponentFlagz` - `ServiceAccountTokenNodeBinding` - `KubeletServiceAccountTokenForCredentialProviders` - `MaxUnavailableStatefulSet` - `KubeletFineGrainedAuthz` - `OpenAPIEnums` - `RelaxedEnvironmentVariableValidation` - `CPUManagerPolicyOptions` - `RetryGenerateName` - `TopologyAwareHints` - `ConsistentListFromCache` - `AuthorizeWithSelectors` - `PodLevelResources` - `ImageVolume` - `DRAResourceClaimDeviceStatus` - `APIServerIdentity` - `ContainerStopSignals` - `ClusterTrustBundle` - `PodAndContainerStatsFromCRI` - `InPlacePodVerticalScaling` - `InPlacePodVerticalScalingAllocatedStatus` - `WinDSR` - `KubeletPodResourcesGet` - `InPlacePodVerticalScalingExclusiveCPUs` - `StorageCapacityScoring` - `PortForwardWebsockets` - `CloudControllerManagerWebhook` - `HPAScaleToZero` - `JobSuccessPolicy` - `CloudControllerManagerWatchBasedRoutesReconciliation` - `DRADeviceBindingConditions` - `ContextualLogging` - `GracefulNodeShutdownBasedOnPodPriority` - `KubeletInUserNamespace` - `NodeLogQuery` - `MultiCIDRServiceAllocator` - `CPUManagerPolicyAlphaOptions` - `CoordinatedLeaderElection` - `DRAExtendedResource` - `ConstrainedImpersonation` - `ProcMountType` - `ResourceHealthStatus` - `HostnameOverride` - `UserNamespacesSupport` - `TaintTolerationComparisonOperators` - `ClearingNominatedNodeNameAfterBinding` - `WindowsCPUAndMemoryAffinity` - `RotateKubeletServerCertificate` - `PodReadyToStartContainersCondition` - `MutatingAdmissionPolicy` - `TranslateStreamCloseWebsocketRequests` - `MutableCSINodeAllocatableCount` - `SchedulerAsyncPreemption` - `RelaxedServiceNameValidation` - `APIServingWithRoutine` - `PodLogsQuerySplitStreams` - `DeploymentReplicaSetTerminatingReplicas` - `CSIMigrationPortworx` - `KubeletPSI` - `RelaxedDNSSearchValidation` - `MutableSchedulingDirectivesForSuspendedJobs` - `SupplementalGroupsPolicy` - `InPlacePodVerticalScalingExclusiveMemory` - `DynamicResourceAllocation` - `MemoryManager` - `CSIVolumeHealth` - `CBORServingAndStorage` - `StructuredAuthorizationConfiguration` - `TopologyManagerPolicyAlphaOptions` - `QOSReserved` - `CrossNamespaceVolumeDataSource` - `StructuredAuthenticationConfiguration` - `StorageVersionAPI` - `PodLifecycleSleepActionAllowZero` - `PreferSameTrafficDistribution` - `EnvFiles` - `DRAPrioritizedList` - `StrictIPCIDRValidation` - `PodTopologyLabelsAdmission` - `RuntimeClassInImageCriAPI` - `ContainerCheckpoint` - `KubeletPodResourcesDynamicResources` ### Kubernetes v1.34 List of supported Feature Gates: - `SELinuxMountReadWriteOncePod` - `UnknownVersionInteroperabilityProxy` - `JobManagedBy` - `KubeletEnsureSecretPulledImages` - `AnonymousAuthConfigurableEndpoints` - `WinOverlay` - `DRAPartitionableDevices` - `PodDeletionCost` - `SidecarContainers` - `ServiceAccountTokenNodeBindingValidation` - `KubeletTracing` - `ImageMaximumGCAge` - `NodeInclusionPolicyInPodTopologySpread` - `HonorPVReclaimPolicy` - `DRAAdminAccess` - `LocalStorageCapacityIsolationFSQuotaMonitoring` - `DRADeviceTaints` - `PodLifecycleSleepAction` - `EventedPLEG` - `ServiceAccountTokenJTI` - `PodSchedulingReadiness` - `StatefulSetAutoDeletePVC` - `NFTablesProxyMode` - `RemoteRequestHeaderUID` - `AnyVolumeDataSource` - `HPAConfigurableTolerance` - `PodObservedGenerationTracking` - `MatchLabelKeysInPodAffinity` - `CRDValidationRatcheting` - `NominatedNodeNameForExpectation` - `TopologyManagerPolicyOptions` - `ExternalServiceAccountTokenSigner` - `StorageVersionMigrator` - `LogarithmicScaleDown` - `JobPodReplacementPolicy` - `APIServerTracing` - `AllowUnsafeMalformedObjectDeletion` - `SELinuxMount` - `InTreePluginPortworxUnregister` - `PodCertificateRequest` - `RecursiveReadOnlyMounts` - `MatchLabelKeysInPodTopologySpread` - `WatchList` - `AuthorizeNodeWithSelectors` - `KubeletCgroupDriverFromCRI` - `MemoryQoS` - `ReduceDefaultCrashLoopBackOffDecay` - `StorageVersionHash` - `SELinuxChangePolicy` - `DisableAllocatorDualWrite` - `GracefulNodeShutdown` - `ServiceTrafficDistribution` - `NodeSwap` - `RecoverVolumeExpansionFailure` - `ClusterTrustBundleProjection` - `CustomResourceFieldSelectors` - `DRAConsumableCapacity` - `ContainerRestartRules` - `LoadBalancerIPMode` - `LoggingAlphaOptions` - `JobBackoffLimitPerIndex` - `WindowsGracefulNodeShutdown` - `CPUCFSQuotaPeriod` - `VolumeAttributesClass` - `ComponentStatusz` - `DisableNodeKubeProxyVersion` - `WindowsHostNetwork` - `KubeletSeparateDiskGC` - `APIResponseCompression` - `ServiceAccountTokenPodNodeInfo` - `KubeletCrashLoopBackOffMax` - `ComponentFlagz` - `ServiceAccountTokenNodeBinding` - `KubeletServiceAccountTokenForCredentialProviders` - `MaxUnavailableStatefulSet` - `UserNamespacesPodSecurityStandards` - `ListFromCacheSnapshot` - `KubeletFineGrainedAuthz` - `OpenAPIEnums` - `RelaxedEnvironmentVariableValidation` - `CPUManagerPolicyOptions` - `RetryGenerateName` - `TopologyAwareHints` - `ConsistentListFromCache` - `AuthorizeWithSelectors` - `PodLevelResources` - `ImageVolume` - `DRAResourceClaimDeviceStatus` - `APIServerIdentity` - `ContainerStopSignals` - `ClusterTrustBundle` - `PodAndContainerStatsFromCRI` - `InPlacePodVerticalScaling` - `InPlacePodVerticalScalingAllocatedStatus` - `SizeMemoryBackedVolumes` - `WinDSR` - `KubeletPodResourcesGet` - `InPlacePodVerticalScalingExclusiveCPUs` - `StorageCapacityScoring` - `PortForwardWebsockets` - `CloudControllerManagerWebhook` - `HPAScaleToZero` - `JobSuccessPolicy` - `DRADeviceBindingConditions` - `ContextualLogging` - `GracefulNodeShutdownBasedOnPodPriority` - `KubeletInUserNamespace` - `NodeLogQuery` - `MultiCIDRServiceAllocator` - `CPUManagerPolicyAlphaOptions` - `CoordinatedLeaderElection` - `DRAExtendedResource` - `ProcMountType` - `ResourceHealthStatus` - `HostnameOverride` - `UserNamespacesSupport` - `ClearingNominatedNodeNameAfterBinding` - `WindowsCPUAndMemoryAffinity` - `RotateKubeletServerCertificate` - `PodReadyToStartContainersCondition` - `MutatingAdmissionPolicy` - `TranslateStreamCloseWebsocketRequests` - `MutableCSINodeAllocatableCount` - `SchedulerAsyncPreemption` - `RelaxedServiceNameValidation` - `APIServingWithRoutine` - `PodLogsQuerySplitStreams` - `DeploymentReplicaSetTerminatingReplicas` - `CSIMigrationPortworx` - `KubeletPSI` - `RelaxedDNSSearchValidation` - `SupplementalGroupsPolicy` - `InPlacePodVerticalScalingExclusiveMemory` - `DynamicResourceAllocation` - `MemoryManager` - `CSIVolumeHealth` - `CBORServingAndStorage` - `StructuredAuthorizationConfiguration` - `TopologyManagerPolicyAlphaOptions` - `QOSReserved` - `CrossNamespaceVolumeDataSource` - `StructuredAuthenticationConfiguration` - `StorageVersionAPI` - `PodLifecycleSleepActionAllowZero` - `PreferSameTrafficDistribution` - `ComponentSLIs` - `EnvFiles` - `DRAPrioritizedList` - `StrictIPCIDRValidation` - `PodTopologyLabelsAdmission` - `RuntimeClassInImageCriAPI` - `ContainerCheckpoint` - `KubeletPodResourcesDynamicResources` ### Kubernetes v1.33 List of supported Feature Gates: - `SELinuxMountReadWriteOncePod` - `UnknownVersionInteroperabilityProxy` - `JobManagedBy` - `KubeletEnsureSecretPulledImages` - `AnonymousAuthConfigurableEndpoints` - `WinOverlay` - `DRAPartitionableDevices` - `PodDeletionCost` - `SidecarContainers` - `ServiceAccountTokenNodeBindingValidation` - `KubeletTracing` - `ImageMaximumGCAge` - `NodeInclusionPolicyInPodTopologySpread` - `HonorPVReclaimPolicy` - `DRAAdminAccess` - `LocalStorageCapacityIsolationFSQuotaMonitoring` - `DRADeviceTaints` - `PodLifecycleSleepAction` - `EventedPLEG` - `ServiceAccountTokenJTI` - `PodSchedulingReadiness` - `StatefulSetAutoDeletePVC` - `NFTablesProxyMode` - `RemoteRequestHeaderUID` - `AnyVolumeDataSource` - `DevicePluginCDIDevices` - `HPAConfigurableTolerance` - `PodObservedGenerationTracking` - `MatchLabelKeysInPodAffinity` - `CRDValidationRatcheting` - `TopologyManagerPolicyOptions` - `ExternalServiceAccountTokenSigner` - `StorageVersionMigrator` - `PodDisruptionConditions` - `LogarithmicScaleDown` - `JobPodReplacementPolicy` - `APIServerTracing` - `AllowUnsafeMalformedObjectDeletion` - `SELinuxMount` - `InTreePluginPortworxUnregister` - `RecursiveReadOnlyMounts` - `MatchLabelKeysInPodTopologySpread` - `WatchList` - `AuthorizeNodeWithSelectors` - `KubeletCgroupDriverFromCRI` - `MemoryQoS` - `ReduceDefaultCrashLoopBackOffDecay` - `StorageVersionHash` - `SELinuxChangePolicy` - `DisableAllocatorDualWrite` - `GracefulNodeShutdown` - `ServiceTrafficDistribution` - `NodeSwap` - `RecoverVolumeExpansionFailure` - `ClusterTrustBundleProjection` - `CustomResourceFieldSelectors` - `LoadBalancerIPMode` - `LoggingAlphaOptions` - `JobBackoffLimitPerIndex` - `WindowsGracefulNodeShutdown` - `CPUCFSQuotaPeriod` - `VolumeAttributesClass` - `ComponentStatusz` - `DisableNodeKubeProxyVersion` - `WindowsHostNetwork` - `KubeletSeparateDiskGC` - `APIResponseCompression` - `ServiceAccountTokenPodNodeInfo` - `KubeletCrashLoopBackOffMax` - `ComponentFlagz` - `ServiceAccountTokenNodeBinding` - `KubeletServiceAccountTokenForCredentialProviders` - `MaxUnavailableStatefulSet` - `UserNamespacesPodSecurityStandards` - `ListFromCacheSnapshot` - `KubeletFineGrainedAuthz` - `OpenAPIEnums` - `RelaxedEnvironmentVariableValidation` - `CPUManagerPolicyOptions` - `RetryGenerateName` - `TopologyAwareHints` - `ConsistentListFromCache` - `AuthorizeWithSelectors` - `PodLevelResources` - `ImageVolume` - `DRAResourceClaimDeviceStatus` - `APIServerIdentity` - `ContainerStopSignals` - `ClusterTrustBundle` - `PodAndContainerStatsFromCRI` - `InPlacePodVerticalScaling` - `InPlacePodVerticalScalingAllocatedStatus` - `SizeMemoryBackedVolumes` - `WinDSR` - `KubeletPodResourcesGet` - `InPlacePodVerticalScalingExclusiveCPUs` - `StorageCapacityScoring` - `PortForwardWebsockets` - `CloudControllerManagerWebhook` - `HPAScaleToZero` - `JobSuccessPolicy` - `ContextualLogging` - `GracefulNodeShutdownBasedOnPodPriority` - `KubeletInUserNamespace` - `StatefulSetStartOrdinal` - `NodeLogQuery` - `MultiCIDRServiceAllocator` - `CPUManagerPolicyAlphaOptions` - `CoordinatedLeaderElection` - `ProcMountType` - `ResourceHealthStatus` - `UserNamespacesSupport` - `WindowsCPUAndMemoryAffinity` - `RotateKubeletServerCertificate` - `PodReadyToStartContainersCondition` - `MutatingAdmissionPolicy` - `TranslateStreamCloseWebsocketRequests` - `MutableCSINodeAllocatableCount` - `SchedulerAsyncPreemption` - `APIServingWithRoutine` - `PodLogsQuerySplitStreams` - `DeploymentReplicaSetTerminatingReplicas` - `CSIMigrationPortworx` - `KubeletPSI` - `RelaxedDNSSearchValidation` - `SupplementalGroupsPolicy` - `DynamicResourceAllocation` - `MemoryManager` - `CSIVolumeHealth` - `CBORServingAndStorage` - `StructuredAuthorizationConfiguration` - `TopologyManagerPolicyAlphaOptions` - `QOSReserved` - `CrossNamespaceVolumeDataSource` - `StructuredAuthenticationConfiguration` - `StorageVersionAPI` - `PodLifecycleSleepActionAllowZero` - `PreferSameTrafficDistribution` - `ComponentSLIs` - `DRAPrioritizedList` - `StrictIPCIDRValidation` - `PodTopologyLabelsAdmission` - `RuntimeClassInImageCriAPI` - `ContainerCheckpoint` - `KubeletPodResourcesDynamicResources` ## Limitations The following limitations are enforced when using Feature Gates with SKS: * Feature Gates use default configuration only; i.e. it is currently not possible to set a custom configuration for a Feature Gate that allows one. * NodePools **MUST** be [upgraded][nodepool-upgrade] to provide the Feature Gates enabled for SKS. * Feature Gates are supported for both classic SKS NodePools and [Karpenter](https://community.exoscale.com/product/compute/containers/how-to/karpenter/index.md) nodes. For Karpenter, Feature Gates are configured via the `spec.kubelet.featureGates` field of the `ExoscaleNodeClass` resource (see [below](#karpenter-support)). [nodepool-upgrade]: ../../operation/lifecycle-management/#nodepool-upgrade ## Karpenter Support Feature Gates can also be applied to nodes provisioned by [Karpenter](https://community.exoscale.com/product/compute/containers/how-to/karpenter/index.md). Instead of being set at the SKS Cluster or NodePool level, they are configured in the `ExoscaleNodeClass` resource under `spec.kubelet.featureGates`: ```yaml apiVersion: karpenter.exoscale.com/v1 kind: ExoscaleNodeClass metadata: name: default spec: # ... kubelet: featureGates: ImageVolume: true MemoryQoS: true ``` The same [supported Feature Gates](#supported-feature-gates) apply. Karpenter nodes will pick up the configured Feature Gates when they are provisioned; existing nodes are not affected until they are replaced. --- url: https://community.exoscale.com/product/compute/containers/how-to/kubernetes-audit/ --- # Use Kubernetes Audit [Kubernetes Audit](https://kubernetes.io/docs/tasks/debug/debug-cluster/audit/) is a feature in K8s' APIserver to audit requests inbound to the apiserver. By choosing the _webhook_ mode, an operator can listen for audit data on a dedicated HTTP webserver as a source for analytics, monitoring, etc. Exoscale packages Kubernetes Audit in SKS. It can be enabled, disabled and configured at no operational cost other than setting up and managing the receiving Webserver. ## Prerequisites - Knowledge of SKS - how to create and/or update an SKS Cluster - An SKS Cluster (new or existing) with the following conditions: **PRO** offering, and Kubernetes version **>= 1.31.0** - A webserver, ideally supporting **Bearer-token authentication** and **HTTPS** ## Configuration At the moment we support the following parameters: - Endpoint: the URL to send the audit data to - Bearer token: the authentication token for the receiving webserver - Initial Backoff: how long to wait before sending the first batch of data (by default 10 seconds) We deploy a static Audit Policy which integrates with [Falco](https://falco.org/) but can be used with bespoke solutions and offers sane defaults. You can check the contents [here](https://github.com/falcosecurity/plugins/blob/main/plugins/k8saudit/rules/k8s_audit_rules.yaml) The first version of the feature supports bearer token authentication exclusively. Other parameters are set to their default value. ## How-to ### Portal In order to **activate and configure** the feature: - For **existing** SKS clusters: navigate to `SKS > your_cluster > Update Cluster`. Enable the _Kubernetes Audit_ toggle and fill-in the form. - For **new** SKS clusters: navigate to `SKS > Add`. Similarly, enable the toggle and fill in the details. In order to **disable** the feature: - Navigate to `SKS > your_cluster > Update Cluster` and disable the toggle. > [!NOTE] > We do not expose the bearer token. Leaving the *Bearer token* field blank > will preserve it. > [!IMPORTANT] > You won't see the toggle unless the prerequisites are met: PRO offering and > Kubernetes >= 1.31. ![](screenshot.png) ### API Our [OpenAPI Spec](https://community.exoscale.com/reference/api/sks/cluster/index.md#update-sks-cluster) describes how to parametrize Kubernetes audit. Tooling integration (Exoscale Terraform Provider and the CLI) is coming soon --- url: https://community.exoscale.com/product/compute/containers/how-to/openid-connect/ --- # Use OpenID Connect Enable external identity-based authentication for your SKS cluster using OpenID Connect (OIDC). ## Enable External Authentication via OpenID Connect Kubernetes natively supports [OpenID Connect](https://kubernetes.io/docs/reference/access-authn-authz/authentication/#openid-connect-tokens) as an authentication method adding external (and more granular) access control to your SKS cluster via the OAuth2 version found in many identity providers. At creation time, you can launch a SKS cluster by specifying the OpenID Connect parameters so that no further configuration is required to access Kubernetes control plane management. ## Configuring OIDC in SKS via CLI To configure OIDC when creating a SKS cluster, use the following CLI flags: ``` --oidc-client-id string OpenID client ID --oidc-groups-claim string OpenID JWT claim to use as the user's group --oidc-groups-prefix string OpenID prefix prepended to group claims --oidc-issuer-url string OpenID provider URL --oidc-required-claim string OpenID Token that describes a required claim in a key=value pair --oidc-username-claim string OpenID JWT claim to use as the user name --oidc-username-prefix string OpenID prefix prepended to username claims ``` Neither Exoscale nor Kubernetes currently provides an OpenID Connect Identity Provider. You can use an existing public OpenID Connect Identity Provider or you can run your own Identity Provider, such as [Dex](https://dexidp.io/docs/kubernetes/) or [Keycloak](https://github.com/keycloak/keycloak). --- url: https://community.exoscale.com/product/compute/containers/how-to/labels/ --- # Use Labels on SKS Labels can be associated with SKS Cluster, SKS NodePool and SKS Karpenter. The intent and the specifics of each object are detailed in the sections below. ## SKS Cluster SKS Cluster labels are informational only and are not intended to be associated with any workload. Each SKS Cluster can have a set of key/value labels defined. Each Key must be unique for a given object and adhere to the [Kubernetes Labels Key syntax and character set][kubernetes-labels-spec]. Quoting from the Kubernetes documentation: > Valid label keys have two segments: an optional prefix and name, separated by > a slash (`/`). The name segment is required and must be 63 characters or > less, beginning and ending with an alphanumeric character (`[a-z0-9A-Z]`) > with dashes (`-`), underscores (`_`), dots (`.`), and alphanumerics between. > The prefix is optional. If specified, the prefix must be a DNS subdomain: a > series of DNS labels separated by dots (`.`), not longer than 253 characters > in total, followed by a slash (`/`). Valid label value: * must be 128 characters or less (can be empty), * unless empty, must **begin** and **end** with an alphanumeric character (`[a-z0-9A-Z]`), * could contain dashes (`-`), underscores (`_`), dots (`.`), slashes (`/`), hash symbols (`#`), colons (`:`), at signs (`@`), and alphanumerics between. For example, below we create an SKS Cluster with two labels `environment: production` and `exoscale.com/sks-cluster-labels-url: https://community.exoscale.com/product/compute/containers/how-to/labels/#sks-cluster`: ```shell ➜ exo compute sks create cluster-001 \ --label environment=production \ --label exoscale.com/sks-cluster-labels-url=https://community.exoscale.com/product/compute/containers/how-to/labels/#sks-cluster ┼───────────────────┼──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┼ │ SKS CLUSTER │ │ ┼───────────────────┼──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┼ │ ID │ 1d963c64-55b5-4c4f-a5e9-ccfdc59bb2df │ │ Name │ cluster-001 │ │ Description │ │ │ Zone │ ch-gva-2 │ │ Creation Date │ 2026-03-02 15:58:30 +0000 UTC │ │ Auto-upgrade │ false │ │ Enable kube-proxy │ true │ │ Endpoint │ 1d963c64-55b5-4c4f-a5e9-ccfdc59bb2df.ppsks-ch-gva-2.exo.io │ │ Version │ 1.35.0 │ │ Service Level │ pro │ │ CNI │ calico │ │ Add-Ons │ exoscale-cloud-controller │ │ │ metrics-server │ │ Audit Enabled │ false │ │ Feature Gates │ │ │ State │ running │ │ Labels │ environment:production │ │ │ exoscale.com/sks-cluster-labels-url:https://community.exoscale.com/product/compute/containers/how-to/labels/#sks-cluster │ │ Nodepools │ n/a │ ┼───────────────────┼──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┼ ``` [kubernetes-labels-spec]: https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/#syntax-and-character-set ## SKS NodePool Labels can be associated with an SKS NodePool to help classify and organize them. Those labels will be associated with Kubernetes Node provisioned via the SKS NodePool. The underlying Compute Instances will also be tagged with SKS NodePool labels when defined. Labels are key/value pairs and must adhere to the [Kubernetes Labels syntax and character set][kubernetes-labels-spec]. Quoting from the documentation – emphasis ours: > Valid label keys have two segments: an optional prefix and name, separated by > a slash (`/`). The name segment is required and must be 63 characters or > less, beginning and ending with an alphanumeric character (`[a-z0-9A-Z]`) > with dashes (`-`), underscores (`_`), dots (`.`), and alphanumerics between. > The prefix is optional. If specified, the prefix must be a DNS subdomain: a > series of DNS labels separated by dots (`.`), not longer than 253 characters > in total, followed by a slash (`/`). > > The `kubernetes.io/` and `k8s.io/` prefixes are **reserved** for Kubernetes > core components. > > Valid label value: > > * must be 63 characters or less (can be empty), > * unless empty, must **begin** and **end** with an alphanumeric character > (`[a-z0-9A-Z]`), > * could contain dashes (`-`), underscores (`_`), dots (`.`), and > alphanumerics between. For example, below we create an SKS NodePool with two labels `environment: production` and `cloud-provider: exoscale`: ```shell ➜ exo compute sks nodepool add cluster-001 nodepool-001 \ --label environment=production \ --label cloud-provider=exoscale ┼─────────────────────────┼──────────────────────────────────────┼ │ SKS NODEPOOL │ │ ┼─────────────────────────┼──────────────────────────────────────┼ │ ID │ 2268bc0c-2f8c-4445-b676-f8503804fba4 │ │ Name │ nodepool-001 │ │ Description │ │ │ Creation Date │ 2026-03-04 10:10:58 +0000 UTC │ │ Instance Pool ID │ 04f6b3f5-1445-4646-afc2-fccb64c2bc9e │ │ Instance Prefix │ pool │ │ Instance Type │ standard.medium │ │ Template │ sks-node-1.35.0-standard │ │ Disk Size │ 50 │ │ Public IP Assignment │ inet4 │ │ IPv6 │ false │ │ Anti Affinity Groups │ n/a │ │ Security Groups │ n/a │ │ Private Networks │ n/a │ │ Version │ 1.35.0 │ │ Size │ 2 │ │ State │ running │ │ Taints │ n/a │ │ Labels │ cloud-provider:exoscale │ │ │ environment:production │ │ Add Ons │ n/a │ │ Image GC Min │ 2m │ │ Image Gc Low Threshold │ 80 │ │ Image Gc High Threshold │ 85 │ ┼─────────────────────────┼──────────────────────────────────────┼ ``` Once the SKS NodePool created, we can verify its labels are propagated to the underlying Compute Instances: ```shell ➜ exo compute instance list ┼──────────────────────────────────────┼──────────────────┼──────────┼─────────────────┼────────────────┼──────────────┼─────────┼ │ ID │ NAME │ ZONE │ TYPE │ IP ADDRESS │ IPV6 ADDRESS │ STATE │ ┼──────────────────────────────────────┼──────────────────┼──────────┼─────────────────┼────────────────┼──────────────┼─────────┼ │ ebcbadfa-6716-4510-8463-75761e17a683 │ pool-04f6b-rtjun │ ch-gva-2 │ standard.medium │ 85.217.161.236 │ - │ running │ │ a1aafd68-c22e-4759-8bf5-27b26c8835b6 │ pool-04f6b-lyadc │ ch-gva-2 │ standard.medium │ 85.217.161.157 │ - │ running │ ┼──────────────────────────────────────┼──────────────────┼──────────┼─────────────────┼────────────────┼──────────────┼─────────┼ ➜ exo compute instance show pool-04f6b-lyadc ┼─────────────────────────────────────────┼──────────────────────────────────────┼ │ COMPUTE INSTANCE │ │ ┼─────────────────────────────────────────┼──────────────────────────────────────┼ │ ID │ a1aafd68-c22e-4759-8bf5-27b26c8835b6 │ │ Name │ pool-04f6b-lyadc │ │ Creation Date │ 2026-03-04 10:11:00 +0000 UTC │ │ Instance Type │ standard.medium │ │ Template │ sks-node-1.35.0-standard │ │ Zone │ ch-gva-2 │ │ Anti-Affinity Groups │ n/a │ │ Deploy Target │ - │ │ Security Groups │ default │ │ Private Instance │ No │ │ Private Networks │ n/a │ │ Elastic IPs │ n/a │ │ Public IP │ inet4 │ │ IP Address │ 85.217.161.157 │ │ IPv6 Address │ - │ │ SSH Key │ - │ │ Disk Size │ 50 GiB │ │ State │ running │ │ Labels │ cloud-provider:exoscale │ │ │ environment:production │ │ Secure Boot │ false │ │ Tpm │ false │ │ Reverse DNS │ │ │ Application-Consistent Snapshot enabled │ false │ ┼─────────────────────────────────────────┼──────────────────────────────────────┼ ➜ exo compute instance show pool-04f6b-rtjun ┼─────────────────────────────────────────┼──────────────────────────────────────┼ │ COMPUTE INSTANCE │ │ ┼─────────────────────────────────────────┼──────────────────────────────────────┼ │ ID │ ebcbadfa-6716-4510-8463-75761e17a683 │ │ Name │ pool-04f6b-rtjun │ │ Creation Date │ 2026-03-04 10:11:00 +0000 UTC │ │ Instance Type │ standard.medium │ │ Template │ sks-node-1.35.0-standard │ │ Zone │ ch-gva-2 │ │ Anti-Affinity Groups │ n/a │ │ Deploy Target │ - │ │ Security Groups │ default │ │ Private Instance │ No │ │ Private Networks │ n/a │ │ Elastic IPs │ n/a │ │ Public IP │ inet4 │ │ IP Address │ 85.217.161.236 │ │ IPv6 Address │ - │ │ SSH Key │ - │ │ Disk Size │ 50 GiB │ │ State │ running │ │ Labels │ cloud-provider:exoscale │ │ │ environment:production │ │ Secure Boot │ false │ │ Tpm │ false │ │ Reverse DNS │ │ │ Application-Consistent Snapshot enabled │ false │ ┼─────────────────────────────────────────┼──────────────────────────────────────┼ ``` We can also ensure the SKS NodePool labels are associated with the Kubernetes Nodes: ```shell ➜ kubectl --kubeconfig kubeconfig.yml get nodes NAME STATUS ROLES AGE VERSION pool-04f6b-lyadc Ready 5m3s v1.35.0 pool-04f6b-rtjun Ready 4m57s v1.35.0 ➜ kubectl --kubeconfig kubeconfig.yml label node/pool-04f6b-lyadc --list [...] environment=production cloud-provider=exoscale ➜ kubectl --kubeconfig kubeconfig.yml label node/pool-04f6b-rtjun --list [...] cloud-provider=exoscale environment=production ``` ## SKS Karpenter add-on Provided [Karpenter add-on is enabled and configured](https://community.exoscale.com/product/compute/containers/how-to/karpenter/index.md) for a given SKS Cluster, labels can be defined in Karpenter NodePools via the [`spec.template.metadata.labels`](https://karpenter.sh/docs/concepts/nodepools/#spectemplatemetadatalabels) configuration. Those labels will then be associated with Kubernetes Node provisioned hereafter. ## Labeling Rules for the `kubernetes.io` Namespace When using labels in the `kubernetes.io` namespace, you must follow strict naming rules. Labels must either: 1. Start with an allowed prefix: * `kubelet.kubernetes.io` * `node.kubernetes.io` **OR** 2. Be part of the explicitly allowed set: * `beta.kubernetes.io/arch` * `beta.kubernetes.io/instance-type` * `beta.kubernetes.io/os` * `failure-domain.beta.kubernetes.io/region` * `failure-domain.beta.kubernetes.io/zone` * `kubernetes.io/arch` * `kubernetes.io/hostname` * `kubernetes.io/os` * `node.kubernetes.io/instance-type` * `topology.kubernetes.io/region` * `topology.kubernetes.io/zone` If you need another label in the `kubernetes.io` namespace, you will need to set it after the Nodes are registered with: ```bash kubectl label node NODE-NAME unallowed.kubernetes.io/prefix=true ``` --- url: https://community.exoscale.com/product/compute/containers/how-to/managed-instance-prefixes/ --- # Set Instance Prefixes By default, Compute instances managed by an SKS nodepool are named `pool--`. This pattern can be customized by passing the `--instance-prefix` parameter during a nodepool creation or update. For example, if you specify `--instance-prefix=applications`, `pool` will be replaced by `applications` in the Compute instance names. --- url: https://community.exoscale.com/product/compute/containers/how-to/kubernetes-taints/ --- # Use Kubernetes Taints Node affinity is a property of pods that attracts them to a set of nodes. Kubernetes taints, on the other hand, are the opposite: they allow a node to repel a set of pods. You can read the [official Kubernetes documentation](https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/) to learn more about taints and toleration. To associate taints on your nodepool workers, you can use the `--nodepool-taint=KEY=VALUE:EFFECT` option. For instances use the `--nodepool-taint=type=GPU:NoSchedule` option. After a taint is registered on a nodepool worker, only pods with [tolerations](https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/) will be scheduled for that worker. Taints options can be repeated in order to register multiple taints. --- url: https://community.exoscale.com/product/compute/containers/how-to/custom-cni-setup/ --- # Use a Custom CNI The **Container Network Interface (CNI)** defines how network plugins integrate with container runtimes. While SKS clusters come with Calico or Cilium by default, you may choose to use a custom CNI plugin or configuration. To ensure compatibility with your SKS cluster, please review the following requirements for custom CNI usage: 1. **Install the CNI Plugin on All Nodes**: Ensure the CNI plugin is installed across all cluster nodes. Remove any `nodeSelector` constraints, and set broad tolerations so it can be scheduled on any node. 2. **Network Interface Configuration**: Set up the CNI plugin to use the correct network interface for outgoing traffic. 3. **Pod CIDR Range**: The Pod CIDR range should be set to `192.168.0.0/16`. 4. **IPAM Management**: Your custom CNI plugin should not manage Pod CIDR ranges. SKS handles IP Address Management (IPAM) automatically. 5. **Kubernetes API Server Connection**: - Set `KUBERNETES_SERVICE_HOST` and `KUBERNETES_SERVICE_PORT` to point to your SKS API server. These values are typically: - **Host**: `.sks-.exo.io` - **Port**: `443` - You can locate these values in the **Generate Kubeconfig** section of the SKS console. 6. **Service Proxy Conflicts**: If your CNI plugin includes a Service Proxy, disable the default `kube-proxy` (details below). 7. **DNS Configuration**: Ensure the DNS suffix is set to `cluster.local` if your CNI plugin requires DNS settings. Following these guidelines will help ensure your custom CNI works seamlessly within your SKS environment. --- url: https://community.exoscale.com/product/compute/containers/how-to/kube-proxy-removal/ --- # Disable kube-proxy Please note that if you deploy your SKS clusters with a pre-configured CNI plugin, this is already handled by our SKS orchestrator and **does not apply** to your situation. An example of support replacement of kube-proxy is a custom deployment of Cilium [with "strict" kube-proxy replacement][cilium-kube-proxy-replacement]. In this specific situation, you would ideally remove kube-proxy **before** any pods are scheduled in the cluster. To achieve this, you can provision your cluster without kube-proxy: - using [the Terraform provider][terraform-provider-no-kube-proxy] SKS resource with the `enable_kube_proxy` attribute set to `false` - using the [Exoscale CLI][exo-cli] with the `--enable-kube-proxy` flag set to `false` when creating your cluster [cilium-kube-proxy-replacement]: https://docs.cilium.io/en/v1.18/network/kubernetes/kubeproxy-free/#kubeproxy-free [terraform-provider-no-kube-proxy]: https://registry.terraform.io/providers/exoscale/exoscale/latest/docs/resources/sks_cluster#enable_kube_proxy-1 [exo-cli]: https://github.com/exoscale/cli --- url: https://community.exoscale.com/product/compute/containers/how-to/user-namespaces/ --- # Use User Namespaces User namespaces provide enhanced security isolation by mapping container user and group IDs to different IDs on the host. This means a process running as root (UID 0) inside a container can be mapped to an unprivileged user on the host, significantly reducing the impact of container escape vulnerabilities. ## Prerequisites User namespaces require SKS nodes with at least kubelet v1.33.1. Ensure your SKS cluster nodes are running a compatible version. ## Enabling User Namespaces SKS clusters support user namespaces out of the box. In order to run a pod with user namespaces enabled, set `hostUsers: false` in your pod specification: ```yaml apiVersion: v1 kind: Pod metadata: name: userns-demo spec: hostUsers: false containers: - name: demo image: alpine command: ["sleep", "infinity"] ``` When `hostUsers: false` is set, the container runs in a separate user namespace. The root user (UID 0) inside the container is mapped to an unprivileged user ID on the host. ## Verifying User Namespace Isolation To verify that user namespaces are working: 1. Deploy the pod: ```bash kubectl apply -f userns-demo.yaml ``` 2. Check the user ID inside the container: ```bash # kubectl exec userns-demo -- id # [output] uid=0(root) gid=0(root) groups=0(root),1(bin),2(daemon),3(sys),4(adm),6(disk),10(wheel),11(floppy),20(dialout),26(tape),27(video) ``` 3. Check the mapped user ID on the host by inspecting the container process: ```bash # kubectl exec userns-demo -- cat /proc/1/uid_map # [output] 0 1317404672 65536 ``` The output shows the UID mapping. The first column is the UID inside the container. The second column is the UID from the host perspective, and the third column is the range length. In the example above, the container is running as root (UID 0), but it maps to UID 1317404672 on the host, indicating that user namespace isolation is active. ## Security Benefits User namespaces provide some security advantages: - Reduced privilege escalation risk since even if a container escape occurs, the attacker gains access to the host as an unprivileged user. - File permission isolation: Host files owned by root remain protected since the container's root user maps to a different unprivileged UID on the host. - Capability limitations: Capabilities like `CAP_SYS_ADMIN` are limited to the pod's user namespace only, not the host. - Pod-to-pod isolation: Each pod gets non-overlapping UID/GID mappings, so escaped containers from different pods run as different unprivileged users on the host. ## Pod Security Standards When using user namespaces, pods running as root inside the container do not violate `runAsNonRoot` constraints in Pod Security Standards. This is because root inside the container (UID 0) maps to an unprivileged user on the host. ## Limitations - Volume file ownership: Files on volumes with UIDs or GIDs outside the mapped 0-65535 range appear as overflow ID (65534) and cannot be modified. ## Further Reading - [Kubernetes User Namespaces Documentation][k8s-userns] - [Linux User Namespaces man page][man-userns] [k8s-userns]: https://kubernetes.io/docs/concepts/workloads/pods/user-namespaces/ [man-userns]: https://man7.org/linux/man-pages/man7/user_namespaces.7.html --- url: https://community.exoscale.com/product/compute/containers/how-to/sks-mountpoint-s3/ --- # Mount SOS in SKS The [Mountpoint for Amazon S3 CSI driver](https://github.com/awslabs/mountpoint-s3-csi-driver) allows you to mount S3-compatible object storage buckets as volumes in Kubernetes. Since Exoscale Simple Object Storage (SOS) is exposing a S3-compatible protocol, you can use this driver to mount SOS buckets in your SKS clusters. This is ideal for read-heavy workloads such as data lakes or machine learning training data. ## Prerequisites - A SKS cluster running in the `ch-gva-2` zone (or adapt the endpoint for your zone) - `kubectl` configured to access your cluster - [Helm](https://helm.sh/) installed - [Exoscale CLI](https://community.exoscale.com/reference/cli/index.md) installed and configured - An existing SOS bucket ## Install the Mountpoint S3 CSI Driver 1. Add the Helm Repository ```bash helm repo add aws-mountpoint-s3-csi-driver https://awslabs.github.io/mountpoint-s3-csi-driver helm repo update ``` 2. Create the namespace ```bash kubectl create namespace mountpoint-s3 ``` 3. Create an IAM Role for SOS Access The API key used by the CSI driver needs an IAM Role with a policy that grants access to your SOS bucket. Create the role with a policy allowing full access to your bucket: ```bash exo iam role create mountpoint-s3-role \ --policy '{ "default-service-strategy": "deny", "services": { "sos": { "type": "rules", "rules": [ { "expression": "parameters.bucket == '\''my-bucket'\''", "action": "allow" } ] } } }' ``` > [!NOTE] > Replace `my-bucket` with your actual SOS bucket name. 4. Create an API key with this role ```bash exo iam api-key create mountpoint-s3-key mountpoint-s3-role ``` Save the key ID and secret from the output - you'll need them in the next step. For more fine-grained access control (read-only, prefix restrictions, etc.), see the [IAM Policy Guide](https://community.exoscale.com/product/security/iam/how-to/policy-guide/index.md). 5. Create the Credentials Secret Create a Kubernetes secret containing your Exoscale API credentials: ```bash kubectl create secret generic exoscale-sos-credentials \ --namespace mountpoint-s3 \ --from-literal=key_id='EXOxxxxxxxxxxxxxxxxxxxxxxxx' \ --from-literal=access_key='your-api-secret-here' ``` > [!NOTE] > Replace the values with your actual Exoscale API key ID and secret from the > previous step. 6. Install the Driver The credentials secret must exist before installing the driver, as credentials are only read at startup. ```bash helm upgrade --install aws-mountpoint-s3-csi-driver \ aws-mountpoint-s3-csi-driver/aws-mountpoint-s3-csi-driver \ --namespace mountpoint-s3 \ --set awsAccessSecret.name=exoscale-sos-credentials ``` > [!NOTE] > If you need to update the credentials later, restart the driver pods: > `kubectl rollout restart daemonset -n mountpoint-s3 s3-csi-node` Verify the driver is running: ```bash kubectl get pods -n mountpoint-s3 ``` You should see one pod per node in the `Running` state (the driver runs as a DaemonSet): ``` NAME READY STATUS RESTARTS AGE s3-csi-controller-5dbf7f4db5-vbxnm 1/1 Running 0 2m36s s3-csi-node-6qmd5 3/3 Running 0 2m36s s3-csi-node-8hwxm 3/3 Running 0 2m36s s3-csi-node-l6wxw 3/3 Running 0 2m36s s3-csi-node-rsk84 3/3 Running 0 2m36s s3-csi-node-sc7zq 3/3 Running 0 2m36s ``` ## Create a PersistentVolume The Mountpoint S3 CSI driver only supports **static provisioning**. This means you must create a `PersistentVolume` (PV) that references an existing SOS bucket. 1. Create a file named `sos-pv.yaml` ```yaml apiVersion: v1 kind: PersistentVolume metadata: name: sos-pv spec: capacity: storage: 100Gi accessModes: - ReadWriteMany mountOptions: - allow-delete - region ch-gva-2 - endpoint-url https://sos-ch-gva-2.exo.io csi: driver: s3.csi.aws.com volumeHandle: s3-csi-driver-volume volumeAttributes: bucketName: my-bucket ``` > [!NOTE] > Replace `my-bucket` with your actual SOS bucket name and adapt the zone in > `region` and `endpoint-url` to match your bucket's zone. The `capacity` field > is informational only as S3 buckets have no fixed size. 2. Apply the manifest ```bash kubectl apply -f sos-pv.yaml ``` Mount Options Key mount options for SOS: | Option | Description | | ------------------------------------------ | ----------------------------------------------------------------------- | | `endpoint-url https://sos-.exo.io` | Required. The SOS endpoint for your zone. | | `region ` | Required. Prevents AWS SDK region auto-detection errors. | | `allow-delete` | Optional. Allows deleting files from the mounted bucket. | | `prefix /` | Optional. Mount only a specific prefix (directory) within the bucket. | The SOS endpoint follows the pattern `https://sos-.exo.io` where `` is your Exoscale zone (e.g., `ch-gva-2`, `ch-dk-2`, `de-fra-1`, `de-muc-1`, `at-vie-1`, `at-vie-2`, etc.). ## Create a PersistentVolumeClaim 1. Create a file named `sos-pvc.yaml` ```yaml apiVersion: v1 kind: PersistentVolumeClaim metadata: name: sos-pvc spec: accessModes: - ReadWriteMany storageClassName: "" resources: requests: storage: 100Gi volumeName: sos-pv ``` 2. Apply the manifest ```bash kubectl apply -f sos-pvc.yaml ``` Verify the PVC is bound: ```bash kubectl get pvc sos-pvc ``` ``` NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS VOLUMEATTRIBUTESCLASS AGE sos-pvc Bound sos-pv 100Gi RWX 47s ``` ## Use the Volume in a Pod 1. Create a file named `sos-pod.yaml` ```yaml apiVersion: v1 kind: Pod metadata: name: sos-app spec: containers: - name: app image: alpine command: ["/bin/sh", "-c", "while true; do ls -la /data; sleep 30; done"] volumeMounts: - name: sos-volume mountPath: /data volumes: - name: sos-volume persistentVolumeClaim: claimName: sos-pvc ``` 2. Apply the manifest ```bash kubectl apply -f sos-pod.yaml ``` 3. Verify the pod is running and can access the bucket ```bash kubectl logs sos-app ``` You should see the contents of your SOS bucket listed. ## Advanced: Using Prefixes for Workload Isolation You can mount multiple `PersistentVolumes` from a single SOS bucket by using different prefixes. Each prefix acts as an isolated "directory" within the bucket. ### Example: Two Workloads Sharing One Bucket Create PVs for two different applications: ```yaml apiVersion: v1 kind: PersistentVolume metadata: name: sos-pv-app1 spec: capacity: storage: 100Gi accessModes: - ReadWriteMany mountOptions: - allow-delete - region ch-gva-2 - endpoint-url https://sos-ch-gva-2.exo.io - prefix app1/data/ csi: driver: s3.csi.aws.com volumeHandle: s3-csi-driver-volume-app1 volumeAttributes: bucketName: shared-bucket --- apiVersion: v1 kind: PersistentVolume metadata: name: sos-pv-app2 spec: capacity: storage: 100Gi accessModes: - ReadWriteMany mountOptions: - allow-delete - region ch-gva-2 - endpoint-url https://sos-ch-gva-2.exo.io - prefix app2/data/ csi: driver: s3.csi.aws.com volumeHandle: s3-csi-driver-volume-app2 volumeAttributes: bucketName: shared-bucket ``` With this configuration: - `sos-pv-app1` only sees objects under `app1/data/` in the bucket - `sos-pv-app2` only sees objects under `app2/data/` in the bucket - Objects are isolated between workloads while sharing the same underlying bucket > [!NOTE] > The prefix must end with a trailing slash (e.g., `app1/data/`). ## Limitations ### Static Provisioning Only The Mountpoint S3 CSI driver does not support dynamic provisioning. You cannot use a `StorageClass` to automatically create PVs or buckets. Each PV must be manually created and mapped to an existing bucket and optionally a key prefix. ### Object Storage Semantics SOS is an object store, not a POSIX filesystem. Hence, some operations behave differently: - **No in-place modifications**: Existing files cannot be modified, only overwritten - **No directory rename**: Directories cannot be renamed - **No symbolic links**: Symlinks are not supported - **No file locking**: Concurrent writes may cause conflicts For detailed differences, please refer to the [Mountpoint documentation](https://github.com/awslabs/mountpoint-s3/blob/main/doc/SEMANTICS.md). ## Further Reading - [Mountpoint for Amazon S3 CSI Driver](https://github.com/awslabs/mountpoint-s3-csi-driver) - [Mountpoint S3 Configuration](https://github.com/awslabs/mountpoint-s3/blob/main/doc/CONFIGURATION.md) - [Exoscale SOS Documentation](https://community.exoscale.com/product/storage/object-storage/index.md) - [Mountpoint for Amazon S3 (host-based)](https://community.exoscale.com/product/storage/object-storage/how-to/mountpoint/index.md) --- url: https://community.exoscale.com/product/compute/containers/how-to/restrict-ingress-traffic-ip-allow-lists/ --- # Restrict ingress traffic with IP allow-lists The [Quick Start Guide](https://community.exoscale.com/product/compute/containers/quick-start/index.md) opens the NodePort range (`30000-32767`) to `0.0.0.0/0` so you can start exposing services right away. That rule is convenient, but it also means **any host on the internet can reach any NodePort on your cluster**, including services you may not have intended to publish. This guide replaces that single open rule with two narrower ones, so traffic is only allowed from a restricted set of sources: | Source | Why it needs access | | -------------------------------------- | --------------------------------------------------------------------------------------- | | Your known IP address(es) | Lets you and your team reach NodePort services directly, e.g. for testing. | | Exoscale's NLB health-check sources | Required if you front your NodePort services with a [Network Load Balancer](https://community.exoscale.com/product/networking/nlb/index.md), so it can perform health checks. | Everything else on the internet is denied by default, which significantly reduces the attack surface of your cluster. ## Replace the open NodePort rule You need to change this rule: ```bash exo compute security-group rule add sks-security-group \ --description "NodePort services" \ --protocol tcp \ --network 0.0.0.0/0 \ --port 30000-32767 ``` and replace it by the two rules below. ### 1. Allow NodePort traffic from your own IP ```bash exo compute security-group rule add sks-security-group \ --description "NodePort TCP services from Internet" \ --protocol tcp \ --network x.x.x.x/32 \ --port 30000-32767 ``` > [!NOTE] > Replace `x.x.x.x/32` with an IP range that should be allowed to reach your service (your workstation, office egress IP, a monitoring probe, etc). Add as many rules as you have ranges to allow. ### 2. Allow NodePort traffic from the NLB health checks ```bash exo compute security-group rule add sks-security-group \ --description "Nodeport TCP services from NLB" \ --protocol tcp \ --port 30000-32767 \ --public-security-group "public-nlb-healthcheck-sources" ``` ## Terraform If you manage your infrastructure as code, the same two rules translate to: ```hcl resource "exoscale_security_group_rule" "nodeport_tcp_nlb" { security_group_id = exoscale_security_group.my_sks_security_group.id description = "Nodeport TCP service from NLB" type = "INGRESS" protocol = "TCP" start_port = 30000 end_port = 32767 public_security_group = "public-nlb-healthcheck-sources" } resource "exoscale_security_group_rule" "nodeport_service_mylaptop" { security_group_id = exoscale_security_group.my_sks_security_group.id description = "NodePort services from a known IP" type = "INGRESS" protocol = "TCP" start_port = 30000 end_port = 32767 cidr = "x.x.x.x/32" } ``` > [!NOTE] > Replace `x.x.x.x/32` with the IP address that should be allowed to reach your service. ## See Also * A breakdown of [Exoscale Security Groups](https://community.exoscale.com/product/networking/security-group/index.md) --- url: https://community.exoscale.com/product/compute/containers/how-to/gpu-mig-partitioning/ --- # Use GPU MIG Partitioning The [Multi-Instance GPU (MIG)](https://www.nvidia.com/en-us/technologies/multi-instance-gpu/) feature allows you to partition a physical GPU into multiple isolated instances, each with dedicated memory and compute resources. MIG is useful when a full GPU would be too large for a single workload. By splitting one physical GPU into multiple GPU instances, you can run multiple smaller workloads on the same SKS node while giving each workload dedicated GPU memory and compute resources. This can improve GPU utilization for some workload types (e.g. AI inference). ## Prerequisites - A GPU instance with a MIG-capable GPU. On Exoscale, the following GPU types support MIG: - [GPU A30](https://www.exoscale.com/gpu/a30/) - [GPU RTX Pro 6000](https://www.exoscale.com/gpu/rtx6000/) MIG helps improve utilization of the GPU resources you provision, but it does not change the billing unit and is only available on MIG-capable GPUs. ## Enable MIG on SKS standard NodePools ### Portal You can enable one of the [supported MIG profiles](https://docs.nvidia.com/datacenter/tesla/mig-user-guide/latest/supported-mig-profiles.html) for your GPU type when you create or update a nodepool: - For **new** nodepools: navigate to `SKS > your_cluster > Add Nodepool`. Select a MIG-capable GPU instance type, then choose the desired MIG profile from the dropdown. - For **existing** nodepools: navigate to `SKS > your_cluster > your_nodepool > Update Nodepool`. Select the desired MIG profile from the dropdown. ![](nvidia-mig-profile-select.png) > [!WARNING] > To make your existing nodepool pick up the newly enabled MIG profile, you have to [cycle your nodes](https://community.exoscale.com/product/compute/containers/how-to/lifecycle-management/index.md#drain-and-destroy-nodes-one-by-one). ### CLI The [Exoscale CLI](https://github.com/exoscale/cli) (starting from version 1.96.0) lets you enable a MIG profile on standard SKS nodepools. You only pass the profile name (e.g. `4g.24gb`). The GPU family (`a30.24gb` or `rtxpro6000.96gb`) is automatically defined from the nodepool instance type. Create a cluster with a default nodepool that has MIG enabled: ```sh exo compute sks create my-cluster \ --zone ch-gva-2 \ --nodepool-name gpu-pool \ --nodepool-size 1 \ --nodepool-instance-type gpua30.large \ --nodepool-nvidia-mig-profile 4g.24gb ``` Add a MIG-enabled nodepool to an existing cluster: ```sh exo compute sks nodepool add my-cluster gpu-pool \ --instance-type gpurtx6000pro.medium \ --size 1 \ --nvidia-mig-profile 1g.24gb ``` Update the MIG profile on an existing nodepool. ```sh exo compute sks nodepool update my-cluster gpu-pool \ --nvidia-mig-profile 2g.48gb ``` You can also pass an empty value to disable MIG. > [!WARNING] > Updating the MIG profile of an existing nodepool only affects newly provisioned nodes. To make current nodes pick up the new profile, you have to [cycle your nodes](https://community.exoscale.com/product/compute/containers/how-to/lifecycle-management/index.md#drain-and-destroy-nodes-one-by-one). ### Terraform provider The [Exoscale Terraform provider](https://registry.terraform.io/providers/exoscale/exoscale/latest/docs/resources/sks_nodepool) (starting from version 0.70.0) enables a MIG profile on a nodepool through the `nvidia_mig_profile` attribute of the `exoscale_sks_nodepool` resource. You only set the profile name (e.g. `4g.24gb`). The GPU family (`a30.24gb` or `rtxpro6000.96gb`) is automatically deduced from the nodepool `instance_type`. Create a MIG-enabled nodepool: ```hcl resource "exoscale_sks_nodepool" "gpu" { zone = "ch-gva-2" cluster_id = exoscale_sks_cluster.my_cluster.id name = "gpu-pool" instance_type = "gpua30.large" size = 1 nvidia_mig_profile = "4g.24gb" } ``` To change the profile, update `nvidia_mig_profile` and apply. To disable MIG, remove the attribute. The current profile is also exposed on the `exoscale_sks_nodepool` data source. > [!WARNING] > Updating the MIG profile of an existing nodepool only affects newly provisioned nodes. To make current nodes pick up the new profile, you have to [cycle your nodes](https://community.exoscale.com/product/compute/containers/how-to/lifecycle-management/index.md#drain-and-destroy-nodes-one-by-one). ## Enable MIG on Karpenter Nodepools The SKS Node operating system is configured with TOML-formatted data. Most critical parts of this data are set by our SKS product orchestrator. As a customer it's possible to enrich this data with additional configuration. In order to enable MIG, you can set specific configuration under both `settings.kubelet-device-plugins.nvidia` and `settings.kubelet-device-plugins.nvidia.mig.profile` entries on `spec.userData` of the Karpenter `ExoscaleNodeClass` resource. For example if you want to select the "1g.24gb" profile on rtxpro6000 nodes, you can define: ```yaml apiVersion: karpenter.exoscale.com/v1 kind: ExoscaleNodeClass metadata: name: nvidia-rtxpro6000-1g-24gb spec: imageTemplateSelector: variant: nvidia diskSize: 100 # Beware that GPU nodes enforce minimum disk size securityGroupSelectorTerms: - id: "5ea00f6e-1e92-456b-9381-2cb2c5937240" # Don't forget to attach proper security group userData: | [settings.kubelet-device-plugins.nvidia] device-partitioning-strategy = "mig" [settings.kubelet-device-plugins.nvidia.mig.profile] "rtxpro6000.96gb" = "1g.24gb" ``` On your referencing NodePool, nothing special except you need to select the proper instance type and reference the NodeClass: ```yaml apiVersion: karpenter.sh/v1 kind: NodePool metadata: name: gpu-rtxpro6000-1g-24gb spec: template: spec: nodeClassRef: group: karpenter.exoscale.com kind: ExoscaleNodeClass name: nvidia-rtxpro6000-1g-24gb requirements: - key: "node.kubernetes.io/instance-type" operator: In values: - "gpurtx6000pro.small" - "gpurtx6000pro.medium" - "gpurtx6000pro.large" taints: - key: "nvidia.com/gpu" value: "true" effect: "NoSchedule" expireAfter: 720h # [...] weight: 10 ``` If a Pod requesting `nvidia.com/gpu` stays pending because of a lack of GPU, Karpenter will trigger the creation of a new GPU node with those characteristics: ```yaml apiVersion: v1 kind: Node # [...] spec: # [...] taints: - effect: NoSchedule key: nvidia.com/gpu value: "true" status: # [...] allocatable: cpu: 35700m ephemeral-storage: "89785012888" hugepages-1Gi: "0" hugepages-2Mi: "0" memory: 123120044Ki nvidia.com/gpu: "4" pods: "110" capacity: cpu: "36" ephemeral-storage: 102083312Ki hugepages-1Gi: "0" hugepages-2Mi: "0" memory: 123632044Ki nvidia.com/gpu: "4" pods: "110" # [...] ``` ## Supported MIG profiles ### A30 MIG profiles The `a30.24gb` device name comes with the available MIG profiles below: | MIG profile | # partitions | | ------------- | -------------- | | `1g.6gb` | 4 | | `1g.6gb+me` | 1 | | `2g.12gb` | 2 | | `2g.12gb+me` | 2 | | `4g.24gb` | 1 | Source: https://docs.nvidia.com/datacenter/tesla/mig-user-guide/latest/supported-mig-profiles.html#a30-mig-profiles ### RTX6000PRO MIG profiles The `rtxpro6000.96gb` device name comes with the available MIG profiles below: | MIG profile | # partitions | | ------------------ | -------------- | | `1g.24gb` | 4 | | `1g.24gb+me` | 1 | | `1g.24gb+gfx` | 4 | | `1g.24gb+me.all` | 1 | | `1g.24gb-me` | 4 | | `2g.48gb` | 2 | | `2g.48gb+gfx` | 2 | | `2g.48gb+me.all` | 1 | | `2g.48gb-me` | 2 | | `4g.96gb` | 1 | | `4g.96gb+gfx` | 1 | Source: https://docs.nvidia.com/datacenter/tesla/mig-user-guide/latest/supported-mig-profiles.html#rtx-pro-6000-blackwell-mig-profiles ### Profile References The profile suffix indicates whether additional GPU capabilities are assigned to the MIG instance or not. * `+me` profiles: Include at least one media engine (NVDEC, NVENC, NVJPG, or OFA). * `+gfx`: Adds support for graphics APIs (new in GB20X). * `+me.all`: Allocates all available media engines to this instance (does not include graphics support). * `-me`: Excludes all media engines for pure compute workloads. Source: https://docs.nvidia.com/datacenter/tesla/mig-user-guide/latest/supported-mig-profiles.html ## Limitations Exoscale currently supports the **single** MIG strategy, i.e., all GPUs within a node are partitioned with the same profile. The **mixed** strategy (different profiles on the same GPU) is currently unsupported. For more details on MIG strategies, see the [NVIDIA MIG configuration documentation](https://run-ai-docs.nvidia.com/self-hosted/platform-management/aiinitiatives/resources/mig-profiles). ## References - [Use GPU MIG Partitioning (on compute instance)](https://community.exoscale.com/product/compute/instances/how-to/gpu-mig-partitioning/index.md). - [Multi-Instance GPU (MIG)](https://www.nvidia.com/en-us/technologies/multi-instance-gpu/) allows you to partition a physical GPU into multiple isolated instances, each with dedicated memory and compute resources. This is particularly useful when you need to run multiple workloads that don't require a full GPU. - [Multi-Instance GPU (MIG) profiles](https://docs.nvidia.com/datacenter/tesla/mig-user-guide/latest/supported-mig-profiles.html) --- url: https://community.exoscale.com/product/compute/containers/service-boundaries/ --- # Service Boundaries - Containers - [Limits and Quotas - Containers](https://community.exoscale.com/product/compute/containers/service-boundaries/limits-and-quotas/index.md) - SKS Limitations and Quotas - [SLA - Containers](https://community.exoscale.com/product/compute/containers/service-boundaries/sla/index.md) - SKS Service Level Agreement - [Pricing ↗︎](https://www.exoscale.com/pricing/#sks) --- url: https://community.exoscale.com/product/compute/containers/service-boundaries/limits-and-quotas/ --- # Limits and Quotas - Containers ## Limits SKS is available with the following limitations to ensure performance and supportability: NodePort Range : The number of SKS clusters per zone is currently limited by the available NodePort range (default 30000-32767). Persistent Volume Attachments : Volumes can be attached to only one node at a time. Online resizing of PersistentVolumes is not supported. Minimun instances size and disk size : Minimum instance size is Small or 2 GB RAM equivalent and minimum instance disk size is 20 GB. No cross-zone stretch support : Each cluster is local to a single zone only. Root Credential Expiry : Root API credentials (including tokens used by cluster automation tools) have a maximum time-to-live (TTL) of 30 days. After that, they expire and must be rotated or renewed to maintain access. ## Quotas | Usage | Quota | | :------ | :------: | | SKS clusters count | 3 | ## Availability SKS is available in all zones. --- url: https://community.exoscale.com/product/compute/containers/service-boundaries/sla/ --- # SLA - Containers ## Scalable Kubernetes Service (SKS) Service Level Agreement (SLA) Those product specific SLAs define the Service Availability Target for Scalable Kubernetes Service (SKS). It applies to each Client using the Service. Capitalized terms used herein but not defined herein shall have the meanings set forth in the Exoscale Terms and Conditions or EUSA whichever applies. ### SLA | Plan | Target | Definition | | -------- | -------- | ------------ | | Starter Cluster | No SLA | The Starter Cluster is a free plan and does not include a Service Level Agreement for its control plane. | | Pro Cluster | >= 99.95% Monthly Uptime Percentage | The percentage of total minutes in a calendar month during which Exoscale SKS Cluster is available and operational for use. This objective ensures consistent access to your cluster, with a monthly overall availability of 99.95%. | | Pro Cluster -SKS API Request Success Rate | ≥ 99.95% (per calendar month) | Percentage of Kubernetes API requests to the SKS control plane returning a successful (non-5xx) response. | ### Non-binding objectives In addition to the SLA defined above, Exoscale provides non-binding Service Level Objectives (SLOs) for its platform APIs, offering transparency into service reliability and performance. General API SLOs are documented on the [API SLO page](https://community.exoscale.com/platform/sla/api-slo/index.md). ##### Nature of the Non-binding Objectives Commitment - This provision does not constitute a legally binding obligation but rather specifies a target value that Exoscale will endeavor to achieve. - In this case, no monetary credits or other compensation will be granted if the objective is missed. - This is provided for transparency and capacity-planning purposes only. ### Client Responsibilities While Exoscale is responsible for operating and maintaining the Scalable Kubernetes Service (SKS) control plane, Client remains responsible for all workloads and resources deployed within their clusters, including but not limited to: * Container images, registries, and application runtimes. * Deployment, configuration, and lifecycle management of Kubernetes workloads. * Management of configuration data and persistent data, including volumes, backups, disaster recovery planning, and restore procedures. * Cluster-level access control, including Kubernetes RBAC, namespaces, service accounts, and credentials. * Security of applications, containers, and third-party components running in the cluster. * Monitoring of resource usage, performance, and costs. The Service Level Agreement applies solely to the availability of the SKS control plane. It does not cover worker nodes, pods, volumes, applications, data, or Client-deployed components. For a complete explanation of the shared responsibility framework between Exoscale and Clients, please see the [Exoscale Shared Responsibility Model documentation](https://community.exoscale.com/platform/compliance/shared-responsability-model/index.md). ### Exclusions No compensation shall be granted to the Client if the failure to comply with the Scalable Kubernetes Service (SKS) Service Level Objective is due to any of the following reasons: * Failures in Kubernetes nodes, volumes, or pods within Scalable Kubernetes Service (SKS) clusters, as the SLA applies to the control plane, and the customer is responsible for worker node management and data backup. For the complete list of SLA exclusions, please refer to the [General Exoscale Terms and Conditions](https://www.exoscale.com/terms/). ### Retribution The standard retribution and SLA credit schemes apply for the Pro Cluster as described under [General Exoscale Terms and Conditions](https://www.exoscale.com/terms/) or [EUSA](https://www.exoscale.com/eusa/) whichever applies. In the event that the Monthly Uptime Percentage for Exoscale Compute Instances falls below the committed Service Level Objective, the Client will be eligible for service credits as follows: | Monthly Uptime Percentage | Service Credit Percentage of Monthly Service Fees for Affected Resources | | -------- | -------- | | from 99.95% to 98.3% | 50% | | below 98.3% | 100% | #### Claim Service Credit To request a service credit, refer to the Service Unavailability Credit section of the [Exoscale Terms and Conditions](https://www.exoscale.com/terms/) or the [EUSA](https://www.exoscale.com/eusa/), whichever applies. --- url: https://community.exoscale.com/product/compute/containers/reference/ --- # Reference - Containers - [API SKS](https://community.exoscale.com/reference/api/sks/index.md) - All operations on SKS. (external) - [CLI SKS](https://community.exoscale.com/reference/cli/exo/compute/sks/index.md) - All commands on SKS. (external) --- url: https://community.exoscale.com/product/compute/containers/recipes/ --- # Recipes - Containers Informational, concise guides showing how to integrate with external tools and solve specific scenarios. - [Setup GitLab Runners](https://community.exoscale.com/product/compute/containers/recipes/autoscaled-gitlab-runners/index.md) - Set up autoscaled GitLab Runners on Exoscale SKS with Karpenter to isolate CI jobs, scale on demand, and optimize costs. - [Use Rancher with SKS](https://community.exoscale.com/product/compute/containers/recipes/rancher-integration/index.md) - Use Rancher with Exoscale SKS to create, import, and manage Kubernetes clusters through the Exoscale cluster driver and UI. --- url: https://community.exoscale.com/product/compute/containers/recipes/autoscaled-gitlab-runners/ --- # Setup GitLab Runners In this documentation, we will guide you through the steps to set up and run autoscaled Gitlab runners on Exoscale SKS clusters with Karpenter. This allows you to efficiently manage your CI/CD workloads by automatically scaling the number of Gitlab runners based on demand, reducing massively operating costs. ## Prerequisites As a prerequisite for the following documentation, you need: - An Exoscale SKS cluster on the Pro plan with Karpenter addon enabled. - Access to your cluster via `kubectl` and `helm`. - Basic Linux knowledge. If you do not have access to an SKS cluster, follow the [Quick Start Guide](https://community.exoscale.com/product/compute/instances/quick-start/index.md). ## Configure Karpenter nodepools We will create one `ExoscaleNodeClass` and two `NodePools`, one for the SKS kube-system and the Gitlab Runner manager, and one for the Gitlab Runner jobs. Here is the configuration: ```yaml --- apiVersion: karpenter.exoscale.com/v1 kind: ExoscaleNodeClass metadata: name: system spec: # imageTemplateSelector automatically selects the appropriate OS template # based on Kubernetes version and variant (alternative to templateID) imageTemplateSelector: # version: Kubernetes version (semver format like "1.34.1") # If omitted (or if you use imageTemplateSelector: {}), the control plane's # current Kubernetes version will be auto-detected at runtime version: "1.35.0" # variant: Template variant (optional, defaults to "standard") # Options: "standard" for regular workloads, "nvidia" for GPU-enabled nodes. variant: "standard" diskSize: 50 securityGroupSelectorTerms: - id: "" # Either you target specific ID - name: "" # Or a security group name # Optional: Define anti-affinity groups to spread nodes across failure domains antiAffinityGroupSelectorTerms: [] --- apiVersion: karpenter.exoscale.com/v1 kind: ExoscaleNodeClass metadata: name: gitlab-runners spec: # imageTemplateSelector automatically selects the appropriate OS template # based on Kubernetes version and variant (alternative to templateID) imageTemplateSelector: # version: Kubernetes version (semver format like "1.34.1") # If omitted (or if you use imageTemplateSelector: {}), the control plane's # current Kubernetes version will be auto-detected at runtime version: "1.35.0" # variant: Template variant (optional, defaults to "standard") # Options: "standard" for regular workloads, "nvidia" for GPU-enabled nodes. variant: "standard" # Disk size in GB (default: 50, min: 10, max: 8000), you can adjust this based on your workload requirements diskSize: 200 securityGroupSelectorTerms: - id: "" # Optional: Define anti-affinity groups to spread nodes across failure domains antiAffinityGroupSelectorTerms: [] --- apiVersion: karpenter.sh/v1 kind: NodePool metadata: name: system spec: template: metadata: labels: nodepool: system spec: nodeClassRef: group: karpenter.exoscale.com kind: ExoscaleNodeClass name: system # Startup taints prevent scheduling until node is fully ready startupTaints: [] taints: - key: CriticalAddonsOnly value: "true" effect: NoSchedule # Instance type requirements requirements: - key: "node.kubernetes.io/instance-type" operator: In values: - "standard.medium" expireAfter: 168h # Recycle nodes every week # Disruption settings for cost optimization disruption: consolidationPolicy: WhenEmptyOrUnderutilized # Wait 30m before consolidating underutilized nodes consolidateAfter: 30m # Limit disruption rate budgets: - nodes: "10%" # Disrupt at most 10% of nodes at once # Weight for prioritization (higher = higher priority) weight: 50 --- apiVersion: karpenter.sh/v1 kind: NodePool metadata: name: gitlab-runners spec: template: metadata: labels: nodepool: gitlab-runners spec: nodeClassRef: group: karpenter.exoscale.com kind: ExoscaleNodeClass name: gitlab-runners taints: - key: workload value: gitlab-runner effect: NoSchedule requirements: - key: "node.kubernetes.io/instance-type" operator: In values: # Choose instance types that are suitable for Gitlab Runner jobs (e.g., more CPU or memory) - "standard.medium" - "standard.large" - "standard.extra-large" expireAfter: 168h # 7 days # Disruption settings for cost optimization disruption: consolidationPolicy: WhenEmpty # Don't consolidate when there is running jobs # Wait 2m before consolidating underutilized nodes, allowing for short-term spikes in demand consolidateAfter: 2m # Limit disruption rate budgets: - nodes: "30%" # Disrupt at most 30% of nodes at once # Weight for prioritization (higher = higher priority) weight: 50 ``` > [!WARNING] > Ensure to modify the `` placeholder with your Security Group ID prior to applying the manifests. After applying this manifest you should have 2 nodes running, supporting system workloads. ## Install Gitlab Runner To install Gitlab Runner on your SKS cluster, you can use the official Helm chart provided by Gitlab. First you will need to gather a Gitlab token in order to permit your instance to access and maange on demand Gitlab Runners. You can obtain this token by going to your Gitlab group, then Builds > Runners, and then clicking on "Create group runner". If you want to be very generic, tick the "Run untagged jobs" checkbox then click on "Create runner". You can then copy the token and use it in the next step. ![Create Gitlab group runner](./gitlab-create-runner.png) Next, we will need a helm values file to configure the Gitlab Runner installation. This file will contain the Gitlab URL, the token we just obtained, and some tolerations to ensure that the Gitlab Runner pods are scheduled on the right nodes. You can create a `gitlab_helm_values.yaml` file with the following content: ```yaml --- gitlabUrl: https://gitlab.com runnerToken: rbac: create: true serviceAccount: create: true tolerations: - key: CriticalAddonsOnly effect: NoSchedule operator: Exists runners: config: | [[runners]] [runners.kubernetes] [runners.kubernetes.node_selector] "nodepool" = "gitlab-runners" [runners.kubernetes.node_tolerations] "workload=gitlab-runner" = "NoSchedule" ``` > [!IMPORTANT] > Ensure to modify the `` placeholder with your runner token prior to applying the manifests. In this configuration we will run Gitlab runners only on a dedicated nodepool tainted with `workload=gitlab-runner:NoSchedule`. This allows us to have better control over the resources used by Gitlab runners and to ensure that they do not interfere with other workloads running on the cluster. You can find more configuration options [here](https://docs.gitlab.com/runner/executors/kubernetes/#other-configtoml-settings). Now we will install the Gitlab runner orchestartor on the SKS cluster: ```bash helm repo add gitlab https://charts.gitlab.io helm repo update helm upgrade --install --namespace gitlab-runner -f gitlab_helm_values.yaml gitlab-runner gitlab/gitlab-runner --create-namespace ``` Now you should have your gitlab-runner pod up and running ```bash kubectl -n gitlab-runner get pod NAME READY STATUS RESTARTS AGE gitlab-runner-5b4fc8b55c-dc84c 1/1 Running 0 15s ``` ## CI in action Let's create a demo CI in a project using this new runner: ```yaml tages: - stress-test run-heavy-tasks: stage: stress-test image: alpine:latest script: - echo Starting simulation..." - sleep 240 - echo "Job completed successfully!" parallel: 15 # Generates 15 jobs in paralllel ``` We now see that there is new nodes and many jobs created in Kubernetes: ```bash ❯ kubectl -n gitlab-runner get node NAME STATUS ROLES AGE VERSION k-gitlab-runners-9z957 Ready 2m v1.35.0 k-gitlab-runners-cwqqp Ready 2m v1.35.0 k-gitlab-runners-fhff7 Ready 2m v1.35.0 k-system-cx6qg Ready 44m v1.35.0 k-system-hqdn5 Ready 51m v1.35.0 ❯ kubectl -n gitlab-runner get pod -w NAME READY STATUS RESTARTS AGE gitlab-runner-5b4fc8b55c-dc84c 1/1 Running 0 4m19s runner-rthqphyc9-project-79409999-concurrent-0-xqczj3vf 0/2 Pending 0 0s runner-rthqphyc9-project-79409999-concurrent-0-xqczj3vf 0/2 Pending 0 0s runner-rthqphyc9-project-79409999-concurrent-1-lm7bbcy6 0/2 Pending 0 0s runner-rthqphyc9-project-79409999-concurrent-1-lm7bbcy6 0/2 Pending 0 0s runner-rthqphyc9-project-79409999-concurrent-2-oen5ne00 0/2 Pending 0 0s runner-rthqphyc9-project-79409999-concurrent-2-oen5ne00 0/2 Pending 0 0s runner-rthqphyc9-project-79409999-concurrent-3-8zy947mw 0/2 Pending 0 0s runner-rthqphyc9-project-79409999-concurrent-3-8zy947mw 0/2 Pending 0 1s runner-rthqphyc9-project-79409999-concurrent-4-ocz5qzj2 0/2 Pending 0 0s runner-rthqphyc9-project-79409999-concurrent-4-ocz5qzj2 0/2 Pending 0 0s runner-rthqphyc9-project-79409999-concurrent-5-e85r7h93 0/2 Pending 0 0s runner-rthqphyc9-project-79409999-concurrent-5-e85r7h93 0/2 Pending 0 0s runner-rthqphyc9-project-79409999-concurrent-6-ybpwijta 0/2 Pending 0 0s runner-rthqphyc9-project-79409999-concurrent-6-ybpwijta 0/2 Pending 0 0s runner-rthqphyc9-project-79409999-concurrent-7-kdgpclbl 0/2 Pending 0 0s runner-rthqphyc9-project-79409999-concurrent-7-kdgpclbl 0/2 Pending 0 0s runner-rthqphyc9-project-79409999-concurrent-8-jqm8z7b2 0/2 Pending 0 1s runner-rthqphyc9-project-79409999-concurrent-8-jqm8z7b2 0/2 Pending 0 1s runner-rthqphyc9-project-79409999-concurrent-9-74ridjck 0/2 Pending 0 0s runner-rthqphyc9-project-79409999-concurrent-9-74ridjck 0/2 Pending 0 0s # Jobs are now running runner-rthqphyc9-project-79409999-concurrent-0-b3bxbvc0 2/2 Running 0 42s runner-rthqphyc9-project-79409999-concurrent-1-mqfcjmqp 2/2 Running 0 46s runner-rthqphyc9-project-79409999-concurrent-2-c5uk73ql 2/2 Running 0 42s runner-rthqphyc9-project-79409999-concurrent-3-b0cdkf06 2/2 Running 0 41s runner-rthqphyc9-project-79409999-concurrent-4-zrd5g4p9 2/2 Running 0 40s runner-rthqphyc9-project-79409999-concurrent-8-0qf7voud 2/2 Running 0 51s runner-rthqphyc9-project-79409999-concurrent-9-e0g8s5oc 2/2 Running 0 51s runner-rthqphyc9-project-79409999-concurrent-9-e0g8s5oc 2/2 Running 0 67s runner-rthqphyc9-project-79409999-concurrent-8-0qf7voud 2/2 Running 0 75s runner-rthqphyc9-project-79409999-concurrent-1-mqfcjmqp 2/2 Running 0 80s ``` --- url: https://community.exoscale.com/product/compute/containers/recipes/rancher-integration/ --- # Use Rancher with SKS This page is a step-by-step guide to creating an SKS cluster using SUSE Rancher. ## Running Rancher To show the entire process, we first need a running Rancher instance. Let's create one. First, we run an Exoscale instance of type Large, named `rancher`. ```bash exo compute instance add rancher --instance-type large ``` > [!TIP] > We can use the shortened version of this command > `exo c i add rancher --instance-type large` Next, we run a shell inside it. The `exo` CLI has a handy command for this purpose. ```bash exo c i ssh rancher ``` Next, we install Docker as we'll use it to run Rancher. ```bash curl -sSL https://get.docker.com | bash ``` Next, we add the current user to the `docker` group. ```bash sudo usermod -aG docker $USER ``` After this step, we need to close the shell and open a new one so the changes are applied. Then, we install Rancher using the [single node installation method](https://ranchermanager.docs.rancher.com/getting-started/installation-and-upgrade/other-installation-methods/rancher-on-a-single-node-with-docker). > [!NOTE] > We should not use the installation method in a production environment, but it > is perfectly fine for this guide. ```bash docker run -d --restart=unless-stopped \ --name rancher \ -p 80:80 -p 443:443 \ --privileged \ rancher/rancher:latest ``` Using the instance's IP address, 194.182.165.191 in this example, we can access Rancher's UI. ![Login](./login-1.png) We retrieve the bootstrap password from the container's logs using the provided command. ```bash ubuntu@rancher:~$ docker logs rancher 2>&1 | grep "Bootstrap Password" 2025/11/06 10:55:55 [INFO] Bootstrap Password: j4s9bqblx2rrl4xkfcgnh8q48dqm5mcvdxmcbn8zb5hxqpskd7glpm ``` We use this password to log in for the first time. Then, we get a new password. ![Login](./login-2.png) We can then access the dashboard. This one shows only a local k3s cluster installed on the instance we created. ![dashboard](./dashboard.png) In the next section, we'll go through the process of managing an SKS cluster with Rancher. ## Adding Exoscale cluster driver By default, if we click the `Create` button, we can only create a hosted cluster in AWS, Azure, or Google. ![Cluster creation](./cluster-creation.png) Rancher lets you enable other providers via the "Drivers" -> "Cluster Drivers" menu. ![Cluster drivers list](./cluster-drivers-list-1.png) Rancher also allows you to add other providers via a cluster driver. To add the [Exoscale cluster driver](https://github.com/exoscale/kontainer-engine-driver-sks), we first click the `Create` button and enter the following information: - Download URL: https://github.com/exoscale/kontainer-engine-driver-sks/releases/download/v0.0.1/kontainer-engine-driver-sks_0.0.1_linux_amd64 - Checksum: 2f1dea48f07baad1634101cf6311a6342bbb444f1039a4fbe2f1d9f02f31d4cc ([from this file](https://github.com/exoscale/kontainer-engine-driver-sks/releases/download/v0.0.1/checksums.txt)) - Custom UI URL: https://exoscale.github.io/kontainer-engine-driver-sks/component.js - whitelist domains: `*.github.io` and `*.exoscale.com` ![Adding cluster driver](./adding-cluster-driver.png) It only takes a few seconds for the new driver to be activated. ![Cluster drivers list](./cluster-drivers-list-2.png) On the cluster creation page, we can now create an Exoscale-managed cluster. ![Cluster creation](./cluster-creation-2.png) Clicking the `Exoscale SKS` button launches the cluster creation assistant. First, we need to provide the cluster's name and an Exoscale key pair (ACCESS_KEY, SECRET_ACCESS_KEY) so Rancher can communicate with the Exoscale API. ![Add SKS cluster](./add-sks-cluster-1.png) > [!NOTE] > If you need to create a key pair, please follow the details below. > > **creating an Exoscale key pair** > > First, go to the [Exoscale portal](https://portal.exoscale.com/) and create a Role with access to the `Compute` Service Class from the IAM menu > > ![Role](/product/compute/containers/recipes/rancher-integration/exo-role.png) > > Next, create a key for this role. > > ![Key](/product/compute/containers/recipes/rancher-integration/exo-key-1.png) > > We then obtain the access key and secret key, which are ready for use. > > ![Key](/product/compute/containers/recipes/rancher-integration/exo-key-2.png) Next, we need to provide the following information: - the zone to launch the cluster - the Kubernetes version we need - the control-plane level (`starter` or `pro`) ![Add SKS cluster](./add-sks-cluster-2.png) Next, we need to specify the Node Pools to create for our cluster. In this example, we create a single Node Pool with 2 Medium instances. ![Add SKS cluster](./add-sks-cluster-3.png) Then, it only takes a few minutes to have the cluster up and running and available in Rancher. ![Add SKS cluster](./add-sks-cluster-5.png) We can now manage our SKS cluster and deploy some applications using Rancher. ![Add SKS cluster](./add-sks-cluster-6.png)