Kubelet Configuration

CAPBK has several ways to configure kubelet.

Pass KubeletConfiguration file via KubeadmConfigSpec.files

You can use KubeadmConfigSpec.files to put any files on nodes. This example puts a KubeletConfiguration file on nodes via KubeadmConfigSpec.files, and makes kubelet use it via KubeadmConfigSpec.kubeletExtraArgs. You can check available configurations of KubeletConfiguration on Kubelet Configuration (v1beta1) | Kubernetes.

This method is easy to replace the whole kubelet configuration generated by kubeadm, but it is not easy to replace only a part of the kubelet configuration.

KubeadmControlPlaneTemplate

apiVersion: controlplane.cluster.x-k8s.io/v1beta1
kind: KubeadmControlPlaneTemplate
metadata:
  name: cloudinit-control-plane
  namespace: default
spec:
  template:
    spec:
      kubeadmConfigSpec:
        files:
        # We put a KubeletConfiguration file on nodes via KubeadmConfigSpec.files
        # In this example, we directly put the file content in the KubeadmConfigSpec.files.content field.
        - path: /etc/kubernetes/kubelet/config.yaml
          owner: "root:root"
          permissions: "0644"
          content: |
            apiVersion: kubelet.config.k8s.io/v1beta1
            kind: KubeletConfiguration
            kubeReserved:
              cpu: "1"
              memory: "2Gi"
              ephemeral-storage: "1Gi"
            systemReserved:
              cpu: "500m"
              memory: "1Gi"
              ephemeral-storage: "1Gi"
            evictionHard:
              memory.available: "500Mi"
              nodefs.available: "10%"
            authentication:
              anonymous:
                enabled: false
              webhook:
                cacheTTL: 0s
                enabled: true
              x509:
                clientCAFile: /etc/kubernetes/pki/ca.crt
            authorization:
              mode: Webhook
              webhook:
                cacheAuthorizedTTL: 0s
                cacheUnauthorizedTTL: 0s
            cgroupDriver: systemd
            clusterDNS:
            - 10.128.0.10
            clusterDomain: cluster.local
            containerRuntimeEndpoint: ""
            cpuManagerReconcilePeriod: 0s
            evictionPressureTransitionPeriod: 0s
            fileCheckFrequency: 0s
            healthzBindAddress: 127.0.0.1
            healthzPort: 10248
            httpCheckFrequency: 0s
            imageMinimumGCAge: 0s
            logging:
              flushFrequency: 0
              options:
                json:
                  infoBufferSize: "0"
              verbosity: 0
            memorySwap: {}
            nodeStatusReportFrequency: 0s
            nodeStatusUpdateFrequency: 0s
            rotateCertificates: true
            runtimeRequestTimeout: 0s
            shutdownGracePeriod: 0s
            shutdownGracePeriodCriticalPods: 0s
            staticPodPath: /etc/kubernetes/manifests
            streamingConnectionIdleTimeout: 0s
            syncFrequency: 0s
            volumeStatsAggPeriod: 0s
        clusterConfiguration:
          controllerManager:
            extraArgs:
              enable-hostpath-provisioner: "true"
        initConfiguration:
          nodeRegistration:
            criSocket: unix:///var/run/containerd/containerd.sock
            # Here we configure kubelet to use the KubeletConfiguration file we put on nodes via KubeadmConfigSpec.files
            kubeletExtraArgs:
              config: "/etc/kubernetes/kubelet/config.yaml"
        joinConfiguration:
          nodeRegistration:
            criSocket: unix:///var/run/containerd/containerd.sock
            # Here we configure kubelet to use the KubeletConfiguration file we put on nodes via KubeadmConfigSpec.files
            kubeletExtraArgs:
              config: "/etc/kubernetes/kubelet/config.yaml"

KubeadmConfigTemplate

apiVersion: bootstrap.cluster.x-k8s.io/v1beta1
kind: KubeadmConfigTemplate
metadata:
  name: cloudinit-default-worker-bootstraptemplate
  namespace: default
spec:
  template:
    spec:
      files:
      # We puts a KubeletConfiguration file on nodes via KubeadmConfigSpec.files
      # In this example, we directly put the file content in the KubeadmConfigSpec.files.content field.
      - path: /etc/kubernetes/kubelet/config.yaml
        owner: "root:root"
        permissions: "0644"
        content: |
          apiVersion: kubelet.config.k8s.io/v1beta1
          kind: KubeletConfiguration
          kubeReserved:
            cpu: "1"
            memory: "2Gi"
            ephemeral-storage: "1Gi"
          systemReserved:
            cpu: "500m"
            memory: "1Gi"
            ephemeral-storage: "1Gi"
          evictionHard:
            memory.available: "500Mi"
            nodefs.available: "10%"
          authentication:
            anonymous:
              enabled: false
            webhook:
              cacheTTL: 0s
              enabled: true
            x509:
              clientCAFile: /etc/kubernetes/pki/ca.crt
          authorization:
            mode: Webhook
            webhook:
              cacheAuthorizedTTL: 0s
              cacheUnauthorizedTTL: 0s
          cgroupDriver: systemd
          clusterDNS:
          - 10.128.0.10
          clusterDomain: cluster.local
          containerRuntimeEndpoint: ""
          cpuManagerReconcilePeriod: 0s
          evictionPressureTransitionPeriod: 0s
          fileCheckFrequency: 0s
          healthzBindAddress: 127.0.0.1
          healthzPort: 10248
          httpCheckFrequency: 0s
          imageMinimumGCAge: 0s
          logging:
            flushFrequency: 0
            options:
              json:
                infoBufferSize: "0"
            verbosity: 0
          memorySwap: {}
          nodeStatusReportFrequency: 0s
          nodeStatusUpdateFrequency: 0s
          rotateCertificates: true
          runtimeRequestTimeout: 0s
          shutdownGracePeriod: 0s
          shutdownGracePeriodCriticalPods: 0s
          staticPodPath: /etc/kubernetes/manifests
          streamingConnectionIdleTimeout: 0s
          syncFrequency: 0s
          volumeStatsAggPeriod: 0s
      joinConfiguration:
        nodeRegistration:
          criSocket: unix:///var/run/containerd/containerd.sock
          # Here we configure kubelet to use the KubeletConfiguration file we put on nodes via KubeadmConfigSpec.files
          kubeletExtraArgs:
            config: "/etc/kubernetes/kubelet/config.yaml"

Set kubelet flags via KubeadmConfigSpec.kubeletExtraArgs

We can pass kubelet command-line flags via KubeadmConfigSpec.kubeletExtraArgs. This example is equivalent to setting --kube-reserved, --system-reserved, and --eviction-hard flags for the kubelet command.

This method is useful when you want to set kubelet flags that are not configurable via the KubeletConfiguration file, however, it is not recommended to use this method to set flags that are configurable via the KubeletConfiguration file.

KubeadmControlPlaneTemplate

apiVersion: controlplane.cluster.x-k8s.io/v1beta1
kind: KubeadmControlPlaneTemplate
metadata:
  name: kubelet-extra-args-control-plane
  namespace: default
spec:
  template:
    spec:
      kubeadmConfigSpec:
        clusterConfiguration:
          controllerManager:
            extraArgs:
              enable-hostpath-provisioner: "true"
        initConfiguration:
          nodeRegistration:
            criSocket: unix:///var/run/containerd/containerd.sock
            # Set kubelet flags via KubeadmConfigSpec.kubeletExtraArgs
            kubeletExtraArgs:
              kube-reserved: cpu=1,memory=2Gi,ephemeral-storage=1Gi
              system-reserved: cpu=500m,memory=1Gi,ephemeral-storage=1Gi
              eviction-hard: memory.available<500Mi,nodefs.available<10%
        joinConfiguration:
          nodeRegistration:
            criSocket: unix:///var/run/containerd/containerd.sock
            # Set kubelet flags via KubeadmConfigSpec.kubeletExtraArgs
            kubeletExtraArgs:
              kube-reserved: cpu=1,memory=2Gi,ephemeral-storage=1Gi
              system-reserved: cpu=500m,memory=1Gi,ephemeral-storage=1Gi
              eviction-hard: memory.available<500Mi,nodefs.available<10%

KubeadmConfigTemplate

apiVersion: bootstrap.cluster.x-k8s.io/v1beta1
kind: KubeadmConfigTemplate
metadata:
  name: kubelet-extra-args-default-worker-bootstraptemplate
  namespace: default
spec:
  template:
    spec:
      joinConfiguration:
        nodeRegistration:
          criSocket: unix:///var/run/containerd/containerd.sock
          # Set kubelet flags via KubeadmConfigSpec.kubeletExtraArgs
          kubeletExtraArgs:
            kube-reserved: cpu=1,memory=2Gi,ephemeral-storage=1Gi
            system-reserved: cpu=500m,memory=1Gi,ephemeral-storage=1Gi
            eviction-hard: memory.available<500Mi,nodefs.available<10%

Use kubeadm’s kubeletconfiguration patch target

We can use kubeadm’s kubeletconfiguration patch target to patch the kubelet configuration file. In this example, we put a patch file for kubeletconfiguration target in strategic patchtype on nodes via KubeadmConfigSpec.files. For more details, see Customizing components with the kubeadm API | Kubernetes

This method is useful when you want to change the kubelet configuration file partially on specific nodes. For example, you can deploy a partially patched kubelet configuration file on specific nodes based on the default configuration used for kubeadm init or kubeadm join.

KubeadmControlPlaneTemplate

apiVersion: controlplane.cluster.x-k8s.io/v1beta1
kind: KubeadmControlPlaneTemplate
metadata:
  name: kubeadm-config-template-control-plane
  namespace: default
spec:
  template:
    spec:
      kubeadmConfigSpec:
        files:
        # Here we put a patch file for kubeletconfiguration target in strategic patchtype on nodes via KubeadmConfigSpec.files
        # The naming convention of the patch file is kubeletconfiguration{suffix}+{patchtype}.json where {suffix} is an string and {patchtype} is one of the following: strategic, merge, json.
        # {suffix} determines the order of the patch files. The patches are applied in the alpha-numerical order of the {suffix}.
        - path: /etc/kubernetes/patches/kubeletconfiguration0+strategic.json
          owner: "root:root"
          permissions: "0644"
          content: |
            {
              "apiVersion": "kubelet.config.k8s.io/v1beta1",
              "kind": "KubeletConfiguration",
              "kubeReserved": {
                "cpu": "1",
                "memory": "2Gi",
                "ephemeral-storage": "1Gi",
              },
              "systemReserved": {
                "cpu": "500m",
                "memory": "1Gi",
                "ephemeral-storage": "1Gi",
              },
              "evictionHard": {
                "memory.available": "500Mi",
                "nodefs.available": "10%",
              },
            }
        clusterConfiguration:
          controllerManager:
            extraArgs:
              enable-hostpath-provisioner: "true"
        initConfiguration:
          nodeRegistration:
            criSocket: unix:///var/run/containerd/containerd.sock
          # Here we specify the directory that contains the patch files
          patches:
            directory: /etc/kubernetes/patches
        joinConfiguration:
          nodeRegistration:
            criSocket: unix:///var/run/containerd/containerd.sock
          # Here we specify the directory that contains the patch files
          patches:
            directory: /etc/kubernetes/patches

KubeadmConfigTemplate

apiVersion: bootstrap.cluster.x-k8s.io/v1beta1
kind: KubeadmConfigTemplate
metadata:
  name: kubeadm-config-template-default-worker-bootstraptemplate
  namespace: default
spec:
  template:
    spec:
      files:
      # Here we put a patch file for kubeletconfiguration target in strategic patchtype on nodes via KubeadmConfigSpec.files
      # The naming convention of the patch file is kubeletconfiguration{suffix}+{patchtype}.json where {suffix} is an string and {patchtype} is one of the following: strategic, merge, json.
      # {suffix} determines the order of the patch files. The patches are applied in the alpha-numerical order of the {suffix}.
      - path: /etc/kubernetes/patches/kubeletconfiguration0+strategic.json
        owner: "root:root"
        permissions: "0644"
        content: |
          {
            "apiVersion": "kubelet.config.k8s.io/v1beta1",
            "kind": "KubeletConfiguration",
            "kubeReserved": {
              "cpu": "1",
              "memory": "2Gi",
              "ephemeral-storage": "1Gi",
            },
            "systemReserved": {
              "cpu": "500m",
              "memory": "1Gi",
              "ephemeral-storage": "1Gi",
            },
            "evictionHard": {
              "memory.available": "500Mi",
              "nodefs.available": "10%",
            },
          }
      joinConfiguration:
        nodeRegistration:
          criSocket: unix:///var/run/containerd/containerd.sock
        # Here we specify the directory that contains the patch files
        patches:
          directory: /etc/kubernetes/patches