Kubelet资源预留

发表于 | 分类于 | 浏览 次

Kubelet Node Allocatable

  • Kubelet Node Allocatable用来为Kube组件和System进程预留资源，从而保证当节点出现满负荷时也能保证Kube和System进程有足够的资源。
  • 目前支持cpu, memory, ephemeral-storage三种资源预留。
  • Node Capacity是Node的所有硬件资源，kube-reserved是给kube组件预留的资源，system-reserved是给System进程预留的资源， eviction-threshold是kubelet eviction的阈值设定，allocatable才是真正scheduler调度Pod时的参考值（保证Node上所有Pods的request resource不超过Allocatable）。
  • Node Allocatable Resource = Node Capacity - Kube-reserved - system-reserved - eviction-threshold

如何配置

  • –enforce-node-allocatable，默认为pods，要为kube组件和System进程预留资源，则需要设置为pods,kube-reserved,system-reserve。
  • –cgroups-per-qos，Enabling QoS and Pod level cgroups，默认开启。开启后，kubelet会将管理所有workload Pods的cgroups。
  • –cgroup-driver，默认为cgroupfs，另一可选项为systemd。取决于容器运行时使用的cgroup driver，kubelet与其保持一致。比如你配置docker使用systemd cgroup driver，那么kubelet也需要配置–cgroup-driver=systemd。
  • –kube-reserved,用于配置为kube组件（kubelet,kube-proxy,dockerd等）预留的资源量，比如—kube-reserved=cpu=1000m,memory=8Gi，ephemeral-storage=16Gi。
  • –kube-reserved-cgroup，如果你设置了–kube-reserved，那么请一定要设置对应的cgroup，并且该cgroup目录要事先创建好，否则kubelet将不会自动创建导致kubelet启动失败。比如设置为kube-reserved-cgroup=/kubelet.service 。
  • –system-reserved，用于配置为System进程预留的资源量，比如—system-reserved=cpu=500m,memory=4Gi,ephemeral-storage=4Gi。
  • –system-reserved-cgroup，如果你设置了–system-reserved，那么请一定要设置对应的cgroup，并且该cgroup目录要事先创建好，否则kubelet将不会自动创建导致kubelet启动失败。比如设置为system-reserved-cgroup=/system.slice。
  • –eviction-hard，用来配置kubelet的hard eviction条件，只支持memory和ephemeral-storage两种不可压缩资源。当出现MemoryPressure时，Scheduler不会调度新的Best-Effort QoS Pods到此节点。当出现DiskPressure时，Scheduler不会调度任何新Pods到此节点。

Sample

1
2
3
4
5
6
7
8
9
10
11
12
kubeadm安装的k8s：
[root@node1 kubelet.service.d]# systemctl status kubelet
● kubelet.service - kubelet: The Kubernetes Node Agent
   Loaded: loaded (/usr/lib/systemd/system/kubelet.service; enabled; vendor preset: disabled)
  Drop-In: /usr/lib/systemd/system/kubelet.service.d
           └─10-kubeadm.conf
   Active: active (running) since Mon 2019-09-09 13:55:37 CST; 18min ago
     Docs: https://kubernetes.io/docs/
 Main PID: 7768 (kubelet)
    Tasks: 20
   Memory: 43.7M
   CGroup: /system.slice/kubelet.service
1
2
3
4
5
6
7
8
9
10
11
12
[root@node1 kubelet.service.d]# cat 10-kubeadm.conf 
# Note: This dropin only works with kubeadm and kubelet v1.11+
[Service]
Environment="KUBELET_KUBECONFIG_ARGS=--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf"
Environment="KUBELET_CONFIG_ARGS=--config=/var/lib/kubelet/config.yaml"
# This is a file that "kubeadm init" and "kubeadm join" generates at runtime, populating the KUBELET_KUBEADM_ARGS variable dynamically
EnvironmentFile=-/var/lib/kubelet/kubeadm-flags.env
# This is a file that the user can use for overrides of the kubelet args as a last resort. Preferably, the user should use
# the .NodeRegistration.KubeletExtraArgs object in the configuration files instead. KUBELET_EXTRA_ARGS should be sourced from this file.
EnvironmentFile=-/etc/sysconfig/kubelet
ExecStart=
ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_KUBEADM_ARGS $KUBELET_EXTRA_ARGS
1
2
[root@node1 kubelet.service.d]# cat /var/lib/kubelet/kubeadm-flags.env
KUBELET_KUBEADM_ARGS="--system-reserved=memory=300Mi         --kube-reserved=memory=400Mi         --eviction-hard=imagefs.available<15%,memory.available<300Mi,nodefs.available<10%,nodefs.inodesFree<5% --cgroup-driver=systemd --network-plugin=cni --pod-infra-container-image=gcr.azk8s.cn/google_containers/pause:3.1"
Donate