目录
一.系统环境
| 服务器版本 | docker软件版本 | Kubernetes(k8s)集群版本 | CPU架构 |
|---|---|---|---|
| CentOS Linux release 7.4.1708 (Core) | Docker version 20.10.12 | v1.21.9 | x86_64 |
Kubernetes集群架构：k8scloude1作为master节点，k8scloude2，k8scloude3作为worker节点
| 服务器 | 操作系统版本 | CPU架构 | 进程 | 功能描述 |
|---|---|---|---|---|
| k8scloude1/192.168.110.130 | CentOS Linux release 7.4.1708 (Core) | x86_64 | docker，kube-apiserver，etcd，kube-scheduler，kube-controller-manager，kubelet，kube-proxy，coredns，calico | k8s master节点 |
| k8scloude2/192.168.110.129 | CentOS Linux release 7.4.1708 (Core) | x86_64 | docker，kubelet，kube-proxy，calico | k8s worker节点 |
| k8scloude3/192.168.110.128 | CentOS Linux release 7.4.1708 (Core) | x86_64 | docker，kubelet，kube-proxy，calico | k8s worker节点 |
二.前言
本文介绍污点taint 与容忍度tolerations，可以影响pod的调度。
使用污点taint 与容忍度tolerations的前提是已经有一套可以正常运行的Kubernetes集群，关于Kubernetes(k8s)集群的安装部署，可以查看博客《Centos7 安装部署Kubernetes(k8s)集群》https://www.cnblogs.com/renshengdezheli/p/16686769.html
三.污点taint
3.1 污点taint概览
节点亲和性 是 Pod 的一种属性，它使 Pod 被吸引到一类特定的节点 （这可能出于一种偏好，也可能是硬性要求）。 污点（Taint） 则相反——它使节点能够排斥一类特定的 Pod。
3.2 给节点添加污点taint
给节点增加一个污点的语法如下：给节点 node1 增加一个污点，它的键名是 key1，键值是 value1，效果是 NoSchedule。 这表示只有拥有和这个污点相匹配的容忍度的 Pod 才能够被分配到 node1 这个节点。
#污点的格式：键=值:NoSchedule kubectl taint nodes node1 key1=value1:NoSchedule #只有键没有值的话，格式为：键:NoSchedule kubectl taint nodes node1 key1:NoSchedule 移除污点语法如下：
kubectl taint nodes node1 key1=value1:NoSchedule- 节点的描述信息里有一个Taints字段，Taints字段表示节点有没有污点
[root@k8scloude1 deploy]# kubectl get nodes -o wide NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME k8scloude1 Ready control-plane,master 8d v1.21.0 192.168.110.130 <none> CentOS Linux 7 (Core) 3.10.0-693.el7.x86_64 docker://20.10.12 k8scloude2 Ready <none> 8d v1.21.0 192.168.110.129 <none> CentOS Linux 7 (Core) 3.10.0-693.el7.x86_64 docker://20.10.12 k8scloude3 Ready <none> 8d v1.21.0 192.168.110.128 <none> CentOS Linux 7 (Core) 3.10.0-693.el7.x86_64 docker://20.10.12 [root@k8scloude1 deploy]# kubectl describe nodes k8scloude1 Name: k8scloude1 Roles: control-plane,master Labels: beta.kubernetes.io/arch=amd64 beta.kubernetes.io/os=linux kubernetes.io/arch=amd64 kubernetes.io/hostname=k8scloude1 kubernetes.io/os=linux node-role.kubernetes.io/control-plane= node-role.kubernetes.io/master= node.kubernetes.io/exclude-from-external-load-balancers= Annotations: kubeadm.alpha.kubernetes.io/cri-socket: /var/run/dockershim.sock node.alpha.kubernetes.io/ttl: 0 projectcalico.org/IPv4Address: 192.168.110.130/24 projectcalico.org/IPv4IPIPTunnelAddr: 10.244.158.64 volumes.kubernetes.io/controller-managed-attach-detach: true CreationTimestamp: Sun, 09 Jan 2022 16:19:06 +0800 Taints: node-role.kubernetes.io/master:NoSchedule Unschedulable: false ...... 查看节点是否有污点，Taints: node-role.kubernetes.io/master:NoSchedule表示k8s集群的master节点有污点，这是默认就存在的污点，这也是master节点为什么不能运行应用pod的原因。
[root@k8scloude1 deploy]# kubectl describe nodes k8scloude2 | grep -i Taints Taints: <none> [root@k8scloude1 deploy]# kubectl describe nodes k8scloude1 | grep -i Taints Taints: node-role.kubernetes.io/master:NoSchedule [root@k8scloude1 deploy]# kubectl describe nodes k8scloude3 | grep -i Taints Taints: <none> 创建pod，nodeSelector:kubernetes.io/hostname: k8scloude1表示pod运行在标签为kubernetes.io/hostname=k8scloude1的节点上。
关于pod的调度详细内容，请查看博客《pod(八)：pod的调度——将 Pod 指派给节点》https://www.cnblogs.com/renshengdezheli/p/16863405.html
[root@k8scloude1 pod]# vim schedulepod4.yaml [root@k8scloude1 pod]# cat schedulepod4.yaml apiVersion: v1 kind: Pod metadata: creationTimestamp: null labels: run: pod1 name: pod1 namespace: pod spec: nodeSelector: kubernetes.io/hostname: k8scloude1 containers: - image: nginx imagePullPolicy: IfNotPresent name: pod1 resources: {} ports: - name: http containerPort: 80 protocol: TCP hostPort: 80 dnsPolicy: ClusterFirst restartPolicy: Always status: {} 标签为kubernetes.io/hostname=k8scloude1的节点为k8scloude1节点
[root@k8scloude1 pod]# kubectl get nodes -l kubernetes.io/hostname=k8scloude1 NAME STATUS ROLES AGE VERSION k8scloude1 Ready control-plane,master 8d v1.21.0 创建pod，因为k8scloude1上有污点，pod1不能运行在k8scloude1上，所以pod1状态为Pending
[root@k8scloude1 pod]# kubectl apply -f schedulepod4.yaml pod/pod1 created #因为k8scloude1上有污点，pod1不能运行在k8scloude1上，所以pod1状态为Pending [root@k8scloude1 pod]# kubectl get pod -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES pod1 0/1 Pending 0 9s <none> <none> <none> <none> [root@k8scloude1 pod]# kubectl delete pod pod1 --force warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. pod "pod1" force deleted [root@k8scloude1 pod]# kubectl get pod -o wide No resources found in pod namespace. 四.容忍度tolerations
4.1 容忍度tolerations概览
容忍度（Toleration） 是应用于 Pod 上的。容忍度允许调度器调度带有对应污点的 Pod。 容忍度允许调度但并不保证调度：作为其功能的一部分， 调度器也会评估其他参数。
污点和容忍度（Toleration）相互配合，可以用来避免 Pod 被分配到不合适的节点上。 每个节点上都可以应用一个或多个污点，这表示对于那些不能容忍这些污点的 Pod， 是不会被该节点接受的。
4.2 设置容忍度tolerations
只有拥有和这个污点相匹配的容忍度的 Pod 才能够被分配到 node节点。
查看k8scloude1节点的污点
[root@k8scloude1 pod]# kubectl describe nodes k8scloude1 | grep -i taint Taints: node-role.kubernetes.io/master:NoSchedule 你可以在 Pod 规约中为 Pod 设置容忍度，创建pod，tolerations参数表示可以容忍污点：node-role.kubernetes.io/master:NoSchedule ，nodeSelector:kubernetes.io/hostname: k8scloude1表示pod运行在标签为kubernetes.io/hostname=k8scloude1的节点上。
[root@k8scloude1 pod]# vim schedulepod4.yaml [root@k8scloude1 pod]# cat schedulepod4.yaml apiVersion: v1 kind: Pod metadata: creationTimestamp: null labels: run: pod1 name: pod1 namespace: pod spec: tolerations: - key: "node-role.kubernetes.io/master" operator: "Equal" value: "" effect: "NoSchedule" nodeSelector: kubernetes.io/hostname: k8scloude1 containers: - image: nginx imagePullPolicy: IfNotPresent name: pod1 resources: {} ports: - name: http containerPort: 80 protocol: TCP hostPort: 80 dnsPolicy: ClusterFirst restartPolicy: Always status: {} [root@k8scloude1 pod]# kubectl get pods -o wide No resources found in pod namespace. [root@k8scloude1 pod]# kubectl apply -f schedulepod4.yaml pod/pod1 created 查看pod，即使k8scloude1节点有污点，pod还是正常运行。
taint污点和cordon，drain的区别：某个节点上有污点，可以设置tolerations容忍度，让pod运行在该节点，某个节点被cordon，drain，则该节点不能被分配出去运行pod。
关于cordon，drain的详细信息，请查看博客《cordon节点，drain驱逐节点，delete 节点》https://www.cnblogs.com/renshengdezheli/p/16860674.html
[root@k8scloude1 pod]# kubectl get pods -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES pod1 1/1 Running 0 4s 10.244.158.84 k8scloude1 <none> <none> [root@k8scloude1 pod]# kubectl delete pod pod1 --force warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. pod "pod1" force deleted [root@k8scloude1 pod]# kubectl get pods -o wide No resources found in pod namespace. 注意，tolerations容忍度有两种写法，任选一种即可:
tolerations: - key: "key1" operator: "Equal" value: "value1" effect: "NoSchedule" tolerations: - key: "key1" operator: "Exists" effect: "NoSchedule" 给k8scloude2节点打标签
[root@k8scloude1 pod]# kubectl label nodes k8scloude2 taint=T node/k8scloude2 labeled [root@k8scloude1 pod]# kubectl get node --show-labels NAME STATUS ROLES AGE VERSION LABELS k8scloude1 Ready control-plane,master 8d v1.21.0 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=k8scloude1,kubernetes.io/os=linux,node-role.kubernetes.io/control-plane=,node-role.kubernetes.io/master=,node.kubernetes.io/exclude-from-external-load-balancers= k8scloude2 Ready <none> 8d v1.21.0 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=k8scloude2,kubernetes.io/os=linux,taint=T k8scloude3 Ready <none> 8d v1.21.0 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=k8scloude3,kubernetes.io/os=linux 对k8scloude2设置污点
#污点taint的格式：键=值:NoSchedule [root@k8scloude1 pod]# kubectl taint node k8scloude2 wudian=true:NoSchedule node/k8scloude2 tainted [root@k8scloude1 pod]# kubectl describe nodes k8scloude2 | grep -i Taints Taints: wudian=true:NoSchedule 创建pod，tolerations参数表示容忍污点wudian=true:NoSchedule，nodeSelector:taint: T参数表示pod运行在标签为nodeSelector=taint: T的节点。
[root@k8scloude1 pod]# vim schedulepod4.yaml [root@k8scloude1 pod]# cat schedulepod4.yaml apiVersion: v1 kind: Pod metadata: creationTimestamp: null labels: run: pod1 name: pod1 namespace: pod spec: tolerations: - key: "wudian" operator: "Equal" value: "true" effect: "NoSchedule" nodeSelector: taint: T containers: - image: nginx imagePullPolicy: IfNotPresent name: pod1 resources: {} ports: - name: http containerPort: 80 protocol: TCP hostPort: 80 dnsPolicy: ClusterFirst restartPolicy: Always status: {} [root@k8scloude1 pod]# kubectl get pod -o wide No resources found in pod namespace. [root@k8scloude1 pod]# kubectl apply -f schedulepod4.yaml pod/pod1 created 查看pod，k8scloude2节点就算有污点也能运行pod
[root@k8scloude1 pod]# kubectl get pods -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES pod1 1/1 Running 0 8s 10.244.112.177 k8scloude2 <none> <none> [root@k8scloude1 pod]# kubectl delete pod pod1 --force warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. pod "pod1" force deleted [root@k8scloude1 pod]# kubectl get pods -o wide No resources found in pod namespace. 污点容忍的另一种写法：operator: "Exists"，没有value值。
[root@k8scloude1 pod]# vim schedulepod4.yaml [root@k8scloude1 pod]# cat schedulepod4.yaml apiVersion: v1 kind: Pod metadata: creationTimestamp: null labels: run: pod1 name: pod1 namespace: pod spec: tolerations: - key: "wudian" operator: "Exists" effect: "NoSchedule" nodeSelector: taint: T containers: - image: nginx imagePullPolicy: IfNotPresent name: pod1 resources: {} ports: - name: http containerPort: 80 protocol: TCP hostPort: 80 dnsPolicy: ClusterFirst restartPolicy: Always status: {} [root@k8scloude1 pod]# kubectl apply -f schedulepod4.yaml pod/pod1 created 查看pod，k8scloude2节点就算有污点也能运行pod
[root@k8scloude1 pod]# kubectl get pods -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES pod1 1/1 Running 0 10s 10.244.112.178 k8scloude2 <none> <none> [root@k8scloude1 pod]# kubectl delete pod pod1 --force warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. pod "pod1" force deleted [root@k8scloude1 pod]# kubectl get pods -o wide No resources found in pod namespace. 给k8scloude2节点再添加一个污点
[root@k8scloude1 pod]# kubectl describe nodes k8scloude2 | grep Taints Taints: wudian=true:NoSchedule [root@k8scloude1 pod]# kubectl taint node k8scloude2 zang=shide:NoSchedule node/k8scloude2 tainted [root@k8scloude1 pod]# kubectl describe nodes k8scloude2 | grep Taints Taints: wudian=true:NoSchedule [root@k8scloude1 pod]# kubectl describe nodes k8scloude2 | grep -A2 Taints Taints: wudian=true:NoSchedule zang=shide:NoSchedule Unschedulable: false [root@k8scloude1 pod]# kubectl describe nodes k8scloude2 | grep -A1 Taints Taints: wudian=true:NoSchedule zang=shide:NoSchedule 创建pod，tolerations参数表示容忍2个污点：wudian=true:NoSchedule和zang=shide:NoSchedule，nodeSelector:taint: T参数表示pod运行在标签为nodeSelector=taint: T的节点。
[root@k8scloude1 pod]# vim schedulepod4.yaml [root@k8scloude1 pod]# cat schedulepod4.yaml apiVersion: v1 kind: Pod metadata: creationTimestamp: null labels: run: pod1 name: pod1 namespace: pod spec: tolerations: - key: "wudian" operator: "Equal" value: "true" effect: "NoSchedule" - key: "zang" operator: "Equal" value: "shide" effect: "NoSchedule" nodeSelector: taint: T containers: - image: nginx imagePullPolicy: IfNotPresent name: pod1 resources: {} ports: - name: http containerPort: 80 protocol: TCP hostPort: 80 dnsPolicy: ClusterFirst restartPolicy: Always status: {} [root@k8scloude1 pod]# kubectl apply -f schedulepod4.yaml pod/pod1 created 查看pod，k8scloude2节点就算有2个污点也能运行pod
[root@k8scloude1 pod]# kubectl get pods -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES pod1 1/1 Running 0 6s 10.244.112.179 k8scloude2 <none> <none> [root@k8scloude1 pod]# kubectl delete pod pod1 --force warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. pod "pod1" force deleted 创建pod，tolerations参数表示容忍污点：wudian=true:NoSchedule，nodeSelector:taint: T参数表示pod运行在标签为nodeSelector=taint: T的节点。
[root@k8scloude1 pod]# vim schedulepod4.yaml [root@k8scloude1 pod]# cat schedulepod4.yaml apiVersion: v1 kind: Pod metadata: creationTimestamp: null labels: run: pod1 name: pod1 namespace: pod spec: tolerations: - key: "wudian" operator: "Equal" value: "true" effect: "NoSchedule" nodeSelector: taint: T containers: - image: nginx imagePullPolicy: IfNotPresent name: pod1 resources: {} ports: - name: http containerPort: 80 protocol: TCP hostPort: 80 dnsPolicy: ClusterFirst restartPolicy: Always status: {} [root@k8scloude1 pod]# kubectl apply -f schedulepod4.yaml pod/pod1 created 查看pod，一个节点有两个污点值，但是yaml文件只容忍一个，所以pod创建不成功。
[root@k8scloude1 pod]# kubectl get pods -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES pod1 0/1 Pending 0 8s <none> <none> <none> <none> [root@k8scloude1 pod]# kubectl delete pod pod1 --force warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. pod "pod1" force deleted [root@k8scloude1 pod]# kubectl get pods -o wide No resources found in pod namespace. 取消k8scloude2污点
[root@k8scloude1 pod]# kubectl describe nodes k8scloude2 | grep -A2 Taints Taints: wudian=true:NoSchedule zang=shide:NoSchedule Unschedulable: false #取消污点 [root@k8scloude1 pod]# kubectl taint node k8scloude2 zang- node/k8scloude2 untainted [root@k8scloude1 pod]# kubectl taint node k8scloude2 wudian- node/k8scloude2 untainted [root@k8scloude1 pod]# kubectl describe nodes k8scloude1 | grep -A2 Taints Taints: node-role.kubernetes.io/master:NoSchedule Unschedulable: false Lease: [root@k8scloude1 pod]# kubectl describe nodes k8scloude2 | grep -A2 Taints Taints: <none> Unschedulable: false Lease: [root@k8scloude1 pod]# kubectl describe nodes k8scloude3 | grep -A2 Taints Taints: <none> Unschedulable: false Lease: Tips:如果自身机器有限，只能有一台机器，则可以把master节点的污点taint取消，就可以在master上运行pod了。
