はじめに
Kubernetes 1.26 にアップグレードする前に、VPC CNI プラグインのバージョンを 1.12 以降にアップグレードする必要があります。以前のバージョンの VPC CNI は、Kubernetes v1.26 から削除された CRI v1alpha2API に依存しているため、CNI がクラッシュする原因になります。
本記事では、EKS クラスタ自体のアップグレードに関する手順は省略させていただきます。
Amazon VPC CNI plugin とは
Amazon VPC CNI plugin for Kubernetes アドオンは Amazon EKS クラスター内の各 Amazon EC2 ノードにデプロイされます。アドオンは Elastic Network Interface を作成し、Amazon EC2 ノードにアタッチします。またアドオンは、VPC のプライベートIPv4 または IPv6 アドレスを各 Pod およびサービスに割り当てます。
アドオンの利用状況確認
クラスターにインストールされているアドオンのバージョンを確認します。
$ kubectl describe daemonset aws-node -n kube-system | grep amazon-k8s-cni: Image: 602401143452.dkr.ecr.ap-northeast-1.amazonaws.com/amazon-k8s-cni:v1.7.5
クラスターに Amazon EKS タイプのアドオンがインストールされていないことを確認しました。
$ aws eks describe-addon --cluster-name **** --addon-name vpc-cni --query addon.addonVersion --output text An error occurred (ResourceNotFoundException) when calling the DescribeAddon operation: No addon: vpc-cni found in cluster: ****
クラスターの状態を確認します。
$ kubectl describe daemonset.apps/aws-node -n kube-system Name: aws-node Selector: k8s-app=aws-node Node-Selector:Labels: k8s-app=aws-node Annotations: deprecated.daemonset.template.generation: 3 Desired Number of Nodes Scheduled: 6 Current Number of Nodes Scheduled: 6 Number of Nodes Scheduled with Up-to-date Pods: 6 Number of Nodes Scheduled with Available Pods: 6 Number of Nodes Misscheduled: 0 Pods Status: 6 Running / 0 Waiting / 0 Succeeded / 0 Failed Pod Template: Labels: k8s-app=aws-node Service Account: aws-node Init Containers: aws-vpc-cni-init: Image: 602401143452.dkr.ecr.ap-northeast-1.amazonaws.com/amazon-k8s-cni-init:v1.7.5 Port: Host Port: Environment: DISABLE_TCP_EARLY_DEMUX: false Mounts: /host/opt/cni/bin from cni-bin-dir (rw) Containers: aws-node: Image: 602401143452.dkr.ecr.ap-northeast-1.amazonaws.com/amazon-k8s-cni:v1.7.5 Port: 61678/TCP Host Port: 61678/TCP Requests: cpu: 10m Liveness: exec [/app/grpc-health-probe -addr=:50051] delay=60s timeout=1s period=10s #success=1 #failure=3 Readiness: exec [/app/grpc-health-probe -addr=:50051] delay=1s timeout=1s period=10s #success=1 #failure=3 Environment: ADDITIONAL_ENI_TAGS: {} AWS_VPC_CNI_NODE_PORT_SUPPORT: true AWS_VPC_ENI_MTU: 9001 AWS_VPC_K8S_CNI_CONFIGURE_RPFILTER: false AWS_VPC_K8S_CNI_CUSTOM_NETWORK_CFG: false AWS_VPC_K8S_CNI_EXTERNALSNAT: false AWS_VPC_K8S_CNI_LOGLEVEL: DEBUG AWS_VPC_K8S_CNI_LOG_FILE: /host/var/log/aws-routed-eni/ipamd.log AWS_VPC_K8S_CNI_RANDOMIZESNAT: prng AWS_VPC_K8S_CNI_VETHPREFIX: eni AWS_VPC_K8S_PLUGIN_LOG_FILE: /var/log/aws-routed-eni/plugin.log AWS_VPC_K8S_PLUGIN_LOG_LEVEL: DEBUG DISABLE_INTROSPECTION: false DISABLE_METRICS: false ENABLE_POD_ENI: false MY_NODE_NAME: (v1:spec.nodeName) WARM_ENI_TARGET: 1 Mounts: /host/etc/cni/net.d from cni-net-dir (rw) /host/opt/cni/bin from cni-bin-dir (rw) /host/var/log/aws-routed-eni from log-dir (rw) /run/xtables.lock from xtables-lock (rw) /var/run/aws-node from run-dir (rw) /var/run/dockershim.sock from dockershim (rw) Volumes: cni-bin-dir: Type: HostPath (bare host directory volume) Path: /opt/cni/bin HostPathType: cni-net-dir: Type: HostPath (bare host directory volume) Path: /etc/cni/net.d HostPathType: dockershim: Type: HostPath (bare host directory volume) Path: /var/run/dockershim.sock HostPathType: xtables-lock: Type: HostPath (bare host directory volume) Path: /run/xtables.lock HostPathType: log-dir: Type: HostPath (bare host directory volume) Path: /var/log/aws-routed-eni HostPathType: DirectoryOrCreate run-dir: Type: HostPath (bare host directory volume) Path: /var/run/aws-node HostPathType: DirectoryOrCreate Priority Class Name: system-node-critical Events:
念の為、現在インストールされているアドオンの設定を保存します。
$ kubectl get daemonset aws-node -n kube-system -o yaml > aws-k8s-cni-old.yaml
アップデート用の設定ファイルの取得
$ curl -o aws-k8s-cni_v1.16.3.yaml https://raw.githubusercontent.com/aws/amazon-vpc-cni-k8s/v1.16.3/config/master/aws-k8s-cni.yaml % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 20218 100 20218 0 0 32319 0 --:--:-- --:--:-- --:--:-- 32348
CNI バージョンを最新バージョンに更新
$ kubectl apply -f aws-k8s-cni_v1.16.3.yaml customresourcedefinition.apiextensions.k8s.io/eniconfigs.crd.k8s.amazonaws.com configured customresourcedefinition.apiextensions.k8s.io/policyendpoints.networking.k8s.aws unchanged serviceaccount/aws-node configured configmap/amazon-vpc-cni created clusterrole.rbac.authorization.k8s.io/aws-node configured clusterrolebinding.rbac.authorization.k8s.io/aws-node configured daemonset.apps/aws-node configured
アドオンの新バージョンの確認
$ kubectl describe daemonset aws-node -n kube-system | grep amazon-k8s-cni: Image: 602401143452.dkr.ecr.us-west-2.amazonaws.com/amazon-k8s-cni:v1.16.3
クラスターの状態の確認：
$ kubectl describe daemonset.apps/aws-node -n kube-system
Name: aws-node
Selector: k8s-app=aws-node
Node-Selector:
Labels: app.kubernetes.io/instance=aws-vpc-cni
app.kubernetes.io/name=aws-node
app.kubernetes.io/version=v1.16.3
k8s-app=aws-node
Annotations: deprecated.daemonset.template.generation: 4
Desired Number of Nodes Scheduled: 6
Current Number of Nodes Scheduled: 6
Number of Nodes Scheduled with Up-to-date Pods: 6
Number of Nodes Scheduled with Available Pods: 6
Number of Nodes Misscheduled: 0
Pods Status: 6 Running / 0 Waiting / 0 Succeeded / 0 Failed
Pod Template:
Labels: app.kubernetes.io/instance=aws-vpc-cni
app.kubernetes.io/name=aws-node
k8s-app=aws-node
Service Account: aws-node
Init Containers:
aws-vpc-cni-init:
Image: 602401143452.dkr.ecr.us-west-2.amazonaws.com/amazon-k8s-cni-init:v1.16.3
Port:
Host Port:
Requests:
cpu: 25m
Environment:
DISABLE_TCP_EARLY_DEMUX: false
ENABLE_IPv6: false
Mounts:
/host/opt/cni/bin from cni-bin-dir (rw)
Containers:
aws-node:
Image: 602401143452.dkr.ecr.us-west-2.amazonaws.com/amazon-k8s-cni:v1.16.3
Port: 61678/TCP
Host Port: 61678/TCP
Requests:
cpu: 25m
Liveness: exec [/app/grpc-health-probe -addr=:50051 -connect-timeout=5s -rpc-timeout=5s] delay=60s timeout=10s period=10s #success=1 #failure=3
Readiness: exec [/app/grpc-health-probe -addr=:50051 -connect-timeout=5s -rpc-timeout=5s] delay=1s timeout=10s period=10s #success=1 #failure=3
Environment:
ADDITIONAL_ENI_TAGS: {}
AWS_VPC_CNI_NODE_PORT_SUPPORT: true
AWS_VPC_ENI_MTU: 9001
AWS_VPC_K8S_CNI_CUSTOM_NETWORK_CFG: false
AWS_VPC_K8S_CNI_EXTERNALSNAT: false
AWS_VPC_K8S_CNI_LOGLEVEL: DEBUG
AWS_VPC_K8S_CNI_LOG_FILE: /host/var/log/aws-routed-eni/ipamd.log
AWS_VPC_K8S_CNI_RANDOMIZESNAT: prng
AWS_VPC_K8S_CNI_VETHPREFIX: eni
AWS_VPC_K8S_PLUGIN_LOG_FILE: /var/log/aws-routed-eni/plugin.log
AWS_VPC_K8S_PLUGIN_LOG_LEVEL: DEBUG
DISABLE_INTROSPECTION: false
DISABLE_METRICS: false
DISABLE_NETWORK_RESOURCE_PROVISIONING: false
ENABLE_IPv4: true
ENABLE_IPv6: false
ENABLE_POD_ENI: false
ENABLE_PREFIX_DELEGATION: false
VPC_CNI_VERSION: v1.16.3
WARM_ENI_TARGET: 1
WARM_PREFIX_TARGET: 1
MY_NODE_NAME: (v1:spec.nodeName)
MY_POD_NAME: (v1:metadata.name)
Mounts:
/host/etc/cni/net.d from cni-net-dir (rw)
/host/opt/cni/bin from cni-bin-dir (rw)
/host/var/log/aws-routed-eni from log-dir (rw)
/run/xtables.lock from xtables-lock (rw)
/var/run/aws-node from run-dir (rw)
aws-eks-nodeagent:
Image: 602401143452.dkr.ecr.us-west-2.amazonaws.com/amazon/aws-network-policy-agent:v1.0.8
Port:
Host Port:
Args:
--enable-ipv6=false
--enable-network-policy=false
--enable-cloudwatch-logs=false
--enable-policy-event-logs=false
--metrics-bind-addr=:8162
--health-probe-bind-addr=:8163
--conntrack-cache-cleanup-period=300
Requests:
cpu: 25m
Environment:
MY_NODE_NAME: (v1:spec.nodeName)
Mounts:
/host/opt/cni/bin from cni-bin-dir (rw)
/sys/fs/bpf from bpf-pin-path (rw)
/var/log/aws-routed-eni from log-dir (rw)
/var/run/aws-node from run-dir (rw)
Volumes:
bpf-pin-path:
Type: HostPath (bare host directory volume)
Path: /sys/fs/bpf
HostPathType:
cni-bin-dir:
Type: HostPath (bare host directory volume)
Path: /opt/cni/bin
HostPathType:
cni-net-dir:
Type: HostPath (bare host directory volume)
Path: /etc/cni/net.d
HostPathType:
log-dir:
Type: HostPath (bare host directory volume)
Path: /var/log/aws-routed-eni
HostPathType: DirectoryOrCreate
run-dir:
Type: HostPath (bare host directory volume)
Path: /var/run/aws-node
HostPathType: DirectoryOrCreate
xtables-lock:
Type: HostPath (bare host directory volume)
Path: /run/xtables.lock
HostPathType:
Priority Class Name: system-node-critical
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal SuccessfulDelete 6m24s daemonset-controller Deleted pod: aws-node-8822m
Normal SuccessfulCreate 6m13s daemonset-controller Created pod: aws-node-m2x45
Normal SuccessfulDelete 5m11s daemonset-controller Deleted pod: aws-node-s7zk2
Normal SuccessfulCreate 5m daemonset-controller Created pod: aws-node-fl5lq
Normal SuccessfulDelete 4m24s daemonset-controller Deleted pod: aws-node-vzg4g
Normal SuccessfulCreate 4m14s daemonset-controller Created pod: aws-node-9gfnp
Normal SuccessfulDelete 3m26s daemonset-controller Deleted pod: aws-node-bqkmx
Normal SuccessfulCreate 3m15s daemonset-controller Created pod: aws-node-k746c
Normal SuccessfulDelete 2m19s daemonset-controller Deleted pod: aws-node-56k7c
Normal SuccessfulCreate 2m8s daemonset-controller Created pod: aws-node-7jzr5
Normal SuccessfulDelete 71s daemonset-controller Deleted pod: aws-node-b8zmw
Normal SuccessfulCreate 60s daemonset-controller Created pod: aws-node-b4plz
