e2e-kops-grid-cilium-eni-amzn2-k26 #1605974498819969024
Recent runs || View in Spyglass
| Result | FAILURE |
exit status 255
from junit_runner.xml
Filter through log files | View test history on testgrid
kubetest2 Down
kubetest2 Up
... skipping 135 lines ... I1222 17:13:50.963973 5510 http.go:37] curl https://storage.googleapis.com/kops-ci/bin/latest-ci-updown-green.txt I1222 17:13:50.965818 5510 http.go:37] curl https://storage.googleapis.com/kops-ci/bin/1.26.0-alpha.3+v1.26.0-alpha.2-339-g6e36b7853a/linux/amd64/kops I1222 17:13:51.937756 5510 local.go:42] ⚙️ ssh-keygen -t ed25519 -N -q -f /tmp/kops/e2e-e2e-kops-grid-cilium-eni-amzn2-k26.test-cncf-aws.k8s.io/id_ed25519 I1222 17:13:51.962623 5510 up.go:44] Cleaning up any leaked resources from previous cluster I1222 17:13:51.962712 5510 dumplogs.go:45] /home/prow/go/src/k8s.io/kops/_rundir/c6ac28b3-821b-11ed-b47a-e2b55b2f9198/kops toolbox dump --name e2e-e2e-kops-grid-cilium-eni-amzn2-k26.test-cncf-aws.k8s.io --dir /logs/artifacts --private-key /tmp/kops/e2e-e2e-kops-grid-cilium-eni-amzn2-k26.test-cncf-aws.k8s.io/id_ed25519 --ssh-user ec2-user I1222 17:13:51.962732 5510 local.go:42] ⚙️ /home/prow/go/src/k8s.io/kops/_rundir/c6ac28b3-821b-11ed-b47a-e2b55b2f9198/kops toolbox dump --name e2e-e2e-kops-grid-cilium-eni-amzn2-k26.test-cncf-aws.k8s.io --dir /logs/artifacts --private-key /tmp/kops/e2e-e2e-kops-grid-cilium-eni-amzn2-k26.test-cncf-aws.k8s.io/id_ed25519 --ssh-user ec2-user W1222 17:13:52.452460 5510 down.go:34] Dumping cluster logs at the start of Down() failed: exit status 1 I1222 17:13:52.452514 5510 down.go:48] /home/prow/go/src/k8s.io/kops/_rundir/c6ac28b3-821b-11ed-b47a-e2b55b2f9198/kops delete cluster --name e2e-e2e-kops-grid-cilium-eni-amzn2-k26.test-cncf-aws.k8s.io --yes I1222 17:13:52.452525 5510 local.go:42] ⚙️ /home/prow/go/src/k8s.io/kops/_rundir/c6ac28b3-821b-11ed-b47a-e2b55b2f9198/kops delete cluster --name e2e-e2e-kops-grid-cilium-eni-amzn2-k26.test-cncf-aws.k8s.io --yes I1222 17:13:52.485218 5543 featureflag.go:166] FeatureFlag "SpecOverrideFlag"=true Error: error reading cluster configuration: Cluster.kops.k8s.io "e2e-e2e-kops-grid-cilium-eni-amzn2-k26.test-cncf-aws.k8s.io" not found I1222 17:13:52.913641 5510 http.go:37] curl http://metadata.google.internal/computeMetadata/v1/instance/network-interfaces/0/access-configs/0/external-ip 2022/12/22 17:13:52 failed to get external ip from metadata service: http://metadata.google.internal/computeMetadata/v1/instance/network-interfaces/0/access-configs/0/external-ip returned 404 I1222 17:13:52.926541 5510 http.go:37] curl https://ip.jsb.workers.dev I1222 17:13:53.035017 5510 up.go:167] /home/prow/go/src/k8s.io/kops/_rundir/c6ac28b3-821b-11ed-b47a-e2b55b2f9198/kops create cluster --name e2e-e2e-kops-grid-cilium-eni-amzn2-k26.test-cncf-aws.k8s.io --cloud aws --kubernetes-version https://storage.googleapis.com/kubernetes-release/release/v1.26.0 --ssh-public-key /tmp/kops/e2e-e2e-kops-grid-cilium-eni-amzn2-k26.test-cncf-aws.k8s.io/id_ed25519.pub --set cluster.spec.nodePortAccess=0.0.0.0/0 --image=137112412989/amzn2-ami-kernel-5.10-hvm-2.0.20221210.1-x86_64-gp2 --channel=alpha --networking=cilium-eni --container-runtime=containerd --admin-access 35.188.115.87/32 --master-count 1 --master-volume-size 48 --node-count 4 --node-volume-size 48 --zones us-west-2a --master-size c5.large I1222 17:13:53.035061 5510 local.go:42] ⚙️ /home/prow/go/src/k8s.io/kops/_rundir/c6ac28b3-821b-11ed-b47a-e2b55b2f9198/kops create cluster --name e2e-e2e-kops-grid-cilium-eni-amzn2-k26.test-cncf-aws.k8s.io --cloud aws --kubernetes-version https://storage.googleapis.com/kubernetes-release/release/v1.26.0 --ssh-public-key /tmp/kops/e2e-e2e-kops-grid-cilium-eni-amzn2-k26.test-cncf-aws.k8s.io/id_ed25519.pub --set cluster.spec.nodePortAccess=0.0.0.0/0 --image=137112412989/amzn2-ami-kernel-5.10-hvm-2.0.20221210.1-x86_64-gp2 --channel=alpha --networking=cilium-eni --container-runtime=containerd --admin-access 35.188.115.87/32 --master-count 1 --master-volume-size 48 --node-count 4 --node-volume-size 48 --zones us-west-2a --master-size c5.large I1222 17:13:53.067722 5553 featureflag.go:166] FeatureFlag "SpecOverrideFlag"=true I1222 17:13:53.083919 5553 create_cluster.go:863] Using SSH public key: /tmp/kops/e2e-e2e-kops-grid-cilium-eni-amzn2-k26.test-cncf-aws.k8s.io/id_ed25519.pub I1222 17:13:53.518839 5553 new_cluster.go:1337] Cloud Provider ID: "aws" ... skipping 516 lines ... NODE STATUS NAME ROLE READY VALIDATION ERRORS KIND NAME MESSAGE dns apiserver Validation Failed The dns-controller Kubernetes deployment has not updated the Kubernetes cluster's API DNS entry to the correct IP address. The API DNS IP address is the placeholder address that kops creates: 203.0.113.123. Please wait about 5-10 minutes for a control plane node to start, dns-controller to launch, and DNS to propagate. The protokube container and dns-controller deployment logs may contain more diagnostic information. Etcd and the API DNS entries must be updated for a kops Kubernetes cluster to start. Validation Failed W1222 17:14:37.095730 5592 validate_cluster.go:232] (will retry): cluster not yet healthy INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-west-2a ControlPlane c5.large 1 1 us-west-2a nodes-us-west-2a Node t3.medium 4 4 us-west-2a NODE STATUS NAME ROLE READY VALIDATION ERRORS KIND NAME MESSAGE dns apiserver Validation Failed The dns-controller Kubernetes deployment has not updated the Kubernetes cluster's API DNS entry to the correct IP address. The API DNS IP address is the placeholder address that kops creates: 203.0.113.123. Please wait about 5-10 minutes for a control plane node to start, dns-controller to launch, and DNS to propagate. The protokube container and dns-controller deployment logs may contain more diagnostic information. Etcd and the API DNS entries must be updated for a kops Kubernetes cluster to start. Validation Failed W1222 17:14:47.145759 5592 validate_cluster.go:232] (will retry): cluster not yet healthy INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-west-2a ControlPlane c5.large 1 1 us-west-2a nodes-us-west-2a Node t3.medium 4 4 us-west-2a NODE STATUS NAME ROLE READY VALIDATION ERRORS KIND NAME MESSAGE dns apiserver Validation Failed The dns-controller Kubernetes deployment has not updated the Kubernetes cluster's API DNS entry to the correct IP address. The API DNS IP address is the placeholder address that kops creates: 203.0.113.123. Please wait about 5-10 minutes for a control plane node to start, dns-controller to launch, and DNS to propagate. The protokube container and dns-controller deployment logs may contain more diagnostic information. Etcd and the API DNS entries must be updated for a kops Kubernetes cluster to start. Validation Failed W1222 17:14:57.188262 5592 validate_cluster.go:232] (will retry): cluster not yet healthy INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-west-2a ControlPlane c5.large 1 1 us-west-2a nodes-us-west-2a Node t3.medium 4 4 us-west-2a NODE STATUS NAME ROLE READY VALIDATION ERRORS KIND NAME MESSAGE dns apiserver Validation Failed The dns-controller Kubernetes deployment has not updated the Kubernetes cluster's API DNS entry to the correct IP address. The API DNS IP address is the placeholder address that kops creates: 203.0.113.123. Please wait about 5-10 minutes for a control plane node to start, dns-controller to launch, and DNS to propagate. The protokube container and dns-controller deployment logs may contain more diagnostic information. Etcd and the API DNS entries must be updated for a kops Kubernetes cluster to start. Validation Failed W1222 17:15:07.220376 5592 validate_cluster.go:232] (will retry): cluster not yet healthy INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-west-2a ControlPlane c5.large 1 1 us-west-2a nodes-us-west-2a Node t3.medium 4 4 us-west-2a NODE STATUS NAME ROLE READY VALIDATION ERRORS KIND NAME MESSAGE dns apiserver Validation Failed The dns-controller Kubernetes deployment has not updated the Kubernetes cluster's API DNS entry to the correct IP address. The API DNS IP address is the placeholder address that kops creates: 203.0.113.123. Please wait about 5-10 minutes for a control plane node to start, dns-controller to launch, and DNS to propagate. The protokube container and dns-controller deployment logs may contain more diagnostic information. Etcd and the API DNS entries must be updated for a kops Kubernetes cluster to start. Validation Failed W1222 17:15:17.266709 5592 validate_cluster.go:232] (will retry): cluster not yet healthy INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-west-2a ControlPlane c5.large 1 1 us-west-2a nodes-us-west-2a Node t3.medium 4 4 us-west-2a NODE STATUS NAME ROLE READY VALIDATION ERRORS KIND NAME MESSAGE dns apiserver Validation Failed The dns-controller Kubernetes deployment has not updated the Kubernetes cluster's API DNS entry to the correct IP address. The API DNS IP address is the placeholder address that kops creates: 203.0.113.123. Please wait about 5-10 minutes for a control plane node to start, dns-controller to launch, and DNS to propagate. The protokube container and dns-controller deployment logs may contain more diagnostic information. Etcd and the API DNS entries must be updated for a kops Kubernetes cluster to start. Validation Failed W1222 17:15:27.301517 5592 validate_cluster.go:232] (will retry): cluster not yet healthy INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-west-2a ControlPlane c5.large 1 1 us-west-2a nodes-us-west-2a Node t3.medium 4 4 us-west-2a NODE STATUS NAME ROLE READY VALIDATION ERRORS KIND NAME MESSAGE dns apiserver Validation Failed The dns-controller Kubernetes deployment has not updated the Kubernetes cluster's API DNS entry to the correct IP address. The API DNS IP address is the placeholder address that kops creates: 203.0.113.123. Please wait about 5-10 minutes for a control plane node to start, dns-controller to launch, and DNS to propagate. The protokube container and dns-controller deployment logs may contain more diagnostic information. Etcd and the API DNS entries must be updated for a kops Kubernetes cluster to start. Validation Failed W1222 17:15:37.362734 5592 validate_cluster.go:232] (will retry): cluster not yet healthy INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-west-2a ControlPlane c5.large 1 1 us-west-2a nodes-us-west-2a Node t3.medium 4 4 us-west-2a NODE STATUS NAME ROLE READY VALIDATION ERRORS KIND NAME MESSAGE dns apiserver Validation Failed The dns-controller Kubernetes deployment has not updated the Kubernetes cluster's API DNS entry to the correct IP address. The API DNS IP address is the placeholder address that kops creates: 203.0.113.123. Please wait about 5-10 minutes for a control plane node to start, dns-controller to launch, and DNS to propagate. The protokube container and dns-controller deployment logs may contain more diagnostic information. Etcd and the API DNS entries must be updated for a kops Kubernetes cluster to start. Validation Failed W1222 17:15:47.409529 5592 validate_cluster.go:232] (will retry): cluster not yet healthy INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-west-2a ControlPlane c5.large 1 1 us-west-2a nodes-us-west-2a Node t3.medium 4 4 us-west-2a NODE STATUS NAME ROLE READY VALIDATION ERRORS KIND NAME MESSAGE dns apiserver Validation Failed The dns-controller Kubernetes deployment has not updated the Kubernetes cluster's API DNS entry to the correct IP address. The API DNS IP address is the placeholder address that kops creates: 203.0.113.123. Please wait about 5-10 minutes for a control plane node to start, dns-controller to launch, and DNS to propagate. The protokube container and dns-controller deployment logs may contain more diagnostic information. Etcd and the API DNS entries must be updated for a kops Kubernetes cluster to start. Validation Failed W1222 17:15:57.445998 5592 validate_cluster.go:232] (will retry): cluster not yet healthy INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-west-2a ControlPlane c5.large 1 1 us-west-2a nodes-us-west-2a Node t3.medium 4 4 us-west-2a NODE STATUS NAME ROLE READY VALIDATION ERRORS KIND NAME MESSAGE dns apiserver Validation Failed The dns-controller Kubernetes deployment has not updated the Kubernetes cluster's API DNS entry to the correct IP address. The API DNS IP address is the placeholder address that kops creates: 203.0.113.123. Please wait about 5-10 minutes for a control plane node to start, dns-controller to launch, and DNS to propagate. The protokube container and dns-controller deployment logs may contain more diagnostic information. Etcd and the API DNS entries must be updated for a kops Kubernetes cluster to start. Validation Failed W1222 17:16:07.484651 5592 validate_cluster.go:232] (will retry): cluster not yet healthy INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-west-2a ControlPlane c5.large 1 1 us-west-2a nodes-us-west-2a Node t3.medium 4 4 us-west-2a NODE STATUS NAME ROLE READY VALIDATION ERRORS KIND NAME MESSAGE dns apiserver Validation Failed The dns-controller Kubernetes deployment has not updated the Kubernetes cluster's API DNS entry to the correct IP address. The API DNS IP address is the placeholder address that kops creates: 203.0.113.123. Please wait about 5-10 minutes for a control plane node to start, dns-controller to launch, and DNS to propagate. The protokube container and dns-controller deployment logs may contain more diagnostic information. Etcd and the API DNS entries must be updated for a kops Kubernetes cluster to start. Validation Failed W1222 17:16:17.530586 5592 validate_cluster.go:232] (will retry): cluster not yet healthy INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-west-2a ControlPlane c5.large 1 1 us-west-2a nodes-us-west-2a Node t3.medium 4 4 us-west-2a NODE STATUS NAME ROLE READY VALIDATION ERRORS KIND NAME MESSAGE dns apiserver Validation Failed The dns-controller Kubernetes deployment has not updated the Kubernetes cluster's API DNS entry to the correct IP address. The API DNS IP address is the placeholder address that kops creates: 203.0.113.123. Please wait about 5-10 minutes for a control plane node to start, dns-controller to launch, and DNS to propagate. The protokube container and dns-controller deployment logs may contain more diagnostic information. Etcd and the API DNS entries must be updated for a kops Kubernetes cluster to start. Validation Failed W1222 17:16:27.579771 5592 validate_cluster.go:232] (will retry): cluster not yet healthy INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-west-2a ControlPlane c5.large 1 1 us-west-2a nodes-us-west-2a Node t3.medium 4 4 us-west-2a NODE STATUS NAME ROLE READY VALIDATION ERRORS KIND NAME MESSAGE dns apiserver Validation Failed The dns-controller Kubernetes deployment has not updated the Kubernetes cluster's API DNS entry to the correct IP address. The API DNS IP address is the placeholder address that kops creates: 203.0.113.123. Please wait about 5-10 minutes for a control plane node to start, dns-controller to launch, and DNS to propagate. The protokube container and dns-controller deployment logs may contain more diagnostic information. Etcd and the API DNS entries must be updated for a kops Kubernetes cluster to start. Validation Failed W1222 17:16:37.617939 5592 validate_cluster.go:232] (will retry): cluster not yet healthy INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-west-2a ControlPlane c5.large 1 1 us-west-2a nodes-us-west-2a Node t3.medium 4 4 us-west-2a NODE STATUS NAME ROLE READY VALIDATION ERRORS KIND NAME MESSAGE dns apiserver Validation Failed The dns-controller Kubernetes deployment has not updated the Kubernetes cluster's API DNS entry to the correct IP address. The API DNS IP address is the placeholder address that kops creates: 203.0.113.123. Please wait about 5-10 minutes for a control plane node to start, dns-controller to launch, and DNS to propagate. The protokube container and dns-controller deployment logs may contain more diagnostic information. Etcd and the API DNS entries must be updated for a kops Kubernetes cluster to start. Validation Failed W1222 17:16:47.667956 5592 validate_cluster.go:232] (will retry): cluster not yet healthy INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-west-2a ControlPlane c5.large 1 1 us-west-2a nodes-us-west-2a Node t3.medium 4 4 us-west-2a NODE STATUS NAME ROLE READY VALIDATION ERRORS KIND NAME MESSAGE dns apiserver Validation Failed The dns-controller Kubernetes deployment has not updated the Kubernetes cluster's API DNS entry to the correct IP address. The API DNS IP address is the placeholder address that kops creates: 203.0.113.123. Please wait about 5-10 minutes for a control plane node to start, dns-controller to launch, and DNS to propagate. The protokube container and dns-controller deployment logs may contain more diagnostic information. Etcd and the API DNS entries must be updated for a kops Kubernetes cluster to start. Validation Failed W1222 17:16:57.706224 5592 validate_cluster.go:232] (will retry): cluster not yet healthy INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-west-2a ControlPlane c5.large 1 1 us-west-2a nodes-us-west-2a Node t3.medium 4 4 us-west-2a NODE STATUS NAME ROLE READY VALIDATION ERRORS KIND NAME MESSAGE dns apiserver Validation Failed The dns-controller Kubernetes deployment has not updated the Kubernetes cluster's API DNS entry to the correct IP address. The API DNS IP address is the placeholder address that kops creates: 203.0.113.123. Please wait about 5-10 minutes for a control plane node to start, dns-controller to launch, and DNS to propagate. The protokube container and dns-controller deployment logs may contain more diagnostic information. Etcd and the API DNS entries must be updated for a kops Kubernetes cluster to start. Validation Failed W1222 17:17:07.757410 5592 validate_cluster.go:232] (will retry): cluster not yet healthy INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-west-2a ControlPlane c5.large 1 1 us-west-2a nodes-us-west-2a Node t3.medium 4 4 us-west-2a NODE STATUS NAME ROLE READY VALIDATION ERRORS KIND NAME MESSAGE dns apiserver Validation Failed The dns-controller Kubernetes deployment has not updated the Kubernetes cluster's API DNS entry to the correct IP address. The API DNS IP address is the placeholder address that kops creates: 203.0.113.123. Please wait about 5-10 minutes for a control plane node to start, dns-controller to launch, and DNS to propagate. The protokube container and dns-controller deployment logs may contain more diagnostic information. Etcd and the API DNS entries must be updated for a kops Kubernetes cluster to start. Validation Failed W1222 17:17:17.793377 5592 validate_cluster.go:232] (will retry): cluster not yet healthy INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-west-2a ControlPlane c5.large 1 1 us-west-2a nodes-us-west-2a Node t3.medium 4 4 us-west-2a NODE STATUS NAME ROLE READY VALIDATION ERRORS KIND NAME MESSAGE dns apiserver Validation Failed The dns-controller Kubernetes deployment has not updated the Kubernetes cluster's API DNS entry to the correct IP address. The API DNS IP address is the placeholder address that kops creates: 203.0.113.123. Please wait about 5-10 minutes for a control plane node to start, dns-controller to launch, and DNS to propagate. The protokube container and dns-controller deployment logs may contain more diagnostic information. Etcd and the API DNS entries must be updated for a kops Kubernetes cluster to start. Validation Failed W1222 17:17:27.844690 5592 validate_cluster.go:232] (will retry): cluster not yet healthy INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-west-2a ControlPlane c5.large 1 1 us-west-2a nodes-us-west-2a Node t3.medium 4 4 us-west-2a NODE STATUS NAME ROLE READY VALIDATION ERRORS KIND NAME MESSAGE dns apiserver Validation Failed The dns-controller Kubernetes deployment has not updated the Kubernetes cluster's API DNS entry to the correct IP address. The API DNS IP address is the placeholder address that kops creates: 203.0.113.123. Please wait about 5-10 minutes for a control plane node to start, dns-controller to launch, and DNS to propagate. The protokube container and dns-controller deployment logs may contain more diagnostic information. Etcd and the API DNS entries must be updated for a kops Kubernetes cluster to start. Validation Failed W1222 17:17:37.886614 5592 validate_cluster.go:232] (will retry): cluster not yet healthy INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-west-2a ControlPlane c5.large 1 1 us-west-2a nodes-us-west-2a Node t3.medium 4 4 us-west-2a NODE STATUS NAME ROLE READY VALIDATION ERRORS KIND NAME MESSAGE dns apiserver Validation Failed The dns-controller Kubernetes deployment has not updated the Kubernetes cluster's API DNS entry to the correct IP address. The API DNS IP address is the placeholder address that kops creates: 203.0.113.123. Please wait about 5-10 minutes for a control plane node to start, dns-controller to launch, and DNS to propagate. The protokube container and dns-controller deployment logs may contain more diagnostic information. Etcd and the API DNS entries must be updated for a kops Kubernetes cluster to start. Validation Failed W1222 17:17:47.930138 5592 validate_cluster.go:232] (will retry): cluster not yet healthy INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-west-2a ControlPlane c5.large 1 1 us-west-2a nodes-us-west-2a Node t3.medium 4 4 us-west-2a ... skipping 20 lines ... Pod kube-system/ebs-csi-node-95t6m system-node-critical pod "ebs-csi-node-95t6m" is pending Pod kube-system/ebs-csi-node-9r92r system-node-critical pod "ebs-csi-node-9r92r" is pending Pod kube-system/ebs-csi-node-9zq79 system-node-critical pod "ebs-csi-node-9zq79" is pending Pod kube-system/ebs-csi-node-f9cc5 system-node-critical pod "ebs-csi-node-f9cc5" is pending Pod kube-system/ebs-csi-node-krz2c system-node-critical pod "ebs-csi-node-krz2c" is pending Validation Failed W1222 17:17:59.964431 5592 validate_cluster.go:232] (will retry): cluster not yet healthy INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-west-2a ControlPlane c5.large 1 1 us-west-2a nodes-us-west-2a Node t3.medium 4 4 us-west-2a ... skipping 19 lines ... Pod kube-system/ebs-csi-node-95t6m system-node-critical pod "ebs-csi-node-95t6m" is pending Pod kube-system/ebs-csi-node-9r92r system-node-critical pod "ebs-csi-node-9r92r" is pending Pod kube-system/ebs-csi-node-9zq79 system-node-critical pod "ebs-csi-node-9zq79" is pending Pod kube-system/ebs-csi-node-f9cc5 system-node-critical pod "ebs-csi-node-f9cc5" is pending Pod kube-system/ebs-csi-node-krz2c system-node-critical pod "ebs-csi-node-krz2c" is pending Validation Failed W1222 17:18:11.554513 5592 validate_cluster.go:232] (will retry): cluster not yet healthy INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-west-2a ControlPlane c5.large 1 1 us-west-2a nodes-us-west-2a Node t3.medium 4 4 us-west-2a ... skipping 15 lines ... Pod kube-system/ebs-csi-node-95t6m system-node-critical pod "ebs-csi-node-95t6m" is pending Pod kube-system/ebs-csi-node-9r92r system-node-critical pod "ebs-csi-node-9r92r" is pending Pod kube-system/ebs-csi-node-9zq79 system-node-critical pod "ebs-csi-node-9zq79" is pending Pod kube-system/ebs-csi-node-f9cc5 system-node-critical pod "ebs-csi-node-f9cc5" is pending Pod kube-system/ebs-csi-node-krz2c system-node-critical pod "ebs-csi-node-krz2c" is pending Validation Failed W1222 17:18:23.178887 5592 validate_cluster.go:232] (will retry): cluster not yet healthy INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-west-2a ControlPlane c5.large 1 1 us-west-2a nodes-us-west-2a Node t3.medium 4 4 us-west-2a ... skipping 11 lines ... Pod kube-system/coredns-559769c974-6xn9f system-cluster-critical pod "coredns-559769c974-6xn9f" is pending Pod kube-system/coredns-autoscaler-7cb5c5b969-fm27z system-cluster-critical pod "coredns-autoscaler-7cb5c5b969-fm27z" is pending Pod kube-system/ebs-csi-controller-86d44d7948-t9g2t system-cluster-critical pod "ebs-csi-controller-86d44d7948-t9g2t" is not ready (ebs-plugin) Pod kube-system/ebs-csi-node-9r92r system-node-critical pod "ebs-csi-node-9r92r" is pending Pod kube-system/ebs-csi-node-f9cc5 system-node-critical pod "ebs-csi-node-f9cc5" is pending Validation Failed W1222 17:18:34.817154 5592 validate_cluster.go:232] (will retry): cluster not yet healthy INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-west-2a ControlPlane c5.large 1 1 us-west-2a nodes-us-west-2a Node t3.medium 4 4 us-west-2a ... skipping 10 lines ... Node i-076110ae730e268e3 node "i-076110ae730e268e3" of role "node" is not ready Pod kube-system/cilium-nrnwh system-node-critical pod "cilium-nrnwh" is not ready (cilium-agent) Pod kube-system/coredns-559769c974-6xn9f system-cluster-critical pod "coredns-559769c974-6xn9f" is pending Pod kube-system/coredns-autoscaler-7cb5c5b969-fm27z system-cluster-critical pod "coredns-autoscaler-7cb5c5b969-fm27z" is pending Pod kube-system/ebs-csi-node-9r92r system-node-critical pod "ebs-csi-node-9r92r" is pending Validation Failed W1222 17:18:46.324892 5592 validate_cluster.go:232] (will retry): cluster not yet healthy INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-west-2a ControlPlane c5.large 1 1 us-west-2a nodes-us-west-2a Node t3.medium 4 4 us-west-2a ... skipping 10 lines ... Node i-076110ae730e268e3 node "i-076110ae730e268e3" of role "node" is not ready Pod kube-system/cilium-nrnwh system-node-critical pod "cilium-nrnwh" is not ready (cilium-agent) Pod kube-system/coredns-559769c974-6xn9f system-cluster-critical pod "coredns-559769c974-6xn9f" is pending Pod kube-system/coredns-autoscaler-7cb5c5b969-fm27z system-cluster-critical pod "coredns-autoscaler-7cb5c5b969-fm27z" is pending Pod kube-system/ebs-csi-node-9r92r system-node-critical pod "ebs-csi-node-9r92r" is pending Validation Failed W1222 17:18:57.810838 5592 validate_cluster.go:232] (will retry): cluster not yet healthy INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-west-2a ControlPlane c5.large 1 1 us-west-2a nodes-us-west-2a Node t3.medium 4 4 us-west-2a ... skipping 8 lines ... VALIDATION ERRORS KIND NAME MESSAGE Pod kube-system/coredns-559769c974-6xn9f system-cluster-critical pod "coredns-559769c974-6xn9f" is pending Pod kube-system/coredns-autoscaler-7cb5c5b969-fm27z system-cluster-critical pod "coredns-autoscaler-7cb5c5b969-fm27z" is pending Pod kube-system/ebs-csi-node-9r92r system-node-critical pod "ebs-csi-node-9r92r" is pending Validation Failed W1222 17:19:09.185215 5592 validate_cluster.go:232] (will retry): cluster not yet healthy INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-west-2a ControlPlane c5.large 1 1 us-west-2a nodes-us-west-2a Node t3.medium 4 4 us-west-2a ... skipping 263 lines ... [sig-storage] In-tree Volumes [38;5;243mtest/e2e/storage/utils/framework.go:23[0m [Driver: hostPathSymlink] [38;5;243mtest/e2e/storage/in_tree_volumes.go:85[0m [38;5;14m[1m[Testpattern: Dynamic PV (delayed binding)] topology [BeforeEach][0m [38;5;243mtest/e2e/storage/framework/testsuite.go:51[0m should fail to schedule a pod which has topologies that conflict with AllowedTopologies [38;5;243mtest/e2e/storage/testsuites/topology.go:191[0m [38;5;243mBegin Captured GinkgoWriter Output >>[0m [BeforeEach] [Testpattern: Dynamic PV (delayed binding)] topology test/e2e/storage/framework/testsuite.go:51 Dec 22 17:21:30.354: INFO: Driver hostPathSymlink doesn't support DynamicPV -- skipping ... skipping 1165 lines ... [sig-storage] In-tree Volumes [38;5;243mtest/e2e/storage/utils/framework.go:23[0m [Driver: azure-disk] [38;5;243mtest/e2e/storage/in_tree_volumes.go:85[0m [38;5;14m[1m[Testpattern: Dynamic PV (delayed binding)] topology [BeforeEach][0m [38;5;243mtest/e2e/storage/framework/testsuite.go:51[0m should fail to schedule a pod which has topologies that conflict with AllowedTopologies [38;5;243mtest/e2e/storage/testsuites/topology.go:191[0m [38;5;243mBegin Captured GinkgoWriter Output >>[0m [BeforeEach] [Testpattern: Dynamic PV (delayed binding)] topology test/e2e/storage/framework/testsuite.go:51 Dec 22 17:21:38.319: INFO: Only supported for providers [azure] (not aws) ... skipping 87 lines ... [38;5;243m------------------------------[0m [38;5;10m• [SLOW TEST] [8.043 seconds][0m [0m[sig-node] Secrets [38;5;243mshould be consumable via the environment [NodeConformance] [Conformance][0m [38;5;243mtest/e2e/common/node/secrets.go:95[0m [38;5;243m------------------------------[0m [38;5;10m• [SLOW TEST] [9.570 seconds][0m [0m[sig-api-machinery] AdmissionWebhook [Privileged:ClusterAdmin] [38;5;243mshould unconditionally reject operations on fail closed webhook [Conformance][0m [38;5;243mtest/e2e/apimachinery/webhook.go:239[0m [38;5;243m------------------------------[0m [38;5;14mS[0m[38;5;14mS[0m [38;5;243m------------------------------[0m [38;5;10m• [SLOW TEST] [10.184 seconds][0m [0m[sig-node] Ephemeral Containers [NodeConformance] [38;5;243mwill start an ephemeral container in an existing pod [Conformance][0m ... skipping 407 lines ... [0m[sig-storage] In-tree Volumes [38;5;243m[Driver: hostPath] [0m[Testpattern: Inline-volume (default fs)] subPath [38;5;243mshould support readOnly file specified in the volumeMount [LinuxOnly][0m [38;5;243mtest/e2e/storage/testsuites/subpath.go:380[0m [38;5;243m------------------------------[0m [38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m [38;5;243m------------------------------[0m [38;5;10m• [SLOW TEST] [16.770 seconds][0m [0m[sig-apps] Job [38;5;243mshould run a job to completion when tasks sometimes fail and are locally restarted [Conformance][0m [38;5;243mtest/e2e/apps/job.go:426[0m [38;5;243m------------------------------[0m [38;5;14mS [SKIPPED] [0.000 seconds][0m [sig-storage] In-tree Volumes [38;5;243mtest/e2e/storage/utils/framework.go:23[0m [Driver: azure-file] ... skipping 306 lines ... Dec 22 17:21:56.812: INFO: The phase of Pod kube-proxy-mode-detector is Running (Ready = true) Dec 22 17:21:56.812: INFO: Pod "kube-proxy-mode-detector" satisfied condition "running and ready" Dec 22 17:21:56.878: INFO: Running '/home/prow/go/src/k8s.io/kops/_rundir/c6ac28b3-821b-11ed-b47a-e2b55b2f9198/kubectl --server=https://api.e2e-e2e-kops-grid-cilium-eni-amzn2-k26.test-cncf-aws.k8s.io --kubeconfig=/root/.kube/config --namespace=sctp-6876 exec kube-proxy-mode-detector -- /bin/sh -x -c curl -q -s --connect-timeout 1 http://localhost:10249/proxyMode' Dec 22 17:21:57.635: INFO: rc: 7 Dec 22 17:21:57.707: INFO: Waiting for pod kube-proxy-mode-detector to disappear Dec 22 17:21:57.773: INFO: Pod kube-proxy-mode-detector no longer exists Dec 22 17:21:57.773: INFO: Couldn't detect KubeProxy mode - skip, error running /home/prow/go/src/k8s.io/kops/_rundir/c6ac28b3-821b-11ed-b47a-e2b55b2f9198/kubectl --server=https://api.e2e-e2e-kops-grid-cilium-eni-amzn2-k26.test-cncf-aws.k8s.io --kubeconfig=/root/.kube/config --namespace=sctp-6876 exec kube-proxy-mode-detector -- /bin/sh -x -c curl -q -s --connect-timeout 1 http://localhost:10249/proxyMode: Command stdout: stderr: + curl -q -s --connect-timeout 1 http://localhost:10249/proxyMode command terminated with exit code 7 error: exit status 7 [AfterEach] [sig-network] SCTP [LinuxOnly] test/e2e/framework/node/init/init.go:32 Dec 22 17:21:57.773: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready [DeferCleanup (Each)] [sig-network] SCTP [LinuxOnly] test/e2e/framework/metrics/init/init.go:33 [DeferCleanup (Each)] [sig-network] SCTP [LinuxOnly] dump namespaces | framework.go:196 [DeferCleanup (Each)] [sig-network] SCTP [LinuxOnly] tear down framework | framework.go:193 [1mSTEP:[0m Destroying namespace "sctp-6876" for this suite. [38;5;243m12/22/22 17:21:57.84[0m [38;5;243m<< End Captured GinkgoWriter Output[0m [38;5;14mCouldn't detect KubeProxy mode - skip, error running /home/prow/go/src/k8s.io/kops/_rundir/c6ac28b3-821b-11ed-b47a-e2b55b2f9198/kubectl --server=https://api.e2e-e2e-kops-grid-cilium-eni-amzn2-k26.test-cncf-aws.k8s.io --kubeconfig=/root/.kube/config --namespace=sctp-6876 exec kube-proxy-mode-detector -- /bin/sh -x -c curl -q -s --connect-timeout 1 http://localhost:10249/proxyMode: Command stdout: stderr: + curl -q -s --connect-timeout 1 http://localhost:10249/proxyMode command terminated with exit code 7 error: exit status 7[0m [38;5;14mIn [1m[It][0m[38;5;14m at: [1mtest/e2e/network/service.go:4454[0m [38;5;243m------------------------------[0m [38;5;10m• [SLOW TEST] [17.190 seconds][0m [0m[sig-apps] DisruptionController [38;5;243mevictions: maxUnavailable allow single eviction, percentage => should allow an eviction[0m [38;5;243mtest/e2e/apps/disruption.go:290[0m ... skipping 962 lines ... [sig-storage] In-tree Volumes [38;5;243mtest/e2e/storage/utils/framework.go:23[0m [Driver: local][LocalVolumeType: block] [38;5;243mtest/e2e/storage/in_tree_volumes.go:85[0m [38;5;14m[1m[Testpattern: Dynamic PV (delayed binding)] topology [BeforeEach][0m [38;5;243mtest/e2e/storage/framework/testsuite.go:51[0m should fail to schedule a pod which has topologies that conflict with AllowedTopologies [38;5;243mtest/e2e/storage/testsuites/topology.go:191[0m [38;5;243mBegin Captured GinkgoWriter Output >>[0m [BeforeEach] [Testpattern: Dynamic PV (delayed binding)] topology test/e2e/storage/framework/testsuite.go:51 Dec 22 17:22:27.826: INFO: Driver local doesn't support DynamicPV -- skipping ... skipping 755 lines ... [sig-storage] In-tree Volumes [38;5;243mtest/e2e/storage/utils/framework.go:23[0m [Driver: azure-file] [38;5;243mtest/e2e/storage/in_tree_volumes.go:85[0m [38;5;14m[1m[Testpattern: Dynamic PV (immediate binding)] topology [BeforeEach][0m [38;5;243mtest/e2e/storage/framework/testsuite.go:51[0m should fail to schedule a pod which has topologies that conflict with AllowedTopologies [38;5;243mtest/e2e/storage/testsuites/topology.go:191[0m [38;5;243mBegin Captured GinkgoWriter Output >>[0m [BeforeEach] [Testpattern: Dynamic PV (immediate binding)] topology test/e2e/storage/framework/testsuite.go:51 Dec 22 17:22:47.176: INFO: Only supported for providers [azure] (not aws) ... skipping 148 lines ... [sig-storage] CSI Volumes [38;5;243mtest/e2e/storage/utils/framework.go:23[0m [Driver: csi-hostpath] [38;5;243mtest/e2e/storage/csi_volumes.go:40[0m [38;5;14m[1m[Testpattern: Dynamic PV (immediate binding)] topology [BeforeEach][0m [38;5;243mtest/e2e/storage/framework/testsuite.go:51[0m should fail to schedule a pod which has topologies that conflict with AllowedTopologies [38;5;243mtest/e2e/storage/testsuites/topology.go:191[0m [38;5;243mBegin Captured GinkgoWriter Output >>[0m [BeforeEach] [Testpattern: Dynamic PV (immediate binding)] topology test/e2e/storage/framework/testsuite.go:51 Dec 22 17:22:47.796: INFO: Driver "csi-hostpath" does not support topology - skipping ... skipping 2539 lines ... [38;5;14mOnly supported for providers [azure] (not aws)[0m [38;5;14mIn [1m[BeforeEach][0m[38;5;14m at: [1mtest/e2e/storage/drivers/in_tree.go:1890[0m [38;5;243m------------------------------[0m [38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m [38;5;243m------------------------------[0m [38;5;10m• [SLOW TEST] [6.606 seconds][0m [0m[sig-storage] PersistentVolumes-local [38;5;243mPod with node different from PV's NodeAffinity [0mshould fail scheduling due to different NodeSelector[0m [38;5;243mtest/e2e/storage/persistent_volumes-local.go:382[0m [38;5;243m------------------------------[0m [38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;10m•[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m [38;5;243m------------------------------[0m [38;5;14mS [SKIPPED] [0.000 seconds][0m [sig-storage] In-tree Volumes ... skipping 1438 lines ... [38;5;243m<< End Captured GinkgoWriter Output[0m [38;5;14mDriver hostPath doesn't support GenericEphemeralVolume -- skipping[0m [38;5;14mIn [1m[BeforeEach][0m[38;5;14m at: [1mtest/e2e/storage/framework/testsuite.go:116[0m [38;5;243m------------------------------[0m [38;5;10m• [SLOW TEST] [8.792 seconds][0m [0m[sig-apps] Job [38;5;243mshould fail to exceed backoffLimit[0m [38;5;243mtest/e2e/apps/job.go:561[0m [38;5;243m------------------------------[0m [38;5;14mS [SKIPPED] [0.000 seconds][0m [sig-storage] In-tree Volumes [38;5;243mtest/e2e/storage/utils/framework.go:23[0m [Driver: hostPath] ... skipping 1816 lines ... [38;5;14mOnly supported for providers [azure] (not aws)[0m [38;5;14mIn [1m[BeforeEach][0m[38;5;14m at: [1mtest/e2e/storage/drivers/in_tree.go:1375[0m [38;5;243m------------------------------[0m [38;5;14mS[0m [38;5;243m------------------------------[0m [38;5;9m• [FAILED] [307.678 seconds][0m [38;5;9m[1m[sig-api-machinery] AdmissionWebhook [Privileged:ClusterAdmin] [BeforeEach][0m [38;5;243mtest/e2e/apimachinery/webhook.go:90[0m patching/updating a validating webhook should work [Conformance] [38;5;243mtest/e2e/apimachinery/webhook.go:413[0m [38;5;243mBegin Captured GinkgoWriter Output >>[0m ... skipping 162 lines ... Dec 22 17:26:24.374: INFO: deployment status: v1.DeploymentStatus{ObservedGeneration:1, Replicas:1, UpdatedReplicas:1, ReadyReplicas:0, AvailableReplicas:0, UnavailableReplicas:1, Conditions:[]v1.DeploymentCondition{v1.DeploymentCondition{Type:"Available", Status:"False", LastUpdateTime:time.Date(2022, time.December, 22, 17, 21, 31, 0, time.Local), LastTransitionTime:time.Date(2022, time.December, 22, 17, 21, 31, 0, time.Local), Reason:"MinimumReplicasUnavailable", Message:"Deployment does not have minimum availability."}, v1.DeploymentCondition{Type:"Progressing", Status:"True", LastUpdateTime:time.Date(2022, time.December, 22, 17, 21, 32, 0, time.Local), LastTransitionTime:time.Date(2022, time.December, 22, 17, 21, 31, 0, time.Local), Reason:"ReplicaSetUpdated", Message:"ReplicaSet \"sample-webhook-deployment-865554f4d9\" is progressing."}}, CollisionCount:(*int32)(nil)} Dec 22 17:26:26.374: INFO: deployment status: v1.DeploymentStatus{ObservedGeneration:1, Replicas:1, UpdatedReplicas:1, ReadyReplicas:0, AvailableReplicas:0, UnavailableReplicas:1, Conditions:[]v1.DeploymentCondition{v1.DeploymentCondition{Type:"Available", Status:"False", LastUpdateTime:time.Date(2022, time.December, 22, 17, 21, 31, 0, time.Local), LastTransitionTime:time.Date(2022, time.December, 22, 17, 21, 31, 0, time.Local), Reason:"MinimumReplicasUnavailable", Message:"Deployment does not have minimum availability."}, v1.DeploymentCondition{Type:"Progressing", Status:"True", LastUpdateTime:time.Date(2022, time.December, 22, 17, 21, 32, 0, time.Local), LastTransitionTime:time.Date(2022, time.December, 22, 17, 21, 31, 0, time.Local), Reason:"ReplicaSetUpdated", Message:"ReplicaSet \"sample-webhook-deployment-865554f4d9\" is progressing."}}, CollisionCount:(*int32)(nil)} Dec 22 17:26:28.375: INFO: deployment status: v1.DeploymentStatus{ObservedGeneration:1, Replicas:1, UpdatedReplicas:1, ReadyReplicas:0, AvailableReplicas:0, UnavailableReplicas:1, Conditions:[]v1.DeploymentCondition{v1.DeploymentCondition{Type:"Available", Status:"False", LastUpdateTime:time.Date(2022, time.December, 22, 17, 21, 31, 0, time.Local), LastTransitionTime:time.Date(2022, time.December, 22, 17, 21, 31, 0, time.Local), Reason:"MinimumReplicasUnavailable", Message:"Deployment does not have minimum availability."}, v1.DeploymentCondition{Type:"Progressing", Status:"True", LastUpdateTime:time.Date(2022, time.December, 22, 17, 21, 32, 0, time.Local), LastTransitionTime:time.Date(2022, time.December, 22, 17, 21, 31, 0, time.Local), Reason:"ReplicaSetUpdated", Message:"ReplicaSet \"sample-webhook-deployment-865554f4d9\" is progressing."}}, CollisionCount:(*int32)(nil)} Dec 22 17:26:30.385: INFO: deployment status: v1.DeploymentStatus{ObservedGeneration:1, Replicas:1, UpdatedReplicas:1, ReadyReplicas:0, AvailableReplicas:0, UnavailableReplicas:1, Conditions:[]v1.DeploymentCondition{v1.DeploymentCondition{Type:"Available", Status:"False", LastUpdateTime:time.Date(2022, time.December, 22, 17, 21, 31, 0, time.Local), LastTransitionTime:time.Date(2022, time.December, 22, 17, 21, 31, 0, time.Local), Reason:"MinimumReplicasUnavailable", Message:"Deployment does not have minimum availability."}, v1.DeploymentCondition{Type:"Progressing", Status:"True", LastUpdateTime:time.Date(2022, time.December, 22, 17, 21, 32, 0, time.Local), LastTransitionTime:time.Date(2022, time.December, 22, 17, 21, 31, 0, time.Local), Reason:"ReplicaSetUpdated", Message:"ReplicaSet \"sample-webhook-deployment-865554f4d9\" is progressing."}}, CollisionCount:(*int32)(nil)} Dec 22 17:26:32.375: INFO: deployment status: v1.DeploymentStatus{ObservedGeneration:1, Replicas:1, UpdatedReplicas:1, ReadyReplicas:0, AvailableReplicas:0, UnavailableReplicas:1, Conditions:[]v1.DeploymentCondition{v1.DeploymentCondition{Type:"Available", Status:"False", LastUpdateTime:time.Date(2022, time.December, 22, 17, 21, 31, 0, time.Local), LastTransitionTime:time.Date(2022, time.December, 22, 17, 21, 31, 0, time.Local), Reason:"MinimumReplicasUnavailable", Message:"Deployment does not have minimum availability."}, v1.DeploymentCondition{Type:"Progressing", Status:"True", LastUpdateTime:time.Date(2022, time.December, 22, 17, 21, 32, 0, time.Local), LastTransitionTime:time.Date(2022, time.December, 22, 17, 21, 31, 0, time.Local), Reason:"ReplicaSetUpdated", Message:"ReplicaSet \"sample-webhook-deployment-865554f4d9\" is progressing."}}, CollisionCount:(*int32)(nil)} Dec 22 17:26:32.438: INFO: deployment status: v1.DeploymentStatus{ObservedGeneration:1, Replicas:1, UpdatedReplicas:1, ReadyReplicas:0, AvailableReplicas:0, UnavailableReplicas:1, Conditions:[]v1.DeploymentCondition{v1.DeploymentCondition{Type:"Available", Status:"False", LastUpdateTime:time.Date(2022, time.December, 22, 17, 21, 31, 0, time.Local), LastTransitionTime:time.Date(2022, time.December, 22, 17, 21, 31, 0, time.Local), Reason:"MinimumReplicasUnavailable", Message:"Deployment does not have minimum availability."}, v1.DeploymentCondition{Type:"Progressing", Status:"True", LastUpdateTime:time.Date(2022, time.December, 22, 17, 21, 32, 0, time.Local), LastTransitionTime:time.Date(2022, time.December, 22, 17, 21, 31, 0, time.Local), Reason:"ReplicaSetUpdated", Message:"ReplicaSet \"sample-webhook-deployment-865554f4d9\" is progressing."}}, CollisionCount:(*int32)(nil)} Dec 22 17:26:32.438: INFO: Unexpected error: waiting for the deployment status valid%!(EXTRA string=registry.k8s.io/e2e-test-images/agnhost:2.43, string=sample-webhook-deployment, string=webhook-2208): <*errors.errorString | 0xc000e5d950>: { s: "error waiting for deployment \"sample-webhook-deployment\" status to match expectation: deployment status: v1.DeploymentStatus{ObservedGeneration:1, Replicas:1, UpdatedReplicas:1, ReadyReplicas:0, AvailableReplicas:0, UnavailableReplicas:1, Conditions:[]v1.DeploymentCondition{v1.DeploymentCondition{Type:\"Available\", Status:\"False\", LastUpdateTime:time.Date(2022, time.December, 22, 17, 21, 31, 0, time.Local), LastTransitionTime:time.Date(2022, time.December, 22, 17, 21, 31, 0, time.Local), Reason:\"MinimumReplicasUnavailable\", Message:\"Deployment does not have minimum availability.\"}, v1.DeploymentCondition{Type:\"Progressing\", Status:\"True\", LastUpdateTime:time.Date(2022, time.December, 22, 17, 21, 32, 0, time.Local), LastTransitionTime:time.Date(2022, time.December, 22, 17, 21, 31, 0, time.Local), Reason:\"ReplicaSetUpdated\", Message:\"ReplicaSet \\\"sample-webhook-deployment-865554f4d9\\\" is progressing.\"}}, CollisionCount:(*int32)(nil)}", } Dec 22 17:26:32.438: FAIL: waiting for the deployment status valid%!(EXTRA string=registry.k8s.io/e2e-test-images/agnhost:2.43, string=sample-webhook-deployment, string=webhook-2208): error waiting for deployment "sample-webhook-deployment" status to match expectation: deployment status: v1.DeploymentStatus{ObservedGeneration:1, Replicas:1, UpdatedReplicas:1, ReadyReplicas:0, AvailableReplicas:0, UnavailableReplicas:1, Conditions:[]v1.DeploymentCondition{v1.DeploymentCondition{Type:"Available", Status:"False", LastUpdateTime:time.Date(2022, time.December, 22, 17, 21, 31, 0, time.Local), LastTransitionTime:time.Date(2022, time.December, 22, 17, 21, 31, 0, time.Local), Reason:"MinimumReplicasUnavailable", Message:"Deployment does not have minimum availability."}, v1.DeploymentCondition{Type:"Progressing", Status:"True", LastUpdateTime:time.Date(2022, time.December, 22, 17, 21, 32, 0, time.Local), LastTransitionTime:time.Date(2022, time.December, 22, 17, 21, 31, 0, time.Local), Reason:"ReplicaSetUpdated", Message:"ReplicaSet \"sample-webhook-deployment-865554f4d9\" is progressing."}}, CollisionCount:(*int32)(nil)} Full Stack Trace k8s.io/kubernetes/test/e2e/apimachinery.deployWebhookAndService(0xc000eb1ef0, {0xc002c83860, 0x2c}, 0xc0040359f0, 0x20fb, 0x20fc) test/e2e/apimachinery/webhook.go:826 +0xed2 k8s.io/kubernetes/test/e2e/apimachinery.glob..func28.1() test/e2e/apimachinery/webhook.go:102 +0x226 ... skipping 13 lines ... Dec 22 17:26:32.882: INFO: At 2022-12-22 17:21:31 +0000 UTC - event for sample-webhook-deployment-865554f4d9: {replicaset-controller } SuccessfulCreate: Created pod: sample-webhook-deployment-865554f4d9-72w82 Dec 22 17:26:32.882: INFO: At 2022-12-22 17:21:31 +0000 UTC - event for sample-webhook-deployment-865554f4d9-72w82: {default-scheduler } Scheduled: Successfully assigned webhook-2208/sample-webhook-deployment-865554f4d9-72w82 to i-076110ae730e268e3 Dec 22 17:26:32.882: INFO: At 2022-12-22 17:21:33 +0000 UTC - event for sample-webhook-deployment-865554f4d9-72w82: {kubelet i-076110ae730e268e3} Pulling: Pulling image "registry.k8s.io/e2e-test-images/agnhost:2.43" Dec 22 17:26:32.882: INFO: At 2022-12-22 17:21:35 +0000 UTC - event for sample-webhook-deployment-865554f4d9-72w82: {kubelet i-076110ae730e268e3} Pulled: Successfully pulled image "registry.k8s.io/e2e-test-images/agnhost:2.43" in 181.609525ms (1.60650627s including waiting) Dec 22 17:26:32.882: INFO: At 2022-12-22 17:21:35 +0000 UTC - event for sample-webhook-deployment-865554f4d9-72w82: {kubelet i-076110ae730e268e3} Created: Created container sample-webhook Dec 22 17:26:32.882: INFO: At 2022-12-22 17:21:35 +0000 UTC - event for sample-webhook-deployment-865554f4d9-72w82: {kubelet i-076110ae730e268e3} Started: Started container sample-webhook Dec 22 17:26:32.882: INFO: At 2022-12-22 17:21:36 +0000 UTC - event for sample-webhook-deployment-865554f4d9-72w82: {kubelet i-076110ae730e268e3} Unhealthy: Readiness probe failed: Get "https://172.20.46.200:8444/readyz": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) Dec 22 17:26:32.882: INFO: At 2022-12-22 17:21:40 +0000 UTC - event for sample-webhook-deployment-865554f4d9-72w82: {kubelet i-076110ae730e268e3} Unhealthy: Readiness probe failed: Get "https://172.20.46.200:8444/readyz": context deadline exceeded Dec 22 17:26:32.945: INFO: POD NODE PHASE GRACE CONDITIONS Dec 22 17:26:32.945: INFO: Dec 22 17:26:33.009: INFO: Logging node info for node i-0243def68c064e8ef Dec 22 17:26:33.072: INFO: Node Info: &Node{ObjectMeta:{i-0243def68c064e8ef f07b8771-29eb-4944-a385-3482a3d3a315 15307 0 2022-12-22 17:17:47 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:us-west-2 failure-domain.beta.kubernetes.io/zone:us-west-2a kubernetes.io/arch:amd64 kubernetes.io/hostname:i-0243def68c064e8ef kubernetes.io/os:linux node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:us-west-2a topology.hostpath.csi/node:i-0243def68c064e8ef topology.kubernetes.io/region:us-west-2 topology.kubernetes.io/zone:us-west-2a] map[alpha.kubernetes.io/provided-node-ip:172.20.35.142 csi.volume.kubernetes.io/nodeid:{"csi-hostpath-ephemeral-547":"i-0243def68c064e8ef","csi-mock-csi-mock-volumes-8885":"i-0243def68c064e8ef","ebs.csi.aws.com":"i-0243def68c064e8ef"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{aws-cloud-controller-manager Update v1 2022-12-22 17:17:47 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2022-12-22 17:17:47 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kops-controller Update v1 2022-12-22 17:17:47 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{}}}} } {kube-controller-manager Update v1 2022-12-22 17:17:47 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.3.0/24\"":{}}}} } {kubelet Update v1 2022-12-22 17:17:47 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:alpha.kubernetes.io/provided-node-ip":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {kubelet Update v1 2022-12-22 17:26:24 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status} {kube-controller-manager Update v1 2022-12-22 17:26:25 +0000 UTC FieldsV1 {"f:status":{"f:volumesAttached":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.3.0/24,DoNotUseExternalID:,ProviderID:aws:///us-west-2a/i-0243def68c064e8ef,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.3.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{51527004160 0} {<nil>} 50319340Ki BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4050636800 0} {<nil>} 3955700Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{46374303668 0} {<nil>} 46374303668 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3945779200 0} {<nil>} 3853300Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-12-22 17:26:18 +0000 UTC,LastTransitionTime:2022-12-22 17:17:37 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-12-22 17:26:18 +0000 UTC,LastTransitionTime:2022-12-22 17:17:37 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-12-22 17:26:18 +0000 UTC,LastTransitionTime:2022-12-22 17:17:37 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-12-22 17:26:18 +0000 UTC,LastTransitionTime:2022-12-22 17:17:59 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.35.142,},NodeAddress{Type:ExternalIP,Address:35.90.235.118,},NodeAddress{Type:InternalDNS,Address:i-0243def68c064e8ef.us-west-2.compute.internal,},NodeAddress{Type:Hostname,Address:i-0243def68c064e8ef.us-west-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-35-90-235-118.us-west-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2c8ae28bb7834ee5f6b4c4ef144a8e,SystemUUID:ec2c8ae2-8bb7-834e-e5f6-b4c4ef144a8e,BootID:5843895c-91be-4c36-b700-9e551cb745de,KernelVersion:5.10.157-139.675.amzn2.x86_64,OSImage:Amazon Linux 2,ContainerRuntimeVersion:containerd://1.6.14,KubeletVersion:v1.26.0,KubeProxyVersion:v1.26.0,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:166719855,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/jessie-dnsutils@sha256:24aaf2626d6b27864c29de2097e8bbb840b3a414271bf7c8995e431e47d8408e registry.k8s.io/e2e-test-images/jessie-dnsutils:1.7],SizeBytes:112030336,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:dd75ec974b0a2a6f6bb47001ba09207976e625db898d1b16735528c009cb171c registry.k8s.io/etcd:3.5.6-0],SizeBytes:102542580,},ContainerImage{Names:[registry.k8s.io/kube-proxy-amd64:v1.26.0],SizeBytes:67205320,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:16bbf38c463a4223d8cfe4da12bc61010b082a79b4bb003e2d3ba3ece5dd5f9e registry.k8s.io/e2e-test-images/agnhost:2.43],SizeBytes:51706353,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nautilus@sha256:80ba6c8c44f9623f06e868a1aa66026c8ec438ad814f9ec95e9333b415fe3550 registry.k8s.io/e2e-test-images/nautilus:1.7],SizeBytes:49641698,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:148b022f5c5da426fc2f3c14b5c0867e58ef05961510c84749ac1fddcb0fef22 registry.k8s.io/e2e-test-images/httpd:2.4.38-4],SizeBytes:40764257,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:10e231cf07c82dfab5141c1afe548dc734e5fc1f67665ec3982a325d1bd31a9a registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.0],SizeBytes:29725756,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:ee3b525d5b89db99da3b8eb521d9cd90cb6e9ef0fbb651e98bb37be78d36b5b8 registry.k8s.io/sig-storage/csi-provisioner:v3.3.0],SizeBytes:25491225,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:425d8f1b769398127767b06ed97ce62578a3179bcb99809ce93a1649e025ffe7 registry.k8s.io/sig-storage/csi-resizer:v1.6.0],SizeBytes:24148884,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:291334908ddf71a4661fd7f6d9d97274de8a5378a2b6fdfeb2ce73414a34f82f registry.k8s.io/sig-storage/csi-snapshotter:v6.1.0],SizeBytes:23881995,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:9a685020911e2725ad019dbce6e4a5ab93d51e3d4557f115e64343345e05781b registry.k8s.io/sig-storage/csi-attacher:v4.0.0],SizeBytes:23847201,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:92257881c1d6493cf18299a24af42330f891166560047902b8d431fb66b01af5 registry.k8s.io/sig-storage/hostpathplugin:v1.9.0],SizeBytes:18758628,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:933940f13b3ea0abc62e656c1aa5c5b47c04b15d71250413a6b821bd0c58b94e registry.k8s.io/sig-storage/livenessprobe:v2.7.0],SizeBytes:8688564,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:5c99cf6a02adda929b10321dbf4ecfa00d87be9ba4fb456006237d530ab4baa1 registry.k8s.io/e2e-test-images/nginx:1.14-4],SizeBytes:6978614,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf registry.k8s.io/e2e-test-images/busybox:1.29-2],SizeBytes:732424,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:2e0f836850e09b8b7cc937681d6194537a09fbd5f6b9e08f4d646a85128e8937 registry.k8s.io/e2e-test-images/busybox:1.29-4],SizeBytes:731990,},ContainerImage{Names:[registry.k8s.io/pause@sha256:7031c1b283388d2c2e09b57badb803c05ebed362dc88d84b480cc47f72a21097 registry.k8s.io/pause:3.9],SizeBytes:321520,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[kubernetes.io/csi/csi-mock-csi-mock-volumes-8885^bc251f85-821d-11ed-b95f-4a799389b639],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/csi-mock-csi-mock-volumes-8885^bc251f85-821d-11ed-b95f-4a799389b639,DevicePath:,},},Config:nil,},} Dec 22 17:26:33.072: INFO: ... skipping 452 lines ... [DeferCleanup (Each)] [sig-api-machinery] AdmissionWebhook [Privileged:ClusterAdmin] tear down framework | framework.go:193 [1mSTEP:[0m Destroying namespace "webhook-2208" for this suite. [38;5;243m12/22/22 17:26:38.054[0m [1mSTEP:[0m Destroying namespace "webhook-2208-markers" for this suite. [38;5;243m12/22/22 17:26:38.118[0m [38;5;243m<< End Captured GinkgoWriter Output[0m [38;5;9mDec 22 17:26:32.438: waiting for the deployment status valid%!(EXTRA string=registry.k8s.io/e2e-test-images/agnhost:2.43, string=sample-webhook-deployment, string=webhook-2208): error waiting for deployment "sample-webhook-deployment" status to match expectation: deployment status: v1.DeploymentStatus{ObservedGeneration:1, Replicas:1, UpdatedReplicas:1, ReadyReplicas:0, AvailableReplicas:0, UnavailableReplicas:1, Conditions:[]v1.DeploymentCondition{v1.DeploymentCondition{Type:"Available", Status:"False", LastUpdateTime:time.Date(2022, time.December, 22, 17, 21, 31, 0, time.Local), LastTransitionTime:time.Date(2022, time.December, 22, 17, 21, 31, 0, time.Local), Reason:"MinimumReplicasUnavailable", Message:"Deployment does not have minimum availability."}, v1.DeploymentCondition{Type:"Progressing", Status:"True", LastUpdateTime:time.Date(2022, time.December, 22, 17, 21, 32, 0, time.Local), LastTransitionTime:time.Date(2022, time.December, 22, 17, 21, 31, 0, time.Local), Reason:"ReplicaSetUpdated", Message:"ReplicaSet \"sample-webhook-deployment-865554f4d9\" is progressing."}}, CollisionCount:(*int32)(nil)}[0m [38;5;9mIn [1m[BeforeEach][0m[38;5;9m at: [1mtest/e2e/apimachinery/webhook.go:826[0m [38;5;243m------------------------------[0m [38;5;14mS[0m [38;5;243m------------------------------[0m [38;5;10m• [SLOW TEST] [26.583 seconds][0m [0m[sig-storage] In-tree Volumes [38;5;243m[Driver: local][LocalVolumeType: tmpfs] [0m[Testpattern: Pre-provisioned PV (default fs)] subPath [38;5;243mshould support existing directories when readOnly specified in the volumeSource[0m ... skipping 216 lines ... [1mSTEP:[0m Waiting for kube-root-ca.crt to be provisioned in namespace [38;5;243m12/22/22 17:26:34.233[0m [BeforeEach] [Testpattern: Inline-volume (default fs)] subPath test/e2e/framework/metrics/init/init.go:31 [It] should support existing directories when readOnly specified in the volumeSource test/e2e/storage/testsuites/subpath.go:395 Dec 22 17:26:34.359: INFO: In-tree plugin kubernetes.io/host-path is not migrated, not validating any metrics Dec 22 17:26:34.495: INFO: Waiting up to 5m0s for pod "hostpath-symlink-prep-provisioning-2596" in namespace "provisioning-2596" to be "Succeeded or Failed" Dec 22 17:26:34.559: INFO: Pod "hostpath-symlink-prep-provisioning-2596": Phase="Pending", Reason="", readiness=false. Elapsed: 64.001008ms Dec 22 17:26:36.624: INFO: Pod "hostpath-symlink-prep-provisioning-2596": Phase="Pending", Reason="", readiness=false. Elapsed: 2.128412331s Dec 22 17:26:38.624: INFO: Pod "hostpath-symlink-prep-provisioning-2596": Phase="Pending", Reason="", readiness=false. Elapsed: 4.128224027s Dec 22 17:26:40.624: INFO: Pod "hostpath-symlink-prep-provisioning-2596": Phase="Pending", Reason="", readiness=false. Elapsed: 6.128496116s Dec 22 17:26:42.624: INFO: Pod "hostpath-symlink-prep-provisioning-2596": Phase="Pending", Reason="", readiness=false. Elapsed: 8.128516139s Dec 22 17:26:44.623: INFO: Pod "hostpath-symlink-prep-provisioning-2596": Phase="Pending", Reason="", readiness=false. Elapsed: 10.12765769s Dec 22 17:26:46.625: INFO: Pod "hostpath-symlink-prep-provisioning-2596": Phase="Pending", Reason="", readiness=false. Elapsed: 12.12915677s Dec 22 17:26:48.624: INFO: Pod "hostpath-symlink-prep-provisioning-2596": Phase="Pending", Reason="", readiness=false. Elapsed: 14.128282692s Dec 22 17:26:50.623: INFO: Pod "hostpath-symlink-prep-provisioning-2596": Phase="Pending", Reason="", readiness=false. Elapsed: 16.128044589s Dec 22 17:26:52.624: INFO: Pod "hostpath-symlink-prep-provisioning-2596": Phase="Succeeded", Reason="", readiness=false. Elapsed: 18.128899094s [1mSTEP:[0m Saw pod success [38;5;243m12/22/22 17:26:52.624[0m Dec 22 17:26:52.624: INFO: Pod "hostpath-symlink-prep-provisioning-2596" satisfied condition "Succeeded or Failed" Dec 22 17:26:52.624: INFO: Deleting pod "hostpath-symlink-prep-provisioning-2596" in namespace "provisioning-2596" Dec 22 17:26:52.696: INFO: Wait up to 5m0s for pod "hostpath-symlink-prep-provisioning-2596" to be fully deleted Dec 22 17:26:52.759: INFO: Creating resource for inline volume Dec 22 17:26:52.759: INFO: Driver hostPathSymlink on volume type InlineVolume doesn't support readOnly source [1mSTEP:[0m Deleting pod [38;5;243m12/22/22 17:26:52.759[0m Dec 22 17:26:52.759: INFO: Deleting pod "pod-subpath-test-inlinevolume-fjmg" in namespace "provisioning-2596" Dec 22 17:26:52.889: INFO: Waiting up to 5m0s for pod "hostpath-symlink-prep-provisioning-2596" in namespace "provisioning-2596" to be "Succeeded or Failed" Dec 22 17:26:52.953: INFO: Pod "hostpath-symlink-prep-provisioning-2596": Phase="Pending", Reason="", readiness=false. Elapsed: 63.561803ms Dec 22 17:26:55.017: INFO: Pod "hostpath-symlink-prep-provisioning-2596": Phase="Pending", Reason="", readiness=false. Elapsed: 2.128183629s Dec 22 17:26:57.040: INFO: Pod "hostpath-symlink-prep-provisioning-2596": Phase="Succeeded", Reason="", readiness=false. Elapsed: 4.151388219s [1mSTEP:[0m Saw pod success [38;5;243m12/22/22 17:26:57.04[0m Dec 22 17:26:57.041: INFO: Pod "hostpath-symlink-prep-provisioning-2596" satisfied condition "Succeeded or Failed" Dec 22 17:26:57.041: INFO: Deleting pod "hostpath-symlink-prep-provisioning-2596" in namespace "provisioning-2596" Dec 22 17:26:57.133: INFO: Wait up to 5m0s for pod "hostpath-symlink-prep-provisioning-2596" to be fully deleted [AfterEach] [Testpattern: Inline-volume (default fs)] subPath test/e2e/framework/node/init/init.go:32 Dec 22 17:26:57.200: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready [DeferCleanup (Each)] [Testpattern: Inline-volume (default fs)] subPath ... skipping 78 lines ... [38;5;14mDriver local doesn't support GenericEphemeralVolume -- skipping[0m [38;5;14mIn [1m[BeforeEach][0m[38;5;14m at: [1mtest/e2e/storage/framework/testsuite.go:116[0m [38;5;243m------------------------------[0m [38;5;14mS[0m [38;5;243m------------------------------[0m [38;5;9m• [FAILED] [58.216 seconds][0m [sig-node] Container Lifecycle Hook [38;5;243mtest/e2e/common/node/framework.go:23[0m when create a pod with lifecycle hook [38;5;243mtest/e2e/common/node/lifecycle_hook.go:46[0m [38;5;9m[1m[It] should execute prestop https hook properly [MinimumKubeletVersion:1.23] [NodeConformance][0m [38;5;243mtest/e2e/common/node/lifecycle_hook.go:234[0m ... skipping 47 lines ... Dec 22 17:26:27.733: INFO: Pod pod-with-prestop-https-hook still exists Dec 22 17:26:29.667: INFO: Waiting for pod pod-with-prestop-https-hook to disappear Dec 22 17:26:29.733: INFO: Pod pod-with-prestop-https-hook still exists Dec 22 17:26:31.668: INFO: Waiting for pod pod-with-prestop-https-hook to disappear Dec 22 17:26:31.734: INFO: Pod pod-with-prestop-https-hook no longer exists [1mSTEP:[0m check prestop hook [38;5;243m12/22/22 17:26:31.734[0m Dec 22 17:27:01.735: FAIL: Timed out after 30.000s. Expected <*errors.errorString | 0xc0013ad8b0>: { s: "failed to match regexp \"GET /echo\\\\?msg=prestop\" in output \"I1222 17:26:08.138316 1 log.go:198] Started HTTP server on port 9090\\nI1222 17:26:08.138993 1 log.go:198] Started UDP server on port 9091\\n\"", } to be nil Full Stack Trace k8s.io/kubernetes/test/e2e/common/node.glob..func12.1.2(0xc000df5680) test/e2e/common/node/lifecycle_hook.go:126 +0x469 ... skipping 242 lines ... [1mSTEP:[0m Destroying namespace "container-lifecycle-hook-2557" for this suite. [38;5;243m12/22/22 17:27:04.628[0m [38;5;243m<< End Captured GinkgoWriter Output[0m [38;5;9mDec 22 17:27:01.735: Timed out after 30.000s. Expected <*errors.errorString | 0xc0013ad8b0>: { s: "failed to match regexp \"GET /echo\\\\?msg=prestop\" in output \"I1222 17:26:08.138316 1 log.go:198] Started HTTP server on port 9090\\nI1222 17:26:08.138993 1 log.go:198] Started UDP server on port 9091\\n\"", } to be nil[0m [38;5;9mIn [1m[It][0m[38;5;9m at: [1mtest/e2e/common/node/lifecycle_hook.go:126[0m [38;5;243m------------------------------[0m [38;5;14mS[0m[38;5;14mS[0m [38;5;243m------------------------------[0m ... skipping 143 lines ... [38;5;14mIn [1m[BeforeEach][0m[38;5;14m at: [1mtest/e2e/storage/drivers/in_tree.go:1890[0m [38;5;243m------------------------------[0m [38;5;10m• [SLOW TEST] [58.994 seconds][0m [0m[sig-node] Sysctls [LinuxOnly] [NodeConformance] [38;5;243mshould support sysctls with slashes as separator [MinimumKubeletVersion:1.23][0m [38;5;243mtest/e2e/common/node/sysctl.go:185[0m [38;5;243m------------------------------[0m [38;5;9m• [FAILED] [304.278 seconds][0m [sig-network] Networking [38;5;243mtest/e2e/network/common/framework.go:23[0m Granular Checks: Services [38;5;243mtest/e2e/network/networking.go:145[0m [38;5;9m[1m[It] should function for endpoint-Service: udp[0m [38;5;243mtest/e2e/network/networking.go:250[0m ... skipping 316 lines ... Dec 22 17:27:15.503: INFO: Pod "netserver-0": Phase="Running", Reason="", readiness=false. Elapsed: 4m58.131078266s Dec 22 17:27:15.503: INFO: The phase of Pod netserver-0 is Running (Ready = false) Dec 22 17:27:17.503: INFO: Pod "netserver-0": Phase="Running", Reason="", readiness=false. Elapsed: 5m0.131623181s Dec 22 17:27:17.504: INFO: The phase of Pod netserver-0 is Running (Ready = false) Dec 22 17:27:17.576: INFO: Pod "netserver-0": Phase="Running", Reason="", readiness=false. Elapsed: 5m0.203897142s Dec 22 17:27:17.576: INFO: The phase of Pod netserver-0 is Running (Ready = false) Dec 22 17:27:17.577: INFO: Unexpected error: <*pod.timeoutError | 0xc0031d1c50>: { msg: "timed out while waiting for pod nettest-7636/netserver-0 to be running and ready", observedObjects: [ <*v1.Pod | 0xc000124480>{ TypeMeta: {Kind: "", APIVersion: ""}, ObjectMeta: { ... skipping 128 lines ... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output Dec 22 17:27:17.577: FAIL: timed out while waiting for pod nettest-7636/netserver-0 to be running and ready Full Stack Trace k8s.io/kubernetes/test/e2e/framework/network.(*NetworkingTestConfig).createNetProxyPods(0xc004a4aa80, {0x75c6f5c, 0x9}, 0xc0037bd710) test/e2e/framework/network/utils.go:866 +0x1d0 k8s.io/kubernetes/test/e2e/framework/network.(*NetworkingTestConfig).setupCore(0xc004a4aa80, 0x7f72ac540698?) test/e2e/framework/network/utils.go:763 +0x55 ... skipping 26 lines ... Dec 22 17:27:17.712: INFO: At 2022-12-22 17:22:18 +0000 UTC - event for netserver-2: {kubelet i-0c76fbcc0b95be727} Created: Created container webserver Dec 22 17:27:17.712: INFO: At 2022-12-22 17:22:18 +0000 UTC - event for netserver-2: {kubelet i-0c76fbcc0b95be727} Pulled: Container image "registry.k8s.io/e2e-test-images/agnhost:2.43" already present on machine Dec 22 17:27:17.712: INFO: At 2022-12-22 17:22:18 +0000 UTC - event for netserver-2: {kubelet i-0c76fbcc0b95be727} Started: Started container webserver Dec 22 17:27:17.712: INFO: At 2022-12-22 17:22:18 +0000 UTC - event for netserver-3: {kubelet i-0ea878d3671f2eda2} Pulled: Container image "registry.k8s.io/e2e-test-images/agnhost:2.43" already present on machine Dec 22 17:27:17.712: INFO: At 2022-12-22 17:22:18 +0000 UTC - event for netserver-3: {kubelet i-0ea878d3671f2eda2} Created: Created container webserver Dec 22 17:27:17.712: INFO: At 2022-12-22 17:22:18 +0000 UTC - event for netserver-3: {kubelet i-0ea878d3671f2eda2} Started: Started container webserver Dec 22 17:27:17.712: INFO: At 2022-12-22 17:22:58 +0000 UTC - event for netserver-0: {kubelet i-0243def68c064e8ef} Unhealthy: Liveness probe failed: Get "http://172.20.37.202:8083/healthz": context deadline exceeded (Client.Timeout exceeded while awaiting headers) Dec 22 17:27:17.712: INFO: At 2022-12-22 17:22:58 +0000 UTC - event for netserver-0: {kubelet i-0243def68c064e8ef} Unhealthy: Readiness probe failed: Get "http://172.20.37.202:8083/healthz": context deadline exceeded (Client.Timeout exceeded while awaiting headers) Dec 22 17:27:17.712: INFO: At 2022-12-22 17:23:58 +0000 UTC - event for netserver-0: {kubelet i-0243def68c064e8ef} Killing: Container webserver failed liveness probe, will be restarted Dec 22 17:27:17.712: INFO: At 2022-12-22 17:24:38 +0000 UTC - event for netserver-0: {kubelet i-0243def68c064e8ef} Unhealthy: Liveness probe failed: Get "http://172.20.37.202:8083/healthz": dial tcp 172.20.37.202:8083: i/o timeout (Client.Timeout exceeded while awaiting headers) Dec 22 17:27:17.712: INFO: At 2022-12-22 17:25:58 +0000 UTC - event for netserver-0: {kubelet i-0243def68c064e8ef} Unhealthy: Readiness probe failed: Get "http://172.20.37.202:8083/healthz": dial tcp 172.20.37.202:8083: i/o timeout (Client.Timeout exceeded while awaiting headers) Dec 22 17:27:17.778: INFO: POD NODE PHASE GRACE CONDITIONS Dec 22 17:27:17.778: INFO: netserver-0 i-0243def68c064e8ef Running [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2022-12-22 17:22:17 +0000 UTC } {Ready False 0001-01-01 00:00:00 +0000 UTC 2022-12-22 17:22:17 +0000 UTC ContainersNotReady containers with unready status: [webserver]} {ContainersReady False 0001-01-01 00:00:00 +0000 UTC 2022-12-22 17:22:17 +0000 UTC ContainersNotReady containers with unready status: [webserver]} {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2022-12-22 17:22:17 +0000 UTC }] Dec 22 17:27:17.778: INFO: netserver-1 i-076110ae730e268e3 Running [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2022-12-22 17:22:17 +0000 UTC } {Ready True 0001-01-01 00:00:00 +0000 UTC 2022-12-22 17:22:28 +0000 UTC } {ContainersReady True 0001-01-01 00:00:00 +0000 UTC 2022-12-22 17:22:28 +0000 UTC } {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2022-12-22 17:22:17 +0000 UTC }] Dec 22 17:27:17.778: INFO: netserver-2 i-0c76fbcc0b95be727 Running [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2022-12-22 17:22:17 +0000 UTC } {Ready True 0001-01-01 00:00:00 +0000 UTC 2022-12-22 17:22:28 +0000 UTC } {ContainersReady True 0001-01-01 00:00:00 +0000 UTC 2022-12-22 17:22:28 +0000 UTC } {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2022-12-22 17:22:17 +0000 UTC }] Dec 22 17:27:17.778: INFO: netserver-3 i-0ea878d3671f2eda2 Running [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2022-12-22 17:22:17 +0000 UTC } {Ready True 0001-01-01 00:00:00 +0000 UTC 2022-12-22 17:22:28 +0000 UTC } {ContainersReady True 0001-01-01 00:00:00 +0000 UTC 2022-12-22 17:22:28 +0000 UTC } {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2022-12-22 17:22:17 +0000 UTC }] Dec 22 17:27:17.778: INFO: ... skipping 515 lines ... [sig-storage] In-tree Volumes [38;5;243mtest/e2e/storage/utils/framework.go:23[0m [Driver: local][LocalVolumeType: blockfs] [38;5;243mtest/e2e/storage/in_tree_volumes.go:85[0m [38;5;14m[1m[Testpattern: Dynamic PV (delayed binding)] topology [BeforeEach][0m [38;5;243mtest/e2e/storage/framework/testsuite.go:51[0m should fail to schedule a pod which has topologies that conflict with AllowedTopologies [38;5;243mtest/e2e/storage/testsuites/topology.go:191[0m [38;5;243mBegin Captured GinkgoWriter Output >>[0m [BeforeEach] [Testpattern: Dynamic PV (delayed binding)] topology test/e2e/storage/framework/testsuite.go:51 Dec 22 17:27:53.114: INFO: Driver local doesn't support DynamicPV -- skipping ... skipping 10 lines ... [sig-storage] In-tree Volumes [38;5;243mtest/e2e/storage/utils/framework.go:23[0m [Driver: local][LocalVolumeType: dir] [38;5;243mtest/e2e/storage/in_tree_volumes.go:85[0m [38;5;14m[1m[Testpattern: Dynamic PV (immediate binding)] topology [BeforeEach][0m [38;5;243mtest/e2e/storage/framework/testsuite.go:51[0m should fail to schedule a pod which has topologies that conflict with AllowedTopologies [38;5;243mtest/e2e/storage/testsuites/topology.go:191[0m [38;5;243mBegin Captured GinkgoWriter Output >>[0m [BeforeEach] [Testpattern: Dynamic PV (immediate binding)] topology test/e2e/storage/framework/testsuite.go:51 Dec 22 17:27:53.116: INFO: Driver local doesn't support DynamicPV -- skipping ... skipping 371 lines ... [sig-storage] In-tree Volumes [38;5;243mtest/e2e/storage/utils/framework.go:23[0m [Driver: hostPathSymlink] [38;5;243mtest/e2e/storage/in_tree_volumes.go:85[0m [38;5;14m[1m[Testpattern: Dynamic PV (immediate binding)] topology [BeforeEach][0m [38;5;243mtest/e2e/storage/framework/testsuite.go:51[0m should fail to schedule a pod which has topologies that conflict with AllowedTopologies [38;5;243mtest/e2e/storage/testsuites/topology.go:191[0m [38;5;243mBegin Captured GinkgoWriter Output >>[0m [BeforeEach] [Testpattern: Dynamic PV (immediate binding)] topology test/e2e/storage/framework/testsuite.go:51 Dec 22 17:28:03.978: INFO: Driver hostPathSymlink doesn't support DynamicPV -- skipping ... skipping 284 lines ... [38;5;14mS[0m [38;5;243m------------------------------[0m [38;5;10m• [SLOW TEST] [5.110 seconds][0m [0m[sig-apps] DisruptionController [38;5;243mshould observe PodDisruptionBudget status updated [Conformance][0m [38;5;243mtest/e2e/apps/disruption.go:141[0m [38;5;243m------------------------------[0m [38;5;9m• [FAILED] [307.977 seconds][0m [38;5;9m[1m[sig-api-machinery] AdmissionWebhook [Privileged:ClusterAdmin] [BeforeEach][0m [38;5;243mtest/e2e/apimachinery/webhook.go:90[0m should mutate configmap [Conformance] [38;5;243mtest/e2e/apimachinery/webhook.go:252[0m [38;5;243mBegin Captured GinkgoWriter Output >>[0m ... skipping 162 lines ... Dec 22 17:28:07.456: INFO: deployment status: v1.DeploymentStatus{ObservedGeneration:1, Replicas:1, UpdatedReplicas:1, ReadyReplicas:0, AvailableReplicas:0, UnavailableReplicas:1, Conditions:[]v1.DeploymentCondition{v1.DeploymentCondition{Type:"Available", Status:"False", LastUpdateTime:time.Date(2022, time.December, 22, 17, 23, 15, 0, time.Local), LastTransitionTime:time.Date(2022, time.December, 22, 17, 23, 15, 0, time.Local), Reason:"MinimumReplicasUnavailable", Message:"Deployment does not have minimum availability."}, v1.DeploymentCondition{Type:"Progressing", Status:"True", LastUpdateTime:time.Date(2022, time.December, 22, 17, 23, 15, 0, time.Local), LastTransitionTime:time.Date(2022, time.December, 22, 17, 23, 15, 0, time.Local), Reason:"ReplicaSetUpdated", Message:"ReplicaSet \"sample-webhook-deployment-865554f4d9\" is progressing."}}, CollisionCount:(*int32)(nil)} Dec 22 17:28:09.456: INFO: deployment status: v1.DeploymentStatus{ObservedGeneration:1, Replicas:1, UpdatedReplicas:1, ReadyReplicas:0, AvailableReplicas:0, UnavailableReplicas:1, Conditions:[]v1.DeploymentCondition{v1.DeploymentCondition{Type:"Available", Status:"False", LastUpdateTime:time.Date(2022, time.December, 22, 17, 23, 15, 0, time.Local), LastTransitionTime:time.Date(2022, time.December, 22, 17, 23, 15, 0, time.Local), Reason:"MinimumReplicasUnavailable", Message:"Deployment does not have minimum availability."}, v1.DeploymentCondition{Type:"Progressing", Status:"True", LastUpdateTime:time.Date(2022, time.December, 22, 17, 23, 15, 0, time.Local), LastTransitionTime:time.Date(2022, time.December, 22, 17, 23, 15, 0, time.Local), Reason:"ReplicaSetUpdated", Message:"ReplicaSet \"sample-webhook-deployment-865554f4d9\" is progressing."}}, CollisionCount:(*int32)(nil)} Dec 22 17:28:11.456: INFO: deployment status: v1.DeploymentStatus{ObservedGeneration:1, Replicas:1, UpdatedReplicas:1, ReadyReplicas:0, AvailableReplicas:0, UnavailableReplicas:1, Conditions:[]v1.DeploymentCondition{v1.DeploymentCondition{Type:"Available", Status:"False", LastUpdateTime:time.Date(2022, time.December, 22, 17, 23, 15, 0, time.Local), LastTransitionTime:time.Date(2022, time.December, 22, 17, 23, 15, 0, time.Local), Reason:"MinimumReplicasUnavailable", Message:"Deployment does not have minimum availability."}, v1.DeploymentCondition{Type:"Progressing", Status:"True", LastUpdateTime:time.Date(2022, time.December, 22, 17, 23, 15, 0, time.Local), LastTransitionTime:time.Date(2022, time.December, 22, 17, 23, 15, 0, time.Local), Reason:"ReplicaSetUpdated", Message:"ReplicaSet \"sample-webhook-deployment-865554f4d9\" is progressing."}}, CollisionCount:(*int32)(nil)} Dec 22 17:28:13.456: INFO: deployment status: v1.DeploymentStatus{ObservedGeneration:1, Replicas:1, UpdatedReplicas:1, ReadyReplicas:0, AvailableReplicas:0, UnavailableReplicas:1, Conditions:[]v1.DeploymentCondition{v1.DeploymentCondition{Type:"Available", Status:"False", LastUpdateTime:time.Date(2022, time.December, 22, 17, 23, 15, 0, time.Local), LastTransitionTime:time.Date(2022, time.December, 22, 17, 23, 15, 0, time.Local), Reason:"MinimumReplicasUnavailable", Message:"Deployment does not have minimum availability."}, v1.DeploymentCondition{Type:"Progressing", Status:"True", LastUpdateTime:time.Date(2022, time.December, 22, 17, 23, 15, 0, time.Local), LastTransitionTime:time.Date(2022, time.December, 22, 17, 23, 15, 0, time.Local), Reason:"ReplicaSetUpdated", Message:"ReplicaSet \"sample-webhook-deployment-865554f4d9\" is progressing."}}, CollisionCount:(*int32)(nil)} Dec 22 17:28:15.455: INFO: deployment status: v1.DeploymentStatus{ObservedGeneration:1, Replicas:1, UpdatedReplicas:1, ReadyReplicas:0, AvailableReplicas:0, UnavailableReplicas:1, Conditions:[]v1.DeploymentCondition{v1.DeploymentCondition{Type:"Available", Status:"False", LastUpdateTime:time.Date(2022, time.December, 22, 17, 23, 15, 0, time.Local), LastTransitionTime:time.Date(2022, time.December, 22, 17, 23, 15, 0, time.Local), Reason:"MinimumReplicasUnavailable", Message:"Deployment does not have minimum availability."}, v1.DeploymentCondition{Type:"Progressing", Status:"True", LastUpdateTime:time.Date(2022, time.December, 22, 17, 23, 15, 0, time.Local), LastTransitionTime:time.Date(2022, time.December, 22, 17, 23, 15, 0, time.Local), Reason:"ReplicaSetUpdated", Message:"ReplicaSet \"sample-webhook-deployment-865554f4d9\" is progressing."}}, CollisionCount:(*int32)(nil)} Dec 22 17:28:15.523: INFO: deployment status: v1.DeploymentStatus{ObservedGeneration:1, Replicas:1, UpdatedReplicas:1, ReadyReplicas:0, AvailableReplicas:0, UnavailableReplicas:1, Conditions:[]v1.DeploymentCondition{v1.DeploymentCondition{Type:"Available", Status:"False", LastUpdateTime:time.Date(2022, time.December, 22, 17, 23, 15, 0, time.Local), LastTransitionTime:time.Date(2022, time.December, 22, 17, 23, 15, 0, time.Local), Reason:"MinimumReplicasUnavailable", Message:"Deployment does not have minimum availability."}, v1.DeploymentCondition{Type:"Progressing", Status:"True", LastUpdateTime:time.Date(2022, time.December, 22, 17, 23, 15, 0, time.Local), LastTransitionTime:time.Date(2022, time.December, 22, 17, 23, 15, 0, time.Local), Reason:"ReplicaSetUpdated", Message:"ReplicaSet \"sample-webhook-deployment-865554f4d9\" is progressing."}}, CollisionCount:(*int32)(nil)} Dec 22 17:28:15.523: INFO: Unexpected error: waiting for the deployment status valid%!(EXTRA string=registry.k8s.io/e2e-test-images/agnhost:2.43, string=sample-webhook-deployment, string=webhook-1411): <*errors.errorString | 0xc001363840>: { s: "error waiting for deployment \"sample-webhook-deployment\" status to match expectation: deployment status: v1.DeploymentStatus{ObservedGeneration:1, Replicas:1, UpdatedReplicas:1, ReadyReplicas:0, AvailableReplicas:0, UnavailableReplicas:1, Conditions:[]v1.DeploymentCondition{v1.DeploymentCondition{Type:\"Available\", Status:\"False\", LastUpdateTime:time.Date(2022, time.December, 22, 17, 23, 15, 0, time.Local), LastTransitionTime:time.Date(2022, time.December, 22, 17, 23, 15, 0, time.Local), Reason:\"MinimumReplicasUnavailable\", Message:\"Deployment does not have minimum availability.\"}, v1.DeploymentCondition{Type:\"Progressing\", Status:\"True\", LastUpdateTime:time.Date(2022, time.December, 22, 17, 23, 15, 0, time.Local), LastTransitionTime:time.Date(2022, time.December, 22, 17, 23, 15, 0, time.Local), Reason:\"ReplicaSetUpdated\", Message:\"ReplicaSet \\\"sample-webhook-deployment-865554f4d9\\\" is progressing.\"}}, CollisionCount:(*int32)(nil)}", } Dec 22 17:28:15.523: FAIL: waiting for the deployment status valid%!(EXTRA string=registry.k8s.io/e2e-test-images/agnhost:2.43, string=sample-webhook-deployment, string=webhook-1411): error waiting for deployment "sample-webhook-deployment" status to match expectation: deployment status: v1.DeploymentStatus{ObservedGeneration:1, Replicas:1, UpdatedReplicas:1, ReadyReplicas:0, AvailableReplicas:0, UnavailableReplicas:1, Conditions:[]v1.DeploymentCondition{v1.DeploymentCondition{Type:"Available", Status:"False", LastUpdateTime:time.Date(2022, time.December, 22, 17, 23, 15, 0, time.Local), LastTransitionTime:time.Date(2022, time.December, 22, 17, 23, 15, 0, time.Local), Reason:"MinimumReplicasUnavailable", Message:"Deployment does not have minimum availability."}, v1.DeploymentCondition{Type:"Progressing", Status:"True", LastUpdateTime:time.Date(2022, time.December, 22, 17, 23, 15, 0, time.Local), LastTransitionTime:time.Date(2022, time.December, 22, 17, 23, 15, 0, time.Local), Reason:"ReplicaSetUpdated", Message:"ReplicaSet \"sample-webhook-deployment-865554f4d9\" is progressing."}}, CollisionCount:(*int32)(nil)} Full Stack Trace k8s.io/kubernetes/test/e2e/apimachinery.deployWebhookAndService(0xc000ac7e00, {0xc001cdebd0, 0x2c}, 0xc00455c9b0, 0x20fb, 0x20fc) test/e2e/apimachinery/webhook.go:826 +0xed2 k8s.io/kubernetes/test/e2e/apimachinery.glob..func28.1() test/e2e/apimachinery/webhook.go:102 +0x226 ... skipping 12 lines ... Dec 22 17:28:15.921: INFO: At 2022-12-22 17:23:15 +0000 UTC - event for sample-webhook-deployment: {deployment-controller } ScalingReplicaSet: Scaled up replica set sample-webhook-deployment-865554f4d9 to 1 Dec 22 17:28:15.921: INFO: At 2022-12-22 17:23:15 +0000 UTC - event for sample-webhook-deployment-865554f4d9: {replicaset-controller } SuccessfulCreate: Created pod: sample-webhook-deployment-865554f4d9-fpsrw Dec 22 17:28:15.921: INFO: At 2022-12-22 17:23:15 +0000 UTC - event for sample-webhook-deployment-865554f4d9-fpsrw: {default-scheduler } Scheduled: Successfully assigned webhook-1411/sample-webhook-deployment-865554f4d9-fpsrw to i-076110ae730e268e3 Dec 22 17:28:15.921: INFO: At 2022-12-22 17:23:16 +0000 UTC - event for sample-webhook-deployment-865554f4d9-fpsrw: {kubelet i-076110ae730e268e3} Pulled: Container image "registry.k8s.io/e2e-test-images/agnhost:2.43" already present on machine Dec 22 17:28:15.921: INFO: At 2022-12-22 17:23:16 +0000 UTC - event for sample-webhook-deployment-865554f4d9-fpsrw: {kubelet i-076110ae730e268e3} Created: Created container sample-webhook Dec 22 17:28:15.921: INFO: At 2022-12-22 17:23:16 +0000 UTC - event for sample-webhook-deployment-865554f4d9-fpsrw: {kubelet i-076110ae730e268e3} Started: Started container sample-webhook Dec 22 17:28:15.921: INFO: At 2022-12-22 17:23:18 +0000 UTC - event for sample-webhook-deployment-865554f4d9-fpsrw: {kubelet i-076110ae730e268e3} Unhealthy: Readiness probe failed: Get "https://172.20.54.167:8444/readyz": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) Dec 22 17:28:15.921: INFO: At 2022-12-22 17:23:25 +0000 UTC - event for sample-webhook-deployment-865554f4d9-fpsrw: {kubelet i-076110ae730e268e3} Unhealthy: Readiness probe failed: Get "https://172.20.54.167:8444/readyz": dial tcp 172.20.54.167:8444: i/o timeout Dec 22 17:28:15.921: INFO: At 2022-12-22 17:23:32 +0000 UTC - event for sample-webhook-deployment-865554f4d9-fpsrw: {kubelet i-076110ae730e268e3} Unhealthy: Readiness probe failed: Get "https://172.20.54.167:8444/readyz": context deadline exceeded Dec 22 17:28:15.986: INFO: POD NODE PHASE GRACE CONDITIONS Dec 22 17:28:15.986: INFO: Dec 22 17:28:16.053: INFO: Logging node info for node i-0243def68c064e8ef Dec 22 17:28:16.118: INFO: Node Info: &Node{ObjectMeta:{i-0243def68c064e8ef f07b8771-29eb-4944-a385-3482a3d3a315 18036 0 2022-12-22 17:17:47 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:us-west-2 failure-domain.beta.kubernetes.io/zone:us-west-2a kubernetes.io/arch:amd64 kubernetes.io/hostname:i-0243def68c064e8ef kubernetes.io/os:linux node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:us-west-2a topology.hostpath.csi/node:i-0243def68c064e8ef topology.kubernetes.io/region:us-west-2 topology.kubernetes.io/zone:us-west-2a] map[alpha.kubernetes.io/provided-node-ip:172.20.35.142 csi.volume.kubernetes.io/nodeid:{"csi-hostpath-ephemeral-2878":"i-0243def68c064e8ef","ebs.csi.aws.com":"i-0243def68c064e8ef"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{aws-cloud-controller-manager Update v1 2022-12-22 17:17:47 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2022-12-22 17:17:47 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kops-controller Update v1 2022-12-22 17:17:47 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{}}}} } {kube-controller-manager Update v1 2022-12-22 17:17:47 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.3.0/24\"":{}}}} } {kubelet Update v1 2022-12-22 17:17:47 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:alpha.kubernetes.io/provided-node-ip":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {kube-controller-manager Update v1 2022-12-22 17:28:11 +0000 UTC FieldsV1 {"f:status":{"f:volumesAttached":{}}} status} {kubelet Update v1 2022-12-22 17:28:11 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.3.0/24,DoNotUseExternalID:,ProviderID:aws:///us-west-2a/i-0243def68c064e8ef,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.3.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{51527004160 0} {<nil>} 50319340Ki BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4050636800 0} {<nil>} 3955700Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{46374303668 0} {<nil>} 46374303668 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3945779200 0} {<nil>} 3853300Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-12-22 17:28:11 +0000 UTC,LastTransitionTime:2022-12-22 17:17:37 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-12-22 17:28:11 +0000 UTC,LastTransitionTime:2022-12-22 17:17:37 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-12-22 17:28:11 +0000 UTC,LastTransitionTime:2022-12-22 17:17:37 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-12-22 17:28:11 +0000 UTC,LastTransitionTime:2022-12-22 17:17:59 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.35.142,},NodeAddress{Type:ExternalIP,Address:35.90.235.118,},NodeAddress{Type:InternalDNS,Address:i-0243def68c064e8ef.us-west-2.compute.internal,},NodeAddress{Type:Hostname,Address:i-0243def68c064e8ef.us-west-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-35-90-235-118.us-west-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2c8ae28bb7834ee5f6b4c4ef144a8e,SystemUUID:ec2c8ae2-8bb7-834e-e5f6-b4c4ef144a8e,BootID:5843895c-91be-4c36-b700-9e551cb745de,KernelVersion:5.10.157-139.675.amzn2.x86_64,OSImage:Amazon Linux 2,ContainerRuntimeVersion:containerd://1.6.14,KubeletVersion:v1.26.0,KubeProxyVersion:v1.26.0,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:166719855,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/jessie-dnsutils@sha256:24aaf2626d6b27864c29de2097e8bbb840b3a414271bf7c8995e431e47d8408e registry.k8s.io/e2e-test-images/jessie-dnsutils:1.7],SizeBytes:112030336,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:dd75ec974b0a2a6f6bb47001ba09207976e625db898d1b16735528c009cb171c registry.k8s.io/etcd:3.5.6-0],SizeBytes:102542580,},ContainerImage{Names:[registry.k8s.io/kube-proxy-amd64:v1.26.0],SizeBytes:67205320,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:16bbf38c463a4223d8cfe4da12bc61010b082a79b4bb003e2d3ba3ece5dd5f9e registry.k8s.io/e2e-test-images/agnhost:2.43],SizeBytes:51706353,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nautilus@sha256:80ba6c8c44f9623f06e868a1aa66026c8ec438ad814f9ec95e9333b415fe3550 registry.k8s.io/e2e-test-images/nautilus:1.7],SizeBytes:49641698,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:3fe7acf013d1264ffded116b80a73dc129a449b0fccdb8d21af8279f2233f36e registry.k8s.io/e2e-test-images/httpd:2.4.39-4],SizeBytes:41901587,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:148b022f5c5da426fc2f3c14b5c0867e58ef05961510c84749ac1fddcb0fef22 registry.k8s.io/e2e-test-images/httpd:2.4.38-4],SizeBytes:40764257,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:10e231cf07c82dfab5141c1afe548dc734e5fc1f67665ec3982a325d1bd31a9a registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.0],SizeBytes:29725756,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:ee3b525d5b89db99da3b8eb521d9cd90cb6e9ef0fbb651e98bb37be78d36b5b8 registry.k8s.io/sig-storage/csi-provisioner:v3.3.0],SizeBytes:25491225,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:425d8f1b769398127767b06ed97ce62578a3179bcb99809ce93a1649e025ffe7 registry.k8s.io/sig-storage/csi-resizer:v1.6.0],SizeBytes:24148884,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:291334908ddf71a4661fd7f6d9d97274de8a5378a2b6fdfeb2ce73414a34f82f registry.k8s.io/sig-storage/csi-snapshotter:v6.1.0],SizeBytes:23881995,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:9a685020911e2725ad019dbce6e4a5ab93d51e3d4557f115e64343345e05781b registry.k8s.io/sig-storage/csi-attacher:v4.0.0],SizeBytes:23847201,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:92257881c1d6493cf18299a24af42330f891166560047902b8d431fb66b01af5 registry.k8s.io/sig-storage/hostpathplugin:v1.9.0],SizeBytes:18758628,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:933940f13b3ea0abc62e656c1aa5c5b47c04b15d71250413a6b821bd0c58b94e registry.k8s.io/sig-storage/livenessprobe:v2.7.0],SizeBytes:8688564,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:5c99cf6a02adda929b10321dbf4ecfa00d87be9ba4fb456006237d530ab4baa1 registry.k8s.io/e2e-test-images/nginx:1.14-4],SizeBytes:6978614,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf registry.k8s.io/e2e-test-images/busybox:1.29-2],SizeBytes:732424,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:2e0f836850e09b8b7cc937681d6194537a09fbd5f6b9e08f4d646a85128e8937 registry.k8s.io/e2e-test-images/busybox:1.29-4],SizeBytes:731990,},ContainerImage{Names:[registry.k8s.io/pause@sha256:7031c1b283388d2c2e09b57badb803c05ebed362dc88d84b480cc47f72a21097 registry.k8s.io/pause:3.9],SizeBytes:321520,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[kubernetes.io/csi/ebs.csi.aws.com^vol-01072d677984dcb84 kubernetes.io/csi/ebs.csi.aws.com^vol-01f61f87220e96e7a],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-01072d677984dcb84,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-01f61f87220e96e7a,DevicePath:,},},Config:nil,},} Dec 22 17:28:16.118: INFO: ... skipping 440 lines ... [DeferCleanup (Each)] [sig-api-machinery] AdmissionWebhook [Privileged:ClusterAdmin] tear down framework | framework.go:193 [1mSTEP:[0m Destroying namespace "webhook-1411" for this suite. [38;5;243m12/22/22 17:28:21.501[0m [1mSTEP:[0m Destroying namespace "webhook-1411-markers" for this suite. [38;5;243m12/22/22 17:28:21.568[0m [38;5;243m<< End Captured GinkgoWriter Output[0m [38;5;9mDec 22 17:28:15.523: waiting for the deployment status valid%!(EXTRA string=registry.k8s.io/e2e-test-images/agnhost:2.43, string=sample-webhook-deployment, string=webhook-1411): error waiting for deployment "sample-webhook-deployment" status to match expectation: deployment status: v1.DeploymentStatus{ObservedGeneration:1, Replicas:1, UpdatedReplicas:1, ReadyReplicas:0, AvailableReplicas:0, UnavailableReplicas:1, Conditions:[]v1.DeploymentCondition{v1.DeploymentCondition{Type:"Available", Status:"False", LastUpdateTime:time.Date(2022, time.December, 22, 17, 23, 15, 0, time.Local), LastTransitionTime:time.Date(2022, time.December, 22, 17, 23, 15, 0, time.Local), Reason:"MinimumReplicasUnavailable", Message:"Deployment does not have minimum availability."}, v1.DeploymentCondition{Type:"Progressing", Status:"True", LastUpdateTime:time.Date(2022, time.December, 22, 17, 23, 15, 0, time.Local), LastTransitionTime:time.Date(2022, time.December, 22, 17, 23, 15, 0, time.Local), Reason:"ReplicaSetUpdated", Message:"ReplicaSet \"sample-webhook-deployment-865554f4d9\" is progressing."}}, CollisionCount:(*int32)(nil)}[0m [38;5;9mIn [1m[BeforeEach][0m[38;5;9m at: [1mtest/e2e/apimachinery/webhook.go:826[0m [38;5;243m------------------------------[0m [38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m [38;5;243m------------------------------[0m [38;5;10m• [SLOW TEST] [10.627 seconds][0m [0m[sig-storage] PersistentVolumes-local [38;5;243m[Volume type: dir-link-bindmounted] [0mTwo pods mounting a local volume one after the other [38;5;243mshould be able to write from pod1 and read from pod2[0m ... skipping 254 lines ... [sig-storage] In-tree Volumes [38;5;243mtest/e2e/storage/utils/framework.go:23[0m [Driver: local][LocalVolumeType: dir-bindmounted] [38;5;243mtest/e2e/storage/in_tree_volumes.go:85[0m [38;5;14m[1m[Testpattern: Dynamic PV (immediate binding)] topology [BeforeEach][0m [38;5;243mtest/e2e/storage/framework/testsuite.go:51[0m should fail to schedule a pod which has topologies that conflict with AllowedTopologies [38;5;243mtest/e2e/storage/testsuites/topology.go:191[0m [38;5;243mBegin Captured GinkgoWriter Output >>[0m [BeforeEach] [Testpattern: Dynamic PV (immediate binding)] topology test/e2e/storage/framework/testsuite.go:51 Dec 22 17:28:49.090: INFO: Driver local doesn't support DynamicPV -- skipping ... skipping 98 lines ... [38;5;243m------------------------------[0m [38;5;14mS [SKIPPED] [0.714 seconds][0m External Storage [Driver: ebs.csi.aws.com] [38;5;243mtest/e2e/storage/external/external.go:173[0m [Testpattern: Dynamic PV (immediate binding)] topology [38;5;243mtest/e2e/storage/framework/testsuite.go:50[0m [38;5;14m[1m[It] should fail to schedule a pod which has topologies that conflict with AllowedTopologies[0m [38;5;243mtest/e2e/storage/testsuites/topology.go:191[0m [38;5;243mBegin Captured GinkgoWriter Output >>[0m [BeforeEach] [Testpattern: Dynamic PV (immediate binding)] topology test/e2e/storage/framework/testsuite.go:51 [BeforeEach] [Testpattern: Dynamic PV (immediate binding)] topology ... skipping 2 lines ... Dec 22 17:28:48.768: INFO: >>> kubeConfig: /root/.kube/config [1mSTEP:[0m Building a namespace api object, basename topology [38;5;243m12/22/22 17:28:48.769[0m [1mSTEP:[0m Waiting for a default service account to be provisioned in namespace [38;5;243m12/22/22 17:28:48.963[0m [1mSTEP:[0m Waiting for kube-root-ca.crt to be provisioned in namespace [38;5;243m12/22/22 17:28:49.091[0m [BeforeEach] [Testpattern: Dynamic PV (immediate binding)] topology test/e2e/framework/metrics/init/init.go:31 [It] should fail to schedule a pod which has topologies that conflict with AllowedTopologies test/e2e/storage/testsuites/topology.go:191 Dec 22 17:28:49.285: INFO: found topology map[topology.ebs.csi.aws.com/zone:us-west-2a] Dec 22 17:28:49.285: INFO: Not enough topologies in cluster -- skipping [1mSTEP:[0m Deleting pvc [38;5;243m12/22/22 17:28:49.286[0m [1mSTEP:[0m Deleting sc [38;5;243m12/22/22 17:28:49.286[0m [AfterEach] [Testpattern: Dynamic PV (immediate binding)] topology ... skipping 101 lines ... [0mExternal Storage [Driver: ebs.csi.aws.com] [38;5;243m[Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy [0m(OnRootMismatch)[LinuxOnly], pod created with an initial fsgroup, volume contents ownership changed via chgrp in first pod, new pod with same fsgroup skips ownership changes to the volume contents[0m [38;5;243mtest/e2e/storage/testsuites/fsgroupchangepolicy.go:210[0m [38;5;243m------------------------------[0m [38;5;14mS[0m[38;5;10m•[0m[38;5;14mS[0m [38;5;243m------------------------------[0m [38;5;10m• [SLOW TEST] [301.191 seconds][0m [0m[sig-storage] Dynamic Provisioning [38;5;243mInvalid AWS KMS key [0mshould report an error and create no PV[0m [38;5;243mtest/e2e/storage/volume_provisioning.go:705[0m [38;5;243m------------------------------[0m [38;5;10m• [SLOW TEST] [5.206 seconds][0m [0m[sig-auth] ServiceAccounts [38;5;243mshould mount an API token into pods [Conformance][0m [38;5;243mtest/e2e/auth/service_accounts.go:78[0m [38;5;243m------------------------------[0m ... skipping 2 lines ... [38;5;10m• [SLOW TEST] [26.087 seconds][0m [0m[sig-storage] In-tree Volumes [38;5;243m[Driver: hostPathSymlink] [0m[Testpattern: Inline-volume (default fs)] volumes [38;5;243mshould store data[0m [38;5;243mtest/e2e/storage/testsuites/volumes.go:158[0m [38;5;243m------------------------------[0m [38;5;14mS[0m [38;5;243m------------------------------[0m [38;5;9m• [FAILED] [318.338 seconds][0m [sig-network] Networking [38;5;243mtest/e2e/network/common/framework.go:23[0m Granular Checks: Services [38;5;243mtest/e2e/network/networking.go:145[0m [38;5;9m[1m[It] should support basic nodePort: udp functionality[0m [38;5;243mtest/e2e/network/networking.go:384[0m ... skipping 334 lines ... Dec 22 17:29:04.592: INFO: Pod "netserver-1": Phase="Running", Reason="", readiness=false. Elapsed: 4m58.129739184s Dec 22 17:29:04.592: INFO: The phase of Pod netserver-1 is Running (Ready = false) Dec 22 17:29:06.593: INFO: Pod "netserver-1": Phase="Running", Reason="", readiness=false. Elapsed: 5m0.130760721s Dec 22 17:29:06.593: INFO: The phase of Pod netserver-1 is Running (Ready = false) Dec 22 17:29:06.657: INFO: Pod "netserver-1": Phase="Running", Reason="", readiness=false. Elapsed: 5m0.194666966s Dec 22 17:29:06.657: INFO: The phase of Pod netserver-1 is Running (Ready = false) Dec 22 17:29:06.658: INFO: Unexpected error: <*pod.timeoutError | 0xc0038be480>: { msg: "timed out while waiting for pod nettest-1737/netserver-1 to be running and ready", observedObjects: [ <*v1.Pod | 0xc003bac000>{ TypeMeta: {Kind: "", APIVersion: ""}, ObjectMeta: { ... skipping 128 lines ... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output Dec 22 17:29:06.658: FAIL: timed out while waiting for pod nettest-1737/netserver-1 to be running and ready Full Stack Trace k8s.io/kubernetes/test/e2e/framework/network.(*NetworkingTestConfig).createNetProxyPods(0xc004075420, {0x75c6f5c, 0x9}, 0xc0065b6cf0) test/e2e/framework/network/utils.go:866 +0x1d0 k8s.io/kubernetes/test/e2e/framework/network.(*NetworkingTestConfig).setupCore(0xc004075420, 0x7fdc6ef53f70?) test/e2e/framework/network/utils.go:763 +0x55 ... skipping 26 lines ... Dec 22 17:29:06.788: INFO: At 2022-12-22 17:23:53 +0000 UTC - event for netserver-3: {kubelet i-0ea878d3671f2eda2} Pulled: Container image "registry.k8s.io/e2e-test-images/agnhost:2.43" already present on machine Dec 22 17:29:06.788: INFO: At 2022-12-22 17:23:53 +0000 UTC - event for netserver-3: {kubelet i-0ea878d3671f2eda2} Created: Created container webserver Dec 22 17:29:06.788: INFO: At 2022-12-22 17:23:53 +0000 UTC - event for netserver-3: {kubelet i-0ea878d3671f2eda2} Started: Started container webserver Dec 22 17:29:06.788: INFO: At 2022-12-22 17:23:54 +0000 UTC - event for netserver-1: {kubelet i-076110ae730e268e3} Pulled: Container image "registry.k8s.io/e2e-test-images/agnhost:2.43" already present on machine Dec 22 17:29:06.788: INFO: At 2022-12-22 17:23:54 +0000 UTC - event for netserver-1: {kubelet i-076110ae730e268e3} Started: Started container webserver Dec 22 17:29:06.788: INFO: At 2022-12-22 17:23:54 +0000 UTC - event for netserver-1: {kubelet i-076110ae730e268e3} Created: Created container webserver Dec 22 17:29:06.788: INFO: At 2022-12-22 17:24:43 +0000 UTC - event for netserver-1: {kubelet i-076110ae730e268e3} Unhealthy: Readiness probe failed: Get "http://172.20.51.11:8083/healthz": context deadline exceeded (Client.Timeout exceeded while awaiting headers) Dec 22 17:29:06.788: INFO: At 2022-12-22 17:24:43 +0000 UTC - event for netserver-1: {kubelet i-076110ae730e268e3} Unhealthy: Liveness probe failed: Get "http://172.20.51.11:8083/healthz": context deadline exceeded (Client.Timeout exceeded while awaiting headers) Dec 22 17:29:06.788: INFO: At 2022-12-22 17:25:43 +0000 UTC - event for netserver-1: {kubelet i-076110ae730e268e3} Unhealthy: Readiness probe failed: Get "http://172.20.51.11:8083/healthz": dial tcp 172.20.51.11:8083: i/o timeout (Client.Timeout exceeded while awaiting headers) Dec 22 17:29:06.788: INFO: At 2022-12-22 17:25:43 +0000 UTC - event for netserver-1: {kubelet i-076110ae730e268e3} Killing: Container webserver failed liveness probe, will be restarted Dec 22 17:29:06.788: INFO: At 2022-12-22 17:26:23 +0000 UTC - event for netserver-1: {kubelet i-076110ae730e268e3} Unhealthy: Liveness probe failed: Get "http://172.20.51.11:8083/healthz": dial tcp 172.20.51.11:8083: i/o timeout (Client.Timeout exceeded while awaiting headers) Dec 22 17:29:06.852: INFO: POD NODE PHASE GRACE CONDITIONS Dec 22 17:29:06.852: INFO: netserver-0 i-0243def68c064e8ef Running [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2022-12-22 17:23:52 +0000 UTC } {Ready True 0001-01-01 00:00:00 +0000 UTC 2022-12-22 17:24:02 +0000 UTC } {ContainersReady True 0001-01-01 00:00:00 +0000 UTC 2022-12-22 17:24:02 +0000 UTC } {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2022-12-22 17:23:52 +0000 UTC }] Dec 22 17:29:06.852: INFO: netserver-1 i-076110ae730e268e3 Running [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2022-12-22 17:23:52 +0000 UTC } {Ready False 0001-01-01 00:00:00 +0000 UTC 2022-12-22 17:23:52 +0000 UTC ContainersNotReady containers with unready status: [webserver]} {ContainersReady False 0001-01-01 00:00:00 +0000 UTC 2022-12-22 17:23:52 +0000 UTC ContainersNotReady containers with unready status: [webserver]} {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2022-12-22 17:23:52 +0000 UTC }] Dec 22 17:29:06.852: INFO: netserver-2 i-0c76fbcc0b95be727 Running [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2022-12-22 17:23:52 +0000 UTC } {Ready True 0001-01-01 00:00:00 +0000 UTC 2022-12-22 17:24:12 +0000 UTC } {ContainersReady True 0001-01-01 00:00:00 +0000 UTC 2022-12-22 17:24:12 +0000 UTC } {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2022-12-22 17:23:52 +0000 UTC }] Dec 22 17:29:06.852: INFO: netserver-3 i-0ea878d3671f2eda2 Running [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2022-12-22 17:23:52 +0000 UTC } {Ready True 0001-01-01 00:00:00 +0000 UTC 2022-12-22 17:24:03 +0000 UTC } {ContainersReady True 0001-01-01 00:00:00 +0000 UTC 2022-12-22 17:24:03 +0000 UTC } {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2022-12-22 17:23:52 +0000 UTC }] Dec 22 17:29:06.852: INFO: ... skipping 783 lines ... [sig-storage] In-tree Volumes [38;5;243mtest/e2e/storage/utils/framework.go:23[0m [Driver: vsphere] [38;5;243mtest/e2e/storage/in_tree_volumes.go:85[0m [38;5;14m[1m[Testpattern: Dynamic PV (delayed binding)] topology [BeforeEach][0m [38;5;243mtest/e2e/storage/framework/testsuite.go:51[0m should fail to schedule a pod which has topologies that conflict with AllowedTopologies [38;5;243mtest/e2e/storage/testsuites/topology.go:191[0m [38;5;243mBegin Captured GinkgoWriter Output >>[0m [BeforeEach] [Testpattern: Dynamic PV (delayed binding)] topology test/e2e/storage/framework/testsuite.go:51 Dec 22 17:29:20.439: INFO: Only supported for providers [vsphere] (not aws) ... skipping 1515 lines ... [sig-storage] In-tree Volumes [38;5;243mtest/e2e/storage/utils/framework.go:23[0m [Driver: local][LocalVolumeType: dir-link] [38;5;243mtest/e2e/storage/in_tree_volumes.go:85[0m [38;5;14m[1m[Testpattern: Dynamic PV (delayed binding)] topology [BeforeEach][0m [38;5;243mtest/e2e/storage/framework/testsuite.go:51[0m should fail to schedule a pod which has topologies that conflict with AllowedTopologies [38;5;243mtest/e2e/storage/testsuites/topology.go:191[0m [38;5;243mBegin Captured GinkgoWriter Output >>[0m [BeforeEach] [Testpattern: Dynamic PV (delayed binding)] topology test/e2e/storage/framework/testsuite.go:51 Dec 22 17:30:05.266: INFO: Driver local doesn't support DynamicPV -- skipping ... skipping 320 lines ... [sig-storage] In-tree Volumes [38;5;243mtest/e2e/storage/utils/framework.go:23[0m [Driver: cinder] [38;5;243mtest/e2e/storage/in_tree_volumes.go:85[0m [38;5;14m[1m[Testpattern: Dynamic PV (delayed binding)] topology [BeforeEach][0m [38;5;243mtest/e2e/storage/framework/testsuite.go:51[0m should fail to schedule a pod which has topologies that conflict with AllowedTopologies [38;5;243mtest/e2e/storage/testsuites/topology.go:191[0m [38;5;243mBegin Captured GinkgoWriter Output >>[0m [BeforeEach] [Testpattern: Dynamic PV (delayed binding)] topology test/e2e/storage/framework/testsuite.go:51 Dec 22 17:30:07.234: INFO: Only supported for providers [openstack] (not aws) ... skipping 804 lines ... [38;5;243m------------------------------[0m [38;5;14mS [SKIPPED] [0.713 seconds][0m External Storage [Driver: ebs.csi.aws.com] [38;5;243mtest/e2e/storage/external/external.go:173[0m [Testpattern: Dynamic PV (delayed binding)] topology [38;5;243mtest/e2e/storage/framework/testsuite.go:50[0m [38;5;14m[1m[It] should fail to schedule a pod which has topologies that conflict with AllowedTopologies[0m [38;5;243mtest/e2e/storage/testsuites/topology.go:191[0m [38;5;243mBegin Captured GinkgoWriter Output >>[0m [BeforeEach] [Testpattern: Dynamic PV (delayed binding)] topology test/e2e/storage/framework/testsuite.go:51 [BeforeEach] [Testpattern: Dynamic PV (delayed binding)] topology ... skipping 2 lines ... Dec 22 17:30:43.763: INFO: >>> kubeConfig: /root/.kube/config [1mSTEP:[0m Building a namespace api object, basename topology [38;5;243m12/22/22 17:30:43.764[0m [1mSTEP:[0m Waiting for a default service account to be provisioned in namespace [38;5;243m12/22/22 17:30:43.957[0m [1mSTEP:[0m Waiting for kube-root-ca.crt to be provisioned in namespace [38;5;243m12/22/22 17:30:44.084[0m [BeforeEach] [Testpattern: Dynamic PV (delayed binding)] topology test/e2e/framework/metrics/init/init.go:31 [It] should fail to schedule a pod which has topologies that conflict with AllowedTopologies test/e2e/storage/testsuites/topology.go:191 Dec 22 17:30:44.277: INFO: found topology map[topology.ebs.csi.aws.com/zone:us-west-2a] Dec 22 17:30:44.277: INFO: Not enough topologies in cluster -- skipping [1mSTEP:[0m Deleting pvc [38;5;243m12/22/22 17:30:44.277[0m [1mSTEP:[0m Deleting sc [38;5;243m12/22/22 17:30:44.277[0m [AfterEach] [Testpattern: Dynamic PV (delayed binding)] topology ... skipping 258 lines ... [sig-storage] In-tree Volumes [38;5;243mtest/e2e/storage/utils/framework.go:23[0m [Driver: local][LocalVolumeType: dir] [38;5;243mtest/e2e/storage/in_tree_volumes.go:85[0m [38;5;14m[1m[Testpattern: Dynamic PV (delayed binding)] topology [BeforeEach][0m [38;5;243mtest/e2e/storage/framework/testsuite.go:51[0m should fail to schedule a pod which has topologies that conflict with AllowedTopologies [38;5;243mtest/e2e/storage/testsuites/topology.go:191[0m [38;5;243mBegin Captured GinkgoWriter Output >>[0m [BeforeEach] [Testpattern: Dynamic PV (delayed binding)] topology test/e2e/storage/framework/testsuite.go:51 Dec 22 17:31:00.301: INFO: Driver local doesn't support DynamicPV -- skipping ... skipping 1528 lines ... [38;5;14mDriver csi-hostpath doesn't support PreprovisionedPV -- skipping[0m [38;5;14mIn [1m[BeforeEach][0m[38;5;14m at: [1mtest/e2e/storage/framework/testsuite.go:116[0m [38;5;243m------------------------------[0m [38;5;14mS [SKIPPED] [0.568 seconds][0m [38;5;14m[1m[sig-node] NodeProblemDetector [BeforeEach][0m [38;5;243mtest/e2e/node/node_problem_detector.go:54[0m should run without error [38;5;243mtest/e2e/node/node_problem_detector.go:62[0m [38;5;243mBegin Captured GinkgoWriter Output >>[0m [BeforeEach] [sig-node] NodeProblemDetector set up framework | framework.go:178 [1mSTEP:[0m Creating a kubernetes client [38;5;243m12/22/22 17:32:23.635[0m ... skipping 921 lines ... [sig-storage] In-tree Volumes [38;5;243mtest/e2e/storage/utils/framework.go:23[0m [Driver: local][LocalVolumeType: tmpfs] [38;5;243mtest/e2e/storage/in_tree_volumes.go:85[0m [38;5;14m[1m[Testpattern: Dynamic PV (delayed binding)] topology [BeforeEach][0m [38;5;243mtest/e2e/storage/framework/testsuite.go:51[0m should fail to schedule a pod which has topologies that conflict with AllowedTopologies [38;5;243mtest/e2e/storage/testsuites/topology.go:191[0m [38;5;243mBegin Captured GinkgoWriter Output >>[0m [BeforeEach] [Testpattern: Dynamic PV (delayed binding)] topology test/e2e/storage/framework/testsuite.go:51 Dec 22 17:32:59.707: INFO: Driver local doesn't support DynamicPV -- skipping ... skipping 251 lines ... [sig-storage] CSI Volumes [38;5;243mtest/e2e/storage/utils/framework.go:23[0m [Driver: csi-hostpath] [38;5;243mtest/e2e/storage/csi_volumes.go:40[0m [38;5;14m[1m[Testpattern: Dynamic PV (delayed binding)] topology [BeforeEach][0m [38;5;243mtest/e2e/storage/framework/testsuite.go:51[0m should fail to schedule a pod which has topologies that conflict with AllowedTopologies [38;5;243mtest/e2e/storage/testsuites/topology.go:191[0m [38;5;243mBegin Captured GinkgoWriter Output >>[0m [BeforeEach] [Testpattern: Dynamic PV (delayed binding)] topology test/e2e/storage/framework/testsuite.go:51 Dec 22 17:33:18.358: INFO: Driver "csi-hostpath" does not support topology - skipping ... skipping 632 lines ... [sig-storage] In-tree Volumes [38;5;243mtest/e2e/storage/utils/framework.go:23[0m [Driver: local][LocalVolumeType: tmpfs] [38;5;243mtest/e2e/storage/in_tree_volumes.go:85[0m [38;5;14m[1m[Testpattern: Dynamic PV (immediate binding)] topology [BeforeEach][0m [38;5;243mtest/e2e/storage/framework/testsuite.go:51[0m should fail to schedule a pod which has topologies that conflict with AllowedTopologies [38;5;243mtest/e2e/storage/testsuites/topology.go:191[0m [38;5;243mBegin Captured GinkgoWriter Output >>[0m [BeforeEach] [Testpattern: Dynamic PV (immediate binding)] topology test/e2e/storage/framework/testsuite.go:51 Dec 22 17:33:28.958: INFO: Driver local doesn't support DynamicPV -- skipping ... skipping 1058 lines ... [38;5;14mDriver local doesn't support DynamicPV -- skipping[0m [38;5;14mIn [1m[BeforeEach][0m[38;5;14m at: [1mtest/e2e/storage/framework/testsuite.go:116[0m [38;5;243m------------------------------[0m [38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m [38;5;243m------------------------------[0m [38;5;10m• [SLOW TEST] [24.793 seconds][0m [0m[sig-apps] Job [38;5;243mshould run a job to completion when tasks sometimes fail and are not locally restarted[0m [38;5;243mtest/e2e/apps/job.go:443[0m [38;5;243m------------------------------[0m [38;5;14mS [SKIPPED] [0.000 seconds][0m [sig-storage] In-tree Volumes [38;5;243mtest/e2e/storage/utils/framework.go:23[0m [Driver: hostPathSymlink] ... skipping 1168 lines ... [38;5;243m<< End Captured GinkgoWriter Output[0m [38;5;14mDriver csi-hostpath doesn't support InlineVolume -- skipping[0m [38;5;14mIn [1m[BeforeEach][0m[38;5;14m at: [1mtest/e2e/storage/framework/testsuite.go:116[0m [38;5;243m------------------------------[0m [38;5;10m• [SLOW TEST] [77.168 seconds][0m [0m[sig-apps] CronJob [38;5;243mshould delete failed finished jobs with limit of one job[0m [38;5;243mtest/e2e/apps/cronjob.go:291[0m [38;5;243m------------------------------[0m [38;5;10m•[0m[38;5;10m•[0m[38;5;14mS[0m [38;5;243m------------------------------[0m [38;5;14mS [SKIPPED] [0.000 seconds][0m [sig-storage] In-tree Volumes ... skipping 356 lines ... [38;5;14mDriver local doesn't support InlineVolume -- skipping[0m [38;5;14mIn [1m[BeforeEach][0m[38;5;14m at: [1mtest/e2e/storage/framework/testsuite.go:116[0m [38;5;243m------------------------------[0m [38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m [38;5;243m------------------------------[0m [38;5;9m• [FAILED] [831.878 seconds][0m [sig-storage] CSI Volumes [38;5;243mtest/e2e/storage/utils/framework.go:23[0m [Driver: csi-hostpath] [38;5;243mtest/e2e/storage/csi_volumes.go:40[0m [Testpattern: Dynamic PV (default fs)(allowExpansion)] volume-expand [38;5;243mtest/e2e/storage/framework/testsuite.go:50[0m ... skipping 87 lines ... [1mSTEP:[0m Deleting the previously created pod [38;5;243m12/22/22 17:22:03.55[0m Dec 22 17:22:03.550: INFO: Deleting pod "pod-c04ee946-2188-47e7-b34c-5a43495db240" in namespace "volume-expand-7749" Dec 22 17:22:03.617: INFO: Wait up to 5m0s for pod "pod-c04ee946-2188-47e7-b34c-5a43495db240" to be fully deleted [1mSTEP:[0m Expanding current pvc [38;5;243m12/22/22 17:22:07.745[0m Dec 22 17:22:07.745: INFO: currentPvcSize {{1073741824 0} {<nil>} 1Gi BinarySI}, newSize {{2147483648 0} {<nil>} BinarySI} [1mSTEP:[0m Waiting for cloudprovider resize to finish [38;5;243m12/22/22 17:22:07.88[0m Dec 22 17:32:08.074: INFO: Unexpected error: While waiting for pvc resize to finish: <*errors.errorString | 0xc000d7a970>: { s: "error while waiting for controller resize to finish: timed out waiting for the condition", } Dec 22 17:32:08.074: FAIL: While waiting for pvc resize to finish: error while waiting for controller resize to finish: timed out waiting for the condition Full Stack Trace k8s.io/kubernetes/test/e2e/storage/testsuites.(*volumeExpandTestSuite).DefineTests.func4() test/e2e/storage/testsuites/volume_expand.go:218 +0xa85 Dec 22 17:32:08.074: INFO: Deleting pod "pod-c04ee946-2188-47e7-b34c-5a43495db240" in namespace "volume-expand-7749" [1mSTEP:[0m Deleting pod [38;5;243m12/22/22 17:32:08.138[0m ... skipping 53 lines ... Dec 22 17:35:05.813: INFO: At 2022-12-22 17:21:45 +0000 UTC - event for pod-c04ee946-2188-47e7-b34c-5a43495db240: {attachdetach-controller } SuccessfulAttachVolume: AttachVolume.Attach succeeded for volume "pvc-c3792eec-cd83-456d-8ca2-a232d22f9c78" Dec 22 17:35:05.813: INFO: At 2022-12-22 17:21:59 +0000 UTC - event for pod-c04ee946-2188-47e7-b34c-5a43495db240: {kubelet i-0c76fbcc0b95be727} Started: Started container write-pod Dec 22 17:35:05.813: INFO: At 2022-12-22 17:21:59 +0000 UTC - event for pod-c04ee946-2188-47e7-b34c-5a43495db240: {kubelet i-0c76fbcc0b95be727} Created: Created container write-pod Dec 22 17:35:05.813: INFO: At 2022-12-22 17:21:59 +0000 UTC - event for pod-c04ee946-2188-47e7-b34c-5a43495db240: {kubelet i-0c76fbcc0b95be727} Pulled: Container image "registry.k8s.io/e2e-test-images/busybox:1.29-4" already present on machine Dec 22 17:35:05.813: INFO: At 2022-12-22 17:22:03 +0000 UTC - event for pod-c04ee946-2188-47e7-b34c-5a43495db240: {kubelet i-0c76fbcc0b95be727} Killing: Stopping container write-pod Dec 22 17:35:05.813: INFO: At 2022-12-22 17:22:07 +0000 UTC - event for csi-hostpathnj7bd: {volume_expand } ExternalExpanding: Ignoring the PVC: didn't find a plugin capable of expanding the volume; waiting for an external controller to process this PVC. Dec 22 17:35:05.813: INFO: At 2022-12-22 17:22:08 +0000 UTC - event for csi-hostpathnj7bd: {external-resizer csi-hostpath-volume-expand-7749 } VolumeResizeFailed: resize volume "pvc-c3792eec-cd83-456d-8ca2-a232d22f9c78" by resizer "csi-hostpath-volume-expand-7749" failed: rpc error: code = NotFound desc = volume id 1b0ede81-821d-11ed-8015-0a72bff03321 does not exist in the volumes list Dec 22 17:35:05.813: INFO: At 2022-12-22 17:22:08 +0000 UTC - event for csi-hostpathnj7bd: {external-resizer csi-hostpath-volume-expand-7749 } Resizing: External resizer is resizing volume pvc-c3792eec-cd83-456d-8ca2-a232d22f9c78 Dec 22 17:35:05.813: INFO: At 2022-12-22 17:22:53 +0000 UTC - event for csi-hostpathnj7bd: {external-resizer csi-hostpath-volume-expand-7749 } VolumeResizeFailed: resize volume "pvc-c3792eec-cd83-456d-8ca2-a232d22f9c78" by resizer "csi-hostpath-volume-expand-7749" failed: rpc error: code = NotFound desc = volume id 1b0ede81-821d-11ed-8015-0a72bff03321 does not exist in the volumes list Dec 22 17:35:05.813: INFO: At 2022-12-22 17:22:53 +0000 UTC - event for csi-hostpathnj7bd: {external-resizer csi-hostpath-volume-expand-7749 } Resizing: External resizer is resizing volume pvc-c3792eec-cd83-456d-8ca2-a232d22f9c78 Dec 22 17:35:05.813: INFO: At 2022-12-22 17:23:25 +0000 UTC - event for csi-hostpathnj7bd: {external-resizer csi-hostpath-volume-expand-7749 } VolumeResizeFailed: resize volume "pvc-c3792eec-cd83-456d-8ca2-a232d22f9c78" by resizer "csi-hostpath-volume-expand-7749" failed: rpc error: code = NotFound desc = volume id 1b0ede81-821d-11ed-8015-0a72bff03321 does not exist in the volumes list Dec 22 17:35:05.813: INFO: At 2022-12-22 17:23:25 +0000 UTC - event for csi-hostpathnj7bd: {external-resizer csi-hostpath-volume-expand-7749 } Resizing: External resizer is resizing volume pvc-c3792eec-cd83-456d-8ca2-a232d22f9c78 Dec 22 17:35:05.813: INFO: At 2022-12-22 17:26:21 +0000 UTC - event for csi-hostpathnj7bd: {external-resizer csi-hostpath-volume-expand-7749 } VolumeResizeFailed: resize volume "pvc-c3792eec-cd83-456d-8ca2-a232d22f9c78" by resizer "csi-hostpath-volume-expand-7749" failed: rpc error: code = NotFound desc = volume id 1b0ede81-821d-11ed-8015-0a72bff03321 does not exist in the volumes list Dec 22 17:35:05.813: INFO: At 2022-12-22 17:26:21 +0000 UTC - event for csi-hostpathnj7bd: {external-resizer csi-hostpath-volume-expand-7749 } Resizing: External resizer is resizing volume pvc-c3792eec-cd83-456d-8ca2-a232d22f9c78 Dec 22 17:35:05.813: INFO: At 2022-12-22 17:29:30 +0000 UTC - event for csi-hostpathnj7bd: {external-resizer csi-hostpath-volume-expand-7749 } Resizing: External resizer is resizing volume pvc-c3792eec-cd83-456d-8ca2-a232d22f9c78 Dec 22 17:35:05.813: INFO: At 2022-12-22 17:29:30 +0000 UTC - event for csi-hostpathnj7bd: {external-resizer csi-hostpath-volume-expand-7749 } VolumeResizeFailed: resize volume "pvc-c3792eec-cd83-456d-8ca2-a232d22f9c78" by resizer "csi-hostpath-volume-expand-7749" failed: rpc error: code = NotFound desc = volume id 1b0ede81-821d-11ed-8015-0a72bff03321 does not exist in the volumes list Dec 22 17:35:05.877: INFO: POD NODE PHASE GRACE CONDITIONS Dec 22 17:35:05.877: INFO: Dec 22 17:35:05.942: INFO: Logging node info for node i-0243def68c064e8ef Dec 22 17:35:06.006: INFO: Node Info: &Node{ObjectMeta:{i-0243def68c064e8ef f07b8771-29eb-4944-a385-3482a3d3a315 32050 0 2022-12-22 17:17:47 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:us-west-2 failure-domain.beta.kubernetes.io/zone:us-west-2a kubernetes.io/arch:amd64 kubernetes.io/hostname:i-0243def68c064e8ef kubernetes.io/os:linux node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:us-west-2a topology.hostpath.csi/node:i-0243def68c064e8ef topology.kubernetes.io/region:us-west-2 topology.kubernetes.io/zone:us-west-2a] map[alpha.kubernetes.io/provided-node-ip:172.20.35.142 csi.volume.kubernetes.io/nodeid:{"csi-hostpath-ephemeral-4812":"i-0243def68c064e8ef","csi-mock-csi-mock-volumes-2740":"i-0243def68c064e8ef","csi-mock-csi-mock-volumes-6994":"i-0243def68c064e8ef","ebs.csi.aws.com":"i-0243def68c064e8ef"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{aws-cloud-controller-manager Update v1 2022-12-22 17:17:47 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2022-12-22 17:17:47 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kops-controller Update v1 2022-12-22 17:17:47 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{}}}} } {kube-controller-manager Update v1 2022-12-22 17:17:47 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.3.0/24\"":{}}}} } {kubelet Update v1 2022-12-22 17:17:47 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:alpha.kubernetes.io/provided-node-ip":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {kubelet Update v1 2022-12-22 17:34:49 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status} {kube-controller-manager Update v1 2022-12-22 17:35:05 +0000 UTC FieldsV1 {"f:status":{"f:volumesAttached":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.3.0/24,DoNotUseExternalID:,ProviderID:aws:///us-west-2a/i-0243def68c064e8ef,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.3.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{51527004160 0} {<nil>} 50319340Ki BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4050636800 0} {<nil>} 3955700Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{46374303668 0} {<nil>} 46374303668 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3945779200 0} {<nil>} 3853300Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-12-22 17:34:49 +0000 UTC,LastTransitionTime:2022-12-22 17:17:37 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-12-22 17:34:49 +0000 UTC,LastTransitionTime:2022-12-22 17:17:37 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-12-22 17:34:49 +0000 UTC,LastTransitionTime:2022-12-22 17:17:37 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-12-22 17:34:49 +0000 UTC,LastTransitionTime:2022-12-22 17:17:59 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.35.142,},NodeAddress{Type:ExternalIP,Address:35.90.235.118,},NodeAddress{Type:InternalDNS,Address:i-0243def68c064e8ef.us-west-2.compute.internal,},NodeAddress{Type:Hostname,Address:i-0243def68c064e8ef.us-west-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-35-90-235-118.us-west-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2c8ae28bb7834ee5f6b4c4ef144a8e,SystemUUID:ec2c8ae2-8bb7-834e-e5f6-b4c4ef144a8e,BootID:5843895c-91be-4c36-b700-9e551cb745de,KernelVersion:5.10.157-139.675.amzn2.x86_64,OSImage:Amazon Linux 2,ContainerRuntimeVersion:containerd://1.6.14,KubeletVersion:v1.26.0,KubeProxyVersion:v1.26.0,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:166719855,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/jessie-dnsutils@sha256:24aaf2626d6b27864c29de2097e8bbb840b3a414271bf7c8995e431e47d8408e registry.k8s.io/e2e-test-images/jessie-dnsutils:1.7],SizeBytes:112030336,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:dd75ec974b0a2a6f6bb47001ba09207976e625db898d1b16735528c009cb171c registry.k8s.io/etcd:3.5.6-0],SizeBytes:102542580,},ContainerImage{Names:[registry.k8s.io/kube-proxy-amd64:v1.26.0],SizeBytes:67205320,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:16bbf38c463a4223d8cfe4da12bc61010b082a79b4bb003e2d3ba3ece5dd5f9e registry.k8s.io/e2e-test-images/agnhost:2.43],SizeBytes:51706353,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nautilus@sha256:80ba6c8c44f9623f06e868a1aa66026c8ec438ad814f9ec95e9333b415fe3550 registry.k8s.io/e2e-test-images/nautilus:1.7],SizeBytes:49641698,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:3fe7acf013d1264ffded116b80a73dc129a449b0fccdb8d21af8279f2233f36e registry.k8s.io/e2e-test-images/httpd:2.4.39-4],SizeBytes:41901587,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:148b022f5c5da426fc2f3c14b5c0867e58ef05961510c84749ac1fddcb0fef22 registry.k8s.io/e2e-test-images/httpd:2.4.38-4],SizeBytes:40764257,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:10e231cf07c82dfab5141c1afe548dc734e5fc1f67665ec3982a325d1bd31a9a registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.0],SizeBytes:29725756,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:ee3b525d5b89db99da3b8eb521d9cd90cb6e9ef0fbb651e98bb37be78d36b5b8 registry.k8s.io/sig-storage/csi-provisioner:v3.3.0],SizeBytes:25491225,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:425d8f1b769398127767b06ed97ce62578a3179bcb99809ce93a1649e025ffe7 registry.k8s.io/sig-storage/csi-resizer:v1.6.0],SizeBytes:24148884,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:291334908ddf71a4661fd7f6d9d97274de8a5378a2b6fdfeb2ce73414a34f82f registry.k8s.io/sig-storage/csi-snapshotter:v6.1.0],SizeBytes:23881995,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:9a685020911e2725ad019dbce6e4a5ab93d51e3d4557f115e64343345e05781b registry.k8s.io/sig-storage/csi-attacher:v4.0.0],SizeBytes:23847201,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:92257881c1d6493cf18299a24af42330f891166560047902b8d431fb66b01af5 registry.k8s.io/sig-storage/hostpathplugin:v1.9.0],SizeBytes:18758628,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nonroot@sha256:ee9f50b3c64b174d296d91ca9f69a914ac30e59095dfb462b2b518ad28a63655 registry.k8s.io/e2e-test-images/nonroot:1.4],SizeBytes:17747885,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:933940f13b3ea0abc62e656c1aa5c5b47c04b15d71250413a6b821bd0c58b94e registry.k8s.io/sig-storage/livenessprobe:v2.7.0],SizeBytes:8688564,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:5c99cf6a02adda929b10321dbf4ecfa00d87be9ba4fb456006237d530ab4baa1 registry.k8s.io/e2e-test-images/nginx:1.14-4],SizeBytes:6978614,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf registry.k8s.io/e2e-test-images/busybox:1.29-2],SizeBytes:732424,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:2e0f836850e09b8b7cc937681d6194537a09fbd5f6b9e08f4d646a85128e8937 registry.k8s.io/e2e-test-images/busybox:1.29-4],SizeBytes:731990,},ContainerImage{Names:[registry.k8s.io/pause@sha256:7031c1b283388d2c2e09b57badb803c05ebed362dc88d84b480cc47f72a21097 registry.k8s.io/pause:3.9],SizeBytes:321520,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[kubernetes.io/csi/csi-hostpath-ephemeral-4812^dc01dbbc-821e-11ed-ad8f-7ee465c920e0 kubernetes.io/csi/csi-hostpath-ephemeral-4812^e6bec19c-821e-11ed-ad8f-7ee465c920e0 kubernetes.io/csi/csi-mock-csi-mock-volumes-2740^b7bf8920-821e-11ed-816b-9e5d9f1f7545 kubernetes.io/csi/csi-mock-csi-mock-volumes-6994^e8ac2db6-821e-11ed-9d9b-2681c30d5db4 kubernetes.io/csi/ebs.csi.aws.com^vol-021017c81d529b622],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-021017c81d529b622,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/csi-mock-csi-mock-volumes-6994^e8ac2db6-821e-11ed-9d9b-2681c30d5db4,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/csi-hostpath-ephemeral-4812^e6bec19c-821e-11ed-ad8f-7ee465c920e0,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/csi-hostpath-ephemeral-4812^dc01dbbc-821e-11ed-ad8f-7ee465c920e0,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/csi-mock-csi-mock-volumes-2740^b7bf8920-821e-11ed-816b-9e5d9f1f7545,DevicePath:,},},Config:nil,},} Dec 22 17:35:06.007: INFO: ... skipping 307 lines ... Dec 22 17:35:17.643: INFO: At 2022-12-22 17:21:45 +0000 UTC - event for csi-hostpathplugin-0: {kubelet i-0c76fbcc0b95be727} Started: Started container csi-resizer Dec 22 17:35:17.643: INFO: At 2022-12-22 17:21:45 +0000 UTC - event for csi-hostpathplugin-0: {kubelet i-0c76fbcc0b95be727} Pulling: Pulling image "registry.k8s.io/sig-storage/csi-snapshotter:v6.1.0" Dec 22 17:35:17.643: INFO: At 2022-12-22 17:21:45 +0000 UTC - event for csi-hostpathplugin-0: {kubelet i-0c76fbcc0b95be727} Created: Created container csi-resizer Dec 22 17:35:17.643: INFO: At 2022-12-22 17:21:47 +0000 UTC - event for csi-hostpathplugin-0: {kubelet i-0c76fbcc0b95be727} Pulled: Successfully pulled image "registry.k8s.io/sig-storage/csi-snapshotter:v6.1.0" in 157.738686ms (1.743248205s including waiting) Dec 22 17:35:17.643: INFO: At 2022-12-22 17:21:47 +0000 UTC - event for csi-hostpathplugin-0: {kubelet i-0c76fbcc0b95be727} Created: Created container csi-snapshotter Dec 22 17:35:17.643: INFO: At 2022-12-22 17:21:47 +0000 UTC - event for csi-hostpathplugin-0: {kubelet i-0c76fbcc0b95be727} Started: Started container csi-snapshotter Dec 22 17:35:17.643: INFO: At 2022-12-22 17:21:54 +0000 UTC - event for csi-hostpathplugin-0: {kubelet i-0c76fbcc0b95be727} Unhealthy: Liveness probe failed: Get "http://172.20.44.56:9898/healthz": context deadline exceeded (Client.Timeout exceeded while awaiting headers) Dec 22 17:35:17.643: INFO: At 2022-12-22 17:23:51 +0000 UTC - event for csi-hostpathplugin-0: {kubelet i-0c76fbcc0b95be727} BackOff: Back-off restarting failed container hostpath in pod csi-hostpathplugin-0_volume-expand-7749-6599(a3f530f3-9976-4ea3-aa1c-312e3e4b2b37) Dec 22 17:35:17.707: INFO: POD NODE PHASE GRACE CONDITIONS Dec 22 17:35:17.707: INFO: csi-hostpathplugin-0 i-0c76fbcc0b95be727 Running 30s [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2022-12-22 17:21:38 +0000 UTC } {Ready True 0001-01-01 00:00:00 +0000 UTC 2022-12-22 17:35:03 +0000 UTC } {ContainersReady True 0001-01-01 00:00:00 +0000 UTC 2022-12-22 17:35:03 +0000 UTC } {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2022-12-22 17:21:38 +0000 UTC }] Dec 22 17:35:17.707: INFO: Dec 22 17:35:17.707: INFO: csi-hostpathplugin-0[volume-expand-7749-6599].container[csi-provisioner]=Lost connection to CSI driver, exiting Dec 22 17:35:17.707: INFO: csi-hostpathplugin-0[volume-expand-7749-6599].container[csi-resizer]=Lost connection to CSI driver, exiting Dec 22 17:35:18.252: INFO: ... skipping 250 lines ... dump namespaces | framework.go:196 [1mSTEP:[0m dump namespace information after failure [38;5;243m12/22/22 17:35:27.038[0m [DeferCleanup (Each)] [Testpattern: Dynamic PV (default fs)(allowExpansion)] volume-expand tear down framework | framework.go:193 [38;5;243m<< End Captured GinkgoWriter Output[0m [38;5;9mDec 22 17:32:08.074: While waiting for pvc resize to finish: error while waiting for controller resize to finish: timed out waiting for the condition[0m [38;5;9mIn [1m[It][0m[38;5;9m at: [1mtest/e2e/storage/testsuites/volume_expand.go:218[0m [38;5;243m------------------------------[0m [38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m [38;5;243m------------------------------[0m [38;5;10m• [SLOW TEST] [65.149 seconds][0m [0m[sig-apps] Job [38;5;243mUsing a pod failure policy to not count some failures towards the backoffLimit [0mIgnore DisruptionTarget condition[0m ... skipping 1207 lines ... [sig-storage] In-tree Volumes [38;5;243mtest/e2e/storage/utils/framework.go:23[0m [Driver: hostPath] [38;5;243mtest/e2e/storage/in_tree_volumes.go:85[0m [38;5;14m[1m[Testpattern: Dynamic PV (immediate binding)] topology [BeforeEach][0m [38;5;243mtest/e2e/storage/framework/testsuite.go:51[0m should fail to schedule a pod which has topologies that conflict with AllowedTopologies [38;5;243mtest/e2e/storage/testsuites/topology.go:191[0m [38;5;243mBegin Captured GinkgoWriter Output >>[0m [BeforeEach] [Testpattern: Dynamic PV (immediate binding)] topology test/e2e/storage/framework/testsuite.go:51 Dec 22 17:36:25.147: INFO: Driver hostPath doesn't support DynamicPV -- skipping ... skipping 44147 lines ... I1222 17:57:44.538016 6685 runners.go:193] Pod cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-s2fqk i-0243def68c064e8ef Pending <nil> I1222 17:57:44.538023 6685 runners.go:193] Pod cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-snhmt i-0ea878d3671f2eda2 Running <nil> I1222 17:57:44.538032 6685 runners.go:193] Pod cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-sx4vk i-0ea878d3671f2eda2 Running <nil> I1222 17:57:44.538038 6685 runners.go:193] Pod cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-t5jgm i-076110ae730e268e3 Running <nil> I1222 17:57:44.538045 6685 runners.go:193] Pod cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-t6sbf i-0ea878d3671f2eda2 Running <nil> I1222 17:57:44.538051 6685 runners.go:193] Pod cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-xjw6k i-076110ae730e268e3 Running <nil> Dec 22 17:57:44.538: INFO: Unexpected error: <*errors.errorString | 0xc0016a5940>: { s: "only 38 pods started out of 40", } Dec 22 17:57:44.538: FAIL: only 38 pods started out of 40 Full Stack Trace k8s.io/kubernetes/test/e2e/node.glob..func5.2.3() test/e2e/node/kubelet.go:359 +0x390 [AfterEach] Clean up pods on node test/e2e/node/kubelet.go:330 ... skipping 63 lines ... Dec 22 17:57:45.404: INFO: At 2022-12-22 17:39:34 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-xjw6k: {default-scheduler } Scheduled: Successfully assigned kubelet-1830/cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-xjw6k to i-076110ae730e268e3 Dec 22 17:57:45.404: INFO: At 2022-12-22 17:39:35 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b: {replication-controller } SuccessfulCreate: (combined from similar events): Created pod: cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-dnwkk Dec 22 17:57:45.404: INFO: At 2022-12-22 17:39:35 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-7rlxn: {default-scheduler } Scheduled: Successfully assigned kubelet-1830/cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-7rlxn to i-076110ae730e268e3 Dec 22 17:57:45.404: INFO: At 2022-12-22 17:39:35 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-7trm8: {kubelet i-0ea878d3671f2eda2} Started: Started container cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b Dec 22 17:57:45.404: INFO: At 2022-12-22 17:39:35 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-7trm8: {kubelet i-0ea878d3671f2eda2} Pulled: Container image "registry.k8s.io/pause:3.9" already present on machine Dec 22 17:57:45.404: INFO: At 2022-12-22 17:39:35 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-7trm8: {kubelet i-0ea878d3671f2eda2} Created: Created container cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b Dec 22 17:57:45.404: INFO: At 2022-12-22 17:39:35 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-cv9l5: {kubelet i-0243def68c064e8ef} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "035a00e7bf8d0fa7dad627e7bdc1f3c74cf814e465786a018314e6e071c742cb": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.404: INFO: At 2022-12-22 17:39:35 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-d9wnv: {kubelet i-0c76fbcc0b95be727} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "c3cc620c1a38c0c2b3bbc2d3d7c1f710e26ea1bac3729de78cdc0e67d8266265": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.404: INFO: At 2022-12-22 17:39:35 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-dzw7n: {default-scheduler } Scheduled: Successfully assigned kubelet-1830/cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-dzw7n to i-0ea878d3671f2eda2 Dec 22 17:57:45.404: INFO: At 2022-12-22 17:39:35 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-fc9kd: {kubelet i-0c76fbcc0b95be727} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "c19ed7a0860d4d1cc9410168c6d156cca1a48a96598bce1e21b98e0b62986761": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.404: INFO: At 2022-12-22 17:39:35 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-h4bp2: {kubelet i-076110ae730e268e3} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "fcf16d1c773d23f433ca00e0da9883e27c5b6236b981ba021d8722a66faae586": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.404: INFO: At 2022-12-22 17:39:35 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-jq2ww: {kubelet i-0243def68c064e8ef} Started: Started container cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b Dec 22 17:57:45.404: INFO: At 2022-12-22 17:39:35 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-jq2ww: {kubelet i-0243def68c064e8ef} Pulled: Container image "registry.k8s.io/pause:3.9" already present on machine Dec 22 17:57:45.404: INFO: At 2022-12-22 17:39:35 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-jq2ww: {kubelet i-0243def68c064e8ef} Created: Created container cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b Dec 22 17:57:45.404: INFO: At 2022-12-22 17:39:35 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-lj45n: {default-scheduler } Scheduled: Successfully assigned kubelet-1830/cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-lj45n to i-0c76fbcc0b95be727 Dec 22 17:57:45.404: INFO: At 2022-12-22 17:39:35 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-n2hcq: {kubelet i-076110ae730e268e3} Started: Started container cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b Dec 22 17:57:45.404: INFO: At 2022-12-22 17:39:35 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-n2hcq: {kubelet i-076110ae730e268e3} Pulled: Container image "registry.k8s.io/pause:3.9" already present on machine Dec 22 17:57:45.404: INFO: At 2022-12-22 17:39:35 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-n2hcq: {kubelet i-076110ae730e268e3} Created: Created container cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b Dec 22 17:57:45.404: INFO: At 2022-12-22 17:39:35 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-n5cfc: {kubelet i-0c76fbcc0b95be727} Started: Started container cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b Dec 22 17:57:45.404: INFO: At 2022-12-22 17:39:35 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-n5cfc: {kubelet i-0c76fbcc0b95be727} Created: Created container cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b Dec 22 17:57:45.404: INFO: At 2022-12-22 17:39:35 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-n5cfc: {kubelet i-0c76fbcc0b95be727} Pulled: Container image "registry.k8s.io/pause:3.9" already present on machine Dec 22 17:57:45.404: INFO: At 2022-12-22 17:39:35 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-nt4t4: {kubelet i-0243def68c064e8ef} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "91fde814a782ffe85132b7c41fdcd11869587529aa66a37c5bba8e2564510cfb": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.404: INFO: At 2022-12-22 17:39:35 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-qkk8s: {kubelet i-0c76fbcc0b95be727} Pulled: Container image "registry.k8s.io/pause:3.9" already present on machine Dec 22 17:57:45.404: INFO: At 2022-12-22 17:39:35 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-qkk8s: {kubelet i-0c76fbcc0b95be727} Created: Created container cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b Dec 22 17:57:45.404: INFO: At 2022-12-22 17:39:35 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-qr2nz: {kubelet i-076110ae730e268e3} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "f4eb357e40bea4c804ae85edee551f019e96af85f6161ee01fe00a00a9c3e18b": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.404: INFO: At 2022-12-22 17:39:35 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-t5jgm: {kubelet i-076110ae730e268e3} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "37a9b2127b8ed57c5356fd6c8bd4a0451b10b04e6e1631a9d8003a7df065dfaa": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.404: INFO: At 2022-12-22 17:39:36 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-2pn4l: {kubelet i-076110ae730e268e3} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "f51a1789300cd612e75922d45ddee34c23e0d5dae4d947770c290412329526c8": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.404: INFO: At 2022-12-22 17:39:36 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-2sdxg: {kubelet i-0ea878d3671f2eda2} Created: Created container cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b Dec 22 17:57:45.404: INFO: At 2022-12-22 17:39:36 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-2sdxg: {kubelet i-0ea878d3671f2eda2} Pulled: Container image "registry.k8s.io/pause:3.9" already present on machine Dec 22 17:57:45.404: INFO: At 2022-12-22 17:39:36 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-4tjzb: {kubelet i-0ea878d3671f2eda2} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "0d1ed6c9041fbcff5f286e9d699a6fb5cce45489abe515bc8af42ff29bc6798a": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.404: INFO: At 2022-12-22 17:39:36 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-6772c: {kubelet i-0243def68c064e8ef} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "12eb3f26e4357e9205def3705b88c7c2e2e75c95e46a39f0edfe5432d88bb0d8": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.404: INFO: At 2022-12-22 17:39:36 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-698nb: {kubelet i-0c76fbcc0b95be727} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "5a2ece9fd80ae6e7a1df27555b163a9fea459d1d73bc42f6811e97ceb45c6f3d": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.404: INFO: At 2022-12-22 17:39:36 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-7bxcw: {kubelet i-0c76fbcc0b95be727} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "00c2375d701f4d270806c84b901b720bc25d774df97a1a9eb1764b4e02eebcf0": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.404: INFO: At 2022-12-22 17:39:36 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-7fg7v: {kubelet i-0c76fbcc0b95be727} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "4b6155093d3dab152967672b0ab89aed9d09879d82e9480072fac1fc10c248b4": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.404: INFO: At 2022-12-22 17:39:36 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-87xst: {kubelet i-076110ae730e268e3} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "a1a2fb5dfaf68c7daaee24be6c75b6213d9bfa6cdda175edcaff24a0b6894299": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.404: INFO: At 2022-12-22 17:39:36 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-89z57: {kubelet i-0243def68c064e8ef} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "6ff4df0dd9c7f6ad54a3430156542901839d0667dd41cf52b03e1b6033a0b8b9": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.404: INFO: At 2022-12-22 17:39:36 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-8dmtf: {kubelet i-0ea878d3671f2eda2} Pulled: Container image "registry.k8s.io/pause:3.9" already present on machine Dec 22 17:57:45.404: INFO: At 2022-12-22 17:39:36 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-8dmtf: {kubelet i-0ea878d3671f2eda2} Created: Created container cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b Dec 22 17:57:45.404: INFO: At 2022-12-22 17:39:36 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-8dmtf: {kubelet i-0ea878d3671f2eda2} Started: Started container cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b Dec 22 17:57:45.404: INFO: At 2022-12-22 17:39:36 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-bk7bk: {kubelet i-0243def68c064e8ef} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "4fe2fc4091506d509c19fbc4b54ea46f8a0825b4d1f92ed2fe58452e52a3f9c2": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.404: INFO: At 2022-12-22 17:39:36 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-dnwkk: {kubelet i-076110ae730e268e3} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "ed3030f33af1f0ec4c6d8ac832793ef6cd5a789e8f4dd2d938c205ef8f0d65ee": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.404: INFO: At 2022-12-22 17:39:36 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-dtkw2: {kubelet i-0c76fbcc0b95be727} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "5dd3f2ea0a93658c9a5d5b74792681871998cac94d51f154eea5e77a7d2aa472": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.404: INFO: At 2022-12-22 17:39:36 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-hxxdw: {kubelet i-0ea878d3671f2eda2} Pulled: Container image "registry.k8s.io/pause:3.9" already present on machine Dec 22 17:57:45.404: INFO: At 2022-12-22 17:39:36 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-hxxdw: {kubelet i-0ea878d3671f2eda2} Created: Created container cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b Dec 22 17:57:45.404: INFO: At 2022-12-22 17:39:36 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-jdnkw: {kubelet i-076110ae730e268e3} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "96f38f190bd6340865068cd97bf43647fdb4d271cfb46ef8ce6cab587975f63d": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.404: INFO: At 2022-12-22 17:39:36 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-nxx2s: {kubelet i-0c76fbcc0b95be727} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "57d4f1afa6fafab2dde398c005d10c623411533ff265f16ec1afb3df7e16401d": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.404: INFO: At 2022-12-22 17:39:36 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-qkk8s: {kubelet i-0c76fbcc0b95be727} Started: Started container cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b Dec 22 17:57:45.404: INFO: At 2022-12-22 17:39:36 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-rnswh: {kubelet i-0ea878d3671f2eda2} Created: Created container cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b Dec 22 17:57:45.404: INFO: At 2022-12-22 17:39:36 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-rnswh: {kubelet i-0ea878d3671f2eda2} Pulled: Container image "registry.k8s.io/pause:3.9" already present on machine Dec 22 17:57:45.404: INFO: At 2022-12-22 17:39:36 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-rnswh: {kubelet i-0ea878d3671f2eda2} Started: Started container cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b Dec 22 17:57:45.404: INFO: At 2022-12-22 17:39:36 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-s2fqk: {kubelet i-0243def68c064e8ef} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "7bdaccc2de9ab0898fccf6ba23478b58951ea41764886debe93af02532380966": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.404: INFO: At 2022-12-22 17:39:36 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-snhmt: {kubelet i-0ea878d3671f2eda2} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "2c954ccce48038872e114161e79c50b6c2b38bce94c227ca34e1c1c82c295ad9": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.404: INFO: At 2022-12-22 17:39:36 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-sx4vk: {kubelet i-0ea878d3671f2eda2} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "562159739f160afc3b7b680bb2f18554f9ce149aa32a9584f7642d27165bf979": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.404: INFO: At 2022-12-22 17:39:36 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-t6sbf: {kubelet i-0ea878d3671f2eda2} Created: Created container cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b Dec 22 17:57:45.404: INFO: At 2022-12-22 17:39:36 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-t6sbf: {kubelet i-0ea878d3671f2eda2} Pulled: Container image "registry.k8s.io/pause:3.9" already present on machine Dec 22 17:57:45.404: INFO: At 2022-12-22 17:39:36 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-xjw6k: {kubelet i-076110ae730e268e3} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "fd7e628fdb9798f8df63a1d60b3cfc1dd5fb8d3a80196fa47721cf51b676f26f": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.404: INFO: At 2022-12-22 17:39:37 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-2sdxg: {kubelet i-0ea878d3671f2eda2} Started: Started container cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b Dec 22 17:57:45.404: INFO: At 2022-12-22 17:39:37 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-7rlxn: {kubelet i-076110ae730e268e3} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "3fb15496f82ec603ecb2f1c6486eb8e38727349cda54adae7af9a73efe152757": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.404: INFO: At 2022-12-22 17:39:37 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-dzw7n: {kubelet i-0ea878d3671f2eda2} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "d573739811a49b3aba9f74b15fea42ca523ed386c500614252e5a128cbe45633": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.404: INFO: At 2022-12-22 17:39:37 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-hxxdw: {kubelet i-0ea878d3671f2eda2} Started: Started container cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b Dec 22 17:57:45.404: INFO: At 2022-12-22 17:39:37 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-lh7g8: {kubelet i-0243def68c064e8ef} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "bfd8d7390610d35bc7e80a2d949575ae574d4fa63623e1cd3ba2bc1c57c7a461": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.404: INFO: At 2022-12-22 17:39:37 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-mttmd: {kubelet i-0243def68c064e8ef} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "a7527255c92c68b524c375de3d13dcb1eeb20ec8761ef2109de4479deb678e20": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.404: INFO: At 2022-12-22 17:39:37 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-qrnbk: {kubelet i-0243def68c064e8ef} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "ea89c050703d4bd36b301dbc567722c9d17c8e5c090765883a81f56299f18aac": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.404: INFO: At 2022-12-22 17:39:37 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-t6sbf: {kubelet i-0ea878d3671f2eda2} Started: Started container cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b Dec 22 17:57:45.404: INFO: At 2022-12-22 17:39:38 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-lj45n: {kubelet i-0c76fbcc0b95be727} Started: Started container cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b Dec 22 17:57:45.404: INFO: At 2022-12-22 17:39:38 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-lj45n: {kubelet i-0c76fbcc0b95be727} Created: Created container cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b Dec 22 17:57:45.404: INFO: At 2022-12-22 17:39:38 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-lj45n: {kubelet i-0c76fbcc0b95be727} Pulled: Container image "registry.k8s.io/pause:3.9" already present on machine Dec 22 17:57:45.404: INFO: At 2022-12-22 17:39:47 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-89z57: {kubelet i-0243def68c064e8ef} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "58231a137b008d6d5356b7acb3f6c873b3f863276248c431b0b88c01b4f13178": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.404: INFO: At 2022-12-22 17:39:47 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-xjw6k: {kubelet i-076110ae730e268e3} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "7df2ec1fadd1c880f4f8727da5df7f26b15fe87f9f69f21b154426a35cd24245": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.404: INFO: At 2022-12-22 17:39:48 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-bk7bk: {kubelet i-0243def68c064e8ef} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "c8e0bc480dcb35e7eb670820dbe810af8f021b16260f54f15d62d9153b2019fb": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.404: INFO: At 2022-12-22 17:39:48 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-d9wnv: {kubelet i-0c76fbcc0b95be727} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "01dd2343a33e38a6eee555c52dcff49bd2327dec1fd2232ca9628436052204fb": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.404: INFO: At 2022-12-22 17:39:48 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-h4bp2: {kubelet i-076110ae730e268e3} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "865ee23bb524e707fe3980934058f1273244cf912e66ac55008b02c4c367f81c": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.404: INFO: At 2022-12-22 17:39:48 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-jdnkw: {kubelet i-076110ae730e268e3} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "9c22f982f5a448d88acec3328e9f686825cceac78f6ef87bdc8c2040593d98c3": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.404: INFO: At 2022-12-22 17:39:48 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-lh7g8: {kubelet i-0243def68c064e8ef} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "cda09ac45dfb5cb2d6cb8d88b78e78b702076be316a8b660c4c6f7b1100eebeb": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.404: INFO: At 2022-12-22 17:39:48 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-sx4vk: {kubelet i-0ea878d3671f2eda2} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "a2f76d727d670d117afeb742a7c3810871cf5f595bf2043ee1ad1cea49d258f1": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.404: INFO: At 2022-12-22 17:39:48 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-t5jgm: {kubelet i-076110ae730e268e3} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "0d632a43cfbcd6cd4f3da8ff0f6c0eb304018095d174eeced9aa488f700f921c": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.404: INFO: At 2022-12-22 17:39:49 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-4tjzb: {kubelet i-0ea878d3671f2eda2} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "de3b53c3b220ac985ab968e745d55f251e00343ca361676d0486beea1aa68d33": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.404: INFO: At 2022-12-22 17:39:49 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-698nb: {kubelet i-0c76fbcc0b95be727} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "74c088be868bfaf8ba1dbde13f5269254f0dba9447272fa104d63ff17394463f": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.404: INFO: At 2022-12-22 17:39:49 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-7bxcw: {kubelet i-0c76fbcc0b95be727} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "d8038074b3cf8b85dda32d531f406de0e227f6f4b6e1b5796dbe2342d928958f": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.404: INFO: At 2022-12-22 17:39:49 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-nt4t4: {kubelet i-0243def68c064e8ef} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "e3711bb7f18c9282b4564c30c50000ff1635db9fb5443cc420582d0592c181d8": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.404: INFO: At 2022-12-22 17:39:49 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-nxx2s: {kubelet i-0c76fbcc0b95be727} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "25ddf29b17bd80d8b6c1e0d374f2d0541b7c88b9f27e926f77c4159a9d9518a2": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.404: INFO: At 2022-12-22 17:39:49 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-s2fqk: {kubelet i-0243def68c064e8ef} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "d019625b1c036220e8dcfdddba6c1c437fb4fa6e601acd309e5ed42f123ba837": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.404: INFO: At 2022-12-22 17:39:49 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-snhmt: {kubelet i-0ea878d3671f2eda2} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "f61496b56de21b0d0c95c7cc03e115a191385e037f9112a550f77e8fc30b138c": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.404: INFO: At 2022-12-22 17:39:50 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-6772c: {kubelet i-0243def68c064e8ef} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "5abdc0e523ffc9a091fda0b594c54670ab7a6f397707a43925bc4170be82403d": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.404: INFO: At 2022-12-22 17:39:50 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-7fg7v: {kubelet i-0c76fbcc0b95be727} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "7ead69dde606ec0dea01bea4f3c7ace738bb9b3fe7891a4cb4012d86607f6dc9": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.404: INFO: At 2022-12-22 17:39:50 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-dnwkk: {kubelet i-076110ae730e268e3} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "f72c1c0267baacc29f5057b42f13f2989db42b239670feadef6cb8d8a77a4438": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.404: INFO: At 2022-12-22 17:39:50 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-qr2nz: {kubelet i-076110ae730e268e3} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "dcce986101b9af24bc94d3e2f79b85a6f90c421f4d44a58f643ec430158666b9": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.404: INFO: At 2022-12-22 17:39:51 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-cv9l5: {kubelet i-0243def68c064e8ef} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "eab1f85a075bb9c6cf39e98a8ddb5005e3ab972fabd8cb86bdc3e2f6aa83bb80": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.404: INFO: At 2022-12-22 17:39:51 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-dtkw2: {kubelet i-0c76fbcc0b95be727} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "7806b36c0eaa12cc79c1ba638352d1a36c34c73b18f026cf4a28cd6c637a4764": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.404: INFO: At 2022-12-22 17:39:51 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-dzw7n: {kubelet i-0ea878d3671f2eda2} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "63765a64087d14542627fbb9ba222efe4f5df92a8f4ee3816a59a0cc963abd9c": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.404: INFO: At 2022-12-22 17:39:51 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-fc9kd: {kubelet i-0c76fbcc0b95be727} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "b32099972d42c10ca39382757052ffb923d854c76a4334aad8ac80e4d2b30c5b": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.404: INFO: At 2022-12-22 17:39:51 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-mttmd: {kubelet i-0243def68c064e8ef} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "a7e88610cc2919eb1b46c66a4ebfff3b32108fdaa19986648b84cb3792a919f4": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.404: INFO: At 2022-12-22 17:39:52 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-2pn4l: {kubelet i-076110ae730e268e3} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "213900268e32c9b693d8e0dab243cba808dbff5beb8b424a7b666f3b7d837306": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.404: INFO: At 2022-12-22 17:39:52 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-7rlxn: {kubelet i-076110ae730e268e3} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "9d7172c760197b43fd8194cf7a0fcde2662abc6c3e86b0b844eab6c4eef435ef": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.404: INFO: At 2022-12-22 17:39:52 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-87xst: {kubelet i-076110ae730e268e3} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "130cebd477a0d68ec98fb6c21b3d13f4f231f43f4c63d2e7cf9e4f6fdc74c0cf": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.404: INFO: At 2022-12-22 17:39:52 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-qrnbk: {kubelet i-0243def68c064e8ef} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "e6867a4dac047dad4afa4cd0e6aa5a9968be97033c7a68827fbbc05584bfe0f5": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.404: INFO: At 2022-12-22 17:39:59 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-d9wnv: {kubelet i-0c76fbcc0b95be727} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "b3f6d1e8c4c03e55216695b9761cd71e95fed1cbcff4528a3abbd70a76daae98": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.404: INFO: At 2022-12-22 17:39:59 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-jdnkw: {kubelet i-076110ae730e268e3} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "a83e59b9b042094fcfba9c51f8bcd92bf4e0082585906f25068f962fbe443539": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.404: INFO: At 2022-12-22 17:39:59 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-lh7g8: {kubelet i-0243def68c064e8ef} Pulled: Container image "registry.k8s.io/pause:3.9" already present on machine Dec 22 17:57:45.404: INFO: At 2022-12-22 17:39:59 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-lh7g8: {kubelet i-0243def68c064e8ef} Created: Created container cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b Dec 22 17:57:45.404: INFO: At 2022-12-22 17:39:59 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-lh7g8: {kubelet i-0243def68c064e8ef} Started: Started container cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b Dec 22 17:57:45.404: INFO: At 2022-12-22 17:40:00 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-698nb: {kubelet i-0c76fbcc0b95be727} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "adc589c482df6bb47a8143a2a56a755237acf94a59c897d14576335fbda122e8": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.404: INFO: At 2022-12-22 17:40:00 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-7bxcw: {kubelet i-0c76fbcc0b95be727} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "8875db185d83cc78485e34b74c590c5058987c893f19b6fa5d379c2594cf733a": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.404: INFO: At 2022-12-22 17:40:00 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-89z57: {kubelet i-0243def68c064e8ef} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "227170ae66f9c266936ea059889929db2a70044f897e80c959bc2718913eda64": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.404: INFO: At 2022-12-22 17:40:00 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-nxx2s: {kubelet i-0c76fbcc0b95be727} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "aa4c0fe11369943fa6f99168cdc64042ed1d10e7bc7e4e8e19942be04ace93bd": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.404: INFO: At 2022-12-22 17:40:00 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-sx4vk: {kubelet i-0ea878d3671f2eda2} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "ce0c231a3bb85a3de586c73fd1f8ce0660ce2af21f8026fe298e1910692cb775": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.404: INFO: At 2022-12-22 17:40:01 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-nt4t4: {kubelet i-0243def68c064e8ef} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "c43a5c0ad853da554aa478b8624788beb5dc763e7537e3bca09df09c01843551": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.404: INFO: At 2022-12-22 17:40:01 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-qr2nz: {kubelet i-076110ae730e268e3} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "dc3747f052f56c231cfb4f873e12e0467da38f0fa1aa55f1edced0ccf41b8dd9": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.404: INFO: At 2022-12-22 17:40:01 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-xjw6k: {kubelet i-076110ae730e268e3} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "0a8ce154af9887c377efe462b0668099f416e5c87e19c1645f7700c569013815": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.404: INFO: At 2022-12-22 17:40:02 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-4tjzb: {kubelet i-0ea878d3671f2eda2} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "a12d7b56df2cbf0c8718f1234ca5ad0503b99d190e2b43108fff42346913be99": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.404: INFO: At 2022-12-22 17:40:02 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-6772c: {kubelet i-0243def68c064e8ef} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "64f459a2f56ec7819b17811eefdc082024e0b4094d5b6c15f7ffe8caaee209a7": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.404: INFO: At 2022-12-22 17:40:02 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-bk7bk: {kubelet i-0243def68c064e8ef} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "977ee520847ec4125f9e83e316ce3f91897eb21b90abe84de2a293f404d20b03": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.404: INFO: At 2022-12-22 17:40:02 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-h4bp2: {kubelet i-076110ae730e268e3} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "f9d3c08084e098c804a3a60589372cfd86ae0020f39193b2bb63c214f8de2031": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.404: INFO: At 2022-12-22 17:40:02 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-t5jgm: {kubelet i-076110ae730e268e3} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "55fed0235247d8a3157b58310865bca7d60659f36dcf2633f03b05243d452f52": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.404: INFO: At 2022-12-22 17:40:03 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-7rlxn: {kubelet i-076110ae730e268e3} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "898e9a8f1b21843541152adebcf82ced30f897a34b964e80b2569d20a93cb597": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.404: INFO: At 2022-12-22 17:40:03 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-cv9l5: {kubelet i-0243def68c064e8ef} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "3938c71528d148c4fdb79dc1483e6de62f4f1b3257fd8d6ba4bf56de8da3d23e": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.404: INFO: At 2022-12-22 17:40:04 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-2pn4l: {kubelet i-076110ae730e268e3} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "3e16683c853e0a3a5931286fbf22b645eb4fbb8b56d000c0325da2ad9eb89a4e": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.404: INFO: At 2022-12-22 17:40:04 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-fc9kd: {kubelet i-0c76fbcc0b95be727} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "a9be085ea202213cdb795e3f5967f63b80d78aa65b4abd0fc17d6d6d59c03de3": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.404: INFO: At 2022-12-22 17:40:04 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-snhmt: {kubelet i-0ea878d3671f2eda2} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "fab2280e15c6e7e27e07b7355d89f29ab67a646c9e0952de38159bb402184851": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.404: INFO: At 2022-12-22 17:40:05 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-7fg7v: {kubelet i-0c76fbcc0b95be727} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "25d28a787a1527db16d7a654c082622bfd3e515928a25a8efb802ef3c790a525": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.404: INFO: At 2022-12-22 17:40:05 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-87xst: {kubelet i-076110ae730e268e3} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "29e128cacc5badd614fa6d7754b2fbce161fe10fcf29d53d4637e460fdc04e17": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.404: INFO: At 2022-12-22 17:40:05 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-dnwkk: {kubelet i-076110ae730e268e3} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "5f0ee14c4ad3324510814b48822c6492dbc44d8b437b538e1d644ccd391a9139": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.404: INFO: At 2022-12-22 17:40:05 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-s2fqk: {kubelet i-0243def68c064e8ef} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "439286e3cef99fac3d5e1b690895f5ce4e4bfa5b7b857c2138458dccfda054b7": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.404: INFO: At 2022-12-22 17:40:06 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-dtkw2: {kubelet i-0c76fbcc0b95be727} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "d175b92d4906bf9ed85847478b3da2658bb6eca46a56f9ba540c3275fecbe39e": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.404: INFO: At 2022-12-22 17:40:06 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-dzw7n: {kubelet i-0ea878d3671f2eda2} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "eb6965ff3dff1ff57ec0dbd20f0e97f7415d43de97d28e83cc3f3d6dfb017e00": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.404: INFO: At 2022-12-22 17:40:06 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-mttmd: {kubelet i-0243def68c064e8ef} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "ed94043866a28784a94bf4348c2bb45d892c26234fd063970a48651be518e17b": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.404: INFO: At 2022-12-22 17:40:07 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-qrnbk: {kubelet i-0243def68c064e8ef} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "cb9802b29ac7f47e27dc3123e2234f69773479e88ba135b25df5771607a15fe9": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.404: INFO: At 2022-12-22 17:40:12 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-sx4vk: {kubelet i-0ea878d3671f2eda2} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "eaef06676875a127d1e71925303c1a7864ef6ab8ce5e54be7fdab69efb3644aa": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.404: INFO: At 2022-12-22 17:40:13 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-jdnkw: {kubelet i-076110ae730e268e3} Started: Started container cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b Dec 22 17:57:45.404: INFO: At 2022-12-22 17:40:13 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-jdnkw: {kubelet i-076110ae730e268e3} Pulled: Container image "registry.k8s.io/pause:3.9" already present on machine Dec 22 17:57:45.404: INFO: At 2022-12-22 17:40:13 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-jdnkw: {kubelet i-076110ae730e268e3} Created: Created container cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b Dec 22 17:57:45.404: INFO: At 2022-12-22 17:40:13 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-qr2nz: {kubelet i-076110ae730e268e3} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "99e8815fdeeb66171269e396b4786f293d0edfa5edb50d3ec94b08a4ad5c5b35": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.404: INFO: At 2022-12-22 17:40:14 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-6772c: {kubelet i-0243def68c064e8ef} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "5b89074b65ddd06beaff5b61f4540ef02e78bc202613564cbd05cf4ab4f93820": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.405: INFO: At 2022-12-22 17:40:14 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-698nb: {kubelet i-0c76fbcc0b95be727} Started: Started container cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b Dec 22 17:57:45.405: INFO: At 2022-12-22 17:40:14 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-698nb: {kubelet i-0c76fbcc0b95be727} Created: Created container cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b Dec 22 17:57:45.405: INFO: At 2022-12-22 17:40:14 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-698nb: {kubelet i-0c76fbcc0b95be727} Pulled: Container image "registry.k8s.io/pause:3.9" already present on machine Dec 22 17:57:45.405: INFO: At 2022-12-22 17:40:14 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-d9wnv: {kubelet i-0c76fbcc0b95be727} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "fefdd225a705a5e612a60980078b32fab86dfb0306e510f8bee2a93d749b4286": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.405: INFO: At 2022-12-22 17:40:14 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-t5jgm: {kubelet i-076110ae730e268e3} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "6d70baf567b2de91e0e52ae7bbfbbbb783274b24cadea3e1c3fc4bb2683510cd": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.405: INFO: At 2022-12-22 17:40:14 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-xjw6k: {kubelet i-076110ae730e268e3} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "1330264f31a88f7ec39bae2cfe150f5fc6d60d32b6cc71f1c94eea3bc9bf637e": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.405: INFO: At 2022-12-22 17:40:15 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-7bxcw: {kubelet i-0c76fbcc0b95be727} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "cc7981c77ee00ddcbc9a3b7186c38c6ab6e0c256f4cdcc9db6a72e72ff74583b": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.405: INFO: At 2022-12-22 17:40:15 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-89z57: {kubelet i-0243def68c064e8ef} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "2882ef65e27e829fd9f218f8061904684575d7897ea5d9aa836bcd3d056fb93e": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.405: INFO: At 2022-12-22 17:40:15 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-h4bp2: {kubelet i-076110ae730e268e3} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "514ac663b74364d4309e04e12073d9cb84d84c1b35fd44d35a476f233e4312d1": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.405: INFO: At 2022-12-22 17:40:15 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-nxx2s: {kubelet i-0c76fbcc0b95be727} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "01e7f3be9c4f515a0ed543c24ffb3dcd213148a34bebc68853f70a31f7c4a497": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.405: INFO: At 2022-12-22 17:40:15 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-snhmt: {kubelet i-0ea878d3671f2eda2} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "a2279d95cfc6a4c29895d035df56765b4a1b7b5a11b1989d7cf69c9aff30044d": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.405: INFO: At 2022-12-22 17:40:16 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-2pn4l: {kubelet i-076110ae730e268e3} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "f4800d2f971380ce286ab569199d233c57212814afadf54f9523dcc481363456": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.405: INFO: At 2022-12-22 17:40:16 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-4tjzb: {kubelet i-0ea878d3671f2eda2} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "0a707c7a51501adffb7a329a182495518ce8b8cadc56a116d9bd837b65c11b7c": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.405: INFO: At 2022-12-22 17:40:16 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-nt4t4: {kubelet i-0243def68c064e8ef} Pulled: Container image "registry.k8s.io/pause:3.9" already present on machine Dec 22 17:57:45.405: INFO: At 2022-12-22 17:40:16 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-nt4t4: {kubelet i-0243def68c064e8ef} Created: Created container cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b Dec 22 17:57:45.405: INFO: At 2022-12-22 17:40:16 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-nt4t4: {kubelet i-0243def68c064e8ef} Started: Started container cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b Dec 22 17:57:45.405: INFO: At 2022-12-22 17:40:17 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-bk7bk: {kubelet i-0243def68c064e8ef} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "69ffafa867fd16313749e9b2c7b4634affe33bf633d3206fc3784d1a6f5a92ae": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.405: INFO: At 2022-12-22 17:40:17 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-dzw7n: {kubelet i-0ea878d3671f2eda2} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "fc23ef36fb80a4f4a47d27995004147282432637fbe79fb4b69c2a340179dd81": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.405: INFO: At 2022-12-22 17:40:17 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-fc9kd: {kubelet i-0c76fbcc0b95be727} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "bbe4bea14d90632917816a956ec50e3eb1c6933ba8367f0ca32466c4ca0d74ee": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.405: INFO: At 2022-12-22 17:40:17 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-s2fqk: {kubelet i-0243def68c064e8ef} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "ef0cf8d10f5ccd2800832d198be4513321f3b5fad688068ef517fccf0b32882b": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.405: INFO: At 2022-12-22 17:40:18 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-7fg7v: {kubelet i-0c76fbcc0b95be727} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "a6aeb82e631f0a570c5392090eddf398d3e21ecfd150feae4dd2125067db6be9": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.405: INFO: At 2022-12-22 17:40:18 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-7rlxn: {kubelet i-076110ae730e268e3} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "dcc1557823587555400da20bda92a5bef9c00e4d8fb120ca49de5ede1946938c": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.405: INFO: At 2022-12-22 17:40:18 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-cv9l5: {kubelet i-0243def68c064e8ef} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "b276e089ce105a19ac49cc26cbce8b3274b6ea35198cc52590d5c4eb9bab2f14": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.405: INFO: At 2022-12-22 17:40:18 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-dnwkk: {kubelet i-076110ae730e268e3} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "5991d1901fcb423f2624aa77722e1547d898b8fa24e66c5f77e4d391313151e2": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.405: INFO: At 2022-12-22 17:40:18 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-qrnbk: {kubelet i-0243def68c064e8ef} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "91b8f62d6bceb5969ac98153a0a02508ce1d4789882fcfc90855ab66dab0b36d": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.405: INFO: At 2022-12-22 17:40:19 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-dtkw2: {kubelet i-0c76fbcc0b95be727} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "3a378a06ce3b07130f3ce8b938f6ba12b73679743c25dec8fb165831967e5829": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.405: INFO: At 2022-12-22 17:40:21 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-87xst: {kubelet i-076110ae730e268e3} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "2dcb3a6d29394df89f2fa28b1b5bd993d287d8687d4f575307f7f6874ddb1e5f": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.405: INFO: At 2022-12-22 17:40:21 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-mttmd: {kubelet i-0243def68c064e8ef} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "87bb7d681433932686c3250afea910db0e39c55dfe2bbdcb4eeb70963f25f53e": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.405: INFO: At 2022-12-22 17:40:26 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-6772c: {kubelet i-0243def68c064e8ef} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "65d1d4e4702c70d66b6fdd5258db137e317d8e13775f2851cd074f3607b729c4": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.405: INFO: At 2022-12-22 17:40:26 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-qr2nz: {kubelet i-076110ae730e268e3} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "d5d74722758a9de8b3e24143645d71064b25bebbb73a9c119f710701e3a2b57f": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.405: INFO: At 2022-12-22 17:40:26 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-snhmt: {kubelet i-0ea878d3671f2eda2} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "f9a6e64578b40c9130a39dee191781e4d71a05e1b8e1ad4bbacef969cf8ca472": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.405: INFO: At 2022-12-22 17:40:27 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-2pn4l: {kubelet i-076110ae730e268e3} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "79ffe007af4678f56400bf34de02c3d0c935b375ae9eb8413c6b6cfa0101c73b": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.405: INFO: At 2022-12-22 17:40:27 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-h4bp2: {kubelet i-076110ae730e268e3} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "90da90bd3a9d69f552b07aca734093db46dec83572d83788354f35b99441a0bc": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.405: INFO: At 2022-12-22 17:40:27 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-sx4vk: {kubelet i-0ea878d3671f2eda2} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "02a456fd2a5ac7453d3659599e30d23e8a85f2b9757df3be96ed46e3cccb7a92": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.405: INFO: At 2022-12-22 17:40:27 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-xjw6k: {kubelet i-076110ae730e268e3} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "d9843e53ff5c8bd3cb29d35910a33e827a3aeac032435783285ec65782288fc2": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.405: INFO: At 2022-12-22 17:40:28 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-d9wnv: {kubelet i-0c76fbcc0b95be727} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "65f47d86986bc00dd047db4d99cc2cb073f905a03cc25d3108b074a3179ddb1b": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.405: INFO: At 2022-12-22 17:40:28 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-dzw7n: {kubelet i-0ea878d3671f2eda2} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "43002abbb76da11dda727e1876bc7a7ade78590e9f2e27ca4bf6e5ff0b6c7886": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.405: INFO: At 2022-12-22 17:40:28 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-t5jgm: {kubelet i-076110ae730e268e3} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "67a6911748fac648023ae7d8a17839857c6e01894c46ad2466e0f7278040f2fb": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.405: INFO: At 2022-12-22 17:40:29 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-7bxcw: {kubelet i-0c76fbcc0b95be727} Pulled: Container image "registry.k8s.io/pause:3.9" already present on machine Dec 22 17:57:45.405: INFO: At 2022-12-22 17:40:29 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-7bxcw: {kubelet i-0c76fbcc0b95be727} Started: Started container cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b Dec 22 17:57:45.405: INFO: At 2022-12-22 17:40:29 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-7bxcw: {kubelet i-0c76fbcc0b95be727} Created: Created container cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b Dec 22 17:57:45.405: INFO: At 2022-12-22 17:40:29 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-dnwkk: {kubelet i-076110ae730e268e3} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "8a7bd7d5bfb033541c77b53a20cf518a05ca260519a4ea8e073c22f4ddefbbd1": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.405: INFO: At 2022-12-22 17:40:30 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-4tjzb: {kubelet i-0ea878d3671f2eda2} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "9eb41ed60970fa2220ded11f1dbbc595f8e4eff20db67437b266dbb5f81689f6": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.405: INFO: At 2022-12-22 17:40:30 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-7rlxn: {kubelet i-076110ae730e268e3} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "07a73eb8bbe26d3591122289dc747ef2b5fe4ef6852266fa89d1cb211c5882cf": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.405: INFO: At 2022-12-22 17:40:30 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-89z57: {kubelet i-0243def68c064e8ef} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "518ec3f334eedf868154af84ff6494fd5f9cf504a98a632cf8696090ae0aa02e": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.405: INFO: At 2022-12-22 17:40:30 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-nxx2s: {kubelet i-0c76fbcc0b95be727} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "cfd7246836539f521ca6bddd5b1196b7ca149859d6f915b7d87ce9b456b279fa": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.405: INFO: At 2022-12-22 17:40:31 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-7fg7v: {kubelet i-0c76fbcc0b95be727} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "082f7cdf76c51d3d81a19f0ded3e24ac87cbe24c543f5658e8a81443130313b0": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.405: INFO: At 2022-12-22 17:40:31 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-qrnbk: {kubelet i-0243def68c064e8ef} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "76d3d098961fc29d2999beadd08acf6d772e65d9ce075d089301be57f5a262c7": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.405: INFO: At 2022-12-22 17:40:31 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-s2fqk: {kubelet i-0243def68c064e8ef} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "5cd947cbec3688affc6194cd7dc0e00c3e76f40702c1e7729a2587f89bfc291a": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.405: INFO: At 2022-12-22 17:40:32 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-87xst: {kubelet i-076110ae730e268e3} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "a7bd80ddc36a98f138b2b8d60d41457b045215f4700212aedaa947ace9656491": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.405: INFO: At 2022-12-22 17:40:32 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-bk7bk: {kubelet i-0243def68c064e8ef} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "cfc959a078f8032076231701266be6a1a0f07922c1c4fb88eb57686377893fdc": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.405: INFO: At 2022-12-22 17:40:32 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-fc9kd: {kubelet i-0c76fbcc0b95be727} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "b583e2c2c58f4fe96bd2616fdc11fdcf33de05a1fc139556dbfcb60770d34f00": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.405: INFO: At 2022-12-22 17:40:33 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-cv9l5: {kubelet i-0243def68c064e8ef} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "066490037abab8d21ca6457278037e55c3c6c594297882ceb5635e43a8603e71": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.405: INFO: At 2022-12-22 17:40:34 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-dtkw2: {kubelet i-0c76fbcc0b95be727} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "6645f623a0a707d05c739cf6c55586612911387158de7bbe8c5ee18f5a333bea": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.405: INFO: At 2022-12-22 17:40:34 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-mttmd: {kubelet i-0243def68c064e8ef} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "68d60435950f73c96c04abd97bc150d5b099aa4ef9617ba26d05c009dd334be0": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.405: INFO: At 2022-12-22 17:40:37 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-6772c: {kubelet i-0243def68c064e8ef} Started: Started container cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b Dec 22 17:57:45.405: INFO: At 2022-12-22 17:40:37 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-6772c: {kubelet i-0243def68c064e8ef} Pulled: Container image "registry.k8s.io/pause:3.9" already present on machine Dec 22 17:57:45.405: INFO: At 2022-12-22 17:40:37 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-6772c: {kubelet i-0243def68c064e8ef} Created: Created container cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b Dec 22 17:57:45.405: INFO: At 2022-12-22 17:40:39 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-qr2nz: {kubelet i-076110ae730e268e3} Pulled: Container image "registry.k8s.io/pause:3.9" already present on machine Dec 22 17:57:45.405: INFO: At 2022-12-22 17:40:39 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-qr2nz: {kubelet i-076110ae730e268e3} Started: Started container cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b Dec 22 17:57:45.405: INFO: At 2022-12-22 17:40:39 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-qr2nz: {kubelet i-076110ae730e268e3} Created: Created container cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b Dec 22 17:57:45.405: INFO: At 2022-12-22 17:40:39 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-snhmt: {kubelet i-0ea878d3671f2eda2} Started: Started container cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b Dec 22 17:57:45.405: INFO: At 2022-12-22 17:40:39 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-snhmt: {kubelet i-0ea878d3671f2eda2} Created: Created container cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b Dec 22 17:57:45.405: INFO: At 2022-12-22 17:40:39 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-snhmt: {kubelet i-0ea878d3671f2eda2} Pulled: Container image "registry.k8s.io/pause:3.9" already present on machine Dec 22 17:57:45.405: INFO: At 2022-12-22 17:40:39 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-t5jgm: {kubelet i-076110ae730e268e3} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "18d032ebcdabe6546577d88a5186cd8e1247a636fd3ef19f9145a39348d2652a": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.405: INFO: At 2022-12-22 17:40:40 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-dnwkk: {kubelet i-076110ae730e268e3} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "43e8e82ba0e836c3ea659b26856576981172a72b64f83cf5afd23a7a78c92f5c": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.405: INFO: At 2022-12-22 17:40:40 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-xjw6k: {kubelet i-076110ae730e268e3} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "4e9bbbf7a7d50cf377ae40fcf05167a69f079bce1b3c800e0bcee2f79aff3559": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.405: INFO: At 2022-12-22 17:40:41 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-2pn4l: {kubelet i-076110ae730e268e3} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "df6adbcc48b0d28f4c71f387d7b63e648d0b8e3a48e6940f9af3d7a196164cbd": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.405: INFO: At 2022-12-22 17:40:42 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-h4bp2: {kubelet i-076110ae730e268e3} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "ae160e52c4f1ea858a3f2093acdde4178097e2be218456395aa0302a1d49d4ea": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.405: INFO: At 2022-12-22 17:40:42 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-sx4vk: {kubelet i-0ea878d3671f2eda2} Pulled: Container image "registry.k8s.io/pause:3.9" already present on machine Dec 22 17:57:45.405: INFO: At 2022-12-22 17:40:42 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-sx4vk: {kubelet i-0ea878d3671f2eda2} Started: Started container cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b Dec 22 17:57:45.405: INFO: At 2022-12-22 17:40:42 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-sx4vk: {kubelet i-0ea878d3671f2eda2} Created: Created container cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b Dec 22 17:57:45.405: INFO: At 2022-12-22 17:40:43 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-4tjzb: {kubelet i-0ea878d3671f2eda2} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "9727417b427531021dc66c96c829966e4cde4b078034f48b1ad779fdf62934a5": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.405: INFO: At 2022-12-22 17:40:43 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-7fg7v: {kubelet i-0c76fbcc0b95be727} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "209e121afba97af164efa865413d4ab6c2baf45685616e37e0dc9532dbc14488": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.405: INFO: At 2022-12-22 17:40:43 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-7rlxn: {kubelet i-076110ae730e268e3} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "5bb08022cb9294de02dd5efd4b4209d2f1e41ec813a287f895263680eaaf77c4": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.405: INFO: At 2022-12-22 17:40:43 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-89z57: {kubelet i-0243def68c064e8ef} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "fdf117869e83e6ec969e09382e67c0db3564f04c9a4a2e10ad469722b482033f": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.405: INFO: At 2022-12-22 17:40:43 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-bk7bk: {kubelet i-0243def68c064e8ef} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "4141b0c931a3f262a15dfe1068210ae39c62f3682a22041ac2aad111dc62abbe": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.405: INFO: At 2022-12-22 17:40:43 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-d9wnv: {kubelet i-0c76fbcc0b95be727} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "f8676a39baee829ee5add7f396e19f88e94958621bc7a221d301c98ba660df6d": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.405: INFO: At 2022-12-22 17:40:43 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-dzw7n: {kubelet i-0ea878d3671f2eda2} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "b1b1c9c9df160b1c6a24e39dc294b48dcd32146f6e1e4895e0308820864e4c86": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.405: INFO: At 2022-12-22 17:40:43 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-s2fqk: {kubelet i-0243def68c064e8ef} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "e6db85e78f86feeb916d6f861340ce444506f4d3c0de86725dc3ea89d027a2c8": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.405: INFO: At 2022-12-22 17:40:44 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-nxx2s: {kubelet i-0c76fbcc0b95be727} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "b5edce85d2afcc2ef606faaaf9395db931ebb8496395fed8e9e16cd1e91ecdbf": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.405: INFO: At 2022-12-22 17:40:45 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-qrnbk: {kubelet i-0243def68c064e8ef} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "a3b9a4e1d3889d49772fa69a4a6833ed33e466c865c7d171ddfd584d19f7471a": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.405: INFO: At 2022-12-22 17:40:47 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-87xst: {kubelet i-076110ae730e268e3} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "dfceafc610d68a357abd686eaa720b68823c1e62dbaa01f9c267de02cea57f32": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.405: INFO: At 2022-12-22 17:40:47 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-fc9kd: {kubelet i-0c76fbcc0b95be727} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "9c649a69682aab1bedde963da66234714e5e709a8e04efadf274afc59de408a3": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.405: INFO: At 2022-12-22 17:40:48 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-cv9l5: {kubelet i-0243def68c064e8ef} Created: Created container cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b Dec 22 17:57:45.405: INFO: At 2022-12-22 17:40:48 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-cv9l5: {kubelet i-0243def68c064e8ef} Pulled: Container image "registry.k8s.io/pause:3.9" already present on machine Dec 22 17:57:45.405: INFO: At 2022-12-22 17:40:48 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-dtkw2: {kubelet i-0c76fbcc0b95be727} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "4f0b402db073c71b4c2c8aae06f5e38aa34516bfa0eaba8d954a8e7ef94cb1ca": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.405: INFO: At 2022-12-22 17:40:49 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-cv9l5: {kubelet i-0243def68c064e8ef} Started: Started container cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b Dec 22 17:57:45.405: INFO: At 2022-12-22 17:40:50 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-mttmd: {kubelet i-0243def68c064e8ef} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "5aa119767105af1342f18cf235e38ccaf148fda78635f064a977746424274a27": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.405: INFO: At 2022-12-22 17:40:52 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-xjw6k: {kubelet i-076110ae730e268e3} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "d07af866b352d73dc0ba443e2c9624f5b870469f47a18366b89fb64e22eb0e11": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.405: INFO: At 2022-12-22 17:40:53 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-dnwkk: {kubelet i-076110ae730e268e3} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "34c97436f3e7a0eb0857e30e66bb59859c7962fae53cd8a7f9e39b07dc6aca14": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.405: INFO: At 2022-12-22 17:40:54 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-2pn4l: {kubelet i-076110ae730e268e3} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "bb31d3689f45d4a81fe64f9137588daeadc93b8628c129df912bf238fb51c9be": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.405: INFO: At 2022-12-22 17:40:55 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-4tjzb: {kubelet i-0ea878d3671f2eda2} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "6cde4d9e26b3daed3f2a183968e3d3aa1092e190f9c525f456ce5ff1299206d2": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.405: INFO: At 2022-12-22 17:40:55 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-dzw7n: {kubelet i-0ea878d3671f2eda2} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "f8159466a48ec55332ecc352e5caefd9502bd4e7f3aac87137666832f199f139": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.405: INFO: At 2022-12-22 17:40:55 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-h4bp2: {kubelet i-076110ae730e268e3} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "bb785d254db475a4d773533f0b56cafce5d76c62a5bb8ba2c0f9e8a11d87439e": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.405: INFO: At 2022-12-22 17:40:55 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-nxx2s: {kubelet i-0c76fbcc0b95be727} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "cb002f33f95f13621c9c527587c70478474edeb6dd90af3bca485ae638c488e9": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.405: INFO: At 2022-12-22 17:40:55 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-t5jgm: {kubelet i-076110ae730e268e3} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "7143d78c142fdb38e6b834f54ae7739fc6a25a2acc5e49dbd58d012c67be6700": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.405: INFO: At 2022-12-22 17:40:56 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-7rlxn: {kubelet i-076110ae730e268e3} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "e012f1e588b1b65730490869dfb132806f8fb4691bbc67ba9767cb4200477b29": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.405: INFO: At 2022-12-22 17:40:56 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-89z57: {kubelet i-0243def68c064e8ef} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "248423ce223c739fa68c4a500281cedc4415d8acde1293405c18fffa689c06f5": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.405: INFO: At 2022-12-22 17:40:56 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-d9wnv: {kubelet i-0c76fbcc0b95be727} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "196e0274f61c47d8f149cdc0e5eae8059f35af88f704509aba52983640e7921b": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.405: INFO: At 2022-12-22 17:40:57 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-7fg7v: {kubelet i-0c76fbcc0b95be727} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "b546d46a08abdc3c12f8ee8b98510adcabba40eb06d1d215e292c72c162f5ac2": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.405: INFO: At 2022-12-22 17:40:58 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-bk7bk: {kubelet i-0243def68c064e8ef} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "c7b1c059cee194b825636f50910f07471e055b2474fb1e4a1e41ee724e01098d": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.405: INFO: At 2022-12-22 17:40:58 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-s2fqk: {kubelet i-0243def68c064e8ef} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "512fcd8b59b1a2fed4008b5bbb1e6317736a3cfa5131e9257d6b0cbf6e5dc772": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.405: INFO: At 2022-12-22 17:40:59 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-87xst: {kubelet i-076110ae730e268e3} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "4b854c960bfca1395f97e195dd4fcc8b923de96ff7fe9f58f730a4cc0a74ae32": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.405: INFO: At 2022-12-22 17:40:59 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-qrnbk: {kubelet i-0243def68c064e8ef} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "d6486c8ee90168b4bf6e21da9e70e5a452118fa36321a667ec2f306defc4d63b": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.405: INFO: At 2022-12-22 17:41:01 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-dtkw2: {kubelet i-0c76fbcc0b95be727} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "10d0ddf3071da4dfdabf3c0c0d9bfd3105a5f56b462805f237d3ee2a6d534d34": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.405: INFO: At 2022-12-22 17:41:03 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-fc9kd: {kubelet i-0c76fbcc0b95be727} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "27796784078f19deb442b5900877c189ac2e6553bf405ab07afea1697320fc35": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.405: INFO: At 2022-12-22 17:41:04 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-mttmd: {kubelet i-0243def68c064e8ef} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "b4f0365f57a825f070af7d0ce5f8c00c22da8066b7c9b8b6c143b6d01047a035": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.405: INFO: At 2022-12-22 17:41:06 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-dzw7n: {kubelet i-0ea878d3671f2eda2} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "4d305d348a8b7f7e76d46ba03203ba74a0373f7ffcd96bf5ee144b15a40688af": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.405: INFO: At 2022-12-22 17:41:07 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-dnwkk: {kubelet i-076110ae730e268e3} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "1f997de148ef95833c892bd28015447fbc9d6df2f6007bc2b5528830664c8f34": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.405: INFO: At 2022-12-22 17:41:07 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-nxx2s: {kubelet i-0c76fbcc0b95be727} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "9e9a6c2e2417527ea87cd816d1ca8f998201616ecddc3497ba24898921f708af": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.405: INFO: At 2022-12-22 17:41:07 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-xjw6k: {kubelet i-076110ae730e268e3} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "f5028c767c96408b4c5180e22462b5d4cec4f5d5e497f7939c37262ec4ad7b42": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.405: INFO: At 2022-12-22 17:41:08 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-2pn4l: {kubelet i-076110ae730e268e3} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "f81f830f7d98089241d10fe431c3ca7278266881ca1ca3c88bb0b61a47ff1a33": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.405: INFO: At 2022-12-22 17:41:08 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-4tjzb: {kubelet i-0ea878d3671f2eda2} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "1d674b511b22da2747ad683840bcb49556352cdf7080cafb86d839f44b20dfcc": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.405: INFO: At 2022-12-22 17:41:08 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-7fg7v: {kubelet i-0c76fbcc0b95be727} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "fd99a9320aa12b1ff1f59074d697fc35a176335630979647fcbb391b7177afc9": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.405: INFO: At 2022-12-22 17:41:08 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-h4bp2: {kubelet i-076110ae730e268e3} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "14de9d6ddfbcd13cd5651ead998678664e6a7afdcb2f98013650beebcf485c30": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.405: INFO: At 2022-12-22 17:41:09 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-7rlxn: {kubelet i-076110ae730e268e3} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "6f4789ff0995241afaf7b81fab47dfaf1dedb358113b2fe3812be0ca8e4dc517": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.405: INFO: At 2022-12-22 17:41:10 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-87xst: {kubelet i-076110ae730e268e3} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "8bb6e7b57ee64262d2228220a965bd8a18e3e6351f87517ed376d5a078917d78": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.405: INFO: At 2022-12-22 17:41:10 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-89z57: {kubelet i-0243def68c064e8ef} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "f5bb953d877d852e5b7a573da669e8f96b4682f209b932aaa5e5a59b5810c305": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.405: INFO: At 2022-12-22 17:41:10 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-d9wnv: {kubelet i-0c76fbcc0b95be727} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "59a5016c465ae5785e96083ef9ee363852b7479f3d8aacf690372fd8bbc33c3f": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.405: INFO: At 2022-12-22 17:41:10 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-qrnbk: {kubelet i-0243def68c064e8ef} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "595940abd851e167095c34b990ea9b22b6a51856a838bd22779ec6e263ce908a": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.405: INFO: At 2022-12-22 17:41:10 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-s2fqk: {kubelet i-0243def68c064e8ef} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "93058f5a6ae178b2f6041a2a490bf3d82e8d2b87ddd304c33dd3b0b8c1ecee90": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.405: INFO: At 2022-12-22 17:41:10 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-t5jgm: {kubelet i-076110ae730e268e3} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "87b5246ea624ab5034b1ba0073f2ac7f1b3844826c8928446a57e11363fc7eda": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.405: INFO: At 2022-12-22 17:41:11 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-bk7bk: {kubelet i-0243def68c064e8ef} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "aa87e4fc48c86d59e09c1ed45bf7709cfa22279314de7ff1b12ba2a4d57af1be": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.405: INFO: At 2022-12-22 17:41:14 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-dtkw2: {kubelet i-0c76fbcc0b95be727} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "a943629b71243a5f2a9e30f61943cce8022144cffc525612322967fab6fc4c36": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.405: INFO: At 2022-12-22 17:41:15 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-mttmd: {kubelet i-0243def68c064e8ef} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "8d6b8af3f8110070f112a81142075af06d12027d4a7e3ba8607b361c165d3544": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.405: INFO: At 2022-12-22 17:41:18 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-fc9kd: {kubelet i-0c76fbcc0b95be727} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "afed154285762ea746cd8d9fac444a254354b859639a2e09cf36acb35fab3083": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.405: INFO: At 2022-12-22 17:41:18 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-xjw6k: {kubelet i-076110ae730e268e3} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "2e3c179b1704f8522ae3fa12929b8cc4f696c59c4a154c1ea11bbc423985f8a0": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.405: INFO: At 2022-12-22 17:41:19 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-dnwkk: {kubelet i-076110ae730e268e3} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "0d1bc6a2a28a3a3e3ada9ec76645174703963159a175efa1eea0f5396d56c858": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.405: INFO: At 2022-12-22 17:41:19 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-dzw7n: {kubelet i-0ea878d3671f2eda2} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "f6e915730755d31f3383dd76bfbcaf6953e5650d664e36f722f3c28608cc7b62": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.405: INFO: At 2022-12-22 17:41:20 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-4tjzb: {kubelet i-0ea878d3671f2eda2} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "66fa91c7c6c77096ff06a6474ebda8ab528a07b7b4f46e0a52b59beeaa9da601": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.405: INFO: At 2022-12-22 17:41:21 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-7fg7v: {kubelet i-0c76fbcc0b95be727} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "cfd08ffaa9a9a72df030bf6e61a534e287ccb0f129429e221845e0d77400633d": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.405: INFO: At 2022-12-22 17:41:21 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-7rlxn: {kubelet i-076110ae730e268e3} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "f183c19a0c910bcc70a397d8940ef349f12c7d2b2f6f55da65e82925fe7d2a99": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.405: INFO: At 2022-12-22 17:41:21 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-89z57: {kubelet i-0243def68c064e8ef} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "b97e0a1738c3f2fc457105a9704abc6d542a69c28f5e7556503af48524380393": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.405: INFO: At 2022-12-22 17:41:21 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-nxx2s: {kubelet i-0c76fbcc0b95be727} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "39bf0053880a9b27dd0f27071d0df2042edf16fcc1875454a090512717ee08e6": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.405: INFO: At 2022-12-22 17:41:22 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-h4bp2: {kubelet i-076110ae730e268e3} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "35c8cbaed2c1b0f7f5fc4ec5875591c73cd4c3da9e97cd2a8eb67b289411b500": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.405: INFO: At 2022-12-22 17:41:22 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-s2fqk: {kubelet i-0243def68c064e8ef} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "65ac89e12e043b38a236bd0f9caae0311768c7b8e8ac0c8e5aaf22f3df642395": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.405: INFO: At 2022-12-22 17:41:23 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-2pn4l: {kubelet i-076110ae730e268e3} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "79120bc410ae577789f9638ebe766f02843f97f94d7ee2ea956828092bc32924": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.405: INFO: At 2022-12-22 17:41:23 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-bk7bk: {kubelet i-0243def68c064e8ef} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "f44572cf1eea204df03d6bd14135db0f7accdd8ef4541180d877e35f8dab1a4b": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.405: INFO: At 2022-12-22 17:41:23 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-t5jgm: {kubelet i-076110ae730e268e3} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "647bffd8647ee5eafbbd5fa7f0138f033fb6748e8340ddf67b1f6a7e5963cd2c": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.405: INFO: At 2022-12-22 17:41:24 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-87xst: {kubelet i-076110ae730e268e3} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "4d44557433b4cfdc8a84260dcb94b5c347156ca46aebd94a69b1781cea788511": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.405: INFO: At 2022-12-22 17:41:25 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-qrnbk: {kubelet i-0243def68c064e8ef} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "8f2f7fbc748789a8778cefd8b7fe615bebca214219a626c2f8d83c98f96072c5": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.405: INFO: At 2022-12-22 17:41:26 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-d9wnv: {kubelet i-0c76fbcc0b95be727} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "f0d09ea85ff350cebd812f795eed48291fdc747fafa3da76dd3a03049e639344": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.405: INFO: At 2022-12-22 17:41:27 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-dtkw2: {kubelet i-0c76fbcc0b95be727} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "7ab3b4894c286faf542550ea0e0d31bd0d87593b59882bd1f783346d62b0f7a6": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.405: INFO: At 2022-12-22 17:41:28 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-mttmd: {kubelet i-0243def68c064e8ef} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "dea8f6441c2c48058b31cfbcbb0a80a4d7fccd59c5a912fbda4de4838bd56b71": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.405: INFO: At 2022-12-22 17:41:30 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-xjw6k: {kubelet i-076110ae730e268e3} FailedCreatePodSandBox: (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "d03f51d9e300ce484498f6f9cc349682e31f5fb54e779fc40cf40b3699dd7321": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.405: INFO: At 2022-12-22 17:41:32 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-dnwkk: {kubelet i-076110ae730e268e3} FailedCreatePodSandBox: (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "221560d4b0153344813c8a3d70c3c5a4438025fdbad4bfa400d305d41ef6f4bb": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.405: INFO: At 2022-12-22 17:41:33 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-fc9kd: {kubelet i-0c76fbcc0b95be727} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "39af85de109b9411725b04b726e424e1b5da76356f1fe398e6526e2796b3c0be": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.405: INFO: At 2022-12-22 17:41:33 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-h4bp2: {kubelet i-076110ae730e268e3} FailedCreatePodSandBox: (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "4d99c29e3465f1d363e598398780e519701943d5ee780c638cf46db941160eb5": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.405: INFO: At 2022-12-22 17:41:34 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-4tjzb: {kubelet i-0ea878d3671f2eda2} FailedCreatePodSandBox: (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "699e6ed0727723e1b729ae7718d83c1fc6717353c86e3d920defc7b9e786713d": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.405: INFO: At 2022-12-22 17:41:34 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-dzw7n: {kubelet i-0ea878d3671f2eda2} FailedCreatePodSandBox: (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "0943f670155737b5b935130a2245c703cab8534bbc366f37435a4e327f363e77": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.405: INFO: At 2022-12-22 17:41:34 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-s2fqk: {kubelet i-0243def68c064e8ef} FailedCreatePodSandBox: (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "240242c4dd2222077b4d82486e0e08f8138bc38c0d10fa94cd7481b0b6da8950": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.405: INFO: At 2022-12-22 17:41:36 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-7fg7v: {kubelet i-0c76fbcc0b95be727} FailedCreatePodSandBox: (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "9174f936d67ea0f3bf1b4437c28669eb44d75d288565d3e0b49cc2b328e23e54": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.405: INFO: At 2022-12-22 17:41:36 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-7rlxn: {kubelet i-076110ae730e268e3} FailedCreatePodSandBox: (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "f19f40714ee49e7167bb3b755ee4ac76d74aefc09cc6cff717bdfa5f77672944": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.405: INFO: At 2022-12-22 17:41:36 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-89z57: {kubelet i-0243def68c064e8ef} FailedCreatePodSandBox: (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "fe557fd502a9e20e96fee992fde9d60e469df4784dff6befb2627797368aec62": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.405: INFO: At 2022-12-22 17:41:37 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-bk7bk: {kubelet i-0243def68c064e8ef} FailedCreatePodSandBox: (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "7017715825c0d34e51b3f965a88cf777893560eed26b0497a9e6a004afe3c8df": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.405: INFO: At 2022-12-22 17:41:37 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-nxx2s: {kubelet i-0c76fbcc0b95be727} FailedCreatePodSandBox: (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "34ef3ce08810c7438211dbe242d7f0267da868b267bc37ee034586367a3a2d44": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.405: INFO: At 2022-12-22 17:41:37 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-t5jgm: {kubelet i-076110ae730e268e3} FailedCreatePodSandBox: (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "da76e3126fc5c6de7ae68ed20f2df38fbababfe6a0ec5d46584da7d105859428": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.405: INFO: At 2022-12-22 17:41:38 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-2pn4l: {kubelet i-076110ae730e268e3} FailedCreatePodSandBox: (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "368a45f35aa6fcd1c83b34c3c9bfac627cc295ec47dd186cc2cd1d265a706a7e": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.405: INFO: At 2022-12-22 17:41:38 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-87xst: {kubelet i-076110ae730e268e3} FailedCreatePodSandBox: (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "00eec8e9ad260d1377104dff6aa30b3c4ef0ca53006ae89c0139f07c2482a25e": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.405: INFO: At 2022-12-22 17:41:38 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-d9wnv: {kubelet i-0c76fbcc0b95be727} FailedCreatePodSandBox: (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "381dd66004820510d6743ed69492a7439964246a4e91954c07c7e1965a340453": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.405: INFO: At 2022-12-22 17:41:40 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-mttmd: {kubelet i-0243def68c064e8ef} FailedCreatePodSandBox: (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "1db878b68ae4b75cb3c27a3ccd19172fe729396f391f9b2c4329631213ecbb07": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.405: INFO: At 2022-12-22 17:41:40 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-qrnbk: {kubelet i-0243def68c064e8ef} FailedCreatePodSandBox: (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "a9dcb8c5cbcab54e1e33bd45c77b138ac0680eb90158ca56f6d82f6deef315b2": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.405: INFO: At 2022-12-22 17:41:41 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-dtkw2: {kubelet i-0c76fbcc0b95be727} FailedCreatePodSandBox: (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "efbf66bb01705e67c94244fc6a92d22e97620e0da6c50e095bae405ef2659738": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:45.405: INFO: At 2022-12-22 17:41:45 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-4tjzb: {kubelet i-0ea878d3671f2eda2} Pulled: Container image "registry.k8s.io/pause:3.9" already present on machine Dec 22 17:57:45.405: INFO: At 2022-12-22 17:41:45 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-4tjzb: {kubelet i-0ea878d3671f2eda2} Started: Started container cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b Dec 22 17:57:45.405: INFO: At 2022-12-22 17:41:45 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-4tjzb: {kubelet i-0ea878d3671f2eda2} Created: Created container cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b Dec 22 17:57:45.405: INFO: At 2022-12-22 17:41:48 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-dnwkk: {kubelet i-076110ae730e268e3} Started: Started container cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b Dec 22 17:57:45.405: INFO: At 2022-12-22 17:41:48 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-dnwkk: {kubelet i-076110ae730e268e3} Pulled: Container image "registry.k8s.io/pause:3.9" already present on machine Dec 22 17:57:45.406: INFO: At 2022-12-22 17:41:48 +0000 UTC - event for cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b-dnwkk: {kubelet i-076110ae730e268e3} Created: Created container cleanup40-42c1bab6-384e-4f20-af47-cf81b057a48b ... skipping 409 lines ... [38;5;14mDriver local doesn't support DynamicPV -- skipping[0m [38;5;14mIn [1m[BeforeEach][0m[38;5;14m at: [1mtest/e2e/storage/framework/testsuite.go:116[0m [38;5;243m------------------------------[0m [38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m [38;5;243m------------------------------[0m [38;5;9m• [FAILED] [365.834 seconds][0m [sig-storage] CSI mock volume [38;5;243mtest/e2e/storage/utils/framework.go:23[0m CSI attach test using mock driver [38;5;243mtest/e2e/storage/csi_mock_volume.go:364[0m [38;5;9m[1m[It] should require VolumeAttach for ephemermal volume and drivers with attachment[0m [38;5;243mtest/e2e/storage/csi_mock_volume.go:392[0m ... skipping 47 lines ... Dec 22 17:51:47.255: INFO: creating *v1.StorageClass: csi-mock-sc-csi-mock-volumes-2425 Dec 22 17:51:47.318: INFO: creating *v1.StatefulSet: csi-mock-volumes-2425-7806/csi-mockplugin Dec 22 17:51:47.384: INFO: creating *v1.CSIDriver: csi-mock-csi-mock-volumes-2425 Dec 22 17:51:47.448: INFO: creating *v1.StatefulSet: csi-mock-volumes-2425-7806/csi-mockplugin-attacher Dec 22 17:51:47.512: INFO: waiting up to 4m0s for CSIDriver "csi-mock-csi-mock-volumes-2425" Dec 22 17:51:47.576: INFO: waiting for CSIDriver csi-mock-csi-mock-volumes-2425 to register on node i-0243def68c064e8ef Dec 22 17:57:30.332: INFO: Unexpected error: Failed to register CSIDriver csi-mock-csi-mock-volumes-2425: <*errors.errorString | 0xc0012982c0>: { s: "error waiting for CSI driver csi-mock-csi-mock-volumes-2425 registration on node i-0243def68c064e8ef: timed out waiting for the condition", } Dec 22 17:57:30.332: FAIL: Failed to register CSIDriver csi-mock-csi-mock-volumes-2425: error waiting for CSI driver csi-mock-csi-mock-volumes-2425 registration on node i-0243def68c064e8ef: timed out waiting for the condition Full Stack Trace k8s.io/kubernetes/test/e2e/storage.glob..func2.1({0x0, 0x0, 0x1, 0x0, 0x0, 0x0, 0x0, {0x0, 0x0}, 0x0, ...}) test/e2e/storage/csi_mock_volume.go:201 +0x81c k8s.io/kubernetes/test/e2e/storage.glob..func2.7.1() test/e2e/storage/csi_mock_volume.go:394 +0x103 ... skipping 352 lines ... Dec 22 17:57:41.156: INFO: deleting *v1.CSIDriver: csi-mock-csi-mock-volumes-2425 Dec 22 17:57:41.222: INFO: deleting *v1.StatefulSet: csi-mock-volumes-2425-7806/csi-mockplugin-attacher [1mSTEP:[0m deleting the driver namespace: csi-mock-volumes-2425-7806 [38;5;243m12/22/22 17:57:41.287[0m [1mSTEP:[0m Collecting events from namespace "csi-mock-volumes-2425-7806". [38;5;243m12/22/22 17:57:41.287[0m [1mSTEP:[0m Found 24 events. [38;5;243m12/22/22 17:57:41.352[0m Dec 22 17:57:41.352: INFO: At 2022-12-22 17:51:47 +0000 UTC - event for csi-mockplugin: {statefulset-controller } SuccessfulCreate: create Pod csi-mockplugin-0 in StatefulSet csi-mockplugin successful Dec 22 17:57:41.352: INFO: At 2022-12-22 17:51:47 +0000 UTC - event for csi-mockplugin-0: {kubelet i-0243def68c064e8ef} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "bf67497a9955130759d730058f5a33f170e4c1c905233228dfbb18fe15a38a85": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:41.352: INFO: At 2022-12-22 17:51:47 +0000 UTC - event for csi-mockplugin-0: {default-scheduler } Scheduled: Successfully assigned csi-mock-volumes-2425-7806/csi-mockplugin-0 to i-0243def68c064e8ef Dec 22 17:57:41.352: INFO: At 2022-12-22 17:51:47 +0000 UTC - event for csi-mockplugin-attacher: {statefulset-controller } SuccessfulCreate: create Pod csi-mockplugin-attacher-0 in StatefulSet csi-mockplugin-attacher successful Dec 22 17:57:41.352: INFO: At 2022-12-22 17:51:47 +0000 UTC - event for csi-mockplugin-attacher-0: {default-scheduler } Scheduled: Successfully assigned csi-mock-volumes-2425-7806/csi-mockplugin-attacher-0 to i-0243def68c064e8ef Dec 22 17:57:41.352: INFO: At 2022-12-22 17:51:47 +0000 UTC - event for csi-mockplugin-attacher-0: {kubelet i-0243def68c064e8ef} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "9018e4fd68a357b8159ea03281cb538b5907e69acf4636072d95d56d7799010e": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:41.352: INFO: At 2022-12-22 17:51:59 +0000 UTC - event for csi-mockplugin-attacher-0: {kubelet i-0243def68c064e8ef} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "e27b2c69a8e0565f65f1b822e5ded306982bf5ac65b7ea4fc993db9de8b7d5e6": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:41.352: INFO: At 2022-12-22 17:52:00 +0000 UTC - event for csi-mockplugin-0: {kubelet i-0243def68c064e8ef} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "357b6416f1872521acee5797bd030b7d8b66ad8d7c00d539a73f6de5dc74f43b": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:41.352: INFO: At 2022-12-22 17:52:11 +0000 UTC - event for csi-mockplugin-attacher-0: {kubelet i-0243def68c064e8ef} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "a08446586d67847b69bd803f67a2d8ae33d772a5a18c260aa7750cafefe560b0": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:41.352: INFO: At 2022-12-22 17:52:15 +0000 UTC - event for csi-mockplugin-0: {kubelet i-0243def68c064e8ef} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "700bbc0a3e00f31a94327205b077f354e36d6f24b9bc3a0838fe5164e5763748": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:41.352: INFO: At 2022-12-22 17:52:26 +0000 UTC - event for csi-mockplugin-attacher-0: {kubelet i-0243def68c064e8ef} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "b52968b1455c2dcbfa5fc2ae5728bf4af2b43643e87d0fd5ceeb6dfff5abc0c6": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:41.352: INFO: At 2022-12-22 17:52:28 +0000 UTC - event for csi-mockplugin-0: {kubelet i-0243def68c064e8ef} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "6331c1dc4efedc5e99c4d287fe247c4c1ae072f3315124af88a64acaccee2b15": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:41.352: INFO: At 2022-12-22 17:52:41 +0000 UTC - event for csi-mockplugin-0: {kubelet i-0243def68c064e8ef} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "33fbb3637fa54434cbbacf31e511198ae37ed345471b0fdef262e13c34a78422": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:41.352: INFO: At 2022-12-22 17:52:41 +0000 UTC - event for csi-mockplugin-attacher-0: {kubelet i-0243def68c064e8ef} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "44fcc9a6fcadcc10d7b6dd34d1e8f27a8fe9e996ee11234fe30eebc958e3c4db": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:41.352: INFO: At 2022-12-22 17:52:53 +0000 UTC - event for csi-mockplugin-0: {kubelet i-0243def68c064e8ef} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "aeacb894b86c30d4ebb43be5947c67a9e2a7adf8c156a291ad0a06f94c71aa56": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:41.352: INFO: At 2022-12-22 17:52:56 +0000 UTC - event for csi-mockplugin-attacher-0: {kubelet i-0243def68c064e8ef} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "cf23ece5c67d658b9e3e5e12dcac835f822bc4a01d70ccce4d3091d69129d46a": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:41.352: INFO: At 2022-12-22 17:53:07 +0000 UTC - event for csi-mockplugin-0: {kubelet i-0243def68c064e8ef} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "8ca0945d88ea3e21323661aa854cdab89f6c165b16a78d6d2ab7be0e825124ba": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:41.352: INFO: At 2022-12-22 17:53:09 +0000 UTC - event for csi-mockplugin-attacher-0: {kubelet i-0243def68c064e8ef} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "db10788cd05acdf1ba1b9c881791eb77ae97a3fffe39d530a4d2bb627d0dddb3": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:41.352: INFO: At 2022-12-22 17:53:20 +0000 UTC - event for csi-mockplugin-0: {kubelet i-0243def68c064e8ef} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "e39d5d85cafa5edc4065716399719cb738a60b42b31452d337e92af71611fc88": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:41.352: INFO: At 2022-12-22 17:53:21 +0000 UTC - event for csi-mockplugin-attacher-0: {kubelet i-0243def68c064e8ef} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "fec24dba9bcd68050e92cdd0ee76271af1bc42b9ca896a24bbf38eeb6735ffef": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:41.352: INFO: At 2022-12-22 17:53:34 +0000 UTC - event for csi-mockplugin-0: {kubelet i-0243def68c064e8ef} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "21ce7c8a6cbc62be0365e55b4a1a4686d244adcf6d6eadcecef4263ac188a71d": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:41.352: INFO: At 2022-12-22 17:53:34 +0000 UTC - event for csi-mockplugin-attacher-0: {kubelet i-0243def68c064e8ef} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "ccd024a219eafff70eef8ab14814d9a36c53eddf8f482c883e037731284f0b52": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:41.352: INFO: At 2022-12-22 17:53:46 +0000 UTC - event for csi-mockplugin-attacher-0: {kubelet i-0243def68c064e8ef} FailedCreatePodSandBox: (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "84c008a32e6af2c315a66d907eb1a1a52305e883e64a5152095c5cf4836db485": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:41.352: INFO: At 2022-12-22 17:53:49 +0000 UTC - event for csi-mockplugin-0: {kubelet i-0243def68c064e8ef} FailedCreatePodSandBox: (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "58044475de8b64baa8df4e9e45a721b5c618d815f104a5fca72cb0d61c60f49a": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:57:41.416: INFO: POD NODE PHASE GRACE CONDITIONS Dec 22 17:57:41.416: INFO: csi-mockplugin-0 i-0243def68c064e8ef Pending 30s [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2022-12-22 17:51:47 +0000 UTC } {Ready False 0001-01-01 00:00:00 +0000 UTC 2022-12-22 17:51:47 +0000 UTC ContainersNotReady containers with unready status: [csi-provisioner driver-registrar mock]} {ContainersReady False 0001-01-01 00:00:00 +0000 UTC 2022-12-22 17:51:47 +0000 UTC ContainersNotReady containers with unready status: [csi-provisioner driver-registrar mock]} {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2022-12-22 17:51:47 +0000 UTC }] Dec 22 17:57:41.416: INFO: csi-mockplugin-attacher-0 i-0243def68c064e8ef Pending 30s [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2022-12-22 17:51:47 +0000 UTC } {Ready False 0001-01-01 00:00:00 +0000 UTC 2022-12-22 17:51:47 +0000 UTC ContainersNotReady containers with unready status: [csi-attacher]} {ContainersReady False 0001-01-01 00:00:00 +0000 UTC 2022-12-22 17:51:47 +0000 UTC ContainersNotReady containers with unready status: [csi-attacher]} {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2022-12-22 17:51:47 +0000 UTC }] Dec 22 17:57:41.416: INFO: Dec 22 17:57:41.485: INFO: Unable to fetch csi-mock-volumes-2425-7806/csi-mockplugin-0/csi-provisioner logs: the server rejected our request for an unknown reason (get pods csi-mockplugin-0) Dec 22 17:57:41.550: INFO: Unable to fetch csi-mock-volumes-2425-7806/csi-mockplugin-0/driver-registrar logs: the server rejected our request for an unknown reason (get pods csi-mockplugin-0) ... skipping 303 lines ... dump namespaces | framework.go:196 [1mSTEP:[0m dump namespace information after failure [38;5;243m12/22/22 17:57:50.338[0m [DeferCleanup (Each)] [sig-storage] CSI mock volume tear down framework | framework.go:193 [38;5;243m<< End Captured GinkgoWriter Output[0m [38;5;9mDec 22 17:57:30.332: Failed to register CSIDriver csi-mock-csi-mock-volumes-2425: error waiting for CSI driver csi-mock-csi-mock-volumes-2425 registration on node i-0243def68c064e8ef: timed out waiting for the condition[0m [38;5;9mIn [1m[It][0m[38;5;9m at: [1mtest/e2e/storage/csi_mock_volume.go:201[0m [38;5;243m------------------------------[0m [38;5;14mS[0m[38;5;14mS[0m [38;5;243m------------------------------[0m [38;5;14mS [SKIPPED] [0.000 seconds][0m [sig-storage] In-tree Volumes ... skipping 13 lines ... test/e2e/framework/node/init/init.go:32 [38;5;243m<< End Captured GinkgoWriter Output[0m [38;5;14mOnly supported for providers [vsphere] (not aws)[0m [38;5;14mIn [1m[BeforeEach][0m[38;5;14m at: [1mtest/e2e/storage/drivers/in_tree.go:1230[0m [38;5;243m------------------------------[0m [38;5;9m• [FAILED] [308.048 seconds][0m [sig-storage] Subpath [38;5;243mtest/e2e/storage/utils/framework.go:23[0m Atomic writer volumes [38;5;243mtest/e2e/storage/subpath.go:36[0m [38;5;9m[1m[It] should support subpaths with downward pod [Conformance][0m [38;5;243mtest/e2e/storage/subpath.go:92[0m ... skipping 12 lines ... test/e2e/storage/subpath.go:40 [1mSTEP:[0m Setting up data [38;5;243m12/22/22 17:52:58.777[0m [It] should support subpaths with downward pod [Conformance] test/e2e/storage/subpath.go:92 [1mSTEP:[0m Creating pod pod-subpath-test-downwardapi-dr4h [38;5;243m12/22/22 17:52:58.906[0m [1mSTEP:[0m Creating a pod to test atomic-volume-subpath [38;5;243m12/22/22 17:52:58.906[0m Dec 22 17:52:58.973: INFO: Waiting up to 5m0s for pod "pod-subpath-test-downwardapi-dr4h" in namespace "subpath-4983" to be "Succeeded or Failed" Dec 22 17:52:59.036: INFO: Pod "pod-subpath-test-downwardapi-dr4h": Phase="Pending", Reason="", readiness=false. Elapsed: 63.308049ms Dec 22 17:53:01.101: INFO: Pod "pod-subpath-test-downwardapi-dr4h": Phase="Pending", Reason="", readiness=false. Elapsed: 2.127886241s Dec 22 17:53:03.101: INFO: Pod "pod-subpath-test-downwardapi-dr4h": Phase="Pending", Reason="", readiness=false. Elapsed: 4.127860955s Dec 22 17:53:05.100: INFO: Pod "pod-subpath-test-downwardapi-dr4h": Phase="Pending", Reason="", readiness=false. Elapsed: 6.127565992s Dec 22 17:53:07.100: INFO: Pod "pod-subpath-test-downwardapi-dr4h": Phase="Pending", Reason="", readiness=false. Elapsed: 8.12679153s Dec 22 17:53:09.101: INFO: Pod "pod-subpath-test-downwardapi-dr4h": Phase="Pending", Reason="", readiness=false. Elapsed: 10.128701799s ... skipping 152 lines ... Dec 22 17:57:59.366: INFO: Waiting for pod pod-subpath-test-downwardapi-dr4h to disappear Dec 22 17:57:59.431: INFO: Pod pod-subpath-test-downwardapi-dr4h still exists Dec 22 17:58:01.432: INFO: Waiting for pod pod-subpath-test-downwardapi-dr4h to disappear Dec 22 17:58:01.497: INFO: Pod pod-subpath-test-downwardapi-dr4h still exists Dec 22 17:58:03.432: INFO: Waiting for pod pod-subpath-test-downwardapi-dr4h to disappear Dec 22 17:58:03.497: INFO: Pod pod-subpath-test-downwardapi-dr4h no longer exists Dec 22 17:58:03.497: INFO: Unexpected error: <*errors.errorString | 0xc006274eb0>: { s: "expected pod \"pod-subpath-test-downwardapi-dr4h\" success: timed out while waiting for pod subpath-4983/pod-subpath-test-downwardapi-dr4h to be Succeeded or Failed", } Dec 22 17:58:03.497: FAIL: expected pod "pod-subpath-test-downwardapi-dr4h" success: timed out while waiting for pod subpath-4983/pod-subpath-test-downwardapi-dr4h to be Succeeded or Failed Full Stack Trace k8s.io/kubernetes/test/e2e/framework/pod/output.TestContainerOutputMatcher(0xc0009f7b00?, {0x7606e1a?, 0x0?}, 0xc0009f7b00, 0x0, {0xc003c13df8, 0x1, 0x1}, 0x0?) test/e2e/framework/pod/output/output.go:237 +0x176 k8s.io/kubernetes/test/e2e/framework/pod/output.TestContainerOutput(...) test/e2e/framework/pod/output/output.go:214 ... skipping 11 lines ... [DeferCleanup (Each)] [sig-storage] Subpath dump namespaces | framework.go:196 [1mSTEP:[0m dump namespace information after failure [38;5;243m12/22/22 17:58:03.564[0m [1mSTEP:[0m Collecting events from namespace "subpath-4983". [38;5;243m12/22/22 17:58:03.564[0m [1mSTEP:[0m Found 12 events. [38;5;243m12/22/22 17:58:03.629[0m Dec 22 17:58:03.629: INFO: At 2022-12-22 17:52:58 +0000 UTC - event for pod-subpath-test-downwardapi-dr4h: {default-scheduler } Scheduled: Successfully assigned subpath-4983/pod-subpath-test-downwardapi-dr4h to i-0ea878d3671f2eda2 Dec 22 17:58:03.629: INFO: At 2022-12-22 17:52:59 +0000 UTC - event for pod-subpath-test-downwardapi-dr4h: {kubelet i-0ea878d3671f2eda2} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "fbe302b62ad243c9f38607d76c7390c527f1043d6de6172dda35684f0eef22a5": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:58:03.629: INFO: At 2022-12-22 17:53:13 +0000 UTC - event for pod-subpath-test-downwardapi-dr4h: {kubelet i-0ea878d3671f2eda2} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "a329aadc139ec2925b0a5877cbfa360bd466771582e6b0187e3209a055164671": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:58:03.629: INFO: At 2022-12-22 17:53:25 +0000 UTC - event for pod-subpath-test-downwardapi-dr4h: {kubelet i-0ea878d3671f2eda2} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "8f23b6463a62b520c2677846187c5d40321f2f3b62ed86a8c04c380aac2eea03": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:58:03.629: INFO: At 2022-12-22 17:53:37 +0000 UTC - event for pod-subpath-test-downwardapi-dr4h: {kubelet i-0ea878d3671f2eda2} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "12523c36bdf37b254ced992eca68f59d4c6b6ff783591c15e3c3f066f142e781": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:58:03.629: INFO: At 2022-12-22 17:53:50 +0000 UTC - event for pod-subpath-test-downwardapi-dr4h: {kubelet i-0ea878d3671f2eda2} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "d3dda12776cf9b5e997fda454f31cd04a455a7f343a719c02fee7e0280ef1724": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:58:03.629: INFO: At 2022-12-22 17:54:02 +0000 UTC - event for pod-subpath-test-downwardapi-dr4h: {kubelet i-0ea878d3671f2eda2} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "f73e54e3e6793205d11ba3675fbfbfd0ddb094d59d62e7ecbabc024159ef35e9": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:58:03.629: INFO: At 2022-12-22 17:54:13 +0000 UTC - event for pod-subpath-test-downwardapi-dr4h: {kubelet i-0ea878d3671f2eda2} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "0073c4cfe1a2d83270d0268021708bee861bc9ac949000028379700162d150f8": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:58:03.629: INFO: At 2022-12-22 17:54:24 +0000 UTC - event for pod-subpath-test-downwardapi-dr4h: {kubelet i-0ea878d3671f2eda2} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "7689ea798e373a0a1a188c0fabaec68395f4161fa5fc83edd80d6dafe54ab1da": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:58:03.629: INFO: At 2022-12-22 17:54:35 +0000 UTC - event for pod-subpath-test-downwardapi-dr4h: {kubelet i-0ea878d3671f2eda2} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "cacd6756a0a6d5dc8c5d73d9e1b6128925dbc2807e103d30310d893582f23560": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:58:03.629: INFO: At 2022-12-22 17:54:46 +0000 UTC - event for pod-subpath-test-downwardapi-dr4h: {kubelet i-0ea878d3671f2eda2} FailedCreatePodSandBox: (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "93195c952237f7aea34fafdc636ff341ab3ee1a1e68be1dabc618e7d8355621f": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 17:58:03.629: INFO: At 2022-12-22 17:57:56 +0000 UTC - event for pod-subpath-test-downwardapi-dr4h: {kubelet i-0ea878d3671f2eda2} Pulled: Container image "registry.k8s.io/e2e-test-images/agnhost:2.43" already present on machine Dec 22 17:58:03.693: INFO: POD NODE PHASE GRACE CONDITIONS Dec 22 17:58:03.694: INFO: Dec 22 17:58:03.759: INFO: Logging node info for node i-0243def68c064e8ef Dec 22 17:58:03.824: INFO: Node Info: &Node{ObjectMeta:{i-0243def68c064e8ef f07b8771-29eb-4944-a385-3482a3d3a315 50387 0 2022-12-22 17:17:47 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:us-west-2 failure-domain.beta.kubernetes.io/zone:us-west-2a kubernetes.io/arch:amd64 kubernetes.io/hostname:i-0243def68c064e8ef kubernetes.io/os:linux node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:us-west-2a topology.hostpath.csi/node:i-0243def68c064e8ef topology.kubernetes.io/region:us-west-2 topology.kubernetes.io/zone:us-west-2a] map[alpha.kubernetes.io/provided-node-ip:172.20.35.142 csi.volume.kubernetes.io/nodeid:{"csi-hostpath-ephemeral-6886":"i-0243def68c064e8ef","csi-hostpath-ephemeral-8407":"i-0243def68c064e8ef","ebs.csi.aws.com":"i-0243def68c064e8ef"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{aws-cloud-controller-manager Update v1 2022-12-22 17:17:47 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2022-12-22 17:17:47 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kops-controller Update v1 2022-12-22 17:17:47 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{}}}} } {kube-controller-manager Update v1 2022-12-22 17:17:47 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.3.0/24\"":{}}}} } {kubelet Update v1 2022-12-22 17:17:47 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:alpha.kubernetes.io/provided-node-ip":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {kube-controller-manager Update v1 2022-12-22 17:51:50 +0000 UTC FieldsV1 {"f:status":{"f:volumesAttached":{}}} status} {kubelet Update v1 2022-12-22 17:56:56 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.3.0/24,DoNotUseExternalID:,ProviderID:aws:///us-west-2a/i-0243def68c064e8ef,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.3.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{51527004160 0} {<nil>} 50319340Ki BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4050636800 0} {<nil>} 3955700Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{46374303668 0} {<nil>} 46374303668 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3945779200 0} {<nil>} 3853300Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-12-22 17:56:56 +0000 UTC,LastTransitionTime:2022-12-22 17:17:37 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-12-22 17:56:56 +0000 UTC,LastTransitionTime:2022-12-22 17:17:37 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-12-22 17:56:56 +0000 UTC,LastTransitionTime:2022-12-22 17:17:37 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-12-22 17:56:56 +0000 UTC,LastTransitionTime:2022-12-22 17:17:59 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.35.142,},NodeAddress{Type:ExternalIP,Address:35.90.235.118,},NodeAddress{Type:InternalDNS,Address:i-0243def68c064e8ef.us-west-2.compute.internal,},NodeAddress{Type:Hostname,Address:i-0243def68c064e8ef.us-west-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-35-90-235-118.us-west-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2c8ae28bb7834ee5f6b4c4ef144a8e,SystemUUID:ec2c8ae2-8bb7-834e-e5f6-b4c4ef144a8e,BootID:5843895c-91be-4c36-b700-9e551cb745de,KernelVersion:5.10.157-139.675.amzn2.x86_64,OSImage:Amazon Linux 2,ContainerRuntimeVersion:containerd://1.6.14,KubeletVersion:v1.26.0,KubeProxyVersion:v1.26.0,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:166719855,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/jessie-dnsutils@sha256:24aaf2626d6b27864c29de2097e8bbb840b3a414271bf7c8995e431e47d8408e registry.k8s.io/e2e-test-images/jessie-dnsutils:1.7],SizeBytes:112030336,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:dd75ec974b0a2a6f6bb47001ba09207976e625db898d1b16735528c009cb171c registry.k8s.io/etcd:3.5.6-0],SizeBytes:102542580,},ContainerImage{Names:[registry.k8s.io/kube-proxy-amd64:v1.26.0],SizeBytes:67205320,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:16bbf38c463a4223d8cfe4da12bc61010b082a79b4bb003e2d3ba3ece5dd5f9e registry.k8s.io/e2e-test-images/agnhost:2.43],SizeBytes:51706353,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nautilus@sha256:80ba6c8c44f9623f06e868a1aa66026c8ec438ad814f9ec95e9333b415fe3550 registry.k8s.io/e2e-test-images/nautilus:1.7],SizeBytes:49641698,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:3fe7acf013d1264ffded116b80a73dc129a449b0fccdb8d21af8279f2233f36e registry.k8s.io/e2e-test-images/httpd:2.4.39-4],SizeBytes:41901587,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:148b022f5c5da426fc2f3c14b5c0867e58ef05961510c84749ac1fddcb0fef22 registry.k8s.io/e2e-test-images/httpd:2.4.38-4],SizeBytes:40764257,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:10e231cf07c82dfab5141c1afe548dc734e5fc1f67665ec3982a325d1bd31a9a registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.0],SizeBytes:29725756,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:ee3b525d5b89db99da3b8eb521d9cd90cb6e9ef0fbb651e98bb37be78d36b5b8 registry.k8s.io/sig-storage/csi-provisioner:v3.3.0],SizeBytes:25491225,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:425d8f1b769398127767b06ed97ce62578a3179bcb99809ce93a1649e025ffe7 registry.k8s.io/sig-storage/csi-resizer:v1.6.0],SizeBytes:24148884,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:291334908ddf71a4661fd7f6d9d97274de8a5378a2b6fdfeb2ce73414a34f82f registry.k8s.io/sig-storage/csi-snapshotter:v6.1.0],SizeBytes:23881995,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:9a685020911e2725ad019dbce6e4a5ab93d51e3d4557f115e64343345e05781b registry.k8s.io/sig-storage/csi-attacher:v4.0.0],SizeBytes:23847201,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:92257881c1d6493cf18299a24af42330f891166560047902b8d431fb66b01af5 registry.k8s.io/sig-storage/hostpathplugin:v1.9.0],SizeBytes:18758628,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nonroot@sha256:ee9f50b3c64b174d296d91ca9f69a914ac30e59095dfb462b2b518ad28a63655 registry.k8s.io/e2e-test-images/nonroot:1.4],SizeBytes:17747885,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:933940f13b3ea0abc62e656c1aa5c5b47c04b15d71250413a6b821bd0c58b94e registry.k8s.io/sig-storage/livenessprobe:v2.7.0],SizeBytes:8688564,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:5c99cf6a02adda929b10321dbf4ecfa00d87be9ba4fb456006237d530ab4baa1 registry.k8s.io/e2e-test-images/nginx:1.14-4],SizeBytes:6978614,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nonewprivs@sha256:8ac1264691820febacf3aea5d152cbde6d10685731ec14966a9401c6f47a68ac registry.k8s.io/e2e-test-images/nonewprivs:1.3],SizeBytes:3263463,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf registry.k8s.io/e2e-test-images/busybox:1.29-2],SizeBytes:732424,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:2e0f836850e09b8b7cc937681d6194537a09fbd5f6b9e08f4d646a85128e8937 registry.k8s.io/e2e-test-images/busybox:1.29-4],SizeBytes:731990,},ContainerImage{Names:[registry.k8s.io/pause@sha256:7031c1b283388d2c2e09b57badb803c05ebed362dc88d84b480cc47f72a21097 registry.k8s.io/pause:3.9],SizeBytes:321520,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[kubernetes.io/csi/csi-hostpath-ephemeral-6886^4e266409-8221-11ed-800d-8a8e987c6028 kubernetes.io/csi/csi-hostpath-ephemeral-8407^9fd9423f-821f-11ed-b5cc-02642e84c89b kubernetes.io/csi/ebs.csi.aws.com^vol-09308d632dcd25d39 kubernetes.io/csi/ebs.csi.aws.com^vol-0ae1955ea2e2f0ecd kubernetes.io/csi/ebs.csi.aws.com^vol-0b74ae25880c6ec78 kubernetes.io/csi/ebs.csi.aws.com^vol-0d3b1f8fbfc34dfdb],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/csi-hostpath-ephemeral-6886^4e266409-8221-11ed-800d-8a8e987c6028,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-0b74ae25880c6ec78,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/csi-hostpath-ephemeral-8407^9fd9423f-821f-11ed-b5cc-02642e84c89b,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-0ae1955ea2e2f0ecd,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-0d3b1f8fbfc34dfdb,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-09308d632dcd25d39,DevicePath:,},},Config:nil,},} ... skipping 219 lines ... Latency metrics for node i-0ea878d3671f2eda2 [DeferCleanup (Each)] [sig-storage] Subpath tear down framework | framework.go:193 [1mSTEP:[0m Destroying namespace "subpath-4983" for this suite. [38;5;243m12/22/22 17:58:06.315[0m [38;5;243m<< End Captured GinkgoWriter Output[0m [38;5;9mDec 22 17:58:03.497: expected pod "pod-subpath-test-downwardapi-dr4h" success: timed out while waiting for pod subpath-4983/pod-subpath-test-downwardapi-dr4h to be Succeeded or Failed[0m [38;5;9mIn [1m[It][0m[38;5;9m at: [1mtest/e2e/framework/pod/output/output.go:237[0m [38;5;243m------------------------------[0m [38;5;14mS [SKIPPED] [0.000 seconds][0m [sig-storage] In-tree Volumes [38;5;243mtest/e2e/storage/utils/framework.go:23[0m [Driver: azure-disk] ... skipping 513 lines ... [sig-storage] In-tree Volumes [38;5;243mtest/e2e/storage/utils/framework.go:23[0m [Driver: hostPath] [38;5;243mtest/e2e/storage/in_tree_volumes.go:85[0m [38;5;14m[1m[Testpattern: Dynamic PV (delayed binding)] topology [BeforeEach][0m [38;5;243mtest/e2e/storage/framework/testsuite.go:51[0m should fail to schedule a pod which has topologies that conflict with AllowedTopologies [38;5;243mtest/e2e/storage/testsuites/topology.go:191[0m [38;5;243mBegin Captured GinkgoWriter Output >>[0m [BeforeEach] [Testpattern: Dynamic PV (delayed binding)] topology test/e2e/storage/framework/testsuite.go:51 Dec 22 17:58:26.217: INFO: Driver hostPath doesn't support DynamicPV -- skipping ... skipping 460 lines ... [38;5;243mtest/e2e/storage/ephemeral_volume.go:58[0m [38;5;243m------------------------------[0m [38;5;10m• [SLOW TEST] [156.163 seconds][0m [0m[sig-storage] CSI mock volume [38;5;243mCSI online volume expansion [0mshould expand volume without restarting pod if attach=off, nodeExpansion=on[0m [38;5;243mtest/e2e/storage/csi_mock_volume.go:799[0m [38;5;243m------------------------------[0m [38;5;9m• [FAILED] [304.498 seconds][0m [38;5;9m[1m[sig-api-machinery] CustomResourceConversionWebhook [Privileged:ClusterAdmin] [BeforeEach][0m [38;5;243mtest/e2e/apimachinery/crd_conversion_webhook.go:128[0m should be able to convert from CR v1 to CR v2 [Conformance] [38;5;243mtest/e2e/apimachinery/crd_conversion_webhook.go:149[0m [38;5;243mBegin Captured GinkgoWriter Output >>[0m ... skipping 161 lines ... Dec 22 18:01:36.181: INFO: deployment status: v1.DeploymentStatus{ObservedGeneration:1, Replicas:1, UpdatedReplicas:1, ReadyReplicas:0, AvailableReplicas:0, UnavailableReplicas:1, Conditions:[]v1.DeploymentCondition{v1.DeploymentCondition{Type:"Available", Status:"False", LastUpdateTime:time.Date(2022, time.December, 22, 17, 56, 43, 0, time.Local), LastTransitionTime:time.Date(2022, time.December, 22, 17, 56, 43, 0, time.Local), Reason:"MinimumReplicasUnavailable", Message:"Deployment does not have minimum availability."}, v1.DeploymentCondition{Type:"Progressing", Status:"True", LastUpdateTime:time.Date(2022, time.December, 22, 17, 56, 43, 0, time.Local), LastTransitionTime:time.Date(2022, time.December, 22, 17, 56, 43, 0, time.Local), Reason:"ReplicaSetUpdated", Message:"ReplicaSet \"sample-crd-conversion-webhook-deployment-74ff66dd47\" is progressing."}}, CollisionCount:(*int32)(nil)} Dec 22 18:01:38.181: INFO: deployment status: v1.DeploymentStatus{ObservedGeneration:1, Replicas:1, UpdatedReplicas:1, ReadyReplicas:0, AvailableReplicas:0, UnavailableReplicas:1, Conditions:[]v1.DeploymentCondition{v1.DeploymentCondition{Type:"Available", Status:"False", LastUpdateTime:time.Date(2022, time.December, 22, 17, 56, 43, 0, time.Local), LastTransitionTime:time.Date(2022, time.December, 22, 17, 56, 43, 0, time.Local), Reason:"MinimumReplicasUnavailable", Message:"Deployment does not have minimum availability."}, v1.DeploymentCondition{Type:"Progressing", Status:"True", LastUpdateTime:time.Date(2022, time.December, 22, 17, 56, 43, 0, time.Local), LastTransitionTime:time.Date(2022, time.December, 22, 17, 56, 43, 0, time.Local), Reason:"ReplicaSetUpdated", Message:"ReplicaSet \"sample-crd-conversion-webhook-deployment-74ff66dd47\" is progressing."}}, CollisionCount:(*int32)(nil)} Dec 22 18:01:40.181: INFO: deployment status: v1.DeploymentStatus{ObservedGeneration:1, Replicas:1, UpdatedReplicas:1, ReadyReplicas:0, AvailableReplicas:0, UnavailableReplicas:1, Conditions:[]v1.DeploymentCondition{v1.DeploymentCondition{Type:"Available", Status:"False", LastUpdateTime:time.Date(2022, time.December, 22, 17, 56, 43, 0, time.Local), LastTransitionTime:time.Date(2022, time.December, 22, 17, 56, 43, 0, time.Local), Reason:"MinimumReplicasUnavailable", Message:"Deployment does not have minimum availability."}, v1.DeploymentCondition{Type:"Progressing", Status:"True", LastUpdateTime:time.Date(2022, time.December, 22, 17, 56, 43, 0, time.Local), LastTransitionTime:time.Date(2022, time.December, 22, 17, 56, 43, 0, time.Local), Reason:"ReplicaSetUpdated", Message:"ReplicaSet \"sample-crd-conversion-webhook-deployment-74ff66dd47\" is progressing."}}, CollisionCount:(*int32)(nil)} Dec 22 18:01:42.181: INFO: deployment status: v1.DeploymentStatus{ObservedGeneration:1, Replicas:1, UpdatedReplicas:1, ReadyReplicas:0, AvailableReplicas:0, UnavailableReplicas:1, Conditions:[]v1.DeploymentCondition{v1.DeploymentCondition{Type:"Available", Status:"False", LastUpdateTime:time.Date(2022, time.December, 22, 17, 56, 43, 0, time.Local), LastTransitionTime:time.Date(2022, time.December, 22, 17, 56, 43, 0, time.Local), Reason:"MinimumReplicasUnavailable", Message:"Deployment does not have minimum availability."}, v1.DeploymentCondition{Type:"Progressing", Status:"True", LastUpdateTime:time.Date(2022, time.December, 22, 17, 56, 43, 0, time.Local), LastTransitionTime:time.Date(2022, time.December, 22, 17, 56, 43, 0, time.Local), Reason:"ReplicaSetUpdated", Message:"ReplicaSet \"sample-crd-conversion-webhook-deployment-74ff66dd47\" is progressing."}}, CollisionCount:(*int32)(nil)} Dec 22 18:01:44.184: INFO: deployment status: v1.DeploymentStatus{ObservedGeneration:1, Replicas:1, UpdatedReplicas:1, ReadyReplicas:0, AvailableReplicas:0, UnavailableReplicas:1, Conditions:[]v1.DeploymentCondition{v1.DeploymentCondition{Type:"Available", Status:"False", LastUpdateTime:time.Date(2022, time.December, 22, 17, 56, 43, 0, time.Local), LastTransitionTime:time.Date(2022, time.December, 22, 17, 56, 43, 0, time.Local), Reason:"MinimumReplicasUnavailable", Message:"Deployment does not have minimum availability."}, v1.DeploymentCondition{Type:"Progressing", Status:"True", LastUpdateTime:time.Date(2022, time.December, 22, 17, 56, 43, 0, time.Local), LastTransitionTime:time.Date(2022, time.December, 22, 17, 56, 43, 0, time.Local), Reason:"ReplicaSetUpdated", Message:"ReplicaSet \"sample-crd-conversion-webhook-deployment-74ff66dd47\" is progressing."}}, CollisionCount:(*int32)(nil)} Dec 22 18:01:44.250: INFO: deployment status: v1.DeploymentStatus{ObservedGeneration:1, Replicas:1, UpdatedReplicas:1, ReadyReplicas:0, AvailableReplicas:0, UnavailableReplicas:1, Conditions:[]v1.DeploymentCondition{v1.DeploymentCondition{Type:"Available", Status:"False", LastUpdateTime:time.Date(2022, time.December, 22, 17, 56, 43, 0, time.Local), LastTransitionTime:time.Date(2022, time.December, 22, 17, 56, 43, 0, time.Local), Reason:"MinimumReplicasUnavailable", Message:"Deployment does not have minimum availability."}, v1.DeploymentCondition{Type:"Progressing", Status:"True", LastUpdateTime:time.Date(2022, time.December, 22, 17, 56, 43, 0, time.Local), LastTransitionTime:time.Date(2022, time.December, 22, 17, 56, 43, 0, time.Local), Reason:"ReplicaSetUpdated", Message:"ReplicaSet \"sample-crd-conversion-webhook-deployment-74ff66dd47\" is progressing."}}, CollisionCount:(*int32)(nil)} Dec 22 18:01:44.250: INFO: Unexpected error: waiting for sample-crd-conversion-webhook-deployment deployment status valid: <*errors.errorString | 0xc005a8ec90>: { s: "error waiting for deployment \"sample-crd-conversion-webhook-deployment\" status to match expectation: deployment status: v1.DeploymentStatus{ObservedGeneration:1, Replicas:1, UpdatedReplicas:1, ReadyReplicas:0, AvailableReplicas:0, UnavailableReplicas:1, Conditions:[]v1.DeploymentCondition{v1.DeploymentCondition{Type:\"Available\", Status:\"False\", LastUpdateTime:time.Date(2022, time.December, 22, 17, 56, 43, 0, time.Local), LastTransitionTime:time.Date(2022, time.December, 22, 17, 56, 43, 0, time.Local), Reason:\"MinimumReplicasUnavailable\", Message:\"Deployment does not have minimum availability.\"}, v1.DeploymentCondition{Type:\"Progressing\", Status:\"True\", LastUpdateTime:time.Date(2022, time.December, 22, 17, 56, 43, 0, time.Local), LastTransitionTime:time.Date(2022, time.December, 22, 17, 56, 43, 0, time.Local), Reason:\"ReplicaSetUpdated\", Message:\"ReplicaSet \\\"sample-crd-conversion-webhook-deployment-74ff66dd47\\\" is progressing.\"}}, CollisionCount:(*int32)(nil)}", } Dec 22 18:01:44.250: FAIL: waiting for sample-crd-conversion-webhook-deployment deployment status valid: error waiting for deployment "sample-crd-conversion-webhook-deployment" status to match expectation: deployment status: v1.DeploymentStatus{ObservedGeneration:1, Replicas:1, UpdatedReplicas:1, ReadyReplicas:0, AvailableReplicas:0, UnavailableReplicas:1, Conditions:[]v1.DeploymentCondition{v1.DeploymentCondition{Type:"Available", Status:"False", LastUpdateTime:time.Date(2022, time.December, 22, 17, 56, 43, 0, time.Local), LastTransitionTime:time.Date(2022, time.December, 22, 17, 56, 43, 0, time.Local), Reason:"MinimumReplicasUnavailable", Message:"Deployment does not have minimum availability."}, v1.DeploymentCondition{Type:"Progressing", Status:"True", LastUpdateTime:time.Date(2022, time.December, 22, 17, 56, 43, 0, time.Local), LastTransitionTime:time.Date(2022, time.December, 22, 17, 56, 43, 0, time.Local), Reason:"ReplicaSetUpdated", Message:"ReplicaSet \"sample-crd-conversion-webhook-deployment-74ff66dd47\" is progressing."}}, CollisionCount:(*int32)(nil)} Full Stack Trace k8s.io/kubernetes/test/e2e/apimachinery.deployCustomResourceWebhookAndService(0xc000c79950, {0xc003a160f0, 0x2c}, 0xc004f62910, 0x24e3, 0x24e4) test/e2e/apimachinery/crd_conversion_webhook.go:327 +0xe19 k8s.io/kubernetes/test/e2e/apimachinery.glob..func5.1() test/e2e/apimachinery/crd_conversion_webhook.go:136 +0x206 ... skipping 9 lines ... [1mSTEP:[0m dump namespace information after failure [38;5;243m12/22/22 18:01:44.586[0m [1mSTEP:[0m Collecting events from namespace "crd-webhook-3312". [38;5;243m12/22/22 18:01:44.587[0m [1mSTEP:[0m Found 16 events. [38;5;243m12/22/22 18:01:44.652[0m Dec 22 18:01:44.652: INFO: At 2022-12-22 17:56:43 +0000 UTC - event for sample-crd-conversion-webhook-deployment: {deployment-controller } ScalingReplicaSet: Scaled up replica set sample-crd-conversion-webhook-deployment-74ff66dd47 to 1 Dec 22 18:01:44.652: INFO: At 2022-12-22 17:56:43 +0000 UTC - event for sample-crd-conversion-webhook-deployment-74ff66dd47: {replicaset-controller } SuccessfulCreate: Created pod: sample-crd-conversion-webhook-deployment-74ff66dd47-qhvwm Dec 22 18:01:44.652: INFO: At 2022-12-22 17:56:43 +0000 UTC - event for sample-crd-conversion-webhook-deployment-74ff66dd47-qhvwm: {default-scheduler } Scheduled: Successfully assigned crd-webhook-3312/sample-crd-conversion-webhook-deployment-74ff66dd47-qhvwm to i-0243def68c064e8ef Dec 22 18:01:44.652: INFO: At 2022-12-22 17:56:44 +0000 UTC - event for sample-crd-conversion-webhook-deployment-74ff66dd47-qhvwm: {kubelet i-0243def68c064e8ef} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "3e3f5fe52ab17547af19298efc4b232ed32aa0c2940fa9efcad66fa5a0956921": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 18:01:44.652: INFO: At 2022-12-22 17:56:58 +0000 UTC - event for sample-crd-conversion-webhook-deployment-74ff66dd47-qhvwm: {kubelet i-0243def68c064e8ef} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "7f51dd6710f49e9d7d68fa04c7dc02165e70e49a215aa2fe43ae04106dc0a28a": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 18:01:44.652: INFO: At 2022-12-22 17:57:12 +0000 UTC - event for sample-crd-conversion-webhook-deployment-74ff66dd47-qhvwm: {kubelet i-0243def68c064e8ef} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "96ff5cb695920e4a7f0ec0112eaeec56d1ed74f268c0500cbf649340c5bfae37": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 18:01:44.652: INFO: At 2022-12-22 17:57:25 +0000 UTC - event for sample-crd-conversion-webhook-deployment-74ff66dd47-qhvwm: {kubelet i-0243def68c064e8ef} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "8a468a63009c7f79a70180dce47134dfd4ccaeb69698a0e052cdf84870c0a078": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 18:01:44.652: INFO: At 2022-12-22 17:57:38 +0000 UTC - event for sample-crd-conversion-webhook-deployment-74ff66dd47-qhvwm: {kubelet i-0243def68c064e8ef} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "8b19b97015f2b3596b135ca25304d80d1a28f792afd55320c5959cf17f4ce630": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 18:01:44.652: INFO: At 2022-12-22 17:57:50 +0000 UTC - event for sample-crd-conversion-webhook-deployment-74ff66dd47-qhvwm: {kubelet i-0243def68c064e8ef} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "9cd47b3a37a2ad7a3340950e6792f783ce4933c2f530e2077f334c023320378c": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 18:01:44.652: INFO: At 2022-12-22 17:58:05 +0000 UTC - event for sample-crd-conversion-webhook-deployment-74ff66dd47-qhvwm: {kubelet i-0243def68c064e8ef} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "9650ac685d040b2e9718bdbb12c46098579c49d54b47c8c6397c738581ed2961": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 18:01:44.652: INFO: At 2022-12-22 17:58:16 +0000 UTC - event for sample-crd-conversion-webhook-deployment-74ff66dd47-qhvwm: {kubelet i-0243def68c064e8ef} Pulled: Container image "registry.k8s.io/e2e-test-images/agnhost:2.43" already present on machine Dec 22 18:01:44.652: INFO: At 2022-12-22 17:58:17 +0000 UTC - event for sample-crd-conversion-webhook-deployment-74ff66dd47-qhvwm: {kubelet i-0243def68c064e8ef} Started: Started container sample-crd-conversion-webhook Dec 22 18:01:44.652: INFO: At 2022-12-22 17:58:17 +0000 UTC - event for sample-crd-conversion-webhook-deployment-74ff66dd47-qhvwm: {kubelet i-0243def68c064e8ef} Created: Created container sample-crd-conversion-webhook Dec 22 18:01:44.652: INFO: At 2022-12-22 17:58:19 +0000 UTC - event for sample-crd-conversion-webhook-deployment-74ff66dd47-qhvwm: {kubelet i-0243def68c064e8ef} Unhealthy: Readiness probe failed: Get "https://172.20.57.60:9444/readyz": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) Dec 22 18:01:44.652: INFO: At 2022-12-22 17:58:29 +0000 UTC - event for sample-crd-conversion-webhook-deployment-74ff66dd47-qhvwm: {kubelet i-0243def68c064e8ef} Unhealthy: Readiness probe failed: Get "https://172.20.57.60:9444/readyz": context deadline exceeded Dec 22 18:01:44.652: INFO: At 2022-12-22 18:01:44 +0000 UTC - event for sample-crd-conversion-webhook-deployment-74ff66dd47-qhvwm: {kubelet i-0243def68c064e8ef} Killing: Stopping container sample-crd-conversion-webhook Dec 22 18:01:44.717: INFO: POD NODE PHASE GRACE CONDITIONS Dec 22 18:01:44.717: INFO: Dec 22 18:01:44.783: INFO: Logging node info for node i-0243def68c064e8ef Dec 22 18:01:44.849: INFO: Node Info: &Node{ObjectMeta:{i-0243def68c064e8ef f07b8771-29eb-4944-a385-3482a3d3a315 52684 0 2022-12-22 17:17:47 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:us-west-2 failure-domain.beta.kubernetes.io/zone:us-west-2a kubernetes.io/arch:amd64 kubernetes.io/hostname:i-0243def68c064e8ef kubernetes.io/os:linux node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:us-west-2a topology.hostpath.csi/node:i-0243def68c064e8ef topology.kubernetes.io/region:us-west-2 topology.kubernetes.io/zone:us-west-2a] map[alpha.kubernetes.io/provided-node-ip:172.20.35.142 csi.volume.kubernetes.io/nodeid:{"csi-hostpath-ephemeral-6886":"i-0243def68c064e8ef","csi-hostpath-ephemeral-8407":"i-0243def68c064e8ef","ebs.csi.aws.com":"i-0243def68c064e8ef"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{aws-cloud-controller-manager Update v1 2022-12-22 17:17:47 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2022-12-22 17:17:47 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kops-controller Update v1 2022-12-22 17:17:47 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{}}}} } {kube-controller-manager Update v1 2022-12-22 17:17:47 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.3.0/24\"":{}}}} } {kubelet Update v1 2022-12-22 17:17:47 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:alpha.kubernetes.io/provided-node-ip":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {kube-controller-manager Update v1 2022-12-22 17:59:50 +0000 UTC FieldsV1 {"f:status":{"f:volumesAttached":{}}} status} {kubelet Update v1 2022-12-22 17:59:50 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.3.0/24,DoNotUseExternalID:,ProviderID:aws:///us-west-2a/i-0243def68c064e8ef,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.3.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{51527004160 0} {<nil>} 50319340Ki BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4050636800 0} {<nil>} 3955700Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{46374303668 0} {<nil>} 46374303668 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3945779200 0} {<nil>} 3853300Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-12-22 17:59:50 +0000 UTC,LastTransitionTime:2022-12-22 17:17:37 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-12-22 17:59:50 +0000 UTC,LastTransitionTime:2022-12-22 17:17:37 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-12-22 17:59:50 +0000 UTC,LastTransitionTime:2022-12-22 17:17:37 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-12-22 17:59:50 +0000 UTC,LastTransitionTime:2022-12-22 17:17:59 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.35.142,},NodeAddress{Type:ExternalIP,Address:35.90.235.118,},NodeAddress{Type:InternalDNS,Address:i-0243def68c064e8ef.us-west-2.compute.internal,},NodeAddress{Type:Hostname,Address:i-0243def68c064e8ef.us-west-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-35-90-235-118.us-west-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2c8ae28bb7834ee5f6b4c4ef144a8e,SystemUUID:ec2c8ae2-8bb7-834e-e5f6-b4c4ef144a8e,BootID:5843895c-91be-4c36-b700-9e551cb745de,KernelVersion:5.10.157-139.675.amzn2.x86_64,OSImage:Amazon Linux 2,ContainerRuntimeVersion:containerd://1.6.14,KubeletVersion:v1.26.0,KubeProxyVersion:v1.26.0,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:166719855,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/jessie-dnsutils@sha256:24aaf2626d6b27864c29de2097e8bbb840b3a414271bf7c8995e431e47d8408e registry.k8s.io/e2e-test-images/jessie-dnsutils:1.7],SizeBytes:112030336,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:dd75ec974b0a2a6f6bb47001ba09207976e625db898d1b16735528c009cb171c registry.k8s.io/etcd:3.5.6-0],SizeBytes:102542580,},ContainerImage{Names:[registry.k8s.io/kube-proxy-amd64:v1.26.0],SizeBytes:67205320,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:16bbf38c463a4223d8cfe4da12bc61010b082a79b4bb003e2d3ba3ece5dd5f9e registry.k8s.io/e2e-test-images/agnhost:2.43],SizeBytes:51706353,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nautilus@sha256:80ba6c8c44f9623f06e868a1aa66026c8ec438ad814f9ec95e9333b415fe3550 registry.k8s.io/e2e-test-images/nautilus:1.7],SizeBytes:49641698,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:3fe7acf013d1264ffded116b80a73dc129a449b0fccdb8d21af8279f2233f36e registry.k8s.io/e2e-test-images/httpd:2.4.39-4],SizeBytes:41901587,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:148b022f5c5da426fc2f3c14b5c0867e58ef05961510c84749ac1fddcb0fef22 registry.k8s.io/e2e-test-images/httpd:2.4.38-4],SizeBytes:40764257,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:10e231cf07c82dfab5141c1afe548dc734e5fc1f67665ec3982a325d1bd31a9a registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.0],SizeBytes:29725756,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:ee3b525d5b89db99da3b8eb521d9cd90cb6e9ef0fbb651e98bb37be78d36b5b8 registry.k8s.io/sig-storage/csi-provisioner:v3.3.0],SizeBytes:25491225,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:425d8f1b769398127767b06ed97ce62578a3179bcb99809ce93a1649e025ffe7 registry.k8s.io/sig-storage/csi-resizer:v1.6.0],SizeBytes:24148884,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:291334908ddf71a4661fd7f6d9d97274de8a5378a2b6fdfeb2ce73414a34f82f registry.k8s.io/sig-storage/csi-snapshotter:v6.1.0],SizeBytes:23881995,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:9a685020911e2725ad019dbce6e4a5ab93d51e3d4557f115e64343345e05781b registry.k8s.io/sig-storage/csi-attacher:v4.0.0],SizeBytes:23847201,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:92257881c1d6493cf18299a24af42330f891166560047902b8d431fb66b01af5 registry.k8s.io/sig-storage/hostpathplugin:v1.9.0],SizeBytes:18758628,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nonroot@sha256:ee9f50b3c64b174d296d91ca9f69a914ac30e59095dfb462b2b518ad28a63655 registry.k8s.io/e2e-test-images/nonroot:1.4],SizeBytes:17747885,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:933940f13b3ea0abc62e656c1aa5c5b47c04b15d71250413a6b821bd0c58b94e registry.k8s.io/sig-storage/livenessprobe:v2.7.0],SizeBytes:8688564,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:5c99cf6a02adda929b10321dbf4ecfa00d87be9ba4fb456006237d530ab4baa1 registry.k8s.io/e2e-test-images/nginx:1.14-4],SizeBytes:6978614,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nonewprivs@sha256:8ac1264691820febacf3aea5d152cbde6d10685731ec14966a9401c6f47a68ac registry.k8s.io/e2e-test-images/nonewprivs:1.3],SizeBytes:3263463,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf registry.k8s.io/e2e-test-images/busybox:1.29-2],SizeBytes:732424,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:2e0f836850e09b8b7cc937681d6194537a09fbd5f6b9e08f4d646a85128e8937 registry.k8s.io/e2e-test-images/busybox:1.29-4],SizeBytes:731990,},ContainerImage{Names:[registry.k8s.io/pause@sha256:7031c1b283388d2c2e09b57badb803c05ebed362dc88d84b480cc47f72a21097 registry.k8s.io/pause:3.9],SizeBytes:321520,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[kubernetes.io/csi/csi-hostpath-ephemeral-6886^4e266409-8221-11ed-800d-8a8e987c6028 kubernetes.io/csi/csi-hostpath-ephemeral-8407^9fd9423f-821f-11ed-b5cc-02642e84c89b],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/csi-hostpath-ephemeral-8407^9fd9423f-821f-11ed-b5cc-02642e84c89b,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/csi-hostpath-ephemeral-6886^4e266409-8221-11ed-800d-8a8e987c6028,DevicePath:,},},Config:nil,},} ... skipping 179 lines ... Latency metrics for node i-0ea878d3671f2eda2 [DeferCleanup (Each)] [sig-api-machinery] CustomResourceConversionWebhook [Privileged:ClusterAdmin] tear down framework | framework.go:193 [1mSTEP:[0m Destroying namespace "crd-webhook-3312" for this suite. [38;5;243m12/22/22 18:01:47.485[0m [38;5;243m<< End Captured GinkgoWriter Output[0m [38;5;9mDec 22 18:01:44.250: waiting for sample-crd-conversion-webhook-deployment deployment status valid: error waiting for deployment "sample-crd-conversion-webhook-deployment" status to match expectation: deployment status: v1.DeploymentStatus{ObservedGeneration:1, Replicas:1, UpdatedReplicas:1, ReadyReplicas:0, AvailableReplicas:0, UnavailableReplicas:1, Conditions:[]v1.DeploymentCondition{v1.DeploymentCondition{Type:"Available", Status:"False", LastUpdateTime:time.Date(2022, time.December, 22, 17, 56, 43, 0, time.Local), LastTransitionTime:time.Date(2022, time.December, 22, 17, 56, 43, 0, time.Local), Reason:"MinimumReplicasUnavailable", Message:"Deployment does not have minimum availability."}, v1.DeploymentCondition{Type:"Progressing", Status:"True", LastUpdateTime:time.Date(2022, time.December, 22, 17, 56, 43, 0, time.Local), LastTransitionTime:time.Date(2022, time.December, 22, 17, 56, 43, 0, time.Local), Reason:"ReplicaSetUpdated", Message:"ReplicaSet \"sample-crd-conversion-webhook-deployment-74ff66dd47\" is progressing."}}, CollisionCount:(*int32)(nil)}[0m [38;5;9mIn [1m[BeforeEach][0m[38;5;9m at: [1mtest/e2e/apimachinery/crd_conversion_webhook.go:327[0m [38;5;243m------------------------------[0m [38;5;9m• [FAILED] [307.384 seconds][0m [38;5;9m[1m[sig-api-machinery] AdmissionWebhook [Privileged:ClusterAdmin] [BeforeEach][0m [38;5;243mtest/e2e/apimachinery/webhook.go:90[0m listing mutating webhooks should work [Conformance] [38;5;243mtest/e2e/apimachinery/webhook.go:656[0m [38;5;243mBegin Captured GinkgoWriter Output >>[0m ... skipping 161 lines ... Dec 22 18:02:39.233: INFO: deployment status: v1.DeploymentStatus{ObservedGeneration:1, Replicas:1, UpdatedReplicas:1, ReadyReplicas:0, AvailableReplicas:0, UnavailableReplicas:1, Conditions:[]v1.DeploymentCondition{v1.DeploymentCondition{Type:"Available", Status:"False", LastUpdateTime:time.Date(2022, time.December, 22, 17, 57, 46, 0, time.Local), LastTransitionTime:time.Date(2022, time.December, 22, 17, 57, 46, 0, time.Local), Reason:"MinimumReplicasUnavailable", Message:"Deployment does not have minimum availability."}, v1.DeploymentCondition{Type:"Progressing", Status:"True", LastUpdateTime:time.Date(2022, time.December, 22, 17, 57, 46, 0, time.Local), LastTransitionTime:time.Date(2022, time.December, 22, 17, 57, 46, 0, time.Local), Reason:"ReplicaSetUpdated", Message:"ReplicaSet \"sample-webhook-deployment-865554f4d9\" is progressing."}}, CollisionCount:(*int32)(nil)} Dec 22 18:02:41.235: INFO: deployment status: v1.DeploymentStatus{ObservedGeneration:1, Replicas:1, UpdatedReplicas:1, ReadyReplicas:0, AvailableReplicas:0, UnavailableReplicas:1, Conditions:[]v1.DeploymentCondition{v1.DeploymentCondition{Type:"Available", Status:"False", LastUpdateTime:time.Date(2022, time.December, 22, 17, 57, 46, 0, time.Local), LastTransitionTime:time.Date(2022, time.December, 22, 17, 57, 46, 0, time.Local), Reason:"MinimumReplicasUnavailable", Message:"Deployment does not have minimum availability."}, v1.DeploymentCondition{Type:"Progressing", Status:"True", LastUpdateTime:time.Date(2022, time.December, 22, 17, 57, 46, 0, time.Local), LastTransitionTime:time.Date(2022, time.December, 22, 17, 57, 46, 0, time.Local), Reason:"ReplicaSetUpdated", Message:"ReplicaSet \"sample-webhook-deployment-865554f4d9\" is progressing."}}, CollisionCount:(*int32)(nil)} Dec 22 18:02:43.234: INFO: deployment status: v1.DeploymentStatus{ObservedGeneration:1, Replicas:1, UpdatedReplicas:1, ReadyReplicas:0, AvailableReplicas:0, UnavailableReplicas:1, Conditions:[]v1.DeploymentCondition{v1.DeploymentCondition{Type:"Available", Status:"False", LastUpdateTime:time.Date(2022, time.December, 22, 17, 57, 46, 0, time.Local), LastTransitionTime:time.Date(2022, time.December, 22, 17, 57, 46, 0, time.Local), Reason:"MinimumReplicasUnavailable", Message:"Deployment does not have minimum availability."}, v1.DeploymentCondition{Type:"Progressing", Status:"True", LastUpdateTime:time.Date(2022, time.December, 22, 17, 57, 46, 0, time.Local), LastTransitionTime:time.Date(2022, time.December, 22, 17, 57, 46, 0, time.Local), Reason:"ReplicaSetUpdated", Message:"ReplicaSet \"sample-webhook-deployment-865554f4d9\" is progressing."}}, CollisionCount:(*int32)(nil)} Dec 22 18:02:45.235: INFO: deployment status: v1.DeploymentStatus{ObservedGeneration:1, Replicas:1, UpdatedReplicas:1, ReadyReplicas:0, AvailableReplicas:0, UnavailableReplicas:1, Conditions:[]v1.DeploymentCondition{v1.DeploymentCondition{Type:"Available", Status:"False", LastUpdateTime:time.Date(2022, time.December, 22, 17, 57, 46, 0, time.Local), LastTransitionTime:time.Date(2022, time.December, 22, 17, 57, 46, 0, time.Local), Reason:"MinimumReplicasUnavailable", Message:"Deployment does not have minimum availability."}, v1.DeploymentCondition{Type:"Progressing", Status:"True", LastUpdateTime:time.Date(2022, time.December, 22, 17, 57, 46, 0, time.Local), LastTransitionTime:time.Date(2022, time.December, 22, 17, 57, 46, 0, time.Local), Reason:"ReplicaSetUpdated", Message:"ReplicaSet \"sample-webhook-deployment-865554f4d9\" is progressing."}}, CollisionCount:(*int32)(nil)} Dec 22 18:02:47.234: INFO: deployment status: v1.DeploymentStatus{ObservedGeneration:1, Replicas:1, UpdatedReplicas:1, ReadyReplicas:0, AvailableReplicas:0, UnavailableReplicas:1, Conditions:[]v1.DeploymentCondition{v1.DeploymentCondition{Type:"Available", Status:"False", LastUpdateTime:time.Date(2022, time.December, 22, 17, 57, 46, 0, time.Local), LastTransitionTime:time.Date(2022, time.December, 22, 17, 57, 46, 0, time.Local), Reason:"MinimumReplicasUnavailable", Message:"Deployment does not have minimum availability."}, v1.DeploymentCondition{Type:"Progressing", Status:"True", LastUpdateTime:time.Date(2022, time.December, 22, 17, 57, 46, 0, time.Local), LastTransitionTime:time.Date(2022, time.December, 22, 17, 57, 46, 0, time.Local), Reason:"ReplicaSetUpdated", Message:"ReplicaSet \"sample-webhook-deployment-865554f4d9\" is progressing."}}, CollisionCount:(*int32)(nil)} Dec 22 18:02:47.298: INFO: deployment status: v1.DeploymentStatus{ObservedGeneration:1, Replicas:1, UpdatedReplicas:1, ReadyReplicas:0, AvailableReplicas:0, UnavailableReplicas:1, Conditions:[]v1.DeploymentCondition{v1.DeploymentCondition{Type:"Available", Status:"False", LastUpdateTime:time.Date(2022, time.December, 22, 17, 57, 46, 0, time.Local), LastTransitionTime:time.Date(2022, time.December, 22, 17, 57, 46, 0, time.Local), Reason:"MinimumReplicasUnavailable", Message:"Deployment does not have minimum availability."}, v1.DeploymentCondition{Type:"Progressing", Status:"True", LastUpdateTime:time.Date(2022, time.December, 22, 17, 57, 46, 0, time.Local), LastTransitionTime:time.Date(2022, time.December, 22, 17, 57, 46, 0, time.Local), Reason:"ReplicaSetUpdated", Message:"ReplicaSet \"sample-webhook-deployment-865554f4d9\" is progressing."}}, CollisionCount:(*int32)(nil)} Dec 22 18:02:47.298: INFO: Unexpected error: waiting for the deployment status valid%!(EXTRA string=registry.k8s.io/e2e-test-images/agnhost:2.43, string=sample-webhook-deployment, string=webhook-5645): <*errors.errorString | 0xc00178d620>: { s: "error waiting for deployment \"sample-webhook-deployment\" status to match expectation: deployment status: v1.DeploymentStatus{ObservedGeneration:1, Replicas:1, UpdatedReplicas:1, ReadyReplicas:0, AvailableReplicas:0, UnavailableReplicas:1, Conditions:[]v1.DeploymentCondition{v1.DeploymentCondition{Type:\"Available\", Status:\"False\", LastUpdateTime:time.Date(2022, time.December, 22, 17, 57, 46, 0, time.Local), LastTransitionTime:time.Date(2022, time.December, 22, 17, 57, 46, 0, time.Local), Reason:\"MinimumReplicasUnavailable\", Message:\"Deployment does not have minimum availability.\"}, v1.DeploymentCondition{Type:\"Progressing\", Status:\"True\", LastUpdateTime:time.Date(2022, time.December, 22, 17, 57, 46, 0, time.Local), LastTransitionTime:time.Date(2022, time.December, 22, 17, 57, 46, 0, time.Local), Reason:\"ReplicaSetUpdated\", Message:\"ReplicaSet \\\"sample-webhook-deployment-865554f4d9\\\" is progressing.\"}}, CollisionCount:(*int32)(nil)}", } Dec 22 18:02:47.299: FAIL: waiting for the deployment status valid%!(EXTRA string=registry.k8s.io/e2e-test-images/agnhost:2.43, string=sample-webhook-deployment, string=webhook-5645): error waiting for deployment "sample-webhook-deployment" status to match expectation: deployment status: v1.DeploymentStatus{ObservedGeneration:1, Replicas:1, UpdatedReplicas:1, ReadyReplicas:0, AvailableReplicas:0, UnavailableReplicas:1, Conditions:[]v1.DeploymentCondition{v1.DeploymentCondition{Type:"Available", Status:"False", LastUpdateTime:time.Date(2022, time.December, 22, 17, 57, 46, 0, time.Local), LastTransitionTime:time.Date(2022, time.December, 22, 17, 57, 46, 0, time.Local), Reason:"MinimumReplicasUnavailable", Message:"Deployment does not have minimum availability."}, v1.DeploymentCondition{Type:"Progressing", Status:"True", LastUpdateTime:time.Date(2022, time.December, 22, 17, 57, 46, 0, time.Local), LastTransitionTime:time.Date(2022, time.December, 22, 17, 57, 46, 0, time.Local), Reason:"ReplicaSetUpdated", Message:"ReplicaSet \"sample-webhook-deployment-865554f4d9\" is progressing."}}, CollisionCount:(*int32)(nil)} Full Stack Trace k8s.io/kubernetes/test/e2e/apimachinery.deployWebhookAndService(0xc000e15ef0, {0xc0027e75f0, 0x2c}, 0xc00535ccd0, 0x20fb, 0x20fc) test/e2e/apimachinery/webhook.go:826 +0xed2 k8s.io/kubernetes/test/e2e/apimachinery.glob..func28.1() test/e2e/apimachinery/webhook.go:102 +0x226 ... skipping 9 lines ... [1mSTEP:[0m dump namespace information after failure [38;5;243m12/22/22 18:02:47.639[0m [1mSTEP:[0m Collecting events from namespace "webhook-5645". [38;5;243m12/22/22 18:02:47.639[0m [1mSTEP:[0m Found 10 events. [38;5;243m12/22/22 18:02:47.703[0m Dec 22 18:02:47.704: INFO: At 2022-12-22 17:57:46 +0000 UTC - event for sample-webhook-deployment: {deployment-controller } ScalingReplicaSet: Scaled up replica set sample-webhook-deployment-865554f4d9 to 1 Dec 22 18:02:47.704: INFO: At 2022-12-22 17:57:46 +0000 UTC - event for sample-webhook-deployment-865554f4d9: {replicaset-controller } SuccessfulCreate: Created pod: sample-webhook-deployment-865554f4d9-j2nt4 Dec 22 18:02:47.704: INFO: At 2022-12-22 17:57:46 +0000 UTC - event for sample-webhook-deployment-865554f4d9-j2nt4: {default-scheduler } Scheduled: Successfully assigned webhook-5645/sample-webhook-deployment-865554f4d9-j2nt4 to i-0c76fbcc0b95be727 Dec 22 18:02:47.704: INFO: At 2022-12-22 17:57:47 +0000 UTC - event for sample-webhook-deployment-865554f4d9-j2nt4: {kubelet i-0c76fbcc0b95be727} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "84b2c8cdebc435cb9ee2194074f8e83e09980b2a175c87b00fc0993b338041f0": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 18:02:47.704: INFO: At 2022-12-22 17:57:59 +0000 UTC - event for sample-webhook-deployment-865554f4d9-j2nt4: {kubelet i-0c76fbcc0b95be727} Pulled: Container image "registry.k8s.io/e2e-test-images/agnhost:2.43" already present on machine Dec 22 18:02:47.704: INFO: At 2022-12-22 17:57:59 +0000 UTC - event for sample-webhook-deployment-865554f4d9-j2nt4: {kubelet i-0c76fbcc0b95be727} Created: Created container sample-webhook Dec 22 18:02:47.704: INFO: At 2022-12-22 17:57:59 +0000 UTC - event for sample-webhook-deployment-865554f4d9-j2nt4: {kubelet i-0c76fbcc0b95be727} Started: Started container sample-webhook Dec 22 18:02:47.704: INFO: At 2022-12-22 17:58:00 +0000 UTC - event for sample-webhook-deployment-865554f4d9-j2nt4: {kubelet i-0c76fbcc0b95be727} Unhealthy: Readiness probe failed: Get "https://172.20.44.56:8444/readyz": dial tcp 172.20.44.56:8444: i/o timeout Dec 22 18:02:47.704: INFO: At 2022-12-22 17:58:01 +0000 UTC - event for sample-webhook-deployment-865554f4d9-j2nt4: {kubelet i-0c76fbcc0b95be727} Unhealthy: Readiness probe failed: Get "https://172.20.44.56:8444/readyz": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) Dec 22 18:02:47.704: INFO: At 2022-12-22 18:02:47 +0000 UTC - event for sample-webhook-deployment-865554f4d9-j2nt4: {kubelet i-0c76fbcc0b95be727} Killing: Stopping container sample-webhook Dec 22 18:02:47.768: INFO: POD NODE PHASE GRACE CONDITIONS Dec 22 18:02:47.768: INFO: Dec 22 18:02:47.833: INFO: Logging node info for node i-0243def68c064e8ef Dec 22 18:02:47.898: INFO: Node Info: &Node{ObjectMeta:{i-0243def68c064e8ef f07b8771-29eb-4944-a385-3482a3d3a315 52684 0 2022-12-22 17:17:47 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:us-west-2 failure-domain.beta.kubernetes.io/zone:us-west-2a kubernetes.io/arch:amd64 kubernetes.io/hostname:i-0243def68c064e8ef kubernetes.io/os:linux node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:us-west-2a topology.hostpath.csi/node:i-0243def68c064e8ef topology.kubernetes.io/region:us-west-2 topology.kubernetes.io/zone:us-west-2a] map[alpha.kubernetes.io/provided-node-ip:172.20.35.142 csi.volume.kubernetes.io/nodeid:{"csi-hostpath-ephemeral-6886":"i-0243def68c064e8ef","csi-hostpath-ephemeral-8407":"i-0243def68c064e8ef","ebs.csi.aws.com":"i-0243def68c064e8ef"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{aws-cloud-controller-manager Update v1 2022-12-22 17:17:47 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2022-12-22 17:17:47 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kops-controller Update v1 2022-12-22 17:17:47 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{}}}} } {kube-controller-manager Update v1 2022-12-22 17:17:47 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.3.0/24\"":{}}}} } {kubelet Update v1 2022-12-22 17:17:47 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:alpha.kubernetes.io/provided-node-ip":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {kube-controller-manager Update v1 2022-12-22 17:59:50 +0000 UTC FieldsV1 {"f:status":{"f:volumesAttached":{}}} status} {kubelet Update v1 2022-12-22 17:59:50 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.3.0/24,DoNotUseExternalID:,ProviderID:aws:///us-west-2a/i-0243def68c064e8ef,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.3.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{51527004160 0} {<nil>} 50319340Ki BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4050636800 0} {<nil>} 3955700Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{46374303668 0} {<nil>} 46374303668 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3945779200 0} {<nil>} 3853300Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-12-22 17:59:50 +0000 UTC,LastTransitionTime:2022-12-22 17:17:37 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-12-22 17:59:50 +0000 UTC,LastTransitionTime:2022-12-22 17:17:37 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-12-22 17:59:50 +0000 UTC,LastTransitionTime:2022-12-22 17:17:37 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-12-22 17:59:50 +0000 UTC,LastTransitionTime:2022-12-22 17:17:59 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.35.142,},NodeAddress{Type:ExternalIP,Address:35.90.235.118,},NodeAddress{Type:InternalDNS,Address:i-0243def68c064e8ef.us-west-2.compute.internal,},NodeAddress{Type:Hostname,Address:i-0243def68c064e8ef.us-west-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-35-90-235-118.us-west-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2c8ae28bb7834ee5f6b4c4ef144a8e,SystemUUID:ec2c8ae2-8bb7-834e-e5f6-b4c4ef144a8e,BootID:5843895c-91be-4c36-b700-9e551cb745de,KernelVersion:5.10.157-139.675.amzn2.x86_64,OSImage:Amazon Linux 2,ContainerRuntimeVersion:containerd://1.6.14,KubeletVersion:v1.26.0,KubeProxyVersion:v1.26.0,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:166719855,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/jessie-dnsutils@sha256:24aaf2626d6b27864c29de2097e8bbb840b3a414271bf7c8995e431e47d8408e registry.k8s.io/e2e-test-images/jessie-dnsutils:1.7],SizeBytes:112030336,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:dd75ec974b0a2a6f6bb47001ba09207976e625db898d1b16735528c009cb171c registry.k8s.io/etcd:3.5.6-0],SizeBytes:102542580,},ContainerImage{Names:[registry.k8s.io/kube-proxy-amd64:v1.26.0],SizeBytes:67205320,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:16bbf38c463a4223d8cfe4da12bc61010b082a79b4bb003e2d3ba3ece5dd5f9e registry.k8s.io/e2e-test-images/agnhost:2.43],SizeBytes:51706353,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nautilus@sha256:80ba6c8c44f9623f06e868a1aa66026c8ec438ad814f9ec95e9333b415fe3550 registry.k8s.io/e2e-test-images/nautilus:1.7],SizeBytes:49641698,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:3fe7acf013d1264ffded116b80a73dc129a449b0fccdb8d21af8279f2233f36e registry.k8s.io/e2e-test-images/httpd:2.4.39-4],SizeBytes:41901587,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:148b022f5c5da426fc2f3c14b5c0867e58ef05961510c84749ac1fddcb0fef22 registry.k8s.io/e2e-test-images/httpd:2.4.38-4],SizeBytes:40764257,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:10e231cf07c82dfab5141c1afe548dc734e5fc1f67665ec3982a325d1bd31a9a registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.0],SizeBytes:29725756,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:ee3b525d5b89db99da3b8eb521d9cd90cb6e9ef0fbb651e98bb37be78d36b5b8 registry.k8s.io/sig-storage/csi-provisioner:v3.3.0],SizeBytes:25491225,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:425d8f1b769398127767b06ed97ce62578a3179bcb99809ce93a1649e025ffe7 registry.k8s.io/sig-storage/csi-resizer:v1.6.0],SizeBytes:24148884,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:291334908ddf71a4661fd7f6d9d97274de8a5378a2b6fdfeb2ce73414a34f82f registry.k8s.io/sig-storage/csi-snapshotter:v6.1.0],SizeBytes:23881995,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:9a685020911e2725ad019dbce6e4a5ab93d51e3d4557f115e64343345e05781b registry.k8s.io/sig-storage/csi-attacher:v4.0.0],SizeBytes:23847201,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:92257881c1d6493cf18299a24af42330f891166560047902b8d431fb66b01af5 registry.k8s.io/sig-storage/hostpathplugin:v1.9.0],SizeBytes:18758628,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nonroot@sha256:ee9f50b3c64b174d296d91ca9f69a914ac30e59095dfb462b2b518ad28a63655 registry.k8s.io/e2e-test-images/nonroot:1.4],SizeBytes:17747885,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:933940f13b3ea0abc62e656c1aa5c5b47c04b15d71250413a6b821bd0c58b94e registry.k8s.io/sig-storage/livenessprobe:v2.7.0],SizeBytes:8688564,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:5c99cf6a02adda929b10321dbf4ecfa00d87be9ba4fb456006237d530ab4baa1 registry.k8s.io/e2e-test-images/nginx:1.14-4],SizeBytes:6978614,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nonewprivs@sha256:8ac1264691820febacf3aea5d152cbde6d10685731ec14966a9401c6f47a68ac registry.k8s.io/e2e-test-images/nonewprivs:1.3],SizeBytes:3263463,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf registry.k8s.io/e2e-test-images/busybox:1.29-2],SizeBytes:732424,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:2e0f836850e09b8b7cc937681d6194537a09fbd5f6b9e08f4d646a85128e8937 registry.k8s.io/e2e-test-images/busybox:1.29-4],SizeBytes:731990,},ContainerImage{Names:[registry.k8s.io/pause@sha256:7031c1b283388d2c2e09b57badb803c05ebed362dc88d84b480cc47f72a21097 registry.k8s.io/pause:3.9],SizeBytes:321520,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[kubernetes.io/csi/csi-hostpath-ephemeral-6886^4e266409-8221-11ed-800d-8a8e987c6028 kubernetes.io/csi/csi-hostpath-ephemeral-8407^9fd9423f-821f-11ed-b5cc-02642e84c89b],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/csi-hostpath-ephemeral-8407^9fd9423f-821f-11ed-b5cc-02642e84c89b,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/csi-hostpath-ephemeral-6886^4e266409-8221-11ed-800d-8a8e987c6028,DevicePath:,},},Config:nil,},} ... skipping 363 lines ... [DeferCleanup (Each)] [sig-api-machinery] AdmissionWebhook [Privileged:ClusterAdmin] tear down framework | framework.go:193 [1mSTEP:[0m Destroying namespace "webhook-5645" for this suite. [38;5;243m12/22/22 18:02:53.005[0m [1mSTEP:[0m Destroying namespace "webhook-5645-markers" for this suite. [38;5;243m12/22/22 18:02:53.072[0m [38;5;243m<< End Captured GinkgoWriter Output[0m [38;5;9mDec 22 18:02:47.299: waiting for the deployment status valid%!(EXTRA string=registry.k8s.io/e2e-test-images/agnhost:2.43, string=sample-webhook-deployment, string=webhook-5645): error waiting for deployment "sample-webhook-deployment" status to match expectation: deployment status: v1.DeploymentStatus{ObservedGeneration:1, Replicas:1, UpdatedReplicas:1, ReadyReplicas:0, AvailableReplicas:0, UnavailableReplicas:1, Conditions:[]v1.DeploymentCondition{v1.DeploymentCondition{Type:"Available", Status:"False", LastUpdateTime:time.Date(2022, time.December, 22, 17, 57, 46, 0, time.Local), LastTransitionTime:time.Date(2022, time.December, 22, 17, 57, 46, 0, time.Local), Reason:"MinimumReplicasUnavailable", Message:"Deployment does not have minimum availability."}, v1.DeploymentCondition{Type:"Progressing", Status:"True", LastUpdateTime:time.Date(2022, time.December, 22, 17, 57, 46, 0, time.Local), LastTransitionTime:time.Date(2022, time.December, 22, 17, 57, 46, 0, time.Local), Reason:"ReplicaSetUpdated", Message:"ReplicaSet \"sample-webhook-deployment-865554f4d9\" is progressing."}}, CollisionCount:(*int32)(nil)}[0m [38;5;9mIn [1m[BeforeEach][0m[38;5;9m at: [1mtest/e2e/apimachinery/webhook.go:826[0m [38;5;243m------------------------------[0m [38;5;9m• [FAILED] [506.263 seconds][0m [sig-network] Networking [38;5;243mtest/e2e/network/common/framework.go:23[0m Granular Checks: Services [38;5;243mtest/e2e/network/networking.go:145[0m [38;5;9m[1m[It] should function for client IP based session affinity: udp [LinuxOnly][0m [38;5;243mtest/e2e/network/networking.go:434[0m ... skipping 522 lines ... Dec 22 18:03:17.696: INFO: Pod "netserver-1": Phase="Running", Reason="", readiness=false. Elapsed: 4m58.12431451s Dec 22 18:03:17.696: INFO: The phase of Pod netserver-1 is Running (Ready = false) Dec 22 18:03:19.695: INFO: Pod "netserver-1": Phase="Running", Reason="", readiness=false. Elapsed: 5m0.123223112s Dec 22 18:03:19.695: INFO: The phase of Pod netserver-1 is Running (Ready = false) Dec 22 18:03:19.757: INFO: Pod "netserver-1": Phase="Running", Reason="", readiness=false. Elapsed: 5m0.18523477s Dec 22 18:03:19.757: INFO: The phase of Pod netserver-1 is Running (Ready = false) Dec 22 18:03:19.758: INFO: Unexpected error: <*pod.timeoutError | 0xc00383e480>: { msg: "timed out while waiting for pod nettest-4932/netserver-1 to be running and ready", observedObjects: [ <*v1.Pod | 0xc00356b200>{ TypeMeta: {Kind: "", APIVersion: ""}, ObjectMeta: { ... skipping 128 lines ... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output Dec 22 18:03:19.759: FAIL: timed out while waiting for pod nettest-4932/netserver-1 to be running and ready Full Stack Trace k8s.io/kubernetes/test/e2e/framework/network.(*NetworkingTestConfig).createNetProxyPods(0xc00339c000, {0x75c6f5c, 0x9}, 0xc0034fc000) test/e2e/framework/network/utils.go:866 +0x1d0 k8s.io/kubernetes/test/e2e/framework/network.(*NetworkingTestConfig).setupCore(0xc00339c000, 0x7f710c0c7a28?) test/e2e/framework/network/utils.go:763 +0x55 ... skipping 10 lines ... test/e2e/framework/metrics/init/init.go:33 [DeferCleanup (Each)] [sig-network] Networking dump namespaces | framework.go:196 [1mSTEP:[0m dump namespace information after failure [38;5;243m12/22/22 18:03:19.822[0m [1mSTEP:[0m Collecting events from namespace "nettest-4932". [38;5;243m12/22/22 18:03:19.822[0m [1mSTEP:[0m Found 68 events. [38;5;243m12/22/22 18:03:19.886[0m Dec 22 18:03:19.886: INFO: At 2022-12-22 17:54:57 +0000 UTC - event for netserver-0: {kubelet i-0243def68c064e8ef} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "cd2af7e8a98a3b974b152c3baf2188855ed470108cef4f20a52e339c2bab78fe": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 18:03:19.886: INFO: At 2022-12-22 17:54:57 +0000 UTC - event for netserver-0: {default-scheduler } Scheduled: Successfully assigned nettest-4932/netserver-0 to i-0243def68c064e8ef Dec 22 18:03:19.886: INFO: At 2022-12-22 17:54:57 +0000 UTC - event for netserver-1: {default-scheduler } Scheduled: Successfully assigned nettest-4932/netserver-1 to i-076110ae730e268e3 Dec 22 18:03:19.886: INFO: At 2022-12-22 17:54:57 +0000 UTC - event for netserver-1: {kubelet i-076110ae730e268e3} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "e28f8d4e8b2dc0f041a0868a1a2d3b86e8831c2c58da9755216c9f78d4984df1": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 18:03:19.886: INFO: At 2022-12-22 17:54:57 +0000 UTC - event for netserver-2: {kubelet i-0c76fbcc0b95be727} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "d739813140029fd6294094c76a65b75c69a5954595d9d585ed26f3e72315cae3": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 18:03:19.886: INFO: At 2022-12-22 17:54:57 +0000 UTC - event for netserver-2: {default-scheduler } Scheduled: Successfully assigned nettest-4932/netserver-2 to i-0c76fbcc0b95be727 Dec 22 18:03:19.886: INFO: At 2022-12-22 17:54:57 +0000 UTC - event for netserver-3: {kubelet i-0ea878d3671f2eda2} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "f3715d381e5d90d6cb3eaabf0833390accad18b1ee6b608267e161795a19d6b6": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 18:03:19.886: INFO: At 2022-12-22 17:54:57 +0000 UTC - event for netserver-3: {default-scheduler } Scheduled: Successfully assigned nettest-4932/netserver-3 to i-0ea878d3671f2eda2 Dec 22 18:03:19.886: INFO: At 2022-12-22 17:55:08 +0000 UTC - event for netserver-2: {kubelet i-0c76fbcc0b95be727} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "f506af17f0e4fbbbfeb547d70dfa233864fd7ea69073342cb7678e0acf535433": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 18:03:19.886: INFO: At 2022-12-22 17:55:10 +0000 UTC - event for netserver-1: {kubelet i-076110ae730e268e3} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "c95046d8982da6f4d322b318f46f289febbe5b8602ca20f2382103ed56efbf6d": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 18:03:19.886: INFO: At 2022-12-22 17:55:11 +0000 UTC - event for netserver-0: {kubelet i-0243def68c064e8ef} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "d8d8526b84932ea97ab10c177156c235afa19eb0b7022d7b89e6978d4d650787": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 18:03:19.886: INFO: At 2022-12-22 17:55:11 +0000 UTC - event for netserver-3: {kubelet i-0ea878d3671f2eda2} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "16a6eff2b32190f8448cf74f16f467c60d7a9159fd9a8d5946da5ca3d640c257": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 18:03:19.886: INFO: At 2022-12-22 17:55:22 +0000 UTC - event for netserver-2: {kubelet i-0c76fbcc0b95be727} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "765759066f52a029d49be7162660fa8da4f49a6fa359865a43d2869570b75c1e": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 18:03:19.886: INFO: At 2022-12-22 17:55:22 +0000 UTC - event for netserver-3: {kubelet i-0ea878d3671f2eda2} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "b9c147df2ddfb0f95ec901c272d6aa4b7ac4c7de96f4a070609647aeb37dc3b6": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 18:03:19.886: INFO: At 2022-12-22 17:55:23 +0000 UTC - event for netserver-0: {kubelet i-0243def68c064e8ef} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "3df29eb36814830d27417b5937e70f281b3a2d09df52f974624d175577bc14d0": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 18:03:19.886: INFO: At 2022-12-22 17:55:24 +0000 UTC - event for netserver-1: {kubelet i-076110ae730e268e3} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "d72b911c4c6e181230072f6a6adadd0f9851a327b621e40482192c66ae5bc3f0": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 18:03:19.886: INFO: At 2022-12-22 17:55:33 +0000 UTC - event for netserver-3: {kubelet i-0ea878d3671f2eda2} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "9558089a32de3e9959a6046a69102e4c5e96a3dfb3d0770715abf65d459ea883": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 18:03:19.886: INFO: At 2022-12-22 17:55:36 +0000 UTC - event for netserver-0: {kubelet i-0243def68c064e8ef} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "e945f9796f2e8bc28cc9b1b65dab4463a546ce79b882738252401e2684a63ee2": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 18:03:19.886: INFO: At 2022-12-22 17:55:36 +0000 UTC - event for netserver-1: {kubelet i-076110ae730e268e3} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "befb4c790a640250e4d3f92fe45ad1275c9cb96151304a4a1038fbfd810f2f39": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 18:03:19.886: INFO: At 2022-12-22 17:55:37 +0000 UTC - event for netserver-2: {kubelet i-0c76fbcc0b95be727} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "6cf8ff8f59cab02613c838300ee3ab6f61174d1789df64002ecbc497c162c1c2": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 18:03:19.886: INFO: At 2022-12-22 17:55:47 +0000 UTC - event for netserver-0: {kubelet i-0243def68c064e8ef} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "041763575f690ed7bb4f73bb9942314e035fa7f918eaaad768a63f301090c7d6": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 18:03:19.886: INFO: At 2022-12-22 17:55:48 +0000 UTC - event for netserver-3: {kubelet i-0ea878d3671f2eda2} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "c03b8e7e16f6edda8b81c8e79c1dff1d0d2eddfc19abb3618c6de4f4e80dcfca": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 18:03:19.886: INFO: At 2022-12-22 17:55:50 +0000 UTC - event for netserver-1: {kubelet i-076110ae730e268e3} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "c6db342e77b185348e85413d52dcab02d4545e47e7a591859883d2dcd8a7e472": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 18:03:19.886: INFO: At 2022-12-22 17:55:52 +0000 UTC - event for netserver-2: {kubelet i-0c76fbcc0b95be727} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "80bda1e3233b511d6205cc24add1efd936fe0b754ba1fd335caa03c2a6058201": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 18:03:19.886: INFO: At 2022-12-22 17:55:59 +0000 UTC - event for netserver-0: {kubelet i-0243def68c064e8ef} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "2dcc449a006063dc37304fac2135d5c52e583ae11c04259fe3a5ae43f8e25a7a": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 18:03:19.886: INFO: At 2022-12-22 17:56:01 +0000 UTC - event for netserver-3: {kubelet i-0ea878d3671f2eda2} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "a21cd679d676e31e8aaafc524d81e2fa06180dcd9c58aefcb07fbd97889fab10": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 18:03:19.886: INFO: At 2022-12-22 17:56:04 +0000 UTC - event for netserver-1: {kubelet i-076110ae730e268e3} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "9154cfe7a8d8b826207bf2d5b41ea95b746e303bce912de2a78bc75ac448553f": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 18:03:19.886: INFO: At 2022-12-22 17:56:05 +0000 UTC - event for netserver-2: {kubelet i-0c76fbcc0b95be727} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "167ab0c5fcf71ac24514b953b223d6a9f34f1ecc7cc8324d4a3a6ce9fbb98c9e": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 18:03:19.886: INFO: At 2022-12-22 17:56:10 +0000 UTC - event for netserver-0: {kubelet i-0243def68c064e8ef} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "ac1494f119b596b0603c72b7092e43cac4aa6aebeb3146c1321de2b045e96aa7": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 18:03:19.886: INFO: At 2022-12-22 17:56:13 +0000 UTC - event for netserver-3: {kubelet i-0ea878d3671f2eda2} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "324eb6c8078846867a169dbfd041ac8879f4ffea7d9cf403b77f452cf6d5db14": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 18:03:19.886: INFO: At 2022-12-22 17:56:17 +0000 UTC - event for netserver-2: {kubelet i-0c76fbcc0b95be727} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "1338402a5abf9d5c5b7f9bcf110bc8257eafbf67d6be5d120ab737c4947fa7f2": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 18:03:19.886: INFO: At 2022-12-22 17:56:18 +0000 UTC - event for netserver-1: {kubelet i-076110ae730e268e3} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "776ebaab084a958e9631c20e6f04d11c5c292cfef183be85659768de670358a4": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 18:03:19.886: INFO: At 2022-12-22 17:56:24 +0000 UTC - event for netserver-0: {kubelet i-0243def68c064e8ef} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "a7dec4047727be3e3d880cb42138930162eebe5f0f6e506a2a768fa2c568ad1e": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 18:03:19.886: INFO: At 2022-12-22 17:56:26 +0000 UTC - event for netserver-3: {kubelet i-0ea878d3671f2eda2} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "e1cc62d56bc09b174094eb23259efb65a062f350ded196c801c6a372f13555f6": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 18:03:19.886: INFO: At 2022-12-22 17:56:29 +0000 UTC - event for netserver-1: {kubelet i-076110ae730e268e3} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "e0e368a69e50e115aad17d5cb585e27580d52cee1f5bd8ce76449a8cd3dd448e": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 18:03:19.886: INFO: At 2022-12-22 17:56:32 +0000 UTC - event for netserver-2: {kubelet i-0c76fbcc0b95be727} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "48a66d33acbc14d048d6accb0e89f3e7dd67142a1004f39c004d451104930fce": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 18:03:19.886: INFO: At 2022-12-22 17:56:35 +0000 UTC - event for netserver-0: {kubelet i-0243def68c064e8ef} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "a673def439aae07686561692a6dc98e7ea7ff61f60f7c2400c0edf58c73d2ef6": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 18:03:19.886: INFO: At 2022-12-22 17:56:39 +0000 UTC - event for netserver-3: {kubelet i-0ea878d3671f2eda2} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "7fba397b6fd1ab9d06ab9e8ce7cb9d94afa4fdae0b719395c8f45553b4d014b1": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 18:03:19.886: INFO: At 2022-12-22 17:56:44 +0000 UTC - event for netserver-1: {kubelet i-076110ae730e268e3} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "35e5899635aeb43c5e7697d46167a4e9b23f70161a1ec9fe6d911fbdab40865b": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 18:03:19.886: INFO: At 2022-12-22 17:56:47 +0000 UTC - event for netserver-2: {kubelet i-0c76fbcc0b95be727} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "2e2bf0aed8629739c1fe89193d601a1200bbf2e4d465df7bd34d2f657199dc5a": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 18:03:19.886: INFO: At 2022-12-22 17:56:50 +0000 UTC - event for netserver-0: {kubelet i-0243def68c064e8ef} FailedCreatePodSandBox: (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "6cc16cbc31e7b1e81f950c552f5c56c59973c6ae06469343b219c16ed95a5c7b": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 18:03:19.886: INFO: At 2022-12-22 17:56:54 +0000 UTC - event for netserver-3: {kubelet i-0ea878d3671f2eda2} FailedCreatePodSandBox: (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "7fad4a6f31ed0c237e0575a78107bb2d58cefee2959930064103a09bddeb24f1": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 18:03:19.886: INFO: At 2022-12-22 17:56:59 +0000 UTC - event for netserver-2: {kubelet i-0c76fbcc0b95be727} FailedCreatePodSandBox: (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "c8daf2538e5ef52e104dcd606e00495b5ff3ee3955fffb70847f0e367269a3bf": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 18:03:19.886: INFO: At 2022-12-22 17:57:00 +0000 UTC - event for netserver-1: {kubelet i-076110ae730e268e3} FailedCreatePodSandBox: (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "37f515c961aa19cb713fb2b5f0e3778875913ecae6502503237cdb9b4eb19d93": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 18:03:19.886: INFO: At 2022-12-22 17:57:06 +0000 UTC - event for netserver-3: {kubelet i-0ea878d3671f2eda2} Pulled: Container image "registry.k8s.io/e2e-test-images/agnhost:2.43" already present on machine Dec 22 18:03:19.886: INFO: At 2022-12-22 17:57:06 +0000 UTC - event for netserver-3: {kubelet i-0ea878d3671f2eda2} Created: Created container webserver Dec 22 18:03:19.886: INFO: At 2022-12-22 17:57:06 +0000 UTC - event for netserver-3: {kubelet i-0ea878d3671f2eda2} Started: Started container webserver Dec 22 18:03:19.886: INFO: At 2022-12-22 17:57:47 +0000 UTC - event for netserver-3: {kubelet i-0ea878d3671f2eda2} Unhealthy: Readiness probe failed: Get "http://172.20.39.179:8083/healthz": dial tcp 172.20.39.179:8083: i/o timeout (Client.Timeout exceeded while awaiting headers) Dec 22 18:03:19.886: INFO: At 2022-12-22 17:57:47 +0000 UTC - event for netserver-3: {kubelet i-0ea878d3671f2eda2} Unhealthy: Liveness probe failed: Get "http://172.20.39.179:8083/healthz": context deadline exceeded (Client.Timeout exceeded while awaiting headers) Dec 22 18:03:19.886: INFO: At 2022-12-22 17:57:54 +0000 UTC - event for netserver-2: {kubelet i-0c76fbcc0b95be727} Created: Created container webserver Dec 22 18:03:19.886: INFO: At 2022-12-22 17:57:54 +0000 UTC - event for netserver-2: {kubelet i-0c76fbcc0b95be727} Pulled: Container image "registry.k8s.io/e2e-test-images/agnhost:2.43" already present on machine Dec 22 18:03:19.886: INFO: At 2022-12-22 17:57:55 +0000 UTC - event for netserver-1: {kubelet i-076110ae730e268e3} Created: Created container webserver Dec 22 18:03:19.886: INFO: At 2022-12-22 17:57:55 +0000 UTC - event for netserver-1: {kubelet i-076110ae730e268e3} Pulled: Container image "registry.k8s.io/e2e-test-images/agnhost:2.43" already present on machine Dec 22 18:03:19.886: INFO: At 2022-12-22 17:57:55 +0000 UTC - event for netserver-1: {kubelet i-076110ae730e268e3} Started: Started container webserver Dec 22 18:03:19.886: INFO: At 2022-12-22 17:57:55 +0000 UTC - event for netserver-2: {kubelet i-0c76fbcc0b95be727} Started: Started container webserver Dec 22 18:03:19.886: INFO: At 2022-12-22 17:58:00 +0000 UTC - event for netserver-0: {kubelet i-0243def68c064e8ef} Pulled: Container image "registry.k8s.io/e2e-test-images/agnhost:2.43" already present on machine Dec 22 18:03:19.886: INFO: At 2022-12-22 17:58:00 +0000 UTC - event for netserver-0: {kubelet i-0243def68c064e8ef} Created: Created container webserver Dec 22 18:03:19.886: INFO: At 2022-12-22 17:58:00 +0000 UTC - event for netserver-0: {kubelet i-0243def68c064e8ef} Started: Started container webserver Dec 22 18:03:19.886: INFO: At 2022-12-22 17:58:17 +0000 UTC - event for netserver-3: {kubelet i-0ea878d3671f2eda2} Unhealthy: Readiness probe failed: Get "http://172.20.39.179:8083/healthz": context deadline exceeded (Client.Timeout exceeded while awaiting headers) Dec 22 18:03:19.886: INFO: At 2022-12-22 17:58:37 +0000 UTC - event for netserver-1: {kubelet i-076110ae730e268e3} Unhealthy: Liveness probe failed: Get "http://172.20.47.29:8083/healthz": context deadline exceeded (Client.Timeout exceeded while awaiting headers) Dec 22 18:03:19.886: INFO: At 2022-12-22 17:58:37 +0000 UTC - event for netserver-1: {kubelet i-076110ae730e268e3} Unhealthy: Readiness probe failed: Get "http://172.20.47.29:8083/healthz": context deadline exceeded (Client.Timeout exceeded while awaiting headers) Dec 22 18:03:19.886: INFO: At 2022-12-22 17:58:37 +0000 UTC - event for netserver-2: {kubelet i-0c76fbcc0b95be727} Unhealthy: Liveness probe failed: Get "http://172.20.56.115:8083/healthz": context deadline exceeded (Client.Timeout exceeded while awaiting headers) Dec 22 18:03:19.886: INFO: At 2022-12-22 17:58:37 +0000 UTC - event for netserver-2: {kubelet i-0c76fbcc0b95be727} Unhealthy: Readiness probe failed: Get "http://172.20.56.115:8083/healthz": dial tcp 172.20.56.115:8083: i/o timeout (Client.Timeout exceeded while awaiting headers) Dec 22 18:03:19.886: INFO: At 2022-12-22 17:58:47 +0000 UTC - event for netserver-3: {kubelet i-0ea878d3671f2eda2} Killing: Container webserver failed liveness probe, will be restarted Dec 22 18:03:19.886: INFO: At 2022-12-22 17:59:07 +0000 UTC - event for netserver-2: {kubelet i-0c76fbcc0b95be727} Unhealthy: Readiness probe failed: Get "http://172.20.56.115:8083/healthz": context deadline exceeded (Client.Timeout exceeded while awaiting headers) Dec 22 18:03:19.886: INFO: At 2022-12-22 17:59:37 +0000 UTC - event for netserver-1: {kubelet i-076110ae730e268e3} Killing: Container webserver failed liveness probe, will be restarted Dec 22 18:03:19.886: INFO: At 2022-12-22 17:59:37 +0000 UTC - event for netserver-2: {kubelet i-0c76fbcc0b95be727} Unhealthy: Liveness probe failed: Get "http://172.20.56.115:8083/healthz": dial tcp 172.20.56.115:8083: i/o timeout (Client.Timeout exceeded while awaiting headers) Dec 22 18:03:19.886: INFO: At 2022-12-22 17:59:37 +0000 UTC - event for netserver-2: {kubelet i-0c76fbcc0b95be727} Killing: Container webserver failed liveness probe, will be restarted Dec 22 18:03:19.949: INFO: POD NODE PHASE GRACE CONDITIONS Dec 22 18:03:19.949: INFO: netserver-0 i-0243def68c064e8ef Running [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2022-12-22 17:54:57 +0000 UTC } {Ready True 0001-01-01 00:00:00 +0000 UTC 2022-12-22 17:58:17 +0000 UTC } {ContainersReady True 0001-01-01 00:00:00 +0000 UTC 2022-12-22 17:58:17 +0000 UTC } {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2022-12-22 17:54:57 +0000 UTC }] Dec 22 18:03:19.949: INFO: netserver-1 i-076110ae730e268e3 Running [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2022-12-22 17:54:57 +0000 UTC } {Ready False 0001-01-01 00:00:00 +0000 UTC 2022-12-22 17:54:57 +0000 UTC ContainersNotReady containers with unready status: [webserver]} {ContainersReady False 0001-01-01 00:00:00 +0000 UTC 2022-12-22 17:54:57 +0000 UTC ContainersNotReady containers with unready status: [webserver]} {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2022-12-22 17:54:57 +0000 UTC }] Dec 22 18:03:19.949: INFO: netserver-2 i-0c76fbcc0b95be727 Running [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2022-12-22 17:54:57 +0000 UTC } {Ready False 0001-01-01 00:00:00 +0000 UTC 2022-12-22 17:54:57 +0000 UTC ContainersNotReady containers with unready status: [webserver]} {ContainersReady False 0001-01-01 00:00:00 +0000 UTC 2022-12-22 17:54:57 +0000 UTC ContainersNotReady containers with unready status: [webserver]} {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2022-12-22 17:54:57 +0000 UTC }] Dec 22 18:03:19.949: INFO: netserver-3 i-0ea878d3671f2eda2 Running [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2022-12-22 17:54:57 +0000 UTC } {Ready False 0001-01-01 00:00:00 +0000 UTC 2022-12-22 17:54:57 +0000 UTC ContainersNotReady containers with unready status: [webserver]} {ContainersReady False 0001-01-01 00:00:00 +0000 UTC 2022-12-22 17:54:57 +0000 UTC ContainersNotReady containers with unready status: [webserver]} {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2022-12-22 17:54:57 +0000 UTC }] Dec 22 18:03:19.949: INFO: ... skipping 181 lines ... [1mSTEP:[0m Destroying namespace "nettest-4932" for this suite. [38;5;243m12/22/22 18:03:22.766[0m [38;5;243m<< End Captured GinkgoWriter Output[0m [38;5;9mDec 22 18:03:19.759: timed out while waiting for pod nettest-4932/netserver-1 to be running and ready[0m [38;5;9mIn [1m[It][0m[38;5;9m at: [1mtest/e2e/framework/network/utils.go:866[0m [38;5;243m------------------------------[0m [38;5;9m• [FAILED] [366.509 seconds][0m [sig-network] Networking [38;5;243mtest/e2e/network/common/framework.go:23[0m Granular Checks: Services [38;5;243mtest/e2e/network/networking.go:145[0m [38;5;9m[1m[It] should function for node-Service: http[0m [38;5;243mtest/e2e/network/networking.go:192[0m ... skipping 382 lines ... Dec 22 18:03:22.393: INFO: Pod "netserver-1": Phase="Running", Reason="", readiness=false. Elapsed: 4m58.131608801s Dec 22 18:03:22.393: INFO: The phase of Pod netserver-1 is Running (Ready = false) Dec 22 18:03:24.389: INFO: Pod "netserver-1": Phase="Running", Reason="", readiness=false. Elapsed: 5m0.12770726s Dec 22 18:03:24.389: INFO: The phase of Pod netserver-1 is Running (Ready = false) Dec 22 18:03:24.454: INFO: Pod "netserver-1": Phase="Running", Reason="", readiness=false. Elapsed: 5m0.192484907s Dec 22 18:03:24.454: INFO: The phase of Pod netserver-1 is Running (Ready = false) Dec 22 18:03:24.456: INFO: Unexpected error: <*pod.timeoutError | 0xc004598540>: { msg: "timed out while waiting for pod nettest-6730/netserver-1 to be running and ready", observedObjects: [ <*v1.Pod | 0xc000d68480>{ TypeMeta: {Kind: "", APIVersion: ""}, ObjectMeta: { ... skipping 128 lines ... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output Dec 22 18:03:24.456: FAIL: timed out while waiting for pod nettest-6730/netserver-1 to be running and ready Full Stack Trace k8s.io/kubernetes/test/e2e/framework/network.(*NetworkingTestConfig).createNetProxyPods(0xc00015e1c0, {0x75c6f5c, 0x9}, 0xc004598450) test/e2e/framework/network/utils.go:866 +0x1d0 k8s.io/kubernetes/test/e2e/framework/network.(*NetworkingTestConfig).setupCore(0xc00015e1c0, 0x7fd8c05597b8?) test/e2e/framework/network/utils.go:763 +0x55 ... skipping 13 lines ... [1mSTEP:[0m dump namespace information after failure [38;5;243m12/22/22 18:03:24.523[0m [1mSTEP:[0m Collecting events from namespace "nettest-6730". [38;5;243m12/22/22 18:03:24.523[0m [1mSTEP:[0m Found 32 events. [38;5;243m12/22/22 18:03:24.588[0m Dec 22 18:03:24.588: INFO: At 2022-12-22 17:57:21 +0000 UTC - event for netserver-0: {default-scheduler } Scheduled: Successfully assigned nettest-6730/netserver-0 to i-0243def68c064e8ef Dec 22 18:03:24.588: INFO: At 2022-12-22 17:57:21 +0000 UTC - event for netserver-1: {default-scheduler } Scheduled: Successfully assigned nettest-6730/netserver-1 to i-076110ae730e268e3 Dec 22 18:03:24.588: INFO: At 2022-12-22 17:57:21 +0000 UTC - event for netserver-2: {default-scheduler } Scheduled: Successfully assigned nettest-6730/netserver-2 to i-0c76fbcc0b95be727 Dec 22 18:03:24.588: INFO: At 2022-12-22 17:57:22 +0000 UTC - event for netserver-0: {kubelet i-0243def68c064e8ef} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "706f3379f08100e182421fa1c56d031b543f3d30525c1c7f63f9539f3ec0a90e": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 18:03:24.588: INFO: At 2022-12-22 17:57:22 +0000 UTC - event for netserver-1: {kubelet i-076110ae730e268e3} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "c8056f389a6562462a72329674fc48c05cfe363858ee20a94f25071f1efe1bdc": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 18:03:24.588: INFO: At 2022-12-22 17:57:22 +0000 UTC - event for netserver-2: {kubelet i-0c76fbcc0b95be727} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "d5131860de233ab6a567735c3827c4f967b59bed1d607483f64217040c3b8ea3": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 18:03:24.588: INFO: At 2022-12-22 17:57:22 +0000 UTC - event for netserver-3: {kubelet i-0ea878d3671f2eda2} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "dcdf145ea1778b61b79da3ab04fccfe3e2d30b1edca32962addf4b7f14c72209": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 18:03:24.588: INFO: At 2022-12-22 17:57:22 +0000 UTC - event for netserver-3: {default-scheduler } Scheduled: Successfully assigned nettest-6730/netserver-3 to i-0ea878d3671f2eda2 Dec 22 18:03:24.588: INFO: At 2022-12-22 17:57:34 +0000 UTC - event for netserver-1: {kubelet i-076110ae730e268e3} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "361ed9d18e8b1b5b77cea3849393aa39d36bbce0ef35e5d754637291ffd3df59": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 18:03:24.588: INFO: At 2022-12-22 17:57:35 +0000 UTC - event for netserver-3: {kubelet i-0ea878d3671f2eda2} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "5d073bae91fdad0cff72eba6ea370b9270ba97383dc7fe4f1f31b512be49e82a": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 18:03:24.588: INFO: At 2022-12-22 17:57:36 +0000 UTC - event for netserver-0: {kubelet i-0243def68c064e8ef} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "2a189e693f8d9eeeb1bee103bdce6a441446f81427f5b63aba0d5e178861795d": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 18:03:24.588: INFO: At 2022-12-22 17:57:37 +0000 UTC - event for netserver-2: {kubelet i-0c76fbcc0b95be727} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "6ef3b1853da7091d7298a569ff16df8211cb488466a07e608fdf932e8760bdcd": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 18:03:24.588: INFO: At 2022-12-22 17:57:47 +0000 UTC - event for netserver-3: {kubelet i-0ea878d3671f2eda2} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "35af2e3fa1d0379ff401dd0ad38860a640ea12f76fdaaa79a5e2a5492ff45c98": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 18:03:24.588: INFO: At 2022-12-22 17:57:48 +0000 UTC - event for netserver-1: {kubelet i-076110ae730e268e3} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "0d81c4e30eefb85bf654580c858e4fb4b7cf38746262255979e29557d6ae3cc5": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 18:03:24.588: INFO: At 2022-12-22 17:57:48 +0000 UTC - event for netserver-2: {kubelet i-0c76fbcc0b95be727} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "3810913fd4a7ecb38c5e4dce0cedea43340edf0a3ba3c6c172f6acf753998280": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 18:03:24.588: INFO: At 2022-12-22 17:57:50 +0000 UTC - event for netserver-0: {kubelet i-0243def68c064e8ef} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "d52b4e2aaee1d18ef837774f80b3c5a23ec21c697a8eba969e35289d488da871": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 18:03:24.588: INFO: At 2022-12-22 17:57:58 +0000 UTC - event for netserver-3: {kubelet i-0ea878d3671f2eda2} Pulled: Container image "registry.k8s.io/e2e-test-images/agnhost:2.43" already present on machine Dec 22 18:03:24.588: INFO: At 2022-12-22 17:57:58 +0000 UTC - event for netserver-3: {kubelet i-0ea878d3671f2eda2} Created: Created container webserver Dec 22 18:03:24.588: INFO: At 2022-12-22 17:57:58 +0000 UTC - event for netserver-3: {kubelet i-0ea878d3671f2eda2} Started: Started container webserver Dec 22 18:03:24.588: INFO: At 2022-12-22 17:58:02 +0000 UTC - event for netserver-0: {kubelet i-0243def68c064e8ef} Created: Created container webserver Dec 22 18:03:24.588: INFO: At 2022-12-22 17:58:02 +0000 UTC - event for netserver-0: {kubelet i-0243def68c064e8ef} Pulled: Container image "registry.k8s.io/e2e-test-images/agnhost:2.43" already present on machine Dec 22 18:03:24.588: INFO: At 2022-12-22 17:58:02 +0000 UTC - event for netserver-2: {kubelet i-0c76fbcc0b95be727} Started: Started container webserver Dec 22 18:03:24.588: INFO: At 2022-12-22 17:58:02 +0000 UTC - event for netserver-2: {kubelet i-0c76fbcc0b95be727} Pulled: Container image "registry.k8s.io/e2e-test-images/agnhost:2.43" already present on machine Dec 22 18:03:24.588: INFO: At 2022-12-22 17:58:02 +0000 UTC - event for netserver-2: {kubelet i-0c76fbcc0b95be727} Created: Created container webserver Dec 22 18:03:24.588: INFO: At 2022-12-22 17:58:03 +0000 UTC - event for netserver-0: {kubelet i-0243def68c064e8ef} Started: Started container webserver Dec 22 18:03:24.588: INFO: At 2022-12-22 17:58:03 +0000 UTC - event for netserver-1: {kubelet i-076110ae730e268e3} Started: Started container webserver Dec 22 18:03:24.588: INFO: At 2022-12-22 17:58:03 +0000 UTC - event for netserver-1: {kubelet i-076110ae730e268e3} Created: Created container webserver Dec 22 18:03:24.588: INFO: At 2022-12-22 17:58:03 +0000 UTC - event for netserver-1: {kubelet i-076110ae730e268e3} Pulled: Container image "registry.k8s.io/e2e-test-images/agnhost:2.43" already present on machine Dec 22 18:03:24.588: INFO: At 2022-12-22 17:58:52 +0000 UTC - event for netserver-1: {kubelet i-076110ae730e268e3} Unhealthy: Readiness probe failed: Get "http://172.20.46.200:8083/healthz": context deadline exceeded (Client.Timeout exceeded while awaiting headers) Dec 22 18:03:24.588: INFO: At 2022-12-22 17:58:52 +0000 UTC - event for netserver-1: {kubelet i-076110ae730e268e3} Unhealthy: Liveness probe failed: Get "http://172.20.46.200:8083/healthz": context deadline exceeded (Client.Timeout exceeded while awaiting headers) Dec 22 18:03:24.588: INFO: At 2022-12-22 17:59:52 +0000 UTC - event for netserver-1: {kubelet i-076110ae730e268e3} Killing: Container webserver failed liveness probe, will be restarted Dec 22 18:03:24.588: INFO: At 2022-12-22 17:59:52 +0000 UTC - event for netserver-1: {kubelet i-076110ae730e268e3} Unhealthy: Liveness probe failed: Get "http://172.20.46.200:8083/healthz": dial tcp 172.20.46.200:8083: i/o timeout (Client.Timeout exceeded while awaiting headers) Dec 22 18:03:24.653: INFO: POD NODE PHASE GRACE CONDITIONS Dec 22 18:03:24.653: INFO: netserver-0 i-0243def68c064e8ef Running [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2022-12-22 17:57:21 +0000 UTC } {Ready True 0001-01-01 00:00:00 +0000 UTC 2022-12-22 17:58:22 +0000 UTC } {ContainersReady True 0001-01-01 00:00:00 +0000 UTC 2022-12-22 17:58:22 +0000 UTC } {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2022-12-22 17:57:21 +0000 UTC }] Dec 22 18:03:24.653: INFO: netserver-1 i-076110ae730e268e3 Running [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2022-12-22 17:57:21 +0000 UTC } {Ready False 0001-01-01 00:00:00 +0000 UTC 2022-12-22 17:57:21 +0000 UTC ContainersNotReady containers with unready status: [webserver]} {ContainersReady False 0001-01-01 00:00:00 +0000 UTC 2022-12-22 17:57:21 +0000 UTC ContainersNotReady containers with unready status: [webserver]} {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2022-12-22 17:57:21 +0000 UTC }] Dec 22 18:03:24.653: INFO: netserver-2 i-0c76fbcc0b95be727 Running [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2022-12-22 17:57:21 +0000 UTC } {Ready True 0001-01-01 00:00:00 +0000 UTC 2022-12-22 17:58:12 +0000 UTC } {ContainersReady True 0001-01-01 00:00:00 +0000 UTC 2022-12-22 17:58:12 +0000 UTC } {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2022-12-22 17:57:21 +0000 UTC }] Dec 22 18:03:24.653: INFO: netserver-3 i-0ea878d3671f2eda2 Running [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2022-12-22 17:57:22 +0000 UTC } {Ready True 0001-01-01 00:00:00 +0000 UTC 2022-12-22 17:58:12 +0000 UTC } {ContainersReady True 0001-01-01 00:00:00 +0000 UTC 2022-12-22 17:58:12 +0000 UTC } {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2022-12-22 17:57:22 +0000 UTC }] Dec 22 18:03:24.653: INFO: ... skipping 181 lines ... [1mSTEP:[0m Destroying namespace "nettest-6730" for this suite. [38;5;243m12/22/22 18:03:27.683[0m [38;5;243m<< End Captured GinkgoWriter Output[0m [38;5;9mDec 22 18:03:24.456: timed out while waiting for pod nettest-6730/netserver-1 to be running and ready[0m [38;5;9mIn [1m[It][0m[38;5;9m at: [1mtest/e2e/framework/network/utils.go:866[0m [38;5;243m------------------------------[0m [38;5;9m• [FAILED] [304.247 seconds][0m [sig-network] Networking [38;5;243mtest/e2e/network/common/framework.go:23[0m Granular Checks: Services [38;5;243mtest/e2e/network/networking.go:145[0m [38;5;9m[1m[It] should function for pod-Service: udp[0m [38;5;243mtest/e2e/network/networking.go:162[0m ... skipping 316 lines ... Dec 22 18:03:55.961: INFO: Pod "netserver-0": Phase="Running", Reason="", readiness=false. Elapsed: 4m58.129495706s Dec 22 18:03:55.961: INFO: The phase of Pod netserver-0 is Running (Ready = false) Dec 22 18:03:57.961: INFO: Pod "netserver-0": Phase="Running", Reason="", readiness=false. Elapsed: 5m0.129173031s Dec 22 18:03:57.961: INFO: The phase of Pod netserver-0 is Running (Ready = false) Dec 22 18:03:58.025: INFO: Pod "netserver-0": Phase="Running", Reason="", readiness=false. Elapsed: 5m0.19353594s Dec 22 18:03:58.025: INFO: The phase of Pod netserver-0 is Running (Ready = false) Dec 22 18:03:58.028: INFO: Unexpected error: <*pod.timeoutError | 0xc001b7ede0>: { msg: "timed out while waiting for pod nettest-4847/netserver-0 to be running and ready", observedObjects: [ <*v1.Pod | 0xc00378fb00>{ TypeMeta: {Kind: "", APIVersion: ""}, ObjectMeta: { ... skipping 128 lines ... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output Dec 22 18:03:58.028: FAIL: timed out while waiting for pod nettest-4847/netserver-0 to be running and ready Full Stack Trace k8s.io/kubernetes/test/e2e/framework/network.(*NetworkingTestConfig).createNetProxyPods(0xc00281a000, {0x75c6f5c, 0x9}, 0xc004c092f0) test/e2e/framework/network/utils.go:866 +0x1d0 k8s.io/kubernetes/test/e2e/framework/network.(*NetworkingTestConfig).setupCore(0xc00281a000, 0x7f1c34077df8?) test/e2e/framework/network/utils.go:763 +0x55 ... skipping 10 lines ... test/e2e/framework/metrics/init/init.go:33 [DeferCleanup (Each)] [sig-network] Networking dump namespaces | framework.go:196 [1mSTEP:[0m dump namespace information after failure [38;5;243m12/22/22 18:03:58.095[0m [1mSTEP:[0m Collecting events from namespace "nettest-4847". [38;5;243m12/22/22 18:03:58.096[0m [1mSTEP:[0m Found 36 events. [38;5;243m12/22/22 18:03:58.161[0m Dec 22 18:03:58.161: INFO: At 2022-12-22 17:58:57 +0000 UTC - event for netserver-0: {kubelet i-0243def68c064e8ef} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "8ad35dc116f6d619e4d27c11586a1b1c6853ff2bfb804a13c109159207e4a614": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 18:03:58.161: INFO: At 2022-12-22 17:58:57 +0000 UTC - event for netserver-0: {default-scheduler } Scheduled: Successfully assigned nettest-4847/netserver-0 to i-0243def68c064e8ef Dec 22 18:03:58.161: INFO: At 2022-12-22 17:58:57 +0000 UTC - event for netserver-1: {default-scheduler } Scheduled: Successfully assigned nettest-4847/netserver-1 to i-076110ae730e268e3 Dec 22 18:03:58.161: INFO: At 2022-12-22 17:58:57 +0000 UTC - event for netserver-2: {default-scheduler } Scheduled: Successfully assigned nettest-4847/netserver-2 to i-0c76fbcc0b95be727 Dec 22 18:03:58.161: INFO: At 2022-12-22 17:58:57 +0000 UTC - event for netserver-3: {default-scheduler } Scheduled: Successfully assigned nettest-4847/netserver-3 to i-0ea878d3671f2eda2 Dec 22 18:03:58.161: INFO: At 2022-12-22 17:58:58 +0000 UTC - event for netserver-1: {kubelet i-076110ae730e268e3} Pulled: Container image "registry.k8s.io/e2e-test-images/agnhost:2.43" already present on machine Dec 22 18:03:58.161: INFO: At 2022-12-22 17:58:58 +0000 UTC - event for netserver-1: {kubelet i-076110ae730e268e3} Started: Started container webserver Dec 22 18:03:58.161: INFO: At 2022-12-22 17:58:58 +0000 UTC - event for netserver-1: {kubelet i-076110ae730e268e3} Created: Created container webserver Dec 22 18:03:58.161: INFO: At 2022-12-22 17:58:58 +0000 UTC - event for netserver-2: {kubelet i-0c76fbcc0b95be727} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "7d7d07a62afb9480d342fee621c4672154eaff810fa8263d9dff5b2950700523": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 18:03:58.161: INFO: At 2022-12-22 17:58:58 +0000 UTC - event for netserver-3: {kubelet i-0ea878d3671f2eda2} Started: Started container webserver Dec 22 18:03:58.161: INFO: At 2022-12-22 17:58:58 +0000 UTC - event for netserver-3: {kubelet i-0ea878d3671f2eda2} Created: Created container webserver Dec 22 18:03:58.161: INFO: At 2022-12-22 17:58:58 +0000 UTC - event for netserver-3: {kubelet i-0ea878d3671f2eda2} Pulled: Container image "registry.k8s.io/e2e-test-images/agnhost:2.43" already present on machine Dec 22 18:03:58.161: INFO: At 2022-12-22 17:59:09 +0000 UTC - event for netserver-0: {kubelet i-0243def68c064e8ef} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "64646b64c1253b3c78f0be8f10efe4a1d507586c5bccd2f23f887751b30bf0c5": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 18:03:58.161: INFO: At 2022-12-22 17:59:13 +0000 UTC - event for netserver-2: {kubelet i-0c76fbcc0b95be727} Created: Created container webserver Dec 22 18:03:58.161: INFO: At 2022-12-22 17:59:13 +0000 UTC - event for netserver-2: {kubelet i-0c76fbcc0b95be727} Pulled: Container image "registry.k8s.io/e2e-test-images/agnhost:2.43" already present on machine Dec 22 18:03:58.161: INFO: At 2022-12-22 17:59:13 +0000 UTC - event for netserver-2: {kubelet i-0c76fbcc0b95be727} Started: Started container webserver Dec 22 18:03:58.161: INFO: At 2022-12-22 17:59:24 +0000 UTC - event for netserver-0: {kubelet i-0243def68c064e8ef} Pulled: Container image "registry.k8s.io/e2e-test-images/agnhost:2.43" already present on machine Dec 22 18:03:58.161: INFO: At 2022-12-22 17:59:24 +0000 UTC - event for netserver-0: {kubelet i-0243def68c064e8ef} Started: Started container webserver Dec 22 18:03:58.161: INFO: At 2022-12-22 17:59:24 +0000 UTC - event for netserver-0: {kubelet i-0243def68c064e8ef} Created: Created container webserver Dec 22 18:03:58.161: INFO: At 2022-12-22 17:59:38 +0000 UTC - event for netserver-1: {kubelet i-076110ae730e268e3} Unhealthy: Liveness probe failed: Get "http://172.20.54.167:8083/healthz": context deadline exceeded (Client.Timeout exceeded while awaiting headers) Dec 22 18:03:58.161: INFO: At 2022-12-22 17:59:38 +0000 UTC - event for netserver-1: {kubelet i-076110ae730e268e3} Unhealthy: Readiness probe failed: Get "http://172.20.54.167:8083/healthz": dial tcp 172.20.54.167:8083: i/o timeout (Client.Timeout exceeded while awaiting headers) Dec 22 18:03:58.161: INFO: At 2022-12-22 17:59:38 +0000 UTC - event for netserver-3: {kubelet i-0ea878d3671f2eda2} Unhealthy: Readiness probe failed: Get "http://172.20.43.253:8083/healthz": context deadline exceeded (Client.Timeout exceeded while awaiting headers) Dec 22 18:03:58.161: INFO: At 2022-12-22 17:59:38 +0000 UTC - event for netserver-3: {kubelet i-0ea878d3671f2eda2} Unhealthy: Liveness probe failed: Get "http://172.20.43.253:8083/healthz": context deadline exceeded (Client.Timeout exceeded while awaiting headers) Dec 22 18:03:58.161: INFO: At 2022-12-22 17:59:58 +0000 UTC - event for netserver-2: {kubelet i-0c76fbcc0b95be727} Unhealthy: Liveness probe failed: Get "http://172.20.34.169:8083/healthz": context deadline exceeded (Client.Timeout exceeded while awaiting headers) Dec 22 18:03:58.161: INFO: At 2022-12-22 17:59:58 +0000 UTC - event for netserver-2: {kubelet i-0c76fbcc0b95be727} Unhealthy: Readiness probe failed: Get "http://172.20.34.169:8083/healthz": context deadline exceeded (Client.Timeout exceeded while awaiting headers) Dec 22 18:03:58.161: INFO: At 2022-12-22 18:00:07 +0000 UTC - event for netserver-0: {kubelet i-0243def68c064e8ef} Unhealthy: Readiness probe failed: Get "http://172.20.38.120:8083/healthz": context deadline exceeded (Client.Timeout exceeded while awaiting headers) Dec 22 18:03:58.161: INFO: At 2022-12-22 18:00:07 +0000 UTC - event for netserver-0: {kubelet i-0243def68c064e8ef} Unhealthy: Liveness probe failed: Get "http://172.20.38.120:8083/healthz": dial tcp 172.20.38.120:8083: i/o timeout (Client.Timeout exceeded while awaiting headers) Dec 22 18:03:58.161: INFO: At 2022-12-22 18:00:08 +0000 UTC - event for netserver-1: {kubelet i-076110ae730e268e3} Unhealthy: Readiness probe failed: Get "http://172.20.54.167:8083/healthz": context deadline exceeded (Client.Timeout exceeded while awaiting headers) Dec 22 18:03:58.161: INFO: At 2022-12-22 18:00:08 +0000 UTC - event for netserver-3: {kubelet i-0ea878d3671f2eda2} Unhealthy: Readiness probe failed: Get "http://172.20.43.253:8083/healthz": dial tcp 172.20.43.253:8083: i/o timeout (Client.Timeout exceeded while awaiting headers) Dec 22 18:03:58.161: INFO: At 2022-12-22 18:00:38 +0000 UTC - event for netserver-1: {kubelet i-076110ae730e268e3} Killing: Container webserver failed liveness probe, will be restarted Dec 22 18:03:58.161: INFO: At 2022-12-22 18:00:38 +0000 UTC - event for netserver-3: {kubelet i-0ea878d3671f2eda2} Killing: Container webserver failed liveness probe, will be restarted Dec 22 18:03:58.161: INFO: At 2022-12-22 18:00:58 +0000 UTC - event for netserver-2: {kubelet i-0c76fbcc0b95be727} Killing: Container webserver failed liveness probe, will be restarted Dec 22 18:03:58.161: INFO: At 2022-12-22 18:00:58 +0000 UTC - event for netserver-2: {kubelet i-0c76fbcc0b95be727} Unhealthy: Readiness probe failed: Get "http://172.20.34.169:8083/healthz": dial tcp 172.20.34.169:8083: i/o timeout (Client.Timeout exceeded while awaiting headers) Dec 22 18:03:58.161: INFO: At 2022-12-22 18:01:07 +0000 UTC - event for netserver-0: {kubelet i-0243def68c064e8ef} Unhealthy: Liveness probe failed: Get "http://172.20.38.120:8083/healthz": context deadline exceeded (Client.Timeout exceeded while awaiting headers) Dec 22 18:03:58.161: INFO: At 2022-12-22 18:01:07 +0000 UTC - event for netserver-0: {kubelet i-0243def68c064e8ef} Killing: Container webserver failed liveness probe, will be restarted Dec 22 18:03:58.161: INFO: At 2022-12-22 18:01:48 +0000 UTC - event for netserver-3: {kubelet i-0ea878d3671f2eda2} Unhealthy: Liveness probe failed: Get "http://172.20.43.253:8083/healthz": dial tcp 172.20.43.253:8083: i/o timeout (Client.Timeout exceeded while awaiting headers) Dec 22 18:03:58.226: INFO: POD NODE PHASE GRACE CONDITIONS Dec 22 18:03:58.226: INFO: netserver-0 i-0243def68c064e8ef Running [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2022-12-22 17:58:57 +0000 UTC } {Ready False 0001-01-01 00:00:00 +0000 UTC 2022-12-22 17:58:57 +0000 UTC ContainersNotReady containers with unready status: [webserver]} {ContainersReady False 0001-01-01 00:00:00 +0000 UTC 2022-12-22 17:58:57 +0000 UTC ContainersNotReady containers with unready status: [webserver]} {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2022-12-22 17:58:57 +0000 UTC }] Dec 22 18:03:58.226: INFO: netserver-1 i-076110ae730e268e3 Running [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2022-12-22 17:58:57 +0000 UTC } {Ready False 0001-01-01 00:00:00 +0000 UTC 2022-12-22 17:58:57 +0000 UTC ContainersNotReady containers with unready status: [webserver]} {ContainersReady False 0001-01-01 00:00:00 +0000 UTC 2022-12-22 17:58:57 +0000 UTC ContainersNotReady containers with unready status: [webserver]} {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2022-12-22 17:58:57 +0000 UTC }] Dec 22 18:03:58.226: INFO: netserver-2 i-0c76fbcc0b95be727 Running [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2022-12-22 17:58:57 +0000 UTC } {Ready False 0001-01-01 00:00:00 +0000 UTC 2022-12-22 17:58:57 +0000 UTC ContainersNotReady containers with unready status: [webserver]} {ContainersReady False 0001-01-01 00:00:00 +0000 UTC 2022-12-22 17:58:57 +0000 UTC ContainersNotReady containers with unready status: [webserver]} {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2022-12-22 17:58:57 +0000 UTC }] Dec 22 18:03:58.226: INFO: netserver-3 i-0ea878d3671f2eda2 Running [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2022-12-22 17:58:57 +0000 UTC } {Ready False 0001-01-01 00:00:00 +0000 UTC 2022-12-22 17:58:57 +0000 UTC ContainersNotReady containers with unready status: [webserver]} {ContainersReady False 0001-01-01 00:00:00 +0000 UTC 2022-12-22 17:58:57 +0000 UTC ContainersNotReady containers with unready status: [webserver]} {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2022-12-22 17:58:57 +0000 UTC }] Dec 22 18:03:58.226: INFO: ... skipping 165 lines ... [1mSTEP:[0m Destroying namespace "nettest-4847" for this suite. [38;5;243m12/22/22 18:04:01.148[0m [38;5;243m<< End Captured GinkgoWriter Output[0m [38;5;9mDec 22 18:03:58.028: timed out while waiting for pod nettest-4847/netserver-0 to be running and ready[0m [38;5;9mIn [1m[It][0m[38;5;9m at: [1mtest/e2e/framework/network/utils.go:866[0m [38;5;243m------------------------------[0m [38;5;9m• [FAILED] [326.311 seconds][0m [sig-network] Networking [38;5;243mtest/e2e/network/common/framework.go:23[0m Granular Checks: Services [38;5;243mtest/e2e/network/networking.go:145[0m [38;5;9m[1m[It] should update endpoints: http[0m [38;5;243mtest/e2e/network/networking.go:328[0m ... skipping 342 lines ... Dec 22 18:04:01.521: INFO: Pod "netserver-1": Phase="Running", Reason="", readiness=false. Elapsed: 4m58.12750056s Dec 22 18:04:01.521: INFO: The phase of Pod netserver-1 is Running (Ready = false) Dec 22 18:04:03.521: INFO: Pod "netserver-1": Phase="Running", Reason="", readiness=false. Elapsed: 5m0.127989271s Dec 22 18:04:03.521: INFO: The phase of Pod netserver-1 is Running (Ready = false) Dec 22 18:04:03.584: INFO: Pod "netserver-1": Phase="Running", Reason="", readiness=false. Elapsed: 5m0.191477799s Dec 22 18:04:03.585: INFO: The phase of Pod netserver-1 is Running (Ready = false) Dec 22 18:04:03.587: INFO: Unexpected error: <*pod.timeoutError | 0xc0033b3710>: { msg: "timed out while waiting for pod nettest-9129/netserver-1 to be running and ready", observedObjects: [ <*v1.Pod | 0xc00537ed80>{ TypeMeta: {Kind: "", APIVersion: ""}, ObjectMeta: { ... skipping 128 lines ... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output Dec 22 18:04:03.587: FAIL: timed out while waiting for pod nettest-9129/netserver-1 to be running and ready Full Stack Trace k8s.io/kubernetes/test/e2e/framework/network.(*NetworkingTestConfig).createNetProxyPods(0xc00026c540, {0x75c6f5c, 0x9}, 0xc0045420f0) test/e2e/framework/network/utils.go:866 +0x1d0 k8s.io/kubernetes/test/e2e/framework/network.(*NetworkingTestConfig).setupCore(0xc00026c540, 0x7f8d08a76bd0?) test/e2e/framework/network/utils.go:763 +0x55 ... skipping 17 lines ... Dec 22 18:04:03.719: INFO: At 2022-12-22 17:58:41 +0000 UTC - event for netserver-0: {kubelet i-0243def68c064e8ef} Pulled: Container image "registry.k8s.io/e2e-test-images/agnhost:2.43" already present on machine Dec 22 18:04:03.719: INFO: At 2022-12-22 17:58:41 +0000 UTC - event for netserver-0: {kubelet i-0243def68c064e8ef} Created: Created container webserver Dec 22 18:04:03.719: INFO: At 2022-12-22 17:58:41 +0000 UTC - event for netserver-1: {default-scheduler } Scheduled: Successfully assigned nettest-9129/netserver-1 to i-076110ae730e268e3 Dec 22 18:04:03.719: INFO: At 2022-12-22 17:58:41 +0000 UTC - event for netserver-1: {kubelet i-076110ae730e268e3} Pulled: Container image "registry.k8s.io/e2e-test-images/agnhost:2.43" already present on machine Dec 22 18:04:03.719: INFO: At 2022-12-22 17:58:41 +0000 UTC - event for netserver-1: {kubelet i-076110ae730e268e3} Created: Created container webserver Dec 22 18:04:03.719: INFO: At 2022-12-22 17:58:41 +0000 UTC - event for netserver-1: {kubelet i-076110ae730e268e3} Started: Started container webserver Dec 22 18:04:03.719: INFO: At 2022-12-22 17:58:41 +0000 UTC - event for netserver-2: {kubelet i-0c76fbcc0b95be727} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "5439f6c1f2f483703297bb98891574f5fa4e493622b833f867d8996f221e8b0c": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 18:04:03.719: INFO: At 2022-12-22 17:58:41 +0000 UTC - event for netserver-2: {default-scheduler } Scheduled: Successfully assigned nettest-9129/netserver-2 to i-0c76fbcc0b95be727 Dec 22 18:04:03.719: INFO: At 2022-12-22 17:58:41 +0000 UTC - event for netserver-3: {kubelet i-0ea878d3671f2eda2} Created: Created container webserver Dec 22 18:04:03.719: INFO: At 2022-12-22 17:58:41 +0000 UTC - event for netserver-3: {kubelet i-0ea878d3671f2eda2} Pulled: Container image "registry.k8s.io/e2e-test-images/agnhost:2.43" already present on machine Dec 22 18:04:03.719: INFO: At 2022-12-22 17:58:41 +0000 UTC - event for netserver-3: {default-scheduler } Scheduled: Successfully assigned nettest-9129/netserver-3 to i-0ea878d3671f2eda2 Dec 22 18:04:03.719: INFO: At 2022-12-22 17:58:41 +0000 UTC - event for netserver-3: {kubelet i-0ea878d3671f2eda2} Started: Started container webserver Dec 22 18:04:03.719: INFO: At 2022-12-22 17:58:42 +0000 UTC - event for netserver-0: {kubelet i-0243def68c064e8ef} Started: Started container webserver Dec 22 18:04:03.719: INFO: At 2022-12-22 17:58:56 +0000 UTC - event for netserver-2: {kubelet i-0c76fbcc0b95be727} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "612c388cb14cbbf1eb451b43df04c33b727156de85f7d2791848815ab2982682": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 18:04:03.719: INFO: At 2022-12-22 17:59:10 +0000 UTC - event for netserver-2: {kubelet i-0c76fbcc0b95be727} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "2b376381a2f8b2380788393c3b0524c2975344f6f306d4beb7f587ff700b17d2": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 18:04:03.719: INFO: At 2022-12-22 17:59:21 +0000 UTC - event for netserver-1: {kubelet i-076110ae730e268e3} Unhealthy: Liveness probe failed: Get "http://172.20.33.201:8083/healthz": context deadline exceeded (Client.Timeout exceeded while awaiting headers) Dec 22 18:04:03.719: INFO: At 2022-12-22 17:59:21 +0000 UTC - event for netserver-1: {kubelet i-076110ae730e268e3} Unhealthy: Readiness probe failed: Get "http://172.20.33.201:8083/healthz": context deadline exceeded (Client.Timeout exceeded while awaiting headers) Dec 22 18:04:03.719: INFO: At 2022-12-22 17:59:22 +0000 UTC - event for netserver-2: {kubelet i-0c76fbcc0b95be727} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "a837621c6e5a6b85c8a5a5533969d2c264894612400b9fb4bb24609d9ff3bddb": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 18:04:03.719: INFO: At 2022-12-22 17:59:36 +0000 UTC - event for netserver-2: {kubelet i-0c76fbcc0b95be727} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "aed55e62a018df5f012637509110889a612867fa9d7812b10c6f21bfd15ea2e9": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 18:04:03.719: INFO: At 2022-12-22 17:59:47 +0000 UTC - event for netserver-2: {kubelet i-0c76fbcc0b95be727} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "9610b469d0443e1ad0084d12a34e5191198a02b42148ed9af7ea6ad2c9fbe84e": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 18:04:03.719: INFO: At 2022-12-22 17:59:51 +0000 UTC - event for netserver-1: {kubelet i-076110ae730e268e3} Unhealthy: Readiness probe failed: Get "http://172.20.33.201:8083/healthz": dial tcp 172.20.33.201:8083: i/o timeout (Client.Timeout exceeded while awaiting headers) Dec 22 18:04:03.719: INFO: At 2022-12-22 18:00:02 +0000 UTC - event for netserver-2: {kubelet i-0c76fbcc0b95be727} Pulled: Container image "registry.k8s.io/e2e-test-images/agnhost:2.43" already present on machine Dec 22 18:04:03.719: INFO: At 2022-12-22 18:00:02 +0000 UTC - event for netserver-2: {kubelet i-0c76fbcc0b95be727} Created: Created container webserver Dec 22 18:04:03.719: INFO: At 2022-12-22 18:00:02 +0000 UTC - event for netserver-2: {kubelet i-0c76fbcc0b95be727} Started: Started container webserver Dec 22 18:04:03.719: INFO: At 2022-12-22 18:00:21 +0000 UTC - event for netserver-1: {kubelet i-076110ae730e268e3} Killing: Container webserver failed liveness probe, will be restarted Dec 22 18:04:03.719: INFO: At 2022-12-22 18:00:51 +0000 UTC - event for netserver-2: {kubelet i-0c76fbcc0b95be727} Unhealthy: Readiness probe failed: Get "http://172.20.37.208:8083/healthz": context deadline exceeded (Client.Timeout exceeded while awaiting headers) Dec 22 18:04:03.719: INFO: At 2022-12-22 18:00:51 +0000 UTC - event for netserver-2: {kubelet i-0c76fbcc0b95be727} Unhealthy: Liveness probe failed: Get "http://172.20.37.208:8083/healthz": context deadline exceeded (Client.Timeout exceeded while awaiting headers) Dec 22 18:04:03.719: INFO: At 2022-12-22 18:01:21 +0000 UTC - event for netserver-2: {kubelet i-0c76fbcc0b95be727} Unhealthy: Readiness probe failed: Get "http://172.20.37.208:8083/healthz": dial tcp 172.20.37.208:8083: i/o timeout (Client.Timeout exceeded while awaiting headers) Dec 22 18:04:03.719: INFO: At 2022-12-22 18:01:21 +0000 UTC - event for netserver-2: {kubelet i-0c76fbcc0b95be727} Unhealthy: Liveness probe failed: Get "http://172.20.37.208:8083/healthz": dial tcp 172.20.37.208:8083: i/o timeout (Client.Timeout exceeded while awaiting headers) Dec 22 18:04:03.719: INFO: At 2022-12-22 18:01:51 +0000 UTC - event for netserver-2: {kubelet i-0c76fbcc0b95be727} Killing: Container webserver failed liveness probe, will be restarted Dec 22 18:04:03.783: INFO: POD NODE PHASE GRACE CONDITIONS Dec 22 18:04:03.783: INFO: netserver-0 i-0243def68c064e8ef Running [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2022-12-22 17:58:40 +0000 UTC } {Ready True 0001-01-01 00:00:00 +0000 UTC 2022-12-22 17:59:01 +0000 UTC } {ContainersReady True 0001-01-01 00:00:00 +0000 UTC 2022-12-22 17:59:01 +0000 UTC } {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2022-12-22 17:58:40 +0000 UTC }] Dec 22 18:04:03.783: INFO: netserver-1 i-076110ae730e268e3 Running [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2022-12-22 17:58:41 +0000 UTC } {Ready False 0001-01-01 00:00:00 +0000 UTC 2022-12-22 17:58:41 +0000 UTC ContainersNotReady containers with unready status: [webserver]} {ContainersReady False 0001-01-01 00:00:00 +0000 UTC 2022-12-22 17:58:41 +0000 UTC ContainersNotReady containers with unready status: [webserver]} {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2022-12-22 17:58:41 +0000 UTC }] Dec 22 18:04:03.783: INFO: netserver-2 i-0c76fbcc0b95be727 Running [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2022-12-22 17:58:41 +0000 UTC } {Ready False 0001-01-01 00:00:00 +0000 UTC 2022-12-22 17:58:41 +0000 UTC ContainersNotReady containers with unready status: [webserver]} {ContainersReady False 0001-01-01 00:00:00 +0000 UTC 2022-12-22 17:58:41 +0000 UTC ContainersNotReady containers with unready status: [webserver]} {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2022-12-22 17:58:41 +0000 UTC }] Dec 22 18:04:03.783: INFO: netserver-3 i-0ea878d3671f2eda2 Running [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2022-12-22 17:58:41 +0000 UTC } {Ready True 0001-01-01 00:00:00 +0000 UTC 2022-12-22 17:58:51 +0000 UTC } {ContainersReady True 0001-01-01 00:00:00 +0000 UTC 2022-12-22 17:58:51 +0000 UTC } {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2022-12-22 17:58:41 +0000 UTC }] Dec 22 18:04:03.783: INFO: ... skipping 165 lines ... [1mSTEP:[0m Destroying namespace "nettest-9129" for this suite. [38;5;243m12/22/22 18:04:06.592[0m [38;5;243m<< End Captured GinkgoWriter Output[0m [38;5;9mDec 22 18:04:03.587: timed out while waiting for pod nettest-9129/netserver-1 to be running and ready[0m [38;5;9mIn [1m[It][0m[38;5;9m at: [1mtest/e2e/framework/network/utils.go:866[0m [38;5;243m------------------------------[0m [38;5;9m• [FAILED] [400.623 seconds][0m External Storage [Driver: ebs.csi.aws.com] [38;5;243mtest/e2e/storage/external/external.go:173[0m [Testpattern: Dynamic PV (default fs)(allowExpansion)] volume-expand [38;5;243mtest/e2e/storage/framework/testsuite.go:50[0m [38;5;9m[1m[It] should resize volume when PVC is edited while pod is using it[0m [38;5;243mtest/e2e/storage/testsuites/volume_expand.go:248[0m ... skipping 97 lines ... Dec 22 18:04:34.786: INFO: PersistentVolume pvc-b417d326-81a1-48eb-9208-dd1cfaf1fc12 found and phase=Released (4m38.624661258s) Dec 22 18:04:39.852: INFO: PersistentVolume pvc-b417d326-81a1-48eb-9208-dd1cfaf1fc12 found and phase=Released (4m43.69161154s) Dec 22 18:04:44.919: INFO: PersistentVolume pvc-b417d326-81a1-48eb-9208-dd1cfaf1fc12 found and phase=Released (4m48.758529103s) Dec 22 18:04:49.986: INFO: PersistentVolume pvc-b417d326-81a1-48eb-9208-dd1cfaf1fc12 found and phase=Released (4m53.825153866s) Dec 22 18:04:55.053: INFO: PersistentVolume pvc-b417d326-81a1-48eb-9208-dd1cfaf1fc12 found and phase=Released (4m58.892327414s) [1mSTEP:[0m Deleting sc [38;5;243m12/22/22 18:05:00.054[0m Dec 22 18:05:00.122: INFO: Unexpected error: while cleaning up resource: <errors.aggregate | len:1, cap:1>: [ <errors.aggregate | len:1, cap:1>[ <*fmt.wrapError | 0xc0047261c0>{ msg: "persistent Volume pvc-b417d326-81a1-48eb-9208-dd1cfaf1fc12 not deleted by dynamic provisioner: PersistentVolume pvc-b417d326-81a1-48eb-9208-dd1cfaf1fc12 still exists within 5m0s", err: <*errors.errorString | 0xc0057de440>{ s: "PersistentVolume pvc-b417d326-81a1-48eb-9208-dd1cfaf1fc12 still exists within 5m0s", }, }, ], ] Dec 22 18:05:00.122: FAIL: while cleaning up resource: persistent Volume pvc-b417d326-81a1-48eb-9208-dd1cfaf1fc12 not deleted by dynamic provisioner: PersistentVolume pvc-b417d326-81a1-48eb-9208-dd1cfaf1fc12 still exists within 5m0s Full Stack Trace k8s.io/kubernetes/test/e2e/storage/testsuites.(*volumeExpandTestSuite).DefineTests.func2() test/e2e/storage/testsuites/volume_expand.go:150 +0x3a6 k8s.io/kubernetes/test/e2e/storage/testsuites.(*volumeExpandTestSuite).DefineTests.func5() test/e2e/storage/testsuites/volume_expand.go:298 +0xb34 ... skipping 176 lines ... [1mSTEP:[0m Destroying namespace "volume-expand-4020" for this suite. [38;5;243m12/22/22 18:05:02.932[0m [38;5;243m<< End Captured GinkgoWriter Output[0m [38;5;9mDec 22 18:05:00.122: while cleaning up resource: persistent Volume pvc-b417d326-81a1-48eb-9208-dd1cfaf1fc12 not deleted by dynamic provisioner: PersistentVolume pvc-b417d326-81a1-48eb-9208-dd1cfaf1fc12 still exists within 5m0s[0m [38;5;9mIn [1m[It][0m[38;5;9m at: [1mtest/e2e/storage/testsuites/volume_expand.go:150[0m [38;5;243m------------------------------[0m [38;5;9m• [FAILED] [1326.907 seconds][0m [sig-storage] CSI Volumes [38;5;243mtest/e2e/storage/utils/framework.go:23[0m [Driver: csi-hostpath] [38;5;243mtest/e2e/storage/csi_volumes.go:40[0m [Testpattern: Generic Ephemeral-volume (default fs) (immediate-binding)] ephemeral [38;5;243mtest/e2e/storage/framework/testsuite.go:50[0m ... skipping 337 lines ... Dec 22 18:07:37.572: INFO: PersistentVolume pvc-4bf151be-a5f3-4cc2-b4d0-293ab5748d60 found and phase=Bound (4m33.537469355s) Dec 22 18:07:42.637: INFO: PersistentVolume pvc-4bf151be-a5f3-4cc2-b4d0-293ab5748d60 found and phase=Bound (4m38.602044037s) Dec 22 18:07:47.701: INFO: PersistentVolume pvc-4bf151be-a5f3-4cc2-b4d0-293ab5748d60 found and phase=Bound (4m43.66630872s) Dec 22 18:07:52.765: INFO: PersistentVolume pvc-4bf151be-a5f3-4cc2-b4d0-293ab5748d60 found and phase=Bound (4m48.730310156s) Dec 22 18:07:57.829: INFO: PersistentVolume pvc-4bf151be-a5f3-4cc2-b4d0-293ab5748d60 found and phase=Bound (4m53.794054862s) Dec 22 18:08:02.893: INFO: PersistentVolume pvc-4bf151be-a5f3-4cc2-b4d0-293ab5748d60 found and phase=Bound (4m58.858038022s) Dec 22 18:08:07.958: FAIL: no dangling PVCs Expected <[]v1.PersistentVolumeClaim | len:1, cap:1>: - metadata: annotations: pv.kubernetes.io/bind-completed: "yes" pv.kubernetes.io/bound-by-controller: "yes" ... skipping 88 lines ... Dec 22 18:08:08.153: INFO: At 2022-12-22 17:51:19 +0000 UTC - event for inline-volume-pcpf6: {default-scheduler } FailedScheduling: 0/5 nodes are available: waiting for ephemeral volume controller to create the persistentvolumeclaim "inline-volume-pcpf6-my-volume". preemption: 0/5 nodes are available: 5 No preemption victims found for incoming pod.. Dec 22 18:08:08.153: INFO: At 2022-12-22 17:51:19 +0000 UTC - event for inline-volume-pcpf6-my-volume: {persistentvolume-controller } ProvisioningFailed: storageclass.storage.k8s.io "no-such-storage-class" not found Dec 22 18:08:08.153: INFO: At 2022-12-22 17:51:22 +0000 UTC - event for inline-volume-tester-6dz24: {default-scheduler } FailedScheduling: 0/5 nodes are available: waiting for ephemeral volume controller to create the persistentvolumeclaim "inline-volume-tester-6dz24-my-volume-0". preemption: 0/5 nodes are available: 5 No preemption victims found for incoming pod.. Dec 22 18:08:08.153: INFO: At 2022-12-22 17:51:22 +0000 UTC - event for inline-volume-tester-6dz24-my-volume-0: {persistentvolume-controller } ExternalProvisioning: waiting for a volume to be created, either by external provisioner "csi-hostpath-ephemeral-6886" or manually created by system administrator Dec 22 18:08:08.153: INFO: At 2022-12-22 17:51:24 +0000 UTC - event for inline-volume-tester-6dz24: {default-scheduler } FailedScheduling: 0/5 nodes are available: pod has unbound immediate PersistentVolumeClaims. preemption: 0/5 nodes are available: 5 No preemption victims found for incoming pod.. Dec 22 18:08:08.153: INFO: At 2022-12-22 17:51:46 +0000 UTC - event for inline-volume-tester-6dz24-my-volume-0: {csi-hostpath-ephemeral-6886_csi-hostpathplugin-0_081ae7f4-bdd4-40af-8239-93020b44b16e } Provisioning: External provisioner is provisioning volume for claim "ephemeral-6886/inline-volume-tester-6dz24-my-volume-0" Dec 22 18:08:08.153: INFO: At 2022-12-22 17:51:46 +0000 UTC - event for inline-volume-tester-6dz24-my-volume-0: {csi-hostpath-ephemeral-6886_csi-hostpathplugin-0_081ae7f4-bdd4-40af-8239-93020b44b16e } ProvisioningFailed: failed to provision volume with StorageClass "ephemeral-6886wcp4d": error generating accessibility requirements: no available topology found Dec 22 18:08:08.153: INFO: At 2022-12-22 17:51:47 +0000 UTC - event for inline-volume-tester-6dz24-my-volume-0: {csi-hostpath-ephemeral-6886_csi-hostpathplugin-0_081ae7f4-bdd4-40af-8239-93020b44b16e } ProvisioningSucceeded: Successfully provisioned volume pvc-4bf151be-a5f3-4cc2-b4d0-293ab5748d60 Dec 22 18:08:08.153: INFO: At 2022-12-22 17:51:48 +0000 UTC - event for inline-volume-tester-6dz24: {default-scheduler } Scheduled: Successfully assigned ephemeral-6886/inline-volume-tester-6dz24 to i-0243def68c064e8ef Dec 22 18:08:08.153: INFO: At 2022-12-22 17:51:49 +0000 UTC - event for inline-volume-tester-6dz24: {attachdetach-controller } SuccessfulAttachVolume: AttachVolume.Attach succeeded for volume "pvc-4bf151be-a5f3-4cc2-b4d0-293ab5748d60" Dec 22 18:08:08.153: INFO: At 2022-12-22 17:51:50 +0000 UTC - event for inline-volume-tester-6dz24: {kubelet i-0243def68c064e8ef} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "c4ab04a7af8fc016ff7a17c46d8b5d408999b00d5189191003e506c34e65d849": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 18:08:08.153: INFO: At 2022-12-22 17:52:04 +0000 UTC - event for inline-volume-tester-6dz24: {kubelet i-0243def68c064e8ef} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "9fc553940d68026c6589a7cd84cfde902a9039b3b402862d193c4f59934b8c78": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 18:08:08.153: INFO: At 2022-12-22 17:52:16 +0000 UTC - event for inline-volume-tester-6dz24: {kubelet i-0243def68c064e8ef} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "2a4371d731d4d2d5b23066056fa34d47d6cdaf264d9c7a766787c4a66b4b1190": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 18:08:08.153: INFO: At 2022-12-22 17:52:30 +0000 UTC - event for inline-volume-tester-6dz24: {kubelet i-0243def68c064e8ef} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "d299da143ac5f0b1058f84ae74d5f8ab0e1241b7125869db657ebcefd84e950c": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 18:08:08.153: INFO: At 2022-12-22 17:52:42 +0000 UTC - event for inline-volume-tester-6dz24: {kubelet i-0243def68c064e8ef} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "427413ff157ee7a697964e9c1218669da3fd8dbbcf9dc4cc6ff3666a1d1cb022": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 18:08:08.153: INFO: At 2022-12-22 17:52:54 +0000 UTC - event for inline-volume-tester-6dz24: {kubelet i-0243def68c064e8ef} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "9571355324ad2cd3911fe094de3018f4054b3ffff283d3b5b347518afdf91fe8": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 18:08:08.153: INFO: At 2022-12-22 17:53:08 +0000 UTC - event for inline-volume-tester-6dz24: {kubelet i-0243def68c064e8ef} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "6f11b4d8968f9d2b3ef7e8f0e04cf5a45133c5decff736c4e849000d6667aa51": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 18:08:08.153: INFO: At 2022-12-22 17:53:20 +0000 UTC - event for inline-volume-tester-6dz24: {kubelet i-0243def68c064e8ef} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "e9b304c88b22ee8c5ea5005e447f1a78b0bdcb5c5ad0cf91a56a5e376edb3073": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 18:08:08.153: INFO: At 2022-12-22 17:53:33 +0000 UTC - event for inline-volume-tester-6dz24: {kubelet i-0243def68c064e8ef} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "7351070c5819ecb58fdea221ba01c41ca4b695a778e4313ec133884c861ec185": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 18:08:08.153: INFO: At 2022-12-22 17:53:48 +0000 UTC - event for inline-volume-tester-6dz24: {kubelet i-0243def68c064e8ef} FailedCreatePodSandBox: (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "0427f4b64b47008f7f09f1bfd0e35a7a6fc5fa9f80f82b5d80b49e0afb08a164": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 18:08:08.217: INFO: POD NODE PHASE GRACE CONDITIONS Dec 22 18:08:08.217: INFO: inline-volume-tester-6dz24 i-0243def68c064e8ef Failed 30s [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2022-12-22 17:51:48 +0000 UTC } {Ready False 0001-01-01 00:00:00 +0000 UTC 2022-12-22 17:58:34 +0000 UTC PodFailed } {ContainersReady False 0001-01-01 00:00:00 +0000 UTC 2022-12-22 17:58:34 +0000 UTC PodFailed } {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2022-12-22 17:51:48 +0000 UTC }] Dec 22 18:08:08.217: INFO: Dec 22 18:08:08.349: INFO: Logging node info for node i-0243def68c064e8ef Dec 22 18:08:08.413: INFO: Node Info: &Node{ObjectMeta:{i-0243def68c064e8ef f07b8771-29eb-4944-a385-3482a3d3a315 54749 0 2022-12-22 17:17:47 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:us-west-2 failure-domain.beta.kubernetes.io/zone:us-west-2a kubernetes.io/arch:amd64 kubernetes.io/hostname:i-0243def68c064e8ef kubernetes.io/os:linux node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:us-west-2a topology.hostpath.csi/node:i-0243def68c064e8ef topology.kubernetes.io/region:us-west-2 topology.kubernetes.io/zone:us-west-2a] map[alpha.kubernetes.io/provided-node-ip:172.20.35.142 csi.volume.kubernetes.io/nodeid:{"csi-hostpath-ephemeral-6886":"i-0243def68c064e8ef","csi-hostpath-ephemeral-8407":"i-0243def68c064e8ef","ebs.csi.aws.com":"i-0243def68c064e8ef"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{aws-cloud-controller-manager Update v1 2022-12-22 17:17:47 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2022-12-22 17:17:47 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kops-controller Update v1 2022-12-22 17:17:47 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{}}}} } {kube-controller-manager Update v1 2022-12-22 17:17:47 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.3.0/24\"":{}}}} } {kubelet Update v1 2022-12-22 17:17:47 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:alpha.kubernetes.io/provided-node-ip":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {kube-controller-manager Update v1 2022-12-22 17:59:50 +0000 UTC FieldsV1 {"f:status":{"f:volumesAttached":{}}} status} {kubelet Update v1 2022-12-22 18:04:55 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.3.0/24,DoNotUseExternalID:,ProviderID:aws:///us-west-2a/i-0243def68c064e8ef,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.3.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{51527004160 0} {<nil>} 50319340Ki BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4050636800 0} {<nil>} 3955700Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{46374303668 0} {<nil>} 46374303668 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3945779200 0} {<nil>} 3853300Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-12-22 18:04:55 +0000 UTC,LastTransitionTime:2022-12-22 17:17:37 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-12-22 18:04:55 +0000 UTC,LastTransitionTime:2022-12-22 17:17:37 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-12-22 18:04:55 +0000 UTC,LastTransitionTime:2022-12-22 17:17:37 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-12-22 18:04:55 +0000 UTC,LastTransitionTime:2022-12-22 17:17:59 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.35.142,},NodeAddress{Type:ExternalIP,Address:35.90.235.118,},NodeAddress{Type:InternalDNS,Address:i-0243def68c064e8ef.us-west-2.compute.internal,},NodeAddress{Type:Hostname,Address:i-0243def68c064e8ef.us-west-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-35-90-235-118.us-west-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2c8ae28bb7834ee5f6b4c4ef144a8e,SystemUUID:ec2c8ae2-8bb7-834e-e5f6-b4c4ef144a8e,BootID:5843895c-91be-4c36-b700-9e551cb745de,KernelVersion:5.10.157-139.675.amzn2.x86_64,OSImage:Amazon Linux 2,ContainerRuntimeVersion:containerd://1.6.14,KubeletVersion:v1.26.0,KubeProxyVersion:v1.26.0,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:166719855,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/jessie-dnsutils@sha256:24aaf2626d6b27864c29de2097e8bbb840b3a414271bf7c8995e431e47d8408e registry.k8s.io/e2e-test-images/jessie-dnsutils:1.7],SizeBytes:112030336,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:dd75ec974b0a2a6f6bb47001ba09207976e625db898d1b16735528c009cb171c registry.k8s.io/etcd:3.5.6-0],SizeBytes:102542580,},ContainerImage{Names:[registry.k8s.io/kube-proxy-amd64:v1.26.0],SizeBytes:67205320,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:16bbf38c463a4223d8cfe4da12bc61010b082a79b4bb003e2d3ba3ece5dd5f9e registry.k8s.io/e2e-test-images/agnhost:2.43],SizeBytes:51706353,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nautilus@sha256:80ba6c8c44f9623f06e868a1aa66026c8ec438ad814f9ec95e9333b415fe3550 registry.k8s.io/e2e-test-images/nautilus:1.7],SizeBytes:49641698,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:3fe7acf013d1264ffded116b80a73dc129a449b0fccdb8d21af8279f2233f36e registry.k8s.io/e2e-test-images/httpd:2.4.39-4],SizeBytes:41901587,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:148b022f5c5da426fc2f3c14b5c0867e58ef05961510c84749ac1fddcb0fef22 registry.k8s.io/e2e-test-images/httpd:2.4.38-4],SizeBytes:40764257,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:10e231cf07c82dfab5141c1afe548dc734e5fc1f67665ec3982a325d1bd31a9a registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.0],SizeBytes:29725756,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:ee3b525d5b89db99da3b8eb521d9cd90cb6e9ef0fbb651e98bb37be78d36b5b8 registry.k8s.io/sig-storage/csi-provisioner:v3.3.0],SizeBytes:25491225,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:425d8f1b769398127767b06ed97ce62578a3179bcb99809ce93a1649e025ffe7 registry.k8s.io/sig-storage/csi-resizer:v1.6.0],SizeBytes:24148884,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:291334908ddf71a4661fd7f6d9d97274de8a5378a2b6fdfeb2ce73414a34f82f registry.k8s.io/sig-storage/csi-snapshotter:v6.1.0],SizeBytes:23881995,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:9a685020911e2725ad019dbce6e4a5ab93d51e3d4557f115e64343345e05781b registry.k8s.io/sig-storage/csi-attacher:v4.0.0],SizeBytes:23847201,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:92257881c1d6493cf18299a24af42330f891166560047902b8d431fb66b01af5 registry.k8s.io/sig-storage/hostpathplugin:v1.9.0],SizeBytes:18758628,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nonroot@sha256:ee9f50b3c64b174d296d91ca9f69a914ac30e59095dfb462b2b518ad28a63655 registry.k8s.io/e2e-test-images/nonroot:1.4],SizeBytes:17747885,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:933940f13b3ea0abc62e656c1aa5c5b47c04b15d71250413a6b821bd0c58b94e registry.k8s.io/sig-storage/livenessprobe:v2.7.0],SizeBytes:8688564,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:5c99cf6a02adda929b10321dbf4ecfa00d87be9ba4fb456006237d530ab4baa1 registry.k8s.io/e2e-test-images/nginx:1.14-4],SizeBytes:6978614,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nonewprivs@sha256:8ac1264691820febacf3aea5d152cbde6d10685731ec14966a9401c6f47a68ac registry.k8s.io/e2e-test-images/nonewprivs:1.3],SizeBytes:3263463,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf registry.k8s.io/e2e-test-images/busybox:1.29-2],SizeBytes:732424,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:2e0f836850e09b8b7cc937681d6194537a09fbd5f6b9e08f4d646a85128e8937 registry.k8s.io/e2e-test-images/busybox:1.29-4],SizeBytes:731990,},ContainerImage{Names:[registry.k8s.io/pause@sha256:7031c1b283388d2c2e09b57badb803c05ebed362dc88d84b480cc47f72a21097 registry.k8s.io/pause:3.9],SizeBytes:321520,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[kubernetes.io/csi/csi-hostpath-ephemeral-6886^4e266409-8221-11ed-800d-8a8e987c6028 kubernetes.io/csi/csi-hostpath-ephemeral-8407^9fd9423f-821f-11ed-b5cc-02642e84c89b],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/csi-hostpath-ephemeral-8407^9fd9423f-821f-11ed-b5cc-02642e84c89b,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/csi-hostpath-ephemeral-6886^4e266409-8221-11ed-800d-8a8e987c6028,DevicePath:,},},Config:nil,},} Dec 22 18:08:08.413: INFO: Logging kubelet events for node i-0243def68c064e8ef ... skipping 133 lines ... Dec 22 18:08:10.574: INFO: Container csi-volume-tester ready: false, restart count 0 Dec 22 18:08:10.574: INFO: coredns-559769c974-wvwxv started at 2022-12-22 17:19:14 +0000 UTC (0+1 container statuses recorded) Dec 22 18:08:10.574: INFO: Container coredns ready: false, restart count 9 Dec 22 18:08:10.830: INFO: Latency metrics for node i-0ea878d3671f2eda2 [1mSTEP:[0m Waiting for namespaces [ephemeral-6886] to vanish [38;5;243m12/22/22 18:08:10.895[0m Dec 22 18:13:11.024: INFO: error deleting namespace ephemeral-6886: timed out waiting for the condition [1mSTEP:[0m uninstalling csi csi-hostpath driver [38;5;243m12/22/22 18:13:11.024[0m Dec 22 18:13:11.024: INFO: deleting *v1.ServiceAccount: ephemeral-6886-8022/csi-attacher Dec 22 18:13:11.089: INFO: deleting *v1.ClusterRole: external-attacher-runner-ephemeral-6886 Dec 22 18:13:11.157: INFO: deleting *v1.ClusterRoleBinding: csi-attacher-role-ephemeral-6886 Dec 22 18:13:11.222: INFO: deleting *v1.Role: ephemeral-6886-8022/external-attacher-cfg-ephemeral-6886 Dec 22 18:13:11.287: INFO: deleting *v1.RoleBinding: ephemeral-6886-8022/csi-attacher-role-cfg ... skipping 33 lines ... Dec 22 18:13:13.500: INFO: deleting *v1.ClusterRoleBinding: psp-csi-hostpath-role-ephemeral-6886 [1mSTEP:[0m deleting the driver namespace: ephemeral-6886-8022 [38;5;243m12/22/22 18:13:13.565[0m [1mSTEP:[0m Collecting events from namespace "ephemeral-6886-8022". [38;5;243m12/22/22 18:13:13.566[0m [1mSTEP:[0m Found 27 events. [38;5;243m12/22/22 18:13:13.63[0m Dec 22 18:13:13.630: INFO: At 2022-12-22 17:51:22 +0000 UTC - event for csi-hostpathplugin: {statefulset-controller } SuccessfulCreate: create Pod csi-hostpathplugin-0 in StatefulSet csi-hostpathplugin successful Dec 22 18:13:13.630: INFO: At 2022-12-22 17:51:22 +0000 UTC - event for csi-hostpathplugin-0: {default-scheduler } Scheduled: Successfully assigned ephemeral-6886-8022/csi-hostpathplugin-0 to i-0243def68c064e8ef Dec 22 18:13:13.630: INFO: At 2022-12-22 17:51:23 +0000 UTC - event for csi-hostpathplugin-0: {kubelet i-0243def68c064e8ef} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "85770d703340c83f1b05f9de28ab8f70516c9df219b32991df93db63f432e656": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 18:13:13.630: INFO: At 2022-12-22 17:51:34 +0000 UTC - event for csi-hostpathplugin-0: {kubelet i-0243def68c064e8ef} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "cbbf270c8d2d8f5aea7b99ee21ae0a0092bcebac05460179a72c5c5212b72ee0": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 18:13:13.630: INFO: At 2022-12-22 17:51:45 +0000 UTC - event for csi-hostpathplugin-0: {kubelet i-0243def68c064e8ef} Pulled: Container image "registry.k8s.io/sig-storage/hostpathplugin:v1.9.0" already present on machine Dec 22 18:13:13.630: INFO: At 2022-12-22 17:51:45 +0000 UTC - event for csi-hostpathplugin-0: {kubelet i-0243def68c064e8ef} Created: Created container hostpath Dec 22 18:13:13.630: INFO: At 2022-12-22 17:51:45 +0000 UTC - event for csi-hostpathplugin-0: {kubelet i-0243def68c064e8ef} Started: Started container hostpath Dec 22 18:13:13.630: INFO: At 2022-12-22 17:51:45 +0000 UTC - event for csi-hostpathplugin-0: {kubelet i-0243def68c064e8ef} Pulled: Container image "registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1" already present on machine Dec 22 18:13:13.630: INFO: At 2022-12-22 17:51:45 +0000 UTC - event for csi-hostpathplugin-0: {kubelet i-0243def68c064e8ef} Created: Created container node-driver-registrar Dec 22 18:13:13.630: INFO: At 2022-12-22 17:51:45 +0000 UTC - event for csi-hostpathplugin-0: {kubelet i-0243def68c064e8ef} Started: Started container node-driver-registrar ... skipping 9 lines ... Dec 22 18:13:13.630: INFO: At 2022-12-22 17:51:46 +0000 UTC - event for csi-hostpathplugin-0: {kubelet i-0243def68c064e8ef} Created: Created container csi-attacher Dec 22 18:13:13.630: INFO: At 2022-12-22 17:51:46 +0000 UTC - event for csi-hostpathplugin-0: {kubelet i-0243def68c064e8ef} Started: Started container csi-resizer Dec 22 18:13:13.630: INFO: At 2022-12-22 17:51:46 +0000 UTC - event for csi-hostpathplugin-0: {kubelet i-0243def68c064e8ef} Pulled: Container image "registry.k8s.io/sig-storage/csi-snapshotter:v6.1.0" already present on machine Dec 22 18:13:13.630: INFO: At 2022-12-22 17:51:46 +0000 UTC - event for csi-hostpathplugin-0: {kubelet i-0243def68c064e8ef} Created: Created container csi-snapshotter Dec 22 18:13:13.630: INFO: At 2022-12-22 17:51:46 +0000 UTC - event for csi-hostpathplugin-0: {kubelet i-0243def68c064e8ef} Started: Started container csi-snapshotter Dec 22 18:13:13.630: INFO: At 2022-12-22 17:51:46 +0000 UTC - event for csi-hostpathplugin-0: {kubelet i-0243def68c064e8ef} Started: Started container liveness-probe Dec 22 18:13:13.630: INFO: At 2022-12-22 17:51:59 +0000 UTC - event for csi-hostpathplugin-0: {kubelet i-0243def68c064e8ef} Unhealthy: Liveness probe failed: Get "http://172.20.43.56:9898/healthz": context deadline exceeded (Client.Timeout exceeded while awaiting headers) Dec 22 18:13:13.630: INFO: At 2022-12-22 17:53:56 +0000 UTC - event for csi-hostpathplugin-0: {kubelet i-0243def68c064e8ef} BackOff: Back-off restarting failed container hostpath in pod csi-hostpathplugin-0_ephemeral-6886-8022(8630e2d8-74fe-40cf-bbde-70c29c9af3dc) Dec 22 18:13:13.694: INFO: POD NODE PHASE GRACE CONDITIONS Dec 22 18:13:13.694: INFO: csi-hostpathplugin-0 i-0243def68c064e8ef Running 30s [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2022-12-22 17:51:22 +0000 UTC } {Ready False 0001-01-01 00:00:00 +0000 UTC 2022-12-22 18:12:06 +0000 UTC ContainersNotReady containers with unready status: [hostpath]} {ContainersReady False 0001-01-01 00:00:00 +0000 UTC 2022-12-22 18:12:06 +0000 UTC ContainersNotReady containers with unready status: [hostpath]} {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2022-12-22 17:51:22 +0000 UTC }] Dec 22 18:13:13.694: INFO: Dec 22 18:13:14.242: INFO: Logging node info for node i-0243def68c064e8ef Dec 22 18:13:14.305: INFO: Node Info: &Node{ObjectMeta:{i-0243def68c064e8ef f07b8771-29eb-4944-a385-3482a3d3a315 56825 0 2022-12-22 17:17:47 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:us-west-2 failure-domain.beta.kubernetes.io/zone:us-west-2a kubernetes.io/arch:amd64 kubernetes.io/hostname:i-0243def68c064e8ef kubernetes.io/os:linux node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:us-west-2a topology.hostpath.csi/node:i-0243def68c064e8ef topology.kubernetes.io/region:us-west-2 topology.kubernetes.io/zone:us-west-2a] map[alpha.kubernetes.io/provided-node-ip:172.20.35.142 csi.volume.kubernetes.io/nodeid:{"csi-hostpath-ephemeral-8407":"i-0243def68c064e8ef","ebs.csi.aws.com":"i-0243def68c064e8ef"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{aws-cloud-controller-manager Update v1 2022-12-22 17:17:47 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2022-12-22 17:17:47 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kops-controller Update v1 2022-12-22 17:17:47 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{}}}} } {kube-controller-manager Update v1 2022-12-22 17:17:47 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.3.0/24\"":{}}}} } {kubelet Update v1 2022-12-22 17:17:47 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:alpha.kubernetes.io/provided-node-ip":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {kube-controller-manager Update v1 2022-12-22 17:59:50 +0000 UTC FieldsV1 {"f:status":{"f:volumesAttached":{}}} status} {kubelet Update v1 2022-12-22 18:13:13 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.3.0/24,DoNotUseExternalID:,ProviderID:aws:///us-west-2a/i-0243def68c064e8ef,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.3.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{51527004160 0} {<nil>} 50319340Ki BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4050636800 0} {<nil>} 3955700Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{46374303668 0} {<nil>} 46374303668 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3945779200 0} {<nil>} 3853300Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-12-22 18:10:01 +0000 UTC,LastTransitionTime:2022-12-22 17:17:37 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-12-22 18:10:01 +0000 UTC,LastTransitionTime:2022-12-22 17:17:37 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-12-22 18:10:01 +0000 UTC,LastTransitionTime:2022-12-22 17:17:37 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-12-22 18:10:01 +0000 UTC,LastTransitionTime:2022-12-22 17:17:59 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.35.142,},NodeAddress{Type:ExternalIP,Address:35.90.235.118,},NodeAddress{Type:InternalDNS,Address:i-0243def68c064e8ef.us-west-2.compute.internal,},NodeAddress{Type:Hostname,Address:i-0243def68c064e8ef.us-west-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-35-90-235-118.us-west-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2c8ae28bb7834ee5f6b4c4ef144a8e,SystemUUID:ec2c8ae2-8bb7-834e-e5f6-b4c4ef144a8e,BootID:5843895c-91be-4c36-b700-9e551cb745de,KernelVersion:5.10.157-139.675.amzn2.x86_64,OSImage:Amazon Linux 2,ContainerRuntimeVersion:containerd://1.6.14,KubeletVersion:v1.26.0,KubeProxyVersion:v1.26.0,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:166719855,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/jessie-dnsutils@sha256:24aaf2626d6b27864c29de2097e8bbb840b3a414271bf7c8995e431e47d8408e registry.k8s.io/e2e-test-images/jessie-dnsutils:1.7],SizeBytes:112030336,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:dd75ec974b0a2a6f6bb47001ba09207976e625db898d1b16735528c009cb171c registry.k8s.io/etcd:3.5.6-0],SizeBytes:102542580,},ContainerImage{Names:[registry.k8s.io/kube-proxy-amd64:v1.26.0],SizeBytes:67205320,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:16bbf38c463a4223d8cfe4da12bc61010b082a79b4bb003e2d3ba3ece5dd5f9e registry.k8s.io/e2e-test-images/agnhost:2.43],SizeBytes:51706353,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nautilus@sha256:80ba6c8c44f9623f06e868a1aa66026c8ec438ad814f9ec95e9333b415fe3550 registry.k8s.io/e2e-test-images/nautilus:1.7],SizeBytes:49641698,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:3fe7acf013d1264ffded116b80a73dc129a449b0fccdb8d21af8279f2233f36e registry.k8s.io/e2e-test-images/httpd:2.4.39-4],SizeBytes:41901587,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:148b022f5c5da426fc2f3c14b5c0867e58ef05961510c84749ac1fddcb0fef22 registry.k8s.io/e2e-test-images/httpd:2.4.38-4],SizeBytes:40764257,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:10e231cf07c82dfab5141c1afe548dc734e5fc1f67665ec3982a325d1bd31a9a registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.0],SizeBytes:29725756,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:ee3b525d5b89db99da3b8eb521d9cd90cb6e9ef0fbb651e98bb37be78d36b5b8 registry.k8s.io/sig-storage/csi-provisioner:v3.3.0],SizeBytes:25491225,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:425d8f1b769398127767b06ed97ce62578a3179bcb99809ce93a1649e025ffe7 registry.k8s.io/sig-storage/csi-resizer:v1.6.0],SizeBytes:24148884,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:291334908ddf71a4661fd7f6d9d97274de8a5378a2b6fdfeb2ce73414a34f82f registry.k8s.io/sig-storage/csi-snapshotter:v6.1.0],SizeBytes:23881995,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:9a685020911e2725ad019dbce6e4a5ab93d51e3d4557f115e64343345e05781b registry.k8s.io/sig-storage/csi-attacher:v4.0.0],SizeBytes:23847201,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:92257881c1d6493cf18299a24af42330f891166560047902b8d431fb66b01af5 registry.k8s.io/sig-storage/hostpathplugin:v1.9.0],SizeBytes:18758628,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nonroot@sha256:ee9f50b3c64b174d296d91ca9f69a914ac30e59095dfb462b2b518ad28a63655 registry.k8s.io/e2e-test-images/nonroot:1.4],SizeBytes:17747885,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:933940f13b3ea0abc62e656c1aa5c5b47c04b15d71250413a6b821bd0c58b94e registry.k8s.io/sig-storage/livenessprobe:v2.7.0],SizeBytes:8688564,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:5c99cf6a02adda929b10321dbf4ecfa00d87be9ba4fb456006237d530ab4baa1 registry.k8s.io/e2e-test-images/nginx:1.14-4],SizeBytes:6978614,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nonewprivs@sha256:8ac1264691820febacf3aea5d152cbde6d10685731ec14966a9401c6f47a68ac registry.k8s.io/e2e-test-images/nonewprivs:1.3],SizeBytes:3263463,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf registry.k8s.io/e2e-test-images/busybox:1.29-2],SizeBytes:732424,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:2e0f836850e09b8b7cc937681d6194537a09fbd5f6b9e08f4d646a85128e8937 registry.k8s.io/e2e-test-images/busybox:1.29-4],SizeBytes:731990,},ContainerImage{Names:[registry.k8s.io/pause@sha256:7031c1b283388d2c2e09b57badb803c05ebed362dc88d84b480cc47f72a21097 registry.k8s.io/pause:3.9],SizeBytes:321520,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[kubernetes.io/csi/csi-hostpath-ephemeral-6886^4e266409-8221-11ed-800d-8a8e987c6028 kubernetes.io/csi/csi-hostpath-ephemeral-8407^9fd9423f-821f-11ed-b5cc-02642e84c89b],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/csi-hostpath-ephemeral-8407^9fd9423f-821f-11ed-b5cc-02642e84c89b,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/csi-hostpath-ephemeral-6886^4e266409-8221-11ed-800d-8a8e987c6028,DevicePath:,},},Config:nil,},} ... skipping 145 lines ... [DeferCleanup (Each)] [Testpattern: Generic Ephemeral-volume (default fs) (immediate-binding)] ephemeral dump namespaces | framework.go:196 [1mSTEP:[0m dump namespace information after failure [38;5;243m12/22/22 18:13:22.859[0m [1mSTEP:[0m Collecting events from namespace "ephemeral-6886". [38;5;243m12/22/22 18:13:22.859[0m [1mSTEP:[0m Found 0 events. [38;5;243m12/22/22 18:13:22.923[0m Dec 22 18:13:22.987: INFO: POD NODE PHASE GRACE CONDITIONS Dec 22 18:13:22.987: INFO: inline-volume-tester-6dz24 i-0243def68c064e8ef Failed 30s [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2022-12-22 17:51:48 +0000 UTC } {Ready False 0001-01-01 00:00:00 +0000 UTC 2022-12-22 17:58:34 +0000 UTC PodFailed } {ContainersReady False 0001-01-01 00:00:00 +0000 UTC 2022-12-22 17:58:34 +0000 UTC PodFailed } {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2022-12-22 17:51:48 +0000 UTC }] Dec 22 18:13:22.987: INFO: Dec 22 18:13:23.117: INFO: Logging node info for node i-0243def68c064e8ef Dec 22 18:13:23.181: INFO: Node Info: &Node{ObjectMeta:{i-0243def68c064e8ef f07b8771-29eb-4944-a385-3482a3d3a315 56825 0 2022-12-22 17:17:47 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:us-west-2 failure-domain.beta.kubernetes.io/zone:us-west-2a kubernetes.io/arch:amd64 kubernetes.io/hostname:i-0243def68c064e8ef kubernetes.io/os:linux node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:us-west-2a topology.hostpath.csi/node:i-0243def68c064e8ef topology.kubernetes.io/region:us-west-2 topology.kubernetes.io/zone:us-west-2a] map[alpha.kubernetes.io/provided-node-ip:172.20.35.142 csi.volume.kubernetes.io/nodeid:{"csi-hostpath-ephemeral-8407":"i-0243def68c064e8ef","ebs.csi.aws.com":"i-0243def68c064e8ef"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{aws-cloud-controller-manager Update v1 2022-12-22 17:17:47 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2022-12-22 17:17:47 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kops-controller Update v1 2022-12-22 17:17:47 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{}}}} } {kube-controller-manager Update v1 2022-12-22 17:17:47 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.3.0/24\"":{}}}} } {kubelet Update v1 2022-12-22 17:17:47 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:alpha.kubernetes.io/provided-node-ip":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {kube-controller-manager Update v1 2022-12-22 17:59:50 +0000 UTC FieldsV1 {"f:status":{"f:volumesAttached":{}}} status} {kubelet Update v1 2022-12-22 18:13:13 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.3.0/24,DoNotUseExternalID:,ProviderID:aws:///us-west-2a/i-0243def68c064e8ef,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.3.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{51527004160 0} {<nil>} 50319340Ki BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4050636800 0} {<nil>} 3955700Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{46374303668 0} {<nil>} 46374303668 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3945779200 0} {<nil>} 3853300Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-12-22 18:10:01 +0000 UTC,LastTransitionTime:2022-12-22 17:17:37 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-12-22 18:10:01 +0000 UTC,LastTransitionTime:2022-12-22 17:17:37 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-12-22 18:10:01 +0000 UTC,LastTransitionTime:2022-12-22 17:17:37 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-12-22 18:10:01 +0000 UTC,LastTransitionTime:2022-12-22 17:17:59 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.35.142,},NodeAddress{Type:ExternalIP,Address:35.90.235.118,},NodeAddress{Type:InternalDNS,Address:i-0243def68c064e8ef.us-west-2.compute.internal,},NodeAddress{Type:Hostname,Address:i-0243def68c064e8ef.us-west-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-35-90-235-118.us-west-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2c8ae28bb7834ee5f6b4c4ef144a8e,SystemUUID:ec2c8ae2-8bb7-834e-e5f6-b4c4ef144a8e,BootID:5843895c-91be-4c36-b700-9e551cb745de,KernelVersion:5.10.157-139.675.amzn2.x86_64,OSImage:Amazon Linux 2,ContainerRuntimeVersion:containerd://1.6.14,KubeletVersion:v1.26.0,KubeProxyVersion:v1.26.0,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:166719855,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/jessie-dnsutils@sha256:24aaf2626d6b27864c29de2097e8bbb840b3a414271bf7c8995e431e47d8408e registry.k8s.io/e2e-test-images/jessie-dnsutils:1.7],SizeBytes:112030336,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:dd75ec974b0a2a6f6bb47001ba09207976e625db898d1b16735528c009cb171c registry.k8s.io/etcd:3.5.6-0],SizeBytes:102542580,},ContainerImage{Names:[registry.k8s.io/kube-proxy-amd64:v1.26.0],SizeBytes:67205320,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:16bbf38c463a4223d8cfe4da12bc61010b082a79b4bb003e2d3ba3ece5dd5f9e registry.k8s.io/e2e-test-images/agnhost:2.43],SizeBytes:51706353,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nautilus@sha256:80ba6c8c44f9623f06e868a1aa66026c8ec438ad814f9ec95e9333b415fe3550 registry.k8s.io/e2e-test-images/nautilus:1.7],SizeBytes:49641698,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:3fe7acf013d1264ffded116b80a73dc129a449b0fccdb8d21af8279f2233f36e registry.k8s.io/e2e-test-images/httpd:2.4.39-4],SizeBytes:41901587,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:148b022f5c5da426fc2f3c14b5c0867e58ef05961510c84749ac1fddcb0fef22 registry.k8s.io/e2e-test-images/httpd:2.4.38-4],SizeBytes:40764257,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:10e231cf07c82dfab5141c1afe548dc734e5fc1f67665ec3982a325d1bd31a9a registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.0],SizeBytes:29725756,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:ee3b525d5b89db99da3b8eb521d9cd90cb6e9ef0fbb651e98bb37be78d36b5b8 registry.k8s.io/sig-storage/csi-provisioner:v3.3.0],SizeBytes:25491225,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:425d8f1b769398127767b06ed97ce62578a3179bcb99809ce93a1649e025ffe7 registry.k8s.io/sig-storage/csi-resizer:v1.6.0],SizeBytes:24148884,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:291334908ddf71a4661fd7f6d9d97274de8a5378a2b6fdfeb2ce73414a34f82f registry.k8s.io/sig-storage/csi-snapshotter:v6.1.0],SizeBytes:23881995,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:9a685020911e2725ad019dbce6e4a5ab93d51e3d4557f115e64343345e05781b registry.k8s.io/sig-storage/csi-attacher:v4.0.0],SizeBytes:23847201,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:92257881c1d6493cf18299a24af42330f891166560047902b8d431fb66b01af5 registry.k8s.io/sig-storage/hostpathplugin:v1.9.0],SizeBytes:18758628,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nonroot@sha256:ee9f50b3c64b174d296d91ca9f69a914ac30e59095dfb462b2b518ad28a63655 registry.k8s.io/e2e-test-images/nonroot:1.4],SizeBytes:17747885,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:933940f13b3ea0abc62e656c1aa5c5b47c04b15d71250413a6b821bd0c58b94e registry.k8s.io/sig-storage/livenessprobe:v2.7.0],SizeBytes:8688564,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:5c99cf6a02adda929b10321dbf4ecfa00d87be9ba4fb456006237d530ab4baa1 registry.k8s.io/e2e-test-images/nginx:1.14-4],SizeBytes:6978614,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nonewprivs@sha256:8ac1264691820febacf3aea5d152cbde6d10685731ec14966a9401c6f47a68ac registry.k8s.io/e2e-test-images/nonewprivs:1.3],SizeBytes:3263463,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf registry.k8s.io/e2e-test-images/busybox:1.29-2],SizeBytes:732424,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:2e0f836850e09b8b7cc937681d6194537a09fbd5f6b9e08f4d646a85128e8937 registry.k8s.io/e2e-test-images/busybox:1.29-4],SizeBytes:731990,},ContainerImage{Names:[registry.k8s.io/pause@sha256:7031c1b283388d2c2e09b57badb803c05ebed362dc88d84b480cc47f72a21097 registry.k8s.io/pause:3.9],SizeBytes:321520,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[kubernetes.io/csi/csi-hostpath-ephemeral-6886^4e266409-8221-11ed-800d-8a8e987c6028 kubernetes.io/csi/csi-hostpath-ephemeral-8407^9fd9423f-821f-11ed-b5cc-02642e84c89b],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/csi-hostpath-ephemeral-8407^9fd9423f-821f-11ed-b5cc-02642e84c89b,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/csi-hostpath-ephemeral-6886^4e266409-8221-11ed-800d-8a8e987c6028,DevicePath:,},},Config:nil,},} Dec 22 18:13:23.182: INFO: Logging kubelet events for node i-0243def68c064e8ef ... skipping 213 lines ... capacity: storage: 1Mi phase: Bound to be empty[0m [38;5;9mIn [1m[It][0m[38;5;9m at: [1mtest/e2e/storage/testsuites/ephemeral.go:431[0m [38;5;243m------------------------------[0m [38;5;9m• [FAILED] [1786.624 seconds][0m [sig-apps] StatefulSet [38;5;243mtest/e2e/apps/framework.go:23[0m [38;5;9m[1mBasic StatefulSet functionality [StatefulSetBasic] [AfterEach][0m [38;5;243mtest/e2e/apps/statefulset.go:124[0m should not deadlock when a pod's predecessor fails [38;5;243mtest/e2e/apps/statefulset.go:258[0m ... skipping 348 lines ... Dec 22 18:16:08.543: INFO: Still waiting for pvs of statefulset to disappear: pvc-5c98ea6d-0d6b-44dd-9393-ec970b741125: {Phase:Released Message: Reason:} pvc-893e0c3a-1f9a-41d0-810f-9fac144cd0d5: {Phase:Released Message: Reason:} Dec 22 18:16:08.608: INFO: Still waiting for pvs of statefulset to disappear: pvc-5c98ea6d-0d6b-44dd-9393-ec970b741125: {Phase:Released Message: Reason:} pvc-893e0c3a-1f9a-41d0-810f-9fac144cd0d5: {Phase:Released Message: Reason:} Dec 22 18:16:08.608: INFO: Unexpected error: <*errors.errorString | 0xc001301a80>: { s: "Timeout waiting for pv provisioner to delete pvs, this might mean the test leaked pvs.", } Dec 22 18:16:08.608: FAIL: Timeout waiting for pv provisioner to delete pvs, this might mean the test leaked pvs. Full Stack Trace k8s.io/kubernetes/test/e2e/framework/statefulset.DeleteAllStatefulSets({0x801e128, 0xc00312b1e0}, {0xc0031b5b80, 0x10}) test/e2e/framework/statefulset/rest.go:141 +0x8d4 k8s.io/kubernetes/test/e2e/apps.glob..func10.2.2() test/e2e/apps/statefulset.go:129 +0x1b2 ... skipping 12 lines ... Dec 22 18:16:08.742: INFO: At 2022-12-22 17:46:25 +0000 UTC - event for datadir-ss-0: {persistentvolume-controller } WaitForFirstConsumer: waiting for first consumer to be created before binding Dec 22 18:16:08.742: INFO: At 2022-12-22 17:46:25 +0000 UTC - event for ss: {statefulset-controller } SuccessfulCreate: create Pod ss-0 in StatefulSet ss successful Dec 22 18:16:08.742: INFO: At 2022-12-22 17:46:25 +0000 UTC - event for ss: {statefulset-controller } SuccessfulCreate: create Claim datadir-ss-0 Pod ss-0 in StatefulSet ss success Dec 22 18:16:08.742: INFO: At 2022-12-22 17:46:28 +0000 UTC - event for datadir-ss-0: {ebs.csi.aws.com_ebs-csi-controller-86d44d7948-t9g2t_b0d2d6cc-f7e5-4849-be1d-32c79c224f40 } ProvisioningSucceeded: Successfully provisioned volume pvc-893e0c3a-1f9a-41d0-810f-9fac144cd0d5 Dec 22 18:16:08.742: INFO: At 2022-12-22 17:46:29 +0000 UTC - event for ss-0: {default-scheduler } Scheduled: Successfully assigned statefulset-8007/ss-0 to i-0243def68c064e8ef Dec 22 18:16:08.742: INFO: At 2022-12-22 17:46:31 +0000 UTC - event for ss-0: {attachdetach-controller } SuccessfulAttachVolume: AttachVolume.Attach succeeded for volume "pvc-893e0c3a-1f9a-41d0-810f-9fac144cd0d5" Dec 22 18:16:08.742: INFO: At 2022-12-22 17:46:34 +0000 UTC - event for ss-0: {kubelet i-0243def68c064e8ef} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "e63da9049832a73725f590fd00e78a179c646082417989aca39a5ff29c58d508": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 18:16:08.742: INFO: At 2022-12-22 17:46:48 +0000 UTC - event for ss-0: {kubelet i-0243def68c064e8ef} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "e526fbc184d02770ec339091a0b5b0443baf6c38c1d7d61889f1ac78b136c2bc": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 18:16:08.742: INFO: At 2022-12-22 17:47:00 +0000 UTC - event for ss-0: {kubelet i-0243def68c064e8ef} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "36d699daabaec86f4294a67f09f6b8011817ec1d6542f082a2f37528bef79307": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 18:16:08.742: INFO: At 2022-12-22 17:47:15 +0000 UTC - event for ss-0: {kubelet i-0243def68c064e8ef} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "e3362aee85f1f79155b7ea3ca0fe0955e8b793931ff1f69d8107df19f445161b": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 18:16:08.742: INFO: At 2022-12-22 17:47:26 +0000 UTC - event for ss-0: {kubelet i-0243def68c064e8ef} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "738713fb5193b62b6817dbebdc854c9818a615cbfba2ea3047c4dae5ec57cdd6": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 18:16:08.742: INFO: At 2022-12-22 17:47:40 +0000 UTC - event for ss-0: {kubelet i-0243def68c064e8ef} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "51caa3d6e46d679a263f6318d69ed8e57cdb8adbbe6d11b5d857ac509487e7cc": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 18:16:08.742: INFO: At 2022-12-22 17:47:51 +0000 UTC - event for ss-0: {kubelet i-0243def68c064e8ef} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "353ac304fd2898b76bddcc676f3d2dc3cde7b52756b3df8088562cb77df7b48f": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 18:16:08.742: INFO: At 2022-12-22 17:48:03 +0000 UTC - event for ss-0: {kubelet i-0243def68c064e8ef} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "64a2830382c2a2da1987d315019a03d43b194b4627107baf52e999c93f21a645": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 18:16:08.742: INFO: At 2022-12-22 17:48:15 +0000 UTC - event for ss-0: {kubelet i-0243def68c064e8ef} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "9687bdbc5e5cf3e8942c6eb6df863e89ccafdb503125824185bbf78449844557": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 18:16:08.742: INFO: At 2022-12-22 17:48:30 +0000 UTC - event for ss-0: {kubelet i-0243def68c064e8ef} FailedCreatePodSandBox: (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "b376bf3af5825db371566eed9161b8489449f2f5d1374fd6ab8a793032d90362": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 18:16:08.742: INFO: At 2022-12-22 17:52:46 +0000 UTC - event for datadir-ss-1: {ebs.csi.aws.com_ebs-csi-controller-86d44d7948-t9g2t_b0d2d6cc-f7e5-4849-be1d-32c79c224f40 } Provisioning: External provisioner is provisioning volume for claim "statefulset-8007/datadir-ss-1" Dec 22 18:16:08.742: INFO: At 2022-12-22 17:52:46 +0000 UTC - event for datadir-ss-1: {persistentvolume-controller } ExternalProvisioning: waiting for a volume to be created, either by external provisioner "ebs.csi.aws.com" or manually created by system administrator Dec 22 18:16:08.742: INFO: At 2022-12-22 17:52:46 +0000 UTC - event for datadir-ss-1: {persistentvolume-controller } WaitForFirstConsumer: waiting for first consumer to be created before binding Dec 22 18:16:08.742: INFO: At 2022-12-22 17:52:46 +0000 UTC - event for ss: {statefulset-controller } SuccessfulCreate: create Pod ss-1 in StatefulSet ss successful Dec 22 18:16:08.742: INFO: At 2022-12-22 17:52:46 +0000 UTC - event for ss: {statefulset-controller } SuccessfulCreate: create Claim datadir-ss-1 Pod ss-1 in StatefulSet ss success Dec 22 18:16:08.742: INFO: At 2022-12-22 17:52:50 +0000 UTC - event for datadir-ss-1: {ebs.csi.aws.com_ebs-csi-controller-86d44d7948-t9g2t_b0d2d6cc-f7e5-4849-be1d-32c79c224f40 } ProvisioningSucceeded: Successfully provisioned volume pvc-5c98ea6d-0d6b-44dd-9393-ec970b741125 Dec 22 18:16:08.742: INFO: At 2022-12-22 17:52:50 +0000 UTC - event for ss-1: {default-scheduler } Scheduled: Successfully assigned statefulset-8007/ss-1 to i-0ea878d3671f2eda2 Dec 22 18:16:08.742: INFO: At 2022-12-22 17:52:53 +0000 UTC - event for ss-1: {attachdetach-controller } SuccessfulAttachVolume: AttachVolume.Attach succeeded for volume "pvc-5c98ea6d-0d6b-44dd-9393-ec970b741125" Dec 22 18:16:08.742: INFO: At 2022-12-22 17:53:00 +0000 UTC - event for ss-1: {kubelet i-0ea878d3671f2eda2} FailedMount: MountVolume.MountDevice failed for volume "pvc-5c98ea6d-0d6b-44dd-9393-ec970b741125" : rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial unix /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock: connect: connection refused" Dec 22 18:16:08.742: INFO: At 2022-12-22 17:54:53 +0000 UTC - event for ss-1: {kubelet i-0ea878d3671f2eda2} FailedMount: Unable to attach or mount volumes: unmounted volumes=[datadir], unattached volumes=[home kube-api-access-lvhkf datadir]: timed out waiting for the condition Dec 22 18:16:08.742: INFO: At 2022-12-22 17:59:13 +0000 UTC - event for ss-1: {kubelet i-0ea878d3671f2eda2} Started: Started container webserver Dec 22 18:16:08.742: INFO: At 2022-12-22 17:59:13 +0000 UTC - event for ss-1: {kubelet i-0ea878d3671f2eda2} Created: Created container webserver Dec 22 18:16:08.742: INFO: At 2022-12-22 17:59:13 +0000 UTC - event for ss-1: {kubelet i-0ea878d3671f2eda2} Pulled: Container image "registry.k8s.io/e2e-test-images/httpd:2.4.38-4" already present on machine Dec 22 18:16:08.742: INFO: At 2022-12-22 17:59:14 +0000 UTC - event for ss-1: {kubelet i-0ea878d3671f2eda2} Unhealthy: Readiness probe failed: Dec 22 18:16:08.742: INFO: At 2022-12-22 17:59:16 +0000 UTC - event for ss-0: {attachdetach-controller } FailedAttachVolume: Multi-Attach error for volume "pvc-893e0c3a-1f9a-41d0-810f-9fac144cd0d5" Volume is already exclusively attached to one node and can't be attached to another Dec 22 18:16:08.742: INFO: At 2022-12-22 17:59:16 +0000 UTC - event for ss-0: {default-scheduler } Scheduled: Successfully assigned statefulset-8007/ss-0 to i-076110ae730e268e3 Dec 22 18:16:08.742: INFO: At 2022-12-22 17:59:16 +0000 UTC - event for ss-0: {kubelet i-0243def68c064e8ef} Killing: Stopping container webserver Dec 22 18:16:08.742: INFO: At 2022-12-22 17:59:16 +0000 UTC - event for test: {endpoint-controller } FailedToUpdateEndpoint: Failed to update endpoint statefulset-8007/test: Operation cannot be fulfilled on endpoints "test": the object has been modified; please apply your changes to the latest version and try again Dec 22 18:16:08.742: INFO: At 2022-12-22 17:59:29 +0000 UTC - event for ss-0: {attachdetach-controller } SuccessfulAttachVolume: AttachVolume.Attach succeeded for volume "pvc-893e0c3a-1f9a-41d0-810f-9fac144cd0d5" Dec 22 18:16:08.742: INFO: At 2022-12-22 17:59:30 +0000 UTC - event for ss-0: {kubelet i-076110ae730e268e3} Pulled: Container image "registry.k8s.io/e2e-test-images/httpd:2.4.38-4" already present on machine Dec 22 18:16:08.742: INFO: At 2022-12-22 17:59:30 +0000 UTC - event for ss-0: {kubelet i-076110ae730e268e3} Created: Created container webserver Dec 22 18:16:08.742: INFO: At 2022-12-22 17:59:30 +0000 UTC - event for ss-0: {kubelet i-076110ae730e268e3} Started: Started container webserver Dec 22 18:16:08.742: INFO: At 2022-12-22 17:59:47 +0000 UTC - event for ss: {statefulset-controller } SuccessfulDelete: delete Pod ss-1 in StatefulSet ss successful Dec 22 18:16:08.742: INFO: At 2022-12-22 18:06:07 +0000 UTC - event for ss: {statefulset-controller } SuccessfulDelete: delete Pod ss-0 in StatefulSet ss successful ... skipping 142 lines ... [1mSTEP:[0m Destroying namespace "statefulset-8007" for this suite. [38;5;243m12/22/22 18:16:11.324[0m [38;5;243m<< End Captured GinkgoWriter Output[0m [38;5;9mDec 22 18:16:08.608: Timeout waiting for pv provisioner to delete pvs, this might mean the test leaked pvs.[0m [38;5;9mIn [1m[AfterEach][0m[38;5;9m at: [1mtest/e2e/framework/statefulset/rest.go:141[0m [38;5;243m------------------------------[0m [38;5;9m• [FAILED] [2263.854 seconds][0m [sig-storage] CSI Volumes [38;5;243mtest/e2e/storage/utils/framework.go:23[0m [Driver: csi-hostpath] [38;5;243mtest/e2e/storage/csi_volumes.go:40[0m [Testpattern: Generic Ephemeral-volume (default fs) (immediate-binding)] ephemeral [38;5;243mtest/e2e/storage/framework/testsuite.go:50[0m ... skipping 441 lines ... Dec 22 17:51:08.694: INFO: Pod "inline-volume-tester-wkvxc": Phase="Running", Reason="", readiness=true. Elapsed: 12m18.124190694s Dec 22 17:51:08.694: INFO: Pod "inline-volume-tester-wkvxc" satisfied condition "running" Dec 22 17:51:08.756: INFO: Running volume expansion checks inline-volume-tester-wkvxc [1mSTEP:[0m Expanding current pvc [38;5;243m12/22/22 17:51:08.817[0m Dec 22 17:51:08.817: INFO: currentPvcSize 1Mi, requested new size 1025Mi [1mSTEP:[0m Waiting for cloudprovider resize to finish [38;5;243m12/22/22 17:51:08.943[0m Dec 22 18:01:09.131: INFO: Unexpected error: While waiting for pvc resize to finish: <*errors.errorString | 0xc000e31e80>: { s: "error while waiting for controller resize to finish: timed out waiting for the condition", } Dec 22 18:01:09.131: FAIL: While waiting for pvc resize to finish: error while waiting for controller resize to finish: timed out waiting for the condition Full Stack Trace k8s.io/kubernetes/test/e2e/storage/testsuites.(*ephemeralTestSuite).DefineTests.func5.1(0xc004407680) test/e2e/storage/testsuites/ephemeral.go:264 +0x8c5 k8s.io/kubernetes/test/e2e/storage/testsuites.EphemeralTest.TestEphemeral({{0x801e128, 0xc00351da00}, 0xc000e1ff80, {0xc003819400, 0xe}, {0x0, 0x0}, 0xc0019325a0, {{0xc00018db60, 0x13}, ...}, ...}, ...) test/e2e/storage/testsuites/ephemeral.go:421 +0x755 ... skipping 76 lines ... Dec 22 18:11:13.449: INFO: At 2022-12-22 17:38:47 +0000 UTC - event for inline-volume-7hppq: {default-scheduler } FailedScheduling: 0/5 nodes are available: waiting for ephemeral volume controller to create the persistentvolumeclaim "inline-volume-7hppq-my-volume". preemption: 0/5 nodes are available: 5 No preemption victims found for incoming pod.. Dec 22 18:11:13.449: INFO: At 2022-12-22 17:38:47 +0000 UTC - event for inline-volume-7hppq-my-volume: {persistentvolume-controller } ProvisioningFailed: storageclass.storage.k8s.io "no-such-storage-class" not found Dec 22 18:11:13.449: INFO: At 2022-12-22 17:38:50 +0000 UTC - event for inline-volume-tester-wkvxc: {default-scheduler } FailedScheduling: 0/5 nodes are available: waiting for ephemeral volume controller to create the persistentvolumeclaim "inline-volume-tester-wkvxc-my-volume-0". preemption: 0/5 nodes are available: 5 No preemption victims found for incoming pod.. Dec 22 18:11:13.449: INFO: At 2022-12-22 17:38:50 +0000 UTC - event for inline-volume-tester-wkvxc-my-volume-0: {persistentvolume-controller } ExternalProvisioning: waiting for a volume to be created, either by external provisioner "csi-hostpath-ephemeral-8407" or manually created by system administrator Dec 22 18:11:13.449: INFO: At 2022-12-22 17:38:52 +0000 UTC - event for inline-volume-tester-wkvxc: {default-scheduler } FailedScheduling: 0/5 nodes are available: pod has unbound immediate PersistentVolumeClaims. preemption: 0/5 nodes are available: 5 No preemption victims found for incoming pod.. Dec 22 18:11:13.449: INFO: At 2022-12-22 17:39:42 +0000 UTC - event for inline-volume-tester-wkvxc-my-volume-0: {csi-hostpath-ephemeral-8407_csi-hostpathplugin-0_36456a46-524e-49d1-8ba2-a3d8a53d05be } Provisioning: External provisioner is provisioning volume for claim "ephemeral-8407/inline-volume-tester-wkvxc-my-volume-0" Dec 22 18:11:13.449: INFO: At 2022-12-22 17:39:42 +0000 UTC - event for inline-volume-tester-wkvxc-my-volume-0: {csi-hostpath-ephemeral-8407_csi-hostpathplugin-0_36456a46-524e-49d1-8ba2-a3d8a53d05be } ProvisioningFailed: failed to provision volume with StorageClass "ephemeral-84074q86s": error generating accessibility requirements: no available topology found Dec 22 18:11:13.449: INFO: At 2022-12-22 17:39:45 +0000 UTC - event for inline-volume-tester-wkvxc-my-volume-0: {csi-hostpath-ephemeral-8407_csi-hostpathplugin-0_36456a46-524e-49d1-8ba2-a3d8a53d05be } ProvisioningSucceeded: Successfully provisioned volume pvc-0399c67b-8772-4298-ab88-56f29af14cf3 Dec 22 18:11:13.449: INFO: At 2022-12-22 17:39:46 +0000 UTC - event for inline-volume-tester-wkvxc: {attachdetach-controller } SuccessfulAttachVolume: AttachVolume.Attach succeeded for volume "pvc-0399c67b-8772-4298-ab88-56f29af14cf3" Dec 22 18:11:13.449: INFO: At 2022-12-22 17:39:46 +0000 UTC - event for inline-volume-tester-wkvxc: {default-scheduler } Scheduled: Successfully assigned ephemeral-8407/inline-volume-tester-wkvxc to i-0243def68c064e8ef Dec 22 18:11:13.449: INFO: At 2022-12-22 17:39:57 +0000 UTC - event for inline-volume-tester-wkvxc: {kubelet i-0243def68c064e8ef} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "d137db52a6d920b5a80c60ae2229ca22dfec4ff0fa4285c43c43080ebaf36b63": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 18:11:13.449: INFO: At 2022-12-22 17:40:12 +0000 UTC - event for inline-volume-tester-wkvxc: {kubelet i-0243def68c064e8ef} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "9d3b3f158769b4b9a9e9eb275a6bbb1d9cfb495dfd7831abad13adfc747dbfc6": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 18:11:13.449: INFO: At 2022-12-22 17:40:23 +0000 UTC - event for inline-volume-tester-wkvxc: {kubelet i-0243def68c064e8ef} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "fe48d7ce3e9d2a2ae120810d67f8c135df87f61f421710c1bf50d1f65812f9a1": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 18:11:13.449: INFO: At 2022-12-22 17:40:34 +0000 UTC - event for inline-volume-tester-wkvxc: {kubelet i-0243def68c064e8ef} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "61bd7c3d0a357b9ca90024e59a06e748713f04926cea3d70c3244689e8c55249": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 18:11:13.449: INFO: At 2022-12-22 17:40:48 +0000 UTC - event for inline-volume-tester-wkvxc: {kubelet i-0243def68c064e8ef} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "095dd4618c2eb0ae5d782a5071441ed40916467b19cb0c6cd19cd4201ef70caa": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 18:11:13.449: INFO: At 2022-12-22 17:41:04 +0000 UTC - event for inline-volume-tester-wkvxc: {kubelet i-0243def68c064e8ef} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "eead9305504bc874f4b51174030a1f1105fed7aa735c88dc6d303ea25c532b62": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 18:11:13.449: INFO: At 2022-12-22 17:41:20 +0000 UTC - event for inline-volume-tester-wkvxc: {kubelet i-0243def68c064e8ef} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "0db000be1573619eeb1f41c43a0439bfd4b1e626b94eb00a0b5f68851a4c3014": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 18:11:13.449: INFO: At 2022-12-22 17:41:32 +0000 UTC - event for inline-volume-tester-wkvxc: {kubelet i-0243def68c064e8ef} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "51cdbf067231c64d05f7a8bd37bd9c699d8a481498e73688f0f5bd333220e8bf": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 18:11:13.449: INFO: At 2022-12-22 17:41:44 +0000 UTC - event for inline-volume-tester-wkvxc: {kubelet i-0243def68c064e8ef} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "b4ce79336928e763708baca2db0749717d8d8dcefcf8e96b637d2be31e446860": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 18:11:13.449: INFO: At 2022-12-22 17:41:57 +0000 UTC - event for inline-volume-tester-wkvxc: {kubelet i-0243def68c064e8ef} FailedCreatePodSandBox: (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "498a77337c9e5cd8028a90df627755fa64a92329d5c7368a7c0c606fa198bc1c": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 18:11:13.449: INFO: At 2022-12-22 17:51:08 +0000 UTC - event for inline-volume-tester-wkvxc-my-volume-0: {volume_expand } ExternalExpanding: Ignoring the PVC: didn't find a plugin capable of expanding the volume; waiting for an external controller to process this PVC. Dec 22 18:11:13.449: INFO: At 2022-12-22 17:51:24 +0000 UTC - event for inline-volume-tester-wkvxc-my-volume-0: {external-resizer csi-hostpath-ephemeral-8407 } Resizing: External resizer is resizing volume pvc-0399c67b-8772-4298-ab88-56f29af14cf3 Dec 22 18:11:13.449: INFO: At 2022-12-22 17:51:24 +0000 UTC - event for inline-volume-tester-wkvxc-my-volume-0: {external-resizer csi-hostpath-ephemeral-8407 } VolumeResizeFailed: resize volume "pvc-0399c67b-8772-4298-ab88-56f29af14cf3" by resizer "csi-hostpath-ephemeral-8407" failed: rpc error: code = NotFound desc = volume id 9fd9423f-821f-11ed-b5cc-02642e84c89b does not exist in the volumes list Dec 22 18:11:13.449: INFO: At 2022-12-22 17:54:40 +0000 UTC - event for inline-volume-tester-wkvxc-my-volume-0: {external-resizer csi-hostpath-ephemeral-8407 } VolumeResizeFailed: resize volume "pvc-0399c67b-8772-4298-ab88-56f29af14cf3" by resizer "csi-hostpath-ephemeral-8407" failed: rpc error: code = NotFound desc = volume id 9fd9423f-821f-11ed-b5cc-02642e84c89b does not exist in the volumes list Dec 22 18:11:13.449: INFO: At 2022-12-22 17:54:40 +0000 UTC - event for inline-volume-tester-wkvxc-my-volume-0: {external-resizer csi-hostpath-ephemeral-8407 } Resizing: External resizer is resizing volume pvc-0399c67b-8772-4298-ab88-56f29af14cf3 Dec 22 18:11:13.449: INFO: At 2022-12-22 18:00:09 +0000 UTC - event for inline-volume-tester-wkvxc-my-volume-0: {external-resizer csi-hostpath-ephemeral-8407 } VolumeResizeFailed: resize volume "pvc-0399c67b-8772-4298-ab88-56f29af14cf3" by resizer "csi-hostpath-ephemeral-8407" failed: rpc error: code = NotFound desc = volume id 9fd9423f-821f-11ed-b5cc-02642e84c89b does not exist in the volumes list Dec 22 18:11:13.449: INFO: At 2022-12-22 18:00:09 +0000 UTC - event for inline-volume-tester-wkvxc-my-volume-0: {external-resizer csi-hostpath-ephemeral-8407 } Resizing: External resizer is resizing volume pvc-0399c67b-8772-4298-ab88-56f29af14cf3 Dec 22 18:11:13.449: INFO: At 2022-12-22 18:01:09 +0000 UTC - event for inline-volume-tester-wkvxc: {kubelet i-0243def68c064e8ef} Killing: Stopping container csi-volume-tester Dec 22 18:11:13.449: INFO: At 2022-12-22 18:06:01 +0000 UTC - event for inline-volume-tester-wkvxc-my-volume-0: {external-resizer csi-hostpath-ephemeral-8407 } VolumeResizeFailed: resize volume "pvc-0399c67b-8772-4298-ab88-56f29af14cf3" by resizer "csi-hostpath-ephemeral-8407" failed: rpc error: code = NotFound desc = volume id 9fd9423f-821f-11ed-b5cc-02642e84c89b does not exist in the volumes list Dec 22 18:11:13.449: INFO: At 2022-12-22 18:06:01 +0000 UTC - event for inline-volume-tester-wkvxc-my-volume-0: {external-resizer csi-hostpath-ephemeral-8407 } Resizing: External resizer is resizing volume pvc-0399c67b-8772-4298-ab88-56f29af14cf3 Dec 22 18:11:13.510: INFO: POD NODE PHASE GRACE CONDITIONS Dec 22 18:11:13.510: INFO: inline-volume-tester-wkvxc i-0243def68c064e8ef Failed 30s [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2022-12-22 17:39:46 +0000 UTC } {Ready False 0001-01-01 00:00:00 +0000 UTC 2022-12-22 18:01:39 +0000 UTC PodFailed } {ContainersReady False 0001-01-01 00:00:00 +0000 UTC 2022-12-22 18:01:39 +0000 UTC PodFailed } {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2022-12-22 17:39:46 +0000 UTC }] Dec 22 18:11:13.510: INFO: Dec 22 18:11:13.643: INFO: Logging node info for node i-0243def68c064e8ef Dec 22 18:11:13.704: INFO: Node Info: &Node{ObjectMeta:{i-0243def68c064e8ef f07b8771-29eb-4944-a385-3482a3d3a315 56043 0 2022-12-22 17:17:47 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:us-west-2 failure-domain.beta.kubernetes.io/zone:us-west-2a kubernetes.io/arch:amd64 kubernetes.io/hostname:i-0243def68c064e8ef kubernetes.io/os:linux node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:us-west-2a topology.hostpath.csi/node:i-0243def68c064e8ef topology.kubernetes.io/region:us-west-2 topology.kubernetes.io/zone:us-west-2a] map[alpha.kubernetes.io/provided-node-ip:172.20.35.142 csi.volume.kubernetes.io/nodeid:{"csi-hostpath-ephemeral-6886":"i-0243def68c064e8ef","csi-hostpath-ephemeral-8407":"i-0243def68c064e8ef","ebs.csi.aws.com":"i-0243def68c064e8ef"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{aws-cloud-controller-manager Update v1 2022-12-22 17:17:47 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2022-12-22 17:17:47 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kops-controller Update v1 2022-12-22 17:17:47 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{}}}} } {kube-controller-manager Update v1 2022-12-22 17:17:47 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.3.0/24\"":{}}}} } {kubelet Update v1 2022-12-22 17:17:47 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:alpha.kubernetes.io/provided-node-ip":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {kube-controller-manager Update v1 2022-12-22 17:59:50 +0000 UTC FieldsV1 {"f:status":{"f:volumesAttached":{}}} status} {kubelet Update v1 2022-12-22 18:10:01 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.3.0/24,DoNotUseExternalID:,ProviderID:aws:///us-west-2a/i-0243def68c064e8ef,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.3.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{51527004160 0} {<nil>} 50319340Ki BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4050636800 0} {<nil>} 3955700Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{46374303668 0} {<nil>} 46374303668 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3945779200 0} {<nil>} 3853300Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-12-22 18:10:01 +0000 UTC,LastTransitionTime:2022-12-22 17:17:37 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-12-22 18:10:01 +0000 UTC,LastTransitionTime:2022-12-22 17:17:37 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-12-22 18:10:01 +0000 UTC,LastTransitionTime:2022-12-22 17:17:37 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-12-22 18:10:01 +0000 UTC,LastTransitionTime:2022-12-22 17:17:59 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.35.142,},NodeAddress{Type:ExternalIP,Address:35.90.235.118,},NodeAddress{Type:InternalDNS,Address:i-0243def68c064e8ef.us-west-2.compute.internal,},NodeAddress{Type:Hostname,Address:i-0243def68c064e8ef.us-west-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-35-90-235-118.us-west-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2c8ae28bb7834ee5f6b4c4ef144a8e,SystemUUID:ec2c8ae2-8bb7-834e-e5f6-b4c4ef144a8e,BootID:5843895c-91be-4c36-b700-9e551cb745de,KernelVersion:5.10.157-139.675.amzn2.x86_64,OSImage:Amazon Linux 2,ContainerRuntimeVersion:containerd://1.6.14,KubeletVersion:v1.26.0,KubeProxyVersion:v1.26.0,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:166719855,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/jessie-dnsutils@sha256:24aaf2626d6b27864c29de2097e8bbb840b3a414271bf7c8995e431e47d8408e registry.k8s.io/e2e-test-images/jessie-dnsutils:1.7],SizeBytes:112030336,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:dd75ec974b0a2a6f6bb47001ba09207976e625db898d1b16735528c009cb171c registry.k8s.io/etcd:3.5.6-0],SizeBytes:102542580,},ContainerImage{Names:[registry.k8s.io/kube-proxy-amd64:v1.26.0],SizeBytes:67205320,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:16bbf38c463a4223d8cfe4da12bc61010b082a79b4bb003e2d3ba3ece5dd5f9e registry.k8s.io/e2e-test-images/agnhost:2.43],SizeBytes:51706353,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nautilus@sha256:80ba6c8c44f9623f06e868a1aa66026c8ec438ad814f9ec95e9333b415fe3550 registry.k8s.io/e2e-test-images/nautilus:1.7],SizeBytes:49641698,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:3fe7acf013d1264ffded116b80a73dc129a449b0fccdb8d21af8279f2233f36e registry.k8s.io/e2e-test-images/httpd:2.4.39-4],SizeBytes:41901587,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:148b022f5c5da426fc2f3c14b5c0867e58ef05961510c84749ac1fddcb0fef22 registry.k8s.io/e2e-test-images/httpd:2.4.38-4],SizeBytes:40764257,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:10e231cf07c82dfab5141c1afe548dc734e5fc1f67665ec3982a325d1bd31a9a registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.0],SizeBytes:29725756,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:ee3b525d5b89db99da3b8eb521d9cd90cb6e9ef0fbb651e98bb37be78d36b5b8 registry.k8s.io/sig-storage/csi-provisioner:v3.3.0],SizeBytes:25491225,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:425d8f1b769398127767b06ed97ce62578a3179bcb99809ce93a1649e025ffe7 registry.k8s.io/sig-storage/csi-resizer:v1.6.0],SizeBytes:24148884,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:291334908ddf71a4661fd7f6d9d97274de8a5378a2b6fdfeb2ce73414a34f82f registry.k8s.io/sig-storage/csi-snapshotter:v6.1.0],SizeBytes:23881995,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:9a685020911e2725ad019dbce6e4a5ab93d51e3d4557f115e64343345e05781b registry.k8s.io/sig-storage/csi-attacher:v4.0.0],SizeBytes:23847201,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:92257881c1d6493cf18299a24af42330f891166560047902b8d431fb66b01af5 registry.k8s.io/sig-storage/hostpathplugin:v1.9.0],SizeBytes:18758628,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nonroot@sha256:ee9f50b3c64b174d296d91ca9f69a914ac30e59095dfb462b2b518ad28a63655 registry.k8s.io/e2e-test-images/nonroot:1.4],SizeBytes:17747885,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:933940f13b3ea0abc62e656c1aa5c5b47c04b15d71250413a6b821bd0c58b94e registry.k8s.io/sig-storage/livenessprobe:v2.7.0],SizeBytes:8688564,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:5c99cf6a02adda929b10321dbf4ecfa00d87be9ba4fb456006237d530ab4baa1 registry.k8s.io/e2e-test-images/nginx:1.14-4],SizeBytes:6978614,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nonewprivs@sha256:8ac1264691820febacf3aea5d152cbde6d10685731ec14966a9401c6f47a68ac registry.k8s.io/e2e-test-images/nonewprivs:1.3],SizeBytes:3263463,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf registry.k8s.io/e2e-test-images/busybox:1.29-2],SizeBytes:732424,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:2e0f836850e09b8b7cc937681d6194537a09fbd5f6b9e08f4d646a85128e8937 registry.k8s.io/e2e-test-images/busybox:1.29-4],SizeBytes:731990,},ContainerImage{Names:[registry.k8s.io/pause@sha256:7031c1b283388d2c2e09b57badb803c05ebed362dc88d84b480cc47f72a21097 registry.k8s.io/pause:3.9],SizeBytes:321520,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[kubernetes.io/csi/csi-hostpath-ephemeral-6886^4e266409-8221-11ed-800d-8a8e987c6028 kubernetes.io/csi/csi-hostpath-ephemeral-8407^9fd9423f-821f-11ed-b5cc-02642e84c89b],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/csi-hostpath-ephemeral-8407^9fd9423f-821f-11ed-b5cc-02642e84c89b,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/csi-hostpath-ephemeral-6886^4e266409-8221-11ed-800d-8a8e987c6028,DevicePath:,},},Config:nil,},} Dec 22 18:11:13.704: INFO: Logging kubelet events for node i-0243def68c064e8ef ... skipping 133 lines ... Dec 22 18:11:15.812: INFO: Container csi-volume-tester ready: false, restart count 0 Dec 22 18:11:15.812: INFO: coredns-559769c974-wvwxv started at 2022-12-22 17:19:14 +0000 UTC (0+1 container statuses recorded) Dec 22 18:11:15.812: INFO: Container coredns ready: false, restart count 9 Dec 22 18:11:16.077: INFO: Latency metrics for node i-0ea878d3671f2eda2 [1mSTEP:[0m Waiting for namespaces [ephemeral-8407] to vanish [38;5;243m12/22/22 18:11:16.141[0m Dec 22 18:16:16.263: INFO: error deleting namespace ephemeral-8407: timed out waiting for the condition [1mSTEP:[0m uninstalling csi csi-hostpath driver [38;5;243m12/22/22 18:16:16.263[0m Dec 22 18:16:16.263: INFO: deleting *v1.ServiceAccount: ephemeral-8407-4387/csi-attacher Dec 22 18:16:16.325: INFO: deleting *v1.ClusterRole: external-attacher-runner-ephemeral-8407 Dec 22 18:16:16.387: INFO: deleting *v1.ClusterRoleBinding: csi-attacher-role-ephemeral-8407 Dec 22 18:16:16.449: INFO: deleting *v1.Role: ephemeral-8407-4387/external-attacher-cfg-ephemeral-8407 Dec 22 18:16:16.511: INFO: deleting *v1.RoleBinding: ephemeral-8407-4387/csi-attacher-role-cfg ... skipping 33 lines ... Dec 22 18:16:18.643: INFO: deleting *v1.ClusterRoleBinding: psp-csi-hostpath-role-ephemeral-8407 [1mSTEP:[0m deleting the driver namespace: ephemeral-8407-4387 [38;5;243m12/22/22 18:16:18.704[0m [1mSTEP:[0m Collecting events from namespace "ephemeral-8407-4387". [38;5;243m12/22/22 18:16:18.705[0m [1mSTEP:[0m Found 29 events. [38;5;243m12/22/22 18:16:18.772[0m Dec 22 18:16:18.772: INFO: At 2022-12-22 17:38:50 +0000 UTC - event for csi-hostpathplugin: {statefulset-controller } SuccessfulCreate: create Pod csi-hostpathplugin-0 in StatefulSet csi-hostpathplugin successful Dec 22 18:16:18.772: INFO: At 2022-12-22 17:38:50 +0000 UTC - event for csi-hostpathplugin-0: {default-scheduler } Scheduled: Successfully assigned ephemeral-8407-4387/csi-hostpathplugin-0 to i-0243def68c064e8ef Dec 22 18:16:18.772: INFO: At 2022-12-22 17:38:51 +0000 UTC - event for csi-hostpathplugin-0: {kubelet i-0243def68c064e8ef} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "e1e156146f58d8497427c5cc364fcd55a9787aa2d417be2debc099062f3cf99e": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 18:16:18.772: INFO: At 2022-12-22 17:39:03 +0000 UTC - event for csi-hostpathplugin-0: {kubelet i-0243def68c064e8ef} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "68859c57e770bdd944bcb1dec58680e746b08c4d0b9c0dbe93014a84378800dc": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 18:16:18.772: INFO: At 2022-12-22 17:39:14 +0000 UTC - event for csi-hostpathplugin-0: {kubelet i-0243def68c064e8ef} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "ef0cf48e2d3637c9740c867eee8207ff59ea27cb426651f8a0caf0e9ce64518f": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 18:16:18.772: INFO: At 2022-12-22 17:39:27 +0000 UTC - event for csi-hostpathplugin-0: {kubelet i-0243def68c064e8ef} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "0f875cd7621af118f113b27fbcd56ca3fb991abda85333d2e494f1b9c7fea3c7": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Dec 22 18:16:18.772: INFO: At 2022-12-22 17:39:41 +0000 UTC - event for csi-hostpathplugin-0: {kubelet i-0243def68c064e8ef} Created: Created container hostpath Dec 22 18:16:18.772: INFO: At 2022-12-22 17:39:41 +0000 UTC - event for csi-hostpathplugin-0: {kubelet i-0243def68c064e8ef} Started: Started container hostpath Dec 22 18:16:18.772: INFO: At 2022-12-22 17:39:41 +0000 UTC - event for csi-hostpathplugin-0: {kubelet i-0243def68c064e8ef} Pulled: Container image "registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1" already present on machine Dec 22 18:16:18.772: INFO: At 2022-12-22 17:39:41 +0000 UTC - event for csi-hostpathplugin-0: {kubelet i-0243def68c064e8ef} Created: Created container node-driver-registrar Dec 22 18:16:18.772: INFO: At 2022-12-22 17:39:41 +0000 UTC - event for csi-hostpathplugin-0: {kubelet i-0243def68c064e8ef} Started: Started container node-driver-registrar Dec 22 18:16:18.772: INFO: At 2022-12-22 17:39:41 +0000 UTC - event for csi-hostpathplugin-0: {kubelet i-0243def68c064e8ef} Pulled: Container image "registry.k8s.io/sig-storage/livenessprobe:v2.7.0" already present on machine ... skipping 9 lines ... Dec 22 18:16:18.772: INFO: At 2022-12-22 17:39:42 +0000 UTC - event for csi-hostpathplugin-0: {kubelet i-0243def68c064e8ef} Created: Created container csi-attacher Dec 22 18:16:18.772: INFO: At 2022-12-22 17:39:42 +0000 UTC - event for csi-hostpathplugin-0: {kubelet i-0243def68c064e8ef} Started: Started container csi-resizer Dec 22 18:16:18.772: INFO: At 2022-12-22 17:39:42 +0000 UTC - event for csi-hostpathplugin-0: {kubelet i-0243def68c064e8ef} Pulled: Container image "registry.k8s.io/sig-storage/csi-snapshotter:v6.1.0" already present on machine Dec 22 18:16:18.772: INFO: At 2022-12-22 17:39:42 +0000 UTC - event for csi-hostpathplugin-0: {kubelet i-0243def68c064e8ef} Created: Created container csi-snapshotter Dec 22 18:16:18.772: INFO: At 2022-12-22 17:39:42 +0000 UTC - event for csi-hostpathplugin-0: {kubelet i-0243def68c064e8ef} Started: Started container csi-snapshotter Dec 22 18:16:18.772: INFO: At 2022-12-22 17:39:42 +0000 UTC - event for csi-hostpathplugin-0: {kubelet i-0243def68c064e8ef} Started: Started container liveness-probe Dec 22 18:16:18.772: INFO: At 2022-12-22 17:46:42 +0000 UTC - event for csi-hostpathplugin-0: {kubelet i-0243def68c064e8ef} Unhealthy: Liveness probe failed: Get "http://172.20.55.34:9898/healthz": context deadline exceeded (Client.Timeout exceeded while awaiting headers) Dec 22 18:16:18.772: INFO: At 2022-12-22 17:48:38 +0000 UTC - event for csi-hostpathplugin-0: {kubelet i-0243def68c064e8ef} BackOff: Back-off restarting failed container hostpath in pod csi-hostpathplugin-0_ephemeral-8407-4387(a53ec5b7-3b80-4a9e-af63-afe0c1986b17) Dec 22 18:16:18.833: INFO: POD NODE PHASE GRACE CONDITIONS Dec 22 18:16:18.833: INFO: csi-hostpathplugin-0 i-0243def68c064e8ef Running 30s [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2022-12-22 17:38:50 +0000 UTC } {Ready False 0001-01-01 00:00:00 +0000 UTC 2022-12-22 18:12:48 +0000 UTC ContainersNotReady containers with unready status: [hostpath]} {ContainersReady False 0001-01-01 00:00:00 +0000 UTC 2022-12-22 18:12:48 +0000 UTC ContainersNotReady containers with unready status: [hostpath]} {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2022-12-22 17:38:50 +0000 UTC }] Dec 22 18:16:18.833: INFO: Dec 22 18:16:18.833: INFO: csi-hostpathplugin-0[ephemeral-8407-4387].container[csi-resizer]=Lost connection to CSI driver, exiting Dec 22 18:16:19.352: INFO: Logging node info for node i-0243def68c064e8ef ... skipping 138 lines ... [DeferCleanup (Each)] [Testpattern: Generic Ephemeral-volume (default fs) (immediate-binding)] ephemeral dump namespaces | framework.go:196 [1mSTEP:[0m dump namespace information after failure [38;5;243m12/22/22 18:16:27.873[0m [1mSTEP:[0m Collecting events from namespace "ephemeral-8407". [38;5;243m12/22/22 18:16:27.874[0m [1mSTEP:[0m Found 0 events. [38;5;243m12/22/22 18:16:27.934[0m Dec 22 18:16:27.994: INFO: POD NODE PHASE GRACE CONDITIONS Dec 22 18:16:27.994: INFO: inline-volume-tester-wkvxc i-0243def68c064e8ef Failed 30s [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2022-12-22 17:39:46 +0000 UTC } {Ready False 0001-01-01 00:00:00 +0000 UTC 2022-12-22 18:01:39 +0000 UTC PodFailed } {ContainersReady False 0001-01-01 00:00:00 +0000 UTC 2022-12-22 18:01:39 +0000 UTC PodFailed } {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2022-12-22 17:39:46 +0000 UTC }] Dec 22 18:16:27.994: INFO: Dec 22 18:16:28.119: INFO: Logging node info for node i-0243def68c064e8ef Dec 22 18:16:28.180: INFO: Node Info: &Node{ObjectMeta:{i-0243def68c064e8ef f07b8771-29eb-4944-a385-3482a3d3a315 57578 0 2022-12-22 17:17:47 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:us-west-2 failure-domain.beta.kubernetes.io/zone:us-west-2a kubernetes.io/arch:amd64 kubernetes.io/hostname:i-0243def68c064e8ef kubernetes.io/os:linux node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:us-west-2a topology.hostpath.csi/node:i-0243def68c064e8ef topology.kubernetes.io/region:us-west-2 topology.kubernetes.io/zone:us-west-2a] map[alpha.kubernetes.io/provided-node-ip:172.20.35.142 csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-0243def68c064e8ef"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{aws-cloud-controller-manager Update v1 2022-12-22 17:17:47 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2022-12-22 17:17:47 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kops-controller Update v1 2022-12-22 17:17:47 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{}}}} } {kube-controller-manager Update v1 2022-12-22 17:17:47 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.3.0/24\"":{}}}} } {kubelet Update v1 2022-12-22 17:17:47 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:alpha.kubernetes.io/provided-node-ip":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {kube-controller-manager Update v1 2022-12-22 17:59:50 +0000 UTC FieldsV1 {"f:status":{"f:volumesAttached":{}}} status} {kubelet Update v1 2022-12-22 18:16:18 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.3.0/24,DoNotUseExternalID:,ProviderID:aws:///us-west-2a/i-0243def68c064e8ef,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.3.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{51527004160 0} {<nil>} 50319340Ki BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4050636800 0} {<nil>} 3955700Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{46374303668 0} {<nil>} 46374303668 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3945779200 0} {<nil>} 3853300Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-12-22 18:15:07 +0000 UTC,LastTransitionTime:2022-12-22 17:17:37 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-12-22 18:15:07 +0000 UTC,LastTransitionTime:2022-12-22 17:17:37 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-12-22 18:15:07 +0000 UTC,LastTransitionTime:2022-12-22 17:17:37 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-12-22 18:15:07 +0000 UTC,LastTransitionTime:2022-12-22 17:17:59 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.35.142,},NodeAddress{Type:ExternalIP,Address:35.90.235.118,},NodeAddress{Type:InternalDNS,Address:i-0243def68c064e8ef.us-west-2.compute.internal,},NodeAddress{Type:Hostname,Address:i-0243def68c064e8ef.us-west-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-35-90-235-118.us-west-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2c8ae28bb7834ee5f6b4c4ef144a8e,SystemUUID:ec2c8ae2-8bb7-834e-e5f6-b4c4ef144a8e,BootID:5843895c-91be-4c36-b700-9e551cb745de,KernelVersion:5.10.157-139.675.amzn2.x86_64,OSImage:Amazon Linux 2,ContainerRuntimeVersion:containerd://1.6.14,KubeletVersion:v1.26.0,KubeProxyVersion:v1.26.0,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:166719855,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/jessie-dnsutils@sha256:24aaf2626d6b27864c29de2097e8bbb840b3a414271bf7c8995e431e47d8408e registry.k8s.io/e2e-test-images/jessie-dnsutils:1.7],SizeBytes:112030336,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:dd75ec974b0a2a6f6bb47001ba09207976e625db898d1b16735528c009cb171c registry.k8s.io/etcd:3.5.6-0],SizeBytes:102542580,},ContainerImage{Names:[registry.k8s.io/kube-proxy-amd64:v1.26.0],SizeBytes:67205320,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:16bbf38c463a4223d8cfe4da12bc61010b082a79b4bb003e2d3ba3ece5dd5f9e registry.k8s.io/e2e-test-images/agnhost:2.43],SizeBytes:51706353,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nautilus@sha256:80ba6c8c44f9623f06e868a1aa66026c8ec438ad814f9ec95e9333b415fe3550 registry.k8s.io/e2e-test-images/nautilus:1.7],SizeBytes:49641698,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:3fe7acf013d1264ffded116b80a73dc129a449b0fccdb8d21af8279f2233f36e registry.k8s.io/e2e-test-images/httpd:2.4.39-4],SizeBytes:41901587,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:148b022f5c5da426fc2f3c14b5c0867e58ef05961510c84749ac1fddcb0fef22 registry.k8s.io/e2e-test-images/httpd:2.4.38-4],SizeBytes:40764257,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:10e231cf07c82dfab5141c1afe548dc734e5fc1f67665ec3982a325d1bd31a9a registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.0],SizeBytes:29725756,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:ee3b525d5b89db99da3b8eb521d9cd90cb6e9ef0fbb651e98bb37be78d36b5b8 registry.k8s.io/sig-storage/csi-provisioner:v3.3.0],SizeBytes:25491225,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:425d8f1b769398127767b06ed97ce62578a3179bcb99809ce93a1649e025ffe7 registry.k8s.io/sig-storage/csi-resizer:v1.6.0],SizeBytes:24148884,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:291334908ddf71a4661fd7f6d9d97274de8a5378a2b6fdfeb2ce73414a34f82f registry.k8s.io/sig-storage/csi-snapshotter:v6.1.0],SizeBytes:23881995,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:9a685020911e2725ad019dbce6e4a5ab93d51e3d4557f115e64343345e05781b registry.k8s.io/sig-storage/csi-attacher:v4.0.0],SizeBytes:23847201,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:92257881c1d6493cf18299a24af42330f891166560047902b8d431fb66b01af5 registry.k8s.io/sig-storage/hostpathplugin:v1.9.0],SizeBytes:18758628,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nonroot@sha256:ee9f50b3c64b174d296d91ca9f69a914ac30e59095dfb462b2b518ad28a63655 registry.k8s.io/e2e-test-images/nonroot:1.4],SizeBytes:17747885,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:933940f13b3ea0abc62e656c1aa5c5b47c04b15d71250413a6b821bd0c58b94e registry.k8s.io/sig-storage/livenessprobe:v2.7.0],SizeBytes:8688564,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:5c99cf6a02adda929b10321dbf4ecfa00d87be9ba4fb456006237d530ab4baa1 registry.k8s.io/e2e-test-images/nginx:1.14-4],SizeBytes:6978614,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nonewprivs@sha256:8ac1264691820febacf3aea5d152cbde6d10685731ec14966a9401c6f47a68ac registry.k8s.io/e2e-test-images/nonewprivs:1.3],SizeBytes:3263463,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf registry.k8s.io/e2e-test-images/busybox:1.29-2],SizeBytes:732424,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:2e0f836850e09b8b7cc937681d6194537a09fbd5f6b9e08f4d646a85128e8937 registry.k8s.io/e2e-test-images/busybox:1.29-4],SizeBytes:731990,},ContainerImage{Names:[registry.k8s.io/pause@sha256:7031c1b283388d2c2e09b57badb803c05ebed362dc88d84b480cc47f72a21097 registry.k8s.io/pause:3.9],SizeBytes:321520,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[kubernetes.io/csi/csi-hostpath-ephemeral-6886^4e266409-8221-11ed-800d-8a8e987c6028 kubernetes.io/csi/csi-hostpath-ephemeral-8407^9fd9423f-821f-11ed-b5cc-02642e84c89b],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/csi-hostpath-ephemeral-8407^9fd9423f-821f-11ed-b5cc-02642e84c89b,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/csi-hostpath-ephemeral-6886^4e266409-8221-11ed-800d-8a8e987c6028,DevicePath:,},},Config:nil,},} Dec 22 18:16:28.180: INFO: Logging kubelet events for node i-0243def68c064e8ef ... skipping 123 lines ... Latency metrics for node i-0ea878d3671f2eda2 [DeferCleanup (Each)] [Testpattern: Generic Ephemeral-volume (default fs) (immediate-binding)] ephemeral tear down framework | framework.go:193 [1mSTEP:[0m Destroying namespace "ephemeral-8407" for this suite. [38;5;243m12/22/22 18:16:30.415[0m [38;5;243m<< End Captured GinkgoWriter Output[0m [38;5;9mDec 22 18:01:09.131: While waiting for pvc resize to finish: error while waiting for controller resize to finish: timed out waiting for the condition[0m [38;5;9mIn [1m[It][0m[38;5;9m at: [1mtest/e2e/storage/testsuites/ephemeral.go:264[0m [38;5;243m------------------------------[0m [1m[38;5;9mGinkgo timed out waiting for all parallel procs to report back[0m [38;5;243mTest suite:[0m e2e (./_rundir/c6ac28b3-821b-11ed-b47a-e2b55b2f9198) ... skipping 11 lines ... Dec 22 17:21:25.718: INFO: Driver loaded from path [/home/prow/go/src/k8s.io/kops/tests/e2e/csi-manifests/aws-ebs/driver.yaml]: &{DriverInfo:{Name:ebs.csi.aws.com InTreePluginName: FeatureTag: MaxFileSize:0 SupportedSizeRange:{Max:16Ti Min:1Gi} SupportedFsType:map[:{} ext4:{} xfs:{}] SupportedMountOption:map[dirsync:{}] RequiredMountOption:map[] Capabilities:map[block:true controllerExpansion:true exec:true fsGroup:true multipods:true nodeExpansion:true offlineExpansion:true onlineExpansion:true persistence:true pvcDataSource:false snapshotDataSource:true topology:true volumeLimits:true] RequiredAccessModes:[] TopologyKeys:[topology.ebs.csi.aws.com/zone] NumAllowedTopologies:0 StressTestOptions:<nil> VolumeSnapshotStressTestOptions:<nil> PerformanceTestOptions:<nil>} StorageClass:{FromName:false FromFile:tests/e2e/csi-manifests/aws-ebs/sc.yaml FromExistingClassName:} SnapshotClass:{FromName:false FromFile: FromExistingClassName:} InlineVolumes:[] ClientNodeName: Timeouts:map[]} I1222 17:21:25.719348 6613 e2e.go:126] Starting e2e run "83af9224-f11c-418d-bdba-4b9b741d28ea" on Ginkgo node 1 [1mOutput from proc 2:[0m Dec 22 17:21:25.536: INFO: Driver loaded from path [/home/prow/go/src/k8s.io/kops/tests/e2e/csi-manifests/aws-ebs/driver.yaml]: &{DriverInfo:{Name:ebs.csi.aws.com InTreePluginName: FeatureTag: MaxFileSize:0 SupportedSizeRange:{Max:16Ti Min:1Gi} SupportedFsType:map[:{} ext4:{} xfs:{}] SupportedMountOption:map[dirsync:{}] RequiredMountOption:map[] Capabilities:map[block:true controllerExpansion:true exec:true fsGroup:true multipods:true nodeExpansion:true offlineExpansion:true onlineExpansion:true persistence:true pvcDataSource:false snapshotDataSource:true topology:true volumeLimits:true] RequiredAccessModes:[] TopologyKeys:[topology.ebs.csi.aws.com/zone] NumAllowedTopologies:0 StressTestOptions:<nil> VolumeSnapshotStressTestOptions:<nil> PerformanceTestOptions:<nil>} StorageClass:{FromName:false FromFile:tests/e2e/csi-manifests/aws-ebs/sc.yaml FromExistingClassName:} SnapshotClass:{FromName:false FromFile: FromExistingClassName:} InlineVolumes:[] ClientNodeName: Timeouts:map[]} I1222 17:21:25.537249 6614 e2e.go:126] Starting e2e run "46f3dd50-2d7d-41a8-8e3d-2fcdf667e591" on Ginkgo node 2 --- FAIL: TestE2E (2522.27s) FAIL [1mOutput from proc 3:[0m Dec 22 17:21:25.713: INFO: Driver loaded from path [/home/prow/go/src/k8s.io/kops/tests/e2e/csi-manifests/aws-ebs/driver.yaml]: &{DriverInfo:{Name:ebs.csi.aws.com InTreePluginName: FeatureTag: MaxFileSize:0 SupportedSizeRange:{Max:16Ti Min:1Gi} SupportedFsType:map[:{} ext4:{} xfs:{}] SupportedMountOption:map[dirsync:{}] RequiredMountOption:map[] Capabilities:map[block:true controllerExpansion:true exec:true fsGroup:true multipods:true nodeExpansion:true offlineExpansion:true onlineExpansion:true persistence:true pvcDataSource:false snapshotDataSource:true topology:true volumeLimits:true] RequiredAccessModes:[] TopologyKeys:[topology.ebs.csi.aws.com/zone] NumAllowedTopologies:0 StressTestOptions:<nil> VolumeSnapshotStressTestOptions:<nil> PerformanceTestOptions:<nil>} StorageClass:{FromName:false FromFile:tests/e2e/csi-manifests/aws-ebs/sc.yaml FromExistingClassName:} SnapshotClass:{FromName:false FromFile: FromExistingClassName:} InlineVolumes:[] ClientNodeName: Timeouts:map[]} I1222 17:21:25.719563 6615 e2e.go:126] Starting e2e run "ba76e4bf-f5f1-481b-ad8c-b59e0fc5c2f2" on Ginkgo node 3 --- FAIL: TestE2E (2276.80s) FAIL [1mOutput from proc 4:[0m Dec 22 17:21:25.712: INFO: Driver loaded from path [/home/prow/go/src/k8s.io/kops/tests/e2e/csi-manifests/aws-ebs/driver.yaml]: &{DriverInfo:{Name:ebs.csi.aws.com InTreePluginName: FeatureTag: MaxFileSize:0 SupportedSizeRange:{Max:16Ti Min:1Gi} SupportedFsType:map[:{} ext4:{} xfs:{}] SupportedMountOption:map[dirsync:{}] RequiredMountOption:map[] Capabilities:map[block:true controllerExpansion:true exec:true fsGroup:true multipods:true nodeExpansion:true offlineExpansion:true onlineExpansion:true persistence:true pvcDataSource:false snapshotDataSource:true topology:true volumeLimits:true] RequiredAccessModes:[] TopologyKeys:[topology.ebs.csi.aws.com/zone] NumAllowedTopologies:0 StressTestOptions:<nil> VolumeSnapshotStressTestOptions:<nil> PerformanceTestOptions:<nil>} StorageClass:{FromName:false FromFile:tests/e2e/csi-manifests/aws-ebs/sc.yaml FromExistingClassName:} SnapshotClass:{FromName:false FromFile: FromExistingClassName:} InlineVolumes:[] ClientNodeName: Timeouts:map[]} I1222 17:21:25.745527 6617 e2e.go:126] Starting e2e run "64390a03-43ea-43cf-acde-f981fb234eb2" on Ginkgo node 4 --- FAIL: TestE2E (2262.67s) FAIL [1mOutput from proc 5:[0m Dec 22 17:21:25.660: INFO: Driver loaded from path [/home/prow/go/src/k8s.io/kops/tests/e2e/csi-manifests/aws-ebs/driver.yaml]: &{DriverInfo:{Name:ebs.csi.aws.com InTreePluginName: FeatureTag: MaxFileSize:0 SupportedSizeRange:{Max:16Ti Min:1Gi} SupportedFsType:map[:{} ext4:{} xfs:{}] SupportedMountOption:map[dirsync:{}] RequiredMountOption:map[] Capabilities:map[block:true controllerExpansion:true exec:true fsGroup:true multipods:true nodeExpansion:true offlineExpansion:true onlineExpansion:true persistence:true pvcDataSource:false snapshotDataSource:true topology:true volumeLimits:true] RequiredAccessModes:[] TopologyKeys:[topology.ebs.csi.aws.com/zone] NumAllowedTopologies:0 StressTestOptions:<nil> VolumeSnapshotStressTestOptions:<nil> PerformanceTestOptions:<nil>} StorageClass:{FromName:false FromFile:tests/e2e/csi-manifests/aws-ebs/sc.yaml FromExistingClassName:} SnapshotClass:{FromName:false FromFile: FromExistingClassName:} InlineVolumes:[] ClientNodeName: Timeouts:map[]} I1222 17:21:25.664797 6619 e2e.go:126] Starting e2e run "56c1eedf-b016-46bf-9202-e8304c2c9b95" on Ginkgo node 5 --- FAIL: TestE2E (2358.51s) FAIL [1mOutput from proc 6:[0m Dec 22 17:21:25.801: INFO: Driver loaded from path [/home/prow/go/src/k8s.io/kops/tests/e2e/csi-manifests/aws-ebs/driver.yaml]: &{DriverInfo:{Name:ebs.csi.aws.com InTreePluginName: FeatureTag: MaxFileSize:0 SupportedSizeRange:{Max:16Ti Min:1Gi} SupportedFsType:map[:{} ext4:{} xfs:{}] SupportedMountOption:map[dirsync:{}] RequiredMountOption:map[] Capabilities:map[block:true controllerExpansion:true exec:true fsGroup:true multipods:true nodeExpansion:true offlineExpansion:true onlineExpansion:true persistence:true pvcDataSource:false snapshotDataSource:true topology:true volumeLimits:true] RequiredAccessModes:[] TopologyKeys:[topology.ebs.csi.aws.com/zone] NumAllowedTopologies:0 StressTestOptions:<nil> VolumeSnapshotStressTestOptions:<nil> PerformanceTestOptions:<nil>} StorageClass:{FromName:false FromFile:tests/e2e/csi-manifests/aws-ebs/sc.yaml FromExistingClassName:} SnapshotClass:{FromName:false FromFile: FromExistingClassName:} InlineVolumes:[] ClientNodeName: Timeouts:map[]} I1222 17:21:25.801973 6621 e2e.go:126] Starting e2e run "403db869-10a2-4585-92dd-ab74fd722ac6" on Ginkgo node 6 --- FAIL: TestE2E (2320.56s) FAIL [1mOutput from proc 7:[0m Dec 22 17:21:25.752: INFO: Driver loaded from path [/home/prow/go/src/k8s.io/kops/tests/e2e/csi-manifests/aws-ebs/driver.yaml]: &{DriverInfo:{Name:ebs.csi.aws.com InTreePluginName: FeatureTag: MaxFileSize:0 SupportedSizeRange:{Max:16Ti Min:1Gi} SupportedFsType:map[:{} ext4:{} xfs:{}] SupportedMountOption:map[dirsync:{}] RequiredMountOption:map[] Capabilities:map[block:true controllerExpansion:true exec:true fsGroup:true multipods:true nodeExpansion:true offlineExpansion:true onlineExpansion:true persistence:true pvcDataSource:false snapshotDataSource:true topology:true volumeLimits:true] RequiredAccessModes:[] TopologyKeys:[topology.ebs.csi.aws.com/zone] NumAllowedTopologies:0 StressTestOptions:<nil> VolumeSnapshotStressTestOptions:<nil> PerformanceTestOptions:<nil>} StorageClass:{FromName:false FromFile:tests/e2e/csi-manifests/aws-ebs/sc.yaml FromExistingClassName:} SnapshotClass:{FromName:false FromFile: FromExistingClassName:} InlineVolumes:[] ClientNodeName: Timeouts:map[]} I1222 17:21:25.756850 6643 e2e.go:126] Starting e2e run "c7f44d0a-a18f-4924-85a6-a14739bacbad" on Ginkgo node 7 --- FAIL: TestE2E (2332.00s) FAIL [1mOutput from proc 8:[0m Dec 22 17:21:25.881: INFO: Driver loaded from path [/home/prow/go/src/k8s.io/kops/tests/e2e/csi-manifests/aws-ebs/driver.yaml]: &{DriverInfo:{Name:ebs.csi.aws.com InTreePluginName: FeatureTag: MaxFileSize:0 SupportedSizeRange:{Max:16Ti Min:1Gi} SupportedFsType:map[:{} ext4:{} xfs:{}] SupportedMountOption:map[dirsync:{}] RequiredMountOption:map[] Capabilities:map[block:true controllerExpansion:true exec:true fsGroup:true multipods:true nodeExpansion:true offlineExpansion:true onlineExpansion:true persistence:true pvcDataSource:false snapshotDataSource:true topology:true volumeLimits:true] RequiredAccessModes:[] TopologyKeys:[topology.ebs.csi.aws.com/zone] NumAllowedTopologies:0 StressTestOptions:<nil> VolumeSnapshotStressTestOptions:<nil> PerformanceTestOptions:<nil>} StorageClass:{FromName:false FromFile:tests/e2e/csi-manifests/aws-ebs/sc.yaml FromExistingClassName:} SnapshotClass:{FromName:false FromFile: FromExistingClassName:} InlineVolumes:[] ClientNodeName: Timeouts:map[]} I1222 17:21:25.883079 6652 e2e.go:126] Starting e2e run "981b8268-9618-47ca-bfac-7a3484d4b680" on Ginkgo node 8 PASS [1mOutput from proc 9:[0m Dec 22 17:21:25.756: INFO: Driver loaded from path [/home/prow/go/src/k8s.io/kops/tests/e2e/csi-manifests/aws-ebs/driver.yaml]: &{DriverInfo:{Name:ebs.csi.aws.com InTreePluginName: FeatureTag: MaxFileSize:0 SupportedSizeRange:{Max:16Ti Min:1Gi} SupportedFsType:map[:{} ext4:{} xfs:{}] SupportedMountOption:map[dirsync:{}] RequiredMountOption:map[] Capabilities:map[block:true controllerExpansion:true exec:true fsGroup:true multipods:true nodeExpansion:true offlineExpansion:true onlineExpansion:true persistence:true pvcDataSource:false snapshotDataSource:true topology:true volumeLimits:true] RequiredAccessModes:[] TopologyKeys:[topology.ebs.csi.aws.com/zone] NumAllowedTopologies:0 StressTestOptions:<nil> VolumeSnapshotStressTestOptions:<nil> PerformanceTestOptions:<nil>} StorageClass:{FromName:false FromFile:tests/e2e/csi-manifests/aws-ebs/sc.yaml FromExistingClassName:} SnapshotClass:{FromName:false FromFile: FromExistingClassName:} InlineVolumes:[] ClientNodeName: Timeouts:map[]} I1222 17:21:25.769374 6659 e2e.go:126] Starting e2e run "9f13b329-7933-401f-a28c-4a64b25670f6" on Ginkgo node 9 --- FAIL: TestE2E (2487.39s) FAIL [1mOutput from proc 10:[0m Dec 22 17:21:25.833: INFO: Driver loaded from path [/home/prow/go/src/k8s.io/kops/tests/e2e/csi-manifests/aws-ebs/driver.yaml]: &{DriverInfo:{Name:ebs.csi.aws.com InTreePluginName: FeatureTag: MaxFileSize:0 SupportedSizeRange:{Max:16Ti Min:1Gi} SupportedFsType:map[:{} ext4:{} xfs:{}] SupportedMountOption:map[dirsync:{}] RequiredMountOption:map[] Capabilities:map[block:true controllerExpansion:true exec:true fsGroup:true multipods:true nodeExpansion:true offlineExpansion:true onlineExpansion:true persistence:true pvcDataSource:false snapshotDataSource:true topology:true volumeLimits:true] RequiredAccessModes:[] TopologyKeys:[topology.ebs.csi.aws.com/zone] NumAllowedTopologies:0 StressTestOptions:<nil> VolumeSnapshotStressTestOptions:<nil> PerformanceTestOptions:<nil>} StorageClass:{FromName:false FromFile:tests/e2e/csi-manifests/aws-ebs/sc.yaml FromExistingClassName:} SnapshotClass:{FromName:false FromFile: FromExistingClassName:} InlineVolumes:[] ClientNodeName: Timeouts:map[]} I1222 17:21:25.834128 6670 e2e.go:126] Starting e2e run "b0c02294-d0a6-4af9-a0bc-56a94d5e1b7c" on Ginkgo node 10 --- FAIL: TestE2E (2555.41s) FAIL [1mOutput from proc 11:[0m Dec 22 17:21:26.085: INFO: Driver loaded from path [/home/prow/go/src/k8s.io/kops/tests/e2e/csi-manifests/aws-ebs/driver.yaml]: &{DriverInfo:{Name:ebs.csi.aws.com InTreePluginName: FeatureTag: MaxFileSize:0 SupportedSizeRange:{Max:16Ti Min:1Gi} SupportedFsType:map[:{} ext4:{} xfs:{}] SupportedMountOption:map[dirsync:{}] RequiredMountOption:map[] Capabilities:map[block:true controllerExpansion:true exec:true fsGroup:true multipods:true nodeExpansion:true offlineExpansion:true onlineExpansion:true persistence:true pvcDataSource:false snapshotDataSource:true topology:true volumeLimits:true] RequiredAccessModes:[] TopologyKeys:[topology.ebs.csi.aws.com/zone] NumAllowedTopologies:0 StressTestOptions:<nil> VolumeSnapshotStressTestOptions:<nil> PerformanceTestOptions:<nil>} StorageClass:{FromName:false FromFile:tests/e2e/csi-manifests/aws-ebs/sc.yaml FromExistingClassName:} SnapshotClass:{FromName:false FromFile: FromExistingClassName:} InlineVolumes:[] ClientNodeName: Timeouts:map[]} I1222 17:21:26.086538 6685 e2e.go:126] Starting e2e run "69f88272-2326-4279-bba3-b1c7cdaac941" on Ginkgo node 11 --- FAIL: TestE2E (2298.75s) FAIL [1mOutput from proc 12:[0m Dec 22 17:21:25.954: INFO: Driver loaded from path [/home/prow/go/src/k8s.io/kops/tests/e2e/csi-manifests/aws-ebs/driver.yaml]: &{DriverInfo:{Name:ebs.csi.aws.com InTreePluginName: FeatureTag: MaxFileSize:0 SupportedSizeRange:{Max:16Ti Min:1Gi} SupportedFsType:map[:{} ext4:{} xfs:{}] SupportedMountOption:map[dirsync:{}] RequiredMountOption:map[] Capabilities:map[block:true controllerExpansion:true exec:true fsGroup:true multipods:true nodeExpansion:true offlineExpansion:true onlineExpansion:true persistence:true pvcDataSource:false snapshotDataSource:true topology:true volumeLimits:true] RequiredAccessModes:[] TopologyKeys:[topology.ebs.csi.aws.com/zone] NumAllowedTopologies:0 StressTestOptions:<nil> VolumeSnapshotStressTestOptions:<nil> PerformanceTestOptions:<nil>} StorageClass:{FromName:false FromFile:tests/e2e/csi-manifests/aws-ebs/sc.yaml FromExistingClassName:} SnapshotClass:{FromName:false FromFile: FromExistingClassName:} InlineVolumes:[] ClientNodeName: Timeouts:map[]} I1222 17:21:25.954876 6687 e2e.go:126] Starting e2e run "619285fa-61af-4bb8-af3d-d24eb5905017" on Ginkgo node 12 --- FAIL: TestE2E (2262.10s) FAIL [1mOutput from proc 13:[0m Dec 22 17:21:26.066: INFO: Driver loaded from path [/home/prow/go/src/k8s.io/kops/tests/e2e/csi-manifests/aws-ebs/driver.yaml]: &{DriverInfo:{Name:ebs.csi.aws.com InTreePluginName: FeatureTag: MaxFileSize:0 SupportedSizeRange:{Max:16Ti Min:1Gi} SupportedFsType:map[:{} ext4:{} xfs:{}] SupportedMountOption:map[dirsync:{}] RequiredMountOption:map[] Capabilities:map[block:true controllerExpansion:true exec:true fsGroup:true multipods:true nodeExpansion:true offlineExpansion:true onlineExpansion:true persistence:true pvcDataSource:false snapshotDataSource:true topology:true volumeLimits:true] RequiredAccessModes:[] TopologyKeys:[topology.ebs.csi.aws.com/zone] NumAllowedTopologies:0 StressTestOptions:<nil> VolumeSnapshotStressTestOptions:<nil> PerformanceTestOptions:<nil>} StorageClass:{FromName:false FromFile:tests/e2e/csi-manifests/aws-ebs/sc.yaml FromExistingClassName:} SnapshotClass:{FromName:false FromFile: FromExistingClassName:} InlineVolumes:[] ClientNodeName: Timeouts:map[]} I1222 17:21:26.084815 6699 e2e.go:126] Starting e2e run "9a50827b-8d64-4eae-ab05-ed3cfb089552" on Ginkgo node 13 --- FAIL: TestE2E (2616.95s) FAIL [1mOutput from proc 14:[0m Dec 22 17:21:26.113: INFO: Driver loaded from path [/home/prow/go/src/k8s.io/kops/tests/e2e/csi-manifests/aws-ebs/driver.yaml]: &{DriverInfo:{Name:ebs.csi.aws.com InTreePluginName: FeatureTag: MaxFileSize:0 SupportedSizeRange:{Max:16Ti Min:1Gi} SupportedFsType:map[:{} ext4:{} xfs:{}] SupportedMountOption:map[dirsync:{}] RequiredMountOption:map[] Capabilities:map[block:true controllerExpansion:true exec:true fsGroup:true multipods:true nodeExpansion:true offlineExpansion:true onlineExpansion:true persistence:true pvcDataSource:false snapshotDataSource:true topology:true volumeLimits:true] RequiredAccessModes:[] TopologyKeys:[topology.ebs.csi.aws.com/zone] NumAllowedTopologies:0 StressTestOptions:<nil> VolumeSnapshotStressTestOptions:<nil> PerformanceTestOptions:<nil>} StorageClass:{FromName:false FromFile:tests/e2e/csi-manifests/aws-ebs/sc.yaml FromExistingClassName:} SnapshotClass:{FromName:false FromFile: FromExistingClassName:} InlineVolumes:[] ClientNodeName: Timeouts:map[]} I1222 17:21:26.114695 6713 e2e.go:126] Starting e2e run "23091f4b-d92b-41ba-9955-89b4b39264c6" on Ginkgo node 14 [1mOutput from proc 15:[0m Dec 22 17:21:26.162: INFO: Driver loaded from path [/home/prow/go/src/k8s.io/kops/tests/e2e/csi-manifests/aws-ebs/driver.yaml]: &{DriverInfo:{Name:ebs.csi.aws.com InTreePluginName: FeatureTag: MaxFileSize:0 SupportedSizeRange:{Max:16Ti Min:1Gi} SupportedFsType:map[:{} ext4:{} xfs:{}] SupportedMountOption:map[dirsync:{}] RequiredMountOption:map[] Capabilities:map[block:true controllerExpansion:true exec:true fsGroup:true multipods:true nodeExpansion:true offlineExpansion:true onlineExpansion:true persistence:true pvcDataSource:false snapshotDataSource:true topology:true volumeLimits:true] RequiredAccessModes:[] TopologyKeys:[topology.ebs.csi.aws.com/zone] NumAllowedTopologies:0 StressTestOptions:<nil> VolumeSnapshotStressTestOptions:<nil> PerformanceTestOptions:<nil>} StorageClass:{FromName:false FromFile:tests/e2e/csi-manifests/aws-ebs/sc.yaml FromExistingClassName:} SnapshotClass:{FromName:false FromFile: FromExistingClassName:} InlineVolumes:[] ClientNodeName: Timeouts:map[]} I1222 17:21:26.163535 6719 e2e.go:126] Starting e2e run "2ed61322-29d9-43f7-bfe9-5d83874f14dc" on Ginkgo node 15 --- FAIL: TestE2E (2290.65s) FAIL [1mOutput from proc 16:[0m Dec 22 17:21:26.289: INFO: Driver loaded from path [/home/prow/go/src/k8s.io/kops/tests/e2e/csi-manifests/aws-ebs/driver.yaml]: &{DriverInfo:{Name:ebs.csi.aws.com InTreePluginName: FeatureTag: MaxFileSize:0 SupportedSizeRange:{Max:16Ti Min:1Gi} SupportedFsType:map[:{} ext4:{} xfs:{}] SupportedMountOption:map[dirsync:{}] RequiredMountOption:map[] Capabilities:map[block:true controllerExpansion:true exec:true fsGroup:true multipods:true nodeExpansion:true offlineExpansion:true onlineExpansion:true persistence:true pvcDataSource:false snapshotDataSource:true topology:true volumeLimits:true] RequiredAccessModes:[] TopologyKeys:[topology.ebs.csi.aws.com/zone] NumAllowedTopologies:0 StressTestOptions:<nil> VolumeSnapshotStressTestOptions:<nil> PerformanceTestOptions:<nil>} StorageClass:{FromName:false FromFile:tests/e2e/csi-manifests/aws-ebs/sc.yaml FromExistingClassName:} SnapshotClass:{FromName:false FromFile: FromExistingClassName:} InlineVolumes:[] ClientNodeName: Timeouts:map[]} I1222 17:21:26.315063 6746 e2e.go:126] Starting e2e run "2be920fe-1170-4a7c-b398-b9b35581976c" on Ginkgo node 16 --- FAIL: TestE2E (2326.25s) FAIL [1mOutput from proc 17:[0m Dec 22 17:21:26.264: INFO: Driver loaded from path [/home/prow/go/src/k8s.io/kops/tests/e2e/csi-manifests/aws-ebs/driver.yaml]: &{DriverInfo:{Name:ebs.csi.aws.com InTreePluginName: FeatureTag: MaxFileSize:0 SupportedSizeRange:{Max:16Ti Min:1Gi} SupportedFsType:map[:{} ext4:{} xfs:{}] SupportedMountOption:map[dirsync:{}] RequiredMountOption:map[] Capabilities:map[block:true controllerExpansion:true exec:true fsGroup:true multipods:true nodeExpansion:true offlineExpansion:true onlineExpansion:true persistence:true pvcDataSource:false snapshotDataSource:true topology:true volumeLimits:true] RequiredAccessModes:[] TopologyKeys:[topology.ebs.csi.aws.com/zone] NumAllowedTopologies:0 StressTestOptions:<nil> VolumeSnapshotStressTestOptions:<nil> PerformanceTestOptions:<nil>} StorageClass:{FromName:false FromFile:tests/e2e/csi-manifests/aws-ebs/sc.yaml FromExistingClassName:} SnapshotClass:{FromName:false FromFile: FromExistingClassName:} InlineVolumes:[] ClientNodeName: Timeouts:map[]} I1222 17:21:26.266496 6748 e2e.go:126] Starting e2e run "9483ad4e-9a37-4cd2-930d-a3ee733721aa" on Ginkgo node 17 --- FAIL: TestE2E (2291.84s) FAIL [1mOutput from proc 18:[0m Dec 22 17:21:26.398: INFO: Driver loaded from path [/home/prow/go/src/k8s.io/kops/tests/e2e/csi-manifests/aws-ebs/driver.yaml]: &{DriverInfo:{Name:ebs.csi.aws.com InTreePluginName: FeatureTag: MaxFileSize:0 SupportedSizeRange:{Max:16Ti Min:1Gi} SupportedFsType:map[:{} ext4:{} xfs:{}] SupportedMountOption:map[dirsync:{}] RequiredMountOption:map[] Capabilities:map[block:true controllerExpansion:true exec:true fsGroup:true multipods:true nodeExpansion:true offlineExpansion:true onlineExpansion:true persistence:true pvcDataSource:false snapshotDataSource:true topology:true volumeLimits:true] RequiredAccessModes:[] TopologyKeys:[topology.ebs.csi.aws.com/zone] NumAllowedTopologies:0 StressTestOptions:<nil> VolumeSnapshotStressTestOptions:<nil> PerformanceTestOptions:<nil>} StorageClass:{FromName:false FromFile:tests/e2e/csi-manifests/aws-ebs/sc.yaml FromExistingClassName:} SnapshotClass:{FromName:false FromFile: FromExistingClassName:} InlineVolumes:[] ClientNodeName: Timeouts:map[]} I1222 17:21:26.405471 6754 e2e.go:126] Starting e2e run "f8a445c5-a27a-4911-84a0-9377a38a58dc" on Ginkgo node 18 --- FAIL: TestE2E (3304.09s) FAIL [1mOutput from proc 19:[0m Dec 22 17:21:26.208: INFO: Driver loaded from path [/home/prow/go/src/k8s.io/kops/tests/e2e/csi-manifests/aws-ebs/driver.yaml]: &{DriverInfo:{Name:ebs.csi.aws.com InTreePluginName: FeatureTag: MaxFileSize:0 SupportedSizeRange:{Max:16Ti Min:1Gi} SupportedFsType:map[:{} ext4:{} xfs:{}] SupportedMountOption:map[dirsync:{}] RequiredMountOption:map[] Capabilities:map[block:true controllerExpansion:true exec:true fsGroup:true multipods:true nodeExpansion:true offlineExpansion:true onlineExpansion:true persistence:true pvcDataSource:false snapshotDataSource:true topology:true volumeLimits:true] RequiredAccessModes:[] TopologyKeys:[topology.ebs.csi.aws.com/zone] NumAllowedTopologies:0 StressTestOptions:<nil> VolumeSnapshotStressTestOptions:<nil> PerformanceTestOptions:<nil>} StorageClass:{FromName:false FromFile:tests/e2e/csi-manifests/aws-ebs/sc.yaml FromExistingClassName:} SnapshotClass:{FromName:false FromFile: FromExistingClassName:} InlineVolumes:[] ClientNodeName: Timeouts:map[]} I1222 17:21:26.220198 6760 e2e.go:126] Starting e2e run "d0f5ffbb-207e-4b5d-b03e-848112068fbc" on Ginkgo node 19 --- FAIL: TestE2E (2516.65s) FAIL [1mOutput from proc 20:[0m Dec 22 17:21:26.484: INFO: Driver loaded from path [/home/prow/go/src/k8s.io/kops/tests/e2e/csi-manifests/aws-ebs/driver.yaml]: &{DriverInfo:{Name:ebs.csi.aws.com InTreePluginName: FeatureTag: MaxFileSize:0 SupportedSizeRange:{Max:16Ti Min:1Gi} SupportedFsType:map[:{} ext4:{} xfs:{}] SupportedMountOption:map[dirsync:{}] RequiredMountOption:map[] Capabilities:map[block:true controllerExpansion:true exec:true fsGroup:true multipods:true nodeExpansion:true offlineExpansion:true onlineExpansion:true persistence:true pvcDataSource:false snapshotDataSource:true topology:true volumeLimits:true] RequiredAccessModes:[] TopologyKeys:[topology.ebs.csi.aws.com/zone] NumAllowedTopologies:0 StressTestOptions:<nil> VolumeSnapshotStressTestOptions:<nil> PerformanceTestOptions:<nil>} StorageClass:{FromName:false FromFile:tests/e2e/csi-manifests/aws-ebs/sc.yaml FromExistingClassName:} SnapshotClass:{FromName:false FromFile: FromExistingClassName:} InlineVolumes:[] ClientNodeName: Timeouts:map[]} I1222 17:21:26.519270 6780 e2e.go:126] Starting e2e run "aff1ccc7-d830-4409-84a7-f5fc2113ddd7" on Ginkgo node 20 --- FAIL: TestE2E (2421.07s) FAIL [1mOutput from proc 21:[0m Dec 22 17:21:26.291: INFO: Driver loaded from path [/home/prow/go/src/k8s.io/kops/tests/e2e/csi-manifests/aws-ebs/driver.yaml]: &{DriverInfo:{Name:ebs.csi.aws.com InTreePluginName: FeatureTag: MaxFileSize:0 SupportedSizeRange:{Max:16Ti Min:1Gi} SupportedFsType:map[:{} ext4:{} xfs:{}] SupportedMountOption:map[dirsync:{}] RequiredMountOption:map[] Capabilities:map[block:true controllerExpansion:true exec:true fsGroup:true multipods:true nodeExpansion:true offlineExpansion:true onlineExpansion:true persistence:true pvcDataSource:false snapshotDataSource:true topology:true volumeLimits:true] RequiredAccessModes:[] TopologyKeys:[topology.ebs.csi.aws.com/zone] NumAllowedTopologies:0 StressTestOptions:<nil> VolumeSnapshotStressTestOptions:<nil> PerformanceTestOptions:<nil>} StorageClass:{FromName:false FromFile:tests/e2e/csi-manifests/aws-ebs/sc.yaml FromExistingClassName:} SnapshotClass:{FromName:false FromFile: FromExistingClassName:} InlineVolumes:[] ClientNodeName: Timeouts:map[]} I1222 17:21:26.295668 6788 e2e.go:126] Starting e2e run "dc16db46-9504-45fd-88d0-5f5d3d139eab" on Ginkgo node 21 --- FAIL: TestE2E (2335.46s) FAIL [1mOutput from proc 22:[0m Dec 22 17:21:26.601: INFO: Driver loaded from path [/home/prow/go/src/k8s.io/kops/tests/e2e/csi-manifests/aws-ebs/driver.yaml]: &{DriverInfo:{Name:ebs.csi.aws.com InTreePluginName: FeatureTag: MaxFileSize:0 SupportedSizeRange:{Max:16Ti Min:1Gi} SupportedFsType:map[:{} ext4:{} xfs:{}] SupportedMountOption:map[dirsync:{}] RequiredMountOption:map[] Capabilities:map[block:true controllerExpansion:true exec:true fsGroup:true multipods:true nodeExpansion:true offlineExpansion:true onlineExpansion:true persistence:true pvcDataSource:false snapshotDataSource:true topology:true volumeLimits:true] RequiredAccessModes:[] TopologyKeys:[topology.ebs.csi.aws.com/zone] NumAllowedTopologies:0 StressTestOptions:<nil> VolumeSnapshotStressTestOptions:<nil> PerformanceTestOptions:<nil>} StorageClass:{FromName:false FromFile:tests/e2e/csi-manifests/aws-ebs/sc.yaml FromExistingClassName:} SnapshotClass:{FromName:false FromFile: FromExistingClassName:} InlineVolumes:[] ClientNodeName: Timeouts:map[]} I1222 17:21:26.602147 6790 e2e.go:126] Starting e2e run "a15f74f8-2a3b-4e04-9170-63a0957d60a0" on Ginkgo node 22 --- FAIL: TestE2E (3119.01s) FAIL [1mOutput from proc 23:[0m Dec 22 17:21:26.581: INFO: Driver loaded from path [/home/prow/go/src/k8s.io/kops/tests/e2e/csi-manifests/aws-ebs/driver.yaml]: &{DriverInfo:{Name:ebs.csi.aws.com InTreePluginName: FeatureTag: MaxFileSize:0 SupportedSizeRange:{Max:16Ti Min:1Gi} SupportedFsType:map[:{} ext4:{} xfs:{}] SupportedMountOption:map[dirsync:{}] RequiredMountOption:map[] Capabilities:map[block:true controllerExpansion:true exec:true fsGroup:true multipods:true nodeExpansion:true offlineExpansion:true onlineExpansion:true persistence:true pvcDataSource:false snapshotDataSource:true topology:true volumeLimits:true] RequiredAccessModes:[] TopologyKeys:[topology.ebs.csi.aws.com/zone] NumAllowedTopologies:0 StressTestOptions:<nil> VolumeSnapshotStressTestOptions:<nil> PerformanceTestOptions:<nil>} StorageClass:{FromName:false FromFile:tests/e2e/csi-manifests/aws-ebs/sc.yaml FromExistingClassName:} SnapshotClass:{FromName:false FromFile: FromExistingClassName:} InlineVolumes:[] ClientNodeName: Timeouts:map[]} I1222 17:21:26.582258 6794 e2e.go:126] Starting e2e run "f54b5695-f996-428e-95db-2163385e3626" on Ginkgo node 23 --- FAIL: TestE2E (3284.84s) FAIL [1mOutput from proc 24:[0m Dec 22 17:21:26.567: INFO: Driver loaded from path [/home/prow/go/src/k8s.io/kops/tests/e2e/csi-manifests/aws-ebs/driver.yaml]: &{DriverInfo:{Name:ebs.csi.aws.com InTreePluginName: FeatureTag: MaxFileSize:0 SupportedSizeRange:{Max:16Ti Min:1Gi} SupportedFsType:map[:{} ext4:{} xfs:{}] SupportedMountOption:map[dirsync:{}] RequiredMountOption:map[] Capabilities:map[block:true controllerExpansion:true exec:true fsGroup:true multipods:true nodeExpansion:true offlineExpansion:true onlineExpansion:true persistence:true pvcDataSource:false snapshotDataSource:true topology:true volumeLimits:true] RequiredAccessModes:[] TopologyKeys:[topology.ebs.csi.aws.com/zone] NumAllowedTopologies:0 StressTestOptions:<nil> VolumeSnapshotStressTestOptions:<nil> PerformanceTestOptions:<nil>} StorageClass:{FromName:false FromFile:tests/e2e/csi-manifests/aws-ebs/sc.yaml FromExistingClassName:} SnapshotClass:{FromName:false FromFile: FromExistingClassName:} InlineVolumes:[] ClientNodeName: Timeouts:map[]} I1222 17:21:26.568174 6800 e2e.go:126] Starting e2e run "26a2c24e-ac55-48ab-a53d-cb1fcf58f694" on Ginkgo node 24 --- FAIL: TestE2E (2255.51s) FAIL [1mOutput from proc 25:[0m Dec 22 17:21:26.513: INFO: Driver loaded from path [/home/prow/go/src/k8s.io/kops/tests/e2e/csi-manifests/aws-ebs/driver.yaml]: &{DriverInfo:{Name:ebs.csi.aws.com InTreePluginName: FeatureTag: MaxFileSize:0 SupportedSizeRange:{Max:16Ti Min:1Gi} SupportedFsType:map[:{} ext4:{} xfs:{}] SupportedMountOption:map[dirsync:{}] RequiredMountOption:map[] Capabilities:map[block:true controllerExpansion:true exec:true fsGroup:true multipods:true nodeExpansion:true offlineExpansion:true onlineExpansion:true persistence:true pvcDataSource:false snapshotDataSource:true topology:true volumeLimits:true] RequiredAccessModes:[] TopologyKeys:[topology.ebs.csi.aws.com/zone] NumAllowedTopologies:0 StressTestOptions:<nil> VolumeSnapshotStressTestOptions:<nil> PerformanceTestOptions:<nil>} StorageClass:{FromName:false FromFile:tests/e2e/csi-manifests/aws-ebs/sc.yaml FromExistingClassName:} SnapshotClass:{FromName:false FromFile: FromExistingClassName:} InlineVolumes:[] ClientNodeName: Timeouts:map[]} I1222 17:21:26.517334 6813 e2e.go:126] Starting e2e run "c26b1aef-4655-4a5f-b7ea-3e463313b44d" on Ginkgo node 25 --- FAIL: TestE2E (2560.16s) FAIL ** End ** Ginkgo ran 1 suite in 1h0m1.864878995s Test Suite Failed [38;5;228mYou're using deprecated Ginkgo functionality:[0m [38;5;228m=============================================[0m [38;5;11m--debug is deprecated[0m [1mLearn more at:[0m [38;5;14m[4mhttps://onsi.github.io/ginkgo/MIGRATING_TO_V2#removed--debug[0m [38;5;243mTo silence deprecations that can be silenced set the following environment variable:[0m [38;5;243mACK_GINKGO_DEPRECATIONS=2.4.0[0m F1222 18:21:27.133410 5602 tester.go:477] failed to run ginkgo tester: exit status 1 I1222 18:21:27.136948 5510 dumplogs.go:45] /home/prow/go/src/k8s.io/kops/_rundir/c6ac28b3-821b-11ed-b47a-e2b55b2f9198/kops toolbox dump --name e2e-e2e-kops-grid-cilium-eni-amzn2-k26.test-cncf-aws.k8s.io --dir /logs/artifacts --private-key /tmp/kops/e2e-e2e-kops-grid-cilium-eni-amzn2-k26.test-cncf-aws.k8s.io/id_ed25519 --ssh-user ec2-user I1222 18:21:27.137226 5510 local.go:42] ⚙️ /home/prow/go/src/k8s.io/kops/_rundir/c6ac28b3-821b-11ed-b47a-e2b55b2f9198/kops toolbox dump --name e2e-e2e-kops-grid-cilium-eni-amzn2-k26.test-cncf-aws.k8s.io --dir /logs/artifacts --private-key /tmp/kops/e2e-e2e-kops-grid-cilium-eni-amzn2-k26.test-cncf-aws.k8s.io/id_ed25519 --ssh-user ec2-user I1222 18:22:23.070490 5510 dumplogs.go:78] /home/prow/go/src/k8s.io/kops/_rundir/c6ac28b3-821b-11ed-b47a-e2b55b2f9198/kops get cluster --name e2e-e2e-kops-grid-cilium-eni-amzn2-k26.test-cncf-aws.k8s.io -o yaml I1222 18:22:23.070539 5510 local.go:42] ⚙️ /home/prow/go/src/k8s.io/kops/_rundir/c6ac28b3-821b-11ed-b47a-e2b55b2f9198/kops get cluster --name e2e-e2e-kops-grid-cilium-eni-amzn2-k26.test-cncf-aws.k8s.io -o yaml I1222 18:22:23.593433 5510 dumplogs.go:78] /home/prow/go/src/k8s.io/kops/_rundir/c6ac28b3-821b-11ed-b47a-e2b55b2f9198/kops get instancegroups --name e2e-e2e-kops-grid-cilium-eni-amzn2-k26.test-cncf-aws.k8s.io -o yaml I1222 18:22:23.593633 5510 local.go:42] ⚙️ /home/prow/go/src/k8s.io/kops/_rundir/c6ac28b3-821b-11ed-b47a-e2b55b2f9198/kops get instancegroups --name e2e-e2e-kops-grid-cilium-eni-amzn2-k26.test-cncf-aws.k8s.io -o yaml ... skipping 348 lines ... route-table:rtb-0e0f5962e7f2a34fb ok vpc:vpc-0adccb4bfca1469bb ok dhcp-options:dopt-079da58283b568fcc ok Deleted kubectl config for e2e-e2e-kops-grid-cilium-eni-amzn2-k26.test-cncf-aws.k8s.io Deleted cluster: "e2e-e2e-kops-grid-cilium-eni-amzn2-k26.test-cncf-aws.k8s.io" Error: exit status 255 + EXIT_VALUE=1 + set +o xtrace