e2e-kops-aws-karpenter #1632414338788626432
Recent runs || View in Spyglass
| Result | FAILURE |
| Tests | 0 failed / 0 succeeded |
| Started | |
| Elapsed | 1h45m |
| Revision | master |
... skipping 140 lines ... I0305 16:16:25.896261 6074 http.go:37] curl https://storage.googleapis.com/kops-ci/bin/1.27.0-alpha.2+v1.27.0-alpha.1-421-g40e9561266/linux/amd64/kops I0305 16:16:26.604613 6074 local.go:42] ⚙️ ssh-keygen -t ed25519 -N -q -f /tmp/kops/e2e-e2e-kops-aws-karpenter.test-cncf-aws.k8s.io/id_ed25519 I0305 16:16:26.611157 6074 up.go:44] Cleaning up any leaked resources from previous cluster I0305 16:16:26.611246 6074 dumplogs.go:45] /home/prow/go/src/k8s.io/kops/_rundir/d5eac005-bb70-11ed-ae50-6e9121926188/kops toolbox dump --name e2e-e2e-kops-aws-karpenter.test-cncf-aws.k8s.io --dir /logs/artifacts --private-key /tmp/kops/e2e-e2e-kops-aws-karpenter.test-cncf-aws.k8s.io/id_ed25519 --ssh-user ubuntu I0305 16:16:26.611335 6074 local.go:42] ⚙️ /home/prow/go/src/k8s.io/kops/_rundir/d5eac005-bb70-11ed-ae50-6e9121926188/kops toolbox dump --name e2e-e2e-kops-aws-karpenter.test-cncf-aws.k8s.io --dir /logs/artifacts --private-key /tmp/kops/e2e-e2e-kops-aws-karpenter.test-cncf-aws.k8s.io/id_ed25519 --ssh-user ubuntu I0305 16:16:26.641546 6095 featureflag.go:164] FeatureFlag "Karpenter"=true Error: Cluster.kops.k8s.io "e2e-e2e-kops-aws-karpenter.test-cncf-aws.k8s.io" not found W0305 16:16:27.124566 6074 down.go:34] Dumping cluster logs at the start of Down() failed: exit status 1 I0305 16:16:27.124617 6074 down.go:48] /home/prow/go/src/k8s.io/kops/_rundir/d5eac005-bb70-11ed-ae50-6e9121926188/kops delete cluster --name e2e-e2e-kops-aws-karpenter.test-cncf-aws.k8s.io --yes I0305 16:16:27.124633 6074 local.go:42] ⚙️ /home/prow/go/src/k8s.io/kops/_rundir/d5eac005-bb70-11ed-ae50-6e9121926188/kops delete cluster --name e2e-e2e-kops-aws-karpenter.test-cncf-aws.k8s.io --yes I0305 16:16:27.154576 6104 featureflag.go:164] FeatureFlag "Karpenter"=true Error: error reading cluster configuration: Cluster.kops.k8s.io "e2e-e2e-kops-aws-karpenter.test-cncf-aws.k8s.io" not found I0305 16:16:27.612418 6074 http.go:37] curl http://metadata.google.internal/computeMetadata/v1/instance/network-interfaces/0/access-configs/0/external-ip 2023/03/05 16:16:27 failed to get external ip from metadata service: http://metadata.google.internal/computeMetadata/v1/instance/network-interfaces/0/access-configs/0/external-ip returned 404 I0305 16:16:27.623998 6074 http.go:37] curl https://ip.jsb.workers.dev I0305 16:16:27.713769 6074 up.go:167] /home/prow/go/src/k8s.io/kops/_rundir/d5eac005-bb70-11ed-ae50-6e9121926188/kops create cluster --name e2e-e2e-kops-aws-karpenter.test-cncf-aws.k8s.io --cloud aws --kubernetes-version https://storage.googleapis.com/k8s-release-dev/ci/v1.27.0-alpha.3.52+d48b8167f7094b --ssh-public-key /tmp/kops/e2e-e2e-kops-aws-karpenter.test-cncf-aws.k8s.io/id_ed25519.pub --set cluster.spec.nodePortAccess=0.0.0.0/0 --image=099720109477/ubuntu/images/hvm-ssd/ubuntu-jammy-22.04-arm64-server-20230303 --channel=alpha --networking=cilium --container-runtime=containerd --instance-manager=karpenter --master-size=c6g.xlarge --discovery-store=s3://k8s-kops-prow/discovery --admin-access 34.170.210.157/32 --master-count 1 --master-volume-size 48 --node-count 4 --node-volume-size 48 --zones eu-central-1a I0305 16:16:27.713808 6074 local.go:42] ⚙️ /home/prow/go/src/k8s.io/kops/_rundir/d5eac005-bb70-11ed-ae50-6e9121926188/kops create cluster --name e2e-e2e-kops-aws-karpenter.test-cncf-aws.k8s.io --cloud aws --kubernetes-version https://storage.googleapis.com/k8s-release-dev/ci/v1.27.0-alpha.3.52+d48b8167f7094b --ssh-public-key /tmp/kops/e2e-e2e-kops-aws-karpenter.test-cncf-aws.k8s.io/id_ed25519.pub --set cluster.spec.nodePortAccess=0.0.0.0/0 --image=099720109477/ubuntu/images/hvm-ssd/ubuntu-jammy-22.04-arm64-server-20230303 --channel=alpha --networking=cilium --container-runtime=containerd --instance-manager=karpenter --master-size=c6g.xlarge --discovery-store=s3://k8s-kops-prow/discovery --admin-access 34.170.210.157/32 --master-count 1 --master-volume-size 48 --node-count 4 --node-volume-size 48 --zones eu-central-1a I0305 16:16:27.752735 6114 featureflag.go:164] FeatureFlag "Karpenter"=true I0305 16:16:27.770963 6114 create_cluster.go:882] Using SSH public key: /tmp/kops/e2e-e2e-kops-aws-karpenter.test-cncf-aws.k8s.io/id_ed25519.pub I0305 16:16:28.210002 6114 new_cluster.go:1347] Cloud Provider ID: "aws" ... skipping 540 lines ... I0305 16:17:13.317516 6074 up.go:251] /home/prow/go/src/k8s.io/kops/_rundir/d5eac005-bb70-11ed-ae50-6e9121926188/kops validate cluster --name e2e-e2e-kops-aws-karpenter.test-cncf-aws.k8s.io --count 10 --wait 15m0s I0305 16:17:13.317579 6074 local.go:42] ⚙️ /home/prow/go/src/k8s.io/kops/_rundir/d5eac005-bb70-11ed-ae50-6e9121926188/kops validate cluster --name e2e-e2e-kops-aws-karpenter.test-cncf-aws.k8s.io --count 10 --wait 15m0s I0305 16:17:13.352818 6156 featureflag.go:164] FeatureFlag "Karpenter"=true Validating cluster e2e-e2e-kops-aws-karpenter.test-cncf-aws.k8s.io W0305 16:17:14.605441 6156 validate_cluster.go:184] (will retry): unexpected error during validation: unable to resolve Kubernetes cluster API URL dns: lookup api.e2e-e2e-kops-aws-karpenter.test-cncf-aws.k8s.io on 10.63.240.10:53: no such host INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-eu-central-1a ControlPlane c6g.xlarge 1 1 eu-central-1a nodes Node - - eu-central-1a NODE STATUS NAME ROLE READY VALIDATION ERRORS KIND NAME MESSAGE dns apiserver Validation Failed The dns-controller Kubernetes deployment has not updated the Kubernetes cluster's API DNS entry to the correct IP address. The API DNS IP address is the placeholder address that kops creates: 203.0.113.123. Please wait about 5-10 minutes for a control plane node to start, dns-controller to launch, and DNS to propagate. The protokube container and dns-controller deployment logs may contain more diagnostic information. Etcd and the API DNS entries must be updated for a kops Kubernetes cluster to start. Validation Failed W0305 16:17:24.638580 6156 validate_cluster.go:232] (will retry): cluster not yet healthy INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-eu-central-1a ControlPlane c6g.xlarge 1 1 eu-central-1a nodes Node - - eu-central-1a NODE STATUS NAME ROLE READY VALIDATION ERRORS KIND NAME MESSAGE dns apiserver Validation Failed The dns-controller Kubernetes deployment has not updated the Kubernetes cluster's API DNS entry to the correct IP address. The API DNS IP address is the placeholder address that kops creates: 203.0.113.123. Please wait about 5-10 minutes for a control plane node to start, dns-controller to launch, and DNS to propagate. The protokube container and dns-controller deployment logs may contain more diagnostic information. Etcd and the API DNS entries must be updated for a kops Kubernetes cluster to start. Validation Failed W0305 16:17:34.669882 6156 validate_cluster.go:232] (will retry): cluster not yet healthy INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-eu-central-1a ControlPlane c6g.xlarge 1 1 eu-central-1a nodes Node - - eu-central-1a NODE STATUS NAME ROLE READY VALIDATION ERRORS KIND NAME MESSAGE dns apiserver Validation Failed The dns-controller Kubernetes deployment has not updated the Kubernetes cluster's API DNS entry to the correct IP address. The API DNS IP address is the placeholder address that kops creates: 203.0.113.123. Please wait about 5-10 minutes for a control plane node to start, dns-controller to launch, and DNS to propagate. The protokube container and dns-controller deployment logs may contain more diagnostic information. Etcd and the API DNS entries must be updated for a kops Kubernetes cluster to start. Validation Failed W0305 16:17:44.709450 6156 validate_cluster.go:232] (will retry): cluster not yet healthy INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-eu-central-1a ControlPlane c6g.xlarge 1 1 eu-central-1a nodes Node - - eu-central-1a NODE STATUS NAME ROLE READY VALIDATION ERRORS KIND NAME MESSAGE dns apiserver Validation Failed The dns-controller Kubernetes deployment has not updated the Kubernetes cluster's API DNS entry to the correct IP address. The API DNS IP address is the placeholder address that kops creates: 203.0.113.123. Please wait about 5-10 minutes for a control plane node to start, dns-controller to launch, and DNS to propagate. The protokube container and dns-controller deployment logs may contain more diagnostic information. Etcd and the API DNS entries must be updated for a kops Kubernetes cluster to start. Validation Failed W0305 16:17:54.744105 6156 validate_cluster.go:232] (will retry): cluster not yet healthy INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-eu-central-1a ControlPlane c6g.xlarge 1 1 eu-central-1a nodes Node - - eu-central-1a NODE STATUS NAME ROLE READY VALIDATION ERRORS KIND NAME MESSAGE dns apiserver Validation Failed The dns-controller Kubernetes deployment has not updated the Kubernetes cluster's API DNS entry to the correct IP address. The API DNS IP address is the placeholder address that kops creates: 203.0.113.123. Please wait about 5-10 minutes for a control plane node to start, dns-controller to launch, and DNS to propagate. The protokube container and dns-controller deployment logs may contain more diagnostic information. Etcd and the API DNS entries must be updated for a kops Kubernetes cluster to start. Validation Failed W0305 16:18:04.801067 6156 validate_cluster.go:232] (will retry): cluster not yet healthy INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-eu-central-1a ControlPlane c6g.xlarge 1 1 eu-central-1a nodes Node - - eu-central-1a NODE STATUS NAME ROLE READY VALIDATION ERRORS KIND NAME MESSAGE dns apiserver Validation Failed The dns-controller Kubernetes deployment has not updated the Kubernetes cluster's API DNS entry to the correct IP address. The API DNS IP address is the placeholder address that kops creates: 203.0.113.123. Please wait about 5-10 minutes for a control plane node to start, dns-controller to launch, and DNS to propagate. The protokube container and dns-controller deployment logs may contain more diagnostic information. Etcd and the API DNS entries must be updated for a kops Kubernetes cluster to start. Validation Failed W0305 16:18:14.835678 6156 validate_cluster.go:232] (will retry): cluster not yet healthy INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-eu-central-1a ControlPlane c6g.xlarge 1 1 eu-central-1a nodes Node - - eu-central-1a NODE STATUS NAME ROLE READY VALIDATION ERRORS KIND NAME MESSAGE dns apiserver Validation Failed The dns-controller Kubernetes deployment has not updated the Kubernetes cluster's API DNS entry to the correct IP address. The API DNS IP address is the placeholder address that kops creates: 203.0.113.123. Please wait about 5-10 minutes for a control plane node to start, dns-controller to launch, and DNS to propagate. The protokube container and dns-controller deployment logs may contain more diagnostic information. Etcd and the API DNS entries must be updated for a kops Kubernetes cluster to start. Validation Failed W0305 16:18:24.892136 6156 validate_cluster.go:232] (will retry): cluster not yet healthy INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-eu-central-1a ControlPlane c6g.xlarge 1 1 eu-central-1a nodes Node - - eu-central-1a NODE STATUS NAME ROLE READY VALIDATION ERRORS KIND NAME MESSAGE dns apiserver Validation Failed The dns-controller Kubernetes deployment has not updated the Kubernetes cluster's API DNS entry to the correct IP address. The API DNS IP address is the placeholder address that kops creates: 203.0.113.123. Please wait about 5-10 minutes for a control plane node to start, dns-controller to launch, and DNS to propagate. The protokube container and dns-controller deployment logs may contain more diagnostic information. Etcd and the API DNS entries must be updated for a kops Kubernetes cluster to start. Validation Failed W0305 16:18:34.923865 6156 validate_cluster.go:232] (will retry): cluster not yet healthy W0305 16:18:44.950116 6156 validate_cluster.go:184] (will retry): unexpected error during validation: unable to resolve Kubernetes cluster API URL dns: lookup api.e2e-e2e-kops-aws-karpenter.test-cncf-aws.k8s.io on 10.63.240.10:53: no such host INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-eu-central-1a ControlPlane c6g.xlarge 1 1 eu-central-1a nodes Node - - eu-central-1a NODE STATUS NAME ROLE READY VALIDATION ERRORS KIND NAME MESSAGE dns apiserver Validation Failed The dns-controller Kubernetes deployment has not updated the Kubernetes cluster's API DNS entry to the correct IP address. The API DNS IP address is the placeholder address that kops creates: 203.0.113.123. Please wait about 5-10 minutes for a control plane node to start, dns-controller to launch, and DNS to propagate. The protokube container and dns-controller deployment logs may contain more diagnostic information. Etcd and the API DNS entries must be updated for a kops Kubernetes cluster to start. Validation Failed W0305 16:18:54.980044 6156 validate_cluster.go:232] (will retry): cluster not yet healthy INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-eu-central-1a ControlPlane c6g.xlarge 1 1 eu-central-1a nodes Node - - eu-central-1a NODE STATUS NAME ROLE READY VALIDATION ERRORS KIND NAME MESSAGE dns apiserver Validation Failed The dns-controller Kubernetes deployment has not updated the Kubernetes cluster's API DNS entry to the correct IP address. The API DNS IP address is the placeholder address that kops creates: 203.0.113.123. Please wait about 5-10 minutes for a control plane node to start, dns-controller to launch, and DNS to propagate. The protokube container and dns-controller deployment logs may contain more diagnostic information. Etcd and the API DNS entries must be updated for a kops Kubernetes cluster to start. Validation Failed W0305 16:19:05.040030 6156 validate_cluster.go:232] (will retry): cluster not yet healthy INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-eu-central-1a ControlPlane c6g.xlarge 1 1 eu-central-1a nodes Node - - eu-central-1a NODE STATUS NAME ROLE READY VALIDATION ERRORS KIND NAME MESSAGE dns apiserver Validation Failed The dns-controller Kubernetes deployment has not updated the Kubernetes cluster's API DNS entry to the correct IP address. The API DNS IP address is the placeholder address that kops creates: 203.0.113.123. Please wait about 5-10 minutes for a control plane node to start, dns-controller to launch, and DNS to propagate. The protokube container and dns-controller deployment logs may contain more diagnostic information. Etcd and the API DNS entries must be updated for a kops Kubernetes cluster to start. Validation Failed W0305 16:19:15.068636 6156 validate_cluster.go:232] (will retry): cluster not yet healthy INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-eu-central-1a ControlPlane c6g.xlarge 1 1 eu-central-1a nodes Node - - eu-central-1a NODE STATUS NAME ROLE READY VALIDATION ERRORS KIND NAME MESSAGE dns apiserver Validation Failed The dns-controller Kubernetes deployment has not updated the Kubernetes cluster's API DNS entry to the correct IP address. The API DNS IP address is the placeholder address that kops creates: 203.0.113.123. Please wait about 5-10 minutes for a control plane node to start, dns-controller to launch, and DNS to propagate. The protokube container and dns-controller deployment logs may contain more diagnostic information. Etcd and the API DNS entries must be updated for a kops Kubernetes cluster to start. Validation Failed W0305 16:19:25.097249 6156 validate_cluster.go:232] (will retry): cluster not yet healthy INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-eu-central-1a ControlPlane c6g.xlarge 1 1 eu-central-1a nodes Node - - eu-central-1a NODE STATUS NAME ROLE READY VALIDATION ERRORS KIND NAME MESSAGE dns apiserver Validation Failed The dns-controller Kubernetes deployment has not updated the Kubernetes cluster's API DNS entry to the correct IP address. The API DNS IP address is the placeholder address that kops creates: 203.0.113.123. Please wait about 5-10 minutes for a control plane node to start, dns-controller to launch, and DNS to propagate. The protokube container and dns-controller deployment logs may contain more diagnostic information. Etcd and the API DNS entries must be updated for a kops Kubernetes cluster to start. Validation Failed W0305 16:19:35.139866 6156 validate_cluster.go:232] (will retry): cluster not yet healthy INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-eu-central-1a ControlPlane c6g.xlarge 1 1 eu-central-1a nodes Node - - eu-central-1a NODE STATUS NAME ROLE READY VALIDATION ERRORS KIND NAME MESSAGE dns apiserver Validation Failed The dns-controller Kubernetes deployment has not updated the Kubernetes cluster's API DNS entry to the correct IP address. The API DNS IP address is the placeholder address that kops creates: 203.0.113.123. Please wait about 5-10 minutes for a control plane node to start, dns-controller to launch, and DNS to propagate. The protokube container and dns-controller deployment logs may contain more diagnostic information. Etcd and the API DNS entries must be updated for a kops Kubernetes cluster to start. Validation Failed W0305 16:19:45.183849 6156 validate_cluster.go:232] (will retry): cluster not yet healthy INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-eu-central-1a ControlPlane c6g.xlarge 1 1 eu-central-1a nodes Node - - eu-central-1a NODE STATUS NAME ROLE READY VALIDATION ERRORS KIND NAME MESSAGE dns apiserver Validation Failed The dns-controller Kubernetes deployment has not updated the Kubernetes cluster's API DNS entry to the correct IP address. The API DNS IP address is the placeholder address that kops creates: 203.0.113.123. Please wait about 5-10 minutes for a control plane node to start, dns-controller to launch, and DNS to propagate. The protokube container and dns-controller deployment logs may contain more diagnostic information. Etcd and the API DNS entries must be updated for a kops Kubernetes cluster to start. Validation Failed W0305 16:19:55.240277 6156 validate_cluster.go:232] (will retry): cluster not yet healthy INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-eu-central-1a ControlPlane c6g.xlarge 1 1 eu-central-1a nodes Node - - eu-central-1a NODE STATUS NAME ROLE READY VALIDATION ERRORS KIND NAME MESSAGE dns apiserver Validation Failed The dns-controller Kubernetes deployment has not updated the Kubernetes cluster's API DNS entry to the correct IP address. The API DNS IP address is the placeholder address that kops creates: 203.0.113.123. Please wait about 5-10 minutes for a control plane node to start, dns-controller to launch, and DNS to propagate. The protokube container and dns-controller deployment logs may contain more diagnostic information. Etcd and the API DNS entries must be updated for a kops Kubernetes cluster to start. Validation Failed W0305 16:20:05.272467 6156 validate_cluster.go:232] (will retry): cluster not yet healthy INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-eu-central-1a ControlPlane c6g.xlarge 1 1 eu-central-1a nodes Node - - eu-central-1a NODE STATUS NAME ROLE READY VALIDATION ERRORS KIND NAME MESSAGE dns apiserver Validation Failed The dns-controller Kubernetes deployment has not updated the Kubernetes cluster's API DNS entry to the correct IP address. The API DNS IP address is the placeholder address that kops creates: 203.0.113.123. Please wait about 5-10 minutes for a control plane node to start, dns-controller to launch, and DNS to propagate. The protokube container and dns-controller deployment logs may contain more diagnostic information. Etcd and the API DNS entries must be updated for a kops Kubernetes cluster to start. Validation Failed W0305 16:20:15.303504 6156 validate_cluster.go:232] (will retry): cluster not yet healthy INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-eu-central-1a ControlPlane c6g.xlarge 1 1 eu-central-1a nodes Node - - eu-central-1a NODE STATUS NAME ROLE READY VALIDATION ERRORS KIND NAME MESSAGE dns apiserver Validation Failed The dns-controller Kubernetes deployment has not updated the Kubernetes cluster's API DNS entry to the correct IP address. The API DNS IP address is the placeholder address that kops creates: 203.0.113.123. Please wait about 5-10 minutes for a control plane node to start, dns-controller to launch, and DNS to propagate. The protokube container and dns-controller deployment logs may contain more diagnostic information. Etcd and the API DNS entries must be updated for a kops Kubernetes cluster to start. Validation Failed W0305 16:20:25.344537 6156 validate_cluster.go:232] (will retry): cluster not yet healthy INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-eu-central-1a ControlPlane c6g.xlarge 1 1 eu-central-1a nodes Node - - eu-central-1a NODE STATUS NAME ROLE READY VALIDATION ERRORS KIND NAME MESSAGE dns apiserver Validation Failed The dns-controller Kubernetes deployment has not updated the Kubernetes cluster's API DNS entry to the correct IP address. The API DNS IP address is the placeholder address that kops creates: 203.0.113.123. Please wait about 5-10 minutes for a control plane node to start, dns-controller to launch, and DNS to propagate. The protokube container and dns-controller deployment logs may contain more diagnostic information. Etcd and the API DNS entries must be updated for a kops Kubernetes cluster to start. Validation Failed W0305 16:20:35.420995 6156 validate_cluster.go:232] (will retry): cluster not yet healthy INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-eu-central-1a ControlPlane c6g.xlarge 1 1 eu-central-1a nodes Node - - eu-central-1a NODE STATUS NAME ROLE READY VALIDATION ERRORS KIND NAME MESSAGE dns apiserver Validation Failed The dns-controller Kubernetes deployment has not updated the Kubernetes cluster's API DNS entry to the correct IP address. The API DNS IP address is the placeholder address that kops creates: 203.0.113.123. Please wait about 5-10 minutes for a control plane node to start, dns-controller to launch, and DNS to propagate. The protokube container and dns-controller deployment logs may contain more diagnostic information. Etcd and the API DNS entries must be updated for a kops Kubernetes cluster to start. Validation Failed W0305 16:20:45.449837 6156 validate_cluster.go:232] (will retry): cluster not yet healthy INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-eu-central-1a ControlPlane c6g.xlarge 1 1 eu-central-1a nodes Node - - eu-central-1a NODE STATUS NAME ROLE READY VALIDATION ERRORS KIND NAME MESSAGE dns apiserver Validation Failed The dns-controller Kubernetes deployment has not updated the Kubernetes cluster's API DNS entry to the correct IP address. The API DNS IP address is the placeholder address that kops creates: 203.0.113.123. Please wait about 5-10 minutes for a control plane node to start, dns-controller to launch, and DNS to propagate. The protokube container and dns-controller deployment logs may contain more diagnostic information. Etcd and the API DNS entries must be updated for a kops Kubernetes cluster to start. Validation Failed W0305 16:20:55.484150 6156 validate_cluster.go:232] (will retry): cluster not yet healthy INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-eu-central-1a ControlPlane c6g.xlarge 1 1 eu-central-1a nodes Node - - eu-central-1a NODE STATUS NAME ROLE READY VALIDATION ERRORS KIND NAME MESSAGE dns apiserver Validation Failed The dns-controller Kubernetes deployment has not updated the Kubernetes cluster's API DNS entry to the correct IP address. The API DNS IP address is the placeholder address that kops creates: 203.0.113.123. Please wait about 5-10 minutes for a control plane node to start, dns-controller to launch, and DNS to propagate. The protokube container and dns-controller deployment logs may contain more diagnostic information. Etcd and the API DNS entries must be updated for a kops Kubernetes cluster to start. Validation Failed W0305 16:21:05.526998 6156 validate_cluster.go:232] (will retry): cluster not yet healthy INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-eu-central-1a ControlPlane c6g.xlarge 1 1 eu-central-1a nodes Node - - eu-central-1a NODE STATUS NAME ROLE READY VALIDATION ERRORS KIND NAME MESSAGE dns apiserver Validation Failed The dns-controller Kubernetes deployment has not updated the Kubernetes cluster's API DNS entry to the correct IP address. The API DNS IP address is the placeholder address that kops creates: 203.0.113.123. Please wait about 5-10 minutes for a control plane node to start, dns-controller to launch, and DNS to propagate. The protokube container and dns-controller deployment logs may contain more diagnostic information. Etcd and the API DNS entries must be updated for a kops Kubernetes cluster to start. Validation Failed W0305 16:21:15.570783 6156 validate_cluster.go:232] (will retry): cluster not yet healthy INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-eu-central-1a ControlPlane c6g.xlarge 1 1 eu-central-1a nodes Node - - eu-central-1a NODE STATUS NAME ROLE READY VALIDATION ERRORS KIND NAME MESSAGE dns apiserver Validation Failed The dns-controller Kubernetes deployment has not updated the Kubernetes cluster's API DNS entry to the correct IP address. The API DNS IP address is the placeholder address that kops creates: 203.0.113.123. Please wait about 5-10 minutes for a control plane node to start, dns-controller to launch, and DNS to propagate. The protokube container and dns-controller deployment logs may contain more diagnostic information. Etcd and the API DNS entries must be updated for a kops Kubernetes cluster to start. Validation Failed W0305 16:21:25.615106 6156 validate_cluster.go:232] (will retry): cluster not yet healthy INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-eu-central-1a ControlPlane c6g.xlarge 1 1 eu-central-1a nodes Node - - eu-central-1a NODE STATUS NAME ROLE READY VALIDATION ERRORS KIND NAME MESSAGE dns apiserver Validation Failed The dns-controller Kubernetes deployment has not updated the Kubernetes cluster's API DNS entry to the correct IP address. The API DNS IP address is the placeholder address that kops creates: 203.0.113.123. Please wait about 5-10 minutes for a control plane node to start, dns-controller to launch, and DNS to propagate. The protokube container and dns-controller deployment logs may contain more diagnostic information. Etcd and the API DNS entries must be updated for a kops Kubernetes cluster to start. Validation Failed W0305 16:21:35.643955 6156 validate_cluster.go:232] (will retry): cluster not yet healthy INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-eu-central-1a ControlPlane c6g.xlarge 1 1 eu-central-1a nodes Node - - eu-central-1a NODE STATUS NAME ROLE READY VALIDATION ERRORS KIND NAME MESSAGE dns apiserver Validation Failed The dns-controller Kubernetes deployment has not updated the Kubernetes cluster's API DNS entry to the correct IP address. The API DNS IP address is the placeholder address that kops creates: 203.0.113.123. Please wait about 5-10 minutes for a control plane node to start, dns-controller to launch, and DNS to propagate. The protokube container and dns-controller deployment logs may contain more diagnostic information. Etcd and the API DNS entries must be updated for a kops Kubernetes cluster to start. Validation Failed W0305 16:21:45.674391 6156 validate_cluster.go:232] (will retry): cluster not yet healthy INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-eu-central-1a ControlPlane c6g.xlarge 1 1 eu-central-1a nodes Node - - eu-central-1a NODE STATUS NAME ROLE READY VALIDATION ERRORS KIND NAME MESSAGE dns apiserver Validation Failed The dns-controller Kubernetes deployment has not updated the Kubernetes cluster's API DNS entry to the correct IP address. The API DNS IP address is the placeholder address that kops creates: 203.0.113.123. Please wait about 5-10 minutes for a control plane node to start, dns-controller to launch, and DNS to propagate. The protokube container and dns-controller deployment logs may contain more diagnostic information. Etcd and the API DNS entries must be updated for a kops Kubernetes cluster to start. Validation Failed W0305 16:21:55.732320 6156 validate_cluster.go:232] (will retry): cluster not yet healthy INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-eu-central-1a ControlPlane c6g.xlarge 1 1 eu-central-1a nodes Node - - eu-central-1a NODE STATUS NAME ROLE READY VALIDATION ERRORS KIND NAME MESSAGE dns apiserver Validation Failed The dns-controller Kubernetes deployment has not updated the Kubernetes cluster's API DNS entry to the correct IP address. The API DNS IP address is the placeholder address that kops creates: 203.0.113.123. Please wait about 5-10 minutes for a control plane node to start, dns-controller to launch, and DNS to propagate. The protokube container and dns-controller deployment logs may contain more diagnostic information. Etcd and the API DNS entries must be updated for a kops Kubernetes cluster to start. Validation Failed W0305 16:22:05.761685 6156 validate_cluster.go:232] (will retry): cluster not yet healthy INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-eu-central-1a ControlPlane c6g.xlarge 1 1 eu-central-1a nodes Node - - eu-central-1a NODE STATUS NAME ROLE READY VALIDATION ERRORS KIND NAME MESSAGE dns apiserver Validation Failed The dns-controller Kubernetes deployment has not updated the Kubernetes cluster's API DNS entry to the correct IP address. The API DNS IP address is the placeholder address that kops creates: 203.0.113.123. Please wait about 5-10 minutes for a control plane node to start, dns-controller to launch, and DNS to propagate. The protokube container and dns-controller deployment logs may contain more diagnostic information. Etcd and the API DNS entries must be updated for a kops Kubernetes cluster to start. Validation Failed W0305 16:22:15.798435 6156 validate_cluster.go:232] (will retry): cluster not yet healthy INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-eu-central-1a ControlPlane c6g.xlarge 1 1 eu-central-1a nodes Node - - eu-central-1a NODE STATUS NAME ROLE READY VALIDATION ERRORS KIND NAME MESSAGE dns apiserver Validation Failed The dns-controller Kubernetes deployment has not updated the Kubernetes cluster's API DNS entry to the correct IP address. The API DNS IP address is the placeholder address that kops creates: 203.0.113.123. Please wait about 5-10 minutes for a control plane node to start, dns-controller to launch, and DNS to propagate. The protokube container and dns-controller deployment logs may contain more diagnostic information. Etcd and the API DNS entries must be updated for a kops Kubernetes cluster to start. Validation Failed W0305 16:22:25.828746 6156 validate_cluster.go:232] (will retry): cluster not yet healthy INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-eu-central-1a ControlPlane c6g.xlarge 1 1 eu-central-1a nodes Node - - eu-central-1a ... skipping 10 lines ... Pod kube-system/ebs-csi-controller-79785c665b-ghrzp system-cluster-critical pod "ebs-csi-controller-79785c665b-ghrzp" is pending Pod kube-system/ebs-csi-node-jsvvw system-node-critical pod "ebs-csi-node-jsvvw" is pending Pod kube-system/etcd-manager-main-i-0d6a32f749839192f system-cluster-critical pod "etcd-manager-main-i-0d6a32f749839192f" is pending Pod kube-system/karpenter-controller-59fd4bd96b-md9wr system-cluster-critical pod "karpenter-controller-59fd4bd96b-md9wr" is pending Pod kube-system/karpenter-webhook-757f994bb9-4shcm system-cluster-critical pod "karpenter-webhook-757f994bb9-4shcm" is pending Validation Failed W0305 16:22:38.611136 6156 validate_cluster.go:232] (will retry): cluster not yet healthy INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-eu-central-1a ControlPlane c6g.xlarge 1 1 eu-central-1a nodes Node - - eu-central-1a ... skipping 9 lines ... Pod kube-system/ebs-csi-controller-79785c665b-7dmsf system-cluster-critical pod "ebs-csi-controller-79785c665b-7dmsf" is pending Pod kube-system/ebs-csi-controller-79785c665b-ghrzp system-cluster-critical pod "ebs-csi-controller-79785c665b-ghrzp" is pending Pod kube-system/ebs-csi-node-jsvvw system-node-critical pod "ebs-csi-node-jsvvw" is pending Pod kube-system/karpenter-controller-59fd4bd96b-md9wr system-cluster-critical pod "karpenter-controller-59fd4bd96b-md9wr" is pending Pod kube-system/karpenter-webhook-757f994bb9-4shcm system-cluster-critical pod "karpenter-webhook-757f994bb9-4shcm" is pending Validation Failed W0305 16:22:50.644384 6156 validate_cluster.go:232] (will retry): cluster not yet healthy INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-eu-central-1a ControlPlane c6g.xlarge 1 1 eu-central-1a nodes Node - - eu-central-1a ... skipping 9 lines ... Pod kube-system/ebs-csi-controller-79785c665b-7dmsf system-cluster-critical pod "ebs-csi-controller-79785c665b-7dmsf" is pending Pod kube-system/ebs-csi-controller-79785c665b-ghrzp system-cluster-critical pod "ebs-csi-controller-79785c665b-ghrzp" is pending Pod kube-system/ebs-csi-node-jsvvw system-node-critical pod "ebs-csi-node-jsvvw" is pending Pod kube-system/karpenter-controller-59fd4bd96b-md9wr system-cluster-critical pod "karpenter-controller-59fd4bd96b-md9wr" is pending Pod kube-system/karpenter-webhook-757f994bb9-4shcm system-cluster-critical pod "karpenter-webhook-757f994bb9-4shcm" is pending Validation Failed W0305 16:23:02.603449 6156 validate_cluster.go:232] (will retry): cluster not yet healthy INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-eu-central-1a ControlPlane c6g.xlarge 1 1 eu-central-1a nodes Node - - eu-central-1a ... skipping 8 lines ... Pod kube-system/ebs-csi-controller-79785c665b-7dmsf system-cluster-critical pod "ebs-csi-controller-79785c665b-7dmsf" is pending Pod kube-system/ebs-csi-controller-79785c665b-ghrzp system-cluster-critical pod "ebs-csi-controller-79785c665b-ghrzp" is pending Pod kube-system/ebs-csi-node-jsvvw system-node-critical pod "ebs-csi-node-jsvvw" is pending Pod kube-system/karpenter-controller-59fd4bd96b-md9wr system-cluster-critical pod "karpenter-controller-59fd4bd96b-md9wr" is pending Pod kube-system/karpenter-webhook-757f994bb9-4shcm system-cluster-critical pod "karpenter-webhook-757f994bb9-4shcm" is pending Validation Failed W0305 16:23:14.541470 6156 validate_cluster.go:232] (will retry): cluster not yet healthy INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-eu-central-1a ControlPlane c6g.xlarge 1 1 eu-central-1a nodes Node - - eu-central-1a ... skipping 7 lines ... Pod kube-system/coredns-autoscaler-8495f5575-plvdw system-cluster-critical pod "coredns-autoscaler-8495f5575-plvdw" is pending Pod kube-system/ebs-csi-controller-79785c665b-7dmsf system-cluster-critical pod "ebs-csi-controller-79785c665b-7dmsf" is pending Pod kube-system/ebs-csi-controller-79785c665b-ghrzp system-cluster-critical pod "ebs-csi-controller-79785c665b-ghrzp" is pending Pod kube-system/ebs-csi-node-jsvvw system-node-critical pod "ebs-csi-node-jsvvw" is pending Pod kube-system/karpenter-webhook-757f994bb9-4shcm system-cluster-critical pod "karpenter-webhook-757f994bb9-4shcm" is not ready (webhook) Validation Failed W0305 16:23:26.519620 6156 validate_cluster.go:232] (will retry): cluster not yet healthy INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-eu-central-1a ControlPlane c6g.xlarge 1 1 eu-central-1a nodes Node - - eu-central-1a ... skipping 5 lines ... KIND NAME MESSAGE Pod kube-system/coredns-745bddcfbb-gm4h2 system-cluster-critical pod "coredns-745bddcfbb-gm4h2" is pending Pod kube-system/coredns-autoscaler-8495f5575-plvdw system-cluster-critical pod "coredns-autoscaler-8495f5575-plvdw" is pending Pod kube-system/ebs-csi-controller-79785c665b-7dmsf system-cluster-critical pod "ebs-csi-controller-79785c665b-7dmsf" is pending Pod kube-system/ebs-csi-controller-79785c665b-ghrzp system-cluster-critical pod "ebs-csi-controller-79785c665b-ghrzp" is pending Validation Failed W0305 16:23:38.452417 6156 validate_cluster.go:232] (will retry): cluster not yet healthy INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-eu-central-1a ControlPlane c6g.xlarge 1 1 eu-central-1a nodes Node - - eu-central-1a ... skipping 5 lines ... KIND NAME MESSAGE Pod kube-system/coredns-745bddcfbb-gm4h2 system-cluster-critical pod "coredns-745bddcfbb-gm4h2" is pending Pod kube-system/coredns-autoscaler-8495f5575-plvdw system-cluster-critical pod "coredns-autoscaler-8495f5575-plvdw" is pending Pod kube-system/ebs-csi-controller-79785c665b-7dmsf system-cluster-critical pod "ebs-csi-controller-79785c665b-7dmsf" is pending Pod kube-system/ebs-csi-controller-79785c665b-ghrzp system-cluster-critical pod "ebs-csi-controller-79785c665b-ghrzp" is pending Validation Failed W0305 16:23:50.388861 6156 validate_cluster.go:232] (will retry): cluster not yet healthy INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-eu-central-1a ControlPlane c6g.xlarge 1 1 eu-central-1a nodes Node - - eu-central-1a ... skipping 5 lines ... KIND NAME MESSAGE Pod kube-system/coredns-745bddcfbb-gm4h2 system-cluster-critical pod "coredns-745bddcfbb-gm4h2" is pending Pod kube-system/coredns-autoscaler-8495f5575-plvdw system-cluster-critical pod "coredns-autoscaler-8495f5575-plvdw" is pending Pod kube-system/ebs-csi-controller-79785c665b-7dmsf system-cluster-critical pod "ebs-csi-controller-79785c665b-7dmsf" is pending Pod kube-system/ebs-csi-controller-79785c665b-ghrzp system-cluster-critical pod "ebs-csi-controller-79785c665b-ghrzp" is pending Validation Failed W0305 16:24:02.337393 6156 validate_cluster.go:232] (will retry): cluster not yet healthy INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-eu-central-1a ControlPlane c6g.xlarge 1 1 eu-central-1a nodes Node - - eu-central-1a ... skipping 5 lines ... KIND NAME MESSAGE Pod kube-system/coredns-745bddcfbb-gm4h2 system-cluster-critical pod "coredns-745bddcfbb-gm4h2" is pending Pod kube-system/coredns-autoscaler-8495f5575-plvdw system-cluster-critical pod "coredns-autoscaler-8495f5575-plvdw" is pending Pod kube-system/ebs-csi-controller-79785c665b-7dmsf system-cluster-critical pod "ebs-csi-controller-79785c665b-7dmsf" is pending Pod kube-system/ebs-csi-controller-79785c665b-ghrzp system-cluster-critical pod "ebs-csi-controller-79785c665b-ghrzp" is pending Validation Failed W0305 16:24:14.287130 6156 validate_cluster.go:232] (will retry): cluster not yet healthy W0305 16:24:26.306479 6156 node_conditions.go:49] v1.NodeReady condition not set on node i-06df568f6e5c59fff W0305 16:24:26.306526 6156 node_conditions.go:49] v1.NodeReady condition not set on node i-056ad61525499081e INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-eu-central-1a ControlPlane c6g.xlarge 1 1 eu-central-1a ... skipping 15 lines ... Pod kube-system/coredns-autoscaler-8495f5575-plvdw system-cluster-critical pod "coredns-autoscaler-8495f5575-plvdw" is pending Pod kube-system/ebs-csi-controller-79785c665b-7dmsf system-cluster-critical pod "ebs-csi-controller-79785c665b-7dmsf" is pending Pod kube-system/ebs-csi-controller-79785c665b-ghrzp system-cluster-critical pod "ebs-csi-controller-79785c665b-ghrzp" is pending Pod kube-system/ebs-csi-node-dkrvw system-node-critical pod "ebs-csi-node-dkrvw" is pending Pod kube-system/ebs-csi-node-n98cq system-node-critical pod "ebs-csi-node-n98cq" is pending Validation Failed W0305 16:24:26.431485 6156 validate_cluster.go:232] (will retry): cluster not yet healthy W0305 16:24:38.239214 6156 node_conditions.go:49] v1.NodeReady condition not set on node i-056ad61525499081e W0305 16:24:38.239263 6156 node_conditions.go:49] v1.NodeReady condition not set on node i-06df568f6e5c59fff INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-eu-central-1a ControlPlane c6g.xlarge 1 1 eu-central-1a ... skipping 15 lines ... Pod kube-system/coredns-autoscaler-8495f5575-plvdw system-cluster-critical pod "coredns-autoscaler-8495f5575-plvdw" is pending Pod kube-system/ebs-csi-controller-79785c665b-7dmsf system-cluster-critical pod "ebs-csi-controller-79785c665b-7dmsf" is pending Pod kube-system/ebs-csi-controller-79785c665b-ghrzp system-cluster-critical pod "ebs-csi-controller-79785c665b-ghrzp" is pending Pod kube-system/ebs-csi-node-dkrvw system-node-critical pod "ebs-csi-node-dkrvw" is pending Pod kube-system/ebs-csi-node-n98cq system-node-critical pod "ebs-csi-node-n98cq" is pending Validation Failed W0305 16:24:38.363611 6156 validate_cluster.go:232] (will retry): cluster not yet healthy W0305 16:24:50.138147 6156 node_conditions.go:49] v1.NodeReady condition not set on node i-056ad61525499081e W0305 16:24:50.138186 6156 node_conditions.go:49] v1.NodeReady condition not set on node i-06df568f6e5c59fff INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-eu-central-1a ControlPlane c6g.xlarge 1 1 eu-central-1a ... skipping 15 lines ... Pod kube-system/coredns-autoscaler-8495f5575-plvdw system-cluster-critical pod "coredns-autoscaler-8495f5575-plvdw" is pending Pod kube-system/ebs-csi-controller-79785c665b-7dmsf system-cluster-critical pod "ebs-csi-controller-79785c665b-7dmsf" is pending Pod kube-system/ebs-csi-controller-79785c665b-ghrzp system-cluster-critical pod "ebs-csi-controller-79785c665b-ghrzp" is pending Pod kube-system/ebs-csi-node-dkrvw system-node-critical pod "ebs-csi-node-dkrvw" is pending Pod kube-system/ebs-csi-node-n98cq system-node-critical pod "ebs-csi-node-n98cq" is pending Validation Failed W0305 16:24:50.268769 6156 validate_cluster.go:232] (will retry): cluster not yet healthy W0305 16:25:02.148419 6156 node_conditions.go:49] v1.NodeReady condition not set on node i-056ad61525499081e W0305 16:25:02.148501 6156 node_conditions.go:49] v1.NodeReady condition not set on node i-06df568f6e5c59fff INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-eu-central-1a ControlPlane c6g.xlarge 1 1 eu-central-1a ... skipping 15 lines ... Pod kube-system/coredns-autoscaler-8495f5575-plvdw system-cluster-critical pod "coredns-autoscaler-8495f5575-plvdw" is pending Pod kube-system/ebs-csi-controller-79785c665b-7dmsf system-cluster-critical pod "ebs-csi-controller-79785c665b-7dmsf" is pending Pod kube-system/ebs-csi-controller-79785c665b-ghrzp system-cluster-critical pod "ebs-csi-controller-79785c665b-ghrzp" is pending Pod kube-system/ebs-csi-node-dkrvw system-node-critical pod "ebs-csi-node-dkrvw" is pending Pod kube-system/ebs-csi-node-n98cq system-node-critical pod "ebs-csi-node-n98cq" is pending Validation Failed W0305 16:25:02.273537 6156 validate_cluster.go:232] (will retry): cluster not yet healthy W0305 16:25:14.165905 6156 node_conditions.go:49] v1.NodeReady condition not set on node i-056ad61525499081e W0305 16:25:14.165977 6156 node_conditions.go:49] v1.NodeReady condition not set on node i-06df568f6e5c59fff INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-eu-central-1a ControlPlane c6g.xlarge 1 1 eu-central-1a ... skipping 15 lines ... Pod kube-system/coredns-autoscaler-8495f5575-plvdw system-cluster-critical pod "coredns-autoscaler-8495f5575-plvdw" is pending Pod kube-system/ebs-csi-controller-79785c665b-7dmsf system-cluster-critical pod "ebs-csi-controller-79785c665b-7dmsf" is pending Pod kube-system/ebs-csi-controller-79785c665b-ghrzp system-cluster-critical pod "ebs-csi-controller-79785c665b-ghrzp" is pending Pod kube-system/ebs-csi-node-dkrvw system-node-critical pod "ebs-csi-node-dkrvw" is pending Pod kube-system/ebs-csi-node-n98cq system-node-critical pod "ebs-csi-node-n98cq" is pending Validation Failed W0305 16:25:14.292375 6156 validate_cluster.go:232] (will retry): cluster not yet healthy INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-eu-central-1a ControlPlane c6g.xlarge 1 1 eu-central-1a nodes Node - - eu-central-1a ... skipping 13 lines ... Pod kube-system/coredns-autoscaler-8495f5575-plvdw system-cluster-critical pod "coredns-autoscaler-8495f5575-plvdw" is pending Pod kube-system/ebs-csi-controller-79785c665b-7dmsf system-cluster-critical pod "ebs-csi-controller-79785c665b-7dmsf" is pending Pod kube-system/ebs-csi-controller-79785c665b-ghrzp system-cluster-critical pod "ebs-csi-controller-79785c665b-ghrzp" is pending Pod kube-system/ebs-csi-node-dkrvw system-node-critical pod "ebs-csi-node-dkrvw" is pending Pod kube-system/ebs-csi-node-n98cq system-node-critical pod "ebs-csi-node-n98cq" is pending Validation Failed W0305 16:25:26.322653 6156 validate_cluster.go:232] (will retry): cluster not yet healthy INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-eu-central-1a ControlPlane c6g.xlarge 1 1 eu-central-1a nodes Node - - eu-central-1a ... skipping 13 lines ... Pod kube-system/coredns-autoscaler-8495f5575-plvdw system-cluster-critical pod "coredns-autoscaler-8495f5575-plvdw" is pending Pod kube-system/ebs-csi-controller-79785c665b-7dmsf system-cluster-critical pod "ebs-csi-controller-79785c665b-7dmsf" is pending Pod kube-system/ebs-csi-controller-79785c665b-ghrzp system-cluster-critical pod "ebs-csi-controller-79785c665b-ghrzp" is pending Pod kube-system/ebs-csi-node-dkrvw system-node-critical pod "ebs-csi-node-dkrvw" is pending Pod kube-system/ebs-csi-node-n98cq system-node-critical pod "ebs-csi-node-n98cq" is pending Validation Failed W0305 16:25:38.333365 6156 validate_cluster.go:232] (will retry): cluster not yet healthy INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-eu-central-1a ControlPlane c6g.xlarge 1 1 eu-central-1a nodes Node - - eu-central-1a ... skipping 13 lines ... Pod kube-system/coredns-autoscaler-8495f5575-plvdw system-cluster-critical pod "coredns-autoscaler-8495f5575-plvdw" is pending Pod kube-system/ebs-csi-controller-79785c665b-7dmsf system-cluster-critical pod "ebs-csi-controller-79785c665b-7dmsf" is pending Pod kube-system/ebs-csi-controller-79785c665b-ghrzp system-cluster-critical pod "ebs-csi-controller-79785c665b-ghrzp" is pending Pod kube-system/ebs-csi-node-dkrvw system-node-critical pod "ebs-csi-node-dkrvw" is pending Pod kube-system/ebs-csi-node-n98cq system-node-critical pod "ebs-csi-node-n98cq" is pending Validation Failed W0305 16:25:50.399498 6156 validate_cluster.go:232] (will retry): cluster not yet healthy INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-eu-central-1a ControlPlane c6g.xlarge 1 1 eu-central-1a nodes Node - - eu-central-1a ... skipping 11 lines ... Pod kube-system/coredns-autoscaler-8495f5575-plvdw system-cluster-critical pod "coredns-autoscaler-8495f5575-plvdw" is pending Pod kube-system/ebs-csi-controller-79785c665b-7dmsf system-cluster-critical pod "ebs-csi-controller-79785c665b-7dmsf" is pending Pod kube-system/ebs-csi-controller-79785c665b-ghrzp system-cluster-critical pod "ebs-csi-controller-79785c665b-ghrzp" is pending Pod kube-system/ebs-csi-node-dkrvw system-node-critical pod "ebs-csi-node-dkrvw" is pending Pod kube-system/ebs-csi-node-n98cq system-node-critical pod "ebs-csi-node-n98cq" is pending Validation Failed W0305 16:26:02.516585 6156 validate_cluster.go:232] (will retry): cluster not yet healthy INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-eu-central-1a ControlPlane c6g.xlarge 1 1 eu-central-1a nodes Node - - eu-central-1a ... skipping 11 lines ... Pod kube-system/coredns-autoscaler-8495f5575-plvdw system-cluster-critical pod "coredns-autoscaler-8495f5575-plvdw" is pending Pod kube-system/ebs-csi-controller-79785c665b-7dmsf system-cluster-critical pod "ebs-csi-controller-79785c665b-7dmsf" is pending Pod kube-system/ebs-csi-controller-79785c665b-ghrzp system-cluster-critical pod "ebs-csi-controller-79785c665b-ghrzp" is pending Pod kube-system/ebs-csi-node-dkrvw system-node-critical pod "ebs-csi-node-dkrvw" is pending Pod kube-system/ebs-csi-node-n98cq system-node-critical pod "ebs-csi-node-n98cq" is pending Validation Failed W0305 16:26:14.447852 6156 validate_cluster.go:232] (will retry): cluster not yet healthy INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-eu-central-1a ControlPlane c6g.xlarge 1 1 eu-central-1a nodes Node - - eu-central-1a ... skipping 7 lines ... KIND NAME MESSAGE Pod kube-system/ebs-csi-controller-79785c665b-7dmsf system-cluster-critical pod "ebs-csi-controller-79785c665b-7dmsf" is pending Pod kube-system/ebs-csi-controller-79785c665b-ghrzp system-cluster-critical pod "ebs-csi-controller-79785c665b-ghrzp" is pending Pod kube-system/ebs-csi-node-dkrvw system-node-critical pod "ebs-csi-node-dkrvw" is pending Pod kube-system/ebs-csi-node-n98cq system-node-critical pod "ebs-csi-node-n98cq" is pending Validation Failed W0305 16:26:26.512748 6156 validate_cluster.go:232] (will retry): cluster not yet healthy INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-eu-central-1a ControlPlane c6g.xlarge 1 1 eu-central-1a nodes Node - - eu-central-1a ... skipping 4 lines ... i-0d6a32f749839192f control-plane True VALIDATION ERRORS KIND NAME MESSAGE Pod kube-system/ebs-csi-controller-79785c665b-ghrzp system-cluster-critical pod "ebs-csi-controller-79785c665b-ghrzp" is not ready (ebs-plugin) Validation Failed W0305 16:26:38.945469 6156 validate_cluster.go:232] (will retry): cluster not yet healthy INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-eu-central-1a ControlPlane c6g.xlarge 1 1 eu-central-1a nodes Node - - eu-central-1a ... skipping 150 lines ... Random Seed: [1m1678033739[0m - will randomize all specs Will run [1m877[0m of [1m7396[0m specs Running in parallel across [1m25[0m processes [38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;10m•[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;10m•[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;10m•[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;10m•[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;10m•[0m[38;5;14mS[0m[38;5;10m•[0m[38;5;14mS[0m[38;5;10m•[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;10m•[0m[38;5;14mS[0m[38;5;10m•[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;10m•[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;10m•[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;10m•[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;10m•[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;10m•[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;10m•[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;10m•[0m[38;5;10m•[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;10m•[0m[38;5;14mS[0m[38;5;10m•[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;10m•[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;10m•[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;10m•[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;10m•[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;10m•[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;10m•[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;10m•[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;10m•[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;10m•[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;10m•[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;10m•[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;10m•[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;10m•[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;10m•[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;10m•[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;10m•[0m[38;5;10m•[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;10m•[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;10m•[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;10m•[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;10m•[0m[38;5;10m•[0m[38;5;14mS[0m[38;5;10m•[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;10m•[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;10m•[0m[38;5;10m•[0m[38;5;10m•[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;10m•[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;10m•[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;10m•[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;10m•[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;10m•[0m[38;5;10m•[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;10m•[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;10m•[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;10m•[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;10m•[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;10m•[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;10m•[0m[38;5;14mS[0m[38;5;10m•[0m[38;5;10m•[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;10m•[0m[38;5;14mS[0m[38;5;10m•[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;10m•[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;10m•[0m[38;5;10m•[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;10m•[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;10m•[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;10m•[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;10m•[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;10m•[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;10m•[0m[38;5;10m•[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;10m•[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;10m•[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;10m•[0m[38;5;14mS[0m[38;5;10m•[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;10m•[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;10m•[0m[38;5;14mS[0m[38;5;10m•[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;10m•[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;10m•[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;10m•[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;10m•[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;10m•[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;10m•[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;10m•[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;10m•[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;10m•[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;10m•[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;10m•[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;10m•[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;10m•[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;10m•[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;10m•[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;10m•[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;10m•[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;10m•[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;10m•[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;10m•[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;10m•[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;10m•[0m[38;5;10m•[0m[38;5;10m•[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;10m•[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;10m•[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;10m•[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;10m•[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;10m•[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;10m•[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;10m•[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;10m•[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;10m•[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;10m•[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;10m•[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;10m•[0m[38;5;14mS[0m[38;5;10m•[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;10m•[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;10m•[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m [38;5;243m------------------------------[0m [38;5;9m• [FAILED] [128.758 seconds][0m [0m[sig-node] Pods [38;5;9m[1m[It] should support pod readiness gates [NodeConformance][0m [38;5;243mtest/e2e/common/node/pods.go:772[0m [38;5;243mTimeline >>[0m [1mSTEP:[0m Creating a kubernetes client [38;5;243m@ 03/05/23 16:31:53.232[0m Mar 5 16:31:53.232: INFO: >>> kubeConfig: /root/.kube/config ... skipping 61 lines ... Mar 5 16:33:11.382: INFO: Expect the Ready condition of pod "pod-ready" to be true, but got false (pod status v1.PodStatus{Phase:"Running", Conditions:[]v1.PodCondition{v1.PodCondition{Type:"k8s.io/test-condition2", Status:"True", LastProbeTime:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), LastTransitionTime:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), Reason:"", Message:""}, v1.PodCondition{Type:"k8s.io/test-condition1", Status:"True", LastProbeTime:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), LastTransitionTime:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), Reason:"", Message:""}, v1.PodCondition{Type:"Initialized", Status:"True", LastProbeTime:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), LastTransitionTime:time.Date(2023, time.March, 5, 16, 31, 54, 0, time.Local), Reason:"", Message:""}, v1.PodCondition{Type:"Ready", Status:"False", LastProbeTime:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), LastTransitionTime:time.Date(2023, time.March, 5, 16, 31, 54, 0, time.Local), Reason:"ReadinessGatesNotReady", Message:"corresponding condition of pod readiness gate \"k8s.io/test-condition2\" does not exist."}, v1.PodCondition{Type:"ContainersReady", Status:"True", LastProbeTime:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), LastTransitionTime:time.Date(2023, time.March, 5, 16, 31, 55, 0, time.Local), Reason:"", Message:""}, v1.PodCondition{Type:"PodScheduled", Status:"True", LastProbeTime:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), LastTransitionTime:time.Date(2023, time.March, 5, 16, 31, 54, 0, time.Local), Reason:"", Message:""}}, Message:"", Reason:"", NominatedNodeName:"", HostIP:"172.20.43.148", PodIP:"100.96.2.132", PodIPs:[]v1.PodIP{v1.PodIP{IP:"100.96.2.132"}}, StartTime:time.Date(2023, time.March, 5, 16, 31, 54, 0, time.Local), InitContainerStatuses:[]v1.ContainerStatus(nil), ContainerStatuses:[]v1.ContainerStatus{v1.ContainerStatus{Name:"pod-readiness-gate", State:v1.ContainerState{Waiting:(*v1.ContainerStateWaiting)(nil), Running:(*v1.ContainerStateRunning)(0xc001652498), Terminated:(*v1.ContainerStateTerminated)(nil)}, LastTerminationState:v1.ContainerState{Waiting:(*v1.ContainerStateWaiting)(nil), Running:(*v1.ContainerStateRunning)(nil), Terminated:(*v1.ContainerStateTerminated)(nil)}, Ready:true, RestartCount:0, Image:"registry.k8s.io/e2e-test-images/busybox:1.29-4", ImageID:"registry.k8s.io/e2e-test-images/busybox@sha256:2e0f836850e09b8b7cc937681d6194537a09fbd5f6b9e08f4d646a85128e8937", ContainerID:"containerd://36e32b3b16c79825d748a4c80c8b32fce109468bc394a7f7c7d973a90e0f5f75", Started:(*bool)(0xc0021e350c), ResourcesAllocated:v1.ResourceList(nil), Resources:(*v1.ResourceRequirements)(nil)}}, QOSClass:"BestEffort", EphemeralContainerStatuses:[]v1.ContainerStatus(nil), Resize:""}) Mar 5 16:33:12.382: INFO: Expect the Ready condition of pod "pod-ready" to be true, but got false (pod status v1.PodStatus{Phase:"Running", Conditions:[]v1.PodCondition{v1.PodCondition{Type:"k8s.io/test-condition2", Status:"True", LastProbeTime:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), LastTransitionTime:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), Reason:"", Message:""}, v1.PodCondition{Type:"k8s.io/test-condition1", Status:"True", LastProbeTime:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), LastTransitionTime:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), Reason:"", Message:""}, v1.PodCondition{Type:"Initialized", Status:"True", LastProbeTime:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), LastTransitionTime:time.Date(2023, time.March, 5, 16, 31, 54, 0, time.Local), Reason:"", Message:""}, v1.PodCondition{Type:"Ready", Status:"False", LastProbeTime:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), LastTransitionTime:time.Date(2023, time.March, 5, 16, 31, 54, 0, time.Local), Reason:"ReadinessGatesNotReady", Message:"corresponding condition of pod readiness gate \"k8s.io/test-condition2\" does not exist."}, v1.PodCondition{Type:"ContainersReady", Status:"True", LastProbeTime:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), LastTransitionTime:time.Date(2023, time.March, 5, 16, 31, 55, 0, time.Local), Reason:"", Message:""}, v1.PodCondition{Type:"PodScheduled", Status:"True", LastProbeTime:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), LastTransitionTime:time.Date(2023, time.March, 5, 16, 31, 54, 0, time.Local), Reason:"", Message:""}}, Message:"", Reason:"", NominatedNodeName:"", HostIP:"172.20.43.148", PodIP:"100.96.2.132", PodIPs:[]v1.PodIP{v1.PodIP{IP:"100.96.2.132"}}, StartTime:time.Date(2023, time.March, 5, 16, 31, 54, 0, time.Local), InitContainerStatuses:[]v1.ContainerStatus(nil), ContainerStatuses:[]v1.ContainerStatus{v1.ContainerStatus{Name:"pod-readiness-gate", State:v1.ContainerState{Waiting:(*v1.ContainerStateWaiting)(nil), Running:(*v1.ContainerStateRunning)(0xc001652870), Terminated:(*v1.ContainerStateTerminated)(nil)}, LastTerminationState:v1.ContainerState{Waiting:(*v1.ContainerStateWaiting)(nil), Running:(*v1.ContainerStateRunning)(nil), Terminated:(*v1.ContainerStateTerminated)(nil)}, Ready:true, RestartCount:0, Image:"registry.k8s.io/e2e-test-images/busybox:1.29-4", ImageID:"registry.k8s.io/e2e-test-images/busybox@sha256:2e0f836850e09b8b7cc937681d6194537a09fbd5f6b9e08f4d646a85128e8937", ContainerID:"containerd://36e32b3b16c79825d748a4c80c8b32fce109468bc394a7f7c7d973a90e0f5f75", Started:(*bool)(0xc0021e39cc), ResourcesAllocated:v1.ResourceList(nil), Resources:(*v1.ResourceRequirements)(nil)}}, QOSClass:"BestEffort", EphemeralContainerStatuses:[]v1.ContainerStatus(nil), Resize:""}) Mar 5 16:33:13.381: INFO: Expect the Ready condition of pod "pod-ready" to be true, but got false (pod status v1.PodStatus{Phase:"Running", Conditions:[]v1.PodCondition{v1.PodCondition{Type:"k8s.io/test-condition2", Status:"True", LastProbeTime:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), LastTransitionTime:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), Reason:"", Message:""}, v1.PodCondition{Type:"k8s.io/test-condition1", Status:"True", LastProbeTime:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), LastTransitionTime:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), Reason:"", Message:""}, v1.PodCondition{Type:"Initialized", Status:"True", LastProbeTime:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), LastTransitionTime:time.Date(2023, time.March, 5, 16, 31, 54, 0, time.Local), Reason:"", Message:""}, v1.PodCondition{Type:"Ready", Status:"False", LastProbeTime:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), LastTransitionTime:time.Date(2023, time.March, 5, 16, 31, 54, 0, time.Local), Reason:"ReadinessGatesNotReady", Message:"corresponding condition of pod readiness gate \"k8s.io/test-condition2\" does not exist."}, v1.PodCondition{Type:"ContainersReady", Status:"True", LastProbeTime:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), LastTransitionTime:time.Date(2023, time.March, 5, 16, 31, 55, 0, time.Local), Reason:"", Message:""}, v1.PodCondition{Type:"PodScheduled", Status:"True", LastProbeTime:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), LastTransitionTime:time.Date(2023, time.March, 5, 16, 31, 54, 0, time.Local), Reason:"", Message:""}}, Message:"", Reason:"", NominatedNodeName:"", HostIP:"172.20.43.148", PodIP:"100.96.2.132", PodIPs:[]v1.PodIP{v1.PodIP{IP:"100.96.2.132"}}, StartTime:time.Date(2023, time.March, 5, 16, 31, 54, 0, time.Local), InitContainerStatuses:[]v1.ContainerStatus(nil), ContainerStatuses:[]v1.ContainerStatus{v1.ContainerStatus{Name:"pod-readiness-gate", State:v1.ContainerState{Waiting:(*v1.ContainerStateWaiting)(nil), Running:(*v1.ContainerStateRunning)(0xc001652be8), Terminated:(*v1.ContainerStateTerminated)(nil)}, LastTerminationState:v1.ContainerState{Waiting:(*v1.ContainerStateWaiting)(nil), Running:(*v1.ContainerStateRunning)(nil), Terminated:(*v1.ContainerStateTerminated)(nil)}, Ready:true, RestartCount:0, Image:"registry.k8s.io/e2e-test-images/busybox:1.29-4", ImageID:"registry.k8s.io/e2e-test-images/busybox@sha256:2e0f836850e09b8b7cc937681d6194537a09fbd5f6b9e08f4d646a85128e8937", ContainerID:"containerd://36e32b3b16c79825d748a4c80c8b32fce109468bc394a7f7c7d973a90e0f5f75", Started:(*bool)(0xc0021e3e1c), ResourcesAllocated:v1.ResourceList(nil), Resources:(*v1.ResourceRequirements)(nil)}}, QOSClass:"BestEffort", EphemeralContainerStatuses:[]v1.ContainerStatus(nil), Resize:""}) Mar 5 16:33:14.384: INFO: Expect the Ready condition of pod "pod-ready" to be true, but got false (pod status v1.PodStatus{Phase:"Running", Conditions:[]v1.PodCondition{v1.PodCondition{Type:"k8s.io/test-condition2", Status:"True", LastProbeTime:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), LastTransitionTime:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), Reason:"", Message:""}, v1.PodCondition{Type:"k8s.io/test-condition1", Status:"True", LastProbeTime:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), LastTransitionTime:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), Reason:"", Message:""}, v1.PodCondition{Type:"Initialized", Status:"True", LastProbeTime:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), LastTransitionTime:time.Date(2023, time.March, 5, 16, 31, 54, 0, time.Local), Reason:"", Message:""}, v1.PodCondition{Type:"Ready", Status:"False", LastProbeTime:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), LastTransitionTime:time.Date(2023, time.March, 5, 16, 31, 54, 0, time.Local), Reason:"ReadinessGatesNotReady", Message:"corresponding condition of pod readiness gate \"k8s.io/test-condition2\" does not exist."}, v1.PodCondition{Type:"ContainersReady", Status:"True", LastProbeTime:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), LastTransitionTime:time.Date(2023, time.March, 5, 16, 31, 55, 0, time.Local), Reason:"", Message:""}, v1.PodCondition{Type:"PodScheduled", Status:"True", LastProbeTime:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), LastTransitionTime:time.Date(2023, time.March, 5, 16, 31, 54, 0, time.Local), Reason:"", Message:""}}, Message:"", Reason:"", NominatedNodeName:"", HostIP:"172.20.43.148", PodIP:"100.96.2.132", PodIPs:[]v1.PodIP{v1.PodIP{IP:"100.96.2.132"}}, StartTime:time.Date(2023, time.March, 5, 16, 31, 54, 0, time.Local), InitContainerStatuses:[]v1.ContainerStatus(nil), ContainerStatuses:[]v1.ContainerStatus{v1.ContainerStatus{Name:"pod-readiness-gate", State:v1.ContainerState{Waiting:(*v1.ContainerStateWaiting)(nil), Running:(*v1.ContainerStateRunning)(0xc0003878c0), Terminated:(*v1.ContainerStateTerminated)(nil)}, LastTerminationState:v1.ContainerState{Waiting:(*v1.ContainerStateWaiting)(nil), Running:(*v1.ContainerStateRunning)(nil), Terminated:(*v1.ContainerStateTerminated)(nil)}, Ready:true, RestartCount:0, Image:"registry.k8s.io/e2e-test-images/busybox:1.29-4", ImageID:"registry.k8s.io/e2e-test-images/busybox@sha256:2e0f836850e09b8b7cc937681d6194537a09fbd5f6b9e08f4d646a85128e8937", ContainerID:"containerd://36e32b3b16c79825d748a4c80c8b32fce109468bc394a7f7c7d973a90e0f5f75", Started:(*bool)(0xc002a2e27c), ResourcesAllocated:v1.ResourceList(nil), Resources:(*v1.ResourceRequirements)(nil)}}, QOSClass:"BestEffort", EphemeralContainerStatuses:[]v1.ContainerStatus(nil), Resize:""}) Mar 5 16:33:15.381: INFO: Expect the Ready condition of pod "pod-ready" to be true, but got false (pod status v1.PodStatus{Phase:"Running", Conditions:[]v1.PodCondition{v1.PodCondition{Type:"k8s.io/test-condition2", Status:"True", LastProbeTime:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), LastTransitionTime:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), Reason:"", Message:""}, v1.PodCondition{Type:"k8s.io/test-condition1", Status:"True", LastProbeTime:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), LastTransitionTime:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), Reason:"", Message:""}, v1.PodCondition{Type:"Initialized", Status:"True", LastProbeTime:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), LastTransitionTime:time.Date(2023, time.March, 5, 16, 31, 54, 0, time.Local), Reason:"", Message:""}, v1.PodCondition{Type:"Ready", Status:"False", LastProbeTime:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), LastTransitionTime:time.Date(2023, time.March, 5, 16, 31, 54, 0, time.Local), Reason:"ReadinessGatesNotReady", Message:"corresponding condition of pod readiness gate \"k8s.io/test-condition2\" does not exist."}, v1.PodCondition{Type:"ContainersReady", Status:"True", LastProbeTime:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), LastTransitionTime:time.Date(2023, time.March, 5, 16, 31, 55, 0, time.Local), Reason:"", Message:""}, v1.PodCondition{Type:"PodScheduled", Status:"True", LastProbeTime:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), LastTransitionTime:time.Date(2023, time.March, 5, 16, 31, 54, 0, time.Local), Reason:"", Message:""}}, Message:"", Reason:"", NominatedNodeName:"", HostIP:"172.20.43.148", PodIP:"100.96.2.132", PodIPs:[]v1.PodIP{v1.PodIP{IP:"100.96.2.132"}}, StartTime:time.Date(2023, time.March, 5, 16, 31, 54, 0, time.Local), InitContainerStatuses:[]v1.ContainerStatus(nil), ContainerStatuses:[]v1.ContainerStatus{v1.ContainerStatus{Name:"pod-readiness-gate", State:v1.ContainerState{Waiting:(*v1.ContainerStateWaiting)(nil), Running:(*v1.ContainerStateRunning)(0xc000387d58), Terminated:(*v1.ContainerStateTerminated)(nil)}, LastTerminationState:v1.ContainerState{Waiting:(*v1.ContainerStateWaiting)(nil), Running:(*v1.ContainerStateRunning)(nil), Terminated:(*v1.ContainerStateTerminated)(nil)}, Ready:true, RestartCount:0, Image:"registry.k8s.io/e2e-test-images/busybox:1.29-4", ImageID:"registry.k8s.io/e2e-test-images/busybox@sha256:2e0f836850e09b8b7cc937681d6194537a09fbd5f6b9e08f4d646a85128e8937", ContainerID:"containerd://36e32b3b16c79825d748a4c80c8b32fce109468bc394a7f7c7d973a90e0f5f75", Started:(*bool)(0xc002a2e71c), ResourcesAllocated:v1.ResourceList(nil), Resources:(*v1.ResourceRequirements)(nil)}}, QOSClass:"BestEffort", EphemeralContainerStatuses:[]v1.ContainerStatus(nil), Resize:""}) Mar 5 16:33:15.493: INFO: Expect the Ready condition of pod "pod-ready" to be true, but got false (pod status v1.PodStatus{Phase:"Running", Conditions:[]v1.PodCondition{v1.PodCondition{Type:"k8s.io/test-condition2", Status:"True", LastProbeTime:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), LastTransitionTime:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), Reason:"", Message:""}, v1.PodCondition{Type:"k8s.io/test-condition1", Status:"True", LastProbeTime:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), LastTransitionTime:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), Reason:"", Message:""}, v1.PodCondition{Type:"Initialized", Status:"True", LastProbeTime:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), LastTransitionTime:time.Date(2023, time.March, 5, 16, 31, 54, 0, time.Local), Reason:"", Message:""}, v1.PodCondition{Type:"Ready", Status:"False", LastProbeTime:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), LastTransitionTime:time.Date(2023, time.March, 5, 16, 31, 54, 0, time.Local), Reason:"ReadinessGatesNotReady", Message:"corresponding condition of pod readiness gate \"k8s.io/test-condition2\" does not exist."}, v1.PodCondition{Type:"ContainersReady", Status:"True", LastProbeTime:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), LastTransitionTime:time.Date(2023, time.March, 5, 16, 31, 55, 0, time.Local), Reason:"", Message:""}, v1.PodCondition{Type:"PodScheduled", Status:"True", LastProbeTime:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), LastTransitionTime:time.Date(2023, time.March, 5, 16, 31, 54, 0, time.Local), Reason:"", Message:""}}, Message:"", Reason:"", NominatedNodeName:"", HostIP:"172.20.43.148", PodIP:"100.96.2.132", PodIPs:[]v1.PodIP{v1.PodIP{IP:"100.96.2.132"}}, StartTime:time.Date(2023, time.March, 5, 16, 31, 54, 0, time.Local), InitContainerStatuses:[]v1.ContainerStatus(nil), ContainerStatuses:[]v1.ContainerStatus{v1.ContainerStatus{Name:"pod-readiness-gate", State:v1.ContainerState{Waiting:(*v1.ContainerStateWaiting)(nil), Running:(*v1.ContainerStateRunning)(0xc000f55ce0), Terminated:(*v1.ContainerStateTerminated)(nil)}, LastTerminationState:v1.ContainerState{Waiting:(*v1.ContainerStateWaiting)(nil), Running:(*v1.ContainerStateRunning)(nil), Terminated:(*v1.ContainerStateTerminated)(nil)}, Ready:true, RestartCount:0, Image:"registry.k8s.io/e2e-test-images/busybox:1.29-4", ImageID:"registry.k8s.io/e2e-test-images/busybox@sha256:2e0f836850e09b8b7cc937681d6194537a09fbd5f6b9e08f4d646a85128e8937", ContainerID:"containerd://36e32b3b16c79825d748a4c80c8b32fce109468bc394a7f7c7d973a90e0f5f75", Started:(*bool)(0xc004f5e80c), ResourcesAllocated:v1.ResourceList(nil), Resources:(*v1.ResourceRequirements)(nil)}}, QOSClass:"BestEffort", EphemeralContainerStatuses:[]v1.ContainerStatus(nil), Resize:""}) Mar 5 16:33:15.493: INFO: Unexpected error: <*errors.errorString | 0xaefe8e0>: { s: "timed out waiting for the condition", } [38;5;9m[FAILED][0m in [It] - test/e2e/common/node/pods.go:808 [38;5;243m@ 03/05/23 16:33:15.494[0m Mar 5 16:33:15.494: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready [1mSTEP:[0m dump namespace information after failure [38;5;243m@ 03/05/23 16:33:15.607[0m [1mSTEP:[0m Collecting events from namespace "pods-9130". [38;5;243m@ 03/05/23 16:33:15.607[0m [1mSTEP:[0m Found 4 events. [38;5;243m@ 03/05/23 16:33:15.723[0m Mar 5 16:33:15.723: INFO: At 2023-03-05 16:31:54 +0000 UTC - event for pod-ready: {default-scheduler } Scheduled: Successfully assigned pods-9130/pod-ready to i-06df568f6e5c59fff Mar 5 16:33:15.723: INFO: At 2023-03-05 16:31:55 +0000 UTC - event for pod-ready: {kubelet i-06df568f6e5c59fff} Pulled: Container image "registry.k8s.io/e2e-test-images/busybox:1.29-4" already present on machine Mar 5 16:33:15.723: INFO: At 2023-03-05 16:31:55 +0000 UTC - event for pod-ready: {kubelet i-06df568f6e5c59fff} Created: Created container pod-readiness-gate Mar 5 16:33:15.723: INFO: At 2023-03-05 16:31:55 +0000 UTC - event for pod-ready: {kubelet i-06df568f6e5c59fff} Started: Started container pod-readiness-gate Mar 5 16:33:15.835: INFO: POD NODE PHASE GRACE CONDITIONS Mar 5 16:33:15.835: INFO: pod-ready i-06df568f6e5c59fff Running [{k8s.io/test-condition2 True 0001-01-01 00:00:00 +0000 UTC 0001-01-01 00:00:00 +0000 UTC } {k8s.io/test-condition1 True 0001-01-01 00:00:00 +0000 UTC 0001-01-01 00:00:00 +0000 UTC } {Initialized True 0001-01-01 00:00:00 +0000 UTC 2023-03-05 16:31:54 +0000 UTC } {Ready False 0001-01-01 00:00:00 +0000 UTC 2023-03-05 16:31:54 +0000 UTC ReadinessGatesNotReady corresponding condition of pod readiness gate "k8s.io/test-condition2" does not exist.} {ContainersReady True 0001-01-01 00:00:00 +0000 UTC 2023-03-05 16:31:55 +0000 UTC } {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2023-03-05 16:31:54 +0000 UTC }] Mar 5 16:33:15.835: INFO: Mar 5 16:33:48.930: INFO: Unable to fetch pods-9130/pod-ready/pod-readiness-gate logs: an error on the server ("unknown") has prevented the request from succeeding (get pods pod-ready) Mar 5 16:33:49.043: INFO: Logging node info for node i-056ad61525499081e Mar 5 16:33:49.155: INFO: Node Info: &Node{ObjectMeta:{i-056ad61525499081e a6ce9ed3-5ae8-4065-b117-6884e24b3632 10762 0 2023-03-05 16:24:16 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:arm64 beta.kubernetes.io/instance-type:t4g.small beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:eu-central-1 failure-domain.beta.kubernetes.io/zone:eu-central-1a karpenter.k8s.aws/instance-category:t karpenter.k8s.aws/instance-cpu:2 karpenter.k8s.aws/instance-family:t4g karpenter.k8s.aws/instance-generation:4 karpenter.k8s.aws/instance-hypervisor:nitro karpenter.k8s.aws/instance-memory:2048 karpenter.k8s.aws/instance-pods:11 karpenter.k8s.aws/instance-size:small karpenter.sh/capacity-type:spot karpenter.sh/initialized:true karpenter.sh/provisioner-name:nodes kubernetes.io/arch:arm64 kubernetes.io/hostname:i-056ad61525499081e kubernetes.io/os:linux node-role.kubernetes.io/node: node-role.kubernetes.io/spot-worker:true node.kubernetes.io/instance-type:t4g.small topology.ebs.csi.aws.com/zone:eu-central-1a topology.hostpath.csi/node:i-056ad61525499081e topology.kubernetes.io/region:eu-central-1 topology.kubernetes.io/zone:eu-central-1a] map[csi.volume.kubernetes.io/nodeid:{"csi-mock-csi-mock-volumes-workload-3941":"i-056ad61525499081e","ebs.csi.aws.com":"i-056ad61525499081e"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [{karpenter.sh/v1alpha5 Provisioner nodes ac5d7599-b2a2-4eae-8da4-1f4c0a54ec82 <nil> 0xc002a2f110}] [karpenter.sh/termination] [{aws-cloud-controller-manager Update v1 2023-03-05 16:24:16 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{}}}} } {kops-controller Update v1 2023-03-05 16:24:16 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{},"f:node-role.kubernetes.io/spot-worker":{}}}} } {karpenter Update v1 2023-03-05 16:25:57 +0000 UTC FieldsV1 {"f:metadata":{"f:finalizers":{".":{},"v:\"karpenter.sh/termination\"":{}},"f:labels":{".":{},"f:karpenter.k8s.aws/instance-category":{},"f:karpenter.k8s.aws/instance-cpu":{},"f:karpenter.k8s.aws/instance-family":{},"f:karpenter.k8s.aws/instance-generation":{},"f:karpenter.k8s.aws/instance-hypervisor":{},"f:karpenter.k8s.aws/instance-memory":{},"f:karpenter.k8s.aws/instance-pods":{},"f:karpenter.k8s.aws/instance-size":{},"f:karpenter.sh/capacity-type":{},"f:karpenter.sh/initialized":{},"f:karpenter.sh/provisioner-name":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}},"f:ownerReferences":{".":{},"k:{\"uid\":\"ac5d7599-b2a2-4eae-8da4-1f4c0a54ec82\"}":{}}},"f:spec":{"f:providerID":{}}} } {kube-controller-manager Update v1 2023-03-05 16:25:57 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:node.alpha.kubernetes.io/ttl":{}},"f:labels":{"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.1.0/24\"":{}}}} } {aws-cloud-controller-manager Update v1 2023-03-05 16:26:48 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{".":{},"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"InternalIP\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kubelet Update v1 2023-03-05 16:33:47 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{"f:kubernetes.io/hostname":{},"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{".":{},"f:cpu":{},"f:ephemeral-storage":{},"f:hugepages-1Gi":{},"f:hugepages-2Mi":{},"f:hugepages-32Mi":{},"f:hugepages-64Ki":{},"f:memory":{},"f:pods":{}},"f:capacity":{".":{},"f:cpu":{},"f:ephemeral-storage":{},"f:hugepages-1Gi":{},"f:hugepages-2Mi":{},"f:hugepages-32Mi":{},"f:hugepages-64Ki":{},"f:memory":{},"f:pods":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:daemonEndpoints":{"f:kubeletEndpoint":{"f:Port":{}}},"f:images":{},"f:nodeInfo":{"f:architecture":{},"f:bootID":{},"f:containerRuntimeVersion":{},"f:kernelVersion":{},"f:kubeProxyVersion":{},"f:kubeletVersion":{},"f:machineID":{},"f:operatingSystem":{},"f:osImage":{},"f:systemUUID":{}},"f:volumesInUse":{}}} status} {kube-controller-manager Update v1 2023-03-05 16:33:48 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{".":{},"k:{\"type\":\"DiskPressure\"}":{".":{},"f:type":{}},"k:{\"type\":\"MemoryPressure\"}":{".":{},"f:type":{}},"k:{\"type\":\"PIDPressure\"}":{".":{},"f:type":{}},"k:{\"type\":\"Ready\"}":{".":{},"f:type":{}}},"f:volumesAttached":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.1.0/24,DoNotUseExternalID:,ProviderID:aws:///eu-central-1a/i-056ad61525499081e,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.1.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49756430336 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-32Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-64Ki: {{0 0} {<nil>} 0 DecimalSI},memory: {{1941540864 0} {<nil>} 1896036Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44780787229 0} {<nil>} 44780787229 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-32Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-64Ki: {{0 0} {<nil>} 0 DecimalSI},memory: {{1836683264 0} {<nil>} 1793636Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-03-05 16:33:47 +0000 UTC,LastTransitionTime:2023-03-05 16:25:57 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-03-05 16:33:47 +0000 UTC,LastTransitionTime:2023-03-05 16:25:48 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-03-05 16:33:47 +0000 UTC,LastTransitionTime:2023-03-05 16:25:48 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-03-05 16:33:47 +0000 UTC,LastTransitionTime:2023-03-05 16:25:48 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.36.179,},NodeAddress{Type:ExternalIP,Address:18.159.37.200,},NodeAddress{Type:InternalDNS,Address:i-056ad61525499081e.eu-central-1.compute.internal,},NodeAddress{Type:Hostname,Address:i-056ad61525499081e.eu-central-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-18-159-37-200.eu-central-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2af3207d1c8c09785c337aae6c6ef0,SystemUUID:ec2af320-7d1c-8c09-785c-337aae6c6ef0,BootID:786f2752-bc30-4729-8aaf-45977b16c253,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 22.04.2 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.27.0-alpha.3.52+d48b8167f7094b,KubeProxyVersion:v1.27.0-alpha.3.52+d48b8167f7094b,OperatingSystem:linux,Architecture:arm64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:157636062,},ContainerImage{Names:[registry.k8s.io/kube-proxy-arm64:v1.27.0-alpha.3.52_d48b8167f7094b],SizeBytes:60554082,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:16bbf38c463a4223d8cfe4da12bc61010b082a79b4bb003e2d3ba3ece5dd5f9e registry.k8s.io/e2e-test-images/agnhost:2.43],SizeBytes:49808364,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:3fe7acf013d1264ffded116b80a73dc129a449b0fccdb8d21af8279f2233f36e registry.k8s.io/e2e-test-images/httpd:2.4.39-4],SizeBytes:41528818,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:148b022f5c5da426fc2f3c14b5c0867e58ef05961510c84749ac1fddcb0fef22 registry.k8s.io/e2e-test-images/httpd:2.4.38-4],SizeBytes:40343601,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:28932904,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:ee3b525d5b89db99da3b8eb521d9cd90cb6e9ef0fbb651e98bb37be78d36b5b8 registry.k8s.io/sig-storage/csi-provisioner:v3.3.0],SizeBytes:23945474,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:425d8f1b769398127767b06ed97ce62578a3179bcb99809ce93a1649e025ffe7 registry.k8s.io/sig-storage/csi-resizer:v1.6.0],SizeBytes:22678632,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:291334908ddf71a4661fd7f6d9d97274de8a5378a2b6fdfeb2ce73414a34f82f registry.k8s.io/sig-storage/csi-snapshotter:v6.1.0],SizeBytes:22434180,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:9a685020911e2725ad019dbce6e4a5ab93d51e3d4557f115e64343345e05781b registry.k8s.io/sig-storage/csi-attacher:v4.0.0],SizeBytes:22394501,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:122bfb8c1edabb3c0edd63f06523e6940d958d19b3957dc7b1d6f81e9f1f6119 registry.k8s.io/sig-storage/csi-provisioner:v3.1.0],SizeBytes:21735644,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:9ebbf9f023e7b41ccee3d52afe39a89e3ddacdbb69269d583abfc25847cfd9e4 registry.k8s.io/sig-storage/csi-resizer:v1.4.0],SizeBytes:20848413,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:8b9c313c05f54fb04f8d430896f5f5904b6cb157df261501b29adc04d2b2dc7b registry.k8s.io/sig-storage/csi-attacher:v3.4.0],SizeBytes:20565480,},ContainerImage{Names:[registry.k8s.io/cpa/cluster-proportional-autoscaler@sha256:68d396900aeaa072c1f27289485fdac29834045a6f3ffe369bf389d830ef572d registry.k8s.io/cpa/cluster-proportional-autoscaler:1.8.6],SizeBytes:19007623,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:92257881c1d6493cf18299a24af42330f891166560047902b8d431fb66b01af5 registry.k8s.io/sig-storage/hostpathplugin:v1.9.0],SizeBytes:18267611,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:9b2d9abda017c28b12f84a344f57cd73fbdb6c2bd7dd5bdd5018246ad1093ba6 registry.k8s.io/sig-storage/hostpathplugin:v1.11.0],SizeBytes:18159085,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:017727efcfeb7d053af68e51436ce8e65edbc6ca573720afb4f79c8594036955 registry.k8s.io/coredns/coredns:v1.10.0],SizeBytes:13766234,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:8502345,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:933940f13b3ea0abc62e656c1aa5c5b47c04b15d71250413a6b821bd0c58b94e registry.k8s.io/sig-storage/livenessprobe:v2.7.0],SizeBytes:8223819,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:7685305,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:5c99cf6a02adda929b10321dbf4ecfa00d87be9ba4fb456006237d530ab4baa1 registry.k8s.io/e2e-test-images/nginx:1.14-4],SizeBytes:6860089,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:2e0f836850e09b8b7cc937681d6194537a09fbd5f6b9e08f4d646a85128e8937 registry.k8s.io/e2e-test-images/busybox:1.29-4],SizeBytes:772933,},ContainerImage{Names:[registry.k8s.io/pause@sha256:7031c1b283388d2c2e09b57badb803c05ebed362dc88d84b480cc47f72a21097 registry.k8s.io/pause:3.9],SizeBytes:268051,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:253553,},},VolumesInUse:[kubernetes.io/csi/csi-mock-csi-mock-volumes-workload-3941^78d47f22-bb73-11ed-a442-2a52b8756b2c kubernetes.io/csi/ebs.csi.aws.com^vol-0997e3e7c22cec909 kubernetes.io/csi/ebs.csi.aws.com^vol-0c3bdf00260f90f1f kubernetes.io/csi/ebs.csi.aws.com^vol-0cfba4c1f44a3d5ed kubernetes.io/csi/ebs.csi.aws.com^vol-0e020446b1cd6abaf kubernetes.io/csi/ebs.csi.aws.com^vol-0efee09c9d1b95e8a],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-0cfba4c1f44a3d5ed,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-0efee09c9d1b95e8a,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/csi-mock-csi-mock-volumes-workload-3941^78d47f22-bb73-11ed-a442-2a52b8756b2c,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-0e020446b1cd6abaf,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-0997e3e7c22cec909,DevicePath:,},},Config:nil,},} Mar 5 16:33:49.156: INFO: Logging kubelet events for node i-056ad61525499081e Mar 5 16:33:49.270: INFO: ... skipping 71 lines ... Logging node info for node i-06df568f6e5c59fff Mar 5 16:33:50.808: INFO: Node Info: &Node{ObjectMeta:{i-06df568f6e5c59fff cb63c681-2e18-474a-b6f3-221325ac0d52 10532 0 2023-03-05 16:24:16 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:arm64 beta.kubernetes.io/instance-type:t4g.small beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:eu-central-1 failure-domain.beta.kubernetes.io/zone:eu-central-1a karpenter.k8s.aws/instance-category:t karpenter.k8s.aws/instance-cpu:2 karpenter.k8s.aws/instance-family:t4g karpenter.k8s.aws/instance-generation:4 karpenter.k8s.aws/instance-hypervisor:nitro karpenter.k8s.aws/instance-memory:2048 karpenter.k8s.aws/instance-pods:11 karpenter.k8s.aws/instance-size:small karpenter.sh/capacity-type:spot karpenter.sh/initialized:true karpenter.sh/provisioner-name:nodes kubernetes.io/arch:arm64 kubernetes.io/hostname:i-06df568f6e5c59fff kubernetes.io/os:linux node-role.kubernetes.io/node: node-role.kubernetes.io/spot-worker:true node.kubernetes.io/instance-type:t4g.small topology.ebs.csi.aws.com/zone:eu-central-1a topology.hostpath.csi/node:i-06df568f6e5c59fff topology.kubernetes.io/region:eu-central-1 topology.kubernetes.io/zone:eu-central-1a] map[csi.volume.kubernetes.io/nodeid:{"csi-hostpath-ephemeral-2147":"i-06df568f6e5c59fff","csi-hostpath-ephemeral-5408":"i-06df568f6e5c59fff","csi-hostpath-ephemeral-6540":"i-06df568f6e5c59fff","csi-mock-csi-mock-volumes-workload-2219":"i-06df568f6e5c59fff","ebs.csi.aws.com":"i-06df568f6e5c59fff"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [{karpenter.sh/v1alpha5 Provisioner nodes ac5d7599-b2a2-4eae-8da4-1f4c0a54ec82 <nil> 0xc003207770}] [karpenter.sh/termination] [{aws-cloud-controller-manager Update v1 2023-03-05 16:24:16 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{}}}} } {kops-controller Update v1 2023-03-05 16:24:16 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{},"f:node-role.kubernetes.io/spot-worker":{}}}} } {karpenter Update v1 2023-03-05 16:25:57 +0000 UTC FieldsV1 {"f:metadata":{"f:finalizers":{".":{},"v:\"karpenter.sh/termination\"":{}},"f:labels":{".":{},"f:karpenter.k8s.aws/instance-category":{},"f:karpenter.k8s.aws/instance-cpu":{},"f:karpenter.k8s.aws/instance-family":{},"f:karpenter.k8s.aws/instance-generation":{},"f:karpenter.k8s.aws/instance-hypervisor":{},"f:karpenter.k8s.aws/instance-memory":{},"f:karpenter.k8s.aws/instance-pods":{},"f:karpenter.k8s.aws/instance-size":{},"f:karpenter.sh/capacity-type":{},"f:karpenter.sh/initialized":{},"f:karpenter.sh/provisioner-name":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}},"f:ownerReferences":{".":{},"k:{\"uid\":\"ac5d7599-b2a2-4eae-8da4-1f4c0a54ec82\"}":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-03-05 16:26:48 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{".":{},"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"InternalIP\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kubelet Update v1 2023-03-05 16:32:27 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{"f:kubernetes.io/hostname":{},"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{".":{},"f:cpu":{},"f:ephemeral-storage":{},"f:hugepages-1Gi":{},"f:hugepages-2Mi":{},"f:hugepages-32Mi":{},"f:hugepages-64Ki":{},"f:memory":{},"f:pods":{}},"f:capacity":{".":{},"f:cpu":{},"f:ephemeral-storage":{},"f:hugepages-1Gi":{},"f:hugepages-2Mi":{},"f:hugepages-32Mi":{},"f:hugepages-64Ki":{},"f:memory":{},"f:pods":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{}}},"f:daemonEndpoints":{"f:kubeletEndpoint":{"f:Port":{}}},"f:images":{},"f:nodeInfo":{"f:architecture":{},"f:bootID":{},"f:containerRuntimeVersion":{},"f:kernelVersion":{},"f:kubeProxyVersion":{},"f:kubeletVersion":{},"f:machineID":{},"f:operatingSystem":{},"f:osImage":{},"f:systemUUID":{}},"f:volumesInUse":{}}} status} {kube-controller-manager Update v1 2023-03-05 16:33:20 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{".":{},"k:{\"type\":\"DiskPressure\"}":{".":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"MemoryPressure\"}":{".":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"PIDPressure\"}":{".":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Ready\"}":{".":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}},"f:volumesAttached":{}}} status} {kube-controller-manager Update v1 2023-03-05 16:33:26 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:node.alpha.kubernetes.io/ttl":{}},"f:labels":{"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.2.0/24\"":{}},"f:taints":{}}} }]},Spec:NodeSpec{PodCIDR:100.96.2.0/24,DoNotUseExternalID:,ProviderID:aws:///eu-central-1a/i-06df568f6e5c59fff,Unschedulable:false,Taints:[]Taint{Taint{Key:node.kubernetes.io/unreachable,Value:,Effect:NoSchedule,TimeAdded:2023-03-05 16:33:20 +0000 UTC,},Taint{Key:node.kubernetes.io/unreachable,Value:,Effect:NoExecute,TimeAdded:2023-03-05 16:33:26 +0000 UTC,},},ConfigSource:nil,PodCIDRs:[100.96.2.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49756430336 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-32Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-64Ki: {{0 0} {<nil>} 0 DecimalSI},memory: {{1941540864 0} {<nil>} 1896036Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44780787229 0} {<nil>} 44780787229 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-32Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-64Ki: {{0 0} {<nil>} 0 DecimalSI},memory: {{1836683264 0} {<nil>} 1793636Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:Ready,Status:Unknown,LastHeartbeatTime:2023-03-05 16:32:27 +0000 UTC,LastTransitionTime:2023-03-05 16:33:20 +0000 UTC,Reason:NodeStatusUnknown,Message:Kubelet stopped posting node status.,},NodeCondition{Type:MemoryPressure,Status:Unknown,LastHeartbeatTime:2023-03-05 16:32:27 +0000 UTC,LastTransitionTime:2023-03-05 16:33:20 +0000 UTC,Reason:NodeStatusUnknown,Message:Kubelet stopped posting node status.,},NodeCondition{Type:DiskPressure,Status:Unknown,LastHeartbeatTime:2023-03-05 16:32:27 +0000 UTC,LastTransitionTime:2023-03-05 16:33:20 +0000 UTC,Reason:NodeStatusUnknown,Message:Kubelet stopped posting node status.,},NodeCondition{Type:PIDPressure,Status:Unknown,LastHeartbeatTime:2023-03-05 16:32:27 +0000 UTC,LastTransitionTime:2023-03-05 16:33:20 +0000 UTC,Reason:NodeStatusUnknown,Message:Kubelet stopped posting node status.,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.43.148,},NodeAddress{Type:ExternalIP,Address:18.192.211.222,},NodeAddress{Type:InternalDNS,Address:i-06df568f6e5c59fff.eu-central-1.compute.internal,},NodeAddress{Type:Hostname,Address:i-06df568f6e5c59fff.eu-central-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-18-192-211-222.eu-central-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec221221864d9409a8ec5f6614a9b94b,SystemUUID:ec221221-864d-9409-a8ec-5f6614a9b94b,BootID:553d31bf-faca-417e-a3db-325839e1b8e4,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 22.04.2 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.27.0-alpha.3.52+d48b8167f7094b,KubeProxyVersion:v1.27.0-alpha.3.52+d48b8167f7094b,OperatingSystem:linux,Architecture:arm64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:157636062,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/jessie-dnsutils@sha256:24aaf2626d6b27864c29de2097e8bbb840b3a414271bf7c8995e431e47d8408e registry.k8s.io/e2e-test-images/jessie-dnsutils:1.7],SizeBytes:116764683,},ContainerImage{Names:[registry.k8s.io/kube-proxy-arm64:v1.27.0-alpha.3.52_d48b8167f7094b],SizeBytes:60554082,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:16bbf38c463a4223d8cfe4da12bc61010b082a79b4bb003e2d3ba3ece5dd5f9e registry.k8s.io/e2e-test-images/agnhost:2.43],SizeBytes:49808364,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:3fe7acf013d1264ffded116b80a73dc129a449b0fccdb8d21af8279f2233f36e registry.k8s.io/e2e-test-images/httpd:2.4.39-4],SizeBytes:41528818,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:148b022f5c5da426fc2f3c14b5c0867e58ef05961510c84749ac1fddcb0fef22 registry.k8s.io/e2e-test-images/httpd:2.4.38-4],SizeBytes:40343601,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:28932904,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:ee3b525d5b89db99da3b8eb521d9cd90cb6e9ef0fbb651e98bb37be78d36b5b8 registry.k8s.io/sig-storage/csi-provisioner:v3.3.0],SizeBytes:23945474,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:425d8f1b769398127767b06ed97ce62578a3179bcb99809ce93a1649e025ffe7 registry.k8s.io/sig-storage/csi-resizer:v1.6.0],SizeBytes:22678632,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:291334908ddf71a4661fd7f6d9d97274de8a5378a2b6fdfeb2ce73414a34f82f registry.k8s.io/sig-storage/csi-snapshotter:v6.1.0],SizeBytes:22434180,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:9a685020911e2725ad019dbce6e4a5ab93d51e3d4557f115e64343345e05781b registry.k8s.io/sig-storage/csi-attacher:v4.0.0],SizeBytes:22394501,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:122bfb8c1edabb3c0edd63f06523e6940d958d19b3957dc7b1d6f81e9f1f6119 registry.k8s.io/sig-storage/csi-provisioner:v3.1.0],SizeBytes:21735644,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:9ebbf9f023e7b41ccee3d52afe39a89e3ddacdbb69269d583abfc25847cfd9e4 registry.k8s.io/sig-storage/csi-resizer:v1.4.0],SizeBytes:20848413,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:8b9c313c05f54fb04f8d430896f5f5904b6cb157df261501b29adc04d2b2dc7b registry.k8s.io/sig-storage/csi-attacher:v3.4.0],SizeBytes:20565480,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nonroot@sha256:ee9f50b3c64b174d296d91ca9f69a914ac30e59095dfb462b2b518ad28a63655 registry.k8s.io/e2e-test-images/nonroot:1.4],SizeBytes:18561972,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:92257881c1d6493cf18299a24af42330f891166560047902b8d431fb66b01af5 registry.k8s.io/sig-storage/hostpathplugin:v1.9.0],SizeBytes:18267611,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:9b2d9abda017c28b12f84a344f57cd73fbdb6c2bd7dd5bdd5018246ad1093ba6 registry.k8s.io/sig-storage/hostpathplugin:v1.11.0],SizeBytes:18159085,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:017727efcfeb7d053af68e51436ce8e65edbc6ca573720afb4f79c8594036955 registry.k8s.io/coredns/coredns:v1.10.0],SizeBytes:13766234,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:8502345,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:933940f13b3ea0abc62e656c1aa5c5b47c04b15d71250413a6b821bd0c58b94e registry.k8s.io/sig-storage/livenessprobe:v2.7.0],SizeBytes:8223819,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:7685305,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:2e0f836850e09b8b7cc937681d6194537a09fbd5f6b9e08f4d646a85128e8937 registry.k8s.io/e2e-test-images/busybox:1.29-4],SizeBytes:772933,},ContainerImage{Names:[registry.k8s.io/pause@sha256:7031c1b283388d2c2e09b57badb803c05ebed362dc88d84b480cc47f72a21097 registry.k8s.io/pause:3.9],SizeBytes:268051,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:253553,},},VolumesInUse:[kubernetes.io/csi/csi-hostpath-ephemeral-5408^443d1daf-bb73-11ed-95c4-5a95d17cbc27 kubernetes.io/csi/csi-hostpath-ephemeral-5408^44411f51-bb73-11ed-95c4-5a95d17cbc27 kubernetes.io/csi/csi-hostpath-ephemeral-6540^4426a07f-bb73-11ed-a604-92687f2c50f8 kubernetes.io/csi/ebs.csi.aws.com^vol-05b67fab5beb6394b],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/csi-hostpath-ephemeral-6540^4426a07f-bb73-11ed-a604-92687f2c50f8,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/csi-hostpath-ephemeral-5408^44411f51-bb73-11ed-95c4-5a95d17cbc27,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/csi-hostpath-ephemeral-5408^443d1daf-bb73-11ed-95c4-5a95d17cbc27,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/csi-hostpath-ephemeral-2147^52df5953-bb73-11ed-8aa7-be49170871b4,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-05b67fab5beb6394b,DevicePath:,},},Config:nil,},} Mar 5 16:33:50.808: INFO: Logging kubelet events for node i-06df568f6e5c59fff Mar 5 16:33:50.923: INFO: Logging pods the kubelet thinks is on node i-06df568f6e5c59fff Mar 5 16:34:01.036: INFO: Unable to retrieve kubelet pods for node i-06df568f6e5c59fff: error trying to reach service: net/http: TLS handshake timeout Mar 5 16:34:01.036: INFO: Logging node info for node i-0d6a32f749839192f Mar 5 16:34:01.149: INFO: Node Info: &Node{ObjectMeta:{i-0d6a32f749839192f 4155fefb-4d3a-4b04-ba53-6ac9e895ff92 10883 0 2023-03-05 16:20:46 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:arm64 beta.kubernetes.io/instance-type:c6g.xlarge beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:eu-central-1 failure-domain.beta.kubernetes.io/zone:eu-central-1a kops.k8s.io/kops-controller-pki: kubernetes.io/arch:arm64 kubernetes.io/hostname:i-0d6a32f749839192f kubernetes.io/os:linux node-role.kubernetes.io/control-plane: node.kubernetes.io/exclude-from-external-load-balancers: node.kubernetes.io/instance-type:c6g.xlarge topology.ebs.csi.aws.com/zone:eu-central-1a topology.kubernetes.io/region:eu-central-1 topology.kubernetes.io/zone:eu-central-1a] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-0d6a32f749839192f"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kubelet Update v1 2023-03-05 16:20:46 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {protokube Update v1 2023-03-05 16:21:25 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kops.k8s.io/kops-controller-pki":{},"f:node-role.kubernetes.io/control-plane":{},"f:node.kubernetes.io/exclude-from-external-load-balancers":{}}}} } {kube-controller-manager Update v1 2023-03-05 16:21:47 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.0.0/24\"":{}}}} } {aws-cloud-controller-manager Update v1 2023-03-05 16:21:48 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{},"f:taints":{}}} } {aws-cloud-controller-manager Update v1 2023-03-05 16:21:48 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kubelet Update v1 2023-03-05 16:34:00 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.0.0/24,DoNotUseExternalID:,ProviderID:aws:///eu-central-1a/i-0d6a32f749839192f,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/control-plane,Value:,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[100.96.0.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{4 0} {<nil>} 4 DecimalSI},ephemeral-storage: {{49756430336 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-32Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-64Ki: {{0 0} {<nil>} 0 DecimalSI},memory: {{8150118400 0} {<nil>} 7959100Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{4 0} {<nil>} 4 DecimalSI},ephemeral-storage: {{44780787229 0} {<nil>} 44780787229 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-32Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-64Ki: {{0 0} {<nil>} 0 DecimalSI},memory: {{8045260800 0} {<nil>} 7856700Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-03-05 16:34:00 +0000 UTC,LastTransitionTime:2023-03-05 16:20:46 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-03-05 16:34:00 +0000 UTC,LastTransitionTime:2023-03-05 16:20:46 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-03-05 16:34:00 +0000 UTC,LastTransitionTime:2023-03-05 16:20:46 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-03-05 16:34:00 +0000 UTC,LastTransitionTime:2023-03-05 16:21:47 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.49.247,},NodeAddress{Type:ExternalIP,Address:18.197.20.82,},NodeAddress{Type:InternalDNS,Address:i-0d6a32f749839192f.eu-central-1.compute.internal,},NodeAddress{Type:Hostname,Address:i-0d6a32f749839192f.eu-central-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-18-197-20-82.eu-central-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2bed249e8ac7b517cb08ccd42b83e5,SystemUUID:ec2bed24-9e8a-c7b5-17cb-08ccd42b83e5,BootID:6f9215c9-dfe8-4b13-84f4-9b2084456198,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 22.04.2 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.27.0-alpha.3.52+d48b8167f7094b,KubeProxyVersion:v1.27.0-alpha.3.52+d48b8167f7094b,OperatingSystem:linux,Architecture:arm64,},Images:[]ContainerImage{ContainerImage{Names:[registry.k8s.io/etcd@sha256:9ce33ba33d8e738a5b85ed50b5080ac746deceed4a7496c550927a7a19ca3b6d registry.k8s.io/etcd:3.5.0-0],SizeBytes:157797353,},ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:157636062,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:4afb99b4690b418ffc2ceb67e1a17376457e441c1f09ab55447f0aaf992fa646 registry.k8s.io/etcd:3.4.3-0],SizeBytes:151677080,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:4ad90a11b55313b182afc186b9876c8e891531b8db4c9bf1541953021618d0e2 registry.k8s.io/etcd:3.4.13-0],SizeBytes:134531559,},ContainerImage{Names:[registry.k8s.io/kube-apiserver-arm64:v1.27.0-alpha.3.52_d48b8167f7094b],SizeBytes:131087707,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:0d2ab6b15b4fe0b903b9e81192d942fb39c29be61fe341cd16db5d0165b0435c registry.k8s.io/etcd:3.3.17-0],SizeBytes:126979107,},ContainerImage{Names:[registry.k8s.io/kube-controller-manager-arm64:v1.27.0-alpha.3.52_d48b8167f7094b],SizeBytes:120667128,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:17da501f5d2a675be46040422a27b7cc21b8a43895ac998b171db1c346f361f7 registry.k8s.io/etcd:3.3.10-0],SizeBytes:81436097,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:6f72b851544986cb0921b53ea655ec04c36131248f16d4ad110cb3ca0c369dc1 registry.k8s.io/etcd:3.5.4-0],SizeBytes:81120118,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:13f53ed1d91e2e11aac476ee9a0269fdda6cc4874eba903efd40daf50c55eee5 registry.k8s.io/etcd:3.5.3-0],SizeBytes:81114891,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:51eae8381dcb1078289fa7b4f3df2630cdc18d09fb56f8e56b41c40e191d6c83 registry.k8s.io/etcd:3.5.7-0],SizeBytes:80665728,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:dd75ec974b0a2a6f6bb47001ba09207976e625db898d1b16735528c009cb171c registry.k8s.io/etcd:3.5.6-0],SizeBytes:80539316,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:905d7ca17fd02bc24c0eba9a062753aba15db3e31422390bc3238eb762339b20 registry.k8s.io/etcd:3.2.24-1],SizeBytes:61500433,},ContainerImage{Names:[registry.k8s.io/etcdadm/etcd-manager-slim@sha256:8727adc059334e0e3329dbccce116a734da4a090da5e86c8cddccaf333caa666 registry.k8s.io/etcdadm/etcd-manager-slim:v3.0.20230201],SizeBytes:60826655,},ContainerImage{Names:[registry.k8s.io/kube-proxy-arm64:v1.27.0-alpha.3.52_d48b8167f7094b],SizeBytes:60554082,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:64b9ea357325d5db9f8a723dcf503b5a449177b17ac87d69481e126bb724c263 registry.k8s.io/etcd:3.5.1-0],SizeBytes:59553904,},ContainerImage{Names:[registry.k8s.io/kube-scheduler-arm64:v1.27.0-alpha.3.52_d48b8167f7094b],SizeBytes:56310763,},ContainerImage{Names:[registry.k8s.io/kops/kops-controller:1.27.0-alpha.1],SizeBytes:41797411,},ContainerImage{Names:[registry.k8s.io/kops/dns-controller:1.27.0-alpha.1],SizeBytes:40267473,},ContainerImage{Names:[public.ecr.aws/karpenter/controller@sha256:68db4f092cf9cc83f5ef9e2fbc5407c2cb682e81f64dfaa700a7602ede38b1cf public.ecr.aws/karpenter/controller:v0.16.3],SizeBytes:33707751,},ContainerImage{Names:[public.ecr.aws/karpenter/webhook@sha256:96a2d9b06d6bc5127801f358f74b1cf2d289b423a2e9ba40c573c0b14b17dafa public.ecr.aws/karpenter/webhook:v0.16.3],SizeBytes:33307187,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:28932904,},ContainerImage{Names:[quay.io/cilium/operator@sha256:a6d24a006a6b92967ac90786b49bc1ac26e5477cf028cd1186efcfc2466484db quay.io/cilium/operator:v1.12.5],SizeBytes:24567287,},ContainerImage{Names:[registry.k8s.io/provider-aws/cloud-controller-manager@sha256:fdeb61e3e42ecd9cca868d550ebdb88dd6341d9e91fcfa9a37e227dab2ad22cb registry.k8s.io/provider-aws/cloud-controller-manager:v1.26.0],SizeBytes:18345226,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:8502345,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:7685305,},ContainerImage{Names:[registry.k8s.io/kops/kube-apiserver-healthcheck:1.27.0-alpha.1],SizeBytes:4713612,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:253553,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Mar 5 16:34:01.149: INFO: Logging kubelet events for node i-0d6a32f749839192f Mar 5 16:34:01.264: INFO: ... skipping 52 lines ... Mar 5 16:34:01.386: INFO: Container controller ready: true, restart count 0 Mar 5 16:34:01.877: INFO: Latency metrics for node i-0d6a32f749839192f [1mSTEP:[0m Destroying namespace "pods-9130" for this suite. [38;5;243m@ 03/05/23 16:34:01.877[0m [38;5;243m<< Timeline[0m [38;5;9m[FAILED] timed out waiting for the condition[0m [38;5;9mIn [1m[It][0m[38;5;9m at: [1mtest/e2e/common/node/pods.go:808[0m [38;5;243m@ 03/05/23 16:33:15.494[0m [38;5;243m------------------------------[0m [38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;10m•[0m[38;5;14mS[0m[38;5;10m•[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;10m•[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;10m•[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;10m•[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;10m•[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;10m•[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;10m•[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;10m•[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;10m•[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;10m•[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;10m•[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;10m•[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;10m•[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;10m•[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;10m•[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;10m•[0m[38;5;10m•[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;10m•[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;10m•[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;10m•[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;10m•[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;10m•[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;10m•[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;10m•[0m[38;5;10m•[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;10m•[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;10m•[0m[38;5;10m•[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;10m•[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;10m•[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;10m•[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;10m•[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;10m•[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;10m•[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;10m•[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;10m•[0m[38;5;10m•[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;10m•[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;10m•[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m [38;5;243m------------------------------[0m [38;5;9m• [FAILED] [608.099 seconds][0m [0m[sig-node] Security Context [38;5;243mwhen creating containers with AllowPrivilegeEscalation [38;5;9m[1m[It] should not allow privilege escalation when false [LinuxOnly] [NodeConformance] [Conformance][0m [38;5;243mtest/e2e/common/node/security_context.go:609[0m [38;5;243mTimeline >>[0m [1mSTEP:[0m Creating a kubernetes client [38;5;243m@ 03/05/23 16:32:18.117[0m Mar 5 16:32:18.117: INFO: >>> kubeConfig: /root/.kube/config [1mSTEP:[0m Building a namespace api object, basename security-context-test [38;5;243m@ 03/05/23 16:32:18.118[0m [1mSTEP:[0m Waiting for a default service account to be provisioned in namespace [38;5;243m@ 03/05/23 16:32:18.456[0m [1mSTEP:[0m Waiting for kube-root-ca.crt to be provisioned in namespace [38;5;243m@ 03/05/23 16:32:18.678[0m [38;5;9m[FAILED][0m in [It] - test/e2e/framework/pod/pod_client.go:238 [38;5;243m@ 03/05/23 16:37:19.022[0m Mar 5 16:37:19.022: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready Mar 5 16:37:19.135: INFO: Condition Ready of node i-06df568f6e5c59fff is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-05 16:33:20 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-05 16:33:26 +0000 UTC}]. Failure Mar 5 16:37:21.250: INFO: Condition Ready of node i-06df568f6e5c59fff is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-05 16:33:20 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-05 16:33:26 +0000 UTC}]. Failure Mar 5 16:37:23.249: INFO: Condition Ready of node i-06df568f6e5c59fff is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-05 16:33:20 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-05 16:33:26 +0000 UTC}]. Failure Mar 5 16:37:25.249: INFO: Condition Ready of node i-06df568f6e5c59fff is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-05 16:33:20 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-05 16:33:26 +0000 UTC}]. Failure Mar 5 16:37:27.249: INFO: Condition Ready of node i-06df568f6e5c59fff is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-05 16:33:20 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-05 16:33:26 +0000 UTC}]. Failure ... skipping 311 lines ... Mar 5 16:40:25.870: INFO: ebs-csi-controller-79785c665b-ghrzp started at 2023-03-05 16:26:00 +0000 UTC (0+5 container statuses recorded) Mar 5 16:40:25.870: INFO: Container csi-attacher ready: true, restart count 0 Mar 5 16:40:25.870: INFO: Container csi-provisioner ready: true, restart count 0 Mar 5 16:40:25.870: INFO: Container csi-resizer ready: true, restart count 0 Mar 5 16:40:25.870: INFO: Container ebs-plugin ready: true, restart count 0 Mar 5 16:40:25.870: INFO: Container liveness-probe ready: true, restart count 0 Mar 5 16:42:26.101: INFO: Unexpected error: <*errors.errorString | 0xc000f7a020>: { s: "Timed out when waiting for proxy to gather metrics from i-06df568f6e5c59fff", } [38;5;9m[FAILED][0m in [DeferCleanup (Each)] - test/e2e/framework/debug/dump.go:139 [38;5;243m@ 03/05/23 16:42:26.102[0m [1mSTEP:[0m Destroying namespace "security-context-test-2781" for this suite. [38;5;243m@ 03/05/23 16:42:26.102[0m [38;5;243m<< Timeline[0m [38;5;9m[FAILED] wait for pod "alpine-nnp-false-697a1ead-6323-4d1c-a634-dd3b69de84b6" to succeed Expected success, but got an error: <framework.FailureError>: { msg: "Timed out after 300.001s.\nexpected pod to be Succeeded or Failed, got instead:\n <*v1.Pod | 0xc000eea900>: \n metadata:\n creationTimestamp: \"2023-03-05T16:32:18Z\"\n managedFields:\n - apiVersion: v1\n fieldsType: FieldsV1\n fieldsV1:\n f:spec:\n f:containers:\n k:{\"name\":\"alpine-nnp-false-697a1ead-6323-4d1c-a634-dd3b69de84b6\"}:\n .: {}\n f:image: {}\n f:imagePullPolicy: {}\n f:name: {}\n f:resources: {}\n f:securityContext:\n .: {}\n f:allowPrivilegeEscalation: {}\n f:runAsUser: {}\n f:terminationMessagePath: {}\n f:terminationMessagePolicy: {}\n f:dnsPolicy: {}\n f:enableServiceLinks: {}\n f:restartPolicy: {}\n f:schedulerName: {}\n f:securityContext: {}\n f:terminationGracePeriodSeconds: {}\n manager: e2e.test\n operation: Update\n time: \"2023-03-05T16:32:18Z\"\n name: alpine-nnp-false-697a1ead-6323-4d1c-a634-dd3b69de84b6\n namespace: security-context-test-2781\n resourceVersion: \"8650\"\n uid: 01c4082e-a081-4954-aa7e-38d4be729673\n spec:\n containers:\n - image: registry.k8s.io/e2e-test-images/nonewprivs:1.3\n imagePullPolicy: IfNotPresent\n name: alpine-nnp-false-697a1ead-6323-4d1c-a634-dd3b69de84b6\n resources: {}\n securityContext:\n allowPrivilegeEscalation: false\n runAsUser: 1000\n terminationMessagePath: /dev/termination-log\n terminationMessagePolicy: File\n volumeMounts:\n - mountPath: /var/run/secrets/kubernetes.io/serviceaccount\n name: kube-api-access-wmfk6\n readOnly: true\n dnsPolicy: ClusterFirst\n enableServiceLinks: true\n nodeName: i-06df568f6e5c59fff\n preemptionPolicy: PreemptLowerPriority\n priority: 0\n restartPolicy: Never\n schedulerName: default-scheduler\n securityContext: {}\n serviceAccount: default\n serviceAccountName: default\n terminationGracePeriodSeconds: 30\n tolerations:\n - effect: NoExecute\n key: node.kubernetes.io/not-ready\n operator: Exists\n tolerationSeconds: 300\n - effect: NoExecute\n key: node.kubernetes.io/unreachable\n operator: Exists\n tolerationSeconds: 300\n volumes:\n - name: kube-api-access-wmfk6\n projected:\n defaultMode: 420\n sources:\n - serviceAccountToken:\n expirationSeconds: 3607\n path: token\n - configMap:\n items:\n - key: ca.crt\n path: ca.crt\n name: kube-root-ca.crt\n - downwardAPI:\n items:\n - fieldRef:\n apiVersion: v1\n fieldPath: metadata.namespace\n path: namespace\n status:\n conditions:\n - lastProbeTime: null\n lastTransitionTime: \"2023-03-05T16:32:18Z\"\n status: \"True\"\n type: PodScheduled\n phase: Pending\n qosClass: BestEffort", fullStackTrace: "k8s.io/kubernetes/test/e2e/framework/pod.WaitForPodCondition({0x7f0b482a2388, 0xc000ce0200}, {0x8252218?, 0xc0014e2000?}, {0xc0036bbdc0, 0x1a}, {0xc004b55800, 0x35}, {0xc004745da0, 0x13}, ...)\n\ttest/e2e/framework/pod/wait.go:228 +0x25f\nk8s.io/kubernetes/test/e2e/framework/pod.(*PodClient).WaitForSuccess(0xc000efc948?, {0x7f0b482a2388, 0xc000ce0200}, {0xc004b55800, 0x35}, 0x1?)\n\ttest/e2e/framework/pod/pod_client.go:227 +0x165\nk8s.io/kubernetes/test/e2e/common/node.glob..func21.7.2({0x7f0b482a2388, 0xc000ce0200}, {0xc004b55800, 0x35}, {0xc004745cf8, 0x13}, 0xc004b12ab8?, 0x2680f2d?)\n\ttest/e2e/common/node/security_context.go:581 +0xa6\nk8s.io/kubernetes/test/e2e/common/node.glob..func21.7.4({0x7f0b482a2388, 0xc000ce0200})\n\ttest/e2e/common/node/security_context.go:612 +0xf7", } Timed out after 300.001s. expected pod to be Succeeded or Failed, got instead: <*v1.Pod | 0xc000eea900>: metadata: creationTimestamp: "2023-03-05T16:32:18Z" managedFields: - apiVersion: v1 fieldsType: FieldsV1 ... skipping 90 lines ... [38;5;9mIn [1m[It][0m[38;5;9m at: [1mtest/e2e/framework/pod/pod_client.go:238[0m [38;5;243m@ 03/05/23 16:37:19.022[0m There were [1m[38;5;9madditional failures[0m detected. To view them in detail run [1mginkgo -vv[0m [38;5;243m------------------------------[0m [38;5;14mS[0m[38;5;14mS[0m[38;5;10m•[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m [38;5;243m------------------------------[0m [38;5;9m• [FAILED] [605.819 seconds][0m [0m[sig-network] Networking [38;5;243mGranular Checks: Services [38;5;9m[1m[It] should function for multiple endpoint-Services with same selector[0m [38;5;243mtest/e2e/network/networking.go:282[0m [38;5;243mTimeline >>[0m [1mSTEP:[0m Creating a kubernetes client [38;5;243m@ 03/05/23 16:32:55.048[0m Mar 5 16:32:55.048: INFO: >>> kubeConfig: /root/.kube/config [1mSTEP:[0m Building a namespace api object, basename nettest [38;5;243m@ 03/05/23 16:32:55.049[0m [1mSTEP:[0m Waiting for a default service account to be provisioned in namespace [38;5;243m@ 03/05/23 16:32:55.387[0m [1mSTEP:[0m Waiting for kube-root-ca.crt to be provisioned in namespace [38;5;243m@ 03/05/23 16:32:55.61[0m [1mSTEP:[0m Performing setup for networking test in namespace nettest-2035 [38;5;243m@ 03/05/23 16:32:55.832[0m [1mSTEP:[0m creating a selector [38;5;243m@ 03/05/23 16:32:55.832[0m [1mSTEP:[0m Creating the service pods in kubernetes [38;5;243m@ 03/05/23 16:32:55.832[0m Mar 5 16:32:55.832: INFO: Waiting up to 10m0s for all (but 0) nodes to be schedulable Mar 5 16:38:19.761: INFO: Failed inside E2E framework: k8s.io/kubernetes/test/e2e/framework/pod.WaitForPodCondition({0x7ffb60583ef0, 0xc00152c340}, {0x8252218?, 0xc004aa16c0?}, {0xc002517be0, 0xc}, {0xc002c87af3, 0xb}, {0x7850c43, 0x11}, ...) test/e2e/framework/pod/wait.go:228 +0x25f k8s.io/kubernetes/test/e2e/framework/pod.WaitTimeoutForPodReadyInNamespace({0x7ffb60583ef0?, 0xc00152c340?}, {0x8252218?, 0xc004aa16c0?}, {0xc002c87af3?, 0x0?}, {0xc002517be0?, 0x7ffb60583ef0?}, 0xc00152c340?) test/e2e/framework/pod/wait.go:492 +0x75 k8s.io/kubernetes/test/e2e/framework/network.(*NetworkingTestConfig).createNetProxyPods(0xc000995340, {0x7ffb60583ef0, 0xc00152c340}, {0x78291ae, 0x9}, 0xc002b6c0f0) test/e2e/framework/network/utils.go:876 +0x5ce ... skipping 2 lines ... k8s.io/kubernetes/test/e2e/framework/network.(*NetworkingTestConfig).setup(0xc000995340, {0x7ffb60583ef0, 0xc00152c340}, 0x1?) test/e2e/framework/network/utils.go:784 +0x4e k8s.io/kubernetes/test/e2e/framework/network.NewNetworkingTestConfig({0x7ffb60583ef0, 0xc00152c340}, 0xc000e19e00, {0x0, 0x0, 0x0?}) test/e2e/framework/network/utils.go:135 +0x145 k8s.io/kubernetes/test/e2e/network.glob..func22.6.10({0x7ffb60583ef0, 0xc00152c340}) test/e2e/network/networking.go:283 +0x4e [38;5;9m[FAILED][0m in [It] - test/e2e/framework/network/utils.go:876 [38;5;243m@ 03/05/23 16:38:19.762[0m Mar 5 16:38:19.762: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready Mar 5 16:38:19.874: INFO: Condition Ready of node i-06df568f6e5c59fff is false, but Node is tainted by NodeController with [{node.kubernetes.io/not-ready NoSchedule 2023-03-05 16:37:27 +0000 UTC} {node.kubernetes.io/not-ready NoExecute 2023-03-05 16:37:31 +0000 UTC}]. Failure Mar 5 16:38:21.988: INFO: Condition Ready of node i-06df568f6e5c59fff is false, but Node is tainted by NodeController with [{node.kubernetes.io/not-ready NoSchedule 2023-03-05 16:37:27 +0000 UTC} {node.kubernetes.io/not-ready NoExecute 2023-03-05 16:37:31 +0000 UTC}]. Failure Mar 5 16:38:23.988: INFO: Condition Ready of node i-06df568f6e5c59fff is false, but Node is tainted by NodeController with [{node.kubernetes.io/not-ready NoSchedule 2023-03-05 16:37:27 +0000 UTC} {node.kubernetes.io/not-ready NoExecute 2023-03-05 16:37:31 +0000 UTC}]. Failure Mar 5 16:38:25.988: INFO: Condition Ready of node i-06df568f6e5c59fff is false, but Node is tainted by NodeController with [{node.kubernetes.io/not-ready NoSchedule 2023-03-05 16:37:27 +0000 UTC} {node.kubernetes.io/not-ready NoExecute 2023-03-05 16:37:31 +0000 UTC}]. Failure Mar 5 16:38:27.988: INFO: Condition Ready of node i-06df568f6e5c59fff is false, but Node is tainted by NodeController with [{node.kubernetes.io/not-ready NoSchedule 2023-03-05 16:37:27 +0000 UTC} {node.kubernetes.io/not-ready NoExecute 2023-03-05 16:37:31 +0000 UTC}]. Failure ... skipping 266 lines ... Mar 5 16:41:38.739: INFO: webhook-to-be-mutated started at 2023-03-05 16:31:01 +0000 UTC (1+1 container statuses recorded) Mar 5 16:41:38.739: INFO: Init container webhook-added-init-container ready: false, restart count 0 Mar 5 16:41:38.739: INFO: Container example ready: false, restart count 0 Mar 5 16:41:38.739: INFO: pvc-volume-tester-nc7w4 started at 2023-03-05 16:31:17 +0000 UTC (0+1 container statuses recorded) Mar 5 16:41:38.739: INFO: Container volume-tester ready: true, restart count 0 Mar 5 16:41:38.739: INFO: csi-hostpathplugin-0 started at <nil> (0+0 container statuses recorded) Mar 5 16:43:00.752: INFO: Unexpected error: <*errors.StatusError | 0xc001284000>: { ErrStatus: code: 503 details: causes: - message: unknown ... skipping 2 lines ... name: i-06df568f6e5c59fff:10250 message: the server is currently unable to handle the request (get nodes i-06df568f6e5c59fff:10250) metadata: {} reason: ServiceUnavailable status: Failure, } [38;5;9m[FAILED][0m in [DeferCleanup (Each)] - test/e2e/framework/debug/dump.go:139 [38;5;243m@ 03/05/23 16:43:00.753[0m [1mSTEP:[0m Destroying namespace "nettest-2035" for this suite. [38;5;243m@ 03/05/23 16:43:00.753[0m [38;5;243m<< Timeline[0m [38;5;9m[FAILED] Timed out after 300.001s. expected pod to be running and ready, got instead: <*v1.Pod | 0xc0043e0d80>: metadata: creationTimestamp: "2023-03-05T16:32:56Z" labels: selector-fcb64c90-7002-425c-a15e-fa86d8a8d550: "true" ... skipping 161 lines ... [38;5;9mIn [1m[It][0m[38;5;9m at: [1mtest/e2e/framework/network/utils.go:876[0m [38;5;243m@ 03/05/23 16:38:19.762[0m There were [1m[38;5;9madditional failures[0m detected. To view them in detail run [1mginkgo -vv[0m [38;5;243m------------------------------[0m [38;5;14mS[0m[38;5;14mS[0m [38;5;243m------------------------------[0m [38;5;9m• [FAILED] [639.727 seconds][0m [0m[sig-storage] Projected configMap [38;5;9m[1m[It] should be consumable from pods in volume with defaultMode set [LinuxOnly] [NodeConformance] [Conformance][0m [38;5;243mtest/e2e/common/storage/projected_configmap.go:57[0m [38;5;243mTimeline >>[0m [1mSTEP:[0m Creating a kubernetes client [38;5;243m@ 03/05/23 16:32:21.143[0m Mar 5 16:32:21.143: INFO: >>> kubeConfig: /root/.kube/config ... skipping 3 lines ... [1mSTEP:[0m Creating configMap with name projected-configmap-test-volume-2ff213b3-30c8-4882-ad84-a92b706995ec [38;5;243m@ 03/05/23 16:32:21.925[0m [1mSTEP:[0m Creating a pod to test consume configMaps [38;5;243m@ 03/05/23 16:32:22.041[0m Mar 5 16:37:36.801: INFO: Output of node "i-06df568f6e5c59fff" pod "pod-projected-configmaps-72d4e2fc-6570-45ff-87cb-7154336523df" container "agnhost-container": mode of file "/etc/projected-configmap-volume/data-1": -r-------- content of file "/etc/projected-configmap-volume/data-1": value-1 [1mSTEP:[0m delete the pod [38;5;243m@ 03/05/23 16:37:36.801[0m Mar 5 16:37:37.031: INFO: Unexpected error: <*errors.errorString | 0xc001175dd0>: { s: "expected pod \"pod-projected-configmaps-72d4e2fc-6570-45ff-87cb-7154336523df\" success: Timed out after 300.113s.\nexpected pod to be Succeeded or Failed, got instead:\n <*v1.Pod | 0xc002384d80>: \n metadata:\n creationTimestamp: \"2023-03-05T16:32:22Z\"\n managedFields:\n - apiVersion: v1\n fieldsType: FieldsV1\n fieldsV1:\n f:spec:\n f:containers:\n k:{\"name\":\"agnhost-container\"}:\n .: {}\n f:args: {}\n f:image: {}\n f:imagePullPolicy: {}\n f:name: {}\n f:resources: {}\n f:securityContext: {}\n f:terminationMessagePath: {}\n f:terminationMessagePolicy: {}\n f:volumeMounts:\n .: {}\n k:{\"mountPath\":\"/etc/projected-configmap-volume\"}:\n .: {}\n f:mountPath: {}\n f:name: {}\n f:readOnly: {}\n f:dnsPolicy: {}\n f:enableServiceLinks: {}\n f:restartPolicy: {}\n f:schedulerName: {}\n f:securityContext: {}\n f:terminationGracePeriodSeconds: {}\n f:volumes:\n .: {}\n k:{\"name\":\"projected-configmap-volume\"}:\n .: {}\n f:name: {}\n f:projected:\n .: {}\n f:defaultMode: {}\n f:sources: {}\n manager: e2e.test\n operation: Update\n time: \"2023-03-05T16:32:22Z\"\n name: pod-projected-configmaps-72d4e2fc-6570-45ff-87cb-7154336523df\n namespace: projected-9688\n resourceVersion: \"8732\"\n uid: 6f0ee25e-31ef-449d-bd63-872cc8f773a0\n spec:\n containers:\n - args:\n - mounttest\n - --file_content=/etc/projected-configmap-volume/data-1\n - --file_mode=/etc/projected-configmap-volume/data-1\n image: registry.k8s.io/e2e-test-images/agnhost:2.43\n imagePullPolicy: IfNotPresent\n name: agnhost-container\n resources: {}\n securityContext: {}\n terminationMessagePath: /dev/termination-log\n terminationMessagePolicy: File\n volumeMounts:\n - mountPath: /etc/projected-configmap-volume\n name: projected-configmap-volume\n readOnly: true\n - mountPath: /var/run/secrets/kubernetes.io/serviceaccount\n name: kube-api-access-ncbvw\n readOnly: true\n dnsPolicy: ClusterFirst\n enableServiceLinks: true\n nodeName: i-06df568f6e5c59fff\n preemptionPolicy: PreemptLowerPriority\n priority: 0\n restartPolicy: Never\n schedulerName: default-scheduler\n securityContext: {}\n serviceAccount: default\n serviceAccountName: default\n terminationGracePeriodSeconds: 0\n tolerations:\n - effect: NoExecute\n key: node.kubernetes.io/not-ready\n operator: Exists\n tolerationSeconds: 300\n - effect: NoExecute\n key: node.kubernetes.io/unreachable\n operator: Exists\n tolerationSeconds: 300\n volumes:\n - name: projected-configmap-volume\n projected:\n defaultMode: 256\n sources:\n - configMap:\n name: projected-configmap-test-volume-2ff213b3-30c8-4882-ad84-a92b706995ec\n - name: kube-api-access-ncbvw\n projected:\n defaultMode: 420\n sources:\n - serviceAccountToken:\n expirationSeconds: 3607\n path: token\n - configMap:\n items:\n - key: ca.crt\n path: ca.crt\n name: kube-root-ca.crt\n - downwardAPI:\n items:\n - fieldRef:\n apiVersion: v1\n fieldPath: metadata.namespace\n path: namespace\n status:\n conditions:\n - lastProbeTime: null\n lastTransitionTime: \"2023-03-05T16:32:22Z\"\n status: \"True\"\n type: PodScheduled\n phase: Pending\n qosClass: BestEffort", } [38;5;9m[FAILED][0m in [It] - test/e2e/framework/pod/output/output.go:238 [38;5;243m@ 03/05/23 16:37:37.031[0m Mar 5 16:37:37.031: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready Mar 5 16:37:37.144: INFO: Condition Ready of node i-06df568f6e5c59fff is false, but Node is tainted by NodeController with [{node.kubernetes.io/not-ready NoSchedule 2023-03-05 16:37:27 +0000 UTC} {node.kubernetes.io/not-ready NoExecute 2023-03-05 16:37:31 +0000 UTC}]. Failure Mar 5 16:37:39.258: INFO: Condition Ready of node i-06df568f6e5c59fff is false, but Node is tainted by NodeController with [{node.kubernetes.io/not-ready NoSchedule 2023-03-05 16:37:27 +0000 UTC} {node.kubernetes.io/not-ready NoExecute 2023-03-05 16:37:31 +0000 UTC}]. Failure Mar 5 16:37:41.258: INFO: Condition Ready of node i-06df568f6e5c59fff is false, but Node is tainted by NodeController with [{node.kubernetes.io/not-ready NoSchedule 2023-03-05 16:37:27 +0000 UTC} {node.kubernetes.io/not-ready NoExecute 2023-03-05 16:37:31 +0000 UTC}]. Failure Mar 5 16:37:43.258: INFO: Condition Ready of node i-06df568f6e5c59fff is false, but Node is tainted by NodeController with [{node.kubernetes.io/not-ready NoSchedule 2023-03-05 16:37:27 +0000 UTC} {node.kubernetes.io/not-ready NoExecute 2023-03-05 16:37:31 +0000 UTC}]. Failure Mar 5 16:37:45.257: INFO: Condition Ready of node i-06df568f6e5c59fff is false, but Node is tainted by NodeController with [{node.kubernetes.io/not-ready NoSchedule 2023-03-05 16:37:27 +0000 UTC} {node.kubernetes.io/not-ready NoExecute 2023-03-05 16:37:31 +0000 UTC}]. Failure ... skipping 78 lines ... Mar 5 16:40:23.258: INFO: Condition Ready of node i-06df568f6e5c59fff is false, but Node is tainted by NodeController with [{node.kubernetes.io/not-ready NoSchedule 2023-03-05 16:37:27 +0000 UTC} {node.kubernetes.io/not-ready NoExecute 2023-03-05 16:37:31 +0000 UTC}]. Failure Mar 5 16:40:25.258: INFO: Condition Ready of node i-06df568f6e5c59fff is false, but Node is tainted by NodeController with [{node.kubernetes.io/not-ready NoSchedule 2023-03-05 16:37:27 +0000 UTC} {node.kubernetes.io/not-ready NoExecute 2023-03-05 16:37:31 +0000 UTC}]. Failure [1mSTEP:[0m dump namespace information after failure [38;5;243m@ 03/05/23 16:40:27.258[0m [1mSTEP:[0m Collecting events from namespace "projected-9688". [38;5;243m@ 03/05/23 16:40:27.258[0m [1mSTEP:[0m Found 6 events. [38;5;243m@ 03/05/23 16:40:27.37[0m Mar 5 16:40:27.370: INFO: At 2023-03-05 16:32:22 +0000 UTC - event for pod-projected-configmaps-72d4e2fc-6570-45ff-87cb-7154336523df: {default-scheduler } Scheduled: Successfully assigned projected-9688/pod-projected-configmaps-72d4e2fc-6570-45ff-87cb-7154336523df to i-06df568f6e5c59fff Mar 5 16:40:27.370: INFO: At 2023-03-05 16:32:23 +0000 UTC - event for pod-projected-configmaps-72d4e2fc-6570-45ff-87cb-7154336523df: {kubelet i-06df568f6e5c59fff} FailedMount: MountVolume.SetUp failed for volume "projected-configmap-volume" : failed to sync configmap cache: timed out waiting for the condition Mar 5 16:40:27.370: INFO: At 2023-03-05 16:32:24 +0000 UTC - event for pod-projected-configmaps-72d4e2fc-6570-45ff-87cb-7154336523df: {kubelet i-06df568f6e5c59fff} Pulled: Container image "registry.k8s.io/e2e-test-images/agnhost:2.43" already present on machine Mar 5 16:40:27.370: INFO: At 2023-03-05 16:32:24 +0000 UTC - event for pod-projected-configmaps-72d4e2fc-6570-45ff-87cb-7154336523df: {kubelet i-06df568f6e5c59fff} Created: Created container agnhost-container Mar 5 16:40:27.370: INFO: At 2023-03-05 16:32:24 +0000 UTC - event for pod-projected-configmaps-72d4e2fc-6570-45ff-87cb-7154336523df: {kubelet i-06df568f6e5c59fff} Started: Started container agnhost-container Mar 5 16:40:27.370: INFO: At 2023-03-05 16:37:36 +0000 UTC - event for pod-projected-configmaps-72d4e2fc-6570-45ff-87cb-7154336523df: {taint-controller } TaintManagerEviction: Cancelling deletion of Pod projected-9688/pod-projected-configmaps-72d4e2fc-6570-45ff-87cb-7154336523df Mar 5 16:40:27.481: INFO: POD NODE PHASE GRACE CONDITIONS Mar 5 16:40:27.481: INFO: ... skipping 194 lines ... Mar 5 16:41:39.043: INFO: simpletest.rc-mrdlj started at <nil> (0+0 container statuses recorded) Mar 5 16:41:39.043: INFO: simpletest.rc-b9nhg started at <nil> (0+0 container statuses recorded) Mar 5 16:41:39.043: INFO: slave started at 2023-03-05 16:31:36 +0000 UTC (0+1 container statuses recorded) Mar 5 16:41:39.043: INFO: Container cntr ready: true, restart count 0 Mar 5 16:41:39.043: INFO: up-down-2-dfrc7 started at 2023-03-05 16:32:02 +0000 UTC (0+1 container statuses recorded) Mar 5 16:41:39.043: INFO: Container up-down-2 ready: true, restart count 0 Mar 5 16:43:00.752: INFO: Unexpected error: <*errors.StatusError | 0xc00105cf00>: { ErrStatus: code: 503 details: causes: - message: unknown ... skipping 2 lines ... name: i-06df568f6e5c59fff:10250 message: the server is currently unable to handle the request (get nodes i-06df568f6e5c59fff:10250) metadata: {} reason: ServiceUnavailable status: Failure, } [38;5;9m[FAILED][0m in [DeferCleanup (Each)] - test/e2e/framework/debug/dump.go:139 [38;5;243m@ 03/05/23 16:43:00.753[0m [1mSTEP:[0m Destroying namespace "projected-9688" for this suite. [38;5;243m@ 03/05/23 16:43:00.753[0m [38;5;243m<< Timeline[0m [38;5;9m[FAILED] expected pod "pod-projected-configmaps-72d4e2fc-6570-45ff-87cb-7154336523df" success: Timed out after 300.113s. expected pod to be Succeeded or Failed, got instead: <*v1.Pod | 0xc002384d80>: metadata: creationTimestamp: "2023-03-05T16:32:22Z" managedFields: - apiVersion: v1 fieldsType: FieldsV1 ... skipping 115 lines ... [38;5;9mIn [1m[It][0m[38;5;9m at: [1mtest/e2e/framework/pod/output/output.go:238[0m [38;5;243m@ 03/05/23 16:37:37.031[0m There were [1m[38;5;9madditional failures[0m detected. To view them in detail run [1mginkgo -vv[0m [38;5;243m------------------------------[0m [38;5;14mS[0m [38;5;243m------------------------------[0m [38;5;9m• [FAILED] [669.268 seconds][0m [0m[sig-network] Services [38;5;9m[1m[It] should be able to up and down services[0m [38;5;243mtest/e2e/network/service.go:1116[0m [38;5;243mTimeline >>[0m [1mSTEP:[0m Creating a kubernetes client [38;5;243m@ 03/05/23 16:31:51.602[0m Mar 5 16:31:51.602: INFO: >>> kubeConfig: /root/.kube/config ... skipping 16 lines ... I0305 16:32:11.297412 7177 runners.go:194] up-down-2 Pods: 3 out of 3 created, 0 running, 3 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0305 16:32:14.298380 7177 runners.go:194] up-down-2 Pods: 3 out of 3 created, 2 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0305 16:32:17.299270 7177 runners.go:194] up-down-2 Pods: 3 out of 3 created, 3 running, 0 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady [1mSTEP:[0m verifying service up-down-1 is up [38;5;243m@ 03/05/23 16:32:17.412[0m Mar 5 16:32:17.412: INFO: Creating new host exec pod Mar 5 16:32:26.103: INFO: Creating new exec pod Mar 5 16:37:26.228: INFO: Unexpected error occurred: Timed out after 300.000s. Expected Pod to be in <v1.PodPhase>: "Running" Got instead: <*v1.Pod | 0xc000591200>: metadata: creationTimestamp: "2023-03-05T16:32:26Z" generateName: verify-service-up-exec-pod- ... skipping 87 lines ... - lastProbeTime: null lastTransitionTime: "2023-03-05T16:32:26Z" status: "True" type: PodScheduled phase: Pending qosClass: BestEffort [38;5;9m[FAILED][0m in [It] - test/e2e/framework/pod/resource.go:369 [38;5;243m@ 03/05/23 16:37:26.228[0m Mar 5 16:37:26.228: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready Mar 5 16:37:26.343: INFO: Condition Ready of node i-06df568f6e5c59fff is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-05 16:33:20 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-05 16:33:26 +0000 UTC}]. Failure Mar 5 16:37:28.457: INFO: Condition Ready of node i-06df568f6e5c59fff is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoExecute 2023-03-05 16:33:26 +0000 UTC} {node.kubernetes.io/not-ready NoSchedule 2023-03-05 16:37:27 +0000 UTC}]. Failure Mar 5 16:37:30.457: INFO: Condition Ready of node i-06df568f6e5c59fff is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoExecute 2023-03-05 16:33:26 +0000 UTC} {node.kubernetes.io/not-ready NoSchedule 2023-03-05 16:37:27 +0000 UTC}]. Failure Mar 5 16:37:32.458: INFO: Condition Ready of node i-06df568f6e5c59fff is false, but Node is tainted by NodeController with [{node.kubernetes.io/not-ready NoSchedule 2023-03-05 16:37:27 +0000 UTC} {node.kubernetes.io/not-ready NoExecute 2023-03-05 16:37:31 +0000 UTC}]. Failure Mar 5 16:37:34.458: INFO: Condition Ready of node i-06df568f6e5c59fff is false, but Node is tainted by NodeController with [{node.kubernetes.io/not-ready NoSchedule 2023-03-05 16:37:27 +0000 UTC} {node.kubernetes.io/not-ready NoExecute 2023-03-05 16:37:31 +0000 UTC}]. Failure ... skipping 133 lines ... Mar 5 16:40:26.687: INFO: At 2023-03-05 16:38:26 +0000 UTC - event for verify-service-up-exec-pod-cxcs2: {taint-controller } TaintManagerEviction: Marking for deletion Pod services-4797/verify-service-up-exec-pod-cxcs2 Mar 5 16:40:26.687: INFO: At 2023-03-05 16:38:29 +0000 UTC - event for up-down-1: {replication-controller } SuccessfulCreate: Created pod: up-down-1-t9k8s Mar 5 16:40:26.687: INFO: At 2023-03-05 16:38:29 +0000 UTC - event for up-down-1-t9k8s: {default-scheduler } Scheduled: Successfully assigned services-4797/up-down-1-t9k8s to i-056ad61525499081e Mar 5 16:40:26.687: INFO: At 2023-03-05 16:38:29 +0000 UTC - event for up-down-1-t9k8s: {kubelet i-056ad61525499081e} Started: Started container up-down-1 Mar 5 16:40:26.687: INFO: At 2023-03-05 16:38:29 +0000 UTC - event for up-down-1-t9k8s: {kubelet i-056ad61525499081e} Pulled: Container image "registry.k8s.io/e2e-test-images/agnhost:2.43" already present on machine Mar 5 16:40:26.687: INFO: At 2023-03-05 16:38:29 +0000 UTC - event for up-down-1-t9k8s: {kubelet i-056ad61525499081e} Created: Created container up-down-1 Mar 5 16:40:26.687: INFO: At 2023-03-05 16:38:29 +0000 UTC - event for up-down-2: {endpoint-controller } FailedToUpdateEndpoint: Failed to update endpoint services-4797/up-down-2: Operation cannot be fulfilled on endpoints "up-down-2": the object has been modified; please apply your changes to the latest version and try again Mar 5 16:40:26.687: INFO: At 2023-03-05 16:38:29 +0000 UTC - event for up-down-2: {replication-controller } SuccessfulCreate: Created pod: up-down-2-cgfqv Mar 5 16:40:26.687: INFO: At 2023-03-05 16:38:29 +0000 UTC - event for up-down-2: {replication-controller } SuccessfulCreate: Created pod: up-down-2-fljd6 Mar 5 16:40:26.687: INFO: At 2023-03-05 16:38:29 +0000 UTC - event for up-down-2-cgfqv: {default-scheduler } Scheduled: Successfully assigned services-4797/up-down-2-cgfqv to i-056ad61525499081e Mar 5 16:40:26.687: INFO: At 2023-03-05 16:38:29 +0000 UTC - event for up-down-2-fljd6: {default-scheduler } Scheduled: Successfully assigned services-4797/up-down-2-fljd6 to i-056ad61525499081e Mar 5 16:40:26.687: INFO: At 2023-03-05 16:38:30 +0000 UTC - event for up-down-1: {replication-controller } SuccessfulCreate: Created pod: up-down-1-rrkmh Mar 5 16:40:26.687: INFO: At 2023-03-05 16:38:30 +0000 UTC - event for up-down-1-rrkmh: {default-scheduler } Scheduled: Successfully assigned services-4797/up-down-1-rrkmh to i-056ad61525499081e ... skipping 231 lines ... Mar 5 16:41:39.028: INFO: simpletest.rc-f7k7c started at 2023-03-05 16:32:25 +0000 UTC (0+1 container statuses recorded) Mar 5 16:41:39.028: INFO: Container nginx ready: false, restart count 0 Mar 5 16:41:39.028: INFO: simpletest.rc-bqr98 started at <nil> (0+0 container statuses recorded) Mar 5 16:41:39.028: INFO: simpletest.rc-gv869 started at <nil> (0+0 container statuses recorded) Mar 5 16:41:39.028: INFO: simpletest.rc-wdfv4 started at <nil> (0+0 container statuses recorded) Mar 5 16:41:39.028: INFO: netserver-1 started at <nil> (0+0 container statuses recorded) Mar 5 16:43:00.752: INFO: Unexpected error: <*errors.StatusError | 0xc001037a40>: { ErrStatus: code: 503 details: causes: - message: unknown ... skipping 2 lines ... name: i-06df568f6e5c59fff:10250 message: the server is currently unable to handle the request (get nodes i-06df568f6e5c59fff:10250) metadata: {} reason: ServiceUnavailable status: Failure, } [38;5;9m[FAILED][0m in [DeferCleanup (Each)] - test/e2e/framework/debug/dump.go:139 [38;5;243m@ 03/05/23 16:43:00.753[0m [1mSTEP:[0m Destroying namespace "services-4797" for this suite. [38;5;243m@ 03/05/23 16:43:00.753[0m [38;5;243m<< Timeline[0m [38;5;9m[FAILED] failed to create new exec pod in namespace: services-4797 Unexpected error: <framework.FailureError>: { msg: "Timed out after 300.000s.\nExpected Pod to be in <v1.PodPhase>: \"Running\"\nGot instead:\n <*v1.Pod | 0xc000591200>: \n metadata:\n creationTimestamp: \"2023-03-05T16:32:26Z\"\n generateName: verify-service-up-exec-pod-\n managedFields:\n - apiVersion: v1\n fieldsType: FieldsV1\n fieldsV1:\n f:metadata:\n f:generateName: {}\n f:spec:\n f:containers:\n k:{\"name\":\"agnhost-container\"}:\n .: {}\n f:args: {}\n f:image: {}\n f:imagePullPolicy: {}\n f:name: {}\n f:resources: {}\n f:securityContext: {}\n f:terminationMessagePath: {}\n f:terminationMessagePolicy: {}\n f:dnsPolicy: {}\n f:enableServiceLinks: {}\n f:restartPolicy: {}\n f:schedulerName: {}\n f:securityContext: {}\n f:terminationGracePeriodSeconds: {}\n manager: e2e.test\n operation: Update\n time: \"2023-03-05T16:32:26Z\"\n name: verify-service-up-exec-pod-cxcs2\n namespace: services-4797\n resourceVersion: \"8923\"\n uid: bad3c7c0-73ee-42f9-92c5-f4739988c24c\n spec:\n containers:\n - args:\n - pause\n image: registry.k8s.io/e2e-test-images/agnhost:2.43\n imagePullPolicy: IfNotPresent\n name: agnhost-container\n resources: {}\n securityContext: {}\n terminationMessagePath: /dev/termination-log\n terminationMessagePolicy: File\n volumeMounts:\n - mountPath: /var/run/secrets/kubernetes.io/serviceaccount\n name: kube-api-access-zzzcq\n readOnly: true\n dnsPolicy: ClusterFirst\n enableServiceLinks: true\n nodeName: i-06df568f6e5c59fff\n preemptionPolicy: PreemptLowerPriority\n priority: 0\n restartPolicy: Always\n schedulerName: default-scheduler\n securityContext: {}\n serviceAccount: default\n serviceAccountName: default\n terminationGracePeriodSeconds: 0\n tolerations:\n - effect: NoExecute\n key: node.kubernetes.io/not-ready\n operator: Exists\n tolerationSeconds: 300\n - effect: NoExecute\n key: node.kubernetes.io/unreachable\n operator: Exists\n tolerationSeconds: 300\n volumes:\n - name: kube-api-access-zzzcq\n projected:\n defaultMode: 420\n sources:\n - serviceAccountToken:\n expirationSeconds: 3607\n path: token\n - configMap:\n items:\n - key: ca.crt\n path: ca.crt\n name: kube-root-ca.crt\n - downwardAPI:\n items:\n - fieldRef:\n apiVersion: v1\n fieldPath: metadata.namespace\n path: namespace\n status:\n conditions:\n - lastProbeTime: null\n lastTransitionTime: \"2023-03-05T16:32:26Z\"\n status: \"True\"\n type: PodScheduled\n phase: Pending\n qosClass: BestEffort", fullStackTrace: "k8s.io/kubernetes/test/e2e/framework/pod.WaitTimeoutForPodRunningInNamespace({0x7efeedeff2d8, 0xc001114100}, {0x8252218?, 0xc004a37860?}, {0xc00466a380, 0x20}, {0xc00189f213, 0xd}, 0x0?)\n\ttest/e2e/framework/pod/wait.go:460 +0x1a4\nk8s.io/kubernetes/test/e2e/framework/pod.WaitForPodNameRunningInNamespace(...)\n\ttest/e2e/framework/pod/wait.go:444\nk8s.io/kubernetes/test/e2e/framework/pod.CreateExecPodOrFail({0x7efeedeff2d8, 0xc001114100}, {0x8252218?, 0xc004a37860}, {0xc005ab5720, 0xd}, {0x7897eb7, 0x1b}, 0x0)\n\ttest/e2e/framework/pod/resource.go:368 +0x2a6\nk8s.io/kubernetes/test/e2e/network.verifyServeHostnameServiceUp({0x7efeedeff2d8, 0xc001114100}, {0x8252218, 0xc004a37860}, {0xc005ab5720, 0xd}, {0xc004a22e70?, 0x3, 0x3}, {0xc002254890, ...}, ...)\n\ttest/e2e/network/service.go:335 +0xf5\nk8s.io/kubernetes/test/e2e/network.glob..func26.8({0x7efeedeff2d8, 0xc001114100})\n\ttest/e2e/network/service.go:1132 +0x445", } Timed out after 300.000s. Expected Pod to be in <v1.PodPhase>: "Running" ... skipping 97 lines ... qosClass: BestEffort occurred[0m [38;5;9mIn [1m[It][0m[38;5;9m at: [1mtest/e2e/framework/pod/resource.go:369[0m [38;5;243m@ 03/05/23 16:37:26.228[0m There were [1m[38;5;9madditional failures[0m detected. To view them in detail run [1mginkgo -vv[0m [38;5;243m------------------------------[0m [38;5;9m• [FAILED] [606.073 seconds][0m [0m[sig-network] DNS [38;5;9m[1m[It] should support configurable pod DNS nameservers [Conformance][0m [38;5;243mtest/e2e/network/dns.go:407[0m [38;5;243mTimeline >>[0m [1mSTEP:[0m Creating a kubernetes client [38;5;243m@ 03/05/23 16:32:54.798[0m Mar 5 16:32:54.798: INFO: >>> kubeConfig: /root/.kube/config [1mSTEP:[0m Building a namespace api object, basename dns [38;5;243m@ 03/05/23 16:32:54.799[0m [1mSTEP:[0m Waiting for a default service account to be provisioned in namespace [38;5;243m@ 03/05/23 16:32:55.137[0m [1mSTEP:[0m Waiting for kube-root-ca.crt to be provisioned in namespace [38;5;243m@ 03/05/23 16:32:55.359[0m [1mSTEP:[0m Creating a pod with dnsPolicy=None and customized dnsConfig... [38;5;243m@ 03/05/23 16:32:55.581[0m Mar 5 16:32:55.697: INFO: Created pod &Pod{ObjectMeta:{test-dns-nameservers dns-4814 22e212b2-29aa-48be-b882-731247492e14 9728 0 2023-03-05 16:32:55 +0000 UTC <nil> <nil> map[] map[] [] [] [{e2e.test Update v1 2023-03-05 16:32:55 +0000 UTC FieldsV1 {"f:spec":{"f:containers":{"k:{\"name\":\"agnhost-container\"}":{".":{},"f:args":{},"f:image":{},"f:imagePullPolicy":{},"f:name":{},"f:resources":{},"f:securityContext":{},"f:terminationMessagePath":{},"f:terminationMessagePolicy":{}}},"f:dnsConfig":{".":{},"f:nameservers":{},"f:searches":{}},"f:dnsPolicy":{},"f:enableServiceLinks":{},"f:restartPolicy":{},"f:schedulerName":{},"f:securityContext":{},"f:terminationGracePeriodSeconds":{}}} }]},Spec:PodSpec{Volumes:[]Volume{Volume{Name:kube-api-access-nnlxb,VolumeSource:VolumeSource{HostPath:nil,EmptyDir:nil,GCEPersistentDisk:nil,AWSElasticBlockStore:nil,GitRepo:nil,Secret:nil,NFS:nil,ISCSI:nil,Glusterfs:nil,PersistentVolumeClaim:nil,RBD:nil,FlexVolume:nil,Cinder:nil,CephFS:nil,Flocker:nil,DownwardAPI:nil,FC:nil,AzureFile:nil,ConfigMap:nil,VsphereVolume:nil,Quobyte:nil,AzureDisk:nil,PhotonPersistentDisk:nil,PortworxVolume:nil,ScaleIO:nil,Projected:&ProjectedVolumeSource{Sources:[]VolumeProjection{VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:nil,ServiceAccountToken:&ServiceAccountTokenProjection{Audience:,ExpirationSeconds:*3607,Path:token,},},VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:&ConfigMapProjection{LocalObjectReference:LocalObjectReference{Name:kube-root-ca.crt,},Items:[]KeyToPath{KeyToPath{Key:ca.crt,Path:ca.crt,Mode:nil,},},Optional:nil,},ServiceAccountToken:nil,},VolumeProjection{Secret:nil,DownwardAPI:&DownwardAPIProjection{Items:[]DownwardAPIVolumeFile{DownwardAPIVolumeFile{Path:namespace,FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,Mode:nil,},},},ConfigMap:nil,ServiceAccountToken:nil,},},DefaultMode:*420,},StorageOS:nil,CSI:nil,Ephemeral:nil,},},},Containers:[]Container{Container{Name:agnhost-container,Image:registry.k8s.io/e2e-test-images/agnhost:2.43,Command:[],Args:[pause],WorkingDir:,Ports:[]ContainerPort{},Env:[]EnvVar{},Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{},Claims:[]ResourceClaim{},},VolumeMounts:[]VolumeMount{VolumeMount{Name:kube-api-access-nnlxb,ReadOnly:true,MountPath:/var/run/secrets/kubernetes.io/serviceaccount,SubPath:,MountPropagation:nil,SubPathExpr:,},},LivenessProbe:nil,ReadinessProbe:nil,Lifecycle:nil,TerminationMessagePath:/dev/termination-log,ImagePullPolicy:IfNotPresent,SecurityContext:&SecurityContext{Capabilities:nil,Privileged:nil,SELinuxOptions:nil,RunAsUser:nil,RunAsNonRoot:nil,ReadOnlyRootFilesystem:nil,AllowPrivilegeEscalation:nil,RunAsGroup:nil,ProcMount:nil,WindowsOptions:nil,SeccompProfile:nil,},Stdin:false,StdinOnce:false,TTY:false,EnvFrom:[]EnvFromSource{},TerminationMessagePolicy:File,VolumeDevices:[]VolumeDevice{},StartupProbe:nil,ResizePolicy:[]ContainerResizePolicy{},},},RestartPolicy:Always,TerminationGracePeriodSeconds:*0,ActiveDeadlineSeconds:nil,DNSPolicy:None,NodeSelector:map[string]string{},ServiceAccountName:default,DeprecatedServiceAccount:default,NodeName:,HostNetwork:false,HostPID:false,HostIPC:false,SecurityContext:&PodSecurityContext{SELinuxOptions:nil,RunAsUser:nil,RunAsNonRoot:nil,SupplementalGroups:[],FSGroup:nil,RunAsGroup:nil,Sysctls:[]Sysctl{},WindowsOptions:nil,FSGroupChangePolicy:nil,SeccompProfile:nil,},ImagePullSecrets:[]LocalObjectReference{},Hostname:,Subdomain:,Affinity:nil,SchedulerName:default-scheduler,InitContainers:[]Container{},AutomountServiceAccountToken:nil,Tolerations:[]Toleration{Toleration{Key:node.kubernetes.io/not-ready,Operator:Exists,Value:,Effect:NoExecute,TolerationSeconds:*300,},Toleration{Key:node.kubernetes.io/unreachable,Operator:Exists,Value:,Effect:NoExecute,TolerationSeconds:*300,},},HostAliases:[]HostAlias{},PriorityClassName:,Priority:*0,DNSConfig:&PodDNSConfig{Nameservers:[1.1.1.1],Searches:[resolv.conf.local],Options:[]PodDNSConfigOption{},},ShareProcessNamespace:nil,ReadinessGates:[]PodReadinessGate{},RuntimeClassName:nil,EnableServiceLinks:*true,PreemptionPolicy:*PreemptLowerPriority,Overhead:ResourceList{},TopologySpreadConstraints:[]TopologySpreadConstraint{},EphemeralContainers:[]EphemeralContainer{},SetHostnameAsFQDN:nil,OS:nil,HostUsers:nil,SchedulingGates:[]PodSchedulingGate{},ResourceClaims:[]PodResourceClaim{},},Status:PodStatus{Phase:Pending,Conditions:[]PodCondition{},Message:,Reason:,HostIP:,PodIP:,StartTime:<nil>,ContainerStatuses:[]ContainerStatus{},QOSClass:BestEffort,InitContainerStatuses:[]ContainerStatus{},NominatedNodeName:,PodIPs:[]PodIP{},EphemeralContainerStatuses:[]ContainerStatus{},Resize:,},} Mar 5 16:37:55.807: INFO: Failed inside E2E framework: k8s.io/kubernetes/test/e2e/framework/pod.WaitForPodCondition({0x7f6944f7a880, 0xc00115c380}, {0x8252218?, 0xc002b4b380?}, {0xc0034fbdc0, 0x8}, {0xc00030d398, 0x14}, {0x7850c43, 0x11}, ...) test/e2e/framework/pod/wait.go:228 +0x25f k8s.io/kubernetes/test/e2e/framework/pod.WaitTimeoutForPodReadyInNamespace({0x7f6944f7a880?, 0xc00115c380?}, {0x8252218?, 0xc002b4b380?}, {0xc00030d398?, 0x1?}, {0xc0034fbdc0?, 0x0?}, 0x0?) test/e2e/framework/pod/wait.go:492 +0x75 k8s.io/kubernetes/test/e2e/network.glob..func2.10({0x7f6944f7a880, 0xc00115c380}) test/e2e/network/dns.go:424 +0x454 [38;5;9m[FAILED][0m in [It] - test/e2e/network/dns.go:425 [38;5;243m@ 03/05/23 16:37:55.807[0m Mar 5 16:37:55.807: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready Mar 5 16:37:55.920: INFO: Condition Ready of node i-06df568f6e5c59fff is false, but Node is tainted by NodeController with [{node.kubernetes.io/not-ready NoSchedule 2023-03-05 16:37:27 +0000 UTC} {node.kubernetes.io/not-ready NoExecute 2023-03-05 16:37:31 +0000 UTC}]. Failure Mar 5 16:37:58.034: INFO: Condition Ready of node i-06df568f6e5c59fff is false, but Node is tainted by NodeController with [{node.kubernetes.io/not-ready NoSchedule 2023-03-05 16:37:27 +0000 UTC} {node.kubernetes.io/not-ready NoExecute 2023-03-05 16:37:31 +0000 UTC}]. Failure Mar 5 16:38:00.034: INFO: Condition Ready of node i-06df568f6e5c59fff is false, but Node is tainted by NodeController with [{node.kubernetes.io/not-ready NoSchedule 2023-03-05 16:37:27 +0000 UTC} {node.kubernetes.io/not-ready NoExecute 2023-03-05 16:37:31 +0000 UTC}]. Failure Mar 5 16:38:02.034: INFO: Condition Ready of node i-06df568f6e5c59fff is false, but Node is tainted by NodeController with [{node.kubernetes.io/not-ready NoSchedule 2023-03-05 16:37:27 +0000 UTC} {node.kubernetes.io/not-ready NoExecute 2023-03-05 16:37:31 +0000 UTC}]. Failure Mar 5 16:38:04.033: INFO: Condition Ready of node i-06df568f6e5c59fff is false, but Node is tainted by NodeController with [{node.kubernetes.io/not-ready NoSchedule 2023-03-05 16:37:27 +0000 UTC} {node.kubernetes.io/not-ready NoExecute 2023-03-05 16:37:31 +0000 UTC}]. Failure ... skipping 66 lines ... Mar 5 16:40:18.034: INFO: Condition Ready of node i-06df568f6e5c59fff is false, but Node is tainted by NodeController with [{node.kubernetes.io/not-ready NoSchedule 2023-03-05 16:37:27 +0000 UTC} {node.kubernetes.io/not-ready NoExecute 2023-03-05 16:37:31 +0000 UTC}]. Failure Mar 5 16:40:20.033: INFO: Condition Ready of node i-06df568f6e5c59fff is false, but Node is tainted by NodeController with [{node.kubernetes.io/not-ready NoSchedule 2023-03-05 16:37:27 +0000 UTC} {node.kubernetes.io/not-ready NoExecute 2023-03-05 16:37:31 +0000 UTC}]. Failure Mar 5 16:40:22.034: INFO: Condition Ready of node i-06df568f6e5c59fff is false, but Node is tainted by NodeController with [{node.kubernetes.io/not-ready NoSchedule 2023-03-05 16:37:27 +0000 UTC} {node.kubernetes.io/not-ready NoExecute 2023-03-05 16:37:31 +0000 UTC}]. Failure Mar 5 16:40:24.034: INFO: Condition Ready of node i-06df568f6e5c59fff is false, but Node is tainted by NodeController with [{node.kubernetes.io/not-ready NoSchedule 2023-03-05 16:37:27 +0000 UTC} {node.kubernetes.io/not-ready NoExecute 2023-03-05 16:37:31 +0000 UTC}]. Failure Mar 5 16:40:26.035: INFO: Condition Ready of node i-06df568f6e5c59fff is true, but Node is tainted by NodeController with [{node.kubernetes.io/not-ready NoExecute 2023-03-05 16:37:31 +0000 UTC}]. Failure Mar 5 16:40:28.033: INFO: Deleting pod test-dns-nameservers... [38;5;9m[FAILED][0m in [DeferCleanup (Each)] - test/e2e/network/dns.go:420 [38;5;243m@ 03/05/23 16:40:28.146[0m [1mSTEP:[0m dump namespace information after failure [38;5;243m@ 03/05/23 16:40:28.147[0m [1mSTEP:[0m Collecting events from namespace "dns-4814". [38;5;243m@ 03/05/23 16:40:28.147[0m [1mSTEP:[0m Found 3 events. [38;5;243m@ 03/05/23 16:40:28.259[0m Mar 5 16:40:28.259: INFO: At 2023-03-05 16:32:55 +0000 UTC - event for test-dns-nameservers: {default-scheduler } Scheduled: Successfully assigned dns-4814/test-dns-nameservers to i-06df568f6e5c59fff Mar 5 16:40:28.259: INFO: At 2023-03-05 16:38:26 +0000 UTC - event for test-dns-nameservers: {taint-controller } TaintManagerEviction: Marking for deletion Pod dns-4814/test-dns-nameservers Mar 5 16:40:28.260: INFO: At 2023-03-05 16:38:30 +0000 UTC - event for test-dns-nameservers: {taint-controller } TaintManagerEviction: Cancelling deletion of Pod dns-4814/test-dns-nameservers ... skipping 195 lines ... Mar 5 16:41:34.003: INFO: pod-ready started at 2023-03-05 16:31:54 +0000 UTC (0+1 container statuses recorded) Mar 5 16:41:34.003: INFO: Container pod-readiness-gate ready: true, restart count 0 Mar 5 16:41:34.003: INFO: simpletest.rc-sh75x started at <nil> (0+0 container statuses recorded) Mar 5 16:41:34.003: INFO: simpletest.rc-cf2v4 started at <nil> (0+0 container statuses recorded) Mar 5 16:41:34.003: INFO: up-down-1-x9pvn started at 2023-03-05 16:31:52 +0000 UTC (0+1 container statuses recorded) Mar 5 16:41:34.003: INFO: Container up-down-1 ready: true, restart count 0 Mar 5 16:43:00.752: INFO: Unexpected error: <*errors.StatusError | 0xc000b55540>: { ErrStatus: code: 503 details: causes: - message: unknown ... skipping 2 lines ... name: i-06df568f6e5c59fff:10250 message: the server is currently unable to handle the request (get nodes i-06df568f6e5c59fff:10250) metadata: {} reason: ServiceUnavailable status: Failure, } [38;5;9m[FAILED][0m in [DeferCleanup (Each)] - test/e2e/framework/debug/dump.go:139 [38;5;243m@ 03/05/23 16:43:00.753[0m [1mSTEP:[0m Destroying namespace "dns-4814" for this suite. [38;5;243m@ 03/05/23 16:43:00.753[0m [38;5;243m<< Timeline[0m [38;5;9m[FAILED] failed to wait for pod test-dns-nameservers to be running: Timed out after 300.105s. expected pod to be running and ready, got instead: <*v1.Pod | 0xc002ebc000>: metadata: creationTimestamp: "2023-03-05T16:32:55Z" managedFields: - apiVersion: v1 ... skipping 98 lines ... [38;5;9mIn [1m[It][0m[38;5;9m at: [1mtest/e2e/network/dns.go:425[0m [38;5;243m@ 03/05/23 16:37:55.807[0m There were [1m[38;5;9madditional failures[0m detected. To view them in detail run [1mginkgo -vv[0m [38;5;243m------------------------------[0m [38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;10m•[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;10m•[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;10m•[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;10m•[0m[38;5;10m•[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;10m•[0m [38;5;243m------------------------------[0m [38;5;9m• [FAILED] [493.841 seconds][0m [0m[sig-node] Container Runtime [38;5;243mblackbox test [0mon terminated container [38;5;9m[1m[It] should report termination message if TerminationMessagePath is set [NodeConformance][0m [38;5;243mtest/e2e/common/node/runtime.go:174[0m [38;5;243mTimeline >>[0m [1mSTEP:[0m Creating a kubernetes client [38;5;243m@ 03/05/23 16:40:27.201[0m Mar 5 16:40:27.201: INFO: >>> kubeConfig: /root/.kube/config [1mSTEP:[0m Building a namespace api object, basename container-runtime [38;5;243m@ 03/05/23 16:40:27.202[0m [1mSTEP:[0m Waiting for a default service account to be provisioned in namespace [38;5;243m@ 03/05/23 16:40:27.54[0m [1mSTEP:[0m Waiting for kube-root-ca.crt to be provisioned in namespace [38;5;243m@ 03/05/23 16:40:27.764[0m [1mSTEP:[0m create the container [38;5;243m@ 03/05/23 16:40:27.988[0m W0305 16:40:28.105526 7366 warnings.go:70] metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must be no more than 63 characters] [1mSTEP:[0m wait for the container to reach Succeeded [38;5;243m@ 03/05/23 16:40:28.105[0m [38;5;9m[FAILED][0m in [It] - test/e2e/common/node/runtime.go:157 [38;5;243m@ 03/05/23 16:45:28.106[0m Mar 5 16:45:28.107: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready Mar 5 16:45:28.220: INFO: Condition Ready of node i-06df568f6e5c59fff is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-05 16:42:51 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-05 16:42:57 +0000 UTC}]. Failure Mar 5 16:45:30.334: INFO: Condition Ready of node i-06df568f6e5c59fff is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-05 16:42:51 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-05 16:42:57 +0000 UTC}]. Failure Mar 5 16:45:32.334: INFO: Condition Ready of node i-06df568f6e5c59fff is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-05 16:42:51 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-05 16:42:57 +0000 UTC}]. Failure Mar 5 16:45:34.334: INFO: Condition Ready of node i-06df568f6e5c59fff is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-05 16:42:51 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-05 16:42:57 +0000 UTC}]. Failure Mar 5 16:45:36.333: INFO: Condition Ready of node i-06df568f6e5c59fff is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-05 16:42:51 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-05 16:42:57 +0000 UTC}]. Failure ... skipping 108 lines ... Mar 5 16:48:29.257: INFO: cleanup20-16a6f148-bb85-4b73-99f5-2adb1cfdeea0-j6cks started at 2023-03-05 16:48:01 +0000 UTC (0+1 container statuses recorded) Mar 5 16:48:29.257: INFO: Container cleanup20-16a6f148-bb85-4b73-99f5-2adb1cfdeea0 ready: true, restart count 0 Mar 5 16:48:29.257: INFO: cleanup20-16a6f148-bb85-4b73-99f5-2adb1cfdeea0-khffj started at 2023-03-05 16:40:29 +0000 UTC (0+1 container statuses recorded) Mar 5 16:48:29.257: INFO: Container cleanup20-16a6f148-bb85-4b73-99f5-2adb1cfdeea0 ready: true, restart count 0 Mar 5 16:48:29.257: INFO: nodeport-test-88ggz started at 2023-03-05 16:40:39 +0000 UTC (0+1 container statuses recorded) Mar 5 16:48:29.257: INFO: Container nodeport-test ready: true, restart count 0 Mar 5 16:48:29.257: INFO: fail-once-local-2mt4w started at 2023-03-05 16:46:28 +0000 UTC (0+1 container statuses recorded) Mar 5 16:48:29.257: INFO: Container c ready: false, restart count 1 Mar 5 16:48:29.257: INFO: cleanup20-16a6f148-bb85-4b73-99f5-2adb1cfdeea0-gnfq6 started at 2023-03-05 16:48:01 +0000 UTC (0+1 container statuses recorded) Mar 5 16:48:29.257: INFO: Container cleanup20-16a6f148-bb85-4b73-99f5-2adb1cfdeea0 ready: true, restart count 0 Mar 5 16:48:29.257: INFO: cleanup20-16a6f148-bb85-4b73-99f5-2adb1cfdeea0-xkdzr started at 2023-03-05 16:40:29 +0000 UTC (0+1 container statuses recorded) Mar 5 16:48:29.257: INFO: Container cleanup20-16a6f148-bb85-4b73-99f5-2adb1cfdeea0 ready: true, restart count 0 Mar 5 16:48:29.257: INFO: ss-0 started at 2023-03-05 16:33:11 +0000 UTC (0+1 container statuses recorded) ... skipping 5 lines ... Mar 5 16:48:29.257: INFO: coredns-745bddcfbb-gm4h2 started at 2023-03-05 16:25:57 +0000 UTC (0+1 container statuses recorded) Mar 5 16:48:29.257: INFO: Container coredns ready: true, restart count 0 Mar 5 16:48:29.257: INFO: cleanup20-16a6f148-bb85-4b73-99f5-2adb1cfdeea0-sqrjl started at 2023-03-05 16:40:29 +0000 UTC (0+1 container statuses recorded) Mar 5 16:48:29.257: INFO: Container cleanup20-16a6f148-bb85-4b73-99f5-2adb1cfdeea0 ready: true, restart count 0 Mar 5 16:48:29.257: INFO: cleanup20-16a6f148-bb85-4b73-99f5-2adb1cfdeea0-n9vbn started at 2023-03-05 16:40:29 +0000 UTC (0+1 container statuses recorded) Mar 5 16:48:29.257: INFO: Container cleanup20-16a6f148-bb85-4b73-99f5-2adb1cfdeea0 ready: true, restart count 0 Mar 5 16:48:29.257: INFO: fail-once-local-gj678 started at 2023-03-05 16:46:33 +0000 UTC (0+1 container statuses recorded) Mar 5 16:48:29.257: INFO: Container c ready: false, restart count 1 Mar 5 16:48:29.257: INFO: cleanup20-16a6f148-bb85-4b73-99f5-2adb1cfdeea0-m6dgl started at 2023-03-05 16:48:01 +0000 UTC (0+1 container statuses recorded) Mar 5 16:48:29.257: INFO: Container cleanup20-16a6f148-bb85-4b73-99f5-2adb1cfdeea0 ready: true, restart count 0 Mar 5 16:48:29.257: INFO: cleanup20-16a6f148-bb85-4b73-99f5-2adb1cfdeea0-cqffq started at 2023-03-05 16:48:02 +0000 UTC (0+1 container statuses recorded) Mar 5 16:48:29.257: INFO: Container cleanup20-16a6f148-bb85-4b73-99f5-2adb1cfdeea0 ready: true, restart count 0 Mar 5 16:48:29.257: INFO: ebs-csi-node-n98cq started at 2023-03-05 16:25:49 +0000 UTC (0+3 container statuses recorded) ... skipping 4 lines ... Mar 5 16:48:29.257: INFO: Init container clean-cilium-state ready: true, restart count 0 Mar 5 16:48:29.257: INFO: Container cilium-agent ready: true, restart count 0 Mar 5 16:48:29.257: INFO: sample-webhook-deployment-7497495989-s9585 started at 2023-03-05 16:46:38 +0000 UTC (0+1 container statuses recorded) Mar 5 16:48:29.257: INFO: Container sample-webhook ready: true, restart count 0 Mar 5 16:48:29.257: INFO: cleanup20-16a6f148-bb85-4b73-99f5-2adb1cfdeea0-xsqgf started at 2023-03-05 16:40:29 +0000 UTC (0+1 container statuses recorded) Mar 5 16:48:29.257: INFO: Container cleanup20-16a6f148-bb85-4b73-99f5-2adb1cfdeea0 ready: true, restart count 0 Mar 5 16:48:29.257: INFO: fail-once-local-kdbgv started at 2023-03-05 16:46:33 +0000 UTC (0+1 container statuses recorded) Mar 5 16:48:29.257: INFO: Container c ready: false, restart count 1 Mar 5 16:48:29.257: INFO: coredns-autoscaler-8495f5575-plvdw started at 2023-03-05 16:25:57 +0000 UTC (0+1 container statuses recorded) Mar 5 16:48:29.257: INFO: Container autoscaler ready: true, restart count 0 Mar 5 16:48:29.257: INFO: cleanup20-16a6f148-bb85-4b73-99f5-2adb1cfdeea0-kgbnw started at 2023-03-05 16:40:29 +0000 UTC (0+1 container statuses recorded) Mar 5 16:48:29.257: INFO: Container cleanup20-16a6f148-bb85-4b73-99f5-2adb1cfdeea0 ready: true, restart count 0 Mar 5 16:48:29.257: INFO: cleanup20-16a6f148-bb85-4b73-99f5-2adb1cfdeea0-42w6f started at 2023-03-05 16:40:29 +0000 UTC (0+1 container statuses recorded) Mar 5 16:48:29.257: INFO: Container cleanup20-16a6f148-bb85-4b73-99f5-2adb1cfdeea0 ready: true, restart count 0 Mar 5 16:48:29.257: INFO: cleanup20-16a6f148-bb85-4b73-99f5-2adb1cfdeea0-6lnc6 started at 2023-03-05 16:48:01 +0000 UTC (0+1 container statuses recorded) Mar 5 16:48:29.257: INFO: Container cleanup20-16a6f148-bb85-4b73-99f5-2adb1cfdeea0 ready: true, restart count 0 Mar 5 16:48:29.257: INFO: cleanup20-16a6f148-bb85-4b73-99f5-2adb1cfdeea0-9qlxz started at 2023-03-05 16:48:01 +0000 UTC (0+1 container statuses recorded) Mar 5 16:48:29.257: INFO: Container cleanup20-16a6f148-bb85-4b73-99f5-2adb1cfdeea0 ready: true, restart count 0 Mar 5 16:48:29.257: INFO: cleanup20-16a6f148-bb85-4b73-99f5-2adb1cfdeea0-7qxdd started at 2023-03-05 16:48:03 +0000 UTC (0+1 container statuses recorded) Mar 5 16:48:29.257: INFO: Container cleanup20-16a6f148-bb85-4b73-99f5-2adb1cfdeea0 ready: true, restart count 0 Mar 5 16:48:29.257: INFO: fail-once-local-69mfr started at 2023-03-05 16:46:28 +0000 UTC (0+1 container statuses recorded) Mar 5 16:48:29.257: INFO: Container c ready: false, restart count 1 Mar 5 16:48:29.257: INFO: cleanup20-16a6f148-bb85-4b73-99f5-2adb1cfdeea0-wmjdk started at 2023-03-05 16:40:29 +0000 UTC (0+1 container statuses recorded) Mar 5 16:48:29.257: INFO: Container cleanup20-16a6f148-bb85-4b73-99f5-2adb1cfdeea0 ready: true, restart count 0 Mar 5 16:48:29.257: INFO: ebs-csi-controller-79785c665b-7dmsf started at 2023-03-05 16:25:57 +0000 UTC (0+5 container statuses recorded) Mar 5 16:48:29.257: INFO: Container csi-attacher ready: true, restart count 0 Mar 5 16:48:29.257: INFO: Container csi-provisioner ready: true, restart count 0 ... skipping 12 lines ... Logging node info for node i-06df568f6e5c59fff Mar 5 16:48:29.831: INFO: Node Info: &Node{ObjectMeta:{i-06df568f6e5c59fff cb63c681-2e18-474a-b6f3-221325ac0d52 17637 0 2023-03-05 16:24:16 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:arm64 beta.kubernetes.io/instance-type:t4g.small beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:eu-central-1 failure-domain.beta.kubernetes.io/zone:eu-central-1a karpenter.k8s.aws/instance-category:t karpenter.k8s.aws/instance-cpu:2 karpenter.k8s.aws/instance-family:t4g karpenter.k8s.aws/instance-generation:4 karpenter.k8s.aws/instance-hypervisor:nitro karpenter.k8s.aws/instance-memory:2048 karpenter.k8s.aws/instance-pods:11 karpenter.k8s.aws/instance-size:small karpenter.sh/capacity-type:spot karpenter.sh/initialized:true karpenter.sh/provisioner-name:nodes kubelet_cleanup:true kubernetes.io/arch:arm64 kubernetes.io/hostname:i-06df568f6e5c59fff kubernetes.io/os:linux node-role.kubernetes.io/node: node-role.kubernetes.io/spot-worker:true node.kubernetes.io/instance-type:t4g.small topology.ebs.csi.aws.com/zone:eu-central-1a topology.hostpath.csi/node:i-06df568f6e5c59fff topology.kubernetes.io/region:eu-central-1 topology.kubernetes.io/zone:eu-central-1a] map[csi.volume.kubernetes.io/nodeid:{"csi-hostpath-ephemeral-2147":"i-06df568f6e5c59fff","csi-hostpath-ephemeral-5408":"i-06df568f6e5c59fff","csi-hostpath-ephemeral-6540":"i-06df568f6e5c59fff","csi-mock-csi-mock-volumes-fsgroup-policy-1058":"i-06df568f6e5c59fff","csi-mock-csi-mock-volumes-workload-2219":"i-06df568f6e5c59fff","ebs.csi.aws.com":"i-06df568f6e5c59fff"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [{karpenter.sh/v1alpha5 Provisioner nodes ac5d7599-b2a2-4eae-8da4-1f4c0a54ec82 <nil> 0xc003a10010}] [karpenter.sh/termination] [{aws-cloud-controller-manager Update v1 2023-03-05 16:24:16 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{}}}} } {kops-controller Update v1 2023-03-05 16:24:16 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{},"f:node-role.kubernetes.io/spot-worker":{}}}} } {karpenter Update v1 2023-03-05 16:25:57 +0000 UTC FieldsV1 {"f:metadata":{"f:finalizers":{".":{},"v:\"karpenter.sh/termination\"":{}},"f:labels":{".":{},"f:karpenter.k8s.aws/instance-category":{},"f:karpenter.k8s.aws/instance-cpu":{},"f:karpenter.k8s.aws/instance-family":{},"f:karpenter.k8s.aws/instance-generation":{},"f:karpenter.k8s.aws/instance-hypervisor":{},"f:karpenter.k8s.aws/instance-memory":{},"f:karpenter.k8s.aws/instance-pods":{},"f:karpenter.k8s.aws/instance-size":{},"f:karpenter.sh/capacity-type":{},"f:karpenter.sh/initialized":{},"f:karpenter.sh/provisioner-name":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}},"f:ownerReferences":{".":{},"k:{\"uid\":\"ac5d7599-b2a2-4eae-8da4-1f4c0a54ec82\"}":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-03-05 16:26:48 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{".":{},"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"InternalIP\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {e2e.test Update v1 2023-03-05 16:40:28 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubelet_cleanup":{}}}} } {kube-controller-manager Update v1 2023-03-05 16:44:29 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{".":{},"k:{\"type\":\"DiskPressure\"}":{".":{},"f:type":{}},"k:{\"type\":\"MemoryPressure\"}":{".":{},"f:type":{}},"k:{\"type\":\"PIDPressure\"}":{".":{},"f:type":{}},"k:{\"type\":\"Ready\"}":{".":{},"f:type":{}}},"f:volumesAttached":{}}} status} {kube-controller-manager Update v1 2023-03-05 16:46:37 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:node.alpha.kubernetes.io/ttl":{}},"f:labels":{"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.2.0/24\"":{}},"f:taints":{}}} } {kubelet Update v1 2023-03-05 16:47:16 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{"f:kubernetes.io/hostname":{},"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{".":{},"f:cpu":{},"f:ephemeral-storage":{},"f:hugepages-1Gi":{},"f:hugepages-2Mi":{},"f:hugepages-32Mi":{},"f:hugepages-64Ki":{},"f:memory":{},"f:pods":{}},"f:capacity":{".":{},"f:cpu":{},"f:ephemeral-storage":{},"f:hugepages-1Gi":{},"f:hugepages-2Mi":{},"f:hugepages-32Mi":{},"f:hugepages-64Ki":{},"f:memory":{},"f:pods":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:daemonEndpoints":{"f:kubeletEndpoint":{"f:Port":{}}},"f:images":{},"f:nodeInfo":{"f:architecture":{},"f:bootID":{},"f:containerRuntimeVersion":{},"f:kernelVersion":{},"f:kubeProxyVersion":{},"f:kubeletVersion":{},"f:machineID":{},"f:operatingSystem":{},"f:osImage":{},"f:systemUUID":{}},"f:volumesInUse":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.2.0/24,DoNotUseExternalID:,ProviderID:aws:///eu-central-1a/i-06df568f6e5c59fff,Unschedulable:false,Taints:[]Taint{Taint{Key:node.kubernetes.io/not-ready,Value:,Effect:NoSchedule,TimeAdded:2023-03-05 16:46:33 +0000 UTC,},Taint{Key:node.kubernetes.io/not-ready,Value:,Effect:NoExecute,TimeAdded:2023-03-05 16:46:37 +0000 UTC,},},ConfigSource:nil,PodCIDRs:[100.96.2.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49756430336 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-32Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-64Ki: {{0 0} {<nil>} 0 DecimalSI},memory: {{1941540864 0} {<nil>} 1896036Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44780787229 0} {<nil>} 44780787229 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-32Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-64Ki: {{0 0} {<nil>} 0 DecimalSI},memory: {{1836683264 0} {<nil>} 1793636Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:Ready,Status:False,LastHeartbeatTime:2023-03-05 16:47:15 +0000 UTC,LastTransitionTime:2023-03-05 16:46:08 +0000 UTC,Reason:KubeletNotReady,Message:PLEG is not healthy: pleg was last seen active 8m10.540757618s ago; threshold is 3m0s,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-03-05 16:47:12 +0000 UTC,LastTransitionTime:2023-03-05 16:47:12 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-03-05 16:47:15 +0000 UTC,LastTransitionTime:2023-03-05 16:47:15 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-03-05 16:47:15 +0000 UTC,LastTransitionTime:2023-03-05 16:47:15 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.43.148,},NodeAddress{Type:ExternalIP,Address:18.192.211.222,},NodeAddress{Type:InternalDNS,Address:i-06df568f6e5c59fff.eu-central-1.compute.internal,},NodeAddress{Type:Hostname,Address:i-06df568f6e5c59fff.eu-central-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-18-192-211-222.eu-central-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec221221864d9409a8ec5f6614a9b94b,SystemUUID:ec221221-864d-9409-a8ec-5f6614a9b94b,BootID:553d31bf-faca-417e-a3db-325839e1b8e4,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 22.04.2 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.27.0-alpha.3.52+d48b8167f7094b,KubeProxyVersion:v1.27.0-alpha.3.52+d48b8167f7094b,OperatingSystem:linux,Architecture:arm64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:157636062,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/jessie-dnsutils@sha256:24aaf2626d6b27864c29de2097e8bbb840b3a414271bf7c8995e431e47d8408e registry.k8s.io/e2e-test-images/jessie-dnsutils:1.7],SizeBytes:116764683,},ContainerImage{Names:[registry.k8s.io/kube-proxy-arm64:v1.27.0-alpha.3.52_d48b8167f7094b],SizeBytes:60554082,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:16bbf38c463a4223d8cfe4da12bc61010b082a79b4bb003e2d3ba3ece5dd5f9e registry.k8s.io/e2e-test-images/agnhost:2.43],SizeBytes:49808364,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:3fe7acf013d1264ffded116b80a73dc129a449b0fccdb8d21af8279f2233f36e registry.k8s.io/e2e-test-images/httpd:2.4.39-4],SizeBytes:41528818,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:148b022f5c5da426fc2f3c14b5c0867e58ef05961510c84749ac1fddcb0fef22 registry.k8s.io/e2e-test-images/httpd:2.4.38-4],SizeBytes:40343601,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:28932904,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:ee3b525d5b89db99da3b8eb521d9cd90cb6e9ef0fbb651e98bb37be78d36b5b8 registry.k8s.io/sig-storage/csi-provisioner:v3.3.0],SizeBytes:23945474,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:425d8f1b769398127767b06ed97ce62578a3179bcb99809ce93a1649e025ffe7 registry.k8s.io/sig-storage/csi-resizer:v1.6.0],SizeBytes:22678632,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:291334908ddf71a4661fd7f6d9d97274de8a5378a2b6fdfeb2ce73414a34f82f registry.k8s.io/sig-storage/csi-snapshotter:v6.1.0],SizeBytes:22434180,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:9a685020911e2725ad019dbce6e4a5ab93d51e3d4557f115e64343345e05781b registry.k8s.io/sig-storage/csi-attacher:v4.0.0],SizeBytes:22394501,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:122bfb8c1edabb3c0edd63f06523e6940d958d19b3957dc7b1d6f81e9f1f6119 registry.k8s.io/sig-storage/csi-provisioner:v3.1.0],SizeBytes:21735644,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:9ebbf9f023e7b41ccee3d52afe39a89e3ddacdbb69269d583abfc25847cfd9e4 registry.k8s.io/sig-storage/csi-resizer:v1.4.0],SizeBytes:20848413,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:8b9c313c05f54fb04f8d430896f5f5904b6cb157df261501b29adc04d2b2dc7b registry.k8s.io/sig-storage/csi-attacher:v3.4.0],SizeBytes:20565480,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nonroot@sha256:ee9f50b3c64b174d296d91ca9f69a914ac30e59095dfb462b2b518ad28a63655 registry.k8s.io/e2e-test-images/nonroot:1.4],SizeBytes:18561972,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:92257881c1d6493cf18299a24af42330f891166560047902b8d431fb66b01af5 registry.k8s.io/sig-storage/hostpathplugin:v1.9.0],SizeBytes:18267611,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:9b2d9abda017c28b12f84a344f57cd73fbdb6c2bd7dd5bdd5018246ad1093ba6 registry.k8s.io/sig-storage/hostpathplugin:v1.11.0],SizeBytes:18159085,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:017727efcfeb7d053af68e51436ce8e65edbc6ca573720afb4f79c8594036955 registry.k8s.io/coredns/coredns:v1.10.0],SizeBytes:13766234,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:8502345,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:933940f13b3ea0abc62e656c1aa5c5b47c04b15d71250413a6b821bd0c58b94e registry.k8s.io/sig-storage/livenessprobe:v2.7.0],SizeBytes:8223819,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:7685305,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:5c99cf6a02adda929b10321dbf4ecfa00d87be9ba4fb456006237d530ab4baa1 registry.k8s.io/e2e-test-images/nginx:1.14-4],SizeBytes:6860089,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nonewprivs@sha256:8ac1264691820febacf3aea5d152cbde6d10685731ec14966a9401c6f47a68ac registry.k8s.io/e2e-test-images/nonewprivs:1.3],SizeBytes:5025027,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:2e0f836850e09b8b7cc937681d6194537a09fbd5f6b9e08f4d646a85128e8937 registry.k8s.io/e2e-test-images/busybox:1.29-4],SizeBytes:772933,},ContainerImage{Names:[registry.k8s.io/pause@sha256:7031c1b283388d2c2e09b57badb803c05ebed362dc88d84b480cc47f72a21097 registry.k8s.io/pause:3.9],SizeBytes:268051,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:253553,},},VolumesInUse:[kubernetes.io/csi/csi-hostpath-ephemeral-2147^52df5953-bb73-11ed-8aa7-be49170871b4 kubernetes.io/csi/csi-hostpath-ephemeral-5408^443d1daf-bb73-11ed-95c4-5a95d17cbc27 kubernetes.io/csi/csi-hostpath-ephemeral-5408^44411f51-bb73-11ed-95c4-5a95d17cbc27 kubernetes.io/csi/csi-hostpath-ephemeral-6540^4426a07f-bb73-11ed-a604-92687f2c50f8 kubernetes.io/csi/ebs.csi.aws.com^vol-05b67fab5beb6394b],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-05b67fab5beb6394b,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/csi-hostpath-ephemeral-6540^4426a07f-bb73-11ed-a604-92687f2c50f8,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/csi-hostpath-ephemeral-5408^44411f51-bb73-11ed-95c4-5a95d17cbc27,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/csi-hostpath-ephemeral-5408^443d1daf-bb73-11ed-95c4-5a95d17cbc27,DevicePath:,},},Config:nil,},} Mar 5 16:48:29.832: INFO: Logging kubelet events for node i-06df568f6e5c59fff Mar 5 16:48:29.950: INFO: Logging pods the kubelet thinks is on node i-06df568f6e5c59fff Mar 5 16:48:40.064: INFO: Unable to retrieve kubelet pods for node i-06df568f6e5c59fff: error trying to reach service: net/http: TLS handshake timeout Mar 5 16:48:40.064: INFO: Logging node info for node i-0d6a32f749839192f Mar 5 16:48:40.176: INFO: Node Info: &Node{ObjectMeta:{i-0d6a32f749839192f 4155fefb-4d3a-4b04-ba53-6ac9e895ff92 16414 0 2023-03-05 16:20:46 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:arm64 beta.kubernetes.io/instance-type:c6g.xlarge beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:eu-central-1 failure-domain.beta.kubernetes.io/zone:eu-central-1a kops.k8s.io/kops-controller-pki: kubernetes.io/arch:arm64 kubernetes.io/hostname:i-0d6a32f749839192f kubernetes.io/os:linux node-role.kubernetes.io/control-plane: node.kubernetes.io/exclude-from-external-load-balancers: node.kubernetes.io/instance-type:c6g.xlarge topology.ebs.csi.aws.com/zone:eu-central-1a topology.kubernetes.io/region:eu-central-1 topology.kubernetes.io/zone:eu-central-1a] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-0d6a32f749839192f"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kubelet Update v1 2023-03-05 16:20:46 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {protokube Update v1 2023-03-05 16:21:25 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kops.k8s.io/kops-controller-pki":{},"f:node-role.kubernetes.io/control-plane":{},"f:node.kubernetes.io/exclude-from-external-load-balancers":{}}}} } {kube-controller-manager Update v1 2023-03-05 16:21:47 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.0.0/24\"":{}}}} } {aws-cloud-controller-manager Update v1 2023-03-05 16:21:48 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{},"f:taints":{}}} } {aws-cloud-controller-manager Update v1 2023-03-05 16:21:48 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kubelet Update v1 2023-03-05 16:44:12 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.0.0/24,DoNotUseExternalID:,ProviderID:aws:///eu-central-1a/i-0d6a32f749839192f,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/control-plane,Value:,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[100.96.0.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{4 0} {<nil>} 4 DecimalSI},ephemeral-storage: {{49756430336 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-32Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-64Ki: {{0 0} {<nil>} 0 DecimalSI},memory: {{8150118400 0} {<nil>} 7959100Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{4 0} {<nil>} 4 DecimalSI},ephemeral-storage: {{44780787229 0} {<nil>} 44780787229 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-32Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-64Ki: {{0 0} {<nil>} 0 DecimalSI},memory: {{8045260800 0} {<nil>} 7856700Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-03-05 16:44:12 +0000 UTC,LastTransitionTime:2023-03-05 16:20:46 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-03-05 16:44:12 +0000 UTC,LastTransitionTime:2023-03-05 16:20:46 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-03-05 16:44:12 +0000 UTC,LastTransitionTime:2023-03-05 16:20:46 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-03-05 16:44:12 +0000 UTC,LastTransitionTime:2023-03-05 16:21:47 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.49.247,},NodeAddress{Type:ExternalIP,Address:18.197.20.82,},NodeAddress{Type:InternalDNS,Address:i-0d6a32f749839192f.eu-central-1.compute.internal,},NodeAddress{Type:Hostname,Address:i-0d6a32f749839192f.eu-central-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-18-197-20-82.eu-central-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2bed249e8ac7b517cb08ccd42b83e5,SystemUUID:ec2bed24-9e8a-c7b5-17cb-08ccd42b83e5,BootID:6f9215c9-dfe8-4b13-84f4-9b2084456198,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 22.04.2 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.27.0-alpha.3.52+d48b8167f7094b,KubeProxyVersion:v1.27.0-alpha.3.52+d48b8167f7094b,OperatingSystem:linux,Architecture:arm64,},Images:[]ContainerImage{ContainerImage{Names:[registry.k8s.io/etcd@sha256:9ce33ba33d8e738a5b85ed50b5080ac746deceed4a7496c550927a7a19ca3b6d registry.k8s.io/etcd:3.5.0-0],SizeBytes:157797353,},ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:157636062,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:4afb99b4690b418ffc2ceb67e1a17376457e441c1f09ab55447f0aaf992fa646 registry.k8s.io/etcd:3.4.3-0],SizeBytes:151677080,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:4ad90a11b55313b182afc186b9876c8e891531b8db4c9bf1541953021618d0e2 registry.k8s.io/etcd:3.4.13-0],SizeBytes:134531559,},ContainerImage{Names:[registry.k8s.io/kube-apiserver-arm64:v1.27.0-alpha.3.52_d48b8167f7094b],SizeBytes:131087707,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:0d2ab6b15b4fe0b903b9e81192d942fb39c29be61fe341cd16db5d0165b0435c registry.k8s.io/etcd:3.3.17-0],SizeBytes:126979107,},ContainerImage{Names:[registry.k8s.io/kube-controller-manager-arm64:v1.27.0-alpha.3.52_d48b8167f7094b],SizeBytes:120667128,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:17da501f5d2a675be46040422a27b7cc21b8a43895ac998b171db1c346f361f7 registry.k8s.io/etcd:3.3.10-0],SizeBytes:81436097,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:6f72b851544986cb0921b53ea655ec04c36131248f16d4ad110cb3ca0c369dc1 registry.k8s.io/etcd:3.5.4-0],SizeBytes:81120118,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:13f53ed1d91e2e11aac476ee9a0269fdda6cc4874eba903efd40daf50c55eee5 registry.k8s.io/etcd:3.5.3-0],SizeBytes:81114891,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:51eae8381dcb1078289fa7b4f3df2630cdc18d09fb56f8e56b41c40e191d6c83 registry.k8s.io/etcd:3.5.7-0],SizeBytes:80665728,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:dd75ec974b0a2a6f6bb47001ba09207976e625db898d1b16735528c009cb171c registry.k8s.io/etcd:3.5.6-0],SizeBytes:80539316,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:905d7ca17fd02bc24c0eba9a062753aba15db3e31422390bc3238eb762339b20 registry.k8s.io/etcd:3.2.24-1],SizeBytes:61500433,},ContainerImage{Names:[registry.k8s.io/etcdadm/etcd-manager-slim@sha256:8727adc059334e0e3329dbccce116a734da4a090da5e86c8cddccaf333caa666 registry.k8s.io/etcdadm/etcd-manager-slim:v3.0.20230201],SizeBytes:60826655,},ContainerImage{Names:[registry.k8s.io/kube-proxy-arm64:v1.27.0-alpha.3.52_d48b8167f7094b],SizeBytes:60554082,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:64b9ea357325d5db9f8a723dcf503b5a449177b17ac87d69481e126bb724c263 registry.k8s.io/etcd:3.5.1-0],SizeBytes:59553904,},ContainerImage{Names:[registry.k8s.io/kube-scheduler-arm64:v1.27.0-alpha.3.52_d48b8167f7094b],SizeBytes:56310763,},ContainerImage{Names:[registry.k8s.io/kops/kops-controller:1.27.0-alpha.1],SizeBytes:41797411,},ContainerImage{Names:[registry.k8s.io/kops/dns-controller:1.27.0-alpha.1],SizeBytes:40267473,},ContainerImage{Names:[public.ecr.aws/karpenter/controller@sha256:68db4f092cf9cc83f5ef9e2fbc5407c2cb682e81f64dfaa700a7602ede38b1cf public.ecr.aws/karpenter/controller:v0.16.3],SizeBytes:33707751,},ContainerImage{Names:[public.ecr.aws/karpenter/webhook@sha256:96a2d9b06d6bc5127801f358f74b1cf2d289b423a2e9ba40c573c0b14b17dafa public.ecr.aws/karpenter/webhook:v0.16.3],SizeBytes:33307187,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:28932904,},ContainerImage{Names:[quay.io/cilium/operator@sha256:a6d24a006a6b92967ac90786b49bc1ac26e5477cf028cd1186efcfc2466484db quay.io/cilium/operator:v1.12.5],SizeBytes:24567287,},ContainerImage{Names:[registry.k8s.io/provider-aws/cloud-controller-manager@sha256:fdeb61e3e42ecd9cca868d550ebdb88dd6341d9e91fcfa9a37e227dab2ad22cb registry.k8s.io/provider-aws/cloud-controller-manager:v1.26.0],SizeBytes:18345226,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:8502345,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:7685305,},ContainerImage{Names:[registry.k8s.io/kops/kube-apiserver-healthcheck:1.27.0-alpha.1],SizeBytes:4713612,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:253553,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Mar 5 16:48:40.177: INFO: Logging kubelet events for node i-0d6a32f749839192f Mar 5 16:48:40.293: INFO: ... skipping 52 lines ... Mar 5 16:48:40.432: INFO: Container controller ready: true, restart count 0 Mar 5 16:48:40.927: INFO: Latency metrics for node i-0d6a32f749839192f [1mSTEP:[0m Destroying namespace "container-runtime-6711" for this suite. [38;5;243m@ 03/05/23 16:48:40.928[0m [38;5;243m<< Timeline[0m [38;5;9m[FAILED] Timed out after 300.001s. Expected <v1.PodPhase>: Pending to equal <v1.PodPhase>: Succeeded[0m [38;5;9mIn [1m[It][0m[38;5;9m at: [1mtest/e2e/common/node/runtime.go:157[0m [38;5;243m@ 03/05/23 16:45:28.106[0m [38;5;243m------------------------------[0m [38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;10m•[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m [38;5;243m------------------------------[0m [38;5;9m• [FAILED] [504.803 seconds][0m [0m[sig-storage] PersistentVolumes-local [38;5;9m[1m[Volume type: blockfswithformat] [BeforeEach] [0mTwo pods mounting a local volume one after the other [38;5;243mshould be able to write from pod1 and read from pod2[0m [38;5;9m[BeforeEach][0m [38;5;243mtest/e2e/storage/persistent_volumes-local.go:198[0m [38;5;243m[It] test/e2e/storage/persistent_volumes-local.go:258[0m [38;5;243mTimeline >>[0m [1mSTEP:[0m Creating a kubernetes client [38;5;243m@ 03/05/23 16:40:36.778[0m Mar 5 16:40:36.778: INFO: >>> kubeConfig: /root/.kube/config [1mSTEP:[0m Building a namespace api object, basename persistent-local-volumes-test [38;5;243m@ 03/05/23 16:40:36.779[0m [1mSTEP:[0m Waiting for a default service account to be provisioned in namespace [38;5;243m@ 03/05/23 16:40:37.116[0m [1mSTEP:[0m Waiting for kube-root-ca.crt to be provisioned in namespace [38;5;243m@ 03/05/23 16:40:37.337[0m [1mSTEP:[0m Initializing test volumes [38;5;243m@ 03/05/23 16:40:37.784[0m [1mSTEP:[0m Creating block device on node "i-06df568f6e5c59fff" using path "/tmp/local-volume-test-602ed0e5-86bc-424c-a6fe-2232304eff7a" [38;5;243m@ 03/05/23 16:40:37.784[0m Mar 5 16:45:37.969: INFO: Failed inside E2E framework: k8s.io/kubernetes/test/e2e/framework/pod.WaitTimeoutForPodRunningInNamespace({0x7f870436e3f8, 0xc003527700}, {0x8252218?, 0xc002f0a4e0?}, {0xc0037897d0, 0x22}, {0xc003789800, 0x22}, 0x0?) test/e2e/framework/pod/wait.go:460 +0x1a4 k8s.io/kubernetes/test/e2e/storage/utils.(*hostExecutor).launchNodeExecPod(0xfd?, {0x7f870436e3f8, 0xc003527700}, {0xc0030510b0, 0x13}) test/e2e/storage/utils/host_exec.go:116 +0x49d k8s.io/kubernetes/test/e2e/storage/utils.(*hostExecutor).exec(0xc000eb79b0, {0x7f870436e3f8, 0xc003527700}, {0xc004acb300, 0xfd}, 0xc002c1de40) test/e2e/storage/utils/host_exec.go:143 +0x135 ... skipping 12 lines ... k8s.io/kubernetes/test/e2e/storage.setupLocalVolumes({0x7f870436e3f8, 0xc003527700}, 0xc002eb3950, {0x784f428, 0x11}, 0x0?, 0x1) test/e2e/storage/persistent_volumes-local.go:844 +0x1a4 k8s.io/kubernetes/test/e2e/storage.setupLocalVolumesPVCsPVs({0x7f870436e3f8, 0xc003527700}, 0xc002eb3950?, {0x784f428, 0x11}, 0x0?, 0x0?, {0x7827855, 0x9}) test/e2e/storage/persistent_volumes-local.go:1110 +0xa5 k8s.io/kubernetes/test/e2e/storage.glob..func23.2.1({0x7f870436e3f8, 0xc003527700}) test/e2e/storage/persistent_volumes-local.go:203 +0x12c [38;5;9m[FAILED][0m in [BeforeEach] - test/e2e/storage/utils/host_exec.go:117 [38;5;243m@ 03/05/23 16:45:37.97[0m [1mSTEP:[0m Cleaning up PVC and PV [38;5;243m@ 03/05/23 16:45:37.97[0m [38;5;13m[PANICKED][0m in [AfterEach] - /usr/local/go/src/runtime/panic.go:260 [38;5;243m@ 03/05/23 16:45:37.97[0m Mar 5 16:45:37.970: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready Mar 5 16:45:38.082: INFO: Condition Ready of node i-06df568f6e5c59fff is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-05 16:42:51 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-05 16:42:57 +0000 UTC}]. Failure Mar 5 16:45:40.195: INFO: Condition Ready of node i-06df568f6e5c59fff is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-05 16:42:51 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-05 16:42:57 +0000 UTC}]. Failure Mar 5 16:45:42.195: INFO: Condition Ready of node i-06df568f6e5c59fff is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-05 16:42:51 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-05 16:42:57 +0000 UTC}]. Failure ... skipping 102 lines ... Mar 5 16:48:38.874: INFO: Logging pods the kubelet thinks is on node i-056ad61525499081e Mar 5 16:48:39.108: INFO: cleanup20-16a6f148-bb85-4b73-99f5-2adb1cfdeea0-sqrjl started at 2023-03-05 16:40:29 +0000 UTC (0+1 container statuses recorded) Mar 5 16:48:39.108: INFO: Container cleanup20-16a6f148-bb85-4b73-99f5-2adb1cfdeea0 ready: true, restart count 0 Mar 5 16:48:39.108: INFO: cleanup20-16a6f148-bb85-4b73-99f5-2adb1cfdeea0-n9vbn started at 2023-03-05 16:40:29 +0000 UTC (0+1 container statuses recorded) Mar 5 16:48:39.108: INFO: Container cleanup20-16a6f148-bb85-4b73-99f5-2adb1cfdeea0 ready: true, restart count 0 Mar 5 16:48:39.108: INFO: fail-once-local-gj678 started at 2023-03-05 16:46:33 +0000 UTC (0+1 container statuses recorded) Mar 5 16:48:39.108: INFO: Container c ready: false, restart count 1 Mar 5 16:48:39.108: INFO: cleanup20-16a6f148-bb85-4b73-99f5-2adb1cfdeea0-m6dgl started at 2023-03-05 16:48:01 +0000 UTC (0+1 container statuses recorded) Mar 5 16:48:39.108: INFO: Container cleanup20-16a6f148-bb85-4b73-99f5-2adb1cfdeea0 ready: true, restart count 0 Mar 5 16:48:39.108: INFO: cleanup20-16a6f148-bb85-4b73-99f5-2adb1cfdeea0-cqffq started at 2023-03-05 16:48:02 +0000 UTC (0+1 container statuses recorded) Mar 5 16:48:39.108: INFO: Container cleanup20-16a6f148-bb85-4b73-99f5-2adb1cfdeea0 ready: true, restart count 0 Mar 5 16:48:39.108: INFO: ebs-csi-node-n98cq started at 2023-03-05 16:25:49 +0000 UTC (0+3 container statuses recorded) ... skipping 2 lines ... Mar 5 16:48:39.108: INFO: Container node-driver-registrar ready: true, restart count 0 Mar 5 16:48:39.108: INFO: cilium-vfgw4 started at 2023-03-05 16:25:49 +0000 UTC (1+1 container statuses recorded) Mar 5 16:48:39.108: INFO: Init container clean-cilium-state ready: true, restart count 0 Mar 5 16:48:39.108: INFO: Container cilium-agent ready: true, restart count 0 Mar 5 16:48:39.108: INFO: sample-webhook-deployment-7497495989-s9585 started at 2023-03-05 16:46:38 +0000 UTC (0+1 container statuses recorded) Mar 5 16:48:39.108: INFO: Container sample-webhook ready: true, restart count 0 Mar 5 16:48:39.108: INFO: fail-once-local-kdbgv started at 2023-03-05 16:46:33 +0000 UTC (0+1 container statuses recorded) Mar 5 16:48:39.108: INFO: Container c ready: false, restart count 1 Mar 5 16:48:39.108: INFO: cleanup20-16a6f148-bb85-4b73-99f5-2adb1cfdeea0-xsqgf started at 2023-03-05 16:40:29 +0000 UTC (0+1 container statuses recorded) Mar 5 16:48:39.108: INFO: Container cleanup20-16a6f148-bb85-4b73-99f5-2adb1cfdeea0 ready: true, restart count 0 Mar 5 16:48:39.108: INFO: cleanup20-16a6f148-bb85-4b73-99f5-2adb1cfdeea0-6lnc6 started at 2023-03-05 16:48:01 +0000 UTC (0+1 container statuses recorded) Mar 5 16:48:39.108: INFO: Container cleanup20-16a6f148-bb85-4b73-99f5-2adb1cfdeea0 ready: true, restart count 0 Mar 5 16:48:39.108: INFO: cleanup20-16a6f148-bb85-4b73-99f5-2adb1cfdeea0-9qlxz started at 2023-03-05 16:48:01 +0000 UTC (0+1 container statuses recorded) ... skipping 3 lines ... Mar 5 16:48:39.108: INFO: coredns-autoscaler-8495f5575-plvdw started at 2023-03-05 16:25:57 +0000 UTC (0+1 container statuses recorded) Mar 5 16:48:39.108: INFO: Container autoscaler ready: true, restart count 0 Mar 5 16:48:39.108: INFO: cleanup20-16a6f148-bb85-4b73-99f5-2adb1cfdeea0-kgbnw started at 2023-03-05 16:40:29 +0000 UTC (0+1 container statuses recorded) Mar 5 16:48:39.108: INFO: Container cleanup20-16a6f148-bb85-4b73-99f5-2adb1cfdeea0 ready: true, restart count 0 Mar 5 16:48:39.108: INFO: cleanup20-16a6f148-bb85-4b73-99f5-2adb1cfdeea0-42w6f started at 2023-03-05 16:40:29 +0000 UTC (0+1 container statuses recorded) Mar 5 16:48:39.108: INFO: Container cleanup20-16a6f148-bb85-4b73-99f5-2adb1cfdeea0 ready: true, restart count 0 Mar 5 16:48:39.108: INFO: fail-once-local-69mfr started at 2023-03-05 16:46:28 +0000 UTC (0+1 container statuses recorded) Mar 5 16:48:39.108: INFO: Container c ready: false, restart count 1 Mar 5 16:48:39.108: INFO: cleanup20-16a6f148-bb85-4b73-99f5-2adb1cfdeea0-wmjdk started at 2023-03-05 16:40:29 +0000 UTC (0+1 container statuses recorded) Mar 5 16:48:39.108: INFO: Container cleanup20-16a6f148-bb85-4b73-99f5-2adb1cfdeea0 ready: true, restart count 0 Mar 5 16:48:39.108: INFO: ebs-csi-controller-79785c665b-7dmsf started at 2023-03-05 16:25:57 +0000 UTC (0+5 container statuses recorded) Mar 5 16:48:39.108: INFO: Container csi-attacher ready: true, restart count 0 Mar 5 16:48:39.108: INFO: Container csi-provisioner ready: true, restart count 0 ... skipping 15 lines ... Mar 5 16:48:39.108: INFO: cleanup20-16a6f148-bb85-4b73-99f5-2adb1cfdeea0-j6cks started at 2023-03-05 16:48:01 +0000 UTC (0+1 container statuses recorded) Mar 5 16:48:39.108: INFO: Container cleanup20-16a6f148-bb85-4b73-99f5-2adb1cfdeea0 ready: true, restart count 0 Mar 5 16:48:39.108: INFO: cleanup20-16a6f148-bb85-4b73-99f5-2adb1cfdeea0-khffj started at 2023-03-05 16:40:29 +0000 UTC (0+1 container statuses recorded) Mar 5 16:48:39.108: INFO: Container cleanup20-16a6f148-bb85-4b73-99f5-2adb1cfdeea0 ready: true, restart count 0 Mar 5 16:48:39.108: INFO: nodeport-test-88ggz started at 2023-03-05 16:40:39 +0000 UTC (0+1 container statuses recorded) Mar 5 16:48:39.108: INFO: Container nodeport-test ready: true, restart count 0 Mar 5 16:48:39.108: INFO: fail-once-local-2mt4w started at 2023-03-05 16:46:28 +0000 UTC (0+1 container statuses recorded) Mar 5 16:48:39.108: INFO: Container c ready: false, restart count 1 Mar 5 16:48:39.108: INFO: cleanup20-16a6f148-bb85-4b73-99f5-2adb1cfdeea0-gnfq6 started at 2023-03-05 16:48:01 +0000 UTC (0+1 container statuses recorded) Mar 5 16:48:39.108: INFO: Container cleanup20-16a6f148-bb85-4b73-99f5-2adb1cfdeea0 ready: true, restart count 0 Mar 5 16:48:39.108: INFO: cleanup20-16a6f148-bb85-4b73-99f5-2adb1cfdeea0-xkdzr started at 2023-03-05 16:40:29 +0000 UTC (0+1 container statuses recorded) Mar 5 16:48:39.108: INFO: Container cleanup20-16a6f148-bb85-4b73-99f5-2adb1cfdeea0 ready: true, restart count 0 Mar 5 16:48:39.108: INFO: ss-0 started at 2023-03-05 16:33:11 +0000 UTC (0+1 container statuses recorded) ... skipping 10 lines ... Logging node info for node i-06df568f6e5c59fff Mar 5 16:48:39.682: INFO: Node Info: &Node{ObjectMeta:{i-06df568f6e5c59fff cb63c681-2e18-474a-b6f3-221325ac0d52 17637 0 2023-03-05 16:24:16 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:arm64 beta.kubernetes.io/instance-type:t4g.small beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:eu-central-1 failure-domain.beta.kubernetes.io/zone:eu-central-1a karpenter.k8s.aws/instance-category:t karpenter.k8s.aws/instance-cpu:2 karpenter.k8s.aws/instance-family:t4g karpenter.k8s.aws/instance-generation:4 karpenter.k8s.aws/instance-hypervisor:nitro karpenter.k8s.aws/instance-memory:2048 karpenter.k8s.aws/instance-pods:11 karpenter.k8s.aws/instance-size:small karpenter.sh/capacity-type:spot karpenter.sh/initialized:true karpenter.sh/provisioner-name:nodes kubelet_cleanup:true kubernetes.io/arch:arm64 kubernetes.io/hostname:i-06df568f6e5c59fff kubernetes.io/os:linux node-role.kubernetes.io/node: node-role.kubernetes.io/spot-worker:true node.kubernetes.io/instance-type:t4g.small topology.ebs.csi.aws.com/zone:eu-central-1a topology.hostpath.csi/node:i-06df568f6e5c59fff topology.kubernetes.io/region:eu-central-1 topology.kubernetes.io/zone:eu-central-1a] map[csi.volume.kubernetes.io/nodeid:{"csi-hostpath-ephemeral-2147":"i-06df568f6e5c59fff","csi-hostpath-ephemeral-5408":"i-06df568f6e5c59fff","csi-hostpath-ephemeral-6540":"i-06df568f6e5c59fff","csi-mock-csi-mock-volumes-fsgroup-policy-1058":"i-06df568f6e5c59fff","csi-mock-csi-mock-volumes-workload-2219":"i-06df568f6e5c59fff","ebs.csi.aws.com":"i-06df568f6e5c59fff"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [{karpenter.sh/v1alpha5 Provisioner nodes ac5d7599-b2a2-4eae-8da4-1f4c0a54ec82 <nil> 0xc003a94b60}] [karpenter.sh/termination] [{aws-cloud-controller-manager Update v1 2023-03-05 16:24:16 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{}}}} } {kops-controller Update v1 2023-03-05 16:24:16 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{},"f:node-role.kubernetes.io/spot-worker":{}}}} } {karpenter Update v1 2023-03-05 16:25:57 +0000 UTC FieldsV1 {"f:metadata":{"f:finalizers":{".":{},"v:\"karpenter.sh/termination\"":{}},"f:labels":{".":{},"f:karpenter.k8s.aws/instance-category":{},"f:karpenter.k8s.aws/instance-cpu":{},"f:karpenter.k8s.aws/instance-family":{},"f:karpenter.k8s.aws/instance-generation":{},"f:karpenter.k8s.aws/instance-hypervisor":{},"f:karpenter.k8s.aws/instance-memory":{},"f:karpenter.k8s.aws/instance-pods":{},"f:karpenter.k8s.aws/instance-size":{},"f:karpenter.sh/capacity-type":{},"f:karpenter.sh/initialized":{},"f:karpenter.sh/provisioner-name":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}},"f:ownerReferences":{".":{},"k:{\"uid\":\"ac5d7599-b2a2-4eae-8da4-1f4c0a54ec82\"}":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-03-05 16:26:48 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{".":{},"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"InternalIP\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {e2e.test Update v1 2023-03-05 16:40:28 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubelet_cleanup":{}}}} } {kube-controller-manager Update v1 2023-03-05 16:44:29 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{".":{},"k:{\"type\":\"DiskPressure\"}":{".":{},"f:type":{}},"k:{\"type\":\"MemoryPressure\"}":{".":{},"f:type":{}},"k:{\"type\":\"PIDPressure\"}":{".":{},"f:type":{}},"k:{\"type\":\"Ready\"}":{".":{},"f:type":{}}},"f:volumesAttached":{}}} status} {kube-controller-manager Update v1 2023-03-05 16:46:37 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:node.alpha.kubernetes.io/ttl":{}},"f:labels":{"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.2.0/24\"":{}},"f:taints":{}}} } {kubelet Update v1 2023-03-05 16:47:16 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{"f:kubernetes.io/hostname":{},"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{".":{},"f:cpu":{},"f:ephemeral-storage":{},"f:hugepages-1Gi":{},"f:hugepages-2Mi":{},"f:hugepages-32Mi":{},"f:hugepages-64Ki":{},"f:memory":{},"f:pods":{}},"f:capacity":{".":{},"f:cpu":{},"f:ephemeral-storage":{},"f:hugepages-1Gi":{},"f:hugepages-2Mi":{},"f:hugepages-32Mi":{},"f:hugepages-64Ki":{},"f:memory":{},"f:pods":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:daemonEndpoints":{"f:kubeletEndpoint":{"f:Port":{}}},"f:images":{},"f:nodeInfo":{"f:architecture":{},"f:bootID":{},"f:containerRuntimeVersion":{},"f:kernelVersion":{},"f:kubeProxyVersion":{},"f:kubeletVersion":{},"f:machineID":{},"f:operatingSystem":{},"f:osImage":{},"f:systemUUID":{}},"f:volumesInUse":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.2.0/24,DoNotUseExternalID:,ProviderID:aws:///eu-central-1a/i-06df568f6e5c59fff,Unschedulable:false,Taints:[]Taint{Taint{Key:node.kubernetes.io/not-ready,Value:,Effect:NoSchedule,TimeAdded:2023-03-05 16:46:33 +0000 UTC,},Taint{Key:node.kubernetes.io/not-ready,Value:,Effect:NoExecute,TimeAdded:2023-03-05 16:46:37 +0000 UTC,},},ConfigSource:nil,PodCIDRs:[100.96.2.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49756430336 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-32Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-64Ki: {{0 0} {<nil>} 0 DecimalSI},memory: {{1941540864 0} {<nil>} 1896036Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44780787229 0} {<nil>} 44780787229 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-32Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-64Ki: {{0 0} {<nil>} 0 DecimalSI},memory: {{1836683264 0} {<nil>} 1793636Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:Ready,Status:False,LastHeartbeatTime:2023-03-05 16:47:15 +0000 UTC,LastTransitionTime:2023-03-05 16:46:08 +0000 UTC,Reason:KubeletNotReady,Message:PLEG is not healthy: pleg was last seen active 8m10.540757618s ago; threshold is 3m0s,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-03-05 16:47:12 +0000 UTC,LastTransitionTime:2023-03-05 16:47:12 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-03-05 16:47:15 +0000 UTC,LastTransitionTime:2023-03-05 16:47:15 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-03-05 16:47:15 +0000 UTC,LastTransitionTime:2023-03-05 16:47:15 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.43.148,},NodeAddress{Type:ExternalIP,Address:18.192.211.222,},NodeAddress{Type:InternalDNS,Address:i-06df568f6e5c59fff.eu-central-1.compute.internal,},NodeAddress{Type:Hostname,Address:i-06df568f6e5c59fff.eu-central-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-18-192-211-222.eu-central-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec221221864d9409a8ec5f6614a9b94b,SystemUUID:ec221221-864d-9409-a8ec-5f6614a9b94b,BootID:553d31bf-faca-417e-a3db-325839e1b8e4,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 22.04.2 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.27.0-alpha.3.52+d48b8167f7094b,KubeProxyVersion:v1.27.0-alpha.3.52+d48b8167f7094b,OperatingSystem:linux,Architecture:arm64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:157636062,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/jessie-dnsutils@sha256:24aaf2626d6b27864c29de2097e8bbb840b3a414271bf7c8995e431e47d8408e registry.k8s.io/e2e-test-images/jessie-dnsutils:1.7],SizeBytes:116764683,},ContainerImage{Names:[registry.k8s.io/kube-proxy-arm64:v1.27.0-alpha.3.52_d48b8167f7094b],SizeBytes:60554082,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:16bbf38c463a4223d8cfe4da12bc61010b082a79b4bb003e2d3ba3ece5dd5f9e registry.k8s.io/e2e-test-images/agnhost:2.43],SizeBytes:49808364,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:3fe7acf013d1264ffded116b80a73dc129a449b0fccdb8d21af8279f2233f36e registry.k8s.io/e2e-test-images/httpd:2.4.39-4],SizeBytes:41528818,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:148b022f5c5da426fc2f3c14b5c0867e58ef05961510c84749ac1fddcb0fef22 registry.k8s.io/e2e-test-images/httpd:2.4.38-4],SizeBytes:40343601,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:28932904,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:ee3b525d5b89db99da3b8eb521d9cd90cb6e9ef0fbb651e98bb37be78d36b5b8 registry.k8s.io/sig-storage/csi-provisioner:v3.3.0],SizeBytes:23945474,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:425d8f1b769398127767b06ed97ce62578a3179bcb99809ce93a1649e025ffe7 registry.k8s.io/sig-storage/csi-resizer:v1.6.0],SizeBytes:22678632,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:291334908ddf71a4661fd7f6d9d97274de8a5378a2b6fdfeb2ce73414a34f82f registry.k8s.io/sig-storage/csi-snapshotter:v6.1.0],SizeBytes:22434180,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:9a685020911e2725ad019dbce6e4a5ab93d51e3d4557f115e64343345e05781b registry.k8s.io/sig-storage/csi-attacher:v4.0.0],SizeBytes:22394501,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:122bfb8c1edabb3c0edd63f06523e6940d958d19b3957dc7b1d6f81e9f1f6119 registry.k8s.io/sig-storage/csi-provisioner:v3.1.0],SizeBytes:21735644,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:9ebbf9f023e7b41ccee3d52afe39a89e3ddacdbb69269d583abfc25847cfd9e4 registry.k8s.io/sig-storage/csi-resizer:v1.4.0],SizeBytes:20848413,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:8b9c313c05f54fb04f8d430896f5f5904b6cb157df261501b29adc04d2b2dc7b registry.k8s.io/sig-storage/csi-attacher:v3.4.0],SizeBytes:20565480,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nonroot@sha256:ee9f50b3c64b174d296d91ca9f69a914ac30e59095dfb462b2b518ad28a63655 registry.k8s.io/e2e-test-images/nonroot:1.4],SizeBytes:18561972,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:92257881c1d6493cf18299a24af42330f891166560047902b8d431fb66b01af5 registry.k8s.io/sig-storage/hostpathplugin:v1.9.0],SizeBytes:18267611,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:9b2d9abda017c28b12f84a344f57cd73fbdb6c2bd7dd5bdd5018246ad1093ba6 registry.k8s.io/sig-storage/hostpathplugin:v1.11.0],SizeBytes:18159085,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:017727efcfeb7d053af68e51436ce8e65edbc6ca573720afb4f79c8594036955 registry.k8s.io/coredns/coredns:v1.10.0],SizeBytes:13766234,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:8502345,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:933940f13b3ea0abc62e656c1aa5c5b47c04b15d71250413a6b821bd0c58b94e registry.k8s.io/sig-storage/livenessprobe:v2.7.0],SizeBytes:8223819,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:7685305,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:5c99cf6a02adda929b10321dbf4ecfa00d87be9ba4fb456006237d530ab4baa1 registry.k8s.io/e2e-test-images/nginx:1.14-4],SizeBytes:6860089,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nonewprivs@sha256:8ac1264691820febacf3aea5d152cbde6d10685731ec14966a9401c6f47a68ac registry.k8s.io/e2e-test-images/nonewprivs:1.3],SizeBytes:5025027,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:2e0f836850e09b8b7cc937681d6194537a09fbd5f6b9e08f4d646a85128e8937 registry.k8s.io/e2e-test-images/busybox:1.29-4],SizeBytes:772933,},ContainerImage{Names:[registry.k8s.io/pause@sha256:7031c1b283388d2c2e09b57badb803c05ebed362dc88d84b480cc47f72a21097 registry.k8s.io/pause:3.9],SizeBytes:268051,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:253553,},},VolumesInUse:[kubernetes.io/csi/csi-hostpath-ephemeral-2147^52df5953-bb73-11ed-8aa7-be49170871b4 kubernetes.io/csi/csi-hostpath-ephemeral-5408^443d1daf-bb73-11ed-95c4-5a95d17cbc27 kubernetes.io/csi/csi-hostpath-ephemeral-5408^44411f51-bb73-11ed-95c4-5a95d17cbc27 kubernetes.io/csi/csi-hostpath-ephemeral-6540^4426a07f-bb73-11ed-a604-92687f2c50f8 kubernetes.io/csi/ebs.csi.aws.com^vol-05b67fab5beb6394b],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-05b67fab5beb6394b,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/csi-hostpath-ephemeral-6540^4426a07f-bb73-11ed-a604-92687f2c50f8,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/csi-hostpath-ephemeral-5408^44411f51-bb73-11ed-95c4-5a95d17cbc27,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/csi-hostpath-ephemeral-5408^443d1daf-bb73-11ed-95c4-5a95d17cbc27,DevicePath:,},},Config:nil,},} Mar 5 16:48:39.683: INFO: Logging kubelet events for node i-06df568f6e5c59fff Mar 5 16:48:39.798: INFO: Logging pods the kubelet thinks is on node i-06df568f6e5c59fff Mar 5 16:49:00.625: INFO: Unable to retrieve kubelet pods for node i-06df568f6e5c59fff: error trying to reach service: unexpected EOF Mar 5 16:49:00.625: INFO: Logging node info for node i-0d6a32f749839192f Mar 5 16:49:00.737: INFO: Node Info: &Node{ObjectMeta:{i-0d6a32f749839192f 4155fefb-4d3a-4b04-ba53-6ac9e895ff92 16414 0 2023-03-05 16:20:46 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:arm64 beta.kubernetes.io/instance-type:c6g.xlarge beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:eu-central-1 failure-domain.beta.kubernetes.io/zone:eu-central-1a kops.k8s.io/kops-controller-pki: kubernetes.io/arch:arm64 kubernetes.io/hostname:i-0d6a32f749839192f kubernetes.io/os:linux node-role.kubernetes.io/control-plane: node.kubernetes.io/exclude-from-external-load-balancers: node.kubernetes.io/instance-type:c6g.xlarge topology.ebs.csi.aws.com/zone:eu-central-1a topology.kubernetes.io/region:eu-central-1 topology.kubernetes.io/zone:eu-central-1a] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-0d6a32f749839192f"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kubelet Update v1 2023-03-05 16:20:46 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {protokube Update v1 2023-03-05 16:21:25 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kops.k8s.io/kops-controller-pki":{},"f:node-role.kubernetes.io/control-plane":{},"f:node.kubernetes.io/exclude-from-external-load-balancers":{}}}} } {kube-controller-manager Update v1 2023-03-05 16:21:47 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.0.0/24\"":{}}}} } {aws-cloud-controller-manager Update v1 2023-03-05 16:21:48 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{},"f:taints":{}}} } {aws-cloud-controller-manager Update v1 2023-03-05 16:21:48 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kubelet Update v1 2023-03-05 16:44:12 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.0.0/24,DoNotUseExternalID:,ProviderID:aws:///eu-central-1a/i-0d6a32f749839192f,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/control-plane,Value:,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[100.96.0.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{4 0} {<nil>} 4 DecimalSI},ephemeral-storage: {{49756430336 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-32Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-64Ki: {{0 0} {<nil>} 0 DecimalSI},memory: {{8150118400 0} {<nil>} 7959100Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{4 0} {<nil>} 4 DecimalSI},ephemeral-storage: {{44780787229 0} {<nil>} 44780787229 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-32Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-64Ki: {{0 0} {<nil>} 0 DecimalSI},memory: {{8045260800 0} {<nil>} 7856700Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-03-05 16:44:12 +0000 UTC,LastTransitionTime:2023-03-05 16:20:46 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-03-05 16:44:12 +0000 UTC,LastTransitionTime:2023-03-05 16:20:46 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-03-05 16:44:12 +0000 UTC,LastTransitionTime:2023-03-05 16:20:46 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-03-05 16:44:12 +0000 UTC,LastTransitionTime:2023-03-05 16:21:47 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.49.247,},NodeAddress{Type:ExternalIP,Address:18.197.20.82,},NodeAddress{Type:InternalDNS,Address:i-0d6a32f749839192f.eu-central-1.compute.internal,},NodeAddress{Type:Hostname,Address:i-0d6a32f749839192f.eu-central-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-18-197-20-82.eu-central-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2bed249e8ac7b517cb08ccd42b83e5,SystemUUID:ec2bed24-9e8a-c7b5-17cb-08ccd42b83e5,BootID:6f9215c9-dfe8-4b13-84f4-9b2084456198,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 22.04.2 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.27.0-alpha.3.52+d48b8167f7094b,KubeProxyVersion:v1.27.0-alpha.3.52+d48b8167f7094b,OperatingSystem:linux,Architecture:arm64,},Images:[]ContainerImage{ContainerImage{Names:[registry.k8s.io/etcd@sha256:9ce33ba33d8e738a5b85ed50b5080ac746deceed4a7496c550927a7a19ca3b6d registry.k8s.io/etcd:3.5.0-0],SizeBytes:157797353,},ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:157636062,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:4afb99b4690b418ffc2ceb67e1a17376457e441c1f09ab55447f0aaf992fa646 registry.k8s.io/etcd:3.4.3-0],SizeBytes:151677080,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:4ad90a11b55313b182afc186b9876c8e891531b8db4c9bf1541953021618d0e2 registry.k8s.io/etcd:3.4.13-0],SizeBytes:134531559,},ContainerImage{Names:[registry.k8s.io/kube-apiserver-arm64:v1.27.0-alpha.3.52_d48b8167f7094b],SizeBytes:131087707,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:0d2ab6b15b4fe0b903b9e81192d942fb39c29be61fe341cd16db5d0165b0435c registry.k8s.io/etcd:3.3.17-0],SizeBytes:126979107,},ContainerImage{Names:[registry.k8s.io/kube-controller-manager-arm64:v1.27.0-alpha.3.52_d48b8167f7094b],SizeBytes:120667128,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:17da501f5d2a675be46040422a27b7cc21b8a43895ac998b171db1c346f361f7 registry.k8s.io/etcd:3.3.10-0],SizeBytes:81436097,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:6f72b851544986cb0921b53ea655ec04c36131248f16d4ad110cb3ca0c369dc1 registry.k8s.io/etcd:3.5.4-0],SizeBytes:81120118,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:13f53ed1d91e2e11aac476ee9a0269fdda6cc4874eba903efd40daf50c55eee5 registry.k8s.io/etcd:3.5.3-0],SizeBytes:81114891,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:51eae8381dcb1078289fa7b4f3df2630cdc18d09fb56f8e56b41c40e191d6c83 registry.k8s.io/etcd:3.5.7-0],SizeBytes:80665728,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:dd75ec974b0a2a6f6bb47001ba09207976e625db898d1b16735528c009cb171c registry.k8s.io/etcd:3.5.6-0],SizeBytes:80539316,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:905d7ca17fd02bc24c0eba9a062753aba15db3e31422390bc3238eb762339b20 registry.k8s.io/etcd:3.2.24-1],SizeBytes:61500433,},ContainerImage{Names:[registry.k8s.io/etcdadm/etcd-manager-slim@sha256:8727adc059334e0e3329dbccce116a734da4a090da5e86c8cddccaf333caa666 registry.k8s.io/etcdadm/etcd-manager-slim:v3.0.20230201],SizeBytes:60826655,},ContainerImage{Names:[registry.k8s.io/kube-proxy-arm64:v1.27.0-alpha.3.52_d48b8167f7094b],SizeBytes:60554082,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:64b9ea357325d5db9f8a723dcf503b5a449177b17ac87d69481e126bb724c263 registry.k8s.io/etcd:3.5.1-0],SizeBytes:59553904,},ContainerImage{Names:[registry.k8s.io/kube-scheduler-arm64:v1.27.0-alpha.3.52_d48b8167f7094b],SizeBytes:56310763,},ContainerImage{Names:[registry.k8s.io/kops/kops-controller:1.27.0-alpha.1],SizeBytes:41797411,},ContainerImage{Names:[registry.k8s.io/kops/dns-controller:1.27.0-alpha.1],SizeBytes:40267473,},ContainerImage{Names:[public.ecr.aws/karpenter/controller@sha256:68db4f092cf9cc83f5ef9e2fbc5407c2cb682e81f64dfaa700a7602ede38b1cf public.ecr.aws/karpenter/controller:v0.16.3],SizeBytes:33707751,},ContainerImage{Names:[public.ecr.aws/karpenter/webhook@sha256:96a2d9b06d6bc5127801f358f74b1cf2d289b423a2e9ba40c573c0b14b17dafa public.ecr.aws/karpenter/webhook:v0.16.3],SizeBytes:33307187,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:28932904,},ContainerImage{Names:[quay.io/cilium/operator@sha256:a6d24a006a6b92967ac90786b49bc1ac26e5477cf028cd1186efcfc2466484db quay.io/cilium/operator:v1.12.5],SizeBytes:24567287,},ContainerImage{Names:[registry.k8s.io/provider-aws/cloud-controller-manager@sha256:fdeb61e3e42ecd9cca868d550ebdb88dd6341d9e91fcfa9a37e227dab2ad22cb registry.k8s.io/provider-aws/cloud-controller-manager:v1.26.0],SizeBytes:18345226,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:8502345,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:7685305,},ContainerImage{Names:[registry.k8s.io/kops/kube-apiserver-healthcheck:1.27.0-alpha.1],SizeBytes:4713612,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:253553,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Mar 5 16:49:00.737: INFO: Logging kubelet events for node i-0d6a32f749839192f Mar 5 16:49:00.852: INFO: ... skipping 52 lines ... Mar 5 16:49:00.973: INFO: Container controller ready: true, restart count 0 Mar 5 16:49:01.467: INFO: Latency metrics for node i-0d6a32f749839192f [1mSTEP:[0m Destroying namespace "persistent-local-volumes-test-7040" for this suite. [38;5;243m@ 03/05/23 16:49:01.467[0m [38;5;243m<< Timeline[0m [38;5;9m[FAILED] Timed out after 300.066s. Expected Pod to be in <v1.PodPhase>: "Running" Got instead: <*v1.Pod | 0xc00356bb00>: metadata: creationTimestamp: "2023-03-05T16:40:37Z" generateName: hostexec-i-06df568f6e5c59fff- ... skipping 133 lines ... phase: Pending qosClass: BestEffort[0m [38;5;9mIn [1m[BeforeEach][0m[38;5;9m at: [1mtest/e2e/storage/utils/host_exec.go:117[0m [38;5;243m@ 03/05/23 16:45:37.97[0m There were [1m[38;5;9madditional failures[0m detected. To view them in detail run [1mginkgo -vv[0m [38;5;243m------------------------------[0m [38;5;9m• [FAILED] [514.416 seconds][0m [0m[sig-network] Services [38;5;9m[1m[It] should be able to create a functioning NodePort service [Conformance][0m [38;5;243mtest/e2e/network/service.go:1280[0m [38;5;243mTimeline >>[0m [1mSTEP:[0m Creating a kubernetes client [38;5;243m@ 03/05/23 16:40:38.67[0m Mar 5 16:40:38.670: INFO: >>> kubeConfig: /root/.kube/config ... skipping 103 lines ... I0305 16:45:33.916277 7334 runners.go:194] nodeport-test Pods: 2 out of 2 created, 1 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0305 16:45:36.917146 7334 runners.go:194] nodeport-test Pods: 2 out of 2 created, 1 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0305 16:45:39.917948 7334 runners.go:194] nodeport-test Pods: 2 out of 2 created, 1 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0305 16:45:42.918683 7334 runners.go:194] nodeport-test Pods: 2 out of 2 created, 1 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0305 16:45:43.031936 7334 runners.go:194] Pod nodeport-test-88ggz i-056ad61525499081e Running <nil> I0305 16:45:43.032009 7334 runners.go:194] Pod nodeport-test-l7dw2 i-06df568f6e5c59fff Pending <nil> Mar 5 16:45:43.032: INFO: Unexpected error: <*errors.errorString | 0xc001184740>: { s: "only 1 pods started out of 2", } [38;5;9m[FAILED][0m in [It] - test/e2e/network/service.go:1295 [38;5;243m@ 03/05/23 16:45:43.032[0m Mar 5 16:45:43.032: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready Mar 5 16:45:43.145: INFO: Condition Ready of node i-06df568f6e5c59fff is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-05 16:42:51 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-05 16:42:57 +0000 UTC}]. Failure Mar 5 16:45:45.260: INFO: Condition Ready of node i-06df568f6e5c59fff is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-05 16:42:51 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-05 16:42:57 +0000 UTC}]. Failure Mar 5 16:45:47.259: INFO: Condition Ready of node i-06df568f6e5c59fff is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-05 16:42:51 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-05 16:42:57 +0000 UTC}]. Failure Mar 5 16:45:49.259: INFO: Condition Ready of node i-06df568f6e5c59fff is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-05 16:42:51 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-05 16:42:57 +0000 UTC}]. Failure Mar 5 16:45:51.259: INFO: Condition Ready of node i-06df568f6e5c59fff is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-05 16:42:51 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-05 16:42:57 +0000 UTC}]. Failure ... skipping 103 lines ... Mar 5 16:48:43.487: INFO: At 2023-03-05 16:48:03 +0000 UTC - event for nodeport-test-nd24m: {kubelet i-056ad61525499081e} Pulled: Container image "registry.k8s.io/e2e-test-images/agnhost:2.43" already present on machine Mar 5 16:48:43.600: INFO: POD NODE PHASE GRACE CONDITIONS Mar 5 16:48:43.600: INFO: nodeport-test-88ggz i-056ad61525499081e Running [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2023-03-05 16:40:39 +0000 UTC } {Ready True 0001-01-01 00:00:00 +0000 UTC 2023-03-05 16:40:41 +0000 UTC } {ContainersReady True 0001-01-01 00:00:00 +0000 UTC 2023-03-05 16:40:41 +0000 UTC } {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2023-03-05 16:40:39 +0000 UTC }] Mar 5 16:48:43.600: INFO: nodeport-test-l7dw2 i-06df568f6e5c59fff Pending 1s [{PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2023-03-05 16:40:39 +0000 UTC } {DisruptionTarget True 0001-01-01 00:00:00 +0000 UTC 2023-03-05 16:47:57 +0000 UTC DeletionByTaintManager Taint manager: deleting due to NoExecute taint}] Mar 5 16:48:43.600: INFO: nodeport-test-nd24m i-056ad61525499081e Running [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2023-03-05 16:48:01 +0000 UTC } {Ready True 0001-01-01 00:00:00 +0000 UTC 2023-03-05 16:48:03 +0000 UTC } {ContainersReady True 0001-01-01 00:00:00 +0000 UTC 2023-03-05 16:48:03 +0000 UTC } {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2023-03-05 16:48:01 +0000 UTC }] Mar 5 16:48:43.600: INFO: Mar 5 16:49:00.626: INFO: Unable to fetch services-2335/nodeport-test-l7dw2/nodeport-test logs: an error on the server ("unknown") has prevented the request from succeeding (get pods nodeport-test-l7dw2) Mar 5 16:49:00.854: INFO: Logging node info for node i-056ad61525499081e Mar 5 16:49:00.966: INFO: Node Info: &Node{ObjectMeta:{i-056ad61525499081e a6ce9ed3-5ae8-4065-b117-6884e24b3632 18352 0 2023-03-05 16:24:16 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:arm64 beta.kubernetes.io/instance-type:t4g.small beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:eu-central-1 failure-domain.beta.kubernetes.io/zone:eu-central-1a karpenter.k8s.aws/instance-category:t karpenter.k8s.aws/instance-cpu:2 karpenter.k8s.aws/instance-family:t4g karpenter.k8s.aws/instance-generation:4 karpenter.k8s.aws/instance-hypervisor:nitro karpenter.k8s.aws/instance-memory:2048 karpenter.k8s.aws/instance-pods:11 karpenter.k8s.aws/instance-size:small karpenter.sh/capacity-type:spot karpenter.sh/initialized:true karpenter.sh/provisioner-name:nodes kubernetes.io/arch:arm64 kubernetes.io/hostname:i-056ad61525499081e kubernetes.io/os:linux node-role.kubernetes.io/node: node-role.kubernetes.io/spot-worker:true node.kubernetes.io/instance-type:t4g.small topology.ebs.csi.aws.com/zone:eu-central-1a topology.hostpath.csi/node:i-056ad61525499081e topology.kubernetes.io/region:eu-central-1 topology.kubernetes.io/zone:eu-central-1a] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-056ad61525499081e"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [{karpenter.sh/v1alpha5 Provisioner nodes ac5d7599-b2a2-4eae-8da4-1f4c0a54ec82 <nil> 0xc002b0fc70}] [karpenter.sh/termination] [{aws-cloud-controller-manager Update v1 2023-03-05 16:24:16 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{}}}} } {kops-controller Update v1 2023-03-05 16:24:16 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{},"f:node-role.kubernetes.io/spot-worker":{}}}} } {karpenter Update v1 2023-03-05 16:25:57 +0000 UTC FieldsV1 {"f:metadata":{"f:finalizers":{".":{},"v:\"karpenter.sh/termination\"":{}},"f:labels":{".":{},"f:karpenter.k8s.aws/instance-category":{},"f:karpenter.k8s.aws/instance-cpu":{},"f:karpenter.k8s.aws/instance-family":{},"f:karpenter.k8s.aws/instance-generation":{},"f:karpenter.k8s.aws/instance-hypervisor":{},"f:karpenter.k8s.aws/instance-memory":{},"f:karpenter.k8s.aws/instance-pods":{},"f:karpenter.k8s.aws/instance-size":{},"f:karpenter.sh/capacity-type":{},"f:karpenter.sh/initialized":{},"f:karpenter.sh/provisioner-name":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}},"f:ownerReferences":{".":{},"k:{\"uid\":\"ac5d7599-b2a2-4eae-8da4-1f4c0a54ec82\"}":{}}},"f:spec":{"f:providerID":{}}} } {kube-controller-manager Update v1 2023-03-05 16:25:57 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:node.alpha.kubernetes.io/ttl":{}},"f:labels":{"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.1.0/24\"":{}}}} } {aws-cloud-controller-manager Update v1 2023-03-05 16:26:48 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{".":{},"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"InternalIP\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kube-controller-manager Update v1 2023-03-05 16:48:56 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{".":{},"k:{\"type\":\"DiskPressure\"}":{".":{},"f:type":{}},"k:{\"type\":\"MemoryPressure\"}":{".":{},"f:type":{}},"k:{\"type\":\"PIDPressure\"}":{".":{},"f:type":{}},"k:{\"type\":\"Ready\"}":{".":{},"f:type":{}}},"f:volumesAttached":{}}} status} {kubelet Update v1 2023-03-05 16:48:56 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{"f:kubernetes.io/hostname":{},"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{".":{},"f:cpu":{},"f:ephemeral-storage":{},"f:hugepages-1Gi":{},"f:hugepages-2Mi":{},"f:hugepages-32Mi":{},"f:hugepages-64Ki":{},"f:memory":{},"f:pods":{}},"f:capacity":{".":{},"f:cpu":{},"f:ephemeral-storage":{},"f:hugepages-1Gi":{},"f:hugepages-2Mi":{},"f:hugepages-32Mi":{},"f:hugepages-64Ki":{},"f:memory":{},"f:pods":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:daemonEndpoints":{"f:kubeletEndpoint":{"f:Port":{}}},"f:images":{},"f:nodeInfo":{"f:architecture":{},"f:bootID":{},"f:containerRuntimeVersion":{},"f:kernelVersion":{},"f:kubeProxyVersion":{},"f:kubeletVersion":{},"f:machineID":{},"f:operatingSystem":{},"f:osImage":{},"f:systemUUID":{}},"f:volumesInUse":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.1.0/24,DoNotUseExternalID:,ProviderID:aws:///eu-central-1a/i-056ad61525499081e,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.1.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49756430336 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-32Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-64Ki: {{0 0} {<nil>} 0 DecimalSI},memory: {{1941540864 0} {<nil>} 1896036Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44780787229 0} {<nil>} 44780787229 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-32Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-64Ki: {{0 0} {<nil>} 0 DecimalSI},memory: {{1836683264 0} {<nil>} 1793636Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-03-05 16:48:56 +0000 UTC,LastTransitionTime:2023-03-05 16:25:57 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-03-05 16:48:56 +0000 UTC,LastTransitionTime:2023-03-05 16:25:48 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-03-05 16:48:56 +0000 UTC,LastTransitionTime:2023-03-05 16:25:48 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-03-05 16:48:56 +0000 UTC,LastTransitionTime:2023-03-05 16:25:48 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.36.179,},NodeAddress{Type:ExternalIP,Address:18.159.37.200,},NodeAddress{Type:InternalDNS,Address:i-056ad61525499081e.eu-central-1.compute.internal,},NodeAddress{Type:Hostname,Address:i-056ad61525499081e.eu-central-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-18-159-37-200.eu-central-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2af3207d1c8c09785c337aae6c6ef0,SystemUUID:ec2af320-7d1c-8c09-785c-337aae6c6ef0,BootID:786f2752-bc30-4729-8aaf-45977b16c253,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 22.04.2 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.27.0-alpha.3.52+d48b8167f7094b,KubeProxyVersion:v1.27.0-alpha.3.52+d48b8167f7094b,OperatingSystem:linux,Architecture:arm64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:157636062,},ContainerImage{Names:[registry.k8s.io/kube-proxy-arm64:v1.27.0-alpha.3.52_d48b8167f7094b],SizeBytes:60554082,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:16bbf38c463a4223d8cfe4da12bc61010b082a79b4bb003e2d3ba3ece5dd5f9e registry.k8s.io/e2e-test-images/agnhost:2.43],SizeBytes:49808364,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:3fe7acf013d1264ffded116b80a73dc129a449b0fccdb8d21af8279f2233f36e registry.k8s.io/e2e-test-images/httpd:2.4.39-4],SizeBytes:41528818,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:148b022f5c5da426fc2f3c14b5c0867e58ef05961510c84749ac1fddcb0fef22 registry.k8s.io/e2e-test-images/httpd:2.4.38-4],SizeBytes:40343601,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:28932904,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:ee3b525d5b89db99da3b8eb521d9cd90cb6e9ef0fbb651e98bb37be78d36b5b8 registry.k8s.io/sig-storage/csi-provisioner:v3.3.0],SizeBytes:23945474,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:425d8f1b769398127767b06ed97ce62578a3179bcb99809ce93a1649e025ffe7 registry.k8s.io/sig-storage/csi-resizer:v1.6.0],SizeBytes:22678632,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:291334908ddf71a4661fd7f6d9d97274de8a5378a2b6fdfeb2ce73414a34f82f registry.k8s.io/sig-storage/csi-snapshotter:v6.1.0],SizeBytes:22434180,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:9a685020911e2725ad019dbce6e4a5ab93d51e3d4557f115e64343345e05781b registry.k8s.io/sig-storage/csi-attacher:v4.0.0],SizeBytes:22394501,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:122bfb8c1edabb3c0edd63f06523e6940d958d19b3957dc7b1d6f81e9f1f6119 registry.k8s.io/sig-storage/csi-provisioner:v3.1.0],SizeBytes:21735644,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:9ebbf9f023e7b41ccee3d52afe39a89e3ddacdbb69269d583abfc25847cfd9e4 registry.k8s.io/sig-storage/csi-resizer:v1.4.0],SizeBytes:20848413,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:8b9c313c05f54fb04f8d430896f5f5904b6cb157df261501b29adc04d2b2dc7b registry.k8s.io/sig-storage/csi-attacher:v3.4.0],SizeBytes:20565480,},ContainerImage{Names:[registry.k8s.io/cpa/cluster-proportional-autoscaler@sha256:68d396900aeaa072c1f27289485fdac29834045a6f3ffe369bf389d830ef572d registry.k8s.io/cpa/cluster-proportional-autoscaler:1.8.6],SizeBytes:19007623,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:92257881c1d6493cf18299a24af42330f891166560047902b8d431fb66b01af5 registry.k8s.io/sig-storage/hostpathplugin:v1.9.0],SizeBytes:18267611,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:9b2d9abda017c28b12f84a344f57cd73fbdb6c2bd7dd5bdd5018246ad1093ba6 registry.k8s.io/sig-storage/hostpathplugin:v1.11.0],SizeBytes:18159085,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:017727efcfeb7d053af68e51436ce8e65edbc6ca573720afb4f79c8594036955 registry.k8s.io/coredns/coredns:v1.10.0],SizeBytes:13766234,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:8502345,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:933940f13b3ea0abc62e656c1aa5c5b47c04b15d71250413a6b821bd0c58b94e registry.k8s.io/sig-storage/livenessprobe:v2.7.0],SizeBytes:8223819,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:7685305,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:5c99cf6a02adda929b10321dbf4ecfa00d87be9ba4fb456006237d530ab4baa1 registry.k8s.io/e2e-test-images/nginx:1.14-4],SizeBytes:6860089,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:2e0f836850e09b8b7cc937681d6194537a09fbd5f6b9e08f4d646a85128e8937 registry.k8s.io/e2e-test-images/busybox:1.29-4],SizeBytes:772933,},ContainerImage{Names:[registry.k8s.io/pause@sha256:7031c1b283388d2c2e09b57badb803c05ebed362dc88d84b480cc47f72a21097 registry.k8s.io/pause:3.9],SizeBytes:268051,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:253553,},},VolumesInUse:[kubernetes.io/csi/ebs.csi.aws.com^vol-0e020446b1cd6abaf],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-0e020446b1cd6abaf,DevicePath:,},},Config:nil,},} Mar 5 16:49:00.967: INFO: Logging kubelet events for node i-056ad61525499081e Mar 5 16:49:01.082: INFO: ... skipping 4 lines ... Mar 5 16:49:01.317: INFO: sample-webhook-deployment-7497495989-s9585 started at 2023-03-05 16:46:38 +0000 UTC (0+1 container statuses recorded) Mar 5 16:49:01.317: INFO: Container sample-webhook ready: true, restart count 0 Mar 5 16:49:01.317: INFO: ebs-csi-node-n98cq started at 2023-03-05 16:25:49 +0000 UTC (0+3 container statuses recorded) Mar 5 16:49:01.317: INFO: Container ebs-plugin ready: true, restart count 0 Mar 5 16:49:01.317: INFO: Container liveness-probe ready: true, restart count 0 Mar 5 16:49:01.317: INFO: Container node-driver-registrar ready: true, restart count 0 Mar 5 16:49:01.317: INFO: fail-once-local-kdbgv started at 2023-03-05 16:46:33 +0000 UTC (0+1 container statuses recorded) Mar 5 16:49:01.317: INFO: Container c ready: false, restart count 1 Mar 5 16:49:01.317: INFO: cleanup20-16a6f148-bb85-4b73-99f5-2adb1cfdeea0-xsqgf started at 2023-03-05 16:40:29 +0000 UTC (0+1 container statuses recorded) Mar 5 16:49:01.317: INFO: Container cleanup20-16a6f148-bb85-4b73-99f5-2adb1cfdeea0 ready: true, restart count 0 Mar 5 16:49:01.317: INFO: cleanup20-16a6f148-bb85-4b73-99f5-2adb1cfdeea0-kgbnw started at 2023-03-05 16:40:29 +0000 UTC (0+1 container statuses recorded) Mar 5 16:49:01.317: INFO: Container cleanup20-16a6f148-bb85-4b73-99f5-2adb1cfdeea0 ready: true, restart count 0 Mar 5 16:49:01.317: INFO: cleanup20-16a6f148-bb85-4b73-99f5-2adb1cfdeea0-42w6f started at 2023-03-05 16:40:29 +0000 UTC (0+1 container statuses recorded) ... skipping 3 lines ... Mar 5 16:49:01.317: INFO: cleanup20-16a6f148-bb85-4b73-99f5-2adb1cfdeea0-9qlxz started at 2023-03-05 16:48:01 +0000 UTC (0+1 container statuses recorded) Mar 5 16:49:01.317: INFO: Container cleanup20-16a6f148-bb85-4b73-99f5-2adb1cfdeea0 ready: true, restart count 0 Mar 5 16:49:01.317: INFO: cleanup20-16a6f148-bb85-4b73-99f5-2adb1cfdeea0-7qxdd started at 2023-03-05 16:48:03 +0000 UTC (0+1 container statuses recorded) Mar 5 16:49:01.317: INFO: Container cleanup20-16a6f148-bb85-4b73-99f5-2adb1cfdeea0 ready: true, restart count 0 Mar 5 16:49:01.317: INFO: coredns-autoscaler-8495f5575-plvdw started at 2023-03-05 16:25:57 +0000 UTC (0+1 container statuses recorded) Mar 5 16:49:01.317: INFO: Container autoscaler ready: true, restart count 0 Mar 5 16:49:01.317: INFO: fail-once-local-69mfr started at 2023-03-05 16:46:28 +0000 UTC (0+1 container statuses recorded) Mar 5 16:49:01.317: INFO: Container c ready: false, restart count 1 Mar 5 16:49:01.317: INFO: cleanup20-16a6f148-bb85-4b73-99f5-2adb1cfdeea0-wmjdk started at 2023-03-05 16:40:29 +0000 UTC (0+1 container statuses recorded) Mar 5 16:49:01.317: INFO: Container cleanup20-16a6f148-bb85-4b73-99f5-2adb1cfdeea0 ready: true, restart count 0 Mar 5 16:49:01.317: INFO: cleanup20-16a6f148-bb85-4b73-99f5-2adb1cfdeea0-x6s5k started at 2023-03-05 16:40:29 +0000 UTC (0+1 container statuses recorded) Mar 5 16:49:01.317: INFO: Container cleanup20-16a6f148-bb85-4b73-99f5-2adb1cfdeea0 ready: true, restart count 0 Mar 5 16:49:01.318: INFO: ebs-csi-controller-79785c665b-7dmsf started at 2023-03-05 16:25:57 +0000 UTC (0+5 container statuses recorded) ... skipping 13 lines ... Mar 5 16:49:01.318: INFO: startup-0f5d0041-93ec-49cb-aeda-dd383b1eb79a started at 2023-03-05 16:46:46 +0000 UTC (0+1 container statuses recorded) Mar 5 16:49:01.318: INFO: Container busybox ready: true, restart count 0 Mar 5 16:49:01.318: INFO: cleanup20-16a6f148-bb85-4b73-99f5-2adb1cfdeea0-j6cks started at 2023-03-05 16:48:01 +0000 UTC (0+1 container statuses recorded) Mar 5 16:49:01.318: INFO: Container cleanup20-16a6f148-bb85-4b73-99f5-2adb1cfdeea0 ready: true, restart count 0 Mar 5 16:49:01.318: INFO: nodeport-test-88ggz started at 2023-03-05 16:40:39 +0000 UTC (0+1 container statuses recorded) Mar 5 16:49:01.318: INFO: Container nodeport-test ready: true, restart count 0 Mar 5 16:49:01.318: INFO: fail-once-local-2mt4w started at 2023-03-05 16:46:28 +0000 UTC (0+1 container statuses recorded) Mar 5 16:49:01.318: INFO: Container c ready: false, restart count 1 Mar 5 16:49:01.318: INFO: cleanup20-16a6f148-bb85-4b73-99f5-2adb1cfdeea0-khffj started at 2023-03-05 16:40:29 +0000 UTC (0+1 container statuses recorded) Mar 5 16:49:01.318: INFO: Container cleanup20-16a6f148-bb85-4b73-99f5-2adb1cfdeea0 ready: true, restart count 0 Mar 5 16:49:01.318: INFO: cleanup20-16a6f148-bb85-4b73-99f5-2adb1cfdeea0-xkdzr started at 2023-03-05 16:40:29 +0000 UTC (0+1 container statuses recorded) Mar 5 16:49:01.318: INFO: Container cleanup20-16a6f148-bb85-4b73-99f5-2adb1cfdeea0 ready: true, restart count 0 Mar 5 16:49:01.318: INFO: cleanup20-16a6f148-bb85-4b73-99f5-2adb1cfdeea0-gnfq6 started at 2023-03-05 16:48:01 +0000 UTC (0+1 container statuses recorded) ... skipping 11 lines ... Mar 5 16:49:01.318: INFO: cleanup20-16a6f148-bb85-4b73-99f5-2adb1cfdeea0-sqrjl started at 2023-03-05 16:40:29 +0000 UTC (0+1 container statuses recorded) Mar 5 16:49:01.318: INFO: Container cleanup20-16a6f148-bb85-4b73-99f5-2adb1cfdeea0 ready: true, restart count 0 Mar 5 16:49:01.318: INFO: cleanup20-16a6f148-bb85-4b73-99f5-2adb1cfdeea0-m6dgl started at 2023-03-05 16:48:01 +0000 UTC (0+1 container statuses recorded) Mar 5 16:49:01.318: INFO: Container cleanup20-16a6f148-bb85-4b73-99f5-2adb1cfdeea0 ready: true, restart count 0 Mar 5 16:49:01.318: INFO: cleanup20-16a6f148-bb85-4b73-99f5-2adb1cfdeea0-cqffq started at 2023-03-05 16:48:02 +0000 UTC (0+1 container statuses recorded) Mar 5 16:49:01.318: INFO: Container cleanup20-16a6f148-bb85-4b73-99f5-2adb1cfdeea0 ready: true, restart count 0 Mar 5 16:49:01.318: INFO: fail-once-local-gj678 started at 2023-03-05 16:46:33 +0000 UTC (0+1 container statuses recorded) Mar 5 16:49:01.318: INFO: Container c ready: false, restart count 1 Mar 5 16:49:01.788: INFO: Latency metrics for node i-056ad61525499081e Mar 5 16:49:01.788: INFO: Logging node info for node i-06df568f6e5c59fff Mar 5 16:49:01.901: INFO: Node Info: &Node{ObjectMeta:{i-06df568f6e5c59fff cb63c681-2e18-474a-b6f3-221325ac0d52 18301 0 2023-03-05 16:24:16 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:arm64 beta.kubernetes.io/instance-type:t4g.small beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:eu-central-1 failure-domain.beta.kubernetes.io/zone:eu-central-1a karpenter.k8s.aws/instance-category:t karpenter.k8s.aws/instance-cpu:2 karpenter.k8s.aws/instance-family:t4g karpenter.k8s.aws/instance-generation:4 karpenter.k8s.aws/instance-hypervisor:nitro karpenter.k8s.aws/instance-memory:2048 karpenter.k8s.aws/instance-pods:11 karpenter.k8s.aws/instance-size:small karpenter.sh/capacity-type:spot karpenter.sh/initialized:true karpenter.sh/provisioner-name:nodes kubernetes.io/arch:arm64 kubernetes.io/hostname:i-06df568f6e5c59fff kubernetes.io/os:linux node-role.kubernetes.io/node: node-role.kubernetes.io/spot-worker:true node.kubernetes.io/instance-type:t4g.small topology.ebs.csi.aws.com/zone:eu-central-1a topology.hostpath.csi/node:i-06df568f6e5c59fff topology.kubernetes.io/region:eu-central-1 topology.kubernetes.io/zone:eu-central-1a] map[csi.volume.kubernetes.io/nodeid:{"csi-hostpath-ephemeral-2147":"i-06df568f6e5c59fff","csi-hostpath-ephemeral-5408":"i-06df568f6e5c59fff","csi-hostpath-ephemeral-6540":"i-06df568f6e5c59fff","csi-mock-csi-mock-volumes-fsgroup-policy-1058":"i-06df568f6e5c59fff","csi-mock-csi-mock-volumes-workload-2219":"i-06df568f6e5c59fff","ebs.csi.aws.com":"i-06df568f6e5c59fff"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [{karpenter.sh/v1alpha5 Provisioner nodes ac5d7599-b2a2-4eae-8da4-1f4c0a54ec82 <nil> 0xc004fe4fa0}] [karpenter.sh/termination] [{aws-cloud-controller-manager Update v1 2023-03-05 16:24:16 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{}}}} } {kops-controller Update v1 2023-03-05 16:24:16 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{},"f:node-role.kubernetes.io/spot-worker":{}}}} } {karpenter Update v1 2023-03-05 16:25:57 +0000 UTC FieldsV1 {"f:metadata":{"f:finalizers":{".":{},"v:\"karpenter.sh/termination\"":{}},"f:labels":{".":{},"f:karpenter.k8s.aws/instance-category":{},"f:karpenter.k8s.aws/instance-cpu":{},"f:karpenter.k8s.aws/instance-family":{},"f:karpenter.k8s.aws/instance-generation":{},"f:karpenter.k8s.aws/instance-hypervisor":{},"f:karpenter.k8s.aws/instance-memory":{},"f:karpenter.k8s.aws/instance-pods":{},"f:karpenter.k8s.aws/instance-size":{},"f:karpenter.sh/capacity-type":{},"f:karpenter.sh/initialized":{},"f:karpenter.sh/provisioner-name":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}},"f:ownerReferences":{".":{},"k:{\"uid\":\"ac5d7599-b2a2-4eae-8da4-1f4c0a54ec82\"}":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-03-05 16:26:48 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{".":{},"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"InternalIP\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kube-controller-manager Update v1 2023-03-05 16:44:29 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{".":{},"k:{\"type\":\"DiskPressure\"}":{".":{},"f:type":{}},"k:{\"type\":\"MemoryPressure\"}":{".":{},"f:type":{}},"k:{\"type\":\"PIDPressure\"}":{".":{},"f:type":{}},"k:{\"type\":\"Ready\"}":{".":{},"f:type":{}}},"f:volumesAttached":{}}} status} {kube-controller-manager Update v1 2023-03-05 16:46:37 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:node.alpha.kubernetes.io/ttl":{}},"f:labels":{"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.2.0/24\"":{}},"f:taints":{}}} } {kubelet Update v1 2023-03-05 16:48:40 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{"f:kubernetes.io/hostname":{},"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{".":{},"f:cpu":{},"f:ephemeral-storage":{},"f:hugepages-1Gi":{},"f:hugepages-2Mi":{},"f:hugepages-32Mi":{},"f:hugepages-64Ki":{},"f:memory":{},"f:pods":{}},"f:capacity":{".":{},"f:cpu":{},"f:ephemeral-storage":{},"f:hugepages-1Gi":{},"f:hugepages-2Mi":{},"f:hugepages-32Mi":{},"f:hugepages-64Ki":{},"f:memory":{},"f:pods":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:daemonEndpoints":{"f:kubeletEndpoint":{"f:Port":{}}},"f:images":{},"f:nodeInfo":{"f:architecture":{},"f:bootID":{},"f:containerRuntimeVersion":{},"f:kernelVersion":{},"f:kubeProxyVersion":{},"f:kubeletVersion":{},"f:machineID":{},"f:operatingSystem":{},"f:osImage":{},"f:systemUUID":{}},"f:volumesInUse":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.2.0/24,DoNotUseExternalID:,ProviderID:aws:///eu-central-1a/i-06df568f6e5c59fff,Unschedulable:false,Taints:[]Taint{Taint{Key:node.kubernetes.io/not-ready,Value:,Effect:NoSchedule,TimeAdded:2023-03-05 16:46:33 +0000 UTC,},Taint{Key:node.kubernetes.io/not-ready,Value:,Effect:NoExecute,TimeAdded:2023-03-05 16:46:37 +0000 UTC,},},ConfigSource:nil,PodCIDRs:[100.96.2.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49756430336 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-32Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-64Ki: {{0 0} {<nil>} 0 DecimalSI},memory: {{1941540864 0} {<nil>} 1896036Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44780787229 0} {<nil>} 44780787229 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-32Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-64Ki: {{0 0} {<nil>} 0 DecimalSI},memory: {{1836683264 0} {<nil>} 1793636Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:Ready,Status:False,LastHeartbeatTime:2023-03-05 16:47:34 +0000 UTC,LastTransitionTime:2023-03-05 16:46:08 +0000 UTC,Reason:KubeletNotReady,Message:PLEG is not healthy: pleg was last seen active 8m29.527212456s ago; threshold is 3m0s,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-03-05 16:47:34 +0000 UTC,LastTransitionTime:2023-03-05 16:47:12 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-03-05 16:47:34 +0000 UTC,LastTransitionTime:2023-03-05 16:47:15 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-03-05 16:47:34 +0000 UTC,LastTransitionTime:2023-03-05 16:47:15 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.43.148,},NodeAddress{Type:ExternalIP,Address:18.192.211.222,},NodeAddress{Type:InternalDNS,Address:i-06df568f6e5c59fff.eu-central-1.compute.internal,},NodeAddress{Type:Hostname,Address:i-06df568f6e5c59fff.eu-central-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-18-192-211-222.eu-central-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec221221864d9409a8ec5f6614a9b94b,SystemUUID:ec221221-864d-9409-a8ec-5f6614a9b94b,BootID:553d31bf-faca-417e-a3db-325839e1b8e4,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 22.04.2 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.27.0-alpha.3.52+d48b8167f7094b,KubeProxyVersion:v1.27.0-alpha.3.52+d48b8167f7094b,OperatingSystem:linux,Architecture:arm64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:157636062,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/jessie-dnsutils@sha256:24aaf2626d6b27864c29de2097e8bbb840b3a414271bf7c8995e431e47d8408e registry.k8s.io/e2e-test-images/jessie-dnsutils:1.7],SizeBytes:116764683,},ContainerImage{Names:[registry.k8s.io/kube-proxy-arm64:v1.27.0-alpha.3.52_d48b8167f7094b],SizeBytes:60554082,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:16bbf38c463a4223d8cfe4da12bc61010b082a79b4bb003e2d3ba3ece5dd5f9e registry.k8s.io/e2e-test-images/agnhost:2.43],SizeBytes:49808364,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:3fe7acf013d1264ffded116b80a73dc129a449b0fccdb8d21af8279f2233f36e registry.k8s.io/e2e-test-images/httpd:2.4.39-4],SizeBytes:41528818,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:148b022f5c5da426fc2f3c14b5c0867e58ef05961510c84749ac1fddcb0fef22 registry.k8s.io/e2e-test-images/httpd:2.4.38-4],SizeBytes:40343601,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:28932904,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:ee3b525d5b89db99da3b8eb521d9cd90cb6e9ef0fbb651e98bb37be78d36b5b8 registry.k8s.io/sig-storage/csi-provisioner:v3.3.0],SizeBytes:23945474,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:425d8f1b769398127767b06ed97ce62578a3179bcb99809ce93a1649e025ffe7 registry.k8s.io/sig-storage/csi-resizer:v1.6.0],SizeBytes:22678632,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:291334908ddf71a4661fd7f6d9d97274de8a5378a2b6fdfeb2ce73414a34f82f registry.k8s.io/sig-storage/csi-snapshotter:v6.1.0],SizeBytes:22434180,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:9a685020911e2725ad019dbce6e4a5ab93d51e3d4557f115e64343345e05781b registry.k8s.io/sig-storage/csi-attacher:v4.0.0],SizeBytes:22394501,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:122bfb8c1edabb3c0edd63f06523e6940d958d19b3957dc7b1d6f81e9f1f6119 registry.k8s.io/sig-storage/csi-provisioner:v3.1.0],SizeBytes:21735644,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:9ebbf9f023e7b41ccee3d52afe39a89e3ddacdbb69269d583abfc25847cfd9e4 registry.k8s.io/sig-storage/csi-resizer:v1.4.0],SizeBytes:20848413,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:8b9c313c05f54fb04f8d430896f5f5904b6cb157df261501b29adc04d2b2dc7b registry.k8s.io/sig-storage/csi-attacher:v3.4.0],SizeBytes:20565480,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nonroot@sha256:ee9f50b3c64b174d296d91ca9f69a914ac30e59095dfb462b2b518ad28a63655 registry.k8s.io/e2e-test-images/nonroot:1.4],SizeBytes:18561972,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:92257881c1d6493cf18299a24af42330f891166560047902b8d431fb66b01af5 registry.k8s.io/sig-storage/hostpathplugin:v1.9.0],SizeBytes:18267611,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:9b2d9abda017c28b12f84a344f57cd73fbdb6c2bd7dd5bdd5018246ad1093ba6 registry.k8s.io/sig-storage/hostpathplugin:v1.11.0],SizeBytes:18159085,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:017727efcfeb7d053af68e51436ce8e65edbc6ca573720afb4f79c8594036955 registry.k8s.io/coredns/coredns:v1.10.0],SizeBytes:13766234,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:8502345,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:933940f13b3ea0abc62e656c1aa5c5b47c04b15d71250413a6b821bd0c58b94e registry.k8s.io/sig-storage/livenessprobe:v2.7.0],SizeBytes:8223819,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:7685305,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:5c99cf6a02adda929b10321dbf4ecfa00d87be9ba4fb456006237d530ab4baa1 registry.k8s.io/e2e-test-images/nginx:1.14-4],SizeBytes:6860089,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nonewprivs@sha256:8ac1264691820febacf3aea5d152cbde6d10685731ec14966a9401c6f47a68ac registry.k8s.io/e2e-test-images/nonewprivs:1.3],SizeBytes:5025027,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:2e0f836850e09b8b7cc937681d6194537a09fbd5f6b9e08f4d646a85128e8937 registry.k8s.io/e2e-test-images/busybox:1.29-4],SizeBytes:772933,},ContainerImage{Names:[registry.k8s.io/pause@sha256:7031c1b283388d2c2e09b57badb803c05ebed362dc88d84b480cc47f72a21097 registry.k8s.io/pause:3.9],SizeBytes:268051,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:253553,},},VolumesInUse:[kubernetes.io/csi/csi-hostpath-ephemeral-2147^52df5953-bb73-11ed-8aa7-be49170871b4 kubernetes.io/csi/csi-hostpath-ephemeral-5408^443d1daf-bb73-11ed-95c4-5a95d17cbc27 kubernetes.io/csi/csi-hostpath-ephemeral-5408^44411f51-bb73-11ed-95c4-5a95d17cbc27 kubernetes.io/csi/csi-hostpath-ephemeral-6540^4426a07f-bb73-11ed-a604-92687f2c50f8 kubernetes.io/csi/ebs.csi.aws.com^vol-05b67fab5beb6394b],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-05b67fab5beb6394b,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/csi-hostpath-ephemeral-6540^4426a07f-bb73-11ed-a604-92687f2c50f8,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/csi-hostpath-ephemeral-5408^44411f51-bb73-11ed-95c4-5a95d17cbc27,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/csi-hostpath-ephemeral-5408^443d1daf-bb73-11ed-95c4-5a95d17cbc27,DevicePath:,},},Config:nil,},} Mar 5 16:49:01.901: INFO: Logging kubelet events for node i-06df568f6e5c59fff Mar 5 16:49:02.017: INFO: Logging pods the kubelet thinks is on node i-06df568f6e5c59fff Mar 5 16:49:12.130: INFO: Unable to retrieve kubelet pods for node i-06df568f6e5c59fff: error trying to reach service: net/http: TLS handshake timeout Mar 5 16:49:12.130: INFO: Logging node info for node i-0d6a32f749839192f Mar 5 16:49:12.243: INFO: Node Info: &Node{ObjectMeta:{i-0d6a32f749839192f 4155fefb-4d3a-4b04-ba53-6ac9e895ff92 16414 0 2023-03-05 16:20:46 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:arm64 beta.kubernetes.io/instance-type:c6g.xlarge beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:eu-central-1 failure-domain.beta.kubernetes.io/zone:eu-central-1a kops.k8s.io/kops-controller-pki: kubernetes.io/arch:arm64 kubernetes.io/hostname:i-0d6a32f749839192f kubernetes.io/os:linux node-role.kubernetes.io/control-plane: node.kubernetes.io/exclude-from-external-load-balancers: node.kubernetes.io/instance-type:c6g.xlarge topology.ebs.csi.aws.com/zone:eu-central-1a topology.kubernetes.io/region:eu-central-1 topology.kubernetes.io/zone:eu-central-1a] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-0d6a32f749839192f"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kubelet Update v1 2023-03-05 16:20:46 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {protokube Update v1 2023-03-05 16:21:25 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kops.k8s.io/kops-controller-pki":{},"f:node-role.kubernetes.io/control-plane":{},"f:node.kubernetes.io/exclude-from-external-load-balancers":{}}}} } {kube-controller-manager Update v1 2023-03-05 16:21:47 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.0.0/24\"":{}}}} } {aws-cloud-controller-manager Update v1 2023-03-05 16:21:48 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{},"f:taints":{}}} } {aws-cloud-controller-manager Update v1 2023-03-05 16:21:48 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kubelet Update v1 2023-03-05 16:44:12 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.0.0/24,DoNotUseExternalID:,ProviderID:aws:///eu-central-1a/i-0d6a32f749839192f,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/control-plane,Value:,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[100.96.0.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{4 0} {<nil>} 4 DecimalSI},ephemeral-storage: {{49756430336 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-32Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-64Ki: {{0 0} {<nil>} 0 DecimalSI},memory: {{8150118400 0} {<nil>} 7959100Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{4 0} {<nil>} 4 DecimalSI},ephemeral-storage: {{44780787229 0} {<nil>} 44780787229 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-32Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-64Ki: {{0 0} {<nil>} 0 DecimalSI},memory: {{8045260800 0} {<nil>} 7856700Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-03-05 16:44:12 +0000 UTC,LastTransitionTime:2023-03-05 16:20:46 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-03-05 16:44:12 +0000 UTC,LastTransitionTime:2023-03-05 16:20:46 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-03-05 16:44:12 +0000 UTC,LastTransitionTime:2023-03-05 16:20:46 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-03-05 16:44:12 +0000 UTC,LastTransitionTime:2023-03-05 16:21:47 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.49.247,},NodeAddress{Type:ExternalIP,Address:18.197.20.82,},NodeAddress{Type:InternalDNS,Address:i-0d6a32f749839192f.eu-central-1.compute.internal,},NodeAddress{Type:Hostname,Address:i-0d6a32f749839192f.eu-central-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-18-197-20-82.eu-central-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2bed249e8ac7b517cb08ccd42b83e5,SystemUUID:ec2bed24-9e8a-c7b5-17cb-08ccd42b83e5,BootID:6f9215c9-dfe8-4b13-84f4-9b2084456198,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 22.04.2 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.27.0-alpha.3.52+d48b8167f7094b,KubeProxyVersion:v1.27.0-alpha.3.52+d48b8167f7094b,OperatingSystem:linux,Architecture:arm64,},Images:[]ContainerImage{ContainerImage{Names:[registry.k8s.io/etcd@sha256:9ce33ba33d8e738a5b85ed50b5080ac746deceed4a7496c550927a7a19ca3b6d registry.k8s.io/etcd:3.5.0-0],SizeBytes:157797353,},ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:157636062,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:4afb99b4690b418ffc2ceb67e1a17376457e441c1f09ab55447f0aaf992fa646 registry.k8s.io/etcd:3.4.3-0],SizeBytes:151677080,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:4ad90a11b55313b182afc186b9876c8e891531b8db4c9bf1541953021618d0e2 registry.k8s.io/etcd:3.4.13-0],SizeBytes:134531559,},ContainerImage{Names:[registry.k8s.io/kube-apiserver-arm64:v1.27.0-alpha.3.52_d48b8167f7094b],SizeBytes:131087707,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:0d2ab6b15b4fe0b903b9e81192d942fb39c29be61fe341cd16db5d0165b0435c registry.k8s.io/etcd:3.3.17-0],SizeBytes:126979107,},ContainerImage{Names:[registry.k8s.io/kube-controller-manager-arm64:v1.27.0-alpha.3.52_d48b8167f7094b],SizeBytes:120667128,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:17da501f5d2a675be46040422a27b7cc21b8a43895ac998b171db1c346f361f7 registry.k8s.io/etcd:3.3.10-0],SizeBytes:81436097,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:6f72b851544986cb0921b53ea655ec04c36131248f16d4ad110cb3ca0c369dc1 registry.k8s.io/etcd:3.5.4-0],SizeBytes:81120118,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:13f53ed1d91e2e11aac476ee9a0269fdda6cc4874eba903efd40daf50c55eee5 registry.k8s.io/etcd:3.5.3-0],SizeBytes:81114891,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:51eae8381dcb1078289fa7b4f3df2630cdc18d09fb56f8e56b41c40e191d6c83 registry.k8s.io/etcd:3.5.7-0],SizeBytes:80665728,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:dd75ec974b0a2a6f6bb47001ba09207976e625db898d1b16735528c009cb171c registry.k8s.io/etcd:3.5.6-0],SizeBytes:80539316,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:905d7ca17fd02bc24c0eba9a062753aba15db3e31422390bc3238eb762339b20 registry.k8s.io/etcd:3.2.24-1],SizeBytes:61500433,},ContainerImage{Names:[registry.k8s.io/etcdadm/etcd-manager-slim@sha256:8727adc059334e0e3329dbccce116a734da4a090da5e86c8cddccaf333caa666 registry.k8s.io/etcdadm/etcd-manager-slim:v3.0.20230201],SizeBytes:60826655,},ContainerImage{Names:[registry.k8s.io/kube-proxy-arm64:v1.27.0-alpha.3.52_d48b8167f7094b],SizeBytes:60554082,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:64b9ea357325d5db9f8a723dcf503b5a449177b17ac87d69481e126bb724c263 registry.k8s.io/etcd:3.5.1-0],SizeBytes:59553904,},ContainerImage{Names:[registry.k8s.io/kube-scheduler-arm64:v1.27.0-alpha.3.52_d48b8167f7094b],SizeBytes:56310763,},ContainerImage{Names:[registry.k8s.io/kops/kops-controller:1.27.0-alpha.1],SizeBytes:41797411,},ContainerImage{Names:[registry.k8s.io/kops/dns-controller:1.27.0-alpha.1],SizeBytes:40267473,},ContainerImage{Names:[public.ecr.aws/karpenter/controller@sha256:68db4f092cf9cc83f5ef9e2fbc5407c2cb682e81f64dfaa700a7602ede38b1cf public.ecr.aws/karpenter/controller:v0.16.3],SizeBytes:33707751,},ContainerImage{Names:[public.ecr.aws/karpenter/webhook@sha256:96a2d9b06d6bc5127801f358f74b1cf2d289b423a2e9ba40c573c0b14b17dafa public.ecr.aws/karpenter/webhook:v0.16.3],SizeBytes:33307187,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:28932904,},ContainerImage{Names:[quay.io/cilium/operator@sha256:a6d24a006a6b92967ac90786b49bc1ac26e5477cf028cd1186efcfc2466484db quay.io/cilium/operator:v1.12.5],SizeBytes:24567287,},ContainerImage{Names:[registry.k8s.io/provider-aws/cloud-controller-manager@sha256:fdeb61e3e42ecd9cca868d550ebdb88dd6341d9e91fcfa9a37e227dab2ad22cb registry.k8s.io/provider-aws/cloud-controller-manager:v1.26.0],SizeBytes:18345226,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:8502345,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:7685305,},ContainerImage{Names:[registry.k8s.io/kops/kube-apiserver-healthcheck:1.27.0-alpha.1],SizeBytes:4713612,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:253553,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Mar 5 16:49:12.243: INFO: Logging kubelet events for node i-0d6a32f749839192f Mar 5 16:49:12.359: INFO: ... skipping 52 lines ... Mar 5 16:49:12.479: INFO: Container controller ready: true, restart count 0 Mar 5 16:49:12.972: INFO: Latency metrics for node i-0d6a32f749839192f [1mSTEP:[0m Destroying namespace "services-2335" for this suite. [38;5;243m@ 03/05/23 16:49:12.972[0m [38;5;243m<< Timeline[0m [38;5;9m[FAILED] only 1 pods started out of 2[0m [38;5;9mIn [1m[It][0m[38;5;9m at: [1mtest/e2e/network/service.go:1295[0m [38;5;243m@ 03/05/23 16:45:43.032[0m [38;5;243m------------------------------[0m [38;5;14mS[0m[38;5;14mS[0m[38;5;10m•[0m[38;5;14mS[0m[38;5;14mS[0m [38;5;243m------------------------------[0m [38;5;9m• [FAILED] [533.205 seconds][0m [0m[sig-storage] Projected configMap [38;5;9m[1m[It] should be consumable from pods in volume [NodeConformance] [Conformance][0m [38;5;243mtest/e2e/common/storage/projected_configmap.go:47[0m [38;5;243mTimeline >>[0m [1mSTEP:[0m Creating a kubernetes client [38;5;243m@ 03/05/23 16:40:28.264[0m Mar 5 16:40:28.264: INFO: >>> kubeConfig: /root/.kube/config [1mSTEP:[0m Building a namespace api object, basename projected [38;5;243m@ 03/05/23 16:40:28.265[0m [1mSTEP:[0m Waiting for a default service account to be provisioned in namespace [38;5;243m@ 03/05/23 16:40:28.6[0m [1mSTEP:[0m Waiting for kube-root-ca.crt to be provisioned in namespace [38;5;243m@ 03/05/23 16:40:28.82[0m [1mSTEP:[0m Creating configMap with name projected-configmap-test-volume-8747160a-ab8b-4abd-ae09-bd6f03d6e62e [38;5;243m@ 03/05/23 16:40:29.041[0m [1mSTEP:[0m Creating a pod to test consume configMaps [38;5;243m@ 03/05/23 16:40:29.174[0m Mar 5 16:46:08.335: INFO: Failed to get logs from node "i-06df568f6e5c59fff" pod "pod-projected-configmaps-404de0a7-eaf4-4291-8862-6e481dc16211" container "agnhost-container": the server could not find the requested resource (get pods pod-projected-configmaps-404de0a7-eaf4-4291-8862-6e481dc16211) [1mSTEP:[0m delete the pod [38;5;243m@ 03/05/23 16:46:08.335[0m Mar 5 16:46:08.563: INFO: Unexpected error: <*errors.errorString | 0xc000e8eb10>: { s: "expected pod \"pod-projected-configmaps-404de0a7-eaf4-4291-8862-6e481dc16211\" success: Timed out after 300.014s.\nexpected pod to be Succeeded or Failed, got instead:\n <*v1.Pod | 0xc0030d2480>: \n metadata:\n creationTimestamp: \"2023-03-05T16:40:29Z\"\n managedFields:\n - apiVersion: v1\n fieldsType: FieldsV1\n fieldsV1:\n f:spec:\n f:containers:\n k:{\"name\":\"agnhost-container\"}:\n .: {}\n f:args: {}\n f:image: {}\n f:imagePullPolicy: {}\n f:name: {}\n f:resources: {}\n f:securityContext: {}\n f:terminationMessagePath: {}\n f:terminationMessagePolicy: {}\n f:volumeMounts:\n .: {}\n k:{\"mountPath\":\"/etc/projected-configmap-volume\"}:\n .: {}\n f:mountPath: {}\n f:name: {}\n f:readOnly: {}\n f:dnsPolicy: {}\n f:enableServiceLinks: {}\n f:restartPolicy: {}\n f:schedulerName: {}\n f:securityContext: {}\n f:terminationGracePeriodSeconds: {}\n f:volumes:\n .: {}\n k:{\"name\":\"projected-configmap-volume\"}:\n .: {}\n f:name: {}\n f:projected:\n .: {}\n f:defaultMode: {}\n f:sources: {}\n manager: e2e.test\n operation: Update\n time: \"2023-03-05T16:40:29Z\"\n name: pod-projected-configmaps-404de0a7-eaf4-4291-8862-6e481dc16211\n namespace: projected-8785\n resourceVersion: \"13986\"\n uid: d162c570-08ea-4788-8a1d-a98fb0665b13\n spec:\n containers:\n - args:\n - mounttest\n - --file_content=/etc/projected-configmap-volume/data-1\n - --file_mode=/etc/projected-configmap-volume/data-1\n image: registry.k8s.io/e2e-test-images/agnhost:2.43\n imagePullPolicy: IfNotPresent\n name: agnhost-container\n resources: {}\n securityContext: {}\n terminationMessagePath: /dev/termination-log\n terminationMessagePolicy: File\n volumeMounts:\n - mountPath: /etc/projected-configmap-volume\n name: projected-configmap-volume\n readOnly: true\n - mountPath: /var/run/secrets/kubernetes.io/serviceaccount\n name: kube-api-access-48lr2\n readOnly: true\n dnsPolicy: ClusterFirst\n enableServiceLinks: true\n nodeName: i-06df568f6e5c59fff\n preemptionPolicy: PreemptLowerPriority\n priority: 0\n restartPolicy: Never\n schedulerName: default-scheduler\n securityContext: {}\n serviceAccount: default\n serviceAccountName: default\n terminationGracePeriodSeconds: 0\n tolerations:\n - effect: NoExecute\n key: node.kubernetes.io/not-ready\n operator: Exists\n tolerationSeconds: 300\n - effect: NoExecute\n key: node.kubernetes.io/unreachable\n operator: Exists\n tolerationSeconds: 300\n volumes:\n - name: projected-configmap-volume\n projected:\n defaultMode: 420\n sources:\n - configMap:\n name: projected-configmap-test-volume-8747160a-ab8b-4abd-ae09-bd6f03d6e62e\n - name: kube-api-access-48lr2\n projected:\n defaultMode: 420\n sources:\n - serviceAccountToken:\n expirationSeconds: 3607\n path: token\n - configMap:\n items:\n - key: ca.crt\n path: ca.crt\n name: kube-root-ca.crt\n - downwardAPI:\n items:\n - fieldRef:\n apiVersion: v1\n fieldPath: metadata.namespace\n path: namespace\n status:\n conditions:\n - lastProbeTime: null\n lastTransitionTime: \"2023-03-05T16:40:29Z\"\n status: \"True\"\n type: PodScheduled\n phase: Pending\n qosClass: BestEffort", } [38;5;9m[FAILED][0m in [It] - test/e2e/framework/pod/output/output.go:238 [38;5;243m@ 03/05/23 16:46:08.563[0m Mar 5 16:46:08.563: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready Mar 5 16:46:08.675: INFO: Condition Ready of node i-06df568f6e5c59fff is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-05 16:42:51 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-05 16:42:57 +0000 UTC}]. Failure Mar 5 16:46:10.789: INFO: Condition Ready of node i-06df568f6e5c59fff is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-05 16:42:51 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-05 16:42:57 +0000 UTC}]. Failure Mar 5 16:46:12.788: INFO: Condition Ready of node i-06df568f6e5c59fff is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-05 16:42:51 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-05 16:42:57 +0000 UTC}]. Failure Mar 5 16:46:14.788: INFO: Condition Ready of node i-06df568f6e5c59fff is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-05 16:42:51 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-05 16:42:57 +0000 UTC}]. Failure Mar 5 16:46:16.788: INFO: Condition Ready of node i-06df568f6e5c59fff is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-05 16:42:51 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-05 16:42:57 +0000 UTC}]. Failure ... skipping 103 lines ... Mar 5 16:49:09.696: INFO: startup-0f5d0041-93ec-49cb-aeda-dd383b1eb79a started at 2023-03-05 16:46:46 +0000 UTC (0+1 container statuses recorded) Mar 5 16:49:09.696: INFO: Container busybox ready: true, restart count 0 Mar 5 16:49:09.696: INFO: cleanup20-16a6f148-bb85-4b73-99f5-2adb1cfdeea0-w749b started at 2023-03-05 16:48:01 +0000 UTC (0+1 container statuses recorded) Mar 5 16:49:09.696: INFO: Container cleanup20-16a6f148-bb85-4b73-99f5-2adb1cfdeea0 ready: true, restart count 0 Mar 5 16:49:09.696: INFO: cleanup20-16a6f148-bb85-4b73-99f5-2adb1cfdeea0-j6cks started at 2023-03-05 16:48:01 +0000 UTC (0+1 container statuses recorded) Mar 5 16:49:09.696: INFO: Container cleanup20-16a6f148-bb85-4b73-99f5-2adb1cfdeea0 ready: true, restart count 0 Mar 5 16:49:09.696: INFO: fail-once-local-2mt4w started at 2023-03-05 16:46:28 +0000 UTC (0+1 container statuses recorded) Mar 5 16:49:09.696: INFO: Container c ready: false, restart count 1 Mar 5 16:49:09.696: INFO: cleanup20-16a6f148-bb85-4b73-99f5-2adb1cfdeea0-khffj started at 2023-03-05 16:40:29 +0000 UTC (0+1 container statuses recorded) Mar 5 16:49:09.696: INFO: Container cleanup20-16a6f148-bb85-4b73-99f5-2adb1cfdeea0 ready: true, restart count 0 Mar 5 16:49:09.696: INFO: nodeport-test-88ggz started at 2023-03-05 16:40:39 +0000 UTC (0+1 container statuses recorded) Mar 5 16:49:09.696: INFO: Container nodeport-test ready: true, restart count 0 Mar 5 16:49:09.696: INFO: cleanup20-16a6f148-bb85-4b73-99f5-2adb1cfdeea0-gnfq6 started at 2023-03-05 16:48:01 +0000 UTC (0+1 container statuses recorded) ... skipping 11 lines ... Mar 5 16:49:09.696: INFO: cleanup20-16a6f148-bb85-4b73-99f5-2adb1cfdeea0-sqrjl started at 2023-03-05 16:40:29 +0000 UTC (0+1 container statuses recorded) Mar 5 16:49:09.696: INFO: Container cleanup20-16a6f148-bb85-4b73-99f5-2adb1cfdeea0 ready: true, restart count 0 Mar 5 16:49:09.696: INFO: cleanup20-16a6f148-bb85-4b73-99f5-2adb1cfdeea0-n9vbn started at 2023-03-05 16:40:29 +0000 UTC (0+1 container statuses recorded) Mar 5 16:49:09.696: INFO: Container cleanup20-16a6f148-bb85-4b73-99f5-2adb1cfdeea0 ready: true, restart count 0 Mar 5 16:49:09.696: INFO: cleanup20-16a6f148-bb85-4b73-99f5-2adb1cfdeea0-cqffq started at 2023-03-05 16:48:02 +0000 UTC (0+1 container statuses recorded) Mar 5 16:49:09.696: INFO: Container cleanup20-16a6f148-bb85-4b73-99f5-2adb1cfdeea0 ready: true, restart count 0 Mar 5 16:49:09.696: INFO: fail-once-local-gj678 started at 2023-03-05 16:46:33 +0000 UTC (0+1 container statuses recorded) Mar 5 16:49:09.696: INFO: Container c ready: false, restart count 1 Mar 5 16:49:09.696: INFO: cleanup20-16a6f148-bb85-4b73-99f5-2adb1cfdeea0-m6dgl started at 2023-03-05 16:48:01 +0000 UTC (0+1 container statuses recorded) Mar 5 16:49:09.696: INFO: Container cleanup20-16a6f148-bb85-4b73-99f5-2adb1cfdeea0 ready: true, restart count 0 Mar 5 16:49:09.696: INFO: sample-webhook-deployment-7497495989-s9585 started at 2023-03-05 16:46:38 +0000 UTC (0+1 container statuses recorded) Mar 5 16:49:09.696: INFO: Container sample-webhook ready: true, restart count 0 Mar 5 16:49:09.696: INFO: ebs-csi-node-n98cq started at 2023-03-05 16:25:49 +0000 UTC (0+3 container statuses recorded) Mar 5 16:49:09.696: INFO: Container ebs-plugin ready: true, restart count 0 Mar 5 16:49:09.696: INFO: Container liveness-probe ready: true, restart count 0 Mar 5 16:49:09.696: INFO: Container node-driver-registrar ready: true, restart count 0 Mar 5 16:49:09.696: INFO: cilium-vfgw4 started at 2023-03-05 16:25:49 +0000 UTC (1+1 container statuses recorded) Mar 5 16:49:09.696: INFO: Init container clean-cilium-state ready: true, restart count 0 Mar 5 16:49:09.696: INFO: Container cilium-agent ready: true, restart count 0 Mar 5 16:49:09.696: INFO: fail-once-local-kdbgv started at 2023-03-05 16:46:33 +0000 UTC (0+1 container statuses recorded) Mar 5 16:49:09.696: INFO: Container c ready: false, restart count 1 Mar 5 16:49:09.696: INFO: cleanup20-16a6f148-bb85-4b73-99f5-2adb1cfdeea0-xsqgf started at 2023-03-05 16:40:29 +0000 UTC (0+1 container statuses recorded) Mar 5 16:49:09.696: INFO: Container cleanup20-16a6f148-bb85-4b73-99f5-2adb1cfdeea0 ready: true, restart count 0 Mar 5 16:49:09.696: INFO: cleanup20-16a6f148-bb85-4b73-99f5-2adb1cfdeea0-42w6f started at 2023-03-05 16:40:29 +0000 UTC (0+1 container statuses recorded) Mar 5 16:49:09.696: INFO: Container cleanup20-16a6f148-bb85-4b73-99f5-2adb1cfdeea0 ready: true, restart count 0 Mar 5 16:49:09.696: INFO: cleanup20-16a6f148-bb85-4b73-99f5-2adb1cfdeea0-6lnc6 started at 2023-03-05 16:48:01 +0000 UTC (0+1 container statuses recorded) ... skipping 3 lines ... Mar 5 16:49:09.696: INFO: cleanup20-16a6f148-bb85-4b73-99f5-2adb1cfdeea0-7qxdd started at 2023-03-05 16:48:03 +0000 UTC (0+1 container statuses recorded) Mar 5 16:49:09.696: INFO: Container cleanup20-16a6f148-bb85-4b73-99f5-2adb1cfdeea0 ready: true, restart count 0 Mar 5 16:49:09.696: INFO: coredns-autoscaler-8495f5575-plvdw started at 2023-03-05 16:25:57 +0000 UTC (0+1 container statuses recorded) Mar 5 16:49:09.696: INFO: Container autoscaler ready: true, restart count 0 Mar 5 16:49:09.696: INFO: cleanup20-16a6f148-bb85-4b73-99f5-2adb1cfdeea0-kgbnw started at 2023-03-05 16:40:29 +0000 UTC (0+1 container statuses recorded) Mar 5 16:49:09.696: INFO: Container cleanup20-16a6f148-bb85-4b73-99f5-2adb1cfdeea0 ready: true, restart count 0 Mar 5 16:49:09.696: INFO: fail-once-local-69mfr started at 2023-03-05 16:46:28 +0000 UTC (0+1 container statuses recorded) Mar 5 16:49:09.696: INFO: Container c ready: false, restart count 1 Mar 5 16:49:09.696: INFO: cleanup20-16a6f148-bb85-4b73-99f5-2adb1cfdeea0-wmjdk started at 2023-03-05 16:40:29 +0000 UTC (0+1 container statuses recorded) Mar 5 16:49:09.696: INFO: Container cleanup20-16a6f148-bb85-4b73-99f5-2adb1cfdeea0 ready: true, restart count 0 Mar 5 16:49:09.696: INFO: cleanup20-16a6f148-bb85-4b73-99f5-2adb1cfdeea0-x6s5k started at 2023-03-05 16:40:29 +0000 UTC (0+1 container statuses recorded) Mar 5 16:49:09.696: INFO: Container cleanup20-16a6f148-bb85-4b73-99f5-2adb1cfdeea0 ready: true, restart count 0 Mar 5 16:49:09.696: INFO: ebs-csi-controller-79785c665b-7dmsf started at 2023-03-05 16:25:57 +0000 UTC (0+5 container statuses recorded) ... skipping 12 lines ... Logging node info for node i-06df568f6e5c59fff Mar 5 16:49:10.277: INFO: Node Info: &Node{ObjectMeta:{i-06df568f6e5c59fff cb63c681-2e18-474a-b6f3-221325ac0d52 18301 0 2023-03-05 16:24:16 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:arm64 beta.kubernetes.io/instance-type:t4g.small beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:eu-central-1 failure-domain.beta.kubernetes.io/zone:eu-central-1a karpenter.k8s.aws/instance-category:t karpenter.k8s.aws/instance-cpu:2 karpenter.k8s.aws/instance-family:t4g karpenter.k8s.aws/instance-generation:4 karpenter.k8s.aws/instance-hypervisor:nitro karpenter.k8s.aws/instance-memory:2048 karpenter.k8s.aws/instance-pods:11 karpenter.k8s.aws/instance-size:small karpenter.sh/capacity-type:spot karpenter.sh/initialized:true karpenter.sh/provisioner-name:nodes kubernetes.io/arch:arm64 kubernetes.io/hostname:i-06df568f6e5c59fff kubernetes.io/os:linux node-role.kubernetes.io/node: node-role.kubernetes.io/spot-worker:true node.kubernetes.io/instance-type:t4g.small topology.ebs.csi.aws.com/zone:eu-central-1a topology.hostpath.csi/node:i-06df568f6e5c59fff topology.kubernetes.io/region:eu-central-1 topology.kubernetes.io/zone:eu-central-1a] map[csi.volume.kubernetes.io/nodeid:{"csi-hostpath-ephemeral-2147":"i-06df568f6e5c59fff","csi-hostpath-ephemeral-5408":"i-06df568f6e5c59fff","csi-hostpath-ephemeral-6540":"i-06df568f6e5c59fff","csi-mock-csi-mock-volumes-fsgroup-policy-1058":"i-06df568f6e5c59fff","csi-mock-csi-mock-volumes-workload-2219":"i-06df568f6e5c59fff","ebs.csi.aws.com":"i-06df568f6e5c59fff"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [{karpenter.sh/v1alpha5 Provisioner nodes ac5d7599-b2a2-4eae-8da4-1f4c0a54ec82 <nil> 0xc001e189f0}] [karpenter.sh/termination] [{aws-cloud-controller-manager Update v1 2023-03-05 16:24:16 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{}}}} } {kops-controller Update v1 2023-03-05 16:24:16 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{},"f:node-role.kubernetes.io/spot-worker":{}}}} } {karpenter Update v1 2023-03-05 16:25:57 +0000 UTC FieldsV1 {"f:metadata":{"f:finalizers":{".":{},"v:\"karpenter.sh/termination\"":{}},"f:labels":{".":{},"f:karpenter.k8s.aws/instance-category":{},"f:karpenter.k8s.aws/instance-cpu":{},"f:karpenter.k8s.aws/instance-family":{},"f:karpenter.k8s.aws/instance-generation":{},"f:karpenter.k8s.aws/instance-hypervisor":{},"f:karpenter.k8s.aws/instance-memory":{},"f:karpenter.k8s.aws/instance-pods":{},"f:karpenter.k8s.aws/instance-size":{},"f:karpenter.sh/capacity-type":{},"f:karpenter.sh/initialized":{},"f:karpenter.sh/provisioner-name":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}},"f:ownerReferences":{".":{},"k:{\"uid\":\"ac5d7599-b2a2-4eae-8da4-1f4c0a54ec82\"}":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-03-05 16:26:48 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{".":{},"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"InternalIP\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kube-controller-manager Update v1 2023-03-05 16:44:29 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{".":{},"k:{\"type\":\"DiskPressure\"}":{".":{},"f:type":{}},"k:{\"type\":\"MemoryPressure\"}":{".":{},"f:type":{}},"k:{\"type\":\"PIDPressure\"}":{".":{},"f:type":{}},"k:{\"type\":\"Ready\"}":{".":{},"f:type":{}}},"f:volumesAttached":{}}} status} {kube-controller-manager Update v1 2023-03-05 16:46:37 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:node.alpha.kubernetes.io/ttl":{}},"f:labels":{"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.2.0/24\"":{}},"f:taints":{}}} } {kubelet Update v1 2023-03-05 16:48:40 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{"f:kubernetes.io/hostname":{},"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{".":{},"f:cpu":{},"f:ephemeral-storage":{},"f:hugepages-1Gi":{},"f:hugepages-2Mi":{},"f:hugepages-32Mi":{},"f:hugepages-64Ki":{},"f:memory":{},"f:pods":{}},"f:capacity":{".":{},"f:cpu":{},"f:ephemeral-storage":{},"f:hugepages-1Gi":{},"f:hugepages-2Mi":{},"f:hugepages-32Mi":{},"f:hugepages-64Ki":{},"f:memory":{},"f:pods":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:daemonEndpoints":{"f:kubeletEndpoint":{"f:Port":{}}},"f:images":{},"f:nodeInfo":{"f:architecture":{},"f:bootID":{},"f:containerRuntimeVersion":{},"f:kernelVersion":{},"f:kubeProxyVersion":{},"f:kubeletVersion":{},"f:machineID":{},"f:operatingSystem":{},"f:osImage":{},"f:systemUUID":{}},"f:volumesInUse":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.2.0/24,DoNotUseExternalID:,ProviderID:aws:///eu-central-1a/i-06df568f6e5c59fff,Unschedulable:false,Taints:[]Taint{Taint{Key:node.kubernetes.io/not-ready,Value:,Effect:NoSchedule,TimeAdded:2023-03-05 16:46:33 +0000 UTC,},Taint{Key:node.kubernetes.io/not-ready,Value:,Effect:NoExecute,TimeAdded:2023-03-05 16:46:37 +0000 UTC,},},ConfigSource:nil,PodCIDRs:[100.96.2.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49756430336 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-32Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-64Ki: {{0 0} {<nil>} 0 DecimalSI},memory: {{1941540864 0} {<nil>} 1896036Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44780787229 0} {<nil>} 44780787229 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-32Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-64Ki: {{0 0} {<nil>} 0 DecimalSI},memory: {{1836683264 0} {<nil>} 1793636Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:Ready,Status:False,LastHeartbeatTime:2023-03-05 16:47:34 +0000 UTC,LastTransitionTime:2023-03-05 16:46:08 +0000 UTC,Reason:KubeletNotReady,Message:PLEG is not healthy: pleg was last seen active 8m29.527212456s ago; threshold is 3m0s,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-03-05 16:47:34 +0000 UTC,LastTransitionTime:2023-03-05 16:47:12 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-03-05 16:47:34 +0000 UTC,LastTransitionTime:2023-03-05 16:47:15 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-03-05 16:47:34 +0000 UTC,LastTransitionTime:2023-03-05 16:47:15 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.43.148,},NodeAddress{Type:ExternalIP,Address:18.192.211.222,},NodeAddress{Type:InternalDNS,Address:i-06df568f6e5c59fff.eu-central-1.compute.internal,},NodeAddress{Type:Hostname,Address:i-06df568f6e5c59fff.eu-central-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-18-192-211-222.eu-central-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec221221864d9409a8ec5f6614a9b94b,SystemUUID:ec221221-864d-9409-a8ec-5f6614a9b94b,BootID:553d31bf-faca-417e-a3db-325839e1b8e4,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 22.04.2 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.27.0-alpha.3.52+d48b8167f7094b,KubeProxyVersion:v1.27.0-alpha.3.52+d48b8167f7094b,OperatingSystem:linux,Architecture:arm64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:157636062,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/jessie-dnsutils@sha256:24aaf2626d6b27864c29de2097e8bbb840b3a414271bf7c8995e431e47d8408e registry.k8s.io/e2e-test-images/jessie-dnsutils:1.7],SizeBytes:116764683,},ContainerImage{Names:[registry.k8s.io/kube-proxy-arm64:v1.27.0-alpha.3.52_d48b8167f7094b],SizeBytes:60554082,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:16bbf38c463a4223d8cfe4da12bc61010b082a79b4bb003e2d3ba3ece5dd5f9e registry.k8s.io/e2e-test-images/agnhost:2.43],SizeBytes:49808364,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:3fe7acf013d1264ffded116b80a73dc129a449b0fccdb8d21af8279f2233f36e registry.k8s.io/e2e-test-images/httpd:2.4.39-4],SizeBytes:41528818,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:148b022f5c5da426fc2f3c14b5c0867e58ef05961510c84749ac1fddcb0fef22 registry.k8s.io/e2e-test-images/httpd:2.4.38-4],SizeBytes:40343601,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:28932904,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:ee3b525d5b89db99da3b8eb521d9cd90cb6e9ef0fbb651e98bb37be78d36b5b8 registry.k8s.io/sig-storage/csi-provisioner:v3.3.0],SizeBytes:23945474,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:425d8f1b769398127767b06ed97ce62578a3179bcb99809ce93a1649e025ffe7 registry.k8s.io/sig-storage/csi-resizer:v1.6.0],SizeBytes:22678632,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:291334908ddf71a4661fd7f6d9d97274de8a5378a2b6fdfeb2ce73414a34f82f registry.k8s.io/sig-storage/csi-snapshotter:v6.1.0],SizeBytes:22434180,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:9a685020911e2725ad019dbce6e4a5ab93d51e3d4557f115e64343345e05781b registry.k8s.io/sig-storage/csi-attacher:v4.0.0],SizeBytes:22394501,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:122bfb8c1edabb3c0edd63f06523e6940d958d19b3957dc7b1d6f81e9f1f6119 registry.k8s.io/sig-storage/csi-provisioner:v3.1.0],SizeBytes:21735644,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:9ebbf9f023e7b41ccee3d52afe39a89e3ddacdbb69269d583abfc25847cfd9e4 registry.k8s.io/sig-storage/csi-resizer:v1.4.0],SizeBytes:20848413,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:8b9c313c05f54fb04f8d430896f5f5904b6cb157df261501b29adc04d2b2dc7b registry.k8s.io/sig-storage/csi-attacher:v3.4.0],SizeBytes:20565480,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nonroot@sha256:ee9f50b3c64b174d296d91ca9f69a914ac30e59095dfb462b2b518ad28a63655 registry.k8s.io/e2e-test-images/nonroot:1.4],SizeBytes:18561972,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:92257881c1d6493cf18299a24af42330f891166560047902b8d431fb66b01af5 registry.k8s.io/sig-storage/hostpathplugin:v1.9.0],SizeBytes:18267611,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:9b2d9abda017c28b12f84a344f57cd73fbdb6c2bd7dd5bdd5018246ad1093ba6 registry.k8s.io/sig-storage/hostpathplugin:v1.11.0],SizeBytes:18159085,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:017727efcfeb7d053af68e51436ce8e65edbc6ca573720afb4f79c8594036955 registry.k8s.io/coredns/coredns:v1.10.0],SizeBytes:13766234,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:8502345,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:933940f13b3ea0abc62e656c1aa5c5b47c04b15d71250413a6b821bd0c58b94e registry.k8s.io/sig-storage/livenessprobe:v2.7.0],SizeBytes:8223819,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:7685305,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:5c99cf6a02adda929b10321dbf4ecfa00d87be9ba4fb456006237d530ab4baa1 registry.k8s.io/e2e-test-images/nginx:1.14-4],SizeBytes:6860089,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nonewprivs@sha256:8ac1264691820febacf3aea5d152cbde6d10685731ec14966a9401c6f47a68ac registry.k8s.io/e2e-test-images/nonewprivs:1.3],SizeBytes:5025027,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:2e0f836850e09b8b7cc937681d6194537a09fbd5f6b9e08f4d646a85128e8937 registry.k8s.io/e2e-test-images/busybox:1.29-4],SizeBytes:772933,},ContainerImage{Names:[registry.k8s.io/pause@sha256:7031c1b283388d2c2e09b57badb803c05ebed362dc88d84b480cc47f72a21097 registry.k8s.io/pause:3.9],SizeBytes:268051,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:253553,},},VolumesInUse:[kubernetes.io/csi/csi-hostpath-ephemeral-2147^52df5953-bb73-11ed-8aa7-be49170871b4 kubernetes.io/csi/csi-hostpath-ephemeral-5408^443d1daf-bb73-11ed-95c4-5a95d17cbc27 kubernetes.io/csi/csi-hostpath-ephemeral-5408^44411f51-bb73-11ed-95c4-5a95d17cbc27 kubernetes.io/csi/csi-hostpath-ephemeral-6540^4426a07f-bb73-11ed-a604-92687f2c50f8 kubernetes.io/csi/ebs.csi.aws.com^vol-05b67fab5beb6394b],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-05b67fab5beb6394b,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/csi-hostpath-ephemeral-6540^4426a07f-bb73-11ed-a604-92687f2c50f8,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/csi-hostpath-ephemeral-5408^44411f51-bb73-11ed-95c4-5a95d17cbc27,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/csi-hostpath-ephemeral-5408^443d1daf-bb73-11ed-95c4-5a95d17cbc27,DevicePath:,},},Config:nil,},} Mar 5 16:49:10.277: INFO: Logging kubelet events for node i-06df568f6e5c59fff Mar 5 16:49:10.391: INFO: Logging pods the kubelet thinks is on node i-06df568f6e5c59fff Mar 5 16:49:20.505: INFO: Unable to retrieve kubelet pods for node i-06df568f6e5c59fff: error trying to reach service: net/http: TLS handshake timeout Mar 5 16:49:20.505: INFO: Logging node info for node i-0d6a32f749839192f Mar 5 16:49:20.616: INFO: Node Info: &Node{ObjectMeta:{i-0d6a32f749839192f 4155fefb-4d3a-4b04-ba53-6ac9e895ff92 18509 0 2023-03-05 16:20:46 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:arm64 beta.kubernetes.io/instance-type:c6g.xlarge beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:eu-central-1 failure-domain.beta.kubernetes.io/zone:eu-central-1a kops.k8s.io/kops-controller-pki: kubernetes.io/arch:arm64 kubernetes.io/hostname:i-0d6a32f749839192f kubernetes.io/os:linux node-role.kubernetes.io/control-plane: node.kubernetes.io/exclude-from-external-load-balancers: node.kubernetes.io/instance-type:c6g.xlarge topology.ebs.csi.aws.com/zone:eu-central-1a topology.kubernetes.io/region:eu-central-1 topology.kubernetes.io/zone:eu-central-1a] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-0d6a32f749839192f"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kubelet Update v1 2023-03-05 16:20:46 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {protokube Update v1 2023-03-05 16:21:25 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kops.k8s.io/kops-controller-pki":{},"f:node-role.kubernetes.io/control-plane":{},"f:node.kubernetes.io/exclude-from-external-load-balancers":{}}}} } {kube-controller-manager Update v1 2023-03-05 16:21:47 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.0.0/24\"":{}}}} } {aws-cloud-controller-manager Update v1 2023-03-05 16:21:48 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{},"f:taints":{}}} } {aws-cloud-controller-manager Update v1 2023-03-05 16:21:48 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kubelet Update v1 2023-03-05 16:49:18 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.0.0/24,DoNotUseExternalID:,ProviderID:aws:///eu-central-1a/i-0d6a32f749839192f,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/control-plane,Value:,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[100.96.0.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{4 0} {<nil>} 4 DecimalSI},ephemeral-storage: {{49756430336 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-32Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-64Ki: {{0 0} {<nil>} 0 DecimalSI},memory: {{8150118400 0} {<nil>} 7959100Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{4 0} {<nil>} 4 DecimalSI},ephemeral-storage: {{44780787229 0} {<nil>} 44780787229 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-32Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-64Ki: {{0 0} {<nil>} 0 DecimalSI},memory: {{8045260800 0} {<nil>} 7856700Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-03-05 16:49:18 +0000 UTC,LastTransitionTime:2023-03-05 16:20:46 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-03-05 16:49:18 +0000 UTC,LastTransitionTime:2023-03-05 16:20:46 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-03-05 16:49:18 +0000 UTC,LastTransitionTime:2023-03-05 16:20:46 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-03-05 16:49:18 +0000 UTC,LastTransitionTime:2023-03-05 16:21:47 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.49.247,},NodeAddress{Type:ExternalIP,Address:18.197.20.82,},NodeAddress{Type:InternalDNS,Address:i-0d6a32f749839192f.eu-central-1.compute.internal,},NodeAddress{Type:Hostname,Address:i-0d6a32f749839192f.eu-central-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-18-197-20-82.eu-central-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2bed249e8ac7b517cb08ccd42b83e5,SystemUUID:ec2bed24-9e8a-c7b5-17cb-08ccd42b83e5,BootID:6f9215c9-dfe8-4b13-84f4-9b2084456198,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 22.04.2 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.27.0-alpha.3.52+d48b8167f7094b,KubeProxyVersion:v1.27.0-alpha.3.52+d48b8167f7094b,OperatingSystem:linux,Architecture:arm64,},Images:[]ContainerImage{ContainerImage{Names:[registry.k8s.io/etcd@sha256:9ce33ba33d8e738a5b85ed50b5080ac746deceed4a7496c550927a7a19ca3b6d registry.k8s.io/etcd:3.5.0-0],SizeBytes:157797353,},ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:157636062,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:4afb99b4690b418ffc2ceb67e1a17376457e441c1f09ab55447f0aaf992fa646 registry.k8s.io/etcd:3.4.3-0],SizeBytes:151677080,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:4ad90a11b55313b182afc186b9876c8e891531b8db4c9bf1541953021618d0e2 registry.k8s.io/etcd:3.4.13-0],SizeBytes:134531559,},ContainerImage{Names:[registry.k8s.io/kube-apiserver-arm64:v1.27.0-alpha.3.52_d48b8167f7094b],SizeBytes:131087707,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:0d2ab6b15b4fe0b903b9e81192d942fb39c29be61fe341cd16db5d0165b0435c registry.k8s.io/etcd:3.3.17-0],SizeBytes:126979107,},ContainerImage{Names:[registry.k8s.io/kube-controller-manager-arm64:v1.27.0-alpha.3.52_d48b8167f7094b],SizeBytes:120667128,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:17da501f5d2a675be46040422a27b7cc21b8a43895ac998b171db1c346f361f7 registry.k8s.io/etcd:3.3.10-0],SizeBytes:81436097,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:6f72b851544986cb0921b53ea655ec04c36131248f16d4ad110cb3ca0c369dc1 registry.k8s.io/etcd:3.5.4-0],SizeBytes:81120118,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:13f53ed1d91e2e11aac476ee9a0269fdda6cc4874eba903efd40daf50c55eee5 registry.k8s.io/etcd:3.5.3-0],SizeBytes:81114891,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:51eae8381dcb1078289fa7b4f3df2630cdc18d09fb56f8e56b41c40e191d6c83 registry.k8s.io/etcd:3.5.7-0],SizeBytes:80665728,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:dd75ec974b0a2a6f6bb47001ba09207976e625db898d1b16735528c009cb171c registry.k8s.io/etcd:3.5.6-0],SizeBytes:80539316,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:905d7ca17fd02bc24c0eba9a062753aba15db3e31422390bc3238eb762339b20 registry.k8s.io/etcd:3.2.24-1],SizeBytes:61500433,},ContainerImage{Names:[registry.k8s.io/etcdadm/etcd-manager-slim@sha256:8727adc059334e0e3329dbccce116a734da4a090da5e86c8cddccaf333caa666 registry.k8s.io/etcdadm/etcd-manager-slim:v3.0.20230201],SizeBytes:60826655,},ContainerImage{Names:[registry.k8s.io/kube-proxy-arm64:v1.27.0-alpha.3.52_d48b8167f7094b],SizeBytes:60554082,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:64b9ea357325d5db9f8a723dcf503b5a449177b17ac87d69481e126bb724c263 registry.k8s.io/etcd:3.5.1-0],SizeBytes:59553904,},ContainerImage{Names:[registry.k8s.io/kube-scheduler-arm64:v1.27.0-alpha.3.52_d48b8167f7094b],SizeBytes:56310763,},ContainerImage{Names:[registry.k8s.io/kops/kops-controller:1.27.0-alpha.1],SizeBytes:41797411,},ContainerImage{Names:[registry.k8s.io/kops/dns-controller:1.27.0-alpha.1],SizeBytes:40267473,},ContainerImage{Names:[public.ecr.aws/karpenter/controller@sha256:68db4f092cf9cc83f5ef9e2fbc5407c2cb682e81f64dfaa700a7602ede38b1cf public.ecr.aws/karpenter/controller:v0.16.3],SizeBytes:33707751,},ContainerImage{Names:[public.ecr.aws/karpenter/webhook@sha256:96a2d9b06d6bc5127801f358f74b1cf2d289b423a2e9ba40c573c0b14b17dafa public.ecr.aws/karpenter/webhook:v0.16.3],SizeBytes:33307187,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:28932904,},ContainerImage{Names:[quay.io/cilium/operator@sha256:a6d24a006a6b92967ac90786b49bc1ac26e5477cf028cd1186efcfc2466484db quay.io/cilium/operator:v1.12.5],SizeBytes:24567287,},ContainerImage{Names:[registry.k8s.io/provider-aws/cloud-controller-manager@sha256:fdeb61e3e42ecd9cca868d550ebdb88dd6341d9e91fcfa9a37e227dab2ad22cb registry.k8s.io/provider-aws/cloud-controller-manager:v1.26.0],SizeBytes:18345226,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:8502345,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:7685305,},ContainerImage{Names:[registry.k8s.io/kops/kube-apiserver-healthcheck:1.27.0-alpha.1],SizeBytes:4713612,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:253553,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Mar 5 16:49:20.616: INFO: Logging kubelet events for node i-0d6a32f749839192f Mar 5 16:49:20.731: INFO: ... skipping 52 lines ... Mar 5 16:49:20.849: INFO: Container kube-scheduler ready: true, restart count 0 Mar 5 16:49:21.356: INFO: Latency metrics for node i-0d6a32f749839192f [1mSTEP:[0m Destroying namespace "projected-8785" for this suite. [38;5;243m@ 03/05/23 16:49:21.356[0m [38;5;243m<< Timeline[0m [38;5;9m[FAILED] expected pod "pod-projected-configmaps-404de0a7-eaf4-4291-8862-6e481dc16211" success: Timed out after 300.014s. expected pod to be Succeeded or Failed, got instead: <*v1.Pod | 0xc0030d2480>: metadata: creationTimestamp: "2023-03-05T16:40:29Z" managedFields: - apiVersion: v1 fieldsType: FieldsV1 ... skipping 113 lines ... phase: Pending qosClass: BestEffort[0m [38;5;9mIn [1m[It][0m[38;5;9m at: [1mtest/e2e/framework/pod/output/output.go:238[0m [38;5;243m@ 03/05/23 16:46:08.563[0m [38;5;243m------------------------------[0m [38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m [38;5;243m------------------------------[0m [38;5;9m• [FAILED] [1024.151 seconds][0m [0m[sig-storage] CSI Mock volume fsgroup policies [38;5;243mCSI FSGroupPolicy [LinuxOnly] [38;5;9m[1m[It] should not modify fsGroup if fsGroupPolicy=None[0m [38;5;243mtest/e2e/storage/csi_mock/csi_fsgroup_policy.go:66[0m [38;5;243mTimeline >>[0m [1mSTEP:[0m Creating a kubernetes client [38;5;243m@ 03/05/23 16:32:28.921[0m Mar 5 16:32:28.921: INFO: >>> kubeConfig: /root/.kube/config ... skipping 36 lines ... Mar 5 16:32:33.648: INFO: creating *v1.ClusterRoleBinding: csi-controller-snapshotter-role-csi-mock-volumes-fsgroup-policy-1058 Mar 5 16:32:33.761: INFO: creating *v1.StorageClass: csi-mock-sc-csi-mock-volumes-fsgroup-policy-1058 Mar 5 16:32:33.875: INFO: creating *v1.StatefulSet: csi-mock-volumes-fsgroup-policy-1058-5723/csi-mockplugin Mar 5 16:32:33.990: INFO: creating *v1.CSIDriver: csi-mock-csi-mock-volumes-fsgroup-policy-1058 Mar 5 16:32:34.102: INFO: waiting up to 4m0s for CSIDriver "csi-mock-csi-mock-volumes-fsgroup-policy-1058" Mar 5 16:32:34.214: INFO: waiting for CSIDriver csi-mock-csi-mock-volumes-fsgroup-policy-1058 to register on node i-06df568f6e5c59fff Mar 5 16:38:17.546: INFO: Unexpected error: Failed to register CSIDriver csi-mock-csi-mock-volumes-fsgroup-policy-1058: <*errors.errorString | 0xc0016ba250>: { s: "error waiting for CSI driver csi-mock-csi-mock-volumes-fsgroup-policy-1058 registration on node i-06df568f6e5c59fff: timed out waiting for the condition", } [38;5;9m[FAILED][0m in [It] - test/e2e/storage/csi_mock/base.go:206 [38;5;243m@ 03/05/23 16:38:17.547[0m Mar 5 16:38:17.547: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready Mar 5 16:38:17.659: INFO: Condition Ready of node i-06df568f6e5c59fff is false, but Node is tainted by NodeController with [{node.kubernetes.io/not-ready NoSchedule 2023-03-05 16:37:27 +0000 UTC} {node.kubernetes.io/not-ready NoExecute 2023-03-05 16:37:31 +0000 UTC}]. Failure Mar 5 16:38:19.772: INFO: Condition Ready of node i-06df568f6e5c59fff is false, but Node is tainted by NodeController with [{node.kubernetes.io/not-ready NoSchedule 2023-03-05 16:37:27 +0000 UTC} {node.kubernetes.io/not-ready NoExecute 2023-03-05 16:37:31 +0000 UTC}]. Failure Mar 5 16:38:21.772: INFO: Condition Ready of node i-06df568f6e5c59fff is false, but Node is tainted by NodeController with [{node.kubernetes.io/not-ready NoSchedule 2023-03-05 16:37:27 +0000 UTC} {node.kubernetes.io/not-ready NoExecute 2023-03-05 16:37:31 +0000 UTC}]. Failure Mar 5 16:38:23.773: INFO: Condition Ready of node i-06df568f6e5c59fff is false, but Node is tainted by NodeController with [{node.kubernetes.io/not-ready NoSchedule 2023-03-05 16:37:27 +0000 UTC} {node.kubernetes.io/not-ready NoExecute 2023-03-05 16:37:31 +0000 UTC}]. Failure Mar 5 16:38:25.773: INFO: Condition Ready of node i-06df568f6e5c59fff is false, but Node is tainted by NodeController with [{node.kubernetes.io/not-ready NoSchedule 2023-03-05 16:37:27 +0000 UTC} {node.kubernetes.io/not-ready NoExecute 2023-03-05 16:37:31 +0000 UTC}]. Failure ... skipping 259 lines ... Mar 5 16:41:34.004: INFO: Container csi-snapshotter ready: true, restart count 0 Mar 5 16:41:34.004: INFO: Container hostpath ready: true, restart count 0 Mar 5 16:41:34.004: INFO: Container liveness-probe ready: true, restart count 0 Mar 5 16:41:34.004: INFO: Container node-driver-registrar ready: true, restart count 0 Mar 5 16:41:34.004: INFO: simpletest.rc-mctk9 started at <nil> (0+0 container statuses recorded) Mar 5 16:41:34.004: INFO: simpletest.rc-lmsnp started at <nil> (0+0 container statuses recorded) Mar 5 16:43:00.753: INFO: Unexpected error: <*errors.StatusError | 0xc000101860>: { ErrStatus: code: 503 details: causes: - message: unknown ... skipping 14 lines ... Mar 5 16:43:07.094: INFO: At 2023-03-05 16:32:33 +0000 UTC - event for csi-mockplugin-0: {default-scheduler } Scheduled: Successfully assigned csi-mock-volumes-fsgroup-policy-1058-5723/csi-mockplugin-0 to i-06df568f6e5c59fff Mar 5 16:43:07.094: INFO: At 2023-03-05 16:38:26 +0000 UTC - event for csi-mockplugin-0: {taint-controller } TaintManagerEviction: Marking for deletion Pod csi-mock-volumes-fsgroup-policy-1058-5723/csi-mockplugin-0 Mar 5 16:43:07.094: INFO: At 2023-03-05 16:41:51 +0000 UTC - event for csi-mockplugin-0: {taint-controller } TaintManagerEviction: Cancelling deletion of Pod csi-mock-volumes-fsgroup-policy-1058-5723/csi-mockplugin-0 Mar 5 16:43:07.207: INFO: POD NODE PHASE GRACE CONDITIONS Mar 5 16:43:07.207: INFO: csi-mockplugin-0 i-06df568f6e5c59fff Pending 30s [{PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2023-03-05 16:32:33 +0000 UTC } {DisruptionTarget True 0001-01-01 00:00:00 +0000 UTC 2023-03-05 16:38:26 +0000 UTC DeletionByTaintManager Taint manager: deleting due to NoExecute taint}] Mar 5 16:43:07.207: INFO: Mar 5 16:43:17.320: INFO: Unable to fetch csi-mock-volumes-fsgroup-policy-1058-5723/csi-mockplugin-0/csi-provisioner logs: an error on the server ("unknown") has prevented the request from succeeding (get pods csi-mockplugin-0) Mar 5 16:43:27.433: INFO: Unable to fetch csi-mock-volumes-fsgroup-policy-1058-5723/csi-mockplugin-0/driver-registrar logs: an error on the server ("unknown") has prevented the request from succeeding (get pods csi-mockplugin-0) Mar 5 16:43:37.547: INFO: Unable to fetch csi-mock-volumes-fsgroup-policy-1058-5723/csi-mockplugin-0/mock logs: an error on the server ("unknown") has prevented the request from succeeding (get pods csi-mockplugin-0) Mar 5 16:43:37.658: INFO: Logging node info for node i-056ad61525499081e Mar 5 16:43:37.770: INFO: Node Info: &Node{ObjectMeta:{i-056ad61525499081e a6ce9ed3-5ae8-4065-b117-6884e24b3632 16141 0 2023-03-05 16:24:16 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:arm64 beta.kubernetes.io/instance-type:t4g.small beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:eu-central-1 failure-domain.beta.kubernetes.io/zone:eu-central-1a karpenter.k8s.aws/instance-category:t karpenter.k8s.aws/instance-cpu:2 karpenter.k8s.aws/instance-family:t4g karpenter.k8s.aws/instance-generation:4 karpenter.k8s.aws/instance-hypervisor:nitro karpenter.k8s.aws/instance-memory:2048 karpenter.k8s.aws/instance-pods:11 karpenter.k8s.aws/instance-size:small karpenter.sh/capacity-type:spot karpenter.sh/initialized:true karpenter.sh/provisioner-name:nodes kubelet_cleanup:true kubernetes.io/arch:arm64 kubernetes.io/hostname:i-056ad61525499081e kubernetes.io/os:linux node-role.kubernetes.io/node: node-role.kubernetes.io/spot-worker:true node.kubernetes.io/instance-type:t4g.small topology.ebs.csi.aws.com/zone:eu-central-1a topology.hostpath.csi/node:i-056ad61525499081e topology.kubernetes.io/region:eu-central-1 topology.kubernetes.io/zone:eu-central-1a] map[csi.volume.kubernetes.io/nodeid:{"csi-hostpath-volumelimits-3711":"i-056ad61525499081e","csi-mock-csi-mock-volumes-attach-7428":"i-056ad61525499081e","ebs.csi.aws.com":"i-056ad61525499081e"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [{karpenter.sh/v1alpha5 Provisioner nodes ac5d7599-b2a2-4eae-8da4-1f4c0a54ec82 <nil> 0xc00329d380}] [karpenter.sh/termination] [{aws-cloud-controller-manager Update v1 2023-03-05 16:24:16 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{}}}} } {kops-controller Update v1 2023-03-05 16:24:16 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{},"f:node-role.kubernetes.io/spot-worker":{}}}} } {karpenter Update v1 2023-03-05 16:25:57 +0000 UTC FieldsV1 {"f:metadata":{"f:finalizers":{".":{},"v:\"karpenter.sh/termination\"":{}},"f:labels":{".":{},"f:karpenter.k8s.aws/instance-category":{},"f:karpenter.k8s.aws/instance-cpu":{},"f:karpenter.k8s.aws/instance-family":{},"f:karpenter.k8s.aws/instance-generation":{},"f:karpenter.k8s.aws/instance-hypervisor":{},"f:karpenter.k8s.aws/instance-memory":{},"f:karpenter.k8s.aws/instance-pods":{},"f:karpenter.k8s.aws/instance-size":{},"f:karpenter.sh/capacity-type":{},"f:karpenter.sh/initialized":{},"f:karpenter.sh/provisioner-name":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}},"f:ownerReferences":{".":{},"k:{\"uid\":\"ac5d7599-b2a2-4eae-8da4-1f4c0a54ec82\"}":{}}},"f:spec":{"f:providerID":{}}} } {kube-controller-manager Update v1 2023-03-05 16:25:57 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:node.alpha.kubernetes.io/ttl":{}},"f:labels":{"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.1.0/24\"":{}}}} } {aws-cloud-controller-manager Update v1 2023-03-05 16:26:48 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{".":{},"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"InternalIP\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {e2e.test Update v1 2023-03-05 16:40:28 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubelet_cleanup":{}}}} } {kube-controller-manager Update v1 2023-03-05 16:43:12 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{".":{},"k:{\"type\":\"DiskPressure\"}":{".":{},"f:type":{}},"k:{\"type\":\"MemoryPressure\"}":{".":{},"f:type":{}},"k:{\"type\":\"PIDPressure\"}":{".":{},"f:type":{}},"k:{\"type\":\"Ready\"}":{".":{},"f:type":{}}},"f:volumesAttached":{}}} status} {kubelet Update v1 2023-03-05 16:43:18 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{"f:kubernetes.io/hostname":{},"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{".":{},"f:cpu":{},"f:ephemeral-storage":{},"f:hugepages-1Gi":{},"f:hugepages-2Mi":{},"f:hugepages-32Mi":{},"f:hugepages-64Ki":{},"f:memory":{},"f:pods":{}},"f:capacity":{".":{},"f:cpu":{},"f:ephemeral-storage":{},"f:hugepages-1Gi":{},"f:hugepages-2Mi":{},"f:hugepages-32Mi":{},"f:hugepages-64Ki":{},"f:memory":{},"f:pods":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:daemonEndpoints":{"f:kubeletEndpoint":{"f:Port":{}}},"f:images":{},"f:nodeInfo":{"f:architecture":{},"f:bootID":{},"f:containerRuntimeVersion":{},"f:kernelVersion":{},"f:kubeProxyVersion":{},"f:kubeletVersion":{},"f:machineID":{},"f:operatingSystem":{},"f:osImage":{},"f:systemUUID":{}},"f:volumesInUse":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.1.0/24,DoNotUseExternalID:,ProviderID:aws:///eu-central-1a/i-056ad61525499081e,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.1.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49756430336 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-32Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-64Ki: {{0 0} {<nil>} 0 DecimalSI},memory: {{1941540864 0} {<nil>} 1896036Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44780787229 0} {<nil>} 44780787229 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-32Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-64Ki: {{0 0} {<nil>} 0 DecimalSI},memory: {{1836683264 0} {<nil>} 1793636Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-03-05 16:43:18 +0000 UTC,LastTransitionTime:2023-03-05 16:25:57 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-03-05 16:43:18 +0000 UTC,LastTransitionTime:2023-03-05 16:25:48 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-03-05 16:43:18 +0000 UTC,LastTransitionTime:2023-03-05 16:25:48 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-03-05 16:43:18 +0000 UTC,LastTransitionTime:2023-03-05 16:25:48 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.36.179,},NodeAddress{Type:ExternalIP,Address:18.159.37.200,},NodeAddress{Type:InternalDNS,Address:i-056ad61525499081e.eu-central-1.compute.internal,},NodeAddress{Type:Hostname,Address:i-056ad61525499081e.eu-central-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-18-159-37-200.eu-central-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2af3207d1c8c09785c337aae6c6ef0,SystemUUID:ec2af320-7d1c-8c09-785c-337aae6c6ef0,BootID:786f2752-bc30-4729-8aaf-45977b16c253,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 22.04.2 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.27.0-alpha.3.52+d48b8167f7094b,KubeProxyVersion:v1.27.0-alpha.3.52+d48b8167f7094b,OperatingSystem:linux,Architecture:arm64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:157636062,},ContainerImage{Names:[registry.k8s.io/kube-proxy-arm64:v1.27.0-alpha.3.52_d48b8167f7094b],SizeBytes:60554082,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:16bbf38c463a4223d8cfe4da12bc61010b082a79b4bb003e2d3ba3ece5dd5f9e registry.k8s.io/e2e-test-images/agnhost:2.43],SizeBytes:49808364,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:3fe7acf013d1264ffded116b80a73dc129a449b0fccdb8d21af8279f2233f36e registry.k8s.io/e2e-test-images/httpd:2.4.39-4],SizeBytes:41528818,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:148b022f5c5da426fc2f3c14b5c0867e58ef05961510c84749ac1fddcb0fef22 registry.k8s.io/e2e-test-images/httpd:2.4.38-4],SizeBytes:40343601,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:28932904,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:ee3b525d5b89db99da3b8eb521d9cd90cb6e9ef0fbb651e98bb37be78d36b5b8 registry.k8s.io/sig-storage/csi-provisioner:v3.3.0],SizeBytes:23945474,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:425d8f1b769398127767b06ed97ce62578a3179bcb99809ce93a1649e025ffe7 registry.k8s.io/sig-storage/csi-resizer:v1.6.0],SizeBytes:22678632,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:291334908ddf71a4661fd7f6d9d97274de8a5378a2b6fdfeb2ce73414a34f82f registry.k8s.io/sig-storage/csi-snapshotter:v6.1.0],SizeBytes:22434180,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:9a685020911e2725ad019dbce6e4a5ab93d51e3d4557f115e64343345e05781b registry.k8s.io/sig-storage/csi-attacher:v4.0.0],SizeBytes:22394501,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:122bfb8c1edabb3c0edd63f06523e6940d958d19b3957dc7b1d6f81e9f1f6119 registry.k8s.io/sig-storage/csi-provisioner:v3.1.0],SizeBytes:21735644,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:9ebbf9f023e7b41ccee3d52afe39a89e3ddacdbb69269d583abfc25847cfd9e4 registry.k8s.io/sig-storage/csi-resizer:v1.4.0],SizeBytes:20848413,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:8b9c313c05f54fb04f8d430896f5f5904b6cb157df261501b29adc04d2b2dc7b registry.k8s.io/sig-storage/csi-attacher:v3.4.0],SizeBytes:20565480,},ContainerImage{Names:[registry.k8s.io/cpa/cluster-proportional-autoscaler@sha256:68d396900aeaa072c1f27289485fdac29834045a6f3ffe369bf389d830ef572d registry.k8s.io/cpa/cluster-proportional-autoscaler:1.8.6],SizeBytes:19007623,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:92257881c1d6493cf18299a24af42330f891166560047902b8d431fb66b01af5 registry.k8s.io/sig-storage/hostpathplugin:v1.9.0],SizeBytes:18267611,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:9b2d9abda017c28b12f84a344f57cd73fbdb6c2bd7dd5bdd5018246ad1093ba6 registry.k8s.io/sig-storage/hostpathplugin:v1.11.0],SizeBytes:18159085,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:017727efcfeb7d053af68e51436ce8e65edbc6ca573720afb4f79c8594036955 registry.k8s.io/coredns/coredns:v1.10.0],SizeBytes:13766234,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:8502345,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:933940f13b3ea0abc62e656c1aa5c5b47c04b15d71250413a6b821bd0c58b94e registry.k8s.io/sig-storage/livenessprobe:v2.7.0],SizeBytes:8223819,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:7685305,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:5c99cf6a02adda929b10321dbf4ecfa00d87be9ba4fb456006237d530ab4baa1 registry.k8s.io/e2e-test-images/nginx:1.14-4],SizeBytes:6860089,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:2e0f836850e09b8b7cc937681d6194537a09fbd5f6b9e08f4d646a85128e8937 registry.k8s.io/e2e-test-images/busybox:1.29-4],SizeBytes:772933,},ContainerImage{Names:[registry.k8s.io/pause@sha256:7031c1b283388d2c2e09b57badb803c05ebed362dc88d84b480cc47f72a21097 registry.k8s.io/pause:3.9],SizeBytes:268051,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:253553,},},VolumesInUse:[kubernetes.io/csi/csi-mock-csi-mock-volumes-attach-7428^d10cad7f-bb74-11ed-a6f8-c6925762327d kubernetes.io/csi/ebs.csi.aws.com^vol-0e020446b1cd6abaf],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-0e020446b1cd6abaf,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/csi-mock-csi-mock-volumes-attach-7428^d10cad7f-bb74-11ed-a6f8-c6925762327d,DevicePath:,},},Config:nil,},} Mar 5 16:43:37.770: INFO: Logging kubelet events for node i-056ad61525499081e Mar 5 16:43:37.885: INFO: ... skipping 63 lines ... Logging node info for node i-06df568f6e5c59fff Mar 5 16:43:38.596: INFO: Node Info: &Node{ObjectMeta:{i-06df568f6e5c59fff cb63c681-2e18-474a-b6f3-221325ac0d52 15784 0 2023-03-05 16:24:16 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:arm64 beta.kubernetes.io/instance-type:t4g.small beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:eu-central-1 failure-domain.beta.kubernetes.io/zone:eu-central-1a karpenter.k8s.aws/instance-category:t karpenter.k8s.aws/instance-cpu:2 karpenter.k8s.aws/instance-family:t4g karpenter.k8s.aws/instance-generation:4 karpenter.k8s.aws/instance-hypervisor:nitro karpenter.k8s.aws/instance-memory:2048 karpenter.k8s.aws/instance-pods:11 karpenter.k8s.aws/instance-size:small karpenter.sh/capacity-type:spot karpenter.sh/initialized:true karpenter.sh/provisioner-name:nodes kubelet_cleanup:true kubernetes.io/arch:arm64 kubernetes.io/hostname:i-06df568f6e5c59fff kubernetes.io/os:linux node-role.kubernetes.io/node: node-role.kubernetes.io/spot-worker:true node.kubernetes.io/instance-type:t4g.small topology.ebs.csi.aws.com/zone:eu-central-1a topology.hostpath.csi/node:i-06df568f6e5c59fff topology.kubernetes.io/region:eu-central-1 topology.kubernetes.io/zone:eu-central-1a] map[csi.volume.kubernetes.io/nodeid:{"csi-hostpath-ephemeral-2147":"i-06df568f6e5c59fff","csi-hostpath-ephemeral-5408":"i-06df568f6e5c59fff","csi-hostpath-ephemeral-6540":"i-06df568f6e5c59fff","csi-mock-csi-mock-volumes-fsgroup-policy-1058":"i-06df568f6e5c59fff","csi-mock-csi-mock-volumes-workload-2219":"i-06df568f6e5c59fff","ebs.csi.aws.com":"i-06df568f6e5c59fff"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [{karpenter.sh/v1alpha5 Provisioner nodes ac5d7599-b2a2-4eae-8da4-1f4c0a54ec82 <nil> 0xc0026a4260}] [karpenter.sh/termination] [{aws-cloud-controller-manager Update v1 2023-03-05 16:24:16 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{}}}} } {kops-controller Update v1 2023-03-05 16:24:16 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{},"f:node-role.kubernetes.io/spot-worker":{}}}} } {karpenter Update v1 2023-03-05 16:25:57 +0000 UTC FieldsV1 {"f:metadata":{"f:finalizers":{".":{},"v:\"karpenter.sh/termination\"":{}},"f:labels":{".":{},"f:karpenter.k8s.aws/instance-category":{},"f:karpenter.k8s.aws/instance-cpu":{},"f:karpenter.k8s.aws/instance-family":{},"f:karpenter.k8s.aws/instance-generation":{},"f:karpenter.k8s.aws/instance-hypervisor":{},"f:karpenter.k8s.aws/instance-memory":{},"f:karpenter.k8s.aws/instance-pods":{},"f:karpenter.k8s.aws/instance-size":{},"f:karpenter.sh/capacity-type":{},"f:karpenter.sh/initialized":{},"f:karpenter.sh/provisioner-name":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}},"f:ownerReferences":{".":{},"k:{\"uid\":\"ac5d7599-b2a2-4eae-8da4-1f4c0a54ec82\"}":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-03-05 16:26:48 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{".":{},"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"InternalIP\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {e2e.test Update v1 2023-03-05 16:40:28 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubelet_cleanup":{}}}} } {kubelet Update v1 2023-03-05 16:41:47 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{"f:kubernetes.io/hostname":{},"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{".":{},"f:cpu":{},"f:ephemeral-storage":{},"f:hugepages-1Gi":{},"f:hugepages-2Mi":{},"f:hugepages-32Mi":{},"f:hugepages-64Ki":{},"f:memory":{},"f:pods":{}},"f:capacity":{".":{},"f:cpu":{},"f:ephemeral-storage":{},"f:hugepages-1Gi":{},"f:hugepages-2Mi":{},"f:hugepages-32Mi":{},"f:hugepages-64Ki":{},"f:memory":{},"f:pods":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{}}},"f:daemonEndpoints":{"f:kubeletEndpoint":{"f:Port":{}}},"f:images":{},"f:nodeInfo":{"f:architecture":{},"f:bootID":{},"f:containerRuntimeVersion":{},"f:kernelVersion":{},"f:kubeProxyVersion":{},"f:kubeletVersion":{},"f:machineID":{},"f:operatingSystem":{},"f:osImage":{},"f:systemUUID":{}},"f:volumesInUse":{}}} status} {kube-controller-manager Update v1 2023-03-05 16:42:51 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{".":{},"k:{\"type\":\"DiskPressure\"}":{".":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"MemoryPressure\"}":{".":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"PIDPressure\"}":{".":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Ready\"}":{".":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}},"f:volumesAttached":{}}} status} {kube-controller-manager Update v1 2023-03-05 16:42:57 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:node.alpha.kubernetes.io/ttl":{}},"f:labels":{"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.2.0/24\"":{}},"f:taints":{}}} }]},Spec:NodeSpec{PodCIDR:100.96.2.0/24,DoNotUseExternalID:,ProviderID:aws:///eu-central-1a/i-06df568f6e5c59fff,Unschedulable:false,Taints:[]Taint{Taint{Key:node.kubernetes.io/unreachable,Value:,Effect:NoSchedule,TimeAdded:2023-03-05 16:42:51 +0000 UTC,},Taint{Key:node.kubernetes.io/unreachable,Value:,Effect:NoExecute,TimeAdded:2023-03-05 16:42:57 +0000 UTC,},},ConfigSource:nil,PodCIDRs:[100.96.2.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49756430336 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-32Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-64Ki: {{0 0} {<nil>} 0 DecimalSI},memory: {{1941540864 0} {<nil>} 1896036Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44780787229 0} {<nil>} 44780787229 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-32Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-64Ki: {{0 0} {<nil>} 0 DecimalSI},memory: {{1836683264 0} {<nil>} 1793636Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:Ready,Status:Unknown,LastHeartbeatTime:2023-03-05 16:41:39 +0000 UTC,LastTransitionTime:2023-03-05 16:42:51 +0000 UTC,Reason:NodeStatusUnknown,Message:Kubelet stopped posting node status.,},NodeCondition{Type:MemoryPressure,Status:Unknown,LastHeartbeatTime:2023-03-05 16:41:39 +0000 UTC,LastTransitionTime:2023-03-05 16:42:51 +0000 UTC,Reason:NodeStatusUnknown,Message:Kubelet stopped posting node status.,},NodeCondition{Type:DiskPressure,Status:Unknown,LastHeartbeatTime:2023-03-05 16:41:39 +0000 UTC,LastTransitionTime:2023-03-05 16:42:51 +0000 UTC,Reason:NodeStatusUnknown,Message:Kubelet stopped posting node status.,},NodeCondition{Type:PIDPressure,Status:Unknown,LastHeartbeatTime:2023-03-05 16:41:39 +0000 UTC,LastTransitionTime:2023-03-05 16:42:51 +0000 UTC,Reason:NodeStatusUnknown,Message:Kubelet stopped posting node status.,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.43.148,},NodeAddress{Type:ExternalIP,Address:18.192.211.222,},NodeAddress{Type:InternalDNS,Address:i-06df568f6e5c59fff.eu-central-1.compute.internal,},NodeAddress{Type:Hostname,Address:i-06df568f6e5c59fff.eu-central-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-18-192-211-222.eu-central-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec221221864d9409a8ec5f6614a9b94b,SystemUUID:ec221221-864d-9409-a8ec-5f6614a9b94b,BootID:553d31bf-faca-417e-a3db-325839e1b8e4,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 22.04.2 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.27.0-alpha.3.52+d48b8167f7094b,KubeProxyVersion:v1.27.0-alpha.3.52+d48b8167f7094b,OperatingSystem:linux,Architecture:arm64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:157636062,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/jessie-dnsutils@sha256:24aaf2626d6b27864c29de2097e8bbb840b3a414271bf7c8995e431e47d8408e registry.k8s.io/e2e-test-images/jessie-dnsutils:1.7],SizeBytes:116764683,},ContainerImage{Names:[registry.k8s.io/kube-proxy-arm64:v1.27.0-alpha.3.52_d48b8167f7094b],SizeBytes:60554082,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:16bbf38c463a4223d8cfe4da12bc61010b082a79b4bb003e2d3ba3ece5dd5f9e registry.k8s.io/e2e-test-images/agnhost:2.43],SizeBytes:49808364,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:3fe7acf013d1264ffded116b80a73dc129a449b0fccdb8d21af8279f2233f36e registry.k8s.io/e2e-test-images/httpd:2.4.39-4],SizeBytes:41528818,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:148b022f5c5da426fc2f3c14b5c0867e58ef05961510c84749ac1fddcb0fef22 registry.k8s.io/e2e-test-images/httpd:2.4.38-4],SizeBytes:40343601,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:28932904,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:ee3b525d5b89db99da3b8eb521d9cd90cb6e9ef0fbb651e98bb37be78d36b5b8 registry.k8s.io/sig-storage/csi-provisioner:v3.3.0],SizeBytes:23945474,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:425d8f1b769398127767b06ed97ce62578a3179bcb99809ce93a1649e025ffe7 registry.k8s.io/sig-storage/csi-resizer:v1.6.0],SizeBytes:22678632,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:291334908ddf71a4661fd7f6d9d97274de8a5378a2b6fdfeb2ce73414a34f82f registry.k8s.io/sig-storage/csi-snapshotter:v6.1.0],SizeBytes:22434180,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:9a685020911e2725ad019dbce6e4a5ab93d51e3d4557f115e64343345e05781b registry.k8s.io/sig-storage/csi-attacher:v4.0.0],SizeBytes:22394501,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:122bfb8c1edabb3c0edd63f06523e6940d958d19b3957dc7b1d6f81e9f1f6119 registry.k8s.io/sig-storage/csi-provisioner:v3.1.0],SizeBytes:21735644,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:9ebbf9f023e7b41ccee3d52afe39a89e3ddacdbb69269d583abfc25847cfd9e4 registry.k8s.io/sig-storage/csi-resizer:v1.4.0],SizeBytes:20848413,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:8b9c313c05f54fb04f8d430896f5f5904b6cb157df261501b29adc04d2b2dc7b registry.k8s.io/sig-storage/csi-attacher:v3.4.0],SizeBytes:20565480,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nonroot@sha256:ee9f50b3c64b174d296d91ca9f69a914ac30e59095dfb462b2b518ad28a63655 registry.k8s.io/e2e-test-images/nonroot:1.4],SizeBytes:18561972,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:92257881c1d6493cf18299a24af42330f891166560047902b8d431fb66b01af5 registry.k8s.io/sig-storage/hostpathplugin:v1.9.0],SizeBytes:18267611,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:9b2d9abda017c28b12f84a344f57cd73fbdb6c2bd7dd5bdd5018246ad1093ba6 registry.k8s.io/sig-storage/hostpathplugin:v1.11.0],SizeBytes:18159085,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:017727efcfeb7d053af68e51436ce8e65edbc6ca573720afb4f79c8594036955 registry.k8s.io/coredns/coredns:v1.10.0],SizeBytes:13766234,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:8502345,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:933940f13b3ea0abc62e656c1aa5c5b47c04b15d71250413a6b821bd0c58b94e registry.k8s.io/sig-storage/livenessprobe:v2.7.0],SizeBytes:8223819,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:7685305,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:5c99cf6a02adda929b10321dbf4ecfa00d87be9ba4fb456006237d530ab4baa1 registry.k8s.io/e2e-test-images/nginx:1.14-4],SizeBytes:6860089,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nonewprivs@sha256:8ac1264691820febacf3aea5d152cbde6d10685731ec14966a9401c6f47a68ac registry.k8s.io/e2e-test-images/nonewprivs:1.3],SizeBytes:5025027,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:2e0f836850e09b8b7cc937681d6194537a09fbd5f6b9e08f4d646a85128e8937 registry.k8s.io/e2e-test-images/busybox:1.29-4],SizeBytes:772933,},ContainerImage{Names:[registry.k8s.io/pause@sha256:7031c1b283388d2c2e09b57badb803c05ebed362dc88d84b480cc47f72a21097 registry.k8s.io/pause:3.9],SizeBytes:268051,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:253553,},},VolumesInUse:[kubernetes.io/csi/csi-hostpath-ephemeral-2147^52df5953-bb73-11ed-8aa7-be49170871b4 kubernetes.io/csi/csi-hostpath-ephemeral-5408^443d1daf-bb73-11ed-95c4-5a95d17cbc27 kubernetes.io/csi/csi-hostpath-ephemeral-5408^44411f51-bb73-11ed-95c4-5a95d17cbc27 kubernetes.io/csi/csi-hostpath-ephemeral-6540^4426a07f-bb73-11ed-a604-92687f2c50f8 kubernetes.io/csi/ebs.csi.aws.com^vol-05b67fab5beb6394b],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/csi-hostpath-ephemeral-6540^4426a07f-bb73-11ed-a604-92687f2c50f8,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/csi-hostpath-ephemeral-5408^44411f51-bb73-11ed-95c4-5a95d17cbc27,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/csi-hostpath-ephemeral-5408^443d1daf-bb73-11ed-95c4-5a95d17cbc27,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/csi-hostpath-ephemeral-2147^52df5953-bb73-11ed-8aa7-be49170871b4,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-05b67fab5beb6394b,DevicePath:,},},Config:nil,},} Mar 5 16:43:38.597: INFO: Logging kubelet events for node i-06df568f6e5c59fff Mar 5 16:43:38.712: INFO: Logging pods the kubelet thinks is on node i-06df568f6e5c59fff Mar 5 16:43:48.825: INFO: Unable to retrieve kubelet pods for node i-06df568f6e5c59fff: error trying to reach service: net/http: TLS handshake timeout Mar 5 16:43:48.825: INFO: Logging node info for node i-0d6a32f749839192f Mar 5 16:43:48.936: INFO: Node Info: &Node{ObjectMeta:{i-0d6a32f749839192f 4155fefb-4d3a-4b04-ba53-6ac9e895ff92 12940 0 2023-03-05 16:20:46 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:arm64 beta.kubernetes.io/instance-type:c6g.xlarge beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:eu-central-1 failure-domain.beta.kubernetes.io/zone:eu-central-1a kops.k8s.io/kops-controller-pki: kubernetes.io/arch:arm64 kubernetes.io/hostname:i-0d6a32f749839192f kubernetes.io/os:linux node-role.kubernetes.io/control-plane: node.kubernetes.io/exclude-from-external-load-balancers: node.kubernetes.io/instance-type:c6g.xlarge topology.ebs.csi.aws.com/zone:eu-central-1a topology.kubernetes.io/region:eu-central-1 topology.kubernetes.io/zone:eu-central-1a] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-0d6a32f749839192f"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kubelet Update v1 2023-03-05 16:20:46 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {protokube Update v1 2023-03-05 16:21:25 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kops.k8s.io/kops-controller-pki":{},"f:node-role.kubernetes.io/control-plane":{},"f:node.kubernetes.io/exclude-from-external-load-balancers":{}}}} } {kube-controller-manager Update v1 2023-03-05 16:21:47 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.0.0/24\"":{}}}} } {aws-cloud-controller-manager Update v1 2023-03-05 16:21:48 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{},"f:taints":{}}} } {aws-cloud-controller-manager Update v1 2023-03-05 16:21:48 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kubelet Update v1 2023-03-05 16:39:06 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.0.0/24,DoNotUseExternalID:,ProviderID:aws:///eu-central-1a/i-0d6a32f749839192f,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/control-plane,Value:,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[100.96.0.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{4 0} {<nil>} 4 DecimalSI},ephemeral-storage: {{49756430336 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-32Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-64Ki: {{0 0} {<nil>} 0 DecimalSI},memory: {{8150118400 0} {<nil>} 7959100Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{4 0} {<nil>} 4 DecimalSI},ephemeral-storage: {{44780787229 0} {<nil>} 44780787229 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-32Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-64Ki: {{0 0} {<nil>} 0 DecimalSI},memory: {{8045260800 0} {<nil>} 7856700Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-03-05 16:39:06 +0000 UTC,LastTransitionTime:2023-03-05 16:20:46 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-03-05 16:39:06 +0000 UTC,LastTransitionTime:2023-03-05 16:20:46 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-03-05 16:39:06 +0000 UTC,LastTransitionTime:2023-03-05 16:20:46 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-03-05 16:39:06 +0000 UTC,LastTransitionTime:2023-03-05 16:21:47 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.49.247,},NodeAddress{Type:ExternalIP,Address:18.197.20.82,},NodeAddress{Type:InternalDNS,Address:i-0d6a32f749839192f.eu-central-1.compute.internal,},NodeAddress{Type:Hostname,Address:i-0d6a32f749839192f.eu-central-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-18-197-20-82.eu-central-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2bed249e8ac7b517cb08ccd42b83e5,SystemUUID:ec2bed24-9e8a-c7b5-17cb-08ccd42b83e5,BootID:6f9215c9-dfe8-4b13-84f4-9b2084456198,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 22.04.2 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.27.0-alpha.3.52+d48b8167f7094b,KubeProxyVersion:v1.27.0-alpha.3.52+d48b8167f7094b,OperatingSystem:linux,Architecture:arm64,},Images:[]ContainerImage{ContainerImage{Names:[registry.k8s.io/etcd@sha256:9ce33ba33d8e738a5b85ed50b5080ac746deceed4a7496c550927a7a19ca3b6d registry.k8s.io/etcd:3.5.0-0],SizeBytes:157797353,},ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:157636062,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:4afb99b4690b418ffc2ceb67e1a17376457e441c1f09ab55447f0aaf992fa646 registry.k8s.io/etcd:3.4.3-0],SizeBytes:151677080,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:4ad90a11b55313b182afc186b9876c8e891531b8db4c9bf1541953021618d0e2 registry.k8s.io/etcd:3.4.13-0],SizeBytes:134531559,},ContainerImage{Names:[registry.k8s.io/kube-apiserver-arm64:v1.27.0-alpha.3.52_d48b8167f7094b],SizeBytes:131087707,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:0d2ab6b15b4fe0b903b9e81192d942fb39c29be61fe341cd16db5d0165b0435c registry.k8s.io/etcd:3.3.17-0],SizeBytes:126979107,},ContainerImage{Names:[registry.k8s.io/kube-controller-manager-arm64:v1.27.0-alpha.3.52_d48b8167f7094b],SizeBytes:120667128,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:17da501f5d2a675be46040422a27b7cc21b8a43895ac998b171db1c346f361f7 registry.k8s.io/etcd:3.3.10-0],SizeBytes:81436097,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:6f72b851544986cb0921b53ea655ec04c36131248f16d4ad110cb3ca0c369dc1 registry.k8s.io/etcd:3.5.4-0],SizeBytes:81120118,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:13f53ed1d91e2e11aac476ee9a0269fdda6cc4874eba903efd40daf50c55eee5 registry.k8s.io/etcd:3.5.3-0],SizeBytes:81114891,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:51eae8381dcb1078289fa7b4f3df2630cdc18d09fb56f8e56b41c40e191d6c83 registry.k8s.io/etcd:3.5.7-0],SizeBytes:80665728,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:dd75ec974b0a2a6f6bb47001ba09207976e625db898d1b16735528c009cb171c registry.k8s.io/etcd:3.5.6-0],SizeBytes:80539316,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:905d7ca17fd02bc24c0eba9a062753aba15db3e31422390bc3238eb762339b20 registry.k8s.io/etcd:3.2.24-1],SizeBytes:61500433,},ContainerImage{Names:[registry.k8s.io/etcdadm/etcd-manager-slim@sha256:8727adc059334e0e3329dbccce116a734da4a090da5e86c8cddccaf333caa666 registry.k8s.io/etcdadm/etcd-manager-slim:v3.0.20230201],SizeBytes:60826655,},ContainerImage{Names:[registry.k8s.io/kube-proxy-arm64:v1.27.0-alpha.3.52_d48b8167f7094b],SizeBytes:60554082,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:64b9ea357325d5db9f8a723dcf503b5a449177b17ac87d69481e126bb724c263 registry.k8s.io/etcd:3.5.1-0],SizeBytes:59553904,},ContainerImage{Names:[registry.k8s.io/kube-scheduler-arm64:v1.27.0-alpha.3.52_d48b8167f7094b],SizeBytes:56310763,},ContainerImage{Names:[registry.k8s.io/kops/kops-controller:1.27.0-alpha.1],SizeBytes:41797411,},ContainerImage{Names:[registry.k8s.io/kops/dns-controller:1.27.0-alpha.1],SizeBytes:40267473,},ContainerImage{Names:[public.ecr.aws/karpenter/controller@sha256:68db4f092cf9cc83f5ef9e2fbc5407c2cb682e81f64dfaa700a7602ede38b1cf public.ecr.aws/karpenter/controller:v0.16.3],SizeBytes:33707751,},ContainerImage{Names:[public.ecr.aws/karpenter/webhook@sha256:96a2d9b06d6bc5127801f358f74b1cf2d289b423a2e9ba40c573c0b14b17dafa public.ecr.aws/karpenter/webhook:v0.16.3],SizeBytes:33307187,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:28932904,},ContainerImage{Names:[quay.io/cilium/operator@sha256:a6d24a006a6b92967ac90786b49bc1ac26e5477cf028cd1186efcfc2466484db quay.io/cilium/operator:v1.12.5],SizeBytes:24567287,},ContainerImage{Names:[registry.k8s.io/provider-aws/cloud-controller-manager@sha256:fdeb61e3e42ecd9cca868d550ebdb88dd6341d9e91fcfa9a37e227dab2ad22cb registry.k8s.io/provider-aws/cloud-controller-manager:v1.26.0],SizeBytes:18345226,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:8502345,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:7685305,},ContainerImage{Names:[registry.k8s.io/kops/kube-apiserver-healthcheck:1.27.0-alpha.1],SizeBytes:4713612,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:253553,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Mar 5 16:43:48.936: INFO: Logging kubelet events for node i-0d6a32f749839192f Mar 5 16:43:49.051: INFO: ... skipping 50 lines ... Mar 5 16:43:49.170: INFO: Container webhook ready: true, restart count 0 Mar 5 16:43:49.170: INFO: kube-scheduler-i-0d6a32f749839192f started at 2023-03-05 16:18:47 +0000 UTC (0+1 container statuses recorded) Mar 5 16:43:49.170: INFO: Container kube-scheduler ready: true, restart count 0 Mar 5 16:43:49.661: INFO: Latency metrics for node i-0d6a32f749839192f [1mSTEP:[0m Waiting for namespaces [csi-mock-volumes-fsgroup-policy-1058-5723] to vanish [38;5;243m@ 03/05/23 16:43:49.775[0m Mar 5 16:48:50.001: INFO: error deleting namespace csi-mock-volumes-fsgroup-policy-1058-5723: timed out waiting for the condition [38;5;9m[FAILED][0m in [DeferCleanup (Each)] - test/e2e/framework/debug/dump.go:139 [38;5;243m@ 03/05/23 16:48:50.001[0m [1mSTEP:[0m dump namespace information after failure [38;5;243m@ 03/05/23 16:48:53.371[0m [1mSTEP:[0m Collecting events from namespace "csi-mock-volumes-fsgroup-policy-1058-5723". [38;5;243m@ 03/05/23 16:48:53.371[0m [1mSTEP:[0m Found 0 events. [38;5;243m@ 03/05/23 16:48:53.482[0m Mar 5 16:48:53.593: INFO: POD NODE PHASE GRACE CONDITIONS Mar 5 16:48:53.593: INFO: csi-mockplugin-0 i-06df568f6e5c59fff Pending 30s [{DisruptionTarget True 0001-01-01 00:00:00 +0000 UTC 2023-03-05 16:47:57 +0000 UTC DeletionByTaintManager Taint manager: deleting due to NoExecute taint} {Initialized True 0001-01-01 00:00:00 +0000 UTC 2023-03-05 16:32:33 +0000 UTC } {Ready False 0001-01-01 00:00:00 +0000 UTC 2023-03-05 16:32:33 +0000 UTC ContainersNotReady containers with unready status: [csi-provisioner driver-registrar mock]} {ContainersReady False 0001-01-01 00:00:00 +0000 UTC 2023-03-05 16:32:33 +0000 UTC ContainersNotReady containers with unready status: [csi-provisioner driver-registrar mock]} {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2023-03-05 16:32:33 +0000 UTC }] Mar 5 16:48:53.594: INFO: Mar 5 16:49:00.627: INFO: Unable to fetch csi-mock-volumes-fsgroup-policy-1058-5723/csi-mockplugin-0/csi-provisioner logs: an error on the server ("unknown") has prevented the request from succeeding (get pods csi-mockplugin-0) Mar 5 16:49:10.742: INFO: Unable to fetch csi-mock-volumes-fsgroup-policy-1058-5723/csi-mockplugin-0/driver-registrar logs: an error on the server ("unknown") has prevented the request from succeeding (get pods csi-mockplugin-0) Mar 5 16:49:20.855: INFO: Unable to fetch csi-mock-volumes-fsgroup-policy-1058-5723/csi-mockplugin-0/mock logs: an error on the server ("unknown") has prevented the request from succeeding (get pods csi-mockplugin-0) Mar 5 16:49:20.967: INFO: Logging node info for node i-056ad61525499081e Mar 5 16:49:21.080: INFO: Node Info: &Node{ObjectMeta:{i-056ad61525499081e a6ce9ed3-5ae8-4065-b117-6884e24b3632 18352 0 2023-03-05 16:24:16 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:arm64 beta.kubernetes.io/instance-type:t4g.small beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:eu-central-1 failure-domain.beta.kubernetes.io/zone:eu-central-1a karpenter.k8s.aws/instance-category:t karpenter.k8s.aws/instance-cpu:2 karpenter.k8s.aws/instance-family:t4g karpenter.k8s.aws/instance-generation:4 karpenter.k8s.aws/instance-hypervisor:nitro karpenter.k8s.aws/instance-memory:2048 karpenter.k8s.aws/instance-pods:11 karpenter.k8s.aws/instance-size:small karpenter.sh/capacity-type:spot karpenter.sh/initialized:true karpenter.sh/provisioner-name:nodes kubernetes.io/arch:arm64 kubernetes.io/hostname:i-056ad61525499081e kubernetes.io/os:linux node-role.kubernetes.io/node: node-role.kubernetes.io/spot-worker:true node.kubernetes.io/instance-type:t4g.small topology.ebs.csi.aws.com/zone:eu-central-1a topology.hostpath.csi/node:i-056ad61525499081e topology.kubernetes.io/region:eu-central-1 topology.kubernetes.io/zone:eu-central-1a] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-056ad61525499081e"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [{karpenter.sh/v1alpha5 Provisioner nodes ac5d7599-b2a2-4eae-8da4-1f4c0a54ec82 <nil> 0xc00470c1c0}] [karpenter.sh/termination] [{aws-cloud-controller-manager Update v1 2023-03-05 16:24:16 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{}}}} } {kops-controller Update v1 2023-03-05 16:24:16 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{},"f:node-role.kubernetes.io/spot-worker":{}}}} } {karpenter Update v1 2023-03-05 16:25:57 +0000 UTC FieldsV1 {"f:metadata":{"f:finalizers":{".":{},"v:\"karpenter.sh/termination\"":{}},"f:labels":{".":{},"f:karpenter.k8s.aws/instance-category":{},"f:karpenter.k8s.aws/instance-cpu":{},"f:karpenter.k8s.aws/instance-family":{},"f:karpenter.k8s.aws/instance-generation":{},"f:karpenter.k8s.aws/instance-hypervisor":{},"f:karpenter.k8s.aws/instance-memory":{},"f:karpenter.k8s.aws/instance-pods":{},"f:karpenter.k8s.aws/instance-size":{},"f:karpenter.sh/capacity-type":{},"f:karpenter.sh/initialized":{},"f:karpenter.sh/provisioner-name":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}},"f:ownerReferences":{".":{},"k:{\"uid\":\"ac5d7599-b2a2-4eae-8da4-1f4c0a54ec82\"}":{}}},"f:spec":{"f:providerID":{}}} } {kube-controller-manager Update v1 2023-03-05 16:25:57 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:node.alpha.kubernetes.io/ttl":{}},"f:labels":{"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.1.0/24\"":{}}}} } {aws-cloud-controller-manager Update v1 2023-03-05 16:26:48 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{".":{},"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"InternalIP\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kube-controller-manager Update v1 2023-03-05 16:48:56 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{".":{},"k:{\"type\":\"DiskPressure\"}":{".":{},"f:type":{}},"k:{\"type\":\"MemoryPressure\"}":{".":{},"f:type":{}},"k:{\"type\":\"PIDPressure\"}":{".":{},"f:type":{}},"k:{\"type\":\"Ready\"}":{".":{},"f:type":{}}},"f:volumesAttached":{}}} status} {kubelet Update v1 2023-03-05 16:48:56 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{"f:kubernetes.io/hostname":{},"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{".":{},"f:cpu":{},"f:ephemeral-storage":{},"f:hugepages-1Gi":{},"f:hugepages-2Mi":{},"f:hugepages-32Mi":{},"f:hugepages-64Ki":{},"f:memory":{},"f:pods":{}},"f:capacity":{".":{},"f:cpu":{},"f:ephemeral-storage":{},"f:hugepages-1Gi":{},"f:hugepages-2Mi":{},"f:hugepages-32Mi":{},"f:hugepages-64Ki":{},"f:memory":{},"f:pods":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:daemonEndpoints":{"f:kubeletEndpoint":{"f:Port":{}}},"f:images":{},"f:nodeInfo":{"f:architecture":{},"f:bootID":{},"f:containerRuntimeVersion":{},"f:kernelVersion":{},"f:kubeProxyVersion":{},"f:kubeletVersion":{},"f:machineID":{},"f:operatingSystem":{},"f:osImage":{},"f:systemUUID":{}},"f:volumesInUse":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.1.0/24,DoNotUseExternalID:,ProviderID:aws:///eu-central-1a/i-056ad61525499081e,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.1.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49756430336 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-32Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-64Ki: {{0 0} {<nil>} 0 DecimalSI},memory: {{1941540864 0} {<nil>} 1896036Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44780787229 0} {<nil>} 44780787229 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-32Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-64Ki: {{0 0} {<nil>} 0 DecimalSI},memory: {{1836683264 0} {<nil>} 1793636Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-03-05 16:48:56 +0000 UTC,LastTransitionTime:2023-03-05 16:25:57 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-03-05 16:48:56 +0000 UTC,LastTransitionTime:2023-03-05 16:25:48 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-03-05 16:48:56 +0000 UTC,LastTransitionTime:2023-03-05 16:25:48 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-03-05 16:48:56 +0000 UTC,LastTransitionTime:2023-03-05 16:25:48 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.36.179,},NodeAddress{Type:ExternalIP,Address:18.159.37.200,},NodeAddress{Type:InternalDNS,Address:i-056ad61525499081e.eu-central-1.compute.internal,},NodeAddress{Type:Hostname,Address:i-056ad61525499081e.eu-central-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-18-159-37-200.eu-central-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2af3207d1c8c09785c337aae6c6ef0,SystemUUID:ec2af320-7d1c-8c09-785c-337aae6c6ef0,BootID:786f2752-bc30-4729-8aaf-45977b16c253,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 22.04.2 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.27.0-alpha.3.52+d48b8167f7094b,KubeProxyVersion:v1.27.0-alpha.3.52+d48b8167f7094b,OperatingSystem:linux,Architecture:arm64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:157636062,},ContainerImage{Names:[registry.k8s.io/kube-proxy-arm64:v1.27.0-alpha.3.52_d48b8167f7094b],SizeBytes:60554082,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:16bbf38c463a4223d8cfe4da12bc61010b082a79b4bb003e2d3ba3ece5dd5f9e registry.k8s.io/e2e-test-images/agnhost:2.43],SizeBytes:49808364,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:3fe7acf013d1264ffded116b80a73dc129a449b0fccdb8d21af8279f2233f36e registry.k8s.io/e2e-test-images/httpd:2.4.39-4],SizeBytes:41528818,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:148b022f5c5da426fc2f3c14b5c0867e58ef05961510c84749ac1fddcb0fef22 registry.k8s.io/e2e-test-images/httpd:2.4.38-4],SizeBytes:40343601,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:28932904,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:ee3b525d5b89db99da3b8eb521d9cd90cb6e9ef0fbb651e98bb37be78d36b5b8 registry.k8s.io/sig-storage/csi-provisioner:v3.3.0],SizeBytes:23945474,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:425d8f1b769398127767b06ed97ce62578a3179bcb99809ce93a1649e025ffe7 registry.k8s.io/sig-storage/csi-resizer:v1.6.0],SizeBytes:22678632,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:291334908ddf71a4661fd7f6d9d97274de8a5378a2b6fdfeb2ce73414a34f82f registry.k8s.io/sig-storage/csi-snapshotter:v6.1.0],SizeBytes:22434180,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:9a685020911e2725ad019dbce6e4a5ab93d51e3d4557f115e64343345e05781b registry.k8s.io/sig-storage/csi-attacher:v4.0.0],SizeBytes:22394501,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:122bfb8c1edabb3c0edd63f06523e6940d958d19b3957dc7b1d6f81e9f1f6119 registry.k8s.io/sig-storage/csi-provisioner:v3.1.0],SizeBytes:21735644,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:9ebbf9f023e7b41ccee3d52afe39a89e3ddacdbb69269d583abfc25847cfd9e4 registry.k8s.io/sig-storage/csi-resizer:v1.4.0],SizeBytes:20848413,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:8b9c313c05f54fb04f8d430896f5f5904b6cb157df261501b29adc04d2b2dc7b registry.k8s.io/sig-storage/csi-attacher:v3.4.0],SizeBytes:20565480,},ContainerImage{Names:[registry.k8s.io/cpa/cluster-proportional-autoscaler@sha256:68d396900aeaa072c1f27289485fdac29834045a6f3ffe369bf389d830ef572d registry.k8s.io/cpa/cluster-proportional-autoscaler:1.8.6],SizeBytes:19007623,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:92257881c1d6493cf18299a24af42330f891166560047902b8d431fb66b01af5 registry.k8s.io/sig-storage/hostpathplugin:v1.9.0],SizeBytes:18267611,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:9b2d9abda017c28b12f84a344f57cd73fbdb6c2bd7dd5bdd5018246ad1093ba6 registry.k8s.io/sig-storage/hostpathplugin:v1.11.0],SizeBytes:18159085,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:017727efcfeb7d053af68e51436ce8e65edbc6ca573720afb4f79c8594036955 registry.k8s.io/coredns/coredns:v1.10.0],SizeBytes:13766234,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:8502345,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:933940f13b3ea0abc62e656c1aa5c5b47c04b15d71250413a6b821bd0c58b94e registry.k8s.io/sig-storage/livenessprobe:v2.7.0],SizeBytes:8223819,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:7685305,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:5c99cf6a02adda929b10321dbf4ecfa00d87be9ba4fb456006237d530ab4baa1 registry.k8s.io/e2e-test-images/nginx:1.14-4],SizeBytes:6860089,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:2e0f836850e09b8b7cc937681d6194537a09fbd5f6b9e08f4d646a85128e8937 registry.k8s.io/e2e-test-images/busybox:1.29-4],SizeBytes:772933,},ContainerImage{Names:[registry.k8s.io/pause@sha256:7031c1b283388d2c2e09b57badb803c05ebed362dc88d84b480cc47f72a21097 registry.k8s.io/pause:3.9],SizeBytes:268051,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:253553,},},VolumesInUse:[kubernetes.io/csi/ebs.csi.aws.com^vol-0e020446b1cd6abaf],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-0e020446b1cd6abaf,DevicePath:,},},Config:nil,},} Mar 5 16:49:21.081: INFO: Logging kubelet events for node i-056ad61525499081e Mar 5 16:49:21.196: INFO: ... skipping 15 lines ... Mar 5 16:49:21.317: INFO: cleanup20-16a6f148-bb85-4b73-99f5-2adb1cfdeea0-w749b started at 2023-03-05 16:48:01 +0000 UTC (0+1 container statuses recorded) Mar 5 16:49:21.317: INFO: Container cleanup20-16a6f148-bb85-4b73-99f5-2adb1cfdeea0 ready: true, restart count 0 Mar 5 16:49:21.317: INFO: cleanup20-16a6f148-bb85-4b73-99f5-2adb1cfdeea0-j6cks started at 2023-03-05 16:48:01 +0000 UTC (0+1 container statuses recorded) Mar 5 16:49:21.317: INFO: Container cleanup20-16a6f148-bb85-4b73-99f5-2adb1cfdeea0 ready: true, restart count 0 Mar 5 16:49:21.317: INFO: cleanup20-16a6f148-bb85-4b73-99f5-2adb1cfdeea0-khffj started at 2023-03-05 16:40:29 +0000 UTC (0+1 container statuses recorded) Mar 5 16:49:21.317: INFO: Container cleanup20-16a6f148-bb85-4b73-99f5-2adb1cfdeea0 ready: true, restart count 0 Mar 5 16:49:21.317: INFO: fail-once-local-2mt4w started at 2023-03-05 16:46:28 +0000 UTC (0+1 container statuses recorded) Mar 5 16:49:21.317: INFO: Container c ready: false, restart count 1 Mar 5 16:49:21.317: INFO: cleanup20-16a6f148-bb85-4b73-99f5-2adb1cfdeea0-gnfq6 started at 2023-03-05 16:48:01 +0000 UTC (0+1 container statuses recorded) Mar 5 16:49:21.317: INFO: Container cleanup20-16a6f148-bb85-4b73-99f5-2adb1cfdeea0 ready: true, restart count 0 Mar 5 16:49:21.317: INFO: cleanup20-16a6f148-bb85-4b73-99f5-2adb1cfdeea0-xkdzr started at 2023-03-05 16:40:29 +0000 UTC (0+1 container statuses recorded) Mar 5 16:49:21.317: INFO: Container cleanup20-16a6f148-bb85-4b73-99f5-2adb1cfdeea0 ready: true, restart count 0 Mar 5 16:49:21.317: INFO: ss-0 started at 2023-03-05 16:33:11 +0000 UTC (0+1 container statuses recorded) ... skipping 5 lines ... Mar 5 16:49:21.317: INFO: coredns-745bddcfbb-gm4h2 started at 2023-03-05 16:25:57 +0000 UTC (0+1 container statuses recorded) Mar 5 16:49:21.317: INFO: Container coredns ready: true, restart count 0 Mar 5 16:49:21.317: INFO: cleanup20-16a6f148-bb85-4b73-99f5-2adb1cfdeea0-sqrjl started at 2023-03-05 16:40:29 +0000 UTC (0+1 container statuses recorded) Mar 5 16:49:21.317: INFO: Container cleanup20-16a6f148-bb85-4b73-99f5-2adb1cfdeea0 ready: true, restart count 0 Mar 5 16:49:21.317: INFO: cleanup20-16a6f148-bb85-4b73-99f5-2adb1cfdeea0-n9vbn started at 2023-03-05 16:40:29 +0000 UTC (0+1 container statuses recorded) Mar 5 16:49:21.317: INFO: Container cleanup20-16a6f148-bb85-4b73-99f5-2adb1cfdeea0 ready: true, restart count 0 Mar 5 16:49:21.317: INFO: fail-once-local-gj678 started at 2023-03-05 16:46:33 +0000 UTC (0+1 container statuses recorded) Mar 5 16:49:21.317: INFO: Container c ready: false, restart count 1 Mar 5 16:49:21.317: INFO: cleanup20-16a6f148-bb85-4b73-99f5-2adb1cfdeea0-m6dgl started at 2023-03-05 16:48:01 +0000 UTC (0+1 container statuses recorded) Mar 5 16:49:21.317: INFO: Container cleanup20-16a6f148-bb85-4b73-99f5-2adb1cfdeea0 ready: true, restart count 0 Mar 5 16:49:21.317: INFO: cleanup20-16a6f148-bb85-4b73-99f5-2adb1cfdeea0-cqffq started at 2023-03-05 16:48:02 +0000 UTC (0+1 container statuses recorded) Mar 5 16:49:21.317: INFO: Container cleanup20-16a6f148-bb85-4b73-99f5-2adb1cfdeea0 ready: true, restart count 0 Mar 5 16:49:21.317: INFO: ebs-csi-node-n98cq started at 2023-03-05 16:25:49 +0000 UTC (0+3 container statuses recorded) ... skipping 2 lines ... Mar 5 16:49:21.317: INFO: Container node-driver-registrar ready: true, restart count 0 Mar 5 16:49:21.317: INFO: cilium-vfgw4 started at 2023-03-05 16:25:49 +0000 UTC (1+1 container statuses recorded) Mar 5 16:49:21.317: INFO: Init container clean-cilium-state ready: true, restart count 0 Mar 5 16:49:21.317: INFO: Container cilium-agent ready: true, restart count 0 Mar 5 16:49:21.317: INFO: sample-webhook-deployment-7497495989-s9585 started at 2023-03-05 16:46:38 +0000 UTC (0+1 container statuses recorded) Mar 5 16:49:21.317: INFO: Container sample-webhook ready: true, restart count 0 Mar 5 16:49:21.317: INFO: fail-once-local-kdbgv started at 2023-03-05 16:46:33 +0000 UTC (0+1 container statuses recorded) Mar 5 16:49:21.317: INFO: Container c ready: false, restart count 1 Mar 5 16:49:21.317: INFO: cleanup20-16a6f148-bb85-4b73-99f5-2adb1cfdeea0-xsqgf started at 2023-03-05 16:40:29 +0000 UTC (0+1 container statuses recorded) Mar 5 16:49:21.317: INFO: Container cleanup20-16a6f148-bb85-4b73-99f5-2adb1cfdeea0 ready: true, restart count 0 Mar 5 16:49:21.317: INFO: cleanup20-16a6f148-bb85-4b73-99f5-2adb1cfdeea0-6lnc6 started at 2023-03-05 16:48:01 +0000 UTC (0+1 container statuses recorded) Mar 5 16:49:21.317: INFO: Container cleanup20-16a6f148-bb85-4b73-99f5-2adb1cfdeea0 ready: true, restart count 0 Mar 5 16:49:21.317: INFO: cleanup20-16a6f148-bb85-4b73-99f5-2adb1cfdeea0-9qlxz started at 2023-03-05 16:48:01 +0000 UTC (0+1 container statuses recorded) ... skipping 3 lines ... Mar 5 16:49:21.317: INFO: coredns-autoscaler-8495f5575-plvdw started at 2023-03-05 16:25:57 +0000 UTC (0+1 container statuses recorded) Mar 5 16:49:21.317: INFO: Container autoscaler ready: true, restart count 0 Mar 5 16:49:21.317: INFO: cleanup20-16a6f148-bb85-4b73-99f5-2adb1cfdeea0-kgbnw started at 2023-03-05 16:40:29 +0000 UTC (0+1 container statuses recorded) Mar 5 16:49:21.317: INFO: Container cleanup20-16a6f148-bb85-4b73-99f5-2adb1cfdeea0 ready: true, restart count 0 Mar 5 16:49:21.317: INFO: cleanup20-16a6f148-bb85-4b73-99f5-2adb1cfdeea0-42w6f started at 2023-03-05 16:40:29 +0000 UTC (0+1 container statuses recorded) Mar 5 16:49:21.317: INFO: Container cleanup20-16a6f148-bb85-4b73-99f5-2adb1cfdeea0 ready: true, restart count 0 Mar 5 16:49:21.317: INFO: fail-once-local-69mfr started at 2023-03-05 16:46:28 +0000 UTC (0+1 container statuses recorded) Mar 5 16:49:21.317: INFO: Container c ready: false, restart count 1 Mar 5 16:49:21.317: INFO: cleanup20-16a6f148-bb85-4b73-99f5-2adb1cfdeea0-wmjdk started at 2023-03-05 16:40:29 +0000 UTC (0+1 container statuses recorded) Mar 5 16:49:21.317: INFO: Container cleanup20-16a6f148-bb85-4b73-99f5-2adb1cfdeea0 ready: true, restart count 0 Mar 5 16:49:21.317: INFO: pod-subpath-test-downwardapi-w7jc started at 2023-03-05 16:49:14 +0000 UTC (0+1 container statuses recorded) Mar 5 16:49:21.317: INFO: Container test-container-subpath-downwardapi-w7jc ready: true, restart count 0 Mar 5 16:49:21.778: INFO: ... skipping 2 lines ... Logging node info for node i-06df568f6e5c59fff Mar 5 16:49:21.894: INFO: Node Info: &Node{ObjectMeta:{i-06df568f6e5c59fff cb63c681-2e18-474a-b6f3-221325ac0d52 18301 0 2023-03-05 16:24:16 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:arm64 beta.kubernetes.io/instance-type:t4g.small beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:eu-central-1 failure-domain.beta.kubernetes.io/zone:eu-central-1a karpenter.k8s.aws/instance-category:t karpenter.k8s.aws/instance-cpu:2 karpenter.k8s.aws/instance-family:t4g karpenter.k8s.aws/instance-generation:4 karpenter.k8s.aws/instance-hypervisor:nitro karpenter.k8s.aws/instance-memory:2048 karpenter.k8s.aws/instance-pods:11 karpenter.k8s.aws/instance-size:small karpenter.sh/capacity-type:spot karpenter.sh/initialized:true karpenter.sh/provisioner-name:nodes kubernetes.io/arch:arm64 kubernetes.io/hostname:i-06df568f6e5c59fff kubernetes.io/os:linux node-role.kubernetes.io/node: node-role.kubernetes.io/spot-worker:true node.kubernetes.io/instance-type:t4g.small topology.ebs.csi.aws.com/zone:eu-central-1a topology.hostpath.csi/node:i-06df568f6e5c59fff topology.kubernetes.io/region:eu-central-1 topology.kubernetes.io/zone:eu-central-1a] map[csi.volume.kubernetes.io/nodeid:{"csi-hostpath-ephemeral-2147":"i-06df568f6e5c59fff","csi-hostpath-ephemeral-5408":"i-06df568f6e5c59fff","csi-hostpath-ephemeral-6540":"i-06df568f6e5c59fff","csi-mock-csi-mock-volumes-fsgroup-policy-1058":"i-06df568f6e5c59fff","csi-mock-csi-mock-volumes-workload-2219":"i-06df568f6e5c59fff","ebs.csi.aws.com":"i-06df568f6e5c59fff"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [{karpenter.sh/v1alpha5 Provisioner nodes ac5d7599-b2a2-4eae-8da4-1f4c0a54ec82 <nil> 0xc0037fc0b0}] [karpenter.sh/termination] [{aws-cloud-controller-manager Update v1 2023-03-05 16:24:16 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{}}}} } {kops-controller Update v1 2023-03-05 16:24:16 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{},"f:node-role.kubernetes.io/spot-worker":{}}}} } {karpenter Update v1 2023-03-05 16:25:57 +0000 UTC FieldsV1 {"f:metadata":{"f:finalizers":{".":{},"v:\"karpenter.sh/termination\"":{}},"f:labels":{".":{},"f:karpenter.k8s.aws/instance-category":{},"f:karpenter.k8s.aws/instance-cpu":{},"f:karpenter.k8s.aws/instance-family":{},"f:karpenter.k8s.aws/instance-generation":{},"f:karpenter.k8s.aws/instance-hypervisor":{},"f:karpenter.k8s.aws/instance-memory":{},"f:karpenter.k8s.aws/instance-pods":{},"f:karpenter.k8s.aws/instance-size":{},"f:karpenter.sh/capacity-type":{},"f:karpenter.sh/initialized":{},"f:karpenter.sh/provisioner-name":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}},"f:ownerReferences":{".":{},"k:{\"uid\":\"ac5d7599-b2a2-4eae-8da4-1f4c0a54ec82\"}":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-03-05 16:26:48 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{".":{},"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"InternalIP\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kube-controller-manager Update v1 2023-03-05 16:44:29 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{".":{},"k:{\"type\":\"DiskPressure\"}":{".":{},"f:type":{}},"k:{\"type\":\"MemoryPressure\"}":{".":{},"f:type":{}},"k:{\"type\":\"PIDPressure\"}":{".":{},"f:type":{}},"k:{\"type\":\"Ready\"}":{".":{},"f:type":{}}},"f:volumesAttached":{}}} status} {kube-controller-manager Update v1 2023-03-05 16:46:37 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:node.alpha.kubernetes.io/ttl":{}},"f:labels":{"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.2.0/24\"":{}},"f:taints":{}}} } {kubelet Update v1 2023-03-05 16:48:40 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{"f:kubernetes.io/hostname":{},"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{".":{},"f:cpu":{},"f:ephemeral-storage":{},"f:hugepages-1Gi":{},"f:hugepages-2Mi":{},"f:hugepages-32Mi":{},"f:hugepages-64Ki":{},"f:memory":{},"f:pods":{}},"f:capacity":{".":{},"f:cpu":{},"f:ephemeral-storage":{},"f:hugepages-1Gi":{},"f:hugepages-2Mi":{},"f:hugepages-32Mi":{},"f:hugepages-64Ki":{},"f:memory":{},"f:pods":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:daemonEndpoints":{"f:kubeletEndpoint":{"f:Port":{}}},"f:images":{},"f:nodeInfo":{"f:architecture":{},"f:bootID":{},"f:containerRuntimeVersion":{},"f:kernelVersion":{},"f:kubeProxyVersion":{},"f:kubeletVersion":{},"f:machineID":{},"f:operatingSystem":{},"f:osImage":{},"f:systemUUID":{}},"f:volumesInUse":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.2.0/24,DoNotUseExternalID:,ProviderID:aws:///eu-central-1a/i-06df568f6e5c59fff,Unschedulable:false,Taints:[]Taint{Taint{Key:node.kubernetes.io/not-ready,Value:,Effect:NoSchedule,TimeAdded:2023-03-05 16:46:33 +0000 UTC,},Taint{Key:node.kubernetes.io/not-ready,Value:,Effect:NoExecute,TimeAdded:2023-03-05 16:46:37 +0000 UTC,},},ConfigSource:nil,PodCIDRs:[100.96.2.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49756430336 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-32Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-64Ki: {{0 0} {<nil>} 0 DecimalSI},memory: {{1941540864 0} {<nil>} 1896036Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44780787229 0} {<nil>} 44780787229 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-32Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-64Ki: {{0 0} {<nil>} 0 DecimalSI},memory: {{1836683264 0} {<nil>} 1793636Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:Ready,Status:False,LastHeartbeatTime:2023-03-05 16:47:34 +0000 UTC,LastTransitionTime:2023-03-05 16:46:08 +0000 UTC,Reason:KubeletNotReady,Message:PLEG is not healthy: pleg was last seen active 8m29.527212456s ago; threshold is 3m0s,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-03-05 16:47:34 +0000 UTC,LastTransitionTime:2023-03-05 16:47:12 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-03-05 16:47:34 +0000 UTC,LastTransitionTime:2023-03-05 16:47:15 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-03-05 16:47:34 +0000 UTC,LastTransitionTime:2023-03-05 16:47:15 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.43.148,},NodeAddress{Type:ExternalIP,Address:18.192.211.222,},NodeAddress{Type:InternalDNS,Address:i-06df568f6e5c59fff.eu-central-1.compute.internal,},NodeAddress{Type:Hostname,Address:i-06df568f6e5c59fff.eu-central-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-18-192-211-222.eu-central-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec221221864d9409a8ec5f6614a9b94b,SystemUUID:ec221221-864d-9409-a8ec-5f6614a9b94b,BootID:553d31bf-faca-417e-a3db-325839e1b8e4,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 22.04.2 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.27.0-alpha.3.52+d48b8167f7094b,KubeProxyVersion:v1.27.0-alpha.3.52+d48b8167f7094b,OperatingSystem:linux,Architecture:arm64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:157636062,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/jessie-dnsutils@sha256:24aaf2626d6b27864c29de2097e8bbb840b3a414271bf7c8995e431e47d8408e registry.k8s.io/e2e-test-images/jessie-dnsutils:1.7],SizeBytes:116764683,},ContainerImage{Names:[registry.k8s.io/kube-proxy-arm64:v1.27.0-alpha.3.52_d48b8167f7094b],SizeBytes:60554082,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:16bbf38c463a4223d8cfe4da12bc61010b082a79b4bb003e2d3ba3ece5dd5f9e registry.k8s.io/e2e-test-images/agnhost:2.43],SizeBytes:49808364,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:3fe7acf013d1264ffded116b80a73dc129a449b0fccdb8d21af8279f2233f36e registry.k8s.io/e2e-test-images/httpd:2.4.39-4],SizeBytes:41528818,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:148b022f5c5da426fc2f3c14b5c0867e58ef05961510c84749ac1fddcb0fef22 registry.k8s.io/e2e-test-images/httpd:2.4.38-4],SizeBytes:40343601,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:28932904,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:ee3b525d5b89db99da3b8eb521d9cd90cb6e9ef0fbb651e98bb37be78d36b5b8 registry.k8s.io/sig-storage/csi-provisioner:v3.3.0],SizeBytes:23945474,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:425d8f1b769398127767b06ed97ce62578a3179bcb99809ce93a1649e025ffe7 registry.k8s.io/sig-storage/csi-resizer:v1.6.0],SizeBytes:22678632,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:291334908ddf71a4661fd7f6d9d97274de8a5378a2b6fdfeb2ce73414a34f82f registry.k8s.io/sig-storage/csi-snapshotter:v6.1.0],SizeBytes:22434180,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:9a685020911e2725ad019dbce6e4a5ab93d51e3d4557f115e64343345e05781b registry.k8s.io/sig-storage/csi-attacher:v4.0.0],SizeBytes:22394501,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:122bfb8c1edabb3c0edd63f06523e6940d958d19b3957dc7b1d6f81e9f1f6119 registry.k8s.io/sig-storage/csi-provisioner:v3.1.0],SizeBytes:21735644,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:9ebbf9f023e7b41ccee3d52afe39a89e3ddacdbb69269d583abfc25847cfd9e4 registry.k8s.io/sig-storage/csi-resizer:v1.4.0],SizeBytes:20848413,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:8b9c313c05f54fb04f8d430896f5f5904b6cb157df261501b29adc04d2b2dc7b registry.k8s.io/sig-storage/csi-attacher:v3.4.0],SizeBytes:20565480,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nonroot@sha256:ee9f50b3c64b174d296d91ca9f69a914ac30e59095dfb462b2b518ad28a63655 registry.k8s.io/e2e-test-images/nonroot:1.4],SizeBytes:18561972,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:92257881c1d6493cf18299a24af42330f891166560047902b8d431fb66b01af5 registry.k8s.io/sig-storage/hostpathplugin:v1.9.0],SizeBytes:18267611,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:9b2d9abda017c28b12f84a344f57cd73fbdb6c2bd7dd5bdd5018246ad1093ba6 registry.k8s.io/sig-storage/hostpathplugin:v1.11.0],SizeBytes:18159085,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:017727efcfeb7d053af68e51436ce8e65edbc6ca573720afb4f79c8594036955 registry.k8s.io/coredns/coredns:v1.10.0],SizeBytes:13766234,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:8502345,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:933940f13b3ea0abc62e656c1aa5c5b47c04b15d71250413a6b821bd0c58b94e registry.k8s.io/sig-storage/livenessprobe:v2.7.0],SizeBytes:8223819,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:7685305,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:5c99cf6a02adda929b10321dbf4ecfa00d87be9ba4fb456006237d530ab4baa1 registry.k8s.io/e2e-test-images/nginx:1.14-4],SizeBytes:6860089,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nonewprivs@sha256:8ac1264691820febacf3aea5d152cbde6d10685731ec14966a9401c6f47a68ac registry.k8s.io/e2e-test-images/nonewprivs:1.3],SizeBytes:5025027,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:2e0f836850e09b8b7cc937681d6194537a09fbd5f6b9e08f4d646a85128e8937 registry.k8s.io/e2e-test-images/busybox:1.29-4],SizeBytes:772933,},ContainerImage{Names:[registry.k8s.io/pause@sha256:7031c1b283388d2c2e09b57badb803c05ebed362dc88d84b480cc47f72a21097 registry.k8s.io/pause:3.9],SizeBytes:268051,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:253553,},},VolumesInUse:[kubernetes.io/csi/csi-hostpath-ephemeral-2147^52df5953-bb73-11ed-8aa7-be49170871b4 kubernetes.io/csi/csi-hostpath-ephemeral-5408^443d1daf-bb73-11ed-95c4-5a95d17cbc27 kubernetes.io/csi/csi-hostpath-ephemeral-5408^44411f51-bb73-11ed-95c4-5a95d17cbc27 kubernetes.io/csi/csi-hostpath-ephemeral-6540^4426a07f-bb73-11ed-a604-92687f2c50f8 kubernetes.io/csi/ebs.csi.aws.com^vol-05b67fab5beb6394b],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-05b67fab5beb6394b,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/csi-hostpath-ephemeral-6540^4426a07f-bb73-11ed-a604-92687f2c50f8,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/csi-hostpath-ephemeral-5408^44411f51-bb73-11ed-95c4-5a95d17cbc27,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/csi-hostpath-ephemeral-5408^443d1daf-bb73-11ed-95c4-5a95d17cbc27,DevicePath:,},},Config:nil,},} Mar 5 16:49:21.895: INFO: Logging kubelet events for node i-06df568f6e5c59fff Mar 5 16:49:22.010: INFO: Logging pods the kubelet thinks is on node i-06df568f6e5c59fff Mar 5 16:49:32.122: INFO: Unable to retrieve kubelet pods for node i-06df568f6e5c59fff: error trying to reach service: net/http: TLS handshake timeout Mar 5 16:49:32.122: INFO: Logging node info for node i-0d6a32f749839192f Mar 5 16:49:32.234: INFO: Node Info: &Node{ObjectMeta:{i-0d6a32f749839192f 4155fefb-4d3a-4b04-ba53-6ac9e895ff92 18509 0 2023-03-05 16:20:46 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:arm64 beta.kubernetes.io/instance-type:c6g.xlarge beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:eu-central-1 failure-domain.beta.kubernetes.io/zone:eu-central-1a kops.k8s.io/kops-controller-pki: kubernetes.io/arch:arm64 kubernetes.io/hostname:i-0d6a32f749839192f kubernetes.io/os:linux node-role.kubernetes.io/control-plane: node.kubernetes.io/exclude-from-external-load-balancers: node.kubernetes.io/instance-type:c6g.xlarge topology.ebs.csi.aws.com/zone:eu-central-1a topology.kubernetes.io/region:eu-central-1 topology.kubernetes.io/zone:eu-central-1a] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-0d6a32f749839192f"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kubelet Update v1 2023-03-05 16:20:46 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {protokube Update v1 2023-03-05 16:21:25 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kops.k8s.io/kops-controller-pki":{},"f:node-role.kubernetes.io/control-plane":{},"f:node.kubernetes.io/exclude-from-external-load-balancers":{}}}} } {kube-controller-manager Update v1 2023-03-05 16:21:47 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.0.0/24\"":{}}}} } {aws-cloud-controller-manager Update v1 2023-03-05 16:21:48 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{},"f:taints":{}}} } {aws-cloud-controller-manager Update v1 2023-03-05 16:21:48 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kubelet Update v1 2023-03-05 16:49:18 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.0.0/24,DoNotUseExternalID:,ProviderID:aws:///eu-central-1a/i-0d6a32f749839192f,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/control-plane,Value:,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[100.96.0.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{4 0} {<nil>} 4 DecimalSI},ephemeral-storage: {{49756430336 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-32Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-64Ki: {{0 0} {<nil>} 0 DecimalSI},memory: {{8150118400 0} {<nil>} 7959100Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{4 0} {<nil>} 4 DecimalSI},ephemeral-storage: {{44780787229 0} {<nil>} 44780787229 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-32Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-64Ki: {{0 0} {<nil>} 0 DecimalSI},memory: {{8045260800 0} {<nil>} 7856700Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-03-05 16:49:18 +0000 UTC,LastTransitionTime:2023-03-05 16:20:46 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-03-05 16:49:18 +0000 UTC,LastTransitionTime:2023-03-05 16:20:46 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-03-05 16:49:18 +0000 UTC,LastTransitionTime:2023-03-05 16:20:46 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-03-05 16:49:18 +0000 UTC,LastTransitionTime:2023-03-05 16:21:47 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.49.247,},NodeAddress{Type:ExternalIP,Address:18.197.20.82,},NodeAddress{Type:InternalDNS,Address:i-0d6a32f749839192f.eu-central-1.compute.internal,},NodeAddress{Type:Hostname,Address:i-0d6a32f749839192f.eu-central-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-18-197-20-82.eu-central-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2bed249e8ac7b517cb08ccd42b83e5,SystemUUID:ec2bed24-9e8a-c7b5-17cb-08ccd42b83e5,BootID:6f9215c9-dfe8-4b13-84f4-9b2084456198,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 22.04.2 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.27.0-alpha.3.52+d48b8167f7094b,KubeProxyVersion:v1.27.0-alpha.3.52+d48b8167f7094b,OperatingSystem:linux,Architecture:arm64,},Images:[]ContainerImage{ContainerImage{Names:[registry.k8s.io/etcd@sha256:9ce33ba33d8e738a5b85ed50b5080ac746deceed4a7496c550927a7a19ca3b6d registry.k8s.io/etcd:3.5.0-0],SizeBytes:157797353,},ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:157636062,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:4afb99b4690b418ffc2ceb67e1a17376457e441c1f09ab55447f0aaf992fa646 registry.k8s.io/etcd:3.4.3-0],SizeBytes:151677080,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:4ad90a11b55313b182afc186b9876c8e891531b8db4c9bf1541953021618d0e2 registry.k8s.io/etcd:3.4.13-0],SizeBytes:134531559,},ContainerImage{Names:[registry.k8s.io/kube-apiserver-arm64:v1.27.0-alpha.3.52_d48b8167f7094b],SizeBytes:131087707,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:0d2ab6b15b4fe0b903b9e81192d942fb39c29be61fe341cd16db5d0165b0435c registry.k8s.io/etcd:3.3.17-0],SizeBytes:126979107,},ContainerImage{Names:[registry.k8s.io/kube-controller-manager-arm64:v1.27.0-alpha.3.52_d48b8167f7094b],SizeBytes:120667128,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:17da501f5d2a675be46040422a27b7cc21b8a43895ac998b171db1c346f361f7 registry.k8s.io/etcd:3.3.10-0],SizeBytes:81436097,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:6f72b851544986cb0921b53ea655ec04c36131248f16d4ad110cb3ca0c369dc1 registry.k8s.io/etcd:3.5.4-0],SizeBytes:81120118,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:13f53ed1d91e2e11aac476ee9a0269fdda6cc4874eba903efd40daf50c55eee5 registry.k8s.io/etcd:3.5.3-0],SizeBytes:81114891,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:51eae8381dcb1078289fa7b4f3df2630cdc18d09fb56f8e56b41c40e191d6c83 registry.k8s.io/etcd:3.5.7-0],SizeBytes:80665728,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:dd75ec974b0a2a6f6bb47001ba09207976e625db898d1b16735528c009cb171c registry.k8s.io/etcd:3.5.6-0],SizeBytes:80539316,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:905d7ca17fd02bc24c0eba9a062753aba15db3e31422390bc3238eb762339b20 registry.k8s.io/etcd:3.2.24-1],SizeBytes:61500433,},ContainerImage{Names:[registry.k8s.io/etcdadm/etcd-manager-slim@sha256:8727adc059334e0e3329dbccce116a734da4a090da5e86c8cddccaf333caa666 registry.k8s.io/etcdadm/etcd-manager-slim:v3.0.20230201],SizeBytes:60826655,},ContainerImage{Names:[registry.k8s.io/kube-proxy-arm64:v1.27.0-alpha.3.52_d48b8167f7094b],SizeBytes:60554082,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:64b9ea357325d5db9f8a723dcf503b5a449177b17ac87d69481e126bb724c263 registry.k8s.io/etcd:3.5.1-0],SizeBytes:59553904,},ContainerImage{Names:[registry.k8s.io/kube-scheduler-arm64:v1.27.0-alpha.3.52_d48b8167f7094b],SizeBytes:56310763,},ContainerImage{Names:[registry.k8s.io/kops/kops-controller:1.27.0-alpha.1],SizeBytes:41797411,},ContainerImage{Names:[registry.k8s.io/kops/dns-controller:1.27.0-alpha.1],SizeBytes:40267473,},ContainerImage{Names:[public.ecr.aws/karpenter/controller@sha256:68db4f092cf9cc83f5ef9e2fbc5407c2cb682e81f64dfaa700a7602ede38b1cf public.ecr.aws/karpenter/controller:v0.16.3],SizeBytes:33707751,},ContainerImage{Names:[public.ecr.aws/karpenter/webhook@sha256:96a2d9b06d6bc5127801f358f74b1cf2d289b423a2e9ba40c573c0b14b17dafa public.ecr.aws/karpenter/webhook:v0.16.3],SizeBytes:33307187,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:28932904,},ContainerImage{Names:[quay.io/cilium/operator@sha256:a6d24a006a6b92967ac90786b49bc1ac26e5477cf028cd1186efcfc2466484db quay.io/cilium/operator:v1.12.5],SizeBytes:24567287,},ContainerImage{Names:[registry.k8s.io/provider-aws/cloud-controller-manager@sha256:fdeb61e3e42ecd9cca868d550ebdb88dd6341d9e91fcfa9a37e227dab2ad22cb registry.k8s.io/provider-aws/cloud-controller-manager:v1.26.0],SizeBytes:18345226,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:8502345,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:7685305,},ContainerImage{Names:[registry.k8s.io/kops/kube-apiserver-healthcheck:1.27.0-alpha.1],SizeBytes:4713612,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:253553,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Mar 5 16:49:32.235: INFO: Logging kubelet events for node i-0d6a32f749839192f Mar 5 16:49:32.350: INFO: ... skipping 52 lines ... Mar 5 16:49:32.469: INFO: Container webhook ready: true, restart count 0 Mar 5 16:49:32.960: INFO: Latency metrics for node i-0d6a32f749839192f [1mSTEP:[0m Destroying namespace "csi-mock-volumes-fsgroup-policy-1058-5723" for this suite. [38;5;243m@ 03/05/23 16:49:32.96[0m [38;5;243m<< Timeline[0m [38;5;9m[FAILED] Failed to register CSIDriver csi-mock-csi-mock-volumes-fsgroup-policy-1058: error waiting for CSI driver csi-mock-csi-mock-volumes-fsgroup-policy-1058 registration on node i-06df568f6e5c59fff: timed out waiting for the condition[0m [38;5;9mIn [1m[It][0m[38;5;9m at: [1mtest/e2e/storage/csi_mock/base.go:206[0m [38;5;243m@ 03/05/23 16:38:17.547[0m There were [1m[38;5;9madditional failures[0m detected. To view them in detail run [1mginkgo -vv[0m [38;5;243m------------------------------[0m [38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m [38;5;243m------------------------------[0m [38;5;9m• [FAILED] [545.847 seconds][0m [0m[sig-node] kubelet [38;5;243mClean up pods on node [38;5;9m[1m[It] kubelet should be able to delete 10 pods per node in 1m0s.[0m [38;5;243mtest/e2e/node/kubelet.go:336[0m [38;5;243mTimeline >>[0m [1mSTEP:[0m Creating a kubernetes client [38;5;243m@ 03/05/23 16:40:27.61[0m Mar 5 16:40:27.610: INFO: >>> kubeConfig: /root/.kube/config ... skipping 75 lines ... Mar 5 16:42:49.282: INFO: Missing info/stats for container "runtime" on node "i-056ad61525499081e" Mar 5 16:42:53.227: INFO: Missing info/stats for container "runtime" on node "i-0d6a32f749839192f" Mar 5 16:42:54.458: INFO: Missing info/stats for container "runtime" on node "i-056ad61525499081e" Mar 5 16:42:58.372: INFO: Missing info/stats for container "runtime" on node "i-0d6a32f749839192f" I0305 16:42:59.268133 7203 runners.go:194] cleanup20-16a6f148-bb85-4b73-99f5-2adb1cfdeea0 Pods: 20 out of 20 created, 10 running, 10 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady Mar 5 16:42:59.634: INFO: Missing info/stats for container "runtime" on node "i-056ad61525499081e" Mar 5 16:43:00.752: INFO: Error getting node stats summary on "i-06df568f6e5c59fff", err: the server is currently unable to handle the request (get nodes i-06df568f6e5c59fff:10250) Mar 5 16:43:03.520: INFO: Missing info/stats for container "runtime" on node "i-0d6a32f749839192f" Mar 5 16:43:04.808: INFO: Missing info/stats for container "runtime" on node "i-056ad61525499081e" Mar 5 16:43:08.663: INFO: Missing info/stats for container "runtime" on node "i-0d6a32f749839192f" I0305 16:43:09.268582 7203 runners.go:194] cleanup20-16a6f148-bb85-4b73-99f5-2adb1cfdeea0 Pods: 20 out of 20 created, 10 running, 10 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady Mar 5 16:43:09.992: INFO: Missing info/stats for container "runtime" on node "i-056ad61525499081e" Mar 5 16:43:13.815: INFO: Missing info/stats for container "runtime" on node "i-0d6a32f749839192f" Mar 5 16:43:15.175: INFO: Missing info/stats for container "runtime" on node "i-056ad61525499081e" Mar 5 16:43:15.867: INFO: Error getting node stats summary on "i-06df568f6e5c59fff", err: the server is currently unable to handle the request (get nodes i-06df568f6e5c59fff:10250) Mar 5 16:43:18.959: INFO: Missing info/stats for container "runtime" on node "i-0d6a32f749839192f" I0305 16:43:19.268925 7203 runners.go:194] cleanup20-16a6f148-bb85-4b73-99f5-2adb1cfdeea0 Pods: 20 out of 20 created, 10 running, 10 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady Mar 5 16:43:20.362: INFO: Missing info/stats for container "runtime" on node "i-056ad61525499081e" Mar 5 16:43:24.105: INFO: Missing info/stats for container "runtime" on node "i-0d6a32f749839192f" Mar 5 16:43:25.543: INFO: Missing info/stats for container "runtime" on node "i-056ad61525499081e" Mar 5 16:43:29.250: INFO: Missing info/stats for container "runtime" on node "i-0d6a32f749839192f" I0305 16:43:29.270123 7203 runners.go:194] cleanup20-16a6f148-bb85-4b73-99f5-2adb1cfdeea0 Pods: 20 out of 20 created, 10 running, 10 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady Mar 5 16:43:30.719: INFO: Missing info/stats for container "runtime" on node "i-056ad61525499081e" Mar 5 16:43:30.986: INFO: Error getting node stats summary on "i-06df568f6e5c59fff", err: the server is currently unable to handle the request (get nodes i-06df568f6e5c59fff:10250) Mar 5 16:43:34.400: INFO: Missing info/stats for container "runtime" on node "i-0d6a32f749839192f" Mar 5 16:43:35.899: INFO: Missing info/stats for container "runtime" on node "i-056ad61525499081e" I0305 16:43:39.270935 7203 runners.go:194] cleanup20-16a6f148-bb85-4b73-99f5-2adb1cfdeea0 Pods: 20 out of 20 created, 10 running, 10 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady Mar 5 16:43:39.544: INFO: Missing info/stats for container "runtime" on node "i-0d6a32f749839192f" Mar 5 16:43:41.078: INFO: Missing info/stats for container "runtime" on node "i-056ad61525499081e" Mar 5 16:43:44.693: INFO: Missing info/stats for container "runtime" on node "i-0d6a32f749839192f" Mar 5 16:43:46.100: INFO: Error getting node stats summary on "i-06df568f6e5c59fff", err: the server is currently unable to handle the request (get nodes i-06df568f6e5c59fff:10250) Mar 5 16:43:46.254: INFO: Missing info/stats for container "runtime" on node "i-056ad61525499081e" I0305 16:43:49.271169 7203 runners.go:194] cleanup20-16a6f148-bb85-4b73-99f5-2adb1cfdeea0 Pods: 20 out of 20 created, 10 running, 10 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady Mar 5 16:43:49.840: INFO: Missing info/stats for container "runtime" on node "i-0d6a32f749839192f" Mar 5 16:43:51.438: INFO: Missing info/stats for container "runtime" on node "i-056ad61525499081e" Mar 5 16:43:54.987: INFO: Missing info/stats for container "runtime" on node "i-0d6a32f749839192f" Mar 5 16:43:56.616: INFO: Missing info/stats for container "runtime" on node "i-056ad61525499081e" I0305 16:43:59.272239 7203 runners.go:194] cleanup20-16a6f148-bb85-4b73-99f5-2adb1cfdeea0 Pods: 20 out of 20 created, 10 running, 10 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady Mar 5 16:44:00.145: INFO: Missing info/stats for container "runtime" on node "i-0d6a32f749839192f" Mar 5 16:44:01.215: INFO: Error getting node stats summary on "i-06df568f6e5c59fff", err: the server is currently unable to handle the request (get nodes i-06df568f6e5c59fff:10250) Mar 5 16:44:01.804: INFO: Missing info/stats for container "runtime" on node "i-056ad61525499081e" Mar 5 16:44:05.289: INFO: Missing info/stats for container "runtime" on node "i-0d6a32f749839192f" Mar 5 16:44:06.985: INFO: Missing info/stats for container "runtime" on node "i-056ad61525499081e" I0305 16:44:09.273153 7203 runners.go:194] cleanup20-16a6f148-bb85-4b73-99f5-2adb1cfdeea0 Pods: 20 out of 20 created, 10 running, 10 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady Mar 5 16:44:10.435: INFO: Missing info/stats for container "runtime" on node "i-0d6a32f749839192f" Mar 5 16:44:12.175: INFO: Missing info/stats for container "runtime" on node "i-056ad61525499081e" Mar 5 16:44:15.582: INFO: Missing info/stats for container "runtime" on node "i-0d6a32f749839192f" Mar 5 16:44:16.329: INFO: Error getting node stats summary on "i-06df568f6e5c59fff", err: the server is currently unable to handle the request (get nodes i-06df568f6e5c59fff:10250) Mar 5 16:44:17.342: INFO: Missing info/stats for container "runtime" on node "i-056ad61525499081e" I0305 16:44:19.273385 7203 runners.go:194] cleanup20-16a6f148-bb85-4b73-99f5-2adb1cfdeea0 Pods: 20 out of 20 created, 10 running, 10 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady Mar 5 16:44:20.727: INFO: Missing info/stats for container "runtime" on node "i-0d6a32f749839192f" Mar 5 16:44:22.518: INFO: Missing info/stats for container "runtime" on node "i-056ad61525499081e" Mar 5 16:44:25.874: INFO: Missing info/stats for container "runtime" on node "i-0d6a32f749839192f" Mar 5 16:44:27.695: INFO: Missing info/stats for container "runtime" on node "i-056ad61525499081e" I0305 16:44:29.273690 7203 runners.go:194] cleanup20-16a6f148-bb85-4b73-99f5-2adb1cfdeea0 Pods: 20 out of 20 created, 10 running, 10 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady Mar 5 16:44:31.019: INFO: Missing info/stats for container "runtime" on node "i-0d6a32f749839192f" Mar 5 16:44:31.444: INFO: Error getting node stats summary on "i-06df568f6e5c59fff", err: the server is currently unable to handle the request (get nodes i-06df568f6e5c59fff:10250) Mar 5 16:44:32.881: INFO: Missing info/stats for container "runtime" on node "i-056ad61525499081e" Mar 5 16:44:36.164: INFO: Missing info/stats for container "runtime" on node "i-0d6a32f749839192f" Mar 5 16:44:38.065: INFO: Missing info/stats for container "runtime" on node "i-056ad61525499081e" I0305 16:44:39.274041 7203 runners.go:194] cleanup20-16a6f148-bb85-4b73-99f5-2adb1cfdeea0 Pods: 20 out of 20 created, 10 running, 10 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady Mar 5 16:44:41.309: INFO: Missing info/stats for container "runtime" on node "i-0d6a32f749839192f" Mar 5 16:44:43.242: INFO: Missing info/stats for container "runtime" on node "i-056ad61525499081e" Mar 5 16:44:46.454: INFO: Missing info/stats for container "runtime" on node "i-0d6a32f749839192f" Mar 5 16:44:46.560: INFO: Error getting node stats summary on "i-06df568f6e5c59fff", err: the server is currently unable to handle the request (get nodes i-06df568f6e5c59fff:10250) Mar 5 16:44:48.425: INFO: Missing info/stats for container "runtime" on node "i-056ad61525499081e" I0305 16:44:49.274988 7203 runners.go:194] cleanup20-16a6f148-bb85-4b73-99f5-2adb1cfdeea0 Pods: 20 out of 20 created, 10 running, 10 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady Mar 5 16:44:51.599: INFO: Missing info/stats for container "runtime" on node "i-0d6a32f749839192f" Mar 5 16:44:53.602: INFO: Missing info/stats for container "runtime" on node "i-056ad61525499081e" Mar 5 16:44:56.743: INFO: Missing info/stats for container "runtime" on node "i-0d6a32f749839192f" Mar 5 16:44:58.781: INFO: Missing info/stats for container "runtime" on node "i-056ad61525499081e" ... skipping 39 lines ... I0305 16:45:49.393831 7203 runners.go:194] Pod cleanup20-16a6f148-bb85-4b73-99f5-2adb1cfdeea0-sqrjl i-056ad61525499081e Running <nil> I0305 16:45:49.393837 7203 runners.go:194] Pod cleanup20-16a6f148-bb85-4b73-99f5-2adb1cfdeea0-wmjdk i-056ad61525499081e Running <nil> I0305 16:45:49.393844 7203 runners.go:194] Pod cleanup20-16a6f148-bb85-4b73-99f5-2adb1cfdeea0-wnmrj i-06df568f6e5c59fff Pending <nil> I0305 16:45:49.393850 7203 runners.go:194] Pod cleanup20-16a6f148-bb85-4b73-99f5-2adb1cfdeea0-x6s5k i-056ad61525499081e Running <nil> I0305 16:45:49.393857 7203 runners.go:194] Pod cleanup20-16a6f148-bb85-4b73-99f5-2adb1cfdeea0-xkdzr i-056ad61525499081e Running <nil> I0305 16:45:49.393866 7203 runners.go:194] Pod cleanup20-16a6f148-bb85-4b73-99f5-2adb1cfdeea0-xsqgf i-056ad61525499081e Running <nil> Mar 5 16:45:49.393: INFO: Unexpected error: <*errors.errorString | 0xc0014b66e0>: { s: "only 10 pods started out of 20", } [38;5;9m[FAILED][0m in [It] - test/e2e/node/kubelet.go:350 [38;5;243m@ 03/05/23 16:45:49.394[0m Mar 5 16:45:49.394: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready Mar 5 16:45:49.394: INFO: Error getting node stats summary on "i-06df568f6e5c59fff", err: Get "https://api.e2e-e2e-kops-aws-karpenter.test-cncf-aws.k8s.io/api/v1/nodes/i-06df568f6e5c59fff:10250/proxy/stats/summary": context canceled Mar 5 16:45:49.508: INFO: Condition Ready of node i-06df568f6e5c59fff is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-05 16:42:51 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-05 16:42:57 +0000 UTC}]. Failure Mar 5 16:45:51.623: INFO: Condition Ready of node i-06df568f6e5c59fff is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-05 16:42:51 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-05 16:42:57 +0000 UTC}]. Failure Mar 5 16:45:53.622: INFO: Condition Ready of node i-06df568f6e5c59fff is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-05 16:42:51 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-05 16:42:57 +0000 UTC}]. Failure Mar 5 16:45:55.622: INFO: Condition Ready of node i-06df568f6e5c59fff is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-05 16:42:51 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-05 16:42:57 +0000 UTC}]. Failure Mar 5 16:45:57.623: INFO: Condition Ready of node i-06df568f6e5c59fff is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-05 16:42:51 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-05 16:42:57 +0000 UTC}]. Failure Mar 5 16:45:59.622: INFO: Condition Ready of node i-06df568f6e5c59fff is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-05 16:42:51 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-05 16:42:57 +0000 UTC}]. Failure ... skipping 239 lines ... Mar 5 16:48:50.664: INFO: cleanup20-16a6f148-bb85-4b73-99f5-2adb1cfdeea0-wmjdk i-056ad61525499081e Running [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2023-03-05 16:40:29 +0000 UTC } {Ready True 0001-01-01 00:00:00 +0000 UTC 2023-03-05 16:40:32 +0000 UTC } {ContainersReady True 0001-01-01 00:00:00 +0000 UTC 2023-03-05 16:40:32 +0000 UTC } {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2023-03-05 16:40:29 +0000 UTC }] Mar 5 16:48:50.664: INFO: cleanup20-16a6f148-bb85-4b73-99f5-2adb1cfdeea0-wnmrj i-06df568f6e5c59fff Pending 1s [{PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2023-03-05 16:40:29 +0000 UTC } {DisruptionTarget True 0001-01-01 00:00:00 +0000 UTC 2023-03-05 16:47:57 +0000 UTC DeletionByTaintManager Taint manager: deleting due to NoExecute taint}] Mar 5 16:48:50.664: INFO: cleanup20-16a6f148-bb85-4b73-99f5-2adb1cfdeea0-x6s5k i-056ad61525499081e Running [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2023-03-05 16:40:29 +0000 UTC } {Ready True 0001-01-01 00:00:00 +0000 UTC 2023-03-05 16:40:32 +0000 UTC } {ContainersReady True 0001-01-01 00:00:00 +0000 UTC 2023-03-05 16:40:32 +0000 UTC } {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2023-03-05 16:40:29 +0000 UTC }] Mar 5 16:48:50.664: INFO: cleanup20-16a6f148-bb85-4b73-99f5-2adb1cfdeea0-xkdzr i-056ad61525499081e Running [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2023-03-05 16:40:29 +0000 UTC } {Ready True 0001-01-01 00:00:00 +0000 UTC 2023-03-05 16:40:32 +0000 UTC } {ContainersReady True 0001-01-01 00:00:00 +0000 UTC 2023-03-05 16:40:32 +0000 UTC } {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2023-03-05 16:40:29 +0000 UTC }] Mar 5 16:48:50.664: INFO: cleanup20-16a6f148-bb85-4b73-99f5-2adb1cfdeea0-xsqgf i-056ad61525499081e Running [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2023-03-05 16:40:29 +0000 UTC } {Ready True 0001-01-01 00:00:00 +0000 UTC 2023-03-05 16:40:33 +0000 UTC } {ContainersReady True 0001-01-01 00:00:00 +0000 UTC 2023-03-05 16:40:33 +0000 UTC } {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2023-03-05 16:40:29 +0000 UTC }] Mar 5 16:48:50.664: INFO: Mar 5 16:49:00.626: INFO: Unable to fetch kubelet-1879/cleanup20-16a6f148-bb85-4b73-99f5-2adb1cfdeea0-29d8m/cleanup20-16a6f148-bb85-4b73-99f5-2adb1cfdeea0 logs: an error on the server ("unknown") has prevented the request from succeeding (get pods cleanup20-16a6f148-bb85-4b73-99f5-2adb1cfdeea0-29d8m) Mar 5 16:49:10.740: INFO: Unable to fetch kubelet-1879/cleanup20-16a6f148-bb85-4b73-99f5-2adb1cfdeea0-2g4th/cleanup20-16a6f148-bb85-4b73-99f5-2adb1cfdeea0 logs: an error on the server ("unknown") has prevented the request from succeeding (get pods cleanup20-16a6f148-bb85-4b73-99f5-2adb1cfdeea0-2g4th) Mar 5 16:49:20.856: INFO: Unable to fetch kubelet-1879/cleanup20-16a6f148-bb85-4b73-99f5-2adb1cfdeea0-2zj7j/cleanup20-16a6f148-bb85-4b73-99f5-2adb1cfdeea0 logs: an error on the server ("unknown") has prevented the request from succeeding (get pods cleanup20-16a6f148-bb85-4b73-99f5-2adb1cfdeea0-2zj7j) Mar 5 16:49:21.208: INFO: Logging node info for node i-056ad61525499081e Mar 5 16:49:21.322: INFO: Node Info: &Node{ObjectMeta:{i-056ad61525499081e a6ce9ed3-5ae8-4065-b117-6884e24b3632 18352 0 2023-03-05 16:24:16 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:arm64 beta.kubernetes.io/instance-type:t4g.small beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:eu-central-1 failure-domain.beta.kubernetes.io/zone:eu-central-1a karpenter.k8s.aws/instance-category:t karpenter.k8s.aws/instance-cpu:2 karpenter.k8s.aws/instance-family:t4g karpenter.k8s.aws/instance-generation:4 karpenter.k8s.aws/instance-hypervisor:nitro karpenter.k8s.aws/instance-memory:2048 karpenter.k8s.aws/instance-pods:11 karpenter.k8s.aws/instance-size:small karpenter.sh/capacity-type:spot karpenter.sh/initialized:true karpenter.sh/provisioner-name:nodes kubernetes.io/arch:arm64 kubernetes.io/hostname:i-056ad61525499081e kubernetes.io/os:linux node-role.kubernetes.io/node: node-role.kubernetes.io/spot-worker:true node.kubernetes.io/instance-type:t4g.small topology.ebs.csi.aws.com/zone:eu-central-1a topology.hostpath.csi/node:i-056ad61525499081e topology.kubernetes.io/region:eu-central-1 topology.kubernetes.io/zone:eu-central-1a] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-056ad61525499081e"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [{karpenter.sh/v1alpha5 Provisioner nodes ac5d7599-b2a2-4eae-8da4-1f4c0a54ec82 <nil> 0xc003487110}] [karpenter.sh/termination] [{aws-cloud-controller-manager Update v1 2023-03-05 16:24:16 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{}}}} } {kops-controller Update v1 2023-03-05 16:24:16 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{},"f:node-role.kubernetes.io/spot-worker":{}}}} } {karpenter Update v1 2023-03-05 16:25:57 +0000 UTC FieldsV1 {"f:metadata":{"f:finalizers":{".":{},"v:\"karpenter.sh/termination\"":{}},"f:labels":{".":{},"f:karpenter.k8s.aws/instance-category":{},"f:karpenter.k8s.aws/instance-cpu":{},"f:karpenter.k8s.aws/instance-family":{},"f:karpenter.k8s.aws/instance-generation":{},"f:karpenter.k8s.aws/instance-hypervisor":{},"f:karpenter.k8s.aws/instance-memory":{},"f:karpenter.k8s.aws/instance-pods":{},"f:karpenter.k8s.aws/instance-size":{},"f:karpenter.sh/capacity-type":{},"f:karpenter.sh/initialized":{},"f:karpenter.sh/provisioner-name":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}},"f:ownerReferences":{".":{},"k:{\"uid\":\"ac5d7599-b2a2-4eae-8da4-1f4c0a54ec82\"}":{}}},"f:spec":{"f:providerID":{}}} } {kube-controller-manager Update v1 2023-03-05 16:25:57 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:node.alpha.kubernetes.io/ttl":{}},"f:labels":{"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.1.0/24\"":{}}}} } {aws-cloud-controller-manager Update v1 2023-03-05 16:26:48 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{".":{},"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"InternalIP\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kube-controller-manager Update v1 2023-03-05 16:48:56 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{".":{},"k:{\"type\":\"DiskPressure\"}":{".":{},"f:type":{}},"k:{\"type\":\"MemoryPressure\"}":{".":{},"f:type":{}},"k:{\"type\":\"PIDPressure\"}":{".":{},"f:type":{}},"k:{\"type\":\"Ready\"}":{".":{},"f:type":{}}},"f:volumesAttached":{}}} status} {kubelet Update v1 2023-03-05 16:48:56 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{"f:kubernetes.io/hostname":{},"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{".":{},"f:cpu":{},"f:ephemeral-storage":{},"f:hugepages-1Gi":{},"f:hugepages-2Mi":{},"f:hugepages-32Mi":{},"f:hugepages-64Ki":{},"f:memory":{},"f:pods":{}},"f:capacity":{".":{},"f:cpu":{},"f:ephemeral-storage":{},"f:hugepages-1Gi":{},"f:hugepages-2Mi":{},"f:hugepages-32Mi":{},"f:hugepages-64Ki":{},"f:memory":{},"f:pods":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:daemonEndpoints":{"f:kubeletEndpoint":{"f:Port":{}}},"f:images":{},"f:nodeInfo":{"f:architecture":{},"f:bootID":{},"f:containerRuntimeVersion":{},"f:kernelVersion":{},"f:kubeProxyVersion":{},"f:kubeletVersion":{},"f:machineID":{},"f:operatingSystem":{},"f:osImage":{},"f:systemUUID":{}},"f:volumesInUse":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.1.0/24,DoNotUseExternalID:,ProviderID:aws:///eu-central-1a/i-056ad61525499081e,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.1.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49756430336 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-32Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-64Ki: {{0 0} {<nil>} 0 DecimalSI},memory: {{1941540864 0} {<nil>} 1896036Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44780787229 0} {<nil>} 44780787229 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-32Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-64Ki: {{0 0} {<nil>} 0 DecimalSI},memory: {{1836683264 0} {<nil>} 1793636Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-03-05 16:48:56 +0000 UTC,LastTransitionTime:2023-03-05 16:25:57 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-03-05 16:48:56 +0000 UTC,LastTransitionTime:2023-03-05 16:25:48 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-03-05 16:48:56 +0000 UTC,LastTransitionTime:2023-03-05 16:25:48 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-03-05 16:48:56 +0000 UTC,LastTransitionTime:2023-03-05 16:25:48 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.36.179,},NodeAddress{Type:ExternalIP,Address:18.159.37.200,},NodeAddress{Type:InternalDNS,Address:i-056ad61525499081e.eu-central-1.compute.internal,},NodeAddress{Type:Hostname,Address:i-056ad61525499081e.eu-central-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-18-159-37-200.eu-central-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2af3207d1c8c09785c337aae6c6ef0,SystemUUID:ec2af320-7d1c-8c09-785c-337aae6c6ef0,BootID:786f2752-bc30-4729-8aaf-45977b16c253,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 22.04.2 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.27.0-alpha.3.52+d48b8167f7094b,KubeProxyVersion:v1.27.0-alpha.3.52+d48b8167f7094b,OperatingSystem:linux,Architecture:arm64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:157636062,},ContainerImage{Names:[registry.k8s.io/kube-proxy-arm64:v1.27.0-alpha.3.52_d48b8167f7094b],SizeBytes:60554082,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:16bbf38c463a4223d8cfe4da12bc61010b082a79b4bb003e2d3ba3ece5dd5f9e registry.k8s.io/e2e-test-images/agnhost:2.43],SizeBytes:49808364,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:3fe7acf013d1264ffded116b80a73dc129a449b0fccdb8d21af8279f2233f36e registry.k8s.io/e2e-test-images/httpd:2.4.39-4],SizeBytes:41528818,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:148b022f5c5da426fc2f3c14b5c0867e58ef05961510c84749ac1fddcb0fef22 registry.k8s.io/e2e-test-images/httpd:2.4.38-4],SizeBytes:40343601,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:28932904,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:ee3b525d5b89db99da3b8eb521d9cd90cb6e9ef0fbb651e98bb37be78d36b5b8 registry.k8s.io/sig-storage/csi-provisioner:v3.3.0],SizeBytes:23945474,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:425d8f1b769398127767b06ed97ce62578a3179bcb99809ce93a1649e025ffe7 registry.k8s.io/sig-storage/csi-resizer:v1.6.0],SizeBytes:22678632,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:291334908ddf71a4661fd7f6d9d97274de8a5378a2b6fdfeb2ce73414a34f82f registry.k8s.io/sig-storage/csi-snapshotter:v6.1.0],SizeBytes:22434180,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:9a685020911e2725ad019dbce6e4a5ab93d51e3d4557f115e64343345e05781b registry.k8s.io/sig-storage/csi-attacher:v4.0.0],SizeBytes:22394501,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:122bfb8c1edabb3c0edd63f06523e6940d958d19b3957dc7b1d6f81e9f1f6119 registry.k8s.io/sig-storage/csi-provisioner:v3.1.0],SizeBytes:21735644,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:9ebbf9f023e7b41ccee3d52afe39a89e3ddacdbb69269d583abfc25847cfd9e4 registry.k8s.io/sig-storage/csi-resizer:v1.4.0],SizeBytes:20848413,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:8b9c313c05f54fb04f8d430896f5f5904b6cb157df261501b29adc04d2b2dc7b registry.k8s.io/sig-storage/csi-attacher:v3.4.0],SizeBytes:20565480,},ContainerImage{Names:[registry.k8s.io/cpa/cluster-proportional-autoscaler@sha256:68d396900aeaa072c1f27289485fdac29834045a6f3ffe369bf389d830ef572d registry.k8s.io/cpa/cluster-proportional-autoscaler:1.8.6],SizeBytes:19007623,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:92257881c1d6493cf18299a24af42330f891166560047902b8d431fb66b01af5 registry.k8s.io/sig-storage/hostpathplugin:v1.9.0],SizeBytes:18267611,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:9b2d9abda017c28b12f84a344f57cd73fbdb6c2bd7dd5bdd5018246ad1093ba6 registry.k8s.io/sig-storage/hostpathplugin:v1.11.0],SizeBytes:18159085,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:017727efcfeb7d053af68e51436ce8e65edbc6ca573720afb4f79c8594036955 registry.k8s.io/coredns/coredns:v1.10.0],SizeBytes:13766234,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:8502345,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:933940f13b3ea0abc62e656c1aa5c5b47c04b15d71250413a6b821bd0c58b94e registry.k8s.io/sig-storage/livenessprobe:v2.7.0],SizeBytes:8223819,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:7685305,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:5c99cf6a02adda929b10321dbf4ecfa00d87be9ba4fb456006237d530ab4baa1 registry.k8s.io/e2e-test-images/nginx:1.14-4],SizeBytes:6860089,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:2e0f836850e09b8b7cc937681d6194537a09fbd5f6b9e08f4d646a85128e8937 registry.k8s.io/e2e-test-images/busybox:1.29-4],SizeBytes:772933,},ContainerImage{Names:[registry.k8s.io/pause@sha256:7031c1b283388d2c2e09b57badb803c05ebed362dc88d84b480cc47f72a21097 registry.k8s.io/pause:3.9],SizeBytes:268051,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:253553,},},VolumesInUse:[kubernetes.io/csi/ebs.csi.aws.com^vol-0e020446b1cd6abaf],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-0e020446b1cd6abaf,DevicePath:,},},Config:nil,},} Mar 5 16:49:21.322: INFO: Logging kubelet events for node i-056ad61525499081e Mar 5 16:49:21.438: INFO: ... skipping 11 lines ... Mar 5 16:49:21.563: INFO: coredns-745bddcfbb-gm4h2 started at 2023-03-05 16:25:57 +0000 UTC (0+1 container statuses recorded) Mar 5 16:49:21.563: INFO: Container coredns ready: true, restart count 0 Mar 5 16:49:21.563: INFO: cleanup20-16a6f148-bb85-4b73-99f5-2adb1cfdeea0-sqrjl started at 2023-03-05 16:40:29 +0000 UTC (0+1 container statuses recorded) Mar 5 16:49:21.563: INFO: Container cleanup20-16a6f148-bb85-4b73-99f5-2adb1cfdeea0 ready: true, restart count 0 Mar 5 16:49:21.563: INFO: cleanup20-16a6f148-bb85-4b73-99f5-2adb1cfdeea0-n9vbn started at 2023-03-05 16:40:29 +0000 UTC (0+1 container statuses recorded) Mar 5 16:49:21.563: INFO: Container cleanup20-16a6f148-bb85-4b73-99f5-2adb1cfdeea0 ready: true, restart count 0 Mar 5 16:49:21.563: INFO: fail-once-local-gj678 started at 2023-03-05 16:46:33 +0000 UTC (0+1 container statuses recorded) Mar 5 16:49:21.563: INFO: Container c ready: false, restart count 1 Mar 5 16:49:21.563: INFO: cleanup20-16a6f148-bb85-4b73-99f5-2adb1cfdeea0-m6dgl started at 2023-03-05 16:48:01 +0000 UTC (0+1 container statuses recorded) Mar 5 16:49:21.563: INFO: Container cleanup20-16a6f148-bb85-4b73-99f5-2adb1cfdeea0 ready: true, restart count 0 Mar 5 16:49:21.563: INFO: cleanup20-16a6f148-bb85-4b73-99f5-2adb1cfdeea0-cqffq started at 2023-03-05 16:48:02 +0000 UTC (0+1 container statuses recorded) Mar 5 16:49:21.563: INFO: Container cleanup20-16a6f148-bb85-4b73-99f5-2adb1cfdeea0 ready: true, restart count 0 Mar 5 16:49:21.563: INFO: ebs-csi-node-n98cq started at 2023-03-05 16:25:49 +0000 UTC (0+3 container statuses recorded) ... skipping 2 lines ... Mar 5 16:49:21.563: INFO: Container node-driver-registrar ready: true, restart count 0 Mar 5 16:49:21.563: INFO: cilium-vfgw4 started at 2023-03-05 16:25:49 +0000 UTC (1+1 container statuses recorded) Mar 5 16:49:21.563: INFO: Init container clean-cilium-state ready: true, restart count 0 Mar 5 16:49:21.563: INFO: Container cilium-agent ready: true, restart count 0 Mar 5 16:49:21.563: INFO: sample-webhook-deployment-7497495989-s9585 started at 2023-03-05 16:46:38 +0000 UTC (0+1 container statuses recorded) Mar 5 16:49:21.563: INFO: Container sample-webhook ready: true, restart count 0 Mar 5 16:49:21.563: INFO: fail-once-local-kdbgv started at 2023-03-05 16:46:33 +0000 UTC (0+1 container statuses recorded) Mar 5 16:49:21.563: INFO: Container c ready: false, restart count 1 Mar 5 16:49:21.563: INFO: cleanup20-16a6f148-bb85-4b73-99f5-2adb1cfdeea0-xsqgf started at 2023-03-05 16:40:29 +0000 UTC (0+1 container statuses recorded) Mar 5 16:49:21.563: INFO: Container cleanup20-16a6f148-bb85-4b73-99f5-2adb1cfdeea0 ready: true, restart count 0 Mar 5 16:49:21.563: INFO: cleanup20-16a6f148-bb85-4b73-99f5-2adb1cfdeea0-6lnc6 started at 2023-03-05 16:48:01 +0000 UTC (0+1 container statuses recorded) Mar 5 16:49:21.563: INFO: Container cleanup20-16a6f148-bb85-4b73-99f5-2adb1cfdeea0 ready: true, restart count 0 Mar 5 16:49:21.563: INFO: cleanup20-16a6f148-bb85-4b73-99f5-2adb1cfdeea0-9qlxz started at 2023-03-05 16:48:01 +0000 UTC (0+1 container statuses recorded) ... skipping 3 lines ... Mar 5 16:49:21.563: INFO: coredns-autoscaler-8495f5575-plvdw started at 2023-03-05 16:25:57 +0000 UTC (0+1 container statuses recorded) Mar 5 16:49:21.563: INFO: Container autoscaler ready: true, restart count 0 Mar 5 16:49:21.563: INFO: cleanup20-16a6f148-bb85-4b73-99f5-2adb1cfdeea0-kgbnw started at 2023-03-05 16:40:29 +0000 UTC (0+1 container statuses recorded) Mar 5 16:49:21.563: INFO: Container cleanup20-16a6f148-bb85-4b73-99f5-2adb1cfdeea0 ready: true, restart count 0 Mar 5 16:49:21.563: INFO: cleanup20-16a6f148-bb85-4b73-99f5-2adb1cfdeea0-42w6f started at 2023-03-05 16:40:29 +0000 UTC (0+1 container statuses recorded) Mar 5 16:49:21.563: INFO: Container cleanup20-16a6f148-bb85-4b73-99f5-2adb1cfdeea0 ready: true, restart count 0 Mar 5 16:49:21.563: INFO: fail-once-local-69mfr started at 2023-03-05 16:46:28 +0000 UTC (0+1 container statuses recorded) Mar 5 16:49:21.563: INFO: Container c ready: false, restart count 1 Mar 5 16:49:21.563: INFO: cleanup20-16a6f148-bb85-4b73-99f5-2adb1cfdeea0-wmjdk started at 2023-03-05 16:40:29 +0000 UTC (0+1 container statuses recorded) Mar 5 16:49:21.563: INFO: Container cleanup20-16a6f148-bb85-4b73-99f5-2adb1cfdeea0 ready: true, restart count 0 Mar 5 16:49:21.563: INFO: pod-subpath-test-downwardapi-w7jc started at 2023-03-05 16:49:14 +0000 UTC (0+1 container statuses recorded) Mar 5 16:49:21.563: INFO: Container test-container-subpath-downwardapi-w7jc ready: true, restart count 0 Mar 5 16:49:21.563: INFO: ebs-csi-controller-79785c665b-7dmsf started at 2023-03-05 16:25:57 +0000 UTC (0+5 container statuses recorded) ... skipping 13 lines ... Mar 5 16:49:21.563: INFO: cleanup20-16a6f148-bb85-4b73-99f5-2adb1cfdeea0-w749b started at 2023-03-05 16:48:01 +0000 UTC (0+1 container statuses recorded) Mar 5 16:49:21.563: INFO: Container cleanup20-16a6f148-bb85-4b73-99f5-2adb1cfdeea0 ready: true, restart count 0 Mar 5 16:49:21.563: INFO: cleanup20-16a6f148-bb85-4b73-99f5-2adb1cfdeea0-j6cks started at 2023-03-05 16:48:01 +0000 UTC (0+1 container statuses recorded) Mar 5 16:49:21.563: INFO: Container cleanup20-16a6f148-bb85-4b73-99f5-2adb1cfdeea0 ready: true, restart count 0 Mar 5 16:49:21.563: INFO: cleanup20-16a6f148-bb85-4b73-99f5-2adb1cfdeea0-khffj started at 2023-03-05 16:40:29 +0000 UTC (0+1 container statuses recorded) Mar 5 16:49:21.563: INFO: Container cleanup20-16a6f148-bb85-4b73-99f5-2adb1cfdeea0 ready: true, restart count 0 Mar 5 16:49:21.563: INFO: fail-once-local-2mt4w started at 2023-03-05 16:46:28 +0000 UTC (0+1 container statuses recorded) Mar 5 16:49:21.563: INFO: Container c ready: false, restart count 1 Mar 5 16:49:22.149: INFO: Latency metrics for node i-056ad61525499081e Mar 5 16:49:22.149: INFO: Logging node info for node i-06df568f6e5c59fff Mar 5 16:49:22.262: INFO: Node Info: &Node{ObjectMeta:{i-06df568f6e5c59fff cb63c681-2e18-474a-b6f3-221325ac0d52 18301 0 2023-03-05 16:24:16 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:arm64 beta.kubernetes.io/instance-type:t4g.small beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:eu-central-1 failure-domain.beta.kubernetes.io/zone:eu-central-1a karpenter.k8s.aws/instance-category:t karpenter.k8s.aws/instance-cpu:2 karpenter.k8s.aws/instance-family:t4g karpenter.k8s.aws/instance-generation:4 karpenter.k8s.aws/instance-hypervisor:nitro karpenter.k8s.aws/instance-memory:2048 karpenter.k8s.aws/instance-pods:11 karpenter.k8s.aws/instance-size:small karpenter.sh/capacity-type:spot karpenter.sh/initialized:true karpenter.sh/provisioner-name:nodes kubernetes.io/arch:arm64 kubernetes.io/hostname:i-06df568f6e5c59fff kubernetes.io/os:linux node-role.kubernetes.io/node: node-role.kubernetes.io/spot-worker:true node.kubernetes.io/instance-type:t4g.small topology.ebs.csi.aws.com/zone:eu-central-1a topology.hostpath.csi/node:i-06df568f6e5c59fff topology.kubernetes.io/region:eu-central-1 topology.kubernetes.io/zone:eu-central-1a] map[csi.volume.kubernetes.io/nodeid:{"csi-hostpath-ephemeral-2147":"i-06df568f6e5c59fff","csi-hostpath-ephemeral-5408":"i-06df568f6e5c59fff","csi-hostpath-ephemeral-6540":"i-06df568f6e5c59fff","csi-mock-csi-mock-volumes-fsgroup-policy-1058":"i-06df568f6e5c59fff","csi-mock-csi-mock-volumes-workload-2219":"i-06df568f6e5c59fff","ebs.csi.aws.com":"i-06df568f6e5c59fff"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [{karpenter.sh/v1alpha5 Provisioner nodes ac5d7599-b2a2-4eae-8da4-1f4c0a54ec82 <nil> 0xc00222e0e0}] [karpenter.sh/termination] [{aws-cloud-controller-manager Update v1 2023-03-05 16:24:16 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{}}}} } {kops-controller Update v1 2023-03-05 16:24:16 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{},"f:node-role.kubernetes.io/spot-worker":{}}}} } {karpenter Update v1 2023-03-05 16:25:57 +0000 UTC FieldsV1 {"f:metadata":{"f:finalizers":{".":{},"v:\"karpenter.sh/termination\"":{}},"f:labels":{".":{},"f:karpenter.k8s.aws/instance-category":{},"f:karpenter.k8s.aws/instance-cpu":{},"f:karpenter.k8s.aws/instance-family":{},"f:karpenter.k8s.aws/instance-generation":{},"f:karpenter.k8s.aws/instance-hypervisor":{},"f:karpenter.k8s.aws/instance-memory":{},"f:karpenter.k8s.aws/instance-pods":{},"f:karpenter.k8s.aws/instance-size":{},"f:karpenter.sh/capacity-type":{},"f:karpenter.sh/initialized":{},"f:karpenter.sh/provisioner-name":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}},"f:ownerReferences":{".":{},"k:{\"uid\":\"ac5d7599-b2a2-4eae-8da4-1f4c0a54ec82\"}":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-03-05 16:26:48 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{".":{},"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"InternalIP\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kube-controller-manager Update v1 2023-03-05 16:44:29 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{".":{},"k:{\"type\":\"DiskPressure\"}":{".":{},"f:type":{}},"k:{\"type\":\"MemoryPressure\"}":{".":{},"f:type":{}},"k:{\"type\":\"PIDPressure\"}":{".":{},"f:type":{}},"k:{\"type\":\"Ready\"}":{".":{},"f:type":{}}},"f:volumesAttached":{}}} status} {kube-controller-manager Update v1 2023-03-05 16:46:37 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:node.alpha.kubernetes.io/ttl":{}},"f:labels":{"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.2.0/24\"":{}},"f:taints":{}}} } {kubelet Update v1 2023-03-05 16:48:40 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{"f:kubernetes.io/hostname":{},"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{".":{},"f:cpu":{},"f:ephemeral-storage":{},"f:hugepages-1Gi":{},"f:hugepages-2Mi":{},"f:hugepages-32Mi":{},"f:hugepages-64Ki":{},"f:memory":{},"f:pods":{}},"f:capacity":{".":{},"f:cpu":{},"f:ephemeral-storage":{},"f:hugepages-1Gi":{},"f:hugepages-2Mi":{},"f:hugepages-32Mi":{},"f:hugepages-64Ki":{},"f:memory":{},"f:pods":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:daemonEndpoints":{"f:kubeletEndpoint":{"f:Port":{}}},"f:images":{},"f:nodeInfo":{"f:architecture":{},"f:bootID":{},"f:containerRuntimeVersion":{},"f:kernelVersion":{},"f:kubeProxyVersion":{},"f:kubeletVersion":{},"f:machineID":{},"f:operatingSystem":{},"f:osImage":{},"f:systemUUID":{}},"f:volumesInUse":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.2.0/24,DoNotUseExternalID:,ProviderID:aws:///eu-central-1a/i-06df568f6e5c59fff,Unschedulable:false,Taints:[]Taint{Taint{Key:node.kubernetes.io/not-ready,Value:,Effect:NoSchedule,TimeAdded:2023-03-05 16:46:33 +0000 UTC,},Taint{Key:node.kubernetes.io/not-ready,Value:,Effect:NoExecute,TimeAdded:2023-03-05 16:46:37 +0000 UTC,},},ConfigSource:nil,PodCIDRs:[100.96.2.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49756430336 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-32Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-64Ki: {{0 0} {<nil>} 0 DecimalSI},memory: {{1941540864 0} {<nil>} 1896036Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44780787229 0} {<nil>} 44780787229 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-32Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-64Ki: {{0 0} {<nil>} 0 DecimalSI},memory: {{1836683264 0} {<nil>} 1793636Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:Ready,Status:False,LastHeartbeatTime:2023-03-05 16:47:34 +0000 UTC,LastTransitionTime:2023-03-05 16:46:08 +0000 UTC,Reason:KubeletNotReady,Message:PLEG is not healthy: pleg was last seen active 8m29.527212456s ago; threshold is 3m0s,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-03-05 16:47:34 +0000 UTC,LastTransitionTime:2023-03-05 16:47:12 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-03-05 16:47:34 +0000 UTC,LastTransitionTime:2023-03-05 16:47:15 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-03-05 16:47:34 +0000 UTC,LastTransitionTime:2023-03-05 16:47:15 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.43.148,},NodeAddress{Type:ExternalIP,Address:18.192.211.222,},NodeAddress{Type:InternalDNS,Address:i-06df568f6e5c59fff.eu-central-1.compute.internal,},NodeAddress{Type:Hostname,Address:i-06df568f6e5c59fff.eu-central-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-18-192-211-222.eu-central-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec221221864d9409a8ec5f6614a9b94b,SystemUUID:ec221221-864d-9409-a8ec-5f6614a9b94b,BootID:553d31bf-faca-417e-a3db-325839e1b8e4,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 22.04.2 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.27.0-alpha.3.52+d48b8167f7094b,KubeProxyVersion:v1.27.0-alpha.3.52+d48b8167f7094b,OperatingSystem:linux,Architecture:arm64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:157636062,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/jessie-dnsutils@sha256:24aaf2626d6b27864c29de2097e8bbb840b3a414271bf7c8995e431e47d8408e registry.k8s.io/e2e-test-images/jessie-dnsutils:1.7],SizeBytes:116764683,},ContainerImage{Names:[registry.k8s.io/kube-proxy-arm64:v1.27.0-alpha.3.52_d48b8167f7094b],SizeBytes:60554082,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:16bbf38c463a4223d8cfe4da12bc61010b082a79b4bb003e2d3ba3ece5dd5f9e registry.k8s.io/e2e-test-images/agnhost:2.43],SizeBytes:49808364,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:3fe7acf013d1264ffded116b80a73dc129a449b0fccdb8d21af8279f2233f36e registry.k8s.io/e2e-test-images/httpd:2.4.39-4],SizeBytes:41528818,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:148b022f5c5da426fc2f3c14b5c0867e58ef05961510c84749ac1fddcb0fef22 registry.k8s.io/e2e-test-images/httpd:2.4.38-4],SizeBytes:40343601,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:28932904,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:ee3b525d5b89db99da3b8eb521d9cd90cb6e9ef0fbb651e98bb37be78d36b5b8 registry.k8s.io/sig-storage/csi-provisioner:v3.3.0],SizeBytes:23945474,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:425d8f1b769398127767b06ed97ce62578a3179bcb99809ce93a1649e025ffe7 registry.k8s.io/sig-storage/csi-resizer:v1.6.0],SizeBytes:22678632,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:291334908ddf71a4661fd7f6d9d97274de8a5378a2b6fdfeb2ce73414a34f82f registry.k8s.io/sig-storage/csi-snapshotter:v6.1.0],SizeBytes:22434180,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:9a685020911e2725ad019dbce6e4a5ab93d51e3d4557f115e64343345e05781b registry.k8s.io/sig-storage/csi-attacher:v4.0.0],SizeBytes:22394501,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:122bfb8c1edabb3c0edd63f06523e6940d958d19b3957dc7b1d6f81e9f1f6119 registry.k8s.io/sig-storage/csi-provisioner:v3.1.0],SizeBytes:21735644,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:9ebbf9f023e7b41ccee3d52afe39a89e3ddacdbb69269d583abfc25847cfd9e4 registry.k8s.io/sig-storage/csi-resizer:v1.4.0],SizeBytes:20848413,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:8b9c313c05f54fb04f8d430896f5f5904b6cb157df261501b29adc04d2b2dc7b registry.k8s.io/sig-storage/csi-attacher:v3.4.0],SizeBytes:20565480,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nonroot@sha256:ee9f50b3c64b174d296d91ca9f69a914ac30e59095dfb462b2b518ad28a63655 registry.k8s.io/e2e-test-images/nonroot:1.4],SizeBytes:18561972,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:92257881c1d6493cf18299a24af42330f891166560047902b8d431fb66b01af5 registry.k8s.io/sig-storage/hostpathplugin:v1.9.0],SizeBytes:18267611,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:9b2d9abda017c28b12f84a344f57cd73fbdb6c2bd7dd5bdd5018246ad1093ba6 registry.k8s.io/sig-storage/hostpathplugin:v1.11.0],SizeBytes:18159085,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:017727efcfeb7d053af68e51436ce8e65edbc6ca573720afb4f79c8594036955 registry.k8s.io/coredns/coredns:v1.10.0],SizeBytes:13766234,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:8502345,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:933940f13b3ea0abc62e656c1aa5c5b47c04b15d71250413a6b821bd0c58b94e registry.k8s.io/sig-storage/livenessprobe:v2.7.0],SizeBytes:8223819,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:7685305,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:5c99cf6a02adda929b10321dbf4ecfa00d87be9ba4fb456006237d530ab4baa1 registry.k8s.io/e2e-test-images/nginx:1.14-4],SizeBytes:6860089,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nonewprivs@sha256:8ac1264691820febacf3aea5d152cbde6d10685731ec14966a9401c6f47a68ac registry.k8s.io/e2e-test-images/nonewprivs:1.3],SizeBytes:5025027,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:2e0f836850e09b8b7cc937681d6194537a09fbd5f6b9e08f4d646a85128e8937 registry.k8s.io/e2e-test-images/busybox:1.29-4],SizeBytes:772933,},ContainerImage{Names:[registry.k8s.io/pause@sha256:7031c1b283388d2c2e09b57badb803c05ebed362dc88d84b480cc47f72a21097 registry.k8s.io/pause:3.9],SizeBytes:268051,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:253553,},},VolumesInUse:[kubernetes.io/csi/csi-hostpath-ephemeral-2147^52df5953-bb73-11ed-8aa7-be49170871b4 kubernetes.io/csi/csi-hostpath-ephemeral-5408^443d1daf-bb73-11ed-95c4-5a95d17cbc27 kubernetes.io/csi/csi-hostpath-ephemeral-5408^44411f51-bb73-11ed-95c4-5a95d17cbc27 kubernetes.io/csi/csi-hostpath-ephemeral-6540^4426a07f-bb73-11ed-a604-92687f2c50f8 kubernetes.io/csi/ebs.csi.aws.com^vol-05b67fab5beb6394b],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-05b67fab5beb6394b,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/csi-hostpath-ephemeral-6540^4426a07f-bb73-11ed-a604-92687f2c50f8,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/csi-hostpath-ephemeral-5408^44411f51-bb73-11ed-95c4-5a95d17cbc27,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/csi-hostpath-ephemeral-5408^443d1daf-bb73-11ed-95c4-5a95d17cbc27,DevicePath:,},},Config:nil,},} Mar 5 16:49:22.262: INFO: Logging kubelet events for node i-06df568f6e5c59fff Mar 5 16:49:22.379: INFO: Logging pods the kubelet thinks is on node i-06df568f6e5c59fff Mar 5 16:49:32.493: INFO: Unable to retrieve kubelet pods for node i-06df568f6e5c59fff: error trying to reach service: net/http: TLS handshake timeout Mar 5 16:49:32.493: INFO: Logging node info for node i-0d6a32f749839192f Mar 5 16:49:32.607: INFO: Node Info: &Node{ObjectMeta:{i-0d6a32f749839192f 4155fefb-4d3a-4b04-ba53-6ac9e895ff92 18509 0 2023-03-05 16:20:46 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:arm64 beta.kubernetes.io/instance-type:c6g.xlarge beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:eu-central-1 failure-domain.beta.kubernetes.io/zone:eu-central-1a kops.k8s.io/kops-controller-pki: kubernetes.io/arch:arm64 kubernetes.io/hostname:i-0d6a32f749839192f kubernetes.io/os:linux node-role.kubernetes.io/control-plane: node.kubernetes.io/exclude-from-external-load-balancers: node.kubernetes.io/instance-type:c6g.xlarge topology.ebs.csi.aws.com/zone:eu-central-1a topology.kubernetes.io/region:eu-central-1 topology.kubernetes.io/zone:eu-central-1a] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-0d6a32f749839192f"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kubelet Update v1 2023-03-05 16:20:46 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {protokube Update v1 2023-03-05 16:21:25 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kops.k8s.io/kops-controller-pki":{},"f:node-role.kubernetes.io/control-plane":{},"f:node.kubernetes.io/exclude-from-external-load-balancers":{}}}} } {kube-controller-manager Update v1 2023-03-05 16:21:47 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.0.0/24\"":{}}}} } {aws-cloud-controller-manager Update v1 2023-03-05 16:21:48 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{},"f:taints":{}}} } {aws-cloud-controller-manager Update v1 2023-03-05 16:21:48 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kubelet Update v1 2023-03-05 16:49:18 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.0.0/24,DoNotUseExternalID:,ProviderID:aws:///eu-central-1a/i-0d6a32f749839192f,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/control-plane,Value:,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[100.96.0.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{4 0} {<nil>} 4 DecimalSI},ephemeral-storage: {{49756430336 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-32Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-64Ki: {{0 0} {<nil>} 0 DecimalSI},memory: {{8150118400 0} {<nil>} 7959100Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{4 0} {<nil>} 4 DecimalSI},ephemeral-storage: {{44780787229 0} {<nil>} 44780787229 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-32Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-64Ki: {{0 0} {<nil>} 0 DecimalSI},memory: {{8045260800 0} {<nil>} 7856700Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-03-05 16:49:18 +0000 UTC,LastTransitionTime:2023-03-05 16:20:46 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-03-05 16:49:18 +0000 UTC,LastTransitionTime:2023-03-05 16:20:46 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-03-05 16:49:18 +0000 UTC,LastTransitionTime:2023-03-05 16:20:46 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-03-05 16:49:18 +0000 UTC,LastTransitionTime:2023-03-05 16:21:47 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.49.247,},NodeAddress{Type:ExternalIP,Address:18.197.20.82,},NodeAddress{Type:InternalDNS,Address:i-0d6a32f749839192f.eu-central-1.compute.internal,},NodeAddress{Type:Hostname,Address:i-0d6a32f749839192f.eu-central-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-18-197-20-82.eu-central-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2bed249e8ac7b517cb08ccd42b83e5,SystemUUID:ec2bed24-9e8a-c7b5-17cb-08ccd42b83e5,BootID:6f9215c9-dfe8-4b13-84f4-9b2084456198,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 22.04.2 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.27.0-alpha.3.52+d48b8167f7094b,KubeProxyVersion:v1.27.0-alpha.3.52+d48b8167f7094b,OperatingSystem:linux,Architecture:arm64,},Images:[]ContainerImage{ContainerImage{Names:[registry.k8s.io/etcd@sha256:9ce33ba33d8e738a5b85ed50b5080ac746deceed4a7496c550927a7a19ca3b6d registry.k8s.io/etcd:3.5.0-0],SizeBytes:157797353,},ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:157636062,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:4afb99b4690b418ffc2ceb67e1a17376457e441c1f09ab55447f0aaf992fa646 registry.k8s.io/etcd:3.4.3-0],SizeBytes:151677080,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:4ad90a11b55313b182afc186b9876c8e891531b8db4c9bf1541953021618d0e2 registry.k8s.io/etcd:3.4.13-0],SizeBytes:134531559,},ContainerImage{Names:[registry.k8s.io/kube-apiserver-arm64:v1.27.0-alpha.3.52_d48b8167f7094b],SizeBytes:131087707,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:0d2ab6b15b4fe0b903b9e81192d942fb39c29be61fe341cd16db5d0165b0435c registry.k8s.io/etcd:3.3.17-0],SizeBytes:126979107,},ContainerImage{Names:[registry.k8s.io/kube-controller-manager-arm64:v1.27.0-alpha.3.52_d48b8167f7094b],SizeBytes:120667128,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:17da501f5d2a675be46040422a27b7cc21b8a43895ac998b171db1c346f361f7 registry.k8s.io/etcd:3.3.10-0],SizeBytes:81436097,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:6f72b851544986cb0921b53ea655ec04c36131248f16d4ad110cb3ca0c369dc1 registry.k8s.io/etcd:3.5.4-0],SizeBytes:81120118,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:13f53ed1d91e2e11aac476ee9a0269fdda6cc4874eba903efd40daf50c55eee5 registry.k8s.io/etcd:3.5.3-0],SizeBytes:81114891,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:51eae8381dcb1078289fa7b4f3df2630cdc18d09fb56f8e56b41c40e191d6c83 registry.k8s.io/etcd:3.5.7-0],SizeBytes:80665728,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:dd75ec974b0a2a6f6bb47001ba09207976e625db898d1b16735528c009cb171c registry.k8s.io/etcd:3.5.6-0],SizeBytes:80539316,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:905d7ca17fd02bc24c0eba9a062753aba15db3e31422390bc3238eb762339b20 registry.k8s.io/etcd:3.2.24-1],SizeBytes:61500433,},ContainerImage{Names:[registry.k8s.io/etcdadm/etcd-manager-slim@sha256:8727adc059334e0e3329dbccce116a734da4a090da5e86c8cddccaf333caa666 registry.k8s.io/etcdadm/etcd-manager-slim:v3.0.20230201],SizeBytes:60826655,},ContainerImage{Names:[registry.k8s.io/kube-proxy-arm64:v1.27.0-alpha.3.52_d48b8167f7094b],SizeBytes:60554082,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:64b9ea357325d5db9f8a723dcf503b5a449177b17ac87d69481e126bb724c263 registry.k8s.io/etcd:3.5.1-0],SizeBytes:59553904,},ContainerImage{Names:[registry.k8s.io/kube-scheduler-arm64:v1.27.0-alpha.3.52_d48b8167f7094b],SizeBytes:56310763,},ContainerImage{Names:[registry.k8s.io/kops/kops-controller:1.27.0-alpha.1],SizeBytes:41797411,},ContainerImage{Names:[registry.k8s.io/kops/dns-controller:1.27.0-alpha.1],SizeBytes:40267473,},ContainerImage{Names:[public.ecr.aws/karpenter/controller@sha256:68db4f092cf9cc83f5ef9e2fbc5407c2cb682e81f64dfaa700a7602ede38b1cf public.ecr.aws/karpenter/controller:v0.16.3],SizeBytes:33707751,},ContainerImage{Names:[public.ecr.aws/karpenter/webhook@sha256:96a2d9b06d6bc5127801f358f74b1cf2d289b423a2e9ba40c573c0b14b17dafa public.ecr.aws/karpenter/webhook:v0.16.3],SizeBytes:33307187,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:28932904,},ContainerImage{Names:[quay.io/cilium/operator@sha256:a6d24a006a6b92967ac90786b49bc1ac26e5477cf028cd1186efcfc2466484db quay.io/cilium/operator:v1.12.5],SizeBytes:24567287,},ContainerImage{Names:[registry.k8s.io/provider-aws/cloud-controller-manager@sha256:fdeb61e3e42ecd9cca868d550ebdb88dd6341d9e91fcfa9a37e227dab2ad22cb registry.k8s.io/provider-aws/cloud-controller-manager:v1.26.0],SizeBytes:18345226,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:8502345,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:7685305,},ContainerImage{Names:[registry.k8s.io/kops/kube-apiserver-healthcheck:1.27.0-alpha.1],SizeBytes:4713612,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:253553,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Mar 5 16:49:32.607: INFO: Logging kubelet events for node i-0d6a32f749839192f Mar 5 16:49:32.724: INFO: ... skipping 52 lines ... Mar 5 16:49:32.844: INFO: Container controller ready: true, restart count 0 Mar 5 16:49:33.343: INFO: Latency metrics for node i-0d6a32f749839192f [1mSTEP:[0m Destroying namespace "kubelet-1879" for this suite. [38;5;243m@ 03/05/23 16:49:33.343[0m [38;5;243m<< Timeline[0m [38;5;9m[FAILED] only 10 pods started out of 20[0m [38;5;9mIn [1m[It][0m[38;5;9m at: [1mtest/e2e/node/kubelet.go:350[0m [38;5;243m@ 03/05/23 16:45:49.394[0m [38;5;243m------------------------------[0m [38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;10m•[0m[38;5;10m•[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;10m•[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;10m•[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;10m•[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;10m•[0m[38;5;14mS[0m[38;5;10m•[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m [38;5;243m------------------------------[0m [38;5;9m• [FAILED] [732.103 seconds][0m [0m[sig-storage] ConfigMap [38;5;9m[1m[It] should be consumable from pods in volume with mappings [NodeConformance] [Conformance][0m [38;5;243mtest/e2e/common/storage/configmap_volume.go:89[0m [38;5;243mTimeline >>[0m [1mSTEP:[0m Creating a kubernetes client [38;5;243m@ 03/05/23 16:40:28.421[0m Mar 5 16:40:28.421: INFO: >>> kubeConfig: /root/.kube/config [1mSTEP:[0m Building a namespace api object, basename configmap [38;5;243m@ 03/05/23 16:40:28.422[0m [1mSTEP:[0m Waiting for a default service account to be provisioned in namespace [38;5;243m@ 03/05/23 16:40:28.759[0m [1mSTEP:[0m Waiting for kube-root-ca.crt to be provisioned in namespace [38;5;243m@ 03/05/23 16:40:28.98[0m [1mSTEP:[0m Creating configMap with name configmap-test-volume-map-346c6e6a-6afd-47ff-b371-f0e3b48532ff [38;5;243m@ 03/05/23 16:40:29.203[0m [1mSTEP:[0m Creating a pod to test consume configMaps [38;5;243m@ 03/05/23 16:40:29.321[0m Mar 5 16:46:17.392: INFO: Failed to get logs from node "i-06df568f6e5c59fff" pod "pod-configmaps-9d1782a6-9e31-4bba-844d-16673efe6cac" container "agnhost-container": the server could not find the requested resource (get pods pod-configmaps-9d1782a6-9e31-4bba-844d-16673efe6cac) [1mSTEP:[0m delete the pod [38;5;243m@ 03/05/23 16:46:17.392[0m Mar 5 16:49:17.511: INFO: Failed inside E2E framework: k8s.io/kubernetes/test/e2e/framework/pod.WaitForPodNotFoundInNamespace({0x7f3319d7bd68, 0xc003f5eb40}, {0x8252218?, 0xc0039cc1a0}, {0xc003a20ec0, 0x33}, {0xc0039065c0, 0xe}, 0x0?) test/e2e/framework/pod/wait.go:538 +0x190 k8s.io/kubernetes/test/e2e/framework/pod.(*PodClient).DeleteSync(0xc0001a7338, {0x7f3319d7bd68, 0xc003f5eb40}, {0xc003a20ec0, 0x33}, {{{0x0, 0x0}, {0x0, 0x0}}, 0x0, ...}, ...) test/e2e/framework/pod/pod_client.go:184 +0x1db k8s.io/kubernetes/test/e2e/framework/pod/output.MatchContainerOutput.func1() test/e2e/framework/pod/output/output.go:160 +0xc9 ... skipping 4 lines ... k8s.io/kubernetes/test/e2e/framework/pod/output.TestContainerOutputRegexp(...) test/e2e/framework/pod/output/output.go:222 k8s.io/kubernetes/test/e2e/common/storage.doConfigMapE2EWithMappings({0x7f3319d7bd68, 0xc003f5eb40}, 0xc000e72a50, 0x0, 0x0, 0x0) test/e2e/common/storage/configmap_volume.go:679 +0x6db k8s.io/kubernetes/test/e2e/common/storage.glob..func1.6({0x7f3319d7bd68?, 0xc003f5eb40?}) test/e2e/common/storage/configmap_volume.go:90 +0x2e [38;5;9m[FAILED][0m in [It] - test/e2e/framework/pod/pod_client.go:184 [38;5;243m@ 03/05/23 16:49:17.511[0m Mar 5 16:49:17.511: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready Mar 5 16:49:17.623: INFO: Condition Ready of node i-06df568f6e5c59fff is false, but Node is tainted by NodeController with [{node.kubernetes.io/not-ready NoSchedule 2023-03-05 16:46:33 +0000 UTC} {node.kubernetes.io/not-ready NoExecute 2023-03-05 16:46:37 +0000 UTC}]. Failure Mar 5 16:49:19.736: INFO: Condition Ready of node i-06df568f6e5c59fff is false, but Node is tainted by NodeController with [{node.kubernetes.io/not-ready NoSchedule 2023-03-05 16:46:33 +0000 UTC} {node.kubernetes.io/not-ready NoExecute 2023-03-05 16:46:37 +0000 UTC}]. Failure Mar 5 16:49:21.737: INFO: Condition Ready of node i-06df568f6e5c59fff is false, but Node is tainted by NodeController with [{node.kubernetes.io/not-ready NoSchedule 2023-03-05 16:46:33 +0000 UTC} {node.kubernetes.io/not-ready NoExecute 2023-03-05 16:46:37 +0000 UTC}]. Failure Mar 5 16:49:23.737: INFO: Condition Ready of node i-06df568f6e5c59fff is false, but Node is tainted by NodeController with [{node.kubernetes.io/not-ready NoSchedule 2023-03-05 16:46:33 +0000 UTC} {node.kubernetes.io/not-ready NoExecute 2023-03-05 16:46:37 +0000 UTC}]. Failure Mar 5 16:49:25.737: INFO: Condition Ready of node i-06df568f6e5c59fff is false, but Node is tainted by NodeController with [{node.kubernetes.io/not-ready NoSchedule 2023-03-05 16:46:33 +0000 UTC} {node.kubernetes.io/not-ready NoExecute 2023-03-05 16:46:37 +0000 UTC}]. Failure ... skipping 90 lines ... Mar 5 16:52:17.961: INFO: At 2023-03-05 16:40:29 +0000 UTC - event for pod-configmaps-9d1782a6-9e31-4bba-844d-16673efe6cac: {default-scheduler } Scheduled: Successfully assigned configmap-1701/pod-configmaps-9d1782a6-9e31-4bba-844d-16673efe6cac to i-06df568f6e5c59fff Mar 5 16:52:17.961: INFO: At 2023-03-05 16:41:51 +0000 UTC - event for pod-configmaps-9d1782a6-9e31-4bba-844d-16673efe6cac: {taint-controller } TaintManagerEviction: Cancelling deletion of Pod configmap-1701/pod-configmaps-9d1782a6-9e31-4bba-844d-16673efe6cac Mar 5 16:52:17.961: INFO: At 2023-03-05 16:47:57 +0000 UTC - event for pod-configmaps-9d1782a6-9e31-4bba-844d-16673efe6cac: {taint-controller } TaintManagerEviction: Marking for deletion Pod configmap-1701/pod-configmaps-9d1782a6-9e31-4bba-844d-16673efe6cac Mar 5 16:52:18.073: INFO: POD NODE PHASE GRACE CONDITIONS Mar 5 16:52:18.073: INFO: pod-configmaps-9d1782a6-9e31-4bba-844d-16673efe6cac i-06df568f6e5c59fff Pending 1s [{PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2023-03-05 16:40:29 +0000 UTC } {DisruptionTarget True 0001-01-01 00:00:00 +0000 UTC 2023-03-05 16:47:57 +0000 UTC DeletionByTaintManager Taint manager: deleting due to NoExecute taint}] Mar 5 16:52:18.073: INFO: Mar 5 16:52:28.187: INFO: Unable to fetch configmap-1701/pod-configmaps-9d1782a6-9e31-4bba-844d-16673efe6cac/agnhost-container logs: an error on the server ("unknown") has prevented the request from succeeding (get pods pod-configmaps-9d1782a6-9e31-4bba-844d-16673efe6cac) Mar 5 16:52:28.300: INFO: Logging node info for node i-056ad61525499081e Mar 5 16:52:28.411: INFO: Node Info: &Node{ObjectMeta:{i-056ad61525499081e a6ce9ed3-5ae8-4065-b117-6884e24b3632 19980 0 2023-03-05 16:24:16 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:arm64 beta.kubernetes.io/instance-type:t4g.small beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:eu-central-1 failure-domain.beta.kubernetes.io/zone:eu-central-1a karpenter.k8s.aws/instance-category:t karpenter.k8s.aws/instance-cpu:2 karpenter.k8s.aws/instance-family:t4g karpenter.k8s.aws/instance-generation:4 karpenter.k8s.aws/instance-hypervisor:nitro karpenter.k8s.aws/instance-memory:2048 karpenter.k8s.aws/instance-pods:11 karpenter.k8s.aws/instance-size:small karpenter.sh/capacity-type:spot karpenter.sh/initialized:true karpenter.sh/provisioner-name:nodes kubernetes.io/arch:arm64 kubernetes.io/hostname:i-056ad61525499081e kubernetes.io/os:linux node-role.kubernetes.io/node: node-role.kubernetes.io/spot-worker:true node.kubernetes.io/instance-type:t4g.small topology.ebs.csi.aws.com/zone:eu-central-1a topology.hostpath.csi/node:i-056ad61525499081e topology.kubernetes.io/region:eu-central-1 topology.kubernetes.io/zone:eu-central-1a] map[csi.volume.kubernetes.io/nodeid:{"csi-hostpath-read-write-once-pod-5175":"i-056ad61525499081e","csi-mock-csi-mock-volumes-expansion-4933":"csi-mock-csi-mock-volumes-expansion-4933","csi-mock-csi-mock-volumes-fsgroup-policy-6623":"i-056ad61525499081e","ebs.csi.aws.com":"i-056ad61525499081e"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [{karpenter.sh/v1alpha5 Provisioner nodes ac5d7599-b2a2-4eae-8da4-1f4c0a54ec82 <nil> 0xc0035cf440}] [karpenter.sh/termination] [{aws-cloud-controller-manager Update v1 2023-03-05 16:24:16 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{}}}} } {kops-controller Update v1 2023-03-05 16:24:16 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{},"f:node-role.kubernetes.io/spot-worker":{}}}} } {karpenter Update v1 2023-03-05 16:25:57 +0000 UTC FieldsV1 {"f:metadata":{"f:finalizers":{".":{},"v:\"karpenter.sh/termination\"":{}},"f:labels":{".":{},"f:karpenter.k8s.aws/instance-category":{},"f:karpenter.k8s.aws/instance-cpu":{},"f:karpenter.k8s.aws/instance-family":{},"f:karpenter.k8s.aws/instance-generation":{},"f:karpenter.k8s.aws/instance-hypervisor":{},"f:karpenter.k8s.aws/instance-memory":{},"f:karpenter.k8s.aws/instance-pods":{},"f:karpenter.k8s.aws/instance-size":{},"f:karpenter.sh/capacity-type":{},"f:karpenter.sh/initialized":{},"f:karpenter.sh/provisioner-name":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}},"f:ownerReferences":{".":{},"k:{\"uid\":\"ac5d7599-b2a2-4eae-8da4-1f4c0a54ec82\"}":{}}},"f:spec":{"f:providerID":{}}} } {kube-controller-manager Update v1 2023-03-05 16:25:57 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:node.alpha.kubernetes.io/ttl":{}},"f:labels":{"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.1.0/24\"":{}}}} } {aws-cloud-controller-manager Update v1 2023-03-05 16:26:48 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{".":{},"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"InternalIP\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kube-controller-manager Update v1 2023-03-05 16:52:18 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{".":{},"k:{\"type\":\"DiskPressure\"}":{".":{},"f:type":{}},"k:{\"type\":\"MemoryPressure\"}":{".":{},"f:type":{}},"k:{\"type\":\"PIDPressure\"}":{".":{},"f:type":{}},"k:{\"type\":\"Ready\"}":{".":{},"f:type":{}}},"f:volumesAttached":{}}} status} {kubelet Update v1 2023-03-05 16:52:21 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{"f:kubernetes.io/hostname":{},"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{".":{},"f:cpu":{},"f:ephemeral-storage":{},"f:hugepages-1Gi":{},"f:hugepages-2Mi":{},"f:hugepages-32Mi":{},"f:hugepages-64Ki":{},"f:memory":{},"f:pods":{}},"f:capacity":{".":{},"f:cpu":{},"f:ephemeral-storage":{},"f:hugepages-1Gi":{},"f:hugepages-2Mi":{},"f:hugepages-32Mi":{},"f:hugepages-64Ki":{},"f:memory":{},"f:pods":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:daemonEndpoints":{"f:kubeletEndpoint":{"f:Port":{}}},"f:images":{},"f:nodeInfo":{"f:architecture":{},"f:bootID":{},"f:containerRuntimeVersion":{},"f:kernelVersion":{},"f:kubeProxyVersion":{},"f:kubeletVersion":{},"f:machineID":{},"f:operatingSystem":{},"f:osImage":{},"f:systemUUID":{}},"f:volumesInUse":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.1.0/24,DoNotUseExternalID:,ProviderID:aws:///eu-central-1a/i-056ad61525499081e,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.1.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49756430336 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-32Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-64Ki: {{0 0} {<nil>} 0 DecimalSI},memory: {{1941540864 0} {<nil>} 1896036Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44780787229 0} {<nil>} 44780787229 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-32Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-64Ki: {{0 0} {<nil>} 0 DecimalSI},memory: {{1836683264 0} {<nil>} 1793636Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-03-05 16:52:21 +0000 UTC,LastTransitionTime:2023-03-05 16:25:57 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-03-05 16:52:21 +0000 UTC,LastTransitionTime:2023-03-05 16:25:48 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-03-05 16:52:21 +0000 UTC,LastTransitionTime:2023-03-05 16:25:48 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-03-05 16:52:21 +0000 UTC,LastTransitionTime:2023-03-05 16:25:48 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.36.179,},NodeAddress{Type:ExternalIP,Address:18.159.37.200,},NodeAddress{Type:InternalDNS,Address:i-056ad61525499081e.eu-central-1.compute.internal,},NodeAddress{Type:Hostname,Address:i-056ad61525499081e.eu-central-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-18-159-37-200.eu-central-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2af3207d1c8c09785c337aae6c6ef0,SystemUUID:ec2af320-7d1c-8c09-785c-337aae6c6ef0,BootID:786f2752-bc30-4729-8aaf-45977b16c253,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 22.04.2 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.27.0-alpha.3.52+d48b8167f7094b,KubeProxyVersion:v1.27.0-alpha.3.52+d48b8167f7094b,OperatingSystem:linux,Architecture:arm64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:157636062,},ContainerImage{Names:[registry.k8s.io/kube-proxy-arm64:v1.27.0-alpha.3.52_d48b8167f7094b],SizeBytes:60554082,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:16bbf38c463a4223d8cfe4da12bc61010b082a79b4bb003e2d3ba3ece5dd5f9e registry.k8s.io/e2e-test-images/agnhost:2.43],SizeBytes:49808364,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:3fe7acf013d1264ffded116b80a73dc129a449b0fccdb8d21af8279f2233f36e registry.k8s.io/e2e-test-images/httpd:2.4.39-4],SizeBytes:41528818,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:148b022f5c5da426fc2f3c14b5c0867e58ef05961510c84749ac1fddcb0fef22 registry.k8s.io/e2e-test-images/httpd:2.4.38-4],SizeBytes:40343601,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:28932904,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:ee3b525d5b89db99da3b8eb521d9cd90cb6e9ef0fbb651e98bb37be78d36b5b8 registry.k8s.io/sig-storage/csi-provisioner:v3.3.0],SizeBytes:23945474,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:425d8f1b769398127767b06ed97ce62578a3179bcb99809ce93a1649e025ffe7 registry.k8s.io/sig-storage/csi-resizer:v1.6.0],SizeBytes:22678632,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:291334908ddf71a4661fd7f6d9d97274de8a5378a2b6fdfeb2ce73414a34f82f registry.k8s.io/sig-storage/csi-snapshotter:v6.1.0],SizeBytes:22434180,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:9a685020911e2725ad019dbce6e4a5ab93d51e3d4557f115e64343345e05781b registry.k8s.io/sig-storage/csi-attacher:v4.0.0],SizeBytes:22394501,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:122bfb8c1edabb3c0edd63f06523e6940d958d19b3957dc7b1d6f81e9f1f6119 registry.k8s.io/sig-storage/csi-provisioner:v3.1.0],SizeBytes:21735644,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:9ebbf9f023e7b41ccee3d52afe39a89e3ddacdbb69269d583abfc25847cfd9e4 registry.k8s.io/sig-storage/csi-resizer:v1.4.0],SizeBytes:20848413,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:8b9c313c05f54fb04f8d430896f5f5904b6cb157df261501b29adc04d2b2dc7b registry.k8s.io/sig-storage/csi-attacher:v3.4.0],SizeBytes:20565480,},ContainerImage{Names:[registry.k8s.io/cpa/cluster-proportional-autoscaler@sha256:68d396900aeaa072c1f27289485fdac29834045a6f3ffe369bf389d830ef572d registry.k8s.io/cpa/cluster-proportional-autoscaler:1.8.6],SizeBytes:19007623,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:92257881c1d6493cf18299a24af42330f891166560047902b8d431fb66b01af5 registry.k8s.io/sig-storage/hostpathplugin:v1.9.0],SizeBytes:18267611,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:9b2d9abda017c28b12f84a344f57cd73fbdb6c2bd7dd5bdd5018246ad1093ba6 registry.k8s.io/sig-storage/hostpathplugin:v1.11.0],SizeBytes:18159085,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:017727efcfeb7d053af68e51436ce8e65edbc6ca573720afb4f79c8594036955 registry.k8s.io/coredns/coredns:v1.10.0],SizeBytes:13766234,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:8502345,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:933940f13b3ea0abc62e656c1aa5c5b47c04b15d71250413a6b821bd0c58b94e registry.k8s.io/sig-storage/livenessprobe:v2.7.0],SizeBytes:8223819,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:7685305,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:5c99cf6a02adda929b10321dbf4ecfa00d87be9ba4fb456006237d530ab4baa1 registry.k8s.io/e2e-test-images/nginx:1.14-4],SizeBytes:6860089,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf registry.k8s.io/e2e-test-images/busybox:1.29-2],SizeBytes:773367,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:2e0f836850e09b8b7cc937681d6194537a09fbd5f6b9e08f4d646a85128e8937 registry.k8s.io/e2e-test-images/busybox:1.29-4],SizeBytes:772933,},ContainerImage{Names:[registry.k8s.io/pause@sha256:7031c1b283388d2c2e09b57badb803c05ebed362dc88d84b480cc47f72a21097 registry.k8s.io/pause:3.9],SizeBytes:268051,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:253553,},},VolumesInUse:[kubernetes.io/csi/csi-hostpath-read-write-once-pod-5175^bc2b4edf-bb75-11ed-8d33-a27fee729fe2 kubernetes.io/csi/csi-mock-csi-mock-volumes-expansion-4933^4 kubernetes.io/csi/ebs.csi.aws.com^vol-0e020446b1cd6abaf],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/csi-mock-csi-mock-volumes-expansion-4933^4,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-0e020446b1cd6abaf,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/csi-hostpath-read-write-once-pod-5175^bc2b4edf-bb75-11ed-8d33-a27fee729fe2,DevicePath:,},},Config:nil,},} Mar 5 16:52:28.412: INFO: Logging kubelet events for node i-056ad61525499081e Mar 5 16:52:28.527: INFO: ... skipping 61 lines ... Logging node info for node i-06df568f6e5c59fff Mar 5 16:52:29.347: INFO: Node Info: &Node{ObjectMeta:{i-06df568f6e5c59fff cb63c681-2e18-474a-b6f3-221325ac0d52 18792 0 2023-03-05 16:24:16 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:arm64 beta.kubernetes.io/instance-type:t4g.small beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:eu-central-1 failure-domain.beta.kubernetes.io/zone:eu-central-1a karpenter.k8s.aws/instance-category:t karpenter.k8s.aws/instance-cpu:2 karpenter.k8s.aws/instance-family:t4g karpenter.k8s.aws/instance-generation:4 karpenter.k8s.aws/instance-hypervisor:nitro karpenter.k8s.aws/instance-memory:2048 karpenter.k8s.aws/instance-pods:11 karpenter.k8s.aws/instance-size:small karpenter.sh/capacity-type:spot karpenter.sh/initialized:true karpenter.sh/provisioner-name:nodes kubernetes.io/arch:arm64 kubernetes.io/hostname:i-06df568f6e5c59fff kubernetes.io/os:linux node-role.kubernetes.io/node: node-role.kubernetes.io/spot-worker:true node.kubernetes.io/instance-type:t4g.small topology.ebs.csi.aws.com/zone:eu-central-1a topology.hostpath.csi/node:i-06df568f6e5c59fff topology.kubernetes.io/region:eu-central-1 topology.kubernetes.io/zone:eu-central-1a] map[csi.volume.kubernetes.io/nodeid:{"csi-hostpath-ephemeral-2147":"i-06df568f6e5c59fff","csi-hostpath-ephemeral-5408":"i-06df568f6e5c59fff","csi-hostpath-ephemeral-6540":"i-06df568f6e5c59fff","csi-mock-csi-mock-volumes-fsgroup-policy-1058":"i-06df568f6e5c59fff","csi-mock-csi-mock-volumes-workload-2219":"i-06df568f6e5c59fff","ebs.csi.aws.com":"i-06df568f6e5c59fff"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [{karpenter.sh/v1alpha5 Provisioner nodes ac5d7599-b2a2-4eae-8da4-1f4c0a54ec82 <nil> 0xc004ca9110}] [karpenter.sh/termination] [{aws-cloud-controller-manager Update v1 2023-03-05 16:24:16 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{}}}} } {kops-controller Update v1 2023-03-05 16:24:16 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{},"f:node-role.kubernetes.io/spot-worker":{}}}} } {karpenter Update v1 2023-03-05 16:25:57 +0000 UTC FieldsV1 {"f:metadata":{"f:finalizers":{".":{},"v:\"karpenter.sh/termination\"":{}},"f:labels":{".":{},"f:karpenter.k8s.aws/instance-category":{},"f:karpenter.k8s.aws/instance-cpu":{},"f:karpenter.k8s.aws/instance-family":{},"f:karpenter.k8s.aws/instance-generation":{},"f:karpenter.k8s.aws/instance-hypervisor":{},"f:karpenter.k8s.aws/instance-memory":{},"f:karpenter.k8s.aws/instance-pods":{},"f:karpenter.k8s.aws/instance-size":{},"f:karpenter.sh/capacity-type":{},"f:karpenter.sh/initialized":{},"f:karpenter.sh/provisioner-name":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}},"f:ownerReferences":{".":{},"k:{\"uid\":\"ac5d7599-b2a2-4eae-8da4-1f4c0a54ec82\"}":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-03-05 16:26:48 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{".":{},"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"InternalIP\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kubelet Update v1 2023-03-05 16:48:40 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{"f:kubernetes.io/hostname":{},"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{".":{},"f:cpu":{},"f:ephemeral-storage":{},"f:hugepages-1Gi":{},"f:hugepages-2Mi":{},"f:hugepages-32Mi":{},"f:hugepages-64Ki":{},"f:memory":{},"f:pods":{}},"f:capacity":{".":{},"f:cpu":{},"f:ephemeral-storage":{},"f:hugepages-1Gi":{},"f:hugepages-2Mi":{},"f:hugepages-32Mi":{},"f:hugepages-64Ki":{},"f:memory":{},"f:pods":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{}}},"f:daemonEndpoints":{"f:kubeletEndpoint":{"f:Port":{}}},"f:images":{},"f:nodeInfo":{"f:architecture":{},"f:bootID":{},"f:containerRuntimeVersion":{},"f:kernelVersion":{},"f:kubeProxyVersion":{},"f:kubeletVersion":{},"f:machineID":{},"f:operatingSystem":{},"f:osImage":{},"f:systemUUID":{}},"f:volumesInUse":{}}} status} {kube-controller-manager Update v1 2023-03-05 16:49:37 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{".":{},"k:{\"type\":\"DiskPressure\"}":{".":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"MemoryPressure\"}":{".":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"PIDPressure\"}":{".":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Ready\"}":{".":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}},"f:volumesAttached":{}}} status} {kube-controller-manager Update v1 2023-03-05 16:49:42 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:node.alpha.kubernetes.io/ttl":{}},"f:labels":{"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.2.0/24\"":{}},"f:taints":{}}} }]},Spec:NodeSpec{PodCIDR:100.96.2.0/24,DoNotUseExternalID:,ProviderID:aws:///eu-central-1a/i-06df568f6e5c59fff,Unschedulable:false,Taints:[]Taint{Taint{Key:node.kubernetes.io/unreachable,Value:,Effect:NoSchedule,TimeAdded:2023-03-05 16:49:37 +0000 UTC,},Taint{Key:node.kubernetes.io/unreachable,Value:,Effect:NoExecute,TimeAdded:2023-03-05 16:49:42 +0000 UTC,},},ConfigSource:nil,PodCIDRs:[100.96.2.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49756430336 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-32Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-64Ki: {{0 0} {<nil>} 0 DecimalSI},memory: {{1941540864 0} {<nil>} 1896036Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44780787229 0} {<nil>} 44780787229 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-32Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-64Ki: {{0 0} {<nil>} 0 DecimalSI},memory: {{1836683264 0} {<nil>} 1793636Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:Ready,Status:Unknown,LastHeartbeatTime:2023-03-05 16:47:34 +0000 UTC,LastTransitionTime:2023-03-05 16:49:37 +0000 UTC,Reason:NodeStatusUnknown,Message:Kubelet stopped posting node status.,},NodeCondition{Type:MemoryPressure,Status:Unknown,LastHeartbeatTime:2023-03-05 16:47:34 +0000 UTC,LastTransitionTime:2023-03-05 16:49:37 +0000 UTC,Reason:NodeStatusUnknown,Message:Kubelet stopped posting node status.,},NodeCondition{Type:DiskPressure,Status:Unknown,LastHeartbeatTime:2023-03-05 16:47:34 +0000 UTC,LastTransitionTime:2023-03-05 16:49:37 +0000 UTC,Reason:NodeStatusUnknown,Message:Kubelet stopped posting node status.,},NodeCondition{Type:PIDPressure,Status:Unknown,LastHeartbeatTime:2023-03-05 16:47:34 +0000 UTC,LastTransitionTime:2023-03-05 16:49:37 +0000 UTC,Reason:NodeStatusUnknown,Message:Kubelet stopped posting node status.,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.43.148,},NodeAddress{Type:ExternalIP,Address:18.192.211.222,},NodeAddress{Type:InternalDNS,Address:i-06df568f6e5c59fff.eu-central-1.compute.internal,},NodeAddress{Type:Hostname,Address:i-06df568f6e5c59fff.eu-central-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-18-192-211-222.eu-central-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec221221864d9409a8ec5f6614a9b94b,SystemUUID:ec221221-864d-9409-a8ec-5f6614a9b94b,BootID:553d31bf-faca-417e-a3db-325839e1b8e4,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 22.04.2 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.27.0-alpha.3.52+d48b8167f7094b,KubeProxyVersion:v1.27.0-alpha.3.52+d48b8167f7094b,OperatingSystem:linux,Architecture:arm64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:157636062,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/jessie-dnsutils@sha256:24aaf2626d6b27864c29de2097e8bbb840b3a414271bf7c8995e431e47d8408e registry.k8s.io/e2e-test-images/jessie-dnsutils:1.7],SizeBytes:116764683,},ContainerImage{Names:[registry.k8s.io/kube-proxy-arm64:v1.27.0-alpha.3.52_d48b8167f7094b],SizeBytes:60554082,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:16bbf38c463a4223d8cfe4da12bc61010b082a79b4bb003e2d3ba3ece5dd5f9e registry.k8s.io/e2e-test-images/agnhost:2.43],SizeBytes:49808364,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:3fe7acf013d1264ffded116b80a73dc129a449b0fccdb8d21af8279f2233f36e registry.k8s.io/e2e-test-images/httpd:2.4.39-4],SizeBytes:41528818,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:148b022f5c5da426fc2f3c14b5c0867e58ef05961510c84749ac1fddcb0fef22 registry.k8s.io/e2e-test-images/httpd:2.4.38-4],SizeBytes:40343601,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:28932904,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:ee3b525d5b89db99da3b8eb521d9cd90cb6e9ef0fbb651e98bb37be78d36b5b8 registry.k8s.io/sig-storage/csi-provisioner:v3.3.0],SizeBytes:23945474,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:425d8f1b769398127767b06ed97ce62578a3179bcb99809ce93a1649e025ffe7 registry.k8s.io/sig-storage/csi-resizer:v1.6.0],SizeBytes:22678632,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:291334908ddf71a4661fd7f6d9d97274de8a5378a2b6fdfeb2ce73414a34f82f registry.k8s.io/sig-storage/csi-snapshotter:v6.1.0],SizeBytes:22434180,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:9a685020911e2725ad019dbce6e4a5ab93d51e3d4557f115e64343345e05781b registry.k8s.io/sig-storage/csi-attacher:v4.0.0],SizeBytes:22394501,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:122bfb8c1edabb3c0edd63f06523e6940d958d19b3957dc7b1d6f81e9f1f6119 registry.k8s.io/sig-storage/csi-provisioner:v3.1.0],SizeBytes:21735644,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:9ebbf9f023e7b41ccee3d52afe39a89e3ddacdbb69269d583abfc25847cfd9e4 registry.k8s.io/sig-storage/csi-resizer:v1.4.0],SizeBytes:20848413,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:8b9c313c05f54fb04f8d430896f5f5904b6cb157df261501b29adc04d2b2dc7b registry.k8s.io/sig-storage/csi-attacher:v3.4.0],SizeBytes:20565480,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nonroot@sha256:ee9f50b3c64b174d296d91ca9f69a914ac30e59095dfb462b2b518ad28a63655 registry.k8s.io/e2e-test-images/nonroot:1.4],SizeBytes:18561972,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:92257881c1d6493cf18299a24af42330f891166560047902b8d431fb66b01af5 registry.k8s.io/sig-storage/hostpathplugin:v1.9.0],SizeBytes:18267611,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:9b2d9abda017c28b12f84a344f57cd73fbdb6c2bd7dd5bdd5018246ad1093ba6 registry.k8s.io/sig-storage/hostpathplugin:v1.11.0],SizeBytes:18159085,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:017727efcfeb7d053af68e51436ce8e65edbc6ca573720afb4f79c8594036955 registry.k8s.io/coredns/coredns:v1.10.0],SizeBytes:13766234,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:8502345,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:933940f13b3ea0abc62e656c1aa5c5b47c04b15d71250413a6b821bd0c58b94e registry.k8s.io/sig-storage/livenessprobe:v2.7.0],SizeBytes:8223819,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:7685305,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:5c99cf6a02adda929b10321dbf4ecfa00d87be9ba4fb456006237d530ab4baa1 registry.k8s.io/e2e-test-images/nginx:1.14-4],SizeBytes:6860089,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nonewprivs@sha256:8ac1264691820febacf3aea5d152cbde6d10685731ec14966a9401c6f47a68ac registry.k8s.io/e2e-test-images/nonewprivs:1.3],SizeBytes:5025027,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:2e0f836850e09b8b7cc937681d6194537a09fbd5f6b9e08f4d646a85128e8937 registry.k8s.io/e2e-test-images/busybox:1.29-4],SizeBytes:772933,},ContainerImage{Names:[registry.k8s.io/pause@sha256:7031c1b283388d2c2e09b57badb803c05ebed362dc88d84b480cc47f72a21097 registry.k8s.io/pause:3.9],SizeBytes:268051,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:253553,},},VolumesInUse:[kubernetes.io/csi/csi-hostpath-ephemeral-2147^52df5953-bb73-11ed-8aa7-be49170871b4 kubernetes.io/csi/csi-hostpath-ephemeral-5408^443d1daf-bb73-11ed-95c4-5a95d17cbc27 kubernetes.io/csi/csi-hostpath-ephemeral-5408^44411f51-bb73-11ed-95c4-5a95d17cbc27 kubernetes.io/csi/csi-hostpath-ephemeral-6540^4426a07f-bb73-11ed-a604-92687f2c50f8 kubernetes.io/csi/ebs.csi.aws.com^vol-05b67fab5beb6394b],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-05b67fab5beb6394b,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/csi-hostpath-ephemeral-6540^4426a07f-bb73-11ed-a604-92687f2c50f8,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/csi-hostpath-ephemeral-5408^44411f51-bb73-11ed-95c4-5a95d17cbc27,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/csi-hostpath-ephemeral-5408^443d1daf-bb73-11ed-95c4-5a95d17cbc27,DevicePath:,},},Config:nil,},} Mar 5 16:52:29.348: INFO: Logging kubelet events for node i-06df568f6e5c59fff Mar 5 16:52:29.463: INFO: Logging pods the kubelet thinks is on node i-06df568f6e5c59fff Mar 5 16:52:39.575: INFO: Unable to retrieve kubelet pods for node i-06df568f6e5c59fff: error trying to reach service: net/http: TLS handshake timeout Mar 5 16:52:39.575: INFO: Logging node info for node i-0d6a32f749839192f Mar 5 16:52:39.687: INFO: Node Info: &Node{ObjectMeta:{i-0d6a32f749839192f 4155fefb-4d3a-4b04-ba53-6ac9e895ff92 18509 0 2023-03-05 16:20:46 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:arm64 beta.kubernetes.io/instance-type:c6g.xlarge beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:eu-central-1 failure-domain.beta.kubernetes.io/zone:eu-central-1a kops.k8s.io/kops-controller-pki: kubernetes.io/arch:arm64 kubernetes.io/hostname:i-0d6a32f749839192f kubernetes.io/os:linux node-role.kubernetes.io/control-plane: node.kubernetes.io/exclude-from-external-load-balancers: node.kubernetes.io/instance-type:c6g.xlarge topology.ebs.csi.aws.com/zone:eu-central-1a topology.kubernetes.io/region:eu-central-1 topology.kubernetes.io/zone:eu-central-1a] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-0d6a32f749839192f"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kubelet Update v1 2023-03-05 16:20:46 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {protokube Update v1 2023-03-05 16:21:25 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kops.k8s.io/kops-controller-pki":{},"f:node-role.kubernetes.io/control-plane":{},"f:node.kubernetes.io/exclude-from-external-load-balancers":{}}}} } {kube-controller-manager Update v1 2023-03-05 16:21:47 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.0.0/24\"":{}}}} } {aws-cloud-controller-manager Update v1 2023-03-05 16:21:48 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{},"f:taints":{}}} } {aws-cloud-controller-manager Update v1 2023-03-05 16:21:48 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kubelet Update v1 2023-03-05 16:49:18 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.0.0/24,DoNotUseExternalID:,ProviderID:aws:///eu-central-1a/i-0d6a32f749839192f,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/control-plane,Value:,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[100.96.0.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{4 0} {<nil>} 4 DecimalSI},ephemeral-storage: {{49756430336 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-32Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-64Ki: {{0 0} {<nil>} 0 DecimalSI},memory: {{8150118400 0} {<nil>} 7959100Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{4 0} {<nil>} 4 DecimalSI},ephemeral-storage: {{44780787229 0} {<nil>} 44780787229 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-32Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-64Ki: {{0 0} {<nil>} 0 DecimalSI},memory: {{8045260800 0} {<nil>} 7856700Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-03-05 16:49:18 +0000 UTC,LastTransitionTime:2023-03-05 16:20:46 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-03-05 16:49:18 +0000 UTC,LastTransitionTime:2023-03-05 16:20:46 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-03-05 16:49:18 +0000 UTC,LastTransitionTime:2023-03-05 16:20:46 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-03-05 16:49:18 +0000 UTC,LastTransitionTime:2023-03-05 16:21:47 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.49.247,},NodeAddress{Type:ExternalIP,Address:18.197.20.82,},NodeAddress{Type:InternalDNS,Address:i-0d6a32f749839192f.eu-central-1.compute.internal,},NodeAddress{Type:Hostname,Address:i-0d6a32f749839192f.eu-central-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-18-197-20-82.eu-central-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2bed249e8ac7b517cb08ccd42b83e5,SystemUUID:ec2bed24-9e8a-c7b5-17cb-08ccd42b83e5,BootID:6f9215c9-dfe8-4b13-84f4-9b2084456198,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 22.04.2 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.27.0-alpha.3.52+d48b8167f7094b,KubeProxyVersion:v1.27.0-alpha.3.52+d48b8167f7094b,OperatingSystem:linux,Architecture:arm64,},Images:[]ContainerImage{ContainerImage{Names:[registry.k8s.io/etcd@sha256:9ce33ba33d8e738a5b85ed50b5080ac746deceed4a7496c550927a7a19ca3b6d registry.k8s.io/etcd:3.5.0-0],SizeBytes:157797353,},ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:157636062,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:4afb99b4690b418ffc2ceb67e1a17376457e441c1f09ab55447f0aaf992fa646 registry.k8s.io/etcd:3.4.3-0],SizeBytes:151677080,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:4ad90a11b55313b182afc186b9876c8e891531b8db4c9bf1541953021618d0e2 registry.k8s.io/etcd:3.4.13-0],SizeBytes:134531559,},ContainerImage{Names:[registry.k8s.io/kube-apiserver-arm64:v1.27.0-alpha.3.52_d48b8167f7094b],SizeBytes:131087707,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:0d2ab6b15b4fe0b903b9e81192d942fb39c29be61fe341cd16db5d0165b0435c registry.k8s.io/etcd:3.3.17-0],SizeBytes:126979107,},ContainerImage{Names:[registry.k8s.io/kube-controller-manager-arm64:v1.27.0-alpha.3.52_d48b8167f7094b],SizeBytes:120667128,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:17da501f5d2a675be46040422a27b7cc21b8a43895ac998b171db1c346f361f7 registry.k8s.io/etcd:3.3.10-0],SizeBytes:81436097,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:6f72b851544986cb0921b53ea655ec04c36131248f16d4ad110cb3ca0c369dc1 registry.k8s.io/etcd:3.5.4-0],SizeBytes:81120118,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:13f53ed1d91e2e11aac476ee9a0269fdda6cc4874eba903efd40daf50c55eee5 registry.k8s.io/etcd:3.5.3-0],SizeBytes:81114891,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:51eae8381dcb1078289fa7b4f3df2630cdc18d09fb56f8e56b41c40e191d6c83 registry.k8s.io/etcd:3.5.7-0],SizeBytes:80665728,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:dd75ec974b0a2a6f6bb47001ba09207976e625db898d1b16735528c009cb171c registry.k8s.io/etcd:3.5.6-0],SizeBytes:80539316,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:905d7ca17fd02bc24c0eba9a062753aba15db3e31422390bc3238eb762339b20 registry.k8s.io/etcd:3.2.24-1],SizeBytes:61500433,},ContainerImage{Names:[registry.k8s.io/etcdadm/etcd-manager-slim@sha256:8727adc059334e0e3329dbccce116a734da4a090da5e86c8cddccaf333caa666 registry.k8s.io/etcdadm/etcd-manager-slim:v3.0.20230201],SizeBytes:60826655,},ContainerImage{Names:[registry.k8s.io/kube-proxy-arm64:v1.27.0-alpha.3.52_d48b8167f7094b],SizeBytes:60554082,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:64b9ea357325d5db9f8a723dcf503b5a449177b17ac87d69481e126bb724c263 registry.k8s.io/etcd:3.5.1-0],SizeBytes:59553904,},ContainerImage{Names:[registry.k8s.io/kube-scheduler-arm64:v1.27.0-alpha.3.52_d48b8167f7094b],SizeBytes:56310763,},ContainerImage{Names:[registry.k8s.io/kops/kops-controller:1.27.0-alpha.1],SizeBytes:41797411,},ContainerImage{Names:[registry.k8s.io/kops/dns-controller:1.27.0-alpha.1],SizeBytes:40267473,},ContainerImage{Names:[public.ecr.aws/karpenter/controller@sha256:68db4f092cf9cc83f5ef9e2fbc5407c2cb682e81f64dfaa700a7602ede38b1cf public.ecr.aws/karpenter/controller:v0.16.3],SizeBytes:33707751,},ContainerImage{Names:[public.ecr.aws/karpenter/webhook@sha256:96a2d9b06d6bc5127801f358f74b1cf2d289b423a2e9ba40c573c0b14b17dafa public.ecr.aws/karpenter/webhook:v0.16.3],SizeBytes:33307187,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:28932904,},ContainerImage{Names:[quay.io/cilium/operator@sha256:a6d24a006a6b92967ac90786b49bc1ac26e5477cf028cd1186efcfc2466484db quay.io/cilium/operator:v1.12.5],SizeBytes:24567287,},ContainerImage{Names:[registry.k8s.io/provider-aws/cloud-controller-manager@sha256:fdeb61e3e42ecd9cca868d550ebdb88dd6341d9e91fcfa9a37e227dab2ad22cb registry.k8s.io/provider-aws/cloud-controller-manager:v1.26.0],SizeBytes:18345226,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:8502345,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:7685305,},ContainerImage{Names:[registry.k8s.io/kops/kube-apiserver-healthcheck:1.27.0-alpha.1],SizeBytes:4713612,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:253553,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Mar 5 16:52:39.687: INFO: Logging kubelet events for node i-0d6a32f749839192f Mar 5 16:52:39.802: INFO: ... skipping 52 lines ... Mar 5 16:52:39.921: INFO: Container kube-scheduler ready: true, restart count 0 Mar 5 16:52:40.410: INFO: Latency metrics for node i-0d6a32f749839192f [1mSTEP:[0m Destroying namespace "configmap-1701" for this suite. [38;5;243m@ 03/05/23 16:52:40.411[0m [38;5;243m<< Timeline[0m [38;5;9m[FAILED] wait for pod "pod-configmaps-9d1782a6-9e31-4bba-844d-16673efe6cac" to disappear: expected pod to not be found: Timed out after 180.001s. Expected <*v1.Pod | 0xc003bc2d80>: metadata: creationTimestamp: "2023-03-05T16:40:29Z" deletionGracePeriodSeconds: 1 deletionTimestamp: "2023-03-05T16:46:18Z" ... skipping 141 lines ... qosClass: BestEffort to be nil[0m [38;5;9mIn [1m[It][0m[38;5;9m at: [1mtest/e2e/framework/pod/pod_client.go:184[0m [38;5;243m@ 03/05/23 16:49:17.511[0m [38;5;243m------------------------------[0m [38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m [38;5;243m------------------------------[0m [38;5;9m• [FAILED] [707.613 seconds][0m [0m[sig-storage] EmptyDir volumes [38;5;9m[1m[It] volume on default medium should have the correct mode [LinuxOnly] [NodeConformance] [Conformance][0m [38;5;243mtest/e2e/common/storage/empty_dir.go:157[0m [38;5;243mTimeline >>[0m [1mSTEP:[0m Creating a kubernetes client [38;5;243m@ 03/05/23 16:40:53.241[0m Mar 5 16:40:53.241: INFO: >>> kubeConfig: /root/.kube/config [1mSTEP:[0m Building a namespace api object, basename emptydir [38;5;243m@ 03/05/23 16:40:53.242[0m [1mSTEP:[0m Waiting for a default service account to be provisioned in namespace [38;5;243m@ 03/05/23 16:40:53.583[0m [1mSTEP:[0m Waiting for kube-root-ca.crt to be provisioned in namespace [38;5;243m@ 03/05/23 16:40:53.808[0m [1mSTEP:[0m Creating a pod to test emptydir volume type on node default medium [38;5;243m@ 03/05/23 16:40:54.034[0m Mar 5 16:46:17.804: INFO: Failed to get logs from node "i-06df568f6e5c59fff" pod "pod-dbbc7aa5-93f4-4e10-9fc3-ab81b04beb68" container "test-container": the server could not find the requested resource (get pods pod-dbbc7aa5-93f4-4e10-9fc3-ab81b04beb68) [1mSTEP:[0m delete the pod [38;5;243m@ 03/05/23 16:46:17.804[0m Mar 5 16:49:17.920: INFO: Failed inside E2E framework: k8s.io/kubernetes/test/e2e/framework/pod.WaitForPodNotFoundInNamespace({0x7fc97c0644f0, 0xc001325a80}, {0x8252218?, 0xc000a53a00}, {0xc002a58120, 0x28}, {0xc00100c620, 0xd}, 0x0?) test/e2e/framework/pod/wait.go:538 +0x190 k8s.io/kubernetes/test/e2e/framework/pod.(*PodClient).DeleteSync(0xc000f9d710, {0x7fc97c0644f0, 0xc001325a80}, {0xc002a58120, 0x28}, {{{0x0, 0x0}, {0x0, 0x0}}, 0x0, ...}, ...) test/e2e/framework/pod/pod_client.go:184 +0x1db k8s.io/kubernetes/test/e2e/framework/pod/output.MatchContainerOutput.func1() test/e2e/framework/pod/output/output.go:160 +0xc9 ... skipping 4 lines ... k8s.io/kubernetes/test/e2e/framework/pod/output.TestContainerOutput(...) test/e2e/framework/pod/output/output.go:215 k8s.io/kubernetes/test/e2e/common/storage.doTestVolumeMode({0x7fc97c0644f0, 0xc001325a80}, 0x2681e96?, 0x6f31140?, {0x0, 0x0}) test/e2e/common/storage/empty_dir.go:502 +0x416 k8s.io/kubernetes/test/e2e/common/storage.glob..func4.9({0x7fc97c0644f0?, 0xc001325a80?}) test/e2e/common/storage/empty_dir.go:158 +0x2e [38;5;9m[FAILED][0m in [It] - test/e2e/framework/pod/pod_client.go:184 [38;5;243m@ 03/05/23 16:49:17.921[0m Mar 5 16:49:17.921: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready Mar 5 16:49:18.035: INFO: Condition Ready of node i-06df568f6e5c59fff is false, but Node is tainted by NodeController with [{node.kubernetes.io/not-ready NoSchedule 2023-03-05 16:46:33 +0000 UTC} {node.kubernetes.io/not-ready NoExecute 2023-03-05 16:46:37 +0000 UTC}]. Failure Mar 5 16:49:20.151: INFO: Condition Ready of node i-06df568f6e5c59fff is false, but Node is tainted by NodeController with [{node.kubernetes.io/not-ready NoSchedule 2023-03-05 16:46:33 +0000 UTC} {node.kubernetes.io/not-ready NoExecute 2023-03-05 16:46:37 +0000 UTC}]. Failure Mar 5 16:49:22.150: INFO: Condition Ready of node i-06df568f6e5c59fff is false, but Node is tainted by NodeController with [{node.kubernetes.io/not-ready NoSchedule 2023-03-05 16:46:33 +0000 UTC} {node.kubernetes.io/not-ready NoExecute 2023-03-05 16:46:37 +0000 UTC}]. Failure Mar 5 16:49:24.150: INFO: Condition Ready of node i-06df568f6e5c59fff is false, but Node is tainted by NodeController with [{node.kubernetes.io/not-ready NoSchedule 2023-03-05 16:46:33 +0000 UTC} {node.kubernetes.io/not-ready NoExecute 2023-03-05 16:46:37 +0000 UTC}]. Failure Mar 5 16:49:26.150: INFO: Condition Ready of node i-06df568f6e5c59fff is false, but Node is tainted by NodeController with [{node.kubernetes.io/not-ready NoSchedule 2023-03-05 16:46:33 +0000 UTC} {node.kubernetes.io/not-ready NoExecute 2023-03-05 16:46:37 +0000 UTC}]. Failure ... skipping 90 lines ... Mar 5 16:52:18.377: INFO: At 2023-03-05 16:40:54 +0000 UTC - event for pod-dbbc7aa5-93f4-4e10-9fc3-ab81b04beb68: {default-scheduler } Scheduled: Successfully assigned emptydir-7027/pod-dbbc7aa5-93f4-4e10-9fc3-ab81b04beb68 to i-06df568f6e5c59fff Mar 5 16:52:18.377: INFO: At 2023-03-05 16:41:51 +0000 UTC - event for pod-dbbc7aa5-93f4-4e10-9fc3-ab81b04beb68: {taint-controller } TaintManagerEviction: Cancelling deletion of Pod emptydir-7027/pod-dbbc7aa5-93f4-4e10-9fc3-ab81b04beb68 Mar 5 16:52:18.377: INFO: At 2023-03-05 16:47:57 +0000 UTC - event for pod-dbbc7aa5-93f4-4e10-9fc3-ab81b04beb68: {taint-controller } TaintManagerEviction: Marking for deletion Pod emptydir-7027/pod-dbbc7aa5-93f4-4e10-9fc3-ab81b04beb68 Mar 5 16:52:18.490: INFO: POD NODE PHASE GRACE CONDITIONS Mar 5 16:52:18.490: INFO: pod-dbbc7aa5-93f4-4e10-9fc3-ab81b04beb68 i-06df568f6e5c59fff Pending 30s [{PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2023-03-05 16:40:54 +0000 UTC } {DisruptionTarget True 0001-01-01 00:00:00 +0000 UTC 2023-03-05 16:47:57 +0000 UTC DeletionByTaintManager Taint manager: deleting due to NoExecute taint}] Mar 5 16:52:18.490: INFO: Mar 5 16:52:28.606: INFO: Unable to fetch emptydir-7027/pod-dbbc7aa5-93f4-4e10-9fc3-ab81b04beb68/test-container logs: an error on the server ("unknown") has prevented the request from succeeding (get pods pod-dbbc7aa5-93f4-4e10-9fc3-ab81b04beb68) Mar 5 16:52:28.719: INFO: Logging node info for node i-056ad61525499081e Mar 5 16:52:28.833: INFO: Node Info: &Node{ObjectMeta:{i-056ad61525499081e a6ce9ed3-5ae8-4065-b117-6884e24b3632 19980 0 2023-03-05 16:24:16 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:arm64 beta.kubernetes.io/instance-type:t4g.small beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:eu-central-1 failure-domain.beta.kubernetes.io/zone:eu-central-1a karpenter.k8s.aws/instance-category:t karpenter.k8s.aws/instance-cpu:2 karpenter.k8s.aws/instance-family:t4g karpenter.k8s.aws/instance-generation:4 karpenter.k8s.aws/instance-hypervisor:nitro karpenter.k8s.aws/instance-memory:2048 karpenter.k8s.aws/instance-pods:11 karpenter.k8s.aws/instance-size:small karpenter.sh/capacity-type:spot karpenter.sh/initialized:true karpenter.sh/provisioner-name:nodes kubernetes.io/arch:arm64 kubernetes.io/hostname:i-056ad61525499081e kubernetes.io/os:linux node-role.kubernetes.io/node: node-role.kubernetes.io/spot-worker:true node.kubernetes.io/instance-type:t4g.small topology.ebs.csi.aws.com/zone:eu-central-1a topology.hostpath.csi/node:i-056ad61525499081e topology.kubernetes.io/region:eu-central-1 topology.kubernetes.io/zone:eu-central-1a] map[csi.volume.kubernetes.io/nodeid:{"csi-hostpath-read-write-once-pod-5175":"i-056ad61525499081e","csi-mock-csi-mock-volumes-expansion-4933":"csi-mock-csi-mock-volumes-expansion-4933","csi-mock-csi-mock-volumes-fsgroup-policy-6623":"i-056ad61525499081e","ebs.csi.aws.com":"i-056ad61525499081e"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [{karpenter.sh/v1alpha5 Provisioner nodes ac5d7599-b2a2-4eae-8da4-1f4c0a54ec82 <nil> 0xc003a5d550}] [karpenter.sh/termination] [{aws-cloud-controller-manager Update v1 2023-03-05 16:24:16 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{}}}} } {kops-controller Update v1 2023-03-05 16:24:16 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{},"f:node-role.kubernetes.io/spot-worker":{}}}} } {karpenter Update v1 2023-03-05 16:25:57 +0000 UTC FieldsV1 {"f:metadata":{"f:finalizers":{".":{},"v:\"karpenter.sh/termination\"":{}},"f:labels":{".":{},"f:karpenter.k8s.aws/instance-category":{},"f:karpenter.k8s.aws/instance-cpu":{},"f:karpenter.k8s.aws/instance-family":{},"f:karpenter.k8s.aws/instance-generation":{},"f:karpenter.k8s.aws/instance-hypervisor":{},"f:karpenter.k8s.aws/instance-memory":{},"f:karpenter.k8s.aws/instance-pods":{},"f:karpenter.k8s.aws/instance-size":{},"f:karpenter.sh/capacity-type":{},"f:karpenter.sh/initialized":{},"f:karpenter.sh/provisioner-name":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}},"f:ownerReferences":{".":{},"k:{\"uid\":\"ac5d7599-b2a2-4eae-8da4-1f4c0a54ec82\"}":{}}},"f:spec":{"f:providerID":{}}} } {kube-controller-manager Update v1 2023-03-05 16:25:57 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:node.alpha.kubernetes.io/ttl":{}},"f:labels":{"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.1.0/24\"":{}}}} } {aws-cloud-controller-manager Update v1 2023-03-05 16:26:48 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{".":{},"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"InternalIP\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kube-controller-manager Update v1 2023-03-05 16:52:18 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{".":{},"k:{\"type\":\"DiskPressure\"}":{".":{},"f:type":{}},"k:{\"type\":\"MemoryPressure\"}":{".":{},"f:type":{}},"k:{\"type\":\"PIDPressure\"}":{".":{},"f:type":{}},"k:{\"type\":\"Ready\"}":{".":{},"f:type":{}}},"f:volumesAttached":{}}} status} {kubelet Update v1 2023-03-05 16:52:21 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{"f:kubernetes.io/hostname":{},"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{".":{},"f:cpu":{},"f:ephemeral-storage":{},"f:hugepages-1Gi":{},"f:hugepages-2Mi":{},"f:hugepages-32Mi":{},"f:hugepages-64Ki":{},"f:memory":{},"f:pods":{}},"f:capacity":{".":{},"f:cpu":{},"f:ephemeral-storage":{},"f:hugepages-1Gi":{},"f:hugepages-2Mi":{},"f:hugepages-32Mi":{},"f:hugepages-64Ki":{},"f:memory":{},"f:pods":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:daemonEndpoints":{"f:kubeletEndpoint":{"f:Port":{}}},"f:images":{},"f:nodeInfo":{"f:architecture":{},"f:bootID":{},"f:containerRuntimeVersion":{},"f:kernelVersion":{},"f:kubeProxyVersion":{},"f:kubeletVersion":{},"f:machineID":{},"f:operatingSystem":{},"f:osImage":{},"f:systemUUID":{}},"f:volumesInUse":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.1.0/24,DoNotUseExternalID:,ProviderID:aws:///eu-central-1a/i-056ad61525499081e,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.1.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49756430336 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-32Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-64Ki: {{0 0} {<nil>} 0 DecimalSI},memory: {{1941540864 0} {<nil>} 1896036Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44780787229 0} {<nil>} 44780787229 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-32Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-64Ki: {{0 0} {<nil>} 0 DecimalSI},memory: {{1836683264 0} {<nil>} 1793636Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-03-05 16:52:21 +0000 UTC,LastTransitionTime:2023-03-05 16:25:57 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-03-05 16:52:21 +0000 UTC,LastTransitionTime:2023-03-05 16:25:48 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-03-05 16:52:21 +0000 UTC,LastTransitionTime:2023-03-05 16:25:48 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-03-05 16:52:21 +0000 UTC,LastTransitionTime:2023-03-05 16:25:48 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.36.179,},NodeAddress{Type:ExternalIP,Address:18.159.37.200,},NodeAddress{Type:InternalDNS,Address:i-056ad61525499081e.eu-central-1.compute.internal,},NodeAddress{Type:Hostname,Address:i-056ad61525499081e.eu-central-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-18-159-37-200.eu-central-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2af3207d1c8c09785c337aae6c6ef0,SystemUUID:ec2af320-7d1c-8c09-785c-337aae6c6ef0,BootID:786f2752-bc30-4729-8aaf-45977b16c253,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 22.04.2 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.27.0-alpha.3.52+d48b8167f7094b,KubeProxyVersion:v1.27.0-alpha.3.52+d48b8167f7094b,OperatingSystem:linux,Architecture:arm64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:157636062,},ContainerImage{Names:[registry.k8s.io/kube-proxy-arm64:v1.27.0-alpha.3.52_d48b8167f7094b],SizeBytes:60554082,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:16bbf38c463a4223d8cfe4da12bc61010b082a79b4bb003e2d3ba3ece5dd5f9e registry.k8s.io/e2e-test-images/agnhost:2.43],SizeBytes:49808364,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:3fe7acf013d1264ffded116b80a73dc129a449b0fccdb8d21af8279f2233f36e registry.k8s.io/e2e-test-images/httpd:2.4.39-4],SizeBytes:41528818,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:148b022f5c5da426fc2f3c14b5c0867e58ef05961510c84749ac1fddcb0fef22 registry.k8s.io/e2e-test-images/httpd:2.4.38-4],SizeBytes:40343601,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:28932904,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:ee3b525d5b89db99da3b8eb521d9cd90cb6e9ef0fbb651e98bb37be78d36b5b8 registry.k8s.io/sig-storage/csi-provisioner:v3.3.0],SizeBytes:23945474,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:425d8f1b769398127767b06ed97ce62578a3179bcb99809ce93a1649e025ffe7 registry.k8s.io/sig-storage/csi-resizer:v1.6.0],SizeBytes:22678632,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:291334908ddf71a4661fd7f6d9d97274de8a5378a2b6fdfeb2ce73414a34f82f registry.k8s.io/sig-storage/csi-snapshotter:v6.1.0],SizeBytes:22434180,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:9a685020911e2725ad019dbce6e4a5ab93d51e3d4557f115e64343345e05781b registry.k8s.io/sig-storage/csi-attacher:v4.0.0],SizeBytes:22394501,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:122bfb8c1edabb3c0edd63f06523e6940d958d19b3957dc7b1d6f81e9f1f6119 registry.k8s.io/sig-storage/csi-provisioner:v3.1.0],SizeBytes:21735644,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:9ebbf9f023e7b41ccee3d52afe39a89e3ddacdbb69269d583abfc25847cfd9e4 registry.k8s.io/sig-storage/csi-resizer:v1.4.0],SizeBytes:20848413,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:8b9c313c05f54fb04f8d430896f5f5904b6cb157df261501b29adc04d2b2dc7b registry.k8s.io/sig-storage/csi-attacher:v3.4.0],SizeBytes:20565480,},ContainerImage{Names:[registry.k8s.io/cpa/cluster-proportional-autoscaler@sha256:68d396900aeaa072c1f27289485fdac29834045a6f3ffe369bf389d830ef572d registry.k8s.io/cpa/cluster-proportional-autoscaler:1.8.6],SizeBytes:19007623,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:92257881c1d6493cf18299a24af42330f891166560047902b8d431fb66b01af5 registry.k8s.io/sig-storage/hostpathplugin:v1.9.0],SizeBytes:18267611,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:9b2d9abda017c28b12f84a344f57cd73fbdb6c2bd7dd5bdd5018246ad1093ba6 registry.k8s.io/sig-storage/hostpathplugin:v1.11.0],SizeBytes:18159085,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:017727efcfeb7d053af68e51436ce8e65edbc6ca573720afb4f79c8594036955 registry.k8s.io/coredns/coredns:v1.10.0],SizeBytes:13766234,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:8502345,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:933940f13b3ea0abc62e656c1aa5c5b47c04b15d71250413a6b821bd0c58b94e registry.k8s.io/sig-storage/livenessprobe:v2.7.0],SizeBytes:8223819,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:7685305,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:5c99cf6a02adda929b10321dbf4ecfa00d87be9ba4fb456006237d530ab4baa1 registry.k8s.io/e2e-test-images/nginx:1.14-4],SizeBytes:6860089,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf registry.k8s.io/e2e-test-images/busybox:1.29-2],SizeBytes:773367,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:2e0f836850e09b8b7cc937681d6194537a09fbd5f6b9e08f4d646a85128e8937 registry.k8s.io/e2e-test-images/busybox:1.29-4],SizeBytes:772933,},ContainerImage{Names:[registry.k8s.io/pause@sha256:7031c1b283388d2c2e09b57badb803c05ebed362dc88d84b480cc47f72a21097 registry.k8s.io/pause:3.9],SizeBytes:268051,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:253553,},},VolumesInUse:[kubernetes.io/csi/csi-hostpath-read-write-once-pod-5175^bc2b4edf-bb75-11ed-8d33-a27fee729fe2 kubernetes.io/csi/csi-mock-csi-mock-volumes-expansion-4933^4 kubernetes.io/csi/ebs.csi.aws.com^vol-0e020446b1cd6abaf],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/csi-mock-csi-mock-volumes-expansion-4933^4,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-0e020446b1cd6abaf,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/csi-hostpath-read-write-once-pod-5175^bc2b4edf-bb75-11ed-8d33-a27fee729fe2,DevicePath:,},},Config:nil,},} Mar 5 16:52:28.833: INFO: Logging kubelet events for node i-056ad61525499081e Mar 5 16:52:28.951: INFO: ... skipping 61 lines ... Logging node info for node i-06df568f6e5c59fff Mar 5 16:52:29.661: INFO: Node Info: &Node{ObjectMeta:{i-06df568f6e5c59fff cb63c681-2e18-474a-b6f3-221325ac0d52 18792 0 2023-03-05 16:24:16 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:arm64 beta.kubernetes.io/instance-type:t4g.small beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:eu-central-1 failure-domain.beta.kubernetes.io/zone:eu-central-1a karpenter.k8s.aws/instance-category:t karpenter.k8s.aws/instance-cpu:2 karpenter.k8s.aws/instance-family:t4g karpenter.k8s.aws/instance-generation:4 karpenter.k8s.aws/instance-hypervisor:nitro karpenter.k8s.aws/instance-memory:2048 karpenter.k8s.aws/instance-pods:11 karpenter.k8s.aws/instance-size:small karpenter.sh/capacity-type:spot karpenter.sh/initialized:true karpenter.sh/provisioner-name:nodes kubernetes.io/arch:arm64 kubernetes.io/hostname:i-06df568f6e5c59fff kubernetes.io/os:linux node-role.kubernetes.io/node: node-role.kubernetes.io/spot-worker:true node.kubernetes.io/instance-type:t4g.small topology.ebs.csi.aws.com/zone:eu-central-1a topology.hostpath.csi/node:i-06df568f6e5c59fff topology.kubernetes.io/region:eu-central-1 topology.kubernetes.io/zone:eu-central-1a] map[csi.volume.kubernetes.io/nodeid:{"csi-hostpath-ephemeral-2147":"i-06df568f6e5c59fff","csi-hostpath-ephemeral-5408":"i-06df568f6e5c59fff","csi-hostpath-ephemeral-6540":"i-06df568f6e5c59fff","csi-mock-csi-mock-volumes-fsgroup-policy-1058":"i-06df568f6e5c59fff","csi-mock-csi-mock-volumes-workload-2219":"i-06df568f6e5c59fff","ebs.csi.aws.com":"i-06df568f6e5c59fff"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [{karpenter.sh/v1alpha5 Provisioner nodes ac5d7599-b2a2-4eae-8da4-1f4c0a54ec82 <nil> 0xc004a1ceb0}] [karpenter.sh/termination] [{aws-cloud-controller-manager Update v1 2023-03-05 16:24:16 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{}}}} } {kops-controller Update v1 2023-03-05 16:24:16 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{},"f:node-role.kubernetes.io/spot-worker":{}}}} } {karpenter Update v1 2023-03-05 16:25:57 +0000 UTC FieldsV1 {"f:metadata":{"f:finalizers":{".":{},"v:\"karpenter.sh/termination\"":{}},"f:labels":{".":{},"f:karpenter.k8s.aws/instance-category":{},"f:karpenter.k8s.aws/instance-cpu":{},"f:karpenter.k8s.aws/instance-family":{},"f:karpenter.k8s.aws/instance-generation":{},"f:karpenter.k8s.aws/instance-hypervisor":{},"f:karpenter.k8s.aws/instance-memory":{},"f:karpenter.k8s.aws/instance-pods":{},"f:karpenter.k8s.aws/instance-size":{},"f:karpenter.sh/capacity-type":{},"f:karpenter.sh/initialized":{},"f:karpenter.sh/provisioner-name":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}},"f:ownerReferences":{".":{},"k:{\"uid\":\"ac5d7599-b2a2-4eae-8da4-1f4c0a54ec82\"}":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-03-05 16:26:48 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{".":{},"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"InternalIP\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kubelet Update v1 2023-03-05 16:48:40 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{"f:kubernetes.io/hostname":{},"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{".":{},"f:cpu":{},"f:ephemeral-storage":{},"f:hugepages-1Gi":{},"f:hugepages-2Mi":{},"f:hugepages-32Mi":{},"f:hugepages-64Ki":{},"f:memory":{},"f:pods":{}},"f:capacity":{".":{},"f:cpu":{},"f:ephemeral-storage":{},"f:hugepages-1Gi":{},"f:hugepages-2Mi":{},"f:hugepages-32Mi":{},"f:hugepages-64Ki":{},"f:memory":{},"f:pods":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{}}},"f:daemonEndpoints":{"f:kubeletEndpoint":{"f:Port":{}}},"f:images":{},"f:nodeInfo":{"f:architecture":{},"f:bootID":{},"f:containerRuntimeVersion":{},"f:kernelVersion":{},"f:kubeProxyVersion":{},"f:kubeletVersion":{},"f:machineID":{},"f:operatingSystem":{},"f:osImage":{},"f:systemUUID":{}},"f:volumesInUse":{}}} status} {kube-controller-manager Update v1 2023-03-05 16:49:37 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{".":{},"k:{\"type\":\"DiskPressure\"}":{".":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"MemoryPressure\"}":{".":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"PIDPressure\"}":{".":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Ready\"}":{".":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}},"f:volumesAttached":{}}} status} {kube-controller-manager Update v1 2023-03-05 16:49:42 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:node.alpha.kubernetes.io/ttl":{}},"f:labels":{"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.2.0/24\"":{}},"f:taints":{}}} }]},Spec:NodeSpec{PodCIDR:100.96.2.0/24,DoNotUseExternalID:,ProviderID:aws:///eu-central-1a/i-06df568f6e5c59fff,Unschedulable:false,Taints:[]Taint{Taint{Key:node.kubernetes.io/unreachable,Value:,Effect:NoSchedule,TimeAdded:2023-03-05 16:49:37 +0000 UTC,},Taint{Key:node.kubernetes.io/unreachable,Value:,Effect:NoExecute,TimeAdded:2023-03-05 16:49:42 +0000 UTC,},},ConfigSource:nil,PodCIDRs:[100.96.2.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49756430336 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-32Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-64Ki: {{0 0} {<nil>} 0 DecimalSI},memory: {{1941540864 0} {<nil>} 1896036Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44780787229 0} {<nil>} 44780787229 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-32Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-64Ki: {{0 0} {<nil>} 0 DecimalSI},memory: {{1836683264 0} {<nil>} 1793636Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:Ready,Status:Unknown,LastHeartbeatTime:2023-03-05 16:47:34 +0000 UTC,LastTransitionTime:2023-03-05 16:49:37 +0000 UTC,Reason:NodeStatusUnknown,Message:Kubelet stopped posting node status.,},NodeCondition{Type:MemoryPressure,Status:Unknown,LastHeartbeatTime:2023-03-05 16:47:34 +0000 UTC,LastTransitionTime:2023-03-05 16:49:37 +0000 UTC,Reason:NodeStatusUnknown,Message:Kubelet stopped posting node status.,},NodeCondition{Type:DiskPressure,Status:Unknown,LastHeartbeatTime:2023-03-05 16:47:34 +0000 UTC,LastTransitionTime:2023-03-05 16:49:37 +0000 UTC,Reason:NodeStatusUnknown,Message:Kubelet stopped posting node status.,},NodeCondition{Type:PIDPressure,Status:Unknown,LastHeartbeatTime:2023-03-05 16:47:34 +0000 UTC,LastTransitionTime:2023-03-05 16:49:37 +0000 UTC,Reason:NodeStatusUnknown,Message:Kubelet stopped posting node status.,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.43.148,},NodeAddress{Type:ExternalIP,Address:18.192.211.222,},NodeAddress{Type:InternalDNS,Address:i-06df568f6e5c59fff.eu-central-1.compute.internal,},NodeAddress{Type:Hostname,Address:i-06df568f6e5c59fff.eu-central-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-18-192-211-222.eu-central-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec221221864d9409a8ec5f6614a9b94b,SystemUUID:ec221221-864d-9409-a8ec-5f6614a9b94b,BootID:553d31bf-faca-417e-a3db-325839e1b8e4,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 22.04.2 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.27.0-alpha.3.52+d48b8167f7094b,KubeProxyVersion:v1.27.0-alpha.3.52+d48b8167f7094b,OperatingSystem:linux,Architecture:arm64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:157636062,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/jessie-dnsutils@sha256:24aaf2626d6b27864c29de2097e8bbb840b3a414271bf7c8995e431e47d8408e registry.k8s.io/e2e-test-images/jessie-dnsutils:1.7],SizeBytes:116764683,},ContainerImage{Names:[registry.k8s.io/kube-proxy-arm64:v1.27.0-alpha.3.52_d48b8167f7094b],SizeBytes:60554082,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:16bbf38c463a4223d8cfe4da12bc61010b082a79b4bb003e2d3ba3ece5dd5f9e registry.k8s.io/e2e-test-images/agnhost:2.43],SizeBytes:49808364,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:3fe7acf013d1264ffded116b80a73dc129a449b0fccdb8d21af8279f2233f36e registry.k8s.io/e2e-test-images/httpd:2.4.39-4],SizeBytes:41528818,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:148b022f5c5da426fc2f3c14b5c0867e58ef05961510c84749ac1fddcb0fef22 registry.k8s.io/e2e-test-images/httpd:2.4.38-4],SizeBytes:40343601,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:28932904,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:ee3b525d5b89db99da3b8eb521d9cd90cb6e9ef0fbb651e98bb37be78d36b5b8 registry.k8s.io/sig-storage/csi-provisioner:v3.3.0],SizeBytes:23945474,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:425d8f1b769398127767b06ed97ce62578a3179bcb99809ce93a1649e025ffe7 registry.k8s.io/sig-storage/csi-resizer:v1.6.0],SizeBytes:22678632,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:291334908ddf71a4661fd7f6d9d97274de8a5378a2b6fdfeb2ce73414a34f82f registry.k8s.io/sig-storage/csi-snapshotter:v6.1.0],SizeBytes:22434180,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:9a685020911e2725ad019dbce6e4a5ab93d51e3d4557f115e64343345e05781b registry.k8s.io/sig-storage/csi-attacher:v4.0.0],SizeBytes:22394501,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:122bfb8c1edabb3c0edd63f06523e6940d958d19b3957dc7b1d6f81e9f1f6119 registry.k8s.io/sig-storage/csi-provisioner:v3.1.0],SizeBytes:21735644,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:9ebbf9f023e7b41ccee3d52afe39a89e3ddacdbb69269d583abfc25847cfd9e4 registry.k8s.io/sig-storage/csi-resizer:v1.4.0],SizeBytes:20848413,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:8b9c313c05f54fb04f8d430896f5f5904b6cb157df261501b29adc04d2b2dc7b registry.k8s.io/sig-storage/csi-attacher:v3.4.0],SizeBytes:20565480,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nonroot@sha256:ee9f50b3c64b174d296d91ca9f69a914ac30e59095dfb462b2b518ad28a63655 registry.k8s.io/e2e-test-images/nonroot:1.4],SizeBytes:18561972,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:92257881c1d6493cf18299a24af42330f891166560047902b8d431fb66b01af5 registry.k8s.io/sig-storage/hostpathplugin:v1.9.0],SizeBytes:18267611,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:9b2d9abda017c28b12f84a344f57cd73fbdb6c2bd7dd5bdd5018246ad1093ba6 registry.k8s.io/sig-storage/hostpathplugin:v1.11.0],SizeBytes:18159085,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:017727efcfeb7d053af68e51436ce8e65edbc6ca573720afb4f79c8594036955 registry.k8s.io/coredns/coredns:v1.10.0],SizeBytes:13766234,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:8502345,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:933940f13b3ea0abc62e656c1aa5c5b47c04b15d71250413a6b821bd0c58b94e registry.k8s.io/sig-storage/livenessprobe:v2.7.0],SizeBytes:8223819,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:7685305,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:5c99cf6a02adda929b10321dbf4ecfa00d87be9ba4fb456006237d530ab4baa1 registry.k8s.io/e2e-test-images/nginx:1.14-4],SizeBytes:6860089,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nonewprivs@sha256:8ac1264691820febacf3aea5d152cbde6d10685731ec14966a9401c6f47a68ac registry.k8s.io/e2e-test-images/nonewprivs:1.3],SizeBytes:5025027,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:2e0f836850e09b8b7cc937681d6194537a09fbd5f6b9e08f4d646a85128e8937 registry.k8s.io/e2e-test-images/busybox:1.29-4],SizeBytes:772933,},ContainerImage{Names:[registry.k8s.io/pause@sha256:7031c1b283388d2c2e09b57badb803c05ebed362dc88d84b480cc47f72a21097 registry.k8s.io/pause:3.9],SizeBytes:268051,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:253553,},},VolumesInUse:[kubernetes.io/csi/csi-hostpath-ephemeral-2147^52df5953-bb73-11ed-8aa7-be49170871b4 kubernetes.io/csi/csi-hostpath-ephemeral-5408^443d1daf-bb73-11ed-95c4-5a95d17cbc27 kubernetes.io/csi/csi-hostpath-ephemeral-5408^44411f51-bb73-11ed-95c4-5a95d17cbc27 kubernetes.io/csi/csi-hostpath-ephemeral-6540^4426a07f-bb73-11ed-a604-92687f2c50f8 kubernetes.io/csi/ebs.csi.aws.com^vol-05b67fab5beb6394b],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-05b67fab5beb6394b,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/csi-hostpath-ephemeral-6540^4426a07f-bb73-11ed-a604-92687f2c50f8,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/csi-hostpath-ephemeral-5408^44411f51-bb73-11ed-95c4-5a95d17cbc27,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/csi-hostpath-ephemeral-5408^443d1daf-bb73-11ed-95c4-5a95d17cbc27,DevicePath:,},},Config:nil,},} Mar 5 16:52:29.662: INFO: Logging kubelet events for node i-06df568f6e5c59fff Mar 5 16:52:29.778: INFO: Logging pods the kubelet thinks is on node i-06df568f6e5c59fff Mar 5 16:52:39.892: INFO: Unable to retrieve kubelet pods for node i-06df568f6e5c59fff: error trying to reach service: net/http: TLS handshake timeout Mar 5 16:52:39.892: INFO: Logging node info for node i-0d6a32f749839192f Mar 5 16:52:40.006: INFO: Node Info: &Node{ObjectMeta:{i-0d6a32f749839192f 4155fefb-4d3a-4b04-ba53-6ac9e895ff92 18509 0 2023-03-05 16:20:46 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:arm64 beta.kubernetes.io/instance-type:c6g.xlarge beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:eu-central-1 failure-domain.beta.kubernetes.io/zone:eu-central-1a kops.k8s.io/kops-controller-pki: kubernetes.io/arch:arm64 kubernetes.io/hostname:i-0d6a32f749839192f kubernetes.io/os:linux node-role.kubernetes.io/control-plane: node.kubernetes.io/exclude-from-external-load-balancers: node.kubernetes.io/instance-type:c6g.xlarge topology.ebs.csi.aws.com/zone:eu-central-1a topology.kubernetes.io/region:eu-central-1 topology.kubernetes.io/zone:eu-central-1a] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-0d6a32f749839192f"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kubelet Update v1 2023-03-05 16:20:46 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {protokube Update v1 2023-03-05 16:21:25 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kops.k8s.io/kops-controller-pki":{},"f:node-role.kubernetes.io/control-plane":{},"f:node.kubernetes.io/exclude-from-external-load-balancers":{}}}} } {kube-controller-manager Update v1 2023-03-05 16:21:47 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.0.0/24\"":{}}}} } {aws-cloud-controller-manager Update v1 2023-03-05 16:21:48 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{},"f:taints":{}}} } {aws-cloud-controller-manager Update v1 2023-03-05 16:21:48 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kubelet Update v1 2023-03-05 16:49:18 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.0.0/24,DoNotUseExternalID:,ProviderID:aws:///eu-central-1a/i-0d6a32f749839192f,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/control-plane,Value:,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[100.96.0.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{4 0} {<nil>} 4 DecimalSI},ephemeral-storage: {{49756430336 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-32Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-64Ki: {{0 0} {<nil>} 0 DecimalSI},memory: {{8150118400 0} {<nil>} 7959100Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{4 0} {<nil>} 4 DecimalSI},ephemeral-storage: {{44780787229 0} {<nil>} 44780787229 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-32Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-64Ki: {{0 0} {<nil>} 0 DecimalSI},memory: {{8045260800 0} {<nil>} 7856700Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-03-05 16:49:18 +0000 UTC,LastTransitionTime:2023-03-05 16:20:46 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-03-05 16:49:18 +0000 UTC,LastTransitionTime:2023-03-05 16:20:46 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-03-05 16:49:18 +0000 UTC,LastTransitionTime:2023-03-05 16:20:46 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-03-05 16:49:18 +0000 UTC,LastTransitionTime:2023-03-05 16:21:47 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.49.247,},NodeAddress{Type:ExternalIP,Address:18.197.20.82,},NodeAddress{Type:InternalDNS,Address:i-0d6a32f749839192f.eu-central-1.compute.internal,},NodeAddress{Type:Hostname,Address:i-0d6a32f749839192f.eu-central-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-18-197-20-82.eu-central-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2bed249e8ac7b517cb08ccd42b83e5,SystemUUID:ec2bed24-9e8a-c7b5-17cb-08ccd42b83e5,BootID:6f9215c9-dfe8-4b13-84f4-9b2084456198,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 22.04.2 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.27.0-alpha.3.52+d48b8167f7094b,KubeProxyVersion:v1.27.0-alpha.3.52+d48b8167f7094b,OperatingSystem:linux,Architecture:arm64,},Images:[]ContainerImage{ContainerImage{Names:[registry.k8s.io/etcd@sha256:9ce33ba33d8e738a5b85ed50b5080ac746deceed4a7496c550927a7a19ca3b6d registry.k8s.io/etcd:3.5.0-0],SizeBytes:157797353,},ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:157636062,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:4afb99b4690b418ffc2ceb67e1a17376457e441c1f09ab55447f0aaf992fa646 registry.k8s.io/etcd:3.4.3-0],SizeBytes:151677080,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:4ad90a11b55313b182afc186b9876c8e891531b8db4c9bf1541953021618d0e2 registry.k8s.io/etcd:3.4.13-0],SizeBytes:134531559,},ContainerImage{Names:[registry.k8s.io/kube-apiserver-arm64:v1.27.0-alpha.3.52_d48b8167f7094b],SizeBytes:131087707,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:0d2ab6b15b4fe0b903b9e81192d942fb39c29be61fe341cd16db5d0165b0435c registry.k8s.io/etcd:3.3.17-0],SizeBytes:126979107,},ContainerImage{Names:[registry.k8s.io/kube-controller-manager-arm64:v1.27.0-alpha.3.52_d48b8167f7094b],SizeBytes:120667128,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:17da501f5d2a675be46040422a27b7cc21b8a43895ac998b171db1c346f361f7 registry.k8s.io/etcd:3.3.10-0],SizeBytes:81436097,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:6f72b851544986cb0921b53ea655ec04c36131248f16d4ad110cb3ca0c369dc1 registry.k8s.io/etcd:3.5.4-0],SizeBytes:81120118,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:13f53ed1d91e2e11aac476ee9a0269fdda6cc4874eba903efd40daf50c55eee5 registry.k8s.io/etcd:3.5.3-0],SizeBytes:81114891,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:51eae8381dcb1078289fa7b4f3df2630cdc18d09fb56f8e56b41c40e191d6c83 registry.k8s.io/etcd:3.5.7-0],SizeBytes:80665728,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:dd75ec974b0a2a6f6bb47001ba09207976e625db898d1b16735528c009cb171c registry.k8s.io/etcd:3.5.6-0],SizeBytes:80539316,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:905d7ca17fd02bc24c0eba9a062753aba15db3e31422390bc3238eb762339b20 registry.k8s.io/etcd:3.2.24-1],SizeBytes:61500433,},ContainerImage{Names:[registry.k8s.io/etcdadm/etcd-manager-slim@sha256:8727adc059334e0e3329dbccce116a734da4a090da5e86c8cddccaf333caa666 registry.k8s.io/etcdadm/etcd-manager-slim:v3.0.20230201],SizeBytes:60826655,},ContainerImage{Names:[registry.k8s.io/kube-proxy-arm64:v1.27.0-alpha.3.52_d48b8167f7094b],SizeBytes:60554082,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:64b9ea357325d5db9f8a723dcf503b5a449177b17ac87d69481e126bb724c263 registry.k8s.io/etcd:3.5.1-0],SizeBytes:59553904,},ContainerImage{Names:[registry.k8s.io/kube-scheduler-arm64:v1.27.0-alpha.3.52_d48b8167f7094b],SizeBytes:56310763,},ContainerImage{Names:[registry.k8s.io/kops/kops-controller:1.27.0-alpha.1],SizeBytes:41797411,},ContainerImage{Names:[registry.k8s.io/kops/dns-controller:1.27.0-alpha.1],SizeBytes:40267473,},ContainerImage{Names:[public.ecr.aws/karpenter/controller@sha256:68db4f092cf9cc83f5ef9e2fbc5407c2cb682e81f64dfaa700a7602ede38b1cf public.ecr.aws/karpenter/controller:v0.16.3],SizeBytes:33707751,},ContainerImage{Names:[public.ecr.aws/karpenter/webhook@sha256:96a2d9b06d6bc5127801f358f74b1cf2d289b423a2e9ba40c573c0b14b17dafa public.ecr.aws/karpenter/webhook:v0.16.3],SizeBytes:33307187,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:28932904,},ContainerImage{Names:[quay.io/cilium/operator@sha256:a6d24a006a6b92967ac90786b49bc1ac26e5477cf028cd1186efcfc2466484db quay.io/cilium/operator:v1.12.5],SizeBytes:24567287,},ContainerImage{Names:[registry.k8s.io/provider-aws/cloud-controller-manager@sha256:fdeb61e3e42ecd9cca868d550ebdb88dd6341d9e91fcfa9a37e227dab2ad22cb registry.k8s.io/provider-aws/cloud-controller-manager:v1.26.0],SizeBytes:18345226,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:8502345,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:7685305,},ContainerImage{Names:[registry.k8s.io/kops/kube-apiserver-healthcheck:1.27.0-alpha.1],SizeBytes:4713612,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:253553,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Mar 5 16:52:40.006: INFO: Logging kubelet events for node i-0d6a32f749839192f Mar 5 16:52:40.122: INFO: ... skipping 52 lines ... Mar 5 16:52:40.246: INFO: Container webhook ready: true, restart count 0 Mar 5 16:52:40.739: INFO: Latency metrics for node i-0d6a32f749839192f [1mSTEP:[0m Destroying namespace "emptydir-7027" for this suite. [38;5;243m@ 03/05/23 16:52:40.739[0m [38;5;243m<< Timeline[0m [38;5;9m[FAILED] wait for pod "pod-dbbc7aa5-93f4-4e10-9fc3-ab81b04beb68" to disappear: expected pod to not be found: Timed out after 180.000s. Expected <*v1.Pod | 0xc0008e4d80>: metadata: creationTimestamp: "2023-03-05T16:40:54Z" deletionGracePeriodSeconds: 30 deletionTimestamp: "2023-03-05T16:46:47Z" ... skipping 139 lines ... [38;5;243m------------------------------[0m [38;5;10m• [232.049 seconds][0m [0m[sig-node] SSH [38;5;10m[1mshould SSH to all nodes and run commands[0m [38;5;243mtest/e2e/node/ssh.go:48[0m [38;5;243mCaptured StdOut/StdErr Output >>[0m error dialing ubuntu@i.do.not.exist: 'dial tcp: address i.do.not.exist: missing port in address', retrying error dialing ubuntu@i.do.not.exist: 'dial tcp: address i.do.not.exist: missing port in address', retrying error dialing ubuntu@i.do.not.exist: 'dial tcp: address i.do.not.exist: missing port in address', retrying error dialing ubuntu@i.do.not.exist: 'dial tcp: address i.do.not.exist: missing port in address', retrying error dialing ubuntu@i.do.not.exist: 'dial tcp: address i.do.not.exist: missing port in address', retrying [38;5;243m<< Captured StdOut/StdErr Output[0m [38;5;243m------------------------------[0m [38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;10m•[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m [38;5;243m------------------------------[0m [38;5;9m• [FAILED] [1299.845 seconds][0m [0m[sig-node] Mount propagation [38;5;9m[1m[It] should propagate mounts within defined scopes[0m [38;5;243mtest/e2e/node/mount_propagation.go:87[0m [38;5;243mTimeline >>[0m [1mSTEP:[0m Creating a kubernetes client [38;5;243m@ 03/05/23 16:31:33.529[0m Mar 5 16:31:33.529: INFO: >>> kubeConfig: /root/.kube/config ... skipping 97 lines ... Mar 5 16:32:05.561: INFO: Exec stderr: "" Mar 5 16:32:05.674: INFO: ExecWithOptions {Command:[/bin/sh -c mount -t tmpfs e2e-mount-propagation-default /mnt/test/default; echo default > /mnt/test/default/file] Namespace:mount-propagation-9879 PodName:default ContainerName:cntr Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false Quiet:false} Mar 5 16:32:05.674: INFO: >>> kubeConfig: /root/.kube/config Mar 5 16:32:05.674: INFO: ExecWithOptions: Clientset creation Mar 5 16:32:05.674: INFO: ExecWithOptions: execute(POST https://api.e2e-e2e-kops-aws-karpenter.test-cncf-aws.k8s.io/api/v1/namespaces/mount-propagation-9879/pods/default/exec?command=%2Fbin%2Fsh&command=-c&command=mount+-t+tmpfs+e2e-mount-propagation-default+%2Fmnt%2Ftest%2Fdefault%3B+echo+default+%3E+%2Fmnt%2Ftest%2Fdefault%2Ffile&container=cntr&container=cntr&stderr=true&stdout=true) Mar 5 16:32:06.435: INFO: Exec stderr: "" Mar 5 16:37:06.556: INFO: Failed inside E2E framework: k8s.io/kubernetes/test/e2e/framework/pod.WaitTimeoutForPodRunningInNamespace({0x7f9bbc567d78, 0xc00101e8c0}, {0x8252218?, 0xc0080a2b60?}, {0xc003b3db00, 0x22}, {0xc00007c600, 0x16}, 0x0?) test/e2e/framework/pod/wait.go:460 +0x1a4 k8s.io/kubernetes/test/e2e/storage/utils.(*hostExecutor).launchNodeExecPod(0xc004b90138?, {0x7f9bbc567d78, 0xc00101e8c0}, {0xc0023e4708, 0x13}) test/e2e/storage/utils/host_exec.go:116 +0x49d k8s.io/kubernetes/test/e2e/storage/utils.(*hostExecutor).exec(0xc000e27240, {0x7f9bbc567d78, 0xc00101e8c0}, {0xc004c3a9c0, 0xcf}, 0xc0003f38c0) test/e2e/storage/utils/host_exec.go:143 +0x135 k8s.io/kubernetes/test/e2e/storage/utils.(*hostExecutor).IssueCommandWithResult(0xc00347bc20?, {0x7f9bbc567d78?, 0xc00101e8c0?}, {0xc004c3a9c0?, 0xc004c3a9c0?}, 0xc004b900d0?) test/e2e/storage/utils/host_exec.go:176 +0x45 k8s.io/kubernetes/test/e2e/storage/utils.(*hostExecutor).IssueCommand(0x7a031ce?, {0x7f9bbc567d78?, 0xc00101e8c0?}, {0xc004c3a9c0?, 0x1?}, 0xc003f7f830?) test/e2e/storage/utils/host_exec.go:185 +0x28 k8s.io/kubernetes/test/e2e/node.glob..func7.1({0x7f9bbc567d78, 0xc00101e8c0}) test/e2e/node/mount_propagation.go:150 +0x893 [38;5;9m[FAILED][0m in [It] - test/e2e/storage/utils/host_exec.go:117 [38;5;243m@ 03/05/23 16:37:06.556[0m Mar 5 16:37:06.556: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready Mar 5 16:37:06.669: INFO: Condition Ready of node i-06df568f6e5c59fff is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-05 16:33:20 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-05 16:33:26 +0000 UTC}]. Failure Mar 5 16:37:08.784: INFO: Condition Ready of node i-06df568f6e5c59fff is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-05 16:33:20 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-05 16:33:26 +0000 UTC}]. Failure Mar 5 16:37:10.783: INFO: Condition Ready of node i-06df568f6e5c59fff is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-05 16:33:20 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-05 16:33:26 +0000 UTC}]. Failure Mar 5 16:37:12.784: INFO: Condition Ready of node i-06df568f6e5c59fff is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-05 16:33:20 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-05 16:33:26 +0000 UTC}]. Failure Mar 5 16:37:14.783: INFO: Condition Ready of node i-06df568f6e5c59fff is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-05 16:33:20 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-05 16:33:26 +0000 UTC}]. Failure ... skipping 101 lines ... Mar 5 16:47:04.867: INFO: Exec stderr: "" Mar 5 16:47:05.251: INFO: ExecWithOptions {Command:[/bin/sh -c umount /mnt/test/master] Namespace:mount-propagation-9879 PodName:master ContainerName:cntr Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false Quiet:false} Mar 5 16:47:05.251: INFO: >>> kubeConfig: /root/.kube/config Mar 5 16:47:05.252: INFO: ExecWithOptions: Clientset creation Mar 5 16:47:05.252: INFO: ExecWithOptions: execute(POST https://api.e2e-e2e-kops-aws-karpenter.test-cncf-aws.k8s.io/api/v1/namespaces/mount-propagation-9879/pods/master/exec?command=%2Fbin%2Fsh&command=-c&command=umount+%2Fmnt%2Ftest%2Fmaster&container=cntr&container=cntr&stderr=true&stdout=true) Mar 5 16:47:20.088: INFO: Exec stderr: "" Mar 5 16:52:20.209: INFO: Failed inside E2E framework: k8s.io/kubernetes/test/e2e/framework/pod.WaitTimeoutForPodRunningInNamespace({0x7f9bbc567d78, 0xc001a90180}, {0x8252218?, 0xc0080a2b60?}, {0xc001d62b40, 0x22}, {0xc000743440, 0x16}, 0x0?) test/e2e/framework/pod/wait.go:460 +0x1a4 k8s.io/kubernetes/test/e2e/storage/utils.(*hostExecutor).launchNodeExecPod(0xc001f082d8?, {0x7f9bbc567d78, 0xc001a90180}, {0xc0023e4708, 0x13}) test/e2e/storage/utils/host_exec.go:116 +0x49d k8s.io/kubernetes/test/e2e/storage/utils.(*hostExecutor).exec(0xc000e27240, {0x7f9bbc567d78, 0xc001a90180}, {0xc001d62180, 0x30}, 0xc0003f38c0) test/e2e/storage/utils/host_exec.go:143 +0x135 ... skipping 4 lines ... k8s.io/kubernetes/test/e2e/node.glob..func7.1.1({0x7f9bbc567d78, 0xc001a90180}) test/e2e/node/mount_propagation.go:113 +0xad reflect.Value.call({0x6a73c20?, 0xc0047e13b0?, 0xc0059cbea0?}, {0x781932d, 0x4}, {0xc0059cbf60, 0x1, 0xc0038f5ba0?}) /usr/local/go/src/reflect/value.go:586 +0xb07 reflect.Value.Call({0x6a73c20?, 0xc0047e13b0?, 0x267cb38?}, {0xc0059cbf60?, 0xc0059cbf48?, 0xc002aa8b01?}) /usr/local/go/src/reflect/value.go:370 +0xbc [38;5;9m[FAILED][0m in [DeferCleanup (Each)] - test/e2e/storage/utils/host_exec.go:117 [38;5;243m@ 03/05/23 16:52:20.21[0m [1mSTEP:[0m dump namespace information after failure [38;5;243m@ 03/05/23 16:52:20.21[0m [1mSTEP:[0m Collecting events from namespace "mount-propagation-9879". [38;5;243m@ 03/05/23 16:52:20.21[0m [1mSTEP:[0m Found 30 events. [38;5;243m@ 03/05/23 16:52:20.432[0m Mar 5 16:52:20.432: INFO: At 2023-03-05 16:31:35 +0000 UTC - event for master: {kubelet i-06df568f6e5c59fff} Created: Created container cntr Mar 5 16:52:20.432: INFO: At 2023-03-05 16:31:35 +0000 UTC - event for master: {kubelet i-06df568f6e5c59fff} Pulled: Container image "registry.k8s.io/e2e-test-images/busybox:1.29-4" already present on machine Mar 5 16:52:20.432: INFO: At 2023-03-05 16:31:35 +0000 UTC - event for master: {kubelet i-06df568f6e5c59fff} Started: Started container cntr ... skipping 28 lines ... Mar 5 16:52:20.545: INFO: default i-06df568f6e5c59fff Running 1s [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2023-03-05 16:31:41 +0000 UTC } {Ready False 0001-01-01 00:00:00 +0000 UTC 2023-03-05 16:33:20 +0000 UTC } {ContainersReady True 0001-01-01 00:00:00 +0000 UTC 2023-03-05 16:31:43 +0000 UTC } {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2023-03-05 16:31:41 +0000 UTC } {DisruptionTarget True 0001-01-01 00:00:00 +0000 UTC 2023-03-05 16:47:57 +0000 UTC DeletionByTaintManager Taint manager: deleting due to NoExecute taint}] Mar 5 16:52:20.545: INFO: hostexec-i-06df568f6e5c59fff-n78gw Pending [{PodScheduled False 0001-01-01 00:00:00 +0000 UTC 2023-03-05 16:47:20 +0000 UTC Unschedulable 0/3 nodes are available: 1 node(s) had untolerated taint {node.kubernetes.io/not-ready: }. preemption: 0/3 nodes are available: 1 Preemption is not helpful for scheduling, 2 No preemption victims found for incoming pod..}] Mar 5 16:52:20.545: INFO: master i-06df568f6e5c59fff Running 1s [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2023-03-05 16:31:34 +0000 UTC } {Ready False 0001-01-01 00:00:00 +0000 UTC 2023-03-05 16:33:20 +0000 UTC } {ContainersReady True 0001-01-01 00:00:00 +0000 UTC 2023-03-05 16:31:35 +0000 UTC } {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2023-03-05 16:31:34 +0000 UTC } {DisruptionTarget True 0001-01-01 00:00:00 +0000 UTC 2023-03-05 16:47:57 +0000 UTC DeletionByTaintManager Taint manager: deleting due to NoExecute taint}] Mar 5 16:52:20.545: INFO: private i-06df568f6e5c59fff Running 1s [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2023-03-05 16:31:39 +0000 UTC } {Ready False 0001-01-01 00:00:00 +0000 UTC 2023-03-05 16:33:21 +0000 UTC } {ContainersReady True 0001-01-01 00:00:00 +0000 UTC 2023-03-05 16:31:40 +0000 UTC } {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2023-03-05 16:31:39 +0000 UTC } {DisruptionTarget True 0001-01-01 00:00:00 +0000 UTC 2023-03-05 16:47:57 +0000 UTC DeletionByTaintManager Taint manager: deleting due to NoExecute taint}] Mar 5 16:52:20.545: INFO: slave i-06df568f6e5c59fff Running 1s [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2023-03-05 16:31:36 +0000 UTC } {Ready False 0001-01-01 00:00:00 +0000 UTC 2023-03-05 16:33:20 +0000 UTC } {ContainersReady True 0001-01-01 00:00:00 +0000 UTC 2023-03-05 16:31:37 +0000 UTC } {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2023-03-05 16:31:36 +0000 UTC } {DisruptionTarget True 0001-01-01 00:00:00 +0000 UTC 2023-03-05 16:47:58 +0000 UTC DeletionByTaintManager Taint manager: deleting due to NoExecute taint}] Mar 5 16:52:20.545: INFO: Mar 5 16:52:30.661: INFO: Unable to fetch mount-propagation-9879/default/cntr logs: an error on the server ("unknown") has prevented the request from succeeding (get pods default) Mar 5 16:52:40.887: INFO: Unable to fetch mount-propagation-9879/master/cntr logs: an error on the server ("unknown") has prevented the request from succeeding (get pods master) Mar 5 16:52:51.001: INFO: Unable to fetch mount-propagation-9879/private/cntr logs: an error on the server ("unknown") has prevented the request from succeeding (get pods private) Mar 5 16:53:01.114: INFO: Unable to fetch mount-propagation-9879/slave/cntr logs: an error on the server ("unknown") has prevented the request from succeeding (get pods slave) Mar 5 16:53:01.227: INFO: Logging node info for node i-056ad61525499081e Mar 5 16:53:01.339: INFO: Node Info: &Node{ObjectMeta:{i-056ad61525499081e a6ce9ed3-5ae8-4065-b117-6884e24b3632 19980 0 2023-03-05 16:24:16 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:arm64 beta.kubernetes.io/instance-type:t4g.small beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:eu-central-1 failure-domain.beta.kubernetes.io/zone:eu-central-1a karpenter.k8s.aws/instance-category:t karpenter.k8s.aws/instance-cpu:2 karpenter.k8s.aws/instance-family:t4g karpenter.k8s.aws/instance-generation:4 karpenter.k8s.aws/instance-hypervisor:nitro karpenter.k8s.aws/instance-memory:2048 karpenter.k8s.aws/instance-pods:11 karpenter.k8s.aws/instance-size:small karpenter.sh/capacity-type:spot karpenter.sh/initialized:true karpenter.sh/provisioner-name:nodes kubernetes.io/arch:arm64 kubernetes.io/hostname:i-056ad61525499081e kubernetes.io/os:linux node-role.kubernetes.io/node: node-role.kubernetes.io/spot-worker:true node.kubernetes.io/instance-type:t4g.small topology.ebs.csi.aws.com/zone:eu-central-1a topology.hostpath.csi/node:i-056ad61525499081e topology.kubernetes.io/region:eu-central-1 topology.kubernetes.io/zone:eu-central-1a] map[csi.volume.kubernetes.io/nodeid:{"csi-hostpath-read-write-once-pod-5175":"i-056ad61525499081e","csi-mock-csi-mock-volumes-expansion-4933":"csi-mock-csi-mock-volumes-expansion-4933","csi-mock-csi-mock-volumes-fsgroup-policy-6623":"i-056ad61525499081e","ebs.csi.aws.com":"i-056ad61525499081e"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [{karpenter.sh/v1alpha5 Provisioner nodes ac5d7599-b2a2-4eae-8da4-1f4c0a54ec82 <nil> 0xc0017b5db0}] [karpenter.sh/termination] [{aws-cloud-controller-manager Update v1 2023-03-05 16:24:16 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{}}}} } {kops-controller Update v1 2023-03-05 16:24:16 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{},"f:node-role.kubernetes.io/spot-worker":{}}}} } {karpenter Update v1 2023-03-05 16:25:57 +0000 UTC FieldsV1 {"f:metadata":{"f:finalizers":{".":{},"v:\"karpenter.sh/termination\"":{}},"f:labels":{".":{},"f:karpenter.k8s.aws/instance-category":{},"f:karpenter.k8s.aws/instance-cpu":{},"f:karpenter.k8s.aws/instance-family":{},"f:karpenter.k8s.aws/instance-generation":{},"f:karpenter.k8s.aws/instance-hypervisor":{},"f:karpenter.k8s.aws/instance-memory":{},"f:karpenter.k8s.aws/instance-pods":{},"f:karpenter.k8s.aws/instance-size":{},"f:karpenter.sh/capacity-type":{},"f:karpenter.sh/initialized":{},"f:karpenter.sh/provisioner-name":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}},"f:ownerReferences":{".":{},"k:{\"uid\":\"ac5d7599-b2a2-4eae-8da4-1f4c0a54ec82\"}":{}}},"f:spec":{"f:providerID":{}}} } {kube-controller-manager Update v1 2023-03-05 16:25:57 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:node.alpha.kubernetes.io/ttl":{}},"f:labels":{"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.1.0/24\"":{}}}} } {aws-cloud-controller-manager Update v1 2023-03-05 16:26:48 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{".":{},"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"InternalIP\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kube-controller-manager Update v1 2023-03-05 16:52:18 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{".":{},"k:{\"type\":\"DiskPressure\"}":{".":{},"f:type":{}},"k:{\"type\":\"MemoryPressure\"}":{".":{},"f:type":{}},"k:{\"type\":\"PIDPressure\"}":{".":{},"f:type":{}},"k:{\"type\":\"Ready\"}":{".":{},"f:type":{}}},"f:volumesAttached":{}}} status} {kubelet Update v1 2023-03-05 16:52:21 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{"f:kubernetes.io/hostname":{},"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{".":{},"f:cpu":{},"f:ephemeral-storage":{},"f:hugepages-1Gi":{},"f:hugepages-2Mi":{},"f:hugepages-32Mi":{},"f:hugepages-64Ki":{},"f:memory":{},"f:pods":{}},"f:capacity":{".":{},"f:cpu":{},"f:ephemeral-storage":{},"f:hugepages-1Gi":{},"f:hugepages-2Mi":{},"f:hugepages-32Mi":{},"f:hugepages-64Ki":{},"f:memory":{},"f:pods":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:daemonEndpoints":{"f:kubeletEndpoint":{"f:Port":{}}},"f:images":{},"f:nodeInfo":{"f:architecture":{},"f:bootID":{},"f:containerRuntimeVersion":{},"f:kernelVersion":{},"f:kubeProxyVersion":{},"f:kubeletVersion":{},"f:machineID":{},"f:operatingSystem":{},"f:osImage":{},"f:systemUUID":{}},"f:volumesInUse":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.1.0/24,DoNotUseExternalID:,ProviderID:aws:///eu-central-1a/i-056ad61525499081e,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.1.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49756430336 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-32Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-64Ki: {{0 0} {<nil>} 0 DecimalSI},memory: {{1941540864 0} {<nil>} 1896036Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44780787229 0} {<nil>} 44780787229 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-32Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-64Ki: {{0 0} {<nil>} 0 DecimalSI},memory: {{1836683264 0} {<nil>} 1793636Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-03-05 16:52:21 +0000 UTC,LastTransitionTime:2023-03-05 16:25:57 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-03-05 16:52:21 +0000 UTC,LastTransitionTime:2023-03-05 16:25:48 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-03-05 16:52:21 +0000 UTC,LastTransitionTime:2023-03-05 16:25:48 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-03-05 16:52:21 +0000 UTC,LastTransitionTime:2023-03-05 16:25:48 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.36.179,},NodeAddress{Type:ExternalIP,Address:18.159.37.200,},NodeAddress{Type:InternalDNS,Address:i-056ad61525499081e.eu-central-1.compute.internal,},NodeAddress{Type:Hostname,Address:i-056ad61525499081e.eu-central-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-18-159-37-200.eu-central-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2af3207d1c8c09785c337aae6c6ef0,SystemUUID:ec2af320-7d1c-8c09-785c-337aae6c6ef0,BootID:786f2752-bc30-4729-8aaf-45977b16c253,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 22.04.2 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.27.0-alpha.3.52+d48b8167f7094b,KubeProxyVersion:v1.27.0-alpha.3.52+d48b8167f7094b,OperatingSystem:linux,Architecture:arm64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:157636062,},ContainerImage{Names:[registry.k8s.io/kube-proxy-arm64:v1.27.0-alpha.3.52_d48b8167f7094b],SizeBytes:60554082,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:16bbf38c463a4223d8cfe4da12bc61010b082a79b4bb003e2d3ba3ece5dd5f9e registry.k8s.io/e2e-test-images/agnhost:2.43],SizeBytes:49808364,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:3fe7acf013d1264ffded116b80a73dc129a449b0fccdb8d21af8279f2233f36e registry.k8s.io/e2e-test-images/httpd:2.4.39-4],SizeBytes:41528818,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:148b022f5c5da426fc2f3c14b5c0867e58ef05961510c84749ac1fddcb0fef22 registry.k8s.io/e2e-test-images/httpd:2.4.38-4],SizeBytes:40343601,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:28932904,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:ee3b525d5b89db99da3b8eb521d9cd90cb6e9ef0fbb651e98bb37be78d36b5b8 registry.k8s.io/sig-storage/csi-provisioner:v3.3.0],SizeBytes:23945474,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:425d8f1b769398127767b06ed97ce62578a3179bcb99809ce93a1649e025ffe7 registry.k8s.io/sig-storage/csi-resizer:v1.6.0],SizeBytes:22678632,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:291334908ddf71a4661fd7f6d9d97274de8a5378a2b6fdfeb2ce73414a34f82f registry.k8s.io/sig-storage/csi-snapshotter:v6.1.0],SizeBytes:22434180,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:9a685020911e2725ad019dbce6e4a5ab93d51e3d4557f115e64343345e05781b registry.k8s.io/sig-storage/csi-attacher:v4.0.0],SizeBytes:22394501,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:122bfb8c1edabb3c0edd63f06523e6940d958d19b3957dc7b1d6f81e9f1f6119 registry.k8s.io/sig-storage/csi-provisioner:v3.1.0],SizeBytes:21735644,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:9ebbf9f023e7b41ccee3d52afe39a89e3ddacdbb69269d583abfc25847cfd9e4 registry.k8s.io/sig-storage/csi-resizer:v1.4.0],SizeBytes:20848413,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:8b9c313c05f54fb04f8d430896f5f5904b6cb157df261501b29adc04d2b2dc7b registry.k8s.io/sig-storage/csi-attacher:v3.4.0],SizeBytes:20565480,},ContainerImage{Names:[registry.k8s.io/cpa/cluster-proportional-autoscaler@sha256:68d396900aeaa072c1f27289485fdac29834045a6f3ffe369bf389d830ef572d registry.k8s.io/cpa/cluster-proportional-autoscaler:1.8.6],SizeBytes:19007623,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:92257881c1d6493cf18299a24af42330f891166560047902b8d431fb66b01af5 registry.k8s.io/sig-storage/hostpathplugin:v1.9.0],SizeBytes:18267611,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:9b2d9abda017c28b12f84a344f57cd73fbdb6c2bd7dd5bdd5018246ad1093ba6 registry.k8s.io/sig-storage/hostpathplugin:v1.11.0],SizeBytes:18159085,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:017727efcfeb7d053af68e51436ce8e65edbc6ca573720afb4f79c8594036955 registry.k8s.io/coredns/coredns:v1.10.0],SizeBytes:13766234,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:8502345,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:933940f13b3ea0abc62e656c1aa5c5b47c04b15d71250413a6b821bd0c58b94e registry.k8s.io/sig-storage/livenessprobe:v2.7.0],SizeBytes:8223819,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:7685305,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:5c99cf6a02adda929b10321dbf4ecfa00d87be9ba4fb456006237d530ab4baa1 registry.k8s.io/e2e-test-images/nginx:1.14-4],SizeBytes:6860089,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf registry.k8s.io/e2e-test-images/busybox:1.29-2],SizeBytes:773367,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:2e0f836850e09b8b7cc937681d6194537a09fbd5f6b9e08f4d646a85128e8937 registry.k8s.io/e2e-test-images/busybox:1.29-4],SizeBytes:772933,},ContainerImage{Names:[registry.k8s.io/pause@sha256:7031c1b283388d2c2e09b57badb803c05ebed362dc88d84b480cc47f72a21097 registry.k8s.io/pause:3.9],SizeBytes:268051,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:253553,},},VolumesInUse:[kubernetes.io/csi/csi-hostpath-read-write-once-pod-5175^bc2b4edf-bb75-11ed-8d33-a27fee729fe2 kubernetes.io/csi/csi-mock-csi-mock-volumes-expansion-4933^4 kubernetes.io/csi/ebs.csi.aws.com^vol-0e020446b1cd6abaf],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/csi-mock-csi-mock-volumes-expansion-4933^4,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-0e020446b1cd6abaf,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/csi-hostpath-read-write-once-pod-5175^bc2b4edf-bb75-11ed-8d33-a27fee729fe2,DevicePath:,},},Config:nil,},} Mar 5 16:53:01.340: INFO: Logging kubelet events for node i-056ad61525499081e Mar 5 16:53:01.455: INFO: ... skipping 57 lines ... Logging node info for node i-06df568f6e5c59fff Mar 5 16:53:02.291: INFO: Node Info: &Node{ObjectMeta:{i-06df568f6e5c59fff cb63c681-2e18-474a-b6f3-221325ac0d52 18792 0 2023-03-05 16:24:16 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:arm64 beta.kubernetes.io/instance-type:t4g.small beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:eu-central-1 failure-domain.beta.kubernetes.io/zone:eu-central-1a karpenter.k8s.aws/instance-category:t karpenter.k8s.aws/instance-cpu:2 karpenter.k8s.aws/instance-family:t4g karpenter.k8s.aws/instance-generation:4 karpenter.k8s.aws/instance-hypervisor:nitro karpenter.k8s.aws/instance-memory:2048 karpenter.k8s.aws/instance-pods:11 karpenter.k8s.aws/instance-size:small karpenter.sh/capacity-type:spot karpenter.sh/initialized:true karpenter.sh/provisioner-name:nodes kubernetes.io/arch:arm64 kubernetes.io/hostname:i-06df568f6e5c59fff kubernetes.io/os:linux node-role.kubernetes.io/node: node-role.kubernetes.io/spot-worker:true node.kubernetes.io/instance-type:t4g.small topology.ebs.csi.aws.com/zone:eu-central-1a topology.hostpath.csi/node:i-06df568f6e5c59fff topology.kubernetes.io/region:eu-central-1 topology.kubernetes.io/zone:eu-central-1a] map[csi.volume.kubernetes.io/nodeid:{"csi-hostpath-ephemeral-2147":"i-06df568f6e5c59fff","csi-hostpath-ephemeral-5408":"i-06df568f6e5c59fff","csi-hostpath-ephemeral-6540":"i-06df568f6e5c59fff","csi-mock-csi-mock-volumes-fsgroup-policy-1058":"i-06df568f6e5c59fff","csi-mock-csi-mock-volumes-workload-2219":"i-06df568f6e5c59fff","ebs.csi.aws.com":"i-06df568f6e5c59fff"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [{karpenter.sh/v1alpha5 Provisioner nodes ac5d7599-b2a2-4eae-8da4-1f4c0a54ec82 <nil> 0xc004baf320}] [karpenter.sh/termination] [{aws-cloud-controller-manager Update v1 2023-03-05 16:24:16 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{}}}} } {kops-controller Update v1 2023-03-05 16:24:16 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{},"f:node-role.kubernetes.io/spot-worker":{}}}} } {karpenter Update v1 2023-03-05 16:25:57 +0000 UTC FieldsV1 {"f:metadata":{"f:finalizers":{".":{},"v:\"karpenter.sh/termination\"":{}},"f:labels":{".":{},"f:karpenter.k8s.aws/instance-category":{},"f:karpenter.k8s.aws/instance-cpu":{},"f:karpenter.k8s.aws/instance-family":{},"f:karpenter.k8s.aws/instance-generation":{},"f:karpenter.k8s.aws/instance-hypervisor":{},"f:karpenter.k8s.aws/instance-memory":{},"f:karpenter.k8s.aws/instance-pods":{},"f:karpenter.k8s.aws/instance-size":{},"f:karpenter.sh/capacity-type":{},"f:karpenter.sh/initialized":{},"f:karpenter.sh/provisioner-name":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}},"f:ownerReferences":{".":{},"k:{\"uid\":\"ac5d7599-b2a2-4eae-8da4-1f4c0a54ec82\"}":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-03-05 16:26:48 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{".":{},"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"InternalIP\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kubelet Update v1 2023-03-05 16:48:40 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{"f:kubernetes.io/hostname":{},"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{".":{},"f:cpu":{},"f:ephemeral-storage":{},"f:hugepages-1Gi":{},"f:hugepages-2Mi":{},"f:hugepages-32Mi":{},"f:hugepages-64Ki":{},"f:memory":{},"f:pods":{}},"f:capacity":{".":{},"f:cpu":{},"f:ephemeral-storage":{},"f:hugepages-1Gi":{},"f:hugepages-2Mi":{},"f:hugepages-32Mi":{},"f:hugepages-64Ki":{},"f:memory":{},"f:pods":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{}}},"f:daemonEndpoints":{"f:kubeletEndpoint":{"f:Port":{}}},"f:images":{},"f:nodeInfo":{"f:architecture":{},"f:bootID":{},"f:containerRuntimeVersion":{},"f:kernelVersion":{},"f:kubeProxyVersion":{},"f:kubeletVersion":{},"f:machineID":{},"f:operatingSystem":{},"f:osImage":{},"f:systemUUID":{}},"f:volumesInUse":{}}} status} {kube-controller-manager Update v1 2023-03-05 16:49:37 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{".":{},"k:{\"type\":\"DiskPressure\"}":{".":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"MemoryPressure\"}":{".":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"PIDPressure\"}":{".":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Ready\"}":{".":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}},"f:volumesAttached":{}}} status} {kube-controller-manager Update v1 2023-03-05 16:49:42 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:node.alpha.kubernetes.io/ttl":{}},"f:labels":{"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.2.0/24\"":{}},"f:taints":{}}} }]},Spec:NodeSpec{PodCIDR:100.96.2.0/24,DoNotUseExternalID:,ProviderID:aws:///eu-central-1a/i-06df568f6e5c59fff,Unschedulable:false,Taints:[]Taint{Taint{Key:node.kubernetes.io/unreachable,Value:,Effect:NoSchedule,TimeAdded:2023-03-05 16:49:37 +0000 UTC,},Taint{Key:node.kubernetes.io/unreachable,Value:,Effect:NoExecute,TimeAdded:2023-03-05 16:49:42 +0000 UTC,},},ConfigSource:nil,PodCIDRs:[100.96.2.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49756430336 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-32Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-64Ki: {{0 0} {<nil>} 0 DecimalSI},memory: {{1941540864 0} {<nil>} 1896036Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44780787229 0} {<nil>} 44780787229 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-32Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-64Ki: {{0 0} {<nil>} 0 DecimalSI},memory: {{1836683264 0} {<nil>} 1793636Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:Ready,Status:Unknown,LastHeartbeatTime:2023-03-05 16:47:34 +0000 UTC,LastTransitionTime:2023-03-05 16:49:37 +0000 UTC,Reason:NodeStatusUnknown,Message:Kubelet stopped posting node status.,},NodeCondition{Type:MemoryPressure,Status:Unknown,LastHeartbeatTime:2023-03-05 16:47:34 +0000 UTC,LastTransitionTime:2023-03-05 16:49:37 +0000 UTC,Reason:NodeStatusUnknown,Message:Kubelet stopped posting node status.,},NodeCondition{Type:DiskPressure,Status:Unknown,LastHeartbeatTime:2023-03-05 16:47:34 +0000 UTC,LastTransitionTime:2023-03-05 16:49:37 +0000 UTC,Reason:NodeStatusUnknown,Message:Kubelet stopped posting node status.,},NodeCondition{Type:PIDPressure,Status:Unknown,LastHeartbeatTime:2023-03-05 16:47:34 +0000 UTC,LastTransitionTime:2023-03-05 16:49:37 +0000 UTC,Reason:NodeStatusUnknown,Message:Kubelet stopped posting node status.,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.43.148,},NodeAddress{Type:ExternalIP,Address:18.192.211.222,},NodeAddress{Type:InternalDNS,Address:i-06df568f6e5c59fff.eu-central-1.compute.internal,},NodeAddress{Type:Hostname,Address:i-06df568f6e5c59fff.eu-central-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-18-192-211-222.eu-central-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec221221864d9409a8ec5f6614a9b94b,SystemUUID:ec221221-864d-9409-a8ec-5f6614a9b94b,BootID:553d31bf-faca-417e-a3db-325839e1b8e4,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 22.04.2 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.27.0-alpha.3.52+d48b8167f7094b,KubeProxyVersion:v1.27.0-alpha.3.52+d48b8167f7094b,OperatingSystem:linux,Architecture:arm64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:157636062,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/jessie-dnsutils@sha256:24aaf2626d6b27864c29de2097e8bbb840b3a414271bf7c8995e431e47d8408e registry.k8s.io/e2e-test-images/jessie-dnsutils:1.7],SizeBytes:116764683,},ContainerImage{Names:[registry.k8s.io/kube-proxy-arm64:v1.27.0-alpha.3.52_d48b8167f7094b],SizeBytes:60554082,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:16bbf38c463a4223d8cfe4da12bc61010b082a79b4bb003e2d3ba3ece5dd5f9e registry.k8s.io/e2e-test-images/agnhost:2.43],SizeBytes:49808364,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:3fe7acf013d1264ffded116b80a73dc129a449b0fccdb8d21af8279f2233f36e registry.k8s.io/e2e-test-images/httpd:2.4.39-4],SizeBytes:41528818,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:148b022f5c5da426fc2f3c14b5c0867e58ef05961510c84749ac1fddcb0fef22 registry.k8s.io/e2e-test-images/httpd:2.4.38-4],SizeBytes:40343601,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:28932904,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:ee3b525d5b89db99da3b8eb521d9cd90cb6e9ef0fbb651e98bb37be78d36b5b8 registry.k8s.io/sig-storage/csi-provisioner:v3.3.0],SizeBytes:23945474,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:425d8f1b769398127767b06ed97ce62578a3179bcb99809ce93a1649e025ffe7 registry.k8s.io/sig-storage/csi-resizer:v1.6.0],SizeBytes:22678632,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:291334908ddf71a4661fd7f6d9d97274de8a5378a2b6fdfeb2ce73414a34f82f registry.k8s.io/sig-storage/csi-snapshotter:v6.1.0],SizeBytes:22434180,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:9a685020911e2725ad019dbce6e4a5ab93d51e3d4557f115e64343345e05781b registry.k8s.io/sig-storage/csi-attacher:v4.0.0],SizeBytes:22394501,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:122bfb8c1edabb3c0edd63f06523e6940d958d19b3957dc7b1d6f81e9f1f6119 registry.k8s.io/sig-storage/csi-provisioner:v3.1.0],SizeBytes:21735644,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:9ebbf9f023e7b41ccee3d52afe39a89e3ddacdbb69269d583abfc25847cfd9e4 registry.k8s.io/sig-storage/csi-resizer:v1.4.0],SizeBytes:20848413,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:8b9c313c05f54fb04f8d430896f5f5904b6cb157df261501b29adc04d2b2dc7b registry.k8s.io/sig-storage/csi-attacher:v3.4.0],SizeBytes:20565480,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nonroot@sha256:ee9f50b3c64b174d296d91ca9f69a914ac30e59095dfb462b2b518ad28a63655 registry.k8s.io/e2e-test-images/nonroot:1.4],SizeBytes:18561972,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:92257881c1d6493cf18299a24af42330f891166560047902b8d431fb66b01af5 registry.k8s.io/sig-storage/hostpathplugin:v1.9.0],SizeBytes:18267611,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:9b2d9abda017c28b12f84a344f57cd73fbdb6c2bd7dd5bdd5018246ad1093ba6 registry.k8s.io/sig-storage/hostpathplugin:v1.11.0],SizeBytes:18159085,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:017727efcfeb7d053af68e51436ce8e65edbc6ca573720afb4f79c8594036955 registry.k8s.io/coredns/coredns:v1.10.0],SizeBytes:13766234,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:8502345,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:933940f13b3ea0abc62e656c1aa5c5b47c04b15d71250413a6b821bd0c58b94e registry.k8s.io/sig-storage/livenessprobe:v2.7.0],SizeBytes:8223819,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:7685305,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:5c99cf6a02adda929b10321dbf4ecfa00d87be9ba4fb456006237d530ab4baa1 registry.k8s.io/e2e-test-images/nginx:1.14-4],SizeBytes:6860089,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nonewprivs@sha256:8ac1264691820febacf3aea5d152cbde6d10685731ec14966a9401c6f47a68ac registry.k8s.io/e2e-test-images/nonewprivs:1.3],SizeBytes:5025027,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:2e0f836850e09b8b7cc937681d6194537a09fbd5f6b9e08f4d646a85128e8937 registry.k8s.io/e2e-test-images/busybox:1.29-4],SizeBytes:772933,},ContainerImage{Names:[registry.k8s.io/pause@sha256:7031c1b283388d2c2e09b57badb803c05ebed362dc88d84b480cc47f72a21097 registry.k8s.io/pause:3.9],SizeBytes:268051,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:253553,},},VolumesInUse:[kubernetes.io/csi/csi-hostpath-ephemeral-2147^52df5953-bb73-11ed-8aa7-be49170871b4 kubernetes.io/csi/csi-hostpath-ephemeral-5408^443d1daf-bb73-11ed-95c4-5a95d17cbc27 kubernetes.io/csi/csi-hostpath-ephemeral-5408^44411f51-bb73-11ed-95c4-5a95d17cbc27 kubernetes.io/csi/csi-hostpath-ephemeral-6540^4426a07f-bb73-11ed-a604-92687f2c50f8 kubernetes.io/csi/ebs.csi.aws.com^vol-05b67fab5beb6394b],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-05b67fab5beb6394b,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/csi-hostpath-ephemeral-6540^4426a07f-bb73-11ed-a604-92687f2c50f8,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/csi-hostpath-ephemeral-5408^44411f51-bb73-11ed-95c4-5a95d17cbc27,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/csi-hostpath-ephemeral-5408^443d1daf-bb73-11ed-95c4-5a95d17cbc27,DevicePath:,},},Config:nil,},} Mar 5 16:53:02.291: INFO: Logging kubelet events for node i-06df568f6e5c59fff Mar 5 16:53:02.406: INFO: Logging pods the kubelet thinks is on node i-06df568f6e5c59fff Mar 5 16:53:12.520: INFO: Unable to retrieve kubelet pods for node i-06df568f6e5c59fff: error trying to reach service: net/http: TLS handshake timeout Mar 5 16:53:12.520: INFO: Logging node info for node i-0d6a32f749839192f Mar 5 16:53:12.632: INFO: Node Info: &Node{ObjectMeta:{i-0d6a32f749839192f 4155fefb-4d3a-4b04-ba53-6ac9e895ff92 18509 0 2023-03-05 16:20:46 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:arm64 beta.kubernetes.io/instance-type:c6g.xlarge beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:eu-central-1 failure-domain.beta.kubernetes.io/zone:eu-central-1a kops.k8s.io/kops-controller-pki: kubernetes.io/arch:arm64 kubernetes.io/hostname:i-0d6a32f749839192f kubernetes.io/os:linux node-role.kubernetes.io/control-plane: node.kubernetes.io/exclude-from-external-load-balancers: node.kubernetes.io/instance-type:c6g.xlarge topology.ebs.csi.aws.com/zone:eu-central-1a topology.kubernetes.io/region:eu-central-1 topology.kubernetes.io/zone:eu-central-1a] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-0d6a32f749839192f"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kubelet Update v1 2023-03-05 16:20:46 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {protokube Update v1 2023-03-05 16:21:25 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kops.k8s.io/kops-controller-pki":{},"f:node-role.kubernetes.io/control-plane":{},"f:node.kubernetes.io/exclude-from-external-load-balancers":{}}}} } {kube-controller-manager Update v1 2023-03-05 16:21:47 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.0.0/24\"":{}}}} } {aws-cloud-controller-manager Update v1 2023-03-05 16:21:48 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{},"f:taints":{}}} } {aws-cloud-controller-manager Update v1 2023-03-05 16:21:48 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kubelet Update v1 2023-03-05 16:49:18 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.0.0/24,DoNotUseExternalID:,ProviderID:aws:///eu-central-1a/i-0d6a32f749839192f,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/control-plane,Value:,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[100.96.0.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{4 0} {<nil>} 4 DecimalSI},ephemeral-storage: {{49756430336 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-32Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-64Ki: {{0 0} {<nil>} 0 DecimalSI},memory: {{8150118400 0} {<nil>} 7959100Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{4 0} {<nil>} 4 DecimalSI},ephemeral-storage: {{44780787229 0} {<nil>} 44780787229 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-32Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-64Ki: {{0 0} {<nil>} 0 DecimalSI},memory: {{8045260800 0} {<nil>} 7856700Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-03-05 16:49:18 +0000 UTC,LastTransitionTime:2023-03-05 16:20:46 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-03-05 16:49:18 +0000 UTC,LastTransitionTime:2023-03-05 16:20:46 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-03-05 16:49:18 +0000 UTC,LastTransitionTime:2023-03-05 16:20:46 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-03-05 16:49:18 +0000 UTC,LastTransitionTime:2023-03-05 16:21:47 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.49.247,},NodeAddress{Type:ExternalIP,Address:18.197.20.82,},NodeAddress{Type:InternalDNS,Address:i-0d6a32f749839192f.eu-central-1.compute.internal,},NodeAddress{Type:Hostname,Address:i-0d6a32f749839192f.eu-central-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-18-197-20-82.eu-central-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2bed249e8ac7b517cb08ccd42b83e5,SystemUUID:ec2bed24-9e8a-c7b5-17cb-08ccd42b83e5,BootID:6f9215c9-dfe8-4b13-84f4-9b2084456198,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 22.04.2 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.27.0-alpha.3.52+d48b8167f7094b,KubeProxyVersion:v1.27.0-alpha.3.52+d48b8167f7094b,OperatingSystem:linux,Architecture:arm64,},Images:[]ContainerImage{ContainerImage{Names:[registry.k8s.io/etcd@sha256:9ce33ba33d8e738a5b85ed50b5080ac746deceed4a7496c550927a7a19ca3b6d registry.k8s.io/etcd:3.5.0-0],SizeBytes:157797353,},ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:157636062,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:4afb99b4690b418ffc2ceb67e1a17376457e441c1f09ab55447f0aaf992fa646 registry.k8s.io/etcd:3.4.3-0],SizeBytes:151677080,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:4ad90a11b55313b182afc186b9876c8e891531b8db4c9bf1541953021618d0e2 registry.k8s.io/etcd:3.4.13-0],SizeBytes:134531559,},ContainerImage{Names:[registry.k8s.io/kube-apiserver-arm64:v1.27.0-alpha.3.52_d48b8167f7094b],SizeBytes:131087707,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:0d2ab6b15b4fe0b903b9e81192d942fb39c29be61fe341cd16db5d0165b0435c registry.k8s.io/etcd:3.3.17-0],SizeBytes:126979107,},ContainerImage{Names:[registry.k8s.io/kube-controller-manager-arm64:v1.27.0-alpha.3.52_d48b8167f7094b],SizeBytes:120667128,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:17da501f5d2a675be46040422a27b7cc21b8a43895ac998b171db1c346f361f7 registry.k8s.io/etcd:3.3.10-0],SizeBytes:81436097,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:6f72b851544986cb0921b53ea655ec04c36131248f16d4ad110cb3ca0c369dc1 registry.k8s.io/etcd:3.5.4-0],SizeBytes:81120118,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:13f53ed1d91e2e11aac476ee9a0269fdda6cc4874eba903efd40daf50c55eee5 registry.k8s.io/etcd:3.5.3-0],SizeBytes:81114891,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:51eae8381dcb1078289fa7b4f3df2630cdc18d09fb56f8e56b41c40e191d6c83 registry.k8s.io/etcd:3.5.7-0],SizeBytes:80665728,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:dd75ec974b0a2a6f6bb47001ba09207976e625db898d1b16735528c009cb171c registry.k8s.io/etcd:3.5.6-0],SizeBytes:80539316,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:905d7ca17fd02bc24c0eba9a062753aba15db3e31422390bc3238eb762339b20 registry.k8s.io/etcd:3.2.24-1],SizeBytes:61500433,},ContainerImage{Names:[registry.k8s.io/etcdadm/etcd-manager-slim@sha256:8727adc059334e0e3329dbccce116a734da4a090da5e86c8cddccaf333caa666 registry.k8s.io/etcdadm/etcd-manager-slim:v3.0.20230201],SizeBytes:60826655,},ContainerImage{Names:[registry.k8s.io/kube-proxy-arm64:v1.27.0-alpha.3.52_d48b8167f7094b],SizeBytes:60554082,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:64b9ea357325d5db9f8a723dcf503b5a449177b17ac87d69481e126bb724c263 registry.k8s.io/etcd:3.5.1-0],SizeBytes:59553904,},ContainerImage{Names:[registry.k8s.io/kube-scheduler-arm64:v1.27.0-alpha.3.52_d48b8167f7094b],SizeBytes:56310763,},ContainerImage{Names:[registry.k8s.io/kops/kops-controller:1.27.0-alpha.1],SizeBytes:41797411,},ContainerImage{Names:[registry.k8s.io/kops/dns-controller:1.27.0-alpha.1],SizeBytes:40267473,},ContainerImage{Names:[public.ecr.aws/karpenter/controller@sha256:68db4f092cf9cc83f5ef9e2fbc5407c2cb682e81f64dfaa700a7602ede38b1cf public.ecr.aws/karpenter/controller:v0.16.3],SizeBytes:33707751,},ContainerImage{Names:[public.ecr.aws/karpenter/webhook@sha256:96a2d9b06d6bc5127801f358f74b1cf2d289b423a2e9ba40c573c0b14b17dafa public.ecr.aws/karpenter/webhook:v0.16.3],SizeBytes:33307187,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:28932904,},ContainerImage{Names:[quay.io/cilium/operator@sha256:a6d24a006a6b92967ac90786b49bc1ac26e5477cf028cd1186efcfc2466484db quay.io/cilium/operator:v1.12.5],SizeBytes:24567287,},ContainerImage{Names:[registry.k8s.io/provider-aws/cloud-controller-manager@sha256:fdeb61e3e42ecd9cca868d550ebdb88dd6341d9e91fcfa9a37e227dab2ad22cb registry.k8s.io/provider-aws/cloud-controller-manager:v1.26.0],SizeBytes:18345226,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:8502345,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:7685305,},ContainerImage{Names:[registry.k8s.io/kops/kube-apiserver-healthcheck:1.27.0-alpha.1],SizeBytes:4713612,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:253553,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Mar 5 16:53:12.633: INFO: Logging kubelet events for node i-0d6a32f749839192f Mar 5 16:53:12.749: INFO: ... skipping 52 lines ... Mar 5 16:53:12.871: INFO: Container controller ready: true, restart count 0 Mar 5 16:53:13.259: INFO: Latency metrics for node i-0d6a32f749839192f [1mSTEP:[0m Destroying namespace "mount-propagation-9879" for this suite. [38;5;243m@ 03/05/23 16:53:13.259[0m [38;5;243m<< Timeline[0m [38;5;9m[FAILED] Timed out after 300.000s. Expected Pod to be in <v1.PodPhase>: "Running" Got instead: <*v1.Pod | 0xc002344900>: metadata: creationTimestamp: "2023-03-05T16:32:06Z" generateName: hostexec-i-06df568f6e5c59fff- ... skipping 207 lines ... [38;5;9mIn [1m[It][0m[38;5;9m at: [1mtest/e2e/storage/utils/host_exec.go:117[0m [38;5;243m@ 03/05/23 16:37:06.556[0m There were [1m[38;5;9madditional failures[0m detected. To view them in detail run [1mginkgo -vv[0m [38;5;243m------------------------------[0m [38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;10m•[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;10m•[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m [38;5;243m------------------------------[0m [38;5;9m• [FAILED] [691.482 seconds][0m [0m[sig-storage] EmptyDir volumes [38;5;9m[1m[It] should support (non-root,0666,default) [LinuxOnly] [NodeConformance] [Conformance][0m [38;5;243mtest/e2e/common/storage/empty_dir.go:207[0m [38;5;243mTimeline >>[0m [1mSTEP:[0m Creating a kubernetes client [38;5;243m@ 03/05/23 16:42:26.222[0m Mar 5 16:42:26.222: INFO: >>> kubeConfig: /root/.kube/config [1mSTEP:[0m Building a namespace api object, basename emptydir [38;5;243m@ 03/05/23 16:42:26.223[0m [1mSTEP:[0m Waiting for a default service account to be provisioned in namespace [38;5;243m@ 03/05/23 16:42:26.562[0m [1mSTEP:[0m Waiting for kube-root-ca.crt to be provisioned in namespace [38;5;243m@ 03/05/23 16:42:26.785[0m [1mSTEP:[0m Creating a pod to test emptydir 0666 on node default medium [38;5;243m@ 03/05/23 16:42:27.008[0m Mar 5 16:47:34.647: INFO: Failed to get logs from node "i-06df568f6e5c59fff" pod "pod-e43f91ea-b3ff-4cde-9686-99368f334635" container "test-container": the server could not find the requested resource (get pods pod-e43f91ea-b3ff-4cde-9686-99368f334635) [1mSTEP:[0m delete the pod [38;5;243m@ 03/05/23 16:47:34.647[0m Mar 5 16:50:34.766: INFO: Failed inside E2E framework: k8s.io/kubernetes/test/e2e/framework/pod.WaitForPodNotFoundInNamespace({0x7f0b482a2388, 0xc003636440}, {0x8252218?, 0xc0014e2680}, {0xc004e6ffb0, 0x28}, {0xc00486f5a0, 0xd}, 0x0?) test/e2e/framework/pod/wait.go:538 +0x190 k8s.io/kubernetes/test/e2e/framework/pod.(*PodClient).DeleteSync(0xc0010950e0, {0x7f0b482a2388, 0xc003636440}, {0xc004e6ffb0, 0x28}, {{{0x0, 0x0}, {0x0, 0x0}}, 0x0, ...}, ...) test/e2e/framework/pod/pod_client.go:184 +0x1db k8s.io/kubernetes/test/e2e/framework/pod/output.MatchContainerOutput.func1() test/e2e/framework/pod/output/output.go:160 +0xc9 ... skipping 4 lines ... k8s.io/kubernetes/test/e2e/framework/pod/output.TestContainerOutput(...) test/e2e/framework/pod/output/output.go:215 k8s.io/kubernetes/test/e2e/common/storage.doTest0666({0x7f0b482a2388, 0xc003636440}, 0x2681e96?, 0x6f31140?, {0x0, 0x0}) test/e2e/common/storage/empty_dir.go:552 +0x55b k8s.io/kubernetes/test/e2e/common/storage.glob..func4.14({0x7f0b482a2388?, 0xc003636440?}) test/e2e/common/storage/empty_dir.go:208 +0x33 [38;5;9m[FAILED][0m in [It] - test/e2e/framework/pod/pod_client.go:184 [38;5;243m@ 03/05/23 16:50:34.766[0m Mar 5 16:50:34.766: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready Mar 5 16:50:34.880: INFO: Condition Ready of node i-06df568f6e5c59fff is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-05 16:49:37 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-05 16:49:42 +0000 UTC}]. Failure Mar 5 16:50:36.994: INFO: Condition Ready of node i-06df568f6e5c59fff is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-05 16:49:37 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-05 16:49:42 +0000 UTC}]. Failure Mar 5 16:50:38.993: INFO: Condition Ready of node i-06df568f6e5c59fff is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-05 16:49:37 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-05 16:49:42 +0000 UTC}]. Failure Mar 5 16:50:40.994: INFO: Condition Ready of node i-06df568f6e5c59fff is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-05 16:49:37 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-05 16:49:42 +0000 UTC}]. Failure Mar 5 16:50:42.994: INFO: Condition Ready of node i-06df568f6e5c59fff is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-05 16:49:37 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-05 16:49:42 +0000 UTC}]. Failure ... skipping 89 lines ... [1mSTEP:[0m Found 2 events. [38;5;243m@ 03/05/23 16:53:35.221[0m Mar 5 16:53:35.221: INFO: At 2023-03-05 16:42:27 +0000 UTC - event for pod-e43f91ea-b3ff-4cde-9686-99368f334635: {default-scheduler } Scheduled: Successfully assigned emptydir-5780/pod-e43f91ea-b3ff-4cde-9686-99368f334635 to i-06df568f6e5c59fff Mar 5 16:53:35.221: INFO: At 2023-03-05 16:47:57 +0000 UTC - event for pod-e43f91ea-b3ff-4cde-9686-99368f334635: {taint-controller } TaintManagerEviction: Marking for deletion Pod emptydir-5780/pod-e43f91ea-b3ff-4cde-9686-99368f334635 Mar 5 16:53:35.333: INFO: POD NODE PHASE GRACE CONDITIONS Mar 5 16:53:35.333: INFO: pod-e43f91ea-b3ff-4cde-9686-99368f334635 i-06df568f6e5c59fff Pending 30s [{PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2023-03-05 16:42:27 +0000 UTC } {DisruptionTarget True 0001-01-01 00:00:00 +0000 UTC 2023-03-05 16:47:57 +0000 UTC DeletionByTaintManager Taint manager: deleting due to NoExecute taint}] Mar 5 16:53:35.333: INFO: Mar 5 16:53:45.447: INFO: Unable to fetch emptydir-5780/pod-e43f91ea-b3ff-4cde-9686-99368f334635/test-container logs: an error on the server ("unknown") has prevented the request from succeeding (get pods pod-e43f91ea-b3ff-4cde-9686-99368f334635) Mar 5 16:53:45.560: INFO: Logging node info for node i-056ad61525499081e Mar 5 16:53:45.672: INFO: Node Info: &Node{ObjectMeta:{i-056ad61525499081e a6ce9ed3-5ae8-4065-b117-6884e24b3632 20555 0 2023-03-05 16:24:16 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:arm64 beta.kubernetes.io/instance-type:t4g.small beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:eu-central-1 failure-domain.beta.kubernetes.io/zone:eu-central-1a karpenter.k8s.aws/instance-category:t karpenter.k8s.aws/instance-cpu:2 karpenter.k8s.aws/instance-family:t4g karpenter.k8s.aws/instance-generation:4 karpenter.k8s.aws/instance-hypervisor:nitro karpenter.k8s.aws/instance-memory:2048 karpenter.k8s.aws/instance-pods:11 karpenter.k8s.aws/instance-size:small karpenter.sh/capacity-type:spot karpenter.sh/initialized:true karpenter.sh/provisioner-name:nodes kubernetes.io/arch:arm64 kubernetes.io/hostname:i-056ad61525499081e kubernetes.io/os:linux node-role.kubernetes.io/node: node-role.kubernetes.io/spot-worker:true node.kubernetes.io/instance-type:t4g.small topology.ebs.csi.aws.com/zone:eu-central-1a topology.hostpath.csi/node:i-056ad61525499081e topology.kubernetes.io/region:eu-central-1 topology.kubernetes.io/zone:eu-central-1a] map[csi.volume.kubernetes.io/nodeid:{"csi-hostpath-ephemeral-7990":"i-056ad61525499081e","csi-mock-csi-mock-volumes-expansion-4933":"csi-mock-csi-mock-volumes-expansion-4933","csi-mock-csi-mock-volumes-fsgroup-policy-6623":"i-056ad61525499081e","ebs.csi.aws.com":"i-056ad61525499081e"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [{karpenter.sh/v1alpha5 Provisioner nodes ac5d7599-b2a2-4eae-8da4-1f4c0a54ec82 <nil> 0xc000d1dff0}] [karpenter.sh/termination] [{aws-cloud-controller-manager Update v1 2023-03-05 16:24:16 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{}}}} } {kops-controller Update v1 2023-03-05 16:24:16 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{},"f:node-role.kubernetes.io/spot-worker":{}}}} } {karpenter Update v1 2023-03-05 16:25:57 +0000 UTC FieldsV1 {"f:metadata":{"f:finalizers":{".":{},"v:\"karpenter.sh/termination\"":{}},"f:labels":{".":{},"f:karpenter.k8s.aws/instance-category":{},"f:karpenter.k8s.aws/instance-cpu":{},"f:karpenter.k8s.aws/instance-family":{},"f:karpenter.k8s.aws/instance-generation":{},"f:karpenter.k8s.aws/instance-hypervisor":{},"f:karpenter.k8s.aws/instance-memory":{},"f:karpenter.k8s.aws/instance-pods":{},"f:karpenter.k8s.aws/instance-size":{},"f:karpenter.sh/capacity-type":{},"f:karpenter.sh/initialized":{},"f:karpenter.sh/provisioner-name":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}},"f:ownerReferences":{".":{},"k:{\"uid\":\"ac5d7599-b2a2-4eae-8da4-1f4c0a54ec82\"}":{}}},"f:spec":{"f:providerID":{}}} } {kube-controller-manager Update v1 2023-03-05 16:25:57 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:node.alpha.kubernetes.io/ttl":{}},"f:labels":{"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.1.0/24\"":{}}}} } {aws-cloud-controller-manager Update v1 2023-03-05 16:26:48 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{".":{},"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"InternalIP\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kube-controller-manager Update v1 2023-03-05 16:53:22 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{".":{},"k:{\"type\":\"DiskPressure\"}":{".":{},"f:type":{}},"k:{\"type\":\"MemoryPressure\"}":{".":{},"f:type":{}},"k:{\"type\":\"PIDPressure\"}":{".":{},"f:type":{}},"k:{\"type\":\"Ready\"}":{".":{},"f:type":{}}},"f:volumesAttached":{}}} status} {kubelet Update v1 2023-03-05 16:53:26 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{"f:kubernetes.io/hostname":{},"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{".":{},"f:cpu":{},"f:ephemeral-storage":{},"f:hugepages-1Gi":{},"f:hugepages-2Mi":{},"f:hugepages-32Mi":{},"f:hugepages-64Ki":{},"f:memory":{},"f:pods":{}},"f:capacity":{".":{},"f:cpu":{},"f:ephemeral-storage":{},"f:hugepages-1Gi":{},"f:hugepages-2Mi":{},"f:hugepages-32Mi":{},"f:hugepages-64Ki":{},"f:memory":{},"f:pods":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:daemonEndpoints":{"f:kubeletEndpoint":{"f:Port":{}}},"f:images":{},"f:nodeInfo":{"f:architecture":{},"f:bootID":{},"f:containerRuntimeVersion":{},"f:kernelVersion":{},"f:kubeProxyVersion":{},"f:kubeletVersion":{},"f:machineID":{},"f:operatingSystem":{},"f:osImage":{},"f:systemUUID":{}},"f:volumesInUse":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.1.0/24,DoNotUseExternalID:,ProviderID:aws:///eu-central-1a/i-056ad61525499081e,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.1.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49756430336 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-32Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-64Ki: {{0 0} {<nil>} 0 DecimalSI},memory: {{1941540864 0} {<nil>} 1896036Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44780787229 0} {<nil>} 44780787229 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-32Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-64Ki: {{0 0} {<nil>} 0 DecimalSI},memory: {{1836683264 0} {<nil>} 1793636Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-03-05 16:53:22 +0000 UTC,LastTransitionTime:2023-03-05 16:25:57 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-03-05 16:53:22 +0000 UTC,LastTransitionTime:2023-03-05 16:25:48 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-03-05 16:53:22 +0000 UTC,LastTransitionTime:2023-03-05 16:25:48 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-03-05 16:53:22 +0000 UTC,LastTransitionTime:2023-03-05 16:25:48 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.36.179,},NodeAddress{Type:ExternalIP,Address:18.159.37.200,},NodeAddress{Type:InternalDNS,Address:i-056ad61525499081e.eu-central-1.compute.internal,},NodeAddress{Type:Hostname,Address:i-056ad61525499081e.eu-central-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-18-159-37-200.eu-central-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2af3207d1c8c09785c337aae6c6ef0,SystemUUID:ec2af320-7d1c-8c09-785c-337aae6c6ef0,BootID:786f2752-bc30-4729-8aaf-45977b16c253,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 22.04.2 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.27.0-alpha.3.52+d48b8167f7094b,KubeProxyVersion:v1.27.0-alpha.3.52+d48b8167f7094b,OperatingSystem:linux,Architecture:arm64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:157636062,},ContainerImage{Names:[registry.k8s.io/kube-proxy-arm64:v1.27.0-alpha.3.52_d48b8167f7094b],SizeBytes:60554082,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:16bbf38c463a4223d8cfe4da12bc61010b082a79b4bb003e2d3ba3ece5dd5f9e registry.k8s.io/e2e-test-images/agnhost:2.43],SizeBytes:49808364,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:3fe7acf013d1264ffded116b80a73dc129a449b0fccdb8d21af8279f2233f36e registry.k8s.io/e2e-test-images/httpd:2.4.39-4],SizeBytes:41528818,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:148b022f5c5da426fc2f3c14b5c0867e58ef05961510c84749ac1fddcb0fef22 registry.k8s.io/e2e-test-images/httpd:2.4.38-4],SizeBytes:40343601,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:28932904,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:ee3b525d5b89db99da3b8eb521d9cd90cb6e9ef0fbb651e98bb37be78d36b5b8 registry.k8s.io/sig-storage/csi-provisioner:v3.3.0],SizeBytes:23945474,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:425d8f1b769398127767b06ed97ce62578a3179bcb99809ce93a1649e025ffe7 registry.k8s.io/sig-storage/csi-resizer:v1.6.0],SizeBytes:22678632,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:291334908ddf71a4661fd7f6d9d97274de8a5378a2b6fdfeb2ce73414a34f82f registry.k8s.io/sig-storage/csi-snapshotter:v6.1.0],SizeBytes:22434180,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:9a685020911e2725ad019dbce6e4a5ab93d51e3d4557f115e64343345e05781b registry.k8s.io/sig-storage/csi-attacher:v4.0.0],SizeBytes:22394501,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:122bfb8c1edabb3c0edd63f06523e6940d958d19b3957dc7b1d6f81e9f1f6119 registry.k8s.io/sig-storage/csi-provisioner:v3.1.0],SizeBytes:21735644,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:9ebbf9f023e7b41ccee3d52afe39a89e3ddacdbb69269d583abfc25847cfd9e4 registry.k8s.io/sig-storage/csi-resizer:v1.4.0],SizeBytes:20848413,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:8b9c313c05f54fb04f8d430896f5f5904b6cb157df261501b29adc04d2b2dc7b registry.k8s.io/sig-storage/csi-attacher:v3.4.0],SizeBytes:20565480,},ContainerImage{Names:[registry.k8s.io/cpa/cluster-proportional-autoscaler@sha256:68d396900aeaa072c1f27289485fdac29834045a6f3ffe369bf389d830ef572d registry.k8s.io/cpa/cluster-proportional-autoscaler:1.8.6],SizeBytes:19007623,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:92257881c1d6493cf18299a24af42330f891166560047902b8d431fb66b01af5 registry.k8s.io/sig-storage/hostpathplugin:v1.9.0],SizeBytes:18267611,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:9b2d9abda017c28b12f84a344f57cd73fbdb6c2bd7dd5bdd5018246ad1093ba6 registry.k8s.io/sig-storage/hostpathplugin:v1.11.0],SizeBytes:18159085,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:017727efcfeb7d053af68e51436ce8e65edbc6ca573720afb4f79c8594036955 registry.k8s.io/coredns/coredns:v1.10.0],SizeBytes:13766234,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:8502345,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:933940f13b3ea0abc62e656c1aa5c5b47c04b15d71250413a6b821bd0c58b94e registry.k8s.io/sig-storage/livenessprobe:v2.7.0],SizeBytes:8223819,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:7685305,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:5c99cf6a02adda929b10321dbf4ecfa00d87be9ba4fb456006237d530ab4baa1 registry.k8s.io/e2e-test-images/nginx:1.14-4],SizeBytes:6860089,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf registry.k8s.io/e2e-test-images/busybox:1.29-2],SizeBytes:773367,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:2e0f836850e09b8b7cc937681d6194537a09fbd5f6b9e08f4d646a85128e8937 registry.k8s.io/e2e-test-images/busybox:1.29-4],SizeBytes:772933,},ContainerImage{Names:[registry.k8s.io/pause@sha256:7031c1b283388d2c2e09b57badb803c05ebed362dc88d84b480cc47f72a21097 registry.k8s.io/pause:3.9],SizeBytes:268051,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:253553,},},VolumesInUse:[kubernetes.io/csi/csi-mock-csi-mock-volumes-expansion-4933^4 kubernetes.io/csi/ebs.csi.aws.com^vol-0e020446b1cd6abaf],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/csi-mock-csi-mock-volumes-expansion-4933^4,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-0e020446b1cd6abaf,DevicePath:,},},Config:nil,},} Mar 5 16:53:45.673: INFO: Logging kubelet events for node i-056ad61525499081e Mar 5 16:53:45.789: INFO: ... skipping 54 lines ... Logging node info for node i-06df568f6e5c59fff Mar 5 16:53:46.504: INFO: Node Info: &Node{ObjectMeta:{i-06df568f6e5c59fff cb63c681-2e18-474a-b6f3-221325ac0d52 18792 0 2023-03-05 16:24:16 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:arm64 beta.kubernetes.io/instance-type:t4g.small beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:eu-central-1 failure-domain.beta.kubernetes.io/zone:eu-central-1a karpenter.k8s.aws/instance-category:t karpenter.k8s.aws/instance-cpu:2 karpenter.k8s.aws/instance-family:t4g karpenter.k8s.aws/instance-generation:4 karpenter.k8s.aws/instance-hypervisor:nitro karpenter.k8s.aws/instance-memory:2048 karpenter.k8s.aws/instance-pods:11 karpenter.k8s.aws/instance-size:small karpenter.sh/capacity-type:spot karpenter.sh/initialized:true karpenter.sh/provisioner-name:nodes kubernetes.io/arch:arm64 kubernetes.io/hostname:i-06df568f6e5c59fff kubernetes.io/os:linux node-role.kubernetes.io/node: node-role.kubernetes.io/spot-worker:true node.kubernetes.io/instance-type:t4g.small topology.ebs.csi.aws.com/zone:eu-central-1a topology.hostpath.csi/node:i-06df568f6e5c59fff topology.kubernetes.io/region:eu-central-1 topology.kubernetes.io/zone:eu-central-1a] map[csi.volume.kubernetes.io/nodeid:{"csi-hostpath-ephemeral-2147":"i-06df568f6e5c59fff","csi-hostpath-ephemeral-5408":"i-06df568f6e5c59fff","csi-hostpath-ephemeral-6540":"i-06df568f6e5c59fff","csi-mock-csi-mock-volumes-fsgroup-policy-1058":"i-06df568f6e5c59fff","csi-mock-csi-mock-volumes-workload-2219":"i-06df568f6e5c59fff","ebs.csi.aws.com":"i-06df568f6e5c59fff"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [{karpenter.sh/v1alpha5 Provisioner nodes ac5d7599-b2a2-4eae-8da4-1f4c0a54ec82 <nil> 0xc00410b950}] [karpenter.sh/termination] [{aws-cloud-controller-manager Update v1 2023-03-05 16:24:16 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{}}}} } {kops-controller Update v1 2023-03-05 16:24:16 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{},"f:node-role.kubernetes.io/spot-worker":{}}}} } {karpenter Update v1 2023-03-05 16:25:57 +0000 UTC FieldsV1 {"f:metadata":{"f:finalizers":{".":{},"v:\"karpenter.sh/termination\"":{}},"f:labels":{".":{},"f:karpenter.k8s.aws/instance-category":{},"f:karpenter.k8s.aws/instance-cpu":{},"f:karpenter.k8s.aws/instance-family":{},"f:karpenter.k8s.aws/instance-generation":{},"f:karpenter.k8s.aws/instance-hypervisor":{},"f:karpenter.k8s.aws/instance-memory":{},"f:karpenter.k8s.aws/instance-pods":{},"f:karpenter.k8s.aws/instance-size":{},"f:karpenter.sh/capacity-type":{},"f:karpenter.sh/initialized":{},"f:karpenter.sh/provisioner-name":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}},"f:ownerReferences":{".":{},"k:{\"uid\":\"ac5d7599-b2a2-4eae-8da4-1f4c0a54ec82\"}":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-03-05 16:26:48 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{".":{},"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"InternalIP\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kubelet Update v1 2023-03-05 16:48:40 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{"f:kubernetes.io/hostname":{},"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{".":{},"f:cpu":{},"f:ephemeral-storage":{},"f:hugepages-1Gi":{},"f:hugepages-2Mi":{},"f:hugepages-32Mi":{},"f:hugepages-64Ki":{},"f:memory":{},"f:pods":{}},"f:capacity":{".":{},"f:cpu":{},"f:ephemeral-storage":{},"f:hugepages-1Gi":{},"f:hugepages-2Mi":{},"f:hugepages-32Mi":{},"f:hugepages-64Ki":{},"f:memory":{},"f:pods":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{}}},"f:daemonEndpoints":{"f:kubeletEndpoint":{"f:Port":{}}},"f:images":{},"f:nodeInfo":{"f:architecture":{},"f:bootID":{},"f:containerRuntimeVersion":{},"f:kernelVersion":{},"f:kubeProxyVersion":{},"f:kubeletVersion":{},"f:machineID":{},"f:operatingSystem":{},"f:osImage":{},"f:systemUUID":{}},"f:volumesInUse":{}}} status} {kube-controller-manager Update v1 2023-03-05 16:49:37 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{".":{},"k:{\"type\":\"DiskPressure\"}":{".":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"MemoryPressure\"}":{".":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"PIDPressure\"}":{".":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Ready\"}":{".":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}},"f:volumesAttached":{}}} status} {kube-controller-manager Update v1 2023-03-05 16:49:42 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:node.alpha.kubernetes.io/ttl":{}},"f:labels":{"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.2.0/24\"":{}},"f:taints":{}}} }]},Spec:NodeSpec{PodCIDR:100.96.2.0/24,DoNotUseExternalID:,ProviderID:aws:///eu-central-1a/i-06df568f6e5c59fff,Unschedulable:false,Taints:[]Taint{Taint{Key:node.kubernetes.io/unreachable,Value:,Effect:NoSchedule,TimeAdded:2023-03-05 16:49:37 +0000 UTC,},Taint{Key:node.kubernetes.io/unreachable,Value:,Effect:NoExecute,TimeAdded:2023-03-05 16:49:42 +0000 UTC,},},ConfigSource:nil,PodCIDRs:[100.96.2.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49756430336 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-32Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-64Ki: {{0 0} {<nil>} 0 DecimalSI},memory: {{1941540864 0} {<nil>} 1896036Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44780787229 0} {<nil>} 44780787229 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-32Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-64Ki: {{0 0} {<nil>} 0 DecimalSI},memory: {{1836683264 0} {<nil>} 1793636Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:Ready,Status:Unknown,LastHeartbeatTime:2023-03-05 16:47:34 +0000 UTC,LastTransitionTime:2023-03-05 16:49:37 +0000 UTC,Reason:NodeStatusUnknown,Message:Kubelet stopped posting node status.,},NodeCondition{Type:MemoryPressure,Status:Unknown,LastHeartbeatTime:2023-03-05 16:47:34 +0000 UTC,LastTransitionTime:2023-03-05 16:49:37 +0000 UTC,Reason:NodeStatusUnknown,Message:Kubelet stopped posting node status.,},NodeCondition{Type:DiskPressure,Status:Unknown,LastHeartbeatTime:2023-03-05 16:47:34 +0000 UTC,LastTransitionTime:2023-03-05 16:49:37 +0000 UTC,Reason:NodeStatusUnknown,Message:Kubelet stopped posting node status.,},NodeCondition{Type:PIDPressure,Status:Unknown,LastHeartbeatTime:2023-03-05 16:47:34 +0000 UTC,LastTransitionTime:2023-03-05 16:49:37 +0000 UTC,Reason:NodeStatusUnknown,Message:Kubelet stopped posting node status.,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.43.148,},NodeAddress{Type:ExternalIP,Address:18.192.211.222,},NodeAddress{Type:InternalDNS,Address:i-06df568f6e5c59fff.eu-central-1.compute.internal,},NodeAddress{Type:Hostname,Address:i-06df568f6e5c59fff.eu-central-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-18-192-211-222.eu-central-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec221221864d9409a8ec5f6614a9b94b,SystemUUID:ec221221-864d-9409-a8ec-5f6614a9b94b,BootID:553d31bf-faca-417e-a3db-325839e1b8e4,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 22.04.2 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.27.0-alpha.3.52+d48b8167f7094b,KubeProxyVersion:v1.27.0-alpha.3.52+d48b8167f7094b,OperatingSystem:linux,Architecture:arm64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:157636062,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/jessie-dnsutils@sha256:24aaf2626d6b27864c29de2097e8bbb840b3a414271bf7c8995e431e47d8408e registry.k8s.io/e2e-test-images/jessie-dnsutils:1.7],SizeBytes:116764683,},ContainerImage{Names:[registry.k8s.io/kube-proxy-arm64:v1.27.0-alpha.3.52_d48b8167f7094b],SizeBytes:60554082,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:16bbf38c463a4223d8cfe4da12bc61010b082a79b4bb003e2d3ba3ece5dd5f9e registry.k8s.io/e2e-test-images/agnhost:2.43],SizeBytes:49808364,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:3fe7acf013d1264ffded116b80a73dc129a449b0fccdb8d21af8279f2233f36e registry.k8s.io/e2e-test-images/httpd:2.4.39-4],SizeBytes:41528818,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:148b022f5c5da426fc2f3c14b5c0867e58ef05961510c84749ac1fddcb0fef22 registry.k8s.io/e2e-test-images/httpd:2.4.38-4],SizeBytes:40343601,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:28932904,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:ee3b525d5b89db99da3b8eb521d9cd90cb6e9ef0fbb651e98bb37be78d36b5b8 registry.k8s.io/sig-storage/csi-provisioner:v3.3.0],SizeBytes:23945474,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:425d8f1b769398127767b06ed97ce62578a3179bcb99809ce93a1649e025ffe7 registry.k8s.io/sig-storage/csi-resizer:v1.6.0],SizeBytes:22678632,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:291334908ddf71a4661fd7f6d9d97274de8a5378a2b6fdfeb2ce73414a34f82f registry.k8s.io/sig-storage/csi-snapshotter:v6.1.0],SizeBytes:22434180,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:9a685020911e2725ad019dbce6e4a5ab93d51e3d4557f115e64343345e05781b registry.k8s.io/sig-storage/csi-attacher:v4.0.0],SizeBytes:22394501,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:122bfb8c1edabb3c0edd63f06523e6940d958d19b3957dc7b1d6f81e9f1f6119 registry.k8s.io/sig-storage/csi-provisioner:v3.1.0],SizeBytes:21735644,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:9ebbf9f023e7b41ccee3d52afe39a89e3ddacdbb69269d583abfc25847cfd9e4 registry.k8s.io/sig-storage/csi-resizer:v1.4.0],SizeBytes:20848413,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:8b9c313c05f54fb04f8d430896f5f5904b6cb157df261501b29adc04d2b2dc7b registry.k8s.io/sig-storage/csi-attacher:v3.4.0],SizeBytes:20565480,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nonroot@sha256:ee9f50b3c64b174d296d91ca9f69a914ac30e59095dfb462b2b518ad28a63655 registry.k8s.io/e2e-test-images/nonroot:1.4],SizeBytes:18561972,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:92257881c1d6493cf18299a24af42330f891166560047902b8d431fb66b01af5 registry.k8s.io/sig-storage/hostpathplugin:v1.9.0],SizeBytes:18267611,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:9b2d9abda017c28b12f84a344f57cd73fbdb6c2bd7dd5bdd5018246ad1093ba6 registry.k8s.io/sig-storage/hostpathplugin:v1.11.0],SizeBytes:18159085,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:017727efcfeb7d053af68e51436ce8e65edbc6ca573720afb4f79c8594036955 registry.k8s.io/coredns/coredns:v1.10.0],SizeBytes:13766234,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:8502345,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:933940f13b3ea0abc62e656c1aa5c5b47c04b15d71250413a6b821bd0c58b94e registry.k8s.io/sig-storage/livenessprobe:v2.7.0],SizeBytes:8223819,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:7685305,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:5c99cf6a02adda929b10321dbf4ecfa00d87be9ba4fb456006237d530ab4baa1 registry.k8s.io/e2e-test-images/nginx:1.14-4],SizeBytes:6860089,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nonewprivs@sha256:8ac1264691820febacf3aea5d152cbde6d10685731ec14966a9401c6f47a68ac registry.k8s.io/e2e-test-images/nonewprivs:1.3],SizeBytes:5025027,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:2e0f836850e09b8b7cc937681d6194537a09fbd5f6b9e08f4d646a85128e8937 registry.k8s.io/e2e-test-images/busybox:1.29-4],SizeBytes:772933,},ContainerImage{Names:[registry.k8s.io/pause@sha256:7031c1b283388d2c2e09b57badb803c05ebed362dc88d84b480cc47f72a21097 registry.k8s.io/pause:3.9],SizeBytes:268051,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:253553,},},VolumesInUse:[kubernetes.io/csi/csi-hostpath-ephemeral-2147^52df5953-bb73-11ed-8aa7-be49170871b4 kubernetes.io/csi/csi-hostpath-ephemeral-5408^443d1daf-bb73-11ed-95c4-5a95d17cbc27 kubernetes.io/csi/csi-hostpath-ephemeral-5408^44411f51-bb73-11ed-95c4-5a95d17cbc27 kubernetes.io/csi/csi-hostpath-ephemeral-6540^4426a07f-bb73-11ed-a604-92687f2c50f8 kubernetes.io/csi/ebs.csi.aws.com^vol-05b67fab5beb6394b],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-05b67fab5beb6394b,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/csi-hostpath-ephemeral-6540^4426a07f-bb73-11ed-a604-92687f2c50f8,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/csi-hostpath-ephemeral-5408^44411f51-bb73-11ed-95c4-5a95d17cbc27,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/csi-hostpath-ephemeral-5408^443d1daf-bb73-11ed-95c4-5a95d17cbc27,DevicePath:,},},Config:nil,},} Mar 5 16:53:46.504: INFO: Logging kubelet events for node i-06df568f6e5c59fff Mar 5 16:53:46.620: INFO: Logging pods the kubelet thinks is on node i-06df568f6e5c59fff Mar 5 16:53:56.733: INFO: Unable to retrieve kubelet pods for node i-06df568f6e5c59fff: error trying to reach service: net/http: TLS handshake timeout Mar 5 16:53:56.733: INFO: Logging node info for node i-0d6a32f749839192f Mar 5 16:53:56.846: INFO: Node Info: &Node{ObjectMeta:{i-0d6a32f749839192f 4155fefb-4d3a-4b04-ba53-6ac9e895ff92 18509 0 2023-03-05 16:20:46 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:arm64 beta.kubernetes.io/instance-type:c6g.xlarge beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:eu-central-1 failure-domain.beta.kubernetes.io/zone:eu-central-1a kops.k8s.io/kops-controller-pki: kubernetes.io/arch:arm64 kubernetes.io/hostname:i-0d6a32f749839192f kubernetes.io/os:linux node-role.kubernetes.io/control-plane: node.kubernetes.io/exclude-from-external-load-balancers: node.kubernetes.io/instance-type:c6g.xlarge topology.ebs.csi.aws.com/zone:eu-central-1a topology.kubernetes.io/region:eu-central-1 topology.kubernetes.io/zone:eu-central-1a] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-0d6a32f749839192f"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kubelet Update v1 2023-03-05 16:20:46 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {protokube Update v1 2023-03-05 16:21:25 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kops.k8s.io/kops-controller-pki":{},"f:node-role.kubernetes.io/control-plane":{},"f:node.kubernetes.io/exclude-from-external-load-balancers":{}}}} } {kube-controller-manager Update v1 2023-03-05 16:21:47 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.0.0/24\"":{}}}} } {aws-cloud-controller-manager Update v1 2023-03-05 16:21:48 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{},"f:taints":{}}} } {aws-cloud-controller-manager Update v1 2023-03-05 16:21:48 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kubelet Update v1 2023-03-05 16:49:18 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.0.0/24,DoNotUseExternalID:,ProviderID:aws:///eu-central-1a/i-0d6a32f749839192f,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/control-plane,Value:,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[100.96.0.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{4 0} {<nil>} 4 DecimalSI},ephemeral-storage: {{49756430336 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-32Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-64Ki: {{0 0} {<nil>} 0 DecimalSI},memory: {{8150118400 0} {<nil>} 7959100Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{4 0} {<nil>} 4 DecimalSI},ephemeral-storage: {{44780787229 0} {<nil>} 44780787229 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-32Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-64Ki: {{0 0} {<nil>} 0 DecimalSI},memory: {{8045260800 0} {<nil>} 7856700Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-03-05 16:49:18 +0000 UTC,LastTransitionTime:2023-03-05 16:20:46 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-03-05 16:49:18 +0000 UTC,LastTransitionTime:2023-03-05 16:20:46 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-03-05 16:49:18 +0000 UTC,LastTransitionTime:2023-03-05 16:20:46 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-03-05 16:49:18 +0000 UTC,LastTransitionTime:2023-03-05 16:21:47 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.49.247,},NodeAddress{Type:ExternalIP,Address:18.197.20.82,},NodeAddress{Type:InternalDNS,Address:i-0d6a32f749839192f.eu-central-1.compute.internal,},NodeAddress{Type:Hostname,Address:i-0d6a32f749839192f.eu-central-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-18-197-20-82.eu-central-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2bed249e8ac7b517cb08ccd42b83e5,SystemUUID:ec2bed24-9e8a-c7b5-17cb-08ccd42b83e5,BootID:6f9215c9-dfe8-4b13-84f4-9b2084456198,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 22.04.2 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.27.0-alpha.3.52+d48b8167f7094b,KubeProxyVersion:v1.27.0-alpha.3.52+d48b8167f7094b,OperatingSystem:linux,Architecture:arm64,},Images:[]ContainerImage{ContainerImage{Names:[registry.k8s.io/etcd@sha256:9ce33ba33d8e738a5b85ed50b5080ac746deceed4a7496c550927a7a19ca3b6d registry.k8s.io/etcd:3.5.0-0],SizeBytes:157797353,},ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:157636062,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:4afb99b4690b418ffc2ceb67e1a17376457e441c1f09ab55447f0aaf992fa646 registry.k8s.io/etcd:3.4.3-0],SizeBytes:151677080,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:4ad90a11b55313b182afc186b9876c8e891531b8db4c9bf1541953021618d0e2 registry.k8s.io/etcd:3.4.13-0],SizeBytes:134531559,},ContainerImage{Names:[registry.k8s.io/kube-apiserver-arm64:v1.27.0-alpha.3.52_d48b8167f7094b],SizeBytes:131087707,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:0d2ab6b15b4fe0b903b9e81192d942fb39c29be61fe341cd16db5d0165b0435c registry.k8s.io/etcd:3.3.17-0],SizeBytes:126979107,},ContainerImage{Names:[registry.k8s.io/kube-controller-manager-arm64:v1.27.0-alpha.3.52_d48b8167f7094b],SizeBytes:120667128,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:17da501f5d2a675be46040422a27b7cc21b8a43895ac998b171db1c346f361f7 registry.k8s.io/etcd:3.3.10-0],SizeBytes:81436097,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:6f72b851544986cb0921b53ea655ec04c36131248f16d4ad110cb3ca0c369dc1 registry.k8s.io/etcd:3.5.4-0],SizeBytes:81120118,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:13f53ed1d91e2e11aac476ee9a0269fdda6cc4874eba903efd40daf50c55eee5 registry.k8s.io/etcd:3.5.3-0],SizeBytes:81114891,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:51eae8381dcb1078289fa7b4f3df2630cdc18d09fb56f8e56b41c40e191d6c83 registry.k8s.io/etcd:3.5.7-0],SizeBytes:80665728,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:dd75ec974b0a2a6f6bb47001ba09207976e625db898d1b16735528c009cb171c registry.k8s.io/etcd:3.5.6-0],SizeBytes:80539316,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:905d7ca17fd02bc24c0eba9a062753aba15db3e31422390bc3238eb762339b20 registry.k8s.io/etcd:3.2.24-1],SizeBytes:61500433,},ContainerImage{Names:[registry.k8s.io/etcdadm/etcd-manager-slim@sha256:8727adc059334e0e3329dbccce116a734da4a090da5e86c8cddccaf333caa666 registry.k8s.io/etcdadm/etcd-manager-slim:v3.0.20230201],SizeBytes:60826655,},ContainerImage{Names:[registry.k8s.io/kube-proxy-arm64:v1.27.0-alpha.3.52_d48b8167f7094b],SizeBytes:60554082,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:64b9ea357325d5db9f8a723dcf503b5a449177b17ac87d69481e126bb724c263 registry.k8s.io/etcd:3.5.1-0],SizeBytes:59553904,},ContainerImage{Names:[registry.k8s.io/kube-scheduler-arm64:v1.27.0-alpha.3.52_d48b8167f7094b],SizeBytes:56310763,},ContainerImage{Names:[registry.k8s.io/kops/kops-controller:1.27.0-alpha.1],SizeBytes:41797411,},ContainerImage{Names:[registry.k8s.io/kops/dns-controller:1.27.0-alpha.1],SizeBytes:40267473,},ContainerImage{Names:[public.ecr.aws/karpenter/controller@sha256:68db4f092cf9cc83f5ef9e2fbc5407c2cb682e81f64dfaa700a7602ede38b1cf public.ecr.aws/karpenter/controller:v0.16.3],SizeBytes:33707751,},ContainerImage{Names:[public.ecr.aws/karpenter/webhook@sha256:96a2d9b06d6bc5127801f358f74b1cf2d289b423a2e9ba40c573c0b14b17dafa public.ecr.aws/karpenter/webhook:v0.16.3],SizeBytes:33307187,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:28932904,},ContainerImage{Names:[quay.io/cilium/operator@sha256:a6d24a006a6b92967ac90786b49bc1ac26e5477cf028cd1186efcfc2466484db quay.io/cilium/operator:v1.12.5],SizeBytes:24567287,},ContainerImage{Names:[registry.k8s.io/provider-aws/cloud-controller-manager@sha256:fdeb61e3e42ecd9cca868d550ebdb88dd6341d9e91fcfa9a37e227dab2ad22cb registry.k8s.io/provider-aws/cloud-controller-manager:v1.26.0],SizeBytes:18345226,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:8502345,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:7685305,},ContainerImage{Names:[registry.k8s.io/kops/kube-apiserver-healthcheck:1.27.0-alpha.1],SizeBytes:4713612,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:253553,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Mar 5 16:53:56.846: INFO: Logging kubelet events for node i-0d6a32f749839192f Mar 5 16:53:56.962: INFO: ... skipping 52 lines ... Mar 5 16:53:57.087: INFO: Container webhook ready: true, restart count 0 Mar 5 16:53:57.588: INFO: Latency metrics for node i-0d6a32f749839192f [1mSTEP:[0m Destroying namespace "emptydir-5780" for this suite. [38;5;243m@ 03/05/23 16:53:57.588[0m [38;5;243m<< Timeline[0m [38;5;9m[FAILED] wait for pod "pod-e43f91ea-b3ff-4cde-9686-99368f334635" to disappear: expected pod to not be found: Timed out after 180.001s. Expected <*v1.Pod | 0xc000d88000>: metadata: creationTimestamp: "2023-03-05T16:42:27Z" deletionGracePeriodSeconds: 30 deletionTimestamp: "2023-03-05T16:48:04Z" ... skipping 137 lines ... qosClass: BestEffort to be nil[0m [38;5;9mIn [1m[It][0m[38;5;9m at: [1mtest/e2e/framework/pod/pod_client.go:184[0m [38;5;243m@ 03/05/23 16:50:34.766[0m [38;5;243m------------------------------[0m [38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;10m•[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;10m•[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m [38;5;243m------------------------------[0m [38;5;9m• [FAILED] [809.543 seconds][0m [0m[sig-apps] StatefulSet [38;5;9m[1m[It] AvailableReplicas should get updated accordingly when MinReadySeconds is enabled[0m [38;5;243mtest/e2e/apps/statefulset.go:1173[0m [38;5;243mTimeline >>[0m [1mSTEP:[0m Creating a kubernetes client [38;5;243m@ 03/05/23 16:42:00.336[0m Mar 5 16:42:00.336: INFO: >>> kubeConfig: /root/.kube/config ... skipping 61 lines ... Mar 5 16:51:26.693: INFO: Waiting for stateful set status.AvailableReplicas to become 2, currently 0 Mar 5 16:51:36.692: INFO: Waiting for stateful set status.AvailableReplicas to become 2, currently 0 Mar 5 16:51:46.693: INFO: Waiting for stateful set status.AvailableReplicas to become 2, currently 0 Mar 5 16:51:56.693: INFO: Waiting for stateful set status.AvailableReplicas to become 2, currently 0 Mar 5 16:52:06.693: INFO: Waiting for stateful set status.AvailableReplicas to become 2, currently 0 Mar 5 16:52:06.806: INFO: Waiting for stateful set status.AvailableReplicas to become 2, currently 0 [38;5;9m[FAILED][0m in [It] - test/e2e/framework/statefulset/wait.go:145 [38;5;243m@ 03/05/23 16:52:06.806[0m Mar 5 16:52:06.806: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready Mar 5 16:52:06.921: INFO: Condition Ready of node i-06df568f6e5c59fff is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-05 16:49:37 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-05 16:49:42 +0000 UTC}]. Failure Mar 5 16:52:09.036: INFO: Condition Ready of node i-06df568f6e5c59fff is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-05 16:49:37 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-05 16:49:42 +0000 UTC}]. Failure Mar 5 16:52:11.035: INFO: Condition Ready of node i-06df568f6e5c59fff is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-05 16:49:37 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-05 16:49:42 +0000 UTC}]. Failure Mar 5 16:52:13.037: INFO: Condition Ready of node i-06df568f6e5c59fff is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-05 16:49:37 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-05 16:49:42 +0000 UTC}]. Failure Mar 5 16:52:15.035: INFO: Condition Ready of node i-06df568f6e5c59fff is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-05 16:49:37 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-05 16:49:42 +0000 UTC}]. Failure ... skipping 90 lines ... Mar 5 16:55:07.262: INFO: At 2023-03-05 16:42:01 +0000 UTC - event for test-ss: {statefulset-controller } SuccessfulCreate: create Pod test-ss-0 in StatefulSet test-ss successful Mar 5 16:55:07.262: INFO: At 2023-03-05 16:42:01 +0000 UTC - event for test-ss-0: {default-scheduler } Scheduled: Successfully assigned statefulset-2394/test-ss-0 to i-06df568f6e5c59fff Mar 5 16:55:07.262: INFO: At 2023-03-05 16:47:57 +0000 UTC - event for test-ss-0: {taint-controller } TaintManagerEviction: Marking for deletion Pod statefulset-2394/test-ss-0 Mar 5 16:55:07.374: INFO: POD NODE PHASE GRACE CONDITIONS Mar 5 16:55:07.374: INFO: test-ss-0 i-06df568f6e5c59fff Pending 30s [{PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2023-03-05 16:42:01 +0000 UTC } {DisruptionTarget True 0001-01-01 00:00:00 +0000 UTC 2023-03-05 16:47:57 +0000 UTC DeletionByTaintManager Taint manager: deleting due to NoExecute taint}] Mar 5 16:55:07.374: INFO: Mar 5 16:55:17.489: INFO: Unable to fetch statefulset-2394/test-ss-0/webserver logs: an error on the server ("unknown") has prevented the request from succeeding (get pods test-ss-0) Mar 5 16:55:17.602: INFO: Logging node info for node i-056ad61525499081e Mar 5 16:55:17.715: INFO: Node Info: &Node{ObjectMeta:{i-056ad61525499081e a6ce9ed3-5ae8-4065-b117-6884e24b3632 21032 0 2023-03-05 16:24:16 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:arm64 beta.kubernetes.io/instance-type:t4g.small beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:eu-central-1 failure-domain.beta.kubernetes.io/zone:eu-central-1a karpenter.k8s.aws/instance-category:t karpenter.k8s.aws/instance-cpu:2 karpenter.k8s.aws/instance-family:t4g karpenter.k8s.aws/instance-generation:4 karpenter.k8s.aws/instance-hypervisor:nitro karpenter.k8s.aws/instance-memory:2048 karpenter.k8s.aws/instance-pods:11 karpenter.k8s.aws/instance-size:small karpenter.sh/capacity-type:spot karpenter.sh/initialized:true karpenter.sh/provisioner-name:nodes kubernetes.io/arch:arm64 kubernetes.io/hostname:i-056ad61525499081e kubernetes.io/os:linux node-role.kubernetes.io/node: node-role.kubernetes.io/spot-worker:true node.kubernetes.io/instance-type:t4g.small topology.ebs.csi.aws.com/zone:eu-central-1a topology.hostpath.csi/node:i-056ad61525499081e topology.kubernetes.io/region:eu-central-1 topology.kubernetes.io/zone:eu-central-1a] map[csi.volume.kubernetes.io/nodeid:{"csi-hostpath-ephemeral-7990":"i-056ad61525499081e","csi-hostpath-provisioning-8616":"i-056ad61525499081e","csi-mock-csi-mock-volumes-expansion-4933":"csi-mock-csi-mock-volumes-expansion-4933","ebs.csi.aws.com":"i-056ad61525499081e"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [{karpenter.sh/v1alpha5 Provisioner nodes ac5d7599-b2a2-4eae-8da4-1f4c0a54ec82 <nil> 0xc003957cf0}] [karpenter.sh/termination] [{aws-cloud-controller-manager Update v1 2023-03-05 16:24:16 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{}}}} } {kops-controller Update v1 2023-03-05 16:24:16 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{},"f:node-role.kubernetes.io/spot-worker":{}}}} } {karpenter Update v1 2023-03-05 16:25:57 +0000 UTC FieldsV1 {"f:metadata":{"f:finalizers":{".":{},"v:\"karpenter.sh/termination\"":{}},"f:labels":{".":{},"f:karpenter.k8s.aws/instance-category":{},"f:karpenter.k8s.aws/instance-cpu":{},"f:karpenter.k8s.aws/instance-family":{},"f:karpenter.k8s.aws/instance-generation":{},"f:karpenter.k8s.aws/instance-hypervisor":{},"f:karpenter.k8s.aws/instance-memory":{},"f:karpenter.k8s.aws/instance-pods":{},"f:karpenter.k8s.aws/instance-size":{},"f:karpenter.sh/capacity-type":{},"f:karpenter.sh/initialized":{},"f:karpenter.sh/provisioner-name":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}},"f:ownerReferences":{".":{},"k:{\"uid\":\"ac5d7599-b2a2-4eae-8da4-1f4c0a54ec82\"}":{}}},"f:spec":{"f:providerID":{}}} } {kube-controller-manager Update v1 2023-03-05 16:25:57 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:node.alpha.kubernetes.io/ttl":{}},"f:labels":{"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.1.0/24\"":{}}}} } {aws-cloud-controller-manager Update v1 2023-03-05 16:26:48 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{".":{},"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"InternalIP\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kube-controller-manager Update v1 2023-03-05 16:54:16 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{".":{},"k:{\"type\":\"DiskPressure\"}":{".":{},"f:type":{}},"k:{\"type\":\"MemoryPressure\"}":{".":{},"f:type":{}},"k:{\"type\":\"PIDPressure\"}":{".":{},"f:type":{}},"k:{\"type\":\"Ready\"}":{".":{},"f:type":{}}},"f:volumesAttached":{}}} status} {kubelet Update v1 2023-03-05 16:54:23 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{"f:kubernetes.io/hostname":{},"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{".":{},"f:cpu":{},"f:ephemeral-storage":{},"f:hugepages-1Gi":{},"f:hugepages-2Mi":{},"f:hugepages-32Mi":{},"f:hugepages-64Ki":{},"f:memory":{},"f:pods":{}},"f:capacity":{".":{},"f:cpu":{},"f:ephemeral-storage":{},"f:hugepages-1Gi":{},"f:hugepages-2Mi":{},"f:hugepages-32Mi":{},"f:hugepages-64Ki":{},"f:memory":{},"f:pods":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:daemonEndpoints":{"f:kubeletEndpoint":{"f:Port":{}}},"f:images":{},"f:nodeInfo":{"f:architecture":{},"f:bootID":{},"f:containerRuntimeVersion":{},"f:kernelVersion":{},"f:kubeProxyVersion":{},"f:kubeletVersion":{},"f:machineID":{},"f:operatingSystem":{},"f:osImage":{},"f:systemUUID":{}},"f:volumesInUse":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.1.0/24,DoNotUseExternalID:,ProviderID:aws:///eu-central-1a/i-056ad61525499081e,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.1.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49756430336 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-32Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-64Ki: {{0 0} {<nil>} 0 DecimalSI},memory: {{1941540864 0} {<nil>} 1896036Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44780787229 0} {<nil>} 44780787229 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-32Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-64Ki: {{0 0} {<nil>} 0 DecimalSI},memory: {{1836683264 0} {<nil>} 1793636Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-03-05 16:54:23 +0000 UTC,LastTransitionTime:2023-03-05 16:25:57 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-03-05 16:54:23 +0000 UTC,LastTransitionTime:2023-03-05 16:25:48 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-03-05 16:54:23 +0000 UTC,LastTransitionTime:2023-03-05 16:25:48 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-03-05 16:54:23 +0000 UTC,LastTransitionTime:2023-03-05 16:25:48 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.36.179,},NodeAddress{Type:ExternalIP,Address:18.159.37.200,},NodeAddress{Type:InternalDNS,Address:i-056ad61525499081e.eu-central-1.compute.internal,},NodeAddress{Type:Hostname,Address:i-056ad61525499081e.eu-central-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-18-159-37-200.eu-central-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2af3207d1c8c09785c337aae6c6ef0,SystemUUID:ec2af320-7d1c-8c09-785c-337aae6c6ef0,BootID:786f2752-bc30-4729-8aaf-45977b16c253,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 22.04.2 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.27.0-alpha.3.52+d48b8167f7094b,KubeProxyVersion:v1.27.0-alpha.3.52+d48b8167f7094b,OperatingSystem:linux,Architecture:arm64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:157636062,},ContainerImage{Names:[registry.k8s.io/kube-proxy-arm64:v1.27.0-alpha.3.52_d48b8167f7094b],SizeBytes:60554082,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:16bbf38c463a4223d8cfe4da12bc61010b082a79b4bb003e2d3ba3ece5dd5f9e registry.k8s.io/e2e-test-images/agnhost:2.43],SizeBytes:49808364,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:3fe7acf013d1264ffded116b80a73dc129a449b0fccdb8d21af8279f2233f36e registry.k8s.io/e2e-test-images/httpd:2.4.39-4],SizeBytes:41528818,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:148b022f5c5da426fc2f3c14b5c0867e58ef05961510c84749ac1fddcb0fef22 registry.k8s.io/e2e-test-images/httpd:2.4.38-4],SizeBytes:40343601,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:28932904,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:ee3b525d5b89db99da3b8eb521d9cd90cb6e9ef0fbb651e98bb37be78d36b5b8 registry.k8s.io/sig-storage/csi-provisioner:v3.3.0],SizeBytes:23945474,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:425d8f1b769398127767b06ed97ce62578a3179bcb99809ce93a1649e025ffe7 registry.k8s.io/sig-storage/csi-resizer:v1.6.0],SizeBytes:22678632,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:291334908ddf71a4661fd7f6d9d97274de8a5378a2b6fdfeb2ce73414a34f82f registry.k8s.io/sig-storage/csi-snapshotter:v6.1.0],SizeBytes:22434180,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:9a685020911e2725ad019dbce6e4a5ab93d51e3d4557f115e64343345e05781b registry.k8s.io/sig-storage/csi-attacher:v4.0.0],SizeBytes:22394501,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:122bfb8c1edabb3c0edd63f06523e6940d958d19b3957dc7b1d6f81e9f1f6119 registry.k8s.io/sig-storage/csi-provisioner:v3.1.0],SizeBytes:21735644,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:9ebbf9f023e7b41ccee3d52afe39a89e3ddacdbb69269d583abfc25847cfd9e4 registry.k8s.io/sig-storage/csi-resizer:v1.4.0],SizeBytes:20848413,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:8b9c313c05f54fb04f8d430896f5f5904b6cb157df261501b29adc04d2b2dc7b registry.k8s.io/sig-storage/csi-attacher:v3.4.0],SizeBytes:20565480,},ContainerImage{Names:[registry.k8s.io/cpa/cluster-proportional-autoscaler@sha256:68d396900aeaa072c1f27289485fdac29834045a6f3ffe369bf389d830ef572d registry.k8s.io/cpa/cluster-proportional-autoscaler:1.8.6],SizeBytes:19007623,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:92257881c1d6493cf18299a24af42330f891166560047902b8d431fb66b01af5 registry.k8s.io/sig-storage/hostpathplugin:v1.9.0],SizeBytes:18267611,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:9b2d9abda017c28b12f84a344f57cd73fbdb6c2bd7dd5bdd5018246ad1093ba6 registry.k8s.io/sig-storage/hostpathplugin:v1.11.0],SizeBytes:18159085,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:017727efcfeb7d053af68e51436ce8e65edbc6ca573720afb4f79c8594036955 registry.k8s.io/coredns/coredns:v1.10.0],SizeBytes:13766234,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:8502345,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:933940f13b3ea0abc62e656c1aa5c5b47c04b15d71250413a6b821bd0c58b94e registry.k8s.io/sig-storage/livenessprobe:v2.7.0],SizeBytes:8223819,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:7685305,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:5c99cf6a02adda929b10321dbf4ecfa00d87be9ba4fb456006237d530ab4baa1 registry.k8s.io/e2e-test-images/nginx:1.14-4],SizeBytes:6860089,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf registry.k8s.io/e2e-test-images/busybox:1.29-2],SizeBytes:773367,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:2e0f836850e09b8b7cc937681d6194537a09fbd5f6b9e08f4d646a85128e8937 registry.k8s.io/e2e-test-images/busybox:1.29-4],SizeBytes:772933,},ContainerImage{Names:[registry.k8s.io/pause@sha256:7031c1b283388d2c2e09b57badb803c05ebed362dc88d84b480cc47f72a21097 registry.k8s.io/pause:3.9],SizeBytes:268051,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:253553,},},VolumesInUse:[kubernetes.io/csi/csi-hostpath-provisioning-8616^5bd42346-bb76-11ed-b199-0ee9a68a9789 kubernetes.io/csi/csi-mock-csi-mock-volumes-expansion-4933^4 kubernetes.io/csi/ebs.csi.aws.com^vol-0e020446b1cd6abaf],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/csi-hostpath-provisioning-8616^5bd42346-bb76-11ed-b199-0ee9a68a9789,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-0e020446b1cd6abaf,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/csi-mock-csi-mock-volumes-expansion-4933^4,DevicePath:,},},Config:nil,},} Mar 5 16:55:17.715: INFO: Logging kubelet events for node i-056ad61525499081e Mar 5 16:55:17.832: INFO: ... skipping 57 lines ... Logging node info for node i-06df568f6e5c59fff Mar 5 16:55:18.676: INFO: Node Info: &Node{ObjectMeta:{i-06df568f6e5c59fff cb63c681-2e18-474a-b6f3-221325ac0d52 18792 0 2023-03-05 16:24:16 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:arm64 beta.kubernetes.io/instance-type:t4g.small beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:eu-central-1 failure-domain.beta.kubernetes.io/zone:eu-central-1a karpenter.k8s.aws/instance-category:t karpenter.k8s.aws/instance-cpu:2 karpenter.k8s.aws/instance-family:t4g karpenter.k8s.aws/instance-generation:4 karpenter.k8s.aws/instance-hypervisor:nitro karpenter.k8s.aws/instance-memory:2048 karpenter.k8s.aws/instance-pods:11 karpenter.k8s.aws/instance-size:small karpenter.sh/capacity-type:spot karpenter.sh/initialized:true karpenter.sh/provisioner-name:nodes kubernetes.io/arch:arm64 kubernetes.io/hostname:i-06df568f6e5c59fff kubernetes.io/os:linux node-role.kubernetes.io/node: node-role.kubernetes.io/spot-worker:true node.kubernetes.io/instance-type:t4g.small topology.ebs.csi.aws.com/zone:eu-central-1a topology.hostpath.csi/node:i-06df568f6e5c59fff topology.kubernetes.io/region:eu-central-1 topology.kubernetes.io/zone:eu-central-1a] map[csi.volume.kubernetes.io/nodeid:{"csi-hostpath-ephemeral-2147":"i-06df568f6e5c59fff","csi-hostpath-ephemeral-5408":"i-06df568f6e5c59fff","csi-hostpath-ephemeral-6540":"i-06df568f6e5c59fff","csi-mock-csi-mock-volumes-fsgroup-policy-1058":"i-06df568f6e5c59fff","csi-mock-csi-mock-volumes-workload-2219":"i-06df568f6e5c59fff","ebs.csi.aws.com":"i-06df568f6e5c59fff"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [{karpenter.sh/v1alpha5 Provisioner nodes ac5d7599-b2a2-4eae-8da4-1f4c0a54ec82 <nil> 0xc001a326d0}] [karpenter.sh/termination] [{aws-cloud-controller-manager Update v1 2023-03-05 16:24:16 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{}}}} } {kops-controller Update v1 2023-03-05 16:24:16 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{},"f:node-role.kubernetes.io/spot-worker":{}}}} } {karpenter Update v1 2023-03-05 16:25:57 +0000 UTC FieldsV1 {"f:metadata":{"f:finalizers":{".":{},"v:\"karpenter.sh/termination\"":{}},"f:labels":{".":{},"f:karpenter.k8s.aws/instance-category":{},"f:karpenter.k8s.aws/instance-cpu":{},"f:karpenter.k8s.aws/instance-family":{},"f:karpenter.k8s.aws/instance-generation":{},"f:karpenter.k8s.aws/instance-hypervisor":{},"f:karpenter.k8s.aws/instance-memory":{},"f:karpenter.k8s.aws/instance-pods":{},"f:karpenter.k8s.aws/instance-size":{},"f:karpenter.sh/capacity-type":{},"f:karpenter.sh/initialized":{},"f:karpenter.sh/provisioner-name":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}},"f:ownerReferences":{".":{},"k:{\"uid\":\"ac5d7599-b2a2-4eae-8da4-1f4c0a54ec82\"}":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-03-05 16:26:48 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{".":{},"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"InternalIP\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kubelet Update v1 2023-03-05 16:48:40 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{"f:kubernetes.io/hostname":{},"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{".":{},"f:cpu":{},"f:ephemeral-storage":{},"f:hugepages-1Gi":{},"f:hugepages-2Mi":{},"f:hugepages-32Mi":{},"f:hugepages-64Ki":{},"f:memory":{},"f:pods":{}},"f:capacity":{".":{},"f:cpu":{},"f:ephemeral-storage":{},"f:hugepages-1Gi":{},"f:hugepages-2Mi":{},"f:hugepages-32Mi":{},"f:hugepages-64Ki":{},"f:memory":{},"f:pods":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{}}},"f:daemonEndpoints":{"f:kubeletEndpoint":{"f:Port":{}}},"f:images":{},"f:nodeInfo":{"f:architecture":{},"f:bootID":{},"f:containerRuntimeVersion":{},"f:kernelVersion":{},"f:kubeProxyVersion":{},"f:kubeletVersion":{},"f:machineID":{},"f:operatingSystem":{},"f:osImage":{},"f:systemUUID":{}},"f:volumesInUse":{}}} status} {kube-controller-manager Update v1 2023-03-05 16:49:37 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{".":{},"k:{\"type\":\"DiskPressure\"}":{".":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"MemoryPressure\"}":{".":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"PIDPressure\"}":{".":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Ready\"}":{".":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}},"f:volumesAttached":{}}} status} {kube-controller-manager Update v1 2023-03-05 16:49:42 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:node.alpha.kubernetes.io/ttl":{}},"f:labels":{"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.2.0/24\"":{}},"f:taints":{}}} }]},Spec:NodeSpec{PodCIDR:100.96.2.0/24,DoNotUseExternalID:,ProviderID:aws:///eu-central-1a/i-06df568f6e5c59fff,Unschedulable:false,Taints:[]Taint{Taint{Key:node.kubernetes.io/unreachable,Value:,Effect:NoSchedule,TimeAdded:2023-03-05 16:49:37 +0000 UTC,},Taint{Key:node.kubernetes.io/unreachable,Value:,Effect:NoExecute,TimeAdded:2023-03-05 16:49:42 +0000 UTC,},},ConfigSource:nil,PodCIDRs:[100.96.2.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49756430336 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-32Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-64Ki: {{0 0} {<nil>} 0 DecimalSI},memory: {{1941540864 0} {<nil>} 1896036Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44780787229 0} {<nil>} 44780787229 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-32Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-64Ki: {{0 0} {<nil>} 0 DecimalSI},memory: {{1836683264 0} {<nil>} 1793636Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:Ready,Status:Unknown,LastHeartbeatTime:2023-03-05 16:47:34 +0000 UTC,LastTransitionTime:2023-03-05 16:49:37 +0000 UTC,Reason:NodeStatusUnknown,Message:Kubelet stopped posting node status.,},NodeCondition{Type:MemoryPressure,Status:Unknown,LastHeartbeatTime:2023-03-05 16:47:34 +0000 UTC,LastTransitionTime:2023-03-05 16:49:37 +0000 UTC,Reason:NodeStatusUnknown,Message:Kubelet stopped posting node status.,},NodeCondition{Type:DiskPressure,Status:Unknown,LastHeartbeatTime:2023-03-05 16:47:34 +0000 UTC,LastTransitionTime:2023-03-05 16:49:37 +0000 UTC,Reason:NodeStatusUnknown,Message:Kubelet stopped posting node status.,},NodeCondition{Type:PIDPressure,Status:Unknown,LastHeartbeatTime:2023-03-05 16:47:34 +0000 UTC,LastTransitionTime:2023-03-05 16:49:37 +0000 UTC,Reason:NodeStatusUnknown,Message:Kubelet stopped posting node status.,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.43.148,},NodeAddress{Type:ExternalIP,Address:18.192.211.222,},NodeAddress{Type:InternalDNS,Address:i-06df568f6e5c59fff.eu-central-1.compute.internal,},NodeAddress{Type:Hostname,Address:i-06df568f6e5c59fff.eu-central-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-18-192-211-222.eu-central-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec221221864d9409a8ec5f6614a9b94b,SystemUUID:ec221221-864d-9409-a8ec-5f6614a9b94b,BootID:553d31bf-faca-417e-a3db-325839e1b8e4,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 22.04.2 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.27.0-alpha.3.52+d48b8167f7094b,KubeProxyVersion:v1.27.0-alpha.3.52+d48b8167f7094b,OperatingSystem:linux,Architecture:arm64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:157636062,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/jessie-dnsutils@sha256:24aaf2626d6b27864c29de2097e8bbb840b3a414271bf7c8995e431e47d8408e registry.k8s.io/e2e-test-images/jessie-dnsutils:1.7],SizeBytes:116764683,},ContainerImage{Names:[registry.k8s.io/kube-proxy-arm64:v1.27.0-alpha.3.52_d48b8167f7094b],SizeBytes:60554082,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:16bbf38c463a4223d8cfe4da12bc61010b082a79b4bb003e2d3ba3ece5dd5f9e registry.k8s.io/e2e-test-images/agnhost:2.43],SizeBytes:49808364,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:3fe7acf013d1264ffded116b80a73dc129a449b0fccdb8d21af8279f2233f36e registry.k8s.io/e2e-test-images/httpd:2.4.39-4],SizeBytes:41528818,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:148b022f5c5da426fc2f3c14b5c0867e58ef05961510c84749ac1fddcb0fef22 registry.k8s.io/e2e-test-images/httpd:2.4.38-4],SizeBytes:40343601,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:28932904,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:ee3b525d5b89db99da3b8eb521d9cd90cb6e9ef0fbb651e98bb37be78d36b5b8 registry.k8s.io/sig-storage/csi-provisioner:v3.3.0],SizeBytes:23945474,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:425d8f1b769398127767b06ed97ce62578a3179bcb99809ce93a1649e025ffe7 registry.k8s.io/sig-storage/csi-resizer:v1.6.0],SizeBytes:22678632,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:291334908ddf71a4661fd7f6d9d97274de8a5378a2b6fdfeb2ce73414a34f82f registry.k8s.io/sig-storage/csi-snapshotter:v6.1.0],SizeBytes:22434180,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:9a685020911e2725ad019dbce6e4a5ab93d51e3d4557f115e64343345e05781b registry.k8s.io/sig-storage/csi-attacher:v4.0.0],SizeBytes:22394501,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:122bfb8c1edabb3c0edd63f06523e6940d958d19b3957dc7b1d6f81e9f1f6119 registry.k8s.io/sig-storage/csi-provisioner:v3.1.0],SizeBytes:21735644,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:9ebbf9f023e7b41ccee3d52afe39a89e3ddacdbb69269d583abfc25847cfd9e4 registry.k8s.io/sig-storage/csi-resizer:v1.4.0],SizeBytes:20848413,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:8b9c313c05f54fb04f8d430896f5f5904b6cb157df261501b29adc04d2b2dc7b registry.k8s.io/sig-storage/csi-attacher:v3.4.0],SizeBytes:20565480,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nonroot@sha256:ee9f50b3c64b174d296d91ca9f69a914ac30e59095dfb462b2b518ad28a63655 registry.k8s.io/e2e-test-images/nonroot:1.4],SizeBytes:18561972,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:92257881c1d6493cf18299a24af42330f891166560047902b8d431fb66b01af5 registry.k8s.io/sig-storage/hostpathplugin:v1.9.0],SizeBytes:18267611,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:9b2d9abda017c28b12f84a344f57cd73fbdb6c2bd7dd5bdd5018246ad1093ba6 registry.k8s.io/sig-storage/hostpathplugin:v1.11.0],SizeBytes:18159085,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:017727efcfeb7d053af68e51436ce8e65edbc6ca573720afb4f79c8594036955 registry.k8s.io/coredns/coredns:v1.10.0],SizeBytes:13766234,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:8502345,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:933940f13b3ea0abc62e656c1aa5c5b47c04b15d71250413a6b821bd0c58b94e registry.k8s.io/sig-storage/livenessprobe:v2.7.0],SizeBytes:8223819,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:7685305,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:5c99cf6a02adda929b10321dbf4ecfa00d87be9ba4fb456006237d530ab4baa1 registry.k8s.io/e2e-test-images/nginx:1.14-4],SizeBytes:6860089,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nonewprivs@sha256:8ac1264691820febacf3aea5d152cbde6d10685731ec14966a9401c6f47a68ac registry.k8s.io/e2e-test-images/nonewprivs:1.3],SizeBytes:5025027,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:2e0f836850e09b8b7cc937681d6194537a09fbd5f6b9e08f4d646a85128e8937 registry.k8s.io/e2e-test-images/busybox:1.29-4],SizeBytes:772933,},ContainerImage{Names:[registry.k8s.io/pause@sha256:7031c1b283388d2c2e09b57badb803c05ebed362dc88d84b480cc47f72a21097 registry.k8s.io/pause:3.9],SizeBytes:268051,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:253553,},},VolumesInUse:[kubernetes.io/csi/csi-hostpath-ephemeral-2147^52df5953-bb73-11ed-8aa7-be49170871b4 kubernetes.io/csi/csi-hostpath-ephemeral-5408^443d1daf-bb73-11ed-95c4-5a95d17cbc27 kubernetes.io/csi/csi-hostpath-ephemeral-5408^44411f51-bb73-11ed-95c4-5a95d17cbc27 kubernetes.io/csi/csi-hostpath-ephemeral-6540^4426a07f-bb73-11ed-a604-92687f2c50f8 kubernetes.io/csi/ebs.csi.aws.com^vol-05b67fab5beb6394b],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-05b67fab5beb6394b,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/csi-hostpath-ephemeral-6540^4426a07f-bb73-11ed-a604-92687f2c50f8,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/csi-hostpath-ephemeral-5408^44411f51-bb73-11ed-95c4-5a95d17cbc27,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/csi-hostpath-ephemeral-5408^443d1daf-bb73-11ed-95c4-5a95d17cbc27,DevicePath:,},},Config:nil,},} Mar 5 16:55:18.676: INFO: Logging kubelet events for node i-06df568f6e5c59fff Mar 5 16:55:18.792: INFO: Logging pods the kubelet thinks is on node i-06df568f6e5c59fff Mar 5 16:55:28.906: INFO: Unable to retrieve kubelet pods for node i-06df568f6e5c59fff: error trying to reach service: net/http: TLS handshake timeout Mar 5 16:55:28.906: INFO: Logging node info for node i-0d6a32f749839192f Mar 5 16:55:29.019: INFO: Node Info: &Node{ObjectMeta:{i-0d6a32f749839192f 4155fefb-4d3a-4b04-ba53-6ac9e895ff92 21031 0 2023-03-05 16:20:46 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:arm64 beta.kubernetes.io/instance-type:c6g.xlarge beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:eu-central-1 failure-domain.beta.kubernetes.io/zone:eu-central-1a kops.k8s.io/kops-controller-pki: kubernetes.io/arch:arm64 kubernetes.io/hostname:i-0d6a32f749839192f kubernetes.io/os:linux node-role.kubernetes.io/control-plane: node.kubernetes.io/exclude-from-external-load-balancers: node.kubernetes.io/instance-type:c6g.xlarge topology.ebs.csi.aws.com/zone:eu-central-1a topology.kubernetes.io/region:eu-central-1 topology.kubernetes.io/zone:eu-central-1a] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-0d6a32f749839192f"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kubelet Update v1 2023-03-05 16:20:46 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {protokube Update v1 2023-03-05 16:21:25 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kops.k8s.io/kops-controller-pki":{},"f:node-role.kubernetes.io/control-plane":{},"f:node.kubernetes.io/exclude-from-external-load-balancers":{}}}} } {kube-controller-manager Update v1 2023-03-05 16:21:47 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.0.0/24\"":{}}}} } {aws-cloud-controller-manager Update v1 2023-03-05 16:21:48 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{},"f:taints":{}}} } {aws-cloud-controller-manager Update v1 2023-03-05 16:21:48 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kubelet Update v1 2023-03-05 16:54:23 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.0.0/24,DoNotUseExternalID:,ProviderID:aws:///eu-central-1a/i-0d6a32f749839192f,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/control-plane,Value:,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[100.96.0.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{4 0} {<nil>} 4 DecimalSI},ephemeral-storage: {{49756430336 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-32Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-64Ki: {{0 0} {<nil>} 0 DecimalSI},memory: {{8150118400 0} {<nil>} 7959100Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{4 0} {<nil>} 4 DecimalSI},ephemeral-storage: {{44780787229 0} {<nil>} 44780787229 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-32Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-64Ki: {{0 0} {<nil>} 0 DecimalSI},memory: {{8045260800 0} {<nil>} 7856700Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-03-05 16:54:23 +0000 UTC,LastTransitionTime:2023-03-05 16:20:46 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-03-05 16:54:23 +0000 UTC,LastTransitionTime:2023-03-05 16:20:46 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-03-05 16:54:23 +0000 UTC,LastTransitionTime:2023-03-05 16:20:46 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-03-05 16:54:23 +0000 UTC,LastTransitionTime:2023-03-05 16:21:47 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.49.247,},NodeAddress{Type:ExternalIP,Address:18.197.20.82,},NodeAddress{Type:InternalDNS,Address:i-0d6a32f749839192f.eu-central-1.compute.internal,},NodeAddress{Type:Hostname,Address:i-0d6a32f749839192f.eu-central-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-18-197-20-82.eu-central-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2bed249e8ac7b517cb08ccd42b83e5,SystemUUID:ec2bed24-9e8a-c7b5-17cb-08ccd42b83e5,BootID:6f9215c9-dfe8-4b13-84f4-9b2084456198,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 22.04.2 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.27.0-alpha.3.52+d48b8167f7094b,KubeProxyVersion:v1.27.0-alpha.3.52+d48b8167f7094b,OperatingSystem:linux,Architecture:arm64,},Images:[]ContainerImage{ContainerImage{Names:[registry.k8s.io/etcd@sha256:9ce33ba33d8e738a5b85ed50b5080ac746deceed4a7496c550927a7a19ca3b6d registry.k8s.io/etcd:3.5.0-0],SizeBytes:157797353,},ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:157636062,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:4afb99b4690b418ffc2ceb67e1a17376457e441c1f09ab55447f0aaf992fa646 registry.k8s.io/etcd:3.4.3-0],SizeBytes:151677080,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:4ad90a11b55313b182afc186b9876c8e891531b8db4c9bf1541953021618d0e2 registry.k8s.io/etcd:3.4.13-0],SizeBytes:134531559,},ContainerImage{Names:[registry.k8s.io/kube-apiserver-arm64:v1.27.0-alpha.3.52_d48b8167f7094b],SizeBytes:131087707,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:0d2ab6b15b4fe0b903b9e81192d942fb39c29be61fe341cd16db5d0165b0435c registry.k8s.io/etcd:3.3.17-0],SizeBytes:126979107,},ContainerImage{Names:[registry.k8s.io/kube-controller-manager-arm64:v1.27.0-alpha.3.52_d48b8167f7094b],SizeBytes:120667128,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:17da501f5d2a675be46040422a27b7cc21b8a43895ac998b171db1c346f361f7 registry.k8s.io/etcd:3.3.10-0],SizeBytes:81436097,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:6f72b851544986cb0921b53ea655ec04c36131248f16d4ad110cb3ca0c369dc1 registry.k8s.io/etcd:3.5.4-0],SizeBytes:81120118,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:13f53ed1d91e2e11aac476ee9a0269fdda6cc4874eba903efd40daf50c55eee5 registry.k8s.io/etcd:3.5.3-0],SizeBytes:81114891,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:51eae8381dcb1078289fa7b4f3df2630cdc18d09fb56f8e56b41c40e191d6c83 registry.k8s.io/etcd:3.5.7-0],SizeBytes:80665728,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:dd75ec974b0a2a6f6bb47001ba09207976e625db898d1b16735528c009cb171c registry.k8s.io/etcd:3.5.6-0],SizeBytes:80539316,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:905d7ca17fd02bc24c0eba9a062753aba15db3e31422390bc3238eb762339b20 registry.k8s.io/etcd:3.2.24-1],SizeBytes:61500433,},ContainerImage{Names:[registry.k8s.io/etcdadm/etcd-manager-slim@sha256:8727adc059334e0e3329dbccce116a734da4a090da5e86c8cddccaf333caa666 registry.k8s.io/etcdadm/etcd-manager-slim:v3.0.20230201],SizeBytes:60826655,},ContainerImage{Names:[registry.k8s.io/kube-proxy-arm64:v1.27.0-alpha.3.52_d48b8167f7094b],SizeBytes:60554082,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:64b9ea357325d5db9f8a723dcf503b5a449177b17ac87d69481e126bb724c263 registry.k8s.io/etcd:3.5.1-0],SizeBytes:59553904,},ContainerImage{Names:[registry.k8s.io/kube-scheduler-arm64:v1.27.0-alpha.3.52_d48b8167f7094b],SizeBytes:56310763,},ContainerImage{Names:[registry.k8s.io/kops/kops-controller:1.27.0-alpha.1],SizeBytes:41797411,},ContainerImage{Names:[registry.k8s.io/kops/dns-controller:1.27.0-alpha.1],SizeBytes:40267473,},ContainerImage{Names:[public.ecr.aws/karpenter/controller@sha256:68db4f092cf9cc83f5ef9e2fbc5407c2cb682e81f64dfaa700a7602ede38b1cf public.ecr.aws/karpenter/controller:v0.16.3],SizeBytes:33707751,},ContainerImage{Names:[public.ecr.aws/karpenter/webhook@sha256:96a2d9b06d6bc5127801f358f74b1cf2d289b423a2e9ba40c573c0b14b17dafa public.ecr.aws/karpenter/webhook:v0.16.3],SizeBytes:33307187,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:28932904,},ContainerImage{Names:[quay.io/cilium/operator@sha256:a6d24a006a6b92967ac90786b49bc1ac26e5477cf028cd1186efcfc2466484db quay.io/cilium/operator:v1.12.5],SizeBytes:24567287,},ContainerImage{Names:[registry.k8s.io/provider-aws/cloud-controller-manager@sha256:fdeb61e3e42ecd9cca868d550ebdb88dd6341d9e91fcfa9a37e227dab2ad22cb registry.k8s.io/provider-aws/cloud-controller-manager:v1.26.0],SizeBytes:18345226,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:8502345,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:7685305,},ContainerImage{Names:[registry.k8s.io/kops/kube-apiserver-healthcheck:1.27.0-alpha.1],SizeBytes:4713612,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:253553,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Mar 5 16:55:29.019: INFO: Logging kubelet events for node i-0d6a32f749839192f Mar 5 16:55:29.135: INFO: ... skipping 52 lines ... Mar 5 16:55:29.257: INFO: Container controller ready: true, restart count 0 Mar 5 16:55:29.762: INFO: Latency metrics for node i-0d6a32f749839192f [1mSTEP:[0m Destroying namespace "statefulset-2394" for this suite. [38;5;243m@ 03/05/23 16:55:29.762[0m [38;5;243m<< Timeline[0m [38;5;9m[FAILED] Failed waiting for stateful set status.AvailableReplicas updated to 2: timed out waiting for the condition[0m [38;5;9mIn [1m[It][0m[38;5;9m at: [1mtest/e2e/framework/statefulset/wait.go:145[0m [38;5;243m@ 03/05/23 16:52:06.806[0m [38;5;243m------------------------------[0m [38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;10m•[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;10m•[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;10m•[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;10m•[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;10m•[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;10m•[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;10m•[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;10m•[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;10m•[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;10m•[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;10m•[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m [38;5;243m------------------------------[0m [38;5;9m• [FAILED] [984.575 seconds][0m [0m[sig-storage] CSI Volumes [38;5;243m[Driver: csi-hostpath] [0m[Testpattern: Dynamic PV (default fs)] volumes [38;5;9m[1m[It] should store data[0m [38;5;243mtest/e2e/storage/testsuites/volumes.go:159[0m [38;5;243mTimeline >>[0m [1mSTEP:[0m Creating a kubernetes client [38;5;243m@ 03/05/23 16:41:57.153[0m Mar 5 16:41:57.153: INFO: >>> kubeConfig: /root/.kube/config ... skipping 193 lines ... Mar 5 16:46:53.147: INFO: PersistentVolumeClaim csi-hostpathrv5x2 found but phase is Pending instead of Bound. Mar 5 16:46:55.260: INFO: PersistentVolumeClaim csi-hostpathrv5x2 found but phase is Pending instead of Bound. Mar 5 16:46:57.372: INFO: PersistentVolumeClaim csi-hostpathrv5x2 found but phase is Pending instead of Bound. Mar 5 16:46:59.484: INFO: PersistentVolumeClaim csi-hostpathrv5x2 found but phase is Pending instead of Bound. Mar 5 16:47:01.596: INFO: PersistentVolumeClaim csi-hostpathrv5x2 found but phase is Pending instead of Bound. Mar 5 16:47:03.709: INFO: PersistentVolumeClaim csi-hostpathrv5x2 found but phase is Pending instead of Bound. Mar 5 16:47:05.710: INFO: Unexpected error: <*errors.errorString | 0xc0010c5430>: { s: "PersistentVolumeClaims [csi-hostpathrv5x2] not all in phase Bound within 5m0s", } [38;5;9m[FAILED][0m in [It] - test/e2e/storage/framework/volume_resource.go:311 [38;5;243m@ 03/05/23 16:47:05.71[0m Mar 5 16:47:05.711: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready Mar 5 16:47:05.823: INFO: Condition Ready of node i-06df568f6e5c59fff is false, but Node is tainted by NodeController with [{node.kubernetes.io/not-ready NoSchedule 2023-03-05 16:46:33 +0000 UTC} {node.kubernetes.io/not-ready NoExecute 2023-03-05 16:46:37 +0000 UTC}]. Failure Mar 5 16:47:07.938: INFO: Condition Ready of node i-06df568f6e5c59fff is false, but Node is tainted by NodeController with [{node.kubernetes.io/not-ready NoSchedule 2023-03-05 16:46:33 +0000 UTC} {node.kubernetes.io/not-ready NoExecute 2023-03-05 16:46:37 +0000 UTC}]. Failure Mar 5 16:47:09.938: INFO: Condition Ready of node i-06df568f6e5c59fff is false, but Node is tainted by NodeController with [{node.kubernetes.io/not-ready NoSchedule 2023-03-05 16:46:33 +0000 UTC} {node.kubernetes.io/not-ready NoExecute 2023-03-05 16:46:37 +0000 UTC}]. Failure Mar 5 16:47:11.937: INFO: Condition Ready of node i-06df568f6e5c59fff is false, but Node is tainted by NodeController with [{node.kubernetes.io/not-ready NoSchedule 2023-03-05 16:46:33 +0000 UTC} {node.kubernetes.io/not-ready NoExecute 2023-03-05 16:46:37 +0000 UTC}]. Failure Mar 5 16:47:13.937: INFO: Condition Ready of node i-06df568f6e5c59fff is false, but Node is tainted by NodeController with [{node.kubernetes.io/not-ready NoSchedule 2023-03-05 16:46:33 +0000 UTC} {node.kubernetes.io/not-ready NoExecute 2023-03-05 16:46:37 +0000 UTC}]. Failure ... skipping 149 lines ... Logging node info for node i-06df568f6e5c59fff Mar 5 16:50:07.302: INFO: Node Info: &Node{ObjectMeta:{i-06df568f6e5c59fff cb63c681-2e18-474a-b6f3-221325ac0d52 18792 0 2023-03-05 16:24:16 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:arm64 beta.kubernetes.io/instance-type:t4g.small beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:eu-central-1 failure-domain.beta.kubernetes.io/zone:eu-central-1a karpenter.k8s.aws/instance-category:t karpenter.k8s.aws/instance-cpu:2 karpenter.k8s.aws/instance-family:t4g karpenter.k8s.aws/instance-generation:4 karpenter.k8s.aws/instance-hypervisor:nitro karpenter.k8s.aws/instance-memory:2048 karpenter.k8s.aws/instance-pods:11 karpenter.k8s.aws/instance-size:small karpenter.sh/capacity-type:spot karpenter.sh/initialized:true karpenter.sh/provisioner-name:nodes kubernetes.io/arch:arm64 kubernetes.io/hostname:i-06df568f6e5c59fff kubernetes.io/os:linux node-role.kubernetes.io/node: node-role.kubernetes.io/spot-worker:true node.kubernetes.io/instance-type:t4g.small topology.ebs.csi.aws.com/zone:eu-central-1a topology.hostpath.csi/node:i-06df568f6e5c59fff topology.kubernetes.io/region:eu-central-1 topology.kubernetes.io/zone:eu-central-1a] map[csi.volume.kubernetes.io/nodeid:{"csi-hostpath-ephemeral-2147":"i-06df568f6e5c59fff","csi-hostpath-ephemeral-5408":"i-06df568f6e5c59fff","csi-hostpath-ephemeral-6540":"i-06df568f6e5c59fff","csi-mock-csi-mock-volumes-fsgroup-policy-1058":"i-06df568f6e5c59fff","csi-mock-csi-mock-volumes-workload-2219":"i-06df568f6e5c59fff","ebs.csi.aws.com":"i-06df568f6e5c59fff"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [{karpenter.sh/v1alpha5 Provisioner nodes ac5d7599-b2a2-4eae-8da4-1f4c0a54ec82 <nil> 0xc0040cde90}] [karpenter.sh/termination] [{aws-cloud-controller-manager Update v1 2023-03-05 16:24:16 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{}}}} } {kops-controller Update v1 2023-03-05 16:24:16 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{},"f:node-role.kubernetes.io/spot-worker":{}}}} } {karpenter Update v1 2023-03-05 16:25:57 +0000 UTC FieldsV1 {"f:metadata":{"f:finalizers":{".":{},"v:\"karpenter.sh/termination\"":{}},"f:labels":{".":{},"f:karpenter.k8s.aws/instance-category":{},"f:karpenter.k8s.aws/instance-cpu":{},"f:karpenter.k8s.aws/instance-family":{},"f:karpenter.k8s.aws/instance-generation":{},"f:karpenter.k8s.aws/instance-hypervisor":{},"f:karpenter.k8s.aws/instance-memory":{},"f:karpenter.k8s.aws/instance-pods":{},"f:karpenter.k8s.aws/instance-size":{},"f:karpenter.sh/capacity-type":{},"f:karpenter.sh/initialized":{},"f:karpenter.sh/provisioner-name":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}},"f:ownerReferences":{".":{},"k:{\"uid\":\"ac5d7599-b2a2-4eae-8da4-1f4c0a54ec82\"}":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-03-05 16:26:48 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{".":{},"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"InternalIP\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kubelet Update v1 2023-03-05 16:48:40 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{"f:kubernetes.io/hostname":{},"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{".":{},"f:cpu":{},"f:ephemeral-storage":{},"f:hugepages-1Gi":{},"f:hugepages-2Mi":{},"f:hugepages-32Mi":{},"f:hugepages-64Ki":{},"f:memory":{},"f:pods":{}},"f:capacity":{".":{},"f:cpu":{},"f:ephemeral-storage":{},"f:hugepages-1Gi":{},"f:hugepages-2Mi":{},"f:hugepages-32Mi":{},"f:hugepages-64Ki":{},"f:memory":{},"f:pods":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{}}},"f:daemonEndpoints":{"f:kubeletEndpoint":{"f:Port":{}}},"f:images":{},"f:nodeInfo":{"f:architecture":{},"f:bootID":{},"f:containerRuntimeVersion":{},"f:kernelVersion":{},"f:kubeProxyVersion":{},"f:kubeletVersion":{},"f:machineID":{},"f:operatingSystem":{},"f:osImage":{},"f:systemUUID":{}},"f:volumesInUse":{}}} status} {kube-controller-manager Update v1 2023-03-05 16:49:37 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{".":{},"k:{\"type\":\"DiskPressure\"}":{".":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"MemoryPressure\"}":{".":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"PIDPressure\"}":{".":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Ready\"}":{".":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}},"f:volumesAttached":{}}} status} {kube-controller-manager Update v1 2023-03-05 16:49:42 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:node.alpha.kubernetes.io/ttl":{}},"f:labels":{"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.2.0/24\"":{}},"f:taints":{}}} }]},Spec:NodeSpec{PodCIDR:100.96.2.0/24,DoNotUseExternalID:,ProviderID:aws:///eu-central-1a/i-06df568f6e5c59fff,Unschedulable:false,Taints:[]Taint{Taint{Key:node.kubernetes.io/unreachable,Value:,Effect:NoSchedule,TimeAdded:2023-03-05 16:49:37 +0000 UTC,},Taint{Key:node.kubernetes.io/unreachable,Value:,Effect:NoExecute,TimeAdded:2023-03-05 16:49:42 +0000 UTC,},},ConfigSource:nil,PodCIDRs:[100.96.2.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49756430336 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-32Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-64Ki: {{0 0} {<nil>} 0 DecimalSI},memory: {{1941540864 0} {<nil>} 1896036Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44780787229 0} {<nil>} 44780787229 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-32Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-64Ki: {{0 0} {<nil>} 0 DecimalSI},memory: {{1836683264 0} {<nil>} 1793636Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:Ready,Status:Unknown,LastHeartbeatTime:2023-03-05 16:47:34 +0000 UTC,LastTransitionTime:2023-03-05 16:49:37 +0000 UTC,Reason:NodeStatusUnknown,Message:Kubelet stopped posting node status.,},NodeCondition{Type:MemoryPressure,Status:Unknown,LastHeartbeatTime:2023-03-05 16:47:34 +0000 UTC,LastTransitionTime:2023-03-05 16:49:37 +0000 UTC,Reason:NodeStatusUnknown,Message:Kubelet stopped posting node status.,},NodeCondition{Type:DiskPressure,Status:Unknown,LastHeartbeatTime:2023-03-05 16:47:34 +0000 UTC,LastTransitionTime:2023-03-05 16:49:37 +0000 UTC,Reason:NodeStatusUnknown,Message:Kubelet stopped posting node status.,},NodeCondition{Type:PIDPressure,Status:Unknown,LastHeartbeatTime:2023-03-05 16:47:34 +0000 UTC,LastTransitionTime:2023-03-05 16:49:37 +0000 UTC,Reason:NodeStatusUnknown,Message:Kubelet stopped posting node status.,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.43.148,},NodeAddress{Type:ExternalIP,Address:18.192.211.222,},NodeAddress{Type:InternalDNS,Address:i-06df568f6e5c59fff.eu-central-1.compute.internal,},NodeAddress{Type:Hostname,Address:i-06df568f6e5c59fff.eu-central-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-18-192-211-222.eu-central-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec221221864d9409a8ec5f6614a9b94b,SystemUUID:ec221221-864d-9409-a8ec-5f6614a9b94b,BootID:553d31bf-faca-417e-a3db-325839e1b8e4,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 22.04.2 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.27.0-alpha.3.52+d48b8167f7094b,KubeProxyVersion:v1.27.0-alpha.3.52+d48b8167f7094b,OperatingSystem:linux,Architecture:arm64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:157636062,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/jessie-dnsutils@sha256:24aaf2626d6b27864c29de2097e8bbb840b3a414271bf7c8995e431e47d8408e registry.k8s.io/e2e-test-images/jessie-dnsutils:1.7],SizeBytes:116764683,},ContainerImage{Names:[registry.k8s.io/kube-proxy-arm64:v1.27.0-alpha.3.52_d48b8167f7094b],SizeBytes:60554082,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:16bbf38c463a4223d8cfe4da12bc61010b082a79b4bb003e2d3ba3ece5dd5f9e registry.k8s.io/e2e-test-images/agnhost:2.43],SizeBytes:49808364,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:3fe7acf013d1264ffded116b80a73dc129a449b0fccdb8d21af8279f2233f36e registry.k8s.io/e2e-test-images/httpd:2.4.39-4],SizeBytes:41528818,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:148b022f5c5da426fc2f3c14b5c0867e58ef05961510c84749ac1fddcb0fef22 registry.k8s.io/e2e-test-images/httpd:2.4.38-4],SizeBytes:40343601,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:28932904,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:ee3b525d5b89db99da3b8eb521d9cd90cb6e9ef0fbb651e98bb37be78d36b5b8 registry.k8s.io/sig-storage/csi-provisioner:v3.3.0],SizeBytes:23945474,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:425d8f1b769398127767b06ed97ce62578a3179bcb99809ce93a1649e025ffe7 registry.k8s.io/sig-storage/csi-resizer:v1.6.0],SizeBytes:22678632,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:291334908ddf71a4661fd7f6d9d97274de8a5378a2b6fdfeb2ce73414a34f82f registry.k8s.io/sig-storage/csi-snapshotter:v6.1.0],SizeBytes:22434180,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:9a685020911e2725ad019dbce6e4a5ab93d51e3d4557f115e64343345e05781b registry.k8s.io/sig-storage/csi-attacher:v4.0.0],SizeBytes:22394501,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:122bfb8c1edabb3c0edd63f06523e6940d958d19b3957dc7b1d6f81e9f1f6119 registry.k8s.io/sig-storage/csi-provisioner:v3.1.0],SizeBytes:21735644,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:9ebbf9f023e7b41ccee3d52afe39a89e3ddacdbb69269d583abfc25847cfd9e4 registry.k8s.io/sig-storage/csi-resizer:v1.4.0],SizeBytes:20848413,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:8b9c313c05f54fb04f8d430896f5f5904b6cb157df261501b29adc04d2b2dc7b registry.k8s.io/sig-storage/csi-attacher:v3.4.0],SizeBytes:20565480,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nonroot@sha256:ee9f50b3c64b174d296d91ca9f69a914ac30e59095dfb462b2b518ad28a63655 registry.k8s.io/e2e-test-images/nonroot:1.4],SizeBytes:18561972,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:92257881c1d6493cf18299a24af42330f891166560047902b8d431fb66b01af5 registry.k8s.io/sig-storage/hostpathplugin:v1.9.0],SizeBytes:18267611,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:9b2d9abda017c28b12f84a344f57cd73fbdb6c2bd7dd5bdd5018246ad1093ba6 registry.k8s.io/sig-storage/hostpathplugin:v1.11.0],SizeBytes:18159085,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:017727efcfeb7d053af68e51436ce8e65edbc6ca573720afb4f79c8594036955 registry.k8s.io/coredns/coredns:v1.10.0],SizeBytes:13766234,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:8502345,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:933940f13b3ea0abc62e656c1aa5c5b47c04b15d71250413a6b821bd0c58b94e registry.k8s.io/sig-storage/livenessprobe:v2.7.0],SizeBytes:8223819,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:7685305,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:5c99cf6a02adda929b10321dbf4ecfa00d87be9ba4fb456006237d530ab4baa1 registry.k8s.io/e2e-test-images/nginx:1.14-4],SizeBytes:6860089,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nonewprivs@sha256:8ac1264691820febacf3aea5d152cbde6d10685731ec14966a9401c6f47a68ac registry.k8s.io/e2e-test-images/nonewprivs:1.3],SizeBytes:5025027,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:2e0f836850e09b8b7cc937681d6194537a09fbd5f6b9e08f4d646a85128e8937 registry.k8s.io/e2e-test-images/busybox:1.29-4],SizeBytes:772933,},ContainerImage{Names:[registry.k8s.io/pause@sha256:7031c1b283388d2c2e09b57badb803c05ebed362dc88d84b480cc47f72a21097 registry.k8s.io/pause:3.9],SizeBytes:268051,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:253553,},},VolumesInUse:[kubernetes.io/csi/csi-hostpath-ephemeral-2147^52df5953-bb73-11ed-8aa7-be49170871b4 kubernetes.io/csi/csi-hostpath-ephemeral-5408^443d1daf-bb73-11ed-95c4-5a95d17cbc27 kubernetes.io/csi/csi-hostpath-ephemeral-5408^44411f51-bb73-11ed-95c4-5a95d17cbc27 kubernetes.io/csi/csi-hostpath-ephemeral-6540^4426a07f-bb73-11ed-a604-92687f2c50f8 kubernetes.io/csi/ebs.csi.aws.com^vol-05b67fab5beb6394b],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-05b67fab5beb6394b,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/csi-hostpath-ephemeral-6540^4426a07f-bb73-11ed-a604-92687f2c50f8,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/csi-hostpath-ephemeral-5408^44411f51-bb73-11ed-95c4-5a95d17cbc27,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/csi-hostpath-ephemeral-5408^443d1daf-bb73-11ed-95c4-5a95d17cbc27,DevicePath:,},},Config:nil,},} Mar 5 16:50:07.302: INFO: Logging kubelet events for node i-06df568f6e5c59fff Mar 5 16:50:07.417: INFO: Logging pods the kubelet thinks is on node i-06df568f6e5c59fff Mar 5 16:50:17.530: INFO: Unable to retrieve kubelet pods for node i-06df568f6e5c59fff: error trying to reach service: net/http: TLS handshake timeout Mar 5 16:50:17.530: INFO: Logging node info for node i-0d6a32f749839192f Mar 5 16:50:17.642: INFO: Node Info: &Node{ObjectMeta:{i-0d6a32f749839192f 4155fefb-4d3a-4b04-ba53-6ac9e895ff92 18509 0 2023-03-05 16:20:46 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:arm64 beta.kubernetes.io/instance-type:c6g.xlarge beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:eu-central-1 failure-domain.beta.kubernetes.io/zone:eu-central-1a kops.k8s.io/kops-controller-pki: kubernetes.io/arch:arm64 kubernetes.io/hostname:i-0d6a32f749839192f kubernetes.io/os:linux node-role.kubernetes.io/control-plane: node.kubernetes.io/exclude-from-external-load-balancers: node.kubernetes.io/instance-type:c6g.xlarge topology.ebs.csi.aws.com/zone:eu-central-1a topology.kubernetes.io/region:eu-central-1 topology.kubernetes.io/zone:eu-central-1a] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-0d6a32f749839192f"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kubelet Update v1 2023-03-05 16:20:46 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {protokube Update v1 2023-03-05 16:21:25 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kops.k8s.io/kops-controller-pki":{},"f:node-role.kubernetes.io/control-plane":{},"f:node.kubernetes.io/exclude-from-external-load-balancers":{}}}} } {kube-controller-manager Update v1 2023-03-05 16:21:47 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.0.0/24\"":{}}}} } {aws-cloud-controller-manager Update v1 2023-03-05 16:21:48 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{},"f:taints":{}}} } {aws-cloud-controller-manager Update v1 2023-03-05 16:21:48 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kubelet Update v1 2023-03-05 16:49:18 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.0.0/24,DoNotUseExternalID:,ProviderID:aws:///eu-central-1a/i-0d6a32f749839192f,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/control-plane,Value:,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[100.96.0.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{4 0} {<nil>} 4 DecimalSI},ephemeral-storage: {{49756430336 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-32Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-64Ki: {{0 0} {<nil>} 0 DecimalSI},memory: {{8150118400 0} {<nil>} 7959100Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{4 0} {<nil>} 4 DecimalSI},ephemeral-storage: {{44780787229 0} {<nil>} 44780787229 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-32Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-64Ki: {{0 0} {<nil>} 0 DecimalSI},memory: {{8045260800 0} {<nil>} 7856700Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-03-05 16:49:18 +0000 UTC,LastTransitionTime:2023-03-05 16:20:46 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-03-05 16:49:18 +0000 UTC,LastTransitionTime:2023-03-05 16:20:46 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-03-05 16:49:18 +0000 UTC,LastTransitionTime:2023-03-05 16:20:46 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-03-05 16:49:18 +0000 UTC,LastTransitionTime:2023-03-05 16:21:47 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.49.247,},NodeAddress{Type:ExternalIP,Address:18.197.20.82,},NodeAddress{Type:InternalDNS,Address:i-0d6a32f749839192f.eu-central-1.compute.internal,},NodeAddress{Type:Hostname,Address:i-0d6a32f749839192f.eu-central-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-18-197-20-82.eu-central-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2bed249e8ac7b517cb08ccd42b83e5,SystemUUID:ec2bed24-9e8a-c7b5-17cb-08ccd42b83e5,BootID:6f9215c9-dfe8-4b13-84f4-9b2084456198,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 22.04.2 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.27.0-alpha.3.52+d48b8167f7094b,KubeProxyVersion:v1.27.0-alpha.3.52+d48b8167f7094b,OperatingSystem:linux,Architecture:arm64,},Images:[]ContainerImage{ContainerImage{Names:[registry.k8s.io/etcd@sha256:9ce33ba33d8e738a5b85ed50b5080ac746deceed4a7496c550927a7a19ca3b6d registry.k8s.io/etcd:3.5.0-0],SizeBytes:157797353,},ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:157636062,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:4afb99b4690b418ffc2ceb67e1a17376457e441c1f09ab55447f0aaf992fa646 registry.k8s.io/etcd:3.4.3-0],SizeBytes:151677080,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:4ad90a11b55313b182afc186b9876c8e891531b8db4c9bf1541953021618d0e2 registry.k8s.io/etcd:3.4.13-0],SizeBytes:134531559,},ContainerImage{Names:[registry.k8s.io/kube-apiserver-arm64:v1.27.0-alpha.3.52_d48b8167f7094b],SizeBytes:131087707,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:0d2ab6b15b4fe0b903b9e81192d942fb39c29be61fe341cd16db5d0165b0435c registry.k8s.io/etcd:3.3.17-0],SizeBytes:126979107,},ContainerImage{Names:[registry.k8s.io/kube-controller-manager-arm64:v1.27.0-alpha.3.52_d48b8167f7094b],SizeBytes:120667128,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:17da501f5d2a675be46040422a27b7cc21b8a43895ac998b171db1c346f361f7 registry.k8s.io/etcd:3.3.10-0],SizeBytes:81436097,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:6f72b851544986cb0921b53ea655ec04c36131248f16d4ad110cb3ca0c369dc1 registry.k8s.io/etcd:3.5.4-0],SizeBytes:81120118,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:13f53ed1d91e2e11aac476ee9a0269fdda6cc4874eba903efd40daf50c55eee5 registry.k8s.io/etcd:3.5.3-0],SizeBytes:81114891,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:51eae8381dcb1078289fa7b4f3df2630cdc18d09fb56f8e56b41c40e191d6c83 registry.k8s.io/etcd:3.5.7-0],SizeBytes:80665728,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:dd75ec974b0a2a6f6bb47001ba09207976e625db898d1b16735528c009cb171c registry.k8s.io/etcd:3.5.6-0],SizeBytes:80539316,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:905d7ca17fd02bc24c0eba9a062753aba15db3e31422390bc3238eb762339b20 registry.k8s.io/etcd:3.2.24-1],SizeBytes:61500433,},ContainerImage{Names:[registry.k8s.io/etcdadm/etcd-manager-slim@sha256:8727adc059334e0e3329dbccce116a734da4a090da5e86c8cddccaf333caa666 registry.k8s.io/etcdadm/etcd-manager-slim:v3.0.20230201],SizeBytes:60826655,},ContainerImage{Names:[registry.k8s.io/kube-proxy-arm64:v1.27.0-alpha.3.52_d48b8167f7094b],SizeBytes:60554082,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:64b9ea357325d5db9f8a723dcf503b5a449177b17ac87d69481e126bb724c263 registry.k8s.io/etcd:3.5.1-0],SizeBytes:59553904,},ContainerImage{Names:[registry.k8s.io/kube-scheduler-arm64:v1.27.0-alpha.3.52_d48b8167f7094b],SizeBytes:56310763,},ContainerImage{Names:[registry.k8s.io/kops/kops-controller:1.27.0-alpha.1],SizeBytes:41797411,},ContainerImage{Names:[registry.k8s.io/kops/dns-controller:1.27.0-alpha.1],SizeBytes:40267473,},ContainerImage{Names:[public.ecr.aws/karpenter/controller@sha256:68db4f092cf9cc83f5ef9e2fbc5407c2cb682e81f64dfaa700a7602ede38b1cf public.ecr.aws/karpenter/controller:v0.16.3],SizeBytes:33707751,},ContainerImage{Names:[public.ecr.aws/karpenter/webhook@sha256:96a2d9b06d6bc5127801f358f74b1cf2d289b423a2e9ba40c573c0b14b17dafa public.ecr.aws/karpenter/webhook:v0.16.3],SizeBytes:33307187,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:28932904,},ContainerImage{Names:[quay.io/cilium/operator@sha256:a6d24a006a6b92967ac90786b49bc1ac26e5477cf028cd1186efcfc2466484db quay.io/cilium/operator:v1.12.5],SizeBytes:24567287,},ContainerImage{Names:[registry.k8s.io/provider-aws/cloud-controller-manager@sha256:fdeb61e3e42ecd9cca868d550ebdb88dd6341d9e91fcfa9a37e227dab2ad22cb registry.k8s.io/provider-aws/cloud-controller-manager:v1.26.0],SizeBytes:18345226,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:8502345,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:7685305,},ContainerImage{Names:[registry.k8s.io/kops/kube-apiserver-healthcheck:1.27.0-alpha.1],SizeBytes:4713612,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:253553,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Mar 5 16:50:17.642: INFO: Logging kubelet events for node i-0d6a32f749839192f Mar 5 16:50:17.757: INFO: ... skipping 60 lines ... Mar 5 16:50:30.712: INFO: At 2023-03-05 16:42:03 +0000 UTC - event for csi-hostpathplugin: {statefulset-controller } SuccessfulCreate: create Pod csi-hostpathplugin-0 in StatefulSet csi-hostpathplugin successful Mar 5 16:50:30.712: INFO: At 2023-03-05 16:42:03 +0000 UTC - event for csi-hostpathplugin-0: {default-scheduler } Scheduled: Successfully assigned volume-2381-9266/csi-hostpathplugin-0 to i-06df568f6e5c59fff Mar 5 16:50:30.712: INFO: At 2023-03-05 16:47:57 +0000 UTC - event for csi-hostpathplugin-0: {taint-controller } TaintManagerEviction: Marking for deletion Pod volume-2381-9266/csi-hostpathplugin-0 Mar 5 16:50:30.824: INFO: POD NODE PHASE GRACE CONDITIONS Mar 5 16:50:30.824: INFO: csi-hostpathplugin-0 i-06df568f6e5c59fff Pending 30s [{PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2023-03-05 16:42:03 +0000 UTC } {DisruptionTarget True 0001-01-01 00:00:00 +0000 UTC 2023-03-05 16:47:57 +0000 UTC DeletionByTaintManager Taint manager: deleting due to NoExecute taint}] Mar 5 16:50:30.824: INFO: Mar 5 16:50:40.938: INFO: Unable to fetch volume-2381-9266/csi-hostpathplugin-0/hostpath logs: an error on the server ("unknown") has prevented the request from succeeding (get pods csi-hostpathplugin-0) Mar 5 16:50:51.052: INFO: Unable to fetch volume-2381-9266/csi-hostpathplugin-0/node-driver-registrar logs: an error on the server ("unknown") has prevented the request from succeeding (get pods csi-hostpathplugin-0) Mar 5 16:51:01.167: INFO: Unable to fetch volume-2381-9266/csi-hostpathplugin-0/liveness-probe logs: an error on the server ("unknown") has prevented the request from succeeding (get pods csi-hostpathplugin-0) Mar 5 16:51:11.284: INFO: Unable to fetch volume-2381-9266/csi-hostpathplugin-0/csi-attacher logs: an error on the server ("unknown") has prevented the request from succeeding (get pods csi-hostpathplugin-0) Mar 5 16:51:21.398: INFO: Unable to fetch volume-2381-9266/csi-hostpathplugin-0/csi-provisioner logs: an error on the server ("unknown") has prevented the request from succeeding (get pods csi-hostpathplugin-0) Mar 5 16:51:31.512: INFO: Unable to fetch volume-2381-9266/csi-hostpathplugin-0/csi-resizer logs: an error on the server ("unknown") has prevented the request from succeeding (get pods csi-hostpathplugin-0) Mar 5 16:51:41.626: INFO: Unable to fetch volume-2381-9266/csi-hostpathplugin-0/csi-snapshotter logs: an error on the server ("unknown") has prevented the request from succeeding (get pods csi-hostpathplugin-0) Mar 5 16:51:41.738: INFO: Logging node info for node i-056ad61525499081e Mar 5 16:51:41.850: INFO: Node Info: &Node{ObjectMeta:{i-056ad61525499081e a6ce9ed3-5ae8-4065-b117-6884e24b3632 18969 0 2023-03-05 16:24:16 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:arm64 beta.kubernetes.io/instance-type:t4g.small beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:eu-central-1 failure-domain.beta.kubernetes.io/zone:eu-central-1a karpenter.k8s.aws/instance-category:t karpenter.k8s.aws/instance-cpu:2 karpenter.k8s.aws/instance-family:t4g karpenter.k8s.aws/instance-generation:4 karpenter.k8s.aws/instance-hypervisor:nitro karpenter.k8s.aws/instance-memory:2048 karpenter.k8s.aws/instance-pods:11 karpenter.k8s.aws/instance-size:small karpenter.sh/capacity-type:spot karpenter.sh/initialized:true karpenter.sh/provisioner-name:nodes kubernetes.io/arch:arm64 kubernetes.io/hostname:i-056ad61525499081e kubernetes.io/os:linux node-role.kubernetes.io/node: node-role.kubernetes.io/spot-worker:true node.kubernetes.io/instance-type:t4g.small topology.ebs.csi.aws.com/zone:eu-central-1a topology.hostpath.csi/node:i-056ad61525499081e topology.kubernetes.io/region:eu-central-1 topology.kubernetes.io/zone:eu-central-1a] map[csi.volume.kubernetes.io/nodeid:{"csi-hostpath-read-write-once-pod-5175":"i-056ad61525499081e","csi-mock-csi-mock-volumes-fsgroup-policy-6623":"i-056ad61525499081e","ebs.csi.aws.com":"i-056ad61525499081e"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [{karpenter.sh/v1alpha5 Provisioner nodes ac5d7599-b2a2-4eae-8da4-1f4c0a54ec82 <nil> 0xc004d23120}] [karpenter.sh/termination] [{aws-cloud-controller-manager Update v1 2023-03-05 16:24:16 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{}}}} } {kops-controller Update v1 2023-03-05 16:24:16 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{},"f:node-role.kubernetes.io/spot-worker":{}}}} } {karpenter Update v1 2023-03-05 16:25:57 +0000 UTC FieldsV1 {"f:metadata":{"f:finalizers":{".":{},"v:\"karpenter.sh/termination\"":{}},"f:labels":{".":{},"f:karpenter.k8s.aws/instance-category":{},"f:karpenter.k8s.aws/instance-cpu":{},"f:karpenter.k8s.aws/instance-family":{},"f:karpenter.k8s.aws/instance-generation":{},"f:karpenter.k8s.aws/instance-hypervisor":{},"f:karpenter.k8s.aws/instance-memory":{},"f:karpenter.k8s.aws/instance-pods":{},"f:karpenter.k8s.aws/instance-size":{},"f:karpenter.sh/capacity-type":{},"f:karpenter.sh/initialized":{},"f:karpenter.sh/provisioner-name":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}},"f:ownerReferences":{".":{},"k:{\"uid\":\"ac5d7599-b2a2-4eae-8da4-1f4c0a54ec82\"}":{}}},"f:spec":{"f:providerID":{}}} } {kube-controller-manager Update v1 2023-03-05 16:25:57 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:node.alpha.kubernetes.io/ttl":{}},"f:labels":{"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.1.0/24\"":{}}}} } {aws-cloud-controller-manager Update v1 2023-03-05 16:26:48 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{".":{},"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"InternalIP\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kube-controller-manager Update v1 2023-03-05 16:49:47 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{".":{},"k:{\"type\":\"DiskPressure\"}":{".":{},"f:type":{}},"k:{\"type\":\"MemoryPressure\"}":{".":{},"f:type":{}},"k:{\"type\":\"PIDPressure\"}":{".":{},"f:type":{}},"k:{\"type\":\"Ready\"}":{".":{},"f:type":{}}},"f:volumesAttached":{}}} status} {kubelet Update v1 2023-03-05 16:49:50 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{"f:kubernetes.io/hostname":{},"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{".":{},"f:cpu":{},"f:ephemeral-storage":{},"f:hugepages-1Gi":{},"f:hugepages-2Mi":{},"f:hugepages-32Mi":{},"f:hugepages-64Ki":{},"f:memory":{},"f:pods":{}},"f:capacity":{".":{},"f:cpu":{},"f:ephemeral-storage":{},"f:hugepages-1Gi":{},"f:hugepages-2Mi":{},"f:hugepages-32Mi":{},"f:hugepages-64Ki":{},"f:memory":{},"f:pods":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:daemonEndpoints":{"f:kubeletEndpoint":{"f:Port":{}}},"f:images":{},"f:nodeInfo":{"f:architecture":{},"f:bootID":{},"f:containerRuntimeVersion":{},"f:kernelVersion":{},"f:kubeProxyVersion":{},"f:kubeletVersion":{},"f:machineID":{},"f:operatingSystem":{},"f:osImage":{},"f:systemUUID":{}},"f:volumesInUse":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.1.0/24,DoNotUseExternalID:,ProviderID:aws:///eu-central-1a/i-056ad61525499081e,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.1.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49756430336 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-32Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-64Ki: {{0 0} {<nil>} 0 DecimalSI},memory: {{1941540864 0} {<nil>} 1896036Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44780787229 0} {<nil>} 44780787229 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-32Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-64Ki: {{0 0} {<nil>} 0 DecimalSI},memory: {{1836683264 0} {<nil>} 1793636Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-03-05 16:49:48 +0000 UTC,LastTransitionTime:2023-03-05 16:25:57 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-03-05 16:49:48 +0000 UTC,LastTransitionTime:2023-03-05 16:25:48 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-03-05 16:49:48 +0000 UTC,LastTransitionTime:2023-03-05 16:25:48 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-03-05 16:49:48 +0000 UTC,LastTransitionTime:2023-03-05 16:25:48 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.36.179,},NodeAddress{Type:ExternalIP,Address:18.159.37.200,},NodeAddress{Type:InternalDNS,Address:i-056ad61525499081e.eu-central-1.compute.internal,},NodeAddress{Type:Hostname,Address:i-056ad61525499081e.eu-central-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-18-159-37-200.eu-central-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2af3207d1c8c09785c337aae6c6ef0,SystemUUID:ec2af320-7d1c-8c09-785c-337aae6c6ef0,BootID:786f2752-bc30-4729-8aaf-45977b16c253,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 22.04.2 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.27.0-alpha.3.52+d48b8167f7094b,KubeProxyVersion:v1.27.0-alpha.3.52+d48b8167f7094b,OperatingSystem:linux,Architecture:arm64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:157636062,},ContainerImage{Names:[registry.k8s.io/kube-proxy-arm64:v1.27.0-alpha.3.52_d48b8167f7094b],SizeBytes:60554082,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:16bbf38c463a4223d8cfe4da12bc61010b082a79b4bb003e2d3ba3ece5dd5f9e registry.k8s.io/e2e-test-images/agnhost:2.43],SizeBytes:49808364,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:3fe7acf013d1264ffded116b80a73dc129a449b0fccdb8d21af8279f2233f36e registry.k8s.io/e2e-test-images/httpd:2.4.39-4],SizeBytes:41528818,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:148b022f5c5da426fc2f3c14b5c0867e58ef05961510c84749ac1fddcb0fef22 registry.k8s.io/e2e-test-images/httpd:2.4.38-4],SizeBytes:40343601,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:28932904,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:ee3b525d5b89db99da3b8eb521d9cd90cb6e9ef0fbb651e98bb37be78d36b5b8 registry.k8s.io/sig-storage/csi-provisioner:v3.3.0],SizeBytes:23945474,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:425d8f1b769398127767b06ed97ce62578a3179bcb99809ce93a1649e025ffe7 registry.k8s.io/sig-storage/csi-resizer:v1.6.0],SizeBytes:22678632,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:291334908ddf71a4661fd7f6d9d97274de8a5378a2b6fdfeb2ce73414a34f82f registry.k8s.io/sig-storage/csi-snapshotter:v6.1.0],SizeBytes:22434180,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:9a685020911e2725ad019dbce6e4a5ab93d51e3d4557f115e64343345e05781b registry.k8s.io/sig-storage/csi-attacher:v4.0.0],SizeBytes:22394501,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:122bfb8c1edabb3c0edd63f06523e6940d958d19b3957dc7b1d6f81e9f1f6119 registry.k8s.io/sig-storage/csi-provisioner:v3.1.0],SizeBytes:21735644,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:9ebbf9f023e7b41ccee3d52afe39a89e3ddacdbb69269d583abfc25847cfd9e4 registry.k8s.io/sig-storage/csi-resizer:v1.4.0],SizeBytes:20848413,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:8b9c313c05f54fb04f8d430896f5f5904b6cb157df261501b29adc04d2b2dc7b registry.k8s.io/sig-storage/csi-attacher:v3.4.0],SizeBytes:20565480,},ContainerImage{Names:[registry.k8s.io/cpa/cluster-proportional-autoscaler@sha256:68d396900aeaa072c1f27289485fdac29834045a6f3ffe369bf389d830ef572d registry.k8s.io/cpa/cluster-proportional-autoscaler:1.8.6],SizeBytes:19007623,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:92257881c1d6493cf18299a24af42330f891166560047902b8d431fb66b01af5 registry.k8s.io/sig-storage/hostpathplugin:v1.9.0],SizeBytes:18267611,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:9b2d9abda017c28b12f84a344f57cd73fbdb6c2bd7dd5bdd5018246ad1093ba6 registry.k8s.io/sig-storage/hostpathplugin:v1.11.0],SizeBytes:18159085,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:017727efcfeb7d053af68e51436ce8e65edbc6ca573720afb4f79c8594036955 registry.k8s.io/coredns/coredns:v1.10.0],SizeBytes:13766234,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:8502345,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:933940f13b3ea0abc62e656c1aa5c5b47c04b15d71250413a6b821bd0c58b94e registry.k8s.io/sig-storage/livenessprobe:v2.7.0],SizeBytes:8223819,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:7685305,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:5c99cf6a02adda929b10321dbf4ecfa00d87be9ba4fb456006237d530ab4baa1 registry.k8s.io/e2e-test-images/nginx:1.14-4],SizeBytes:6860089,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:2e0f836850e09b8b7cc937681d6194537a09fbd5f6b9e08f4d646a85128e8937 registry.k8s.io/e2e-test-images/busybox:1.29-4],SizeBytes:772933,},ContainerImage{Names:[registry.k8s.io/pause@sha256:7031c1b283388d2c2e09b57badb803c05ebed362dc88d84b480cc47f72a21097 registry.k8s.io/pause:3.9],SizeBytes:268051,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:253553,},},VolumesInUse:[kubernetes.io/csi/csi-hostpath-read-write-once-pod-5175^bc2b4edf-bb75-11ed-8d33-a27fee729fe2 kubernetes.io/csi/ebs.csi.aws.com^vol-0e020446b1cd6abaf],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/csi-hostpath-read-write-once-pod-5175^bc2b4edf-bb75-11ed-8d33-a27fee729fe2,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-0e020446b1cd6abaf,DevicePath:,},},Config:nil,},} Mar 5 16:51:41.850: INFO: Logging kubelet events for node i-056ad61525499081e Mar 5 16:51:41.966: INFO: ... skipping 50 lines ... Logging node info for node i-06df568f6e5c59fff Mar 5 16:51:42.664: INFO: Node Info: &Node{ObjectMeta:{i-06df568f6e5c59fff cb63c681-2e18-474a-b6f3-221325ac0d52 18792 0 2023-03-05 16:24:16 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:arm64 beta.kubernetes.io/instance-type:t4g.small beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:eu-central-1 failure-domain.beta.kubernetes.io/zone:eu-central-1a karpenter.k8s.aws/instance-category:t karpenter.k8s.aws/instance-cpu:2 karpenter.k8s.aws/instance-family:t4g karpenter.k8s.aws/instance-generation:4 karpenter.k8s.aws/instance-hypervisor:nitro karpenter.k8s.aws/instance-memory:2048 karpenter.k8s.aws/instance-pods:11 karpenter.k8s.aws/instance-size:small karpenter.sh/capacity-type:spot karpenter.sh/initialized:true karpenter.sh/provisioner-name:nodes kubernetes.io/arch:arm64 kubernetes.io/hostname:i-06df568f6e5c59fff kubernetes.io/os:linux node-role.kubernetes.io/node: node-role.kubernetes.io/spot-worker:true node.kubernetes.io/instance-type:t4g.small topology.ebs.csi.aws.com/zone:eu-central-1a topology.hostpath.csi/node:i-06df568f6e5c59fff topology.kubernetes.io/region:eu-central-1 topology.kubernetes.io/zone:eu-central-1a] map[csi.volume.kubernetes.io/nodeid:{"csi-hostpath-ephemeral-2147":"i-06df568f6e5c59fff","csi-hostpath-ephemeral-5408":"i-06df568f6e5c59fff","csi-hostpath-ephemeral-6540":"i-06df568f6e5c59fff","csi-mock-csi-mock-volumes-fsgroup-policy-1058":"i-06df568f6e5c59fff","csi-mock-csi-mock-volumes-workload-2219":"i-06df568f6e5c59fff","ebs.csi.aws.com":"i-06df568f6e5c59fff"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [{karpenter.sh/v1alpha5 Provisioner nodes ac5d7599-b2a2-4eae-8da4-1f4c0a54ec82 <nil> 0xc002f7b9d0}] [karpenter.sh/termination] [{aws-cloud-controller-manager Update v1 2023-03-05 16:24:16 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{}}}} } {kops-controller Update v1 2023-03-05 16:24:16 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{},"f:node-role.kubernetes.io/spot-worker":{}}}} } {karpenter Update v1 2023-03-05 16:25:57 +0000 UTC FieldsV1 {"f:metadata":{"f:finalizers":{".":{},"v:\"karpenter.sh/termination\"":{}},"f:labels":{".":{},"f:karpenter.k8s.aws/instance-category":{},"f:karpenter.k8s.aws/instance-cpu":{},"f:karpenter.k8s.aws/instance-family":{},"f:karpenter.k8s.aws/instance-generation":{},"f:karpenter.k8s.aws/instance-hypervisor":{},"f:karpenter.k8s.aws/instance-memory":{},"f:karpenter.k8s.aws/instance-pods":{},"f:karpenter.k8s.aws/instance-size":{},"f:karpenter.sh/capacity-type":{},"f:karpenter.sh/initialized":{},"f:karpenter.sh/provisioner-name":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}},"f:ownerReferences":{".":{},"k:{\"uid\":\"ac5d7599-b2a2-4eae-8da4-1f4c0a54ec82\"}":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-03-05 16:26:48 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{".":{},"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"InternalIP\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kubelet Update v1 2023-03-05 16:48:40 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{"f:kubernetes.io/hostname":{},"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{".":{},"f:cpu":{},"f:ephemeral-storage":{},"f:hugepages-1Gi":{},"f:hugepages-2Mi":{},"f:hugepages-32Mi":{},"f:hugepages-64Ki":{},"f:memory":{},"f:pods":{}},"f:capacity":{".":{},"f:cpu":{},"f:ephemeral-storage":{},"f:hugepages-1Gi":{},"f:hugepages-2Mi":{},"f:hugepages-32Mi":{},"f:hugepages-64Ki":{},"f:memory":{},"f:pods":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{}}},"f:daemonEndpoints":{"f:kubeletEndpoint":{"f:Port":{}}},"f:images":{},"f:nodeInfo":{"f:architecture":{},"f:bootID":{},"f:containerRuntimeVersion":{},"f:kernelVersion":{},"f:kubeProxyVersion":{},"f:kubeletVersion":{},"f:machineID":{},"f:operatingSystem":{},"f:osImage":{},"f:systemUUID":{}},"f:volumesInUse":{}}} status} {kube-controller-manager Update v1 2023-03-05 16:49:37 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{".":{},"k:{\"type\":\"DiskPressure\"}":{".":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"MemoryPressure\"}":{".":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"PIDPressure\"}":{".":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Ready\"}":{".":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}},"f:volumesAttached":{}}} status} {kube-controller-manager Update v1 2023-03-05 16:49:42 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:node.alpha.kubernetes.io/ttl":{}},"f:labels":{"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.2.0/24\"":{}},"f:taints":{}}} }]},Spec:NodeSpec{PodCIDR:100.96.2.0/24,DoNotUseExternalID:,ProviderID:aws:///eu-central-1a/i-06df568f6e5c59fff,Unschedulable:false,Taints:[]Taint{Taint{Key:node.kubernetes.io/unreachable,Value:,Effect:NoSchedule,TimeAdded:2023-03-05 16:49:37 +0000 UTC,},Taint{Key:node.kubernetes.io/unreachable,Value:,Effect:NoExecute,TimeAdded:2023-03-05 16:49:42 +0000 UTC,},},ConfigSource:nil,PodCIDRs:[100.96.2.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49756430336 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-32Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-64Ki: {{0 0} {<nil>} 0 DecimalSI},memory: {{1941540864 0} {<nil>} 1896036Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44780787229 0} {<nil>} 44780787229 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-32Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-64Ki: {{0 0} {<nil>} 0 DecimalSI},memory: {{1836683264 0} {<nil>} 1793636Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:Ready,Status:Unknown,LastHeartbeatTime:2023-03-05 16:47:34 +0000 UTC,LastTransitionTime:2023-03-05 16:49:37 +0000 UTC,Reason:NodeStatusUnknown,Message:Kubelet stopped posting node status.,},NodeCondition{Type:MemoryPressure,Status:Unknown,LastHeartbeatTime:2023-03-05 16:47:34 +0000 UTC,LastTransitionTime:2023-03-05 16:49:37 +0000 UTC,Reason:NodeStatusUnknown,Message:Kubelet stopped posting node status.,},NodeCondition{Type:DiskPressure,Status:Unknown,LastHeartbeatTime:2023-03-05 16:47:34 +0000 UTC,LastTransitionTime:2023-03-05 16:49:37 +0000 UTC,Reason:NodeStatusUnknown,Message:Kubelet stopped posting node status.,},NodeCondition{Type:PIDPressure,Status:Unknown,LastHeartbeatTime:2023-03-05 16:47:34 +0000 UTC,LastTransitionTime:2023-03-05 16:49:37 +0000 UTC,Reason:NodeStatusUnknown,Message:Kubelet stopped posting node status.,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.43.148,},NodeAddress{Type:ExternalIP,Address:18.192.211.222,},NodeAddress{Type:InternalDNS,Address:i-06df568f6e5c59fff.eu-central-1.compute.internal,},NodeAddress{Type:Hostname,Address:i-06df568f6e5c59fff.eu-central-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-18-192-211-222.eu-central-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec221221864d9409a8ec5f6614a9b94b,SystemUUID:ec221221-864d-9409-a8ec-5f6614a9b94b,BootID:553d31bf-faca-417e-a3db-325839e1b8e4,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 22.04.2 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.27.0-alpha.3.52+d48b8167f7094b,KubeProxyVersion:v1.27.0-alpha.3.52+d48b8167f7094b,OperatingSystem:linux,Architecture:arm64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:157636062,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/jessie-dnsutils@sha256:24aaf2626d6b27864c29de2097e8bbb840b3a414271bf7c8995e431e47d8408e registry.k8s.io/e2e-test-images/jessie-dnsutils:1.7],SizeBytes:116764683,},ContainerImage{Names:[registry.k8s.io/kube-proxy-arm64:v1.27.0-alpha.3.52_d48b8167f7094b],SizeBytes:60554082,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:16bbf38c463a4223d8cfe4da12bc61010b082a79b4bb003e2d3ba3ece5dd5f9e registry.k8s.io/e2e-test-images/agnhost:2.43],SizeBytes:49808364,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:3fe7acf013d1264ffded116b80a73dc129a449b0fccdb8d21af8279f2233f36e registry.k8s.io/e2e-test-images/httpd:2.4.39-4],SizeBytes:41528818,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:148b022f5c5da426fc2f3c14b5c0867e58ef05961510c84749ac1fddcb0fef22 registry.k8s.io/e2e-test-images/httpd:2.4.38-4],SizeBytes:40343601,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:28932904,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:ee3b525d5b89db99da3b8eb521d9cd90cb6e9ef0fbb651e98bb37be78d36b5b8 registry.k8s.io/sig-storage/csi-provisioner:v3.3.0],SizeBytes:23945474,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:425d8f1b769398127767b06ed97ce62578a3179bcb99809ce93a1649e025ffe7 registry.k8s.io/sig-storage/csi-resizer:v1.6.0],SizeBytes:22678632,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:291334908ddf71a4661fd7f6d9d97274de8a5378a2b6fdfeb2ce73414a34f82f registry.k8s.io/sig-storage/csi-snapshotter:v6.1.0],SizeBytes:22434180,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:9a685020911e2725ad019dbce6e4a5ab93d51e3d4557f115e64343345e05781b registry.k8s.io/sig-storage/csi-attacher:v4.0.0],SizeBytes:22394501,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:122bfb8c1edabb3c0edd63f06523e6940d958d19b3957dc7b1d6f81e9f1f6119 registry.k8s.io/sig-storage/csi-provisioner:v3.1.0],SizeBytes:21735644,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:9ebbf9f023e7b41ccee3d52afe39a89e3ddacdbb69269d583abfc25847cfd9e4 registry.k8s.io/sig-storage/csi-resizer:v1.4.0],SizeBytes:20848413,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:8b9c313c05f54fb04f8d430896f5f5904b6cb157df261501b29adc04d2b2dc7b registry.k8s.io/sig-storage/csi-attacher:v3.4.0],SizeBytes:20565480,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nonroot@sha256:ee9f50b3c64b174d296d91ca9f69a914ac30e59095dfb462b2b518ad28a63655 registry.k8s.io/e2e-test-images/nonroot:1.4],SizeBytes:18561972,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:92257881c1d6493cf18299a24af42330f891166560047902b8d431fb66b01af5 registry.k8s.io/sig-storage/hostpathplugin:v1.9.0],SizeBytes:18267611,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:9b2d9abda017c28b12f84a344f57cd73fbdb6c2bd7dd5bdd5018246ad1093ba6 registry.k8s.io/sig-storage/hostpathplugin:v1.11.0],SizeBytes:18159085,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:017727efcfeb7d053af68e51436ce8e65edbc6ca573720afb4f79c8594036955 registry.k8s.io/coredns/coredns:v1.10.0],SizeBytes:13766234,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:8502345,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:933940f13b3ea0abc62e656c1aa5c5b47c04b15d71250413a6b821bd0c58b94e registry.k8s.io/sig-storage/livenessprobe:v2.7.0],SizeBytes:8223819,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:7685305,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:5c99cf6a02adda929b10321dbf4ecfa00d87be9ba4fb456006237d530ab4baa1 registry.k8s.io/e2e-test-images/nginx:1.14-4],SizeBytes:6860089,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nonewprivs@sha256:8ac1264691820febacf3aea5d152cbde6d10685731ec14966a9401c6f47a68ac registry.k8s.io/e2e-test-images/nonewprivs:1.3],SizeBytes:5025027,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:2e0f836850e09b8b7cc937681d6194537a09fbd5f6b9e08f4d646a85128e8937 registry.k8s.io/e2e-test-images/busybox:1.29-4],SizeBytes:772933,},ContainerImage{Names:[registry.k8s.io/pause@sha256:7031c1b283388d2c2e09b57badb803c05ebed362dc88d84b480cc47f72a21097 registry.k8s.io/pause:3.9],SizeBytes:268051,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:253553,},},VolumesInUse:[kubernetes.io/csi/csi-hostpath-ephemeral-2147^52df5953-bb73-11ed-8aa7-be49170871b4 kubernetes.io/csi/csi-hostpath-ephemeral-5408^443d1daf-bb73-11ed-95c4-5a95d17cbc27 kubernetes.io/csi/csi-hostpath-ephemeral-5408^44411f51-bb73-11ed-95c4-5a95d17cbc27 kubernetes.io/csi/csi-hostpath-ephemeral-6540^4426a07f-bb73-11ed-a604-92687f2c50f8 kubernetes.io/csi/ebs.csi.aws.com^vol-05b67fab5beb6394b],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-05b67fab5beb6394b,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/csi-hostpath-ephemeral-6540^4426a07f-bb73-11ed-a604-92687f2c50f8,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/csi-hostpath-ephemeral-5408^44411f51-bb73-11ed-95c4-5a95d17cbc27,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/csi-hostpath-ephemeral-5408^443d1daf-bb73-11ed-95c4-5a95d17cbc27,DevicePath:,},},Config:nil,},} Mar 5 16:51:42.664: INFO: Logging kubelet events for node i-06df568f6e5c59fff Mar 5 16:51:42.780: INFO: Logging pods the kubelet thinks is on node i-06df568f6e5c59fff Mar 5 16:51:52.893: INFO: Unable to retrieve kubelet pods for node i-06df568f6e5c59fff: error trying to reach service: net/http: TLS handshake timeout Mar 5 16:51:52.893: INFO: Logging node info for node i-0d6a32f749839192f Mar 5 16:51:53.005: INFO: Node Info: &Node{ObjectMeta:{i-0d6a32f749839192f 4155fefb-4d3a-4b04-ba53-6ac9e895ff92 18509 0 2023-03-05 16:20:46 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:arm64 beta.kubernetes.io/instance-type:c6g.xlarge beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:eu-central-1 failure-domain.beta.kubernetes.io/zone:eu-central-1a kops.k8s.io/kops-controller-pki: kubernetes.io/arch:arm64 kubernetes.io/hostname:i-0d6a32f749839192f kubernetes.io/os:linux node-role.kubernetes.io/control-plane: node.kubernetes.io/exclude-from-external-load-balancers: node.kubernetes.io/instance-type:c6g.xlarge topology.ebs.csi.aws.com/zone:eu-central-1a topology.kubernetes.io/region:eu-central-1 topology.kubernetes.io/zone:eu-central-1a] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-0d6a32f749839192f"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kubelet Update v1 2023-03-05 16:20:46 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {protokube Update v1 2023-03-05 16:21:25 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kops.k8s.io/kops-controller-pki":{},"f:node-role.kubernetes.io/control-plane":{},"f:node.kubernetes.io/exclude-from-external-load-balancers":{}}}} } {kube-controller-manager Update v1 2023-03-05 16:21:47 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.0.0/24\"":{}}}} } {aws-cloud-controller-manager Update v1 2023-03-05 16:21:48 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{},"f:taints":{}}} } {aws-cloud-controller-manager Update v1 2023-03-05 16:21:48 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kubelet Update v1 2023-03-05 16:49:18 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.0.0/24,DoNotUseExternalID:,ProviderID:aws:///eu-central-1a/i-0d6a32f749839192f,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/control-plane,Value:,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[100.96.0.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{4 0} {<nil>} 4 DecimalSI},ephemeral-storage: {{49756430336 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-32Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-64Ki: {{0 0} {<nil>} 0 DecimalSI},memory: {{8150118400 0} {<nil>} 7959100Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{4 0} {<nil>} 4 DecimalSI},ephemeral-storage: {{44780787229 0} {<nil>} 44780787229 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-32Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-64Ki: {{0 0} {<nil>} 0 DecimalSI},memory: {{8045260800 0} {<nil>} 7856700Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-03-05 16:49:18 +0000 UTC,LastTransitionTime:2023-03-05 16:20:46 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-03-05 16:49:18 +0000 UTC,LastTransitionTime:2023-03-05 16:20:46 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-03-05 16:49:18 +0000 UTC,LastTransitionTime:2023-03-05 16:20:46 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-03-05 16:49:18 +0000 UTC,LastTransitionTime:2023-03-05 16:21:47 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.49.247,},NodeAddress{Type:ExternalIP,Address:18.197.20.82,},NodeAddress{Type:InternalDNS,Address:i-0d6a32f749839192f.eu-central-1.compute.internal,},NodeAddress{Type:Hostname,Address:i-0d6a32f749839192f.eu-central-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-18-197-20-82.eu-central-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2bed249e8ac7b517cb08ccd42b83e5,SystemUUID:ec2bed24-9e8a-c7b5-17cb-08ccd42b83e5,BootID:6f9215c9-dfe8-4b13-84f4-9b2084456198,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 22.04.2 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.27.0-alpha.3.52+d48b8167f7094b,KubeProxyVersion:v1.27.0-alpha.3.52+d48b8167f7094b,OperatingSystem:linux,Architecture:arm64,},Images:[]ContainerImage{ContainerImage{Names:[registry.k8s.io/etcd@sha256:9ce33ba33d8e738a5b85ed50b5080ac746deceed4a7496c550927a7a19ca3b6d registry.k8s.io/etcd:3.5.0-0],SizeBytes:157797353,},ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:157636062,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:4afb99b4690b418ffc2ceb67e1a17376457e441c1f09ab55447f0aaf992fa646 registry.k8s.io/etcd:3.4.3-0],SizeBytes:151677080,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:4ad90a11b55313b182afc186b9876c8e891531b8db4c9bf1541953021618d0e2 registry.k8s.io/etcd:3.4.13-0],SizeBytes:134531559,},ContainerImage{Names:[registry.k8s.io/kube-apiserver-arm64:v1.27.0-alpha.3.52_d48b8167f7094b],SizeBytes:131087707,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:0d2ab6b15b4fe0b903b9e81192d942fb39c29be61fe341cd16db5d0165b0435c registry.k8s.io/etcd:3.3.17-0],SizeBytes:126979107,},ContainerImage{Names:[registry.k8s.io/kube-controller-manager-arm64:v1.27.0-alpha.3.52_d48b8167f7094b],SizeBytes:120667128,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:17da501f5d2a675be46040422a27b7cc21b8a43895ac998b171db1c346f361f7 registry.k8s.io/etcd:3.3.10-0],SizeBytes:81436097,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:6f72b851544986cb0921b53ea655ec04c36131248f16d4ad110cb3ca0c369dc1 registry.k8s.io/etcd:3.5.4-0],SizeBytes:81120118,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:13f53ed1d91e2e11aac476ee9a0269fdda6cc4874eba903efd40daf50c55eee5 registry.k8s.io/etcd:3.5.3-0],SizeBytes:81114891,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:51eae8381dcb1078289fa7b4f3df2630cdc18d09fb56f8e56b41c40e191d6c83 registry.k8s.io/etcd:3.5.7-0],SizeBytes:80665728,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:dd75ec974b0a2a6f6bb47001ba09207976e625db898d1b16735528c009cb171c registry.k8s.io/etcd:3.5.6-0],SizeBytes:80539316,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:905d7ca17fd02bc24c0eba9a062753aba15db3e31422390bc3238eb762339b20 registry.k8s.io/etcd:3.2.24-1],SizeBytes:61500433,},ContainerImage{Names:[registry.k8s.io/etcdadm/etcd-manager-slim@sha256:8727adc059334e0e3329dbccce116a734da4a090da5e86c8cddccaf333caa666 registry.k8s.io/etcdadm/etcd-manager-slim:v3.0.20230201],SizeBytes:60826655,},ContainerImage{Names:[registry.k8s.io/kube-proxy-arm64:v1.27.0-alpha.3.52_d48b8167f7094b],SizeBytes:60554082,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:64b9ea357325d5db9f8a723dcf503b5a449177b17ac87d69481e126bb724c263 registry.k8s.io/etcd:3.5.1-0],SizeBytes:59553904,},ContainerImage{Names:[registry.k8s.io/kube-scheduler-arm64:v1.27.0-alpha.3.52_d48b8167f7094b],SizeBytes:56310763,},ContainerImage{Names:[registry.k8s.io/kops/kops-controller:1.27.0-alpha.1],SizeBytes:41797411,},ContainerImage{Names:[registry.k8s.io/kops/dns-controller:1.27.0-alpha.1],SizeBytes:40267473,},ContainerImage{Names:[public.ecr.aws/karpenter/controller@sha256:68db4f092cf9cc83f5ef9e2fbc5407c2cb682e81f64dfaa700a7602ede38b1cf public.ecr.aws/karpenter/controller:v0.16.3],SizeBytes:33707751,},ContainerImage{Names:[public.ecr.aws/karpenter/webhook@sha256:96a2d9b06d6bc5127801f358f74b1cf2d289b423a2e9ba40c573c0b14b17dafa public.ecr.aws/karpenter/webhook:v0.16.3],SizeBytes:33307187,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:28932904,},ContainerImage{Names:[quay.io/cilium/operator@sha256:a6d24a006a6b92967ac90786b49bc1ac26e5477cf028cd1186efcfc2466484db quay.io/cilium/operator:v1.12.5],SizeBytes:24567287,},ContainerImage{Names:[registry.k8s.io/provider-aws/cloud-controller-manager@sha256:fdeb61e3e42ecd9cca868d550ebdb88dd6341d9e91fcfa9a37e227dab2ad22cb registry.k8s.io/provider-aws/cloud-controller-manager:v1.26.0],SizeBytes:18345226,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:8502345,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:7685305,},ContainerImage{Names:[registry.k8s.io/kops/kube-apiserver-healthcheck:1.27.0-alpha.1],SizeBytes:4713612,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:253553,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Mar 5 16:51:53.005: INFO: Logging kubelet events for node i-0d6a32f749839192f Mar 5 16:51:53.120: INFO: ... skipping 50 lines ... Mar 5 16:51:53.240: INFO: Container webhook ready: true, restart count 0 Mar 5 16:51:53.240: INFO: kube-scheduler-i-0d6a32f749839192f started at 2023-03-05 16:18:47 +0000 UTC (0+1 container statuses recorded) Mar 5 16:51:53.240: INFO: Container kube-scheduler ready: true, restart count 0 Mar 5 16:51:53.728: INFO: Latency metrics for node i-0d6a32f749839192f [1mSTEP:[0m Waiting for namespaces [volume-2381-9266] to vanish [38;5;243m@ 03/05/23 16:51:53.842[0m Mar 5 16:56:54.069: INFO: error deleting namespace volume-2381-9266: timed out waiting for the condition [1mSTEP:[0m dump namespace information after failure [38;5;243m@ 03/05/23 16:56:58.456[0m [1mSTEP:[0m Collecting events from namespace "volume-2381-9266". [38;5;243m@ 03/05/23 16:56:58.456[0m [1mSTEP:[0m Found 0 events. [38;5;243m@ 03/05/23 16:56:58.568[0m Mar 5 16:56:58.680: INFO: POD NODE PHASE GRACE CONDITIONS Mar 5 16:56:58.680: INFO: csi-hostpathplugin-0 i-06df568f6e5c59fff Pending 30s [{PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2023-03-05 16:42:03 +0000 UTC } {DisruptionTarget True 0001-01-01 00:00:00 +0000 UTC 2023-03-05 16:47:57 +0000 UTC DeletionByTaintManager Taint manager: deleting due to NoExecute taint}] Mar 5 16:56:58.680: INFO: Mar 5 16:57:08.794: INFO: Unable to fetch volume-2381-9266/csi-hostpathplugin-0/hostpath logs: an error on the server ("unknown") has prevented the request from succeeding (get pods csi-hostpathplugin-0) Mar 5 16:57:18.908: INFO: Unable to fetch volume-2381-9266/csi-hostpathplugin-0/node-driver-registrar logs: an error on the server ("unknown") has prevented the request from succeeding (get pods csi-hostpathplugin-0) Mar 5 16:57:29.021: INFO: Unable to fetch volume-2381-9266/csi-hostpathplugin-0/liveness-probe logs: an error on the server ("unknown") has prevented the request from succeeding (get pods csi-hostpathplugin-0) Mar 5 16:57:39.135: INFO: Unable to fetch volume-2381-9266/csi-hostpathplugin-0/csi-attacher logs: an error on the server ("unknown") has prevented the request from succeeding (get pods csi-hostpathplugin-0) Mar 5 16:57:49.249: INFO: Unable to fetch volume-2381-9266/csi-hostpathplugin-0/csi-provisioner logs: an error on the server ("unknown") has prevented the request from succeeding (get pods csi-hostpathplugin-0) Mar 5 16:57:59.363: INFO: Unable to fetch volume-2381-9266/csi-hostpathplugin-0/csi-resizer logs: an error on the server ("unknown") has prevented the request from succeeding (get pods csi-hostpathplugin-0) Mar 5 16:58:09.477: INFO: Unable to fetch volume-2381-9266/csi-hostpathplugin-0/csi-snapshotter logs: an error on the server ("unknown") has prevented the request from succeeding (get pods csi-hostpathplugin-0) Mar 5 16:58:09.589: INFO: Logging node info for node i-056ad61525499081e Mar 5 16:58:09.701: INFO: Node Info: &Node{ObjectMeta:{i-056ad61525499081e a6ce9ed3-5ae8-4065-b117-6884e24b3632 23247 0 2023-03-05 16:24:16 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:arm64 beta.kubernetes.io/instance-type:t4g.small beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:eu-central-1 failure-domain.beta.kubernetes.io/zone:eu-central-1a karpenter.k8s.aws/instance-category:t karpenter.k8s.aws/instance-cpu:2 karpenter.k8s.aws/instance-family:t4g karpenter.k8s.aws/instance-generation:4 karpenter.k8s.aws/instance-hypervisor:nitro karpenter.k8s.aws/instance-memory:2048 karpenter.k8s.aws/instance-pods:11 karpenter.k8s.aws/instance-size:small karpenter.sh/capacity-type:spot karpenter.sh/initialized:true karpenter.sh/provisioner-name:nodes kubernetes.io/arch:arm64 kubernetes.io/hostname:i-056ad61525499081e kubernetes.io/os:linux node-role.kubernetes.io/node: node-role.kubernetes.io/spot-worker:true node.kubernetes.io/instance-type:t4g.small topology.ebs.csi.aws.com/zone:eu-central-1a topology.hostpath.csi/node:i-056ad61525499081e topology.kubernetes.io/region:eu-central-1 topology.kubernetes.io/zone:eu-central-1a] map[csi.volume.kubernetes.io/nodeid:{"csi-hostpath-provisioning-9504":"i-056ad61525499081e","csi-mock-csi-mock-volumes-workload-1975":"i-056ad61525499081e","ebs.csi.aws.com":"i-056ad61525499081e"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [{karpenter.sh/v1alpha5 Provisioner nodes ac5d7599-b2a2-4eae-8da4-1f4c0a54ec82 <nil> 0xc004258410}] [karpenter.sh/termination] [{aws-cloud-controller-manager Update v1 2023-03-05 16:24:16 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{}}}} } {kops-controller Update v1 2023-03-05 16:24:16 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{},"f:node-role.kubernetes.io/spot-worker":{}}}} } {karpenter Update v1 2023-03-05 16:25:57 +0000 UTC FieldsV1 {"f:metadata":{"f:finalizers":{".":{},"v:\"karpenter.sh/termination\"":{}},"f:labels":{".":{},"f:karpenter.k8s.aws/instance-category":{},"f:karpenter.k8s.aws/instance-cpu":{},"f:karpenter.k8s.aws/instance-family":{},"f:karpenter.k8s.aws/instance-generation":{},"f:karpenter.k8s.aws/instance-hypervisor":{},"f:karpenter.k8s.aws/instance-memory":{},"f:karpenter.k8s.aws/instance-pods":{},"f:karpenter.k8s.aws/instance-size":{},"f:karpenter.sh/capacity-type":{},"f:karpenter.sh/initialized":{},"f:karpenter.sh/provisioner-name":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}},"f:ownerReferences":{".":{},"k:{\"uid\":\"ac5d7599-b2a2-4eae-8da4-1f4c0a54ec82\"}":{}}},"f:spec":{"f:providerID":{}}} } {kube-controller-manager Update v1 2023-03-05 16:25:57 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:node.alpha.kubernetes.io/ttl":{}},"f:labels":{"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.1.0/24\"":{}}}} } {aws-cloud-controller-manager Update v1 2023-03-05 16:26:48 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{".":{},"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"InternalIP\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kube-controller-manager Update v1 2023-03-05 16:57:37 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{".":{},"k:{\"type\":\"DiskPressure\"}":{".":{},"f:type":{}},"k:{\"type\":\"MemoryPressure\"}":{".":{},"f:type":{}},"k:{\"type\":\"PIDPressure\"}":{".":{},"f:type":{}},"k:{\"type\":\"Ready\"}":{".":{},"f:type":{}}},"f:volumesAttached":{}}} status} {kubelet Update v1 2023-03-05 16:57:42 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{"f:kubernetes.io/hostname":{},"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{".":{},"f:cpu":{},"f:ephemeral-storage":{},"f:hugepages-1Gi":{},"f:hugepages-2Mi":{},"f:hugepages-32Mi":{},"f:hugepages-64Ki":{},"f:memory":{},"f:pods":{}},"f:capacity":{".":{},"f:cpu":{},"f:ephemeral-storage":{},"f:hugepages-1Gi":{},"f:hugepages-2Mi":{},"f:hugepages-32Mi":{},"f:hugepages-64Ki":{},"f:memory":{},"f:pods":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:daemonEndpoints":{"f:kubeletEndpoint":{"f:Port":{}}},"f:images":{},"f:nodeInfo":{"f:architecture":{},"f:bootID":{},"f:containerRuntimeVersion":{},"f:kernelVersion":{},"f:kubeProxyVersion":{},"f:kubeletVersion":{},"f:machineID":{},"f:operatingSystem":{},"f:osImage":{},"f:systemUUID":{}},"f:volumesInUse":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.1.0/24,DoNotUseExternalID:,ProviderID:aws:///eu-central-1a/i-056ad61525499081e,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.1.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49756430336 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-32Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-64Ki: {{0 0} {<nil>} 0 DecimalSI},memory: {{1941540864 0} {<nil>} 1896036Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44780787229 0} {<nil>} 44780787229 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-32Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-64Ki: {{0 0} {<nil>} 0 DecimalSI},memory: {{1836683264 0} {<nil>} 1793636Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-03-05 16:57:37 +0000 UTC,LastTransitionTime:2023-03-05 16:25:57 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-03-05 16:57:37 +0000 UTC,LastTransitionTime:2023-03-05 16:25:48 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-03-05 16:57:37 +0000 UTC,LastTransitionTime:2023-03-05 16:25:48 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-03-05 16:57:37 +0000 UTC,LastTransitionTime:2023-03-05 16:25:48 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.36.179,},NodeAddress{Type:ExternalIP,Address:18.159.37.200,},NodeAddress{Type:InternalDNS,Address:i-056ad61525499081e.eu-central-1.compute.internal,},NodeAddress{Type:Hostname,Address:i-056ad61525499081e.eu-central-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-18-159-37-200.eu-central-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2af3207d1c8c09785c337aae6c6ef0,SystemUUID:ec2af320-7d1c-8c09-785c-337aae6c6ef0,BootID:786f2752-bc30-4729-8aaf-45977b16c253,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 22.04.2 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.27.0-alpha.3.52+d48b8167f7094b,KubeProxyVersion:v1.27.0-alpha.3.52+d48b8167f7094b,OperatingSystem:linux,Architecture:arm64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:157636062,},ContainerImage{Names:[registry.k8s.io/kube-proxy-arm64:v1.27.0-alpha.3.52_d48b8167f7094b],SizeBytes:60554082,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:16bbf38c463a4223d8cfe4da12bc61010b082a79b4bb003e2d3ba3ece5dd5f9e registry.k8s.io/e2e-test-images/agnhost:2.43],SizeBytes:49808364,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:3fe7acf013d1264ffded116b80a73dc129a449b0fccdb8d21af8279f2233f36e registry.k8s.io/e2e-test-images/httpd:2.4.39-4],SizeBytes:41528818,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:148b022f5c5da426fc2f3c14b5c0867e58ef05961510c84749ac1fddcb0fef22 registry.k8s.io/e2e-test-images/httpd:2.4.38-4],SizeBytes:40343601,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:28932904,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:ee3b525d5b89db99da3b8eb521d9cd90cb6e9ef0fbb651e98bb37be78d36b5b8 registry.k8s.io/sig-storage/csi-provisioner:v3.3.0],SizeBytes:23945474,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:425d8f1b769398127767b06ed97ce62578a3179bcb99809ce93a1649e025ffe7 registry.k8s.io/sig-storage/csi-resizer:v1.6.0],SizeBytes:22678632,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:291334908ddf71a4661fd7f6d9d97274de8a5378a2b6fdfeb2ce73414a34f82f registry.k8s.io/sig-storage/csi-snapshotter:v6.1.0],SizeBytes:22434180,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:9a685020911e2725ad019dbce6e4a5ab93d51e3d4557f115e64343345e05781b registry.k8s.io/sig-storage/csi-attacher:v4.0.0],SizeBytes:22394501,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:122bfb8c1edabb3c0edd63f06523e6940d958d19b3957dc7b1d6f81e9f1f6119 registry.k8s.io/sig-storage/csi-provisioner:v3.1.0],SizeBytes:21735644,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:9ebbf9f023e7b41ccee3d52afe39a89e3ddacdbb69269d583abfc25847cfd9e4 registry.k8s.io/sig-storage/csi-resizer:v1.4.0],SizeBytes:20848413,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:8b9c313c05f54fb04f8d430896f5f5904b6cb157df261501b29adc04d2b2dc7b registry.k8s.io/sig-storage/csi-attacher:v3.4.0],SizeBytes:20565480,},ContainerImage{Names:[registry.k8s.io/cpa/cluster-proportional-autoscaler@sha256:68d396900aeaa072c1f27289485fdac29834045a6f3ffe369bf389d830ef572d registry.k8s.io/cpa/cluster-proportional-autoscaler:1.8.6],SizeBytes:19007623,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:92257881c1d6493cf18299a24af42330f891166560047902b8d431fb66b01af5 registry.k8s.io/sig-storage/hostpathplugin:v1.9.0],SizeBytes:18267611,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:9b2d9abda017c28b12f84a344f57cd73fbdb6c2bd7dd5bdd5018246ad1093ba6 registry.k8s.io/sig-storage/hostpathplugin:v1.11.0],SizeBytes:18159085,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:017727efcfeb7d053af68e51436ce8e65edbc6ca573720afb4f79c8594036955 registry.k8s.io/coredns/coredns:v1.10.0],SizeBytes:13766234,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:8502345,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:933940f13b3ea0abc62e656c1aa5c5b47c04b15d71250413a6b821bd0c58b94e registry.k8s.io/sig-storage/livenessprobe:v2.7.0],SizeBytes:8223819,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:7685305,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:5c99cf6a02adda929b10321dbf4ecfa00d87be9ba4fb456006237d530ab4baa1 registry.k8s.io/e2e-test-images/nginx:1.14-4],SizeBytes:6860089,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf registry.k8s.io/e2e-test-images/busybox:1.29-2],SizeBytes:773367,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:2e0f836850e09b8b7cc937681d6194537a09fbd5f6b9e08f4d646a85128e8937 registry.k8s.io/e2e-test-images/busybox:1.29-4],SizeBytes:772933,},ContainerImage{Names:[registry.k8s.io/pause@sha256:7031c1b283388d2c2e09b57badb803c05ebed362dc88d84b480cc47f72a21097 registry.k8s.io/pause:3.9],SizeBytes:268051,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:253553,},},VolumesInUse:[kubernetes.io/csi/ebs.csi.aws.com^vol-0e020446b1cd6abaf],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-0e020446b1cd6abaf,DevicePath:,},},Config:nil,},} Mar 5 16:58:09.702: INFO: Logging kubelet events for node i-056ad61525499081e Mar 5 16:58:09.817: INFO: ... skipping 65 lines ... Logging node info for node i-06df568f6e5c59fff Mar 5 16:58:10.528: INFO: Node Info: &Node{ObjectMeta:{i-06df568f6e5c59fff cb63c681-2e18-474a-b6f3-221325ac0d52 18792 0 2023-03-05 16:24:16 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:arm64 beta.kubernetes.io/instance-type:t4g.small beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:eu-central-1 failure-domain.beta.kubernetes.io/zone:eu-central-1a karpenter.k8s.aws/instance-category:t karpenter.k8s.aws/instance-cpu:2 karpenter.k8s.aws/instance-family:t4g karpenter.k8s.aws/instance-generation:4 karpenter.k8s.aws/instance-hypervisor:nitro karpenter.k8s.aws/instance-memory:2048 karpenter.k8s.aws/instance-pods:11 karpenter.k8s.aws/instance-size:small karpenter.sh/capacity-type:spot karpenter.sh/initialized:true karpenter.sh/provisioner-name:nodes kubernetes.io/arch:arm64 kubernetes.io/hostname:i-06df568f6e5c59fff kubernetes.io/os:linux node-role.kubernetes.io/node: node-role.kubernetes.io/spot-worker:true node.kubernetes.io/instance-type:t4g.small topology.ebs.csi.aws.com/zone:eu-central-1a topology.hostpath.csi/node:i-06df568f6e5c59fff topology.kubernetes.io/region:eu-central-1 topology.kubernetes.io/zone:eu-central-1a] map[csi.volume.kubernetes.io/nodeid:{"csi-hostpath-ephemeral-2147":"i-06df568f6e5c59fff","csi-hostpath-ephemeral-5408":"i-06df568f6e5c59fff","csi-hostpath-ephemeral-6540":"i-06df568f6e5c59fff","csi-mock-csi-mock-volumes-fsgroup-policy-1058":"i-06df568f6e5c59fff","csi-mock-csi-mock-volumes-workload-2219":"i-06df568f6e5c59fff","ebs.csi.aws.com":"i-06df568f6e5c59fff"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [{karpenter.sh/v1alpha5 Provisioner nodes ac5d7599-b2a2-4eae-8da4-1f4c0a54ec82 <nil> 0xc0035f4350}] [karpenter.sh/termination] [{aws-cloud-controller-manager Update v1 2023-03-05 16:24:16 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{}}}} } {kops-controller Update v1 2023-03-05 16:24:16 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{},"f:node-role.kubernetes.io/spot-worker":{}}}} } {karpenter Update v1 2023-03-05 16:25:57 +0000 UTC FieldsV1 {"f:metadata":{"f:finalizers":{".":{},"v:\"karpenter.sh/termination\"":{}},"f:labels":{".":{},"f:karpenter.k8s.aws/instance-category":{},"f:karpenter.k8s.aws/instance-cpu":{},"f:karpenter.k8s.aws/instance-family":{},"f:karpenter.k8s.aws/instance-generation":{},"f:karpenter.k8s.aws/instance-hypervisor":{},"f:karpenter.k8s.aws/instance-memory":{},"f:karpenter.k8s.aws/instance-pods":{},"f:karpenter.k8s.aws/instance-size":{},"f:karpenter.sh/capacity-type":{},"f:karpenter.sh/initialized":{},"f:karpenter.sh/provisioner-name":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}},"f:ownerReferences":{".":{},"k:{\"uid\":\"ac5d7599-b2a2-4eae-8da4-1f4c0a54ec82\"}":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-03-05 16:26:48 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{".":{},"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"InternalIP\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kubelet Update v1 2023-03-05 16:48:40 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{"f:kubernetes.io/hostname":{},"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{".":{},"f:cpu":{},"f:ephemeral-storage":{},"f:hugepages-1Gi":{},"f:hugepages-2Mi":{},"f:hugepages-32Mi":{},"f:hugepages-64Ki":{},"f:memory":{},"f:pods":{}},"f:capacity":{".":{},"f:cpu":{},"f:ephemeral-storage":{},"f:hugepages-1Gi":{},"f:hugepages-2Mi":{},"f:hugepages-32Mi":{},"f:hugepages-64Ki":{},"f:memory":{},"f:pods":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{}}},"f:daemonEndpoints":{"f:kubeletEndpoint":{"f:Port":{}}},"f:images":{},"f:nodeInfo":{"f:architecture":{},"f:bootID":{},"f:containerRuntimeVersion":{},"f:kernelVersion":{},"f:kubeProxyVersion":{},"f:kubeletVersion":{},"f:machineID":{},"f:operatingSystem":{},"f:osImage":{},"f:systemUUID":{}},"f:volumesInUse":{}}} status} {kube-controller-manager Update v1 2023-03-05 16:49:37 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{".":{},"k:{\"type\":\"DiskPressure\"}":{".":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"MemoryPressure\"}":{".":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"PIDPressure\"}":{".":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Ready\"}":{".":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}},"f:volumesAttached":{}}} status} {kube-controller-manager Update v1 2023-03-05 16:49:42 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:node.alpha.kubernetes.io/ttl":{}},"f:labels":{"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.2.0/24\"":{}},"f:taints":{}}} }]},Spec:NodeSpec{PodCIDR:100.96.2.0/24,DoNotUseExternalID:,ProviderID:aws:///eu-central-1a/i-06df568f6e5c59fff,Unschedulable:false,Taints:[]Taint{Taint{Key:node.kubernetes.io/unreachable,Value:,Effect:NoSchedule,TimeAdded:2023-03-05 16:49:37 +0000 UTC,},Taint{Key:node.kubernetes.io/unreachable,Value:,Effect:NoExecute,TimeAdded:2023-03-05 16:49:42 +0000 UTC,},},ConfigSource:nil,PodCIDRs:[100.96.2.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49756430336 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-32Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-64Ki: {{0 0} {<nil>} 0 DecimalSI},memory: {{1941540864 0} {<nil>} 1896036Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44780787229 0} {<nil>} 44780787229 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-32Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-64Ki: {{0 0} {<nil>} 0 DecimalSI},memory: {{1836683264 0} {<nil>} 1793636Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:Ready,Status:Unknown,LastHeartbeatTime:2023-03-05 16:47:34 +0000 UTC,LastTransitionTime:2023-03-05 16:49:37 +0000 UTC,Reason:NodeStatusUnknown,Message:Kubelet stopped posting node status.,},NodeCondition{Type:MemoryPressure,Status:Unknown,LastHeartbeatTime:2023-03-05 16:47:34 +0000 UTC,LastTransitionTime:2023-03-05 16:49:37 +0000 UTC,Reason:NodeStatusUnknown,Message:Kubelet stopped posting node status.,},NodeCondition{Type:DiskPressure,Status:Unknown,LastHeartbeatTime:2023-03-05 16:47:34 +0000 UTC,LastTransitionTime:2023-03-05 16:49:37 +0000 UTC,Reason:NodeStatusUnknown,Message:Kubelet stopped posting node status.,},NodeCondition{Type:PIDPressure,Status:Unknown,LastHeartbeatTime:2023-03-05 16:47:34 +0000 UTC,LastTransitionTime:2023-03-05 16:49:37 +0000 UTC,Reason:NodeStatusUnknown,Message:Kubelet stopped posting node status.,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.43.148,},NodeAddress{Type:ExternalIP,Address:18.192.211.222,},NodeAddress{Type:InternalDNS,Address:i-06df568f6e5c59fff.eu-central-1.compute.internal,},NodeAddress{Type:Hostname,Address:i-06df568f6e5c59fff.eu-central-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-18-192-211-222.eu-central-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec221221864d9409a8ec5f6614a9b94b,SystemUUID:ec221221-864d-9409-a8ec-5f6614a9b94b,BootID:553d31bf-faca-417e-a3db-325839e1b8e4,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 22.04.2 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.27.0-alpha.3.52+d48b8167f7094b,KubeProxyVersion:v1.27.0-alpha.3.52+d48b8167f7094b,OperatingSystem:linux,Architecture:arm64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:157636062,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/jessie-dnsutils@sha256:24aaf2626d6b27864c29de2097e8bbb840b3a414271bf7c8995e431e47d8408e registry.k8s.io/e2e-test-images/jessie-dnsutils:1.7],SizeBytes:116764683,},ContainerImage{Names:[registry.k8s.io/kube-proxy-arm64:v1.27.0-alpha.3.52_d48b8167f7094b],SizeBytes:60554082,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:16bbf38c463a4223d8cfe4da12bc61010b082a79b4bb003e2d3ba3ece5dd5f9e registry.k8s.io/e2e-test-images/agnhost:2.43],SizeBytes:49808364,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:3fe7acf013d1264ffded116b80a73dc129a449b0fccdb8d21af8279f2233f36e registry.k8s.io/e2e-test-images/httpd:2.4.39-4],SizeBytes:41528818,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:148b022f5c5da426fc2f3c14b5c0867e58ef05961510c84749ac1fddcb0fef22 registry.k8s.io/e2e-test-images/httpd:2.4.38-4],SizeBytes:40343601,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:28932904,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:ee3b525d5b89db99da3b8eb521d9cd90cb6e9ef0fbb651e98bb37be78d36b5b8 registry.k8s.io/sig-storage/csi-provisioner:v3.3.0],SizeBytes:23945474,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:425d8f1b769398127767b06ed97ce62578a3179bcb99809ce93a1649e025ffe7 registry.k8s.io/sig-storage/csi-resizer:v1.6.0],SizeBytes:22678632,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:291334908ddf71a4661fd7f6d9d97274de8a5378a2b6fdfeb2ce73414a34f82f registry.k8s.io/sig-storage/csi-snapshotter:v6.1.0],SizeBytes:22434180,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:9a685020911e2725ad019dbce6e4a5ab93d51e3d4557f115e64343345e05781b registry.k8s.io/sig-storage/csi-attacher:v4.0.0],SizeBytes:22394501,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:122bfb8c1edabb3c0edd63f06523e6940d958d19b3957dc7b1d6f81e9f1f6119 registry.k8s.io/sig-storage/csi-provisioner:v3.1.0],SizeBytes:21735644,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:9ebbf9f023e7b41ccee3d52afe39a89e3ddacdbb69269d583abfc25847cfd9e4 registry.k8s.io/sig-storage/csi-resizer:v1.4.0],SizeBytes:20848413,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:8b9c313c05f54fb04f8d430896f5f5904b6cb157df261501b29adc04d2b2dc7b registry.k8s.io/sig-storage/csi-attacher:v3.4.0],SizeBytes:20565480,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nonroot@sha256:ee9f50b3c64b174d296d91ca9f69a914ac30e59095dfb462b2b518ad28a63655 registry.k8s.io/e2e-test-images/nonroot:1.4],SizeBytes:18561972,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:92257881c1d6493cf18299a24af42330f891166560047902b8d431fb66b01af5 registry.k8s.io/sig-storage/hostpathplugin:v1.9.0],SizeBytes:18267611,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:9b2d9abda017c28b12f84a344f57cd73fbdb6c2bd7dd5bdd5018246ad1093ba6 registry.k8s.io/sig-storage/hostpathplugin:v1.11.0],SizeBytes:18159085,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:017727efcfeb7d053af68e51436ce8e65edbc6ca573720afb4f79c8594036955 registry.k8s.io/coredns/coredns:v1.10.0],SizeBytes:13766234,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:8502345,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:933940f13b3ea0abc62e656c1aa5c5b47c04b15d71250413a6b821bd0c58b94e registry.k8s.io/sig-storage/livenessprobe:v2.7.0],SizeBytes:8223819,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:7685305,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:5c99cf6a02adda929b10321dbf4ecfa00d87be9ba4fb456006237d530ab4baa1 registry.k8s.io/e2e-test-images/nginx:1.14-4],SizeBytes:6860089,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nonewprivs@sha256:8ac1264691820febacf3aea5d152cbde6d10685731ec14966a9401c6f47a68ac registry.k8s.io/e2e-test-images/nonewprivs:1.3],SizeBytes:5025027,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:2e0f836850e09b8b7cc937681d6194537a09fbd5f6b9e08f4d646a85128e8937 registry.k8s.io/e2e-test-images/busybox:1.29-4],SizeBytes:772933,},ContainerImage{Names:[registry.k8s.io/pause@sha256:7031c1b283388d2c2e09b57badb803c05ebed362dc88d84b480cc47f72a21097 registry.k8s.io/pause:3.9],SizeBytes:268051,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:253553,},},VolumesInUse:[kubernetes.io/csi/csi-hostpath-ephemeral-2147^52df5953-bb73-11ed-8aa7-be49170871b4 kubernetes.io/csi/csi-hostpath-ephemeral-5408^443d1daf-bb73-11ed-95c4-5a95d17cbc27 kubernetes.io/csi/csi-hostpath-ephemeral-5408^44411f51-bb73-11ed-95c4-5a95d17cbc27 kubernetes.io/csi/csi-hostpath-ephemeral-6540^4426a07f-bb73-11ed-a604-92687f2c50f8 kubernetes.io/csi/ebs.csi.aws.com^vol-05b67fab5beb6394b],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-05b67fab5beb6394b,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/csi-hostpath-ephemeral-6540^4426a07f-bb73-11ed-a604-92687f2c50f8,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/csi-hostpath-ephemeral-5408^44411f51-bb73-11ed-95c4-5a95d17cbc27,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/csi-hostpath-ephemeral-5408^443d1daf-bb73-11ed-95c4-5a95d17cbc27,DevicePath:,},},Config:nil,},} Mar 5 16:58:10.529: INFO: Logging kubelet events for node i-06df568f6e5c59fff Mar 5 16:58:10.644: INFO: Logging pods the kubelet thinks is on node i-06df568f6e5c59fff Mar 5 16:58:20.757: INFO: Unable to retrieve kubelet pods for node i-06df568f6e5c59fff: error trying to reach service: net/http: TLS handshake timeout Mar 5 16:58:20.757: INFO: Logging node info for node i-0d6a32f749839192f Mar 5 16:58:20.869: INFO: Node Info: &Node{ObjectMeta:{i-0d6a32f749839192f 4155fefb-4d3a-4b04-ba53-6ac9e895ff92 21031 0 2023-03-05 16:20:46 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:arm64 beta.kubernetes.io/instance-type:c6g.xlarge beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:eu-central-1 failure-domain.beta.kubernetes.io/zone:eu-central-1a kops.k8s.io/kops-controller-pki: kubernetes.io/arch:arm64 kubernetes.io/hostname:i-0d6a32f749839192f kubernetes.io/os:linux node-role.kubernetes.io/control-plane: node.kubernetes.io/exclude-from-external-load-balancers: node.kubernetes.io/instance-type:c6g.xlarge topology.ebs.csi.aws.com/zone:eu-central-1a topology.kubernetes.io/region:eu-central-1 topology.kubernetes.io/zone:eu-central-1a] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-0d6a32f749839192f"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kubelet Update v1 2023-03-05 16:20:46 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {protokube Update v1 2023-03-05 16:21:25 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kops.k8s.io/kops-controller-pki":{},"f:node-role.kubernetes.io/control-plane":{},"f:node.kubernetes.io/exclude-from-external-load-balancers":{}}}} } {kube-controller-manager Update v1 2023-03-05 16:21:47 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.0.0/24\"":{}}}} } {aws-cloud-controller-manager Update v1 2023-03-05 16:21:48 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{},"f:taints":{}}} } {aws-cloud-controller-manager Update v1 2023-03-05 16:21:48 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kubelet Update v1 2023-03-05 16:54:23 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.0.0/24,DoNotUseExternalID:,ProviderID:aws:///eu-central-1a/i-0d6a32f749839192f,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/control-plane,Value:,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[100.96.0.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{4 0} {<nil>} 4 DecimalSI},ephemeral-storage: {{49756430336 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-32Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-64Ki: {{0 0} {<nil>} 0 DecimalSI},memory: {{8150118400 0} {<nil>} 7959100Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{4 0} {<nil>} 4 DecimalSI},ephemeral-storage: {{44780787229 0} {<nil>} 44780787229 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-32Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-64Ki: {{0 0} {<nil>} 0 DecimalSI},memory: {{8045260800 0} {<nil>} 7856700Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-03-05 16:54:23 +0000 UTC,LastTransitionTime:2023-03-05 16:20:46 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-03-05 16:54:23 +0000 UTC,LastTransitionTime:2023-03-05 16:20:46 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-03-05 16:54:23 +0000 UTC,LastTransitionTime:2023-03-05 16:20:46 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-03-05 16:54:23 +0000 UTC,LastTransitionTime:2023-03-05 16:21:47 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.49.247,},NodeAddress{Type:ExternalIP,Address:18.197.20.82,},NodeAddress{Type:InternalDNS,Address:i-0d6a32f749839192f.eu-central-1.compute.internal,},NodeAddress{Type:Hostname,Address:i-0d6a32f749839192f.eu-central-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-18-197-20-82.eu-central-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2bed249e8ac7b517cb08ccd42b83e5,SystemUUID:ec2bed24-9e8a-c7b5-17cb-08ccd42b83e5,BootID:6f9215c9-dfe8-4b13-84f4-9b2084456198,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 22.04.2 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.27.0-alpha.3.52+d48b8167f7094b,KubeProxyVersion:v1.27.0-alpha.3.52+d48b8167f7094b,OperatingSystem:linux,Architecture:arm64,},Images:[]ContainerImage{ContainerImage{Names:[registry.k8s.io/etcd@sha256:9ce33ba33d8e738a5b85ed50b5080ac746deceed4a7496c550927a7a19ca3b6d registry.k8s.io/etcd:3.5.0-0],SizeBytes:157797353,},ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:157636062,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:4afb99b4690b418ffc2ceb67e1a17376457e441c1f09ab55447f0aaf992fa646 registry.k8s.io/etcd:3.4.3-0],SizeBytes:151677080,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:4ad90a11b55313b182afc186b9876c8e891531b8db4c9bf1541953021618d0e2 registry.k8s.io/etcd:3.4.13-0],SizeBytes:134531559,},ContainerImage{Names:[registry.k8s.io/kube-apiserver-arm64:v1.27.0-alpha.3.52_d48b8167f7094b],SizeBytes:131087707,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:0d2ab6b15b4fe0b903b9e81192d942fb39c29be61fe341cd16db5d0165b0435c registry.k8s.io/etcd:3.3.17-0],SizeBytes:126979107,},ContainerImage{Names:[registry.k8s.io/kube-controller-manager-arm64:v1.27.0-alpha.3.52_d48b8167f7094b],SizeBytes:120667128,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:17da501f5d2a675be46040422a27b7cc21b8a43895ac998b171db1c346f361f7 registry.k8s.io/etcd:3.3.10-0],SizeBytes:81436097,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:6f72b851544986cb0921b53ea655ec04c36131248f16d4ad110cb3ca0c369dc1 registry.k8s.io/etcd:3.5.4-0],SizeBytes:81120118,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:13f53ed1d91e2e11aac476ee9a0269fdda6cc4874eba903efd40daf50c55eee5 registry.k8s.io/etcd:3.5.3-0],SizeBytes:81114891,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:51eae8381dcb1078289fa7b4f3df2630cdc18d09fb56f8e56b41c40e191d6c83 registry.k8s.io/etcd:3.5.7-0],SizeBytes:80665728,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:dd75ec974b0a2a6f6bb47001ba09207976e625db898d1b16735528c009cb171c registry.k8s.io/etcd:3.5.6-0],SizeBytes:80539316,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:905d7ca17fd02bc24c0eba9a062753aba15db3e31422390bc3238eb762339b20 registry.k8s.io/etcd:3.2.24-1],SizeBytes:61500433,},ContainerImage{Names:[registry.k8s.io/etcdadm/etcd-manager-slim@sha256:8727adc059334e0e3329dbccce116a734da4a090da5e86c8cddccaf333caa666 registry.k8s.io/etcdadm/etcd-manager-slim:v3.0.20230201],SizeBytes:60826655,},ContainerImage{Names:[registry.k8s.io/kube-proxy-arm64:v1.27.0-alpha.3.52_d48b8167f7094b],SizeBytes:60554082,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:64b9ea357325d5db9f8a723dcf503b5a449177b17ac87d69481e126bb724c263 registry.k8s.io/etcd:3.5.1-0],SizeBytes:59553904,},ContainerImage{Names:[registry.k8s.io/kube-scheduler-arm64:v1.27.0-alpha.3.52_d48b8167f7094b],SizeBytes:56310763,},ContainerImage{Names:[registry.k8s.io/kops/kops-controller:1.27.0-alpha.1],SizeBytes:41797411,},ContainerImage{Names:[registry.k8s.io/kops/dns-controller:1.27.0-alpha.1],SizeBytes:40267473,},ContainerImage{Names:[public.ecr.aws/karpenter/controller@sha256:68db4f092cf9cc83f5ef9e2fbc5407c2cb682e81f64dfaa700a7602ede38b1cf public.ecr.aws/karpenter/controller:v0.16.3],SizeBytes:33707751,},ContainerImage{Names:[public.ecr.aws/karpenter/webhook@sha256:96a2d9b06d6bc5127801f358f74b1cf2d289b423a2e9ba40c573c0b14b17dafa public.ecr.aws/karpenter/webhook:v0.16.3],SizeBytes:33307187,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:28932904,},ContainerImage{Names:[quay.io/cilium/operator@sha256:a6d24a006a6b92967ac90786b49bc1ac26e5477cf028cd1186efcfc2466484db quay.io/cilium/operator:v1.12.5],SizeBytes:24567287,},ContainerImage{Names:[registry.k8s.io/provider-aws/cloud-controller-manager@sha256:fdeb61e3e42ecd9cca868d550ebdb88dd6341d9e91fcfa9a37e227dab2ad22cb registry.k8s.io/provider-aws/cloud-controller-manager:v1.26.0],SizeBytes:18345226,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:8502345,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:7685305,},ContainerImage{Names:[registry.k8s.io/kops/kube-apiserver-healthcheck:1.27.0-alpha.1],SizeBytes:4713612,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:253553,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Mar 5 16:58:20.869: INFO: Logging kubelet events for node i-0d6a32f749839192f Mar 5 16:58:20.984: INFO: ... skipping 52 lines ... Mar 5 16:58:21.103: INFO: Container webhook ready: true, restart count 0 Mar 5 16:58:21.613: INFO: Latency metrics for node i-0d6a32f749839192f [1mSTEP:[0m Destroying namespace "volume-2381-9266" for this suite. [38;5;243m@ 03/05/23 16:58:21.613[0m [38;5;243m<< Timeline[0m [38;5;9m[FAILED] PersistentVolumeClaims [csi-hostpathrv5x2] not all in phase Bound within 5m0s[0m [38;5;9mIn [1m[It][0m[38;5;9m at: [1mtest/e2e/storage/framework/volume_resource.go:311[0m [38;5;243m@ 03/05/23 16:47:05.71[0m [38;5;243m------------------------------[0m [38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;10m•[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m [38;5;243m------------------------------[0m [38;5;9m• [FAILED] [1094.401 seconds][0m [0m[sig-network] DNS [38;5;9m[1m[It] should resolve DNS of partial qualified names for services [LinuxOnly] [Conformance][0m [38;5;243mtest/e2e/network/dns.go:191[0m [38;5;243mTimeline >>[0m [1mSTEP:[0m Creating a kubernetes client [38;5;243m@ 03/05/23 16:40:27.832[0m Mar 5 16:40:27.832: INFO: >>> kubeConfig: /root/.kube/config ... skipping 4 lines ... [1mSTEP:[0m Running these commands on wheezy: for i in `seq 1 600`; do check="$$(dig +notcp +noall +answer +search dns-test-service A)" && test -n "$$check" && echo OK > /results/wheezy_udp@dns-test-service;check="$$(dig +tcp +noall +answer +search dns-test-service A)" && test -n "$$check" && echo OK > /results/wheezy_tcp@dns-test-service;check="$$(dig +notcp +noall +answer +search dns-test-service.dns-8819 A)" && test -n "$$check" && echo OK > /results/wheezy_udp@dns-test-service.dns-8819;check="$$(dig +tcp +noall +answer +search dns-test-service.dns-8819 A)" && test -n "$$check" && echo OK > /results/wheezy_tcp@dns-test-service.dns-8819;check="$$(dig +notcp +noall +answer +search dns-test-service.dns-8819.svc A)" && test -n "$$check" && echo OK > /results/wheezy_udp@dns-test-service.dns-8819.svc;check="$$(dig +tcp +noall +answer +search dns-test-service.dns-8819.svc A)" && test -n "$$check" && echo OK > /results/wheezy_tcp@dns-test-service.dns-8819.svc;check="$$(dig +notcp +noall +answer +search _http._tcp.dns-test-service.dns-8819.svc SRV)" && test -n "$$check" && echo OK > /results/wheezy_udp@_http._tcp.dns-test-service.dns-8819.svc;check="$$(dig +tcp +noall +answer +search _http._tcp.dns-test-service.dns-8819.svc SRV)" && test -n "$$check" && echo OK > /results/wheezy_tcp@_http._tcp.dns-test-service.dns-8819.svc;check="$$(dig +notcp +noall +answer +search _http._tcp.test-service-2.dns-8819.svc SRV)" && test -n "$$check" && echo OK > /results/wheezy_udp@_http._tcp.test-service-2.dns-8819.svc;check="$$(dig +tcp +noall +answer +search _http._tcp.test-service-2.dns-8819.svc SRV)" && test -n "$$check" && echo OK > /results/wheezy_tcp@_http._tcp.test-service-2.dns-8819.svc;check="$$(dig +notcp +noall +answer +search 235.242.64.100.in-addr.arpa. PTR)" && test -n "$$check" && echo OK > /results/100.64.242.235_udp@PTR;check="$$(dig +tcp +noall +answer +search 235.242.64.100.in-addr.arpa. PTR)" && test -n "$$check" && echo OK > /results/100.64.242.235_tcp@PTR;sleep 1; done [38;5;243m@ 03/05/23 16:40:28.855[0m [1mSTEP:[0m Running these commands on jessie: for i in `seq 1 600`; do check="$$(dig +notcp +noall +answer +search dns-test-service A)" && test -n "$$check" && echo OK > /results/jessie_udp@dns-test-service;check="$$(dig +tcp +noall +answer +search dns-test-service A)" && test -n "$$check" && echo OK > /results/jessie_tcp@dns-test-service;check="$$(dig +notcp +noall +answer +search dns-test-service.dns-8819 A)" && test -n "$$check" && echo OK > /results/jessie_udp@dns-test-service.dns-8819;check="$$(dig +tcp +noall +answer +search dns-test-service.dns-8819 A)" && test -n "$$check" && echo OK > /results/jessie_tcp@dns-test-service.dns-8819;check="$$(dig +notcp +noall +answer +search dns-test-service.dns-8819.svc A)" && test -n "$$check" && echo OK > /results/jessie_udp@dns-test-service.dns-8819.svc;check="$$(dig +tcp +noall +answer +search dns-test-service.dns-8819.svc A)" && test -n "$$check" && echo OK > /results/jessie_tcp@dns-test-service.dns-8819.svc;check="$$(dig +notcp +noall +answer +search _http._tcp.dns-test-service.dns-8819.svc SRV)" && test -n "$$check" && echo OK > /results/jessie_udp@_http._tcp.dns-test-service.dns-8819.svc;check="$$(dig +tcp +noall +answer +search _http._tcp.dns-test-service.dns-8819.svc SRV)" && test -n "$$check" && echo OK > /results/jessie_tcp@_http._tcp.dns-test-service.dns-8819.svc;check="$$(dig +notcp +noall +answer +search _http._tcp.test-service-2.dns-8819.svc SRV)" && test -n "$$check" && echo OK > /results/jessie_udp@_http._tcp.test-service-2.dns-8819.svc;check="$$(dig +tcp +noall +answer +search _http._tcp.test-service-2.dns-8819.svc SRV)" && test -n "$$check" && echo OK > /results/jessie_tcp@_http._tcp.test-service-2.dns-8819.svc;check="$$(dig +notcp +noall +answer +search 235.242.64.100.in-addr.arpa. PTR)" && test -n "$$check" && echo OK > /results/100.64.242.235_udp@PTR;check="$$(dig +tcp +noall +answer +search 235.242.64.100.in-addr.arpa. PTR)" && test -n "$$check" && echo OK > /results/100.64.242.235_tcp@PTR;sleep 1; done [38;5;243m@ 03/05/23 16:40:28.855[0m [1mSTEP:[0m creating a pod to probe DNS [38;5;243m@ 03/05/23 16:40:28.855[0m [1mSTEP:[0m submitting the pod to kubernetes [38;5;243m@ 03/05/23 16:40:28.855[0m Mar 5 16:55:28.978: INFO: Failed inside E2E framework: k8s.io/kubernetes/test/e2e/framework/pod.WaitTimeoutForPodRunningInNamespace({0x7f2524084d10, 0xc002f491c0}, {0x8252218?, 0xc003505520?}, {0xc0038edec0, 0x2d}, {0xc003128100, 0x8}, 0x0?) test/e2e/framework/pod/wait.go:460 +0x1a4 k8s.io/kubernetes/test/e2e/framework/pod.WaitForPodRunningInNamespaceSlow(...) test/e2e/framework/pod/wait.go:451 k8s.io/kubernetes/test/e2e/network.validateDNSResults({0x7f2524084d10, 0xc002f491c0}, 0xc000efb1d0, 0xc003f61680, {0xc0038b1180, 0x18, 0x28}) test/e2e/network/dns_common.go:506 +0x335 k8s.io/kubernetes/test/e2e/network.glob..func2.6({0x7f2524084d10, 0xc002f491c0}) test/e2e/network/dns.go:235 +0xcee [38;5;9m[FAILED][0m in [It] - test/e2e/network/dns_common.go:506 [38;5;243m@ 03/05/23 16:55:28.978[0m Mar 5 16:55:28.979: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready Mar 5 16:55:29.094: INFO: Condition Ready of node i-06df568f6e5c59fff is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-05 16:49:37 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-05 16:49:42 +0000 UTC}]. Failure Mar 5 16:55:31.210: INFO: Condition Ready of node i-06df568f6e5c59fff is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-05 16:49:37 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-05 16:49:42 +0000 UTC}]. Failure Mar 5 16:55:33.210: INFO: Condition Ready of node i-06df568f6e5c59fff is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-05 16:49:37 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-05 16:49:42 +0000 UTC}]. Failure Mar 5 16:55:35.210: INFO: Condition Ready of node i-06df568f6e5c59fff is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-05 16:49:37 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-05 16:49:42 +0000 UTC}]. Failure Mar 5 16:55:37.209: INFO: Condition Ready of node i-06df568f6e5c59fff is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-05 16:49:37 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-05 16:49:42 +0000 UTC}]. Failure ... skipping 82 lines ... Mar 5 16:58:23.210: INFO: Condition Ready of node i-06df568f6e5c59fff is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-05 16:49:37 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-05 16:49:42 +0000 UTC}]. Failure Mar 5 16:58:25.209: INFO: Condition Ready of node i-06df568f6e5c59fff is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-05 16:49:37 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-05 16:49:42 +0000 UTC}]. Failure Mar 5 16:58:27.210: INFO: Condition Ready of node i-06df568f6e5c59fff is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-05 16:49:37 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-05 16:49:42 +0000 UTC}]. Failure Mar 5 16:58:29.210: INFO: Condition Ready of node i-06df568f6e5c59fff is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-05 16:49:37 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-05 16:49:42 +0000 UTC}]. Failure Mar 5 16:58:29.325: INFO: Condition Ready of node i-06df568f6e5c59fff is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-05 16:49:37 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-05 16:49:42 +0000 UTC}]. Failure [1mSTEP:[0m deleting the pod [38;5;243m@ 03/05/23 16:58:29.325[0m [38;5;9m[FAILED][0m in [DeferCleanup (Each)] - test/e2e/network/dns_common.go:498 [38;5;243m@ 03/05/23 16:58:29.44[0m [1mSTEP:[0m deleting the test service [38;5;243m@ 03/05/23 16:58:29.44[0m [1mSTEP:[0m deleting the test headless service [38;5;243m@ 03/05/23 16:58:29.568[0m [1mSTEP:[0m dump namespace information after failure [38;5;243m@ 03/05/23 16:58:29.687[0m [1mSTEP:[0m Collecting events from namespace "dns-8819". [38;5;243m@ 03/05/23 16:58:29.687[0m [1mSTEP:[0m Found 3 events. [38;5;243m@ 03/05/23 16:58:29.801[0m Mar 5 16:58:29.801: INFO: At 2023-03-05 16:40:28 +0000 UTC - event for dns-test-cb157f07-8473-48eb-a0a2-8df8dabcd1aa: {default-scheduler } Scheduled: Successfully assigned dns-8819/dns-test-cb157f07-8473-48eb-a0a2-8df8dabcd1aa to i-06df568f6e5c59fff ... skipping 77 lines ... Logging node info for node i-06df568f6e5c59fff Mar 5 16:58:31.019: INFO: Node Info: &Node{ObjectMeta:{i-06df568f6e5c59fff cb63c681-2e18-474a-b6f3-221325ac0d52 18792 0 2023-03-05 16:24:16 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:arm64 beta.kubernetes.io/instance-type:t4g.small beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:eu-central-1 failure-domain.beta.kubernetes.io/zone:eu-central-1a karpenter.k8s.aws/instance-category:t karpenter.k8s.aws/instance-cpu:2 karpenter.k8s.aws/instance-family:t4g karpenter.k8s.aws/instance-generation:4 karpenter.k8s.aws/instance-hypervisor:nitro karpenter.k8s.aws/instance-memory:2048 karpenter.k8s.aws/instance-pods:11 karpenter.k8s.aws/instance-size:small karpenter.sh/capacity-type:spot karpenter.sh/initialized:true karpenter.sh/provisioner-name:nodes kubernetes.io/arch:arm64 kubernetes.io/hostname:i-06df568f6e5c59fff kubernetes.io/os:linux node-role.kubernetes.io/node: node-role.kubernetes.io/spot-worker:true node.kubernetes.io/instance-type:t4g.small topology.ebs.csi.aws.com/zone:eu-central-1a topology.hostpath.csi/node:i-06df568f6e5c59fff topology.kubernetes.io/region:eu-central-1 topology.kubernetes.io/zone:eu-central-1a] map[csi.volume.kubernetes.io/nodeid:{"csi-hostpath-ephemeral-2147":"i-06df568f6e5c59fff","csi-hostpath-ephemeral-5408":"i-06df568f6e5c59fff","csi-hostpath-ephemeral-6540":"i-06df568f6e5c59fff","csi-mock-csi-mock-volumes-fsgroup-policy-1058":"i-06df568f6e5c59fff","csi-mock-csi-mock-volumes-workload-2219":"i-06df568f6e5c59fff","ebs.csi.aws.com":"i-06df568f6e5c59fff"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [{karpenter.sh/v1alpha5 Provisioner nodes ac5d7599-b2a2-4eae-8da4-1f4c0a54ec82 <nil> 0xc0054dbc20}] [karpenter.sh/termination] [{aws-cloud-controller-manager Update v1 2023-03-05 16:24:16 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{}}}} } {kops-controller Update v1 2023-03-05 16:24:16 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{},"f:node-role.kubernetes.io/spot-worker":{}}}} } {karpenter Update v1 2023-03-05 16:25:57 +0000 UTC FieldsV1 {"f:metadata":{"f:finalizers":{".":{},"v:\"karpenter.sh/termination\"":{}},"f:labels":{".":{},"f:karpenter.k8s.aws/instance-category":{},"f:karpenter.k8s.aws/instance-cpu":{},"f:karpenter.k8s.aws/instance-family":{},"f:karpenter.k8s.aws/instance-generation":{},"f:karpenter.k8s.aws/instance-hypervisor":{},"f:karpenter.k8s.aws/instance-memory":{},"f:karpenter.k8s.aws/instance-pods":{},"f:karpenter.k8s.aws/instance-size":{},"f:karpenter.sh/capacity-type":{},"f:karpenter.sh/initialized":{},"f:karpenter.sh/provisioner-name":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}},"f:ownerReferences":{".":{},"k:{\"uid\":\"ac5d7599-b2a2-4eae-8da4-1f4c0a54ec82\"}":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-03-05 16:26:48 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{".":{},"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"InternalIP\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kubelet Update v1 2023-03-05 16:48:40 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{"f:kubernetes.io/hostname":{},"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{".":{},"f:cpu":{},"f:ephemeral-storage":{},"f:hugepages-1Gi":{},"f:hugepages-2Mi":{},"f:hugepages-32Mi":{},"f:hugepages-64Ki":{},"f:memory":{},"f:pods":{}},"f:capacity":{".":{},"f:cpu":{},"f:ephemeral-storage":{},"f:hugepages-1Gi":{},"f:hugepages-2Mi":{},"f:hugepages-32Mi":{},"f:hugepages-64Ki":{},"f:memory":{},"f:pods":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{}}},"f:daemonEndpoints":{"f:kubeletEndpoint":{"f:Port":{}}},"f:images":{},"f:nodeInfo":{"f:architecture":{},"f:bootID":{},"f:containerRuntimeVersion":{},"f:kernelVersion":{},"f:kubeProxyVersion":{},"f:kubeletVersion":{},"f:machineID":{},"f:operatingSystem":{},"f:osImage":{},"f:systemUUID":{}},"f:volumesInUse":{}}} status} {kube-controller-manager Update v1 2023-03-05 16:49:37 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{".":{},"k:{\"type\":\"DiskPressure\"}":{".":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"MemoryPressure\"}":{".":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"PIDPressure\"}":{".":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Ready\"}":{".":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}},"f:volumesAttached":{}}} status} {kube-controller-manager Update v1 2023-03-05 16:49:42 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:node.alpha.kubernetes.io/ttl":{}},"f:labels":{"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.2.0/24\"":{}},"f:taints":{}}} }]},Spec:NodeSpec{PodCIDR:100.96.2.0/24,DoNotUseExternalID:,ProviderID:aws:///eu-central-1a/i-06df568f6e5c59fff,Unschedulable:false,Taints:[]Taint{Taint{Key:node.kubernetes.io/unreachable,Value:,Effect:NoSchedule,TimeAdded:2023-03-05 16:49:37 +0000 UTC,},Taint{Key:node.kubernetes.io/unreachable,Value:,Effect:NoExecute,TimeAdded:2023-03-05 16:49:42 +0000 UTC,},},ConfigSource:nil,PodCIDRs:[100.96.2.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49756430336 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-32Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-64Ki: {{0 0} {<nil>} 0 DecimalSI},memory: {{1941540864 0} {<nil>} 1896036Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44780787229 0} {<nil>} 44780787229 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-32Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-64Ki: {{0 0} {<nil>} 0 DecimalSI},memory: {{1836683264 0} {<nil>} 1793636Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:Ready,Status:Unknown,LastHeartbeatTime:2023-03-05 16:47:34 +0000 UTC,LastTransitionTime:2023-03-05 16:49:37 +0000 UTC,Reason:NodeStatusUnknown,Message:Kubelet stopped posting node status.,},NodeCondition{Type:MemoryPressure,Status:Unknown,LastHeartbeatTime:2023-03-05 16:47:34 +0000 UTC,LastTransitionTime:2023-03-05 16:49:37 +0000 UTC,Reason:NodeStatusUnknown,Message:Kubelet stopped posting node status.,},NodeCondition{Type:DiskPressure,Status:Unknown,LastHeartbeatTime:2023-03-05 16:47:34 +0000 UTC,LastTransitionTime:2023-03-05 16:49:37 +0000 UTC,Reason:NodeStatusUnknown,Message:Kubelet stopped posting node status.,},NodeCondition{Type:PIDPressure,Status:Unknown,LastHeartbeatTime:2023-03-05 16:47:34 +0000 UTC,LastTransitionTime:2023-03-05 16:49:37 +0000 UTC,Reason:NodeStatusUnknown,Message:Kubelet stopped posting node status.,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.43.148,},NodeAddress{Type:ExternalIP,Address:18.192.211.222,},NodeAddress{Type:InternalDNS,Address:i-06df568f6e5c59fff.eu-central-1.compute.internal,},NodeAddress{Type:Hostname,Address:i-06df568f6e5c59fff.eu-central-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-18-192-211-222.eu-central-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec221221864d9409a8ec5f6614a9b94b,SystemUUID:ec221221-864d-9409-a8ec-5f6614a9b94b,BootID:553d31bf-faca-417e-a3db-325839e1b8e4,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 22.04.2 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.27.0-alpha.3.52+d48b8167f7094b,KubeProxyVersion:v1.27.0-alpha.3.52+d48b8167f7094b,OperatingSystem:linux,Architecture:arm64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:157636062,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/jessie-dnsutils@sha256:24aaf2626d6b27864c29de2097e8bbb840b3a414271bf7c8995e431e47d8408e registry.k8s.io/e2e-test-images/jessie-dnsutils:1.7],SizeBytes:116764683,},ContainerImage{Names:[registry.k8s.io/kube-proxy-arm64:v1.27.0-alpha.3.52_d48b8167f7094b],SizeBytes:60554082,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:16bbf38c463a4223d8cfe4da12bc61010b082a79b4bb003e2d3ba3ece5dd5f9e registry.k8s.io/e2e-test-images/agnhost:2.43],SizeBytes:49808364,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:3fe7acf013d1264ffded116b80a73dc129a449b0fccdb8d21af8279f2233f36e registry.k8s.io/e2e-test-images/httpd:2.4.39-4],SizeBytes:41528818,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:148b022f5c5da426fc2f3c14b5c0867e58ef05961510c84749ac1fddcb0fef22 registry.k8s.io/e2e-test-images/httpd:2.4.38-4],SizeBytes:40343601,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:28932904,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:ee3b525d5b89db99da3b8eb521d9cd90cb6e9ef0fbb651e98bb37be78d36b5b8 registry.k8s.io/sig-storage/csi-provisioner:v3.3.0],SizeBytes:23945474,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:425d8f1b769398127767b06ed97ce62578a3179bcb99809ce93a1649e025ffe7 registry.k8s.io/sig-storage/csi-resizer:v1.6.0],SizeBytes:22678632,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:291334908ddf71a4661fd7f6d9d97274de8a5378a2b6fdfeb2ce73414a34f82f registry.k8s.io/sig-storage/csi-snapshotter:v6.1.0],SizeBytes:22434180,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:9a685020911e2725ad019dbce6e4a5ab93d51e3d4557f115e64343345e05781b registry.k8s.io/sig-storage/csi-attacher:v4.0.0],SizeBytes:22394501,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:122bfb8c1edabb3c0edd63f06523e6940d958d19b3957dc7b1d6f81e9f1f6119 registry.k8s.io/sig-storage/csi-provisioner:v3.1.0],SizeBytes:21735644,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:9ebbf9f023e7b41ccee3d52afe39a89e3ddacdbb69269d583abfc25847cfd9e4 registry.k8s.io/sig-storage/csi-resizer:v1.4.0],SizeBytes:20848413,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:8b9c313c05f54fb04f8d430896f5f5904b6cb157df261501b29adc04d2b2dc7b registry.k8s.io/sig-storage/csi-attacher:v3.4.0],SizeBytes:20565480,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nonroot@sha256:ee9f50b3c64b174d296d91ca9f69a914ac30e59095dfb462b2b518ad28a63655 registry.k8s.io/e2e-test-images/nonroot:1.4],SizeBytes:18561972,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:92257881c1d6493cf18299a24af42330f891166560047902b8d431fb66b01af5 registry.k8s.io/sig-storage/hostpathplugin:v1.9.0],SizeBytes:18267611,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:9b2d9abda017c28b12f84a344f57cd73fbdb6c2bd7dd5bdd5018246ad1093ba6 registry.k8s.io/sig-storage/hostpathplugin:v1.11.0],SizeBytes:18159085,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:017727efcfeb7d053af68e51436ce8e65edbc6ca573720afb4f79c8594036955 registry.k8s.io/coredns/coredns:v1.10.0],SizeBytes:13766234,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:8502345,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:933940f13b3ea0abc62e656c1aa5c5b47c04b15d71250413a6b821bd0c58b94e registry.k8s.io/sig-storage/livenessprobe:v2.7.0],SizeBytes:8223819,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:7685305,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:5c99cf6a02adda929b10321dbf4ecfa00d87be9ba4fb456006237d530ab4baa1 registry.k8s.io/e2e-test-images/nginx:1.14-4],SizeBytes:6860089,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nonewprivs@sha256:8ac1264691820febacf3aea5d152cbde6d10685731ec14966a9401c6f47a68ac registry.k8s.io/e2e-test-images/nonewprivs:1.3],SizeBytes:5025027,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:2e0f836850e09b8b7cc937681d6194537a09fbd5f6b9e08f4d646a85128e8937 registry.k8s.io/e2e-test-images/busybox:1.29-4],SizeBytes:772933,},ContainerImage{Names:[registry.k8s.io/pause@sha256:7031c1b283388d2c2e09b57badb803c05ebed362dc88d84b480cc47f72a21097 registry.k8s.io/pause:3.9],SizeBytes:268051,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:253553,},},VolumesInUse:[kubernetes.io/csi/csi-hostpath-ephemeral-2147^52df5953-bb73-11ed-8aa7-be49170871b4 kubernetes.io/csi/csi-hostpath-ephemeral-5408^443d1daf-bb73-11ed-95c4-5a95d17cbc27 kubernetes.io/csi/csi-hostpath-ephemeral-5408^44411f51-bb73-11ed-95c4-5a95d17cbc27 kubernetes.io/csi/csi-hostpath-ephemeral-6540^4426a07f-bb73-11ed-a604-92687f2c50f8 kubernetes.io/csi/ebs.csi.aws.com^vol-05b67fab5beb6394b],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-05b67fab5beb6394b,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/csi-hostpath-ephemeral-6540^4426a07f-bb73-11ed-a604-92687f2c50f8,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/csi-hostpath-ephemeral-5408^44411f51-bb73-11ed-95c4-5a95d17cbc27,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/csi-hostpath-ephemeral-5408^443d1daf-bb73-11ed-95c4-5a95d17cbc27,DevicePath:,},},Config:nil,},} Mar 5 16:58:31.020: INFO: Logging kubelet events for node i-06df568f6e5c59fff Mar 5 16:58:31.138: INFO: Logging pods the kubelet thinks is on node i-06df568f6e5c59fff Mar 5 16:58:41.254: INFO: Unable to retrieve kubelet pods for node i-06df568f6e5c59fff: error trying to reach service: net/http: TLS handshake timeout Mar 5 16:58:41.254: INFO: Logging node info for node i-0d6a32f749839192f Mar 5 16:58:41.369: INFO: Node Info: &Node{ObjectMeta:{i-0d6a32f749839192f 4155fefb-4d3a-4b04-ba53-6ac9e895ff92 21031 0 2023-03-05 16:20:46 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:arm64 beta.kubernetes.io/instance-type:c6g.xlarge beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:eu-central-1 failure-domain.beta.kubernetes.io/zone:eu-central-1a kops.k8s.io/kops-controller-pki: kubernetes.io/arch:arm64 kubernetes.io/hostname:i-0d6a32f749839192f kubernetes.io/os:linux node-role.kubernetes.io/control-plane: node.kubernetes.io/exclude-from-external-load-balancers: node.kubernetes.io/instance-type:c6g.xlarge topology.ebs.csi.aws.com/zone:eu-central-1a topology.kubernetes.io/region:eu-central-1 topology.kubernetes.io/zone:eu-central-1a] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-0d6a32f749839192f"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kubelet Update v1 2023-03-05 16:20:46 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {protokube Update v1 2023-03-05 16:21:25 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kops.k8s.io/kops-controller-pki":{},"f:node-role.kubernetes.io/control-plane":{},"f:node.kubernetes.io/exclude-from-external-load-balancers":{}}}} } {kube-controller-manager Update v1 2023-03-05 16:21:47 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.0.0/24\"":{}}}} } {aws-cloud-controller-manager Update v1 2023-03-05 16:21:48 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{},"f:taints":{}}} } {aws-cloud-controller-manager Update v1 2023-03-05 16:21:48 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kubelet Update v1 2023-03-05 16:54:23 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.0.0/24,DoNotUseExternalID:,ProviderID:aws:///eu-central-1a/i-0d6a32f749839192f,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/control-plane,Value:,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[100.96.0.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{4 0} {<nil>} 4 DecimalSI},ephemeral-storage: {{49756430336 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-32Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-64Ki: {{0 0} {<nil>} 0 DecimalSI},memory: {{8150118400 0} {<nil>} 7959100Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{4 0} {<nil>} 4 DecimalSI},ephemeral-storage: {{44780787229 0} {<nil>} 44780787229 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-32Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-64Ki: {{0 0} {<nil>} 0 DecimalSI},memory: {{8045260800 0} {<nil>} 7856700Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-03-05 16:54:23 +0000 UTC,LastTransitionTime:2023-03-05 16:20:46 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-03-05 16:54:23 +0000 UTC,LastTransitionTime:2023-03-05 16:20:46 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-03-05 16:54:23 +0000 UTC,LastTransitionTime:2023-03-05 16:20:46 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-03-05 16:54:23 +0000 UTC,LastTransitionTime:2023-03-05 16:21:47 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.49.247,},NodeAddress{Type:ExternalIP,Address:18.197.20.82,},NodeAddress{Type:InternalDNS,Address:i-0d6a32f749839192f.eu-central-1.compute.internal,},NodeAddress{Type:Hostname,Address:i-0d6a32f749839192f.eu-central-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-18-197-20-82.eu-central-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2bed249e8ac7b517cb08ccd42b83e5,SystemUUID:ec2bed24-9e8a-c7b5-17cb-08ccd42b83e5,BootID:6f9215c9-dfe8-4b13-84f4-9b2084456198,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 22.04.2 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.27.0-alpha.3.52+d48b8167f7094b,KubeProxyVersion:v1.27.0-alpha.3.52+d48b8167f7094b,OperatingSystem:linux,Architecture:arm64,},Images:[]ContainerImage{ContainerImage{Names:[registry.k8s.io/etcd@sha256:9ce33ba33d8e738a5b85ed50b5080ac746deceed4a7496c550927a7a19ca3b6d registry.k8s.io/etcd:3.5.0-0],SizeBytes:157797353,},ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:157636062,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:4afb99b4690b418ffc2ceb67e1a17376457e441c1f09ab55447f0aaf992fa646 registry.k8s.io/etcd:3.4.3-0],SizeBytes:151677080,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:4ad90a11b55313b182afc186b9876c8e891531b8db4c9bf1541953021618d0e2 registry.k8s.io/etcd:3.4.13-0],SizeBytes:134531559,},ContainerImage{Names:[registry.k8s.io/kube-apiserver-arm64:v1.27.0-alpha.3.52_d48b8167f7094b],SizeBytes:131087707,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:0d2ab6b15b4fe0b903b9e81192d942fb39c29be61fe341cd16db5d0165b0435c registry.k8s.io/etcd:3.3.17-0],SizeBytes:126979107,},ContainerImage{Names:[registry.k8s.io/kube-controller-manager-arm64:v1.27.0-alpha.3.52_d48b8167f7094b],SizeBytes:120667128,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:17da501f5d2a675be46040422a27b7cc21b8a43895ac998b171db1c346f361f7 registry.k8s.io/etcd:3.3.10-0],SizeBytes:81436097,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:6f72b851544986cb0921b53ea655ec04c36131248f16d4ad110cb3ca0c369dc1 registry.k8s.io/etcd:3.5.4-0],SizeBytes:81120118,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:13f53ed1d91e2e11aac476ee9a0269fdda6cc4874eba903efd40daf50c55eee5 registry.k8s.io/etcd:3.5.3-0],SizeBytes:81114891,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:51eae8381dcb1078289fa7b4f3df2630cdc18d09fb56f8e56b41c40e191d6c83 registry.k8s.io/etcd:3.5.7-0],SizeBytes:80665728,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:dd75ec974b0a2a6f6bb47001ba09207976e625db898d1b16735528c009cb171c registry.k8s.io/etcd:3.5.6-0],SizeBytes:80539316,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:905d7ca17fd02bc24c0eba9a062753aba15db3e31422390bc3238eb762339b20 registry.k8s.io/etcd:3.2.24-1],SizeBytes:61500433,},ContainerImage{Names:[registry.k8s.io/etcdadm/etcd-manager-slim@sha256:8727adc059334e0e3329dbccce116a734da4a090da5e86c8cddccaf333caa666 registry.k8s.io/etcdadm/etcd-manager-slim:v3.0.20230201],SizeBytes:60826655,},ContainerImage{Names:[registry.k8s.io/kube-proxy-arm64:v1.27.0-alpha.3.52_d48b8167f7094b],SizeBytes:60554082,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:64b9ea357325d5db9f8a723dcf503b5a449177b17ac87d69481e126bb724c263 registry.k8s.io/etcd:3.5.1-0],SizeBytes:59553904,},ContainerImage{Names:[registry.k8s.io/kube-scheduler-arm64:v1.27.0-alpha.3.52_d48b8167f7094b],SizeBytes:56310763,},ContainerImage{Names:[registry.k8s.io/kops/kops-controller:1.27.0-alpha.1],SizeBytes:41797411,},ContainerImage{Names:[registry.k8s.io/kops/dns-controller:1.27.0-alpha.1],SizeBytes:40267473,},ContainerImage{Names:[public.ecr.aws/karpenter/controller@sha256:68db4f092cf9cc83f5ef9e2fbc5407c2cb682e81f64dfaa700a7602ede38b1cf public.ecr.aws/karpenter/controller:v0.16.3],SizeBytes:33707751,},ContainerImage{Names:[public.ecr.aws/karpenter/webhook@sha256:96a2d9b06d6bc5127801f358f74b1cf2d289b423a2e9ba40c573c0b14b17dafa public.ecr.aws/karpenter/webhook:v0.16.3],SizeBytes:33307187,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:28932904,},ContainerImage{Names:[quay.io/cilium/operator@sha256:a6d24a006a6b92967ac90786b49bc1ac26e5477cf028cd1186efcfc2466484db quay.io/cilium/operator:v1.12.5],SizeBytes:24567287,},ContainerImage{Names:[registry.k8s.io/provider-aws/cloud-controller-manager@sha256:fdeb61e3e42ecd9cca868d550ebdb88dd6341d9e91fcfa9a37e227dab2ad22cb registry.k8s.io/provider-aws/cloud-controller-manager:v1.26.0],SizeBytes:18345226,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:8502345,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:7685305,},ContainerImage{Names:[registry.k8s.io/kops/kube-apiserver-healthcheck:1.27.0-alpha.1],SizeBytes:4713612,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:253553,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Mar 5 16:58:41.370: INFO: Logging kubelet events for node i-0d6a32f749839192f Mar 5 16:58:41.488: INFO: ... skipping 52 lines ... Mar 5 16:58:41.618: INFO: Container controller ready: true, restart count 0 Mar 5 16:58:42.117: INFO: Latency metrics for node i-0d6a32f749839192f [1mSTEP:[0m Destroying namespace "dns-8819" for this suite. [38;5;243m@ 03/05/23 16:58:42.117[0m [38;5;243m<< Timeline[0m [38;5;9m[FAILED] Timed out after 900.000s. The function passed to Eventually returned the following error: pods "dns-test-cb157f07-8473-48eb-a0a2-8df8dabcd1aa" not found <framework.transientError>: { error: <*errors.StatusError | 0xc00461dae0>{ ErrStatus: { TypeMeta: {Kind: "", APIVersion: ""}, ListMeta: { SelfLink: "", ResourceVersion: "", Continue: "", ... skipping 12 lines ... }, Code: 404, }, }, } At one point, however, the function did return successfully. Yet, Eventually failed because the matcher was not satisfied: Expected Pod to be in <v1.PodPhase>: "Running" Got instead: <*v1.Pod | 0xc00025c480>: metadata: creationTimestamp: "2023-03-05T16:40:28Z" labels: ... skipping 248 lines ... [38;5;9mIn [1m[It][0m[38;5;9m at: [1mtest/e2e/network/dns_common.go:506[0m [38;5;243m@ 03/05/23 16:55:28.978[0m There were [1m[38;5;9madditional failures[0m detected. To view them in detail run [1mginkgo -vv[0m [38;5;243m------------------------------[0m [38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;10m•[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;10m•[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m[38;5;14mS[0m [38;5;243m------------------------------[0m [38;5;9m• [FAILED] [1694.435 seconds][0m [0m[sig-storage] CSI Mock workload info [38;5;243mCSI workload information using mock driver [38;5;9m[1m[It] contain ephemeral=true when using inline volume[0m [38;5;243mtest/e2e/storage/csi_mock/csi_workload.go:85[0m [38;5;243mTimeline >>[0m [1mSTEP:[0m Creating a kubernetes client [38;5;243m@ 03/05/23 16:31:02.14[0m Mar 5 16:31:02.140: INFO: >>> kubeConfig: /root/.kube/config ... skipping 39 lines ... Mar 5 16:31:07.293: INFO: creating *v1.CSIDriver: csi-mock-csi-mock-volumes-workload-2219 Mar 5 16:31:07.408: INFO: creating *v1.StatefulSet: csi-mock-volumes-workload-2219-9598/csi-mockplugin-attacher Mar 5 16:31:07.523: INFO: waiting up to 4m0s for CSIDriver "csi-mock-csi-mock-volumes-workload-2219" Mar 5 16:31:07.636: INFO: waiting for CSIDriver csi-mock-csi-mock-volumes-workload-2219 to register on node i-06df568f6e5c59fff [1mSTEP:[0m Creating pod [38;5;243m@ 03/05/23 16:31:17.591[0m [1mSTEP:[0m checking for CSIInlineVolumes feature [38;5;243m@ 03/05/23 16:31:22.051[0m Mar 5 16:31:22.282: INFO: Error getting logs for pod inline-volume-l54xx: the server rejected our request for an unknown reason (get pods inline-volume-l54xx) Mar 5 16:31:22.396: INFO: Deleting pod "inline-volume-l54xx" in namespace "csi-mock-volumes-workload-2219" Mar 5 16:31:22.517: INFO: Wait up to 5m0s for pod "inline-volume-l54xx" to be fully deleted [1mSTEP:[0m Deleting the previously created pod [38;5;243m@ 03/05/23 16:33:35.838[0m Mar 5 16:33:35.838: INFO: Deleting pod "pvc-volume-tester-nc7w4" in namespace "csi-mock-volumes-workload-2219" Mar 5 16:33:35.955: INFO: Wait up to 5m0s for pod "pvc-volume-tester-nc7w4" to be fully deleted Mar 5 16:38:35.959: INFO: Failed inside E2E framework: k8s.io/kubernetes/test/e2e/framework/pod.WaitForPodNotFoundInNamespace({0x7f1dc82c0498, 0xc00336a8c0}, {0x8252218?, 0xc0034009c0}, {0xc003e9a408, 0x17}, {0xc003d9ac20, 0x1e}, 0x0?) test/e2e/framework/pod/wait.go:538 +0x190 k8s.io/kubernetes/test/e2e/framework/pod.DeletePodWithWaitByName({0x7f1dc82c0498, 0xc00336a8c0}, {0x8252218?, 0xc0034009c0?}, {0xc003e9a408, 0x17}, {0xc003d9ac20, 0x1e}) test/e2e/framework/pod/delete.go:71 +0x31b k8s.io/kubernetes/test/e2e/framework/pod.DeletePodWithWait({0x7f1dc82c0498?, 0xc00336a8c0?}, {0x8252218?, 0xc0034009c0?}, 0x0?) test/e2e/framework/pod/delete.go:56 +0x45 k8s.io/kubernetes/test/e2e/storage/csi_mock.glob..func11.1.1({0x7f1dc82c0498, 0xc00336a8c0}) test/e2e/storage/csi_mock/csi_workload.go:114 +0x4aa [38;5;9m[FAILED][0m in [It] - test/e2e/storage/csi_mock/csi_workload.go:115 [38;5;243m@ 03/05/23 16:38:35.959[0m Mar 5 16:38:35.959: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready WARNING: pod log: csi-mockplugin-0/csi-provisioner: Get "https://api.e2e-e2e-kops-aws-karpenter.test-cncf-aws.k8s.io/api/v1/namespaces/csi-mock-volumes-workload-2219-9598/pods/csi-mockplugin-0/log?container=csi-provisioner&follow=true": context canceled WARNING: pod log: csi-mockplugin-0/driver-registrar: client rate limiter Wait returned an error: context canceled WARNING: pod log: csi-mockplugin-0/mock: client rate limiter Wait returned an error: context canceled WARNING: pod log: csi-mockplugin-attacher-0/csi-attacher: client rate limiter Wait returned an error: context canceled Mar 5 16:38:36.073: INFO: Condition Ready of node i-06df568f6e5c59fff is false, but Node is tainted by NodeController with [{node.kubernetes.io/not-ready NoSchedule 2023-03-05 16:37:27 +0000 UTC} {node.kubernetes.io/not-ready NoExecute 2023-03-05 16:37:31 +0000 UTC}]. Failure Mar 5 16:38:38.188: INFO: Condition Ready of node i-06df568f6e5c59fff is false, but Node is tainted by NodeController with [{node.kubernetes.io/not-ready NoSchedule 2023-03-05 16:37:27 +0000 UTC} {node.kubernetes.io/not-ready NoExecute 2023-03-05 16:37:31 +0000 UTC}]. Failure Mar 5 16:38:40.189: INFO: Condition Ready of node i-06df568f6e5c59fff is false, but Node is tainted by NodeController with [{node.kubernetes.io/not-ready NoSchedule 2023-03-05 16:37:27 +0000 UTC} {node.kubernetes.io/not-ready NoExecute 2023-03-05 16:37:31 +0000 UTC}]. Failure Mar 5 16:38:42.188: INFO: Condition Ready of node i-06df568f6e5c59fff is false, but Node is tainted by NodeController with [{node.kubernetes.io/not-ready NoSchedule 2023-03-05 16:37:27 +0000 UTC} {node.kubernetes.io/not-ready NoExecute 2023-03-05 16:37:31 +0000 UTC}]. Failure Mar 5 16:38:44.189: INFO: Condition Ready of node i-06df568f6e5c59fff is false, but Node is tainted by NodeController with [{node.kubernetes.io/not-ready NoSchedule 2023-03-05 16:37:27 +0000 UTC} {node.kubernetes.io/not-ready NoExecute 2023-03-05 16:37:31 +0000 UTC}]. Failure Mar 5 16:38:46.189: INFO: Condition Ready of node i-06df568f6e5c59fff is false, but Node is tainted by NodeController with [{node.kubernetes.io/not-ready NoSchedule 2023-03-05 16:37:27 +0000 UTC} {node.kubernetes.io/not-ready NoExecute 2023-03-05 16:37:31 +0000 UTC}]. Failure ... skipping 47 lines ... Mar 5 16:40:22.189: INFO: Condition Ready of node i-06df568f6e5c59fff is false, but Node is tainted by NodeController with [{node.kubernetes.io/not-ready NoSchedule 2023-03-05 16:37:27 +0000 UTC} {node.kubernetes.io/not-ready NoExecute 2023-03-05 16:37:31 +0000 UTC}]. Failure Mar 5 16:40:24.189: INFO: Condition Ready of node i-06df568f6e5c59fff is false, but Node is tainted by NodeController with [{node.kubernetes.io/not-ready NoSchedule 2023-03-05 16:37:27 +0000 UTC} {node.kubernetes.io/not-ready NoExecute 2023-03-05 16:37:31 +0000 UTC}]. Failure Mar 5 16:40:26.189: INFO: Condition Ready of node i-06df568f6e5c59fff is true, but Node is tainted by NodeController with [{node.kubernetes.io/not-ready NoExecute 2023-03-05 16:37:31 +0000 UTC}]. Failure [1mSTEP:[0m Deleting pod pvc-volume-tester-nc7w4 [38;5;243m@ 03/05/23 16:40:28.189[0m Mar 5 16:40:28.189: INFO: Deleting pod "pvc-volume-tester-nc7w4" in namespace "csi-mock-volumes-workload-2219" Mar 5 16:40:28.302: INFO: Wait up to 5m0s for pod "pvc-volume-tester-nc7w4" to be fully deleted [38;5;9m[FAILED][0m in [DeferCleanup (Each)] - test/e2e/storage/csi_mock/base.go:242 [38;5;243m@ 03/05/23 16:45:28.305[0m Mar 5 16:45:28.419: INFO: deleting .: csi-mock-csi-mock-volumes-workload-2219 [1mSTEP:[0m deleting the test namespace: csi-mock-volumes-workload-2219 [38;5;243m@ 03/05/23 16:45:28.534[0m [1mSTEP:[0m Collecting events from namespace "csi-mock-volumes-workload-2219". [38;5;243m@ 03/05/23 16:45:28.534[0m [1mSTEP:[0m Found 10 events. [38;5;243m@ 03/05/23 16:45:28.647[0m Mar 5 16:45:28.647: INFO: At 2023-03-05 16:31:17 +0000 UTC - event for pvc-volume-tester-nc7w4: {default-scheduler } Scheduled: Successfully assigned csi-mock-volumes-workload-2219/pvc-volume-tester-nc7w4 to i-06df568f6e5c59fff Mar 5 16:45:28.647: INFO: At 2023-03-05 16:31:18 +0000 UTC - event for pvc-volume-tester-nc7w4: {kubelet i-06df568f6e5c59fff} Pulled: Container image "registry.k8s.io/pause:3.9" already present on machine Mar 5 16:45:28.647: INFO: At 2023-03-05 16:31:18 +0000 UTC - event for pvc-volume-tester-nc7w4: {kubelet i-06df568f6e5c59fff} Created: Created container volume-tester Mar 5 16:45:28.647: INFO: At 2023-03-05 16:31:18 +0000 UTC - event for pvc-volume-tester-nc7w4: {kubelet i-06df568f6e5c59fff} Started: Started container volume-tester Mar 5 16:45:28.647: INFO: At 2023-03-05 16:31:22 +0000 UTC - event for inline-volume-l54xx: {default-scheduler } Scheduled: Successfully assigned csi-mock-volumes-workload-2219/inline-volume-l54xx to i-056ad61525499081e Mar 5 16:45:28.647: INFO: At 2023-03-05 16:31:22 +0000 UTC - event for inline-volume-l54xx: {kubelet i-056ad61525499081e} FailedMount: MountVolume.SetUp failed for volume "my-volume" : kubernetes.io/csi: mounter.SetUpAt failed to get CSI client: driver name no-such-driver.example.com not found in the list of registered CSI drivers Mar 5 16:45:28.647: INFO: At 2023-03-05 16:33:21 +0000 UTC - event for pvc-volume-tester-nc7w4: {node-controller } NodeNotReady: Node is not ready Mar 5 16:45:28.647: INFO: At 2023-03-05 16:33:25 +0000 UTC - event for inline-volume-l54xx: {kubelet i-056ad61525499081e} FailedMount: Unable to attach or mount volumes: unmounted volumes=[my-volume], unattached volumes=[], failed to process volumes=[]: timed out waiting for the condition Mar 5 16:45:28.647: INFO: At 2023-03-05 16:38:26 +0000 UTC - event for pvc-volume-tester-nc7w4: {taint-controller } TaintManagerEviction: Marking for deletion Pod csi-mock-volumes-workload-2219/pvc-volume-tester-nc7w4 Mar 5 16:45:28.647: INFO: At 2023-03-05 16:41:51 +0000 UTC - event for pvc-volume-tester-nc7w4: {taint-controller } TaintManagerEviction: Cancelling deletion of Pod csi-mock-volumes-workload-2219/pvc-volume-tester-nc7w4 Mar 5 16:45:28.761: INFO: POD NODE PHASE GRACE CONDITIONS Mar 5 16:45:28.761: INFO: pvc-volume-tester-nc7w4 i-06df568f6e5c59fff Running 30s [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2023-03-05 16:31:17 +0000 UTC } {Ready False 0001-01-01 00:00:00 +0000 UTC 2023-03-05 16:33:21 +0000 UTC } {ContainersReady True 0001-01-01 00:00:00 +0000 UTC 2023-03-05 16:31:19 +0000 UTC } {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2023-03-05 16:31:17 +0000 UTC } {DisruptionTarget True 0001-01-01 00:00:00 +0000 UTC 2023-03-05 16:38:26 +0000 UTC DeletionByTaintManager Taint manager: deleting due to NoExecute taint}] Mar 5 16:45:28.761: INFO: Mar 5 16:46:18.190: INFO: ... skipping 72 lines ... Logging node info for node i-06df568f6e5c59fff Mar 5 16:46:19.254: INFO: Node Info: &Node{ObjectMeta:{i-06df568f6e5c59fff cb63c681-2e18-474a-b6f3-221325ac0d52 16488 0 2023-03-05 16:24:16 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:arm64 beta.kubernetes.io/instance-type:t4g.small beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:eu-central-1 failure-domain.beta.kubernetes.io/zone:eu-central-1a karpenter.k8s.aws/instance-category:t karpenter.k8s.aws/instance-cpu:2 karpenter.k8s.aws/instance-family:t4g karpenter.k8s.aws/instance-generation:4 karpenter.k8s.aws/instance-hypervisor:nitro karpenter.k8s.aws/instance-memory:2048 karpenter.k8s.aws/instance-pods:11 karpenter.k8s.aws/instance-size:small karpenter.sh/capacity-type:spot karpenter.sh/initialized:true karpenter.sh/provisioner-name:nodes kubelet_cleanup:true kubernetes.io/arch:arm64 kubernetes.io/hostname:i-06df568f6e5c59fff kubernetes.io/os:linux node-role.kubernetes.io/node: node-role.kubernetes.io/spot-worker:true node.kubernetes.io/instance-type:t4g.small topology.ebs.csi.aws.com/zone:eu-central-1a topology.hostpath.csi/node:i-06df568f6e5c59fff topology.kubernetes.io/region:eu-central-1 topology.kubernetes.io/zone:eu-central-1a] map[csi.volume.kubernetes.io/nodeid:{"csi-hostpath-ephemeral-2147":"i-06df568f6e5c59fff","csi-hostpath-ephemeral-5408":"i-06df568f6e5c59fff","csi-hostpath-ephemeral-6540":"i-06df568f6e5c59fff","csi-mock-csi-mock-volumes-fsgroup-policy-1058":"i-06df568f6e5c59fff","csi-mock-csi-mock-volumes-workload-2219":"i-06df568f6e5c59fff","ebs.csi.aws.com":"i-06df568f6e5c59fff"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [{karpenter.sh/v1alpha5 Provisioner nodes ac5d7599-b2a2-4eae-8da4-1f4c0a54ec82 <nil> 0xc004110950}] [karpenter.sh/termination] [{aws-cloud-controller-manager Update v1 2023-03-05 16:24:16 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{}}}} } {kops-controller Update v1 2023-03-05 16:24:16 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{},"f:node-role.kubernetes.io/spot-worker":{}}}} } {karpenter Update v1 2023-03-05 16:25:57 +0000 UTC FieldsV1 {"f:metadata":{"f:finalizers":{".":{},"v:\"karpenter.sh/termination\"":{}},"f:labels":{".":{},"f:karpenter.k8s.aws/instance-category":{},"f:karpenter.k8s.aws/instance-cpu":{},"f:karpenter.k8s.aws/instance-family":{},"f:karpenter.k8s.aws/instance-generation":{},"f:karpenter.k8s.aws/instance-hypervisor":{},"f:karpenter.k8s.aws/instance-memory":{},"f:karpenter.k8s.aws/instance-pods":{},"f:karpenter.k8s.aws/instance-size":{},"f:karpenter.sh/capacity-type":{},"f:karpenter.sh/initialized":{},"f:karpenter.sh/provisioner-name":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}},"f:ownerReferences":{".":{},"k:{\"uid\":\"ac5d7599-b2a2-4eae-8da4-1f4c0a54ec82\"}":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-03-05 16:26:48 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{".":{},"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"InternalIP\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {e2e.test Update v1 2023-03-05 16:40:28 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubelet_cleanup":{}}}} } {kubelet Update v1 2023-03-05 16:41:47 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{"f:kubernetes.io/hostname":{},"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{".":{},"f:cpu":{},"f:ephemeral-storage":{},"f:hugepages-1Gi":{},"f:hugepages-2Mi":{},"f:hugepages-32Mi":{},"f:hugepages-64Ki":{},"f:memory":{},"f:pods":{}},"f:capacity":{".":{},"f:cpu":{},"f:ephemeral-storage":{},"f:hugepages-1Gi":{},"f:hugepages-2Mi":{},"f:hugepages-32Mi":{},"f:hugepages-64Ki":{},"f:memory":{},"f:pods":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{}}},"f:daemonEndpoints":{"f:kubeletEndpoint":{"f:Port":{}}},"f:images":{},"f:nodeInfo":{"f:architecture":{},"f:bootID":{},"f:containerRuntimeVersion":{},"f:kernelVersion":{},"f:kubeProxyVersion":{},"f:kubeletVersion":{},"f:machineID":{},"f:operatingSystem":{},"f:osImage":{},"f:systemUUID":{}},"f:volumesInUse":{}}} status} {kube-controller-manager Update v1 2023-03-05 16:42:57 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:node.alpha.kubernetes.io/ttl":{}},"f:labels":{"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.2.0/24\"":{}},"f:taints":{}}} } {kube-controller-manager Update v1 2023-03-05 16:44:29 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{".":{},"k:{\"type\":\"DiskPressure\"}":{".":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"MemoryPressure\"}":{".":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"PIDPressure\"}":{".":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Ready\"}":{".":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}},"f:volumesAttached":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.2.0/24,DoNotUseExternalID:,ProviderID:aws:///eu-central-1a/i-06df568f6e5c59fff,Unschedulable:false,Taints:[]Taint{Taint{Key:node.kubernetes.io/unreachable,Value:,Effect:NoSchedule,TimeAdded:2023-03-05 16:42:51 +0000 UTC,},Taint{Key:node.kubernetes.io/unreachable,Value:,Effect:NoExecute,TimeAdded:2023-03-05 16:42:57 +0000 UTC,},},ConfigSource:nil,PodCIDRs:[100.96.2.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49756430336 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-32Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-64Ki: {{0 0} {<nil>} 0 DecimalSI},memory: {{1941540864 0} {<nil>} 1896036Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44780787229 0} {<nil>} 44780787229 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-32Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-64Ki: {{0 0} {<nil>} 0 DecimalSI},memory: {{1836683264 0} {<nil>} 1793636Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:Ready,Status:Unknown,LastHeartbeatTime:2023-03-05 16:41:39 +0000 UTC,LastTransitionTime:2023-03-05 16:42:51 +0000 UTC,Reason:NodeStatusUnknown,Message:Kubelet stopped posting node status.,},NodeCondition{Type:MemoryPressure,Status:Unknown,LastHeartbeatTime:2023-03-05 16:41:39 +0000 UTC,LastTransitionTime:2023-03-05 16:42:51 +0000 UTC,Reason:NodeStatusUnknown,Message:Kubelet stopped posting node status.,},NodeCondition{Type:DiskPressure,Status:Unknown,LastHeartbeatTime:2023-03-05 16:41:39 +0000 UTC,LastTransitionTime:2023-03-05 16:42:51 +0000 UTC,Reason:NodeStatusUnknown,Message:Kubelet stopped posting node status.,},NodeCondition{Type:PIDPressure,Status:Unknown,LastHeartbeatTime:2023-03-05 16:41:39 +0000 UTC,LastTransitionTime:2023-03-05 16:42:51 +0000 UTC,Reason:NodeStatusUnknown,Message:Kubelet stopped posting node status.,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.43.148,},NodeAddress{Type:ExternalIP,Address:18.192.211.222,},NodeAddress{Type:InternalDNS,Address:i-06df568f6e5c59fff.eu-central-1.compute.internal,},NodeAddress{Type:Hostname,Address:i-06df568f6e5c59fff.eu-central-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-18-192-211-222.eu-central-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec221221864d9409a8ec5f6614a9b94b,SystemUUID:ec221221-864d-9409-a8ec-5f6614a9b94b,BootID:553d31bf-faca-417e-a3db-325839e1b8e4,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 22.04.2 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.27.0-alpha.3.52+d48b8167f7094b,KubeProxyVersion:v1.27.0-alpha.3.52+d48b8167f7094b,OperatingSystem:linux,Architecture:arm64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:157636062,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/jessie-dnsutils@sha256:24aaf2626d6b27864c29de2097e8bbb840b3a414271bf7c8995e431e47d8408e registry.k8s.io/e2e-test-images/jessie-dnsutils:1.7],SizeBytes:116764683,},ContainerImage{Names:[registry.k8s.io/kube-proxy-arm64:v1.27.0-alpha.3.52_d48b8167f7094b],SizeBytes:60554082,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:16bbf38c463a4223d8cfe4da12bc61010b082a79b4bb003e2d3ba3ece5dd5f9e registry.k8s.io/e2e-test-images/agnhost:2.43],SizeBytes:49808364,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:3fe7acf013d1264ffded116b80a73dc129a449b0fccdb8d21af8279f2233f36e registry.k8s.io/e2e-test-images/httpd:2.4.39-4],SizeBytes:41528818,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:148b022f5c5da426fc2f3c14b5c0867e58ef05961510c84749ac1fddcb0fef22 registry.k8s.io/e2e-test-images/httpd:2.4.38-4],SizeBytes:40343601,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:28932904,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:ee3b525d5b89db99da3b8eb521d9cd90cb6e9ef0fbb651e98bb37be78d36b5b8 registry.k8s.io/sig-storage/csi-provisioner:v3.3.0],SizeBytes:23945474,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:425d8f1b769398127767b06ed97ce62578a3179bcb99809ce93a1649e025ffe7 registry.k8s.io/sig-storage/csi-resizer:v1.6.0],SizeBytes:22678632,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:291334908ddf71a4661fd7f6d9d97274de8a5378a2b6fdfeb2ce73414a34f82f registry.k8s.io/sig-storage/csi-snapshotter:v6.1.0],SizeBytes:22434180,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:9a685020911e2725ad019dbce6e4a5ab93d51e3d4557f115e64343345e05781b registry.k8s.io/sig-storage/csi-attacher:v4.0.0],SizeBytes:22394501,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:122bfb8c1edabb3c0edd63f06523e6940d958d19b3957dc7b1d6f81e9f1f6119 registry.k8s.io/sig-storage/csi-provisioner:v3.1.0],SizeBytes:21735644,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:9ebbf9f023e7b41ccee3d52afe39a89e3ddacdbb69269d583abfc25847cfd9e4 registry.k8s.io/sig-storage/csi-resizer:v1.4.0],SizeBytes:20848413,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:8b9c313c05f54fb04f8d430896f5f5904b6cb157df261501b29adc04d2b2dc7b registry.k8s.io/sig-storage/csi-attacher:v3.4.0],SizeBytes:20565480,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nonroot@sha256:ee9f50b3c64b174d296d91ca9f69a914ac30e59095dfb462b2b518ad28a63655 registry.k8s.io/e2e-test-images/nonroot:1.4],SizeBytes:18561972,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:92257881c1d6493cf18299a24af42330f891166560047902b8d431fb66b01af5 registry.k8s.io/sig-storage/hostpathplugin:v1.9.0],SizeBytes:18267611,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:9b2d9abda017c28b12f84a344f57cd73fbdb6c2bd7dd5bdd5018246ad1093ba6 registry.k8s.io/sig-storage/hostpathplugin:v1.11.0],SizeBytes:18159085,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:017727efcfeb7d053af68e51436ce8e65edbc6ca573720afb4f79c8594036955 registry.k8s.io/coredns/coredns:v1.10.0],SizeBytes:13766234,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:8502345,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:933940f13b3ea0abc62e656c1aa5c5b47c04b15d71250413a6b821bd0c58b94e registry.k8s.io/sig-storage/livenessprobe:v2.7.0],SizeBytes:8223819,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:7685305,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:5c99cf6a02adda929b10321dbf4ecfa00d87be9ba4fb456006237d530ab4baa1 registry.k8s.io/e2e-test-images/nginx:1.14-4],SizeBytes:6860089,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nonewprivs@sha256:8ac1264691820febacf3aea5d152cbde6d10685731ec14966a9401c6f47a68ac registry.k8s.io/e2e-test-images/nonewprivs:1.3],SizeBytes:5025027,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:2e0f836850e09b8b7cc937681d6194537a09fbd5f6b9e08f4d646a85128e8937 registry.k8s.io/e2e-test-images/busybox:1.29-4],SizeBytes:772933,},ContainerImage{Names:[registry.k8s.io/pause@sha256:7031c1b283388d2c2e09b57badb803c05ebed362dc88d84b480cc47f72a21097 registry.k8s.io/pause:3.9],SizeBytes:268051,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:253553,},},VolumesInUse:[kubernetes.io/csi/csi-hostpath-ephemeral-2147^52df5953-bb73-11ed-8aa7-be49170871b4 kubernetes.io/csi/csi-hostpath-ephemeral-5408^443d1daf-bb73-11ed-95c4-5a95d17cbc27 kubernetes.io/csi/csi-hostpath-ephemeral-5408^44411f51-bb73-11ed-95c4-5a95d17cbc27 kubernetes.io/csi/csi-hostpath-ephemeral-6540^4426a07f-bb73-11ed-a604-92687f2c50f8 kubernetes.io/csi/ebs.csi.aws.com^vol-05b67fab5beb6394b],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-05b67fab5beb6394b,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/csi-hostpath-ephemeral-6540^4426a07f-bb73-11ed-a604-92687f2c50f8,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/csi-hostpath-ephemeral-5408^44411f51-bb73-11ed-95c4-5a95d17cbc27,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/csi-hostpath-ephemeral-5408^443d1daf-bb73-11ed-95c4-5a95d17cbc27,DevicePath:,},},Config:nil,},} Mar 5 16:46:19.254: INFO: Logging kubelet events for node i-06df568f6e5c59fff Mar 5 16:46:19.371: INFO: Logging pods the kubelet thinks is on node i-06df568f6e5c59fff Mar 5 16:47:03.076: INFO: Unable to retrieve kubelet pods for node i-06df568f6e5c59fff: error trying to reach service: http2: client connection lost Mar 5 16:47:03.076: INFO: Logging node info for node i-0d6a32f749839192f Mar 5 16:47:03.190: INFO: Node Info: &Node{ObjectMeta:{i-0d6a32f749839192f 4155fefb-4d3a-4b04-ba53-6ac9e895ff92 16414 0 2023-03-05 16:20:46 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:arm64 beta.kubernetes.io/instance-type:c6g.xlarge beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:eu-central-1 failure-domain.beta.kubernetes.io/zone:eu-central-1a kops.k8s.io/kops-controller-pki: kubernetes.io/arch:arm64 kubernetes.io/hostname:i-0d6a32f749839192f kubernetes.io/os:linux node-role.kubernetes.io/control-plane: node.kubernetes.io/exclude-from-external-load-balancers: node.kubernetes.io/instance-type:c6g.xlarge topology.ebs.csi.aws.com/zone:eu-central-1a topology.kubernetes.io/region:eu-central-1 topology.kubernetes.io/zone:eu-central-1a] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-0d6a32f749839192f"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kubelet Update v1 2023-03-05 16:20:46 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {protokube Update v1 2023-03-05 16:21:25 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kops.k8s.io/kops-controller-pki":{},"f:node-role.kubernetes.io/control-plane":{},"f:node.kubernetes.io/exclude-from-external-load-balancers":{}}}} } {kube-controller-manager Update v1 2023-03-05 16:21:47 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.0.0/24\"":{}}}} } {aws-cloud-controller-manager Update v1 2023-03-05 16:21:48 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{},"f:taints":{}}} } {aws-cloud-controller-manager Update v1 2023-03-05 16:21:48 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kubelet Update v1 2023-03-05 16:44:12 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.0.0/24,DoNotUseExternalID:,ProviderID:aws:///eu-central-1a/i-0d6a32f749839192f,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/control-plane,Value:,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[100.96.0.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{4 0} {<nil>} 4 DecimalSI},ephemeral-storage: {{49756430336 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-32Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-64Ki: {{0 0} {<nil>} 0 DecimalSI},memory: {{8150118400 0} {<nil>} 7959100Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{4 0} {<nil>} 4 DecimalSI},ephemeral-storage: {{44780787229 0} {<nil>} 44780787229 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-32Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-64Ki: {{0 0} {<nil>} 0 DecimalSI},memory: {{8045260800 0} {<nil>} 7856700Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-03-05 16:44:12 +0000 UTC,LastTransitionTime:2023-03-05 16:20:46 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-03-05 16:44:12 +0000 UTC,LastTransitionTime:2023-03-05 16:20:46 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-03-05 16:44:12 +0000 UTC,LastTransitionTime:2023-03-05 16:20:46 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-03-05 16:44:12 +0000 UTC,LastTransitionTime:2023-03-05 16:21:47 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.49.247,},NodeAddress{Type:ExternalIP,Address:18.197.20.82,},NodeAddress{Type:InternalDNS,Address:i-0d6a32f749839192f.eu-central-1.compute.internal,},NodeAddress{Type:Hostname,Address:i-0d6a32f749839192f.eu-central-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-18-197-20-82.eu-central-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2bed249e8ac7b517cb08ccd42b83e5,SystemUUID:ec2bed24-9e8a-c7b5-17cb-08ccd42b83e5,BootID:6f9215c9-dfe8-4b13-84f4-9b2084456198,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 22.04.2 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.27.0-alpha.3.52+d48b8167f7094b,KubeProxyVersion:v1.27.0-alpha.3.52+d48b8167f7094b,OperatingSystem:linux,Architecture:arm64,},Images:[]ContainerImage{ContainerImage{Names:[registry.k8s.io/etcd@sha256:9ce33ba33d8e738a5b85ed50b5080ac746deceed4a7496c550927a7a19ca3b6d registry.k8s.io/etcd:3.5.0-0],SizeBytes:157797353,},ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:157636062,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:4afb99b4690b418ffc2ceb67e1a17376457e441c1f09ab55447f0aaf992fa646 registry.k8s.io/etcd:3.4.3-0],SizeBytes:151677080,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:4ad90a11b55313b182afc186b9876c8e891531b8db4c9bf1541953021618d0e2 registry.k8s.io/etcd:3.4.13-0],SizeBytes:134531559,},ContainerImage{Names:[registry.k8s.io/kube-apiserver-arm64:v1.27.0-alpha.3.52_d48b8167f7094b],SizeBytes:131087707,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:0d2ab6b15b4fe0b903b9e81192d942fb39c29be61fe341cd16db5d0165b0435c registry.k8s.io/etcd:3.3.17-0],SizeBytes:126979107,},ContainerImage{Names:[registry.k8s.io/kube-controller-manager-arm64:v1.27.0-alpha.3.52_d48b8167f7094b],SizeBytes:120667128,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:17da501f5d2a675be46040422a27b7cc21b8a43895ac998b171db1c346f361f7 registry.k8s.io/etcd:3.3.10-0],SizeBytes:81436097,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:6f72b851544986cb0921b53ea655ec04c36131248f16d4ad110cb3ca0c369dc1 registry.k8s.io/etcd:3.5.4-0],SizeBytes:81120118,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:13f53ed1d91e2e11aac476ee9a0269fdda6cc4874eba903efd40daf50c55eee5 registry.k8s.io/etcd:3.5.3-0],SizeBytes:81114891,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:51eae8381dcb1078289fa7b4f3df2630cdc18d09fb56f8e56b41c40e191d6c83 registry.k8s.io/etcd:3.5.7-0],SizeBytes:80665728,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:dd75ec974b0a2a6f6bb47001ba09207976e625db898d1b16735528c009cb171c registry.k8s.io/etcd:3.5.6-0],SizeBytes:80539316,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:905d7ca17fd02bc24c0eba9a062753aba15db3e31422390bc3238eb762339b20 registry.k8s.io/etcd:3.2.24-1],SizeBytes:61500433,},ContainerImage{Names:[registry.k8s.io/etcdadm/etcd-manager-slim@sha256:8727adc059334e0e3329dbccce116a734da4a090da5e86c8cddccaf333caa666 registry.k8s.io/etcdadm/etcd-manager-slim:v3.0.20230201],SizeBytes:60826655,},ContainerImage{Names:[registry.k8s.io/kube-proxy-arm64:v1.27.0-alpha.3.52_d48b8167f7094b],SizeBytes:60554082,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:64b9ea357325d5db9f8a723dcf503b5a449177b17ac87d69481e126bb724c263 registry.k8s.io/etcd:3.5.1-0],SizeBytes:59553904,},ContainerImage{Names:[registry.k8s.io/kube-scheduler-arm64:v1.27.0-alpha.3.52_d48b8167f7094b],SizeBytes:56310763,},ContainerImage{Names:[registry.k8s.io/kops/kops-controller:1.27.0-alpha.1],SizeBytes:41797411,},ContainerImage{Names:[registry.k8s.io/kops/dns-controller:1.27.0-alpha.1],SizeBytes:40267473,},ContainerImage{Names:[public.ecr.aws/karpenter/controller@sha256:68db4f092cf9cc83f5ef9e2fbc5407c2cb682e81f64dfaa700a7602ede38b1cf public.ecr.aws/karpenter/controller:v0.16.3],SizeBytes:33707751,},ContainerImage{Names:[public.ecr.aws/karpenter/webhook@sha256:96a2d9b06d6bc5127801f358f74b1cf2d289b423a2e9ba40c573c0b14b17dafa public.ecr.aws/karpenter/webhook:v0.16.3],SizeBytes:33307187,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:28932904,},ContainerImage{Names:[quay.io/cilium/operator@sha256:a6d24a006a6b92967ac90786b49bc1ac26e5477cf028cd1186efcfc2466484db quay.io/cilium/operator:v1.12.5],SizeBytes:24567287,},ContainerImage{Names:[registry.k8s.io/provider-aws/cloud-controller-manager@sha256:fdeb61e3e42ecd9cca868d550ebdb88dd6341d9e91fcfa9a37e227dab2ad22cb registry.k8s.io/provider-aws/cloud-controller-manager:v1.26.0],SizeBytes:18345226,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:8502345,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:7685305,},ContainerImage{Names:[registry.k8s.io/kops/kube-apiserver-healthcheck:1.27.0-alpha.1],SizeBytes:4713612,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:253553,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Mar 5 16:47:03.191: INFO: Logging kubelet events for node i-0d6a32f749839192f Mar 5 16:47:03.307: INFO: ... skipping 50 lines ... Mar 5 16:47:03.428: INFO: Container cilium-agent ready: true, restart count 1 Mar 5 16:47:03.428: INFO: karpenter-webhook-757f994bb9-4shcm started at 2023-03-05 16:21:48 +0000 UTC (0+1 container statuses recorded) Mar 5 16:47:03.428: INFO: Container webhook ready: true, restart count 0 Mar 5 16:47:03.928: INFO: Latency metrics for node i-0d6a32f749839192f [1mSTEP:[0m Waiting for namespaces [csi-mock-volumes-workload-2219] to vanish [38;5;243m@ 03/05/23 16:47:04.044[0m Mar 5 16:52:04.274: INFO: error deleting namespace csi-mock-volumes-workload-2219: timed out waiting for the condition [1mSTEP:[0m uninstalling csi mock driver [38;5;243m@ 03/05/23 16:52:04.275[0m [1mSTEP:[0m deleting the driver namespace: csi-mock-volumes-workload-2219-9598 [38;5;243m@ 03/05/23 16:52:04.275[0m [1mSTEP:[0m Collecting events from namespace "csi-mock-volumes-workload-2219-9598". [38;5;243m@ 03/05/23 16:52:04.275[0m [1mSTEP:[0m Found 22 events. [38;5;243m@ 03/05/23 16:52:04.389[0m Mar 5 16:52:04.389: INFO: At 2023-03-05 16:31:07 +0000 UTC - event for csi-mockplugin: {statefulset-controller } SuccessfulCreate: create Pod csi-mockplugin-0 in StatefulSet csi-mockplugin successful Mar 5 16:52:04.389: INFO: At 2023-03-05 16:31:07 +0000 UTC - event for csi-mockplugin-0: {default-scheduler } Scheduled: Successfully assigned csi-mock-volumes-workload-2219-9598/csi-mockplugin-0 to i-06df568f6e5c59fff ... skipping 18 lines ... Mar 5 16:52:04.389: INFO: At 2023-03-05 16:41:51 +0000 UTC - event for csi-mockplugin-0: {taint-controller } TaintManagerEviction: Cancelling deletion of Pod csi-mock-volumes-workload-2219-9598/csi-mockplugin-0 Mar 5 16:52:04.389: INFO: At 2023-03-05 16:41:51 +0000 UTC - event for csi-mockplugin-attacher-0: {taint-controller } TaintManagerEviction: Cancelling deletion of Pod csi-mock-volumes-workload-2219-9598/csi-mockplugin-attacher-0 Mar 5 16:52:04.502: INFO: POD NODE PHASE GRACE CONDITIONS Mar 5 16:52:04.502: INFO: csi-mockplugin-0 i-06df568f6e5c59fff Running 30s [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2023-03-05 16:31:07 +0000 UTC } {Ready False 0001-01-01 00:00:00 +0000 UTC 2023-03-05 16:33:21 +0000 UTC } {ContainersReady True 0001-01-01 00:00:00 +0000 UTC 2023-03-05 16:31:09 +0000 UTC } {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2023-03-05 16:31:07 +0000 UTC } {DisruptionTarget True 0001-01-01 00:00:00 +0000 UTC 2023-03-05 16:47:57 +0000 UTC DeletionByTaintManager Taint manager: deleting due to NoExecute taint}] Mar 5 16:52:04.502: INFO: csi-mockplugin-attacher-0 i-06df568f6e5c59fff Running 30s [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2023-03-05 16:31:07 +0000 UTC } {Ready False 0001-01-01 00:00:00 +0000 UTC 2023-03-05 16:33:20 +0000 UTC } {ContainersReady True 0001-01-01 00:00:00 +0000 UTC 2023-03-05 16:31:09 +0000 UTC } {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2023-03-05 16:31:07 +0000 UTC } {DisruptionTarget True 0001-01-01 00:00:00 +0000 UTC 2023-03-05 16:47:57 +0000 UTC DeletionByTaintManager Taint manager: deleting due to NoExecute taint}] Mar 5 16:52:04.502: INFO: Mar 5 16:52:14.618: INFO: Unable to fetch csi-mock-volumes-workload-2219-9598/csi-mockplugin-0/csi-provisioner logs: an error on the server ("unknown") has prevented the request from succeeding (get pods csi-mockplugin-0) Mar 5 16:52:24.736: INFO: Unable to fetch csi-mock-volumes-workload-2219-9598/csi-mockplugin-0/driver-registrar logs: an error on the server ("unknown") has prevented the request from succeeding (get pods csi-mockplugin-0) Mar 5 16:52:34.852: INFO: Unable to fetch csi-mock-volumes-workload-2219-9598/csi-mockplugin-0/mock logs: an error on the server ("unknown") has prevented the request from succeeding (get pods csi-mockplugin-0) Mar 5 16:52:44.967: INFO: Unable to fetch csi-mock-volumes-workload-2219-9598/csi-mockplugin-attacher-0/csi-attacher logs: an error on the server ("unknown") has prevented the request from succeeding (get pods csi-mockplugin-attacher-0) Mar 5 16:52:45.