ci-kubernetes-e2e-gci-gce-serial #1620106158725402624 Results

Test Failures

Kubernetes e2e suite [It] [sig-cloud-provider-gcp] Restart [Disruptive] should restart all nodes and ensure all nodes and pods recover 7m15s

go run hack/e2e.go -v --test --test_args='--ginkgo.focus=Kubernetes\se2e\ssuite\s\[It\]\s\[sig\-cloud\-provider\-gcp\]\sRestart\s\[Disruptive\]\sshould\srestart\sall\snodes\sand\sensure\sall\snodes\sand\spods\srecover$'

[FAILED] couldn't find 25 pods within 5m0s; last error: expected to find 25 pods but found only 24
In [It] at: test/e2e/cloud/gcp/restart.go:115 @ 01/30/23 20:33:49.945

				> Enter [BeforeEach] [sig-cloud-provider-gcp] Restart [Disruptive] - set up framework | framework.go:188 @ 01/30/23 20:26:35.831
STEP: Creating a kubernetes client - test/e2e/framework/framework.go:208 @ 01/30/23 20:26:35.831
Jan 30 20:26:35.831: INFO: >>> kubeConfig: /workspace/.kube/config
STEP: Building a namespace api object, basename restart - test/e2e/framework/framework.go:247 @ 01/30/23 20:26:35.833
STEP: Waiting for a default service account to be provisioned in namespace - test/e2e/framework/framework.go:256 @ 01/30/23 20:26:35.959
STEP: Waiting for kube-root-ca.crt to be provisioned in namespace - test/e2e/framework/framework.go:259 @ 01/30/23 20:26:36.041
< Exit [BeforeEach] [sig-cloud-provider-gcp] Restart [Disruptive] - set up framework | framework.go:188 @ 01/30/23 20:26:36.125 (294ms)
> Enter [BeforeEach] [sig-cloud-provider-gcp] Restart [Disruptive] - test/e2e/framework/metrics/init/init.go:33 @ 01/30/23 20:26:36.125
< Exit [BeforeEach] [sig-cloud-provider-gcp] Restart [Disruptive] - test/e2e/framework/metrics/init/init.go:33 @ 01/30/23 20:26:36.125 (0s)
> Enter [BeforeEach] [sig-cloud-provider-gcp] Restart [Disruptive] - test/e2e/cloud/gcp/restart.go:55 @ 01/30/23 20:26:36.125
STEP: ensuring all nodes are ready - test/e2e/cloud/gcp/restart.go:66 @ 01/30/23 20:26:36.22
Jan 30 20:26:36.264: INFO: Cluster has reached the desired number of ready nodes 3
Jan 30 20:26:36.264: INFO: Got the following nodes before restart: [bootstrap-e2e-minion-group-5sqf bootstrap-e2e-minion-group-gr7w bootstrap-e2e-minion-group-hwhs]
STEP: ensuring all pods are running and ready - test/e2e/cloud/gcp/restart.go:71 @ 01/30/23 20:26:36.264
Jan 30 20:26:36.264: INFO: Waiting up to 5m0s for 25 pods to be running and ready, or succeeded: [kube-scheduler-bootstrap-e2e-master metadata-proxy-v0.1-g24sg volume-snapshot-controller-0 konnectivity-agent-9r6m6 konnectivity-agent-m7rcz konnectivity-server-bootstrap-e2e-master kube-controller-manager-bootstrap-e2e-master l7-default-backend-8549d69d99-xzb2m l7-lb-controller-bootstrap-e2e-master coredns-6846b5b5f-zwbrf etcd-server-bootstrap-e2e-master kube-apiserver-bootstrap-e2e-master metrics-server-v0.5.2-867b8754b9-dmjcn coredns-6846b5b5f-bkw98 konnectivity-agent-pb855 kube-addon-manager-bootstrap-e2e-master kube-dns-autoscaler-5f6455f985-7jdxx kube-proxy-bootstrap-e2e-minion-group-5sqf kube-proxy-bootstrap-e2e-minion-group-gr7w kube-proxy-bootstrap-e2e-minion-group-hwhs metadata-proxy-v0.1-g6h7m coredns-6846b5b5f-zkslx etcd-server-events-bootstrap-e2e-master metadata-proxy-v0.1-j8mg4 metadata-proxy-v0.1-tp2qz]
Jan 30 20:26:36.264: INFO: Waiting up to 5m0s for pod "metadata-proxy-v0.1-tp2qz" in namespace "kube-system" to be "running and ready, or succeeded"
Jan 30 20:26:36.264: INFO: Waiting up to 5m0s for pod "coredns-6846b5b5f-zwbrf" in namespace "kube-system" to be "running and ready, or succeeded"
Jan 30 20:26:36.264: INFO: Waiting up to 5m0s for pod "konnectivity-agent-m7rcz" in namespace "kube-system" to be "running and ready, or succeeded"
Jan 30 20:26:36.264: INFO: Waiting up to 5m0s for pod "kube-controller-manager-bootstrap-e2e-master" in namespace "kube-system" to be "running and ready, or succeeded"
Jan 30 20:26:36.264: INFO: Waiting up to 5m0s for pod "metadata-proxy-v0.1-g24sg" in namespace "kube-system" to be "running and ready, or succeeded"
Jan 30 20:26:36.264: INFO: Waiting up to 5m0s for pod "l7-default-backend-8549d69d99-xzb2m" in namespace "kube-system" to be "running and ready, or succeeded"
Jan 30 20:26:36.264: INFO: Waiting up to 5m0s for pod "volume-snapshot-controller-0" in namespace "kube-system" to be "running and ready, or succeeded"
Jan 30 20:26:36.264: INFO: Waiting up to 5m0s for pod "konnectivity-server-bootstrap-e2e-master" in namespace "kube-system" to be "running and ready, or succeeded"
Jan 30 20:26:36.265: INFO: Waiting up to 5m0s for pod "kube-scheduler-bootstrap-e2e-master" in namespace "kube-system" to be "running and ready, or succeeded"
Jan 30 20:26:36.265: INFO: Waiting up to 5m0s for pod "l7-lb-controller-bootstrap-e2e-master" in namespace "kube-system" to be "running and ready, or succeeded"
Jan 30 20:26:36.265: INFO: Waiting up to 5m0s for pod "konnectivity-agent-9r6m6" in namespace "kube-system" to be "running and ready, or succeeded"
Jan 30 20:26:36.265: INFO: Waiting up to 5m0s for pod "metrics-server-v0.5.2-867b8754b9-dmjcn" in namespace "kube-system" to be "running and ready, or succeeded"
Jan 30 20:26:36.265: INFO: Waiting up to 5m0s for pod "kube-apiserver-bootstrap-e2e-master" in namespace "kube-system" to be "running and ready, or succeeded"
Jan 30 20:26:36.265: INFO: Waiting up to 5m0s for pod "etcd-server-bootstrap-e2e-master" in namespace "kube-system" to be "running and ready, or succeeded"
Jan 30 20:26:36.265: INFO: Waiting up to 5m0s for pod "konnectivity-agent-pb855" in namespace "kube-system" to be "running and ready, or succeeded"
Jan 30 20:26:36.265: INFO: Waiting up to 5m0s for pod "kube-dns-autoscaler-5f6455f985-7jdxx" in namespace "kube-system" to be "running and ready, or succeeded"
Jan 30 20:26:36.265: INFO: Waiting up to 5m0s for pod "kube-addon-manager-bootstrap-e2e-master" in namespace "kube-system" to be "running and ready, or succeeded"
Jan 30 20:26:36.265: INFO: Waiting up to 5m0s for pod "coredns-6846b5b5f-bkw98" in namespace "kube-system" to be "running and ready, or succeeded"
Jan 30 20:26:36.265: INFO: Waiting up to 5m0s for pod "metadata-proxy-v0.1-g6h7m" in namespace "kube-system" to be "running and ready, or succeeded"
Jan 30 20:26:36.265: INFO: Waiting up to 5m0s for pod "kube-proxy-bootstrap-e2e-minion-group-gr7w" in namespace "kube-system" to be "running and ready, or succeeded"
Jan 30 20:26:36.265: INFO: Waiting up to 5m0s for pod "coredns-6846b5b5f-zkslx" in namespace "kube-system" to be "running and ready, or succeeded"
Jan 30 20:26:36.265: INFO: Waiting up to 5m0s for pod "kube-proxy-bootstrap-e2e-minion-group-hwhs" in namespace "kube-system" to be "running and ready, or succeeded"
Jan 30 20:26:36.265: INFO: Waiting up to 5m0s for pod "etcd-server-events-bootstrap-e2e-master" in namespace "kube-system" to be "running and ready, or succeeded"
Jan 30 20:26:36.265: INFO: Waiting up to 5m0s for pod "kube-proxy-bootstrap-e2e-minion-group-5sqf" in namespace "kube-system" to be "running and ready, or succeeded"
Jan 30 20:26:36.265: INFO: Waiting up to 5m0s for pod "metadata-proxy-v0.1-j8mg4" in namespace "kube-system" to be "running and ready, or succeeded"
Jan 30 20:26:36.310: INFO: Pod "l7-default-backend-8549d69d99-xzb2m": Phase="Running", Reason="", readiness=true. Elapsed: 45.652353ms
Jan 30 20:26:36.310: INFO: Pod "l7-default-backend-8549d69d99-xzb2m" satisfied condition "running and ready, or succeeded"
Jan 30 20:26:36.315: INFO: Pod "metadata-proxy-v0.1-tp2qz": Phase="Running", Reason="", readiness=true. Elapsed: 51.068739ms
Jan 30 20:26:36.315: INFO: Pod "metadata-proxy-v0.1-tp2qz" satisfied condition "running and ready, or succeeded"
Jan 30 20:26:36.315: INFO: Pod "konnectivity-agent-m7rcz": Phase="Running", Reason="", readiness=true. Elapsed: 51.076042ms
Jan 30 20:26:36.315: INFO: Pod "konnectivity-agent-m7rcz" satisfied condition "running and ready, or succeeded"
Jan 30 20:26:36.315: INFO: Pod "metadata-proxy-v0.1-g24sg": Phase="Running", Reason="", readiness=true. Elapsed: 51.040429ms
Jan 30 20:26:36.315: INFO: Pod "metadata-proxy-v0.1-g24sg" satisfied condition "running and ready, or succeeded"
Jan 30 20:26:36.316: INFO: Pod "kube-controller-manager-bootstrap-e2e-master": Phase="Running", Reason="", readiness=true. Elapsed: 51.125586ms
Jan 30 20:26:36.316: INFO: Pod "kube-controller-manager-bootstrap-e2e-master" satisfied condition "running and ready, or succeeded"
Jan 30 20:26:36.316: INFO: Pod "coredns-6846b5b5f-zwbrf": Phase="Pending", Reason="", readiness=false. Elapsed: 51.26483ms
Jan 30 20:26:36.316: INFO: Error evaluating pod condition running and ready, or succeeded: want pod 'coredns-6846b5b5f-zwbrf' on 'bootstrap-e2e-minion-group-hwhs' to be 'Running' but was 'Pending'
Jan 30 20:26:36.364: INFO: Pod "etcd-server-events-bootstrap-e2e-master": Phase="Running", Reason="", readiness=true. Elapsed: 98.842963ms
Jan 30 20:26:36.364: INFO: Pod "etcd-server-events-bootstrap-e2e-master" satisfied condition "running and ready, or succeeded"
Jan 30 20:26:36.364: INFO: Pod "volume-snapshot-controller-0": Phase="Running", Reason="", readiness=true. Elapsed: 99.077606ms
Jan 30 20:26:36.364: INFO: Pod "volume-snapshot-controller-0" satisfied condition "running and ready, or succeeded"
Jan 30 20:26:36.365: INFO: Pod "kube-dns-autoscaler-5f6455f985-7jdxx": Phase="Running", Reason="", readiness=true. Elapsed: 100.147738ms
Jan 30 20:26:36.365: INFO: Pod "kube-dns-autoscaler-5f6455f985-7jdxx" satisfied condition "running and ready, or succeeded"
Jan 30 20:26:36.370: INFO: Pod "kube-proxy-bootstrap-e2e-minion-group-5sqf": Phase="Running", Reason="", readiness=true. Elapsed: 105.110825ms
Jan 30 20:26:36.370: INFO: Pod "kube-proxy-bootstrap-e2e-minion-group-5sqf" satisfied condition "running and ready, or succeeded"
Jan 30 20:26:36.370: INFO: Pod "kube-proxy-bootstrap-e2e-minion-group-hwhs": Phase="Running", Reason="", readiness=true. Elapsed: 105.201467ms
Jan 30 20:26:36.370: INFO: Pod "kube-proxy-bootstrap-e2e-minion-group-hwhs" satisfied condition "running and ready, or succeeded"
Jan 30 20:26:36.370: INFO: Pod "metadata-proxy-v0.1-j8mg4": Phase="Running", Reason="", readiness=true. Elapsed: 105.202086ms
Jan 30 20:26:36.370: INFO: Pod "metadata-proxy-v0.1-j8mg4" satisfied condition "running and ready, or succeeded"
Jan 30 20:26:36.370: INFO: Pod "metrics-server-v0.5.2-867b8754b9-dmjcn": Phase="Running", Reason="", readiness=true. Elapsed: 105.404977ms
Jan 30 20:26:36.370: INFO: Pod "metrics-server-v0.5.2-867b8754b9-dmjcn" satisfied condition "running and ready, or succeeded"
Jan 30 20:26:36.370: INFO: Pod "kube-scheduler-bootstrap-e2e-master": Phase="Running", Reason="", readiness=true. Elapsed: 105.484552ms
Jan 30 20:26:36.370: INFO: Pod "kube-scheduler-bootstrap-e2e-master" satisfied condition "running and ready, or succeeded"
Jan 30 20:26:36.370: INFO: Pod "coredns-6846b5b5f-zkslx": Phase="Running", Reason="", readiness=true. Elapsed: 105.3485ms
Jan 30 20:26:36.370: INFO: Pod "coredns-6846b5b5f-zkslx" satisfied condition "running and ready, or succeeded"
Jan 30 20:26:36.370: INFO: Pod "konnectivity-server-bootstrap-e2e-master": Phase="Running", Reason="", readiness=true. Elapsed: 105.615831ms
Jan 30 20:26:36.370: INFO: Pod "konnectivity-server-bootstrap-e2e-master" satisfied condition "running and ready, or succeeded"
Jan 30 20:26:36.373: INFO: Pod "kube-proxy-bootstrap-e2e-minion-group-gr7w": Phase="Running", Reason="", readiness=true. Elapsed: 108.131203ms
Jan 30 20:26:36.373: INFO: Pod "kube-proxy-bootstrap-e2e-minion-group-gr7w" satisfied condition "running and ready, or succeeded"
Jan 30 20:26:36.373: INFO: Pod "konnectivity-agent-pb855": Phase="Running", Reason="", readiness=true. Elapsed: 108.257416ms
Jan 30 20:26:36.373: INFO: Pod "konnectivity-agent-pb855" satisfied condition "running and ready, or succeeded"
Jan 30 20:26:36.373: INFO: Pod "metadata-proxy-v0.1-g6h7m": Phase="Running", Reason="", readiness=true. Elapsed: 108.221554ms
Jan 30 20:26:36.373: INFO: Pod "metadata-proxy-v0.1-g6h7m" satisfied condition "running and ready, or succeeded"
Jan 30 20:26:36.373: INFO: Pod "coredns-6846b5b5f-bkw98": Phase="Running", Reason="", readiness=true. Elapsed: 108.261592ms
Jan 30 20:26:36.373: INFO: Pod "coredns-6846b5b5f-bkw98" satisfied condition "running and ready, or succeeded"
Jan 30 20:26:36.373: INFO: Pod "l7-lb-controller-bootstrap-e2e-master": Phase="Running", Reason="", readiness=true. Elapsed: 108.388434ms
Jan 30 20:26:36.373: INFO: Pod "etcd-server-bootstrap-e2e-master": Phase="Running", Reason="", readiness=true. Elapsed: 108.33426ms
Jan 30 20:26:36.373: INFO: Pod "l7-lb-controller-bootstrap-e2e-master" satisfied condition "running and ready, or succeeded"
Jan 30 20:26:36.373: INFO: Pod "etcd-server-bootstrap-e2e-master" satisfied condition "running and ready, or succeeded"
Jan 30 20:26:36.373: INFO: Pod "kube-addon-manager-bootstrap-e2e-master": Phase="Running", Reason="", readiness=true. Elapsed: 108.320757ms
Jan 30 20:26:36.373: INFO: Pod "kube-addon-manager-bootstrap-e2e-master" satisfied condition "running and ready, or succeeded"
Jan 30 20:26:36.373: INFO: Pod "konnectivity-agent-9r6m6": Phase="Running", Reason="", readiness=true. Elapsed: 108.390517ms
Jan 30 20:26:36.373: INFO: Pod "konnectivity-agent-9r6m6" satisfied condition "running and ready, or succeeded"
Jan 30 20:26:36.373: INFO: Pod "kube-apiserver-bootstrap-e2e-master": Phase="Running", Reason="", readiness=true. Elapsed: 108.40052ms
Jan 30 20:26:36.373: INFO: Pod "kube-apiserver-bootstrap-e2e-master" satisfied condition "running and ready, or succeeded"
Jan 30 20:26:38.360: INFO: Pod "coredns-6846b5b5f-zwbrf": Phase="Running", Reason="", readiness=true. Elapsed: 2.095349051s
Jan 30 20:26:38.360: INFO: Pod "coredns-6846b5b5f-zwbrf" satisfied condition "running and ready, or succeeded"
Jan 30 20:26:38.360: INFO: Wanted all 25 pods to be running and ready, or succeeded. Result: true. Pods: [kube-scheduler-bootstrap-e2e-master metadata-proxy-v0.1-g24sg volume-snapshot-controller-0 konnectivity-agent-9r6m6 konnectivity-agent-m7rcz konnectivity-server-bootstrap-e2e-master kube-controller-manager-bootstrap-e2e-master l7-default-backend-8549d69d99-xzb2m l7-lb-controller-bootstrap-e2e-master coredns-6846b5b5f-zwbrf etcd-server-bootstrap-e2e-master kube-apiserver-bootstrap-e2e-master metrics-server-v0.5.2-867b8754b9-dmjcn coredns-6846b5b5f-bkw98 konnectivity-agent-pb855 kube-addon-manager-bootstrap-e2e-master kube-dns-autoscaler-5f6455f985-7jdxx kube-proxy-bootstrap-e2e-minion-group-5sqf kube-proxy-bootstrap-e2e-minion-group-gr7w kube-proxy-bootstrap-e2e-minion-group-hwhs metadata-proxy-v0.1-g6h7m coredns-6846b5b5f-zkslx etcd-server-events-bootstrap-e2e-master metadata-proxy-v0.1-j8mg4 metadata-proxy-v0.1-tp2qz]
< Exit [BeforeEach] [sig-cloud-provider-gcp] Restart [Disruptive] - test/e2e/cloud/gcp/restart.go:55 @ 01/30/23 20:26:38.36 (2.234s)
> Enter [It] should restart all nodes and ensure all nodes and pods recover - test/e2e/cloud/gcp/restart.go:91 @ 01/30/23 20:26:38.36
STEP: restarting all of the nodes - test/e2e/cloud/gcp/restart.go:92 @ 01/30/23 20:26:38.36
Jan 30 20:26:38.360: INFO: Running gcloud [compute --project=k8s-infra-e2e-boskos-080 instances reset bootstrap-e2e-minion-group-5sqf bootstrap-e2e-minion-group-gr7w bootstrap-e2e-minion-group-hwhs --zone=us-west1-b]
STEP: ensuring all nodes are ready after the restart - test/e2e/cloud/gcp/restart.go:96 @ 01/30/23 20:28:49.899
Jan 30 20:28:49.943: INFO: Cluster has reached the desired number of ready nodes 3
Jan 30 20:28:49.943: INFO: Got the following nodes after restart: [bootstrap-e2e-minion-group-5sqf bootstrap-e2e-minion-group-gr7w bootstrap-e2e-minion-group-hwhs]
STEP: ensuring the same number of nodes exist after the restart - test/e2e/cloud/gcp/restart.go:103 @ 01/30/23 20:28:49.943
STEP: ensuring the same number of pods are running and ready after restart - test/e2e/cloud/gcp/restart.go:112 @ 01/30/23 20:28:49.943
Jan 30 20:28:51.944: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:28:53.945: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:28:55.945: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:28:57.944: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:28:59.945: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:29:01.944: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:29:03.945: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:29:05.944: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:29:07.944: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:29:09.945: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:29:11.944: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:29:13.945: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:29:15.945: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:29:17.945: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:29:19.945: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:29:21.944: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:29:23.945: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:29:25.945: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:29:27.944: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:29:29.945: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:29:31.945: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:29:33.945: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:29:35.944: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:29:37.944: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:29:39.945: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:29:41.944: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:29:43.945: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:29:45.946: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:29:47.945: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:29:49.945: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:29:51.944: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:29:53.945: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:29:55.945: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:29:57.944: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:29:59.945: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:30:01.944: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:30:03.945: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:30:05.944: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:30:07.944: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:30:09.945: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:30:11.944: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:30:13.945: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:30:15.945: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:30:17.945: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:30:19.945: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:30:21.944: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:30:23.945: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:30:25.945: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:30:27.944: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:30:29.944: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:30:31.944: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:30:33.945: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:30:35.944: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:30:37.944: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:30:39.945: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:30:41.945: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:30:43.945: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:30:45.945: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:30:47.944: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:30:49.944: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:30:51.944: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:30:53.945: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:30:55.944: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:30:57.944: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:30:59.944: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:31:01.944: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:31:03.945: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:31:05.944: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:31:07.944: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:31:09.945: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:31:11.944: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:31:13.945: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:31:15.945: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:31:17.944: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:31:19.945: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:31:21.944: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:31:23.944: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:31:25.945: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:31:27.944: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:31:29.945: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:31:31.944: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:31:33.944: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:31:35.945: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:31:37.944: INFO: Error getting pods: expected to find 25 pods but found only 24
Automatically polling progress:
  [sig-cloud-provider-gcp] Restart [Disruptive] should restart all nodes and ensure all nodes and pods recover (Spec Runtime: 5m2.529s)
    test/e2e/cloud/gcp/restart.go:91
    In [It] (Node Runtime: 5m0.001s)
      test/e2e/cloud/gcp/restart.go:91
      At [By Step] ensuring the same number of pods are running and ready after restart (Step Runtime: 2m48.417s)
        test/e2e/cloud/gcp/restart.go:112

      Spec Goroutine
      goroutine 44796 [select]
        k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.waitForWithContext({0x7f2fec1df798, 0xc008507640}, 0xc00634db00, 0x302ee4a?)
          vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:647
        k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.poll({0x7f2fec1df798, 0xc008507640}, 0x44?, 0x0?, 0x0?)
          vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:600
        k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.PollWithContext({0x7f2fec1df798, 0xc008507640}, 0x0?, 0x0?, 0x0?)
          vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:468
        k8s.io/kubernetes/test/e2e/framework/pod.WaitForNRestartablePods({0x7f2fec1df798?, 0xc008507640?}, 0x0?, 0x19, 0xc005555f20?)
          test/e2e/framework/pod/wait.go:757
      > k8s.io/kubernetes/test/e2e/cloud/gcp.glob..func11.3({0x7f2fec1df798, 0xc008507640})
          test/e2e/cloud/gcp/restart.go:114
        k8s.io/kubernetes/vendor/github.com/onsi/ginkgo/v2/internal.extractBodyFunction.func2({0x8111f08?, 0xc008507640})
          vendor/github.com/onsi/ginkgo/v2/internal/node.go:452
        k8s.io/kubernetes/vendor/github.com/onsi/ginkgo/v2/internal.(*Suite).runNode.func3()
          vendor/github.com/onsi/ginkgo/v2/internal/suite.go:854
        k8s.io/kubernetes/vendor/github.com/onsi/ginkgo/v2/internal.(*Suite).runNode
          vendor/github.com/onsi/ginkgo/v2/internal/suite.go:841
Jan 30 20:31:39.944: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:31:41.945: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:31:43.945: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:31:45.945: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:31:47.944: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:31:49.945: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:31:51.944: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:31:53.945: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:31:55.945: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:31:57.945: INFO: Error getting pods: expected to find 25 pods but found only 24
Automatically polling progress:
  [sig-cloud-provider-gcp] Restart [Disruptive] should restart all nodes and ensure all nodes and pods recover (Spec Runtime: 5m22.531s)
    test/e2e/cloud/gcp/restart.go:91
    In [It] (Node Runtime: 5m20.002s)
      test/e2e/cloud/gcp/restart.go:91
      At [By Step] ensuring the same number of pods are running and ready after restart (Step Runtime: 3m8.419s)
        test/e2e/cloud/gcp/restart.go:112

      Spec Goroutine
      goroutine 44796 [select]
        k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.waitForWithContext({0x7f2fec1df798, 0xc008507640}, 0xc00634db00, 0x302ee4a?)
          vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:647
        k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.poll({0x7f2fec1df798, 0xc008507640}, 0x44?, 0x0?, 0x0?)
          vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:600
        k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.PollWithContext({0x7f2fec1df798, 0xc008507640}, 0x0?, 0x0?, 0x0?)
          vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:468
        k8s.io/kubernetes/test/e2e/framework/pod.WaitForNRestartablePods({0x7f2fec1df798?, 0xc008507640?}, 0x0?, 0x19, 0xc005555f20?)
          test/e2e/framework/pod/wait.go:757
      > k8s.io/kubernetes/test/e2e/cloud/gcp.glob..func11.3({0x7f2fec1df798, 0xc008507640})
          test/e2e/cloud/gcp/restart.go:114
        k8s.io/kubernetes/vendor/github.com/onsi/ginkgo/v2/internal.extractBodyFunction.func2({0x8111f08?, 0xc008507640})
          vendor/github.com/onsi/ginkgo/v2/internal/node.go:452
        k8s.io/kubernetes/vendor/github.com/onsi/ginkgo/v2/internal.(*Suite).runNode.func3()
          vendor/github.com/onsi/ginkgo/v2/internal/suite.go:854
        k8s.io/kubernetes/vendor/github.com/onsi/ginkgo/v2/internal.(*Suite).runNode
          vendor/github.com/onsi/ginkgo/v2/internal/suite.go:841
Jan 30 20:31:59.944: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:32:01.944: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:32:03.945: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:32:05.944: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:32:07.944: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:32:09.945: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:32:11.944: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:32:13.944: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:32:15.945: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:32:17.944: INFO: Error getting pods: expected to find 25 pods but found only 24
Automatically polling progress:
  [sig-cloud-provider-gcp] Restart [Disruptive] should restart all nodes and ensure all nodes and pods recover (Spec Runtime: 5m42.532s)
    test/e2e/cloud/gcp/restart.go:91
    In [It] (Node Runtime: 5m40.004s)
      test/e2e/cloud/gcp/restart.go:91
      At [By Step] ensuring the same number of pods are running and ready after restart (Step Runtime: 3m28.42s)
        test/e2e/cloud/gcp/restart.go:112

      Spec Goroutine
      goroutine 44796 [select]
        k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.waitForWithContext({0x7f2fec1df798, 0xc008507640}, 0xc00634db00, 0x302ee4a?)
          vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:647
        k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.poll({0x7f2fec1df798, 0xc008507640}, 0x44?, 0x0?, 0x0?)
          vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:600
        k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.PollWithContext({0x7f2fec1df798, 0xc008507640}, 0x0?, 0x0?, 0x0?)
          vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:468
        k8s.io/kubernetes/test/e2e/framework/pod.WaitForNRestartablePods({0x7f2fec1df798?, 0xc008507640?}, 0x0?, 0x19, 0xc005555f20?)
          test/e2e/framework/pod/wait.go:757
      > k8s.io/kubernetes/test/e2e/cloud/gcp.glob..func11.3({0x7f2fec1df798, 0xc008507640})
          test/e2e/cloud/gcp/restart.go:114
        k8s.io/kubernetes/vendor/github.com/onsi/ginkgo/v2/internal.extractBodyFunction.func2({0x8111f08?, 0xc008507640})
          vendor/github.com/onsi/ginkgo/v2/internal/node.go:452
        k8s.io/kubernetes/vendor/github.com/onsi/ginkgo/v2/internal.(*Suite).runNode.func3()
          vendor/github.com/onsi/ginkgo/v2/internal/suite.go:854
        k8s.io/kubernetes/vendor/github.com/onsi/ginkgo/v2/internal.(*Suite).runNode
          vendor/github.com/onsi/ginkgo/v2/internal/suite.go:841
Jan 30 20:32:19.944: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:32:21.944: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:32:23.944: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:32:25.944: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:32:27.944: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:32:29.944: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:32:31.944: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:32:33.944: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:32:35.945: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:32:37.945: INFO: Error getting pods: expected to find 25 pods but found only 24
Automatically polling progress:
  [sig-cloud-provider-gcp] Restart [Disruptive] should restart all nodes and ensure all nodes and pods recover (Spec Runtime: 6m2.535s)
    test/e2e/cloud/gcp/restart.go:91
    In [It] (Node Runtime: 6m0.006s)
      test/e2e/cloud/gcp/restart.go:91
      At [By Step] ensuring the same number of pods are running and ready after restart (Step Runtime: 3m48.423s)
        test/e2e/cloud/gcp/restart.go:112

      Spec Goroutine
      goroutine 44796 [select]
        k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.waitForWithContext({0x7f2fec1df798, 0xc008507640}, 0xc00634db00, 0x302ee4a?)
          vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:647
        k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.poll({0x7f2fec1df798, 0xc008507640}, 0x44?, 0x0?, 0x0?)
          vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:600
        k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.PollWithContext({0x7f2fec1df798, 0xc008507640}, 0x0?, 0x0?, 0x0?)
          vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:468
        k8s.io/kubernetes/test/e2e/framework/pod.WaitForNRestartablePods({0x7f2fec1df798?, 0xc008507640?}, 0x0?, 0x19, 0xc005555f20?)
          test/e2e/framework/pod/wait.go:757
      > k8s.io/kubernetes/test/e2e/cloud/gcp.glob..func11.3({0x7f2fec1df798, 0xc008507640})
          test/e2e/cloud/gcp/restart.go:114
        k8s.io/kubernetes/vendor/github.com/onsi/ginkgo/v2/internal.extractBodyFunction.func2({0x8111f08?, 0xc008507640})
          vendor/github.com/onsi/ginkgo/v2/internal/node.go:452
        k8s.io/kubernetes/vendor/github.com/onsi/ginkgo/v2/internal.(*Suite).runNode.func3()
          vendor/github.com/onsi/ginkgo/v2/internal/suite.go:854
        k8s.io/kubernetes/vendor/github.com/onsi/ginkgo/v2/internal.(*Suite).runNode
          vendor/github.com/onsi/ginkgo/v2/internal/suite.go:841
Jan 30 20:32:39.944: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:32:41.944: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:32:43.944: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:32:45.944: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:32:47.944: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:32:49.944: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:32:51.945: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:32:53.945: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:32:55.945: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:32:57.944: INFO: Error getting pods: expected to find 25 pods but found only 24
Automatically polling progress:
  [sig-cloud-provider-gcp] Restart [Disruptive] should restart all nodes and ensure all nodes and pods recover (Spec Runtime: 6m22.537s)
    test/e2e/cloud/gcp/restart.go:91
    In [It] (Node Runtime: 6m20.008s)
      test/e2e/cloud/gcp/restart.go:91
      At [By Step] ensuring the same number of pods are running and ready after restart (Step Runtime: 4m8.425s)
        test/e2e/cloud/gcp/restart.go:112

      Spec Goroutine
      goroutine 44796 [select]
        k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.waitForWithContext({0x7f2fec1df798, 0xc008507640}, 0xc00634db00, 0x302ee4a?)
          vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:647
        k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.poll({0x7f2fec1df798, 0xc008507640}, 0x44?, 0x0?, 0x0?)
          vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:600
        k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.PollWithContext({0x7f2fec1df798, 0xc008507640}, 0x0?, 0x0?, 0x0?)
          vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:468
        k8s.io/kubernetes/test/e2e/framework/pod.WaitForNRestartablePods({0x7f2fec1df798?, 0xc008507640?}, 0x0?, 0x19, 0xc005555f20?)
          test/e2e/framework/pod/wait.go:757
      > k8s.io/kubernetes/test/e2e/cloud/gcp.glob..func11.3({0x7f2fec1df798, 0xc008507640})
          test/e2e/cloud/gcp/restart.go:114
        k8s.io/kubernetes/vendor/github.com/onsi/ginkgo/v2/internal.extractBodyFunction.func2({0x8111f08?, 0xc008507640})
          vendor/github.com/onsi/ginkgo/v2/internal/node.go:452
        k8s.io/kubernetes/vendor/github.com/onsi/ginkgo/v2/internal.(*Suite).runNode.func3()
          vendor/github.com/onsi/ginkgo/v2/internal/suite.go:854
        k8s.io/kubernetes/vendor/github.com/onsi/ginkgo/v2/internal.(*Suite).runNode
          vendor/github.com/onsi/ginkgo/v2/internal/suite.go:841
Jan 30 20:32:59.945: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:33:01.945: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:33:03.945: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:33:05.944: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:33:07.944: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:33:09.945: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:33:11.944: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:33:13.944: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:33:15.945: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:33:17.945: INFO: Error getting pods: expected to find 25 pods but found only 24
Automatically polling progress:
  [sig-cloud-provider-gcp] Restart [Disruptive] should restart all nodes and ensure all nodes and pods recover (Spec Runtime: 6m42.539s)
    test/e2e/cloud/gcp/restart.go:91
    In [It] (Node Runtime: 6m40.01s)
      test/e2e/cloud/gcp/restart.go:91
      At [By Step] ensuring the same number of pods are running and ready after restart (Step Runtime: 4m28.427s)
        test/e2e/cloud/gcp/restart.go:112

      Spec Goroutine
      goroutine 44796 [select]
        k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.waitForWithContext({0x7f2fec1df798, 0xc008507640}, 0xc00634db00, 0x302ee4a?)
          vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:647
        k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.poll({0x7f2fec1df798, 0xc008507640}, 0x44?, 0x0?, 0x0?)
          vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:600
        k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.PollWithContext({0x7f2fec1df798, 0xc008507640}, 0x0?, 0x0?, 0x0?)
          vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:468
        k8s.io/kubernetes/test/e2e/framework/pod.WaitForNRestartablePods({0x7f2fec1df798?, 0xc008507640?}, 0x0?, 0x19, 0xc005555f20?)
          test/e2e/framework/pod/wait.go:757
      > k8s.io/kubernetes/test/e2e/cloud/gcp.glob..func11.3({0x7f2fec1df798, 0xc008507640})
          test/e2e/cloud/gcp/restart.go:114
        k8s.io/kubernetes/vendor/github.com/onsi/ginkgo/v2/internal.extractBodyFunction.func2({0x8111f08?, 0xc008507640})
          vendor/github.com/onsi/ginkgo/v2/internal/node.go:452
        k8s.io/kubernetes/vendor/github.com/onsi/ginkgo/v2/internal.(*Suite).runNode.func3()
          vendor/github.com/onsi/ginkgo/v2/internal/suite.go:854
        k8s.io/kubernetes/vendor/github.com/onsi/ginkgo/v2/internal.(*Suite).runNode
          vendor/github.com/onsi/ginkgo/v2/internal/suite.go:841
Jan 30 20:33:19.945: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:33:21.944: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:33:23.944: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:33:25.945: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:33:27.944: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:33:29.944: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:33:31.944: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:33:33.944: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:33:35.945: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:33:37.945: INFO: Error getting pods: expected to find 25 pods but found only 24
Automatically polling progress:
  [sig-cloud-provider-gcp] Restart [Disruptive] should restart all nodes and ensure all nodes and pods recover (Spec Runtime: 7m2.541s)
    test/e2e/cloud/gcp/restart.go:91
    In [It] (Node Runtime: 7m0.012s)
      test/e2e/cloud/gcp/restart.go:91
      At [By Step] ensuring the same number of pods are running and ready after restart (Step Runtime: 4m48.429s)
        test/e2e/cloud/gcp/restart.go:112

      Spec Goroutine
      goroutine 44796 [select]
        k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.waitForWithContext({0x7f2fec1df798, 0xc008507640}, 0xc00634db00, 0x302ee4a?)
          vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:647
        k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.poll({0x7f2fec1df798, 0xc008507640}, 0x44?, 0x0?, 0x0?)
          vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:600
        k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.PollWithContext({0x7f2fec1df798, 0xc008507640}, 0x0?, 0x0?, 0x0?)
          vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:468
        k8s.io/kubernetes/test/e2e/framework/pod.WaitForNRestartablePods({0x7f2fec1df798?, 0xc008507640?}, 0x0?, 0x19, 0xc005555f20?)
          test/e2e/framework/pod/wait.go:757
      > k8s.io/kubernetes/test/e2e/cloud/gcp.glob..func11.3({0x7f2fec1df798, 0xc008507640})
          test/e2e/cloud/gcp/restart.go:114
        k8s.io/kubernetes/vendor/github.com/onsi/ginkgo/v2/internal.extractBodyFunction.func2({0x8111f08?, 0xc008507640})
          vendor/github.com/onsi/ginkgo/v2/internal/node.go:452
        k8s.io/kubernetes/vendor/github.com/onsi/ginkgo/v2/internal.(*Suite).runNode.func3()
          vendor/github.com/onsi/ginkgo/v2/internal/suite.go:854
        k8s.io/kubernetes/vendor/github.com/onsi/ginkgo/v2/internal.(*Suite).runNode
          vendor/github.com/onsi/ginkgo/v2/internal/suite.go:841
Jan 30 20:33:39.944: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:33:41.944: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:33:43.945: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:33:45.944: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:33:47.945: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:33:49.945: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:33:49.945: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:33:49.945: INFO: Unexpected error: 
    <*errors.errorString | 0xc0014c8a80>: {
        s: "couldn't find 25 pods within 5m0s; last error: expected to find 25 pods but found only 24",
    }
[FAILED] couldn't find 25 pods within 5m0s; last error: expected to find 25 pods but found only 24
In [It] at: test/e2e/cloud/gcp/restart.go:115 @ 01/30/23 20:33:49.945
< Exit [It] should restart all nodes and ensure all nodes and pods recover - test/e2e/cloud/gcp/restart.go:91 @ 01/30/23 20:33:49.945 (7m11.585s)
> Enter [AfterEach] [sig-cloud-provider-gcp] Restart [Disruptive] - test/e2e/framework/node/init/init.go:33 @ 01/30/23 20:33:49.945
Jan 30 20:33:49.945: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready
< Exit [AfterEach] [sig-cloud-provider-gcp] Restart [Disruptive] - test/e2e/framework/node/init/init.go:33 @ 01/30/23 20:33:49.99 (45ms)
> Enter [AfterEach] [sig-cloud-provider-gcp] Restart [Disruptive] - test/e2e/cloud/gcp/restart.go:85 @ 01/30/23 20:33:49.99
< Exit [AfterEach] [sig-cloud-provider-gcp] Restart [Disruptive] - test/e2e/cloud/gcp/restart.go:85 @ 01/30/23 20:33:49.99 (0s)
> Enter [DeferCleanup (Each)] [sig-cloud-provider-gcp] Restart [Disruptive] - test/e2e/framework/metrics/init/init.go:35 @ 01/30/23 20:33:49.99
< Exit [DeferCleanup (Each)] [sig-cloud-provider-gcp] Restart [Disruptive] - test/e2e/framework/metrics/init/init.go:35 @ 01/30/23 20:33:49.99 (0s)
> Enter [DeferCleanup (Each)] [sig-cloud-provider-gcp] Restart [Disruptive] - dump namespaces | framework.go:206 @ 01/30/23 20:33:49.99
STEP: dump namespace information after failure - test/e2e/framework/framework.go:284 @ 01/30/23 20:33:49.99
STEP: Collecting events from namespace "restart-7764". - test/e2e/framework/debug/dump.go:42 @ 01/30/23 20:33:49.99
STEP: Found 0 events. - test/e2e/framework/debug/dump.go:46 @ 01/30/23 20:33:50.032
Jan 30 20:33:50.074: INFO: POD  NODE  PHASE  GRACE  CONDITIONS
Jan 30 20:33:50.074: INFO: 
Jan 30 20:33:50.117: INFO: 
Logging node info for node bootstrap-e2e-master
Jan 30 20:33:50.160: INFO: Node Info: &Node{ObjectMeta:{bootstrap-e2e-master    11f6bd22-6cdc-4ea1-9f3a-81217ad4db12 52894 0 2023-01-30 17:11:34 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:n1-standard-1 beta.kubernetes.io/os:linux cloud.google.com/metadata-proxy-ready:true failure-domain.beta.kubernetes.io/region:us-west1 failure-domain.beta.kubernetes.io/zone:us-west1-b kubernetes.io/arch:amd64 kubernetes.io/hostname:bootstrap-e2e-master kubernetes.io/os:linux node.kubernetes.io/instance-type:n1-standard-1 topology.gke.io/zone:us-west1-b topology.kubernetes.io/region:us-west1 topology.kubernetes.io/zone:us-west1-b] map[node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kubelet Update v1 2023-01-30 17:11:34 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:cloud.google.com/metadata-proxy-ready":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{},"f:unschedulable":{}}} } {kube-controller-manager Update v1 2023-01-30 17:11:45 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}}}} status} {kube-controller-manager Update v1 2023-01-30 17:11:52 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"10.64.0.0/24\"":{}},"f:taints":{}}} } {kubelet Update v1 2023-01-30 20:33:19 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:topology.gke.io/zone":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:10.64.0.0/24,DoNotUseExternalID:,ProviderID:gce://k8s-infra-e2e-boskos-080/us-west1-b/bootstrap-e2e-master,Unschedulable:true,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/master,Value:,Effect:NoSchedule,TimeAdded:<nil>,},Taint{Key:node.kubernetes.io/unschedulable,Value:,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[10.64.0.0/24],},Status:NodeStatus{Capacity:ResourceList{attachable-volumes-gce-pd: {{127 0} {<nil>} 127 DecimalSI},cpu: {{1 0} {<nil>} 1 DecimalSI},ephemeral-storage: {{16656896000 0} {<nil>} 16266500Ki BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3858378752 0} {<nil>} 3767948Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{attachable-volumes-gce-pd: {{127 0} {<nil>} 127 DecimalSI},cpu: {{1 0} {<nil>} 1 DecimalSI},ephemeral-storage: {{14991206376 0} {<nil>} 14991206376 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3596234752 0} {<nil>} 3511948Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2023-01-30 17:11:45 +0000 UTC,LastTransitionTime:2023-01-30 17:11:45 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-01-30 20:33:19 +0000 UTC,LastTransitionTime:2023-01-30 17:11:34 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-01-30 20:33:19 +0000 UTC,LastTransitionTime:2023-01-30 17:11:34 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-01-30 20:33:19 +0000 UTC,LastTransitionTime:2023-01-30 17:11:34 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-01-30 20:33:19 +0000 UTC,LastTransitionTime:2023-01-30 17:11:47 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:10.138.0.2,},NodeAddress{Type:ExternalIP,Address:34.83.218.233,},NodeAddress{Type:InternalDNS,Address:bootstrap-e2e-master.c.k8s-infra-e2e-boskos-080.internal,},NodeAddress{Type:Hostname,Address:bootstrap-e2e-master.c.k8s-infra-e2e-boskos-080.internal,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:098aa58fbfa61c6753f7d9aa561aa43a,SystemUUID:098aa58f-bfa6-1c67-53f7-d9aa561aa43a,BootID:161bcd60-e3c0-44ef-b398-9c12c675a9b3,KernelVersion:5.10.123+,OSImage:Container-Optimized OS from Google,ContainerRuntimeVersion:containerd://1.6.6,KubeletVersion:v1.27.0-alpha.1.90+c829397f7a1fa9,KubeProxyVersion:v1.27.0-alpha.1.90+c829397f7a1fa9,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[registry.k8s.io/kube-apiserver-amd64:v1.27.0-alpha.1.90_c829397f7a1fa9],SizeBytes:135973331,},ContainerImage{Names:[registry.k8s.io/kube-controller-manager-amd64:v1.27.0-alpha.1.90_c829397f7a1fa9],SizeBytes:125279545,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:51eae8381dcb1078289fa7b4f3df2630cdc18d09fb56f8e56b41c40e191d6c83 registry.k8s.io/etcd:3.5.7-0],SizeBytes:101639218,},ContainerImage{Names:[registry.k8s.io/cloud-provider-gcp/gcp-compute-persistent-disk-csi-driver@sha256:e9c355dd1ce57de91a0f9164bd2fe688938723e7f9bd40bb13652d603f41d25a registry.k8s.io/cloud-provider-gcp/gcp-compute-persistent-disk-csi-driver:v1.2.2],SizeBytes:73354282,},ContainerImage{Names:[registry.k8s.io/kube-scheduler-amd64:v1.27.0-alpha.1.90_c829397f7a1fa9],SizeBytes:57551672,},ContainerImage{Names:[gke.gcr.io/prometheus-to-sd@sha256:e739643c3939ba0b161425f45a1989eedfc4a3b166db9a7100863296b4c70510 gke.gcr.io/prometheus-to-sd:v0.11.1-gke.1],SizeBytes:48742566,},ContainerImage{Names:[gcr.io/k8s-ingress-image-push/ingress-gce-glbc-amd64@sha256:5db27383add6d9f4ebdf0286409ac31f7f5d273690204b341a4e37998917693b gcr.io/k8s-ingress-image-push/ingress-gce-glbc-amd64:v1.20.1],SizeBytes:36598135,},ContainerImage{Names:[registry.k8s.io/addon-manager/kube-addon-manager@sha256:49cc4e6e4a3745b427ce14b0141476ab339bb65c6bc05033019e046c8727dcb0 registry.k8s.io/addon-manager/kube-addon-manager:v9.1.6],SizeBytes:30464183,},ContainerImage{Names:[registry.k8s.io/kas-network-proxy/proxy-server@sha256:b1389e7014425a1752aac55f5043ef4c52edaef0e223bf4d48ed1324e298087c registry.k8s.io/kas-network-proxy/proxy-server:v0.1.1],SizeBytes:21875112,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/metadata-proxy@sha256:e914645f22e946bce5165737e1b244e0a296ad1f0f81a9531adc57af2780978a registry.k8s.io/metadata-proxy:v0.1.12],SizeBytes:5301657,},ContainerImage{Names:[registry.k8s.io/pause@sha256:7031c1b283388d2c2e09b57badb803c05ebed362dc88d84b480cc47f72a21097 registry.k8s.io/pause:3.9],SizeBytes:321520,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},}
Jan 30 20:33:50.160: INFO: 
Logging kubelet events for node bootstrap-e2e-master
Jan 30 20:33:50.204: INFO: 
Logging pods the kubelet thinks is on node bootstrap-e2e-master
Jan 30 20:33:50.297: INFO: etcd-server-events-bootstrap-e2e-master started at 2023-01-30 17:10:44 +0000 UTC (0+1 container statuses recorded)
Jan 30 20:33:50.297: INFO: 	Container etcd-container ready: true, restart count 0
Jan 30 20:33:50.297: INFO: etcd-server-bootstrap-e2e-master started at 2023-01-30 17:10:44 +0000 UTC (0+1 container statuses recorded)
Jan 30 20:33:50.297: INFO: 	Container etcd-container ready: true, restart count 0
Jan 30 20:33:50.297: INFO: konnectivity-server-bootstrap-e2e-master started at 2023-01-30 17:10:44 +0000 UTC (0+1 container statuses recorded)
Jan 30 20:33:50.297: INFO: 	Container konnectivity-server-container ready: true, restart count 0
Jan 30 20:33:50.297: INFO: kube-scheduler-bootstrap-e2e-master started at 2023-01-30 17:10:44 +0000 UTC (0+1 container statuses recorded)
Jan 30 20:33:50.297: INFO: 	Container kube-scheduler ready: true, restart count 1
Jan 30 20:33:50.297: INFO: l7-lb-controller-bootstrap-e2e-master started at 2023-01-30 17:11:04 +0000 UTC (0+1 container statuses recorded)
Jan 30 20:33:50.297: INFO: 	Container l7-lb-controller ready: true, restart count 2
Jan 30 20:33:50.297: INFO: kube-apiserver-bootstrap-e2e-master started at 2023-01-30 17:10:43 +0000 UTC (0+1 container statuses recorded)
Jan 30 20:33:50.297: INFO: 	Container kube-apiserver ready: true, restart count 0
Jan 30 20:33:50.297: INFO: kube-controller-manager-bootstrap-e2e-master started at 2023-01-30 17:10:44 +0000 UTC (0+1 container statuses recorded)
Jan 30 20:33:50.297: INFO: 	Container kube-controller-manager ready: true, restart count 3
Jan 30 20:33:50.297: INFO: kube-addon-manager-bootstrap-e2e-master started at 2023-01-30 17:11:03 +0000 UTC (0+1 container statuses recorded)
Jan 30 20:33:50.297: INFO: 	Container kube-addon-manager ready: true, restart count 0
Jan 30 20:33:50.297: INFO: metadata-proxy-v0.1-g24sg started at 2023-01-30 17:11:48 +0000 UTC (0+2 container statuses recorded)
Jan 30 20:33:50.297: INFO: 	Container metadata-proxy ready: true, restart count 0
Jan 30 20:33:50.297: INFO: 	Container prometheus-to-sd-exporter ready: true, restart count 0
Jan 30 20:33:50.484: INFO: 
Latency metrics for node bootstrap-e2e-master
Jan 30 20:33:50.484: INFO: 
Logging node info for node bootstrap-e2e-minion-group-5sqf
Jan 30 20:33:50.527: INFO: Node Info: &Node{ObjectMeta:{bootstrap-e2e-minion-group-5sqf    316fbe79-ed9c-499e-aac1-84966e803f96 52839 0 2023-01-30 19:47:23 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:n1-standard-2 beta.kubernetes.io/os:linux cloud.google.com/metadata-proxy-ready:true failure-domain.beta.kubernetes.io/region:us-west1 failure-domain.beta.kubernetes.io/zone:us-west1-b kubernetes.io/arch:amd64 kubernetes.io/hostname:bootstrap-e2e-minion-group-5sqf kubernetes.io/os:linux node.kubernetes.io/instance-type:n1-standard-2 topology.gke.io/zone:us-west1-b topology.hostpath.csi/node:bootstrap-e2e-minion-group-5sqf topology.kubernetes.io/region:us-west1 topology.kubernetes.io/zone:us-west1-b] map[node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kubelet Update v1 2023-01-30 19:47:23 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:cloud.google.com/metadata-proxy-ready":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {kube-controller-manager Update v1 2023-01-30 20:27:20 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}}}} status} {kube-controller-manager Update v1 2023-01-30 20:27:49 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"10.64.4.0/24\"":{}}}} } {node-problem-detector Update v1 2023-01-30 20:32:49 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"CorruptDockerOverlay2\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"FrequentContainerdRestart\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"FrequentDockerRestart\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"FrequentKubeletRestart\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"FrequentUnregisterNetDevice\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"KernelDeadlock\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"ReadonlyFilesystem\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kubelet Update v1 2023-01-30 20:32:55 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:topology.gke.io/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:nodeInfo":{"f:bootID":{}}}} status}]},Spec:NodeSpec{PodCIDR:10.64.4.0/24,DoNotUseExternalID:,ProviderID:gce://k8s-infra-e2e-boskos-080/us-west1-b/bootstrap-e2e-minion-group-5sqf,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[10.64.4.0/24],},Status:NodeStatus{Capacity:ResourceList{attachable-volumes-gce-pd: {{127 0} {<nil>} 127 DecimalSI},cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{101203873792 0} {<nil>} 98831908Ki BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{7815426048 0} {<nil>} 7632252Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{attachable-volumes-gce-pd: {{127 0} {<nil>} 127 DecimalSI},cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{91083486262 0} {<nil>} 91083486262 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{7553282048 0} {<nil>} 7376252Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:KernelDeadlock,Status:False,LastHeartbeatTime:2023-01-30 20:32:49 +0000 UTC,LastTransitionTime:2023-01-30 20:27:47 +0000 UTC,Reason:KernelHasNoDeadlock,Message:kernel has no deadlock,},NodeCondition{Type:ReadonlyFilesystem,Status:False,LastHeartbeatTime:2023-01-30 20:32:49 +0000 UTC,LastTransitionTime:2023-01-30 20:27:47 +0000 UTC,Reason:FilesystemIsNotReadOnly,Message:Filesystem is not read-only,},NodeCondition{Type:CorruptDockerOverlay2,Status:False,LastHeartbeatTime:2023-01-30 20:32:49 +0000 UTC,LastTransitionTime:2023-01-30 20:27:47 +0000 UTC,Reason:NoCorruptDockerOverlay2,Message:docker overlay2 is functioning properly,},NodeCondition{Type:FrequentUnregisterNetDevice,Status:False,LastHeartbeatTime:2023-01-30 20:32:49 +0000 UTC,LastTransitionTime:2023-01-30 20:27:47 +0000 UTC,Reason:NoFrequentUnregisterNetDevice,Message:node is functioning properly,},NodeCondition{Type:FrequentKubeletRestart,Status:False,LastHeartbeatTime:2023-01-30 20:32:49 +0000 UTC,LastTransitionTime:2023-01-30 20:27:47 +0000 UTC,Reason:NoFrequentKubeletRestart,Message:kubelet is functioning properly,},NodeCondition{Type:FrequentDockerRestart,Status:False,LastHeartbeatTime:2023-01-30 20:32:49 +0000 UTC,LastTransitionTime:2023-01-30 20:27:47 +0000 UTC,Reason:NoFrequentDockerRestart,Message:docker is functioning properly,},NodeCondition{Type:FrequentContainerdRestart,Status:False,LastHeartbeatTime:2023-01-30 20:32:49 +0000 UTC,LastTransitionTime:2023-01-30 20:27:47 +0000 UTC,Reason:NoFrequentContainerdRestart,Message:containerd is functioning properly,},NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2023-01-30 19:47:33 +0000 UTC,LastTransitionTime:2023-01-30 19:47:33 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-01-30 20:32:55 +0000 UTC,LastTransitionTime:2023-01-30 20:27:49 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-01-30 20:32:55 +0000 UTC,LastTransitionTime:2023-01-30 20:27:49 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-01-30 20:32:55 +0000 UTC,LastTransitionTime:2023-01-30 20:27:49 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-01-30 20:32:55 +0000 UTC,LastTransitionTime:2023-01-30 20:27:49 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:10.138.0.9,},NodeAddress{Type:ExternalIP,Address:35.227.149.86,},NodeAddress{Type:InternalDNS,Address:bootstrap-e2e-minion-group-5sqf.c.k8s-infra-e2e-boskos-080.internal,},NodeAddress{Type:Hostname,Address:bootstrap-e2e-minion-group-5sqf.c.k8s-infra-e2e-boskos-080.internal,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:939aca5c4e6f48f6e7efe71a9260f4cf,SystemUUID:939aca5c-4e6f-48f6-e7ef-e71a9260f4cf,BootID:bcc84782-293d-401e-b1a7-1198eeaeeac8,KernelVersion:5.10.123+,OSImage:Container-Optimized OS from Google,ContainerRuntimeVersion:containerd://1.6.6,KubeletVersion:v1.27.0-alpha.1.90+c829397f7a1fa9,KubeProxyVersion:v1.27.0-alpha.1.90+c829397f7a1fa9,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[registry.k8s.io/cloud-provider-gcp/gcp-compute-persistent-disk-csi-driver@sha256:e9c355dd1ce57de91a0f9164bd2fe688938723e7f9bd40bb13652d603f41d25a registry.k8s.io/cloud-provider-gcp/gcp-compute-persistent-disk-csi-driver:v1.2.2],SizeBytes:73354282,},ContainerImage{Names:[registry.k8s.io/kube-proxy-amd64:v1.27.0-alpha.1.90_c829397f7a1fa9],SizeBytes:66988744,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:16bbf38c463a4223d8cfe4da12bc61010b082a79b4bb003e2d3ba3ece5dd5f9e registry.k8s.io/e2e-test-images/agnhost:2.43],SizeBytes:51706353,},ContainerImage{Names:[gke.gcr.io/prometheus-to-sd@sha256:e739643c3939ba0b161425f45a1989eedfc4a3b166db9a7100863296b4c70510 gke.gcr.io/prometheus-to-sd:v0.11.1-gke.1],SizeBytes:48742566,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:148b022f5c5da426fc2f3c14b5c0867e58ef05961510c84749ac1fddcb0fef22 registry.k8s.io/e2e-test-images/httpd:2.4.38-4],SizeBytes:40764257,},ContainerImage{Names:[registry.k8s.io/cloud-provider-gcp/gcp-compute-persistent-disk-csi-driver@sha256:dc0301d9a43e23e389abb7690cd631f98af76ea3429259c7f5f643fd31fc0a3e registry.k8s.io/cloud-provider-gcp/gcp-compute-persistent-disk-csi-driver:v1.4.0],SizeBytes:27914152,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:ee3b525d5b89db99da3b8eb521d9cd90cb6e9ef0fbb651e98bb37be78d36b5b8 registry.k8s.io/sig-storage/csi-provisioner:v3.3.0],SizeBytes:25491225,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:425d8f1b769398127767b06ed97ce62578a3179bcb99809ce93a1649e025ffe7 registry.k8s.io/sig-storage/csi-resizer:v1.6.0],SizeBytes:24148884,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:291334908ddf71a4661fd7f6d9d97274de8a5378a2b6fdfeb2ce73414a34f82f registry.k8s.io/sig-storage/csi-snapshotter:v6.1.0],SizeBytes:23881995,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:9a685020911e2725ad019dbce6e4a5ab93d51e3d4557f115e64343345e05781b registry.k8s.io/sig-storage/csi-attacher:v4.0.0],SizeBytes:23847201,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:89e900a160a986a1a7a4eba7f5259e510398fa87ca9b8a729e7dec59e04c7709 registry.k8s.io/sig-storage/csi-snapshotter:v5.0.1],SizeBytes:22163966,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:92257881c1d6493cf18299a24af42330f891166560047902b8d431fb66b01af5 registry.k8s.io/sig-storage/hostpathplugin:v1.9.0],SizeBytes:18758628,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:017727efcfeb7d053af68e51436ce8e65edbc6ca573720afb4f79c8594036955 registry.k8s.io/coredns/coredns:v1.10.0],SizeBytes:15273057,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:933940f13b3ea0abc62e656c1aa5c5b47c04b15d71250413a6b821bd0c58b94e registry.k8s.io/sig-storage/livenessprobe:v2.7.0],SizeBytes:8688564,},ContainerImage{Names:[registry.k8s.io/kas-network-proxy/proxy-agent@sha256:939c42e815e6b6af3181f074652c0d18fe429fcee9b49c1392aee7e92887cfef registry.k8s.io/kas-network-proxy/proxy-agent:v0.1.1],SizeBytes:8364694,},ContainerImage{Names:[registry.k8s.io/metadata-proxy@sha256:e914645f22e946bce5165737e1b244e0a296ad1f0f81a9531adc57af2780978a registry.k8s.io/metadata-proxy:v0.1.12],SizeBytes:5301657,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:2e0f836850e09b8b7cc937681d6194537a09fbd5f6b9e08f4d646a85128e8937 registry.k8s.io/e2e-test-images/busybox:1.29-4],SizeBytes:731990,},ContainerImage{Names:[registry.k8s.io/pause@sha256:7031c1b283388d2c2e09b57badb803c05ebed362dc88d84b480cc47f72a21097 registry.k8s.io/pause:3.9],SizeBytes:321520,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},}
Jan 30 20:33:50.527: INFO: 
Logging kubelet events for node bootstrap-e2e-minion-group-5sqf
Jan 30 20:33:50.571: INFO: 
Logging pods the kubelet thinks is on node bootstrap-e2e-minion-group-5sqf
Jan 30 20:33:50.639: INFO: metadata-proxy-v0.1-j8mg4 started at 2023-01-30 19:47:24 +0000 UTC (0+2 container statuses recorded)
Jan 30 20:33:50.639: INFO: 	Container metadata-proxy ready: true, restart count 1
Jan 30 20:33:50.639: INFO: 	Container prometheus-to-sd-exporter ready: true, restart count 1
Jan 30 20:33:50.639: INFO: konnectivity-agent-m7rcz started at 2023-01-30 19:47:33 +0000 UTC (0+1 container statuses recorded)
Jan 30 20:33:50.639: INFO: 	Container konnectivity-agent ready: true, restart count 1
Jan 30 20:33:50.639: INFO: kube-proxy-bootstrap-e2e-minion-group-5sqf started at 2023-01-30 19:47:23 +0000 UTC (0+1 container statuses recorded)
Jan 30 20:33:50.639: INFO: 	Container kube-proxy ready: true, restart count 1
Jan 30 20:33:50.836: INFO: 
Latency metrics for node bootstrap-e2e-minion-group-5sqf
Jan 30 20:33:50.836: INFO: 
Logging node info for node bootstrap-e2e-minion-group-gr7w
Jan 30 20:33:50.879: INFO: Node Info: &Node{ObjectMeta:{bootstrap-e2e-minion-group-gr7w    0a4b32b4-348b-4c5b-93fb-de4d0701e2eb 52860 0 2023-01-30 17:11:35 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:n1-standard-2 beta.kubernetes.io/os:linux cloud.google.com/metadata-proxy-ready:true failure-domain.beta.kubernetes.io/region:us-west1 failure-domain.beta.kubernetes.io/zone:us-west1-b kubernetes.io/arch:amd64 kubernetes.io/hostname:bootstrap-e2e-minion-group-gr7w kubernetes.io/os:linux node.kubernetes.io/instance-type:n1-standard-2 topology.gke.io/zone:us-west1-b topology.hostpath.csi/node:bootstrap-e2e-minion-group-gr7w topology.kubernetes.io/region:us-west1 topology.kubernetes.io/zone:us-west1-b] map[node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kubelet Update v1 2023-01-30 17:11:35 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:cloud.google.com/metadata-proxy-ready":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {e2e.test Update v1 2023-01-30 20:15:46 +0000 UTC FieldsV1 {"f:status":{"f:allocatable":{"f:example.com/fakePTSRes":{}},"f:capacity":{"f:example.com/fakePTSRes":{}}}} status} {kube-controller-manager Update v1 2023-01-30 20:27:25 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}}}} status} {kube-controller-manager Update v1 2023-01-30 20:27:56 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"10.64.1.0/24\"":{}}}} } {node-problem-detector Update v1 2023-01-30 20:32:57 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"CorruptDockerOverlay2\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"FrequentContainerdRestart\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"FrequentDockerRestart\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"FrequentKubeletRestart\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"FrequentUnregisterNetDevice\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"KernelDeadlock\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"ReadonlyFilesystem\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kubelet Update v1 2023-01-30 20:33:03 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:topology.gke.io/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{},"f:memory":{}},"f:capacity":{"f:memory":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:nodeInfo":{"f:bootID":{}}}} status}]},Spec:NodeSpec{PodCIDR:10.64.1.0/24,DoNotUseExternalID:,ProviderID:gce://k8s-infra-e2e-boskos-080/us-west1-b/bootstrap-e2e-minion-group-gr7w,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[10.64.1.0/24],},Status:NodeStatus{Capacity:ResourceList{attachable-volumes-gce-pd: {{127 0} {<nil>} 127 DecimalSI},cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{101203873792 0} {<nil>} 98831908Ki BinarySI},example.com/fakePTSRes: {{10 0} {<nil>} 10 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{7815426048 0} {<nil>} 7632252Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{attachable-volumes-gce-pd: {{127 0} {<nil>} 127 DecimalSI},cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{91083486262 0} {<nil>} 91083486262 DecimalSI},example.com/fakePTSRes: {{10 0} {<nil>} 10 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{7553282048 0} {<nil>} 7376252Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:FrequentContainerdRestart,Status:False,LastHeartbeatTime:2023-01-30 20:32:57 +0000 UTC,LastTransitionTime:2023-01-30 20:27:55 +0000 UTC,Reason:NoFrequentContainerdRestart,Message:containerd is functioning properly,},NodeCondition{Type:KernelDeadlock,Status:False,LastHeartbeatTime:2023-01-30 20:32:57 +0000 UTC,LastTransitionTime:2023-01-30 20:27:55 +0000 UTC,Reason:KernelHasNoDeadlock,Message:kernel has no deadlock,},NodeCondition{Type:ReadonlyFilesystem,Status:False,LastHeartbeatTime:2023-01-30 20:32:57 +0000 UTC,LastTransitionTime:2023-01-30 20:27:55 +0000 UTC,Reason:FilesystemIsNotReadOnly,Message:Filesystem is not read-only,},NodeCondition{Type:CorruptDockerOverlay2,Status:False,LastHeartbeatTime:2023-01-30 20:32:57 +0000 UTC,LastTransitionTime:2023-01-30 20:27:55 +0000 UTC,Reason:NoCorruptDockerOverlay2,Message:docker overlay2 is functioning properly,},NodeCondition{Type:FrequentUnregisterNetDevice,Status:False,LastHeartbeatTime:2023-01-30 20:32:57 +0000 UTC,LastTransitionTime:2023-01-30 20:27:55 +0000 UTC,Reason:NoFrequentUnregisterNetDevice,Message:node is functioning properly,},NodeCondition{Type:FrequentKubeletRestart,Status:False,LastHeartbeatTime:2023-01-30 20:32:57 +0000 UTC,LastTransitionTime:2023-01-30 20:27:55 +0000 UTC,Reason:NoFrequentKubeletRestart,Message:kubelet is functioning properly,},NodeCondition{Type:FrequentDockerRestart,Status:False,LastHeartbeatTime:2023-01-30 20:32:57 +0000 UTC,LastTransitionTime:2023-01-30 20:27:55 +0000 UTC,Reason:NoFrequentDockerRestart,Message:docker is functioning properly,},NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2023-01-30 17:11:45 +0000 UTC,LastTransitionTime:2023-01-30 17:11:45 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-01-30 20:33:03 +0000 UTC,LastTransitionTime:2023-01-30 20:27:56 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-01-30 20:33:03 +0000 UTC,LastTransitionTime:2023-01-30 20:27:56 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-01-30 20:33:03 +0000 UTC,LastTransitionTime:2023-01-30 20:27:56 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-01-30 20:33:03 +0000 UTC,LastTransitionTime:2023-01-30 20:27:56 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:10.138.0.4,},NodeAddress{Type:ExternalIP,Address:35.197.7.140,},NodeAddress{Type:InternalDNS,Address:bootstrap-e2e-minion-group-gr7w.c.k8s-infra-e2e-boskos-080.internal,},NodeAddress{Type:Hostname,Address:bootstrap-e2e-minion-group-gr7w.c.k8s-infra-e2e-boskos-080.internal,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:b7fa42720df0ccfbfadca51fa5993af0,SystemUUID:b7fa4272-0df0-ccfb-fadc-a51fa5993af0,BootID:24c065d9-7b4d-4dfd-a3ab-600e2ef1a9f9,KernelVersion:5.10.123+,OSImage:Container-Optimized OS from Google,ContainerRuntimeVersion:containerd://1.6.6,KubeletVersion:v1.27.0-alpha.1.90+c829397f7a1fa9,KubeProxyVersion:v1.27.0-alpha.1.90+c829397f7a1fa9,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[registry.k8s.io/e2e-test-images/jessie-dnsutils@sha256:24aaf2626d6b27864c29de2097e8bbb840b3a414271bf7c8995e431e47d8408e registry.k8s.io/e2e-test-images/jessie-dnsutils:1.7],SizeBytes:112030336,},ContainerImage{Names:[registry.k8s.io/cloud-provider-gcp/gcp-compute-persistent-disk-csi-driver@sha256:e9c355dd1ce57de91a0f9164bd2fe688938723e7f9bd40bb13652d603f41d25a registry.k8s.io/cloud-provider-gcp/gcp-compute-persistent-disk-csi-driver:v1.2.2],SizeBytes:73354282,},ContainerImage{Names:[registry.k8s.io/kube-proxy-amd64:v1.27.0-alpha.1.90_c829397f7a1fa9],SizeBytes:66988744,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:16bbf38c463a4223d8cfe4da12bc61010b082a79b4bb003e2d3ba3ece5dd5f9e registry.k8s.io/e2e-test-images/agnhost:2.43],SizeBytes:51706353,},ContainerImage{Names:[gke.gcr.io/prometheus-to-sd@sha256:e739643c3939ba0b161425f45a1989eedfc4a3b166db9a7100863296b4c70510 gke.gcr.io/prometheus-to-sd:v0.11.1-gke.1],SizeBytes:48742566,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:148b022f5c5da426fc2f3c14b5c0867e58ef05961510c84749ac1fddcb0fef22 registry.k8s.io/e2e-test-images/httpd:2.4.38-4],SizeBytes:40764257,},ContainerImage{Names:[registry.k8s.io/sig-storage/volume-data-source-validator@sha256:d35884236461ad14397194cb4abc13aefb28a54bf3f6698d673c2ba353e78e18 registry.k8s.io/sig-storage/volume-data-source-validator:v1.0.0],SizeBytes:28662933,},ContainerImage{Names:[registry.k8s.io/cloud-provider-gcp/gcp-compute-persistent-disk-csi-driver@sha256:dc0301d9a43e23e389abb7690cd631f98af76ea3429259c7f5f643fd31fc0a3e registry.k8s.io/cloud-provider-gcp/gcp-compute-persistent-disk-csi-driver:v1.4.0],SizeBytes:27914152,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:ee3b525d5b89db99da3b8eb521d9cd90cb6e9ef0fbb651e98bb37be78d36b5b8 registry.k8s.io/sig-storage/csi-provisioner:v3.3.0],SizeBytes:25491225,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:425d8f1b769398127767b06ed97ce62578a3179bcb99809ce93a1649e025ffe7 registry.k8s.io/sig-storage/csi-resizer:v1.6.0],SizeBytes:24148884,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:291334908ddf71a4661fd7f6d9d97274de8a5378a2b6fdfeb2ce73414a34f82f registry.k8s.io/sig-storage/csi-snapshotter:v6.1.0],SizeBytes:23881995,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:9a685020911e2725ad019dbce6e4a5ab93d51e3d4557f115e64343345e05781b registry.k8s.io/sig-storage/csi-attacher:v4.0.0],SizeBytes:23847201,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:89e900a160a986a1a7a4eba7f5259e510398fa87ca9b8a729e7dec59e04c7709 registry.k8s.io/sig-storage/csi-snapshotter:v5.0.1],SizeBytes:22163966,},ContainerImage{Names:[registry.k8s.io/sig-storage/hello-populator@sha256:42d275bc2569b00a11c8058e5b77c50cdee39bf75d80c912b80813f03e7fd7d1 registry.k8s.io/sig-storage/hello-populator:v1.0.1],SizeBytes:20511089,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:92257881c1d6493cf18299a24af42330f891166560047902b8d431fb66b01af5 registry.k8s.io/sig-storage/hostpathplugin:v1.9.0],SizeBytes:18758628,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:017727efcfeb7d053af68e51436ce8e65edbc6ca573720afb4f79c8594036955 registry.k8s.io/coredns/coredns:v1.10.0],SizeBytes:15273057,},ContainerImage{Names:[registry.k8s.io/cpa/cluster-proportional-autoscaler@sha256:fd636b33485c7826fb20ef0688a83ee0910317dbb6c0c6f3ad14661c1db25def registry.k8s.io/cpa/cluster-proportional-autoscaler:1.8.4],SizeBytes:15209393,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:933940f13b3ea0abc62e656c1aa5c5b47c04b15d71250413a6b821bd0c58b94e registry.k8s.io/sig-storage/livenessprobe:v2.7.0],SizeBytes:8688564,},ContainerImage{Names:[registry.k8s.io/kas-network-proxy/proxy-agent@sha256:939c42e815e6b6af3181f074652c0d18fe429fcee9b49c1392aee7e92887cfef registry.k8s.io/kas-network-proxy/proxy-agent:v0.1.1],SizeBytes:8364694,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:5c99cf6a02adda929b10321dbf4ecfa00d87be9ba4fb456006237d530ab4baa1 registry.k8s.io/e2e-test-images/nginx:1.14-4],SizeBytes:6978614,},ContainerImage{Names:[registry.k8s.io/networking/ingress-gce-404-server-with-metrics-amd64@sha256:7eb7b3cee4d33c10c49893ad3c386232b86d4067de5251294d4c620d6e072b93 registry.k8s.io/networking/ingress-gce-404-server-with-metrics-amd64:v1.10.11],SizeBytes:6463068,},ContainerImage{Names:[registry.k8s.io/metadata-proxy@sha256:e914645f22e946bce5165737e1b244e0a296ad1f0f81a9531adc57af2780978a registry.k8s.io/metadata-proxy:v0.1.12],SizeBytes:5301657,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:2e0f836850e09b8b7cc937681d6194537a09fbd5f6b9e08f4d646a85128e8937 registry.k8s.io/e2e-test-images/busybox:1.29-4],SizeBytes:731990,},ContainerImage{Names:[registry.k8s.io/pause@sha256:7031c1b283388d2c2e09b57badb803c05ebed362dc88d84b480cc47f72a21097 registry.k8s.io/pause:3.9],SizeBytes:321520,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},}
Jan 30 20:33:50.879: INFO: 
Logging kubelet events for node bootstrap-e2e-minion-group-gr7w
Jan 30 20:33:50.923: INFO: 
Logging pods the kubelet thinks is on node bootstrap-e2e-minion-group-gr7w
Jan 30 20:33:51.014: INFO: metadata-proxy-v0.1-g6h7m started at 2023-01-30 17:11:36 +0000 UTC (0+2 container statuses recorded)
Jan 30 20:33:51.014: INFO: 	Container metadata-proxy ready: true, restart count 1
Jan 30 20:33:51.014: INFO: 	Container prometheus-to-sd-exporter ready: true, restart count 1
Jan 30 20:33:51.014: INFO: konnectivity-agent-pb855 started at 2023-01-30 17:11:46 +0000 UTC (0+1 container statuses recorded)
Jan 30 20:33:51.014: INFO: 	Container konnectivity-agent ready: true, restart count 1
Jan 30 20:33:51.014: INFO: kube-dns-autoscaler-5f6455f985-7jdxx started at 2023-01-30 18:36:57 +0000 UTC (0+1 container statuses recorded)
Jan 30 20:33:51.014: INFO: 	Container autoscaler ready: true, restart count 1
Jan 30 20:33:51.014: INFO: l7-default-backend-8549d69d99-xzb2m started at 2023-01-30 18:36:57 +0000 UTC (0+1 container statuses recorded)
Jan 30 20:33:51.014: INFO: 	Container default-http-backend ready: true, restart count 1
Jan 30 20:33:51.014: INFO: coredns-6846b5b5f-bkw98 started at 2023-01-30 20:20:45 +0000 UTC (0+1 container statuses recorded)
Jan 30 20:33:51.014: INFO: 	Container coredns ready: true, restart count 1
Jan 30 20:33:51.014: INFO: kube-proxy-bootstrap-e2e-minion-group-gr7w started at 2023-01-30 17:11:35 +0000 UTC (0+1 container statuses recorded)
Jan 30 20:33:51.014: INFO: 	Container kube-proxy ready: true, restart count 1
Jan 30 20:33:51.197: INFO: 
Latency metrics for node bootstrap-e2e-minion-group-gr7w
Jan 30 20:33:51.197: INFO: 
Logging node info for node bootstrap-e2e-minion-group-hwhs
Jan 30 20:33:51.241: INFO: Node Info: &Node{ObjectMeta:{bootstrap-e2e-minion-group-hwhs    7bbe7962-b092-4381-b012-d5ff6aa9582a 52851 0 2023-01-30 17:11:38 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:n1-standard-2 beta.kubernetes.io/os:linux cloud.google.com/metadata-proxy-ready:true failure-domain.beta.kubernetes.io/region:us-west1 failure-domain.beta.kubernetes.io/zone:us-west1-b kubernetes.io/arch:amd64 kubernetes.io/hostname:bootstrap-e2e-minion-group-hwhs kubernetes.io/os:linux node.kubernetes.io/instance-type:n1-standard-2 topology.gke.io/zone:us-west1-b topology.hostpath.csi/node:bootstrap-e2e-minion-group-hwhs topology.kubernetes.io/region:us-west1 topology.kubernetes.io/zone:us-west1-b] map[node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kubelet Update v1 2023-01-30 17:11:38 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:cloud.google.com/metadata-proxy-ready":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {kube-controller-manager Update v1 2023-01-30 20:27:15 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}}}} status} {kube-controller-manager Update v1 2023-01-30 20:27:52 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"10.64.3.0/24\"":{}}}} } {node-problem-detector Update v1 2023-01-30 20:32:52 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"CorruptDockerOverlay2\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"FrequentContainerdRestart\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"FrequentDockerRestart\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"FrequentKubeletRestart\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"FrequentUnregisterNetDevice\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"KernelDeadlock\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"ReadonlyFilesystem\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kubelet Update v1 2023-01-30 20:32:59 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:topology.gke.io/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{},"f:memory":{}},"f:capacity":{"f:memory":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:nodeInfo":{"f:bootID":{}}}} status}]},Spec:NodeSpec{PodCIDR:10.64.3.0/24,DoNotUseExternalID:,ProviderID:gce://k8s-infra-e2e-boskos-080/us-west1-b/bootstrap-e2e-minion-group-hwhs,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[10.64.3.0/24],},Status:NodeStatus{Capacity:ResourceList{attachable-volumes-gce-pd: {{127 0} {<nil>} 127 DecimalSI},cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{101203873792 0} {<nil>} 98831908Ki BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{7815426048 0} {<nil>} 7632252Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{attachable-volumes-gce-pd: {{127 0} {<nil>} 127 DecimalSI},cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{91083486262 0} {<nil>} 91083486262 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{7553282048 0} {<nil>} 7376252Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:KernelDeadlock,Status:False,LastHeartbeatTime:2023-01-30 20:32:52 +0000 UTC,LastTransitionTime:2023-01-30 20:27:51 +0000 UTC,Reason:KernelHasNoDeadlock,Message:kernel has no deadlock,},NodeCondition{Type:ReadonlyFilesystem,Status:False,LastHeartbeatTime:2023-01-30 20:32:52 +0000 UTC,LastTransitionTime:2023-01-30 20:27:51 +0000 UTC,Reason:FilesystemIsNotReadOnly,Message:Filesystem is not read-only,},NodeCondition{Type:CorruptDockerOverlay2,Status:False,LastHeartbeatTime:2023-01-30 20:32:52 +0000 UTC,LastTransitionTime:2023-01-30 20:27:51 +0000 UTC,Reason:NoCorruptDockerOverlay2,Message:docker overlay2 is functioning properly,},NodeCondition{Type:FrequentUnregisterNetDevice,Status:False,LastHeartbeatTime:2023-01-30 20:32:52 +0000 UTC,LastTransitionTime:2023-01-30 20:27:51 +0000 UTC,Reason:NoFrequentUnregisterNetDevice,Message:node is functioning properly,},NodeCondition{Type:FrequentKubeletRestart,Status:False,LastHeartbeatTime:2023-01-30 20:32:52 +0000 UTC,LastTransitionTime:2023-01-30 20:27:51 +0000 UTC,Reason:NoFrequentKubeletRestart,Message:kubelet is functioning properly,},NodeCondition{Type:FrequentDockerRestart,Status:False,LastHeartbeatTime:2023-01-30 20:32:52 +0000 UTC,LastTransitionTime:2023-01-30 20:27:51 +0000 UTC,Reason:NoFrequentDockerRestart,Message:docker is functioning properly,},NodeCondition{Type:FrequentContainerdRestart,Status:False,LastHeartbeatTime:2023-01-30 20:32:52 +0000 UTC,LastTransitionTime:2023-01-30 20:27:51 +0000 UTC,Reason:NoFrequentContainerdRestart,Message:containerd is functioning properly,},NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2023-01-30 17:11:54 +0000 UTC,LastTransitionTime:2023-01-30 17:11:54 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-01-30 20:32:59 +0000 UTC,LastTransitionTime:2023-01-30 20:27:52 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-01-30 20:32:59 +0000 UTC,LastTransitionTime:2023-01-30 20:27:52 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-01-30 20:32:59 +0000 UTC,LastTransitionTime:2023-01-30 20:27:52 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-01-30 20:32:59 +0000 UTC,LastTransitionTime:2023-01-30 20:27:52 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:10.138.0.5,},NodeAddress{Type:ExternalIP,Address:34.82.53.160,},NodeAddress{Type:InternalDNS,Address:bootstrap-e2e-minion-group-hwhs.c.k8s-infra-e2e-boskos-080.internal,},NodeAddress{Type:Hostname,Address:bootstrap-e2e-minion-group-hwhs.c.k8s-infra-e2e-boskos-080.internal,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:0306c1b4b16dc97a101d208104b2a5f0,SystemUUID:0306c1b4-b16d-c97a-101d-208104b2a5f0,BootID:4cb2635d-07e6-4b8f-a686-d021bbfef301,KernelVersion:5.10.123+,OSImage:Container-Optimized OS from Google,ContainerRuntimeVersion:containerd://1.6.6,KubeletVersion:v1.27.0-alpha.1.90+c829397f7a1fa9,KubeProxyVersion:v1.27.0-alpha.1.90+c829397f7a1fa9,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[registry.k8s.io/cloud-provider-gcp/gcp-compute-persistent-disk-csi-driver@sha256:e9c355dd1ce57de91a0f9164bd2fe688938723e7f9bd40bb13652d603f41d25a registry.k8s.io/cloud-provider-gcp/gcp-compute-persistent-disk-csi-driver:v1.2.2],SizeBytes:73354282,},ContainerImage{Names:[registry.k8s.io/kube-proxy-amd64:v1.27.0-alpha.1.90_c829397f7a1fa9],SizeBytes:66988744,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:16bbf38c463a4223d8cfe4da12bc61010b082a79b4bb003e2d3ba3ece5dd5f9e registry.k8s.io/e2e-test-images/agnhost:2.43],SizeBytes:51706353,},ContainerImage{Names:[gke.gcr.io/prometheus-to-sd@sha256:e739643c3939ba0b161425f45a1989eedfc4a3b166db9a7100863296b4c70510 gke.gcr.io/prometheus-to-sd:v0.11.1-gke.1],SizeBytes:48742566,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:148b022f5c5da426fc2f3c14b5c0867e58ef05961510c84749ac1fddcb0fef22 registry.k8s.io/e2e-test-images/httpd:2.4.38-4],SizeBytes:40764257,},ContainerImage{Names:[registry.k8s.io/cloud-provider-gcp/gcp-compute-persistent-disk-csi-driver@sha256:dc0301d9a43e23e389abb7690cd631f98af76ea3429259c7f5f643fd31fc0a3e registry.k8s.io/cloud-provider-gcp/gcp-compute-persistent-disk-csi-driver:v1.4.0],SizeBytes:27914152,},ContainerImage{Names:[registry.k8s.io/metrics-server/metrics-server@sha256:6385aec64bb97040a5e692947107b81e178555c7a5b71caa90d733e4130efc10 registry.k8s.io/metrics-server/metrics-server:v0.5.2],SizeBytes:26023008,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:ee3b525d5b89db99da3b8eb521d9cd90cb6e9ef0fbb651e98bb37be78d36b5b8 registry.k8s.io/sig-storage/csi-provisioner:v3.3.0],SizeBytes:25491225,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:425d8f1b769398127767b06ed97ce62578a3179bcb99809ce93a1649e025ffe7 registry.k8s.io/sig-storage/csi-resizer:v1.6.0],SizeBytes:24148884,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:291334908ddf71a4661fd7f6d9d97274de8a5378a2b6fdfeb2ce73414a34f82f registry.k8s.io/sig-storage/csi-snapshotter:v6.1.0],SizeBytes:23881995,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:9a685020911e2725ad019dbce6e4a5ab93d51e3d4557f115e64343345e05781b registry.k8s.io/sig-storage/csi-attacher:v4.0.0],SizeBytes:23847201,},ContainerImage{Names:[registry.k8s.io/sig-storage/snapshot-controller@sha256:823c75d0c45d1427f6d850070956d9ca657140a7bbf828381541d1d808475280 registry.k8s.io/sig-storage/snapshot-controller:v6.1.0],SizeBytes:22620891,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:89e900a160a986a1a7a4eba7f5259e510398fa87ca9b8a729e7dec59e04c7709 registry.k8s.io/sig-storage/csi-snapshotter:v5.0.1],SizeBytes:22163966,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:92257881c1d6493cf18299a24af42330f891166560047902b8d431fb66b01af5 registry.k8s.io/sig-storage/hostpathplugin:v1.9.0],SizeBytes:18758628,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:017727efcfeb7d053af68e51436ce8e65edbc6ca573720afb4f79c8594036955 registry.k8s.io/coredns/coredns:v1.10.0],SizeBytes:15273057,},ContainerImage{Names:[registry.k8s.io/autoscaling/addon-resizer@sha256:43f129b81d28f0fdd54de6d8e7eacd5728030782e03db16087fc241ad747d3d6 registry.k8s.io/autoscaling/addon-resizer:1.8.14],SizeBytes:10153852,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:933940f13b3ea0abc62e656c1aa5c5b47c04b15d71250413a6b821bd0c58b94e registry.k8s.io/sig-storage/livenessprobe:v2.7.0],SizeBytes:8688564,},ContainerImage{Names:[registry.k8s.io/kas-network-proxy/proxy-agent@sha256:939c42e815e6b6af3181f074652c0d18fe429fcee9b49c1392aee7e92887cfef registry.k8s.io/kas-network-proxy/proxy-agent:v0.1.1],SizeBytes:8364694,},ContainerImage{Names:[registry.k8s.io/metadata-proxy@sha256:e914645f22e946bce5165737e1b244e0a296ad1f0f81a9531adc57af2780978a registry.k8s.io/metadata-proxy:v0.1.12],SizeBytes:5301657,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:2e0f836850e09b8b7cc937681d6194537a09fbd5f6b9e08f4d646a85128e8937 registry.k8s.io/e2e-test-images/busybox:1.29-4],SizeBytes:731990,},ContainerImage{Names:[registry.k8s.io/pause@sha256:7031c1b283388d2c2e09b57badb803c05ebed362dc88d84b480cc47f72a21097 registry.k8s.io/pause:3.9],SizeBytes:321520,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},}
Jan 30 20:33:51.241: INFO: 
Logging kubelet events for node bootstrap-e2e-minion-group-hwhs
Jan 30 20:33:51.289: INFO: 
Logging pods the kubelet thinks is on node bootstrap-e2e-minion-group-hwhs
Jan 30 20:33:51.433: INFO: kube-proxy-bootstrap-e2e-minion-group-hwhs started at 2023-01-30 17:11:38 +0000 UTC (0+1 container statuses recorded)
Jan 30 20:33:51.433: INFO: 	Container kube-proxy ready: true, restart count 1
Jan 30 20:33:51.433: INFO: metadata-proxy-v0.1-tp2qz started at 2023-01-30 17:11:39 +0000 UTC (0+2 container statuses recorded)
Jan 30 20:33:51.433: INFO: 	Container metadata-proxy ready: true, restart count 1
Jan 30 20:33:51.433: INFO: 	Container prometheus-to-sd-exporter ready: true, restart count 1
Jan 30 20:33:51.433: INFO: konnectivity-agent-9r6m6 started at 2023-01-30 17:11:54 +0000 UTC (0+1 container statuses recorded)
Jan 30 20:33:51.433: INFO: 	Container konnectivity-agent ready: true, restart count 1
Jan 30 20:33:51.433: INFO: metrics-server-v0.5.2-867b8754b9-dmjcn started at 2023-01-30 17:12:01 +0000 UTC (0+2 container statuses recorded)
Jan 30 20:33:51.433: INFO: 	Container metrics-server ready: true, restart count 1
Jan 30 20:33:51.433: INFO: 	Container metrics-server-nanny ready: true, restart count 1
Jan 30 20:33:51.433: INFO: volume-snapshot-controller-0 started at 2023-01-30 17:52:56 +0000 UTC (0+1 container statuses recorded)
Jan 30 20:33:51.433: INFO: 	Container volume-snapshot-controller ready: true, restart count 1
Jan 30 20:33:51.433: INFO: coredns-6846b5b5f-zwbrf started at 2023-01-30 20:26:34 +0000 UTC (0+1 container statuses recorded)
Jan 30 20:33:51.433: INFO: 	Container coredns ready: true, restart count 1
Jan 30 20:33:51.609: INFO: 
Latency metrics for node bootstrap-e2e-minion-group-hwhs
END STEP: dump namespace information after failure - test/e2e/framework/framework.go:284 @ 01/30/23 20:33:51.609 (1.619s)
< Exit [DeferCleanup (Each)] [sig-cloud-provider-gcp] Restart [Disruptive] - dump namespaces | framework.go:206 @ 01/30/23 20:33:51.609 (1.619s)
> Enter [DeferCleanup (Each)] [sig-cloud-provider-gcp] Restart [Disruptive] - tear down framework | framework.go:203 @ 01/30/23 20:33:51.609
STEP: Destroying namespace "restart-7764" for this suite. - test/e2e/framework/framework.go:347 @ 01/30/23 20:33:51.609
< Exit [DeferCleanup (Each)] [sig-cloud-provider-gcp] Restart [Disruptive] - tear down framework | framework.go:203 @ 01/30/23 20:33:51.658 (49ms)
> Enter [ReportAfterEach] TOP-LEVEL - test/e2e/e2e_test.go:144 @ 01/30/23 20:33:51.658
< Exit [ReportAfterEach] TOP-LEVEL - test/e2e/e2e_test.go:144 @ 01/30/23 20:33:51.658 (0s)

				Click to see stdout/stderrfrom ginkgo_report.xml

Filter through log files | View test history on testgrid

Kubernetes e2e suite [It] [sig-cloud-provider-gcp] Restart [Disruptive] should restart all nodes and ensure all nodes and pods recover 7m15s

go run hack/e2e.go -v --test --test_args='--ginkgo.focus=Kubernetes\se2e\ssuite\s\[It\]\s\[sig\-cloud\-provider\-gcp\]\sRestart\s\[Disruptive\]\sshould\srestart\sall\snodes\sand\sensure\sall\snodes\sand\spods\srecover$'

[FAILED] couldn't find 25 pods within 5m0s; last error: expected to find 25 pods but found only 24
In [It] at: test/e2e/cloud/gcp/restart.go:115 @ 01/30/23 20:33:49.945

				> Enter [BeforeEach] [sig-cloud-provider-gcp] Restart [Disruptive] - set up framework | framework.go:188 @ 01/30/23 20:26:35.831
STEP: Creating a kubernetes client - test/e2e/framework/framework.go:208 @ 01/30/23 20:26:35.831
Jan 30 20:26:35.831: INFO: >>> kubeConfig: /workspace/.kube/config
STEP: Building a namespace api object, basename restart - test/e2e/framework/framework.go:247 @ 01/30/23 20:26:35.833
STEP: Waiting for a default service account to be provisioned in namespace - test/e2e/framework/framework.go:256 @ 01/30/23 20:26:35.959
STEP: Waiting for kube-root-ca.crt to be provisioned in namespace - test/e2e/framework/framework.go:259 @ 01/30/23 20:26:36.041
< Exit [BeforeEach] [sig-cloud-provider-gcp] Restart [Disruptive] - set up framework | framework.go:188 @ 01/30/23 20:26:36.125 (294ms)
> Enter [BeforeEach] [sig-cloud-provider-gcp] Restart [Disruptive] - test/e2e/framework/metrics/init/init.go:33 @ 01/30/23 20:26:36.125
< Exit [BeforeEach] [sig-cloud-provider-gcp] Restart [Disruptive] - test/e2e/framework/metrics/init/init.go:33 @ 01/30/23 20:26:36.125 (0s)
> Enter [BeforeEach] [sig-cloud-provider-gcp] Restart [Disruptive] - test/e2e/cloud/gcp/restart.go:55 @ 01/30/23 20:26:36.125
STEP: ensuring all nodes are ready - test/e2e/cloud/gcp/restart.go:66 @ 01/30/23 20:26:36.22
Jan 30 20:26:36.264: INFO: Cluster has reached the desired number of ready nodes 3
Jan 30 20:26:36.264: INFO: Got the following nodes before restart: [bootstrap-e2e-minion-group-5sqf bootstrap-e2e-minion-group-gr7w bootstrap-e2e-minion-group-hwhs]
STEP: ensuring all pods are running and ready - test/e2e/cloud/gcp/restart.go:71 @ 01/30/23 20:26:36.264
Jan 30 20:26:36.264: INFO: Waiting up to 5m0s for 25 pods to be running and ready, or succeeded: [kube-scheduler-bootstrap-e2e-master metadata-proxy-v0.1-g24sg volume-snapshot-controller-0 konnectivity-agent-9r6m6 konnectivity-agent-m7rcz konnectivity-server-bootstrap-e2e-master kube-controller-manager-bootstrap-e2e-master l7-default-backend-8549d69d99-xzb2m l7-lb-controller-bootstrap-e2e-master coredns-6846b5b5f-zwbrf etcd-server-bootstrap-e2e-master kube-apiserver-bootstrap-e2e-master metrics-server-v0.5.2-867b8754b9-dmjcn coredns-6846b5b5f-bkw98 konnectivity-agent-pb855 kube-addon-manager-bootstrap-e2e-master kube-dns-autoscaler-5f6455f985-7jdxx kube-proxy-bootstrap-e2e-minion-group-5sqf kube-proxy-bootstrap-e2e-minion-group-gr7w kube-proxy-bootstrap-e2e-minion-group-hwhs metadata-proxy-v0.1-g6h7m coredns-6846b5b5f-zkslx etcd-server-events-bootstrap-e2e-master metadata-proxy-v0.1-j8mg4 metadata-proxy-v0.1-tp2qz]
Jan 30 20:26:36.264: INFO: Waiting up to 5m0s for pod "metadata-proxy-v0.1-tp2qz" in namespace "kube-system" to be "running and ready, or succeeded"
Jan 30 20:26:36.264: INFO: Waiting up to 5m0s for pod "coredns-6846b5b5f-zwbrf" in namespace "kube-system" to be "running and ready, or succeeded"
Jan 30 20:26:36.264: INFO: Waiting up to 5m0s for pod "konnectivity-agent-m7rcz" in namespace "kube-system" to be "running and ready, or succeeded"
Jan 30 20:26:36.264: INFO: Waiting up to 5m0s for pod "kube-controller-manager-bootstrap-e2e-master" in namespace "kube-system" to be "running and ready, or succeeded"
Jan 30 20:26:36.264: INFO: Waiting up to 5m0s for pod "metadata-proxy-v0.1-g24sg" in namespace "kube-system" to be "running and ready, or succeeded"
Jan 30 20:26:36.264: INFO: Waiting up to 5m0s for pod "l7-default-backend-8549d69d99-xzb2m" in namespace "kube-system" to be "running and ready, or succeeded"
Jan 30 20:26:36.264: INFO: Waiting up to 5m0s for pod "volume-snapshot-controller-0" in namespace "kube-system" to be "running and ready, or succeeded"
Jan 30 20:26:36.264: INFO: Waiting up to 5m0s for pod "konnectivity-server-bootstrap-e2e-master" in namespace "kube-system" to be "running and ready, or succeeded"
Jan 30 20:26:36.265: INFO: Waiting up to 5m0s for pod "kube-scheduler-bootstrap-e2e-master" in namespace "kube-system" to be "running and ready, or succeeded"
Jan 30 20:26:36.265: INFO: Waiting up to 5m0s for pod "l7-lb-controller-bootstrap-e2e-master" in namespace "kube-system" to be "running and ready, or succeeded"
Jan 30 20:26:36.265: INFO: Waiting up to 5m0s for pod "konnectivity-agent-9r6m6" in namespace "kube-system" to be "running and ready, or succeeded"
Jan 30 20:26:36.265: INFO: Waiting up to 5m0s for pod "metrics-server-v0.5.2-867b8754b9-dmjcn" in namespace "kube-system" to be "running and ready, or succeeded"
Jan 30 20:26:36.265: INFO: Waiting up to 5m0s for pod "kube-apiserver-bootstrap-e2e-master" in namespace "kube-system" to be "running and ready, or succeeded"
Jan 30 20:26:36.265: INFO: Waiting up to 5m0s for pod "etcd-server-bootstrap-e2e-master" in namespace "kube-system" to be "running and ready, or succeeded"
Jan 30 20:26:36.265: INFO: Waiting up to 5m0s for pod "konnectivity-agent-pb855" in namespace "kube-system" to be "running and ready, or succeeded"
Jan 30 20:26:36.265: INFO: Waiting up to 5m0s for pod "kube-dns-autoscaler-5f6455f985-7jdxx" in namespace "kube-system" to be "running and ready, or succeeded"
Jan 30 20:26:36.265: INFO: Waiting up to 5m0s for pod "kube-addon-manager-bootstrap-e2e-master" in namespace "kube-system" to be "running and ready, or succeeded"
Jan 30 20:26:36.265: INFO: Waiting up to 5m0s for pod "coredns-6846b5b5f-bkw98" in namespace "kube-system" to be "running and ready, or succeeded"
Jan 30 20:26:36.265: INFO: Waiting up to 5m0s for pod "metadata-proxy-v0.1-g6h7m" in namespace "kube-system" to be "running and ready, or succeeded"
Jan 30 20:26:36.265: INFO: Waiting up to 5m0s for pod "kube-proxy-bootstrap-e2e-minion-group-gr7w" in namespace "kube-system" to be "running and ready, or succeeded"
Jan 30 20:26:36.265: INFO: Waiting up to 5m0s for pod "coredns-6846b5b5f-zkslx" in namespace "kube-system" to be "running and ready, or succeeded"
Jan 30 20:26:36.265: INFO: Waiting up to 5m0s for pod "kube-proxy-bootstrap-e2e-minion-group-hwhs" in namespace "kube-system" to be "running and ready, or succeeded"
Jan 30 20:26:36.265: INFO: Waiting up to 5m0s for pod "etcd-server-events-bootstrap-e2e-master" in namespace "kube-system" to be "running and ready, or succeeded"
Jan 30 20:26:36.265: INFO: Waiting up to 5m0s for pod "kube-proxy-bootstrap-e2e-minion-group-5sqf" in namespace "kube-system" to be "running and ready, or succeeded"
Jan 30 20:26:36.265: INFO: Waiting up to 5m0s for pod "metadata-proxy-v0.1-j8mg4" in namespace "kube-system" to be "running and ready, or succeeded"
Jan 30 20:26:36.310: INFO: Pod "l7-default-backend-8549d69d99-xzb2m": Phase="Running", Reason="", readiness=true. Elapsed: 45.652353ms
Jan 30 20:26:36.310: INFO: Pod "l7-default-backend-8549d69d99-xzb2m" satisfied condition "running and ready, or succeeded"
Jan 30 20:26:36.315: INFO: Pod "metadata-proxy-v0.1-tp2qz": Phase="Running", Reason="", readiness=true. Elapsed: 51.068739ms
Jan 30 20:26:36.315: INFO: Pod "metadata-proxy-v0.1-tp2qz" satisfied condition "running and ready, or succeeded"
Jan 30 20:26:36.315: INFO: Pod "konnectivity-agent-m7rcz": Phase="Running", Reason="", readiness=true. Elapsed: 51.076042ms
Jan 30 20:26:36.315: INFO: Pod "konnectivity-agent-m7rcz" satisfied condition "running and ready, or succeeded"
Jan 30 20:26:36.315: INFO: Pod "metadata-proxy-v0.1-g24sg": Phase="Running", Reason="", readiness=true. Elapsed: 51.040429ms
Jan 30 20:26:36.315: INFO: Pod "metadata-proxy-v0.1-g24sg" satisfied condition "running and ready, or succeeded"
Jan 30 20:26:36.316: INFO: Pod "kube-controller-manager-bootstrap-e2e-master": Phase="Running", Reason="", readiness=true. Elapsed: 51.125586ms
Jan 30 20:26:36.316: INFO: Pod "kube-controller-manager-bootstrap-e2e-master" satisfied condition "running and ready, or succeeded"
Jan 30 20:26:36.316: INFO: Pod "coredns-6846b5b5f-zwbrf": Phase="Pending", Reason="", readiness=false. Elapsed: 51.26483ms
Jan 30 20:26:36.316: INFO: Error evaluating pod condition running and ready, or succeeded: want pod 'coredns-6846b5b5f-zwbrf' on 'bootstrap-e2e-minion-group-hwhs' to be 'Running' but was 'Pending'
Jan 30 20:26:36.364: INFO: Pod "etcd-server-events-bootstrap-e2e-master": Phase="Running", Reason="", readiness=true. Elapsed: 98.842963ms
Jan 30 20:26:36.364: INFO: Pod "etcd-server-events-bootstrap-e2e-master" satisfied condition "running and ready, or succeeded"
Jan 30 20:26:36.364: INFO: Pod "volume-snapshot-controller-0": Phase="Running", Reason="", readiness=true. Elapsed: 99.077606ms
Jan 30 20:26:36.364: INFO: Pod "volume-snapshot-controller-0" satisfied condition "running and ready, or succeeded"
Jan 30 20:26:36.365: INFO: Pod "kube-dns-autoscaler-5f6455f985-7jdxx": Phase="Running", Reason="", readiness=true. Elapsed: 100.147738ms
Jan 30 20:26:36.365: INFO: Pod "kube-dns-autoscaler-5f6455f985-7jdxx" satisfied condition "running and ready, or succeeded"
Jan 30 20:26:36.370: INFO: Pod "kube-proxy-bootstrap-e2e-minion-group-5sqf": Phase="Running", Reason="", readiness=true. Elapsed: 105.110825ms
Jan 30 20:26:36.370: INFO: Pod "kube-proxy-bootstrap-e2e-minion-group-5sqf" satisfied condition "running and ready, or succeeded"
Jan 30 20:26:36.370: INFO: Pod "kube-proxy-bootstrap-e2e-minion-group-hwhs": Phase="Running", Reason="", readiness=true. Elapsed: 105.201467ms
Jan 30 20:26:36.370: INFO: Pod "kube-proxy-bootstrap-e2e-minion-group-hwhs" satisfied condition "running and ready, or succeeded"
Jan 30 20:26:36.370: INFO: Pod "metadata-proxy-v0.1-j8mg4": Phase="Running", Reason="", readiness=true. Elapsed: 105.202086ms
Jan 30 20:26:36.370: INFO: Pod "metadata-proxy-v0.1-j8mg4" satisfied condition "running and ready, or succeeded"
Jan 30 20:26:36.370: INFO: Pod "metrics-server-v0.5.2-867b8754b9-dmjcn": Phase="Running", Reason="", readiness=true. Elapsed: 105.404977ms
Jan 30 20:26:36.370: INFO: Pod "metrics-server-v0.5.2-867b8754b9-dmjcn" satisfied condition "running and ready, or succeeded"
Jan 30 20:26:36.370: INFO: Pod "kube-scheduler-bootstrap-e2e-master": Phase="Running", Reason="", readiness=true. Elapsed: 105.484552ms
Jan 30 20:26:36.370: INFO: Pod "kube-scheduler-bootstrap-e2e-master" satisfied condition "running and ready, or succeeded"
Jan 30 20:26:36.370: INFO: Pod "coredns-6846b5b5f-zkslx": Phase="Running", Reason="", readiness=true. Elapsed: 105.3485ms
Jan 30 20:26:36.370: INFO: Pod "coredns-6846b5b5f-zkslx" satisfied condition "running and ready, or succeeded"
Jan 30 20:26:36.370: INFO: Pod "konnectivity-server-bootstrap-e2e-master": Phase="Running", Reason="", readiness=true. Elapsed: 105.615831ms
Jan 30 20:26:36.370: INFO: Pod "konnectivity-server-bootstrap-e2e-master" satisfied condition "running and ready, or succeeded"
Jan 30 20:26:36.373: INFO: Pod "kube-proxy-bootstrap-e2e-minion-group-gr7w": Phase="Running", Reason="", readiness=true. Elapsed: 108.131203ms
Jan 30 20:26:36.373: INFO: Pod "kube-proxy-bootstrap-e2e-minion-group-gr7w" satisfied condition "running and ready, or succeeded"
Jan 30 20:26:36.373: INFO: Pod "konnectivity-agent-pb855": Phase="Running", Reason="", readiness=true. Elapsed: 108.257416ms
Jan 30 20:26:36.373: INFO: Pod "konnectivity-agent-pb855" satisfied condition "running and ready, or succeeded"
Jan 30 20:26:36.373: INFO: Pod "metadata-proxy-v0.1-g6h7m": Phase="Running", Reason="", readiness=true. Elapsed: 108.221554ms
Jan 30 20:26:36.373: INFO: Pod "metadata-proxy-v0.1-g6h7m" satisfied condition "running and ready, or succeeded"
Jan 30 20:26:36.373: INFO: Pod "coredns-6846b5b5f-bkw98": Phase="Running", Reason="", readiness=true. Elapsed: 108.261592ms
Jan 30 20:26:36.373: INFO: Pod "coredns-6846b5b5f-bkw98" satisfied condition "running and ready, or succeeded"
Jan 30 20:26:36.373: INFO: Pod "l7-lb-controller-bootstrap-e2e-master": Phase="Running", Reason="", readiness=true. Elapsed: 108.388434ms
Jan 30 20:26:36.373: INFO: Pod "etcd-server-bootstrap-e2e-master": Phase="Running", Reason="", readiness=true. Elapsed: 108.33426ms
Jan 30 20:26:36.373: INFO: Pod "l7-lb-controller-bootstrap-e2e-master" satisfied condition "running and ready, or succeeded"
Jan 30 20:26:36.373: INFO: Pod "etcd-server-bootstrap-e2e-master" satisfied condition "running and ready, or succeeded"
Jan 30 20:26:36.373: INFO: Pod "kube-addon-manager-bootstrap-e2e-master": Phase="Running", Reason="", readiness=true. Elapsed: 108.320757ms
Jan 30 20:26:36.373: INFO: Pod "kube-addon-manager-bootstrap-e2e-master" satisfied condition "running and ready, or succeeded"
Jan 30 20:26:36.373: INFO: Pod "konnectivity-agent-9r6m6": Phase="Running", Reason="", readiness=true. Elapsed: 108.390517ms
Jan 30 20:26:36.373: INFO: Pod "konnectivity-agent-9r6m6" satisfied condition "running and ready, or succeeded"
Jan 30 20:26:36.373: INFO: Pod "kube-apiserver-bootstrap-e2e-master": Phase="Running", Reason="", readiness=true. Elapsed: 108.40052ms
Jan 30 20:26:36.373: INFO: Pod "kube-apiserver-bootstrap-e2e-master" satisfied condition "running and ready, or succeeded"
Jan 30 20:26:38.360: INFO: Pod "coredns-6846b5b5f-zwbrf": Phase="Running", Reason="", readiness=true. Elapsed: 2.095349051s
Jan 30 20:26:38.360: INFO: Pod "coredns-6846b5b5f-zwbrf" satisfied condition "running and ready, or succeeded"
Jan 30 20:26:38.360: INFO: Wanted all 25 pods to be running and ready, or succeeded. Result: true. Pods: [kube-scheduler-bootstrap-e2e-master metadata-proxy-v0.1-g24sg volume-snapshot-controller-0 konnectivity-agent-9r6m6 konnectivity-agent-m7rcz konnectivity-server-bootstrap-e2e-master kube-controller-manager-bootstrap-e2e-master l7-default-backend-8549d69d99-xzb2m l7-lb-controller-bootstrap-e2e-master coredns-6846b5b5f-zwbrf etcd-server-bootstrap-e2e-master kube-apiserver-bootstrap-e2e-master metrics-server-v0.5.2-867b8754b9-dmjcn coredns-6846b5b5f-bkw98 konnectivity-agent-pb855 kube-addon-manager-bootstrap-e2e-master kube-dns-autoscaler-5f6455f985-7jdxx kube-proxy-bootstrap-e2e-minion-group-5sqf kube-proxy-bootstrap-e2e-minion-group-gr7w kube-proxy-bootstrap-e2e-minion-group-hwhs metadata-proxy-v0.1-g6h7m coredns-6846b5b5f-zkslx etcd-server-events-bootstrap-e2e-master metadata-proxy-v0.1-j8mg4 metadata-proxy-v0.1-tp2qz]
< Exit [BeforeEach] [sig-cloud-provider-gcp] Restart [Disruptive] - test/e2e/cloud/gcp/restart.go:55 @ 01/30/23 20:26:38.36 (2.234s)
> Enter [It] should restart all nodes and ensure all nodes and pods recover - test/e2e/cloud/gcp/restart.go:91 @ 01/30/23 20:26:38.36
STEP: restarting all of the nodes - test/e2e/cloud/gcp/restart.go:92 @ 01/30/23 20:26:38.36
Jan 30 20:26:38.360: INFO: Running gcloud [compute --project=k8s-infra-e2e-boskos-080 instances reset bootstrap-e2e-minion-group-5sqf bootstrap-e2e-minion-group-gr7w bootstrap-e2e-minion-group-hwhs --zone=us-west1-b]
STEP: ensuring all nodes are ready after the restart - test/e2e/cloud/gcp/restart.go:96 @ 01/30/23 20:28:49.899
Jan 30 20:28:49.943: INFO: Cluster has reached the desired number of ready nodes 3
Jan 30 20:28:49.943: INFO: Got the following nodes after restart: [bootstrap-e2e-minion-group-5sqf bootstrap-e2e-minion-group-gr7w bootstrap-e2e-minion-group-hwhs]
STEP: ensuring the same number of nodes exist after the restart - test/e2e/cloud/gcp/restart.go:103 @ 01/30/23 20:28:49.943
STEP: ensuring the same number of pods are running and ready after restart - test/e2e/cloud/gcp/restart.go:112 @ 01/30/23 20:28:49.943
Jan 30 20:28:51.944: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:28:53.945: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:28:55.945: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:28:57.944: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:28:59.945: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:29:01.944: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:29:03.945: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:29:05.944: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:29:07.944: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:29:09.945: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:29:11.944: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:29:13.945: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:29:15.945: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:29:17.945: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:29:19.945: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:29:21.944: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:29:23.945: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:29:25.945: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:29:27.944: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:29:29.945: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:29:31.945: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:29:33.945: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:29:35.944: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:29:37.944: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:29:39.945: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:29:41.944: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:29:43.945: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:29:45.946: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:29:47.945: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:29:49.945: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:29:51.944: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:29:53.945: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:29:55.945: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:29:57.944: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:29:59.945: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:30:01.944: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:30:03.945: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:30:05.944: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:30:07.944: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:30:09.945: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:30:11.944: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:30:13.945: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:30:15.945: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:30:17.945: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:30:19.945: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:30:21.944: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:30:23.945: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:30:25.945: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:30:27.944: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:30:29.944: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:30:31.944: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:30:33.945: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:30:35.944: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:30:37.944: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:30:39.945: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:30:41.945: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:30:43.945: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:30:45.945: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:30:47.944: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:30:49.944: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:30:51.944: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:30:53.945: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:30:55.944: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:30:57.944: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:30:59.944: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:31:01.944: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:31:03.945: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:31:05.944: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:31:07.944: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:31:09.945: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:31:11.944: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:31:13.945: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:31:15.945: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:31:17.944: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:31:19.945: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:31:21.944: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:31:23.944: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:31:25.945: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:31:27.944: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:31:29.945: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:31:31.944: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:31:33.944: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:31:35.945: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:31:37.944: INFO: Error getting pods: expected to find 25 pods but found only 24
Automatically polling progress:
  [sig-cloud-provider-gcp] Restart [Disruptive] should restart all nodes and ensure all nodes and pods recover (Spec Runtime: 5m2.529s)
    test/e2e/cloud/gcp/restart.go:91
    In [It] (Node Runtime: 5m0.001s)
      test/e2e/cloud/gcp/restart.go:91
      At [By Step] ensuring the same number of pods are running and ready after restart (Step Runtime: 2m48.417s)
        test/e2e/cloud/gcp/restart.go:112

      Spec Goroutine
      goroutine 44796 [select]
        k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.waitForWithContext({0x7f2fec1df798, 0xc008507640}, 0xc00634db00, 0x302ee4a?)
          vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:647
        k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.poll({0x7f2fec1df798, 0xc008507640}, 0x44?, 0x0?, 0x0?)
          vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:600
        k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.PollWithContext({0x7f2fec1df798, 0xc008507640}, 0x0?, 0x0?, 0x0?)
          vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:468
        k8s.io/kubernetes/test/e2e/framework/pod.WaitForNRestartablePods({0x7f2fec1df798?, 0xc008507640?}, 0x0?, 0x19, 0xc005555f20?)
          test/e2e/framework/pod/wait.go:757
      > k8s.io/kubernetes/test/e2e/cloud/gcp.glob..func11.3({0x7f2fec1df798, 0xc008507640})
          test/e2e/cloud/gcp/restart.go:114
        k8s.io/kubernetes/vendor/github.com/onsi/ginkgo/v2/internal.extractBodyFunction.func2({0x8111f08?, 0xc008507640})
          vendor/github.com/onsi/ginkgo/v2/internal/node.go:452
        k8s.io/kubernetes/vendor/github.com/onsi/ginkgo/v2/internal.(*Suite).runNode.func3()
          vendor/github.com/onsi/ginkgo/v2/internal/suite.go:854
        k8s.io/kubernetes/vendor/github.com/onsi/ginkgo/v2/internal.(*Suite).runNode
          vendor/github.com/onsi/ginkgo/v2/internal/suite.go:841
Jan 30 20:31:39.944: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:31:41.945: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:31:43.945: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:31:45.945: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:31:47.944: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:31:49.945: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:31:51.944: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:31:53.945: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:31:55.945: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:31:57.945: INFO: Error getting pods: expected to find 25 pods but found only 24
Automatically polling progress:
  [sig-cloud-provider-gcp] Restart [Disruptive] should restart all nodes and ensure all nodes and pods recover (Spec Runtime: 5m22.531s)
    test/e2e/cloud/gcp/restart.go:91
    In [It] (Node Runtime: 5m20.002s)
      test/e2e/cloud/gcp/restart.go:91
      At [By Step] ensuring the same number of pods are running and ready after restart (Step Runtime: 3m8.419s)
        test/e2e/cloud/gcp/restart.go:112

      Spec Goroutine
      goroutine 44796 [select]
        k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.waitForWithContext({0x7f2fec1df798, 0xc008507640}, 0xc00634db00, 0x302ee4a?)
          vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:647
        k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.poll({0x7f2fec1df798, 0xc008507640}, 0x44?, 0x0?, 0x0?)
          vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:600
        k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.PollWithContext({0x7f2fec1df798, 0xc008507640}, 0x0?, 0x0?, 0x0?)
          vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:468
        k8s.io/kubernetes/test/e2e/framework/pod.WaitForNRestartablePods({0x7f2fec1df798?, 0xc008507640?}, 0x0?, 0x19, 0xc005555f20?)
          test/e2e/framework/pod/wait.go:757
      > k8s.io/kubernetes/test/e2e/cloud/gcp.glob..func11.3({0x7f2fec1df798, 0xc008507640})
          test/e2e/cloud/gcp/restart.go:114
        k8s.io/kubernetes/vendor/github.com/onsi/ginkgo/v2/internal.extractBodyFunction.func2({0x8111f08?, 0xc008507640})
          vendor/github.com/onsi/ginkgo/v2/internal/node.go:452
        k8s.io/kubernetes/vendor/github.com/onsi/ginkgo/v2/internal.(*Suite).runNode.func3()
          vendor/github.com/onsi/ginkgo/v2/internal/suite.go:854
        k8s.io/kubernetes/vendor/github.com/onsi/ginkgo/v2/internal.(*Suite).runNode
          vendor/github.com/onsi/ginkgo/v2/internal/suite.go:841
Jan 30 20:31:59.944: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:32:01.944: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:32:03.945: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:32:05.944: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:32:07.944: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:32:09.945: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:32:11.944: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:32:13.944: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:32:15.945: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:32:17.944: INFO: Error getting pods: expected to find 25 pods but found only 24
Automatically polling progress:
  [sig-cloud-provider-gcp] Restart [Disruptive] should restart all nodes and ensure all nodes and pods recover (Spec Runtime: 5m42.532s)
    test/e2e/cloud/gcp/restart.go:91
    In [It] (Node Runtime: 5m40.004s)
      test/e2e/cloud/gcp/restart.go:91
      At [By Step] ensuring the same number of pods are running and ready after restart (Step Runtime: 3m28.42s)
        test/e2e/cloud/gcp/restart.go:112

      Spec Goroutine
      goroutine 44796 [select]
        k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.waitForWithContext({0x7f2fec1df798, 0xc008507640}, 0xc00634db00, 0x302ee4a?)
          vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:647
        k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.poll({0x7f2fec1df798, 0xc008507640}, 0x44?, 0x0?, 0x0?)
          vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:600
        k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.PollWithContext({0x7f2fec1df798, 0xc008507640}, 0x0?, 0x0?, 0x0?)
          vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:468
        k8s.io/kubernetes/test/e2e/framework/pod.WaitForNRestartablePods({0x7f2fec1df798?, 0xc008507640?}, 0x0?, 0x19, 0xc005555f20?)
          test/e2e/framework/pod/wait.go:757
      > k8s.io/kubernetes/test/e2e/cloud/gcp.glob..func11.3({0x7f2fec1df798, 0xc008507640})
          test/e2e/cloud/gcp/restart.go:114
        k8s.io/kubernetes/vendor/github.com/onsi/ginkgo/v2/internal.extractBodyFunction.func2({0x8111f08?, 0xc008507640})
          vendor/github.com/onsi/ginkgo/v2/internal/node.go:452
        k8s.io/kubernetes/vendor/github.com/onsi/ginkgo/v2/internal.(*Suite).runNode.func3()
          vendor/github.com/onsi/ginkgo/v2/internal/suite.go:854
        k8s.io/kubernetes/vendor/github.com/onsi/ginkgo/v2/internal.(*Suite).runNode
          vendor/github.com/onsi/ginkgo/v2/internal/suite.go:841
Jan 30 20:32:19.944: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:32:21.944: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:32:23.944: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:32:25.944: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:32:27.944: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:32:29.944: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:32:31.944: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:32:33.944: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:32:35.945: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:32:37.945: INFO: Error getting pods: expected to find 25 pods but found only 24
Automatically polling progress:
  [sig-cloud-provider-gcp] Restart [Disruptive] should restart all nodes and ensure all nodes and pods recover (Spec Runtime: 6m2.535s)
    test/e2e/cloud/gcp/restart.go:91
    In [It] (Node Runtime: 6m0.006s)
      test/e2e/cloud/gcp/restart.go:91
      At [By Step] ensuring the same number of pods are running and ready after restart (Step Runtime: 3m48.423s)
        test/e2e/cloud/gcp/restart.go:112

      Spec Goroutine
      goroutine 44796 [select]
        k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.waitForWithContext({0x7f2fec1df798, 0xc008507640}, 0xc00634db00, 0x302ee4a?)
          vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:647
        k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.poll({0x7f2fec1df798, 0xc008507640}, 0x44?, 0x0?, 0x0?)
          vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:600
        k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.PollWithContext({0x7f2fec1df798, 0xc008507640}, 0x0?, 0x0?, 0x0?)
          vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:468
        k8s.io/kubernetes/test/e2e/framework/pod.WaitForNRestartablePods({0x7f2fec1df798?, 0xc008507640?}, 0x0?, 0x19, 0xc005555f20?)
          test/e2e/framework/pod/wait.go:757
      > k8s.io/kubernetes/test/e2e/cloud/gcp.glob..func11.3({0x7f2fec1df798, 0xc008507640})
          test/e2e/cloud/gcp/restart.go:114
        k8s.io/kubernetes/vendor/github.com/onsi/ginkgo/v2/internal.extractBodyFunction.func2({0x8111f08?, 0xc008507640})
          vendor/github.com/onsi/ginkgo/v2/internal/node.go:452
        k8s.io/kubernetes/vendor/github.com/onsi/ginkgo/v2/internal.(*Suite).runNode.func3()
          vendor/github.com/onsi/ginkgo/v2/internal/suite.go:854
        k8s.io/kubernetes/vendor/github.com/onsi/ginkgo/v2/internal.(*Suite).runNode
          vendor/github.com/onsi/ginkgo/v2/internal/suite.go:841
Jan 30 20:32:39.944: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:32:41.944: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:32:43.944: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:32:45.944: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:32:47.944: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:32:49.944: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:32:51.945: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:32:53.945: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:32:55.945: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:32:57.944: INFO: Error getting pods: expected to find 25 pods but found only 24
Automatically polling progress:
  [sig-cloud-provider-gcp] Restart [Disruptive] should restart all nodes and ensure all nodes and pods recover (Spec Runtime: 6m22.537s)
    test/e2e/cloud/gcp/restart.go:91
    In [It] (Node Runtime: 6m20.008s)
      test/e2e/cloud/gcp/restart.go:91
      At [By Step] ensuring the same number of pods are running and ready after restart (Step Runtime: 4m8.425s)
        test/e2e/cloud/gcp/restart.go:112

      Spec Goroutine
      goroutine 44796 [select]
        k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.waitForWithContext({0x7f2fec1df798, 0xc008507640}, 0xc00634db00, 0x302ee4a?)
          vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:647
        k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.poll({0x7f2fec1df798, 0xc008507640}, 0x44?, 0x0?, 0x0?)
          vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:600
        k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.PollWithContext({0x7f2fec1df798, 0xc008507640}, 0x0?, 0x0?, 0x0?)
          vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:468
        k8s.io/kubernetes/test/e2e/framework/pod.WaitForNRestartablePods({0x7f2fec1df798?, 0xc008507640?}, 0x0?, 0x19, 0xc005555f20?)
          test/e2e/framework/pod/wait.go:757
      > k8s.io/kubernetes/test/e2e/cloud/gcp.glob..func11.3({0x7f2fec1df798, 0xc008507640})
          test/e2e/cloud/gcp/restart.go:114
        k8s.io/kubernetes/vendor/github.com/onsi/ginkgo/v2/internal.extractBodyFunction.func2({0x8111f08?, 0xc008507640})
          vendor/github.com/onsi/ginkgo/v2/internal/node.go:452
        k8s.io/kubernetes/vendor/github.com/onsi/ginkgo/v2/internal.(*Suite).runNode.func3()
          vendor/github.com/onsi/ginkgo/v2/internal/suite.go:854
        k8s.io/kubernetes/vendor/github.com/onsi/ginkgo/v2/internal.(*Suite).runNode
          vendor/github.com/onsi/ginkgo/v2/internal/suite.go:841
Jan 30 20:32:59.945: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:33:01.945: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:33:03.945: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:33:05.944: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:33:07.944: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:33:09.945: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:33:11.944: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:33:13.944: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:33:15.945: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:33:17.945: INFO: Error getting pods: expected to find 25 pods but found only 24
Automatically polling progress:
  [sig-cloud-provider-gcp] Restart [Disruptive] should restart all nodes and ensure all nodes and pods recover (Spec Runtime: 6m42.539s)
    test/e2e/cloud/gcp/restart.go:91
    In [It] (Node Runtime: 6m40.01s)
      test/e2e/cloud/gcp/restart.go:91
      At [By Step] ensuring the same number of pods are running and ready after restart (Step Runtime: 4m28.427s)
        test/e2e/cloud/gcp/restart.go:112

      Spec Goroutine
      goroutine 44796 [select]
        k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.waitForWithContext({0x7f2fec1df798, 0xc008507640}, 0xc00634db00, 0x302ee4a?)
          vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:647
        k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.poll({0x7f2fec1df798, 0xc008507640}, 0x44?, 0x0?, 0x0?)
          vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:600
        k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.PollWithContext({0x7f2fec1df798, 0xc008507640}, 0x0?, 0x0?, 0x0?)
          vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:468
        k8s.io/kubernetes/test/e2e/framework/pod.WaitForNRestartablePods({0x7f2fec1df798?, 0xc008507640?}, 0x0?, 0x19, 0xc005555f20?)
          test/e2e/framework/pod/wait.go:757
      > k8s.io/kubernetes/test/e2e/cloud/gcp.glob..func11.3({0x7f2fec1df798, 0xc008507640})
          test/e2e/cloud/gcp/restart.go:114
        k8s.io/kubernetes/vendor/github.com/onsi/ginkgo/v2/internal.extractBodyFunction.func2({0x8111f08?, 0xc008507640})
          vendor/github.com/onsi/ginkgo/v2/internal/node.go:452
        k8s.io/kubernetes/vendor/github.com/onsi/ginkgo/v2/internal.(*Suite).runNode.func3()
          vendor/github.com/onsi/ginkgo/v2/internal/suite.go:854
        k8s.io/kubernetes/vendor/github.com/onsi/ginkgo/v2/internal.(*Suite).runNode
          vendor/github.com/onsi/ginkgo/v2/internal/suite.go:841
Jan 30 20:33:19.945: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:33:21.944: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:33:23.944: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:33:25.945: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:33:27.944: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:33:29.944: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:33:31.944: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:33:33.944: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:33:35.945: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:33:37.945: INFO: Error getting pods: expected to find 25 pods but found only 24
Automatically polling progress:
  [sig-cloud-provider-gcp] Restart [Disruptive] should restart all nodes and ensure all nodes and pods recover (Spec Runtime: 7m2.541s)
    test/e2e/cloud/gcp/restart.go:91
    In [It] (Node Runtime: 7m0.012s)
      test/e2e/cloud/gcp/restart.go:91
      At [By Step] ensuring the same number of pods are running and ready after restart (Step Runtime: 4m48.429s)
        test/e2e/cloud/gcp/restart.go:112

      Spec Goroutine
      goroutine 44796 [select]
        k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.waitForWithContext({0x7f2fec1df798, 0xc008507640}, 0xc00634db00, 0x302ee4a?)
          vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:647
        k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.poll({0x7f2fec1df798, 0xc008507640}, 0x44?, 0x0?, 0x0?)
          vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:600
        k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.PollWithContext({0x7f2fec1df798, 0xc008507640}, 0x0?, 0x0?, 0x0?)
          vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:468
        k8s.io/kubernetes/test/e2e/framework/pod.WaitForNRestartablePods({0x7f2fec1df798?, 0xc008507640?}, 0x0?, 0x19, 0xc005555f20?)
          test/e2e/framework/pod/wait.go:757
      > k8s.io/kubernetes/test/e2e/cloud/gcp.glob..func11.3({0x7f2fec1df798, 0xc008507640})
          test/e2e/cloud/gcp/restart.go:114
        k8s.io/kubernetes/vendor/github.com/onsi/ginkgo/v2/internal.extractBodyFunction.func2({0x8111f08?, 0xc008507640})
          vendor/github.com/onsi/ginkgo/v2/internal/node.go:452
        k8s.io/kubernetes/vendor/github.com/onsi/ginkgo/v2/internal.(*Suite).runNode.func3()
          vendor/github.com/onsi/ginkgo/v2/internal/suite.go:854
        k8s.io/kubernetes/vendor/github.com/onsi/ginkgo/v2/internal.(*Suite).runNode
          vendor/github.com/onsi/ginkgo/v2/internal/suite.go:841
Jan 30 20:33:39.944: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:33:41.944: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:33:43.945: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:33:45.944: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:33:47.945: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:33:49.945: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:33:49.945: INFO: Error getting pods: expected to find 25 pods but found only 24
Jan 30 20:33:49.945: INFO: Unexpected error: 
    <*errors.errorString | 0xc0014c8a80>: {
        s: "couldn't find 25 pods within 5m0s; last error: expected to find 25 pods but found only 24",
    }
[FAILED] couldn't find 25 pods within 5m0s; last error: expected to find 25 pods but found only 24
In [It] at: test/e2e/cloud/gcp/restart.go:115 @ 01/30/23 20:33:49.945
< Exit [It] should restart all nodes and ensure all nodes and pods recover - test/e2e/cloud/gcp/restart.go:91 @ 01/30/23 20:33:49.945 (7m11.585s)
> Enter [AfterEach] [sig-cloud-provider-gcp] Restart [Disruptive] - test/e2e/framework/node/init/init.go:33 @ 01/30/23 20:33:49.945
Jan 30 20:33:49.945: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready
< Exit [AfterEach] [sig-cloud-provider-gcp] Restart [Disruptive] - test/e2e/framework/node/init/init.go:33 @ 01/30/23 20:33:49.99 (45ms)
> Enter [AfterEach] [sig-cloud-provider-gcp] Restart [Disruptive] - test/e2e/cloud/gcp/restart.go:85 @ 01/30/23 20:33:49.99
< Exit [AfterEach] [sig-cloud-provider-gcp] Restart [Disruptive] - test/e2e/cloud/gcp/restart.go:85 @ 01/30/23 20:33:49.99 (0s)
> Enter [DeferCleanup (Each)] [sig-cloud-provider-gcp] Restart [Disruptive] - test/e2e/framework/metrics/init/init.go:35 @ 01/30/23 20:33:49.99
< Exit [DeferCleanup (Each)] [sig-cloud-provider-gcp] Restart [Disruptive] - test/e2e/framework/metrics/init/init.go:35 @ 01/30/23 20:33:49.99 (0s)
> Enter [DeferCleanup (Each)] [sig-cloud-provider-gcp] Restart [Disruptive] - dump namespaces | framework.go:206 @ 01/30/23 20:33:49.99
STEP: dump namespace information after failure - test/e2e/framework/framework.go:284 @ 01/30/23 20:33:49.99
STEP: Collecting events from namespace "restart-7764". - test/e2e/framework/debug/dump.go:42 @ 01/30/23 20:33:49.99
STEP: Found 0 events. - test/e2e/framework/debug/dump.go:46 @ 01/30/23 20:33:50.032
Jan 30 20:33:50.074: INFO: POD  NODE  PHASE  GRACE  CONDITIONS
Jan 30 20:33:50.074: INFO: 
Jan 30 20:33:50.117: INFO: 
Logging node info for node bootstrap-e2e-master
Jan 30 20:33:50.160: INFO: Node Info: &Node{ObjectMeta:{bootstrap-e2e-master    11f6bd22-6cdc-4ea1-9f3a-81217ad4db12 52894 0 2023-01-30 17:11:34 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:n1-standard-1 beta.kubernetes.io/os:linux cloud.google.com/metadata-proxy-ready:true failure-domain.beta.kubernetes.io/region:us-west1 failure-domain.beta.kubernetes.io/zone:us-west1-b kubernetes.io/arch:amd64 kubernetes.io/hostname:bootstrap-e2e-master kubernetes.io/os:linux node.kubernetes.io/instance-type:n1-standard-1 topology.gke.io/zone:us-west1-b topology.kubernetes.io/region:us-west1 topology.kubernetes.io/zone:us-west1-b] map[node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kubelet Update v1 2023-01-30 17:11:34 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:cloud.google.com/metadata-proxy-ready":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{},"f:unschedulable":{}}} } {kube-controller-manager Update v1 2023-01-30 17:11:45 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}}}} status} {kube-controller-manager Update v1 2023-01-30 17:11:52 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"10.64.0.0/24\"":{}},"f:taints":{}}} } {kubelet Update v1 2023-01-30 20:33:19 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:topology.gke.io/zone":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:10.64.0.0/24,DoNotUseExternalID:,ProviderID:gce://k8s-infra-e2e-boskos-080/us-west1-b/bootstrap-e2e-master,Unschedulable:true,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/master,Value:,Effect:NoSchedule,TimeAdded:<nil>,},Taint{Key:node.kubernetes.io/unschedulable,Value:,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[10.64.0.0/24],},Status:NodeStatus{Capacity:ResourceList{attachable-volumes-gce-pd: {{127 0} {<nil>} 127 DecimalSI},cpu: {{1 0} {<nil>} 1 DecimalSI},ephemeral-storage: {{16656896000 0} {<nil>} 16266500Ki BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3858378752 0} {<nil>} 3767948Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{attachable-volumes-gce-pd: {{127 0} {<nil>} 127 DecimalSI},cpu: {{1 0} {<nil>} 1 DecimalSI},ephemeral-storage: {{14991206376 0} {<nil>} 14991206376 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3596234752 0} {<nil>} 3511948Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2023-01-30 17:11:45 +0000 UTC,LastTransitionTime:2023-01-30 17:11:45 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-01-30 20:33:19 +0000 UTC,LastTransitionTime:2023-01-30 17:11:34 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-01-30 20:33:19 +0000 UTC,LastTransitionTime:2023-01-30 17:11:34 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-01-30 20:33:19 +0000 UTC,LastTransitionTime:2023-01-30 17:11:34 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-01-30 20:33:19 +0000 UTC,LastTransitionTime:2023-01-30 17:11:47 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:10.138.0.2,},NodeAddress{Type:ExternalIP,Address:34.83.218.233,},NodeAddress{Type:InternalDNS,Address:bootstrap-e2e-master.c.k8s-infra-e2e-boskos-080.internal,},NodeAddress{Type:Hostname,Address:bootstrap-e2e-master.c.k8s-infra-e2e-boskos-080.internal,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:098aa58fbfa61c6753f7d9aa561aa43a,SystemUUID:098aa58f-bfa6-1c67-53f7-d9aa561aa43a,BootID:161bcd60-e3c0-44ef-b398-9c12c675a9b3,KernelVersion:5.10.123+,OSImage:Container-Optimized OS from Google,ContainerRuntimeVersion:containerd://1.6.6,KubeletVersion:v1.27.0-alpha.1.90+c829397f7a1fa9,KubeProxyVersion:v1.27.0-alpha.1.90+c829397f7a1fa9,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[registry.k8s.io/kube-apiserver-amd64:v1.27.0-alpha.1.90_c829397f7a1fa9],SizeBytes:135973331,},ContainerImage{Names:[registry.k8s.io/kube-controller-manager-amd64:v1.27.0-alpha.1.90_c829397f7a1fa9],SizeBytes:125279545,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:51eae8381dcb1078289fa7b4f3df2630cdc18d09fb56f8e56b41c40e191d6c83 registry.k8s.io/etcd:3.5.7-0],SizeBytes:101639218,},ContainerImage{Names:[registry.k8s.io/cloud-provider-gcp/gcp-compute-persistent-disk-csi-driver@sha256:e9c355dd1ce57de91a0f9164bd2fe688938723e7f9bd40bb13652d603f41d25a registry.k8s.io/cloud-provider-gcp/gcp-compute-persistent-disk-csi-driver:v1.2.2],SizeBytes:73354282,},ContainerImage{Names:[registry.k8s.io/kube-scheduler-amd64:v1.27.0-alpha.1.90_c829397f7a1fa9],SizeBytes:57551672,},ContainerImage{Names:[gke.gcr.io/prometheus-to-sd@sha256:e739643c3939ba0b161425f45a1989eedfc4a3b166db9a7100863296b4c70510 gke.gcr.io/prometheus-to-sd:v0.11.1-gke.1],SizeBytes:48742566,},ContainerImage{Names:[gcr.io/k8s-ingress-image-push/ingress-gce-glbc-amd64@sha256:5db27383add6d9f4ebdf0286409ac31f7f5d273690204b341a4e37998917693b gcr.io/k8s-ingress-image-push/ingress-gce-glbc-amd64:v1.20.1],SizeBytes:36598135,},ContainerImage{Names:[registry.k8s.io/addon-manager/kube-addon-manager@sha256:49cc4e6e4a3745b427ce14b0141476ab339bb65c6bc05033019e046c8727dcb0 registry.k8s.io/addon-manager/kube-addon-manager:v9.1.6],SizeBytes:30464183,},ContainerImage{Names:[registry.k8s.io/kas-network-proxy/proxy-server@sha256:b1389e7014425a1752aac55f5043ef4c52edaef0e223bf4d48ed1324e298087c registry.k8s.io/kas-network-proxy/proxy-server:v0.1.1],SizeBytes:21875112,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/metadata-proxy@sha256:e914645f22e946bce5165737e1b244e0a296ad1f0f81a9531adc57af2780978a registry.k8s.io/metadata-proxy:v0.1.12],SizeBytes:5301657,},ContainerImage{Names:[registry.k8s.io/pause@sha256:7031c1b283388d2c2e09b57badb803c05ebed362dc88d84b480cc47f72a21097 registry.k8s.io/pause:3.9],SizeBytes:321520,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},}
Jan 30 20:33:50.160: INFO: 
Logging kubelet events for node bootstrap-e2e-master
Jan 30 20:33:50.204: INFO: 
Logging pods the kubelet thinks is on node bootstrap-e2e-master
Jan 30 20:33:50.297: INFO: etcd-server-events-bootstrap-e2e-master started at 2023-01-30 17:10:44 +0000 UTC (0+1 container statuses recorded)
Jan 30 20:33:50.297: INFO: 	Container etcd-container ready: true, restart count 0
Jan 30 20:33:50.297: INFO: etcd-server-bootstrap-e2e-master started at 2023-01-30 17:10:44 +0000 UTC (0+1 container statuses recorded)
Jan 30 20:33:50.297: INFO: 	Container etcd-container ready: true, restart count 0
Jan 30 20:33:50.297: INFO: konnectivity-server-bootstrap-e2e-master started at 2023-01-30 17:10:44 +0000 UTC (0+1 container statuses recorded)
Jan 30 20:33:50.297: INFO: 	Container konnectivity-server-container ready: true, restart count 0
Jan 30 20:33:50.297: INFO: kube-scheduler-bootstrap-e2e-master started at 2023-01-30 17:10:44 +0000 UTC (0+1 container statuses recorded)
Jan 30 20:33:50.297: INFO: 	Container kube-scheduler ready: true, restart count 1
Jan 30 20:33:50.297: INFO: l7-lb-controller-bootstrap-e2e-master started at 2023-01-30 17:11:04 +0000 UTC (0+1 container statuses recorded)
Jan 30 20:33:50.297: INFO: 	Container l7-lb-controller ready: true, restart count 2
Jan 30 20:33:50.297: INFO: kube-apiserver-bootstrap-e2e-master started at 2023-01-30 17:10:43 +0000 UTC (0+1 container statuses recorded)
Jan 30 20:33:50.297: INFO: 	Container kube-apiserver ready: true, restart count 0
Jan 30 20:33:50.297: INFO: kube-controller-manager-bootstrap-e2e-master started at 2023-01-30 17:10:44 +0000 UTC (0+1 container statuses recorded)
Jan 30 20:33:50.297: INFO: 	Container kube-controller-manager ready: true, restart count 3
Jan 30 20:33:50.297: INFO: kube-addon-manager-bootstrap-e2e-master started at 2023-01-30 17:11:03 +0000 UTC (0+1 container statuses recorded)
Jan 30 20:33:50.297: INFO: 	Container kube-addon-manager ready: true, restart count 0
Jan 30 20:33:50.297: INFO: metadata-proxy-v0.1-g24sg started at 2023-01-30 17:11:48 +0000 UTC (0+2 container statuses recorded)
Jan 30 20:33:50.297: INFO: 	Container metadata-proxy ready: true, restart count 0
Jan 30 20:33:50.297: INFO: 	Container prometheus-to-sd-exporter ready: true, restart count 0
Jan 30 20:33:50.484: INFO: 
Latency metrics for node bootstrap-e2e-master
Jan 30 20:33:50.484: INFO: 
Logging node info for node bootstrap-e2e-minion-group-5sqf
Jan 30 20:33:50.527: INFO: Node Info: &Node{ObjectMeta:{bootstrap-e2e-minion-group-5sqf    316fbe79-ed9c-499e-aac1-84966e803f96 52839 0 2023-01-30 19:47:23 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:n1-standard-2 beta.kubernetes.io/os:linux cloud.google.com/metadata-proxy-ready:true failure-domain.beta.kubernetes.io/region:us-west1 failure-domain.beta.kubernetes.io/zone:us-west1-b kubernetes.io/arch:amd64 kubernetes.io/hostname:bootstrap-e2e-minion-group-5sqf kubernetes.io/os:linux node.kubernetes.io/instance-type:n1-standard-2 topology.gke.io/zone:us-west1-b topology.hostpath.csi/node:bootstrap-e2e-minion-group-5sqf topology.kubernetes.io/region:us-west1 topology.kubernetes.io/zone:us-west1-b] map[node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kubelet Update v1 2023-01-30 19:47:23 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:cloud.google.com/metadata-proxy-ready":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {kube-controller-manager Update v1 2023-01-30 20:27:20 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}}}} status} {kube-controller-manager Update v1 2023-01-30 20:27:49 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"10.64.4.0/24\"":{}}}} } {node-problem-detector Update v1 2023-01-30 20:32:49 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"CorruptDockerOverlay2\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"FrequentContainerdRestart\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"FrequentDockerRestart\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"FrequentKubeletRestart\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"FrequentUnregisterNetDevice\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"KernelDeadlock\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"ReadonlyFilesystem\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kubelet Update v1 2023-01-30 20:32:55 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:topology.gke.io/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:nodeInfo":{"f:bootID":{}}}} status}]},Spec:NodeSpec{PodCIDR:10.64.4.0/24,DoNotUseExternalID:,ProviderID:gce://k8s-infra-e2e-boskos-080/us-west1-b/bootstrap-e2e-minion-group-5sqf,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[10.64.4.0/24],},Status:NodeStatus{Capacity:ResourceList{attachable-volumes-gce-pd: {{127 0} {<nil>} 127 DecimalSI},cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{101203873792 0} {<nil>} 98831908Ki BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{7815426048 0} {<nil>} 7632252Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{attachable-volumes-gce-pd: {{127 0} {<nil>} 127 DecimalSI},cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{91083486262 0} {<nil>} 91083486262 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{7553282048 0} {<nil>} 7376252Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:KernelDeadlock,Status:False,LastHeartbeatTime:2023-01-30 20:32:49 +0000 UTC,LastTransitionTime:2023-01-30 20:27:47 +0000 UTC,Reason:KernelHasNoDeadlock,Message:kernel has no deadlock,},NodeCondition{Type:ReadonlyFilesystem,Status:False,LastHeartbeatTime:2023-01-30 20:32:49 +0000 UTC,LastTransitionTime:2023-01-30 20:27:47 +0000 UTC,Reason:FilesystemIsNotReadOnly,Message:Filesystem is not read-only,},NodeCondition{Type:CorruptDockerOverlay2,Status:False,LastHeartbeatTime:2023-01-30 20:32:49 +0000 UTC,LastTransitionTime:2023-01-30 20:27:47 +0000 UTC,Reason:NoCorruptDockerOverlay2,Message:docker overlay2 is functioning properly,},NodeCondition{Type:FrequentUnregisterNetDevice,Status:False,LastHeartbeatTime:2023-01-30 20:32:49 +0000 UTC,LastTransitionTime:2023-01-30 20:27:47 +0000 UTC,Reason:NoFrequentUnregisterNetDevice,Message:node is functioning properly,},NodeCondition{Type:FrequentKubeletRestart,Status:False,LastHeartbeatTime:2023-01-30 20:32:49 +0000 UTC,LastTransitionTime:2023-01-30 20:27:47 +0000 UTC,Reason:NoFrequentKubeletRestart,Message:kubelet is functioning properly,},NodeCondition{Type:FrequentDockerRestart,Status:False,LastHeartbeatTime:2023-01-30 20:32:49 +0000 UTC,LastTransitionTime:2023-01-30 20:27:47 +0000 UTC,Reason:NoFrequentDockerRestart,Message:docker is functioning properly,},NodeCondition{Type:FrequentContainerdRestart,Status:False,LastHeartbeatTime:2023-01-30 20:32:49 +0000 UTC,LastTransitionTime:2023-01-30 20:27:47 +0000 UTC,Reason:NoFrequentContainerdRestart,Message:containerd is functioning properly,},NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2023-01-30 19:47:33 +0000 UTC,LastTransitionTime:2023-01-30 19:47:33 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-01-30 20:32:55 +0000 UTC,LastTransitionTime:2023-01-30 20:27:49 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-01-30 20:32:55 +0000 UTC,LastTransitionTime:2023-01-30 20:27:49 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-01-30 20:32:55 +0000 UTC,LastTransitionTime:2023-01-30 20:27:49 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-01-30 20:32:55 +0000 UTC,LastTransitionTime:2023-01-30 20:27:49 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:10.138.0.9,},NodeAddress{Type:ExternalIP,Address:35.227.149.86,},NodeAddress{Type:InternalDNS,Address:bootstrap-e2e-minion-group-5sqf.c.k8s-infra-e2e-boskos-080.internal,},NodeAddress{Type:Hostname,Address:bootstrap-e2e-minion-group-5sqf.c.k8s-infra-e2e-boskos-080.internal,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:939aca5c4e6f48f6e7efe71a9260f4cf,SystemUUID:939aca5c-4e6f-48f6-e7ef-e71a9260f4cf,BootID:bcc84782-293d-401e-b1a7-1198eeaeeac8,KernelVersion:5.10.123+,OSImage:Container-Optimized OS from Google,ContainerRuntimeVersion:containerd://1.6.6,KubeletVersion:v1.27.0-alpha.1.90+c829397f7a1fa9,KubeProxyVersion:v1.27.0-alpha.1.90+c829397f7a1fa9,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[registry.k8s.io/cloud-provider-gcp/gcp-compute-persistent-disk-csi-driver@sha256:e9c355dd1ce57de91a0f9164bd2fe688938723e7f9bd40bb13652d603f41d25a registry.k8s.io/cloud-provider-gcp/gcp-compute-persistent-disk-csi-driver:v1.2.2],SizeBytes:73354282,},ContainerImage{Names:[registry.k8s.io/kube-proxy-amd64:v1.27.0-alpha.1.90_c829397f7a1fa9],SizeBytes:66988744,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:16bbf38c463a4223d8cfe4da12bc61010b082a79b4bb003e2d3ba3ece5dd5f9e registry.k8s.io/e2e-test-images/agnhost:2.43],SizeBytes:51706353,},ContainerImage{Names:[gke.gcr.io/prometheus-to-sd@sha256:e739643c3939ba0b161425f45a1989eedfc4a3b166db9a7100863296b4c70510 gke.gcr.io/prometheus-to-sd:v0.11.1-gke.1],SizeBytes:48742566,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:148b022f5c5da426fc2f3c14b5c0867e58ef05961510c84749ac1fddcb0fef22 registry.k8s.io/e2e-test-images/httpd:2.4.38-4],SizeBytes:40764257,},ContainerImage{Names:[registry.k8s.io/cloud-provider-gcp/gcp-compute-persistent-disk-csi-driver@sha256:dc0301d9a43e23e389abb7690cd631f98af76ea3429259c7f5f643fd31fc0a3e registry.k8s.io/cloud-provider-gcp/gcp-compute-persistent-disk-csi-driver:v1.4.0],SizeBytes:27914152,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:ee3b525d5b89db99da3b8eb521d9cd90cb6e9ef0fbb651e98bb37be78d36b5b8 registry.k8s.io/sig-storage/csi-provisioner:v3.3.0],SizeBytes:25491225,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:425d8f1b769398127767b06ed97ce62578a3179bcb99809ce93a1649e025ffe7 registry.k8s.io/sig-storage/csi-resizer:v1.6.0],SizeBytes:24148884,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:291334908ddf71a4661fd7f6d9d97274de8a5378a2b6fdfeb2ce73414a34f82f registry.k8s.io/sig-storage/csi-snapshotter:v6.1.0],SizeBytes:23881995,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:9a685020911e2725ad019dbce6e4a5ab93d51e3d4557f115e64343345e05781b registry.k8s.io/sig-storage/csi-attacher:v4.0.0],SizeBytes:23847201,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:89e900a160a986a1a7a4eba7f5259e510398fa87ca9b8a729e7dec59e04c7709 registry.k8s.io/sig-storage/csi-snapshotter:v5.0.1],SizeBytes:22163966,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:92257881c1d6493cf18299a24af42330f891166560047902b8d431fb66b01af5 registry.k8s.io/sig-storage/hostpathplugin:v1.9.0],SizeBytes:18758628,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:017727efcfeb7d053af68e51436ce8e65edbc6ca573720afb4f79c8594036955 registry.k8s.io/coredns/coredns:v1.10.0],SizeBytes:15273057,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:933940f13b3ea0abc62e656c1aa5c5b47c04b15d71250413a6b821bd0c58b94e registry.k8s.io/sig-storage/livenessprobe:v2.7.0],SizeBytes:8688564,},ContainerImage{Names:[registry.k8s.io/kas-network-proxy/proxy-agent@sha256:939c42e815e6b6af3181f074652c0d18fe429fcee9b49c1392aee7e92887cfef registry.k8s.io/kas-network-proxy/proxy-agent:v0.1.1],SizeBytes:8364694,},ContainerImage{Names:[registry.k8s.io/metadata-proxy@sha256:e914645f22e946bce5165737e1b244e0a296ad1f0f81a9531adc57af2780978a registry.k8s.io/metadata-proxy:v0.1.12],SizeBytes:5301657,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:2e0f836850e09b8b7cc937681d6194537a09fbd5f6b9e08f4d646a85128e8937 registry.k8s.io/e2e-test-images/busybox:1.29-4],SizeBytes:731990,},ContainerImage{Names:[registry.k8s.io/pause@sha256:7031c1b283388d2c2e09b57badb803c05ebed362dc88d84b480cc47f72a21097 registry.k8s.io/pause:3.9],SizeBytes:321520,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},}
Jan 30 20:33:50.527: INFO: 
Logging kubelet events for node bootstrap-e2e-minion-group-5sqf
Jan 30 20:33:50.571: INFO: 
Logging pods the kubelet thinks is on node bootstrap-e2e-minion-group-5sqf
Jan 30 20:33:50.639: INFO: metadata-proxy-v0.1-j8mg4 started at 2023-01-30 19:47:24 +0000 UTC (0+2 container statuses recorded)
Jan 30 20:33:50.639: INFO: 	Container metadata-proxy ready: true, restart count 1
Jan 30 20:33:50.639: INFO: 	Container prometheus-to-sd-exporter ready: true, restart count 1
Jan 30 20:33:50.639: INFO: konnectivity-agent-m7rcz started at 2023-01-30 19:47:33 +0000 UTC (0+1 container statuses recorded)
Jan 30 20:33:50.639: INFO: 	Container konnectivity-agent ready: true, restart count 1
Jan 30 20:33:50.639: INFO: kube-proxy-bootstrap-e2e-minion-group-5sqf started at 2023-01-30 19:47:23 +0000 UTC (0+1 container statuses recorded)
Jan 30 20:33:50.639: INFO: 	Container kube-proxy ready: true, restart count 1
Jan 30 20:33:50.836: INFO: 
Latency metrics for node bootstrap-e2e-minion-group-5sqf
Jan 30 20:33:50.836: INFO: 
Logging node info for node bootstrap-e2e-minion-group-gr7w
Jan 30 20:33:50.879: INFO: Node Info: &Node{ObjectMeta:{bootstrap-e2e-minion-group-gr7w    0a4b32b4-348b-4c5b-93fb-de4d0701e2eb 52860 0 2023-01-30 17:11:35 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:n1-standard-2 beta.kubernetes.io/os:linux cloud.google.com/metadata-proxy-ready:true failure-domain.beta.kubernetes.io/region:us-west1 failure-domain.beta.kubernetes.io/zone:us-west1-b kubernetes.io/arch:amd64 kubernetes.io/hostname:bootstrap-e2e-minion-group-gr7w kubernetes.io/os:linux node.kubernetes.io/instance-type:n1-standard-2 topology.gke.io/zone:us-west1-b topology.hostpath.csi/node:bootstrap-e2e-minion-group-gr7w topology.kubernetes.io/region:us-west1 topology.kubernetes.io/zone:us-west1-b] map[node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kubelet Update v1 2023-01-30 17:11:35 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:cloud.google.com/metadata-proxy-ready":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {e2e.test Update v1 2023-01-30 20:15:46 +0000 UTC FieldsV1 {"f:status":{"f:allocatable":{"f:example.com/fakePTSRes":{}},"f:capacity":{"f:example.com/fakePTSRes":{}}}} status} {kube-controller-manager Update v1 2023-01-30 20:27:25 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}}}} status} {kube-controller-manager Update v1 2023-01-30 20:27:56 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"10.64.1.0/24\"":{}}}} } {node-problem-detector Update v1 2023-01-30 20:32:57 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"CorruptDockerOverlay2\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"FrequentContainerdRestart\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"FrequentDockerRestart\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"FrequentKubeletRestart\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"FrequentUnregisterNetDevice\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"KernelDeadlock\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"ReadonlyFilesystem\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kubelet Update v1 2023-01-30 20:33:03 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:topology.gke.io/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{},"f:memory":{}},"f:capacity":{"f:memory":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:nodeInfo":{"f:bootID":{}}}} status}]},Spec:NodeSpec{PodCIDR:10.64.1.0/24,DoNotUseExternalID:,ProviderID:gce://k8s-infra-e2e-boskos-080/us-west1-b/bootstrap-e2e-minion-group-gr7w,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[10.64.1.0/24],},Status:NodeStatus{Capacity:ResourceList{attachable-volumes-gce-pd: {{127 0} {<nil>} 127 DecimalSI},cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{101203873792 0} {<nil>} 98831908Ki BinarySI},example.com/fakePTSRes: {{10 0} {<nil>} 10 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{7815426048 0} {<nil>} 7632252Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{attachable-volumes-gce-pd: {{127 0} {<nil>} 127 DecimalSI},cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{91083486262 0} {<nil>} 91083486262 DecimalSI},example.com/fakePTSRes: {{10 0} {<nil>} 10 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{7553282048 0} {<nil>} 7376252Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:FrequentContainerdRestart,Status:False,LastHeartbeatTime:2023-01-30 20:32:57 +0000 UTC,LastTransitionTime:2023-01-30 20:27:55 +0000 UTC,Reason:NoFrequentContainerdRestart,Message:containerd is functioning properly,},NodeCondition{Type:KernelDeadlock,Status:False,LastHeartbeatTime:2023-01-30 20:32:57 +0000 UTC,LastTransitionTime:2023-01-30 20:27:55 +0000 UTC,Reason:KernelHasNoDeadlock,Message:kernel has no deadlock,},NodeCondition{Type:ReadonlyFilesystem,Status:False,LastHeartbeatTime:2023-01-30 20:32:57 +0000 UTC,LastTransitionTime:2023-01-30 20:27:55 +0000 UTC,Reason:FilesystemIsNotReadOnly,Message:Filesystem is not read-only,},NodeCondition{Type:CorruptDockerOverlay2,Status:False,LastHeartbeatTime:2023-01-30 20:32:57 +0000 UTC,LastTransitionTime:2023-01-30 20:27:55 +0000 UTC,Reason:NoCorruptDockerOverlay2,Message:docker overlay2 is functioning properly,},NodeCondition{Type:FrequentUnregisterNetDevice,Status:False,LastHeartbeatTime:2023-01-30 20:32:57 +0000 UTC,LastTransitionTime:2023-01-30 20:27:55 +0000 UTC,Reason:NoFrequentUnregisterNetDevice,Message:node is functioning properly,},NodeCondition{Type:FrequentKubeletRestart,Status:False,LastHeartbeatTime:2023-01-30 20:32:57 +0000 UTC,LastTransitionTime:2023-01-30 20:27:55 +0000 UTC,Reason:NoFrequentKubeletRestart,Message:kubelet is functioning properly,},NodeCondition{Type:FrequentDockerRestart,Status:False,LastHeartbeatTime:2023-01-30 20:32:57 +0000 UTC,LastTransitionTime:2023-01-30 20:27:55 +0000 UTC,Reason:NoFrequentDockerRestart,Message:docker is functioning properly,},NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2023-01-30 17:11:45 +0000 UTC,LastTransitionTime:2023-01-30 17:11:45 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-01-30 20:33:03 +0000 UTC,LastTransitionTime:2023-01-30 20:27:56 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-01-30 20:33:03 +0000 UTC,LastTransitionTime:2023-01-30 20:27:56 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-01-30 20:33:03 +0000 UTC,LastTransitionTime:2023-01-30 20:27:56 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-01-30 20:33:03 +0000 UTC,LastTransitionTime:2023-01-30 20:27:56 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:10.138.0.4,},NodeAddress{Type:ExternalIP,Address:35.197.7.140,},NodeAddress{Type:InternalDNS,Address:bootstrap-e2e-minion-group-gr7w.c.k8s-infra-e2e-boskos-080.internal,},NodeAddress{Type:Hostname,Address:bootstrap-e2e-minion-group-gr7w.c.k8s-infra-e2e-boskos-080.internal,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:b7fa42720df0ccfbfadca51fa5993af0,SystemUUID:b7fa4272-0df0-ccfb-fadc-a51fa5993af0,BootID:24c065d9-7b4d-4dfd-a3ab-600e2ef1a9f9,KernelVersion:5.10.123+,OSImage:Container-Optimized OS from Google,ContainerRuntimeVersion:containerd://1.6.6,KubeletVersion:v1.27.0-alpha.1.90+c829397f7a1fa9,KubeProxyVersion:v1.27.0-alpha.1.90+c829397f7a1fa9,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[registry.k8s.io/e2e-test-images/jessie-dnsutils@sha256:24aaf2626d6b27864c29de2097e8bbb840b3a414271bf7c8995e431e47d8408e registry.k8s.io/e2e-test-images/jessie-dnsutils:1.7],SizeBytes:112030336,},ContainerImage{Names:[registry.k8s.io/cloud-provider-gcp/gcp-compute-persistent-disk-csi-driver@sha256:e9c355dd1ce57de91a0f9164bd2fe688938723e7f9bd40bb13652d603f41d25a registry.k8s.io/cloud-provider-gcp/gcp-compute-persistent-disk-csi-driver:v1.2.2],SizeBytes:73354282,},ContainerImage{Names:[registry.k8s.io/kube-proxy-amd64:v1.27.0-alpha.1.90_c829397f7a1fa9],SizeBytes:66988744,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:16bbf38c463a4223d8cfe4da12bc61010b082a79b4bb003e2d3ba3ece5dd5f9e registry.k8s.io/e2e-test-images/agnhost:2.43],SizeBytes:51706353,},ContainerImage{Names:[gke.gcr.io/prometheus-to-sd@sha256:e739643c3939ba0b161425f45a1989eedfc4a3b166db9a7100863296b4c70510 gke.gcr.io/prometheus-to-sd:v0.11.1-gke.1],SizeBytes:48742566,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:148b022f5c5da426fc2f3c14b5c0867e58ef05961510c84749ac1fddcb0fef22 registry.k8s.io/e2e-test-images/httpd:2.4.38-4],SizeBytes:40764257,},ContainerImage{Names:[registry.k8s.io/sig-storage/volume-data-source-validator@sha256:d35884236461ad14397194cb4abc13aefb28a54bf3f6698d673c2ba353e78e18 registry.k8s.io/sig-storage/volume-data-source-validator:v1.0.0],SizeBytes:28662933,},ContainerImage{Names:[registry.k8s.io/cloud-provider-gcp/gcp-compute-persistent-disk-csi-driver@sha256:dc0301d9a43e23e389abb7690cd631f98af76ea3429259c7f5f643fd31fc0a3e registry.k8s.io/cloud-provider-gcp/gcp-compute-persistent-disk-csi-driver:v1.4.0],SizeBytes:27914152,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:ee3b525d5b89db99da3b8eb521d9cd90cb6e9ef0fbb651e98bb37be78d36b5b8 registry.k8s.io/sig-storage/csi-provisioner:v3.3.0],SizeBytes:25491225,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:425d8f1b769398127767b06ed97ce62578a3179bcb99809ce93a1649e025ffe7 registry.k8s.io/sig-storage/csi-resizer:v1.6.0],SizeBytes:24148884,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:291334908ddf71a4661fd7f6d9d97274de8a5378a2b6fdfeb2ce73414a34f82f registry.k8s.io/sig-storage/csi-snapshotter:v6.1.0],SizeBytes:23881995,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:9a685020911e2725ad019dbce6e4a5ab93d51e3d4557f115e64343345e05781b registry.k8s.io/sig-storage/csi-attacher:v4.0.0],SizeBytes:23847201,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:89e900a160a986a1a7a4eba7f5259e510398fa87ca9b8a729e7dec59e04c7709 registry.k8s.io/sig-storage/csi-snapshotter:v5.0.1],SizeBytes:22163966,},ContainerImage{Names:[registry.k8s.io/sig-storage/hello-populator@sha256:42d275bc2569b00a11c8058e5b77c50cdee39bf75d80c912b80813f03e7fd7d1 registry.k8s.io/sig-storage/hello-populator:v1.0.1],SizeBytes:20511089,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:92257881c1d6493cf18299a24af42330f891166560047902b8d431fb66b01af5 registry.k8s.io/sig-storage/hostpathplugin:v1.9.0],SizeBytes:18758628,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:017727efcfeb7d053af68e51436ce8e65edbc6ca573720afb4f79c8594036955 registry.k8s.io/coredns/coredns:v1.10.0],SizeBytes:15273057,},ContainerImage{Names:[registry.k8s.io/cpa/cluster-proportional-autoscaler@sha256:fd636b33485c7826fb20ef0688a83ee0910317dbb6c0c6f3ad14661c1db25def registry.k8s.io/cpa/cluster-proportional-autoscaler:1.8.4],SizeBytes:15209393,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:933940f13b3ea0abc62e656c1aa5c5b47c04b15d71250413a6b821bd0c58b94e registry.k8s.io/sig-storage/livenessprobe:v2.7.0],SizeBytes:8688564,},ContainerImage{Names:[registry.k8s.io/kas-network-proxy/proxy-agent@sha256:939c42e815e6b6af3181f074652c0d18fe429fcee9b49c1392aee7e92887cfef registry.k8s.io/kas-network-proxy/proxy-agent:v0.1.1],SizeBytes:8364694,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:5c99cf6a02adda929b10321dbf4ecfa00d87be9ba4fb456006237d530ab4baa1 registry.k8s.io/e2e-test-images/nginx:1.14-4],SizeBytes:6978614,},ContainerImage{Names:[registry.k8s.io/networking/ingress-gce-404-server-with-metrics-amd64@sha256:7eb7b3cee4d33c10c49893ad3c386232b86d4067de5251294d4c620d6e072b93 registry.k8s.io/networking/ingress-gce-404-server-with-metrics-amd64:v1.10.11],SizeBytes:6463068,},ContainerImage{Names:[registry.k8s.io/metadata-proxy@sha256:e914645f22e946bce5165737e1b244e0a296ad1f0f81a9531adc57af2780978a registry.k8s.io/metadata-proxy:v0.1.12],SizeBytes:5301657,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:2e0f836850e09b8b7cc937681d6194537a09fbd5f6b9e08f4d646a85128e8937 registry.k8s.io/e2e-test-images/busybox:1.29-4],SizeBytes:731990,},ContainerImage{Names:[registry.k8s.io/pause@sha256:7031c1b283388d2c2e09b57badb803c05ebed362dc88d84b480cc47f72a21097 registry.k8s.io/pause:3.9],SizeBytes:321520,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},}
Jan 30 20:33:50.879: INFO: 
Logging kubelet events for node bootstrap-e2e-minion-group-gr7w
Jan 30 20:33:50.923: INFO: 
Logging pods the kubelet thinks is on node bootstrap-e2e-minion-group-gr7w
Jan 30 20:33:51.014: INFO: metadata-proxy-v0.1-g6h7m started at 2023-01-30 17:11:36 +0000 UTC (0+2 container statuses recorded)
Jan 30 20:33:51.014: INFO: 	Container metadata-proxy ready: true, restart count 1
Jan 30 20:33:51.014: INFO: 	Container prometheus-to-sd-exporter ready: true, restart count 1
Jan 30 20:33:51.014: INFO: konnectivity-agent-pb855 started at 2023-01-30 17:11:46 +0000 UTC (0+1 container statuses recorded)
Jan 30 20:33:51.014: INFO: 	Container konnectivity-agent ready: true, restart count 1
Jan 30 20:33:51.014: INFO: kube-dns-autoscaler-5f6455f985-7jdxx started at 2023-01-30 18:36:57 +0000 UTC (0+1 container statuses recorded)
Jan 30 20:33:51.014: INFO: 	Container autoscaler ready: true, restart count 1
Jan 30 20:33:51.014: INFO: l7-default-backend-8549d69d99-xzb2m started at 2023-01-30 18:36:57 +0000 UTC (0+1 container statuses recorded)
Jan 30 20:33:51.014: INFO: 	Container default-http-backend ready: true, restart count 1
Jan 30 20:33:51.014: INFO: coredns-6846b5b5f-bkw98 started at 2023-01-30 20:20:45 +0000 UTC (0+1 container statuses recorded)
Jan 30 20:33:51.014: INFO: 	Container coredns ready: true, restart count 1
Jan 30 20:33:51.014: INFO: kube-proxy-bootstrap-e2e-minion-group-gr7w started at 2023-01-30 17:11:35 +0000 UTC (0+1 container statuses recorded)
Jan 30 20:33:51.014: INFO: 	Container kube-proxy ready: true, restart count 1
Jan 30 20:33:51.197: INFO: 
Latency metrics for node bootstrap-e2e-minion-group-gr7w
Jan 30 20:33:51.197: INFO: 
Logging node info for node bootstrap-e2e-minion-group-hwhs
Jan 30 20:33:51.241: INFO: Node Info: &Node{ObjectMeta:{bootstrap-e2e-minion-group-hwhs    7bbe7962-b092-4381-b012-d5ff6aa9582a 52851 0 2023-01-30 17:11:38 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:n1-standard-2 beta.kubernetes.io/os:linux cloud.google.com/metadata-proxy-ready:true failure-domain.beta.kubernetes.io/region:us-west1 failure-domain.beta.kubernetes.io/zone:us-west1-b kubernetes.io/arch:amd64 kubernetes.io/hostname:bootstrap-e2e-minion-group-hwhs kubernetes.io/os:linux node.kubernetes.io/instance-type:n1-standard-2 topology.gke.io/zone:us-west1-b topology.hostpath.csi/node:bootstrap-e2e-minion-group-hwhs topology.kubernetes.io/region:us-west1 topology.kubernetes.io/zone:us-west1-b] map[node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kubelet Update v1 2023-01-30 17:11:38 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:cloud.google.com/metadata-proxy-ready":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {kube-controller-manager Update v1 2023-01-30 20:27:15 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}}}} status} {kube-controller-manager Update v1 2023-01-30 20:27:52 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"10.64.3.0/24\"":{}}}} } {node-problem-detector Update v1 2023-01-30 20:32:52 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"CorruptDockerOverlay2\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"FrequentContainerdRestart\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"FrequentDockerRestart\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"FrequentKubeletRestart\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"FrequentUnregisterNetDevice\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"KernelDeadlock\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"ReadonlyFilesystem\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kubelet Update v1 2023-01-30 20:32:59 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:topology.gke.io/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{},"f:memory":{}},"f:capacity":{"f:memory":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:nodeInfo":{"f:bootID":{}}}} status}]},Spec:NodeSpec{PodCIDR:10.64.3.0/24,DoNotUseExternalID:,ProviderID:gce://k8s-infra-e2e-boskos-080/us-west1-b/bootstrap-e2e-minion-group-hwhs,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[10.64.3.0/24],},Status:NodeStatus{Capacity:ResourceList{attachable-volumes-gce-pd: {{127 0} {<nil>} 127 DecimalSI},cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{101203873792 0} {<nil>} 98831908Ki BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{7815426048 0} {<nil>} 7632252Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{attachable-volumes-gce-pd: {{127 0} {<nil>} 127 DecimalSI},cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{91083486262 0} {<nil>} 91083486262 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{7553282048 0} {<nil>} 7376252Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:KernelDeadlock,Status:False,LastHeartbeatTime:2023-01-30 20:32:52 +0000 UTC,LastTransitionTime:2023-01-30 20:27:51 +0000 UTC,Reason:KernelHasNoDeadlock,Message:kernel has no deadlock,},NodeCondition{Type:ReadonlyFilesystem,Status:False,LastHeartbeatTime:2023-01-30 20:32:52 +0000 UTC,LastTransitionTime:2023-01-30 20:27:51 +0000 UTC,Reason:FilesystemIsNotReadOnly,Message:Filesystem is not read-only,},NodeCondition{Type:CorruptDockerOverlay2,Status:False,LastHeartbeatTime:2023-01-30 20:32:52 +0000 UTC,LastTransitionTime:2023-01-30 20:27:51 +0000 UTC,Reason:NoCorruptDockerOverlay2,Message:docker overlay2 is functioning properly,},NodeCondition{Type:FrequentUnregisterNetDevice,Status:False,LastHeartbeatTime:2023-01-30 20:32:52 +0000 UTC,LastTransitionTime:2023-01-30 20:27:51 +0000 UTC,Reason:NoFrequentUnregisterNetDevice,Message:node is functioning properly,},NodeCondition{Type:FrequentKubeletRestart,Status:False,LastHeartbeatTime:2023-01-30 20:32:52 +0000 UTC,LastTransitionTime:2023-01-30 20:27:51 +0000 UTC,Reason:NoFrequentKubeletRestart,Message:kubelet is functioning properly,},NodeCondition{Type:FrequentDockerRestart,Status:False,LastHeartbeatTime:2023-01-30 20:32:52 +0000 UTC,LastTransitionTime:2023-01-30 20:27:51 +0000 UTC,Reason:NoFrequentDockerRestart,Message:docker is functioning properly,},NodeCondition{Type:FrequentContainerdRestart,Status:False,LastHeartbeatTime:2023-01-30 20:32:52 +0000 UTC,LastTransitionTime:2023-01-30 20:27:51 +0000 UTC,Reason:NoFrequentContainerdRestart,Message:containerd is functioning properly,},NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2023-01-30 17:11:54 +0000 UTC,LastTransitionTime:2023-01-30 17:11:54 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-01-30 20:32:59 +0000 UTC,LastTransitionTime:2023-01-30 20:27:52 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-01-30 20:32:59 +0000 UTC,LastTransitionTime:2023-01-30 20:27:52 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-01-30 20:32:59 +0000 UTC,LastTransitionTime:2023-01-30 20:27:52 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-01-30 20:32:59 +0000 UTC,LastTransitionTime:2023-01-30 20:27:52 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:10.138.0.5,},NodeAddress{Type:ExternalIP,Address:34.82.53.160,},NodeAddress{Type:InternalDNS,Address:bootstrap-e2e-minion-group-hwhs.c.k8s-infra-e2e-boskos-080.internal,},NodeAddress{Type:Hostname,Address:bootstrap-e2e-minion-group-hwhs.c.k8s-infra-e2e-boskos-080.internal,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:0306c1b4b16dc97a101d208104b2a5f0,SystemUUID:0306c1b4-b16d-c97a-101d-208104b2a5f0,BootID:4cb2635d-07e6-4b8f-a686-d021bbfef301,KernelVersion:5.10.123+,OSImage:Container-Optimized OS from Google,ContainerRuntimeVersion:containerd://1.6.6,KubeletVersion:v1.27.0-alpha.1.90+c829397f7a1fa9,KubeProxyVersion:v1.27.0-alpha.1.90+c829397f7a1fa9,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[registry.k8s.io/cloud-provider-gcp/gcp-compute-persistent-disk-csi-driver@sha256:e9c355dd1ce57de91a0f9164bd2fe688938723e7f9bd40bb13652d603f41d25a registry.k8s.io/cloud-provider-gcp/gcp-compute-persistent-disk-csi-driver:v1.2.2],SizeBytes:73354282,},ContainerImage{Names:[registry.k8s.io/kube-proxy-amd64:v1.27.0-alpha.1.90_c829397f7a1fa9],SizeBytes:66988744,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:16bbf38c463a4223d8cfe4da12bc61010b082a79b4bb003e2d3ba3ece5dd5f9e registry.k8s.io/e2e-test-images/agnhost:2.43],SizeBytes:51706353,},ContainerImage{Names:[gke.gcr.io/prometheus-to-sd@sha256:e739643c3939ba0b161425f45a1989eedfc4a3b166db9a7100863296b4c70510 gke.gcr.io/prometheus-to-sd:v0.11.1-gke.1],SizeBytes:48742566,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:148b022f5c5da426fc2f3c14b5c0867e58ef05961510c84749ac1fddcb0fef22 registry.k8s.io/e2e-test-images/httpd:2.4.38-4],SizeBytes:40764257,},ContainerImage{Names:[registry.k8s.io/cloud-provider-gcp/gcp-compute-persistent-disk-csi-driver@sha256:dc0301d9a43e23e389abb7690cd631f98af76ea3429259c7f5f643fd31fc0a3e registry.k8s.io/cloud-provider-gcp/gcp-compute-persistent-disk-csi-driver:v1.4.0],SizeBytes:27914152,},ContainerImage{Names:[registry.k8s.io/metrics-server/metrics-server@sha256:6385aec64bb97040a5e692947107b81e178555c7a5b71caa90d733e4130efc10 registry.k8s.io/metrics-server/metrics-server:v0.5.2],SizeBytes:26023008,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:ee3b525d5b89db99da3b8eb521d9cd90cb6e9ef0fbb651e98bb37be78d36b5b8 registry.k8s.io/sig-storage/csi-provisioner:v3.3.0],SizeBytes:25491225,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:425d8f1b769398127767b06ed97ce62578a3179bcb99809ce93a1649e025ffe7 registry.k8s.io/sig-storage/csi-resizer:v1.6.0],SizeBytes:24148884,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:291334908ddf71a4661fd7f6d9d97274de8a5378a2b6fdfeb2ce73414a34f82f registry.k8s.io/sig-storage/csi-snapshotter:v6.1.0],SizeBytes:23881995,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:9a685020911e2725ad019dbce6e4a5ab93d51e3d4557f115e64343345e05781b registry.k8s.io/sig-storage/csi-attacher:v4.0.0],SizeBytes:23847201,},ContainerImage{Names:[registry.k8s.io/sig-storage/snapshot-controller@sha256:823c75d0c45d1427f6d850070956d9ca657140a7bbf828381541d1d808475280 registry.k8s.io/sig-storage/snapshot-controller:v6.1.0],SizeBytes:22620891,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:89e900a160a986a1a7a4eba7f5259e510398fa87ca9b8a729e7dec59e04c7709 registry.k8s.io/sig-storage/csi-snapshotter:v5.0.1],SizeBytes:22163966,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:92257881c1d6493cf18299a24af42330f891166560047902b8d431fb66b01af5 registry.k8s.io/sig-storage/hostpathplugin:v1.9.0],SizeBytes:18758628,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:017727efcfeb7d053af68e51436ce8e65edbc6ca573720afb4f79c8594036955 registry.k8s.io/coredns/coredns:v1.10.0],SizeBytes:15273057,},ContainerImage{Names:[registry.k8s.io/autoscaling/addon-resizer@sha256:43f129b81d28f0fdd54de6d8e7eacd5728030782e03db16087fc241ad747d3d6 registry.k8s.io/autoscaling/addon-resizer:1.8.14],SizeBytes:10153852,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:933940f13b3ea0abc62e656c1aa5c5b47c04b15d71250413a6b821bd0c58b94e registry.k8s.io/sig-storage/livenessprobe:v2.7.0],SizeBytes:8688564,},ContainerImage{Names:[registry.k8s.io/kas-network-proxy/proxy-agent@sha256:939c42e815e6b6af3181f074652c0d18fe429fcee9b49c1392aee7e92887cfef registry.k8s.io/kas-network-proxy/proxy-agent:v0.1.1],SizeBytes:8364694,},ContainerImage{Names:[registry.k8s.io/metadata-proxy@sha256:e914645f22e946bce5165737e1b244e0a296ad1f0f81a9531adc57af2780978a registry.k8s.io/metadata-proxy:v0.1.12],SizeBytes:5301657,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:2e0f836850e09b8b7cc937681d6194537a09fbd5f6b9e08f4d646a85128e8937 registry.k8s.io/e2e-test-images/busybox:1.29-4],SizeBytes:731990,},ContainerImage{Names:[registry.k8s.io/pause@sha256:7031c1b283388d2c2e09b57badb803c05ebed362dc88d84b480cc47f72a21097 registry.k8s.io/pause:3.9],SizeBytes:321520,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},}
Jan 30 20:33:51.241: INFO: 
Logging kubelet events for node bootstrap-e2e-minion-group-hwhs
Jan 30 20:33:51.289: INFO: 
Logging pods the kubelet thinks is on node bootstrap-e2e-minion-group-hwhs
Jan 30 20:33:51.433: INFO: kube-proxy-bootstrap-e2e-minion-group-hwhs started at 2023-01-30 17:11:38 +0000 UTC (0+1 container statuses recorded)
Jan 30 20:33:51.433: INFO: 	Container kube-proxy ready: true, restart count 1
Jan 30 20:33:51.433: INFO: metadata-proxy-v0.1-tp2qz started at 2023-01-30 17:11:39 +0000 UTC (0+2 container statuses recorded)
Jan 30 20:33:51.433: INFO: 	Container metadata-proxy ready: true, restart count 1
Jan 30 20:33:51.433: INFO: 	Container prometheus-to-sd-exporter ready: true, restart count 1
Jan 30 20:33:51.433: INFO: konnectivity-agent-9r6m6 started at 2023-01-30 17:11:54 +0000 UTC (0+1 container statuses recorded)
Jan 30 20:33:51.433: INFO: 	Container konnectivity-agent ready: true, restart count 1
Jan 30 20:33:51.433: INFO: metrics-server-v0.5.2-867b8754b9-dmjcn started at 2023-01-30 17:12:01 +0000 UTC (0+2 container statuses recorded)
Jan 30 20:33:51.433: INFO: 	Container metrics-server ready: true, restart count 1
Jan 30 20:33:51.433: INFO: 	Container metrics-server-nanny ready: true, restart count 1
Jan 30 20:33:51.433: INFO: volume-snapshot-controller-0 started at 2023-01-30 17:52:56 +0000 UTC (0+1 container statuses recorded)
Jan 30 20:33:51.433: INFO: 	Container volume-snapshot-controller ready: true, restart count 1
Jan 30 20:33:51.433: INFO: coredns-6846b5b5f-zwbrf started at 2023-01-30 20:26:34 +0000 UTC (0+1 container statuses recorded)
Jan 30 20:33:51.433: INFO: 	Container coredns ready: true, restart count 1
Jan 30 20:33:51.609: INFO: 
Latency metrics for node bootstrap-e2e-minion-group-hwhs
END STEP: dump namespace information after failure - test/e2e/framework/framework.go:284 @ 01/30/23 20:33:51.609 (1.619s)
< Exit [DeferCleanup (Each)] [sig-cloud-provider-gcp] Restart [Disruptive] - dump namespaces | framework.go:206 @ 01/30/23 20:33:51.609 (1.619s)
> Enter [DeferCleanup (Each)] [sig-cloud-provider-gcp] Restart [Disruptive] - tear down framework | framework.go:203 @ 01/30/23 20:33:51.609
STEP: Destroying namespace "restart-7764" for this suite. - test/e2e/framework/framework.go:347 @ 01/30/23 20:33:51.609
< Exit [DeferCleanup (Each)] [sig-cloud-provider-gcp] Restart [Disruptive] - tear down framework | framework.go:203 @ 01/30/23 20:33:51.658 (49ms)
> Enter [ReportAfterEach] TOP-LEVEL - test/e2e/e2e_test.go:144 @ 01/30/23 20:33:51.658
< Exit [ReportAfterEach] TOP-LEVEL - test/e2e/e2e_test.go:144 @ 01/30/23 20:33:51.658 (0s)

				Click to see stdout/stderrfrom junit_01.xml

Filter through log files | View test history on testgrid

kubetest Test 5h24m

error during ./hack/ginkgo-e2e.sh --ginkgo.focus=\[Serial\]|\[Disruptive\] --ginkgo.skip=\[Driver:.gcepd\]|\[Flaky\]|\[Feature:.+\] --minStartupPods=8 --report-dir=/workspace/_artifacts --disable-log-dump=true: exit status 1
				from junit_runner.xml

Filter through log files | View test history on testgrid

Show 13698 Skipped Tests

Result	FAILURE
Tests	3 failed / 494 succeeded
Started	2023-01-30 17:06
Elapsed	5h47m
Revision
Builder	6c55144c-a0c0-11ed-9a0c-f25a29267a91
control_plane_node_os_image	cos-97-16919-103-16
infra-commit	3bf935f41
job-version	v1.27.0-alpha.1.90+c829397f7a1fa9
kubetest-version	v20230127-9396ca613c
revision	v1.27.0-alpha.1.90+c829397f7a1fa9
worker_node_os_image	cos-97-16919-103-16

ci-kubernetes-e2e-gci-gce-serial #1620106158725402624

Test Failures

Kubernetes e2e suite [It] [sig-cloud-provider-gcp] Restart [Disruptive] should restart all nodes and ensure all nodes and pods recover 7m15s

Kubernetes e2e suite [It] [sig-cloud-provider-gcp] Restart [Disruptive] should restart all nodes and ensure all nodes and pods recover 7m15s

kubetest Test 5h24m

Passed

Skipped