ci-kubernetes-e2e-gci-gce-serial #1622323746830290944 Results

Test Failures

Kubernetes e2e suite [It] [sig-storage] In-tree Volumes [Driver: local][LocalVolumeType: dir] [Testpattern: Pre-provisioned PV (default fs)] subPath should unmount if pod is gracefully deleted while kubelet is down [Disruptive][Slow][LinuxOnly] 2m15s

go run hack/e2e.go -v --test --test_args='--ginkgo.focus=Kubernetes\se2e\ssuite\s\[It\]\s\[sig\-storage\]\sIn\-tree\sVolumes\s\[Driver\:\slocal\]\[LocalVolumeType\:\sdir\]\s\[Testpattern\:\sPre\-provisioned\sPV\s\(default\sfs\)\]\ssubPath\sshould\sunmount\sif\spod\sis\sgracefully\sdeleted\swhile\skubelet\sis\sdown\s\[Disruptive\]\[Slow\]\[LinuxOnly\]$'

[FAILED] SSH to Node "bootstrap-e2e-minion-group-7p0n" errored.: error getting SSH client to prow@34.118.193.103:22: timed out waiting for the condition
In [DeferCleanup (Each)] at: test/e2e/storage/utils/pod.go:117 @ 02/05/23 23:56:44.852

				> Enter [BeforeEach] [Testpattern: Pre-provisioned PV (default fs)] subPath - test/e2e/storage/framework/testsuite.go:51 @ 02/05/23 23:54:31.361
< Exit [BeforeEach] [Testpattern: Pre-provisioned PV (default fs)] subPath - test/e2e/storage/framework/testsuite.go:51 @ 02/05/23 23:54:31.361 (0s)
> Enter [BeforeEach] [Testpattern: Pre-provisioned PV (default fs)] subPath - set up framework | framework.go:188 @ 02/05/23 23:54:31.361
STEP: Creating a kubernetes client - test/e2e/framework/framework.go:208 @ 02/05/23 23:54:31.361
Feb  5 23:54:31.361: INFO: >>> kubeConfig: /workspace/.kube/config
STEP: Building a namespace api object, basename provisioning - test/e2e/framework/framework.go:247 @ 02/05/23 23:54:31.362
STEP: Waiting for a default service account to be provisioned in namespace - test/e2e/framework/framework.go:256 @ 02/05/23 23:54:31.487
STEP: Waiting for kube-root-ca.crt to be provisioned in namespace - test/e2e/framework/framework.go:259 @ 02/05/23 23:54:31.568
< Exit [BeforeEach] [Testpattern: Pre-provisioned PV (default fs)] subPath - set up framework | framework.go:188 @ 02/05/23 23:54:31.648 (287ms)
> Enter [BeforeEach] [Testpattern: Pre-provisioned PV (default fs)] subPath - test/e2e/framework/metrics/init/init.go:33 @ 02/05/23 23:54:31.648
< Exit [BeforeEach] [Testpattern: Pre-provisioned PV (default fs)] subPath - test/e2e/framework/metrics/init/init.go:33 @ 02/05/23 23:54:31.648 (0s)
> Enter [It] should unmount if pod is gracefully deleted while kubelet is down [Disruptive][Slow][LinuxOnly] - test/e2e/storage/testsuites/subpath.go:341 @ 02/05/23 23:54:31.648
Feb  5 23:54:31.693: INFO: In-tree plugin kubernetes.io/local-volume is not migrated, not validating any metrics
Feb  5 23:54:33.823: INFO: ExecWithOptions {Command:[nsenter --mount=/rootfs/proc/1/ns/mnt -- sh -c mkdir -p /tmp/local-driver-e8e50194-9ead-40de-af7c-8bcb8ceb62ca] Namespace:provisioning-9333 PodName:hostexec-bootstrap-e2e-minion-group-7p0n-6lr4r ContainerName:agnhost-container Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:true Quiet:false}
Feb  5 23:54:33.823: INFO: >>> kubeConfig: /workspace/.kube/config
Feb  5 23:54:33.824: INFO: ExecWithOptions: Clientset creation
Feb  5 23:54:33.824: INFO: ExecWithOptions: execute(POST https://35.247.68.66/api/v1/namespaces/provisioning-9333/pods/hostexec-bootstrap-e2e-minion-group-7p0n-6lr4r/exec?command=nsenter&command=--mount%3D%2Frootfs%2Fproc%2F1%2Fns%2Fmnt&command=--&command=sh&command=-c&command=mkdir+-p+%2Ftmp%2Flocal-driver-e8e50194-9ead-40de-af7c-8bcb8ceb62ca&container=agnhost-container&container=agnhost-container&stderr=true&stdout=true)
Feb  5 23:54:34.154: INFO: Creating resource for pre-provisioned PV
Feb  5 23:54:34.154: INFO: Creating PVC and PV
STEP: Creating a PVC followed by a PV - test/e2e/framework/pv/pv.go:360 @ 02/05/23 23:54:34.154
Feb  5 23:54:34.239: INFO: Waiting for PV local-szhdn to bind to PVC pvc-6wbt5
Feb  5 23:54:34.239: INFO: Waiting up to timeout=3m0s for PersistentVolumeClaims [pvc-6wbt5] to have phase Bound
Feb  5 23:54:34.280: INFO: PersistentVolumeClaim pvc-6wbt5 found but phase is Pending instead of Bound.
Feb  5 23:54:36.322: INFO: PersistentVolumeClaim pvc-6wbt5 found but phase is Pending instead of Bound.
Feb  5 23:54:38.363: INFO: PersistentVolumeClaim pvc-6wbt5 found but phase is Pending instead of Bound.
Feb  5 23:54:40.406: INFO: PersistentVolumeClaim pvc-6wbt5 found but phase is Pending instead of Bound.
Feb  5 23:54:42.448: INFO: PersistentVolumeClaim pvc-6wbt5 found but phase is Pending instead of Bound.
Feb  5 23:54:44.491: INFO: PersistentVolumeClaim pvc-6wbt5 found but phase is Pending instead of Bound.
Feb  5 23:54:46.534: INFO: PersistentVolumeClaim pvc-6wbt5 found and phase=Bound (12.294546325s)
Feb  5 23:54:46.534: INFO: Waiting up to 3m0s for PersistentVolume local-szhdn to have phase Bound
Feb  5 23:54:46.575: INFO: PersistentVolume local-szhdn found and phase=Bound (41.144816ms)
Feb  5 23:54:48.882: INFO: ExecWithOptions {Command:[nsenter --mount=/rootfs/proc/1/ns/mnt -- sh -c find /var/lib/kubelet/plugins -type d -exec mountpoint {} \; | grep 'is a mountpoint$' || true] Namespace:provisioning-9333 PodName:hostexec-bootstrap-e2e-minion-group-7p0n-7jcfh ContainerName:agnhost-container Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:true Quiet:false}
Feb  5 23:54:48.882: INFO: >>> kubeConfig: /workspace/.kube/config
Feb  5 23:54:48.883: INFO: ExecWithOptions: Clientset creation
Feb  5 23:54:48.884: INFO: ExecWithOptions: execute(POST https://35.247.68.66/api/v1/namespaces/provisioning-9333/pods/hostexec-bootstrap-e2e-minion-group-7p0n-7jcfh/exec?command=nsenter&command=--mount%3D%2Frootfs%2Fproc%2F1%2Fns%2Fmnt&command=--&command=sh&command=-c&command=find+%2Fvar%2Flib%2Fkubelet%2Fplugins+-type+d+-exec+mountpoint+%7B%7D+%5C%3B+%7C+grep+%27is+a+mountpoint%24%27+%7C%7C+true&container=agnhost-container&container=agnhost-container&stderr=true&stdout=true)
Feb  5 23:54:51.359: INFO: ExecWithOptions {Command:[nsenter --mount=/rootfs/proc/1/ns/mnt -- sh -c find /var/lib/kubelet/plugins -type d -exec mountpoint {} \; | grep 'is a mountpoint$' || true] Namespace:provisioning-9333 PodName:hostexec-bootstrap-e2e-minion-group-f2xv-jqtng ContainerName:agnhost-container Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:true Quiet:false}
Feb  5 23:54:51.359: INFO: >>> kubeConfig: /workspace/.kube/config
Feb  5 23:54:51.360: INFO: ExecWithOptions: Clientset creation
Feb  5 23:54:51.360: INFO: ExecWithOptions: execute(POST https://35.247.68.66/api/v1/namespaces/provisioning-9333/pods/hostexec-bootstrap-e2e-minion-group-f2xv-jqtng/exec?command=nsenter&command=--mount%3D%2Frootfs%2Fproc%2F1%2Fns%2Fmnt&command=--&command=sh&command=-c&command=find+%2Fvar%2Flib%2Fkubelet%2Fplugins+-type+d+-exec+mountpoint+%7B%7D+%5C%3B+%7C+grep+%27is+a+mountpoint%24%27+%7C%7C+true&container=agnhost-container&container=agnhost-container&stderr=true&stdout=true)
Feb  5 23:54:53.817: INFO: ExecWithOptions {Command:[nsenter --mount=/rootfs/proc/1/ns/mnt -- sh -c find /var/lib/kubelet/plugins -type d -exec mountpoint {} \; | grep 'is a mountpoint$' || true] Namespace:provisioning-9333 PodName:hostexec-bootstrap-e2e-minion-group-vc67-p67vj ContainerName:agnhost-container Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:true Quiet:false}
Feb  5 23:54:53.817: INFO: >>> kubeConfig: /workspace/.kube/config
Feb  5 23:54:53.818: INFO: ExecWithOptions: Clientset creation
Feb  5 23:54:53.818: INFO: ExecWithOptions: execute(POST https://35.247.68.66/api/v1/namespaces/provisioning-9333/pods/hostexec-bootstrap-e2e-minion-group-vc67-p67vj/exec?command=nsenter&command=--mount%3D%2Frootfs%2Fproc%2F1%2Fns%2Fmnt&command=--&command=sh&command=-c&command=find+%2Fvar%2Flib%2Fkubelet%2Fplugins+-type+d+-exec+mountpoint+%7B%7D+%5C%3B+%7C+grep+%27is+a+mountpoint%24%27+%7C%7C+true&container=agnhost-container&container=agnhost-container&stderr=true&stdout=true)
STEP: Creating pod pod-subpath-test-preprovisionedpv-q5d2 - test/e2e/storage/testsuites/subpath.go:985 @ 02/05/23 23:54:54.166
STEP: Expecting the volume mount to be found. - test/e2e/storage/utils/utils.go:140 @ 02/05/23 23:54:58.424
Feb  5 23:54:58.949: INFO: ssh prow@34.118.193.103:22: command:   mount | grep bea0ae1f-c8a1-4863-a0fa-28331261466e | grep -v volume-subpaths
Feb  5 23:54:58.949: INFO: ssh prow@34.118.193.103:22: stdout:    "tmpfs on /var/lib/kubelet/pods/bea0ae1f-c8a1-4863-a0fa-28331261466e/volumes/kubernetes.io~local-volume/local-szhdn type tmpfs (rw)\ntmpfs on /var/lib/kubelet/pods/bea0ae1f-c8a1-4863-a0fa-28331261466e/volumes/kubernetes.io~local-volume/local-szhdn type tmpfs (rw,nosuid,nodev,noexec)\ntmpfs on /home/kubernetes/containerized_mounter/rootfs/var/lib/kubelet/pods/bea0ae1f-c8a1-4863-a0fa-28331261466e/volumes/kubernetes.io~local-volume/local-szhdn type tmpfs (rw,nosuid,nodev,noexec)\ntmpfs on /home/kubernetes/containerized_mounter/rootfs/var/lib/kubelet/pods/bea0ae1f-c8a1-4863-a0fa-28331261466e/volumes/kubernetes.io~local-volume/local-szhdn type tmpfs (rw,nosuid,nodev,noexec)\ntmpfs on /var/lib/kubelet/pods/bea0ae1f-c8a1-4863-a0fa-28331261466e/volumes/kubernetes.io~projected/kube-api-access-mfp7v type tmpfs (rw,relatime,size=7376260k)\ntmpfs on /var/lib/kubelet/pods/bea0ae1f-c8a1-4863-a0fa-28331261466e/volumes/kubernetes.io~projected/kube-api-access-mfp7v type tmpfs (rw,relatime,size=7376260k)\ntmpfs on /home/kubernetes/containerized_mounter/rootfs/var/lib/kubelet/pods/bea0ae1f-c8a1-4863-a0fa-28331261466e/volumes/kubernetes.io~projected/kube-api-access-mfp7v type tmpfs (rw,relatime,size=7376260k)\ntmpfs on /home/kubernetes/containerized_mounter/rootfs/var/lib/kubelet/pods/bea0ae1f-c8a1-4863-a0fa-28331261466e/volumes/kubernetes.io~projected/kube-api-access-mfp7v type tmpfs (rw,relatime,size=7376260k)\n"
Feb  5 23:54:58.949: INFO: ssh prow@34.118.193.103:22: stderr:    ""
Feb  5 23:54:58.949: INFO: ssh prow@34.118.193.103:22: exit code: 0
STEP: Expecting the volume subpath mount to be found. - test/e2e/storage/utils/utils.go:147 @ 02/05/23 23:54:58.949
Feb  5 23:54:59.466: INFO: ssh prow@34.118.193.103:22: command:   cat /proc/self/mountinfo | grep bea0ae1f-c8a1-4863-a0fa-28331261466e | grep volume-subpaths
Feb  5 23:54:59.466: INFO: ssh prow@34.118.193.103:22: stdout:    "1515 457 0:30 /local-driver-e8e50194-9ead-40de-af7c-8bcb8ceb62ca/provisioning-9333 /var/lib/kubelet/pods/bea0ae1f-c8a1-4863-a0fa-28331261466e/volume-subpaths/local-szhdn/test-container-subpath-preprovisionedpv-q5d2/0 rw shared:18 - tmpfs tmpfs rw\n1518 161 0:30 /local-driver-e8e50194-9ead-40de-af7c-8bcb8ceb62ca/provisioning-9333 /var/lib/kubelet/pods/bea0ae1f-c8a1-4863-a0fa-28331261466e/volume-subpaths/local-szhdn/test-container-subpath-preprovisionedpv-q5d2/0 rw shared:18 - tmpfs tmpfs rw\n1517 500 0:30 /local-driver-e8e50194-9ead-40de-af7c-8bcb8ceb62ca/provisioning-9333 /home/kubernetes/containerized_mounter/rootfs/var/lib/kubelet/pods/bea0ae1f-c8a1-4863-a0fa-28331261466e/volume-subpaths/local-szhdn/test-container-subpath-preprovisionedpv-q5d2/0 rw shared:18 - tmpfs tmpfs rw\n1516 498 0:30 /local-driver-e8e50194-9ead-40de-af7c-8bcb8ceb62ca/provisioning-9333 /home/kubernetes/containerized_mounter/rootfs/var/lib/kubelet/pods/bea0ae1f-c8a1-4863-a0fa-28331261466e/volume-subpaths/local-szhdn/test-container-subpath-preprovisionedpv-q5d2/0 rw shared:18 - tmpfs tmpfs rw\n"
Feb  5 23:54:59.466: INFO: ssh prow@34.118.193.103:22: stderr:    ""
Feb  5 23:54:59.466: INFO: ssh prow@34.118.193.103:22: exit code: 0
STEP: Writing to the volume. - test/e2e/storage/utils/utils.go:154 @ 02/05/23 23:54:59.466
Feb  5 23:54:59.466: INFO: ExecWithOptions {Command:[/bin/sh -c echo 2MEQHYuLwl3E4xURkbaZRtABsmgRZUrQHVG7Ba4i3ISQ+OE6sqE6d6MMVk/9zWlZbuKbfayuCnEVsNUTCCJxEw== | base64 -d | sha256sum] Namespace:provisioning-9333 PodName:pod-subpath-test-preprovisionedpv-q5d2 ContainerName:test-container-subpath-preprovisionedpv-q5d2 Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false Quiet:false}
Feb  5 23:54:59.466: INFO: >>> kubeConfig: /workspace/.kube/config
Feb  5 23:54:59.467: INFO: ExecWithOptions: Clientset creation
Feb  5 23:54:59.467: INFO: ExecWithOptions: execute(POST https://35.247.68.66/api/v1/namespaces/provisioning-9333/pods/pod-subpath-test-preprovisionedpv-q5d2/exec?command=%2Fbin%2Fsh&command=-c&command=echo+2MEQHYuLwl3E4xURkbaZRtABsmgRZUrQHVG7Ba4i3ISQ%2BOE6sqE6d6MMVk%2F9zWlZbuKbfayuCnEVsNUTCCJxEw%3D%3D+%7C+base64+-d+%7C+sha256sum&container=test-container-subpath-preprovisionedpv-q5d2&container=test-container-subpath-preprovisionedpv-q5d2&stderr=true&stdout=true)
Feb  5 23:54:59.770: INFO: ExecWithOptions {Command:[/bin/sh -c echo 2MEQHYuLwl3E4xURkbaZRtABsmgRZUrQHVG7Ba4i3ISQ+OE6sqE6d6MMVk/9zWlZbuKbfayuCnEVsNUTCCJxEw== | base64 -d | dd of=/test-volume/file1.txt  bs=64 count=1] Namespace:provisioning-9333 PodName:pod-subpath-test-preprovisionedpv-q5d2 ContainerName:test-container-subpath-preprovisionedpv-q5d2 Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false Quiet:false}
Feb  5 23:54:59.770: INFO: >>> kubeConfig: /workspace/.kube/config
Feb  5 23:54:59.771: INFO: ExecWithOptions: Clientset creation
Feb  5 23:54:59.771: INFO: ExecWithOptions: execute(POST https://35.247.68.66/api/v1/namespaces/provisioning-9333/pods/pod-subpath-test-preprovisionedpv-q5d2/exec?command=%2Fbin%2Fsh&command=-c&command=echo+2MEQHYuLwl3E4xURkbaZRtABsmgRZUrQHVG7Ba4i3ISQ%2BOE6sqE6d6MMVk%2F9zWlZbuKbfayuCnEVsNUTCCJxEw%3D%3D+%7C+base64+-d+%7C+dd+of%3D%2Ftest-volume%2Ffile1.txt++bs%3D64+count%3D1&container=test-container-subpath-preprovisionedpv-q5d2&container=test-container-subpath-preprovisionedpv-q5d2&stderr=true&stdout=true)
STEP: Stopping the kubelet. - test/e2e/storage/utils/utils.go:161 @ 02/05/23 23:55:00.077
Feb  5 23:55:00.119: INFO: Checking if systemctl command is present
Feb  5 23:55:00.642: INFO: Checking if sudo command is present
Feb  5 23:55:01.163: INFO: Attempting `sudo systemctl stop kubelet`
Feb  5 23:55:01.722: INFO: ssh prow@34.118.193.103:22: command:   sudo systemctl stop kubelet
Feb  5 23:55:01.722: INFO: ssh prow@34.118.193.103:22: stdout:    ""
Feb  5 23:55:01.722: INFO: ssh prow@34.118.193.103:22: stderr:    ""
Feb  5 23:55:01.722: INFO: ssh prow@34.118.193.103:22: exit code: 0
Feb  5 23:55:01.722: INFO: Waiting up to 1m0s for node bootstrap-e2e-minion-group-7p0n condition Ready to be false
Feb  5 23:55:01.764: INFO: Condition Ready of node bootstrap-e2e-minion-group-7p0n is true instead of false. Reason: KubeletReady, message: kubelet is posting ready status. AppArmor enabled
Feb  5 23:55:03.808: INFO: Condition Ready of node bootstrap-e2e-minion-group-7p0n is true instead of false. Reason: KubeletReady, message: kubelet is posting ready status. AppArmor enabled
Feb  5 23:55:05.850: INFO: Condition Ready of node bootstrap-e2e-minion-group-7p0n is true instead of false. Reason: KubeletReady, message: kubelet is posting ready status. AppArmor enabled
Feb  5 23:55:07.895: INFO: Condition Ready of node bootstrap-e2e-minion-group-7p0n is true instead of false. Reason: KubeletReady, message: kubelet is posting ready status. AppArmor enabled
Feb  5 23:55:09.937: INFO: Condition Ready of node bootstrap-e2e-minion-group-7p0n is true instead of false. Reason: KubeletReady, message: kubelet is posting ready status. AppArmor enabled
Feb  5 23:55:11.982: INFO: Condition Ready of node bootstrap-e2e-minion-group-7p0n is true instead of false. Reason: KubeletReady, message: kubelet is posting ready status. AppArmor enabled
Feb  5 23:55:14.026: INFO: Condition Ready of node bootstrap-e2e-minion-group-7p0n is true instead of false. Reason: KubeletReady, message: kubelet is posting ready status. AppArmor enabled
Feb  5 23:55:16.069: INFO: Condition Ready of node bootstrap-e2e-minion-group-7p0n is true instead of false. Reason: KubeletReady, message: kubelet is posting ready status. AppArmor enabled
Feb  5 23:55:18.112: INFO: Condition Ready of node bootstrap-e2e-minion-group-7p0n is true instead of false. Reason: KubeletReady, message: kubelet is posting ready status. AppArmor enabled
Feb  5 23:55:20.155: INFO: Condition Ready of node bootstrap-e2e-minion-group-7p0n is true instead of false. Reason: KubeletReady, message: kubelet is posting ready status. AppArmor enabled
Feb  5 23:55:22.197: INFO: Condition Ready of node bootstrap-e2e-minion-group-7p0n is true instead of false. Reason: KubeletReady, message: kubelet is posting ready status. AppArmor enabled
Feb  5 23:55:24.240: INFO: Condition Ready of node bootstrap-e2e-minion-group-7p0n is true instead of false. Reason: KubeletReady, message: kubelet is posting ready status. AppArmor enabled
Feb  5 23:55:26.283: INFO: Condition Ready of node bootstrap-e2e-minion-group-7p0n is true instead of false. Reason: KubeletReady, message: kubelet is posting ready status. AppArmor enabled
Feb  5 23:55:28.327: INFO: Condition Ready of node bootstrap-e2e-minion-group-7p0n is true instead of false. Reason: KubeletReady, message: kubelet is posting ready status. AppArmor enabled
Feb  5 23:55:30.370: INFO: Condition Ready of node bootstrap-e2e-minion-group-7p0n is true instead of false. Reason: KubeletReady, message: kubelet is posting ready status. AppArmor enabled
Feb  5 23:55:32.414: INFO: Condition Ready of node bootstrap-e2e-minion-group-7p0n is true instead of false. Reason: KubeletReady, message: kubelet is posting ready status. AppArmor enabled
Feb  5 23:55:34.457: INFO: Condition Ready of node bootstrap-e2e-minion-group-7p0n is true instead of false. Reason: KubeletReady, message: kubelet is posting ready status. AppArmor enabled
Feb  5 23:55:36.500: INFO: Condition Ready of node bootstrap-e2e-minion-group-7p0n is true instead of false. Reason: KubeletReady, message: kubelet is posting ready status. AppArmor enabled
STEP: Deleting Pod "pod-subpath-test-preprovisionedpv-q5d2" - test/e2e/storage/utils/utils.go:170 @ 02/05/23 23:55:38.543
STEP: Starting the kubelet and waiting for pod to delete. - test/e2e/storage/utils/utils.go:178 @ 02/05/23 23:55:38.588
Feb  5 23:55:38.634: INFO: Checking if systemctl command is present
Feb  5 23:55:39.163: INFO: Checking if sudo command is present
Feb  5 23:55:39.687: INFO: Attempting `sudo systemctl start kubelet`
Feb  5 23:55:40.225: INFO: ssh prow@34.118.193.103:22: command:   sudo systemctl start kubelet
Feb  5 23:55:40.225: INFO: ssh prow@34.118.193.103:22: stdout:    ""
Feb  5 23:55:40.225: INFO: ssh prow@34.118.193.103:22: stderr:    ""
Feb  5 23:55:40.225: INFO: ssh prow@34.118.193.103:22: exit code: 0
Feb  5 23:55:40.225: INFO: Waiting up to 1m0s for node bootstrap-e2e-minion-group-7p0n condition Ready to be true
Feb  5 23:55:40.268: INFO: Condition Ready of node bootstrap-e2e-minion-group-7p0n is false instead of true. Reason: NodeStatusUnknown, message: Kubelet stopped posting node status.
STEP: Expecting the volume mount not to be found. - test/e2e/storage/utils/utils.go:214 @ 02/05/23 23:56:13.03
Feb  5 23:56:23.630: INFO: ssh prow@34.118.193.103:22: command:   mount | grep bea0ae1f-c8a1-4863-a0fa-28331261466e | grep -v volume-subpaths
Feb  5 23:56:23.630: INFO: ssh prow@34.118.193.103:22: stdout:    ""
Feb  5 23:56:23.630: INFO: ssh prow@34.118.193.103:22: stderr:    ""
Feb  5 23:56:23.630: INFO: ssh prow@34.118.193.103:22: exit code: 1
Feb  5 23:56:23.630: INFO: Volume unmounted on node bootstrap-e2e-minion-group-7p0n
STEP: Expecting the volume subpath mount not to be found. - test/e2e/storage/utils/utils.go:222 @ 02/05/23 23:56:23.63
Feb  5 23:56:24.150: INFO: ssh prow@34.118.193.103:22: command:   cat /proc/self/mountinfo | grep bea0ae1f-c8a1-4863-a0fa-28331261466e | grep volume-subpaths
Feb  5 23:56:24.150: INFO: ssh prow@34.118.193.103:22: stdout:    ""
Feb  5 23:56:24.150: INFO: ssh prow@34.118.193.103:22: stderr:    ""
Feb  5 23:56:24.150: INFO: ssh prow@34.118.193.103:22: exit code: 1
Feb  5 23:56:24.150: INFO: Subpath volume unmounted on node bootstrap-e2e-minion-group-7p0n
Feb  5 23:56:24.150: INFO: ExecWithOptions {Command:[nsenter --mount=/rootfs/proc/1/ns/mnt -- sh -c find /var/lib/kubelet/plugins -type d -exec mountpoint {} \; | grep 'is a mountpoint$' || true] Namespace:provisioning-9333 PodName:hostexec-bootstrap-e2e-minion-group-7p0n-7jcfh ContainerName:agnhost-container Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:true Quiet:false}
Feb  5 23:56:24.150: INFO: >>> kubeConfig: /workspace/.kube/config
Feb  5 23:56:24.151: INFO: ExecWithOptions: Clientset creation
Feb  5 23:56:24.151: INFO: ExecWithOptions: execute(POST https://35.247.68.66/api/v1/namespaces/provisioning-9333/pods/hostexec-bootstrap-e2e-minion-group-7p0n-7jcfh/exec?command=nsenter&command=--mount%3D%2Frootfs%2Fproc%2F1%2Fns%2Fmnt&command=--&command=sh&command=-c&command=find+%2Fvar%2Flib%2Fkubelet%2Fplugins+-type+d+-exec+mountpoint+%7B%7D+%5C%3B+%7C+grep+%27is+a+mountpoint%24%27+%7C%7C+true&container=agnhost-container&container=agnhost-container&stderr=true&stdout=true)
< Exit [It] should unmount if pod is gracefully deleted while kubelet is down [Disruptive][Slow][LinuxOnly] - test/e2e/storage/testsuites/subpath.go:341 @ 02/05/23 23:56:24.53 (1m52.881s)
> Enter [AfterEach] [Testpattern: Pre-provisioned PV (default fs)] subPath - test/e2e/framework/node/init/init.go:33 @ 02/05/23 23:56:24.53
Feb  5 23:56:24.530: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready
< Exit [AfterEach] [Testpattern: Pre-provisioned PV (default fs)] subPath - test/e2e/framework/node/init/init.go:33 @ 02/05/23 23:56:24.573 (43ms)
> Enter [DeferCleanup (Each)] [Testpattern: Pre-provisioned PV (default fs)] subPath - test/e2e/storage/utils/utils.go:160 @ 02/05/23 23:56:24.573
Feb  5 23:56:24.615: INFO: Checking if systemctl command is present
Feb  5 23:56:44.852: INFO: Unexpected error: SSH to Node "bootstrap-e2e-minion-group-7p0n" errored.: 
    <*fmt.wrapError | 0xc00495b520>: {
        msg: "error getting SSH client to prow@34.118.193.103:22: timed out waiting for the condition",
        err: <*errors.errorString | 0xae80790>{
            s: "timed out waiting for the condition",
        },
    }
[FAILED] SSH to Node "bootstrap-e2e-minion-group-7p0n" errored.: error getting SSH client to prow@34.118.193.103:22: timed out waiting for the condition
In [DeferCleanup (Each)] at: test/e2e/storage/utils/pod.go:117 @ 02/05/23 23:56:44.852
< Exit [DeferCleanup (Each)] [Testpattern: Pre-provisioned PV (default fs)] subPath - test/e2e/storage/utils/utils.go:160 @ 02/05/23 23:56:44.852 (20.279s)
> Enter [DeferCleanup (Each)] [Testpattern: Pre-provisioned PV (default fs)] subPath - test/e2e/storage/testsuites/subpath.go:344 @ 02/05/23 23:56:44.852
STEP: Deleting pod - test/e2e/storage/testsuites/subpath.go:172 @ 02/05/23 23:56:44.852
Feb  5 23:56:44.852: INFO: Deleting pod "pod-subpath-test-preprovisionedpv-q5d2" in namespace "provisioning-9333"
STEP: Deleting pv and pvc - test/e2e/storage/framework/volume_resource.go:176 @ 02/05/23 23:56:44.895
Feb  5 23:56:44.895: INFO: Deleting PersistentVolumeClaim "pvc-6wbt5"
Feb  5 23:56:44.940: INFO: Deleting PersistentVolume "local-szhdn"
STEP: Removing the test directory - test/e2e/storage/utils/local.go:209 @ 02/05/23 23:56:44.984
Feb  5 23:56:44.984: INFO: ExecWithOptions {Command:[nsenter --mount=/rootfs/proc/1/ns/mnt -- sh -c rm -r /tmp/local-driver-e8e50194-9ead-40de-af7c-8bcb8ceb62ca] Namespace:provisioning-9333 PodName:hostexec-bootstrap-e2e-minion-group-7p0n-6lr4r ContainerName:agnhost-container Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:true Quiet:false}
Feb  5 23:56:44.984: INFO: >>> kubeConfig: /workspace/.kube/config
Feb  5 23:56:44.985: INFO: ExecWithOptions: Clientset creation
Feb  5 23:56:44.985: INFO: ExecWithOptions: execute(POST https://35.247.68.66/api/v1/namespaces/provisioning-9333/pods/hostexec-bootstrap-e2e-minion-group-7p0n-6lr4r/exec?command=nsenter&command=--mount%3D%2Frootfs%2Fproc%2F1%2Fns%2Fmnt&command=--&command=sh&command=-c&command=rm+-r+%2Ftmp%2Flocal-driver-e8e50194-9ead-40de-af7c-8bcb8ceb62ca&container=agnhost-container&container=agnhost-container&stderr=true&stdout=true)
STEP: Deleting pod hostexec-bootstrap-e2e-minion-group-7p0n-7jcfh in namespace provisioning-9333 - test/e2e/framework/pod/delete.go:41 @ 02/05/23 23:56:45.3
STEP: Deleting pod hostexec-bootstrap-e2e-minion-group-f2xv-jqtng in namespace provisioning-9333 - test/e2e/framework/pod/delete.go:41 @ 02/05/23 23:56:45.35
STEP: Deleting pod hostexec-bootstrap-e2e-minion-group-vc67-p67vj in namespace provisioning-9333 - test/e2e/framework/pod/delete.go:41 @ 02/05/23 23:56:45.4
< Exit [DeferCleanup (Each)] [Testpattern: Pre-provisioned PV (default fs)] subPath - test/e2e/storage/testsuites/subpath.go:344 @ 02/05/23 23:56:45.45 (598ms)
> Enter [DeferCleanup (Each)] [Testpattern: Pre-provisioned PV (default fs)] subPath - test/e2e/storage/drivers/in_tree.go:1729 @ 02/05/23 23:56:45.45
STEP: Deleting pod hostexec-bootstrap-e2e-minion-group-7p0n-6lr4r in namespace provisioning-9333 - test/e2e/framework/pod/delete.go:41 @ 02/05/23 23:56:45.451
< Exit [DeferCleanup (Each)] [Testpattern: Pre-provisioned PV (default fs)] subPath - test/e2e/storage/drivers/in_tree.go:1729 @ 02/05/23 23:56:45.499 (49ms)
> Enter [DeferCleanup (Each)] [Testpattern: Pre-provisioned PV (default fs)] subPath - test/e2e/framework/metrics/init/init.go:35 @ 02/05/23 23:56:45.499
< Exit [DeferCleanup (Each)] [Testpattern: Pre-provisioned PV (default fs)] subPath - test/e2e/framework/metrics/init/init.go:35 @ 02/05/23 23:56:45.5 (0s)
> Enter [DeferCleanup (Each)] [Testpattern: Pre-provisioned PV (default fs)] subPath - dump namespaces | framework.go:206 @ 02/05/23 23:56:45.5
STEP: dump namespace information after failure - test/e2e/framework/framework.go:284 @ 02/05/23 23:56:45.5
STEP: Collecting events from namespace "provisioning-9333". - test/e2e/framework/debug/dump.go:42 @ 02/05/23 23:56:45.5
STEP: Found 34 events. - test/e2e/framework/debug/dump.go:46 @ 02/05/23 23:56:45.542
Feb  5 23:56:45.542: INFO: At 2023-02-05 23:54:31 +0000 UTC - event for hostexec-bootstrap-e2e-minion-group-7p0n-6lr4r: {default-scheduler } Scheduled: Successfully assigned provisioning-9333/hostexec-bootstrap-e2e-minion-group-7p0n-6lr4r to bootstrap-e2e-minion-group-7p0n
Feb  5 23:56:45.542: INFO: At 2023-02-05 23:54:32 +0000 UTC - event for hostexec-bootstrap-e2e-minion-group-7p0n-6lr4r: {kubelet bootstrap-e2e-minion-group-7p0n} Pulled: Container image "registry.k8s.io/e2e-test-images/agnhost:2.43" already present on machine
Feb  5 23:56:45.542: INFO: At 2023-02-05 23:54:32 +0000 UTC - event for hostexec-bootstrap-e2e-minion-group-7p0n-6lr4r: {kubelet bootstrap-e2e-minion-group-7p0n} Created: Created container agnhost-container
Feb  5 23:56:45.542: INFO: At 2023-02-05 23:54:32 +0000 UTC - event for hostexec-bootstrap-e2e-minion-group-7p0n-6lr4r: {kubelet bootstrap-e2e-minion-group-7p0n} Started: Started container agnhost-container
Feb  5 23:56:45.542: INFO: At 2023-02-05 23:54:34 +0000 UTC - event for pvc-6wbt5: {persistentvolume-controller } ProvisioningFailed: storageclass.storage.k8s.io "provisioning-9333" not found
Feb  5 23:56:45.542: INFO: At 2023-02-05 23:54:46 +0000 UTC - event for hostexec-bootstrap-e2e-minion-group-7p0n-7jcfh: {default-scheduler } Scheduled: Successfully assigned provisioning-9333/hostexec-bootstrap-e2e-minion-group-7p0n-7jcfh to bootstrap-e2e-minion-group-7p0n
Feb  5 23:56:45.542: INFO: At 2023-02-05 23:54:47 +0000 UTC - event for hostexec-bootstrap-e2e-minion-group-7p0n-7jcfh: {kubelet bootstrap-e2e-minion-group-7p0n} Created: Created container agnhost-container
Feb  5 23:56:45.542: INFO: At 2023-02-05 23:54:47 +0000 UTC - event for hostexec-bootstrap-e2e-minion-group-7p0n-7jcfh: {kubelet bootstrap-e2e-minion-group-7p0n} Pulled: Container image "registry.k8s.io/e2e-test-images/agnhost:2.43" already present on machine
Feb  5 23:56:45.542: INFO: At 2023-02-05 23:54:47 +0000 UTC - event for hostexec-bootstrap-e2e-minion-group-7p0n-7jcfh: {kubelet bootstrap-e2e-minion-group-7p0n} Started: Started container agnhost-container
Feb  5 23:56:45.542: INFO: At 2023-02-05 23:54:49 +0000 UTC - event for hostexec-bootstrap-e2e-minion-group-f2xv-jqtng: {default-scheduler } Scheduled: Successfully assigned provisioning-9333/hostexec-bootstrap-e2e-minion-group-f2xv-jqtng to bootstrap-e2e-minion-group-f2xv
Feb  5 23:56:45.542: INFO: At 2023-02-05 23:54:49 +0000 UTC - event for hostexec-bootstrap-e2e-minion-group-f2xv-jqtng: {kubelet bootstrap-e2e-minion-group-f2xv} Pulled: Container image "registry.k8s.io/e2e-test-images/agnhost:2.43" already present on machine
Feb  5 23:56:45.542: INFO: At 2023-02-05 23:54:49 +0000 UTC - event for hostexec-bootstrap-e2e-minion-group-f2xv-jqtng: {kubelet bootstrap-e2e-minion-group-f2xv} Created: Created container agnhost-container
Feb  5 23:56:45.542: INFO: At 2023-02-05 23:54:49 +0000 UTC - event for hostexec-bootstrap-e2e-minion-group-f2xv-jqtng: {kubelet bootstrap-e2e-minion-group-f2xv} Started: Started container agnhost-container
Feb  5 23:56:45.542: INFO: At 2023-02-05 23:54:51 +0000 UTC - event for hostexec-bootstrap-e2e-minion-group-vc67-p67vj: {default-scheduler } Scheduled: Successfully assigned provisioning-9333/hostexec-bootstrap-e2e-minion-group-vc67-p67vj to bootstrap-e2e-minion-group-vc67
Feb  5 23:56:45.542: INFO: At 2023-02-05 23:54:52 +0000 UTC - event for hostexec-bootstrap-e2e-minion-group-vc67-p67vj: {kubelet bootstrap-e2e-minion-group-vc67} Created: Created container agnhost-container
Feb  5 23:56:45.542: INFO: At 2023-02-05 23:54:52 +0000 UTC - event for hostexec-bootstrap-e2e-minion-group-vc67-p67vj: {kubelet bootstrap-e2e-minion-group-vc67} Started: Started container agnhost-container
Feb  5 23:56:45.542: INFO: At 2023-02-05 23:54:52 +0000 UTC - event for hostexec-bootstrap-e2e-minion-group-vc67-p67vj: {kubelet bootstrap-e2e-minion-group-vc67} Pulled: Container image "registry.k8s.io/e2e-test-images/agnhost:2.43" already present on machine
Feb  5 23:56:45.542: INFO: At 2023-02-05 23:54:54 +0000 UTC - event for pod-subpath-test-preprovisionedpv-q5d2: {kubelet bootstrap-e2e-minion-group-7p0n} Pulling: Pulling image "registry.k8s.io/e2e-test-images/busybox:1.29-4"
Feb  5 23:56:45.542: INFO: At 2023-02-05 23:54:54 +0000 UTC - event for pod-subpath-test-preprovisionedpv-q5d2: {default-scheduler } Scheduled: Successfully assigned provisioning-9333/pod-subpath-test-preprovisionedpv-q5d2 to bootstrap-e2e-minion-group-7p0n
Feb  5 23:56:45.542: INFO: At 2023-02-05 23:54:55 +0000 UTC - event for pod-subpath-test-preprovisionedpv-q5d2: {kubelet bootstrap-e2e-minion-group-7p0n} Pulled: Successfully pulled image "registry.k8s.io/e2e-test-images/busybox:1.29-4" in 423.146995ms (423.15762ms including waiting)
Feb  5 23:56:45.542: INFO: At 2023-02-05 23:54:55 +0000 UTC - event for pod-subpath-test-preprovisionedpv-q5d2: {kubelet bootstrap-e2e-minion-group-7p0n} Started: Started container test-container-volume-preprovisionedpv-q5d2
Feb  5 23:56:45.542: INFO: At 2023-02-05 23:54:55 +0000 UTC - event for pod-subpath-test-preprovisionedpv-q5d2: {kubelet bootstrap-e2e-minion-group-7p0n} Created: Created container test-container-volume-preprovisionedpv-q5d2
Feb  5 23:56:45.542: INFO: At 2023-02-05 23:54:55 +0000 UTC - event for pod-subpath-test-preprovisionedpv-q5d2: {kubelet bootstrap-e2e-minion-group-7p0n} Pulled: Container image "registry.k8s.io/e2e-test-images/busybox:1.29-4" already present on machine
Feb  5 23:56:45.542: INFO: At 2023-02-05 23:54:55 +0000 UTC - event for pod-subpath-test-preprovisionedpv-q5d2: {kubelet bootstrap-e2e-minion-group-7p0n} Started: Started container test-container-subpath-preprovisionedpv-q5d2
Feb  5 23:56:45.542: INFO: At 2023-02-05 23:54:55 +0000 UTC - event for pod-subpath-test-preprovisionedpv-q5d2: {kubelet bootstrap-e2e-minion-group-7p0n} Created: Created container test-container-subpath-preprovisionedpv-q5d2
Feb  5 23:56:45.542: INFO: At 2023-02-05 23:55:37 +0000 UTC - event for hostexec-bootstrap-e2e-minion-group-7p0n-6lr4r: {node-controller } NodeNotReady: Node is not ready
Feb  5 23:56:45.542: INFO: At 2023-02-05 23:55:37 +0000 UTC - event for hostexec-bootstrap-e2e-minion-group-7p0n-7jcfh: {node-controller } NodeNotReady: Node is not ready
Feb  5 23:56:45.542: INFO: At 2023-02-05 23:55:37 +0000 UTC - event for pod-subpath-test-preprovisionedpv-q5d2: {node-controller } NodeNotReady: Node is not ready
Feb  5 23:56:45.542: INFO: At 2023-02-05 23:55:41 +0000 UTC - event for pod-subpath-test-preprovisionedpv-q5d2: {kubelet bootstrap-e2e-minion-group-7p0n} Killing: Stopping container test-container-subpath-preprovisionedpv-q5d2
Feb  5 23:56:45.542: INFO: At 2023-02-05 23:55:41 +0000 UTC - event for pod-subpath-test-preprovisionedpv-q5d2: {kubelet bootstrap-e2e-minion-group-7p0n} Killing: Stopping container test-container-volume-preprovisionedpv-q5d2
Feb  5 23:56:45.542: INFO: At 2023-02-05 23:56:45 +0000 UTC - event for hostexec-bootstrap-e2e-minion-group-7p0n-6lr4r: {kubelet bootstrap-e2e-minion-group-7p0n} Killing: Stopping container agnhost-container
Feb  5 23:56:45.542: INFO: At 2023-02-05 23:56:45 +0000 UTC - event for hostexec-bootstrap-e2e-minion-group-7p0n-7jcfh: {kubelet bootstrap-e2e-minion-group-7p0n} Killing: Stopping container agnhost-container
Feb  5 23:56:45.542: INFO: At 2023-02-05 23:56:45 +0000 UTC - event for hostexec-bootstrap-e2e-minion-group-f2xv-jqtng: {kubelet bootstrap-e2e-minion-group-f2xv} Killing: Stopping container agnhost-container
Feb  5 23:56:45.542: INFO: At 2023-02-05 23:56:45 +0000 UTC - event for hostexec-bootstrap-e2e-minion-group-vc67-p67vj: {kubelet bootstrap-e2e-minion-group-vc67} Killing: Stopping container agnhost-container
Feb  5 23:56:45.583: INFO: POD  NODE  PHASE  GRACE  CONDITIONS
Feb  5 23:56:45.583: INFO: 
Feb  5 23:56:45.625: INFO: 
Logging node info for node bootstrap-e2e-master
Feb  5 23:56:45.667: INFO: Node Info: &Node{ObjectMeta:{bootstrap-e2e-master    cff6e042-47eb-4a9a-b932-5766273d8cbd 62051 0 2023-02-05 20:03:54 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:n1-standard-1 beta.kubernetes.io/os:linux cloud.google.com/metadata-proxy-ready:true failure-domain.beta.kubernetes.io/region:us-west1 failure-domain.beta.kubernetes.io/zone:us-west1-b kubernetes.io/arch:amd64 kubernetes.io/hostname:bootstrap-e2e-master kubernetes.io/os:linux node.kubernetes.io/instance-type:n1-standard-1 topology.gke.io/zone:us-west1-b topology.kubernetes.io/region:us-west1 topology.kubernetes.io/zone:us-west1-b] map[node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kubelet Update v1 2023-02-05 20:03:54 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:cloud.google.com/metadata-proxy-ready":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{},"f:unschedulable":{}}} } {kube-controller-manager Update v1 2023-02-05 20:04:10 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"10.64.3.0/24\"":{}},"f:taints":{}}} } {kube-controller-manager Update v1 2023-02-05 20:04:10 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}}}} status} {kubelet Update v1 2023-02-05 23:55:24 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:topology.gke.io/zone":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:10.64.3.0/24,DoNotUseExternalID:,ProviderID:gce://k8s-infra-e2e-boskos-001/us-west1-b/bootstrap-e2e-master,Unschedulable:true,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/master,Value:,Effect:NoSchedule,TimeAdded:<nil>,},Taint{Key:node.kubernetes.io/unschedulable,Value:,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[10.64.3.0/24],},Status:NodeStatus{Capacity:ResourceList{attachable-volumes-gce-pd: {{127 0} {<nil>} 127 DecimalSI},cpu: {{1 0} {<nil>} 1 DecimalSI},ephemeral-storage: {{16656896000 0} {<nil>} 16266500Ki BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3858370560 0} {<nil>} 3767940Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{attachable-volumes-gce-pd: {{127 0} {<nil>} 127 DecimalSI},cpu: {{1 0} {<nil>} 1 DecimalSI},ephemeral-storage: {{14991206376 0} {<nil>} 14991206376 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3596226560 0} {<nil>} 3511940Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2023-02-05 20:04:10 +0000 UTC,LastTransitionTime:2023-02-05 20:04:10 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-02-05 23:55:24 +0000 UTC,LastTransitionTime:2023-02-05 20:03:54 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-02-05 23:55:24 +0000 UTC,LastTransitionTime:2023-02-05 20:03:54 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-02-05 23:55:24 +0000 UTC,LastTransitionTime:2023-02-05 20:03:54 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-02-05 23:55:24 +0000 UTC,LastTransitionTime:2023-02-05 20:03:56 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:10.138.0.2,},NodeAddress{Type:ExternalIP,Address:35.247.68.66,},NodeAddress{Type:InternalDNS,Address:bootstrap-e2e-master.c.k8s-infra-e2e-boskos-001.internal,},NodeAddress{Type:Hostname,Address:bootstrap-e2e-master.c.k8s-infra-e2e-boskos-001.internal,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:6c56a7468028f8335ce3a87f603c6ce3,SystemUUID:6c56a746-8028-f833-5ce3-a87f603c6ce3,BootID:5e24514d-7978-4290-b3ac-f42b99d56778,KernelVersion:5.10.123+,OSImage:Container-Optimized OS from Google,ContainerRuntimeVersion:containerd://1.6.6,KubeletVersion:v1.27.0-alpha.1.249+cf14b50b0d2215,KubeProxyVersion:v1.27.0-alpha.1.249+cf14b50b0d2215,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[registry.k8s.io/kube-apiserver-amd64:v1.27.0-alpha.1.249_cf14b50b0d2215],SizeBytes:136985951,},ContainerImage{Names:[registry.k8s.io/kube-controller-manager-amd64:v1.27.0-alpha.1.249_cf14b50b0d2215],SizeBytes:126037497,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:51eae8381dcb1078289fa7b4f3df2630cdc18d09fb56f8e56b41c40e191d6c83 registry.k8s.io/etcd:3.5.7-0],SizeBytes:101639218,},ContainerImage{Names:[registry.k8s.io/cloud-provider-gcp/gcp-compute-persistent-disk-csi-driver@sha256:e9c355dd1ce57de91a0f9164bd2fe688938723e7f9bd40bb13652d603f41d25a registry.k8s.io/cloud-provider-gcp/gcp-compute-persistent-disk-csi-driver:v1.2.2],SizeBytes:73354282,},ContainerImage{Names:[registry.k8s.io/kube-scheduler-amd64:v1.27.0-alpha.1.249_cf14b50b0d2215],SizeBytes:58412524,},ContainerImage{Names:[gke.gcr.io/prometheus-to-sd@sha256:e739643c3939ba0b161425f45a1989eedfc4a3b166db9a7100863296b4c70510 gke.gcr.io/prometheus-to-sd:v0.11.1-gke.1],SizeBytes:48742566,},ContainerImage{Names:[gcr.io/k8s-ingress-image-push/ingress-gce-glbc-amd64@sha256:5db27383add6d9f4ebdf0286409ac31f7f5d273690204b341a4e37998917693b gcr.io/k8s-ingress-image-push/ingress-gce-glbc-amd64:v1.20.1],SizeBytes:36598135,},ContainerImage{Names:[registry.k8s.io/addon-manager/kube-addon-manager@sha256:49cc4e6e4a3745b427ce14b0141476ab339bb65c6bc05033019e046c8727dcb0 registry.k8s.io/addon-manager/kube-addon-manager:v9.1.6],SizeBytes:30464183,},ContainerImage{Names:[registry.k8s.io/kas-network-proxy/proxy-server@sha256:b1389e7014425a1752aac55f5043ef4c52edaef0e223bf4d48ed1324e298087c registry.k8s.io/kas-network-proxy/proxy-server:v0.1.1],SizeBytes:21875112,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/metadata-proxy@sha256:e914645f22e946bce5165737e1b244e0a296ad1f0f81a9531adc57af2780978a registry.k8s.io/metadata-proxy:v0.1.12],SizeBytes:5301657,},ContainerImage{Names:[registry.k8s.io/pause@sha256:7031c1b283388d2c2e09b57badb803c05ebed362dc88d84b480cc47f72a21097 registry.k8s.io/pause:3.9],SizeBytes:321520,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},}
Feb  5 23:56:45.667: INFO: 
Logging kubelet events for node bootstrap-e2e-master
Feb  5 23:56:45.709: INFO: 
Logging pods the kubelet thinks is on node bootstrap-e2e-master
Feb  5 23:56:45.765: INFO: etcd-server-bootstrap-e2e-master started at 2023-02-05 20:03:12 +0000 UTC (0+1 container statuses recorded)
Feb  5 23:56:45.765: INFO: 	Container etcd-container ready: true, restart count 0
Feb  5 23:56:45.765: INFO: l7-lb-controller-bootstrap-e2e-master started at 2023-02-05 20:03:28 +0000 UTC (0+1 container statuses recorded)
Feb  5 23:56:45.765: INFO: 	Container l7-lb-controller ready: true, restart count 2
Feb  5 23:56:45.765: INFO: kube-controller-manager-bootstrap-e2e-master started at 2023-02-05 20:03:12 +0000 UTC (0+1 container statuses recorded)
Feb  5 23:56:45.765: INFO: 	Container kube-controller-manager ready: true, restart count 3
Feb  5 23:56:45.765: INFO: kube-scheduler-bootstrap-e2e-master started at 2023-02-05 20:03:12 +0000 UTC (0+1 container statuses recorded)
Feb  5 23:56:45.765: INFO: 	Container kube-scheduler ready: true, restart count 0
Feb  5 23:56:45.765: INFO: etcd-server-events-bootstrap-e2e-master started at 2023-02-05 20:03:12 +0000 UTC (0+1 container statuses recorded)
Feb  5 23:56:45.765: INFO: 	Container etcd-container ready: true, restart count 0
Feb  5 23:56:45.765: INFO: konnectivity-server-bootstrap-e2e-master started at 2023-02-05 20:03:12 +0000 UTC (0+1 container statuses recorded)
Feb  5 23:56:45.765: INFO: 	Container konnectivity-server-container ready: true, restart count 0
Feb  5 23:56:45.765: INFO: kube-apiserver-bootstrap-e2e-master started at 2023-02-05 20:03:12 +0000 UTC (0+1 container statuses recorded)
Feb  5 23:56:45.765: INFO: 	Container kube-apiserver ready: true, restart count 0
Feb  5 23:56:45.765: INFO: kube-addon-manager-bootstrap-e2e-master started at 2023-02-05 20:03:28 +0000 UTC (0+1 container statuses recorded)
Feb  5 23:56:45.765: INFO: 	Container kube-addon-manager ready: true, restart count 0
Feb  5 23:56:45.765: INFO: metadata-proxy-v0.1-xv5df started at 2023-02-05 20:03:55 +0000 UTC (0+2 container statuses recorded)
Feb  5 23:56:45.765: INFO: 	Container metadata-proxy ready: true, restart count 0
Feb  5 23:56:45.765: INFO: 	Container prometheus-to-sd-exporter ready: true, restart count 0
Feb  5 23:56:45.991: INFO: 
Latency metrics for node bootstrap-e2e-master
Feb  5 23:56:45.991: INFO: 
Logging node info for node bootstrap-e2e-minion-group-7p0n
Feb  5 23:56:46.033: INFO: Node Info: &Node{ObjectMeta:{bootstrap-e2e-minion-group-7p0n    1b0c4b6f-6ac7-457b-aaed-c0715264fb74 62102 0 2023-02-05 23:49:36 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:n1-standard-2 beta.kubernetes.io/os:linux cloud.google.com/metadata-proxy-ready:true failure-domain.beta.kubernetes.io/region:us-west1 failure-domain.beta.kubernetes.io/zone:us-west1-b kubernetes.io/arch:amd64 kubernetes.io/hostname:bootstrap-e2e-minion-group-7p0n kubernetes.io/os:linux node.kubernetes.io/instance-type:n1-standard-2 topology.gke.io/zone:us-west1-b topology.kubernetes.io/region:us-west1 topology.kubernetes.io/zone:us-west1-b] map[node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kubelet Update v1 2023-02-05 23:49:36 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:cloud.google.com/metadata-proxy-ready":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {node-problem-detector Update v1 2023-02-05 23:54:41 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"CorruptDockerOverlay2\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"FrequentContainerdRestart\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"FrequentDockerRestart\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"FrequentKubeletRestart\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"FrequentUnregisterNetDevice\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"KernelDeadlock\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"ReadonlyFilesystem\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kube-controller-manager Update v1 2023-02-05 23:55:37 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}}}} status} {kube-controller-manager Update v1 2023-02-05 23:55:40 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"10.64.2.0/24\"":{}}}} } {kubelet Update v1 2023-02-05 23:55:40 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:topology.gke.io/zone":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:10.64.2.0/24,DoNotUseExternalID:,ProviderID:gce://k8s-infra-e2e-boskos-001/us-west1-b/bootstrap-e2e-minion-group-7p0n,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[10.64.2.0/24],},Status:NodeStatus{Capacity:ResourceList{attachable-volumes-gce-pd: {{127 0} {<nil>} 127 DecimalSI},cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{101203873792 0} {<nil>} 98831908Ki BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{7815434240 0} {<nil>} 7632260Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{attachable-volumes-gce-pd: {{127 0} {<nil>} 127 DecimalSI},cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{91083486262 0} {<nil>} 91083486262 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{7553290240 0} {<nil>} 7376260Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:FrequentDockerRestart,Status:False,LastHeartbeatTime:2023-02-05 23:54:41 +0000 UTC,LastTransitionTime:2023-02-05 23:49:40 +0000 UTC,Reason:NoFrequentDockerRestart,Message:docker is functioning properly,},NodeCondition{Type:FrequentContainerdRestart,Status:False,LastHeartbeatTime:2023-02-05 23:54:41 +0000 UTC,LastTransitionTime:2023-02-05 23:49:40 +0000 UTC,Reason:NoFrequentContainerdRestart,Message:containerd is functioning properly,},NodeCondition{Type:CorruptDockerOverlay2,Status:False,LastHeartbeatTime:2023-02-05 23:54:41 +0000 UTC,LastTransitionTime:2023-02-05 23:49:40 +0000 UTC,Reason:NoCorruptDockerOverlay2,Message:docker overlay2 is functioning properly,},NodeCondition{Type:KernelDeadlock,Status:False,LastHeartbeatTime:2023-02-05 23:54:41 +0000 UTC,LastTransitionTime:2023-02-05 23:49:40 +0000 UTC,Reason:KernelHasNoDeadlock,Message:kernel has no deadlock,},NodeCondition{Type:ReadonlyFilesystem,Status:False,LastHeartbeatTime:2023-02-05 23:54:41 +0000 UTC,LastTransitionTime:2023-02-05 23:49:40 +0000 UTC,Reason:FilesystemIsNotReadOnly,Message:Filesystem is not read-only,},NodeCondition{Type:FrequentUnregisterNetDevice,Status:False,LastHeartbeatTime:2023-02-05 23:54:41 +0000 UTC,LastTransitionTime:2023-02-05 23:49:40 +0000 UTC,Reason:NoFrequentUnregisterNetDevice,Message:node is functioning properly,},NodeCondition{Type:FrequentKubeletRestart,Status:False,LastHeartbeatTime:2023-02-05 23:54:41 +0000 UTC,LastTransitionTime:2023-02-05 23:49:40 +0000 UTC,Reason:NoFrequentKubeletRestart,Message:kubelet is functioning properly,},NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2023-02-05 23:49:45 +0000 UTC,LastTransitionTime:2023-02-05 23:49:45 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-02-05 23:55:40 +0000 UTC,LastTransitionTime:2023-02-05 23:55:40 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-02-05 23:55:40 +0000 UTC,LastTransitionTime:2023-02-05 23:55:40 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-02-05 23:55:40 +0000 UTC,LastTransitionTime:2023-02-05 23:55:40 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-02-05 23:55:40 +0000 UTC,LastTransitionTime:2023-02-05 23:55:40 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:10.138.0.7,},NodeAddress{Type:ExternalIP,Address:34.118.193.103,},NodeAddress{Type:InternalDNS,Address:bootstrap-e2e-minion-group-7p0n.c.k8s-infra-e2e-boskos-001.internal,},NodeAddress{Type:Hostname,Address:bootstrap-e2e-minion-group-7p0n.c.k8s-infra-e2e-boskos-001.internal,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:a7360e0504037fe783766a874b55420f,SystemUUID:a7360e05-0403-7fe7-8376-6a874b55420f,BootID:21a9d9ec-9d9a-4d80-a6ba-4d65cdd17d89,KernelVersion:5.10.123+,OSImage:Container-Optimized OS from Google,ContainerRuntimeVersion:containerd://1.6.6,KubeletVersion:v1.27.0-alpha.1.249+cf14b50b0d2215,KubeProxyVersion:v1.27.0-alpha.1.249+cf14b50b0d2215,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[registry.k8s.io/cloud-provider-gcp/gcp-compute-persistent-disk-csi-driver@sha256:e9c355dd1ce57de91a0f9164bd2fe688938723e7f9bd40bb13652d603f41d25a registry.k8s.io/cloud-provider-gcp/gcp-compute-persistent-disk-csi-driver:v1.2.2],SizeBytes:73354282,},ContainerImage{Names:[registry.k8s.io/kube-proxy-amd64:v1.27.0-alpha.1.249_cf14b50b0d2215],SizeBytes:67583329,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:16bbf38c463a4223d8cfe4da12bc61010b082a79b4bb003e2d3ba3ece5dd5f9e registry.k8s.io/e2e-test-images/agnhost:2.43],SizeBytes:51706353,},ContainerImage{Names:[gke.gcr.io/prometheus-to-sd@sha256:e739643c3939ba0b161425f45a1989eedfc4a3b166db9a7100863296b4c70510 gke.gcr.io/prometheus-to-sd:v0.11.1-gke.1],SizeBytes:48742566,},ContainerImage{Names:[registry.k8s.io/cloud-provider-gcp/gcp-compute-persistent-disk-csi-driver@sha256:dc0301d9a43e23e389abb7690cd631f98af76ea3429259c7f5f643fd31fc0a3e registry.k8s.io/cloud-provider-gcp/gcp-compute-persistent-disk-csi-driver:v1.4.0],SizeBytes:27914152,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:ee3b525d5b89db99da3b8eb521d9cd90cb6e9ef0fbb651e98bb37be78d36b5b8 registry.k8s.io/sig-storage/csi-provisioner:v3.3.0],SizeBytes:25491225,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:425d8f1b769398127767b06ed97ce62578a3179bcb99809ce93a1649e025ffe7 registry.k8s.io/sig-storage/csi-resizer:v1.6.0],SizeBytes:24148884,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:9a685020911e2725ad019dbce6e4a5ab93d51e3d4557f115e64343345e05781b registry.k8s.io/sig-storage/csi-attacher:v4.0.0],SizeBytes:23847201,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:89e900a160a986a1a7a4eba7f5259e510398fa87ca9b8a729e7dec59e04c7709 registry.k8s.io/sig-storage/csi-snapshotter:v5.0.1],SizeBytes:22163966,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/kas-network-proxy/proxy-agent@sha256:939c42e815e6b6af3181f074652c0d18fe429fcee9b49c1392aee7e92887cfef registry.k8s.io/kas-network-proxy/proxy-agent:v0.1.1],SizeBytes:8364694,},ContainerImage{Names:[registry.k8s.io/metadata-proxy@sha256:e914645f22e946bce5165737e1b244e0a296ad1f0f81a9531adc57af2780978a registry.k8s.io/metadata-proxy:v0.1.12],SizeBytes:5301657,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:2e0f836850e09b8b7cc937681d6194537a09fbd5f6b9e08f4d646a85128e8937 registry.k8s.io/e2e-test-images/busybox:1.29-4],SizeBytes:731990,},ContainerImage{Names:[registry.k8s.io/pause@sha256:7031c1b283388d2c2e09b57badb803c05ebed362dc88d84b480cc47f72a21097 registry.k8s.io/pause:3.9],SizeBytes:321520,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},}
Feb  5 23:56:46.033: INFO: 
Logging kubelet events for node bootstrap-e2e-minion-group-7p0n
Feb  5 23:56:46.076: INFO: 
Logging pods the kubelet thinks is on node bootstrap-e2e-minion-group-7p0n
Feb  5 23:56:46.139: INFO: kube-proxy-bootstrap-e2e-minion-group-7p0n started at 2023-02-05 23:49:36 +0000 UTC (0+1 container statuses recorded)
Feb  5 23:56:46.139: INFO: 	Container kube-proxy ready: true, restart count 0
Feb  5 23:56:46.139: INFO: metadata-proxy-v0.1-2vz8w started at 2023-02-05 23:49:37 +0000 UTC (0+2 container statuses recorded)
Feb  5 23:56:46.139: INFO: 	Container metadata-proxy ready: true, restart count 0
Feb  5 23:56:46.139: INFO: 	Container prometheus-to-sd-exporter ready: true, restart count 0
Feb  5 23:56:46.139: INFO: konnectivity-agent-fljgx started at 2023-02-05 23:49:45 +0000 UTC (0+1 container statuses recorded)
Feb  5 23:56:46.139: INFO: 	Container konnectivity-agent ready: true, restart count 0
Feb  5 23:56:46.308: INFO: 
Latency metrics for node bootstrap-e2e-minion-group-7p0n
Feb  5 23:56:46.308: INFO: 
Logging node info for node bootstrap-e2e-minion-group-f2xv
Feb  5 23:56:46.399: INFO: Node Info: &Node{ObjectMeta:{bootstrap-e2e-minion-group-f2xv    e03f6c2e-c82f-4648-923b-3ca1f93c77cb 61865 0 2023-02-05 20:03:50 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:n1-standard-2 beta.kubernetes.io/os:linux cloud.google.com/metadata-proxy-ready:true failure-domain.beta.kubernetes.io/region:us-west1 failure-domain.beta.kubernetes.io/zone:us-west1-b kubernetes.io/arch:amd64 kubernetes.io/hostname:bootstrap-e2e-minion-group-f2xv kubernetes.io/os:linux node.kubernetes.io/instance-type:n1-standard-2 topology.gke.io/zone:us-west1-b topology.hostpath.csi/node:bootstrap-e2e-minion-group-f2xv topology.kubernetes.io/region:us-west1 topology.kubernetes.io/zone:us-west1-b] map[node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kubelet Update v1 2023-02-05 20:03:50 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:cloud.google.com/metadata-proxy-ready":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {kube-controller-manager Update v1 2023-02-05 23:19:26 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}}}} status} {kube-controller-manager Update v1 2023-02-05 23:44:07 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"10.64.0.0/24\"":{}}}} } {node-problem-detector Update v1 2023-02-05 23:53:48 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"CorruptDockerOverlay2\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"FrequentContainerdRestart\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"FrequentDockerRestart\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"FrequentKubeletRestart\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"FrequentUnregisterNetDevice\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"KernelDeadlock\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"ReadonlyFilesystem\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kubelet Update v1 2023-02-05 23:54:19 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:topology.gke.io/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:nodeInfo":{"f:bootID":{}}}} status}]},Spec:NodeSpec{PodCIDR:10.64.0.0/24,DoNotUseExternalID:,ProviderID:gce://k8s-infra-e2e-boskos-001/us-west1-b/bootstrap-e2e-minion-group-f2xv,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[10.64.0.0/24],},Status:NodeStatus{Capacity:ResourceList{attachable-volumes-gce-pd: {{127 0} {<nil>} 127 DecimalSI},cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{101203873792 0} {<nil>} 98831908Ki BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{7815426048 0} {<nil>} 7632252Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{attachable-volumes-gce-pd: {{127 0} {<nil>} 127 DecimalSI},cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{91083486262 0} {<nil>} 91083486262 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{7553282048 0} {<nil>} 7376252Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:CorruptDockerOverlay2,Status:False,LastHeartbeatTime:2023-02-05 23:53:48 +0000 UTC,LastTransitionTime:2023-02-05 22:58:40 +0000 UTC,Reason:NoCorruptDockerOverlay2,Message:docker overlay2 is functioning properly,},NodeCondition{Type:FrequentUnregisterNetDevice,Status:False,LastHeartbeatTime:2023-02-05 23:53:48 +0000 UTC,LastTransitionTime:2023-02-05 22:58:40 +0000 UTC,Reason:NoFrequentUnregisterNetDevice,Message:node is functioning properly,},NodeCondition{Type:FrequentKubeletRestart,Status:False,LastHeartbeatTime:2023-02-05 23:53:48 +0000 UTC,LastTransitionTime:2023-02-05 22:58:40 +0000 UTC,Reason:NoFrequentKubeletRestart,Message:kubelet is functioning properly,},NodeCondition{Type:FrequentDockerRestart,Status:False,LastHeartbeatTime:2023-02-05 23:53:48 +0000 UTC,LastTransitionTime:2023-02-05 22:58:40 +0000 UTC,Reason:NoFrequentDockerRestart,Message:docker is functioning properly,},NodeCondition{Type:FrequentContainerdRestart,Status:False,LastHeartbeatTime:2023-02-05 23:53:48 +0000 UTC,LastTransitionTime:2023-02-05 22:58:40 +0000 UTC,Reason:NoFrequentContainerdRestart,Message:containerd is functioning properly,},NodeCondition{Type:KernelDeadlock,Status:False,LastHeartbeatTime:2023-02-05 23:53:48 +0000 UTC,LastTransitionTime:2023-02-05 22:58:40 +0000 UTC,Reason:KernelHasNoDeadlock,Message:kernel has no deadlock,},NodeCondition{Type:ReadonlyFilesystem,Status:False,LastHeartbeatTime:2023-02-05 23:53:48 +0000 UTC,LastTransitionTime:2023-02-05 22:58:40 +0000 UTC,Reason:FilesystemIsNotReadOnly,Message:Filesystem is not read-only,},NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2023-02-05 20:03:58 +0000 UTC,LastTransitionTime:2023-02-05 20:03:58 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-02-05 23:54:19 +0000 UTC,LastTransitionTime:2023-02-05 23:19:30 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-02-05 23:54:19 +0000 UTC,LastTransitionTime:2023-02-05 23:19:30 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-02-05 23:54:19 +0000 UTC,LastTransitionTime:2023-02-05 23:19:30 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-02-05 23:54:19 +0000 UTC,LastTransitionTime:2023-02-05 23:44:07 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:10.138.0.5,},NodeAddress{Type:ExternalIP,Address:34.83.102.119,},NodeAddress{Type:InternalDNS,Address:bootstrap-e2e-minion-group-f2xv.c.k8s-infra-e2e-boskos-001.internal,},NodeAddress{Type:Hostname,Address:bootstrap-e2e-minion-group-f2xv.c.k8s-infra-e2e-boskos-001.internal,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:376621d36b55b947abdcfdac7e13ae34,SystemUUID:376621d3-6b55-b947-abdc-fdac7e13ae34,BootID:5bccce71-ce5b-47fa-9ed2-b6d61be88210,KernelVersion:5.10.123+,OSImage:Container-Optimized OS from Google,ContainerRuntimeVersion:containerd://1.6.6,KubeletVersion:v1.27.0-alpha.1.249+cf14b50b0d2215,KubeProxyVersion:v1.27.0-alpha.1.249+cf14b50b0d2215,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[registry.k8s.io/cloud-provider-gcp/gcp-compute-persistent-disk-csi-driver@sha256:e9c355dd1ce57de91a0f9164bd2fe688938723e7f9bd40bb13652d603f41d25a registry.k8s.io/cloud-provider-gcp/gcp-compute-persistent-disk-csi-driver:v1.2.2],SizeBytes:73354282,},ContainerImage{Names:[registry.k8s.io/kube-proxy-amd64:v1.27.0-alpha.1.249_cf14b50b0d2215],SizeBytes:67583329,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:16bbf38c463a4223d8cfe4da12bc61010b082a79b4bb003e2d3ba3ece5dd5f9e registry.k8s.io/e2e-test-images/agnhost:2.43],SizeBytes:51706353,},ContainerImage{Names:[gke.gcr.io/prometheus-to-sd@sha256:e739643c3939ba0b161425f45a1989eedfc4a3b166db9a7100863296b4c70510 gke.gcr.io/prometheus-to-sd:v0.11.1-gke.1],SizeBytes:48742566,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:148b022f5c5da426fc2f3c14b5c0867e58ef05961510c84749ac1fddcb0fef22 registry.k8s.io/e2e-test-images/httpd:2.4.38-4],SizeBytes:40764257,},ContainerImage{Names:[registry.k8s.io/metrics-server/metrics-server@sha256:6385aec64bb97040a5e692947107b81e178555c7a5b71caa90d733e4130efc10 registry.k8s.io/metrics-server/metrics-server:v0.5.2],SizeBytes:26023008,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:ee3b525d5b89db99da3b8eb521d9cd90cb6e9ef0fbb651e98bb37be78d36b5b8 registry.k8s.io/sig-storage/csi-provisioner:v3.3.0],SizeBytes:25491225,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:425d8f1b769398127767b06ed97ce62578a3179bcb99809ce93a1649e025ffe7 registry.k8s.io/sig-storage/csi-resizer:v1.6.0],SizeBytes:24148884,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:291334908ddf71a4661fd7f6d9d97274de8a5378a2b6fdfeb2ce73414a34f82f registry.k8s.io/sig-storage/csi-snapshotter:v6.1.0],SizeBytes:23881995,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:9a685020911e2725ad019dbce6e4a5ab93d51e3d4557f115e64343345e05781b registry.k8s.io/sig-storage/csi-attacher:v4.0.0],SizeBytes:23847201,},ContainerImage{Names:[registry.k8s.io/sig-storage/snapshot-controller@sha256:823c75d0c45d1427f6d850070956d9ca657140a7bbf828381541d1d808475280 registry.k8s.io/sig-storage/snapshot-controller:v6.1.0],SizeBytes:22620891,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:92257881c1d6493cf18299a24af42330f891166560047902b8d431fb66b01af5 registry.k8s.io/sig-storage/hostpathplugin:v1.9.0],SizeBytes:18758628,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:017727efcfeb7d053af68e51436ce8e65edbc6ca573720afb4f79c8594036955 registry.k8s.io/coredns/coredns:v1.10.0],SizeBytes:15273057,},ContainerImage{Names:[registry.k8s.io/cpa/cluster-proportional-autoscaler@sha256:fd636b33485c7826fb20ef0688a83ee0910317dbb6c0c6f3ad14661c1db25def registry.k8s.io/cpa/cluster-proportional-autoscaler:1.8.4],SizeBytes:15209393,},ContainerImage{Names:[registry.k8s.io/autoscaling/addon-resizer@sha256:43f129b81d28f0fdd54de6d8e7eacd5728030782e03db16087fc241ad747d3d6 registry.k8s.io/autoscaling/addon-resizer:1.8.14],SizeBytes:10153852,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:933940f13b3ea0abc62e656c1aa5c5b47c04b15d71250413a6b821bd0c58b94e registry.k8s.io/sig-storage/livenessprobe:v2.7.0],SizeBytes:8688564,},ContainerImage{Names:[registry.k8s.io/kas-network-proxy/proxy-agent@sha256:939c42e815e6b6af3181f074652c0d18fe429fcee9b49c1392aee7e92887cfef registry.k8s.io/kas-network-proxy/proxy-agent:v0.1.1],SizeBytes:8364694,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:5c99cf6a02adda929b10321dbf4ecfa00d87be9ba4fb456006237d530ab4baa1 registry.k8s.io/e2e-test-images/nginx:1.14-4],SizeBytes:6978614,},ContainerImage{Names:[registry.k8s.io/networking/ingress-gce-404-server-with-metrics-amd64@sha256:7eb7b3cee4d33c10c49893ad3c386232b86d4067de5251294d4c620d6e072b93 registry.k8s.io/networking/ingress-gce-404-server-with-metrics-amd64:v1.10.11],SizeBytes:6463068,},ContainerImage{Names:[registry.k8s.io/metadata-proxy@sha256:e914645f22e946bce5165737e1b244e0a296ad1f0f81a9531adc57af2780978a registry.k8s.io/metadata-proxy:v0.1.12],SizeBytes:5301657,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:2e0f836850e09b8b7cc937681d6194537a09fbd5f6b9e08f4d646a85128e8937 registry.k8s.io/e2e-test-images/busybox:1.29-4],SizeBytes:731990,},ContainerImage{Names:[registry.k8s.io/pause@sha256:7031c1b283388d2c2e09b57badb803c05ebed362dc88d84b480cc47f72a21097 registry.k8s.io/pause:3.9],SizeBytes:321520,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},}
Feb  5 23:56:46.399: INFO: 
Logging kubelet events for node bootstrap-e2e-minion-group-f2xv
Feb  5 23:56:46.443: INFO: 
Logging pods the kubelet thinks is on node bootstrap-e2e-minion-group-f2xv
Feb  5 23:56:46.511: INFO: kube-proxy-bootstrap-e2e-minion-group-f2xv started at 2023-02-05 20:03:51 +0000 UTC (0+1 container statuses recorded)
Feb  5 23:56:46.511: INFO: 	Container kube-proxy ready: true, restart count 1
Feb  5 23:56:46.511: INFO: l7-default-backend-8549d69d99-ptkh8 started at 2023-02-05 20:03:58 +0000 UTC (0+1 container statuses recorded)
Feb  5 23:56:46.511: INFO: 	Container default-http-backend ready: true, restart count 1
Feb  5 23:56:46.511: INFO: volume-snapshot-controller-0 started at 2023-02-05 20:03:58 +0000 UTC (0+1 container statuses recorded)
Feb  5 23:56:46.511: INFO: 	Container volume-snapshot-controller ready: true, restart count 1
Feb  5 23:56:46.511: INFO: kube-dns-autoscaler-5f6455f985-ddvmr started at 2023-02-05 20:03:58 +0000 UTC (0+1 container statuses recorded)
Feb  5 23:56:46.511: INFO: 	Container autoscaler ready: true, restart count 1
Feb  5 23:56:46.511: INFO: coredns-6846b5b5f-r8m7l started at 2023-02-05 23:48:50 +0000 UTC (0+1 container statuses recorded)
Feb  5 23:56:46.511: INFO: 	Container coredns ready: true, restart count 0
Feb  5 23:56:46.511: INFO: metadata-proxy-v0.1-46fnl started at 2023-02-05 20:03:51 +0000 UTC (0+2 container statuses recorded)
Feb  5 23:56:46.511: INFO: 	Container metadata-proxy ready: true, restart count 1
Feb  5 23:56:46.511: INFO: 	Container prometheus-to-sd-exporter ready: true, restart count 1
Feb  5 23:56:46.511: INFO: konnectivity-agent-tp4wv started at 2023-02-05 20:03:59 +0000 UTC (0+1 container statuses recorded)
Feb  5 23:56:46.511: INFO: 	Container konnectivity-agent ready: true, restart count 1
Feb  5 23:56:46.511: INFO: metrics-server-v0.5.2-867b8754b9-ss6lj started at 2023-02-05 20:09:11 +0000 UTC (0+2 container statuses recorded)
Feb  5 23:56:46.511: INFO: 	Container metrics-server ready: true, restart count 2
Feb  5 23:56:46.511: INFO: 	Container metrics-server-nanny ready: true, restart count 1
Feb  5 23:56:46.695: INFO: 
Latency metrics for node bootstrap-e2e-minion-group-f2xv
Feb  5 23:56:46.695: INFO: 
Logging node info for node bootstrap-e2e-minion-group-vc67
Feb  5 23:56:46.745: INFO: Node Info: &Node{ObjectMeta:{bootstrap-e2e-minion-group-vc67    d427fdfc-daa1-44d1-9847-d6ca5f6726b7 61838 0 2023-02-05 20:03:54 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:n1-standard-2 beta.kubernetes.io/os:linux cloud.google.com/metadata-proxy-ready:true failure-domain.beta.kubernetes.io/region:us-west1 failure-domain.beta.kubernetes.io/zone:us-west1-b kubernetes.io/arch:amd64 kubernetes.io/hostname:bootstrap-e2e-minion-group-vc67 kubernetes.io/os:linux node.kubernetes.io/instance-type:n1-standard-2 topology.gke.io/zone:us-west1-b topology.hostpath.csi/node:bootstrap-e2e-minion-group-vc67 topology.kubernetes.io/region:us-west1 topology.kubernetes.io/zone:us-west1-b] map[node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kubelet Update v1 2023-02-05 20:03:54 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:cloud.google.com/metadata-proxy-ready":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {e2e.test Update v1 2023-02-05 23:01:21 +0000 UTC FieldsV1 {"f:status":{"f:allocatable":{"f:example.com/fakePTSRes":{},"f:example.com/fakecpu":{}},"f:capacity":{"f:example.com/fakePTSRes":{},"f:example.com/fakecpu":{}}}} status} {node-problem-detector Update v1 2023-02-05 23:53:48 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"CorruptDockerOverlay2\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"FrequentContainerdRestart\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"FrequentDockerRestart\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"FrequentKubeletRestart\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"FrequentUnregisterNetDevice\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"KernelDeadlock\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"ReadonlyFilesystem\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kube-controller-manager Update v1 2023-02-05 23:53:52 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}}}} status} {kube-controller-manager Update v1 2023-02-05 23:53:55 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"10.64.1.0/24\"":{}}}} } {kubelet Update v1 2023-02-05 23:54:06 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:topology.gke.io/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{},"f:memory":{}},"f:capacity":{"f:memory":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:nodeInfo":{"f:bootID":{}}}} status}]},Spec:NodeSpec{PodCIDR:10.64.1.0/24,DoNotUseExternalID:,ProviderID:gce://k8s-infra-e2e-boskos-001/us-west1-b/bootstrap-e2e-minion-group-vc67,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[10.64.1.0/24],},Status:NodeStatus{Capacity:ResourceList{attachable-volumes-gce-pd: {{127 0} {<nil>} 127 DecimalSI},cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{101203873792 0} {<nil>} 98831908Ki BinarySI},example.com/fakePTSRes: {{10 0} {<nil>} 10 DecimalSI},example.com/fakecpu: {{1 3} {<nil>} 1k DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{7815434240 0} {<nil>} 7632260Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{attachable-volumes-gce-pd: {{127 0} {<nil>} 127 DecimalSI},cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{91083486262 0} {<nil>} 91083486262 DecimalSI},example.com/fakePTSRes: {{10 0} {<nil>} 10 DecimalSI},example.com/fakecpu: {{1 3} {<nil>} 1k DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{7553290240 0} {<nil>} 7376260Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:CorruptDockerOverlay2,Status:False,LastHeartbeatTime:2023-02-05 23:53:48 +0000 UTC,LastTransitionTime:2023-02-05 22:58:39 +0000 UTC,Reason:NoCorruptDockerOverlay2,Message:docker overlay2 is functioning properly,},NodeCondition{Type:FrequentUnregisterNetDevice,Status:False,LastHeartbeatTime:2023-02-05 23:53:48 +0000 UTC,LastTransitionTime:2023-02-05 22:58:39 +0000 UTC,Reason:NoFrequentUnregisterNetDevice,Message:node is functioning properly,},NodeCondition{Type:FrequentKubeletRestart,Status:False,LastHeartbeatTime:2023-02-05 23:53:48 +0000 UTC,LastTransitionTime:2023-02-05 22:58:39 +0000 UTC,Reason:NoFrequentKubeletRestart,Message:kubelet is functioning properly,},NodeCondition{Type:FrequentDockerRestart,Status:False,LastHeartbeatTime:2023-02-05 23:53:48 +0000 UTC,LastTransitionTime:2023-02-05 22:58:39 +0000 UTC,Reason:NoFrequentDockerRestart,Message:docker is functioning properly,},NodeCondition{Type:FrequentContainerdRestart,Status:False,LastHeartbeatTime:2023-02-05 23:53:48 +0000 UTC,LastTransitionTime:2023-02-05 22:58:39 +0000 UTC,Reason:NoFrequentContainerdRestart,Message:containerd is functioning properly,},NodeCondition{Type:KernelDeadlock,Status:False,LastHeartbeatTime:2023-02-05 23:53:48 +0000 UTC,LastTransitionTime:2023-02-05 22:58:39 +0000 UTC,Reason:KernelHasNoDeadlock,Message:kernel has no deadlock,},NodeCondition{Type:ReadonlyFilesystem,Status:False,LastHeartbeatTime:2023-02-05 23:53:48 +0000 UTC,LastTransitionTime:2023-02-05 22:58:39 +0000 UTC,Reason:FilesystemIsNotReadOnly,Message:Filesystem is not read-only,},NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2023-02-05 20:04:10 +0000 UTC,LastTransitionTime:2023-02-05 20:04:10 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-02-05 23:54:06 +0000 UTC,LastTransitionTime:2023-02-05 23:53:55 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-02-05 23:54:06 +0000 UTC,LastTransitionTime:2023-02-05 23:53:55 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-02-05 23:54:06 +0000 UTC,LastTransitionTime:2023-02-05 23:53:55 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-02-05 23:54:06 +0000 UTC,LastTransitionTime:2023-02-05 23:53:55 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:10.138.0.3,},NodeAddress{Type:ExternalIP,Address:34.145.100.19,},NodeAddress{Type:InternalDNS,Address:bootstrap-e2e-minion-group-vc67.c.k8s-infra-e2e-boskos-001.internal,},NodeAddress{Type:Hostname,Address:bootstrap-e2e-minion-group-vc67.c.k8s-infra-e2e-boskos-001.internal,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:a65546ee19f4e3edcf02f690a7396782,SystemUUID:a65546ee-19f4-e3ed-cf02-f690a7396782,BootID:c872f346-7124-458f-a158-1ef112f88295,KernelVersion:5.10.123+,OSImage:Container-Optimized OS from Google,ContainerRuntimeVersion:containerd://1.6.6,KubeletVersion:v1.27.0-alpha.1.249+cf14b50b0d2215,KubeProxyVersion:v1.27.0-alpha.1.249+cf14b50b0d2215,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[registry.k8s.io/e2e-test-images/jessie-dnsutils@sha256:24aaf2626d6b27864c29de2097e8bbb840b3a414271bf7c8995e431e47d8408e registry.k8s.io/e2e-test-images/jessie-dnsutils:1.7],SizeBytes:112030336,},ContainerImage{Names:[registry.k8s.io/sig-storage/nfs-provisioner@sha256:e943bb77c7df05ebdc8c7888b2db289b13bf9f012d6a3a5a74f14d4d5743d439 registry.k8s.io/sig-storage/nfs-provisioner:v3.0.1],SizeBytes:90632047,},ContainerImage{Names:[registry.k8s.io/cloud-provider-gcp/gcp-compute-persistent-disk-csi-driver@sha256:e9c355dd1ce57de91a0f9164bd2fe688938723e7f9bd40bb13652d603f41d25a registry.k8s.io/cloud-provider-gcp/gcp-compute-persistent-disk-csi-driver:v1.2.2],SizeBytes:73354282,},ContainerImage{Names:[registry.k8s.io/kube-proxy-amd64:v1.27.0-alpha.1.249_cf14b50b0d2215],SizeBytes:67583329,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:16bbf38c463a4223d8cfe4da12bc61010b082a79b4bb003e2d3ba3ece5dd5f9e registry.k8s.io/e2e-test-images/agnhost:2.43],SizeBytes:51706353,},ContainerImage{Names:[gke.gcr.io/prometheus-to-sd@sha256:e739643c3939ba0b161425f45a1989eedfc4a3b166db9a7100863296b4c70510 gke.gcr.io/prometheus-to-sd:v0.11.1-gke.1],SizeBytes:48742566,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:148b022f5c5da426fc2f3c14b5c0867e58ef05961510c84749ac1fddcb0fef22 registry.k8s.io/e2e-test-images/httpd:2.4.38-4],SizeBytes:40764257,},ContainerImage{Names:[registry.k8s.io/sig-storage/volume-data-source-validator@sha256:d35884236461ad14397194cb4abc13aefb28a54bf3f6698d673c2ba353e78e18 registry.k8s.io/sig-storage/volume-data-source-validator:v1.0.0],SizeBytes:28662933,},ContainerImage{Names:[registry.k8s.io/cloud-provider-gcp/gcp-compute-persistent-disk-csi-driver@sha256:dc0301d9a43e23e389abb7690cd631f98af76ea3429259c7f5f643fd31fc0a3e registry.k8s.io/cloud-provider-gcp/gcp-compute-persistent-disk-csi-driver:v1.4.0],SizeBytes:27914152,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:ee3b525d5b89db99da3b8eb521d9cd90cb6e9ef0fbb651e98bb37be78d36b5b8 registry.k8s.io/sig-storage/csi-provisioner:v3.3.0],SizeBytes:25491225,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:425d8f1b769398127767b06ed97ce62578a3179bcb99809ce93a1649e025ffe7 registry.k8s.io/sig-storage/csi-resizer:v1.6.0],SizeBytes:24148884,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:291334908ddf71a4661fd7f6d9d97274de8a5378a2b6fdfeb2ce73414a34f82f registry.k8s.io/sig-storage/csi-snapshotter:v6.1.0],SizeBytes:23881995,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:9a685020911e2725ad019dbce6e4a5ab93d51e3d4557f115e64343345e05781b registry.k8s.io/sig-storage/csi-attacher:v4.0.0],SizeBytes:23847201,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:89e900a160a986a1a7a4eba7f5259e510398fa87ca9b8a729e7dec59e04c7709 registry.k8s.io/sig-storage/csi-snapshotter:v5.0.1],SizeBytes:22163966,},ContainerImage{Names:[registry.k8s.io/sig-storage/hello-populator@sha256:42d275bc2569b00a11c8058e5b77c50cdee39bf75d80c912b80813f03e7fd7d1 registry.k8s.io/sig-storage/hello-populator:v1.0.1],SizeBytes:20511089,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:92257881c1d6493cf18299a24af42330f891166560047902b8d431fb66b01af5 registry.k8s.io/sig-storage/hostpathplugin:v1.9.0],SizeBytes:18758628,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:017727efcfeb7d053af68e51436ce8e65edbc6ca573720afb4f79c8594036955 registry.k8s.io/coredns/coredns:v1.10.0],SizeBytes:15273057,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:933940f13b3ea0abc62e656c1aa5c5b47c04b15d71250413a6b821bd0c58b94e registry.k8s.io/sig-storage/livenessprobe:v2.7.0],SizeBytes:8688564,},ContainerImage{Names:[registry.k8s.io/kas-network-proxy/proxy-agent@sha256:939c42e815e6b6af3181f074652c0d18fe429fcee9b49c1392aee7e92887cfef registry.k8s.io/kas-network-proxy/proxy-agent:v0.1.1],SizeBytes:8364694,},ContainerImage{Names:[registry.k8s.io/metadata-proxy@sha256:e914645f22e946bce5165737e1b244e0a296ad1f0f81a9531adc57af2780978a registry.k8s.io/metadata-proxy:v0.1.12],SizeBytes:5301657,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:2e0f836850e09b8b7cc937681d6194537a09fbd5f6b9e08f4d646a85128e8937 registry.k8s.io/e2e-test-images/busybox:1.29-4],SizeBytes:731990,},ContainerImage{Names:[registry.k8s.io/pause@sha256:7031c1b283388d2c2e09b57badb803c05ebed362dc88d84b480cc47f72a21097 registry.k8s.io/pause:3.9],SizeBytes:321520,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},}
Feb  5 23:56:46.745: INFO: 
Logging kubelet events for node bootstrap-e2e-minion-group-vc67
Feb  5 23:56:46.791: INFO: 
Logging pods the kubelet thinks is on node bootstrap-e2e-minion-group-vc67
Feb  5 23:56:46.856: INFO: kube-proxy-bootstrap-e2e-minion-group-vc67 started at 2023-02-05 20:03:54 +0000 UTC (0+1 container statuses recorded)
Feb  5 23:56:46.856: INFO: 	Container kube-proxy ready: true, restart count 1
Feb  5 23:56:46.856: INFO: metadata-proxy-v0.1-kjcsh started at 2023-02-05 20:03:55 +0000 UTC (0+2 container statuses recorded)
Feb  5 23:56:46.856: INFO: 	Container metadata-proxy ready: true, restart count 1
Feb  5 23:56:46.856: INFO: 	Container prometheus-to-sd-exporter ready: true, restart count 1
Feb  5 23:56:46.856: INFO: konnectivity-agent-tmn2w started at 2023-02-05 20:04:10 +0000 UTC (0+1 container statuses recorded)
Feb  5 23:56:46.856: INFO: 	Container konnectivity-agent ready: true, restart count 1
Feb  5 23:56:46.856: INFO: coredns-6846b5b5f-94td7 started at 2023-02-05 23:44:58 +0000 UTC (0+1 container statuses recorded)
Feb  5 23:56:46.856: INFO: 	Container coredns ready: true, restart count 0
Feb  5 23:56:47.024: INFO: 
Latency metrics for node bootstrap-e2e-minion-group-vc67
END STEP: dump namespace information after failure - test/e2e/framework/framework.go:284 @ 02/05/23 23:56:47.024 (1.524s)
< Exit [DeferCleanup (Each)] [Testpattern: Pre-provisioned PV (default fs)] subPath - dump namespaces | framework.go:206 @ 02/05/23 23:56:47.024 (1.525s)
> Enter [DeferCleanup (Each)] [Testpattern: Pre-provisioned PV (default fs)] subPath - tear down framework | framework.go:203 @ 02/05/23 23:56:47.024
STEP: Destroying namespace "provisioning-9333" for this suite. - test/e2e/framework/framework.go:347 @ 02/05/23 23:56:47.024
< Exit [DeferCleanup (Each)] [Testpattern: Pre-provisioned PV (default fs)] subPath - tear down framework | framework.go:203 @ 02/05/23 23:56:47.067 (43ms)
> Enter [ReportAfterEach] TOP-LEVEL - test/e2e/e2e_test.go:144 @ 02/05/23 23:56:47.067
< Exit [ReportAfterEach] TOP-LEVEL - test/e2e/e2e_test.go:144 @ 02/05/23 23:56:47.067 (0s)

				Click to see stdout/stderrfrom ginkgo_report.xml

Filter through log files | View test history on testgrid

Kubernetes e2e suite [It] [sig-storage] In-tree Volumes [Driver: local][LocalVolumeType: dir] [Testpattern: Pre-provisioned PV (default fs)] subPath should unmount if pod is gracefully deleted while kubelet is down [Disruptive][Slow][LinuxOnly] 2m15s

go run hack/e2e.go -v --test --test_args='--ginkgo.focus=Kubernetes\se2e\ssuite\s\[It\]\s\[sig\-storage\]\sIn\-tree\sVolumes\s\[Driver\:\slocal\]\[LocalVolumeType\:\sdir\]\s\[Testpattern\:\sPre\-provisioned\sPV\s\(default\sfs\)\]\ssubPath\sshould\sunmount\sif\spod\sis\sgracefully\sdeleted\swhile\skubelet\sis\sdown\s\[Disruptive\]\[Slow\]\[LinuxOnly\]$'

[FAILED] SSH to Node "bootstrap-e2e-minion-group-7p0n" errored.: error getting SSH client to prow@34.118.193.103:22: timed out waiting for the condition
In [DeferCleanup (Each)] at: test/e2e/storage/utils/pod.go:117 @ 02/05/23 23:56:44.852

				> Enter [BeforeEach] [Testpattern: Pre-provisioned PV (default fs)] subPath - test/e2e/storage/framework/testsuite.go:51 @ 02/05/23 23:54:31.361
< Exit [BeforeEach] [Testpattern: Pre-provisioned PV (default fs)] subPath - test/e2e/storage/framework/testsuite.go:51 @ 02/05/23 23:54:31.361 (0s)
> Enter [BeforeEach] [Testpattern: Pre-provisioned PV (default fs)] subPath - set up framework | framework.go:188 @ 02/05/23 23:54:31.361
STEP: Creating a kubernetes client - test/e2e/framework/framework.go:208 @ 02/05/23 23:54:31.361
Feb  5 23:54:31.361: INFO: >>> kubeConfig: /workspace/.kube/config
STEP: Building a namespace api object, basename provisioning - test/e2e/framework/framework.go:247 @ 02/05/23 23:54:31.362
STEP: Waiting for a default service account to be provisioned in namespace - test/e2e/framework/framework.go:256 @ 02/05/23 23:54:31.487
STEP: Waiting for kube-root-ca.crt to be provisioned in namespace - test/e2e/framework/framework.go:259 @ 02/05/23 23:54:31.568
< Exit [BeforeEach] [Testpattern: Pre-provisioned PV (default fs)] subPath - set up framework | framework.go:188 @ 02/05/23 23:54:31.648 (287ms)
> Enter [BeforeEach] [Testpattern: Pre-provisioned PV (default fs)] subPath - test/e2e/framework/metrics/init/init.go:33 @ 02/05/23 23:54:31.648
< Exit [BeforeEach] [Testpattern: Pre-provisioned PV (default fs)] subPath - test/e2e/framework/metrics/init/init.go:33 @ 02/05/23 23:54:31.648 (0s)
> Enter [It] should unmount if pod is gracefully deleted while kubelet is down [Disruptive][Slow][LinuxOnly] - test/e2e/storage/testsuites/subpath.go:341 @ 02/05/23 23:54:31.648
Feb  5 23:54:31.693: INFO: In-tree plugin kubernetes.io/local-volume is not migrated, not validating any metrics
Feb  5 23:54:33.823: INFO: ExecWithOptions {Command:[nsenter --mount=/rootfs/proc/1/ns/mnt -- sh -c mkdir -p /tmp/local-driver-e8e50194-9ead-40de-af7c-8bcb8ceb62ca] Namespace:provisioning-9333 PodName:hostexec-bootstrap-e2e-minion-group-7p0n-6lr4r ContainerName:agnhost-container Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:true Quiet:false}
Feb  5 23:54:33.823: INFO: >>> kubeConfig: /workspace/.kube/config
Feb  5 23:54:33.824: INFO: ExecWithOptions: Clientset creation
Feb  5 23:54:33.824: INFO: ExecWithOptions: execute(POST https://35.247.68.66/api/v1/namespaces/provisioning-9333/pods/hostexec-bootstrap-e2e-minion-group-7p0n-6lr4r/exec?command=nsenter&command=--mount%3D%2Frootfs%2Fproc%2F1%2Fns%2Fmnt&command=--&command=sh&command=-c&command=mkdir+-p+%2Ftmp%2Flocal-driver-e8e50194-9ead-40de-af7c-8bcb8ceb62ca&container=agnhost-container&container=agnhost-container&stderr=true&stdout=true)
Feb  5 23:54:34.154: INFO: Creating resource for pre-provisioned PV
Feb  5 23:54:34.154: INFO: Creating PVC and PV
STEP: Creating a PVC followed by a PV - test/e2e/framework/pv/pv.go:360 @ 02/05/23 23:54:34.154
Feb  5 23:54:34.239: INFO: Waiting for PV local-szhdn to bind to PVC pvc-6wbt5
Feb  5 23:54:34.239: INFO: Waiting up to timeout=3m0s for PersistentVolumeClaims [pvc-6wbt5] to have phase Bound
Feb  5 23:54:34.280: INFO: PersistentVolumeClaim pvc-6wbt5 found but phase is Pending instead of Bound.
Feb  5 23:54:36.322: INFO: PersistentVolumeClaim pvc-6wbt5 found but phase is Pending instead of Bound.
Feb  5 23:54:38.363: INFO: PersistentVolumeClaim pvc-6wbt5 found but phase is Pending instead of Bound.
Feb  5 23:54:40.406: INFO: PersistentVolumeClaim pvc-6wbt5 found but phase is Pending instead of Bound.
Feb  5 23:54:42.448: INFO: PersistentVolumeClaim pvc-6wbt5 found but phase is Pending instead of Bound.
Feb  5 23:54:44.491: INFO: PersistentVolumeClaim pvc-6wbt5 found but phase is Pending instead of Bound.
Feb  5 23:54:46.534: INFO: PersistentVolumeClaim pvc-6wbt5 found and phase=Bound (12.294546325s)
Feb  5 23:54:46.534: INFO: Waiting up to 3m0s for PersistentVolume local-szhdn to have phase Bound
Feb  5 23:54:46.575: INFO: PersistentVolume local-szhdn found and phase=Bound (41.144816ms)
Feb  5 23:54:48.882: INFO: ExecWithOptions {Command:[nsenter --mount=/rootfs/proc/1/ns/mnt -- sh -c find /var/lib/kubelet/plugins -type d -exec mountpoint {} \; | grep 'is a mountpoint$' || true] Namespace:provisioning-9333 PodName:hostexec-bootstrap-e2e-minion-group-7p0n-7jcfh ContainerName:agnhost-container Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:true Quiet:false}
Feb  5 23:54:48.882: INFO: >>> kubeConfig: /workspace/.kube/config
Feb  5 23:54:48.883: INFO: ExecWithOptions: Clientset creation
Feb  5 23:54:48.884: INFO: ExecWithOptions: execute(POST https://35.247.68.66/api/v1/namespaces/provisioning-9333/pods/hostexec-bootstrap-e2e-minion-group-7p0n-7jcfh/exec?command=nsenter&command=--mount%3D%2Frootfs%2Fproc%2F1%2Fns%2Fmnt&command=--&command=sh&command=-c&command=find+%2Fvar%2Flib%2Fkubelet%2Fplugins+-type+d+-exec+mountpoint+%7B%7D+%5C%3B+%7C+grep+%27is+a+mountpoint%24%27+%7C%7C+true&container=agnhost-container&container=agnhost-container&stderr=true&stdout=true)
Feb  5 23:54:51.359: INFO: ExecWithOptions {Command:[nsenter --mount=/rootfs/proc/1/ns/mnt -- sh -c find /var/lib/kubelet/plugins -type d -exec mountpoint {} \; | grep 'is a mountpoint$' || true] Namespace:provisioning-9333 PodName:hostexec-bootstrap-e2e-minion-group-f2xv-jqtng ContainerName:agnhost-container Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:true Quiet:false}
Feb  5 23:54:51.359: INFO: >>> kubeConfig: /workspace/.kube/config
Feb  5 23:54:51.360: INFO: ExecWithOptions: Clientset creation
Feb  5 23:54:51.360: INFO: ExecWithOptions: execute(POST https://35.247.68.66/api/v1/namespaces/provisioning-9333/pods/hostexec-bootstrap-e2e-minion-group-f2xv-jqtng/exec?command=nsenter&command=--mount%3D%2Frootfs%2Fproc%2F1%2Fns%2Fmnt&command=--&command=sh&command=-c&command=find+%2Fvar%2Flib%2Fkubelet%2Fplugins+-type+d+-exec+mountpoint+%7B%7D+%5C%3B+%7C+grep+%27is+a+mountpoint%24%27+%7C%7C+true&container=agnhost-container&container=agnhost-container&stderr=true&stdout=true)
Feb  5 23:54:53.817: INFO: ExecWithOptions {Command:[nsenter --mount=/rootfs/proc/1/ns/mnt -- sh -c find /var/lib/kubelet/plugins -type d -exec mountpoint {} \; | grep 'is a mountpoint$' || true] Namespace:provisioning-9333 PodName:hostexec-bootstrap-e2e-minion-group-vc67-p67vj ContainerName:agnhost-container Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:true Quiet:false}
Feb  5 23:54:53.817: INFO: >>> kubeConfig: /workspace/.kube/config
Feb  5 23:54:53.818: INFO: ExecWithOptions: Clientset creation
Feb  5 23:54:53.818: INFO: ExecWithOptions: execute(POST https://35.247.68.66/api/v1/namespaces/provisioning-9333/pods/hostexec-bootstrap-e2e-minion-group-vc67-p67vj/exec?command=nsenter&command=--mount%3D%2Frootfs%2Fproc%2F1%2Fns%2Fmnt&command=--&command=sh&command=-c&command=find+%2Fvar%2Flib%2Fkubelet%2Fplugins+-type+d+-exec+mountpoint+%7B%7D+%5C%3B+%7C+grep+%27is+a+mountpoint%24%27+%7C%7C+true&container=agnhost-container&container=agnhost-container&stderr=true&stdout=true)
STEP: Creating pod pod-subpath-test-preprovisionedpv-q5d2 - test/e2e/storage/testsuites/subpath.go:985 @ 02/05/23 23:54:54.166
STEP: Expecting the volume mount to be found. - test/e2e/storage/utils/utils.go:140 @ 02/05/23 23:54:58.424
Feb  5 23:54:58.949: INFO: ssh prow@34.118.193.103:22: command:   mount | grep bea0ae1f-c8a1-4863-a0fa-28331261466e | grep -v volume-subpaths
Feb  5 23:54:58.949: INFO: ssh prow@34.118.193.103:22: stdout:    "tmpfs on /var/lib/kubelet/pods/bea0ae1f-c8a1-4863-a0fa-28331261466e/volumes/kubernetes.io~local-volume/local-szhdn type tmpfs (rw)\ntmpfs on /var/lib/kubelet/pods/bea0ae1f-c8a1-4863-a0fa-28331261466e/volumes/kubernetes.io~local-volume/local-szhdn type tmpfs (rw,nosuid,nodev,noexec)\ntmpfs on /home/kubernetes/containerized_mounter/rootfs/var/lib/kubelet/pods/bea0ae1f-c8a1-4863-a0fa-28331261466e/volumes/kubernetes.io~local-volume/local-szhdn type tmpfs (rw,nosuid,nodev,noexec)\ntmpfs on /home/kubernetes/containerized_mounter/rootfs/var/lib/kubelet/pods/bea0ae1f-c8a1-4863-a0fa-28331261466e/volumes/kubernetes.io~local-volume/local-szhdn type tmpfs (rw,nosuid,nodev,noexec)\ntmpfs on /var/lib/kubelet/pods/bea0ae1f-c8a1-4863-a0fa-28331261466e/volumes/kubernetes.io~projected/kube-api-access-mfp7v type tmpfs (rw,relatime,size=7376260k)\ntmpfs on /var/lib/kubelet/pods/bea0ae1f-c8a1-4863-a0fa-28331261466e/volumes/kubernetes.io~projected/kube-api-access-mfp7v type tmpfs (rw,relatime,size=7376260k)\ntmpfs on /home/kubernetes/containerized_mounter/rootfs/var/lib/kubelet/pods/bea0ae1f-c8a1-4863-a0fa-28331261466e/volumes/kubernetes.io~projected/kube-api-access-mfp7v type tmpfs (rw,relatime,size=7376260k)\ntmpfs on /home/kubernetes/containerized_mounter/rootfs/var/lib/kubelet/pods/bea0ae1f-c8a1-4863-a0fa-28331261466e/volumes/kubernetes.io~projected/kube-api-access-mfp7v type tmpfs (rw,relatime,size=7376260k)\n"
Feb  5 23:54:58.949: INFO: ssh prow@34.118.193.103:22: stderr:    ""
Feb  5 23:54:58.949: INFO: ssh prow@34.118.193.103:22: exit code: 0
STEP: Expecting the volume subpath mount to be found. - test/e2e/storage/utils/utils.go:147 @ 02/05/23 23:54:58.949
Feb  5 23:54:59.466: INFO: ssh prow@34.118.193.103:22: command:   cat /proc/self/mountinfo | grep bea0ae1f-c8a1-4863-a0fa-28331261466e | grep volume-subpaths
Feb  5 23:54:59.466: INFO: ssh prow@34.118.193.103:22: stdout:    "1515 457 0:30 /local-driver-e8e50194-9ead-40de-af7c-8bcb8ceb62ca/provisioning-9333 /var/lib/kubelet/pods/bea0ae1f-c8a1-4863-a0fa-28331261466e/volume-subpaths/local-szhdn/test-container-subpath-preprovisionedpv-q5d2/0 rw shared:18 - tmpfs tmpfs rw\n1518 161 0:30 /local-driver-e8e50194-9ead-40de-af7c-8bcb8ceb62ca/provisioning-9333 /var/lib/kubelet/pods/bea0ae1f-c8a1-4863-a0fa-28331261466e/volume-subpaths/local-szhdn/test-container-subpath-preprovisionedpv-q5d2/0 rw shared:18 - tmpfs tmpfs rw\n1517 500 0:30 /local-driver-e8e50194-9ead-40de-af7c-8bcb8ceb62ca/provisioning-9333 /home/kubernetes/containerized_mounter/rootfs/var/lib/kubelet/pods/bea0ae1f-c8a1-4863-a0fa-28331261466e/volume-subpaths/local-szhdn/test-container-subpath-preprovisionedpv-q5d2/0 rw shared:18 - tmpfs tmpfs rw\n1516 498 0:30 /local-driver-e8e50194-9ead-40de-af7c-8bcb8ceb62ca/provisioning-9333 /home/kubernetes/containerized_mounter/rootfs/var/lib/kubelet/pods/bea0ae1f-c8a1-4863-a0fa-28331261466e/volume-subpaths/local-szhdn/test-container-subpath-preprovisionedpv-q5d2/0 rw shared:18 - tmpfs tmpfs rw\n"
Feb  5 23:54:59.466: INFO: ssh prow@34.118.193.103:22: stderr:    ""
Feb  5 23:54:59.466: INFO: ssh prow@34.118.193.103:22: exit code: 0
STEP: Writing to the volume. - test/e2e/storage/utils/utils.go:154 @ 02/05/23 23:54:59.466
Feb  5 23:54:59.466: INFO: ExecWithOptions {Command:[/bin/sh -c echo 2MEQHYuLwl3E4xURkbaZRtABsmgRZUrQHVG7Ba4i3ISQ+OE6sqE6d6MMVk/9zWlZbuKbfayuCnEVsNUTCCJxEw== | base64 -d | sha256sum] Namespace:provisioning-9333 PodName:pod-subpath-test-preprovisionedpv-q5d2 ContainerName:test-container-subpath-preprovisionedpv-q5d2 Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false Quiet:false}
Feb  5 23:54:59.466: INFO: >>> kubeConfig: /workspace/.kube/config
Feb  5 23:54:59.467: INFO: ExecWithOptions: Clientset creation
Feb  5 23:54:59.467: INFO: ExecWithOptions: execute(POST https://35.247.68.66/api/v1/namespaces/provisioning-9333/pods/pod-subpath-test-preprovisionedpv-q5d2/exec?command=%2Fbin%2Fsh&command=-c&command=echo+2MEQHYuLwl3E4xURkbaZRtABsmgRZUrQHVG7Ba4i3ISQ%2BOE6sqE6d6MMVk%2F9zWlZbuKbfayuCnEVsNUTCCJxEw%3D%3D+%7C+base64+-d+%7C+sha256sum&container=test-container-subpath-preprovisionedpv-q5d2&container=test-container-subpath-preprovisionedpv-q5d2&stderr=true&stdout=true)
Feb  5 23:54:59.770: INFO: ExecWithOptions {Command:[/bin/sh -c echo 2MEQHYuLwl3E4xURkbaZRtABsmgRZUrQHVG7Ba4i3ISQ+OE6sqE6d6MMVk/9zWlZbuKbfayuCnEVsNUTCCJxEw== | base64 -d | dd of=/test-volume/file1.txt  bs=64 count=1] Namespace:provisioning-9333 PodName:pod-subpath-test-preprovisionedpv-q5d2 ContainerName:test-container-subpath-preprovisionedpv-q5d2 Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false Quiet:false}
Feb  5 23:54:59.770: INFO: >>> kubeConfig: /workspace/.kube/config
Feb  5 23:54:59.771: INFO: ExecWithOptions: Clientset creation
Feb  5 23:54:59.771: INFO: ExecWithOptions: execute(POST https://35.247.68.66/api/v1/namespaces/provisioning-9333/pods/pod-subpath-test-preprovisionedpv-q5d2/exec?command=%2Fbin%2Fsh&command=-c&command=echo+2MEQHYuLwl3E4xURkbaZRtABsmgRZUrQHVG7Ba4i3ISQ%2BOE6sqE6d6MMVk%2F9zWlZbuKbfayuCnEVsNUTCCJxEw%3D%3D+%7C+base64+-d+%7C+dd+of%3D%2Ftest-volume%2Ffile1.txt++bs%3D64+count%3D1&container=test-container-subpath-preprovisionedpv-q5d2&container=test-container-subpath-preprovisionedpv-q5d2&stderr=true&stdout=true)
STEP: Stopping the kubelet. - test/e2e/storage/utils/utils.go:161 @ 02/05/23 23:55:00.077
Feb  5 23:55:00.119: INFO: Checking if systemctl command is present
Feb  5 23:55:00.642: INFO: Checking if sudo command is present
Feb  5 23:55:01.163: INFO: Attempting `sudo systemctl stop kubelet`
Feb  5 23:55:01.722: INFO: ssh prow@34.118.193.103:22: command:   sudo systemctl stop kubelet
Feb  5 23:55:01.722: INFO: ssh prow@34.118.193.103:22: stdout:    ""
Feb  5 23:55:01.722: INFO: ssh prow@34.118.193.103:22: stderr:    ""
Feb  5 23:55:01.722: INFO: ssh prow@34.118.193.103:22: exit code: 0
Feb  5 23:55:01.722: INFO: Waiting up to 1m0s for node bootstrap-e2e-minion-group-7p0n condition Ready to be false
Feb  5 23:55:01.764: INFO: Condition Ready of node bootstrap-e2e-minion-group-7p0n is true instead of false. Reason: KubeletReady, message: kubelet is posting ready status. AppArmor enabled
Feb  5 23:55:03.808: INFO: Condition Ready of node bootstrap-e2e-minion-group-7p0n is true instead of false. Reason: KubeletReady, message: kubelet is posting ready status. AppArmor enabled
Feb  5 23:55:05.850: INFO: Condition Ready of node bootstrap-e2e-minion-group-7p0n is true instead of false. Reason: KubeletReady, message: kubelet is posting ready status. AppArmor enabled
Feb  5 23:55:07.895: INFO: Condition Ready of node bootstrap-e2e-minion-group-7p0n is true instead of false. Reason: KubeletReady, message: kubelet is posting ready status. AppArmor enabled
Feb  5 23:55:09.937: INFO: Condition Ready of node bootstrap-e2e-minion-group-7p0n is true instead of false. Reason: KubeletReady, message: kubelet is posting ready status. AppArmor enabled
Feb  5 23:55:11.982: INFO: Condition Ready of node bootstrap-e2e-minion-group-7p0n is true instead of false. Reason: KubeletReady, message: kubelet is posting ready status. AppArmor enabled
Feb  5 23:55:14.026: INFO: Condition Ready of node bootstrap-e2e-minion-group-7p0n is true instead of false. Reason: KubeletReady, message: kubelet is posting ready status. AppArmor enabled
Feb  5 23:55:16.069: INFO: Condition Ready of node bootstrap-e2e-minion-group-7p0n is true instead of false. Reason: KubeletReady, message: kubelet is posting ready status. AppArmor enabled
Feb  5 23:55:18.112: INFO: Condition Ready of node bootstrap-e2e-minion-group-7p0n is true instead of false. Reason: KubeletReady, message: kubelet is posting ready status. AppArmor enabled
Feb  5 23:55:20.155: INFO: Condition Ready of node bootstrap-e2e-minion-group-7p0n is true instead of false. Reason: KubeletReady, message: kubelet is posting ready status. AppArmor enabled
Feb  5 23:55:22.197: INFO: Condition Ready of node bootstrap-e2e-minion-group-7p0n is true instead of false. Reason: KubeletReady, message: kubelet is posting ready status. AppArmor enabled
Feb  5 23:55:24.240: INFO: Condition Ready of node bootstrap-e2e-minion-group-7p0n is true instead of false. Reason: KubeletReady, message: kubelet is posting ready status. AppArmor enabled
Feb  5 23:55:26.283: INFO: Condition Ready of node bootstrap-e2e-minion-group-7p0n is true instead of false. Reason: KubeletReady, message: kubelet is posting ready status. AppArmor enabled
Feb  5 23:55:28.327: INFO: Condition Ready of node bootstrap-e2e-minion-group-7p0n is true instead of false. Reason: KubeletReady, message: kubelet is posting ready status. AppArmor enabled
Feb  5 23:55:30.370: INFO: Condition Ready of node bootstrap-e2e-minion-group-7p0n is true instead of false. Reason: KubeletReady, message: kubelet is posting ready status. AppArmor enabled
Feb  5 23:55:32.414: INFO: Condition Ready of node bootstrap-e2e-minion-group-7p0n is true instead of false. Reason: KubeletReady, message: kubelet is posting ready status. AppArmor enabled
Feb  5 23:55:34.457: INFO: Condition Ready of node bootstrap-e2e-minion-group-7p0n is true instead of false. Reason: KubeletReady, message: kubelet is posting ready status. AppArmor enabled
Feb  5 23:55:36.500: INFO: Condition Ready of node bootstrap-e2e-minion-group-7p0n is true instead of false. Reason: KubeletReady, message: kubelet is posting ready status. AppArmor enabled
STEP: Deleting Pod "pod-subpath-test-preprovisionedpv-q5d2" - test/e2e/storage/utils/utils.go:170 @ 02/05/23 23:55:38.543
STEP: Starting the kubelet and waiting for pod to delete. - test/e2e/storage/utils/utils.go:178 @ 02/05/23 23:55:38.588
Feb  5 23:55:38.634: INFO: Checking if systemctl command is present
Feb  5 23:55:39.163: INFO: Checking if sudo command is present
Feb  5 23:55:39.687: INFO: Attempting `sudo systemctl start kubelet`
Feb  5 23:55:40.225: INFO: ssh prow@34.118.193.103:22: command:   sudo systemctl start kubelet
Feb  5 23:55:40.225: INFO: ssh prow@34.118.193.103:22: stdout:    ""
Feb  5 23:55:40.225: INFO: ssh prow@34.118.193.103:22: stderr:    ""
Feb  5 23:55:40.225: INFO: ssh prow@34.118.193.103:22: exit code: 0
Feb  5 23:55:40.225: INFO: Waiting up to 1m0s for node bootstrap-e2e-minion-group-7p0n condition Ready to be true
Feb  5 23:55:40.268: INFO: Condition Ready of node bootstrap-e2e-minion-group-7p0n is false instead of true. Reason: NodeStatusUnknown, message: Kubelet stopped posting node status.
STEP: Expecting the volume mount not to be found. - test/e2e/storage/utils/utils.go:214 @ 02/05/23 23:56:13.03
Feb  5 23:56:23.630: INFO: ssh prow@34.118.193.103:22: command:   mount | grep bea0ae1f-c8a1-4863-a0fa-28331261466e | grep -v volume-subpaths
Feb  5 23:56:23.630: INFO: ssh prow@34.118.193.103:22: stdout:    ""
Feb  5 23:56:23.630: INFO: ssh prow@34.118.193.103:22: stderr:    ""
Feb  5 23:56:23.630: INFO: ssh prow@34.118.193.103:22: exit code: 1
Feb  5 23:56:23.630: INFO: Volume unmounted on node bootstrap-e2e-minion-group-7p0n
STEP: Expecting the volume subpath mount not to be found. - test/e2e/storage/utils/utils.go:222 @ 02/05/23 23:56:23.63
Feb  5 23:56:24.150: INFO: ssh prow@34.118.193.103:22: command:   cat /proc/self/mountinfo | grep bea0ae1f-c8a1-4863-a0fa-28331261466e | grep volume-subpaths
Feb  5 23:56:24.150: INFO: ssh prow@34.118.193.103:22: stdout:    ""
Feb  5 23:56:24.150: INFO: ssh prow@34.118.193.103:22: stderr:    ""
Feb  5 23:56:24.150: INFO: ssh prow@34.118.193.103:22: exit code: 1
Feb  5 23:56:24.150: INFO: Subpath volume unmounted on node bootstrap-e2e-minion-group-7p0n
Feb  5 23:56:24.150: INFO: ExecWithOptions {Command:[nsenter --mount=/rootfs/proc/1/ns/mnt -- sh -c find /var/lib/kubelet/plugins -type d -exec mountpoint {} \; | grep 'is a mountpoint$' || true] Namespace:provisioning-9333 PodName:hostexec-bootstrap-e2e-minion-group-7p0n-7jcfh ContainerName:agnhost-container Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:true Quiet:false}
Feb  5 23:56:24.150: INFO: >>> kubeConfig: /workspace/.kube/config
Feb  5 23:56:24.151: INFO: ExecWithOptions: Clientset creation
Feb  5 23:56:24.151: INFO: ExecWithOptions: execute(POST https://35.247.68.66/api/v1/namespaces/provisioning-9333/pods/hostexec-bootstrap-e2e-minion-group-7p0n-7jcfh/exec?command=nsenter&command=--mount%3D%2Frootfs%2Fproc%2F1%2Fns%2Fmnt&command=--&command=sh&command=-c&command=find+%2Fvar%2Flib%2Fkubelet%2Fplugins+-type+d+-exec+mountpoint+%7B%7D+%5C%3B+%7C+grep+%27is+a+mountpoint%24%27+%7C%7C+true&container=agnhost-container&container=agnhost-container&stderr=true&stdout=true)
< Exit [It] should unmount if pod is gracefully deleted while kubelet is down [Disruptive][Slow][LinuxOnly] - test/e2e/storage/testsuites/subpath.go:341 @ 02/05/23 23:56:24.53 (1m52.881s)
> Enter [AfterEach] [Testpattern: Pre-provisioned PV (default fs)] subPath - test/e2e/framework/node/init/init.go:33 @ 02/05/23 23:56:24.53
Feb  5 23:56:24.530: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready
< Exit [AfterEach] [Testpattern: Pre-provisioned PV (default fs)] subPath - test/e2e/framework/node/init/init.go:33 @ 02/05/23 23:56:24.573 (43ms)
> Enter [DeferCleanup (Each)] [Testpattern: Pre-provisioned PV (default fs)] subPath - test/e2e/storage/utils/utils.go:160 @ 02/05/23 23:56:24.573
Feb  5 23:56:24.615: INFO: Checking if systemctl command is present
Feb  5 23:56:44.852: INFO: Unexpected error: SSH to Node "bootstrap-e2e-minion-group-7p0n" errored.: 
    <*fmt.wrapError | 0xc00495b520>: {
        msg: "error getting SSH client to prow@34.118.193.103:22: timed out waiting for the condition",
        err: <*errors.errorString | 0xae80790>{
            s: "timed out waiting for the condition",
        },
    }
[FAILED] SSH to Node "bootstrap-e2e-minion-group-7p0n" errored.: error getting SSH client to prow@34.118.193.103:22: timed out waiting for the condition
In [DeferCleanup (Each)] at: test/e2e/storage/utils/pod.go:117 @ 02/05/23 23:56:44.852
< Exit [DeferCleanup (Each)] [Testpattern: Pre-provisioned PV (default fs)] subPath - test/e2e/storage/utils/utils.go:160 @ 02/05/23 23:56:44.852 (20.279s)
> Enter [DeferCleanup (Each)] [Testpattern: Pre-provisioned PV (default fs)] subPath - test/e2e/storage/testsuites/subpath.go:344 @ 02/05/23 23:56:44.852
STEP: Deleting pod - test/e2e/storage/testsuites/subpath.go:172 @ 02/05/23 23:56:44.852
Feb  5 23:56:44.852: INFO: Deleting pod "pod-subpath-test-preprovisionedpv-q5d2" in namespace "provisioning-9333"
STEP: Deleting pv and pvc - test/e2e/storage/framework/volume_resource.go:176 @ 02/05/23 23:56:44.895
Feb  5 23:56:44.895: INFO: Deleting PersistentVolumeClaim "pvc-6wbt5"
Feb  5 23:56:44.940: INFO: Deleting PersistentVolume "local-szhdn"
STEP: Removing the test directory - test/e2e/storage/utils/local.go:209 @ 02/05/23 23:56:44.984
Feb  5 23:56:44.984: INFO: ExecWithOptions {Command:[nsenter --mount=/rootfs/proc/1/ns/mnt -- sh -c rm -r /tmp/local-driver-e8e50194-9ead-40de-af7c-8bcb8ceb62ca] Namespace:provisioning-9333 PodName:hostexec-bootstrap-e2e-minion-group-7p0n-6lr4r ContainerName:agnhost-container Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:true Quiet:false}
Feb  5 23:56:44.984: INFO: >>> kubeConfig: /workspace/.kube/config
Feb  5 23:56:44.985: INFO: ExecWithOptions: Clientset creation
Feb  5 23:56:44.985: INFO: ExecWithOptions: execute(POST https://35.247.68.66/api/v1/namespaces/provisioning-9333/pods/hostexec-bootstrap-e2e-minion-group-7p0n-6lr4r/exec?command=nsenter&command=--mount%3D%2Frootfs%2Fproc%2F1%2Fns%2Fmnt&command=--&command=sh&command=-c&command=rm+-r+%2Ftmp%2Flocal-driver-e8e50194-9ead-40de-af7c-8bcb8ceb62ca&container=agnhost-container&container=agnhost-container&stderr=true&stdout=true)
STEP: Deleting pod hostexec-bootstrap-e2e-minion-group-7p0n-7jcfh in namespace provisioning-9333 - test/e2e/framework/pod/delete.go:41 @ 02/05/23 23:56:45.3
STEP: Deleting pod hostexec-bootstrap-e2e-minion-group-f2xv-jqtng in namespace provisioning-9333 - test/e2e/framework/pod/delete.go:41 @ 02/05/23 23:56:45.35
STEP: Deleting pod hostexec-bootstrap-e2e-minion-group-vc67-p67vj in namespace provisioning-9333 - test/e2e/framework/pod/delete.go:41 @ 02/05/23 23:56:45.4
< Exit [DeferCleanup (Each)] [Testpattern: Pre-provisioned PV (default fs)] subPath - test/e2e/storage/testsuites/subpath.go:344 @ 02/05/23 23:56:45.45 (598ms)
> Enter [DeferCleanup (Each)] [Testpattern: Pre-provisioned PV (default fs)] subPath - test/e2e/storage/drivers/in_tree.go:1729 @ 02/05/23 23:56:45.45
STEP: Deleting pod hostexec-bootstrap-e2e-minion-group-7p0n-6lr4r in namespace provisioning-9333 - test/e2e/framework/pod/delete.go:41 @ 02/05/23 23:56:45.451
< Exit [DeferCleanup (Each)] [Testpattern: Pre-provisioned PV (default fs)] subPath - test/e2e/storage/drivers/in_tree.go:1729 @ 02/05/23 23:56:45.499 (49ms)
> Enter [DeferCleanup (Each)] [Testpattern: Pre-provisioned PV (default fs)] subPath - test/e2e/framework/metrics/init/init.go:35 @ 02/05/23 23:56:45.499
< Exit [DeferCleanup (Each)] [Testpattern: Pre-provisioned PV (default fs)] subPath - test/e2e/framework/metrics/init/init.go:35 @ 02/05/23 23:56:45.5 (0s)
> Enter [DeferCleanup (Each)] [Testpattern: Pre-provisioned PV (default fs)] subPath - dump namespaces | framework.go:206 @ 02/05/23 23:56:45.5
STEP: dump namespace information after failure - test/e2e/framework/framework.go:284 @ 02/05/23 23:56:45.5
STEP: Collecting events from namespace "provisioning-9333". - test/e2e/framework/debug/dump.go:42 @ 02/05/23 23:56:45.5
STEP: Found 34 events. - test/e2e/framework/debug/dump.go:46 @ 02/05/23 23:56:45.542
Feb  5 23:56:45.542: INFO: At 2023-02-05 23:54:31 +0000 UTC - event for hostexec-bootstrap-e2e-minion-group-7p0n-6lr4r: {default-scheduler } Scheduled: Successfully assigned provisioning-9333/hostexec-bootstrap-e2e-minion-group-7p0n-6lr4r to bootstrap-e2e-minion-group-7p0n
Feb  5 23:56:45.542: INFO: At 2023-02-05 23:54:32 +0000 UTC - event for hostexec-bootstrap-e2e-minion-group-7p0n-6lr4r: {kubelet bootstrap-e2e-minion-group-7p0n} Pulled: Container image "registry.k8s.io/e2e-test-images/agnhost:2.43" already present on machine
Feb  5 23:56:45.542: INFO: At 2023-02-05 23:54:32 +0000 UTC - event for hostexec-bootstrap-e2e-minion-group-7p0n-6lr4r: {kubelet bootstrap-e2e-minion-group-7p0n} Created: Created container agnhost-container
Feb  5 23:56:45.542: INFO: At 2023-02-05 23:54:32 +0000 UTC - event for hostexec-bootstrap-e2e-minion-group-7p0n-6lr4r: {kubelet bootstrap-e2e-minion-group-7p0n} Started: Started container agnhost-container
Feb  5 23:56:45.542: INFO: At 2023-02-05 23:54:34 +0000 UTC - event for pvc-6wbt5: {persistentvolume-controller } ProvisioningFailed: storageclass.storage.k8s.io "provisioning-9333" not found
Feb  5 23:56:45.542: INFO: At 2023-02-05 23:54:46 +0000 UTC - event for hostexec-bootstrap-e2e-minion-group-7p0n-7jcfh: {default-scheduler } Scheduled: Successfully assigned provisioning-9333/hostexec-bootstrap-e2e-minion-group-7p0n-7jcfh to bootstrap-e2e-minion-group-7p0n
Feb  5 23:56:45.542: INFO: At 2023-02-05 23:54:47 +0000 UTC - event for hostexec-bootstrap-e2e-minion-group-7p0n-7jcfh: {kubelet bootstrap-e2e-minion-group-7p0n} Created: Created container agnhost-container
Feb  5 23:56:45.542: INFO: At 2023-02-05 23:54:47 +0000 UTC - event for hostexec-bootstrap-e2e-minion-group-7p0n-7jcfh: {kubelet bootstrap-e2e-minion-group-7p0n} Pulled: Container image "registry.k8s.io/e2e-test-images/agnhost:2.43" already present on machine
Feb  5 23:56:45.542: INFO: At 2023-02-05 23:54:47 +0000 UTC - event for hostexec-bootstrap-e2e-minion-group-7p0n-7jcfh: {kubelet bootstrap-e2e-minion-group-7p0n} Started: Started container agnhost-container
Feb  5 23:56:45.542: INFO: At 2023-02-05 23:54:49 +0000 UTC - event for hostexec-bootstrap-e2e-minion-group-f2xv-jqtng: {default-scheduler } Scheduled: Successfully assigned provisioning-9333/hostexec-bootstrap-e2e-minion-group-f2xv-jqtng to bootstrap-e2e-minion-group-f2xv
Feb  5 23:56:45.542: INFO: At 2023-02-05 23:54:49 +0000 UTC - event for hostexec-bootstrap-e2e-minion-group-f2xv-jqtng: {kubelet bootstrap-e2e-minion-group-f2xv} Pulled: Container image "registry.k8s.io/e2e-test-images/agnhost:2.43" already present on machine
Feb  5 23:56:45.542: INFO: At 2023-02-05 23:54:49 +0000 UTC - event for hostexec-bootstrap-e2e-minion-group-f2xv-jqtng: {kubelet bootstrap-e2e-minion-group-f2xv} Created: Created container agnhost-container
Feb  5 23:56:45.542: INFO: At 2023-02-05 23:54:49 +0000 UTC - event for hostexec-bootstrap-e2e-minion-group-f2xv-jqtng: {kubelet bootstrap-e2e-minion-group-f2xv} Started: Started container agnhost-container
Feb  5 23:56:45.542: INFO: At 2023-02-05 23:54:51 +0000 UTC - event for hostexec-bootstrap-e2e-minion-group-vc67-p67vj: {default-scheduler } Scheduled: Successfully assigned provisioning-9333/hostexec-bootstrap-e2e-minion-group-vc67-p67vj to bootstrap-e2e-minion-group-vc67
Feb  5 23:56:45.542: INFO: At 2023-02-05 23:54:52 +0000 UTC - event for hostexec-bootstrap-e2e-minion-group-vc67-p67vj: {kubelet bootstrap-e2e-minion-group-vc67} Created: Created container agnhost-container
Feb  5 23:56:45.542: INFO: At 2023-02-05 23:54:52 +0000 UTC - event for hostexec-bootstrap-e2e-minion-group-vc67-p67vj: {kubelet bootstrap-e2e-minion-group-vc67} Started: Started container agnhost-container
Feb  5 23:56:45.542: INFO: At 2023-02-05 23:54:52 +0000 UTC - event for hostexec-bootstrap-e2e-minion-group-vc67-p67vj: {kubelet bootstrap-e2e-minion-group-vc67} Pulled: Container image "registry.k8s.io/e2e-test-images/agnhost:2.43" already present on machine
Feb  5 23:56:45.542: INFO: At 2023-02-05 23:54:54 +0000 UTC - event for pod-subpath-test-preprovisionedpv-q5d2: {kubelet bootstrap-e2e-minion-group-7p0n} Pulling: Pulling image "registry.k8s.io/e2e-test-images/busybox:1.29-4"
Feb  5 23:56:45.542: INFO: At 2023-02-05 23:54:54 +0000 UTC - event for pod-subpath-test-preprovisionedpv-q5d2: {default-scheduler } Scheduled: Successfully assigned provisioning-9333/pod-subpath-test-preprovisionedpv-q5d2 to bootstrap-e2e-minion-group-7p0n
Feb  5 23:56:45.542: INFO: At 2023-02-05 23:54:55 +0000 UTC - event for pod-subpath-test-preprovisionedpv-q5d2: {kubelet bootstrap-e2e-minion-group-7p0n} Pulled: Successfully pulled image "registry.k8s.io/e2e-test-images/busybox:1.29-4" in 423.146995ms (423.15762ms including waiting)
Feb  5 23:56:45.542: INFO: At 2023-02-05 23:54:55 +0000 UTC - event for pod-subpath-test-preprovisionedpv-q5d2: {kubelet bootstrap-e2e-minion-group-7p0n} Started: Started container test-container-volume-preprovisionedpv-q5d2
Feb  5 23:56:45.542: INFO: At 2023-02-05 23:54:55 +0000 UTC - event for pod-subpath-test-preprovisionedpv-q5d2: {kubelet bootstrap-e2e-minion-group-7p0n} Created: Created container test-container-volume-preprovisionedpv-q5d2
Feb  5 23:56:45.542: INFO: At 2023-02-05 23:54:55 +0000 UTC - event for pod-subpath-test-preprovisionedpv-q5d2: {kubelet bootstrap-e2e-minion-group-7p0n} Pulled: Container image "registry.k8s.io/e2e-test-images/busybox:1.29-4" already present on machine
Feb  5 23:56:45.542: INFO: At 2023-02-05 23:54:55 +0000 UTC - event for pod-subpath-test-preprovisionedpv-q5d2: {kubelet bootstrap-e2e-minion-group-7p0n} Started: Started container test-container-subpath-preprovisionedpv-q5d2
Feb  5 23:56:45.542: INFO: At 2023-02-05 23:54:55 +0000 UTC - event for pod-subpath-test-preprovisionedpv-q5d2: {kubelet bootstrap-e2e-minion-group-7p0n} Created: Created container test-container-subpath-preprovisionedpv-q5d2
Feb  5 23:56:45.542: INFO: At 2023-02-05 23:55:37 +0000 UTC - event for hostexec-bootstrap-e2e-minion-group-7p0n-6lr4r: {node-controller } NodeNotReady: Node is not ready
Feb  5 23:56:45.542: INFO: At 2023-02-05 23:55:37 +0000 UTC - event for hostexec-bootstrap-e2e-minion-group-7p0n-7jcfh: {node-controller } NodeNotReady: Node is not ready
Feb  5 23:56:45.542: INFO: At 2023-02-05 23:55:37 +0000 UTC - event for pod-subpath-test-preprovisionedpv-q5d2: {node-controller } NodeNotReady: Node is not ready
Feb  5 23:56:45.542: INFO: At 2023-02-05 23:55:41 +0000 UTC - event for pod-subpath-test-preprovisionedpv-q5d2: {kubelet bootstrap-e2e-minion-group-7p0n} Killing: Stopping container test-container-subpath-preprovisionedpv-q5d2
Feb  5 23:56:45.542: INFO: At 2023-02-05 23:55:41 +0000 UTC - event for pod-subpath-test-preprovisionedpv-q5d2: {kubelet bootstrap-e2e-minion-group-7p0n} Killing: Stopping container test-container-volume-preprovisionedpv-q5d2
Feb  5 23:56:45.542: INFO: At 2023-02-05 23:56:45 +0000 UTC - event for hostexec-bootstrap-e2e-minion-group-7p0n-6lr4r: {kubelet bootstrap-e2e-minion-group-7p0n} Killing: Stopping container agnhost-container
Feb  5 23:56:45.542: INFO: At 2023-02-05 23:56:45 +0000 UTC - event for hostexec-bootstrap-e2e-minion-group-7p0n-7jcfh: {kubelet bootstrap-e2e-minion-group-7p0n} Killing: Stopping container agnhost-container
Feb  5 23:56:45.542: INFO: At 2023-02-05 23:56:45 +0000 UTC - event for hostexec-bootstrap-e2e-minion-group-f2xv-jqtng: {kubelet bootstrap-e2e-minion-group-f2xv} Killing: Stopping container agnhost-container
Feb  5 23:56:45.542: INFO: At 2023-02-05 23:56:45 +0000 UTC - event for hostexec-bootstrap-e2e-minion-group-vc67-p67vj: {kubelet bootstrap-e2e-minion-group-vc67} Killing: Stopping container agnhost-container
Feb  5 23:56:45.583: INFO: POD  NODE  PHASE  GRACE  CONDITIONS
Feb  5 23:56:45.583: INFO: 
Feb  5 23:56:45.625: INFO: 
Logging node info for node bootstrap-e2e-master
Feb  5 23:56:45.667: INFO: Node Info: &Node{ObjectMeta:{bootstrap-e2e-master    cff6e042-47eb-4a9a-b932-5766273d8cbd 62051 0 2023-02-05 20:03:54 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:n1-standard-1 beta.kubernetes.io/os:linux cloud.google.com/metadata-proxy-ready:true failure-domain.beta.kubernetes.io/region:us-west1 failure-domain.beta.kubernetes.io/zone:us-west1-b kubernetes.io/arch:amd64 kubernetes.io/hostname:bootstrap-e2e-master kubernetes.io/os:linux node.kubernetes.io/instance-type:n1-standard-1 topology.gke.io/zone:us-west1-b topology.kubernetes.io/region:us-west1 topology.kubernetes.io/zone:us-west1-b] map[node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kubelet Update v1 2023-02-05 20:03:54 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:cloud.google.com/metadata-proxy-ready":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{},"f:unschedulable":{}}} } {kube-controller-manager Update v1 2023-02-05 20:04:10 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"10.64.3.0/24\"":{}},"f:taints":{}}} } {kube-controller-manager Update v1 2023-02-05 20:04:10 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}}}} status} {kubelet Update v1 2023-02-05 23:55:24 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:topology.gke.io/zone":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:10.64.3.0/24,DoNotUseExternalID:,ProviderID:gce://k8s-infra-e2e-boskos-001/us-west1-b/bootstrap-e2e-master,Unschedulable:true,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/master,Value:,Effect:NoSchedule,TimeAdded:<nil>,},Taint{Key:node.kubernetes.io/unschedulable,Value:,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[10.64.3.0/24],},Status:NodeStatus{Capacity:ResourceList{attachable-volumes-gce-pd: {{127 0} {<nil>} 127 DecimalSI},cpu: {{1 0} {<nil>} 1 DecimalSI},ephemeral-storage: {{16656896000 0} {<nil>} 16266500Ki BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3858370560 0} {<nil>} 3767940Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{attachable-volumes-gce-pd: {{127 0} {<nil>} 127 DecimalSI},cpu: {{1 0} {<nil>} 1 DecimalSI},ephemeral-storage: {{14991206376 0} {<nil>} 14991206376 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3596226560 0} {<nil>} 3511940Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2023-02-05 20:04:10 +0000 UTC,LastTransitionTime:2023-02-05 20:04:10 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-02-05 23:55:24 +0000 UTC,LastTransitionTime:2023-02-05 20:03:54 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-02-05 23:55:24 +0000 UTC,LastTransitionTime:2023-02-05 20:03:54 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-02-05 23:55:24 +0000 UTC,LastTransitionTime:2023-02-05 20:03:54 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-02-05 23:55:24 +0000 UTC,LastTransitionTime:2023-02-05 20:03:56 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:10.138.0.2,},NodeAddress{Type:ExternalIP,Address:35.247.68.66,},NodeAddress{Type:InternalDNS,Address:bootstrap-e2e-master.c.k8s-infra-e2e-boskos-001.internal,},NodeAddress{Type:Hostname,Address:bootstrap-e2e-master.c.k8s-infra-e2e-boskos-001.internal,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:6c56a7468028f8335ce3a87f603c6ce3,SystemUUID:6c56a746-8028-f833-5ce3-a87f603c6ce3,BootID:5e24514d-7978-4290-b3ac-f42b99d56778,KernelVersion:5.10.123+,OSImage:Container-Optimized OS from Google,ContainerRuntimeVersion:containerd://1.6.6,KubeletVersion:v1.27.0-alpha.1.249+cf14b50b0d2215,KubeProxyVersion:v1.27.0-alpha.1.249+cf14b50b0d2215,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[registry.k8s.io/kube-apiserver-amd64:v1.27.0-alpha.1.249_cf14b50b0d2215],SizeBytes:136985951,},ContainerImage{Names:[registry.k8s.io/kube-controller-manager-amd64:v1.27.0-alpha.1.249_cf14b50b0d2215],SizeBytes:126037497,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:51eae8381dcb1078289fa7b4f3df2630cdc18d09fb56f8e56b41c40e191d6c83 registry.k8s.io/etcd:3.5.7-0],SizeBytes:101639218,},ContainerImage{Names:[registry.k8s.io/cloud-provider-gcp/gcp-compute-persistent-disk-csi-driver@sha256:e9c355dd1ce57de91a0f9164bd2fe688938723e7f9bd40bb13652d603f41d25a registry.k8s.io/cloud-provider-gcp/gcp-compute-persistent-disk-csi-driver:v1.2.2],SizeBytes:73354282,},ContainerImage{Names:[registry.k8s.io/kube-scheduler-amd64:v1.27.0-alpha.1.249_cf14b50b0d2215],SizeBytes:58412524,},ContainerImage{Names:[gke.gcr.io/prometheus-to-sd@sha256:e739643c3939ba0b161425f45a1989eedfc4a3b166db9a7100863296b4c70510 gke.gcr.io/prometheus-to-sd:v0.11.1-gke.1],SizeBytes:48742566,},ContainerImage{Names:[gcr.io/k8s-ingress-image-push/ingress-gce-glbc-amd64@sha256:5db27383add6d9f4ebdf0286409ac31f7f5d273690204b341a4e37998917693b gcr.io/k8s-ingress-image-push/ingress-gce-glbc-amd64:v1.20.1],SizeBytes:36598135,},ContainerImage{Names:[registry.k8s.io/addon-manager/kube-addon-manager@sha256:49cc4e6e4a3745b427ce14b0141476ab339bb65c6bc05033019e046c8727dcb0 registry.k8s.io/addon-manager/kube-addon-manager:v9.1.6],SizeBytes:30464183,},ContainerImage{Names:[registry.k8s.io/kas-network-proxy/proxy-server@sha256:b1389e7014425a1752aac55f5043ef4c52edaef0e223bf4d48ed1324e298087c registry.k8s.io/kas-network-proxy/proxy-server:v0.1.1],SizeBytes:21875112,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/metadata-proxy@sha256:e914645f22e946bce5165737e1b244e0a296ad1f0f81a9531adc57af2780978a registry.k8s.io/metadata-proxy:v0.1.12],SizeBytes:5301657,},ContainerImage{Names:[registry.k8s.io/pause@sha256:7031c1b283388d2c2e09b57badb803c05ebed362dc88d84b480cc47f72a21097 registry.k8s.io/pause:3.9],SizeBytes:321520,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},}
Feb  5 23:56:45.667: INFO: 
Logging kubelet events for node bootstrap-e2e-master
Feb  5 23:56:45.709: INFO: 
Logging pods the kubelet thinks is on node bootstrap-e2e-master
Feb  5 23:56:45.765: INFO: etcd-server-bootstrap-e2e-master started at 2023-02-05 20:03:12 +0000 UTC (0+1 container statuses recorded)
Feb  5 23:56:45.765: INFO: 	Container etcd-container ready: true, restart count 0
Feb  5 23:56:45.765: INFO: l7-lb-controller-bootstrap-e2e-master started at 2023-02-05 20:03:28 +0000 UTC (0+1 container statuses recorded)
Feb  5 23:56:45.765: INFO: 	Container l7-lb-controller ready: true, restart count 2
Feb  5 23:56:45.765: INFO: kube-controller-manager-bootstrap-e2e-master started at 2023-02-05 20:03:12 +0000 UTC (0+1 container statuses recorded)
Feb  5 23:56:45.765: INFO: 	Container kube-controller-manager ready: true, restart count 3
Feb  5 23:56:45.765: INFO: kube-scheduler-bootstrap-e2e-master started at 2023-02-05 20:03:12 +0000 UTC (0+1 container statuses recorded)
Feb  5 23:56:45.765: INFO: 	Container kube-scheduler ready: true, restart count 0
Feb  5 23:56:45.765: INFO: etcd-server-events-bootstrap-e2e-master started at 2023-02-05 20:03:12 +0000 UTC (0+1 container statuses recorded)
Feb  5 23:56:45.765: INFO: 	Container etcd-container ready: true, restart count 0
Feb  5 23:56:45.765: INFO: konnectivity-server-bootstrap-e2e-master started at 2023-02-05 20:03:12 +0000 UTC (0+1 container statuses recorded)
Feb  5 23:56:45.765: INFO: 	Container konnectivity-server-container ready: true, restart count 0
Feb  5 23:56:45.765: INFO: kube-apiserver-bootstrap-e2e-master started at 2023-02-05 20:03:12 +0000 UTC (0+1 container statuses recorded)
Feb  5 23:56:45.765: INFO: 	Container kube-apiserver ready: true, restart count 0
Feb  5 23:56:45.765: INFO: kube-addon-manager-bootstrap-e2e-master started at 2023-02-05 20:03:28 +0000 UTC (0+1 container statuses recorded)
Feb  5 23:56:45.765: INFO: 	Container kube-addon-manager ready: true, restart count 0
Feb  5 23:56:45.765: INFO: metadata-proxy-v0.1-xv5df started at 2023-02-05 20:03:55 +0000 UTC (0+2 container statuses recorded)
Feb  5 23:56:45.765: INFO: 	Container metadata-proxy ready: true, restart count 0
Feb  5 23:56:45.765: INFO: 	Container prometheus-to-sd-exporter ready: true, restart count 0
Feb  5 23:56:45.991: INFO: 
Latency metrics for node bootstrap-e2e-master
Feb  5 23:56:45.991: INFO: 
Logging node info for node bootstrap-e2e-minion-group-7p0n
Feb  5 23:56:46.033: INFO: Node Info: &Node{ObjectMeta:{bootstrap-e2e-minion-group-7p0n    1b0c4b6f-6ac7-457b-aaed-c0715264fb74 62102 0 2023-02-05 23:49:36 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:n1-standard-2 beta.kubernetes.io/os:linux cloud.google.com/metadata-proxy-ready:true failure-domain.beta.kubernetes.io/region:us-west1 failure-domain.beta.kubernetes.io/zone:us-west1-b kubernetes.io/arch:amd64 kubernetes.io/hostname:bootstrap-e2e-minion-group-7p0n kubernetes.io/os:linux node.kubernetes.io/instance-type:n1-standard-2 topology.gke.io/zone:us-west1-b topology.kubernetes.io/region:us-west1 topology.kubernetes.io/zone:us-west1-b] map[node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kubelet Update v1 2023-02-05 23:49:36 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:cloud.google.com/metadata-proxy-ready":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {node-problem-detector Update v1 2023-02-05 23:54:41 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"CorruptDockerOverlay2\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"FrequentContainerdRestart\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"FrequentDockerRestart\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"FrequentKubeletRestart\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"FrequentUnregisterNetDevice\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"KernelDeadlock\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"ReadonlyFilesystem\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kube-controller-manager Update v1 2023-02-05 23:55:37 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}}}} status} {kube-controller-manager Update v1 2023-02-05 23:55:40 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"10.64.2.0/24\"":{}}}} } {kubelet Update v1 2023-02-05 23:55:40 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:topology.gke.io/zone":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:10.64.2.0/24,DoNotUseExternalID:,ProviderID:gce://k8s-infra-e2e-boskos-001/us-west1-b/bootstrap-e2e-minion-group-7p0n,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[10.64.2.0/24],},Status:NodeStatus{Capacity:ResourceList{attachable-volumes-gce-pd: {{127 0} {<nil>} 127 DecimalSI},cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{101203873792 0} {<nil>} 98831908Ki BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{7815434240 0} {<nil>} 7632260Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{attachable-volumes-gce-pd: {{127 0} {<nil>} 127 DecimalSI},cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{91083486262 0} {<nil>} 91083486262 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{7553290240 0} {<nil>} 7376260Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:FrequentDockerRestart,Status:False,LastHeartbeatTime:2023-02-05 23:54:41 +0000 UTC,LastTransitionTime:2023-02-05 23:49:40 +0000 UTC,Reason:NoFrequentDockerRestart,Message:docker is functioning properly,},NodeCondition{Type:FrequentContainerdRestart,Status:False,LastHeartbeatTime:2023-02-05 23:54:41 +0000 UTC,LastTransitionTime:2023-02-05 23:49:40 +0000 UTC,Reason:NoFrequentContainerdRestart,Message:containerd is functioning properly,},NodeCondition{Type:CorruptDockerOverlay2,Status:False,LastHeartbeatTime:2023-02-05 23:54:41 +0000 UTC,LastTransitionTime:2023-02-05 23:49:40 +0000 UTC,Reason:NoCorruptDockerOverlay2,Message:docker overlay2 is functioning properly,},NodeCondition{Type:KernelDeadlock,Status:False,LastHeartbeatTime:2023-02-05 23:54:41 +0000 UTC,LastTransitionTime:2023-02-05 23:49:40 +0000 UTC,Reason:KernelHasNoDeadlock,Message:kernel has no deadlock,},NodeCondition{Type:ReadonlyFilesystem,Status:False,LastHeartbeatTime:2023-02-05 23:54:41 +0000 UTC,LastTransitionTime:2023-02-05 23:49:40 +0000 UTC,Reason:FilesystemIsNotReadOnly,Message:Filesystem is not read-only,},NodeCondition{Type:FrequentUnregisterNetDevice,Status:False,LastHeartbeatTime:2023-02-05 23:54:41 +0000 UTC,LastTransitionTime:2023-02-05 23:49:40 +0000 UTC,Reason:NoFrequentUnregisterNetDevice,Message:node is functioning properly,},NodeCondition{Type:FrequentKubeletRestart,Status:False,LastHeartbeatTime:2023-02-05 23:54:41 +0000 UTC,LastTransitionTime:2023-02-05 23:49:40 +0000 UTC,Reason:NoFrequentKubeletRestart,Message:kubelet is functioning properly,},NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2023-02-05 23:49:45 +0000 UTC,LastTransitionTime:2023-02-05 23:49:45 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-02-05 23:55:40 +0000 UTC,LastTransitionTime:2023-02-05 23:55:40 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-02-05 23:55:40 +0000 UTC,LastTransitionTime:2023-02-05 23:55:40 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-02-05 23:55:40 +0000 UTC,LastTransitionTime:2023-02-05 23:55:40 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-02-05 23:55:40 +0000 UTC,LastTransitionTime:2023-02-05 23:55:40 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:10.138.0.7,},NodeAddress{Type:ExternalIP,Address:34.118.193.103,},NodeAddress{Type:InternalDNS,Address:bootstrap-e2e-minion-group-7p0n.c.k8s-infra-e2e-boskos-001.internal,},NodeAddress{Type:Hostname,Address:bootstrap-e2e-minion-group-7p0n.c.k8s-infra-e2e-boskos-001.internal,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:a7360e0504037fe783766a874b55420f,SystemUUID:a7360e05-0403-7fe7-8376-6a874b55420f,BootID:21a9d9ec-9d9a-4d80-a6ba-4d65cdd17d89,KernelVersion:5.10.123+,OSImage:Container-Optimized OS from Google,ContainerRuntimeVersion:containerd://1.6.6,KubeletVersion:v1.27.0-alpha.1.249+cf14b50b0d2215,KubeProxyVersion:v1.27.0-alpha.1.249+cf14b50b0d2215,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[registry.k8s.io/cloud-provider-gcp/gcp-compute-persistent-disk-csi-driver@sha256:e9c355dd1ce57de91a0f9164bd2fe688938723e7f9bd40bb13652d603f41d25a registry.k8s.io/cloud-provider-gcp/gcp-compute-persistent-disk-csi-driver:v1.2.2],SizeBytes:73354282,},ContainerImage{Names:[registry.k8s.io/kube-proxy-amd64:v1.27.0-alpha.1.249_cf14b50b0d2215],SizeBytes:67583329,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:16bbf38c463a4223d8cfe4da12bc61010b082a79b4bb003e2d3ba3ece5dd5f9e registry.k8s.io/e2e-test-images/agnhost:2.43],SizeBytes:51706353,},ContainerImage{Names:[gke.gcr.io/prometheus-to-sd@sha256:e739643c3939ba0b161425f45a1989eedfc4a3b166db9a7100863296b4c70510 gke.gcr.io/prometheus-to-sd:v0.11.1-gke.1],SizeBytes:48742566,},ContainerImage{Names:[registry.k8s.io/cloud-provider-gcp/gcp-compute-persistent-disk-csi-driver@sha256:dc0301d9a43e23e389abb7690cd631f98af76ea3429259c7f5f643fd31fc0a3e registry.k8s.io/cloud-provider-gcp/gcp-compute-persistent-disk-csi-driver:v1.4.0],SizeBytes:27914152,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:ee3b525d5b89db99da3b8eb521d9cd90cb6e9ef0fbb651e98bb37be78d36b5b8 registry.k8s.io/sig-storage/csi-provisioner:v3.3.0],SizeBytes:25491225,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:425d8f1b769398127767b06ed97ce62578a3179bcb99809ce93a1649e025ffe7 registry.k8s.io/sig-storage/csi-resizer:v1.6.0],SizeBytes:24148884,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:9a685020911e2725ad019dbce6e4a5ab93d51e3d4557f115e64343345e05781b registry.k8s.io/sig-storage/csi-attacher:v4.0.0],SizeBytes:23847201,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:89e900a160a986a1a7a4eba7f5259e510398fa87ca9b8a729e7dec59e04c7709 registry.k8s.io/sig-storage/csi-snapshotter:v5.0.1],SizeBytes:22163966,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/kas-network-proxy/proxy-agent@sha256:939c42e815e6b6af3181f074652c0d18fe429fcee9b49c1392aee7e92887cfef registry.k8s.io/kas-network-proxy/proxy-agent:v0.1.1],SizeBytes:8364694,},ContainerImage{Names:[registry.k8s.io/metadata-proxy@sha256:e914645f22e946bce5165737e1b244e0a296ad1f0f81a9531adc57af2780978a registry.k8s.io/metadata-proxy:v0.1.12],SizeBytes:5301657,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:2e0f836850e09b8b7cc937681d6194537a09fbd5f6b9e08f4d646a85128e8937 registry.k8s.io/e2e-test-images/busybox:1.29-4],SizeBytes:731990,},ContainerImage{Names:[registry.k8s.io/pause@sha256:7031c1b283388d2c2e09b57badb803c05ebed362dc88d84b480cc47f72a21097 registry.k8s.io/pause:3.9],SizeBytes:321520,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},}
Feb  5 23:56:46.033: INFO: 
Logging kubelet events for node bootstrap-e2e-minion-group-7p0n
Feb  5 23:56:46.076: INFO: 
Logging pods the kubelet thinks is on node bootstrap-e2e-minion-group-7p0n
Feb  5 23:56:46.139: INFO: kube-proxy-bootstrap-e2e-minion-group-7p0n started at 2023-02-05 23:49:36 +0000 UTC (0+1 container statuses recorded)
Feb  5 23:56:46.139: INFO: 	Container kube-proxy ready: true, restart count 0
Feb  5 23:56:46.139: INFO: metadata-proxy-v0.1-2vz8w started at 2023-02-05 23:49:37 +0000 UTC (0+2 container statuses recorded)
Feb  5 23:56:46.139: INFO: 	Container metadata-proxy ready: true, restart count 0
Feb  5 23:56:46.139: INFO: 	Container prometheus-to-sd-exporter ready: true, restart count 0
Feb  5 23:56:46.139: INFO: konnectivity-agent-fljgx started at 2023-02-05 23:49:45 +0000 UTC (0+1 container statuses recorded)
Feb  5 23:56:46.139: INFO: 	Container konnectivity-agent ready: true, restart count 0
Feb  5 23:56:46.308: INFO: 
Latency metrics for node bootstrap-e2e-minion-group-7p0n
Feb  5 23:56:46.308: INFO: 
Logging node info for node bootstrap-e2e-minion-group-f2xv
Feb  5 23:56:46.399: INFO: Node Info: &Node{ObjectMeta:{bootstrap-e2e-minion-group-f2xv    e03f6c2e-c82f-4648-923b-3ca1f93c77cb 61865 0 2023-02-05 20:03:50 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:n1-standard-2 beta.kubernetes.io/os:linux cloud.google.com/metadata-proxy-ready:true failure-domain.beta.kubernetes.io/region:us-west1 failure-domain.beta.kubernetes.io/zone:us-west1-b kubernetes.io/arch:amd64 kubernetes.io/hostname:bootstrap-e2e-minion-group-f2xv kubernetes.io/os:linux node.kubernetes.io/instance-type:n1-standard-2 topology.gke.io/zone:us-west1-b topology.hostpath.csi/node:bootstrap-e2e-minion-group-f2xv topology.kubernetes.io/region:us-west1 topology.kubernetes.io/zone:us-west1-b] map[node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kubelet Update v1 2023-02-05 20:03:50 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:cloud.google.com/metadata-proxy-ready":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {kube-controller-manager Update v1 2023-02-05 23:19:26 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}}}} status} {kube-controller-manager Update v1 2023-02-05 23:44:07 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"10.64.0.0/24\"":{}}}} } {node-problem-detector Update v1 2023-02-05 23:53:48 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"CorruptDockerOverlay2\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"FrequentContainerdRestart\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"FrequentDockerRestart\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"FrequentKubeletRestart\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"FrequentUnregisterNetDevice\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"KernelDeadlock\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"ReadonlyFilesystem\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kubelet Update v1 2023-02-05 23:54:19 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:topology.gke.io/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:nodeInfo":{"f:bootID":{}}}} status}]},Spec:NodeSpec{PodCIDR:10.64.0.0/24,DoNotUseExternalID:,ProviderID:gce://k8s-infra-e2e-boskos-001/us-west1-b/bootstrap-e2e-minion-group-f2xv,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[10.64.0.0/24],},Status:NodeStatus{Capacity:ResourceList{attachable-volumes-gce-pd: {{127 0} {<nil>} 127 DecimalSI},cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{101203873792 0} {<nil>} 98831908Ki BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{7815426048 0} {<nil>} 7632252Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{attachable-volumes-gce-pd: {{127 0} {<nil>} 127 DecimalSI},cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{91083486262 0} {<nil>} 91083486262 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{7553282048 0} {<nil>} 7376252Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:CorruptDockerOverlay2,Status:False,LastHeartbeatTime:2023-02-05 23:53:48 +0000 UTC,LastTransitionTime:2023-02-05 22:58:40 +0000 UTC,Reason:NoCorruptDockerOverlay2,Message:docker overlay2 is functioning properly,},NodeCondition{Type:FrequentUnregisterNetDevice,Status:False,LastHeartbeatTime:2023-02-05 23:53:48 +0000 UTC,LastTransitionTime:2023-02-05 22:58:40 +0000 UTC,Reason:NoFrequentUnregisterNetDevice,Message:node is functioning properly,},NodeCondition{Type:FrequentKubeletRestart,Status:False,LastHeartbeatTime:2023-02-05 23:53:48 +0000 UTC,LastTransitionTime:2023-02-05 22:58:40 +0000 UTC,Reason:NoFrequentKubeletRestart,Message:kubelet is functioning properly,},NodeCondition{Type:FrequentDockerRestart,Status:False,LastHeartbeatTime:2023-02-05 23:53:48 +0000 UTC,LastTransitionTime:2023-02-05 22:58:40 +0000 UTC,Reason:NoFrequentDockerRestart,Message:docker is functioning properly,},NodeCondition{Type:FrequentContainerdRestart,Status:False,LastHeartbeatTime:2023-02-05 23:53:48 +0000 UTC,LastTransitionTime:2023-02-05 22:58:40 +0000 UTC,Reason:NoFrequentContainerdRestart,Message:containerd is functioning properly,},NodeCondition{Type:KernelDeadlock,Status:False,LastHeartbeatTime:2023-02-05 23:53:48 +0000 UTC,LastTransitionTime:2023-02-05 22:58:40 +0000 UTC,Reason:KernelHasNoDeadlock,Message:kernel has no deadlock,},NodeCondition{Type:ReadonlyFilesystem,Status:False,LastHeartbeatTime:2023-02-05 23:53:48 +0000 UTC,LastTransitionTime:2023-02-05 22:58:40 +0000 UTC,Reason:FilesystemIsNotReadOnly,Message:Filesystem is not read-only,},NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2023-02-05 20:03:58 +0000 UTC,LastTransitionTime:2023-02-05 20:03:58 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-02-05 23:54:19 +0000 UTC,LastTransitionTime:2023-02-05 23:19:30 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-02-05 23:54:19 +0000 UTC,LastTransitionTime:2023-02-05 23:19:30 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-02-05 23:54:19 +0000 UTC,LastTransitionTime:2023-02-05 23:19:30 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-02-05 23:54:19 +0000 UTC,LastTransitionTime:2023-02-05 23:44:07 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:10.138.0.5,},NodeAddress{Type:ExternalIP,Address:34.83.102.119,},NodeAddress{Type:InternalDNS,Address:bootstrap-e2e-minion-group-f2xv.c.k8s-infra-e2e-boskos-001.internal,},NodeAddress{Type:Hostname,Address:bootstrap-e2e-minion-group-f2xv.c.k8s-infra-e2e-boskos-001.internal,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:376621d36b55b947abdcfdac7e13ae34,SystemUUID:376621d3-6b55-b947-abdc-fdac7e13ae34,BootID:5bccce71-ce5b-47fa-9ed2-b6d61be88210,KernelVersion:5.10.123+,OSImage:Container-Optimized OS from Google,ContainerRuntimeVersion:containerd://1.6.6,KubeletVersion:v1.27.0-alpha.1.249+cf14b50b0d2215,KubeProxyVersion:v1.27.0-alpha.1.249+cf14b50b0d2215,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[registry.k8s.io/cloud-provider-gcp/gcp-compute-persistent-disk-csi-driver@sha256:e9c355dd1ce57de91a0f9164bd2fe688938723e7f9bd40bb13652d603f41d25a registry.k8s.io/cloud-provider-gcp/gcp-compute-persistent-disk-csi-driver:v1.2.2],SizeBytes:73354282,},ContainerImage{Names:[registry.k8s.io/kube-proxy-amd64:v1.27.0-alpha.1.249_cf14b50b0d2215],SizeBytes:67583329,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:16bbf38c463a4223d8cfe4da12bc61010b082a79b4bb003e2d3ba3ece5dd5f9e registry.k8s.io/e2e-test-images/agnhost:2.43],SizeBytes:51706353,},ContainerImage{Names:[gke.gcr.io/prometheus-to-sd@sha256:e739643c3939ba0b161425f45a1989eedfc4a3b166db9a7100863296b4c70510 gke.gcr.io/prometheus-to-sd:v0.11.1-gke.1],SizeBytes:48742566,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:148b022f5c5da426fc2f3c14b5c0867e58ef05961510c84749ac1fddcb0fef22 registry.k8s.io/e2e-test-images/httpd:2.4.38-4],SizeBytes:40764257,},ContainerImage{Names:[registry.k8s.io/metrics-server/metrics-server@sha256:6385aec64bb97040a5e692947107b81e178555c7a5b71caa90d733e4130efc10 registry.k8s.io/metrics-server/metrics-server:v0.5.2],SizeBytes:26023008,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:ee3b525d5b89db99da3b8eb521d9cd90cb6e9ef0fbb651e98bb37be78d36b5b8 registry.k8s.io/sig-storage/csi-provisioner:v3.3.0],SizeBytes:25491225,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:425d8f1b769398127767b06ed97ce62578a3179bcb99809ce93a1649e025ffe7 registry.k8s.io/sig-storage/csi-resizer:v1.6.0],SizeBytes:24148884,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:291334908ddf71a4661fd7f6d9d97274de8a5378a2b6fdfeb2ce73414a34f82f registry.k8s.io/sig-storage/csi-snapshotter:v6.1.0],SizeBytes:23881995,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:9a685020911e2725ad019dbce6e4a5ab93d51e3d4557f115e64343345e05781b registry.k8s.io/sig-storage/csi-attacher:v4.0.0],SizeBytes:23847201,},ContainerImage{Names:[registry.k8s.io/sig-storage/snapshot-controller@sha256:823c75d0c45d1427f6d850070956d9ca657140a7bbf828381541d1d808475280 registry.k8s.io/sig-storage/snapshot-controller:v6.1.0],SizeBytes:22620891,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:92257881c1d6493cf18299a24af42330f891166560047902b8d431fb66b01af5 registry.k8s.io/sig-storage/hostpathplugin:v1.9.0],SizeBytes:18758628,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:017727efcfeb7d053af68e51436ce8e65edbc6ca573720afb4f79c8594036955 registry.k8s.io/coredns/coredns:v1.10.0],SizeBytes:15273057,},ContainerImage{Names:[registry.k8s.io/cpa/cluster-proportional-autoscaler@sha256:fd636b33485c7826fb20ef0688a83ee0910317dbb6c0c6f3ad14661c1db25def registry.k8s.io/cpa/cluster-proportional-autoscaler:1.8.4],SizeBytes:15209393,},ContainerImage{Names:[registry.k8s.io/autoscaling/addon-resizer@sha256:43f129b81d28f0fdd54de6d8e7eacd5728030782e03db16087fc241ad747d3d6 registry.k8s.io/autoscaling/addon-resizer:1.8.14],SizeBytes:10153852,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:933940f13b3ea0abc62e656c1aa5c5b47c04b15d71250413a6b821bd0c58b94e registry.k8s.io/sig-storage/livenessprobe:v2.7.0],SizeBytes:8688564,},ContainerImage{Names:[registry.k8s.io/kas-network-proxy/proxy-agent@sha256:939c42e815e6b6af3181f074652c0d18fe429fcee9b49c1392aee7e92887cfef registry.k8s.io/kas-network-proxy/proxy-agent:v0.1.1],SizeBytes:8364694,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:5c99cf6a02adda929b10321dbf4ecfa00d87be9ba4fb456006237d530ab4baa1 registry.k8s.io/e2e-test-images/nginx:1.14-4],SizeBytes:6978614,},ContainerImage{Names:[registry.k8s.io/networking/ingress-gce-404-server-with-metrics-amd64@sha256:7eb7b3cee4d33c10c49893ad3c386232b86d4067de5251294d4c620d6e072b93 registry.k8s.io/networking/ingress-gce-404-server-with-metrics-amd64:v1.10.11],SizeBytes:6463068,},ContainerImage{Names:[registry.k8s.io/metadata-proxy@sha256:e914645f22e946bce5165737e1b244e0a296ad1f0f81a9531adc57af2780978a registry.k8s.io/metadata-proxy:v0.1.12],SizeBytes:5301657,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:2e0f836850e09b8b7cc937681d6194537a09fbd5f6b9e08f4d646a85128e8937 registry.k8s.io/e2e-test-images/busybox:1.29-4],SizeBytes:731990,},ContainerImage{Names:[registry.k8s.io/pause@sha256:7031c1b283388d2c2e09b57badb803c05ebed362dc88d84b480cc47f72a21097 registry.k8s.io/pause:3.9],SizeBytes:321520,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},}
Feb  5 23:56:46.399: INFO: 
Logging kubelet events for node bootstrap-e2e-minion-group-f2xv
Feb  5 23:56:46.443: INFO: 
Logging pods the kubelet thinks is on node bootstrap-e2e-minion-group-f2xv
Feb  5 23:56:46.511: INFO: kube-proxy-bootstrap-e2e-minion-group-f2xv started at 2023-02-05 20:03:51 +0000 UTC (0+1 container statuses recorded)
Feb  5 23:56:46.511: INFO: 	Container kube-proxy ready: true, restart count 1
Feb  5 23:56:46.511: INFO: l7-default-backend-8549d69d99-ptkh8 started at 2023-02-05 20:03:58 +0000 UTC (0+1 container statuses recorded)
Feb  5 23:56:46.511: INFO: 	Container default-http-backend ready: true, restart count 1
Feb  5 23:56:46.511: INFO: volume-snapshot-controller-0 started at 2023-02-05 20:03:58 +0000 UTC (0+1 container statuses recorded)
Feb  5 23:56:46.511: INFO: 	Container volume-snapshot-controller ready: true, restart count 1
Feb  5 23:56:46.511: INFO: kube-dns-autoscaler-5f6455f985-ddvmr started at 2023-02-05 20:03:58 +0000 UTC (0+1 container statuses recorded)
Feb  5 23:56:46.511: INFO: 	Container autoscaler ready: true, restart count 1
Feb  5 23:56:46.511: INFO: coredns-6846b5b5f-r8m7l started at 2023-02-05 23:48:50 +0000 UTC (0+1 container statuses recorded)
Feb  5 23:56:46.511: INFO: 	Container coredns ready: true, restart count 0
Feb  5 23:56:46.511: INFO: metadata-proxy-v0.1-46fnl started at 2023-02-05 20:03:51 +0000 UTC (0+2 container statuses recorded)
Feb  5 23:56:46.511: INFO: 	Container metadata-proxy ready: true, restart count 1
Feb  5 23:56:46.511: INFO: 	Container prometheus-to-sd-exporter ready: true, restart count 1
Feb  5 23:56:46.511: INFO: konnectivity-agent-tp4wv started at 2023-02-05 20:03:59 +0000 UTC (0+1 container statuses recorded)
Feb  5 23:56:46.511: INFO: 	Container konnectivity-agent ready: true, restart count 1
Feb  5 23:56:46.511: INFO: metrics-server-v0.5.2-867b8754b9-ss6lj started at 2023-02-05 20:09:11 +0000 UTC (0+2 container statuses recorded)
Feb  5 23:56:46.511: INFO: 	Container metrics-server ready: true, restart count 2
Feb  5 23:56:46.511: INFO: 	Container metrics-server-nanny ready: true, restart count 1
Feb  5 23:56:46.695: INFO: 
Latency metrics for node bootstrap-e2e-minion-group-f2xv
Feb  5 23:56:46.695: INFO: 
Logging node info for node bootstrap-e2e-minion-group-vc67
Feb  5 23:56:46.745: INFO: Node Info: &Node{ObjectMeta:{bootstrap-e2e-minion-group-vc67    d427fdfc-daa1-44d1-9847-d6ca5f6726b7 61838 0 2023-02-05 20:03:54 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:n1-standard-2 beta.kubernetes.io/os:linux cloud.google.com/metadata-proxy-ready:true failure-domain.beta.kubernetes.io/region:us-west1 failure-domain.beta.kubernetes.io/zone:us-west1-b kubernetes.io/arch:amd64 kubernetes.io/hostname:bootstrap-e2e-minion-group-vc67 kubernetes.io/os:linux node.kubernetes.io/instance-type:n1-standard-2 topology.gke.io/zone:us-west1-b topology.hostpath.csi/node:bootstrap-e2e-minion-group-vc67 topology.kubernetes.io/region:us-west1 topology.kubernetes.io/zone:us-west1-b] map[node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kubelet Update v1 2023-02-05 20:03:54 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:cloud.google.com/metadata-proxy-ready":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {e2e.test Update v1 2023-02-05 23:01:21 +0000 UTC FieldsV1 {"f:status":{"f:allocatable":{"f:example.com/fakePTSRes":{},"f:example.com/fakecpu":{}},"f:capacity":{"f:example.com/fakePTSRes":{},"f:example.com/fakecpu":{}}}} status} {node-problem-detector Update v1 2023-02-05 23:53:48 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"CorruptDockerOverlay2\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"FrequentContainerdRestart\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"FrequentDockerRestart\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"FrequentKubeletRestart\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"FrequentUnregisterNetDevice\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"KernelDeadlock\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"ReadonlyFilesystem\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kube-controller-manager Update v1 2023-02-05 23:53:52 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}}}} status} {kube-controller-manager Update v1 2023-02-05 23:53:55 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"10.64.1.0/24\"":{}}}} } {kubelet Update v1 2023-02-05 23:54:06 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:topology.gke.io/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{},"f:memory":{}},"f:capacity":{"f:memory":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:nodeInfo":{"f:bootID":{}}}} status}]},Spec:NodeSpec{PodCIDR:10.64.1.0/24,DoNotUseExternalID:,ProviderID:gce://k8s-infra-e2e-boskos-001/us-west1-b/bootstrap-e2e-minion-group-vc67,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[10.64.1.0/24],},Status:NodeStatus{Capacity:ResourceList{attachable-volumes-gce-pd: {{127 0} {<nil>} 127 DecimalSI},cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{101203873792 0} {<nil>} 98831908Ki BinarySI},example.com/fakePTSRes: {{10 0} {<nil>} 10 DecimalSI},example.com/fakecpu: {{1 3} {<nil>} 1k DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{7815434240 0} {<nil>} 7632260Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{attachable-volumes-gce-pd: {{127 0} {<nil>} 127 DecimalSI},cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{91083486262 0} {<nil>} 91083486262 DecimalSI},example.com/fakePTSRes: {{10 0} {<nil>} 10 DecimalSI},example.com/fakecpu: {{1 3} {<nil>} 1k DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{7553290240 0} {<nil>} 7376260Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:CorruptDockerOverlay2,Status:False,LastHeartbeatTime:2023-02-05 23:53:48 +0000 UTC,LastTransitionTime:2023-02-05 22:58:39 +0000 UTC,Reason:NoCorruptDockerOverlay2,Message:docker overlay2 is functioning properly,},NodeCondition{Type:FrequentUnregisterNetDevice,Status:False,LastHeartbeatTime:2023-02-05 23:53:48 +0000 UTC,LastTransitionTime:2023-02-05 22:58:39 +0000 UTC,Reason:NoFrequentUnregisterNetDevice,Message:node is functioning properly,},NodeCondition{Type:FrequentKubeletRestart,Status:False,LastHeartbeatTime:2023-02-05 23:53:48 +0000 UTC,LastTransitionTime:2023-02-05 22:58:39 +0000 UTC,Reason:NoFrequentKubeletRestart,Message:kubelet is functioning properly,},NodeCondition{Type:FrequentDockerRestart,Status:False,LastHeartbeatTime:2023-02-05 23:53:48 +0000 UTC,LastTransitionTime:2023-02-05 22:58:39 +0000 UTC,Reason:NoFrequentDockerRestart,Message:docker is functioning properly,},NodeCondition{Type:FrequentContainerdRestart,Status:False,LastHeartbeatTime:2023-02-05 23:53:48 +0000 UTC,LastTransitionTime:2023-02-05 22:58:39 +0000 UTC,Reason:NoFrequentContainerdRestart,Message:containerd is functioning properly,},NodeCondition{Type:KernelDeadlock,Status:False,LastHeartbeatTime:2023-02-05 23:53:48 +0000 UTC,LastTransitionTime:2023-02-05 22:58:39 +0000 UTC,Reason:KernelHasNoDeadlock,Message:kernel has no deadlock,},NodeCondition{Type:ReadonlyFilesystem,Status:False,LastHeartbeatTime:2023-02-05 23:53:48 +0000 UTC,LastTransitionTime:2023-02-05 22:58:39 +0000 UTC,Reason:FilesystemIsNotReadOnly,Message:Filesystem is not read-only,},NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2023-02-05 20:04:10 +0000 UTC,LastTransitionTime:2023-02-05 20:04:10 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-02-05 23:54:06 +0000 UTC,LastTransitionTime:2023-02-05 23:53:55 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-02-05 23:54:06 +0000 UTC,LastTransitionTime:2023-02-05 23:53:55 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-02-05 23:54:06 +0000 UTC,LastTransitionTime:2023-02-05 23:53:55 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-02-05 23:54:06 +0000 UTC,LastTransitionTime:2023-02-05 23:53:55 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:10.138.0.3,},NodeAddress{Type:ExternalIP,Address:34.145.100.19,},NodeAddress{Type:InternalDNS,Address:bootstrap-e2e-minion-group-vc67.c.k8s-infra-e2e-boskos-001.internal,},NodeAddress{Type:Hostname,Address:bootstrap-e2e-minion-group-vc67.c.k8s-infra-e2e-boskos-001.internal,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:a65546ee19f4e3edcf02f690a7396782,SystemUUID:a65546ee-19f4-e3ed-cf02-f690a7396782,BootID:c872f346-7124-458f-a158-1ef112f88295,KernelVersion:5.10.123+,OSImage:Container-Optimized OS from Google,ContainerRuntimeVersion:containerd://1.6.6,KubeletVersion:v1.27.0-alpha.1.249+cf14b50b0d2215,KubeProxyVersion:v1.27.0-alpha.1.249+cf14b50b0d2215,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[registry.k8s.io/e2e-test-images/jessie-dnsutils@sha256:24aaf2626d6b27864c29de2097e8bbb840b3a414271bf7c8995e431e47d8408e registry.k8s.io/e2e-test-images/jessie-dnsutils:1.7],SizeBytes:112030336,},ContainerImage{Names:[registry.k8s.io/sig-storage/nfs-provisioner@sha256:e943bb77c7df05ebdc8c7888b2db289b13bf9f012d6a3a5a74f14d4d5743d439 registry.k8s.io/sig-storage/nfs-provisioner:v3.0.1],SizeBytes:90632047,},ContainerImage{Names:[registry.k8s.io/cloud-provider-gcp/gcp-compute-persistent-disk-csi-driver@sha256:e9c355dd1ce57de91a0f9164bd2fe688938723e7f9bd40bb13652d603f41d25a registry.k8s.io/cloud-provider-gcp/gcp-compute-persistent-disk-csi-driver:v1.2.2],SizeBytes:73354282,},ContainerImage{Names:[registry.k8s.io/kube-proxy-amd64:v1.27.0-alpha.1.249_cf14b50b0d2215],SizeBytes:67583329,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:16bbf38c463a4223d8cfe4da12bc61010b082a79b4bb003e2d3ba3ece5dd5f9e registry.k8s.io/e2e-test-images/agnhost:2.43],SizeBytes:51706353,},ContainerImage{Names:[gke.gcr.io/prometheus-to-sd@sha256:e739643c3939ba0b161425f45a1989eedfc4a3b166db9a7100863296b4c70510 gke.gcr.io/prometheus-to-sd:v0.11.1-gke.1],SizeBytes:48742566,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:148b022f5c5da426fc2f3c14b5c0867e58ef05961510c84749ac1fddcb0fef22 registry.k8s.io/e2e-test-images/httpd:2.4.38-4],SizeBytes:40764257,},ContainerImage{Names:[registry.k8s.io/sig-storage/volume-data-source-validator@sha256:d35884236461ad14397194cb4abc13aefb28a54bf3f6698d673c2ba353e78e18 registry.k8s.io/sig-storage/volume-data-source-validator:v1.0.0],SizeBytes:28662933,},ContainerImage{Names:[registry.k8s.io/cloud-provider-gcp/gcp-compute-persistent-disk-csi-driver@sha256:dc0301d9a43e23e389abb7690cd631f98af76ea3429259c7f5f643fd31fc0a3e registry.k8s.io/cloud-provider-gcp/gcp-compute-persistent-disk-csi-driver:v1.4.0],SizeBytes:27914152,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:ee3b525d5b89db99da3b8eb521d9cd90cb6e9ef0fbb651e98bb37be78d36b5b8 registry.k8s.io/sig-storage/csi-provisioner:v3.3.0],SizeBytes:25491225,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:425d8f1b769398127767b06ed97ce62578a3179bcb99809ce93a1649e025ffe7 registry.k8s.io/sig-storage/csi-resizer:v1.6.0],SizeBytes:24148884,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:291334908ddf71a4661fd7f6d9d97274de8a5378a2b6fdfeb2ce73414a34f82f registry.k8s.io/sig-storage/csi-snapshotter:v6.1.0],SizeBytes:23881995,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:9a685020911e2725ad019dbce6e4a5ab93d51e3d4557f115e64343345e05781b registry.k8s.io/sig-storage/csi-attacher:v4.0.0],SizeBytes:23847201,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:89e900a160a986a1a7a4eba7f5259e510398fa87ca9b8a729e7dec59e04c7709 registry.k8s.io/sig-storage/csi-snapshotter:v5.0.1],SizeBytes:22163966,},ContainerImage{Names:[registry.k8s.io/sig-storage/hello-populator@sha256:42d275bc2569b00a11c8058e5b77c50cdee39bf75d80c912b80813f03e7fd7d1 registry.k8s.io/sig-storage/hello-populator:v1.0.1],SizeBytes:20511089,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:92257881c1d6493cf18299a24af42330f891166560047902b8d431fb66b01af5 registry.k8s.io/sig-storage/hostpathplugin:v1.9.0],SizeBytes:18758628,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:017727efcfeb7d053af68e51436ce8e65edbc6ca573720afb4f79c8594036955 registry.k8s.io/coredns/coredns:v1.10.0],SizeBytes:15273057,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:933940f13b3ea0abc62e656c1aa5c5b47c04b15d71250413a6b821bd0c58b94e registry.k8s.io/sig-storage/livenessprobe:v2.7.0],SizeBytes:8688564,},ContainerImage{Names:[registry.k8s.io/kas-network-proxy/proxy-agent@sha256:939c42e815e6b6af3181f074652c0d18fe429fcee9b49c1392aee7e92887cfef registry.k8s.io/kas-network-proxy/proxy-agent:v0.1.1],SizeBytes:8364694,},ContainerImage{Names:[registry.k8s.io/metadata-proxy@sha256:e914645f22e946bce5165737e1b244e0a296ad1f0f81a9531adc57af2780978a registry.k8s.io/metadata-proxy:v0.1.12],SizeBytes:5301657,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:2e0f836850e09b8b7cc937681d6194537a09fbd5f6b9e08f4d646a85128e8937 registry.k8s.io/e2e-test-images/busybox:1.29-4],SizeBytes:731990,},ContainerImage{Names:[registry.k8s.io/pause@sha256:7031c1b283388d2c2e09b57badb803c05ebed362dc88d84b480cc47f72a21097 registry.k8s.io/pause:3.9],SizeBytes:321520,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},}
Feb  5 23:56:46.745: INFO: 
Logging kubelet events for node bootstrap-e2e-minion-group-vc67
Feb  5 23:56:46.791: INFO: 
Logging pods the kubelet thinks is on node bootstrap-e2e-minion-group-vc67
Feb  5 23:56:46.856: INFO: kube-proxy-bootstrap-e2e-minion-group-vc67 started at 2023-02-05 20:03:54 +0000 UTC (0+1 container statuses recorded)
Feb  5 23:56:46.856: INFO: 	Container kube-proxy ready: true, restart count 1
Feb  5 23:56:46.856: INFO: metadata-proxy-v0.1-kjcsh started at 2023-02-05 20:03:55 +0000 UTC (0+2 container statuses recorded)
Feb  5 23:56:46.856: INFO: 	Container metadata-proxy ready: true, restart count 1
Feb  5 23:56:46.856: INFO: 	Container prometheus-to-sd-exporter ready: true, restart count 1
Feb  5 23:56:46.856: INFO: konnectivity-agent-tmn2w started at 2023-02-05 20:04:10 +0000 UTC (0+1 container statuses recorded)
Feb  5 23:56:46.856: INFO: 	Container konnectivity-agent ready: true, restart count 1
Feb  5 23:56:46.856: INFO: coredns-6846b5b5f-94td7 started at 2023-02-05 23:44:58 +0000 UTC (0+1 container statuses recorded)
Feb  5 23:56:46.856: INFO: 	Container coredns ready: true, restart count 0
Feb  5 23:56:47.024: INFO: 
Latency metrics for node bootstrap-e2e-minion-group-vc67
END STEP: dump namespace information after failure - test/e2e/framework/framework.go:284 @ 02/05/23 23:56:47.024 (1.524s)
< Exit [DeferCleanup (Each)] [Testpattern: Pre-provisioned PV (default fs)] subPath - dump namespaces | framework.go:206 @ 02/05/23 23:56:47.024 (1.525s)
> Enter [DeferCleanup (Each)] [Testpattern: Pre-provisioned PV (default fs)] subPath - tear down framework | framework.go:203 @ 02/05/23 23:56:47.024
STEP: Destroying namespace "provisioning-9333" for this suite. - test/e2e/framework/framework.go:347 @ 02/05/23 23:56:47.024
< Exit [DeferCleanup (Each)] [Testpattern: Pre-provisioned PV (default fs)] subPath - tear down framework | framework.go:203 @ 02/05/23 23:56:47.067 (43ms)
> Enter [ReportAfterEach] TOP-LEVEL - test/e2e/e2e_test.go:144 @ 02/05/23 23:56:47.067
< Exit [ReportAfterEach] TOP-LEVEL - test/e2e/e2e_test.go:144 @ 02/05/23 23:56:47.067 (0s)

				Click to see stdout/stderrfrom junit_01.xml

Filter through log files | View test history on testgrid

kubetest Test 5h5m

error during ./hack/ginkgo-e2e.sh --ginkgo.focus=\[Serial\]|\[Disruptive\] --ginkgo.skip=\[Driver:.gcepd\]|\[Flaky\]|\[Feature:.+\] --minStartupPods=8 --report-dir=/workspace/_artifacts --disable-log-dump=true: exit status 1
				from junit_runner.xml

Filter through log files | View test history on testgrid

Show 13708 Skipped Tests

Result	FAILURE
Tests	3 failed / 486 succeeded
Started	2023-02-05 19:58
Elapsed	5h26m
Revision
Builder	6e8e8674-a58f-11ed-987c-c2f4825a9650
control_plane_node_os_image	cos-97-16919-103-16
infra-commit	1a18f0ced
job-version	v1.27.0-alpha.1.249+cf14b50b0d2215
kubetest-version	v20230127-9396ca613c
revision	v1.27.0-alpha.1.249+cf14b50b0d2215
worker_node_os_image	cos-97-16919-103-16

ci-kubernetes-e2e-gci-gce-serial #1622323746830290944

Test Failures

Kubernetes e2e suite [It] [sig-storage] In-tree Volumes [Driver: local][LocalVolumeType: dir] [Testpattern: Pre-provisioned PV (default fs)] subPath should unmount if pod is gracefully deleted while kubelet is down [Disruptive][Slow][LinuxOnly] 2m15s

Kubernetes e2e suite [It] [sig-storage] In-tree Volumes [Driver: local][LocalVolumeType: dir] [Testpattern: Pre-provisioned PV (default fs)] subPath should unmount if pod is gracefully deleted while kubelet is down [Disruptive][Slow][LinuxOnly] 2m15s

kubetest Test 5h5m

Passed

Skipped