Result | FAILURE |
Tests | 18 failed / 776 succeeded |
Started | |
Elapsed | 1h45m |
Revision | master |
go run hack/e2e.go -v --test --test_args='--ginkgo.focus=Kubernetes\se2e\ssuite\s\[It\]\sExternal\sStorage\s\[Driver\:\sebs\.csi\.aws\.com\]\s\[Testpattern\:\sDynamic\sPV\s\(default\sfs\)\]\sfsgroupchangepolicy\s\(OnRootMismatch\)\[LinuxOnly\]\,\spod\screated\swith\san\sinitial\sfsgroup\,\svolume\scontents\sownership\schanged\svia\schgrp\sin\sfirst\spod\,\snew\spod\swith\sdifferent\sfsgroup\sapplied\sto\sthe\svolume\scontents$'
[FAILED] pod "pod-c7a5ea42-8a55-4f11-bbe2-e4757a42cdda" is not Running: Timed out after 300.000s. Expected Pod to be in <v1.PodPhase>: "Running" Got instead: <*v1.Pod | 0xc002ec3b00>: metadata: creationTimestamp: "2023-03-11T16:44:24Z" managedFields: - apiVersion: v1 fieldsType: FieldsV1 fieldsV1: f:spec: f:containers: k:{"name":"write-pod"}: .: {} f:command: {} f:image: {} f:imagePullPolicy: {} f:name: {} f:resources: {} f:securityContext: .: {} f:privileged: {} f:terminationMessagePath: {} f:terminationMessagePolicy: {} f:volumeMounts: .: {} k:{"mountPath":"/mnt/volume1"}: .: {} f:mountPath: {} f:name: {} f:dnsPolicy: {} f:enableServiceLinks: {} f:nodeSelector: {} f:restartPolicy: {} f:schedulerName: {} f:securityContext: .: {} f:fsGroup: {} f:fsGroupChangePolicy: {} f:terminationGracePeriodSeconds: {} f:volumes: .: {} k:{"name":"volume1"}: .: {} f:name: {} f:persistentVolumeClaim: .: {} f:claimName: {} manager: e2e.test operation: Update time: "2023-03-11T16:44:24Z" - apiVersion: v1 fieldsType: FieldsV1 fieldsV1: f:status: f:conditions: k:{"type":"ContainersReady"}: .: {} f:lastProbeTime: {} f:lastTransitionTime: {} f:message: {} f:reason: {} f:status: {} f:type: {} k:{"type":"Initialized"}: .: {} f:lastProbeTime: {} f:lastTransitionTime: {} f:status: {} f:type: {} k:{"type":"Ready"}: .: {} f:lastProbeTime: {} f:lastTransitionTime: {} f:message: {} f:reason: {} f:status: {} f:type: {} f:containerStatuses: {} f:hostIP: {} f:startTime: {} manager: kubelet operation: Update subresource: status time: "2023-03-11T16:44:28Z" name: pod-c7a5ea42-8a55-4f11-bbe2-e4757a42cdda namespace: fsgroupchangepolicy-6906 resourceVersion: "46217" uid: dd9b3f1f-1f84-42ad-b82c-c9817ed4eee3 spec: containers: - command: - /bin/sh - -c - trap exit TERM; while true; do sleep 1; done image: registry.k8s.io/e2e-test-images/busybox:1.29-4 imagePullPolicy: IfNotPresent name: write-pod resources: {} securityContext: privileged: false terminationMessagePath: /dev/termination-log terminationMessagePolicy: File volumeMounts: - mountPath: /mnt/volume1 name: volume1 - mountPath: /var/run/secrets/kubernetes.io/serviceaccount name: kube-api-access-svfpv readOnly: true dnsPolicy: ClusterFirst enableServiceLinks: true nodeName: i-04523d1dffb0fe725 nodeSelector: kubernetes.io/os: linux preemptionPolicy: PreemptLowerPriority priority: 0 restartPolicy: OnFailure schedulerName: default-scheduler securityContext: fsGroup: 1000 fsGroupChangePolicy: OnRootMismatch serviceAccount: default serviceAccountName: default terminationGracePeriodSeconds: 30 tolerations: - effect: NoExecute key: node.kubernetes.io/not-ready operator: Exists tolerationSeconds: 300 - effect: NoExecute key: node.kubernetes.io/unreachable operator: Exists tolerationSeconds: 300 volumes: - name: volume1 persistentVolumeClaim: claimName: ebs.csi.aws.comzvh2k - name: kube-api-access-svfpv projected: defaultMode: 420 sources: - serviceAccountToken: expirationSeconds: 3607 path: token - configMap: items: - key: ca.crt path: ca.crt name: kube-root-ca.crt - downwardAPI: items: - fieldRef: apiVersion: v1 fieldPath: metadata.namespace path: namespace status: conditions: - lastProbeTime: null lastTransitionTime: "2023-03-11T16:44:28Z" status: "True" type: Initialized - lastProbeTime: null lastTransitionTime: "2023-03-11T16:44:28Z" message: 'containers with unready status: [write-pod]' reason: ContainersNotReady status: "False" type: Ready - lastProbeTime: null lastTransitionTime: "2023-03-11T16:44:28Z" message: 'containers with unready status: [write-pod]' reason: ContainersNotReady status: "False" type: ContainersReady - lastProbeTime: null lastTransitionTime: "2023-03-11T16:44:28Z" status: "True" type: PodScheduled containerStatuses: - image: registry.k8s.io/e2e-test-images/busybox:1.29-4 imageID: "" lastState: {} name: write-pod ready: false restartCount: 0 started: false state: waiting: reason: ContainerCreating hostIP: 172.20.48.168 phase: Pending qosClass: BestEffort startTime: "2023-03-11T16:44:28Z" In [It] at: test/e2e/storage/testsuites/fsgroupchangepolicy.go:259 @ 03/11/23 16:49:24.959 There were additional failures detected after the initial failure. These are visible in the timelinefrom junit_01.xml
> Enter [BeforeEach] [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy - test/e2e/storage/framework/testsuite.go:51 @ 03/11/23 16:44:23.276 < Exit [BeforeEach] [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy - test/e2e/storage/framework/testsuite.go:51 @ 03/11/23 16:44:23.276 (0s) > Enter [BeforeEach] [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy - set up framework | framework.go:191 @ 03/11/23 16:44:23.276 STEP: Creating a kubernetes client - test/e2e/framework/framework.go:211 @ 03/11/23 16:44:23.276 Mar 11 16:44:23.276: INFO: >>> kubeConfig: /root/.kube/config STEP: Building a namespace api object, basename fsgroupchangepolicy - test/e2e/framework/framework.go:250 @ 03/11/23 16:44:23.277 STEP: Waiting for a default service account to be provisioned in namespace - test/e2e/framework/framework.go:259 @ 03/11/23 16:44:23.733 STEP: Waiting for kube-root-ca.crt to be provisioned in namespace - test/e2e/framework/framework.go:262 @ 03/11/23 16:44:24.031 < Exit [BeforeEach] [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy - set up framework | framework.go:191 @ 03/11/23 16:44:24.327 (1.051s) > Enter [BeforeEach] [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy - test/e2e/framework/metrics/init/init.go:33 @ 03/11/23 16:44:24.327 < Exit [BeforeEach] [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy - test/e2e/framework/metrics/init/init.go:33 @ 03/11/23 16:44:24.327 (0s) > Enter [It] (OnRootMismatch)[LinuxOnly], pod created with an initial fsgroup, volume contents ownership changed via chgrp in first pod, new pod with different fsgroup applied to the volume contents - test/e2e/storage/testsuites/fsgroupchangepolicy.go:211 @ 03/11/23 16:44:24.327 Mar 11 16:44:24.327: INFO: Creating resource for dynamic PV Mar 11 16:44:24.327: INFO: Using claimSize:1Gi, test suite supported size:{ 1Mi}, driver(ebs.csi.aws.com) supported size:{ 1Mi} STEP: creating a StorageClass fsgroupchangepolicy-6906-e2e-scjc7nr - test/e2e/storage/framework/volume_resource.go:102 @ 03/11/23 16:44:24.327 STEP: creating a claim - test/e2e/storage/framework/volume_resource.go:294 @ 03/11/23 16:44:24.481 Mar 11 16:44:24.481: INFO: Warning: Making PVC: VolumeMode specified as invalid empty string, treating as nil STEP: Creating Pod in namespace fsgroupchangepolicy-6906 with fsgroup 1000 - test/e2e/storage/testsuites/fsgroupchangepolicy.go:257 @ 03/11/23 16:44:24.799 Mar 11 16:49:24.959: INFO: Failed inside E2E framework: k8s.io/kubernetes/test/e2e/framework/pod.WaitTimeoutForPodRunningInNamespace({0x7f08e1b4c1f8, 0xc000f7c100}, {0x722adb8?, 0xc0029bf040?}, {0xc003be9350, 0x28}, {0xc0037dfea8, 0x18}, 0x0?) test/e2e/framework/pod/wait.go:460 +0x1a4 k8s.io/kubernetes/test/e2e/framework/pod.CreateSecPodWithNodeSelection({0x7f08e1b4c1f8, 0xc000f7c100}, {0x722adb8?, 0xc0029bf040?}, 0xc002457e00, 0x2?) test/e2e/framework/pod/create.go:117 +0x1fc k8s.io/kubernetes/test/e2e/storage/testsuites.createPodAndVerifyContentGid({0x7f08e1b4c1f8, 0xc000f7c100}, 0xc0016f6f00, 0xc0017c3e00, 0x1, {0x0, 0x0}, {0x0, 0x0}) test/e2e/storage/testsuites/fsgroupchangepolicy.go:258 +0x165 k8s.io/kubernetes/test/e2e/storage/testsuites.(*fsGroupChangePolicyTestSuite).DefineTests.func3({0x7f08e1b4c1f8, 0xc000f7c100}) test/e2e/storage/testsuites/fsgroupchangepolicy.go:230 +0x2eb [FAILED] pod "pod-c7a5ea42-8a55-4f11-bbe2-e4757a42cdda" is not Running: Timed out after 300.000s. Expected Pod to be in <v1.PodPhase>: "Running" Got instead: <*v1.Pod | 0xc002ec3b00>: metadata: creationTimestamp: "2023-03-11T16:44:24Z" managedFields: - apiVersion: v1 fieldsType: FieldsV1 fieldsV1: f:spec: f:containers: k:{"name":"write-pod"}: .: {} f:command: {} f:image: {} f:imagePullPolicy: {} f:name: {} f:resources: {} f:securityContext: .: {} f:privileged: {} f:terminationMessagePath: {} f:terminationMessagePolicy: {} f:volumeMounts: .: {} k:{"mountPath":"/mnt/volume1"}: .: {} f:mountPath: {} f:name: {} f:dnsPolicy: {} f:enableServiceLinks: {} f:nodeSelector: {} f:restartPolicy: {} f:schedulerName: {} f:securityContext: .: {} f:fsGroup: {} f:fsGroupChangePolicy: {} f:terminationGracePeriodSeconds: {} f:volumes: .: {} k:{"name":"volume1"}: .: {} f:name: {} f:persistentVolumeClaim: .: {} f:claimName: {} manager: e2e.test operation: Update time: "2023-03-11T16:44:24Z" - apiVersion: v1 fieldsType: FieldsV1 fieldsV1: f:status: f:conditions: k:{"type":"ContainersReady"}: .: {} f:lastProbeTime: {} f:lastTransitionTime: {} f:message: {} f:reason: {} f:status: {} f:type: {} k:{"type":"Initialized"}: .: {} f:lastProbeTime: {} f:lastTransitionTime: {} f:status: {} f:type: {} k:{"type":"Ready"}: .: {} f:lastProbeTime: {} f:lastTransitionTime: {} f:message: {} f:reason: {} f:status: {} f:type: {} f:containerStatuses: {} f:hostIP: {} f:startTime: {} manager: kubelet operation: Update subresource: status time: "2023-03-11T16:44:28Z" name: pod-c7a5ea42-8a55-4f11-bbe2-e4757a42cdda namespace: fsgroupchangepolicy-6906 resourceVersion: "46217" uid: dd9b3f1f-1f84-42ad-b82c-c9817ed4eee3 spec: containers: - command: - /bin/sh - -c - trap exit TERM; while true; do sleep 1; done image: registry.k8s.io/e2e-test-images/busybox:1.29-4 imagePullPolicy: IfNotPresent name: write-pod resources: {} securityContext: privileged: false terminationMessagePath: /dev/termination-log terminationMessagePolicy: File volumeMounts: - mountPath: /mnt/volume1 name: volume1 - mountPath: /var/run/secrets/kubernetes.io/serviceaccount name: kube-api-access-svfpv readOnly: true dnsPolicy: ClusterFirst enableServiceLinks: true nodeName: i-04523d1dffb0fe725 nodeSelector: kubernetes.io/os: linux preemptionPolicy: PreemptLowerPriority priority: 0 restartPolicy: OnFailure schedulerName: default-scheduler securityContext: fsGroup: 1000 fsGroupChangePolicy: OnRootMismatch serviceAccount: default serviceAccountName: default terminationGracePeriodSeconds: 30 tolerations: - effect: NoExecute key: node.kubernetes.io/not-ready operator: Exists tolerationSeconds: 300 - effect: NoExecute key: node.kubernetes.io/unreachable operator: Exists tolerationSeconds: 300 volumes: - name: volume1 persistentVolumeClaim: claimName: ebs.csi.aws.comzvh2k - name: kube-api-access-svfpv projected: defaultMode: 420 sources: - serviceAccountToken: expirationSeconds: 3607 path: token - configMap: items: - key: ca.crt path: ca.crt name: kube-root-ca.crt - downwardAPI: items: - fieldRef: apiVersion: v1 fieldPath: metadata.namespace path: namespace status: conditions: - lastProbeTime: null lastTransitionTime: "2023-03-11T16:44:28Z" status: "True" type: Initialized - lastProbeTime: null lastTransitionTime: "2023-03-11T16:44:28Z" message: 'containers with unready status: [write-pod]' reason: ContainersNotReady status: "False" type: Ready - lastProbeTime: null lastTransitionTime: "2023-03-11T16:44:28Z" message: 'containers with unready status: [write-pod]' reason: ContainersNotReady status: "False" type: ContainersReady - lastProbeTime: null lastTransitionTime: "2023-03-11T16:44:28Z" status: "True" type: PodScheduled containerStatuses: - image: registry.k8s.io/e2e-test-images/busybox:1.29-4 imageID: "" lastState: {} name: write-pod ready: false restartCount: 0 started: false state: waiting: reason: ContainerCreating hostIP: 172.20.48.168 phase: Pending qosClass: BestEffort startTime: "2023-03-11T16:44:28Z" In [It] at: test/e2e/storage/testsuites/fsgroupchangepolicy.go:259 @ 03/11/23 16:49:24.959 < Exit [It] (OnRootMismatch)[LinuxOnly], pod created with an initial fsgroup, volume contents ownership changed via chgrp in first pod, new pod with different fsgroup applied to the volume contents - test/e2e/storage/testsuites/fsgroupchangepolicy.go:211 @ 03/11/23 16:49:24.959 (5m0.633s) > Enter [AfterEach] [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy - test/e2e/framework/node/init/init.go:33 @ 03/11/23 16:49:24.959 Mar 11 16:49:24.959: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready Mar 11 16:49:25.109: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:49:25.109: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:49:27.259: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:49:27.259: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:49:29.259: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:49:29.259: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:49:31.260: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:49:31.260: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:49:33.259: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:49:33.259: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:49:35.259: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:49:35.259: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:49:37.260: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:49:37.260: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:49:39.259: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:49:39.259: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:49:41.260: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:49:41.260: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:49:43.259: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:49:43.259: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:49:45.259: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:49:45.259: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:49:47.260: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:49:47.260: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:49:49.260: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:49:49.260: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:49:51.260: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:49:51.260: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:49:53.260: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:49:53.260: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:49:55.259: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:49:55.259: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:49:57.259: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:49:57.259: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:49:59.259: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:49:59.259: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:50:01.260: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:50:01.260: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:50:03.259: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:50:03.259: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:50:05.259: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:50:05.259: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:50:07.259: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:50:07.259: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:50:09.260: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:50:09.260: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:50:11.260: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:50:11.260: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:50:13.259: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:50:13.259: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:50:15.260: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:50:15.260: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:50:17.262: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:50:17.262: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:50:19.260: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:50:19.260: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:50:21.260: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:50:21.260: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:50:23.259: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:50:23.259: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:50:25.259: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:50:25.259: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:50:27.260: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:50:27.260: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:50:29.260: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:50:29.260: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:50:31.259: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:50:31.259: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:50:33.259: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:50:33.259: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:50:35.259: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:50:35.259: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:50:37.260: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:50:37.260: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:50:39.260: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:50:39.260: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:50:41.260: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:50:41.260: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:50:43.259: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:50:43.259: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:50:45.259: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:50:45.259: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:50:47.260: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:50:47.260: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:50:49.259: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:50:49.259: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:50:51.259: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:50:51.259: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:50:53.260: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:50:53.260: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:50:55.259: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:50:55.259: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:50:57.259: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:50:57.259: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:50:59.259: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:50:59.259: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:51:01.260: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:51:01.260: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:51:03.260: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:51:03.260: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:51:05.259: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:51:05.259: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:51:07.260: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:51:07.260: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:51:09.259: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:51:09.259: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:51:11.259: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:51:11.259: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:51:13.260: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:51:13.260: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:51:15.260: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:51:15.260: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:51:17.260: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:51:17.260: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:51:19.260: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:51:19.260: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:51:21.260: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:51:21.260: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:51:23.259: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:51:23.260: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:51:25.259: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:51:25.260: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:51:27.262: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:51:27.262: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:51:29.260: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:51:29.260: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:51:31.260: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:51:31.260: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:51:33.260: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:51:33.260: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:51:35.260: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:51:35.260: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:51:37.259: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:51:37.259: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:51:39.260: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:51:39.260: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:51:41.260: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:51:41.260: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:51:43.260: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:51:43.260: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:51:45.260: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:51:45.260: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:51:47.260: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:51:47.260: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:51:49.260: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:51:49.260: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:51:51.260: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:51:51.260: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:51:53.259: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:51:53.259: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:51:55.259: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:51:55.259: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:51:57.260: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:51:57.260: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:51:59.260: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:51:59.260: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:52:01.260: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:52:01.260: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:52:03.260: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:52:03.260: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:52:05.260: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:52:05.260: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:52:07.260: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:52:07.260: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:52:09.260: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:52:09.260: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:52:11.259: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:52:11.259: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:52:13.259: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:52:13.259: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:52:15.259: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:52:15.259: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:52:17.259: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:52:17.259: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:52:19.259: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:52:19.259: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:52:21.259: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:52:21.259: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:52:23.260: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:52:23.260: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:52:25.259: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:52:25.259: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:52:25.408: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:52:25.408: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure < Exit [AfterEach] [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy - test/e2e/framework/node/init/init.go:33 @ 03/11/23 16:52:25.408 (3m0.449s) > Enter [DeferCleanup (Each)] [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy - test/e2e/storage/testsuites/fsgroupchangepolicy.go:221 @ 03/11/23 16:52:25.409 STEP: Deleting pvc - test/e2e/storage/framework/volume_resource.go:181 @ 03/11/23 16:52:25.409 Mar 11 16:52:25.706: INFO: Deleting PersistentVolumeClaim "ebs.csi.aws.comzvh2k" Mar 11 16:52:25.857: INFO: Waiting up to 5m0s for PersistentVolume pvc-dbb9b83e-52bb-4231-a392-5cddc1d510e5 to get deleted Mar 11 16:52:26.005: INFO: PersistentVolume pvc-dbb9b83e-52bb-4231-a392-5cddc1d510e5 found and phase=Bound (148.717888ms) Mar 11 16:52:31.154: INFO: PersistentVolume pvc-dbb9b83e-52bb-4231-a392-5cddc1d510e5 found and phase=Bound (5.297658081s) Mar 11 16:52:36.305: INFO: PersistentVolume pvc-dbb9b83e-52bb-4231-a392-5cddc1d510e5 found and phase=Bound (10.447935066s) Mar 11 16:52:41.453: INFO: PersistentVolume pvc-dbb9b83e-52bb-4231-a392-5cddc1d510e5 found and phase=Bound (15.596685915s) Mar 11 16:52:46.603: INFO: PersistentVolume pvc-dbb9b83e-52bb-4231-a392-5cddc1d510e5 found and phase=Bound (20.746115815s) Mar 11 16:52:51.752: INFO: PersistentVolume pvc-dbb9b83e-52bb-4231-a392-5cddc1d510e5 found and phase=Bound (25.895641584s) Mar 11 16:52:56.901: INFO: PersistentVolume pvc-dbb9b83e-52bb-4231-a392-5cddc1d510e5 found and phase=Bound (31.04467563s) Mar 11 16:53:02.051: INFO: PersistentVolume pvc-dbb9b83e-52bb-4231-a392-5cddc1d510e5 found and phase=Bound (36.194362932s) Mar 11 16:53:07.201: INFO: PersistentVolume pvc-dbb9b83e-52bb-4231-a392-5cddc1d510e5 found and phase=Bound (41.344373299s) Mar 11 16:53:12.350: INFO: PersistentVolume pvc-dbb9b83e-52bb-4231-a392-5cddc1d510e5 found and phase=Bound (46.493396906s) Mar 11 16:53:17.499: INFO: PersistentVolume pvc-dbb9b83e-52bb-4231-a392-5cddc1d510e5 found and phase=Bound (51.642868015s) Mar 11 16:53:22.649: INFO: PersistentVolume pvc-dbb9b83e-52bb-4231-a392-5cddc1d510e5 found and phase=Bound (56.792265996s) Mar 11 16:53:27.798: INFO: PersistentVolume pvc-dbb9b83e-52bb-4231-a392-5cddc1d510e5 found and phase=Bound (1m1.941208142s) Mar 11 16:53:32.947: INFO: PersistentVolume pvc-dbb9b83e-52bb-4231-a392-5cddc1d510e5 found and phase=Bound (1m7.090689874s) Mar 11 16:53:38.097: INFO: PersistentVolume pvc-dbb9b83e-52bb-4231-a392-5cddc1d510e5 found and phase=Bound (1m12.240550206s) Mar 11 16:53:43.246: INFO: PersistentVolume pvc-dbb9b83e-52bb-4231-a392-5cddc1d510e5 found and phase=Bound (1m17.389753997s) Mar 11 16:53:48.395: INFO: PersistentVolume pvc-dbb9b83e-52bb-4231-a392-5cddc1d510e5 found and phase=Bound (1m22.538526541s) Mar 11 16:53:53.544: INFO: PersistentVolume pvc-dbb9b83e-52bb-4231-a392-5cddc1d510e5 found and phase=Bound (1m27.687452194s) Mar 11 16:53:58.694: INFO: PersistentVolume pvc-dbb9b83e-52bb-4231-a392-5cddc1d510e5 found and phase=Bound (1m32.837173246s) Mar 11 16:54:03.844: INFO: PersistentVolume pvc-dbb9b83e-52bb-4231-a392-5cddc1d510e5 found and phase=Bound (1m37.986931375s) Mar 11 16:54:08.994: INFO: PersistentVolume pvc-dbb9b83e-52bb-4231-a392-5cddc1d510e5 found and phase=Bound (1m43.137194658s) Mar 11 16:54:14.142: INFO: PersistentVolume pvc-dbb9b83e-52bb-4231-a392-5cddc1d510e5 found and phase=Bound (1m48.285449593s) Mar 11 16:54:19.290: INFO: PersistentVolume pvc-dbb9b83e-52bb-4231-a392-5cddc1d510e5 found and phase=Bound (1m53.433673628s) Mar 11 16:54:24.439: INFO: PersistentVolume pvc-dbb9b83e-52bb-4231-a392-5cddc1d510e5 found and phase=Bound (1m58.582644548s) Mar 11 16:54:29.587: INFO: PersistentVolume pvc-dbb9b83e-52bb-4231-a392-5cddc1d510e5 found and phase=Bound (2m3.730731247s) Mar 11 16:54:34.736: INFO: PersistentVolume pvc-dbb9b83e-52bb-4231-a392-5cddc1d510e5 found and phase=Bound (2m8.879464164s) Mar 11 16:54:39.885: INFO: PersistentVolume pvc-dbb9b83e-52bb-4231-a392-5cddc1d510e5 found and phase=Bound (2m14.027947919s) Mar 11 16:54:45.033: INFO: PersistentVolume pvc-dbb9b83e-52bb-4231-a392-5cddc1d510e5 found and phase=Bound (2m19.17647606s) Mar 11 16:54:50.181: INFO: PersistentVolume pvc-dbb9b83e-52bb-4231-a392-5cddc1d510e5 found and phase=Bound (2m24.324675937s) Mar 11 16:54:55.330: INFO: PersistentVolume pvc-dbb9b83e-52bb-4231-a392-5cddc1d510e5 found and phase=Bound (2m29.4734183s) Mar 11 16:55:00.478: INFO: PersistentVolume pvc-dbb9b83e-52bb-4231-a392-5cddc1d510e5 found and phase=Bound (2m34.621635511s) Mar 11 16:55:05.627: INFO: PersistentVolume pvc-dbb9b83e-52bb-4231-a392-5cddc1d510e5 found and phase=Bound (2m39.77011747s) Mar 11 16:55:10.776: INFO: PersistentVolume pvc-dbb9b83e-52bb-4231-a392-5cddc1d510e5 found and phase=Bound (2m44.919147139s) Mar 11 16:55:15.924: INFO: PersistentVolume pvc-dbb9b83e-52bb-4231-a392-5cddc1d510e5 found and phase=Bound (2m50.067875714s) Mar 11 16:55:21.073: INFO: PersistentVolume pvc-dbb9b83e-52bb-4231-a392-5cddc1d510e5 found and phase=Bound (2m55.216623403s) Mar 11 16:55:26.222: INFO: PersistentVolume pvc-dbb9b83e-52bb-4231-a392-5cddc1d510e5 found and phase=Bound (3m0.365761826s) Mar 11 16:55:31.371: INFO: PersistentVolume pvc-dbb9b83e-52bb-4231-a392-5cddc1d510e5 found and phase=Bound (3m5.514278506s) Mar 11 16:55:36.520: INFO: PersistentVolume pvc-dbb9b83e-52bb-4231-a392-5cddc1d510e5 found and phase=Bound (3m10.66297032s) Mar 11 16:55:41.669: INFO: PersistentVolume pvc-dbb9b83e-52bb-4231-a392-5cddc1d510e5 found and phase=Bound (3m15.812129792s) Mar 11 16:55:46.818: INFO: PersistentVolume pvc-dbb9b83e-52bb-4231-a392-5cddc1d510e5 found and phase=Bound (3m20.961084846s) Mar 11 16:55:51.967: INFO: PersistentVolume pvc-dbb9b83e-52bb-4231-a392-5cddc1d510e5 found and phase=Bound (3m26.109992825s) Mar 11 16:55:57.115: INFO: PersistentVolume pvc-dbb9b83e-52bb-4231-a392-5cddc1d510e5 found and phase=Bound (3m31.258803355s) Mar 11 16:56:02.264: INFO: PersistentVolume pvc-dbb9b83e-52bb-4231-a392-5cddc1d510e5 found and phase=Bound (3m36.407793892s) Mar 11 16:56:07.413: INFO: PersistentVolume pvc-dbb9b83e-52bb-4231-a392-5cddc1d510e5 found and phase=Bound (3m41.556592988s) Mar 11 16:56:12.562: INFO: PersistentVolume pvc-dbb9b83e-52bb-4231-a392-5cddc1d510e5 found and phase=Bound (3m46.705341057s) Mar 11 16:56:17.710: INFO: PersistentVolume pvc-dbb9b83e-52bb-4231-a392-5cddc1d510e5 found and phase=Bound (3m51.853452168s) Mar 11 16:56:22.858: INFO: PersistentVolume pvc-dbb9b83e-52bb-4231-a392-5cddc1d510e5 found and phase=Bound (3m57.001601912s) Mar 11 16:56:28.007: INFO: PersistentVolume pvc-dbb9b83e-52bb-4231-a392-5cddc1d510e5 found and phase=Bound (4m2.15029689s) Mar 11 16:56:33.155: INFO: PersistentVolume pvc-dbb9b83e-52bb-4231-a392-5cddc1d510e5 found and phase=Bound (4m7.298845206s) Mar 11 16:56:38.304: INFO: PersistentVolume pvc-dbb9b83e-52bb-4231-a392-5cddc1d510e5 found and phase=Bound (4m12.447326446s) Mar 11 16:56:43.452: INFO: PersistentVolume pvc-dbb9b83e-52bb-4231-a392-5cddc1d510e5 found and phase=Bound (4m17.595812823s) Mar 11 16:56:48.602: INFO: PersistentVolume pvc-dbb9b83e-52bb-4231-a392-5cddc1d510e5 found and phase=Bound (4m22.745026933s) Mar 11 16:56:53.750: INFO: PersistentVolume pvc-dbb9b83e-52bb-4231-a392-5cddc1d510e5 found and phase=Bound (4m27.893652385s) Mar 11 16:56:58.898: INFO: PersistentVolume pvc-dbb9b83e-52bb-4231-a392-5cddc1d510e5 found and phase=Bound (4m33.041886811s) Mar 11 16:57:04.047: INFO: PersistentVolume pvc-dbb9b83e-52bb-4231-a392-5cddc1d510e5 found and phase=Bound (4m38.190579053s) Mar 11 16:57:09.196: INFO: PersistentVolume pvc-dbb9b83e-52bb-4231-a392-5cddc1d510e5 found and phase=Bound (4m43.339872587s) Mar 11 16:57:14.345: INFO: PersistentVolume pvc-dbb9b83e-52bb-4231-a392-5cddc1d510e5 found and phase=Bound (4m48.488395948s) Mar 11 16:57:19.494: INFO: PersistentVolume pvc-dbb9b83e-52bb-4231-a392-5cddc1d510e5 found and phase=Bound (4m53.637160804s) Mar 11 16:57:24.642: INFO: PersistentVolume pvc-dbb9b83e-52bb-4231-a392-5cddc1d510e5 found and phase=Bound (4m58.785343015s) STEP: Deleting sc - test/e2e/storage/framework/volume_resource.go:228 @ 03/11/23 16:57:29.642 Mar 11 16:57:29.792: INFO: Unexpected error: while cleanup resource: <errors.aggregate | len:1, cap:1>: [ <errors.aggregate | len:1, cap:1>[ <*fmt.wrapError | 0xc002f2a000>{ msg: "persistent Volume pvc-dbb9b83e-52bb-4231-a392-5cddc1d510e5 not deleted by dynamic provisioner: PersistentVolume pvc-dbb9b83e-52bb-4231-a392-5cddc1d510e5 still exists within 5m0s", err: <*errors.errorString | 0xc0010d2010>{ s: "PersistentVolume pvc-dbb9b83e-52bb-4231-a392-5cddc1d510e5 still exists within 5m0s", }, }, ], ] [FAILED] while cleanup resource: persistent Volume pvc-dbb9b83e-52bb-4231-a392-5cddc1d510e5 not deleted by dynamic provisioner: PersistentVolume pvc-dbb9b83e-52bb-4231-a392-5cddc1d510e5 still exists within 5m0s In [DeferCleanup (Each)] at: test/e2e/storage/testsuites/fsgroupchangepolicy.go:129 @ 03/11/23 16:57:29.792 < Exit [DeferCleanup (Each)] [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy - test/e2e/storage/testsuites/fsgroupchangepolicy.go:221 @ 03/11/23 16:57:29.792 (5m4.383s) > Enter [DeferCleanup (Each)] [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy - test/e2e/framework/metrics/init/init.go:35 @ 03/11/23 16:57:29.792 < Exit [DeferCleanup (Each)] [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy - test/e2e/framework/metrics/init/init.go:35 @ 03/11/23 16:57:29.792 (0s) > Enter [DeferCleanup (Each)] [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy - dump namespaces | framework.go:209 @ 03/11/23 16:57:29.792 STEP: dump namespace information after failure - test/e2e/framework/framework.go:288 @ 03/11/23 16:57:29.792 STEP: Collecting events from namespace "fsgroupchangepolicy-6906". - test/e2e/framework/debug/dump.go:42 @ 03/11/23 16:57:29.792 STEP: Found 7 events. - test/e2e/framework/debug/dump.go:46 @ 03/11/23 16:57:29.94 Mar 11 16:57:29.940: INFO: At 2023-03-11 16:44:24 +0000 UTC - event for ebs.csi.aws.comzvh2k: {persistentvolume-controller } WaitForFirstConsumer: waiting for first consumer to be created before binding Mar 11 16:57:29.940: INFO: At 2023-03-11 16:44:24 +0000 UTC - event for ebs.csi.aws.comzvh2k: {persistentvolume-controller } ExternalProvisioning: waiting for a volume to be created, either by external provisioner "ebs.csi.aws.com" or manually created by system administrator Mar 11 16:57:29.940: INFO: At 2023-03-11 16:44:24 +0000 UTC - event for ebs.csi.aws.comzvh2k: {ebs.csi.aws.com_ebs-csi-controller-79785c665b-lpmmf_0e0dbfd5-dc63-4852-b921-aa9f41d3903f } Provisioning: External provisioner is provisioning volume for claim "fsgroupchangepolicy-6906/ebs.csi.aws.comzvh2k" Mar 11 16:57:29.940: INFO: At 2023-03-11 16:44:28 +0000 UTC - event for ebs.csi.aws.comzvh2k: {ebs.csi.aws.com_ebs-csi-controller-79785c665b-lpmmf_0e0dbfd5-dc63-4852-b921-aa9f41d3903f } ProvisioningSucceeded: Successfully provisioned volume pvc-dbb9b83e-52bb-4231-a392-5cddc1d510e5 Mar 11 16:57:29.940: INFO: At 2023-03-11 16:44:28 +0000 UTC - event for pod-c7a5ea42-8a55-4f11-bbe2-e4757a42cdda: {default-scheduler } Scheduled: Successfully assigned fsgroupchangepolicy-6906/pod-c7a5ea42-8a55-4f11-bbe2-e4757a42cdda to i-04523d1dffb0fe725 Mar 11 16:57:29.940: INFO: At 2023-03-11 16:44:31 +0000 UTC - event for pod-c7a5ea42-8a55-4f11-bbe2-e4757a42cdda: {attachdetach-controller } SuccessfulAttachVolume: AttachVolume.Attach succeeded for volume "pvc-dbb9b83e-52bb-4231-a392-5cddc1d510e5" Mar 11 16:57:29.940: INFO: At 2023-03-11 16:50:36 +0000 UTC - event for pod-c7a5ea42-8a55-4f11-bbe2-e4757a42cdda: {taint-controller } TaintManagerEviction: Marking for deletion Pod fsgroupchangepolicy-6906/pod-c7a5ea42-8a55-4f11-bbe2-e4757a42cdda Mar 11 16:57:30.089: INFO: POD NODE PHASE GRACE CONDITIONS Mar 11 16:57:30.089: INFO: pod-c7a5ea42-8a55-4f11-bbe2-e4757a42cdda i-04523d1dffb0fe725 Pending 30s [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2023-03-11 16:44:28 +0000 UTC } {Ready False 0001-01-01 00:00:00 +0000 UTC 2023-03-11 16:44:28 +0000 UTC ContainersNotReady containers with unready status: [write-pod]} {ContainersReady False 0001-01-01 00:00:00 +0000 UTC 2023-03-11 16:44:28 +0000 UTC ContainersNotReady containers with unready status: [write-pod]} {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2023-03-11 16:44:28 +0000 UTC } {DisruptionTarget True 0001-01-01 00:00:00 +0000 UTC 2023-03-11 16:50:36 +0000 UTC DeletionByTaintManager Taint manager: deleting due to NoExecute taint}] Mar 11 16:57:30.089: INFO: Mar 11 16:57:40.238: INFO: Unable to fetch fsgroupchangepolicy-6906/pod-c7a5ea42-8a55-4f11-bbe2-e4757a42cdda/write-pod logs: an error on the server ("unknown") has prevented the request from succeeding (get pods pod-c7a5ea42-8a55-4f11-bbe2-e4757a42cdda) Mar 11 16:57:40.387: INFO: Logging node info for node i-04523d1dffb0fe725 Mar 11 16:57:40.535: INFO: Node Info: &Node{ObjectMeta:{i-04523d1dffb0fe725 5d37640f-a7d9-4c26-86c5-9da21f7ccc18 51243 0 2023-03-11 16:24:20 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:arm64 beta.kubernetes.io/instance-type:t4g.small beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-northeast-1 failure-domain.beta.kubernetes.io/zone:ap-northeast-1a karpenter.k8s.aws/instance-category:t karpenter.k8s.aws/instance-cpu:2 karpenter.k8s.aws/instance-family:t4g karpenter.k8s.aws/instance-generation:4 karpenter.k8s.aws/instance-hypervisor:nitro karpenter.k8s.aws/instance-memory:2048 karpenter.k8s.aws/instance-pods:11 karpenter.k8s.aws/instance-size:small karpenter.sh/capacity-type:spot karpenter.sh/initialized:true karpenter.sh/provisioner-name:nodes kubernetes.io/arch:arm64 kubernetes.io/hostname:i-04523d1dffb0fe725 kubernetes.io/os:linux node-role.kubernetes.io/node: node-role.kubernetes.io/spot-worker:true node.kubernetes.io/instance-type:t4g.small topology.ebs.csi.aws.com/zone:ap-northeast-1a topology.hostpath.csi/node:i-04523d1dffb0fe725 topology.kubernetes.io/region:ap-northeast-1 topology.kubernetes.io/zone:ap-northeast-1a] map[csi.volume.kubernetes.io/nodeid:{"csi-hostpath-read-write-once-pod-7172":"i-04523d1dffb0fe725","csi-hostpath-volume-expand-1673":"i-04523d1dffb0fe725","ebs.csi.aws.com":"i-04523d1dffb0fe725"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [{karpenter.sh/v1alpha5 Provisioner nodes ac7283b2-eed0-4a9f-9ddd-616f33a0991a <nil> 0xc0035f30a5}] [karpenter.sh/termination] [{aws-cloud-controller-manager Update v1 2023-03-11 16:24:20 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{}}}} } {kops-controller Update v1 2023-03-11 16:24:20 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{},"f:node-role.kubernetes.io/spot-worker":{}}}} } {karpenter Update v1 2023-03-11 16:26:12 +0000 UTC FieldsV1 {"f:metadata":{"f:finalizers":{".":{},"v:\"karpenter.sh/termination\"":{}},"f:labels":{".":{},"f:karpenter.k8s.aws/instance-category":{},"f:karpenter.k8s.aws/instance-cpu":{},"f:karpenter.k8s.aws/instance-family":{},"f:karpenter.k8s.aws/instance-generation":{},"f:karpenter.k8s.aws/instance-hypervisor":{},"f:karpenter.k8s.aws/instance-memory":{},"f:karpenter.k8s.aws/instance-pods":{},"f:karpenter.k8s.aws/instance-size":{},"f:karpenter.sh/capacity-type":{},"f:karpenter.sh/initialized":{},"f:karpenter.sh/provisioner-name":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}},"f:ownerReferences":{".":{},"k:{\"uid\":\"ac7283b2-eed0-4a9f-9ddd-616f33a0991a\"}":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-03-11 16:26:50 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{".":{},"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"InternalIP\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kubelet Update v1 2023-03-11 16:44:47 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{"f:kubernetes.io/hostname":{},"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{".":{},"f:cpu":{},"f:ephemeral-storage":{},"f:hugepages-1Gi":{},"f:hugepages-2Mi":{},"f:hugepages-32Mi":{},"f:hugepages-64Ki":{},"f:memory":{},"f:pods":{}},"f:capacity":{".":{},"f:cpu":{},"f:ephemeral-storage":{},"f:hugepages-1Gi":{},"f:hugepages-2Mi":{},"f:hugepages-32Mi":{},"f:hugepages-64Ki":{},"f:memory":{},"f:pods":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{}}},"f:daemonEndpoints":{"f:kubeletEndpoint":{"f:Port":{}}},"f:images":{},"f:nodeInfo":{"f:architecture":{},"f:bootID":{},"f:containerRuntimeVersion":{},"f:kernelVersion":{},"f:kubeProxyVersion":{},"f:kubeletVersion":{},"f:machineID":{},"f:operatingSystem":{},"f:osImage":{},"f:systemUUID":{}},"f:volumesInUse":{}}} status} {kube-controller-manager Update v1 2023-03-11 16:45:36 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:node.alpha.kubernetes.io/ttl":{}},"f:labels":{"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.2.0/24\"":{}},"f:taints":{}}} } {kube-controller-manager Update v1 2023-03-11 16:57:03 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{".":{},"k:{\"type\":\"DiskPressure\"}":{".":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"MemoryPressure\"}":{".":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"PIDPressure\"}":{".":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Ready\"}":{".":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status}]},Spec:NodeSpec{PodCIDR:100.96.2.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-northeast-1a/i-04523d1dffb0fe725,Unschedulable:false,Taints:[]Taint{Taint{Key:node.kubernetes.io/unreachable,Value:,Effect:NoSchedule,TimeAdded:2023-03-11 16:45:31 +0000 UTC,},Taint{Key:node.kubernetes.io/unreachable,Value:,Effect:NoExecute,TimeAdded:2023-03-11 16:45:36 +0000 UTC,},},ConfigSource:nil,PodCIDRs:[100.96.2.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49756430336 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-32Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-64Ki: {{0 0} {<nil>} 0 DecimalSI},memory: {{1941540864 0} {<nil>} 1896036Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44780787229 0} {<nil>} 44780787229 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-32Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-64Ki: {{0 0} {<nil>} 0 DecimalSI},memory: {{1836683264 0} {<nil>} 1793636Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:Ready,Status:Unknown,LastHeartbeatTime:2023-03-11 16:44:47 +0000 UTC,LastTransitionTime:2023-03-11 16:45:31 +0000 UTC,Reason:NodeStatusUnknown,Message:Kubelet stopped posting node status.,},NodeCondition{Type:MemoryPressure,Status:Unknown,LastHeartbeatTime:2023-03-11 16:44:47 +0000 UTC,LastTransitionTime:2023-03-11 16:45:31 +0000 UTC,Reason:NodeStatusUnknown,Message:Kubelet stopped posting node status.,},NodeCondition{Type:DiskPressure,Status:Unknown,LastHeartbeatTime:2023-03-11 16:44:47 +0000 UTC,LastTransitionTime:2023-03-11 16:45:31 +0000 UTC,Reason:NodeStatusUnknown,Message:Kubelet stopped posting node status.,},NodeCondition{Type:PIDPressure,Status:Unknown,LastHeartbeatTime:2023-03-11 16:44:47 +0000 UTC,LastTransitionTime:2023-03-11 16:45:31 +0000 UTC,Reason:NodeStatusUnknown,Message:Kubelet stopped posting node status.,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.48.168,},NodeAddress{Type:ExternalIP,Address:52.199.253.156,},NodeAddress{Type:InternalDNS,Address:i-04523d1dffb0fe725.ap-northeast-1.compute.internal,},NodeAddress{Type:Hostname,Address:i-04523d1dffb0fe725.ap-northeast-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-52-199-253-156.ap-northeast-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2c8fb0e6b2e9c4af310f65b6161cd8,SystemUUID:ec2c8fb0-e6b2-e9c4-af31-0f65b6161cd8,BootID:a3b8de83-558b-44c4-a46a-d097c3f98225,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 22.04.2 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.27.0-alpha.3.420+bb6c6fad2c19b3,KubeProxyVersion:v1.27.0-alpha.3.420+bb6c6fad2c19b3,OperatingSystem:linux,Architecture:arm64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:157636062,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/jessie-dnsutils@sha256:24aaf2626d6b27864c29de2097e8bbb840b3a414271bf7c8995e431e47d8408e registry.k8s.io/e2e-test-images/jessie-dnsutils:1.7],SizeBytes:116764683,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:51eae8381dcb1078289fa7b4f3df2630cdc18d09fb56f8e56b41c40e191d6c83 registry.k8s.io/etcd:3.5.7-0],SizeBytes:80665728,},ContainerImage{Names:[registry.k8s.io/kube-proxy-arm64:v1.27.0-alpha.3.420_bb6c6fad2c19b3],SizeBytes:60947298,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:16bbf38c463a4223d8cfe4da12bc61010b082a79b4bb003e2d3ba3ece5dd5f9e registry.k8s.io/e2e-test-images/agnhost:2.43],SizeBytes:49808364,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nautilus@sha256:80ba6c8c44f9623f06e868a1aa66026c8ec438ad814f9ec95e9333b415fe3550 registry.k8s.io/e2e-test-images/nautilus:1.7],SizeBytes:47953400,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:3fe7acf013d1264ffded116b80a73dc129a449b0fccdb8d21af8279f2233f36e registry.k8s.io/e2e-test-images/httpd:2.4.39-4],SizeBytes:41528818,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:148b022f5c5da426fc2f3c14b5c0867e58ef05961510c84749ac1fddcb0fef22 registry.k8s.io/e2e-test-images/httpd:2.4.38-4],SizeBytes:40343601,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:28932904,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:ee3b525d5b89db99da3b8eb521d9cd90cb6e9ef0fbb651e98bb37be78d36b5b8 registry.k8s.io/sig-storage/csi-provisioner:v3.3.0],SizeBytes:23945474,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/sample-apiserver@sha256:8d70890151aa5d096f331cb9da1b9cd5be0412b7363fe67b5c3befdcaa2a28d0 registry.k8s.io/e2e-test-images/sample-apiserver:1.17.7],SizeBytes:23857462,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:425d8f1b769398127767b06ed97ce62578a3179bcb99809ce93a1649e025ffe7 registry.k8s.io/sig-storage/csi-resizer:v1.6.0],SizeBytes:22678632,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:291334908ddf71a4661fd7f6d9d97274de8a5378a2b6fdfeb2ce73414a34f82f registry.k8s.io/sig-storage/csi-snapshotter:v6.1.0],SizeBytes:22434180,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:9a685020911e2725ad019dbce6e4a5ab93d51e3d4557f115e64343345e05781b registry.k8s.io/sig-storage/csi-attacher:v4.0.0],SizeBytes:22394501,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:122bfb8c1edabb3c0edd63f06523e6940d958d19b3957dc7b1d6f81e9f1f6119 registry.k8s.io/sig-storage/csi-provisioner:v3.1.0],SizeBytes:21735644,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:9ebbf9f023e7b41ccee3d52afe39a89e3ddacdbb69269d583abfc25847cfd9e4 registry.k8s.io/sig-storage/csi-resizer:v1.4.0],SizeBytes:20848413,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:8b9c313c05f54fb04f8d430896f5f5904b6cb157df261501b29adc04d2b2dc7b registry.k8s.io/sig-storage/csi-attacher:v3.4.0],SizeBytes:20565480,},ContainerImage{Names:[registry.k8s.io/cpa/cluster-proportional-autoscaler@sha256:68d396900aeaa072c1f27289485fdac29834045a6f3ffe369bf389d830ef572d registry.k8s.io/cpa/cluster-proportional-autoscaler:1.8.6],SizeBytes:19007623,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nonroot@sha256:ee9f50b3c64b174d296d91ca9f69a914ac30e59095dfb462b2b518ad28a63655 registry.k8s.io/e2e-test-images/nonroot:1.4],SizeBytes:18561972,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:92257881c1d6493cf18299a24af42330f891166560047902b8d431fb66b01af5 registry.k8s.io/sig-storage/hostpathplugin:v1.9.0],SizeBytes:18267611,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:9b2d9abda017c28b12f84a344f57cd73fbdb6c2bd7dd5bdd5018246ad1093ba6 registry.k8s.io/sig-storage/hostpathplugin:v1.11.0],SizeBytes:18159085,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:017727efcfeb7d053af68e51436ce8e65edbc6ca573720afb4f79c8594036955 registry.k8s.io/coredns/coredns:v1.10.0],SizeBytes:13766234,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:8502345,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:933940f13b3ea0abc62e656c1aa5c5b47c04b15d71250413a6b821bd0c58b94e registry.k8s.io/sig-storage/livenessprobe:v2.7.0],SizeBytes:8223819,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:7685305,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:db048754ae68ae337d8fa96494c96d2a1204c3320f5dcf7e8e71085adec85da6 registry.k8s.io/e2e-test-images/nginx:1.15-4],SizeBytes:6883347,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:5c99cf6a02adda929b10321dbf4ecfa00d87be9ba4fb456006237d530ab4baa1 registry.k8s.io/e2e-test-images/nginx:1.14-4],SizeBytes:6860089,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nonewprivs@sha256:8ac1264691820febacf3aea5d152cbde6d10685731ec14966a9401c6f47a68ac registry.k8s.io/e2e-test-images/nonewprivs:1.3],SizeBytes:5025027,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf registry.k8s.io/e2e-test-images/busybox:1.29-2],SizeBytes:773367,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:2e0f836850e09b8b7cc937681d6194537a09fbd5f6b9e08f4d646a85128e8937 registry.k8s.io/e2e-test-images/busybox:1.29-4],SizeBytes:772933,},ContainerImage{Names:[registry.k8s.io/pause@sha256:7031c1b283388d2c2e09b57badb803c05ebed362dc88d84b480cc47f72a21097 registry.k8s.io/pause:3.9],SizeBytes:268051,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:253553,},},VolumesInUse:[kubernetes.io/csi/csi-hostpath-read-write-once-pod-7172^038fde52-c02c-11ed-b028-5eca9e37a97f kubernetes.io/csi/csi-hostpath-volume-expand-1673^f8193b68-c02b-11ed-8784-9e8522e7cc37 kubernetes.io/csi/ebs.csi.aws.com^vol-0293af09ce81fe75e],VolumesAttached:[]AttachedVolume{},Config:nil,},} Mar 11 16:57:40.535: INFO: Logging kubelet events for node i-04523d1dffb0fe725 Mar 11 16:57:40.687: INFO: Logging pods the kubelet thinks is on node i-04523d1dffb0fe725 Mar 11 16:57:50.836: INFO: Unable to retrieve kubelet pods for node i-04523d1dffb0fe725: error trying to reach service: net/http: TLS handshake timeout Mar 11 16:57:50.836: INFO: Logging node info for node i-07e2b6e650d7ef56d Mar 11 16:57:50.984: INFO: Node Info: &Node{ObjectMeta:{i-07e2b6e650d7ef56d fcc65272-6f1b-4228-ac59-e9e3e92cdb2c 47965 0 2023-03-11 16:24:19 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:arm64 beta.kubernetes.io/instance-type:t4g.small beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-northeast-1 failure-domain.beta.kubernetes.io/zone:ap-northeast-1a io.kubernetes.storage.mock/node:some-mock-node karpenter.k8s.aws/instance-category:t karpenter.k8s.aws/instance-cpu:2 karpenter.k8s.aws/instance-family:t4g karpenter.k8s.aws/instance-generation:4 karpenter.k8s.aws/instance-hypervisor:nitro karpenter.k8s.aws/instance-memory:2048 karpenter.k8s.aws/instance-pods:11 karpenter.k8s.aws/instance-size:small karpenter.sh/capacity-type:spot karpenter.sh/initialized:true karpenter.sh/provisioner-name:nodes kubernetes.io/arch:arm64 kubernetes.io/hostname:i-07e2b6e650d7ef56d kubernetes.io/os:linux node-role.kubernetes.io/node: node-role.kubernetes.io/spot-worker:true node.kubernetes.io/instance-type:t4g.small topology.ebs.csi.aws.com/zone:ap-northeast-1a topology.hostpath.csi/node:i-07e2b6e650d7ef56d topology.kubernetes.io/region:ap-northeast-1 topology.kubernetes.io/zone:ap-northeast-1a] map[csi.volume.kubernetes.io/nodeid:{"csi-hostpath-ephemeral-7952":"i-07e2b6e650d7ef56d","csi-hostpath-volume-expand-703":"i-07e2b6e650d7ef56d","ebs.csi.aws.com":"i-07e2b6e650d7ef56d"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [{karpenter.sh/v1alpha5 Provisioner nodes ac7283b2-eed0-4a9f-9ddd-616f33a0991a <nil> 0xc0035f39f5}] [karpenter.sh/termination] [{aws-cloud-controller-manager Update v1 2023-03-11 16:24:20 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{}}}} } {kops-controller Update v1 2023-03-11 16:24:20 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{},"f:node-role.kubernetes.io/spot-worker":{}}}} } {karpenter Update v1 2023-03-11 16:26:15 +0000 UTC FieldsV1 {"f:metadata":{"f:finalizers":{".":{},"v:\"karpenter.sh/termination\"":{}},"f:labels":{".":{},"f:karpenter.k8s.aws/instance-category":{},"f:karpenter.k8s.aws/instance-cpu":{},"f:karpenter.k8s.aws/instance-family":{},"f:karpenter.k8s.aws/instance-generation":{},"f:karpenter.k8s.aws/instance-hypervisor":{},"f:karpenter.k8s.aws/instance-memory":{},"f:karpenter.k8s.aws/instance-pods":{},"f:karpenter.k8s.aws/instance-size":{},"f:karpenter.sh/capacity-type":{},"f:karpenter.sh/initialized":{},"f:karpenter.sh/provisioner-name":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}},"f:ownerReferences":{".":{},"k:{\"uid\":\"ac7283b2-eed0-4a9f-9ddd-616f33a0991a\"}":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-03-11 16:26:50 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{".":{},"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"InternalIP\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kubelet Update v1 2023-03-11 16:44:22 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{"f:io.kubernetes.storage.mock/node":{},"f:kubernetes.io/hostname":{},"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{".":{},"f:cpu":{},"f:ephemeral-storage":{},"f:hugepages-1Gi":{},"f:hugepages-2Mi":{},"f:hugepages-32Mi":{},"f:hugepages-64Ki":{},"f:memory":{},"f:pods":{}},"f:capacity":{".":{},"f:cpu":{},"f:ephemeral-storage":{},"f:hugepages-1Gi":{},"f:hugepages-2Mi":{},"f:hugepages-32Mi":{},"f:hugepages-64Ki":{},"f:memory":{},"f:pods":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{}}},"f:daemonEndpoints":{"f:kubeletEndpoint":{"f:Port":{}}},"f:images":{},"f:nodeInfo":{"f:architecture":{},"f:bootID":{},"f:containerRuntimeVersion":{},"f:kernelVersion":{},"f:kubeProxyVersion":{},"f:kubeletVersion":{},"f:machineID":{},"f:operatingSystem":{},"f:osImage":{},"f:systemUUID":{}},"f:volumesInUse":{}}} status} {kube-controller-manager Update v1 2023-03-11 16:45:36 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{".":{},"k:{\"type\":\"DiskPressure\"}":{".":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"MemoryPressure\"}":{".":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"PIDPressure\"}":{".":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Ready\"}":{".":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}},"f:volumesAttached":{}}} status} {kube-controller-manager Update v1 2023-03-11 16:45:46 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:node.alpha.kubernetes.io/ttl":{}},"f:labels":{"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.1.0/24\"":{}},"f:taints":{}}} }]},Spec:NodeSpec{PodCIDR:100.96.1.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-northeast-1a/i-07e2b6e650d7ef56d,Unschedulable:false,Taints:[]Taint{Taint{Key:node.kubernetes.io/unreachable,Value:,Effect:NoSchedule,TimeAdded:2023-03-11 16:45:36 +0000 UTC,},Taint{Key:node.kubernetes.io/unreachable,Value:,Effect:NoExecute,TimeAdded:2023-03-11 16:45:46 +0000 UTC,},},ConfigSource:nil,PodCIDRs:[100.96.1.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49756430336 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-32Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-64Ki: {{0 0} {<nil>} 0 DecimalSI},memory: {{1941549056 0} {<nil>} 1896044Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44780787229 0} {<nil>} 44780787229 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-32Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-64Ki: {{0 0} {<nil>} 0 DecimalSI},memory: {{1836691456 0} {<nil>} 1793644Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:Ready,Status:Unknown,LastHeartbeatTime:2023-03-11 16:44:12 +0000 UTC,LastTransitionTime:2023-03-11 16:45:36 +0000 UTC,Reason:NodeStatusUnknown,Message:Kubelet stopped posting node status.,},NodeCondition{Type:MemoryPressure,Status:Unknown,LastHeartbeatTime:2023-03-11 16:44:12 +0000 UTC,LastTransitionTime:2023-03-11 16:45:36 +0000 UTC,Reason:NodeStatusUnknown,Message:Kubelet stopped posting node status.,},NodeCondition{Type:DiskPressure,Status:Unknown,LastHeartbeatTime:2023-03-11 16:44:12 +0000 UTC,LastTransitionTime:2023-03-11 16:45:36 +0000 UTC,Reason:NodeStatusUnknown,Message:Kubelet stopped posting node status.,},NodeCondition{Type:PIDPressure,Status:Unknown,LastHeartbeatTime:2023-03-11 16:44:12 +0000 UTC,LastTransitionTime:2023-03-11 16:45:36 +0000 UTC,Reason:NodeStatusUnknown,Message:Kubelet stopped posting node status.,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.35.76,},NodeAddress{Type:ExternalIP,Address:54.238.106.83,},NodeAddress{Type:InternalDNS,Address:i-07e2b6e650d7ef56d.ap-northeast-1.compute.internal,},NodeAddress{Type:Hostname,Address:i-07e2b6e650d7ef56d.ap-northeast-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-54-238-106-83.ap-northeast-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2083a8e9d27d02fac967340367f875,SystemUUID:ec2083a8-e9d2-7d02-fac9-67340367f875,BootID:ea264ed6-5b0d-47d6-ad5d-6bd2a1f7c11b,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 22.04.2 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.27.0-alpha.3.420+bb6c6fad2c19b3,KubeProxyVersion:v1.27.0-alpha.3.420+bb6c6fad2c19b3,OperatingSystem:linux,Architecture:arm64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:157636062,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/jessie-dnsutils@sha256:24aaf2626d6b27864c29de2097e8bbb840b3a414271bf7c8995e431e47d8408e registry.k8s.io/e2e-test-images/jessie-dnsutils:1.7],SizeBytes:116764683,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:51eae8381dcb1078289fa7b4f3df2630cdc18d09fb56f8e56b41c40e191d6c83 registry.k8s.io/etcd:3.5.7-0],SizeBytes:80665728,},ContainerImage{Names:[registry.k8s.io/kube-proxy-arm64:v1.27.0-alpha.3.420_bb6c6fad2c19b3],SizeBytes:60947298,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:16bbf38c463a4223d8cfe4da12bc61010b082a79b4bb003e2d3ba3ece5dd5f9e registry.k8s.io/e2e-test-images/agnhost:2.43],SizeBytes:49808364,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nautilus@sha256:80ba6c8c44f9623f06e868a1aa66026c8ec438ad814f9ec95e9333b415fe3550 registry.k8s.io/e2e-test-images/nautilus:1.7],SizeBytes:47953400,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:3fe7acf013d1264ffded116b80a73dc129a449b0fccdb8d21af8279f2233f36e registry.k8s.io/e2e-test-images/httpd:2.4.39-4],SizeBytes:41528818,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:148b022f5c5da426fc2f3c14b5c0867e58ef05961510c84749ac1fddcb0fef22 registry.k8s.io/e2e-test-images/httpd:2.4.38-4],SizeBytes:40343601,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:28932904,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:ee3b525d5b89db99da3b8eb521d9cd90cb6e9ef0fbb651e98bb37be78d36b5b8 registry.k8s.io/sig-storage/csi-provisioner:v3.3.0],SizeBytes:23945474,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/sample-apiserver@sha256:8d70890151aa5d096f331cb9da1b9cd5be0412b7363fe67b5c3befdcaa2a28d0 registry.k8s.io/e2e-test-images/sample-apiserver:1.17.7],SizeBytes:23857462,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:425d8f1b769398127767b06ed97ce62578a3179bcb99809ce93a1649e025ffe7 registry.k8s.io/sig-storage/csi-resizer:v1.6.0],SizeBytes:22678632,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:291334908ddf71a4661fd7f6d9d97274de8a5378a2b6fdfeb2ce73414a34f82f registry.k8s.io/sig-storage/csi-snapshotter:v6.1.0],SizeBytes:22434180,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:9a685020911e2725ad019dbce6e4a5ab93d51e3d4557f115e64343345e05781b registry.k8s.io/sig-storage/csi-attacher:v4.0.0],SizeBytes:22394501,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:122bfb8c1edabb3c0edd63f06523e6940d958d19b3957dc7b1d6f81e9f1f6119 registry.k8s.io/sig-storage/csi-provisioner:v3.1.0],SizeBytes:21735644,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/regression-issue-74839@sha256:b4f1d8d61bdad84bd50442d161d5460e4019d53e989b64220fdbc62fc87d76bf registry.k8s.io/e2e-test-images/regression-issue-74839:1.2],SizeBytes:21331383,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:9ebbf9f023e7b41ccee3d52afe39a89e3ddacdbb69269d583abfc25847cfd9e4 registry.k8s.io/sig-storage/csi-resizer:v1.4.0],SizeBytes:20848413,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:8b9c313c05f54fb04f8d430896f5f5904b6cb157df261501b29adc04d2b2dc7b registry.k8s.io/sig-storage/csi-attacher:v3.4.0],SizeBytes:20565480,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nonroot@sha256:ee9f50b3c64b174d296d91ca9f69a914ac30e59095dfb462b2b518ad28a63655 registry.k8s.io/e2e-test-images/nonroot:1.4],SizeBytes:18561972,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:92257881c1d6493cf18299a24af42330f891166560047902b8d431fb66b01af5 registry.k8s.io/sig-storage/hostpathplugin:v1.9.0],SizeBytes:18267611,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:9b2d9abda017c28b12f84a344f57cd73fbdb6c2bd7dd5bdd5018246ad1093ba6 registry.k8s.io/sig-storage/hostpathplugin:v1.11.0],SizeBytes:18159085,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:017727efcfeb7d053af68e51436ce8e65edbc6ca573720afb4f79c8594036955 registry.k8s.io/coredns/coredns:v1.10.0],SizeBytes:13766234,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:8502345,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:933940f13b3ea0abc62e656c1aa5c5b47c04b15d71250413a6b821bd0c58b94e registry.k8s.io/sig-storage/livenessprobe:v2.7.0],SizeBytes:8223819,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:7685305,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:db048754ae68ae337d8fa96494c96d2a1204c3320f5dcf7e8e71085adec85da6 registry.k8s.io/e2e-test-images/nginx:1.15-4],SizeBytes:6883347,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:5c99cf6a02adda929b10321dbf4ecfa00d87be9ba4fb456006237d530ab4baa1 registry.k8s.io/e2e-test-images/nginx:1.14-4],SizeBytes:6860089,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nonewprivs@sha256:8ac1264691820febacf3aea5d152cbde6d10685731ec14966a9401c6f47a68ac registry.k8s.io/e2e-test-images/nonewprivs:1.3],SizeBytes:5025027,},ContainerImage{Names:[gcr.io/authenticated-image-pulling/alpine@sha256:7ff177862cb50c602bfe81f805969412e619c054a2bbead977d0c276988aa4a0 gcr.io/authenticated-image-pulling/alpine:3.7],SizeBytes:2003505,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf registry.k8s.io/e2e-test-images/busybox:1.29-2],SizeBytes:773367,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:2e0f836850e09b8b7cc937681d6194537a09fbd5f6b9e08f4d646a85128e8937 registry.k8s.io/e2e-test-images/busybox:1.29-4],SizeBytes:772933,},ContainerImage{Names:[registry.k8s.io/pause@sha256:7031c1b283388d2c2e09b57badb803c05ebed362dc88d84b480cc47f72a21097 registry.k8s.io/pause:3.9],SizeBytes:268051,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:253553,},},VolumesInUse:[kubernetes.io/csi/csi-hostpath-ephemeral-7952^eca2344a-c02b-11ed-b483-1e4a85f27a89 kubernetes.io/csi/csi-hostpath-ephemeral-7952^f0ac2401-c02b-11ed-b483-1e4a85f27a89 kubernetes.io/csi/ebs.csi.aws.com^vol-06cba8db23f1489ad],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-06cba8db23f1489ad,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/csi-hostpath-ephemeral-7952^f0ac2401-c02b-11ed-b483-1e4a85f27a89,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/csi-hostpath-ephemeral-7952^eca2344a-c02b-11ed-b483-1e4a85f27a89,DevicePath:,},},Config:nil,},} Mar 11 16:57:50.985: INFO: Logging kubelet events for node i-07e2b6e650d7ef56d Mar 11 16:57:51.136: INFO: Logging pods the kubelet thinks is on node i-07e2b6e650d7ef56d Mar 11 16:58:01.285: INFO: Unable to retrieve kubelet pods for node i-07e2b6e650d7ef56d: error trying to reach service: net/http: TLS handshake timeout Mar 11 16:58:01.285: INFO: Logging node info for node i-0e3467ec552af797e Mar 11 16:58:01.433: INFO: Node Info: &Node{ObjectMeta:{i-0e3467ec552af797e 6c9b0cc5-2c4e-4037-bf31-61bac12a7775 50325 0 2023-03-11 16:20:46 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:arm64 beta.kubernetes.io/instance-type:c6g.xlarge beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-northeast-1 failure-domain.beta.kubernetes.io/zone:ap-northeast-1a kops.k8s.io/kops-controller-pki: kubernetes.io/arch:arm64 kubernetes.io/hostname:i-0e3467ec552af797e kubernetes.io/os:linux node-role.kubernetes.io/control-plane: node.kubernetes.io/exclude-from-external-load-balancers: node.kubernetes.io/instance-type:c6g.xlarge topology.ebs.csi.aws.com/zone:ap-northeast-1a topology.kubernetes.io/region:ap-northeast-1 topology.kubernetes.io/zone:ap-northeast-1a] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-0e3467ec552af797e"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kubelet Update v1 2023-03-11 16:20:46 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {protokube Update v1 2023-03-11 16:21:30 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kops.k8s.io/kops-controller-pki":{},"f:node-role.kubernetes.io/control-plane":{},"f:node.kubernetes.io/exclude-from-external-load-balancers":{}}}} } {aws-cloud-controller-manager Update v1 2023-03-11 16:21:50 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-03-11 16:21:50 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kube-controller-manager Update v1 2023-03-11 16:21:58 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.0.0/24\"":{}},"f:taints":{}}} } {kubelet Update v1 2023-03-11 16:53:56 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.0.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-northeast-1a/i-0e3467ec552af797e,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/control-plane,Value:,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[100.96.0.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{4 0} {<nil>} 4 DecimalSI},ephemeral-storage: {{49756430336 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-32Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-64Ki: {{0 0} {<nil>} 0 DecimalSI},memory: {{8150110208 0} {<nil>} 7959092Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{4 0} {<nil>} 4 DecimalSI},ephemeral-storage: {{44780787229 0} {<nil>} 44780787229 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-32Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-64Ki: {{0 0} {<nil>} 0 DecimalSI},memory: {{8045252608 0} {<nil>} 7856692Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-03-11 16:53:56 +0000 UTC,LastTransitionTime:2023-03-11 16:20:46 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-03-11 16:53:56 +0000 UTC,LastTransitionTime:2023-03-11 16:20:46 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-03-11 16:53:56 +0000 UTC,LastTransitionTime:2023-03-11 16:20:46 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-03-11 16:53:56 +0000 UTC,LastTransitionTime:2023-03-11 16:21:58 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.40.18,},NodeAddress{Type:ExternalIP,Address:52.194.213.7,},NodeAddress{Type:InternalDNS,Address:i-0e3467ec552af797e.ap-northeast-1.compute.internal,},NodeAddress{Type:Hostname,Address:i-0e3467ec552af797e.ap-northeast-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-52-194-213-7.ap-northeast-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2abf2e564965592c7716accd56507e,SystemUUID:ec2abf2e-5649-6559-2c77-16accd56507e,BootID:fbada3ff-8655-404c-a98d-876577d4c322,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 22.04.2 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.27.0-alpha.3.420+bb6c6fad2c19b3,KubeProxyVersion:v1.27.0-alpha.3.420+bb6c6fad2c19b3,OperatingSystem:linux,Architecture:arm64,},Images:[]ContainerImage{ContainerImage{Names:[registry.k8s.io/etcd@sha256:9ce33ba33d8e738a5b85ed50b5080ac746deceed4a7496c550927a7a19ca3b6d registry.k8s.io/etcd:3.5.0-0],SizeBytes:157797353,},ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:157636062,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:4afb99b4690b418ffc2ceb67e1a17376457e441c1f09ab55447f0aaf992fa646 registry.k8s.io/etcd:3.4.3-0],SizeBytes:151677080,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:4ad90a11b55313b182afc186b9876c8e891531b8db4c9bf1541953021618d0e2 registry.k8s.io/etcd:3.4.13-0],SizeBytes:134531559,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:0d2ab6b15b4fe0b903b9e81192d942fb39c29be61fe341cd16db5d0165b0435c registry.k8s.io/etcd:3.3.17-0],SizeBytes:126979107,},ContainerImage{Names:[registry.k8s.io/kube-apiserver-arm64:v1.27.0-alpha.3.420_bb6c6fad2c19b3],SizeBytes:115123555,},ContainerImage{Names:[registry.k8s.io/kube-controller-manager-arm64:v1.27.0-alpha.3.420_bb6c6fad2c19b3],SizeBytes:107914235,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:17da501f5d2a675be46040422a27b7cc21b8a43895ac998b171db1c346f361f7 registry.k8s.io/etcd:3.3.10-0],SizeBytes:81436097,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:6f72b851544986cb0921b53ea655ec04c36131248f16d4ad110cb3ca0c369dc1 registry.k8s.io/etcd:3.5.4-0],SizeBytes:81120118,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:13f53ed1d91e2e11aac476ee9a0269fdda6cc4874eba903efd40daf50c55eee5 registry.k8s.io/etcd:3.5.3-0],SizeBytes:81114891,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:51eae8381dcb1078289fa7b4f3df2630cdc18d09fb56f8e56b41c40e191d6c83 registry.k8s.io/etcd:3.5.7-0],SizeBytes:80665728,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:dd75ec974b0a2a6f6bb47001ba09207976e625db898d1b16735528c009cb171c registry.k8s.io/etcd:3.5.6-0],SizeBytes:80539316,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:905d7ca17fd02bc24c0eba9a062753aba15db3e31422390bc3238eb762339b20 registry.k8s.io/etcd:3.2.24-1],SizeBytes:61500433,},ContainerImage{Names:[registry.k8s.io/kube-proxy-arm64:v1.27.0-alpha.3.420_bb6c6fad2c19b3],SizeBytes:60947298,},ContainerImage{Names:[registry.k8s.io/etcdadm/etcd-manager-slim@sha256:8727adc059334e0e3329dbccce116a734da4a090da5e86c8cddccaf333caa666 registry.k8s.io/etcdadm/etcd-manager-slim:v3.0.20230201],SizeBytes:60826655,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:64b9ea357325d5db9f8a723dcf503b5a449177b17ac87d69481e126bb724c263 registry.k8s.io/etcd:3.5.1-0],SizeBytes:59553904,},ContainerImage{Names:[registry.k8s.io/kube-scheduler-arm64:v1.27.0-alpha.3.420_bb6c6fad2c19b3],SizeBytes:56731120,},ContainerImage{Names:[registry.k8s.io/kops/kops-controller:1.27.0-alpha.1],SizeBytes:41885191,},ContainerImage{Names:[registry.k8s.io/kops/dns-controller:1.27.0-alpha.1],SizeBytes:40376038,},ContainerImage{Names:[public.ecr.aws/karpenter/controller@sha256:68db4f092cf9cc83f5ef9e2fbc5407c2cb682e81f64dfaa700a7602ede38b1cf public.ecr.aws/karpenter/controller:v0.16.3],SizeBytes:33707751,},ContainerImage{Names:[public.ecr.aws/karpenter/webhook@sha256:96a2d9b06d6bc5127801f358f74b1cf2d289b423a2e9ba40c573c0b14b17dafa public.ecr.aws/karpenter/webhook:v0.16.3],SizeBytes:33307187,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:28932904,},ContainerImage{Names:[quay.io/cilium/operator@sha256:a6d24a006a6b92967ac90786b49bc1ac26e5477cf028cd1186efcfc2466484db quay.io/cilium/operator:v1.12.5],SizeBytes:24567287,},ContainerImage{Names:[registry.k8s.io/provider-aws/cloud-controller-manager@sha256:fdeb61e3e42ecd9cca868d550ebdb88dd6341d9e91fcfa9a37e227dab2ad22cb registry.k8s.io/provider-aws/cloud-controller-manager:v1.26.0],SizeBytes:18345226,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:8502345,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:7685305,},ContainerImage{Names:[registry.k8s.io/kops/kube-apiserver-healthcheck:1.27.0-alpha.1],SizeBytes:4740762,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:253553,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Mar 11 16:58:01.434: INFO: Logging kubelet events for node i-0e3467ec552af797e Mar 11 16:58:01.586: INFO: Logging pods the kubelet thinks is on node i-0e3467ec552af797e Mar 11 16:58:01.741: INFO: cilium-qrsfn started at 2023-03-11 16:21:37 +0000 UTC (1+1 container statuses recorded) Mar 11 16:58:01.741: INFO: Init container clean-cilium-state ready: true, restart count 0 Mar 11 16:58:01.741: INFO: Container cilium-agent ready: true, restart count 0 Mar 11 16:58:01.741: INFO: cilium-operator-8496cc65f5-qhdg8 started at 2023-03-11 16:21:37 +0000 UTC (0+1 container statuses recorded) Mar 11 16:58:01.741: INFO: Container cilium-operator ready: true, restart count 0 Mar 11 16:58:01.741: INFO: kops-controller-76gkw started at 2023-03-11 16:21:37 +0000 UTC (0+1 container statuses recorded) Mar 11 16:58:01.741: INFO: Container kops-controller ready: true, restart count 0 Mar 11 16:58:01.741: INFO: aws-cloud-controller-manager-kwbw2 started at 2023-03-11 16:21:37 +0000 UTC (0+1 container statuses recorded) Mar 11 16:58:01.741: INFO: Container aws-cloud-controller-manager ready: true, restart count 0 Mar 11 16:58:01.741: INFO: kube-apiserver-i-0e3467ec552af797e started at 2023-03-11 16:18:44 +0000 UTC (0+2 container statuses recorded) Mar 11 16:58:01.741: INFO: Container healthcheck ready: true, restart count 0 Mar 11 16:58:01.741: INFO: Container kube-apiserver ready: true, restart count 3 Mar 11 16:58:01.741: INFO: kube-scheduler-i-0e3467ec552af797e started at 2023-03-11 16:18:44 +0000 UTC (0+1 container statuses recorded) Mar 11 16:58:01.741: INFO: Container kube-scheduler ready: true, restart count 0 Mar 11 16:58:01.741: INFO: dns-controller-7dfd64d999-q9hh7 started at 2023-03-11 16:21:37 +0000 UTC (0+1 container statuses recorded) Mar 11 16:58:01.741: INFO: Container dns-controller ready: true, restart count 0 Mar 11 16:58:01.741: INFO: ebs-csi-node-lpwz9 started at 2023-03-11 16:21:37 +0000 UTC (0+3 container statuses recorded) Mar 11 16:58:01.741: INFO: Container ebs-plugin ready: true, restart count 0 Mar 11 16:58:01.741: INFO: Container liveness-probe ready: true, restart count 0 Mar 11 16:58:01.741: INFO: Container node-driver-registrar ready: true, restart count 0 Mar 11 16:58:01.741: INFO: karpenter-webhook-5b646cd4f7-6sg2k started at 2023-03-11 16:21:58 +0000 UTC (0+1 container statuses recorded) Mar 11 16:58:01.741: INFO: Container webhook ready: true, restart count 0 Mar 11 16:58:01.742: INFO: karpenter-controller-567749677d-d9szs started at 2023-03-11 16:21:58 +0000 UTC (0+1 container statuses recorded) Mar 11 16:58:01.742: INFO: Container controller ready: true, restart count 0 Mar 11 16:58:01.742: INFO: kube-controller-manager-i-0e3467ec552af797e started at 2023-03-11 16:18:44 +0000 UTC (0+1 container statuses recorded) Mar 11 16:58:01.742: INFO: Container kube-controller-manager ready: true, restart count 4 Mar 11 16:58:01.742: INFO: etcd-manager-events-i-0e3467ec552af797e started at 2023-03-11 16:18:44 +0000 UTC (11+1 container statuses recorded) Mar 11 16:58:01.742: INFO: Init container init-etcd-3-2-24 ready: true, restart count 0 Mar 11 16:58:01.742: INFO: Init container init-etcd-3-3-10 ready: true, restart count 0 Mar 11 16:58:01.742: INFO: Init container init-etcd-3-3-17 ready: true, restart count 0 Mar 11 16:58:01.742: INFO: Init container init-etcd-3-4-13 ready: true, restart count 0 Mar 11 16:58:01.742: INFO: Init container init-etcd-3-4-3 ready: true, restart count 0 Mar 11 16:58:01.742: INFO: Init container init-etcd-3-5-0 ready: true, restart count 0 Mar 11 16:58:01.742: INFO: Init container init-etcd-3-5-1 ready: true, restart count 0 Mar 11 16:58:01.742: INFO: Init container init-etcd-3-5-3 ready: true, restart count 0 Mar 11 16:58:01.742: INFO: Init container init-etcd-3-5-4 ready: true, restart count 0 Mar 11 16:58:01.742: INFO: Init container init-etcd-3-5-6 ready: true, restart count 0 Mar 11 16:58:01.742: INFO: Init container init-etcd-3-5-7 ready: true, restart count 0 Mar 11 16:58:01.742: INFO: Container etcd-manager ready: true, restart count 0 Mar 11 16:58:01.742: INFO: etcd-manager-main-i-0e3467ec552af797e started at 2023-03-11 16:18:44 +0000 UTC (11+1 container statuses recorded) Mar 11 16:58:01.742: INFO: Init container init-etcd-3-2-24 ready: true, restart count 0 Mar 11 16:58:01.742: INFO: Init container init-etcd-3-3-10 ready: true, restart count 0 Mar 11 16:58:01.742: INFO: Init container init-etcd-3-3-17 ready: true, restart count 0 Mar 11 16:58:01.742: INFO: Init container init-etcd-3-4-13 ready: true, restart count 0 Mar 11 16:58:01.742: INFO: Init container init-etcd-3-4-3 ready: true, restart count 0 Mar 11 16:58:01.742: INFO: Init container init-etcd-3-5-0 ready: true, restart count 0 Mar 11 16:58:01.742: INFO: Init container init-etcd-3-5-1 ready: true, restart count 0 Mar 11 16:58:01.742: INFO: Init container init-etcd-3-5-3 ready: true, restart count 0 Mar 11 16:58:01.742: INFO: Init container init-etcd-3-5-4 ready: true, restart count 0 Mar 11 16:58:01.742: INFO: Init container init-etcd-3-5-6 ready: true, restart count 0 Mar 11 16:58:01.742: INFO: Init container init-etcd-3-5-7 ready: true, restart count 0 Mar 11 16:58:01.742: INFO: Container etcd-manager ready: true, restart count 0 Mar 11 16:58:02.376: INFO: Latency metrics for node i-0e3467ec552af797e END STEP: dump namespace information after failure - test/e2e/framework/framework.go:288 @ 03/11/23 16:58:02.376 (32.584s) < Exit [DeferCleanup (Each)] [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy - dump namespaces | framework.go:209 @ 03/11/23 16:58:02.376 (32.584s) > Enter [DeferCleanup (Each)] [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy - tear down framework | framework.go:206 @ 03/11/23 16:58:02.376 STEP: Destroying namespace "fsgroupchangepolicy-6906" for this suite. - test/e2e/framework/framework.go:351 @ 03/11/23 16:58:02.376 < Exit [DeferCleanup (Each)] [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy - tear down framework | framework.go:206 @ 03/11/23 16:58:02.527 (151ms) > Enter [ReportAfterEach] TOP-LEVEL - test/e2e/e2e_test.go:144 @ 03/11/23 16:58:02.528 < Exit [ReportAfterEach] TOP-LEVEL - test/e2e/e2e_test.go:144 @ 03/11/23 16:58:02.528 (0s)
Find pod-c7a5ea42-8a55-4f11-bbe2-e4757a42cdda mentions in log files | View test history on testgrid
go run hack/e2e.go -v --test --test_args='--ginkgo.focus=Kubernetes\se2e\ssuite\s\[It\]\sExternal\sStorage\s\[Driver\:\sebs\.csi\.aws\.com\]\s\[Testpattern\:\sGeneric\sEphemeral\-volume\s\(default\sfs\)\s\(late\-binding\)\]\sephemeral\sshould\screate\sread\-only\sinline\sephemeral\svolume$'
[FAILED] expected pod to not be found: Timed out after 300.000s. Expected <*v1.Pod | 0xc003553680>: metadata: creationTimestamp: "2023-03-11T16:44:01Z" deletionGracePeriodSeconds: 30 deletionTimestamp: "2023-03-11T16:44:46Z" finalizers: - foregroundDeletion generateName: inline-volume-tester- labels: app: inline-volume-tester managedFields: - apiVersion: v1 fieldsType: FieldsV1 fieldsV1: f:metadata: f:generateName: {} f:labels: .: {} f:app: {} f:spec: f:containers: k:{"name":"csi-volume-tester"}: .: {} f:command: {} f:image: {} f:imagePullPolicy: {} f:name: {} f:resources: {} f:terminationMessagePath: {} f:terminationMessagePolicy: {} f:volumeMounts: .: {} k:{"mountPath":"/mnt/test-0"}: .: {} f:mountPath: {} f:name: {} f:readOnly: {} f:dnsPolicy: {} f:enableServiceLinks: {} f:nodeSelector: {} f:restartPolicy: {} f:schedulerName: {} f:securityContext: {} f:terminationGracePeriodSeconds: {} f:volumes: .: {} k:{"name":"my-volume-0"}: .: {} f:ephemeral: .: {} f:volumeClaimTemplate: .: {} f:metadata: .: {} f:creationTimestamp: {} f:spec: .: {} f:accessModes: {} f:resources: .: {} f:requests: .: {} f:storage: {} f:storageClassName: {} f:volumeMode: {} f:name: {} manager: e2e.test operation: Update time: "2023-03-11T16:44:01Z" - apiVersion: v1 fieldsType: FieldsV1 fieldsV1: f:status: f:conditions: .: {} k:{"type":"PodScheduled"}: .: {} f:lastProbeTime: {} f:lastTransitionTime: {} f:message: {} f:reason: {} f:status: {} f:type: {} manager: kube-scheduler operation: Update subresource: status time: "2023-03-11T16:44:01Z" - apiVersion: v1 fieldsType: FieldsV1 fieldsV1: f:status: f:conditions: k:{"type":"ContainersReady"}: .: {} f:lastProbeTime: {} f:lastTransitionTime: {} f:status: {} f:type: {} k:{"type":"Initialized"}: .: {} f:lastProbeTime: {} f:lastTransitionTime: {} f:status: {} f:type: {} k:{"type":"Ready"}: .: {} f:lastProbeTime: {} f:type: {} f:containerStatuses: {} f:hostIP: {} f:phase: {} f:podIP: {} f:podIPs: .: {} k:{"ip":"100.96.1.67"}: .: {} f:ip: {} f:startTime: {} manager: kubelet operation: Update subresource: status time: "2023-03-11T16:44:14Z" - apiVersion: v1 fieldsType: FieldsV1 fieldsV1: f:status: f:conditions: k:{"type":"Ready"}: f:lastTransitionTime: {} f:status: {} manager: kube-controller-manager operation: Update subresource: status time: "2023-03-11T16:45:36Z" name: inline-volume-tester-7hvcm namespace: ephemeral-234 resourceVersion: "47909" uid: 4510f2fb-5323-4ecc-87bf-7b298c53a2dd spec: containers: - command: - /bin/sh - -c - sleep 10000 image: registry.k8s.io/e2e-test-images/busybox:1.29-4 imagePullPolicy: IfNotPresent name: csi-volume-tester resources: {} terminationMessagePath: /dev/termination-log terminationMessagePolicy: File volumeMounts: - mountPath: /mnt/test-0 name: my-volume-0 readOnly: true - mountPath: /var/run/secrets/kubernetes.io/serviceaccount name: kube-api-access-f2czn readOnly: true dnsPolicy: ClusterFirst enableServiceLinks: true nodeName: i-07e2b6e650d7ef56d nodeSelector: kubernetes.io/os: linux preemptionPolicy: PreemptLowerPriority priority: 0 restartPolicy: Never schedulerName: default-scheduler securityContext: {} serviceAccount: default serviceAccountName: default terminationGracePeriodSeconds: 30 tolerations: - effect: NoExecute key: node.kubernetes.io/not-ready operator: Exists tolerationSeconds: 300 - effect: NoExecute key: node.kubernetes.io/unreachable operator: Exists tolerationSeconds: 300 volumes: - ephemeral: volumeClaimTemplate: metadata: creationTimestamp: null spec: accessModes: - ReadWriteOnce resources: requests: storage: 1Gi storageClassName: ephemeral-234-e2e-scschlr volumeMode: Filesystem name: my-volume-0 - name: kube-api-access-f2czn projected: defaultMode: 420 sources: - serviceAccountToken: expirationSeconds: 3607 path: token - configMap: items: - key: ca.crt path: ca.crt name: kube-root-ca.crt - downwardAPI: items: - fieldRef: apiVersion: v1 fieldPath: metadata.namespace path: namespace status: conditions: - lastProbeTime: null lastTransitionTime: "2023-03-11T16:44:07Z" status: "True" type: Initialized - lastProbeTime: null lastTransitionTime: "2023-03-11T16:45:36Z" status: "False" type: Ready - lastProbeTime: null lastTransitionTime: "2023-03-11T16:44:14Z" status: "True" type: ContainersReady - lastProbeTime: null lastTransitionTime: "2023-03-11T16:44:07Z" status: "True" type: PodScheduled containerStatuses: - containerID: containerd://5e190abd623baf6ecf57baf5a6856d50cafbaaedac2cdfe1914792d691acbb12 image: registry.k8s.io/e2e-test-images/busybox:1.29-4 imageID: registry.k8s.io/e2e-test-images/busybox@sha256:2e0f836850e09b8b7cc937681d6194537a09fbd5f6b9e08f4d646a85128e8937 lastState: {} name: csi-volume-tester ready: true restartCount: 0 started: true state: running: startedAt: "2023-03-11T16:44:13Z" hostIP: 172.20.35.76 phase: Running podIP: 100.96.1.67 podIPs: - ip: 100.96.1.67 qosClass: BestEffort startTime: "2023-03-11T16:44:07Z" to be nil In [It] at: test/e2e/storage/testsuites/provisioning.go:1010 @ 03/11/23 16:54:26.751from junit_01.xml
> Enter [BeforeEach] [Testpattern: Generic Ephemeral-volume (default fs) (late-binding)] ephemeral - test/e2e/storage/framework/testsuite.go:51 @ 03/11/23 16:43:59.27 < Exit [BeforeEach] [Testpattern: Generic Ephemeral-volume (default fs) (late-binding)] ephemeral - test/e2e/storage/framework/testsuite.go:51 @ 03/11/23 16:43:59.27 (0s) > Enter [BeforeEach] [Testpattern: Generic Ephemeral-volume (default fs) (late-binding)] ephemeral - set up framework | framework.go:191 @ 03/11/23 16:43:59.27 STEP: Creating a kubernetes client - test/e2e/framework/framework.go:211 @ 03/11/23 16:43:59.27 Mar 11 16:43:59.270: INFO: >>> kubeConfig: /root/.kube/config STEP: Building a namespace api object, basename ephemeral - test/e2e/framework/framework.go:250 @ 03/11/23 16:43:59.271 STEP: Waiting for a default service account to be provisioned in namespace - test/e2e/framework/framework.go:259 @ 03/11/23 16:43:59.722 STEP: Waiting for kube-root-ca.crt to be provisioned in namespace - test/e2e/framework/framework.go:262 @ 03/11/23 16:44:00.018 < Exit [BeforeEach] [Testpattern: Generic Ephemeral-volume (default fs) (late-binding)] ephemeral - set up framework | framework.go:191 @ 03/11/23 16:44:00.314 (1.044s) > Enter [BeforeEach] [Testpattern: Generic Ephemeral-volume (default fs) (late-binding)] ephemeral - test/e2e/framework/metrics/init/init.go:33 @ 03/11/23 16:44:00.314 < Exit [BeforeEach] [Testpattern: Generic Ephemeral-volume (default fs) (late-binding)] ephemeral - test/e2e/framework/metrics/init/init.go:33 @ 03/11/23 16:44:00.314 (0s) > Enter [It] should create read-only inline ephemeral volume - test/e2e/storage/testsuites/ephemeral.go:182 @ 03/11/23 16:44:00.314 Mar 11 16:44:00.616: INFO: Pod inline-volume-frt6r has the following logs: Mar 11 16:44:00.765: INFO: Deleting pod "inline-volume-frt6r" in namespace "ephemeral-234" Mar 11 16:44:00.917: INFO: Wait up to 5m0s for pod "inline-volume-frt6r" to be fully deleted Mar 11 16:44:01.066: INFO: Creating resource for dynamic PV Mar 11 16:44:01.066: INFO: Using claimSize:1Gi, test suite supported size:{ }, driver(ebs.csi.aws.com) supported size:{ } STEP: creating a StorageClass ephemeral-234-e2e-scschlr - test/e2e/storage/framework/volume_resource.go:102 @ 03/11/23 16:44:01.067 STEP: checking the requested inline volume exists in the pod running on node {Name: Selector:map[kubernetes.io/os:linux] Affinity:nil} - test/e2e/storage/testsuites/ephemeral.go:389 @ 03/11/23 16:44:01.222 Mar 11 16:44:14.571: INFO: ExecWithOptions {Command:[/bin/sh -c mount | grep /mnt/test | grep ro,] Namespace:ephemeral-234 PodName:inline-volume-tester-7hvcm ContainerName:csi-volume-tester Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false Quiet:false} Mar 11 16:44:14.571: INFO: >>> kubeConfig: /root/.kube/config Mar 11 16:44:14.572: INFO: ExecWithOptions: Clientset creation Mar 11 16:44:14.572: INFO: ExecWithOptions: execute(POST https://api.e2e-e2e-kops-aws-karpenter.test-cncf-aws.k8s.io/api/v1/namespaces/ephemeral-234/pods/inline-volume-tester-7hvcm/exec?command=%2Fbin%2Fsh&command=-c&command=mount+%7C+grep+%2Fmnt%2Ftest+%7C+grep+ro%2C&container=csi-volume-tester&container=csi-volume-tester&stderr=true&stdout=true) Mar 11 16:44:15.689: INFO: Pod inline-volume-tester-7hvcm has the following logs: Mar 11 16:44:15.988: INFO: Deleting pod "inline-volume-tester-7hvcm" in namespace "ephemeral-234" Mar 11 16:44:16.142: INFO: Wait up to 5m0s for pod "inline-volume-tester-7hvcm" to be fully deleted Mar 11 16:49:16.145: INFO: Failed inside E2E framework: k8s.io/kubernetes/test/e2e/framework/pod.WaitForPodNotFoundInNamespace({0x7f44bc56ed48, 0xc0054016c0}, {0x722adb8?, 0xc0030d1860}, {0xc00397df40, 0x1a}, {0xc005814ed3, 0xd}, 0x0?) test/e2e/framework/pod/wait.go:538 +0x190 k8s.io/kubernetes/test/e2e/storage/testsuites.StopPodAndDependents({0x7f44bc56ed48, 0xc0054016c0}, {0x722adb8?, 0xc0030d1860?}, 0xc0010f6aa0, 0xc0027c4d80) test/e2e/storage/testsuites/provisioning.go:1010 +0xa8e k8s.io/kubernetes/test/e2e/storage/testsuites.EphemeralTest.TestEphemeral({{0x722adb8, 0xc0030d1860}, 0xc0010f6aa0, {0xc003791090, 0xd}, {0x0, 0x0}, 0xc0039ca000, {{0x0, 0x0}, ...}, ...}, ...) test/e2e/storage/testsuites/ephemeral.go:434 +0x7aa k8s.io/kubernetes/test/e2e/storage/testsuites.(*ephemeralTestSuite).DefineTests.func3({0x7f44bc56ed48, 0xc0054016c0}) test/e2e/storage/testsuites/ephemeral.go:200 +0x1d4 Mar 11 16:49:26.300: INFO: Error getting logs for pod inline-volume-tester-7hvcm: an error on the server ("unknown") has prevented the request from succeeding (get pods inline-volume-tester-7hvcm) Mar 11 16:49:26.599: INFO: Deleting pod "inline-volume-tester-7hvcm" in namespace "ephemeral-234" Mar 11 16:49:26.748: INFO: Wait up to 5m0s for pod "inline-volume-tester-7hvcm" to be fully deleted Mar 11 16:54:26.750: INFO: Failed inside E2E framework: k8s.io/kubernetes/test/e2e/framework/pod.WaitForPodNotFoundInNamespace({0x7f44bc56ed48, 0xc0054016c0}, {0x722adb8?, 0xc0030d1860}, {0xc00397df40, 0x1a}, {0xc005814ed3, 0xd}, 0x0?) test/e2e/framework/pod/wait.go:538 +0x190 k8s.io/kubernetes/test/e2e/storage/testsuites.StopPodAndDependents({0x7f44bc56ed48, 0xc0054016c0}, {0x722adb8?, 0xc0030d1860?}, 0xc0010f6aa0, 0xc0027c4d80) test/e2e/storage/testsuites/provisioning.go:1010 +0xa8e k8s.io/kubernetes/test/e2e/storage/testsuites.EphemeralTest.TestEphemeral.func1() test/e2e/storage/testsuites/ephemeral.go:421 +0x38 panic({0x65bb9e0, 0xc000338a80}) /usr/local/go/src/runtime/panic.go:884 +0x213 k8s.io/kubernetes/test/e2e/framework.ExpectNoErrorWithOffset(0x1, {0x71b9f20, 0xc004b8d860}, {0x0?, 0xc00397df40?, 0x1a?}) test/e2e/framework/expect.go:363 +0x363 k8s.io/kubernetes/test/e2e/framework.ExpectNoError(...) test/e2e/framework/expect.go:320 k8s.io/kubernetes/test/e2e/storage/testsuites.StopPodAndDependents({0x7f44bc56ed48, 0xc0054016c0}, {0x722adb8?, 0xc0030d1860?}, 0xc0010f6aa0, 0xc0027c4d80) test/e2e/storage/testsuites/provisioning.go:1010 +0xaa5 k8s.io/kubernetes/test/e2e/storage/testsuites.EphemeralTest.TestEphemeral({{0x722adb8, 0xc0030d1860}, 0xc0010f6aa0, {0xc003791090, 0xd}, {0x0, 0x0}, 0xc0039ca000, {{0x0, 0x0}, ...}, ...}, ...) test/e2e/storage/testsuites/ephemeral.go:434 +0x7aa k8s.io/kubernetes/test/e2e/storage/testsuites.(*ephemeralTestSuite).DefineTests.func3({0x7f44bc56ed48, 0xc0054016c0}) test/e2e/storage/testsuites/ephemeral.go:200 +0x1d4 [FAILED] expected pod to not be found: Timed out after 300.000s. Expected <*v1.Pod | 0xc003553680>: metadata: creationTimestamp: "2023-03-11T16:44:01Z" deletionGracePeriodSeconds: 30 deletionTimestamp: "2023-03-11T16:44:46Z" finalizers: - foregroundDeletion generateName: inline-volume-tester- labels: app: inline-volume-tester managedFields: - apiVersion: v1 fieldsType: FieldsV1 fieldsV1: f:metadata: f:generateName: {} f:labels: .: {} f:app: {} f:spec: f:containers: k:{"name":"csi-volume-tester"}: .: {} f:command: {} f:image: {} f:imagePullPolicy: {} f:name: {} f:resources: {} f:terminationMessagePath: {} f:terminationMessagePolicy: {} f:volumeMounts: .: {} k:{"mountPath":"/mnt/test-0"}: .: {} f:mountPath: {} f:name: {} f:readOnly: {} f:dnsPolicy: {} f:enableServiceLinks: {} f:nodeSelector: {} f:restartPolicy: {} f:schedulerName: {} f:securityContext: {} f:terminationGracePeriodSeconds: {} f:volumes: .: {} k:{"name":"my-volume-0"}: .: {} f:ephemeral: .: {} f:volumeClaimTemplate: .: {} f:metadata: .: {} f:creationTimestamp: {} f:spec: .: {} f:accessModes: {} f:resources: .: {} f:requests: .: {} f:storage: {} f:storageClassName: {} f:volumeMode: {} f:name: {} manager: e2e.test operation: Update time: "2023-03-11T16:44:01Z" - apiVersion: v1 fieldsType: FieldsV1 fieldsV1: f:status: f:conditions: .: {} k:{"type":"PodScheduled"}: .: {} f:lastProbeTime: {} f:lastTransitionTime: {} f:message: {} f:reason: {} f:status: {} f:type: {} manager: kube-scheduler operation: Update subresource: status time: "2023-03-11T16:44:01Z" - apiVersion: v1 fieldsType: FieldsV1 fieldsV1: f:status: f:conditions: k:{"type":"ContainersReady"}: .: {} f:lastProbeTime: {} f:lastTransitionTime: {} f:status: {} f:type: {} k:{"type":"Initialized"}: .: {} f:lastProbeTime: {} f:lastTransitionTime: {} f:status: {} f:type: {} k:{"type":"Ready"}: .: {} f:lastProbeTime: {} f:type: {} f:containerStatuses: {} f:hostIP: {} f:phase: {} f:podIP: {} f:podIPs: .: {} k:{"ip":"100.96.1.67"}: .: {} f:ip: {} f:startTime: {} manager: kubelet operation: Update subresource: status time: "2023-03-11T16:44:14Z" - apiVersion: v1 fieldsType: FieldsV1 fieldsV1: f:status: f:conditions: k:{"type":"Ready"}: f:lastTransitionTime: {} f:status: {} manager: kube-controller-manager operation: Update subresource: status time: "2023-03-11T16:45:36Z" name: inline-volume-tester-7hvcm namespace: ephemeral-234 resourceVersion: "47909" uid: 4510f2fb-5323-4ecc-87bf-7b298c53a2dd spec: containers: - command: - /bin/sh - -c - sleep 10000 image: registry.k8s.io/e2e-test-images/busybox:1.29-4 imagePullPolicy: IfNotPresent name: csi-volume-tester resources: {} terminationMessagePath: /dev/termination-log terminationMessagePolicy: File volumeMounts: - mountPath: /mnt/test-0 name: my-volume-0 readOnly: true - mountPath: /var/run/secrets/kubernetes.io/serviceaccount name: kube-api-access-f2czn readOnly: true dnsPolicy: ClusterFirst enableServiceLinks: true nodeName: i-07e2b6e650d7ef56d nodeSelector: kubernetes.io/os: linux preemptionPolicy: PreemptLowerPriority priority: 0 restartPolicy: Never schedulerName: default-scheduler securityContext: {} serviceAccount: default serviceAccountName: default terminationGracePeriodSeconds: 30 tolerations: - effect: NoExecute key: node.kubernetes.io/not-ready operator: Exists tolerationSeconds: 300 - effect: NoExecute key: node.kubernetes.io/unreachable operator: Exists tolerationSeconds: 300 volumes: - ephemeral: volumeClaimTemplate: metadata: creationTimestamp: null spec: accessModes: - ReadWriteOnce resources: requests: storage: 1Gi storageClassName: ephemeral-234-e2e-scschlr volumeMode: Filesystem name: my-volume-0 - name: kube-api-access-f2czn projected: defaultMode: 420 sources: - serviceAccountToken: expirationSeconds: 3607 path: token - configMap: items: - key: ca.crt path: ca.crt name: kube-root-ca.crt - downwardAPI: items: - fieldRef: apiVersion: v1 fieldPath: metadata.namespace path: namespace status: conditions: - lastProbeTime: null lastTransitionTime: "2023-03-11T16:44:07Z" status: "True" type: Initialized - lastProbeTime: null lastTransitionTime: "2023-03-11T16:45:36Z" status: "False" type: Ready - lastProbeTime: null lastTransitionTime: "2023-03-11T16:44:14Z" status: "True" type: ContainersReady - lastProbeTime: null lastTransitionTime: "2023-03-11T16:44:07Z" status: "True" type: PodScheduled containerStatuses: - containerID: containerd://5e190abd623baf6ecf57baf5a6856d50cafbaaedac2cdfe1914792d691acbb12 image: registry.k8s.io/e2e-test-images/busybox:1.29-4 imageID: registry.k8s.io/e2e-test-images/busybox@sha256:2e0f836850e09b8b7cc937681d6194537a09fbd5f6b9e08f4d646a85128e8937 lastState: {} name: csi-volume-tester ready: true restartCount: 0 started: true state: running: startedAt: "2023-03-11T16:44:13Z" hostIP: 172.20.35.76 phase: Running podIP: 100.96.1.67 podIPs: - ip: 100.96.1.67 qosClass: BestEffort startTime: "2023-03-11T16:44:07Z" to be nil In [It] at: test/e2e/storage/testsuites/provisioning.go:1010 @ 03/11/23 16:54:26.751 < Exit [It] should create read-only inline ephemeral volume - test/e2e/storage/testsuites/ephemeral.go:182 @ 03/11/23 16:54:26.751 (10m26.437s) > Enter [AfterEach] [Testpattern: Generic Ephemeral-volume (default fs) (late-binding)] ephemeral - test/e2e/framework/node/init/init.go:33 @ 03/11/23 16:54:26.751 Mar 11 16:54:26.751: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready Mar 11 16:54:26.901: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:54:26.901: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:54:29.051: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:54:29.051: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:54:31.052: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:54:31.052: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:54:33.051: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:54:33.051: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:54:35.052: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:54:35.052: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:54:37.052: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:54:37.052: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:54:39.052: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:54:39.052: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:54:41.051: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:54:41.051: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:54:43.052: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:54:43.052: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:54:45.051: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:54:45.051: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:54:47.051: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:54:47.051: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:54:49.051: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:54:49.051: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:54:51.052: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:54:51.052: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:54:53.052: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:54:53.052: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:54:55.052: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:54:55.052: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:54:57.052: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:54:57.052: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:54:59.052: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:54:59.052: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:55:01.052: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:55:01.052: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:55:03.052: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:55:03.052: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:55:05.052: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:55:05.052: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:55:07.052: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:55:07.052: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:55:09.052: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:55:09.052: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:55:11.052: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:55:11.052: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:55:13.052: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:55:13.052: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:55:15.052: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:55:15.052: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:55:17.052: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:55:17.052: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:55:19.052: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:55:19.052: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:55:21.052: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:55:21.052: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:55:23.052: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:55:23.052: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:55:25.052: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:55:25.052: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:55:27.052: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:55:27.052: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:55:29.052: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:55:29.052: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:55:31.052: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:55:31.052: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:55:33.052: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:55:33.052: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:55:35.052: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:55:35.052: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:55:37.052: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:55:37.052: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:55:39.052: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:55:39.052: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:55:41.051: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:55:41.051: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:55:43.051: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:55:43.051: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:55:45.052: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:55:45.052: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:55:47.052: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:55:47.052: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:55:49.052: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:55:49.052: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:55:51.052: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:55:51.052: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:55:53.052: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:55:53.052: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:55:55.052: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:55:55.052: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:55:57.052: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:55:57.052: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:55:59.051: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:55:59.051: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:56:01.052: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:56:01.052: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:56:03.052: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:56:03.052: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:56:05.052: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:56:05.052: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:56:07.051: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:56:07.051: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:56:09.052: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:56:09.052: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:56:11.051: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:56:11.051: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:56:13.051: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:56:13.051: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:56:15.051: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:56:15.051: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:56:17.052: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:56:17.052: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:56:19.051: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:56:19.051: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:56:21.051: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:56:21.051: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:56:23.051: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:56:23.051: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:56:25.051: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:56:25.051: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:56:27.051: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:56:27.051: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:56:29.052: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:56:29.052: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:56:31.051: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:56:31.051: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:56:33.051: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:56:33.051: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:56:35.051: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:56:35.051: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:56:37.051: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:56:37.051: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:56:39.051: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:56:39.051: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:56:41.052: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:56:41.052: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:56:43.052: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:56:43.052: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:56:45.051: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:56:45.051: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:56:47.051: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:56:47.051: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:56:49.051: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:56:49.051: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:56:51.051: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:56:51.051: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:56:53.051: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:56:53.051: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:56:55.052: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:56:55.052: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:56:57.051: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:56:57.052: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:56:59.051: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:56:59.051: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:57:01.051: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:57:01.051: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:57:03.052: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:57:03.052: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:57:05.051: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:57:05.051: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:57:07.052: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:57:07.052: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:57:09.051: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:57:09.051: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:57:11.051: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:57:11.051: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:57:13.051: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:57:13.051: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:57:15.051: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:57:15.051: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:57:17.051: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:57:17.051: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:57:19.051: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:57:19.051: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:57:21.051: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:57:21.051: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:57:23.052: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:57:23.052: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:57:25.051: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:57:25.051: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:57:27.052: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:57:27.052: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:57:27.201: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:57:27.201: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure < Exit [AfterEach] [Testpattern: Generic Ephemeral-volume (default fs) (late-binding)] ephemeral - test/e2e/framework/node/init/init.go:33 @ 03/11/23 16:57:27.201 (3m0.45s) > Enter [DeferCleanup (Each)] [Testpattern: Generic Ephemeral-volume (default fs) (late-binding)] ephemeral - test/e2e/storage/testsuites/ephemeral.go:188 @ 03/11/23 16:57:27.201 STEP: Deleting sc - test/e2e/storage/framework/volume_resource.go:228 @ 03/11/23 16:57:27.201 < Exit [DeferCleanup (Each)] [Testpattern: Generic Ephemeral-volume (default fs) (late-binding)] ephemeral - test/e2e/storage/testsuites/ephemeral.go:188 @ 03/11/23 16:57:27.351 (150ms) > Enter [DeferCleanup (Each)] [Testpattern: Generic Ephemeral-volume (default fs) (late-binding)] ephemeral - test/e2e/framework/metrics/init/init.go:35 @ 03/11/23 16:57:27.351 < Exit [DeferCleanup (Each)] [Testpattern: Generic Ephemeral-volume (default fs) (late-binding)] ephemeral - test/e2e/framework/metrics/init/init.go:35 @ 03/11/23 16:57:27.351 (0s) > Enter [DeferCleanup (Each)] [Testpattern: Generic Ephemeral-volume (default fs) (late-binding)] ephemeral - dump namespaces | framework.go:209 @ 03/11/23 16:57:27.351 STEP: dump namespace information after failure - test/e2e/framework/framework.go:288 @ 03/11/23 16:57:27.351 STEP: Collecting events from namespace "ephemeral-234". - test/e2e/framework/debug/dump.go:42 @ 03/11/23 16:57:27.351 STEP: Found 16 events. - test/e2e/framework/debug/dump.go:46 @ 03/11/23 16:57:27.5 Mar 11 16:57:27.500: INFO: At 2023-03-11 16:44:00 +0000 UTC - event for inline-volume-frt6r: {default-scheduler } FailedScheduling: 0/3 nodes are available: waiting for ephemeral volume controller to create the persistentvolumeclaim "inline-volume-frt6r-my-volume". preemption: 0/3 nodes are available: 3 No preemption victims found for incoming pod.. Mar 11 16:57:27.500: INFO: At 2023-03-11 16:44:00 +0000 UTC - event for inline-volume-frt6r-my-volume: {persistentvolume-controller } ProvisioningFailed: storageclass.storage.k8s.io "no-such-storage-class" not found Mar 11 16:57:27.500: INFO: At 2023-03-11 16:44:01 +0000 UTC - event for inline-volume-tester-7hvcm: {default-scheduler } FailedScheduling: 0/3 nodes are available: waiting for ephemeral volume controller to create the persistentvolumeclaim "inline-volume-tester-7hvcm-my-volume-0". preemption: 0/3 nodes are available: 3 No preemption victims found for incoming pod.. Mar 11 16:57:27.500: INFO: At 2023-03-11 16:44:01 +0000 UTC - event for inline-volume-tester-7hvcm-my-volume-0: {persistentvolume-controller } WaitForPodScheduled: waiting for pod inline-volume-tester-7hvcm to be scheduled Mar 11 16:57:27.500: INFO: At 2023-03-11 16:44:02 +0000 UTC - event for inline-volume-tester-7hvcm: {karpenter } Nominate: Pod should schedule on i-07e2b6e650d7ef56d Mar 11 16:57:27.500: INFO: At 2023-03-11 16:44:03 +0000 UTC - event for inline-volume-tester-7hvcm-my-volume-0: {persistentvolume-controller } ExternalProvisioning: waiting for a volume to be created, either by external provisioner "ebs.csi.aws.com" or manually created by system administrator Mar 11 16:57:27.500: INFO: At 2023-03-11 16:44:03 +0000 UTC - event for inline-volume-tester-7hvcm-my-volume-0: {ebs.csi.aws.com_ebs-csi-controller-79785c665b-lpmmf_0e0dbfd5-dc63-4852-b921-aa9f41d3903f } Provisioning: External provisioner is provisioning volume for claim "ephemeral-234/inline-volume-tester-7hvcm-my-volume-0" Mar 11 16:57:27.500: INFO: At 2023-03-11 16:44:06 +0000 UTC - event for inline-volume-tester-7hvcm-my-volume-0: {ebs.csi.aws.com_ebs-csi-controller-79785c665b-lpmmf_0e0dbfd5-dc63-4852-b921-aa9f41d3903f } ProvisioningSucceeded: Successfully provisioned volume pvc-c724d977-74bf-4584-af55-fc714a6928d9 Mar 11 16:57:27.500: INFO: At 2023-03-11 16:44:07 +0000 UTC - event for inline-volume-tester-7hvcm: {default-scheduler } Scheduled: Successfully assigned ephemeral-234/inline-volume-tester-7hvcm to i-07e2b6e650d7ef56d Mar 11 16:57:27.500: INFO: At 2023-03-11 16:44:09 +0000 UTC - event for inline-volume-tester-7hvcm: {attachdetach-controller } SuccessfulAttachVolume: AttachVolume.Attach succeeded for volume "pvc-c724d977-74bf-4584-af55-fc714a6928d9" Mar 11 16:57:27.500: INFO: At 2023-03-11 16:44:13 +0000 UTC - event for inline-volume-tester-7hvcm: {kubelet i-07e2b6e650d7ef56d} Pulled: Container image "registry.k8s.io/e2e-test-images/busybox:1.29-4" already present on machine Mar 11 16:57:27.500: INFO: At 2023-03-11 16:44:13 +0000 UTC - event for inline-volume-tester-7hvcm: {kubelet i-07e2b6e650d7ef56d} Created: Created container csi-volume-tester Mar 11 16:57:27.500: INFO: At 2023-03-11 16:44:13 +0000 UTC - event for inline-volume-tester-7hvcm: {kubelet i-07e2b6e650d7ef56d} Started: Started container csi-volume-tester Mar 11 16:57:27.500: INFO: At 2023-03-11 16:44:16 +0000 UTC - event for inline-volume-tester-7hvcm: {kubelet i-07e2b6e650d7ef56d} Killing: Stopping container csi-volume-tester Mar 11 16:57:27.500: INFO: At 2023-03-11 16:45:36 +0000 UTC - event for inline-volume-tester-7hvcm: {node-controller } NodeNotReady: Node is not ready Mar 11 16:57:27.500: INFO: At 2023-03-11 16:50:46 +0000 UTC - event for inline-volume-tester-7hvcm: {taint-controller } TaintManagerEviction: Marking for deletion Pod ephemeral-234/inline-volume-tester-7hvcm Mar 11 16:57:27.649: INFO: POD NODE PHASE GRACE CONDITIONS Mar 11 16:57:27.649: INFO: inline-volume-tester-7hvcm i-07e2b6e650d7ef56d Running 30s [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2023-03-11 16:44:07 +0000 UTC } {Ready False 0001-01-01 00:00:00 +0000 UTC 2023-03-11 16:45:36 +0000 UTC } {ContainersReady True 0001-01-01 00:00:00 +0000 UTC 2023-03-11 16:44:14 +0000 UTC } {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2023-03-11 16:44:07 +0000 UTC } {DisruptionTarget True 0001-01-01 00:00:00 +0000 UTC 2023-03-11 16:50:46 +0000 UTC DeletionByTaintManager Taint manager: deleting due to NoExecute taint}] Mar 11 16:57:27.649: INFO: Mar 11 16:57:37.799: INFO: Unable to fetch ephemeral-234/inline-volume-tester-7hvcm/csi-volume-tester logs: an error on the server ("unknown") has prevented the request from succeeding (get pods inline-volume-tester-7hvcm) Mar 11 16:57:37.948: INFO: Logging node info for node i-04523d1dffb0fe725 Mar 11 16:57:38.097: INFO: Node Info: &Node{ObjectMeta:{i-04523d1dffb0fe725 5d37640f-a7d9-4c26-86c5-9da21f7ccc18 51243 0 2023-03-11 16:24:20 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:arm64 beta.kubernetes.io/instance-type:t4g.small beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-northeast-1 failure-domain.beta.kubernetes.io/zone:ap-northeast-1a karpenter.k8s.aws/instance-category:t karpenter.k8s.aws/instance-cpu:2 karpenter.k8s.aws/instance-family:t4g karpenter.k8s.aws/instance-generation:4 karpenter.k8s.aws/instance-hypervisor:nitro karpenter.k8s.aws/instance-memory:2048 karpenter.k8s.aws/instance-pods:11 karpenter.k8s.aws/instance-size:small karpenter.sh/capacity-type:spot karpenter.sh/initialized:true karpenter.sh/provisioner-name:nodes kubernetes.io/arch:arm64 kubernetes.io/hostname:i-04523d1dffb0fe725 kubernetes.io/os:linux node-role.kubernetes.io/node: node-role.kubernetes.io/spot-worker:true node.kubernetes.io/instance-type:t4g.small topology.ebs.csi.aws.com/zone:ap-northeast-1a topology.hostpath.csi/node:i-04523d1dffb0fe725 topology.kubernetes.io/region:ap-northeast-1 topology.kubernetes.io/zone:ap-northeast-1a] map[csi.volume.kubernetes.io/nodeid:{"csi-hostpath-read-write-once-pod-7172":"i-04523d1dffb0fe725","csi-hostpath-volume-expand-1673":"i-04523d1dffb0fe725","ebs.csi.aws.com":"i-04523d1dffb0fe725"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [{karpenter.sh/v1alpha5 Provisioner nodes ac7283b2-eed0-4a9f-9ddd-616f33a0991a <nil> 0xc003258185}] [karpenter.sh/termination] [{aws-cloud-controller-manager Update v1 2023-03-11 16:24:20 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{}}}} } {kops-controller Update v1 2023-03-11 16:24:20 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{},"f:node-role.kubernetes.io/spot-worker":{}}}} } {karpenter Update v1 2023-03-11 16:26:12 +0000 UTC FieldsV1 {"f:metadata":{"f:finalizers":{".":{},"v:\"karpenter.sh/termination\"":{}},"f:labels":{".":{},"f:karpenter.k8s.aws/instance-category":{},"f:karpenter.k8s.aws/instance-cpu":{},"f:karpenter.k8s.aws/instance-family":{},"f:karpenter.k8s.aws/instance-generation":{},"f:karpenter.k8s.aws/instance-hypervisor":{},"f:karpenter.k8s.aws/instance-memory":{},"f:karpenter.k8s.aws/instance-pods":{},"f:karpenter.k8s.aws/instance-size":{},"f:karpenter.sh/capacity-type":{},"f:karpenter.sh/initialized":{},"f:karpenter.sh/provisioner-name":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}},"f:ownerReferences":{".":{},"k:{\"uid\":\"ac7283b2-eed0-4a9f-9ddd-616f33a0991a\"}":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-03-11 16:26:50 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{".":{},"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"InternalIP\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kubelet Update v1 2023-03-11 16:44:47 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{"f:kubernetes.io/hostname":{},"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{".":{},"f:cpu":{},"f:ephemeral-storage":{},"f:hugepages-1Gi":{},"f:hugepages-2Mi":{},"f:hugepages-32Mi":{},"f:hugepages-64Ki":{},"f:memory":{},"f:pods":{}},"f:capacity":{".":{},"f:cpu":{},"f:ephemeral-storage":{},"f:hugepages-1Gi":{},"f:hugepages-2Mi":{},"f:hugepages-32Mi":{},"f:hugepages-64Ki":{},"f:memory":{},"f:pods":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{}}},"f:daemonEndpoints":{"f:kubeletEndpoint":{"f:Port":{}}},"f:images":{},"f:nodeInfo":{"f:architecture":{},"f:bootID":{},"f:containerRuntimeVersion":{},"f:kernelVersion":{},"f:kubeProxyVersion":{},"f:kubeletVersion":{},"f:machineID":{},"f:operatingSystem":{},"f:osImage":{},"f:systemUUID":{}},"f:volumesInUse":{}}} status} {kube-controller-manager Update v1 2023-03-11 16:45:36 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:node.alpha.kubernetes.io/ttl":{}},"f:labels":{"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.2.0/24\"":{}},"f:taints":{}}} } {kube-controller-manager Update v1 2023-03-11 16:57:03 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{".":{},"k:{\"type\":\"DiskPressure\"}":{".":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"MemoryPressure\"}":{".":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"PIDPressure\"}":{".":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Ready\"}":{".":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status}]},Spec:NodeSpec{PodCIDR:100.96.2.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-northeast-1a/i-04523d1dffb0fe725,Unschedulable:false,Taints:[]Taint{Taint{Key:node.kubernetes.io/unreachable,Value:,Effect:NoSchedule,TimeAdded:2023-03-11 16:45:31 +0000 UTC,},Taint{Key:node.kubernetes.io/unreachable,Value:,Effect:NoExecute,TimeAdded:2023-03-11 16:45:36 +0000 UTC,},},ConfigSource:nil,PodCIDRs:[100.96.2.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49756430336 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-32Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-64Ki: {{0 0} {<nil>} 0 DecimalSI},memory: {{1941540864 0} {<nil>} 1896036Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44780787229 0} {<nil>} 44780787229 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-32Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-64Ki: {{0 0} {<nil>} 0 DecimalSI},memory: {{1836683264 0} {<nil>} 1793636Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:Ready,Status:Unknown,LastHeartbeatTime:2023-03-11 16:44:47 +0000 UTC,LastTransitionTime:2023-03-11 16:45:31 +0000 UTC,Reason:NodeStatusUnknown,Message:Kubelet stopped posting node status.,},NodeCondition{Type:MemoryPressure,Status:Unknown,LastHeartbeatTime:2023-03-11 16:44:47 +0000 UTC,LastTransitionTime:2023-03-11 16:45:31 +0000 UTC,Reason:NodeStatusUnknown,Message:Kubelet stopped posting node status.,},NodeCondition{Type:DiskPressure,Status:Unknown,LastHeartbeatTime:2023-03-11 16:44:47 +0000 UTC,LastTransitionTime:2023-03-11 16:45:31 +0000 UTC,Reason:NodeStatusUnknown,Message:Kubelet stopped posting node status.,},NodeCondition{Type:PIDPressure,Status:Unknown,LastHeartbeatTime:2023-03-11 16:44:47 +0000 UTC,LastTransitionTime:2023-03-11 16:45:31 +0000 UTC,Reason:NodeStatusUnknown,Message:Kubelet stopped posting node status.,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.48.168,},NodeAddress{Type:ExternalIP,Address:52.199.253.156,},NodeAddress{Type:InternalDNS,Address:i-04523d1dffb0fe725.ap-northeast-1.compute.internal,},NodeAddress{Type:Hostname,Address:i-04523d1dffb0fe725.ap-northeast-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-52-199-253-156.ap-northeast-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2c8fb0e6b2e9c4af310f65b6161cd8,SystemUUID:ec2c8fb0-e6b2-e9c4-af31-0f65b6161cd8,BootID:a3b8de83-558b-44c4-a46a-d097c3f98225,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 22.04.2 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.27.0-alpha.3.420+bb6c6fad2c19b3,KubeProxyVersion:v1.27.0-alpha.3.420+bb6c6fad2c19b3,OperatingSystem:linux,Architecture:arm64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:157636062,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/jessie-dnsutils@sha256:24aaf2626d6b27864c29de2097e8bbb840b3a414271bf7c8995e431e47d8408e registry.k8s.io/e2e-test-images/jessie-dnsutils:1.7],SizeBytes:116764683,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:51eae8381dcb1078289fa7b4f3df2630cdc18d09fb56f8e56b41c40e191d6c83 registry.k8s.io/etcd:3.5.7-0],SizeBytes:80665728,},ContainerImage{Names:[registry.k8s.io/kube-proxy-arm64:v1.27.0-alpha.3.420_bb6c6fad2c19b3],SizeBytes:60947298,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:16bbf38c463a4223d8cfe4da12bc61010b082a79b4bb003e2d3ba3ece5dd5f9e registry.k8s.io/e2e-test-images/agnhost:2.43],SizeBytes:49808364,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nautilus@sha256:80ba6c8c44f9623f06e868a1aa66026c8ec438ad814f9ec95e9333b415fe3550 registry.k8s.io/e2e-test-images/nautilus:1.7],SizeBytes:47953400,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:3fe7acf013d1264ffded116b80a73dc129a449b0fccdb8d21af8279f2233f36e registry.k8s.io/e2e-test-images/httpd:2.4.39-4],SizeBytes:41528818,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:148b022f5c5da426fc2f3c14b5c0867e58ef05961510c84749ac1fddcb0fef22 registry.k8s.io/e2e-test-images/httpd:2.4.38-4],SizeBytes:40343601,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:28932904,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:ee3b525d5b89db99da3b8eb521d9cd90cb6e9ef0fbb651e98bb37be78d36b5b8 registry.k8s.io/sig-storage/csi-provisioner:v3.3.0],SizeBytes:23945474,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/sample-apiserver@sha256:8d70890151aa5d096f331cb9da1b9cd5be0412b7363fe67b5c3befdcaa2a28d0 registry.k8s.io/e2e-test-images/sample-apiserver:1.17.7],SizeBytes:23857462,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:425d8f1b769398127767b06ed97ce62578a3179bcb99809ce93a1649e025ffe7 registry.k8s.io/sig-storage/csi-resizer:v1.6.0],SizeBytes:22678632,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:291334908ddf71a4661fd7f6d9d97274de8a5378a2b6fdfeb2ce73414a34f82f registry.k8s.io/sig-storage/csi-snapshotter:v6.1.0],SizeBytes:22434180,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:9a685020911e2725ad019dbce6e4a5ab93d51e3d4557f115e64343345e05781b registry.k8s.io/sig-storage/csi-attacher:v4.0.0],SizeBytes:22394501,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:122bfb8c1edabb3c0edd63f06523e6940d958d19b3957dc7b1d6f81e9f1f6119 registry.k8s.io/sig-storage/csi-provisioner:v3.1.0],SizeBytes:21735644,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:9ebbf9f023e7b41ccee3d52afe39a89e3ddacdbb69269d583abfc25847cfd9e4 registry.k8s.io/sig-storage/csi-resizer:v1.4.0],SizeBytes:20848413,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:8b9c313c05f54fb04f8d430896f5f5904b6cb157df261501b29adc04d2b2dc7b registry.k8s.io/sig-storage/csi-attacher:v3.4.0],SizeBytes:20565480,},ContainerImage{Names:[registry.k8s.io/cpa/cluster-proportional-autoscaler@sha256:68d396900aeaa072c1f27289485fdac29834045a6f3ffe369bf389d830ef572d registry.k8s.io/cpa/cluster-proportional-autoscaler:1.8.6],SizeBytes:19007623,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nonroot@sha256:ee9f50b3c64b174d296d91ca9f69a914ac30e59095dfb462b2b518ad28a63655 registry.k8s.io/e2e-test-images/nonroot:1.4],SizeBytes:18561972,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:92257881c1d6493cf18299a24af42330f891166560047902b8d431fb66b01af5 registry.k8s.io/sig-storage/hostpathplugin:v1.9.0],SizeBytes:18267611,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:9b2d9abda017c28b12f84a344f57cd73fbdb6c2bd7dd5bdd5018246ad1093ba6 registry.k8s.io/sig-storage/hostpathplugin:v1.11.0],SizeBytes:18159085,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:017727efcfeb7d053af68e51436ce8e65edbc6ca573720afb4f79c8594036955 registry.k8s.io/coredns/coredns:v1.10.0],SizeBytes:13766234,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:8502345,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:933940f13b3ea0abc62e656c1aa5c5b47c04b15d71250413a6b821bd0c58b94e registry.k8s.io/sig-storage/livenessprobe:v2.7.0],SizeBytes:8223819,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:7685305,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:db048754ae68ae337d8fa96494c96d2a1204c3320f5dcf7e8e71085adec85da6 registry.k8s.io/e2e-test-images/nginx:1.15-4],SizeBytes:6883347,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:5c99cf6a02adda929b10321dbf4ecfa00d87be9ba4fb456006237d530ab4baa1 registry.k8s.io/e2e-test-images/nginx:1.14-4],SizeBytes:6860089,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nonewprivs@sha256:8ac1264691820febacf3aea5d152cbde6d10685731ec14966a9401c6f47a68ac registry.k8s.io/e2e-test-images/nonewprivs:1.3],SizeBytes:5025027,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf registry.k8s.io/e2e-test-images/busybox:1.29-2],SizeBytes:773367,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:2e0f836850e09b8b7cc937681d6194537a09fbd5f6b9e08f4d646a85128e8937 registry.k8s.io/e2e-test-images/busybox:1.29-4],SizeBytes:772933,},ContainerImage{Names:[registry.k8s.io/pause@sha256:7031c1b283388d2c2e09b57badb803c05ebed362dc88d84b480cc47f72a21097 registry.k8s.io/pause:3.9],SizeBytes:268051,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:253553,},},VolumesInUse:[kubernetes.io/csi/csi-hostpath-read-write-once-pod-7172^038fde52-c02c-11ed-b028-5eca9e37a97f kubernetes.io/csi/csi-hostpath-volume-expand-1673^f8193b68-c02b-11ed-8784-9e8522e7cc37 kubernetes.io/csi/ebs.csi.aws.com^vol-0293af09ce81fe75e],VolumesAttached:[]AttachedVolume{},Config:nil,},} Mar 11 16:57:38.097: INFO: Logging kubelet events for node i-04523d1dffb0fe725 Mar 11 16:57:38.249: INFO: Logging pods the kubelet thinks is on node i-04523d1dffb0fe725 Mar 11 16:57:48.399: INFO: Unable to retrieve kubelet pods for node i-04523d1dffb0fe725: error trying to reach service: net/http: TLS handshake timeout Mar 11 16:57:48.399: INFO: Logging node info for node i-07e2b6e650d7ef56d Mar 11 16:57:48.548: INFO: Node Info: &Node{ObjectMeta:{i-07e2b6e650d7ef56d fcc65272-6f1b-4228-ac59-e9e3e92cdb2c 47965 0 2023-03-11 16:24:19 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:arm64 beta.kubernetes.io/instance-type:t4g.small beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-northeast-1 failure-domain.beta.kubernetes.io/zone:ap-northeast-1a io.kubernetes.storage.mock/node:some-mock-node karpenter.k8s.aws/instance-category:t karpenter.k8s.aws/instance-cpu:2 karpenter.k8s.aws/instance-family:t4g karpenter.k8s.aws/instance-generation:4 karpenter.k8s.aws/instance-hypervisor:nitro karpenter.k8s.aws/instance-memory:2048 karpenter.k8s.aws/instance-pods:11 karpenter.k8s.aws/instance-size:small karpenter.sh/capacity-type:spot karpenter.sh/initialized:true karpenter.sh/provisioner-name:nodes kubernetes.io/arch:arm64 kubernetes.io/hostname:i-07e2b6e650d7ef56d kubernetes.io/os:linux node-role.kubernetes.io/node: node-role.kubernetes.io/spot-worker:true node.kubernetes.io/instance-type:t4g.small topology.ebs.csi.aws.com/zone:ap-northeast-1a topology.hostpath.csi/node:i-07e2b6e650d7ef56d topology.kubernetes.io/region:ap-northeast-1 topology.kubernetes.io/zone:ap-northeast-1a] map[csi.volume.kubernetes.io/nodeid:{"csi-hostpath-ephemeral-7952":"i-07e2b6e650d7ef56d","csi-hostpath-volume-expand-703":"i-07e2b6e650d7ef56d","ebs.csi.aws.com":"i-07e2b6e650d7ef56d"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [{karpenter.sh/v1alpha5 Provisioner nodes ac7283b2-eed0-4a9f-9ddd-616f33a0991a <nil> 0xc000e7e365}] [karpenter.sh/termination] [{aws-cloud-controller-manager Update v1 2023-03-11 16:24:20 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{}}}} } {kops-controller Update v1 2023-03-11 16:24:20 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{},"f:node-role.kubernetes.io/spot-worker":{}}}} } {karpenter Update v1 2023-03-11 16:26:15 +0000 UTC FieldsV1 {"f:metadata":{"f:finalizers":{".":{},"v:\"karpenter.sh/termination\"":{}},"f:labels":{".":{},"f:karpenter.k8s.aws/instance-category":{},"f:karpenter.k8s.aws/instance-cpu":{},"f:karpenter.k8s.aws/instance-family":{},"f:karpenter.k8s.aws/instance-generation":{},"f:karpenter.k8s.aws/instance-hypervisor":{},"f:karpenter.k8s.aws/instance-memory":{},"f:karpenter.k8s.aws/instance-pods":{},"f:karpenter.k8s.aws/instance-size":{},"f:karpenter.sh/capacity-type":{},"f:karpenter.sh/initialized":{},"f:karpenter.sh/provisioner-name":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}},"f:ownerReferences":{".":{},"k:{\"uid\":\"ac7283b2-eed0-4a9f-9ddd-616f33a0991a\"}":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-03-11 16:26:50 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{".":{},"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"InternalIP\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kubelet Update v1 2023-03-11 16:44:22 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{"f:io.kubernetes.storage.mock/node":{},"f:kubernetes.io/hostname":{},"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{".":{},"f:cpu":{},"f:ephemeral-storage":{},"f:hugepages-1Gi":{},"f:hugepages-2Mi":{},"f:hugepages-32Mi":{},"f:hugepages-64Ki":{},"f:memory":{},"f:pods":{}},"f:capacity":{".":{},"f:cpu":{},"f:ephemeral-storage":{},"f:hugepages-1Gi":{},"f:hugepages-2Mi":{},"f:hugepages-32Mi":{},"f:hugepages-64Ki":{},"f:memory":{},"f:pods":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{}}},"f:daemonEndpoints":{"f:kubeletEndpoint":{"f:Port":{}}},"f:images":{},"f:nodeInfo":{"f:architecture":{},"f:bootID":{},"f:containerRuntimeVersion":{},"f:kernelVersion":{},"f:kubeProxyVersion":{},"f:kubeletVersion":{},"f:machineID":{},"f:operatingSystem":{},"f:osImage":{},"f:systemUUID":{}},"f:volumesInUse":{}}} status} {kube-controller-manager Update v1 2023-03-11 16:45:36 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{".":{},"k:{\"type\":\"DiskPressure\"}":{".":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"MemoryPressure\"}":{".":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"PIDPressure\"}":{".":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Ready\"}":{".":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}},"f:volumesAttached":{}}} status} {kube-controller-manager Update v1 2023-03-11 16:45:46 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:node.alpha.kubernetes.io/ttl":{}},"f:labels":{"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.1.0/24\"":{}},"f:taints":{}}} }]},Spec:NodeSpec{PodCIDR:100.96.1.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-northeast-1a/i-07e2b6e650d7ef56d,Unschedulable:false,Taints:[]Taint{Taint{Key:node.kubernetes.io/unreachable,Value:,Effect:NoSchedule,TimeAdded:2023-03-11 16:45:36 +0000 UTC,},Taint{Key:node.kubernetes.io/unreachable,Value:,Effect:NoExecute,TimeAdded:2023-03-11 16:45:46 +0000 UTC,},},ConfigSource:nil,PodCIDRs:[100.96.1.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49756430336 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-32Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-64Ki: {{0 0} {<nil>} 0 DecimalSI},memory: {{1941549056 0} {<nil>} 1896044Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44780787229 0} {<nil>} 44780787229 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-32Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-64Ki: {{0 0} {<nil>} 0 DecimalSI},memory: {{1836691456 0} {<nil>} 1793644Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:Ready,Status:Unknown,LastHeartbeatTime:2023-03-11 16:44:12 +0000 UTC,LastTransitionTime:2023-03-11 16:45:36 +0000 UTC,Reason:NodeStatusUnknown,Message:Kubelet stopped posting node status.,},NodeCondition{Type:MemoryPressure,Status:Unknown,LastHeartbeatTime:2023-03-11 16:44:12 +0000 UTC,LastTransitionTime:2023-03-11 16:45:36 +0000 UTC,Reason:NodeStatusUnknown,Message:Kubelet stopped posting node status.,},NodeCondition{Type:DiskPressure,Status:Unknown,LastHeartbeatTime:2023-03-11 16:44:12 +0000 UTC,LastTransitionTime:2023-03-11 16:45:36 +0000 UTC,Reason:NodeStatusUnknown,Message:Kubelet stopped posting node status.,},NodeCondition{Type:PIDPressure,Status:Unknown,LastHeartbeatTime:2023-03-11 16:44:12 +0000 UTC,LastTransitionTime:2023-03-11 16:45:36 +0000 UTC,Reason:NodeStatusUnknown,Message:Kubelet stopped posting node status.,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.35.76,},NodeAddress{Type:ExternalIP,Address:54.238.106.83,},NodeAddress{Type:InternalDNS,Address:i-07e2b6e650d7ef56d.ap-northeast-1.compute.internal,},NodeAddress{Type:Hostname,Address:i-07e2b6e650d7ef56d.ap-northeast-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-54-238-106-83.ap-northeast-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2083a8e9d27d02fac967340367f875,SystemUUID:ec2083a8-e9d2-7d02-fac9-67340367f875,BootID:ea264ed6-5b0d-47d6-ad5d-6bd2a1f7c11b,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 22.04.2 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.27.0-alpha.3.420+bb6c6fad2c19b3,KubeProxyVersion:v1.27.0-alpha.3.420+bb6c6fad2c19b3,OperatingSystem:linux,Architecture:arm64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:157636062,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/jessie-dnsutils@sha256:24aaf2626d6b27864c29de2097e8bbb840b3a414271bf7c8995e431e47d8408e registry.k8s.io/e2e-test-images/jessie-dnsutils:1.7],SizeBytes:116764683,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:51eae8381dcb1078289fa7b4f3df2630cdc18d09fb56f8e56b41c40e191d6c83 registry.k8s.io/etcd:3.5.7-0],SizeBytes:80665728,},ContainerImage{Names:[registry.k8s.io/kube-proxy-arm64:v1.27.0-alpha.3.420_bb6c6fad2c19b3],SizeBytes:60947298,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:16bbf38c463a4223d8cfe4da12bc61010b082a79b4bb003e2d3ba3ece5dd5f9e registry.k8s.io/e2e-test-images/agnhost:2.43],SizeBytes:49808364,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nautilus@sha256:80ba6c8c44f9623f06e868a1aa66026c8ec438ad814f9ec95e9333b415fe3550 registry.k8s.io/e2e-test-images/nautilus:1.7],SizeBytes:47953400,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:3fe7acf013d1264ffded116b80a73dc129a449b0fccdb8d21af8279f2233f36e registry.k8s.io/e2e-test-images/httpd:2.4.39-4],SizeBytes:41528818,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:148b022f5c5da426fc2f3c14b5c0867e58ef05961510c84749ac1fddcb0fef22 registry.k8s.io/e2e-test-images/httpd:2.4.38-4],SizeBytes:40343601,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:28932904,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:ee3b525d5b89db99da3b8eb521d9cd90cb6e9ef0fbb651e98bb37be78d36b5b8 registry.k8s.io/sig-storage/csi-provisioner:v3.3.0],SizeBytes:23945474,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/sample-apiserver@sha256:8d70890151aa5d096f331cb9da1b9cd5be0412b7363fe67b5c3befdcaa2a28d0 registry.k8s.io/e2e-test-images/sample-apiserver:1.17.7],SizeBytes:23857462,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:425d8f1b769398127767b06ed97ce62578a3179bcb99809ce93a1649e025ffe7 registry.k8s.io/sig-storage/csi-resizer:v1.6.0],SizeBytes:22678632,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:291334908ddf71a4661fd7f6d9d97274de8a5378a2b6fdfeb2ce73414a34f82f registry.k8s.io/sig-storage/csi-snapshotter:v6.1.0],SizeBytes:22434180,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:9a685020911e2725ad019dbce6e4a5ab93d51e3d4557f115e64343345e05781b registry.k8s.io/sig-storage/csi-attacher:v4.0.0],SizeBytes:22394501,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:122bfb8c1edabb3c0edd63f06523e6940d958d19b3957dc7b1d6f81e9f1f6119 registry.k8s.io/sig-storage/csi-provisioner:v3.1.0],SizeBytes:21735644,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/regression-issue-74839@sha256:b4f1d8d61bdad84bd50442d161d5460e4019d53e989b64220fdbc62fc87d76bf registry.k8s.io/e2e-test-images/regression-issue-74839:1.2],SizeBytes:21331383,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:9ebbf9f023e7b41ccee3d52afe39a89e3ddacdbb69269d583abfc25847cfd9e4 registry.k8s.io/sig-storage/csi-resizer:v1.4.0],SizeBytes:20848413,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:8b9c313c05f54fb04f8d430896f5f5904b6cb157df261501b29adc04d2b2dc7b registry.k8s.io/sig-storage/csi-attacher:v3.4.0],SizeBytes:20565480,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nonroot@sha256:ee9f50b3c64b174d296d91ca9f69a914ac30e59095dfb462b2b518ad28a63655 registry.k8s.io/e2e-test-images/nonroot:1.4],SizeBytes:18561972,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:92257881c1d6493cf18299a24af42330f891166560047902b8d431fb66b01af5 registry.k8s.io/sig-storage/hostpathplugin:v1.9.0],SizeBytes:18267611,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:9b2d9abda017c28b12f84a344f57cd73fbdb6c2bd7dd5bdd5018246ad1093ba6 registry.k8s.io/sig-storage/hostpathplugin:v1.11.0],SizeBytes:18159085,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:017727efcfeb7d053af68e51436ce8e65edbc6ca573720afb4f79c8594036955 registry.k8s.io/coredns/coredns:v1.10.0],SizeBytes:13766234,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:8502345,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:933940f13b3ea0abc62e656c1aa5c5b47c04b15d71250413a6b821bd0c58b94e registry.k8s.io/sig-storage/livenessprobe:v2.7.0],SizeBytes:8223819,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:7685305,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:db048754ae68ae337d8fa96494c96d2a1204c3320f5dcf7e8e71085adec85da6 registry.k8s.io/e2e-test-images/nginx:1.15-4],SizeBytes:6883347,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:5c99cf6a02adda929b10321dbf4ecfa00d87be9ba4fb456006237d530ab4baa1 registry.k8s.io/e2e-test-images/nginx:1.14-4],SizeBytes:6860089,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nonewprivs@sha256:8ac1264691820febacf3aea5d152cbde6d10685731ec14966a9401c6f47a68ac registry.k8s.io/e2e-test-images/nonewprivs:1.3],SizeBytes:5025027,},ContainerImage{Names:[gcr.io/authenticated-image-pulling/alpine@sha256:7ff177862cb50c602bfe81f805969412e619c054a2bbead977d0c276988aa4a0 gcr.io/authenticated-image-pulling/alpine:3.7],SizeBytes:2003505,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf registry.k8s.io/e2e-test-images/busybox:1.29-2],SizeBytes:773367,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:2e0f836850e09b8b7cc937681d6194537a09fbd5f6b9e08f4d646a85128e8937 registry.k8s.io/e2e-test-images/busybox:1.29-4],SizeBytes:772933,},ContainerImage{Names:[registry.k8s.io/pause@sha256:7031c1b283388d2c2e09b57badb803c05ebed362dc88d84b480cc47f72a21097 registry.k8s.io/pause:3.9],SizeBytes:268051,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:253553,},},VolumesInUse:[kubernetes.io/csi/csi-hostpath-ephemeral-7952^eca2344a-c02b-11ed-b483-1e4a85f27a89 kubernetes.io/csi/csi-hostpath-ephemeral-7952^f0ac2401-c02b-11ed-b483-1e4a85f27a89 kubernetes.io/csi/ebs.csi.aws.com^vol-06cba8db23f1489ad],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-06cba8db23f1489ad,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/csi-hostpath-ephemeral-7952^f0ac2401-c02b-11ed-b483-1e4a85f27a89,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/csi-hostpath-ephemeral-7952^eca2344a-c02b-11ed-b483-1e4a85f27a89,DevicePath:,},},Config:nil,},} Mar 11 16:57:48.548: INFO: Logging kubelet events for node i-07e2b6e650d7ef56d Mar 11 16:57:48.703: INFO: Logging pods the kubelet thinks is on node i-07e2b6e650d7ef56d Mar 11 16:57:58.853: INFO: Unable to retrieve kubelet pods for node i-07e2b6e650d7ef56d: error trying to reach service: net/http: TLS handshake timeout Mar 11 16:57:58.853: INFO: Logging node info for node i-0e3467ec552af797e Mar 11 16:57:59.001: INFO: Node Info: &Node{ObjectMeta:{i-0e3467ec552af797e 6c9b0cc5-2c4e-4037-bf31-61bac12a7775 50325 0 2023-03-11 16:20:46 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:arm64 beta.kubernetes.io/instance-type:c6g.xlarge beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-northeast-1 failure-domain.beta.kubernetes.io/zone:ap-northeast-1a kops.k8s.io/kops-controller-pki: kubernetes.io/arch:arm64 kubernetes.io/hostname:i-0e3467ec552af797e kubernetes.io/os:linux node-role.kubernetes.io/control-plane: node.kubernetes.io/exclude-from-external-load-balancers: node.kubernetes.io/instance-type:c6g.xlarge topology.ebs.csi.aws.com/zone:ap-northeast-1a topology.kubernetes.io/region:ap-northeast-1 topology.kubernetes.io/zone:ap-northeast-1a] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-0e3467ec552af797e"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kubelet Update v1 2023-03-11 16:20:46 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {protokube Update v1 2023-03-11 16:21:30 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kops.k8s.io/kops-controller-pki":{},"f:node-role.kubernetes.io/control-plane":{},"f:node.kubernetes.io/exclude-from-external-load-balancers":{}}}} } {aws-cloud-controller-manager Update v1 2023-03-11 16:21:50 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-03-11 16:21:50 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kube-controller-manager Update v1 2023-03-11 16:21:58 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.0.0/24\"":{}},"f:taints":{}}} } {kubelet Update v1 2023-03-11 16:53:56 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.0.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-northeast-1a/i-0e3467ec552af797e,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/control-plane,Value:,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[100.96.0.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{4 0} {<nil>} 4 DecimalSI},ephemeral-storage: {{49756430336 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-32Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-64Ki: {{0 0} {<nil>} 0 DecimalSI},memory: {{8150110208 0} {<nil>} 7959092Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{4 0} {<nil>} 4 DecimalSI},ephemeral-storage: {{44780787229 0} {<nil>} 44780787229 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-32Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-64Ki: {{0 0} {<nil>} 0 DecimalSI},memory: {{8045252608 0} {<nil>} 7856692Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-03-11 16:53:56 +0000 UTC,LastTransitionTime:2023-03-11 16:20:46 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-03-11 16:53:56 +0000 UTC,LastTransitionTime:2023-03-11 16:20:46 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-03-11 16:53:56 +0000 UTC,LastTransitionTime:2023-03-11 16:20:46 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-03-11 16:53:56 +0000 UTC,LastTransitionTime:2023-03-11 16:21:58 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.40.18,},NodeAddress{Type:ExternalIP,Address:52.194.213.7,},NodeAddress{Type:InternalDNS,Address:i-0e3467ec552af797e.ap-northeast-1.compute.internal,},NodeAddress{Type:Hostname,Address:i-0e3467ec552af797e.ap-northeast-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-52-194-213-7.ap-northeast-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2abf2e564965592c7716accd56507e,SystemUUID:ec2abf2e-5649-6559-2c77-16accd56507e,BootID:fbada3ff-8655-404c-a98d-876577d4c322,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 22.04.2 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.27.0-alpha.3.420+bb6c6fad2c19b3,KubeProxyVersion:v1.27.0-alpha.3.420+bb6c6fad2c19b3,OperatingSystem:linux,Architecture:arm64,},Images:[]ContainerImage{ContainerImage{Names:[registry.k8s.io/etcd@sha256:9ce33ba33d8e738a5b85ed50b5080ac746deceed4a7496c550927a7a19ca3b6d registry.k8s.io/etcd:3.5.0-0],SizeBytes:157797353,},ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:157636062,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:4afb99b4690b418ffc2ceb67e1a17376457e441c1f09ab55447f0aaf992fa646 registry.k8s.io/etcd:3.4.3-0],SizeBytes:151677080,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:4ad90a11b55313b182afc186b9876c8e891531b8db4c9bf1541953021618d0e2 registry.k8s.io/etcd:3.4.13-0],SizeBytes:134531559,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:0d2ab6b15b4fe0b903b9e81192d942fb39c29be61fe341cd16db5d0165b0435c registry.k8s.io/etcd:3.3.17-0],SizeBytes:126979107,},ContainerImage{Names:[registry.k8s.io/kube-apiserver-arm64:v1.27.0-alpha.3.420_bb6c6fad2c19b3],SizeBytes:115123555,},ContainerImage{Names:[registry.k8s.io/kube-controller-manager-arm64:v1.27.0-alpha.3.420_bb6c6fad2c19b3],SizeBytes:107914235,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:17da501f5d2a675be46040422a27b7cc21b8a43895ac998b171db1c346f361f7 registry.k8s.io/etcd:3.3.10-0],SizeBytes:81436097,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:6f72b851544986cb0921b53ea655ec04c36131248f16d4ad110cb3ca0c369dc1 registry.k8s.io/etcd:3.5.4-0],SizeBytes:81120118,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:13f53ed1d91e2e11aac476ee9a0269fdda6cc4874eba903efd40daf50c55eee5 registry.k8s.io/etcd:3.5.3-0],SizeBytes:81114891,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:51eae8381dcb1078289fa7b4f3df2630cdc18d09fb56f8e56b41c40e191d6c83 registry.k8s.io/etcd:3.5.7-0],SizeBytes:80665728,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:dd75ec974b0a2a6f6bb47001ba09207976e625db898d1b16735528c009cb171c registry.k8s.io/etcd:3.5.6-0],SizeBytes:80539316,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:905d7ca17fd02bc24c0eba9a062753aba15db3e31422390bc3238eb762339b20 registry.k8s.io/etcd:3.2.24-1],SizeBytes:61500433,},ContainerImage{Names:[registry.k8s.io/kube-proxy-arm64:v1.27.0-alpha.3.420_bb6c6fad2c19b3],SizeBytes:60947298,},ContainerImage{Names:[registry.k8s.io/etcdadm/etcd-manager-slim@sha256:8727adc059334e0e3329dbccce116a734da4a090da5e86c8cddccaf333caa666 registry.k8s.io/etcdadm/etcd-manager-slim:v3.0.20230201],SizeBytes:60826655,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:64b9ea357325d5db9f8a723dcf503b5a449177b17ac87d69481e126bb724c263 registry.k8s.io/etcd:3.5.1-0],SizeBytes:59553904,},ContainerImage{Names:[registry.k8s.io/kube-scheduler-arm64:v1.27.0-alpha.3.420_bb6c6fad2c19b3],SizeBytes:56731120,},ContainerImage{Names:[registry.k8s.io/kops/kops-controller:1.27.0-alpha.1],SizeBytes:41885191,},ContainerImage{Names:[registry.k8s.io/kops/dns-controller:1.27.0-alpha.1],SizeBytes:40376038,},ContainerImage{Names:[public.ecr.aws/karpenter/controller@sha256:68db4f092cf9cc83f5ef9e2fbc5407c2cb682e81f64dfaa700a7602ede38b1cf public.ecr.aws/karpenter/controller:v0.16.3],SizeBytes:33707751,},ContainerImage{Names:[public.ecr.aws/karpenter/webhook@sha256:96a2d9b06d6bc5127801f358f74b1cf2d289b423a2e9ba40c573c0b14b17dafa public.ecr.aws/karpenter/webhook:v0.16.3],SizeBytes:33307187,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:28932904,},ContainerImage{Names:[quay.io/cilium/operator@sha256:a6d24a006a6b92967ac90786b49bc1ac26e5477cf028cd1186efcfc2466484db quay.io/cilium/operator:v1.12.5],SizeBytes:24567287,},ContainerImage{Names:[registry.k8s.io/provider-aws/cloud-controller-manager@sha256:fdeb61e3e42ecd9cca868d550ebdb88dd6341d9e91fcfa9a37e227dab2ad22cb registry.k8s.io/provider-aws/cloud-controller-manager:v1.26.0],SizeBytes:18345226,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:8502345,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:7685305,},ContainerImage{Names:[registry.k8s.io/kops/kube-apiserver-healthcheck:1.27.0-alpha.1],SizeBytes:4740762,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:253553,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Mar 11 16:57:59.001: INFO: Logging kubelet events for node i-0e3467ec552af797e Mar 11 16:57:59.154: INFO: Logging pods the kubelet thinks is on node i-0e3467ec552af797e Mar 11 16:57:59.325: INFO: kube-controller-manager-i-0e3467ec552af797e started at 2023-03-11 16:18:44 +0000 UTC (0+1 container statuses recorded) Mar 11 16:57:59.326: INFO: Container kube-controller-manager ready: true, restart count 4 Mar 11 16:57:59.326: INFO: etcd-manager-events-i-0e3467ec552af797e started at 2023-03-11 16:18:44 +0000 UTC (11+1 container statuses recorded) Mar 11 16:57:59.326: INFO: Init container init-etcd-3-2-24 ready: true, restart count 0 Mar 11 16:57:59.326: INFO: Init container init-etcd-3-3-10 ready: true, restart count 0 Mar 11 16:57:59.326: INFO: Init container init-etcd-3-3-17 ready: true, restart count 0 Mar 11 16:57:59.326: INFO: Init container init-etcd-3-4-13 ready: true, restart count 0 Mar 11 16:57:59.326: INFO: Init container init-etcd-3-4-3 ready: true, restart count 0 Mar 11 16:57:59.326: INFO: Init container init-etcd-3-5-0 ready: true, restart count 0 Mar 11 16:57:59.326: INFO: Init container init-etcd-3-5-1 ready: true, restart count 0 Mar 11 16:57:59.326: INFO: Init container init-etcd-3-5-3 ready: true, restart count 0 Mar 11 16:57:59.326: INFO: Init container init-etcd-3-5-4 ready: true, restart count 0 Mar 11 16:57:59.326: INFO: Init container init-etcd-3-5-6 ready: true, restart count 0 Mar 11 16:57:59.326: INFO: Init container init-etcd-3-5-7 ready: true, restart count 0 Mar 11 16:57:59.326: INFO: Container etcd-manager ready: true, restart count 0 Mar 11 16:57:59.326: INFO: etcd-manager-main-i-0e3467ec552af797e started at 2023-03-11 16:18:44 +0000 UTC (11+1 container statuses recorded) Mar 11 16:57:59.326: INFO: Init container init-etcd-3-2-24 ready: true, restart count 0 Mar 11 16:57:59.326: INFO: Init container init-etcd-3-3-10 ready: true, restart count 0 Mar 11 16:57:59.326: INFO: Init container init-etcd-3-3-17 ready: true, restart count 0 Mar 11 16:57:59.326: INFO: Init container init-etcd-3-4-13 ready: true, restart count 0 Mar 11 16:57:59.326: INFO: Init container init-etcd-3-4-3 ready: true, restart count 0 Mar 11 16:57:59.326: INFO: Init container init-etcd-3-5-0 ready: true, restart count 0 Mar 11 16:57:59.326: INFO: Init container init-etcd-3-5-1 ready: true, restart count 0 Mar 11 16:57:59.326: INFO: Init container init-etcd-3-5-3 ready: true, restart count 0 Mar 11 16:57:59.326: INFO: Init container init-etcd-3-5-4 ready: true, restart count 0 Mar 11 16:57:59.326: INFO: Init container init-etcd-3-5-6 ready: true, restart count 0 Mar 11 16:57:59.326: INFO: Init container init-etcd-3-5-7 ready: true, restart count 0 Mar 11 16:57:59.326: INFO: Container etcd-manager ready: true, restart count 0 Mar 11 16:57:59.326: INFO: ebs-csi-node-lpwz9 started at 2023-03-11 16:21:37 +0000 UTC (0+3 container statuses recorded) Mar 11 16:57:59.326: INFO: Container ebs-plugin ready: true, restart count 0 Mar 11 16:57:59.326: INFO: Container liveness-probe ready: true, restart count 0 Mar 11 16:57:59.326: INFO: Container node-driver-registrar ready: true, restart count 0 Mar 11 16:57:59.326: INFO: karpenter-webhook-5b646cd4f7-6sg2k started at 2023-03-11 16:21:58 +0000 UTC (0+1 container statuses recorded) Mar 11 16:57:59.326: INFO: Container webhook ready: true, restart count 0 Mar 11 16:57:59.326: INFO: karpenter-controller-567749677d-d9szs started at 2023-03-11 16:21:58 +0000 UTC (0+1 container statuses recorded) Mar 11 16:57:59.326: INFO: Container controller ready: true, restart count 0 Mar 11 16:57:59.326: INFO: aws-cloud-controller-manager-kwbw2 started at 2023-03-11 16:21:37 +0000 UTC (0+1 container statuses recorded) Mar 11 16:57:59.326: INFO: Container aws-cloud-controller-manager ready: true, restart count 0 Mar 11 16:57:59.326: INFO: kube-apiserver-i-0e3467ec552af797e started at 2023-03-11 16:18:44 +0000 UTC (0+2 container statuses recorded) Mar 11 16:57:59.326: INFO: Container healthcheck ready: true, restart count 0 Mar 11 16:57:59.326: INFO: Container kube-apiserver ready: true, restart count 3 Mar 11 16:57:59.326: INFO: kube-scheduler-i-0e3467ec552af797e started at 2023-03-11 16:18:44 +0000 UTC (0+1 container statuses recorded) Mar 11 16:57:59.326: INFO: Container kube-scheduler ready: true, restart count 0 Mar 11 16:57:59.326: INFO: dns-controller-7dfd64d999-q9hh7 started at 2023-03-11 16:21:37 +0000 UTC (0+1 container statuses recorded) Mar 11 16:57:59.326: INFO: Container dns-controller ready: true, restart count 0 Mar 11 16:57:59.326: INFO: cilium-qrsfn started at 2023-03-11 16:21:37 +0000 UTC (1+1 container statuses recorded) Mar 11 16:57:59.326: INFO: Init container clean-cilium-state ready: true, restart count 0 Mar 11 16:57:59.326: INFO: Container cilium-agent ready: true, restart count 0 Mar 11 16:57:59.326: INFO: cilium-operator-8496cc65f5-qhdg8 started at 2023-03-11 16:21:37 +0000 UTC (0+1 container statuses recorded) Mar 11 16:57:59.326: INFO: Container cilium-operator ready: true, restart count 0 Mar 11 16:57:59.326: INFO: kops-controller-76gkw started at 2023-03-11 16:21:37 +0000 UTC (0+1 container statuses recorded) Mar 11 16:57:59.326: INFO: Container kops-controller ready: true, restart count 0 Mar 11 16:57:59.965: INFO: Latency metrics for node i-0e3467ec552af797e END STEP: dump namespace information after failure - test/e2e/framework/framework.go:288 @ 03/11/23 16:57:59.965 (32.614s) < Exit [DeferCleanup (Each)] [Testpattern: Generic Ephemeral-volume (default fs) (late-binding)] ephemeral - dump namespaces | framework.go:209 @ 03/11/23 16:57:59.965 (32.614s) > Enter [DeferCleanup (Each)] [Testpattern: Generic Ephemeral-volume (default fs) (late-binding)] ephemeral - tear down framework | framework.go:206 @ 03/11/23 16:57:59.965 STEP: Destroying namespace "ephemeral-234" for this suite. - test/e2e/framework/framework.go:351 @ 03/11/23 16:57:59.966 < Exit [DeferCleanup (Each)] [Testpattern: Generic Ephemeral-volume (default fs) (late-binding)] ephemeral - tear down framework | framework.go:206 @ 03/11/23 16:58:00.115 (150ms) > Enter [ReportAfterEach] TOP-LEVEL - test/e2e/e2e_test.go:144 @ 03/11/23 16:58:00.116 < Exit [ReportAfterEach] TOP-LEVEL - test/e2e/e2e_test.go:144 @ 03/11/23 16:58:00.116 (0s)
Find to mentions in log files | View test history on testgrid
go run hack/e2e.go -v --test --test_args='--ginkgo.focus=Kubernetes\se2e\ssuite\s\[It\]\sExternal\sStorage\s\[Driver\:\sebs\.csi\.aws\.com\]\s\[Testpattern\:\sGeneric\sEphemeral\-volume\s\(default\sfs\)\s\(late\-binding\)\]\sephemeral\sshould\ssupport\stwo\spods\swhich\shave\sthe\ssame\svolume\sdefinition$'
[FAILED] PersistentVolume pvc-d3807d41-b762-46ce-88e9-2aac574f5da4 still exists within 5m0s In [It] at: test/e2e/storage/testsuites/provisioning.go:1016 @ 03/11/23 16:49:50.661from junit_01.xml
> Enter [BeforeEach] [Testpattern: Generic Ephemeral-volume (default fs) (late-binding)] ephemeral - test/e2e/storage/framework/testsuite.go:51 @ 03/11/23 16:42:43.284 < Exit [BeforeEach] [Testpattern: Generic Ephemeral-volume (default fs) (late-binding)] ephemeral - test/e2e/storage/framework/testsuite.go:51 @ 03/11/23 16:42:43.284 (0s) > Enter [BeforeEach] [Testpattern: Generic Ephemeral-volume (default fs) (late-binding)] ephemeral - set up framework | framework.go:191 @ 03/11/23 16:42:43.284 STEP: Creating a kubernetes client - test/e2e/framework/framework.go:211 @ 03/11/23 16:42:43.285 Mar 11 16:42:43.285: INFO: >>> kubeConfig: /root/.kube/config STEP: Building a namespace api object, basename ephemeral - test/e2e/framework/framework.go:250 @ 03/11/23 16:42:43.286 STEP: Waiting for a default service account to be provisioned in namespace - test/e2e/framework/framework.go:259 @ 03/11/23 16:42:43.738 STEP: Waiting for kube-root-ca.crt to be provisioned in namespace - test/e2e/framework/framework.go:262 @ 03/11/23 16:42:44.033 < Exit [BeforeEach] [Testpattern: Generic Ephemeral-volume (default fs) (late-binding)] ephemeral - set up framework | framework.go:191 @ 03/11/23 16:42:44.329 (1.045s) > Enter [BeforeEach] [Testpattern: Generic Ephemeral-volume (default fs) (late-binding)] ephemeral - test/e2e/framework/metrics/init/init.go:33 @ 03/11/23 16:42:44.329 < Exit [BeforeEach] [Testpattern: Generic Ephemeral-volume (default fs) (late-binding)] ephemeral - test/e2e/framework/metrics/init/init.go:33 @ 03/11/23 16:42:44.329 (0s) > Enter [It] should support two pods which have the same volume definition - test/e2e/storage/testsuites/ephemeral.go:284 @ 03/11/23 16:42:44.329 Mar 11 16:42:44.630: INFO: Pod inline-volume-mtj6g has the following logs: Mar 11 16:42:44.779: INFO: Deleting pod "inline-volume-mtj6g" in namespace "ephemeral-9978" Mar 11 16:42:44.929: INFO: Wait up to 5m0s for pod "inline-volume-mtj6g" to be fully deleted Mar 11 16:42:45.078: INFO: Creating resource for dynamic PV Mar 11 16:42:45.078: INFO: Using claimSize:1Gi, test suite supported size:{ }, driver(ebs.csi.aws.com) supported size:{ } STEP: creating a StorageClass ephemeral-9978-e2e-scdplzc - test/e2e/storage/framework/volume_resource.go:102 @ 03/11/23 16:42:45.078 STEP: checking the requested inline volume exists in the pod running on node {Name: Selector:map[kubernetes.io/os:linux] Affinity:nil} - test/e2e/storage/testsuites/ephemeral.go:389 @ 03/11/23 16:42:45.228 STEP: writing data in one pod and checking the second does not see it (it should get its own volume) - test/e2e/storage/testsuites/ephemeral.go:310 @ 03/11/23 16:43:09.624 Mar 11 16:43:09.624: INFO: ExecWithOptions {Command:[/bin/sh -c touch /mnt/test-0/hello-world] Namespace:ephemeral-9978 PodName:inline-volume-tester-qnld2 ContainerName:csi-volume-tester Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false Quiet:false} Mar 11 16:43:09.624: INFO: >>> kubeConfig: /root/.kube/config Mar 11 16:43:09.625: INFO: ExecWithOptions: Clientset creation Mar 11 16:43:09.625: INFO: ExecWithOptions: execute(POST https://api.e2e-e2e-kops-aws-karpenter.test-cncf-aws.k8s.io/api/v1/namespaces/ephemeral-9978/pods/inline-volume-tester-qnld2/exec?command=%2Fbin%2Fsh&command=-c&command=touch+%2Fmnt%2Ftest-0%2Fhello-world&container=csi-volume-tester&container=csi-volume-tester&stderr=true&stdout=true) Mar 11 16:43:10.615: INFO: ExecWithOptions {Command:[/bin/sh -c [ ! -f /mnt/test-0/hello-world ]] Namespace:ephemeral-9978 PodName:inline-volume-tester2-xcfnv ContainerName:csi-volume-tester Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false Quiet:false} Mar 11 16:43:10.615: INFO: >>> kubeConfig: /root/.kube/config Mar 11 16:43:10.616: INFO: ExecWithOptions: Clientset creation Mar 11 16:43:10.616: INFO: ExecWithOptions: execute(POST https://api.e2e-e2e-kops-aws-karpenter.test-cncf-aws.k8s.io/api/v1/namespaces/ephemeral-9978/pods/inline-volume-tester2-xcfnv/exec?command=%2Fbin%2Fsh&command=-c&command=%5B+%21+-f+%2Fmnt%2Ftest-0%2Fhello-world+%5D&container=csi-volume-tester&container=csi-volume-tester&stderr=true&stdout=true) Mar 11 16:43:11.729: INFO: Pod inline-volume-tester2-xcfnv has the following logs: Mar 11 16:43:12.176: INFO: Deleting pod "inline-volume-tester2-xcfnv" in namespace "ephemeral-9978" Mar 11 16:43:12.331: INFO: Wait up to 5m0s for pod "inline-volume-tester2-xcfnv" to be fully deleted Mar 11 16:43:44.724: INFO: Wait up to 5m0s for pod PV pvc-34cc0889-3c80-4638-8f17-73cf161c1ca4 to be fully deleted Mar 11 16:43:44.724: INFO: Waiting up to 5m0s for PersistentVolume pvc-34cc0889-3c80-4638-8f17-73cf161c1ca4 to get deleted Mar 11 16:43:44.874: INFO: PersistentVolume pvc-34cc0889-3c80-4638-8f17-73cf161c1ca4 found and phase=Released (150.032042ms) Mar 11 16:43:50.023: INFO: PersistentVolume pvc-34cc0889-3c80-4638-8f17-73cf161c1ca4 found and phase=Released (5.299663311s) Mar 11 16:43:55.172: INFO: PersistentVolume pvc-34cc0889-3c80-4638-8f17-73cf161c1ca4 found and phase=Released (10.448655929s) Mar 11 16:44:00.322: INFO: PersistentVolume pvc-34cc0889-3c80-4638-8f17-73cf161c1ca4 was removed Mar 11 16:44:00.495: INFO: Pod inline-volume-tester-qnld2 has the following logs: Mar 11 16:44:00.792: INFO: Deleting pod "inline-volume-tester-qnld2" in namespace "ephemeral-9978" Mar 11 16:44:00.945: INFO: Wait up to 5m0s for pod "inline-volume-tester-qnld2" to be fully deleted Mar 11 16:44:46.244: INFO: Wait up to 5m0s for pod PV pvc-d3807d41-b762-46ce-88e9-2aac574f5da4 to be fully deleted Mar 11 16:44:46.244: INFO: Waiting up to 5m0s for PersistentVolume pvc-d3807d41-b762-46ce-88e9-2aac574f5da4 to get deleted Mar 11 16:44:46.392: INFO: PersistentVolume pvc-d3807d41-b762-46ce-88e9-2aac574f5da4 found and phase=Released (148.460658ms) Mar 11 16:44:51.541: INFO: PersistentVolume pvc-d3807d41-b762-46ce-88e9-2aac574f5da4 found and phase=Released (5.297460351s) Mar 11 16:44:56.690: INFO: PersistentVolume pvc-d3807d41-b762-46ce-88e9-2aac574f5da4 found and phase=Released (10.44686076s) Mar 11 16:45:01.840: INFO: PersistentVolume pvc-d3807d41-b762-46ce-88e9-2aac574f5da4 found and phase=Released (15.596737543s) Mar 11 16:45:06.990: INFO: PersistentVolume pvc-d3807d41-b762-46ce-88e9-2aac574f5da4 found and phase=Released (20.746001767s) Mar 11 16:45:12.139: INFO: PersistentVolume pvc-d3807d41-b762-46ce-88e9-2aac574f5da4 found and phase=Released (25.895131277s) Mar 11 16:45:17.288: INFO: PersistentVolume pvc-d3807d41-b762-46ce-88e9-2aac574f5da4 found and phase=Released (31.044087174s) Mar 11 16:45:22.436: INFO: PersistentVolume pvc-d3807d41-b762-46ce-88e9-2aac574f5da4 found and phase=Released (36.19283975s) Mar 11 16:45:27.585: INFO: PersistentVolume pvc-d3807d41-b762-46ce-88e9-2aac574f5da4 found and phase=Released (41.341768595s) Mar 11 16:45:32.735: INFO: PersistentVolume pvc-d3807d41-b762-46ce-88e9-2aac574f5da4 found and phase=Released (46.491529694s) Mar 11 16:45:37.884: INFO: PersistentVolume pvc-d3807d41-b762-46ce-88e9-2aac574f5da4 found and phase=Released (51.640896002s) Mar 11 16:45:43.034: INFO: PersistentVolume pvc-d3807d41-b762-46ce-88e9-2aac574f5da4 found and phase=Released (56.790421334s) Mar 11 16:45:48.183: INFO: PersistentVolume pvc-d3807d41-b762-46ce-88e9-2aac574f5da4 found and phase=Released (1m1.93977318s) Mar 11 16:45:53.332: INFO: PersistentVolume pvc-d3807d41-b762-46ce-88e9-2aac574f5da4 found and phase=Released (1m7.088693882s) Mar 11 16:45:58.482: INFO: PersistentVolume pvc-d3807d41-b762-46ce-88e9-2aac574f5da4 found and phase=Released (1m12.238950672s) Mar 11 16:46:03.633: INFO: PersistentVolume pvc-d3807d41-b762-46ce-88e9-2aac574f5da4 found and phase=Released (1m17.388990687s) Mar 11 16:46:08.781: INFO: PersistentVolume pvc-d3807d41-b762-46ce-88e9-2aac574f5da4 found and phase=Released (1m22.537887258s) Mar 11 16:46:13.931: INFO: PersistentVolume pvc-d3807d41-b762-46ce-88e9-2aac574f5da4 found and phase=Released (1m27.687776219s) Mar 11 16:46:19.081: INFO: PersistentVolume pvc-d3807d41-b762-46ce-88e9-2aac574f5da4 found and phase=Released (1m32.837196392s) Mar 11 16:46:24.231: INFO: PersistentVolume pvc-d3807d41-b762-46ce-88e9-2aac574f5da4 found and phase=Released (1m37.987349704s) Mar 11 16:46:29.381: INFO: PersistentVolume pvc-d3807d41-b762-46ce-88e9-2aac574f5da4 found and phase=Released (1m43.137082264s) Mar 11 16:46:34.529: INFO: PersistentVolume pvc-d3807d41-b762-46ce-88e9-2aac574f5da4 found and phase=Released (1m48.285948313s) Mar 11 16:46:39.679: INFO: PersistentVolume pvc-d3807d41-b762-46ce-88e9-2aac574f5da4 found and phase=Released (1m53.435164001s) Mar 11 16:46:44.828: INFO: PersistentVolume pvc-d3807d41-b762-46ce-88e9-2aac574f5da4 found and phase=Released (1m58.584781592s) Mar 11 16:46:49.977: INFO: PersistentVolume pvc-d3807d41-b762-46ce-88e9-2aac574f5da4 found and phase=Released (2m3.733779968s) Mar 11 16:46:55.127: INFO: PersistentVolume pvc-d3807d41-b762-46ce-88e9-2aac574f5da4 found and phase=Released (2m8.883025657s) Mar 11 16:47:00.276: INFO: PersistentVolume pvc-d3807d41-b762-46ce-88e9-2aac574f5da4 found and phase=Released (2m14.032879876s) Mar 11 16:47:05.426: INFO: PersistentVolume pvc-d3807d41-b762-46ce-88e9-2aac574f5da4 found and phase=Released (2m19.182778622s) Mar 11 16:47:10.576: INFO: PersistentVolume pvc-d3807d41-b762-46ce-88e9-2aac574f5da4 found and phase=Released (2m24.332110436s) Mar 11 16:47:15.726: INFO: PersistentVolume pvc-d3807d41-b762-46ce-88e9-2aac574f5da4 found and phase=Released (2m29.481971049s) Mar 11 16:47:20.875: INFO: PersistentVolume pvc-d3807d41-b762-46ce-88e9-2aac574f5da4 found and phase=Released (2m34.631811195s) Mar 11 16:47:26.025: INFO: PersistentVolume pvc-d3807d41-b762-46ce-88e9-2aac574f5da4 found and phase=Released (2m39.781783808s) Mar 11 16:47:31.174: INFO: PersistentVolume pvc-d3807d41-b762-46ce-88e9-2aac574f5da4 found and phase=Released (2m44.930688237s) Mar 11 16:47:36.324: INFO: PersistentVolume pvc-d3807d41-b762-46ce-88e9-2aac574f5da4 found and phase=Released (2m50.080594923s) Mar 11 16:47:41.474: INFO: PersistentVolume pvc-d3807d41-b762-46ce-88e9-2aac574f5da4 found and phase=Released (2m55.23020252s) Mar 11 16:47:46.623: INFO: PersistentVolume pvc-d3807d41-b762-46ce-88e9-2aac574f5da4 found and phase=Released (3m0.379475194s) Mar 11 16:47:51.773: INFO: PersistentVolume pvc-d3807d41-b762-46ce-88e9-2aac574f5da4 found and phase=Released (3m5.52921473s) Mar 11 16:47:56.923: INFO: PersistentVolume pvc-d3807d41-b762-46ce-88e9-2aac574f5da4 found and phase=Released (3m10.678984519s) Mar 11 16:48:02.072: INFO: PersistentVolume pvc-d3807d41-b762-46ce-88e9-2aac574f5da4 found and phase=Released (3m15.828482046s) Mar 11 16:48:07.221: INFO: PersistentVolume pvc-d3807d41-b762-46ce-88e9-2aac574f5da4 found and phase=Released (3m20.977807352s) Mar 11 16:48:12.371: INFO: PersistentVolume pvc-d3807d41-b762-46ce-88e9-2aac574f5da4 found and phase=Released (3m26.127656728s) Mar 11 16:48:17.521: INFO: PersistentVolume pvc-d3807d41-b762-46ce-88e9-2aac574f5da4 found and phase=Released (3m31.277330707s) Mar 11 16:48:22.670: INFO: PersistentVolume pvc-d3807d41-b762-46ce-88e9-2aac574f5da4 found and phase=Released (3m36.42649601s) Mar 11 16:48:27.820: INFO: PersistentVolume pvc-d3807d41-b762-46ce-88e9-2aac574f5da4 found and phase=Released (3m41.576180287s) Mar 11 16:48:32.969: INFO: PersistentVolume pvc-d3807d41-b762-46ce-88e9-2aac574f5da4 found and phase=Released (3m46.725442546s) Mar 11 16:48:38.119: INFO: PersistentVolume pvc-d3807d41-b762-46ce-88e9-2aac574f5da4 found and phase=Released (3m51.875303649s) Mar 11 16:48:43.268: INFO: PersistentVolume pvc-d3807d41-b762-46ce-88e9-2aac574f5da4 found and phase=Released (3m57.024947295s) Mar 11 16:48:48.418: INFO: PersistentVolume pvc-d3807d41-b762-46ce-88e9-2aac574f5da4 found and phase=Released (4m2.17458338s) Mar 11 16:48:53.568: INFO: PersistentVolume pvc-d3807d41-b762-46ce-88e9-2aac574f5da4 found and phase=Released (4m7.32493796s) Mar 11 16:48:58.718: INFO: PersistentVolume pvc-d3807d41-b762-46ce-88e9-2aac574f5da4 found and phase=Released (4m12.474703579s) Mar 11 16:49:03.867: INFO: PersistentVolume pvc-d3807d41-b762-46ce-88e9-2aac574f5da4 found and phase=Released (4m17.623920329s) Mar 11 16:49:09.017: INFO: PersistentVolume pvc-d3807d41-b762-46ce-88e9-2aac574f5da4 found and phase=Released (4m22.772980012s) Mar 11 16:49:14.167: INFO: PersistentVolume pvc-d3807d41-b762-46ce-88e9-2aac574f5da4 found and phase=Released (4m27.92297411s) Mar 11 16:49:19.316: INFO: PersistentVolume pvc-d3807d41-b762-46ce-88e9-2aac574f5da4 found and phase=Released (4m33.072372674s) Mar 11 16:49:24.466: INFO: PersistentVolume pvc-d3807d41-b762-46ce-88e9-2aac574f5da4 found and phase=Released (4m38.222331939s) Mar 11 16:49:29.615: INFO: PersistentVolume pvc-d3807d41-b762-46ce-88e9-2aac574f5da4 found and phase=Released (4m43.37189885s) Mar 11 16:49:34.765: INFO: PersistentVolume pvc-d3807d41-b762-46ce-88e9-2aac574f5da4 found and phase=Released (4m48.521761768s) Mar 11 16:49:39.915: INFO: PersistentVolume pvc-d3807d41-b762-46ce-88e9-2aac574f5da4 found and phase=Released (4m53.671326637s) Mar 11 16:49:45.065: INFO: PersistentVolume pvc-d3807d41-b762-46ce-88e9-2aac574f5da4 found and phase=Released (4m58.82127173s) Mar 11 16:49:50.065: INFO: Unexpected error: <*errors.errorString | 0xc0013629d0>: { s: "PersistentVolume pvc-d3807d41-b762-46ce-88e9-2aac574f5da4 still exists within 5m0s", } Mar 11 16:49:50.214: INFO: Error getting logs for pod inline-volume-tester-qnld2: the server could not find the requested resource (get pods inline-volume-tester-qnld2) Mar 11 16:49:50.512: INFO: Deleting pod "inline-volume-tester-qnld2" in namespace "ephemeral-9978" [FAILED] PersistentVolume pvc-d3807d41-b762-46ce-88e9-2aac574f5da4 still exists within 5m0s In [It] at: test/e2e/storage/testsuites/provisioning.go:1016 @ 03/11/23 16:49:50.661 < Exit [It] should support two pods which have the same volume definition - test/e2e/storage/testsuites/ephemeral.go:284 @ 03/11/23 16:49:50.661 (7m6.332s) > Enter [AfterEach] [Testpattern: Generic Ephemeral-volume (default fs) (late-binding)] ephemeral - test/e2e/framework/node/init/init.go:33 @ 03/11/23 16:49:50.661 Mar 11 16:49:50.661: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready Mar 11 16:49:50.811: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:49:50.811: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:49:52.961: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:49:52.961: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:49:54.961: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:49:54.961: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:49:56.961: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:49:56.961: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:49:58.961: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:49:58.961: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:50:00.961: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:50:00.961: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:50:02.962: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:50:02.962: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:50:04.961: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:50:04.961: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:50:06.961: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:50:06.961: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:50:08.961: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:50:08.961: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:50:10.961: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:50:10.961: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:50:12.961: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:50:12.961: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:50:14.962: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:50:14.962: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:50:16.961: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:50:16.961: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:50:18.961: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:50:18.961: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:50:20.961: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:50:20.961: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:50:22.961: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:50:22.961: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:50:24.961: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:50:24.961: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:50:26.961: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:50:26.961: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:50:28.962: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:50:28.962: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:50:30.962: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:50:30.962: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:50:32.961: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:50:32.961: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:50:34.961: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:50:34.961: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:50:36.961: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:50:36.961: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:50:38.961: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:50:38.961: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:50:40.961: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:50:40.961: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:50:42.961: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:50:42.961: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:50:44.961: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:50:44.961: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:50:46.961: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:50:46.961: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:50:48.961: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:50:48.961: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:50:50.961: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:50:50.961: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:50:52.961: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:50:52.961: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:50:54.961: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:50:54.961: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:50:56.961: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:50:56.961: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:50:58.961: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:50:58.961: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:51:00.961: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:51:00.961: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:51:02.963: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:51:02.963: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:51:04.961: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:51:04.961: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:51:06.961: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:51:06.961: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:51:08.962: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:51:08.962: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:51:10.962: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:51:10.962: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:51:12.961: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:51:12.961: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:51:14.961: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:51:14.961: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:51:16.961: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:51:16.961: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:51:18.961: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:51:18.961: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:51:20.961: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:51:20.961: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:51:22.961: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:51:22.961: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:51:24.962: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:51:24.962: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:51:26.961: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:51:26.961: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:51:28.962: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:51:28.962: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:51:30.961: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:51:30.961: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:51:32.961: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:51:32.961: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:51:34.961: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:51:34.961: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:51:36.962: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:51:36.962: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:51:38.962: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:51:38.962: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:51:40.962: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:51:40.962: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:51:42.961: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:51:42.961: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:51:44.962: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:51:44.962: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:51:46.962: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:51:46.962: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:51:48.962: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:51:48.962: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:51:50.961: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:51:50.961: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:51:52.962: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:51:52.962: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:51:54.961: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:51:54.961: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:51:56.961: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:51:56.961: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:51:58.961: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:51:58.961: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:52:00.961: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:52:00.961: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:52:02.961: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:52:02.961: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:52:04.961: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:52:04.961: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:52:06.961: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:52:06.961: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:52:08.961: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:52:08.961: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:52:10.961: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:52:10.961: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:52:12.961: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:52:12.961: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:52:14.961: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:52:14.961: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:52:16.961: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:52:16.961: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:52:18.961: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:52:18.961: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:52:20.961: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:52:20.961: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:52:22.961: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:52:22.961: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:52:24.961: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:52:24.961: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:52:26.961: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:52:26.961: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:52:28.961: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:52:28.961: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:52:30.961: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:52:30.961: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:52:32.961: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:52:32.961: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:52:34.961: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:52:34.961: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:52:36.961: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:52:36.961: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:52:38.961: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:52:38.961: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:52:40.961: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:52:40.961: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:52:42.961: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:52:42.961: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:52:44.961: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:52:44.961: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:52:46.961: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:52:46.961: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:52:48.961: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:52:48.961: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:52:50.961: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:52:50.961: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:52:51.111: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:52:51.111: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure < Exit [AfterEach] [Testpattern: Generic Ephemeral-volume (default fs) (late-binding)] ephemeral - test/e2e/framework/node/init/init.go:33 @ 03/11/23 16:52:51.111 (3m0.45s) > Enter [DeferCleanup (Each)] [Testpattern: Generic Ephemeral-volume (default fs) (late-binding)] ephemeral - test/e2e/storage/testsuites/ephemeral.go:286 @ 03/11/23 16:52:51.111 STEP: Deleting sc - test/e2e/storage/framework/volume_resource.go:228 @ 03/11/23 16:52:51.111 < Exit [DeferCleanup (Each)] [Testpattern: Generic Ephemeral-volume (default fs) (late-binding)] ephemeral - test/e2e/storage/testsuites/ephemeral.go:286 @ 03/11/23 16:52:51.262 (151ms) > Enter [DeferCleanup (Each)] [Testpattern: Generic Ephemeral-volume (default fs) (late-binding)] ephemeral - test/e2e/framework/metrics/init/init.go:35 @ 03/11/23 16:52:51.262 < Exit [DeferCleanup (Each)] [Testpattern: Generic Ephemeral-volume (default fs) (late-binding)] ephemeral - test/e2e/framework/metrics/init/init.go:35 @ 03/11/23 16:52:51.262 (0s) > Enter [DeferCleanup (Each)] [Testpattern: Generic Ephemeral-volume (default fs) (late-binding)] ephemeral - dump namespaces | framework.go:209 @ 03/11/23 16:52:51.262 STEP: dump namespace information after failure - test/e2e/framework/framework.go:288 @ 03/11/23 16:52:51.263 STEP: Collecting events from namespace "ephemeral-9978". - test/e2e/framework/debug/dump.go:42 @ 03/11/23 16:52:51.263 STEP: Found 26 events. - test/e2e/framework/debug/dump.go:46 @ 03/11/23 16:52:51.412 Mar 11 16:52:51.412: INFO: At 2023-03-11 16:42:44 +0000 UTC - event for inline-volume-mtj6g: {default-scheduler } FailedScheduling: 0/3 nodes are available: waiting for ephemeral volume controller to create the persistentvolumeclaim "inline-volume-mtj6g-my-volume". preemption: 0/3 nodes are available: 3 No preemption victims found for incoming pod.. Mar 11 16:52:51.412: INFO: At 2023-03-11 16:42:44 +0000 UTC - event for inline-volume-mtj6g-my-volume: {persistentvolume-controller } ProvisioningFailed: storageclass.storage.k8s.io "no-such-storage-class" not found Mar 11 16:52:51.412: INFO: At 2023-03-11 16:42:45 +0000 UTC - event for inline-volume-tester-qnld2: {default-scheduler } FailedScheduling: 0/3 nodes are available: waiting for ephemeral volume controller to create the persistentvolumeclaim "inline-volume-tester-qnld2-my-volume-0". preemption: 0/3 nodes are available: 3 No preemption victims found for incoming pod.. Mar 11 16:52:51.412: INFO: At 2023-03-11 16:42:45 +0000 UTC - event for inline-volume-tester-qnld2-my-volume-0: {persistentvolume-controller } WaitForPodScheduled: waiting for pod inline-volume-tester-qnld2 to be scheduled Mar 11 16:52:51.412: INFO: At 2023-03-11 16:42:46 +0000 UTC - event for inline-volume-tester-qnld2: {karpenter } Nominate: Pod should schedule on i-07e2b6e650d7ef56d Mar 11 16:52:51.412: INFO: At 2023-03-11 16:42:47 +0000 UTC - event for inline-volume-tester-qnld2-my-volume-0: {persistentvolume-controller } ExternalProvisioning: waiting for a volume to be created, either by external provisioner "ebs.csi.aws.com" or manually created by system administrator Mar 11 16:52:51.412: INFO: At 2023-03-11 16:42:47 +0000 UTC - event for inline-volume-tester-qnld2-my-volume-0: {ebs.csi.aws.com_ebs-csi-controller-79785c665b-lpmmf_0e0dbfd5-dc63-4852-b921-aa9f41d3903f } Provisioning: External provisioner is provisioning volume for claim "ephemeral-9978/inline-volume-tester-qnld2-my-volume-0" Mar 11 16:52:51.412: INFO: At 2023-03-11 16:42:50 +0000 UTC - event for inline-volume-tester-qnld2-my-volume-0: {ebs.csi.aws.com_ebs-csi-controller-79785c665b-lpmmf_0e0dbfd5-dc63-4852-b921-aa9f41d3903f } ProvisioningSucceeded: Successfully provisioned volume pvc-d3807d41-b762-46ce-88e9-2aac574f5da4 Mar 11 16:52:51.412: INFO: At 2023-03-11 16:42:51 +0000 UTC - event for inline-volume-tester-qnld2: {default-scheduler } Scheduled: Successfully assigned ephemeral-9978/inline-volume-tester-qnld2 to i-04523d1dffb0fe725 Mar 11 16:52:51.412: INFO: At 2023-03-11 16:42:54 +0000 UTC - event for inline-volume-tester-qnld2: {attachdetach-controller } SuccessfulAttachVolume: AttachVolume.Attach succeeded for volume "pvc-d3807d41-b762-46ce-88e9-2aac574f5da4" Mar 11 16:52:51.412: INFO: At 2023-03-11 16:42:55 +0000 UTC - event for inline-volume-tester-qnld2: {kubelet i-04523d1dffb0fe725} Started: Started container csi-volume-tester Mar 11 16:52:51.412: INFO: At 2023-03-11 16:42:55 +0000 UTC - event for inline-volume-tester-qnld2: {kubelet i-04523d1dffb0fe725} Pulled: Container image "registry.k8s.io/e2e-test-images/busybox:1.29-4" already present on machine Mar 11 16:52:51.412: INFO: At 2023-03-11 16:42:55 +0000 UTC - event for inline-volume-tester-qnld2: {kubelet i-04523d1dffb0fe725} Created: Created container csi-volume-tester Mar 11 16:52:51.412: INFO: At 2023-03-11 16:42:58 +0000 UTC - event for inline-volume-tester2-xcfnv: {default-scheduler } FailedScheduling: 0/3 nodes are available: waiting for ephemeral volume controller to create the persistentvolumeclaim "inline-volume-tester2-xcfnv-my-volume-0". preemption: 0/3 nodes are available: 3 No preemption victims found for incoming pod.. Mar 11 16:52:51.412: INFO: At 2023-03-11 16:42:58 +0000 UTC - event for inline-volume-tester2-xcfnv-my-volume-0: {persistentvolume-controller } WaitForPodScheduled: waiting for pod inline-volume-tester2-xcfnv to be scheduled Mar 11 16:52:51.412: INFO: At 2023-03-11 16:42:59 +0000 UTC - event for inline-volume-tester2-xcfnv: {karpenter } Nominate: Pod should schedule on i-07e2b6e650d7ef56d Mar 11 16:52:51.412: INFO: At 2023-03-11 16:43:00 +0000 UTC - event for inline-volume-tester2-xcfnv-my-volume-0: {persistentvolume-controller } ExternalProvisioning: waiting for a volume to be created, either by external provisioner "ebs.csi.aws.com" or manually created by system administrator Mar 11 16:52:51.412: INFO: At 2023-03-11 16:43:00 +0000 UTC - event for inline-volume-tester2-xcfnv-my-volume-0: {ebs.csi.aws.com_ebs-csi-controller-79785c665b-lpmmf_0e0dbfd5-dc63-4852-b921-aa9f41d3903f } Provisioning: External provisioner is provisioning volume for claim "ephemeral-9978/inline-volume-tester2-xcfnv-my-volume-0" Mar 11 16:52:51.412: INFO: At 2023-03-11 16:43:03 +0000 UTC - event for inline-volume-tester2-xcfnv-my-volume-0: {ebs.csi.aws.com_ebs-csi-controller-79785c665b-lpmmf_0e0dbfd5-dc63-4852-b921-aa9f41d3903f } ProvisioningSucceeded: Successfully provisioned volume pvc-34cc0889-3c80-4638-8f17-73cf161c1ca4 Mar 11 16:52:51.412: INFO: At 2023-03-11 16:43:04 +0000 UTC - event for inline-volume-tester2-xcfnv: {default-scheduler } Scheduled: Successfully assigned ephemeral-9978/inline-volume-tester2-xcfnv to i-04523d1dffb0fe725 Mar 11 16:52:51.412: INFO: At 2023-03-11 16:43:06 +0000 UTC - event for inline-volume-tester2-xcfnv: {attachdetach-controller } SuccessfulAttachVolume: AttachVolume.Attach succeeded for volume "pvc-34cc0889-3c80-4638-8f17-73cf161c1ca4" Mar 11 16:52:51.412: INFO: At 2023-03-11 16:43:07 +0000 UTC - event for inline-volume-tester2-xcfnv: {kubelet i-04523d1dffb0fe725} Pulled: Container image "registry.k8s.io/e2e-test-images/busybox:1.29-4" already present on machine Mar 11 16:52:51.412: INFO: At 2023-03-11 16:43:07 +0000 UTC - event for inline-volume-tester2-xcfnv: {kubelet i-04523d1dffb0fe725} Created: Created container csi-volume-tester Mar 11 16:52:51.412: INFO: At 2023-03-11 16:43:07 +0000 UTC - event for inline-volume-tester2-xcfnv: {kubelet i-04523d1dffb0fe725} Started: Started container csi-volume-tester Mar 11 16:52:51.412: INFO: At 2023-03-11 16:43:12 +0000 UTC - event for inline-volume-tester2-xcfnv: {kubelet i-04523d1dffb0fe725} Killing: Stopping container csi-volume-tester Mar 11 16:52:51.412: INFO: At 2023-03-11 16:44:00 +0000 UTC - event for inline-volume-tester-qnld2: {kubelet i-04523d1dffb0fe725} Killing: Stopping container csi-volume-tester Mar 11 16:52:51.561: INFO: POD NODE PHASE GRACE CONDITIONS Mar 11 16:52:51.561: INFO: Mar 11 16:52:51.711: INFO: Logging node info for node i-04523d1dffb0fe725 Mar 11 16:52:51.860: INFO: Node Info: &Node{ObjectMeta:{i-04523d1dffb0fe725 5d37640f-a7d9-4c26-86c5-9da21f7ccc18 50006 0 2023-03-11 16:24:20 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:arm64 beta.kubernetes.io/instance-type:t4g.small beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-northeast-1 failure-domain.beta.kubernetes.io/zone:ap-northeast-1a karpenter.k8s.aws/instance-category:t karpenter.k8s.aws/instance-cpu:2 karpenter.k8s.aws/instance-family:t4g karpenter.k8s.aws/instance-generation:4 karpenter.k8s.aws/instance-hypervisor:nitro karpenter.k8s.aws/instance-memory:2048 karpenter.k8s.aws/instance-pods:11 karpenter.k8s.aws/instance-size:small karpenter.sh/capacity-type:spot karpenter.sh/initialized:true karpenter.sh/provisioner-name:nodes kubernetes.io/arch:arm64 kubernetes.io/hostname:i-04523d1dffb0fe725 kubernetes.io/os:linux node-role.kubernetes.io/node: node-role.kubernetes.io/spot-worker:true node.kubernetes.io/instance-type:t4g.small topology.ebs.csi.aws.com/zone:ap-northeast-1a topology.hostpath.csi/node:i-04523d1dffb0fe725 topology.kubernetes.io/region:ap-northeast-1 topology.kubernetes.io/zone:ap-northeast-1a] map[csi.volume.kubernetes.io/nodeid:{"csi-hostpath-read-write-once-pod-7172":"i-04523d1dffb0fe725","csi-hostpath-volume-expand-1673":"i-04523d1dffb0fe725","ebs.csi.aws.com":"i-04523d1dffb0fe725"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [{karpenter.sh/v1alpha5 Provisioner nodes ac7283b2-eed0-4a9f-9ddd-616f33a0991a <nil> 0xc00510f1f5}] [karpenter.sh/termination] [{aws-cloud-controller-manager Update v1 2023-03-11 16:24:20 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{}}}} } {kops-controller Update v1 2023-03-11 16:24:20 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{},"f:node-role.kubernetes.io/spot-worker":{}}}} } {karpenter Update v1 2023-03-11 16:26:12 +0000 UTC FieldsV1 {"f:metadata":{"f:finalizers":{".":{},"v:\"karpenter.sh/termination\"":{}},"f:labels":{".":{},"f:karpenter.k8s.aws/instance-category":{},"f:karpenter.k8s.aws/instance-cpu":{},"f:karpenter.k8s.aws/instance-family":{},"f:karpenter.k8s.aws/instance-generation":{},"f:karpenter.k8s.aws/instance-hypervisor":{},"f:karpenter.k8s.aws/instance-memory":{},"f:karpenter.k8s.aws/instance-pods":{},"f:karpenter.k8s.aws/instance-size":{},"f:karpenter.sh/capacity-type":{},"f:karpenter.sh/initialized":{},"f:karpenter.sh/provisioner-name":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}},"f:ownerReferences":{".":{},"k:{\"uid\":\"ac7283b2-eed0-4a9f-9ddd-616f33a0991a\"}":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-03-11 16:26:50 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{".":{},"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"InternalIP\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kubelet Update v1 2023-03-11 16:44:47 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{"f:kubernetes.io/hostname":{},"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{".":{},"f:cpu":{},"f:ephemeral-storage":{},"f:hugepages-1Gi":{},"f:hugepages-2Mi":{},"f:hugepages-32Mi":{},"f:hugepages-64Ki":{},"f:memory":{},"f:pods":{}},"f:capacity":{".":{},"f:cpu":{},"f:ephemeral-storage":{},"f:hugepages-1Gi":{},"f:hugepages-2Mi":{},"f:hugepages-32Mi":{},"f:hugepages-64Ki":{},"f:memory":{},"f:pods":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{}}},"f:daemonEndpoints":{"f:kubeletEndpoint":{"f:Port":{}}},"f:images":{},"f:nodeInfo":{"f:architecture":{},"f:bootID":{},"f:containerRuntimeVersion":{},"f:kernelVersion":{},"f:kubeProxyVersion":{},"f:kubeletVersion":{},"f:machineID":{},"f:operatingSystem":{},"f:osImage":{},"f:systemUUID":{}},"f:volumesInUse":{}}} status} {kube-controller-manager Update v1 2023-03-11 16:45:36 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:node.alpha.kubernetes.io/ttl":{}},"f:labels":{"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.2.0/24\"":{}},"f:taints":{}}} } {kube-controller-manager Update v1 2023-03-11 16:52:51 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{".":{},"k:{\"type\":\"DiskPressure\"}":{".":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"MemoryPressure\"}":{".":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"PIDPressure\"}":{".":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Ready\"}":{".":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}},"f:volumesAttached":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.2.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-northeast-1a/i-04523d1dffb0fe725,Unschedulable:false,Taints:[]Taint{Taint{Key:node.kubernetes.io/unreachable,Value:,Effect:NoSchedule,TimeAdded:2023-03-11 16:45:31 +0000 UTC,},Taint{Key:node.kubernetes.io/unreachable,Value:,Effect:NoExecute,TimeAdded:2023-03-11 16:45:36 +0000 UTC,},},ConfigSource:nil,PodCIDRs:[100.96.2.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49756430336 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-32Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-64Ki: {{0 0} {<nil>} 0 DecimalSI},memory: {{1941540864 0} {<nil>} 1896036Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44780787229 0} {<nil>} 44780787229 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-32Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-64Ki: {{0 0} {<nil>} 0 DecimalSI},memory: {{1836683264 0} {<nil>} 1793636Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:Ready,Status:Unknown,LastHeartbeatTime:2023-03-11 16:44:47 +0000 UTC,LastTransitionTime:2023-03-11 16:45:31 +0000 UTC,Reason:NodeStatusUnknown,Message:Kubelet stopped posting node status.,},NodeCondition{Type:MemoryPressure,Status:Unknown,LastHeartbeatTime:2023-03-11 16:44:47 +0000 UTC,LastTransitionTime:2023-03-11 16:45:31 +0000 UTC,Reason:NodeStatusUnknown,Message:Kubelet stopped posting node status.,},NodeCondition{Type:DiskPressure,Status:Unknown,LastHeartbeatTime:2023-03-11 16:44:47 +0000 UTC,LastTransitionTime:2023-03-11 16:45:31 +0000 UTC,Reason:NodeStatusUnknown,Message:Kubelet stopped posting node status.,},NodeCondition{Type:PIDPressure,Status:Unknown,LastHeartbeatTime:2023-03-11 16:44:47 +0000 UTC,LastTransitionTime:2023-03-11 16:45:31 +0000 UTC,Reason:NodeStatusUnknown,Message:Kubelet stopped posting node status.,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.48.168,},NodeAddress{Type:ExternalIP,Address:52.199.253.156,},NodeAddress{Type:InternalDNS,Address:i-04523d1dffb0fe725.ap-northeast-1.compute.internal,},NodeAddress{Type:Hostname,Address:i-04523d1dffb0fe725.ap-northeast-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-52-199-253-156.ap-northeast-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2c8fb0e6b2e9c4af310f65b6161cd8,SystemUUID:ec2c8fb0-e6b2-e9c4-af31-0f65b6161cd8,BootID:a3b8de83-558b-44c4-a46a-d097c3f98225,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 22.04.2 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.27.0-alpha.3.420+bb6c6fad2c19b3,KubeProxyVersion:v1.27.0-alpha.3.420+bb6c6fad2c19b3,OperatingSystem:linux,Architecture:arm64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:157636062,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/jessie-dnsutils@sha256:24aaf2626d6b27864c29de2097e8bbb840b3a414271bf7c8995e431e47d8408e registry.k8s.io/e2e-test-images/jessie-dnsutils:1.7],SizeBytes:116764683,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:51eae8381dcb1078289fa7b4f3df2630cdc18d09fb56f8e56b41c40e191d6c83 registry.k8s.io/etcd:3.5.7-0],SizeBytes:80665728,},ContainerImage{Names:[registry.k8s.io/kube-proxy-arm64:v1.27.0-alpha.3.420_bb6c6fad2c19b3],SizeBytes:60947298,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:16bbf38c463a4223d8cfe4da12bc61010b082a79b4bb003e2d3ba3ece5dd5f9e registry.k8s.io/e2e-test-images/agnhost:2.43],SizeBytes:49808364,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nautilus@sha256:80ba6c8c44f9623f06e868a1aa66026c8ec438ad814f9ec95e9333b415fe3550 registry.k8s.io/e2e-test-images/nautilus:1.7],SizeBytes:47953400,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:3fe7acf013d1264ffded116b80a73dc129a449b0fccdb8d21af8279f2233f36e registry.k8s.io/e2e-test-images/httpd:2.4.39-4],SizeBytes:41528818,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:148b022f5c5da426fc2f3c14b5c0867e58ef05961510c84749ac1fddcb0fef22 registry.k8s.io/e2e-test-images/httpd:2.4.38-4],SizeBytes:40343601,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:28932904,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:ee3b525d5b89db99da3b8eb521d9cd90cb6e9ef0fbb651e98bb37be78d36b5b8 registry.k8s.io/sig-storage/csi-provisioner:v3.3.0],SizeBytes:23945474,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/sample-apiserver@sha256:8d70890151aa5d096f331cb9da1b9cd5be0412b7363fe67b5c3befdcaa2a28d0 registry.k8s.io/e2e-test-images/sample-apiserver:1.17.7],SizeBytes:23857462,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:425d8f1b769398127767b06ed97ce62578a3179bcb99809ce93a1649e025ffe7 registry.k8s.io/sig-storage/csi-resizer:v1.6.0],SizeBytes:22678632,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:291334908ddf71a4661fd7f6d9d97274de8a5378a2b6fdfeb2ce73414a34f82f registry.k8s.io/sig-storage/csi-snapshotter:v6.1.0],SizeBytes:22434180,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:9a685020911e2725ad019dbce6e4a5ab93d51e3d4557f115e64343345e05781b registry.k8s.io/sig-storage/csi-attacher:v4.0.0],SizeBytes:22394501,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:122bfb8c1edabb3c0edd63f06523e6940d958d19b3957dc7b1d6f81e9f1f6119 registry.k8s.io/sig-storage/csi-provisioner:v3.1.0],SizeBytes:21735644,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:9ebbf9f023e7b41ccee3d52afe39a89e3ddacdbb69269d583abfc25847cfd9e4 registry.k8s.io/sig-storage/csi-resizer:v1.4.0],SizeBytes:20848413,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:8b9c313c05f54fb04f8d430896f5f5904b6cb157df261501b29adc04d2b2dc7b registry.k8s.io/sig-storage/csi-attacher:v3.4.0],SizeBytes:20565480,},ContainerImage{Names:[registry.k8s.io/cpa/cluster-proportional-autoscaler@sha256:68d396900aeaa072c1f27289485fdac29834045a6f3ffe369bf389d830ef572d registry.k8s.io/cpa/cluster-proportional-autoscaler:1.8.6],SizeBytes:19007623,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nonroot@sha256:ee9f50b3c64b174d296d91ca9f69a914ac30e59095dfb462b2b518ad28a63655 registry.k8s.io/e2e-test-images/nonroot:1.4],SizeBytes:18561972,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:92257881c1d6493cf18299a24af42330f891166560047902b8d431fb66b01af5 registry.k8s.io/sig-storage/hostpathplugin:v1.9.0],SizeBytes:18267611,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:9b2d9abda017c28b12f84a344f57cd73fbdb6c2bd7dd5bdd5018246ad1093ba6 registry.k8s.io/sig-storage/hostpathplugin:v1.11.0],SizeBytes:18159085,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:017727efcfeb7d053af68e51436ce8e65edbc6ca573720afb4f79c8594036955 registry.k8s.io/coredns/coredns:v1.10.0],SizeBytes:13766234,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:8502345,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:933940f13b3ea0abc62e656c1aa5c5b47c04b15d71250413a6b821bd0c58b94e registry.k8s.io/sig-storage/livenessprobe:v2.7.0],SizeBytes:8223819,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:7685305,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:db048754ae68ae337d8fa96494c96d2a1204c3320f5dcf7e8e71085adec85da6 registry.k8s.io/e2e-test-images/nginx:1.15-4],SizeBytes:6883347,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:5c99cf6a02adda929b10321dbf4ecfa00d87be9ba4fb456006237d530ab4baa1 registry.k8s.io/e2e-test-images/nginx:1.14-4],SizeBytes:6860089,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nonewprivs@sha256:8ac1264691820febacf3aea5d152cbde6d10685731ec14966a9401c6f47a68ac registry.k8s.io/e2e-test-images/nonewprivs:1.3],SizeBytes:5025027,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf registry.k8s.io/e2e-test-images/busybox:1.29-2],SizeBytes:773367,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:2e0f836850e09b8b7cc937681d6194537a09fbd5f6b9e08f4d646a85128e8937 registry.k8s.io/e2e-test-images/busybox:1.29-4],SizeBytes:772933,},ContainerImage{Names:[registry.k8s.io/pause@sha256:7031c1b283388d2c2e09b57badb803c05ebed362dc88d84b480cc47f72a21097 registry.k8s.io/pause:3.9],SizeBytes:268051,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:253553,},},VolumesInUse:[kubernetes.io/csi/csi-hostpath-read-write-once-pod-7172^038fde52-c02c-11ed-b028-5eca9e37a97f kubernetes.io/csi/csi-hostpath-volume-expand-1673^f8193b68-c02b-11ed-8784-9e8522e7cc37 kubernetes.io/csi/ebs.csi.aws.com^vol-0293af09ce81fe75e],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-0d0346dd53f1fd7bb,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/csi-hostpath-volume-expand-1673^f8193b68-c02b-11ed-8784-9e8522e7cc37,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-0293af09ce81fe75e,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/csi-hostpath-read-write-once-pod-7172^038fde52-c02c-11ed-b028-5eca9e37a97f,DevicePath:,},},Config:nil,},} Mar 11 16:52:51.861: INFO: Logging kubelet events for node i-04523d1dffb0fe725 Mar 11 16:52:52.013: INFO: Logging pods the kubelet thinks is on node i-04523d1dffb0fe725 Mar 11 16:53:02.162: INFO: Unable to retrieve kubelet pods for node i-04523d1dffb0fe725: error trying to reach service: net/http: TLS handshake timeout Mar 11 16:53:02.162: INFO: Logging node info for node i-07e2b6e650d7ef56d Mar 11 16:53:02.312: INFO: Node Info: &Node{ObjectMeta:{i-07e2b6e650d7ef56d fcc65272-6f1b-4228-ac59-e9e3e92cdb2c 47965 0 2023-03-11 16:24:19 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:arm64 beta.kubernetes.io/instance-type:t4g.small beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-northeast-1 failure-domain.beta.kubernetes.io/zone:ap-northeast-1a io.kubernetes.storage.mock/node:some-mock-node karpenter.k8s.aws/instance-category:t karpenter.k8s.aws/instance-cpu:2 karpenter.k8s.aws/instance-family:t4g karpenter.k8s.aws/instance-generation:4 karpenter.k8s.aws/instance-hypervisor:nitro karpenter.k8s.aws/instance-memory:2048 karpenter.k8s.aws/instance-pods:11 karpenter.k8s.aws/instance-size:small karpenter.sh/capacity-type:spot karpenter.sh/initialized:true karpenter.sh/provisioner-name:nodes kubernetes.io/arch:arm64 kubernetes.io/hostname:i-07e2b6e650d7ef56d kubernetes.io/os:linux node-role.kubernetes.io/node: node-role.kubernetes.io/spot-worker:true node.kubernetes.io/instance-type:t4g.small topology.ebs.csi.aws.com/zone:ap-northeast-1a topology.hostpath.csi/node:i-07e2b6e650d7ef56d topology.kubernetes.io/region:ap-northeast-1 topology.kubernetes.io/zone:ap-northeast-1a] map[csi.volume.kubernetes.io/nodeid:{"csi-hostpath-ephemeral-7952":"i-07e2b6e650d7ef56d","csi-hostpath-volume-expand-703":"i-07e2b6e650d7ef56d","ebs.csi.aws.com":"i-07e2b6e650d7ef56d"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [{karpenter.sh/v1alpha5 Provisioner nodes ac7283b2-eed0-4a9f-9ddd-616f33a0991a <nil> 0xc00510fd25}] [karpenter.sh/termination] [{aws-cloud-controller-manager Update v1 2023-03-11 16:24:20 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{}}}} } {kops-controller Update v1 2023-03-11 16:24:20 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{},"f:node-role.kubernetes.io/spot-worker":{}}}} } {karpenter Update v1 2023-03-11 16:26:15 +0000 UTC FieldsV1 {"f:metadata":{"f:finalizers":{".":{},"v:\"karpenter.sh/termination\"":{}},"f:labels":{".":{},"f:karpenter.k8s.aws/instance-category":{},"f:karpenter.k8s.aws/instance-cpu":{},"f:karpenter.k8s.aws/instance-family":{},"f:karpenter.k8s.aws/instance-generation":{},"f:karpenter.k8s.aws/instance-hypervisor":{},"f:karpenter.k8s.aws/instance-memory":{},"f:karpenter.k8s.aws/instance-pods":{},"f:karpenter.k8s.aws/instance-size":{},"f:karpenter.sh/capacity-type":{},"f:karpenter.sh/initialized":{},"f:karpenter.sh/provisioner-name":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}},"f:ownerReferences":{".":{},"k:{\"uid\":\"ac7283b2-eed0-4a9f-9ddd-616f33a0991a\"}":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-03-11 16:26:50 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{".":{},"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"InternalIP\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kubelet Update v1 2023-03-11 16:44:22 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{"f:io.kubernetes.storage.mock/node":{},"f:kubernetes.io/hostname":{},"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{".":{},"f:cpu":{},"f:ephemeral-storage":{},"f:hugepages-1Gi":{},"f:hugepages-2Mi":{},"f:hugepages-32Mi":{},"f:hugepages-64Ki":{},"f:memory":{},"f:pods":{}},"f:capacity":{".":{},"f:cpu":{},"f:ephemeral-storage":{},"f:hugepages-1Gi":{},"f:hugepages-2Mi":{},"f:hugepages-32Mi":{},"f:hugepages-64Ki":{},"f:memory":{},"f:pods":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{}}},"f:daemonEndpoints":{"f:kubeletEndpoint":{"f:Port":{}}},"f:images":{},"f:nodeInfo":{"f:architecture":{},"f:bootID":{},"f:containerRuntimeVersion":{},"f:kernelVersion":{},"f:kubeProxyVersion":{},"f:kubeletVersion":{},"f:machineID":{},"f:operatingSystem":{},"f:osImage":{},"f:systemUUID":{}},"f:volumesInUse":{}}} status} {kube-controller-manager Update v1 2023-03-11 16:45:36 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{".":{},"k:{\"type\":\"DiskPressure\"}":{".":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"MemoryPressure\"}":{".":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"PIDPressure\"}":{".":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Ready\"}":{".":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}},"f:volumesAttached":{}}} status} {kube-controller-manager Update v1 2023-03-11 16:45:46 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:node.alpha.kubernetes.io/ttl":{}},"f:labels":{"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.1.0/24\"":{}},"f:taints":{}}} }]},Spec:NodeSpec{PodCIDR:100.96.1.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-northeast-1a/i-07e2b6e650d7ef56d,Unschedulable:false,Taints:[]Taint{Taint{Key:node.kubernetes.io/unreachable,Value:,Effect:NoSchedule,TimeAdded:2023-03-11 16:45:36 +0000 UTC,},Taint{Key:node.kubernetes.io/unreachable,Value:,Effect:NoExecute,TimeAdded:2023-03-11 16:45:46 +0000 UTC,},},ConfigSource:nil,PodCIDRs:[100.96.1.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49756430336 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-32Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-64Ki: {{0 0} {<nil>} 0 DecimalSI},memory: {{1941549056 0} {<nil>} 1896044Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44780787229 0} {<nil>} 44780787229 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-32Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-64Ki: {{0 0} {<nil>} 0 DecimalSI},memory: {{1836691456 0} {<nil>} 1793644Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:Ready,Status:Unknown,LastHeartbeatTime:2023-03-11 16:44:12 +0000 UTC,LastTransitionTime:2023-03-11 16:45:36 +0000 UTC,Reason:NodeStatusUnknown,Message:Kubelet stopped posting node status.,},NodeCondition{Type:MemoryPressure,Status:Unknown,LastHeartbeatTime:2023-03-11 16:44:12 +0000 UTC,LastTransitionTime:2023-03-11 16:45:36 +0000 UTC,Reason:NodeStatusUnknown,Message:Kubelet stopped posting node status.,},NodeCondition{Type:DiskPressure,Status:Unknown,LastHeartbeatTime:2023-03-11 16:44:12 +0000 UTC,LastTransitionTime:2023-03-11 16:45:36 +0000 UTC,Reason:NodeStatusUnknown,Message:Kubelet stopped posting node status.,},NodeCondition{Type:PIDPressure,Status:Unknown,LastHeartbeatTime:2023-03-11 16:44:12 +0000 UTC,LastTransitionTime:2023-03-11 16:45:36 +0000 UTC,Reason:NodeStatusUnknown,Message:Kubelet stopped posting node status.,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.35.76,},NodeAddress{Type:ExternalIP,Address:54.238.106.83,},NodeAddress{Type:InternalDNS,Address:i-07e2b6e650d7ef56d.ap-northeast-1.compute.internal,},NodeAddress{Type:Hostname,Address:i-07e2b6e650d7ef56d.ap-northeast-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-54-238-106-83.ap-northeast-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2083a8e9d27d02fac967340367f875,SystemUUID:ec2083a8-e9d2-7d02-fac9-67340367f875,BootID:ea264ed6-5b0d-47d6-ad5d-6bd2a1f7c11b,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 22.04.2 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.27.0-alpha.3.420+bb6c6fad2c19b3,KubeProxyVersion:v1.27.0-alpha.3.420+bb6c6fad2c19b3,OperatingSystem:linux,Architecture:arm64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:157636062,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/jessie-dnsutils@sha256:24aaf2626d6b27864c29de2097e8bbb840b3a414271bf7c8995e431e47d8408e registry.k8s.io/e2e-test-images/jessie-dnsutils:1.7],SizeBytes:116764683,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:51eae8381dcb1078289fa7b4f3df2630cdc18d09fb56f8e56b41c40e191d6c83 registry.k8s.io/etcd:3.5.7-0],SizeBytes:80665728,},ContainerImage{Names:[registry.k8s.io/kube-proxy-arm64:v1.27.0-alpha.3.420_bb6c6fad2c19b3],SizeBytes:60947298,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:16bbf38c463a4223d8cfe4da12bc61010b082a79b4bb003e2d3ba3ece5dd5f9e registry.k8s.io/e2e-test-images/agnhost:2.43],SizeBytes:49808364,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nautilus@sha256:80ba6c8c44f9623f06e868a1aa66026c8ec438ad814f9ec95e9333b415fe3550 registry.k8s.io/e2e-test-images/nautilus:1.7],SizeBytes:47953400,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:3fe7acf013d1264ffded116b80a73dc129a449b0fccdb8d21af8279f2233f36e registry.k8s.io/e2e-test-images/httpd:2.4.39-4],SizeBytes:41528818,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:148b022f5c5da426fc2f3c14b5c0867e58ef05961510c84749ac1fddcb0fef22 registry.k8s.io/e2e-test-images/httpd:2.4.38-4],SizeBytes:40343601,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:28932904,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:ee3b525d5b89db99da3b8eb521d9cd90cb6e9ef0fbb651e98bb37be78d36b5b8 registry.k8s.io/sig-storage/csi-provisioner:v3.3.0],SizeBytes:23945474,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/sample-apiserver@sha256:8d70890151aa5d096f331cb9da1b9cd5be0412b7363fe67b5c3befdcaa2a28d0 registry.k8s.io/e2e-test-images/sample-apiserver:1.17.7],SizeBytes:23857462,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:425d8f1b769398127767b06ed97ce62578a3179bcb99809ce93a1649e025ffe7 registry.k8s.io/sig-storage/csi-resizer:v1.6.0],SizeBytes:22678632,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:291334908ddf71a4661fd7f6d9d97274de8a5378a2b6fdfeb2ce73414a34f82f registry.k8s.io/sig-storage/csi-snapshotter:v6.1.0],SizeBytes:22434180,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:9a685020911e2725ad019dbce6e4a5ab93d51e3d4557f115e64343345e05781b registry.k8s.io/sig-storage/csi-attacher:v4.0.0],SizeBytes:22394501,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:122bfb8c1edabb3c0edd63f06523e6940d958d19b3957dc7b1d6f81e9f1f6119 registry.k8s.io/sig-storage/csi-provisioner:v3.1.0],SizeBytes:21735644,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/regression-issue-74839@sha256:b4f1d8d61bdad84bd50442d161d5460e4019d53e989b64220fdbc62fc87d76bf registry.k8s.io/e2e-test-images/regression-issue-74839:1.2],SizeBytes:21331383,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:9ebbf9f023e7b41ccee3d52afe39a89e3ddacdbb69269d583abfc25847cfd9e4 registry.k8s.io/sig-storage/csi-resizer:v1.4.0],SizeBytes:20848413,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:8b9c313c05f54fb04f8d430896f5f5904b6cb157df261501b29adc04d2b2dc7b registry.k8s.io/sig-storage/csi-attacher:v3.4.0],SizeBytes:20565480,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nonroot@sha256:ee9f50b3c64b174d296d91ca9f69a914ac30e59095dfb462b2b518ad28a63655 registry.k8s.io/e2e-test-images/nonroot:1.4],SizeBytes:18561972,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:92257881c1d6493cf18299a24af42330f891166560047902b8d431fb66b01af5 registry.k8s.io/sig-storage/hostpathplugin:v1.9.0],SizeBytes:18267611,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:9b2d9abda017c28b12f84a344f57cd73fbdb6c2bd7dd5bdd5018246ad1093ba6 registry.k8s.io/sig-storage/hostpathplugin:v1.11.0],SizeBytes:18159085,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:017727efcfeb7d053af68e51436ce8e65edbc6ca573720afb4f79c8594036955 registry.k8s.io/coredns/coredns:v1.10.0],SizeBytes:13766234,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:8502345,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:933940f13b3ea0abc62e656c1aa5c5b47c04b15d71250413a6b821bd0c58b94e registry.k8s.io/sig-storage/livenessprobe:v2.7.0],SizeBytes:8223819,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:7685305,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:db048754ae68ae337d8fa96494c96d2a1204c3320f5dcf7e8e71085adec85da6 registry.k8s.io/e2e-test-images/nginx:1.15-4],SizeBytes:6883347,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:5c99cf6a02adda929b10321dbf4ecfa00d87be9ba4fb456006237d530ab4baa1 registry.k8s.io/e2e-test-images/nginx:1.14-4],SizeBytes:6860089,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nonewprivs@sha256:8ac1264691820febacf3aea5d152cbde6d10685731ec14966a9401c6f47a68ac registry.k8s.io/e2e-test-images/nonewprivs:1.3],SizeBytes:5025027,},ContainerImage{Names:[gcr.io/authenticated-image-pulling/alpine@sha256:7ff177862cb50c602bfe81f805969412e619c054a2bbead977d0c276988aa4a0 gcr.io/authenticated-image-pulling/alpine:3.7],SizeBytes:2003505,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf registry.k8s.io/e2e-test-images/busybox:1.29-2],SizeBytes:773367,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:2e0f836850e09b8b7cc937681d6194537a09fbd5f6b9e08f4d646a85128e8937 registry.k8s.io/e2e-test-images/busybox:1.29-4],SizeBytes:772933,},ContainerImage{Names:[registry.k8s.io/pause@sha256:7031c1b283388d2c2e09b57badb803c05ebed362dc88d84b480cc47f72a21097 registry.k8s.io/pause:3.9],SizeBytes:268051,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:253553,},},VolumesInUse:[kubernetes.io/csi/csi-hostpath-ephemeral-7952^eca2344a-c02b-11ed-b483-1e4a85f27a89 kubernetes.io/csi/csi-hostpath-ephemeral-7952^f0ac2401-c02b-11ed-b483-1e4a85f27a89 kubernetes.io/csi/ebs.csi.aws.com^vol-06cba8db23f1489ad],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-06cba8db23f1489ad,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/csi-hostpath-ephemeral-7952^f0ac2401-c02b-11ed-b483-1e4a85f27a89,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/csi-hostpath-ephemeral-7952^eca2344a-c02b-11ed-b483-1e4a85f27a89,DevicePath:,},},Config:nil,},} Mar 11 16:53:02.312: INFO: Logging kubelet events for node i-07e2b6e650d7ef56d Mar 11 16:53:02.464: INFO: Logging pods the kubelet thinks is on node i-07e2b6e650d7ef56d Mar 11 16:53:12.614: INFO: Unable to retrieve kubelet pods for node i-07e2b6e650d7ef56d: error trying to reach service: net/http: TLS handshake timeout Mar 11 16:53:12.614: INFO: Logging node info for node i-0e3467ec552af797e Mar 11 16:53:12.763: INFO: Node Info: &Node{ObjectMeta:{i-0e3467ec552af797e 6c9b0cc5-2c4e-4037-bf31-61bac12a7775 48787 0 2023-03-11 16:20:46 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:arm64 beta.kubernetes.io/instance-type:c6g.xlarge beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-northeast-1 failure-domain.beta.kubernetes.io/zone:ap-northeast-1a kops.k8s.io/kops-controller-pki: kubernetes.io/arch:arm64 kubernetes.io/hostname:i-0e3467ec552af797e kubernetes.io/os:linux node-role.kubernetes.io/control-plane: node.kubernetes.io/exclude-from-external-load-balancers: node.kubernetes.io/instance-type:c6g.xlarge topology.ebs.csi.aws.com/zone:ap-northeast-1a topology.kubernetes.io/region:ap-northeast-1 topology.kubernetes.io/zone:ap-northeast-1a] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-0e3467ec552af797e"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kubelet Update v1 2023-03-11 16:20:46 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {protokube Update v1 2023-03-11 16:21:30 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kops.k8s.io/kops-controller-pki":{},"f:node-role.kubernetes.io/control-plane":{},"f:node.kubernetes.io/exclude-from-external-load-balancers":{}}}} } {aws-cloud-controller-manager Update v1 2023-03-11 16:21:50 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-03-11 16:21:50 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kube-controller-manager Update v1 2023-03-11 16:21:58 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.0.0/24\"":{}},"f:taints":{}}} } {kubelet Update v1 2023-03-11 16:48:50 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.0.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-northeast-1a/i-0e3467ec552af797e,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/control-plane,Value:,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[100.96.0.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{4 0} {<nil>} 4 DecimalSI},ephemeral-storage: {{49756430336 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-32Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-64Ki: {{0 0} {<nil>} 0 DecimalSI},memory: {{8150110208 0} {<nil>} 7959092Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{4 0} {<nil>} 4 DecimalSI},ephemeral-storage: {{44780787229 0} {<nil>} 44780787229 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-32Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-64Ki: {{0 0} {<nil>} 0 DecimalSI},memory: {{8045252608 0} {<nil>} 7856692Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-03-11 16:48:50 +0000 UTC,LastTransitionTime:2023-03-11 16:20:46 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-03-11 16:48:50 +0000 UTC,LastTransitionTime:2023-03-11 16:20:46 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-03-11 16:48:50 +0000 UTC,LastTransitionTime:2023-03-11 16:20:46 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-03-11 16:48:50 +0000 UTC,LastTransitionTime:2023-03-11 16:21:58 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.40.18,},NodeAddress{Type:ExternalIP,Address:52.194.213.7,},NodeAddress{Type:InternalDNS,Address:i-0e3467ec552af797e.ap-northeast-1.compute.internal,},NodeAddress{Type:Hostname,Address:i-0e3467ec552af797e.ap-northeast-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-52-194-213-7.ap-northeast-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2abf2e564965592c7716accd56507e,SystemUUID:ec2abf2e-5649-6559-2c77-16accd56507e,BootID:fbada3ff-8655-404c-a98d-876577d4c322,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 22.04.2 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.27.0-alpha.3.420+bb6c6fad2c19b3,KubeProxyVersion:v1.27.0-alpha.3.420+bb6c6fad2c19b3,OperatingSystem:linux,Architecture:arm64,},Images:[]ContainerImage{ContainerImage{Names:[registry.k8s.io/etcd@sha256:9ce33ba33d8e738a5b85ed50b5080ac746deceed4a7496c550927a7a19ca3b6d registry.k8s.io/etcd:3.5.0-0],SizeBytes:157797353,},ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:157636062,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:4afb99b4690b418ffc2ceb67e1a17376457e441c1f09ab55447f0aaf992fa646 registry.k8s.io/etcd:3.4.3-0],SizeBytes:151677080,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:4ad90a11b55313b182afc186b9876c8e891531b8db4c9bf1541953021618d0e2 registry.k8s.io/etcd:3.4.13-0],SizeBytes:134531559,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:0d2ab6b15b4fe0b903b9e81192d942fb39c29be61fe341cd16db5d0165b0435c registry.k8s.io/etcd:3.3.17-0],SizeBytes:126979107,},ContainerImage{Names:[registry.k8s.io/kube-apiserver-arm64:v1.27.0-alpha.3.420_bb6c6fad2c19b3],SizeBytes:115123555,},ContainerImage{Names:[registry.k8s.io/kube-controller-manager-arm64:v1.27.0-alpha.3.420_bb6c6fad2c19b3],SizeBytes:107914235,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:17da501f5d2a675be46040422a27b7cc21b8a43895ac998b171db1c346f361f7 registry.k8s.io/etcd:3.3.10-0],SizeBytes:81436097,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:6f72b851544986cb0921b53ea655ec04c36131248f16d4ad110cb3ca0c369dc1 registry.k8s.io/etcd:3.5.4-0],SizeBytes:81120118,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:13f53ed1d91e2e11aac476ee9a0269fdda6cc4874eba903efd40daf50c55eee5 registry.k8s.io/etcd:3.5.3-0],SizeBytes:81114891,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:51eae8381dcb1078289fa7b4f3df2630cdc18d09fb56f8e56b41c40e191d6c83 registry.k8s.io/etcd:3.5.7-0],SizeBytes:80665728,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:dd75ec974b0a2a6f6bb47001ba09207976e625db898d1b16735528c009cb171c registry.k8s.io/etcd:3.5.6-0],SizeBytes:80539316,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:905d7ca17fd02bc24c0eba9a062753aba15db3e31422390bc3238eb762339b20 registry.k8s.io/etcd:3.2.24-1],SizeBytes:61500433,},ContainerImage{Names:[registry.k8s.io/kube-proxy-arm64:v1.27.0-alpha.3.420_bb6c6fad2c19b3],SizeBytes:60947298,},ContainerImage{Names:[registry.k8s.io/etcdadm/etcd-manager-slim@sha256:8727adc059334e0e3329dbccce116a734da4a090da5e86c8cddccaf333caa666 registry.k8s.io/etcdadm/etcd-manager-slim:v3.0.20230201],SizeBytes:60826655,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:64b9ea357325d5db9f8a723dcf503b5a449177b17ac87d69481e126bb724c263 registry.k8s.io/etcd:3.5.1-0],SizeBytes:59553904,},ContainerImage{Names:[registry.k8s.io/kube-scheduler-arm64:v1.27.0-alpha.3.420_bb6c6fad2c19b3],SizeBytes:56731120,},ContainerImage{Names:[registry.k8s.io/kops/kops-controller:1.27.0-alpha.1],SizeBytes:41885191,},ContainerImage{Names:[registry.k8s.io/kops/dns-controller:1.27.0-alpha.1],SizeBytes:40376038,},ContainerImage{Names:[public.ecr.aws/karpenter/controller@sha256:68db4f092cf9cc83f5ef9e2fbc5407c2cb682e81f64dfaa700a7602ede38b1cf public.ecr.aws/karpenter/controller:v0.16.3],SizeBytes:33707751,},ContainerImage{Names:[public.ecr.aws/karpenter/webhook@sha256:96a2d9b06d6bc5127801f358f74b1cf2d289b423a2e9ba40c573c0b14b17dafa public.ecr.aws/karpenter/webhook:v0.16.3],SizeBytes:33307187,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:28932904,},ContainerImage{Names:[quay.io/cilium/operator@sha256:a6d24a006a6b92967ac90786b49bc1ac26e5477cf028cd1186efcfc2466484db quay.io/cilium/operator:v1.12.5],SizeBytes:24567287,},ContainerImage{Names:[registry.k8s.io/provider-aws/cloud-controller-manager@sha256:fdeb61e3e42ecd9cca868d550ebdb88dd6341d9e91fcfa9a37e227dab2ad22cb registry.k8s.io/provider-aws/cloud-controller-manager:v1.26.0],SizeBytes:18345226,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:8502345,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:7685305,},ContainerImage{Names:[registry.k8s.io/kops/kube-apiserver-healthcheck:1.27.0-alpha.1],SizeBytes:4740762,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:253553,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Mar 11 16:53:12.763: INFO: Logging kubelet events for node i-0e3467ec552af797e Mar 11 16:53:12.916: INFO: Logging pods the kubelet thinks is on node i-0e3467ec552af797e Mar 11 16:53:13.220: INFO: cilium-qrsfn started at 2023-03-11 16:21:37 +0000 UTC (1+1 container statuses recorded) Mar 11 16:53:13.220: INFO: Init container clean-cilium-state ready: true, restart count 0 Mar 11 16:53:13.220: INFO: Container cilium-agent ready: true, restart count 0 Mar 11 16:53:13.220: INFO: cilium-operator-8496cc65f5-qhdg8 started at 2023-03-11 16:21:37 +0000 UTC (0+1 container statuses recorded) Mar 11 16:53:13.220: INFO: Container cilium-operator ready: true, restart count 0 Mar 11 16:53:13.220: INFO: kops-controller-76gkw started at 2023-03-11 16:21:37 +0000 UTC (0+1 container statuses recorded) Mar 11 16:53:13.220: INFO: Container kops-controller ready: true, restart count 0 Mar 11 16:53:13.220: INFO: aws-cloud-controller-manager-kwbw2 started at 2023-03-11 16:21:37 +0000 UTC (0+1 container statuses recorded) Mar 11 16:53:13.220: INFO: Container aws-cloud-controller-manager ready: true, restart count 0 Mar 11 16:53:13.220: INFO: kube-apiserver-i-0e3467ec552af797e started at 2023-03-11 16:18:44 +0000 UTC (0+2 container statuses recorded) Mar 11 16:53:13.220: INFO: Container healthcheck ready: true, restart count 0 Mar 11 16:53:13.220: INFO: Container kube-apiserver ready: true, restart count 3 Mar 11 16:53:13.220: INFO: kube-scheduler-i-0e3467ec552af797e started at 2023-03-11 16:18:44 +0000 UTC (0+1 container statuses recorded) Mar 11 16:53:13.220: INFO: Container kube-scheduler ready: true, restart count 0 Mar 11 16:53:13.220: INFO: dns-controller-7dfd64d999-q9hh7 started at 2023-03-11 16:21:37 +0000 UTC (0+1 container statuses recorded) Mar 11 16:53:13.220: INFO: Container dns-controller ready: true, restart count 0 Mar 11 16:53:13.220: INFO: ebs-csi-node-lpwz9 started at 2023-03-11 16:21:37 +0000 UTC (0+3 container statuses recorded) Mar 11 16:53:13.220: INFO: Container ebs-plugin ready: true, restart count 0 Mar 11 16:53:13.220: INFO: Container liveness-probe ready: true, restart count 0 Mar 11 16:53:13.220: INFO: Container node-driver-registrar ready: true, restart count 0 Mar 11 16:53:13.220: INFO: karpenter-webhook-5b646cd4f7-6sg2k started at 2023-03-11 16:21:58 +0000 UTC (0+1 container statuses recorded) Mar 11 16:53:13.220: INFO: Container webhook ready: true, restart count 0 Mar 11 16:53:13.220: INFO: karpenter-controller-567749677d-d9szs started at 2023-03-11 16:21:58 +0000 UTC (0+1 container statuses recorded) Mar 11 16:53:13.220: INFO: Container controller ready: true, restart count 0 Mar 11 16:53:13.220: INFO: kube-controller-manager-i-0e3467ec552af797e started at 2023-03-11 16:18:44 +0000 UTC (0+1 container statuses recorded) Mar 11 16:53:13.220: INFO: Container kube-controller-manager ready: true, restart count 4 Mar 11 16:53:13.220: INFO: etcd-manager-events-i-0e3467ec552af797e started at 2023-03-11 16:18:44 +0000 UTC (11+1 container statuses recorded) Mar 11 16:53:13.220: INFO: Init container init-etcd-3-2-24 ready: true, restart count 0 Mar 11 16:53:13.220: INFO: Init container init-etcd-3-3-10 ready: true, restart count 0 Mar 11 16:53:13.220: INFO: Init container init-etcd-3-3-17 ready: true, restart count 0 Mar 11 16:53:13.220: INFO: Init container init-etcd-3-4-13 ready: true, restart count 0 Mar 11 16:53:13.220: INFO: Init container init-etcd-3-4-3 ready: true, restart count 0 Mar 11 16:53:13.220: INFO: Init container init-etcd-3-5-0 ready: true, restart count 0 Mar 11 16:53:13.220: INFO: Init container init-etcd-3-5-1 ready: true, restart count 0 Mar 11 16:53:13.220: INFO: Init container init-etcd-3-5-3 ready: true, restart count 0 Mar 11 16:53:13.220: INFO: Init container init-etcd-3-5-4 ready: true, restart count 0 Mar 11 16:53:13.220: INFO: Init container init-etcd-3-5-6 ready: true, restart count 0 Mar 11 16:53:13.220: INFO: Init container init-etcd-3-5-7 ready: true, restart count 0 Mar 11 16:53:13.220: INFO: Container etcd-manager ready: true, restart count 0 Mar 11 16:53:13.220: INFO: etcd-manager-main-i-0e3467ec552af797e started at 2023-03-11 16:18:44 +0000 UTC (11+1 container statuses recorded) Mar 11 16:53:13.220: INFO: Init container init-etcd-3-2-24 ready: true, restart count 0 Mar 11 16:53:13.220: INFO: Init container init-etcd-3-3-10 ready: true, restart count 0 Mar 11 16:53:13.220: INFO: Init container init-etcd-3-3-17 ready: true, restart count 0 Mar 11 16:53:13.220: INFO: Init container init-etcd-3-4-13 ready: true, restart count 0 Mar 11 16:53:13.220: INFO: Init container init-etcd-3-4-3 ready: true, restart count 0 Mar 11 16:53:13.220: INFO: Init container init-etcd-3-5-0 ready: true, restart count 0 Mar 11 16:53:13.220: INFO: Init container init-etcd-3-5-1 ready: true, restart count 0 Mar 11 16:53:13.220: INFO: Init container init-etcd-3-5-3 ready: true, restart count 0 Mar 11 16:53:13.220: INFO: Init container init-etcd-3-5-4 ready: true, restart count 0 Mar 11 16:53:13.220: INFO: Init container init-etcd-3-5-6 ready: true, restart count 0 Mar 11 16:53:13.220: INFO: Init container init-etcd-3-5-7 ready: true, restart count 0 Mar 11 16:53:13.220: INFO: Container etcd-manager ready: true, restart count 0 Mar 11 16:53:13.716: INFO: Latency metrics for node i-0e3467ec552af797e END STEP: dump namespace information after failure - test/e2e/framework/framework.go:288 @ 03/11/23 16:53:13.716 (22.453s) < Exit [DeferCleanup (Each)] [Testpattern: Generic Ephemeral-volume (default fs) (late-binding)] ephemeral - dump namespaces | framework.go:209 @ 03/11/23 16:53:13.716 (22.453s) > Enter [DeferCleanup (Each)] [Testpattern: Generic Ephemeral-volume (default fs) (late-binding)] ephemeral - tear down framework | framework.go:206 @ 03/11/23 16:53:13.716 STEP: Destroying namespace "ephemeral-9978" for this suite. - test/e2e/framework/framework.go:351 @ 03/11/23 16:53:13.716 < Exit [DeferCleanup (Each)] [Testpattern: Generic Ephemeral-volume (default fs) (late-binding)] ephemeral - tear down framework | framework.go:206 @ 03/11/23 16:53:13.868 (152ms) > Enter [ReportAfterEach] TOP-LEVEL - test/e2e/e2e_test.go:144 @ 03/11/23 16:53:13.868 < Exit [ReportAfterEach] TOP-LEVEL - test/e2e/e2e_test.go:144 @ 03/11/23 16:53:13.868 (0s)
Filter through log files | View test history on testgrid
go run hack/e2e.go -v --test --test_args='--ginkgo.focus=Kubernetes\se2e\ssuite\s\[It\]\s\[sig\-network\]\sEndpointSlice\sshould\ssupport\sa\sService\swith\smultiple\sports\sspecified\sin\smultiple\sEndpointSlices$'
[FAILED] error running /home/prow/go/src/k8s.io/kops/_rundir/c9b7e3a4-c027-11ed-9d91-c209e83e3776/kubectl --server=https://api.e2e-e2e-kops-aws-karpenter.test-cncf-aws.k8s.io --kubeconfig=/root/.kube/config --namespace=endpointslice-2030 exec pause-pod-0 -- /bin/sh -x -c curl -q -s --connect-timeout 30 100.65.232.188:80/hostname: Command stdout: stderr: + curl -q -s --connect-timeout 30 100.65.232.188:80/hostname command terminated with exit code 28 error: exit status 28 In [It] at: test/e2e/network/util.go:180 @ 03/11/23 16:35:32.292from junit_01.xml
> Enter [BeforeEach] [sig-network] EndpointSlice - set up framework | framework.go:191 @ 03/11/23 16:33:11.353 STEP: Creating a kubernetes client - test/e2e/framework/framework.go:211 @ 03/11/23 16:33:11.353 Mar 11 16:33:11.353: INFO: >>> kubeConfig: /root/.kube/config STEP: Building a namespace api object, basename endpointslice - test/e2e/framework/framework.go:250 @ 03/11/23 16:33:11.354 STEP: Waiting for a default service account to be provisioned in namespace - test/e2e/framework/framework.go:259 @ 03/11/23 16:33:11.801 STEP: Waiting for kube-root-ca.crt to be provisioned in namespace - test/e2e/framework/framework.go:262 @ 03/11/23 16:33:12.096 < Exit [BeforeEach] [sig-network] EndpointSlice - set up framework | framework.go:191 @ 03/11/23 16:33:12.392 (1.04s) > Enter [BeforeEach] [sig-network] EndpointSlice - test/e2e/framework/metrics/init/init.go:33 @ 03/11/23 16:33:12.392 < Exit [BeforeEach] [sig-network] EndpointSlice - test/e2e/framework/metrics/init/init.go:33 @ 03/11/23 16:33:12.392 (0s) > Enter [BeforeEach] [sig-network] EndpointSlice - test/e2e/network/endpointslice.go:54 @ 03/11/23 16:33:12.392 < Exit [BeforeEach] [sig-network] EndpointSlice - test/e2e/network/endpointslice.go:54 @ 03/11/23 16:33:12.392 (0s) > Enter [It] should support a Service with multiple ports specified in multiple EndpointSlices - test/e2e/network/endpointslice.go:530 @ 03/11/23 16:33:12.392 STEP: creating - test/e2e/network/endpointslice.go:597 @ 03/11/23 16:33:15.147 STEP: Creating a pause pods that will try to connect to the webserver - test/e2e/network/endpointslice.go:618 @ 03/11/23 16:33:15.449 Mar 11 16:33:18.050: INFO: Waiting up to 2m0s to get response from 100.65.232.188:80 Mar 11 16:33:18.050: INFO: Running '/home/prow/go/src/k8s.io/kops/_rundir/c9b7e3a4-c027-11ed-9d91-c209e83e3776/kubectl --server=https://api.e2e-e2e-kops-aws-karpenter.test-cncf-aws.k8s.io --kubeconfig=/root/.kube/config --namespace=endpointslice-2030 exec pause-pod-0 -- /bin/sh -x -c curl -q -s --connect-timeout 30 100.65.232.188:80/hostname' Mar 11 16:33:49.578: INFO: rc: 28 Mar 11 16:33:49.578: INFO: got err: error running /home/prow/go/src/k8s.io/kops/_rundir/c9b7e3a4-c027-11ed-9d91-c209e83e3776/kubectl --server=https://api.e2e-e2e-kops-aws-karpenter.test-cncf-aws.k8s.io --kubeconfig=/root/.kube/config --namespace=endpointslice-2030 exec pause-pod-0 -- /bin/sh -x -c curl -q -s --connect-timeout 30 100.65.232.188:80/hostname: Command stdout: stderr: + curl -q -s --connect-timeout 30 100.65.232.188:80/hostname command terminated with exit code 28 error: exit status 28, retry until timeout Mar 11 16:33:51.578: INFO: Running '/home/prow/go/src/k8s.io/kops/_rundir/c9b7e3a4-c027-11ed-9d91-c209e83e3776/kubectl --server=https://api.e2e-e2e-kops-aws-karpenter.test-cncf-aws.k8s.io --kubeconfig=/root/.kube/config --namespace=endpointslice-2030 exec pause-pod-0 -- /bin/sh -x -c curl -q -s --connect-timeout 30 100.65.232.188:80/hostname' Mar 11 16:34:23.209: INFO: rc: 28 Mar 11 16:34:23.209: INFO: got err: error running /home/prow/go/src/k8s.io/kops/_rundir/c9b7e3a4-c027-11ed-9d91-c209e83e3776/kubectl --server=https://api.e2e-e2e-kops-aws-karpenter.test-cncf-aws.k8s.io --kubeconfig=/root/.kube/config --namespace=endpointslice-2030 exec pause-pod-0 -- /bin/sh -x -c curl -q -s --connect-timeout 30 100.65.232.188:80/hostname: Command stdout: stderr: + curl -q -s --connect-timeout 30 100.65.232.188:80/hostname command terminated with exit code 28 error: exit status 28, retry until timeout Mar 11 16:34:25.210: INFO: Running '/home/prow/go/src/k8s.io/kops/_rundir/c9b7e3a4-c027-11ed-9d91-c209e83e3776/kubectl --server=https://api.e2e-e2e-kops-aws-karpenter.test-cncf-aws.k8s.io --kubeconfig=/root/.kube/config --namespace=endpointslice-2030 exec pause-pod-0 -- /bin/sh -x -c curl -q -s --connect-timeout 30 100.65.232.188:80/hostname' Mar 11 16:34:56.727: INFO: rc: 28 Mar 11 16:34:56.727: INFO: got err: error running /home/prow/go/src/k8s.io/kops/_rundir/c9b7e3a4-c027-11ed-9d91-c209e83e3776/kubectl --server=https://api.e2e-e2e-kops-aws-karpenter.test-cncf-aws.k8s.io --kubeconfig=/root/.kube/config --namespace=endpointslice-2030 exec pause-pod-0 -- /bin/sh -x -c curl -q -s --connect-timeout 30 100.65.232.188:80/hostname: Command stdout: stderr: + curl -q -s --connect-timeout 30 100.65.232.188:80/hostname command terminated with exit code 28 error: exit status 28, retry until timeout Mar 11 16:34:58.727: INFO: Running '/home/prow/go/src/k8s.io/kops/_rundir/c9b7e3a4-c027-11ed-9d91-c209e83e3776/kubectl --server=https://api.e2e-e2e-kops-aws-karpenter.test-cncf-aws.k8s.io --kubeconfig=/root/.kube/config --namespace=endpointslice-2030 exec pause-pod-0 -- /bin/sh -x -c curl -q -s --connect-timeout 30 100.65.232.188:80/hostname' Mar 11 16:35:30.290: INFO: rc: 28 Mar 11 16:35:30.291: INFO: got err: error running /home/prow/go/src/k8s.io/kops/_rundir/c9b7e3a4-c027-11ed-9d91-c209e83e3776/kubectl --server=https://api.e2e-e2e-kops-aws-karpenter.test-cncf-aws.k8s.io --kubeconfig=/root/.kube/config --namespace=endpointslice-2030 exec pause-pod-0 -- /bin/sh -x -c curl -q -s --connect-timeout 30 100.65.232.188:80/hostname: Command stdout: stderr: + curl -q -s --connect-timeout 30 100.65.232.188:80/hostname command terminated with exit code 28 error: exit status 28, retry until timeout Mar 11 16:35:32.292: INFO: Unexpected error: <exec.CodeExitError>: { Err: <*errors.errorString | 0xc00137a010>{ s: "error running /home/prow/go/src/k8s.io/kops/_rundir/c9b7e3a4-c027-11ed-9d91-c209e83e3776/kubectl --server=https://api.e2e-e2e-kops-aws-karpenter.test-cncf-aws.k8s.io --kubeconfig=/root/.kube/config --namespace=endpointslice-2030 exec pause-pod-0 -- /bin/sh -x -c curl -q -s --connect-timeout 30 100.65.232.188:80/hostname:\nCommand stdout:\n\nstderr:\n+ curl -q -s --connect-timeout 30 100.65.232.188:80/hostname\ncommand terminated with exit code 28\n\nerror:\nexit status 28", }, Code: 28, } [FAILED] error running /home/prow/go/src/k8s.io/kops/_rundir/c9b7e3a4-c027-11ed-9d91-c209e83e3776/kubectl --server=https://api.e2e-e2e-kops-aws-karpenter.test-cncf-aws.k8s.io --kubeconfig=/root/.kube/config --namespace=endpointslice-2030 exec pause-pod-0 -- /bin/sh -x -c curl -q -s --connect-timeout 30 100.65.232.188:80/hostname: Command stdout: stderr: + curl -q -s --connect-timeout 30 100.65.232.188:80/hostname command terminated with exit code 28 error: exit status 28 In [It] at: test/e2e/network/util.go:180 @ 03/11/23 16:35:32.292 < Exit [It] should support a Service with multiple ports specified in multiple EndpointSlices - test/e2e/network/endpointslice.go:530 @ 03/11/23 16:35:32.292 (2m19.9s) > Enter [AfterEach] [sig-network] EndpointSlice - test/e2e/framework/node/init/init.go:33 @ 03/11/23 16:35:32.292 Mar 11 16:35:32.292: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready < Exit [AfterEach] [sig-network] EndpointSlice - test/e2e/framework/node/init/init.go:33 @ 03/11/23 16:35:32.442 (150ms) > Enter [DeferCleanup (Each)] [sig-network] EndpointSlice - test/e2e/framework/metrics/init/init.go:35 @ 03/11/23 16:35:32.442 < Exit [DeferCleanup (Each)] [sig-network] EndpointSlice - test/e2e/framework/metrics/init/init.go:35 @ 03/11/23 16:35:32.442 (0s) > Enter [DeferCleanup (Each)] [sig-network] EndpointSlice - dump namespaces | framework.go:209 @ 03/11/23 16:35:32.442 STEP: dump namespace information after failure - test/e2e/framework/framework.go:288 @ 03/11/23 16:35:32.442 STEP: Collecting events from namespace "endpointslice-2030". - test/e2e/framework/debug/dump.go:42 @ 03/11/23 16:35:32.442 STEP: Found 11 events. - test/e2e/framework/debug/dump.go:46 @ 03/11/23 16:35:32.591 Mar 11 16:35:32.591: INFO: At 2023-03-11 16:33:12 +0000 UTC - event for pod-handle-http-request: {default-scheduler } Scheduled: Successfully assigned endpointslice-2030/pod-handle-http-request to i-04523d1dffb0fe725 Mar 11 16:35:32.591: INFO: At 2023-03-11 16:33:13 +0000 UTC - event for pod-handle-http-request: {kubelet i-04523d1dffb0fe725} Pulled: Container image "registry.k8s.io/e2e-test-images/agnhost:2.43" already present on machine Mar 11 16:35:32.591: INFO: At 2023-03-11 16:33:13 +0000 UTC - event for pod-handle-http-request: {kubelet i-04523d1dffb0fe725} Created: Created container container-handle-8090-request Mar 11 16:35:32.591: INFO: At 2023-03-11 16:33:13 +0000 UTC - event for pod-handle-http-request: {kubelet i-04523d1dffb0fe725} Started: Started container container-handle-8090-request Mar 11 16:35:32.591: INFO: At 2023-03-11 16:33:13 +0000 UTC - event for pod-handle-http-request: {kubelet i-04523d1dffb0fe725} Pulled: Container image "registry.k8s.io/e2e-test-images/agnhost:2.43" already present on machine Mar 11 16:35:32.591: INFO: At 2023-03-11 16:33:13 +0000 UTC - event for pod-handle-http-request: {kubelet i-04523d1dffb0fe725} Created: Created container container-handle-9090-request Mar 11 16:35:32.591: INFO: At 2023-03-11 16:33:13 +0000 UTC - event for pod-handle-http-request: {kubelet i-04523d1dffb0fe725} Started: Started container container-handle-9090-request Mar 11 16:35:32.591: INFO: At 2023-03-11 16:33:15 +0000 UTC - event for pause-pod-0: {default-scheduler } Scheduled: Successfully assigned endpointslice-2030/pause-pod-0 to i-07e2b6e650d7ef56d Mar 11 16:35:32.591: INFO: At 2023-03-11 16:33:16 +0000 UTC - event for pause-pod-0: {kubelet i-07e2b6e650d7ef56d} Pulled: Container image "registry.k8s.io/e2e-test-images/agnhost:2.43" already present on machine Mar 11 16:35:32.591: INFO: At 2023-03-11 16:33:16 +0000 UTC - event for pause-pod-0: {kubelet i-07e2b6e650d7ef56d} Created: Created container agnhost-container Mar 11 16:35:32.591: INFO: At 2023-03-11 16:33:16 +0000 UTC - event for pause-pod-0: {kubelet i-07e2b6e650d7ef56d} Started: Started container agnhost-container Mar 11 16:35:32.739: INFO: POD NODE PHASE GRACE CONDITIONS Mar 11 16:35:32.739: INFO: pause-pod-0 i-07e2b6e650d7ef56d Running [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2023-03-11 16:33:15 +0000 UTC } {Ready True 0001-01-01 00:00:00 +0000 UTC 2023-03-11 16:33:17 +0000 UTC } {ContainersReady True 0001-01-01 00:00:00 +0000 UTC 2023-03-11 16:33:17 +0000 UTC } {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2023-03-11 16:33:15 +0000 UTC }] Mar 11 16:35:32.739: INFO: pod-handle-http-request i-04523d1dffb0fe725 Running [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2023-03-11 16:33:12 +0000 UTC } {Ready True 0001-01-01 00:00:00 +0000 UTC 2023-03-11 16:33:14 +0000 UTC } {ContainersReady True 0001-01-01 00:00:00 +0000 UTC 2023-03-11 16:33:14 +0000 UTC } {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2023-03-11 16:33:12 +0000 UTC }] Mar 11 16:35:32.739: INFO: Mar 11 16:35:33.359: INFO: Logging node info for node i-04523d1dffb0fe725 Mar 11 16:35:33.508: INFO: Node Info: &Node{ObjectMeta:{i-04523d1dffb0fe725 5d37640f-a7d9-4c26-86c5-9da21f7ccc18 22934 0 2023-03-11 16:24:20 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:arm64 beta.kubernetes.io/instance-type:t4g.small beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-northeast-1 failure-domain.beta.kubernetes.io/zone:ap-northeast-1a karpenter.k8s.aws/instance-category:t karpenter.k8s.aws/instance-cpu:2 karpenter.k8s.aws/instance-family:t4g karpenter.k8s.aws/instance-generation:4 karpenter.k8s.aws/instance-hypervisor:nitro karpenter.k8s.aws/instance-memory:2048 karpenter.k8s.aws/instance-pods:11 karpenter.k8s.aws/instance-size:small karpenter.sh/capacity-type:spot karpenter.sh/initialized:true karpenter.sh/provisioner-name:nodes kubernetes.io/arch:arm64 kubernetes.io/hostname:i-04523d1dffb0fe725 kubernetes.io/os:linux node-role.kubernetes.io/node: node-role.kubernetes.io/spot-worker:true node.kubernetes.io/instance-type:t4g.small topology.ebs.csi.aws.com/zone:ap-northeast-1a topology.hostpath.csi/node:i-04523d1dffb0fe725 topology.kubernetes.io/region:ap-northeast-1 topology.kubernetes.io/zone:ap-northeast-1a] map[csi.volume.kubernetes.io/nodeid:{"csi-hostpath-ephemeral-7532":"i-04523d1dffb0fe725","csi-hostpath-provisioning-482":"i-04523d1dffb0fe725","ebs.csi.aws.com":"i-04523d1dffb0fe725"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [{karpenter.sh/v1alpha5 Provisioner nodes ac7283b2-eed0-4a9f-9ddd-616f33a0991a <nil> 0xc0031abab5}] [karpenter.sh/termination] [{aws-cloud-controller-manager Update v1 2023-03-11 16:24:20 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{}}}} } {kops-controller Update v1 2023-03-11 16:24:20 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{},"f:node-role.kubernetes.io/spot-worker":{}}}} } {karpenter Update v1 2023-03-11 16:26:12 +0000 UTC FieldsV1 {"f:metadata":{"f:finalizers":{".":{},"v:\"karpenter.sh/termination\"":{}},"f:labels":{".":{},"f:karpenter.k8s.aws/instance-category":{},"f:karpenter.k8s.aws/instance-cpu":{},"f:karpenter.k8s.aws/instance-family":{},"f:karpenter.k8s.aws/instance-generation":{},"f:karpenter.k8s.aws/instance-hypervisor":{},"f:karpenter.k8s.aws/instance-memory":{},"f:karpenter.k8s.aws/instance-pods":{},"f:karpenter.k8s.aws/instance-size":{},"f:karpenter.sh/capacity-type":{},"f:karpenter.sh/initialized":{},"f:karpenter.sh/provisioner-name":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}},"f:ownerReferences":{".":{},"k:{\"uid\":\"ac7283b2-eed0-4a9f-9ddd-616f33a0991a\"}":{}}},"f:spec":{"f:providerID":{}}} } {kube-controller-manager Update v1 2023-03-11 16:26:12 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:node.alpha.kubernetes.io/ttl":{}},"f:labels":{"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.2.0/24\"":{}}}} } {aws-cloud-controller-manager Update v1 2023-03-11 16:26:50 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{".":{},"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"InternalIP\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kube-controller-manager Update v1 2023-03-11 16:35:25 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{".":{},"k:{\"type\":\"DiskPressure\"}":{".":{},"f:type":{}},"k:{\"type\":\"MemoryPressure\"}":{".":{},"f:type":{}},"k:{\"type\":\"PIDPressure\"}":{".":{},"f:type":{}},"k:{\"type\":\"Ready\"}":{".":{},"f:type":{}}},"f:volumesAttached":{}}} status} {kubelet Update v1 2023-03-11 16:35:32 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{"f:kubernetes.io/hostname":{},"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{".":{},"f:cpu":{},"f:ephemeral-storage":{},"f:hugepages-1Gi":{},"f:hugepages-2Mi":{},"f:hugepages-32Mi":{},"f:hugepages-64Ki":{},"f:memory":{},"f:pods":{}},"f:capacity":{".":{},"f:cpu":{},"f:ephemeral-storage":{},"f:hugepages-1Gi":{},"f:hugepages-2Mi":{},"f:hugepages-32Mi":{},"f:hugepages-64Ki":{},"f:memory":{},"f:pods":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:daemonEndpoints":{"f:kubeletEndpoint":{"f:Port":{}}},"f:images":{},"f:nodeInfo":{"f:architecture":{},"f:bootID":{},"f:containerRuntimeVersion":{},"f:kernelVersion":{},"f:kubeProxyVersion":{},"f:kubeletVersion":{},"f:machineID":{},"f:operatingSystem":{},"f:osImage":{},"f:systemUUID":{}},"f:volumesInUse":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.2.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-northeast-1a/i-04523d1dffb0fe725,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.2.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49756430336 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-32Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-64Ki: {{0 0} {<nil>} 0 DecimalSI},memory: {{1941540864 0} {<nil>} 1896036Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44780787229 0} {<nil>} 44780787229 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-32Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-64Ki: {{0 0} {<nil>} 0 DecimalSI},memory: {{1836683264 0} {<nil>} 1793636Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-03-11 16:35:23 +0000 UTC,LastTransitionTime:2023-03-11 16:26:12 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-03-11 16:35:23 +0000 UTC,LastTransitionTime:2023-03-11 16:26:02 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-03-11 16:35:23 +0000 UTC,LastTransitionTime:2023-03-11 16:26:02 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-03-11 16:35:23 +0000 UTC,LastTransitionTime:2023-03-11 16:26:02 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.48.168,},NodeAddress{Type:ExternalIP,Address:52.199.253.156,},NodeAddress{Type:InternalDNS,Address:i-04523d1dffb0fe725.ap-northeast-1.compute.internal,},NodeAddress{Type:Hostname,Address:i-04523d1dffb0fe725.ap-northeast-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-52-199-253-156.ap-northeast-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2c8fb0e6b2e9c4af310f65b6161cd8,SystemUUID:ec2c8fb0-e6b2-e9c4-af31-0f65b6161cd8,BootID:a3b8de83-558b-44c4-a46a-d097c3f98225,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 22.04.2 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.27.0-alpha.3.420+bb6c6fad2c19b3,KubeProxyVersion:v1.27.0-alpha.3.420+bb6c6fad2c19b3,OperatingSystem:linux,Architecture:arm64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:157636062,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/jessie-dnsutils@sha256:24aaf2626d6b27864c29de2097e8bbb840b3a414271bf7c8995e431e47d8408e registry.k8s.io/e2e-test-images/jessie-dnsutils:1.7],SizeBytes:116764683,},ContainerImage{Names:[registry.k8s.io/kube-proxy-arm64:v1.27.0-alpha.3.420_bb6c6fad2c19b3],SizeBytes:60947298,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:16bbf38c463a4223d8cfe4da12bc61010b082a79b4bb003e2d3ba3ece5dd5f9e registry.k8s.io/e2e-test-images/agnhost:2.43],SizeBytes:49808364,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nautilus@sha256:80ba6c8c44f9623f06e868a1aa66026c8ec438ad814f9ec95e9333b415fe3550 registry.k8s.io/e2e-test-images/nautilus:1.7],SizeBytes:47953400,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:3fe7acf013d1264ffded116b80a73dc129a449b0fccdb8d21af8279f2233f36e registry.k8s.io/e2e-test-images/httpd:2.4.39-4],SizeBytes:41528818,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:148b022f5c5da426fc2f3c14b5c0867e58ef05961510c84749ac1fddcb0fef22 registry.k8s.io/e2e-test-images/httpd:2.4.38-4],SizeBytes:40343601,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:28932904,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:ee3b525d5b89db99da3b8eb521d9cd90cb6e9ef0fbb651e98bb37be78d36b5b8 registry.k8s.io/sig-storage/csi-provisioner:v3.3.0],SizeBytes:23945474,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:425d8f1b769398127767b06ed97ce62578a3179bcb99809ce93a1649e025ffe7 registry.k8s.io/sig-storage/csi-resizer:v1.6.0],SizeBytes:22678632,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:291334908ddf71a4661fd7f6d9d97274de8a5378a2b6fdfeb2ce73414a34f82f registry.k8s.io/sig-storage/csi-snapshotter:v6.1.0],SizeBytes:22434180,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:9a685020911e2725ad019dbce6e4a5ab93d51e3d4557f115e64343345e05781b registry.k8s.io/sig-storage/csi-attacher:v4.0.0],SizeBytes:22394501,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:122bfb8c1edabb3c0edd63f06523e6940d958d19b3957dc7b1d6f81e9f1f6119 registry.k8s.io/sig-storage/csi-provisioner:v3.1.0],SizeBytes:21735644,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:9ebbf9f023e7b41ccee3d52afe39a89e3ddacdbb69269d583abfc25847cfd9e4 registry.k8s.io/sig-storage/csi-resizer:v1.4.0],SizeBytes:20848413,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:8b9c313c05f54fb04f8d430896f5f5904b6cb157df261501b29adc04d2b2dc7b registry.k8s.io/sig-storage/csi-attacher:v3.4.0],SizeBytes:20565480,},ContainerImage{Names:[registry.k8s.io/cpa/cluster-proportional-autoscaler@sha256:68d396900aeaa072c1f27289485fdac29834045a6f3ffe369bf389d830ef572d registry.k8s.io/cpa/cluster-proportional-autoscaler:1.8.6],SizeBytes:19007623,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nonroot@sha256:ee9f50b3c64b174d296d91ca9f69a914ac30e59095dfb462b2b518ad28a63655 registry.k8s.io/e2e-test-images/nonroot:1.4],SizeBytes:18561972,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:92257881c1d6493cf18299a24af42330f891166560047902b8d431fb66b01af5 registry.k8s.io/sig-storage/hostpathplugin:v1.9.0],SizeBytes:18267611,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:9b2d9abda017c28b12f84a344f57cd73fbdb6c2bd7dd5bdd5018246ad1093ba6 registry.k8s.io/sig-storage/hostpathplugin:v1.11.0],SizeBytes:18159085,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:017727efcfeb7d053af68e51436ce8e65edbc6ca573720afb4f79c8594036955 registry.k8s.io/coredns/coredns:v1.10.0],SizeBytes:13766234,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:8502345,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:933940f13b3ea0abc62e656c1aa5c5b47c04b15d71250413a6b821bd0c58b94e registry.k8s.io/sig-storage/livenessprobe:v2.7.0],SizeBytes:8223819,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:7685305,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:db048754ae68ae337d8fa96494c96d2a1204c3320f5dcf7e8e71085adec85da6 registry.k8s.io/e2e-test-images/nginx:1.15-4],SizeBytes:6883347,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:5c99cf6a02adda929b10321dbf4ecfa00d87be9ba4fb456006237d530ab4baa1 registry.k8s.io/e2e-test-images/nginx:1.14-4],SizeBytes:6860089,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf registry.k8s.io/e2e-test-images/busybox:1.29-2],SizeBytes:773367,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:2e0f836850e09b8b7cc937681d6194537a09fbd5f6b9e08f4d646a85128e8937 registry.k8s.io/e2e-test-images/busybox:1.29-4],SizeBytes:772933,},ContainerImage{Names:[registry.k8s.io/pause@sha256:7031c1b283388d2c2e09b57badb803c05ebed362dc88d84b480cc47f72a21097 registry.k8s.io/pause:3.9],SizeBytes:268051,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:253553,},},VolumesInUse:[kubernetes.io/csi/ebs.csi.aws.com^vol-00ca46dcf58a4c997 kubernetes.io/csi/ebs.csi.aws.com^vol-01f4724617628c8e4 kubernetes.io/csi/ebs.csi.aws.com^vol-0a99c16a1826516dd kubernetes.io/csi/ebs.csi.aws.com^vol-0b9068d8f2ae1da97 kubernetes.io/csi/ebs.csi.aws.com^vol-0cc38fbba557b7a10],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-0b9068d8f2ae1da97,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-00ca46dcf58a4c997,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-01f4724617628c8e4,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-0a99c16a1826516dd,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-0cc38fbba557b7a10,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/csi-hostpath-provisioning-482^b8f08d61-c02a-11ed-bd24-e6d627d37248,DevicePath:,},},Config:nil,},} Mar 11 16:35:33.509: INFO: Logging kubelet events for node i-04523d1dffb0fe725 Mar 11 16:35:33.661: INFO: Logging pods the kubelet thinks is on node i-04523d1dffb0fe725 Mar 11 16:35:33.819: INFO: pod-6d61f539-1934-403b-851e-db7da48a34c7 started at 2023-03-11 16:35:08 +0000 UTC (0+1 container statuses recorded) Mar 11 16:35:33.819: INFO: Container write-pod ready: true, restart count 0 Mar 11 16:35:33.819: INFO: netserver-0 started at 2023-03-11 16:35:08 +0000 UTC (0+1 container statuses recorded) Mar 11 16:35:33.819: INFO: Container webserver ready: true, restart count 0 Mar 11 16:35:33.819: INFO: pod-e6269945-3bac-473e-b684-4b3b68b9c558 started at 2023-03-11 16:35:17 +0000 UTC (0+1 container statuses recorded) Mar 11 16:35:33.819: INFO: Container write-pod ready: true, restart count 0 Mar 11 16:35:33.819: INFO: ebs-csi-node-2qlbt started at 2023-03-11 16:26:03 +0000 UTC (0+3 container statuses recorded) Mar 11 16:35:33.819: INFO: Container ebs-plugin ready: true, restart count 0 Mar 11 16:35:33.819: INFO: Container liveness-probe ready: true, restart count 0 Mar 11 16:35:33.819: INFO: Container node-driver-registrar ready: true, restart count 0 Mar 11 16:35:33.819: INFO: coredns-745bddcfbb-s57v5 started at 2023-03-11 16:26:12 +0000 UTC (0+1 container statuses recorded) Mar 11 16:35:33.819: INFO: Container coredns ready: true, restart count 0 Mar 11 16:35:33.819: INFO: hostpath-client started at 2023-03-11 16:35:24 +0000 UTC (0+1 container statuses recorded) Mar 11 16:35:33.819: INFO: Container hostpath-client ready: false, restart count 0 Mar 11 16:35:33.819: INFO: cilium-kfwlg started at 2023-03-11 16:26:03 +0000 UTC (1+1 container statuses recorded) Mar 11 16:35:33.819: INFO: Init container clean-cilium-state ready: true, restart count 0 Mar 11 16:35:33.819: INFO: Container cilium-agent ready: true, restart count 0 Mar 11 16:35:33.819: INFO: webserver-pod started at 2023-03-11 16:35:07 +0000 UTC (0+1 container statuses recorded) Mar 11 16:35:33.819: INFO: Container agnhost ready: true, restart count 0 Mar 11 16:35:33.819: INFO: csi-hostpathplugin-0 started at 2023-03-11 16:34:55 +0000 UTC (0+7 container statuses recorded) Mar 11 16:35:33.819: INFO: Container csi-attacher ready: true, restart count 0 Mar 11 16:35:33.819: INFO: Container csi-provisioner ready: true, restart count 0 Mar 11 16:35:33.819: INFO: Container csi-resizer ready: true, restart count 0 Mar 11 16:35:33.819: INFO: Container csi-snapshotter ready: true, restart count 0 Mar 11 16:35:33.819: INFO: Container hostpath ready: true, restart count 0 Mar 11 16:35:33.819: INFO: Container liveness-probe ready: true, restart count 0 Mar 11 16:35:33.819: INFO: Container node-driver-registrar ready: true, restart count 0 Mar 11 16:35:33.819: INFO: pod-exec-websocket-3a71d612-9f93-4f6b-b9ab-2a0679effc51 started at 2023-03-11 16:35:22 +0000 UTC (0+1 container statuses recorded) Mar 11 16:35:33.819: INFO: Container main ready: true, restart count 0 Mar 11 16:35:33.819: INFO: pod-handle-http-request started at 2023-03-11 16:33:12 +0000 UTC (0+2 container statuses recorded) Mar 11 16:35:33.819: INFO: Container container-handle-8090-request ready: true, restart count 0 Mar 11 16:35:33.819: INFO: Container container-handle-9090-request ready: true, restart count 0 Mar 11 16:35:33.819: INFO: test-container-pod started at 2023-03-11 16:35:31 +0000 UTC (0+1 container statuses recorded) Mar 11 16:35:33.819: INFO: Container webserver ready: true, restart count 0 Mar 11 16:35:33.819: INFO: fail-once-non-local-gcwdg started at 2023-03-11 16:35:29 +0000 UTC (0+1 container statuses recorded) Mar 11 16:35:33.819: INFO: Container c ready: false, restart count 0 Mar 11 16:35:33.819: INFO: ebs-csi-controller-79785c665b-2wwsh started at 2023-03-11 16:26:12 +0000 UTC (0+5 container statuses recorded) Mar 11 16:35:33.819: INFO: Container csi-attacher ready: true, restart count 0 Mar 11 16:35:33.819: INFO: Container csi-provisioner ready: true, restart count 0 Mar 11 16:35:33.819: INFO: Container csi-resizer ready: true, restart count 0 Mar 11 16:35:33.819: INFO: Container ebs-plugin ready: true, restart count 0 Mar 11 16:35:33.819: INFO: Container liveness-probe ready: true, restart count 0 Mar 11 16:35:33.819: INFO: hostexec-i-04523d1dffb0fe725-l5b7g started at 2023-03-11 16:35:01 +0000 UTC (0+1 container statuses recorded) Mar 11 16:35:33.819: INFO: Container agnhost-container ready: true, restart count 0 Mar 11 16:35:33.819: INFO: ss-0 started at 2023-03-11 16:35:13 +0000 UTC (0+1 container statuses recorded) Mar 11 16:35:33.819: INFO: Container webserver ready: true, restart count 0 Mar 11 16:35:33.819: INFO: fail-once-non-local-qssph started at 2023-03-11 16:35:29 +0000 UTC (0+1 container statuses recorded) Mar 11 16:35:33.819: INFO: Container c ready: false, restart count 0 Mar 11 16:35:33.819: INFO: csi-hostpathplugin-0 started at 2023-03-11 16:35:31 +0000 UTC (0+7 container statuses recorded) Mar 11 16:35:33.819: INFO: Container csi-attacher ready: false, restart count 0 Mar 11 16:35:33.819: INFO: Container csi-provisioner ready: false, restart count 0 Mar 11 16:35:33.819: INFO: Container csi-resizer ready: false, restart count 0 Mar 11 16:35:33.819: INFO: Container csi-snapshotter ready: false, restart count 0 Mar 11 16:35:33.819: INFO: Container hostpath ready: false, restart count 0 Mar 11 16:35:33.819: INFO: Container liveness-probe ready: false, restart count 0 Mar 11 16:35:33.819: INFO: Container node-driver-registrar ready: false, restart count 0 Mar 11 16:35:33.819: INFO: test-ss-0 started at 2023-03-11 16:34:46 +0000 UTC (0+1 container statuses recorded) Mar 11 16:35:33.819: INFO: Container webserver ready: true, restart count 0 Mar 11 16:35:33.819: INFO: ss-2 started at 2023-03-11 16:35:16 +0000 UTC (0+1 container statuses recorded) Mar 11 16:35:33.819: INFO: Container webserver ready: true, restart count 0 Mar 11 16:35:33.819: INFO: e2e-gpgcj-j4rqp started at 2023-03-11 16:35:25 +0000 UTC (0+1 container statuses recorded) Mar 11 16:35:33.819: INFO: Container c ready: false, restart count 0 Mar 11 16:35:33.819: INFO: pod-ephm-test-configmap-tzlh started at 2023-03-11 16:35:06 +0000 UTC (0+1 container statuses recorded) Mar 11 16:35:33.819: INFO: Container test-container-subpath-configmap-tzlh ready: false, restart count 0 Mar 11 16:35:33.819: INFO: coredns-autoscaler-8495f5575-d8gqb started at 2023-03-11 16:26:12 +0000 UTC (0+1 container statuses recorded) Mar 11 16:35:33.819: INFO: Container autoscaler ready: true, restart count 0 Mar 11 16:35:34.531: INFO: Latency metrics for node i-04523d1dffb0fe725 Mar 11 16:35:34.531: INFO: Logging node info for node i-07e2b6e650d7ef56d Mar 11 16:35:34.682: INFO: Node Info: &Node{ObjectMeta:{i-07e2b6e650d7ef56d fcc65272-6f1b-4228-ac59-e9e3e92cdb2c 22319 0 2023-03-11 16:24:19 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:arm64 beta.kubernetes.io/instance-type:t4g.small beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-northeast-1 failure-domain.beta.kubernetes.io/zone:ap-northeast-1a karpenter.k8s.aws/instance-category:t karpenter.k8s.aws/instance-cpu:2 karpenter.k8s.aws/instance-family:t4g karpenter.k8s.aws/instance-generation:4 karpenter.k8s.aws/instance-hypervisor:nitro karpenter.k8s.aws/instance-memory:2048 karpenter.k8s.aws/instance-pods:11 karpenter.k8s.aws/instance-size:small karpenter.sh/capacity-type:spot karpenter.sh/initialized:true karpenter.sh/provisioner-name:nodes kubernetes.io/arch:arm64 kubernetes.io/hostname:i-07e2b6e650d7ef56d kubernetes.io/os:linux node-role.kubernetes.io/node: node-role.kubernetes.io/spot-worker:true node.kubernetes.io/instance-type:t4g.small topology.ebs.csi.aws.com/zone:ap-northeast-1a topology.hostpath.csi/node:i-07e2b6e650d7ef56d topology.kubernetes.io/region:ap-northeast-1 topology.kubernetes.io/zone:ap-northeast-1a] map[csi.volume.kubernetes.io/nodeid:{"csi-hostpath-ephemeral-7431":"i-07e2b6e650d7ef56d","csi-mock-csi-mock-volumes-expansion-6425":"csi-mock-csi-mock-volumes-expansion-6425","csi-mock-csi-mock-volumes-fsgroup-policy-9457":"i-07e2b6e650d7ef56d","ebs.csi.aws.com":"i-07e2b6e650d7ef56d"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [{karpenter.sh/v1alpha5 Provisioner nodes ac7283b2-eed0-4a9f-9ddd-616f33a0991a <nil> 0xc003916525}] [karpenter.sh/termination] [{aws-cloud-controller-manager Update v1 2023-03-11 16:24:20 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{}}}} } {kops-controller Update v1 2023-03-11 16:24:20 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{},"f:node-role.kubernetes.io/spot-worker":{}}}} } {karpenter Update v1 2023-03-11 16:26:15 +0000 UTC FieldsV1 {"f:metadata":{"f:finalizers":{".":{},"v:\"karpenter.sh/termination\"":{}},"f:labels":{".":{},"f:karpenter.k8s.aws/instance-category":{},"f:karpenter.k8s.aws/instance-cpu":{},"f:karpenter.k8s.aws/instance-family":{},"f:karpenter.k8s.aws/instance-generation":{},"f:karpenter.k8s.aws/instance-hypervisor":{},"f:karpenter.k8s.aws/instance-memory":{},"f:karpenter.k8s.aws/instance-pods":{},"f:karpenter.k8s.aws/instance-size":{},"f:karpenter.sh/capacity-type":{},"f:karpenter.sh/initialized":{},"f:karpenter.sh/provisioner-name":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}},"f:ownerReferences":{".":{},"k:{\"uid\":\"ac7283b2-eed0-4a9f-9ddd-616f33a0991a\"}":{}}},"f:spec":{"f:providerID":{}}} } {kube-controller-manager Update v1 2023-03-11 16:26:15 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:node.alpha.kubernetes.io/ttl":{}},"f:labels":{"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.1.0/24\"":{}}}} } {aws-cloud-controller-manager Update v1 2023-03-11 16:26:50 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{".":{},"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"InternalIP\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kube-controller-manager Update v1 2023-03-11 16:35:07 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{".":{},"k:{\"type\":\"DiskPressure\"}":{".":{},"f:type":{}},"k:{\"type\":\"MemoryPressure\"}":{".":{},"f:type":{}},"k:{\"type\":\"PIDPressure\"}":{".":{},"f:type":{}},"k:{\"type\":\"Ready\"}":{".":{},"f:type":{}}},"f:volumesAttached":{}}} status} {kubelet Update v1 2023-03-11 16:35:16 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{"f:kubernetes.io/hostname":{},"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{".":{},"f:cpu":{},"f:ephemeral-storage":{},"f:hugepages-1Gi":{},"f:hugepages-2Mi":{},"f:hugepages-32Mi":{},"f:hugepages-64Ki":{},"f:memory":{},"f:pods":{}},"f:capacity":{".":{},"f:cpu":{},"f:ephemeral-storage":{},"f:hugepages-1Gi":{},"f:hugepages-2Mi":{},"f:hugepages-32Mi":{},"f:hugepages-64Ki":{},"f:memory":{},"f:pods":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:daemonEndpoints":{"f:kubeletEndpoint":{"f:Port":{}}},"f:images":{},"f:nodeInfo":{"f:architecture":{},"f:bootID":{},"f:containerRuntimeVersion":{},"f:kernelVersion":{},"f:kubeProxyVersion":{},"f:kubeletVersion":{},"f:machineID":{},"f:operatingSystem":{},"f:osImage":{},"f:systemUUID":{}},"f:volumesInUse":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.1.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-northeast-1a/i-07e2b6e650d7ef56d,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.1.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49756430336 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-32Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-64Ki: {{0 0} {<nil>} 0 DecimalSI},memory: {{1941549056 0} {<nil>} 1896044Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44780787229 0} {<nil>} 44780787229 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-32Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-64Ki: {{0 0} {<nil>} 0 DecimalSI},memory: {{1836691456 0} {<nil>} 1793644Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-03-11 16:35:15 +0000 UTC,LastTransitionTime:2023-03-11 16:26:15 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-03-11 16:35:15 +0000 UTC,LastTransitionTime:2023-03-11 16:26:05 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-03-11 16:35:15 +0000 UTC,LastTransitionTime:2023-03-11 16:26:05 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-03-11 16:35:15 +0000 UTC,LastTransitionTime:2023-03-11 16:26:05 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.35.76,},NodeAddress{Type:ExternalIP,Address:54.238.106.83,},NodeAddress{Type:InternalDNS,Address:i-07e2b6e650d7ef56d.ap-northeast-1.compute.internal,},NodeAddress{Type:Hostname,Address:i-07e2b6e650d7ef56d.ap-northeast-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-54-238-106-83.ap-northeast-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2083a8e9d27d02fac967340367f875,SystemUUID:ec2083a8-e9d2-7d02-fac9-67340367f875,BootID:ea264ed6-5b0d-47d6-ad5d-6bd2a1f7c11b,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 22.04.2 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.27.0-alpha.3.420+bb6c6fad2c19b3,KubeProxyVersion:v1.27.0-alpha.3.420+bb6c6fad2c19b3,OperatingSystem:linux,Architecture:arm64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:157636062,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/jessie-dnsutils@sha256:24aaf2626d6b27864c29de2097e8bbb840b3a414271bf7c8995e431e47d8408e registry.k8s.io/e2e-test-images/jessie-dnsutils:1.7],SizeBytes:116764683,},ContainerImage{Names:[registry.k8s.io/kube-proxy-arm64:v1.27.0-alpha.3.420_bb6c6fad2c19b3],SizeBytes:60947298,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:16bbf38c463a4223d8cfe4da12bc61010b082a79b4bb003e2d3ba3ece5dd5f9e registry.k8s.io/e2e-test-images/agnhost:2.43],SizeBytes:49808364,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nautilus@sha256:80ba6c8c44f9623f06e868a1aa66026c8ec438ad814f9ec95e9333b415fe3550 registry.k8s.io/e2e-test-images/nautilus:1.7],SizeBytes:47953400,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:3fe7acf013d1264ffded116b80a73dc129a449b0fccdb8d21af8279f2233f36e registry.k8s.io/e2e-test-images/httpd:2.4.39-4],SizeBytes:41528818,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:148b022f5c5da426fc2f3c14b5c0867e58ef05961510c84749ac1fddcb0fef22 registry.k8s.io/e2e-test-images/httpd:2.4.38-4],SizeBytes:40343601,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:28932904,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:ee3b525d5b89db99da3b8eb521d9cd90cb6e9ef0fbb651e98bb37be78d36b5b8 registry.k8s.io/sig-storage/csi-provisioner:v3.3.0],SizeBytes:23945474,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:425d8f1b769398127767b06ed97ce62578a3179bcb99809ce93a1649e025ffe7 registry.k8s.io/sig-storage/csi-resizer:v1.6.0],SizeBytes:22678632,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:291334908ddf71a4661fd7f6d9d97274de8a5378a2b6fdfeb2ce73414a34f82f registry.k8s.io/sig-storage/csi-snapshotter:v6.1.0],SizeBytes:22434180,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:9a685020911e2725ad019dbce6e4a5ab93d51e3d4557f115e64343345e05781b registry.k8s.io/sig-storage/csi-attacher:v4.0.0],SizeBytes:22394501,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:122bfb8c1edabb3c0edd63f06523e6940d958d19b3957dc7b1d6f81e9f1f6119 registry.k8s.io/sig-storage/csi-provisioner:v3.1.0],SizeBytes:21735644,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:9ebbf9f023e7b41ccee3d52afe39a89e3ddacdbb69269d583abfc25847cfd9e4 registry.k8s.io/sig-storage/csi-resizer:v1.4.0],SizeBytes:20848413,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:8b9c313c05f54fb04f8d430896f5f5904b6cb157df261501b29adc04d2b2dc7b registry.k8s.io/sig-storage/csi-attacher:v3.4.0],SizeBytes:20565480,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nonroot@sha256:ee9f50b3c64b174d296d91ca9f69a914ac30e59095dfb462b2b518ad28a63655 registry.k8s.io/e2e-test-images/nonroot:1.4],SizeBytes:18561972,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:92257881c1d6493cf18299a24af42330f891166560047902b8d431fb66b01af5 registry.k8s.io/sig-storage/hostpathplugin:v1.9.0],SizeBytes:18267611,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:9b2d9abda017c28b12f84a344f57cd73fbdb6c2bd7dd5bdd5018246ad1093ba6 registry.k8s.io/sig-storage/hostpathplugin:v1.11.0],SizeBytes:18159085,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:017727efcfeb7d053af68e51436ce8e65edbc6ca573720afb4f79c8594036955 registry.k8s.io/coredns/coredns:v1.10.0],SizeBytes:13766234,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:8502345,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:933940f13b3ea0abc62e656c1aa5c5b47c04b15d71250413a6b821bd0c58b94e registry.k8s.io/sig-storage/livenessprobe:v2.7.0],SizeBytes:8223819,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:7685305,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:db048754ae68ae337d8fa96494c96d2a1204c3320f5dcf7e8e71085adec85da6 registry.k8s.io/e2e-test-images/nginx:1.15-4],SizeBytes:6883347,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:5c99cf6a02adda929b10321dbf4ecfa00d87be9ba4fb456006237d530ab4baa1 registry.k8s.io/e2e-test-images/nginx:1.14-4],SizeBytes:6860089,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nonewprivs@sha256:8ac1264691820febacf3aea5d152cbde6d10685731ec14966a9401c6f47a68ac registry.k8s.io/e2e-test-images/nonewprivs:1.3],SizeBytes:5025027,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf registry.k8s.io/e2e-test-images/busybox:1.29-2],SizeBytes:773367,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:2e0f836850e09b8b7cc937681d6194537a09fbd5f6b9e08f4d646a85128e8937 registry.k8s.io/e2e-test-images/busybox:1.29-4],SizeBytes:772933,},ContainerImage{Names:[registry.k8s.io/pause@sha256:7031c1b283388d2c2e09b57badb803c05ebed362dc88d84b480cc47f72a21097 registry.k8s.io/pause:3.9],SizeBytes:268051,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:253553,},},VolumesInUse:[kubernetes.io/csi/csi-hostpath-ephemeral-7431^adfaaa1d-c02a-11ed-9622-6a5c3c829c29 kubernetes.io/csi/csi-mock-csi-mock-volumes-expansion-6425^4 kubernetes.io/csi/ebs.csi.aws.com^vol-0629a42d1a184a26f kubernetes.io/csi/ebs.csi.aws.com^vol-08bebe1064d14d4df],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/csi-mock-csi-mock-volumes-expansion-6425^4,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-08bebe1064d14d4df,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/csi-hostpath-ephemeral-7431^adfaaa1d-c02a-11ed-9622-6a5c3c829c29,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-0629a42d1a184a26f,DevicePath:,},},Config:nil,},} Mar 11 16:35:34.682: INFO: Logging kubelet events for node i-07e2b6e650d7ef56d Mar 11 16:35:34.834: INFO: Logging pods the kubelet thinks is on node i-07e2b6e650d7ef56d Mar 11 16:35:34.992: INFO: netserver-1 started at 2023-03-11 16:35:09 +0000 UTC (0+1 container statuses recorded) Mar 11 16:35:34.992: INFO: Container webserver ready: true, restart count 0 Mar 11 16:35:34.992: INFO: pod-37d913c8-128e-43bd-9425-b59ff4283eaa started at 2023-03-11 16:35:28 +0000 UTC (0+1 container statuses recorded) Mar 11 16:35:34.992: INFO: Container write-pod ready: false, restart count 0 Mar 11 16:35:34.992: INFO: csi-mockplugin-0 started at 2023-03-11 16:35:11 +0000 UTC (0+3 container statuses recorded) Mar 11 16:35:34.992: INFO: Container csi-provisioner ready: true, restart count 0 Mar 11 16:35:34.992: INFO: Container driver-registrar ready: true, restart count 0 Mar 11 16:35:34.992: INFO: Container mock ready: true, restart count 0 Mar 11 16:35:34.992: INFO: ss-1 started at 2023-03-11 16:35:25 +0000 UTC (0+1 container statuses recorded) Mar 11 16:35:34.992: INFO: Container webserver ready: true, restart count 0 Mar 11 16:35:34.992: INFO: pod-ephm-test-projected-6dkr started at 2023-03-11 16:33:33 +0000 UTC (0+1 container statuses recorded) Mar 11 16:35:34.992: INFO: Container test-container-subpath-projected-6dkr ready: false, restart count 0 Mar 11 16:35:34.992: INFO: csi-mockplugin-0 started at 2023-03-11 16:33:31 +0000 UTC (0+4 container statuses recorded) Mar 11 16:35:34.992: INFO: Container busybox ready: true, restart count 0 Mar 11 16:35:34.992: INFO: Container csi-provisioner ready: true, restart count 0 Mar 11 16:35:34.992: INFO: Container driver-registrar ready: true, restart count 0 Mar 11 16:35:34.992: INFO: Container mock ready: true, restart count 0 Mar 11 16:35:34.992: INFO: ss-0 started at 2023-03-11 16:35:28 +0000 UTC (0+1 container statuses recorded) Mar 11 16:35:34.992: INFO: Container webserver ready: false, restart count 0 Mar 11 16:35:34.992: INFO: ebs-csi-controller-79785c665b-lpmmf started at 2023-03-11 16:26:15 +0000 UTC (0+5 container statuses recorded) Mar 11 16:35:34.992: INFO: Container csi-attacher ready: true, restart count 0 Mar 11 16:35:34.992: INFO: Container csi-provisioner ready: true, restart count 0 Mar 11 16:35:34.992: INFO: Container csi-resizer ready: true, restart count 0 Mar 11 16:35:34.992: INFO: Container ebs-plugin ready: true, restart count 0 Mar 11 16:35:34.992: INFO: Container liveness-probe ready: true, restart count 0 Mar 11 16:35:34.992: INFO: ss-1 started at 2023-03-11 16:34:57 +0000 UTC (0+1 container statuses recorded) Mar 11 16:35:34.992: INFO: Container webserver ready: true, restart count 0 Mar 11 16:35:34.992: INFO: cilium-mvm8l started at 2023-03-11 16:26:06 +0000 UTC (1+1 container statuses recorded) Mar 11 16:35:34.992: INFO: Init container clean-cilium-state ready: true, restart count 0 Mar 11 16:35:34.992: INFO: Container cilium-agent ready: true, restart count 0 Mar 11 16:35:34.992: INFO: inline-volume-tester-nbrpx started at 2023-03-11 16:35:06 +0000 UTC (0+1 container statuses recorded) Mar 11 16:35:34.992: INFO: Container csi-volume-tester ready: true, restart count 0 Mar 11 16:35:34.992: INFO: ebs-csi-node-m5b9f started at 2023-03-11 16:26:06 +0000 UTC (0+3 container statuses recorded) Mar 11 16:35:34.992: INFO: Container ebs-plugin ready: true, restart count 0 Mar 11 16:35:34.992: INFO: Container liveness-probe ready: true, restart count 0 Mar 11 16:35:34.992: INFO: Container node-driver-registrar ready: true, restart count 0 Mar 11 16:35:34.992: INFO: pause-pod-0 started at 2023-03-11 16:33:15 +0000 UTC (0+1 container statuses recorded) Mar 11 16:35:34.992: INFO: Container agnhost-container ready: true, restart count 0 Mar 11 16:35:34.992: INFO: csi-mockplugin-attacher-0 started at 2023-03-11 16:33:31 +0000 UTC (0+1 container statuses recorded) Mar 11 16:35:34.992: INFO: Container csi-attacher ready: true, restart count 0 Mar 11 16:35:34.992: INFO: pvc-volume-tester-hngnv started at 2023-03-11 16:33:44 +0000 UTC (0+1 container statuses recorded) Mar 11 16:35:34.992: INFO: Container volume-tester ready: true, restart count 0 Mar 11 16:35:34.992: INFO: pause-pod-1 started at 2023-03-11 16:35:12 +0000 UTC (0+1 container statuses recorded) Mar 11 16:35:34.992: INFO: Container agnhost-container ready: true, restart count 0 Mar 11 16:35:34.992: INFO: liveness-0a3782e9-b416-48f3-a764-4a0383907484 started at 2023-03-11 16:32:41 +0000 UTC (0+1 container statuses recorded) Mar 11 16:35:34.992: INFO: Container agnhost-container ready: true, restart count 0 Mar 11 16:35:34.992: INFO: csi-hostpathplugin-0 started at 2023-03-11 16:35:04 +0000 UTC (0+7 container statuses recorded) Mar 11 16:35:34.992: INFO: Container csi-attacher ready: true, restart count 0 Mar 11 16:35:34.992: INFO: Container csi-provisioner ready: true, restart count 0 Mar 11 16:35:34.992: INFO: Container csi-resizer ready: true, restart count 0 Mar 11 16:35:34.992: INFO: Container csi-snapshotter ready: true, restart count 0 Mar 11 16:35:34.992: INFO: Container hostpath ready: true, restart count 0 Mar 11 16:35:34.992: INFO: Container liveness-probe ready: true, restart count 0 Mar 11 16:35:34.992: INFO: Container node-driver-registrar ready: true, restart count 0 Mar 11 16:35:34.992: INFO: pvc-volume-tester-955ss started at 2023-03-11 16:35:19 +0000 UTC (0+1 container statuses recorded) Mar 11 16:35:34.992: INFO: Container volume-tester ready: true, restart count 0 Mar 11 16:35:34.992: INFO: coredns-745bddcfbb-lkbct started at 2023-03-11 16:26:41 +0000 UTC (0+1 container statuses recorded) Mar 11 16:35:34.992: INFO: Container coredns ready: true, restart count 0 Mar 11 16:35:35.594: INFO: Latency metrics for node i-07e2b6e650d7ef56d Mar 11 16:35:35.594: INFO: Logging node info for node i-0e3467ec552af797e Mar 11 16:35:35.744: INFO: Node Info: &Node{ObjectMeta:{i-0e3467ec552af797e 6c9b0cc5-2c4e-4037-bf31-61bac12a7775 17626 0 2023-03-11 16:20:46 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:arm64 beta.kubernetes.io/instance-type:c6g.xlarge beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-northeast-1 failure-domain.beta.kubernetes.io/zone:ap-northeast-1a kops.k8s.io/kops-controller-pki: kubernetes.io/arch:arm64 kubernetes.io/hostname:i-0e3467ec552af797e kubernetes.io/os:linux node-role.kubernetes.io/control-plane: node.kubernetes.io/exclude-from-external-load-balancers: node.kubernetes.io/instance-type:c6g.xlarge topology.ebs.csi.aws.com/zone:ap-northeast-1a topology.kubernetes.io/region:ap-northeast-1 topology.kubernetes.io/zone:ap-northeast-1a] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-0e3467ec552af797e"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kubelet Update v1 2023-03-11 16:20:46 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {protokube Update v1 2023-03-11 16:21:30 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kops.k8s.io/kops-controller-pki":{},"f:node-role.kubernetes.io/control-plane":{},"f:node.kubernetes.io/exclude-from-external-load-balancers":{}}}} } {aws-cloud-controller-manager Update v1 2023-03-11 16:21:50 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-03-11 16:21:50 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kube-controller-manager Update v1 2023-03-11 16:21:58 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.0.0/24\"":{}},"f:taints":{}}} } {kubelet Update v1 2023-03-11 16:33:33 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.0.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-northeast-1a/i-0e3467ec552af797e,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/control-plane,Value:,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[100.96.0.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{4 0} {<nil>} 4 DecimalSI},ephemeral-storage: {{49756430336 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-32Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-64Ki: {{0 0} {<nil>} 0 DecimalSI},memory: {{8150110208 0} {<nil>} 7959092Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{4 0} {<nil>} 4 DecimalSI},ephemeral-storage: {{44780787229 0} {<nil>} 44780787229 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-32Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-64Ki: {{0 0} {<nil>} 0 DecimalSI},memory: {{8045252608 0} {<nil>} 7856692Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-03-11 16:33:33 +0000 UTC,LastTransitionTime:2023-03-11 16:20:46 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-03-11 16:33:33 +0000 UTC,LastTransitionTime:2023-03-11 16:20:46 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-03-11 16:33:33 +0000 UTC,LastTransitionTime:2023-03-11 16:20:46 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-03-11 16:33:33 +0000 UTC,LastTransitionTime:2023-03-11 16:21:58 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.40.18,},NodeAddress{Type:ExternalIP,Address:52.194.213.7,},NodeAddress{Type:InternalDNS,Address:i-0e3467ec552af797e.ap-northeast-1.compute.internal,},NodeAddress{Type:Hostname,Address:i-0e3467ec552af797e.ap-northeast-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-52-194-213-7.ap-northeast-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2abf2e564965592c7716accd56507e,SystemUUID:ec2abf2e-5649-6559-2c77-16accd56507e,BootID:fbada3ff-8655-404c-a98d-876577d4c322,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 22.04.2 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.27.0-alpha.3.420+bb6c6fad2c19b3,KubeProxyVersion:v1.27.0-alpha.3.420+bb6c6fad2c19b3,OperatingSystem:linux,Architecture:arm64,},Images:[]ContainerImage{ContainerImage{Names:[registry.k8s.io/etcd@sha256:9ce33ba33d8e738a5b85ed50b5080ac746deceed4a7496c550927a7a19ca3b6d registry.k8s.io/etcd:3.5.0-0],SizeBytes:157797353,},ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:157636062,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:4afb99b4690b418ffc2ceb67e1a17376457e441c1f09ab55447f0aaf992fa646 registry.k8s.io/etcd:3.4.3-0],SizeBytes:151677080,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:4ad90a11b55313b182afc186b9876c8e891531b8db4c9bf1541953021618d0e2 registry.k8s.io/etcd:3.4.13-0],SizeBytes:134531559,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:0d2ab6b15b4fe0b903b9e81192d942fb39c29be61fe341cd16db5d0165b0435c registry.k8s.io/etcd:3.3.17-0],SizeBytes:126979107,},ContainerImage{Names:[registry.k8s.io/kube-apiserver-arm64:v1.27.0-alpha.3.420_bb6c6fad2c19b3],SizeBytes:115123555,},ContainerImage{Names:[registry.k8s.io/kube-controller-manager-arm64:v1.27.0-alpha.3.420_bb6c6fad2c19b3],SizeBytes:107914235,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:17da501f5d2a675be46040422a27b7cc21b8a43895ac998b171db1c346f361f7 registry.k8s.io/etcd:3.3.10-0],SizeBytes:81436097,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:6f72b851544986cb0921b53ea655ec04c36131248f16d4ad110cb3ca0c369dc1 registry.k8s.io/etcd:3.5.4-0],SizeBytes:81120118,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:13f53ed1d91e2e11aac476ee9a0269fdda6cc4874eba903efd40daf50c55eee5 registry.k8s.io/etcd:3.5.3-0],SizeBytes:81114891,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:51eae8381dcb1078289fa7b4f3df2630cdc18d09fb56f8e56b41c40e191d6c83 registry.k8s.io/etcd:3.5.7-0],SizeBytes:80665728,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:dd75ec974b0a2a6f6bb47001ba09207976e625db898d1b16735528c009cb171c registry.k8s.io/etcd:3.5.6-0],SizeBytes:80539316,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:905d7ca17fd02bc24c0eba9a062753aba15db3e31422390bc3238eb762339b20 registry.k8s.io/etcd:3.2.24-1],SizeBytes:61500433,},ContainerImage{Names:[registry.k8s.io/kube-proxy-arm64:v1.27.0-alpha.3.420_bb6c6fad2c19b3],SizeBytes:60947298,},ContainerImage{Names:[registry.k8s.io/etcdadm/etcd-manager-slim@sha256:8727adc059334e0e3329dbccce116a734da4a090da5e86c8cddccaf333caa666 registry.k8s.io/etcdadm/etcd-manager-slim:v3.0.20230201],SizeBytes:60826655,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:64b9ea357325d5db9f8a723dcf503b5a449177b17ac87d69481e126bb724c263 registry.k8s.io/etcd:3.5.1-0],SizeBytes:59553904,},ContainerImage{Names:[registry.k8s.io/kube-scheduler-arm64:v1.27.0-alpha.3.420_bb6c6fad2c19b3],SizeBytes:56731120,},ContainerImage{Names:[registry.k8s.io/kops/kops-controller:1.27.0-alpha.1],SizeBytes:41885191,},ContainerImage{Names:[registry.k8s.io/kops/dns-controller:1.27.0-alpha.1],SizeBytes:40376038,},ContainerImage{Names:[public.ecr.aws/karpenter/controller@sha256:68db4f092cf9cc83f5ef9e2fbc5407c2cb682e81f64dfaa700a7602ede38b1cf public.ecr.aws/karpenter/controller:v0.16.3],SizeBytes:33707751,},ContainerImage{Names:[public.ecr.aws/karpenter/webhook@sha256:96a2d9b06d6bc5127801f358f74b1cf2d289b423a2e9ba40c573c0b14b17dafa public.ecr.aws/karpenter/webhook:v0.16.3],SizeBytes:33307187,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:28932904,},ContainerImage{Names:[quay.io/cilium/operator@sha256:a6d24a006a6b92967ac90786b49bc1ac26e5477cf028cd1186efcfc2466484db quay.io/cilium/operator:v1.12.5],SizeBytes:24567287,},ContainerImage{Names:[registry.k8s.io/provider-aws/cloud-controller-manager@sha256:fdeb61e3e42ecd9cca868d550ebdb88dd6341d9e91fcfa9a37e227dab2ad22cb registry.k8s.io/provider-aws/cloud-controller-manager:v1.26.0],SizeBytes:18345226,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:8502345,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:7685305,},ContainerImage{Names:[registry.k8s.io/kops/kube-apiserver-healthcheck:1.27.0-alpha.1],SizeBytes:4740762,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:253553,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Mar 11 16:35:35.744: INFO: Logging kubelet events for node i-0e3467ec552af797e Mar 11 16:35:35.895: INFO: Logging pods the kubelet thinks is on node i-0e3467ec552af797e Mar 11 16:35:36.057: INFO: kube-controller-manager-i-0e3467ec552af797e started at 2023-03-11 16:18:44 +0000 UTC (0+1 container statuses recorded) Mar 11 16:35:36.057: INFO: Container kube-controller-manager ready: true, restart count 4 Mar 11 16:35:36.057: INFO: etcd-manager-events-i-0e3467ec552af797e started at 2023-03-11 16:18:44 +0000 UTC (11+1 container statuses recorded) Mar 11 16:35:36.057: INFO: Init container init-etcd-3-2-24 ready: true, restart count 0 Mar 11 16:35:36.057: INFO: Init container init-etcd-3-3-10 ready: true, restart count 0 Mar 11 16:35:36.057: INFO: Init container init-etcd-3-3-17 ready: true, restart count 0 Mar 11 16:35:36.057: INFO: Init container init-etcd-3-4-13 ready: true, restart count 0 Mar 11 16:35:36.057: INFO: Init container init-etcd-3-4-3 ready: true, restart count 0 Mar 11 16:35:36.057: INFO: Init container init-etcd-3-5-0 ready: true, restart count 0 Mar 11 16:35:36.057: INFO: Init container init-etcd-3-5-1 ready: true, restart count 0 Mar 11 16:35:36.057: INFO: Init container init-etcd-3-5-3 ready: true, restart count 0 Mar 11 16:35:36.057: INFO: Init container init-etcd-3-5-4 ready: true, restart count 0 Mar 11 16:35:36.057: INFO: Init container init-etcd-3-5-6 ready: true, restart count 0 Mar 11 16:35:36.057: INFO: Init container init-etcd-3-5-7 ready: true, restart count 0 Mar 11 16:35:36.057: INFO: Container etcd-manager ready: true, restart count 0 Mar 11 16:35:36.057: INFO: etcd-manager-main-i-0e3467ec552af797e started at 2023-03-11 16:18:44 +0000 UTC (11+1 container statuses recorded) Mar 11 16:35:36.057: INFO: Init container init-etcd-3-2-24 ready: true, restart count 0 Mar 11 16:35:36.057: INFO: Init container init-etcd-3-3-10 ready: true, restart count 0 Mar 11 16:35:36.057: INFO: Init container init-etcd-3-3-17 ready: true, restart count 0 Mar 11 16:35:36.057: INFO: Init container init-etcd-3-4-13 ready: true, restart count 0 Mar 11 16:35:36.057: INFO: Init container init-etcd-3-4-3 ready: true, restart count 0 Mar 11 16:35:36.057: INFO: Init container init-etcd-3-5-0 ready: true, restart count 0 Mar 11 16:35:36.057: INFO: Init container init-etcd-3-5-1 ready: true, restart count 0 Mar 11 16:35:36.057: INFO: Init container init-etcd-3-5-3 ready: true, restart count 0 Mar 11 16:35:36.057: INFO: Init container init-etcd-3-5-4 ready: true, restart count 0 Mar 11 16:35:36.057: INFO: Init container init-etcd-3-5-6 ready: true, restart count 0 Mar 11 16:35:36.057: INFO: Init container init-etcd-3-5-7 ready: true, restart count 0 Mar 11 16:35:36.057: INFO: Container etcd-manager ready: true, restart count 0 Mar 11 16:35:36.057: INFO: ebs-csi-node-lpwz9 started at 2023-03-11 16:21:37 +0000 UTC (0+3 container statuses recorded) Mar 11 16:35:36.057: INFO: Container ebs-plugin ready: true, restart count 0 Mar 11 16:35:36.057: INFO: Container liveness-probe ready: true, restart count 0 Mar 11 16:35:36.057: INFO: Container node-driver-registrar ready: true, restart count 0 Mar 11 16:35:36.057: INFO: karpenter-webhook-5b646cd4f7-6sg2k started at 2023-03-11 16:21:58 +0000 UTC (0+1 container statuses recorded) Mar 11 16:35:36.057: INFO: Container webhook ready: true, restart count 0 Mar 11 16:35:36.057: INFO: karpenter-controller-567749677d-d9szs started at 2023-03-11 16:21:58 +0000 UTC (0+1 container statuses recorded) Mar 11 16:35:36.057: INFO: Container controller ready: true, restart count 0 Mar 11 16:35:36.057: INFO: kube-apiserver-i-0e3467ec552af797e started at 2023-03-11 16:18:44 +0000 UTC (0+2 container statuses recorded) Mar 11 16:35:36.057: INFO: Container healthcheck ready: true, restart count 0 Mar 11 16:35:36.057: INFO: Container kube-apiserver ready: true, restart count 3 Mar 11 16:35:36.057: INFO: kube-scheduler-i-0e3467ec552af797e started at 2023-03-11 16:18:44 +0000 UTC (0+1 container statuses recorded) Mar 11 16:35:36.057: INFO: Container kube-scheduler ready: true, restart count 0 Mar 11 16:35:36.057: INFO: dns-controller-7dfd64d999-q9hh7 started at 2023-03-11 16:21:37 +0000 UTC (0+1 container statuses recorded) Mar 11 16:35:36.057: INFO: Container dns-controller ready: true, restart count 0 Mar 11 16:35:36.057: INFO: cilium-qrsfn started at 2023-03-11 16:21:37 +0000 UTC (1+1 container statuses recorded) Mar 11 16:35:36.057: INFO: Init container clean-cilium-state ready: true, restart count 0 Mar 11 16:35:36.057: INFO: Container cilium-agent ready: true, restart count 0 Mar 11 16:35:36.057: INFO: cilium-operator-8496cc65f5-qhdg8 started at 2023-03-11 16:21:37 +0000 UTC (0+1 container statuses recorded) Mar 11 16:35:36.057: INFO: Container cilium-operator ready: true, restart count 0 Mar 11 16:35:36.057: INFO: kops-controller-76gkw started at 2023-03-11 16:21:37 +0000 UTC (0+1 container statuses recorded) Mar 11 16:35:36.057: INFO: Container kops-controller ready: true, restart count 0 Mar 11 16:35:36.057: INFO: aws-cloud-controller-manager-kwbw2 started at 2023-03-11 16:21:37 +0000 UTC (0+1 container statuses recorded) Mar 11 16:35:36.057: INFO: Container aws-cloud-controller-manager ready: true, restart count 0 Mar 11 16:35:36.572: INFO: Latency metrics for node i-0e3467ec552af797e END STEP: dump namespace information after failure - test/e2e/framework/framework.go:288 @ 03/11/23 16:35:36.572 (4.13s) < Exit [DeferCleanup (Each)] [sig-network] EndpointSlice - dump namespaces | framework.go:209 @ 03/11/23 16:35:36.572 (4.13s) > Enter [DeferCleanup (Each)] [sig-network] EndpointSlice - tear down framework | framework.go:206 @ 03/11/23 16:35:36.572 STEP: Destroying namespace "endpointslice-2030" for this suite. - test/e2e/framework/framework.go:351 @ 03/11/23 16:35:36.572 < Exit [DeferCleanup (Each)] [sig-network] EndpointSlice - tear down framework | framework.go:206 @ 03/11/23 16:35:36.724 (152ms) > Enter [ReportAfterEach] TOP-LEVEL - test/e2e/e2e_test.go:144 @ 03/11/23 16:35:36.725 < Exit [ReportAfterEach] TOP-LEVEL - test/e2e/e2e_test.go:144 @ 03/11/23 16:35:36.725 (0s)
Filter through log files | View test history on testgrid
go run hack/e2e.go -v --test --test_args='--ginkgo.focus=Kubernetes\se2e\ssuite\s\[It\]\s\[sig\-network\]\sServices\sshould\sbe\sable\sto\sswitch\ssession\saffinity\sfor\sNodePort\sservice\s\[LinuxOnly\]\s\[Conformance\]$'
[FAILED] failed to create replication controller with service in the namespace: services-6839: only 0 pods started out of 3 In [It] at: test/e2e/network/service.go:3967 @ 03/11/23 16:49:30.095from junit_01.xml
> Enter [BeforeEach] [sig-network] Services - set up framework | framework.go:191 @ 03/11/23 16:44:25.372 STEP: Creating a kubernetes client - test/e2e/framework/framework.go:211 @ 03/11/23 16:44:25.372 Mar 11 16:44:25.372: INFO: >>> kubeConfig: /root/.kube/config STEP: Building a namespace api object, basename services - test/e2e/framework/framework.go:250 @ 03/11/23 16:44:25.373 STEP: Waiting for a default service account to be provisioned in namespace - test/e2e/framework/framework.go:259 @ 03/11/23 16:44:25.821 STEP: Waiting for kube-root-ca.crt to be provisioned in namespace - test/e2e/framework/framework.go:262 @ 03/11/23 16:44:26.118 < Exit [BeforeEach] [sig-network] Services - set up framework | framework.go:191 @ 03/11/23 16:44:26.415 (1.043s) > Enter [BeforeEach] [sig-network] Services - test/e2e/framework/metrics/init/init.go:33 @ 03/11/23 16:44:26.415 < Exit [BeforeEach] [sig-network] Services - test/e2e/framework/metrics/init/init.go:33 @ 03/11/23 16:44:26.415 (0s) > Enter [BeforeEach] [sig-network] Services - test/e2e/network/service.go:764 @ 03/11/23 16:44:26.415 < Exit [BeforeEach] [sig-network] Services - test/e2e/network/service.go:764 @ 03/11/23 16:44:26.415 (0s) > Enter [It] should be able to switch session affinity for NodePort service [LinuxOnly] [Conformance] - test/e2e/network/service.go:2224 @ 03/11/23 16:44:26.415 STEP: creating service in namespace services-6839 - test/e2e/network/service.go:3963 @ 03/11/23 16:44:26.415 STEP: creating service affinity-nodeport-transition in namespace services-6839 - test/e2e/network/service.go:270 @ 03/11/23 16:44:26.415 STEP: creating replication controller affinity-nodeport-transition in namespace services-6839 - test/e2e/framework/rc/rc_utils.go:88 @ 03/11/23 16:44:26.572 I0311 16:44:26.723441 7237 runners.go:194] Created replication controller with name: affinity-nodeport-transition, namespace: services-6839, replica count: 3 I0311 16:44:29.874183 7237 runners.go:194] affinity-nodeport-transition Pods: 3 out of 3 created, 0 running, 3 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0311 16:44:32.875364 7237 runners.go:194] affinity-nodeport-transition Pods: 3 out of 3 created, 0 running, 3 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0311 16:44:35.875978 7237 runners.go:194] affinity-nodeport-transition Pods: 3 out of 3 created, 0 running, 3 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0311 16:44:38.876918 7237 runners.go:194] affinity-nodeport-transition Pods: 3 out of 3 created, 0 running, 3 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0311 16:44:41.877004 7237 runners.go:194] affinity-nodeport-transition Pods: 3 out of 3 created, 0 running, 3 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0311 16:44:44.877155 7237 runners.go:194] affinity-nodeport-transition Pods: 3 out of 3 created, 0 running, 3 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0311 16:44:47.878276 7237 runners.go:194] affinity-nodeport-transition Pods: 3 out of 3 created, 0 running, 3 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0311 16:44:50.878929 7237 runners.go:194] affinity-nodeport-transition Pods: 3 out of 3 created, 0 running, 3 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0311 16:44:53.879696 7237 runners.go:194] affinity-nodeport-transition Pods: 3 out of 3 created, 0 running, 3 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0311 16:44:56.880357 7237 runners.go:194] affinity-nodeport-transition Pods: 3 out of 3 created, 0 running, 3 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0311 16:44:59.881389 7237 runners.go:194] affinity-nodeport-transition Pods: 3 out of 3 created, 0 running, 3 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0311 16:45:02.882339 7237 runners.go:194] affinity-nodeport-transition Pods: 3 out of 3 created, 0 running, 3 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0311 16:45:05.883302 7237 runners.go:194] affinity-nodeport-transition Pods: 3 out of 3 created, 0 running, 3 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0311 16:45:08.883659 7237 runners.go:194] affinity-nodeport-transition Pods: 3 out of 3 created, 0 running, 3 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0311 16:45:11.884232 7237 runners.go:194] affinity-nodeport-transition Pods: 3 out of 3 created, 0 running, 3 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0311 16:45:14.884790 7237 runners.go:194] affinity-nodeport-transition Pods: 3 out of 3 created, 0 running, 3 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0311 16:45:17.885014 7237 runners.go:194] affinity-nodeport-transition Pods: 3 out of 3 created, 0 running, 3 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0311 16:45:20.885351 7237 runners.go:194] affinity-nodeport-transition Pods: 3 out of 3 created, 0 running, 3 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0311 16:45:23.885594 7237 runners.go:194] affinity-nodeport-transition Pods: 3 out of 3 created, 0 running, 3 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0311 16:45:26.886394 7237 runners.go:194] affinity-nodeport-transition Pods: 3 out of 3 created, 0 running, 3 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0311 16:45:29.887309 7237 runners.go:194] affinity-nodeport-transition Pods: 3 out of 3 created, 0 running, 3 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0311 16:45:32.888210 7237 runners.go:194] affinity-nodeport-transition Pods: 3 out of 3 created, 0 running, 3 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0311 16:45:35.889243 7237 runners.go:194] affinity-nodeport-transition Pods: 3 out of 3 created, 0 running, 3 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0311 16:45:38.889601 7237 runners.go:194] affinity-nodeport-transition Pods: 3 out of 3 created, 0 running, 3 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0311 16:45:41.890232 7237 runners.go:194] affinity-nodeport-transition Pods: 3 out of 3 created, 0 running, 3 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0311 16:45:44.891185 7237 runners.go:194] affinity-nodeport-transition Pods: 3 out of 3 created, 0 running, 3 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0311 16:45:47.892359 7237 runners.go:194] affinity-nodeport-transition Pods: 3 out of 3 created, 0 running, 3 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0311 16:45:50.893246 7237 runners.go:194] affinity-nodeport-transition Pods: 3 out of 3 created, 0 running, 3 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0311 16:45:53.893846 7237 runners.go:194] affinity-nodeport-transition Pods: 3 out of 3 created, 0 running, 3 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0311 16:45:56.894101 7237 runners.go:194] affinity-nodeport-transition Pods: 3 out of 3 created, 0 running, 3 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0311 16:45:59.894309 7237 runners.go:194] affinity-nodeport-transition Pods: 3 out of 3 created, 0 running, 3 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0311 16:46:02.895436 7237 runners.go:194] affinity-nodeport-transition Pods: 3 out of 3 created, 0 running, 3 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0311 16:46:05.896200 7237 runners.go:194] affinity-nodeport-transition Pods: 3 out of 3 created, 0 running, 3 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0311 16:46:08.896388 7237 runners.go:194] affinity-nodeport-transition Pods: 3 out of 3 created, 0 running, 3 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0311 16:46:11.897382 7237 runners.go:194] affinity-nodeport-transition Pods: 3 out of 3 created, 0 running, 3 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0311 16:46:14.898284 7237 runners.go:194] affinity-nodeport-transition Pods: 3 out of 3 created, 0 running, 3 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0311 16:46:17.899376 7237 runners.go:194] affinity-nodeport-transition Pods: 3 out of 3 created, 0 running, 3 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0311 16:46:20.900051 7237 runners.go:194] affinity-nodeport-transition Pods: 3 out of 3 created, 0 running, 3 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0311 16:46:23.900924 7237 runners.go:194] affinity-nodeport-transition Pods: 3 out of 3 created, 0 running, 3 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0311 16:46:26.901133 7237 runners.go:194] affinity-nodeport-transition Pods: 3 out of 3 created, 0 running, 3 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0311 16:46:29.902312 7237 runners.go:194] affinity-nodeport-transition Pods: 3 out of 3 created, 0 running, 3 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0311 16:46:32.903297 7237 runners.go:194] affinity-nodeport-transition Pods: 3 out of 3 created, 0 running, 3 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0311 16:46:35.903974 7237 runners.go:194] affinity-nodeport-transition Pods: 3 out of 3 created, 0 running, 3 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0311 16:46:38.904212 7237 runners.go:194] affinity-nodeport-transition Pods: 3 out of 3 created, 0 running, 3 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0311 16:46:41.905258 7237 runners.go:194] affinity-nodeport-transition Pods: 3 out of 3 created, 0 running, 3 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0311 16:46:44.905971 7237 runners.go:194] affinity-nodeport-transition Pods: 3 out of 3 created, 0 running, 3 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0311 16:46:47.906341 7237 runners.go:194] affinity-nodeport-transition Pods: 3 out of 3 created, 0 running, 3 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0311 16:46:50.907094 7237 runners.go:194] affinity-nodeport-transition Pods: 3 out of 3 created, 0 running, 3 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0311 16:46:53.907303 7237 runners.go:194] affinity-nodeport-transition Pods: 3 out of 3 created, 0 running, 3 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0311 16:46:56.908144 7237 runners.go:194] affinity-nodeport-transition Pods: 3 out of 3 created, 0 running, 3 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0311 16:46:59.908250 7237 runners.go:194] affinity-nodeport-transition Pods: 3 out of 3 created, 0 running, 3 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0311 16:47:02.909317 7237 runners.go:194] affinity-nodeport-transition Pods: 3 out of 3 created, 0 running, 3 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0311 16:47:05.910185 7237 runners.go:194] affinity-nodeport-transition Pods: 3 out of 3 created, 0 running, 3 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0311 16:47:08.910375 7237 runners.go:194] affinity-nodeport-transition Pods: 3 out of 3 created, 0 running, 3 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0311 16:47:11.911148 7237 runners.go:194] affinity-nodeport-transition Pods: 3 out of 3 created, 0 running, 3 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0311 16:47:14.911706 7237 runners.go:194] affinity-nodeport-transition Pods: 3 out of 3 created, 0 running, 3 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0311 16:47:17.912091 7237 runners.go:194] affinity-nodeport-transition Pods: 3 out of 3 created, 0 running, 3 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0311 16:47:20.913280 7237 runners.go:194] affinity-nodeport-transition Pods: 3 out of 3 created, 0 running, 3 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0311 16:47:23.913658 7237 runners.go:194] affinity-nodeport-transition Pods: 3 out of 3 created, 0 running, 3 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0311 16:47:26.914637 7237 runners.go:194] affinity-nodeport-transition Pods: 3 out of 3 created, 0 running, 3 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0311 16:47:29.915423 7237 runners.go:194] affinity-nodeport-transition Pods: 3 out of 3 created, 0 running, 3 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0311 16:47:32.916180 7237 runners.go:194] affinity-nodeport-transition Pods: 3 out of 3 created, 0 running, 3 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0311 16:47:35.916404 7237 runners.go:194] affinity-nodeport-transition Pods: 3 out of 3 created, 0 running, 3 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0311 16:47:38.917376 7237 runners.go:194] affinity-nodeport-transition Pods: 3 out of 3 created, 0 running, 3 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0311 16:47:41.918381 7237 runners.go:194] affinity-nodeport-transition Pods: 3 out of 3 created, 0 running, 3 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0311 16:47:44.919017 7237 runners.go:194] affinity-nodeport-transition Pods: 3 out of 3 created, 0 running, 3 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0311 16:47:47.919205 7237 runners.go:194] affinity-nodeport-transition Pods: 3 out of 3 created, 0 running, 3 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0311 16:47:50.919874 7237 runners.go:194] affinity-nodeport-transition Pods: 3 out of 3 created, 0 running, 3 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0311 16:47:53.920738 7237 runners.go:194] affinity-nodeport-transition Pods: 3 out of 3 created, 0 running, 3 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0311 16:47:56.921387 7237 runners.go:194] affinity-nodeport-transition Pods: 3 out of 3 created, 0 running, 3 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0311 16:47:59.921859 7237 runners.go:194] affinity-nodeport-transition Pods: 3 out of 3 created, 0 running, 3 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0311 16:48:02.922304 7237 runners.go:194] affinity-nodeport-transition Pods: 3 out of 3 created, 0 running, 3 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0311 16:48:05.923326 7237 runners.go:194] affinity-nodeport-transition Pods: 3 out of 3 created, 0 running, 3 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0311 16:48:08.924345 7237 runners.go:194] affinity-nodeport-transition Pods: 3 out of 3 created, 0 running, 3 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0311 16:48:11.925401 7237 runners.go:194] affinity-nodeport-transition Pods: 3 out of 3 created, 0 running, 3 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0311 16:48:14.926526 7237 runners.go:194] affinity-nodeport-transition Pods: 3 out of 3 created, 0 running, 3 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0311 16:48:17.927292 7237 runners.go:194] affinity-nodeport-transition Pods: 3 out of 3 created, 0 running, 3 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0311 16:48:20.928307 7237 runners.go:194] affinity-nodeport-transition Pods: 3 out of 3 created, 0 running, 3 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0311 16:48:23.928571 7237 runners.go:194] affinity-nodeport-transition Pods: 3 out of 3 created, 0 running, 3 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0311 16:48:26.929360 7237 runners.go:194] affinity-nodeport-transition Pods: 3 out of 3 created, 0 running, 3 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0311 16:48:29.930226 7237 runners.go:194] affinity-nodeport-transition Pods: 3 out of 3 created, 0 running, 3 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0311 16:48:32.930358 7237 runners.go:194] affinity-nodeport-transition Pods: 3 out of 3 created, 0 running, 3 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0311 16:48:35.931310 7237 runners.go:194] affinity-nodeport-transition Pods: 3 out of 3 created, 0 running, 3 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0311 16:48:38.931524 7237 runners.go:194] affinity-nodeport-transition Pods: 3 out of 3 created, 0 running, 3 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0311 16:48:41.932246 7237 runners.go:194] affinity-nodeport-transition Pods: 3 out of 3 created, 0 running, 3 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0311 16:48:44.933289 7237 runners.go:194] affinity-nodeport-transition Pods: 3 out of 3 created, 0 running, 3 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0311 16:48:47.934365 7237 runners.go:194] affinity-nodeport-transition Pods: 3 out of 3 created, 0 running, 3 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0311 16:48:50.935049 7237 runners.go:194] affinity-nodeport-transition Pods: 3 out of 3 created, 0 running, 3 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0311 16:48:53.935836 7237 runners.go:194] affinity-nodeport-transition Pods: 3 out of 3 created, 0 running, 3 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0311 16:48:56.936083 7237 runners.go:194] affinity-nodeport-transition Pods: 3 out of 3 created, 0 running, 3 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0311 16:48:59.936857 7237 runners.go:194] affinity-nodeport-transition Pods: 3 out of 3 created, 0 running, 3 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0311 16:49:02.937517 7237 runners.go:194] affinity-nodeport-transition Pods: 3 out of 3 created, 0 running, 3 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0311 16:49:05.938210 7237 runners.go:194] affinity-nodeport-transition Pods: 3 out of 3 created, 0 running, 3 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0311 16:49:08.938867 7237 runners.go:194] affinity-nodeport-transition Pods: 3 out of 3 created, 0 running, 3 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0311 16:49:11.939255 7237 runners.go:194] affinity-nodeport-transition Pods: 3 out of 3 created, 0 running, 3 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0311 16:49:14.940356 7237 runners.go:194] affinity-nodeport-transition Pods: 3 out of 3 created, 0 running, 3 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0311 16:49:17.941270 7237 runners.go:194] affinity-nodeport-transition Pods: 3 out of 3 created, 0 running, 3 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0311 16:49:20.942356 7237 runners.go:194] affinity-nodeport-transition Pods: 3 out of 3 created, 0 running, 3 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0311 16:49:23.942605 7237 runners.go:194] affinity-nodeport-transition Pods: 3 out of 3 created, 0 running, 3 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0311 16:49:26.943340 7237 runners.go:194] affinity-nodeport-transition Pods: 3 out of 3 created, 0 running, 3 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0311 16:49:29.944364 7237 runners.go:194] affinity-nodeport-transition Pods: 3 out of 3 created, 0 running, 3 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0311 16:49:30.095170 7237 runners.go:194] Pod affinity-nodeport-transition-4x8jj i-04523d1dffb0fe725 Pending <nil> I0311 16:49:30.095224 7237 runners.go:194] Pod affinity-nodeport-transition-tqldd i-04523d1dffb0fe725 Pending <nil> I0311 16:49:30.095232 7237 runners.go:194] Pod affinity-nodeport-transition-vz5tg i-07e2b6e650d7ef56d Pending <nil> Mar 11 16:49:30.095: INFO: Unexpected error: failed to create replication controller with service in the namespace: services-6839: <*errors.errorString | 0xc0011f8610>: { s: "only 0 pods started out of 3", } [FAILED] failed to create replication controller with service in the namespace: services-6839: only 0 pods started out of 3 In [It] at: test/e2e/network/service.go:3967 @ 03/11/23 16:49:30.095 < Exit [It] should be able to switch session affinity for NodePort service [LinuxOnly] [Conformance] - test/e2e/network/service.go:2224 @ 03/11/23 16:49:30.095 (5m3.68s) > Enter [AfterEach] [sig-network] Services - test/e2e/framework/node/init/init.go:33 @ 03/11/23 16:49:30.095 Mar 11 16:49:30.095: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready Mar 11 16:49:30.246: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:49:30.246: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:49:32.398: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:49:32.398: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:49:34.398: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:49:34.398: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:49:36.397: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:49:36.397: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:49:38.398: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:49:38.398: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:49:40.397: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:49:40.397: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:49:42.397: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:49:42.397: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:49:44.398: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:49:44.398: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:49:46.398: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:49:46.398: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:49:48.398: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:49:48.398: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:49:50.397: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:49:50.397: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:49:52.398: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:49:52.398: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:49:54.398: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:49:54.398: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:49:56.398: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:49:56.398: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:49:58.400: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:49:58.400: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:50:00.398: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:50:00.398: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:50:02.398: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:50:02.398: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:50:04.398: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:50:04.398: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:50:06.398: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:50:06.398: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:50:08.398: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:50:08.398: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:50:10.398: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:50:10.398: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:50:12.398: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:50:12.398: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:50:14.398: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:50:14.398: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:50:16.398: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:50:16.398: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:50:18.398: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:50:18.398: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:50:20.398: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:50:20.398: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:50:22.398: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:50:22.398: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:50:24.398: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:50:24.398: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:50:26.398: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:50:26.398: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:50:28.397: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:50:28.397: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:50:30.398: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:50:30.398: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:50:32.398: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:50:32.398: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:50:34.398: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:50:34.398: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:50:36.398: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:50:36.398: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:50:38.398: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:50:38.398: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:50:40.398: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:50:40.398: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:50:42.398: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:50:42.398: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:50:44.399: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:50:44.399: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:50:46.398: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:50:46.398: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:50:48.398: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:50:48.398: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:50:50.398: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:50:50.398: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:50:52.398: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:50:52.398: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:50:54.398: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:50:54.398: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:50:56.398: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:50:56.398: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:50:58.398: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:50:58.398: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:51:00.398: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:51:00.398: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:51:02.398: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:51:02.398: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:51:04.398: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:51:04.398: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:51:06.398: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:51:06.398: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:51:08.398: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:51:08.398: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:51:10.398: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:51:10.398: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:51:12.402: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:51:12.402: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:51:14.397: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:51:14.397: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:51:16.398: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:51:16.398: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:51:18.397: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:51:18.397: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:51:20.398: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:51:20.398: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:51:22.397: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:51:22.397: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:51:24.398: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:51:24.398: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:51:26.398: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:51:26.398: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:51:28.398: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:51:28.398: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:51:30.398: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:51:30.398: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:51:32.398: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:51:32.398: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:51:34.398: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:51:34.398: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:51:36.398: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:51:36.398: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:51:38.398: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:51:38.398: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:51:40.398: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:51:40.398: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:51:42.398: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:51:42.398: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:51:44.398: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:51:44.398: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:51:46.398: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:51:46.398: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:51:48.398: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:51:48.398: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:51:50.398: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:51:50.398: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:51:52.398: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:51:52.398: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:51:54.398: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:51:54.398: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:51:56.398: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:51:56.398: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:51:58.409: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:51:58.409: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:52:00.397: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:52:00.397: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:52:02.398: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:52:02.398: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:52:04.398: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:52:04.398: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:52:06.398: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:52:06.398: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:52:08.398: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:52:08.398: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:52:10.398: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:52:10.398: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:52:12.398: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:52:12.398: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:52:14.398: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:52:14.398: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:52:16.398: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:52:16.398: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:52:18.398: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:52:18.398: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:52:20.398: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:52:20.398: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:52:22.398: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:52:22.398: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:52:24.398: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:52:24.398: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:52:26.398: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:52:26.398: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:52:28.398: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:52:28.398: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:52:30.398: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:52:30.398: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:52:30.549: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:52:30.549: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure < Exit [AfterEach] [sig-network] Services - test/e2e/framework/node/init/init.go:33 @ 03/11/23 16:52:30.549 (3m0.454s) > Enter [DeferCleanup (Each)] [sig-network] Services - test/e2e/framework/metrics/init/init.go:35 @ 03/11/23 16:52:30.549 < Exit [DeferCleanup (Each)] [sig-network] Services - test/e2e/framework/metrics/init/init.go:35 @ 03/11/23 16:52:30.549 (0s) > Enter [DeferCleanup (Each)] [sig-network] Services - dump namespaces | framework.go:209 @ 03/11/23 16:52:30.549 STEP: dump namespace information after failure - test/e2e/framework/framework.go:288 @ 03/11/23 16:52:30.549 STEP: Collecting events from namespace "services-6839". - test/e2e/framework/debug/dump.go:42 @ 03/11/23 16:52:30.549 STEP: Found 24 events. - test/e2e/framework/debug/dump.go:46 @ 03/11/23 16:52:30.7 Mar 11 16:52:30.700: INFO: At 2023-03-11 16:44:27 +0000 UTC - event for affinity-nodeport-transition: {replication-controller } SuccessfulCreate: Created pod: affinity-nodeport-transition-4x8jj Mar 11 16:52:30.700: INFO: At 2023-03-11 16:44:27 +0000 UTC - event for affinity-nodeport-transition: {replication-controller } SuccessfulCreate: Created pod: affinity-nodeport-transition-vz5tg Mar 11 16:52:30.700: INFO: At 2023-03-11 16:44:27 +0000 UTC - event for affinity-nodeport-transition: {replication-controller } SuccessfulCreate: Created pod: affinity-nodeport-transition-tqldd Mar 11 16:52:30.700: INFO: At 2023-03-11 16:44:27 +0000 UTC - event for affinity-nodeport-transition-4x8jj: {default-scheduler } Scheduled: Successfully assigned services-6839/affinity-nodeport-transition-4x8jj to i-04523d1dffb0fe725 Mar 11 16:52:30.700: INFO: At 2023-03-11 16:44:27 +0000 UTC - event for affinity-nodeport-transition-tqldd: {default-scheduler } Scheduled: Successfully assigned services-6839/affinity-nodeport-transition-tqldd to i-04523d1dffb0fe725 Mar 11 16:52:30.700: INFO: At 2023-03-11 16:44:27 +0000 UTC - event for affinity-nodeport-transition-vz5tg: {default-scheduler } Scheduled: Successfully assigned services-6839/affinity-nodeport-transition-vz5tg to i-07e2b6e650d7ef56d Mar 11 16:52:30.700: INFO: At 2023-03-11 16:44:46 +0000 UTC - event for affinity-nodeport-transition-vz5tg: {kubelet i-07e2b6e650d7ef56d} Pulled: Container image "registry.k8s.io/e2e-test-images/agnhost:2.43" already present on machine Mar 11 16:52:30.700: INFO: At 2023-03-11 16:44:47 +0000 UTC - event for affinity-nodeport-transition-vz5tg: {kubelet i-07e2b6e650d7ef56d} Created: Created container affinity-nodeport-transition Mar 11 16:52:30.700: INFO: At 2023-03-11 16:44:48 +0000 UTC - event for affinity-nodeport-transition-tqldd: {kubelet i-04523d1dffb0fe725} Pulled: Container image "registry.k8s.io/e2e-test-images/agnhost:2.43" already present on machine Mar 11 16:52:30.700: INFO: At 2023-03-11 16:44:49 +0000 UTC - event for affinity-nodeport-transition-4x8jj: {kubelet i-04523d1dffb0fe725} Pulled: Container image "registry.k8s.io/e2e-test-images/agnhost:2.43" already present on machine Mar 11 16:52:30.700: INFO: At 2023-03-11 16:44:50 +0000 UTC - event for affinity-nodeport-transition-4x8jj: {kubelet i-04523d1dffb0fe725} Created: Created container affinity-nodeport-transition Mar 11 16:52:30.700: INFO: At 2023-03-11 16:44:50 +0000 UTC - event for affinity-nodeport-transition-tqldd: {kubelet i-04523d1dffb0fe725} Created: Created container affinity-nodeport-transition Mar 11 16:52:30.700: INFO: At 2023-03-11 16:50:36 +0000 UTC - event for affinity-nodeport-transition-4x8jj: {taint-controller } TaintManagerEviction: Marking for deletion Pod services-6839/affinity-nodeport-transition-4x8jj Mar 11 16:52:30.700: INFO: At 2023-03-11 16:50:36 +0000 UTC - event for affinity-nodeport-transition-tqldd: {taint-controller } TaintManagerEviction: Marking for deletion Pod services-6839/affinity-nodeport-transition-tqldd Mar 11 16:52:30.700: INFO: At 2023-03-11 16:50:37 +0000 UTC - event for affinity-nodeport-transition: {replication-controller } SuccessfulCreate: Created pod: affinity-nodeport-transition-dwrt9 Mar 11 16:52:30.700: INFO: At 2023-03-11 16:50:37 +0000 UTC - event for affinity-nodeport-transition: {replication-controller } SuccessfulCreate: Created pod: affinity-nodeport-transition-5b8kj Mar 11 16:52:30.700: INFO: At 2023-03-11 16:50:37 +0000 UTC - event for affinity-nodeport-transition-5b8kj: {default-scheduler } FailedScheduling: 0/3 nodes are available: 1 node(s) had untolerated taint {node-role.kubernetes.io/control-plane: }, 2 node(s) had untolerated taint {node.kubernetes.io/unreachable: }. preemption: 0/3 nodes are available: 3 Preemption is not helpful for scheduling.. Mar 11 16:52:30.700: INFO: At 2023-03-11 16:50:37 +0000 UTC - event for affinity-nodeport-transition-dwrt9: {default-scheduler } FailedScheduling: 0/3 nodes are available: 1 node(s) had untolerated taint {node-role.kubernetes.io/control-plane: }, 2 node(s) had untolerated taint {node.kubernetes.io/unreachable: }. preemption: 0/3 nodes are available: 3 Preemption is not helpful for scheduling.. Mar 11 16:52:30.700: INFO: At 2023-03-11 16:50:38 +0000 UTC - event for affinity-nodeport-transition-5b8kj: {karpenter } Nominate: Pod should schedule on i-07e2b6e650d7ef56d Mar 11 16:52:30.700: INFO: At 2023-03-11 16:50:38 +0000 UTC - event for affinity-nodeport-transition-dwrt9: {karpenter } Nominate: Pod should schedule on i-07e2b6e650d7ef56d Mar 11 16:52:30.700: INFO: At 2023-03-11 16:50:46 +0000 UTC - event for affinity-nodeport-transition-vz5tg: {taint-controller } TaintManagerEviction: Marking for deletion Pod services-6839/affinity-nodeport-transition-vz5tg Mar 11 16:52:30.700: INFO: At 2023-03-11 16:50:47 +0000 UTC - event for affinity-nodeport-transition: {replication-controller } SuccessfulCreate: Created pod: affinity-nodeport-transition-xvvs5 Mar 11 16:52:30.700: INFO: At 2023-03-11 16:50:47 +0000 UTC - event for affinity-nodeport-transition-xvvs5: {default-scheduler } FailedScheduling: 0/3 nodes are available: 1 node(s) had untolerated taint {node-role.kubernetes.io/control-plane: }, 2 node(s) had untolerated taint {node.kubernetes.io/unreachable: }. preemption: 0/3 nodes are available: 3 Preemption is not helpful for scheduling.. Mar 11 16:52:30.700: INFO: At 2023-03-11 16:50:48 +0000 UTC - event for affinity-nodeport-transition-xvvs5: {karpenter } Nominate: Pod should schedule on i-07e2b6e650d7ef56d Mar 11 16:52:30.850: INFO: POD NODE PHASE GRACE CONDITIONS Mar 11 16:52:30.850: INFO: affinity-nodeport-transition-4x8jj i-04523d1dffb0fe725 Pending 1s [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2023-03-11 16:44:27 +0000 UTC } {Ready False 0001-01-01 00:00:00 +0000 UTC 2023-03-11 16:44:27 +0000 UTC ContainersNotReady containers with unready status: [affinity-nodeport-transition]} {ContainersReady False 0001-01-01 00:00:00 +0000 UTC 2023-03-11 16:44:27 +0000 UTC ContainersNotReady containers with unready status: [affinity-nodeport-transition]} {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2023-03-11 16:44:27 +0000 UTC } {DisruptionTarget True 0001-01-01 00:00:00 +0000 UTC 2023-03-11 16:50:36 +0000 UTC DeletionByTaintManager Taint manager: deleting due to NoExecute taint}] Mar 11 16:52:30.851: INFO: affinity-nodeport-transition-5b8kj Pending [{PodScheduled False 0001-01-01 00:00:00 +0000 UTC 2023-03-11 16:50:37 +0000 UTC Unschedulable 0/3 nodes are available: 1 node(s) had untolerated taint {node-role.kubernetes.io/control-plane: }, 2 node(s) had untolerated taint {node.kubernetes.io/unreachable: }. preemption: 0/3 nodes are available: 3 Preemption is not helpful for scheduling..}] Mar 11 16:52:30.851: INFO: affinity-nodeport-transition-dwrt9 Pending [{PodScheduled False 0001-01-01 00:00:00 +0000 UTC 2023-03-11 16:50:37 +0000 UTC Unschedulable 0/3 nodes are available: 1 node(s) had untolerated taint {node-role.kubernetes.io/control-plane: }, 2 node(s) had untolerated taint {node.kubernetes.io/unreachable: }. preemption: 0/3 nodes are available: 3 Preemption is not helpful for scheduling..}] Mar 11 16:52:30.851: INFO: affinity-nodeport-transition-tqldd i-04523d1dffb0fe725 Pending 1s [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2023-03-11 16:44:27 +0000 UTC } {Ready False 0001-01-01 00:00:00 +0000 UTC 2023-03-11 16:44:27 +0000 UTC ContainersNotReady containers with unready status: [affinity-nodeport-transition]} {ContainersReady False 0001-01-01 00:00:00 +0000 UTC 2023-03-11 16:44:27 +0000 UTC ContainersNotReady containers with unready status: [affinity-nodeport-transition]} {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2023-03-11 16:44:27 +0000 UTC } {DisruptionTarget True 0001-01-01 00:00:00 +0000 UTC 2023-03-11 16:50:36 +0000 UTC DeletionByTaintManager Taint manager: deleting due to NoExecute taint}] Mar 11 16:52:30.851: INFO: affinity-nodeport-transition-vz5tg i-07e2b6e650d7ef56d Pending 1s [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2023-03-11 16:44:27 +0000 UTC } {Ready False 0001-01-01 00:00:00 +0000 UTC 2023-03-11 16:44:27 +0000 UTC ContainersNotReady containers with unready status: [affinity-nodeport-transition]} {ContainersReady False 0001-01-01 00:00:00 +0000 UTC 2023-03-11 16:44:27 +0000 UTC ContainersNotReady containers with unready status: [affinity-nodeport-transition]} {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2023-03-11 16:44:27 +0000 UTC } {DisruptionTarget True 0001-01-01 00:00:00 +0000 UTC 2023-03-11 16:50:46 +0000 UTC DeletionByTaintManager Taint manager: deleting due to NoExecute taint}] Mar 11 16:52:30.851: INFO: affinity-nodeport-transition-xvvs5 Pending [{PodScheduled False 0001-01-01 00:00:00 +0000 UTC 2023-03-11 16:50:47 +0000 UTC Unschedulable 0/3 nodes are available: 1 node(s) had untolerated taint {node-role.kubernetes.io/control-plane: }, 2 node(s) had untolerated taint {node.kubernetes.io/unreachable: }. preemption: 0/3 nodes are available: 3 Preemption is not helpful for scheduling..}] Mar 11 16:52:30.851: INFO: Mar 11 16:52:41.002: INFO: Unable to fetch services-6839/affinity-nodeport-transition-4x8jj/affinity-nodeport-transition logs: an error on the server ("unknown") has prevented the request from succeeding (get pods affinity-nodeport-transition-4x8jj) Mar 11 16:52:51.454: INFO: Unable to fetch services-6839/affinity-nodeport-transition-tqldd/affinity-nodeport-transition logs: an error on the server ("unknown") has prevented the request from succeeding (get pods affinity-nodeport-transition-tqldd) Mar 11 16:53:01.606: INFO: Unable to fetch services-6839/affinity-nodeport-transition-vz5tg/affinity-nodeport-transition logs: an error on the server ("unknown") has prevented the request from succeeding (get pods affinity-nodeport-transition-vz5tg) Mar 11 16:53:01.757: INFO: Logging node info for node i-04523d1dffb0fe725 Mar 11 16:53:01.907: INFO: Node Info: &Node{ObjectMeta:{i-04523d1dffb0fe725 5d37640f-a7d9-4c26-86c5-9da21f7ccc18 50021 0 2023-03-11 16:24:20 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:arm64 beta.kubernetes.io/instance-type:t4g.small beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-northeast-1 failure-domain.beta.kubernetes.io/zone:ap-northeast-1a karpenter.k8s.aws/instance-category:t karpenter.k8s.aws/instance-cpu:2 karpenter.k8s.aws/instance-family:t4g karpenter.k8s.aws/instance-generation:4 karpenter.k8s.aws/instance-hypervisor:nitro karpenter.k8s.aws/instance-memory:2048 karpenter.k8s.aws/instance-pods:11 karpenter.k8s.aws/instance-size:small karpenter.sh/capacity-type:spot karpenter.sh/initialized:true karpenter.sh/provisioner-name:nodes kubernetes.io/arch:arm64 kubernetes.io/hostname:i-04523d1dffb0fe725 kubernetes.io/os:linux node-role.kubernetes.io/node: node-role.kubernetes.io/spot-worker:true node.kubernetes.io/instance-type:t4g.small topology.ebs.csi.aws.com/zone:ap-northeast-1a topology.hostpath.csi/node:i-04523d1dffb0fe725 topology.kubernetes.io/region:ap-northeast-1 topology.kubernetes.io/zone:ap-northeast-1a] map[csi.volume.kubernetes.io/nodeid:{"csi-hostpath-read-write-once-pod-7172":"i-04523d1dffb0fe725","csi-hostpath-volume-expand-1673":"i-04523d1dffb0fe725","ebs.csi.aws.com":"i-04523d1dffb0fe725"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [{karpenter.sh/v1alpha5 Provisioner nodes ac7283b2-eed0-4a9f-9ddd-616f33a0991a <nil> 0xc006719555}] [karpenter.sh/termination] [{aws-cloud-controller-manager Update v1 2023-03-11 16:24:20 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{}}}} } {kops-controller Update v1 2023-03-11 16:24:20 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{},"f:node-role.kubernetes.io/spot-worker":{}}}} } {karpenter Update v1 2023-03-11 16:26:12 +0000 UTC FieldsV1 {"f:metadata":{"f:finalizers":{".":{},"v:\"karpenter.sh/termination\"":{}},"f:labels":{".":{},"f:karpenter.k8s.aws/instance-category":{},"f:karpenter.k8s.aws/instance-cpu":{},"f:karpenter.k8s.aws/instance-family":{},"f:karpenter.k8s.aws/instance-generation":{},"f:karpenter.k8s.aws/instance-hypervisor":{},"f:karpenter.k8s.aws/instance-memory":{},"f:karpenter.k8s.aws/instance-pods":{},"f:karpenter.k8s.aws/instance-size":{},"f:karpenter.sh/capacity-type":{},"f:karpenter.sh/initialized":{},"f:karpenter.sh/provisioner-name":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}},"f:ownerReferences":{".":{},"k:{\"uid\":\"ac7283b2-eed0-4a9f-9ddd-616f33a0991a\"}":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-03-11 16:26:50 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{".":{},"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"InternalIP\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kubelet Update v1 2023-03-11 16:44:47 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{"f:kubernetes.io/hostname":{},"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{".":{},"f:cpu":{},"f:ephemeral-storage":{},"f:hugepages-1Gi":{},"f:hugepages-2Mi":{},"f:hugepages-32Mi":{},"f:hugepages-64Ki":{},"f:memory":{},"f:pods":{}},"f:capacity":{".":{},"f:cpu":{},"f:ephemeral-storage":{},"f:hugepages-1Gi":{},"f:hugepages-2Mi":{},"f:hugepages-32Mi":{},"f:hugepages-64Ki":{},"f:memory":{},"f:pods":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{}}},"f:daemonEndpoints":{"f:kubeletEndpoint":{"f:Port":{}}},"f:images":{},"f:nodeInfo":{"f:architecture":{},"f:bootID":{},"f:containerRuntimeVersion":{},"f:kernelVersion":{},"f:kubeProxyVersion":{},"f:kubeletVersion":{},"f:machineID":{},"f:operatingSystem":{},"f:osImage":{},"f:systemUUID":{}},"f:volumesInUse":{}}} status} {kube-controller-manager Update v1 2023-03-11 16:45:36 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:node.alpha.kubernetes.io/ttl":{}},"f:labels":{"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.2.0/24\"":{}},"f:taints":{}}} } {kube-controller-manager Update v1 2023-03-11 16:52:55 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{".":{},"k:{\"type\":\"DiskPressure\"}":{".":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"MemoryPressure\"}":{".":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"PIDPressure\"}":{".":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Ready\"}":{".":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}},"f:volumesAttached":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.2.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-northeast-1a/i-04523d1dffb0fe725,Unschedulable:false,Taints:[]Taint{Taint{Key:node.kubernetes.io/unreachable,Value:,Effect:NoSchedule,TimeAdded:2023-03-11 16:45:31 +0000 UTC,},Taint{Key:node.kubernetes.io/unreachable,Value:,Effect:NoExecute,TimeAdded:2023-03-11 16:45:36 +0000 UTC,},},ConfigSource:nil,PodCIDRs:[100.96.2.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49756430336 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-32Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-64Ki: {{0 0} {<nil>} 0 DecimalSI},memory: {{1941540864 0} {<nil>} 1896036Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44780787229 0} {<nil>} 44780787229 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-32Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-64Ki: {{0 0} {<nil>} 0 DecimalSI},memory: {{1836683264 0} {<nil>} 1793636Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:Ready,Status:Unknown,LastHeartbeatTime:2023-03-11 16:44:47 +0000 UTC,LastTransitionTime:2023-03-11 16:45:31 +0000 UTC,Reason:NodeStatusUnknown,Message:Kubelet stopped posting node status.,},NodeCondition{Type:MemoryPressure,Status:Unknown,LastHeartbeatTime:2023-03-11 16:44:47 +0000 UTC,LastTransitionTime:2023-03-11 16:45:31 +0000 UTC,Reason:NodeStatusUnknown,Message:Kubelet stopped posting node status.,},NodeCondition{Type:DiskPressure,Status:Unknown,LastHeartbeatTime:2023-03-11 16:44:47 +0000 UTC,LastTransitionTime:2023-03-11 16:45:31 +0000 UTC,Reason:NodeStatusUnknown,Message:Kubelet stopped posting node status.,},NodeCondition{Type:PIDPressure,Status:Unknown,LastHeartbeatTime:2023-03-11 16:44:47 +0000 UTC,LastTransitionTime:2023-03-11 16:45:31 +0000 UTC,Reason:NodeStatusUnknown,Message:Kubelet stopped posting node status.,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.48.168,},NodeAddress{Type:ExternalIP,Address:52.199.253.156,},NodeAddress{Type:InternalDNS,Address:i-04523d1dffb0fe725.ap-northeast-1.compute.internal,},NodeAddress{Type:Hostname,Address:i-04523d1dffb0fe725.ap-northeast-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-52-199-253-156.ap-northeast-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2c8fb0e6b2e9c4af310f65b6161cd8,SystemUUID:ec2c8fb0-e6b2-e9c4-af31-0f65b6161cd8,BootID:a3b8de83-558b-44c4-a46a-d097c3f98225,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 22.04.2 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.27.0-alpha.3.420+bb6c6fad2c19b3,KubeProxyVersion:v1.27.0-alpha.3.420+bb6c6fad2c19b3,OperatingSystem:linux,Architecture:arm64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:157636062,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/jessie-dnsutils@sha256:24aaf2626d6b27864c29de2097e8bbb840b3a414271bf7c8995e431e47d8408e registry.k8s.io/e2e-test-images/jessie-dnsutils:1.7],SizeBytes:116764683,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:51eae8381dcb1078289fa7b4f3df2630cdc18d09fb56f8e56b41c40e191d6c83 registry.k8s.io/etcd:3.5.7-0],SizeBytes:80665728,},ContainerImage{Names:[registry.k8s.io/kube-proxy-arm64:v1.27.0-alpha.3.420_bb6c6fad2c19b3],SizeBytes:60947298,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:16bbf38c463a4223d8cfe4da12bc61010b082a79b4bb003e2d3ba3ece5dd5f9e registry.k8s.io/e2e-test-images/agnhost:2.43],SizeBytes:49808364,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nautilus@sha256:80ba6c8c44f9623f06e868a1aa66026c8ec438ad814f9ec95e9333b415fe3550 registry.k8s.io/e2e-test-images/nautilus:1.7],SizeBytes:47953400,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:3fe7acf013d1264ffded116b80a73dc129a449b0fccdb8d21af8279f2233f36e registry.k8s.io/e2e-test-images/httpd:2.4.39-4],SizeBytes:41528818,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:148b022f5c5da426fc2f3c14b5c0867e58ef05961510c84749ac1fddcb0fef22 registry.k8s.io/e2e-test-images/httpd:2.4.38-4],SizeBytes:40343601,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:28932904,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:ee3b525d5b89db99da3b8eb521d9cd90cb6e9ef0fbb651e98bb37be78d36b5b8 registry.k8s.io/sig-storage/csi-provisioner:v3.3.0],SizeBytes:23945474,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/sample-apiserver@sha256:8d70890151aa5d096f331cb9da1b9cd5be0412b7363fe67b5c3befdcaa2a28d0 registry.k8s.io/e2e-test-images/sample-apiserver:1.17.7],SizeBytes:23857462,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:425d8f1b769398127767b06ed97ce62578a3179bcb99809ce93a1649e025ffe7 registry.k8s.io/sig-storage/csi-resizer:v1.6.0],SizeBytes:22678632,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:291334908ddf71a4661fd7f6d9d97274de8a5378a2b6fdfeb2ce73414a34f82f registry.k8s.io/sig-storage/csi-snapshotter:v6.1.0],SizeBytes:22434180,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:9a685020911e2725ad019dbce6e4a5ab93d51e3d4557f115e64343345e05781b registry.k8s.io/sig-storage/csi-attacher:v4.0.0],SizeBytes:22394501,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:122bfb8c1edabb3c0edd63f06523e6940d958d19b3957dc7b1d6f81e9f1f6119 registry.k8s.io/sig-storage/csi-provisioner:v3.1.0],SizeBytes:21735644,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:9ebbf9f023e7b41ccee3d52afe39a89e3ddacdbb69269d583abfc25847cfd9e4 registry.k8s.io/sig-storage/csi-resizer:v1.4.0],SizeBytes:20848413,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:8b9c313c05f54fb04f8d430896f5f5904b6cb157df261501b29adc04d2b2dc7b registry.k8s.io/sig-storage/csi-attacher:v3.4.0],SizeBytes:20565480,},ContainerImage{Names:[registry.k8s.io/cpa/cluster-proportional-autoscaler@sha256:68d396900aeaa072c1f27289485fdac29834045a6f3ffe369bf389d830ef572d registry.k8s.io/cpa/cluster-proportional-autoscaler:1.8.6],SizeBytes:19007623,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nonroot@sha256:ee9f50b3c64b174d296d91ca9f69a914ac30e59095dfb462b2b518ad28a63655 registry.k8s.io/e2e-test-images/nonroot:1.4],SizeBytes:18561972,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:92257881c1d6493cf18299a24af42330f891166560047902b8d431fb66b01af5 registry.k8s.io/sig-storage/hostpathplugin:v1.9.0],SizeBytes:18267611,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:9b2d9abda017c28b12f84a344f57cd73fbdb6c2bd7dd5bdd5018246ad1093ba6 registry.k8s.io/sig-storage/hostpathplugin:v1.11.0],SizeBytes:18159085,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:017727efcfeb7d053af68e51436ce8e65edbc6ca573720afb4f79c8594036955 registry.k8s.io/coredns/coredns:v1.10.0],SizeBytes:13766234,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:8502345,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:933940f13b3ea0abc62e656c1aa5c5b47c04b15d71250413a6b821bd0c58b94e registry.k8s.io/sig-storage/livenessprobe:v2.7.0],SizeBytes:8223819,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:7685305,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:db048754ae68ae337d8fa96494c96d2a1204c3320f5dcf7e8e71085adec85da6 registry.k8s.io/e2e-test-images/nginx:1.15-4],SizeBytes:6883347,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:5c99cf6a02adda929b10321dbf4ecfa00d87be9ba4fb456006237d530ab4baa1 registry.k8s.io/e2e-test-images/nginx:1.14-4],SizeBytes:6860089,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nonewprivs@sha256:8ac1264691820febacf3aea5d152cbde6d10685731ec14966a9401c6f47a68ac registry.k8s.io/e2e-test-images/nonewprivs:1.3],SizeBytes:5025027,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf registry.k8s.io/e2e-test-images/busybox:1.29-2],SizeBytes:773367,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:2e0f836850e09b8b7cc937681d6194537a09fbd5f6b9e08f4d646a85128e8937 registry.k8s.io/e2e-test-images/busybox:1.29-4],SizeBytes:772933,},ContainerImage{Names:[registry.k8s.io/pause@sha256:7031c1b283388d2c2e09b57badb803c05ebed362dc88d84b480cc47f72a21097 registry.k8s.io/pause:3.9],SizeBytes:268051,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:253553,},},VolumesInUse:[kubernetes.io/csi/csi-hostpath-read-write-once-pod-7172^038fde52-c02c-11ed-b028-5eca9e37a97f kubernetes.io/csi/csi-hostpath-volume-expand-1673^f8193b68-c02b-11ed-8784-9e8522e7cc37 kubernetes.io/csi/ebs.csi.aws.com^vol-0293af09ce81fe75e],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/csi-hostpath-volume-expand-1673^f8193b68-c02b-11ed-8784-9e8522e7cc37,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-0293af09ce81fe75e,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/csi-hostpath-read-write-once-pod-7172^038fde52-c02c-11ed-b028-5eca9e37a97f,DevicePath:,},},Config:nil,},} Mar 11 16:53:01.908: INFO: Logging kubelet events for node i-04523d1dffb0fe725 Mar 11 16:53:02.061: INFO: Logging pods the kubelet thinks is on node i-04523d1dffb0fe725 Mar 11 16:53:12.212: INFO: Unable to retrieve kubelet pods for node i-04523d1dffb0fe725: error trying to reach service: net/http: TLS handshake timeout Mar 11 16:53:12.213: INFO: Logging node info for node i-07e2b6e650d7ef56d Mar 11 16:53:12.363: INFO: Node Info: &Node{ObjectMeta:{i-07e2b6e650d7ef56d fcc65272-6f1b-4228-ac59-e9e3e92cdb2c 47965 0 2023-03-11 16:24:19 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:arm64 beta.kubernetes.io/instance-type:t4g.small beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-northeast-1 failure-domain.beta.kubernetes.io/zone:ap-northeast-1a io.kubernetes.storage.mock/node:some-mock-node karpenter.k8s.aws/instance-category:t karpenter.k8s.aws/instance-cpu:2 karpenter.k8s.aws/instance-family:t4g karpenter.k8s.aws/instance-generation:4 karpenter.k8s.aws/instance-hypervisor:nitro karpenter.k8s.aws/instance-memory:2048 karpenter.k8s.aws/instance-pods:11 karpenter.k8s.aws/instance-size:small karpenter.sh/capacity-type:spot karpenter.sh/initialized:true karpenter.sh/provisioner-name:nodes kubernetes.io/arch:arm64 kubernetes.io/hostname:i-07e2b6e650d7ef56d kubernetes.io/os:linux node-role.kubernetes.io/node: node-role.kubernetes.io/spot-worker:true node.kubernetes.io/instance-type:t4g.small topology.ebs.csi.aws.com/zone:ap-northeast-1a topology.hostpath.csi/node:i-07e2b6e650d7ef56d topology.kubernetes.io/region:ap-northeast-1 topology.kubernetes.io/zone:ap-northeast-1a] map[csi.volume.kubernetes.io/nodeid:{"csi-hostpath-ephemeral-7952":"i-07e2b6e650d7ef56d","csi-hostpath-volume-expand-703":"i-07e2b6e650d7ef56d","ebs.csi.aws.com":"i-07e2b6e650d7ef56d"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [{karpenter.sh/v1alpha5 Provisioner nodes ac7283b2-eed0-4a9f-9ddd-616f33a0991a <nil> 0xc0035c9185}] [karpenter.sh/termination] [{aws-cloud-controller-manager Update v1 2023-03-11 16:24:20 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{}}}} } {kops-controller Update v1 2023-03-11 16:24:20 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{},"f:node-role.kubernetes.io/spot-worker":{}}}} } {karpenter Update v1 2023-03-11 16:26:15 +0000 UTC FieldsV1 {"f:metadata":{"f:finalizers":{".":{},"v:\"karpenter.sh/termination\"":{}},"f:labels":{".":{},"f:karpenter.k8s.aws/instance-category":{},"f:karpenter.k8s.aws/instance-cpu":{},"f:karpenter.k8s.aws/instance-family":{},"f:karpenter.k8s.aws/instance-generation":{},"f:karpenter.k8s.aws/instance-hypervisor":{},"f:karpenter.k8s.aws/instance-memory":{},"f:karpenter.k8s.aws/instance-pods":{},"f:karpenter.k8s.aws/instance-size":{},"f:karpenter.sh/capacity-type":{},"f:karpenter.sh/initialized":{},"f:karpenter.sh/provisioner-name":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}},"f:ownerReferences":{".":{},"k:{\"uid\":\"ac7283b2-eed0-4a9f-9ddd-616f33a0991a\"}":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-03-11 16:26:50 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{".":{},"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"InternalIP\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kubelet Update v1 2023-03-11 16:44:22 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{"f:io.kubernetes.storage.mock/node":{},"f:kubernetes.io/hostname":{},"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{".":{},"f:cpu":{},"f:ephemeral-storage":{},"f:hugepages-1Gi":{},"f:hugepages-2Mi":{},"f:hugepages-32Mi":{},"f:hugepages-64Ki":{},"f:memory":{},"f:pods":{}},"f:capacity":{".":{},"f:cpu":{},"f:ephemeral-storage":{},"f:hugepages-1Gi":{},"f:hugepages-2Mi":{},"f:hugepages-32Mi":{},"f:hugepages-64Ki":{},"f:memory":{},"f:pods":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{}}},"f:daemonEndpoints":{"f:kubeletEndpoint":{"f:Port":{}}},"f:images":{},"f:nodeInfo":{"f:architecture":{},"f:bootID":{},"f:containerRuntimeVersion":{},"f:kernelVersion":{},"f:kubeProxyVersion":{},"f:kubeletVersion":{},"f:machineID":{},"f:operatingSystem":{},"f:osImage":{},"f:systemUUID":{}},"f:volumesInUse":{}}} status} {kube-controller-manager Update v1 2023-03-11 16:45:36 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{".":{},"k:{\"type\":\"DiskPressure\"}":{".":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"MemoryPressure\"}":{".":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"PIDPressure\"}":{".":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Ready\"}":{".":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}},"f:volumesAttached":{}}} status} {kube-controller-manager Update v1 2023-03-11 16:45:46 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:node.alpha.kubernetes.io/ttl":{}},"f:labels":{"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.1.0/24\"":{}},"f:taints":{}}} }]},Spec:NodeSpec{PodCIDR:100.96.1.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-northeast-1a/i-07e2b6e650d7ef56d,Unschedulable:false,Taints:[]Taint{Taint{Key:node.kubernetes.io/unreachable,Value:,Effect:NoSchedule,TimeAdded:2023-03-11 16:45:36 +0000 UTC,},Taint{Key:node.kubernetes.io/unreachable,Value:,Effect:NoExecute,TimeAdded:2023-03-11 16:45:46 +0000 UTC,},},ConfigSource:nil,PodCIDRs:[100.96.1.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49756430336 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-32Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-64Ki: {{0 0} {<nil>} 0 DecimalSI},memory: {{1941549056 0} {<nil>} 1896044Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44780787229 0} {<nil>} 44780787229 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-32Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-64Ki: {{0 0} {<nil>} 0 DecimalSI},memory: {{1836691456 0} {<nil>} 1793644Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:Ready,Status:Unknown,LastHeartbeatTime:2023-03-11 16:44:12 +0000 UTC,LastTransitionTime:2023-03-11 16:45:36 +0000 UTC,Reason:NodeStatusUnknown,Message:Kubelet stopped posting node status.,},NodeCondition{Type:MemoryPressure,Status:Unknown,LastHeartbeatTime:2023-03-11 16:44:12 +0000 UTC,LastTransitionTime:2023-03-11 16:45:36 +0000 UTC,Reason:NodeStatusUnknown,Message:Kubelet stopped posting node status.,},NodeCondition{Type:DiskPressure,Status:Unknown,LastHeartbeatTime:2023-03-11 16:44:12 +0000 UTC,LastTransitionTime:2023-03-11 16:45:36 +0000 UTC,Reason:NodeStatusUnknown,Message:Kubelet stopped posting node status.,},NodeCondition{Type:PIDPressure,Status:Unknown,LastHeartbeatTime:2023-03-11 16:44:12 +0000 UTC,LastTransitionTime:2023-03-11 16:45:36 +0000 UTC,Reason:NodeStatusUnknown,Message:Kubelet stopped posting node status.,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.35.76,},NodeAddress{Type:ExternalIP,Address:54.238.106.83,},NodeAddress{Type:InternalDNS,Address:i-07e2b6e650d7ef56d.ap-northeast-1.compute.internal,},NodeAddress{Type:Hostname,Address:i-07e2b6e650d7ef56d.ap-northeast-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-54-238-106-83.ap-northeast-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2083a8e9d27d02fac967340367f875,SystemUUID:ec2083a8-e9d2-7d02-fac9-67340367f875,BootID:ea264ed6-5b0d-47d6-ad5d-6bd2a1f7c11b,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 22.04.2 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.27.0-alpha.3.420+bb6c6fad2c19b3,KubeProxyVersion:v1.27.0-alpha.3.420+bb6c6fad2c19b3,OperatingSystem:linux,Architecture:arm64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:157636062,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/jessie-dnsutils@sha256:24aaf2626d6b27864c29de2097e8bbb840b3a414271bf7c8995e431e47d8408e registry.k8s.io/e2e-test-images/jessie-dnsutils:1.7],SizeBytes:116764683,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:51eae8381dcb1078289fa7b4f3df2630cdc18d09fb56f8e56b41c40e191d6c83 registry.k8s.io/etcd:3.5.7-0],SizeBytes:80665728,},ContainerImage{Names:[registry.k8s.io/kube-proxy-arm64:v1.27.0-alpha.3.420_bb6c6fad2c19b3],SizeBytes:60947298,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:16bbf38c463a4223d8cfe4da12bc61010b082a79b4bb003e2d3ba3ece5dd5f9e registry.k8s.io/e2e-test-images/agnhost:2.43],SizeBytes:49808364,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nautilus@sha256:80ba6c8c44f9623f06e868a1aa66026c8ec438ad814f9ec95e9333b415fe3550 registry.k8s.io/e2e-test-images/nautilus:1.7],SizeBytes:47953400,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:3fe7acf013d1264ffded116b80a73dc129a449b0fccdb8d21af8279f2233f36e registry.k8s.io/e2e-test-images/httpd:2.4.39-4],SizeBytes:41528818,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:148b022f5c5da426fc2f3c14b5c0867e58ef05961510c84749ac1fddcb0fef22 registry.k8s.io/e2e-test-images/httpd:2.4.38-4],SizeBytes:40343601,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:28932904,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:ee3b525d5b89db99da3b8eb521d9cd90cb6e9ef0fbb651e98bb37be78d36b5b8 registry.k8s.io/sig-storage/csi-provisioner:v3.3.0],SizeBytes:23945474,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/sample-apiserver@sha256:8d70890151aa5d096f331cb9da1b9cd5be0412b7363fe67b5c3befdcaa2a28d0 registry.k8s.io/e2e-test-images/sample-apiserver:1.17.7],SizeBytes:23857462,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:425d8f1b769398127767b06ed97ce62578a3179bcb99809ce93a1649e025ffe7 registry.k8s.io/sig-storage/csi-resizer:v1.6.0],SizeBytes:22678632,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:291334908ddf71a4661fd7f6d9d97274de8a5378a2b6fdfeb2ce73414a34f82f registry.k8s.io/sig-storage/csi-snapshotter:v6.1.0],SizeBytes:22434180,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:9a685020911e2725ad019dbce6e4a5ab93d51e3d4557f115e64343345e05781b registry.k8s.io/sig-storage/csi-attacher:v4.0.0],SizeBytes:22394501,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:122bfb8c1edabb3c0edd63f06523e6940d958d19b3957dc7b1d6f81e9f1f6119 registry.k8s.io/sig-storage/csi-provisioner:v3.1.0],SizeBytes:21735644,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/regression-issue-74839@sha256:b4f1d8d61bdad84bd50442d161d5460e4019d53e989b64220fdbc62fc87d76bf registry.k8s.io/e2e-test-images/regression-issue-74839:1.2],SizeBytes:21331383,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:9ebbf9f023e7b41ccee3d52afe39a89e3ddacdbb69269d583abfc25847cfd9e4 registry.k8s.io/sig-storage/csi-resizer:v1.4.0],SizeBytes:20848413,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:8b9c313c05f54fb04f8d430896f5f5904b6cb157df261501b29adc04d2b2dc7b registry.k8s.io/sig-storage/csi-attacher:v3.4.0],SizeBytes:20565480,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nonroot@sha256:ee9f50b3c64b174d296d91ca9f69a914ac30e59095dfb462b2b518ad28a63655 registry.k8s.io/e2e-test-images/nonroot:1.4],SizeBytes:18561972,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:92257881c1d6493cf18299a24af42330f891166560047902b8d431fb66b01af5 registry.k8s.io/sig-storage/hostpathplugin:v1.9.0],SizeBytes:18267611,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:9b2d9abda017c28b12f84a344f57cd73fbdb6c2bd7dd5bdd5018246ad1093ba6 registry.k8s.io/sig-storage/hostpathplugin:v1.11.0],SizeBytes:18159085,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:017727efcfeb7d053af68e51436ce8e65edbc6ca573720afb4f79c8594036955 registry.k8s.io/coredns/coredns:v1.10.0],SizeBytes:13766234,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:8502345,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:933940f13b3ea0abc62e656c1aa5c5b47c04b15d71250413a6b821bd0c58b94e registry.k8s.io/sig-storage/livenessprobe:v2.7.0],SizeBytes:8223819,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:7685305,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:db048754ae68ae337d8fa96494c96d2a1204c3320f5dcf7e8e71085adec85da6 registry.k8s.io/e2e-test-images/nginx:1.15-4],SizeBytes:6883347,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:5c99cf6a02adda929b10321dbf4ecfa00d87be9ba4fb456006237d530ab4baa1 registry.k8s.io/e2e-test-images/nginx:1.14-4],SizeBytes:6860089,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nonewprivs@sha256:8ac1264691820febacf3aea5d152cbde6d10685731ec14966a9401c6f47a68ac registry.k8s.io/e2e-test-images/nonewprivs:1.3],SizeBytes:5025027,},ContainerImage{Names:[gcr.io/authenticated-image-pulling/alpine@sha256:7ff177862cb50c602bfe81f805969412e619c054a2bbead977d0c276988aa4a0 gcr.io/authenticated-image-pulling/alpine:3.7],SizeBytes:2003505,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf registry.k8s.io/e2e-test-images/busybox:1.29-2],SizeBytes:773367,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:2e0f836850e09b8b7cc937681d6194537a09fbd5f6b9e08f4d646a85128e8937 registry.k8s.io/e2e-test-images/busybox:1.29-4],SizeBytes:772933,},ContainerImage{Names:[registry.k8s.io/pause@sha256:7031c1b283388d2c2e09b57badb803c05ebed362dc88d84b480cc47f72a21097 registry.k8s.io/pause:3.9],SizeBytes:268051,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:253553,},},VolumesInUse:[kubernetes.io/csi/csi-hostpath-ephemeral-7952^eca2344a-c02b-11ed-b483-1e4a85f27a89 kubernetes.io/csi/csi-hostpath-ephemeral-7952^f0ac2401-c02b-11ed-b483-1e4a85f27a89 kubernetes.io/csi/ebs.csi.aws.com^vol-06cba8db23f1489ad],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-06cba8db23f1489ad,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/csi-hostpath-ephemeral-7952^f0ac2401-c02b-11ed-b483-1e4a85f27a89,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/csi-hostpath-ephemeral-7952^eca2344a-c02b-11ed-b483-1e4a85f27a89,DevicePath:,},},Config:nil,},} Mar 11 16:53:12.364: INFO: Logging kubelet events for node i-07e2b6e650d7ef56d Mar 11 16:53:12.517: INFO: Logging pods the kubelet thinks is on node i-07e2b6e650d7ef56d Mar 11 16:53:22.668: INFO: Unable to retrieve kubelet pods for node i-07e2b6e650d7ef56d: error trying to reach service: net/http: TLS handshake timeout Mar 11 16:53:22.668: INFO: Logging node info for node i-0e3467ec552af797e Mar 11 16:53:22.819: INFO: Node Info: &Node{ObjectMeta:{i-0e3467ec552af797e 6c9b0cc5-2c4e-4037-bf31-61bac12a7775 48787 0 2023-03-11 16:20:46 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:arm64 beta.kubernetes.io/instance-type:c6g.xlarge beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-northeast-1 failure-domain.beta.kubernetes.io/zone:ap-northeast-1a kops.k8s.io/kops-controller-pki: kubernetes.io/arch:arm64 kubernetes.io/hostname:i-0e3467ec552af797e kubernetes.io/os:linux node-role.kubernetes.io/control-plane: node.kubernetes.io/exclude-from-external-load-balancers: node.kubernetes.io/instance-type:c6g.xlarge topology.ebs.csi.aws.com/zone:ap-northeast-1a topology.kubernetes.io/region:ap-northeast-1 topology.kubernetes.io/zone:ap-northeast-1a] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-0e3467ec552af797e"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kubelet Update v1 2023-03-11 16:20:46 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {protokube Update v1 2023-03-11 16:21:30 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kops.k8s.io/kops-controller-pki":{},"f:node-role.kubernetes.io/control-plane":{},"f:node.kubernetes.io/exclude-from-external-load-balancers":{}}}} } {aws-cloud-controller-manager Update v1 2023-03-11 16:21:50 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-03-11 16:21:50 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kube-controller-manager Update v1 2023-03-11 16:21:58 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.0.0/24\"":{}},"f:taints":{}}} } {kubelet Update v1 2023-03-11 16:48:50 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.0.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-northeast-1a/i-0e3467ec552af797e,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/control-plane,Value:,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[100.96.0.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{4 0} {<nil>} 4 DecimalSI},ephemeral-storage: {{49756430336 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-32Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-64Ki: {{0 0} {<nil>} 0 DecimalSI},memory: {{8150110208 0} {<nil>} 7959092Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{4 0} {<nil>} 4 DecimalSI},ephemeral-storage: {{44780787229 0} {<nil>} 44780787229 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-32Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-64Ki: {{0 0} {<nil>} 0 DecimalSI},memory: {{8045252608 0} {<nil>} 7856692Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-03-11 16:48:50 +0000 UTC,LastTransitionTime:2023-03-11 16:20:46 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-03-11 16:48:50 +0000 UTC,LastTransitionTime:2023-03-11 16:20:46 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-03-11 16:48:50 +0000 UTC,LastTransitionTime:2023-03-11 16:20:46 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-03-11 16:48:50 +0000 UTC,LastTransitionTime:2023-03-11 16:21:58 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.40.18,},NodeAddress{Type:ExternalIP,Address:52.194.213.7,},NodeAddress{Type:InternalDNS,Address:i-0e3467ec552af797e.ap-northeast-1.compute.internal,},NodeAddress{Type:Hostname,Address:i-0e3467ec552af797e.ap-northeast-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-52-194-213-7.ap-northeast-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2abf2e564965592c7716accd56507e,SystemUUID:ec2abf2e-5649-6559-2c77-16accd56507e,BootID:fbada3ff-8655-404c-a98d-876577d4c322,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 22.04.2 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.27.0-alpha.3.420+bb6c6fad2c19b3,KubeProxyVersion:v1.27.0-alpha.3.420+bb6c6fad2c19b3,OperatingSystem:linux,Architecture:arm64,},Images:[]ContainerImage{ContainerImage{Names:[registry.k8s.io/etcd@sha256:9ce33ba33d8e738a5b85ed50b5080ac746deceed4a7496c550927a7a19ca3b6d registry.k8s.io/etcd:3.5.0-0],SizeBytes:157797353,},ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:157636062,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:4afb99b4690b418ffc2ceb67e1a17376457e441c1f09ab55447f0aaf992fa646 registry.k8s.io/etcd:3.4.3-0],SizeBytes:151677080,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:4ad90a11b55313b182afc186b9876c8e891531b8db4c9bf1541953021618d0e2 registry.k8s.io/etcd:3.4.13-0],SizeBytes:134531559,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:0d2ab6b15b4fe0b903b9e81192d942fb39c29be61fe341cd16db5d0165b0435c registry.k8s.io/etcd:3.3.17-0],SizeBytes:126979107,},ContainerImage{Names:[registry.k8s.io/kube-apiserver-arm64:v1.27.0-alpha.3.420_bb6c6fad2c19b3],SizeBytes:115123555,},ContainerImage{Names:[registry.k8s.io/kube-controller-manager-arm64:v1.27.0-alpha.3.420_bb6c6fad2c19b3],SizeBytes:107914235,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:17da501f5d2a675be46040422a27b7cc21b8a43895ac998b171db1c346f361f7 registry.k8s.io/etcd:3.3.10-0],SizeBytes:81436097,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:6f72b851544986cb0921b53ea655ec04c36131248f16d4ad110cb3ca0c369dc1 registry.k8s.io/etcd:3.5.4-0],SizeBytes:81120118,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:13f53ed1d91e2e11aac476ee9a0269fdda6cc4874eba903efd40daf50c55eee5 registry.k8s.io/etcd:3.5.3-0],SizeBytes:81114891,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:51eae8381dcb1078289fa7b4f3df2630cdc18d09fb56f8e56b41c40e191d6c83 registry.k8s.io/etcd:3.5.7-0],SizeBytes:80665728,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:dd75ec974b0a2a6f6bb47001ba09207976e625db898d1b16735528c009cb171c registry.k8s.io/etcd:3.5.6-0],SizeBytes:80539316,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:905d7ca17fd02bc24c0eba9a062753aba15db3e31422390bc3238eb762339b20 registry.k8s.io/etcd:3.2.24-1],SizeBytes:61500433,},ContainerImage{Names:[registry.k8s.io/kube-proxy-arm64:v1.27.0-alpha.3.420_bb6c6fad2c19b3],SizeBytes:60947298,},ContainerImage{Names:[registry.k8s.io/etcdadm/etcd-manager-slim@sha256:8727adc059334e0e3329dbccce116a734da4a090da5e86c8cddccaf333caa666 registry.k8s.io/etcdadm/etcd-manager-slim:v3.0.20230201],SizeBytes:60826655,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:64b9ea357325d5db9f8a723dcf503b5a449177b17ac87d69481e126bb724c263 registry.k8s.io/etcd:3.5.1-0],SizeBytes:59553904,},ContainerImage{Names:[registry.k8s.io/kube-scheduler-arm64:v1.27.0-alpha.3.420_bb6c6fad2c19b3],SizeBytes:56731120,},ContainerImage{Names:[registry.k8s.io/kops/kops-controller:1.27.0-alpha.1],SizeBytes:41885191,},ContainerImage{Names:[registry.k8s.io/kops/dns-controller:1.27.0-alpha.1],SizeBytes:40376038,},ContainerImage{Names:[public.ecr.aws/karpenter/controller@sha256:68db4f092cf9cc83f5ef9e2fbc5407c2cb682e81f64dfaa700a7602ede38b1cf public.ecr.aws/karpenter/controller:v0.16.3],SizeBytes:33707751,},ContainerImage{Names:[public.ecr.aws/karpenter/webhook@sha256:96a2d9b06d6bc5127801f358f74b1cf2d289b423a2e9ba40c573c0b14b17dafa public.ecr.aws/karpenter/webhook:v0.16.3],SizeBytes:33307187,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:28932904,},ContainerImage{Names:[quay.io/cilium/operator@sha256:a6d24a006a6b92967ac90786b49bc1ac26e5477cf028cd1186efcfc2466484db quay.io/cilium/operator:v1.12.5],SizeBytes:24567287,},ContainerImage{Names:[registry.k8s.io/provider-aws/cloud-controller-manager@sha256:fdeb61e3e42ecd9cca868d550ebdb88dd6341d9e91fcfa9a37e227dab2ad22cb registry.k8s.io/provider-aws/cloud-controller-manager:v1.26.0],SizeBytes:18345226,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:8502345,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:7685305,},ContainerImage{Names:[registry.k8s.io/kops/kube-apiserver-healthcheck:1.27.0-alpha.1],SizeBytes:4740762,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:253553,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Mar 11 16:53:22.819: INFO: Logging kubelet events for node i-0e3467ec552af797e Mar 11 16:53:22.973: INFO: Logging pods the kubelet thinks is on node i-0e3467ec552af797e Mar 11 16:53:23.130: INFO: kube-controller-manager-i-0e3467ec552af797e started at 2023-03-11 16:18:44 +0000 UTC (0+1 container statuses recorded) Mar 11 16:53:23.130: INFO: Container kube-controller-manager ready: true, restart count 4 Mar 11 16:53:23.130: INFO: etcd-manager-events-i-0e3467ec552af797e started at 2023-03-11 16:18:44 +0000 UTC (11+1 container statuses recorded) Mar 11 16:53:23.130: INFO: Init container init-etcd-3-2-24 ready: true, restart count 0 Mar 11 16:53:23.130: INFO: Init container init-etcd-3-3-10 ready: true, restart count 0 Mar 11 16:53:23.130: INFO: Init container init-etcd-3-3-17 ready: true, restart count 0 Mar 11 16:53:23.130: INFO: Init container init-etcd-3-4-13 ready: true, restart count 0 Mar 11 16:53:23.130: INFO: Init container init-etcd-3-4-3 ready: true, restart count 0 Mar 11 16:53:23.130: INFO: Init container init-etcd-3-5-0 ready: true, restart count 0 Mar 11 16:53:23.130: INFO: Init container init-etcd-3-5-1 ready: true, restart count 0 Mar 11 16:53:23.130: INFO: Init container init-etcd-3-5-3 ready: true, restart count 0 Mar 11 16:53:23.130: INFO: Init container init-etcd-3-5-4 ready: true, restart count 0 Mar 11 16:53:23.130: INFO: Init container init-etcd-3-5-6 ready: true, restart count 0 Mar 11 16:53:23.130: INFO: Init container init-etcd-3-5-7 ready: true, restart count 0 Mar 11 16:53:23.130: INFO: Container etcd-manager ready: true, restart count 0 Mar 11 16:53:23.130: INFO: etcd-manager-main-i-0e3467ec552af797e started at 2023-03-11 16:18:44 +0000 UTC (11+1 container statuses recorded) Mar 11 16:53:23.130: INFO: Init container init-etcd-3-2-24 ready: true, restart count 0 Mar 11 16:53:23.130: INFO: Init container init-etcd-3-3-10 ready: true, restart count 0 Mar 11 16:53:23.130: INFO: Init container init-etcd-3-3-17 ready: true, restart count 0 Mar 11 16:53:23.130: INFO: Init container init-etcd-3-4-13 ready: true, restart count 0 Mar 11 16:53:23.130: INFO: Init container init-etcd-3-4-3 ready: true, restart count 0 Mar 11 16:53:23.130: INFO: Init container init-etcd-3-5-0 ready: true, restart count 0 Mar 11 16:53:23.130: INFO: Init container init-etcd-3-5-1 ready: true, restart count 0 Mar 11 16:53:23.130: INFO: Init container init-etcd-3-5-3 ready: true, restart count 0 Mar 11 16:53:23.130: INFO: Init container init-etcd-3-5-4 ready: true, restart count 0 Mar 11 16:53:23.130: INFO: Init container init-etcd-3-5-6 ready: true, restart count 0 Mar 11 16:53:23.130: INFO: Init container init-etcd-3-5-7 ready: true, restart count 0 Mar 11 16:53:23.130: INFO: Container etcd-manager ready: true, restart count 0 Mar 11 16:53:23.130: INFO: ebs-csi-node-lpwz9 started at 2023-03-11 16:21:37 +0000 UTC (0+3 container statuses recorded) Mar 11 16:53:23.130: INFO: Container ebs-plugin ready: true, restart count 0 Mar 11 16:53:23.130: INFO: Container liveness-probe ready: true, restart count 0 Mar 11 16:53:23.130: INFO: Container node-driver-registrar ready: true, restart count 0 Mar 11 16:53:23.130: INFO: karpenter-webhook-5b646cd4f7-6sg2k started at 2023-03-11 16:21:58 +0000 UTC (0+1 container statuses recorded) Mar 11 16:53:23.130: INFO: Container webhook ready: true, restart count 0 Mar 11 16:53:23.130: INFO: karpenter-controller-567749677d-d9szs started at 2023-03-11 16:21:58 +0000 UTC (0+1 container statuses recorded) Mar 11 16:53:23.130: INFO: Container controller ready: true, restart count 0 Mar 11 16:53:23.130: INFO: kube-apiserver-i-0e3467ec552af797e started at 2023-03-11 16:18:44 +0000 UTC (0+2 container statuses recorded) Mar 11 16:53:23.130: INFO: Container healthcheck ready: true, restart count 0 Mar 11 16:53:23.130: INFO: Container kube-apiserver ready: true, restart count 3 Mar 11 16:53:23.130: INFO: kube-scheduler-i-0e3467ec552af797e started at 2023-03-11 16:18:44 +0000 UTC (0+1 container statuses recorded) Mar 11 16:53:23.130: INFO: Container kube-scheduler ready: true, restart count 0 Mar 11 16:53:23.130: INFO: dns-controller-7dfd64d999-q9hh7 started at 2023-03-11 16:21:37 +0000 UTC (0+1 container statuses recorded) Mar 11 16:53:23.130: INFO: Container dns-controller ready: true, restart count 0 Mar 11 16:53:23.130: INFO: cilium-qrsfn started at 2023-03-11 16:21:37 +0000 UTC (1+1 container statuses recorded) Mar 11 16:53:23.130: INFO: Init container clean-cilium-state ready: true, restart count 0 Mar 11 16:53:23.130: INFO: Container cilium-agent ready: true, restart count 0 Mar 11 16:53:23.130: INFO: cilium-operator-8496cc65f5-qhdg8 started at 2023-03-11 16:21:37 +0000 UTC (0+1 container statuses recorded) Mar 11 16:53:23.130: INFO: Container cilium-operator ready: true, restart count 0 Mar 11 16:53:23.130: INFO: kops-controller-76gkw started at 2023-03-11 16:21:37 +0000 UTC (0+1 container statuses recorded) Mar 11 16:53:23.130: INFO: Container kops-controller ready: true, restart count 0 Mar 11 16:53:23.130: INFO: aws-cloud-controller-manager-kwbw2 started at 2023-03-11 16:21:37 +0000 UTC (0+1 container statuses recorded) Mar 11 16:53:23.130: INFO: Container aws-cloud-controller-manager ready: true, restart count 0 Mar 11 16:53:23.789: INFO: Latency metrics for node i-0e3467ec552af797e END STEP: dump namespace information after failure - test/e2e/framework/framework.go:288 @ 03/11/23 16:53:23.789 (53.239s) < Exit [DeferCleanup (Each)] [sig-network] Services - dump namespaces | framework.go:209 @ 03/11/23 16:53:23.789 (53.239s) > Enter [DeferCleanup (Each)] [sig-network] Services - tear down framework | framework.go:206 @ 03/11/23 16:53:23.789 STEP: Destroying namespace "services-6839" for this suite. - test/e2e/framework/framework.go:351 @ 03/11/23 16:53:23.789 < Exit [DeferCleanup (Each)] [sig-network] Services - tear down framework | framework.go:206 @ 03/11/23 16:53:23.94 (151ms) > Enter [ReportAfterEach] TOP-LEVEL - test/e2e/e2e_test.go:144 @ 03/11/23 16:53:23.94 < Exit [ReportAfterEach] TOP-LEVEL - test/e2e/e2e_test.go:144 @ 03/11/23 16:53:23.94 (0s)
Filter through log files | View test history on testgrid
go run hack/e2e.go -v --test --test_args='--ginkgo.focus=Kubernetes\se2e\ssuite\s\[It\]\s\[sig\-network\]\sServices\sshould\shave\ssession\saffinity\swork\sfor\sservice\swith\stype\sclusterIP\s\[LinuxOnly\]\s\[Conformance\]$'
[FAILED] DeferCleanup callback returned error: error while waiting for pods gone affinity-clusterip: there are 3 pods left. E.g. "affinity-clusterip-f8cft" on node "i-04523d1dffb0fe725" In [DeferCleanup (Each)] at: test/e2e/network/service.go:3968 @ 03/11/23 17:04:36.3from junit_01.xml
> Enter [BeforeEach] [sig-network] Services - set up framework | framework.go:191 @ 03/11/23 16:44:12.758 STEP: Creating a kubernetes client - test/e2e/framework/framework.go:211 @ 03/11/23 16:44:12.758 Mar 11 16:44:12.758: INFO: >>> kubeConfig: /root/.kube/config STEP: Building a namespace api object, basename services - test/e2e/framework/framework.go:250 @ 03/11/23 16:44:12.759 STEP: Waiting for a default service account to be provisioned in namespace - test/e2e/framework/framework.go:259 @ 03/11/23 16:44:13.206 STEP: Waiting for kube-root-ca.crt to be provisioned in namespace - test/e2e/framework/framework.go:262 @ 03/11/23 16:44:13.502 < Exit [BeforeEach] [sig-network] Services - set up framework | framework.go:191 @ 03/11/23 16:44:13.798 (1.04s) > Enter [BeforeEach] [sig-network] Services - test/e2e/framework/metrics/init/init.go:33 @ 03/11/23 16:44:13.798 < Exit [BeforeEach] [sig-network] Services - test/e2e/framework/metrics/init/init.go:33 @ 03/11/23 16:44:13.798 (0s) > Enter [BeforeEach] [sig-network] Services - test/e2e/network/service.go:764 @ 03/11/23 16:44:13.798 < Exit [BeforeEach] [sig-network] Services - test/e2e/network/service.go:764 @ 03/11/23 16:44:13.798 (0s) > Enter [It] should have session affinity work for service with type clusterIP [LinuxOnly] [Conformance] - test/e2e/network/service.go:2165 @ 03/11/23 16:44:13.798 STEP: creating service in namespace services-4707 - test/e2e/network/service.go:3963 @ 03/11/23 16:44:13.798 STEP: creating service affinity-clusterip in namespace services-4707 - test/e2e/network/service.go:270 @ 03/11/23 16:44:13.798 STEP: creating replication controller affinity-clusterip in namespace services-4707 - test/e2e/framework/rc/rc_utils.go:88 @ 03/11/23 16:44:13.953 I0311 16:44:14.103601 7304 runners.go:194] Created replication controller with name: affinity-clusterip, namespace: services-4707, replica count: 3 I0311 16:44:17.255751 7304 runners.go:194] affinity-clusterip Pods: 3 out of 3 created, 3 running, 0 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady Mar 11 16:44:17.552: INFO: Creating new exec pod Mar 11 16:44:21.002: INFO: Running '/home/prow/go/src/k8s.io/kops/_rundir/c9b7e3a4-c027-11ed-9d91-c209e83e3776/kubectl --server=https://api.e2e-e2e-kops-aws-karpenter.test-cncf-aws.k8s.io --kubeconfig=/root/.kube/config --namespace=services-4707 exec execpod-affinity9xxf5 -- /bin/sh -x -c echo hostName | nc -v -t -w 2 affinity-clusterip 80' Mar 11 16:44:22.512: INFO: stderr: "+ echo hostName\n+ nc -v -t -w 2 affinity-clusterip 80\nConnection to affinity-clusterip 80 port [tcp/http] succeeded!\n" Mar 11 16:44:22.512: INFO: stdout: "HTTP/1.1 400 Bad Request\r\nContent-Type: text/plain; charset=utf-8\r\nConnection: close\r\n\r\n400 Bad Request" Mar 11 16:44:22.512: INFO: Running '/home/prow/go/src/k8s.io/kops/_rundir/c9b7e3a4-c027-11ed-9d91-c209e83e3776/kubectl --server=https://api.e2e-e2e-kops-aws-karpenter.test-cncf-aws.k8s.io --kubeconfig=/root/.kube/config --namespace=services-4707 exec execpod-affinity9xxf5 -- /bin/sh -x -c echo hostName | nc -v -t -w 2 100.64.196.240 80' Mar 11 16:44:24.543: INFO: stderr: "+ nc -v -t -w 2 100.64.196.240 80\nConnection to 100.64.196.240 80 port [tcp/http] succeeded!\n+ echo hostName\n" Mar 11 16:44:24.543: INFO: stdout: "HTTP/1.1 400 Bad Request\r\nContent-Type: text/plain; charset=utf-8\r\nConnection: close\r\n\r\n400 Bad Request" Mar 11 16:44:24.543: INFO: Running '/home/prow/go/src/k8s.io/kops/_rundir/c9b7e3a4-c027-11ed-9d91-c209e83e3776/kubectl --server=https://api.e2e-e2e-kops-aws-karpenter.test-cncf-aws.k8s.io --kubeconfig=/root/.kube/config --namespace=services-4707 exec execpod-affinity9xxf5 -- /bin/sh -x -c for i in $(seq 0 15); do echo; curl -q -s --connect-timeout 2 http://100.64.196.240:80/ ; done' Mar 11 16:44:28.386: INFO: stderr: "+ seq 0 15\n+ echo\n+ curl -q -s --connect-timeout 2 http://100.64.196.240:80/\n+ echo\n+ curl -q -s --connect-timeout 2 http://100.64.196.240:80/\n+ echo\n+ curl -q -s --connect-timeout 2 http://100.64.196.240:80/\n+ echo\n+ curl -q -s --connect-timeout 2 http://100.64.196.240:80/\n+ echo\n+ curl -q -s --connect-timeout 2 http://100.64.196.240:80/\n+ echo\n+ curl -q -s --connect-timeout 2 http://100.64.196.240:80/\n+ echo\n+ curl -q -s --connect-timeout 2 http://100.64.196.240:80/\n+ echo\n+ curl -q -s --connect-timeout 2 http://100.64.196.240:80/\n+ echo\n+ curl -q -s --connect-timeout 2 http://100.64.196.240:80/\n+ echo\n+ curl -q -s --connect-timeout 2 http://100.64.196.240:80/\n+ echo\n+ curl -q -s --connect-timeout 2 http://100.64.196.240:80/\n+ echo\n+ curl -q -s --connect-timeout 2 http://100.64.196.240:80/\n+ echo\n+ curl -q -s --connect-timeout 2 http://100.64.196.240:80/\n+ echo\n+ curl -q -s --connect-timeout 2 http://100.64.196.240:80/\n+ echo\n+ curl -q -s --connect-timeout 2 http://100.64.196.240:80/\n+ echo\n+ curl -q -s --connect-timeout 2 http://100.64.196.240:80/\n" Mar 11 16:44:28.386: INFO: stdout: "\naffinity-clusterip-f8cft\naffinity-clusterip-f8cft\naffinity-clusterip-f8cft\naffinity-clusterip-f8cft\naffinity-clusterip-f8cft\naffinity-clusterip-f8cft\naffinity-clusterip-f8cft\naffinity-clusterip-f8cft\naffinity-clusterip-f8cft\naffinity-clusterip-f8cft\naffinity-clusterip-f8cft\naffinity-clusterip-f8cft\naffinity-clusterip-f8cft\naffinity-clusterip-f8cft\naffinity-clusterip-f8cft\naffinity-clusterip-f8cft" Mar 11 16:44:28.386: INFO: Received response from host: affinity-clusterip-f8cft Mar 11 16:44:28.386: INFO: Received response from host: affinity-clusterip-f8cft Mar 11 16:44:28.386: INFO: Received response from host: affinity-clusterip-f8cft Mar 11 16:44:28.386: INFO: Received response from host: affinity-clusterip-f8cft Mar 11 16:44:28.386: INFO: Received response from host: affinity-clusterip-f8cft Mar 11 16:44:28.386: INFO: Received response from host: affinity-clusterip-f8cft Mar 11 16:44:28.386: INFO: Received response from host: affinity-clusterip-f8cft Mar 11 16:44:28.386: INFO: Received response from host: affinity-clusterip-f8cft Mar 11 16:44:28.386: INFO: Received response from host: affinity-clusterip-f8cft Mar 11 16:44:28.386: INFO: Received response from host: affinity-clusterip-f8cft Mar 11 16:44:28.386: INFO: Received response from host: affinity-clusterip-f8cft Mar 11 16:44:28.386: INFO: Received response from host: affinity-clusterip-f8cft Mar 11 16:44:28.386: INFO: Received response from host: affinity-clusterip-f8cft Mar 11 16:44:28.386: INFO: Received response from host: affinity-clusterip-f8cft Mar 11 16:44:28.386: INFO: Received response from host: affinity-clusterip-f8cft Mar 11 16:44:28.386: INFO: Received response from host: affinity-clusterip-f8cft < Exit [It] should have session affinity work for service with type clusterIP [LinuxOnly] [Conformance] - test/e2e/network/service.go:2165 @ 03/11/23 16:44:28.386 (14.588s) > Enter [AfterEach] [sig-network] Services - test/e2e/framework/node/init/init.go:33 @ 03/11/23 16:44:28.386 Mar 11 16:44:28.386: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready < Exit [AfterEach] [sig-network] Services - test/e2e/framework/node/init/init.go:33 @ 03/11/23 16:44:28.539 (153ms) > Enter [DeferCleanup (Each)] [sig-network] Services - test/e2e/network/service.go:3989 @ 03/11/23 16:44:28.539 Mar 11 16:44:28.539: INFO: Cleaning up the exec pod < Exit [DeferCleanup (Each)] [sig-network] Services - test/e2e/network/service.go:3989 @ 03/11/23 16:44:28.845 (307ms) > Enter [DeferCleanup (Each)] [sig-network] Services - test/e2e/network/service.go:3968 @ 03/11/23 16:44:28.845 STEP: deleting ReplicationController affinity-clusterip in namespace services-4707, will wait for the garbage collector to delete the pods - test/e2e/framework/resource/resources.go:70 @ 03/11/23 16:44:28.845 Mar 11 16:44:29.299: INFO: Deleting ReplicationController affinity-clusterip took: 153.216009ms Mar 11 16:44:36.300: INFO: Terminating ReplicationController affinity-clusterip pods took: 7.000762644s Mar 11 17:04:36.300: INFO: ERROR: Pod "affinity-clusterip-f8cft" still exists. Node: "i-04523d1dffb0fe725" Mar 11 17:04:36.300: INFO: ERROR: Pod "affinity-clusterip-lqkg5" still exists. Node: "i-04523d1dffb0fe725" Mar 11 17:04:36.300: INFO: ERROR: Pod "affinity-clusterip-x7nb2" still exists. Node: "i-07e2b6e650d7ef56d" [FAILED] DeferCleanup callback returned error: error while waiting for pods gone affinity-clusterip: there are 3 pods left. E.g. "affinity-clusterip-f8cft" on node "i-04523d1dffb0fe725" In [DeferCleanup (Each)] at: test/e2e/network/service.go:3968 @ 03/11/23 17:04:36.3 < Exit [DeferCleanup (Each)] [sig-network] Services - test/e2e/network/service.go:3968 @ 03/11/23 17:04:36.3 (20m7.455s) > Enter [DeferCleanup (Each)] [sig-network] Services - test/e2e/framework/metrics/init/init.go:35 @ 03/11/23 17:04:36.3 < Exit [DeferCleanup (Each)] [sig-network] Services - test/e2e/framework/metrics/init/init.go:35 @ 03/11/23 17:04:36.3 (0s) > Enter [DeferCleanup (Each)] [sig-network] Services - dump namespaces | framework.go:209 @ 03/11/23 17:04:36.3 STEP: dump namespace information after failure - test/e2e/framework/framework.go:288 @ 03/11/23 17:04:36.3 STEP: Collecting events from namespace "services-4707". - test/e2e/framework/debug/dump.go:42 @ 03/11/23 17:04:36.3 STEP: Found 29 events. - test/e2e/framework/debug/dump.go:46 @ 03/11/23 17:04:36.45 Mar 11 17:04:36.450: INFO: At 2023-03-11 16:44:14 +0000 UTC - event for affinity-clusterip: {replication-controller } SuccessfulCreate: Created pod: affinity-clusterip-f8cft Mar 11 17:04:36.450: INFO: At 2023-03-11 16:44:14 +0000 UTC - event for affinity-clusterip: {replication-controller } SuccessfulCreate: Created pod: affinity-clusterip-x7nb2 Mar 11 17:04:36.450: INFO: At 2023-03-11 16:44:14 +0000 UTC - event for affinity-clusterip: {replication-controller } SuccessfulCreate: Created pod: affinity-clusterip-lqkg5 Mar 11 17:04:36.450: INFO: At 2023-03-11 16:44:14 +0000 UTC - event for affinity-clusterip-f8cft: {default-scheduler } Scheduled: Successfully assigned services-4707/affinity-clusterip-f8cft to i-04523d1dffb0fe725 Mar 11 17:04:36.450: INFO: At 2023-03-11 16:44:14 +0000 UTC - event for affinity-clusterip-f8cft: {kubelet i-04523d1dffb0fe725} Pulled: Container image "registry.k8s.io/e2e-test-images/agnhost:2.43" already present on machine Mar 11 17:04:36.450: INFO: At 2023-03-11 16:44:14 +0000 UTC - event for affinity-clusterip-f8cft: {kubelet i-04523d1dffb0fe725} Created: Created container affinity-clusterip Mar 11 17:04:36.450: INFO: At 2023-03-11 16:44:14 +0000 UTC - event for affinity-clusterip-f8cft: {kubelet i-04523d1dffb0fe725} Started: Started container affinity-clusterip Mar 11 17:04:36.450: INFO: At 2023-03-11 16:44:14 +0000 UTC - event for affinity-clusterip-lqkg5: {default-scheduler } Scheduled: Successfully assigned services-4707/affinity-clusterip-lqkg5 to i-04523d1dffb0fe725 Mar 11 17:04:36.450: INFO: At 2023-03-11 16:44:14 +0000 UTC - event for affinity-clusterip-lqkg5: {kubelet i-04523d1dffb0fe725} Pulled: Container image "registry.k8s.io/e2e-test-images/agnhost:2.43" already present on machine Mar 11 17:04:36.450: INFO: At 2023-03-11 16:44:14 +0000 UTC - event for affinity-clusterip-lqkg5: {kubelet i-04523d1dffb0fe725} Created: Created container affinity-clusterip Mar 11 17:04:36.450: INFO: At 2023-03-11 16:44:14 +0000 UTC - event for affinity-clusterip-lqkg5: {kubelet i-04523d1dffb0fe725} Started: Started container affinity-clusterip Mar 11 17:04:36.450: INFO: At 2023-03-11 16:44:14 +0000 UTC - event for affinity-clusterip-x7nb2: {kubelet i-07e2b6e650d7ef56d} Started: Started container affinity-clusterip Mar 11 17:04:36.450: INFO: At 2023-03-11 16:44:14 +0000 UTC - event for affinity-clusterip-x7nb2: {default-scheduler } Scheduled: Successfully assigned services-4707/affinity-clusterip-x7nb2 to i-07e2b6e650d7ef56d Mar 11 17:04:36.450: INFO: At 2023-03-11 16:44:14 +0000 UTC - event for affinity-clusterip-x7nb2: {kubelet i-07e2b6e650d7ef56d} Pulled: Container image "registry.k8s.io/e2e-test-images/agnhost:2.43" already present on machine Mar 11 17:04:36.450: INFO: At 2023-03-11 16:44:14 +0000 UTC - event for affinity-clusterip-x7nb2: {kubelet i-07e2b6e650d7ef56d} Created: Created container affinity-clusterip Mar 11 17:04:36.450: INFO: At 2023-03-11 16:44:17 +0000 UTC - event for execpod-affinity9xxf5: {default-scheduler } Scheduled: Successfully assigned services-4707/execpod-affinity9xxf5 to i-04523d1dffb0fe725 Mar 11 17:04:36.450: INFO: At 2023-03-11 16:44:18 +0000 UTC - event for execpod-affinity9xxf5: {kubelet i-04523d1dffb0fe725} Started: Started container agnhost-container Mar 11 17:04:36.450: INFO: At 2023-03-11 16:44:18 +0000 UTC - event for execpod-affinity9xxf5: {kubelet i-04523d1dffb0fe725} Created: Created container agnhost-container Mar 11 17:04:36.450: INFO: At 2023-03-11 16:44:18 +0000 UTC - event for execpod-affinity9xxf5: {kubelet i-04523d1dffb0fe725} Pulled: Container image "registry.k8s.io/e2e-test-images/agnhost:2.43" already present on machine Mar 11 17:04:36.450: INFO: At 2023-03-11 16:44:28 +0000 UTC - event for execpod-affinity9xxf5: {kubelet i-04523d1dffb0fe725} Killing: Stopping container agnhost-container Mar 11 17:04:36.450: INFO: At 2023-03-11 16:44:36 +0000 UTC - event for affinity-clusterip-f8cft: {kubelet i-04523d1dffb0fe725} Killing: Stopping container affinity-clusterip Mar 11 17:04:36.450: INFO: At 2023-03-11 16:44:36 +0000 UTC - event for affinity-clusterip-lqkg5: {kubelet i-04523d1dffb0fe725} Killing: Stopping container affinity-clusterip Mar 11 17:04:36.450: INFO: At 2023-03-11 16:44:36 +0000 UTC - event for affinity-clusterip-x7nb2: {kubelet i-07e2b6e650d7ef56d} Killing: Stopping container affinity-clusterip Mar 11 17:04:36.450: INFO: At 2023-03-11 16:45:31 +0000 UTC - event for affinity-clusterip-f8cft: {node-controller } NodeNotReady: Node is not ready Mar 11 17:04:36.450: INFO: At 2023-03-11 16:45:31 +0000 UTC - event for affinity-clusterip-lqkg5: {node-controller } NodeNotReady: Node is not ready Mar 11 17:04:36.450: INFO: At 2023-03-11 16:45:36 +0000 UTC - event for affinity-clusterip-x7nb2: {node-controller } NodeNotReady: Node is not ready Mar 11 17:04:36.450: INFO: At 2023-03-11 16:50:36 +0000 UTC - event for affinity-clusterip-f8cft: {taint-controller } TaintManagerEviction: Marking for deletion Pod services-4707/affinity-clusterip-f8cft Mar 11 17:04:36.450: INFO: At 2023-03-11 16:50:36 +0000 UTC - event for affinity-clusterip-lqkg5: {taint-controller } TaintManagerEviction: Marking for deletion Pod services-4707/affinity-clusterip-lqkg5 Mar 11 17:04:36.450: INFO: At 2023-03-11 16:50:46 +0000 UTC - event for affinity-clusterip-x7nb2: {taint-controller } TaintManagerEviction: Marking for deletion Pod services-4707/affinity-clusterip-x7nb2 Mar 11 17:04:36.599: INFO: POD NODE PHASE GRACE CONDITIONS Mar 11 17:04:36.599: INFO: affinity-clusterip-f8cft i-04523d1dffb0fe725 Running 1s [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2023-03-11 16:44:14 +0000 UTC } {Ready False 0001-01-01 00:00:00 +0000 UTC 2023-03-11 16:45:31 +0000 UTC } {ContainersReady True 0001-01-01 00:00:00 +0000 UTC 2023-03-11 16:44:15 +0000 UTC } {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2023-03-11 16:44:14 +0000 UTC } {DisruptionTarget True 0001-01-01 00:00:00 +0000 UTC 2023-03-11 16:50:36 +0000 UTC DeletionByTaintManager Taint manager: deleting due to NoExecute taint}] Mar 11 17:04:36.599: INFO: affinity-clusterip-lqkg5 i-04523d1dffb0fe725 Running 1s [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2023-03-11 16:44:14 +0000 UTC } {Ready False 0001-01-01 00:00:00 +0000 UTC 2023-03-11 16:45:31 +0000 UTC } {ContainersReady True 0001-01-01 00:00:00 +0000 UTC 2023-03-11 16:44:15 +0000 UTC } {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2023-03-11 16:44:14 +0000 UTC } {DisruptionTarget True 0001-01-01 00:00:00 +0000 UTC 2023-03-11 16:50:36 +0000 UTC DeletionByTaintManager Taint manager: deleting due to NoExecute taint}] Mar 11 17:04:36.599: INFO: affinity-clusterip-x7nb2 i-07e2b6e650d7ef56d Running 1s [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2023-03-11 16:44:14 +0000 UTC } {Ready False 0001-01-01 00:00:00 +0000 UTC 2023-03-11 16:45:36 +0000 UTC } {ContainersReady True 0001-01-01 00:00:00 +0000 UTC 2023-03-11 16:44:15 +0000 UTC } {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2023-03-11 16:44:14 +0000 UTC } {DisruptionTarget True 0001-01-01 00:00:00 +0000 UTC 2023-03-11 16:50:46 +0000 UTC DeletionByTaintManager Taint manager: deleting due to NoExecute taint}] Mar 11 17:04:36.599: INFO: Mar 11 17:04:46.750: INFO: Unable to fetch services-4707/affinity-clusterip-f8cft/affinity-clusterip logs: an error on the server ("unknown") has prevented the request from succeeding (get pods affinity-clusterip-f8cft) Mar 11 17:04:56.901: INFO: Unable to fetch services-4707/affinity-clusterip-lqkg5/affinity-clusterip logs: an error on the server ("unknown") has prevented the request from succeeding (get pods affinity-clusterip-lqkg5) Mar 11 17:05:07.052: INFO: Unable to fetch services-4707/affinity-clusterip-x7nb2/affinity-clusterip logs: an error on the server ("unknown") has prevented the request from succeeding (get pods affinity-clusterip-x7nb2) Mar 11 17:05:07.202: INFO: Logging node info for node i-04523d1dffb0fe725 Mar 11 17:05:07.351: INFO: Node Info: &Node{ObjectMeta:{i-04523d1dffb0fe725 5d37640f-a7d9-4c26-86c5-9da21f7ccc18 53266 0 2023-03-11 16:24:20 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:arm64 beta.kubernetes.io/instance-type:t4g.small beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-northeast-1 failure-domain.beta.kubernetes.io/zone:ap-northeast-1a karpenter.k8s.aws/instance-category:t karpenter.k8s.aws/instance-cpu:2 karpenter.k8s.aws/instance-family:t4g karpenter.k8s.aws/instance-generation:4 karpenter.k8s.aws/instance-hypervisor:nitro karpenter.k8s.aws/instance-memory:2048 karpenter.k8s.aws/instance-pods:11 karpenter.k8s.aws/instance-size:small karpenter.sh/capacity-type:spot karpenter.sh/initialized:true karpenter.sh/provisioner-name:nodes kubernetes.io/arch:arm64 kubernetes.io/hostname:i-04523d1dffb0fe725 kubernetes.io/os:linux node-role.kubernetes.io/node: node-role.kubernetes.io/spot-worker:true node.kubernetes.io/instance-type:t4g.small topology.ebs.csi.aws.com/zone:ap-northeast-1a topology.hostpath.csi/node:i-04523d1dffb0fe725 topology.kubernetes.io/region:ap-northeast-1 topology.kubernetes.io/zone:ap-northeast-1a] map[csi.volume.kubernetes.io/nodeid:{"csi-hostpath-read-write-once-pod-7172":"i-04523d1dffb0fe725","csi-hostpath-volume-expand-1673":"i-04523d1dffb0fe725","ebs.csi.aws.com":"i-04523d1dffb0fe725"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [{karpenter.sh/v1alpha5 Provisioner nodes ac7283b2-eed0-4a9f-9ddd-616f33a0991a <nil> 0xc0031ec1f5}] [karpenter.sh/termination] [{aws-cloud-controller-manager Update v1 2023-03-11 16:24:20 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{}}}} } {kops-controller Update v1 2023-03-11 16:24:20 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{},"f:node-role.kubernetes.io/spot-worker":{}}}} } {karpenter Update v1 2023-03-11 16:26:12 +0000 UTC FieldsV1 {"f:metadata":{"f:finalizers":{".":{},"v:\"karpenter.sh/termination\"":{}},"f:labels":{".":{},"f:karpenter.k8s.aws/instance-category":{},"f:karpenter.k8s.aws/instance-cpu":{},"f:karpenter.k8s.aws/instance-family":{},"f:karpenter.k8s.aws/instance-generation":{},"f:karpenter.k8s.aws/instance-hypervisor":{},"f:karpenter.k8s.aws/instance-memory":{},"f:karpenter.k8s.aws/instance-pods":{},"f:karpenter.k8s.aws/instance-size":{},"f:karpenter.sh/capacity-type":{},"f:karpenter.sh/initialized":{},"f:karpenter.sh/provisioner-name":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}},"f:ownerReferences":{".":{},"k:{\"uid\":\"ac7283b2-eed0-4a9f-9ddd-616f33a0991a\"}":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-03-11 16:26:50 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{".":{},"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"InternalIP\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kubelet Update v1 2023-03-11 16:44:47 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{"f:kubernetes.io/hostname":{},"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{".":{},"f:cpu":{},"f:ephemeral-storage":{},"f:hugepages-1Gi":{},"f:hugepages-2Mi":{},"f:hugepages-32Mi":{},"f:hugepages-64Ki":{},"f:memory":{},"f:pods":{}},"f:capacity":{".":{},"f:cpu":{},"f:ephemeral-storage":{},"f:hugepages-1Gi":{},"f:hugepages-2Mi":{},"f:hugepages-32Mi":{},"f:hugepages-64Ki":{},"f:memory":{},"f:pods":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{}}},"f:daemonEndpoints":{"f:kubeletEndpoint":{"f:Port":{}}},"f:images":{},"f:nodeInfo":{"f:architecture":{},"f:bootID":{},"f:containerRuntimeVersion":{},"f:kernelVersion":{},"f:kubeProxyVersion":{},"f:kubeletVersion":{},"f:machineID":{},"f:operatingSystem":{},"f:osImage":{},"f:systemUUID":{}},"f:volumesInUse":{}}} status} {kube-controller-manager Update v1 2023-03-11 16:45:36 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:node.alpha.kubernetes.io/ttl":{}},"f:labels":{"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.2.0/24\"":{}},"f:taints":{}}} } {kube-controller-manager Update v1 2023-03-11 17:04:56 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{".":{},"k:{\"type\":\"DiskPressure\"}":{".":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"MemoryPressure\"}":{".":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"PIDPressure\"}":{".":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Ready\"}":{".":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}},"f:volumesAttached":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.2.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-northeast-1a/i-04523d1dffb0fe725,Unschedulable:false,Taints:[]Taint{Taint{Key:node.kubernetes.io/unreachable,Value:,Effect:NoSchedule,TimeAdded:2023-03-11 16:45:31 +0000 UTC,},Taint{Key:node.kubernetes.io/unreachable,Value:,Effect:NoExecute,TimeAdded:2023-03-11 16:45:36 +0000 UTC,},},ConfigSource:nil,PodCIDRs:[100.96.2.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49756430336 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-32Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-64Ki: {{0 0} {<nil>} 0 DecimalSI},memory: {{1941540864 0} {<nil>} 1896036Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44780787229 0} {<nil>} 44780787229 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-32Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-64Ki: {{0 0} {<nil>} 0 DecimalSI},memory: {{1836683264 0} {<nil>} 1793636Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:Ready,Status:Unknown,LastHeartbeatTime:2023-03-11 16:44:47 +0000 UTC,LastTransitionTime:2023-03-11 16:45:31 +0000 UTC,Reason:NodeStatusUnknown,Message:Kubelet stopped posting node status.,},NodeCondition{Type:MemoryPressure,Status:Unknown,LastHeartbeatTime:2023-03-11 16:44:47 +0000 UTC,LastTransitionTime:2023-03-11 16:45:31 +0000 UTC,Reason:NodeStatusUnknown,Message:Kubelet stopped posting node status.,},NodeCondition{Type:DiskPressure,Status:Unknown,LastHeartbeatTime:2023-03-11 16:44:47 +0000 UTC,LastTransitionTime:2023-03-11 16:45:31 +0000 UTC,Reason:NodeStatusUnknown,Message:Kubelet stopped posting node status.,},NodeCondition{Type:PIDPressure,Status:Unknown,LastHeartbeatTime:2023-03-11 16:44:47 +0000 UTC,LastTransitionTime:2023-03-11 16:45:31 +0000 UTC,Reason:NodeStatusUnknown,Message:Kubelet stopped posting node status.,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.48.168,},NodeAddress{Type:ExternalIP,Address:52.199.253.156,},NodeAddress{Type:InternalDNS,Address:i-04523d1dffb0fe725.ap-northeast-1.compute.internal,},NodeAddress{Type:Hostname,Address:i-04523d1dffb0fe725.ap-northeast-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-52-199-253-156.ap-northeast-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2c8fb0e6b2e9c4af310f65b6161cd8,SystemUUID:ec2c8fb0-e6b2-e9c4-af31-0f65b6161cd8,BootID:a3b8de83-558b-44c4-a46a-d097c3f98225,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 22.04.2 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.27.0-alpha.3.420+bb6c6fad2c19b3,KubeProxyVersion:v1.27.0-alpha.3.420+bb6c6fad2c19b3,OperatingSystem:linux,Architecture:arm64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:157636062,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/jessie-dnsutils@sha256:24aaf2626d6b27864c29de2097e8bbb840b3a414271bf7c8995e431e47d8408e registry.k8s.io/e2e-test-images/jessie-dnsutils:1.7],SizeBytes:116764683,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:51eae8381dcb1078289fa7b4f3df2630cdc18d09fb56f8e56b41c40e191d6c83 registry.k8s.io/etcd:3.5.7-0],SizeBytes:80665728,},ContainerImage{Names:[registry.k8s.io/kube-proxy-arm64:v1.27.0-alpha.3.420_bb6c6fad2c19b3],SizeBytes:60947298,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:16bbf38c463a4223d8cfe4da12bc61010b082a79b4bb003e2d3ba3ece5dd5f9e registry.k8s.io/e2e-test-images/agnhost:2.43],SizeBytes:49808364,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nautilus@sha256:80ba6c8c44f9623f06e868a1aa66026c8ec438ad814f9ec95e9333b415fe3550 registry.k8s.io/e2e-test-images/nautilus:1.7],SizeBytes:47953400,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:3fe7acf013d1264ffded116b80a73dc129a449b0fccdb8d21af8279f2233f36e registry.k8s.io/e2e-test-images/httpd:2.4.39-4],SizeBytes:41528818,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:148b022f5c5da426fc2f3c14b5c0867e58ef05961510c84749ac1fddcb0fef22 registry.k8s.io/e2e-test-images/httpd:2.4.38-4],SizeBytes:40343601,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:28932904,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:ee3b525d5b89db99da3b8eb521d9cd90cb6e9ef0fbb651e98bb37be78d36b5b8 registry.k8s.io/sig-storage/csi-provisioner:v3.3.0],SizeBytes:23945474,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/sample-apiserver@sha256:8d70890151aa5d096f331cb9da1b9cd5be0412b7363fe67b5c3befdcaa2a28d0 registry.k8s.io/e2e-test-images/sample-apiserver:1.17.7],SizeBytes:23857462,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:425d8f1b769398127767b06ed97ce62578a3179bcb99809ce93a1649e025ffe7 registry.k8s.io/sig-storage/csi-resizer:v1.6.0],SizeBytes:22678632,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:291334908ddf71a4661fd7f6d9d97274de8a5378a2b6fdfeb2ce73414a34f82f registry.k8s.io/sig-storage/csi-snapshotter:v6.1.0],SizeBytes:22434180,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:9a685020911e2725ad019dbce6e4a5ab93d51e3d4557f115e64343345e05781b registry.k8s.io/sig-storage/csi-attacher:v4.0.0],SizeBytes:22394501,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:122bfb8c1edabb3c0edd63f06523e6940d958d19b3957dc7b1d6f81e9f1f6119 registry.k8s.io/sig-storage/csi-provisioner:v3.1.0],SizeBytes:21735644,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:9ebbf9f023e7b41ccee3d52afe39a89e3ddacdbb69269d583abfc25847cfd9e4 registry.k8s.io/sig-storage/csi-resizer:v1.4.0],SizeBytes:20848413,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:8b9c313c05f54fb04f8d430896f5f5904b6cb157df261501b29adc04d2b2dc7b registry.k8s.io/sig-storage/csi-attacher:v3.4.0],SizeBytes:20565480,},ContainerImage{Names:[registry.k8s.io/cpa/cluster-proportional-autoscaler@sha256:68d396900aeaa072c1f27289485fdac29834045a6f3ffe369bf389d830ef572d registry.k8s.io/cpa/cluster-proportional-autoscaler:1.8.6],SizeBytes:19007623,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nonroot@sha256:ee9f50b3c64b174d296d91ca9f69a914ac30e59095dfb462b2b518ad28a63655 registry.k8s.io/e2e-test-images/nonroot:1.4],SizeBytes:18561972,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:92257881c1d6493cf18299a24af42330f891166560047902b8d431fb66b01af5 registry.k8s.io/sig-storage/hostpathplugin:v1.9.0],SizeBytes:18267611,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:9b2d9abda017c28b12f84a344f57cd73fbdb6c2bd7dd5bdd5018246ad1093ba6 registry.k8s.io/sig-storage/hostpathplugin:v1.11.0],SizeBytes:18159085,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:017727efcfeb7d053af68e51436ce8e65edbc6ca573720afb4f79c8594036955 registry.k8s.io/coredns/coredns:v1.10.0],SizeBytes:13766234,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:8502345,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:933940f13b3ea0abc62e656c1aa5c5b47c04b15d71250413a6b821bd0c58b94e registry.k8s.io/sig-storage/livenessprobe:v2.7.0],SizeBytes:8223819,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:7685305,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:db048754ae68ae337d8fa96494c96d2a1204c3320f5dcf7e8e71085adec85da6 registry.k8s.io/e2e-test-images/nginx:1.15-4],SizeBytes:6883347,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:5c99cf6a02adda929b10321dbf4ecfa00d87be9ba4fb456006237d530ab4baa1 registry.k8s.io/e2e-test-images/nginx:1.14-4],SizeBytes:6860089,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nonewprivs@sha256:8ac1264691820febacf3aea5d152cbde6d10685731ec14966a9401c6f47a68ac registry.k8s.io/e2e-test-images/nonewprivs:1.3],SizeBytes:5025027,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf registry.k8s.io/e2e-test-images/busybox:1.29-2],SizeBytes:773367,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:2e0f836850e09b8b7cc937681d6194537a09fbd5f6b9e08f4d646a85128e8937 registry.k8s.io/e2e-test-images/busybox:1.29-4],SizeBytes:772933,},ContainerImage{Names:[registry.k8s.io/pause@sha256:7031c1b283388d2c2e09b57badb803c05ebed362dc88d84b480cc47f72a21097 registry.k8s.io/pause:3.9],SizeBytes:268051,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:253553,},},VolumesInUse:[kubernetes.io/csi/csi-hostpath-read-write-once-pod-7172^038fde52-c02c-11ed-b028-5eca9e37a97f kubernetes.io/csi/csi-hostpath-volume-expand-1673^f8193b68-c02b-11ed-8784-9e8522e7cc37 kubernetes.io/csi/ebs.csi.aws.com^vol-0293af09ce81fe75e],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-0d0346dd53f1fd7bb,DevicePath:,},},Config:nil,},} Mar 11 17:05:07.351: INFO: Logging kubelet events for node i-04523d1dffb0fe725 Mar 11 17:05:07.504: INFO: Logging pods the kubelet thinks is on node i-04523d1dffb0fe725 Mar 11 17:05:17.654: INFO: Unable to retrieve kubelet pods for node i-04523d1dffb0fe725: error trying to reach service: net/http: TLS handshake timeout Mar 11 17:05:17.654: INFO: Logging node info for node i-07e2b6e650d7ef56d Mar 11 17:05:17.803: INFO: Node Info: &Node{ObjectMeta:{i-07e2b6e650d7ef56d fcc65272-6f1b-4228-ac59-e9e3e92cdb2c 47965 0 2023-03-11 16:24:19 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:arm64 beta.kubernetes.io/instance-type:t4g.small beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-northeast-1 failure-domain.beta.kubernetes.io/zone:ap-northeast-1a io.kubernetes.storage.mock/node:some-mock-node karpenter.k8s.aws/instance-category:t karpenter.k8s.aws/instance-cpu:2 karpenter.k8s.aws/instance-family:t4g karpenter.k8s.aws/instance-generation:4 karpenter.k8s.aws/instance-hypervisor:nitro karpenter.k8s.aws/instance-memory:2048 karpenter.k8s.aws/instance-pods:11 karpenter.k8s.aws/instance-size:small karpenter.sh/capacity-type:spot karpenter.sh/initialized:true karpenter.sh/provisioner-name:nodes kubernetes.io/arch:arm64 kubernetes.io/hostname:i-07e2b6e650d7ef56d kubernetes.io/os:linux node-role.kubernetes.io/node: node-role.kubernetes.io/spot-worker:true node.kubernetes.io/instance-type:t4g.small topology.ebs.csi.aws.com/zone:ap-northeast-1a topology.hostpath.csi/node:i-07e2b6e650d7ef56d topology.kubernetes.io/region:ap-northeast-1 topology.kubernetes.io/zone:ap-northeast-1a] map[csi.volume.kubernetes.io/nodeid:{"csi-hostpath-ephemeral-7952":"i-07e2b6e650d7ef56d","csi-hostpath-volume-expand-703":"i-07e2b6e650d7ef56d","ebs.csi.aws.com":"i-07e2b6e650d7ef56d"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [{karpenter.sh/v1alpha5 Provisioner nodes ac7283b2-eed0-4a9f-9ddd-616f33a0991a <nil> 0xc00443e5e5}] [karpenter.sh/termination] [{aws-cloud-controller-manager Update v1 2023-03-11 16:24:20 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{}}}} } {kops-controller Update v1 2023-03-11 16:24:20 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{},"f:node-role.kubernetes.io/spot-worker":{}}}} } {karpenter Update v1 2023-03-11 16:26:15 +0000 UTC FieldsV1 {"f:metadata":{"f:finalizers":{".":{},"v:\"karpenter.sh/termination\"":{}},"f:labels":{".":{},"f:karpenter.k8s.aws/instance-category":{},"f:karpenter.k8s.aws/instance-cpu":{},"f:karpenter.k8s.aws/instance-family":{},"f:karpenter.k8s.aws/instance-generation":{},"f:karpenter.k8s.aws/instance-hypervisor":{},"f:karpenter.k8s.aws/instance-memory":{},"f:karpenter.k8s.aws/instance-pods":{},"f:karpenter.k8s.aws/instance-size":{},"f:karpenter.sh/capacity-type":{},"f:karpenter.sh/initialized":{},"f:karpenter.sh/provisioner-name":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}},"f:ownerReferences":{".":{},"k:{\"uid\":\"ac7283b2-eed0-4a9f-9ddd-616f33a0991a\"}":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-03-11 16:26:50 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{".":{},"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"InternalIP\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kubelet Update v1 2023-03-11 16:44:22 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{"f:io.kubernetes.storage.mock/node":{},"f:kubernetes.io/hostname":{},"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{".":{},"f:cpu":{},"f:ephemeral-storage":{},"f:hugepages-1Gi":{},"f:hugepages-2Mi":{},"f:hugepages-32Mi":{},"f:hugepages-64Ki":{},"f:memory":{},"f:pods":{}},"f:capacity":{".":{},"f:cpu":{},"f:ephemeral-storage":{},"f:hugepages-1Gi":{},"f:hugepages-2Mi":{},"f:hugepages-32Mi":{},"f:hugepages-64Ki":{},"f:memory":{},"f:pods":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{}}},"f:daemonEndpoints":{"f:kubeletEndpoint":{"f:Port":{}}},"f:images":{},"f:nodeInfo":{"f:architecture":{},"f:bootID":{},"f:containerRuntimeVersion":{},"f:kernelVersion":{},"f:kubeProxyVersion":{},"f:kubeletVersion":{},"f:machineID":{},"f:operatingSystem":{},"f:osImage":{},"f:systemUUID":{}},"f:volumesInUse":{}}} status} {kube-controller-manager Update v1 2023-03-11 16:45:36 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{".":{},"k:{\"type\":\"DiskPressure\"}":{".":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"MemoryPressure\"}":{".":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"PIDPressure\"}":{".":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Ready\"}":{".":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}},"f:volumesAttached":{}}} status} {kube-controller-manager Update v1 2023-03-11 16:45:46 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:node.alpha.kubernetes.io/ttl":{}},"f:labels":{"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.1.0/24\"":{}},"f:taints":{}}} }]},Spec:NodeSpec{PodCIDR:100.96.1.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-northeast-1a/i-07e2b6e650d7ef56d,Unschedulable:false,Taints:[]Taint{Taint{Key:node.kubernetes.io/unreachable,Value:,Effect:NoSchedule,TimeAdded:2023-03-11 16:45:36 +0000 UTC,},Taint{Key:node.kubernetes.io/unreachable,Value:,Effect:NoExecute,TimeAdded:2023-03-11 16:45:46 +0000 UTC,},},ConfigSource:nil,PodCIDRs:[100.96.1.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49756430336 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-32Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-64Ki: {{0 0} {<nil>} 0 DecimalSI},memory: {{1941549056 0} {<nil>} 1896044Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44780787229 0} {<nil>} 44780787229 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-32Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-64Ki: {{0 0} {<nil>} 0 DecimalSI},memory: {{1836691456 0} {<nil>} 1793644Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:Ready,Status:Unknown,LastHeartbeatTime:2023-03-11 16:44:12 +0000 UTC,LastTransitionTime:2023-03-11 16:45:36 +0000 UTC,Reason:NodeStatusUnknown,Message:Kubelet stopped posting node status.,},NodeCondition{Type:MemoryPressure,Status:Unknown,LastHeartbeatTime:2023-03-11 16:44:12 +0000 UTC,LastTransitionTime:2023-03-11 16:45:36 +0000 UTC,Reason:NodeStatusUnknown,Message:Kubelet stopped posting node status.,},NodeCondition{Type:DiskPressure,Status:Unknown,LastHeartbeatTime:2023-03-11 16:44:12 +0000 UTC,LastTransitionTime:2023-03-11 16:45:36 +0000 UTC,Reason:NodeStatusUnknown,Message:Kubelet stopped posting node status.,},NodeCondition{Type:PIDPressure,Status:Unknown,LastHeartbeatTime:2023-03-11 16:44:12 +0000 UTC,LastTransitionTime:2023-03-11 16:45:36 +0000 UTC,Reason:NodeStatusUnknown,Message:Kubelet stopped posting node status.,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.35.76,},NodeAddress{Type:ExternalIP,Address:54.238.106.83,},NodeAddress{Type:InternalDNS,Address:i-07e2b6e650d7ef56d.ap-northeast-1.compute.internal,},NodeAddress{Type:Hostname,Address:i-07e2b6e650d7ef56d.ap-northeast-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-54-238-106-83.ap-northeast-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2083a8e9d27d02fac967340367f875,SystemUUID:ec2083a8-e9d2-7d02-fac9-67340367f875,BootID:ea264ed6-5b0d-47d6-ad5d-6bd2a1f7c11b,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 22.04.2 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.27.0-alpha.3.420+bb6c6fad2c19b3,KubeProxyVersion:v1.27.0-alpha.3.420+bb6c6fad2c19b3,OperatingSystem:linux,Architecture:arm64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:157636062,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/jessie-dnsutils@sha256:24aaf2626d6b27864c29de2097e8bbb840b3a414271bf7c8995e431e47d8408e registry.k8s.io/e2e-test-images/jessie-dnsutils:1.7],SizeBytes:116764683,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:51eae8381dcb1078289fa7b4f3df2630cdc18d09fb56f8e56b41c40e191d6c83 registry.k8s.io/etcd:3.5.7-0],SizeBytes:80665728,},ContainerImage{Names:[registry.k8s.io/kube-proxy-arm64:v1.27.0-alpha.3.420_bb6c6fad2c19b3],SizeBytes:60947298,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:16bbf38c463a4223d8cfe4da12bc61010b082a79b4bb003e2d3ba3ece5dd5f9e registry.k8s.io/e2e-test-images/agnhost:2.43],SizeBytes:49808364,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nautilus@sha256:80ba6c8c44f9623f06e868a1aa66026c8ec438ad814f9ec95e9333b415fe3550 registry.k8s.io/e2e-test-images/nautilus:1.7],SizeBytes:47953400,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:3fe7acf013d1264ffded116b80a73dc129a449b0fccdb8d21af8279f2233f36e registry.k8s.io/e2e-test-images/httpd:2.4.39-4],SizeBytes:41528818,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:148b022f5c5da426fc2f3c14b5c0867e58ef05961510c84749ac1fddcb0fef22 registry.k8s.io/e2e-test-images/httpd:2.4.38-4],SizeBytes:40343601,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:28932904,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:ee3b525d5b89db99da3b8eb521d9cd90cb6e9ef0fbb651e98bb37be78d36b5b8 registry.k8s.io/sig-storage/csi-provisioner:v3.3.0],SizeBytes:23945474,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/sample-apiserver@sha256:8d70890151aa5d096f331cb9da1b9cd5be0412b7363fe67b5c3befdcaa2a28d0 registry.k8s.io/e2e-test-images/sample-apiserver:1.17.7],SizeBytes:23857462,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:425d8f1b769398127767b06ed97ce62578a3179bcb99809ce93a1649e025ffe7 registry.k8s.io/sig-storage/csi-resizer:v1.6.0],SizeBytes:22678632,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:291334908ddf71a4661fd7f6d9d97274de8a5378a2b6fdfeb2ce73414a34f82f registry.k8s.io/sig-storage/csi-snapshotter:v6.1.0],SizeBytes:22434180,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:9a685020911e2725ad019dbce6e4a5ab93d51e3d4557f115e64343345e05781b registry.k8s.io/sig-storage/csi-attacher:v4.0.0],SizeBytes:22394501,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:122bfb8c1edabb3c0edd63f06523e6940d958d19b3957dc7b1d6f81e9f1f6119 registry.k8s.io/sig-storage/csi-provisioner:v3.1.0],SizeBytes:21735644,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/regression-issue-74839@sha256:b4f1d8d61bdad84bd50442d161d5460e4019d53e989b64220fdbc62fc87d76bf registry.k8s.io/e2e-test-images/regression-issue-74839:1.2],SizeBytes:21331383,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:9ebbf9f023e7b41ccee3d52afe39a89e3ddacdbb69269d583abfc25847cfd9e4 registry.k8s.io/sig-storage/csi-resizer:v1.4.0],SizeBytes:20848413,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:8b9c313c05f54fb04f8d430896f5f5904b6cb157df261501b29adc04d2b2dc7b registry.k8s.io/sig-storage/csi-attacher:v3.4.0],SizeBytes:20565480,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nonroot@sha256:ee9f50b3c64b174d296d91ca9f69a914ac30e59095dfb462b2b518ad28a63655 registry.k8s.io/e2e-test-images/nonroot:1.4],SizeBytes:18561972,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:92257881c1d6493cf18299a24af42330f891166560047902b8d431fb66b01af5 registry.k8s.io/sig-storage/hostpathplugin:v1.9.0],SizeBytes:18267611,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:9b2d9abda017c28b12f84a344f57cd73fbdb6c2bd7dd5bdd5018246ad1093ba6 registry.k8s.io/sig-storage/hostpathplugin:v1.11.0],SizeBytes:18159085,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:017727efcfeb7d053af68e51436ce8e65edbc6ca573720afb4f79c8594036955 registry.k8s.io/coredns/coredns:v1.10.0],SizeBytes:13766234,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:8502345,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:933940f13b3ea0abc62e656c1aa5c5b47c04b15d71250413a6b821bd0c58b94e registry.k8s.io/sig-storage/livenessprobe:v2.7.0],SizeBytes:8223819,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:7685305,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:db048754ae68ae337d8fa96494c96d2a1204c3320f5dcf7e8e71085adec85da6 registry.k8s.io/e2e-test-images/nginx:1.15-4],SizeBytes:6883347,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:5c99cf6a02adda929b10321dbf4ecfa00d87be9ba4fb456006237d530ab4baa1 registry.k8s.io/e2e-test-images/nginx:1.14-4],SizeBytes:6860089,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nonewprivs@sha256:8ac1264691820febacf3aea5d152cbde6d10685731ec14966a9401c6f47a68ac registry.k8s.io/e2e-test-images/nonewprivs:1.3],SizeBytes:5025027,},ContainerImage{Names:[gcr.io/authenticated-image-pulling/alpine@sha256:7ff177862cb50c602bfe81f805969412e619c054a2bbead977d0c276988aa4a0 gcr.io/authenticated-image-pulling/alpine:3.7],SizeBytes:2003505,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf registry.k8s.io/e2e-test-images/busybox:1.29-2],SizeBytes:773367,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:2e0f836850e09b8b7cc937681d6194537a09fbd5f6b9e08f4d646a85128e8937 registry.k8s.io/e2e-test-images/busybox:1.29-4],SizeBytes:772933,},ContainerImage{Names:[registry.k8s.io/pause@sha256:7031c1b283388d2c2e09b57badb803c05ebed362dc88d84b480cc47f72a21097 registry.k8s.io/pause:3.9],SizeBytes:268051,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:253553,},},VolumesInUse:[kubernetes.io/csi/csi-hostpath-ephemeral-7952^eca2344a-c02b-11ed-b483-1e4a85f27a89 kubernetes.io/csi/csi-hostpath-ephemeral-7952^f0ac2401-c02b-11ed-b483-1e4a85f27a89 kubernetes.io/csi/ebs.csi.aws.com^vol-06cba8db23f1489ad],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-06cba8db23f1489ad,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/csi-hostpath-ephemeral-7952^f0ac2401-c02b-11ed-b483-1e4a85f27a89,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/csi-hostpath-ephemeral-7952^eca2344a-c02b-11ed-b483-1e4a85f27a89,DevicePath:,},},Config:nil,},} Mar 11 17:05:17.804: INFO: Logging kubelet events for node i-07e2b6e650d7ef56d Mar 11 17:05:17.957: INFO: Logging pods the kubelet thinks is on node i-07e2b6e650d7ef56d Mar 11 17:05:28.106: INFO: Unable to retrieve kubelet pods for node i-07e2b6e650d7ef56d: error trying to reach service: net/http: TLS handshake timeout Mar 11 17:05:28.106: INFO: Logging node info for node i-0e3467ec552af797e Mar 11 17:05:28.255: INFO: Node Info: &Node{ObjectMeta:{i-0e3467ec552af797e 6c9b0cc5-2c4e-4037-bf31-61bac12a7775 53046 0 2023-03-11 16:20:46 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:arm64 beta.kubernetes.io/instance-type:c6g.xlarge beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-northeast-1 failure-domain.beta.kubernetes.io/zone:ap-northeast-1a kops.k8s.io/kops-controller-pki: kubernetes.io/arch:arm64 kubernetes.io/hostname:i-0e3467ec552af797e kubernetes.io/os:linux node-role.kubernetes.io/control-plane: node.kubernetes.io/exclude-from-external-load-balancers: node.kubernetes.io/instance-type:c6g.xlarge topology.ebs.csi.aws.com/zone:ap-northeast-1a topology.kubernetes.io/region:ap-northeast-1 topology.kubernetes.io/zone:ap-northeast-1a] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-0e3467ec552af797e"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kubelet Update v1 2023-03-11 16:20:46 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {protokube Update v1 2023-03-11 16:21:30 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kops.k8s.io/kops-controller-pki":{},"f:node-role.kubernetes.io/control-plane":{},"f:node.kubernetes.io/exclude-from-external-load-balancers":{}}}} } {aws-cloud-controller-manager Update v1 2023-03-11 16:21:50 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-03-11 16:21:50 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kube-controller-manager Update v1 2023-03-11 16:21:58 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.0.0/24\"":{}},"f:taints":{}}} } {kubelet Update v1 2023-03-11 17:04:07 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.0.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-northeast-1a/i-0e3467ec552af797e,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/control-plane,Value:,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[100.96.0.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{4 0} {<nil>} 4 DecimalSI},ephemeral-storage: {{49756430336 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-32Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-64Ki: {{0 0} {<nil>} 0 DecimalSI},memory: {{8150110208 0} {<nil>} 7959092Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{4 0} {<nil>} 4 DecimalSI},ephemeral-storage: {{44780787229 0} {<nil>} 44780787229 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-32Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-64Ki: {{0 0} {<nil>} 0 DecimalSI},memory: {{8045252608 0} {<nil>} 7856692Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-03-11 17:04:07 +0000 UTC,LastTransitionTime:2023-03-11 16:20:46 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-03-11 17:04:07 +0000 UTC,LastTransitionTime:2023-03-11 16:20:46 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-03-11 17:04:07 +0000 UTC,LastTransitionTime:2023-03-11 16:20:46 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-03-11 17:04:07 +0000 UTC,LastTransitionTime:2023-03-11 16:21:58 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.40.18,},NodeAddress{Type:ExternalIP,Address:52.194.213.7,},NodeAddress{Type:InternalDNS,Address:i-0e3467ec552af797e.ap-northeast-1.compute.internal,},NodeAddress{Type:Hostname,Address:i-0e3467ec552af797e.ap-northeast-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-52-194-213-7.ap-northeast-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2abf2e564965592c7716accd56507e,SystemUUID:ec2abf2e-5649-6559-2c77-16accd56507e,BootID:fbada3ff-8655-404c-a98d-876577d4c322,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 22.04.2 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.27.0-alpha.3.420+bb6c6fad2c19b3,KubeProxyVersion:v1.27.0-alpha.3.420+bb6c6fad2c19b3,OperatingSystem:linux,Architecture:arm64,},Images:[]ContainerImage{ContainerImage{Names:[registry.k8s.io/etcd@sha256:9ce33ba33d8e738a5b85ed50b5080ac746deceed4a7496c550927a7a19ca3b6d registry.k8s.io/etcd:3.5.0-0],SizeBytes:157797353,},ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:157636062,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:4afb99b4690b418ffc2ceb67e1a17376457e441c1f09ab55447f0aaf992fa646 registry.k8s.io/etcd:3.4.3-0],SizeBytes:151677080,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:4ad90a11b55313b182afc186b9876c8e891531b8db4c9bf1541953021618d0e2 registry.k8s.io/etcd:3.4.13-0],SizeBytes:134531559,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:0d2ab6b15b4fe0b903b9e81192d942fb39c29be61fe341cd16db5d0165b0435c registry.k8s.io/etcd:3.3.17-0],SizeBytes:126979107,},ContainerImage{Names:[registry.k8s.io/kube-apiserver-arm64:v1.27.0-alpha.3.420_bb6c6fad2c19b3],SizeBytes:115123555,},ContainerImage{Names:[registry.k8s.io/kube-controller-manager-arm64:v1.27.0-alpha.3.420_bb6c6fad2c19b3],SizeBytes:107914235,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:17da501f5d2a675be46040422a27b7cc21b8a43895ac998b171db1c346f361f7 registry.k8s.io/etcd:3.3.10-0],SizeBytes:81436097,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:6f72b851544986cb0921b53ea655ec04c36131248f16d4ad110cb3ca0c369dc1 registry.k8s.io/etcd:3.5.4-0],SizeBytes:81120118,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:13f53ed1d91e2e11aac476ee9a0269fdda6cc4874eba903efd40daf50c55eee5 registry.k8s.io/etcd:3.5.3-0],SizeBytes:81114891,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:51eae8381dcb1078289fa7b4f3df2630cdc18d09fb56f8e56b41c40e191d6c83 registry.k8s.io/etcd:3.5.7-0],SizeBytes:80665728,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:dd75ec974b0a2a6f6bb47001ba09207976e625db898d1b16735528c009cb171c registry.k8s.io/etcd:3.5.6-0],SizeBytes:80539316,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:905d7ca17fd02bc24c0eba9a062753aba15db3e31422390bc3238eb762339b20 registry.k8s.io/etcd:3.2.24-1],SizeBytes:61500433,},ContainerImage{Names:[registry.k8s.io/kube-proxy-arm64:v1.27.0-alpha.3.420_bb6c6fad2c19b3],SizeBytes:60947298,},ContainerImage{Names:[registry.k8s.io/etcdadm/etcd-manager-slim@sha256:8727adc059334e0e3329dbccce116a734da4a090da5e86c8cddccaf333caa666 registry.k8s.io/etcdadm/etcd-manager-slim:v3.0.20230201],SizeBytes:60826655,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:64b9ea357325d5db9f8a723dcf503b5a449177b17ac87d69481e126bb724c263 registry.k8s.io/etcd:3.5.1-0],SizeBytes:59553904,},ContainerImage{Names:[registry.k8s.io/kube-scheduler-arm64:v1.27.0-alpha.3.420_bb6c6fad2c19b3],SizeBytes:56731120,},ContainerImage{Names:[registry.k8s.io/kops/kops-controller:1.27.0-alpha.1],SizeBytes:41885191,},ContainerImage{Names:[registry.k8s.io/kops/dns-controller:1.27.0-alpha.1],SizeBytes:40376038,},ContainerImage{Names:[public.ecr.aws/karpenter/controller@sha256:68db4f092cf9cc83f5ef9e2fbc5407c2cb682e81f64dfaa700a7602ede38b1cf public.ecr.aws/karpenter/controller:v0.16.3],SizeBytes:33707751,},ContainerImage{Names:[public.ecr.aws/karpenter/webhook@sha256:96a2d9b06d6bc5127801f358f74b1cf2d289b423a2e9ba40c573c0b14b17dafa public.ecr.aws/karpenter/webhook:v0.16.3],SizeBytes:33307187,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:28932904,},ContainerImage{Names:[quay.io/cilium/operator@sha256:a6d24a006a6b92967ac90786b49bc1ac26e5477cf028cd1186efcfc2466484db quay.io/cilium/operator:v1.12.5],SizeBytes:24567287,},ContainerImage{Names:[registry.k8s.io/provider-aws/cloud-controller-manager@sha256:fdeb61e3e42ecd9cca868d550ebdb88dd6341d9e91fcfa9a37e227dab2ad22cb registry.k8s.io/provider-aws/cloud-controller-manager:v1.26.0],SizeBytes:18345226,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:8502345,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:7685305,},ContainerImage{Names:[registry.k8s.io/kops/kube-apiserver-healthcheck:1.27.0-alpha.1],SizeBytes:4740762,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:253553,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Mar 11 17:05:28.256: INFO: Logging kubelet events for node i-0e3467ec552af797e Mar 11 17:05:28.408: INFO: Logging pods the kubelet thinks is on node i-0e3467ec552af797e Mar 11 17:05:28.564: INFO: cilium-operator-8496cc65f5-qhdg8 started at 2023-03-11 16:21:37 +0000 UTC (0+1 container statuses recorded) Mar 11 17:05:28.564: INFO: Container cilium-operator ready: true, restart count 0 Mar 11 17:05:28.564: INFO: kops-controller-76gkw started at 2023-03-11 16:21:37 +0000 UTC (0+1 container statuses recorded) Mar 11 17:05:28.564: INFO: Container kops-controller ready: true, restart count 0 Mar 11 17:05:28.564: INFO: aws-cloud-controller-manager-kwbw2 started at 2023-03-11 16:21:37 +0000 UTC (0+1 container statuses recorded) Mar 11 17:05:28.564: INFO: Container aws-cloud-controller-manager ready: true, restart count 0 Mar 11 17:05:28.564: INFO: kube-apiserver-i-0e3467ec552af797e started at 2023-03-11 16:18:44 +0000 UTC (0+2 container statuses recorded) Mar 11 17:05:28.564: INFO: Container healthcheck ready: true, restart count 0 Mar 11 17:05:28.564: INFO: Container kube-apiserver ready: true, restart count 3 Mar 11 17:05:28.564: INFO: kube-scheduler-i-0e3467ec552af797e started at 2023-03-11 16:18:44 +0000 UTC (0+1 container statuses recorded) Mar 11 17:05:28.564: INFO: Container kube-scheduler ready: true, restart count 0 Mar 11 17:05:28.564: INFO: dns-controller-7dfd64d999-q9hh7 started at 2023-03-11 16:21:37 +0000 UTC (0+1 container statuses recorded) Mar 11 17:05:28.564: INFO: Container dns-controller ready: true, restart count 0 Mar 11 17:05:28.564: INFO: cilium-qrsfn started at 2023-03-11 16:21:37 +0000 UTC (1+1 container statuses recorded) Mar 11 17:05:28.564: INFO: Init container clean-cilium-state ready: true, restart count 0 Mar 11 17:05:28.564: INFO: Container cilium-agent ready: true, restart count 0 Mar 11 17:05:28.564: INFO: karpenter-webhook-5b646cd4f7-6sg2k started at 2023-03-11 16:21:58 +0000 UTC (0+1 container statuses recorded) Mar 11 17:05:28.564: INFO: Container webhook ready: true, restart count 0 Mar 11 17:05:28.564: INFO: karpenter-controller-567749677d-d9szs started at 2023-03-11 16:21:58 +0000 UTC (0+1 container statuses recorded) Mar 11 17:05:28.564: INFO: Container controller ready: true, restart count 0 Mar 11 17:05:28.564: INFO: kube-controller-manager-i-0e3467ec552af797e started at 2023-03-11 16:18:44 +0000 UTC (0+1 container statuses recorded) Mar 11 17:05:28.564: INFO: Container kube-controller-manager ready: true, restart count 4 Mar 11 17:05:28.564: INFO: etcd-manager-events-i-0e3467ec552af797e started at 2023-03-11 16:18:44 +0000 UTC (11+1 container statuses recorded) Mar 11 17:05:28.564: INFO: Init container init-etcd-3-2-24 ready: true, restart count 0 Mar 11 17:05:28.564: INFO: Init container init-etcd-3-3-10 ready: true, restart count 0 Mar 11 17:05:28.564: INFO: Init container init-etcd-3-3-17 ready: true, restart count 0 Mar 11 17:05:28.564: INFO: Init container init-etcd-3-4-13 ready: true, restart count 0 Mar 11 17:05:28.564: INFO: Init container init-etcd-3-4-3 ready: true, restart count 0 Mar 11 17:05:28.564: INFO: Init container init-etcd-3-5-0 ready: true, restart count 0 Mar 11 17:05:28.564: INFO: Init container init-etcd-3-5-1 ready: true, restart count 0 Mar 11 17:05:28.564: INFO: Init container init-etcd-3-5-3 ready: true, restart count 0 Mar 11 17:05:28.564: INFO: Init container init-etcd-3-5-4 ready: true, restart count 0 Mar 11 17:05:28.564: INFO: Init container init-etcd-3-5-6 ready: true, restart count 0 Mar 11 17:05:28.564: INFO: Init container init-etcd-3-5-7 ready: true, restart count 0 Mar 11 17:05:28.564: INFO: Container etcd-manager ready: true, restart count 0 Mar 11 17:05:28.564: INFO: etcd-manager-main-i-0e3467ec552af797e started at 2023-03-11 16:18:44 +0000 UTC (11+1 container statuses recorded) Mar 11 17:05:28.564: INFO: Init container init-etcd-3-2-24 ready: true, restart count 0 Mar 11 17:05:28.564: INFO: Init container init-etcd-3-3-10 ready: true, restart count 0 Mar 11 17:05:28.564: INFO: Init container init-etcd-3-3-17 ready: true, restart count 0 Mar 11 17:05:28.564: INFO: Init container init-etcd-3-4-13 ready: true, restart count 0 Mar 11 17:05:28.564: INFO: Init container init-etcd-3-4-3 ready: true, restart count 0 Mar 11 17:05:28.564: INFO: Init container init-etcd-3-5-0 ready: true, restart count 0 Mar 11 17:05:28.564: INFO: Init container init-etcd-3-5-1 ready: true, restart count 0 Mar 11 17:05:28.564: INFO: Init container init-etcd-3-5-3 ready: true, restart count 0 Mar 11 17:05:28.564: INFO: Init container init-etcd-3-5-4 ready: true, restart count 0 Mar 11 17:05:28.564: INFO: Init container init-etcd-3-5-6 ready: true, restart count 0 Mar 11 17:05:28.564: INFO: Init container init-etcd-3-5-7 ready: true, restart count 0 Mar 11 17:05:28.564: INFO: Container etcd-manager ready: true, restart count 0 Mar 11 17:05:28.564: INFO: ebs-csi-node-lpwz9 started at 2023-03-11 16:21:37 +0000 UTC (0+3 container statuses recorded) Mar 11 17:05:28.564: INFO: Container ebs-plugin ready: true, restart count 0 Mar 11 17:05:28.564: INFO: Container liveness-probe ready: true, restart count 0 Mar 11 17:05:28.564: INFO: Container node-driver-registrar ready: true, restart count 0 Mar 11 17:05:29.206: INFO: Latency metrics for node i-0e3467ec552af797e END STEP: dump namespace information after failure - test/e2e/framework/framework.go:288 @ 03/11/23 17:05:29.206 (52.905s) < Exit [DeferCleanup (Each)] [sig-network] Services - dump namespaces | framework.go:209 @ 03/11/23 17:05:29.206 (52.905s) > Enter [DeferCleanup (Each)] [sig-network] Services - tear down framework | framework.go:206 @ 03/11/23 17:05:29.206 STEP: Destroying namespace "services-4707" for this suite. - test/e2e/framework/framework.go:351 @ 03/11/23 17:05:29.206 < Exit [DeferCleanup (Each)] [sig-network] Services - tear down framework | framework.go:206 @ 03/11/23 17:05:29.356 (151ms) > Enter [ReportAfterEach] TOP-LEVEL - test/e2e/e2e_test.go:144 @ 03/11/23 17:05:29.357 < Exit [ReportAfterEach] TOP-LEVEL - test/e2e/e2e_test.go:144 @ 03/11/23 17:05:29.357 (0s)
Filter through log files | View test history on testgrid
go run hack/e2e.go -v --test --test_args='--ginkgo.focus=Kubernetes\se2e\ssuite\s\[It\]\s\[sig\-network\]\sServices\sshould\sserve\sendpoints\son\ssame\sport\sand\sdifferent\sprotocols$'
[FAILED] Timed out after 30.001s. Expected <nil>: nil not to be nil In [It] at: test/e2e/network/service.go:3703 @ 03/11/23 16:33:07.152from junit_01.xml
> Enter [BeforeEach] [sig-network] Services - set up framework | framework.go:191 @ 03/11/23 16:32:21.391 STEP: Creating a kubernetes client - test/e2e/framework/framework.go:211 @ 03/11/23 16:32:21.391 Mar 11 16:32:21.391: INFO: >>> kubeConfig: /root/.kube/config STEP: Building a namespace api object, basename services - test/e2e/framework/framework.go:250 @ 03/11/23 16:32:21.392 STEP: Waiting for a default service account to be provisioned in namespace - test/e2e/framework/framework.go:259 @ 03/11/23 16:32:21.839 STEP: Waiting for kube-root-ca.crt to be provisioned in namespace - test/e2e/framework/framework.go:262 @ 03/11/23 16:32:22.135 < Exit [BeforeEach] [sig-network] Services - set up framework | framework.go:191 @ 03/11/23 16:32:22.431 (1.04s) > Enter [BeforeEach] [sig-network] Services - test/e2e/framework/metrics/init/init.go:33 @ 03/11/23 16:32:22.431 < Exit [BeforeEach] [sig-network] Services - test/e2e/framework/metrics/init/init.go:33 @ 03/11/23 16:32:22.431 (0s) > Enter [BeforeEach] [sig-network] Services - test/e2e/network/service.go:764 @ 03/11/23 16:32:22.431 < Exit [BeforeEach] [sig-network] Services - test/e2e/network/service.go:764 @ 03/11/23 16:32:22.431 (0s) > Enter [It] should serve endpoints on same port and different protocols - test/e2e/network/service.go:3622 @ 03/11/23 16:32:22.431 STEP: creating service multiprotocol-test in namespace services-2895 - test/e2e/network/service.go:3641 @ 03/11/23 16:32:22.431 STEP: creating pod pod1 in namespace services-2895 - test/e2e/network/service.go:3667 @ 03/11/23 16:32:22.585 STEP: Creating pod pod1 in namespace services-2895 - test/e2e/network/service.go:4094 @ 03/11/23 16:32:22.585 STEP: waiting up to 3m0s for service multiprotocol-test in namespace services-2895 to expose endpoints map[pod1:[{tcp-port 0 80 TCP } {udp-port 0 80 UDP }]] - test/e2e/network/service.go:4286 @ 03/11/23 16:32:25.185 Mar 11 16:32:25.779: INFO: successfully validated that service multiprotocol-test in namespace services-2895 exposes endpoints map[pod1:[{tcp-port 0 80 TCP } {udp-port 0 80 UDP }]] STEP: Checking if the Service forwards traffic to the TCP and UDP port - test/e2e/network/service.go:3671 @ 03/11/23 16:32:25.779 Mar 11 16:32:25.779: INFO: Creating new exec pod Mar 11 16:32:28.228: INFO: Running '/home/prow/go/src/k8s.io/kops/_rundir/c9b7e3a4-c027-11ed-9d91-c209e83e3776/kubectl --server=https://api.e2e-e2e-kops-aws-karpenter.test-cncf-aws.k8s.io --kubeconfig=/root/.kube/config --namespace=services-2895 exec execpodkprtv -- /bin/sh -x -c echo hostName | nc -v -t -w 2 100.66.27.166 80' Mar 11 16:32:29.766: INFO: stderr: "+ echo hostName\n+ nc -v -t -w 2 100.66.27.166 80\nConnection to 100.66.27.166 80 port [tcp/http] succeeded!\n" Mar 11 16:32:29.766: INFO: stdout: "HTTP/1.1 400 Bad Request\r\nContent-Type: text/plain; charset=utf-8\r\nConnection: close\r\n\r\n400 Bad Request" Mar 11 16:32:29.766: INFO: Running '/home/prow/go/src/k8s.io/kops/_rundir/c9b7e3a4-c027-11ed-9d91-c209e83e3776/kubectl --server=https://api.e2e-e2e-kops-aws-karpenter.test-cncf-aws.k8s.io --kubeconfig=/root/.kube/config --namespace=services-2895 exec execpodkprtv -- /bin/sh -x -c echo hostName | nc -v -u -w 2 100.66.27.166 80' Mar 11 16:32:35.292: INFO: stderr: "+ echo hostName\n+ nc -v -u -w 2 100.66.27.166 80\nConnection to 100.66.27.166 80 port [udp/*] succeeded!\n" Mar 11 16:32:35.293: INFO: stdout: "pod1" STEP: Checking if the Service forwards traffic to TCP only - test/e2e/network/service.go:3682 @ 03/11/23 16:32:35.293 Mar 11 16:32:35.591: INFO: Running '/home/prow/go/src/k8s.io/kops/_rundir/c9b7e3a4-c027-11ed-9d91-c209e83e3776/kubectl --server=https://api.e2e-e2e-kops-aws-karpenter.test-cncf-aws.k8s.io --kubeconfig=/root/.kube/config --namespace=services-2895 exec execpodkprtv -- /bin/sh -x -c echo hostName | nc -v -t -w 2 100.66.27.166 80' Mar 11 16:32:37.150: INFO: stderr: "+ echo hostName\n+ nc -v -t -w 2 100.66.27.166 80\nConnection to 100.66.27.166 80 port [tcp/http] succeeded!\n" Mar 11 16:32:37.150: INFO: stdout: "HTTP/1.1 400 Bad Request\r\nContent-Type: text/plain; charset=utf-8\r\nConnection: close\r\n\r\n400 Bad Request" Mar 11 16:32:37.150: INFO: Running '/home/prow/go/src/k8s.io/kops/_rundir/c9b7e3a4-c027-11ed-9d91-c209e83e3776/kubectl --server=https://api.e2e-e2e-kops-aws-karpenter.test-cncf-aws.k8s.io --kubeconfig=/root/.kube/config --namespace=services-2895 exec execpodkprtv -- /bin/sh -x -c echo hostName | nc -v -u -w 2 100.66.27.166 80' Mar 11 16:32:42.696: INFO: stderr: "+ echo hostName\n+ nc -v -u -w 2 100.66.27.166 80\nConnection to 100.66.27.166 80 port [udp/*] succeeded!\n" Mar 11 16:32:42.696: INFO: stdout: "pod1" Mar 11 16:32:47.697: INFO: Running '/home/prow/go/src/k8s.io/kops/_rundir/c9b7e3a4-c027-11ed-9d91-c209e83e3776/kubectl --server=https://api.e2e-e2e-kops-aws-karpenter.test-cncf-aws.k8s.io --kubeconfig=/root/.kube/config --namespace=services-2895 exec execpodkprtv -- /bin/sh -x -c echo hostName | nc -v -u -w 2 100.66.27.166 80' Mar 11 16:32:53.223: INFO: stderr: "+ nc -v -u -w 2 100.66.27.166 80\n+ echo hostName\nConnection to 100.66.27.166 80 port [udp/*] succeeded!\n" Mar 11 16:32:53.223: INFO: stdout: "pod1" Mar 11 16:32:58.223: INFO: Running '/home/prow/go/src/k8s.io/kops/_rundir/c9b7e3a4-c027-11ed-9d91-c209e83e3776/kubectl --server=https://api.e2e-e2e-kops-aws-karpenter.test-cncf-aws.k8s.io --kubeconfig=/root/.kube/config --namespace=services-2895 exec execpodkprtv -- /bin/sh -x -c echo hostName | nc -v -u -w 2 100.66.27.166 80' Mar 11 16:33:03.763: INFO: stderr: "+ echo hostName+ nc\n -v -u -w 2 100.66.27.166 80\nConnection to 100.66.27.166 80 port [udp/*] succeeded!\n" Mar 11 16:33:03.763: INFO: stdout: "pod1" [FAILED] Timed out after 30.001s. Expected <nil>: nil not to be nil In [It] at: test/e2e/network/service.go:3703 @ 03/11/23 16:33:07.152 < Exit [It] should serve endpoints on same port and different protocols - test/e2e/network/service.go:3622 @ 03/11/23 16:33:07.152 (44.72s) > Enter [AfterEach] [sig-network] Services - test/e2e/framework/node/init/init.go:33 @ 03/11/23 16:33:07.152 Mar 11 16:33:07.152: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready < Exit [AfterEach] [sig-network] Services - test/e2e/framework/node/init/init.go:33 @ 03/11/23 16:33:07.301 (149ms) > Enter [DeferCleanup (Each)] [sig-network] Services - test/e2e/framework/metrics/init/init.go:35 @ 03/11/23 16:33:07.301 < Exit [DeferCleanup (Each)] [sig-network] Services - test/e2e/framework/metrics/init/init.go:35 @ 03/11/23 16:33:07.301 (0s) > Enter [DeferCleanup (Each)] [sig-network] Services - dump namespaces | framework.go:209 @ 03/11/23 16:33:07.301 STEP: dump namespace information after failure - test/e2e/framework/framework.go:288 @ 03/11/23 16:33:07.301 STEP: Collecting events from namespace "services-2895". - test/e2e/framework/debug/dump.go:42 @ 03/11/23 16:33:07.301 STEP: Found 8 events. - test/e2e/framework/debug/dump.go:46 @ 03/11/23 16:33:07.45 Mar 11 16:33:07.450: INFO: At 2023-03-11 16:32:22 +0000 UTC - event for pod1: {default-scheduler } Scheduled: Successfully assigned services-2895/pod1 to i-07e2b6e650d7ef56d Mar 11 16:33:07.450: INFO: At 2023-03-11 16:32:23 +0000 UTC - event for pod1: {kubelet i-07e2b6e650d7ef56d} Pulled: Container image "registry.k8s.io/e2e-test-images/agnhost:2.43" already present on machine Mar 11 16:33:07.450: INFO: At 2023-03-11 16:32:23 +0000 UTC - event for pod1: {kubelet i-07e2b6e650d7ef56d} Created: Created container agnhost-container Mar 11 16:33:07.450: INFO: At 2023-03-11 16:32:23 +0000 UTC - event for pod1: {kubelet i-07e2b6e650d7ef56d} Started: Started container agnhost-container Mar 11 16:33:07.450: INFO: At 2023-03-11 16:32:25 +0000 UTC - event for execpodkprtv: {default-scheduler } Scheduled: Successfully assigned services-2895/execpodkprtv to i-07e2b6e650d7ef56d Mar 11 16:33:07.450: INFO: At 2023-03-11 16:32:26 +0000 UTC - event for execpodkprtv: {kubelet i-07e2b6e650d7ef56d} Pulled: Container image "registry.k8s.io/e2e-test-images/agnhost:2.43" already present on machine Mar 11 16:33:07.450: INFO: At 2023-03-11 16:32:26 +0000 UTC - event for execpodkprtv: {kubelet i-07e2b6e650d7ef56d} Created: Created container agnhost-container Mar 11 16:33:07.450: INFO: At 2023-03-11 16:32:26 +0000 UTC - event for execpodkprtv: {kubelet i-07e2b6e650d7ef56d} Started: Started container agnhost-container Mar 11 16:33:07.600: INFO: POD NODE PHASE GRACE CONDITIONS Mar 11 16:33:07.600: INFO: execpodkprtv i-07e2b6e650d7ef56d Running [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2023-03-11 16:32:25 +0000 UTC } {Ready True 0001-01-01 00:00:00 +0000 UTC 2023-03-11 16:32:27 +0000 UTC } {ContainersReady True 0001-01-01 00:00:00 +0000 UTC 2023-03-11 16:32:27 +0000 UTC } {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2023-03-11 16:32:25 +0000 UTC }] Mar 11 16:33:07.600: INFO: pod1 i-07e2b6e650d7ef56d Running [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2023-03-11 16:32:22 +0000 UTC } {Ready True 0001-01-01 00:00:00 +0000 UTC 2023-03-11 16:32:23 +0000 UTC } {ContainersReady True 0001-01-01 00:00:00 +0000 UTC 2023-03-11 16:32:23 +0000 UTC } {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2023-03-11 16:32:22 +0000 UTC }] Mar 11 16:33:07.600: INFO: Mar 11 16:33:08.054: INFO: Logging node info for node i-04523d1dffb0fe725 Mar 11 16:33:08.203: INFO: Node Info: &Node{ObjectMeta:{i-04523d1dffb0fe725 5d37640f-a7d9-4c26-86c5-9da21f7ccc18 16266 0 2023-03-11 16:24:20 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:arm64 beta.kubernetes.io/instance-type:t4g.small beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-northeast-1 failure-domain.beta.kubernetes.io/zone:ap-northeast-1a karpenter.k8s.aws/instance-category:t karpenter.k8s.aws/instance-cpu:2 karpenter.k8s.aws/instance-family:t4g karpenter.k8s.aws/instance-generation:4 karpenter.k8s.aws/instance-hypervisor:nitro karpenter.k8s.aws/instance-memory:2048 karpenter.k8s.aws/instance-pods:11 karpenter.k8s.aws/instance-size:small karpenter.sh/capacity-type:spot karpenter.sh/initialized:true karpenter.sh/provisioner-name:nodes kubernetes.io/arch:arm64 kubernetes.io/hostname:i-04523d1dffb0fe725 kubernetes.io/os:linux node-role.kubernetes.io/node: node-role.kubernetes.io/spot-worker:true node.kubernetes.io/instance-type:t4g.small topology.ebs.csi.aws.com/zone:ap-northeast-1a topology.hostpath.csi/node:i-04523d1dffb0fe725 topology.kubernetes.io/region:ap-northeast-1 topology.kubernetes.io/zone:ap-northeast-1a] map[csi.volume.kubernetes.io/nodeid:{"csi-hostpath-read-write-once-pod-9470":"i-04523d1dffb0fe725","csi-mock-csi-mock-volumes-workload-584":"i-04523d1dffb0fe725","ebs.csi.aws.com":"i-04523d1dffb0fe725"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [{karpenter.sh/v1alpha5 Provisioner nodes ac7283b2-eed0-4a9f-9ddd-616f33a0991a <nil> 0xc00513deb5}] [karpenter.sh/termination] [{aws-cloud-controller-manager Update v1 2023-03-11 16:24:20 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{}}}} } {kops-controller Update v1 2023-03-11 16:24:20 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{},"f:node-role.kubernetes.io/spot-worker":{}}}} } {karpenter Update v1 2023-03-11 16:26:12 +0000 UTC FieldsV1 {"f:metadata":{"f:finalizers":{".":{},"v:\"karpenter.sh/termination\"":{}},"f:labels":{".":{},"f:karpenter.k8s.aws/instance-category":{},"f:karpenter.k8s.aws/instance-cpu":{},"f:karpenter.k8s.aws/instance-family":{},"f:karpenter.k8s.aws/instance-generation":{},"f:karpenter.k8s.aws/instance-hypervisor":{},"f:karpenter.k8s.aws/instance-memory":{},"f:karpenter.k8s.aws/instance-pods":{},"f:karpenter.k8s.aws/instance-size":{},"f:karpenter.sh/capacity-type":{},"f:karpenter.sh/initialized":{},"f:karpenter.sh/provisioner-name":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}},"f:ownerReferences":{".":{},"k:{\"uid\":\"ac7283b2-eed0-4a9f-9ddd-616f33a0991a\"}":{}}},"f:spec":{"f:providerID":{}}} } {kube-controller-manager Update v1 2023-03-11 16:26:12 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:node.alpha.kubernetes.io/ttl":{}},"f:labels":{"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.2.0/24\"":{}}}} } {aws-cloud-controller-manager Update v1 2023-03-11 16:26:50 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{".":{},"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"InternalIP\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kubelet Update v1 2023-03-11 16:33:01 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{"f:kubernetes.io/hostname":{},"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{".":{},"f:cpu":{},"f:ephemeral-storage":{},"f:hugepages-1Gi":{},"f:hugepages-2Mi":{},"f:hugepages-32Mi":{},"f:hugepages-64Ki":{},"f:memory":{},"f:pods":{}},"f:capacity":{".":{},"f:cpu":{},"f:ephemeral-storage":{},"f:hugepages-1Gi":{},"f:hugepages-2Mi":{},"f:hugepages-32Mi":{},"f:hugepages-64Ki":{},"f:memory":{},"f:pods":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:daemonEndpoints":{"f:kubeletEndpoint":{"f:Port":{}}},"f:images":{},"f:nodeInfo":{"f:architecture":{},"f:bootID":{},"f:containerRuntimeVersion":{},"f:kernelVersion":{},"f:kubeProxyVersion":{},"f:kubeletVersion":{},"f:machineID":{},"f:operatingSystem":{},"f:osImage":{},"f:systemUUID":{}},"f:volumesInUse":{}}} status} {kube-controller-manager Update v1 2023-03-11 16:33:02 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{".":{},"k:{\"type\":\"DiskPressure\"}":{".":{},"f:type":{}},"k:{\"type\":\"MemoryPressure\"}":{".":{},"f:type":{}},"k:{\"type\":\"PIDPressure\"}":{".":{},"f:type":{}},"k:{\"type\":\"Ready\"}":{".":{},"f:type":{}}},"f:volumesAttached":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.2.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-northeast-1a/i-04523d1dffb0fe725,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.2.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49756430336 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-32Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-64Ki: {{0 0} {<nil>} 0 DecimalSI},memory: {{1941540864 0} {<nil>} 1896036Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44780787229 0} {<nil>} 44780787229 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-32Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-64Ki: {{0 0} {<nil>} 0 DecimalSI},memory: {{1836683264 0} {<nil>} 1793636Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-03-11 16:33:01 +0000 UTC,LastTransitionTime:2023-03-11 16:26:12 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-03-11 16:33:01 +0000 UTC,LastTransitionTime:2023-03-11 16:26:02 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-03-11 16:33:01 +0000 UTC,LastTransitionTime:2023-03-11 16:26:02 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-03-11 16:33:01 +0000 UTC,LastTransitionTime:2023-03-11 16:26:02 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.48.168,},NodeAddress{Type:ExternalIP,Address:52.199.253.156,},NodeAddress{Type:InternalDNS,Address:i-04523d1dffb0fe725.ap-northeast-1.compute.internal,},NodeAddress{Type:Hostname,Address:i-04523d1dffb0fe725.ap-northeast-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-52-199-253-156.ap-northeast-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2c8fb0e6b2e9c4af310f65b6161cd8,SystemUUID:ec2c8fb0-e6b2-e9c4-af31-0f65b6161cd8,BootID:a3b8de83-558b-44c4-a46a-d097c3f98225,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 22.04.2 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.27.0-alpha.3.420+bb6c6fad2c19b3,KubeProxyVersion:v1.27.0-alpha.3.420+bb6c6fad2c19b3,OperatingSystem:linux,Architecture:arm64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:157636062,},ContainerImage{Names:[registry.k8s.io/kube-proxy-arm64:v1.27.0-alpha.3.420_bb6c6fad2c19b3],SizeBytes:60947298,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:16bbf38c463a4223d8cfe4da12bc61010b082a79b4bb003e2d3ba3ece5dd5f9e registry.k8s.io/e2e-test-images/agnhost:2.43],SizeBytes:49808364,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nautilus@sha256:80ba6c8c44f9623f06e868a1aa66026c8ec438ad814f9ec95e9333b415fe3550 registry.k8s.io/e2e-test-images/nautilus:1.7],SizeBytes:47953400,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:148b022f5c5da426fc2f3c14b5c0867e58ef05961510c84749ac1fddcb0fef22 registry.k8s.io/e2e-test-images/httpd:2.4.38-4],SizeBytes:40343601,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:28932904,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:ee3b525d5b89db99da3b8eb521d9cd90cb6e9ef0fbb651e98bb37be78d36b5b8 registry.k8s.io/sig-storage/csi-provisioner:v3.3.0],SizeBytes:23945474,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:425d8f1b769398127767b06ed97ce62578a3179bcb99809ce93a1649e025ffe7 registry.k8s.io/sig-storage/csi-resizer:v1.6.0],SizeBytes:22678632,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:291334908ddf71a4661fd7f6d9d97274de8a5378a2b6fdfeb2ce73414a34f82f registry.k8s.io/sig-storage/csi-snapshotter:v6.1.0],SizeBytes:22434180,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:9a685020911e2725ad019dbce6e4a5ab93d51e3d4557f115e64343345e05781b registry.k8s.io/sig-storage/csi-attacher:v4.0.0],SizeBytes:22394501,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:122bfb8c1edabb3c0edd63f06523e6940d958d19b3957dc7b1d6f81e9f1f6119 registry.k8s.io/sig-storage/csi-provisioner:v3.1.0],SizeBytes:21735644,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:9ebbf9f023e7b41ccee3d52afe39a89e3ddacdbb69269d583abfc25847cfd9e4 registry.k8s.io/sig-storage/csi-resizer:v1.4.0],SizeBytes:20848413,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:8b9c313c05f54fb04f8d430896f5f5904b6cb157df261501b29adc04d2b2dc7b registry.k8s.io/sig-storage/csi-attacher:v3.4.0],SizeBytes:20565480,},ContainerImage{Names:[registry.k8s.io/cpa/cluster-proportional-autoscaler@sha256:68d396900aeaa072c1f27289485fdac29834045a6f3ffe369bf389d830ef572d registry.k8s.io/cpa/cluster-proportional-autoscaler:1.8.6],SizeBytes:19007623,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nonroot@sha256:ee9f50b3c64b174d296d91ca9f69a914ac30e59095dfb462b2b518ad28a63655 registry.k8s.io/e2e-test-images/nonroot:1.4],SizeBytes:18561972,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:92257881c1d6493cf18299a24af42330f891166560047902b8d431fb66b01af5 registry.k8s.io/sig-storage/hostpathplugin:v1.9.0],SizeBytes:18267611,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:9b2d9abda017c28b12f84a344f57cd73fbdb6c2bd7dd5bdd5018246ad1093ba6 registry.k8s.io/sig-storage/hostpathplugin:v1.11.0],SizeBytes:18159085,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:017727efcfeb7d053af68e51436ce8e65edbc6ca573720afb4f79c8594036955 registry.k8s.io/coredns/coredns:v1.10.0],SizeBytes:13766234,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:8502345,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:933940f13b3ea0abc62e656c1aa5c5b47c04b15d71250413a6b821bd0c58b94e registry.k8s.io/sig-storage/livenessprobe:v2.7.0],SizeBytes:8223819,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:7685305,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:5c99cf6a02adda929b10321dbf4ecfa00d87be9ba4fb456006237d530ab4baa1 registry.k8s.io/e2e-test-images/nginx:1.14-4],SizeBytes:6860089,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf registry.k8s.io/e2e-test-images/busybox:1.29-2],SizeBytes:773367,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:2e0f836850e09b8b7cc937681d6194537a09fbd5f6b9e08f4d646a85128e8937 registry.k8s.io/e2e-test-images/busybox:1.29-4],SizeBytes:772933,},ContainerImage{Names:[registry.k8s.io/pause@sha256:7031c1b283388d2c2e09b57badb803c05ebed362dc88d84b480cc47f72a21097 registry.k8s.io/pause:3.9],SizeBytes:268051,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:253553,},},VolumesInUse:[kubernetes.io/csi/csi-mock-csi-mock-volumes-workload-584^4bb8154e-c02a-11ed-980e-eea8c801c21c],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/csi-mock-csi-mock-volumes-workload-584^4bb8154e-c02a-11ed-980e-eea8c801c21c,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/csi-hostpath-read-write-once-pod-9470^630254be-c02a-11ed-9889-ba0a319aac75,DevicePath:,},},Config:nil,},} Mar 11 16:33:08.203: INFO: Logging kubelet events for node i-04523d1dffb0fe725 Mar 11 16:33:08.355: INFO: Logging pods the kubelet thinks is on node i-04523d1dffb0fe725 Mar 11 16:33:08.520: INFO: webserver-pod started at 2023-03-11 16:31:55 +0000 UTC (0+1 container statuses recorded) Mar 11 16:33:08.520: INFO: Container agnhost-container ready: true, restart count 0 Mar 11 16:33:08.520: INFO: pod-4e160366-1fd3-4153-b6e8-6380ee212ae0 started at 2023-03-11 16:33:01 +0000 UTC (0+1 container statuses recorded) Mar 11 16:33:08.520: INFO: Container write-pod ready: false, restart count 0 Mar 11 16:33:08.520: INFO: pod-subpath-test-configmap-zblz started at 2023-03-11 16:31:37 +0000 UTC (1+2 container statuses recorded) Mar 11 16:33:08.520: INFO: Init container init-volume-configmap-zblz ready: true, restart count 0 Mar 11 16:33:08.520: INFO: Container test-container-subpath-configmap-zblz ready: true, restart count 3 Mar 11 16:33:08.520: INFO: Container test-container-volume-configmap-zblz ready: true, restart count 0 Mar 11 16:33:08.520: INFO: csi-mockplugin-0 started at 2023-03-11 16:32:14 +0000 UTC (0+3 container statuses recorded) Mar 11 16:33:08.520: INFO: Container csi-provisioner ready: true, restart count 0 Mar 11 16:33:08.520: INFO: Container driver-registrar ready: true, restart count 0 Mar 11 16:33:08.520: INFO: Container mock ready: true, restart count 0 Mar 11 16:33:08.520: INFO: pvc-volume-tester-bpth7 started at 2023-03-11 16:32:21 +0000 UTC (0+1 container statuses recorded) Mar 11 16:33:08.520: INFO: Container volume-tester ready: true, restart count 0 Mar 11 16:33:08.520: INFO: pod-configmaps-750f6974-b335-42a2-8954-8025c9aea3e7 started at 2023-03-11 16:33:05 +0000 UTC (0+1 container statuses recorded) Mar 11 16:33:08.520: INFO: Container agnhost-container ready: false, restart count 0 Mar 11 16:33:08.520: INFO: ebs-csi-controller-79785c665b-2wwsh started at 2023-03-11 16:26:12 +0000 UTC (0+5 container statuses recorded) Mar 11 16:33:08.520: INFO: Container csi-attacher ready: true, restart count 0 Mar 11 16:33:08.520: INFO: Container csi-provisioner ready: true, restart count 0 Mar 11 16:33:08.520: INFO: Container csi-resizer ready: true, restart count 0 Mar 11 16:33:08.520: INFO: Container ebs-plugin ready: true, restart count 0 Mar 11 16:33:08.520: INFO: Container liveness-probe ready: true, restart count 0 Mar 11 16:33:08.520: INFO: csi-mockplugin-attacher-0 started at 2023-03-11 16:32:15 +0000 UTC (0+1 container statuses recorded) Mar 11 16:33:08.520: INFO: Container csi-attacher ready: true, restart count 0 Mar 11 16:33:08.520: INFO: pod2 started at 2023-03-11 16:32:55 +0000 UTC (0+1 container statuses recorded) Mar 11 16:33:08.520: INFO: Container agnhost-container ready: true, restart count 0 Mar 11 16:33:08.520: INFO: ephemeral-containers-target-pod started at 2023-03-11 16:33:08 +0000 UTC (0+1 container statuses recorded) Mar 11 16:33:08.520: INFO: Container test-container-1 ready: false, restart count 0 Mar 11 16:33:08.520: INFO: hostexec-i-04523d1dffb0fe725-v26nm started at 2023-03-11 16:33:01 +0000 UTC (0+1 container statuses recorded) Mar 11 16:33:08.520: INFO: Container agnhost-container ready: true, restart count 0 Mar 11 16:33:08.520: INFO: inline-volume-trmmn started at 2023-03-11 16:32:32 +0000 UTC (0+1 container statuses recorded) Mar 11 16:33:08.520: INFO: Container volume-tester ready: false, restart count 0 Mar 11 16:33:08.520: INFO: to-be-attached-pod started at 2023-03-11 16:33:02 +0000 UTC (0+1 container statuses recorded) Mar 11 16:33:08.520: INFO: Container container1 ready: true, restart count 0 Mar 11 16:33:08.520: INFO: coredns-autoscaler-8495f5575-d8gqb started at 2023-03-11 16:26:12 +0000 UTC (0+1 container statuses recorded) Mar 11 16:33:08.520: INFO: Container autoscaler ready: true, restart count 0 Mar 11 16:33:08.520: INFO: pod-client started at 2023-03-11 16:32:41 +0000 UTC (0+1 container statuses recorded) Mar 11 16:33:08.520: INFO: Container pod-client ready: true, restart count 0 Mar 11 16:33:08.520: INFO: sysctl-ee62e2f0-bc6c-4c42-8518-b697588b2bb3 started at 2023-03-11 16:32:59 +0000 UTC (0+1 container statuses recorded) Mar 11 16:33:08.520: INFO: Container test-container ready: false, restart count 0 Mar 11 16:33:08.520: INFO: pod-7da43fb0-5a1e-4d87-9550-95f6008335ec started at 2023-03-11 16:33:05 +0000 UTC (0+1 container statuses recorded) Mar 11 16:33:08.520: INFO: Container write-pod ready: true, restart count 0 Mar 11 16:33:08.520: INFO: ss2-2 started at 2023-03-11 16:32:51 +0000 UTC (0+1 container statuses recorded) Mar 11 16:33:08.520: INFO: Container webserver ready: true, restart count 0 Mar 11 16:33:08.520: INFO: csi-hostpathplugin-0 started at 2023-03-11 16:32:58 +0000 UTC (0+7 container statuses recorded) Mar 11 16:33:08.520: INFO: Container csi-attacher ready: true, restart count 0 Mar 11 16:33:08.520: INFO: Container csi-provisioner ready: true, restart count 0 Mar 11 16:33:08.520: INFO: Container csi-resizer ready: true, restart count 0 Mar 11 16:33:08.520: INFO: Container csi-snapshotter ready: true, restart count 0 Mar 11 16:33:08.520: INFO: Container hostpath ready: true, restart count 0 Mar 11 16:33:08.520: INFO: Container liveness-probe ready: true, restart count 0 Mar 11 16:33:08.520: INFO: Container node-driver-registrar ready: true, restart count 0 Mar 11 16:33:08.520: INFO: cilium-kfwlg started at 2023-03-11 16:26:03 +0000 UTC (1+1 container statuses recorded) Mar 11 16:33:08.520: INFO: Init container clean-cilium-state ready: true, restart count 0 Mar 11 16:33:08.520: INFO: Container cilium-agent ready: true, restart count 0 Mar 11 16:33:08.520: INFO: ebs-csi-node-2qlbt started at 2023-03-11 16:26:03 +0000 UTC (0+3 container statuses recorded) Mar 11 16:33:08.520: INFO: Container ebs-plugin ready: true, restart count 0 Mar 11 16:33:08.520: INFO: Container liveness-probe ready: true, restart count 0 Mar 11 16:33:08.520: INFO: Container node-driver-registrar ready: true, restart count 0 Mar 11 16:33:08.520: INFO: coredns-745bddcfbb-s57v5 started at 2023-03-11 16:26:12 +0000 UTC (0+1 container statuses recorded) Mar 11 16:33:08.520: INFO: Container coredns ready: true, restart count 0 Mar 11 16:33:09.166: INFO: Latency metrics for node i-04523d1dffb0fe725 Mar 11 16:33:09.166: INFO: Logging node info for node i-07e2b6e650d7ef56d Mar 11 16:33:09.315: INFO: Node Info: &Node{ObjectMeta:{i-07e2b6e650d7ef56d fcc65272-6f1b-4228-ac59-e9e3e92cdb2c 16309 0 2023-03-11 16:24:19 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:arm64 beta.kubernetes.io/instance-type:t4g.small beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-northeast-1 failure-domain.beta.kubernetes.io/zone:ap-northeast-1a karpenter.k8s.aws/instance-category:t karpenter.k8s.aws/instance-cpu:2 karpenter.k8s.aws/instance-family:t4g karpenter.k8s.aws/instance-generation:4 karpenter.k8s.aws/instance-hypervisor:nitro karpenter.k8s.aws/instance-memory:2048 karpenter.k8s.aws/instance-pods:11 karpenter.k8s.aws/instance-size:small karpenter.sh/capacity-type:spot karpenter.sh/initialized:true karpenter.sh/provisioner-name:nodes kubernetes.io/arch:arm64 kubernetes.io/hostname:i-07e2b6e650d7ef56d kubernetes.io/os:linux node-role.kubernetes.io/node: node-role.kubernetes.io/spot-worker:true node.kubernetes.io/instance-type:t4g.small topology.ebs.csi.aws.com/zone:ap-northeast-1a topology.hostpath.csi/node:i-07e2b6e650d7ef56d topology.kubernetes.io/region:ap-northeast-1 topology.kubernetes.io/zone:ap-northeast-1a] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-07e2b6e650d7ef56d"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [{karpenter.sh/v1alpha5 Provisioner nodes ac7283b2-eed0-4a9f-9ddd-616f33a0991a <nil> 0xc004a7f5d5}] [karpenter.sh/termination] [{aws-cloud-controller-manager Update v1 2023-03-11 16:24:20 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{}}}} } {kops-controller Update v1 2023-03-11 16:24:20 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{},"f:node-role.kubernetes.io/spot-worker":{}}}} } {karpenter Update v1 2023-03-11 16:26:15 +0000 UTC FieldsV1 {"f:metadata":{"f:finalizers":{".":{},"v:\"karpenter.sh/termination\"":{}},"f:labels":{".":{},"f:karpenter.k8s.aws/instance-category":{},"f:karpenter.k8s.aws/instance-cpu":{},"f:karpenter.k8s.aws/instance-family":{},"f:karpenter.k8s.aws/instance-generation":{},"f:karpenter.k8s.aws/instance-hypervisor":{},"f:karpenter.k8s.aws/instance-memory":{},"f:karpenter.k8s.aws/instance-pods":{},"f:karpenter.k8s.aws/instance-size":{},"f:karpenter.sh/capacity-type":{},"f:karpenter.sh/initialized":{},"f:karpenter.sh/provisioner-name":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}},"f:ownerReferences":{".":{},"k:{\"uid\":\"ac7283b2-eed0-4a9f-9ddd-616f33a0991a\"}":{}}},"f:spec":{"f:providerID":{}}} } {kube-controller-manager Update v1 2023-03-11 16:26:15 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:node.alpha.kubernetes.io/ttl":{}},"f:labels":{"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.1.0/24\"":{}}}} } {aws-cloud-controller-manager Update v1 2023-03-11 16:26:50 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{".":{},"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"InternalIP\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kube-controller-manager Update v1 2023-03-11 16:33:03 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{".":{},"k:{\"type\":\"DiskPressure\"}":{".":{},"f:type":{}},"k:{\"type\":\"MemoryPressure\"}":{".":{},"f:type":{}},"k:{\"type\":\"PIDPressure\"}":{".":{},"f:type":{}},"k:{\"type\":\"Ready\"}":{".":{},"f:type":{}}},"f:volumesAttached":{}}} status} {kubelet Update v1 2023-03-11 16:33:03 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{"f:kubernetes.io/hostname":{},"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{".":{},"f:cpu":{},"f:ephemeral-storage":{},"f:hugepages-1Gi":{},"f:hugepages-2Mi":{},"f:hugepages-32Mi":{},"f:hugepages-64Ki":{},"f:memory":{},"f:pods":{}},"f:capacity":{".":{},"f:cpu":{},"f:ephemeral-storage":{},"f:hugepages-1Gi":{},"f:hugepages-2Mi":{},"f:hugepages-32Mi":{},"f:hugepages-64Ki":{},"f:memory":{},"f:pods":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:daemonEndpoints":{"f:kubeletEndpoint":{"f:Port":{}}},"f:images":{},"f:nodeInfo":{"f:architecture":{},"f:bootID":{},"f:containerRuntimeVersion":{},"f:kernelVersion":{},"f:kubeProxyVersion":{},"f:kubeletVersion":{},"f:machineID":{},"f:operatingSystem":{},"f:osImage":{},"f:systemUUID":{}},"f:volumesInUse":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.1.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-northeast-1a/i-07e2b6e650d7ef56d,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.1.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49756430336 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-32Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-64Ki: {{0 0} {<nil>} 0 DecimalSI},memory: {{1941549056 0} {<nil>} 1896044Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44780787229 0} {<nil>} 44780787229 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-32Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-64Ki: {{0 0} {<nil>} 0 DecimalSI},memory: {{1836691456 0} {<nil>} 1793644Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-03-11 16:33:03 +0000 UTC,LastTransitionTime:2023-03-11 16:26:15 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-03-11 16:33:03 +0000 UTC,LastTransitionTime:2023-03-11 16:26:05 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-03-11 16:33:03 +0000 UTC,LastTransitionTime:2023-03-11 16:26:05 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-03-11 16:33:03 +0000 UTC,LastTransitionTime:2023-03-11 16:26:05 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.35.76,},NodeAddress{Type:ExternalIP,Address:54.238.106.83,},NodeAddress{Type:InternalDNS,Address:i-07e2b6e650d7ef56d.ap-northeast-1.compute.internal,},NodeAddress{Type:Hostname,Address:i-07e2b6e650d7ef56d.ap-northeast-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-54-238-106-83.ap-northeast-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2083a8e9d27d02fac967340367f875,SystemUUID:ec2083a8-e9d2-7d02-fac9-67340367f875,BootID:ea264ed6-5b0d-47d6-ad5d-6bd2a1f7c11b,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 22.04.2 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.27.0-alpha.3.420+bb6c6fad2c19b3,KubeProxyVersion:v1.27.0-alpha.3.420+bb6c6fad2c19b3,OperatingSystem:linux,Architecture:arm64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:157636062,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/jessie-dnsutils@sha256:24aaf2626d6b27864c29de2097e8bbb840b3a414271bf7c8995e431e47d8408e registry.k8s.io/e2e-test-images/jessie-dnsutils:1.7],SizeBytes:116764683,},ContainerImage{Names:[registry.k8s.io/kube-proxy-arm64:v1.27.0-alpha.3.420_bb6c6fad2c19b3],SizeBytes:60947298,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:16bbf38c463a4223d8cfe4da12bc61010b082a79b4bb003e2d3ba3ece5dd5f9e registry.k8s.io/e2e-test-images/agnhost:2.43],SizeBytes:49808364,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nautilus@sha256:80ba6c8c44f9623f06e868a1aa66026c8ec438ad814f9ec95e9333b415fe3550 registry.k8s.io/e2e-test-images/nautilus:1.7],SizeBytes:47953400,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:148b022f5c5da426fc2f3c14b5c0867e58ef05961510c84749ac1fddcb0fef22 registry.k8s.io/e2e-test-images/httpd:2.4.38-4],SizeBytes:40343601,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:28932904,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:ee3b525d5b89db99da3b8eb521d9cd90cb6e9ef0fbb651e98bb37be78d36b5b8 registry.k8s.io/sig-storage/csi-provisioner:v3.3.0],SizeBytes:23945474,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:425d8f1b769398127767b06ed97ce62578a3179bcb99809ce93a1649e025ffe7 registry.k8s.io/sig-storage/csi-resizer:v1.6.0],SizeBytes:22678632,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:291334908ddf71a4661fd7f6d9d97274de8a5378a2b6fdfeb2ce73414a34f82f registry.k8s.io/sig-storage/csi-snapshotter:v6.1.0],SizeBytes:22434180,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:9a685020911e2725ad019dbce6e4a5ab93d51e3d4557f115e64343345e05781b registry.k8s.io/sig-storage/csi-attacher:v4.0.0],SizeBytes:22394501,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:122bfb8c1edabb3c0edd63f06523e6940d958d19b3957dc7b1d6f81e9f1f6119 registry.k8s.io/sig-storage/csi-provisioner:v3.1.0],SizeBytes:21735644,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:9ebbf9f023e7b41ccee3d52afe39a89e3ddacdbb69269d583abfc25847cfd9e4 registry.k8s.io/sig-storage/csi-resizer:v1.4.0],SizeBytes:20848413,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:8b9c313c05f54fb04f8d430896f5f5904b6cb157df261501b29adc04d2b2dc7b registry.k8s.io/sig-storage/csi-attacher:v3.4.0],SizeBytes:20565480,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:92257881c1d6493cf18299a24af42330f891166560047902b8d431fb66b01af5 registry.k8s.io/sig-storage/hostpathplugin:v1.9.0],SizeBytes:18267611,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:9b2d9abda017c28b12f84a344f57cd73fbdb6c2bd7dd5bdd5018246ad1093ba6 registry.k8s.io/sig-storage/hostpathplugin:v1.11.0],SizeBytes:18159085,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:017727efcfeb7d053af68e51436ce8e65edbc6ca573720afb4f79c8594036955 registry.k8s.io/coredns/coredns:v1.10.0],SizeBytes:13766234,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:8502345,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:933940f13b3ea0abc62e656c1aa5c5b47c04b15d71250413a6b821bd0c58b94e registry.k8s.io/sig-storage/livenessprobe:v2.7.0],SizeBytes:8223819,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:7685305,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:5c99cf6a02adda929b10321dbf4ecfa00d87be9ba4fb456006237d530ab4baa1 registry.k8s.io/e2e-test-images/nginx:1.14-4],SizeBytes:6860089,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf registry.k8s.io/e2e-test-images/busybox:1.29-2],SizeBytes:773367,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:2e0f836850e09b8b7cc937681d6194537a09fbd5f6b9e08f4d646a85128e8937 registry.k8s.io/e2e-test-images/busybox:1.29-4],SizeBytes:772933,},ContainerImage{Names:[registry.k8s.io/pause@sha256:7031c1b283388d2c2e09b57badb803c05ebed362dc88d84b480cc47f72a21097 registry.k8s.io/pause:3.9],SizeBytes:268051,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:253553,},},VolumesInUse:[kubernetes.io/csi/ebs.csi.aws.com^vol-00dd094a7765d1f07 kubernetes.io/csi/ebs.csi.aws.com^vol-052e2a8d13a3d22a1],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-052e2a8d13a3d22a1,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-00dd094a7765d1f07,DevicePath:,},},Config:nil,},} Mar 11 16:33:09.315: INFO: Logging kubelet events for node i-07e2b6e650d7ef56d Mar 11 16:33:09.467: INFO: Logging pods the kubelet thinks is on node i-07e2b6e650d7ef56d Mar 11 16:33:09.624: INFO: pod-b72de422-b57e-40d9-8662-3503d5b015dc started at 2023-03-11 16:33:08 +0000 UTC (0+1 container statuses recorded) Mar 11 16:33:09.624: INFO: Container write-pod ready: false, restart count 0 Mar 11 16:33:09.624: INFO: pod1 started at 2023-03-11 16:32:45 +0000 UTC (0+1 container statuses recorded) Mar 11 16:33:09.624: INFO: Container agnhost-container ready: true, restart count 0 Mar 11 16:33:09.624: INFO: cilium-mvm8l started at 2023-03-11 16:26:06 +0000 UTC (1+1 container statuses recorded) Mar 11 16:33:09.624: INFO: Init container clean-cilium-state ready: true, restart count 0 Mar 11 16:33:09.624: INFO: Container cilium-agent ready: true, restart count 0 Mar 11 16:33:09.624: INFO: ebs-csi-controller-79785c665b-lpmmf started at 2023-03-11 16:26:15 +0000 UTC (0+5 container statuses recorded) Mar 11 16:33:09.624: INFO: Container csi-attacher ready: true, restart count 0 Mar 11 16:33:09.624: INFO: Container csi-provisioner ready: true, restart count 0 Mar 11 16:33:09.624: INFO: Container csi-resizer ready: true, restart count 0 Mar 11 16:33:09.624: INFO: Container ebs-plugin ready: true, restart count 0 Mar 11 16:33:09.624: INFO: Container liveness-probe ready: true, restart count 0 Mar 11 16:33:09.624: INFO: execpodbr99k started at 2023-03-11 16:32:49 +0000 UTC (0+1 container statuses recorded) Mar 11 16:33:09.624: INFO: Container agnhost-container ready: true, restart count 0 Mar 11 16:33:09.624: INFO: foo-jtwpl started at 2023-03-11 16:32:56 +0000 UTC (0+1 container statuses recorded) Mar 11 16:33:09.624: INFO: Container c ready: true, restart count 0 Mar 11 16:33:09.624: INFO: inline-volume-tester-bkt86 started at 2023-03-11 16:32:57 +0000 UTC (0+1 container statuses recorded) Mar 11 16:33:09.624: INFO: Container csi-volume-tester ready: true, restart count 0 Mar 11 16:33:09.624: INFO: ebs-csi-node-m5b9f started at 2023-03-11 16:26:06 +0000 UTC (0+3 container statuses recorded) Mar 11 16:33:09.624: INFO: Container ebs-plugin ready: true, restart count 0 Mar 11 16:33:09.624: INFO: Container liveness-probe ready: true, restart count 0 Mar 11 16:33:09.624: INFO: Container node-driver-registrar ready: true, restart count 0 Mar 11 16:33:09.624: INFO: pod-configmaps-ffc90c6a-95c4-4caf-a7d2-4696ac8a79a1 started at 2023-03-11 16:33:07 +0000 UTC (0+1 container statuses recorded) Mar 11 16:33:09.624: INFO: Container agnhost-container ready: false, restart count 0 Mar 11 16:33:09.624: INFO: external-injector started at 2023-03-11 16:32:55 +0000 UTC (0+1 container statuses recorded) Mar 11 16:33:09.624: INFO: Container external-injector ready: true, restart count 0 Mar 11 16:33:09.624: INFO: dns-test-c44cc70e-176a-43e4-b289-53637798644e started at 2023-03-11 16:32:41 +0000 UTC (0+3 container statuses recorded) Mar 11 16:33:09.624: INFO: Container jessie-querier ready: true, restart count 0 Mar 11 16:33:09.624: INFO: Container querier ready: true, restart count 0 Mar 11 16:33:09.624: INFO: Container webserver ready: true, restart count 0 Mar 11 16:33:09.624: INFO: pod1 started at 2023-03-11 16:32:22 +0000 UTC (0+1 container statuses recorded) Mar 11 16:33:09.624: INFO: Container agnhost-container ready: true, restart count 0 Mar 11 16:33:09.624: INFO: execpodkprtv started at 2023-03-11 16:32:25 +0000 UTC (0+1 container statuses recorded) Mar 11 16:33:09.624: INFO: Container agnhost-container ready: true, restart count 0 Mar 11 16:33:09.624: INFO: ss2-0 started at 2023-03-11 16:32:50 +0000 UTC (0+1 container statuses recorded) Mar 11 16:33:09.624: INFO: Container webserver ready: true, restart count 0 Mar 11 16:33:09.624: INFO: ss2-1 started at 2023-03-11 16:33:02 +0000 UTC (0+1 container statuses recorded) Mar 11 16:33:09.624: INFO: Container webserver ready: true, restart count 0 Mar 11 16:33:09.624: INFO: coredns-745bddcfbb-lkbct started at 2023-03-11 16:26:41 +0000 UTC (0+1 container statuses recorded) Mar 11 16:33:09.624: INFO: Container coredns ready: true, restart count 0 Mar 11 16:33:09.624: INFO: liveness-0a3782e9-b416-48f3-a764-4a0383907484 started at 2023-03-11 16:32:41 +0000 UTC (0+1 container statuses recorded) Mar 11 16:33:09.624: INFO: Container agnhost-container ready: true, restart count 0 Mar 11 16:33:09.624: INFO: pod-configmaps-a0ced038-b190-4461-bc15-47ac173b0236 started at 2023-03-11 16:32:18 +0000 UTC (0+3 container statuses recorded) Mar 11 16:33:09.624: INFO: Container createcm-volume-test ready: true, restart count 0 Mar 11 16:33:09.624: INFO: Container delcm-volume-test ready: true, restart count 0 Mar 11 16:33:09.624: INFO: Container updcm-volume-test ready: true, restart count 0 Mar 11 16:33:09.624: INFO: foo-fbhcn started at 2023-03-11 16:32:56 +0000 UTC (0+1 container statuses recorded) Mar 11 16:33:09.624: INFO: Container c ready: true, restart count 0 Mar 11 16:33:09.624: INFO: pod-server-2 started at 2023-03-11 16:32:53 +0000 UTC (0+1 container statuses recorded) Mar 11 16:33:09.624: INFO: Container agnhost-container ready: true, restart count 0 Mar 11 16:33:10.233: INFO: Latency metrics for node i-07e2b6e650d7ef56d Mar 11 16:33:10.233: INFO: Logging node info for node i-0e3467ec552af797e Mar 11 16:33:10.382: INFO: Node Info: &Node{ObjectMeta:{i-0e3467ec552af797e 6c9b0cc5-2c4e-4037-bf31-61bac12a7775 2458 0 2023-03-11 16:20:46 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:arm64 beta.kubernetes.io/instance-type:c6g.xlarge beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-northeast-1 failure-domain.beta.kubernetes.io/zone:ap-northeast-1a kops.k8s.io/kops-controller-pki: kubernetes.io/arch:arm64 kubernetes.io/hostname:i-0e3467ec552af797e kubernetes.io/os:linux node-role.kubernetes.io/control-plane: node.kubernetes.io/exclude-from-external-load-balancers: node.kubernetes.io/instance-type:c6g.xlarge topology.ebs.csi.aws.com/zone:ap-northeast-1a topology.kubernetes.io/region:ap-northeast-1 topology.kubernetes.io/zone:ap-northeast-1a] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-0e3467ec552af797e"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kubelet Update v1 2023-03-11 16:20:46 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {protokube Update v1 2023-03-11 16:21:30 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kops.k8s.io/kops-controller-pki":{},"f:node-role.kubernetes.io/control-plane":{},"f:node.kubernetes.io/exclude-from-external-load-balancers":{}}}} } {aws-cloud-controller-manager Update v1 2023-03-11 16:21:50 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-03-11 16:21:50 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kube-controller-manager Update v1 2023-03-11 16:21:58 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.0.0/24\"":{}},"f:taints":{}}} } {kubelet Update v1 2023-03-11 16:28:26 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.0.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-northeast-1a/i-0e3467ec552af797e,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/control-plane,Value:,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[100.96.0.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{4 0} {<nil>} 4 DecimalSI},ephemeral-storage: {{49756430336 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-32Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-64Ki: {{0 0} {<nil>} 0 DecimalSI},memory: {{8150110208 0} {<nil>} 7959092Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{4 0} {<nil>} 4 DecimalSI},ephemeral-storage: {{44780787229 0} {<nil>} 44780787229 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-32Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-64Ki: {{0 0} {<nil>} 0 DecimalSI},memory: {{8045252608 0} {<nil>} 7856692Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-03-11 16:28:26 +0000 UTC,LastTransitionTime:2023-03-11 16:20:46 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-03-11 16:28:26 +0000 UTC,LastTransitionTime:2023-03-11 16:20:46 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-03-11 16:28:26 +0000 UTC,LastTransitionTime:2023-03-11 16:20:46 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-03-11 16:28:26 +0000 UTC,LastTransitionTime:2023-03-11 16:21:58 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.40.18,},NodeAddress{Type:ExternalIP,Address:52.194.213.7,},NodeAddress{Type:InternalDNS,Address:i-0e3467ec552af797e.ap-northeast-1.compute.internal,},NodeAddress{Type:Hostname,Address:i-0e3467ec552af797e.ap-northeast-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-52-194-213-7.ap-northeast-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2abf2e564965592c7716accd56507e,SystemUUID:ec2abf2e-5649-6559-2c77-16accd56507e,BootID:fbada3ff-8655-404c-a98d-876577d4c322,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 22.04.2 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.27.0-alpha.3.420+bb6c6fad2c19b3,KubeProxyVersion:v1.27.0-alpha.3.420+bb6c6fad2c19b3,OperatingSystem:linux,Architecture:arm64,},Images:[]ContainerImage{ContainerImage{Names:[registry.k8s.io/etcd@sha256:9ce33ba33d8e738a5b85ed50b5080ac746deceed4a7496c550927a7a19ca3b6d registry.k8s.io/etcd:3.5.0-0],SizeBytes:157797353,},ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:157636062,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:4afb99b4690b418ffc2ceb67e1a17376457e441c1f09ab55447f0aaf992fa646 registry.k8s.io/etcd:3.4.3-0],SizeBytes:151677080,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:4ad90a11b55313b182afc186b9876c8e891531b8db4c9bf1541953021618d0e2 registry.k8s.io/etcd:3.4.13-0],SizeBytes:134531559,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:0d2ab6b15b4fe0b903b9e81192d942fb39c29be61fe341cd16db5d0165b0435c registry.k8s.io/etcd:3.3.17-0],SizeBytes:126979107,},ContainerImage{Names:[registry.k8s.io/kube-apiserver-arm64:v1.27.0-alpha.3.420_bb6c6fad2c19b3],SizeBytes:115123555,},ContainerImage{Names:[registry.k8s.io/kube-controller-manager-arm64:v1.27.0-alpha.3.420_bb6c6fad2c19b3],SizeBytes:107914235,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:17da501f5d2a675be46040422a27b7cc21b8a43895ac998b171db1c346f361f7 registry.k8s.io/etcd:3.3.10-0],SizeBytes:81436097,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:6f72b851544986cb0921b53ea655ec04c36131248f16d4ad110cb3ca0c369dc1 registry.k8s.io/etcd:3.5.4-0],SizeBytes:81120118,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:13f53ed1d91e2e11aac476ee9a0269fdda6cc4874eba903efd40daf50c55eee5 registry.k8s.io/etcd:3.5.3-0],SizeBytes:81114891,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:51eae8381dcb1078289fa7b4f3df2630cdc18d09fb56f8e56b41c40e191d6c83 registry.k8s.io/etcd:3.5.7-0],SizeBytes:80665728,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:dd75ec974b0a2a6f6bb47001ba09207976e625db898d1b16735528c009cb171c registry.k8s.io/etcd:3.5.6-0],SizeBytes:80539316,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:905d7ca17fd02bc24c0eba9a062753aba15db3e31422390bc3238eb762339b20 registry.k8s.io/etcd:3.2.24-1],SizeBytes:61500433,},ContainerImage{Names:[registry.k8s.io/kube-proxy-arm64:v1.27.0-alpha.3.420_bb6c6fad2c19b3],SizeBytes:60947298,},ContainerImage{Names:[registry.k8s.io/etcdadm/etcd-manager-slim@sha256:8727adc059334e0e3329dbccce116a734da4a090da5e86c8cddccaf333caa666 registry.k8s.io/etcdadm/etcd-manager-slim:v3.0.20230201],SizeBytes:60826655,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:64b9ea357325d5db9f8a723dcf503b5a449177b17ac87d69481e126bb724c263 registry.k8s.io/etcd:3.5.1-0],SizeBytes:59553904,},ContainerImage{Names:[registry.k8s.io/kube-scheduler-arm64:v1.27.0-alpha.3.420_bb6c6fad2c19b3],SizeBytes:56731120,},ContainerImage{Names:[registry.k8s.io/kops/kops-controller:1.27.0-alpha.1],SizeBytes:41885191,},ContainerImage{Names:[registry.k8s.io/kops/dns-controller:1.27.0-alpha.1],SizeBytes:40376038,},ContainerImage{Names:[public.ecr.aws/karpenter/controller@sha256:68db4f092cf9cc83f5ef9e2fbc5407c2cb682e81f64dfaa700a7602ede38b1cf public.ecr.aws/karpenter/controller:v0.16.3],SizeBytes:33707751,},ContainerImage{Names:[public.ecr.aws/karpenter/webhook@sha256:96a2d9b06d6bc5127801f358f74b1cf2d289b423a2e9ba40c573c0b14b17dafa public.ecr.aws/karpenter/webhook:v0.16.3],SizeBytes:33307187,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:28932904,},ContainerImage{Names:[quay.io/cilium/operator@sha256:a6d24a006a6b92967ac90786b49bc1ac26e5477cf028cd1186efcfc2466484db quay.io/cilium/operator:v1.12.5],SizeBytes:24567287,},ContainerImage{Names:[registry.k8s.io/provider-aws/cloud-controller-manager@sha256:fdeb61e3e42ecd9cca868d550ebdb88dd6341d9e91fcfa9a37e227dab2ad22cb registry.k8s.io/provider-aws/cloud-controller-manager:v1.26.0],SizeBytes:18345226,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:8502345,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:7685305,},ContainerImage{Names:[registry.k8s.io/kops/kube-apiserver-healthcheck:1.27.0-alpha.1],SizeBytes:4740762,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:253553,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Mar 11 16:33:10.383: INFO: Logging kubelet events for node i-0e3467ec552af797e Mar 11 16:33:10.536: INFO: Logging pods the kubelet thinks is on node i-0e3467ec552af797e Mar 11 16:33:10.692: INFO: kops-controller-76gkw started at 2023-03-11 16:21:37 +0000 UTC (0+1 container statuses recorded) Mar 11 16:33:10.692: INFO: Container kops-controller ready: true, restart count 0 Mar 11 16:33:10.692: INFO: aws-cloud-controller-manager-kwbw2 started at 2023-03-11 16:21:37 +0000 UTC (0+1 container statuses recorded) Mar 11 16:33:10.692: INFO: Container aws-cloud-controller-manager ready: true, restart count 0 Mar 11 16:33:10.692: INFO: kube-apiserver-i-0e3467ec552af797e started at 2023-03-11 16:18:44 +0000 UTC (0+2 container statuses recorded) Mar 11 16:33:10.692: INFO: Container healthcheck ready: true, restart count 0 Mar 11 16:33:10.692: INFO: Container kube-apiserver ready: true, restart count 3 Mar 11 16:33:10.692: INFO: kube-scheduler-i-0e3467ec552af797e started at 2023-03-11 16:18:44 +0000 UTC (0+1 container statuses recorded) Mar 11 16:33:10.692: INFO: Container kube-scheduler ready: true, restart count 0 Mar 11 16:33:10.692: INFO: dns-controller-7dfd64d999-q9hh7 started at 2023-03-11 16:21:37 +0000 UTC (0+1 container statuses recorded) Mar 11 16:33:10.692: INFO: Container dns-controller ready: true, restart count 0 Mar 11 16:33:10.692: INFO: cilium-qrsfn started at 2023-03-11 16:21:37 +0000 UTC (1+1 container statuses recorded) Mar 11 16:33:10.692: INFO: Init container clean-cilium-state ready: true, restart count 0 Mar 11 16:33:10.692: INFO: Container cilium-agent ready: true, restart count 0 Mar 11 16:33:10.692: INFO: cilium-operator-8496cc65f5-qhdg8 started at 2023-03-11 16:21:37 +0000 UTC (0+1 container statuses recorded) Mar 11 16:33:10.692: INFO: Container cilium-operator ready: true, restart count 0 Mar 11 16:33:10.692: INFO: karpenter-controller-567749677d-d9szs started at 2023-03-11 16:21:58 +0000 UTC (0+1 container statuses recorded) Mar 11 16:33:10.692: INFO: Container controller ready: true, restart count 0 Mar 11 16:33:10.692: INFO: kube-controller-manager-i-0e3467ec552af797e started at 2023-03-11 16:18:44 +0000 UTC (0+1 container statuses recorded) Mar 11 16:33:10.692: INFO: Container kube-controller-manager ready: true, restart count 4 Mar 11 16:33:10.692: INFO: etcd-manager-events-i-0e3467ec552af797e started at 2023-03-11 16:18:44 +0000 UTC (11+1 container statuses recorded) Mar 11 16:33:10.692: INFO: Init container init-etcd-3-2-24 ready: true, restart count 0 Mar 11 16:33:10.692: INFO: Init container init-etcd-3-3-10 ready: true, restart count 0 Mar 11 16:33:10.692: INFO: Init container init-etcd-3-3-17 ready: true, restart count 0 Mar 11 16:33:10.692: INFO: Init container init-etcd-3-4-13 ready: true, restart count 0 Mar 11 16:33:10.692: INFO: Init container init-etcd-3-4-3 ready: true, restart count 0 Mar 11 16:33:10.692: INFO: Init container init-etcd-3-5-0 ready: true, restart count 0 Mar 11 16:33:10.692: INFO: Init container init-etcd-3-5-1 ready: true, restart count 0 Mar 11 16:33:10.692: INFO: Init container init-etcd-3-5-3 ready: true, restart count 0 Mar 11 16:33:10.692: INFO: Init container init-etcd-3-5-4 ready: true, restart count 0 Mar 11 16:33:10.692: INFO: Init container init-etcd-3-5-6 ready: true, restart count 0 Mar 11 16:33:10.692: INFO: Init container init-etcd-3-5-7 ready: true, restart count 0 Mar 11 16:33:10.692: INFO: Container etcd-manager ready: true, restart count 0 Mar 11 16:33:10.692: INFO: etcd-manager-main-i-0e3467ec552af797e started at 2023-03-11 16:18:44 +0000 UTC (11+1 container statuses recorded) Mar 11 16:33:10.692: INFO: Init container init-etcd-3-2-24 ready: true, restart count 0 Mar 11 16:33:10.692: INFO: Init container init-etcd-3-3-10 ready: true, restart count 0 Mar 11 16:33:10.692: INFO: Init container init-etcd-3-3-17 ready: true, restart count 0 Mar 11 16:33:10.692: INFO: Init container init-etcd-3-4-13 ready: true, restart count 0 Mar 11 16:33:10.692: INFO: Init container init-etcd-3-4-3 ready: true, restart count 0 Mar 11 16:33:10.692: INFO: Init container init-etcd-3-5-0 ready: true, restart count 0 Mar 11 16:33:10.692: INFO: Init container init-etcd-3-5-1 ready: true, restart count 0 Mar 11 16:33:10.692: INFO: Init container init-etcd-3-5-3 ready: true, restart count 0 Mar 11 16:33:10.692: INFO: Init container init-etcd-3-5-4 ready: true, restart count 0 Mar 11 16:33:10.692: INFO: Init container init-etcd-3-5-6 ready: true, restart count 0 Mar 11 16:33:10.692: INFO: Init container init-etcd-3-5-7 ready: true, restart count 0 Mar 11 16:33:10.692: INFO: Container etcd-manager ready: true, restart count 0 Mar 11 16:33:10.692: INFO: ebs-csi-node-lpwz9 started at 2023-03-11 16:21:37 +0000 UTC (0+3 container statuses recorded) Mar 11 16:33:10.692: INFO: Container ebs-plugin ready: true, restart count 0 Mar 11 16:33:10.692: INFO: Container liveness-probe ready: true, restart count 0 Mar 11 16:33:10.692: INFO: Container node-driver-registrar ready: true, restart count 0 Mar 11 16:33:10.692: INFO: karpenter-webhook-5b646cd4f7-6sg2k started at 2023-03-11 16:21:58 +0000 UTC (0+1 container statuses recorded) Mar 11 16:33:10.692: INFO: Container webhook ready: true, restart count 0 Mar 11 16:33:11.194: INFO: Latency metrics for node i-0e3467ec552af797e END STEP: dump namespace information after failure - test/e2e/framework/framework.go:288 @ 03/11/23 16:33:11.194 (3.893s) < Exit [DeferCleanup (Each)] [sig-network] Services - dump namespaces | framework.go:209 @ 03/11/23 16:33:11.194 (3.893s) > Enter [DeferCleanup (Each)] [sig-network] Services - tear down framework | framework.go:206 @ 03/11/23 16:33:11.194 STEP: Destroying namespace "services-2895" for this suite. - test/e2e/framework/framework.go:351 @ 03/11/23 16:33:11.194 < Exit [DeferCleanup (Each)] [sig-network] Services - tear down framework | framework.go:206 @ 03/11/23 16:33:11.347 (153ms) > Enter [ReportAfterEach] TOP-LEVEL - test/e2e/e2e_test.go:144 @ 03/11/23 16:33:11.347 < Exit [ReportAfterEach] TOP-LEVEL - test/e2e/e2e_test.go:144 @ 03/11/23 16:33:11.347 (0s)
Filter through log files | View test history on testgrid
go run hack/e2e.go -v --test --test_args='--ginkgo.focus=Kubernetes\se2e\ssuite\s\[It\]\s\[sig\-node\]\sProbing\scontainer\sshould\s\*not\*\sbe\srestarted\swith\sa\sGRPC\sliveness\sprobe\s\[NodeConformance\]\s\[Conformance\]$'
[FAILED] pod container-probe-4792/test-grpc-21e6bf52-ac24-444f-9335-0685035e1c19 - expected number of restarts: 0, found restarts: 1 In [It] at: test/e2e/common/node/container_probe.go:1000 @ 03/11/23 16:41:55.206from junit_01.xml
> Enter [BeforeEach] [sig-node] Probing container - set up framework | framework.go:191 @ 03/11/23 16:37:50.907 STEP: Creating a kubernetes client - test/e2e/framework/framework.go:211 @ 03/11/23 16:37:50.907 Mar 11 16:37:50.907: INFO: >>> kubeConfig: /root/.kube/config STEP: Building a namespace api object, basename container-probe - test/e2e/framework/framework.go:250 @ 03/11/23 16:37:50.908 STEP: Waiting for a default service account to be provisioned in namespace - test/e2e/framework/framework.go:259 @ 03/11/23 16:37:51.352 STEP: Waiting for kube-root-ca.crt to be provisioned in namespace - test/e2e/framework/framework.go:262 @ 03/11/23 16:37:51.646 < Exit [BeforeEach] [sig-node] Probing container - set up framework | framework.go:191 @ 03/11/23 16:37:51.94 (1.033s) > Enter [BeforeEach] [sig-node] Probing container - test/e2e/framework/metrics/init/init.go:33 @ 03/11/23 16:37:51.94 < Exit [BeforeEach] [sig-node] Probing container - test/e2e/framework/metrics/init/init.go:33 @ 03/11/23 16:37:51.94 (0s) > Enter [BeforeEach] [sig-node] Probing container - test/e2e/common/node/container_probe.go:62 @ 03/11/23 16:37:51.94 < Exit [BeforeEach] [sig-node] Probing container - test/e2e/common/node/container_probe.go:62 @ 03/11/23 16:37:51.94 (0s) > Enter [It] should *not* be restarted with a GRPC liveness probe [NodeConformance] [Conformance] - test/e2e/common/node/container_probe.go:523 @ 03/11/23 16:37:51.94 STEP: Creating pod test-grpc-21e6bf52-ac24-444f-9335-0685035e1c19 in namespace container-probe-4792 - test/e2e/common/node/container_probe.go:955 @ 03/11/23 16:37:51.94 Mar 11 16:37:54.399: INFO: Started pod test-grpc-21e6bf52-ac24-444f-9335-0685035e1c19 in namespace container-probe-4792 STEP: checking the pod's current state and verifying that restartCount is present - test/e2e/common/node/container_probe.go:966 @ 03/11/23 16:37:54.399 Mar 11 16:37:54.546: INFO: Initial restart count of pod test-grpc-21e6bf52-ac24-444f-9335-0685035e1c19 is 0 Mar 11 16:40:07.920: INFO: Restart count of pod container-probe-4792/test-grpc-21e6bf52-ac24-444f-9335-0685035e1c19 is now 1 (2m13.373564885s elapsed) [FAILED] pod container-probe-4792/test-grpc-21e6bf52-ac24-444f-9335-0685035e1c19 - expected number of restarts: 0, found restarts: 1 In [It] at: test/e2e/common/node/container_probe.go:1000 @ 03/11/23 16:41:55.206 < Exit [It] should *not* be restarted with a GRPC liveness probe [NodeConformance] [Conformance] - test/e2e/common/node/container_probe.go:523 @ 03/11/23 16:41:55.206 (4m3.266s) > Enter [AfterEach] [sig-node] Probing container - test/e2e/framework/node/init/init.go:33 @ 03/11/23 16:41:55.206 Mar 11 16:41:55.206: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready < Exit [AfterEach] [sig-node] Probing container - test/e2e/framework/node/init/init.go:33 @ 03/11/23 16:41:55.357 (151ms) > Enter [DeferCleanup (Each)] [sig-node] Probing container - test/e2e/common/node/container_probe.go:951 @ 03/11/23 16:41:55.357 STEP: deleting the pod - test/e2e/common/node/container_probe.go:952 @ 03/11/23 16:41:55.357 < Exit [DeferCleanup (Each)] [sig-node] Probing container - test/e2e/common/node/container_probe.go:951 @ 03/11/23 16:41:55.512 (155ms) > Enter [DeferCleanup (Each)] [sig-node] Probing container - test/e2e/framework/metrics/init/init.go:35 @ 03/11/23 16:41:55.512 < Exit [DeferCleanup (Each)] [sig-node] Probing container - test/e2e/framework/metrics/init/init.go:35 @ 03/11/23 16:41:55.512 (0s) > Enter [DeferCleanup (Each)] [sig-node] Probing container - dump namespaces | framework.go:209 @ 03/11/23 16:41:55.512 STEP: dump namespace information after failure - test/e2e/framework/framework.go:288 @ 03/11/23 16:41:55.512 STEP: Collecting events from namespace "container-probe-4792". - test/e2e/framework/debug/dump.go:42 @ 03/11/23 16:41:55.512 STEP: Found 7 events. - test/e2e/framework/debug/dump.go:46 @ 03/11/23 16:41:55.66 Mar 11 16:41:55.660: INFO: At 2023-03-11 16:37:52 +0000 UTC - event for test-grpc-21e6bf52-ac24-444f-9335-0685035e1c19: {default-scheduler } Scheduled: Successfully assigned container-probe-4792/test-grpc-21e6bf52-ac24-444f-9335-0685035e1c19 to i-07e2b6e650d7ef56d Mar 11 16:41:55.660: INFO: At 2023-03-11 16:37:52 +0000 UTC - event for test-grpc-21e6bf52-ac24-444f-9335-0685035e1c19: {kubelet i-07e2b6e650d7ef56d} Pulled: Container image "registry.k8s.io/e2e-test-images/agnhost:2.43" already present on machine Mar 11 16:41:55.660: INFO: At 2023-03-11 16:37:52 +0000 UTC - event for test-grpc-21e6bf52-ac24-444f-9335-0685035e1c19: {kubelet i-07e2b6e650d7ef56d} Created: Created container agnhost Mar 11 16:41:55.660: INFO: At 2023-03-11 16:37:52 +0000 UTC - event for test-grpc-21e6bf52-ac24-444f-9335-0685035e1c19: {kubelet i-07e2b6e650d7ef56d} Started: Started container agnhost Mar 11 16:41:55.660: INFO: At 2023-03-11 16:38:57 +0000 UTC - event for test-grpc-21e6bf52-ac24-444f-9335-0685035e1c19: {kubelet i-07e2b6e650d7ef56d} Unhealthy: Liveness probe failed: timeout: failed to connect service "100.96.1.184:5000" within 5s: context deadline exceeded Mar 11 16:41:55.660: INFO: At 2023-03-11 16:39:25 +0000 UTC - event for test-grpc-21e6bf52-ac24-444f-9335-0685035e1c19: {kubelet i-07e2b6e650d7ef56d} Killing: Container agnhost failed liveness probe, will be restarted Mar 11 16:41:55.660: INFO: At 2023-03-11 16:41:55 +0000 UTC - event for test-grpc-21e6bf52-ac24-444f-9335-0685035e1c19: {kubelet i-07e2b6e650d7ef56d} Killing: Stopping container agnhost Mar 11 16:41:55.808: INFO: POD NODE PHASE GRACE CONDITIONS Mar 11 16:41:55.808: INFO: Mar 11 16:41:55.959: INFO: Logging node info for node i-04523d1dffb0fe725 Mar 11 16:41:56.107: INFO: Node Info: &Node{ObjectMeta:{i-04523d1dffb0fe725 5d37640f-a7d9-4c26-86c5-9da21f7ccc18 37780 0 2023-03-11 16:24:20 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:arm64 beta.kubernetes.io/instance-type:t4g.small beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-northeast-1 failure-domain.beta.kubernetes.io/zone:ap-northeast-1a karpenter.k8s.aws/instance-category:t karpenter.k8s.aws/instance-cpu:2 karpenter.k8s.aws/instance-family:t4g karpenter.k8s.aws/instance-generation:4 karpenter.k8s.aws/instance-hypervisor:nitro karpenter.k8s.aws/instance-memory:2048 karpenter.k8s.aws/instance-pods:11 karpenter.k8s.aws/instance-size:small karpenter.sh/capacity-type:spot karpenter.sh/initialized:true karpenter.sh/provisioner-name:nodes kubernetes.io/arch:arm64 kubernetes.io/hostname:i-04523d1dffb0fe725 kubernetes.io/os:linux node-role.kubernetes.io/node: node-role.kubernetes.io/spot-worker:true node.kubernetes.io/instance-type:t4g.small topology.ebs.csi.aws.com/zone:ap-northeast-1a topology.hostpath.csi/node:i-04523d1dffb0fe725 topology.kubernetes.io/region:ap-northeast-1 topology.kubernetes.io/zone:ap-northeast-1a] map[csi.volume.kubernetes.io/nodeid:{"csi-hostpath-provisioning-3750":"i-04523d1dffb0fe725","csi-mock-csi-mock-volumes-expansion-1028":"i-04523d1dffb0fe725","ebs.csi.aws.com":"i-04523d1dffb0fe725"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [{karpenter.sh/v1alpha5 Provisioner nodes ac7283b2-eed0-4a9f-9ddd-616f33a0991a <nil> 0xc0048095a5}] [karpenter.sh/termination] [{aws-cloud-controller-manager Update v1 2023-03-11 16:24:20 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{}}}} } {kops-controller Update v1 2023-03-11 16:24:20 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{},"f:node-role.kubernetes.io/spot-worker":{}}}} } {karpenter Update v1 2023-03-11 16:26:12 +0000 UTC FieldsV1 {"f:metadata":{"f:finalizers":{".":{},"v:\"karpenter.sh/termination\"":{}},"f:labels":{".":{},"f:karpenter.k8s.aws/instance-category":{},"f:karpenter.k8s.aws/instance-cpu":{},"f:karpenter.k8s.aws/instance-family":{},"f:karpenter.k8s.aws/instance-generation":{},"f:karpenter.k8s.aws/instance-hypervisor":{},"f:karpenter.k8s.aws/instance-memory":{},"f:karpenter.k8s.aws/instance-pods":{},"f:karpenter.k8s.aws/instance-size":{},"f:karpenter.sh/capacity-type":{},"f:karpenter.sh/initialized":{},"f:karpenter.sh/provisioner-name":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}},"f:ownerReferences":{".":{},"k:{\"uid\":\"ac7283b2-eed0-4a9f-9ddd-616f33a0991a\"}":{}}},"f:spec":{"f:providerID":{}}} } {kube-controller-manager Update v1 2023-03-11 16:26:12 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:node.alpha.kubernetes.io/ttl":{}},"f:labels":{"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.2.0/24\"":{}}}} } {aws-cloud-controller-manager Update v1 2023-03-11 16:26:50 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{".":{},"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"InternalIP\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kube-controller-manager Update v1 2023-03-11 16:41:52 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{".":{},"k:{\"type\":\"DiskPressure\"}":{".":{},"f:type":{}},"k:{\"type\":\"MemoryPressure\"}":{".":{},"f:type":{}},"k:{\"type\":\"PIDPressure\"}":{".":{},"f:type":{}},"k:{\"type\":\"Ready\"}":{".":{},"f:type":{}}},"f:volumesAttached":{}}} status} {kubelet Update v1 2023-03-11 16:41:52 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{"f:kubernetes.io/hostname":{},"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{".":{},"f:cpu":{},"f:ephemeral-storage":{},"f:hugepages-1Gi":{},"f:hugepages-2Mi":{},"f:hugepages-32Mi":{},"f:hugepages-64Ki":{},"f:memory":{},"f:pods":{}},"f:capacity":{".":{},"f:cpu":{},"f:ephemeral-storage":{},"f:hugepages-1Gi":{},"f:hugepages-2Mi":{},"f:hugepages-32Mi":{},"f:hugepages-64Ki":{},"f:memory":{},"f:pods":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:daemonEndpoints":{"f:kubeletEndpoint":{"f:Port":{}}},"f:images":{},"f:nodeInfo":{"f:architecture":{},"f:bootID":{},"f:containerRuntimeVersion":{},"f:kernelVersion":{},"f:kubeProxyVersion":{},"f:kubeletVersion":{},"f:machineID":{},"f:operatingSystem":{},"f:osImage":{},"f:systemUUID":{}},"f:volumesInUse":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.2.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-northeast-1a/i-04523d1dffb0fe725,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.2.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49756430336 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-32Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-64Ki: {{0 0} {<nil>} 0 DecimalSI},memory: {{1941540864 0} {<nil>} 1896036Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44780787229 0} {<nil>} 44780787229 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-32Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-64Ki: {{0 0} {<nil>} 0 DecimalSI},memory: {{1836683264 0} {<nil>} 1793636Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-03-11 16:41:52 +0000 UTC,LastTransitionTime:2023-03-11 16:26:12 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-03-11 16:41:52 +0000 UTC,LastTransitionTime:2023-03-11 16:26:02 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-03-11 16:41:52 +0000 UTC,LastTransitionTime:2023-03-11 16:26:02 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-03-11 16:41:52 +0000 UTC,LastTransitionTime:2023-03-11 16:26:02 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.48.168,},NodeAddress{Type:ExternalIP,Address:52.199.253.156,},NodeAddress{Type:InternalDNS,Address:i-04523d1dffb0fe725.ap-northeast-1.compute.internal,},NodeAddress{Type:Hostname,Address:i-04523d1dffb0fe725.ap-northeast-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-52-199-253-156.ap-northeast-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2c8fb0e6b2e9c4af310f65b6161cd8,SystemUUID:ec2c8fb0-e6b2-e9c4-af31-0f65b6161cd8,BootID:a3b8de83-558b-44c4-a46a-d097c3f98225,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 22.04.2 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.27.0-alpha.3.420+bb6c6fad2c19b3,KubeProxyVersion:v1.27.0-alpha.3.420+bb6c6fad2c19b3,OperatingSystem:linux,Architecture:arm64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:157636062,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/jessie-dnsutils@sha256:24aaf2626d6b27864c29de2097e8bbb840b3a414271bf7c8995e431e47d8408e registry.k8s.io/e2e-test-images/jessie-dnsutils:1.7],SizeBytes:116764683,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:51eae8381dcb1078289fa7b4f3df2630cdc18d09fb56f8e56b41c40e191d6c83 registry.k8s.io/etcd:3.5.7-0],SizeBytes:80665728,},ContainerImage{Names:[registry.k8s.io/kube-proxy-arm64:v1.27.0-alpha.3.420_bb6c6fad2c19b3],SizeBytes:60947298,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:16bbf38c463a4223d8cfe4da12bc61010b082a79b4bb003e2d3ba3ece5dd5f9e registry.k8s.io/e2e-test-images/agnhost:2.43],SizeBytes:49808364,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nautilus@sha256:80ba6c8c44f9623f06e868a1aa66026c8ec438ad814f9ec95e9333b415fe3550 registry.k8s.io/e2e-test-images/nautilus:1.7],SizeBytes:47953400,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:3fe7acf013d1264ffded116b80a73dc129a449b0fccdb8d21af8279f2233f36e registry.k8s.io/e2e-test-images/httpd:2.4.39-4],SizeBytes:41528818,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:148b022f5c5da426fc2f3c14b5c0867e58ef05961510c84749ac1fddcb0fef22 registry.k8s.io/e2e-test-images/httpd:2.4.38-4],SizeBytes:40343601,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:28932904,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:ee3b525d5b89db99da3b8eb521d9cd90cb6e9ef0fbb651e98bb37be78d36b5b8 registry.k8s.io/sig-storage/csi-provisioner:v3.3.0],SizeBytes:23945474,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/sample-apiserver@sha256:8d70890151aa5d096f331cb9da1b9cd5be0412b7363fe67b5c3befdcaa2a28d0 registry.k8s.io/e2e-test-images/sample-apiserver:1.17.7],SizeBytes:23857462,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:425d8f1b769398127767b06ed97ce62578a3179bcb99809ce93a1649e025ffe7 registry.k8s.io/sig-storage/csi-resizer:v1.6.0],SizeBytes:22678632,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:291334908ddf71a4661fd7f6d9d97274de8a5378a2b6fdfeb2ce73414a34f82f registry.k8s.io/sig-storage/csi-snapshotter:v6.1.0],SizeBytes:22434180,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:9a685020911e2725ad019dbce6e4a5ab93d51e3d4557f115e64343345e05781b registry.k8s.io/sig-storage/csi-attacher:v4.0.0],SizeBytes:22394501,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:122bfb8c1edabb3c0edd63f06523e6940d958d19b3957dc7b1d6f81e9f1f6119 registry.k8s.io/sig-storage/csi-provisioner:v3.1.0],SizeBytes:21735644,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:9ebbf9f023e7b41ccee3d52afe39a89e3ddacdbb69269d583abfc25847cfd9e4 registry.k8s.io/sig-storage/csi-resizer:v1.4.0],SizeBytes:20848413,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:8b9c313c05f54fb04f8d430896f5f5904b6cb157df261501b29adc04d2b2dc7b registry.k8s.io/sig-storage/csi-attacher:v3.4.0],SizeBytes:20565480,},ContainerImage{Names:[registry.k8s.io/cpa/cluster-proportional-autoscaler@sha256:68d396900aeaa072c1f27289485fdac29834045a6f3ffe369bf389d830ef572d registry.k8s.io/cpa/cluster-proportional-autoscaler:1.8.6],SizeBytes:19007623,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nonroot@sha256:ee9f50b3c64b174d296d91ca9f69a914ac30e59095dfb462b2b518ad28a63655 registry.k8s.io/e2e-test-images/nonroot:1.4],SizeBytes:18561972,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:92257881c1d6493cf18299a24af42330f891166560047902b8d431fb66b01af5 registry.k8s.io/sig-storage/hostpathplugin:v1.9.0],SizeBytes:18267611,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:9b2d9abda017c28b12f84a344f57cd73fbdb6c2bd7dd5bdd5018246ad1093ba6 registry.k8s.io/sig-storage/hostpathplugin:v1.11.0],SizeBytes:18159085,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:017727efcfeb7d053af68e51436ce8e65edbc6ca573720afb4f79c8594036955 registry.k8s.io/coredns/coredns:v1.10.0],SizeBytes:13766234,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:8502345,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:933940f13b3ea0abc62e656c1aa5c5b47c04b15d71250413a6b821bd0c58b94e registry.k8s.io/sig-storage/livenessprobe:v2.7.0],SizeBytes:8223819,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:7685305,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:db048754ae68ae337d8fa96494c96d2a1204c3320f5dcf7e8e71085adec85da6 registry.k8s.io/e2e-test-images/nginx:1.15-4],SizeBytes:6883347,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:5c99cf6a02adda929b10321dbf4ecfa00d87be9ba4fb456006237d530ab4baa1 registry.k8s.io/e2e-test-images/nginx:1.14-4],SizeBytes:6860089,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf registry.k8s.io/e2e-test-images/busybox:1.29-2],SizeBytes:773367,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:2e0f836850e09b8b7cc937681d6194537a09fbd5f6b9e08f4d646a85128e8937 registry.k8s.io/e2e-test-images/busybox:1.29-4],SizeBytes:772933,},ContainerImage{Names:[registry.k8s.io/pause@sha256:7031c1b283388d2c2e09b57badb803c05ebed362dc88d84b480cc47f72a21097 registry.k8s.io/pause:3.9],SizeBytes:268051,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:253553,},},VolumesInUse:[kubernetes.io/csi/csi-hostpath-provisioning-3750^9b4f98a4-c02b-11ed-825d-2ad5e7954bcb kubernetes.io/csi/ebs.csi.aws.com^vol-09348d3cba5a1265a],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/csi-hostpath-provisioning-3750^9b4f98a4-c02b-11ed-825d-2ad5e7954bcb,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-09348d3cba5a1265a,DevicePath:,},},Config:nil,},} Mar 11 16:41:56.107: INFO: Logging kubelet events for node i-04523d1dffb0fe725 Mar 11 16:41:56.258: INFO: Logging pods the kubelet thinks is on node i-04523d1dffb0fe725 Mar 11 16:41:56.415: INFO: pod-secrets-e98ccbd1-9959-4bb7-bce6-4839b50d0c44 started at 2023-03-11 16:41:51 +0000 UTC (0+1 container statuses recorded) Mar 11 16:41:56.415: INFO: Container secret-volume-test ready: false, restart count 0 Mar 11 16:41:56.415: INFO: csi-mockplugin-resizer-0 started at 2023-03-11 16:40:30 +0000 UTC (0+1 container statuses recorded) Mar 11 16:41:56.415: INFO: Container csi-resizer ready: true, restart count 0 Mar 11 16:41:56.415: INFO: pod-1 started at 2023-03-11 16:41:47 +0000 UTC (0+1 container statuses recorded) Mar 11 16:41:56.415: INFO: Container donothing ready: true, restart count 0 Mar 11 16:41:56.415: INFO: coredns-autoscaler-8495f5575-d8gqb started at 2023-03-11 16:26:12 +0000 UTC (0+1 container statuses recorded) Mar 11 16:41:56.415: INFO: Container autoscaler ready: true, restart count 0 Mar 11 16:41:56.415: INFO: external-client started at 2023-03-11 16:41:33 +0000 UTC (0+1 container statuses recorded) Mar 11 16:41:56.415: INFO: Container external-client ready: true, restart count 0 Mar 11 16:41:56.415: INFO: coredns-745bddcfbb-s57v5 started at 2023-03-11 16:26:12 +0000 UTC (0+1 container statuses recorded) Mar 11 16:41:56.415: INFO: Container coredns ready: true, restart count 0 Mar 11 16:41:56.415: INFO: execpod59vnr started at 2023-03-11 16:37:56 +0000 UTC (0+1 container statuses recorded) Mar 11 16:41:56.415: INFO: Container agnhost-container ready: true, restart count 0 Mar 11 16:41:56.415: INFO: probe-test-5acff1c9-0661-4994-a50d-95ff62b20366 started at 2023-03-11 16:41:35 +0000 UTC (0+1 container statuses recorded) Mar 11 16:41:56.415: INFO: Container probe-test-5acff1c9-0661-4994-a50d-95ff62b20366 ready: false, restart count 0 Mar 11 16:41:56.415: INFO: cilium-kfwlg started at 2023-03-11 16:26:03 +0000 UTC (1+1 container statuses recorded) Mar 11 16:41:56.415: INFO: Init container clean-cilium-state ready: true, restart count 0 Mar 11 16:41:56.415: INFO: Container cilium-agent ready: true, restart count 0 Mar 11 16:41:56.415: INFO: ebs-csi-controller-79785c665b-2wwsh started at 2023-03-11 16:26:12 +0000 UTC (0+5 container statuses recorded) Mar 11 16:41:56.415: INFO: Container csi-attacher ready: true, restart count 0 Mar 11 16:41:56.415: INFO: Container csi-provisioner ready: true, restart count 0 Mar 11 16:41:56.415: INFO: Container csi-resizer ready: true, restart count 0 Mar 11 16:41:56.415: INFO: Container ebs-plugin ready: true, restart count 0 Mar 11 16:41:56.415: INFO: Container liveness-probe ready: true, restart count 0 Mar 11 16:41:56.415: INFO: alpine-nnp-false-0927d54c-fbca-4f4d-bf96-2ad5c96a2d34 started at 2023-03-11 16:41:51 +0000 UTC (0+1 container statuses recorded) Mar 11 16:41:56.415: INFO: Container alpine-nnp-false-0927d54c-fbca-4f4d-bf96-2ad5c96a2d34 ready: false, restart count 0 Mar 11 16:41:56.415: INFO: pause-pod-1 started at 2023-03-11 16:41:49 +0000 UTC (0+1 container statuses recorded) Mar 11 16:41:56.415: INFO: Container agnhost-container ready: true, restart count 0 Mar 11 16:41:56.415: INFO: pod-subpath-test-dynamicpv-h5r5 started at 2023-03-11 16:41:45 +0000 UTC (0+2 container statuses recorded) Mar 11 16:41:56.415: INFO: Container test-container-subpath-dynamicpv-h5r5 ready: false, restart count 0 Mar 11 16:41:56.415: INFO: Container test-container-volume-dynamicpv-h5r5 ready: true, restart count 0 Mar 11 16:41:56.415: INFO: csi-mockplugin-attacher-0 started at 2023-03-11 16:40:30 +0000 UTC (0+1 container statuses recorded) Mar 11 16:41:56.415: INFO: Container csi-attacher ready: true, restart count 0 Mar 11 16:41:56.415: INFO: pod-0 started at 2023-03-11 16:41:47 +0000 UTC (0+1 container statuses recorded) Mar 11 16:41:56.415: INFO: Container donothing ready: true, restart count 0 Mar 11 16:41:56.415: INFO: hostexec-i-04523d1dffb0fe725-hcf9g started at 2023-03-11 16:41:52 +0000 UTC (0+1 container statuses recorded) Mar 11 16:41:56.415: INFO: Container agnhost-container ready: true, restart count 0 Mar 11 16:41:56.415: INFO: ebs-csi-node-2qlbt started at 2023-03-11 16:26:03 +0000 UTC (0+3 container statuses recorded) Mar 11 16:41:56.415: INFO: Container ebs-plugin ready: true, restart count 0 Mar 11 16:41:56.415: INFO: Container liveness-probe ready: true, restart count 0 Mar 11 16:41:56.415: INFO: Container node-driver-registrar ready: true, restart count 0 Mar 11 16:41:56.415: INFO: csi-hostpathplugin-0 started at 2023-03-11 16:41:41 +0000 UTC (0+7 container statuses recorded) Mar 11 16:41:56.415: INFO: Container csi-attacher ready: true, restart count 0 Mar 11 16:41:56.415: INFO: Container csi-provisioner ready: true, restart count 0 Mar 11 16:41:56.415: INFO: Container csi-resizer ready: true, restart count 0 Mar 11 16:41:56.415: INFO: Container csi-snapshotter ready: true, restart count 0 Mar 11 16:41:56.415: INFO: Container hostpath ready: true, restart count 0 Mar 11 16:41:56.415: INFO: Container liveness-probe ready: true, restart count 0 Mar 11 16:41:56.415: INFO: Container node-driver-registrar ready: true, restart count 0 Mar 11 16:41:56.415: INFO: csi-mockplugin-0 started at 2023-03-11 16:40:30 +0000 UTC (0+3 container statuses recorded) Mar 11 16:41:56.415: INFO: Container csi-provisioner ready: true, restart count 0 Mar 11 16:41:56.415: INFO: Container driver-registrar ready: true, restart count 0 Mar 11 16:41:56.415: INFO: Container mock ready: true, restart count 0 Mar 11 16:41:56.415: INFO: webserver-pod started at 2023-03-11 16:41:47 +0000 UTC (0+1 container statuses recorded) Mar 11 16:41:56.415: INFO: Container agnhost-container ready: true, restart count 0 Mar 11 16:41:57.114: INFO: Latency metrics for node i-04523d1dffb0fe725 Mar 11 16:41:57.114: INFO: Logging node info for node i-07e2b6e650d7ef56d Mar 11 16:41:57.262: INFO: Node Info: &Node{ObjectMeta:{i-07e2b6e650d7ef56d fcc65272-6f1b-4228-ac59-e9e3e92cdb2c 37671 0 2023-03-11 16:24:19 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:arm64 beta.kubernetes.io/instance-type:t4g.small beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-northeast-1 failure-domain.beta.kubernetes.io/zone:ap-northeast-1a karpenter.k8s.aws/instance-category:t karpenter.k8s.aws/instance-cpu:2 karpenter.k8s.aws/instance-family:t4g karpenter.k8s.aws/instance-generation:4 karpenter.k8s.aws/instance-hypervisor:nitro karpenter.k8s.aws/instance-memory:2048 karpenter.k8s.aws/instance-pods:11 karpenter.k8s.aws/instance-size:small karpenter.sh/capacity-type:spot karpenter.sh/initialized:true karpenter.sh/provisioner-name:nodes kubernetes.io/arch:arm64 kubernetes.io/hostname:i-07e2b6e650d7ef56d kubernetes.io/os:linux node-role.kubernetes.io/node: node-role.kubernetes.io/spot-worker:true node.kubernetes.io/instance-type:t4g.small topology.ebs.csi.aws.com/zone:ap-northeast-1a topology.hostpath.csi/node:i-07e2b6e650d7ef56d topology.kubernetes.io/region:ap-northeast-1 topology.kubernetes.io/zone:ap-northeast-1a] map[csi.volume.kubernetes.io/nodeid:{"csi-mock-csi-mock-volumes-service-token-9125":"i-07e2b6e650d7ef56d","csi-mock-csi-mock-volumes-workload-4674":"i-07e2b6e650d7ef56d","ebs.csi.aws.com":"i-07e2b6e650d7ef56d"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [{karpenter.sh/v1alpha5 Provisioner nodes ac7283b2-eed0-4a9f-9ddd-616f33a0991a <nil> 0xc00468a4f5}] [karpenter.sh/termination] [{aws-cloud-controller-manager Update v1 2023-03-11 16:24:20 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{}}}} } {kops-controller Update v1 2023-03-11 16:24:20 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{},"f:node-role.kubernetes.io/spot-worker":{}}}} } {karpenter Update v1 2023-03-11 16:26:15 +0000 UTC FieldsV1 {"f:metadata":{"f:finalizers":{".":{},"v:\"karpenter.sh/termination\"":{}},"f:labels":{".":{},"f:karpenter.k8s.aws/instance-category":{},"f:karpenter.k8s.aws/instance-cpu":{},"f:karpenter.k8s.aws/instance-family":{},"f:karpenter.k8s.aws/instance-generation":{},"f:karpenter.k8s.aws/instance-hypervisor":{},"f:karpenter.k8s.aws/instance-memory":{},"f:karpenter.k8s.aws/instance-pods":{},"f:karpenter.k8s.aws/instance-size":{},"f:karpenter.sh/capacity-type":{},"f:karpenter.sh/initialized":{},"f:karpenter.sh/provisioner-name":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}},"f:ownerReferences":{".":{},"k:{\"uid\":\"ac7283b2-eed0-4a9f-9ddd-616f33a0991a\"}":{}}},"f:spec":{"f:providerID":{}}} } {kube-controller-manager Update v1 2023-03-11 16:26:15 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:node.alpha.kubernetes.io/ttl":{}},"f:labels":{"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.1.0/24\"":{}}}} } {aws-cloud-controller-manager Update v1 2023-03-11 16:26:50 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{".":{},"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"InternalIP\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kube-controller-manager Update v1 2023-03-11 16:41:45 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{".":{},"k:{\"type\":\"DiskPressure\"}":{".":{},"f:type":{}},"k:{\"type\":\"MemoryPressure\"}":{".":{},"f:type":{}},"k:{\"type\":\"PIDPressure\"}":{".":{},"f:type":{}},"k:{\"type\":\"Ready\"}":{".":{},"f:type":{}}},"f:volumesAttached":{}}} status} {kubelet Update v1 2023-03-11 16:41:50 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{"f:kubernetes.io/hostname":{},"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{".":{},"f:cpu":{},"f:ephemeral-storage":{},"f:hugepages-1Gi":{},"f:hugepages-2Mi":{},"f:hugepages-32Mi":{},"f:hugepages-64Ki":{},"f:memory":{},"f:pods":{}},"f:capacity":{".":{},"f:cpu":{},"f:ephemeral-storage":{},"f:hugepages-1Gi":{},"f:hugepages-2Mi":{},"f:hugepages-32Mi":{},"f:hugepages-64Ki":{},"f:memory":{},"f:pods":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:daemonEndpoints":{"f:kubeletEndpoint":{"f:Port":{}}},"f:images":{},"f:nodeInfo":{"f:architecture":{},"f:bootID":{},"f:containerRuntimeVersion":{},"f:kernelVersion":{},"f:kubeProxyVersion":{},"f:kubeletVersion":{},"f:machineID":{},"f:operatingSystem":{},"f:osImage":{},"f:systemUUID":{}},"f:volumesInUse":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.1.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-northeast-1a/i-07e2b6e650d7ef56d,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.1.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49756430336 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-32Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-64Ki: {{0 0} {<nil>} 0 DecimalSI},memory: {{1941549056 0} {<nil>} 1896044Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44780787229 0} {<nil>} 44780787229 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-32Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-64Ki: {{0 0} {<nil>} 0 DecimalSI},memory: {{1836691456 0} {<nil>} 1793644Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-03-11 16:41:49 +0000 UTC,LastTransitionTime:2023-03-11 16:26:15 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-03-11 16:41:49 +0000 UTC,LastTransitionTime:2023-03-11 16:26:05 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-03-11 16:41:49 +0000 UTC,LastTransitionTime:2023-03-11 16:26:05 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-03-11 16:41:49 +0000 UTC,LastTransitionTime:2023-03-11 16:26:05 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.35.76,},NodeAddress{Type:ExternalIP,Address:54.238.106.83,},NodeAddress{Type:InternalDNS,Address:i-07e2b6e650d7ef56d.ap-northeast-1.compute.internal,},NodeAddress{Type:Hostname,Address:i-07e2b6e650d7ef56d.ap-northeast-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-54-238-106-83.ap-northeast-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2083a8e9d27d02fac967340367f875,SystemUUID:ec2083a8-e9d2-7d02-fac9-67340367f875,BootID:ea264ed6-5b0d-47d6-ad5d-6bd2a1f7c11b,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 22.04.2 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.27.0-alpha.3.420+bb6c6fad2c19b3,KubeProxyVersion:v1.27.0-alpha.3.420+bb6c6fad2c19b3,OperatingSystem:linux,Architecture:arm64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:157636062,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/jessie-dnsutils@sha256:24aaf2626d6b27864c29de2097e8bbb840b3a414271bf7c8995e431e47d8408e registry.k8s.io/e2e-test-images/jessie-dnsutils:1.7],SizeBytes:116764683,},ContainerImage{Names:[registry.k8s.io/kube-proxy-arm64:v1.27.0-alpha.3.420_bb6c6fad2c19b3],SizeBytes:60947298,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:16bbf38c463a4223d8cfe4da12bc61010b082a79b4bb003e2d3ba3ece5dd5f9e registry.k8s.io/e2e-test-images/agnhost:2.43],SizeBytes:49808364,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nautilus@sha256:80ba6c8c44f9623f06e868a1aa66026c8ec438ad814f9ec95e9333b415fe3550 registry.k8s.io/e2e-test-images/nautilus:1.7],SizeBytes:47953400,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:3fe7acf013d1264ffded116b80a73dc129a449b0fccdb8d21af8279f2233f36e registry.k8s.io/e2e-test-images/httpd:2.4.39-4],SizeBytes:41528818,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:148b022f5c5da426fc2f3c14b5c0867e58ef05961510c84749ac1fddcb0fef22 registry.k8s.io/e2e-test-images/httpd:2.4.38-4],SizeBytes:40343601,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:28932904,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:ee3b525d5b89db99da3b8eb521d9cd90cb6e9ef0fbb651e98bb37be78d36b5b8 registry.k8s.io/sig-storage/csi-provisioner:v3.3.0],SizeBytes:23945474,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:425d8f1b769398127767b06ed97ce62578a3179bcb99809ce93a1649e025ffe7 registry.k8s.io/sig-storage/csi-resizer:v1.6.0],SizeBytes:22678632,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:291334908ddf71a4661fd7f6d9d97274de8a5378a2b6fdfeb2ce73414a34f82f registry.k8s.io/sig-storage/csi-snapshotter:v6.1.0],SizeBytes:22434180,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:9a685020911e2725ad019dbce6e4a5ab93d51e3d4557f115e64343345e05781b registry.k8s.io/sig-storage/csi-attacher:v4.0.0],SizeBytes:22394501,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:122bfb8c1edabb3c0edd63f06523e6940d958d19b3957dc7b1d6f81e9f1f6119 registry.k8s.io/sig-storage/csi-provisioner:v3.1.0],SizeBytes:21735644,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:9ebbf9f023e7b41ccee3d52afe39a89e3ddacdbb69269d583abfc25847cfd9e4 registry.k8s.io/sig-storage/csi-resizer:v1.4.0],SizeBytes:20848413,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:8b9c313c05f54fb04f8d430896f5f5904b6cb157df261501b29adc04d2b2dc7b registry.k8s.io/sig-storage/csi-attacher:v3.4.0],SizeBytes:20565480,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nonroot@sha256:ee9f50b3c64b174d296d91ca9f69a914ac30e59095dfb462b2b518ad28a63655 registry.k8s.io/e2e-test-images/nonroot:1.4],SizeBytes:18561972,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:92257881c1d6493cf18299a24af42330f891166560047902b8d431fb66b01af5 registry.k8s.io/sig-storage/hostpathplugin:v1.9.0],SizeBytes:18267611,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:9b2d9abda017c28b12f84a344f57cd73fbdb6c2bd7dd5bdd5018246ad1093ba6 registry.k8s.io/sig-storage/hostpathplugin:v1.11.0],SizeBytes:18159085,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:017727efcfeb7d053af68e51436ce8e65edbc6ca573720afb4f79c8594036955 registry.k8s.io/coredns/coredns:v1.10.0],SizeBytes:13766234,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:8502345,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:933940f13b3ea0abc62e656c1aa5c5b47c04b15d71250413a6b821bd0c58b94e registry.k8s.io/sig-storage/livenessprobe:v2.7.0],SizeBytes:8223819,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:7685305,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:db048754ae68ae337d8fa96494c96d2a1204c3320f5dcf7e8e71085adec85da6 registry.k8s.io/e2e-test-images/nginx:1.15-4],SizeBytes:6883347,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:5c99cf6a02adda929b10321dbf4ecfa00d87be9ba4fb456006237d530ab4baa1 registry.k8s.io/e2e-test-images/nginx:1.14-4],SizeBytes:6860089,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nonewprivs@sha256:8ac1264691820febacf3aea5d152cbde6d10685731ec14966a9401c6f47a68ac registry.k8s.io/e2e-test-images/nonewprivs:1.3],SizeBytes:5025027,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf registry.k8s.io/e2e-test-images/busybox:1.29-2],SizeBytes:773367,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:2e0f836850e09b8b7cc937681d6194537a09fbd5f6b9e08f4d646a85128e8937 registry.k8s.io/e2e-test-images/busybox:1.29-4],SizeBytes:772933,},ContainerImage{Names:[registry.k8s.io/pause@sha256:7031c1b283388d2c2e09b57badb803c05ebed362dc88d84b480cc47f72a21097 registry.k8s.io/pause:3.9],SizeBytes:268051,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:253553,},},VolumesInUse:[kubernetes.io/csi/ebs.csi.aws.com^vol-03fbd90ab608db8d0 kubernetes.io/csi/ebs.csi.aws.com^vol-05f5719c7bc983c37],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-05f5719c7bc983c37,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-03fbd90ab608db8d0,DevicePath:,},},Config:nil,},} Mar 11 16:41:57.263: INFO: Logging kubelet events for node i-07e2b6e650d7ef56d Mar 11 16:41:57.421: INFO: Logging pods the kubelet thinks is on node i-07e2b6e650d7ef56d Mar 11 16:41:57.579: INFO: inline-volume-gkrb2 started at 2023-03-11 16:41:30 +0000 UTC (0+1 container statuses recorded) Mar 11 16:41:57.579: INFO: Container volume-tester ready: false, restart count 0 Mar 11 16:41:57.579: INFO: coredns-745bddcfbb-lkbct started at 2023-03-11 16:26:41 +0000 UTC (0+1 container statuses recorded) Mar 11 16:41:57.579: INFO: Container coredns ready: true, restart count 0 Mar 11 16:41:57.579: INFO: pod-0 started at 2023-03-11 16:41:26 +0000 UTC (0+1 container statuses recorded) Mar 11 16:41:57.579: INFO: Container donothing ready: true, restart count 0 Mar 11 16:41:57.579: INFO: pvc-volume-tester-zq7k2 started at 2023-03-11 16:41:28 +0000 UTC (0+1 container statuses recorded) Mar 11 16:41:57.579: INFO: Container volume-tester ready: true, restart count 0 Mar 11 16:41:57.579: INFO: cilium-mvm8l started at 2023-03-11 16:26:06 +0000 UTC (1+1 container statuses recorded) Mar 11 16:41:57.579: INFO: Init container clean-cilium-state ready: true, restart count 0 Mar 11 16:41:57.579: INFO: Container cilium-agent ready: true, restart count 0 Mar 11 16:41:57.579: INFO: ebs-csi-node-m5b9f started at 2023-03-11 16:26:06 +0000 UTC (0+3 container statuses recorded) Mar 11 16:41:57.579: INFO: Container ebs-plugin ready: true, restart count 1 Mar 11 16:41:57.579: INFO: Container liveness-probe ready: true, restart count 0 Mar 11 16:41:57.579: INFO: Container node-driver-registrar ready: true, restart count 0 Mar 11 16:41:57.579: INFO: csi-mockplugin-attacher-0 started at 2023-03-11 16:41:25 +0000 UTC (0+1 container statuses recorded) Mar 11 16:41:57.579: INFO: Container csi-attacher ready: true, restart count 0 Mar 11 16:41:57.579: INFO: pod-2 started at 2023-03-11 16:41:27 +0000 UTC (0+1 container statuses recorded) Mar 11 16:41:57.579: INFO: Container donothing ready: true, restart count 0 Mar 11 16:41:57.579: INFO: funny-ip-9w5ps started at 2023-03-11 16:37:52 +0000 UTC (0+1 container statuses recorded) Mar 11 16:41:57.579: INFO: Container funny-ip ready: true, restart count 0 Mar 11 16:41:57.579: INFO: csi-mockplugin-0 started at 2023-03-11 16:41:15 +0000 UTC (0+3 container statuses recorded) Mar 11 16:41:57.579: INFO: Container csi-provisioner ready: true, restart count 0 Mar 11 16:41:57.579: INFO: Container driver-registrar ready: true, restart count 0 Mar 11 16:41:57.579: INFO: Container mock ready: true, restart count 0 Mar 11 16:41:57.579: INFO: pod-c6b5080f-dd59-4d03-a447-8dfa4716d6ab started at 2023-03-11 16:41:19 +0000 UTC (0+1 container statuses recorded) Mar 11 16:41:57.579: INFO: Container write-pod ready: true, restart count 0 Mar 11 16:41:57.579: INFO: pod-1 started at 2023-03-11 16:41:26 +0000 UTC (0+1 container statuses recorded) Mar 11 16:41:57.579: INFO: Container donothing ready: true, restart count 0 Mar 11 16:41:57.579: INFO: csi-mockplugin-attacher-0 started at 2023-03-11 16:41:15 +0000 UTC (0+1 container statuses recorded) Mar 11 16:41:57.579: INFO: Container csi-attacher ready: true, restart count 0 Mar 11 16:41:57.579: INFO: run-test started at 2023-03-11 16:41:55 +0000 UTC (0+1 container statuses recorded) Mar 11 16:41:57.579: INFO: Container run-test ready: true, restart count 0 Mar 11 16:41:57.579: INFO: test-grpc-3f5bb381-2f83-49fc-af8c-a754031c970f started at 2023-03-11 16:41:46 +0000 UTC (0+1 container statuses recorded) Mar 11 16:41:57.579: INFO: Container agnhost ready: true, restart count 0 Mar 11 16:41:57.579: INFO: ebs-csi-controller-79785c665b-lpmmf started at 2023-03-11 16:26:15 +0000 UTC (0+5 container statuses recorded) Mar 11 16:41:57.579: INFO: Container csi-attacher ready: true, restart count 0 Mar 11 16:41:57.579: INFO: Container csi-provisioner ready: true, restart count 0 Mar 11 16:41:57.579: INFO: Container csi-resizer ready: true, restart count 0 Mar 11 16:41:57.579: INFO: Container ebs-plugin ready: true, restart count 0 Mar 11 16:41:57.579: INFO: Container liveness-probe ready: true, restart count 0 Mar 11 16:41:57.579: INFO: csi-hostpathplugin-0 started at 2023-03-11 16:41:57 +0000 UTC (0+7 container statuses recorded) Mar 11 16:41:57.579: INFO: Container csi-attacher ready: false, restart count 0 Mar 11 16:41:57.579: INFO: Container csi-provisioner ready: false, restart count 0 Mar 11 16:41:57.579: INFO: Container csi-resizer ready: false, restart count 0 Mar 11 16:41:57.579: INFO: Container csi-snapshotter ready: false, restart count 0 Mar 11 16:41:57.579: INFO: Container hostpath ready: false, restart count 0 Mar 11 16:41:57.579: INFO: Container liveness-probe ready: false, restart count 0 Mar 11 16:41:57.579: INFO: Container node-driver-registrar ready: false, restart count 0 Mar 11 16:41:57.579: INFO: httpd started at 2023-03-11 16:41:43 +0000 UTC (0+1 container statuses recorded) Mar 11 16:41:57.579: INFO: Container httpd ready: true, restart count 0 Mar 11 16:41:57.579: INFO: csi-mockplugin-0 started at 2023-03-11 16:41:25 +0000 UTC (0+3 container statuses recorded) Mar 11 16:41:57.579: INFO: Container csi-provisioner ready: true, restart count 0 Mar 11 16:41:57.579: INFO: Container driver-registrar ready: true, restart count 0 Mar 11 16:41:57.579: INFO: Container mock ready: true, restart count 0 Mar 11 16:41:58.214: INFO: Latency metrics for node i-07e2b6e650d7ef56d Mar 11 16:41:58.214: INFO: Logging node info for node i-0e3467ec552af797e Mar 11 16:41:58.367: INFO: Node Info: &Node{ObjectMeta:{i-0e3467ec552af797e 6c9b0cc5-2c4e-4037-bf31-61bac12a7775 31369 0 2023-03-11 16:20:46 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:arm64 beta.kubernetes.io/instance-type:c6g.xlarge beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-northeast-1 failure-domain.beta.kubernetes.io/zone:ap-northeast-1a kops.k8s.io/kops-controller-pki: kubernetes.io/arch:arm64 kubernetes.io/hostname:i-0e3467ec552af797e kubernetes.io/os:linux node-role.kubernetes.io/control-plane: node.kubernetes.io/exclude-from-external-load-balancers: node.kubernetes.io/instance-type:c6g.xlarge topology.ebs.csi.aws.com/zone:ap-northeast-1a topology.kubernetes.io/region:ap-northeast-1 topology.kubernetes.io/zone:ap-northeast-1a] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-0e3467ec552af797e"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kubelet Update v1 2023-03-11 16:20:46 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {protokube Update v1 2023-03-11 16:21:30 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kops.k8s.io/kops-controller-pki":{},"f:node-role.kubernetes.io/control-plane":{},"f:node.kubernetes.io/exclude-from-external-load-balancers":{}}}} } {aws-cloud-controller-manager Update v1 2023-03-11 16:21:50 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-03-11 16:21:50 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kube-controller-manager Update v1 2023-03-11 16:21:58 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.0.0/24\"":{}},"f:taints":{}}} } {kubelet Update v1 2023-03-11 16:38:39 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.0.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-northeast-1a/i-0e3467ec552af797e,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/control-plane,Value:,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[100.96.0.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{4 0} {<nil>} 4 DecimalSI},ephemeral-storage: {{49756430336 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-32Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-64Ki: {{0 0} {<nil>} 0 DecimalSI},memory: {{8150110208 0} {<nil>} 7959092Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{4 0} {<nil>} 4 DecimalSI},ephemeral-storage: {{44780787229 0} {<nil>} 44780787229 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-32Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-64Ki: {{0 0} {<nil>} 0 DecimalSI},memory: {{8045252608 0} {<nil>} 7856692Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-03-11 16:38:39 +0000 UTC,LastTransitionTime:2023-03-11 16:20:46 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-03-11 16:38:39 +0000 UTC,LastTransitionTime:2023-03-11 16:20:46 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-03-11 16:38:39 +0000 UTC,LastTransitionTime:2023-03-11 16:20:46 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-03-11 16:38:39 +0000 UTC,LastTransitionTime:2023-03-11 16:21:58 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.40.18,},NodeAddress{Type:ExternalIP,Address:52.194.213.7,},NodeAddress{Type:InternalDNS,Address:i-0e3467ec552af797e.ap-northeast-1.compute.internal,},NodeAddress{Type:Hostname,Address:i-0e3467ec552af797e.ap-northeast-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-52-194-213-7.ap-northeast-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2abf2e564965592c7716accd56507e,SystemUUID:ec2abf2e-5649-6559-2c77-16accd56507e,BootID:fbada3ff-8655-404c-a98d-876577d4c322,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 22.04.2 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.27.0-alpha.3.420+bb6c6fad2c19b3,KubeProxyVersion:v1.27.0-alpha.3.420+bb6c6fad2c19b3,OperatingSystem:linux,Architecture:arm64,},Images:[]ContainerImage{ContainerImage{Names:[registry.k8s.io/etcd@sha256:9ce33ba33d8e738a5b85ed50b5080ac746deceed4a7496c550927a7a19ca3b6d registry.k8s.io/etcd:3.5.0-0],SizeBytes:157797353,},ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:157636062,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:4afb99b4690b418ffc2ceb67e1a17376457e441c1f09ab55447f0aaf992fa646 registry.k8s.io/etcd:3.4.3-0],SizeBytes:151677080,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:4ad90a11b55313b182afc186b9876c8e891531b8db4c9bf1541953021618d0e2 registry.k8s.io/etcd:3.4.13-0],SizeBytes:134531559,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:0d2ab6b15b4fe0b903b9e81192d942fb39c29be61fe341cd16db5d0165b0435c registry.k8s.io/etcd:3.3.17-0],SizeBytes:126979107,},ContainerImage{Names:[registry.k8s.io/kube-apiserver-arm64:v1.27.0-alpha.3.420_bb6c6fad2c19b3],SizeBytes:115123555,},ContainerImage{Names:[registry.k8s.io/kube-controller-manager-arm64:v1.27.0-alpha.3.420_bb6c6fad2c19b3],SizeBytes:107914235,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:17da501f5d2a675be46040422a27b7cc21b8a43895ac998b171db1c346f361f7 registry.k8s.io/etcd:3.3.10-0],SizeBytes:81436097,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:6f72b851544986cb0921b53ea655ec04c36131248f16d4ad110cb3ca0c369dc1 registry.k8s.io/etcd:3.5.4-0],SizeBytes:81120118,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:13f53ed1d91e2e11aac476ee9a0269fdda6cc4874eba903efd40daf50c55eee5 registry.k8s.io/etcd:3.5.3-0],SizeBytes:81114891,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:51eae8381dcb1078289fa7b4f3df2630cdc18d09fb56f8e56b41c40e191d6c83 registry.k8s.io/etcd:3.5.7-0],SizeBytes:80665728,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:dd75ec974b0a2a6f6bb47001ba09207976e625db898d1b16735528c009cb171c registry.k8s.io/etcd:3.5.6-0],SizeBytes:80539316,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:905d7ca17fd02bc24c0eba9a062753aba15db3e31422390bc3238eb762339b20 registry.k8s.io/etcd:3.2.24-1],SizeBytes:61500433,},ContainerImage{Names:[registry.k8s.io/kube-proxy-arm64:v1.27.0-alpha.3.420_bb6c6fad2c19b3],SizeBytes:60947298,},ContainerImage{Names:[registry.k8s.io/etcdadm/etcd-manager-slim@sha256:8727adc059334e0e3329dbccce116a734da4a090da5e86c8cddccaf333caa666 registry.k8s.io/etcdadm/etcd-manager-slim:v3.0.20230201],SizeBytes:60826655,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:64b9ea357325d5db9f8a723dcf503b5a449177b17ac87d69481e126bb724c263 registry.k8s.io/etcd:3.5.1-0],SizeBytes:59553904,},ContainerImage{Names:[registry.k8s.io/kube-scheduler-arm64:v1.27.0-alpha.3.420_bb6c6fad2c19b3],SizeBytes:56731120,},ContainerImage{Names:[registry.k8s.io/kops/kops-controller:1.27.0-alpha.1],SizeBytes:41885191,},ContainerImage{Names:[registry.k8s.io/kops/dns-controller:1.27.0-alpha.1],SizeBytes:40376038,},ContainerImage{Names:[public.ecr.aws/karpenter/controller@sha256:68db4f092cf9cc83f5ef9e2fbc5407c2cb682e81f64dfaa700a7602ede38b1cf public.ecr.aws/karpenter/controller:v0.16.3],SizeBytes:33707751,},ContainerImage{Names:[public.ecr.aws/karpenter/webhook@sha256:96a2d9b06d6bc5127801f358f74b1cf2d289b423a2e9ba40c573c0b14b17dafa public.ecr.aws/karpenter/webhook:v0.16.3],SizeBytes:33307187,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:28932904,},ContainerImage{Names:[quay.io/cilium/operator@sha256:a6d24a006a6b92967ac90786b49bc1ac26e5477cf028cd1186efcfc2466484db quay.io/cilium/operator:v1.12.5],SizeBytes:24567287,},ContainerImage{Names:[registry.k8s.io/provider-aws/cloud-controller-manager@sha256:fdeb61e3e42ecd9cca868d550ebdb88dd6341d9e91fcfa9a37e227dab2ad22cb registry.k8s.io/provider-aws/cloud-controller-manager:v1.26.0],SizeBytes:18345226,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:8502345,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:7685305,},ContainerImage{Names:[registry.k8s.io/kops/kube-apiserver-healthcheck:1.27.0-alpha.1],SizeBytes:4740762,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:253553,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Mar 11 16:41:58.367: INFO: Logging kubelet events for node i-0e3467ec552af797e Mar 11 16:41:58.519: INFO: Logging pods the kubelet thinks is on node i-0e3467ec552af797e Mar 11 16:41:58.674: INFO: dns-controller-7dfd64d999-q9hh7 started at 2023-03-11 16:21:37 +0000 UTC (0+1 container statuses recorded) Mar 11 16:41:58.674: INFO: Container dns-controller ready: true, restart count 0 Mar 11 16:41:58.674: INFO: cilium-qrsfn started at 2023-03-11 16:21:37 +0000 UTC (1+1 container statuses recorded) Mar 11 16:41:58.674: INFO: Init container clean-cilium-state ready: true, restart count 0 Mar 11 16:41:58.674: INFO: Container cilium-agent ready: true, restart count 0 Mar 11 16:41:58.674: INFO: cilium-operator-8496cc65f5-qhdg8 started at 2023-03-11 16:21:37 +0000 UTC (0+1 container statuses recorded) Mar 11 16:41:58.674: INFO: Container cilium-operator ready: true, restart count 0 Mar 11 16:41:58.674: INFO: kops-controller-76gkw started at 2023-03-11 16:21:37 +0000 UTC (0+1 container statuses recorded) Mar 11 16:41:58.674: INFO: Container kops-controller ready: true, restart count 0 Mar 11 16:41:58.674: INFO: aws-cloud-controller-manager-kwbw2 started at 2023-03-11 16:21:37 +0000 UTC (0+1 container statuses recorded) Mar 11 16:41:58.674: INFO: Container aws-cloud-controller-manager ready: true, restart count 0 Mar 11 16:41:58.674: INFO: kube-apiserver-i-0e3467ec552af797e started at 2023-03-11 16:18:44 +0000 UTC (0+2 container statuses recorded) Mar 11 16:41:58.674: INFO: Container healthcheck ready: true, restart count 0 Mar 11 16:41:58.674: INFO: Container kube-apiserver ready: true, restart count 3 Mar 11 16:41:58.674: INFO: kube-scheduler-i-0e3467ec552af797e started at 2023-03-11 16:18:44 +0000 UTC (0+1 container statuses recorded) Mar 11 16:41:58.674: INFO: Container kube-scheduler ready: true, restart count 0 Mar 11 16:41:58.674: INFO: etcd-manager-main-i-0e3467ec552af797e started at 2023-03-11 16:18:44 +0000 UTC (11+1 container statuses recorded) Mar 11 16:41:58.674: INFO: Init container init-etcd-3-2-24 ready: true, restart count 0 Mar 11 16:41:58.674: INFO: Init container init-etcd-3-3-10 ready: true, restart count 0 Mar 11 16:41:58.674: INFO: Init container init-etcd-3-3-17 ready: true, restart count 0 Mar 11 16:41:58.674: INFO: Init container init-etcd-3-4-13 ready: true, restart count 0 Mar 11 16:41:58.674: INFO: Init container init-etcd-3-4-3 ready: true, restart count 0 Mar 11 16:41:58.674: INFO: Init container init-etcd-3-5-0 ready: true, restart count 0 Mar 11 16:41:58.674: INFO: Init container init-etcd-3-5-1 ready: true, restart count 0 Mar 11 16:41:58.674: INFO: Init container init-etcd-3-5-3 ready: true, restart count 0 Mar 11 16:41:58.674: INFO: Init container init-etcd-3-5-4 ready: true, restart count 0 Mar 11 16:41:58.674: INFO: Init container init-etcd-3-5-6 ready: true, restart count 0 Mar 11 16:41:58.674: INFO: Init container init-etcd-3-5-7 ready: true, restart count 0 Mar 11 16:41:58.674: INFO: Container etcd-manager ready: true, restart count 0 Mar 11 16:41:58.674: INFO: ebs-csi-node-lpwz9 started at 2023-03-11 16:21:37 +0000 UTC (0+3 container statuses recorded) Mar 11 16:41:58.674: INFO: Container ebs-plugin ready: true, restart count 0 Mar 11 16:41:58.674: INFO: Container liveness-probe ready: true, restart count 0 Mar 11 16:41:58.674: INFO: Container node-driver-registrar ready: true, restart count 0 Mar 11 16:41:58.674: INFO: karpenter-webhook-5b646cd4f7-6sg2k started at 2023-03-11 16:21:58 +0000 UTC (0+1 container statuses recorded) Mar 11 16:41:58.674: INFO: Container webhook ready: true, restart count 0 Mar 11 16:41:58.674: INFO: karpenter-controller-567749677d-d9szs started at 2023-03-11 16:21:58 +0000 UTC (0+1 container statuses recorded) Mar 11 16:41:58.674: INFO: Container controller ready: true, restart count 0 Mar 11 16:41:58.674: INFO: kube-controller-manager-i-0e3467ec552af797e started at 2023-03-11 16:18:44 +0000 UTC (0+1 container statuses recorded) Mar 11 16:41:58.674: INFO: Container kube-controller-manager ready: true, restart count 4 Mar 11 16:41:58.674: INFO: etcd-manager-events-i-0e3467ec552af797e started at 2023-03-11 16:18:44 +0000 UTC (11+1 container statuses recorded) Mar 11 16:41:58.674: INFO: Init container init-etcd-3-2-24 ready: true, restart count 0 Mar 11 16:41:58.674: INFO: Init container init-etcd-3-3-10 ready: true, restart count 0 Mar 11 16:41:58.674: INFO: Init container init-etcd-3-3-17 ready: true, restart count 0 Mar 11 16:41:58.674: INFO: Init container init-etcd-3-4-13 ready: true, restart count 0 Mar 11 16:41:58.674: INFO: Init container init-etcd-3-4-3 ready: true, restart count 0 Mar 11 16:41:58.674: INFO: Init container init-etcd-3-5-0 ready: true, restart count 0 Mar 11 16:41:58.674: INFO: Init container init-etcd-3-5-1 ready: true, restart count 0 Mar 11 16:41:58.674: INFO: Init container init-etcd-3-5-3 ready: true, restart count 0 Mar 11 16:41:58.674: INFO: Init container init-etcd-3-5-4 ready: true, restart count 0 Mar 11 16:41:58.674: INFO: Init container init-etcd-3-5-6 ready: true, restart count 0 Mar 11 16:41:58.674: INFO: Init container init-etcd-3-5-7 ready: true, restart count 0 Mar 11 16:41:58.674: INFO: Container etcd-manager ready: true, restart count 0 Mar 11 16:41:59.166: INFO: Latency metrics for node i-0e3467ec552af797e END STEP: dump namespace information after failure - test/e2e/framework/framework.go:288 @ 03/11/23 16:41:59.166 (3.654s) < Exit [DeferCleanup (Each)] [sig-node] Probing container - dump namespaces | framework.go:209 @ 03/11/23 16:41:59.166 (3.654s) > Enter [DeferCleanup (Each)] [sig-node] Probing container - tear down framework | framework.go:206 @ 03/11/23 16:41:59.166 STEP: Destroying namespace "container-probe-4792" for this suite. - test/e2e/framework/framework.go:351 @ 03/11/23 16:41:59.166 < Exit [DeferCleanup (Each)] [sig-node] Probing container - tear down framework | framework.go:206 @ 03/11/23 16:41:59.316 (150ms) > Enter [ReportAfterEach] TOP-LEVEL - test/e2e/e2e_test.go:144 @ 03/11/23 16:41:59.316 < Exit [ReportAfterEach] TOP-LEVEL - test/e2e/e2e_test.go:144 @ 03/11/23 16:41:59.316 (0s)
Find container-probe-4792/test-grpc-21e6bf52-ac24-444f-9335-0685035e1c19 mentions in log files | View test history on testgrid
go run hack/e2e.go -v --test --test_args='--ginkgo.focus=Kubernetes\se2e\ssuite\s\[It\]\s\[sig\-node\]\sProbing\scontainer\sshould\s\*not\*\sbe\srestarted\swith\sa\stcp\:8080\sliveness\sprobe\s\[NodeConformance\]\s\[Conformance\]$'
[FAILED] pod container-probe-271/liveness-ecf7c9aa-a3c6-4146-b171-4bb7b637f384 - expected number of restarts: 0, found restarts: 1 In [It] at: test/e2e/common/node/container_probe.go:1000 @ 03/11/23 16:40:26.981from junit_01.xml
> Enter [BeforeEach] [sig-node] Probing container - set up framework | framework.go:191 @ 03/11/23 16:36:22.806 STEP: Creating a kubernetes client - test/e2e/framework/framework.go:211 @ 03/11/23 16:36:22.806 Mar 11 16:36:22.806: INFO: >>> kubeConfig: /root/.kube/config STEP: Building a namespace api object, basename container-probe - test/e2e/framework/framework.go:250 @ 03/11/23 16:36:22.807 STEP: Waiting for a default service account to be provisioned in namespace - test/e2e/framework/framework.go:259 @ 03/11/23 16:36:23.25 STEP: Waiting for kube-root-ca.crt to be provisioned in namespace - test/e2e/framework/framework.go:262 @ 03/11/23 16:36:23.542 < Exit [BeforeEach] [sig-node] Probing container - set up framework | framework.go:191 @ 03/11/23 16:36:23.835 (1.028s) > Enter [BeforeEach] [sig-node] Probing container - test/e2e/framework/metrics/init/init.go:33 @ 03/11/23 16:36:23.835 < Exit [BeforeEach] [sig-node] Probing container - test/e2e/framework/metrics/init/init.go:33 @ 03/11/23 16:36:23.835 (0s) > Enter [BeforeEach] [sig-node] Probing container - test/e2e/common/node/container_probe.go:62 @ 03/11/23 16:36:23.835 < Exit [BeforeEach] [sig-node] Probing container - test/e2e/common/node/container_probe.go:62 @ 03/11/23 16:36:23.835 (0s) > Enter [It] should *not* be restarted with a tcp:8080 liveness probe [NodeConformance] [Conformance] - test/e2e/common/node/container_probe.go:183 @ 03/11/23 16:36:23.835 STEP: Creating pod liveness-ecf7c9aa-a3c6-4146-b171-4bb7b637f384 in namespace container-probe-271 - test/e2e/common/node/container_probe.go:955 @ 03/11/23 16:36:23.835 Mar 11 16:36:26.282: INFO: Started pod liveness-ecf7c9aa-a3c6-4146-b171-4bb7b637f384 in namespace container-probe-271 STEP: checking the pod's current state and verifying that restartCount is present - test/e2e/common/node/container_probe.go:966 @ 03/11/23 16:36:26.282 Mar 11 16:36:26.429: INFO: Initial restart count of pod liveness-ecf7c9aa-a3c6-4146-b171-4bb7b637f384 is 0 Mar 11 16:40:07.797: INFO: Restart count of pod container-probe-271/liveness-ecf7c9aa-a3c6-4146-b171-4bb7b637f384 is now 1 (3m41.368066679s elapsed) [FAILED] pod container-probe-271/liveness-ecf7c9aa-a3c6-4146-b171-4bb7b637f384 - expected number of restarts: 0, found restarts: 1 In [It] at: test/e2e/common/node/container_probe.go:1000 @ 03/11/23 16:40:26.981 < Exit [It] should *not* be restarted with a tcp:8080 liveness probe [NodeConformance] [Conformance] - test/e2e/common/node/container_probe.go:183 @ 03/11/23 16:40:26.981 (4m3.146s) > Enter [AfterEach] [sig-node] Probing container - test/e2e/framework/node/init/init.go:33 @ 03/11/23 16:40:26.981 Mar 11 16:40:26.981: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready < Exit [AfterEach] [sig-node] Probing container - test/e2e/framework/node/init/init.go:33 @ 03/11/23 16:40:27.129 (148ms) > Enter [DeferCleanup (Each)] [sig-node] Probing container - test/e2e/common/node/container_probe.go:951 @ 03/11/23 16:40:27.129 STEP: deleting the pod - test/e2e/common/node/container_probe.go:952 @ 03/11/23 16:40:27.129 < Exit [DeferCleanup (Each)] [sig-node] Probing container - test/e2e/common/node/container_probe.go:951 @ 03/11/23 16:40:27.284 (155ms) > Enter [DeferCleanup (Each)] [sig-node] Probing container - test/e2e/framework/metrics/init/init.go:35 @ 03/11/23 16:40:27.284 < Exit [DeferCleanup (Each)] [sig-node] Probing container - test/e2e/framework/metrics/init/init.go:35 @ 03/11/23 16:40:27.284 (0s) > Enter [DeferCleanup (Each)] [sig-node] Probing container - dump namespaces | framework.go:209 @ 03/11/23 16:40:27.284 STEP: dump namespace information after failure - test/e2e/framework/framework.go:288 @ 03/11/23 16:40:27.284 STEP: Collecting events from namespace "container-probe-271". - test/e2e/framework/debug/dump.go:42 @ 03/11/23 16:40:27.284 STEP: Found 7 events. - test/e2e/framework/debug/dump.go:46 @ 03/11/23 16:40:27.431 Mar 11 16:40:27.431: INFO: At 2023-03-11 16:36:23 +0000 UTC - event for liveness-ecf7c9aa-a3c6-4146-b171-4bb7b637f384: {default-scheduler } Scheduled: Successfully assigned container-probe-271/liveness-ecf7c9aa-a3c6-4146-b171-4bb7b637f384 to i-07e2b6e650d7ef56d Mar 11 16:40:27.431: INFO: At 2023-03-11 16:36:24 +0000 UTC - event for liveness-ecf7c9aa-a3c6-4146-b171-4bb7b637f384: {kubelet i-07e2b6e650d7ef56d} Pulled: Container image "registry.k8s.io/e2e-test-images/agnhost:2.43" already present on machine Mar 11 16:40:27.431: INFO: At 2023-03-11 16:36:24 +0000 UTC - event for liveness-ecf7c9aa-a3c6-4146-b171-4bb7b637f384: {kubelet i-07e2b6e650d7ef56d} Created: Created container agnhost-container Mar 11 16:40:27.431: INFO: At 2023-03-11 16:36:24 +0000 UTC - event for liveness-ecf7c9aa-a3c6-4146-b171-4bb7b637f384: {kubelet i-07e2b6e650d7ef56d} Started: Started container agnhost-container Mar 11 16:40:27.431: INFO: At 2023-03-11 16:38:57 +0000 UTC - event for liveness-ecf7c9aa-a3c6-4146-b171-4bb7b637f384: {kubelet i-07e2b6e650d7ef56d} Unhealthy: Liveness probe failed: dial tcp 100.96.1.150:8080: i/o timeout Mar 11 16:40:27.431: INFO: At 2023-03-11 16:39:25 +0000 UTC - event for liveness-ecf7c9aa-a3c6-4146-b171-4bb7b637f384: {kubelet i-07e2b6e650d7ef56d} Killing: Container agnhost-container failed liveness probe, will be restarted Mar 11 16:40:27.431: INFO: At 2023-03-11 16:40:27 +0000 UTC - event for liveness-ecf7c9aa-a3c6-4146-b171-4bb7b637f384: {kubelet i-07e2b6e650d7ef56d} Killing: Stopping container agnhost-container Mar 11 16:40:27.578: INFO: POD NODE PHASE GRACE CONDITIONS Mar 11 16:40:27.578: INFO: Mar 11 16:40:27.725: INFO: Logging node info for node i-04523d1dffb0fe725 Mar 11 16:40:27.873: INFO: Node Info: &Node{ObjectMeta:{i-04523d1dffb0fe725 5d37640f-a7d9-4c26-86c5-9da21f7ccc18 33302 0 2023-03-11 16:24:20 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:arm64 beta.kubernetes.io/instance-type:t4g.small beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-northeast-1 failure-domain.beta.kubernetes.io/zone:ap-northeast-1a karpenter.k8s.aws/instance-category:t karpenter.k8s.aws/instance-cpu:2 karpenter.k8s.aws/instance-family:t4g karpenter.k8s.aws/instance-generation:4 karpenter.k8s.aws/instance-hypervisor:nitro karpenter.k8s.aws/instance-memory:2048 karpenter.k8s.aws/instance-pods:11 karpenter.k8s.aws/instance-size:small karpenter.sh/capacity-type:spot karpenter.sh/initialized:true karpenter.sh/provisioner-name:nodes kubelet_cleanup:true kubernetes.io/arch:arm64 kubernetes.io/hostname:i-04523d1dffb0fe725 kubernetes.io/os:linux node-role.kubernetes.io/node: node-role.kubernetes.io/spot-worker:true node.kubernetes.io/instance-type:t4g.small topology.ebs.csi.aws.com/zone:ap-northeast-1a topology.hostpath.csi/node:i-04523d1dffb0fe725 topology.kubernetes.io/region:ap-northeast-1 topology.kubernetes.io/zone:ap-northeast-1a] map[csi.volume.kubernetes.io/nodeid:{"csi-hostpath-ephemeral-305":"i-04523d1dffb0fe725","csi-hostpath-ephemeral-3506":"i-04523d1dffb0fe725","csi-mock-csi-mock-volumes-fsgroup-policy-9933":"i-04523d1dffb0fe725","ebs.csi.aws.com":"i-04523d1dffb0fe725"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [{karpenter.sh/v1alpha5 Provisioner nodes ac7283b2-eed0-4a9f-9ddd-616f33a0991a <nil> 0xc001f98120}] [karpenter.sh/termination] [{aws-cloud-controller-manager Update v1 2023-03-11 16:24:20 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{}}}} } {kops-controller Update v1 2023-03-11 16:24:20 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{},"f:node-role.kubernetes.io/spot-worker":{}}}} } {karpenter Update v1 2023-03-11 16:26:12 +0000 UTC FieldsV1 {"f:metadata":{"f:finalizers":{".":{},"v:\"karpenter.sh/termination\"":{}},"f:labels":{".":{},"f:karpenter.k8s.aws/instance-category":{},"f:karpenter.k8s.aws/instance-cpu":{},"f:karpenter.k8s.aws/instance-family":{},"f:karpenter.k8s.aws/instance-generation":{},"f:karpenter.k8s.aws/instance-hypervisor":{},"f:karpenter.k8s.aws/instance-memory":{},"f:karpenter.k8s.aws/instance-pods":{},"f:karpenter.k8s.aws/instance-size":{},"f:karpenter.sh/capacity-type":{},"f:karpenter.sh/initialized":{},"f:karpenter.sh/provisioner-name":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}},"f:ownerReferences":{".":{},"k:{\"uid\":\"ac7283b2-eed0-4a9f-9ddd-616f33a0991a\"}":{}}},"f:spec":{"f:providerID":{}}} } {kube-controller-manager Update v1 2023-03-11 16:26:12 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:node.alpha.kubernetes.io/ttl":{}},"f:labels":{"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.2.0/24\"":{}}}} } {aws-cloud-controller-manager Update v1 2023-03-11 16:26:50 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{".":{},"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"InternalIP\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {e2e.test Update v1 2023-03-11 16:38:42 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubelet_cleanup":{}}}} } {kube-controller-manager Update v1 2023-03-11 16:40:10 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{".":{},"k:{\"type\":\"DiskPressure\"}":{".":{},"f:type":{}},"k:{\"type\":\"MemoryPressure\"}":{".":{},"f:type":{}},"k:{\"type\":\"PIDPressure\"}":{".":{},"f:type":{}},"k:{\"type\":\"Ready\"}":{".":{},"f:type":{}}},"f:volumesAttached":{}}} status} {kubelet Update v1 2023-03-11 16:40:10 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{"f:kubernetes.io/hostname":{},"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{".":{},"f:cpu":{},"f:ephemeral-storage":{},"f:hugepages-1Gi":{},"f:hugepages-2Mi":{},"f:hugepages-32Mi":{},"f:hugepages-64Ki":{},"f:memory":{},"f:pods":{}},"f:capacity":{".":{},"f:cpu":{},"f:ephemeral-storage":{},"f:hugepages-1Gi":{},"f:hugepages-2Mi":{},"f:hugepages-32Mi":{},"f:hugepages-64Ki":{},"f:memory":{},"f:pods":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:daemonEndpoints":{"f:kubeletEndpoint":{"f:Port":{}}},"f:images":{},"f:nodeInfo":{"f:architecture":{},"f:bootID":{},"f:containerRuntimeVersion":{},"f:kernelVersion":{},"f:kubeProxyVersion":{},"f:kubeletVersion":{},"f:machineID":{},"f:operatingSystem":{},"f:osImage":{},"f:systemUUID":{}},"f:volumesInUse":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.2.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-northeast-1a/i-04523d1dffb0fe725,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.2.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49756430336 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-32Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-64Ki: {{0 0} {<nil>} 0 DecimalSI},memory: {{1941540864 0} {<nil>} 1896036Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44780787229 0} {<nil>} 44780787229 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-32Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-64Ki: {{0 0} {<nil>} 0 DecimalSI},memory: {{1836683264 0} {<nil>} 1793636Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-03-11 16:40:10 +0000 UTC,LastTransitionTime:2023-03-11 16:26:12 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-03-11 16:40:10 +0000 UTC,LastTransitionTime:2023-03-11 16:26:02 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-03-11 16:40:10 +0000 UTC,LastTransitionTime:2023-03-11 16:26:02 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-03-11 16:40:10 +0000 UTC,LastTransitionTime:2023-03-11 16:26:02 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.48.168,},NodeAddress{Type:ExternalIP,Address:52.199.253.156,},NodeAddress{Type:InternalDNS,Address:i-04523d1dffb0fe725.ap-northeast-1.compute.internal,},NodeAddress{Type:Hostname,Address:i-04523d1dffb0fe725.ap-northeast-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-52-199-253-156.ap-northeast-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2c8fb0e6b2e9c4af310f65b6161cd8,SystemUUID:ec2c8fb0-e6b2-e9c4-af31-0f65b6161cd8,BootID:a3b8de83-558b-44c4-a46a-d097c3f98225,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 22.04.2 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.27.0-alpha.3.420+bb6c6fad2c19b3,KubeProxyVersion:v1.27.0-alpha.3.420+bb6c6fad2c19b3,OperatingSystem:linux,Architecture:arm64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:157636062,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/jessie-dnsutils@sha256:24aaf2626d6b27864c29de2097e8bbb840b3a414271bf7c8995e431e47d8408e registry.k8s.io/e2e-test-images/jessie-dnsutils:1.7],SizeBytes:116764683,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:51eae8381dcb1078289fa7b4f3df2630cdc18d09fb56f8e56b41c40e191d6c83 registry.k8s.io/etcd:3.5.7-0],SizeBytes:80665728,},ContainerImage{Names:[registry.k8s.io/kube-proxy-arm64:v1.27.0-alpha.3.420_bb6c6fad2c19b3],SizeBytes:60947298,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:16bbf38c463a4223d8cfe4da12bc61010b082a79b4bb003e2d3ba3ece5dd5f9e registry.k8s.io/e2e-test-images/agnhost:2.43],SizeBytes:49808364,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nautilus@sha256:80ba6c8c44f9623f06e868a1aa66026c8ec438ad814f9ec95e9333b415fe3550 registry.k8s.io/e2e-test-images/nautilus:1.7],SizeBytes:47953400,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:3fe7acf013d1264ffded116b80a73dc129a449b0fccdb8d21af8279f2233f36e registry.k8s.io/e2e-test-images/httpd:2.4.39-4],SizeBytes:41528818,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:148b022f5c5da426fc2f3c14b5c0867e58ef05961510c84749ac1fddcb0fef22 registry.k8s.io/e2e-test-images/httpd:2.4.38-4],SizeBytes:40343601,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:28932904,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:ee3b525d5b89db99da3b8eb521d9cd90cb6e9ef0fbb651e98bb37be78d36b5b8 registry.k8s.io/sig-storage/csi-provisioner:v3.3.0],SizeBytes:23945474,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/sample-apiserver@sha256:8d70890151aa5d096f331cb9da1b9cd5be0412b7363fe67b5c3befdcaa2a28d0 registry.k8s.io/e2e-test-images/sample-apiserver:1.17.7],SizeBytes:23857462,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:425d8f1b769398127767b06ed97ce62578a3179bcb99809ce93a1649e025ffe7 registry.k8s.io/sig-storage/csi-resizer:v1.6.0],SizeBytes:22678632,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:291334908ddf71a4661fd7f6d9d97274de8a5378a2b6fdfeb2ce73414a34f82f registry.k8s.io/sig-storage/csi-snapshotter:v6.1.0],SizeBytes:22434180,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:9a685020911e2725ad019dbce6e4a5ab93d51e3d4557f115e64343345e05781b registry.k8s.io/sig-storage/csi-attacher:v4.0.0],SizeBytes:22394501,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:122bfb8c1edabb3c0edd63f06523e6940d958d19b3957dc7b1d6f81e9f1f6119 registry.k8s.io/sig-storage/csi-provisioner:v3.1.0],SizeBytes:21735644,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:9ebbf9f023e7b41ccee3d52afe39a89e3ddacdbb69269d583abfc25847cfd9e4 registry.k8s.io/sig-storage/csi-resizer:v1.4.0],SizeBytes:20848413,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:8b9c313c05f54fb04f8d430896f5f5904b6cb157df261501b29adc04d2b2dc7b registry.k8s.io/sig-storage/csi-attacher:v3.4.0],SizeBytes:20565480,},ContainerImage{Names:[registry.k8s.io/cpa/cluster-proportional-autoscaler@sha256:68d396900aeaa072c1f27289485fdac29834045a6f3ffe369bf389d830ef572d registry.k8s.io/cpa/cluster-proportional-autoscaler:1.8.6],SizeBytes:19007623,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nonroot@sha256:ee9f50b3c64b174d296d91ca9f69a914ac30e59095dfb462b2b518ad28a63655 registry.k8s.io/e2e-test-images/nonroot:1.4],SizeBytes:18561972,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:92257881c1d6493cf18299a24af42330f891166560047902b8d431fb66b01af5 registry.k8s.io/sig-storage/hostpathplugin:v1.9.0],SizeBytes:18267611,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:9b2d9abda017c28b12f84a344f57cd73fbdb6c2bd7dd5bdd5018246ad1093ba6 registry.k8s.io/sig-storage/hostpathplugin:v1.11.0],SizeBytes:18159085,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:017727efcfeb7d053af68e51436ce8e65edbc6ca573720afb4f79c8594036955 registry.k8s.io/coredns/coredns:v1.10.0],SizeBytes:13766234,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:8502345,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:933940f13b3ea0abc62e656c1aa5c5b47c04b15d71250413a6b821bd0c58b94e registry.k8s.io/sig-storage/livenessprobe:v2.7.0],SizeBytes:8223819,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:7685305,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:db048754ae68ae337d8fa96494c96d2a1204c3320f5dcf7e8e71085adec85da6 registry.k8s.io/e2e-test-images/nginx:1.15-4],SizeBytes:6883347,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:5c99cf6a02adda929b10321dbf4ecfa00d87be9ba4fb456006237d530ab4baa1 registry.k8s.io/e2e-test-images/nginx:1.14-4],SizeBytes:6860089,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf registry.k8s.io/e2e-test-images/busybox:1.29-2],SizeBytes:773367,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:2e0f836850e09b8b7cc937681d6194537a09fbd5f6b9e08f4d646a85128e8937 registry.k8s.io/e2e-test-images/busybox:1.29-4],SizeBytes:772933,},ContainerImage{Names:[registry.k8s.io/pause@sha256:7031c1b283388d2c2e09b57badb803c05ebed362dc88d84b480cc47f72a21097 registry.k8s.io/pause:3.9],SizeBytes:268051,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:253553,},},VolumesInUse:[kubernetes.io/csi/csi-hostpath-ephemeral-305^5a7aeb13-c02b-11ed-a1cb-46207272bd57 kubernetes.io/csi/csi-hostpath-ephemeral-3506^5033904d-c02b-11ed-a7b7-be3e7bde3e06],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/csi-hostpath-ephemeral-3506^5033904d-c02b-11ed-a7b7-be3e7bde3e06,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/csi-hostpath-ephemeral-305^5a7aeb13-c02b-11ed-a1cb-46207272bd57,DevicePath:,},},Config:nil,},} Mar 11 16:40:27.873: INFO: Logging kubelet events for node i-04523d1dffb0fe725 Mar 11 16:40:28.023: INFO: Logging pods the kubelet thinks is on node i-04523d1dffb0fe725 Mar 11 16:40:28.184: INFO: cilium-kfwlg started at 2023-03-11 16:26:03 +0000 UTC (1+1 container statuses recorded) Mar 11 16:40:28.184: INFO: Init container clean-cilium-state ready: true, restart count 0 Mar 11 16:40:28.184: INFO: Container cilium-agent ready: true, restart count 0 Mar 11 16:40:28.184: INFO: coredns-745bddcfbb-s57v5 started at 2023-03-11 16:26:12 +0000 UTC (0+1 container statuses recorded) Mar 11 16:40:28.184: INFO: Container coredns ready: true, restart count 0 Mar 11 16:40:28.184: INFO: execpod59vnr started at 2023-03-11 16:37:56 +0000 UTC (0+1 container statuses recorded) Mar 11 16:40:28.184: INFO: Container agnhost-container ready: true, restart count 0 Mar 11 16:40:28.184: INFO: inline-volume-tester-prv4m started at 2023-03-11 16:39:38 +0000 UTC (0+1 container statuses recorded) Mar 11 16:40:28.184: INFO: Container csi-volume-tester ready: true, restart count 0 Mar 11 16:40:28.184: INFO: inline-volume-tester-6fw2d started at 2023-03-11 16:39:55 +0000 UTC (0+1 container statuses recorded) Mar 11 16:40:28.184: INFO: Container csi-volume-tester ready: true, restart count 0 Mar 11 16:40:28.184: INFO: cleanup20-94fd1eef-e542-41c7-9753-4b0cafd57772-t6bqh started at 2023-03-11 16:38:43 +0000 UTC (0+1 container statuses recorded) Mar 11 16:40:28.184: INFO: Container cleanup20-94fd1eef-e542-41c7-9753-4b0cafd57772 ready: true, restart count 0 Mar 11 16:40:28.184: INFO: cleanup20-94fd1eef-e542-41c7-9753-4b0cafd57772-pbcb4 started at 2023-03-11 16:38:43 +0000 UTC (0+1 container statuses recorded) Mar 11 16:40:28.184: INFO: Container cleanup20-94fd1eef-e542-41c7-9753-4b0cafd57772 ready: true, restart count 0 Mar 11 16:40:28.184: INFO: ebs-csi-controller-79785c665b-2wwsh started at 2023-03-11 16:26:12 +0000 UTC (0+5 container statuses recorded) Mar 11 16:40:28.184: INFO: Container csi-attacher ready: true, restart count 0 Mar 11 16:40:28.184: INFO: Container csi-provisioner ready: true, restart count 0 Mar 11 16:40:28.184: INFO: Container csi-resizer ready: true, restart count 0 Mar 11 16:40:28.184: INFO: Container ebs-plugin ready: true, restart count 0 Mar 11 16:40:28.184: INFO: Container liveness-probe ready: true, restart count 0 Mar 11 16:40:28.184: INFO: csi-hostpathplugin-0 started at 2023-03-11 16:39:35 +0000 UTC (0+7 container statuses recorded) Mar 11 16:40:28.184: INFO: Container csi-attacher ready: true, restart count 0 Mar 11 16:40:28.184: INFO: Container csi-provisioner ready: true, restart count 0 Mar 11 16:40:28.184: INFO: Container csi-resizer ready: true, restart count 0 Mar 11 16:40:28.184: INFO: Container csi-snapshotter ready: true, restart count 0 Mar 11 16:40:28.184: INFO: Container hostpath ready: true, restart count 0 Mar 11 16:40:28.184: INFO: Container liveness-probe ready: true, restart count 0 Mar 11 16:40:28.184: INFO: Container node-driver-registrar ready: true, restart count 0 Mar 11 16:40:28.184: INFO: csi-hostpathplugin-0 started at 2023-03-11 16:39:53 +0000 UTC (0+7 container statuses recorded) Mar 11 16:40:28.184: INFO: Container csi-attacher ready: true, restart count 0 Mar 11 16:40:28.184: INFO: Container csi-provisioner ready: true, restart count 0 Mar 11 16:40:28.184: INFO: Container csi-resizer ready: true, restart count 0 Mar 11 16:40:28.184: INFO: Container csi-snapshotter ready: true, restart count 0 Mar 11 16:40:28.184: INFO: Container hostpath ready: true, restart count 0 Mar 11 16:40:28.184: INFO: Container liveness-probe ready: true, restart count 0 Mar 11 16:40:28.184: INFO: Container node-driver-registrar ready: true, restart count 0 Mar 11 16:40:28.184: INFO: test-rollover-controller-c66n4 started at 2023-03-11 16:40:21 +0000 UTC (0+1 container statuses recorded) Mar 11 16:40:28.184: INFO: Container httpd ready: true, restart count 0 Mar 11 16:40:28.184: INFO: pvc-volume-tester-hw4hq started at 2023-03-11 16:40:04 +0000 UTC (0+1 container statuses recorded) Mar 11 16:40:28.184: INFO: Container volume-tester ready: true, restart count 0 Mar 11 16:40:28.184: INFO: service-proxy-disabled-lk9pg started at 2023-03-11 16:37:29 +0000 UTC (0+1 container statuses recorded) Mar 11 16:40:28.184: INFO: Container service-proxy-disabled ready: true, restart count 0 Mar 11 16:40:28.184: INFO: cleanup20-94fd1eef-e542-41c7-9753-4b0cafd57772-vdckh started at 2023-03-11 16:38:43 +0000 UTC (0+1 container statuses recorded) Mar 11 16:40:28.184: INFO: Container cleanup20-94fd1eef-e542-41c7-9753-4b0cafd57772 ready: true, restart count 0 Mar 11 16:40:28.184: INFO: cleanup20-94fd1eef-e542-41c7-9753-4b0cafd57772-4nfw6 started at 2023-03-11 16:38:43 +0000 UTC (0+1 container statuses recorded) Mar 11 16:40:28.184: INFO: Container cleanup20-94fd1eef-e542-41c7-9753-4b0cafd57772 ready: true, restart count 0 Mar 11 16:40:28.184: INFO: ebs-csi-node-2qlbt started at 2023-03-11 16:26:03 +0000 UTC (0+3 container statuses recorded) Mar 11 16:40:28.184: INFO: Container ebs-plugin ready: true, restart count 0 Mar 11 16:40:28.184: INFO: Container liveness-probe ready: true, restart count 0 Mar 11 16:40:28.184: INFO: Container node-driver-registrar ready: true, restart count 0 Mar 11 16:40:28.184: INFO: cleanup20-94fd1eef-e542-41c7-9753-4b0cafd57772-l2jkm started at 2023-03-11 16:38:43 +0000 UTC (0+1 container statuses recorded) Mar 11 16:40:28.184: INFO: Container cleanup20-94fd1eef-e542-41c7-9753-4b0cafd57772 ready: true, restart count 0 Mar 11 16:40:28.184: INFO: cleanup20-94fd1eef-e542-41c7-9753-4b0cafd57772-j5j54 started at 2023-03-11 16:38:43 +0000 UTC (0+1 container statuses recorded) Mar 11 16:40:28.184: INFO: Container cleanup20-94fd1eef-e542-41c7-9753-4b0cafd57772 ready: true, restart count 0 Mar 11 16:40:28.184: INFO: cleanup20-94fd1eef-e542-41c7-9753-4b0cafd57772-6fnkj started at 2023-03-11 16:38:43 +0000 UTC (0+1 container statuses recorded) Mar 11 16:40:28.184: INFO: Container cleanup20-94fd1eef-e542-41c7-9753-4b0cafd57772 ready: true, restart count 0 Mar 11 16:40:28.184: INFO: csi-mockplugin-0 started at 2023-03-11 16:39:53 +0000 UTC (0+3 container statuses recorded) Mar 11 16:40:28.184: INFO: Container csi-provisioner ready: true, restart count 0 Mar 11 16:40:28.184: INFO: Container driver-registrar ready: true, restart count 0 Mar 11 16:40:28.184: INFO: Container mock ready: true, restart count 0 Mar 11 16:40:28.184: INFO: service-proxy-disabled-rvkzp started at 2023-03-11 16:37:29 +0000 UTC (0+1 container statuses recorded) Mar 11 16:40:28.184: INFO: Container service-proxy-disabled ready: true, restart count 0 Mar 11 16:40:28.184: INFO: cleanup20-94fd1eef-e542-41c7-9753-4b0cafd57772-rrh87 started at 2023-03-11 16:38:43 +0000 UTC (0+1 container statuses recorded) Mar 11 16:40:28.184: INFO: Container cleanup20-94fd1eef-e542-41c7-9753-4b0cafd57772 ready: true, restart count 0 Mar 11 16:40:28.184: INFO: cleanup20-94fd1eef-e542-41c7-9753-4b0cafd57772-2xmt8 started at 2023-03-11 16:38:43 +0000 UTC (0+1 container statuses recorded) Mar 11 16:40:28.184: INFO: Container cleanup20-94fd1eef-e542-41c7-9753-4b0cafd57772 ready: true, restart count 0 Mar 11 16:40:28.184: INFO: service-proxy-toggled-nj9m9 started at 2023-03-11 16:37:32 +0000 UTC (0+1 container statuses recorded) Mar 11 16:40:28.184: INFO: Container service-proxy-toggled ready: true, restart count 0 Mar 11 16:40:28.184: INFO: coredns-autoscaler-8495f5575-d8gqb started at 2023-03-11 16:26:12 +0000 UTC (0+1 container statuses recorded) Mar 11 16:40:28.184: INFO: Container autoscaler ready: true, restart count 0 Mar 11 16:40:28.184: INFO: cleanup20-94fd1eef-e542-41c7-9753-4b0cafd57772-mgv2f started at 2023-03-11 16:38:43 +0000 UTC (0+1 container statuses recorded) Mar 11 16:40:28.184: INFO: Container cleanup20-94fd1eef-e542-41c7-9753-4b0cafd57772 ready: true, restart count 0 Mar 11 16:40:28.184: INFO: service-proxy-toggled-pxtkv started at 2023-03-11 16:37:32 +0000 UTC (0+1 container statuses recorded) Mar 11 16:40:28.184: INFO: Container service-proxy-toggled ready: true, restart count 0 Mar 11 16:40:28.831: INFO: Latency metrics for node i-04523d1dffb0fe725 Mar 11 16:40:28.831: INFO: Logging node info for node i-07e2b6e650d7ef56d Mar 11 16:40:28.978: INFO: Node Info: &Node{ObjectMeta:{i-07e2b6e650d7ef56d fcc65272-6f1b-4228-ac59-e9e3e92cdb2c 33740 0 2023-03-11 16:24:19 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:arm64 beta.kubernetes.io/instance-type:t4g.small beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-northeast-1 failure-domain.beta.kubernetes.io/zone:ap-northeast-1a karpenter.k8s.aws/instance-category:t karpenter.k8s.aws/instance-cpu:2 karpenter.k8s.aws/instance-family:t4g karpenter.k8s.aws/instance-generation:4 karpenter.k8s.aws/instance-hypervisor:nitro karpenter.k8s.aws/instance-memory:2048 karpenter.k8s.aws/instance-pods:11 karpenter.k8s.aws/instance-size:small karpenter.sh/capacity-type:spot karpenter.sh/initialized:true karpenter.sh/provisioner-name:nodes kubelet_cleanup:true kubernetes.io/arch:arm64 kubernetes.io/hostname:i-07e2b6e650d7ef56d kubernetes.io/os:linux node-role.kubernetes.io/node: node-role.kubernetes.io/spot-worker:true node.kubernetes.io/instance-type:t4g.small topology.ebs.csi.aws.com/zone:ap-northeast-1a topology.hostpath.csi/node:i-07e2b6e650d7ef56d topology.kubernetes.io/region:ap-northeast-1 topology.kubernetes.io/zone:ap-northeast-1a] map[csi.volume.kubernetes.io/nodeid:{"csi-hostpath-provisioning-4792":"i-07e2b6e650d7ef56d","csi-mock-csi-mock-volumes-expansion-9638":"csi-mock-csi-mock-volumes-expansion-9638","ebs.csi.aws.com":"i-07e2b6e650d7ef56d"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [{karpenter.sh/v1alpha5 Provisioner nodes ac7283b2-eed0-4a9f-9ddd-616f33a0991a <nil> 0xc0048e8910}] [karpenter.sh/termination] [{aws-cloud-controller-manager Update v1 2023-03-11 16:24:20 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{}}}} } {kops-controller Update v1 2023-03-11 16:24:20 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{},"f:node-role.kubernetes.io/spot-worker":{}}}} } {karpenter Update v1 2023-03-11 16:26:15 +0000 UTC FieldsV1 {"f:metadata":{"f:finalizers":{".":{},"v:\"karpenter.sh/termination\"":{}},"f:labels":{".":{},"f:karpenter.k8s.aws/instance-category":{},"f:karpenter.k8s.aws/instance-cpu":{},"f:karpenter.k8s.aws/instance-family":{},"f:karpenter.k8s.aws/instance-generation":{},"f:karpenter.k8s.aws/instance-hypervisor":{},"f:karpenter.k8s.aws/instance-memory":{},"f:karpenter.k8s.aws/instance-pods":{},"f:karpenter.k8s.aws/instance-size":{},"f:karpenter.sh/capacity-type":{},"f:karpenter.sh/initialized":{},"f:karpenter.sh/provisioner-name":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}},"f:ownerReferences":{".":{},"k:{\"uid\":\"ac7283b2-eed0-4a9f-9ddd-616f33a0991a\"}":{}}},"f:spec":{"f:providerID":{}}} } {kube-controller-manager Update v1 2023-03-11 16:26:15 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:node.alpha.kubernetes.io/ttl":{}},"f:labels":{"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.1.0/24\"":{}}}} } {aws-cloud-controller-manager Update v1 2023-03-11 16:26:50 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{".":{},"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"InternalIP\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {e2e.test Update v1 2023-03-11 16:38:42 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubelet_cleanup":{}}}} } {kube-controller-manager Update v1 2023-03-11 16:40:25 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{".":{},"k:{\"type\":\"DiskPressure\"}":{".":{},"f:type":{}},"k:{\"type\":\"MemoryPressure\"}":{".":{},"f:type":{}},"k:{\"type\":\"PIDPressure\"}":{".":{},"f:type":{}},"k:{\"type\":\"Ready\"}":{".":{},"f:type":{}}},"f:volumesAttached":{}}} status} {kubelet Update v1 2023-03-11 16:40:27 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{"f:kubernetes.io/hostname":{},"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{".":{},"f:cpu":{},"f:ephemeral-storage":{},"f:hugepages-1Gi":{},"f:hugepages-2Mi":{},"f:hugepages-32Mi":{},"f:hugepages-64Ki":{},"f:memory":{},"f:pods":{}},"f:capacity":{".":{},"f:cpu":{},"f:ephemeral-storage":{},"f:hugepages-1Gi":{},"f:hugepages-2Mi":{},"f:hugepages-32Mi":{},"f:hugepages-64Ki":{},"f:memory":{},"f:pods":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:daemonEndpoints":{"f:kubeletEndpoint":{"f:Port":{}}},"f:images":{},"f:nodeInfo":{"f:architecture":{},"f:bootID":{},"f:containerRuntimeVersion":{},"f:kernelVersion":{},"f:kubeProxyVersion":{},"f:kubeletVersion":{},"f:machineID":{},"f:operatingSystem":{},"f:osImage":{},"f:systemUUID":{}},"f:volumesInUse":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.1.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-northeast-1a/i-07e2b6e650d7ef56d,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.1.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49756430336 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-32Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-64Ki: {{0 0} {<nil>} 0 DecimalSI},memory: {{1941549056 0} {<nil>} 1896044Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44780787229 0} {<nil>} 44780787229 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-32Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-64Ki: {{0 0} {<nil>} 0 DecimalSI},memory: {{1836691456 0} {<nil>} 1793644Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-03-11 16:40:27 +0000 UTC,LastTransitionTime:2023-03-11 16:26:15 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-03-11 16:40:27 +0000 UTC,LastTransitionTime:2023-03-11 16:26:05 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-03-11 16:40:27 +0000 UTC,LastTransitionTime:2023-03-11 16:26:05 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-03-11 16:40:27 +0000 UTC,LastTransitionTime:2023-03-11 16:26:05 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.35.76,},NodeAddress{Type:ExternalIP,Address:54.238.106.83,},NodeAddress{Type:InternalDNS,Address:i-07e2b6e650d7ef56d.ap-northeast-1.compute.internal,},NodeAddress{Type:Hostname,Address:i-07e2b6e650d7ef56d.ap-northeast-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-54-238-106-83.ap-northeast-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2083a8e9d27d02fac967340367f875,SystemUUID:ec2083a8-e9d2-7d02-fac9-67340367f875,BootID:ea264ed6-5b0d-47d6-ad5d-6bd2a1f7c11b,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 22.04.2 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.27.0-alpha.3.420+bb6c6fad2c19b3,KubeProxyVersion:v1.27.0-alpha.3.420+bb6c6fad2c19b3,OperatingSystem:linux,Architecture:arm64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:157636062,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/jessie-dnsutils@sha256:24aaf2626d6b27864c29de2097e8bbb840b3a414271bf7c8995e431e47d8408e registry.k8s.io/e2e-test-images/jessie-dnsutils:1.7],SizeBytes:116764683,},ContainerImage{Names:[registry.k8s.io/kube-proxy-arm64:v1.27.0-alpha.3.420_bb6c6fad2c19b3],SizeBytes:60947298,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:16bbf38c463a4223d8cfe4da12bc61010b082a79b4bb003e2d3ba3ece5dd5f9e registry.k8s.io/e2e-test-images/agnhost:2.43],SizeBytes:49808364,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nautilus@sha256:80ba6c8c44f9623f06e868a1aa66026c8ec438ad814f9ec95e9333b415fe3550 registry.k8s.io/e2e-test-images/nautilus:1.7],SizeBytes:47953400,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:3fe7acf013d1264ffded116b80a73dc129a449b0fccdb8d21af8279f2233f36e registry.k8s.io/e2e-test-images/httpd:2.4.39-4],SizeBytes:41528818,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:148b022f5c5da426fc2f3c14b5c0867e58ef05961510c84749ac1fddcb0fef22 registry.k8s.io/e2e-test-images/httpd:2.4.38-4],SizeBytes:40343601,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:28932904,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:ee3b525d5b89db99da3b8eb521d9cd90cb6e9ef0fbb651e98bb37be78d36b5b8 registry.k8s.io/sig-storage/csi-provisioner:v3.3.0],SizeBytes:23945474,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:425d8f1b769398127767b06ed97ce62578a3179bcb99809ce93a1649e025ffe7 registry.k8s.io/sig-storage/csi-resizer:v1.6.0],SizeBytes:22678632,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:291334908ddf71a4661fd7f6d9d97274de8a5378a2b6fdfeb2ce73414a34f82f registry.k8s.io/sig-storage/csi-snapshotter:v6.1.0],SizeBytes:22434180,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:9a685020911e2725ad019dbce6e4a5ab93d51e3d4557f115e64343345e05781b registry.k8s.io/sig-storage/csi-attacher:v4.0.0],SizeBytes:22394501,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:122bfb8c1edabb3c0edd63f06523e6940d958d19b3957dc7b1d6f81e9f1f6119 registry.k8s.io/sig-storage/csi-provisioner:v3.1.0],SizeBytes:21735644,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:9ebbf9f023e7b41ccee3d52afe39a89e3ddacdbb69269d583abfc25847cfd9e4 registry.k8s.io/sig-storage/csi-resizer:v1.4.0],SizeBytes:20848413,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:8b9c313c05f54fb04f8d430896f5f5904b6cb157df261501b29adc04d2b2dc7b registry.k8s.io/sig-storage/csi-attacher:v3.4.0],SizeBytes:20565480,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nonroot@sha256:ee9f50b3c64b174d296d91ca9f69a914ac30e59095dfb462b2b518ad28a63655 registry.k8s.io/e2e-test-images/nonroot:1.4],SizeBytes:18561972,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:92257881c1d6493cf18299a24af42330f891166560047902b8d431fb66b01af5 registry.k8s.io/sig-storage/hostpathplugin:v1.9.0],SizeBytes:18267611,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:9b2d9abda017c28b12f84a344f57cd73fbdb6c2bd7dd5bdd5018246ad1093ba6 registry.k8s.io/sig-storage/hostpathplugin:v1.11.0],SizeBytes:18159085,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:017727efcfeb7d053af68e51436ce8e65edbc6ca573720afb4f79c8594036955 registry.k8s.io/coredns/coredns:v1.10.0],SizeBytes:13766234,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:8502345,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:933940f13b3ea0abc62e656c1aa5c5b47c04b15d71250413a6b821bd0c58b94e registry.k8s.io/sig-storage/livenessprobe:v2.7.0],SizeBytes:8223819,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:7685305,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:db048754ae68ae337d8fa96494c96d2a1204c3320f5dcf7e8e71085adec85da6 registry.k8s.io/e2e-test-images/nginx:1.15-4],SizeBytes:6883347,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:5c99cf6a02adda929b10321dbf4ecfa00d87be9ba4fb456006237d530ab4baa1 registry.k8s.io/e2e-test-images/nginx:1.14-4],SizeBytes:6860089,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nonewprivs@sha256:8ac1264691820febacf3aea5d152cbde6d10685731ec14966a9401c6f47a68ac registry.k8s.io/e2e-test-images/nonewprivs:1.3],SizeBytes:5025027,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf registry.k8s.io/e2e-test-images/busybox:1.29-2],SizeBytes:773367,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:2e0f836850e09b8b7cc937681d6194537a09fbd5f6b9e08f4d646a85128e8937 registry.k8s.io/e2e-test-images/busybox:1.29-4],SizeBytes:772933,},ContainerImage{Names:[registry.k8s.io/pause@sha256:7031c1b283388d2c2e09b57badb803c05ebed362dc88d84b480cc47f72a21097 registry.k8s.io/pause:3.9],SizeBytes:268051,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:253553,},},VolumesInUse:[kubernetes.io/csi/csi-hostpath-provisioning-4792^669f339d-c02b-11ed-9bb8-66151c847aca kubernetes.io/csi/ebs.csi.aws.com^vol-0af26e592f763b1a9],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/csi-hostpath-provisioning-4792^669f339d-c02b-11ed-9bb8-66151c847aca,DevicePath:,},},Config:nil,},} Mar 11 16:40:28.979: INFO: Logging kubelet events for node i-07e2b6e650d7ef56d Mar 11 16:40:29.168: INFO: Logging pods the kubelet thinks is on node i-07e2b6e650d7ef56d Mar 11 16:40:29.328: INFO: csi-hostpathplugin-0 started at 2023-03-11 16:38:04 +0000 UTC (0+7 container statuses recorded) Mar 11 16:40:29.328: INFO: Container csi-attacher ready: true, restart count 0 Mar 11 16:40:29.328: INFO: Container csi-provisioner ready: true, restart count 0 Mar 11 16:40:29.328: INFO: Container csi-resizer ready: true, restart count 0 Mar 11 16:40:29.328: INFO: Container csi-snapshotter ready: true, restart count 0 Mar 11 16:40:29.328: INFO: Container hostpath ready: true, restart count 0 Mar 11 16:40:29.328: INFO: Container liveness-probe ready: true, restart count 0 Mar 11 16:40:29.328: INFO: Container node-driver-registrar ready: true, restart count 0 Mar 11 16:40:29.328: INFO: cilium-mvm8l started at 2023-03-11 16:26:06 +0000 UTC (1+1 container statuses recorded) Mar 11 16:40:29.328: INFO: Init container clean-cilium-state ready: true, restart count 0 Mar 11 16:40:29.328: INFO: Container cilium-agent ready: true, restart count 0 Mar 11 16:40:29.328: INFO: pod-secrets-4d2edcac-66e2-49bb-bc24-b64218dc5774 started at 2023-03-11 16:40:26 +0000 UTC (0+1 container statuses recorded) Mar 11 16:40:29.328: INFO: Container secret-volume-test ready: false, restart count 0 Mar 11 16:40:29.328: INFO: ebs-csi-node-m5b9f started at 2023-03-11 16:26:06 +0000 UTC (0+3 container statuses recorded) Mar 11 16:40:29.328: INFO: Container ebs-plugin ready: true, restart count 1 Mar 11 16:40:29.328: INFO: Container liveness-probe ready: true, restart count 0 Mar 11 16:40:29.328: INFO: Container node-driver-registrar ready: true, restart count 0 Mar 11 16:40:29.328: INFO: pod-subpath-test-dynamicpv-x2sc started at 2023-03-11 16:40:16 +0000 UTC (1+1 container statuses recorded) Mar 11 16:40:29.328: INFO: Init container init-volume-dynamicpv-x2sc ready: false, restart count 0 Mar 11 16:40:29.328: INFO: Container test-container-subpath-dynamicpv-x2sc ready: false, restart count 0 Mar 11 16:40:29.328: INFO: test-rollover-deployment-57777854c9-7fdr2 started at 2023-03-11 16:40:27 +0000 UTC (0+1 container statuses recorded) Mar 11 16:40:29.328: INFO: Container agnhost ready: false, restart count 0 Mar 11 16:40:29.328: INFO: csi-mockplugin-0 started at 2023-03-11 16:37:24 +0000 UTC (0+4 container statuses recorded) Mar 11 16:40:29.328: INFO: Container busybox ready: true, restart count 0 Mar 11 16:40:29.328: INFO: Container csi-provisioner ready: true, restart count 0 Mar 11 16:40:29.328: INFO: Container driver-registrar ready: true, restart count 0 Mar 11 16:40:29.328: INFO: Container mock ready: true, restart count 0 Mar 11 16:40:29.328: INFO: image-pull-test8223cc69-b3b4-4e95-bd26-e216c714de83 started at 2023-03-11 16:40:13 +0000 UTC (0+1 container statuses recorded) Mar 11 16:40:29.328: INFO: Container image-pull-test ready: false, restart count 0 Mar 11 16:40:29.328: INFO: cleanup20-94fd1eef-e542-41c7-9753-4b0cafd57772-dhtkv started at 2023-03-11 16:39:33 +0000 UTC (0+1 container statuses recorded) Mar 11 16:40:29.328: INFO: Container cleanup20-94fd1eef-e542-41c7-9753-4b0cafd57772 ready: false, restart count 0 Mar 11 16:40:29.328: INFO: cleanup20-94fd1eef-e542-41c7-9753-4b0cafd57772-nrq4h started at 2023-03-11 16:39:34 +0000 UTC (0+1 container statuses recorded) Mar 11 16:40:29.328: INFO: Container cleanup20-94fd1eef-e542-41c7-9753-4b0cafd57772 ready: false, restart count 0 Mar 11 16:40:29.328: INFO: cleanup20-94fd1eef-e542-41c7-9753-4b0cafd57772-65b9d started at 2023-03-11 16:39:34 +0000 UTC (0+1 container statuses recorded) Mar 11 16:40:29.328: INFO: Container cleanup20-94fd1eef-e542-41c7-9753-4b0cafd57772 ready: false, restart count 0 Mar 11 16:40:29.328: INFO: cleanup20-94fd1eef-e542-41c7-9753-4b0cafd57772-cwwd2 started at 2023-03-11 16:39:34 +0000 UTC (0+1 container statuses recorded) Mar 11 16:40:29.328: INFO: Container cleanup20-94fd1eef-e542-41c7-9753-4b0cafd57772 ready: false, restart count 0 Mar 11 16:40:29.328: INFO: pod-secrets-66207379-4694-41ae-92fa-b53d37b18b75 started at 2023-03-11 16:40:22 +0000 UTC (0+1 container statuses recorded) Mar 11 16:40:29.328: INFO: Container secret-test ready: false, restart count 0 Mar 11 16:40:29.328: INFO: sample-webhook-deployment-7497495989-ggnfh started at 2023-03-11 16:39:53 +0000 UTC (0+1 container statuses recorded) Mar 11 16:40:29.328: INFO: Container sample-webhook ready: false, restart count 0 Mar 11 16:40:29.328: INFO: funny-ip-9w5ps started at 2023-03-11 16:37:52 +0000 UTC (0+1 container statuses recorded) Mar 11 16:40:29.328: INFO: Container funny-ip ready: true, restart count 0 Mar 11 16:40:29.328: INFO: cleanup20-94fd1eef-e542-41c7-9753-4b0cafd57772-t589q started at 2023-03-11 16:39:34 +0000 UTC (0+1 container statuses recorded) Mar 11 16:40:29.328: INFO: Container cleanup20-94fd1eef-e542-41c7-9753-4b0cafd57772 ready: false, restart count 0 Mar 11 16:40:29.328: INFO: csi-mockplugin-attacher-0 started at 2023-03-11 16:37:24 +0000 UTC (0+1 container statuses recorded) Mar 11 16:40:29.328: INFO: Container csi-attacher ready: true, restart count 0 Mar 11 16:40:29.328: INFO: cleanup20-94fd1eef-e542-41c7-9753-4b0cafd57772-xhwxr started at 2023-03-11 16:39:34 +0000 UTC (0+1 container statuses recorded) Mar 11 16:40:29.328: INFO: Container cleanup20-94fd1eef-e542-41c7-9753-4b0cafd57772 ready: false, restart count 0 Mar 11 16:40:29.328: INFO: ebs-csi-controller-79785c665b-lpmmf started at 2023-03-11 16:26:15 +0000 UTC (0+5 container statuses recorded) Mar 11 16:40:29.328: INFO: Container csi-attacher ready: true, restart count 0 Mar 11 16:40:29.328: INFO: Container csi-provisioner ready: true, restart count 0 Mar 11 16:40:29.328: INFO: Container csi-resizer ready: true, restart count 0 Mar 11 16:40:29.328: INFO: Container ebs-plugin ready: true, restart count 0 Mar 11 16:40:29.328: INFO: Container liveness-probe ready: true, restart count 0 Mar 11 16:40:29.328: INFO: cleanup20-94fd1eef-e542-41c7-9753-4b0cafd57772-l7d9x started at 2023-03-11 16:39:34 +0000 UTC (0+1 container statuses recorded) Mar 11 16:40:29.328: INFO: Container cleanup20-94fd1eef-e542-41c7-9753-4b0cafd57772 ready: false, restart count 0 Mar 11 16:40:29.328: INFO: pod-44bb9b77-9e64-4d2e-9b83-4ef43ae7375e started at 2023-03-11 16:40:29 +0000 UTC (0+1 container statuses recorded) Mar 11 16:40:29.328: INFO: Container write-pod ready: false, restart count 0 Mar 11 16:40:29.328: INFO: cleanup20-94fd1eef-e542-41c7-9753-4b0cafd57772-vpmwb started at 2023-03-11 16:39:34 +0000 UTC (0+1 container statuses recorded) Mar 11 16:40:29.328: INFO: Container cleanup20-94fd1eef-e542-41c7-9753-4b0cafd57772 ready: true, restart count 0 Mar 11 16:40:29.328: INFO: cleanup20-94fd1eef-e542-41c7-9753-4b0cafd57772-tcvrh started at 2023-03-11 16:39:34 +0000 UTC (0+1 container statuses recorded) Mar 11 16:40:29.328: INFO: Container cleanup20-94fd1eef-e542-41c7-9753-4b0cafd57772 ready: true, restart count 0 Mar 11 16:40:29.328: INFO: test-webserver-a9d538d3-8801-47a1-b3e0-a52bd8a08bf3 started at 2023-03-11 16:36:55 +0000 UTC (0+1 container statuses recorded) Mar 11 16:40:29.328: INFO: Container test-webserver ready: true, restart count 0 Mar 11 16:40:29.328: INFO: csi-mockplugin-resizer-0 started at 2023-03-11 16:37:24 +0000 UTC (0+1 container statuses recorded) Mar 11 16:40:29.328: INFO: Container csi-resizer ready: true, restart count 0 Mar 11 16:40:29.328: INFO: pod-ephm-test-secret-vtrk started at 2023-03-11 16:38:33 +0000 UTC (0+1 container statuses recorded) Mar 11 16:40:29.328: INFO: Container test-container-subpath-secret-vtrk ready: false, restart count 0 Mar 11 16:40:29.328: INFO: coredns-745bddcfbb-lkbct started at 2023-03-11 16:26:41 +0000 UTC (0+1 container statuses recorded) Mar 11 16:40:29.328: INFO: Container coredns ready: true, restart count 0 Mar 11 16:40:29.328: INFO: test-grpc-21e6bf52-ac24-444f-9335-0685035e1c19 started at 2023-03-11 16:37:52 +0000 UTC (0+1 container statuses recorded) Mar 11 16:40:29.328: INFO: Container agnhost ready: true, restart count 1 Mar 11 16:40:29.328: INFO: all-succeed-gmh8p started at 2023-03-11 16:40:20 +0000 UTC (0+1 container statuses recorded) Mar 11 16:40:29.328: INFO: Container c ready: false, restart count 0 Mar 11 16:40:29.328: INFO: all-succeed-j5ghd started at 2023-03-11 16:40:20 +0000 UTC (0+1 container statuses recorded) Mar 11 16:40:29.328: INFO: Container c ready: false, restart count 0 Mar 11 16:40:29.328: INFO: httpd started at 2023-03-11 16:39:53 +0000 UTC (0+1 container statuses recorded) Mar 11 16:40:29.328: INFO: Container httpd ready: false, restart count 0 Mar 11 16:40:29.328: INFO: pod-subpath-test-dynamicpv-gpt9 started at 2023-03-11 16:40:25 +0000 UTC (1+2 container statuses recorded) Mar 11 16:40:29.328: INFO: Init container test-init-subpath-dynamicpv-gpt9 ready: false, restart count 0 Mar 11 16:40:29.328: INFO: Container test-container-subpath-dynamicpv-gpt9 ready: false, restart count 0 Mar 11 16:40:29.328: INFO: Container test-container-volume-dynamicpv-gpt9 ready: false, restart count 0 Mar 11 16:40:29.328: INFO: cleanup20-94fd1eef-e542-41c7-9753-4b0cafd57772-zgrfk started at 2023-03-11 16:39:34 +0000 UTC (0+1 container statuses recorded) Mar 11 16:40:29.328: INFO: Container cleanup20-94fd1eef-e542-41c7-9753-4b0cafd57772 ready: true, restart count 0 Mar 11 16:40:29.939: INFO: Latency metrics for node i-07e2b6e650d7ef56d Mar 11 16:40:29.939: INFO: Logging node info for node i-0e3467ec552af797e Mar 11 16:40:30.086: INFO: Node Info: &Node{ObjectMeta:{i-0e3467ec552af797e 6c9b0cc5-2c4e-4037-bf31-61bac12a7775 31369 0 2023-03-11 16:20:46 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:arm64 beta.kubernetes.io/instance-type:c6g.xlarge beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-northeast-1 failure-domain.beta.kubernetes.io/zone:ap-northeast-1a kops.k8s.io/kops-controller-pki: kubernetes.io/arch:arm64 kubernetes.io/hostname:i-0e3467ec552af797e kubernetes.io/os:linux node-role.kubernetes.io/control-plane: node.kubernetes.io/exclude-from-external-load-balancers: node.kubernetes.io/instance-type:c6g.xlarge topology.ebs.csi.aws.com/zone:ap-northeast-1a topology.kubernetes.io/region:ap-northeast-1 topology.kubernetes.io/zone:ap-northeast-1a] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-0e3467ec552af797e"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kubelet Update v1 2023-03-11 16:20:46 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {protokube Update v1 2023-03-11 16:21:30 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kops.k8s.io/kops-controller-pki":{},"f:node-role.kubernetes.io/control-plane":{},"f:node.kubernetes.io/exclude-from-external-load-balancers":{}}}} } {aws-cloud-controller-manager Update v1 2023-03-11 16:21:50 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-03-11 16:21:50 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kube-controller-manager Update v1 2023-03-11 16:21:58 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.0.0/24\"":{}},"f:taints":{}}} } {kubelet Update v1 2023-03-11 16:38:39 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.0.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-northeast-1a/i-0e3467ec552af797e,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/control-plane,Value:,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[100.96.0.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{4 0} {<nil>} 4 DecimalSI},ephemeral-storage: {{49756430336 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-32Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-64Ki: {{0 0} {<nil>} 0 DecimalSI},memory: {{8150110208 0} {<nil>} 7959092Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{4 0} {<nil>} 4 DecimalSI},ephemeral-storage: {{44780787229 0} {<nil>} 44780787229 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-32Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-64Ki: {{0 0} {<nil>} 0 DecimalSI},memory: {{8045252608 0} {<nil>} 7856692Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-03-11 16:38:39 +0000 UTC,LastTransitionTime:2023-03-11 16:20:46 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-03-11 16:38:39 +0000 UTC,LastTransitionTime:2023-03-11 16:20:46 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-03-11 16:38:39 +0000 UTC,LastTransitionTime:2023-03-11 16:20:46 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-03-11 16:38:39 +0000 UTC,LastTransitionTime:2023-03-11 16:21:58 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.40.18,},NodeAddress{Type:ExternalIP,Address:52.194.213.7,},NodeAddress{Type:InternalDNS,Address:i-0e3467ec552af797e.ap-northeast-1.compute.internal,},NodeAddress{Type:Hostname,Address:i-0e3467ec552af797e.ap-northeast-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-52-194-213-7.ap-northeast-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2abf2e564965592c7716accd56507e,SystemUUID:ec2abf2e-5649-6559-2c77-16accd56507e,BootID:fbada3ff-8655-404c-a98d-876577d4c322,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 22.04.2 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.27.0-alpha.3.420+bb6c6fad2c19b3,KubeProxyVersion:v1.27.0-alpha.3.420+bb6c6fad2c19b3,OperatingSystem:linux,Architecture:arm64,},Images:[]ContainerImage{ContainerImage{Names:[registry.k8s.io/etcd@sha256:9ce33ba33d8e738a5b85ed50b5080ac746deceed4a7496c550927a7a19ca3b6d registry.k8s.io/etcd:3.5.0-0],SizeBytes:157797353,},ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:157636062,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:4afb99b4690b418ffc2ceb67e1a17376457e441c1f09ab55447f0aaf992fa646 registry.k8s.io/etcd:3.4.3-0],SizeBytes:151677080,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:4ad90a11b55313b182afc186b9876c8e891531b8db4c9bf1541953021618d0e2 registry.k8s.io/etcd:3.4.13-0],SizeBytes:134531559,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:0d2ab6b15b4fe0b903b9e81192d942fb39c29be61fe341cd16db5d0165b0435c registry.k8s.io/etcd:3.3.17-0],SizeBytes:126979107,},ContainerImage{Names:[registry.k8s.io/kube-apiserver-arm64:v1.27.0-alpha.3.420_bb6c6fad2c19b3],SizeBytes:115123555,},ContainerImage{Names:[registry.k8s.io/kube-controller-manager-arm64:v1.27.0-alpha.3.420_bb6c6fad2c19b3],SizeBytes:107914235,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:17da501f5d2a675be46040422a27b7cc21b8a43895ac998b171db1c346f361f7 registry.k8s.io/etcd:3.3.10-0],SizeBytes:81436097,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:6f72b851544986cb0921b53ea655ec04c36131248f16d4ad110cb3ca0c369dc1 registry.k8s.io/etcd:3.5.4-0],SizeBytes:81120118,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:13f53ed1d91e2e11aac476ee9a0269fdda6cc4874eba903efd40daf50c55eee5 registry.k8s.io/etcd:3.5.3-0],SizeBytes:81114891,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:51eae8381dcb1078289fa7b4f3df2630cdc18d09fb56f8e56b41c40e191d6c83 registry.k8s.io/etcd:3.5.7-0],SizeBytes:80665728,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:dd75ec974b0a2a6f6bb47001ba09207976e625db898d1b16735528c009cb171c registry.k8s.io/etcd:3.5.6-0],SizeBytes:80539316,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:905d7ca17fd02bc24c0eba9a062753aba15db3e31422390bc3238eb762339b20 registry.k8s.io/etcd:3.2.24-1],SizeBytes:61500433,},ContainerImage{Names:[registry.k8s.io/kube-proxy-arm64:v1.27.0-alpha.3.420_bb6c6fad2c19b3],SizeBytes:60947298,},ContainerImage{Names:[registry.k8s.io/etcdadm/etcd-manager-slim@sha256:8727adc059334e0e3329dbccce116a734da4a090da5e86c8cddccaf333caa666 registry.k8s.io/etcdadm/etcd-manager-slim:v3.0.20230201],SizeBytes:60826655,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:64b9ea357325d5db9f8a723dcf503b5a449177b17ac87d69481e126bb724c263 registry.k8s.io/etcd:3.5.1-0],SizeBytes:59553904,},ContainerImage{Names:[registry.k8s.io/kube-scheduler-arm64:v1.27.0-alpha.3.420_bb6c6fad2c19b3],SizeBytes:56731120,},ContainerImage{Names:[registry.k8s.io/kops/kops-controller:1.27.0-alpha.1],SizeBytes:41885191,},ContainerImage{Names:[registry.k8s.io/kops/dns-controller:1.27.0-alpha.1],SizeBytes:40376038,},ContainerImage{Names:[public.ecr.aws/karpenter/controller@sha256:68db4f092cf9cc83f5ef9e2fbc5407c2cb682e81f64dfaa700a7602ede38b1cf public.ecr.aws/karpenter/controller:v0.16.3],SizeBytes:33707751,},ContainerImage{Names:[public.ecr.aws/karpenter/webhook@sha256:96a2d9b06d6bc5127801f358f74b1cf2d289b423a2e9ba40c573c0b14b17dafa public.ecr.aws/karpenter/webhook:v0.16.3],SizeBytes:33307187,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:28932904,},ContainerImage{Names:[quay.io/cilium/operator@sha256:a6d24a006a6b92967ac90786b49bc1ac26e5477cf028cd1186efcfc2466484db quay.io/cilium/operator:v1.12.5],SizeBytes:24567287,},ContainerImage{Names:[registry.k8s.io/provider-aws/cloud-controller-manager@sha256:fdeb61e3e42ecd9cca868d550ebdb88dd6341d9e91fcfa9a37e227dab2ad22cb registry.k8s.io/provider-aws/cloud-controller-manager:v1.26.0],SizeBytes:18345226,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:8502345,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:7685305,},ContainerImage{Names:[registry.k8s.io/kops/kube-apiserver-healthcheck:1.27.0-alpha.1],SizeBytes:4740762,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:253553,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Mar 11 16:40:30.087: INFO: Logging kubelet events for node i-0e3467ec552af797e Mar 11 16:40:30.237: INFO: Logging pods the kubelet thinks is on node i-0e3467ec552af797e Mar 11 16:40:30.391: INFO: etcd-manager-events-i-0e3467ec552af797e started at 2023-03-11 16:18:44 +0000 UTC (11+1 container statuses recorded) Mar 11 16:40:30.391: INFO: Init container init-etcd-3-2-24 ready: true, restart count 0 Mar 11 16:40:30.391: INFO: Init container init-etcd-3-3-10 ready: true, restart count 0 Mar 11 16:40:30.391: INFO: Init container init-etcd-3-3-17 ready: true, restart count 0 Mar 11 16:40:30.391: INFO: Init container init-etcd-3-4-13 ready: true, restart count 0 Mar 11 16:40:30.391: INFO: Init container init-etcd-3-4-3 ready: true, restart count 0 Mar 11 16:40:30.391: INFO: Init container init-etcd-3-5-0 ready: true, restart count 0 Mar 11 16:40:30.391: INFO: Init container init-etcd-3-5-1 ready: true, restart count 0 Mar 11 16:40:30.391: INFO: Init container init-etcd-3-5-3 ready: true, restart count 0 Mar 11 16:40:30.391: INFO: Init container init-etcd-3-5-4 ready: true, restart count 0 Mar 11 16:40:30.391: INFO: Init container init-etcd-3-5-6 ready: true, restart count 0 Mar 11 16:40:30.391: INFO: Init container init-etcd-3-5-7 ready: true, restart count 0 Mar 11 16:40:30.391: INFO: Container etcd-manager ready: true, restart count 0 Mar 11 16:40:30.391: INFO: etcd-manager-main-i-0e3467ec552af797e started at 2023-03-11 16:18:44 +0000 UTC (11+1 container statuses recorded) Mar 11 16:40:30.391: INFO: Init container init-etcd-3-2-24 ready: true, restart count 0 Mar 11 16:40:30.391: INFO: Init container init-etcd-3-3-10 ready: true, restart count 0 Mar 11 16:40:30.391: INFO: Init container init-etcd-3-3-17 ready: true, restart count 0 Mar 11 16:40:30.391: INFO: Init container init-etcd-3-4-13 ready: true, restart count 0 Mar 11 16:40:30.391: INFO: Init container init-etcd-3-4-3 ready: true, restart count 0 Mar 11 16:40:30.391: INFO: Init container init-etcd-3-5-0 ready: true, restart count 0 Mar 11 16:40:30.391: INFO: Init container init-etcd-3-5-1 ready: true, restart count 0 Mar 11 16:40:30.391: INFO: Init container init-etcd-3-5-3 ready: true, restart count 0 Mar 11 16:40:30.391: INFO: Init container init-etcd-3-5-4 ready: true, restart count 0 Mar 11 16:40:30.391: INFO: Init container init-etcd-3-5-6 ready: true, restart count 0 Mar 11 16:40:30.391: INFO: Init container init-etcd-3-5-7 ready: true, restart count 0 Mar 11 16:40:30.391: INFO: Container etcd-manager ready: true, restart count 0 Mar 11 16:40:30.391: INFO: ebs-csi-node-lpwz9 started at 2023-03-11 16:21:37 +0000 UTC (0+3 container statuses recorded) Mar 11 16:40:30.391: INFO: Container ebs-plugin ready: true, restart count 0 Mar 11 16:40:30.391: INFO: Container liveness-probe ready: true, restart count 0 Mar 11 16:40:30.391: INFO: Container node-driver-registrar ready: true, restart count 0 Mar 11 16:40:30.391: INFO: karpenter-webhook-5b646cd4f7-6sg2k started at 2023-03-11 16:21:58 +0000 UTC (0+1 container statuses recorded) Mar 11 16:40:30.391: INFO: Container webhook ready: true, restart count 0 Mar 11 16:40:30.391: INFO: karpenter-controller-567749677d-d9szs started at 2023-03-11 16:21:58 +0000 UTC (0+1 container statuses recorded) Mar 11 16:40:30.391: INFO: Container controller ready: true, restart count 0 Mar 11 16:40:30.391: INFO: kube-controller-manager-i-0e3467ec552af797e started at 2023-03-11 16:18:44 +0000 UTC (0+1 container statuses recorded) Mar 11 16:40:30.391: INFO: Container kube-controller-manager ready: true, restart count 4 Mar 11 16:40:30.391: INFO: kube-scheduler-i-0e3467ec552af797e started at 2023-03-11 16:18:44 +0000 UTC (0+1 container statuses recorded) Mar 11 16:40:30.391: INFO: Container kube-scheduler ready: true, restart count 0 Mar 11 16:40:30.391: INFO: dns-controller-7dfd64d999-q9hh7 started at 2023-03-11 16:21:37 +0000 UTC (0+1 container statuses recorded) Mar 11 16:40:30.391: INFO: Container dns-controller ready: true, restart count 0 Mar 11 16:40:30.391: INFO: cilium-qrsfn started at 2023-03-11 16:21:37 +0000 UTC (1+1 container statuses recorded) Mar 11 16:40:30.391: INFO: Init container clean-cilium-state ready: true, restart count 0 Mar 11 16:40:30.391: INFO: Container cilium-agent ready: true, restart count 0 Mar 11 16:40:30.391: INFO: cilium-operator-8496cc65f5-qhdg8 started at 2023-03-11 16:21:37 +0000 UTC (0+1 container statuses recorded) Mar 11 16:40:30.391: INFO: Container cilium-operator ready: true, restart count 0 Mar 11 16:40:30.391: INFO: kops-controller-76gkw started at 2023-03-11 16:21:37 +0000 UTC (0+1 container statuses recorded) Mar 11 16:40:30.391: INFO: Container kops-controller ready: true, restart count 0 Mar 11 16:40:30.391: INFO: aws-cloud-controller-manager-kwbw2 started at 2023-03-11 16:21:37 +0000 UTC (0+1 container statuses recorded) Mar 11 16:40:30.391: INFO: Container aws-cloud-controller-manager ready: true, restart count 0 Mar 11 16:40:30.391: INFO: kube-apiserver-i-0e3467ec552af797e started at 2023-03-11 16:18:44 +0000 UTC (0+2 container statuses recorded) Mar 11 16:40:30.391: INFO: Container healthcheck ready: true, restart count 0 Mar 11 16:40:30.391: INFO: Container kube-apiserver ready: true, restart count 3 Mar 11 16:40:30.901: INFO: Latency metrics for node i-0e3467ec552af797e END STEP: dump namespace information after failure - test/e2e/framework/framework.go:288 @ 03/11/23 16:40:30.901 (3.617s) < Exit [DeferCleanup (Each)] [sig-node] Probing container - dump namespaces | framework.go:209 @ 03/11/23 16:40:30.901 (3.617s) > Enter [DeferCleanup (Each)] [sig-node] Probing container - tear down framework | framework.go:206 @ 03/11/23 16:40:30.901 STEP: Destroying namespace "container-probe-271" for this suite. - test/e2e/framework/framework.go:351 @ 03/11/23 16:40:30.901 < Exit [DeferCleanup (Each)] [sig-node] Probing container - tear down framework | framework.go:206 @ 03/11/23 16:40:31.051 (150ms) > Enter [ReportAfterEach] TOP-LEVEL - test/e2e/e2e_test.go:144 @ 03/11/23 16:40:31.051 < Exit [ReportAfterEach] TOP-LEVEL - test/e2e/e2e_test.go:144 @ 03/11/23 16:40:31.051 (0s)
Find container-probe-271/liveness-ecf7c9aa-a3c6-4146-b171-4bb7b637f384 mentions in log files | View test history on testgrid
go run hack/e2e.go -v --test --test_args='--ginkgo.focus=Kubernetes\se2e\ssuite\s\[It\]\s\[sig\-node\]\sProbing\scontainer\sshould\sbe\srestarted\sby\sliveness\sprobe\safter\sstartup\sprobe\senables\sit$'
[FAILED] pod container-probe-4831/startup-81ef738d-6302-42b0-9bac-a06875bbc4dd - expected number of restarts: 1, found restarts: 0 In [It] at: test/e2e/common/node/container_probe.go:1000 @ 03/11/23 16:48:06.752from junit_01.xml
> Enter [BeforeEach] [sig-node] Probing container - set up framework | framework.go:191 @ 03/11/23 16:44:02.471 STEP: Creating a kubernetes client - test/e2e/framework/framework.go:211 @ 03/11/23 16:44:02.471 Mar 11 16:44:02.471: INFO: >>> kubeConfig: /root/.kube/config STEP: Building a namespace api object, basename container-probe - test/e2e/framework/framework.go:250 @ 03/11/23 16:44:02.472 STEP: Waiting for a default service account to be provisioned in namespace - test/e2e/framework/framework.go:259 @ 03/11/23 16:44:02.924 STEP: Waiting for kube-root-ca.crt to be provisioned in namespace - test/e2e/framework/framework.go:262 @ 03/11/23 16:44:03.218 < Exit [BeforeEach] [sig-node] Probing container - set up framework | framework.go:191 @ 03/11/23 16:44:03.512 (1.041s) > Enter [BeforeEach] [sig-node] Probing container - test/e2e/framework/metrics/init/init.go:33 @ 03/11/23 16:44:03.512 < Exit [BeforeEach] [sig-node] Probing container - test/e2e/framework/metrics/init/init.go:33 @ 03/11/23 16:44:03.512 (0s) > Enter [BeforeEach] [sig-node] Probing container - test/e2e/common/node/container_probe.go:62 @ 03/11/23 16:44:03.512 < Exit [BeforeEach] [sig-node] Probing container - test/e2e/common/node/container_probe.go:62 @ 03/11/23 16:44:03.512 (0s) > Enter [It] should be restarted by liveness probe after startup probe enables it - test/e2e/common/node/container_probe.go:378 @ 03/11/23 16:44:03.512 STEP: Creating pod startup-81ef738d-6302-42b0-9bac-a06875bbc4dd in namespace container-probe-4831 - test/e2e/common/node/container_probe.go:955 @ 03/11/23 16:44:03.512 Mar 11 16:44:05.959: INFO: Started pod startup-81ef738d-6302-42b0-9bac-a06875bbc4dd in namespace container-probe-4831 STEP: checking the pod's current state and verifying that restartCount is present - test/e2e/common/node/container_probe.go:966 @ 03/11/23 16:44:05.959 Mar 11 16:44:06.107: INFO: Initial restart count of pod startup-81ef738d-6302-42b0-9bac-a06875bbc4dd is 0 [FAILED] pod container-probe-4831/startup-81ef738d-6302-42b0-9bac-a06875bbc4dd - expected number of restarts: 1, found restarts: 0 In [It] at: test/e2e/common/node/container_probe.go:1000 @ 03/11/23 16:48:06.752 < Exit [It] should be restarted by liveness probe after startup probe enables it - test/e2e/common/node/container_probe.go:378 @ 03/11/23 16:48:06.752 (4m3.24s) > Enter [AfterEach] [sig-node] Probing container - test/e2e/framework/node/init/init.go:33 @ 03/11/23 16:48:06.752 Mar 11 16:48:06.752: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready Mar 11 16:48:06.901: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:48:06.901: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:48:09.050: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:48:09.050: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:48:11.050: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:48:11.050: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:48:13.050: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:48:13.050: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:48:15.051: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:48:15.051: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:48:17.050: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:48:17.050: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:48:19.050: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:48:19.050: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:48:21.050: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:48:21.050: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:48:23.050: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:48:23.050: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:48:25.051: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:48:25.051: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:48:27.051: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:48:27.051: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:48:29.050: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:48:29.050: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:48:31.050: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:48:31.050: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:48:33.050: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:48:33.050: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:48:35.051: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:48:35.051: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:48:37.051: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:48:37.051: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:48:39.051: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:48:39.051: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:48:41.050: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:48:41.050: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:48:43.050: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:48:43.050: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:48:45.050: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:48:45.050: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:48:47.051: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:48:47.051: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:48:49.051: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:48:49.051: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:48:51.051: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:48:51.051: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:48:53.050: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:48:53.050: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:48:55.051: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:48:55.051: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:48:57.050: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:48:57.050: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:48:59.050: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:48:59.050: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:49:01.050: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:49:01.050: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:49:03.051: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:49:03.051: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:49:05.050: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:49:05.050: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:49:07.050: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:49:07.050: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:49:09.050: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:49:09.050: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:49:11.050: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:49:11.050: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:49:13.050: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:49:13.050: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:49:15.051: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:49:15.051: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:49:17.050: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:49:17.050: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:49:19.050: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:49:19.050: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:49:21.051: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:49:21.051: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:49:23.050: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:49:23.050: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:49:25.051: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:49:25.051: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:49:27.050: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:49:27.050: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:49:29.050: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:49:29.050: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:49:31.050: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:49:31.050: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:49:33.050: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:49:33.050: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:49:35.050: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:49:35.050: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:49:37.050: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:49:37.050: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:49:39.050: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:49:39.050: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:49:41.050: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:49:41.050: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:49:43.050: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:49:43.050: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:49:45.050: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:49:45.050: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:49:47.050: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:49:47.050: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:49:49.051: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:49:49.051: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:49:51.050: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:49:51.050: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:49:53.050: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:49:53.050: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:49:55.051: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:49:55.051: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:49:57.051: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:49:57.051: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:49:59.051: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:49:59.051: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:50:01.051: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:50:01.051: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:50:03.050: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:50:03.050: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:50:05.050: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:50:05.050: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:50:07.051: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:50:07.051: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:50:09.050: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:50:09.050: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:50:11.051: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:50:11.051: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:50:13.050: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:50:13.050: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:50:15.051: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:50:15.051: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:50:17.051: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:50:17.051: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:50:19.050: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:50:19.050: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:50:21.051: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:50:21.051: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:50:23.050: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:50:23.050: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:50:25.051: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:50:25.051: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:50:27.050: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:50:27.050: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:50:29.051: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:50:29.051: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:50:31.050: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:50:31.050: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:50:33.050: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:50:33.050: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:50:35.050: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:50:35.050: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:50:37.050: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:50:37.050: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:50:39.050: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:50:39.050: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:50:41.050: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:50:41.051: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:50:43.050: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:50:43.050: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:50:45.050: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:50:45.050: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:50:47.055: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:50:47.055: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:50:49.050: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:50:49.050: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:50:51.050: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:50:51.050: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:50:53.050: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:50:53.050: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:50:55.051: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:50:55.051: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:50:57.050: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:50:57.050: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:50:59.050: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:50:59.050: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:51:01.050: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:51:01.050: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:51:03.050: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:51:03.050: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:51:05.051: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:51:05.051: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:51:07.050: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:51:07.051: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:51:07.199: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:51:07.199: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure < Exit [AfterEach] [sig-node] Probing container - test/e2e/framework/node/init/init.go:33 @ 03/11/23 16:51:07.199 (3m0.447s) > Enter [DeferCleanup (Each)] [sig-node] Probing container - test/e2e/common/node/container_probe.go:951 @ 03/11/23 16:51:07.199 STEP: deleting the pod - test/e2e/common/node/container_probe.go:952 @ 03/11/23 16:51:07.199 < Exit [DeferCleanup (Each)] [sig-node] Probing container - test/e2e/common/node/container_probe.go:951 @ 03/11/23 16:51:07.354 (155ms) > Enter [DeferCleanup (Each)] [sig-node] Probing container - test/e2e/framework/metrics/init/init.go:35 @ 03/11/23 16:51:07.354 < Exit [DeferCleanup (Each)] [sig-node] Probing container - test/e2e/framework/metrics/init/init.go:35 @ 03/11/23 16:51:07.354 (0s) > Enter [DeferCleanup (Each)] [sig-node] Probing container - dump namespaces | framework.go:209 @ 03/11/23 16:51:07.354 STEP: dump namespace information after failure - test/e2e/framework/framework.go:288 @ 03/11/23 16:51:07.354 STEP: Collecting events from namespace "container-probe-4831". - test/e2e/framework/debug/dump.go:42 @ 03/11/23 16:51:07.354 STEP: Found 8 events. - test/e2e/framework/debug/dump.go:46 @ 03/11/23 16:51:07.502 Mar 11 16:51:07.502: INFO: At 2023-03-11 16:44:03 +0000 UTC - event for startup-81ef738d-6302-42b0-9bac-a06875bbc4dd: {default-scheduler } Scheduled: Successfully assigned container-probe-4831/startup-81ef738d-6302-42b0-9bac-a06875bbc4dd to i-07e2b6e650d7ef56d Mar 11 16:51:07.502: INFO: At 2023-03-11 16:44:04 +0000 UTC - event for startup-81ef738d-6302-42b0-9bac-a06875bbc4dd: {kubelet i-07e2b6e650d7ef56d} Pulled: Container image "registry.k8s.io/e2e-test-images/busybox:1.29-4" already present on machine Mar 11 16:51:07.502: INFO: At 2023-03-11 16:44:04 +0000 UTC - event for startup-81ef738d-6302-42b0-9bac-a06875bbc4dd: {kubelet i-07e2b6e650d7ef56d} Created: Created container busybox Mar 11 16:51:07.502: INFO: At 2023-03-11 16:44:04 +0000 UTC - event for startup-81ef738d-6302-42b0-9bac-a06875bbc4dd: {kubelet i-07e2b6e650d7ef56d} Started: Started container busybox Mar 11 16:51:07.502: INFO: At 2023-03-11 16:44:34 +0000 UTC - event for startup-81ef738d-6302-42b0-9bac-a06875bbc4dd: {kubelet i-07e2b6e650d7ef56d} Unhealthy: Liveness probe failed: Mar 11 16:51:07.502: INFO: At 2023-03-11 16:44:34 +0000 UTC - event for startup-81ef738d-6302-42b0-9bac-a06875bbc4dd: {kubelet i-07e2b6e650d7ef56d} Killing: Container busybox failed liveness probe, will be restarted Mar 11 16:51:07.502: INFO: At 2023-03-11 16:45:36 +0000 UTC - event for startup-81ef738d-6302-42b0-9bac-a06875bbc4dd: {node-controller } NodeNotReady: Node is not ready Mar 11 16:51:07.502: INFO: At 2023-03-11 16:50:46 +0000 UTC - event for startup-81ef738d-6302-42b0-9bac-a06875bbc4dd: {taint-controller } TaintManagerEviction: Marking for deletion Pod container-probe-4831/startup-81ef738d-6302-42b0-9bac-a06875bbc4dd Mar 11 16:51:07.650: INFO: POD NODE PHASE GRACE CONDITIONS Mar 11 16:51:07.650: INFO: Mar 11 16:51:07.798: INFO: Logging node info for node i-04523d1dffb0fe725 Mar 11 16:51:07.946: INFO: Node Info: &Node{ObjectMeta:{i-04523d1dffb0fe725 5d37640f-a7d9-4c26-86c5-9da21f7ccc18 49426 0 2023-03-11 16:24:20 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:arm64 beta.kubernetes.io/instance-type:t4g.small beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-northeast-1 failure-domain.beta.kubernetes.io/zone:ap-northeast-1a karpenter.k8s.aws/instance-category:t karpenter.k8s.aws/instance-cpu:2 karpenter.k8s.aws/instance-family:t4g karpenter.k8s.aws/instance-generation:4 karpenter.k8s.aws/instance-hypervisor:nitro karpenter.k8s.aws/instance-memory:2048 karpenter.k8s.aws/instance-pods:11 karpenter.k8s.aws/instance-size:small karpenter.sh/capacity-type:spot karpenter.sh/initialized:true karpenter.sh/provisioner-name:nodes kubernetes.io/arch:arm64 kubernetes.io/hostname:i-04523d1dffb0fe725 kubernetes.io/os:linux node-role.kubernetes.io/node: node-role.kubernetes.io/spot-worker:true node.kubernetes.io/instance-type:t4g.small topology.ebs.csi.aws.com/zone:ap-northeast-1a topology.hostpath.csi/node:i-04523d1dffb0fe725 topology.kubernetes.io/region:ap-northeast-1 topology.kubernetes.io/zone:ap-northeast-1a] map[csi.volume.kubernetes.io/nodeid:{"csi-hostpath-read-write-once-pod-7172":"i-04523d1dffb0fe725","csi-hostpath-volume-expand-1673":"i-04523d1dffb0fe725","ebs.csi.aws.com":"i-04523d1dffb0fe725"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [{karpenter.sh/v1alpha5 Provisioner nodes ac7283b2-eed0-4a9f-9ddd-616f33a0991a <nil> 0xc003a0a9e5}] [karpenter.sh/termination] [{aws-cloud-controller-manager Update v1 2023-03-11 16:24:20 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{}}}} } {kops-controller Update v1 2023-03-11 16:24:20 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{},"f:node-role.kubernetes.io/spot-worker":{}}}} } {karpenter Update v1 2023-03-11 16:26:12 +0000 UTC FieldsV1 {"f:metadata":{"f:finalizers":{".":{},"v:\"karpenter.sh/termination\"":{}},"f:labels":{".":{},"f:karpenter.k8s.aws/instance-category":{},"f:karpenter.k8s.aws/instance-cpu":{},"f:karpenter.k8s.aws/instance-family":{},"f:karpenter.k8s.aws/instance-generation":{},"f:karpenter.k8s.aws/instance-hypervisor":{},"f:karpenter.k8s.aws/instance-memory":{},"f:karpenter.k8s.aws/instance-pods":{},"f:karpenter.k8s.aws/instance-size":{},"f:karpenter.sh/capacity-type":{},"f:karpenter.sh/initialized":{},"f:karpenter.sh/provisioner-name":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}},"f:ownerReferences":{".":{},"k:{\"uid\":\"ac7283b2-eed0-4a9f-9ddd-616f33a0991a\"}":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-03-11 16:26:50 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{".":{},"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"InternalIP\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kubelet Update v1 2023-03-11 16:44:47 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{"f:kubernetes.io/hostname":{},"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{".":{},"f:cpu":{},"f:ephemeral-storage":{},"f:hugepages-1Gi":{},"f:hugepages-2Mi":{},"f:hugepages-32Mi":{},"f:hugepages-64Ki":{},"f:memory":{},"f:pods":{}},"f:capacity":{".":{},"f:cpu":{},"f:ephemeral-storage":{},"f:hugepages-1Gi":{},"f:hugepages-2Mi":{},"f:hugepages-32Mi":{},"f:hugepages-64Ki":{},"f:memory":{},"f:pods":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{}}},"f:daemonEndpoints":{"f:kubeletEndpoint":{"f:Port":{}}},"f:images":{},"f:nodeInfo":{"f:architecture":{},"f:bootID":{},"f:containerRuntimeVersion":{},"f:kernelVersion":{},"f:kubeProxyVersion":{},"f:kubeletVersion":{},"f:machineID":{},"f:operatingSystem":{},"f:osImage":{},"f:systemUUID":{}},"f:volumesInUse":{}}} status} {kube-controller-manager Update v1 2023-03-11 16:45:36 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:node.alpha.kubernetes.io/ttl":{}},"f:labels":{"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.2.0/24\"":{}},"f:taints":{}}} } {kube-controller-manager Update v1 2023-03-11 16:50:51 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{".":{},"k:{\"type\":\"DiskPressure\"}":{".":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"MemoryPressure\"}":{".":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"PIDPressure\"}":{".":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Ready\"}":{".":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}},"f:volumesAttached":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.2.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-northeast-1a/i-04523d1dffb0fe725,Unschedulable:false,Taints:[]Taint{Taint{Key:node.kubernetes.io/unreachable,Value:,Effect:NoSchedule,TimeAdded:2023-03-11 16:45:31 +0000 UTC,},Taint{Key:node.kubernetes.io/unreachable,Value:,Effect:NoExecute,TimeAdded:2023-03-11 16:45:36 +0000 UTC,},},ConfigSource:nil,PodCIDRs:[100.96.2.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49756430336 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-32Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-64Ki: {{0 0} {<nil>} 0 DecimalSI},memory: {{1941540864 0} {<nil>} 1896036Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44780787229 0} {<nil>} 44780787229 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-32Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-64Ki: {{0 0} {<nil>} 0 DecimalSI},memory: {{1836683264 0} {<nil>} 1793636Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:Ready,Status:Unknown,LastHeartbeatTime:2023-03-11 16:44:47 +0000 UTC,LastTransitionTime:2023-03-11 16:45:31 +0000 UTC,Reason:NodeStatusUnknown,Message:Kubelet stopped posting node status.,},NodeCondition{Type:MemoryPressure,Status:Unknown,LastHeartbeatTime:2023-03-11 16:44:47 +0000 UTC,LastTransitionTime:2023-03-11 16:45:31 +0000 UTC,Reason:NodeStatusUnknown,Message:Kubelet stopped posting node status.,},NodeCondition{Type:DiskPressure,Status:Unknown,LastHeartbeatTime:2023-03-11 16:44:47 +0000 UTC,LastTransitionTime:2023-03-11 16:45:31 +0000 UTC,Reason:NodeStatusUnknown,Message:Kubelet stopped posting node status.,},NodeCondition{Type:PIDPressure,Status:Unknown,LastHeartbeatTime:2023-03-11 16:44:47 +0000 UTC,LastTransitionTime:2023-03-11 16:45:31 +0000 UTC,Reason:NodeStatusUnknown,Message:Kubelet stopped posting node status.,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.48.168,},NodeAddress{Type:ExternalIP,Address:52.199.253.156,},NodeAddress{Type:InternalDNS,Address:i-04523d1dffb0fe725.ap-northeast-1.compute.internal,},NodeAddress{Type:Hostname,Address:i-04523d1dffb0fe725.ap-northeast-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-52-199-253-156.ap-northeast-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2c8fb0e6b2e9c4af310f65b6161cd8,SystemUUID:ec2c8fb0-e6b2-e9c4-af31-0f65b6161cd8,BootID:a3b8de83-558b-44c4-a46a-d097c3f98225,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 22.04.2 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.27.0-alpha.3.420+bb6c6fad2c19b3,KubeProxyVersion:v1.27.0-alpha.3.420+bb6c6fad2c19b3,OperatingSystem:linux,Architecture:arm64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:157636062,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/jessie-dnsutils@sha256:24aaf2626d6b27864c29de2097e8bbb840b3a414271bf7c8995e431e47d8408e registry.k8s.io/e2e-test-images/jessie-dnsutils:1.7],SizeBytes:116764683,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:51eae8381dcb1078289fa7b4f3df2630cdc18d09fb56f8e56b41c40e191d6c83 registry.k8s.io/etcd:3.5.7-0],SizeBytes:80665728,},ContainerImage{Names:[registry.k8s.io/kube-proxy-arm64:v1.27.0-alpha.3.420_bb6c6fad2c19b3],SizeBytes:60947298,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:16bbf38c463a4223d8cfe4da12bc61010b082a79b4bb003e2d3ba3ece5dd5f9e registry.k8s.io/e2e-test-images/agnhost:2.43],SizeBytes:49808364,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nautilus@sha256:80ba6c8c44f9623f06e868a1aa66026c8ec438ad814f9ec95e9333b415fe3550 registry.k8s.io/e2e-test-images/nautilus:1.7],SizeBytes:47953400,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:3fe7acf013d1264ffded116b80a73dc129a449b0fccdb8d21af8279f2233f36e registry.k8s.io/e2e-test-images/httpd:2.4.39-4],SizeBytes:41528818,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:148b022f5c5da426fc2f3c14b5c0867e58ef05961510c84749ac1fddcb0fef22 registry.k8s.io/e2e-test-images/httpd:2.4.38-4],SizeBytes:40343601,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:28932904,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:ee3b525d5b89db99da3b8eb521d9cd90cb6e9ef0fbb651e98bb37be78d36b5b8 registry.k8s.io/sig-storage/csi-provisioner:v3.3.0],SizeBytes:23945474,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/sample-apiserver@sha256:8d70890151aa5d096f331cb9da1b9cd5be0412b7363fe67b5c3befdcaa2a28d0 registry.k8s.io/e2e-test-images/sample-apiserver:1.17.7],SizeBytes:23857462,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:425d8f1b769398127767b06ed97ce62578a3179bcb99809ce93a1649e025ffe7 registry.k8s.io/sig-storage/csi-resizer:v1.6.0],SizeBytes:22678632,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:291334908ddf71a4661fd7f6d9d97274de8a5378a2b6fdfeb2ce73414a34f82f registry.k8s.io/sig-storage/csi-snapshotter:v6.1.0],SizeBytes:22434180,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:9a685020911e2725ad019dbce6e4a5ab93d51e3d4557f115e64343345e05781b registry.k8s.io/sig-storage/csi-attacher:v4.0.0],SizeBytes:22394501,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:122bfb8c1edabb3c0edd63f06523e6940d958d19b3957dc7b1d6f81e9f1f6119 registry.k8s.io/sig-storage/csi-provisioner:v3.1.0],SizeBytes:21735644,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:9ebbf9f023e7b41ccee3d52afe39a89e3ddacdbb69269d583abfc25847cfd9e4 registry.k8s.io/sig-storage/csi-resizer:v1.4.0],SizeBytes:20848413,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:8b9c313c05f54fb04f8d430896f5f5904b6cb157df261501b29adc04d2b2dc7b registry.k8s.io/sig-storage/csi-attacher:v3.4.0],SizeBytes:20565480,},ContainerImage{Names:[registry.k8s.io/cpa/cluster-proportional-autoscaler@sha256:68d396900aeaa072c1f27289485fdac29834045a6f3ffe369bf389d830ef572d registry.k8s.io/cpa/cluster-proportional-autoscaler:1.8.6],SizeBytes:19007623,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nonroot@sha256:ee9f50b3c64b174d296d91ca9f69a914ac30e59095dfb462b2b518ad28a63655 registry.k8s.io/e2e-test-images/nonroot:1.4],SizeBytes:18561972,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:92257881c1d6493cf18299a24af42330f891166560047902b8d431fb66b01af5 registry.k8s.io/sig-storage/hostpathplugin:v1.9.0],SizeBytes:18267611,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:9b2d9abda017c28b12f84a344f57cd73fbdb6c2bd7dd5bdd5018246ad1093ba6 registry.k8s.io/sig-storage/hostpathplugin:v1.11.0],SizeBytes:18159085,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:017727efcfeb7d053af68e51436ce8e65edbc6ca573720afb4f79c8594036955 registry.k8s.io/coredns/coredns:v1.10.0],SizeBytes:13766234,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:8502345,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:933940f13b3ea0abc62e656c1aa5c5b47c04b15d71250413a6b821bd0c58b94e registry.k8s.io/sig-storage/livenessprobe:v2.7.0],SizeBytes:8223819,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:7685305,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:db048754ae68ae337d8fa96494c96d2a1204c3320f5dcf7e8e71085adec85da6 registry.k8s.io/e2e-test-images/nginx:1.15-4],SizeBytes:6883347,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:5c99cf6a02adda929b10321dbf4ecfa00d87be9ba4fb456006237d530ab4baa1 registry.k8s.io/e2e-test-images/nginx:1.14-4],SizeBytes:6860089,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nonewprivs@sha256:8ac1264691820febacf3aea5d152cbde6d10685731ec14966a9401c6f47a68ac registry.k8s.io/e2e-test-images/nonewprivs:1.3],SizeBytes:5025027,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf registry.k8s.io/e2e-test-images/busybox:1.29-2],SizeBytes:773367,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:2e0f836850e09b8b7cc937681d6194537a09fbd5f6b9e08f4d646a85128e8937 registry.k8s.io/e2e-test-images/busybox:1.29-4],SizeBytes:772933,},ContainerImage{Names:[registry.k8s.io/pause@sha256:7031c1b283388d2c2e09b57badb803c05ebed362dc88d84b480cc47f72a21097 registry.k8s.io/pause:3.9],SizeBytes:268051,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:253553,},},VolumesInUse:[kubernetes.io/csi/csi-hostpath-read-write-once-pod-7172^038fde52-c02c-11ed-b028-5eca9e37a97f kubernetes.io/csi/csi-hostpath-volume-expand-1673^f8193b68-c02b-11ed-8784-9e8522e7cc37 kubernetes.io/csi/ebs.csi.aws.com^vol-0293af09ce81fe75e],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/csi-hostpath-read-write-once-pod-7172^038fde52-c02c-11ed-b028-5eca9e37a97f,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/csi-hostpath-volume-expand-1673^f8193b68-c02b-11ed-8784-9e8522e7cc37,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-0293af09ce81fe75e,DevicePath:,},},Config:nil,},} Mar 11 16:51:07.947: INFO: Logging kubelet events for node i-04523d1dffb0fe725 Mar 11 16:51:08.098: INFO: Logging pods the kubelet thinks is on node i-04523d1dffb0fe725 Mar 11 16:51:18.247: INFO: Unable to retrieve kubelet pods for node i-04523d1dffb0fe725: error trying to reach service: net/http: TLS handshake timeout Mar 11 16:51:18.247: INFO: Logging node info for node i-07e2b6e650d7ef56d Mar 11 16:51:18.395: INFO: Node Info: &Node{ObjectMeta:{i-07e2b6e650d7ef56d fcc65272-6f1b-4228-ac59-e9e3e92cdb2c 47965 0 2023-03-11 16:24:19 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:arm64 beta.kubernetes.io/instance-type:t4g.small beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-northeast-1 failure-domain.beta.kubernetes.io/zone:ap-northeast-1a io.kubernetes.storage.mock/node:some-mock-node karpenter.k8s.aws/instance-category:t karpenter.k8s.aws/instance-cpu:2 karpenter.k8s.aws/instance-family:t4g karpenter.k8s.aws/instance-generation:4 karpenter.k8s.aws/instance-hypervisor:nitro karpenter.k8s.aws/instance-memory:2048 karpenter.k8s.aws/instance-pods:11 karpenter.k8s.aws/instance-size:small karpenter.sh/capacity-type:spot karpenter.sh/initialized:true karpenter.sh/provisioner-name:nodes kubernetes.io/arch:arm64 kubernetes.io/hostname:i-07e2b6e650d7ef56d kubernetes.io/os:linux node-role.kubernetes.io/node: node-role.kubernetes.io/spot-worker:true node.kubernetes.io/instance-type:t4g.small topology.ebs.csi.aws.com/zone:ap-northeast-1a topology.hostpath.csi/node:i-07e2b6e650d7ef56d topology.kubernetes.io/region:ap-northeast-1 topology.kubernetes.io/zone:ap-northeast-1a] map[csi.volume.kubernetes.io/nodeid:{"csi-hostpath-ephemeral-7952":"i-07e2b6e650d7ef56d","csi-hostpath-volume-expand-703":"i-07e2b6e650d7ef56d","ebs.csi.aws.com":"i-07e2b6e650d7ef56d"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [{karpenter.sh/v1alpha5 Provisioner nodes ac7283b2-eed0-4a9f-9ddd-616f33a0991a <nil> 0xc003acce45}] [karpenter.sh/termination] [{aws-cloud-controller-manager Update v1 2023-03-11 16:24:20 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{}}}} } {kops-controller Update v1 2023-03-11 16:24:20 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{},"f:node-role.kubernetes.io/spot-worker":{}}}} } {karpenter Update v1 2023-03-11 16:26:15 +0000 UTC FieldsV1 {"f:metadata":{"f:finalizers":{".":{},"v:\"karpenter.sh/termination\"":{}},"f:labels":{".":{},"f:karpenter.k8s.aws/instance-category":{},"f:karpenter.k8s.aws/instance-cpu":{},"f:karpenter.k8s.aws/instance-family":{},"f:karpenter.k8s.aws/instance-generation":{},"f:karpenter.k8s.aws/instance-hypervisor":{},"f:karpenter.k8s.aws/instance-memory":{},"f:karpenter.k8s.aws/instance-pods":{},"f:karpenter.k8s.aws/instance-size":{},"f:karpenter.sh/capacity-type":{},"f:karpenter.sh/initialized":{},"f:karpenter.sh/provisioner-name":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}},"f:ownerReferences":{".":{},"k:{\"uid\":\"ac7283b2-eed0-4a9f-9ddd-616f33a0991a\"}":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-03-11 16:26:50 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{".":{},"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"InternalIP\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kubelet Update v1 2023-03-11 16:44:22 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{"f:io.kubernetes.storage.mock/node":{},"f:kubernetes.io/hostname":{},"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{".":{},"f:cpu":{},"f:ephemeral-storage":{},"f:hugepages-1Gi":{},"f:hugepages-2Mi":{},"f:hugepages-32Mi":{},"f:hugepages-64Ki":{},"f:memory":{},"f:pods":{}},"f:capacity":{".":{},"f:cpu":{},"f:ephemeral-storage":{},"f:hugepages-1Gi":{},"f:hugepages-2Mi":{},"f:hugepages-32Mi":{},"f:hugepages-64Ki":{},"f:memory":{},"f:pods":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{}}},"f:daemonEndpoints":{"f:kubeletEndpoint":{"f:Port":{}}},"f:images":{},"f:nodeInfo":{"f:architecture":{},"f:bootID":{},"f:containerRuntimeVersion":{},"f:kernelVersion":{},"f:kubeProxyVersion":{},"f:kubeletVersion":{},"f:machineID":{},"f:operatingSystem":{},"f:osImage":{},"f:systemUUID":{}},"f:volumesInUse":{}}} status} {kube-controller-manager Update v1 2023-03-11 16:45:36 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{".":{},"k:{\"type\":\"DiskPressure\"}":{".":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"MemoryPressure\"}":{".":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"PIDPressure\"}":{".":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Ready\"}":{".":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}},"f:volumesAttached":{}}} status} {kube-controller-manager Update v1 2023-03-11 16:45:46 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:node.alpha.kubernetes.io/ttl":{}},"f:labels":{"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.1.0/24\"":{}},"f:taints":{}}} }]},Spec:NodeSpec{PodCIDR:100.96.1.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-northeast-1a/i-07e2b6e650d7ef56d,Unschedulable:false,Taints:[]Taint{Taint{Key:node.kubernetes.io/unreachable,Value:,Effect:NoSchedule,TimeAdded:2023-03-11 16:45:36 +0000 UTC,},Taint{Key:node.kubernetes.io/unreachable,Value:,Effect:NoExecute,TimeAdded:2023-03-11 16:45:46 +0000 UTC,},},ConfigSource:nil,PodCIDRs:[100.96.1.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49756430336 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-32Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-64Ki: {{0 0} {<nil>} 0 DecimalSI},memory: {{1941549056 0} {<nil>} 1896044Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44780787229 0} {<nil>} 44780787229 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-32Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-64Ki: {{0 0} {<nil>} 0 DecimalSI},memory: {{1836691456 0} {<nil>} 1793644Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:Ready,Status:Unknown,LastHeartbeatTime:2023-03-11 16:44:12 +0000 UTC,LastTransitionTime:2023-03-11 16:45:36 +0000 UTC,Reason:NodeStatusUnknown,Message:Kubelet stopped posting node status.,},NodeCondition{Type:MemoryPressure,Status:Unknown,LastHeartbeatTime:2023-03-11 16:44:12 +0000 UTC,LastTransitionTime:2023-03-11 16:45:36 +0000 UTC,Reason:NodeStatusUnknown,Message:Kubelet stopped posting node status.,},NodeCondition{Type:DiskPressure,Status:Unknown,LastHeartbeatTime:2023-03-11 16:44:12 +0000 UTC,LastTransitionTime:2023-03-11 16:45:36 +0000 UTC,Reason:NodeStatusUnknown,Message:Kubelet stopped posting node status.,},NodeCondition{Type:PIDPressure,Status:Unknown,LastHeartbeatTime:2023-03-11 16:44:12 +0000 UTC,LastTransitionTime:2023-03-11 16:45:36 +0000 UTC,Reason:NodeStatusUnknown,Message:Kubelet stopped posting node status.,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.35.76,},NodeAddress{Type:ExternalIP,Address:54.238.106.83,},NodeAddress{Type:InternalDNS,Address:i-07e2b6e650d7ef56d.ap-northeast-1.compute.internal,},NodeAddress{Type:Hostname,Address:i-07e2b6e650d7ef56d.ap-northeast-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-54-238-106-83.ap-northeast-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2083a8e9d27d02fac967340367f875,SystemUUID:ec2083a8-e9d2-7d02-fac9-67340367f875,BootID:ea264ed6-5b0d-47d6-ad5d-6bd2a1f7c11b,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 22.04.2 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.27.0-alpha.3.420+bb6c6fad2c19b3,KubeProxyVersion:v1.27.0-alpha.3.420+bb6c6fad2c19b3,OperatingSystem:linux,Architecture:arm64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:157636062,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/jessie-dnsutils@sha256:24aaf2626d6b27864c29de2097e8bbb840b3a414271bf7c8995e431e47d8408e registry.k8s.io/e2e-test-images/jessie-dnsutils:1.7],SizeBytes:116764683,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:51eae8381dcb1078289fa7b4f3df2630cdc18d09fb56f8e56b41c40e191d6c83 registry.k8s.io/etcd:3.5.7-0],SizeBytes:80665728,},ContainerImage{Names:[registry.k8s.io/kube-proxy-arm64:v1.27.0-alpha.3.420_bb6c6fad2c19b3],SizeBytes:60947298,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:16bbf38c463a4223d8cfe4da12bc61010b082a79b4bb003e2d3ba3ece5dd5f9e registry.k8s.io/e2e-test-images/agnhost:2.43],SizeBytes:49808364,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nautilus@sha256:80ba6c8c44f9623f06e868a1aa66026c8ec438ad814f9ec95e9333b415fe3550 registry.k8s.io/e2e-test-images/nautilus:1.7],SizeBytes:47953400,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:3fe7acf013d1264ffded116b80a73dc129a449b0fccdb8d21af8279f2233f36e registry.k8s.io/e2e-test-images/httpd:2.4.39-4],SizeBytes:41528818,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:148b022f5c5da426fc2f3c14b5c0867e58ef05961510c84749ac1fddcb0fef22 registry.k8s.io/e2e-test-images/httpd:2.4.38-4],SizeBytes:40343601,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:28932904,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:ee3b525d5b89db99da3b8eb521d9cd90cb6e9ef0fbb651e98bb37be78d36b5b8 registry.k8s.io/sig-storage/csi-provisioner:v3.3.0],SizeBytes:23945474,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/sample-apiserver@sha256:8d70890151aa5d096f331cb9da1b9cd5be0412b7363fe67b5c3befdcaa2a28d0 registry.k8s.io/e2e-test-images/sample-apiserver:1.17.7],SizeBytes:23857462,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:425d8f1b769398127767b06ed97ce62578a3179bcb99809ce93a1649e025ffe7 registry.k8s.io/sig-storage/csi-resizer:v1.6.0],SizeBytes:22678632,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:291334908ddf71a4661fd7f6d9d97274de8a5378a2b6fdfeb2ce73414a34f82f registry.k8s.io/sig-storage/csi-snapshotter:v6.1.0],SizeBytes:22434180,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:9a685020911e2725ad019dbce6e4a5ab93d51e3d4557f115e64343345e05781b registry.k8s.io/sig-storage/csi-attacher:v4.0.0],SizeBytes:22394501,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:122bfb8c1edabb3c0edd63f06523e6940d958d19b3957dc7b1d6f81e9f1f6119 registry.k8s.io/sig-storage/csi-provisioner:v3.1.0],SizeBytes:21735644,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/regression-issue-74839@sha256:b4f1d8d61bdad84bd50442d161d5460e4019d53e989b64220fdbc62fc87d76bf registry.k8s.io/e2e-test-images/regression-issue-74839:1.2],SizeBytes:21331383,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:9ebbf9f023e7b41ccee3d52afe39a89e3ddacdbb69269d583abfc25847cfd9e4 registry.k8s.io/sig-storage/csi-resizer:v1.4.0],SizeBytes:20848413,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:8b9c313c05f54fb04f8d430896f5f5904b6cb157df261501b29adc04d2b2dc7b registry.k8s.io/sig-storage/csi-attacher:v3.4.0],SizeBytes:20565480,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nonroot@sha256:ee9f50b3c64b174d296d91ca9f69a914ac30e59095dfb462b2b518ad28a63655 registry.k8s.io/e2e-test-images/nonroot:1.4],SizeBytes:18561972,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:92257881c1d6493cf18299a24af42330f891166560047902b8d431fb66b01af5 registry.k8s.io/sig-storage/hostpathplugin:v1.9.0],SizeBytes:18267611,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:9b2d9abda017c28b12f84a344f57cd73fbdb6c2bd7dd5bdd5018246ad1093ba6 registry.k8s.io/sig-storage/hostpathplugin:v1.11.0],SizeBytes:18159085,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:017727efcfeb7d053af68e51436ce8e65edbc6ca573720afb4f79c8594036955 registry.k8s.io/coredns/coredns:v1.10.0],SizeBytes:13766234,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:8502345,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:933940f13b3ea0abc62e656c1aa5c5b47c04b15d71250413a6b821bd0c58b94e registry.k8s.io/sig-storage/livenessprobe:v2.7.0],SizeBytes:8223819,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:7685305,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:db048754ae68ae337d8fa96494c96d2a1204c3320f5dcf7e8e71085adec85da6 registry.k8s.io/e2e-test-images/nginx:1.15-4],SizeBytes:6883347,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:5c99cf6a02adda929b10321dbf4ecfa00d87be9ba4fb456006237d530ab4baa1 registry.k8s.io/e2e-test-images/nginx:1.14-4],SizeBytes:6860089,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nonewprivs@sha256:8ac1264691820febacf3aea5d152cbde6d10685731ec14966a9401c6f47a68ac registry.k8s.io/e2e-test-images/nonewprivs:1.3],SizeBytes:5025027,},ContainerImage{Names:[gcr.io/authenticated-image-pulling/alpine@sha256:7ff177862cb50c602bfe81f805969412e619c054a2bbead977d0c276988aa4a0 gcr.io/authenticated-image-pulling/alpine:3.7],SizeBytes:2003505,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf registry.k8s.io/e2e-test-images/busybox:1.29-2],SizeBytes:773367,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:2e0f836850e09b8b7cc937681d6194537a09fbd5f6b9e08f4d646a85128e8937 registry.k8s.io/e2e-test-images/busybox:1.29-4],SizeBytes:772933,},ContainerImage{Names:[registry.k8s.io/pause@sha256:7031c1b283388d2c2e09b57badb803c05ebed362dc88d84b480cc47f72a21097 registry.k8s.io/pause:3.9],SizeBytes:268051,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:253553,},},VolumesInUse:[kubernetes.io/csi/csi-hostpath-ephemeral-7952^eca2344a-c02b-11ed-b483-1e4a85f27a89 kubernetes.io/csi/csi-hostpath-ephemeral-7952^f0ac2401-c02b-11ed-b483-1e4a85f27a89 kubernetes.io/csi/ebs.csi.aws.com^vol-06cba8db23f1489ad],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-06cba8db23f1489ad,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/csi-hostpath-ephemeral-7952^f0ac2401-c02b-11ed-b483-1e4a85f27a89,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/csi-hostpath-ephemeral-7952^eca2344a-c02b-11ed-b483-1e4a85f27a89,DevicePath:,},},Config:nil,},} Mar 11 16:51:18.396: INFO: Logging kubelet events for node i-07e2b6e650d7ef56d Mar 11 16:51:18.547: INFO: Logging pods the kubelet thinks is on node i-07e2b6e650d7ef56d Mar 11 16:51:28.696: INFO: Unable to retrieve kubelet pods for node i-07e2b6e650d7ef56d: error trying to reach service: net/http: TLS handshake timeout Mar 11 16:51:28.696: INFO: Logging node info for node i-0e3467ec552af797e Mar 11 16:51:28.845: INFO: Node Info: &Node{ObjectMeta:{i-0e3467ec552af797e 6c9b0cc5-2c4e-4037-bf31-61bac12a7775 48787 0 2023-03-11 16:20:46 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:arm64 beta.kubernetes.io/instance-type:c6g.xlarge beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-northeast-1 failure-domain.beta.kubernetes.io/zone:ap-northeast-1a kops.k8s.io/kops-controller-pki: kubernetes.io/arch:arm64 kubernetes.io/hostname:i-0e3467ec552af797e kubernetes.io/os:linux node-role.kubernetes.io/control-plane: node.kubernetes.io/exclude-from-external-load-balancers: node.kubernetes.io/instance-type:c6g.xlarge topology.ebs.csi.aws.com/zone:ap-northeast-1a topology.kubernetes.io/region:ap-northeast-1 topology.kubernetes.io/zone:ap-northeast-1a] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-0e3467ec552af797e"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kubelet Update v1 2023-03-11 16:20:46 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {protokube Update v1 2023-03-11 16:21:30 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kops.k8s.io/kops-controller-pki":{},"f:node-role.kubernetes.io/control-plane":{},"f:node.kubernetes.io/exclude-from-external-load-balancers":{}}}} } {aws-cloud-controller-manager Update v1 2023-03-11 16:21:50 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-03-11 16:21:50 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kube-controller-manager Update v1 2023-03-11 16:21:58 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.0.0/24\"":{}},"f:taints":{}}} } {kubelet Update v1 2023-03-11 16:48:50 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.0.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-northeast-1a/i-0e3467ec552af797e,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/control-plane,Value:,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[100.96.0.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{4 0} {<nil>} 4 DecimalSI},ephemeral-storage: {{49756430336 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-32Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-64Ki: {{0 0} {<nil>} 0 DecimalSI},memory: {{8150110208 0} {<nil>} 7959092Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{4 0} {<nil>} 4 DecimalSI},ephemeral-storage: {{44780787229 0} {<nil>} 44780787229 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-32Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-64Ki: {{0 0} {<nil>} 0 DecimalSI},memory: {{8045252608 0} {<nil>} 7856692Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-03-11 16:48:50 +0000 UTC,LastTransitionTime:2023-03-11 16:20:46 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-03-11 16:48:50 +0000 UTC,LastTransitionTime:2023-03-11 16:20:46 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-03-11 16:48:50 +0000 UTC,LastTransitionTime:2023-03-11 16:20:46 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-03-11 16:48:50 +0000 UTC,LastTransitionTime:2023-03-11 16:21:58 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.40.18,},NodeAddress{Type:ExternalIP,Address:52.194.213.7,},NodeAddress{Type:InternalDNS,Address:i-0e3467ec552af797e.ap-northeast-1.compute.internal,},NodeAddress{Type:Hostname,Address:i-0e3467ec552af797e.ap-northeast-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-52-194-213-7.ap-northeast-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2abf2e564965592c7716accd56507e,SystemUUID:ec2abf2e-5649-6559-2c77-16accd56507e,BootID:fbada3ff-8655-404c-a98d-876577d4c322,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 22.04.2 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.27.0-alpha.3.420+bb6c6fad2c19b3,KubeProxyVersion:v1.27.0-alpha.3.420+bb6c6fad2c19b3,OperatingSystem:linux,Architecture:arm64,},Images:[]ContainerImage{ContainerImage{Names:[registry.k8s.io/etcd@sha256:9ce33ba33d8e738a5b85ed50b5080ac746deceed4a7496c550927a7a19ca3b6d registry.k8s.io/etcd:3.5.0-0],SizeBytes:157797353,},ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:157636062,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:4afb99b4690b418ffc2ceb67e1a17376457e441c1f09ab55447f0aaf992fa646 registry.k8s.io/etcd:3.4.3-0],SizeBytes:151677080,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:4ad90a11b55313b182afc186b9876c8e891531b8db4c9bf1541953021618d0e2 registry.k8s.io/etcd:3.4.13-0],SizeBytes:134531559,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:0d2ab6b15b4fe0b903b9e81192d942fb39c29be61fe341cd16db5d0165b0435c registry.k8s.io/etcd:3.3.17-0],SizeBytes:126979107,},ContainerImage{Names:[registry.k8s.io/kube-apiserver-arm64:v1.27.0-alpha.3.420_bb6c6fad2c19b3],SizeBytes:115123555,},ContainerImage{Names:[registry.k8s.io/kube-controller-manager-arm64:v1.27.0-alpha.3.420_bb6c6fad2c19b3],SizeBytes:107914235,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:17da501f5d2a675be46040422a27b7cc21b8a43895ac998b171db1c346f361f7 registry.k8s.io/etcd:3.3.10-0],SizeBytes:81436097,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:6f72b851544986cb0921b53ea655ec04c36131248f16d4ad110cb3ca0c369dc1 registry.k8s.io/etcd:3.5.4-0],SizeBytes:81120118,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:13f53ed1d91e2e11aac476ee9a0269fdda6cc4874eba903efd40daf50c55eee5 registry.k8s.io/etcd:3.5.3-0],SizeBytes:81114891,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:51eae8381dcb1078289fa7b4f3df2630cdc18d09fb56f8e56b41c40e191d6c83 registry.k8s.io/etcd:3.5.7-0],SizeBytes:80665728,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:dd75ec974b0a2a6f6bb47001ba09207976e625db898d1b16735528c009cb171c registry.k8s.io/etcd:3.5.6-0],SizeBytes:80539316,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:905d7ca17fd02bc24c0eba9a062753aba15db3e31422390bc3238eb762339b20 registry.k8s.io/etcd:3.2.24-1],SizeBytes:61500433,},ContainerImage{Names:[registry.k8s.io/kube-proxy-arm64:v1.27.0-alpha.3.420_bb6c6fad2c19b3],SizeBytes:60947298,},ContainerImage{Names:[registry.k8s.io/etcdadm/etcd-manager-slim@sha256:8727adc059334e0e3329dbccce116a734da4a090da5e86c8cddccaf333caa666 registry.k8s.io/etcdadm/etcd-manager-slim:v3.0.20230201],SizeBytes:60826655,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:64b9ea357325d5db9f8a723dcf503b5a449177b17ac87d69481e126bb724c263 registry.k8s.io/etcd:3.5.1-0],SizeBytes:59553904,},ContainerImage{Names:[registry.k8s.io/kube-scheduler-arm64:v1.27.0-alpha.3.420_bb6c6fad2c19b3],SizeBytes:56731120,},ContainerImage{Names:[registry.k8s.io/kops/kops-controller:1.27.0-alpha.1],SizeBytes:41885191,},ContainerImage{Names:[registry.k8s.io/kops/dns-controller:1.27.0-alpha.1],SizeBytes:40376038,},ContainerImage{Names:[public.ecr.aws/karpenter/controller@sha256:68db4f092cf9cc83f5ef9e2fbc5407c2cb682e81f64dfaa700a7602ede38b1cf public.ecr.aws/karpenter/controller:v0.16.3],SizeBytes:33707751,},ContainerImage{Names:[public.ecr.aws/karpenter/webhook@sha256:96a2d9b06d6bc5127801f358f74b1cf2d289b423a2e9ba40c573c0b14b17dafa public.ecr.aws/karpenter/webhook:v0.16.3],SizeBytes:33307187,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:28932904,},ContainerImage{Names:[quay.io/cilium/operator@sha256:a6d24a006a6b92967ac90786b49bc1ac26e5477cf028cd1186efcfc2466484db quay.io/cilium/operator:v1.12.5],SizeBytes:24567287,},ContainerImage{Names:[registry.k8s.io/provider-aws/cloud-controller-manager@sha256:fdeb61e3e42ecd9cca868d550ebdb88dd6341d9e91fcfa9a37e227dab2ad22cb registry.k8s.io/provider-aws/cloud-controller-manager:v1.26.0],SizeBytes:18345226,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:8502345,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:7685305,},ContainerImage{Names:[registry.k8s.io/kops/kube-apiserver-healthcheck:1.27.0-alpha.1],SizeBytes:4740762,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:253553,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Mar 11 16:51:28.845: INFO: Logging kubelet events for node i-0e3467ec552af797e Mar 11 16:51:28.997: INFO: Logging pods the kubelet thinks is on node i-0e3467ec552af797e Mar 11 16:51:29.155: INFO: karpenter-webhook-5b646cd4f7-6sg2k started at 2023-03-11 16:21:58 +0000 UTC (0+1 container statuses recorded) Mar 11 16:51:29.155: INFO: Container webhook ready: true, restart count 0 Mar 11 16:51:29.155: INFO: karpenter-controller-567749677d-d9szs started at 2023-03-11 16:21:58 +0000 UTC (0+1 container statuses recorded) Mar 11 16:51:29.155: INFO: Container controller ready: true, restart count 0 Mar 11 16:51:29.155: INFO: kube-controller-manager-i-0e3467ec552af797e started at 2023-03-11 16:18:44 +0000 UTC (0+1 container statuses recorded) Mar 11 16:51:29.155: INFO: Container kube-controller-manager ready: true, restart count 4 Mar 11 16:51:29.155: INFO: etcd-manager-events-i-0e3467ec552af797e started at 2023-03-11 16:18:44 +0000 UTC (11+1 container statuses recorded) Mar 11 16:51:29.155: INFO: Init container init-etcd-3-2-24 ready: true, restart count 0 Mar 11 16:51:29.155: INFO: Init container init-etcd-3-3-10 ready: true, restart count 0 Mar 11 16:51:29.155: INFO: Init container init-etcd-3-3-17 ready: true, restart count 0 Mar 11 16:51:29.155: INFO: Init container init-etcd-3-4-13 ready: true, restart count 0 Mar 11 16:51:29.155: INFO: Init container init-etcd-3-4-3 ready: true, restart count 0 Mar 11 16:51:29.155: INFO: Init container init-etcd-3-5-0 ready: true, restart count 0 Mar 11 16:51:29.155: INFO: Init container init-etcd-3-5-1 ready: true, restart count 0 Mar 11 16:51:29.155: INFO: Init container init-etcd-3-5-3 ready: true, restart count 0 Mar 11 16:51:29.155: INFO: Init container init-etcd-3-5-4 ready: true, restart count 0 Mar 11 16:51:29.155: INFO: Init container init-etcd-3-5-6 ready: true, restart count 0 Mar 11 16:51:29.155: INFO: Init container init-etcd-3-5-7 ready: true, restart count 0 Mar 11 16:51:29.155: INFO: Container etcd-manager ready: true, restart count 0 Mar 11 16:51:29.155: INFO: etcd-manager-main-i-0e3467ec552af797e started at 2023-03-11 16:18:44 +0000 UTC (11+1 container statuses recorded) Mar 11 16:51:29.155: INFO: Init container init-etcd-3-2-24 ready: true, restart count 0 Mar 11 16:51:29.155: INFO: Init container init-etcd-3-3-10 ready: true, restart count 0 Mar 11 16:51:29.155: INFO: Init container init-etcd-3-3-17 ready: true, restart count 0 Mar 11 16:51:29.155: INFO: Init container init-etcd-3-4-13 ready: true, restart count 0 Mar 11 16:51:29.155: INFO: Init container init-etcd-3-4-3 ready: true, restart count 0 Mar 11 16:51:29.155: INFO: Init container init-etcd-3-5-0 ready: true, restart count 0 Mar 11 16:51:29.155: INFO: Init container init-etcd-3-5-1 ready: true, restart count 0 Mar 11 16:51:29.155: INFO: Init container init-etcd-3-5-3 ready: true, restart count 0 Mar 11 16:51:29.155: INFO: Init container init-etcd-3-5-4 ready: true, restart count 0 Mar 11 16:51:29.155: INFO: Init container init-etcd-3-5-6 ready: true, restart count 0 Mar 11 16:51:29.155: INFO: Init container init-etcd-3-5-7 ready: true, restart count 0 Mar 11 16:51:29.155: INFO: Container etcd-manager ready: true, restart count 0 Mar 11 16:51:29.155: INFO: ebs-csi-node-lpwz9 started at 2023-03-11 16:21:37 +0000 UTC (0+3 container statuses recorded) Mar 11 16:51:29.155: INFO: Container ebs-plugin ready: true, restart count 0 Mar 11 16:51:29.155: INFO: Container liveness-probe ready: true, restart count 0 Mar 11 16:51:29.155: INFO: Container node-driver-registrar ready: true, restart count 0 Mar 11 16:51:29.155: INFO: cilium-operator-8496cc65f5-qhdg8 started at 2023-03-11 16:21:37 +0000 UTC (0+1 container statuses recorded) Mar 11 16:51:29.155: INFO: Container cilium-operator ready: true, restart count 0 Mar 11 16:51:29.155: INFO: kops-controller-76gkw started at 2023-03-11 16:21:37 +0000 UTC (0+1 container statuses recorded) Mar 11 16:51:29.156: INFO: Container kops-controller ready: true, restart count 0 Mar 11 16:51:29.156: INFO: aws-cloud-controller-manager-kwbw2 started at 2023-03-11 16:21:37 +0000 UTC (0+1 container statuses recorded) Mar 11 16:51:29.156: INFO: Container aws-cloud-controller-manager ready: true, restart count 0 Mar 11 16:51:29.156: INFO: kube-apiserver-i-0e3467ec552af797e started at 2023-03-11 16:18:44 +0000 UTC (0+2 container statuses recorded) Mar 11 16:51:29.156: INFO: Container healthcheck ready: true, restart count 0 Mar 11 16:51:29.156: INFO: Container kube-apiserver ready: true, restart count 3 Mar 11 16:51:29.156: INFO: kube-scheduler-i-0e3467ec552af797e started at 2023-03-11 16:18:44 +0000 UTC (0+1 container statuses recorded) Mar 11 16:51:29.156: INFO: Container kube-scheduler ready: true, restart count 0 Mar 11 16:51:29.156: INFO: dns-controller-7dfd64d999-q9hh7 started at 2023-03-11 16:21:37 +0000 UTC (0+1 container statuses recorded) Mar 11 16:51:29.156: INFO: Container dns-controller ready: true, restart count 0 Mar 11 16:51:29.156: INFO: cilium-qrsfn started at 2023-03-11 16:21:37 +0000 UTC (1+1 container statuses recorded) Mar 11 16:51:29.156: INFO: Init container clean-cilium-state ready: true, restart count 0 Mar 11 16:51:29.156: INFO: Container cilium-agent ready: true, restart count 0 Mar 11 16:51:29.794: INFO: Latency metrics for node i-0e3467ec552af797e END STEP: dump namespace information after failure - test/e2e/framework/framework.go:288 @ 03/11/23 16:51:29.794 (22.44s) < Exit [DeferCleanup (Each)] [sig-node] Probing container - dump namespaces | framework.go:209 @ 03/11/23 16:51:29.794 (22.44s) > Enter [DeferCleanup (Each)] [sig-node] Probing container - tear down framework | framework.go:206 @ 03/11/23 16:51:29.794 STEP: Destroying namespace "container-probe-4831" for this suite. - test/e2e/framework/framework.go:351 @ 03/11/23 16:51:29.794 < Exit [DeferCleanup (Each)] [sig-node] Probing container - tear down framework | framework.go:206 @ 03/11/23 16:51:29.944 (150ms) > Enter [ReportAfterEach] TOP-LEVEL - test/e2e/e2e_test.go:144 @ 03/11/23 16:51:29.944 < Exit [ReportAfterEach] TOP-LEVEL - test/e2e/e2e_test.go:144 @ 03/11/23 16:51:29.944 (0s)
Find container-probe-4831/startup-81ef738d-6302-42b0-9bac-a06875bbc4dd mentions in log files | View test history on testgrid
go run hack/e2e.go -v --test --test_args='--ginkgo.focus=Kubernetes\se2e\ssuite\s\[It\]\s\[sig\-node\]\sProbing\scontainer\sshould\shave\smonotonically\sincreasing\srestart\scount\s\[NodeConformance\]\s\[Conformance\]$'
[FAILED] getting pod : pods "liveness-a3f2610d-e3b8-4480-af3b-568d9b8080d3" not found In [It] at: test/e2e/common/node/container_probe.go:978 @ 03/11/23 16:50:48.317 There were additional failures detected after the initial failure. These are visible in the timeline
> Enter [BeforeEach] [sig-node] Probing container - set up framework | framework.go:191 @ 03/11/23 16:42:27.976 STEP: Creating a kubernetes client - test/e2e/framework/framework.go:211 @ 03/11/23 16:42:27.976 Mar 11 16:42:27.976: INFO: >>> kubeConfig: /root/.kube/config STEP: Building a namespace api object, basename container-probe - test/e2e/framework/framework.go:250 @ 03/11/23 16:42:27.977 STEP: Waiting for a default service account to be provisioned in namespace - test/e2e/framework/framework.go:259 @ 03/11/23 16:42:28.422 STEP: Waiting for kube-root-ca.crt to be provisioned in namespace - test/e2e/framework/framework.go:262 @ 03/11/23 16:42:28.717 < Exit [BeforeEach] [sig-node] Probing container - set up framework | framework.go:191 @ 03/11/23 16:42:29.012 (1.036s) > Enter [BeforeEach] [sig-node] Probing container - test/e2e/framework/metrics/init/init.go:33 @ 03/11/23 16:42:29.012 < Exit [BeforeEach] [sig-node] Probing container - test/e2e/framework/metrics/init/init.go:33 @ 03/11/23 16:42:29.012 (0s) > Enter [BeforeEach] [sig-node] Probing container - test/e2e/common/node/container_probe.go:62 @ 03/11/23 16:42:29.012 < Exit [BeforeEach] [sig-node] Probing container - test/e2e/common/node/container_probe.go:62 @ 03/11/23 16:42:29.012 (0s) > Enter [It] should have monotonically increasing restart count [NodeConformance] [Conformance] - test/e2e/common/node/container_probe.go:198 @ 03/11/23 16:42:29.012 STEP: Creating pod liveness-a3f2610d-e3b8-4480-af3b-568d9b8080d3 in namespace container-probe-1905 - test/e2e/common/node/container_probe.go:955 @ 03/11/23 16:42:29.012 Mar 11 16:42:31.462: INFO: Started pod liveness-a3f2610d-e3b8-4480-af3b-568d9b8080d3 in namespace container-probe-1905 STEP: checking the pod's current state and verifying that restartCount is present - test/e2e/common/node/container_probe.go:966 @ 03/11/23 16:42:31.462 Mar 11 16:42:31.610: INFO: Initial restart count of pod liveness-a3f2610d-e3b8-4480-af3b-568d9b8080d3 is 0 Mar 11 16:42:51.098: INFO: Restart count of pod container-probe-1905/liveness-a3f2610d-e3b8-4480-af3b-568d9b8080d3 is now 1 (19.487621825s elapsed) Mar 11 16:43:10.445: INFO: Restart count of pod container-probe-1905/liveness-a3f2610d-e3b8-4480-af3b-568d9b8080d3 is now 2 (38.834392521s elapsed) Mar 11 16:43:31.936: INFO: Restart count of pod container-probe-1905/liveness-a3f2610d-e3b8-4480-af3b-568d9b8080d3 is now 3 (1m0.325347978s elapsed) Mar 11 16:43:51.277: INFO: Restart count of pod container-probe-1905/liveness-a3f2610d-e3b8-4480-af3b-568d9b8080d3 is now 4 (1m19.666611477s elapsed) Mar 11 16:50:48.317: INFO: Unexpected error: getting pod : <*errors.StatusError | 0xc003478c80>: { ErrStatus: code: 404 details: kind: pods name: liveness-a3f2610d-e3b8-4480-af3b-568d9b8080d3 message: pods "liveness-a3f2610d-e3b8-4480-af3b-568d9b8080d3" not found metadata: {} reason: NotFound status: Failure, } [FAILED] getting pod : pods "liveness-a3f2610d-e3b8-4480-af3b-568d9b8080d3" not found In [It] at: test/e2e/common/node/container_probe.go:978 @ 03/11/23 16:50:48.317 < Exit [It] should have monotonically increasing restart count [NodeConformance] [Conformance] - test/e2e/common/node/container_probe.go:198 @ 03/11/23 16:50:48.317 (8m19.305s) > Enter [AfterEach] [sig-node] Probing container - test/e2e/framework/node/init/init.go:33 @ 03/11/23 16:50:48.317 Mar 11 16:50:48.317: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready Mar 11 16:50:48.467: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:50:48.467: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:50:50.617: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:50:50.617: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:50:52.618: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:50:52.618: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:50:54.617: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:50:54.617: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:50:56.617: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:50:56.617: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:50:58.617: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:50:58.617: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:51:00.617: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:51:00.617: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:51:02.618: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:51:02.618: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:51:04.617: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:51:04.617: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:51:06.618: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:51:06.618: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:51:08.618: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:51:08.618: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:51:10.618: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:51:10.618: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:51:12.617: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:51:12.617: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:51:14.617: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:51:14.617: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:51:16.618: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:51:16.618: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:51:18.618: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:51:18.618: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:51:20.618: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:51:20.618: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:51:22.617: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:51:22.617: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:51:24.617: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:51:24.617: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:51:26.617: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:51:26.617: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:51:28.618: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:51:28.618: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:51:30.617: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:51:30.617: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:51:32.618: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:51:32.618: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:51:34.617: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:51:34.617: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failure Mar 11 16:51:36.617: INFO: Condition Ready of node i-04523d1dffb0fe725 is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:31 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:36 +0000 UTC}]. Failure Mar 11 16:51:36.617: INFO: Condition Ready of node i-07e2b6e650d7ef56d is false, but Node is tainted by NodeController with [{node.kubernetes.io/unreachable NoSchedule 2023-03-11 16:45:36 +0000 UTC} {node.kubernetes.io/unreachable NoExecute 2023-03-11 16:45:46 +0000 UTC}]. Failu