Result | FAILURE |
Tests | 48 failed / 736 succeeded |
Started | |
Elapsed | 45m17s |
Revision | master |
go run hack/e2e.go -v --test --test_args='--ginkgo.focus=Kubernetes\se2e\ssuite\sExternal\sStorage\s\[Driver\:\sebs\.csi\.aws\.com\]\s\[Testpattern\:\sDynamic\sPV\s\(default\sfs\)\]\sfsgroupchangepolicy\s\(Always\)\[LinuxOnly\]\,\spod\screated\swith\san\sinitial\sfsgroup\,\snew\spod\sfsgroup\sapplied\sto\svolume\scontents$'
/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/fsgroupchangepolicy.go:214 Jan 9 23:08:13.132: Unexpected error: <*errors.errorString | 0xc004e90650>: { s: "pod \"pod-d16d48d3-63ef-4a76-95b8-d80f8b4aec62\" is not Running: timed out waiting for the condition", } pod "pod-d16d48d3-63ef-4a76-95b8-d80f8b4aec62" is not Running: timed out waiting for the condition occurred /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/fsgroupchangepolicy.go:262from junit_17.xml
[BeforeEach] [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/framework/testsuite.go:51 [BeforeEach] [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:185 �[1mSTEP�[0m: Creating a kubernetes client Jan 9 23:03:12.680: INFO: >>> kubeConfig: /root/.kube/config �[1mSTEP�[0m: Building a namespace api object, basename fsgroupchangepolicy �[1mSTEP�[0m: Waiting for a default service account to be provisioned in namespace �[1mSTEP�[0m: Waiting for kube-root-ca.crt to be provisioned in namespace [It] (Always)[LinuxOnly], pod created with an initial fsgroup, new pod fsgroup applied to volume contents /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/fsgroupchangepolicy.go:214 Jan 9 23:03:12.902: INFO: Creating resource for dynamic PV Jan 9 23:03:12.902: INFO: Using claimSize:1Gi, test suite supported size:{ 1Mi}, driver(ebs.csi.aws.com) supported size:{ 1Mi} �[1mSTEP�[0m: creating a StorageClass fsgroupchangepolicy-7772-e2e-scpplpf �[1mSTEP�[0m: creating a claim Jan 9 23:03:12.937: INFO: Warning: Making PVC: VolumeMode specified as invalid empty string, treating as nil �[1mSTEP�[0m: Creating Pod in namespace fsgroupchangepolicy-7772 with fsgroup 1000 Jan 9 23:08:13.132: FAIL: Unexpected error: <*errors.errorString | 0xc004e90650>: { s: "pod \"pod-d16d48d3-63ef-4a76-95b8-d80f8b4aec62\" is not Running: timed out waiting for the condition", } pod "pod-d16d48d3-63ef-4a76-95b8-d80f8b4aec62" is not Running: timed out waiting for the condition occurred Full Stack Trace k8s.io/kubernetes/test/e2e/storage/testsuites.createPodAndVerifyContentGid(0xc0049709a0, 0xc00198f070, 0x1, {0x0, 0x0}, {0x0, 0x0}) /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/fsgroupchangepolicy.go:262 +0x15c k8s.io/kubernetes/test/e2e/storage/testsuites.(*fsGroupChangePolicyTestSuite).DefineTests.func3() /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/fsgroupchangepolicy.go:233 +0x296 k8s.io/kubernetes/test/e2e.RunE2ETests(0x24c66d7) _output/dockerized/go/src/k8s.io/kubernetes/test/e2e/e2e.go:133 +0x697 k8s.io/kubernetes/test/e2e.TestE2E(0x243a8f9) _output/dockerized/go/src/k8s.io/kubernetes/test/e2e/e2e_test.go:136 +0x19 testing.tRunner(0xc0005c0680, 0x735e880) /usr/local/go/src/testing/testing.go:1259 +0x102 created by testing.(*T).Run /usr/local/go/src/testing/testing.go:1306 +0x35a �[1mSTEP�[0m: Deleting pvc Jan 9 23:08:13.199: INFO: Deleting PersistentVolumeClaim "ebs.csi.aws.comx82nx" Jan 9 23:08:13.233: INFO: Waiting up to 3m0s for PersistentVolume pvc-7d5ee7da-e48a-48b5-90f3-b7fa94aa7a2c to get deleted Jan 9 23:08:13.264: INFO: PersistentVolume pvc-7d5ee7da-e48a-48b5-90f3-b7fa94aa7a2c found and phase=Bound (31.006834ms) Jan 9 23:08:18.296: INFO: PersistentVolume pvc-7d5ee7da-e48a-48b5-90f3-b7fa94aa7a2c found and phase=Bound (5.063455101s) Jan 9 23:08:23.328: INFO: PersistentVolume pvc-7d5ee7da-e48a-48b5-90f3-b7fa94aa7a2c found and phase=Bound (10.094915618s) Jan 9 23:08:28.360: INFO: PersistentVolume pvc-7d5ee7da-e48a-48b5-90f3-b7fa94aa7a2c found and phase=Bound (15.127595743s) Jan 9 23:08:33.392: INFO: PersistentVolume pvc-7d5ee7da-e48a-48b5-90f3-b7fa94aa7a2c found and phase=Bound (20.159514206s) Jan 9 23:08:38.425: INFO: PersistentVolume pvc-7d5ee7da-e48a-48b5-90f3-b7fa94aa7a2c found and phase=Bound (25.192394604s) Jan 9 23:08:43.457: INFO: PersistentVolume pvc-7d5ee7da-e48a-48b5-90f3-b7fa94aa7a2c found and phase=Bound (30.224335063s) Jan 9 23:08:48.489: INFO: PersistentVolume pvc-7d5ee7da-e48a-48b5-90f3-b7fa94aa7a2c found and phase=Bound (35.256703252s) Jan 9 23:08:53.522: INFO: PersistentVolume pvc-7d5ee7da-e48a-48b5-90f3-b7fa94aa7a2c found and phase=Bound (40.289539567s) Jan 9 23:08:58.554: INFO: PersistentVolume pvc-7d5ee7da-e48a-48b5-90f3-b7fa94aa7a2c found and phase=Bound (45.321586505s) Jan 9 23:09:03.586: INFO: PersistentVolume pvc-7d5ee7da-e48a-48b5-90f3-b7fa94aa7a2c found and phase=Bound (50.353587663s) Jan 9 23:09:08.619: INFO: PersistentVolume pvc-7d5ee7da-e48a-48b5-90f3-b7fa94aa7a2c found and phase=Bound (55.385884134s) Jan 9 23:09:13.651: INFO: PersistentVolume pvc-7d5ee7da-e48a-48b5-90f3-b7fa94aa7a2c found and phase=Bound (1m0.4179813s) Jan 9 23:09:18.682: INFO: PersistentVolume pvc-7d5ee7da-e48a-48b5-90f3-b7fa94aa7a2c found and phase=Bound (1m5.449367864s) Jan 9 23:09:23.714: INFO: PersistentVolume pvc-7d5ee7da-e48a-48b5-90f3-b7fa94aa7a2c found and phase=Bound (1m10.481039991s) Jan 9 23:09:28.746: INFO: PersistentVolume pvc-7d5ee7da-e48a-48b5-90f3-b7fa94aa7a2c found and phase=Bound (1m15.51287576s) Jan 9 23:09:33.778: INFO: PersistentVolume pvc-7d5ee7da-e48a-48b5-90f3-b7fa94aa7a2c found and phase=Bound (1m20.545012732s) Jan 9 23:09:38.815: INFO: PersistentVolume pvc-7d5ee7da-e48a-48b5-90f3-b7fa94aa7a2c found and phase=Bound (1m25.582253082s) Jan 9 23:09:43.847: INFO: PersistentVolume pvc-7d5ee7da-e48a-48b5-90f3-b7fa94aa7a2c found and phase=Bound (1m30.614226245s) Jan 9 23:09:48.894: INFO: PersistentVolume pvc-7d5ee7da-e48a-48b5-90f3-b7fa94aa7a2c found and phase=Bound (1m35.66160062s) Jan 9 23:09:53.927: INFO: PersistentVolume pvc-7d5ee7da-e48a-48b5-90f3-b7fa94aa7a2c found and phase=Bound (1m40.693930172s) Jan 9 23:09:58.959: INFO: PersistentVolume pvc-7d5ee7da-e48a-48b5-90f3-b7fa94aa7a2c found and phase=Bound (1m45.725947372s) Jan 9 23:10:03.991: INFO: PersistentVolume pvc-7d5ee7da-e48a-48b5-90f3-b7fa94aa7a2c found and phase=Bound (1m50.75875338s) Jan 9 23:10:09.023: INFO: PersistentVolume pvc-7d5ee7da-e48a-48b5-90f3-b7fa94aa7a2c found and phase=Bound (1m55.790640082s) Jan 9 23:10:14.055: INFO: PersistentVolume pvc-7d5ee7da-e48a-48b5-90f3-b7fa94aa7a2c found and phase=Bound (2m0.822473664s) Jan 9 23:10:19.087: INFO: PersistentVolume pvc-7d5ee7da-e48a-48b5-90f3-b7fa94aa7a2c found and phase=Bound (2m5.854352562s) Jan 9 23:10:24.120: INFO: PersistentVolume pvc-7d5ee7da-e48a-48b5-90f3-b7fa94aa7a2c found and phase=Bound (2m10.887707296s) Jan 9 23:10:29.152: INFO: PersistentVolume pvc-7d5ee7da-e48a-48b5-90f3-b7fa94aa7a2c found and phase=Bound (2m15.91925655s) Jan 9 23:10:34.184: INFO: PersistentVolume pvc-7d5ee7da-e48a-48b5-90f3-b7fa94aa7a2c found and phase=Bound (2m20.951304913s) Jan 9 23:10:39.216: INFO: PersistentVolume pvc-7d5ee7da-e48a-48b5-90f3-b7fa94aa7a2c found and phase=Bound (2m25.983265322s) Jan 9 23:10:44.249: INFO: PersistentVolume pvc-7d5ee7da-e48a-48b5-90f3-b7fa94aa7a2c found and phase=Bound (2m31.015960362s) Jan 9 23:10:49.281: INFO: PersistentVolume pvc-7d5ee7da-e48a-48b5-90f3-b7fa94aa7a2c found and phase=Bound (2m36.048391967s) Jan 9 23:10:54.313: INFO: PersistentVolume pvc-7d5ee7da-e48a-48b5-90f3-b7fa94aa7a2c found and phase=Bound (2m41.080045224s) Jan 9 23:10:59.345: INFO: PersistentVolume pvc-7d5ee7da-e48a-48b5-90f3-b7fa94aa7a2c found and phase=Bound (2m46.112477564s) Jan 9 23:11:04.379: INFO: PersistentVolume pvc-7d5ee7da-e48a-48b5-90f3-b7fa94aa7a2c found and phase=Bound (2m51.146534884s) Jan 9 23:11:09.411: INFO: PersistentVolume pvc-7d5ee7da-e48a-48b5-90f3-b7fa94aa7a2c found and phase=Bound (2m56.178646997s) �[1mSTEP�[0m: Deleting sc Jan 9 23:11:14.446: FAIL: while cleanup resource Unexpected error: <errors.aggregate | len:1, cap:1>: [ [ { msg: "persistent Volume pvc-7d5ee7da-e48a-48b5-90f3-b7fa94aa7a2c not deleted by dynamic provisioner: PersistentVolume pvc-7d5ee7da-e48a-48b5-90f3-b7fa94aa7a2c still exists within 3m0s", err: { s: "PersistentVolume pvc-7d5ee7da-e48a-48b5-90f3-b7fa94aa7a2c still exists within 3m0s", }, }, ], ] persistent Volume pvc-7d5ee7da-e48a-48b5-90f3-b7fa94aa7a2c not deleted by dynamic provisioner: PersistentVolume pvc-7d5ee7da-e48a-48b5-90f3-b7fa94aa7a2c still exists within 3m0s occurred Full Stack Trace k8s.io/kubernetes/test/e2e/storage/testsuites.(*fsGroupChangePolicyTestSuite).DefineTests.func2() /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/fsgroupchangepolicy.go:132 +0x20f panic({0x6c3d000, 0xc002eaffc0}) /usr/local/go/src/runtime/panic.go:1038 +0x215 k8s.io/kubernetes/test/e2e/framework/ginkgowrapper.Fail.func1() /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/ginkgowrapper/wrapper.go:63 +0x73 panic({0x62d5960, 0x78abec0}) /usr/local/go/src/runtime/panic.go:1038 +0x215 k8s.io/kubernetes/test/e2e/framework/ginkgowrapper.Fail({0xc00406a500, 0x13b}, {0xc006de6d30, 0x70cbf6a, 0xc006de6d50}) /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/ginkgowrapper/wrapper.go:67 +0x1a7 k8s.io/kubernetes/test/e2e/framework.Fail({0xc00406a3c0, 0x126}, {0xc004d9ec70, 0xc00406a3c0, 0xc003fb20d0}) /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/log.go:63 +0x149 k8s.io/kubernetes/vendor/github.com/onsi/gomega/internal/assertion.(*Assertion).match(0xc006de6e98, {0x79beb98, 0xaa12408}, 0x0, {0x0, 0x0, 0x0}) /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/onsi/gomega/internal/assertion/assertion.go:79 +0x1bd k8s.io/kubernetes/vendor/github.com/onsi/gomega/internal/assertion.(*Assertion).NotTo(0xc006de6e98, {0x79beb98, 0xaa12408}, {0x0, 0x0, 0x0}) /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/onsi/gomega/internal/assertion/assertion.go:48 +0x92 k8s.io/kubernetes/test/e2e/framework.ExpectNoErrorWithOffset(0x7b06bd0, {0x78b2200, 0xc004e90650}, {0x0, 0x2, 0xc003faad20}) /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/expect.go:46 +0xa9 k8s.io/kubernetes/test/e2e/framework.ExpectNoError(...) /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/expect.go:40 k8s.io/kubernetes/test/e2e/storage/testsuites.createPodAndVerifyContentGid(0xc0049709a0, 0xc00198f070, 0x1, {0x0, 0x0}, {0x0, 0x0}) /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/fsgroupchangepolicy.go:262 +0x15c k8s.io/kubernetes/test/e2e/storage/testsuites.(*fsGroupChangePolicyTestSuite).DefineTests.func3() /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/fsgroupchangepolicy.go:233 +0x296 k8s.io/kubernetes/test/e2e.RunE2ETests(0x24c66d7) _output/dockerized/go/src/k8s.io/kubernetes/test/e2e/e2e.go:133 +0x697 k8s.io/kubernetes/test/e2e.TestE2E(0x243a8f9) _output/dockerized/go/src/k8s.io/kubernetes/test/e2e/e2e_test.go:136 +0x19 testing.tRunner(0xc0005c0680, 0x735e880) /usr/local/go/src/testing/testing.go:1259 +0x102 created by testing.(*T).Run /usr/local/go/src/testing/testing.go:1306 +0x35a [AfterEach] [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:186 �[1mSTEP�[0m: Collecting events from namespace "fsgroupchangepolicy-7772". �[1mSTEP�[0m: Found 17 events. Jan 9 23:11:14.480: INFO: At 2023-01-09 23:03:12 +0000 UTC - event for ebs.csi.aws.comx82nx: {persistentvolume-controller } WaitForFirstConsumer: waiting for first consumer to be created before binding Jan 9 23:11:14.480: INFO: At 2023-01-09 23:03:13 +0000 UTC - event for ebs.csi.aws.comx82nx: {persistentvolume-controller } ExternalProvisioning: waiting for a volume to be created, either by external provisioner "ebs.csi.aws.com" or manually created by system administrator Jan 9 23:11:14.480: INFO: At 2023-01-09 23:03:13 +0000 UTC - event for ebs.csi.aws.comx82nx: {ebs.csi.aws.com_ebs-csi-controller-5d8d7cc949-8wxx2_35215d1d-5b9d-46a9-b0b7-d7677b375a3a } Provisioning: External provisioner is provisioning volume for claim "fsgroupchangepolicy-7772/ebs.csi.aws.comx82nx" Jan 9 23:11:14.480: INFO: At 2023-01-09 23:03:16 +0000 UTC - event for ebs.csi.aws.comx82nx: {ebs.csi.aws.com_ebs-csi-controller-5d8d7cc949-8wxx2_35215d1d-5b9d-46a9-b0b7-d7677b375a3a } ProvisioningSucceeded: Successfully provisioned volume pvc-7d5ee7da-e48a-48b5-90f3-b7fa94aa7a2c Jan 9 23:11:14.480: INFO: At 2023-01-09 23:03:17 +0000 UTC - event for pod-d16d48d3-63ef-4a76-95b8-d80f8b4aec62: {default-scheduler } Scheduled: Successfully assigned fsgroupchangepolicy-7772/pod-d16d48d3-63ef-4a76-95b8-d80f8b4aec62 to ip-172-20-57-251.ca-central-1.compute.internal Jan 9 23:11:14.480: INFO: At 2023-01-09 23:03:19 +0000 UTC - event for pod-d16d48d3-63ef-4a76-95b8-d80f8b4aec62: {attachdetach-controller } SuccessfulAttachVolume: AttachVolume.Attach succeeded for volume "pvc-7d5ee7da-e48a-48b5-90f3-b7fa94aa7a2c" Jan 9 23:11:14.480: INFO: At 2023-01-09 23:03:24 +0000 UTC - event for pod-d16d48d3-63ef-4a76-95b8-d80f8b4aec62: {kubelet ip-172-20-57-251.ca-central-1.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "abd55b9661108b87af9a74922a8c70f9a1ac379271084ad0424fc5ebc1cd79fd" network for pod "pod-d16d48d3-63ef-4a76-95b8-d80f8b4aec62": networkPlugin cni failed to set up pod "pod-d16d48d3-63ef-4a76-95b8-d80f8b4aec62_fsgroupchangepolicy-7772" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 9 23:11:14.480: INFO: At 2023-01-09 23:03:24 +0000 UTC - event for pod-d16d48d3-63ef-4a76-95b8-d80f8b4aec62: {kubelet ip-172-20-57-251.ca-central-1.compute.internal} SandboxChanged: Pod sandbox changed, it will be killed and re-created. Jan 9 23:11:14.480: INFO: At 2023-01-09 23:03:27 +0000 UTC - event for pod-d16d48d3-63ef-4a76-95b8-d80f8b4aec62: {kubelet ip-172-20-57-251.ca-central-1.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "fed3a96732c493eb32bd630e2cc21a82d481640404c2e66a725f4dbd5b85436c" network for pod "pod-d16d48d3-63ef-4a76-95b8-d80f8b4aec62": networkPlugin cni failed to set up pod "pod-d16d48d3-63ef-4a76-95b8-d80f8b4aec62_fsgroupchangepolicy-7772" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 9 23:11:14.480: INFO: At 2023-01-09 23:03:30 +0000 UTC - event for pod-d16d48d3-63ef-4a76-95b8-d80f8b4aec62: {kubelet ip-172-20-57-251.ca-central-1.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "8170b8ebc7b7d1c7ecc09286b39f36e96094450a615c540acd9f5da5eac075ed" network for pod "pod-d16d48d3-63ef-4a76-95b8-d80f8b4aec62": networkPlugin cni failed to set up pod "pod-d16d48d3-63ef-4a76-95b8-d80f8b4aec62_fsgroupchangepolicy-7772" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 9 23:11:14.480: INFO: At 2023-01-09 23:03:34 +0000 UTC - event for pod-d16d48d3-63ef-4a76-95b8-d80f8b4aec62: {kubelet ip-172-20-57-251.ca-central-1.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "b060a383f746869d15e6e6aaa510fa4f95f74e6ee68a4b6dd3529a9f2bc895f2" network for pod "pod-d16d48d3-63ef-4a76-95b8-d80f8b4aec62": networkPlugin cni failed to set up pod "pod-d16d48d3-63ef-4a76-95b8-d80f8b4aec62_fsgroupchangepolicy-7772" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 9 23:11:14.480: INFO: At 2023-01-09 23:03:36 +0000 UTC - event for pod-d16d48d3-63ef-4a76-95b8-d80f8b4aec62: {kubelet ip-172-20-57-251.ca-central-1.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "5876801c4f1c710bab7f1530c00a357a78b67a5e54c1b2739713848e676408af" network for pod "pod-d16d48d3-63ef-4a76-95b8-d80f8b4aec62": networkPlugin cni failed to set up pod "pod-d16d48d3-63ef-4a76-95b8-d80f8b4aec62_fsgroupchangepolicy-7772" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 9 23:11:14.480: INFO: At 2023-01-09 23:03:39 +0000 UTC - event for pod-d16d48d3-63ef-4a76-95b8-d80f8b4aec62: {kubelet ip-172-20-57-251.ca-central-1.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "e5ee7d4bf9f8c796d6ac5469211538f93760977272674c304fcee0843fcd26d9" network for pod "pod-d16d48d3-63ef-4a76-95b8-d80f8b4aec62": networkPlugin cni failed to set up pod "pod-d16d48d3-63ef-4a76-95b8-d80f8b4aec62_fsgroupchangepolicy-7772" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 9 23:11:14.480: INFO: At 2023-01-09 23:03:40 +0000 UTC - event for pod-d16d48d3-63ef-4a76-95b8-d80f8b4aec62: {kubelet ip-172-20-57-251.ca-central-1.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "50019a64cd62a53c335fd617ad3ce8044ce18fb4150939530676a9b116e1ed29" network for pod "pod-d16d48d3-63ef-4a76-95b8-d80f8b4aec62": networkPlugin cni failed to set up pod "pod-d16d48d3-63ef-4a76-95b8-d80f8b4aec62_fsgroupchangepolicy-7772" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 9 23:11:14.480: INFO: At 2023-01-09 23:03:42 +0000 UTC - event for pod-d16d48d3-63ef-4a76-95b8-d80f8b4aec62: {kubelet ip-172-20-57-251.ca-central-1.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "b52e9e80e014627fe169ccff06340a9127e82a7b3345ff731f84dc2cf6e0800a" network for pod "pod-d16d48d3-63ef-4a76-95b8-d80f8b4aec62": networkPlugin cni failed to set up pod "pod-d16d48d3-63ef-4a76-95b8-d80f8b4aec62_fsgroupchangepolicy-7772" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 9 23:11:14.480: INFO: At 2023-01-09 23:03:44 +0000 UTC - event for pod-d16d48d3-63ef-4a76-95b8-d80f8b4aec62: {kubelet ip-172-20-57-251.ca-central-1.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "2e7839ccf44a1b432054f6a96f039cfde6514488e01f13d9ef771e40df0f2575" network for pod "pod-d16d48d3-63ef-4a76-95b8-d80f8b4aec62": networkPlugin cni failed to set up pod "pod-d16d48d3-63ef-4a76-95b8-d80f8b4aec62_fsgroupchangepolicy-7772" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 9 23:11:14.481: INFO: At 2023-01-09 23:03:47 +0000 UTC - event for pod-d16d48d3-63ef-4a76-95b8-d80f8b4aec62: {kubelet ip-172-20-57-251.ca-central-1.compute.internal} FailedCreatePodSandBox: (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "24d270c7f1b115f35b364e12312be7c6681c3cf4c23c75660a82f9b5efe03758" network for pod "pod-d16d48d3-63ef-4a76-95b8-d80f8b4aec62": networkPlugin cni failed to set up pod "pod-d16d48d3-63ef-4a76-95b8-d80f8b4aec62_fsgroupchangepolicy-7772" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 9 23:11:14.512: INFO: POD NODE PHASE GRACE CONDITIONS Jan 9 23:11:14.512: INFO: pod-d16d48d3-63ef-4a76-95b8-d80f8b4aec62 ip-172-20-57-251.ca-central-1.compute.internal Running [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2023-01-09 23:03:17 +0000 UTC } {Ready True 0001-01-01 00:00:00 +0000 UTC 2023-01-09 23:10:26 +0000 UTC } {ContainersReady True 0001-01-01 00:00:00 +0000 UTC 2023-01-09 23:10:26 +0000 UTC } {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2023-01-09 23:03:17 +0000 UTC }] Jan 9 23:11:14.512: INFO: Jan 9 23:11:14.590: INFO: Logging node info for node ip-172-20-32-67.ca-central-1.compute.internal Jan 9 23:11:14.622: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-32-67.ca-central-1.compute.internal 1f51981b-a01d-404c-b3a4-6649f301f65d 45435 0 2023-01-09 22:38:45 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ca-central-1 failure-domain.beta.kubernetes.io/zone:ca-central-1a kubelet_cleanup:true kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-32-67.ca-central-1.compute.internal kubernetes.io/os:linux kubernetes.io/role:node node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:ca-central-1a topology.hostpath.csi/node:ip-172-20-32-67.ca-central-1.compute.internal topology.kubernetes.io/region:ca-central-1 topology.kubernetes.io/zone:ca-central-1a] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-0272cebea4c6c3d5d"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kops-controller Update v1 2023-01-09 22:38:45 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}} } {kube-controller-manager Update v1 2023-01-09 22:38:45 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.3.0/24\"":{}}}} } {kubelet Update v1 2023-01-09 22:38:45 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {e2e.test Update v1 2023-01-09 22:54:42 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubelet_cleanup":{}}}} } {kube-controller-manager Update v1 2023-01-09 22:55:59 +0000 UTC FieldsV1 {"f:status":{"f:volumesAttached":{}}} status} {kubelet Update v1 2023-01-09 22:56:00 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:capacity":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.3.0/24,DoNotUseExternalID:,ProviderID:aws:///ca-central-1a/i-0272cebea4c6c3d5d,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.3.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051644416 0} {<nil>} 3956684Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946786816 0} {<nil>} 3854284Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-01-09 23:06:12 +0000 UTC,LastTransitionTime:2023-01-09 22:38:45 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-01-09 23:06:12 +0000 UTC,LastTransitionTime:2023-01-09 22:38:45 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-01-09 23:06:12 +0000 UTC,LastTransitionTime:2023-01-09 22:38:45 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-01-09 23:06:12 +0000 UTC,LastTransitionTime:2023-01-09 22:39:06 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.32.67,},NodeAddress{Type:ExternalIP,Address:35.182.2.102,},NodeAddress{Type:Hostname,Address:ip-172-20-32-67.ca-central-1.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-32-67.ca-central-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-35-182-2-102.ca-central-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2536b48bd6b6dade0ed68af2555085,SystemUUID:ec2536b4-8bd6-b6da-de0e-d68af2555085,BootID:bfb3b0df-aafa-481c-9ac6-f85e5e4580e3,KernelVersion:5.15.0-1026-aws,OSImage:Ubuntu 22.04.1 LTS,ContainerRuntimeVersion:docker://20.10.17,KubeletVersion:v1.23.15,KubeProxyVersion:v1.23.15,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:456569528,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/jessie-dnsutils@sha256:11e6a66017ba4e4b938c1612b7a54a3befcefd354796c04e1dba76873a13518e k8s.gcr.io/e2e-test-images/jessie-dnsutils:1.5],SizeBytes:253371792,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:7e8bdd271312fd25fc5ff5a8f04727be84044eb3d7d8d03611972a6752e2e11e k8s.gcr.io/e2e-test-images/agnhost:2.39],SizeBytes:126872991,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nautilus@sha256:99c0d6f1ad24a1aa1905d9c6534d193f268f7b23f9add2ae6bb41f31094bdd5c k8s.gcr.io/e2e-test-images/nautilus:1.5],SizeBytes:124758741,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:1b9d1b2f36cb2dbee1960e82a9344aeb11bd4c4c03abf5e1853e0559c23855e3 k8s.gcr.io/e2e-test-images/httpd:2.4.38-2],SizeBytes:123781643,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.23.15 registry.k8s.io/kube-proxy-amd64:v1.23.15],SizeBytes:112336036,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:10e231cf07c82dfab5141c1afe548dc734e5fc1f67665ec3982a325d1bd31a9a registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.0],SizeBytes:94471559,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:c8e03f60afa90a28e4bb6ec9a8d0fc36d89de4b7475cf2d613afa793ec969fe0 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.0],SizeBytes:56382329,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:4e74c0492bceddc598de1c90cc5bc14dcda94cb49fa9c5bad9d117c4834b5e08 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.1],SizeBytes:56379269,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:36c31f7e1f433c9634d24f876353e8646246d81a03c4e351202c2644daff1620 k8s.gcr.io/sig-storage/csi-resizer:v1.2.0],SizeBytes:53956308,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:ed98431376c9e944e19a465fe8ea944806714dd95416a0821096c78d66b579bd k8s.gcr.io/sig-storage/csi-snapshotter:v4.1.1],SizeBytes:53531990,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:c5be65d6679efabb969d9b019300d187437ae876f992c40911fd2892bbef3b36 k8s.gcr.io/sig-storage/csi-attacher:v3.2.0],SizeBytes:53500826,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:60ab9b3e6a030d3038c87c0d6bca2930f58d1d72823e6a4af09767dc83b696a2 k8s.gcr.io/sig-storage/csi-attacher:v3.2.1],SizeBytes:53499570,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:017727efcfeb7d053af68e51436ce8e65edbc6ca573720afb4f79c8594036955 registry.k8s.io/coredns/coredns:v1.10.0],SizeBytes:50249337,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:7a5ba58a44e0d749e0767e4e37315bcf6a61f33ce3185c1991848af4db0fb70a k8s.gcr.io/sig-storage/csi-resizer:v1.1.0],SizeBytes:49176472,},ContainerImage{Names:[k8s.gcr.io/sig-storage/hostpathplugin@sha256:232fe80174d60d520d36043103853a1d7ab4b7f3782cf43e45034f04ccda58ce k8s.gcr.io/sig-storage/hostpathplugin:v1.7.1],SizeBytes:29977921,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:a61d309da54641db41fb8f35718f744e9f730d4d0384f8c4b186ddc9f06cbd5f k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.1.0],SizeBytes:19662887,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:19582112,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:2dee3fe5fe861bb66c3a4ac51114f3447a4cd35870e0f2e2b558c7a400d89589 k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.2.0],SizeBytes:18699436,},ContainerImage{Names:[k8s.gcr.io/sig-storage/mock-driver@sha256:a7b517f9e0f42ffade802eef9cefa271372386b85f55b702b493241e58459793 k8s.gcr.io/sig-storage/mock-driver:v4.1.0],SizeBytes:17680993,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:17611980,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:529be2c9770add0cdd0c989115222ea9fc1be430c11095eb9f6dafcf98a36e2b k8s.gcr.io/sig-storage/livenessprobe:v2.4.0],SizeBytes:17175488,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:13616070e3f29de4417eee434a8ef472221c9e51b3d037b5a6b46cef08eb7443 k8s.gcr.io/e2e-test-images/nginx:1.14-2],SizeBytes:16032814,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nonewprivs@sha256:8ac1264691820febacf3aea5d152cbde6d10685731ec14966a9401c6f47a68ac k8s.gcr.io/e2e-test-images/nonewprivs:1.3],SizeBytes:7107254,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf k8s.gcr.io/e2e-test-images/busybox:1.29-2],SizeBytes:1154361,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:39e1e963e5310e9c313bad51523be012ede7b35bb9316517d19089a010356592 k8s.gcr.io/e2e-test-images/busybox:1.29-1],SizeBytes:1154361,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db k8s.gcr.io/pause:3.6 registry.k8s.io/pause:3.6],SizeBytes:682696,},},VolumesInUse:[kubernetes.io/csi/ebs.csi.aws.com^vol-01e8a4a8ff1da0e10],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-01e8a4a8ff1da0e10,DevicePath:,},},Config:nil,},} Jan 9 23:11:14.623: INFO: Logging kubelet events for node ip-172-20-32-67.ca-central-1.compute.internal Jan 9 23:11:14.657: INFO: Logging pods the kubelet thinks is on node ip-172-20-32-67.ca-central-1.compute.internal Jan 9 23:11:14.696: INFO: coredns-867df8f45c-sckrz started at 2023-01-09 22:39:23 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:14.696: INFO: Container coredns ready: true, restart count 0 Jan 9 23:11:14.696: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-44ccv started at 2023-01-09 22:54:43 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:14.696: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:11:14.696: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-fzrxx started at 2023-01-09 22:54:43 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:14.696: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:11:14.696: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-mlcl4 started at 2023-01-09 22:54:43 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:14.696: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:11:14.696: INFO: netserver-0 started at 2023-01-09 23:07:18 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:14.696: INFO: Container webserver ready: false, restart count 0 Jan 9 23:11:14.696: INFO: netserver-0 started at 2023-01-09 23:08:45 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:14.696: INFO: Container webserver ready: false, restart count 0 Jan 9 23:11:14.696: INFO: ebs-csi-node-lc4rb started at 2023-01-09 22:38:46 +0000 UTC (0+3 container statuses recorded) Jan 9 23:11:14.696: INFO: Container ebs-plugin ready: true, restart count 0 Jan 9 23:11:14.696: INFO: Container liveness-probe ready: true, restart count 0 Jan 9 23:11:14.696: INFO: Container node-driver-registrar ready: true, restart count 0 Jan 9 23:11:14.696: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-5pr42 started at 2023-01-09 22:54:43 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:14.696: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:11:14.696: INFO: inline-volume-tester-5g6j6 started at 2023-01-09 22:55:56 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:14.696: INFO: Container csi-volume-tester ready: true, restart count 0 Jan 9 23:11:14.696: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-sddjv started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:14.696: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:11:14.696: INFO: csi-hostpathplugin-0 started at 2023-01-09 23:05:41 +0000 UTC (0+7 container statuses recorded) Jan 9 23:11:14.696: INFO: Container csi-attacher ready: false, restart count 0 Jan 9 23:11:14.696: INFO: Container csi-provisioner ready: false, restart count 0 Jan 9 23:11:14.696: INFO: Container csi-resizer ready: false, restart count 0 Jan 9 23:11:14.696: INFO: Container csi-snapshotter ready: false, restart count 0 Jan 9 23:11:14.696: INFO: Container hostpath ready: false, restart count 0 Jan 9 23:11:14.696: INFO: Container liveness-probe ready: false, restart count 0 Jan 9 23:11:14.696: INFO: Container node-driver-registrar ready: false, restart count 0 Jan 9 23:11:14.696: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-jb6kp started at 2023-01-09 22:54:43 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:14.696: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:11:14.696: INFO: cilium-dvsfs started at 2023-01-09 22:38:46 +0000 UTC (1+1 container statuses recorded) Jan 9 23:11:14.696: INFO: Init container clean-cilium-state ready: true, restart count 0 Jan 9 23:11:14.696: INFO: Container cilium-agent ready: true, restart count 0 Jan 9 23:11:14.696: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-gjc2h started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:14.696: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:11:14.697: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-8vzqv started at 2023-01-09 22:54:43 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:14.697: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:11:14.697: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-hwbbn started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:14.697: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:11:14.697: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-fr98x started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:14.697: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:11:14.839: INFO: Latency metrics for node ip-172-20-32-67.ca-central-1.compute.internal Jan 9 23:11:14.839: INFO: Logging node info for node ip-172-20-37-64.ca-central-1.compute.internal Jan 9 23:11:14.870: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-37-64.ca-central-1.compute.internal d817613a-0fe5-40c0-a5e4-7b85b9f8506b 46487 0 2023-01-09 22:36:54 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:c5.large beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ca-central-1 failure-domain.beta.kubernetes.io/zone:ca-central-1a kops.k8s.io/kops-controller-pki: kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-37-64.ca-central-1.compute.internal kubernetes.io/os:linux kubernetes.io/role:master node-role.kubernetes.io/control-plane: node-role.kubernetes.io/master: node.kubernetes.io/exclude-from-external-load-balancers: node.kubernetes.io/instance-type:c5.large topology.ebs.csi.aws.com/zone:ca-central-1a topology.kubernetes.io/region:ca-central-1 topology.kubernetes.io/zone:ca-central-1a] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-06217673764fdafeb"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kubelet Update v1 2023-01-09 22:36:54 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {protokube Update v1 2023-01-09 22:37:19 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kops.k8s.io/kops-controller-pki":{},"f:node-role.kubernetes.io/control-plane":{},"f:node.kubernetes.io/exclude-from-external-load-balancers":{}}}} } {kops-controller Update v1 2023-01-09 22:37:21 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/master":{}}}} } {kube-controller-manager Update v1 2023-01-09 22:37:35 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.0.0/24\"":{}},"f:taints":{}}} } {kubelet Update v1 2023-01-09 22:38:39 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.0.0/24,DoNotUseExternalID:,ProviderID:aws:///ca-central-1a/i-06217673764fdafeb,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/master,Value:,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[100.96.0.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3862904832 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3758047232 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-01-09 23:09:24 +0000 UTC,LastTransitionTime:2023-01-09 22:36:48 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-01-09 23:09:24 +0000 UTC,LastTransitionTime:2023-01-09 22:36:48 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-01-09 23:09:24 +0000 UTC,LastTransitionTime:2023-01-09 22:36:48 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-01-09 23:09:24 +0000 UTC,LastTransitionTime:2023-01-09 22:37:35 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.37.64,},NodeAddress{Type:ExternalIP,Address:3.96.216.165,},NodeAddress{Type:Hostname,Address:ip-172-20-37-64.ca-central-1.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-37-64.ca-central-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-3-96-216-165.ca-central-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec27a483f61319a98e5ba2ee2d59e6a3,SystemUUID:ec27a483-f613-19a9-8e5b-a2ee2d59e6a3,BootID:9b41e339-1aaa-4374-8f11-8d34e12cd62a,KernelVersion:5.15.0-1026-aws,OSImage:Ubuntu 22.04.1 LTS,ContainerRuntimeVersion:docker://20.10.17,KubeletVersion:v1.23.15,KubeProxyVersion:v1.23.15,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[registry.k8s.io/etcdadm/etcd-manager@sha256:66a453db625abb268f4b3bbefc5a34a171d81e6e8796cecca54cfd71775c77c4 registry.k8s.io/etcdadm/etcd-manager:v3.0.20221209],SizeBytes:662209237,},ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:456569528,},ContainerImage{Names:[k8s.gcr.io/kube-apiserver-amd64:v1.23.15 registry.k8s.io/kube-apiserver-amd64:v1.23.15],SizeBytes:135240444,},ContainerImage{Names:[k8s.gcr.io/kube-controller-manager-amd64:v1.23.15 registry.k8s.io/kube-controller-manager-amd64:v1.23.15],SizeBytes:125046097,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.23.15 registry.k8s.io/kube-proxy-amd64:v1.23.15],SizeBytes:112336036,},ContainerImage{Names:[quay.io/cilium/operator@sha256:a6d24a006a6b92967ac90786b49bc1ac26e5477cf028cd1186efcfc2466484db quay.io/cilium/operator:v1.12.5],SizeBytes:110016956,},ContainerImage{Names:[registry.k8s.io/kops/kops-controller:1.26.0-beta.1],SizeBytes:104212800,},ContainerImage{Names:[registry.k8s.io/kops/dns-controller:1.26.0-beta.1],SizeBytes:102566451,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:10e231cf07c82dfab5141c1afe548dc734e5fc1f67665ec3982a325d1bd31a9a registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.0],SizeBytes:94471559,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:122bfb8c1edabb3c0edd63f06523e6940d958d19b3957dc7b1d6f81e9f1f6119 registry.k8s.io/sig-storage/csi-provisioner:v3.1.0],SizeBytes:57736070,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:9ebbf9f023e7b41ccee3d52afe39a89e3ddacdbb69269d583abfc25847cfd9e4 registry.k8s.io/sig-storage/csi-resizer:v1.4.0],SizeBytes:55464778,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:8b9c313c05f54fb04f8d430896f5f5904b6cb157df261501b29adc04d2b2dc7b registry.k8s.io/sig-storage/csi-attacher:v3.4.0],SizeBytes:54848595,},ContainerImage{Names:[k8s.gcr.io/kube-scheduler-amd64:v1.23.15 registry.k8s.io/kube-scheduler-amd64:v1.23.15],SizeBytes:53521430,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:19582112,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:17611980,},ContainerImage{Names:[registry.k8s.io/kops/kube-apiserver-healthcheck:1.26.0-beta.1],SizeBytes:8787111,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:682696,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Jan 9 23:11:14.871: INFO: Logging kubelet events for node ip-172-20-37-64.ca-central-1.compute.internal Jan 9 23:11:14.904: INFO: Logging pods the kubelet thinks is on node ip-172-20-37-64.ca-central-1.compute.internal Jan 9 23:11:14.941: INFO: ebs-csi-controller-5d8d7cc949-8wxx2 started at 2023-01-09 22:37:19 +0000 UTC (0+5 container statuses recorded) Jan 9 23:11:14.941: INFO: Container csi-attacher ready: true, restart count 0 Jan 9 23:11:14.941: INFO: Container csi-provisioner ready: true, restart count 0 Jan 9 23:11:14.941: INFO: Container csi-resizer ready: true, restart count 0 Jan 9 23:11:14.941: INFO: Container ebs-plugin ready: true, restart count 0 Jan 9 23:11:14.941: INFO: Container liveness-probe ready: true, restart count 0 Jan 9 23:11:14.941: INFO: dns-controller-867784b75c-sxxcl started at 2023-01-09 22:37:20 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:14.941: INFO: Container dns-controller ready: true, restart count 0 Jan 9 23:11:14.941: INFO: kube-scheduler-ip-172-20-37-64.ca-central-1.compute.internal started at 2023-01-09 22:36:15 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:14.941: INFO: Container kube-scheduler ready: true, restart count 0 Jan 9 23:11:14.941: INFO: etcd-manager-events-ip-172-20-37-64.ca-central-1.compute.internal started at 2023-01-09 22:36:15 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:14.941: INFO: Container etcd-manager ready: true, restart count 0 Jan 9 23:11:14.941: INFO: etcd-manager-main-ip-172-20-37-64.ca-central-1.compute.internal started at 2023-01-09 22:36:15 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:14.941: INFO: Container etcd-manager ready: true, restart count 0 Jan 9 23:11:14.941: INFO: ebs-csi-node-8q4pg started at 2023-01-09 22:37:17 +0000 UTC (0+3 container statuses recorded) Jan 9 23:11:14.941: INFO: Container ebs-plugin ready: true, restart count 0 Jan 9 23:11:14.941: INFO: Container liveness-probe ready: true, restart count 0 Jan 9 23:11:14.941: INFO: Container node-driver-registrar ready: true, restart count 0 Jan 9 23:11:14.941: INFO: cilium-q7lrg started at 2023-01-09 22:37:17 +0000 UTC (1+1 container statuses recorded) Jan 9 23:11:14.941: INFO: Init container clean-cilium-state ready: true, restart count 0 Jan 9 23:11:14.942: INFO: Container cilium-agent ready: true, restart count 0 Jan 9 23:11:14.942: INFO: cilium-operator-c7bfc9f44-w4nqz started at 2023-01-09 22:37:19 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:14.942: INFO: Container cilium-operator ready: true, restart count 0 Jan 9 23:11:14.942: INFO: kube-apiserver-ip-172-20-37-64.ca-central-1.compute.internal started at 2023-01-09 22:36:15 +0000 UTC (0+2 container statuses recorded) Jan 9 23:11:14.942: INFO: Container healthcheck ready: true, restart count 0 Jan 9 23:11:14.942: INFO: Container kube-apiserver ready: true, restart count 1 Jan 9 23:11:14.942: INFO: kube-controller-manager-ip-172-20-37-64.ca-central-1.compute.internal started at 2023-01-09 22:36:15 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:14.942: INFO: Container kube-controller-manager ready: true, restart count 2 Jan 9 23:11:14.942: INFO: kops-controller-cqrv8 started at 2023-01-09 22:37:19 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:14.942: INFO: Container kops-controller ready: true, restart count 0 Jan 9 23:11:15.057: INFO: Latency metrics for node ip-172-20-37-64.ca-central-1.compute.internal Jan 9 23:11:15.057: INFO: Logging node info for node ip-172-20-55-216.ca-central-1.compute.internal Jan 9 23:11:15.089: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-55-216.ca-central-1.compute.internal 2ceff781-505a-4706-ad8b-e38e37f72571 47173 0 2023-01-09 22:38:47 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ca-central-1 failure-domain.beta.kubernetes.io/zone:ca-central-1a kubelet_cleanup:true kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-55-216.ca-central-1.compute.internal kubernetes.io/os:linux kubernetes.io/role:node node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:ca-central-1a topology.hostpath.csi/node:ip-172-20-55-216.ca-central-1.compute.internal topology.kubernetes.io/region:ca-central-1 topology.kubernetes.io/zone:ca-central-1a] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-0c78741e44993ae7f"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kops-controller Update v1 2023-01-09 22:38:47 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}} } {kube-controller-manager Update v1 2023-01-09 22:38:47 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.4.0/24\"":{}}}} } {kubelet Update v1 2023-01-09 22:38:47 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {e2e.test Update v1 2023-01-09 22:54:42 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubelet_cleanup":{}}}} } {kube-controller-manager Update v1 2023-01-09 22:56:45 +0000 UTC FieldsV1 {"f:status":{"f:volumesAttached":{}}} status} {kubelet Update v1 2023-01-09 22:56:51 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:capacity":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.4.0/24,DoNotUseExternalID:,ProviderID:aws:///ca-central-1a/i-0c78741e44993ae7f,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.4.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051644416 0} {<nil>} 3956684Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946786816 0} {<nil>} 3854284Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-01-09 23:11:11 +0000 UTC,LastTransitionTime:2023-01-09 22:38:47 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-01-09 23:11:11 +0000 UTC,LastTransitionTime:2023-01-09 22:38:47 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-01-09 23:11:11 +0000 UTC,LastTransitionTime:2023-01-09 22:38:47 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-01-09 23:11:11 +0000 UTC,LastTransitionTime:2023-01-09 22:39:07 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.55.216,},NodeAddress{Type:ExternalIP,Address:15.222.60.56,},NodeAddress{Type:Hostname,Address:ip-172-20-55-216.ca-central-1.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-55-216.ca-central-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-15-222-60-56.ca-central-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec249e6a3570a28ebd4276206d3ae523,SystemUUID:ec249e6a-3570-a28e-bd42-76206d3ae523,BootID:d58dcef2-849d-4807-9ba5-0438ebaa05b9,KernelVersion:5.15.0-1026-aws,OSImage:Ubuntu 22.04.1 LTS,ContainerRuntimeVersion:docker://20.10.17,KubeletVersion:v1.23.15,KubeProxyVersion:v1.23.15,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:456569528,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/jessie-dnsutils@sha256:11e6a66017ba4e4b938c1612b7a54a3befcefd354796c04e1dba76873a13518e k8s.gcr.io/e2e-test-images/jessie-dnsutils:1.5],SizeBytes:253371792,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:20f25f275d46aa728f7615a1ccc19c78b2ed89435bf943a44b339f70f45508e6 k8s.gcr.io/e2e-test-images/httpd:2.4.39-2],SizeBytes:126894770,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:7e8bdd271312fd25fc5ff5a8f04727be84044eb3d7d8d03611972a6752e2e11e k8s.gcr.io/e2e-test-images/agnhost:2.39],SizeBytes:126872991,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nautilus@sha256:99c0d6f1ad24a1aa1905d9c6534d193f268f7b23f9add2ae6bb41f31094bdd5c k8s.gcr.io/e2e-test-images/nautilus:1.5],SizeBytes:124758741,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:1b9d1b2f36cb2dbee1960e82a9344aeb11bd4c4c03abf5e1853e0559c23855e3 k8s.gcr.io/e2e-test-images/httpd:2.4.38-2],SizeBytes:123781643,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.23.15 registry.k8s.io/kube-proxy-amd64:v1.23.15],SizeBytes:112336036,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:10e231cf07c82dfab5141c1afe548dc734e5fc1f67665ec3982a325d1bd31a9a registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.0],SizeBytes:94471559,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:c8e03f60afa90a28e4bb6ec9a8d0fc36d89de4b7475cf2d613afa793ec969fe0 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.0],SizeBytes:56382329,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:4e74c0492bceddc598de1c90cc5bc14dcda94cb49fa9c5bad9d117c4834b5e08 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.1],SizeBytes:56379269,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:36c31f7e1f433c9634d24f876353e8646246d81a03c4e351202c2644daff1620 k8s.gcr.io/sig-storage/csi-resizer:v1.2.0],SizeBytes:53956308,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:ed98431376c9e944e19a465fe8ea944806714dd95416a0821096c78d66b579bd k8s.gcr.io/sig-storage/csi-snapshotter:v4.1.1],SizeBytes:53531990,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:c5be65d6679efabb969d9b019300d187437ae876f992c40911fd2892bbef3b36 k8s.gcr.io/sig-storage/csi-attacher:v3.2.0],SizeBytes:53500826,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:60ab9b3e6a030d3038c87c0d6bca2930f58d1d72823e6a4af09767dc83b696a2 k8s.gcr.io/sig-storage/csi-attacher:v3.2.1],SizeBytes:53499570,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:7a5ba58a44e0d749e0767e4e37315bcf6a61f33ce3185c1991848af4db0fb70a k8s.gcr.io/sig-storage/csi-resizer:v1.1.0],SizeBytes:49176472,},ContainerImage{Names:[k8s.gcr.io/sig-storage/hostpathplugin@sha256:232fe80174d60d520d36043103853a1d7ab4b7f3782cf43e45034f04ccda58ce k8s.gcr.io/sig-storage/hostpathplugin:v1.7.1],SizeBytes:29977921,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:a61d309da54641db41fb8f35718f744e9f730d4d0384f8c4b186ddc9f06cbd5f k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.1.0],SizeBytes:19662887,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:19582112,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:2dee3fe5fe861bb66c3a4ac51114f3447a4cd35870e0f2e2b558c7a400d89589 k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.2.0],SizeBytes:18699436,},ContainerImage{Names:[k8s.gcr.io/sig-storage/mock-driver@sha256:a7b517f9e0f42ffade802eef9cefa271372386b85f55b702b493241e58459793 k8s.gcr.io/sig-storage/mock-driver:v4.1.0],SizeBytes:17680993,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:17611980,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:529be2c9770add0cdd0c989115222ea9fc1be430c11095eb9f6dafcf98a36e2b k8s.gcr.io/sig-storage/livenessprobe:v2.4.0],SizeBytes:17175488,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:13616070e3f29de4417eee434a8ef472221c9e51b3d037b5a6b46cef08eb7443 k8s.gcr.io/e2e-test-images/nginx:1.14-2],SizeBytes:16032814,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf k8s.gcr.io/e2e-test-images/busybox:1.29-2],SizeBytes:1154361,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db k8s.gcr.io/pause:3.6 registry.k8s.io/pause:3.6],SizeBytes:682696,},},VolumesInUse:[kubernetes.io/csi/ebs.csi.aws.com^vol-0a0867bd913d60721 kubernetes.io/csi/ebs.csi.aws.com^vol-0c8e4ae9d44bf0644],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-0a0867bd913d60721,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-0c8e4ae9d44bf0644,DevicePath:,},},Config:nil,},} Jan 9 23:11:15.090: INFO: Logging kubelet events for node ip-172-20-55-216.ca-central-1.compute.internal Jan 9 23:11:15.124: INFO: Logging pods the kubelet thinks is on node ip-172-20-55-216.ca-central-1.compute.internal Jan 9 23:11:15.166: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-qglk4 started at 2023-01-09 22:54:43 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:15.166: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:11:15.166: INFO: netserver-1 started at 2023-01-09 23:08:45 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:15.167: INFO: Container webserver ready: false, restart count 0 Jan 9 23:11:15.167: INFO: cilium-dqpw4 started at 2023-01-09 22:38:48 +0000 UTC (1+1 container statuses recorded) Jan 9 23:11:15.167: INFO: Init container clean-cilium-state ready: true, restart count 0 Jan 9 23:11:15.167: INFO: Container cilium-agent ready: true, restart count 0 Jan 9 23:11:15.167: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-b7bxr started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:15.167: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:11:15.167: INFO: inline-volume-tester-7mx4s started at 2023-01-09 22:56:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:15.167: INFO: Container csi-volume-tester ready: false, restart count 0 Jan 9 23:11:15.167: INFO: hostexec-ip-172-20-55-216.ca-central-1.compute.internal-ddnj4 started at 2023-01-09 23:10:19 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:15.167: INFO: Container agnhost-container ready: true, restart count 0 Jan 9 23:11:15.167: INFO: inline-volume-tester2-9qph2 started at 2023-01-09 23:10:30 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:15.167: INFO: Container csi-volume-tester ready: false, restart count 0 Jan 9 23:11:15.167: INFO: netserver-1 started at 2023-01-09 23:07:18 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:15.167: INFO: Container webserver ready: false, restart count 0 Jan 9 23:11:15.167: INFO: fail-once-non-local-mh6qc started at 2023-01-09 22:56:53 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:15.167: INFO: Container c ready: false, restart count 0 Jan 9 23:11:15.167: INFO: pod-8ade382c-2fb2-4f02-8d50-57f7b6a15696 started at 2023-01-09 23:06:54 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:15.167: INFO: Container test-container ready: false, restart count 0 Jan 9 23:11:15.167: INFO: csi-mockplugin-0 started at 2023-01-09 23:10:49 +0000 UTC (0+3 container statuses recorded) Jan 9 23:11:15.167: INFO: Container csi-provisioner ready: false, restart count 0 Jan 9 23:11:15.167: INFO: Container driver-registrar ready: false, restart count 0 Jan 9 23:11:15.167: INFO: Container mock ready: false, restart count 0 Jan 9 23:11:15.167: INFO: startup-ff2f8ca6-ae0a-4fae-9483-a7d80b37fdc1 started at 2023-01-09 23:07:24 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:15.167: INFO: Container busybox ready: false, restart count 0 Jan 9 23:11:15.167: INFO: pod-90864581-af4c-4524-a3f6-27ae0e060f71 started at 2023-01-09 23:10:24 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:15.167: INFO: Container write-pod ready: false, restart count 0 Jan 9 23:11:15.167: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-mcd88 started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:15.167: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:11:15.167: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-6nqvb started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:15.167: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:11:15.167: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-8kb86 started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:15.167: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:11:15.167: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-vwd6k started at 2023-01-09 22:54:43 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:15.167: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:11:15.167: INFO: image-pull-test99b7b050-d558-462d-a744-2f374728e347 started at 2023-01-09 23:07:56 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:15.167: INFO: Container image-pull-test ready: false, restart count 0 Jan 9 23:11:15.167: INFO: hostpath-symlink-prep-provisioning-3662 started at 2023-01-09 23:10:57 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:15.167: INFO: Container init-volume-provisioning-3662 ready: false, restart count 0 Jan 9 23:11:15.167: INFO: fail-once-non-local-nx7cr started at 2023-01-09 22:56:53 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:15.167: INFO: Container c ready: false, restart count 0 Jan 9 23:11:15.167: INFO: ebs-csi-node-psq97 started at 2023-01-09 22:38:48 +0000 UTC (0+3 container statuses recorded) Jan 9 23:11:15.167: INFO: Container ebs-plugin ready: true, restart count 0 Jan 9 23:11:15.167: INFO: Container liveness-probe ready: true, restart count 0 Jan 9 23:11:15.167: INFO: Container node-driver-registrar ready: true, restart count 0 Jan 9 23:11:15.167: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-qj2bw started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:15.167: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:11:15.167: INFO: pod-secrets-44fccaea-993f-49ad-91c1-2dc3743d909d started at 2023-01-09 23:10:24 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:15.167: INFO: Container secret-volume-test ready: false, restart count 0 Jan 9 23:11:15.167: INFO: fail-once-local-gvnwq started at 2023-01-09 23:10:38 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:15.167: INFO: Container c ready: false, restart count 1 Jan 9 23:11:15.167: INFO: csi-mockplugin-attacher-0 started at 2023-01-09 23:10:49 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:15.167: INFO: Container csi-attacher ready: false, restart count 0 Jan 9 23:11:15.167: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-6ks7b started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:15.167: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:11:15.167: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-72kwd started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:15.167: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:11:15.167: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-krwzl started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:15.167: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:11:15.167: INFO: pod1 started at 2023-01-09 23:11:07 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:15.167: INFO: Container agnhost-container ready: false, restart count 0 Jan 9 23:11:15.335: INFO: Latency metrics for node ip-172-20-55-216.ca-central-1.compute.internal Jan 9 23:11:15.336: INFO: Logging node info for node ip-172-20-57-251.ca-central-1.compute.internal Jan 9 23:11:15.367: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-57-251.ca-central-1.compute.internal eaaaf6eb-cbd5-4763-9c39-fb0e1a4ce44a 47193 0 2023-01-09 22:38:45 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ca-central-1 failure-domain.beta.kubernetes.io/zone:ca-central-1a kubelet_cleanup:true kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-57-251.ca-central-1.compute.internal kubernetes.io/os:linux kubernetes.io/role:node node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:ca-central-1a topology.hostpath.csi/node:ip-172-20-57-251.ca-central-1.compute.internal topology.kubernetes.io/region:ca-central-1 topology.kubernetes.io/zone:ca-central-1a] map[csi.volume.kubernetes.io/nodeid:{"csi-mock-csi-mock-volumes-9402":"csi-mock-csi-mock-volumes-9402","ebs.csi.aws.com":"i-0513e7d6e5eba8eec"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kops-controller Update v1 2023-01-09 22:38:45 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}} } {kube-controller-manager Update v1 2023-01-09 22:38:45 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.2.0/24\"":{}}}} } {kubelet Update v1 2023-01-09 22:38:45 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {e2e.test Update v1 2023-01-09 22:54:42 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubelet_cleanup":{}}}} } {kube-controller-manager Update v1 2023-01-09 23:00:23 +0000 UTC FieldsV1 {"f:status":{"f:volumesAttached":{}}} status} {kubelet Update v1 2023-01-09 23:00:27 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.2.0/24,DoNotUseExternalID:,ProviderID:aws:///ca-central-1a/i-0513e7d6e5eba8eec,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.2.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051652608 0} {<nil>} 3956692Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946795008 0} {<nil>} 3854292Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-01-09 23:11:14 +0000 UTC,LastTransitionTime:2023-01-09 22:38:45 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-01-09 23:11:14 +0000 UTC,LastTransitionTime:2023-01-09 22:38:45 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-01-09 23:11:14 +0000 UTC,LastTransitionTime:2023-01-09 22:38:45 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-01-09 23:11:14 +0000 UTC,LastTransitionTime:2023-01-09 22:39:05 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.57.251,},NodeAddress{Type:ExternalIP,Address:3.96.144.209,},NodeAddress{Type:Hostname,Address:ip-172-20-57-251.ca-central-1.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-57-251.ca-central-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-3-96-144-209.ca-central-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2f0d7ba95b37a96a1d9360f6071d84,SystemUUID:ec2f0d7b-a95b-37a9-6a1d-9360f6071d84,BootID:67865282-1911-4e38-8675-a7b78d5ae45e,KernelVersion:5.15.0-1026-aws,OSImage:Ubuntu 22.04.1 LTS,ContainerRuntimeVersion:docker://20.10.17,KubeletVersion:v1.23.15,KubeProxyVersion:v1.23.15,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:456569528,},ContainerImage{Names:[k8s.gcr.io/etcd@sha256:dd75ec974b0a2a6f6bb47001ba09207976e625db898d1b16735528c009cb171c k8s.gcr.io/etcd:3.5.6-0],SizeBytes:299475478,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/jessie-dnsutils@sha256:11e6a66017ba4e4b938c1612b7a54a3befcefd354796c04e1dba76873a13518e k8s.gcr.io/e2e-test-images/jessie-dnsutils:1.5],SizeBytes:253371792,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:20f25f275d46aa728f7615a1ccc19c78b2ed89435bf943a44b339f70f45508e6 k8s.gcr.io/e2e-test-images/httpd:2.4.39-2],SizeBytes:126894770,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:7e8bdd271312fd25fc5ff5a8f04727be84044eb3d7d8d03611972a6752e2e11e k8s.gcr.io/e2e-test-images/agnhost:2.39],SizeBytes:126872991,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nautilus@sha256:99c0d6f1ad24a1aa1905d9c6534d193f268f7b23f9add2ae6bb41f31094bdd5c k8s.gcr.io/e2e-test-images/nautilus:1.5],SizeBytes:124758741,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:1b9d1b2f36cb2dbee1960e82a9344aeb11bd4c4c03abf5e1853e0559c23855e3 k8s.gcr.io/e2e-test-images/httpd:2.4.38-2],SizeBytes:123781643,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.23.15 registry.k8s.io/kube-proxy-amd64:v1.23.15],SizeBytes:112336036,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:10e231cf07c82dfab5141c1afe548dc734e5fc1f67665ec3982a325d1bd31a9a registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.0],SizeBytes:94471559,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/sample-apiserver@sha256:f9c93b92b6ff750b41a93c4e4fe0bfe384597aeb841e2539d5444815c55b2d8f k8s.gcr.io/e2e-test-images/sample-apiserver:1.17.5],SizeBytes:56504541,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:c8e03f60afa90a28e4bb6ec9a8d0fc36d89de4b7475cf2d613afa793ec969fe0 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.0],SizeBytes:56382329,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:4e74c0492bceddc598de1c90cc5bc14dcda94cb49fa9c5bad9d117c4834b5e08 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.1],SizeBytes:56379269,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:36c31f7e1f433c9634d24f876353e8646246d81a03c4e351202c2644daff1620 k8s.gcr.io/sig-storage/csi-resizer:v1.2.0],SizeBytes:53956308,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:ed98431376c9e944e19a465fe8ea944806714dd95416a0821096c78d66b579bd k8s.gcr.io/sig-storage/csi-snapshotter:v4.1.1],SizeBytes:53531990,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:c5be65d6679efabb969d9b019300d187437ae876f992c40911fd2892bbef3b36 k8s.gcr.io/sig-storage/csi-attacher:v3.2.0],SizeBytes:53500826,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:60ab9b3e6a030d3038c87c0d6bca2930f58d1d72823e6a4af09767dc83b696a2 k8s.gcr.io/sig-storage/csi-attacher:v3.2.1],SizeBytes:53499570,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:7a5ba58a44e0d749e0767e4e37315bcf6a61f33ce3185c1991848af4db0fb70a k8s.gcr.io/sig-storage/csi-resizer:v1.1.0],SizeBytes:49176472,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/regression-issue-74839@sha256:b4f1d8d61bdad84bd50442d161d5460e4019d53e989b64220fdbc62fc87d76bf k8s.gcr.io/e2e-test-images/regression-issue-74839:1.2],SizeBytes:44576952,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nonroot@sha256:b9e2958a3dd879e3cf11142228c6d073d0fc4ea2e857c3be6f4fb0ab5fb2c937 k8s.gcr.io/e2e-test-images/nonroot:1.2],SizeBytes:42321438,},ContainerImage{Names:[k8s.gcr.io/sig-storage/hostpathplugin@sha256:232fe80174d60d520d36043103853a1d7ab4b7f3782cf43e45034f04ccda58ce k8s.gcr.io/sig-storage/hostpathplugin:v1.7.1],SizeBytes:29977921,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:a61d309da54641db41fb8f35718f744e9f730d4d0384f8c4b186ddc9f06cbd5f k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.1.0],SizeBytes:19662887,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:19582112,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:2dee3fe5fe861bb66c3a4ac51114f3447a4cd35870e0f2e2b558c7a400d89589 k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.2.0],SizeBytes:18699436,},ContainerImage{Names:[k8s.gcr.io/sig-storage/mock-driver@sha256:a7b517f9e0f42ffade802eef9cefa271372386b85f55b702b493241e58459793 k8s.gcr.io/sig-storage/mock-driver:v4.1.0],SizeBytes:17680993,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:17611980,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:529be2c9770add0cdd0c989115222ea9fc1be430c11095eb9f6dafcf98a36e2b k8s.gcr.io/sig-storage/livenessprobe:v2.4.0],SizeBytes:17175488,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:13616070e3f29de4417eee434a8ef472221c9e51b3d037b5a6b46cef08eb7443 k8s.gcr.io/e2e-test-images/nginx:1.14-2],SizeBytes:16032814,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nonewprivs@sha256:8ac1264691820febacf3aea5d152cbde6d10685731ec14966a9401c6f47a68ac k8s.gcr.io/e2e-test-images/nonewprivs:1.3],SizeBytes:7107254,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf k8s.gcr.io/e2e-test-images/busybox:1.29-2],SizeBytes:1154361,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:39e1e963e5310e9c313bad51523be012ede7b35bb9316517d19089a010356592 k8s.gcr.io/e2e-test-images/busybox:1.29-1],SizeBytes:1154361,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db k8s.gcr.io/pause:3.6 registry.k8s.io/pause:3.6],SizeBytes:682696,},},VolumesInUse:[kubernetes.io/csi/csi-mock-csi-mock-volumes-9402^4 kubernetes.io/csi/ebs.csi.aws.com^vol-02daff5e76e72db98],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-02daff5e76e72db98,DevicePath:,},},Config:nil,},} Jan 9 23:11:15.368: INFO: Logging kubelet events for node ip-172-20-57-251.ca-central-1.compute.internal Jan 9 23:11:15.402: INFO: Logging pods the kubelet thinks is on node ip-172-20-57-251.ca-central-1.compute.internal Jan 9 23:11:15.443: INFO: cilium-6p9jz started at 2023-01-09 22:38:46 +0000 UTC (1+1 container statuses recorded) Jan 9 23:11:15.443: INFO: Init container clean-cilium-state ready: true, restart count 0 Jan 9 23:11:15.443: INFO: Container cilium-agent ready: true, restart count 0 Jan 9 23:11:15.443: INFO: hostpath-symlink-prep-volume-6746 started at 2023-01-09 23:07:44 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:15.443: INFO: Container init-volume-volume-6746 ready: false, restart count 0 Jan 9 23:11:15.443: INFO: pvc-volume-tester-2vlf5 started at 2023-01-09 23:06:05 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:15.443: INFO: Container volume-tester ready: false, restart count 0 Jan 9 23:11:15.443: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-wvflq started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:15.443: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:11:15.443: INFO: hostexec-ip-172-20-57-251.ca-central-1.compute.internal-f9vsk started at 2023-01-09 23:08:31 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:15.443: INFO: Container agnhost-container ready: true, restart count 0 Jan 9 23:11:15.443: INFO: pod-subpath-test-preprovisionedpv-ssct started at 2023-01-09 23:08:48 +0000 UTC (0+2 container statuses recorded) Jan 9 23:11:15.443: INFO: Container test-container-subpath-preprovisionedpv-ssct ready: false, restart count 0 Jan 9 23:11:15.443: INFO: Container test-container-volume-preprovisionedpv-ssct ready: false, restart count 0 Jan 9 23:11:15.443: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-tvxfq started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:15.443: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:11:15.443: INFO: csi-mockplugin-0 started at 2023-01-09 23:00:20 +0000 UTC (0+3 container statuses recorded) Jan 9 23:11:15.443: INFO: Container csi-provisioner ready: true, restart count 0 Jan 9 23:11:15.443: INFO: Container driver-registrar ready: true, restart count 0 Jan 9 23:11:15.443: INFO: Container mock ready: true, restart count 0 Jan 9 23:11:15.443: INFO: fail-once-local-szrdn started at 2023-01-09 23:10:38 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:15.443: INFO: Container c ready: false, restart count 0 Jan 9 23:11:15.443: INFO: fail-once-local-wj6xl started at 2023-01-09 23:10:57 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:15.443: INFO: Container c ready: false, restart count 0 Jan 9 23:11:15.443: INFO: ebs-csi-node-g4ngc started at 2023-01-09 22:38:46 +0000 UTC (0+3 container statuses recorded) Jan 9 23:11:15.443: INFO: Container ebs-plugin ready: true, restart count 0 Jan 9 23:11:15.443: INFO: Container liveness-probe ready: true, restart count 0 Jan 9 23:11:15.443: INFO: Container node-driver-registrar ready: true, restart count 0 Jan 9 23:11:15.443: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-wmqwf started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:15.443: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:11:15.443: INFO: pod-d16d48d3-63ef-4a76-95b8-d80f8b4aec62 started at 2023-01-09 23:03:17 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:15.443: INFO: Container write-pod ready: true, restart count 0 Jan 9 23:11:15.443: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-m8cfz started at 2023-01-09 22:54:43 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:15.443: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:11:15.443: INFO: netserver-2 started at 2023-01-09 23:07:18 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:15.443: INFO: Container webserver ready: false, restart count 0 Jan 9 23:11:15.443: INFO: csi-mockplugin-attacher-0 started at 2023-01-09 23:00:20 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:15.443: INFO: Container csi-attacher ready: false, restart count 0 Jan 9 23:11:15.443: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-rbqs2 started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:15.443: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:11:15.443: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-96np5 started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:15.443: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:11:15.443: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-f9w29 started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:15.443: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:11:15.443: INFO: csi-mockplugin-0 started at 2023-01-09 23:05:43 +0000 UTC (0+3 container statuses recorded) Jan 9 23:11:15.443: INFO: Container csi-provisioner ready: false, restart count 0 Jan 9 23:11:15.443: INFO: Container driver-registrar ready: false, restart count 0 Jan 9 23:11:15.443: INFO: Container mock ready: false, restart count 0 Jan 9 23:11:15.443: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-zx8gm started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:15.443: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:11:15.443: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-qzf5t started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:15.443: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:11:15.443: INFO: busybox-scheduling-6815563e-4dc7-43f5-a3cd-3e998a31b84b started at 2023-01-09 23:06:39 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:15.443: INFO: Container busybox-scheduling-6815563e-4dc7-43f5-a3cd-3e998a31b84b ready: false, restart count 0 Jan 9 23:11:15.443: INFO: netserver-2 started at 2023-01-09 23:08:45 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:15.443: INFO: Container webserver ready: false, restart count 0 Jan 9 23:11:15.443: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-zvfdw started at 2023-01-09 22:54:43 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:15.443: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:11:15.656: INFO: Latency metrics for node ip-172-20-57-251.ca-central-1.compute.internal Jan 9 23:11:15.656: INFO: Logging node info for node ip-172-20-59-208.ca-central-1.compute.internal Jan 9 23:11:15.689: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-59-208.ca-central-1.compute.internal d061a4d4-9ff6-43fc-89cc-422c61f7441c 45677 0 2023-01-09 22:38:43 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ca-central-1 failure-domain.beta.kubernetes.io/zone:ca-central-1a kubelet_cleanup:true kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-59-208.ca-central-1.compute.internal kubernetes.io/os:linux kubernetes.io/role:node node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:ca-central-1a topology.hostpath.csi/node:ip-172-20-59-208.ca-central-1.compute.internal topology.kubernetes.io/region:ca-central-1 topology.kubernetes.io/zone:ca-central-1a] map[csi.volume.kubernetes.io/nodeid:{"csi-hostpath-ephemeral-7223":"ip-172-20-59-208.ca-central-1.compute.internal","ebs.csi.aws.com":"i-0f00ea8bafbacdd11"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kops-controller Update v1 2023-01-09 22:38:43 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}} } {kube-controller-manager Update v1 2023-01-09 22:38:43 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.1.0/24\"":{}}}} } {kubelet Update v1 2023-01-09 22:38:43 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {e2e.test Update v1 2023-01-09 22:54:42 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubelet_cleanup":{}}}} } {kube-controller-manager Update v1 2023-01-09 23:06:36 +0000 UTC FieldsV1 {"f:status":{"f:volumesAttached":{}}} status} {kubelet Update v1 2023-01-09 23:06:40 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:capacity":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.1.0/24,DoNotUseExternalID:,ProviderID:aws:///ca-central-1a/i-0f00ea8bafbacdd11,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.1.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051652608 0} {<nil>} 3956692Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946795008 0} {<nil>} 3854292Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-01-09 23:06:40 +0000 UTC,LastTransitionTime:2023-01-09 22:38:43 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-01-09 23:06:40 +0000 UTC,LastTransitionTime:2023-01-09 22:38:43 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-01-09 23:06:40 +0000 UTC,LastTransitionTime:2023-01-09 22:38:43 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-01-09 23:06:40 +0000 UTC,LastTransitionTime:2023-01-09 22:39:03 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.59.208,},NodeAddress{Type:ExternalIP,Address:99.79.51.95,},NodeAddress{Type:Hostname,Address:ip-172-20-59-208.ca-central-1.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-59-208.ca-central-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-99-79-51-95.ca-central-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec28d5ecfae27d4bcedd89eca1a365f0,SystemUUID:ec28d5ec-fae2-7d4b-cedd-89eca1a365f0,BootID:560320f7-12c5-44f2-a7f6-a7a40031e204,KernelVersion:5.15.0-1026-aws,OSImage:Ubuntu 22.04.1 LTS,ContainerRuntimeVersion:docker://20.10.17,KubeletVersion:v1.23.15,KubeProxyVersion:v1.23.15,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:456569528,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:20f25f275d46aa728f7615a1ccc19c78b2ed89435bf943a44b339f70f45508e6 k8s.gcr.io/e2e-test-images/httpd:2.4.39-2],SizeBytes:126894770,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:7e8bdd271312fd25fc5ff5a8f04727be84044eb3d7d8d03611972a6752e2e11e k8s.gcr.io/e2e-test-images/agnhost:2.39],SizeBytes:126872991,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:1b9d1b2f36cb2dbee1960e82a9344aeb11bd4c4c03abf5e1853e0559c23855e3 k8s.gcr.io/e2e-test-images/httpd:2.4.38-2],SizeBytes:123781643,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.23.15 registry.k8s.io/kube-proxy-amd64:v1.23.15],SizeBytes:112336036,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:10e231cf07c82dfab5141c1afe548dc734e5fc1f67665ec3982a325d1bd31a9a registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.0],SizeBytes:94471559,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:c8e03f60afa90a28e4bb6ec9a8d0fc36d89de4b7475cf2d613afa793ec969fe0 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.0],SizeBytes:56382329,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:4e74c0492bceddc598de1c90cc5bc14dcda94cb49fa9c5bad9d117c4834b5e08 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.1],SizeBytes:56379269,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:36c31f7e1f433c9634d24f876353e8646246d81a03c4e351202c2644daff1620 k8s.gcr.io/sig-storage/csi-resizer:v1.2.0],SizeBytes:53956308,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:ed98431376c9e944e19a465fe8ea944806714dd95416a0821096c78d66b579bd k8s.gcr.io/sig-storage/csi-snapshotter:v4.1.1],SizeBytes:53531990,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:c5be65d6679efabb969d9b019300d187437ae876f992c40911fd2892bbef3b36 k8s.gcr.io/sig-storage/csi-attacher:v3.2.0],SizeBytes:53500826,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:60ab9b3e6a030d3038c87c0d6bca2930f58d1d72823e6a4af09767dc83b696a2 k8s.gcr.io/sig-storage/csi-attacher:v3.2.1],SizeBytes:53499570,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:017727efcfeb7d053af68e51436ce8e65edbc6ca573720afb4f79c8594036955 registry.k8s.io/coredns/coredns:v1.10.0],SizeBytes:50249337,},ContainerImage{Names:[registry.k8s.io/cpa/cluster-proportional-autoscaler@sha256:68d396900aeaa072c1f27289485fdac29834045a6f3ffe369bf389d830ef572d registry.k8s.io/cpa/cluster-proportional-autoscaler:1.8.6],SizeBytes:47910609,},ContainerImage{Names:[k8s.gcr.io/sig-storage/hostpathplugin@sha256:232fe80174d60d520d36043103853a1d7ab4b7f3782cf43e45034f04ccda58ce k8s.gcr.io/sig-storage/hostpathplugin:v1.7.1],SizeBytes:29977921,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:a61d309da54641db41fb8f35718f744e9f730d4d0384f8c4b186ddc9f06cbd5f k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.1.0],SizeBytes:19662887,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:19582112,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:2dee3fe5fe861bb66c3a4ac51114f3447a4cd35870e0f2e2b558c7a400d89589 k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.2.0],SizeBytes:18699436,},ContainerImage{Names:[k8s.gcr.io/sig-storage/mock-driver@sha256:a7b517f9e0f42ffade802eef9cefa271372386b85f55b702b493241e58459793 k8s.gcr.io/sig-storage/mock-driver:v4.1.0],SizeBytes:17680993,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:17611980,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:529be2c9770add0cdd0c989115222ea9fc1be430c11095eb9f6dafcf98a36e2b k8s.gcr.io/sig-storage/livenessprobe:v2.4.0],SizeBytes:17175488,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:13616070e3f29de4417eee434a8ef472221c9e51b3d037b5a6b46cef08eb7443 k8s.gcr.io/e2e-test-images/nginx:1.14-2],SizeBytes:16032814,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nonewprivs@sha256:8ac1264691820febacf3aea5d152cbde6d10685731ec14966a9401c6f47a68ac k8s.gcr.io/e2e-test-images/nonewprivs:1.3],SizeBytes:7107254,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf k8s.gcr.io/e2e-test-images/busybox:1.29-2],SizeBytes:1154361,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:39e1e963e5310e9c313bad51523be012ede7b35bb9316517d19089a010356592 k8s.gcr.io/e2e-test-images/busybox:1.29-1],SizeBytes:1154361,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db k8s.gcr.io/pause:3.6 registry.k8s.io/pause:3.6],SizeBytes:682696,},},VolumesInUse:[kubernetes.io/csi/csi-hostpath-ephemeral-7223^43b8adc9-9072-11ed-b4b6-0a454e7330ed kubernetes.io/csi/csi-hostpath-ephemeral-7223^43b8d4d5-9072-11ed-b4b6-0a454e7330ed],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/csi-hostpath-ephemeral-7223^43b8d4d5-9072-11ed-b4b6-0a454e7330ed,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/csi-hostpath-ephemeral-7223^43b8adc9-9072-11ed-b4b6-0a454e7330ed,DevicePath:,},},Config:nil,},} Jan 9 23:11:15.689: INFO: Logging kubelet events for node ip-172-20-59-208.ca-central-1.compute.internal Jan 9 23:11:15.759: INFO: Logging pods the kubelet thinks is on node ip-172-20-59-208.ca-central-1.compute.internal Jan 9 23:11:15.801: INFO: cilium-z2ltf started at 2023-01-09 22:38:44 +0000 UTC (1+1 container statuses recorded) Jan 9 23:11:15.801: INFO: Init container clean-cilium-state ready: true, restart count 0 Jan 9 23:11:15.801: INFO: Container cilium-agent ready: true, restart count 0 Jan 9 23:11:15.801: INFO: csi-hostpathplugin-0 started at 2023-01-09 23:03:05 +0000 UTC (0+7 container statuses recorded) Jan 9 23:11:15.801: INFO: Container csi-attacher ready: true, restart count 0 Jan 9 23:11:15.801: INFO: Container csi-provisioner ready: true, restart count 0 Jan 9 23:11:15.801: INFO: Container csi-resizer ready: true, restart count 0 Jan 9 23:11:15.801: INFO: Container csi-snapshotter ready: true, restart count 0 Jan 9 23:11:15.801: INFO: Container hostpath ready: true, restart count 0 Jan 9 23:11:15.801: INFO: Container liveness-probe ready: true, restart count 0 Jan 9 23:11:15.801: INFO: Container node-driver-registrar ready: true, restart count 0 Jan 9 23:11:15.801: INFO: hostexec-ip-172-20-59-208.ca-central-1.compute.internal-sn7w7 started at 2023-01-09 23:06:26 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:15.801: INFO: Container agnhost-container ready: true, restart count 0 Jan 9 23:11:15.801: INFO: inline-volume-tester-xnqfb started at 2023-01-09 23:06:35 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:15.801: INFO: Container csi-volume-tester ready: false, restart count 0 Jan 9 23:11:15.801: INFO: coredns-867df8f45c-tmrqb started at 2023-01-09 22:39:03 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:15.801: INFO: Container coredns ready: true, restart count 0 Jan 9 23:11:15.801: INFO: netserver-3 started at 2023-01-09 23:08:45 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:15.801: INFO: Container webserver ready: false, restart count 0 Jan 9 23:11:15.801: INFO: coredns-autoscaler-557ccb4c66-jnjcc started at 2023-01-09 22:39:03 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:15.801: INFO: Container autoscaler ready: true, restart count 0 Jan 9 23:11:15.801: INFO: netserver-3 started at 2023-01-09 23:07:18 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:15.801: INFO: Container webserver ready: false, restart count 0 Jan 9 23:11:15.801: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-hszdf started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:15.801: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:11:15.801: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-fc6pg started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:15.801: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:11:15.801: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-8qp6l started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:15.801: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: false, restart count 0 Jan 9 23:11:15.801: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-bhxkw started at 2023-01-09 22:54:43 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:15.801: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:11:15.801: INFO: pod-e1224014-09b8-42e1-b210-71a9acc98e21 started at 2023-01-09 23:06:29 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:15.801: INFO: Container write-pod ready: false, restart count 0 Jan 9 23:11:15.801: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-rzgjf started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:15.801: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:11:15.801: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-ptfgl started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:15.801: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:11:15.801: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-t5g85 started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:15.801: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:11:15.801: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-hjsbl started at 2023-01-09 22:54:43 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:15.801: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:11:15.801: INFO: ebs-csi-node-prrrf started at 2023-01-09 22:38:44 +0000 UTC (0+3 container statuses recorded) Jan 9 23:11:15.801: INFO: Container ebs-plugin ready: true, restart count 0 Jan 9 23:11:15.801: INFO: Container liveness-probe ready: true, restart count 0 Jan 9 23:11:15.801: INFO: Container node-driver-registrar ready: true, restart count 0 Jan 9 23:11:15.801: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-x682f started at 2023-01-09 22:54:43 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:15.801: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:11:15.801: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-tx2mm started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:15.801: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:11:15.973: INFO: Latency metrics for node ip-172-20-59-208.ca-central-1.compute.internal Jan 9 23:11:15.973: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready �[1mSTEP�[0m: Destroying namespace "fsgroupchangepolicy-7772" for this suite.
Find pod-d16d48d3-63ef-4a76-95b8-d80f8b4aec62 mentions in log files
go run hack/e2e.go -v --test --test_args='--ginkgo.focus=Kubernetes\se2e\ssuite\sExternal\sStorage\s\[Driver\:\sebs\.csi\.aws\.com\]\s\[Testpattern\:\sDynamic\sPV\s\(default\sfs\)\]\sprovisioning\sshould\sprovision\sstorage\swith\smount\soptions$'
/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/provisioning.go:180 Jan 9 23:07:41.774: Unexpected error: <*errors.errorString | 0xc00369e730>: { s: "pod \"pod-9ba7c444-540c-4612-ad00-c7c5dd3438eb\" is not Running: timed out waiting for the condition", } pod "pod-9ba7c444-540c-4612-ad00-c7c5dd3438eb" is not Running: timed out waiting for the condition occurred /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/provisioning.go:418from junit_18.xml
[BeforeEach] [Testpattern: Dynamic PV (default fs)] provisioning /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/framework/testsuite.go:51 [BeforeEach] [Testpattern: Dynamic PV (default fs)] provisioning /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:185 �[1mSTEP�[0m: Creating a kubernetes client Jan 9 23:02:41.260: INFO: >>> kubeConfig: /root/.kube/config �[1mSTEP�[0m: Building a namespace api object, basename provisioning �[1mSTEP�[0m: Waiting for a default service account to be provisioned in namespace �[1mSTEP�[0m: Waiting for kube-root-ca.crt to be provisioned in namespace [It] should provision storage with mount options /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/provisioning.go:180 Jan 9 23:02:41.483: INFO: Warning: Making PVC: VolumeMode specified as invalid empty string, treating as nil Jan 9 23:02:41.483: INFO: Warning: Making PVC: VolumeMode specified as invalid empty string, treating as nil Jan 9 23:02:41.483: INFO: In creating storage class object and pvc objects for driver - sc: &StorageClass{ObjectMeta:{provisioning-3409-e2e-scfcrwv 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},Provisioner:ebs.csi.aws.com,Parameters:map[string]string{encrypted: true,type: gp3,},ReclaimPolicy:nil,MountOptions:[],AllowVolumeExpansion:*true,VolumeBindingMode:*WaitForFirstConsumer,AllowedTopologies:[]TopologySelectorTerm{},}, pvc: &PersistentVolumeClaim{ObjectMeta:{ pvc- provisioning-3409 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},Spec:PersistentVolumeClaimSpec{AccessModes:[ReadWriteOnce],Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{storage: {{1073741824 0} {<nil>} 1Gi BinarySI},},},VolumeName:,Selector:nil,StorageClassName:*provisioning-3409-e2e-scfcrwv,VolumeMode:nil,DataSource:nil,DataSourceRef:nil,},Status:PersistentVolumeClaimStatus{Phase:,AccessModes:[],Capacity:ResourceList{},Conditions:[]PersistentVolumeClaimCondition{},AllocatedResources:ResourceList{},ResizeStatus:nil,},}, src-pvc: &PersistentVolumeClaim{ObjectMeta:{ pvc- provisioning-3409 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},Spec:PersistentVolumeClaimSpec{AccessModes:[ReadWriteOnce],Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{storage: {{1073741824 0} {<nil>} 1Gi BinarySI},},},VolumeName:,Selector:nil,StorageClassName:*provisioning-3409-e2e-scfcrwv,VolumeMode:nil,DataSource:nil,DataSourceRef:nil,},Status:PersistentVolumeClaimStatus{Phase:,AccessModes:[],Capacity:ResourceList{},Conditions:[]PersistentVolumeClaimCondition{},AllocatedResources:ResourceList{},ResizeStatus:nil,},} �[1mSTEP�[0m: Creating a StorageClass �[1mSTEP�[0m: creating claim=&PersistentVolumeClaim{ObjectMeta:{ pvc- provisioning-3409 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},Spec:PersistentVolumeClaimSpec{AccessModes:[ReadWriteOnce],Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{storage: {{1073741824 0} {<nil>} 1Gi BinarySI},},},VolumeName:,Selector:nil,StorageClassName:*provisioning-3409-e2e-scfcrwv,VolumeMode:nil,DataSource:nil,DataSourceRef:nil,},Status:PersistentVolumeClaimStatus{Phase:,AccessModes:[],Capacity:ResourceList{},Conditions:[]PersistentVolumeClaimCondition{},AllocatedResources:ResourceList{},ResizeStatus:nil,},} �[1mSTEP�[0m: creating a pod referring to the class=&StorageClass{ObjectMeta:{provisioning-3409-e2e-scfcrwv d83c0bcc-3d3f-48d2-bfa2-c9eee8a4fd4f 44029 0 2023-01-09 23:02:41 +0000 UTC <nil> <nil> map[] map[] [] [] [{e2e.test Update storage.k8s.io/v1 2023-01-09 23:02:41 +0000 UTC FieldsV1 {"f:allowVolumeExpansion":{},"f:mountOptions":{},"f:parameters":{".":{},"f:encrypted":{},"f:type":{}},"f:provisioner":{},"f:reclaimPolicy":{},"f:volumeBindingMode":{}} }]},Provisioner:ebs.csi.aws.com,Parameters:map[string]string{encrypted: true,type: gp3,},ReclaimPolicy:*Delete,MountOptions:[dirsync],AllowVolumeExpansion:*true,VolumeBindingMode:*WaitForFirstConsumer,AllowedTopologies:[]TopologySelectorTerm{},} claim=&PersistentVolumeClaim{ObjectMeta:{pvc-kpl4w pvc- provisioning-3409 05f6b48d-b934-4774-866a-ff53aa7bd022 44030 0 2023-01-09 23:02:41 +0000 UTC <nil> <nil> map[] map[] [] [kubernetes.io/pvc-protection] [{e2e.test Update v1 2023-01-09 23:02:41 +0000 UTC FieldsV1 {"f:metadata":{"f:generateName":{}},"f:spec":{"f:accessModes":{},"f:resources":{"f:requests":{".":{},"f:storage":{}}},"f:storageClassName":{},"f:volumeMode":{}}} }]},Spec:PersistentVolumeClaimSpec{AccessModes:[ReadWriteOnce],Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{storage: {{1073741824 0} {<nil>} 1Gi BinarySI},},},VolumeName:,Selector:nil,StorageClassName:*provisioning-3409-e2e-scfcrwv,VolumeMode:*Filesystem,DataSource:nil,DataSourceRef:nil,},Status:PersistentVolumeClaimStatus{Phase:Pending,AccessModes:[],Capacity:ResourceList{},Conditions:[]PersistentVolumeClaimCondition{},AllocatedResources:ResourceList{},ResizeStatus:nil,},} Jan 9 23:07:41.774: FAIL: Unexpected error: <*errors.errorString | 0xc00369e730>: { s: "pod \"pod-9ba7c444-540c-4612-ad00-c7c5dd3438eb\" is not Running: timed out waiting for the condition", } pod "pod-9ba7c444-540c-4612-ad00-c7c5dd3438eb" is not Running: timed out waiting for the condition occurred Full Stack Trace k8s.io/kubernetes/test/e2e/storage/testsuites.StorageClassTest.TestDynamicProvisioning({{0x7b06bd0, 0xc003860180}, 0xc002d7ca80, 0xc000b40960, 0xc000b40b40, 0xc004421380, {0x0, 0x0}, {0x0, 0x0, ...}, ...}) /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/provisioning.go:418 +0x830 k8s.io/kubernetes/test/e2e/storage/testsuites.(*provisioningTestSuite).DefineTests.func3() /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/provisioning.go:198 +0x238 k8s.io/kubernetes/test/e2e.RunE2ETests(0x24c66d7) _output/dockerized/go/src/k8s.io/kubernetes/test/e2e/e2e.go:133 +0x697 k8s.io/kubernetes/test/e2e.TestE2E(0x243a8f9) _output/dockerized/go/src/k8s.io/kubernetes/test/e2e/e2e_test.go:136 +0x19 testing.tRunner(0xc0001036c0, 0x735e880) /usr/local/go/src/testing/testing.go:1259 +0x102 created by testing.(*T).Run /usr/local/go/src/testing/testing.go:1306 +0x35a Jan 9 23:07:41.775: INFO: deleting claim "provisioning-3409"/"pvc-kpl4w" Jan 9 23:07:41.809: INFO: deleting storage class provisioning-3409-e2e-scfcrwv [AfterEach] [Testpattern: Dynamic PV (default fs)] provisioning /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:186 �[1mSTEP�[0m: Collecting events from namespace "provisioning-3409". �[1mSTEP�[0m: Found 17 events. Jan 9 23:07:41.875: INFO: At 2023-01-09 23:02:41 +0000 UTC - event for pvc-kpl4w: {ebs.csi.aws.com_ebs-csi-controller-5d8d7cc949-8wxx2_35215d1d-5b9d-46a9-b0b7-d7677b375a3a } Provisioning: External provisioner is provisioning volume for claim "provisioning-3409/pvc-kpl4w" Jan 9 23:07:41.875: INFO: At 2023-01-09 23:02:41 +0000 UTC - event for pvc-kpl4w: {persistentvolume-controller } ExternalProvisioning: waiting for a volume to be created, either by external provisioner "ebs.csi.aws.com" or manually created by system administrator Jan 9 23:07:41.875: INFO: At 2023-01-09 23:02:41 +0000 UTC - event for pvc-kpl4w: {persistentvolume-controller } WaitForFirstConsumer: waiting for first consumer to be created before binding Jan 9 23:07:41.875: INFO: At 2023-01-09 23:02:45 +0000 UTC - event for pod-9ba7c444-540c-4612-ad00-c7c5dd3438eb: {default-scheduler } Scheduled: Successfully assigned provisioning-3409/pod-9ba7c444-540c-4612-ad00-c7c5dd3438eb to ip-172-20-55-216.ca-central-1.compute.internal Jan 9 23:07:41.875: INFO: At 2023-01-09 23:02:45 +0000 UTC - event for pvc-kpl4w: {ebs.csi.aws.com_ebs-csi-controller-5d8d7cc949-8wxx2_35215d1d-5b9d-46a9-b0b7-d7677b375a3a } ProvisioningSucceeded: Successfully provisioned volume pvc-05f6b48d-b934-4774-866a-ff53aa7bd022 Jan 9 23:07:41.875: INFO: At 2023-01-09 23:02:47 +0000 UTC - event for pod-9ba7c444-540c-4612-ad00-c7c5dd3438eb: {attachdetach-controller } SuccessfulAttachVolume: AttachVolume.Attach succeeded for volume "pvc-05f6b48d-b934-4774-866a-ff53aa7bd022" Jan 9 23:07:41.875: INFO: At 2023-01-09 23:02:51 +0000 UTC - event for pod-9ba7c444-540c-4612-ad00-c7c5dd3438eb: {kubelet ip-172-20-55-216.ca-central-1.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "94166d0fd0de882f582839b078417f8adcccf6396edbadd16aae19af5fc98b52" network for pod "pod-9ba7c444-540c-4612-ad00-c7c5dd3438eb": networkPlugin cni failed to set up pod "pod-9ba7c444-540c-4612-ad00-c7c5dd3438eb_provisioning-3409" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 9 23:07:41.875: INFO: At 2023-01-09 23:02:52 +0000 UTC - event for pod-9ba7c444-540c-4612-ad00-c7c5dd3438eb: {kubelet ip-172-20-55-216.ca-central-1.compute.internal} SandboxChanged: Pod sandbox changed, it will be killed and re-created. Jan 9 23:07:41.875: INFO: At 2023-01-09 23:02:54 +0000 UTC - event for pod-9ba7c444-540c-4612-ad00-c7c5dd3438eb: {kubelet ip-172-20-55-216.ca-central-1.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "ccca0189985eba0136cc26324557f9ea80eec8cd1c5ebdbe1c0c81a37da0a9b6" network for pod "pod-9ba7c444-540c-4612-ad00-c7c5dd3438eb": networkPlugin cni failed to set up pod "pod-9ba7c444-540c-4612-ad00-c7c5dd3438eb_provisioning-3409" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 9 23:07:41.875: INFO: At 2023-01-09 23:02:57 +0000 UTC - event for pod-9ba7c444-540c-4612-ad00-c7c5dd3438eb: {kubelet ip-172-20-55-216.ca-central-1.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "eee2eb1e89ed12d60f81a19a099ef2e6182b639d197cbddbec5302405fe18abf" network for pod "pod-9ba7c444-540c-4612-ad00-c7c5dd3438eb": networkPlugin cni failed to set up pod "pod-9ba7c444-540c-4612-ad00-c7c5dd3438eb_provisioning-3409" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 9 23:07:41.875: INFO: At 2023-01-09 23:02:59 +0000 UTC - event for pod-9ba7c444-540c-4612-ad00-c7c5dd3438eb: {kubelet ip-172-20-55-216.ca-central-1.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "806c54d9794fed2f734e6c5d616429be4b85c9330589bcf9f82df9f08121c8e2" network for pod "pod-9ba7c444-540c-4612-ad00-c7c5dd3438eb": networkPlugin cni failed to set up pod "pod-9ba7c444-540c-4612-ad00-c7c5dd3438eb_provisioning-3409" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 9 23:07:41.875: INFO: At 2023-01-09 23:03:02 +0000 UTC - event for pod-9ba7c444-540c-4612-ad00-c7c5dd3438eb: {kubelet ip-172-20-55-216.ca-central-1.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "222128ae65519e8dc6cb3d3da72dbda9ece1e383c7bea10061760f10ffb118de" network for pod "pod-9ba7c444-540c-4612-ad00-c7c5dd3438eb": networkPlugin cni failed to set up pod "pod-9ba7c444-540c-4612-ad00-c7c5dd3438eb_provisioning-3409" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 9 23:07:41.875: INFO: At 2023-01-09 23:03:07 +0000 UTC - event for pod-9ba7c444-540c-4612-ad00-c7c5dd3438eb: {kubelet ip-172-20-55-216.ca-central-1.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "cb8886c26ac800eba74d559dcb1e5d2fcc896dcf85a6bee40709c0513d40f11e" network for pod "pod-9ba7c444-540c-4612-ad00-c7c5dd3438eb": networkPlugin cni failed to set up pod "pod-9ba7c444-540c-4612-ad00-c7c5dd3438eb_provisioning-3409" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 9 23:07:41.875: INFO: At 2023-01-09 23:03:09 +0000 UTC - event for pod-9ba7c444-540c-4612-ad00-c7c5dd3438eb: {kubelet ip-172-20-55-216.ca-central-1.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "7a992acc543351426e74062cd34e21ed1d496f4049952c72bf454216a59b5ce4" network for pod "pod-9ba7c444-540c-4612-ad00-c7c5dd3438eb": networkPlugin cni failed to set up pod "pod-9ba7c444-540c-4612-ad00-c7c5dd3438eb_provisioning-3409" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 9 23:07:41.875: INFO: At 2023-01-09 23:03:12 +0000 UTC - event for pod-9ba7c444-540c-4612-ad00-c7c5dd3438eb: {kubelet ip-172-20-55-216.ca-central-1.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "58d93216fb2d50c7b3dd2bf6ed1d3dec3ea8f4daa064d9db3406e147d8fae1c8" network for pod "pod-9ba7c444-540c-4612-ad00-c7c5dd3438eb": networkPlugin cni failed to set up pod "pod-9ba7c444-540c-4612-ad00-c7c5dd3438eb_provisioning-3409" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 9 23:07:41.875: INFO: At 2023-01-09 23:03:17 +0000 UTC - event for pod-9ba7c444-540c-4612-ad00-c7c5dd3438eb: {kubelet ip-172-20-55-216.ca-central-1.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "84bf3827fbb1592348479ce195ce1100aeadc0626f0f003bb00615e789393adf" network for pod "pod-9ba7c444-540c-4612-ad00-c7c5dd3438eb": networkPlugin cni failed to set up pod "pod-9ba7c444-540c-4612-ad00-c7c5dd3438eb_provisioning-3409" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 9 23:07:41.875: INFO: At 2023-01-09 23:03:22 +0000 UTC - event for pod-9ba7c444-540c-4612-ad00-c7c5dd3438eb: {kubelet ip-172-20-55-216.ca-central-1.compute.internal} FailedCreatePodSandBox: (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "facea25de068f1591cad060548765d3bbdef1f194907570e1bde926a0c2c8e99" network for pod "pod-9ba7c444-540c-4612-ad00-c7c5dd3438eb": networkPlugin cni failed to set up pod "pod-9ba7c444-540c-4612-ad00-c7c5dd3438eb_provisioning-3409" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 9 23:07:41.908: INFO: POD NODE PHASE GRACE CONDITIONS Jan 9 23:07:41.908: INFO: pod-9ba7c444-540c-4612-ad00-c7c5dd3438eb ip-172-20-55-216.ca-central-1.compute.internal Pending [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2023-01-09 23:02:45 +0000 UTC } {Ready False 0001-01-01 00:00:00 +0000 UTC 2023-01-09 23:02:45 +0000 UTC ContainersNotReady containers with unready status: [write-pod]} {ContainersReady False 0001-01-01 00:00:00 +0000 UTC 2023-01-09 23:02:45 +0000 UTC ContainersNotReady containers with unready status: [write-pod]} {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2023-01-09 23:02:45 +0000 UTC }] Jan 9 23:07:41.908: INFO: Jan 9 23:07:41.942: INFO: Unable to fetch provisioning-3409/pod-9ba7c444-540c-4612-ad00-c7c5dd3438eb/write-pod logs: the server rejected our request for an unknown reason (get pods pod-9ba7c444-540c-4612-ad00-c7c5dd3438eb) Jan 9 23:07:41.975: INFO: Logging node info for node ip-172-20-32-67.ca-central-1.compute.internal Jan 9 23:07:42.007: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-32-67.ca-central-1.compute.internal 1f51981b-a01d-404c-b3a4-6649f301f65d 45435 0 2023-01-09 22:38:45 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ca-central-1 failure-domain.beta.kubernetes.io/zone:ca-central-1a kubelet_cleanup:true kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-32-67.ca-central-1.compute.internal kubernetes.io/os:linux kubernetes.io/role:node node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:ca-central-1a topology.hostpath.csi/node:ip-172-20-32-67.ca-central-1.compute.internal topology.kubernetes.io/region:ca-central-1 topology.kubernetes.io/zone:ca-central-1a] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-0272cebea4c6c3d5d"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kops-controller Update v1 2023-01-09 22:38:45 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}} } {kube-controller-manager Update v1 2023-01-09 22:38:45 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.3.0/24\"":{}}}} } {kubelet Update v1 2023-01-09 22:38:45 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {e2e.test Update v1 2023-01-09 22:54:42 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubelet_cleanup":{}}}} } {kube-controller-manager Update v1 2023-01-09 22:55:59 +0000 UTC FieldsV1 {"f:status":{"f:volumesAttached":{}}} status} {kubelet Update v1 2023-01-09 22:56:00 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:capacity":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.3.0/24,DoNotUseExternalID:,ProviderID:aws:///ca-central-1a/i-0272cebea4c6c3d5d,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.3.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051644416 0} {<nil>} 3956684Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946786816 0} {<nil>} 3854284Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-01-09 23:06:12 +0000 UTC,LastTransitionTime:2023-01-09 22:38:45 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-01-09 23:06:12 +0000 UTC,LastTransitionTime:2023-01-09 22:38:45 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-01-09 23:06:12 +0000 UTC,LastTransitionTime:2023-01-09 22:38:45 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-01-09 23:06:12 +0000 UTC,LastTransitionTime:2023-01-09 22:39:06 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.32.67,},NodeAddress{Type:ExternalIP,Address:35.182.2.102,},NodeAddress{Type:Hostname,Address:ip-172-20-32-67.ca-central-1.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-32-67.ca-central-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-35-182-2-102.ca-central-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2536b48bd6b6dade0ed68af2555085,SystemUUID:ec2536b4-8bd6-b6da-de0e-d68af2555085,BootID:bfb3b0df-aafa-481c-9ac6-f85e5e4580e3,KernelVersion:5.15.0-1026-aws,OSImage:Ubuntu 22.04.1 LTS,ContainerRuntimeVersion:docker://20.10.17,KubeletVersion:v1.23.15,KubeProxyVersion:v1.23.15,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:456569528,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/jessie-dnsutils@sha256:11e6a66017ba4e4b938c1612b7a54a3befcefd354796c04e1dba76873a13518e k8s.gcr.io/e2e-test-images/jessie-dnsutils:1.5],SizeBytes:253371792,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:7e8bdd271312fd25fc5ff5a8f04727be84044eb3d7d8d03611972a6752e2e11e k8s.gcr.io/e2e-test-images/agnhost:2.39],SizeBytes:126872991,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nautilus@sha256:99c0d6f1ad24a1aa1905d9c6534d193f268f7b23f9add2ae6bb41f31094bdd5c k8s.gcr.io/e2e-test-images/nautilus:1.5],SizeBytes:124758741,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:1b9d1b2f36cb2dbee1960e82a9344aeb11bd4c4c03abf5e1853e0559c23855e3 k8s.gcr.io/e2e-test-images/httpd:2.4.38-2],SizeBytes:123781643,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.23.15 registry.k8s.io/kube-proxy-amd64:v1.23.15],SizeBytes:112336036,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:10e231cf07c82dfab5141c1afe548dc734e5fc1f67665ec3982a325d1bd31a9a registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.0],SizeBytes:94471559,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:c8e03f60afa90a28e4bb6ec9a8d0fc36d89de4b7475cf2d613afa793ec969fe0 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.0],SizeBytes:56382329,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:4e74c0492bceddc598de1c90cc5bc14dcda94cb49fa9c5bad9d117c4834b5e08 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.1],SizeBytes:56379269,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:36c31f7e1f433c9634d24f876353e8646246d81a03c4e351202c2644daff1620 k8s.gcr.io/sig-storage/csi-resizer:v1.2.0],SizeBytes:53956308,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:ed98431376c9e944e19a465fe8ea944806714dd95416a0821096c78d66b579bd k8s.gcr.io/sig-storage/csi-snapshotter:v4.1.1],SizeBytes:53531990,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:c5be65d6679efabb969d9b019300d187437ae876f992c40911fd2892bbef3b36 k8s.gcr.io/sig-storage/csi-attacher:v3.2.0],SizeBytes:53500826,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:60ab9b3e6a030d3038c87c0d6bca2930f58d1d72823e6a4af09767dc83b696a2 k8s.gcr.io/sig-storage/csi-attacher:v3.2.1],SizeBytes:53499570,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:017727efcfeb7d053af68e51436ce8e65edbc6ca573720afb4f79c8594036955 registry.k8s.io/coredns/coredns:v1.10.0],SizeBytes:50249337,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:7a5ba58a44e0d749e0767e4e37315bcf6a61f33ce3185c1991848af4db0fb70a k8s.gcr.io/sig-storage/csi-resizer:v1.1.0],SizeBytes:49176472,},ContainerImage{Names:[k8s.gcr.io/sig-storage/hostpathplugin@sha256:232fe80174d60d520d36043103853a1d7ab4b7f3782cf43e45034f04ccda58ce k8s.gcr.io/sig-storage/hostpathplugin:v1.7.1],SizeBytes:29977921,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:a61d309da54641db41fb8f35718f744e9f730d4d0384f8c4b186ddc9f06cbd5f k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.1.0],SizeBytes:19662887,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:19582112,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:2dee3fe5fe861bb66c3a4ac51114f3447a4cd35870e0f2e2b558c7a400d89589 k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.2.0],SizeBytes:18699436,},ContainerImage{Names:[k8s.gcr.io/sig-storage/mock-driver@sha256:a7b517f9e0f42ffade802eef9cefa271372386b85f55b702b493241e58459793 k8s.gcr.io/sig-storage/mock-driver:v4.1.0],SizeBytes:17680993,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:17611980,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:529be2c9770add0cdd0c989115222ea9fc1be430c11095eb9f6dafcf98a36e2b k8s.gcr.io/sig-storage/livenessprobe:v2.4.0],SizeBytes:17175488,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:13616070e3f29de4417eee434a8ef472221c9e51b3d037b5a6b46cef08eb7443 k8s.gcr.io/e2e-test-images/nginx:1.14-2],SizeBytes:16032814,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nonewprivs@sha256:8ac1264691820febacf3aea5d152cbde6d10685731ec14966a9401c6f47a68ac k8s.gcr.io/e2e-test-images/nonewprivs:1.3],SizeBytes:7107254,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf k8s.gcr.io/e2e-test-images/busybox:1.29-2],SizeBytes:1154361,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:39e1e963e5310e9c313bad51523be012ede7b35bb9316517d19089a010356592 k8s.gcr.io/e2e-test-images/busybox:1.29-1],SizeBytes:1154361,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db k8s.gcr.io/pause:3.6 registry.k8s.io/pause:3.6],SizeBytes:682696,},},VolumesInUse:[kubernetes.io/csi/ebs.csi.aws.com^vol-01e8a4a8ff1da0e10],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-01e8a4a8ff1da0e10,DevicePath:,},},Config:nil,},} Jan 9 23:07:42.007: INFO: Logging kubelet events for node ip-172-20-32-67.ca-central-1.compute.internal Jan 9 23:07:42.041: INFO: Logging pods the kubelet thinks is on node ip-172-20-32-67.ca-central-1.compute.internal Jan 9 23:07:42.080: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-sddjv started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:07:42.080: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:07:42.080: INFO: csi-hostpathplugin-0 started at 2023-01-09 23:05:41 +0000 UTC (0+7 container statuses recorded) Jan 9 23:07:42.080: INFO: Container csi-attacher ready: false, restart count 0 Jan 9 23:07:42.080: INFO: Container csi-provisioner ready: false, restart count 0 Jan 9 23:07:42.080: INFO: Container csi-resizer ready: false, restart count 0 Jan 9 23:07:42.080: INFO: Container csi-snapshotter ready: false, restart count 0 Jan 9 23:07:42.080: INFO: Container hostpath ready: false, restart count 0 Jan 9 23:07:42.080: INFO: Container liveness-probe ready: false, restart count 0 Jan 9 23:07:42.080: INFO: Container node-driver-registrar ready: false, restart count 0 Jan 9 23:07:42.080: INFO: inline-volume-tester-5g6j6 started at 2023-01-09 22:55:56 +0000 UTC (0+1 container statuses recorded) Jan 9 23:07:42.080: INFO: Container csi-volume-tester ready: false, restart count 0 Jan 9 23:07:42.080: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-gjc2h started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:07:42.080: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:07:42.080: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-8vzqv started at 2023-01-09 22:54:43 +0000 UTC (0+1 container statuses recorded) Jan 9 23:07:42.080: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:07:42.080: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-jb6kp started at 2023-01-09 22:54:43 +0000 UTC (0+1 container statuses recorded) Jan 9 23:07:42.080: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:07:42.080: INFO: cilium-dvsfs started at 2023-01-09 22:38:46 +0000 UTC (1+1 container statuses recorded) Jan 9 23:07:42.080: INFO: Init container clean-cilium-state ready: true, restart count 0 Jan 9 23:07:42.080: INFO: Container cilium-agent ready: true, restart count 0 Jan 9 23:07:42.080: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-fr98x started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:07:42.080: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:07:42.080: INFO: rs-zn5kz started at 2023-01-09 22:59:56 +0000 UTC (0+1 container statuses recorded) Jan 9 23:07:42.080: INFO: Container donothing ready: true, restart count 0 Jan 9 23:07:42.080: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-hwbbn started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:07:42.080: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:07:42.080: INFO: coredns-867df8f45c-sckrz started at 2023-01-09 22:39:23 +0000 UTC (0+1 container statuses recorded) Jan 9 23:07:42.080: INFO: Container coredns ready: true, restart count 0 Jan 9 23:07:42.080: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-44ccv started at 2023-01-09 22:54:43 +0000 UTC (0+1 container statuses recorded) Jan 9 23:07:42.080: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:07:42.080: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-fzrxx started at 2023-01-09 22:54:43 +0000 UTC (0+1 container statuses recorded) Jan 9 23:07:42.080: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:07:42.080: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-mlcl4 started at 2023-01-09 22:54:43 +0000 UTC (0+1 container statuses recorded) Jan 9 23:07:42.080: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:07:42.080: INFO: rs-gxqdg started at 2023-01-09 22:59:56 +0000 UTC (0+1 container statuses recorded) Jan 9 23:07:42.080: INFO: Container donothing ready: false, restart count 0 Jan 9 23:07:42.080: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-5pr42 started at 2023-01-09 22:54:43 +0000 UTC (0+1 container statuses recorded) Jan 9 23:07:42.080: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:07:42.080: INFO: netserver-0 started at 2023-01-09 23:07:18 +0000 UTC (0+1 container statuses recorded) Jan 9 23:07:42.080: INFO: Container webserver ready: false, restart count 0 Jan 9 23:07:42.080: INFO: ebs-csi-node-lc4rb started at 2023-01-09 22:38:46 +0000 UTC (0+3 container statuses recorded) Jan 9 23:07:42.080: INFO: Container ebs-plugin ready: true, restart count 0 Jan 9 23:07:42.080: INFO: Container liveness-probe ready: true, restart count 0 Jan 9 23:07:42.080: INFO: Container node-driver-registrar ready: true, restart count 0 Jan 9 23:07:42.336: INFO: Latency metrics for node ip-172-20-32-67.ca-central-1.compute.internal Jan 9 23:07:42.336: INFO: Logging node info for node ip-172-20-37-64.ca-central-1.compute.internal Jan 9 23:07:42.367: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-37-64.ca-central-1.compute.internal d817613a-0fe5-40c0-a5e4-7b85b9f8506b 44622 0 2023-01-09 22:36:54 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:c5.large beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ca-central-1 failure-domain.beta.kubernetes.io/zone:ca-central-1a kops.k8s.io/kops-controller-pki: kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-37-64.ca-central-1.compute.internal kubernetes.io/os:linux kubernetes.io/role:master node-role.kubernetes.io/control-plane: node-role.kubernetes.io/master: node.kubernetes.io/exclude-from-external-load-balancers: node.kubernetes.io/instance-type:c5.large topology.ebs.csi.aws.com/zone:ca-central-1a topology.kubernetes.io/region:ca-central-1 topology.kubernetes.io/zone:ca-central-1a] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-06217673764fdafeb"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kubelet Update v1 2023-01-09 22:36:54 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {protokube Update v1 2023-01-09 22:37:19 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kops.k8s.io/kops-controller-pki":{},"f:node-role.kubernetes.io/control-plane":{},"f:node.kubernetes.io/exclude-from-external-load-balancers":{}}}} } {kops-controller Update v1 2023-01-09 22:37:21 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/master":{}}}} } {kube-controller-manager Update v1 2023-01-09 22:37:35 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.0.0/24\"":{}},"f:taints":{}}} } {kubelet Update v1 2023-01-09 22:38:39 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.0.0/24,DoNotUseExternalID:,ProviderID:aws:///ca-central-1a/i-06217673764fdafeb,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/master,Value:,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[100.96.0.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3862904832 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3758047232 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-01-09 23:04:17 +0000 UTC,LastTransitionTime:2023-01-09 22:36:48 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-01-09 23:04:17 +0000 UTC,LastTransitionTime:2023-01-09 22:36:48 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-01-09 23:04:17 +0000 UTC,LastTransitionTime:2023-01-09 22:36:48 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-01-09 23:04:17 +0000 UTC,LastTransitionTime:2023-01-09 22:37:35 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.37.64,},NodeAddress{Type:ExternalIP,Address:3.96.216.165,},NodeAddress{Type:Hostname,Address:ip-172-20-37-64.ca-central-1.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-37-64.ca-central-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-3-96-216-165.ca-central-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec27a483f61319a98e5ba2ee2d59e6a3,SystemUUID:ec27a483-f613-19a9-8e5b-a2ee2d59e6a3,BootID:9b41e339-1aaa-4374-8f11-8d34e12cd62a,KernelVersion:5.15.0-1026-aws,OSImage:Ubuntu 22.04.1 LTS,ContainerRuntimeVersion:docker://20.10.17,KubeletVersion:v1.23.15,KubeProxyVersion:v1.23.15,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[registry.k8s.io/etcdadm/etcd-manager@sha256:66a453db625abb268f4b3bbefc5a34a171d81e6e8796cecca54cfd71775c77c4 registry.k8s.io/etcdadm/etcd-manager:v3.0.20221209],SizeBytes:662209237,},ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:456569528,},ContainerImage{Names:[k8s.gcr.io/kube-apiserver-amd64:v1.23.15 registry.k8s.io/kube-apiserver-amd64:v1.23.15],SizeBytes:135240444,},ContainerImage{Names:[k8s.gcr.io/kube-controller-manager-amd64:v1.23.15 registry.k8s.io/kube-controller-manager-amd64:v1.23.15],SizeBytes:125046097,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.23.15 registry.k8s.io/kube-proxy-amd64:v1.23.15],SizeBytes:112336036,},ContainerImage{Names:[quay.io/cilium/operator@sha256:a6d24a006a6b92967ac90786b49bc1ac26e5477cf028cd1186efcfc2466484db quay.io/cilium/operator:v1.12.5],SizeBytes:110016956,},ContainerImage{Names:[registry.k8s.io/kops/kops-controller:1.26.0-beta.1],SizeBytes:104212800,},ContainerImage{Names:[registry.k8s.io/kops/dns-controller:1.26.0-beta.1],SizeBytes:102566451,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:10e231cf07c82dfab5141c1afe548dc734e5fc1f67665ec3982a325d1bd31a9a registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.0],SizeBytes:94471559,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:122bfb8c1edabb3c0edd63f06523e6940d958d19b3957dc7b1d6f81e9f1f6119 registry.k8s.io/sig-storage/csi-provisioner:v3.1.0],SizeBytes:57736070,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:9ebbf9f023e7b41ccee3d52afe39a89e3ddacdbb69269d583abfc25847cfd9e4 registry.k8s.io/sig-storage/csi-resizer:v1.4.0],SizeBytes:55464778,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:8b9c313c05f54fb04f8d430896f5f5904b6cb157df261501b29adc04d2b2dc7b registry.k8s.io/sig-storage/csi-attacher:v3.4.0],SizeBytes:54848595,},ContainerImage{Names:[k8s.gcr.io/kube-scheduler-amd64:v1.23.15 registry.k8s.io/kube-scheduler-amd64:v1.23.15],SizeBytes:53521430,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:19582112,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:17611980,},ContainerImage{Names:[registry.k8s.io/kops/kube-apiserver-healthcheck:1.26.0-beta.1],SizeBytes:8787111,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:682696,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Jan 9 23:07:42.368: INFO: Logging kubelet events for node ip-172-20-37-64.ca-central-1.compute.internal Jan 9 23:07:42.401: INFO: Logging pods the kubelet thinks is on node ip-172-20-37-64.ca-central-1.compute.internal Jan 9 23:07:42.438: INFO: cilium-q7lrg started at 2023-01-09 22:37:17 +0000 UTC (1+1 container statuses recorded) Jan 9 23:07:42.439: INFO: Init container clean-cilium-state ready: true, restart count 0 Jan 9 23:07:42.439: INFO: Container cilium-agent ready: true, restart count 0 Jan 9 23:07:42.439: INFO: cilium-operator-c7bfc9f44-w4nqz started at 2023-01-09 22:37:19 +0000 UTC (0+1 container statuses recorded) Jan 9 23:07:42.439: INFO: Container cilium-operator ready: true, restart count 0 Jan 9 23:07:42.439: INFO: ebs-csi-controller-5d8d7cc949-8wxx2 started at 2023-01-09 22:37:19 +0000 UTC (0+5 container statuses recorded) Jan 9 23:07:42.439: INFO: Container csi-attacher ready: true, restart count 0 Jan 9 23:07:42.439: INFO: Container csi-provisioner ready: true, restart count 0 Jan 9 23:07:42.439: INFO: Container csi-resizer ready: true, restart count 0 Jan 9 23:07:42.439: INFO: Container ebs-plugin ready: true, restart count 0 Jan 9 23:07:42.439: INFO: Container liveness-probe ready: true, restart count 0 Jan 9 23:07:42.439: INFO: dns-controller-867784b75c-sxxcl started at 2023-01-09 22:37:20 +0000 UTC (0+1 container statuses recorded) Jan 9 23:07:42.439: INFO: Container dns-controller ready: true, restart count 0 Jan 9 23:07:42.439: INFO: kube-scheduler-ip-172-20-37-64.ca-central-1.compute.internal started at 2023-01-09 22:36:15 +0000 UTC (0+1 container statuses recorded) Jan 9 23:07:42.439: INFO: Container kube-scheduler ready: true, restart count 0 Jan 9 23:07:42.439: INFO: etcd-manager-events-ip-172-20-37-64.ca-central-1.compute.internal started at 2023-01-09 22:36:15 +0000 UTC (0+1 container statuses recorded) Jan 9 23:07:42.439: INFO: Container etcd-manager ready: true, restart count 0 Jan 9 23:07:42.439: INFO: etcd-manager-main-ip-172-20-37-64.ca-central-1.compute.internal started at 2023-01-09 22:36:15 +0000 UTC (0+1 container statuses recorded) Jan 9 23:07:42.439: INFO: Container etcd-manager ready: true, restart count 0 Jan 9 23:07:42.439: INFO: ebs-csi-node-8q4pg started at 2023-01-09 22:37:17 +0000 UTC (0+3 container statuses recorded) Jan 9 23:07:42.439: INFO: Container ebs-plugin ready: true, restart count 0 Jan 9 23:07:42.439: INFO: Container liveness-probe ready: true, restart count 0 Jan 9 23:07:42.439: INFO: Container node-driver-registrar ready: true, restart count 0 Jan 9 23:07:42.439: INFO: kube-apiserver-ip-172-20-37-64.ca-central-1.compute.internal started at 2023-01-09 22:36:15 +0000 UTC (0+2 container statuses recorded) Jan 9 23:07:42.439: INFO: Container healthcheck ready: true, restart count 0 Jan 9 23:07:42.439: INFO: Container kube-apiserver ready: true, restart count 1 Jan 9 23:07:42.439: INFO: kube-controller-manager-ip-172-20-37-64.ca-central-1.compute.internal started at 2023-01-09 22:36:15 +0000 UTC (0+1 container statuses recorded) Jan 9 23:07:42.439: INFO: Container kube-controller-manager ready: true, restart count 2 Jan 9 23:07:42.439: INFO: kops-controller-cqrv8 started at 2023-01-09 22:37:19 +0000 UTC (0+1 container statuses recorded) Jan 9 23:07:42.439: INFO: Container kops-controller ready: true, restart count 0 Jan 9 23:07:42.553: INFO: Latency metrics for node ip-172-20-37-64.ca-central-1.compute.internal Jan 9 23:07:42.553: INFO: Logging node info for node ip-172-20-55-216.ca-central-1.compute.internal Jan 9 23:07:42.585: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-55-216.ca-central-1.compute.internal 2ceff781-505a-4706-ad8b-e38e37f72571 44855 0 2023-01-09 22:38:47 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ca-central-1 failure-domain.beta.kubernetes.io/zone:ca-central-1a kubelet_cleanup:true kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-55-216.ca-central-1.compute.internal kubernetes.io/os:linux kubernetes.io/role:node node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:ca-central-1a topology.hostpath.csi/node:ip-172-20-55-216.ca-central-1.compute.internal topology.kubernetes.io/region:ca-central-1 topology.kubernetes.io/zone:ca-central-1a] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-0c78741e44993ae7f"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kops-controller Update v1 2023-01-09 22:38:47 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}} } {kube-controller-manager Update v1 2023-01-09 22:38:47 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.4.0/24\"":{}}}} } {kubelet Update v1 2023-01-09 22:38:47 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {e2e.test Update v1 2023-01-09 22:54:42 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubelet_cleanup":{}}}} } {kube-controller-manager Update v1 2023-01-09 22:56:45 +0000 UTC FieldsV1 {"f:status":{"f:volumesAttached":{}}} status} {kubelet Update v1 2023-01-09 22:56:51 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:capacity":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.4.0/24,DoNotUseExternalID:,ProviderID:aws:///ca-central-1a/i-0c78741e44993ae7f,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.4.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051644416 0} {<nil>} 3956684Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946786816 0} {<nil>} 3854284Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-01-09 23:05:11 +0000 UTC,LastTransitionTime:2023-01-09 22:38:47 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-01-09 23:05:11 +0000 UTC,LastTransitionTime:2023-01-09 22:38:47 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-01-09 23:05:11 +0000 UTC,LastTransitionTime:2023-01-09 22:38:47 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-01-09 23:05:11 +0000 UTC,LastTransitionTime:2023-01-09 22:39:07 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.55.216,},NodeAddress{Type:ExternalIP,Address:15.222.60.56,},NodeAddress{Type:Hostname,Address:ip-172-20-55-216.ca-central-1.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-55-216.ca-central-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-15-222-60-56.ca-central-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec249e6a3570a28ebd4276206d3ae523,SystemUUID:ec249e6a-3570-a28e-bd42-76206d3ae523,BootID:d58dcef2-849d-4807-9ba5-0438ebaa05b9,KernelVersion:5.15.0-1026-aws,OSImage:Ubuntu 22.04.1 LTS,ContainerRuntimeVersion:docker://20.10.17,KubeletVersion:v1.23.15,KubeProxyVersion:v1.23.15,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:456569528,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/jessie-dnsutils@sha256:11e6a66017ba4e4b938c1612b7a54a3befcefd354796c04e1dba76873a13518e k8s.gcr.io/e2e-test-images/jessie-dnsutils:1.5],SizeBytes:253371792,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:20f25f275d46aa728f7615a1ccc19c78b2ed89435bf943a44b339f70f45508e6 k8s.gcr.io/e2e-test-images/httpd:2.4.39-2],SizeBytes:126894770,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:7e8bdd271312fd25fc5ff5a8f04727be84044eb3d7d8d03611972a6752e2e11e k8s.gcr.io/e2e-test-images/agnhost:2.39],SizeBytes:126872991,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nautilus@sha256:99c0d6f1ad24a1aa1905d9c6534d193f268f7b23f9add2ae6bb41f31094bdd5c k8s.gcr.io/e2e-test-images/nautilus:1.5],SizeBytes:124758741,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:1b9d1b2f36cb2dbee1960e82a9344aeb11bd4c4c03abf5e1853e0559c23855e3 k8s.gcr.io/e2e-test-images/httpd:2.4.38-2],SizeBytes:123781643,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.23.15 registry.k8s.io/kube-proxy-amd64:v1.23.15],SizeBytes:112336036,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:10e231cf07c82dfab5141c1afe548dc734e5fc1f67665ec3982a325d1bd31a9a registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.0],SizeBytes:94471559,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:c8e03f60afa90a28e4bb6ec9a8d0fc36d89de4b7475cf2d613afa793ec969fe0 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.0],SizeBytes:56382329,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:4e74c0492bceddc598de1c90cc5bc14dcda94cb49fa9c5bad9d117c4834b5e08 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.1],SizeBytes:56379269,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:36c31f7e1f433c9634d24f876353e8646246d81a03c4e351202c2644daff1620 k8s.gcr.io/sig-storage/csi-resizer:v1.2.0],SizeBytes:53956308,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:ed98431376c9e944e19a465fe8ea944806714dd95416a0821096c78d66b579bd k8s.gcr.io/sig-storage/csi-snapshotter:v4.1.1],SizeBytes:53531990,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:c5be65d6679efabb969d9b019300d187437ae876f992c40911fd2892bbef3b36 k8s.gcr.io/sig-storage/csi-attacher:v3.2.0],SizeBytes:53500826,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:60ab9b3e6a030d3038c87c0d6bca2930f58d1d72823e6a4af09767dc83b696a2 k8s.gcr.io/sig-storage/csi-attacher:v3.2.1],SizeBytes:53499570,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:7a5ba58a44e0d749e0767e4e37315bcf6a61f33ce3185c1991848af4db0fb70a k8s.gcr.io/sig-storage/csi-resizer:v1.1.0],SizeBytes:49176472,},ContainerImage{Names:[k8s.gcr.io/sig-storage/hostpathplugin@sha256:232fe80174d60d520d36043103853a1d7ab4b7f3782cf43e45034f04ccda58ce k8s.gcr.io/sig-storage/hostpathplugin:v1.7.1],SizeBytes:29977921,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:a61d309da54641db41fb8f35718f744e9f730d4d0384f8c4b186ddc9f06cbd5f k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.1.0],SizeBytes:19662887,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:19582112,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:2dee3fe5fe861bb66c3a4ac51114f3447a4cd35870e0f2e2b558c7a400d89589 k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.2.0],SizeBytes:18699436,},ContainerImage{Names:[k8s.gcr.io/sig-storage/mock-driver@sha256:a7b517f9e0f42ffade802eef9cefa271372386b85f55b702b493241e58459793 k8s.gcr.io/sig-storage/mock-driver:v4.1.0],SizeBytes:17680993,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:17611980,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:529be2c9770add0cdd0c989115222ea9fc1be430c11095eb9f6dafcf98a36e2b k8s.gcr.io/sig-storage/livenessprobe:v2.4.0],SizeBytes:17175488,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:13616070e3f29de4417eee434a8ef472221c9e51b3d037b5a6b46cef08eb7443 k8s.gcr.io/e2e-test-images/nginx:1.14-2],SizeBytes:16032814,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf k8s.gcr.io/e2e-test-images/busybox:1.29-2],SizeBytes:1154361,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db k8s.gcr.io/pause:3.6 registry.k8s.io/pause:3.6],SizeBytes:682696,},},VolumesInUse:[kubernetes.io/csi/ebs.csi.aws.com^vol-07a7954e90ced7ccc kubernetes.io/csi/ebs.csi.aws.com^vol-0c8e4ae9d44bf0644 kubernetes.io/csi/ebs.csi.aws.com^vol-0d0fbf37c343045c2],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-0c8e4ae9d44bf0644,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-07a7954e90ced7ccc,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-0d0fbf37c343045c2,DevicePath:,},},Config:nil,},} Jan 9 23:07:42.585: INFO: Logging kubelet events for node ip-172-20-55-216.ca-central-1.compute.internal Jan 9 23:07:42.619: INFO: Logging pods the kubelet thinks is on node ip-172-20-55-216.ca-central-1.compute.internal Jan 9 23:07:42.666: INFO: fail-once-non-local-nx7cr started at 2023-01-09 22:56:53 +0000 UTC (0+1 container statuses recorded) Jan 9 23:07:42.666: INFO: Container c ready: false, restart count 0 Jan 9 23:07:42.666: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-6nqvb started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:07:42.666: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:07:42.666: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-8kb86 started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:07:42.666: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:07:42.666: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-vwd6k started at 2023-01-09 22:54:43 +0000 UTC (0+1 container statuses recorded) Jan 9 23:07:42.666: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:07:42.666: INFO: rs-ctw9c started at 2023-01-09 22:59:56 +0000 UTC (0+1 container statuses recorded) Jan 9 23:07:42.666: INFO: Container donothing ready: true, restart count 0 Jan 9 23:07:42.666: INFO: ebs-csi-node-psq97 started at 2023-01-09 22:38:48 +0000 UTC (0+3 container statuses recorded) Jan 9 23:07:42.666: INFO: Container ebs-plugin ready: true, restart count 0 Jan 9 23:07:42.666: INFO: Container liveness-probe ready: true, restart count 0 Jan 9 23:07:42.666: INFO: Container node-driver-registrar ready: true, restart count 0 Jan 9 23:07:42.666: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-qj2bw started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:07:42.666: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:07:42.666: INFO: ss2-0 started at 2023-01-09 22:57:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:07:42.666: INFO: Container webserver ready: false, restart count 0 Jan 9 23:07:42.666: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-krwzl started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:07:42.666: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:07:42.666: INFO: rs-j4qxp started at 2023-01-09 22:59:56 +0000 UTC (0+1 container statuses recorded) Jan 9 23:07:42.666: INFO: Container donothing ready: false, restart count 0 Jan 9 23:07:42.666: INFO: csi-mockplugin-resizer-0 started at 2023-01-09 23:02:58 +0000 UTC (0+1 container statuses recorded) Jan 9 23:07:42.666: INFO: Container csi-resizer ready: false, restart count 0 Jan 9 23:07:42.666: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-6ks7b started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:07:42.666: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:07:42.666: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-72kwd started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:07:42.666: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:07:42.666: INFO: pod-adoption-release started at 2023-01-09 23:05:21 +0000 UTC (0+1 container statuses recorded) Jan 9 23:07:42.666: INFO: Container pod-adoption-release ready: false, restart count 0 Jan 9 23:07:42.666: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-qglk4 started at 2023-01-09 22:54:43 +0000 UTC (0+1 container statuses recorded) Jan 9 23:07:42.666: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:07:42.666: INFO: httpd started at 2023-01-09 23:06:03 +0000 UTC (0+1 container statuses recorded) Jan 9 23:07:42.666: INFO: Container httpd ready: false, restart count 0 Jan 9 23:07:42.666: INFO: cilium-dqpw4 started at 2023-01-09 22:38:48 +0000 UTC (1+1 container statuses recorded) Jan 9 23:07:42.666: INFO: Init container clean-cilium-state ready: true, restart count 0 Jan 9 23:07:42.666: INFO: Container cilium-agent ready: true, restart count 0 Jan 9 23:07:42.666: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-b7bxr started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:07:42.666: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:07:42.666: INFO: pod-subpath-test-inlinevolume-x5g6 started at 2023-01-09 23:05:45 +0000 UTC (2+1 container statuses recorded) Jan 9 23:07:42.666: INFO: Init container init-volume-inlinevolume-x5g6 ready: false, restart count 0 Jan 9 23:07:42.666: INFO: Init container test-init-volume-inlinevolume-x5g6 ready: false, restart count 0 Jan 9 23:07:42.666: INFO: Container test-container-subpath-inlinevolume-x5g6 ready: false, restart count 0 Jan 9 23:07:42.666: INFO: inline-volume-tester-7mx4s started at 2023-01-09 22:56:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:07:42.666: INFO: Container csi-volume-tester ready: false, restart count 0 Jan 9 23:07:42.666: INFO: fail-once-non-local-mh6qc started at 2023-01-09 22:56:53 +0000 UTC (0+1 container statuses recorded) Jan 9 23:07:42.666: INFO: Container c ready: false, restart count 0 Jan 9 23:07:42.666: INFO: rs-lf7br started at 2023-01-09 22:59:56 +0000 UTC (0+1 container statuses recorded) Jan 9 23:07:42.666: INFO: Container donothing ready: false, restart count 0 Jan 9 23:07:42.666: INFO: csi-mockplugin-0 started at 2023-01-09 23:02:58 +0000 UTC (0+3 container statuses recorded) Jan 9 23:07:42.666: INFO: Container csi-provisioner ready: false, restart count 0 Jan 9 23:07:42.666: INFO: Container driver-registrar ready: false, restart count 0 Jan 9 23:07:42.666: INFO: Container mock ready: false, restart count 0 Jan 9 23:07:42.666: INFO: pod-subpath-test-projected-tqfk started at 2023-01-09 23:03:15 +0000 UTC (0+1 container statuses recorded) Jan 9 23:07:42.666: INFO: Container test-container-subpath-projected-tqfk ready: false, restart count 0 Jan 9 23:07:42.666: INFO: inline-volume-tester-crzhr started at 2023-01-09 23:05:03 +0000 UTC (0+1 container statuses recorded) Jan 9 23:07:42.666: INFO: Container csi-volume-tester ready: false, restart count 0 Jan 9 23:07:42.666: INFO: netserver-1 started at 2023-01-09 23:07:18 +0000 UTC (0+1 container statuses recorded) Jan 9 23:07:42.666: INFO: Container webserver ready: false, restart count 0 Jan 9 23:07:42.666: INFO: pod-9ba7c444-540c-4612-ad00-c7c5dd3438eb started at 2023-01-09 23:02:45 +0000 UTC (0+1 container statuses recorded) Jan 9 23:07:42.666: INFO: Container write-pod ready: false, restart count 0 Jan 9 23:07:42.666: INFO: pause-pod-78b7f9b75c-l4st2 started at 2023-01-09 23:05:44 +0000 UTC (0+1 container statuses recorded) Jan 9 23:07:42.666: INFO: Container agnhost-pause ready: false, restart count 0 Jan 9 23:07:42.666: INFO: logs-generator started at 2023-01-09 23:05:28 +0000 UTC (0+1 container statuses recorded) Jan 9 23:07:42.666: INFO: Container logs-generator ready: false, restart count 0 Jan 9 23:07:42.666: INFO: pod-8ade382c-2fb2-4f02-8d50-57f7b6a15696 started at 2023-01-09 23:06:54 +0000 UTC (0+1 container statuses recorded) Jan 9 23:07:42.666: INFO: Container test-container ready: false, restart count 0 Jan 9 23:07:42.666: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-mcd88 started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:07:42.666: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:07:42.666: INFO: echo-sourceip started at 2023-01-09 23:00:54 +0000 UTC (0+1 container statuses recorded) Jan 9 23:07:42.666: INFO: Container agnhost-container ready: true, restart count 0 Jan 9 23:07:42.666: INFO: startup-ff2f8ca6-ae0a-4fae-9483-a7d80b37fdc1 started at 2023-01-09 23:07:24 +0000 UTC (0+1 container statuses recorded) Jan 9 23:07:42.666: INFO: Container busybox ready: false, restart count 0 Jan 9 23:07:42.840: INFO: Latency metrics for node ip-172-20-55-216.ca-central-1.compute.internal Jan 9 23:07:42.840: INFO: Logging node info for node ip-172-20-57-251.ca-central-1.compute.internal Jan 9 23:07:42.872: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-57-251.ca-central-1.compute.internal eaaaf6eb-cbd5-4763-9c39-fb0e1a4ce44a 45391 0 2023-01-09 22:38:45 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ca-central-1 failure-domain.beta.kubernetes.io/zone:ca-central-1a kubelet_cleanup:true kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-57-251.ca-central-1.compute.internal kubernetes.io/os:linux kubernetes.io/role:node node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:ca-central-1a topology.hostpath.csi/node:ip-172-20-57-251.ca-central-1.compute.internal topology.kubernetes.io/region:ca-central-1 topology.kubernetes.io/zone:ca-central-1a] map[csi.volume.kubernetes.io/nodeid:{"csi-mock-csi-mock-volumes-9402":"csi-mock-csi-mock-volumes-9402","ebs.csi.aws.com":"i-0513e7d6e5eba8eec"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kops-controller Update v1 2023-01-09 22:38:45 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}} } {kube-controller-manager Update v1 2023-01-09 22:38:45 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.2.0/24\"":{}}}} } {kubelet Update v1 2023-01-09 22:38:45 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {e2e.test Update v1 2023-01-09 22:54:42 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubelet_cleanup":{}}}} } {kube-controller-manager Update v1 2023-01-09 23:00:23 +0000 UTC FieldsV1 {"f:status":{"f:volumesAttached":{}}} status} {kubelet Update v1 2023-01-09 23:00:27 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.2.0/24,DoNotUseExternalID:,ProviderID:aws:///ca-central-1a/i-0513e7d6e5eba8eec,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.2.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051652608 0} {<nil>} 3956692Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946795008 0} {<nil>} 3854292Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-01-09 23:06:06 +0000 UTC,LastTransitionTime:2023-01-09 22:38:45 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-01-09 23:06:06 +0000 UTC,LastTransitionTime:2023-01-09 22:38:45 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-01-09 23:06:06 +0000 UTC,LastTransitionTime:2023-01-09 22:38:45 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-01-09 23:06:06 +0000 UTC,LastTransitionTime:2023-01-09 22:39:05 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.57.251,},NodeAddress{Type:ExternalIP,Address:3.96.144.209,},NodeAddress{Type:Hostname,Address:ip-172-20-57-251.ca-central-1.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-57-251.ca-central-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-3-96-144-209.ca-central-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2f0d7ba95b37a96a1d9360f6071d84,SystemUUID:ec2f0d7b-a95b-37a9-6a1d-9360f6071d84,BootID:67865282-1911-4e38-8675-a7b78d5ae45e,KernelVersion:5.15.0-1026-aws,OSImage:Ubuntu 22.04.1 LTS,ContainerRuntimeVersion:docker://20.10.17,KubeletVersion:v1.23.15,KubeProxyVersion:v1.23.15,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:456569528,},ContainerImage{Names:[k8s.gcr.io/etcd@sha256:dd75ec974b0a2a6f6bb47001ba09207976e625db898d1b16735528c009cb171c k8s.gcr.io/etcd:3.5.6-0],SizeBytes:299475478,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/jessie-dnsutils@sha256:11e6a66017ba4e4b938c1612b7a54a3befcefd354796c04e1dba76873a13518e k8s.gcr.io/e2e-test-images/jessie-dnsutils:1.5],SizeBytes:253371792,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:20f25f275d46aa728f7615a1ccc19c78b2ed89435bf943a44b339f70f45508e6 k8s.gcr.io/e2e-test-images/httpd:2.4.39-2],SizeBytes:126894770,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:7e8bdd271312fd25fc5ff5a8f04727be84044eb3d7d8d03611972a6752e2e11e k8s.gcr.io/e2e-test-images/agnhost:2.39],SizeBytes:126872991,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nautilus@sha256:99c0d6f1ad24a1aa1905d9c6534d193f268f7b23f9add2ae6bb41f31094bdd5c k8s.gcr.io/e2e-test-images/nautilus:1.5],SizeBytes:124758741,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:1b9d1b2f36cb2dbee1960e82a9344aeb11bd4c4c03abf5e1853e0559c23855e3 k8s.gcr.io/e2e-test-images/httpd:2.4.38-2],SizeBytes:123781643,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.23.15 registry.k8s.io/kube-proxy-amd64:v1.23.15],SizeBytes:112336036,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:10e231cf07c82dfab5141c1afe548dc734e5fc1f67665ec3982a325d1bd31a9a registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.0],SizeBytes:94471559,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/sample-apiserver@sha256:f9c93b92b6ff750b41a93c4e4fe0bfe384597aeb841e2539d5444815c55b2d8f k8s.gcr.io/e2e-test-images/sample-apiserver:1.17.5],SizeBytes:56504541,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:c8e03f60afa90a28e4bb6ec9a8d0fc36d89de4b7475cf2d613afa793ec969fe0 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.0],SizeBytes:56382329,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:4e74c0492bceddc598de1c90cc5bc14dcda94cb49fa9c5bad9d117c4834b5e08 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.1],SizeBytes:56379269,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:36c31f7e1f433c9634d24f876353e8646246d81a03c4e351202c2644daff1620 k8s.gcr.io/sig-storage/csi-resizer:v1.2.0],SizeBytes:53956308,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:ed98431376c9e944e19a465fe8ea944806714dd95416a0821096c78d66b579bd k8s.gcr.io/sig-storage/csi-snapshotter:v4.1.1],SizeBytes:53531990,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:c5be65d6679efabb969d9b019300d187437ae876f992c40911fd2892bbef3b36 k8s.gcr.io/sig-storage/csi-attacher:v3.2.0],SizeBytes:53500826,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:60ab9b3e6a030d3038c87c0d6bca2930f58d1d72823e6a4af09767dc83b696a2 k8s.gcr.io/sig-storage/csi-attacher:v3.2.1],SizeBytes:53499570,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:7a5ba58a44e0d749e0767e4e37315bcf6a61f33ce3185c1991848af4db0fb70a k8s.gcr.io/sig-storage/csi-resizer:v1.1.0],SizeBytes:49176472,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/regression-issue-74839@sha256:b4f1d8d61bdad84bd50442d161d5460e4019d53e989b64220fdbc62fc87d76bf k8s.gcr.io/e2e-test-images/regression-issue-74839:1.2],SizeBytes:44576952,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nonroot@sha256:b9e2958a3dd879e3cf11142228c6d073d0fc4ea2e857c3be6f4fb0ab5fb2c937 k8s.gcr.io/e2e-test-images/nonroot:1.2],SizeBytes:42321438,},ContainerImage{Names:[k8s.gcr.io/sig-storage/hostpathplugin@sha256:232fe80174d60d520d36043103853a1d7ab4b7f3782cf43e45034f04ccda58ce k8s.gcr.io/sig-storage/hostpathplugin:v1.7.1],SizeBytes:29977921,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:a61d309da54641db41fb8f35718f744e9f730d4d0384f8c4b186ddc9f06cbd5f k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.1.0],SizeBytes:19662887,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:19582112,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:2dee3fe5fe861bb66c3a4ac51114f3447a4cd35870e0f2e2b558c7a400d89589 k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.2.0],SizeBytes:18699436,},ContainerImage{Names:[k8s.gcr.io/sig-storage/mock-driver@sha256:a7b517f9e0f42ffade802eef9cefa271372386b85f55b702b493241e58459793 k8s.gcr.io/sig-storage/mock-driver:v4.1.0],SizeBytes:17680993,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:17611980,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:529be2c9770add0cdd0c989115222ea9fc1be430c11095eb9f6dafcf98a36e2b k8s.gcr.io/sig-storage/livenessprobe:v2.4.0],SizeBytes:17175488,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:13616070e3f29de4417eee434a8ef472221c9e51b3d037b5a6b46cef08eb7443 k8s.gcr.io/e2e-test-images/nginx:1.14-2],SizeBytes:16032814,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nonewprivs@sha256:8ac1264691820febacf3aea5d152cbde6d10685731ec14966a9401c6f47a68ac k8s.gcr.io/e2e-test-images/nonewprivs:1.3],SizeBytes:7107254,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf k8s.gcr.io/e2e-test-images/busybox:1.29-2],SizeBytes:1154361,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:39e1e963e5310e9c313bad51523be012ede7b35bb9316517d19089a010356592 k8s.gcr.io/e2e-test-images/busybox:1.29-1],SizeBytes:1154361,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db k8s.gcr.io/pause:3.6 registry.k8s.io/pause:3.6],SizeBytes:682696,},},VolumesInUse:[kubernetes.io/csi/csi-mock-csi-mock-volumes-9402^4 kubernetes.io/csi/ebs.csi.aws.com^vol-02daff5e76e72db98],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-02daff5e76e72db98,DevicePath:,},},Config:nil,},} Jan 9 23:07:42.872: INFO: Logging kubelet events for node ip-172-20-57-251.ca-central-1.compute.internal Jan 9 23:07:42.906: INFO: Logging pods the kubelet thinks is on node ip-172-20-57-251.ca-central-1.compute.internal Jan 9 23:07:42.948: INFO: rs-dzbwc started at 2023-01-09 22:59:56 +0000 UTC (0+1 container statuses recorded) Jan 9 23:07:42.949: INFO: Container donothing ready: false, restart count 0 Jan 9 23:07:42.949: INFO: ebs-csi-node-g4ngc started at 2023-01-09 22:38:46 +0000 UTC (0+3 container statuses recorded) Jan 9 23:07:42.949: INFO: Container ebs-plugin ready: true, restart count 0 Jan 9 23:07:42.949: INFO: Container liveness-probe ready: true, restart count 0 Jan 9 23:07:42.949: INFO: Container node-driver-registrar ready: true, restart count 0 Jan 9 23:07:42.949: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-wmqwf started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:07:42.949: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:07:42.949: INFO: pod-d16d48d3-63ef-4a76-95b8-d80f8b4aec62 started at 2023-01-09 23:03:17 +0000 UTC (0+1 container statuses recorded) Jan 9 23:07:42.949: INFO: Container write-pod ready: false, restart count 0 Jan 9 23:07:42.949: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-m8cfz started at 2023-01-09 22:54:43 +0000 UTC (0+1 container statuses recorded) Jan 9 23:07:42.949: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:07:42.949: INFO: rs-49kqp started at 2023-01-09 22:59:56 +0000 UTC (0+1 container statuses recorded) Jan 9 23:07:42.949: INFO: Container donothing ready: false, restart count 0 Jan 9 23:07:42.949: INFO: netserver-2 started at 2023-01-09 23:07:18 +0000 UTC (0+1 container statuses recorded) Jan 9 23:07:42.949: INFO: Container webserver ready: false, restart count 0 Jan 9 23:07:42.949: INFO: csi-mockplugin-attacher-0 started at 2023-01-09 23:00:20 +0000 UTC (0+1 container statuses recorded) Jan 9 23:07:42.949: INFO: Container csi-attacher ready: false, restart count 0 Jan 9 23:07:42.949: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-rbqs2 started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:07:42.949: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:07:42.949: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-96np5 started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:07:42.949: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:07:42.949: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-f9w29 started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:07:42.949: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:07:42.949: INFO: pause-pod-78b7f9b75c-fckd2 started at 2023-01-09 23:05:44 +0000 UTC (0+1 container statuses recorded) Jan 9 23:07:42.949: INFO: Container agnhost-pause ready: false, restart count 0 Jan 9 23:07:42.949: INFO: csi-mockplugin-0 started at 2023-01-09 23:05:43 +0000 UTC (0+3 container statuses recorded) Jan 9 23:07:42.949: INFO: Container csi-provisioner ready: false, restart count 0 Jan 9 23:07:42.949: INFO: Container driver-registrar ready: false, restart count 0 Jan 9 23:07:42.949: INFO: Container mock ready: false, restart count 0 Jan 9 23:07:42.949: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-zx8gm started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:07:42.949: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:07:42.949: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-qzf5t started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:07:42.949: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:07:42.949: INFO: busybox-scheduling-6815563e-4dc7-43f5-a3cd-3e998a31b84b started at 2023-01-09 23:06:39 +0000 UTC (0+1 container statuses recorded) Jan 9 23:07:42.949: INFO: Container busybox-scheduling-6815563e-4dc7-43f5-a3cd-3e998a31b84b ready: false, restart count 0 Jan 9 23:07:42.949: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-zvfdw started at 2023-01-09 22:54:43 +0000 UTC (0+1 container statuses recorded) Jan 9 23:07:42.949: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:07:42.949: INFO: cilium-6p9jz started at 2023-01-09 22:38:46 +0000 UTC (1+1 container statuses recorded) Jan 9 23:07:42.949: INFO: Init container clean-cilium-state ready: true, restart count 0 Jan 9 23:07:42.949: INFO: Container cilium-agent ready: true, restart count 0 Jan 9 23:07:42.949: INFO: pvc-volume-tester-2vlf5 started at 2023-01-09 23:06:05 +0000 UTC (0+1 container statuses recorded) Jan 9 23:07:42.949: INFO: Container volume-tester ready: false, restart count 0 Jan 9 23:07:42.949: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-wvflq started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:07:42.949: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:07:42.949: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-tvxfq started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:07:42.949: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:07:42.949: INFO: csi-mockplugin-0 started at 2023-01-09 23:00:20 +0000 UTC (0+3 container statuses recorded) Jan 9 23:07:42.949: INFO: Container csi-provisioner ready: true, restart count 0 Jan 9 23:07:42.949: INFO: Container driver-registrar ready: true, restart count 0 Jan 9 23:07:42.949: INFO: Container mock ready: true, restart count 0 Jan 9 23:07:42.949: INFO: rs-56t9b started at 2023-01-09 22:59:56 +0000 UTC (0+1 container statuses recorded) Jan 9 23:07:42.949: INFO: Container donothing ready: true, restart count 0 Jan 9 23:07:43.519: INFO: Latency metrics for node ip-172-20-57-251.ca-central-1.compute.internal Jan 9 23:07:43.519: INFO: Logging node info for node ip-172-20-59-208.ca-central-1.compute.internal Jan 9 23:07:43.551: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-59-208.ca-central-1.compute.internal d061a4d4-9ff6-43fc-89cc-422c61f7441c 45677 0 2023-01-09 22:38:43 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ca-central-1 failure-domain.beta.kubernetes.io/zone:ca-central-1a kubelet_cleanup:true kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-59-208.ca-central-1.compute.internal kubernetes.io/os:linux kubernetes.io/role:node node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:ca-central-1a topology.hostpath.csi/node:ip-172-20-59-208.ca-central-1.compute.internal topology.kubernetes.io/region:ca-central-1 topology.kubernetes.io/zone:ca-central-1a] map[csi.volume.kubernetes.io/nodeid:{"csi-hostpath-ephemeral-7223":"ip-172-20-59-208.ca-central-1.compute.internal","ebs.csi.aws.com":"i-0f00ea8bafbacdd11"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kops-controller Update v1 2023-01-09 22:38:43 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}} } {kube-controller-manager Update v1 2023-01-09 22:38:43 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.1.0/24\"":{}}}} } {kubelet Update v1 2023-01-09 22:38:43 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {e2e.test Update v1 2023-01-09 22:54:42 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubelet_cleanup":{}}}} } {kube-controller-manager Update v1 2023-01-09 23:06:36 +0000 UTC FieldsV1 {"f:status":{"f:volumesAttached":{}}} status} {kubelet Update v1 2023-01-09 23:06:40 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:capacity":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.1.0/24,DoNotUseExternalID:,ProviderID:aws:///ca-central-1a/i-0f00ea8bafbacdd11,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.1.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051652608 0} {<nil>} 3956692Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946795008 0} {<nil>} 3854292Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-01-09 23:06:40 +0000 UTC,LastTransitionTime:2023-01-09 22:38:43 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-01-09 23:06:40 +0000 UTC,LastTransitionTime:2023-01-09 22:38:43 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-01-09 23:06:40 +0000 UTC,LastTransitionTime:2023-01-09 22:38:43 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-01-09 23:06:40 +0000 UTC,LastTransitionTime:2023-01-09 22:39:03 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.59.208,},NodeAddress{Type:ExternalIP,Address:99.79.51.95,},NodeAddress{Type:Hostname,Address:ip-172-20-59-208.ca-central-1.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-59-208.ca-central-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-99-79-51-95.ca-central-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec28d5ecfae27d4bcedd89eca1a365f0,SystemUUID:ec28d5ec-fae2-7d4b-cedd-89eca1a365f0,BootID:560320f7-12c5-44f2-a7f6-a7a40031e204,KernelVersion:5.15.0-1026-aws,OSImage:Ubuntu 22.04.1 LTS,ContainerRuntimeVersion:docker://20.10.17,KubeletVersion:v1.23.15,KubeProxyVersion:v1.23.15,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:456569528,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:20f25f275d46aa728f7615a1ccc19c78b2ed89435bf943a44b339f70f45508e6 k8s.gcr.io/e2e-test-images/httpd:2.4.39-2],SizeBytes:126894770,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:7e8bdd271312fd25fc5ff5a8f04727be84044eb3d7d8d03611972a6752e2e11e k8s.gcr.io/e2e-test-images/agnhost:2.39],SizeBytes:126872991,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:1b9d1b2f36cb2dbee1960e82a9344aeb11bd4c4c03abf5e1853e0559c23855e3 k8s.gcr.io/e2e-test-images/httpd:2.4.38-2],SizeBytes:123781643,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.23.15 registry.k8s.io/kube-proxy-amd64:v1.23.15],SizeBytes:112336036,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:10e231cf07c82dfab5141c1afe548dc734e5fc1f67665ec3982a325d1bd31a9a registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.0],SizeBytes:94471559,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:c8e03f60afa90a28e4bb6ec9a8d0fc36d89de4b7475cf2d613afa793ec969fe0 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.0],SizeBytes:56382329,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:4e74c0492bceddc598de1c90cc5bc14dcda94cb49fa9c5bad9d117c4834b5e08 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.1],SizeBytes:56379269,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:36c31f7e1f433c9634d24f876353e8646246d81a03c4e351202c2644daff1620 k8s.gcr.io/sig-storage/csi-resizer:v1.2.0],SizeBytes:53956308,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:ed98431376c9e944e19a465fe8ea944806714dd95416a0821096c78d66b579bd k8s.gcr.io/sig-storage/csi-snapshotter:v4.1.1],SizeBytes:53531990,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:c5be65d6679efabb969d9b019300d187437ae876f992c40911fd2892bbef3b36 k8s.gcr.io/sig-storage/csi-attacher:v3.2.0],SizeBytes:53500826,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:60ab9b3e6a030d3038c87c0d6bca2930f58d1d72823e6a4af09767dc83b696a2 k8s.gcr.io/sig-storage/csi-attacher:v3.2.1],SizeBytes:53499570,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:017727efcfeb7d053af68e51436ce8e65edbc6ca573720afb4f79c8594036955 registry.k8s.io/coredns/coredns:v1.10.0],SizeBytes:50249337,},ContainerImage{Names:[registry.k8s.io/cpa/cluster-proportional-autoscaler@sha256:68d396900aeaa072c1f27289485fdac29834045a6f3ffe369bf389d830ef572d registry.k8s.io/cpa/cluster-proportional-autoscaler:1.8.6],SizeBytes:47910609,},ContainerImage{Names:[k8s.gcr.io/sig-storage/hostpathplugin@sha256:232fe80174d60d520d36043103853a1d7ab4b7f3782cf43e45034f04ccda58ce k8s.gcr.io/sig-storage/hostpathplugin:v1.7.1],SizeBytes:29977921,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:a61d309da54641db41fb8f35718f744e9f730d4d0384f8c4b186ddc9f06cbd5f k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.1.0],SizeBytes:19662887,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:19582112,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:2dee3fe5fe861bb66c3a4ac51114f3447a4cd35870e0f2e2b558c7a400d89589 k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.2.0],SizeBytes:18699436,},ContainerImage{Names:[k8s.gcr.io/sig-storage/mock-driver@sha256:a7b517f9e0f42ffade802eef9cefa271372386b85f55b702b493241e58459793 k8s.gcr.io/sig-storage/mock-driver:v4.1.0],SizeBytes:17680993,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:17611980,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:529be2c9770add0cdd0c989115222ea9fc1be430c11095eb9f6dafcf98a36e2b k8s.gcr.io/sig-storage/livenessprobe:v2.4.0],SizeBytes:17175488,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:13616070e3f29de4417eee434a8ef472221c9e51b3d037b5a6b46cef08eb7443 k8s.gcr.io/e2e-test-images/nginx:1.14-2],SizeBytes:16032814,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nonewprivs@sha256:8ac1264691820febacf3aea5d152cbde6d10685731ec14966a9401c6f47a68ac k8s.gcr.io/e2e-test-images/nonewprivs:1.3],SizeBytes:7107254,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf k8s.gcr.io/e2e-test-images/busybox:1.29-2],SizeBytes:1154361,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:39e1e963e5310e9c313bad51523be012ede7b35bb9316517d19089a010356592 k8s.gcr.io/e2e-test-images/busybox:1.29-1],SizeBytes:1154361,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db k8s.gcr.io/pause:3.6 registry.k8s.io/pause:3.6],SizeBytes:682696,},},VolumesInUse:[kubernetes.io/csi/csi-hostpath-ephemeral-7223^43b8adc9-9072-11ed-b4b6-0a454e7330ed kubernetes.io/csi/csi-hostpath-ephemeral-7223^43b8d4d5-9072-11ed-b4b6-0a454e7330ed],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/csi-hostpath-ephemeral-7223^43b8d4d5-9072-11ed-b4b6-0a454e7330ed,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/csi-hostpath-ephemeral-7223^43b8adc9-9072-11ed-b4b6-0a454e7330ed,DevicePath:,},},Config:nil,},} Jan 9 23:07:43.551: INFO: Logging kubelet events for node ip-172-20-59-208.ca-central-1.compute.internal Jan 9 23:07:43.621: INFO: Logging pods the kubelet thinks is on node ip-172-20-59-208.ca-central-1.compute.internal Jan 9 23:07:43.673: INFO: cilium-z2ltf started at 2023-01-09 22:38:44 +0000 UTC (1+1 container statuses recorded) Jan 9 23:07:43.673: INFO: Init container clean-cilium-state ready: true, restart count 0 Jan 9 23:07:43.673: INFO: Container cilium-agent ready: true, restart count 0 Jan 9 23:07:43.673: INFO: csi-hostpathplugin-0 started at 2023-01-09 23:03:05 +0000 UTC (0+7 container statuses recorded) Jan 9 23:07:43.673: INFO: Container csi-attacher ready: true, restart count 0 Jan 9 23:07:43.673: INFO: Container csi-provisioner ready: true, restart count 0 Jan 9 23:07:43.673: INFO: Container csi-resizer ready: true, restart count 0 Jan 9 23:07:43.673: INFO: Container csi-snapshotter ready: true, restart count 0 Jan 9 23:07:43.673: INFO: Container hostpath ready: true, restart count 0 Jan 9 23:07:43.673: INFO: Container liveness-probe ready: true, restart count 0 Jan 9 23:07:43.673: INFO: Container node-driver-registrar ready: true, restart count 0 Jan 9 23:07:43.673: INFO: hostexec-ip-172-20-59-208.ca-central-1.compute.internal-sn7w7 started at 2023-01-09 23:06:26 +0000 UTC (0+1 container statuses recorded) Jan 9 23:07:43.673: INFO: Container agnhost-container ready: true, restart count 0 Jan 9 23:07:43.673: INFO: inline-volume-tester-xnqfb started at 2023-01-09 23:06:35 +0000 UTC (0+1 container statuses recorded) Jan 9 23:07:43.673: INFO: Container csi-volume-tester ready: false, restart count 0 Jan 9 23:07:43.673: INFO: coredns-867df8f45c-tmrqb started at 2023-01-09 22:39:03 +0000 UTC (0+1 container statuses recorded) Jan 9 23:07:43.673: INFO: Container coredns ready: true, restart count 0 Jan 9 23:07:43.673: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-8qp6l started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:07:43.673: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: false, restart count 0 Jan 9 23:07:43.673: INFO: rs-nd6n2 started at 2023-01-09 22:59:56 +0000 UTC (0+1 container statuses recorded) Jan 9 23:07:43.673: INFO: Container donothing ready: false, restart count 0 Jan 9 23:07:43.673: INFO: coredns-autoscaler-557ccb4c66-jnjcc started at 2023-01-09 22:39:03 +0000 UTC (0+1 container statuses recorded) Jan 9 23:07:43.673: INFO: Container autoscaler ready: true, restart count 0 Jan 9 23:07:43.673: INFO: netserver-3 started at 2023-01-09 23:07:18 +0000 UTC (0+1 container statuses recorded) Jan 9 23:07:43.673: INFO: Container webserver ready: false, restart count 0 Jan 9 23:07:43.673: INFO: rs-x69wn started at 2023-01-09 22:59:56 +0000 UTC (0+1 container statuses recorded) Jan 9 23:07:43.673: INFO: Container donothing ready: false, restart count 0 Jan 9 23:07:43.673: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-hszdf started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:07:43.673: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:07:43.673: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-fc6pg started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:07:43.673: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:07:43.673: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-bhxkw started at 2023-01-09 22:54:43 +0000 UTC (0+1 container statuses recorded) Jan 9 23:07:43.673: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:07:43.673: INFO: pod-e1224014-09b8-42e1-b210-71a9acc98e21 started at 2023-01-09 23:06:29 +0000 UTC (0+1 container statuses recorded) Jan 9 23:07:43.673: INFO: Container write-pod ready: false, restart count 0 Jan 9 23:07:43.673: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-rzgjf started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:07:43.673: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:07:43.673: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-ptfgl started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:07:43.673: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:07:43.673: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-t5g85 started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:07:43.673: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:07:43.673: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-hjsbl started at 2023-01-09 22:54:43 +0000 UTC (0+1 container statuses recorded) Jan 9 23:07:43.673: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:07:43.673: INFO: ebs-csi-node-prrrf started at 2023-01-09 22:38:44 +0000 UTC (0+3 container statuses recorded) Jan 9 23:07:43.673: INFO: Container ebs-plugin ready: true, restart count 0 Jan 9 23:07:43.673: INFO: Container liveness-probe ready: true, restart count 0 Jan 9 23:07:43.673: INFO: Container node-driver-registrar ready: true, restart count 0 Jan 9 23:07:43.673: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-x682f started at 2023-01-09 22:54:43 +0000 UTC (0+1 container statuses recorded) Jan 9 23:07:43.673: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:07:43.673: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-tx2mm started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:07:43.673: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:07:43.854: INFO: Latency metrics for node ip-172-20-59-208.ca-central-1.compute.internal Jan 9 23:07:43.855: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready �[1mSTEP�[0m: Destroying namespace "provisioning-3409" for this suite.
Find pod-9ba7c444-540c-4612-ad00-c7c5dd3438eb mentions in log files
go run hack/e2e.go -v --test --test_args='--ginkgo.focus=Kubernetes\se2e\ssuite\sExternal\sStorage\s\[Driver\:\sebs\.csi\.aws\.com\]\s\[Testpattern\:\sGeneric\sEphemeral\-volume\s\(default\sfs\)\s\(immediate\-binding\)\]\sephemeral\sshould\screate\sread\-only\sinline\sephemeral\svolume$'
/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/ephemeral.go:173 Jan 9 23:11:34.559: waiting for pod with inline volume Unexpected error: <*errors.errorString | 0xc0002b82b0>: { s: "timed out waiting for the condition", } timed out waiting for the condition occurred /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/ephemeral.go:349from junit_20.xml
[BeforeEach] [Testpattern: Generic Ephemeral-volume (default fs) (immediate-binding)] ephemeral /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/framework/testsuite.go:51 [BeforeEach] [Testpattern: Generic Ephemeral-volume (default fs) (immediate-binding)] ephemeral /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:185 �[1mSTEP�[0m: Creating a kubernetes client Jan 9 22:56:31.922: INFO: >>> kubeConfig: /root/.kube/config �[1mSTEP�[0m: Building a namespace api object, basename ephemeral �[1mSTEP�[0m: Waiting for a default service account to be provisioned in namespace �[1mSTEP�[0m: Waiting for kube-root-ca.crt to be provisioned in namespace [It] should create read-only inline ephemeral volume /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/ephemeral.go:173 Jan 9 22:56:32.252: INFO: Pod inline-volume-dq9q9 has the following logs: Jan 9 22:56:32.285: INFO: Deleting pod "inline-volume-dq9q9" in namespace "ephemeral-9413" Jan 9 22:56:32.333: INFO: Wait up to 5m0s for pod "inline-volume-dq9q9" to be fully deleted Jan 9 22:56:34.397: INFO: Creating resource for dynamic PV Jan 9 22:56:34.397: INFO: Using claimSize:1Gi, test suite supported size:{ }, driver(ebs.csi.aws.com) supported size:{ } �[1mSTEP�[0m: creating a StorageClass ephemeral-9413-e2e-scdk4ks �[1mSTEP�[0m: checking the requested inline volume exists in the pod running on node {Name: Selector:map[] Affinity:nil} Jan 9 23:11:34.559: FAIL: waiting for pod with inline volume Unexpected error: <*errors.errorString | 0xc0002b82b0>: { s: "timed out waiting for the condition", } timed out waiting for the condition occurred Full Stack Trace k8s.io/kubernetes/test/e2e/storage/testsuites.EphemeralTest.TestEphemeral({{0x7b06bd0, 0xc001f5a180}, 0xc002346d80, {0xc002879460, 0xe}, {0x0, 0x0}, 0xc001462f00, {{0x0, 0x0}, ...}, ...}) /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/ephemeral.go:349 +0x60a k8s.io/kubernetes/test/e2e/storage/testsuites.(*ephemeralTestSuite).DefineTests.func3() /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/ephemeral.go:191 +0x1d8 k8s.io/kubernetes/test/e2e.RunE2ETests(0x24c66d7) _output/dockerized/go/src/k8s.io/kubernetes/test/e2e/e2e.go:133 +0x697 k8s.io/kubernetes/test/e2e.TestE2E(0x0) _output/dockerized/go/src/k8s.io/kubernetes/test/e2e/e2e_test.go:136 +0x19 testing.tRunner(0xc0001cad00, 0x735e880) /usr/local/go/src/testing/testing.go:1259 +0x102 created by testing.(*T).Run /usr/local/go/src/testing/testing.go:1306 +0x35a Jan 9 23:11:34.593: INFO: Error getting logs for pod inline-volume-tester-7mx4s: the server rejected our request for an unknown reason (get pods inline-volume-tester-7mx4s) Jan 9 23:11:34.657: INFO: Deleting pod "inline-volume-tester-7mx4s" in namespace "ephemeral-9413" Jan 9 23:11:34.690: INFO: Wait up to 5m0s for pod "inline-volume-tester-7mx4s" to be fully deleted Jan 9 23:11:44.755: INFO: Wait up to 5m0s for pod PV pvc-4bcb7fdd-1982-4f6b-8a6e-5860526c6820 to be fully deleted Jan 9 23:11:44.755: INFO: Waiting up to 5m0s for PersistentVolume pvc-4bcb7fdd-1982-4f6b-8a6e-5860526c6820 to get deleted Jan 9 23:11:44.786: INFO: PersistentVolume pvc-4bcb7fdd-1982-4f6b-8a6e-5860526c6820 found and phase=Released (31.106031ms) Jan 9 23:11:49.819: INFO: PersistentVolume pvc-4bcb7fdd-1982-4f6b-8a6e-5860526c6820 found and phase=Released (5.063801326s) Jan 9 23:11:54.961: INFO: PersistentVolume pvc-4bcb7fdd-1982-4f6b-8a6e-5860526c6820 was removed �[1mSTEP�[0m: Deleting sc [AfterEach] [Testpattern: Generic Ephemeral-volume (default fs) (immediate-binding)] ephemeral /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:186 �[1mSTEP�[0m: Collecting events from namespace "ephemeral-9413". �[1mSTEP�[0m: Found 20 events. Jan 9 23:11:55.042: INFO: At 2023-01-09 22:56:32 +0000 UTC - event for inline-volume-dq9q9: {default-scheduler } FailedScheduling: 0/5 nodes are available: 5 waiting for ephemeral volume controller to create the persistentvolumeclaim "inline-volume-dq9q9-my-volume". Jan 9 23:11:55.042: INFO: At 2023-01-09 22:56:32 +0000 UTC - event for inline-volume-dq9q9-my-volume: {persistentvolume-controller } ProvisioningFailed: storageclass.storage.k8s.io "no-such-storage-class" not found Jan 9 23:11:55.042: INFO: At 2023-01-09 22:56:34 +0000 UTC - event for inline-volume-tester-7mx4s: {default-scheduler } FailedScheduling: 0/5 nodes are available: 5 waiting for ephemeral volume controller to create the persistentvolumeclaim "inline-volume-tester-7mx4s-my-volume-0". Jan 9 23:11:55.042: INFO: At 2023-01-09 22:56:34 +0000 UTC - event for inline-volume-tester-7mx4s-my-volume-0: {persistentvolume-controller } ExternalProvisioning: waiting for a volume to be created, either by external provisioner "ebs.csi.aws.com" or manually created by system administrator Jan 9 23:11:55.042: INFO: At 2023-01-09 22:56:34 +0000 UTC - event for inline-volume-tester-7mx4s-my-volume-0: {ebs.csi.aws.com_ebs-csi-controller-5d8d7cc949-8wxx2_35215d1d-5b9d-46a9-b0b7-d7677b375a3a } Provisioning: External provisioner is provisioning volume for claim "ephemeral-9413/inline-volume-tester-7mx4s-my-volume-0" Jan 9 23:11:55.042: INFO: At 2023-01-09 22:56:35 +0000 UTC - event for inline-volume-tester-7mx4s: {default-scheduler } FailedScheduling: 0/5 nodes are available: 5 pod has unbound immediate PersistentVolumeClaims. Jan 9 23:11:55.042: INFO: At 2023-01-09 22:56:37 +0000 UTC - event for inline-volume-tester-7mx4s-my-volume-0: {ebs.csi.aws.com_ebs-csi-controller-5d8d7cc949-8wxx2_35215d1d-5b9d-46a9-b0b7-d7677b375a3a } ProvisioningSucceeded: Successfully provisioned volume pvc-4bcb7fdd-1982-4f6b-8a6e-5860526c6820 Jan 9 23:11:55.042: INFO: At 2023-01-09 22:56:42 +0000 UTC - event for inline-volume-tester-7mx4s: {default-scheduler } Scheduled: Successfully assigned ephemeral-9413/inline-volume-tester-7mx4s to ip-172-20-55-216.ca-central-1.compute.internal Jan 9 23:11:55.042: INFO: At 2023-01-09 22:56:45 +0000 UTC - event for inline-volume-tester-7mx4s: {attachdetach-controller } SuccessfulAttachVolume: AttachVolume.Attach succeeded for volume "pvc-4bcb7fdd-1982-4f6b-8a6e-5860526c6820" Jan 9 23:11:55.042: INFO: At 2023-01-09 22:56:53 +0000 UTC - event for inline-volume-tester-7mx4s: {kubelet ip-172-20-55-216.ca-central-1.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "084a8260083327b03f190101424b8e133deb6e7bc094a14f79c7a77061dcfc3d" network for pod "inline-volume-tester-7mx4s": networkPlugin cni failed to set up pod "inline-volume-tester-7mx4s_ephemeral-9413" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 9 23:11:55.042: INFO: At 2023-01-09 22:56:55 +0000 UTC - event for inline-volume-tester-7mx4s: {kubelet ip-172-20-55-216.ca-central-1.compute.internal} SandboxChanged: Pod sandbox changed, it will be killed and re-created. Jan 9 23:11:55.042: INFO: At 2023-01-09 22:56:57 +0000 UTC - event for inline-volume-tester-7mx4s: {kubelet ip-172-20-55-216.ca-central-1.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "5eb0b7c26d607b0849d86765c27b70c5b376470e435da45059e3fccbfa32f990" network for pod "inline-volume-tester-7mx4s": networkPlugin cni failed to set up pod "inline-volume-tester-7mx4s_ephemeral-9413" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 9 23:11:55.042: INFO: At 2023-01-09 22:57:01 +0000 UTC - event for inline-volume-tester-7mx4s: {kubelet ip-172-20-55-216.ca-central-1.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "3d28f86aa4d87b2bdb7a91a381093e8aa3acb86e8a0037ec7161884c869c2613" network for pod "inline-volume-tester-7mx4s": networkPlugin cni failed to set up pod "inline-volume-tester-7mx4s_ephemeral-9413" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 9 23:11:55.042: INFO: At 2023-01-09 22:57:06 +0000 UTC - event for inline-volume-tester-7mx4s: {kubelet ip-172-20-55-216.ca-central-1.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "81f6161e3894ebfc1617925748640da5145e8cd40ec75d5d141c3c269532fc18" network for pod "inline-volume-tester-7mx4s": networkPlugin cni failed to set up pod "inline-volume-tester-7mx4s_ephemeral-9413" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 9 23:11:55.042: INFO: At 2023-01-09 22:57:08 +0000 UTC - event for inline-volume-tester-7mx4s: {kubelet ip-172-20-55-216.ca-central-1.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "6de07f289557dcb050c6e44d29ec328f957ece045a6d203f39f6324295f17790" network for pod "inline-volume-tester-7mx4s": networkPlugin cni failed to set up pod "inline-volume-tester-7mx4s_ephemeral-9413" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 9 23:11:55.042: INFO: At 2023-01-09 22:57:10 +0000 UTC - event for inline-volume-tester-7mx4s: {kubelet ip-172-20-55-216.ca-central-1.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "e05eeb5299b53a99262f160c6f1f5b71f64efab5e7bd0de2f1800ca8d48fba05" network for pod "inline-volume-tester-7mx4s": networkPlugin cni failed to set up pod "inline-volume-tester-7mx4s_ephemeral-9413" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 9 23:11:55.042: INFO: At 2023-01-09 22:57:12 +0000 UTC - event for inline-volume-tester-7mx4s: {kubelet ip-172-20-55-216.ca-central-1.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "88c24dfc8b27c3472d876d8dce59c8e73577afb7d11aebfdb3f76b58f4db8fa4" network for pod "inline-volume-tester-7mx4s": networkPlugin cni failed to set up pod "inline-volume-tester-7mx4s_ephemeral-9413" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 9 23:11:55.042: INFO: At 2023-01-09 22:57:14 +0000 UTC - event for inline-volume-tester-7mx4s: {kubelet ip-172-20-55-216.ca-central-1.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "50f47642ebe53f28ed846fa16401ca5988b3e2a178df231549d23e515700fb60" network for pod "inline-volume-tester-7mx4s": networkPlugin cni failed to set up pod "inline-volume-tester-7mx4s_ephemeral-9413" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 9 23:11:55.042: INFO: At 2023-01-09 22:57:17 +0000 UTC - event for inline-volume-tester-7mx4s: {kubelet ip-172-20-55-216.ca-central-1.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "8b6655785dcce6fb5bdbfe9f9d6cc3726d3f9a0c0c8e2d7090468786d05db8ac" network for pod "inline-volume-tester-7mx4s": networkPlugin cni failed to set up pod "inline-volume-tester-7mx4s_ephemeral-9413" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 9 23:11:55.042: INFO: At 2023-01-09 22:57:20 +0000 UTC - event for inline-volume-tester-7mx4s: {kubelet ip-172-20-55-216.ca-central-1.compute.internal} FailedCreatePodSandBox: (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "4bbf0c63efe00a1040b22482b79d5908b3c037ce439385968fae2173617ab977" network for pod "inline-volume-tester-7mx4s": networkPlugin cni failed to set up pod "inline-volume-tester-7mx4s_ephemeral-9413" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 9 23:11:55.079: INFO: POD NODE PHASE GRACE CONDITIONS Jan 9 23:11:55.079: INFO: Jan 9 23:11:55.126: INFO: Logging node info for node ip-172-20-32-67.ca-central-1.compute.internal Jan 9 23:11:55.164: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-32-67.ca-central-1.compute.internal 1f51981b-a01d-404c-b3a4-6649f301f65d 47775 0 2023-01-09 22:38:45 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ca-central-1 failure-domain.beta.kubernetes.io/zone:ca-central-1a kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-32-67.ca-central-1.compute.internal kubernetes.io/os:linux kubernetes.io/role:node node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:ca-central-1a topology.hostpath.csi/node:ip-172-20-32-67.ca-central-1.compute.internal topology.kubernetes.io/region:ca-central-1 topology.kubernetes.io/zone:ca-central-1a] map[csi.volume.kubernetes.io/nodeid:{"csi-hostpath-ephemeral-7839":"ip-172-20-32-67.ca-central-1.compute.internal","ebs.csi.aws.com":"i-0272cebea4c6c3d5d"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kops-controller Update v1 2023-01-09 22:38:45 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}} } {kube-controller-manager Update v1 2023-01-09 22:38:45 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.3.0/24\"":{}}}} } {kubelet Update v1 2023-01-09 22:38:45 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {kube-controller-manager Update v1 2023-01-09 22:55:59 +0000 UTC FieldsV1 {"f:status":{"f:volumesAttached":{}}} status} {kubelet Update v1 2023-01-09 22:56:00 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:capacity":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.3.0/24,DoNotUseExternalID:,ProviderID:aws:///ca-central-1a/i-0272cebea4c6c3d5d,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.3.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051644416 0} {<nil>} 3956684Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946786816 0} {<nil>} 3854284Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-01-09 23:11:18 +0000 UTC,LastTransitionTime:2023-01-09 22:38:45 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-01-09 23:11:18 +0000 UTC,LastTransitionTime:2023-01-09 22:38:45 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-01-09 23:11:18 +0000 UTC,LastTransitionTime:2023-01-09 22:38:45 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-01-09 23:11:18 +0000 UTC,LastTransitionTime:2023-01-09 22:39:06 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.32.67,},NodeAddress{Type:ExternalIP,Address:35.182.2.102,},NodeAddress{Type:Hostname,Address:ip-172-20-32-67.ca-central-1.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-32-67.ca-central-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-35-182-2-102.ca-central-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2536b48bd6b6dade0ed68af2555085,SystemUUID:ec2536b4-8bd6-b6da-de0e-d68af2555085,BootID:bfb3b0df-aafa-481c-9ac6-f85e5e4580e3,KernelVersion:5.15.0-1026-aws,OSImage:Ubuntu 22.04.1 LTS,ContainerRuntimeVersion:docker://20.10.17,KubeletVersion:v1.23.15,KubeProxyVersion:v1.23.15,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:456569528,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/jessie-dnsutils@sha256:11e6a66017ba4e4b938c1612b7a54a3befcefd354796c04e1dba76873a13518e k8s.gcr.io/e2e-test-images/jessie-dnsutils:1.5],SizeBytes:253371792,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:7e8bdd271312fd25fc5ff5a8f04727be84044eb3d7d8d03611972a6752e2e11e k8s.gcr.io/e2e-test-images/agnhost:2.39],SizeBytes:126872991,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nautilus@sha256:99c0d6f1ad24a1aa1905d9c6534d193f268f7b23f9add2ae6bb41f31094bdd5c k8s.gcr.io/e2e-test-images/nautilus:1.5],SizeBytes:124758741,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:1b9d1b2f36cb2dbee1960e82a9344aeb11bd4c4c03abf5e1853e0559c23855e3 k8s.gcr.io/e2e-test-images/httpd:2.4.38-2],SizeBytes:123781643,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.23.15 registry.k8s.io/kube-proxy-amd64:v1.23.15],SizeBytes:112336036,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:10e231cf07c82dfab5141c1afe548dc734e5fc1f67665ec3982a325d1bd31a9a registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.0],SizeBytes:94471559,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:c8e03f60afa90a28e4bb6ec9a8d0fc36d89de4b7475cf2d613afa793ec969fe0 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.0],SizeBytes:56382329,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:4e74c0492bceddc598de1c90cc5bc14dcda94cb49fa9c5bad9d117c4834b5e08 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.1],SizeBytes:56379269,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:36c31f7e1f433c9634d24f876353e8646246d81a03c4e351202c2644daff1620 k8s.gcr.io/sig-storage/csi-resizer:v1.2.0],SizeBytes:53956308,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:ed98431376c9e944e19a465fe8ea944806714dd95416a0821096c78d66b579bd k8s.gcr.io/sig-storage/csi-snapshotter:v4.1.1],SizeBytes:53531990,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:c5be65d6679efabb969d9b019300d187437ae876f992c40911fd2892bbef3b36 k8s.gcr.io/sig-storage/csi-attacher:v3.2.0],SizeBytes:53500826,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:60ab9b3e6a030d3038c87c0d6bca2930f58d1d72823e6a4af09767dc83b696a2 k8s.gcr.io/sig-storage/csi-attacher:v3.2.1],SizeBytes:53499570,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:017727efcfeb7d053af68e51436ce8e65edbc6ca573720afb4f79c8594036955 registry.k8s.io/coredns/coredns:v1.10.0],SizeBytes:50249337,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:7a5ba58a44e0d749e0767e4e37315bcf6a61f33ce3185c1991848af4db0fb70a k8s.gcr.io/sig-storage/csi-resizer:v1.1.0],SizeBytes:49176472,},ContainerImage{Names:[k8s.gcr.io/sig-storage/hostpathplugin@sha256:232fe80174d60d520d36043103853a1d7ab4b7f3782cf43e45034f04ccda58ce k8s.gcr.io/sig-storage/hostpathplugin:v1.7.1],SizeBytes:29977921,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:a61d309da54641db41fb8f35718f744e9f730d4d0384f8c4b186ddc9f06cbd5f k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.1.0],SizeBytes:19662887,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:19582112,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:2dee3fe5fe861bb66c3a4ac51114f3447a4cd35870e0f2e2b558c7a400d89589 k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.2.0],SizeBytes:18699436,},ContainerImage{Names:[k8s.gcr.io/sig-storage/mock-driver@sha256:a7b517f9e0f42ffade802eef9cefa271372386b85f55b702b493241e58459793 k8s.gcr.io/sig-storage/mock-driver:v4.1.0],SizeBytes:17680993,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:17611980,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:529be2c9770add0cdd0c989115222ea9fc1be430c11095eb9f6dafcf98a36e2b k8s.gcr.io/sig-storage/livenessprobe:v2.4.0],SizeBytes:17175488,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:13616070e3f29de4417eee434a8ef472221c9e51b3d037b5a6b46cef08eb7443 k8s.gcr.io/e2e-test-images/nginx:1.14-2],SizeBytes:16032814,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nonewprivs@sha256:8ac1264691820febacf3aea5d152cbde6d10685731ec14966a9401c6f47a68ac k8s.gcr.io/e2e-test-images/nonewprivs:1.3],SizeBytes:7107254,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf k8s.gcr.io/e2e-test-images/busybox:1.29-2],SizeBytes:1154361,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:39e1e963e5310e9c313bad51523be012ede7b35bb9316517d19089a010356592 k8s.gcr.io/e2e-test-images/busybox:1.29-1],SizeBytes:1154361,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db k8s.gcr.io/pause:3.6 registry.k8s.io/pause:3.6],SizeBytes:682696,},},VolumesInUse:[kubernetes.io/csi/ebs.csi.aws.com^vol-01e8a4a8ff1da0e10],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-01e8a4a8ff1da0e10,DevicePath:,},},Config:nil,},} Jan 9 23:11:55.165: INFO: Logging kubelet events for node ip-172-20-32-67.ca-central-1.compute.internal Jan 9 23:11:55.202: INFO: Logging pods the kubelet thinks is on node ip-172-20-32-67.ca-central-1.compute.internal Jan 9 23:11:55.258: INFO: ebs-csi-node-lc4rb started at 2023-01-09 22:38:46 +0000 UTC (0+3 container statuses recorded) Jan 9 23:11:55.258: INFO: Container ebs-plugin ready: true, restart count 0 Jan 9 23:11:55.258: INFO: Container liveness-probe ready: true, restart count 0 Jan 9 23:11:55.258: INFO: Container node-driver-registrar ready: true, restart count 0 Jan 9 23:11:55.258: INFO: netserver-0 started at 2023-01-09 23:07:18 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:55.258: INFO: Container webserver ready: false, restart count 0 Jan 9 23:11:55.258: INFO: netserver-0 started at 2023-01-09 23:08:45 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:55.258: INFO: Container webserver ready: false, restart count 0 Jan 9 23:11:55.258: INFO: inline-volume-tester-5g6j6 started at 2023-01-09 22:55:56 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:55.258: INFO: Container csi-volume-tester ready: true, restart count 0 Jan 9 23:11:55.258: INFO: csi-hostpathplugin-0 started at 2023-01-09 23:05:41 +0000 UTC (0+7 container statuses recorded) Jan 9 23:11:55.258: INFO: Container csi-attacher ready: false, restart count 0 Jan 9 23:11:55.258: INFO: Container csi-provisioner ready: false, restart count 0 Jan 9 23:11:55.258: INFO: Container csi-resizer ready: false, restart count 0 Jan 9 23:11:55.258: INFO: Container csi-snapshotter ready: false, restart count 0 Jan 9 23:11:55.258: INFO: Container hostpath ready: false, restart count 0 Jan 9 23:11:55.258: INFO: Container liveness-probe ready: false, restart count 0 Jan 9 23:11:55.258: INFO: Container node-driver-registrar ready: false, restart count 0 Jan 9 23:11:55.258: INFO: cilium-dvsfs started at 2023-01-09 22:38:46 +0000 UTC (1+1 container statuses recorded) Jan 9 23:11:55.258: INFO: Init container clean-cilium-state ready: true, restart count 0 Jan 9 23:11:55.258: INFO: Container cilium-agent ready: true, restart count 0 Jan 9 23:11:55.258: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-gjc2h started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:55.258: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:11:55.258: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-jb6kp started at 2023-01-09 22:54:43 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:55.258: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:11:55.258: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-hwbbn started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:55.258: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:11:55.258: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-fr98x started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:55.258: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:11:55.258: INFO: coredns-867df8f45c-sckrz started at 2023-01-09 22:39:23 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:55.258: INFO: Container coredns ready: true, restart count 0 Jan 9 23:11:55.258: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-fzrxx started at 2023-01-09 22:54:43 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:55.258: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:11:55.258: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-mlcl4 started at 2023-01-09 22:54:43 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:55.258: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:11:55.428: INFO: Latency metrics for node ip-172-20-32-67.ca-central-1.compute.internal Jan 9 23:11:55.428: INFO: Logging node info for node ip-172-20-37-64.ca-central-1.compute.internal Jan 9 23:11:55.469: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-37-64.ca-central-1.compute.internal d817613a-0fe5-40c0-a5e4-7b85b9f8506b 46487 0 2023-01-09 22:36:54 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:c5.large beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ca-central-1 failure-domain.beta.kubernetes.io/zone:ca-central-1a kops.k8s.io/kops-controller-pki: kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-37-64.ca-central-1.compute.internal kubernetes.io/os:linux kubernetes.io/role:master node-role.kubernetes.io/control-plane: node-role.kubernetes.io/master: node.kubernetes.io/exclude-from-external-load-balancers: node.kubernetes.io/instance-type:c5.large topology.ebs.csi.aws.com/zone:ca-central-1a topology.kubernetes.io/region:ca-central-1 topology.kubernetes.io/zone:ca-central-1a] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-06217673764fdafeb"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kubelet Update v1 2023-01-09 22:36:54 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {protokube Update v1 2023-01-09 22:37:19 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kops.k8s.io/kops-controller-pki":{},"f:node-role.kubernetes.io/control-plane":{},"f:node.kubernetes.io/exclude-from-external-load-balancers":{}}}} } {kops-controller Update v1 2023-01-09 22:37:21 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/master":{}}}} } {kube-controller-manager Update v1 2023-01-09 22:37:35 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.0.0/24\"":{}},"f:taints":{}}} } {kubelet Update v1 2023-01-09 22:38:39 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.0.0/24,DoNotUseExternalID:,ProviderID:aws:///ca-central-1a/i-06217673764fdafeb,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/master,Value:,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[100.96.0.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3862904832 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3758047232 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-01-09 23:09:24 +0000 UTC,LastTransitionTime:2023-01-09 22:36:48 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-01-09 23:09:24 +0000 UTC,LastTransitionTime:2023-01-09 22:36:48 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-01-09 23:09:24 +0000 UTC,LastTransitionTime:2023-01-09 22:36:48 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-01-09 23:09:24 +0000 UTC,LastTransitionTime:2023-01-09 22:37:35 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.37.64,},NodeAddress{Type:ExternalIP,Address:3.96.216.165,},NodeAddress{Type:Hostname,Address:ip-172-20-37-64.ca-central-1.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-37-64.ca-central-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-3-96-216-165.ca-central-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec27a483f61319a98e5ba2ee2d59e6a3,SystemUUID:ec27a483-f613-19a9-8e5b-a2ee2d59e6a3,BootID:9b41e339-1aaa-4374-8f11-8d34e12cd62a,KernelVersion:5.15.0-1026-aws,OSImage:Ubuntu 22.04.1 LTS,ContainerRuntimeVersion:docker://20.10.17,KubeletVersion:v1.23.15,KubeProxyVersion:v1.23.15,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[registry.k8s.io/etcdadm/etcd-manager@sha256:66a453db625abb268f4b3bbefc5a34a171d81e6e8796cecca54cfd71775c77c4 registry.k8s.io/etcdadm/etcd-manager:v3.0.20221209],SizeBytes:662209237,},ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:456569528,},ContainerImage{Names:[k8s.gcr.io/kube-apiserver-amd64:v1.23.15 registry.k8s.io/kube-apiserver-amd64:v1.23.15],SizeBytes:135240444,},ContainerImage{Names:[k8s.gcr.io/kube-controller-manager-amd64:v1.23.15 registry.k8s.io/kube-controller-manager-amd64:v1.23.15],SizeBytes:125046097,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.23.15 registry.k8s.io/kube-proxy-amd64:v1.23.15],SizeBytes:112336036,},ContainerImage{Names:[quay.io/cilium/operator@sha256:a6d24a006a6b92967ac90786b49bc1ac26e5477cf028cd1186efcfc2466484db quay.io/cilium/operator:v1.12.5],SizeBytes:110016956,},ContainerImage{Names:[registry.k8s.io/kops/kops-controller:1.26.0-beta.1],SizeBytes:104212800,},ContainerImage{Names:[registry.k8s.io/kops/dns-controller:1.26.0-beta.1],SizeBytes:102566451,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:10e231cf07c82dfab5141c1afe548dc734e5fc1f67665ec3982a325d1bd31a9a registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.0],SizeBytes:94471559,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:122bfb8c1edabb3c0edd63f06523e6940d958d19b3957dc7b1d6f81e9f1f6119 registry.k8s.io/sig-storage/csi-provisioner:v3.1.0],SizeBytes:57736070,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:9ebbf9f023e7b41ccee3d52afe39a89e3ddacdbb69269d583abfc25847cfd9e4 registry.k8s.io/sig-storage/csi-resizer:v1.4.0],SizeBytes:55464778,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:8b9c313c05f54fb04f8d430896f5f5904b6cb157df261501b29adc04d2b2dc7b registry.k8s.io/sig-storage/csi-attacher:v3.4.0],SizeBytes:54848595,},ContainerImage{Names:[k8s.gcr.io/kube-scheduler-amd64:v1.23.15 registry.k8s.io/kube-scheduler-amd64:v1.23.15],SizeBytes:53521430,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:19582112,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:17611980,},ContainerImage{Names:[registry.k8s.io/kops/kube-apiserver-healthcheck:1.26.0-beta.1],SizeBytes:8787111,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:682696,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Jan 9 23:11:55.470: INFO: Logging kubelet events for node ip-172-20-37-64.ca-central-1.compute.internal Jan 9 23:11:55.504: INFO: Logging pods the kubelet thinks is on node ip-172-20-37-64.ca-central-1.compute.internal Jan 9 23:11:55.546: INFO: ebs-csi-node-8q4pg started at 2023-01-09 22:37:17 +0000 UTC (0+3 container statuses recorded) Jan 9 23:11:55.546: INFO: Container ebs-plugin ready: true, restart count 0 Jan 9 23:11:55.546: INFO: Container liveness-probe ready: true, restart count 0 Jan 9 23:11:55.546: INFO: Container node-driver-registrar ready: true, restart count 0 Jan 9 23:11:55.546: INFO: cilium-q7lrg started at 2023-01-09 22:37:17 +0000 UTC (1+1 container statuses recorded) Jan 9 23:11:55.546: INFO: Init container clean-cilium-state ready: true, restart count 0 Jan 9 23:11:55.546: INFO: Container cilium-agent ready: true, restart count 0 Jan 9 23:11:55.546: INFO: cilium-operator-c7bfc9f44-w4nqz started at 2023-01-09 22:37:19 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:55.546: INFO: Container cilium-operator ready: true, restart count 0 Jan 9 23:11:55.546: INFO: ebs-csi-controller-5d8d7cc949-8wxx2 started at 2023-01-09 22:37:19 +0000 UTC (0+5 container statuses recorded) Jan 9 23:11:55.546: INFO: Container csi-attacher ready: true, restart count 0 Jan 9 23:11:55.546: INFO: Container csi-provisioner ready: true, restart count 0 Jan 9 23:11:55.546: INFO: Container csi-resizer ready: true, restart count 0 Jan 9 23:11:55.546: INFO: Container ebs-plugin ready: true, restart count 0 Jan 9 23:11:55.546: INFO: Container liveness-probe ready: true, restart count 0 Jan 9 23:11:55.546: INFO: dns-controller-867784b75c-sxxcl started at 2023-01-09 22:37:20 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:55.547: INFO: Container dns-controller ready: true, restart count 0 Jan 9 23:11:55.547: INFO: kube-scheduler-ip-172-20-37-64.ca-central-1.compute.internal started at 2023-01-09 22:36:15 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:55.547: INFO: Container kube-scheduler ready: true, restart count 0 Jan 9 23:11:55.547: INFO: etcd-manager-events-ip-172-20-37-64.ca-central-1.compute.internal started at 2023-01-09 22:36:15 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:55.547: INFO: Container etcd-manager ready: true, restart count 0 Jan 9 23:11:55.547: INFO: etcd-manager-main-ip-172-20-37-64.ca-central-1.compute.internal started at 2023-01-09 22:36:15 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:55.547: INFO: Container etcd-manager ready: true, restart count 0 Jan 9 23:11:55.547: INFO: kube-apiserver-ip-172-20-37-64.ca-central-1.compute.internal started at 2023-01-09 22:36:15 +0000 UTC (0+2 container statuses recorded) Jan 9 23:11:55.547: INFO: Container healthcheck ready: true, restart count 0 Jan 9 23:11:55.547: INFO: Container kube-apiserver ready: true, restart count 1 Jan 9 23:11:55.547: INFO: kube-controller-manager-ip-172-20-37-64.ca-central-1.compute.internal started at 2023-01-09 22:36:15 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:55.547: INFO: Container kube-controller-manager ready: true, restart count 2 Jan 9 23:11:55.547: INFO: kops-controller-cqrv8 started at 2023-01-09 22:37:19 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:55.547: INFO: Container kops-controller ready: true, restart count 0 Jan 9 23:11:55.666: INFO: Latency metrics for node ip-172-20-37-64.ca-central-1.compute.internal Jan 9 23:11:55.666: INFO: Logging node info for node ip-172-20-55-216.ca-central-1.compute.internal Jan 9 23:11:55.700: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-55-216.ca-central-1.compute.internal 2ceff781-505a-4706-ad8b-e38e37f72571 47493 0 2023-01-09 22:38:47 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ca-central-1 failure-domain.beta.kubernetes.io/zone:ca-central-1a kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-55-216.ca-central-1.compute.internal kubernetes.io/os:linux kubernetes.io/role:node node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:ca-central-1a topology.hostpath.csi/node:ip-172-20-55-216.ca-central-1.compute.internal topology.kubernetes.io/region:ca-central-1 topology.kubernetes.io/zone:ca-central-1a] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-0c78741e44993ae7f"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kops-controller Update v1 2023-01-09 22:38:47 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}} } {kube-controller-manager Update v1 2023-01-09 22:38:47 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.4.0/24\"":{}}}} } {kubelet Update v1 2023-01-09 22:38:47 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {kube-controller-manager Update v1 2023-01-09 22:56:45 +0000 UTC FieldsV1 {"f:status":{"f:volumesAttached":{}}} status} {kubelet Update v1 2023-01-09 22:56:51 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:capacity":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.4.0/24,DoNotUseExternalID:,ProviderID:aws:///ca-central-1a/i-0c78741e44993ae7f,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.4.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051644416 0} {<nil>} 3956684Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946786816 0} {<nil>} 3854284Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-01-09 23:11:41 +0000 UTC,LastTransitionTime:2023-01-09 22:38:47 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-01-09 23:11:41 +0000 UTC,LastTransitionTime:2023-01-09 22:38:47 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-01-09 23:11:41 +0000 UTC,LastTransitionTime:2023-01-09 22:38:47 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-01-09 23:11:41 +0000 UTC,LastTransitionTime:2023-01-09 22:39:07 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.55.216,},NodeAddress{Type:ExternalIP,Address:15.222.60.56,},NodeAddress{Type:Hostname,Address:ip-172-20-55-216.ca-central-1.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-55-216.ca-central-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-15-222-60-56.ca-central-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec249e6a3570a28ebd4276206d3ae523,SystemUUID:ec249e6a-3570-a28e-bd42-76206d3ae523,BootID:d58dcef2-849d-4807-9ba5-0438ebaa05b9,KernelVersion:5.15.0-1026-aws,OSImage:Ubuntu 22.04.1 LTS,ContainerRuntimeVersion:docker://20.10.17,KubeletVersion:v1.23.15,KubeProxyVersion:v1.23.15,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:456569528,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/jessie-dnsutils@sha256:11e6a66017ba4e4b938c1612b7a54a3befcefd354796c04e1dba76873a13518e k8s.gcr.io/e2e-test-images/jessie-dnsutils:1.5],SizeBytes:253371792,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:20f25f275d46aa728f7615a1ccc19c78b2ed89435bf943a44b339f70f45508e6 k8s.gcr.io/e2e-test-images/httpd:2.4.39-2],SizeBytes:126894770,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:7e8bdd271312fd25fc5ff5a8f04727be84044eb3d7d8d03611972a6752e2e11e k8s.gcr.io/e2e-test-images/agnhost:2.39],SizeBytes:126872991,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nautilus@sha256:99c0d6f1ad24a1aa1905d9c6534d193f268f7b23f9add2ae6bb41f31094bdd5c k8s.gcr.io/e2e-test-images/nautilus:1.5],SizeBytes:124758741,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:1b9d1b2f36cb2dbee1960e82a9344aeb11bd4c4c03abf5e1853e0559c23855e3 k8s.gcr.io/e2e-test-images/httpd:2.4.38-2],SizeBytes:123781643,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.23.15 registry.k8s.io/kube-proxy-amd64:v1.23.15],SizeBytes:112336036,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:10e231cf07c82dfab5141c1afe548dc734e5fc1f67665ec3982a325d1bd31a9a registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.0],SizeBytes:94471559,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:c8e03f60afa90a28e4bb6ec9a8d0fc36d89de4b7475cf2d613afa793ec969fe0 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.0],SizeBytes:56382329,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:4e74c0492bceddc598de1c90cc5bc14dcda94cb49fa9c5bad9d117c4834b5e08 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.1],SizeBytes:56379269,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:36c31f7e1f433c9634d24f876353e8646246d81a03c4e351202c2644daff1620 k8s.gcr.io/sig-storage/csi-resizer:v1.2.0],SizeBytes:53956308,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:ed98431376c9e944e19a465fe8ea944806714dd95416a0821096c78d66b579bd k8s.gcr.io/sig-storage/csi-snapshotter:v4.1.1],SizeBytes:53531990,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:c5be65d6679efabb969d9b019300d187437ae876f992c40911fd2892bbef3b36 k8s.gcr.io/sig-storage/csi-attacher:v3.2.0],SizeBytes:53500826,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:60ab9b3e6a030d3038c87c0d6bca2930f58d1d72823e6a4af09767dc83b696a2 k8s.gcr.io/sig-storage/csi-attacher:v3.2.1],SizeBytes:53499570,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:7a5ba58a44e0d749e0767e4e37315bcf6a61f33ce3185c1991848af4db0fb70a k8s.gcr.io/sig-storage/csi-resizer:v1.1.0],SizeBytes:49176472,},ContainerImage{Names:[k8s.gcr.io/sig-storage/hostpathplugin@sha256:232fe80174d60d520d36043103853a1d7ab4b7f3782cf43e45034f04ccda58ce k8s.gcr.io/sig-storage/hostpathplugin:v1.7.1],SizeBytes:29977921,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:a61d309da54641db41fb8f35718f744e9f730d4d0384f8c4b186ddc9f06cbd5f k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.1.0],SizeBytes:19662887,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:19582112,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:2dee3fe5fe861bb66c3a4ac51114f3447a4cd35870e0f2e2b558c7a400d89589 k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.2.0],SizeBytes:18699436,},ContainerImage{Names:[k8s.gcr.io/sig-storage/mock-driver@sha256:a7b517f9e0f42ffade802eef9cefa271372386b85f55b702b493241e58459793 k8s.gcr.io/sig-storage/mock-driver:v4.1.0],SizeBytes:17680993,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:17611980,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:529be2c9770add0cdd0c989115222ea9fc1be430c11095eb9f6dafcf98a36e2b k8s.gcr.io/sig-storage/livenessprobe:v2.4.0],SizeBytes:17175488,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:13616070e3f29de4417eee434a8ef472221c9e51b3d037b5a6b46cef08eb7443 k8s.gcr.io/e2e-test-images/nginx:1.14-2],SizeBytes:16032814,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf k8s.gcr.io/e2e-test-images/busybox:1.29-2],SizeBytes:1154361,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db k8s.gcr.io/pause:3.6 registry.k8s.io/pause:3.6],SizeBytes:682696,},},VolumesInUse:[kubernetes.io/csi/ebs.csi.aws.com^vol-08c82b2633ea0246d kubernetes.io/csi/ebs.csi.aws.com^vol-0a0867bd913d60721],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-0a0867bd913d60721,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-08c82b2633ea0246d,DevicePath:,},},Config:nil,},} Jan 9 23:11:55.700: INFO: Logging kubelet events for node ip-172-20-55-216.ca-central-1.compute.internal Jan 9 23:11:55.735: INFO: Logging pods the kubelet thinks is on node ip-172-20-55-216.ca-central-1.compute.internal Jan 9 23:11:55.847: INFO: pod-8ade382c-2fb2-4f02-8d50-57f7b6a15696 started at 2023-01-09 23:06:54 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:55.847: INFO: Container test-container ready: false, restart count 0 Jan 9 23:11:55.847: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-mcd88 started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:55.847: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:11:55.847: INFO: csi-mockplugin-0 started at 2023-01-09 23:10:49 +0000 UTC (0+3 container statuses recorded) Jan 9 23:11:55.847: INFO: Container csi-provisioner ready: false, restart count 0 Jan 9 23:11:55.847: INFO: Container driver-registrar ready: false, restart count 0 Jan 9 23:11:55.847: INFO: Container mock ready: false, restart count 0 Jan 9 23:11:55.847: INFO: startup-ff2f8ca6-ae0a-4fae-9483-a7d80b37fdc1 started at 2023-01-09 23:07:24 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:55.847: INFO: Container busybox ready: true, restart count 0 Jan 9 23:11:55.847: INFO: pod-90864581-af4c-4524-a3f6-27ae0e060f71 started at 2023-01-09 23:10:24 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:55.847: INFO: Container write-pod ready: false, restart count 0 Jan 9 23:11:55.847: INFO: hostpath-symlink-prep-provisioning-3662 started at 2023-01-09 23:10:57 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:55.847: INFO: Container init-volume-provisioning-3662 ready: false, restart count 0 Jan 9 23:11:55.847: INFO: fail-once-non-local-nx7cr started at 2023-01-09 22:56:53 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:55.847: INFO: Container c ready: false, restart count 0 Jan 9 23:11:55.847: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-6nqvb started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:55.847: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:11:55.847: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-8kb86 started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:55.847: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:11:55.847: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-vwd6k started at 2023-01-09 22:54:43 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:55.847: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:11:55.847: INFO: image-pull-test99b7b050-d558-462d-a744-2f374728e347 started at 2023-01-09 23:07:56 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:55.847: INFO: Container image-pull-test ready: false, restart count 0 Jan 9 23:11:55.847: INFO: ebs-csi-node-psq97 started at 2023-01-09 22:38:48 +0000 UTC (0+3 container statuses recorded) Jan 9 23:11:55.847: INFO: Container ebs-plugin ready: true, restart count 0 Jan 9 23:11:55.847: INFO: Container liveness-probe ready: true, restart count 0 Jan 9 23:11:55.847: INFO: Container node-driver-registrar ready: true, restart count 0 Jan 9 23:11:55.847: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-qj2bw started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:55.847: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:11:55.847: INFO: pod-secrets-44fccaea-993f-49ad-91c1-2dc3743d909d started at 2023-01-09 23:10:24 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:55.847: INFO: Container secret-volume-test ready: false, restart count 0 Jan 9 23:11:55.847: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-krwzl started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:55.847: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:11:55.847: INFO: pod1 started at 2023-01-09 23:11:07 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:55.847: INFO: Container agnhost-container ready: false, restart count 0 Jan 9 23:11:55.847: INFO: fail-once-local-gvnwq started at 2023-01-09 23:10:38 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:55.847: INFO: Container c ready: false, restart count 1 Jan 9 23:11:55.847: INFO: csi-mockplugin-attacher-0 started at 2023-01-09 23:10:49 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:55.847: INFO: Container csi-attacher ready: false, restart count 0 Jan 9 23:11:55.847: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-6ks7b started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:55.847: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:11:55.847: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-72kwd started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:55.847: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:11:55.847: INFO: pod-53d27e26-4bc7-4dfb-aa03-5bd21df067c1 started at 2023-01-09 23:11:20 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:55.847: INFO: Container write-pod ready: false, restart count 0 Jan 9 23:11:55.847: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-qglk4 started at 2023-01-09 22:54:43 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:55.847: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:11:55.847: INFO: netserver-1 started at 2023-01-09 23:08:45 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:55.847: INFO: Container webserver ready: true, restart count 0 Jan 9 23:11:55.847: INFO: hostexec-ip-172-20-55-216.ca-central-1.compute.internal-ddnj4 started at 2023-01-09 23:10:19 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:55.847: INFO: Container agnhost-container ready: true, restart count 0 Jan 9 23:11:55.847: INFO: cilium-dqpw4 started at 2023-01-09 22:38:48 +0000 UTC (1+1 container statuses recorded) Jan 9 23:11:55.847: INFO: Init container clean-cilium-state ready: true, restart count 0 Jan 9 23:11:55.847: INFO: Container cilium-agent ready: true, restart count 0 Jan 9 23:11:55.847: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-b7bxr started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:55.847: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:11:55.847: INFO: test-rolling-update-controller-782nh started at 2023-01-09 23:11:33 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:55.847: INFO: Container httpd ready: false, restart count 0 Jan 9 23:11:55.847: INFO: fail-once-non-local-mh6qc started at 2023-01-09 22:56:53 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:55.847: INFO: Container c ready: false, restart count 0 Jan 9 23:11:55.847: INFO: inline-volume-tester2-9qph2 started at 2023-01-09 23:10:30 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:55.847: INFO: Container csi-volume-tester ready: false, restart count 0 Jan 9 23:11:55.847: INFO: netserver-1 started at 2023-01-09 23:07:18 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:55.847: INFO: Container webserver ready: false, restart count 0 Jan 9 23:11:56.115: INFO: Latency metrics for node ip-172-20-55-216.ca-central-1.compute.internal Jan 9 23:11:56.115: INFO: Logging node info for node ip-172-20-57-251.ca-central-1.compute.internal Jan 9 23:11:56.147: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-57-251.ca-central-1.compute.internal eaaaf6eb-cbd5-4763-9c39-fb0e1a4ce44a 47846 0 2023-01-09 22:38:45 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ca-central-1 failure-domain.beta.kubernetes.io/zone:ca-central-1a kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-57-251.ca-central-1.compute.internal kubernetes.io/os:linux kubernetes.io/role:node node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:ca-central-1a topology.hostpath.csi/node:ip-172-20-57-251.ca-central-1.compute.internal topology.kubernetes.io/region:ca-central-1 topology.kubernetes.io/zone:ca-central-1a] map[csi.volume.kubernetes.io/nodeid:{"csi-mock-csi-mock-volumes-9402":"csi-mock-csi-mock-volumes-9402","ebs.csi.aws.com":"i-0513e7d6e5eba8eec"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kops-controller Update v1 2023-01-09 22:38:45 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}} } {kube-controller-manager Update v1 2023-01-09 22:38:45 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.2.0/24\"":{}}}} } {kubelet Update v1 2023-01-09 22:38:45 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {kubelet Update v1 2023-01-09 23:00:27 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status} {kube-controller-manager Update v1 2023-01-09 23:11:55 +0000 UTC FieldsV1 {"f:status":{"f:volumesAttached":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.2.0/24,DoNotUseExternalID:,ProviderID:aws:///ca-central-1a/i-0513e7d6e5eba8eec,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.2.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051652608 0} {<nil>} 3956692Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946795008 0} {<nil>} 3854292Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-01-09 23:11:24 +0000 UTC,LastTransitionTime:2023-01-09 22:38:45 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-01-09 23:11:24 +0000 UTC,LastTransitionTime:2023-01-09 22:38:45 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-01-09 23:11:24 +0000 UTC,LastTransitionTime:2023-01-09 22:38:45 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-01-09 23:11:24 +0000 UTC,LastTransitionTime:2023-01-09 22:39:05 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.57.251,},NodeAddress{Type:ExternalIP,Address:3.96.144.209,},NodeAddress{Type:Hostname,Address:ip-172-20-57-251.ca-central-1.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-57-251.ca-central-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-3-96-144-209.ca-central-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2f0d7ba95b37a96a1d9360f6071d84,SystemUUID:ec2f0d7b-a95b-37a9-6a1d-9360f6071d84,BootID:67865282-1911-4e38-8675-a7b78d5ae45e,KernelVersion:5.15.0-1026-aws,OSImage:Ubuntu 22.04.1 LTS,ContainerRuntimeVersion:docker://20.10.17,KubeletVersion:v1.23.15,KubeProxyVersion:v1.23.15,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:456569528,},ContainerImage{Names:[k8s.gcr.io/etcd@sha256:dd75ec974b0a2a6f6bb47001ba09207976e625db898d1b16735528c009cb171c k8s.gcr.io/etcd:3.5.6-0],SizeBytes:299475478,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/jessie-dnsutils@sha256:11e6a66017ba4e4b938c1612b7a54a3befcefd354796c04e1dba76873a13518e k8s.gcr.io/e2e-test-images/jessie-dnsutils:1.5],SizeBytes:253371792,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:20f25f275d46aa728f7615a1ccc19c78b2ed89435bf943a44b339f70f45508e6 k8s.gcr.io/e2e-test-images/httpd:2.4.39-2],SizeBytes:126894770,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:7e8bdd271312fd25fc5ff5a8f04727be84044eb3d7d8d03611972a6752e2e11e k8s.gcr.io/e2e-test-images/agnhost:2.39],SizeBytes:126872991,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nautilus@sha256:99c0d6f1ad24a1aa1905d9c6534d193f268f7b23f9add2ae6bb41f31094bdd5c k8s.gcr.io/e2e-test-images/nautilus:1.5],SizeBytes:124758741,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:1b9d1b2f36cb2dbee1960e82a9344aeb11bd4c4c03abf5e1853e0559c23855e3 k8s.gcr.io/e2e-test-images/httpd:2.4.38-2],SizeBytes:123781643,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.23.15 registry.k8s.io/kube-proxy-amd64:v1.23.15],SizeBytes:112336036,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:10e231cf07c82dfab5141c1afe548dc734e5fc1f67665ec3982a325d1bd31a9a registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.0],SizeBytes:94471559,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/sample-apiserver@sha256:f9c93b92b6ff750b41a93c4e4fe0bfe384597aeb841e2539d5444815c55b2d8f k8s.gcr.io/e2e-test-images/sample-apiserver:1.17.5],SizeBytes:56504541,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:c8e03f60afa90a28e4bb6ec9a8d0fc36d89de4b7475cf2d613afa793ec969fe0 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.0],SizeBytes:56382329,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:4e74c0492bceddc598de1c90cc5bc14dcda94cb49fa9c5bad9d117c4834b5e08 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.1],SizeBytes:56379269,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:36c31f7e1f433c9634d24f876353e8646246d81a03c4e351202c2644daff1620 k8s.gcr.io/sig-storage/csi-resizer:v1.2.0],SizeBytes:53956308,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:ed98431376c9e944e19a465fe8ea944806714dd95416a0821096c78d66b579bd k8s.gcr.io/sig-storage/csi-snapshotter:v4.1.1],SizeBytes:53531990,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:c5be65d6679efabb969d9b019300d187437ae876f992c40911fd2892bbef3b36 k8s.gcr.io/sig-storage/csi-attacher:v3.2.0],SizeBytes:53500826,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:60ab9b3e6a030d3038c87c0d6bca2930f58d1d72823e6a4af09767dc83b696a2 k8s.gcr.io/sig-storage/csi-attacher:v3.2.1],SizeBytes:53499570,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:7a5ba58a44e0d749e0767e4e37315bcf6a61f33ce3185c1991848af4db0fb70a k8s.gcr.io/sig-storage/csi-resizer:v1.1.0],SizeBytes:49176472,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/regression-issue-74839@sha256:b4f1d8d61bdad84bd50442d161d5460e4019d53e989b64220fdbc62fc87d76bf k8s.gcr.io/e2e-test-images/regression-issue-74839:1.2],SizeBytes:44576952,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nonroot@sha256:b9e2958a3dd879e3cf11142228c6d073d0fc4ea2e857c3be6f4fb0ab5fb2c937 k8s.gcr.io/e2e-test-images/nonroot:1.2],SizeBytes:42321438,},ContainerImage{Names:[k8s.gcr.io/sig-storage/hostpathplugin@sha256:232fe80174d60d520d36043103853a1d7ab4b7f3782cf43e45034f04ccda58ce k8s.gcr.io/sig-storage/hostpathplugin:v1.7.1],SizeBytes:29977921,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:a61d309da54641db41fb8f35718f744e9f730d4d0384f8c4b186ddc9f06cbd5f k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.1.0],SizeBytes:19662887,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:19582112,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:2dee3fe5fe861bb66c3a4ac51114f3447a4cd35870e0f2e2b558c7a400d89589 k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.2.0],SizeBytes:18699436,},ContainerImage{Names:[k8s.gcr.io/sig-storage/mock-driver@sha256:a7b517f9e0f42ffade802eef9cefa271372386b85f55b702b493241e58459793 k8s.gcr.io/sig-storage/mock-driver:v4.1.0],SizeBytes:17680993,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:17611980,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:529be2c9770add0cdd0c989115222ea9fc1be430c11095eb9f6dafcf98a36e2b k8s.gcr.io/sig-storage/livenessprobe:v2.4.0],SizeBytes:17175488,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:13616070e3f29de4417eee434a8ef472221c9e51b3d037b5a6b46cef08eb7443 k8s.gcr.io/e2e-test-images/nginx:1.14-2],SizeBytes:16032814,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nonewprivs@sha256:8ac1264691820febacf3aea5d152cbde6d10685731ec14966a9401c6f47a68ac k8s.gcr.io/e2e-test-images/nonewprivs:1.3],SizeBytes:7107254,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf k8s.gcr.io/e2e-test-images/busybox:1.29-2],SizeBytes:1154361,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:39e1e963e5310e9c313bad51523be012ede7b35bb9316517d19089a010356592 k8s.gcr.io/e2e-test-images/busybox:1.29-1],SizeBytes:1154361,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db k8s.gcr.io/pause:3.6 registry.k8s.io/pause:3.6],SizeBytes:682696,},},VolumesInUse:[kubernetes.io/csi/csi-mock-csi-mock-volumes-9402^4],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/csi-mock-csi-mock-volumes-9402^4,DevicePath:,},},Config:nil,},} Jan 9 23:11:56.148: INFO: Logging kubelet events for node ip-172-20-57-251.ca-central-1.compute.internal Jan 9 23:11:56.183: INFO: Logging pods the kubelet thinks is on node ip-172-20-57-251.ca-central-1.compute.internal Jan 9 23:11:56.241: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-zx8gm started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:56.241: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:11:56.241: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-qzf5t started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:56.241: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:11:56.241: INFO: netserver-2 started at 2023-01-09 23:08:45 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:56.241: INFO: Container webserver ready: false, restart count 0 Jan 9 23:11:56.241: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-zvfdw started at 2023-01-09 22:54:43 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:56.241: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:11:56.241: INFO: cilium-6p9jz started at 2023-01-09 22:38:46 +0000 UTC (1+1 container statuses recorded) Jan 9 23:11:56.241: INFO: Init container clean-cilium-state ready: true, restart count 0 Jan 9 23:11:56.241: INFO: Container cilium-agent ready: true, restart count 0 Jan 9 23:11:56.241: INFO: sample-webhook-deployment-6c69dbd86b-wkgxj started at 2023-01-09 23:11:47 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:56.241: INFO: Container sample-webhook ready: true, restart count 0 Jan 9 23:11:56.241: INFO: hostpath-symlink-prep-volume-6746 started at 2023-01-09 23:07:44 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:56.241: INFO: Container init-volume-volume-6746 ready: false, restart count 0 Jan 9 23:11:56.241: INFO: pvc-volume-tester-2vlf5 started at 2023-01-09 23:06:05 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:56.241: INFO: Container volume-tester ready: false, restart count 0 Jan 9 23:11:56.241: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-wvflq started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:56.241: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:11:56.241: INFO: hostexec-ip-172-20-57-251.ca-central-1.compute.internal-f9vsk started at 2023-01-09 23:08:31 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:56.241: INFO: Container agnhost-container ready: true, restart count 0 Jan 9 23:11:56.241: INFO: pod-subpath-test-preprovisionedpv-ssct started at 2023-01-09 23:08:48 +0000 UTC (0+2 container statuses recorded) Jan 9 23:11:56.241: INFO: Container test-container-subpath-preprovisionedpv-ssct ready: false, restart count 0 Jan 9 23:11:56.241: INFO: Container test-container-volume-preprovisionedpv-ssct ready: false, restart count 0 Jan 9 23:11:56.241: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-tvxfq started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:56.241: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:11:56.241: INFO: csi-mockplugin-0 started at 2023-01-09 23:00:20 +0000 UTC (0+3 container statuses recorded) Jan 9 23:11:56.241: INFO: Container csi-provisioner ready: true, restart count 0 Jan 9 23:11:56.241: INFO: Container driver-registrar ready: true, restart count 0 Jan 9 23:11:56.241: INFO: Container mock ready: true, restart count 0 Jan 9 23:11:56.241: INFO: fail-once-local-szrdn started at 2023-01-09 23:10:38 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:56.241: INFO: Container c ready: false, restart count 0 Jan 9 23:11:56.241: INFO: fail-once-local-wj6xl started at 2023-01-09 23:10:57 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:56.241: INFO: Container c ready: false, restart count 0 Jan 9 23:11:56.242: INFO: test-pod started at 2023-01-09 23:11:31 +0000 UTC (0+3 container statuses recorded) Jan 9 23:11:56.242: INFO: Container busybox-1 ready: false, restart count 0 Jan 9 23:11:56.242: INFO: Container busybox-2 ready: false, restart count 0 Jan 9 23:11:56.242: INFO: Container busybox-3 ready: false, restart count 0 Jan 9 23:11:56.242: INFO: ebs-csi-node-g4ngc started at 2023-01-09 22:38:46 +0000 UTC (0+3 container statuses recorded) Jan 9 23:11:56.242: INFO: Container ebs-plugin ready: true, restart count 0 Jan 9 23:11:56.242: INFO: Container liveness-probe ready: true, restart count 0 Jan 9 23:11:56.242: INFO: Container node-driver-registrar ready: true, restart count 0 Jan 9 23:11:56.242: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-wmqwf started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:56.242: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:11:56.242: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-m8cfz started at 2023-01-09 22:54:43 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:56.242: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:11:56.242: INFO: pod-11edca44-2fb9-4737-9beb-f344cd567f1c started at 2023-01-09 23:11:21 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:56.242: INFO: Container test-container ready: false, restart count 0 Jan 9 23:11:56.242: INFO: csi-mockplugin-attacher-0 started at 2023-01-09 23:00:20 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:56.242: INFO: Container csi-attacher ready: true, restart count 0 Jan 9 23:11:56.242: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-rbqs2 started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:56.242: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:11:56.242: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-f9w29 started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:56.242: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:11:56.242: INFO: netserver-2 started at 2023-01-09 23:07:18 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:56.242: INFO: Container webserver ready: true, restart count 0 Jan 9 23:11:56.421: INFO: Latency metrics for node ip-172-20-57-251.ca-central-1.compute.internal Jan 9 23:11:56.421: INFO: Logging node info for node ip-172-20-59-208.ca-central-1.compute.internal Jan 9 23:11:56.452: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-59-208.ca-central-1.compute.internal d061a4d4-9ff6-43fc-89cc-422c61f7441c 47576 0 2023-01-09 22:38:43 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ca-central-1 failure-domain.beta.kubernetes.io/zone:ca-central-1a kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-59-208.ca-central-1.compute.internal kubernetes.io/os:linux kubernetes.io/role:node node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:ca-central-1a topology.hostpath.csi/node:ip-172-20-59-208.ca-central-1.compute.internal topology.kubernetes.io/region:ca-central-1 topology.kubernetes.io/zone:ca-central-1a] map[csi.volume.kubernetes.io/nodeid:{"csi-hostpath-ephemeral-7223":"ip-172-20-59-208.ca-central-1.compute.internal","ebs.csi.aws.com":"i-0f00ea8bafbacdd11"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kops-controller Update v1 2023-01-09 22:38:43 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}} } {kube-controller-manager Update v1 2023-01-09 22:38:43 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.1.0/24\"":{}}}} } {kubelet Update v1 2023-01-09 22:38:43 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {kube-controller-manager Update v1 2023-01-09 23:06:36 +0000 UTC FieldsV1 {"f:status":{"f:volumesAttached":{}}} status} {kubelet Update v1 2023-01-09 23:06:40 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:capacity":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.1.0/24,DoNotUseExternalID:,ProviderID:aws:///ca-central-1a/i-0f00ea8bafbacdd11,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.1.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051652608 0} {<nil>} 3956692Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946795008 0} {<nil>} 3854292Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-01-09 23:11:47 +0000 UTC,LastTransitionTime:2023-01-09 22:38:43 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-01-09 23:11:47 +0000 UTC,LastTransitionTime:2023-01-09 22:38:43 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-01-09 23:11:47 +0000 UTC,LastTransitionTime:2023-01-09 22:38:43 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-01-09 23:11:47 +0000 UTC,LastTransitionTime:2023-01-09 22:39:03 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.59.208,},NodeAddress{Type:ExternalIP,Address:99.79.51.95,},NodeAddress{Type:Hostname,Address:ip-172-20-59-208.ca-central-1.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-59-208.ca-central-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-99-79-51-95.ca-central-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec28d5ecfae27d4bcedd89eca1a365f0,SystemUUID:ec28d5ec-fae2-7d4b-cedd-89eca1a365f0,BootID:560320f7-12c5-44f2-a7f6-a7a40031e204,KernelVersion:5.15.0-1026-aws,OSImage:Ubuntu 22.04.1 LTS,ContainerRuntimeVersion:docker://20.10.17,KubeletVersion:v1.23.15,KubeProxyVersion:v1.23.15,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:456569528,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:20f25f275d46aa728f7615a1ccc19c78b2ed89435bf943a44b339f70f45508e6 k8s.gcr.io/e2e-test-images/httpd:2.4.39-2],SizeBytes:126894770,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:7e8bdd271312fd25fc5ff5a8f04727be84044eb3d7d8d03611972a6752e2e11e k8s.gcr.io/e2e-test-images/agnhost:2.39],SizeBytes:126872991,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:1b9d1b2f36cb2dbee1960e82a9344aeb11bd4c4c03abf5e1853e0559c23855e3 k8s.gcr.io/e2e-test-images/httpd:2.4.38-2],SizeBytes:123781643,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.23.15 registry.k8s.io/kube-proxy-amd64:v1.23.15],SizeBytes:112336036,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:10e231cf07c82dfab5141c1afe548dc734e5fc1f67665ec3982a325d1bd31a9a registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.0],SizeBytes:94471559,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:c8e03f60afa90a28e4bb6ec9a8d0fc36d89de4b7475cf2d613afa793ec969fe0 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.0],SizeBytes:56382329,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:4e74c0492bceddc598de1c90cc5bc14dcda94cb49fa9c5bad9d117c4834b5e08 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.1],SizeBytes:56379269,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:36c31f7e1f433c9634d24f876353e8646246d81a03c4e351202c2644daff1620 k8s.gcr.io/sig-storage/csi-resizer:v1.2.0],SizeBytes:53956308,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:ed98431376c9e944e19a465fe8ea944806714dd95416a0821096c78d66b579bd k8s.gcr.io/sig-storage/csi-snapshotter:v4.1.1],SizeBytes:53531990,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:c5be65d6679efabb969d9b019300d187437ae876f992c40911fd2892bbef3b36 k8s.gcr.io/sig-storage/csi-attacher:v3.2.0],SizeBytes:53500826,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:60ab9b3e6a030d3038c87c0d6bca2930f58d1d72823e6a4af09767dc83b696a2 k8s.gcr.io/sig-storage/csi-attacher:v3.2.1],SizeBytes:53499570,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:017727efcfeb7d053af68e51436ce8e65edbc6ca573720afb4f79c8594036955 registry.k8s.io/coredns/coredns:v1.10.0],SizeBytes:50249337,},ContainerImage{Names:[registry.k8s.io/cpa/cluster-proportional-autoscaler@sha256:68d396900aeaa072c1f27289485fdac29834045a6f3ffe369bf389d830ef572d registry.k8s.io/cpa/cluster-proportional-autoscaler:1.8.6],SizeBytes:47910609,},ContainerImage{Names:[k8s.gcr.io/sig-storage/hostpathplugin@sha256:232fe80174d60d520d36043103853a1d7ab4b7f3782cf43e45034f04ccda58ce k8s.gcr.io/sig-storage/hostpathplugin:v1.7.1],SizeBytes:29977921,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:a61d309da54641db41fb8f35718f744e9f730d4d0384f8c4b186ddc9f06cbd5f k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.1.0],SizeBytes:19662887,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:19582112,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:2dee3fe5fe861bb66c3a4ac51114f3447a4cd35870e0f2e2b558c7a400d89589 k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.2.0],SizeBytes:18699436,},ContainerImage{Names:[k8s.gcr.io/sig-storage/mock-driver@sha256:a7b517f9e0f42ffade802eef9cefa271372386b85f55b702b493241e58459793 k8s.gcr.io/sig-storage/mock-driver:v4.1.0],SizeBytes:17680993,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:17611980,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:529be2c9770add0cdd0c989115222ea9fc1be430c11095eb9f6dafcf98a36e2b k8s.gcr.io/sig-storage/livenessprobe:v2.4.0],SizeBytes:17175488,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:13616070e3f29de4417eee434a8ef472221c9e51b3d037b5a6b46cef08eb7443 k8s.gcr.io/e2e-test-images/nginx:1.14-2],SizeBytes:16032814,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nonewprivs@sha256:8ac1264691820febacf3aea5d152cbde6d10685731ec14966a9401c6f47a68ac k8s.gcr.io/e2e-test-images/nonewprivs:1.3],SizeBytes:7107254,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf k8s.gcr.io/e2e-test-images/busybox:1.29-2],SizeBytes:1154361,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:39e1e963e5310e9c313bad51523be012ede7b35bb9316517d19089a010356592 k8s.gcr.io/e2e-test-images/busybox:1.29-1],SizeBytes:1154361,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db k8s.gcr.io/pause:3.6 registry.k8s.io/pause:3.6],SizeBytes:682696,},},VolumesInUse:[kubernetes.io/csi/csi-hostpath-ephemeral-7223^43b8adc9-9072-11ed-b4b6-0a454e7330ed kubernetes.io/csi/csi-hostpath-ephemeral-7223^43b8d4d5-9072-11ed-b4b6-0a454e7330ed],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/csi-hostpath-ephemeral-7223^43b8d4d5-9072-11ed-b4b6-0a454e7330ed,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/csi-hostpath-ephemeral-7223^43b8adc9-9072-11ed-b4b6-0a454e7330ed,DevicePath:,},},Config:nil,},} Jan 9 23:11:56.453: INFO: Logging kubelet events for node ip-172-20-59-208.ca-central-1.compute.internal Jan 9 23:11:56.487: INFO: Logging pods the kubelet thinks is on node ip-172-20-59-208.ca-central-1.compute.internal Jan 9 23:11:56.525: INFO: coredns-867df8f45c-tmrqb started at 2023-01-09 22:39:03 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:56.525: INFO: Container coredns ready: true, restart count 0 Jan 9 23:11:56.525: INFO: netserver-3 started at 2023-01-09 23:08:45 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:56.525: INFO: Container webserver ready: false, restart count 0 Jan 9 23:11:56.525: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-8qp6l started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:56.525: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: false, restart count 0 Jan 9 23:11:56.525: INFO: coredns-autoscaler-557ccb4c66-jnjcc started at 2023-01-09 22:39:03 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:56.525: INFO: Container autoscaler ready: true, restart count 0 Jan 9 23:11:56.525: INFO: netserver-3 started at 2023-01-09 23:07:18 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:56.525: INFO: Container webserver ready: false, restart count 0 Jan 9 23:11:56.525: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-ptfgl started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:56.525: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:11:56.525: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-t5g85 started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:56.525: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:11:56.525: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-rzgjf started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:56.525: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:11:56.525: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-x682f started at 2023-01-09 22:54:43 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:56.525: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:11:56.525: INFO: ebs-csi-node-prrrf started at 2023-01-09 22:38:44 +0000 UTC (0+3 container statuses recorded) Jan 9 23:11:56.525: INFO: Container ebs-plugin ready: true, restart count 0 Jan 9 23:11:56.525: INFO: Container liveness-probe ready: true, restart count 0 Jan 9 23:11:56.525: INFO: Container node-driver-registrar ready: true, restart count 0 Jan 9 23:11:56.525: INFO: inline-volume-tester-xnqfb started at 2023-01-09 23:06:35 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:56.525: INFO: Container csi-volume-tester ready: false, restart count 0 Jan 9 23:11:56.525: INFO: cilium-z2ltf started at 2023-01-09 22:38:44 +0000 UTC (1+1 container statuses recorded) Jan 9 23:11:56.525: INFO: Init container clean-cilium-state ready: true, restart count 0 Jan 9 23:11:56.525: INFO: Container cilium-agent ready: true, restart count 0 Jan 9 23:11:56.525: INFO: csi-hostpathplugin-0 started at 2023-01-09 23:03:05 +0000 UTC (0+7 container statuses recorded) Jan 9 23:11:56.525: INFO: Container csi-attacher ready: true, restart count 0 Jan 9 23:11:56.525: INFO: Container csi-provisioner ready: true, restart count 0 Jan 9 23:11:56.525: INFO: Container csi-resizer ready: true, restart count 0 Jan 9 23:11:56.525: INFO: Container csi-snapshotter ready: true, restart count 0 Jan 9 23:11:56.525: INFO: Container hostpath ready: true, restart count 0 Jan 9 23:11:56.525: INFO: Container liveness-probe ready: true, restart count 0 Jan 9 23:11:56.525: INFO: Container node-driver-registrar ready: true, restart count 0 Jan 9 23:11:56.676: INFO: Latency metrics for node ip-172-20-59-208.ca-central-1.compute.internal Jan 9 23:11:56.676: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready �[1mSTEP�[0m: Destroying namespace "ephemeral-9413" for this suite.
Find with mentions in log files
go run hack/e2e.go -v --test --test_args='--ginkgo.focus=Kubernetes\se2e\ssuite\s\[sig\-apps\]\sDisruptionController\sevictions\:\senough\spods\,\sreplicaSet\,\spercentage\s\=\>\sshould\sallow\san\seviction$'
/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/apps/disruption.go:286 Jan 9 23:10:16.604: Waiting for pods in namespace "disruption-3102" to be ready Unexpected error: <*errors.errorString | 0xc0002482c0>: { s: "timed out waiting for the condition", } timed out waiting for the condition occurred /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/apps/disruption.go:576from junit_07.xml
[BeforeEach] [sig-apps] DisruptionController /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:185 �[1mSTEP�[0m: Creating a kubernetes client Jan 9 22:59:56.095: INFO: >>> kubeConfig: /root/.kube/config �[1mSTEP�[0m: Building a namespace api object, basename disruption �[1mSTEP�[0m: Waiting for a default service account to be provisioned in namespace �[1mSTEP�[0m: Waiting for kube-root-ca.crt to be provisioned in namespace [BeforeEach] [sig-apps] DisruptionController /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/apps/disruption.go:69 [It] evictions: enough pods, replicaSet, percentage => should allow an eviction /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/apps/disruption.go:286 �[1mSTEP�[0m: Waiting for the pdb to be processed �[1mSTEP�[0m: locating a running pod �[1mSTEP�[0m: Waiting for all pods to be running Jan 9 23:00:16.537: INFO: running pods: 2 < 10 Jan 9 23:00:18.571: INFO: running pods: 2 < 10 Jan 9 23:00:20.571: INFO: running pods: 2 < 10 Jan 9 23:00:22.573: INFO: running pods: 2 < 10 Jan 9 23:00:24.570: INFO: running pods: 2 < 10 Jan 9 23:00:26.570: INFO: running pods: 2 < 10 Jan 9 23:00:28.571: INFO: running pods: 2 < 10 Jan 9 23:00:30.571: INFO: running pods: 2 < 10 Jan 9 23:00:32.571: INFO: running pods: 2 < 10 Jan 9 23:00:34.570: INFO: running pods: 2 < 10 Jan 9 23:00:36.570: INFO: running pods: 2 < 10 Jan 9 23:00:38.570: INFO: running pods: 2 < 10 Jan 9 23:00:40.571: INFO: running pods: 2 < 10 Jan 9 23:00:42.569: INFO: running pods: 2 < 10 Jan 9 23:00:44.571: INFO: running pods: 2 < 10 Jan 9 23:00:46.571: INFO: running pods: 2 < 10 Jan 9 23:00:48.571: INFO: running pods: 2 < 10 Jan 9 23:00:50.571: INFO: running pods: 2 < 10 Jan 9 23:00:52.569: INFO: running pods: 2 < 10 Jan 9 23:00:54.570: INFO: running pods: 2 < 10 Jan 9 23:00:56.570: INFO: running pods: 2 < 10 Jan 9 23:00:58.569: INFO: running pods: 2 < 10 Jan 9 23:01:00.569: INFO: running pods: 2 < 10 Jan 9 23:01:02.570: INFO: running pods: 2 < 10 Jan 9 23:01:04.570: INFO: running pods: 2 < 10 Jan 9 23:01:06.569: INFO: running pods: 2 < 10 Jan 9 23:01:08.570: INFO: running pods: 2 < 10 Jan 9 23:01:10.570: INFO: running pods: 2 < 10 Jan 9 23:01:12.571: INFO: running pods: 2 < 10 Jan 9 23:01:14.570: INFO: running pods: 2 < 10 Jan 9 23:01:16.571: INFO: running pods: 2 < 10 Jan 9 23:01:18.570: INFO: running pods: 2 < 10 Jan 9 23:01:20.570: INFO: running pods: 2 < 10 Jan 9 23:01:22.572: INFO: running pods: 2 < 10 Jan 9 23:01:24.570: INFO: running pods: 2 < 10 Jan 9 23:01:26.570: INFO: running pods: 2 < 10 Jan 9 23:01:28.570: INFO: running pods: 2 < 10 Jan 9 23:01:30.571: INFO: running pods: 2 < 10 Jan 9 23:01:32.571: INFO: running pods: 2 < 10 Jan 9 23:01:34.569: INFO: running pods: 2 < 10 Jan 9 23:01:36.570: INFO: running pods: 2 < 10 Jan 9 23:01:38.570: INFO: running pods: 2 < 10 Jan 9 23:01:40.570: INFO: running pods: 2 < 10 Jan 9 23:01:42.571: INFO: running pods: 2 < 10 Jan 9 23:01:44.570: INFO: running pods: 2 < 10 Jan 9 23:01:46.570: INFO: running pods: 2 < 10 Jan 9 23:01:48.571: INFO: running pods: 2 < 10 Jan 9 23:01:50.571: INFO: running pods: 2 < 10 Jan 9 23:01:52.573: INFO: running pods: 2 < 10 Jan 9 23:01:54.570: INFO: running pods: 2 < 10 Jan 9 23:01:56.570: INFO: running pods: 2 < 10 Jan 9 23:01:58.570: INFO: running pods: 2 < 10 Jan 9 23:02:00.571: INFO: running pods: 2 < 10 Jan 9 23:02:02.570: INFO: running pods: 2 < 10 Jan 9 23:02:04.569: INFO: running pods: 2 < 10 Jan 9 23:02:06.570: INFO: running pods: 2 < 10 Jan 9 23:02:08.570: INFO: running pods: 2 < 10 Jan 9 23:02:10.570: INFO: running pods: 2 < 10 Jan 9 23:02:12.571: INFO: running pods: 2 < 10 Jan 9 23:02:14.570: INFO: running pods: 3 < 10 Jan 9 23:02:16.574: INFO: running pods: 3 < 10 Jan 9 23:02:18.571: INFO: running pods: 3 < 10 Jan 9 23:02:20.571: INFO: running pods: 3 < 10 Jan 9 23:02:22.570: INFO: running pods: 3 < 10 Jan 9 23:02:24.570: INFO: running pods: 3 < 10 Jan 9 23:02:26.570: INFO: running pods: 3 < 10 Jan 9 23:02:28.577: INFO: running pods: 3 < 10 Jan 9 23:02:30.571: INFO: running pods: 3 < 10 Jan 9 23:02:32.571: INFO: running pods: 3 < 10 Jan 9 23:02:34.571: INFO: running pods: 3 < 10 Jan 9 23:02:36.570: INFO: running pods: 3 < 10 Jan 9 23:02:38.571: INFO: running pods: 3 < 10 Jan 9 23:02:40.571: INFO: running pods: 3 < 10 Jan 9 23:02:42.570: INFO: running pods: 3 < 10 Jan 9 23:02:44.570: INFO: running pods: 3 < 10 Jan 9 23:02:46.571: INFO: running pods: 3 < 10 Jan 9 23:02:48.571: INFO: running pods: 3 < 10 Jan 9 23:02:50.571: INFO: running pods: 3 < 10 Jan 9 23:02:52.570: INFO: running pods: 3 < 10 Jan 9 23:02:54.570: INFO: running pods: 3 < 10 Jan 9 23:02:56.570: INFO: running pods: 3 < 10 Jan 9 23:02:58.570: INFO: running pods: 3 < 10 Jan 9 23:03:00.571: INFO: running pods: 3 < 10 Jan 9 23:03:02.572: INFO: running pods: 3 < 10 Jan 9 23:03:04.571: INFO: running pods: 3 < 10 Jan 9 23:03:06.572: INFO: running pods: 3 < 10 Jan 9 23:03:08.571: INFO: running pods: 3 < 10 Jan 9 23:03:10.571: INFO: running pods: 3 < 10 Jan 9 23:03:12.570: INFO: running pods: 3 < 10 Jan 9 23:03:14.570: INFO: running pods: 3 < 10 Jan 9 23:03:16.570: INFO: running pods: 3 < 10 Jan 9 23:03:18.570: INFO: running pods: 3 < 10 Jan 9 23:03:20.571: INFO: running pods: 3 < 10 Jan 9 23:03:22.571: INFO: running pods: 3 < 10 Jan 9 23:03:24.571: INFO: running pods: 3 < 10 Jan 9 23:03:26.571: INFO: running pods: 3 < 10 Jan 9 23:03:28.571: INFO: running pods: 3 < 10 Jan 9 23:03:30.571: INFO: running pods: 3 < 10 Jan 9 23:03:32.571: INFO: running pods: 3 < 10 Jan 9 23:03:34.569: INFO: running pods: 3 < 10 Jan 9 23:03:36.572: INFO: running pods: 3 < 10 Jan 9 23:03:38.570: INFO: running pods: 3 < 10 Jan 9 23:03:40.571: INFO: running pods: 3 < 10 Jan 9 23:03:42.571: INFO: running pods: 3 < 10 Jan 9 23:03:44.570: INFO: running pods: 3 < 10 Jan 9 23:03:46.571: INFO: running pods: 3 < 10 Jan 9 23:03:48.570: INFO: running pods: 3 < 10 Jan 9 23:03:50.571: INFO: running pods: 3 < 10 Jan 9 23:03:52.571: INFO: running pods: 3 < 10 Jan 9 23:03:54.570: INFO: running pods: 3 < 10 Jan 9 23:03:56.570: INFO: running pods: 3 < 10 Jan 9 23:03:58.570: INFO: running pods: 3 < 10 Jan 9 23:04:00.570: INFO: running pods: 3 < 10 Jan 9 23:04:02.571: INFO: running pods: 3 < 10 Jan 9 23:04:04.570: INFO: running pods: 3 < 10 Jan 9 23:04:06.571: INFO: running pods: 3 < 10 Jan 9 23:04:08.570: INFO: running pods: 3 < 10 Jan 9 23:04:10.571: INFO: running pods: 3 < 10 Jan 9 23:04:12.570: INFO: running pods: 3 < 10 Jan 9 23:04:14.570: INFO: running pods: 3 < 10 Jan 9 23:04:16.571: INFO: running pods: 3 < 10 Jan 9 23:04:18.571: INFO: running pods: 3 < 10 Jan 9 23:04:20.571: INFO: running pods: 3 < 10 Jan 9 23:04:22.570: INFO: running pods: 3 < 10 Jan 9 23:04:24.571: INFO: running pods: 3 < 10 Jan 9 23:04:26.571: INFO: running pods: 3 < 10 Jan 9 23:04:28.571: INFO: running pods: 3 < 10 Jan 9 23:04:30.572: INFO: running pods: 3 < 10 Jan 9 23:04:32.572: INFO: running pods: 3 < 10 Jan 9 23:04:34.570: INFO: running pods: 3 < 10 Jan 9 23:04:36.570: INFO: running pods: 3 < 10 Jan 9 23:04:38.570: INFO: running pods: 3 < 10 Jan 9 23:04:40.571: INFO: running pods: 3 < 10 Jan 9 23:04:42.571: INFO: running pods: 3 < 10 Jan 9 23:04:44.571: INFO: running pods: 3 < 10 Jan 9 23:04:46.570: INFO: running pods: 3 < 10 Jan 9 23:04:48.570: INFO: running pods: 3 < 10 Jan 9 23:04:50.570: INFO: running pods: 3 < 10 Jan 9 23:04:52.570: INFO: running pods: 3 < 10 Jan 9 23:04:54.570: INFO: running pods: 3 < 10 Jan 9 23:04:56.572: INFO: running pods: 3 < 10 Jan 9 23:04:58.570: INFO: running pods: 3 < 10 Jan 9 23:05:00.572: INFO: running pods: 3 < 10 Jan 9 23:05:02.572: INFO: running pods: 3 < 10 Jan 9 23:05:04.570: INFO: running pods: 3 < 10 Jan 9 23:05:06.571: INFO: running pods: 3 < 10 Jan 9 23:05:08.570: INFO: running pods: 3 < 10 Jan 9 23:05:10.572: INFO: running pods: 3 < 10 Jan 9 23:05:12.570: INFO: running pods: 3 < 10 Jan 9 23:05:14.569: INFO: running pods: 3 < 10 Jan 9 23:05:16.572: INFO: running pods: 3 < 10 Jan 9 23:05:18.570: INFO: running pods: 3 < 10 Jan 9 23:05:20.572: INFO: running pods: 3 < 10 Jan 9 23:05:22.569: INFO: running pods: 3 < 10 Jan 9 23:05:24.571: INFO: running pods: 3 < 10 Jan 9 23:05:26.571: INFO: running pods: 3 < 10 Jan 9 23:05:28.576: INFO: running pods: 3 < 10 Jan 9 23:05:30.571: INFO: running pods: 3 < 10 Jan 9 23:05:32.572: INFO: running pods: 3 < 10 Jan 9 23:05:34.570: INFO: running pods: 3 < 10 Jan 9 23:05:36.570: INFO: running pods: 3 < 10 Jan 9 23:05:38.570: INFO: running pods: 3 < 10 Jan 9 23:05:40.571: INFO: running pods: 3 < 10 Jan 9 23:05:42.571: INFO: running pods: 3 < 10 Jan 9 23:05:44.570: INFO: running pods: 3 < 10 Jan 9 23:05:46.572: INFO: running pods: 3 < 10 Jan 9 23:05:48.570: INFO: running pods: 3 < 10 Jan 9 23:05:50.571: INFO: running pods: 3 < 10 Jan 9 23:05:52.571: INFO: running pods: 3 < 10 Jan 9 23:05:54.570: INFO: running pods: 3 < 10 Jan 9 23:05:56.578: INFO: running pods: 3 < 10 Jan 9 23:05:58.570: INFO: running pods: 3 < 10 Jan 9 23:06:00.570: INFO: running pods: 3 < 10 Jan 9 23:06:02.572: INFO: running pods: 3 < 10 Jan 9 23:06:04.570: INFO: running pods: 3 < 10 Jan 9 23:06:06.571: INFO: running pods: 3 < 10 Jan 9 23:06:08.570: INFO: running pods: 3 < 10 Jan 9 23:06:10.571: INFO: running pods: 3 < 10 Jan 9 23:06:12.572: INFO: running pods: 3 < 10 Jan 9 23:06:14.570: INFO: running pods: 3 < 10 Jan 9 23:06:16.571: INFO: running pods: 3 < 10 Jan 9 23:06:18.571: INFO: running pods: 3 < 10 Jan 9 23:06:20.572: INFO: running pods: 3 < 10 Jan 9 23:06:22.573: INFO: running pods: 3 < 10 Jan 9 23:06:24.570: INFO: running pods: 3 < 10 Jan 9 23:06:26.570: INFO: running pods: 3 < 10 Jan 9 23:06:28.570: INFO: running pods: 3 < 10 Jan 9 23:06:30.570: INFO: running pods: 3 < 10 Jan 9 23:06:32.570: INFO: running pods: 3 < 10 Jan 9 23:06:34.570: INFO: running pods: 3 < 10 Jan 9 23:06:36.571: INFO: running pods: 3 < 10 Jan 9 23:06:38.571: INFO: running pods: 3 < 10 Jan 9 23:06:40.571: INFO: running pods: 3 < 10 Jan 9 23:06:42.570: INFO: running pods: 3 < 10 Jan 9 23:06:44.571: INFO: running pods: 3 < 10 Jan 9 23:06:46.570: INFO: running pods: 3 < 10 Jan 9 23:06:48.571: INFO: running pods: 3 < 10 Jan 9 23:06:50.571: INFO: running pods: 3 < 10 Jan 9 23:06:52.571: INFO: running pods: 3 < 10 Jan 9 23:06:54.570: INFO: running pods: 3 < 10 Jan 9 23:06:56.574: INFO: running pods: 3 < 10 Jan 9 23:06:58.570: INFO: running pods: 3 < 10 Jan 9 23:07:00.572: INFO: running pods: 3 < 10 Jan 9 23:07:02.570: INFO: running pods: 3 < 10 Jan 9 23:07:04.571: INFO: running pods: 3 < 10 Jan 9 23:07:06.571: INFO: running pods: 3 < 10 Jan 9 23:07:08.570: INFO: running pods: 3 < 10 Jan 9 23:07:10.570: INFO: running pods: 3 < 10 Jan 9 23:07:12.572: INFO: running pods: 3 < 10 Jan 9 23:07:14.570: INFO: running pods: 3 < 10 Jan 9 23:07:16.571: INFO: running pods: 3 < 10 Jan 9 23:07:18.570: INFO: running pods: 3 < 10 Jan 9 23:07:20.571: INFO: running pods: 3 < 10 Jan 9 23:07:22.571: INFO: running pods: 3 < 10 Jan 9 23:07:24.571: INFO: running pods: 3 < 10 Jan 9 23:07:26.571: INFO: running pods: 3 < 10 Jan 9 23:07:28.570: INFO: running pods: 3 < 10 Jan 9 23:07:30.570: INFO: running pods: 3 < 10 Jan 9 23:07:32.571: INFO: running pods: 3 < 10 Jan 9 23:07:34.570: INFO: running pods: 3 < 10 Jan 9 23:07:36.570: INFO: running pods: 3 < 10 Jan 9 23:07:38.570: INFO: running pods: 3 < 10 Jan 9 23:07:40.571: INFO: running pods: 3 < 10 Jan 9 23:07:42.571: INFO: running pods: 3 < 10 Jan 9 23:07:44.570: INFO: running pods: 3 < 10 Jan 9 23:07:46.570: INFO: running pods: 3 < 10 Jan 9 23:07:48.572: INFO: running pods: 3 < 10 Jan 9 23:07:50.571: INFO: running pods: 3 < 10 Jan 9 23:07:52.571: INFO: running pods: 3 < 10 Jan 9 23:07:54.569: INFO: running pods: 3 < 10 Jan 9 23:07:56.571: INFO: running pods: 3 < 10 Jan 9 23:07:58.571: INFO: running pods: 3 < 10 Jan 9 23:08:00.571: INFO: running pods: 3 < 10 Jan 9 23:08:02.571: INFO: running pods: 3 < 10 Jan 9 23:08:04.572: INFO: running pods: 3 < 10 Jan 9 23:08:06.570: INFO: running pods: 3 < 10 Jan 9 23:08:08.570: INFO: running pods: 3 < 10 Jan 9 23:08:10.570: INFO: running pods: 3 < 10 Jan 9 23:08:12.571: INFO: running pods: 3 < 10 Jan 9 23:08:14.570: INFO: running pods: 3 < 10 Jan 9 23:08:16.571: INFO: running pods: 3 < 10 Jan 9 23:08:18.571: INFO: running pods: 3 < 10 Jan 9 23:08:20.573: INFO: running pods: 3 < 10 Jan 9 23:08:22.571: INFO: running pods: 3 < 10 Jan 9 23:08:24.570: INFO: running pods: 3 < 10 Jan 9 23:08:26.571: INFO: running pods: 3 < 10 Jan 9 23:08:28.570: INFO: running pods: 3 < 10 Jan 9 23:08:30.571: INFO: running pods: 3 < 10 Jan 9 23:08:32.573: INFO: running pods: 3 < 10 Jan 9 23:08:34.570: INFO: running pods: 3 < 10 Jan 9 23:08:36.570: INFO: running pods: 3 < 10 Jan 9 23:08:38.570: INFO: running pods: 3 < 10 Jan 9 23:08:40.570: INFO: running pods: 3 < 10 Jan 9 23:08:42.569: INFO: running pods: 3 < 10 Jan 9 23:08:44.570: INFO: running pods: 3 < 10 Jan 9 23:08:46.570: INFO: running pods: 3 < 10 Jan 9 23:08:48.570: INFO: running pods: 3 < 10 Jan 9 23:08:50.571: INFO: running pods: 3 < 10 Jan 9 23:08:52.571: INFO: running pods: 3 < 10 Jan 9 23:08:54.570: INFO: running pods: 3 < 10 Jan 9 23:08:56.571: INFO: running pods: 3 < 10 Jan 9 23:08:58.570: INFO: running pods: 3 < 10 Jan 9 23:09:00.571: INFO: running pods: 3 < 10 Jan 9 23:09:02.570: INFO: running pods: 3 < 10 Jan 9 23:09:04.571: INFO: running pods: 3 < 10 Jan 9 23:09:06.572: INFO: running pods: 3 < 10 Jan 9 23:09:08.570: INFO: running pods: 3 < 10 Jan 9 23:09:10.571: INFO: running pods: 3 < 10 Jan 9 23:09:12.571: INFO: running pods: 3 < 10 Jan 9 23:09:14.570: INFO: running pods: 3 < 10 Jan 9 23:09:16.570: INFO: running pods: 3 < 10 Jan 9 23:09:18.571: INFO: running pods: 3 < 10 Jan 9 23:09:20.570: INFO: running pods: 3 < 10 Jan 9 23:09:22.570: INFO: running pods: 3 < 10 Jan 9 23:09:24.570: INFO: running pods: 3 < 10 Jan 9 23:09:26.571: INFO: running pods: 3 < 10 Jan 9 23:09:28.570: INFO: running pods: 3 < 10 Jan 9 23:09:30.570: INFO: running pods: 3 < 10 Jan 9 23:09:32.572: INFO: running pods: 3 < 10 Jan 9 23:09:34.570: INFO: running pods: 3 < 10 Jan 9 23:09:36.570: INFO: running pods: 3 < 10 Jan 9 23:09:38.570: INFO: running pods: 3 < 10 Jan 9 23:09:40.570: INFO: running pods: 3 < 10 Jan 9 23:09:42.571: INFO: running pods: 3 < 10 Jan 9 23:09:44.571: INFO: running pods: 3 < 10 Jan 9 23:09:46.571: INFO: running pods: 3 < 10 Jan 9 23:09:48.570: INFO: running pods: 3 < 10 Jan 9 23:09:50.572: INFO: running pods: 3 < 10 Jan 9 23:09:52.570: INFO: running pods: 3 < 10 Jan 9 23:09:54.570: INFO: running pods: 3 < 10 Jan 9 23:09:56.571: INFO: running pods: 3 < 10 Jan 9 23:09:58.570: INFO: running pods: 3 < 10 Jan 9 23:10:00.572: INFO: running pods: 3 < 10 Jan 9 23:10:02.571: INFO: running pods: 3 < 10 Jan 9 23:10:04.571: INFO: running pods: 3 < 10 Jan 9 23:10:06.571: INFO: running pods: 3 < 10 Jan 9 23:10:08.570: INFO: running pods: 3 < 10 Jan 9 23:10:10.570: INFO: running pods: 3 < 10 Jan 9 23:10:12.570: INFO: running pods: 3 < 10 Jan 9 23:10:14.571: INFO: running pods: 3 < 10 Jan 9 23:10:16.571: INFO: running pods: 3 < 10 Jan 9 23:10:16.604: INFO: running pods: 3 < 10 Jan 9 23:10:16.604: FAIL: Waiting for pods in namespace "disruption-3102" to be ready Unexpected error: <*errors.errorString | 0xc0002482c0>: { s: "timed out waiting for the condition", } timed out waiting for the condition occurred Full Stack Trace k8s.io/kubernetes/test/e2e/apps.waitForPodsOrDie({0x7b06bd0, 0xc003282c00}, {0xc00218be10, 0xf}, 0xa) /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/apps/disruption.go:576 +0x165 k8s.io/kubernetes/test/e2e/apps.glob..func5.7() /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/apps/disruption.go:322 +0x477 k8s.io/kubernetes/test/e2e.RunE2ETests(0x24c66d7) _output/dockerized/go/src/k8s.io/kubernetes/test/e2e/e2e.go:133 +0x697 k8s.io/kubernetes/test/e2e.TestE2E(0x0) _output/dockerized/go/src/k8s.io/kubernetes/test/e2e/e2e_test.go:136 +0x19 testing.tRunner(0xc000001d40, 0x735e880) /usr/local/go/src/testing/testing.go:1259 +0x102 created by testing.(*T).Run /usr/local/go/src/testing/testing.go:1306 +0x35a [AfterEach] [sig-apps] DisruptionController /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:186 �[1mSTEP�[0m: Collecting events from namespace "disruption-3102". �[1mSTEP�[0m: Found 130 events. Jan 9 23:10:16.639: INFO: At 2023-01-09 22:59:56 +0000 UTC - event for rs: {replicaset-controller } SuccessfulCreate: Created pod: rs-nd6n2 Jan 9 23:10:16.639: INFO: At 2023-01-09 22:59:56 +0000 UTC - event for rs: {replicaset-controller } SuccessfulCreate: Created pod: rs-49kqp Jan 9 23:10:16.639: INFO: At 2023-01-09 22:59:56 +0000 UTC - event for rs: {replicaset-controller } SuccessfulCreate: (combined from similar events): Created pod: rs-gxqdg Jan 9 23:10:16.639: INFO: At 2023-01-09 22:59:56 +0000 UTC - event for rs: {replicaset-controller } SuccessfulCreate: Created pod: rs-zn5kz Jan 9 23:10:16.639: INFO: At 2023-01-09 22:59:56 +0000 UTC - event for rs: {replicaset-controller } SuccessfulCreate: Created pod: rs-dzbwc Jan 9 23:10:16.639: INFO: At 2023-01-09 22:59:56 +0000 UTC - event for rs: {replicaset-controller } SuccessfulCreate: Created pod: rs-ctw9c Jan 9 23:10:16.639: INFO: At 2023-01-09 22:59:56 +0000 UTC - event for rs: {replicaset-controller } SuccessfulCreate: Created pod: rs-j4qxp Jan 9 23:10:16.639: INFO: At 2023-01-09 22:59:56 +0000 UTC - event for rs: {replicaset-controller } SuccessfulCreate: Created pod: rs-56t9b Jan 9 23:10:16.639: INFO: At 2023-01-09 22:59:56 +0000 UTC - event for rs: {replicaset-controller } SuccessfulCreate: Created pod: rs-x69wn Jan 9 23:10:16.639: INFO: At 2023-01-09 22:59:56 +0000 UTC - event for rs: {replicaset-controller } SuccessfulCreate: Created pod: rs-lf7br Jan 9 23:10:16.639: INFO: At 2023-01-09 22:59:56 +0000 UTC - event for rs-49kqp: {default-scheduler } Scheduled: Successfully assigned disruption-3102/rs-49kqp to ip-172-20-57-251.ca-central-1.compute.internal Jan 9 23:10:16.639: INFO: At 2023-01-09 22:59:56 +0000 UTC - event for rs-56t9b: {default-scheduler } Scheduled: Successfully assigned disruption-3102/rs-56t9b to ip-172-20-57-251.ca-central-1.compute.internal Jan 9 23:10:16.639: INFO: At 2023-01-09 22:59:56 +0000 UTC - event for rs-ctw9c: {default-scheduler } Scheduled: Successfully assigned disruption-3102/rs-ctw9c to ip-172-20-55-216.ca-central-1.compute.internal Jan 9 23:10:16.639: INFO: At 2023-01-09 22:59:56 +0000 UTC - event for rs-dzbwc: {default-scheduler } Scheduled: Successfully assigned disruption-3102/rs-dzbwc to ip-172-20-57-251.ca-central-1.compute.internal Jan 9 23:10:16.639: INFO: At 2023-01-09 22:59:56 +0000 UTC - event for rs-gxqdg: {default-scheduler } Scheduled: Successfully assigned disruption-3102/rs-gxqdg to ip-172-20-32-67.ca-central-1.compute.internal Jan 9 23:10:16.639: INFO: At 2023-01-09 22:59:56 +0000 UTC - event for rs-j4qxp: {default-scheduler } Scheduled: Successfully assigned disruption-3102/rs-j4qxp to ip-172-20-55-216.ca-central-1.compute.internal Jan 9 23:10:16.639: INFO: At 2023-01-09 22:59:56 +0000 UTC - event for rs-lf7br: {default-scheduler } Scheduled: Successfully assigned disruption-3102/rs-lf7br to ip-172-20-55-216.ca-central-1.compute.internal Jan 9 23:10:16.639: INFO: At 2023-01-09 22:59:56 +0000 UTC - event for rs-nd6n2: {default-scheduler } Scheduled: Successfully assigned disruption-3102/rs-nd6n2 to ip-172-20-59-208.ca-central-1.compute.internal Jan 9 23:10:16.639: INFO: At 2023-01-09 22:59:56 +0000 UTC - event for rs-x69wn: {default-scheduler } Scheduled: Successfully assigned disruption-3102/rs-x69wn to ip-172-20-59-208.ca-central-1.compute.internal Jan 9 23:10:16.639: INFO: At 2023-01-09 22:59:56 +0000 UTC - event for rs-zn5kz: {default-scheduler } Scheduled: Successfully assigned disruption-3102/rs-zn5kz to ip-172-20-32-67.ca-central-1.compute.internal Jan 9 23:10:16.639: INFO: At 2023-01-09 22:59:57 +0000 UTC - event for rs-49kqp: {kubelet ip-172-20-57-251.ca-central-1.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "5872b305df626fd9f8fb3229edefc8920ad42a15f1e85a986686a1dcc0adf5d9" network for pod "rs-49kqp": networkPlugin cni failed to set up pod "rs-49kqp_disruption-3102" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 9 23:10:16.639: INFO: At 2023-01-09 22:59:57 +0000 UTC - event for rs-dzbwc: {kubelet ip-172-20-57-251.ca-central-1.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "abdcc755f2346b2e697523b797d510614abcda13d1be3d826ca782808374080b" network for pod "rs-dzbwc": networkPlugin cni failed to set up pod "rs-dzbwc_disruption-3102" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 9 23:10:16.639: INFO: At 2023-01-09 22:59:57 +0000 UTC - event for rs-nd6n2: {kubelet ip-172-20-59-208.ca-central-1.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "d621017f6198157aeadc48dc33450dc1428e6511b12dbeff540d6eaed02ff503" network for pod "rs-nd6n2": networkPlugin cni failed to set up pod "rs-nd6n2_disruption-3102" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 9 23:10:16.639: INFO: At 2023-01-09 22:59:57 +0000 UTC - event for rs-x69wn: {kubelet ip-172-20-59-208.ca-central-1.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "397dd60e9764cf1a2e74373166a85eb67b615f8e3618e54a3f5b09920239fc47" network for pod "rs-x69wn": networkPlugin cni failed to set up pod "rs-x69wn_disruption-3102" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 9 23:10:16.639: INFO: At 2023-01-09 22:59:57 +0000 UTC - event for rs-zn5kz: {kubelet ip-172-20-32-67.ca-central-1.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "dbce7bfe85703600fac193c09bd5b50f6db9b8e7ed2ebd65ebb5fee8e3b85e77" network for pod "rs-zn5kz": networkPlugin cni failed to set up pod "rs-zn5kz_disruption-3102" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 9 23:10:16.639: INFO: At 2023-01-09 22:59:58 +0000 UTC - event for rs-56t9b: {kubelet ip-172-20-57-251.ca-central-1.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "ace42e7085f65e4dee4cd091630fc0486c838dc13336fffac4e8905aaaa2eca9" network for pod "rs-56t9b": networkPlugin cni failed to set up pod "rs-56t9b_disruption-3102" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 9 23:10:16.639: INFO: At 2023-01-09 22:59:58 +0000 UTC - event for rs-gxqdg: {kubelet ip-172-20-32-67.ca-central-1.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "a234704a73432f550f305da6d40b4447c517d3a8088b1b28b098551aafbcd285" network for pod "rs-gxqdg": networkPlugin cni failed to set up pod "rs-gxqdg_disruption-3102" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 9 23:10:16.639: INFO: At 2023-01-09 22:59:58 +0000 UTC - event for rs-nd6n2: {kubelet ip-172-20-59-208.ca-central-1.compute.internal} SandboxChanged: Pod sandbox changed, it will be killed and re-created. Jan 9 23:10:16.639: INFO: At 2023-01-09 22:59:58 +0000 UTC - event for rs-x69wn: {kubelet ip-172-20-59-208.ca-central-1.compute.internal} SandboxChanged: Pod sandbox changed, it will be killed and re-created. Jan 9 23:10:16.639: INFO: At 2023-01-09 22:59:58 +0000 UTC - event for rs-zn5kz: {kubelet ip-172-20-32-67.ca-central-1.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "46af8380bd4c4dc63962a6b70ae47804deaa476843347d5b32cfa24d414d04fa" network for pod "rs-zn5kz": networkPlugin cni failed to set up pod "rs-zn5kz_disruption-3102" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 9 23:10:16.639: INFO: At 2023-01-09 22:59:58 +0000 UTC - event for rs-zn5kz: {kubelet ip-172-20-32-67.ca-central-1.compute.internal} SandboxChanged: Pod sandbox changed, it will be killed and re-created. Jan 9 23:10:16.639: INFO: At 2023-01-09 22:59:59 +0000 UTC - event for rs-49kqp: {kubelet ip-172-20-57-251.ca-central-1.compute.internal} SandboxChanged: Pod sandbox changed, it will be killed and re-created. Jan 9 23:10:16.639: INFO: At 2023-01-09 22:59:59 +0000 UTC - event for rs-56t9b: {kubelet ip-172-20-57-251.ca-central-1.compute.internal} SandboxChanged: Pod sandbox changed, it will be killed and re-created. Jan 9 23:10:16.639: INFO: At 2023-01-09 22:59:59 +0000 UTC - event for rs-ctw9c: {kubelet ip-172-20-55-216.ca-central-1.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "44e50ede2392524802cccbd4756e84672d11823ff0275045a49e77e42d74b3b7" network for pod "rs-ctw9c": networkPlugin cni failed to set up pod "rs-ctw9c_disruption-3102" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 9 23:10:16.639: INFO: At 2023-01-09 22:59:59 +0000 UTC - event for rs-dzbwc: {kubelet ip-172-20-57-251.ca-central-1.compute.internal} SandboxChanged: Pod sandbox changed, it will be killed and re-created. Jan 9 23:10:16.639: INFO: At 2023-01-09 22:59:59 +0000 UTC - event for rs-gxqdg: {kubelet ip-172-20-32-67.ca-central-1.compute.internal} SandboxChanged: Pod sandbox changed, it will be killed and re-created. Jan 9 23:10:16.639: INFO: At 2023-01-09 22:59:59 +0000 UTC - event for rs-j4qxp: {kubelet ip-172-20-55-216.ca-central-1.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "e8055c621fe63fbf2ede9da1817a057a03f9c14267f94eacf6b4da7716907d46" network for pod "rs-j4qxp": networkPlugin cni failed to set up pod "rs-j4qxp_disruption-3102" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 9 23:10:16.639: INFO: At 2023-01-09 22:59:59 +0000 UTC - event for rs-lf7br: {kubelet ip-172-20-55-216.ca-central-1.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "057bb48d6265a00d3267a8cfb117ab522db13028090e550dd7b045f5c9774ca5" network for pod "rs-lf7br": networkPlugin cni failed to set up pod "rs-lf7br_disruption-3102" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 9 23:10:16.639: INFO: At 2023-01-09 23:00:00 +0000 UTC - event for rs-49kqp: {kubelet ip-172-20-57-251.ca-central-1.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "a328e4b993dfc8ffa9a3343b9bf5986a255565cc98dcb9aa3cf3702c4ab57dd7" network for pod "rs-49kqp": networkPlugin cni failed to set up pod "rs-49kqp_disruption-3102" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 9 23:10:16.639: INFO: At 2023-01-09 23:00:00 +0000 UTC - event for rs-56t9b: {kubelet ip-172-20-57-251.ca-central-1.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "4a14d4f23a9f2e3d42d9befb852637fa7cb8b7874d1192eacdf646c9c3a712db" network for pod "rs-56t9b": networkPlugin cni failed to set up pod "rs-56t9b_disruption-3102" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 9 23:10:16.639: INFO: At 2023-01-09 23:00:00 +0000 UTC - event for rs-dzbwc: {kubelet ip-172-20-57-251.ca-central-1.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "d63cea3a62db4d7084ac6137c809c2283d1f37bf84d5d60bade1e62dd61877cd" network for pod "rs-dzbwc": networkPlugin cni failed to set up pod "rs-dzbwc_disruption-3102" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 9 23:10:16.639: INFO: At 2023-01-09 23:00:00 +0000 UTC - event for rs-nd6n2: {kubelet ip-172-20-59-208.ca-central-1.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "913a8afaabf16a7734e92da27199081073405c175d1d130bb9056607377574c2" network for pod "rs-nd6n2": networkPlugin cni failed to set up pod "rs-nd6n2_disruption-3102" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 9 23:10:16.639: INFO: At 2023-01-09 23:00:00 +0000 UTC - event for rs-x69wn: {kubelet ip-172-20-59-208.ca-central-1.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "bbbd4c82056861729ecd0a2eddeeb8e5eb2e1741f7a40dc66cd4bb7ff6a34970" network for pod "rs-x69wn": networkPlugin cni failed to set up pod "rs-x69wn_disruption-3102" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 9 23:10:16.639: INFO: At 2023-01-09 23:00:01 +0000 UTC - event for rs-dzbwc: {kubelet ip-172-20-57-251.ca-central-1.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "a2033add93560756525d5e3b247f94dd500ebc7799441bdb8cf902cfe4e014b3" network for pod "rs-dzbwc": networkPlugin cni failed to set up pod "rs-dzbwc_disruption-3102" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 9 23:10:16.639: INFO: At 2023-01-09 23:00:01 +0000 UTC - event for rs-gxqdg: {kubelet ip-172-20-32-67.ca-central-1.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "392228cf24e74b631bfa466f2c1b7ba775bcf9804b2c8372418ea4a50038dbdd" network for pod "rs-gxqdg": networkPlugin cni failed to set up pod "rs-gxqdg_disruption-3102" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 9 23:10:16.639: INFO: At 2023-01-09 23:00:01 +0000 UTC - event for rs-j4qxp: {kubelet ip-172-20-55-216.ca-central-1.compute.internal} SandboxChanged: Pod sandbox changed, it will be killed and re-created. Jan 9 23:10:16.639: INFO: At 2023-01-09 23:00:01 +0000 UTC - event for rs-x69wn: {kubelet ip-172-20-59-208.ca-central-1.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "abb93421e8dd18e62cecc7cd015e84a7591d60169508d8ce0fc8109949efa1d0" network for pod "rs-x69wn": networkPlugin cni failed to set up pod "rs-x69wn_disruption-3102" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 9 23:10:16.639: INFO: At 2023-01-09 23:00:01 +0000 UTC - event for rs-zn5kz: {kubelet ip-172-20-32-67.ca-central-1.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "f651dd9c0fbd60fb36a294e747c65ce4814d5e557943742995df8eb4df006124" network for pod "rs-zn5kz": networkPlugin cni failed to set up pod "rs-zn5kz_disruption-3102" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 9 23:10:16.639: INFO: At 2023-01-09 23:00:02 +0000 UTC - event for rs-49kqp: {kubelet ip-172-20-57-251.ca-central-1.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "b0cb63900674edf8f2115dde95ea5ef0977d208fa5a7b9588f7301cc780cb1e4" network for pod "rs-49kqp": networkPlugin cni failed to set up pod "rs-49kqp_disruption-3102" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 9 23:10:16.639: INFO: At 2023-01-09 23:00:02 +0000 UTC - event for rs-zn5kz: {kubelet ip-172-20-32-67.ca-central-1.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "757b48fbfc7c9b71ed338692e618aeb9e9d9835a3291b12e12b81c539b472ee5" network for pod "rs-zn5kz": networkPlugin cni failed to set up pod "rs-zn5kz_disruption-3102" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 9 23:10:16.639: INFO: At 2023-01-09 23:00:04 +0000 UTC - event for rs-49kqp: {kubelet ip-172-20-57-251.ca-central-1.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "5d775e4834ef2e7a613bdb5c6741fcf1cedfd06129454e69828c3dc0bad323aa" network for pod "rs-49kqp": networkPlugin cni failed to set up pod "rs-49kqp_disruption-3102" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 9 23:10:16.639: INFO: At 2023-01-09 23:00:04 +0000 UTC - event for rs-nd6n2: {kubelet ip-172-20-59-208.ca-central-1.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "d7e92e422c85b3b0b65436c6b27c2ea963ff44e302ea181f07bf8e36bb864102" network for pod "rs-nd6n2": networkPlugin cni failed to set up pod "rs-nd6n2_disruption-3102" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 9 23:10:16.639: INFO: At 2023-01-09 23:00:05 +0000 UTC - event for rs-56t9b: {kubelet ip-172-20-57-251.ca-central-1.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "37a74b0f991fc76281bd149f36e58d436a6c95c535b7aad3a005d7e41819465e" network for pod "rs-56t9b": networkPlugin cni failed to set up pod "rs-56t9b_disruption-3102" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 9 23:10:16.639: INFO: At 2023-01-09 23:00:05 +0000 UTC - event for rs-dzbwc: {kubelet ip-172-20-57-251.ca-central-1.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "3247a8df234549ef5b35fcb757efdba04d31324c1c29b9229976edcac1012454" network for pod "rs-dzbwc": networkPlugin cni failed to set up pod "rs-dzbwc_disruption-3102" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 9 23:10:16.639: INFO: At 2023-01-09 23:00:05 +0000 UTC - event for rs-gxqdg: {kubelet ip-172-20-32-67.ca-central-1.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "5b1ec1afd7140236786d3567122b382562058badbbc6d403dc8e9bca5b0a2e7a" network for pod "rs-gxqdg": networkPlugin cni failed to set up pod "rs-gxqdg_disruption-3102" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 9 23:10:16.639: INFO: At 2023-01-09 23:00:05 +0000 UTC - event for rs-x69wn: {kubelet ip-172-20-59-208.ca-central-1.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "00927fbb8c5c9f52417c0ec4e8ca66c255ae3416ce0046e8f2ea65f09f6377c4" network for pod "rs-x69wn": networkPlugin cni failed to set up pod "rs-x69wn_disruption-3102" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 9 23:10:16.639: INFO: At 2023-01-09 23:00:05 +0000 UTC - event for rs-zn5kz: {kubelet ip-172-20-32-67.ca-central-1.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "9650fa058544be3691f2ade7a9dd538a5ddf39c4dbaa26659e7a2e277eaf780c" network for pod "rs-zn5kz": networkPlugin cni failed to set up pod "rs-zn5kz_disruption-3102" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 9 23:10:16.639: INFO: At 2023-01-09 23:00:06 +0000 UTC - event for rs-j4qxp: {kubelet ip-172-20-55-216.ca-central-1.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "d91676afd84e083fff99bb4a282efb78a441e100831ba3c64db64861c6ae561e" network for pod "rs-j4qxp": networkPlugin cni failed to set up pod "rs-j4qxp_disruption-3102" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 9 23:10:16.639: INFO: At 2023-01-09 23:00:07 +0000 UTC - event for rs-49kqp: {kubelet ip-172-20-57-251.ca-central-1.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "6482e0e9f8dd2e956c0447213072374afaf0603d260db3dd5f39915e773fd453" network for pod "rs-49kqp": networkPlugin cni failed to set up pod "rs-49kqp_disruption-3102" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 9 23:10:16.639: INFO: At 2023-01-09 23:00:07 +0000 UTC - event for rs-dzbwc: {kubelet ip-172-20-57-251.ca-central-1.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "973dd1151e70f8edce429270d16b789a1007f60ebe512d9051212756bb8ae712" network for pod "rs-dzbwc": networkPlugin cni failed to set up pod "rs-dzbwc_disruption-3102" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 9 23:10:16.639: INFO: At 2023-01-09 23:00:07 +0000 UTC - event for rs-gxqdg: {kubelet ip-172-20-32-67.ca-central-1.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "fd07eba4bc047736520690a1cd6e4edc8d4927039e53f57dce6d068601e908d6" network for pod "rs-gxqdg": networkPlugin cni failed to set up pod "rs-gxqdg_disruption-3102" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 9 23:10:16.639: INFO: At 2023-01-09 23:00:07 +0000 UTC - event for rs-nd6n2: {kubelet ip-172-20-59-208.ca-central-1.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "330884874b0e7c178de7f2285790fcaba2cd90969b7a0e782bdd78fbf1e1f408" network for pod "rs-nd6n2": networkPlugin cni failed to set up pod "rs-nd6n2_disruption-3102" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 9 23:10:16.639: INFO: At 2023-01-09 23:00:07 +0000 UTC - event for rs-x69wn: {kubelet ip-172-20-59-208.ca-central-1.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "563fd3437e3c38dad13bde69432fd071fcab7f19524637095e87a5d79cfccd9b" network for pod "rs-x69wn": networkPlugin cni failed to set up pod "rs-x69wn_disruption-3102" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 9 23:10:16.639: INFO: At 2023-01-09 23:00:07 +0000 UTC - event for rs-zn5kz: {kubelet ip-172-20-32-67.ca-central-1.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "2d4c3523abf3f63f6f13ddae43dc72e61a8be3110112987cfd5f502ed0af7f10" network for pod "rs-zn5kz": networkPlugin cni failed to set up pod "rs-zn5kz_disruption-3102" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 9 23:10:16.639: INFO: At 2023-01-09 23:00:08 +0000 UTC - event for rs-56t9b: {kubelet ip-172-20-57-251.ca-central-1.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "d6445931790fec00a916120dae2c8ef526ee10c8e229fdda501abf6c1ed8fc90" network for pod "rs-56t9b": networkPlugin cni failed to set up pod "rs-56t9b_disruption-3102" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 9 23:10:16.639: INFO: At 2023-01-09 23:00:08 +0000 UTC - event for rs-ctw9c: {kubelet ip-172-20-55-216.ca-central-1.compute.internal} SandboxChanged: Pod sandbox changed, it will be killed and re-created. Jan 9 23:10:16.639: INFO: At 2023-01-09 23:00:08 +0000 UTC - event for rs-dzbwc: {kubelet ip-172-20-57-251.ca-central-1.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "05d0d848b388e98e72d3b3d3ad4e43da81ca24a0362ed909046b93555fcd6f00" network for pod "rs-dzbwc": networkPlugin cni failed to set up pod "rs-dzbwc_disruption-3102" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 9 23:10:16.639: INFO: At 2023-01-09 23:00:08 +0000 UTC - event for rs-gxqdg: {kubelet ip-172-20-32-67.ca-central-1.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "aae3f8586314ef294a4f1401f14da0a266a2b58b48ff40cfb7e9501cd4a5902f" network for pod "rs-gxqdg": networkPlugin cni failed to set up pod "rs-gxqdg_disruption-3102" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 9 23:10:16.639: INFO: At 2023-01-09 23:00:08 +0000 UTC - event for rs-lf7br: {kubelet ip-172-20-55-216.ca-central-1.compute.internal} SandboxChanged: Pod sandbox changed, it will be killed and re-created. Jan 9 23:10:16.639: INFO: At 2023-01-09 23:00:08 +0000 UTC - event for rs-nd6n2: {kubelet ip-172-20-59-208.ca-central-1.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "ed7f1e4b91435c1462eaaf0d38104de9912c9e2d51d85c65da0ec6f94e7d139d" network for pod "rs-nd6n2": networkPlugin cni failed to set up pod "rs-nd6n2_disruption-3102" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 9 23:10:16.639: INFO: At 2023-01-09 23:00:08 +0000 UTC - event for rs-x69wn: {kubelet ip-172-20-59-208.ca-central-1.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "77fb851e02ecafeb6d386f617cf2adb3b2d33b72df875bcb94e4df874d0db17a" network for pod "rs-x69wn": networkPlugin cni failed to set up pod "rs-x69wn_disruption-3102" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 9 23:10:16.639: INFO: At 2023-01-09 23:00:08 +0000 UTC - event for rs-zn5kz: {kubelet ip-172-20-32-67.ca-central-1.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "d38aaaab2f66bbc9e43e84a6dcd608280066b32fc0b8b34bf7e5bd33cdc6276c" network for pod "rs-zn5kz": networkPlugin cni failed to set up pod "rs-zn5kz_disruption-3102" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 9 23:10:16.639: INFO: At 2023-01-09 23:00:09 +0000 UTC - event for rs-49kqp: {kubelet ip-172-20-57-251.ca-central-1.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "e787534d35f98003c3a8bdba1d95d747c81068154d1f3b5687e115d24ee65cfa" network for pod "rs-49kqp": networkPlugin cni failed to set up pod "rs-49kqp_disruption-3102" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 9 23:10:16.639: INFO: At 2023-01-09 23:00:09 +0000 UTC - event for rs-56t9b: {kubelet ip-172-20-57-251.ca-central-1.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "dbc3d9c741d015b44485cd4dbcd2c50d04d1e38580e553b40e3b2f725565c0a7" network for pod "rs-56t9b": networkPlugin cni failed to set up pod "rs-56t9b_disruption-3102" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 9 23:10:16.639: INFO: At 2023-01-09 23:00:10 +0000 UTC - event for rs-gxqdg: {kubelet ip-172-20-32-67.ca-central-1.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "4d2bbdd202d0df7956cc3ef3da4f662f8ddb40a053c59999a85633d3011cf743" network for pod "rs-gxqdg": networkPlugin cni failed to set up pod "rs-gxqdg_disruption-3102" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 9 23:10:16.639: INFO: At 2023-01-09 23:00:10 +0000 UTC - event for rs-nd6n2: {kubelet ip-172-20-59-208.ca-central-1.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "773c02c53e5f5dda3760d6fe7b3b744b0da80295b18813d6f13496d8dfb8a7d9" network for pod "rs-nd6n2": networkPlugin cni failed to set up pod "rs-nd6n2_disruption-3102" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 9 23:10:16.639: INFO: At 2023-01-09 23:00:10 +0000 UTC - event for rs-zn5kz: {kubelet ip-172-20-32-67.ca-central-1.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "3bc44c203004cbb5a1ecdae21a24cf5f3c1463bf97b4a7379b37709a0877a60f" network for pod "rs-zn5kz": networkPlugin cni failed to set up pod "rs-zn5kz_disruption-3102" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 9 23:10:16.639: INFO: At 2023-01-09 23:00:11 +0000 UTC - event for rs-49kqp: {kubelet ip-172-20-57-251.ca-central-1.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "af0ac3c3ea085ccc5eabf3872972d48418a50a5689f3c799ea3b18d1d423066c" network for pod "rs-49kqp": networkPlugin cni failed to set up pod "rs-49kqp_disruption-3102" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 9 23:10:16.639: INFO: At 2023-01-09 23:00:11 +0000 UTC - event for rs-ctw9c: {kubelet ip-172-20-55-216.ca-central-1.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "41530887119b0b8b1333d900557f7a4b00d6e47c3520e8f3b2c76ae12ee3759d" network for pod "rs-ctw9c": networkPlugin cni failed to set up pod "rs-ctw9c_disruption-3102" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 9 23:10:16.639: INFO: At 2023-01-09 23:00:11 +0000 UTC - event for rs-lf7br: {kubelet ip-172-20-55-216.ca-central-1.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "84c4310a789ded4f54ae68f2742ff23c5ba6148420a5bc150e0633562ff70d7b" network for pod "rs-lf7br": networkPlugin cni failed to set up pod "rs-lf7br_disruption-3102" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 9 23:10:16.639: INFO: At 2023-01-09 23:00:11 +0000 UTC - event for rs-nd6n2: {kubelet ip-172-20-59-208.ca-central-1.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "749ac438b7cd3e3331936aa61a11494e71c4f81935a133f8e3d4eb2317287bd9" network for pod "rs-nd6n2": networkPlugin cni failed to set up pod "rs-nd6n2_disruption-3102" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 9 23:10:16.639: INFO: At 2023-01-09 23:00:11 +0000 UTC - event for rs-x69wn: {kubelet ip-172-20-59-208.ca-central-1.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "f93754cd85feb7f899835e5736ea4717c5d64dfb436872768c55ff0548a322ce" network for pod "rs-x69wn": networkPlugin cni failed to set up pod "rs-x69wn_disruption-3102" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 9 23:10:16.639: INFO: At 2023-01-09 23:00:11 +0000 UTC - event for rs-zn5kz: {kubelet ip-172-20-32-67.ca-central-1.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "dfd65f2410a4dd18ddbb5ef561759cc1c44c5638413d5d69af580ca604a04164" network for pod "rs-zn5kz": networkPlugin cni failed to set up pod "rs-zn5kz_disruption-3102" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 9 23:10:16.639: INFO: At 2023-01-09 23:00:12 +0000 UTC - event for rs-dzbwc: {kubelet ip-172-20-57-251.ca-central-1.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "9f8340e960bed2f011c2e435ed588f212c1219e003b2761cfc1375ae44b3e793" network for pod "rs-dzbwc": networkPlugin cni failed to set up pod "rs-dzbwc_disruption-3102" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 9 23:10:16.639: INFO: At 2023-01-09 23:00:12 +0000 UTC - event for rs-nd6n2: {kubelet ip-172-20-59-208.ca-central-1.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "5ff56c0d6d5a98604f7d38cfa68f9bcb38bdf1d7c8774718e7de7f1d6487f038" network for pod "rs-nd6n2": networkPlugin cni failed to set up pod "rs-nd6n2_disruption-3102" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 9 23:10:16.639: INFO: At 2023-01-09 23:00:12 +0000 UTC - event for rs-x69wn: {kubelet ip-172-20-59-208.ca-central-1.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "ac217c8693753348f3952c4494536b3616047a1c9b12272958fb2d4c637bd57d" network for pod "rs-x69wn": networkPlugin cni failed to set up pod "rs-x69wn_disruption-3102" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 9 23:10:16.639: INFO: At 2023-01-09 23:00:13 +0000 UTC - event for rs-49kqp: {kubelet ip-172-20-57-251.ca-central-1.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "484ad9dd3f2e5f87ef69a52b2ca86b13220d85f264a0587e5aa3f2a83d7dfaeb" network for pod "rs-49kqp": networkPlugin cni failed to set up pod "rs-49kqp_disruption-3102" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 9 23:10:16.639: INFO: At 2023-01-09 23:00:13 +0000 UTC - event for rs-56t9b: {kubelet ip-172-20-57-251.ca-central-1.compute.internal} Pulled: Container image "k8s.gcr.io/pause:3.6" already present on machine Jan 9 23:10:16.639: INFO: At 2023-01-09 23:00:13 +0000 UTC - event for rs-56t9b: {kubelet ip-172-20-57-251.ca-central-1.compute.internal} Created: Created container donothing Jan 9 23:10:16.639: INFO: At 2023-01-09 23:00:13 +0000 UTC - event for rs-56t9b: {kubelet ip-172-20-57-251.ca-central-1.compute.internal} Started: Started container donothing Jan 9 23:10:16.639: INFO: At 2023-01-09 23:00:13 +0000 UTC - event for rs-zn5kz: {kubelet ip-172-20-32-67.ca-central-1.compute.internal} Created: Created container donothing Jan 9 23:10:16.639: INFO: At 2023-01-09 23:00:13 +0000 UTC - event for rs-zn5kz: {kubelet ip-172-20-32-67.ca-central-1.compute.internal} Pulled: Container image "k8s.gcr.io/pause:3.6" already present on machine Jan 9 23:10:16.639: INFO: At 2023-01-09 23:00:13 +0000 UTC - event for rs-zn5kz: {kubelet ip-172-20-32-67.ca-central-1.compute.internal} Started: Started container donothing Jan 9 23:10:16.639: INFO: At 2023-01-09 23:00:14 +0000 UTC - event for rs-49kqp: {kubelet ip-172-20-57-251.ca-central-1.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "54453d34a4f5d86aa37da068e87f5e2b19f9a98cf1cbbf41672538d27352a5ec" network for pod "rs-49kqp": networkPlugin cni failed to set up pod "rs-49kqp_disruption-3102" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 9 23:10:16.639: INFO: At 2023-01-09 23:00:14 +0000 UTC - event for rs-gxqdg: {kubelet ip-172-20-32-67.ca-central-1.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "4ff8caff4e959ea5a96bda821263a2c82a8267bacba676be4c4dcafff18cba88" network for pod "rs-gxqdg": networkPlugin cni failed to set up pod "rs-gxqdg_disruption-3102" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 9 23:10:16.639: INFO: At 2023-01-09 23:00:14 +0000 UTC - event for rs-nd6n2: {kubelet ip-172-20-59-208.ca-central-1.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "c007452950b1d777d660fe235dea5a27a9f1e10b79ebf323f300a2d69e47be35" network for pod "rs-nd6n2": networkPlugin cni failed to set up pod "rs-nd6n2_disruption-3102" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 9 23:10:16.639: INFO: At 2023-01-09 23:00:14 +0000 UTC - event for rs-x69wn: {kubelet ip-172-20-59-208.ca-central-1.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "6b28bfa8f98cc534b268a0325f5a3b4ffffcd694adbafa3dad6f4b25c4cb8190" network for pod "rs-x69wn": networkPlugin cni failed to set up pod "rs-x69wn_disruption-3102" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 9 23:10:16.639: INFO: At 2023-01-09 23:00:15 +0000 UTC - event for rs-dzbwc: {kubelet ip-172-20-57-251.ca-central-1.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "988e9b711c97485c71293588c2e4dcd059f96560a0267e1132340071fe26bb4f" network for pod "rs-dzbwc": networkPlugin cni failed to set up pod "rs-dzbwc_disruption-3102" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 9 23:10:16.639: INFO: At 2023-01-09 23:00:15 +0000 UTC - event for rs-gxqdg: {kubelet ip-172-20-32-67.ca-central-1.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "1dbdd71112c0ba206c78f270207633dfc8f2ec2621990e79e28ddcc9e2e3a088" network for pod "rs-gxqdg": networkPlugin cni failed to set up pod "rs-gxqdg_disruption-3102" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 9 23:10:16.639: INFO: At 2023-01-09 23:00:15 +0000 UTC - event for rs-j4qxp: {kubelet ip-172-20-55-216.ca-central-1.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "4d1174a0651ef5af76ee198db228621ee10b2610c75a68de78e801a8111aaf22" network for pod "rs-j4qxp": networkPlugin cni failed to set up pod "rs-j4qxp_disruption-3102" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 9 23:10:16.639: INFO: At 2023-01-09 23:00:16 +0000 UTC - event for rs-49kqp: {kubelet ip-172-20-57-251.ca-central-1.compute.internal} FailedCreatePodSandBox: (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "e347eace917c47a497d8bdc128e46a34c7e43b42a8bfc4ffc9de665ea34aa8a1" network for pod "rs-49kqp": networkPlugin cni failed to set up pod "rs-49kqp_disruption-3102" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 9 23:10:16.639: INFO: At 2023-01-09 23:00:16 +0000 UTC - event for rs-nd6n2: {kubelet ip-172-20-59-208.ca-central-1.compute.internal} FailedCreatePodSandBox: (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "b08456ec02c407ae459c530ca164ab3f7f9eee66e49aa978024439f4331c3ed1" network for pod "rs-nd6n2": networkPlugin cni failed to set up pod "rs-nd6n2_disruption-3102" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 9 23:10:16.639: INFO: At 2023-01-09 23:00:16 +0000 UTC - event for rs-x69wn: {kubelet ip-172-20-59-208.ca-central-1.compute.internal} FailedCreatePodSandBox: (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "a1c5b544908ae78f1bfff455b91f7a9c8ec8a3e1e790e5ccbb28bf7e72d9f547" network for pod "rs-x69wn": networkPlugin cni failed to set up pod "rs-x69wn_disruption-3102" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 9 23:10:16.639: INFO: At 2023-01-09 23:00:17 +0000 UTC - event for rs-gxqdg: {kubelet ip-172-20-32-67.ca-central-1.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "443f18216160647e5de75fcfe747c9fdad075061d6f68e4998bca3678d42411a" network for pod "rs-gxqdg": networkPlugin cni failed to set up pod "rs-gxqdg_disruption-3102" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 9 23:10:16.639: INFO: At 2023-01-09 23:00:17 +0000 UTC - event for rs-j4qxp: {kubelet ip-172-20-55-216.ca-central-1.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "a24c211496fe64873612458afd5b6cb619c53c7d8074a6fa98aa61f458306852" network for pod "rs-j4qxp": networkPlugin cni failed to set up pod "rs-j4qxp_disruption-3102" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 9 23:10:16.639: INFO: At 2023-01-09 23:00:17 +0000 UTC - event for rs-lf7br: {kubelet ip-172-20-55-216.ca-central-1.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "f63251d32672c8c50163951c14820c422a04a0a733aceccd947c425ea9af0a75" network for pod "rs-lf7br": networkPlugin cni failed to set up pod "rs-lf7br_disruption-3102" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 9 23:10:16.639: INFO: At 2023-01-09 23:00:18 +0000 UTC - event for rs-ctw9c: {kubelet ip-172-20-55-216.ca-central-1.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "aeab9f4968b96638a98c72855ecede6874192db659e3c89a8137c40eb0077132" network for pod "rs-ctw9c": networkPlugin cni failed to set up pod "rs-ctw9c_disruption-3102" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 9 23:10:16.639: INFO: At 2023-01-09 23:00:18 +0000 UTC - event for rs-dzbwc: {kubelet ip-172-20-57-251.ca-central-1.compute.internal} FailedCreatePodSandBox: (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "98f38c7af7f44e88532af258f4f7b8fdbf487333c9327cbe979557997e77de41" network for pod "rs-dzbwc": networkPlugin cni failed to set up pod "rs-dzbwc_disruption-3102" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 9 23:10:16.639: INFO: At 2023-01-09 23:00:18 +0000 UTC - event for rs-dzbwc: {kubelet ip-172-20-57-251.ca-central-1.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "5529810f93152831a52c7d6b02da5b6567cab6ddda839ca513bc2616dc3d5d76" network for pod "rs-dzbwc": networkPlugin cni failed to set up pod "rs-dzbwc_disruption-3102" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 9 23:10:16.639: INFO: At 2023-01-09 23:00:18 +0000 UTC - event for rs-gxqdg: {kubelet ip-172-20-32-67.ca-central-1.compute.internal} FailedCreatePodSandBox: (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "7cc21f7e6df338bd53034cb1bdc22127190d1a8dcff8a62c234d886d1175739e" network for pod "rs-gxqdg": networkPlugin cni failed to set up pod "rs-gxqdg_disruption-3102" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 9 23:10:16.639: INFO: At 2023-01-09 23:00:21 +0000 UTC - event for rs-j4qxp: {kubelet ip-172-20-55-216.ca-central-1.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "ad239a71cc1ae1fc673728dae5998b29335d6337052aa421be44705dd171e61f" network for pod "rs-j4qxp": networkPlugin cni failed to set up pod "rs-j4qxp_disruption-3102" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 9 23:10:16.639: INFO: At 2023-01-09 23:00:21 +0000 UTC - event for rs-lf7br: {kubelet ip-172-20-55-216.ca-central-1.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "3afb59fce148f73633503c616767e78b28e20a399580c0737cbd4ac291612d0a" network for pod "rs-lf7br": networkPlugin cni failed to set up pod "rs-lf7br_disruption-3102" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 9 23:10:16.639: INFO: At 2023-01-09 23:00:25 +0000 UTC - event for rs-ctw9c: {kubelet ip-172-20-55-216.ca-central-1.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "3188bc29449e7e6e4a5057714b6f8fb94ef4318783d649f8ee27341ba73b5aa9" network for pod "rs-ctw9c": networkPlugin cni failed to set up pod "rs-ctw9c_disruption-3102" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 9 23:10:16.639: INFO: At 2023-01-09 23:00:25 +0000 UTC - event for rs-j4qxp: {kubelet ip-172-20-55-216.ca-central-1.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "c9f4b233a537792b8acd8e80fb732faaa78666a7752a3bcf9be4a1860c55fccf" network for pod "rs-j4qxp": networkPlugin cni failed to set up pod "rs-j4qxp_disruption-3102" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 9 23:10:16.639: INFO: At 2023-01-09 23:00:27 +0000 UTC - event for rs-lf7br: {kubelet ip-172-20-55-216.ca-central-1.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "07898203928d49d5f90ecd7387517e901cdfd9b77dc494f9829349466ea04cb0" network for pod "rs-lf7br": networkPlugin cni failed to set up pod "rs-lf7br_disruption-3102" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 9 23:10:16.639: INFO: At 2023-01-09 23:00:29 +0000 UTC - event for rs-lf7br: {kubelet ip-172-20-55-216.ca-central-1.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "fa04f31d718ee688783bdd0296e96fffa52d3f2191f0cb7f546783a682e8a938" network for pod "rs-lf7br": networkPlugin cni failed to set up pod "rs-lf7br_disruption-3102" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 9 23:10:16.639: INFO: At 2023-01-09 23:00:30 +0000 UTC - event for rs-ctw9c: {kubelet ip-172-20-55-216.ca-central-1.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "00b1e9cbe6a2d4235606d734daeb80c2143a864a6864a0c9a84e8cb756a515c3" network for pod "rs-ctw9c": networkPlugin cni failed to set up pod "rs-ctw9c_disruption-3102" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 9 23:10:16.639: INFO: At 2023-01-09 23:00:30 +0000 UTC - event for rs-j4qxp: {kubelet ip-172-20-55-216.ca-central-1.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "947d4a3f78a7591e9c1aaf8217c695b11251ca80518bd02e25006a0075593920" network for pod "rs-j4qxp": networkPlugin cni failed to set up pod "rs-j4qxp_disruption-3102" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 9 23:10:16.639: INFO: At 2023-01-09 23:00:33 +0000 UTC - event for rs-ctw9c: {kubelet ip-172-20-55-216.ca-central-1.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "ef83257b2ea26873c747707bf0884dc5e05708c8ac7a639614b8d5ef4a90c239" network for pod "rs-ctw9c": networkPlugin cni failed to set up pod "rs-ctw9c_disruption-3102" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 9 23:10:16.639: INFO: At 2023-01-09 23:00:33 +0000 UTC - event for rs-j4qxp: {kubelet ip-172-20-55-216.ca-central-1.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "e8b69d89333f04639e9d828723743c1c21acc50a788b0c3760536259f2d6cee7" network for pod "rs-j4qxp": networkPlugin cni failed to set up pod "rs-j4qxp_disruption-3102" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 9 23:10:16.639: INFO: At 2023-01-09 23:00:33 +0000 UTC - event for rs-lf7br: {kubelet ip-172-20-55-216.ca-central-1.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "2dc96025eb992f2eacce0cd59b1ce2fe90dc88ad0f147935ce754782b8c1ce00" network for pod "rs-lf7br": networkPlugin cni failed to set up pod "rs-lf7br_disruption-3102" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 9 23:10:16.639: INFO: At 2023-01-09 23:00:35 +0000 UTC - event for rs-j4qxp: {kubelet ip-172-20-55-216.ca-central-1.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "dc329c11237a0a255d1eb96c3bc584a7656ea51d0aa9bbe0714eceea4f40b0af" network for pod "rs-j4qxp": networkPlugin cni failed to set up pod "rs-j4qxp_disruption-3102" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 9 23:10:16.639: INFO: At 2023-01-09 23:00:35 +0000 UTC - event for rs-lf7br: {kubelet ip-172-20-55-216.ca-central-1.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "0ba8b7190175c8d48eba773537705602250cb67862c756fa45773adc04f021e2" network for pod "rs-lf7br": networkPlugin cni failed to set up pod "rs-lf7br_disruption-3102" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 9 23:10:16.639: INFO: At 2023-01-09 23:00:37 +0000 UTC - event for rs-lf7br: {kubelet ip-172-20-55-216.ca-central-1.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "98091240b6d61fb1246daf89fd7c62ff6a842e8642d8fc9f02252d467a306f02" network for pod "rs-lf7br": networkPlugin cni failed to set up pod "rs-lf7br_disruption-3102" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 9 23:10:16.639: INFO: At 2023-01-09 23:00:38 +0000 UTC - event for rs-ctw9c: {kubelet ip-172-20-55-216.ca-central-1.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "6c6208c973e6c85e6f3cb8d7a0dcdab25550d3cf0559cd78946205a58f4c8b12" network for pod "rs-ctw9c": networkPlugin cni failed to set up pod "rs-ctw9c_disruption-3102" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 9 23:10:16.639: INFO: At 2023-01-09 23:00:38 +0000 UTC - event for rs-j4qxp: {kubelet ip-172-20-55-216.ca-central-1.compute.internal} FailedCreatePodSandBox: (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "448f2583c9e8e3515f1c8bc3d4e0cda75549a8ef03c466378d05af3b0ac673c1" network for pod "rs-j4qxp": networkPlugin cni failed to set up pod "rs-j4qxp_disruption-3102" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 9 23:10:16.639: INFO: At 2023-01-09 23:00:40 +0000 UTC - event for rs-ctw9c: {kubelet ip-172-20-55-216.ca-central-1.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "99a372f0a114e7f0ccbb708cb96b3ec4429c13bcb1042826701d7980bd043878" network for pod "rs-ctw9c": networkPlugin cni failed to set up pod "rs-ctw9c_disruption-3102" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 9 23:10:16.639: INFO: At 2023-01-09 23:00:40 +0000 UTC - event for rs-lf7br: {kubelet ip-172-20-55-216.ca-central-1.compute.internal} FailedCreatePodSandBox: (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "73e5c76b1001b69e23653cb6f82adbafc1a3d29443ed6df75e8a6dd63fb084b4" network for pod "rs-lf7br": networkPlugin cni failed to set up pod "rs-lf7br_disruption-3102" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 9 23:10:16.639: INFO: At 2023-01-09 23:00:43 +0000 UTC - event for rs-ctw9c: {kubelet ip-172-20-55-216.ca-central-1.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "cf7c7659304f154728acc90322c6baecb08ff605fdacae23b9c3cb67879a87f9" network for pod "rs-ctw9c": networkPlugin cni failed to set up pod "rs-ctw9c_disruption-3102" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 9 23:10:16.639: INFO: At 2023-01-09 23:00:49 +0000 UTC - event for rs-ctw9c: {kubelet ip-172-20-55-216.ca-central-1.compute.internal} FailedCreatePodSandBox: (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "e3d91eacff062f18aa3dc4c9c511bffee7b2450f914cc072126944b8e0a26115" network for pod "rs-ctw9c": networkPlugin cni failed to set up pod "rs-ctw9c_disruption-3102" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 9 23:10:16.672: INFO: POD NODE PHASE GRACE CONDITIONS Jan 9 23:10:16.672: INFO: rs-49kqp ip-172-20-57-251.ca-central-1.compute.internal Pending [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2023-01-09 22:59:56 +0000 UTC } {Ready False 0001-01-01 00:00:00 +0000 UTC 2023-01-09 22:59:56 +0000 UTC ContainersNotReady containers with unready status: [donothing]} {ContainersReady False 0001-01-01 00:00:00 +0000 UTC 2023-01-09 22:59:56 +0000 UTC ContainersNotReady containers with unready status: [donothing]} {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2023-01-09 22:59:56 +0000 UTC }] Jan 9 23:10:16.672: INFO: rs-56t9b ip-172-20-57-251.ca-central-1.compute.internal Running [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2023-01-09 22:59:56 +0000 UTC } {Ready True 0001-01-01 00:00:00 +0000 UTC 2023-01-09 23:00:15 +0000 UTC } {ContainersReady True 0001-01-01 00:00:00 +0000 UTC 2023-01-09 23:00:15 +0000 UTC } {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2023-01-09 22:59:56 +0000 UTC }] Jan 9 23:10:16.672: INFO: rs-ctw9c ip-172-20-55-216.ca-central-1.compute.internal Running [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2023-01-09 22:59:56 +0000 UTC } {Ready True 0001-01-01 00:00:00 +0000 UTC 2023-01-09 23:02:13 +0000 UTC } {ContainersReady True 0001-01-01 00:00:00 +0000 UTC 2023-01-09 23:02:13 +0000 UTC } {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2023-01-09 22:59:56 +0000 UTC }] Jan 9 23:10:16.672: INFO: rs-dzbwc ip-172-20-57-251.ca-central-1.compute.internal Pending [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2023-01-09 22:59:56 +0000 UTC } {Ready False 0001-01-01 00:00:00 +0000 UTC 2023-01-09 22:59:56 +0000 UTC ContainersNotReady containers with unready status: [donothing]} {ContainersReady False 0001-01-01 00:00:00 +0000 UTC 2023-01-09 22:59:56 +0000 UTC ContainersNotReady containers with unready status: [donothing]} {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2023-01-09 22:59:56 +0000 UTC }] Jan 9 23:10:16.672: INFO: rs-gxqdg ip-172-20-32-67.ca-central-1.compute.internal Pending [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2023-01-09 22:59:56 +0000 UTC } {Ready False 0001-01-01 00:00:00 +0000 UTC 2023-01-09 22:59:56 +0000 UTC ContainersNotReady containers with unready status: [donothing]} {ContainersReady False 0001-01-01 00:00:00 +0000 UTC 2023-01-09 22:59:56 +0000 UTC ContainersNotReady containers with unready status: [donothing]} {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2023-01-09 22:59:56 +0000 UTC }] Jan 9 23:10:16.672: INFO: rs-j4qxp ip-172-20-55-216.ca-central-1.compute.internal Pending [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2023-01-09 22:59:56 +0000 UTC } {Ready False 0001-01-01 00:00:00 +0000 UTC 2023-01-09 22:59:56 +0000 UTC ContainersNotReady containers with unready status: [donothing]} {ContainersReady False 0001-01-01 00:00:00 +0000 UTC 2023-01-09 22:59:56 +0000 UTC ContainersNotReady containers with unready status: [donothing]} {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2023-01-09 22:59:56 +0000 UTC }] Jan 9 23:10:16.672: INFO: rs-lf7br ip-172-20-55-216.ca-central-1.compute.internal Pending [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2023-01-09 22:59:56 +0000 UTC } {Ready False 0001-01-01 00:00:00 +0000 UTC 2023-01-09 22:59:56 +0000 UTC ContainersNotReady containers with unready status: [donothing]} {ContainersReady False 0001-01-01 00:00:00 +0000 UTC 2023-01-09 22:59:56 +0000 UTC ContainersNotReady containers with unready status: [donothing]} {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2023-01-09 22:59:56 +0000 UTC }] Jan 9 23:10:16.672: INFO: rs-nd6n2 ip-172-20-59-208.ca-central-1.compute.internal Pending [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2023-01-09 22:59:56 +0000 UTC } {Ready False 0001-01-01 00:00:00 +0000 UTC 2023-01-09 22:59:56 +0000 UTC ContainersNotReady containers with unready status: [donothing]} {ContainersReady False 0001-01-01 00:00:00 +0000 UTC 2023-01-09 22:59:56 +0000 UTC ContainersNotReady containers with unready status: [donothing]} {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2023-01-09 22:59:56 +0000 UTC }] Jan 9 23:10:16.672: INFO: rs-x69wn ip-172-20-59-208.ca-central-1.compute.internal Pending [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2023-01-09 22:59:56 +0000 UTC } {Ready False 0001-01-01 00:00:00 +0000 UTC 2023-01-09 22:59:56 +0000 UTC ContainersNotReady containers with unready status: [donothing]} {ContainersReady False 0001-01-01 00:00:00 +0000 UTC 2023-01-09 22:59:56 +0000 UTC ContainersNotReady containers with unready status: [donothing]} {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2023-01-09 22:59:56 +0000 UTC }] Jan 9 23:10:16.672: INFO: rs-zn5kz ip-172-20-32-67.ca-central-1.compute.internal Running [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2023-01-09 22:59:56 +0000 UTC } {Ready True 0001-01-01 00:00:00 +0000 UTC 2023-01-09 23:00:15 +0000 UTC } {ContainersReady True 0001-01-01 00:00:00 +0000 UTC 2023-01-09 23:00:15 +0000 UTC } {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2023-01-09 22:59:56 +0000 UTC }] Jan 9 23:10:16.672: INFO: Jan 9 23:10:16.705: INFO: Unable to fetch disruption-3102/rs-49kqp/donothing logs: the server rejected our request for an unknown reason (get pods rs-49kqp) Jan 9 23:10:16.834: INFO: Unable to fetch disruption-3102/rs-dzbwc/donothing logs: the server rejected our request for an unknown reason (get pods rs-dzbwc) Jan 9 23:10:16.871: INFO: Unable to fetch disruption-3102/rs-gxqdg/donothing logs: the server rejected our request for an unknown reason (get pods rs-gxqdg) Jan 9 23:10:16.904: INFO: Logging node info for node ip-172-20-32-67.ca-central-1.compute.internal Jan 9 23:10:16.935: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-32-67.ca-central-1.compute.internal 1f51981b-a01d-404c-b3a4-6649f301f65d 45435 0 2023-01-09 22:38:45 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ca-central-1 failure-domain.beta.kubernetes.io/zone:ca-central-1a kubelet_cleanup:true kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-32-67.ca-central-1.compute.internal kubernetes.io/os:linux kubernetes.io/role:node node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:ca-central-1a topology.hostpath.csi/node:ip-172-20-32-67.ca-central-1.compute.internal topology.kubernetes.io/region:ca-central-1 topology.kubernetes.io/zone:ca-central-1a] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-0272cebea4c6c3d5d"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kops-controller Update v1 2023-01-09 22:38:45 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}} } {kube-controller-manager Update v1 2023-01-09 22:38:45 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.3.0/24\"":{}}}} } {kubelet Update v1 2023-01-09 22:38:45 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {e2e.test Update v1 2023-01-09 22:54:42 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubelet_cleanup":{}}}} } {kube-controller-manager Update v1 2023-01-09 22:55:59 +0000 UTC FieldsV1 {"f:status":{"f:volumesAttached":{}}} status} {kubelet Update v1 2023-01-09 22:56:00 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:capacity":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.3.0/24,DoNotUseExternalID:,ProviderID:aws:///ca-central-1a/i-0272cebea4c6c3d5d,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.3.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051644416 0} {<nil>} 3956684Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946786816 0} {<nil>} 3854284Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-01-09 23:06:12 +0000 UTC,LastTransitionTime:2023-01-09 22:38:45 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-01-09 23:06:12 +0000 UTC,LastTransitionTime:2023-01-09 22:38:45 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-01-09 23:06:12 +0000 UTC,LastTransitionTime:2023-01-09 22:38:45 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-01-09 23:06:12 +0000 UTC,LastTransitionTime:2023-01-09 22:39:06 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.32.67,},NodeAddress{Type:ExternalIP,Address:35.182.2.102,},NodeAddress{Type:Hostname,Address:ip-172-20-32-67.ca-central-1.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-32-67.ca-central-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-35-182-2-102.ca-central-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2536b48bd6b6dade0ed68af2555085,SystemUUID:ec2536b4-8bd6-b6da-de0e-d68af2555085,BootID:bfb3b0df-aafa-481c-9ac6-f85e5e4580e3,KernelVersion:5.15.0-1026-aws,OSImage:Ubuntu 22.04.1 LTS,ContainerRuntimeVersion:docker://20.10.17,KubeletVersion:v1.23.15,KubeProxyVersion:v1.23.15,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:456569528,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/jessie-dnsutils@sha256:11e6a66017ba4e4b938c1612b7a54a3befcefd354796c04e1dba76873a13518e k8s.gcr.io/e2e-test-images/jessie-dnsutils:1.5],SizeBytes:253371792,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:7e8bdd271312fd25fc5ff5a8f04727be84044eb3d7d8d03611972a6752e2e11e k8s.gcr.io/e2e-test-images/agnhost:2.39],SizeBytes:126872991,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nautilus@sha256:99c0d6f1ad24a1aa1905d9c6534d193f268f7b23f9add2ae6bb41f31094bdd5c k8s.gcr.io/e2e-test-images/nautilus:1.5],SizeBytes:124758741,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:1b9d1b2f36cb2dbee1960e82a9344aeb11bd4c4c03abf5e1853e0559c23855e3 k8s.gcr.io/e2e-test-images/httpd:2.4.38-2],SizeBytes:123781643,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.23.15 registry.k8s.io/kube-proxy-amd64:v1.23.15],SizeBytes:112336036,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:10e231cf07c82dfab5141c1afe548dc734e5fc1f67665ec3982a325d1bd31a9a registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.0],SizeBytes:94471559,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:c8e03f60afa90a28e4bb6ec9a8d0fc36d89de4b7475cf2d613afa793ec969fe0 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.0],SizeBytes:56382329,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:4e74c0492bceddc598de1c90cc5bc14dcda94cb49fa9c5bad9d117c4834b5e08 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.1],SizeBytes:56379269,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:36c31f7e1f433c9634d24f876353e8646246d81a03c4e351202c2644daff1620 k8s.gcr.io/sig-storage/csi-resizer:v1.2.0],SizeBytes:53956308,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:ed98431376c9e944e19a465fe8ea944806714dd95416a0821096c78d66b579bd k8s.gcr.io/sig-storage/csi-snapshotter:v4.1.1],SizeBytes:53531990,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:c5be65d6679efabb969d9b019300d187437ae876f992c40911fd2892bbef3b36 k8s.gcr.io/sig-storage/csi-attacher:v3.2.0],SizeBytes:53500826,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:60ab9b3e6a030d3038c87c0d6bca2930f58d1d72823e6a4af09767dc83b696a2 k8s.gcr.io/sig-storage/csi-attacher:v3.2.1],SizeBytes:53499570,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:017727efcfeb7d053af68e51436ce8e65edbc6ca573720afb4f79c8594036955 registry.k8s.io/coredns/coredns:v1.10.0],SizeBytes:50249337,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:7a5ba58a44e0d749e0767e4e37315bcf6a61f33ce3185c1991848af4db0fb70a k8s.gcr.io/sig-storage/csi-resizer:v1.1.0],SizeBytes:49176472,},ContainerImage{Names:[k8s.gcr.io/sig-storage/hostpathplugin@sha256:232fe80174d60d520d36043103853a1d7ab4b7f3782cf43e45034f04ccda58ce k8s.gcr.io/sig-storage/hostpathplugin:v1.7.1],SizeBytes:29977921,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:a61d309da54641db41fb8f35718f744e9f730d4d0384f8c4b186ddc9f06cbd5f k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.1.0],SizeBytes:19662887,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:19582112,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:2dee3fe5fe861bb66c3a4ac51114f3447a4cd35870e0f2e2b558c7a400d89589 k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.2.0],SizeBytes:18699436,},ContainerImage{Names:[k8s.gcr.io/sig-storage/mock-driver@sha256:a7b517f9e0f42ffade802eef9cefa271372386b85f55b702b493241e58459793 k8s.gcr.io/sig-storage/mock-driver:v4.1.0],SizeBytes:17680993,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:17611980,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:529be2c9770add0cdd0c989115222ea9fc1be430c11095eb9f6dafcf98a36e2b k8s.gcr.io/sig-storage/livenessprobe:v2.4.0],SizeBytes:17175488,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:13616070e3f29de4417eee434a8ef472221c9e51b3d037b5a6b46cef08eb7443 k8s.gcr.io/e2e-test-images/nginx:1.14-2],SizeBytes:16032814,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nonewprivs@sha256:8ac1264691820febacf3aea5d152cbde6d10685731ec14966a9401c6f47a68ac k8s.gcr.io/e2e-test-images/nonewprivs:1.3],SizeBytes:7107254,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf k8s.gcr.io/e2e-test-images/busybox:1.29-2],SizeBytes:1154361,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:39e1e963e5310e9c313bad51523be012ede7b35bb9316517d19089a010356592 k8s.gcr.io/e2e-test-images/busybox:1.29-1],SizeBytes:1154361,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db k8s.gcr.io/pause:3.6 registry.k8s.io/pause:3.6],SizeBytes:682696,},},VolumesInUse:[kubernetes.io/csi/ebs.csi.aws.com^vol-01e8a4a8ff1da0e10],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-01e8a4a8ff1da0e10,DevicePath:,},},Config:nil,},} Jan 9 23:10:16.936: INFO: Logging kubelet events for node ip-172-20-32-67.ca-central-1.compute.internal Jan 9 23:10:16.969: INFO: Logging pods the kubelet thinks is on node ip-172-20-32-67.ca-central-1.compute.internal Jan 9 23:10:17.010: INFO: ebs-csi-node-lc4rb started at 2023-01-09 22:38:46 +0000 UTC (0+3 container statuses recorded) Jan 9 23:10:17.010: INFO: Container ebs-plugin ready: true, restart count 0 Jan 9 23:10:17.010: INFO: Container liveness-probe ready: true, restart count 0 Jan 9 23:10:17.010: INFO: Container node-driver-registrar ready: true, restart count 0 Jan 9 23:10:17.010: INFO: rs-gxqdg started at 2023-01-09 22:59:56 +0000 UTC (0+1 container statuses recorded) Jan 9 23:10:17.010: INFO: Container donothing ready: false, restart count 0 Jan 9 23:10:17.010: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-5pr42 started at 2023-01-09 22:54:43 +0000 UTC (0+1 container statuses recorded) Jan 9 23:10:17.010: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:10:17.010: INFO: netserver-0 started at 2023-01-09 23:07:18 +0000 UTC (0+1 container statuses recorded) Jan 9 23:10:17.010: INFO: Container webserver ready: false, restart count 0 Jan 9 23:10:17.010: INFO: netserver-0 started at 2023-01-09 23:08:45 +0000 UTC (0+1 container statuses recorded) Jan 9 23:10:17.010: INFO: Container webserver ready: false, restart count 0 Jan 9 23:10:17.010: INFO: inline-volume-tester-5g6j6 started at 2023-01-09 22:55:56 +0000 UTC (0+1 container statuses recorded) Jan 9 23:10:17.010: INFO: Container csi-volume-tester ready: false, restart count 0 Jan 9 23:10:17.010: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-sddjv started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:10:17.010: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:10:17.010: INFO: csi-hostpathplugin-0 started at 2023-01-09 23:05:41 +0000 UTC (0+7 container statuses recorded) Jan 9 23:10:17.010: INFO: Container csi-attacher ready: false, restart count 0 Jan 9 23:10:17.010: INFO: Container csi-provisioner ready: false, restart count 0 Jan 9 23:10:17.010: INFO: Container csi-resizer ready: false, restart count 0 Jan 9 23:10:17.010: INFO: Container csi-snapshotter ready: false, restart count 0 Jan 9 23:10:17.010: INFO: Container hostpath ready: false, restart count 0 Jan 9 23:10:17.010: INFO: Container liveness-probe ready: false, restart count 0 Jan 9 23:10:17.010: INFO: Container node-driver-registrar ready: false, restart count 0 Jan 9 23:10:17.010: INFO: cilium-dvsfs started at 2023-01-09 22:38:46 +0000 UTC (1+1 container statuses recorded) Jan 9 23:10:17.010: INFO: Init container clean-cilium-state ready: true, restart count 0 Jan 9 23:10:17.010: INFO: Container cilium-agent ready: true, restart count 0 Jan 9 23:10:17.010: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-gjc2h started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:10:17.010: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:10:17.010: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-8vzqv started at 2023-01-09 22:54:43 +0000 UTC (0+1 container statuses recorded) Jan 9 23:10:17.010: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:10:17.010: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-jb6kp started at 2023-01-09 22:54:43 +0000 UTC (0+1 container statuses recorded) Jan 9 23:10:17.010: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:10:17.010: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-hwbbn started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:10:17.010: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:10:17.010: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-fr98x started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:10:17.010: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:10:17.010: INFO: rs-zn5kz started at 2023-01-09 22:59:56 +0000 UTC (0+1 container statuses recorded) Jan 9 23:10:17.010: INFO: Container donothing ready: true, restart count 0 Jan 9 23:10:17.010: INFO: coredns-867df8f45c-sckrz started at 2023-01-09 22:39:23 +0000 UTC (0+1 container statuses recorded) Jan 9 23:10:17.010: INFO: Container coredns ready: true, restart count 0 Jan 9 23:10:17.010: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-44ccv started at 2023-01-09 22:54:43 +0000 UTC (0+1 container statuses recorded) Jan 9 23:10:17.010: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:10:17.010: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-fzrxx started at 2023-01-09 22:54:43 +0000 UTC (0+1 container statuses recorded) Jan 9 23:10:17.010: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:10:17.010: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-mlcl4 started at 2023-01-09 22:54:43 +0000 UTC (0+1 container statuses recorded) Jan 9 23:10:17.010: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:10:17.271: INFO: Latency metrics for node ip-172-20-32-67.ca-central-1.compute.internal Jan 9 23:10:17.272: INFO: Logging node info for node ip-172-20-37-64.ca-central-1.compute.internal Jan 9 23:10:17.304: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-37-64.ca-central-1.compute.internal d817613a-0fe5-40c0-a5e4-7b85b9f8506b 46487 0 2023-01-09 22:36:54 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:c5.large beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ca-central-1 failure-domain.beta.kubernetes.io/zone:ca-central-1a kops.k8s.io/kops-controller-pki: kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-37-64.ca-central-1.compute.internal kubernetes.io/os:linux kubernetes.io/role:master node-role.kubernetes.io/control-plane: node-role.kubernetes.io/master: node.kubernetes.io/exclude-from-external-load-balancers: node.kubernetes.io/instance-type:c5.large topology.ebs.csi.aws.com/zone:ca-central-1a topology.kubernetes.io/region:ca-central-1 topology.kubernetes.io/zone:ca-central-1a] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-06217673764fdafeb"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kubelet Update v1 2023-01-09 22:36:54 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {protokube Update v1 2023-01-09 22:37:19 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kops.k8s.io/kops-controller-pki":{},"f:node-role.kubernetes.io/control-plane":{},"f:node.kubernetes.io/exclude-from-external-load-balancers":{}}}} } {kops-controller Update v1 2023-01-09 22:37:21 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/master":{}}}} } {kube-controller-manager Update v1 2023-01-09 22:37:35 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.0.0/24\"":{}},"f:taints":{}}} } {kubelet Update v1 2023-01-09 22:38:39 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.0.0/24,DoNotUseExternalID:,ProviderID:aws:///ca-central-1a/i-06217673764fdafeb,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/master,Value:,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[100.96.0.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3862904832 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3758047232 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-01-09 23:09:24 +0000 UTC,LastTransitionTime:2023-01-09 22:36:48 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-01-09 23:09:24 +0000 UTC,LastTransitionTime:2023-01-09 22:36:48 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-01-09 23:09:24 +0000 UTC,LastTransitionTime:2023-01-09 22:36:48 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-01-09 23:09:24 +0000 UTC,LastTransitionTime:2023-01-09 22:37:35 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.37.64,},NodeAddress{Type:ExternalIP,Address:3.96.216.165,},NodeAddress{Type:Hostname,Address:ip-172-20-37-64.ca-central-1.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-37-64.ca-central-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-3-96-216-165.ca-central-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec27a483f61319a98e5ba2ee2d59e6a3,SystemUUID:ec27a483-f613-19a9-8e5b-a2ee2d59e6a3,BootID:9b41e339-1aaa-4374-8f11-8d34e12cd62a,KernelVersion:5.15.0-1026-aws,OSImage:Ubuntu 22.04.1 LTS,ContainerRuntimeVersion:docker://20.10.17,KubeletVersion:v1.23.15,KubeProxyVersion:v1.23.15,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[registry.k8s.io/etcdadm/etcd-manager@sha256:66a453db625abb268f4b3bbefc5a34a171d81e6e8796cecca54cfd71775c77c4 registry.k8s.io/etcdadm/etcd-manager:v3.0.20221209],SizeBytes:662209237,},ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:456569528,},ContainerImage{Names:[k8s.gcr.io/kube-apiserver-amd64:v1.23.15 registry.k8s.io/kube-apiserver-amd64:v1.23.15],SizeBytes:135240444,},ContainerImage{Names:[k8s.gcr.io/kube-controller-manager-amd64:v1.23.15 registry.k8s.io/kube-controller-manager-amd64:v1.23.15],SizeBytes:125046097,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.23.15 registry.k8s.io/kube-proxy-amd64:v1.23.15],SizeBytes:112336036,},ContainerImage{Names:[quay.io/cilium/operator@sha256:a6d24a006a6b92967ac90786b49bc1ac26e5477cf028cd1186efcfc2466484db quay.io/cilium/operator:v1.12.5],SizeBytes:110016956,},ContainerImage{Names:[registry.k8s.io/kops/kops-controller:1.26.0-beta.1],SizeBytes:104212800,},ContainerImage{Names:[registry.k8s.io/kops/dns-controller:1.26.0-beta.1],SizeBytes:102566451,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:10e231cf07c82dfab5141c1afe548dc734e5fc1f67665ec3982a325d1bd31a9a registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.0],SizeBytes:94471559,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:122bfb8c1edabb3c0edd63f06523e6940d958d19b3957dc7b1d6f81e9f1f6119 registry.k8s.io/sig-storage/csi-provisioner:v3.1.0],SizeBytes:57736070,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:9ebbf9f023e7b41ccee3d52afe39a89e3ddacdbb69269d583abfc25847cfd9e4 registry.k8s.io/sig-storage/csi-resizer:v1.4.0],SizeBytes:55464778,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:8b9c313c05f54fb04f8d430896f5f5904b6cb157df261501b29adc04d2b2dc7b registry.k8s.io/sig-storage/csi-attacher:v3.4.0],SizeBytes:54848595,},ContainerImage{Names:[k8s.gcr.io/kube-scheduler-amd64:v1.23.15 registry.k8s.io/kube-scheduler-amd64:v1.23.15],SizeBytes:53521430,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:19582112,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:17611980,},ContainerImage{Names:[registry.k8s.io/kops/kube-apiserver-healthcheck:1.26.0-beta.1],SizeBytes:8787111,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:682696,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Jan 9 23:10:17.304: INFO: Logging kubelet events for node ip-172-20-37-64.ca-central-1.compute.internal Jan 9 23:10:17.338: INFO: Logging pods the kubelet thinks is on node ip-172-20-37-64.ca-central-1.compute.internal Jan 9 23:10:17.378: INFO: ebs-csi-controller-5d8d7cc949-8wxx2 started at 2023-01-09 22:37:19 +0000 UTC (0+5 container statuses recorded) Jan 9 23:10:17.378: INFO: Container csi-attacher ready: true, restart count 0 Jan 9 23:10:17.378: INFO: Container csi-provisioner ready: true, restart count 0 Jan 9 23:10:17.378: INFO: Container csi-resizer ready: true, restart count 0 Jan 9 23:10:17.378: INFO: Container ebs-plugin ready: true, restart count 0 Jan 9 23:10:17.378: INFO: Container liveness-probe ready: true, restart count 0 Jan 9 23:10:17.378: INFO: dns-controller-867784b75c-sxxcl started at 2023-01-09 22:37:20 +0000 UTC (0+1 container statuses recorded) Jan 9 23:10:17.378: INFO: Container dns-controller ready: true, restart count 0 Jan 9 23:10:17.378: INFO: kube-scheduler-ip-172-20-37-64.ca-central-1.compute.internal started at 2023-01-09 22:36:15 +0000 UTC (0+1 container statuses recorded) Jan 9 23:10:17.378: INFO: Container kube-scheduler ready: true, restart count 0 Jan 9 23:10:17.378: INFO: etcd-manager-events-ip-172-20-37-64.ca-central-1.compute.internal started at 2023-01-09 22:36:15 +0000 UTC (0+1 container statuses recorded) Jan 9 23:10:17.378: INFO: Container etcd-manager ready: true, restart count 0 Jan 9 23:10:17.378: INFO: etcd-manager-main-ip-172-20-37-64.ca-central-1.compute.internal started at 2023-01-09 22:36:15 +0000 UTC (0+1 container statuses recorded) Jan 9 23:10:17.378: INFO: Container etcd-manager ready: true, restart count 0 Jan 9 23:10:17.378: INFO: ebs-csi-node-8q4pg started at 2023-01-09 22:37:17 +0000 UTC (0+3 container statuses recorded) Jan 9 23:10:17.378: INFO: Container ebs-plugin ready: true, restart count 0 Jan 9 23:10:17.378: INFO: Container liveness-probe ready: true, restart count 0 Jan 9 23:10:17.378: INFO: Container node-driver-registrar ready: true, restart count 0 Jan 9 23:10:17.378: INFO: cilium-q7lrg started at 2023-01-09 22:37:17 +0000 UTC (1+1 container statuses recorded) Jan 9 23:10:17.378: INFO: Init container clean-cilium-state ready: true, restart count 0 Jan 9 23:10:17.378: INFO: Container cilium-agent ready: true, restart count 0 Jan 9 23:10:17.378: INFO: cilium-operator-c7bfc9f44-w4nqz started at 2023-01-09 22:37:19 +0000 UTC (0+1 container statuses recorded) Jan 9 23:10:17.378: INFO: Container cilium-operator ready: true, restart count 0 Jan 9 23:10:17.378: INFO: kube-apiserver-ip-172-20-37-64.ca-central-1.compute.internal started at 2023-01-09 22:36:15 +0000 UTC (0+2 container statuses recorded) Jan 9 23:10:17.378: INFO: Container healthcheck ready: true, restart count 0 Jan 9 23:10:17.378: INFO: Container kube-apiserver ready: true, restart count 1 Jan 9 23:10:17.378: INFO: kube-controller-manager-ip-172-20-37-64.ca-central-1.compute.internal started at 2023-01-09 22:36:15 +0000 UTC (0+1 container statuses recorded) Jan 9 23:10:17.378: INFO: Container kube-controller-manager ready: true, restart count 2 Jan 9 23:10:17.378: INFO: kops-controller-cqrv8 started at 2023-01-09 22:37:19 +0000 UTC (0+1 container statuses recorded) Jan 9 23:10:17.378: INFO: Container kops-controller ready: true, restart count 0 Jan 9 23:10:17.494: INFO: Latency metrics for node ip-172-20-37-64.ca-central-1.compute.internal Jan 9 23:10:17.494: INFO: Logging node info for node ip-172-20-55-216.ca-central-1.compute.internal Jan 9 23:10:17.525: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-55-216.ca-central-1.compute.internal 2ceff781-505a-4706-ad8b-e38e37f72571 46096 0 2023-01-09 22:38:47 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ca-central-1 failure-domain.beta.kubernetes.io/zone:ca-central-1a kubelet_cleanup:true kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-55-216.ca-central-1.compute.internal kubernetes.io/os:linux kubernetes.io/role:node node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:ca-central-1a topology.hostpath.csi/node:ip-172-20-55-216.ca-central-1.compute.internal topology.kubernetes.io/region:ca-central-1 topology.kubernetes.io/zone:ca-central-1a] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-0c78741e44993ae7f"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kops-controller Update v1 2023-01-09 22:38:47 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}} } {kube-controller-manager Update v1 2023-01-09 22:38:47 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.4.0/24\"":{}}}} } {kubelet Update v1 2023-01-09 22:38:47 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {e2e.test Update v1 2023-01-09 22:54:42 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubelet_cleanup":{}}}} } {kube-controller-manager Update v1 2023-01-09 22:56:45 +0000 UTC FieldsV1 {"f:status":{"f:volumesAttached":{}}} status} {kubelet Update v1 2023-01-09 22:56:51 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:capacity":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.4.0/24,DoNotUseExternalID:,ProviderID:aws:///ca-central-1a/i-0c78741e44993ae7f,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.4.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051644416 0} {<nil>} 3956684Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946786816 0} {<nil>} 3854284Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-01-09 23:08:06 +0000 UTC,LastTransitionTime:2023-01-09 22:38:47 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-01-09 23:08:06 +0000 UTC,LastTransitionTime:2023-01-09 22:38:47 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-01-09 23:08:06 +0000 UTC,LastTransitionTime:2023-01-09 22:38:47 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-01-09 23:08:06 +0000 UTC,LastTransitionTime:2023-01-09 22:39:07 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.55.216,},NodeAddress{Type:ExternalIP,Address:15.222.60.56,},NodeAddress{Type:Hostname,Address:ip-172-20-55-216.ca-central-1.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-55-216.ca-central-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-15-222-60-56.ca-central-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec249e6a3570a28ebd4276206d3ae523,SystemUUID:ec249e6a-3570-a28e-bd42-76206d3ae523,BootID:d58dcef2-849d-4807-9ba5-0438ebaa05b9,KernelVersion:5.15.0-1026-aws,OSImage:Ubuntu 22.04.1 LTS,ContainerRuntimeVersion:docker://20.10.17,KubeletVersion:v1.23.15,KubeProxyVersion:v1.23.15,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:456569528,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/jessie-dnsutils@sha256:11e6a66017ba4e4b938c1612b7a54a3befcefd354796c04e1dba76873a13518e k8s.gcr.io/e2e-test-images/jessie-dnsutils:1.5],SizeBytes:253371792,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:20f25f275d46aa728f7615a1ccc19c78b2ed89435bf943a44b339f70f45508e6 k8s.gcr.io/e2e-test-images/httpd:2.4.39-2],SizeBytes:126894770,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:7e8bdd271312fd25fc5ff5a8f04727be84044eb3d7d8d03611972a6752e2e11e k8s.gcr.io/e2e-test-images/agnhost:2.39],SizeBytes:126872991,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nautilus@sha256:99c0d6f1ad24a1aa1905d9c6534d193f268f7b23f9add2ae6bb41f31094bdd5c k8s.gcr.io/e2e-test-images/nautilus:1.5],SizeBytes:124758741,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:1b9d1b2f36cb2dbee1960e82a9344aeb11bd4c4c03abf5e1853e0559c23855e3 k8s.gcr.io/e2e-test-images/httpd:2.4.38-2],SizeBytes:123781643,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.23.15 registry.k8s.io/kube-proxy-amd64:v1.23.15],SizeBytes:112336036,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:10e231cf07c82dfab5141c1afe548dc734e5fc1f67665ec3982a325d1bd31a9a registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.0],SizeBytes:94471559,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:c8e03f60afa90a28e4bb6ec9a8d0fc36d89de4b7475cf2d613afa793ec969fe0 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.0],SizeBytes:56382329,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:4e74c0492bceddc598de1c90cc5bc14dcda94cb49fa9c5bad9d117c4834b5e08 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.1],SizeBytes:56379269,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:36c31f7e1f433c9634d24f876353e8646246d81a03c4e351202c2644daff1620 k8s.gcr.io/sig-storage/csi-resizer:v1.2.0],SizeBytes:53956308,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:ed98431376c9e944e19a465fe8ea944806714dd95416a0821096c78d66b579bd k8s.gcr.io/sig-storage/csi-snapshotter:v4.1.1],SizeBytes:53531990,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:c5be65d6679efabb969d9b019300d187437ae876f992c40911fd2892bbef3b36 k8s.gcr.io/sig-storage/csi-attacher:v3.2.0],SizeBytes:53500826,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:60ab9b3e6a030d3038c87c0d6bca2930f58d1d72823e6a4af09767dc83b696a2 k8s.gcr.io/sig-storage/csi-attacher:v3.2.1],SizeBytes:53499570,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:7a5ba58a44e0d749e0767e4e37315bcf6a61f33ce3185c1991848af4db0fb70a k8s.gcr.io/sig-storage/csi-resizer:v1.1.0],SizeBytes:49176472,},ContainerImage{Names:[k8s.gcr.io/sig-storage/hostpathplugin@sha256:232fe80174d60d520d36043103853a1d7ab4b7f3782cf43e45034f04ccda58ce k8s.gcr.io/sig-storage/hostpathplugin:v1.7.1],SizeBytes:29977921,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:a61d309da54641db41fb8f35718f744e9f730d4d0384f8c4b186ddc9f06cbd5f k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.1.0],SizeBytes:19662887,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:19582112,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:2dee3fe5fe861bb66c3a4ac51114f3447a4cd35870e0f2e2b558c7a400d89589 k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.2.0],SizeBytes:18699436,},ContainerImage{Names:[k8s.gcr.io/sig-storage/mock-driver@sha256:a7b517f9e0f42ffade802eef9cefa271372386b85f55b702b493241e58459793 k8s.gcr.io/sig-storage/mock-driver:v4.1.0],SizeBytes:17680993,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:17611980,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:529be2c9770add0cdd0c989115222ea9fc1be430c11095eb9f6dafcf98a36e2b k8s.gcr.io/sig-storage/livenessprobe:v2.4.0],SizeBytes:17175488,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:13616070e3f29de4417eee434a8ef472221c9e51b3d037b5a6b46cef08eb7443 k8s.gcr.io/e2e-test-images/nginx:1.14-2],SizeBytes:16032814,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf k8s.gcr.io/e2e-test-images/busybox:1.29-2],SizeBytes:1154361,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db k8s.gcr.io/pause:3.6 registry.k8s.io/pause:3.6],SizeBytes:682696,},},VolumesInUse:[kubernetes.io/csi/ebs.csi.aws.com^vol-0c8e4ae9d44bf0644 kubernetes.io/csi/ebs.csi.aws.com^vol-0d0fbf37c343045c2],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-0d0fbf37c343045c2,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-0c8e4ae9d44bf0644,DevicePath:,},},Config:nil,},} Jan 9 23:10:17.526: INFO: Logging kubelet events for node ip-172-20-55-216.ca-central-1.compute.internal Jan 9 23:10:17.559: INFO: Logging pods the kubelet thinks is on node ip-172-20-55-216.ca-central-1.compute.internal Jan 9 23:10:17.602: INFO: pause-pod-78b7f9b75c-l4st2 started at 2023-01-09 23:05:44 +0000 UTC (0+1 container statuses recorded) Jan 9 23:10:17.602: INFO: Container agnhost-pause ready: false, restart count 0 Jan 9 23:10:17.602: INFO: logs-generator started at 2023-01-09 23:05:28 +0000 UTC (0+1 container statuses recorded) Jan 9 23:10:17.602: INFO: Container logs-generator ready: false, restart count 0 Jan 9 23:10:17.602: INFO: pod-8ade382c-2fb2-4f02-8d50-57f7b6a15696 started at 2023-01-09 23:06:54 +0000 UTC (0+1 container statuses recorded) Jan 9 23:10:17.602: INFO: Container test-container ready: false, restart count 0 Jan 9 23:10:17.602: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-mcd88 started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:10:17.602: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:10:17.602: INFO: echo-sourceip started at 2023-01-09 23:00:54 +0000 UTC (0+1 container statuses recorded) Jan 9 23:10:17.602: INFO: Container agnhost-container ready: true, restart count 0 Jan 9 23:10:17.602: INFO: startup-ff2f8ca6-ae0a-4fae-9483-a7d80b37fdc1 started at 2023-01-09 23:07:24 +0000 UTC (0+1 container statuses recorded) Jan 9 23:10:17.602: INFO: Container busybox ready: false, restart count 0 Jan 9 23:10:17.602: INFO: fail-once-non-local-nx7cr started at 2023-01-09 22:56:53 +0000 UTC (0+1 container statuses recorded) Jan 9 23:10:17.602: INFO: Container c ready: false, restart count 0 Jan 9 23:10:17.602: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-6nqvb started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:10:17.602: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:10:17.602: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-8kb86 started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:10:17.602: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:10:17.602: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-vwd6k started at 2023-01-09 22:54:43 +0000 UTC (0+1 container statuses recorded) Jan 9 23:10:17.602: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:10:17.602: INFO: image-pull-test99b7b050-d558-462d-a744-2f374728e347 started at 2023-01-09 23:07:56 +0000 UTC (0+1 container statuses recorded) Jan 9 23:10:17.602: INFO: Container image-pull-test ready: false, restart count 0 Jan 9 23:10:17.602: INFO: rs-ctw9c started at 2023-01-09 22:59:56 +0000 UTC (0+1 container statuses recorded) Jan 9 23:10:17.602: INFO: Container donothing ready: true, restart count 0 Jan 9 23:10:17.602: INFO: ebs-csi-node-psq97 started at 2023-01-09 22:38:48 +0000 UTC (0+3 container statuses recorded) Jan 9 23:10:17.602: INFO: Container ebs-plugin ready: true, restart count 0 Jan 9 23:10:17.602: INFO: Container liveness-probe ready: true, restart count 0 Jan 9 23:10:17.602: INFO: Container node-driver-registrar ready: true, restart count 0 Jan 9 23:10:17.602: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-qj2bw started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:10:17.602: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:10:17.602: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-krwzl started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:10:17.602: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:10:17.602: INFO: rs-j4qxp started at 2023-01-09 22:59:56 +0000 UTC (0+1 container statuses recorded) Jan 9 23:10:17.602: INFO: Container donothing ready: false, restart count 0 Jan 9 23:10:17.602: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-6ks7b started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:10:17.602: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:10:17.602: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-72kwd started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:10:17.602: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:10:17.602: INFO: pod-adoption-release started at 2023-01-09 23:05:21 +0000 UTC (0+1 container statuses recorded) Jan 9 23:10:17.602: INFO: Container pod-adoption-release ready: false, restart count 0 Jan 9 23:10:17.602: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-qglk4 started at 2023-01-09 22:54:43 +0000 UTC (0+1 container statuses recorded) Jan 9 23:10:17.602: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:10:17.602: INFO: httpd started at 2023-01-09 23:06:03 +0000 UTC (0+1 container statuses recorded) Jan 9 23:10:17.602: INFO: Container httpd ready: false, restart count 0 Jan 9 23:10:17.602: INFO: netserver-1 started at 2023-01-09 23:08:45 +0000 UTC (0+1 container statuses recorded) Jan 9 23:10:17.602: INFO: Container webserver ready: false, restart count 0 Jan 9 23:10:17.602: INFO: cilium-dqpw4 started at 2023-01-09 22:38:48 +0000 UTC (1+1 container statuses recorded) Jan 9 23:10:17.602: INFO: Init container clean-cilium-state ready: true, restart count 0 Jan 9 23:10:17.602: INFO: Container cilium-agent ready: true, restart count 0 Jan 9 23:10:17.602: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-b7bxr started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:10:17.602: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:10:17.602: INFO: pod-subpath-test-inlinevolume-x5g6 started at 2023-01-09 23:05:45 +0000 UTC (2+1 container statuses recorded) Jan 9 23:10:17.602: INFO: Init container init-volume-inlinevolume-x5g6 ready: false, restart count 0 Jan 9 23:10:17.602: INFO: Init container test-init-volume-inlinevolume-x5g6 ready: false, restart count 0 Jan 9 23:10:17.602: INFO: Container test-container-subpath-inlinevolume-x5g6 ready: false, restart count 0 Jan 9 23:10:17.602: INFO: inline-volume-tester-7mx4s started at 2023-01-09 22:56:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:10:17.602: INFO: Container csi-volume-tester ready: false, restart count 0 Jan 9 23:10:17.602: INFO: fail-once-non-local-mh6qc started at 2023-01-09 22:56:53 +0000 UTC (0+1 container statuses recorded) Jan 9 23:10:17.602: INFO: Container c ready: false, restart count 0 Jan 9 23:10:17.602: INFO: rs-lf7br started at 2023-01-09 22:59:56 +0000 UTC (0+1 container statuses recorded) Jan 9 23:10:17.602: INFO: Container donothing ready: false, restart count 0 Jan 9 23:10:17.602: INFO: inline-volume-tester-crzhr started at 2023-01-09 23:05:03 +0000 UTC (0+1 container statuses recorded) Jan 9 23:10:17.602: INFO: Container csi-volume-tester ready: false, restart count 0 Jan 9 23:10:17.602: INFO: netserver-1 started at 2023-01-09 23:07:18 +0000 UTC (0+1 container statuses recorded) Jan 9 23:10:17.602: INFO: Container webserver ready: false, restart count 0 Jan 9 23:10:17.877: INFO: Latency metrics for node ip-172-20-55-216.ca-central-1.compute.internal Jan 9 23:10:17.877: INFO: Logging node info for node ip-172-20-57-251.ca-central-1.compute.internal Jan 9 23:10:17.909: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-57-251.ca-central-1.compute.internal eaaaf6eb-cbd5-4763-9c39-fb0e1a4ce44a 45391 0 2023-01-09 22:38:45 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ca-central-1 failure-domain.beta.kubernetes.io/zone:ca-central-1a kubelet_cleanup:true kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-57-251.ca-central-1.compute.internal kubernetes.io/os:linux kubernetes.io/role:node node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:ca-central-1a topology.hostpath.csi/node:ip-172-20-57-251.ca-central-1.compute.internal topology.kubernetes.io/region:ca-central-1 topology.kubernetes.io/zone:ca-central-1a] map[csi.volume.kubernetes.io/nodeid:{"csi-mock-csi-mock-volumes-9402":"csi-mock-csi-mock-volumes-9402","ebs.csi.aws.com":"i-0513e7d6e5eba8eec"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kops-controller Update v1 2023-01-09 22:38:45 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}} } {kube-controller-manager Update v1 2023-01-09 22:38:45 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.2.0/24\"":{}}}} } {kubelet Update v1 2023-01-09 22:38:45 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {e2e.test Update v1 2023-01-09 22:54:42 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubelet_cleanup":{}}}} } {kube-controller-manager Update v1 2023-01-09 23:00:23 +0000 UTC FieldsV1 {"f:status":{"f:volumesAttached":{}}} status} {kubelet Update v1 2023-01-09 23:00:27 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.2.0/24,DoNotUseExternalID:,ProviderID:aws:///ca-central-1a/i-0513e7d6e5eba8eec,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.2.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051652608 0} {<nil>} 3956692Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946795008 0} {<nil>} 3854292Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-01-09 23:06:06 +0000 UTC,LastTransitionTime:2023-01-09 22:38:45 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-01-09 23:06:06 +0000 UTC,LastTransitionTime:2023-01-09 22:38:45 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-01-09 23:06:06 +0000 UTC,LastTransitionTime:2023-01-09 22:38:45 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-01-09 23:06:06 +0000 UTC,LastTransitionTime:2023-01-09 22:39:05 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.57.251,},NodeAddress{Type:ExternalIP,Address:3.96.144.209,},NodeAddress{Type:Hostname,Address:ip-172-20-57-251.ca-central-1.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-57-251.ca-central-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-3-96-144-209.ca-central-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2f0d7ba95b37a96a1d9360f6071d84,SystemUUID:ec2f0d7b-a95b-37a9-6a1d-9360f6071d84,BootID:67865282-1911-4e38-8675-a7b78d5ae45e,KernelVersion:5.15.0-1026-aws,OSImage:Ubuntu 22.04.1 LTS,ContainerRuntimeVersion:docker://20.10.17,KubeletVersion:v1.23.15,KubeProxyVersion:v1.23.15,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:456569528,},ContainerImage{Names:[k8s.gcr.io/etcd@sha256:dd75ec974b0a2a6f6bb47001ba09207976e625db898d1b16735528c009cb171c k8s.gcr.io/etcd:3.5.6-0],SizeBytes:299475478,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/jessie-dnsutils@sha256:11e6a66017ba4e4b938c1612b7a54a3befcefd354796c04e1dba76873a13518e k8s.gcr.io/e2e-test-images/jessie-dnsutils:1.5],SizeBytes:253371792,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:20f25f275d46aa728f7615a1ccc19c78b2ed89435bf943a44b339f70f45508e6 k8s.gcr.io/e2e-test-images/httpd:2.4.39-2],SizeBytes:126894770,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:7e8bdd271312fd25fc5ff5a8f04727be84044eb3d7d8d03611972a6752e2e11e k8s.gcr.io/e2e-test-images/agnhost:2.39],SizeBytes:126872991,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nautilus@sha256:99c0d6f1ad24a1aa1905d9c6534d193f268f7b23f9add2ae6bb41f31094bdd5c k8s.gcr.io/e2e-test-images/nautilus:1.5],SizeBytes:124758741,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:1b9d1b2f36cb2dbee1960e82a9344aeb11bd4c4c03abf5e1853e0559c23855e3 k8s.gcr.io/e2e-test-images/httpd:2.4.38-2],SizeBytes:123781643,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.23.15 registry.k8s.io/kube-proxy-amd64:v1.23.15],SizeBytes:112336036,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:10e231cf07c82dfab5141c1afe548dc734e5fc1f67665ec3982a325d1bd31a9a registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.0],SizeBytes:94471559,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/sample-apiserver@sha256:f9c93b92b6ff750b41a93c4e4fe0bfe384597aeb841e2539d5444815c55b2d8f k8s.gcr.io/e2e-test-images/sample-apiserver:1.17.5],SizeBytes:56504541,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:c8e03f60afa90a28e4bb6ec9a8d0fc36d89de4b7475cf2d613afa793ec969fe0 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.0],SizeBytes:56382329,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:4e74c0492bceddc598de1c90cc5bc14dcda94cb49fa9c5bad9d117c4834b5e08 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.1],SizeBytes:56379269,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:36c31f7e1f433c9634d24f876353e8646246d81a03c4e351202c2644daff1620 k8s.gcr.io/sig-storage/csi-resizer:v1.2.0],SizeBytes:53956308,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:ed98431376c9e944e19a465fe8ea944806714dd95416a0821096c78d66b579bd k8s.gcr.io/sig-storage/csi-snapshotter:v4.1.1],SizeBytes:53531990,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:c5be65d6679efabb969d9b019300d187437ae876f992c40911fd2892bbef3b36 k8s.gcr.io/sig-storage/csi-attacher:v3.2.0],SizeBytes:53500826,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:60ab9b3e6a030d3038c87c0d6bca2930f58d1d72823e6a4af09767dc83b696a2 k8s.gcr.io/sig-storage/csi-attacher:v3.2.1],SizeBytes:53499570,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:7a5ba58a44e0d749e0767e4e37315bcf6a61f33ce3185c1991848af4db0fb70a k8s.gcr.io/sig-storage/csi-resizer:v1.1.0],SizeBytes:49176472,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/regression-issue-74839@sha256:b4f1d8d61bdad84bd50442d161d5460e4019d53e989b64220fdbc62fc87d76bf k8s.gcr.io/e2e-test-images/regression-issue-74839:1.2],SizeBytes:44576952,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nonroot@sha256:b9e2958a3dd879e3cf11142228c6d073d0fc4ea2e857c3be6f4fb0ab5fb2c937 k8s.gcr.io/e2e-test-images/nonroot:1.2],SizeBytes:42321438,},ContainerImage{Names:[k8s.gcr.io/sig-storage/hostpathplugin@sha256:232fe80174d60d520d36043103853a1d7ab4b7f3782cf43e45034f04ccda58ce k8s.gcr.io/sig-storage/hostpathplugin:v1.7.1],SizeBytes:29977921,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:a61d309da54641db41fb8f35718f744e9f730d4d0384f8c4b186ddc9f06cbd5f k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.1.0],SizeBytes:19662887,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:19582112,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:2dee3fe5fe861bb66c3a4ac51114f3447a4cd35870e0f2e2b558c7a400d89589 k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.2.0],SizeBytes:18699436,},ContainerImage{Names:[k8s.gcr.io/sig-storage/mock-driver@sha256:a7b517f9e0f42ffade802eef9cefa271372386b85f55b702b493241e58459793 k8s.gcr.io/sig-storage/mock-driver:v4.1.0],SizeBytes:17680993,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:17611980,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:529be2c9770add0cdd0c989115222ea9fc1be430c11095eb9f6dafcf98a36e2b k8s.gcr.io/sig-storage/livenessprobe:v2.4.0],SizeBytes:17175488,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:13616070e3f29de4417eee434a8ef472221c9e51b3d037b5a6b46cef08eb7443 k8s.gcr.io/e2e-test-images/nginx:1.14-2],SizeBytes:16032814,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nonewprivs@sha256:8ac1264691820febacf3aea5d152cbde6d10685731ec14966a9401c6f47a68ac k8s.gcr.io/e2e-test-images/nonewprivs:1.3],SizeBytes:7107254,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf k8s.gcr.io/e2e-test-images/busybox:1.29-2],SizeBytes:1154361,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:39e1e963e5310e9c313bad51523be012ede7b35bb9316517d19089a010356592 k8s.gcr.io/e2e-test-images/busybox:1.29-1],SizeBytes:1154361,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db k8s.gcr.io/pause:3.6 registry.k8s.io/pause:3.6],SizeBytes:682696,},},VolumesInUse:[kubernetes.io/csi/csi-mock-csi-mock-volumes-9402^4 kubernetes.io/csi/ebs.csi.aws.com^vol-02daff5e76e72db98],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-02daff5e76e72db98,DevicePath:,},},Config:nil,},} Jan 9 23:10:17.909: INFO: Logging kubelet events for node ip-172-20-57-251.ca-central-1.compute.internal Jan 9 23:10:17.943: INFO: Logging pods the kubelet thinks is on node ip-172-20-57-251.ca-central-1.compute.internal Jan 9 23:10:17.990: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-96np5 started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:10:17.990: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:10:17.990: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-f9w29 started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:10:17.990: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:10:17.990: INFO: pause-pod-78b7f9b75c-fckd2 started at 2023-01-09 23:05:44 +0000 UTC (0+1 container statuses recorded) Jan 9 23:10:17.990: INFO: Container agnhost-pause ready: false, restart count 0 Jan 9 23:10:17.990: INFO: netserver-2 started at 2023-01-09 23:07:18 +0000 UTC (0+1 container statuses recorded) Jan 9 23:10:17.990: INFO: Container webserver ready: false, restart count 0 Jan 9 23:10:17.990: INFO: csi-mockplugin-attacher-0 started at 2023-01-09 23:00:20 +0000 UTC (0+1 container statuses recorded) Jan 9 23:10:17.990: INFO: Container csi-attacher ready: false, restart count 0 Jan 9 23:10:17.990: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-rbqs2 started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:10:17.990: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:10:17.990: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-qzf5t started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:10:17.990: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:10:17.990: INFO: busybox-scheduling-6815563e-4dc7-43f5-a3cd-3e998a31b84b started at 2023-01-09 23:06:39 +0000 UTC (0+1 container statuses recorded) Jan 9 23:10:17.990: INFO: Container busybox-scheduling-6815563e-4dc7-43f5-a3cd-3e998a31b84b ready: false, restart count 0 Jan 9 23:10:17.990: INFO: netserver-2 started at 2023-01-09 23:08:45 +0000 UTC (0+1 container statuses recorded) Jan 9 23:10:17.990: INFO: Container webserver ready: false, restart count 0 Jan 9 23:10:17.990: INFO: csi-mockplugin-0 started at 2023-01-09 23:05:43 +0000 UTC (0+3 container statuses recorded) Jan 9 23:10:17.990: INFO: Container csi-provisioner ready: false, restart count 0 Jan 9 23:10:17.990: INFO: Container driver-registrar ready: false, restart count 0 Jan 9 23:10:17.990: INFO: Container mock ready: false, restart count 0 Jan 9 23:10:17.990: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-zx8gm started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:10:17.990: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:10:17.990: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-zvfdw started at 2023-01-09 22:54:43 +0000 UTC (0+1 container statuses recorded) Jan 9 23:10:17.990: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:10:17.990: INFO: cilium-6p9jz started at 2023-01-09 22:38:46 +0000 UTC (1+1 container statuses recorded) Jan 9 23:10:17.990: INFO: Init container clean-cilium-state ready: true, restart count 0 Jan 9 23:10:17.990: INFO: Container cilium-agent ready: true, restart count 0 Jan 9 23:10:17.990: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-wvflq started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:10:17.991: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:10:17.991: INFO: hostexec-ip-172-20-57-251.ca-central-1.compute.internal-f9vsk started at 2023-01-09 23:08:31 +0000 UTC (0+1 container statuses recorded) Jan 9 23:10:17.991: INFO: Container agnhost-container ready: true, restart count 0 Jan 9 23:10:17.991: INFO: pod-subpath-test-preprovisionedpv-ssct started at 2023-01-09 23:08:48 +0000 UTC (0+2 container statuses recorded) Jan 9 23:10:17.991: INFO: Container test-container-subpath-preprovisionedpv-ssct ready: false, restart count 0 Jan 9 23:10:17.991: INFO: Container test-container-volume-preprovisionedpv-ssct ready: false, restart count 0 Jan 9 23:10:17.991: INFO: hostpath-symlink-prep-volume-6746 started at 2023-01-09 23:07:44 +0000 UTC (0+1 container statuses recorded) Jan 9 23:10:17.991: INFO: Container init-volume-volume-6746 ready: false, restart count 0 Jan 9 23:10:17.991: INFO: pvc-volume-tester-2vlf5 started at 2023-01-09 23:06:05 +0000 UTC (0+1 container statuses recorded) Jan 9 23:10:17.991: INFO: Container volume-tester ready: false, restart count 0 Jan 9 23:10:17.991: INFO: rs-56t9b started at 2023-01-09 22:59:56 +0000 UTC (0+1 container statuses recorded) Jan 9 23:10:17.991: INFO: Container donothing ready: true, restart count 0 Jan 9 23:10:17.991: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-tvxfq started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:10:17.991: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:10:17.991: INFO: csi-mockplugin-0 started at 2023-01-09 23:00:20 +0000 UTC (0+3 container statuses recorded) Jan 9 23:10:17.991: INFO: Container csi-provisioner ready: true, restart count 0 Jan 9 23:10:17.991: INFO: Container driver-registrar ready: true, restart count 0 Jan 9 23:10:17.991: INFO: Container mock ready: true, restart count 0 Jan 9 23:10:17.991: INFO: pod-d16d48d3-63ef-4a76-95b8-d80f8b4aec62 started at 2023-01-09 23:03:17 +0000 UTC (0+1 container statuses recorded) Jan 9 23:10:17.991: INFO: Container write-pod ready: false, restart count 0 Jan 9 23:10:17.991: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-m8cfz started at 2023-01-09 22:54:43 +0000 UTC (0+1 container statuses recorded) Jan 9 23:10:17.991: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:10:17.991: INFO: rs-49kqp started at 2023-01-09 22:59:56 +0000 UTC (0+1 container statuses recorded) Jan 9 23:10:17.991: INFO: Container donothing ready: false, restart count 0 Jan 9 23:10:17.991: INFO: rs-dzbwc started at 2023-01-09 22:59:56 +0000 UTC (0+1 container statuses recorded) Jan 9 23:10:17.991: INFO: Container donothing ready: false, restart count 0 Jan 9 23:10:17.991: INFO: ebs-csi-node-g4ngc started at 2023-01-09 22:38:46 +0000 UTC (0+3 container statuses recorded) Jan 9 23:10:17.991: INFO: Container ebs-plugin ready: true, restart count 0 Jan 9 23:10:17.991: INFO: Container liveness-probe ready: true, restart count 0 Jan 9 23:10:17.991: INFO: Container node-driver-registrar ready: true, restart count 0 Jan 9 23:10:17.991: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-wmqwf started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:10:17.991: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:10:18.270: INFO: Latency metrics for node ip-172-20-57-251.ca-central-1.compute.internal Jan 9 23:10:18.270: INFO: Logging node info for node ip-172-20-59-208.ca-central-1.compute.internal Jan 9 23:10:18.302: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-59-208.ca-central-1.compute.internal d061a4d4-9ff6-43fc-89cc-422c61f7441c 45677 0 2023-01-09 22:38:43 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ca-central-1 failure-domain.beta.kubernetes.io/zone:ca-central-1a kubelet_cleanup:true kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-59-208.ca-central-1.compute.internal kubernetes.io/os:linux kubernetes.io/role:node node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:ca-central-1a topology.hostpath.csi/node:ip-172-20-59-208.ca-central-1.compute.internal topology.kubernetes.io/region:ca-central-1 topology.kubernetes.io/zone:ca-central-1a] map[csi.volume.kubernetes.io/nodeid:{"csi-hostpath-ephemeral-7223":"ip-172-20-59-208.ca-central-1.compute.internal","ebs.csi.aws.com":"i-0f00ea8bafbacdd11"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kops-controller Update v1 2023-01-09 22:38:43 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}} } {kube-controller-manager Update v1 2023-01-09 22:38:43 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.1.0/24\"":{}}}} } {kubelet Update v1 2023-01-09 22:38:43 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {e2e.test Update v1 2023-01-09 22:54:42 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubelet_cleanup":{}}}} } {kube-controller-manager Update v1 2023-01-09 23:06:36 +0000 UTC FieldsV1 {"f:status":{"f:volumesAttached":{}}} status} {kubelet Update v1 2023-01-09 23:06:40 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:capacity":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.1.0/24,DoNotUseExternalID:,ProviderID:aws:///ca-central-1a/i-0f00ea8bafbacdd11,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.1.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051652608 0} {<nil>} 3956692Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946795008 0} {<nil>} 3854292Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-01-09 23:06:40 +0000 UTC,LastTransitionTime:2023-01-09 22:38:43 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-01-09 23:06:40 +0000 UTC,LastTransitionTime:2023-01-09 22:38:43 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-01-09 23:06:40 +0000 UTC,LastTransitionTime:2023-01-09 22:38:43 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-01-09 23:06:40 +0000 UTC,LastTransitionTime:2023-01-09 22:39:03 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.59.208,},NodeAddress{Type:ExternalIP,Address:99.79.51.95,},NodeAddress{Type:Hostname,Address:ip-172-20-59-208.ca-central-1.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-59-208.ca-central-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-99-79-51-95.ca-central-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec28d5ecfae27d4bcedd89eca1a365f0,SystemUUID:ec28d5ec-fae2-7d4b-cedd-89eca1a365f0,BootID:560320f7-12c5-44f2-a7f6-a7a40031e204,KernelVersion:5.15.0-1026-aws,OSImage:Ubuntu 22.04.1 LTS,ContainerRuntimeVersion:docker://20.10.17,KubeletVersion:v1.23.15,KubeProxyVersion:v1.23.15,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:456569528,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:20f25f275d46aa728f7615a1ccc19c78b2ed89435bf943a44b339f70f45508e6 k8s.gcr.io/e2e-test-images/httpd:2.4.39-2],SizeBytes:126894770,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:7e8bdd271312fd25fc5ff5a8f04727be84044eb3d7d8d03611972a6752e2e11e k8s.gcr.io/e2e-test-images/agnhost:2.39],SizeBytes:126872991,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:1b9d1b2f36cb2dbee1960e82a9344aeb11bd4c4c03abf5e1853e0559c23855e3 k8s.gcr.io/e2e-test-images/httpd:2.4.38-2],SizeBytes:123781643,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.23.15 registry.k8s.io/kube-proxy-amd64:v1.23.15],SizeBytes:112336036,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:10e231cf07c82dfab5141c1afe548dc734e5fc1f67665ec3982a325d1bd31a9a registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.0],SizeBytes:94471559,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:c8e03f60afa90a28e4bb6ec9a8d0fc36d89de4b7475cf2d613afa793ec969fe0 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.0],SizeBytes:56382329,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:4e74c0492bceddc598de1c90cc5bc14dcda94cb49fa9c5bad9d117c4834b5e08 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.1],SizeBytes:56379269,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:36c31f7e1f433c9634d24f876353e8646246d81a03c4e351202c2644daff1620 k8s.gcr.io/sig-storage/csi-resizer:v1.2.0],SizeBytes:53956308,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:ed98431376c9e944e19a465fe8ea944806714dd95416a0821096c78d66b579bd k8s.gcr.io/sig-storage/csi-snapshotter:v4.1.1],SizeBytes:53531990,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:c5be65d6679efabb969d9b019300d187437ae876f992c40911fd2892bbef3b36 k8s.gcr.io/sig-storage/csi-attacher:v3.2.0],SizeBytes:53500826,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:60ab9b3e6a030d3038c87c0d6bca2930f58d1d72823e6a4af09767dc83b696a2 k8s.gcr.io/sig-storage/csi-attacher:v3.2.1],SizeBytes:53499570,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:017727efcfeb7d053af68e51436ce8e65edbc6ca573720afb4f79c8594036955 registry.k8s.io/coredns/coredns:v1.10.0],SizeBytes:50249337,},ContainerImage{Names:[registry.k8s.io/cpa/cluster-proportional-autoscaler@sha256:68d396900aeaa072c1f27289485fdac29834045a6f3ffe369bf389d830ef572d registry.k8s.io/cpa/cluster-proportional-autoscaler:1.8.6],SizeBytes:47910609,},ContainerImage{Names:[k8s.gcr.io/sig-storage/hostpathplugin@sha256:232fe80174d60d520d36043103853a1d7ab4b7f3782cf43e45034f04ccda58ce k8s.gcr.io/sig-storage/hostpathplugin:v1.7.1],SizeBytes:29977921,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:a61d309da54641db41fb8f35718f744e9f730d4d0384f8c4b186ddc9f06cbd5f k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.1.0],SizeBytes:19662887,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:19582112,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:2dee3fe5fe861bb66c3a4ac51114f3447a4cd35870e0f2e2b558c7a400d89589 k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.2.0],SizeBytes:18699436,},ContainerImage{Names:[k8s.gcr.io/sig-storage/mock-driver@sha256:a7b517f9e0f42ffade802eef9cefa271372386b85f55b702b493241e58459793 k8s.gcr.io/sig-storage/mock-driver:v4.1.0],SizeBytes:17680993,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:17611980,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:529be2c9770add0cdd0c989115222ea9fc1be430c11095eb9f6dafcf98a36e2b k8s.gcr.io/sig-storage/livenessprobe:v2.4.0],SizeBytes:17175488,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:13616070e3f29de4417eee434a8ef472221c9e51b3d037b5a6b46cef08eb7443 k8s.gcr.io/e2e-test-images/nginx:1.14-2],SizeBytes:16032814,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nonewprivs@sha256:8ac1264691820febacf3aea5d152cbde6d10685731ec14966a9401c6f47a68ac k8s.gcr.io/e2e-test-images/nonewprivs:1.3],SizeBytes:7107254,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf k8s.gcr.io/e2e-test-images/busybox:1.29-2],SizeBytes:1154361,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:39e1e963e5310e9c313bad51523be012ede7b35bb9316517d19089a010356592 k8s.gcr.io/e2e-test-images/busybox:1.29-1],SizeBytes:1154361,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db k8s.gcr.io/pause:3.6 registry.k8s.io/pause:3.6],SizeBytes:682696,},},VolumesInUse:[kubernetes.io/csi/csi-hostpath-ephemeral-7223^43b8adc9-9072-11ed-b4b6-0a454e7330ed kubernetes.io/csi/csi-hostpath-ephemeral-7223^43b8d4d5-9072-11ed-b4b6-0a454e7330ed],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/csi-hostpath-ephemeral-7223^43b8d4d5-9072-11ed-b4b6-0a454e7330ed,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/csi-hostpath-ephemeral-7223^43b8adc9-9072-11ed-b4b6-0a454e7330ed,DevicePath:,},},Config:nil,},} Jan 9 23:10:18.302: INFO: Logging kubelet events for node ip-172-20-59-208.ca-central-1.compute.internal Jan 9 23:10:18.336: INFO: Logging pods the kubelet thinks is on node ip-172-20-59-208.ca-central-1.compute.internal Jan 9 23:10:18.386: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-bhxkw started at 2023-01-09 22:54:43 +0000 UTC (0+1 container statuses recorded) Jan 9 23:10:18.386: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:10:18.386: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-rzgjf started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:10:18.386: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:10:18.386: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-ptfgl started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:10:18.386: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:10:18.386: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-t5g85 started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:10:18.386: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:10:18.386: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-hjsbl started at 2023-01-09 22:54:43 +0000 UTC (0+1 container statuses recorded) Jan 9 23:10:18.386: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:10:18.386: INFO: pod-e1224014-09b8-42e1-b210-71a9acc98e21 started at 2023-01-09 23:06:29 +0000 UTC (0+1 container statuses recorded) Jan 9 23:10:18.386: INFO: Container write-pod ready: false, restart count 0 Jan 9 23:10:18.386: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-x682f started at 2023-01-09 22:54:43 +0000 UTC (0+1 container statuses recorded) Jan 9 23:10:18.386: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:10:18.386: INFO: ebs-csi-node-prrrf started at 2023-01-09 22:38:44 +0000 UTC (0+3 container statuses recorded) Jan 9 23:10:18.386: INFO: Container ebs-plugin ready: true, restart count 0 Jan 9 23:10:18.386: INFO: Container liveness-probe ready: true, restart count 0 Jan 9 23:10:18.386: INFO: Container node-driver-registrar ready: true, restart count 0 Jan 9 23:10:18.386: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-tx2mm started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:10:18.386: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:10:18.386: INFO: csi-hostpathplugin-0 started at 2023-01-09 23:03:05 +0000 UTC (0+7 container statuses recorded) Jan 9 23:10:18.386: INFO: Container csi-attacher ready: true, restart count 0 Jan 9 23:10:18.386: INFO: Container csi-provisioner ready: true, restart count 0 Jan 9 23:10:18.386: INFO: Container csi-resizer ready: true, restart count 0 Jan 9 23:10:18.386: INFO: Container csi-snapshotter ready: true, restart count 0 Jan 9 23:10:18.386: INFO: Container hostpath ready: true, restart count 0 Jan 9 23:10:18.386: INFO: Container liveness-probe ready: true, restart count 0 Jan 9 23:10:18.386: INFO: Container node-driver-registrar ready: true, restart count 0 Jan 9 23:10:18.386: INFO: hostexec-ip-172-20-59-208.ca-central-1.compute.internal-sn7w7 started at 2023-01-09 23:06:26 +0000 UTC (0+1 container statuses recorded) Jan 9 23:10:18.386: INFO: Container agnhost-container ready: true, restart count 0 Jan 9 23:10:18.386: INFO: inline-volume-tester-xnqfb started at 2023-01-09 23:06:35 +0000 UTC (0+1 container statuses recorded) Jan 9 23:10:18.386: INFO: Container csi-volume-tester ready: false, restart count 0 Jan 9 23:10:18.386: INFO: cilium-z2ltf started at 2023-01-09 22:38:44 +0000 UTC (1+1 container statuses recorded) Jan 9 23:10:18.386: INFO: Init container clean-cilium-state ready: true, restart count 0 Jan 9 23:10:18.386: INFO: Container cilium-agent ready: true, restart count 0 Jan 9 23:10:18.386: INFO: netserver-3 started at 2023-01-09 23:08:45 +0000 UTC (0+1 container statuses recorded) Jan 9 23:10:18.386: INFO: Container webserver ready: false, restart count 0 Jan 9 23:10:18.386: INFO: coredns-867df8f45c-tmrqb started at 2023-01-09 22:39:03 +0000 UTC (0+1 container statuses recorded) Jan 9 23:10:18.386: INFO: Container coredns ready: true, restart count 0 Jan 9 23:10:18.386: INFO: netserver-3 started at 2023-01-09 23:07:18 +0000 UTC (0+1 container statuses recorded) Jan 9 23:10:18.386: INFO: Container webserver ready: false, restart count 0 Jan 9 23:10:18.387: INFO: rs-x69wn started at 2023-01-09 22:59:56 +0000 UTC (0+1 container statuses recorded) Jan 9 23:10:18.387: INFO: Container donothing ready: false, restart count 0 Jan 9 23:10:18.387: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-hszdf started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:10:18.387: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:10:18.387: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-fc6pg started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:10:18.387: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:10:18.387: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-8qp6l started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:10:18.387: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: false, restart count 0 Jan 9 23:10:18.387: INFO: rs-nd6n2 started at 2023-01-09 22:59:56 +0000 UTC (0+1 container statuses recorded) Jan 9 23:10:18.387: INFO: Container donothing ready: false, restart count 0 Jan 9 23:10:18.387: INFO: coredns-autoscaler-557ccb4c66-jnjcc started at 2023-01-09 22:39:03 +0000 UTC (0+1 container statuses recorded) Jan 9 23:10:18.387: INFO: Container autoscaler ready: true, restart count 0 Jan 9 23:10:18.614: INFO: Latency metrics for node ip-172-20-59-208.ca-central-1.compute.internal Jan 9 23:10:18.615: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready �[1mSTEP�[0m: Destroying namespace "disruption-3102" for this suite.
Filter through log files
go run hack/e2e.go -v --test --test_args='--ginkgo.focus=Kubernetes\se2e\ssuite\s\[sig\-apps\]\sJob\sshould\srun\sa\sjob\sto\scompletion\swhen\stasks\ssometimes\sfail\sand\sare\snot\slocally\srestarted$'
/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/apps/job.go:233 Jan 9 23:11:53.184: failed to ensure job completion in namespace: job-4121 Unexpected error: <*errors.errorString | 0xc0002462d0>: { s: "timed out waiting for the condition", } timed out waiting for the condition occurred /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/apps/job.go:252from junit_11.xml
[BeforeEach] [sig-apps] Job /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:185 �[1mSTEP�[0m: Creating a kubernetes client Jan 9 22:56:52.660: INFO: >>> kubeConfig: /root/.kube/config �[1mSTEP�[0m: Building a namespace api object, basename job �[1mSTEP�[0m: Waiting for a default service account to be provisioned in namespace �[1mSTEP�[0m: Waiting for kube-root-ca.crt to be provisioned in namespace [It] should run a job to completion when tasks sometimes fail and are not locally restarted /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/apps/job.go:233 �[1mSTEP�[0m: Looking for a node to schedule job pod �[1mSTEP�[0m: Creating a job �[1mSTEP�[0m: Ensuring job reaches completions Jan 9 23:11:53.184: FAIL: failed to ensure job completion in namespace: job-4121 Unexpected error: <*errors.errorString | 0xc0002462d0>: { s: "timed out waiting for the condition", } timed out waiting for the condition occurred Full Stack Trace k8s.io/kubernetes/test/e2e/apps.glob..func6.7() /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/apps/job.go:252 +0x29f k8s.io/kubernetes/test/e2e.RunE2ETests(0x24c66d7) _output/dockerized/go/src/k8s.io/kubernetes/test/e2e/e2e.go:133 +0x697 k8s.io/kubernetes/test/e2e.TestE2E(0x9e000002c9) _output/dockerized/go/src/k8s.io/kubernetes/test/e2e/e2e_test.go:136 +0x19 testing.tRunner(0xc000838ea0, 0x735e880) /usr/local/go/src/testing/testing.go:1259 +0x102 created by testing.(*T).Run /usr/local/go/src/testing/testing.go:1306 +0x35a [AfterEach] [sig-apps] Job /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:186 �[1mSTEP�[0m: Collecting events from namespace "job-4121". �[1mSTEP�[0m: Found 24 events. Jan 9 23:11:53.218: INFO: At 2023-01-09 22:56:53 +0000 UTC - event for fail-once-non-local: {job-controller } SuccessfulCreate: Created pod: fail-once-non-local-nx7cr Jan 9 23:11:53.218: INFO: At 2023-01-09 22:56:53 +0000 UTC - event for fail-once-non-local: {job-controller } SuccessfulCreate: Created pod: fail-once-non-local-mh6qc Jan 9 23:11:53.218: INFO: At 2023-01-09 22:56:55 +0000 UTC - event for fail-once-non-local-mh6qc: {kubelet ip-172-20-55-216.ca-central-1.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "04454ddff346b26e8a927b425452692b56a78b2ba26182b4ca55bf4c081ee2ec" network for pod "fail-once-non-local-mh6qc": networkPlugin cni failed to set up pod "fail-once-non-local-mh6qc_job-4121" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 9 23:11:53.218: INFO: At 2023-01-09 22:56:55 +0000 UTC - event for fail-once-non-local-nx7cr: {kubelet ip-172-20-55-216.ca-central-1.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "a1394b312552cc689c8322d3b58247ce577b05611f0fa807466bf27cb98ee925" network for pod "fail-once-non-local-nx7cr": networkPlugin cni failed to set up pod "fail-once-non-local-nx7cr_job-4121" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 9 23:11:53.218: INFO: At 2023-01-09 22:56:57 +0000 UTC - event for fail-once-non-local-mh6qc: {kubelet ip-172-20-55-216.ca-central-1.compute.internal} SandboxChanged: Pod sandbox changed, it will be killed and re-created. Jan 9 23:11:53.218: INFO: At 2023-01-09 22:56:57 +0000 UTC - event for fail-once-non-local-nx7cr: {kubelet ip-172-20-55-216.ca-central-1.compute.internal} SandboxChanged: Pod sandbox changed, it will be killed and re-created. Jan 9 23:11:53.218: INFO: At 2023-01-09 22:56:59 +0000 UTC - event for fail-once-non-local-mh6qc: {kubelet ip-172-20-55-216.ca-central-1.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "b640b0ea5be51e6fe69122adcb3cb02b0e1589baa1e7c0faa2696af6c0466398" network for pod "fail-once-non-local-mh6qc": networkPlugin cni failed to set up pod "fail-once-non-local-mh6qc_job-4121" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 9 23:11:53.218: INFO: At 2023-01-09 22:56:59 +0000 UTC - event for fail-once-non-local-nx7cr: {kubelet ip-172-20-55-216.ca-central-1.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "2de9c9e57e6560e958815f5dace80f155fd8284c351beffe5cadd25a78d6f37f" network for pod "fail-once-non-local-nx7cr": networkPlugin cni failed to set up pod "fail-once-non-local-nx7cr_job-4121" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 9 23:11:53.218: INFO: At 2023-01-09 22:57:01 +0000 UTC - event for fail-once-non-local-mh6qc: {kubelet ip-172-20-55-216.ca-central-1.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "1bffa4b13cc7afa84d2f11de3cacfebba7ff67cc4e1c7480a71c006d2a3681c8" network for pod "fail-once-non-local-mh6qc": networkPlugin cni failed to set up pod "fail-once-non-local-mh6qc_job-4121" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 9 23:11:53.218: INFO: At 2023-01-09 22:57:04 +0000 UTC - event for fail-once-non-local-nx7cr: {kubelet ip-172-20-55-216.ca-central-1.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "338de6ca8d1ec9d07a349cd7f98490fa44503f962e129e40bc4abe5bb9ec5755" network for pod "fail-once-non-local-nx7cr": networkPlugin cni failed to set up pod "fail-once-non-local-nx7cr_job-4121" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 9 23:11:53.218: INFO: At 2023-01-09 22:57:05 +0000 UTC - event for fail-once-non-local-mh6qc: {kubelet ip-172-20-55-216.ca-central-1.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "0f078d6b543e1624701d1800f00a1d0eb09e4b2a45ecb564b8942d651d12168f" network for pod "fail-once-non-local-mh6qc": networkPlugin cni failed to set up pod "fail-once-non-local-mh6qc_job-4121" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 9 23:11:53.218: INFO: At 2023-01-09 22:57:05 +0000 UTC - event for fail-once-non-local-nx7cr: {kubelet ip-172-20-55-216.ca-central-1.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "3c67c59497ccf0a31fffffe6db4be5edf9c8bbe7a657e20e2791d0a798b7d25b" network for pod "fail-once-non-local-nx7cr": networkPlugin cni failed to set up pod "fail-once-non-local-nx7cr_job-4121" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 9 23:11:53.218: INFO: At 2023-01-09 22:57:08 +0000 UTC - event for fail-once-non-local-mh6qc: {kubelet ip-172-20-55-216.ca-central-1.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "b4ec7a8e59d6a9dfcfb1c7407734d71aedb89b6873a6ce77d2521ff389811625" network for pod "fail-once-non-local-mh6qc": networkPlugin cni failed to set up pod "fail-once-non-local-mh6qc_job-4121" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 9 23:11:53.218: INFO: At 2023-01-09 22:57:08 +0000 UTC - event for fail-once-non-local-nx7cr: {kubelet ip-172-20-55-216.ca-central-1.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "e883da0ee30582e8715fe49c23b26c3798f8c7575301d327ef09a6a2acee4f0d" network for pod "fail-once-non-local-nx7cr": networkPlugin cni failed to set up pod "fail-once-non-local-nx7cr_job-4121" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 9 23:11:53.218: INFO: At 2023-01-09 22:57:10 +0000 UTC - event for fail-once-non-local-mh6qc: {kubelet ip-172-20-55-216.ca-central-1.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "7531fe00465ca91f1ba00a59c7a53b8b0d5ebf91b1d7b92bd155049128b65a83" network for pod "fail-once-non-local-mh6qc": networkPlugin cni failed to set up pod "fail-once-non-local-mh6qc_job-4121" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 9 23:11:53.218: INFO: At 2023-01-09 22:57:10 +0000 UTC - event for fail-once-non-local-nx7cr: {kubelet ip-172-20-55-216.ca-central-1.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "f0f83179d108cf413d7f3ed2d015808cf1dc936e3881f652da46530ee20141e3" network for pod "fail-once-non-local-nx7cr": networkPlugin cni failed to set up pod "fail-once-non-local-nx7cr_job-4121" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 9 23:11:53.218: INFO: At 2023-01-09 22:57:12 +0000 UTC - event for fail-once-non-local-nx7cr: {kubelet ip-172-20-55-216.ca-central-1.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "743e36ac851beadb5bcf3ce6107a524ab459e9925f2a3e03913a66b20bbed818" network for pod "fail-once-non-local-nx7cr": networkPlugin cni failed to set up pod "fail-once-non-local-nx7cr_job-4121" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 9 23:11:53.218: INFO: At 2023-01-09 22:57:13 +0000 UTC - event for fail-once-non-local-mh6qc: {kubelet ip-172-20-55-216.ca-central-1.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "4cb31b450182402b2be6fe18196523271ce3cfb3781babf7d9d3b6185753cb6b" network for pod "fail-once-non-local-mh6qc": networkPlugin cni failed to set up pod "fail-once-non-local-mh6qc_job-4121" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 9 23:11:53.218: INFO: At 2023-01-09 22:57:15 +0000 UTC - event for fail-once-non-local-nx7cr: {kubelet ip-172-20-55-216.ca-central-1.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "bc8c9bf60eab1be4f930e96283115be071a31ec84563ddba374543588b9d8ff5" network for pod "fail-once-non-local-nx7cr": networkPlugin cni failed to set up pod "fail-once-non-local-nx7cr_job-4121" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 9 23:11:53.218: INFO: At 2023-01-09 22:57:17 +0000 UTC - event for fail-once-non-local-mh6qc: {kubelet ip-172-20-55-216.ca-central-1.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "3db46f3cf4b293fcb2012a47819e24d79da0748b5c11dc607aaed557e55a277e" network for pod "fail-once-non-local-mh6qc": networkPlugin cni failed to set up pod "fail-once-non-local-mh6qc_job-4121" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 9 23:11:53.218: INFO: At 2023-01-09 22:57:18 +0000 UTC - event for fail-once-non-local-nx7cr: {kubelet ip-172-20-55-216.ca-central-1.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "12db788f8bddeeee3c0c5159a800c671622dbb58522234ee2137b2f41850c10e" network for pod "fail-once-non-local-nx7cr": networkPlugin cni failed to set up pod "fail-once-non-local-nx7cr_job-4121" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 9 23:11:53.218: INFO: At 2023-01-09 22:57:19 +0000 UTC - event for fail-once-non-local-mh6qc: {kubelet ip-172-20-55-216.ca-central-1.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "4efd55478c74698b52b8ce1171855cb52efa7741152e6d6d93cdd18426f2f339" network for pod "fail-once-non-local-mh6qc": networkPlugin cni failed to set up pod "fail-once-non-local-mh6qc_job-4121" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 9 23:11:53.218: INFO: At 2023-01-09 22:57:23 +0000 UTC - event for fail-once-non-local-mh6qc: {kubelet ip-172-20-55-216.ca-central-1.compute.internal} FailedCreatePodSandBox: (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "8e5d9f501ab78501671083e0c60e35b5d91e3833da6be332231b9882de37fc47" network for pod "fail-once-non-local-mh6qc": networkPlugin cni failed to set up pod "fail-once-non-local-mh6qc_job-4121" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 9 23:11:53.218: INFO: At 2023-01-09 22:57:23 +0000 UTC - event for fail-once-non-local-nx7cr: {kubelet ip-172-20-55-216.ca-central-1.compute.internal} FailedCreatePodSandBox: (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "75a1fdf2305dfb151679c3fed8a1b242ba3050ddf51a20e9a4ddd9636877991c" network for pod "fail-once-non-local-nx7cr": networkPlugin cni failed to set up pod "fail-once-non-local-nx7cr_job-4121" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 9 23:11:53.250: INFO: POD NODE PHASE GRACE CONDITIONS Jan 9 23:11:53.250: INFO: fail-once-non-local-mh6qc ip-172-20-55-216.ca-central-1.compute.internal Pending [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2023-01-09 22:56:53 +0000 UTC } {Ready False 0001-01-01 00:00:00 +0000 UTC 2023-01-09 22:56:53 +0000 UTC ContainersNotReady containers with unready status: [c]} {ContainersReady False 0001-01-01 00:00:00 +0000 UTC 2023-01-09 22:56:53 +0000 UTC ContainersNotReady containers with unready status: [c]} {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2023-01-09 22:56:53 +0000 UTC }] Jan 9 23:11:53.250: INFO: fail-once-non-local-nx7cr ip-172-20-55-216.ca-central-1.compute.internal Pending [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2023-01-09 22:56:53 +0000 UTC } {Ready False 0001-01-01 00:00:00 +0000 UTC 2023-01-09 22:56:53 +0000 UTC ContainersNotReady containers with unready status: [c]} {ContainersReady False 0001-01-01 00:00:00 +0000 UTC 2023-01-09 22:56:53 +0000 UTC ContainersNotReady containers with unready status: [c]} {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2023-01-09 22:56:53 +0000 UTC }] Jan 9 23:11:53.250: INFO: Jan 9 23:11:53.284: INFO: Unable to fetch job-4121/fail-once-non-local-mh6qc/c logs: the server rejected our request for an unknown reason (get pods fail-once-non-local-mh6qc) Jan 9 23:11:53.336: INFO: Unable to fetch job-4121/fail-once-non-local-nx7cr/c logs: the server rejected our request for an unknown reason (get pods fail-once-non-local-nx7cr) Jan 9 23:11:53.371: INFO: Logging node info for node ip-172-20-32-67.ca-central-1.compute.internal Jan 9 23:11:53.403: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-32-67.ca-central-1.compute.internal 1f51981b-a01d-404c-b3a4-6649f301f65d 47491 0 2023-01-09 22:38:45 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ca-central-1 failure-domain.beta.kubernetes.io/zone:ca-central-1a kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-32-67.ca-central-1.compute.internal kubernetes.io/os:linux kubernetes.io/role:node node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:ca-central-1a topology.hostpath.csi/node:ip-172-20-32-67.ca-central-1.compute.internal topology.kubernetes.io/region:ca-central-1 topology.kubernetes.io/zone:ca-central-1a] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-0272cebea4c6c3d5d"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kops-controller Update v1 2023-01-09 22:38:45 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}} } {kube-controller-manager Update v1 2023-01-09 22:38:45 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.3.0/24\"":{}}}} } {kubelet Update v1 2023-01-09 22:38:45 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {kube-controller-manager Update v1 2023-01-09 22:55:59 +0000 UTC FieldsV1 {"f:status":{"f:volumesAttached":{}}} status} {kubelet Update v1 2023-01-09 22:56:00 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:capacity":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.3.0/24,DoNotUseExternalID:,ProviderID:aws:///ca-central-1a/i-0272cebea4c6c3d5d,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.3.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051644416 0} {<nil>} 3956684Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946786816 0} {<nil>} 3854284Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-01-09 23:11:18 +0000 UTC,LastTransitionTime:2023-01-09 22:38:45 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-01-09 23:11:18 +0000 UTC,LastTransitionTime:2023-01-09 22:38:45 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-01-09 23:11:18 +0000 UTC,LastTransitionTime:2023-01-09 22:38:45 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-01-09 23:11:18 +0000 UTC,LastTransitionTime:2023-01-09 22:39:06 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.32.67,},NodeAddress{Type:ExternalIP,Address:35.182.2.102,},NodeAddress{Type:Hostname,Address:ip-172-20-32-67.ca-central-1.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-32-67.ca-central-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-35-182-2-102.ca-central-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2536b48bd6b6dade0ed68af2555085,SystemUUID:ec2536b4-8bd6-b6da-de0e-d68af2555085,BootID:bfb3b0df-aafa-481c-9ac6-f85e5e4580e3,KernelVersion:5.15.0-1026-aws,OSImage:Ubuntu 22.04.1 LTS,ContainerRuntimeVersion:docker://20.10.17,KubeletVersion:v1.23.15,KubeProxyVersion:v1.23.15,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:456569528,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/jessie-dnsutils@sha256:11e6a66017ba4e4b938c1612b7a54a3befcefd354796c04e1dba76873a13518e k8s.gcr.io/e2e-test-images/jessie-dnsutils:1.5],SizeBytes:253371792,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:7e8bdd271312fd25fc5ff5a8f04727be84044eb3d7d8d03611972a6752e2e11e k8s.gcr.io/e2e-test-images/agnhost:2.39],SizeBytes:126872991,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nautilus@sha256:99c0d6f1ad24a1aa1905d9c6534d193f268f7b23f9add2ae6bb41f31094bdd5c k8s.gcr.io/e2e-test-images/nautilus:1.5],SizeBytes:124758741,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:1b9d1b2f36cb2dbee1960e82a9344aeb11bd4c4c03abf5e1853e0559c23855e3 k8s.gcr.io/e2e-test-images/httpd:2.4.38-2],SizeBytes:123781643,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.23.15 registry.k8s.io/kube-proxy-amd64:v1.23.15],SizeBytes:112336036,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:10e231cf07c82dfab5141c1afe548dc734e5fc1f67665ec3982a325d1bd31a9a registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.0],SizeBytes:94471559,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:c8e03f60afa90a28e4bb6ec9a8d0fc36d89de4b7475cf2d613afa793ec969fe0 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.0],SizeBytes:56382329,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:4e74c0492bceddc598de1c90cc5bc14dcda94cb49fa9c5bad9d117c4834b5e08 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.1],SizeBytes:56379269,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:36c31f7e1f433c9634d24f876353e8646246d81a03c4e351202c2644daff1620 k8s.gcr.io/sig-storage/csi-resizer:v1.2.0],SizeBytes:53956308,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:ed98431376c9e944e19a465fe8ea944806714dd95416a0821096c78d66b579bd k8s.gcr.io/sig-storage/csi-snapshotter:v4.1.1],SizeBytes:53531990,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:c5be65d6679efabb969d9b019300d187437ae876f992c40911fd2892bbef3b36 k8s.gcr.io/sig-storage/csi-attacher:v3.2.0],SizeBytes:53500826,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:60ab9b3e6a030d3038c87c0d6bca2930f58d1d72823e6a4af09767dc83b696a2 k8s.gcr.io/sig-storage/csi-attacher:v3.2.1],SizeBytes:53499570,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:017727efcfeb7d053af68e51436ce8e65edbc6ca573720afb4f79c8594036955 registry.k8s.io/coredns/coredns:v1.10.0],SizeBytes:50249337,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:7a5ba58a44e0d749e0767e4e37315bcf6a61f33ce3185c1991848af4db0fb70a k8s.gcr.io/sig-storage/csi-resizer:v1.1.0],SizeBytes:49176472,},ContainerImage{Names:[k8s.gcr.io/sig-storage/hostpathplugin@sha256:232fe80174d60d520d36043103853a1d7ab4b7f3782cf43e45034f04ccda58ce k8s.gcr.io/sig-storage/hostpathplugin:v1.7.1],SizeBytes:29977921,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:a61d309da54641db41fb8f35718f744e9f730d4d0384f8c4b186ddc9f06cbd5f k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.1.0],SizeBytes:19662887,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:19582112,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:2dee3fe5fe861bb66c3a4ac51114f3447a4cd35870e0f2e2b558c7a400d89589 k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.2.0],SizeBytes:18699436,},ContainerImage{Names:[k8s.gcr.io/sig-storage/mock-driver@sha256:a7b517f9e0f42ffade802eef9cefa271372386b85f55b702b493241e58459793 k8s.gcr.io/sig-storage/mock-driver:v4.1.0],SizeBytes:17680993,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:17611980,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:529be2c9770add0cdd0c989115222ea9fc1be430c11095eb9f6dafcf98a36e2b k8s.gcr.io/sig-storage/livenessprobe:v2.4.0],SizeBytes:17175488,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:13616070e3f29de4417eee434a8ef472221c9e51b3d037b5a6b46cef08eb7443 k8s.gcr.io/e2e-test-images/nginx:1.14-2],SizeBytes:16032814,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nonewprivs@sha256:8ac1264691820febacf3aea5d152cbde6d10685731ec14966a9401c6f47a68ac k8s.gcr.io/e2e-test-images/nonewprivs:1.3],SizeBytes:7107254,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf k8s.gcr.io/e2e-test-images/busybox:1.29-2],SizeBytes:1154361,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:39e1e963e5310e9c313bad51523be012ede7b35bb9316517d19089a010356592 k8s.gcr.io/e2e-test-images/busybox:1.29-1],SizeBytes:1154361,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db k8s.gcr.io/pause:3.6 registry.k8s.io/pause:3.6],SizeBytes:682696,},},VolumesInUse:[kubernetes.io/csi/ebs.csi.aws.com^vol-01e8a4a8ff1da0e10],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-01e8a4a8ff1da0e10,DevicePath:,},},Config:nil,},} Jan 9 23:11:53.403: INFO: Logging kubelet events for node ip-172-20-32-67.ca-central-1.compute.internal Jan 9 23:11:53.444: INFO: Logging pods the kubelet thinks is on node ip-172-20-32-67.ca-central-1.compute.internal Jan 9 23:11:53.523: INFO: coredns-867df8f45c-sckrz started at 2023-01-09 22:39:23 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:53.524: INFO: Container coredns ready: true, restart count 0 Jan 9 23:11:53.524: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-fzrxx started at 2023-01-09 22:54:43 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:53.524: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:11:53.524: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-mlcl4 started at 2023-01-09 22:54:43 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:53.524: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:11:53.524: INFO: ebs-csi-node-lc4rb started at 2023-01-09 22:38:46 +0000 UTC (0+3 container statuses recorded) Jan 9 23:11:53.524: INFO: Container ebs-plugin ready: true, restart count 0 Jan 9 23:11:53.524: INFO: Container liveness-probe ready: true, restart count 0 Jan 9 23:11:53.524: INFO: Container node-driver-registrar ready: true, restart count 0 Jan 9 23:11:53.524: INFO: netserver-0 started at 2023-01-09 23:07:18 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:53.524: INFO: Container webserver ready: false, restart count 0 Jan 9 23:11:53.524: INFO: netserver-0 started at 2023-01-09 23:08:45 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:53.524: INFO: Container webserver ready: false, restart count 0 Jan 9 23:11:53.524: INFO: inline-volume-tester-5g6j6 started at 2023-01-09 22:55:56 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:53.524: INFO: Container csi-volume-tester ready: true, restart count 0 Jan 9 23:11:53.524: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-sddjv started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:53.524: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:11:53.524: INFO: csi-hostpathplugin-0 started at 2023-01-09 23:05:41 +0000 UTC (0+7 container statuses recorded) Jan 9 23:11:53.524: INFO: Container csi-attacher ready: false, restart count 0 Jan 9 23:11:53.524: INFO: Container csi-provisioner ready: false, restart count 0 Jan 9 23:11:53.524: INFO: Container csi-resizer ready: false, restart count 0 Jan 9 23:11:53.524: INFO: Container csi-snapshotter ready: false, restart count 0 Jan 9 23:11:53.524: INFO: Container hostpath ready: false, restart count 0 Jan 9 23:11:53.524: INFO: Container liveness-probe ready: false, restart count 0 Jan 9 23:11:53.524: INFO: Container node-driver-registrar ready: false, restart count 0 Jan 9 23:11:53.524: INFO: cilium-dvsfs started at 2023-01-09 22:38:46 +0000 UTC (1+1 container statuses recorded) Jan 9 23:11:53.524: INFO: Init container clean-cilium-state ready: true, restart count 0 Jan 9 23:11:53.524: INFO: Container cilium-agent ready: true, restart count 0 Jan 9 23:11:53.524: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-gjc2h started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:53.524: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:11:53.524: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-jb6kp started at 2023-01-09 22:54:43 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:53.524: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:11:53.524: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-hwbbn started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:53.524: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:11:53.524: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-fr98x started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:53.524: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:11:53.716: INFO: Latency metrics for node ip-172-20-32-67.ca-central-1.compute.internal Jan 9 23:11:53.717: INFO: Logging node info for node ip-172-20-37-64.ca-central-1.compute.internal Jan 9 23:11:53.748: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-37-64.ca-central-1.compute.internal d817613a-0fe5-40c0-a5e4-7b85b9f8506b 46487 0 2023-01-09 22:36:54 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:c5.large beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ca-central-1 failure-domain.beta.kubernetes.io/zone:ca-central-1a kops.k8s.io/kops-controller-pki: kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-37-64.ca-central-1.compute.internal kubernetes.io/os:linux kubernetes.io/role:master node-role.kubernetes.io/control-plane: node-role.kubernetes.io/master: node.kubernetes.io/exclude-from-external-load-balancers: node.kubernetes.io/instance-type:c5.large topology.ebs.csi.aws.com/zone:ca-central-1a topology.kubernetes.io/region:ca-central-1 topology.kubernetes.io/zone:ca-central-1a] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-06217673764fdafeb"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kubelet Update v1 2023-01-09 22:36:54 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {protokube Update v1 2023-01-09 22:37:19 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kops.k8s.io/kops-controller-pki":{},"f:node-role.kubernetes.io/control-plane":{},"f:node.kubernetes.io/exclude-from-external-load-balancers":{}}}} } {kops-controller Update v1 2023-01-09 22:37:21 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/master":{}}}} } {kube-controller-manager Update v1 2023-01-09 22:37:35 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.0.0/24\"":{}},"f:taints":{}}} } {kubelet Update v1 2023-01-09 22:38:39 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.0.0/24,DoNotUseExternalID:,ProviderID:aws:///ca-central-1a/i-06217673764fdafeb,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/master,Value:,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[100.96.0.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3862904832 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3758047232 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-01-09 23:09:24 +0000 UTC,LastTransitionTime:2023-01-09 22:36:48 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-01-09 23:09:24 +0000 UTC,LastTransitionTime:2023-01-09 22:36:48 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-01-09 23:09:24 +0000 UTC,LastTransitionTime:2023-01-09 22:36:48 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-01-09 23:09:24 +0000 UTC,LastTransitionTime:2023-01-09 22:37:35 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.37.64,},NodeAddress{Type:ExternalIP,Address:3.96.216.165,},NodeAddress{Type:Hostname,Address:ip-172-20-37-64.ca-central-1.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-37-64.ca-central-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-3-96-216-165.ca-central-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec27a483f61319a98e5ba2ee2d59e6a3,SystemUUID:ec27a483-f613-19a9-8e5b-a2ee2d59e6a3,BootID:9b41e339-1aaa-4374-8f11-8d34e12cd62a,KernelVersion:5.15.0-1026-aws,OSImage:Ubuntu 22.04.1 LTS,ContainerRuntimeVersion:docker://20.10.17,KubeletVersion:v1.23.15,KubeProxyVersion:v1.23.15,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[registry.k8s.io/etcdadm/etcd-manager@sha256:66a453db625abb268f4b3bbefc5a34a171d81e6e8796cecca54cfd71775c77c4 registry.k8s.io/etcdadm/etcd-manager:v3.0.20221209],SizeBytes:662209237,},ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:456569528,},ContainerImage{Names:[k8s.gcr.io/kube-apiserver-amd64:v1.23.15 registry.k8s.io/kube-apiserver-amd64:v1.23.15],SizeBytes:135240444,},ContainerImage{Names:[k8s.gcr.io/kube-controller-manager-amd64:v1.23.15 registry.k8s.io/kube-controller-manager-amd64:v1.23.15],SizeBytes:125046097,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.23.15 registry.k8s.io/kube-proxy-amd64:v1.23.15],SizeBytes:112336036,},ContainerImage{Names:[quay.io/cilium/operator@sha256:a6d24a006a6b92967ac90786b49bc1ac26e5477cf028cd1186efcfc2466484db quay.io/cilium/operator:v1.12.5],SizeBytes:110016956,},ContainerImage{Names:[registry.k8s.io/kops/kops-controller:1.26.0-beta.1],SizeBytes:104212800,},ContainerImage{Names:[registry.k8s.io/kops/dns-controller:1.26.0-beta.1],SizeBytes:102566451,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:10e231cf07c82dfab5141c1afe548dc734e5fc1f67665ec3982a325d1bd31a9a registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.0],SizeBytes:94471559,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:122bfb8c1edabb3c0edd63f06523e6940d958d19b3957dc7b1d6f81e9f1f6119 registry.k8s.io/sig-storage/csi-provisioner:v3.1.0],SizeBytes:57736070,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:9ebbf9f023e7b41ccee3d52afe39a89e3ddacdbb69269d583abfc25847cfd9e4 registry.k8s.io/sig-storage/csi-resizer:v1.4.0],SizeBytes:55464778,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:8b9c313c05f54fb04f8d430896f5f5904b6cb157df261501b29adc04d2b2dc7b registry.k8s.io/sig-storage/csi-attacher:v3.4.0],SizeBytes:54848595,},ContainerImage{Names:[k8s.gcr.io/kube-scheduler-amd64:v1.23.15 registry.k8s.io/kube-scheduler-amd64:v1.23.15],SizeBytes:53521430,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:19582112,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:17611980,},ContainerImage{Names:[registry.k8s.io/kops/kube-apiserver-healthcheck:1.26.0-beta.1],SizeBytes:8787111,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:682696,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Jan 9 23:11:53.749: INFO: Logging kubelet events for node ip-172-20-37-64.ca-central-1.compute.internal Jan 9 23:11:53.784: INFO: Logging pods the kubelet thinks is on node ip-172-20-37-64.ca-central-1.compute.internal Jan 9 23:11:53.821: INFO: kube-apiserver-ip-172-20-37-64.ca-central-1.compute.internal started at 2023-01-09 22:36:15 +0000 UTC (0+2 container statuses recorded) Jan 9 23:11:53.821: INFO: Container healthcheck ready: true, restart count 0 Jan 9 23:11:53.821: INFO: Container kube-apiserver ready: true, restart count 1 Jan 9 23:11:53.821: INFO: kube-controller-manager-ip-172-20-37-64.ca-central-1.compute.internal started at 2023-01-09 22:36:15 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:53.822: INFO: Container kube-controller-manager ready: true, restart count 2 Jan 9 23:11:53.822: INFO: kops-controller-cqrv8 started at 2023-01-09 22:37:19 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:53.822: INFO: Container kops-controller ready: true, restart count 0 Jan 9 23:11:53.822: INFO: cilium-operator-c7bfc9f44-w4nqz started at 2023-01-09 22:37:19 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:53.822: INFO: Container cilium-operator ready: true, restart count 0 Jan 9 23:11:53.822: INFO: ebs-csi-controller-5d8d7cc949-8wxx2 started at 2023-01-09 22:37:19 +0000 UTC (0+5 container statuses recorded) Jan 9 23:11:53.822: INFO: Container csi-attacher ready: true, restart count 0 Jan 9 23:11:53.822: INFO: Container csi-provisioner ready: true, restart count 0 Jan 9 23:11:53.822: INFO: Container csi-resizer ready: true, restart count 0 Jan 9 23:11:53.822: INFO: Container ebs-plugin ready: true, restart count 0 Jan 9 23:11:53.822: INFO: Container liveness-probe ready: true, restart count 0 Jan 9 23:11:53.822: INFO: dns-controller-867784b75c-sxxcl started at 2023-01-09 22:37:20 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:53.822: INFO: Container dns-controller ready: true, restart count 0 Jan 9 23:11:53.822: INFO: kube-scheduler-ip-172-20-37-64.ca-central-1.compute.internal started at 2023-01-09 22:36:15 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:53.822: INFO: Container kube-scheduler ready: true, restart count 0 Jan 9 23:11:53.822: INFO: etcd-manager-events-ip-172-20-37-64.ca-central-1.compute.internal started at 2023-01-09 22:36:15 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:53.822: INFO: Container etcd-manager ready: true, restart count 0 Jan 9 23:11:53.822: INFO: etcd-manager-main-ip-172-20-37-64.ca-central-1.compute.internal started at 2023-01-09 22:36:15 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:53.822: INFO: Container etcd-manager ready: true, restart count 0 Jan 9 23:11:53.822: INFO: ebs-csi-node-8q4pg started at 2023-01-09 22:37:17 +0000 UTC (0+3 container statuses recorded) Jan 9 23:11:53.822: INFO: Container ebs-plugin ready: true, restart count 0 Jan 9 23:11:53.822: INFO: Container liveness-probe ready: true, restart count 0 Jan 9 23:11:53.822: INFO: Container node-driver-registrar ready: true, restart count 0 Jan 9 23:11:53.822: INFO: cilium-q7lrg started at 2023-01-09 22:37:17 +0000 UTC (1+1 container statuses recorded) Jan 9 23:11:53.822: INFO: Init container clean-cilium-state ready: true, restart count 0 Jan 9 23:11:53.822: INFO: Container cilium-agent ready: true, restart count 0 Jan 9 23:11:53.944: INFO: Latency metrics for node ip-172-20-37-64.ca-central-1.compute.internal Jan 9 23:11:53.945: INFO: Logging node info for node ip-172-20-55-216.ca-central-1.compute.internal Jan 9 23:11:53.977: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-55-216.ca-central-1.compute.internal 2ceff781-505a-4706-ad8b-e38e37f72571 47493 0 2023-01-09 22:38:47 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ca-central-1 failure-domain.beta.kubernetes.io/zone:ca-central-1a kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-55-216.ca-central-1.compute.internal kubernetes.io/os:linux kubernetes.io/role:node node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:ca-central-1a topology.hostpath.csi/node:ip-172-20-55-216.ca-central-1.compute.internal topology.kubernetes.io/region:ca-central-1 topology.kubernetes.io/zone:ca-central-1a] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-0c78741e44993ae7f"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kops-controller Update v1 2023-01-09 22:38:47 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}} } {kube-controller-manager Update v1 2023-01-09 22:38:47 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.4.0/24\"":{}}}} } {kubelet Update v1 2023-01-09 22:38:47 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {kube-controller-manager Update v1 2023-01-09 22:56:45 +0000 UTC FieldsV1 {"f:status":{"f:volumesAttached":{}}} status} {kubelet Update v1 2023-01-09 22:56:51 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:capacity":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.4.0/24,DoNotUseExternalID:,ProviderID:aws:///ca-central-1a/i-0c78741e44993ae7f,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.4.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051644416 0} {<nil>} 3956684Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946786816 0} {<nil>} 3854284Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-01-09 23:11:41 +0000 UTC,LastTransitionTime:2023-01-09 22:38:47 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-01-09 23:11:41 +0000 UTC,LastTransitionTime:2023-01-09 22:38:47 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-01-09 23:11:41 +0000 UTC,LastTransitionTime:2023-01-09 22:38:47 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-01-09 23:11:41 +0000 UTC,LastTransitionTime:2023-01-09 22:39:07 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.55.216,},NodeAddress{Type:ExternalIP,Address:15.222.60.56,},NodeAddress{Type:Hostname,Address:ip-172-20-55-216.ca-central-1.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-55-216.ca-central-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-15-222-60-56.ca-central-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec249e6a3570a28ebd4276206d3ae523,SystemUUID:ec249e6a-3570-a28e-bd42-76206d3ae523,BootID:d58dcef2-849d-4807-9ba5-0438ebaa05b9,KernelVersion:5.15.0-1026-aws,OSImage:Ubuntu 22.04.1 LTS,ContainerRuntimeVersion:docker://20.10.17,KubeletVersion:v1.23.15,KubeProxyVersion:v1.23.15,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:456569528,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/jessie-dnsutils@sha256:11e6a66017ba4e4b938c1612b7a54a3befcefd354796c04e1dba76873a13518e k8s.gcr.io/e2e-test-images/jessie-dnsutils:1.5],SizeBytes:253371792,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:20f25f275d46aa728f7615a1ccc19c78b2ed89435bf943a44b339f70f45508e6 k8s.gcr.io/e2e-test-images/httpd:2.4.39-2],SizeBytes:126894770,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:7e8bdd271312fd25fc5ff5a8f04727be84044eb3d7d8d03611972a6752e2e11e k8s.gcr.io/e2e-test-images/agnhost:2.39],SizeBytes:126872991,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nautilus@sha256:99c0d6f1ad24a1aa1905d9c6534d193f268f7b23f9add2ae6bb41f31094bdd5c k8s.gcr.io/e2e-test-images/nautilus:1.5],SizeBytes:124758741,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:1b9d1b2f36cb2dbee1960e82a9344aeb11bd4c4c03abf5e1853e0559c23855e3 k8s.gcr.io/e2e-test-images/httpd:2.4.38-2],SizeBytes:123781643,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.23.15 registry.k8s.io/kube-proxy-amd64:v1.23.15],SizeBytes:112336036,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:10e231cf07c82dfab5141c1afe548dc734e5fc1f67665ec3982a325d1bd31a9a registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.0],SizeBytes:94471559,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:c8e03f60afa90a28e4bb6ec9a8d0fc36d89de4b7475cf2d613afa793ec969fe0 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.0],SizeBytes:56382329,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:4e74c0492bceddc598de1c90cc5bc14dcda94cb49fa9c5bad9d117c4834b5e08 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.1],SizeBytes:56379269,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:36c31f7e1f433c9634d24f876353e8646246d81a03c4e351202c2644daff1620 k8s.gcr.io/sig-storage/csi-resizer:v1.2.0],SizeBytes:53956308,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:ed98431376c9e944e19a465fe8ea944806714dd95416a0821096c78d66b579bd k8s.gcr.io/sig-storage/csi-snapshotter:v4.1.1],SizeBytes:53531990,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:c5be65d6679efabb969d9b019300d187437ae876f992c40911fd2892bbef3b36 k8s.gcr.io/sig-storage/csi-attacher:v3.2.0],SizeBytes:53500826,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:60ab9b3e6a030d3038c87c0d6bca2930f58d1d72823e6a4af09767dc83b696a2 k8s.gcr.io/sig-storage/csi-attacher:v3.2.1],SizeBytes:53499570,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:7a5ba58a44e0d749e0767e4e37315bcf6a61f33ce3185c1991848af4db0fb70a k8s.gcr.io/sig-storage/csi-resizer:v1.1.0],SizeBytes:49176472,},ContainerImage{Names:[k8s.gcr.io/sig-storage/hostpathplugin@sha256:232fe80174d60d520d36043103853a1d7ab4b7f3782cf43e45034f04ccda58ce k8s.gcr.io/sig-storage/hostpathplugin:v1.7.1],SizeBytes:29977921,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:a61d309da54641db41fb8f35718f744e9f730d4d0384f8c4b186ddc9f06cbd5f k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.1.0],SizeBytes:19662887,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:19582112,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:2dee3fe5fe861bb66c3a4ac51114f3447a4cd35870e0f2e2b558c7a400d89589 k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.2.0],SizeBytes:18699436,},ContainerImage{Names:[k8s.gcr.io/sig-storage/mock-driver@sha256:a7b517f9e0f42ffade802eef9cefa271372386b85f55b702b493241e58459793 k8s.gcr.io/sig-storage/mock-driver:v4.1.0],SizeBytes:17680993,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:17611980,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:529be2c9770add0cdd0c989115222ea9fc1be430c11095eb9f6dafcf98a36e2b k8s.gcr.io/sig-storage/livenessprobe:v2.4.0],SizeBytes:17175488,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:13616070e3f29de4417eee434a8ef472221c9e51b3d037b5a6b46cef08eb7443 k8s.gcr.io/e2e-test-images/nginx:1.14-2],SizeBytes:16032814,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf k8s.gcr.io/e2e-test-images/busybox:1.29-2],SizeBytes:1154361,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db k8s.gcr.io/pause:3.6 registry.k8s.io/pause:3.6],SizeBytes:682696,},},VolumesInUse:[kubernetes.io/csi/ebs.csi.aws.com^vol-08c82b2633ea0246d kubernetes.io/csi/ebs.csi.aws.com^vol-0a0867bd913d60721],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-0a0867bd913d60721,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-08c82b2633ea0246d,DevicePath:,},},Config:nil,},} Jan 9 23:11:53.978: INFO: Logging kubelet events for node ip-172-20-55-216.ca-central-1.compute.internal Jan 9 23:11:54.013: INFO: Logging pods the kubelet thinks is on node ip-172-20-55-216.ca-central-1.compute.internal Jan 9 23:11:54.068: INFO: pod-8ade382c-2fb2-4f02-8d50-57f7b6a15696 started at 2023-01-09 23:06:54 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:54.068: INFO: Container test-container ready: false, restart count 0 Jan 9 23:11:54.068: INFO: csi-mockplugin-0 started at 2023-01-09 23:10:49 +0000 UTC (0+3 container statuses recorded) Jan 9 23:11:54.068: INFO: Container csi-provisioner ready: false, restart count 0 Jan 9 23:11:54.068: INFO: Container driver-registrar ready: false, restart count 0 Jan 9 23:11:54.068: INFO: Container mock ready: false, restart count 0 Jan 9 23:11:54.068: INFO: startup-ff2f8ca6-ae0a-4fae-9483-a7d80b37fdc1 started at 2023-01-09 23:07:24 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:54.068: INFO: Container busybox ready: true, restart count 0 Jan 9 23:11:54.068: INFO: pod-90864581-af4c-4524-a3f6-27ae0e060f71 started at 2023-01-09 23:10:24 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:54.068: INFO: Container write-pod ready: false, restart count 0 Jan 9 23:11:54.068: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-mcd88 started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:54.068: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:11:54.068: INFO: fail-once-non-local-nx7cr started at 2023-01-09 22:56:53 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:54.068: INFO: Container c ready: false, restart count 0 Jan 9 23:11:54.068: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-6nqvb started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:54.068: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:11:54.068: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-8kb86 started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:54.068: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:11:54.068: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-vwd6k started at 2023-01-09 22:54:43 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:54.068: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:11:54.068: INFO: image-pull-test99b7b050-d558-462d-a744-2f374728e347 started at 2023-01-09 23:07:56 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:54.068: INFO: Container image-pull-test ready: false, restart count 0 Jan 9 23:11:54.068: INFO: hostpath-symlink-prep-provisioning-3662 started at 2023-01-09 23:10:57 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:54.068: INFO: Container init-volume-provisioning-3662 ready: false, restart count 0 Jan 9 23:11:54.068: INFO: ebs-csi-node-psq97 started at 2023-01-09 22:38:48 +0000 UTC (0+3 container statuses recorded) Jan 9 23:11:54.068: INFO: Container ebs-plugin ready: true, restart count 0 Jan 9 23:11:54.068: INFO: Container liveness-probe ready: true, restart count 0 Jan 9 23:11:54.068: INFO: Container node-driver-registrar ready: true, restart count 0 Jan 9 23:11:54.068: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-qj2bw started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:54.068: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:11:54.068: INFO: pod-secrets-44fccaea-993f-49ad-91c1-2dc3743d909d started at 2023-01-09 23:10:24 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:54.068: INFO: Container secret-volume-test ready: false, restart count 0 Jan 9 23:11:54.068: INFO: pod1 started at 2023-01-09 23:11:07 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:54.068: INFO: Container agnhost-container ready: false, restart count 0 Jan 9 23:11:54.068: INFO: fail-once-local-gvnwq started at 2023-01-09 23:10:38 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:54.068: INFO: Container c ready: false, restart count 1 Jan 9 23:11:54.068: INFO: csi-mockplugin-attacher-0 started at 2023-01-09 23:10:49 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:54.068: INFO: Container csi-attacher ready: false, restart count 0 Jan 9 23:11:54.068: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-6ks7b started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:54.068: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:11:54.068: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-72kwd started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:54.068: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:11:54.068: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-krwzl started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:54.068: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:11:54.068: INFO: pod-53d27e26-4bc7-4dfb-aa03-5bd21df067c1 started at 2023-01-09 23:11:20 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:54.068: INFO: Container write-pod ready: false, restart count 0 Jan 9 23:11:54.068: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-qglk4 started at 2023-01-09 22:54:43 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:54.069: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:11:54.069: INFO: netserver-1 started at 2023-01-09 23:08:45 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:54.069: INFO: Container webserver ready: true, restart count 0 Jan 9 23:11:54.069: INFO: cilium-dqpw4 started at 2023-01-09 22:38:48 +0000 UTC (1+1 container statuses recorded) Jan 9 23:11:54.069: INFO: Init container clean-cilium-state ready: true, restart count 0 Jan 9 23:11:54.069: INFO: Container cilium-agent ready: true, restart count 0 Jan 9 23:11:54.069: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-b7bxr started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:54.069: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:11:54.069: INFO: test-rolling-update-controller-782nh started at 2023-01-09 23:11:33 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:54.069: INFO: Container httpd ready: false, restart count 0 Jan 9 23:11:54.069: INFO: hostexec-ip-172-20-55-216.ca-central-1.compute.internal-ddnj4 started at 2023-01-09 23:10:19 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:54.069: INFO: Container agnhost-container ready: true, restart count 0 Jan 9 23:11:54.069: INFO: inline-volume-tester2-9qph2 started at 2023-01-09 23:10:30 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:54.069: INFO: Container csi-volume-tester ready: false, restart count 0 Jan 9 23:11:54.069: INFO: netserver-1 started at 2023-01-09 23:07:18 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:54.069: INFO: Container webserver ready: false, restart count 0 Jan 9 23:11:54.069: INFO: fail-once-non-local-mh6qc started at 2023-01-09 22:56:53 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:54.069: INFO: Container c ready: false, restart count 0 Jan 9 23:11:54.320: INFO: Latency metrics for node ip-172-20-55-216.ca-central-1.compute.internal Jan 9 23:11:54.320: INFO: Logging node info for node ip-172-20-57-251.ca-central-1.compute.internal Jan 9 23:11:54.352: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-57-251.ca-central-1.compute.internal eaaaf6eb-cbd5-4763-9c39-fb0e1a4ce44a 47495 0 2023-01-09 22:38:45 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ca-central-1 failure-domain.beta.kubernetes.io/zone:ca-central-1a kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-57-251.ca-central-1.compute.internal kubernetes.io/os:linux kubernetes.io/role:node node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:ca-central-1a topology.hostpath.csi/node:ip-172-20-57-251.ca-central-1.compute.internal topology.kubernetes.io/region:ca-central-1 topology.kubernetes.io/zone:ca-central-1a] map[csi.volume.kubernetes.io/nodeid:{"csi-mock-csi-mock-volumes-9402":"csi-mock-csi-mock-volumes-9402","ebs.csi.aws.com":"i-0513e7d6e5eba8eec"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kops-controller Update v1 2023-01-09 22:38:45 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}} } {kube-controller-manager Update v1 2023-01-09 22:38:45 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.2.0/24\"":{}}}} } {kubelet Update v1 2023-01-09 22:38:45 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {kubelet Update v1 2023-01-09 23:00:27 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.2.0/24,DoNotUseExternalID:,ProviderID:aws:///ca-central-1a/i-0513e7d6e5eba8eec,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.2.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051652608 0} {<nil>} 3956692Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946795008 0} {<nil>} 3854292Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-01-09 23:11:24 +0000 UTC,LastTransitionTime:2023-01-09 22:38:45 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-01-09 23:11:24 +0000 UTC,LastTransitionTime:2023-01-09 22:38:45 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-01-09 23:11:24 +0000 UTC,LastTransitionTime:2023-01-09 22:38:45 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-01-09 23:11:24 +0000 UTC,LastTransitionTime:2023-01-09 22:39:05 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.57.251,},NodeAddress{Type:ExternalIP,Address:3.96.144.209,},NodeAddress{Type:Hostname,Address:ip-172-20-57-251.ca-central-1.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-57-251.ca-central-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-3-96-144-209.ca-central-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2f0d7ba95b37a96a1d9360f6071d84,SystemUUID:ec2f0d7b-a95b-37a9-6a1d-9360f6071d84,BootID:67865282-1911-4e38-8675-a7b78d5ae45e,KernelVersion:5.15.0-1026-aws,OSImage:Ubuntu 22.04.1 LTS,ContainerRuntimeVersion:docker://20.10.17,KubeletVersion:v1.23.15,KubeProxyVersion:v1.23.15,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:456569528,},ContainerImage{Names:[k8s.gcr.io/etcd@sha256:dd75ec974b0a2a6f6bb47001ba09207976e625db898d1b16735528c009cb171c k8s.gcr.io/etcd:3.5.6-0],SizeBytes:299475478,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/jessie-dnsutils@sha256:11e6a66017ba4e4b938c1612b7a54a3befcefd354796c04e1dba76873a13518e k8s.gcr.io/e2e-test-images/jessie-dnsutils:1.5],SizeBytes:253371792,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:20f25f275d46aa728f7615a1ccc19c78b2ed89435bf943a44b339f70f45508e6 k8s.gcr.io/e2e-test-images/httpd:2.4.39-2],SizeBytes:126894770,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:7e8bdd271312fd25fc5ff5a8f04727be84044eb3d7d8d03611972a6752e2e11e k8s.gcr.io/e2e-test-images/agnhost:2.39],SizeBytes:126872991,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nautilus@sha256:99c0d6f1ad24a1aa1905d9c6534d193f268f7b23f9add2ae6bb41f31094bdd5c k8s.gcr.io/e2e-test-images/nautilus:1.5],SizeBytes:124758741,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:1b9d1b2f36cb2dbee1960e82a9344aeb11bd4c4c03abf5e1853e0559c23855e3 k8s.gcr.io/e2e-test-images/httpd:2.4.38-2],SizeBytes:123781643,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.23.15 registry.k8s.io/kube-proxy-amd64:v1.23.15],SizeBytes:112336036,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:10e231cf07c82dfab5141c1afe548dc734e5fc1f67665ec3982a325d1bd31a9a registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.0],SizeBytes:94471559,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/sample-apiserver@sha256:f9c93b92b6ff750b41a93c4e4fe0bfe384597aeb841e2539d5444815c55b2d8f k8s.gcr.io/e2e-test-images/sample-apiserver:1.17.5],SizeBytes:56504541,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:c8e03f60afa90a28e4bb6ec9a8d0fc36d89de4b7475cf2d613afa793ec969fe0 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.0],SizeBytes:56382329,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:4e74c0492bceddc598de1c90cc5bc14dcda94cb49fa9c5bad9d117c4834b5e08 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.1],SizeBytes:56379269,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:36c31f7e1f433c9634d24f876353e8646246d81a03c4e351202c2644daff1620 k8s.gcr.io/sig-storage/csi-resizer:v1.2.0],SizeBytes:53956308,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:ed98431376c9e944e19a465fe8ea944806714dd95416a0821096c78d66b579bd k8s.gcr.io/sig-storage/csi-snapshotter:v4.1.1],SizeBytes:53531990,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:c5be65d6679efabb969d9b019300d187437ae876f992c40911fd2892bbef3b36 k8s.gcr.io/sig-storage/csi-attacher:v3.2.0],SizeBytes:53500826,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:60ab9b3e6a030d3038c87c0d6bca2930f58d1d72823e6a4af09767dc83b696a2 k8s.gcr.io/sig-storage/csi-attacher:v3.2.1],SizeBytes:53499570,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:7a5ba58a44e0d749e0767e4e37315bcf6a61f33ce3185c1991848af4db0fb70a k8s.gcr.io/sig-storage/csi-resizer:v1.1.0],SizeBytes:49176472,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/regression-issue-74839@sha256:b4f1d8d61bdad84bd50442d161d5460e4019d53e989b64220fdbc62fc87d76bf k8s.gcr.io/e2e-test-images/regression-issue-74839:1.2],SizeBytes:44576952,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nonroot@sha256:b9e2958a3dd879e3cf11142228c6d073d0fc4ea2e857c3be6f4fb0ab5fb2c937 k8s.gcr.io/e2e-test-images/nonroot:1.2],SizeBytes:42321438,},ContainerImage{Names:[k8s.gcr.io/sig-storage/hostpathplugin@sha256:232fe80174d60d520d36043103853a1d7ab4b7f3782cf43e45034f04ccda58ce k8s.gcr.io/sig-storage/hostpathplugin:v1.7.1],SizeBytes:29977921,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:a61d309da54641db41fb8f35718f744e9f730d4d0384f8c4b186ddc9f06cbd5f k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.1.0],SizeBytes:19662887,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:19582112,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:2dee3fe5fe861bb66c3a4ac51114f3447a4cd35870e0f2e2b558c7a400d89589 k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.2.0],SizeBytes:18699436,},ContainerImage{Names:[k8s.gcr.io/sig-storage/mock-driver@sha256:a7b517f9e0f42ffade802eef9cefa271372386b85f55b702b493241e58459793 k8s.gcr.io/sig-storage/mock-driver:v4.1.0],SizeBytes:17680993,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:17611980,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:529be2c9770add0cdd0c989115222ea9fc1be430c11095eb9f6dafcf98a36e2b k8s.gcr.io/sig-storage/livenessprobe:v2.4.0],SizeBytes:17175488,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:13616070e3f29de4417eee434a8ef472221c9e51b3d037b5a6b46cef08eb7443 k8s.gcr.io/e2e-test-images/nginx:1.14-2],SizeBytes:16032814,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nonewprivs@sha256:8ac1264691820febacf3aea5d152cbde6d10685731ec14966a9401c6f47a68ac k8s.gcr.io/e2e-test-images/nonewprivs:1.3],SizeBytes:7107254,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf k8s.gcr.io/e2e-test-images/busybox:1.29-2],SizeBytes:1154361,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:39e1e963e5310e9c313bad51523be012ede7b35bb9316517d19089a010356592 k8s.gcr.io/e2e-test-images/busybox:1.29-1],SizeBytes:1154361,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db k8s.gcr.io/pause:3.6 registry.k8s.io/pause:3.6],SizeBytes:682696,},},VolumesInUse:[kubernetes.io/csi/csi-mock-csi-mock-volumes-9402^4],VolumesAttached:[]AttachedVolume{},Config:nil,},} Jan 9 23:11:54.353: INFO: Logging kubelet events for node ip-172-20-57-251.ca-central-1.compute.internal Jan 9 23:11:54.391: INFO: Logging pods the kubelet thinks is on node ip-172-20-57-251.ca-central-1.compute.internal Jan 9 23:11:54.434: INFO: hostpath-symlink-prep-volume-6746 started at 2023-01-09 23:07:44 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:54.434: INFO: Container init-volume-volume-6746 ready: false, restart count 0 Jan 9 23:11:54.434: INFO: pvc-volume-tester-2vlf5 started at 2023-01-09 23:06:05 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:54.434: INFO: Container volume-tester ready: false, restart count 0 Jan 9 23:11:54.434: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-wvflq started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:54.434: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:11:54.434: INFO: hostexec-ip-172-20-57-251.ca-central-1.compute.internal-f9vsk started at 2023-01-09 23:08:31 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:54.434: INFO: Container agnhost-container ready: true, restart count 0 Jan 9 23:11:54.434: INFO: pod-subpath-test-preprovisionedpv-ssct started at 2023-01-09 23:08:48 +0000 UTC (0+2 container statuses recorded) Jan 9 23:11:54.434: INFO: Container test-container-subpath-preprovisionedpv-ssct ready: false, restart count 0 Jan 9 23:11:54.434: INFO: Container test-container-volume-preprovisionedpv-ssct ready: false, restart count 0 Jan 9 23:11:54.435: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-tvxfq started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:54.435: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:11:54.435: INFO: csi-mockplugin-0 started at 2023-01-09 23:00:20 +0000 UTC (0+3 container statuses recorded) Jan 9 23:11:54.435: INFO: Container csi-provisioner ready: true, restart count 0 Jan 9 23:11:54.435: INFO: Container driver-registrar ready: true, restart count 0 Jan 9 23:11:54.435: INFO: Container mock ready: true, restart count 0 Jan 9 23:11:54.435: INFO: fail-once-local-szrdn started at 2023-01-09 23:10:38 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:54.435: INFO: Container c ready: false, restart count 0 Jan 9 23:11:54.435: INFO: fail-once-local-wj6xl started at 2023-01-09 23:10:57 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:54.435: INFO: Container c ready: false, restart count 0 Jan 9 23:11:54.435: INFO: test-pod started at 2023-01-09 23:11:31 +0000 UTC (0+3 container statuses recorded) Jan 9 23:11:54.435: INFO: Container busybox-1 ready: false, restart count 0 Jan 9 23:11:54.435: INFO: Container busybox-2 ready: false, restart count 0 Jan 9 23:11:54.435: INFO: Container busybox-3 ready: false, restart count 0 Jan 9 23:11:54.435: INFO: ebs-csi-node-g4ngc started at 2023-01-09 22:38:46 +0000 UTC (0+3 container statuses recorded) Jan 9 23:11:54.435: INFO: Container ebs-plugin ready: true, restart count 0 Jan 9 23:11:54.435: INFO: Container liveness-probe ready: true, restart count 0 Jan 9 23:11:54.435: INFO: Container node-driver-registrar ready: true, restart count 0 Jan 9 23:11:54.435: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-wmqwf started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:54.435: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:11:54.435: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-m8cfz started at 2023-01-09 22:54:43 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:54.435: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:11:54.435: INFO: pod-11edca44-2fb9-4737-9beb-f344cd567f1c started at 2023-01-09 23:11:21 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:54.435: INFO: Container test-container ready: false, restart count 0 Jan 9 23:11:54.435: INFO: csi-mockplugin-attacher-0 started at 2023-01-09 23:00:20 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:54.435: INFO: Container csi-attacher ready: false, restart count 0 Jan 9 23:11:54.435: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-rbqs2 started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:54.435: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:11:54.435: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-96np5 started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:54.435: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:11:54.435: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-f9w29 started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:54.435: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:11:54.435: INFO: netserver-2 started at 2023-01-09 23:07:18 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:54.435: INFO: Container webserver ready: true, restart count 0 Jan 9 23:11:54.435: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-zx8gm started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:54.435: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:11:54.435: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-qzf5t started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:54.435: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:11:54.435: INFO: netserver-2 started at 2023-01-09 23:08:45 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:54.435: INFO: Container webserver ready: false, restart count 0 Jan 9 23:11:54.435: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-zvfdw started at 2023-01-09 22:54:43 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:54.435: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:11:54.435: INFO: cilium-6p9jz started at 2023-01-09 22:38:46 +0000 UTC (1+1 container statuses recorded) Jan 9 23:11:54.435: INFO: Init container clean-cilium-state ready: true, restart count 0 Jan 9 23:11:54.435: INFO: Container cilium-agent ready: true, restart count 0 Jan 9 23:11:54.435: INFO: sample-webhook-deployment-6c69dbd86b-wkgxj started at 2023-01-09 23:11:47 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:54.435: INFO: Container sample-webhook ready: false, restart count 0 Jan 9 23:11:55.040: INFO: Latency metrics for node ip-172-20-57-251.ca-central-1.compute.internal Jan 9 23:11:55.040: INFO: Logging node info for node ip-172-20-59-208.ca-central-1.compute.internal Jan 9 23:11:55.077: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-59-208.ca-central-1.compute.internal d061a4d4-9ff6-43fc-89cc-422c61f7441c 47576 0 2023-01-09 22:38:43 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ca-central-1 failure-domain.beta.kubernetes.io/zone:ca-central-1a kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-59-208.ca-central-1.compute.internal kubernetes.io/os:linux kubernetes.io/role:node node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:ca-central-1a topology.hostpath.csi/node:ip-172-20-59-208.ca-central-1.compute.internal topology.kubernetes.io/region:ca-central-1 topology.kubernetes.io/zone:ca-central-1a] map[csi.volume.kubernetes.io/nodeid:{"csi-hostpath-ephemeral-7223":"ip-172-20-59-208.ca-central-1.compute.internal","ebs.csi.aws.com":"i-0f00ea8bafbacdd11"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kops-controller Update v1 2023-01-09 22:38:43 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}} } {kube-controller-manager Update v1 2023-01-09 22:38:43 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.1.0/24\"":{}}}} } {kubelet Update v1 2023-01-09 22:38:43 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {kube-controller-manager Update v1 2023-01-09 23:06:36 +0000 UTC FieldsV1 {"f:status":{"f:volumesAttached":{}}} status} {kubelet Update v1 2023-01-09 23:06:40 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:capacity":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.1.0/24,DoNotUseExternalID:,ProviderID:aws:///ca-central-1a/i-0f00ea8bafbacdd11,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.1.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051652608 0} {<nil>} 3956692Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946795008 0} {<nil>} 3854292Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-01-09 23:11:47 +0000 UTC,LastTransitionTime:2023-01-09 22:38:43 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-01-09 23:11:47 +0000 UTC,LastTransitionTime:2023-01-09 22:38:43 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-01-09 23:11:47 +0000 UTC,LastTransitionTime:2023-01-09 22:38:43 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-01-09 23:11:47 +0000 UTC,LastTransitionTime:2023-01-09 22:39:03 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.59.208,},NodeAddress{Type:ExternalIP,Address:99.79.51.95,},NodeAddress{Type:Hostname,Address:ip-172-20-59-208.ca-central-1.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-59-208.ca-central-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-99-79-51-95.ca-central-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec28d5ecfae27d4bcedd89eca1a365f0,SystemUUID:ec28d5ec-fae2-7d4b-cedd-89eca1a365f0,BootID:560320f7-12c5-44f2-a7f6-a7a40031e204,KernelVersion:5.15.0-1026-aws,OSImage:Ubuntu 22.04.1 LTS,ContainerRuntimeVersion:docker://20.10.17,KubeletVersion:v1.23.15,KubeProxyVersion:v1.23.15,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:456569528,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:20f25f275d46aa728f7615a1ccc19c78b2ed89435bf943a44b339f70f45508e6 k8s.gcr.io/e2e-test-images/httpd:2.4.39-2],SizeBytes:126894770,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:7e8bdd271312fd25fc5ff5a8f04727be84044eb3d7d8d03611972a6752e2e11e k8s.gcr.io/e2e-test-images/agnhost:2.39],SizeBytes:126872991,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:1b9d1b2f36cb2dbee1960e82a9344aeb11bd4c4c03abf5e1853e0559c23855e3 k8s.gcr.io/e2e-test-images/httpd:2.4.38-2],SizeBytes:123781643,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.23.15 registry.k8s.io/kube-proxy-amd64:v1.23.15],SizeBytes:112336036,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:10e231cf07c82dfab5141c1afe548dc734e5fc1f67665ec3982a325d1bd31a9a registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.0],SizeBytes:94471559,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:c8e03f60afa90a28e4bb6ec9a8d0fc36d89de4b7475cf2d613afa793ec969fe0 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.0],SizeBytes:56382329,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:4e74c0492bceddc598de1c90cc5bc14dcda94cb49fa9c5bad9d117c4834b5e08 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.1],SizeBytes:56379269,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:36c31f7e1f433c9634d24f876353e8646246d81a03c4e351202c2644daff1620 k8s.gcr.io/sig-storage/csi-resizer:v1.2.0],SizeBytes:53956308,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:ed98431376c9e944e19a465fe8ea944806714dd95416a0821096c78d66b579bd k8s.gcr.io/sig-storage/csi-snapshotter:v4.1.1],SizeBytes:53531990,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:c5be65d6679efabb969d9b019300d187437ae876f992c40911fd2892bbef3b36 k8s.gcr.io/sig-storage/csi-attacher:v3.2.0],SizeBytes:53500826,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:60ab9b3e6a030d3038c87c0d6bca2930f58d1d72823e6a4af09767dc83b696a2 k8s.gcr.io/sig-storage/csi-attacher:v3.2.1],SizeBytes:53499570,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:017727efcfeb7d053af68e51436ce8e65edbc6ca573720afb4f79c8594036955 registry.k8s.io/coredns/coredns:v1.10.0],SizeBytes:50249337,},ContainerImage{Names:[registry.k8s.io/cpa/cluster-proportional-autoscaler@sha256:68d396900aeaa072c1f27289485fdac29834045a6f3ffe369bf389d830ef572d registry.k8s.io/cpa/cluster-proportional-autoscaler:1.8.6],SizeBytes:47910609,},ContainerImage{Names:[k8s.gcr.io/sig-storage/hostpathplugin@sha256:232fe80174d60d520d36043103853a1d7ab4b7f3782cf43e45034f04ccda58ce k8s.gcr.io/sig-storage/hostpathplugin:v1.7.1],SizeBytes:29977921,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:a61d309da54641db41fb8f35718f744e9f730d4d0384f8c4b186ddc9f06cbd5f k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.1.0],SizeBytes:19662887,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:19582112,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:2dee3fe5fe861bb66c3a4ac51114f3447a4cd35870e0f2e2b558c7a400d89589 k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.2.0],SizeBytes:18699436,},ContainerImage{Names:[k8s.gcr.io/sig-storage/mock-driver@sha256:a7b517f9e0f42ffade802eef9cefa271372386b85f55b702b493241e58459793 k8s.gcr.io/sig-storage/mock-driver:v4.1.0],SizeBytes:17680993,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:17611980,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:529be2c9770add0cdd0c989115222ea9fc1be430c11095eb9f6dafcf98a36e2b k8s.gcr.io/sig-storage/livenessprobe:v2.4.0],SizeBytes:17175488,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:13616070e3f29de4417eee434a8ef472221c9e51b3d037b5a6b46cef08eb7443 k8s.gcr.io/e2e-test-images/nginx:1.14-2],SizeBytes:16032814,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nonewprivs@sha256:8ac1264691820febacf3aea5d152cbde6d10685731ec14966a9401c6f47a68ac k8s.gcr.io/e2e-test-images/nonewprivs:1.3],SizeBytes:7107254,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf k8s.gcr.io/e2e-test-images/busybox:1.29-2],SizeBytes:1154361,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:39e1e963e5310e9c313bad51523be012ede7b35bb9316517d19089a010356592 k8s.gcr.io/e2e-test-images/busybox:1.29-1],SizeBytes:1154361,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db k8s.gcr.io/pause:3.6 registry.k8s.io/pause:3.6],SizeBytes:682696,},},VolumesInUse:[kubernetes.io/csi/csi-hostpath-ephemeral-7223^43b8adc9-9072-11ed-b4b6-0a454e7330ed kubernetes.io/csi/csi-hostpath-ephemeral-7223^43b8d4d5-9072-11ed-b4b6-0a454e7330ed],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/csi-hostpath-ephemeral-7223^43b8d4d5-9072-11ed-b4b6-0a454e7330ed,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/csi-hostpath-ephemeral-7223^43b8adc9-9072-11ed-b4b6-0a454e7330ed,DevicePath:,},},Config:nil,},} Jan 9 23:11:55.078: INFO: Logging kubelet events for node ip-172-20-59-208.ca-central-1.compute.internal Jan 9 23:11:55.124: INFO: Logging pods the kubelet thinks is on node ip-172-20-59-208.ca-central-1.compute.internal Jan 9 23:11:55.169: INFO: coredns-867df8f45c-tmrqb started at 2023-01-09 22:39:03 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:55.169: INFO: Container coredns ready: true, restart count 0 Jan 9 23:11:55.169: INFO: netserver-3 started at 2023-01-09 23:08:45 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:55.169: INFO: Container webserver ready: false, restart count 0 Jan 9 23:11:55.169: INFO: coredns-autoscaler-557ccb4c66-jnjcc started at 2023-01-09 22:39:03 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:55.169: INFO: Container autoscaler ready: true, restart count 0 Jan 9 23:11:55.169: INFO: netserver-3 started at 2023-01-09 23:07:18 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:55.169: INFO: Container webserver ready: false, restart count 0 Jan 9 23:11:55.169: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-fc6pg started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:55.169: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:11:55.169: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-8qp6l started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:55.169: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: false, restart count 0 Jan 9 23:11:55.169: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-rzgjf started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:55.169: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:11:55.169: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-ptfgl started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:55.169: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:11:55.169: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-t5g85 started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:55.169: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:11:55.169: INFO: ebs-csi-node-prrrf started at 2023-01-09 22:38:44 +0000 UTC (0+3 container statuses recorded) Jan 9 23:11:55.169: INFO: Container ebs-plugin ready: true, restart count 0 Jan 9 23:11:55.169: INFO: Container liveness-probe ready: true, restart count 0 Jan 9 23:11:55.169: INFO: Container node-driver-registrar ready: true, restart count 0 Jan 9 23:11:55.169: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-x682f started at 2023-01-09 22:54:43 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:55.169: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:11:55.169: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-tx2mm started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:55.169: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: false, restart count 0 Jan 9 23:11:55.169: INFO: cilium-z2ltf started at 2023-01-09 22:38:44 +0000 UTC (1+1 container statuses recorded) Jan 9 23:11:55.169: INFO: Init container clean-cilium-state ready: true, restart count 0 Jan 9 23:11:55.169: INFO: Container cilium-agent ready: true, restart count 0 Jan 9 23:11:55.169: INFO: csi-hostpathplugin-0 started at 2023-01-09 23:03:05 +0000 UTC (0+7 container statuses recorded) Jan 9 23:11:55.169: INFO: Container csi-attacher ready: true, restart count 0 Jan 9 23:11:55.169: INFO: Container csi-provisioner ready: true, restart count 0 Jan 9 23:11:55.169: INFO: Container csi-resizer ready: true, restart count 0 Jan 9 23:11:55.169: INFO: Container csi-snapshotter ready: true, restart count 0 Jan 9 23:11:55.169: INFO: Container hostpath ready: true, restart count 0 Jan 9 23:11:55.169: INFO: Container liveness-probe ready: true, restart count 0 Jan 9 23:11:55.169: INFO: Container node-driver-registrar ready: true, restart count 0 Jan 9 23:11:55.169: INFO: inline-volume-tester-xnqfb started at 2023-01-09 23:06:35 +0000 UTC (0+1 container statuses recorded) Jan 9 23:11:55.169: INFO: Container csi-volume-tester ready: false, restart count 0 Jan 9 23:11:55.350: INFO: Latency metrics for node ip-172-20-59-208.ca-central-1.compute.internal Jan 9 23:11:55.350: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready �[1mSTEP�[0m: Destroying namespace "job-4121" for this suite.
Filter through log files
go run hack/e2e.go -v --test --test_args='--ginkgo.focus=Kubernetes\se2e\ssuite\s\[sig\-apps\]\sReplicaSet\sshould\sadopt\smatching\spods\son\screation\sand\srelease\sno\slonger\smatching\spods\s\[Conformance\]$'
/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:633 Jan 9 23:10:21.737: Unexpected error: <*errors.errorString | 0xc0002482c0>: { s: "timed out waiting for the condition", } timed out waiting for the condition occurred /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/pods.go:107from junit_13.xml
[BeforeEach] [sig-apps] ReplicaSet /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:185 �[1mSTEP�[0m: Creating a kubernetes client Jan 9 23:05:21.382: INFO: >>> kubeConfig: /root/.kube/config �[1mSTEP�[0m: Building a namespace api object, basename replicaset �[1mSTEP�[0m: Waiting for a default service account to be provisioned in namespace �[1mSTEP�[0m: Waiting for kube-root-ca.crt to be provisioned in namespace [It] should adopt matching pods on creation and release no longer matching pods [Conformance] /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:633 �[1mSTEP�[0m: Given a Pod with a 'name' label pod-adoption-release is created Jan 9 23:05:21.671: INFO: The status of Pod pod-adoption-release is Pending, waiting for it to be Running (with Ready = true) Jan 9 23:05:23.703: INFO: The status of Pod pod-adoption-release is Pending, waiting for it to be Running (with Ready = true) Jan 9 23:05:25.705: INFO: The status of Pod pod-adoption-release is Pending, waiting for it to be Running (with Ready = true) Jan 9 23:05:27.703: INFO: The status of Pod pod-adoption-release is Pending, waiting for it to be Running (with Ready = true) Jan 9 23:05:29.703: INFO: The status of Pod pod-adoption-release is Pending, waiting for it to be Running (with Ready = true) Jan 9 23:05:31.704: INFO: The status of Pod pod-adoption-release is Pending, waiting for it to be Running (with Ready = true) Jan 9 23:05:33.704: INFO: The status of Pod pod-adoption-release is Pending, waiting for it to be Running (with Ready = true) Jan 9 23:05:35.703: INFO: The status of Pod pod-adoption-release is Pending, waiting for it to be Running (with Ready = true) Jan 9 23:05:37.703: INFO: The status of Pod pod-adoption-release is Pending, waiting for it to be Running (with Ready = true) Jan 9 23:05:39.704: INFO: The status of Pod pod-adoption-release is Pending, waiting for it to be Running (with Ready = true) Jan 9 23:05:41.704: INFO: The status of Pod pod-adoption-release is Pending, waiting for it to be Running (with Ready = true) Jan 9 23:05:43.709: INFO: The status of Pod pod-adoption-release is Pending, waiting for it to be Running (with Ready = true) Jan 9 23:05:45.705: INFO: The status of Pod pod-adoption-release is Pending, waiting for it to be Running (with Ready = true) Jan 9 23:05:47.704: INFO: The status of Pod pod-adoption-release is Pending, waiting for it to be Running (with Ready = true) Jan 9 23:05:49.703: INFO: The status of Pod pod-adoption-release is Pending, waiting for it to be Running (with Ready = true) Jan 9 23:05:51.711: INFO: The status of Pod pod-adoption-release is Pending, waiting for it to be Running (with Ready = true) Jan 9 23:05:53.704: INFO: The status of Pod pod-adoption-release is Pending, waiting for it to be Running (with Ready = true) Jan 9 23:05:55.704: INFO: The status of Pod pod-adoption-release is Pending, waiting for it to be Running (with Ready = true) Jan 9 23:05:57.703: INFO: The status of Pod pod-adoption-release is Pending, waiting for it to be Running (with Ready = true) Jan 9 23:05:59.703: INFO: The status of Pod pod-adoption-release is Pending, waiting for it to be Running (with Ready = true) Jan 9 23:06:01.705: INFO: The status of Pod pod-adoption-release is Pending, waiting for it to be Running (with Ready = true) Jan 9 23:06:03.703: INFO: The status of Pod pod-adoption-release is Pending, waiting for it to be Running (with Ready = true) Jan 9 23:06:05.705: INFO: The status of Pod pod-adoption-release is Pending, waiting for it to be Running (with Ready = true) Jan 9 23:06:07.703: INFO: The status of Pod pod-adoption-release is Pending, waiting for it to be Running (with Ready = true) Jan 9 23:06:09.703: INFO: The status of Pod pod-adoption-release is Pending, waiting for it to be Running (with Ready = true) Jan 9 23:06:11.705: INFO: The status of Pod pod-adoption-release is Pending, waiting for it to be Running (with Ready = true) Jan 9 23:06:13.702: INFO: The status of Pod pod-adoption-release is Pending, waiting for it to be Running (with Ready = true) Jan 9 23:06:15.704: INFO: The status of Pod pod-adoption-release is Pending, waiting for it to be Running (with Ready = true) Jan 9 23:06:17.703: INFO: The status of Pod pod-adoption-release is Pending, waiting for it to be Running (with Ready = true) Jan 9 23:06:19.704: INFO: The status of Pod pod-adoption-release is Pending, waiting for it to be Running (with Ready = true) Jan 9 23:06:21.704: INFO: The status of Pod pod-adoption-release is Pending, waiting for it to be Running (with Ready = true) Jan 9 23:06:23.703: INFO: The status of Pod pod-adoption-release is Pending, waiting for it to be Running (with Ready = true) Jan 9 23:06:25.705: INFO: The status of Pod pod-adoption-release is Pending, waiting for it to be Running (with Ready = true) Jan 9 23:06:27.703: INFO: The status of Pod pod-adoption-release is Pending, waiting for it to be Running (with Ready = true) Jan 9 23:06:29.703: INFO: The status of Pod pod-adoption-release is Pending, waiting for it to be Running (with Ready = true) Jan 9 23:06:31.705: INFO: The status of Pod pod-adoption-release is Pending, waiting for it to be Running (with Ready = true) Jan 9 23:06:33.703: INFO: The status of Pod pod-adoption-release is Pending, waiting for it to be Running (with Ready = true) Jan 9 23:06:35.705: INFO: The status of Pod pod-adoption-release is Pending, waiting for it to be Running (with Ready = true) Jan 9 23:06:37.704: INFO: The status of Pod pod-adoption-release is Pending, waiting for it to be Running (with Ready = true) Jan 9 23:06:39.703: INFO: The status of Pod pod-adoption-release is Pending, waiting for it to be Running (with Ready = true) Jan 9 23:06:41.703: INFO: The status of Pod pod-adoption-release is Pending, waiting for it to be Running (with Ready = true) Jan 9 23:06:43.703: INFO: The status of Pod pod-adoption-release is Pending, waiting for it to be Running (with Ready = true) Jan 9 23:06:45.704: INFO: The status of Pod pod-adoption-release is Pending, waiting for it to be Running (with Ready = true) Jan 9 23:06:47.703: INFO: The status of Pod pod-adoption-release is Pending, waiting for it to be Running (with Ready = true) Jan 9 23:06:49.704: INFO: The status of Pod pod-adoption-release is Pending, waiting for it to be Running (with Ready = true) Jan 9 23:06:51.704: INFO: The status of Pod pod-adoption-release is Pending, waiting for it to be Running (with Ready = true) Jan 9 23:06:53.703: INFO: The status of Pod pod-adoption-release is Pending, waiting for it to be Running (with Ready = true) Jan 9 23:06:55.705: INFO: The status of Pod pod-adoption-release is Pending, waiting for it to be Running (with Ready = true) Jan 9 23:06:57.703: INFO: The status of Pod pod-adoption-release is Pending, waiting for it to be Running (with Ready = true) Jan 9 23:06:59.703: INFO: The status of Pod pod-adoption-release is Pending, waiting for it to be Running (with Ready = true) Jan 9 23:07:01.704: INFO: The status of Pod pod-adoption-release is Pending, waiting for it to be Running (with Ready = true) Jan 9 23:07:03.703: INFO: The status of Pod pod-adoption-release is Pending, waiting for it to be Running (with Ready = true) Jan 9 23:07:05.704: INFO: The status of Pod pod-adoption-release is Pending, waiting for it to be Running (with Ready = true) Jan 9 23:07:07.703: INFO: The status of Pod pod-adoption-release is Pending, waiting for it to be Running (with Ready = true) Jan 9 23:07:09.703: INFO: The status of Pod pod-adoption-release is Pending, waiting for it to be Running (with Ready = true) Jan 9 23:07:11.704: INFO: The status of Pod pod-adoption-release is Pending, waiting for it to be Running (with Ready = true) Jan 9 23:07:13.703: INFO: The status of Pod pod-adoption-release is Pending, waiting for it to be Running (with Ready = true) Jan 9 23:07:15.712: INFO: The status of Pod pod-adoption-release is Pending, waiting for it to be Running (with Ready = true) Jan 9 23:07:17.703: INFO: The status of Pod pod-adoption-release is Pending, waiting for it to be Running (with Ready = true) Jan 9 23:07:19.704: INFO: The status of Pod pod-adoption-release is Pending, waiting for it to be Running (with Ready = true) Jan 9 23:07:21.704: INFO: The status of Pod pod-adoption-release is Pending, waiting for it to be Running (with Ready = true) Jan 9 23:07:23.703: INFO: The status of Pod pod-adoption-release is Pending, waiting for it to be Running (with Ready = true) Jan 9 23:07:25.705: INFO: The status of Pod pod-adoption-release is Pending, waiting for it to be Running (with Ready = true) Jan 9 23:07:27.704: INFO: The status of Pod pod-adoption-release is Pending, waiting for it to be Running (with Ready = true) Jan 9 23:07:29.706: INFO: The status of Pod pod-adoption-release is Pending, waiting for it to be Running (with Ready = true) Jan 9 23:07:31.704: INFO: The status of Pod pod-adoption-release is Pending, waiting for it to be Running (with Ready = true) Jan 9 23:07:33.704: INFO: The status of Pod pod-adoption-release is Pending, waiting for it to be Running (with Ready = true) Jan 9 23:07:35.704: INFO: The status of Pod pod-adoption-release is Pending, waiting for it to be Running (with Ready = true) Jan 9 23:07:37.704: INFO: The status of Pod pod-adoption-release is Pending, waiting for it to be Running (with Ready = true) Jan 9 23:07:39.704: INFO: The status of Pod pod-adoption-release is Pending, waiting for it to be Running (with Ready = true) Jan 9 23:07:41.703: INFO: The status of Pod pod-adoption-release is Pending, waiting for it to be Running (with Ready = true) Jan 9 23:07:43.705: INFO: The status of Pod pod-adoption-release is Pending, waiting for it to be Running (with Ready = true) Jan 9 23:07:45.705: INFO: The status of Pod pod-adoption-release is Pending, waiting for it to be Running (with Ready = true) Jan 9 23:07:47.703: INFO: The status of Pod pod-adoption-release is Pending, waiting for it to be Running (with Ready = true) Jan 9 23:07:49.703: INFO: The status of Pod pod-adoption-release is Pending, waiting for it to be Running (with Ready = true) Jan 9 23:07:51.703: INFO: The status of Pod pod-adoption-release is Pending, waiting for it to be Running (with Ready = true) Jan 9 23:07:53.703: INFO: The status of Pod pod-adoption-release is Pending, waiting for it to be Running (with Ready = true) Jan 9 23:07:55.703: INFO: The status of Pod pod-adoption-release is Pending, waiting for it to be Running (with Ready = true) Jan 9 23:07:57.704: INFO: The status of Pod pod-adoption-release is Pending, waiting for it to be Running (with Ready = true) Jan 9 23:07:59.704: INFO: The status of Pod pod-adoption-release is Pending, waiting for it to be Running (with Ready = true) Jan 9 23:08:01.703: INFO: The status of Pod pod-adoption-release is Pending, waiting for it to be Running (with Ready = true) Jan 9 23:08:03.703: INFO: The status of Pod pod-adoption-release is Pending, waiting for it to be Running (with Ready = true) Jan 9 23:08:05.704: INFO: The status of Pod pod-adoption-release is Pending, waiting for it to be Running (with Ready = true) Jan 9 23:08:07.703: INFO: The status of Pod pod-adoption-release is Pending, waiting for it to be Running (with Ready = true) Jan 9 23:08:09.704: INFO: The status of Pod pod-adoption-release is Pending, waiting for it to be Running (with Ready = true) Jan 9 23:08:11.703: INFO: The status of Pod pod-adoption-release is Pending, waiting for it to be Running (with Ready = true) Jan 9 23:08:13.703: INFO: The status of Pod pod-adoption-release is Pending, waiting for it to be Running (with Ready = true) Jan 9 23:08:15.704: INFO: The status of Pod pod-adoption-release is Pending, waiting for it to be Running (with Ready = true) Jan 9 23:08:17.703: INFO: The status of Pod pod-adoption-release is Pending, waiting for it to be Running (with Ready = true) Jan 9 23:08:19.704: INFO: The status of Pod pod-adoption-release is Pending, waiting for it to be Running (with Ready = true) Jan 9 23:08:21.704: INFO: The status of Pod pod-adoption-release is Pending, waiting for it to be Running (with Ready = true) Jan 9 23:08:23.703: INFO: The status of Pod pod-adoption-release is Pending, waiting for it to be Running (with Ready = true) Jan 9 23:08:25.704: INFO: The status of Pod pod-adoption-release is Pending, waiting for it to be Running (with Ready = true) Jan 9 23:08:27.704: INFO: The status of Pod pod-adoption-release is Pending, waiting for it to be Running (with Ready = true) Jan 9 23:08:29.703: INFO: The status of Pod pod-adoption-release is Pending, waiting for it to be Running (with Ready = true) Jan 9 23:08:31.705: INFO: The status of Pod pod-adoption-release is Pending, waiting for it to be Running (with Ready = true) Jan 9 23:08:33.703: INFO: The status of Pod pod-adoption-release is Pending, waiting for it to be Running (with Ready = true) Jan 9 23:08:35.703: INFO: The status of Pod pod-adoption-release is Pending, waiting for it to be Running (with Ready = true) Jan 9 23:08:37.704: INFO: The status of Pod pod-adoption-release is Pending, waiting for it to be Running (with Ready = true) Jan 9 23:08:39.703: INFO: The status of Pod pod-adoption-release is Pending, waiting for it to be Running (with Ready = true) Jan 9 23:08:41.704: INFO: The status of Pod pod-adoption-release is Pending, waiting for it to be Running (with Ready = true) Jan 9 23:08:43.703: INFO: The status of Pod pod-adoption-release is Pending, waiting for it to be Running (with Ready = true) Jan 9 23:08:45.703: INFO: The status of Pod pod-adoption-release is Pending, waiting for it to be Running (with Ready = true) Jan 9 23:08:47.704: INFO: The status of Pod pod-adoption-release is Pending, waiting for it to be Running (with Ready = true) Jan 9 23:08:49.704: INFO: The status of Pod pod-adoption-release is Pending, waiting for it to be Running (with Ready = true) Jan 9 23:08:51.704: INFO: The status of Pod pod-adoption-release is Pending, waiting for it to be Running (with Ready = true) Jan 9 23:08:53.705: INFO: The status of Pod pod-adoption-release is Pending, waiting for it to be Running (with Ready = true) Jan 9 23:08:55.704: INFO: The status of Pod pod-adoption-release is Pending, waiting for it to be Running (with Ready = true) Jan 9 23:08:57.704: INFO: The status of Pod pod-adoption-release is Pending, waiting for it to be Running (with Ready = true) Jan 9 23:08:59.704: INFO: The status of Pod pod-adoption-release is Pending, waiting for it to be Running (with Ready = true) Jan 9 23:09:01.704: INFO: The status of Pod pod-adoption-release is Pending, waiting for it to be Running (with Ready = true) Jan 9 23:09:03.703: INFO: The status of Pod pod-adoption-release is Pending, waiting for it to be Running (with Ready = true) Jan 9 23:09:05.704: INFO: The status of Pod pod-adoption-release is Pending, waiting for it to be Running (with Ready = true) Jan 9 23:09:07.705: INFO: The status of Pod pod-adoption-release is Pending, waiting for it to be Running (with Ready = true) Jan 9 23:09:09.703: INFO: The status of Pod pod-adoption-release is Pending, waiting for it to be Running (with Ready = true) Jan 9 23:09:11.705: INFO: The status of Pod pod-adoption-release is Pending, waiting for it to be Running (with Ready = true) Jan 9 23:09:13.703: INFO: The status of Pod pod-adoption-release is Pending, waiting for it to be Running (with Ready = true) Jan 9 23:09:15.707: INFO: The status of Pod pod-adoption-release is Pending, waiting for it to be Running (with Ready = true) Jan 9 23:09:17.704: INFO: The status of Pod pod-adoption-release is Pending, waiting for it to be Running (with Ready = true) Jan 9 23:09:19.704: INFO: The status of Pod pod-adoption-release is Pending, waiting for it to be Running (with Ready = true) Jan 9 23:09:21.706: INFO: The status of Pod pod-adoption-release is Pending, waiting for it to be Running (with Ready = true) Jan 9 23:09:23.703: INFO: The status of Pod pod-adoption-release is Pending, waiting for it to be Running (with Ready = true) Jan 9 23:09:25.704: INFO: The status of Pod pod-adoption-release is Pending, waiting for it to be Running (with Ready = true) Jan 9 23:09:27.703: INFO: The status of Pod pod-adoption-release is Pending, waiting for it to be Running (with Ready = true) Jan 9 23:09:29.703: INFO: The status of Pod pod-adoption-release is Pending, waiting for it to be Running (with Ready = true) Jan 9 23:09:31.704: INFO: The status of Pod pod-adoption-release is Pending, waiting for it to be Running (with Ready = true) Jan 9 23:09:33.703: INFO: The status of Pod pod-adoption-release is Pending, waiting for it to be Running (with Ready = true) Jan 9 23:09:35.705: INFO: The status of Pod pod-adoption-release is Pending, waiting for it to be Running (with Ready = true) Jan 9 23:09:37.704: INFO: The status of Pod pod-adoption-release is Pending, waiting for it to be Running (with Ready = true) Jan 9 23:09:39.703: INFO: The status of Pod pod-adoption-release is Pending, waiting for it to be Running (with Ready = true) Jan 9 23:09:41.704: INFO: The status of Pod pod-adoption-release is Pending, waiting for it to be Running (with Ready = true) Jan 9 23:09:43.703: INFO: The status of Pod pod-adoption-release is Pending, waiting for it to be Running (with Ready = true) Jan 9 23:09:45.704: INFO: The status of Pod pod-adoption-release is Pending, waiting for it to be Running (with Ready = true) Jan 9 23:09:47.703: INFO: The status of Pod pod-adoption-release is Pending, waiting for it to be Running (with Ready = true) Jan 9 23:09:49.704: INFO: The status of Pod pod-adoption-release is Pending, waiting for it to be Running (with Ready = true) Jan 9 23:09:51.704: INFO: The status of Pod pod-adoption-release is Pending, waiting for it to be Running (with Ready = true) Jan 9 23:09:53.703: INFO: The status of Pod pod-adoption-release is Pending, waiting for it to be Running (with Ready = true) Jan 9 23:09:55.705: INFO: The status of Pod pod-adoption-release is Pending, waiting for it to be Running (with Ready = true) Jan 9 23:09:57.703: INFO: The status of Pod pod-adoption-release is Pending, waiting for it to be Running (with Ready = true) Jan 9 23:09:59.703: INFO: The status of Pod pod-adoption-release is Pending, waiting for it to be Running (with Ready = true) Jan 9 23:10:01.705: INFO: The status of Pod pod-adoption-release is Pending, waiting for it to be Running (with Ready = true) Jan 9 23:10:03.703: INFO: The status of Pod pod-adoption-release is Pending, waiting for it to be Running (with Ready = true) Jan 9 23:10:05.703: INFO: The status of Pod pod-adoption-release is Pending, waiting for it to be Running (with Ready = true) Jan 9 23:10:07.703: INFO: The status of Pod pod-adoption-release is Pending, waiting for it to be Running (with Ready = true) Jan 9 23:10:09.703: INFO: The status of Pod pod-adoption-release is Pending, waiting for it to be Running (with Ready = true) Jan 9 23:10:11.703: INFO: The status of Pod pod-adoption-release is Pending, waiting for it to be Running (with Ready = true) Jan 9 23:10:13.704: INFO: The status of Pod pod-adoption-release is Pending, waiting for it to be Running (with Ready = true) Jan 9 23:10:15.704: INFO: The status of Pod pod-adoption-release is Pending, waiting for it to be Running (with Ready = true) Jan 9 23:10:17.704: INFO: The status of Pod pod-adoption-release is Pending, waiting for it to be Running (with Ready = true) Jan 9 23:10:19.703: INFO: The status of Pod pod-adoption-release is Pending, waiting for it to be Running (with Ready = true) Jan 9 23:10:21.705: INFO: The status of Pod pod-adoption-release is Pending, waiting for it to be Running (with Ready = true) Jan 9 23:10:21.736: INFO: The status of Pod pod-adoption-release is Pending, waiting for it to be Running (with Ready = true) Jan 9 23:10:21.737: FAIL: Unexpected error: <*errors.errorString | 0xc0002482c0>: { s: "timed out waiting for the condition", } timed out waiting for the condition occurred Full Stack Trace k8s.io/kubernetes/test/e2e/framework.(*PodClient).CreateSync(0xc0030176c8, 0xc0056d2750) /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/pods.go:107 +0x94 k8s.io/kubernetes/test/e2e/apps.testRSAdoptMatchingAndReleaseNotMatching(0xc00080a580) /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/apps/replica_set.go:324 +0x1fb k8s.io/kubernetes/test/e2e/apps.glob..func8.4() /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/apps/replica_set.go:130 +0x1d k8s.io/kubernetes/test/e2e.RunE2ETests(0x24c66d7) _output/dockerized/go/src/k8s.io/kubernetes/test/e2e/e2e.go:133 +0x697 k8s.io/kubernetes/test/e2e.TestE2E(0x0) _output/dockerized/go/src/k8s.io/kubernetes/test/e2e/e2e_test.go:136 +0x19 testing.tRunner(0xc0005916c0, 0x735e880) /usr/local/go/src/testing/testing.go:1259 +0x102 created by testing.(*T).Run /usr/local/go/src/testing/testing.go:1306 +0x35a [AfterEach] [sig-apps] ReplicaSet /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:186 �[1mSTEP�[0m: Collecting events from namespace "replicaset-2552". �[1mSTEP�[0m: Found 12 events. Jan 9 23:10:21.769: INFO: At 2023-01-09 23:05:21 +0000 UTC - event for pod-adoption-release: {default-scheduler } Scheduled: Successfully assigned replicaset-2552/pod-adoption-release to ip-172-20-55-216.ca-central-1.compute.internal Jan 9 23:10:21.769: INFO: At 2023-01-09 23:05:25 +0000 UTC - event for pod-adoption-release: {kubelet ip-172-20-55-216.ca-central-1.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "a694135ec5ce589c90ace7aca94ae4031a6b0331d1ddf629b2b6547c51677ca3" network for pod "pod-adoption-release": networkPlugin cni failed to set up pod "pod-adoption-release_replicaset-2552" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 9 23:10:21.769: INFO: At 2023-01-09 23:05:26 +0000 UTC - event for pod-adoption-release: {kubelet ip-172-20-55-216.ca-central-1.compute.internal} SandboxChanged: Pod sandbox changed, it will be killed and re-created. Jan 9 23:10:21.769: INFO: At 2023-01-09 23:05:29 +0000 UTC - event for pod-adoption-release: {kubelet ip-172-20-55-216.ca-central-1.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "52ae0138ac4d8e6c51cb959ef2fda302d95178eb10ef397a6786611935302ce2" network for pod "pod-adoption-release": networkPlugin cni failed to set up pod "pod-adoption-release_replicaset-2552" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 9 23:10:21.769: INFO: At 2023-01-09 23:05:36 +0000 UTC - event for pod-adoption-release: {kubelet ip-172-20-55-216.ca-central-1.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "41febe39a741ae152258b8a54862a762c44c1812532169242cdb89f62bf2f36e" network for pod "pod-adoption-release": networkPlugin cni failed to set up pod "pod-adoption-release_replicaset-2552" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 9 23:10:21.769: INFO: At 2023-01-09 23:05:40 +0000 UTC - event for pod-adoption-release: {kubelet ip-172-20-55-216.ca-central-1.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "ae638b30e77d8a3463d6d3bc8867c5859892502d91a2ed51ef3dc0309dd88316" network for pod "pod-adoption-release": networkPlugin cni failed to set up pod "pod-adoption-release_replicaset-2552" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 9 23:10:21.769: INFO: At 2023-01-09 23:05:46 +0000 UTC - event for pod-adoption-release: {kubelet ip-172-20-55-216.ca-central-1.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "e086491b60f80b50aae17ba1555f0fc0a4dfeeaf7d782bb28d892a8be08c5263" network for pod "pod-adoption-release": networkPlugin cni failed to set up pod "pod-adoption-release_replicaset-2552" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 9 23:10:21.769: INFO: At 2023-01-09 23:05:51 +0000 UTC - event for pod-adoption-release: {kubelet ip-172-20-55-216.ca-central-1.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "fcb4a5b09e1113878a2545685f8444c30f75461bcce7376208950d933671e4c2" network for pod "pod-adoption-release": networkPlugin cni failed to set up pod "pod-adoption-release_replicaset-2552" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 9 23:10:21.769: INFO: At 2023-01-09 23:05:58 +0000 UTC - event for pod-adoption-release: {kubelet ip-172-20-55-216.ca-central-1.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "7c6202845719195927f5cd197746c3fb9bb60600d3b8e7df00dd4b07980ccac4" network for pod "pod-adoption-release": networkPlugin cni failed to set up pod "pod-adoption-release_replicaset-2552" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 9 23:10:21.769: INFO: At 2023-01-09 23:06:01 +0000 UTC - event for pod-adoption-release: {kubelet ip-172-20-55-216.ca-central-1.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "6f8e119896728525a6b0ce41c6ca4057529daed1a1e468525b06b27aa9b7a48b" network for pod "pod-adoption-release": networkPlugin cni failed to set up pod "pod-adoption-release_replicaset-2552" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 9 23:10:21.769: INFO: At 2023-01-09 23:06:08 +0000 UTC - event for pod-adoption-release: {kubelet ip-172-20-55-216.ca-central-1.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "097b3daa1f41e97cd3e5868928ba2d9a7d184059003c16a9712c23aff8c80b5d" network for pod "pod-adoption-release": networkPlugin cni failed to set up pod "pod-adoption-release_replicaset-2552" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 9 23:10:21.769: INFO: At 2023-01-09 23:06:11 +0000 UTC - event for pod-adoption-release: {kubelet ip-172-20-55-216.ca-central-1.compute.internal} FailedCreatePodSandBox: (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "57ea77b4c31ce6634c574b870277e0d3c356499fe082d0a569cd0840299720d5" network for pod "pod-adoption-release": networkPlugin cni failed to set up pod "pod-adoption-release_replicaset-2552" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 9 23:10:21.801: INFO: POD NODE PHASE GRACE CONDITIONS Jan 9 23:10:21.801: INFO: pod-adoption-release ip-172-20-55-216.ca-central-1.compute.internal Pending [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2023-01-09 23:05:21 +0000 UTC } {Ready False 0001-01-01 00:00:00 +0000 UTC 2023-01-09 23:05:21 +0000 UTC ContainersNotReady containers with unready status: [pod-adoption-release]} {ContainersReady False 0001-01-01 00:00:00 +0000 UTC 2023-01-09 23:05:21 +0000 UTC ContainersNotReady containers with unready status: [pod-adoption-release]} {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2023-01-09 23:05:21 +0000 UTC }] Jan 9 23:10:21.801: INFO: Jan 9 23:10:21.838: INFO: Unable to fetch replicaset-2552/pod-adoption-release/pod-adoption-release logs: the server rejected our request for an unknown reason (get pods pod-adoption-release) Jan 9 23:10:21.871: INFO: Logging node info for node ip-172-20-32-67.ca-central-1.compute.internal Jan 9 23:10:21.903: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-32-67.ca-central-1.compute.internal 1f51981b-a01d-404c-b3a4-6649f301f65d 45435 0 2023-01-09 22:38:45 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ca-central-1 failure-domain.beta.kubernetes.io/zone:ca-central-1a kubelet_cleanup:true kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-32-67.ca-central-1.compute.internal kubernetes.io/os:linux kubernetes.io/role:node node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:ca-central-1a topology.hostpath.csi/node:ip-172-20-32-67.ca-central-1.compute.internal topology.kubernetes.io/region:ca-central-1 topology.kubernetes.io/zone:ca-central-1a] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-0272cebea4c6c3d5d"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kops-controller Update v1 2023-01-09 22:38:45 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}} } {kube-controller-manager Update v1 2023-01-09 22:38:45 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.3.0/24\"":{}}}} } {kubelet Update v1 2023-01-09 22:38:45 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {e2e.test Update v1 2023-01-09 22:54:42 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubelet_cleanup":{}}}} } {kube-controller-manager Update v1 2023-01-09 22:55:59 +0000 UTC FieldsV1 {"f:status":{"f:volumesAttached":{}}} status} {kubelet Update v1 2023-01-09 22:56:00 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:capacity":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.3.0/24,DoNotUseExternalID:,ProviderID:aws:///ca-central-1a/i-0272cebea4c6c3d5d,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.3.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051644416 0} {<nil>} 3956684Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946786816 0} {<nil>} 3854284Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-01-09 23:06:12 +0000 UTC,LastTransitionTime:2023-01-09 22:38:45 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-01-09 23:06:12 +0000 UTC,LastTransitionTime:2023-01-09 22:38:45 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-01-09 23:06:12 +0000 UTC,LastTransitionTime:2023-01-09 22:38:45 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-01-09 23:06:12 +0000 UTC,LastTransitionTime:2023-01-09 22:39:06 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.32.67,},NodeAddress{Type:ExternalIP,Address:35.182.2.102,},NodeAddress{Type:Hostname,Address:ip-172-20-32-67.ca-central-1.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-32-67.ca-central-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-35-182-2-102.ca-central-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2536b48bd6b6dade0ed68af2555085,SystemUUID:ec2536b4-8bd6-b6da-de0e-d68af2555085,BootID:bfb3b0df-aafa-481c-9ac6-f85e5e4580e3,KernelVersion:5.15.0-1026-aws,OSImage:Ubuntu 22.04.1 LTS,ContainerRuntimeVersion:docker://20.10.17,KubeletVersion:v1.23.15,KubeProxyVersion:v1.23.15,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:456569528,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/jessie-dnsutils@sha256:11e6a66017ba4e4b938c1612b7a54a3befcefd354796c04e1dba76873a13518e k8s.gcr.io/e2e-test-images/jessie-dnsutils:1.5],SizeBytes:253371792,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:7e8bdd271312fd25fc5ff5a8f04727be84044eb3d7d8d03611972a6752e2e11e k8s.gcr.io/e2e-test-images/agnhost:2.39],SizeBytes:126872991,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nautilus@sha256:99c0d6f1ad24a1aa1905d9c6534d193f268f7b23f9add2ae6bb41f31094bdd5c k8s.gcr.io/e2e-test-images/nautilus:1.5],SizeBytes:124758741,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:1b9d1b2f36cb2dbee1960e82a9344aeb11bd4c4c03abf5e1853e0559c23855e3 k8s.gcr.io/e2e-test-images/httpd:2.4.38-2],SizeBytes:123781643,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.23.15 registry.k8s.io/kube-proxy-amd64:v1.23.15],SizeBytes:112336036,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:10e231cf07c82dfab5141c1afe548dc734e5fc1f67665ec3982a325d1bd31a9a registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.0],SizeBytes:94471559,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:c8e03f60afa90a28e4bb6ec9a8d0fc36d89de4b7475cf2d613afa793ec969fe0 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.0],SizeBytes:56382329,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:4e74c0492bceddc598de1c90cc5bc14dcda94cb49fa9c5bad9d117c4834b5e08 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.1],SizeBytes:56379269,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:36c31f7e1f433c9634d24f876353e8646246d81a03c4e351202c2644daff1620 k8s.gcr.io/sig-storage/csi-resizer:v1.2.0],SizeBytes:53956308,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:ed98431376c9e944e19a465fe8ea944806714dd95416a0821096c78d66b579bd k8s.gcr.io/sig-storage/csi-snapshotter:v4.1.1],SizeBytes:53531990,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:c5be65d6679efabb969d9b019300d187437ae876f992c40911fd2892bbef3b36 k8s.gcr.io/sig-storage/csi-attacher:v3.2.0],SizeBytes:53500826,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:60ab9b3e6a030d3038c87c0d6bca2930f58d1d72823e6a4af09767dc83b696a2 k8s.gcr.io/sig-storage/csi-attacher:v3.2.1],SizeBytes:53499570,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:017727efcfeb7d053af68e51436ce8e65edbc6ca573720afb4f79c8594036955 registry.k8s.io/coredns/coredns:v1.10.0],SizeBytes:50249337,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:7a5ba58a44e0d749e0767e4e37315bcf6a61f33ce3185c1991848af4db0fb70a k8s.gcr.io/sig-storage/csi-resizer:v1.1.0],SizeBytes:49176472,},ContainerImage{Names:[k8s.gcr.io/sig-storage/hostpathplugin@sha256:232fe80174d60d520d36043103853a1d7ab4b7f3782cf43e45034f04ccda58ce k8s.gcr.io/sig-storage/hostpathplugin:v1.7.1],SizeBytes:29977921,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:a61d309da54641db41fb8f35718f744e9f730d4d0384f8c4b186ddc9f06cbd5f k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.1.0],SizeBytes:19662887,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:19582112,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:2dee3fe5fe861bb66c3a4ac51114f3447a4cd35870e0f2e2b558c7a400d89589 k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.2.0],SizeBytes:18699436,},ContainerImage{Names:[k8s.gcr.io/sig-storage/mock-driver@sha256:a7b517f9e0f42ffade802eef9cefa271372386b85f55b702b493241e58459793 k8s.gcr.io/sig-storage/mock-driver:v4.1.0],SizeBytes:17680993,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:17611980,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:529be2c9770add0cdd0c989115222ea9fc1be430c11095eb9f6dafcf98a36e2b k8s.gcr.io/sig-storage/livenessprobe:v2.4.0],SizeBytes:17175488,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:13616070e3f29de4417eee434a8ef472221c9e51b3d037b5a6b46cef08eb7443 k8s.gcr.io/e2e-test-images/nginx:1.14-2],SizeBytes:16032814,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nonewprivs@sha256:8ac1264691820febacf3aea5d152cbde6d10685731ec14966a9401c6f47a68ac k8s.gcr.io/e2e-test-images/nonewprivs:1.3],SizeBytes:7107254,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf k8s.gcr.io/e2e-test-images/busybox:1.29-2],SizeBytes:1154361,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:39e1e963e5310e9c313bad51523be012ede7b35bb9316517d19089a010356592 k8s.gcr.io/e2e-test-images/busybox:1.29-1],SizeBytes:1154361,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db k8s.gcr.io/pause:3.6 registry.k8s.io/pause:3.6],SizeBytes:682696,},},VolumesInUse:[kubernetes.io/csi/ebs.csi.aws.com^vol-01e8a4a8ff1da0e10],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-01e8a4a8ff1da0e10,DevicePath:,},},Config:nil,},} Jan 9 23:10:21.903: INFO: Logging kubelet events for node ip-172-20-32-67.ca-central-1.compute.internal Jan 9 23:10:21.937: INFO: Logging pods the kubelet thinks is on node ip-172-20-32-67.ca-central-1.compute.internal Jan 9 23:10:21.977: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-gjc2h started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:10:21.977: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:10:21.977: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-8vzqv started at 2023-01-09 22:54:43 +0000 UTC (0+1 container statuses recorded) Jan 9 23:10:21.977: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:10:21.977: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-jb6kp started at 2023-01-09 22:54:43 +0000 UTC (0+1 container statuses recorded) Jan 9 23:10:21.977: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:10:21.977: INFO: cilium-dvsfs started at 2023-01-09 22:38:46 +0000 UTC (1+1 container statuses recorded) Jan 9 23:10:21.977: INFO: Init container clean-cilium-state ready: true, restart count 0 Jan 9 23:10:21.977: INFO: Container cilium-agent ready: true, restart count 0 Jan 9 23:10:21.977: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-fr98x started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:10:21.977: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:10:21.977: INFO: rs-zn5kz started at 2023-01-09 22:59:56 +0000 UTC (0+1 container statuses recorded) Jan 9 23:10:21.977: INFO: Container donothing ready: true, restart count 0 Jan 9 23:10:21.977: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-hwbbn started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:10:21.977: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:10:21.977: INFO: coredns-867df8f45c-sckrz started at 2023-01-09 22:39:23 +0000 UTC (0+1 container statuses recorded) Jan 9 23:10:21.977: INFO: Container coredns ready: true, restart count 0 Jan 9 23:10:21.977: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-44ccv started at 2023-01-09 22:54:43 +0000 UTC (0+1 container statuses recorded) Jan 9 23:10:21.977: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:10:21.977: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-fzrxx started at 2023-01-09 22:54:43 +0000 UTC (0+1 container statuses recorded) Jan 9 23:10:21.977: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:10:21.977: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-mlcl4 started at 2023-01-09 22:54:43 +0000 UTC (0+1 container statuses recorded) Jan 9 23:10:21.977: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:10:21.977: INFO: rs-gxqdg started at 2023-01-09 22:59:56 +0000 UTC (0+1 container statuses recorded) Jan 9 23:10:21.977: INFO: Container donothing ready: false, restart count 0 Jan 9 23:10:21.977: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-5pr42 started at 2023-01-09 22:54:43 +0000 UTC (0+1 container statuses recorded) Jan 9 23:10:21.977: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:10:21.977: INFO: netserver-0 started at 2023-01-09 23:07:18 +0000 UTC (0+1 container statuses recorded) Jan 9 23:10:21.977: INFO: Container webserver ready: false, restart count 0 Jan 9 23:10:21.977: INFO: netserver-0 started at 2023-01-09 23:08:45 +0000 UTC (0+1 container statuses recorded) Jan 9 23:10:21.977: INFO: Container webserver ready: false, restart count 0 Jan 9 23:10:21.977: INFO: ebs-csi-node-lc4rb started at 2023-01-09 22:38:46 +0000 UTC (0+3 container statuses recorded) Jan 9 23:10:21.977: INFO: Container ebs-plugin ready: true, restart count 0 Jan 9 23:10:21.977: INFO: Container liveness-probe ready: true, restart count 0 Jan 9 23:10:21.977: INFO: Container node-driver-registrar ready: true, restart count 0 Jan 9 23:10:21.977: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-sddjv started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:10:21.977: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:10:21.977: INFO: csi-hostpathplugin-0 started at 2023-01-09 23:05:41 +0000 UTC (0+7 container statuses recorded) Jan 9 23:10:21.977: INFO: Container csi-attacher ready: false, restart count 0 Jan 9 23:10:21.977: INFO: Container csi-provisioner ready: false, restart count 0 Jan 9 23:10:21.977: INFO: Container csi-resizer ready: false, restart count 0 Jan 9 23:10:21.977: INFO: Container csi-snapshotter ready: false, restart count 0 Jan 9 23:10:21.977: INFO: Container hostpath ready: false, restart count 0 Jan 9 23:10:21.977: INFO: Container liveness-probe ready: false, restart count 0 Jan 9 23:10:21.977: INFO: Container node-driver-registrar ready: false, restart count 0 Jan 9 23:10:21.977: INFO: inline-volume-tester-5g6j6 started at 2023-01-09 22:55:56 +0000 UTC (0+1 container statuses recorded) Jan 9 23:10:21.977: INFO: Container csi-volume-tester ready: false, restart count 0 Jan 9 23:10:22.261: INFO: Latency metrics for node ip-172-20-32-67.ca-central-1.compute.internal Jan 9 23:10:22.261: INFO: Logging node info for node ip-172-20-37-64.ca-central-1.compute.internal Jan 9 23:10:22.293: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-37-64.ca-central-1.compute.internal d817613a-0fe5-40c0-a5e4-7b85b9f8506b 46487 0 2023-01-09 22:36:54 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:c5.large beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ca-central-1 failure-domain.beta.kubernetes.io/zone:ca-central-1a kops.k8s.io/kops-controller-pki: kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-37-64.ca-central-1.compute.internal kubernetes.io/os:linux kubernetes.io/role:master node-role.kubernetes.io/control-plane: node-role.kubernetes.io/master: node.kubernetes.io/exclude-from-external-load-balancers: node.kubernetes.io/instance-type:c5.large topology.ebs.csi.aws.com/zone:ca-central-1a topology.kubernetes.io/region:ca-central-1 topology.kubernetes.io/zone:ca-central-1a] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-06217673764fdafeb"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kubelet Update v1 2023-01-09 22:36:54 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {protokube Update v1 2023-01-09 22:37:19 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kops.k8s.io/kops-controller-pki":{},"f:node-role.kubernetes.io/control-plane":{},"f:node.kubernetes.io/exclude-from-external-load-balancers":{}}}} } {kops-controller Update v1 2023-01-09 22:37:21 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/master":{}}}} } {kube-controller-manager Update v1 2023-01-09 22:37:35 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.0.0/24\"":{}},"f:taints":{}}} } {kubelet Update v1 2023-01-09 22:38:39 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.0.0/24,DoNotUseExternalID:,ProviderID:aws:///ca-central-1a/i-06217673764fdafeb,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/master,Value:,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[100.96.0.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3862904832 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3758047232 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-01-09 23:09:24 +0000 UTC,LastTransitionTime:2023-01-09 22:36:48 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-01-09 23:09:24 +0000 UTC,LastTransitionTime:2023-01-09 22:36:48 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-01-09 23:09:24 +0000 UTC,LastTransitionTime:2023-01-09 22:36:48 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-01-09 23:09:24 +0000 UTC,LastTransitionTime:2023-01-09 22:37:35 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.37.64,},NodeAddress{Type:ExternalIP,Address:3.96.216.165,},NodeAddress{Type:Hostname,Address:ip-172-20-37-64.ca-central-1.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-37-64.ca-central-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-3-96-216-165.ca-central-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec27a483f61319a98e5ba2ee2d59e6a3,SystemUUID:ec27a483-f613-19a9-8e5b-a2ee2d59e6a3,BootID:9b41e339-1aaa-4374-8f11-8d34e12cd62a,KernelVersion:5.15.0-1026-aws,OSImage:Ubuntu 22.04.1 LTS,ContainerRuntimeVersion:docker://20.10.17,KubeletVersion:v1.23.15,KubeProxyVersion:v1.23.15,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[registry.k8s.io/etcdadm/etcd-manager@sha256:66a453db625abb268f4b3bbefc5a34a171d81e6e8796cecca54cfd71775c77c4 registry.k8s.io/etcdadm/etcd-manager:v3.0.20221209],SizeBytes:662209237,},ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:456569528,},ContainerImage{Names:[k8s.gcr.io/kube-apiserver-amd64:v1.23.15 registry.k8s.io/kube-apiserver-amd64:v1.23.15],SizeBytes:135240444,},ContainerImage{Names:[k8s.gcr.io/kube-controller-manager-amd64:v1.23.15 registry.k8s.io/kube-controller-manager-amd64:v1.23.15],SizeBytes:125046097,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.23.15 registry.k8s.io/kube-proxy-amd64:v1.23.15],SizeBytes:112336036,},ContainerImage{Names:[quay.io/cilium/operator@sha256:a6d24a006a6b92967ac90786b49bc1ac26e5477cf028cd1186efcfc2466484db quay.io/cilium/operator:v1.12.5],SizeBytes:110016956,},ContainerImage{Names:[registry.k8s.io/kops/kops-controller:1.26.0-beta.1],SizeBytes:104212800,},ContainerImage{Names:[registry.k8s.io/kops/dns-controller:1.26.0-beta.1],SizeBytes:102566451,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:10e231cf07c82dfab5141c1afe548dc734e5fc1f67665ec3982a325d1bd31a9a registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.0],SizeBytes:94471559,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:122bfb8c1edabb3c0edd63f06523e6940d958d19b3957dc7b1d6f81e9f1f6119 registry.k8s.io/sig-storage/csi-provisioner:v3.1.0],SizeBytes:57736070,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:9ebbf9f023e7b41ccee3d52afe39a89e3ddacdbb69269d583abfc25847cfd9e4 registry.k8s.io/sig-storage/csi-resizer:v1.4.0],SizeBytes:55464778,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:8b9c313c05f54fb04f8d430896f5f5904b6cb157df261501b29adc04d2b2dc7b registry.k8s.io/sig-storage/csi-attacher:v3.4.0],SizeBytes:54848595,},ContainerImage{Names:[k8s.gcr.io/kube-scheduler-amd64:v1.23.15 registry.k8s.io/kube-scheduler-amd64:v1.23.15],SizeBytes:53521430,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:19582112,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:17611980,},ContainerImage{Names:[registry.k8s.io/kops/kube-apiserver-healthcheck:1.26.0-beta.1],SizeBytes:8787111,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:682696,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Jan 9 23:10:22.294: INFO: Logging kubelet events for node ip-172-20-37-64.ca-central-1.compute.internal Jan 9 23:10:22.328: INFO: Logging pods the kubelet thinks is on node ip-172-20-37-64.ca-central-1.compute.internal Jan 9 23:10:22.367: INFO: kube-apiserver-ip-172-20-37-64.ca-central-1.compute.internal started at 2023-01-09 22:36:15 +0000 UTC (0+2 container statuses recorded) Jan 9 23:10:22.367: INFO: Container healthcheck ready: true, restart count 0 Jan 9 23:10:22.367: INFO: Container kube-apiserver ready: true, restart count 1 Jan 9 23:10:22.367: INFO: kube-controller-manager-ip-172-20-37-64.ca-central-1.compute.internal started at 2023-01-09 22:36:15 +0000 UTC (0+1 container statuses recorded) Jan 9 23:10:22.367: INFO: Container kube-controller-manager ready: true, restart count 2 Jan 9 23:10:22.367: INFO: kops-controller-cqrv8 started at 2023-01-09 22:37:19 +0000 UTC (0+1 container statuses recorded) Jan 9 23:10:22.367: INFO: Container kops-controller ready: true, restart count 0 Jan 9 23:10:22.367: INFO: cilium-q7lrg started at 2023-01-09 22:37:17 +0000 UTC (1+1 container statuses recorded) Jan 9 23:10:22.367: INFO: Init container clean-cilium-state ready: true, restart count 0 Jan 9 23:10:22.367: INFO: Container cilium-agent ready: true, restart count 0 Jan 9 23:10:22.367: INFO: cilium-operator-c7bfc9f44-w4nqz started at 2023-01-09 22:37:19 +0000 UTC (0+1 container statuses recorded) Jan 9 23:10:22.367: INFO: Container cilium-operator ready: true, restart count 0 Jan 9 23:10:22.367: INFO: ebs-csi-controller-5d8d7cc949-8wxx2 started at 2023-01-09 22:37:19 +0000 UTC (0+5 container statuses recorded) Jan 9 23:10:22.367: INFO: Container csi-attacher ready: true, restart count 0 Jan 9 23:10:22.367: INFO: Container csi-provisioner ready: true, restart count 0 Jan 9 23:10:22.367: INFO: Container csi-resizer ready: true, restart count 0 Jan 9 23:10:22.367: INFO: Container ebs-plugin ready: true, restart count 0 Jan 9 23:10:22.367: INFO: Container liveness-probe ready: true, restart count 0 Jan 9 23:10:22.367: INFO: dns-controller-867784b75c-sxxcl started at 2023-01-09 22:37:20 +0000 UTC (0+1 container statuses recorded) Jan 9 23:10:22.367: INFO: Container dns-controller ready: true, restart count 0 Jan 9 23:10:22.367: INFO: kube-scheduler-ip-172-20-37-64.ca-central-1.compute.internal started at 2023-01-09 22:36:15 +0000 UTC (0+1 container statuses recorded) Jan 9 23:10:22.367: INFO: Container kube-scheduler ready: true, restart count 0 Jan 9 23:10:22.367: INFO: etcd-manager-events-ip-172-20-37-64.ca-central-1.compute.internal started at 2023-01-09 22:36:15 +0000 UTC (0+1 container statuses recorded) Jan 9 23:10:22.367: INFO: Container etcd-manager ready: true, restart count 0 Jan 9 23:10:22.367: INFO: etcd-manager-main-ip-172-20-37-64.ca-central-1.compute.internal started at 2023-01-09 22:36:15 +0000 UTC (0+1 container statuses recorded) Jan 9 23:10:22.367: INFO: Container etcd-manager ready: true, restart count 0 Jan 9 23:10:22.367: INFO: ebs-csi-node-8q4pg started at 2023-01-09 22:37:17 +0000 UTC (0+3 container statuses recorded) Jan 9 23:10:22.367: INFO: Container ebs-plugin ready: true, restart count 0 Jan 9 23:10:22.367: INFO: Container liveness-probe ready: true, restart count 0 Jan 9 23:10:22.367: INFO: Container node-driver-registrar ready: true, restart count 0 Jan 9 23:10:22.486: INFO: Latency metrics for node ip-172-20-37-64.ca-central-1.compute.internal Jan 9 23:10:22.486: INFO: Logging node info for node ip-172-20-55-216.ca-central-1.compute.internal Jan 9 23:10:22.517: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-55-216.ca-central-1.compute.internal 2ceff781-505a-4706-ad8b-e38e37f72571 46096 0 2023-01-09 22:38:47 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ca-central-1 failure-domain.beta.kubernetes.io/zone:ca-central-1a kubelet_cleanup:true kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-55-216.ca-central-1.compute.internal kubernetes.io/os:linux kubernetes.io/role:node node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:ca-central-1a topology.hostpath.csi/node:ip-172-20-55-216.ca-central-1.compute.internal topology.kubernetes.io/region:ca-central-1 topology.kubernetes.io/zone:ca-central-1a] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-0c78741e44993ae7f"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kops-controller Update v1 2023-01-09 22:38:47 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}} } {kube-controller-manager Update v1 2023-01-09 22:38:47 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.4.0/24\"":{}}}} } {kubelet Update v1 2023-01-09 22:38:47 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {e2e.test Update v1 2023-01-09 22:54:42 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubelet_cleanup":{}}}} } {kube-controller-manager Update v1 2023-01-09 22:56:45 +0000 UTC FieldsV1 {"f:status":{"f:volumesAttached":{}}} status} {kubelet Update v1 2023-01-09 22:56:51 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:capacity":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.4.0/24,DoNotUseExternalID:,ProviderID:aws:///ca-central-1a/i-0c78741e44993ae7f,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.4.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051644416 0} {<nil>} 3956684Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946786816 0} {<nil>} 3854284Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-01-09 23:08:06 +0000 UTC,LastTransitionTime:2023-01-09 22:38:47 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-01-09 23:08:06 +0000 UTC,LastTransitionTime:2023-01-09 22:38:47 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-01-09 23:08:06 +0000 UTC,LastTransitionTime:2023-01-09 22:38:47 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-01-09 23:08:06 +0000 UTC,LastTransitionTime:2023-01-09 22:39:07 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.55.216,},NodeAddress{Type:ExternalIP,Address:15.222.60.56,},NodeAddress{Type:Hostname,Address:ip-172-20-55-216.ca-central-1.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-55-216.ca-central-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-15-222-60-56.ca-central-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec249e6a3570a28ebd4276206d3ae523,SystemUUID:ec249e6a-3570-a28e-bd42-76206d3ae523,BootID:d58dcef2-849d-4807-9ba5-0438ebaa05b9,KernelVersion:5.15.0-1026-aws,OSImage:Ubuntu 22.04.1 LTS,ContainerRuntimeVersion:docker://20.10.17,KubeletVersion:v1.23.15,KubeProxyVersion:v1.23.15,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:456569528,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/jessie-dnsutils@sha256:11e6a66017ba4e4b938c1612b7a54a3befcefd354796c04e1dba76873a13518e k8s.gcr.io/e2e-test-images/jessie-dnsutils:1.5],SizeBytes:253371792,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:20f25f275d46aa728f7615a1ccc19c78b2ed89435bf943a44b339f70f45508e6 k8s.gcr.io/e2e-test-images/httpd:2.4.39-2],SizeBytes:126894770,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:7e8bdd271312fd25fc5ff5a8f04727be84044eb3d7d8d03611972a6752e2e11e k8s.gcr.io/e2e-test-images/agnhost:2.39],SizeBytes:126872991,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nautilus@sha256:99c0d6f1ad24a1aa1905d9c6534d193f268f7b23f9add2ae6bb41f31094bdd5c k8s.gcr.io/e2e-test-images/nautilus:1.5],SizeBytes:124758741,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:1b9d1b2f36cb2dbee1960e82a9344aeb11bd4c4c03abf5e1853e0559c23855e3 k8s.gcr.io/e2e-test-images/httpd:2.4.38-2],SizeBytes:123781643,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.23.15 registry.k8s.io/kube-proxy-amd64:v1.23.15],SizeBytes:112336036,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:10e231cf07c82dfab5141c1afe548dc734e5fc1f67665ec3982a325d1bd31a9a registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.0],SizeBytes:94471559,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:c8e03f60afa90a28e4bb6ec9a8d0fc36d89de4b7475cf2d613afa793ec969fe0 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.0],SizeBytes:56382329,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:4e74c0492bceddc598de1c90cc5bc14dcda94cb49fa9c5bad9d117c4834b5e08 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.1],SizeBytes:56379269,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:36c31f7e1f433c9634d24f876353e8646246d81a03c4e351202c2644daff1620 k8s.gcr.io/sig-storage/csi-resizer:v1.2.0],SizeBytes:53956308,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:ed98431376c9e944e19a465fe8ea944806714dd95416a0821096c78d66b579bd k8s.gcr.io/sig-storage/csi-snapshotter:v4.1.1],SizeBytes:53531990,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:c5be65d6679efabb969d9b019300d187437ae876f992c40911fd2892bbef3b36 k8s.gcr.io/sig-storage/csi-attacher:v3.2.0],SizeBytes:53500826,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:60ab9b3e6a030d3038c87c0d6bca2930f58d1d72823e6a4af09767dc83b696a2 k8s.gcr.io/sig-storage/csi-attacher:v3.2.1],SizeBytes:53499570,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:7a5ba58a44e0d749e0767e4e37315bcf6a61f33ce3185c1991848af4db0fb70a k8s.gcr.io/sig-storage/csi-resizer:v1.1.0],SizeBytes:49176472,},ContainerImage{Names:[k8s.gcr.io/sig-storage/hostpathplugin@sha256:232fe80174d60d520d36043103853a1d7ab4b7f3782cf43e45034f04ccda58ce k8s.gcr.io/sig-storage/hostpathplugin:v1.7.1],SizeBytes:29977921,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:a61d309da54641db41fb8f35718f744e9f730d4d0384f8c4b186ddc9f06cbd5f k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.1.0],SizeBytes:19662887,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:19582112,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:2dee3fe5fe861bb66c3a4ac51114f3447a4cd35870e0f2e2b558c7a400d89589 k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.2.0],SizeBytes:18699436,},ContainerImage{Names:[k8s.gcr.io/sig-storage/mock-driver@sha256:a7b517f9e0f42ffade802eef9cefa271372386b85f55b702b493241e58459793 k8s.gcr.io/sig-storage/mock-driver:v4.1.0],SizeBytes:17680993,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:17611980,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:529be2c9770add0cdd0c989115222ea9fc1be430c11095eb9f6dafcf98a36e2b k8s.gcr.io/sig-storage/livenessprobe:v2.4.0],SizeBytes:17175488,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:13616070e3f29de4417eee434a8ef472221c9e51b3d037b5a6b46cef08eb7443 k8s.gcr.io/e2e-test-images/nginx:1.14-2],SizeBytes:16032814,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf k8s.gcr.io/e2e-test-images/busybox:1.29-2],SizeBytes:1154361,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db k8s.gcr.io/pause:3.6 registry.k8s.io/pause:3.6],SizeBytes:682696,},},VolumesInUse:[kubernetes.io/csi/ebs.csi.aws.com^vol-0c8e4ae9d44bf0644 kubernetes.io/csi/ebs.csi.aws.com^vol-0d0fbf37c343045c2],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-0d0fbf37c343045c2,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-0c8e4ae9d44bf0644,DevicePath:,},},Config:nil,},} Jan 9 23:10:22.518: INFO: Logging kubelet events for node ip-172-20-55-216.ca-central-1.compute.internal Jan 9 23:10:22.553: INFO: Logging pods the kubelet thinks is on node ip-172-20-55-216.ca-central-1.compute.internal Jan 9 23:10:22.596: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-6ks7b started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:10:22.596: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:10:22.596: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-72kwd started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:10:22.596: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:10:22.596: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-krwzl started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:10:22.596: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:10:22.596: INFO: rs-j4qxp started at 2023-01-09 22:59:56 +0000 UTC (0+1 container statuses recorded) Jan 9 23:10:22.596: INFO: Container donothing ready: false, restart count 0 Jan 9 23:10:22.596: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-qglk4 started at 2023-01-09 22:54:43 +0000 UTC (0+1 container statuses recorded) Jan 9 23:10:22.596: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:10:22.596: INFO: httpd started at 2023-01-09 23:06:03 +0000 UTC (0+1 container statuses recorded) Jan 9 23:10:22.596: INFO: Container httpd ready: false, restart count 0 Jan 9 23:10:22.596: INFO: netserver-1 started at 2023-01-09 23:08:45 +0000 UTC (0+1 container statuses recorded) Jan 9 23:10:22.596: INFO: Container webserver ready: false, restart count 0 Jan 9 23:10:22.596: INFO: pod-adoption-release started at 2023-01-09 23:05:21 +0000 UTC (0+1 container statuses recorded) Jan 9 23:10:22.596: INFO: Container pod-adoption-release ready: false, restart count 0 Jan 9 23:10:22.596: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-b7bxr started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:10:22.596: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:10:22.596: INFO: pod-subpath-test-inlinevolume-x5g6 started at 2023-01-09 23:05:45 +0000 UTC (2+1 container statuses recorded) Jan 9 23:10:22.596: INFO: Init container init-volume-inlinevolume-x5g6 ready: false, restart count 0 Jan 9 23:10:22.596: INFO: Init container test-init-volume-inlinevolume-x5g6 ready: false, restart count 0 Jan 9 23:10:22.596: INFO: Container test-container-subpath-inlinevolume-x5g6 ready: false, restart count 0 Jan 9 23:10:22.596: INFO: inline-volume-tester-7mx4s started at 2023-01-09 22:56:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:10:22.596: INFO: Container csi-volume-tester ready: false, restart count 0 Jan 9 23:10:22.596: INFO: hostexec-ip-172-20-55-216.ca-central-1.compute.internal-ddnj4 started at 2023-01-09 23:10:19 +0000 UTC (0+1 container statuses recorded) Jan 9 23:10:22.596: INFO: Container agnhost-container ready: true, restart count 0 Jan 9 23:10:22.596: INFO: cilium-dqpw4 started at 2023-01-09 22:38:48 +0000 UTC (1+1 container statuses recorded) Jan 9 23:10:22.596: INFO: Init container clean-cilium-state ready: true, restart count 0 Jan 9 23:10:22.596: INFO: Container cilium-agent ready: true, restart count 0 Jan 9 23:10:22.596: INFO: inline-volume-tester-crzhr started at 2023-01-09 23:05:03 +0000 UTC (0+1 container statuses recorded) Jan 9 23:10:22.596: INFO: Container csi-volume-tester ready: false, restart count 0 Jan 9 23:10:22.596: INFO: netserver-1 started at 2023-01-09 23:07:18 +0000 UTC (0+1 container statuses recorded) Jan 9 23:10:22.596: INFO: Container webserver ready: false, restart count 0 Jan 9 23:10:22.596: INFO: fail-once-non-local-mh6qc started at 2023-01-09 22:56:53 +0000 UTC (0+1 container statuses recorded) Jan 9 23:10:22.596: INFO: Container c ready: false, restart count 0 Jan 9 23:10:22.596: INFO: rs-lf7br started at 2023-01-09 22:59:56 +0000 UTC (0+1 container statuses recorded) Jan 9 23:10:22.596: INFO: Container donothing ready: false, restart count 0 Jan 9 23:10:22.596: INFO: pause-pod-78b7f9b75c-l4st2 started at 2023-01-09 23:05:44 +0000 UTC (0+1 container statuses recorded) Jan 9 23:10:22.596: INFO: Container agnhost-pause ready: false, restart count 0 Jan 9 23:10:22.596: INFO: logs-generator started at 2023-01-09 23:05:28 +0000 UTC (0+1 container statuses recorded) Jan 9 23:10:22.596: INFO: Container logs-generator ready: false, restart count 0 Jan 9 23:10:22.596: INFO: pod-8ade382c-2fb2-4f02-8d50-57f7b6a15696 started at 2023-01-09 23:06:54 +0000 UTC (0+1 container statuses recorded) Jan 9 23:10:22.596: INFO: Container test-container ready: false, restart count 0 Jan 9 23:10:22.596: INFO: startup-ff2f8ca6-ae0a-4fae-9483-a7d80b37fdc1 started at 2023-01-09 23:07:24 +0000 UTC (0+1 container statuses recorded) Jan 9 23:10:22.596: INFO: Container busybox ready: false, restart count 0 Jan 9 23:10:22.596: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-mcd88 started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:10:22.596: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:10:22.596: INFO: echo-sourceip started at 2023-01-09 23:00:54 +0000 UTC (0+1 container statuses recorded) Jan 9 23:10:22.596: INFO: Container agnhost-container ready: true, restart count 0 Jan 9 23:10:22.596: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-8kb86 started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:10:22.596: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:10:22.596: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-vwd6k started at 2023-01-09 22:54:43 +0000 UTC (0+1 container statuses recorded) Jan 9 23:10:22.596: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:10:22.596: INFO: image-pull-test99b7b050-d558-462d-a744-2f374728e347 started at 2023-01-09 23:07:56 +0000 UTC (0+1 container statuses recorded) Jan 9 23:10:22.596: INFO: Container image-pull-test ready: false, restart count 0 Jan 9 23:10:22.596: INFO: fail-once-non-local-nx7cr started at 2023-01-09 22:56:53 +0000 UTC (0+1 container statuses recorded) Jan 9 23:10:22.596: INFO: Container c ready: false, restart count 0 Jan 9 23:10:22.596: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-6nqvb started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:10:22.596: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:10:22.596: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-qj2bw started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:10:22.596: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:10:22.596: INFO: rs-ctw9c started at 2023-01-09 22:59:56 +0000 UTC (0+1 container statuses recorded) Jan 9 23:10:22.596: INFO: Container donothing ready: true, restart count 0 Jan 9 23:10:22.596: INFO: ebs-csi-node-psq97 started at 2023-01-09 22:38:48 +0000 UTC (0+3 container statuses recorded) Jan 9 23:10:22.596: INFO: Container ebs-plugin ready: true, restart count 0 Jan 9 23:10:22.596: INFO: Container liveness-probe ready: true, restart count 0 Jan 9 23:10:22.596: INFO: Container node-driver-registrar ready: true, restart count 0 Jan 9 23:10:23.289: INFO: Latency metrics for node ip-172-20-55-216.ca-central-1.compute.internal Jan 9 23:10:23.289: INFO: Logging node info for node ip-172-20-57-251.ca-central-1.compute.internal Jan 9 23:10:23.321: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-57-251.ca-central-1.compute.internal eaaaf6eb-cbd5-4763-9c39-fb0e1a4ce44a 45391 0 2023-01-09 22:38:45 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ca-central-1 failure-domain.beta.kubernetes.io/zone:ca-central-1a kubelet_cleanup:true kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-57-251.ca-central-1.compute.internal kubernetes.io/os:linux kubernetes.io/role:node node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:ca-central-1a topology.hostpath.csi/node:ip-172-20-57-251.ca-central-1.compute.internal topology.kubernetes.io/region:ca-central-1 topology.kubernetes.io/zone:ca-central-1a] map[csi.volume.kubernetes.io/nodeid:{"csi-mock-csi-mock-volumes-9402":"csi-mock-csi-mock-volumes-9402","ebs.csi.aws.com":"i-0513e7d6e5eba8eec"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kops-controller Update v1 2023-01-09 22:38:45 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}} } {kube-controller-manager Update v1 2023-01-09 22:38:45 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.2.0/24\"":{}}}} } {kubelet Update v1 2023-01-09 22:38:45 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {e2e.test Update v1 2023-01-09 22:54:42 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubelet_cleanup":{}}}} } {kube-controller-manager Update v1 2023-01-09 23:00:23 +0000 UTC FieldsV1 {"f:status":{"f:volumesAttached":{}}} status} {kubelet Update v1 2023-01-09 23:00:27 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.2.0/24,DoNotUseExternalID:,ProviderID:aws:///ca-central-1a/i-0513e7d6e5eba8eec,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.2.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051652608 0} {<nil>} 3956692Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946795008 0} {<nil>} 3854292Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-01-09 23:06:06 +0000 UTC,LastTransitionTime:2023-01-09 22:38:45 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-01-09 23:06:06 +0000 UTC,LastTransitionTime:2023-01-09 22:38:45 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-01-09 23:06:06 +0000 UTC,LastTransitionTime:2023-01-09 22:38:45 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-01-09 23:06:06 +0000 UTC,LastTransitionTime:2023-01-09 22:39:05 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.57.251,},NodeAddress{Type:ExternalIP,Address:3.96.144.209,},NodeAddress{Type:Hostname,Address:ip-172-20-57-251.ca-central-1.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-57-251.ca-central-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-3-96-144-209.ca-central-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2f0d7ba95b37a96a1d9360f6071d84,SystemUUID:ec2f0d7b-a95b-37a9-6a1d-9360f6071d84,BootID:67865282-1911-4e38-8675-a7b78d5ae45e,KernelVersion:5.15.0-1026-aws,OSImage:Ubuntu 22.04.1 LTS,ContainerRuntimeVersion:docker://20.10.17,KubeletVersion:v1.23.15,KubeProxyVersion:v1.23.15,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:456569528,},ContainerImage{Names:[k8s.gcr.io/etcd@sha256:dd75ec974b0a2a6f6bb47001ba09207976e625db898d1b16735528c009cb171c k8s.gcr.io/etcd:3.5.6-0],SizeBytes:299475478,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/jessie-dnsutils@sha256:11e6a66017ba4e4b938c1612b7a54a3befcefd354796c04e1dba76873a13518e k8s.gcr.io/e2e-test-images/jessie-dnsutils:1.5],SizeBytes:253371792,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:20f25f275d46aa728f7615a1ccc19c78b2ed89435bf943a44b339f70f45508e6 k8s.gcr.io/e2e-test-images/httpd:2.4.39-2],SizeBytes:126894770,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:7e8bdd271312fd25fc5ff5a8f04727be84044eb3d7d8d03611972a6752e2e11e k8s.gcr.io/e2e-test-images/agnhost:2.39],SizeBytes:126872991,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nautilus@sha256:99c0d6f1ad24a1aa1905d9c6534d193f268f7b23f9add2ae6bb41f31094bdd5c k8s.gcr.io/e2e-test-images/nautilus:1.5],SizeBytes:124758741,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:1b9d1b2f36cb2dbee1960e82a9344aeb11bd4c4c03abf5e1853e0559c23855e3 k8s.gcr.io/e2e-test-images/httpd:2.4.38-2],SizeBytes:123781643,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.23.15 registry.k8s.io/kube-proxy-amd64:v1.23.15],SizeBytes:112336036,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:10e231cf07c82dfab5141c1afe548dc734e5fc1f67665ec3982a325d1bd31a9a registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.0],SizeBytes:94471559,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/sample-apiserver@sha256:f9c93b92b6ff750b41a93c4e4fe0bfe384597aeb841e2539d5444815c55b2d8f k8s.gcr.io/e2e-test-images/sample-apiserver:1.17.5],SizeBytes:56504541,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:c8e03f60afa90a28e4bb6ec9a8d0fc36d89de4b7475cf2d613afa793ec969fe0 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.0],SizeBytes:56382329,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:4e74c0492bceddc598de1c90cc5bc14dcda94cb49fa9c5bad9d117c4834b5e08 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.1],SizeBytes:56379269,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:36c31f7e1f433c9634d24f876353e8646246d81a03c4e351202c2644daff1620 k8s.gcr.io/sig-storage/csi-resizer:v1.2.0],SizeBytes:53956308,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:ed98431376c9e944e19a465fe8ea944806714dd95416a0821096c78d66b579bd k8s.gcr.io/sig-storage/csi-snapshotter:v4.1.1],SizeBytes:53531990,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:c5be65d6679efabb969d9b019300d187437ae876f992c40911fd2892bbef3b36 k8s.gcr.io/sig-storage/csi-attacher:v3.2.0],SizeBytes:53500826,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:60ab9b3e6a030d3038c87c0d6bca2930f58d1d72823e6a4af09767dc83b696a2 k8s.gcr.io/sig-storage/csi-attacher:v3.2.1],SizeBytes:53499570,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:7a5ba58a44e0d749e0767e4e37315bcf6a61f33ce3185c1991848af4db0fb70a k8s.gcr.io/sig-storage/csi-resizer:v1.1.0],SizeBytes:49176472,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/regression-issue-74839@sha256:b4f1d8d61bdad84bd50442d161d5460e4019d53e989b64220fdbc62fc87d76bf k8s.gcr.io/e2e-test-images/regression-issue-74839:1.2],SizeBytes:44576952,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nonroot@sha256:b9e2958a3dd879e3cf11142228c6d073d0fc4ea2e857c3be6f4fb0ab5fb2c937 k8s.gcr.io/e2e-test-images/nonroot:1.2],SizeBytes:42321438,},ContainerImage{Names:[k8s.gcr.io/sig-storage/hostpathplugin@sha256:232fe80174d60d520d36043103853a1d7ab4b7f3782cf43e45034f04ccda58ce k8s.gcr.io/sig-storage/hostpathplugin:v1.7.1],SizeBytes:29977921,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:a61d309da54641db41fb8f35718f744e9f730d4d0384f8c4b186ddc9f06cbd5f k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.1.0],SizeBytes:19662887,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:19582112,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:2dee3fe5fe861bb66c3a4ac51114f3447a4cd35870e0f2e2b558c7a400d89589 k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.2.0],SizeBytes:18699436,},ContainerImage{Names:[k8s.gcr.io/sig-storage/mock-driver@sha256:a7b517f9e0f42ffade802eef9cefa271372386b85f55b702b493241e58459793 k8s.gcr.io/sig-storage/mock-driver:v4.1.0],SizeBytes:17680993,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:17611980,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:529be2c9770add0cdd0c989115222ea9fc1be430c11095eb9f6dafcf98a36e2b k8s.gcr.io/sig-storage/livenessprobe:v2.4.0],SizeBytes:17175488,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:13616070e3f29de4417eee434a8ef472221c9e51b3d037b5a6b46cef08eb7443 k8s.gcr.io/e2e-test-images/nginx:1.14-2],SizeBytes:16032814,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nonewprivs@sha256:8ac1264691820febacf3aea5d152cbde6d10685731ec14966a9401c6f47a68ac k8s.gcr.io/e2e-test-images/nonewprivs:1.3],SizeBytes:7107254,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf k8s.gcr.io/e2e-test-images/busybox:1.29-2],SizeBytes:1154361,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:39e1e963e5310e9c313bad51523be012ede7b35bb9316517d19089a010356592 k8s.gcr.io/e2e-test-images/busybox:1.29-1],SizeBytes:1154361,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db k8s.gcr.io/pause:3.6 registry.k8s.io/pause:3.6],SizeBytes:682696,},},VolumesInUse:[kubernetes.io/csi/csi-mock-csi-mock-volumes-9402^4 kubernetes.io/csi/ebs.csi.aws.com^vol-02daff5e76e72db98],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-02daff5e76e72db98,DevicePath:,},},Config:nil,},} Jan 9 23:10:23.322: INFO: Logging kubelet events for node ip-172-20-57-251.ca-central-1.compute.internal Jan 9 23:10:23.356: INFO: Logging pods the kubelet thinks is on node ip-172-20-57-251.ca-central-1.compute.internal Jan 9 23:10:23.432: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-wvflq started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:10:23.432: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:10:23.432: INFO: hostexec-ip-172-20-57-251.ca-central-1.compute.internal-f9vsk started at 2023-01-09 23:08:31 +0000 UTC (0+1 container statuses recorded) Jan 9 23:10:23.432: INFO: Container agnhost-container ready: true, restart count 0 Jan 9 23:10:23.432: INFO: pod-subpath-test-preprovisionedpv-ssct started at 2023-01-09 23:08:48 +0000 UTC (0+2 container statuses recorded) Jan 9 23:10:23.432: INFO: Container test-container-subpath-preprovisionedpv-ssct ready: false, restart count 0 Jan 9 23:10:23.432: INFO: Container test-container-volume-preprovisionedpv-ssct ready: false, restart count 0 Jan 9 23:10:23.432: INFO: hostpath-symlink-prep-volume-6746 started at 2023-01-09 23:07:44 +0000 UTC (0+1 container statuses recorded) Jan 9 23:10:23.432: INFO: Container init-volume-volume-6746 ready: false, restart count 0 Jan 9 23:10:23.432: INFO: pvc-volume-tester-2vlf5 started at 2023-01-09 23:06:05 +0000 UTC (0+1 container statuses recorded) Jan 9 23:10:23.432: INFO: Container volume-tester ready: false, restart count 0 Jan 9 23:10:23.432: INFO: rs-56t9b started at 2023-01-09 22:59:56 +0000 UTC (0+1 container statuses recorded) Jan 9 23:10:23.432: INFO: Container donothing ready: true, restart count 0 Jan 9 23:10:23.432: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-tvxfq started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:10:23.432: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:10:23.432: INFO: csi-mockplugin-0 started at 2023-01-09 23:00:20 +0000 UTC (0+3 container statuses recorded) Jan 9 23:10:23.432: INFO: Container csi-provisioner ready: true, restart count 0 Jan 9 23:10:23.432: INFO: Container driver-registrar ready: true, restart count 0 Jan 9 23:10:23.432: INFO: Container mock ready: true, restart count 0 Jan 9 23:10:23.432: INFO: pod-d16d48d3-63ef-4a76-95b8-d80f8b4aec62 started at 2023-01-09 23:03:17 +0000 UTC (0+1 container statuses recorded) Jan 9 23:10:23.432: INFO: Container write-pod ready: false, restart count 0 Jan 9 23:10:23.432: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-m8cfz started at 2023-01-09 22:54:43 +0000 UTC (0+1 container statuses recorded) Jan 9 23:10:23.432: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:10:23.432: INFO: rs-49kqp started at 2023-01-09 22:59:56 +0000 UTC (0+1 container statuses recorded) Jan 9 23:10:23.432: INFO: Container donothing ready: false, restart count 0 Jan 9 23:10:23.432: INFO: rs-dzbwc started at 2023-01-09 22:59:56 +0000 UTC (0+1 container statuses recorded) Jan 9 23:10:23.432: INFO: Container donothing ready: false, restart count 0 Jan 9 23:10:23.432: INFO: ebs-csi-node-g4ngc started at 2023-01-09 22:38:46 +0000 UTC (0+3 container statuses recorded) Jan 9 23:10:23.432: INFO: Container ebs-plugin ready: true, restart count 0 Jan 9 23:10:23.432: INFO: Container liveness-probe ready: true, restart count 0 Jan 9 23:10:23.432: INFO: Container node-driver-registrar ready: true, restart count 0 Jan 9 23:10:23.432: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-wmqwf started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:10:23.432: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:10:23.432: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-96np5 started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:10:23.432: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:10:23.432: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-f9w29 started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:10:23.432: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:10:23.432: INFO: pause-pod-78b7f9b75c-fckd2 started at 2023-01-09 23:05:44 +0000 UTC (0+1 container statuses recorded) Jan 9 23:10:23.432: INFO: Container agnhost-pause ready: false, restart count 0 Jan 9 23:10:23.432: INFO: netserver-2 started at 2023-01-09 23:07:18 +0000 UTC (0+1 container statuses recorded) Jan 9 23:10:23.432: INFO: Container webserver ready: false, restart count 0 Jan 9 23:10:23.432: INFO: csi-mockplugin-attacher-0 started at 2023-01-09 23:00:20 +0000 UTC (0+1 container statuses recorded) Jan 9 23:10:23.432: INFO: Container csi-attacher ready: false, restart count 0 Jan 9 23:10:23.432: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-rbqs2 started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:10:23.432: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:10:23.432: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-qzf5t started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:10:23.432: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:10:23.432: INFO: busybox-scheduling-6815563e-4dc7-43f5-a3cd-3e998a31b84b started at 2023-01-09 23:06:39 +0000 UTC (0+1 container statuses recorded) Jan 9 23:10:23.432: INFO: Container busybox-scheduling-6815563e-4dc7-43f5-a3cd-3e998a31b84b ready: false, restart count 0 Jan 9 23:10:23.432: INFO: netserver-2 started at 2023-01-09 23:08:45 +0000 UTC (0+1 container statuses recorded) Jan 9 23:10:23.432: INFO: Container webserver ready: false, restart count 0 Jan 9 23:10:23.432: INFO: csi-mockplugin-0 started at 2023-01-09 23:05:43 +0000 UTC (0+3 container statuses recorded) Jan 9 23:10:23.432: INFO: Container csi-provisioner ready: false, restart count 0 Jan 9 23:10:23.432: INFO: Container driver-registrar ready: false, restart count 0 Jan 9 23:10:23.432: INFO: Container mock ready: false, restart count 0 Jan 9 23:10:23.432: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-zx8gm started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:10:23.432: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:10:23.432: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-zvfdw started at 2023-01-09 22:54:43 +0000 UTC (0+1 container statuses recorded) Jan 9 23:10:23.432: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:10:23.432: INFO: cilium-6p9jz started at 2023-01-09 22:38:46 +0000 UTC (1+1 container statuses recorded) Jan 9 23:10:23.432: INFO: Init container clean-cilium-state ready: true, restart count 0 Jan 9 23:10:23.432: INFO: Container cilium-agent ready: true, restart count 0 Jan 9 23:10:24.012: INFO: Latency metrics for node ip-172-20-57-251.ca-central-1.compute.internal Jan 9 23:10:24.012: INFO: Logging node info for node ip-172-20-59-208.ca-central-1.compute.internal Jan 9 23:10:24.045: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-59-208.ca-central-1.compute.internal d061a4d4-9ff6-43fc-89cc-422c61f7441c 45677 0 2023-01-09 22:38:43 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ca-central-1 failure-domain.beta.kubernetes.io/zone:ca-central-1a kubelet_cleanup:true kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-59-208.ca-central-1.compute.internal kubernetes.io/os:linux kubernetes.io/role:node node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:ca-central-1a topology.hostpath.csi/node:ip-172-20-59-208.ca-central-1.compute.internal topology.kubernetes.io/region:ca-central-1 topology.kubernetes.io/zone:ca-central-1a] map[csi.volume.kubernetes.io/nodeid:{"csi-hostpath-ephemeral-7223":"ip-172-20-59-208.ca-central-1.compute.internal","ebs.csi.aws.com":"i-0f00ea8bafbacdd11"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kops-controller Update v1 2023-01-09 22:38:43 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}} } {kube-controller-manager Update v1 2023-01-09 22:38:43 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.1.0/24\"":{}}}} } {kubelet Update v1 2023-01-09 22:38:43 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {e2e.test Update v1 2023-01-09 22:54:42 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubelet_cleanup":{}}}} } {kube-controller-manager Update v1 2023-01-09 23:06:36 +0000 UTC FieldsV1 {"f:status":{"f:volumesAttached":{}}} status} {kubelet Update v1 2023-01-09 23:06:40 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:capacity":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.1.0/24,DoNotUseExternalID:,ProviderID:aws:///ca-central-1a/i-0f00ea8bafbacdd11,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.1.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051652608 0} {<nil>} 3956692Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946795008 0} {<nil>} 3854292Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-01-09 23:06:40 +0000 UTC,LastTransitionTime:2023-01-09 22:38:43 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-01-09 23:06:40 +0000 UTC,LastTransitionTime:2023-01-09 22:38:43 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-01-09 23:06:40 +0000 UTC,LastTransitionTime:2023-01-09 22:38:43 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-01-09 23:06:40 +0000 UTC,LastTransitionTime:2023-01-09 22:39:03 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.59.208,},NodeAddress{Type:ExternalIP,Address:99.79.51.95,},NodeAddress{Type:Hostname,Address:ip-172-20-59-208.ca-central-1.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-59-208.ca-central-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-99-79-51-95.ca-central-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec28d5ecfae27d4bcedd89eca1a365f0,SystemUUID:ec28d5ec-fae2-7d4b-cedd-89eca1a365f0,BootID:560320f7-12c5-44f2-a7f6-a7a40031e204,KernelVersion:5.15.0-1026-aws,OSImage:Ubuntu 22.04.1 LTS,ContainerRuntimeVersion:docker://20.10.17,KubeletVersion:v1.23.15,KubeProxyVersion:v1.23.15,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:456569528,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:20f25f275d46aa728f7615a1ccc19c78b2ed89435bf943a44b339f70f45508e6 k8s.gcr.io/e2e-test-images/httpd:2.4.39-2],SizeBytes:126894770,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:7e8bdd271312fd25fc5ff5a8f04727be84044eb3d7d8d03611972a6752e2e11e k8s.gcr.io/e2e-test-images/agnhost:2.39],SizeBytes:126872991,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:1b9d1b2f36cb2dbee1960e82a9344aeb11bd4c4c03abf5e1853e0559c23855e3 k8s.gcr.io/e2e-test-images/httpd:2.4.38-2],SizeBytes:123781643,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.23.15 registry.k8s.io/kube-proxy-amd64:v1.23.15],SizeBytes:112336036,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:10e231cf07c82dfab5141c1afe548dc734e5fc1f67665ec3982a325d1bd31a9a registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.0],SizeBytes:94471559,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:c8e03f60afa90a28e4bb6ec9a8d0fc36d89de4b7475cf2d613afa793ec969fe0 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.0],SizeBytes:56382329,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:4e74c0492bceddc598de1c90cc5bc14dcda94cb49fa9c5bad9d117c4834b5e08 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.1],SizeBytes:56379269,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:36c31f7e1f433c9634d24f876353e8646246d81a03c4e351202c2644daff1620 k8s.gcr.io/sig-storage/csi-resizer:v1.2.0],SizeBytes:53956308,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:ed98431376c9e944e19a465fe8ea944806714dd95416a0821096c78d66b579bd k8s.gcr.io/sig-storage/csi-snapshotter:v4.1.1],SizeBytes:53531990,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:c5be65d6679efabb969d9b019300d187437ae876f992c40911fd2892bbef3b36 k8s.gcr.io/sig-storage/csi-attacher:v3.2.0],SizeBytes:53500826,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:60ab9b3e6a030d3038c87c0d6bca2930f58d1d72823e6a4af09767dc83b696a2 k8s.gcr.io/sig-storage/csi-attacher:v3.2.1],SizeBytes:53499570,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:017727efcfeb7d053af68e51436ce8e65edbc6ca573720afb4f79c8594036955 registry.k8s.io/coredns/coredns:v1.10.0],SizeBytes:50249337,},ContainerImage{Names:[registry.k8s.io/cpa/cluster-proportional-autoscaler@sha256:68d396900aeaa072c1f27289485fdac29834045a6f3ffe369bf389d830ef572d registry.k8s.io/cpa/cluster-proportional-autoscaler:1.8.6],SizeBytes:47910609,},ContainerImage{Names:[k8s.gcr.io/sig-storage/hostpathplugin@sha256:232fe80174d60d520d36043103853a1d7ab4b7f3782cf43e45034f04ccda58ce k8s.gcr.io/sig-storage/hostpathplugin:v1.7.1],SizeBytes:29977921,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:a61d309da54641db41fb8f35718f744e9f730d4d0384f8c4b186ddc9f06cbd5f k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.1.0],SizeBytes:19662887,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:19582112,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:2dee3fe5fe861bb66c3a4ac51114f3447a4cd35870e0f2e2b558c7a400d89589 k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.2.0],SizeBytes:18699436,},ContainerImage{Names:[k8s.gcr.io/sig-storage/mock-driver@sha256:a7b517f9e0f42ffade802eef9cefa271372386b85f55b702b493241e58459793 k8s.gcr.io/sig-storage/mock-driver:v4.1.0],SizeBytes:17680993,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:17611980,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:529be2c9770add0cdd0c989115222ea9fc1be430c11095eb9f6dafcf98a36e2b k8s.gcr.io/sig-storage/livenessprobe:v2.4.0],SizeBytes:17175488,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:13616070e3f29de4417eee434a8ef472221c9e51b3d037b5a6b46cef08eb7443 k8s.gcr.io/e2e-test-images/nginx:1.14-2],SizeBytes:16032814,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nonewprivs@sha256:8ac1264691820febacf3aea5d152cbde6d10685731ec14966a9401c6f47a68ac k8s.gcr.io/e2e-test-images/nonewprivs:1.3],SizeBytes:7107254,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf k8s.gcr.io/e2e-test-images/busybox:1.29-2],SizeBytes:1154361,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:39e1e963e5310e9c313bad51523be012ede7b35bb9316517d19089a010356592 k8s.gcr.io/e2e-test-images/busybox:1.29-1],SizeBytes:1154361,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db k8s.gcr.io/pause:3.6 registry.k8s.io/pause:3.6],SizeBytes:682696,},},VolumesInUse:[kubernetes.io/csi/csi-hostpath-ephemeral-7223^43b8adc9-9072-11ed-b4b6-0a454e7330ed kubernetes.io/csi/csi-hostpath-ephemeral-7223^43b8d4d5-9072-11ed-b4b6-0a454e7330ed],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/csi-hostpath-ephemeral-7223^43b8d4d5-9072-11ed-b4b6-0a454e7330ed,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/csi-hostpath-ephemeral-7223^43b8adc9-9072-11ed-b4b6-0a454e7330ed,DevicePath:,},},Config:nil,},} Jan 9 23:10:24.046: INFO: Logging kubelet events for node ip-172-20-59-208.ca-central-1.compute.internal Jan 9 23:10:24.084: INFO: Logging pods the kubelet thinks is on node ip-172-20-59-208.ca-central-1.compute.internal Jan 9 23:10:24.133: INFO: coredns-867df8f45c-tmrqb started at 2023-01-09 22:39:03 +0000 UTC (0+1 container statuses recorded) Jan 9 23:10:24.133: INFO: Container coredns ready: true, restart count 0 Jan 9 23:10:24.133: INFO: netserver-3 started at 2023-01-09 23:08:45 +0000 UTC (0+1 container statuses recorded) Jan 9 23:10:24.133: INFO: Container webserver ready: false, restart count 0 Jan 9 23:10:24.133: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-fc6pg started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:10:24.133: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:10:24.133: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-8qp6l started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:10:24.133: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: false, restart count 0 Jan 9 23:10:24.133: INFO: rs-nd6n2 started at 2023-01-09 22:59:56 +0000 UTC (0+1 container statuses recorded) Jan 9 23:10:24.133: INFO: Container donothing ready: false, restart count 0 Jan 9 23:10:24.133: INFO: coredns-autoscaler-557ccb4c66-jnjcc started at 2023-01-09 22:39:03 +0000 UTC (0+1 container statuses recorded) Jan 9 23:10:24.133: INFO: Container autoscaler ready: true, restart count 0 Jan 9 23:10:24.133: INFO: netserver-3 started at 2023-01-09 23:07:18 +0000 UTC (0+1 container statuses recorded) Jan 9 23:10:24.133: INFO: Container webserver ready: false, restart count 0 Jan 9 23:10:24.133: INFO: rs-x69wn started at 2023-01-09 22:59:56 +0000 UTC (0+1 container statuses recorded) Jan 9 23:10:24.133: INFO: Container donothing ready: false, restart count 0 Jan 9 23:10:24.133: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-hszdf started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:10:24.133: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:10:24.133: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-bhxkw started at 2023-01-09 22:54:43 +0000 UTC (0+1 container statuses recorded) Jan 9 23:10:24.133: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:10:24.133: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-hjsbl started at 2023-01-09 22:54:43 +0000 UTC (0+1 container statuses recorded) Jan 9 23:10:24.133: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:10:24.133: INFO: pod-e1224014-09b8-42e1-b210-71a9acc98e21 started at 2023-01-09 23:06:29 +0000 UTC (0+1 container statuses recorded) Jan 9 23:10:24.133: INFO: Container write-pod ready: false, restart count 0 Jan 9 23:10:24.133: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-rzgjf started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:10:24.133: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:10:24.133: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-ptfgl started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:10:24.133: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:10:24.133: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-t5g85 started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:10:24.133: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:10:24.133: INFO: ebs-csi-node-prrrf started at 2023-01-09 22:38:44 +0000 UTC (0+3 container statuses recorded) Jan 9 23:10:24.133: INFO: Container ebs-plugin ready: true, restart count 0 Jan 9 23:10:24.133: INFO: Container liveness-probe ready: true, restart count 0 Jan 9 23:10:24.133: INFO: Container node-driver-registrar ready: true, restart count 0 Jan 9 23:10:24.133: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-x682f started at 2023-01-09 22:54:43 +0000 UTC (0+1 container statuses recorded) Jan 9 23:10:24.133: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:10:24.134: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-tx2mm started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:10:24.134: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:10:24.134: INFO: cilium-z2ltf started at 2023-01-09 22:38:44 +0000 UTC (1+1 container statuses recorded) Jan 9 23:10:24.134: INFO: Init container clean-cilium-state ready: true, restart count 0 Jan 9 23:10:24.134: INFO: Container cilium-agent ready: true, restart count 0 Jan 9 23:10:24.134: INFO: csi-hostpathplugin-0 started at 2023-01-09 23:03:05 +0000 UTC (0+7 container statuses recorded) Jan 9 23:10:24.134: INFO: Container csi-attacher ready: true, restart count 0 Jan 9 23:10:24.134: INFO: Container csi-provisioner ready: true, restart count 0 Jan 9 23:10:24.134: INFO: Container csi-resizer ready: true, restart count 0 Jan 9 23:10:24.134: INFO: Container csi-snapshotter ready: true, restart count 0 Jan 9 23:10:24.134: INFO: Container hostpath ready: true, restart count 0 Jan 9 23:10:24.134: INFO: Container liveness-probe ready: true, restart count 0 Jan 9 23:10:24.134: INFO: Container node-driver-registrar ready: true, restart count 0 Jan 9 23:10:24.134: INFO: hostexec-ip-172-20-59-208.ca-central-1.compute.internal-sn7w7 started at 2023-01-09 23:06:26 +0000 UTC (0+1 container statuses recorded) Jan 9 23:10:24.134: INFO: Container agnhost-container ready: true, restart count 0 Jan 9 23:10:24.134: INFO: inline-volume-tester-xnqfb started at 2023-01-09 23:06:35 +0000 UTC (0+1 container statuses recorded) Jan 9 23:10:24.134: INFO: Container csi-volume-tester ready: false, restart count 0 Jan 9 23:10:24.557: INFO: Latency metrics for node ip-172-20-59-208.ca-central-1.compute.internal Jan 9 23:10:24.557: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready �[1mSTEP�[0m: Destroying namespace "replicaset-2552" for this suite.
Filter through log files
go run hack/e2e.go -v --test --test_args='--ginkgo.focus=Kubernetes\se2e\ssuite\s\[sig\-apps\]\sReplicationController\sshould\sserve\sa\sbasic\simage\son\seach\sreplica\swith\sa\spublic\simage\s\s\[Conformance\]$'
/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:633 Jan 9 23:00:09.165: Unexpected error: <*errors.errorString | 0xc005999270>: { s: "pod \"my-hostname-basic-e6b46933-d037-4e81-8a71-4b04c21b3646-rqpsj\" never run (phase: Pending, conditions: [{Type:Initialized Status:True LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2023-01-09 22:55:09 +0000 UTC Reason: Message:} {Type:Ready Status:False LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2023-01-09 22:55:09 +0000 UTC Reason:ContainersNotReady Message:containers with unready status: [my-hostname-basic-e6b46933-d037-4e81-8a71-4b04c21b3646]} {Type:ContainersReady Status:False LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2023-01-09 22:55:09 +0000 UTC Reason:ContainersNotReady Message:containers with unready status: [my-hostname-basic-e6b46933-d037-4e81-8a71-4b04c21b3646]} {Type:PodScheduled Status:True LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2023-01-09 22:55:09 +0000 UTC Reason: Message:}]): timed out waiting for the condition", } pod "my-hostname-basic-e6b46933-d037-4e81-8a71-4b04c21b3646-rqpsj" never run (phase: Pending, conditions: [{Type:Initialized Status:True LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2023-01-09 22:55:09 +0000 UTC Reason: Message:} {Type:Ready Status:False LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2023-01-09 22:55:09 +0000 UTC Reason:ContainersNotReady Message:containers with unready status: [my-hostname-basic-e6b46933-d037-4e81-8a71-4b04c21b3646]} {Type:ContainersReady Status:False LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2023-01-09 22:55:09 +0000 UTC Reason:ContainersNotReady Message:containers with unready status: [my-hostname-basic-e6b46933-d037-4e81-8a71-4b04c21b3646]} {Type:PodScheduled Status:True LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2023-01-09 22:55:09 +0000 UTC Reason: Message:}]): timed out waiting for the condition occurred /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/apps/rc.go:459from junit_25.xml
[BeforeEach] [sig-apps] ReplicationController /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:185 �[1mSTEP�[0m: Creating a kubernetes client Jan 9 22:55:08.755: INFO: >>> kubeConfig: /root/.kube/config �[1mSTEP�[0m: Building a namespace api object, basename replication-controller �[1mSTEP�[0m: Waiting for a default service account to be provisioned in namespace �[1mSTEP�[0m: Waiting for kube-root-ca.crt to be provisioned in namespace [BeforeEach] [sig-apps] ReplicationController /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/apps/rc.go:54 [It] should serve a basic image on each replica with a public image [Conformance] /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:633 �[1mSTEP�[0m: Creating replication controller my-hostname-basic-e6b46933-d037-4e81-8a71-4b04c21b3646 Jan 9 22:55:09.037: INFO: Pod name my-hostname-basic-e6b46933-d037-4e81-8a71-4b04c21b3646: Found 1 pods out of 1 Jan 9 22:55:09.038: INFO: Ensuring all pods for ReplicationController "my-hostname-basic-e6b46933-d037-4e81-8a71-4b04c21b3646" are running Jan 9 23:00:09.165: FAIL: Unexpected error: <*errors.errorString | 0xc005999270>: { s: "pod \"my-hostname-basic-e6b46933-d037-4e81-8a71-4b04c21b3646-rqpsj\" never run (phase: Pending, conditions: [{Type:Initialized Status:True LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2023-01-09 22:55:09 +0000 UTC Reason: Message:} {Type:Ready Status:False LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2023-01-09 22:55:09 +0000 UTC Reason:ContainersNotReady Message:containers with unready status: [my-hostname-basic-e6b46933-d037-4e81-8a71-4b04c21b3646]} {Type:ContainersReady Status:False LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2023-01-09 22:55:09 +0000 UTC Reason:ContainersNotReady Message:containers with unready status: [my-hostname-basic-e6b46933-d037-4e81-8a71-4b04c21b3646]} {Type:PodScheduled Status:True LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2023-01-09 22:55:09 +0000 UTC Reason: Message:}]): timed out waiting for the condition", } pod "my-hostname-basic-e6b46933-d037-4e81-8a71-4b04c21b3646-rqpsj" never run (phase: Pending, conditions: [{Type:Initialized Status:True LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2023-01-09 22:55:09 +0000 UTC Reason: Message:} {Type:Ready Status:False LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2023-01-09 22:55:09 +0000 UTC Reason:ContainersNotReady Message:containers with unready status: [my-hostname-basic-e6b46933-d037-4e81-8a71-4b04c21b3646]} {Type:ContainersReady Status:False LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2023-01-09 22:55:09 +0000 UTC Reason:ContainersNotReady Message:containers with unready status: [my-hostname-basic-e6b46933-d037-4e81-8a71-4b04c21b3646]} {Type:PodScheduled Status:True LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2023-01-09 22:55:09 +0000 UTC Reason: Message:}]): timed out waiting for the condition occurred Full Stack Trace k8s.io/kubernetes/test/e2e/apps.TestReplicationControllerServeImageOrFail(0xc0000da2c0, {0x70cf457, 0x5}, {0xc0003038c0, 0x27}) /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/apps/rc.go:459 +0xba5 k8s.io/kubernetes/test/e2e/apps.glob..func7.2() /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/apps/rc.go:65 +0x37 k8s.io/kubernetes/test/e2e.RunE2ETests(0x24c66d7) _output/dockerized/go/src/k8s.io/kubernetes/test/e2e/e2e.go:133 +0x697 k8s.io/kubernetes/test/e2e.TestE2E(0x243a8f9) _output/dockerized/go/src/k8s.io/kubernetes/test/e2e/e2e_test.go:136 +0x19 testing.tRunner(0xc000d12b60, 0x735e880) /usr/local/go/src/testing/testing.go:1259 +0x102 created by testing.(*T).Run /usr/local/go/src/testing/testing.go:1306 +0x35a [AfterEach] [sig-apps] ReplicationController /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:186 �[1mSTEP�[0m: Collecting events from namespace "replication-controller-5218". �[1mSTEP�[0m: Found 13 events. Jan 9 23:00:09.199: INFO: At 2023-01-09 22:55:08 +0000 UTC - event for my-hostname-basic-e6b46933-d037-4e81-8a71-4b04c21b3646: {replication-controller } SuccessfulCreate: Created pod: my-hostname-basic-e6b46933-d037-4e81-8a71-4b04c21b3646-rqpsj Jan 9 23:00:09.199: INFO: At 2023-01-09 22:55:09 +0000 UTC - event for my-hostname-basic-e6b46933-d037-4e81-8a71-4b04c21b3646-rqpsj: {kubelet ip-172-20-55-216.ca-central-1.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "80ae00f1871d464c2028340b1fd0d268d9b83fbe7118107bbf46a9f3b95dca9f" network for pod "my-hostname-basic-e6b46933-d037-4e81-8a71-4b04c21b3646-rqpsj": networkPlugin cni failed to set up pod "my-hostname-basic-e6b46933-d037-4e81-8a71-4b04c21b3646-rqpsj_replication-controller-5218" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 9 23:00:09.199: INFO: At 2023-01-09 22:55:09 +0000 UTC - event for my-hostname-basic-e6b46933-d037-4e81-8a71-4b04c21b3646-rqpsj: {default-scheduler } Scheduled: Successfully assigned replication-controller-5218/my-hostname-basic-e6b46933-d037-4e81-8a71-4b04c21b3646-rqpsj to ip-172-20-55-216.ca-central-1.compute.internal Jan 9 23:00:09.199: INFO: At 2023-01-09 22:55:10 +0000 UTC - event for my-hostname-basic-e6b46933-d037-4e81-8a71-4b04c21b3646-rqpsj: {kubelet ip-172-20-55-216.ca-central-1.compute.internal} SandboxChanged: Pod sandbox changed, it will be killed and re-created. Jan 9 23:00:09.199: INFO: At 2023-01-09 22:55:12 +0000 UTC - event for my-hostname-basic-e6b46933-d037-4e81-8a71-4b04c21b3646-rqpsj: {kubelet ip-172-20-55-216.ca-central-1.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "b8489362744f0ab18bd99a05fd51093c5fe4a2327530c9996a6c77e9cdd52043" network for pod "my-hostname-basic-e6b46933-d037-4e81-8a71-4b04c21b3646-rqpsj": networkPlugin cni failed to set up pod "my-hostname-basic-e6b46933-d037-4e81-8a71-4b04c21b3646-rqpsj_replication-controller-5218" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 9 23:00:09.199: INFO: At 2023-01-09 22:55:14 +0000 UTC - event for my-hostname-basic-e6b46933-d037-4e81-8a71-4b04c21b3646-rqpsj: {kubelet ip-172-20-55-216.ca-central-1.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "6d966de595ef2281f41a57dc098d239508f74374f4a0cda6317613c7bab3370f" network for pod "my-hostname-basic-e6b46933-d037-4e81-8a71-4b04c21b3646-rqpsj": networkPlugin cni failed to set up pod "my-hostname-basic-e6b46933-d037-4e81-8a71-4b04c21b3646-rqpsj_replication-controller-5218" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 9 23:00:09.199: INFO: At 2023-01-09 22:55:15 +0000 UTC - event for my-hostname-basic-e6b46933-d037-4e81-8a71-4b04c21b3646-rqpsj: {kubelet ip-172-20-55-216.ca-central-1.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "e7ab2e33625371f81be3c7350d04038fcd76a1755ecaa2c9e4a76e81b96f8194" network for pod "my-hostname-basic-e6b46933-d037-4e81-8a71-4b04c21b3646-rqpsj": networkPlugin cni failed to set up pod "my-hostname-basic-e6b46933-d037-4e81-8a71-4b04c21b3646-rqpsj_replication-controller-5218" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 9 23:00:09.199: INFO: At 2023-01-09 22:55:17 +0000 UTC - event for my-hostname-basic-e6b46933-d037-4e81-8a71-4b04c21b3646-rqpsj: {kubelet ip-172-20-55-216.ca-central-1.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "4979029ac1dc4e724ecc1c3cb9208ca9b26bd01f2a939efdfa011d4d19c9b402" network for pod "my-hostname-basic-e6b46933-d037-4e81-8a71-4b04c21b3646-rqpsj": networkPlugin cni failed to set up pod "my-hostname-basic-e6b46933-d037-4e81-8a71-4b04c21b3646-rqpsj_replication-controller-5218" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 9 23:00:09.199: INFO: At 2023-01-09 22:55:19 +0000 UTC - event for my-hostname-basic-e6b46933-d037-4e81-8a71-4b04c21b3646-rqpsj: {kubelet ip-172-20-55-216.ca-central-1.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "b02c5cf421d8eaffacdfdfe4e1076bbbd9833eb5a3c9eeb1682ecb4aea0f9b4f" network for pod "my-hostname-basic-e6b46933-d037-4e81-8a71-4b04c21b3646-rqpsj": networkPlugin cni failed to set up pod "my-hostname-basic-e6b46933-d037-4e81-8a71-4b04c21b3646-rqpsj_replication-controller-5218" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 9 23:00:09.199: INFO: At 2023-01-09 22:55:22 +0000 UTC - event for my-hostname-basic-e6b46933-d037-4e81-8a71-4b04c21b3646-rqpsj: {kubelet ip-172-20-55-216.ca-central-1.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "34f63de82a26c85a9551facac641d1e3c87ee6e26c1c46d5f28bca93a7f9e81e" network for pod "my-hostname-basic-e6b46933-d037-4e81-8a71-4b04c21b3646-rqpsj": networkPlugin cni failed to set up pod "my-hostname-basic-e6b46933-d037-4e81-8a71-4b04c21b3646-rqpsj_replication-controller-5218" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 9 23:00:09.199: INFO: At 2023-01-09 22:55:25 +0000 UTC - event for my-hostname-basic-e6b46933-d037-4e81-8a71-4b04c21b3646-rqpsj: {kubelet ip-172-20-55-216.ca-central-1.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "aff793775419b0e5475c418b7dc759ddbc2a95177bd486146130693451199652" network for pod "my-hostname-basic-e6b46933-d037-4e81-8a71-4b04c21b3646-rqpsj": networkPlugin cni failed to set up pod "my-hostname-basic-e6b46933-d037-4e81-8a71-4b04c21b3646-rqpsj_replication-controller-5218" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 9 23:00:09.199: INFO: At 2023-01-09 22:55:29 +0000 UTC - event for my-hostname-basic-e6b46933-d037-4e81-8a71-4b04c21b3646-rqpsj: {kubelet ip-172-20-55-216.ca-central-1.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "08ca155dcacb08cc83bd974de6465c3a2826bfa2a0fc8a669b16e482aa181995" network for pod "my-hostname-basic-e6b46933-d037-4e81-8a71-4b04c21b3646-rqpsj": networkPlugin cni failed to set up pod "my-hostname-basic-e6b46933-d037-4e81-8a71-4b04c21b3646-rqpsj_replication-controller-5218" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 9 23:00:09.199: INFO: At 2023-01-09 22:55:42 +0000 UTC - event for my-hostname-basic-e6b46933-d037-4e81-8a71-4b04c21b3646-rqpsj: {kubelet ip-172-20-55-216.ca-central-1.compute.internal} FailedCreatePodSandBox: (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "b262e0f669fbb2fa2209f5a370cb1d75c8ecf9382810c5aa6df93e1a84554be4" network for pod "my-hostname-basic-e6b46933-d037-4e81-8a71-4b04c21b3646-rqpsj": networkPlugin cni failed to set up pod "my-hostname-basic-e6b46933-d037-4e81-8a71-4b04c21b3646-rqpsj_replication-controller-5218" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 9 23:00:09.231: INFO: POD NODE PHASE GRACE CONDITIONS Jan 9 23:00:09.231: INFO: my-hostname-basic-e6b46933-d037-4e81-8a71-4b04c21b3646-rqpsj ip-172-20-55-216.ca-central-1.compute.internal Pending [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2023-01-09 22:55:09 +0000 UTC } {Ready False 0001-01-01 00:00:00 +0000 UTC 2023-01-09 22:55:09 +0000 UTC ContainersNotReady containers with unready status: [my-hostname-basic-e6b46933-d037-4e81-8a71-4b04c21b3646]} {ContainersReady False 0001-01-01 00:00:00 +0000 UTC 2023-01-09 22:55:09 +0000 UTC ContainersNotReady containers with unready status: [my-hostname-basic-e6b46933-d037-4e81-8a71-4b04c21b3646]} {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2023-01-09 22:55:09 +0000 UTC }] Jan 9 23:00:09.231: INFO: Jan 9 23:00:09.268: INFO: Unable to fetch replication-controller-5218/my-hostname-basic-e6b46933-d037-4e81-8a71-4b04c21b3646-rqpsj/my-hostname-basic-e6b46933-d037-4e81-8a71-4b04c21b3646 logs: the server rejected our request for an unknown reason (get pods my-hostname-basic-e6b46933-d037-4e81-8a71-4b04c21b3646-rqpsj) Jan 9 23:00:09.300: INFO: Logging node info for node ip-172-20-32-67.ca-central-1.compute.internal Jan 9 23:00:09.332: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-32-67.ca-central-1.compute.internal 1f51981b-a01d-404c-b3a4-6649f301f65d 41447 0 2023-01-09 22:38:45 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ca-central-1 failure-domain.beta.kubernetes.io/zone:ca-central-1a kubelet_cleanup:true kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-32-67.ca-central-1.compute.internal kubernetes.io/os:linux kubernetes.io/role:node node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:ca-central-1a topology.hostpath.csi/node:ip-172-20-32-67.ca-central-1.compute.internal topology.kubernetes.io/region:ca-central-1 topology.kubernetes.io/zone:ca-central-1a] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-0272cebea4c6c3d5d"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kops-controller Update v1 2023-01-09 22:38:45 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}} } {kube-controller-manager Update v1 2023-01-09 22:38:45 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.3.0/24\"":{}}}} } {kubelet Update v1 2023-01-09 22:38:45 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {e2e.test Update v1 2023-01-09 22:54:42 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubelet_cleanup":{}}}} } {kube-controller-manager Update v1 2023-01-09 22:55:59 +0000 UTC FieldsV1 {"f:status":{"f:volumesAttached":{}}} status} {kubelet Update v1 2023-01-09 22:56:00 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:capacity":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.3.0/24,DoNotUseExternalID:,ProviderID:aws:///ca-central-1a/i-0272cebea4c6c3d5d,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.3.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051644416 0} {<nil>} 3956684Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946786816 0} {<nil>} 3854284Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-01-09 22:56:00 +0000 UTC,LastTransitionTime:2023-01-09 22:38:45 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-01-09 22:56:00 +0000 UTC,LastTransitionTime:2023-01-09 22:38:45 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-01-09 22:56:00 +0000 UTC,LastTransitionTime:2023-01-09 22:38:45 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-01-09 22:56:00 +0000 UTC,LastTransitionTime:2023-01-09 22:39:06 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.32.67,},NodeAddress{Type:ExternalIP,Address:35.182.2.102,},NodeAddress{Type:Hostname,Address:ip-172-20-32-67.ca-central-1.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-32-67.ca-central-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-35-182-2-102.ca-central-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2536b48bd6b6dade0ed68af2555085,SystemUUID:ec2536b4-8bd6-b6da-de0e-d68af2555085,BootID:bfb3b0df-aafa-481c-9ac6-f85e5e4580e3,KernelVersion:5.15.0-1026-aws,OSImage:Ubuntu 22.04.1 LTS,ContainerRuntimeVersion:docker://20.10.17,KubeletVersion:v1.23.15,KubeProxyVersion:v1.23.15,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:456569528,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/jessie-dnsutils@sha256:11e6a66017ba4e4b938c1612b7a54a3befcefd354796c04e1dba76873a13518e k8s.gcr.io/e2e-test-images/jessie-dnsutils:1.5],SizeBytes:253371792,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:7e8bdd271312fd25fc5ff5a8f04727be84044eb3d7d8d03611972a6752e2e11e k8s.gcr.io/e2e-test-images/agnhost:2.39],SizeBytes:126872991,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nautilus@sha256:99c0d6f1ad24a1aa1905d9c6534d193f268f7b23f9add2ae6bb41f31094bdd5c k8s.gcr.io/e2e-test-images/nautilus:1.5],SizeBytes:124758741,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:1b9d1b2f36cb2dbee1960e82a9344aeb11bd4c4c03abf5e1853e0559c23855e3 k8s.gcr.io/e2e-test-images/httpd:2.4.38-2],SizeBytes:123781643,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.23.15 registry.k8s.io/kube-proxy-amd64:v1.23.15],SizeBytes:112336036,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:10e231cf07c82dfab5141c1afe548dc734e5fc1f67665ec3982a325d1bd31a9a registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.0],SizeBytes:94471559,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:c8e03f60afa90a28e4bb6ec9a8d0fc36d89de4b7475cf2d613afa793ec969fe0 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.0],SizeBytes:56382329,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:4e74c0492bceddc598de1c90cc5bc14dcda94cb49fa9c5bad9d117c4834b5e08 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.1],SizeBytes:56379269,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:36c31f7e1f433c9634d24f876353e8646246d81a03c4e351202c2644daff1620 k8s.gcr.io/sig-storage/csi-resizer:v1.2.0],SizeBytes:53956308,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:ed98431376c9e944e19a465fe8ea944806714dd95416a0821096c78d66b579bd k8s.gcr.io/sig-storage/csi-snapshotter:v4.1.1],SizeBytes:53531990,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:c5be65d6679efabb969d9b019300d187437ae876f992c40911fd2892bbef3b36 k8s.gcr.io/sig-storage/csi-attacher:v3.2.0],SizeBytes:53500826,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:60ab9b3e6a030d3038c87c0d6bca2930f58d1d72823e6a4af09767dc83b696a2 k8s.gcr.io/sig-storage/csi-attacher:v3.2.1],SizeBytes:53499570,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:017727efcfeb7d053af68e51436ce8e65edbc6ca573720afb4f79c8594036955 registry.k8s.io/coredns/coredns:v1.10.0],SizeBytes:50249337,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:7a5ba58a44e0d749e0767e4e37315bcf6a61f33ce3185c1991848af4db0fb70a k8s.gcr.io/sig-storage/csi-resizer:v1.1.0],SizeBytes:49176472,},ContainerImage{Names:[k8s.gcr.io/sig-storage/hostpathplugin@sha256:232fe80174d60d520d36043103853a1d7ab4b7f3782cf43e45034f04ccda58ce k8s.gcr.io/sig-storage/hostpathplugin:v1.7.1],SizeBytes:29977921,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:a61d309da54641db41fb8f35718f744e9f730d4d0384f8c4b186ddc9f06cbd5f k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.1.0],SizeBytes:19662887,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:19582112,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:2dee3fe5fe861bb66c3a4ac51114f3447a4cd35870e0f2e2b558c7a400d89589 k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.2.0],SizeBytes:18699436,},ContainerImage{Names:[k8s.gcr.io/sig-storage/mock-driver@sha256:a7b517f9e0f42ffade802eef9cefa271372386b85f55b702b493241e58459793 k8s.gcr.io/sig-storage/mock-driver:v4.1.0],SizeBytes:17680993,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:17611980,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:529be2c9770add0cdd0c989115222ea9fc1be430c11095eb9f6dafcf98a36e2b k8s.gcr.io/sig-storage/livenessprobe:v2.4.0],SizeBytes:17175488,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:13616070e3f29de4417eee434a8ef472221c9e51b3d037b5a6b46cef08eb7443 k8s.gcr.io/e2e-test-images/nginx:1.14-2],SizeBytes:16032814,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nonewprivs@sha256:8ac1264691820febacf3aea5d152cbde6d10685731ec14966a9401c6f47a68ac k8s.gcr.io/e2e-test-images/nonewprivs:1.3],SizeBytes:7107254,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf k8s.gcr.io/e2e-test-images/busybox:1.29-2],SizeBytes:1154361,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:39e1e963e5310e9c313bad51523be012ede7b35bb9316517d19089a010356592 k8s.gcr.io/e2e-test-images/busybox:1.29-1],SizeBytes:1154361,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db k8s.gcr.io/pause:3.6 registry.k8s.io/pause:3.6],SizeBytes:682696,},},VolumesInUse:[kubernetes.io/csi/ebs.csi.aws.com^vol-01e8a4a8ff1da0e10],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-01e8a4a8ff1da0e10,DevicePath:,},},Config:nil,},} Jan 9 23:00:09.333: INFO: Logging kubelet events for node ip-172-20-32-67.ca-central-1.compute.internal Jan 9 23:00:09.368: INFO: Logging pods the kubelet thinks is on node ip-172-20-32-67.ca-central-1.compute.internal Jan 9 23:00:09.409: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-mlcl4 started at 2023-01-09 22:54:43 +0000 UTC (0+1 container statuses recorded) Jan 9 23:00:09.409: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:00:09.409: INFO: security-context-e67b5208-ab97-4fd9-a5df-beeeec3d467f started at 2023-01-09 22:57:35 +0000 UTC (0+1 container statuses recorded) Jan 9 23:00:09.409: INFO: Container test-container ready: false, restart count 0 Jan 9 23:00:09.409: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-fzrxx started at 2023-01-09 22:54:43 +0000 UTC (0+1 container statuses recorded) Jan 9 23:00:09.409: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:00:09.409: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-5pr42 started at 2023-01-09 22:54:43 +0000 UTC (0+1 container statuses recorded) Jan 9 23:00:09.409: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:00:09.409: INFO: ebs-csi-node-lc4rb started at 2023-01-09 22:38:46 +0000 UTC (0+3 container statuses recorded) Jan 9 23:00:09.409: INFO: Container ebs-plugin ready: true, restart count 0 Jan 9 23:00:09.409: INFO: Container liveness-probe ready: true, restart count 0 Jan 9 23:00:09.409: INFO: Container node-driver-registrar ready: true, restart count 0 Jan 9 23:00:09.409: INFO: rs-gxqdg started at 2023-01-09 22:59:56 +0000 UTC (0+1 container statuses recorded) Jan 9 23:00:09.409: INFO: Container donothing ready: false, restart count 0 Jan 9 23:00:09.409: INFO: hostexec-ip-172-20-32-67.ca-central-1.compute.internal-l4bzx started at 2023-01-09 22:57:44 +0000 UTC (0+1 container statuses recorded) Jan 9 23:00:09.409: INFO: Container agnhost-container ready: true, restart count 0 Jan 9 23:00:09.409: INFO: inline-volume-tester-5g6j6 started at 2023-01-09 22:55:56 +0000 UTC (0+1 container statuses recorded) Jan 9 23:00:09.409: INFO: Container csi-volume-tester ready: false, restart count 0 Jan 9 23:00:09.409: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-sddjv started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:00:09.409: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:00:09.409: INFO: busybox-c48a8654-4481-4f1d-8e36-a143d7ab957d started at 2023-01-09 22:55:53 +0000 UTC (0+1 container statuses recorded) Jan 9 23:00:09.409: INFO: Container busybox ready: false, restart count 0 Jan 9 23:00:09.409: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-8vzqv started at 2023-01-09 22:54:43 +0000 UTC (0+1 container statuses recorded) Jan 9 23:00:09.409: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:00:09.409: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-jb6kp started at 2023-01-09 22:54:43 +0000 UTC (0+1 container statuses recorded) Jan 9 23:00:09.409: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:00:09.409: INFO: cilium-dvsfs started at 2023-01-09 22:38:46 +0000 UTC (1+1 container statuses recorded) Jan 9 23:00:09.409: INFO: Init container clean-cilium-state ready: true, restart count 0 Jan 9 23:00:09.409: INFO: Container cilium-agent ready: true, restart count 0 Jan 9 23:00:09.409: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-gjc2h started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:00:09.409: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:00:09.409: INFO: rs-zn5kz started at 2023-01-09 22:59:56 +0000 UTC (0+1 container statuses recorded) Jan 9 23:00:09.409: INFO: Container donothing ready: false, restart count 0 Jan 9 23:00:09.409: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-hwbbn started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:00:09.409: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:00:09.409: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-fr98x started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:00:09.409: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:00:09.409: INFO: coredns-867df8f45c-sckrz started at 2023-01-09 22:39:23 +0000 UTC (0+1 container statuses recorded) Jan 9 23:00:09.409: INFO: Container coredns ready: true, restart count 0 Jan 9 23:00:09.409: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-44ccv started at 2023-01-09 22:54:43 +0000 UTC (0+1 container statuses recorded) Jan 9 23:00:09.409: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:00:09.409: INFO: netserver-0 started at 2023-01-09 22:54:37 +0000 UTC (0+1 container statuses recorded) Jan 9 23:00:09.409: INFO: Container webserver ready: true, restart count 0 Jan 9 23:00:09.409: INFO: pod-subpath-test-preprovisionedpv-cmgx started at 2023-01-09 22:57:48 +0000 UTC (2+1 container statuses recorded) Jan 9 23:00:09.409: INFO: Init container init-volume-preprovisionedpv-cmgx ready: false, restart count 0 Jan 9 23:00:09.409: INFO: Init container test-init-subpath-preprovisionedpv-cmgx ready: false, restart count 0 Jan 9 23:00:09.409: INFO: Container test-container-subpath-preprovisionedpv-cmgx ready: false, restart count 0 Jan 9 23:00:10.034: INFO: Latency metrics for node ip-172-20-32-67.ca-central-1.compute.internal Jan 9 23:00:10.034: INFO: Logging node info for node ip-172-20-37-64.ca-central-1.compute.internal Jan 9 23:00:10.065: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-37-64.ca-central-1.compute.internal d817613a-0fe5-40c0-a5e4-7b85b9f8506b 42613 0 2023-01-09 22:36:54 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:c5.large beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ca-central-1 failure-domain.beta.kubernetes.io/zone:ca-central-1a kops.k8s.io/kops-controller-pki: kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-37-64.ca-central-1.compute.internal kubernetes.io/os:linux kubernetes.io/role:master node-role.kubernetes.io/control-plane: node-role.kubernetes.io/master: node.kubernetes.io/exclude-from-external-load-balancers: node.kubernetes.io/instance-type:c5.large topology.ebs.csi.aws.com/zone:ca-central-1a topology.kubernetes.io/region:ca-central-1 topology.kubernetes.io/zone:ca-central-1a] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-06217673764fdafeb"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kubelet Update v1 2023-01-09 22:36:54 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {protokube Update v1 2023-01-09 22:37:19 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kops.k8s.io/kops-controller-pki":{},"f:node-role.kubernetes.io/control-plane":{},"f:node.kubernetes.io/exclude-from-external-load-balancers":{}}}} } {kops-controller Update v1 2023-01-09 22:37:21 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/master":{}}}} } {kube-controller-manager Update v1 2023-01-09 22:37:35 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.0.0/24\"":{}},"f:taints":{}}} } {kubelet Update v1 2023-01-09 22:38:39 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.0.0/24,DoNotUseExternalID:,ProviderID:aws:///ca-central-1a/i-06217673764fdafeb,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/master,Value:,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[100.96.0.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3862904832 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3758047232 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-01-09 22:59:10 +0000 UTC,LastTransitionTime:2023-01-09 22:36:48 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-01-09 22:59:10 +0000 UTC,LastTransitionTime:2023-01-09 22:36:48 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-01-09 22:59:10 +0000 UTC,LastTransitionTime:2023-01-09 22:36:48 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-01-09 22:59:10 +0000 UTC,LastTransitionTime:2023-01-09 22:37:35 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.37.64,},NodeAddress{Type:ExternalIP,Address:3.96.216.165,},NodeAddress{Type:Hostname,Address:ip-172-20-37-64.ca-central-1.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-37-64.ca-central-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-3-96-216-165.ca-central-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec27a483f61319a98e5ba2ee2d59e6a3,SystemUUID:ec27a483-f613-19a9-8e5b-a2ee2d59e6a3,BootID:9b41e339-1aaa-4374-8f11-8d34e12cd62a,KernelVersion:5.15.0-1026-aws,OSImage:Ubuntu 22.04.1 LTS,ContainerRuntimeVersion:docker://20.10.17,KubeletVersion:v1.23.15,KubeProxyVersion:v1.23.15,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[registry.k8s.io/etcdadm/etcd-manager@sha256:66a453db625abb268f4b3bbefc5a34a171d81e6e8796cecca54cfd71775c77c4 registry.k8s.io/etcdadm/etcd-manager:v3.0.20221209],SizeBytes:662209237,},ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:456569528,},ContainerImage{Names:[k8s.gcr.io/kube-apiserver-amd64:v1.23.15 registry.k8s.io/kube-apiserver-amd64:v1.23.15],SizeBytes:135240444,},ContainerImage{Names:[k8s.gcr.io/kube-controller-manager-amd64:v1.23.15 registry.k8s.io/kube-controller-manager-amd64:v1.23.15],SizeBytes:125046097,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.23.15 registry.k8s.io/kube-proxy-amd64:v1.23.15],SizeBytes:112336036,},ContainerImage{Names:[quay.io/cilium/operator@sha256:a6d24a006a6b92967ac90786b49bc1ac26e5477cf028cd1186efcfc2466484db quay.io/cilium/operator:v1.12.5],SizeBytes:110016956,},ContainerImage{Names:[registry.k8s.io/kops/kops-controller:1.26.0-beta.1],SizeBytes:104212800,},ContainerImage{Names:[registry.k8s.io/kops/dns-controller:1.26.0-beta.1],SizeBytes:102566451,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:10e231cf07c82dfab5141c1afe548dc734e5fc1f67665ec3982a325d1bd31a9a registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.0],SizeBytes:94471559,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:122bfb8c1edabb3c0edd63f06523e6940d958d19b3957dc7b1d6f81e9f1f6119 registry.k8s.io/sig-storage/csi-provisioner:v3.1.0],SizeBytes:57736070,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:9ebbf9f023e7b41ccee3d52afe39a89e3ddacdbb69269d583abfc25847cfd9e4 registry.k8s.io/sig-storage/csi-resizer:v1.4.0],SizeBytes:55464778,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:8b9c313c05f54fb04f8d430896f5f5904b6cb157df261501b29adc04d2b2dc7b registry.k8s.io/sig-storage/csi-attacher:v3.4.0],SizeBytes:54848595,},ContainerImage{Names:[k8s.gcr.io/kube-scheduler-amd64:v1.23.15 registry.k8s.io/kube-scheduler-amd64:v1.23.15],SizeBytes:53521430,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:19582112,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:17611980,},ContainerImage{Names:[registry.k8s.io/kops/kube-apiserver-healthcheck:1.26.0-beta.1],SizeBytes:8787111,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:682696,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Jan 9 23:00:10.066: INFO: Logging kubelet events for node ip-172-20-37-64.ca-central-1.compute.internal Jan 9 23:00:10.100: INFO: Logging pods the kubelet thinks is on node ip-172-20-37-64.ca-central-1.compute.internal Jan 9 23:00:10.138: INFO: kube-apiserver-ip-172-20-37-64.ca-central-1.compute.internal started at 2023-01-09 22:36:15 +0000 UTC (0+2 container statuses recorded) Jan 9 23:00:10.138: INFO: Container healthcheck ready: true, restart count 0 Jan 9 23:00:10.138: INFO: Container kube-apiserver ready: true, restart count 1 Jan 9 23:00:10.138: INFO: kube-controller-manager-ip-172-20-37-64.ca-central-1.compute.internal started at 2023-01-09 22:36:15 +0000 UTC (0+1 container statuses recorded) Jan 9 23:00:10.138: INFO: Container kube-controller-manager ready: true, restart count 2 Jan 9 23:00:10.138: INFO: kops-controller-cqrv8 started at 2023-01-09 22:37:19 +0000 UTC (0+1 container statuses recorded) Jan 9 23:00:10.138: INFO: Container kops-controller ready: true, restart count 0 Jan 9 23:00:10.138: INFO: ebs-csi-node-8q4pg started at 2023-01-09 22:37:17 +0000 UTC (0+3 container statuses recorded) Jan 9 23:00:10.138: INFO: Container ebs-plugin ready: true, restart count 0 Jan 9 23:00:10.138: INFO: Container liveness-probe ready: true, restart count 0 Jan 9 23:00:10.138: INFO: Container node-driver-registrar ready: true, restart count 0 Jan 9 23:00:10.138: INFO: cilium-q7lrg started at 2023-01-09 22:37:17 +0000 UTC (1+1 container statuses recorded) Jan 9 23:00:10.138: INFO: Init container clean-cilium-state ready: true, restart count 0 Jan 9 23:00:10.138: INFO: Container cilium-agent ready: true, restart count 0 Jan 9 23:00:10.138: INFO: cilium-operator-c7bfc9f44-w4nqz started at 2023-01-09 22:37:19 +0000 UTC (0+1 container statuses recorded) Jan 9 23:00:10.138: INFO: Container cilium-operator ready: true, restart count 0 Jan 9 23:00:10.138: INFO: ebs-csi-controller-5d8d7cc949-8wxx2 started at 2023-01-09 22:37:19 +0000 UTC (0+5 container statuses recorded) Jan 9 23:00:10.138: INFO: Container csi-attacher ready: true, restart count 0 Jan 9 23:00:10.138: INFO: Container csi-provisioner ready: true, restart count 0 Jan 9 23:00:10.138: INFO: Container csi-resizer ready: true, restart count 0 Jan 9 23:00:10.138: INFO: Container ebs-plugin ready: true, restart count 0 Jan 9 23:00:10.138: INFO: Container liveness-probe ready: true, restart count 0 Jan 9 23:00:10.138: INFO: dns-controller-867784b75c-sxxcl started at 2023-01-09 22:37:20 +0000 UTC (0+1 container statuses recorded) Jan 9 23:00:10.138: INFO: Container dns-controller ready: true, restart count 0 Jan 9 23:00:10.138: INFO: kube-scheduler-ip-172-20-37-64.ca-central-1.compute.internal started at 2023-01-09 22:36:15 +0000 UTC (0+1 container statuses recorded) Jan 9 23:00:10.138: INFO: Container kube-scheduler ready: true, restart count 0 Jan 9 23:00:10.138: INFO: etcd-manager-events-ip-172-20-37-64.ca-central-1.compute.internal started at 2023-01-09 22:36:15 +0000 UTC (0+1 container statuses recorded) Jan 9 23:00:10.138: INFO: Container etcd-manager ready: true, restart count 0 Jan 9 23:00:10.138: INFO: etcd-manager-main-ip-172-20-37-64.ca-central-1.compute.internal started at 2023-01-09 22:36:15 +0000 UTC (0+1 container statuses recorded) Jan 9 23:00:10.138: INFO: Container etcd-manager ready: true, restart count 0 Jan 9 23:00:10.255: INFO: Latency metrics for node ip-172-20-37-64.ca-central-1.compute.internal Jan 9 23:00:10.255: INFO: Logging node info for node ip-172-20-55-216.ca-central-1.compute.internal Jan 9 23:00:10.286: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-55-216.ca-central-1.compute.internal 2ceff781-505a-4706-ad8b-e38e37f72571 42017 0 2023-01-09 22:38:47 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ca-central-1 failure-domain.beta.kubernetes.io/zone:ca-central-1a kubelet_cleanup:true kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-55-216.ca-central-1.compute.internal kubernetes.io/os:linux kubernetes.io/role:node node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:ca-central-1a topology.hostpath.csi/node:ip-172-20-55-216.ca-central-1.compute.internal topology.kubernetes.io/region:ca-central-1 topology.kubernetes.io/zone:ca-central-1a] map[csi.volume.kubernetes.io/nodeid:{"csi-mock-csi-mock-volumes-6368":"csi-mock-csi-mock-volumes-6368","ebs.csi.aws.com":"i-0c78741e44993ae7f"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kops-controller Update v1 2023-01-09 22:38:47 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}} } {kube-controller-manager Update v1 2023-01-09 22:38:47 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.4.0/24\"":{}}}} } {kubelet Update v1 2023-01-09 22:38:47 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {e2e.test Update v1 2023-01-09 22:54:42 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubelet_cleanup":{}}}} } {kube-controller-manager Update v1 2023-01-09 22:56:45 +0000 UTC FieldsV1 {"f:status":{"f:volumesAttached":{}}} status} {kubelet Update v1 2023-01-09 22:56:51 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:capacity":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.4.0/24,DoNotUseExternalID:,ProviderID:aws:///ca-central-1a/i-0c78741e44993ae7f,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.4.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051644416 0} {<nil>} 3956684Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946786816 0} {<nil>} 3854284Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-01-09 22:57:01 +0000 UTC,LastTransitionTime:2023-01-09 22:38:47 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-01-09 22:57:01 +0000 UTC,LastTransitionTime:2023-01-09 22:38:47 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-01-09 22:57:01 +0000 UTC,LastTransitionTime:2023-01-09 22:38:47 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-01-09 22:57:01 +0000 UTC,LastTransitionTime:2023-01-09 22:39:07 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.55.216,},NodeAddress{Type:ExternalIP,Address:15.222.60.56,},NodeAddress{Type:Hostname,Address:ip-172-20-55-216.ca-central-1.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-55-216.ca-central-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-15-222-60-56.ca-central-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec249e6a3570a28ebd4276206d3ae523,SystemUUID:ec249e6a-3570-a28e-bd42-76206d3ae523,BootID:d58dcef2-849d-4807-9ba5-0438ebaa05b9,KernelVersion:5.15.0-1026-aws,OSImage:Ubuntu 22.04.1 LTS,ContainerRuntimeVersion:docker://20.10.17,KubeletVersion:v1.23.15,KubeProxyVersion:v1.23.15,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:456569528,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/jessie-dnsutils@sha256:11e6a66017ba4e4b938c1612b7a54a3befcefd354796c04e1dba76873a13518e k8s.gcr.io/e2e-test-images/jessie-dnsutils:1.5],SizeBytes:253371792,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:20f25f275d46aa728f7615a1ccc19c78b2ed89435bf943a44b339f70f45508e6 k8s.gcr.io/e2e-test-images/httpd:2.4.39-2],SizeBytes:126894770,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:7e8bdd271312fd25fc5ff5a8f04727be84044eb3d7d8d03611972a6752e2e11e k8s.gcr.io/e2e-test-images/agnhost:2.39],SizeBytes:126872991,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nautilus@sha256:99c0d6f1ad24a1aa1905d9c6534d193f268f7b23f9add2ae6bb41f31094bdd5c k8s.gcr.io/e2e-test-images/nautilus:1.5],SizeBytes:124758741,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:1b9d1b2f36cb2dbee1960e82a9344aeb11bd4c4c03abf5e1853e0559c23855e3 k8s.gcr.io/e2e-test-images/httpd:2.4.38-2],SizeBytes:123781643,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.23.15 registry.k8s.io/kube-proxy-amd64:v1.23.15],SizeBytes:112336036,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:10e231cf07c82dfab5141c1afe548dc734e5fc1f67665ec3982a325d1bd31a9a registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.0],SizeBytes:94471559,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:c8e03f60afa90a28e4bb6ec9a8d0fc36d89de4b7475cf2d613afa793ec969fe0 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.0],SizeBytes:56382329,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:4e74c0492bceddc598de1c90cc5bc14dcda94cb49fa9c5bad9d117c4834b5e08 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.1],SizeBytes:56379269,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:36c31f7e1f433c9634d24f876353e8646246d81a03c4e351202c2644daff1620 k8s.gcr.io/sig-storage/csi-resizer:v1.2.0],SizeBytes:53956308,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:ed98431376c9e944e19a465fe8ea944806714dd95416a0821096c78d66b579bd k8s.gcr.io/sig-storage/csi-snapshotter:v4.1.1],SizeBytes:53531990,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:c5be65d6679efabb969d9b019300d187437ae876f992c40911fd2892bbef3b36 k8s.gcr.io/sig-storage/csi-attacher:v3.2.0],SizeBytes:53500826,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:60ab9b3e6a030d3038c87c0d6bca2930f58d1d72823e6a4af09767dc83b696a2 k8s.gcr.io/sig-storage/csi-attacher:v3.2.1],SizeBytes:53499570,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:7a5ba58a44e0d749e0767e4e37315bcf6a61f33ce3185c1991848af4db0fb70a k8s.gcr.io/sig-storage/csi-resizer:v1.1.0],SizeBytes:49176472,},ContainerImage{Names:[k8s.gcr.io/sig-storage/hostpathplugin@sha256:232fe80174d60d520d36043103853a1d7ab4b7f3782cf43e45034f04ccda58ce k8s.gcr.io/sig-storage/hostpathplugin:v1.7.1],SizeBytes:29977921,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:a61d309da54641db41fb8f35718f744e9f730d4d0384f8c4b186ddc9f06cbd5f k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.1.0],SizeBytes:19662887,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:19582112,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:2dee3fe5fe861bb66c3a4ac51114f3447a4cd35870e0f2e2b558c7a400d89589 k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.2.0],SizeBytes:18699436,},ContainerImage{Names:[k8s.gcr.io/sig-storage/mock-driver@sha256:a7b517f9e0f42ffade802eef9cefa271372386b85f55b702b493241e58459793 k8s.gcr.io/sig-storage/mock-driver:v4.1.0],SizeBytes:17680993,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:17611980,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:529be2c9770add0cdd0c989115222ea9fc1be430c11095eb9f6dafcf98a36e2b k8s.gcr.io/sig-storage/livenessprobe:v2.4.0],SizeBytes:17175488,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:13616070e3f29de4417eee434a8ef472221c9e51b3d037b5a6b46cef08eb7443 k8s.gcr.io/e2e-test-images/nginx:1.14-2],SizeBytes:16032814,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf k8s.gcr.io/e2e-test-images/busybox:1.29-2],SizeBytes:1154361,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db k8s.gcr.io/pause:3.6 registry.k8s.io/pause:3.6],SizeBytes:682696,},},VolumesInUse:[kubernetes.io/csi/csi-mock-csi-mock-volumes-6368^4 kubernetes.io/csi/ebs.csi.aws.com^vol-0c8e4ae9d44bf0644],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-0c8e4ae9d44bf0644,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/csi-mock-csi-mock-volumes-6368^4,DevicePath:,},},Config:nil,},} Jan 9 23:00:10.287: INFO: Logging kubelet events for node ip-172-20-55-216.ca-central-1.compute.internal Jan 9 23:00:10.321: INFO: Logging pods the kubelet thinks is on node ip-172-20-55-216.ca-central-1.compute.internal Jan 9 23:00:10.365: INFO: cilium-dqpw4 started at 2023-01-09 22:38:48 +0000 UTC (1+1 container statuses recorded) Jan 9 23:00:10.365: INFO: Init container clean-cilium-state ready: true, restart count 0 Jan 9 23:00:10.365: INFO: Container cilium-agent ready: true, restart count 0 Jan 9 23:00:10.365: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-b7bxr started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:00:10.365: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:00:10.365: INFO: externalname-service-cfh5p started at 2023-01-09 22:56:39 +0000 UTC (0+1 container statuses recorded) Jan 9 23:00:10.365: INFO: Container externalname-service ready: false, restart count 0 Jan 9 23:00:10.365: INFO: inline-volume-tester-7mx4s started at 2023-01-09 22:56:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:00:10.365: INFO: Container csi-volume-tester ready: false, restart count 0 Jan 9 23:00:10.365: INFO: csi-mockplugin-0 started at 2023-01-09 22:55:43 +0000 UTC (0+3 container statuses recorded) Jan 9 23:00:10.365: INFO: Container csi-provisioner ready: true, restart count 0 Jan 9 23:00:10.365: INFO: Container driver-registrar ready: true, restart count 0 Jan 9 23:00:10.365: INFO: Container mock ready: true, restart count 0 Jan 9 23:00:10.365: INFO: pod-4f81b2a8-6f5c-4506-9892-41ac72807c71 started at 2023-01-09 22:56:41 +0000 UTC (0+1 container statuses recorded) Jan 9 23:00:10.365: INFO: Container test-container ready: false, restart count 0 Jan 9 23:00:10.365: INFO: csi-mockplugin-resizer-0 started at 2023-01-09 22:55:01 +0000 UTC (0+1 container statuses recorded) Jan 9 23:00:10.365: INFO: Container csi-resizer ready: true, restart count 0 Jan 9 23:00:10.365: INFO: my-hostname-basic-e6b46933-d037-4e81-8a71-4b04c21b3646-rqpsj started at 2023-01-09 22:55:09 +0000 UTC (0+1 container statuses recorded) Jan 9 23:00:10.365: INFO: Container my-hostname-basic-e6b46933-d037-4e81-8a71-4b04c21b3646 ready: false, restart count 0 Jan 9 23:00:10.365: INFO: fail-once-non-local-mh6qc started at 2023-01-09 22:56:53 +0000 UTC (0+1 container statuses recorded) Jan 9 23:00:10.365: INFO: Container c ready: false, restart count 0 Jan 9 23:00:10.365: INFO: hostexec-ip-172-20-55-216.ca-central-1.compute.internal-wq7sx started at 2023-01-09 22:57:49 +0000 UTC (0+1 container statuses recorded) Jan 9 23:00:10.365: INFO: Container agnhost-container ready: true, restart count 0 Jan 9 23:00:10.365: INFO: rs-lf7br started at 2023-01-09 22:59:56 +0000 UTC (0+1 container statuses recorded) Jan 9 23:00:10.365: INFO: Container donothing ready: false, restart count 0 Jan 9 23:00:10.365: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-mcd88 started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:00:10.365: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:00:10.365: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-6nqvb started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:00:10.365: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:00:10.365: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-8kb86 started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:00:10.365: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:00:10.365: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-vwd6k started at 2023-01-09 22:54:43 +0000 UTC (0+1 container statuses recorded) Jan 9 23:00:10.365: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:00:10.365: INFO: csi-mockplugin-0 started at 2023-01-09 22:55:01 +0000 UTC (0+3 container statuses recorded) Jan 9 23:00:10.365: INFO: Container csi-provisioner ready: false, restart count 0 Jan 9 23:00:10.365: INFO: Container driver-registrar ready: false, restart count 0 Jan 9 23:00:10.365: INFO: Container mock ready: false, restart count 0 Jan 9 23:00:10.365: INFO: csi-mockplugin-attacher-0 started at 2023-01-09 22:55:43 +0000 UTC (0+1 container statuses recorded) Jan 9 23:00:10.365: INFO: Container csi-attacher ready: true, restart count 0 Jan 9 23:00:10.365: INFO: fail-once-non-local-nx7cr started at 2023-01-09 22:56:53 +0000 UTC (0+1 container statuses recorded) Jan 9 23:00:10.365: INFO: Container c ready: false, restart count 0 Jan 9 23:00:10.365: INFO: ebs-csi-node-psq97 started at 2023-01-09 22:38:48 +0000 UTC (0+3 container statuses recorded) Jan 9 23:00:10.365: INFO: Container ebs-plugin ready: true, restart count 0 Jan 9 23:00:10.365: INFO: Container liveness-probe ready: true, restart count 0 Jan 9 23:00:10.365: INFO: Container node-driver-registrar ready: true, restart count 0 Jan 9 23:00:10.365: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-qj2bw started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:00:10.365: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:00:10.365: INFO: ss2-0 started at 2023-01-09 22:57:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:00:10.365: INFO: Container webserver ready: false, restart count 0 Jan 9 23:00:10.365: INFO: test-container-pod started at 2023-01-09 22:55:03 +0000 UTC (0+1 container statuses recorded) Jan 9 23:00:10.365: INFO: Container webserver ready: false, restart count 0 Jan 9 23:00:10.365: INFO: rs-ctw9c started at 2023-01-09 22:59:56 +0000 UTC (0+1 container statuses recorded) Jan 9 23:00:10.365: INFO: Container donothing ready: false, restart count 0 Jan 9 23:00:10.365: INFO: netserver-1 started at 2023-01-09 22:54:37 +0000 UTC (0+1 container statuses recorded) Jan 9 23:00:10.365: INFO: Container webserver ready: true, restart count 0 Jan 9 23:00:10.365: INFO: pvc-volume-tester-x8kmj started at 2023-01-09 22:56:54 +0000 UTC (0+1 container statuses recorded) Jan 9 23:00:10.365: INFO: Container volume-tester ready: false, restart count 0 Jan 9 23:00:10.365: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-6ks7b started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:00:10.365: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:00:10.365: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-72kwd started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:00:10.365: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:00:10.365: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-krwzl started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:00:10.365: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: false, restart count 0 Jan 9 23:00:10.365: INFO: pod-subpath-test-preprovisionedpv-9bjc started at 2023-01-09 22:58:02 +0000 UTC (1+1 container statuses recorded) Jan 9 23:00:10.365: INFO: Init container init-volume-preprovisionedpv-9bjc ready: false, restart count 0 Jan 9 23:00:10.365: INFO: Container test-container-subpath-preprovisionedpv-9bjc ready: false, restart count 0 Jan 9 23:00:10.365: INFO: rs-j4qxp started at 2023-01-09 22:59:56 +0000 UTC (0+1 container statuses recorded) Jan 9 23:00:10.365: INFO: Container donothing ready: false, restart count 0 Jan 9 23:00:10.365: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-qglk4 started at 2023-01-09 22:54:43 +0000 UTC (0+1 container statuses recorded) Jan 9 23:00:10.365: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: false, restart count 0 Jan 9 23:00:10.365: INFO: csi-mockplugin-attacher-0 started at 2023-01-09 22:55:01 +0000 UTC (0+1 container statuses recorded) Jan 9 23:00:10.365: INFO: Container csi-attacher ready: false, restart count 0 Jan 9 23:00:10.365: INFO: host-test-container-pod started at 2023-01-09 22:55:03 +0000 UTC (0+1 container statuses recorded) Jan 9 23:00:10.365: INFO: Container agnhost-container ready: true, restart count 0 Jan 9 23:00:10.675: INFO: Latency metrics for node ip-172-20-55-216.ca-central-1.compute.internal Jan 9 23:00:10.675: INFO: Logging node info for node ip-172-20-57-251.ca-central-1.compute.internal Jan 9 23:00:10.709: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-57-251.ca-central-1.compute.internal eaaaf6eb-cbd5-4763-9c39-fb0e1a4ce44a 41445 0 2023-01-09 22:38:45 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ca-central-1 failure-domain.beta.kubernetes.io/zone:ca-central-1a kubelet_cleanup:true kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-57-251.ca-central-1.compute.internal kubernetes.io/os:linux kubernetes.io/role:node node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:ca-central-1a topology.hostpath.csi/node:ip-172-20-57-251.ca-central-1.compute.internal topology.kubernetes.io/region:ca-central-1 topology.kubernetes.io/zone:ca-central-1a] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-0513e7d6e5eba8eec"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kops-controller Update v1 2023-01-09 22:38:45 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}} } {kube-controller-manager Update v1 2023-01-09 22:38:45 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.2.0/24\"":{}}}} } {kubelet Update v1 2023-01-09 22:38:45 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {e2e.test Update v1 2023-01-09 22:54:42 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubelet_cleanup":{}}}} } {kubelet Update v1 2023-01-09 22:56:00 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.2.0/24,DoNotUseExternalID:,ProviderID:aws:///ca-central-1a/i-0513e7d6e5eba8eec,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.2.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051652608 0} {<nil>} 3956692Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946795008 0} {<nil>} 3854292Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-01-09 22:56:00 +0000 UTC,LastTransitionTime:2023-01-09 22:38:45 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-01-09 22:56:00 +0000 UTC,LastTransitionTime:2023-01-09 22:38:45 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-01-09 22:56:00 +0000 UTC,LastTransitionTime:2023-01-09 22:38:45 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-01-09 22:56:00 +0000 UTC,LastTransitionTime:2023-01-09 22:39:05 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.57.251,},NodeAddress{Type:ExternalIP,Address:3.96.144.209,},NodeAddress{Type:Hostname,Address:ip-172-20-57-251.ca-central-1.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-57-251.ca-central-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-3-96-144-209.ca-central-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2f0d7ba95b37a96a1d9360f6071d84,SystemUUID:ec2f0d7b-a95b-37a9-6a1d-9360f6071d84,BootID:67865282-1911-4e38-8675-a7b78d5ae45e,KernelVersion:5.15.0-1026-aws,OSImage:Ubuntu 22.04.1 LTS,ContainerRuntimeVersion:docker://20.10.17,KubeletVersion:v1.23.15,KubeProxyVersion:v1.23.15,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:456569528,},ContainerImage{Names:[k8s.gcr.io/etcd@sha256:dd75ec974b0a2a6f6bb47001ba09207976e625db898d1b16735528c009cb171c k8s.gcr.io/etcd:3.5.6-0],SizeBytes:299475478,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/jessie-dnsutils@sha256:11e6a66017ba4e4b938c1612b7a54a3befcefd354796c04e1dba76873a13518e k8s.gcr.io/e2e-test-images/jessie-dnsutils:1.5],SizeBytes:253371792,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:20f25f275d46aa728f7615a1ccc19c78b2ed89435bf943a44b339f70f45508e6 k8s.gcr.io/e2e-test-images/httpd:2.4.39-2],SizeBytes:126894770,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:7e8bdd271312fd25fc5ff5a8f04727be84044eb3d7d8d03611972a6752e2e11e k8s.gcr.io/e2e-test-images/agnhost:2.39],SizeBytes:126872991,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nautilus@sha256:99c0d6f1ad24a1aa1905d9c6534d193f268f7b23f9add2ae6bb41f31094bdd5c k8s.gcr.io/e2e-test-images/nautilus:1.5],SizeBytes:124758741,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:1b9d1b2f36cb2dbee1960e82a9344aeb11bd4c4c03abf5e1853e0559c23855e3 k8s.gcr.io/e2e-test-images/httpd:2.4.38-2],SizeBytes:123781643,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.23.15 registry.k8s.io/kube-proxy-amd64:v1.23.15],SizeBytes:112336036,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:10e231cf07c82dfab5141c1afe548dc734e5fc1f67665ec3982a325d1bd31a9a registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.0],SizeBytes:94471559,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/sample-apiserver@sha256:f9c93b92b6ff750b41a93c4e4fe0bfe384597aeb841e2539d5444815c55b2d8f k8s.gcr.io/e2e-test-images/sample-apiserver:1.17.5],SizeBytes:56504541,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:c8e03f60afa90a28e4bb6ec9a8d0fc36d89de4b7475cf2d613afa793ec969fe0 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.0],SizeBytes:56382329,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:4e74c0492bceddc598de1c90cc5bc14dcda94cb49fa9c5bad9d117c4834b5e08 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.1],SizeBytes:56379269,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:36c31f7e1f433c9634d24f876353e8646246d81a03c4e351202c2644daff1620 k8s.gcr.io/sig-storage/csi-resizer:v1.2.0],SizeBytes:53956308,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:ed98431376c9e944e19a465fe8ea944806714dd95416a0821096c78d66b579bd k8s.gcr.io/sig-storage/csi-snapshotter:v4.1.1],SizeBytes:53531990,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:c5be65d6679efabb969d9b019300d187437ae876f992c40911fd2892bbef3b36 k8s.gcr.io/sig-storage/csi-attacher:v3.2.0],SizeBytes:53500826,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:60ab9b3e6a030d3038c87c0d6bca2930f58d1d72823e6a4af09767dc83b696a2 k8s.gcr.io/sig-storage/csi-attacher:v3.2.1],SizeBytes:53499570,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:7a5ba58a44e0d749e0767e4e37315bcf6a61f33ce3185c1991848af4db0fb70a k8s.gcr.io/sig-storage/csi-resizer:v1.1.0],SizeBytes:49176472,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/regression-issue-74839@sha256:b4f1d8d61bdad84bd50442d161d5460e4019d53e989b64220fdbc62fc87d76bf k8s.gcr.io/e2e-test-images/regression-issue-74839:1.2],SizeBytes:44576952,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nonroot@sha256:b9e2958a3dd879e3cf11142228c6d073d0fc4ea2e857c3be6f4fb0ab5fb2c937 k8s.gcr.io/e2e-test-images/nonroot:1.2],SizeBytes:42321438,},ContainerImage{Names:[k8s.gcr.io/sig-storage/hostpathplugin@sha256:232fe80174d60d520d36043103853a1d7ab4b7f3782cf43e45034f04ccda58ce k8s.gcr.io/sig-storage/hostpathplugin:v1.7.1],SizeBytes:29977921,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:a61d309da54641db41fb8f35718f744e9f730d4d0384f8c4b186ddc9f06cbd5f k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.1.0],SizeBytes:19662887,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:19582112,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:2dee3fe5fe861bb66c3a4ac51114f3447a4cd35870e0f2e2b558c7a400d89589 k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.2.0],SizeBytes:18699436,},ContainerImage{Names:[k8s.gcr.io/sig-storage/mock-driver@sha256:a7b517f9e0f42ffade802eef9cefa271372386b85f55b702b493241e58459793 k8s.gcr.io/sig-storage/mock-driver:v4.1.0],SizeBytes:17680993,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:17611980,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:529be2c9770add0cdd0c989115222ea9fc1be430c11095eb9f6dafcf98a36e2b k8s.gcr.io/sig-storage/livenessprobe:v2.4.0],SizeBytes:17175488,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:13616070e3f29de4417eee434a8ef472221c9e51b3d037b5a6b46cef08eb7443 k8s.gcr.io/e2e-test-images/nginx:1.14-2],SizeBytes:16032814,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nonewprivs@sha256:8ac1264691820febacf3aea5d152cbde6d10685731ec14966a9401c6f47a68ac k8s.gcr.io/e2e-test-images/nonewprivs:1.3],SizeBytes:7107254,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf k8s.gcr.io/e2e-test-images/busybox:1.29-2],SizeBytes:1154361,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:39e1e963e5310e9c313bad51523be012ede7b35bb9316517d19089a010356592 k8s.gcr.io/e2e-test-images/busybox:1.29-1],SizeBytes:1154361,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db k8s.gcr.io/pause:3.6 registry.k8s.io/pause:3.6],SizeBytes:682696,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Jan 9 23:00:10.709: INFO: Logging kubelet events for node ip-172-20-57-251.ca-central-1.compute.internal Jan 9 23:00:10.744: INFO: Logging pods the kubelet thinks is on node ip-172-20-57-251.ca-central-1.compute.internal Jan 9 23:00:10.784: INFO: pod-projected-configmaps-42a5aed6-787c-4898-951d-d57d8df41be9 started at 2023-01-09 22:55:15 +0000 UTC (0+1 container statuses recorded) Jan 9 23:00:10.784: INFO: Container agnhost-container ready: false, restart count 0 Jan 9 23:00:10.784: INFO: rs-56t9b started at 2023-01-09 22:59:56 +0000 UTC (0+1 container statuses recorded) Jan 9 23:00:10.784: INFO: Container donothing ready: false, restart count 0 Jan 9 23:00:10.784: INFO: externalname-service-k4b9s started at 2023-01-09 22:56:39 +0000 UTC (0+1 container statuses recorded) Jan 9 23:00:10.784: INFO: Container externalname-service ready: true, restart count 0 Jan 9 23:00:10.784: INFO: hostexec-ip-172-20-57-251.ca-central-1.compute.internal-d92vh started at 2023-01-09 23:00:07 +0000 UTC (0+1 container statuses recorded) Jan 9 23:00:10.784: INFO: Container agnhost-container ready: true, restart count 0 Jan 9 23:00:10.784: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-tvxfq started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:00:10.784: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:00:10.784: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-wmqwf started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:00:10.784: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:00:10.784: INFO: hostexec-ip-172-20-57-251.ca-central-1.compute.internal-rvlgt started at 2023-01-09 22:55:08 +0000 UTC (0+1 container statuses recorded) Jan 9 23:00:10.784: INFO: Container agnhost-container ready: true, restart count 0 Jan 9 23:00:10.784: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-m8cfz started at 2023-01-09 22:54:43 +0000 UTC (0+1 container statuses recorded) Jan 9 23:00:10.784: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:00:10.784: INFO: rs-49kqp started at 2023-01-09 22:59:56 +0000 UTC (0+1 container statuses recorded) Jan 9 23:00:10.784: INFO: Container donothing ready: false, restart count 0 Jan 9 23:00:10.784: INFO: hostexec-ip-172-20-57-251.ca-central-1.compute.internal-q7sw2 started at 2023-01-09 22:54:49 +0000 UTC (0+1 container statuses recorded) Jan 9 23:00:10.784: INFO: Container agnhost-container ready: true, restart count 0 Jan 9 23:00:10.784: INFO: rs-dzbwc started at 2023-01-09 22:59:56 +0000 UTC (0+1 container statuses recorded) Jan 9 23:00:10.784: INFO: Container donothing ready: false, restart count 0 Jan 9 23:00:10.784: INFO: ebs-csi-node-g4ngc started at 2023-01-09 22:38:46 +0000 UTC (0+3 container statuses recorded) Jan 9 23:00:10.784: INFO: Container ebs-plugin ready: true, restart count 0 Jan 9 23:00:10.784: INFO: Container liveness-probe ready: true, restart count 0 Jan 9 23:00:10.784: INFO: Container node-driver-registrar ready: true, restart count 0 Jan 9 23:00:10.784: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-rbqs2 started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:00:10.784: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:00:10.784: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-96np5 started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:00:10.784: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:00:10.784: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-f9w29 started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:00:10.784: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:00:10.784: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-zx8gm started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:00:10.784: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:00:10.784: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-qzf5t started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:00:10.784: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:00:10.784: INFO: terminate-cmd-rpof190988ae-62a8-4c20-8047-f3d403e08a0f started at 2023-01-09 22:58:10 +0000 UTC (0+1 container statuses recorded) Jan 9 23:00:10.784: INFO: Container terminate-cmd-rpof ready: false, restart count 0 Jan 9 23:00:10.784: INFO: netserver-2 started at 2023-01-09 22:54:37 +0000 UTC (0+1 container statuses recorded) Jan 9 23:00:10.784: INFO: Container webserver ready: true, restart count 0 Jan 9 23:00:10.784: INFO: pod-d57ea68f-7c7a-4031-8def-db582c490286 started at 2023-01-09 22:55:13 +0000 UTC (0+1 container statuses recorded) Jan 9 23:00:10.784: INFO: Container write-pod ready: false, restart count 0 Jan 9 23:00:10.784: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-zvfdw started at 2023-01-09 22:54:43 +0000 UTC (0+1 container statuses recorded) Jan 9 23:00:10.784: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:00:10.784: INFO: explicit-root-uid started at 2023-01-09 22:56:01 +0000 UTC (0+1 container statuses recorded) Jan 9 23:00:10.784: INFO: Container explicit-root-uid ready: false, restart count 0 Jan 9 23:00:10.784: INFO: pod-projected-secrets-37f76895-190e-4b64-b08c-ceae234836df started at 2023-01-09 22:57:53 +0000 UTC (0+1 container statuses recorded) Jan 9 23:00:10.784: INFO: Container projected-secret-volume-test ready: false, restart count 0 Jan 9 23:00:10.784: INFO: cilium-6p9jz started at 2023-01-09 22:38:46 +0000 UTC (1+1 container statuses recorded) Jan 9 23:00:10.784: INFO: Init container clean-cilium-state ready: true, restart count 0 Jan 9 23:00:10.784: INFO: Container cilium-agent ready: true, restart count 0 Jan 9 23:00:10.784: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-wvflq started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:00:10.784: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:00:11.613: INFO: Latency metrics for node ip-172-20-57-251.ca-central-1.compute.internal Jan 9 23:00:11.613: INFO: Logging node info for node ip-172-20-59-208.ca-central-1.compute.internal Jan 9 23:00:11.644: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-59-208.ca-central-1.compute.internal d061a4d4-9ff6-43fc-89cc-422c61f7441c 41293 0 2023-01-09 22:38:43 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ca-central-1 failure-domain.beta.kubernetes.io/zone:ca-central-1a kubelet_cleanup:true kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-59-208.ca-central-1.compute.internal kubernetes.io/os:linux kubernetes.io/role:node node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:ca-central-1a topology.hostpath.csi/node:ip-172-20-59-208.ca-central-1.compute.internal topology.kubernetes.io/region:ca-central-1 topology.kubernetes.io/zone:ca-central-1a] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-0f00ea8bafbacdd11"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kops-controller Update v1 2023-01-09 22:38:43 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}} } {kube-controller-manager Update v1 2023-01-09 22:38:43 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.1.0/24\"":{}}}} } {kubelet Update v1 2023-01-09 22:38:43 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {e2e.test Update v1 2023-01-09 22:54:42 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubelet_cleanup":{}}}} } {kubelet Update v1 2023-01-09 22:55:47 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:capacity":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.1.0/24,DoNotUseExternalID:,ProviderID:aws:///ca-central-1a/i-0f00ea8bafbacdd11,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.1.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051652608 0} {<nil>} 3956692Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946795008 0} {<nil>} 3854292Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-01-09 22:55:47 +0000 UTC,LastTransitionTime:2023-01-09 22:38:43 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-01-09 22:55:47 +0000 UTC,LastTransitionTime:2023-01-09 22:38:43 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-01-09 22:55:47 +0000 UTC,LastTransitionTime:2023-01-09 22:38:43 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-01-09 22:55:47 +0000 UTC,LastTransitionTime:2023-01-09 22:39:03 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.59.208,},NodeAddress{Type:ExternalIP,Address:99.79.51.95,},NodeAddress{Type:Hostname,Address:ip-172-20-59-208.ca-central-1.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-59-208.ca-central-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-99-79-51-95.ca-central-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec28d5ecfae27d4bcedd89eca1a365f0,SystemUUID:ec28d5ec-fae2-7d4b-cedd-89eca1a365f0,BootID:560320f7-12c5-44f2-a7f6-a7a40031e204,KernelVersion:5.15.0-1026-aws,OSImage:Ubuntu 22.04.1 LTS,ContainerRuntimeVersion:docker://20.10.17,KubeletVersion:v1.23.15,KubeProxyVersion:v1.23.15,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:456569528,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:20f25f275d46aa728f7615a1ccc19c78b2ed89435bf943a44b339f70f45508e6 k8s.gcr.io/e2e-test-images/httpd:2.4.39-2],SizeBytes:126894770,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:7e8bdd271312fd25fc5ff5a8f04727be84044eb3d7d8d03611972a6752e2e11e k8s.gcr.io/e2e-test-images/agnhost:2.39],SizeBytes:126872991,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:1b9d1b2f36cb2dbee1960e82a9344aeb11bd4c4c03abf5e1853e0559c23855e3 k8s.gcr.io/e2e-test-images/httpd:2.4.38-2],SizeBytes:123781643,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.23.15 registry.k8s.io/kube-proxy-amd64:v1.23.15],SizeBytes:112336036,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:10e231cf07c82dfab5141c1afe548dc734e5fc1f67665ec3982a325d1bd31a9a registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.0],SizeBytes:94471559,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:c8e03f60afa90a28e4bb6ec9a8d0fc36d89de4b7475cf2d613afa793ec969fe0 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.0],SizeBytes:56382329,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:4e74c0492bceddc598de1c90cc5bc14dcda94cb49fa9c5bad9d117c4834b5e08 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.1],SizeBytes:56379269,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:36c31f7e1f433c9634d24f876353e8646246d81a03c4e351202c2644daff1620 k8s.gcr.io/sig-storage/csi-resizer:v1.2.0],SizeBytes:53956308,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:ed98431376c9e944e19a465fe8ea944806714dd95416a0821096c78d66b579bd k8s.gcr.io/sig-storage/csi-snapshotter:v4.1.1],SizeBytes:53531990,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:c5be65d6679efabb969d9b019300d187437ae876f992c40911fd2892bbef3b36 k8s.gcr.io/sig-storage/csi-attacher:v3.2.0],SizeBytes:53500826,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:60ab9b3e6a030d3038c87c0d6bca2930f58d1d72823e6a4af09767dc83b696a2 k8s.gcr.io/sig-storage/csi-attacher:v3.2.1],SizeBytes:53499570,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:017727efcfeb7d053af68e51436ce8e65edbc6ca573720afb4f79c8594036955 registry.k8s.io/coredns/coredns:v1.10.0],SizeBytes:50249337,},ContainerImage{Names:[registry.k8s.io/cpa/cluster-proportional-autoscaler@sha256:68d396900aeaa072c1f27289485fdac29834045a6f3ffe369bf389d830ef572d registry.k8s.io/cpa/cluster-proportional-autoscaler:1.8.6],SizeBytes:47910609,},ContainerImage{Names:[k8s.gcr.io/sig-storage/hostpathplugin@sha256:232fe80174d60d520d36043103853a1d7ab4b7f3782cf43e45034f04ccda58ce k8s.gcr.io/sig-storage/hostpathplugin:v1.7.1],SizeBytes:29977921,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:a61d309da54641db41fb8f35718f744e9f730d4d0384f8c4b186ddc9f06cbd5f k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.1.0],SizeBytes:19662887,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:19582112,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:2dee3fe5fe861bb66c3a4ac51114f3447a4cd35870e0f2e2b558c7a400d89589 k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.2.0],SizeBytes:18699436,},ContainerImage{Names:[k8s.gcr.io/sig-storage/mock-driver@sha256:a7b517f9e0f42ffade802eef9cefa271372386b85f55b702b493241e58459793 k8s.gcr.io/sig-storage/mock-driver:v4.1.0],SizeBytes:17680993,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:17611980,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:529be2c9770add0cdd0c989115222ea9fc1be430c11095eb9f6dafcf98a36e2b k8s.gcr.io/sig-storage/livenessprobe:v2.4.0],SizeBytes:17175488,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:13616070e3f29de4417eee434a8ef472221c9e51b3d037b5a6b46cef08eb7443 k8s.gcr.io/e2e-test-images/nginx:1.14-2],SizeBytes:16032814,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nonewprivs@sha256:8ac1264691820febacf3aea5d152cbde6d10685731ec14966a9401c6f47a68ac k8s.gcr.io/e2e-test-images/nonewprivs:1.3],SizeBytes:7107254,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf k8s.gcr.io/e2e-test-images/busybox:1.29-2],SizeBytes:1154361,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:39e1e963e5310e9c313bad51523be012ede7b35bb9316517d19089a010356592 k8s.gcr.io/e2e-test-images/busybox:1.29-1],SizeBytes:1154361,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db k8s.gcr.io/pause:3.6 registry.k8s.io/pause:3.6],SizeBytes:682696,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Jan 9 23:00:11.645: INFO: Logging kubelet events for node ip-172-20-59-208.ca-central-1.compute.internal Jan 9 23:00:11.680: INFO: Logging pods the kubelet thinks is on node ip-172-20-59-208.ca-central-1.compute.internal Jan 9 23:00:11.723: INFO: rs-nd6n2 started at 2023-01-09 22:59:56 +0000 UTC (0+1 container statuses recorded) Jan 9 23:00:11.723: INFO: Container donothing ready: false, restart count 0 Jan 9 23:00:11.723: INFO: coredns-autoscaler-557ccb4c66-jnjcc started at 2023-01-09 22:39:03 +0000 UTC (0+1 container statuses recorded) Jan 9 23:00:11.723: INFO: Container autoscaler ready: true, restart count 0 Jan 9 23:00:11.723: INFO: csi-mockplugin-attacher-0 started at 2023-01-09 22:55:28 +0000 UTC (0+1 container statuses recorded) Jan 9 23:00:11.723: INFO: Container csi-attacher ready: true, restart count 0 Jan 9 23:00:11.723: INFO: rs-x69wn started at 2023-01-09 22:59:56 +0000 UTC (0+1 container statuses recorded) Jan 9 23:00:11.723: INFO: Container donothing ready: false, restart count 0 Jan 9 23:00:11.723: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-hszdf started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:00:11.723: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:00:11.723: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-fc6pg started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:00:11.723: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:00:11.723: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-8qp6l started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:00:11.723: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: false, restart count 0 Jan 9 23:00:11.723: INFO: ss2-0 started at 2023-01-09 22:54:44 +0000 UTC (0+1 container statuses recorded) Jan 9 23:00:11.723: INFO: Container webserver ready: false, restart count 0 Jan 9 23:00:11.723: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-bhxkw started at 2023-01-09 22:54:43 +0000 UTC (0+1 container statuses recorded) Jan 9 23:00:11.723: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:00:11.723: INFO: hostexec-ip-172-20-59-208.ca-central-1.compute.internal-jn4pk started at 2023-01-09 22:56:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:00:11.723: INFO: Container agnhost-container ready: true, restart count 0 Jan 9 23:00:11.723: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-rzgjf started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:00:11.723: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:00:11.723: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-ptfgl started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:00:11.723: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:00:11.723: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-t5g85 started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:00:11.723: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:00:11.723: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-hjsbl started at 2023-01-09 22:54:43 +0000 UTC (0+1 container statuses recorded) Jan 9 23:00:11.723: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: false, restart count 0 Jan 9 23:00:11.723: INFO: ebs-csi-node-prrrf started at 2023-01-09 22:38:44 +0000 UTC (0+3 container statuses recorded) Jan 9 23:00:11.723: INFO: Container ebs-plugin ready: true, restart count 0 Jan 9 23:00:11.723: INFO: Container liveness-probe ready: true, restart count 0 Jan 9 23:00:11.723: INFO: Container node-driver-registrar ready: true, restart count 0 Jan 9 23:00:11.723: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-x682f started at 2023-01-09 22:54:43 +0000 UTC (0+1 container statuses recorded) Jan 9 23:00:11.723: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:00:11.723: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-tx2mm started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:00:11.723: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:00:11.723: INFO: csi-mockplugin-0 started at 2023-01-09 22:55:28 +0000 UTC (0+3 container statuses recorded) Jan 9 23:00:11.723: INFO: Container csi-provisioner ready: false, restart count 0 Jan 9 23:00:11.723: INFO: Container driver-registrar ready: false, restart count 0 Jan 9 23:00:11.723: INFO: Container mock ready: false, restart count 0 Jan 9 23:00:11.723: INFO: cilium-z2ltf started at 2023-01-09 22:38:44 +0000 UTC (1+1 container statuses recorded) Jan 9 23:00:11.723: INFO: Init container clean-cilium-state ready: true, restart count 0 Jan 9 23:00:11.723: INFO: Container cilium-agent ready: true, restart count 0 Jan 9 23:00:11.723: INFO: netserver-3 started at 2023-01-09 22:54:37 +0000 UTC (0+1 container statuses recorded) Jan 9 23:00:11.723: INFO: Container webserver ready: true, restart count 0 Jan 9 23:00:11.723: INFO: coredns-867df8f45c-tmrqb started at 2023-01-09 22:39:03 +0000 UTC (0+1 container statuses recorded) Jan 9 23:00:11.723: INFO: Container coredns ready: true, restart count 0 Jan 9 23:00:11.723: INFO: pod-subpath-test-preprovisionedpv-gh54 started at 2023-01-09 22:56:48 +0000 UTC (2+2 container statuses recorded) Jan 9 23:00:11.723: INFO: Init container init-volume-preprovisionedpv-gh54 ready: false, restart count 0 Jan 9 23:00:11.723: INFO: Init container test-init-subpath-preprovisionedpv-gh54 ready: false, restart count 0 Jan 9 23:00:11.723: INFO: Container test-container-subpath-preprovisionedpv-gh54 ready: false, restart count 0 Jan 9 23:00:11.723: INFO: Container test-container-volume-preprovisionedpv-gh54 ready: false, restart count 0 Jan 9 23:00:11.952: INFO: Latency metrics for node ip-172-20-59-208.ca-central-1.compute.internal Jan 9 23:00:11.952: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready �[1mSTEP�[0m: Destroying namespace "replication-controller-5218" for this suite.
Find my-hostname-basic-e6b46933-d037-4e81-8a71-4b04c21b3646-rqpsj mentions in log files
go run hack/e2e.go -v --test --test_args='--ginkgo.focus=Kubernetes\se2e\ssuite\s\[sig\-apps\]\sStatefulSet\sBasic\sStatefulSet\sfunctionality\s\[StatefulSetBasic\]\sshould\simplement\slegacy\sreplacement\swhen\sthe\supdate\sstrategy\sis\sOnDelete$'
/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/apps/statefulset.go:505 Jan 9 23:04:44.259: Failed waiting for pods to enter running: timed out waiting for the condition /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/statefulset/wait.go:80from junit_02.xml
{"msg":"PASSED [sig-storage] Downward API volume should set DefaultMode on files [LinuxOnly] [NodeConformance] [Conformance]","total":-1,"completed":28,"skipped":224,"failed":0} [BeforeEach] [sig-apps] StatefulSet /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:185 �[1mSTEP�[0m: Creating a kubernetes client Jan 9 22:54:23.555: INFO: >>> kubeConfig: /root/.kube/config �[1mSTEP�[0m: Building a namespace api object, basename statefulset �[1mSTEP�[0m: Waiting for a default service account to be provisioned in namespace �[1mSTEP�[0m: Waiting for kube-root-ca.crt to be provisioned in namespace [BeforeEach] [sig-apps] StatefulSet /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/apps/statefulset.go:94 [BeforeEach] Basic StatefulSet functionality [StatefulSetBasic] /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/apps/statefulset.go:109 �[1mSTEP�[0m: Creating service test in namespace statefulset-6319 [It] should implement legacy replacement when the update strategy is OnDelete /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/apps/statefulset.go:505 �[1mSTEP�[0m: Creating a new StatefulSet Jan 9 22:54:23.897: INFO: Found 1 stateful pods, waiting for 3 Jan 9 22:54:33.942: INFO: Waiting for pod ss2-0 to enter Running - Ready=true, currently Running - Ready=true Jan 9 22:54:33.942: INFO: Waiting for pod ss2-1 to enter Running - Ready=true, currently Running - Ready=true Jan 9 22:54:33.942: INFO: Waiting for pod ss2-2 to enter Running - Ready=true, currently Pending - Ready=false Jan 9 22:54:43.930: INFO: Waiting for pod ss2-0 to enter Running - Ready=true, currently Running - Ready=true Jan 9 22:54:43.930: INFO: Waiting for pod ss2-1 to enter Running - Ready=true, currently Running - Ready=true Jan 9 22:54:43.930: INFO: Waiting for pod ss2-2 to enter Running - Ready=true, currently Running - Ready=true �[1mSTEP�[0m: Restoring Pods to the current revision Jan 9 22:54:44.193: INFO: Found 1 stateful pods, waiting for 3 Jan 9 22:54:54.226: INFO: Found 1 stateful pods, waiting for 3 Jan 9 22:55:04.227: INFO: Found 1 stateful pods, waiting for 3 Jan 9 22:55:14.227: INFO: Found 1 stateful pods, waiting for 3 Jan 9 22:55:24.228: INFO: Found 1 stateful pods, waiting for 3 Jan 9 22:55:34.227: INFO: Found 1 stateful pods, waiting for 3 Jan 9 22:55:44.226: INFO: Found 1 stateful pods, waiting for 3 Jan 9 22:55:54.226: INFO: Found 1 stateful pods, waiting for 3 Jan 9 22:56:04.225: INFO: Found 1 stateful pods, waiting for 3 Jan 9 22:56:14.227: INFO: Found 1 stateful pods, waiting for 3 Jan 9 22:56:24.226: INFO: Found 1 stateful pods, waiting for 3 Jan 9 22:56:34.227: INFO: Found 1 stateful pods, waiting for 3 Jan 9 22:56:44.227: INFO: Found 1 stateful pods, waiting for 3 Jan 9 22:56:54.226: INFO: Found 1 stateful pods, waiting for 3 Jan 9 22:57:04.226: INFO: Found 1 stateful pods, waiting for 3 Jan 9 22:57:14.226: INFO: Found 1 stateful pods, waiting for 3 Jan 9 22:57:24.226: INFO: Found 1 stateful pods, waiting for 3 Jan 9 22:57:34.228: INFO: Found 1 stateful pods, waiting for 3 Jan 9 22:57:44.226: INFO: Found 1 stateful pods, waiting for 3 Jan 9 22:57:54.227: INFO: Found 1 stateful pods, waiting for 3 Jan 9 22:58:04.227: INFO: Found 1 stateful pods, waiting for 3 Jan 9 22:58:14.226: INFO: Found 1 stateful pods, waiting for 3 Jan 9 22:58:24.227: INFO: Found 1 stateful pods, waiting for 3 Jan 9 22:58:34.227: INFO: Found 1 stateful pods, waiting for 3 Jan 9 22:58:44.229: INFO: Found 1 stateful pods, waiting for 3 Jan 9 22:58:54.227: INFO: Found 1 stateful pods, waiting for 3 Jan 9 22:59:04.226: INFO: Found 1 stateful pods, waiting for 3 Jan 9 22:59:14.227: INFO: Found 1 stateful pods, waiting for 3 Jan 9 22:59:24.227: INFO: Found 1 stateful pods, waiting for 3 Jan 9 22:59:34.227: INFO: Found 1 stateful pods, waiting for 3 Jan 9 22:59:44.226: INFO: Found 1 stateful pods, waiting for 3 Jan 9 22:59:54.226: INFO: Found 1 stateful pods, waiting for 3 Jan 9 23:00:04.226: INFO: Found 1 stateful pods, waiting for 3 Jan 9 23:00:14.227: INFO: Found 1 stateful pods, waiting for 3 Jan 9 23:00:24.228: INFO: Found 1 stateful pods, waiting for 3 Jan 9 23:00:34.226: INFO: Found 1 stateful pods, waiting for 3 Jan 9 23:00:44.226: INFO: Found 1 stateful pods, waiting for 3 Jan 9 23:00:54.226: INFO: Found 1 stateful pods, waiting for 3 Jan 9 23:01:04.226: INFO: Found 1 stateful pods, waiting for 3 Jan 9 23:01:14.227: INFO: Found 1 stateful pods, waiting for 3 Jan 9 23:01:24.225: INFO: Found 1 stateful pods, waiting for 3 Jan 9 23:01:34.225: INFO: Found 1 stateful pods, waiting for 3 Jan 9 23:01:44.225: INFO: Found 1 stateful pods, waiting for 3 Jan 9 23:01:54.226: INFO: Found 1 stateful pods, waiting for 3 Jan 9 23:02:04.226: INFO: Found 1 stateful pods, waiting for 3 Jan 9 23:02:14.226: INFO: Found 1 stateful pods, waiting for 3 Jan 9 23:02:24.227: INFO: Found 1 stateful pods, waiting for 3 Jan 9 23:02:34.227: INFO: Found 1 stateful pods, waiting for 3 Jan 9 23:02:44.226: INFO: Found 1 stateful pods, waiting for 3 Jan 9 23:02:54.225: INFO: Found 1 stateful pods, waiting for 3 Jan 9 23:03:04.226: INFO: Found 1 stateful pods, waiting for 3 Jan 9 23:03:14.225: INFO: Found 1 stateful pods, waiting for 3 Jan 9 23:03:24.226: INFO: Found 1 stateful pods, waiting for 3 Jan 9 23:03:34.225: INFO: Found 1 stateful pods, waiting for 3 Jan 9 23:03:44.225: INFO: Found 1 stateful pods, waiting for 3 Jan 9 23:03:54.226: INFO: Found 1 stateful pods, waiting for 3 Jan 9 23:04:04.225: INFO: Found 1 stateful pods, waiting for 3 Jan 9 23:04:14.225: INFO: Found 1 stateful pods, waiting for 3 Jan 9 23:04:24.226: INFO: Found 1 stateful pods, waiting for 3 Jan 9 23:04:34.226: INFO: Found 1 stateful pods, waiting for 3 Jan 9 23:04:44.227: INFO: Found 1 stateful pods, waiting for 3 Jan 9 23:04:44.259: INFO: Found 1 stateful pods, waiting for 3 Jan 9 23:04:44.259: FAIL: Failed waiting for pods to enter running: timed out waiting for the condition Full Stack Trace k8s.io/kubernetes/test/e2e/framework/statefulset.WaitForRunningAndReady(...) /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/statefulset/wait.go:80 k8s.io/kubernetes/test/e2e/apps.glob..func9.2.9() /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/apps/statefulset.go:532 +0x7aa k8s.io/kubernetes/test/e2e.RunE2ETests(0x24c66d7) _output/dockerized/go/src/k8s.io/kubernetes/test/e2e/e2e.go:133 +0x697 k8s.io/kubernetes/test/e2e.TestE2E(0x243a8f9) _output/dockerized/go/src/k8s.io/kubernetes/test/e2e/e2e_test.go:136 +0x19 testing.tRunner(0xc0000cf040, 0x735e880) /usr/local/go/src/testing/testing.go:1259 +0x102 created by testing.(*T).Run /usr/local/go/src/testing/testing.go:1306 +0x35a [AfterEach] Basic StatefulSet functionality [StatefulSetBasic] /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/apps/statefulset.go:120 Jan 9 23:04:44.292: INFO: Running '/home/prow/go/src/k8s.io/kops/_rundir/792a86bd-906d-11ed-84f7-72cdabe2d9ae/kubectl --server=https://api.e2e-e2e-kops-grid-cilium-eni-u2204-k23-ko26-docker.test-cncf-aws.k8s.io --kubeconfig=/root/.kube/config --namespace=statefulset-6319 describe po ss2-0' Jan 9 23:04:44.561: INFO: stderr: "" Jan 9 23:04:44.561: INFO: stdout: "Name: ss2-0\nNamespace: statefulset-6319\nPriority: 0\nNode: ip-172-20-59-208.ca-central-1.compute.internal/172.20.59.208\nStart Time: Mon, 09 Jan 2023 22:54:44 +0000\nLabels: baz=blah\n controller-revision-hash=ss2-57bbdd95cb\n foo=bar\n statefulset.kubernetes.io/pod-name=ss2-0\nAnnotations: <none>\nStatus: Pending\nIP: \nIPs: <none>\nControlled By: StatefulSet/ss2\nContainers:\n webserver:\n Container ID: \n Image: k8s.gcr.io/e2e-test-images/httpd:2.4.38-2\n Image ID: \n Port: <none>\n Host Port: <none>\n State: Waiting\n Reason: ContainerCreating\n Ready: False\n Restart Count: 0\n Readiness: http-get http://:80/index.html delay=0s timeout=1s period=1s #success=1 #failure=1\n Environment: <none>\n Mounts:\n /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-kmb6k (ro)\nConditions:\n Type Status\n Initialized True \n Ready False \n ContainersReady False \n PodScheduled True \nVolumes:\n kube-api-access-kmb6k:\n Type: Projected (a volume that contains injected data from multiple sources)\n TokenExpirationSeconds: 3607\n ConfigMapName: kube-root-ca.crt\n ConfigMapOptional: <nil>\n DownwardAPI: true\nQoS Class: BestEffort\nNode-Selectors: <none>\nTolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s\n node.kubernetes.io/unreachable:NoExecute op=Exists for 300s\nEvents:\n Type Reason Age From Message\n ---- ------ ---- ---- -------\n Normal Scheduled 10m default-scheduler Successfully assigned statefulset-6319/ss2-0 to ip-172-20-59-208.ca-central-1.compute.internal\n Warning FailedCreatePodSandBox 9m57s kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container \"c35dac424c2b30dd2fc4077cceb670bfbbae6e45451bb837f793c76558560cba\" network for pod \"ss2-0\": networkPlugin cni failed to set up pod \"ss2-0_statefulset-6319\" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available\n Warning FailedCreatePodSandBox 9m54s kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container \"88930abcf06245185597d8e0e3a3a532e197fe46c01473262bbdba8cc9b281df\" network for pod \"ss2-0\": networkPlugin cni failed to set up pod \"ss2-0_statefulset-6319\" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available\n Warning FailedCreatePodSandBox 9m51s kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container \"03cd87f2d6c69be8f56d834740feb4014a6c4d779cb470f30efb0a1f39fbd263\" network for pod \"ss2-0\": networkPlugin cni failed to set up pod \"ss2-0_statefulset-6319\" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available\n Warning FailedCreatePodSandBox 9m49s kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container \"d710b0efa74b726ad4f16014c54c0ac082445373a732ccb1ec18a053a027c4d4\" network for pod \"ss2-0\": networkPlugin cni failed to set up pod \"ss2-0_statefulset-6319\" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available\n Warning FailedCreatePodSandBox 9m46s kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container \"d7d2c56f79d8b91e1a1994e8536bfd8b629ca04d94c2b1ea9ef1c2a7006cbde3\" network for pod \"ss2-0\": networkPlugin cni failed to set up pod \"ss2-0_statefulset-6319\" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available\n Warning FailedCreatePodSandBox 9m43s kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container \"656a7faff0fd9cf6ddb6a0d92b812352bd215ec8671baec32ce01959f23a72bf\" network for pod \"ss2-0\": networkPlugin cni failed to set up pod \"ss2-0_statefulset-6319\" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available\n Warning FailedCreatePodSandBox 9m40s kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container \"1961d45d3ce51bdbb01e319c484de31acfce5f388703e5f50faa3a58dba40223\" network for pod \"ss2-0\": networkPlugin cni failed to set up pod \"ss2-0_statefulset-6319\" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available\n Warning FailedCreatePodSandBox 9m39s kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container \"56bd3e03775104ab7f9e54c93d9c2247a3b4bec3eb90b2a5e2f3922bba4de759\" network for pod \"ss2-0\": networkPlugin cni failed to set up pod \"ss2-0_statefulset-6319\" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available\n Warning FailedCreatePodSandBox 9m36s kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container \"e9c1665e0041f5fa8e28ec5fd9a458e8af8413e2f695b6c573cae0ffcbb57572\" network for pod \"ss2-0\": networkPlugin cni failed to set up pod \"ss2-0_statefulset-6319\" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available\n Warning FailedCreatePodSandBox 9m16s (x4 over 9m24s) kubelet (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container \"5dbe5ae2b090d1bbb6b556e440124c6e1aeaad4f93e458eb812fda38fc9b55fe\" network for pod \"ss2-0\": networkPlugin cni failed to set up pod \"ss2-0_statefulset-6319\" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available\n Normal SandboxChanged 4m55s (x142 over 9m56s) kubelet Pod sandbox changed, it will be killed and re-created.\n" Jan 9 23:04:44.561: INFO: Output of kubectl describe ss2-0: Name: ss2-0 Namespace: statefulset-6319 Priority: 0 Node: ip-172-20-59-208.ca-central-1.compute.internal/172.20.59.208 Start Time: Mon, 09 Jan 2023 22:54:44 +0000 Labels: baz=blah controller-revision-hash=ss2-57bbdd95cb foo=bar statefulset.kubernetes.io/pod-name=ss2-0 Annotations: <none> Status: Pending IP: IPs: <none> Controlled By: StatefulSet/ss2 Containers: webserver: Container ID: Image: k8s.gcr.io/e2e-test-images/httpd:2.4.38-2 Image ID: Port: <none> Host Port: <none> State: Waiting Reason: ContainerCreating Ready: False Restart Count: 0 Readiness: http-get http://:80/index.html delay=0s timeout=1s period=1s #success=1 #failure=1 Environment: <none> Mounts: /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-kmb6k (ro) Conditions: Type Status Initialized True Ready False ContainersReady False PodScheduled True Volumes: kube-api-access-kmb6k: Type: Projected (a volume that contains injected data from multiple sources) TokenExpirationSeconds: 3607 ConfigMapName: kube-root-ca.crt ConfigMapOptional: <nil> DownwardAPI: true QoS Class: BestEffort Node-Selectors: <none> Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s node.kubernetes.io/unreachable:NoExecute op=Exists for 300s Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 10m default-scheduler Successfully assigned statefulset-6319/ss2-0 to ip-172-20-59-208.ca-central-1.compute.internal Warning FailedCreatePodSandBox 9m57s kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "c35dac424c2b30dd2fc4077cceb670bfbbae6e45451bb837f793c76558560cba" network for pod "ss2-0": networkPlugin cni failed to set up pod "ss2-0_statefulset-6319" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Warning FailedCreatePodSandBox 9m54s kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "88930abcf06245185597d8e0e3a3a532e197fe46c01473262bbdba8cc9b281df" network for pod "ss2-0": networkPlugin cni failed to set up pod "ss2-0_statefulset-6319" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Warning FailedCreatePodSandBox 9m51s kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "03cd87f2d6c69be8f56d834740feb4014a6c4d779cb470f30efb0a1f39fbd263" network for pod "ss2-0": networkPlugin cni failed to set up pod "ss2-0_statefulset-6319" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Warning FailedCreatePodSandBox 9m49s kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "d710b0efa74b726ad4f16014c54c0ac082445373a732ccb1ec18a053a027c4d4" network for pod "ss2-0": networkPlugin cni failed to set up pod "ss2-0_statefulset-6319" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Warning FailedCreatePodSandBox 9m46s kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "d7d2c56f79d8b91e1a1994e8536bfd8b629ca04d94c2b1ea9ef1c2a7006cbde3" network for pod "ss2-0": networkPlugin cni failed to set up pod "ss2-0_statefulset-6319" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Warning FailedCreatePodSandBox 9m43s kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "656a7faff0fd9cf6ddb6a0d92b812352bd215ec8671baec32ce01959f23a72bf" network for pod "ss2-0": networkPlugin cni failed to set up pod "ss2-0_statefulset-6319" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Warning FailedCreatePodSandBox 9m40s kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "1961d45d3ce51bdbb01e319c484de31acfce5f388703e5f50faa3a58dba40223" network for pod "ss2-0": networkPlugin cni failed to set up pod "ss2-0_statefulset-6319" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Warning FailedCreatePodSandBox 9m39s kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "56bd3e03775104ab7f9e54c93d9c2247a3b4bec3eb90b2a5e2f3922bba4de759" network for pod "ss2-0": networkPlugin cni failed to set up pod "ss2-0_statefulset-6319" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Warning FailedCreatePodSandBox 9m36s kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "e9c1665e0041f5fa8e28ec5fd9a458e8af8413e2f695b6c573cae0ffcbb57572" network for pod "ss2-0": networkPlugin cni failed to set up pod "ss2-0_statefulset-6319" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Warning FailedCreatePodSandBox 9m16s (x4 over 9m24s) kubelet (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "5dbe5ae2b090d1bbb6b556e440124c6e1aeaad4f93e458eb812fda38fc9b55fe" network for pod "ss2-0": networkPlugin cni failed to set up pod "ss2-0_statefulset-6319" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Normal SandboxChanged 4m55s (x142 over 9m56s) kubelet Pod sandbox changed, it will be killed and re-created. Jan 9 23:04:44.561: INFO: Running '/home/prow/go/src/k8s.io/kops/_rundir/792a86bd-906d-11ed-84f7-72cdabe2d9ae/kubectl --server=https://api.e2e-e2e-kops-grid-cilium-eni-u2204-k23-ko26-docker.test-cncf-aws.k8s.io --kubeconfig=/root/.kube/config --namespace=statefulset-6319 logs ss2-0 --tail=100' Jan 9 23:04:44.807: INFO: rc: 1 Jan 9 23:04:44.807: INFO: Last 100 log lines of ss2-0: Jan 9 23:04:44.807: INFO: Deleting all statefulset in ns statefulset-6319 Jan 9 23:04:44.838: INFO: Scaling statefulset ss2 to 0 Jan 9 23:04:54.969: INFO: Waiting for statefulset status.replicas updated to 0 Jan 9 23:04:55.000: INFO: Deleting statefulset ss2 [AfterEach] [sig-apps] StatefulSet /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:186 �[1mSTEP�[0m: Collecting events from namespace "statefulset-6319". �[1mSTEP�[0m: Found 35 events. Jan 9 23:04:55.128: INFO: At 2023-01-09 22:54:23 +0000 UTC - event for ss2: {statefulset-controller } SuccessfulCreate: create Pod ss2-0 in StatefulSet ss2 successful Jan 9 23:04:55.128: INFO: At 2023-01-09 22:54:23 +0000 UTC - event for ss2-0: {default-scheduler } Scheduled: Successfully assigned statefulset-6319/ss2-0 to ip-172-20-32-67.ca-central-1.compute.internal Jan 9 23:04:55.128: INFO: At 2023-01-09 22:54:25 +0000 UTC - event for ss2-0: {kubelet ip-172-20-32-67.ca-central-1.compute.internal} Pulled: Container image "k8s.gcr.io/e2e-test-images/httpd:2.4.38-2" already present on machine Jan 9 23:04:55.128: INFO: At 2023-01-09 22:54:25 +0000 UTC - event for ss2-0: {kubelet ip-172-20-32-67.ca-central-1.compute.internal} Created: Created container webserver Jan 9 23:04:55.128: INFO: At 2023-01-09 22:54:25 +0000 UTC - event for ss2-0: {kubelet ip-172-20-32-67.ca-central-1.compute.internal} Started: Started container webserver Jan 9 23:04:55.128: INFO: At 2023-01-09 22:54:31 +0000 UTC - event for ss2: {statefulset-controller } SuccessfulCreate: create Pod ss2-1 in StatefulSet ss2 successful Jan 9 23:04:55.128: INFO: At 2023-01-09 22:54:31 +0000 UTC - event for ss2-1: {default-scheduler } Scheduled: Successfully assigned statefulset-6319/ss2-1 to ip-172-20-55-216.ca-central-1.compute.internal Jan 9 23:04:55.128: INFO: At 2023-01-09 22:54:31 +0000 UTC - event for test: {endpoint-controller } FailedToUpdateEndpoint: Failed to update endpoint statefulset-6319/test: Operation cannot be fulfilled on endpoints "test": the object has been modified; please apply your changes to the latest version and try again Jan 9 23:04:55.128: INFO: At 2023-01-09 22:54:32 +0000 UTC - event for ss2-1: {kubelet ip-172-20-55-216.ca-central-1.compute.internal} Started: Started container webserver Jan 9 23:04:55.128: INFO: At 2023-01-09 22:54:32 +0000 UTC - event for ss2-1: {kubelet ip-172-20-55-216.ca-central-1.compute.internal} Created: Created container webserver Jan 9 23:04:55.128: INFO: At 2023-01-09 22:54:32 +0000 UTC - event for ss2-1: {kubelet ip-172-20-55-216.ca-central-1.compute.internal} Pulled: Container image "k8s.gcr.io/e2e-test-images/httpd:2.4.38-2" already present on machine Jan 9 23:04:55.128: INFO: At 2023-01-09 22:54:33 +0000 UTC - event for ss2: {statefulset-controller } SuccessfulCreate: create Pod ss2-2 in StatefulSet ss2 successful Jan 9 23:04:55.128: INFO: At 2023-01-09 22:54:33 +0000 UTC - event for ss2-2: {default-scheduler } Scheduled: Successfully assigned statefulset-6319/ss2-2 to ip-172-20-57-251.ca-central-1.compute.internal Jan 9 23:04:55.128: INFO: At 2023-01-09 22:54:35 +0000 UTC - event for ss2-2: {kubelet ip-172-20-57-251.ca-central-1.compute.internal} Created: Created container webserver Jan 9 23:04:55.128: INFO: At 2023-01-09 22:54:35 +0000 UTC - event for ss2-2: {kubelet ip-172-20-57-251.ca-central-1.compute.internal} Started: Started container webserver Jan 9 23:04:55.128: INFO: At 2023-01-09 22:54:35 +0000 UTC - event for ss2-2: {kubelet ip-172-20-57-251.ca-central-1.compute.internal} Pulled: Container image "k8s.gcr.io/e2e-test-images/httpd:2.4.38-2" already present on machine Jan 9 23:04:55.128: INFO: At 2023-01-09 22:54:44 +0000 UTC - event for ss2-0: {kubelet ip-172-20-32-67.ca-central-1.compute.internal} Unhealthy: Readiness probe failed: Get "http://172.20.44.253:80/index.html": read tcp 172.20.32.67:52398->172.20.44.253:80: read: connection reset by peer Jan 9 23:04:55.128: INFO: At 2023-01-09 22:54:44 +0000 UTC - event for ss2-0: {default-scheduler } Scheduled: Successfully assigned statefulset-6319/ss2-0 to ip-172-20-59-208.ca-central-1.compute.internal Jan 9 23:04:55.128: INFO: At 2023-01-09 22:54:44 +0000 UTC - event for ss2-0: {kubelet ip-172-20-32-67.ca-central-1.compute.internal} Killing: Stopping container webserver Jan 9 23:04:55.128: INFO: At 2023-01-09 22:54:44 +0000 UTC - event for ss2-1: {kubelet ip-172-20-55-216.ca-central-1.compute.internal} Killing: Stopping container webserver Jan 9 23:04:55.128: INFO: At 2023-01-09 22:54:44 +0000 UTC - event for ss2-2: {kubelet ip-172-20-57-251.ca-central-1.compute.internal} Unhealthy: Readiness probe failed: Get "http://172.20.61.215:80/index.html": dial tcp 172.20.61.215:80: connect: connection refused Jan 9 23:04:55.128: INFO: At 2023-01-09 22:54:44 +0000 UTC - event for ss2-2: {kubelet ip-172-20-57-251.ca-central-1.compute.internal} Killing: Stopping container webserver Jan 9 23:04:55.128: INFO: At 2023-01-09 22:54:45 +0000 UTC - event for ss2-1: {kubelet ip-172-20-55-216.ca-central-1.compute.internal} Unhealthy: Readiness probe failed: Get "http://172.20.59.35:80/index.html": context deadline exceeded (Client.Timeout exceeded while awaiting headers) Jan 9 23:04:55.128: INFO: At 2023-01-09 22:54:47 +0000 UTC - event for ss2-0: {kubelet ip-172-20-59-208.ca-central-1.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "c35dac424c2b30dd2fc4077cceb670bfbbae6e45451bb837f793c76558560cba" network for pod "ss2-0": networkPlugin cni failed to set up pod "ss2-0_statefulset-6319" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 9 23:04:55.128: INFO: At 2023-01-09 22:54:48 +0000 UTC - event for ss2-0: {kubelet ip-172-20-59-208.ca-central-1.compute.internal} SandboxChanged: Pod sandbox changed, it will be killed and re-created. Jan 9 23:04:55.128: INFO: At 2023-01-09 22:54:50 +0000 UTC - event for ss2-0: {kubelet ip-172-20-59-208.ca-central-1.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "88930abcf06245185597d8e0e3a3a532e197fe46c01473262bbdba8cc9b281df" network for pod "ss2-0": networkPlugin cni failed to set up pod "ss2-0_statefulset-6319" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 9 23:04:55.128: INFO: At 2023-01-09 22:54:53 +0000 UTC - event for ss2-0: {kubelet ip-172-20-59-208.ca-central-1.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "03cd87f2d6c69be8f56d834740feb4014a6c4d779cb470f30efb0a1f39fbd263" network for pod "ss2-0": networkPlugin cni failed to set up pod "ss2-0_statefulset-6319" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 9 23:04:55.128: INFO: At 2023-01-09 22:54:55 +0000 UTC - event for ss2-0: {kubelet ip-172-20-59-208.ca-central-1.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "d710b0efa74b726ad4f16014c54c0ac082445373a732ccb1ec18a053a027c4d4" network for pod "ss2-0": networkPlugin cni failed to set up pod "ss2-0_statefulset-6319" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 9 23:04:55.128: INFO: At 2023-01-09 22:54:58 +0000 UTC - event for ss2-0: {kubelet ip-172-20-59-208.ca-central-1.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "d7d2c56f79d8b91e1a1994e8536bfd8b629ca04d94c2b1ea9ef1c2a7006cbde3" network for pod "ss2-0": networkPlugin cni failed to set up pod "ss2-0_statefulset-6319" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 9 23:04:55.128: INFO: At 2023-01-09 22:55:01 +0000 UTC - event for ss2-0: {kubelet ip-172-20-59-208.ca-central-1.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "656a7faff0fd9cf6ddb6a0d92b812352bd215ec8671baec32ce01959f23a72bf" network for pod "ss2-0": networkPlugin cni failed to set up pod "ss2-0_statefulset-6319" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 9 23:04:55.128: INFO: At 2023-01-09 22:55:04 +0000 UTC - event for ss2-0: {kubelet ip-172-20-59-208.ca-central-1.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "1961d45d3ce51bdbb01e319c484de31acfce5f388703e5f50faa3a58dba40223" network for pod "ss2-0": networkPlugin cni failed to set up pod "ss2-0_statefulset-6319" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 9 23:04:55.128: INFO: At 2023-01-09 22:55:05 +0000 UTC - event for ss2-0: {kubelet ip-172-20-59-208.ca-central-1.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "56bd3e03775104ab7f9e54c93d9c2247a3b4bec3eb90b2a5e2f3922bba4de759" network for pod "ss2-0": networkPlugin cni failed to set up pod "ss2-0_statefulset-6319" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 9 23:04:55.128: INFO: At 2023-01-09 22:55:08 +0000 UTC - event for ss2-0: {kubelet ip-172-20-59-208.ca-central-1.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "e9c1665e0041f5fa8e28ec5fd9a458e8af8413e2f695b6c573cae0ffcbb57572" network for pod "ss2-0": networkPlugin cni failed to set up pod "ss2-0_statefulset-6319" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 9 23:04:55.128: INFO: At 2023-01-09 22:55:20 +0000 UTC - event for ss2-0: {kubelet ip-172-20-59-208.ca-central-1.compute.internal} FailedCreatePodSandBox: (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "5dbe5ae2b090d1bbb6b556e440124c6e1aeaad4f93e458eb812fda38fc9b55fe" network for pod "ss2-0": networkPlugin cni failed to set up pod "ss2-0_statefulset-6319" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 9 23:04:55.128: INFO: At 2023-01-09 23:04:44 +0000 UTC - event for ss2: {statefulset-controller } SuccessfulDelete: delete Pod ss2-0 in StatefulSet ss2 successful Jan 9 23:04:55.159: INFO: POD NODE PHASE GRACE CONDITIONS Jan 9 23:04:55.159: INFO: Jan 9 23:04:55.191: INFO: Logging node info for node ip-172-20-32-67.ca-central-1.compute.internal Jan 9 23:04:55.223: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-32-67.ca-central-1.compute.internal 1f51981b-a01d-404c-b3a4-6649f301f65d 43460 0 2023-01-09 22:38:45 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ca-central-1 failure-domain.beta.kubernetes.io/zone:ca-central-1a kubelet_cleanup:true kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-32-67.ca-central-1.compute.internal kubernetes.io/os:linux kubernetes.io/role:node node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:ca-central-1a topology.hostpath.csi/node:ip-172-20-32-67.ca-central-1.compute.internal topology.kubernetes.io/region:ca-central-1 topology.kubernetes.io/zone:ca-central-1a] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-0272cebea4c6c3d5d"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kops-controller Update v1 2023-01-09 22:38:45 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}} } {kube-controller-manager Update v1 2023-01-09 22:38:45 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.3.0/24\"":{}}}} } {kubelet Update v1 2023-01-09 22:38:45 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {e2e.test Update v1 2023-01-09 22:54:42 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubelet_cleanup":{}}}} } {kube-controller-manager Update v1 2023-01-09 22:55:59 +0000 UTC FieldsV1 {"f:status":{"f:volumesAttached":{}}} status} {kubelet Update v1 2023-01-09 22:56:00 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:capacity":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.3.0/24,DoNotUseExternalID:,ProviderID:aws:///ca-central-1a/i-0272cebea4c6c3d5d,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.3.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051644416 0} {<nil>} 3956684Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946786816 0} {<nil>} 3854284Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-01-09 23:01:06 +0000 UTC,LastTransitionTime:2023-01-09 22:38:45 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-01-09 23:01:06 +0000 UTC,LastTransitionTime:2023-01-09 22:38:45 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-01-09 23:01:06 +0000 UTC,LastTransitionTime:2023-01-09 22:38:45 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-01-09 23:01:06 +0000 UTC,LastTransitionTime:2023-01-09 22:39:06 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.32.67,},NodeAddress{Type:ExternalIP,Address:35.182.2.102,},NodeAddress{Type:Hostname,Address:ip-172-20-32-67.ca-central-1.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-32-67.ca-central-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-35-182-2-102.ca-central-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2536b48bd6b6dade0ed68af2555085,SystemUUID:ec2536b4-8bd6-b6da-de0e-d68af2555085,BootID:bfb3b0df-aafa-481c-9ac6-f85e5e4580e3,KernelVersion:5.15.0-1026-aws,OSImage:Ubuntu 22.04.1 LTS,ContainerRuntimeVersion:docker://20.10.17,KubeletVersion:v1.23.15,KubeProxyVersion:v1.23.15,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:456569528,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/jessie-dnsutils@sha256:11e6a66017ba4e4b938c1612b7a54a3befcefd354796c04e1dba76873a13518e k8s.gcr.io/e2e-test-images/jessie-dnsutils:1.5],SizeBytes:253371792,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:7e8bdd271312fd25fc5ff5a8f04727be84044eb3d7d8d03611972a6752e2e11e k8s.gcr.io/e2e-test-images/agnhost:2.39],SizeBytes:126872991,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nautilus@sha256:99c0d6f1ad24a1aa1905d9c6534d193f268f7b23f9add2ae6bb41f31094bdd5c k8s.gcr.io/e2e-test-images/nautilus:1.5],SizeBytes:124758741,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:1b9d1b2f36cb2dbee1960e82a9344aeb11bd4c4c03abf5e1853e0559c23855e3 k8s.gcr.io/e2e-test-images/httpd:2.4.38-2],SizeBytes:123781643,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.23.15 registry.k8s.io/kube-proxy-amd64:v1.23.15],SizeBytes:112336036,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:10e231cf07c82dfab5141c1afe548dc734e5fc1f67665ec3982a325d1bd31a9a registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.0],SizeBytes:94471559,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:c8e03f60afa90a28e4bb6ec9a8d0fc36d89de4b7475cf2d613afa793ec969fe0 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.0],SizeBytes:56382329,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:4e74c0492bceddc598de1c90cc5bc14dcda94cb49fa9c5bad9d117c4834b5e08 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.1],SizeBytes:56379269,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:36c31f7e1f433c9634d24f876353e8646246d81a03c4e351202c2644daff1620 k8s.gcr.io/sig-storage/csi-resizer:v1.2.0],SizeBytes:53956308,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:ed98431376c9e944e19a465fe8ea944806714dd95416a0821096c78d66b579bd k8s.gcr.io/sig-storage/csi-snapshotter:v4.1.1],SizeBytes:53531990,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:c5be65d6679efabb969d9b019300d187437ae876f992c40911fd2892bbef3b36 k8s.gcr.io/sig-storage/csi-attacher:v3.2.0],SizeBytes:53500826,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:60ab9b3e6a030d3038c87c0d6bca2930f58d1d72823e6a4af09767dc83b696a2 k8s.gcr.io/sig-storage/csi-attacher:v3.2.1],SizeBytes:53499570,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:017727efcfeb7d053af68e51436ce8e65edbc6ca573720afb4f79c8594036955 registry.k8s.io/coredns/coredns:v1.10.0],SizeBytes:50249337,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:7a5ba58a44e0d749e0767e4e37315bcf6a61f33ce3185c1991848af4db0fb70a k8s.gcr.io/sig-storage/csi-resizer:v1.1.0],SizeBytes:49176472,},ContainerImage{Names:[k8s.gcr.io/sig-storage/hostpathplugin@sha256:232fe80174d60d520d36043103853a1d7ab4b7f3782cf43e45034f04ccda58ce k8s.gcr.io/sig-storage/hostpathplugin:v1.7.1],SizeBytes:29977921,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:a61d309da54641db41fb8f35718f744e9f730d4d0384f8c4b186ddc9f06cbd5f k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.1.0],SizeBytes:19662887,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:19582112,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:2dee3fe5fe861bb66c3a4ac51114f3447a4cd35870e0f2e2b558c7a400d89589 k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.2.0],SizeBytes:18699436,},ContainerImage{Names:[k8s.gcr.io/sig-storage/mock-driver@sha256:a7b517f9e0f42ffade802eef9cefa271372386b85f55b702b493241e58459793 k8s.gcr.io/sig-storage/mock-driver:v4.1.0],SizeBytes:17680993,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:17611980,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:529be2c9770add0cdd0c989115222ea9fc1be430c11095eb9f6dafcf98a36e2b k8s.gcr.io/sig-storage/livenessprobe:v2.4.0],SizeBytes:17175488,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:13616070e3f29de4417eee434a8ef472221c9e51b3d037b5a6b46cef08eb7443 k8s.gcr.io/e2e-test-images/nginx:1.14-2],SizeBytes:16032814,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nonewprivs@sha256:8ac1264691820febacf3aea5d152cbde6d10685731ec14966a9401c6f47a68ac k8s.gcr.io/e2e-test-images/nonewprivs:1.3],SizeBytes:7107254,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf k8s.gcr.io/e2e-test-images/busybox:1.29-2],SizeBytes:1154361,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:39e1e963e5310e9c313bad51523be012ede7b35bb9316517d19089a010356592 k8s.gcr.io/e2e-test-images/busybox:1.29-1],SizeBytes:1154361,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db k8s.gcr.io/pause:3.6 registry.k8s.io/pause:3.6],SizeBytes:682696,},},VolumesInUse:[kubernetes.io/csi/ebs.csi.aws.com^vol-01e8a4a8ff1da0e10],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-01e8a4a8ff1da0e10,DevicePath:,},},Config:nil,},} Jan 9 23:04:55.224: INFO: Logging kubelet events for node ip-172-20-32-67.ca-central-1.compute.internal Jan 9 23:04:55.257: INFO: Logging pods the kubelet thinks is on node ip-172-20-32-67.ca-central-1.compute.internal Jan 9 23:04:55.298: INFO: pod-subpath-test-preprovisionedpv-d8ks started at 2023-01-09 23:00:32 +0000 UTC (1+1 container statuses recorded) Jan 9 23:04:55.298: INFO: Init container init-volume-preprovisionedpv-d8ks ready: false, restart count 0 Jan 9 23:04:55.298: INFO: Container test-container-subpath-preprovisionedpv-d8ks ready: false, restart count 0 Jan 9 23:04:55.298: INFO: cilium-dvsfs started at 2023-01-09 22:38:46 +0000 UTC (1+1 container statuses recorded) Jan 9 23:04:55.298: INFO: Init container clean-cilium-state ready: true, restart count 0 Jan 9 23:04:55.298: INFO: Container cilium-agent ready: true, restart count 0 Jan 9 23:04:55.298: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-gjc2h started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:04:55.298: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:04:55.298: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-8vzqv started at 2023-01-09 22:54:43 +0000 UTC (0+1 container statuses recorded) Jan 9 23:04:55.298: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:04:55.298: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-jb6kp started at 2023-01-09 22:54:43 +0000 UTC (0+1 container statuses recorded) Jan 9 23:04:55.298: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:04:55.298: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-hwbbn started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:04:55.298: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:04:55.298: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-fr98x started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:04:55.298: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:04:55.298: INFO: rs-zn5kz started at 2023-01-09 22:59:56 +0000 UTC (0+1 container statuses recorded) Jan 9 23:04:55.298: INFO: Container donothing ready: true, restart count 0 Jan 9 23:04:55.298: INFO: webserver-pod started at 2023-01-09 23:02:15 +0000 UTC (0+1 container statuses recorded) Jan 9 23:04:55.298: INFO: Container agnhost ready: false, restart count 0 Jan 9 23:04:55.298: INFO: coredns-867df8f45c-sckrz started at 2023-01-09 22:39:23 +0000 UTC (0+1 container statuses recorded) Jan 9 23:04:55.298: INFO: Container coredns ready: true, restart count 0 Jan 9 23:04:55.298: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-44ccv started at 2023-01-09 22:54:43 +0000 UTC (0+1 container statuses recorded) Jan 9 23:04:55.298: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:04:55.298: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-fzrxx started at 2023-01-09 22:54:43 +0000 UTC (0+1 container statuses recorded) Jan 9 23:04:55.298: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:04:55.298: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-mlcl4 started at 2023-01-09 22:54:43 +0000 UTC (0+1 container statuses recorded) Jan 9 23:04:55.298: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:04:55.298: INFO: ebs-csi-node-lc4rb started at 2023-01-09 22:38:46 +0000 UTC (0+3 container statuses recorded) Jan 9 23:04:55.298: INFO: Container ebs-plugin ready: true, restart count 0 Jan 9 23:04:55.298: INFO: Container liveness-probe ready: true, restart count 0 Jan 9 23:04:55.298: INFO: Container node-driver-registrar ready: true, restart count 0 Jan 9 23:04:55.298: INFO: rs-gxqdg started at 2023-01-09 22:59:56 +0000 UTC (0+1 container statuses recorded) Jan 9 23:04:55.298: INFO: Container donothing ready: false, restart count 0 Jan 9 23:04:55.298: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-5pr42 started at 2023-01-09 22:54:43 +0000 UTC (0+1 container statuses recorded) Jan 9 23:04:55.298: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:04:55.298: INFO: inline-volume-tester-5g6j6 started at 2023-01-09 22:55:56 +0000 UTC (0+1 container statuses recorded) Jan 9 23:04:55.298: INFO: Container csi-volume-tester ready: false, restart count 0 Jan 9 23:04:55.298: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-sddjv started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:04:55.298: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:04:55.298: INFO: hostexec-ip-172-20-32-67.ca-central-1.compute.internal-qtq5v started at 2023-01-09 23:00:13 +0000 UTC (0+1 container statuses recorded) Jan 9 23:04:55.298: INFO: Container agnhost-container ready: true, restart count 0 Jan 9 23:04:55.439: INFO: Latency metrics for node ip-172-20-32-67.ca-central-1.compute.internal Jan 9 23:04:55.439: INFO: Logging node info for node ip-172-20-37-64.ca-central-1.compute.internal Jan 9 23:04:55.471: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-37-64.ca-central-1.compute.internal d817613a-0fe5-40c0-a5e4-7b85b9f8506b 44622 0 2023-01-09 22:36:54 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:c5.large beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ca-central-1 failure-domain.beta.kubernetes.io/zone:ca-central-1a kops.k8s.io/kops-controller-pki: kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-37-64.ca-central-1.compute.internal kubernetes.io/os:linux kubernetes.io/role:master node-role.kubernetes.io/control-plane: node-role.kubernetes.io/master: node.kubernetes.io/exclude-from-external-load-balancers: node.kubernetes.io/instance-type:c5.large topology.ebs.csi.aws.com/zone:ca-central-1a topology.kubernetes.io/region:ca-central-1 topology.kubernetes.io/zone:ca-central-1a] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-06217673764fdafeb"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kubelet Update v1 2023-01-09 22:36:54 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {protokube Update v1 2023-01-09 22:37:19 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kops.k8s.io/kops-controller-pki":{},"f:node-role.kubernetes.io/control-plane":{},"f:node.kubernetes.io/exclude-from-external-load-balancers":{}}}} } {kops-controller Update v1 2023-01-09 22:37:21 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/master":{}}}} } {kube-controller-manager Update v1 2023-01-09 22:37:35 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.0.0/24\"":{}},"f:taints":{}}} } {kubelet Update v1 2023-01-09 22:38:39 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.0.0/24,DoNotUseExternalID:,ProviderID:aws:///ca-central-1a/i-06217673764fdafeb,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/master,Value:,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[100.96.0.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3862904832 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3758047232 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-01-09 23:04:17 +0000 UTC,LastTransitionTime:2023-01-09 22:36:48 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-01-09 23:04:17 +0000 UTC,LastTransitionTime:2023-01-09 22:36:48 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-01-09 23:04:17 +0000 UTC,LastTransitionTime:2023-01-09 22:36:48 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-01-09 23:04:17 +0000 UTC,LastTransitionTime:2023-01-09 22:37:35 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.37.64,},NodeAddress{Type:ExternalIP,Address:3.96.216.165,},NodeAddress{Type:Hostname,Address:ip-172-20-37-64.ca-central-1.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-37-64.ca-central-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-3-96-216-165.ca-central-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec27a483f61319a98e5ba2ee2d59e6a3,SystemUUID:ec27a483-f613-19a9-8e5b-a2ee2d59e6a3,BootID:9b41e339-1aaa-4374-8f11-8d34e12cd62a,KernelVersion:5.15.0-1026-aws,OSImage:Ubuntu 22.04.1 LTS,ContainerRuntimeVersion:docker://20.10.17,KubeletVersion:v1.23.15,KubeProxyVersion:v1.23.15,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[registry.k8s.io/etcdadm/etcd-manager@sha256:66a453db625abb268f4b3bbefc5a34a171d81e6e8796cecca54cfd71775c77c4 registry.k8s.io/etcdadm/etcd-manager:v3.0.20221209],SizeBytes:662209237,},ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:456569528,},ContainerImage{Names:[k8s.gcr.io/kube-apiserver-amd64:v1.23.15 registry.k8s.io/kube-apiserver-amd64:v1.23.15],SizeBytes:135240444,},ContainerImage{Names:[k8s.gcr.io/kube-controller-manager-amd64:v1.23.15 registry.k8s.io/kube-controller-manager-amd64:v1.23.15],SizeBytes:125046097,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.23.15 registry.k8s.io/kube-proxy-amd64:v1.23.15],SizeBytes:112336036,},ContainerImage{Names:[quay.io/cilium/operator@sha256:a6d24a006a6b92967ac90786b49bc1ac26e5477cf028cd1186efcfc2466484db quay.io/cilium/operator:v1.12.5],SizeBytes:110016956,},ContainerImage{Names:[registry.k8s.io/kops/kops-controller:1.26.0-beta.1],SizeBytes:104212800,},ContainerImage{Names:[registry.k8s.io/kops/dns-controller:1.26.0-beta.1],SizeBytes:102566451,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:10e231cf07c82dfab5141c1afe548dc734e5fc1f67665ec3982a325d1bd31a9a registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.0],SizeBytes:94471559,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:122bfb8c1edabb3c0edd63f06523e6940d958d19b3957dc7b1d6f81e9f1f6119 registry.k8s.io/sig-storage/csi-provisioner:v3.1.0],SizeBytes:57736070,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:9ebbf9f023e7b41ccee3d52afe39a89e3ddacdbb69269d583abfc25847cfd9e4 registry.k8s.io/sig-storage/csi-resizer:v1.4.0],SizeBytes:55464778,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:8b9c313c05f54fb04f8d430896f5f5904b6cb157df261501b29adc04d2b2dc7b registry.k8s.io/sig-storage/csi-attacher:v3.4.0],SizeBytes:54848595,},ContainerImage{Names:[k8s.gcr.io/kube-scheduler-amd64:v1.23.15 registry.k8s.io/kube-scheduler-amd64:v1.23.15],SizeBytes:53521430,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:19582112,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:17611980,},ContainerImage{Names:[registry.k8s.io/kops/kube-apiserver-healthcheck:1.26.0-beta.1],SizeBytes:8787111,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:682696,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Jan 9 23:04:55.471: INFO: Logging kubelet events for node ip-172-20-37-64.ca-central-1.compute.internal Jan 9 23:04:55.505: INFO: Logging pods the kubelet thinks is on node ip-172-20-37-64.ca-central-1.compute.internal Jan 9 23:04:55.543: INFO: etcd-manager-events-ip-172-20-37-64.ca-central-1.compute.internal started at 2023-01-09 22:36:15 +0000 UTC (0+1 container statuses recorded) Jan 9 23:04:55.543: INFO: Container etcd-manager ready: true, restart count 0 Jan 9 23:04:55.543: INFO: etcd-manager-main-ip-172-20-37-64.ca-central-1.compute.internal started at 2023-01-09 22:36:15 +0000 UTC (0+1 container statuses recorded) Jan 9 23:04:55.543: INFO: Container etcd-manager ready: true, restart count 0 Jan 9 23:04:55.543: INFO: ebs-csi-node-8q4pg started at 2023-01-09 22:37:17 +0000 UTC (0+3 container statuses recorded) Jan 9 23:04:55.543: INFO: Container ebs-plugin ready: true, restart count 0 Jan 9 23:04:55.543: INFO: Container liveness-probe ready: true, restart count 0 Jan 9 23:04:55.543: INFO: Container node-driver-registrar ready: true, restart count 0 Jan 9 23:04:55.543: INFO: cilium-q7lrg started at 2023-01-09 22:37:17 +0000 UTC (1+1 container statuses recorded) Jan 9 23:04:55.543: INFO: Init container clean-cilium-state ready: true, restart count 0 Jan 9 23:04:55.543: INFO: Container cilium-agent ready: true, restart count 0 Jan 9 23:04:55.543: INFO: cilium-operator-c7bfc9f44-w4nqz started at 2023-01-09 22:37:19 +0000 UTC (0+1 container statuses recorded) Jan 9 23:04:55.543: INFO: Container cilium-operator ready: true, restart count 0 Jan 9 23:04:55.543: INFO: ebs-csi-controller-5d8d7cc949-8wxx2 started at 2023-01-09 22:37:19 +0000 UTC (0+5 container statuses recorded) Jan 9 23:04:55.543: INFO: Container csi-attacher ready: true, restart count 0 Jan 9 23:04:55.543: INFO: Container csi-provisioner ready: true, restart count 0 Jan 9 23:04:55.543: INFO: Container csi-resizer ready: true, restart count 0 Jan 9 23:04:55.543: INFO: Container ebs-plugin ready: true, restart count 0 Jan 9 23:04:55.543: INFO: Container liveness-probe ready: true, restart count 0 Jan 9 23:04:55.543: INFO: dns-controller-867784b75c-sxxcl started at 2023-01-09 22:37:20 +0000 UTC (0+1 container statuses recorded) Jan 9 23:04:55.543: INFO: Container dns-controller ready: true, restart count 0 Jan 9 23:04:55.543: INFO: kube-scheduler-ip-172-20-37-64.ca-central-1.compute.internal started at 2023-01-09 22:36:15 +0000 UTC (0+1 container statuses recorded) Jan 9 23:04:55.543: INFO: Container kube-scheduler ready: true, restart count 0 Jan 9 23:04:55.543: INFO: kube-controller-manager-ip-172-20-37-64.ca-central-1.compute.internal started at 2023-01-09 22:36:15 +0000 UTC (0+1 container statuses recorded) Jan 9 23:04:55.543: INFO: Container kube-controller-manager ready: true, restart count 2 Jan 9 23:04:55.543: INFO: kops-controller-cqrv8 started at 2023-01-09 22:37:19 +0000 UTC (0+1 container statuses recorded) Jan 9 23:04:55.543: INFO: Container kops-controller ready: true, restart count 0 Jan 9 23:04:55.543: INFO: kube-apiserver-ip-172-20-37-64.ca-central-1.compute.internal started at 2023-01-09 22:36:15 +0000 UTC (0+2 container statuses recorded) Jan 9 23:04:55.543: INFO: Container healthcheck ready: true, restart count 0 Jan 9 23:04:55.543: INFO: Container kube-apiserver ready: true, restart count 1 Jan 9 23:04:55.660: INFO: Latency metrics for node ip-172-20-37-64.ca-central-1.compute.internal Jan 9 23:04:55.660: INFO: Logging node info for node ip-172-20-55-216.ca-central-1.compute.internal Jan 9 23:04:55.694: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-55-216.ca-central-1.compute.internal 2ceff781-505a-4706-ad8b-e38e37f72571 44079 0 2023-01-09 22:38:47 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ca-central-1 failure-domain.beta.kubernetes.io/zone:ca-central-1a kubelet_cleanup:true kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-55-216.ca-central-1.compute.internal kubernetes.io/os:linux kubernetes.io/role:node node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:ca-central-1a topology.hostpath.csi/node:ip-172-20-55-216.ca-central-1.compute.internal topology.kubernetes.io/region:ca-central-1 topology.kubernetes.io/zone:ca-central-1a] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-0c78741e44993ae7f"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kops-controller Update v1 2023-01-09 22:38:47 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}} } {kube-controller-manager Update v1 2023-01-09 22:38:47 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.4.0/24\"":{}}}} } {kubelet Update v1 2023-01-09 22:38:47 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {e2e.test Update v1 2023-01-09 22:54:42 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubelet_cleanup":{}}}} } {kube-controller-manager Update v1 2023-01-09 22:56:45 +0000 UTC FieldsV1 {"f:status":{"f:volumesAttached":{}}} status} {kubelet Update v1 2023-01-09 22:56:51 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:capacity":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.4.0/24,DoNotUseExternalID:,ProviderID:aws:///ca-central-1a/i-0c78741e44993ae7f,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.4.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051644416 0} {<nil>} 3956684Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946786816 0} {<nil>} 3854284Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-01-09 23:02:48 +0000 UTC,LastTransitionTime:2023-01-09 22:38:47 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-01-09 23:02:48 +0000 UTC,LastTransitionTime:2023-01-09 22:38:47 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-01-09 23:02:48 +0000 UTC,LastTransitionTime:2023-01-09 22:38:47 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-01-09 23:02:48 +0000 UTC,LastTransitionTime:2023-01-09 22:39:07 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.55.216,},NodeAddress{Type:ExternalIP,Address:15.222.60.56,},NodeAddress{Type:Hostname,Address:ip-172-20-55-216.ca-central-1.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-55-216.ca-central-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-15-222-60-56.ca-central-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec249e6a3570a28ebd4276206d3ae523,SystemUUID:ec249e6a-3570-a28e-bd42-76206d3ae523,BootID:d58dcef2-849d-4807-9ba5-0438ebaa05b9,KernelVersion:5.15.0-1026-aws,OSImage:Ubuntu 22.04.1 LTS,ContainerRuntimeVersion:docker://20.10.17,KubeletVersion:v1.23.15,KubeProxyVersion:v1.23.15,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:456569528,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/jessie-dnsutils@sha256:11e6a66017ba4e4b938c1612b7a54a3befcefd354796c04e1dba76873a13518e k8s.gcr.io/e2e-test-images/jessie-dnsutils:1.5],SizeBytes:253371792,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:20f25f275d46aa728f7615a1ccc19c78b2ed89435bf943a44b339f70f45508e6 k8s.gcr.io/e2e-test-images/httpd:2.4.39-2],SizeBytes:126894770,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:7e8bdd271312fd25fc5ff5a8f04727be84044eb3d7d8d03611972a6752e2e11e k8s.gcr.io/e2e-test-images/agnhost:2.39],SizeBytes:126872991,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nautilus@sha256:99c0d6f1ad24a1aa1905d9c6534d193f268f7b23f9add2ae6bb41f31094bdd5c k8s.gcr.io/e2e-test-images/nautilus:1.5],SizeBytes:124758741,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:1b9d1b2f36cb2dbee1960e82a9344aeb11bd4c4c03abf5e1853e0559c23855e3 k8s.gcr.io/e2e-test-images/httpd:2.4.38-2],SizeBytes:123781643,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.23.15 registry.k8s.io/kube-proxy-amd64:v1.23.15],SizeBytes:112336036,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:10e231cf07c82dfab5141c1afe548dc734e5fc1f67665ec3982a325d1bd31a9a registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.0],SizeBytes:94471559,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:c8e03f60afa90a28e4bb6ec9a8d0fc36d89de4b7475cf2d613afa793ec969fe0 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.0],SizeBytes:56382329,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:4e74c0492bceddc598de1c90cc5bc14dcda94cb49fa9c5bad9d117c4834b5e08 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.1],SizeBytes:56379269,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:36c31f7e1f433c9634d24f876353e8646246d81a03c4e351202c2644daff1620 k8s.gcr.io/sig-storage/csi-resizer:v1.2.0],SizeBytes:53956308,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:ed98431376c9e944e19a465fe8ea944806714dd95416a0821096c78d66b579bd k8s.gcr.io/sig-storage/csi-snapshotter:v4.1.1],SizeBytes:53531990,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:c5be65d6679efabb969d9b019300d187437ae876f992c40911fd2892bbef3b36 k8s.gcr.io/sig-storage/csi-attacher:v3.2.0],SizeBytes:53500826,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:60ab9b3e6a030d3038c87c0d6bca2930f58d1d72823e6a4af09767dc83b696a2 k8s.gcr.io/sig-storage/csi-attacher:v3.2.1],SizeBytes:53499570,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:7a5ba58a44e0d749e0767e4e37315bcf6a61f33ce3185c1991848af4db0fb70a k8s.gcr.io/sig-storage/csi-resizer:v1.1.0],SizeBytes:49176472,},ContainerImage{Names:[k8s.gcr.io/sig-storage/hostpathplugin@sha256:232fe80174d60d520d36043103853a1d7ab4b7f3782cf43e45034f04ccda58ce k8s.gcr.io/sig-storage/hostpathplugin:v1.7.1],SizeBytes:29977921,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:a61d309da54641db41fb8f35718f744e9f730d4d0384f8c4b186ddc9f06cbd5f k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.1.0],SizeBytes:19662887,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:19582112,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:2dee3fe5fe861bb66c3a4ac51114f3447a4cd35870e0f2e2b558c7a400d89589 k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.2.0],SizeBytes:18699436,},ContainerImage{Names:[k8s.gcr.io/sig-storage/mock-driver@sha256:a7b517f9e0f42ffade802eef9cefa271372386b85f55b702b493241e58459793 k8s.gcr.io/sig-storage/mock-driver:v4.1.0],SizeBytes:17680993,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:17611980,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:529be2c9770add0cdd0c989115222ea9fc1be430c11095eb9f6dafcf98a36e2b k8s.gcr.io/sig-storage/livenessprobe:v2.4.0],SizeBytes:17175488,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:13616070e3f29de4417eee434a8ef472221c9e51b3d037b5a6b46cef08eb7443 k8s.gcr.io/e2e-test-images/nginx:1.14-2],SizeBytes:16032814,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf k8s.gcr.io/e2e-test-images/busybox:1.29-2],SizeBytes:1154361,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db k8s.gcr.io/pause:3.6 registry.k8s.io/pause:3.6],SizeBytes:682696,},},VolumesInUse:[kubernetes.io/csi/ebs.csi.aws.com^vol-07a7954e90ced7ccc kubernetes.io/csi/ebs.csi.aws.com^vol-0c8e4ae9d44bf0644],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-0c8e4ae9d44bf0644,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-07a7954e90ced7ccc,DevicePath:,},},Config:nil,},} Jan 9 23:04:55.694: INFO: Logging kubelet events for node ip-172-20-55-216.ca-central-1.compute.internal Jan 9 23:04:55.728: INFO: Logging pods the kubelet thinks is on node ip-172-20-55-216.ca-central-1.compute.internal Jan 9 23:04:55.776: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-qglk4 started at 2023-01-09 22:54:43 +0000 UTC (0+1 container statuses recorded) Jan 9 23:04:55.776: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:04:55.776: INFO: cilium-dqpw4 started at 2023-01-09 22:38:48 +0000 UTC (1+1 container statuses recorded) Jan 9 23:04:55.776: INFO: Init container clean-cilium-state ready: true, restart count 0 Jan 9 23:04:55.776: INFO: Container cilium-agent ready: true, restart count 0 Jan 9 23:04:55.776: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-b7bxr started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:04:55.776: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:04:55.776: INFO: externalname-service-cfh5p started at 2023-01-09 22:56:39 +0000 UTC (0+1 container statuses recorded) Jan 9 23:04:55.776: INFO: Container externalname-service ready: true, restart count 0 Jan 9 23:04:55.776: INFO: inline-volume-tester-7mx4s started at 2023-01-09 22:56:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:04:55.776: INFO: Container csi-volume-tester ready: false, restart count 0 Jan 9 23:04:55.776: INFO: up-down-1-xblpx started at 2023-01-09 23:00:56 +0000 UTC (0+1 container statuses recorded) Jan 9 23:04:55.776: INFO: Container up-down-1 ready: false, restart count 0 Jan 9 23:04:55.776: INFO: rs-lf7br started at 2023-01-09 22:59:56 +0000 UTC (0+1 container statuses recorded) Jan 9 23:04:55.776: INFO: Container donothing ready: false, restart count 0 Jan 9 23:04:55.776: INFO: csi-mockplugin-0 started at 2023-01-09 23:02:58 +0000 UTC (0+3 container statuses recorded) Jan 9 23:04:55.776: INFO: Container csi-provisioner ready: false, restart count 0 Jan 9 23:04:55.776: INFO: Container driver-registrar ready: false, restart count 0 Jan 9 23:04:55.776: INFO: Container mock ready: false, restart count 0 Jan 9 23:04:55.776: INFO: pod-subpath-test-projected-tqfk started at 2023-01-09 23:03:15 +0000 UTC (0+1 container statuses recorded) Jan 9 23:04:55.776: INFO: Container test-container-subpath-projected-tqfk ready: false, restart count 0 Jan 9 23:04:55.776: INFO: fail-once-non-local-mh6qc started at 2023-01-09 22:56:53 +0000 UTC (0+1 container statuses recorded) Jan 9 23:04:55.776: INFO: Container c ready: false, restart count 0 Jan 9 23:04:55.776: INFO: pod-9ba7c444-540c-4612-ad00-c7c5dd3438eb started at 2023-01-09 23:02:45 +0000 UTC (0+1 container statuses recorded) Jan 9 23:04:55.776: INFO: Container write-pod ready: false, restart count 0 Jan 9 23:04:55.776: INFO: echo-sourceip started at 2023-01-09 23:00:54 +0000 UTC (0+1 container statuses recorded) Jan 9 23:04:55.776: INFO: Container agnhost-container ready: false, restart count 0 Jan 9 23:04:55.776: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-mcd88 started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:04:55.776: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:04:55.776: INFO: fail-once-non-local-nx7cr started at 2023-01-09 22:56:53 +0000 UTC (0+1 container statuses recorded) Jan 9 23:04:55.776: INFO: Container c ready: false, restart count 0 Jan 9 23:04:55.776: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-6nqvb started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:04:55.776: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:04:55.776: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-8kb86 started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:04:55.776: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:04:55.776: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-vwd6k started at 2023-01-09 22:54:43 +0000 UTC (0+1 container statuses recorded) Jan 9 23:04:55.776: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:04:55.776: INFO: hostpath-symlink-prep-provisioning-3662 started at 2023-01-09 23:02:14 +0000 UTC (0+1 container statuses recorded) Jan 9 23:04:55.776: INFO: Container init-volume-provisioning-3662 ready: false, restart count 0 Jan 9 23:04:55.776: INFO: ebs-csi-node-psq97 started at 2023-01-09 22:38:48 +0000 UTC (0+3 container statuses recorded) Jan 9 23:04:55.776: INFO: Container ebs-plugin ready: true, restart count 0 Jan 9 23:04:55.776: INFO: Container liveness-probe ready: true, restart count 0 Jan 9 23:04:55.776: INFO: Container node-driver-registrar ready: true, restart count 0 Jan 9 23:04:55.776: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-qj2bw started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:04:55.776: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:04:55.776: INFO: ss2-0 started at 2023-01-09 22:57:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:04:55.776: INFO: Container webserver ready: false, restart count 0 Jan 9 23:04:55.776: INFO: execpod7wsq2 started at 2023-01-09 23:00:27 +0000 UTC (0+1 container statuses recorded) Jan 9 23:04:55.776: INFO: Container agnhost-container ready: false, restart count 0 Jan 9 23:04:55.776: INFO: rs-ctw9c started at 2023-01-09 22:59:56 +0000 UTC (0+1 container statuses recorded) Jan 9 23:04:55.776: INFO: Container donothing ready: true, restart count 0 Jan 9 23:04:55.776: INFO: rs-j4qxp started at 2023-01-09 22:59:56 +0000 UTC (0+1 container statuses recorded) Jan 9 23:04:55.776: INFO: Container donothing ready: false, restart count 0 Jan 9 23:04:55.776: INFO: pod-projected-configmaps-5a177488-8c20-4e67-9a1a-4a9090cd5bc9 started at 2023-01-09 23:02:21 +0000 UTC (0+1 container statuses recorded) Jan 9 23:04:55.776: INFO: Container agnhost-container ready: false, restart count 0 Jan 9 23:04:55.776: INFO: csi-mockplugin-resizer-0 started at 2023-01-09 23:02:58 +0000 UTC (0+1 container statuses recorded) Jan 9 23:04:55.776: INFO: Container csi-resizer ready: false, restart count 0 Jan 9 23:04:55.776: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-6ks7b started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:04:55.776: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:04:55.776: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-72kwd started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:04:55.776: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:04:55.776: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-krwzl started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:04:55.777: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:04:56.310: INFO: Latency metrics for node ip-172-20-55-216.ca-central-1.compute.internal Jan 9 23:04:56.310: INFO: Logging node info for node ip-172-20-57-251.ca-central-1.compute.internal Jan 9 23:04:56.342: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-57-251.ca-central-1.compute.internal eaaaf6eb-cbd5-4763-9c39-fb0e1a4ce44a 44424 0 2023-01-09 22:38:45 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ca-central-1 failure-domain.beta.kubernetes.io/zone:ca-central-1a kubelet_cleanup:true kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-57-251.ca-central-1.compute.internal kubernetes.io/os:linux kubernetes.io/role:node node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:ca-central-1a topology.hostpath.csi/node:ip-172-20-57-251.ca-central-1.compute.internal topology.kubernetes.io/region:ca-central-1 topology.kubernetes.io/zone:ca-central-1a] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-0513e7d6e5eba8eec"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kops-controller Update v1 2023-01-09 22:38:45 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}} } {kube-controller-manager Update v1 2023-01-09 22:38:45 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.2.0/24\"":{}}}} } {kubelet Update v1 2023-01-09 22:38:45 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {e2e.test Update v1 2023-01-09 22:54:42 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubelet_cleanup":{}}}} } {kube-controller-manager Update v1 2023-01-09 23:00:23 +0000 UTC FieldsV1 {"f:status":{"f:volumesAttached":{}}} status} {kubelet Update v1 2023-01-09 23:00:27 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.2.0/24,DoNotUseExternalID:,ProviderID:aws:///ca-central-1a/i-0513e7d6e5eba8eec,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.2.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051652608 0} {<nil>} 3956692Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946795008 0} {<nil>} 3854292Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-01-09 23:03:22 +0000 UTC,LastTransitionTime:2023-01-09 22:38:45 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-01-09 23:03:22 +0000 UTC,LastTransitionTime:2023-01-09 22:38:45 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-01-09 23:03:22 +0000 UTC,LastTransitionTime:2023-01-09 22:38:45 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-01-09 23:03:22 +0000 UTC,LastTransitionTime:2023-01-09 22:39:05 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.57.251,},NodeAddress{Type:ExternalIP,Address:3.96.144.209,},NodeAddress{Type:Hostname,Address:ip-172-20-57-251.ca-central-1.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-57-251.ca-central-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-3-96-144-209.ca-central-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2f0d7ba95b37a96a1d9360f6071d84,SystemUUID:ec2f0d7b-a95b-37a9-6a1d-9360f6071d84,BootID:67865282-1911-4e38-8675-a7b78d5ae45e,KernelVersion:5.15.0-1026-aws,OSImage:Ubuntu 22.04.1 LTS,ContainerRuntimeVersion:docker://20.10.17,KubeletVersion:v1.23.15,KubeProxyVersion:v1.23.15,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:456569528,},ContainerImage{Names:[k8s.gcr.io/etcd@sha256:dd75ec974b0a2a6f6bb47001ba09207976e625db898d1b16735528c009cb171c k8s.gcr.io/etcd:3.5.6-0],SizeBytes:299475478,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/jessie-dnsutils@sha256:11e6a66017ba4e4b938c1612b7a54a3befcefd354796c04e1dba76873a13518e k8s.gcr.io/e2e-test-images/jessie-dnsutils:1.5],SizeBytes:253371792,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:20f25f275d46aa728f7615a1ccc19c78b2ed89435bf943a44b339f70f45508e6 k8s.gcr.io/e2e-test-images/httpd:2.4.39-2],SizeBytes:126894770,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:7e8bdd271312fd25fc5ff5a8f04727be84044eb3d7d8d03611972a6752e2e11e k8s.gcr.io/e2e-test-images/agnhost:2.39],SizeBytes:126872991,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nautilus@sha256:99c0d6f1ad24a1aa1905d9c6534d193f268f7b23f9add2ae6bb41f31094bdd5c k8s.gcr.io/e2e-test-images/nautilus:1.5],SizeBytes:124758741,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:1b9d1b2f36cb2dbee1960e82a9344aeb11bd4c4c03abf5e1853e0559c23855e3 k8s.gcr.io/e2e-test-images/httpd:2.4.38-2],SizeBytes:123781643,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.23.15 registry.k8s.io/kube-proxy-amd64:v1.23.15],SizeBytes:112336036,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:10e231cf07c82dfab5141c1afe548dc734e5fc1f67665ec3982a325d1bd31a9a registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.0],SizeBytes:94471559,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/sample-apiserver@sha256:f9c93b92b6ff750b41a93c4e4fe0bfe384597aeb841e2539d5444815c55b2d8f k8s.gcr.io/e2e-test-images/sample-apiserver:1.17.5],SizeBytes:56504541,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:c8e03f60afa90a28e4bb6ec9a8d0fc36d89de4b7475cf2d613afa793ec969fe0 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.0],SizeBytes:56382329,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:4e74c0492bceddc598de1c90cc5bc14dcda94cb49fa9c5bad9d117c4834b5e08 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.1],SizeBytes:56379269,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:36c31f7e1f433c9634d24f876353e8646246d81a03c4e351202c2644daff1620 k8s.gcr.io/sig-storage/csi-resizer:v1.2.0],SizeBytes:53956308,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:ed98431376c9e944e19a465fe8ea944806714dd95416a0821096c78d66b579bd k8s.gcr.io/sig-storage/csi-snapshotter:v4.1.1],SizeBytes:53531990,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:c5be65d6679efabb969d9b019300d187437ae876f992c40911fd2892bbef3b36 k8s.gcr.io/sig-storage/csi-attacher:v3.2.0],SizeBytes:53500826,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:60ab9b3e6a030d3038c87c0d6bca2930f58d1d72823e6a4af09767dc83b696a2 k8s.gcr.io/sig-storage/csi-attacher:v3.2.1],SizeBytes:53499570,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:7a5ba58a44e0d749e0767e4e37315bcf6a61f33ce3185c1991848af4db0fb70a k8s.gcr.io/sig-storage/csi-resizer:v1.1.0],SizeBytes:49176472,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/regression-issue-74839@sha256:b4f1d8d61bdad84bd50442d161d5460e4019d53e989b64220fdbc62fc87d76bf k8s.gcr.io/e2e-test-images/regression-issue-74839:1.2],SizeBytes:44576952,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nonroot@sha256:b9e2958a3dd879e3cf11142228c6d073d0fc4ea2e857c3be6f4fb0ab5fb2c937 k8s.gcr.io/e2e-test-images/nonroot:1.2],SizeBytes:42321438,},ContainerImage{Names:[k8s.gcr.io/sig-storage/hostpathplugin@sha256:232fe80174d60d520d36043103853a1d7ab4b7f3782cf43e45034f04ccda58ce k8s.gcr.io/sig-storage/hostpathplugin:v1.7.1],SizeBytes:29977921,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:a61d309da54641db41fb8f35718f744e9f730d4d0384f8c4b186ddc9f06cbd5f k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.1.0],SizeBytes:19662887,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:19582112,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:2dee3fe5fe861bb66c3a4ac51114f3447a4cd35870e0f2e2b558c7a400d89589 k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.2.0],SizeBytes:18699436,},ContainerImage{Names:[k8s.gcr.io/sig-storage/mock-driver@sha256:a7b517f9e0f42ffade802eef9cefa271372386b85f55b702b493241e58459793 k8s.gcr.io/sig-storage/mock-driver:v4.1.0],SizeBytes:17680993,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:17611980,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:529be2c9770add0cdd0c989115222ea9fc1be430c11095eb9f6dafcf98a36e2b k8s.gcr.io/sig-storage/livenessprobe:v2.4.0],SizeBytes:17175488,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:13616070e3f29de4417eee434a8ef472221c9e51b3d037b5a6b46cef08eb7443 k8s.gcr.io/e2e-test-images/nginx:1.14-2],SizeBytes:16032814,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nonewprivs@sha256:8ac1264691820febacf3aea5d152cbde6d10685731ec14966a9401c6f47a68ac k8s.gcr.io/e2e-test-images/nonewprivs:1.3],SizeBytes:7107254,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf k8s.gcr.io/e2e-test-images/busybox:1.29-2],SizeBytes:1154361,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:39e1e963e5310e9c313bad51523be012ede7b35bb9316517d19089a010356592 k8s.gcr.io/e2e-test-images/busybox:1.29-1],SizeBytes:1154361,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db k8s.gcr.io/pause:3.6 registry.k8s.io/pause:3.6],SizeBytes:682696,},},VolumesInUse:[kubernetes.io/csi/ebs.csi.aws.com^vol-02daff5e76e72db98 kubernetes.io/csi/ebs.csi.aws.com^vol-0f7b5a5273a8a182c],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-0f7b5a5273a8a182c,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-02daff5e76e72db98,DevicePath:,},},Config:nil,},} Jan 9 23:04:56.342: INFO: Logging kubelet events for node ip-172-20-57-251.ca-central-1.compute.internal Jan 9 23:04:56.376: INFO: Logging pods the kubelet thinks is on node ip-172-20-57-251.ca-central-1.compute.internal Jan 9 23:04:56.417: INFO: pvc-tester-p5xsb started at 2023-01-09 23:00:21 +0000 UTC (0+1 container statuses recorded) Jan 9 23:04:56.417: INFO: Container write-pod ready: false, restart count 0 Jan 9 23:04:56.417: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-zvfdw started at 2023-01-09 22:54:43 +0000 UTC (0+1 container statuses recorded) Jan 9 23:04:56.417: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:04:56.417: INFO: pod-subpath-test-preprovisionedpv-vjmv started at 2023-01-09 23:00:18 +0000 UTC (2+1 container statuses recorded) Jan 9 23:04:56.417: INFO: Init container init-volume-preprovisionedpv-vjmv ready: false, restart count 0 Jan 9 23:04:56.417: INFO: Init container test-init-volume-preprovisionedpv-vjmv ready: false, restart count 0 Jan 9 23:04:56.417: INFO: Container test-container-subpath-preprovisionedpv-vjmv ready: false, restart count 0 Jan 9 23:04:56.417: INFO: cilium-6p9jz started at 2023-01-09 22:38:46 +0000 UTC (1+1 container statuses recorded) Jan 9 23:04:56.417: INFO: Init container clean-cilium-state ready: true, restart count 0 Jan 9 23:04:56.417: INFO: Container cilium-agent ready: true, restart count 0 Jan 9 23:04:56.417: INFO: pod-submit-remove-4957ac0e-2b88-4d9b-ad07-823bdb0ff487 started at 2023-01-09 23:01:51 +0000 UTC (0+1 container statuses recorded) Jan 9 23:04:56.417: INFO: Container agnhost-container ready: false, restart count 0 Jan 9 23:04:56.417: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-wvflq started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:04:56.417: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:04:56.417: INFO: pod-configmaps-a60c74d4-3fef-4d36-bcd7-7b71e3017d26 started at 2023-01-09 23:01:16 +0000 UTC (0+1 container statuses recorded) Jan 9 23:04:56.417: INFO: Container agnhost-container ready: false, restart count 0 Jan 9 23:04:56.417: INFO: up-down-1-s85rz started at 2023-01-09 23:00:56 +0000 UTC (0+1 container statuses recorded) Jan 9 23:04:56.417: INFO: Container up-down-1 ready: false, restart count 0 Jan 9 23:04:56.417: INFO: rs-56t9b started at 2023-01-09 22:59:56 +0000 UTC (0+1 container statuses recorded) Jan 9 23:04:56.417: INFO: Container donothing ready: true, restart count 0 Jan 9 23:04:56.417: INFO: externalname-service-k4b9s started at 2023-01-09 22:56:39 +0000 UTC (0+1 container statuses recorded) Jan 9 23:04:56.417: INFO: Container externalname-service ready: true, restart count 0 Jan 9 23:04:56.417: INFO: hostexec-ip-172-20-57-251.ca-central-1.compute.internal-d92vh started at 2023-01-09 23:00:07 +0000 UTC (0+1 container statuses recorded) Jan 9 23:04:56.417: INFO: Container agnhost-container ready: true, restart count 0 Jan 9 23:04:56.417: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-tvxfq started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:04:56.417: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:04:56.417: INFO: csi-mockplugin-0 started at 2023-01-09 23:00:20 +0000 UTC (0+3 container statuses recorded) Jan 9 23:04:56.417: INFO: Container csi-provisioner ready: false, restart count 0 Jan 9 23:04:56.417: INFO: Container driver-registrar ready: false, restart count 0 Jan 9 23:04:56.417: INFO: Container mock ready: false, restart count 0 Jan 9 23:04:56.417: INFO: pod-d16d48d3-63ef-4a76-95b8-d80f8b4aec62 started at 2023-01-09 23:03:17 +0000 UTC (0+1 container statuses recorded) Jan 9 23:04:56.417: INFO: Container write-pod ready: false, restart count 0 Jan 9 23:04:56.417: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-m8cfz started at 2023-01-09 22:54:43 +0000 UTC (0+1 container statuses recorded) Jan 9 23:04:56.417: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:04:56.417: INFO: rs-49kqp started at 2023-01-09 22:59:56 +0000 UTC (0+1 container statuses recorded) Jan 9 23:04:56.417: INFO: Container donothing ready: false, restart count 0 Jan 9 23:04:56.417: INFO: rs-dzbwc started at 2023-01-09 22:59:56 +0000 UTC (0+1 container statuses recorded) Jan 9 23:04:56.417: INFO: Container donothing ready: false, restart count 0 Jan 9 23:04:56.417: INFO: ebs-csi-node-g4ngc started at 2023-01-09 22:38:46 +0000 UTC (0+3 container statuses recorded) Jan 9 23:04:56.417: INFO: Container ebs-plugin ready: true, restart count 0 Jan 9 23:04:56.417: INFO: Container liveness-probe ready: true, restart count 0 Jan 9 23:04:56.417: INFO: Container node-driver-registrar ready: true, restart count 0 Jan 9 23:04:56.417: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-wmqwf started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:04:56.417: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:04:56.417: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-96np5 started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:04:56.417: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:04:56.417: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-f9w29 started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:04:56.417: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:04:56.417: INFO: csi-mockplugin-attacher-0 started at 2023-01-09 23:00:20 +0000 UTC (0+1 container statuses recorded) Jan 9 23:04:56.417: INFO: Container csi-attacher ready: false, restart count 0 Jan 9 23:04:56.417: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-rbqs2 started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:04:56.417: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:04:56.417: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-qzf5t started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:04:56.417: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:04:56.417: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-zx8gm started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:04:56.417: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:04:56.693: INFO: Latency metrics for node ip-172-20-57-251.ca-central-1.compute.internal Jan 9 23:04:56.693: INFO: Logging node info for node ip-172-20-59-208.ca-central-1.compute.internal Jan 9 23:04:56.724: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-59-208.ca-central-1.compute.internal d061a4d4-9ff6-43fc-89cc-422c61f7441c 43505 0 2023-01-09 22:38:43 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ca-central-1 failure-domain.beta.kubernetes.io/zone:ca-central-1a kubelet_cleanup:true kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-59-208.ca-central-1.compute.internal kubernetes.io/os:linux kubernetes.io/role:node node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:ca-central-1a topology.hostpath.csi/node:ip-172-20-59-208.ca-central-1.compute.internal topology.kubernetes.io/region:ca-central-1 topology.kubernetes.io/zone:ca-central-1a] map[csi.volume.kubernetes.io/nodeid:{"csi-mock-csi-mock-volumes-5722":"csi-mock-csi-mock-volumes-5722","ebs.csi.aws.com":"i-0f00ea8bafbacdd11"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kops-controller Update v1 2023-01-09 22:38:43 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}} } {kube-controller-manager Update v1 2023-01-09 22:38:43 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.1.0/24\"":{}}}} } {kubelet Update v1 2023-01-09 22:38:43 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {e2e.test Update v1 2023-01-09 22:54:42 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubelet_cleanup":{}}}} } {kube-controller-manager Update v1 2023-01-09 23:01:13 +0000 UTC FieldsV1 {"f:status":{"f:volumesAttached":{}}} status} {kubelet Update v1 2023-01-09 23:01:13 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:capacity":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.1.0/24,DoNotUseExternalID:,ProviderID:aws:///ca-central-1a/i-0f00ea8bafbacdd11,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.1.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051652608 0} {<nil>} 3956692Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946795008 0} {<nil>} 3854292Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-01-09 23:01:13 +0000 UTC,LastTransitionTime:2023-01-09 22:38:43 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-01-09 23:01:13 +0000 UTC,LastTransitionTime:2023-01-09 22:38:43 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-01-09 23:01:13 +0000 UTC,LastTransitionTime:2023-01-09 22:38:43 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-01-09 23:01:13 +0000 UTC,LastTransitionTime:2023-01-09 22:39:03 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.59.208,},NodeAddress{Type:ExternalIP,Address:99.79.51.95,},NodeAddress{Type:Hostname,Address:ip-172-20-59-208.ca-central-1.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-59-208.ca-central-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-99-79-51-95.ca-central-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec28d5ecfae27d4bcedd89eca1a365f0,SystemUUID:ec28d5ec-fae2-7d4b-cedd-89eca1a365f0,BootID:560320f7-12c5-44f2-a7f6-a7a40031e204,KernelVersion:5.15.0-1026-aws,OSImage:Ubuntu 22.04.1 LTS,ContainerRuntimeVersion:docker://20.10.17,KubeletVersion:v1.23.15,KubeProxyVersion:v1.23.15,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:456569528,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:20f25f275d46aa728f7615a1ccc19c78b2ed89435bf943a44b339f70f45508e6 k8s.gcr.io/e2e-test-images/httpd:2.4.39-2],SizeBytes:126894770,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:7e8bdd271312fd25fc5ff5a8f04727be84044eb3d7d8d03611972a6752e2e11e k8s.gcr.io/e2e-test-images/agnhost:2.39],SizeBytes:126872991,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:1b9d1b2f36cb2dbee1960e82a9344aeb11bd4c4c03abf5e1853e0559c23855e3 k8s.gcr.io/e2e-test-images/httpd:2.4.38-2],SizeBytes:123781643,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.23.15 registry.k8s.io/kube-proxy-amd64:v1.23.15],SizeBytes:112336036,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:10e231cf07c82dfab5141c1afe548dc734e5fc1f67665ec3982a325d1bd31a9a registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.0],SizeBytes:94471559,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:c8e03f60afa90a28e4bb6ec9a8d0fc36d89de4b7475cf2d613afa793ec969fe0 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.0],SizeBytes:56382329,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:4e74c0492bceddc598de1c90cc5bc14dcda94cb49fa9c5bad9d117c4834b5e08 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.1],SizeBytes:56379269,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:36c31f7e1f433c9634d24f876353e8646246d81a03c4e351202c2644daff1620 k8s.gcr.io/sig-storage/csi-resizer:v1.2.0],SizeBytes:53956308,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:ed98431376c9e944e19a465fe8ea944806714dd95416a0821096c78d66b579bd k8s.gcr.io/sig-storage/csi-snapshotter:v4.1.1],SizeBytes:53531990,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:c5be65d6679efabb969d9b019300d187437ae876f992c40911fd2892bbef3b36 k8s.gcr.io/sig-storage/csi-attacher:v3.2.0],SizeBytes:53500826,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:60ab9b3e6a030d3038c87c0d6bca2930f58d1d72823e6a4af09767dc83b696a2 k8s.gcr.io/sig-storage/csi-attacher:v3.2.1],SizeBytes:53499570,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:017727efcfeb7d053af68e51436ce8e65edbc6ca573720afb4f79c8594036955 registry.k8s.io/coredns/coredns:v1.10.0],SizeBytes:50249337,},ContainerImage{Names:[registry.k8s.io/cpa/cluster-proportional-autoscaler@sha256:68d396900aeaa072c1f27289485fdac29834045a6f3ffe369bf389d830ef572d registry.k8s.io/cpa/cluster-proportional-autoscaler:1.8.6],SizeBytes:47910609,},ContainerImage{Names:[k8s.gcr.io/sig-storage/hostpathplugin@sha256:232fe80174d60d520d36043103853a1d7ab4b7f3782cf43e45034f04ccda58ce k8s.gcr.io/sig-storage/hostpathplugin:v1.7.1],SizeBytes:29977921,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:a61d309da54641db41fb8f35718f744e9f730d4d0384f8c4b186ddc9f06cbd5f k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.1.0],SizeBytes:19662887,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:19582112,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:2dee3fe5fe861bb66c3a4ac51114f3447a4cd35870e0f2e2b558c7a400d89589 k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.2.0],SizeBytes:18699436,},ContainerImage{Names:[k8s.gcr.io/sig-storage/mock-driver@sha256:a7b517f9e0f42ffade802eef9cefa271372386b85f55b702b493241e58459793 k8s.gcr.io/sig-storage/mock-driver:v4.1.0],SizeBytes:17680993,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:17611980,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:529be2c9770add0cdd0c989115222ea9fc1be430c11095eb9f6dafcf98a36e2b k8s.gcr.io/sig-storage/livenessprobe:v2.4.0],SizeBytes:17175488,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:13616070e3f29de4417eee434a8ef472221c9e51b3d037b5a6b46cef08eb7443 k8s.gcr.io/e2e-test-images/nginx:1.14-2],SizeBytes:16032814,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nonewprivs@sha256:8ac1264691820febacf3aea5d152cbde6d10685731ec14966a9401c6f47a68ac k8s.gcr.io/e2e-test-images/nonewprivs:1.3],SizeBytes:7107254,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf k8s.gcr.io/e2e-test-images/busybox:1.29-2],SizeBytes:1154361,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:39e1e963e5310e9c313bad51523be012ede7b35bb9316517d19089a010356592 k8s.gcr.io/e2e-test-images/busybox:1.29-1],SizeBytes:1154361,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db k8s.gcr.io/pause:3.6 registry.k8s.io/pause:3.6],SizeBytes:682696,},},VolumesInUse:[kubernetes.io/csi/csi-mock-csi-mock-volumes-5722^4],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/csi-mock-csi-mock-volumes-5722^4,DevicePath:,},},Config:nil,},} Jan 9 23:04:56.725: INFO: Logging kubelet events for node ip-172-20-59-208.ca-central-1.compute.internal Jan 9 23:04:56.759: INFO: Logging pods the kubelet thinks is on node ip-172-20-59-208.ca-central-1.compute.internal Jan 9 23:04:56.803: INFO: coredns-867df8f45c-tmrqb started at 2023-01-09 22:39:03 +0000 UTC (0+1 container statuses recorded) Jan 9 23:04:56.804: INFO: Container coredns ready: true, restart count 0 Jan 9 23:04:56.804: INFO: rs-x69wn started at 2023-01-09 22:59:56 +0000 UTC (0+1 container statuses recorded) Jan 9 23:04:56.804: INFO: Container donothing ready: false, restart count 0 Jan 9 23:04:56.804: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-hszdf started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:04:56.804: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:04:56.804: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-fc6pg started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:04:56.804: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:04:56.804: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-8qp6l started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:04:56.804: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: false, restart count 0 Jan 9 23:04:56.804: INFO: rs-nd6n2 started at 2023-01-09 22:59:56 +0000 UTC (0+1 container statuses recorded) Jan 9 23:04:56.804: INFO: Container donothing ready: false, restart count 0 Jan 9 23:04:56.804: INFO: coredns-autoscaler-557ccb4c66-jnjcc started at 2023-01-09 22:39:03 +0000 UTC (0+1 container statuses recorded) Jan 9 23:04:56.804: INFO: Container autoscaler ready: true, restart count 0 Jan 9 23:04:56.804: INFO: csi-mockplugin-attacher-0 started at 2023-01-09 22:55:28 +0000 UTC (0+1 container statuses recorded) Jan 9 23:04:56.804: INFO: Container csi-attacher ready: true, restart count 0 Jan 9 23:04:56.804: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-bhxkw started at 2023-01-09 22:54:43 +0000 UTC (0+1 container statuses recorded) Jan 9 23:04:56.804: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:04:56.804: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-ptfgl started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:04:56.804: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:04:56.804: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-t5g85 started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:04:56.804: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:04:56.804: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-hjsbl started at 2023-01-09 22:54:43 +0000 UTC (0+1 container statuses recorded) Jan 9 23:04:56.804: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: false, restart count 0 Jan 9 23:04:56.804: INFO: pvc-volume-tester-6qzcw started at 2023-01-09 23:01:13 +0000 UTC (0+1 container statuses recorded) Jan 9 23:04:56.804: INFO: Container volume-tester ready: false, restart count 0 Jan 9 23:04:56.804: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-rzgjf started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:04:56.804: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:04:56.804: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-x682f started at 2023-01-09 22:54:43 +0000 UTC (0+1 container statuses recorded) Jan 9 23:04:56.804: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:04:56.804: INFO: ebs-csi-node-prrrf started at 2023-01-09 22:38:44 +0000 UTC (0+3 container statuses recorded) Jan 9 23:04:56.804: INFO: Container ebs-plugin ready: true, restart count 0 Jan 9 23:04:56.804: INFO: Container liveness-probe ready: true, restart count 0 Jan 9 23:04:56.804: INFO: Container node-driver-registrar ready: true, restart count 0 Jan 9 23:04:56.804: INFO: up-down-1-l4sgq started at 2023-01-09 23:00:56 +0000 UTC (0+1 container statuses recorded) Jan 9 23:04:56.804: INFO: Container up-down-1 ready: false, restart count 0 Jan 9 23:04:56.804: INFO: csi-mockplugin-0 started at 2023-01-09 22:55:28 +0000 UTC (0+3 container statuses recorded) Jan 9 23:04:56.804: INFO: Container csi-provisioner ready: true, restart count 0 Jan 9 23:04:56.804: INFO: Container driver-registrar ready: true, restart count 0 Jan 9 23:04:56.804: INFO: Container mock ready: true, restart count 0 Jan 9 23:04:56.804: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-tx2mm started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:04:56.804: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:04:56.804: INFO: cilium-z2ltf started at 2023-01-09 22:38:44 +0000 UTC (1+1 container statuses recorded) Jan 9 23:04:56.804: INFO: Init container clean-cilium-state ready: true, restart count 0 Jan 9 23:04:56.804: INFO: Container cilium-agent ready: true, restart count 0 Jan 9 23:04:56.804: INFO: csi-hostpathplugin-0 started at 2023-01-09 23:03:05 +0000 UTC (0+7 container statuses recorded) Jan 9 23:04:56.804: INFO: Container csi-attacher ready: false, restart count 0 Jan 9 23:04:56.804: INFO: Container csi-provisioner ready: false, restart count 0 Jan 9 23:04:56.804: INFO: Container csi-resizer ready: false, restart count 0 Jan 9 23:04:56.804: INFO: Container csi-snapshotter ready: false, restart count 0 Jan 9 23:04:56.804: INFO: Container hostpath ready: false, restart count 0 Jan 9 23:04:56.804: INFO: Container liveness-probe ready: false, restart count 0 Jan 9 23:04:56.804: INFO: Container node-driver-registrar ready: false, restart count 0 Jan 9 23:04:57.056: INFO: Latency metrics for node ip-172-20-59-208.ca-central-1.compute.internal Jan 9 23:04:57.056: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready �[1mSTEP�[0m: Destroying namespace "statefulset-6319" for this suite.
Filter through log files
go run hack/e2e.go -v --test --test_args='--ginkgo.focus=Kubernetes\se2e\ssuite\s\[sig\-apps\]\sStatefulSet\sBasic\sStatefulSet\sfunctionality\s\[StatefulSetBasic\]\sshould\sperform\srolling\supdates\sand\sroll\sbacks\sof\stemplate\smodifications\s\[Conformance\]$'
/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:633 Jan 9 23:07:42.951: Failed waiting for pods to enter running: timed out waiting for the condition /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/statefulset/wait.go:80
[BeforeEach] [sig-apps] StatefulSet /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:185 �[1mSTEP�[0m: Creating a kubernetes client Jan 9 22:57:42.536: INFO: >>> kubeConfig: /root/.kube/config �[1mSTEP�[0m: Building a namespace api object, basename statefulset �[1mSTEP�[0m: Waiting for a default service account to be provisioned in namespace �[1mSTEP�[0m: Waiting for kube-root-ca.crt to be provisioned in namespace [BeforeEach] [sig-apps] StatefulSet /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/apps/statefulset.go:94 [BeforeEach] Basic StatefulSet functionality [StatefulSetBasic] /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/apps/statefulset.go:109 �[1mSTEP�[0m: Creating service test in namespace statefulset-3951 [It] should perform rolling updates and roll backs of template modifications [Conformance] /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:633 �[1mSTEP�[0m: Creating a new StatefulSet Jan 9 22:57:42.887: INFO: Found 1 stateful pods, waiting for 3 Jan 9 22:57:52.919: INFO: Found 1 stateful pods, waiting for 3 Jan 9 22:58:02.919: INFO: Found 1 stateful pods, waiting for 3 Jan 9 22:58:12.920: INFO: Found 1 stateful pods, waiting for 3 Jan 9 22:58:22.919: INFO: Found 1 stateful pods, waiting for 3 Jan 9 22:58:32.919: INFO: Found 1 stateful pods, waiting for 3 Jan 9 22:58:42.920: INFO: Found 1 stateful pods, waiting for 3 Jan 9 22:58:52.920: INFO: Found 1 stateful pods, waiting for 3 Jan 9 22:59:02.920: INFO: Found 1 stateful pods, waiting for 3 Jan 9 22:59:12.920: INFO: Found 1 stateful pods, waiting for 3 Jan 9 22:59:22.920: INFO: Found 1 stateful pods, waiting for 3 Jan 9 22:59:32.919: INFO: Found 1 stateful pods, waiting for 3 Jan 9 22:59:42.920: INFO: Found 1 stateful pods, waiting for 3 Jan 9 22:59:52.919: INFO: Found 1 stateful pods, waiting for 3 Jan 9 23:00:02.920: INFO: Found 1 stateful pods, waiting for 3 Jan 9 23:00:12.919: INFO: Found 1 stateful pods, waiting for 3 Jan 9 23:00:22.919: INFO: Found 1 stateful pods, waiting for 3 Jan 9 23:00:32.919: INFO: Found 1 stateful pods, waiting for 3 Jan 9 23:00:42.920: INFO: Found 1 stateful pods, waiting for 3 Jan 9 23:00:52.920: INFO: Found 1 stateful pods, waiting for 3 Jan 9 23:01:02.920: INFO: Found 1 stateful pods, waiting for 3 Jan 9 23:01:12.920: INFO: Found 1 stateful pods, waiting for 3 Jan 9 23:01:22.919: INFO: Found 1 stateful pods, waiting for 3 Jan 9 23:01:32.920: INFO: Found 1 stateful pods, waiting for 3 Jan 9 23:01:42.919: INFO: Found 1 stateful pods, waiting for 3 Jan 9 23:01:52.920: INFO: Found 1 stateful pods, waiting for 3 Jan 9 23:02:02.919: INFO: Found 1 stateful pods, waiting for 3 Jan 9 23:02:12.920: INFO: Found 1 stateful pods, waiting for 3 Jan 9 23:02:22.920: INFO: Found 1 stateful pods, waiting for 3 Jan 9 23:02:32.919: INFO: Found 1 stateful pods, waiting for 3 Jan 9 23:02:42.919: INFO: Found 1 stateful pods, waiting for 3 Jan 9 23:02:52.920: INFO: Found 1 stateful pods, waiting for 3 Jan 9 23:03:02.920: INFO: Found 1 stateful pods, waiting for 3 Jan 9 23:03:12.921: INFO: Found 1 stateful pods, waiting for 3 Jan 9 23:03:22.920: INFO: Found 1 stateful pods, waiting for 3 Jan 9 23:03:32.919: INFO: Found 1 stateful pods, waiting for 3 Jan 9 23:03:42.920: INFO: Found 1 stateful pods, waiting for 3 Jan 9 23:03:52.919: INFO: Found 1 stateful pods, waiting for 3 Jan 9 23:04:02.919: INFO: Found 1 stateful pods, waiting for 3 Jan 9 23:04:12.920: INFO: Found 1 stateful pods, waiting for 3 Jan 9 23:04:22.920: INFO: Found 1 stateful pods, waiting for 3 Jan 9 23:04:32.920: INFO: Found 1 stateful pods, waiting for 3 Jan 9 23:04:42.919: INFO: Found 1 stateful pods, waiting for 3 Jan 9 23:04:52.920: INFO: Found 1 stateful pods, waiting for 3 Jan 9 23:05:02.919: INFO: Found 1 stateful pods, waiting for 3 Jan 9 23:05:12.919: INFO: Found 1 stateful pods, waiting for 3 Jan 9 23:05:22.919: INFO: Found 1 stateful pods, waiting for 3 Jan 9 23:05:32.919: INFO: Found 1 stateful pods, waiting for 3 Jan 9 23:05:42.919: INFO: Found 1 stateful pods, waiting for 3 Jan 9 23:05:52.919: INFO: Found 1 stateful pods, waiting for 3 Jan 9 23:06:02.920: INFO: Found 1 stateful pods, waiting for 3 Jan 9 23:06:12.920: INFO: Found 1 stateful pods, waiting for 3 Jan 9 23:06:22.920: INFO: Found 1 stateful pods, waiting for 3 Jan 9 23:06:32.919: INFO: Found 1 stateful pods, waiting for 3 Jan 9 23:06:42.919: INFO: Found 1 stateful pods, waiting for 3 Jan 9 23:06:52.919: INFO: Found 1 stateful pods, waiting for 3 Jan 9 23:07:02.920: INFO: Found 1 stateful pods, waiting for 3 Jan 9 23:07:12.919: INFO: Found 1 stateful pods, waiting for 3 Jan 9 23:07:22.919: INFO: Found 1 stateful pods, waiting for 3 Jan 9 23:07:32.920: INFO: Found 1 stateful pods, waiting for 3 Jan 9 23:07:42.919: INFO: Found 1 stateful pods, waiting for 3 Jan 9 23:07:42.951: INFO: Found 1 stateful pods, waiting for 3 Jan 9 23:07:42.951: FAIL: Failed waiting for pods to enter running: timed out waiting for the condition Full Stack Trace k8s.io/kubernetes/test/e2e/framework/statefulset.WaitForRunningAndReady(...) /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/statefulset/wait.go:80 k8s.io/kubernetes/test/e2e/apps.rollbackTest({0x7b06bd0, 0xc0021cb500}, {0xc00684ea20, 0x10}, 0xc000602a00) /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/apps/statefulset.go:1565 +0x15c k8s.io/kubernetes/test/e2e/apps.glob..func9.2.7() /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/apps/statefulset.go:305 +0xe6 k8s.io/kubernetes/test/e2e.RunE2ETests(0x24c66d7) _output/dockerized/go/src/k8s.io/kubernetes/test/e2e/e2e.go:133 +0x697 k8s.io/kubernetes/test/e2e.TestE2E(0x243a8f9) _output/dockerized/go/src/k8s.io/kubernetes/test/e2e/e2e_test.go:136 +0x19 testing.tRunner(0xc0001d0d00, 0x735e880) /usr/local/go/src/testing/testing.go:1259 +0x102 created by testing.(*T).Run /usr/local/go/src/testing/testing.go:1306 +0x35a [AfterEach] Basic StatefulSet functionality [StatefulSetBasic] /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/apps/statefulset.go:120 Jan 9 23:07:42.984: INFO: Running '/home/prow/go/src/k8s.io/kops/_rundir/792a86bd-906d-11ed-84f7-72cdabe2d9ae/kubectl --server=https://api.e2e-e2e-kops-grid-cilium-eni-u2204-k23-ko26-docker.test-cncf-aws.k8s.io --kubeconfig=/root/.kube/config --namespace=statefulset-3951 describe po ss2-0' Jan 9 23:07:43.253: INFO: stderr: "" Jan 9 23:07:43.253: INFO: stdout: "Name: ss2-0\nNamespace: statefulset-3951\nPriority: 0\nNode: ip-172-20-55-216.ca-central-1.compute.internal/172.20.55.216\nStart Time: Mon, 09 Jan 2023 22:57:42 +0000\nLabels: baz=blah\n controller-revision-hash=ss2-57bbdd95cb\n foo=bar\n statefulset.kubernetes.io/pod-name=ss2-0\nAnnotations: <none>\nStatus: Pending\nIP: \nIPs: <none>\nControlled By: StatefulSet/ss2\nContainers:\n webserver:\n Container ID: \n Image: k8s.gcr.io/e2e-test-images/httpd:2.4.38-2\n Image ID: \n Port: <none>\n Host Port: <none>\n State: Waiting\n Reason: ContainerCreating\n Ready: False\n Restart Count: 0\n Readiness: http-get http://:80/index.html delay=0s timeout=1s period=1s #success=1 #failure=1\n Environment: <none>\n Mounts:\n /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-jx776 (ro)\nConditions:\n Type Status\n Initialized True \n Ready False \n ContainersReady False \n PodScheduled True \nVolumes:\n kube-api-access-jx776:\n Type: Projected (a volume that contains injected data from multiple sources)\n TokenExpirationSeconds: 3607\n ConfigMapName: kube-root-ca.crt\n ConfigMapOptional: <nil>\n DownwardAPI: true\nQoS Class: BestEffort\nNode-Selectors: <none>\nTolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s\n node.kubernetes.io/unreachable:NoExecute op=Exists for 300s\nEvents:\n Type Reason Age From Message\n ---- ------ ---- ---- -------\n Normal Scheduled 10m default-scheduler Successfully assigned statefulset-3951/ss2-0 to ip-172-20-55-216.ca-central-1.compute.internal\n Warning FailedCreatePodSandBox 9m59s kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container \"b87613a504d1e0cc2c4a7de6ed12cc6cab99648e24f4a4dd202d336ec2a35797\" network for pod \"ss2-0\": networkPlugin cni failed to set up pod \"ss2-0_statefulset-3951\" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available\n Warning FailedCreatePodSandBox 9m57s kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container \"7c794c1d90f5631ea1d4c0e8eae8d1b0bf73f17f7232ed59a04b4237e8c6a3ee\" network for pod \"ss2-0\": networkPlugin cni failed to set up pod \"ss2-0_statefulset-3951\" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available\n Warning FailedCreatePodSandBox 9m55s kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container \"6bf74824b1911d4a4f911d30fa4422642636f55dbb2beb45083e0c13ff0586f3\" network for pod \"ss2-0\": networkPlugin cni failed to set up pod \"ss2-0_statefulset-3951\" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available\n Warning FailedCreatePodSandBox 9m52s kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container \"2428651d9adeb14853a2646b26d87a00240fb0535097bf3e8c5e8b86aab85de6\" network for pod \"ss2-0\": networkPlugin cni failed to set up pod \"ss2-0_statefulset-3951\" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available\n Warning FailedCreatePodSandBox 9m48s kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container \"59bdd0d2c7cc4ad57db2e172e546e77949587eb83e98f7324869793a0e0d48c0\" network for pod \"ss2-0\": networkPlugin cni failed to set up pod \"ss2-0_statefulset-3951\" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available\n Warning FailedCreatePodSandBox 9m45s kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container \"88be0a14fb01610a43434d2267bce8bd6556f14710f8aed8d8a4e7d9a22f6cb8\" network for pod \"ss2-0\": networkPlugin cni failed to set up pod \"ss2-0_statefulset-3951\" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available\n Warning FailedCreatePodSandBox 9m43s kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container \"5f333cb94294d1fe9570b645043936474fe0391a64bc3f31353b52612ae9031d\" network for pod \"ss2-0\": networkPlugin cni failed to set up pod \"ss2-0_statefulset-3951\" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available\n Warning FailedCreatePodSandBox 9m40s kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container \"a16e81a7cf3d4ad68c2b0709b2f59cad4644a71f15f41a422150e7ccfdc16ede\" network for pod \"ss2-0\": networkPlugin cni failed to set up pod \"ss2-0_statefulset-3951\" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available\n Warning FailedCreatePodSandBox 9m35s kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container \"1f56a73edaa44e6c307d8009af45414d3568974ae79e9e1d72c9a22190b3d97e\" network for pod \"ss2-0\": networkPlugin cni failed to set up pod \"ss2-0_statefulset-3951\" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available\n Warning FailedCreatePodSandBox 9m24s (x4 over 9m32s) kubelet (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container \"ad3ed745e69e200a064d827d1dcef70b1aa66a6e0dfb3ba5753e5ea5bb5ef18a\" network for pod \"ss2-0\": networkPlugin cni failed to set up pod \"ss2-0_statefulset-3951\" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available\n Normal SandboxChanged 4m59s (x87 over 9m59s) kubelet Pod sandbox changed, it will be killed and re-created.\n" Jan 9 23:07:43.253: INFO: Output of kubectl describe ss2-0: Name: ss2-0 Namespace: statefulset-3951 Priority: 0 Node: ip-172-20-55-216.ca-central-1.compute.internal/172.20.55.216 Start Time: Mon, 09 Jan 2023 22:57:42 +0000 Labels: baz=blah controller-revision-hash=ss2-57bbdd95cb foo=bar statefulset.kubernetes.io/pod-name=ss2-0 Annotations: <none> Status: Pending IP: IPs: <none> Controlled By: StatefulSet/ss2 Containers: webserver: Container ID: Image: k8s.gcr.io/e2e-test-images/httpd:2.4.38-2 Image ID: Port: <none> Host Port: <none> State: Waiting Reason: ContainerCreating Ready: False Restart Count: 0 Readiness: http-get http://:80/index.html delay=0s timeout=1s period=1s #success=1 #failure=1 Environment: <none> Mounts: /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-jx776 (ro) Conditions: Type Status Initialized True Ready False ContainersReady False PodScheduled True Volumes: kube-api-access-jx776: Type: Projected (a volume that contains injected data from multiple sources) TokenExpirationSeconds: 3607 ConfigMapName: kube-root-ca.crt ConfigMapOptional: <nil> DownwardAPI: true QoS Class: BestEffort Node-Selectors: <none> Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s node.kubernetes.io/unreachable:NoExecute op=Exists for 300s Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 10m default-scheduler Successfully assigned statefulset-3951/ss2-0 to ip-172-20-55-216.ca-central-1.compute.internal Warning FailedCreatePodSandBox 9m59s kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "b87613a504d1e0cc2c4a7de6ed12cc6cab99648e24f4a4dd202d336ec2a35797" network for pod "ss2-0": networkPlugin cni failed to set up pod "ss2-0_statefulset-3951" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Warning FailedCreatePodSandBox 9m57s kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "7c794c1d90f5631ea1d4c0e8eae8d1b0bf73f17f7232ed59a04b4237e8c6a3ee" network for pod "ss2-0": networkPlugin cni failed to set up pod "ss2-0_statefulset-3951" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Warning FailedCreatePodSandBox 9m55s kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "6bf74824b1911d4a4f911d30fa4422642636f55dbb2beb45083e0c13ff0586f3" network for pod "ss2-0": networkPlugin cni failed to set up pod "ss2-0_statefulset-3951" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Warning FailedCreatePodSandBox 9m52s kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "2428651d9adeb14853a2646b26d87a00240fb0535097bf3e8c5e8b86aab85de6" network for pod "ss2-0": networkPlugin cni failed to set up pod "ss2-0_statefulset-3951" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Warning FailedCreatePodSandBox 9m48s kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "59bdd0d2c7cc4ad57db2e172e546e77949587eb83e98f7324869793a0e0d48c0" network for pod "ss2-0": networkPlugin cni failed to set up pod "ss2-0_statefulset-3951" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Warning FailedCreatePodSandBox 9m45s kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "88be0a14fb01610a43434d2267bce8bd6556f14710f8aed8d8a4e7d9a22f6cb8" network for pod "ss2-0": networkPlugin cni failed to set up pod "ss2-0_statefulset-3951" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Warning FailedCreatePodSandBox 9m43s kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "5f333cb94294d1fe9570b645043936474fe0391a64bc3f31353b52612ae9031d" network for pod "ss2-0": networkPlugin cni failed to set up pod "ss2-0_statefulset-3951" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Warning FailedCreatePodSandBox 9m40s kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "a16e81a7cf3d4ad68c2b0709b2f59cad4644a71f15f41a422150e7ccfdc16ede" network for pod "ss2-0": networkPlugin cni failed to set up pod "ss2-0_statefulset-3951" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Warning FailedCreatePodSandBox 9m35s kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "1f56a73edaa44e6c307d8009af45414d3568974ae79e9e1d72c9a22190b3d97e" network for pod "ss2-0": networkPlugin cni failed to set up pod "ss2-0_statefulset-3951" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Warning FailedCreatePodSandBox 9m24s (x4 over 9m32s) kubelet (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "ad3ed745e69e200a064d827d1dcef70b1aa66a6e0dfb3ba5753e5ea5bb5ef18a" network for pod "ss2-0": networkPlugin cni failed to set up pod "ss2-0_statefulset-3951" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Normal SandboxChanged 4m59s (x87 over 9m59s) kubelet Pod sandbox changed, it will be killed and re-created. Jan 9 23:07:43.253: INFO: Running '/home/prow/go/src/k8s.io/kops/_rundir/792a86bd-906d-11ed-84f7-72cdabe2d9ae/kubectl --server=https://api.e2e-e2e-kops-grid-cilium-eni-u2204-k23-ko26-docker.test-cncf-aws.k8s.io --kubeconfig=/root/.kube/config --namespace=statefulset-3951 logs ss2-0 --tail=100' Jan 9 23:07:43.485: INFO: rc: 1 Jan 9 23:07:43.485: INFO: Last 100 log lines of ss2-0: Jan 9 23:07:43.485: INFO: Deleting all statefulset in ns statefulset-3951 Jan 9 23:07:43.519: INFO: Scaling statefulset ss2 to 0 Jan 9 23:07:53.677: INFO: Waiting for statefulset status.replicas updated to 0 Jan 9 23:07:53.708: INFO: Deleting statefulset ss2 [AfterEach] [sig-apps] StatefulSet /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:186 �[1mSTEP�[0m: Collecting events from namespace "statefulset-3951". �[1mSTEP�[0m: Found 14 events. Jan 9 23:07:53.836: INFO: At 2023-01-09 22:57:42 +0000 UTC - event for ss2: {statefulset-controller } SuccessfulCreate: create Pod ss2-0 in StatefulSet ss2 successful Jan 9 23:07:53.836: INFO: At 2023-01-09 22:57:42 +0000 UTC - event for ss2-0: {default-scheduler } Scheduled: Successfully assigned statefulset-3951/ss2-0 to ip-172-20-55-216.ca-central-1.compute.internal Jan 9 23:07:53.836: INFO: At 2023-01-09 22:57:44 +0000 UTC - event for ss2-0: {kubelet ip-172-20-55-216.ca-central-1.compute.internal} SandboxChanged: Pod sandbox changed, it will be killed and re-created. Jan 9 23:07:53.836: INFO: At 2023-01-09 22:57:44 +0000 UTC - event for ss2-0: {kubelet ip-172-20-55-216.ca-central-1.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "b87613a504d1e0cc2c4a7de6ed12cc6cab99648e24f4a4dd202d336ec2a35797" network for pod "ss2-0": networkPlugin cni failed to set up pod "ss2-0_statefulset-3951" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 9 23:07:53.836: INFO: At 2023-01-09 22:57:46 +0000 UTC - event for ss2-0: {kubelet ip-172-20-55-216.ca-central-1.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "7c794c1d90f5631ea1d4c0e8eae8d1b0bf73f17f7232ed59a04b4237e8c6a3ee" network for pod "ss2-0": networkPlugin cni failed to set up pod "ss2-0_statefulset-3951" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 9 23:07:53.836: INFO: At 2023-01-09 22:57:48 +0000 UTC - event for ss2-0: {kubelet ip-172-20-55-216.ca-central-1.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "6bf74824b1911d4a4f911d30fa4422642636f55dbb2beb45083e0c13ff0586f3" network for pod "ss2-0": networkPlugin cni failed to set up pod "ss2-0_statefulset-3951" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 9 23:07:53.836: INFO: At 2023-01-09 22:57:51 +0000 UTC - event for ss2-0: {kubelet ip-172-20-55-216.ca-central-1.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "2428651d9adeb14853a2646b26d87a00240fb0535097bf3e8c5e8b86aab85de6" network for pod "ss2-0": networkPlugin cni failed to set up pod "ss2-0_statefulset-3951" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 9 23:07:53.836: INFO: At 2023-01-09 22:57:55 +0000 UTC - event for ss2-0: {kubelet ip-172-20-55-216.ca-central-1.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "59bdd0d2c7cc4ad57db2e172e546e77949587eb83e98f7324869793a0e0d48c0" network for pod "ss2-0": networkPlugin cni failed to set up pod "ss2-0_statefulset-3951" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 9 23:07:53.836: INFO: At 2023-01-09 22:57:58 +0000 UTC - event for ss2-0: {kubelet ip-172-20-55-216.ca-central-1.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "88be0a14fb01610a43434d2267bce8bd6556f14710f8aed8d8a4e7d9a22f6cb8" network for pod "ss2-0": networkPlugin cni failed to set up pod "ss2-0_statefulset-3951" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 9 23:07:53.836: INFO: At 2023-01-09 22:58:00 +0000 UTC - event for ss2-0: {kubelet ip-172-20-55-216.ca-central-1.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "5f333cb94294d1fe9570b645043936474fe0391a64bc3f31353b52612ae9031d" network for pod "ss2-0": networkPlugin cni failed to set up pod "ss2-0_statefulset-3951" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 9 23:07:53.836: INFO: At 2023-01-09 22:58:03 +0000 UTC - event for ss2-0: {kubelet ip-172-20-55-216.ca-central-1.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "a16e81a7cf3d4ad68c2b0709b2f59cad4644a71f15f41a422150e7ccfdc16ede" network for pod "ss2-0": networkPlugin cni failed to set up pod "ss2-0_statefulset-3951" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 9 23:07:53.836: INFO: At 2023-01-09 22:58:08 +0000 UTC - event for ss2-0: {kubelet ip-172-20-55-216.ca-central-1.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "1f56a73edaa44e6c307d8009af45414d3568974ae79e9e1d72c9a22190b3d97e" network for pod "ss2-0": networkPlugin cni failed to set up pod "ss2-0_statefulset-3951" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 9 23:07:53.836: INFO: At 2023-01-09 22:58:11 +0000 UTC - event for ss2-0: {kubelet ip-172-20-55-216.ca-central-1.compute.internal} FailedCreatePodSandBox: (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "ad3ed745e69e200a064d827d1dcef70b1aa66a6e0dfb3ba5753e5ea5bb5ef18a" network for pod "ss2-0": networkPlugin cni failed to set up pod "ss2-0_statefulset-3951" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 9 23:07:53.836: INFO: At 2023-01-09 23:07:43 +0000 UTC - event for ss2: {statefulset-controller } SuccessfulDelete: delete Pod ss2-0 in StatefulSet ss2 successful Jan 9 23:07:53.867: INFO: POD NODE PHASE GRACE CONDITIONS Jan 9 23:07:53.867: INFO: Jan 9 23:07:53.900: INFO: Logging node info for node ip-172-20-32-67.ca-central-1.compute.internal Jan 9 23:07:53.931: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-32-67.ca-central-1.compute.internal 1f51981b-a01d-404c-b3a4-6649f301f65d 45435 0 2023-01-09 22:38:45 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ca-central-1 failure-domain.beta.kubernetes.io/zone:ca-central-1a kubelet_cleanup:true kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-32-67.ca-central-1.compute.internal kubernetes.io/os:linux kubernetes.io/role:node node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:ca-central-1a topology.hostpath.csi/node:ip-172-20-32-67.ca-central-1.compute.internal topology.kubernetes.io/region:ca-central-1 topology.kubernetes.io/zone:ca-central-1a] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-0272cebea4c6c3d5d"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kops-controller Update v1 2023-01-09 22:38:45 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}} } {kube-controller-manager Update v1 2023-01-09 22:38:45 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.3.0/24\"":{}}}} } {kubelet Update v1 2023-01-09 22:38:45 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {e2e.test Update v1 2023-01-09 22:54:42 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubelet_cleanup":{}}}} } {kube-controller-manager Update v1 2023-01-09 22:55:59 +0000 UTC FieldsV1 {"f:status":{"f:volumesAttached":{}}} status} {kubelet Update v1 2023-01-09 22:56:00 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:capacity":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.3.0/24,DoNotUseExternalID:,ProviderID:aws:///ca-central-1a/i-0272cebea4c6c3d5d,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.3.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051644416 0} {<nil>} 3956684Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946786816 0} {<nil>} 3854284Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-01-09 23:06:12 +0000 UTC,LastTransitionTime:2023-01-09 22:38:45 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-01-09 23:06:12 +0000 UTC,LastTransitionTime:2023-01-09 22:38:45 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-01-09 23:06:12 +0000 UTC,LastTransitionTime:2023-01-09 22:38:45 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-01-09 23:06:12 +0000 UTC,LastTransitionTime:2023-01-09 22:39:06 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.32.67,},NodeAddress{Type:ExternalIP,Address:35.182.2.102,},NodeAddress{Type:Hostname,Address:ip-172-20-32-67.ca-central-1.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-32-67.ca-central-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-35-182-2-102.ca-central-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2536b48bd6b6dade0ed68af2555085,SystemUUID:ec2536b4-8bd6-b6da-de0e-d68af2555085,BootID:bfb3b0df-aafa-481c-9ac6-f85e5e4580e3,KernelVersion:5.15.0-1026-aws,OSImage:Ubuntu 22.04.1 LTS,ContainerRuntimeVersion:docker://20.10.17,KubeletVersion:v1.23.15,KubeProxyVersion:v1.23.15,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:456569528,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/jessie-dnsutils@sha256:11e6a66017ba4e4b938c1612b7a54a3befcefd354796c04e1dba76873a13518e k8s.gcr.io/e2e-test-images/jessie-dnsutils:1.5],SizeBytes:253371792,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:7e8bdd271312fd25fc5ff5a8f04727be84044eb3d7d8d03611972a6752e2e11e k8s.gcr.io/e2e-test-images/agnhost:2.39],SizeBytes:126872991,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nautilus@sha256:99c0d6f1ad24a1aa1905d9c6534d193f268f7b23f9add2ae6bb41f31094bdd5c k8s.gcr.io/e2e-test-images/nautilus:1.5],SizeBytes:124758741,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:1b9d1b2f36cb2dbee1960e82a9344aeb11bd4c4c03abf5e1853e0559c23855e3 k8s.gcr.io/e2e-test-images/httpd:2.4.38-2],SizeBytes:123781643,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.23.15 registry.k8s.io/kube-proxy-amd64:v1.23.15],SizeBytes:112336036,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:10e231cf07c82dfab5141c1afe548dc734e5fc1f67665ec3982a325d1bd31a9a registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.0],SizeBytes:94471559,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:c8e03f60afa90a28e4bb6ec9a8d0fc36d89de4b7475cf2d613afa793ec969fe0 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.0],SizeBytes:56382329,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:4e74c0492bceddc598de1c90cc5bc14dcda94cb49fa9c5bad9d117c4834b5e08 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.1],SizeBytes:56379269,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:36c31f7e1f433c9634d24f876353e8646246d81a03c4e351202c2644daff1620 k8s.gcr.io/sig-storage/csi-resizer:v1.2.0],SizeBytes:53956308,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:ed98431376c9e944e19a465fe8ea944806714dd95416a0821096c78d66b579bd k8s.gcr.io/sig-storage/csi-snapshotter:v4.1.1],SizeBytes:53531990,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:c5be65d6679efabb969d9b019300d187437ae876f992c40911fd2892bbef3b36 k8s.gcr.io/sig-storage/csi-attacher:v3.2.0],SizeBytes:53500826,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:60ab9b3e6a030d3038c87c0d6bca2930f58d1d72823e6a4af09767dc83b696a2 k8s.gcr.io/sig-storage/csi-attacher:v3.2.1],SizeBytes:53499570,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:017727efcfeb7d053af68e51436ce8e65edbc6ca573720afb4f79c8594036955 registry.k8s.io/coredns/coredns:v1.10.0],SizeBytes:50249337,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:7a5ba58a44e0d749e0767e4e37315bcf6a61f33ce3185c1991848af4db0fb70a k8s.gcr.io/sig-storage/csi-resizer:v1.1.0],SizeBytes:49176472,},ContainerImage{Names:[k8s.gcr.io/sig-storage/hostpathplugin@sha256:232fe80174d60d520d36043103853a1d7ab4b7f3782cf43e45034f04ccda58ce k8s.gcr.io/sig-storage/hostpathplugin:v1.7.1],SizeBytes:29977921,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:a61d309da54641db41fb8f35718f744e9f730d4d0384f8c4b186ddc9f06cbd5f k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.1.0],SizeBytes:19662887,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:19582112,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:2dee3fe5fe861bb66c3a4ac51114f3447a4cd35870e0f2e2b558c7a400d89589 k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.2.0],SizeBytes:18699436,},ContainerImage{Names:[k8s.gcr.io/sig-storage/mock-driver@sha256:a7b517f9e0f42ffade802eef9cefa271372386b85f55b702b493241e58459793 k8s.gcr.io/sig-storage/mock-driver:v4.1.0],SizeBytes:17680993,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:17611980,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:529be2c9770add0cdd0c989115222ea9fc1be430c11095eb9f6dafcf98a36e2b k8s.gcr.io/sig-storage/livenessprobe:v2.4.0],SizeBytes:17175488,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:13616070e3f29de4417eee434a8ef472221c9e51b3d037b5a6b46cef08eb7443 k8s.gcr.io/e2e-test-images/nginx:1.14-2],SizeBytes:16032814,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nonewprivs@sha256:8ac1264691820febacf3aea5d152cbde6d10685731ec14966a9401c6f47a68ac k8s.gcr.io/e2e-test-images/nonewprivs:1.3],SizeBytes:7107254,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf k8s.gcr.io/e2e-test-images/busybox:1.29-2],SizeBytes:1154361,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:39e1e963e5310e9c313bad51523be012ede7b35bb9316517d19089a010356592 k8s.gcr.io/e2e-test-images/busybox:1.29-1],SizeBytes:1154361,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db k8s.gcr.io/pause:3.6 registry.k8s.io/pause:3.6],SizeBytes:682696,},},VolumesInUse:[kubernetes.io/csi/ebs.csi.aws.com^vol-01e8a4a8ff1da0e10],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-01e8a4a8ff1da0e10,DevicePath:,},},Config:nil,},} Jan 9 23:07:53.932: INFO: Logging kubelet events for node ip-172-20-32-67.ca-central-1.compute.internal Jan 9 23:07:53.967: INFO: Logging pods the kubelet thinks is on node ip-172-20-32-67.ca-central-1.compute.internal Jan 9 23:07:54.006: INFO: coredns-867df8f45c-sckrz started at 2023-01-09 22:39:23 +0000 UTC (0+1 container statuses recorded) Jan 9 23:07:54.006: INFO: Container coredns ready: true, restart count 0 Jan 9 23:07:54.006: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-44ccv started at 2023-01-09 22:54:43 +0000 UTC (0+1 container statuses recorded) Jan 9 23:07:54.006: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:07:54.006: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-mlcl4 started at 2023-01-09 22:54:43 +0000 UTC (0+1 container statuses recorded) Jan 9 23:07:54.006: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:07:54.006: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-fzrxx started at 2023-01-09 22:54:43 +0000 UTC (0+1 container statuses recorded) Jan 9 23:07:54.006: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:07:54.006: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-5pr42 started at 2023-01-09 22:54:43 +0000 UTC (0+1 container statuses recorded) Jan 9 23:07:54.006: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:07:54.006: INFO: netserver-0 started at 2023-01-09 23:07:18 +0000 UTC (0+1 container statuses recorded) Jan 9 23:07:54.006: INFO: Container webserver ready: false, restart count 0 Jan 9 23:07:54.006: INFO: ebs-csi-node-lc4rb started at 2023-01-09 22:38:46 +0000 UTC (0+3 container statuses recorded) Jan 9 23:07:54.006: INFO: Container ebs-plugin ready: true, restart count 0 Jan 9 23:07:54.006: INFO: Container liveness-probe ready: true, restart count 0 Jan 9 23:07:54.006: INFO: Container node-driver-registrar ready: true, restart count 0 Jan 9 23:07:54.006: INFO: rs-gxqdg started at 2023-01-09 22:59:56 +0000 UTC (0+1 container statuses recorded) Jan 9 23:07:54.006: INFO: Container donothing ready: false, restart count 0 Jan 9 23:07:54.006: INFO: csi-hostpathplugin-0 started at 2023-01-09 23:05:41 +0000 UTC (0+7 container statuses recorded) Jan 9 23:07:54.006: INFO: Container csi-attacher ready: false, restart count 0 Jan 9 23:07:54.006: INFO: Container csi-provisioner ready: false, restart count 0 Jan 9 23:07:54.006: INFO: Container csi-resizer ready: false, restart count 0 Jan 9 23:07:54.006: INFO: Container csi-snapshotter ready: false, restart count 0 Jan 9 23:07:54.006: INFO: Container hostpath ready: false, restart count 0 Jan 9 23:07:54.006: INFO: Container liveness-probe ready: false, restart count 0 Jan 9 23:07:54.006: INFO: Container node-driver-registrar ready: false, restart count 0 Jan 9 23:07:54.006: INFO: inline-volume-tester-5g6j6 started at 2023-01-09 22:55:56 +0000 UTC (0+1 container statuses recorded) Jan 9 23:07:54.006: INFO: Container csi-volume-tester ready: false, restart count 0 Jan 9 23:07:54.006: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-sddjv started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:07:54.006: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:07:54.006: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-8vzqv started at 2023-01-09 22:54:43 +0000 UTC (0+1 container statuses recorded) Jan 9 23:07:54.006: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:07:54.006: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-jb6kp started at 2023-01-09 22:54:43 +0000 UTC (0+1 container statuses recorded) Jan 9 23:07:54.006: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:07:54.006: INFO: cilium-dvsfs started at 2023-01-09 22:38:46 +0000 UTC (1+1 container statuses recorded) Jan 9 23:07:54.006: INFO: Init container clean-cilium-state ready: true, restart count 0 Jan 9 23:07:54.006: INFO: Container cilium-agent ready: true, restart count 0 Jan 9 23:07:54.006: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-gjc2h started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:07:54.006: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:07:54.006: INFO: rs-zn5kz started at 2023-01-09 22:59:56 +0000 UTC (0+1 container statuses recorded) Jan 9 23:07:54.006: INFO: Container donothing ready: true, restart count 0 Jan 9 23:07:54.006: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-hwbbn started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:07:54.006: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:07:54.006: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-fr98x started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:07:54.006: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:07:54.160: INFO: Latency metrics for node ip-172-20-32-67.ca-central-1.compute.internal Jan 9 23:07:54.160: INFO: Logging node info for node ip-172-20-37-64.ca-central-1.compute.internal Jan 9 23:07:54.196: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-37-64.ca-central-1.compute.internal d817613a-0fe5-40c0-a5e4-7b85b9f8506b 44622 0 2023-01-09 22:36:54 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:c5.large beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ca-central-1 failure-domain.beta.kubernetes.io/zone:ca-central-1a kops.k8s.io/kops-controller-pki: kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-37-64.ca-central-1.compute.internal kubernetes.io/os:linux kubernetes.io/role:master node-role.kubernetes.io/control-plane: node-role.kubernetes.io/master: node.kubernetes.io/exclude-from-external-load-balancers: node.kubernetes.io/instance-type:c5.large topology.ebs.csi.aws.com/zone:ca-central-1a topology.kubernetes.io/region:ca-central-1 topology.kubernetes.io/zone:ca-central-1a] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-06217673764fdafeb"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kubelet Update v1 2023-01-09 22:36:54 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {protokube Update v1 2023-01-09 22:37:19 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kops.k8s.io/kops-controller-pki":{},"f:node-role.kubernetes.io/control-plane":{},"f:node.kubernetes.io/exclude-from-external-load-balancers":{}}}} } {kops-controller Update v1 2023-01-09 22:37:21 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/master":{}}}} } {kube-controller-manager Update v1 2023-01-09 22:37:35 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.0.0/24\"":{}},"f:taints":{}}} } {kubelet Update v1 2023-01-09 22:38:39 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.0.0/24,DoNotUseExternalID:,ProviderID:aws:///ca-central-1a/i-06217673764fdafeb,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/master,Value:,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[100.96.0.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3862904832 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3758047232 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-01-09 23:04:17 +0000 UTC,LastTransitionTime:2023-01-09 22:36:48 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-01-09 23:04:17 +0000 UTC,LastTransitionTime:2023-01-09 22:36:48 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-01-09 23:04:17 +0000 UTC,LastTransitionTime:2023-01-09 22:36:48 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-01-09 23:04:17 +0000 UTC,LastTransitionTime:2023-01-09 22:37:35 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.37.64,},NodeAddress{Type:ExternalIP,Address:3.96.216.165,},NodeAddress{Type:Hostname,Address:ip-172-20-37-64.ca-central-1.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-37-64.ca-central-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-3-96-216-165.ca-central-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec27a483f61319a98e5ba2ee2d59e6a3,SystemUUID:ec27a483-f613-19a9-8e5b-a2ee2d59e6a3,BootID:9b41e339-1aaa-4374-8f11-8d34e12cd62a,KernelVersion:5.15.0-1026-aws,OSImage:Ubuntu 22.04.1 LTS,ContainerRuntimeVersion:docker://20.10.17,KubeletVersion:v1.23.15,KubeProxyVersion:v1.23.15,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[registry.k8s.io/etcdadm/etcd-manager@sha256:66a453db625abb268f4b3bbefc5a34a171d81e6e8796cecca54cfd71775c77c4 registry.k8s.io/etcdadm/etcd-manager:v3.0.20221209],SizeBytes:662209237,},ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:456569528,},ContainerImage{Names:[k8s.gcr.io/kube-apiserver-amd64:v1.23.15 registry.k8s.io/kube-apiserver-amd64:v1.23.15],SizeBytes:135240444,},ContainerImage{Names:[k8s.gcr.io/kube-controller-manager-amd64:v1.23.15 registry.k8s.io/kube-controller-manager-amd64:v1.23.15],SizeBytes:125046097,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.23.15 registry.k8s.io/kube-proxy-amd64:v1.23.15],SizeBytes:112336036,},ContainerImage{Names:[quay.io/cilium/operator@sha256:a6d24a006a6b92967ac90786b49bc1ac26e5477cf028cd1186efcfc2466484db quay.io/cilium/operator:v1.12.5],SizeBytes:110016956,},ContainerImage{Names:[registry.k8s.io/kops/kops-controller:1.26.0-beta.1],SizeBytes:104212800,},ContainerImage{Names:[registry.k8s.io/kops/dns-controller:1.26.0-beta.1],SizeBytes:102566451,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:10e231cf07c82dfab5141c1afe548dc734e5fc1f67665ec3982a325d1bd31a9a registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.0],SizeBytes:94471559,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:122bfb8c1edabb3c0edd63f06523e6940d958d19b3957dc7b1d6f81e9f1f6119 registry.k8s.io/sig-storage/csi-provisioner:v3.1.0],SizeBytes:57736070,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:9ebbf9f023e7b41ccee3d52afe39a89e3ddacdbb69269d583abfc25847cfd9e4 registry.k8s.io/sig-storage/csi-resizer:v1.4.0],SizeBytes:55464778,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:8b9c313c05f54fb04f8d430896f5f5904b6cb157df261501b29adc04d2b2dc7b registry.k8s.io/sig-storage/csi-attacher:v3.4.0],SizeBytes:54848595,},ContainerImage{Names:[k8s.gcr.io/kube-scheduler-amd64:v1.23.15 registry.k8s.io/kube-scheduler-amd64:v1.23.15],SizeBytes:53521430,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:19582112,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:17611980,},ContainerImage{Names:[registry.k8s.io/kops/kube-apiserver-healthcheck:1.26.0-beta.1],SizeBytes:8787111,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:682696,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Jan 9 23:07:54.197: INFO: Logging kubelet events for node ip-172-20-37-64.ca-central-1.compute.internal Jan 9 23:07:54.232: INFO: Logging pods the kubelet thinks is on node ip-172-20-37-64.ca-central-1.compute.internal Jan 9 23:07:54.269: INFO: kube-apiserver-ip-172-20-37-64.ca-central-1.compute.internal started at 2023-01-09 22:36:15 +0000 UTC (0+2 container statuses recorded) Jan 9 23:07:54.269: INFO: Container healthcheck ready: true, restart count 0 Jan 9 23:07:54.269: INFO: Container kube-apiserver ready: true, restart count 1 Jan 9 23:07:54.269: INFO: kube-controller-manager-ip-172-20-37-64.ca-central-1.compute.internal started at 2023-01-09 22:36:15 +0000 UTC (0+1 container statuses recorded) Jan 9 23:07:54.270: INFO: Container kube-controller-manager ready: true, restart count 2 Jan 9 23:07:54.270: INFO: kops-controller-cqrv8 started at 2023-01-09 22:37:19 +0000 UTC (0+1 container statuses recorded) Jan 9 23:07:54.270: INFO: Container kops-controller ready: true, restart count 0 Jan 9 23:07:54.270: INFO: dns-controller-867784b75c-sxxcl started at 2023-01-09 22:37:20 +0000 UTC (0+1 container statuses recorded) Jan 9 23:07:54.270: INFO: Container dns-controller ready: true, restart count 0 Jan 9 23:07:54.270: INFO: kube-scheduler-ip-172-20-37-64.ca-central-1.compute.internal started at 2023-01-09 22:36:15 +0000 UTC (0+1 container statuses recorded) Jan 9 23:07:54.270: INFO: Container kube-scheduler ready: true, restart count 0 Jan 9 23:07:54.270: INFO: etcd-manager-events-ip-172-20-37-64.ca-central-1.compute.internal started at 2023-01-09 22:36:15 +0000 UTC (0+1 container statuses recorded) Jan 9 23:07:54.270: INFO: Container etcd-manager ready: true, restart count 0 Jan 9 23:07:54.270: INFO: etcd-manager-main-ip-172-20-37-64.ca-central-1.compute.internal started at 2023-01-09 22:36:15 +0000 UTC (0+1 container statuses recorded) Jan 9 23:07:54.270: INFO: Container etcd-manager ready: true, restart count 0 Jan 9 23:07:54.270: INFO: ebs-csi-node-8q4pg started at 2023-01-09 22:37:17 +0000 UTC (0+3 container statuses recorded) Jan 9 23:07:54.270: INFO: Container ebs-plugin ready: true, restart count 0 Jan 9 23:07:54.270: INFO: Container liveness-probe ready: true, restart count 0 Jan 9 23:07:54.270: INFO: Container node-driver-registrar ready: true, restart count 0 Jan 9 23:07:54.270: INFO: cilium-q7lrg started at 2023-01-09 22:37:17 +0000 UTC (1+1 container statuses recorded) Jan 9 23:07:54.270: INFO: Init container clean-cilium-state ready: true, restart count 0 Jan 9 23:07:54.270: INFO: Container cilium-agent ready: true, restart count 0 Jan 9 23:07:54.270: INFO: cilium-operator-c7bfc9f44-w4nqz started at 2023-01-09 22:37:19 +0000 UTC (0+1 container statuses recorded) Jan 9 23:07:54.270: INFO: Container cilium-operator ready: true, restart count 0 Jan 9 23:07:54.270: INFO: ebs-csi-controller-5d8d7cc949-8wxx2 started at 2023-01-09 22:37:19 +0000 UTC (0+5 container statuses recorded) Jan 9 23:07:54.270: INFO: Container csi-attacher ready: true, restart count 0 Jan 9 23:07:54.270: INFO: Container csi-provisioner ready: true, restart count 0 Jan 9 23:07:54.270: INFO: Container csi-resizer ready: true, restart count 0 Jan 9 23:07:54.270: INFO: Container ebs-plugin ready: true, restart count 0 Jan 9 23:07:54.270: INFO: Container liveness-probe ready: true, restart count 0 Jan 9 23:07:54.384: INFO: Latency metrics for node ip-172-20-37-64.ca-central-1.compute.internal Jan 9 23:07:54.384: INFO: Logging node info for node ip-172-20-55-216.ca-central-1.compute.internal Jan 9 23:07:54.416: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-55-216.ca-central-1.compute.internal 2ceff781-505a-4706-ad8b-e38e37f72571 44855 0 2023-01-09 22:38:47 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ca-central-1 failure-domain.beta.kubernetes.io/zone:ca-central-1a kubelet_cleanup:true kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-55-216.ca-central-1.compute.internal kubernetes.io/os:linux kubernetes.io/role:node node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:ca-central-1a topology.hostpath.csi/node:ip-172-20-55-216.ca-central-1.compute.internal topology.kubernetes.io/region:ca-central-1 topology.kubernetes.io/zone:ca-central-1a] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-0c78741e44993ae7f"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kops-controller Update v1 2023-01-09 22:38:47 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}} } {kube-controller-manager Update v1 2023-01-09 22:38:47 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.4.0/24\"":{}}}} } {kubelet Update v1 2023-01-09 22:38:47 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {e2e.test Update v1 2023-01-09 22:54:42 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubelet_cleanup":{}}}} } {kube-controller-manager Update v1 2023-01-09 22:56:45 +0000 UTC FieldsV1 {"f:status":{"f:volumesAttached":{}}} status} {kubelet Update v1 2023-01-09 22:56:51 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:capacity":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.4.0/24,DoNotUseExternalID:,ProviderID:aws:///ca-central-1a/i-0c78741e44993ae7f,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.4.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051644416 0} {<nil>} 3956684Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946786816 0} {<nil>} 3854284Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-01-09 23:05:11 +0000 UTC,LastTransitionTime:2023-01-09 22:38:47 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-01-09 23:05:11 +0000 UTC,LastTransitionTime:2023-01-09 22:38:47 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-01-09 23:05:11 +0000 UTC,LastTransitionTime:2023-01-09 22:38:47 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-01-09 23:05:11 +0000 UTC,LastTransitionTime:2023-01-09 22:39:07 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.55.216,},NodeAddress{Type:ExternalIP,Address:15.222.60.56,},NodeAddress{Type:Hostname,Address:ip-172-20-55-216.ca-central-1.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-55-216.ca-central-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-15-222-60-56.ca-central-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec249e6a3570a28ebd4276206d3ae523,SystemUUID:ec249e6a-3570-a28e-bd42-76206d3ae523,BootID:d58dcef2-849d-4807-9ba5-0438ebaa05b9,KernelVersion:5.15.0-1026-aws,OSImage:Ubuntu 22.04.1 LTS,ContainerRuntimeVersion:docker://20.10.17,KubeletVersion:v1.23.15,KubeProxyVersion:v1.23.15,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:456569528,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/jessie-dnsutils@sha256:11e6a66017ba4e4b938c1612b7a54a3befcefd354796c04e1dba76873a13518e k8s.gcr.io/e2e-test-images/jessie-dnsutils:1.5],SizeBytes:253371792,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:20f25f275d46aa728f7615a1ccc19c78b2ed89435bf943a44b339f70f45508e6 k8s.gcr.io/e2e-test-images/httpd:2.4.39-2],SizeBytes:126894770,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:7e8bdd271312fd25fc5ff5a8f04727be84044eb3d7d8d03611972a6752e2e11e k8s.gcr.io/e2e-test-images/agnhost:2.39],SizeBytes:126872991,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nautilus@sha256:99c0d6f1ad24a1aa1905d9c6534d193f268f7b23f9add2ae6bb41f31094bdd5c k8s.gcr.io/e2e-test-images/nautilus:1.5],SizeBytes:124758741,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:1b9d1b2f36cb2dbee1960e82a9344aeb11bd4c4c03abf5e1853e0559c23855e3 k8s.gcr.io/e2e-test-images/httpd:2.4.38-2],SizeBytes:123781643,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.23.15 registry.k8s.io/kube-proxy-amd64:v1.23.15],SizeBytes:112336036,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:10e231cf07c82dfab5141c1afe548dc734e5fc1f67665ec3982a325d1bd31a9a registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.0],SizeBytes:94471559,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:c8e03f60afa90a28e4bb6ec9a8d0fc36d89de4b7475cf2d613afa793ec969fe0 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.0],SizeBytes:56382329,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:4e74c0492bceddc598de1c90cc5bc14dcda94cb49fa9c5bad9d117c4834b5e08 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.1],SizeBytes:56379269,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:36c31f7e1f433c9634d24f876353e8646246d81a03c4e351202c2644daff1620 k8s.gcr.io/sig-storage/csi-resizer:v1.2.0],SizeBytes:53956308,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:ed98431376c9e944e19a465fe8ea944806714dd95416a0821096c78d66b579bd k8s.gcr.io/sig-storage/csi-snapshotter:v4.1.1],SizeBytes:53531990,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:c5be65d6679efabb969d9b019300d187437ae876f992c40911fd2892bbef3b36 k8s.gcr.io/sig-storage/csi-attacher:v3.2.0],SizeBytes:53500826,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:60ab9b3e6a030d3038c87c0d6bca2930f58d1d72823e6a4af09767dc83b696a2 k8s.gcr.io/sig-storage/csi-attacher:v3.2.1],SizeBytes:53499570,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:7a5ba58a44e0d749e0767e4e37315bcf6a61f33ce3185c1991848af4db0fb70a k8s.gcr.io/sig-storage/csi-resizer:v1.1.0],SizeBytes:49176472,},ContainerImage{Names:[k8s.gcr.io/sig-storage/hostpathplugin@sha256:232fe80174d60d520d36043103853a1d7ab4b7f3782cf43e45034f04ccda58ce k8s.gcr.io/sig-storage/hostpathplugin:v1.7.1],SizeBytes:29977921,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:a61d309da54641db41fb8f35718f744e9f730d4d0384f8c4b186ddc9f06cbd5f k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.1.0],SizeBytes:19662887,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:19582112,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:2dee3fe5fe861bb66c3a4ac51114f3447a4cd35870e0f2e2b558c7a400d89589 k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.2.0],SizeBytes:18699436,},ContainerImage{Names:[k8s.gcr.io/sig-storage/mock-driver@sha256:a7b517f9e0f42ffade802eef9cefa271372386b85f55b702b493241e58459793 k8s.gcr.io/sig-storage/mock-driver:v4.1.0],SizeBytes:17680993,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:17611980,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:529be2c9770add0cdd0c989115222ea9fc1be430c11095eb9f6dafcf98a36e2b k8s.gcr.io/sig-storage/livenessprobe:v2.4.0],SizeBytes:17175488,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:13616070e3f29de4417eee434a8ef472221c9e51b3d037b5a6b46cef08eb7443 k8s.gcr.io/e2e-test-images/nginx:1.14-2],SizeBytes:16032814,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf k8s.gcr.io/e2e-test-images/busybox:1.29-2],SizeBytes:1154361,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db k8s.gcr.io/pause:3.6 registry.k8s.io/pause:3.6],SizeBytes:682696,},},VolumesInUse:[kubernetes.io/csi/ebs.csi.aws.com^vol-07a7954e90ced7ccc kubernetes.io/csi/ebs.csi.aws.com^vol-0c8e4ae9d44bf0644 kubernetes.io/csi/ebs.csi.aws.com^vol-0d0fbf37c343045c2],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-0c8e4ae9d44bf0644,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-07a7954e90ced7ccc,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-0d0fbf37c343045c2,DevicePath:,},},Config:nil,},} Jan 9 23:07:54.416: INFO: Logging kubelet events for node ip-172-20-55-216.ca-central-1.compute.internal Jan 9 23:07:54.450: INFO: Logging pods the kubelet thinks is on node ip-172-20-55-216.ca-central-1.compute.internal Jan 9 23:07:54.494: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-qglk4 started at 2023-01-09 22:54:43 +0000 UTC (0+1 container statuses recorded) Jan 9 23:07:54.495: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:07:54.495: INFO: httpd started at 2023-01-09 23:06:03 +0000 UTC (0+1 container statuses recorded) Jan 9 23:07:54.495: INFO: Container httpd ready: false, restart count 0 Jan 9 23:07:54.495: INFO: pod-adoption-release started at 2023-01-09 23:05:21 +0000 UTC (0+1 container statuses recorded) Jan 9 23:07:54.495: INFO: Container pod-adoption-release ready: false, restart count 0 Jan 9 23:07:54.495: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-b7bxr started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:07:54.495: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:07:54.495: INFO: pod-subpath-test-inlinevolume-x5g6 started at 2023-01-09 23:05:45 +0000 UTC (2+1 container statuses recorded) Jan 9 23:07:54.495: INFO: Init container init-volume-inlinevolume-x5g6 ready: false, restart count 0 Jan 9 23:07:54.495: INFO: Init container test-init-volume-inlinevolume-x5g6 ready: false, restart count 0 Jan 9 23:07:54.495: INFO: Container test-container-subpath-inlinevolume-x5g6 ready: false, restart count 0 Jan 9 23:07:54.495: INFO: inline-volume-tester-7mx4s started at 2023-01-09 22:56:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:07:54.495: INFO: Container csi-volume-tester ready: false, restart count 0 Jan 9 23:07:54.495: INFO: cilium-dqpw4 started at 2023-01-09 22:38:48 +0000 UTC (1+1 container statuses recorded) Jan 9 23:07:54.495: INFO: Init container clean-cilium-state ready: true, restart count 0 Jan 9 23:07:54.495: INFO: Container cilium-agent ready: true, restart count 0 Jan 9 23:07:54.495: INFO: pod-subpath-test-projected-tqfk started at 2023-01-09 23:03:15 +0000 UTC (0+1 container statuses recorded) Jan 9 23:07:54.495: INFO: Container test-container-subpath-projected-tqfk ready: false, restart count 0 Jan 9 23:07:54.495: INFO: inline-volume-tester-crzhr started at 2023-01-09 23:05:03 +0000 UTC (0+1 container statuses recorded) Jan 9 23:07:54.495: INFO: Container csi-volume-tester ready: false, restart count 0 Jan 9 23:07:54.495: INFO: netserver-1 started at 2023-01-09 23:07:18 +0000 UTC (0+1 container statuses recorded) Jan 9 23:07:54.495: INFO: Container webserver ready: false, restart count 0 Jan 9 23:07:54.495: INFO: fail-once-non-local-mh6qc started at 2023-01-09 22:56:53 +0000 UTC (0+1 container statuses recorded) Jan 9 23:07:54.495: INFO: Container c ready: false, restart count 0 Jan 9 23:07:54.495: INFO: rs-lf7br started at 2023-01-09 22:59:56 +0000 UTC (0+1 container statuses recorded) Jan 9 23:07:54.495: INFO: Container donothing ready: false, restart count 0 Jan 9 23:07:54.495: INFO: csi-mockplugin-0 started at 2023-01-09 23:02:58 +0000 UTC (0+3 container statuses recorded) Jan 9 23:07:54.495: INFO: Container csi-provisioner ready: false, restart count 0 Jan 9 23:07:54.495: INFO: Container driver-registrar ready: false, restart count 0 Jan 9 23:07:54.495: INFO: Container mock ready: false, restart count 0 Jan 9 23:07:54.495: INFO: pause-pod-78b7f9b75c-l4st2 started at 2023-01-09 23:05:44 +0000 UTC (0+1 container statuses recorded) Jan 9 23:07:54.495: INFO: Container agnhost-pause ready: false, restart count 0 Jan 9 23:07:54.495: INFO: logs-generator started at 2023-01-09 23:05:28 +0000 UTC (0+1 container statuses recorded) Jan 9 23:07:54.495: INFO: Container logs-generator ready: false, restart count 0 Jan 9 23:07:54.495: INFO: pod-8ade382c-2fb2-4f02-8d50-57f7b6a15696 started at 2023-01-09 23:06:54 +0000 UTC (0+1 container statuses recorded) Jan 9 23:07:54.495: INFO: Container test-container ready: false, restart count 0 Jan 9 23:07:54.495: INFO: pod-9ba7c444-540c-4612-ad00-c7c5dd3438eb started at 2023-01-09 23:02:45 +0000 UTC (0+1 container statuses recorded) Jan 9 23:07:54.495: INFO: Container write-pod ready: false, restart count 0 Jan 9 23:07:54.495: INFO: startup-ff2f8ca6-ae0a-4fae-9483-a7d80b37fdc1 started at 2023-01-09 23:07:24 +0000 UTC (0+1 container statuses recorded) Jan 9 23:07:54.495: INFO: Container busybox ready: false, restart count 0 Jan 9 23:07:54.495: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-mcd88 started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:07:54.495: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:07:54.495: INFO: echo-sourceip started at 2023-01-09 23:00:54 +0000 UTC (0+1 container statuses recorded) Jan 9 23:07:54.495: INFO: Container agnhost-container ready: true, restart count 0 Jan 9 23:07:54.495: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-8kb86 started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:07:54.495: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:07:54.495: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-vwd6k started at 2023-01-09 22:54:43 +0000 UTC (0+1 container statuses recorded) Jan 9 23:07:54.495: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:07:54.495: INFO: fail-once-non-local-nx7cr started at 2023-01-09 22:56:53 +0000 UTC (0+1 container statuses recorded) Jan 9 23:07:54.495: INFO: Container c ready: false, restart count 0 Jan 9 23:07:54.495: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-6nqvb started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:07:54.495: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:07:54.495: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-qj2bw started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:07:54.495: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:07:54.495: INFO: rs-ctw9c started at 2023-01-09 22:59:56 +0000 UTC (0+1 container statuses recorded) Jan 9 23:07:54.495: INFO: Container donothing ready: true, restart count 0 Jan 9 23:07:54.495: INFO: ebs-csi-node-psq97 started at 2023-01-09 22:38:48 +0000 UTC (0+3 container statuses recorded) Jan 9 23:07:54.495: INFO: Container ebs-plugin ready: true, restart count 0 Jan 9 23:07:54.495: INFO: Container liveness-probe ready: true, restart count 0 Jan 9 23:07:54.495: INFO: Container node-driver-registrar ready: true, restart count 0 Jan 9 23:07:54.495: INFO: csi-mockplugin-resizer-0 started at 2023-01-09 23:02:58 +0000 UTC (0+1 container statuses recorded) Jan 9 23:07:54.495: INFO: Container csi-resizer ready: false, restart count 0 Jan 9 23:07:54.495: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-6ks7b started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:07:54.495: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:07:54.495: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-72kwd started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:07:54.495: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:07:54.495: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-krwzl started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:07:54.495: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:07:54.495: INFO: rs-j4qxp started at 2023-01-09 22:59:56 +0000 UTC (0+1 container statuses recorded) Jan 9 23:07:54.495: INFO: Container donothing ready: false, restart count 0 Jan 9 23:07:54.789: INFO: Latency metrics for node ip-172-20-55-216.ca-central-1.compute.internal Jan 9 23:07:54.789: INFO: Logging node info for node ip-172-20-57-251.ca-central-1.compute.internal Jan 9 23:07:54.821: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-57-251.ca-central-1.compute.internal eaaaf6eb-cbd5-4763-9c39-fb0e1a4ce44a 45391 0 2023-01-09 22:38:45 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ca-central-1 failure-domain.beta.kubernetes.io/zone:ca-central-1a kubelet_cleanup:true kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-57-251.ca-central-1.compute.internal kubernetes.io/os:linux kubernetes.io/role:node node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:ca-central-1a topology.hostpath.csi/node:ip-172-20-57-251.ca-central-1.compute.internal topology.kubernetes.io/region:ca-central-1 topology.kubernetes.io/zone:ca-central-1a] map[csi.volume.kubernetes.io/nodeid:{"csi-mock-csi-mock-volumes-9402":"csi-mock-csi-mock-volumes-9402","ebs.csi.aws.com":"i-0513e7d6e5eba8eec"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kops-controller Update v1 2023-01-09 22:38:45 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}} } {kube-controller-manager Update v1 2023-01-09 22:38:45 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.2.0/24\"":{}}}} } {kubelet Update v1 2023-01-09 22:38:45 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {e2e.test Update v1 2023-01-09 22:54:42 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubelet_cleanup":{}}}} } {kube-controller-manager Update v1 2023-01-09 23:00:23 +0000 UTC FieldsV1 {"f:status":{"f:volumesAttached":{}}} status} {kubelet Update v1 2023-01-09 23:00:27 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.2.0/24,DoNotUseExternalID:,ProviderID:aws:///ca-central-1a/i-0513e7d6e5eba8eec,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.2.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051652608 0} {<nil>} 3956692Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946795008 0} {<nil>} 3854292Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-01-09 23:06:06 +0000 UTC,LastTransitionTime:2023-01-09 22:38:45 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-01-09 23:06:06 +0000 UTC,LastTransitionTime:2023-01-09 22:38:45 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-01-09 23:06:06 +0000 UTC,LastTransitionTime:2023-01-09 22:38:45 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-01-09 23:06:06 +0000 UTC,LastTransitionTime:2023-01-09 22:39:05 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.57.251,},NodeAddress{Type:ExternalIP,Address:3.96.144.209,},NodeAddress{Type:Hostname,Address:ip-172-20-57-251.ca-central-1.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-57-251.ca-central-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-3-96-144-209.ca-central-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2f0d7ba95b37a96a1d9360f6071d84,SystemUUID:ec2f0d7b-a95b-37a9-6a1d-9360f6071d84,BootID:67865282-1911-4e38-8675-a7b78d5ae45e,KernelVersion:5.15.0-1026-aws,OSImage:Ubuntu 22.04.1 LTS,ContainerRuntimeVersion:docker://20.10.17,KubeletVersion:v1.23.15,KubeProxyVersion:v1.23.15,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:456569528,},ContainerImage{Names:[k8s.gcr.io/etcd@sha256:dd75ec974b0a2a6f6bb47001ba09207976e625db898d1b16735528c009cb171c k8s.gcr.io/etcd:3.5.6-0],SizeBytes:299475478,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/jessie-dnsutils@sha256:11e6a66017ba4e4b938c1612b7a54a3befcefd354796c04e1dba76873a13518e k8s.gcr.io/e2e-test-images/jessie-dnsutils:1.5],SizeBytes:253371792,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:20f25f275d46aa728f7615a1ccc19c78b2ed89435bf943a44b339f70f45508e6 k8s.gcr.io/e2e-test-images/httpd:2.4.39-2],SizeBytes:126894770,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:7e8bdd271312fd25fc5ff5a8f04727be84044eb3d7d8d03611972a6752e2e11e k8s.gcr.io/e2e-test-images/agnhost:2.39],SizeBytes:126872991,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nautilus@sha256:99c0d6f1ad24a1aa1905d9c6534d193f268f7b23f9add2ae6bb41f31094bdd5c k8s.gcr.io/e2e-test-images/nautilus:1.5],SizeBytes:124758741,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:1b9d1b2f36cb2dbee1960e82a9344aeb11bd4c4c03abf5e1853e0559c23855e3 k8s.gcr.io/e2e-test-images/httpd:2.4.38-2],SizeBytes:123781643,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.23.15 registry.k8s.io/kube-proxy-amd64:v1.23.15],SizeBytes:112336036,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:10e231cf07c82dfab5141c1afe548dc734e5fc1f67665ec3982a325d1bd31a9a registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.0],SizeBytes:94471559,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/sample-apiserver@sha256:f9c93b92b6ff750b41a93c4e4fe0bfe384597aeb841e2539d5444815c55b2d8f k8s.gcr.io/e2e-test-images/sample-apiserver:1.17.5],SizeBytes:56504541,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:c8e03f60afa90a28e4bb6ec9a8d0fc36d89de4b7475cf2d613afa793ec969fe0 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.0],SizeBytes:56382329,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:4e74c0492bceddc598de1c90cc5bc14dcda94cb49fa9c5bad9d117c4834b5e08 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.1],SizeBytes:56379269,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:36c31f7e1f433c9634d24f876353e8646246d81a03c4e351202c2644daff1620 k8s.gcr.io/sig-storage/csi-resizer:v1.2.0],SizeBytes:53956308,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:ed98431376c9e944e19a465fe8ea944806714dd95416a0821096c78d66b579bd k8s.gcr.io/sig-storage/csi-snapshotter:v4.1.1],SizeBytes:53531990,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:c5be65d6679efabb969d9b019300d187437ae876f992c40911fd2892bbef3b36 k8s.gcr.io/sig-storage/csi-attacher:v3.2.0],SizeBytes:53500826,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:60ab9b3e6a030d3038c87c0d6bca2930f58d1d72823e6a4af09767dc83b696a2 k8s.gcr.io/sig-storage/csi-attacher:v3.2.1],SizeBytes:53499570,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:7a5ba58a44e0d749e0767e4e37315bcf6a61f33ce3185c1991848af4db0fb70a k8s.gcr.io/sig-storage/csi-resizer:v1.1.0],SizeBytes:49176472,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/regression-issue-74839@sha256:b4f1d8d61bdad84bd50442d161d5460e4019d53e989b64220fdbc62fc87d76bf k8s.gcr.io/e2e-test-images/regression-issue-74839:1.2],SizeBytes:44576952,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nonroot@sha256:b9e2958a3dd879e3cf11142228c6d073d0fc4ea2e857c3be6f4fb0ab5fb2c937 k8s.gcr.io/e2e-test-images/nonroot:1.2],SizeBytes:42321438,},ContainerImage{Names:[k8s.gcr.io/sig-storage/hostpathplugin@sha256:232fe80174d60d520d36043103853a1d7ab4b7f3782cf43e45034f04ccda58ce k8s.gcr.io/sig-storage/hostpathplugin:v1.7.1],SizeBytes:29977921,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:a61d309da54641db41fb8f35718f744e9f730d4d0384f8c4b186ddc9f06cbd5f k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.1.0],SizeBytes:19662887,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:19582112,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:2dee3fe5fe861bb66c3a4ac51114f3447a4cd35870e0f2e2b558c7a400d89589 k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.2.0],SizeBytes:18699436,},ContainerImage{Names:[k8s.gcr.io/sig-storage/mock-driver@sha256:a7b517f9e0f42ffade802eef9cefa271372386b85f55b702b493241e58459793 k8s.gcr.io/sig-storage/mock-driver:v4.1.0],SizeBytes:17680993,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:17611980,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:529be2c9770add0cdd0c989115222ea9fc1be430c11095eb9f6dafcf98a36e2b k8s.gcr.io/sig-storage/livenessprobe:v2.4.0],SizeBytes:17175488,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:13616070e3f29de4417eee434a8ef472221c9e51b3d037b5a6b46cef08eb7443 k8s.gcr.io/e2e-test-images/nginx:1.14-2],SizeBytes:16032814,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nonewprivs@sha256:8ac1264691820febacf3aea5d152cbde6d10685731ec14966a9401c6f47a68ac k8s.gcr.io/e2e-test-images/nonewprivs:1.3],SizeBytes:7107254,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf k8s.gcr.io/e2e-test-images/busybox:1.29-2],SizeBytes:1154361,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:39e1e963e5310e9c313bad51523be012ede7b35bb9316517d19089a010356592 k8s.gcr.io/e2e-test-images/busybox:1.29-1],SizeBytes:1154361,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db k8s.gcr.io/pause:3.6 registry.k8s.io/pause:3.6],SizeBytes:682696,},},VolumesInUse:[kubernetes.io/csi/csi-mock-csi-mock-volumes-9402^4 kubernetes.io/csi/ebs.csi.aws.com^vol-02daff5e76e72db98],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-02daff5e76e72db98,DevicePath:,},},Config:nil,},} Jan 9 23:07:54.821: INFO: Logging kubelet events for node ip-172-20-57-251.ca-central-1.compute.internal Jan 9 23:07:54.858: INFO: Logging pods the kubelet thinks is on node ip-172-20-57-251.ca-central-1.compute.internal Jan 9 23:07:54.901: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-tvxfq started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:07:54.901: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:07:54.901: INFO: csi-mockplugin-0 started at 2023-01-09 23:00:20 +0000 UTC (0+3 container statuses recorded) Jan 9 23:07:54.901: INFO: Container csi-provisioner ready: true, restart count 0 Jan 9 23:07:54.901: INFO: Container driver-registrar ready: true, restart count 0 Jan 9 23:07:54.901: INFO: Container mock ready: true, restart count 0 Jan 9 23:07:54.901: INFO: rs-56t9b started at 2023-01-09 22:59:56 +0000 UTC (0+1 container statuses recorded) Jan 9 23:07:54.901: INFO: Container donothing ready: true, restart count 0 Jan 9 23:07:54.901: INFO: ebs-csi-node-g4ngc started at 2023-01-09 22:38:46 +0000 UTC (0+3 container statuses recorded) Jan 9 23:07:54.901: INFO: Container ebs-plugin ready: true, restart count 0 Jan 9 23:07:54.901: INFO: Container liveness-probe ready: true, restart count 0 Jan 9 23:07:54.901: INFO: Container node-driver-registrar ready: true, restart count 0 Jan 9 23:07:54.901: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-wmqwf started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:07:54.901: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:07:54.901: INFO: pod-d16d48d3-63ef-4a76-95b8-d80f8b4aec62 started at 2023-01-09 23:03:17 +0000 UTC (0+1 container statuses recorded) Jan 9 23:07:54.901: INFO: Container write-pod ready: false, restart count 0 Jan 9 23:07:54.901: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-m8cfz started at 2023-01-09 22:54:43 +0000 UTC (0+1 container statuses recorded) Jan 9 23:07:54.901: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:07:54.901: INFO: rs-49kqp started at 2023-01-09 22:59:56 +0000 UTC (0+1 container statuses recorded) Jan 9 23:07:54.901: INFO: Container donothing ready: false, restart count 0 Jan 9 23:07:54.901: INFO: rs-dzbwc started at 2023-01-09 22:59:56 +0000 UTC (0+1 container statuses recorded) Jan 9 23:07:54.901: INFO: Container donothing ready: false, restart count 0 Jan 9 23:07:54.901: INFO: csi-mockplugin-attacher-0 started at 2023-01-09 23:00:20 +0000 UTC (0+1 container statuses recorded) Jan 9 23:07:54.901: INFO: Container csi-attacher ready: false, restart count 0 Jan 9 23:07:54.901: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-rbqs2 started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:07:54.901: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:07:54.901: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-96np5 started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:07:54.901: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:07:54.901: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-f9w29 started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:07:54.901: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:07:54.901: INFO: pause-pod-78b7f9b75c-fckd2 started at 2023-01-09 23:05:44 +0000 UTC (0+1 container statuses recorded) Jan 9 23:07:54.901: INFO: Container agnhost-pause ready: false, restart count 0 Jan 9 23:07:54.901: INFO: netserver-2 started at 2023-01-09 23:07:18 +0000 UTC (0+1 container statuses recorded) Jan 9 23:07:54.901: INFO: Container webserver ready: false, restart count 0 Jan 9 23:07:54.901: INFO: csi-mockplugin-0 started at 2023-01-09 23:05:43 +0000 UTC (0+3 container statuses recorded) Jan 9 23:07:54.901: INFO: Container csi-provisioner ready: false, restart count 0 Jan 9 23:07:54.901: INFO: Container driver-registrar ready: false, restart count 0 Jan 9 23:07:54.901: INFO: Container mock ready: false, restart count 0 Jan 9 23:07:54.901: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-zx8gm started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:07:54.901: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:07:54.901: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-qzf5t started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:07:54.901: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:07:54.901: INFO: busybox-scheduling-6815563e-4dc7-43f5-a3cd-3e998a31b84b started at 2023-01-09 23:06:39 +0000 UTC (0+1 container statuses recorded) Jan 9 23:07:54.901: INFO: Container busybox-scheduling-6815563e-4dc7-43f5-a3cd-3e998a31b84b ready: false, restart count 0 Jan 9 23:07:54.901: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-zvfdw started at 2023-01-09 22:54:43 +0000 UTC (0+1 container statuses recorded) Jan 9 23:07:54.901: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:07:54.901: INFO: cilium-6p9jz started at 2023-01-09 22:38:46 +0000 UTC (1+1 container statuses recorded) Jan 9 23:07:54.901: INFO: Init container clean-cilium-state ready: true, restart count 0 Jan 9 23:07:54.901: INFO: Container cilium-agent ready: true, restart count 0 Jan 9 23:07:54.901: INFO: hostpath-symlink-prep-volume-6746 started at 2023-01-09 23:07:44 +0000 UTC (0+1 container statuses recorded) Jan 9 23:07:54.901: INFO: Container init-volume-volume-6746 ready: false, restart count 0 Jan 9 23:07:54.901: INFO: pvc-volume-tester-2vlf5 started at 2023-01-09 23:06:05 +0000 UTC (0+1 container statuses recorded) Jan 9 23:07:54.901: INFO: Container volume-tester ready: false, restart count 0 Jan 9 23:07:54.901: INFO: cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4-wvflq started at 2023-01-09 22:54:42 +0000 UTC (0+1 container statuses recorded) Jan 9 23:07:54.901: INFO: Container cleanup40-29a73290-8a44-4367-84d2-d90ec32514c4 ready: true, restart count 0 Jan 9 23:07:55.352: INFO: Latency metrics for node ip-172-20-57-251.ca-central-1.compute.internal Jan 9 23:07:55.352: INFO: Logging node info for node ip-172-20-59-208.ca-central-1.compute.internal Jan 9 23:07:55.385: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-59-208.ca-central-1.compute.internal d061a4d4-9ff6-43fc-89cc-422c61f7441c 45677 0 2023-01-09 22:38:43 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ca-central-1 failure-domain.beta.kubernetes.io/zone:ca-central-1a kubelet_cleanup:true kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-59-208.ca-central-1.compute.internal kubernetes.io/os:linux kubernetes.io/role:node node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:ca-central-1a topology.hostpath.csi/node:ip-172-20-59-208.ca-central-1.compute.internal topology.kubernetes.io/region:ca-central-1 topology.kubernetes.io/zone:ca-central-1a] map[csi.volume.kubernetes.io/nodeid:{"csi-hostpath-ephemeral-7223":"ip-172-20-59-208.ca-central-1.compute.internal","ebs.csi.aws.com":"i-0f00ea8bafbacdd11"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kops-controller Update v1 2023-01-09 22:38:43 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}} } {kube-controller-manager Update v1 2023-01-09 22:38:43 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.1.0/24\"":{}}}} } {kubelet Update v1 2023-01-09 22:38:43 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {e2e.test Update v1 2023-01-09 22:54:42 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubelet_cleanup":{}}}} } {kube-controller-manager Update v1 2023-01-09 23:06:36 +0000 UTC FieldsV1 {"f:status":{"f:volumesAttached":{}}} status} {kubelet Update v1 2023-01-09 23:06:40 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:capacity":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.1.0/24,DoNotUseExternalID:,ProviderID:aws:///ca-central-1a/i-0f00ea8bafbacdd11,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.1.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051652608 0} {<nil>} 3956692Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946795008 0} {<nil>} 3854292Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-01-09 23:06:40 +0000 UTC,LastTransitionTime:2023-01-09 22:38:43 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-01-09 23:06:40 +0000 UTC,LastTransitionTime:2023-01-09 22:38:43 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-01-09 23:06:40 +0000 UTC,LastTransitionTime:2023-01-09 22:38:43 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-01-09 23:06:40 +0000 UTC,LastTransitionTime:2023-01-09 22:39:03 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.59.208,},NodeAddress{Type:ExternalIP,Address:99.79.51.95,},NodeAddress{Type:Hostname,Address:ip-172-20-59-208.ca-central-1.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-59-208.ca-central-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-99-79-51-95.ca-central-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec28d5ecfae27d4bcedd89eca1a365f0,SystemUUID:ec28d5ec-fae2-7d4b-cedd-89eca1a365f0,BootID:560320f7-12c5-44f2-a7f6-a7a40031e204,KernelVersion:5.15.0-1026-aws,OSImage:Ubuntu 22.04.1 LTS,ContainerRuntimeVersion:docker://20.10.17,KubeletVersion:v1.23.15,KubeProxyVersion:v1.23.15,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:456569528,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:20f25f275d46aa728f7615a1ccc19c78b2ed89435bf943a44b339f70f45508e6 k8s.gcr.io/e2e-test-images/httpd:2.4.39-2],SizeBytes:126894770,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:7e8bdd271312fd25fc5ff5a8f04727be84044eb3d7d8d03611972a6752e2e11e k8s.gcr.io/e2e-test-images/agnhost:2.39],SizeBytes:126872991,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:1b9d1b2f36cb2dbee1960e82a9344aeb11bd4c4c03abf5e1853e0559c23855e3 k8s.gcr.io/e2e-test-images/httpd:2.4.38-2],SizeBytes:123781643,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.23.15 registry.k8s.io/kube-proxy-amd64:v1.23.15],SizeBytes:112336036,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:10e231cf07c82dfab5141c1afe548dc734e5fc1f67665ec3982a325d1bd31a9a registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.0],SizeBytes:94471559,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:c8e03f60afa90a28e4bb6ec9a8d0fc36d89de4b7475cf2d613afa793ec969fe0 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.0],SizeBytes:56382329,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:4e74c0492bceddc598de1c90cc5bc14dcda94cb49fa9c5bad9d117c4834b5e08 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.1],SizeBytes:56379269,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:36c31f7e1f433c9634d24f876353e8646246d81a03c4e351202c2644daff1620 k8s.gcr.io/sig-storage/csi-resizer:v1.2.0],SizeBytes:53956308,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:ed98431376c9e944e19a465fe8ea944806714dd95416a0821096c78d66b579bd k8s.gcr.io/sig-storage/csi-snapshotter:v4.1.1],SizeBytes:53531990,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:c5be65d6679efabb969d9b019300d187437ae876f992c40911fd2892bbef3b36 k8s.gcr.io/sig-storage/csi-attacher:v3.2.0],SizeBytes:53500826,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:60ab9b3e6a030d3038c87c0d6bca2930f58d1d72823e6a4af09767dc83b696a2 k8s.gcr.io/sig-storage/csi-attacher:v3.2.1],SizeBytes:53499570,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:017727efcfeb7d053af68e51436ce8e65edbc6ca573720afb4f79c8594036955 registry.k8s.io/coredns/coredns:v1.10.0],SizeBytes:50249337,},ContainerImage{Names:[registry.k8s.io/cpa/cluster-proportional-autoscaler@sha256:68d396900aeaa072c1f27289485fdac29834045a6f3ffe369bf389d830ef572d registry.k8s.io/cpa/cluster-proportional-autoscaler:1.8.6],SizeBytes:47910609,},ContainerImage{Names:[k8s.gcr.io/sig-storage/hostpathplugin@sha256:232fe80174d60d520d36043103853a1d7ab4b7f3782cf43e45034f04ccda58ce k8s.gcr.io/sig-storage/hostpathplugin:v1.7.1],SizeBytes:29977921,},ContainerImage