Result | FAILURE |
Tests | 27 failed / 758 succeeded |
Started | |
Elapsed | 44m24s |
Revision | master |
go run hack/e2e.go -v --test --test_args='--ginkgo.focus=Kubernetes\se2e\ssuite\sExternal\sStorage\s\[Driver\:\sebs\.csi\.aws\.com\]\s\[Testpattern\:\sDynamic\sPV\s\(default\sfs\)\]\sfsgroupchangepolicy\s\(Always\)\[LinuxOnly\]\,\spod\screated\swith\san\sinitial\sfsgroup\,\svolume\scontents\sownership\schanged\svia\schgrp\sin\sfirst\spod\,\snew\spod\swith\sdifferent\sfsgroup\sapplied\sto\sthe\svolume\scontents$'
/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/fsgroupchangepolicy.go:214 Jan 1 02:35:56.026: Unexpected error: <*errors.errorString | 0xc002629220>: { s: "pod \"pod-d05d599f-1b25-4249-b9c7-c64a79abebd3\" is not Running: timed out waiting for the condition", } pod "pod-d05d599f-1b25-4249-b9c7-c64a79abebd3" is not Running: timed out waiting for the condition occurred /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/fsgroupchangepolicy.go:262from junit_06.xml
[BeforeEach] [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/framework/testsuite.go:51 [BeforeEach] [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:185 �[1mSTEP�[0m: Creating a kubernetes client Jan 1 02:30:54.576: INFO: >>> kubeConfig: /root/.kube/config �[1mSTEP�[0m: Building a namespace api object, basename fsgroupchangepolicy �[1mSTEP�[0m: Waiting for a default service account to be provisioned in namespace �[1mSTEP�[0m: Waiting for kube-root-ca.crt to be provisioned in namespace [It] (Always)[LinuxOnly], pod created with an initial fsgroup, volume contents ownership changed via chgrp in first pod, new pod with different fsgroup applied to the volume contents /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/fsgroupchangepolicy.go:214 Jan 1 02:30:55.297: INFO: Creating resource for dynamic PV Jan 1 02:30:55.297: INFO: Using claimSize:1Gi, test suite supported size:{ 1Mi}, driver(ebs.csi.aws.com) supported size:{ 1Mi} �[1mSTEP�[0m: creating a StorageClass fsgroupchangepolicy-1819-e2e-scmhhwp �[1mSTEP�[0m: creating a claim Jan 1 02:30:55.403: INFO: Warning: Making PVC: VolumeMode specified as invalid empty string, treating as nil �[1mSTEP�[0m: Creating Pod in namespace fsgroupchangepolicy-1819 with fsgroup 1000 Jan 1 02:35:56.026: FAIL: Unexpected error: <*errors.errorString | 0xc002629220>: { s: "pod \"pod-d05d599f-1b25-4249-b9c7-c64a79abebd3\" is not Running: timed out waiting for the condition", } pod "pod-d05d599f-1b25-4249-b9c7-c64a79abebd3" is not Running: timed out waiting for the condition occurred Full Stack Trace k8s.io/kubernetes/test/e2e/storage/testsuites.createPodAndVerifyContentGid(0xc003e602c0, 0xc0047b5070, 0x1, {0x0, 0x0}, {0x0, 0x0}) /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/fsgroupchangepolicy.go:262 +0x15c k8s.io/kubernetes/test/e2e/storage/testsuites.(*fsGroupChangePolicyTestSuite).DefineTests.func3() /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/fsgroupchangepolicy.go:233 +0x296 k8s.io/kubernetes/test/e2e.RunE2ETests(0x24c66d7) _output/dockerized/go/src/k8s.io/kubernetes/test/e2e/e2e.go:133 +0x697 k8s.io/kubernetes/test/e2e.TestE2E(0x243a8f9) _output/dockerized/go/src/k8s.io/kubernetes/test/e2e/e2e_test.go:136 +0x19 testing.tRunner(0xc000d81380, 0x735e880) /usr/local/go/src/testing/testing.go:1259 +0x102 created by testing.(*T).Run /usr/local/go/src/testing/testing.go:1306 +0x35a �[1mSTEP�[0m: Deleting pvc Jan 1 02:35:56.232: INFO: Deleting PersistentVolumeClaim "ebs.csi.aws.combdbck" Jan 1 02:35:56.337: INFO: Waiting up to 3m0s for PersistentVolume pvc-c052a5c8-cc1d-4487-99d3-e231ddab71b8 to get deleted Jan 1 02:35:56.440: INFO: PersistentVolume pvc-c052a5c8-cc1d-4487-99d3-e231ddab71b8 found and phase=Bound (102.744901ms) Jan 1 02:36:01.543: INFO: PersistentVolume pvc-c052a5c8-cc1d-4487-99d3-e231ddab71b8 found and phase=Bound (5.205883503s) Jan 1 02:36:06.646: INFO: PersistentVolume pvc-c052a5c8-cc1d-4487-99d3-e231ddab71b8 found and phase=Bound (10.308752627s) Jan 1 02:36:11.752: INFO: PersistentVolume pvc-c052a5c8-cc1d-4487-99d3-e231ddab71b8 found and phase=Bound (15.414939106s) Jan 1 02:36:16.863: INFO: PersistentVolume pvc-c052a5c8-cc1d-4487-99d3-e231ddab71b8 found and phase=Bound (20.526446353s) Jan 1 02:36:21.966: INFO: PersistentVolume pvc-c052a5c8-cc1d-4487-99d3-e231ddab71b8 found and phase=Bound (25.629016191s) Jan 1 02:36:27.069: INFO: PersistentVolume pvc-c052a5c8-cc1d-4487-99d3-e231ddab71b8 found and phase=Bound (30.732483009s) Jan 1 02:36:32.173: INFO: PersistentVolume pvc-c052a5c8-cc1d-4487-99d3-e231ddab71b8 found and phase=Bound (35.835886352s) Jan 1 02:36:37.276: INFO: PersistentVolume pvc-c052a5c8-cc1d-4487-99d3-e231ddab71b8 found and phase=Bound (40.939442406s) Jan 1 02:36:42.379: INFO: PersistentVolume pvc-c052a5c8-cc1d-4487-99d3-e231ddab71b8 found and phase=Bound (46.042216648s) Jan 1 02:36:47.483: INFO: PersistentVolume pvc-c052a5c8-cc1d-4487-99d3-e231ddab71b8 found and phase=Bound (51.145899734s) Jan 1 02:36:52.595: INFO: PersistentVolume pvc-c052a5c8-cc1d-4487-99d3-e231ddab71b8 found and phase=Bound (56.257927146s) Jan 1 02:36:57.701: INFO: PersistentVolume pvc-c052a5c8-cc1d-4487-99d3-e231ddab71b8 found and phase=Bound (1m1.364427426s) Jan 1 02:37:02.805: INFO: PersistentVolume pvc-c052a5c8-cc1d-4487-99d3-e231ddab71b8 found and phase=Bound (1m6.468421541s) Jan 1 02:37:07.908: INFO: PersistentVolume pvc-c052a5c8-cc1d-4487-99d3-e231ddab71b8 found and phase=Bound (1m11.57141802s) Jan 1 02:37:13.012: INFO: PersistentVolume pvc-c052a5c8-cc1d-4487-99d3-e231ddab71b8 found and phase=Bound (1m16.675483543s) Jan 1 02:37:18.117: INFO: PersistentVolume pvc-c052a5c8-cc1d-4487-99d3-e231ddab71b8 found and phase=Bound (1m21.780275558s) Jan 1 02:37:23.221: INFO: PersistentVolume pvc-c052a5c8-cc1d-4487-99d3-e231ddab71b8 found and phase=Bound (1m26.88445673s) Jan 1 02:37:28.324: INFO: PersistentVolume pvc-c052a5c8-cc1d-4487-99d3-e231ddab71b8 found and phase=Bound (1m31.987169278s) Jan 1 02:37:33.428: INFO: PersistentVolume pvc-c052a5c8-cc1d-4487-99d3-e231ddab71b8 found and phase=Bound (1m37.090742818s) Jan 1 02:37:38.530: INFO: PersistentVolume pvc-c052a5c8-cc1d-4487-99d3-e231ddab71b8 found and phase=Bound (1m42.193597565s) Jan 1 02:37:43.634: INFO: PersistentVolume pvc-c052a5c8-cc1d-4487-99d3-e231ddab71b8 found and phase=Bound (1m47.296786553s) Jan 1 02:37:48.737: INFO: PersistentVolume pvc-c052a5c8-cc1d-4487-99d3-e231ddab71b8 found and phase=Bound (1m52.400163717s) Jan 1 02:37:53.841: INFO: PersistentVolume pvc-c052a5c8-cc1d-4487-99d3-e231ddab71b8 found and phase=Bound (1m57.50458902s) Jan 1 02:37:58.944: INFO: PersistentVolume pvc-c052a5c8-cc1d-4487-99d3-e231ddab71b8 found and phase=Bound (2m2.607582818s) Jan 1 02:38:04.048: INFO: PersistentVolume pvc-c052a5c8-cc1d-4487-99d3-e231ddab71b8 found and phase=Bound (2m7.710826359s) Jan 1 02:38:09.151: INFO: PersistentVolume pvc-c052a5c8-cc1d-4487-99d3-e231ddab71b8 found and phase=Bound (2m12.813902202s) Jan 1 02:38:14.254: INFO: PersistentVolume pvc-c052a5c8-cc1d-4487-99d3-e231ddab71b8 found and phase=Bound (2m17.917252914s) Jan 1 02:38:19.358: INFO: PersistentVolume pvc-c052a5c8-cc1d-4487-99d3-e231ddab71b8 found and phase=Bound (2m23.021115723s) Jan 1 02:38:24.461: INFO: PersistentVolume pvc-c052a5c8-cc1d-4487-99d3-e231ddab71b8 found and phase=Bound (2m28.124027s) Jan 1 02:38:29.563: INFO: PersistentVolume pvc-c052a5c8-cc1d-4487-99d3-e231ddab71b8 found and phase=Bound (2m33.226709786s) Jan 1 02:38:34.669: INFO: PersistentVolume pvc-c052a5c8-cc1d-4487-99d3-e231ddab71b8 found and phase=Bound (2m38.331894658s) Jan 1 02:38:39.772: INFO: PersistentVolume pvc-c052a5c8-cc1d-4487-99d3-e231ddab71b8 found and phase=Bound (2m43.434979061s) Jan 1 02:38:44.875: INFO: PersistentVolume pvc-c052a5c8-cc1d-4487-99d3-e231ddab71b8 found and phase=Bound (2m48.537767537s) Jan 1 02:38:49.977: INFO: PersistentVolume pvc-c052a5c8-cc1d-4487-99d3-e231ddab71b8 found and phase=Bound (2m53.640619375s) Jan 1 02:38:55.081: INFO: PersistentVolume pvc-c052a5c8-cc1d-4487-99d3-e231ddab71b8 found and phase=Bound (2m58.743841217s) �[1mSTEP�[0m: Deleting sc Jan 1 02:39:00.190: FAIL: while cleanup resource Unexpected error: <errors.aggregate | len:1, cap:1>: [ [ { msg: "persistent Volume pvc-c052a5c8-cc1d-4487-99d3-e231ddab71b8 not deleted by dynamic provisioner: PersistentVolume pvc-c052a5c8-cc1d-4487-99d3-e231ddab71b8 still exists within 3m0s", err: { s: "PersistentVolume pvc-c052a5c8-cc1d-4487-99d3-e231ddab71b8 still exists within 3m0s", }, }, ], ] persistent Volume pvc-c052a5c8-cc1d-4487-99d3-e231ddab71b8 not deleted by dynamic provisioner: PersistentVolume pvc-c052a5c8-cc1d-4487-99d3-e231ddab71b8 still exists within 3m0s occurred Full Stack Trace k8s.io/kubernetes/test/e2e/storage/testsuites.(*fsGroupChangePolicyTestSuite).DefineTests.func2() /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/fsgroupchangepolicy.go:132 +0x20f panic({0x6c3d000, 0xc0039cfa00}) /usr/local/go/src/runtime/panic.go:1038 +0x215 k8s.io/kubernetes/test/e2e/framework/ginkgowrapper.Fail.func1() /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/ginkgowrapper/wrapper.go:63 +0x73 panic({0x62d5960, 0x78abec0}) /usr/local/go/src/runtime/panic.go:1038 +0x215 k8s.io/kubernetes/test/e2e/framework/ginkgowrapper.Fail({0xc00023c3c0, 0x13b}, {0xc00253ad30, 0x70cbf6a, 0xc00253ad50}) /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/ginkgowrapper/wrapper.go:67 +0x1a7 k8s.io/kubernetes/test/e2e/framework.Fail({0xc00023c280, 0x126}, {0xc002e01da0, 0xc00023c280, 0xc0041760d0}) /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/log.go:63 +0x149 k8s.io/kubernetes/vendor/github.com/onsi/gomega/internal/assertion.(*Assertion).match(0xc00253ae98, {0x79beb98, 0xaa12408}, 0x0, {0x0, 0x0, 0x0}) /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/onsi/gomega/internal/assertion/assertion.go:79 +0x1bd k8s.io/kubernetes/vendor/github.com/onsi/gomega/internal/assertion.(*Assertion).NotTo(0xc00253ae98, {0x79beb98, 0xaa12408}, {0x0, 0x0, 0x0}) /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/onsi/gomega/internal/assertion/assertion.go:48 +0x92 k8s.io/kubernetes/test/e2e/framework.ExpectNoErrorWithOffset(0x7b06bd0, {0x78b2200, 0xc002629220}, {0x0, 0x2, 0xc003d5ad20}) /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/expect.go:46 +0xa9 k8s.io/kubernetes/test/e2e/framework.ExpectNoError(...) /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/expect.go:40 k8s.io/kubernetes/test/e2e/storage/testsuites.createPodAndVerifyContentGid(0xc003e602c0, 0xc0047b5070, 0x1, {0x0, 0x0}, {0x0, 0x0}) /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/fsgroupchangepolicy.go:262 +0x15c k8s.io/kubernetes/test/e2e/storage/testsuites.(*fsGroupChangePolicyTestSuite).DefineTests.func3() /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/fsgroupchangepolicy.go:233 +0x296 k8s.io/kubernetes/test/e2e.RunE2ETests(0x24c66d7) _output/dockerized/go/src/k8s.io/kubernetes/test/e2e/e2e.go:133 +0x697 k8s.io/kubernetes/test/e2e.TestE2E(0x243a8f9) _output/dockerized/go/src/k8s.io/kubernetes/test/e2e/e2e_test.go:136 +0x19 testing.tRunner(0xc000d81380, 0x735e880) /usr/local/go/src/testing/testing.go:1259 +0x102 created by testing.(*T).Run /usr/local/go/src/testing/testing.go:1306 +0x35a [AfterEach] [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:186 �[1mSTEP�[0m: Collecting events from namespace "fsgroupchangepolicy-1819". �[1mSTEP�[0m: Found 17 events. Jan 1 02:39:00.295: INFO: At 2023-01-01 02:30:55 +0000 UTC - event for ebs.csi.aws.combdbck: {persistentvolume-controller } WaitForFirstConsumer: waiting for first consumer to be created before binding Jan 1 02:39:00.295: INFO: At 2023-01-01 02:30:55 +0000 UTC - event for ebs.csi.aws.combdbck: {persistentvolume-controller } ExternalProvisioning: waiting for a volume to be created, either by external provisioner "ebs.csi.aws.com" or manually created by system administrator Jan 1 02:39:00.295: INFO: At 2023-01-01 02:30:55 +0000 UTC - event for ebs.csi.aws.combdbck: {ebs.csi.aws.com_ebs-csi-controller-6dc545b75-5qqh8_70f1c4b8-e6fd-4c89-86a5-21e55725c724 } Provisioning: External provisioner is provisioning volume for claim "fsgroupchangepolicy-1819/ebs.csi.aws.combdbck" Jan 1 02:39:00.295: INFO: At 2023-01-01 02:30:59 +0000 UTC - event for ebs.csi.aws.combdbck: {ebs.csi.aws.com_ebs-csi-controller-6dc545b75-5qqh8_70f1c4b8-e6fd-4c89-86a5-21e55725c724 } ProvisioningSucceeded: Successfully provisioned volume pvc-c052a5c8-cc1d-4487-99d3-e231ddab71b8 Jan 1 02:39:00.295: INFO: At 2023-01-01 02:30:59 +0000 UTC - event for pod-d05d599f-1b25-4249-b9c7-c64a79abebd3: {default-scheduler } Scheduled: Successfully assigned fsgroupchangepolicy-1819/pod-d05d599f-1b25-4249-b9c7-c64a79abebd3 to ip-172-20-45-112.eu-west-3.compute.internal Jan 1 02:39:00.295: INFO: At 2023-01-01 02:31:02 +0000 UTC - event for pod-d05d599f-1b25-4249-b9c7-c64a79abebd3: {attachdetach-controller } SuccessfulAttachVolume: AttachVolume.Attach succeeded for volume "pvc-c052a5c8-cc1d-4487-99d3-e231ddab71b8" Jan 1 02:39:00.295: INFO: At 2023-01-01 02:31:03 +0000 UTC - event for pod-d05d599f-1b25-4249-b9c7-c64a79abebd3: {kubelet ip-172-20-45-112.eu-west-3.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "27c4cd9fb3caaa286ba34b29e6d0786588b1f8c9f199caaa9fa427677da12152" network for pod "pod-d05d599f-1b25-4249-b9c7-c64a79abebd3": networkPlugin cni failed to set up pod "pod-d05d599f-1b25-4249-b9c7-c64a79abebd3_fsgroupchangepolicy-1819" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 1 02:39:00.295: INFO: At 2023-01-01 02:31:04 +0000 UTC - event for pod-d05d599f-1b25-4249-b9c7-c64a79abebd3: {kubelet ip-172-20-45-112.eu-west-3.compute.internal} SandboxChanged: Pod sandbox changed, it will be killed and re-created. Jan 1 02:39:00.295: INFO: At 2023-01-01 02:31:04 +0000 UTC - event for pod-d05d599f-1b25-4249-b9c7-c64a79abebd3: {kubelet ip-172-20-45-112.eu-west-3.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "20400af7c60a66067e1c02fef9605b4a8107904e4f8bef2eda646715b63694a5" network for pod "pod-d05d599f-1b25-4249-b9c7-c64a79abebd3": networkPlugin cni failed to set up pod "pod-d05d599f-1b25-4249-b9c7-c64a79abebd3_fsgroupchangepolicy-1819" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 1 02:39:00.295: INFO: At 2023-01-01 02:31:05 +0000 UTC - event for pod-d05d599f-1b25-4249-b9c7-c64a79abebd3: {kubelet ip-172-20-45-112.eu-west-3.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "67385fc8ef03d5ec422117a2366abdd8cf7b6d1146154f1ab5222b748f2fcc90" network for pod "pod-d05d599f-1b25-4249-b9c7-c64a79abebd3": networkPlugin cni failed to set up pod "pod-d05d599f-1b25-4249-b9c7-c64a79abebd3_fsgroupchangepolicy-1819" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 1 02:39:00.295: INFO: At 2023-01-01 02:31:06 +0000 UTC - event for pod-d05d599f-1b25-4249-b9c7-c64a79abebd3: {kubelet ip-172-20-45-112.eu-west-3.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "d04731d235adfaf391a66bef2982209eb109a635f3056a08531f65ada4c222c5" network for pod "pod-d05d599f-1b25-4249-b9c7-c64a79abebd3": networkPlugin cni failed to set up pod "pod-d05d599f-1b25-4249-b9c7-c64a79abebd3_fsgroupchangepolicy-1819" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 1 02:39:00.295: INFO: At 2023-01-01 02:31:07 +0000 UTC - event for pod-d05d599f-1b25-4249-b9c7-c64a79abebd3: {kubelet ip-172-20-45-112.eu-west-3.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "36c21ddb0943fb1be92950f3b74f85f49cf892d0abbf0d31731cc45f8967aaa8" network for pod "pod-d05d599f-1b25-4249-b9c7-c64a79abebd3": networkPlugin cni failed to set up pod "pod-d05d599f-1b25-4249-b9c7-c64a79abebd3_fsgroupchangepolicy-1819" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 1 02:39:00.295: INFO: At 2023-01-01 02:31:08 +0000 UTC - event for pod-d05d599f-1b25-4249-b9c7-c64a79abebd3: {kubelet ip-172-20-45-112.eu-west-3.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "01731bbc662484207a6eeff5b6c94d69e807bba6929fd34dd6376a021270779b" network for pod "pod-d05d599f-1b25-4249-b9c7-c64a79abebd3": networkPlugin cni failed to set up pod "pod-d05d599f-1b25-4249-b9c7-c64a79abebd3_fsgroupchangepolicy-1819" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 1 02:39:00.295: INFO: At 2023-01-01 02:31:09 +0000 UTC - event for pod-d05d599f-1b25-4249-b9c7-c64a79abebd3: {kubelet ip-172-20-45-112.eu-west-3.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "cf1c6ec8aa29f095f89e672765042affbffbc7e0f52e6f847825dd5b8593f246" network for pod "pod-d05d599f-1b25-4249-b9c7-c64a79abebd3": networkPlugin cni failed to set up pod "pod-d05d599f-1b25-4249-b9c7-c64a79abebd3_fsgroupchangepolicy-1819" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 1 02:39:00.295: INFO: At 2023-01-01 02:31:10 +0000 UTC - event for pod-d05d599f-1b25-4249-b9c7-c64a79abebd3: {kubelet ip-172-20-45-112.eu-west-3.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "20d3c66ca82115e0c28206588a5b37ae20af036420a60c64596c84e942258533" network for pod "pod-d05d599f-1b25-4249-b9c7-c64a79abebd3": networkPlugin cni failed to set up pod "pod-d05d599f-1b25-4249-b9c7-c64a79abebd3_fsgroupchangepolicy-1819" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 1 02:39:00.295: INFO: At 2023-01-01 02:31:11 +0000 UTC - event for pod-d05d599f-1b25-4249-b9c7-c64a79abebd3: {kubelet ip-172-20-45-112.eu-west-3.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "294db2f88b239cfb095f5f60d3d7ede0f303588a26eaed483c6a7527c3969cea" network for pod "pod-d05d599f-1b25-4249-b9c7-c64a79abebd3": networkPlugin cni failed to set up pod "pod-d05d599f-1b25-4249-b9c7-c64a79abebd3_fsgroupchangepolicy-1819" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 1 02:39:00.295: INFO: At 2023-01-01 02:31:12 +0000 UTC - event for pod-d05d599f-1b25-4249-b9c7-c64a79abebd3: {kubelet ip-172-20-45-112.eu-west-3.compute.internal} FailedCreatePodSandBox: (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "35e8b8e06923f71e605990517b49cd534b6f4c58d2779feffd7eec8fec275994" network for pod "pod-d05d599f-1b25-4249-b9c7-c64a79abebd3": networkPlugin cni failed to set up pod "pod-d05d599f-1b25-4249-b9c7-c64a79abebd3_fsgroupchangepolicy-1819" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 1 02:39:00.398: INFO: POD NODE PHASE GRACE CONDITIONS Jan 1 02:39:00.398: INFO: pod-d05d599f-1b25-4249-b9c7-c64a79abebd3 ip-172-20-45-112.eu-west-3.compute.internal Running [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2023-01-01 02:30:59 +0000 UTC } {Ready True 0001-01-01 00:00:00 +0000 UTC 2023-01-01 02:36:22 +0000 UTC } {ContainersReady True 0001-01-01 00:00:00 +0000 UTC 2023-01-01 02:36:22 +0000 UTC } {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2023-01-01 02:30:59 +0000 UTC }] Jan 1 02:39:00.398: INFO: Jan 1 02:39:00.615: INFO: Logging node info for node ip-172-20-33-66.eu-west-3.compute.internal Jan 1 02:39:00.717: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-33-66.eu-west-3.compute.internal ec4b59b3-62db-4040-aad2-4c3249f80fe8 16278 0 2023-01-01 02:20:25 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:eu-west-3 failure-domain.beta.kubernetes.io/zone:eu-west-3a io.kubernetes.storage.mock/node:some-mock-node kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-33-66.eu-west-3.compute.internal kubernetes.io/os:linux kubernetes.io/role:node node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:eu-west-3a topology.hostpath.csi/node:ip-172-20-33-66.eu-west-3.compute.internal topology.kubernetes.io/region:eu-west-3 topology.kubernetes.io/zone:eu-west-3a] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-028313673124ac4ad"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kops-controller Update v1 2023-01-01 02:20:25 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}} } {kube-controller-manager Update v1 2023-01-01 02:20:25 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.1.0/24\"":{}}}} } {kubelet Update v1 2023-01-01 02:20:25 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {kubelet Update v1 2023-01-01 02:38:29 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:io.kubernetes.storage.mock/node":{},"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.1.0/24,DoNotUseExternalID:,ProviderID:aws:///eu-west-3a/i-028313673124ac4ad,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.1.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{50478186496 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4078190592 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{45430367772 0} {<nil>} 45430367772 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3973332992 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-01-01 02:38:29 +0000 UTC,LastTransitionTime:2023-01-01 02:20:25 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-01-01 02:38:29 +0000 UTC,LastTransitionTime:2023-01-01 02:20:25 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-01-01 02:38:29 +0000 UTC,LastTransitionTime:2023-01-01 02:20:25 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-01-01 02:38:29 +0000 UTC,LastTransitionTime:2023-01-01 02:20:46 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.33.66,},NodeAddress{Type:ExternalIP,Address:52.47.140.112,},NodeAddress{Type:Hostname,Address:ip-172-20-33-66.eu-west-3.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-33-66.eu-west-3.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-52-47-140-112.eu-west-3.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2ad2ce1a87d508c178acf6215c5767,SystemUUID:ec2ad2ce-1a87-d508-c178-acf6215c5767,BootID:b3b60783-d786-412b-a36a-802bcfae604d,KernelVersion:4.19.0-23-cloud-amd64,OSImage:Debian GNU/Linux 10 (buster),ContainerRuntimeVersion:docker://20.10.17,KubeletVersion:v1.23.15,KubeProxyVersion:v1.23.15,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:456569528,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/jessie-dnsutils@sha256:11e6a66017ba4e4b938c1612b7a54a3befcefd354796c04e1dba76873a13518e k8s.gcr.io/e2e-test-images/jessie-dnsutils:1.5],SizeBytes:253371792,},ContainerImage{Names:[nginx@sha256:0047b729188a15da49380d9506d65959cce6d40291ccfb4e039f5dc7efd33286 nginx:latest],SizeBytes:141812353,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:7e8bdd271312fd25fc5ff5a8f04727be84044eb3d7d8d03611972a6752e2e11e k8s.gcr.io/e2e-test-images/agnhost:2.39],SizeBytes:126872991,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:1b9d1b2f36cb2dbee1960e82a9344aeb11bd4c4c03abf5e1853e0559c23855e3 k8s.gcr.io/e2e-test-images/httpd:2.4.38-2],SizeBytes:123781643,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.23.15 registry.k8s.io/kube-proxy-amd64:v1.23.15],SizeBytes:112336036,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:10e231cf07c82dfab5141c1afe548dc734e5fc1f67665ec3982a325d1bd31a9a registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.0],SizeBytes:94471559,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:c8e03f60afa90a28e4bb6ec9a8d0fc36d89de4b7475cf2d613afa793ec969fe0 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.0],SizeBytes:56382329,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:4e74c0492bceddc598de1c90cc5bc14dcda94cb49fa9c5bad9d117c4834b5e08 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.1],SizeBytes:56379269,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:36c31f7e1f433c9634d24f876353e8646246d81a03c4e351202c2644daff1620 k8s.gcr.io/sig-storage/csi-resizer:v1.2.0],SizeBytes:53956308,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:ed98431376c9e944e19a465fe8ea944806714dd95416a0821096c78d66b579bd k8s.gcr.io/sig-storage/csi-snapshotter:v4.1.1],SizeBytes:53531990,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:c5be65d6679efabb969d9b019300d187437ae876f992c40911fd2892bbef3b36 k8s.gcr.io/sig-storage/csi-attacher:v3.2.0],SizeBytes:53500826,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:60ab9b3e6a030d3038c87c0d6bca2930f58d1d72823e6a4af09767dc83b696a2 k8s.gcr.io/sig-storage/csi-attacher:v3.2.1],SizeBytes:53499570,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:017727efcfeb7d053af68e51436ce8e65edbc6ca573720afb4f79c8594036955 registry.k8s.io/coredns/coredns:v1.10.0],SizeBytes:50249337,},ContainerImage{Names:[k8s.gcr.io/sig-storage/hostpathplugin@sha256:232fe80174d60d520d36043103853a1d7ab4b7f3782cf43e45034f04ccda58ce k8s.gcr.io/sig-storage/hostpathplugin:v1.7.1],SizeBytes:29977921,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:a61d309da54641db41fb8f35718f744e9f730d4d0384f8c4b186ddc9f06cbd5f k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.1.0],SizeBytes:19662887,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:19582112,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:2dee3fe5fe861bb66c3a4ac51114f3447a4cd35870e0f2e2b558c7a400d89589 k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.2.0],SizeBytes:18699436,},ContainerImage{Names:[k8s.gcr.io/sig-storage/mock-driver@sha256:a7b517f9e0f42ffade802eef9cefa271372386b85f55b702b493241e58459793 k8s.gcr.io/sig-storage/mock-driver:v4.1.0],SizeBytes:17680993,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:17611980,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:529be2c9770add0cdd0c989115222ea9fc1be430c11095eb9f6dafcf98a36e2b k8s.gcr.io/sig-storage/livenessprobe:v2.4.0],SizeBytes:17175488,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:13616070e3f29de4417eee434a8ef472221c9e51b3d037b5a6b46cef08eb7443 k8s.gcr.io/e2e-test-images/nginx:1.14-2],SizeBytes:16032814,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nonewprivs@sha256:8ac1264691820febacf3aea5d152cbde6d10685731ec14966a9401c6f47a68ac k8s.gcr.io/e2e-test-images/nonewprivs:1.3],SizeBytes:7107254,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf k8s.gcr.io/e2e-test-images/busybox:1.29-2],SizeBytes:1154361,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:39e1e963e5310e9c313bad51523be012ede7b35bb9316517d19089a010356592 k8s.gcr.io/e2e-test-images/busybox:1.29-1],SizeBytes:1154361,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db k8s.gcr.io/pause:3.6 registry.k8s.io/pause:3.6],SizeBytes:682696,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Jan 1 02:39:00.718: INFO: Logging kubelet events for node ip-172-20-33-66.eu-west-3.compute.internal Jan 1 02:39:00.822: INFO: Logging pods the kubelet thinks is on node ip-172-20-33-66.eu-west-3.compute.internal Jan 1 02:39:00.932: INFO: httpd-deployment-95bc5655f-d8hbt started at 2023-01-01 02:38:59 +0000 UTC (0+1 container statuses recorded) Jan 1 02:39:00.932: INFO: Container httpd ready: false, restart count 0 Jan 1 02:39:00.932: INFO: inline-volume-pswlf started at 2023-01-01 02:38:47 +0000 UTC (0+1 container statuses recorded) Jan 1 02:39:00.932: INFO: Container volume-tester ready: false, restart count 0 Jan 1 02:39:00.932: INFO: pod-subpath-test-configmap-lgmv started at 2023-01-01 02:38:54 +0000 UTC (0+1 container statuses recorded) Jan 1 02:39:00.932: INFO: Container test-container-subpath-configmap-lgmv ready: true, restart count 0 Jan 1 02:39:00.932: INFO: ebs-csi-node-vsxjc started at 2023-01-01 02:20:26 +0000 UTC (0+3 container statuses recorded) Jan 1 02:39:00.932: INFO: Container ebs-plugin ready: true, restart count 0 Jan 1 02:39:00.932: INFO: Container liveness-probe ready: true, restart count 0 Jan 1 02:39:00.932: INFO: Container node-driver-registrar ready: true, restart count 0 Jan 1 02:39:00.932: INFO: labelsupdate42c69795-0951-44b0-bb43-a5a7799919c4 started at 2023-01-01 02:38:46 +0000 UTC (0+1 container statuses recorded) Jan 1 02:39:00.932: INFO: Container client-container ready: false, restart count 0 Jan 1 02:39:00.932: INFO: service-proxy-disabled-w9dz8 started at 2023-01-01 02:37:17 +0000 UTC (0+1 container statuses recorded) Jan 1 02:39:00.932: INFO: Container service-proxy-disabled ready: true, restart count 0 Jan 1 02:39:00.932: INFO: cilium-vbcgc started at 2023-01-01 02:20:26 +0000 UTC (1+1 container statuses recorded) Jan 1 02:39:00.932: INFO: Init container clean-cilium-state ready: true, restart count 0 Jan 1 02:39:00.932: INFO: Container cilium-agent ready: true, restart count 0 Jan 1 02:39:00.932: INFO: httpd-deployment-5bf95dfb4d-lv6jv started at 2023-01-01 02:38:57 +0000 UTC (0+1 container statuses recorded) Jan 1 02:39:00.932: INFO: Container httpd ready: false, restart count 0 Jan 1 02:39:00.932: INFO: failed-jobs-history-limit-27875679-mb22w started at <nil> (0+0 container statuses recorded) Jan 1 02:39:00.932: INFO: ss2-0 started at 2023-01-01 02:37:44 +0000 UTC (0+1 container statuses recorded) Jan 1 02:39:00.932: INFO: Container webserver ready: true, restart count 0 Jan 1 02:39:00.932: INFO: coredns-867df8f45c-cqmp7 started at 2023-01-01 02:21:18 +0000 UTC (0+1 container statuses recorded) Jan 1 02:39:00.932: INFO: Container coredns ready: true, restart count 0 Jan 1 02:39:00.932: INFO: inline-volume-nj465 started at 2023-01-01 02:37:12 +0000 UTC (0+1 container statuses recorded) Jan 1 02:39:00.932: INFO: Container volume-tester ready: false, restart count 0 Jan 1 02:39:00.932: INFO: hostexec-ip-172-20-33-66.eu-west-3.compute.internal-8h6jr started at 2023-01-01 02:38:48 +0000 UTC (0+1 container statuses recorded) Jan 1 02:39:00.932: INFO: Container agnhost-container ready: true, restart count 0 Jan 1 02:39:00.932: INFO: netserver-0 started at 2023-01-01 02:38:39 +0000 UTC (0+1 container statuses recorded) Jan 1 02:39:00.932: INFO: Container webserver ready: true, restart count 0 Jan 1 02:39:00.932: INFO: service-proxy-toggled-tcpkz started at 2023-01-01 02:37:24 +0000 UTC (0+1 container statuses recorded) Jan 1 02:39:00.932: INFO: Container service-proxy-toggled ready: true, restart count 0 Jan 1 02:39:01.295: INFO: Latency metrics for node ip-172-20-33-66.eu-west-3.compute.internal Jan 1 02:39:01.295: INFO: Logging node info for node ip-172-20-38-238.eu-west-3.compute.internal Jan 1 02:39:01.399: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-38-238.eu-west-3.compute.internal 0eb713b4-64bf-47b7-8090-bab8111218ee 11610 0 2023-01-01 02:18:51 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:c5.large beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:eu-west-3 failure-domain.beta.kubernetes.io/zone:eu-west-3a kops.k8s.io/kops-controller-pki: kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-38-238.eu-west-3.compute.internal kubernetes.io/os:linux kubernetes.io/role:master node-role.kubernetes.io/control-plane: node-role.kubernetes.io/master: node.kubernetes.io/exclude-from-external-load-balancers: node.kubernetes.io/instance-type:c5.large topology.ebs.csi.aws.com/zone:eu-west-3a topology.kubernetes.io/region:eu-west-3 topology.kubernetes.io/zone:eu-west-3a] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-0f70ebb30a9076648"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kubelet Update v1 2023-01-01 02:18:50 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {protokube Update v1 2023-01-01 02:19:16 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kops.k8s.io/kops-controller-pki":{},"f:node-role.kubernetes.io/control-plane":{},"f:node.kubernetes.io/exclude-from-external-load-balancers":{}}}} } {kops-controller Update v1 2023-01-01 02:19:21 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/master":{}}}} } {kube-controller-manager Update v1 2023-01-01 02:19:31 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.0.0/24\"":{}},"f:taints":{}}} } {kubelet Update v1 2023-01-01 02:20:36 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.0.0/24,DoNotUseExternalID:,ProviderID:aws:///eu-west-3a/i-0f70ebb30a9076648,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/master,Value:,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[100.96.0.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{50478186496 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3918811136 0} {<nil>} 3826964Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{45430367772 0} {<nil>} 45430367772 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3813953536 0} {<nil>} 3724564Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-01-01 02:36:32 +0000 UTC,LastTransitionTime:2023-01-01 02:18:45 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-01-01 02:36:32 +0000 UTC,LastTransitionTime:2023-01-01 02:18:45 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-01-01 02:36:32 +0000 UTC,LastTransitionTime:2023-01-01 02:18:45 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-01-01 02:36:32 +0000 UTC,LastTransitionTime:2023-01-01 02:19:31 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.38.238,},NodeAddress{Type:ExternalIP,Address:15.237.26.58,},NodeAddress{Type:Hostname,Address:ip-172-20-38-238.eu-west-3.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-38-238.eu-west-3.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-15-237-26-58.eu-west-3.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2e6c64b5b811bf0c78b2aa7ccdb943,SystemUUID:ec2e6c64-b5b8-11bf-0c78-b2aa7ccdb943,BootID:9db01fa3-d136-4230-b21a-b03abbfb6bff,KernelVersion:4.19.0-23-cloud-amd64,OSImage:Debian GNU/Linux 10 (buster),ContainerRuntimeVersion:docker://20.10.17,KubeletVersion:v1.23.15,KubeProxyVersion:v1.23.15,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[registry.k8s.io/etcdadm/etcd-manager@sha256:66a453db625abb268f4b3bbefc5a34a171d81e6e8796cecca54cfd71775c77c4 registry.k8s.io/etcdadm/etcd-manager:v3.0.20221209],SizeBytes:662209237,},ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:456569528,},ContainerImage{Names:[k8s.gcr.io/kube-apiserver-amd64:v1.23.15 registry.k8s.io/kube-apiserver-amd64:v1.23.15],SizeBytes:135240444,},ContainerImage{Names:[k8s.gcr.io/kube-controller-manager-amd64:v1.23.15 registry.k8s.io/kube-controller-manager-amd64:v1.23.15],SizeBytes:125046097,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.23.15 registry.k8s.io/kube-proxy-amd64:v1.23.15],SizeBytes:112336036,},ContainerImage{Names:[quay.io/cilium/operator@sha256:a6d24a006a6b92967ac90786b49bc1ac26e5477cf028cd1186efcfc2466484db quay.io/cilium/operator:v1.12.5],SizeBytes:110016956,},ContainerImage{Names:[registry.k8s.io/kops/kops-controller:1.27.0-alpha.1],SizeBytes:105197757,},ContainerImage{Names:[registry.k8s.io/kops/dns-controller:1.27.0-alpha.1],SizeBytes:102577531,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:10e231cf07c82dfab5141c1afe548dc734e5fc1f67665ec3982a325d1bd31a9a registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.0],SizeBytes:94471559,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:122bfb8c1edabb3c0edd63f06523e6940d958d19b3957dc7b1d6f81e9f1f6119 registry.k8s.io/sig-storage/csi-provisioner:v3.1.0],SizeBytes:57736070,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:9ebbf9f023e7b41ccee3d52afe39a89e3ddacdbb69269d583abfc25847cfd9e4 registry.k8s.io/sig-storage/csi-resizer:v1.4.0],SizeBytes:55464778,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:8b9c313c05f54fb04f8d430896f5f5904b6cb157df261501b29adc04d2b2dc7b registry.k8s.io/sig-storage/csi-attacher:v3.4.0],SizeBytes:54848595,},ContainerImage{Names:[k8s.gcr.io/kube-scheduler-amd64:v1.23.15 registry.k8s.io/kube-scheduler-amd64:v1.23.15],SizeBytes:53521430,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:19582112,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:17611980,},ContainerImage{Names:[registry.k8s.io/kops/kube-apiserver-healthcheck:1.27.0-alpha.1],SizeBytes:8787111,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:682696,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Jan 1 02:39:01.399: INFO: Logging kubelet events for node ip-172-20-38-238.eu-west-3.compute.internal Jan 1 02:39:01.508: INFO: Logging pods the kubelet thinks is on node ip-172-20-38-238.eu-west-3.compute.internal Jan 1 02:39:01.624: INFO: cilium-qk59v started at 2023-01-01 02:19:20 +0000 UTC (1+1 container statuses recorded) Jan 1 02:39:01.624: INFO: Init container clean-cilium-state ready: true, restart count 0 Jan 1 02:39:01.624: INFO: Container cilium-agent ready: true, restart count 0 Jan 1 02:39:01.624: INFO: ebs-csi-node-k87k6 started at 2023-01-01 02:19:20 +0000 UTC (0+3 container statuses recorded) Jan 1 02:39:01.624: INFO: Container ebs-plugin ready: true, restart count 0 Jan 1 02:39:01.624: INFO: Container liveness-probe ready: true, restart count 0 Jan 1 02:39:01.624: INFO: Container node-driver-registrar ready: true, restart count 0 Jan 1 02:39:01.624: INFO: dns-controller-7db8b74b99-58zzw started at 2023-01-01 02:19:20 +0000 UTC (0+1 container statuses recorded) Jan 1 02:39:01.624: INFO: Container dns-controller ready: true, restart count 0 Jan 1 02:39:01.624: INFO: cilium-operator-d47487c76-j84nq started at 2023-01-01 02:19:20 +0000 UTC (0+1 container statuses recorded) Jan 1 02:39:01.624: INFO: Container cilium-operator ready: true, restart count 0 Jan 1 02:39:01.624: INFO: etcd-manager-events-ip-172-20-38-238.eu-west-3.compute.internal started at 2023-01-01 02:18:11 +0000 UTC (0+1 container statuses recorded) Jan 1 02:39:01.624: INFO: Container etcd-manager ready: true, restart count 0 Jan 1 02:39:01.624: INFO: etcd-manager-main-ip-172-20-38-238.eu-west-3.compute.internal started at 2023-01-01 02:18:11 +0000 UTC (0+1 container statuses recorded) Jan 1 02:39:01.624: INFO: Container etcd-manager ready: true, restart count 0 Jan 1 02:39:01.624: INFO: kube-apiserver-ip-172-20-38-238.eu-west-3.compute.internal started at 2023-01-01 02:18:11 +0000 UTC (0+2 container statuses recorded) Jan 1 02:39:01.624: INFO: Container healthcheck ready: true, restart count 0 Jan 1 02:39:01.624: INFO: Container kube-apiserver ready: true, restart count 1 Jan 1 02:39:01.624: INFO: kops-controller-26gzg started at 2023-01-01 02:19:20 +0000 UTC (0+1 container statuses recorded) Jan 1 02:39:01.624: INFO: Container kops-controller ready: true, restart count 0 Jan 1 02:39:01.624: INFO: ebs-csi-controller-6dc545b75-5qqh8 started at 2023-01-01 02:19:20 +0000 UTC (0+5 container statuses recorded) Jan 1 02:39:01.624: INFO: Container csi-attacher ready: true, restart count 0 Jan 1 02:39:01.624: INFO: Container csi-provisioner ready: true, restart count 0 Jan 1 02:39:01.624: INFO: Container csi-resizer ready: true, restart count 0 Jan 1 02:39:01.624: INFO: Container ebs-plugin ready: true, restart count 0 Jan 1 02:39:01.624: INFO: Container liveness-probe ready: true, restart count 0 Jan 1 02:39:01.624: INFO: kube-controller-manager-ip-172-20-38-238.eu-west-3.compute.internal started at 2023-01-01 02:18:11 +0000 UTC (0+1 container statuses recorded) Jan 1 02:39:01.624: INFO: Container kube-controller-manager ready: true, restart count 2 Jan 1 02:39:01.624: INFO: kube-scheduler-ip-172-20-38-238.eu-west-3.compute.internal started at 2023-01-01 02:18:11 +0000 UTC (0+1 container statuses recorded) Jan 1 02:39:01.624: INFO: Container kube-scheduler ready: true, restart count 0 Jan 1 02:39:01.963: INFO: Latency metrics for node ip-172-20-38-238.eu-west-3.compute.internal Jan 1 02:39:01.963: INFO: Logging node info for node ip-172-20-45-112.eu-west-3.compute.internal Jan 1 02:39:02.067: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-45-112.eu-west-3.compute.internal 48eed937-05bb-4d8a-bddd-aa94a2e6ab92 16355 0 2023-01-01 02:20:27 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:eu-west-3 failure-domain.beta.kubernetes.io/zone:eu-west-3a kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-45-112.eu-west-3.compute.internal kubernetes.io/os:linux kubernetes.io/role:node node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:eu-west-3a topology.hostpath.csi/node:ip-172-20-45-112.eu-west-3.compute.internal topology.kubernetes.io/region:eu-west-3 topology.kubernetes.io/zone:eu-west-3a] map[csi.volume.kubernetes.io/nodeid:{"csi-hostpath-ephemeral-5709":"ip-172-20-45-112.eu-west-3.compute.internal","ebs.csi.aws.com":"i-0778adf2085fa7f34"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kube-controller-manager Update v1 2023-01-01 02:20:27 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.3.0/24\"":{}}}} } {kubelet Update v1 2023-01-01 02:20:27 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {kops-controller Update v1 2023-01-01 02:20:28 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}} } {kube-controller-manager Update v1 2023-01-01 02:30:49 +0000 UTC FieldsV1 {"f:status":{"f:volumesAttached":{}}} status} {kubelet Update v1 2023-01-01 02:30:52 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:capacity":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.3.0/24,DoNotUseExternalID:,ProviderID:aws:///eu-west-3a/i-0778adf2085fa7f34,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.3.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{50478186496 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4078190592 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{45430367772 0} {<nil>} 45430367772 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3973332992 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-01-01 02:39:01 +0000 UTC,LastTransitionTime:2023-01-01 02:20:27 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-01-01 02:39:01 +0000 UTC,LastTransitionTime:2023-01-01 02:20:27 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-01-01 02:39:01 +0000 UTC,LastTransitionTime:2023-01-01 02:20:27 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-01-01 02:39:01 +0000 UTC,LastTransitionTime:2023-01-01 02:20:48 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.45.112,},NodeAddress{Type:ExternalIP,Address:13.38.32.14,},NodeAddress{Type:Hostname,Address:ip-172-20-45-112.eu-west-3.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-45-112.eu-west-3.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-13-38-32-14.eu-west-3.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2dbdf846e8ff614f4555e5bf9dbbf4,SystemUUID:ec2dbdf8-46e8-ff61-4f45-55e5bf9dbbf4,BootID:10a054ae-621a-47ce-9566-b0048cc5101b,KernelVersion:4.19.0-23-cloud-amd64,OSImage:Debian GNU/Linux 10 (buster),ContainerRuntimeVersion:docker://20.10.17,KubeletVersion:v1.23.15,KubeProxyVersion:v1.23.15,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:456569528,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/jessie-dnsutils@sha256:11e6a66017ba4e4b938c1612b7a54a3befcefd354796c04e1dba76873a13518e k8s.gcr.io/e2e-test-images/jessie-dnsutils:1.5],SizeBytes:253371792,},ContainerImage{Names:[nginx@sha256:0047b729188a15da49380d9506d65959cce6d40291ccfb4e039f5dc7efd33286 nginx:latest],SizeBytes:141812353,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:7e8bdd271312fd25fc5ff5a8f04727be84044eb3d7d8d03611972a6752e2e11e k8s.gcr.io/e2e-test-images/agnhost:2.39],SizeBytes:126872991,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:1b9d1b2f36cb2dbee1960e82a9344aeb11bd4c4c03abf5e1853e0559c23855e3 k8s.gcr.io/e2e-test-images/httpd:2.4.38-2],SizeBytes:123781643,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.23.15 registry.k8s.io/kube-proxy-amd64:v1.23.15],SizeBytes:112336036,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:10e231cf07c82dfab5141c1afe548dc734e5fc1f67665ec3982a325d1bd31a9a registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.0],SizeBytes:94471559,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:c8e03f60afa90a28e4bb6ec9a8d0fc36d89de4b7475cf2d613afa793ec969fe0 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.0],SizeBytes:56382329,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:4e74c0492bceddc598de1c90cc5bc14dcda94cb49fa9c5bad9d117c4834b5e08 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.1],SizeBytes:56379269,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:36c31f7e1f433c9634d24f876353e8646246d81a03c4e351202c2644daff1620 k8s.gcr.io/sig-storage/csi-resizer:v1.2.0],SizeBytes:53956308,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:ed98431376c9e944e19a465fe8ea944806714dd95416a0821096c78d66b579bd k8s.gcr.io/sig-storage/csi-snapshotter:v4.1.1],SizeBytes:53531990,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:c5be65d6679efabb969d9b019300d187437ae876f992c40911fd2892bbef3b36 k8s.gcr.io/sig-storage/csi-attacher:v3.2.0],SizeBytes:53500826,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:60ab9b3e6a030d3038c87c0d6bca2930f58d1d72823e6a4af09767dc83b696a2 k8s.gcr.io/sig-storage/csi-attacher:v3.2.1],SizeBytes:53499570,},ContainerImage{Names:[k8s.gcr.io/sig-storage/hostpathplugin@sha256:232fe80174d60d520d36043103853a1d7ab4b7f3782cf43e45034f04ccda58ce k8s.gcr.io/sig-storage/hostpathplugin:v1.7.1],SizeBytes:29977921,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:a61d309da54641db41fb8f35718f744e9f730d4d0384f8c4b186ddc9f06cbd5f k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.1.0],SizeBytes:19662887,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:19582112,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:2dee3fe5fe861bb66c3a4ac51114f3447a4cd35870e0f2e2b558c7a400d89589 k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.2.0],SizeBytes:18699436,},ContainerImage{Names:[k8s.gcr.io/sig-storage/mock-driver@sha256:a7b517f9e0f42ffade802eef9cefa271372386b85f55b702b493241e58459793 k8s.gcr.io/sig-storage/mock-driver:v4.1.0],SizeBytes:17680993,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:17611980,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:529be2c9770add0cdd0c989115222ea9fc1be430c11095eb9f6dafcf98a36e2b k8s.gcr.io/sig-storage/livenessprobe:v2.4.0],SizeBytes:17175488,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:13616070e3f29de4417eee434a8ef472221c9e51b3d037b5a6b46cef08eb7443 k8s.gcr.io/e2e-test-images/nginx:1.14-2],SizeBytes:16032814,},ContainerImage{Names:[gcr.io/authenticated-image-pulling/alpine@sha256:7ff177862cb50c602bfe81f805969412e619c054a2bbead977d0c276988aa4a0 gcr.io/authenticated-image-pulling/alpine:3.7],SizeBytes:4206620,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf k8s.gcr.io/e2e-test-images/busybox:1.29-2],SizeBytes:1154361,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:39e1e963e5310e9c313bad51523be012ede7b35bb9316517d19089a010356592 k8s.gcr.io/e2e-test-images/busybox:1.29-1],SizeBytes:1154361,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db k8s.gcr.io/pause:3.6 registry.k8s.io/pause:3.6],SizeBytes:682696,},},VolumesInUse:[kubernetes.io/csi/ebs.csi.aws.com^vol-0627618389b1d4de1 kubernetes.io/csi/ebs.csi.aws.com^vol-08601772399ba2131],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-08601772399ba2131,DevicePath:,},},Config:nil,},} Jan 1 02:39:02.067: INFO: Logging kubelet events for node ip-172-20-45-112.eu-west-3.compute.internal Jan 1 02:39:02.172: INFO: Logging pods the kubelet thinks is on node ip-172-20-45-112.eu-west-3.compute.internal Jan 1 02:39:02.284: INFO: simpletest.rc-59rx4 started at 2023-01-01 02:38:36 +0000 UTC (0+1 container statuses recorded) Jan 1 02:39:02.284: INFO: Container nginx ready: true, restart count 0 Jan 1 02:39:02.284: INFO: netserver-1 started at 2023-01-01 02:38:39 +0000 UTC (0+1 container statuses recorded) Jan 1 02:39:02.284: INFO: Container webserver ready: true, restart count 0 Jan 1 02:39:02.284: INFO: external-injector started at 2023-01-01 02:39:01 +0000 UTC (0+1 container statuses recorded) Jan 1 02:39:02.284: INFO: Container external-injector ready: false, restart count 0 Jan 1 02:39:02.284: INFO: pod-d05d599f-1b25-4249-b9c7-c64a79abebd3 started at 2023-01-01 02:30:59 +0000 UTC (0+1 container statuses recorded) Jan 1 02:39:02.284: INFO: Container write-pod ready: true, restart count 0 Jan 1 02:39:02.284: INFO: ebs-csi-node-7clrr started at 2023-01-01 02:20:28 +0000 UTC (0+3 container statuses recorded) Jan 1 02:39:02.284: INFO: Container ebs-plugin ready: true, restart count 0 Jan 1 02:39:02.284: INFO: Container liveness-probe ready: true, restart count 0 Jan 1 02:39:02.284: INFO: Container node-driver-registrar ready: true, restart count 0 Jan 1 02:39:02.284: INFO: test-rolling-update-with-lb-79bf9f6988-zljpg started at 2023-01-01 02:36:11 +0000 UTC (0+1 container statuses recorded) Jan 1 02:39:02.284: INFO: Container agnhost ready: true, restart count 0 Jan 1 02:39:02.284: INFO: httpd-deployment-5bf95dfb4d-jck92 started at 2023-01-01 02:38:57 +0000 UTC (0+1 container statuses recorded) Jan 1 02:39:02.284: INFO: Container httpd ready: false, restart count 0 Jan 1 02:39:02.284: INFO: csi-hostpathplugin-0 started at 2023-01-01 02:37:35 +0000 UTC (0+7 container statuses recorded) Jan 1 02:39:02.284: INFO: Container csi-attacher ready: true, restart count 0 Jan 1 02:39:02.284: INFO: Container csi-provisioner ready: true, restart count 0 Jan 1 02:39:02.284: INFO: Container csi-resizer ready: true, restart count 0 Jan 1 02:39:02.284: INFO: Container csi-snapshotter ready: true, restart count 0 Jan 1 02:39:02.284: INFO: Container hostpath ready: true, restart count 0 Jan 1 02:39:02.284: INFO: Container liveness-probe ready: true, restart count 0 Jan 1 02:39:02.284: INFO: Container node-driver-registrar ready: true, restart count 0 Jan 1 02:39:02.284: INFO: hostexec-ip-172-20-45-112.eu-west-3.compute.internal-cx2rk started at 2023-01-01 02:39:01 +0000 UTC (0+1 container statuses recorded) Jan 1 02:39:02.284: INFO: Container agnhost-container ready: false, restart count 0 Jan 1 02:39:02.284: INFO: service-proxy-toggled-hfxf7 started at 2023-01-01 02:37:24 +0000 UTC (0+1 container statuses recorded) Jan 1 02:39:02.284: INFO: Container service-proxy-toggled ready: true, restart count 0 Jan 1 02:39:02.284: INFO: cilium-slx98 started at 2023-01-01 02:20:28 +0000 UTC (1+1 container statuses recorded) Jan 1 02:39:02.284: INFO: Init container clean-cilium-state ready: true, restart count 0 Jan 1 02:39:02.284: INFO: Container cilium-agent ready: true, restart count 0 Jan 1 02:39:02.284: INFO: service-proxy-disabled-4bzcz started at 2023-01-01 02:37:17 +0000 UTC (0+1 container statuses recorded) Jan 1 02:39:02.284: INFO: Container service-proxy-disabled ready: true, restart count 0 Jan 1 02:39:02.648: INFO: Latency metrics for node ip-172-20-45-112.eu-west-3.compute.internal Jan 1 02:39:02.648: INFO: Logging node info for node ip-172-20-47-178.eu-west-3.compute.internal Jan 1 02:39:02.751: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-47-178.eu-west-3.compute.internal 835e5337-d2b9-45ad-bf0e-9d32514f71ba 16109 0 2023-01-01 02:20:25 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:eu-west-3 failure-domain.beta.kubernetes.io/zone:eu-west-3a kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-47-178.eu-west-3.compute.internal kubernetes.io/os:linux kubernetes.io/role:node node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:eu-west-3a topology.hostpath.csi/node:ip-172-20-47-178.eu-west-3.compute.internal topology.kubernetes.io/region:eu-west-3 topology.kubernetes.io/zone:eu-west-3a] map[csi.volume.kubernetes.io/nodeid:{"csi-hostpath-volume-expand-8523":"ip-172-20-47-178.eu-west-3.compute.internal","ebs.csi.aws.com":"i-00e1e60199f5a4d04"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kops-controller Update v1 2023-01-01 02:20:25 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}} } {kube-controller-manager Update v1 2023-01-01 02:20:25 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.2.0/24\"":{}}}} } {kubelet Update v1 2023-01-01 02:20:25 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {kubelet Update v1 2023-01-01 02:37:46 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:capacity":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.2.0/24,DoNotUseExternalID:,ProviderID:aws:///eu-west-3a/i-00e1e60199f5a4d04,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.2.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{50478186496 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4078190592 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{45430367772 0} {<nil>} 45430367772 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3973332992 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-01-01 02:38:27 +0000 UTC,LastTransitionTime:2023-01-01 02:20:25 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-01-01 02:38:27 +0000 UTC,LastTransitionTime:2023-01-01 02:20:25 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-01-01 02:38:27 +0000 UTC,LastTransitionTime:2023-01-01 02:20:25 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-01-01 02:38:27 +0000 UTC,LastTransitionTime:2023-01-01 02:20:46 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.47.178,},NodeAddress{Type:ExternalIP,Address:13.36.177.101,},NodeAddress{Type:Hostname,Address:ip-172-20-47-178.eu-west-3.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-47-178.eu-west-3.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-13-36-177-101.eu-west-3.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec243eda226c96d5573f3eaa3203df0d,SystemUUID:ec243eda-226c-96d5-573f-3eaa3203df0d,BootID:15dccaa7-d7c7-419d-b7de-76be5781a51a,KernelVersion:4.19.0-23-cloud-amd64,OSImage:Debian GNU/Linux 10 (buster),ContainerRuntimeVersion:docker://20.10.17,KubeletVersion:v1.23.15,KubeProxyVersion:v1.23.15,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:456569528,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/jessie-dnsutils@sha256:11e6a66017ba4e4b938c1612b7a54a3befcefd354796c04e1dba76873a13518e k8s.gcr.io/e2e-test-images/jessie-dnsutils:1.5],SizeBytes:253371792,},ContainerImage{Names:[nginx@sha256:0047b729188a15da49380d9506d65959cce6d40291ccfb4e039f5dc7efd33286 nginx:latest],SizeBytes:141812353,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:7e8bdd271312fd25fc5ff5a8f04727be84044eb3d7d8d03611972a6752e2e11e k8s.gcr.io/e2e-test-images/agnhost:2.39],SizeBytes:126872991,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:1b9d1b2f36cb2dbee1960e82a9344aeb11bd4c4c03abf5e1853e0559c23855e3 k8s.gcr.io/e2e-test-images/httpd:2.4.38-2],SizeBytes:123781643,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.23.15 registry.k8s.io/kube-proxy-amd64:v1.23.15],SizeBytes:112336036,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:10e231cf07c82dfab5141c1afe548dc734e5fc1f67665ec3982a325d1bd31a9a registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.0],SizeBytes:94471559,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:c8e03f60afa90a28e4bb6ec9a8d0fc36d89de4b7475cf2d613afa793ec969fe0 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.0],SizeBytes:56382329,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:4e74c0492bceddc598de1c90cc5bc14dcda94cb49fa9c5bad9d117c4834b5e08 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.1],SizeBytes:56379269,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:36c31f7e1f433c9634d24f876353e8646246d81a03c4e351202c2644daff1620 k8s.gcr.io/sig-storage/csi-resizer:v1.2.0],SizeBytes:53956308,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:ed98431376c9e944e19a465fe8ea944806714dd95416a0821096c78d66b579bd k8s.gcr.io/sig-storage/csi-snapshotter:v4.1.1],SizeBytes:53531990,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:c5be65d6679efabb969d9b019300d187437ae876f992c40911fd2892bbef3b36 k8s.gcr.io/sig-storage/csi-attacher:v3.2.0],SizeBytes:53500826,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:60ab9b3e6a030d3038c87c0d6bca2930f58d1d72823e6a4af09767dc83b696a2 k8s.gcr.io/sig-storage/csi-attacher:v3.2.1],SizeBytes:53499570,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:017727efcfeb7d053af68e51436ce8e65edbc6ca573720afb4f79c8594036955 registry.k8s.io/coredns/coredns:v1.10.0],SizeBytes:50249337,},ContainerImage{Names:[registry.k8s.io/cpa/cluster-proportional-autoscaler@sha256:68d396900aeaa072c1f27289485fdac29834045a6f3ffe369bf389d830ef572d registry.k8s.io/cpa/cluster-proportional-autoscaler:1.8.6],SizeBytes:47910609,},ContainerImage{Names:[k8s.gcr.io/sig-storage/hostpathplugin@sha256:232fe80174d60d520d36043103853a1d7ab4b7f3782cf43e45034f04ccda58ce k8s.gcr.io/sig-storage/hostpathplugin:v1.7.1],SizeBytes:29977921,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:a61d309da54641db41fb8f35718f744e9f730d4d0384f8c4b186ddc9f06cbd5f k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.1.0],SizeBytes:19662887,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:19582112,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:2dee3fe5fe861bb66c3a4ac51114f3447a4cd35870e0f2e2b558c7a400d89589 k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.2.0],SizeBytes:18699436,},ContainerImage{Names:[k8s.gcr.io/sig-storage/mock-driver@sha256:a7b517f9e0f42ffade802eef9cefa271372386b85f55b702b493241e58459793 k8s.gcr.io/sig-storage/mock-driver:v4.1.0],SizeBytes:17680993,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:17611980,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:529be2c9770add0cdd0c989115222ea9fc1be430c11095eb9f6dafcf98a36e2b k8s.gcr.io/sig-storage/livenessprobe:v2.4.0],SizeBytes:17175488,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:13616070e3f29de4417eee434a8ef472221c9e51b3d037b5a6b46cef08eb7443 k8s.gcr.io/e2e-test-images/nginx:1.14-2],SizeBytes:16032814,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf k8s.gcr.io/e2e-test-images/busybox:1.29-2],SizeBytes:1154361,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db k8s.gcr.io/pause:3.6 registry.k8s.io/pause:3.6],SizeBytes:682696,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Jan 1 02:39:02.751: INFO: Logging kubelet events for node ip-172-20-47-178.eu-west-3.compute.internal Jan 1 02:39:02.856: INFO: Logging pods the kubelet thinks is on node ip-172-20-47-178.eu-west-3.compute.internal Jan 1 02:39:02.965: INFO: cilium-kk2c2 started at 2023-01-01 02:20:26 +0000 UTC (1+1 container statuses recorded) Jan 1 02:39:02.965: INFO: Init container clean-cilium-state ready: true, restart count 0 Jan 1 02:39:02.966: INFO: Container cilium-agent ready: true, restart count 0 Jan 1 02:39:02.966: INFO: verify-service-up-host-exec-pod started at 2023-01-01 02:38:22 +0000 UTC (0+1 container statuses recorded) Jan 1 02:39:02.966: INFO: Container agnhost-container ready: true, restart count 0 Jan 1 02:39:02.966: INFO: coredns-867df8f45c-wzntb started at 2023-01-01 02:20:46 +0000 UTC (0+1 container statuses recorded) Jan 1 02:39:02.966: INFO: Container coredns ready: true, restart count 0 Jan 1 02:39:02.966: INFO: coredns-autoscaler-557ccb4c66-k4lzb started at 2023-01-01 02:20:46 +0000 UTC (0+1 container statuses recorded) Jan 1 02:39:02.966: INFO: Container autoscaler ready: true, restart count 0 Jan 1 02:39:02.966: INFO: test-rolling-update-with-lb-79bf9f6988-7nqmz started at 2023-01-01 02:36:11 +0000 UTC (0+1 container statuses recorded) Jan 1 02:39:02.966: INFO: Container agnhost ready: true, restart count 0 Jan 1 02:39:02.966: INFO: csi-hostpathplugin-0 started at 2023-01-01 02:38:49 +0000 UTC (0+7 container statuses recorded) Jan 1 02:39:02.966: INFO: Container csi-attacher ready: true, restart count 0 Jan 1 02:39:02.966: INFO: Container csi-provisioner ready: true, restart count 0 Jan 1 02:39:02.966: INFO: Container csi-resizer ready: true, restart count 0 Jan 1 02:39:02.966: INFO: Container csi-snapshotter ready: true, restart count 0 Jan 1 02:39:02.966: INFO: Container hostpath ready: true, restart count 0 Jan 1 02:39:02.966: INFO: Container liveness-probe ready: true, restart count 0 Jan 1 02:39:02.966: INFO: Container node-driver-registrar ready: true, restart count 0 Jan 1 02:39:02.966: INFO: pod-configmaps-25ed06ed-ccd8-4b11-869a-2971e465492a started at 2023-01-01 02:37:40 +0000 UTC (0+3 container statuses recorded) Jan 1 02:39:02.966: INFO: Container createcm-volume-test ready: true, restart count 0 Jan 1 02:39:02.966: INFO: Container delcm-volume-test ready: true, restart count 0 Jan 1 02:39:02.966: INFO: Container updcm-volume-test ready: true, restart count 0 Jan 1 02:39:02.966: INFO: ss2-1 started at 2023-01-01 02:37:49 +0000 UTC (0+1 container statuses recorded) Jan 1 02:39:02.966: INFO: Container webserver ready: true, restart count 0 Jan 1 02:39:02.966: INFO: simpletest.rc-nrfsq started at 2023-01-01 02:38:36 +0000 UTC (0+1 container statuses recorded) Jan 1 02:39:02.966: INFO: Container nginx ready: true, restart count 0 Jan 1 02:39:02.966: INFO: ebs-csi-node-vcprc started at 2023-01-01 02:20:26 +0000 UTC (0+3 container statuses recorded) Jan 1 02:39:02.966: INFO: Container ebs-plugin ready: true, restart count 0 Jan 1 02:39:02.966: INFO: Container liveness-probe ready: true, restart count 0 Jan 1 02:39:02.966: INFO: Container node-driver-registrar ready: true, restart count 0 Jan 1 02:39:02.966: INFO: startup-c7083725-97b9-412f-8e5d-eb07c577873c started at 2023-01-01 02:38:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:39:02.966: INFO: Container busybox ready: true, restart count 0 Jan 1 02:39:02.966: INFO: netserver-2 started at 2023-01-01 02:38:39 +0000 UTC (0+1 container statuses recorded) Jan 1 02:39:02.966: INFO: Container webserver ready: true, restart count 0 Jan 1 02:39:03.318: INFO: Latency metrics for node ip-172-20-47-178.eu-west-3.compute.internal Jan 1 02:39:03.318: INFO: Logging node info for node ip-172-20-49-224.eu-west-3.compute.internal Jan 1 02:39:03.422: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-49-224.eu-west-3.compute.internal d7d57146-c54a-4155-94f8-0f322216bfcf 14819 0 2023-01-01 02:20:28 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:eu-west-3 failure-domain.beta.kubernetes.io/zone:eu-west-3a kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-49-224.eu-west-3.compute.internal kubernetes.io/os:linux kubernetes.io/role:node node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:eu-west-3a topology.hostpath.csi/node:ip-172-20-49-224.eu-west-3.compute.internal topology.kubernetes.io/region:eu-west-3 topology.kubernetes.io/zone:eu-west-3a] map[csi.volume.kubernetes.io/nodeid:{"csi-mock-csi-mock-volumes-2230":"csi-mock-csi-mock-volumes-2230","csi-mock-csi-mock-volumes-316":"csi-mock-csi-mock-volumes-316","ebs.csi.aws.com":"i-089a107939b27c28a"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kops-controller Update v1 2023-01-01 02:20:28 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}} } {kube-controller-manager Update v1 2023-01-01 02:20:28 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.4.0/24\"":{}}}} } {kubelet Update v1 2023-01-01 02:20:28 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {kube-controller-manager Update v1 2023-01-01 02:38:04 +0000 UTC FieldsV1 {"f:status":{"f:volumesAttached":{}}} status} {kubelet Update v1 2023-01-01 02:38:04 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:capacity":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.4.0/24,DoNotUseExternalID:,ProviderID:aws:///eu-west-3a/i-089a107939b27c28a,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.4.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{50478186496 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4078206976 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{45430367772 0} {<nil>} 45430367772 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3973349376 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-01-01 02:38:04 +0000 UTC,LastTransitionTime:2023-01-01 02:20:28 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-01-01 02:38:04 +0000 UTC,LastTransitionTime:2023-01-01 02:20:28 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-01-01 02:38:04 +0000 UTC,LastTransitionTime:2023-01-01 02:20:28 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-01-01 02:38:04 +0000 UTC,LastTransitionTime:2023-01-01 02:20:48 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.49.224,},NodeAddress{Type:ExternalIP,Address:15.237.95.101,},NodeAddress{Type:Hostname,Address:ip-172-20-49-224.eu-west-3.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-49-224.eu-west-3.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-15-237-95-101.eu-west-3.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2d0a1bb33dd26f5a12c3896530131b,SystemUUID:ec2d0a1b-b33d-d26f-5a12-c3896530131b,BootID:9f64e91d-26f0-45aa-aec4-a54433e67034,KernelVersion:4.19.0-23-cloud-amd64,OSImage:Debian GNU/Linux 10 (buster),ContainerRuntimeVersion:docker://20.10.17,KubeletVersion:v1.23.15,KubeProxyVersion:v1.23.15,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:456569528,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/jessie-dnsutils@sha256:11e6a66017ba4e4b938c1612b7a54a3befcefd354796c04e1dba76873a13518e k8s.gcr.io/e2e-test-images/jessie-dnsutils:1.5],SizeBytes:253371792,},ContainerImage{Names:[nginx@sha256:0047b729188a15da49380d9506d65959cce6d40291ccfb4e039f5dc7efd33286 nginx:latest],SizeBytes:141812353,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:7e8bdd271312fd25fc5ff5a8f04727be84044eb3d7d8d03611972a6752e2e11e k8s.gcr.io/e2e-test-images/agnhost:2.39],SizeBytes:126872991,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:1b9d1b2f36cb2dbee1960e82a9344aeb11bd4c4c03abf5e1853e0559c23855e3 k8s.gcr.io/e2e-test-images/httpd:2.4.38-2],SizeBytes:123781643,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.23.15 registry.k8s.io/kube-proxy-amd64:v1.23.15],SizeBytes:112336036,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:10e231cf07c82dfab5141c1afe548dc734e5fc1f67665ec3982a325d1bd31a9a registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.0],SizeBytes:94471559,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:c8e03f60afa90a28e4bb6ec9a8d0fc36d89de4b7475cf2d613afa793ec969fe0 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.0],SizeBytes:56382329,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:4e74c0492bceddc598de1c90cc5bc14dcda94cb49fa9c5bad9d117c4834b5e08 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.1],SizeBytes:56379269,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:36c31f7e1f433c9634d24f876353e8646246d81a03c4e351202c2644daff1620 k8s.gcr.io/sig-storage/csi-resizer:v1.2.0],SizeBytes:53956308,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:ed98431376c9e944e19a465fe8ea944806714dd95416a0821096c78d66b579bd k8s.gcr.io/sig-storage/csi-snapshotter:v4.1.1],SizeBytes:53531990,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:c5be65d6679efabb969d9b019300d187437ae876f992c40911fd2892bbef3b36 k8s.gcr.io/sig-storage/csi-attacher:v3.2.0],SizeBytes:53500826,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:60ab9b3e6a030d3038c87c0d6bca2930f58d1d72823e6a4af09767dc83b696a2 k8s.gcr.io/sig-storage/csi-attacher:v3.2.1],SizeBytes:53499570,},ContainerImage{Names:[k8s.gcr.io/sig-storage/hostpathplugin@sha256:232fe80174d60d520d36043103853a1d7ab4b7f3782cf43e45034f04ccda58ce k8s.gcr.io/sig-storage/hostpathplugin:v1.7.1],SizeBytes:29977921,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:a61d309da54641db41fb8f35718f744e9f730d4d0384f8c4b186ddc9f06cbd5f k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.1.0],SizeBytes:19662887,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:19582112,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:2dee3fe5fe861bb66c3a4ac51114f3447a4cd35870e0f2e2b558c7a400d89589 k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.2.0],SizeBytes:18699436,},ContainerImage{Names:[k8s.gcr.io/sig-storage/mock-driver@sha256:a7b517f9e0f42ffade802eef9cefa271372386b85f55b702b493241e58459793 k8s.gcr.io/sig-storage/mock-driver:v4.1.0],SizeBytes:17680993,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:17611980,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:529be2c9770add0cdd0c989115222ea9fc1be430c11095eb9f6dafcf98a36e2b k8s.gcr.io/sig-storage/livenessprobe:v2.4.0],SizeBytes:17175488,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:13616070e3f29de4417eee434a8ef472221c9e51b3d037b5a6b46cef08eb7443 k8s.gcr.io/e2e-test-images/nginx:1.14-2],SizeBytes:16032814,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf k8s.gcr.io/e2e-test-images/busybox:1.29-2],SizeBytes:1154361,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db k8s.gcr.io/pause:3.6 registry.k8s.io/pause:3.6],SizeBytes:682696,},},VolumesInUse:[kubernetes.io/csi/csi-mock-csi-mock-volumes-2230^4],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/csi-mock-csi-mock-volumes-2230^4,DevicePath:,},},Config:nil,},} Jan 1 02:39:03.422: INFO: Logging kubelet events for node ip-172-20-49-224.eu-west-3.compute.internal Jan 1 02:39:03.535: INFO: Logging pods the kubelet thinks is on node ip-172-20-49-224.eu-west-3.compute.internal Jan 1 02:39:03.652: INFO: csi-hostpathplugin-0 started at 2023-01-01 02:37:50 +0000 UTC (0+7 container statuses recorded) Jan 1 02:39:03.652: INFO: Container csi-attacher ready: false, restart count 0 Jan 1 02:39:03.652: INFO: Container csi-provisioner ready: false, restart count 0 Jan 1 02:39:03.652: INFO: Container csi-resizer ready: false, restart count 0 Jan 1 02:39:03.652: INFO: Container csi-snapshotter ready: false, restart count 0 Jan 1 02:39:03.652: INFO: Container hostpath ready: false, restart count 0 Jan 1 02:39:03.652: INFO: Container liveness-probe ready: false, restart count 0 Jan 1 02:39:03.652: INFO: Container node-driver-registrar ready: false, restart count 0 Jan 1 02:39:03.652: INFO: downwardapi-volume-222d4d03-147d-4c1f-8fda-e53a7e62a80c started at 2023-01-01 02:37:54 +0000 UTC (0+1 container statuses recorded) Jan 1 02:39:03.652: INFO: Container client-container ready: false, restart count 0 Jan 1 02:39:03.652: INFO: csi-mockplugin-attacher-0 started at 2023-01-01 02:37:35 +0000 UTC (0+1 container statuses recorded) Jan 1 02:39:03.652: INFO: Container csi-attacher ready: true, restart count 0 Jan 1 02:39:03.652: INFO: httpd-deployment-5bf95dfb4d-xsvvl started at 2023-01-01 02:38:58 +0000 UTC (0+1 container statuses recorded) Jan 1 02:39:03.652: INFO: Container httpd ready: false, restart count 0 Jan 1 02:39:03.652: INFO: hostexec-ip-172-20-49-224.eu-west-3.compute.internal-htrs6 started at 2023-01-01 02:37:39 +0000 UTC (0+1 container statuses recorded) Jan 1 02:39:03.652: INFO: Container agnhost-container ready: true, restart count 0 Jan 1 02:39:03.652: INFO: verify-service-up-exec-pod-n7wwh started at 2023-01-01 02:38:24 +0000 UTC (0+1 container statuses recorded) Jan 1 02:39:03.652: INFO: Container agnhost-container ready: false, restart count 0 Jan 1 02:39:03.652: INFO: pod-b5541035-84e2-4398-9961-0e8a5b7d2e7b started at 2023-01-01 02:38:46 +0000 UTC (0+1 container statuses recorded) Jan 1 02:39:03.652: INFO: Container write-pod ready: false, restart count 0 Jan 1 02:39:03.652: INFO: cilium-78jll started at 2023-01-01 02:20:29 +0000 UTC (1+1 container statuses recorded) Jan 1 02:39:03.652: INFO: Init container clean-cilium-state ready: true, restart count 0 Jan 1 02:39:03.652: INFO: Container cilium-agent ready: true, restart count 0 Jan 1 02:39:03.652: INFO: ebs-csi-node-tzntf started at 2023-01-01 02:20:29 +0000 UTC (0+3 container statuses recorded) Jan 1 02:39:03.652: INFO: Container ebs-plugin ready: true, restart count 0 Jan 1 02:39:03.652: INFO: Container liveness-probe ready: true, restart count 0 Jan 1 02:39:03.652: INFO: Container node-driver-registrar ready: true, restart count 0 Jan 1 02:39:03.652: INFO: service-proxy-toggled-lzms2 started at 2023-01-01 02:37:24 +0000 UTC (0+1 container statuses recorded) Jan 1 02:39:03.652: INFO: Container service-proxy-toggled ready: true, restart count 0 Jan 1 02:39:03.652: INFO: pod-projected-configmaps-2daa203d-dbcb-4d20-80ba-19d57fbb12ef started at 2023-01-01 02:37:38 +0000 UTC (0+1 container statuses recorded) Jan 1 02:39:03.652: INFO: Container agnhost-container ready: true, restart count 0 Jan 1 02:39:03.652: INFO: ss2-2 started at 2023-01-01 02:37:56 +0000 UTC (0+1 container statuses recorded) Jan 1 02:39:03.652: INFO: Container webserver ready: false, restart count 0 Jan 1 02:39:03.652: INFO: pod-host-path-test started at 2023-01-01 02:37:55 +0000 UTC (0+2 container statuses recorded) Jan 1 02:39:03.652: INFO: Container test-container-1 ready: false, restart count 0 Jan 1 02:39:03.652: INFO: Container test-container-2 ready: false, restart count 0 Jan 1 02:39:03.652: INFO: csi-mockplugin-0 started at 2023-01-01 02:37:35 +0000 UTC (0+3 container statuses recorded) Jan 1 02:39:03.652: INFO: Container csi-provisioner ready: true, restart count 0 Jan 1 02:39:03.652: INFO: Container driver-registrar ready: true, restart count 0 Jan 1 02:39:03.652: INFO: Container mock ready: true, restart count 0 Jan 1 02:39:03.652: INFO: netserver-3 started at 2023-01-01 02:38:39 +0000 UTC (0+1 container statuses recorded) Jan 1 02:39:03.652: INFO: Container webserver ready: false, restart count 0 Jan 1 02:39:03.652: INFO: liveness-b61a07c8-7116-4bf3-bf05-31114b2bce91 started at 2023-01-01 02:36:05 +0000 UTC (0+1 container statuses recorded) Jan 1 02:39:03.652: INFO: Container agnhost-container ready: true, restart count 0 Jan 1 02:39:03.652: INFO: test-rolling-update-with-lb-79bf9f6988-wgh57 started at 2023-01-01 02:36:11 +0000 UTC (0+1 container statuses recorded) Jan 1 02:39:03.653: INFO: Container agnhost ready: true, restart count 0 Jan 1 02:39:03.653: INFO: service-proxy-disabled-jm2xc started at 2023-01-01 02:37:17 +0000 UTC (0+1 container statuses recorded) Jan 1 02:39:03.653: INFO: Container service-proxy-disabled ready: true, restart count 0 Jan 1 02:39:03.653: INFO: pod-45ef951d-5be5-4d7e-a9ec-96715e495526 started at 2023-01-01 02:38:10 +0000 UTC (0+1 container statuses recorded) Jan 1 02:39:03.653: INFO: Container write-pod ready: false, restart count 0 Jan 1 02:39:03.653: INFO: csi-mockplugin-attacher-0 started at 2023-01-01 02:31:26 +0000 UTC (0+1 container statuses recorded) Jan 1 02:39:03.653: INFO: Container csi-attacher ready: true, restart count 0 Jan 1 02:39:03.653: INFO: pvc-volume-tester-5l7pg started at 2023-01-01 02:38:03 +0000 UTC (0+1 container statuses recorded) Jan 1 02:39:03.653: INFO: Container volume-tester ready: true, restart count 0 Jan 1 02:39:03.653: INFO: csi-mockplugin-0 started at 2023-01-01 02:31:26 +0000 UTC (0+3 container statuses recorded) Jan 1 02:39:03.653: INFO: Container csi-provisioner ready: true, restart count 0 Jan 1 02:39:03.653: INFO: Container driver-registrar ready: true, restart count 0 Jan 1 02:39:03.653: INFO: Container mock ready: true, restart count 0 Jan 1 02:39:03.653: INFO: hostexec-ip-172-20-49-224.eu-west-3.compute.internal-nqr5w started at 2023-01-01 02:38:40 +0000 UTC (0+1 container statuses recorded) Jan 1 02:39:03.653: INFO: Container agnhost-container ready: true, restart count 0 Jan 1 02:39:03.653: INFO: pvc-volume-tester-c5cxh started at 2023-01-01 02:37:09 +0000 UTC (0+1 container statuses recorded) Jan 1 02:39:03.653: INFO: Container volume-tester ready: true, restart count 0 Jan 1 02:39:03.653: INFO: pod-23787fb4-0bb3-4a60-934a-76bbe2673115 started at 2023-01-01 02:37:45 +0000 UTC (0+1 container statuses recorded) Jan 1 02:39:03.653: INFO: Container write-pod ready: true, restart count 0 Jan 1 02:39:04.129: INFO: Latency metrics for node ip-172-20-49-224.eu-west-3.compute.internal Jan 1 02:39:04.129: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready �[1mSTEP�[0m: Destroying namespace "fsgroupchangepolicy-1819" for this suite.
Find pod-d05d599f-1b25-4249-b9c7-c64a79abebd3 mentions in log files
go run hack/e2e.go -v --test --test_args='--ginkgo.focus=Kubernetes\se2e\ssuite\s\[sig\-cli\]\sKubectl\sclient\sSimple\spod\sshould\ssupport\sexec\sthrough\skubectl\sproxy$'
/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/kubectl/kubectl.go:381 Jan 1 02:31:25.363: Expected <bool>: false to equal <bool>: true /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/kubectl/kubectl.go:385from junit_18.xml
[BeforeEach] [sig-cli] Kubectl client /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:185 �[1mSTEP�[0m: Creating a kubernetes client Jan 1 02:26:22.865: INFO: >>> kubeConfig: /root/.kube/config �[1mSTEP�[0m: Building a namespace api object, basename kubectl �[1mSTEP�[0m: Waiting for a default service account to be provisioned in namespace �[1mSTEP�[0m: Waiting for kube-root-ca.crt to be provisioned in namespace [BeforeEach] [sig-cli] Kubectl client /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/kubectl/kubectl.go:244 [BeforeEach] Simple pod /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/kubectl/kubectl.go:381 �[1mSTEP�[0m: creating the pod from Jan 1 02:26:23.601: INFO: Running '/home/prow/go/src/k8s.io/kops/_rundir/08fd59e0-897a-11ed-b47a-e2b55b2f9198/kubectl --server=https://api.e2e-e2e-kops-grid-cilium-eni-deb10-k23-docker.test-cncf-aws.k8s.io --kubeconfig=/root/.kube/config --namespace=kubectl-1532 create -f -' Jan 1 02:26:24.250: INFO: stderr: "" Jan 1 02:26:24.250: INFO: stdout: "pod/httpd created\n" Jan 1 02:26:24.250: INFO: Waiting up to 5m0s for 1 pods to be running and ready: [httpd] Jan 1 02:26:24.250: INFO: Waiting up to 5m0s for pod "httpd" in namespace "kubectl-1532" to be "running and ready" Jan 1 02:26:24.355: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 104.765301ms Jan 1 02:26:26.462: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 2.211472452s Jan 1 02:26:28.568: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 4.317081046s Jan 1 02:26:30.674: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 6.423441804s Jan 1 02:26:32.781: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 8.530685844s Jan 1 02:26:34.886: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 10.635621893s Jan 1 02:26:36.992: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 12.741183436s Jan 1 02:26:39.098: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 14.847540227s Jan 1 02:26:41.204: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 16.953593466s Jan 1 02:26:43.309: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 19.058702562s Jan 1 02:26:45.415: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 21.16451487s Jan 1 02:26:47.520: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 23.269731406s Jan 1 02:26:49.625: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 25.374917032s Jan 1 02:26:51.731: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 27.480325631s Jan 1 02:26:53.837: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 29.586104477s Jan 1 02:26:55.942: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 31.691678026s Jan 1 02:26:58.047: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 33.796612781s Jan 1 02:27:00.152: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 35.901961603s Jan 1 02:27:02.258: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 38.007491848s Jan 1 02:27:04.363: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 40.112615438s Jan 1 02:27:06.469: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 42.218655102s Jan 1 02:27:08.575: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 44.324837481s Jan 1 02:27:10.682: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 46.431093085s Jan 1 02:27:12.787: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 48.536431402s Jan 1 02:27:14.893: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 50.642247014s Jan 1 02:27:16.998: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 52.747611663s Jan 1 02:27:19.103: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 54.852602902s Jan 1 02:27:21.208: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 56.957880712s Jan 1 02:27:23.313: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 59.062966414s Jan 1 02:27:25.420: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 1m1.169343281s Jan 1 02:27:27.525: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 1m3.274258354s Jan 1 02:27:29.630: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 1m5.379318723s Jan 1 02:27:31.735: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 1m7.484796273s Jan 1 02:27:33.841: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 1m9.590233262s Jan 1 02:27:35.946: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 1m11.695091197s Jan 1 02:27:38.051: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 1m13.800388587s Jan 1 02:27:40.158: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 1m15.907617227s Jan 1 02:27:42.263: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 1m18.012799171s Jan 1 02:27:44.370: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 1m20.119128838s Jan 1 02:27:46.474: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 1m22.224022319s Jan 1 02:27:48.580: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 1m24.329393829s Jan 1 02:27:50.685: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 1m26.434621739s Jan 1 02:27:52.790: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 1m28.539887587s Jan 1 02:27:54.895: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 1m30.644526505s Jan 1 02:27:57.001: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 1m32.750486462s Jan 1 02:27:59.106: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 1m34.855628707s Jan 1 02:28:01.211: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 1m36.960553911s Jan 1 02:28:03.317: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 1m39.06635456s Jan 1 02:28:05.425: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 1m41.174125578s Jan 1 02:28:07.530: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 1m43.279694904s Jan 1 02:28:09.635: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 1m45.384415322s Jan 1 02:28:11.740: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 1m47.489287388s Jan 1 02:28:13.845: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 1m49.59492652s Jan 1 02:28:15.951: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 1m51.700979816s Jan 1 02:28:18.058: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 1m53.807065356s Jan 1 02:28:20.163: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 1m55.912735194s Jan 1 02:28:22.271: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 1m58.020460129s Jan 1 02:28:24.376: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 2m0.12541437s Jan 1 02:28:26.481: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 2m2.230891037s Jan 1 02:28:28.586: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 2m4.335634654s Jan 1 02:28:30.692: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 2m6.441628064s Jan 1 02:28:32.797: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 2m8.546406821s Jan 1 02:28:34.903: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 2m10.652195073s Jan 1 02:28:37.008: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 2m12.75758673s Jan 1 02:28:39.114: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 2m14.863259356s Jan 1 02:28:41.221: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 2m16.970072093s Jan 1 02:28:43.326: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 2m19.075216309s Jan 1 02:28:45.431: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 2m21.180495598s Jan 1 02:28:47.536: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 2m23.285567326s Jan 1 02:28:49.641: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 2m25.390802144s Jan 1 02:28:51.747: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 2m27.49606136s Jan 1 02:28:53.853: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 2m29.602318469s Jan 1 02:28:55.959: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 2m31.708137175s Jan 1 02:28:58.064: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 2m33.813671276s Jan 1 02:29:00.170: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 2m35.919734265s Jan 1 02:29:02.276: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 2m38.025233915s Jan 1 02:29:04.381: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 2m40.130571615s Jan 1 02:29:06.487: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 2m42.236527555s Jan 1 02:29:08.592: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 2m44.341934227s Jan 1 02:29:10.698: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 2m46.447127658s Jan 1 02:29:12.803: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 2m48.553034984s Jan 1 02:29:14.910: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 2m50.659106035s Jan 1 02:29:17.015: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 2m52.764433558s Jan 1 02:29:19.120: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 2m54.869713891s Jan 1 02:29:21.226: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 2m56.975901168s Jan 1 02:29:23.332: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 2m59.081230327s Jan 1 02:29:25.438: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 3m1.187236413s Jan 1 02:29:27.544: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 3m3.293058393s Jan 1 02:29:29.648: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 3m5.398010716s Jan 1 02:29:31.754: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 3m7.503791556s Jan 1 02:29:33.859: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 3m9.608950598s Jan 1 02:29:35.965: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 3m11.71500263s Jan 1 02:29:38.071: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 3m13.820959306s Jan 1 02:29:40.178: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 3m15.927099118s Jan 1 02:29:42.284: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 3m18.033203691s Jan 1 02:29:44.390: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 3m20.139510929s Jan 1 02:29:46.496: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 3m22.245324385s Jan 1 02:29:48.602: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 3m24.351400682s Jan 1 02:29:50.707: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 3m26.456758951s Jan 1 02:29:52.813: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 3m28.562933465s Jan 1 02:29:54.919: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 3m30.668160803s Jan 1 02:29:57.025: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 3m32.774495292s Jan 1 02:29:59.130: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 3m34.87959859s Jan 1 02:30:01.237: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 3m36.98642855s Jan 1 02:30:03.343: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 3m39.092525624s Jan 1 02:30:05.449: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 3m41.198507042s Jan 1 02:30:07.555: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 3m43.304974154s Jan 1 02:30:09.662: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 3m45.411266047s Jan 1 02:30:11.767: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 3m47.517008192s Jan 1 02:30:13.873: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 3m49.622902294s Jan 1 02:30:15.979: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 3m51.728287026s Jan 1 02:30:18.084: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 3m53.833344968s Jan 1 02:30:20.189: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 3m55.938750192s Jan 1 02:30:22.294: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 3m58.04371965s Jan 1 02:30:24.400: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 4m0.149952458s Jan 1 02:30:26.507: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 4m2.256308315s Jan 1 02:30:28.612: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 4m4.361224601s Jan 1 02:30:30.718: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 4m6.467639584s Jan 1 02:30:32.824: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 4m8.573666005s Jan 1 02:30:34.929: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 4m10.678982946s Jan 1 02:30:37.035: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 4m12.784503816s Jan 1 02:30:39.141: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 4m14.890340018s Jan 1 02:30:41.246: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 4m16.995268025s Jan 1 02:30:43.352: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 4m19.101522603s Jan 1 02:30:45.458: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 4m21.207792532s Jan 1 02:30:47.564: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 4m23.313604234s Jan 1 02:30:49.670: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 4m25.419324837s Jan 1 02:30:51.776: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 4m27.525468428s Jan 1 02:30:53.881: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 4m29.630228111s Jan 1 02:30:55.986: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 4m31.736018112s Jan 1 02:30:58.093: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 4m33.842127048s Jan 1 02:31:00.198: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 4m35.947717892s Jan 1 02:31:02.305: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 4m38.05409641s Jan 1 02:31:04.411: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 4m40.160077424s Jan 1 02:31:06.517: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 4m42.266385075s Jan 1 02:31:08.622: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 4m44.371595313s Jan 1 02:31:10.727: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 4m46.476940969s Jan 1 02:31:12.835: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 4m48.584064798s Jan 1 02:31:14.940: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 4m50.68938843s Jan 1 02:31:17.045: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 4m52.794393753s Jan 1 02:31:19.150: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 4m54.899593515s Jan 1 02:31:21.256: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 4m57.00517743s Jan 1 02:31:23.362: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 4m59.111146042s Jan 1 02:31:25.362: INFO: Pod httpd failed to be running and ready. Jan 1 02:31:25.362: INFO: Wanted all 1 pods to be running and ready. Result: false. Pods: [httpd] Jan 1 02:31:25.363: FAIL: Expected <bool>: false to equal <bool>: true Full Stack Trace k8s.io/kubernetes/test/e2e/kubectl.glob..func1.8.1() /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/kubectl/kubectl.go:385 +0x251 k8s.io/kubernetes/test/e2e.RunE2ETests(0xc000741fb0) _output/dockerized/go/src/k8s.io/kubernetes/test/e2e/e2e.go:133 +0x697 k8s.io/kubernetes/test/e2e.TestE2E(0x243a8f9) _output/dockerized/go/src/k8s.io/kubernetes/test/e2e/e2e_test.go:136 +0x19 testing.tRunner(0xc000e5e1a0, 0x735e880) /usr/local/go/src/testing/testing.go:1259 +0x102 created by testing.(*T).Run /usr/local/go/src/testing/testing.go:1306 +0x35a [AfterEach] Simple pod /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/kubectl/kubectl.go:387 �[1mSTEP�[0m: using delete to clean up resources Jan 1 02:31:25.364: INFO: Running '/home/prow/go/src/k8s.io/kops/_rundir/08fd59e0-897a-11ed-b47a-e2b55b2f9198/kubectl --server=https://api.e2e-e2e-kops-grid-cilium-eni-deb10-k23-docker.test-cncf-aws.k8s.io --kubeconfig=/root/.kube/config --namespace=kubectl-1532 delete --grace-period=0 --force -f -' Jan 1 02:31:25.881: INFO: stderr: "warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely.\n" Jan 1 02:31:25.881: INFO: stdout: "pod \"httpd\" force deleted\n" Jan 1 02:31:25.881: INFO: Running '/home/prow/go/src/k8s.io/kops/_rundir/08fd59e0-897a-11ed-b47a-e2b55b2f9198/kubectl --server=https://api.e2e-e2e-kops-grid-cilium-eni-deb10-k23-docker.test-cncf-aws.k8s.io --kubeconfig=/root/.kube/config --namespace=kubectl-1532 get rc,svc -l name=httpd --no-headers' Jan 1 02:31:26.382: INFO: stderr: "No resources found in kubectl-1532 namespace.\n" Jan 1 02:31:26.382: INFO: stdout: "" Jan 1 02:31:26.382: INFO: Running '/home/prow/go/src/k8s.io/kops/_rundir/08fd59e0-897a-11ed-b47a-e2b55b2f9198/kubectl --server=https://api.e2e-e2e-kops-grid-cilium-eni-deb10-k23-docker.test-cncf-aws.k8s.io --kubeconfig=/root/.kube/config --namespace=kubectl-1532 get pods -l name=httpd -o go-template={{ range .items }}{{ if not .metadata.deletionTimestamp }}{{ .metadata.name }}{{ "\n" }}{{ end }}{{ end }}' Jan 1 02:31:26.769: INFO: stderr: "" Jan 1 02:31:26.769: INFO: stdout: "" [AfterEach] [sig-cli] Kubectl client /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:186 �[1mSTEP�[0m: Collecting events from namespace "kubectl-1532". �[1mSTEP�[0m: Found 12 events. Jan 1 02:31:26.874: INFO: At 2023-01-01 02:26:24 +0000 UTC - event for httpd: {default-scheduler } Scheduled: Successfully assigned kubectl-1532/httpd to ip-172-20-49-224.eu-west-3.compute.internal Jan 1 02:31:26.875: INFO: At 2023-01-01 02:26:25 +0000 UTC - event for httpd: {kubelet ip-172-20-49-224.eu-west-3.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "6660a4a5c75b9f4828c80b770529942620f80cc71fba29387515119c7b499e3a" network for pod "httpd": networkPlugin cni failed to set up pod "httpd_kubectl-1532" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 1 02:31:26.875: INFO: At 2023-01-01 02:26:25 +0000 UTC - event for httpd: {kubelet ip-172-20-49-224.eu-west-3.compute.internal} SandboxChanged: Pod sandbox changed, it will be killed and re-created. Jan 1 02:31:26.875: INFO: At 2023-01-01 02:26:26 +0000 UTC - event for httpd: {kubelet ip-172-20-49-224.eu-west-3.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "401f97b26f1d6cd7dec05acac7e3761750fe4cef0349c19f92bece3958dcb4b3" network for pod "httpd": networkPlugin cni failed to set up pod "httpd_kubectl-1532" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 1 02:31:26.875: INFO: At 2023-01-01 02:26:27 +0000 UTC - event for httpd: {kubelet ip-172-20-49-224.eu-west-3.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "74db193e1cdbbb5c7ce53ff0355c2723aab5d947549292cfef1ce2e7346b69ea" network for pod "httpd": networkPlugin cni failed to set up pod "httpd_kubectl-1532" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 1 02:31:26.875: INFO: At 2023-01-01 02:26:29 +0000 UTC - event for httpd: {kubelet ip-172-20-49-224.eu-west-3.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "994082cdcf8d1bec2a5ccbfd68250c8e2987a59b1296baba6512ffaff11730bc" network for pod "httpd": networkPlugin cni failed to set up pod "httpd_kubectl-1532" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 1 02:31:26.875: INFO: At 2023-01-01 02:26:31 +0000 UTC - event for httpd: {kubelet ip-172-20-49-224.eu-west-3.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "5e6b734b75d1f55489efa45083336e387da3eb212d7f8e144214ca522d6434ed" network for pod "httpd": networkPlugin cni failed to set up pod "httpd_kubectl-1532" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 1 02:31:26.875: INFO: At 2023-01-01 02:26:32 +0000 UTC - event for httpd: {kubelet ip-172-20-49-224.eu-west-3.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "1e41b2fce5cef45dd165d2e2c1a5433c98c9a1220287719cdf5a49cc2a08fb15" network for pod "httpd": networkPlugin cni failed to set up pod "httpd_kubectl-1532" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 1 02:31:26.875: INFO: At 2023-01-01 02:26:34 +0000 UTC - event for httpd: {kubelet ip-172-20-49-224.eu-west-3.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "676d5c28174e96def658e89eba8e47e1ebbab07435cf55995c5245c842e5ac08" network for pod "httpd": networkPlugin cni failed to set up pod "httpd_kubectl-1532" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 1 02:31:26.875: INFO: At 2023-01-01 02:26:35 +0000 UTC - event for httpd: {kubelet ip-172-20-49-224.eu-west-3.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "d00a70bf610817910e7a540a625bf192469aff73fb69026ee040e219a549e260" network for pod "httpd": networkPlugin cni failed to set up pod "httpd_kubectl-1532" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 1 02:31:26.875: INFO: At 2023-01-01 02:26:37 +0000 UTC - event for httpd: {kubelet ip-172-20-49-224.eu-west-3.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "32b53046c508130166401b8a33460263a8004f48dd99217ae661319aba9ee089" network for pod "httpd": networkPlugin cni failed to set up pod "httpd_kubectl-1532" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 1 02:31:26.875: INFO: At 2023-01-01 02:26:38 +0000 UTC - event for httpd: {kubelet ip-172-20-49-224.eu-west-3.compute.internal} FailedCreatePodSandBox: (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "279e5fb45677df99635cba25119842bdc2fa91fc5419a29eae7b473d2f26dd41" network for pod "httpd": networkPlugin cni failed to set up pod "httpd_kubectl-1532" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 1 02:31:26.979: INFO: POD NODE PHASE GRACE CONDITIONS Jan 1 02:31:26.979: INFO: Jan 1 02:31:27.086: INFO: Logging node info for node ip-172-20-33-66.eu-west-3.compute.internal Jan 1 02:31:27.192: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-33-66.eu-west-3.compute.internal ec4b59b3-62db-4040-aad2-4c3249f80fe8 9096 0 2023-01-01 02:20:25 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:eu-west-3 failure-domain.beta.kubernetes.io/zone:eu-west-3a io.kubernetes.storage.mock/node:some-mock-node kubelet_cleanup:true kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-33-66.eu-west-3.compute.internal kubernetes.io/os:linux kubernetes.io/role:node node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:eu-west-3a topology.kubernetes.io/region:eu-west-3 topology.kubernetes.io/zone:eu-west-3a] map[csi.volume.kubernetes.io/nodeid:{"csi-mock-csi-mock-volumes-8129":"csi-mock-csi-mock-volumes-8129","ebs.csi.aws.com":"i-028313673124ac4ad"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kops-controller Update v1 2023-01-01 02:20:25 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}} } {kube-controller-manager Update v1 2023-01-01 02:20:25 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.1.0/24\"":{}}}} } {kubelet Update v1 2023-01-01 02:20:25 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {e2e.test Update v1 2023-01-01 02:25:15 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubelet_cleanup":{}}}} } {kubelet Update v1 2023-01-01 02:31:10 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:io.kubernetes.storage.mock/node":{},"f:topology.ebs.csi.aws.com/zone":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.1.0/24,DoNotUseExternalID:,ProviderID:aws:///eu-west-3a/i-028313673124ac4ad,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.1.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{50478186496 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4078190592 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{45430367772 0} {<nil>} 45430367772 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3973332992 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-01-01 02:30:38 +0000 UTC,LastTransitionTime:2023-01-01 02:20:25 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-01-01 02:30:38 +0000 UTC,LastTransitionTime:2023-01-01 02:20:25 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-01-01 02:30:38 +0000 UTC,LastTransitionTime:2023-01-01 02:20:25 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-01-01 02:30:38 +0000 UTC,LastTransitionTime:2023-01-01 02:20:46 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.33.66,},NodeAddress{Type:ExternalIP,Address:52.47.140.112,},NodeAddress{Type:Hostname,Address:ip-172-20-33-66.eu-west-3.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-33-66.eu-west-3.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-52-47-140-112.eu-west-3.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2ad2ce1a87d508c178acf6215c5767,SystemUUID:ec2ad2ce-1a87-d508-c178-acf6215c5767,BootID:b3b60783-d786-412b-a36a-802bcfae604d,KernelVersion:4.19.0-23-cloud-amd64,OSImage:Debian GNU/Linux 10 (buster),ContainerRuntimeVersion:docker://20.10.17,KubeletVersion:v1.23.15,KubeProxyVersion:v1.23.15,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:456569528,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/jessie-dnsutils@sha256:11e6a66017ba4e4b938c1612b7a54a3befcefd354796c04e1dba76873a13518e k8s.gcr.io/e2e-test-images/jessie-dnsutils:1.5],SizeBytes:253371792,},ContainerImage{Names:[nginx@sha256:0047b729188a15da49380d9506d65959cce6d40291ccfb4e039f5dc7efd33286 nginx:latest],SizeBytes:141812353,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:7e8bdd271312fd25fc5ff5a8f04727be84044eb3d7d8d03611972a6752e2e11e k8s.gcr.io/e2e-test-images/agnhost:2.39],SizeBytes:126872991,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:1b9d1b2f36cb2dbee1960e82a9344aeb11bd4c4c03abf5e1853e0559c23855e3 k8s.gcr.io/e2e-test-images/httpd:2.4.38-2],SizeBytes:123781643,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.23.15 registry.k8s.io/kube-proxy-amd64:v1.23.15],SizeBytes:112336036,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:10e231cf07c82dfab5141c1afe548dc734e5fc1f67665ec3982a325d1bd31a9a registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.0],SizeBytes:94471559,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:c8e03f60afa90a28e4bb6ec9a8d0fc36d89de4b7475cf2d613afa793ec969fe0 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.0],SizeBytes:56382329,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:c5be65d6679efabb969d9b019300d187437ae876f992c40911fd2892bbef3b36 k8s.gcr.io/sig-storage/csi-attacher:v3.2.0],SizeBytes:53500826,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:017727efcfeb7d053af68e51436ce8e65edbc6ca573720afb4f79c8594036955 registry.k8s.io/coredns/coredns:v1.10.0],SizeBytes:50249337,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:a61d309da54641db41fb8f35718f744e9f730d4d0384f8c4b186ddc9f06cbd5f k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.1.0],SizeBytes:19662887,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:19582112,},ContainerImage{Names:[k8s.gcr.io/sig-storage/mock-driver@sha256:a7b517f9e0f42ffade802eef9cefa271372386b85f55b702b493241e58459793 k8s.gcr.io/sig-storage/mock-driver:v4.1.0],SizeBytes:17680993,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:17611980,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:13616070e3f29de4417eee434a8ef472221c9e51b3d037b5a6b46cef08eb7443 k8s.gcr.io/e2e-test-images/nginx:1.14-2],SizeBytes:16032814,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nonewprivs@sha256:8ac1264691820febacf3aea5d152cbde6d10685731ec14966a9401c6f47a68ac k8s.gcr.io/e2e-test-images/nonewprivs:1.3],SizeBytes:7107254,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf k8s.gcr.io/e2e-test-images/busybox:1.29-2],SizeBytes:1154361,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db k8s.gcr.io/pause:3.6 registry.k8s.io/pause:3.6],SizeBytes:682696,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Jan 1 02:31:27.192: INFO: Logging kubelet events for node ip-172-20-33-66.eu-west-3.compute.internal Jan 1 02:31:27.299: INFO: Logging pods the kubelet thinks is on node ip-172-20-33-66.eu-west-3.compute.internal Jan 1 02:31:27.417: INFO: coredns-867df8f45c-cqmp7 started at 2023-01-01 02:21:18 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:27.417: INFO: Container coredns ready: true, restart count 0 Jan 1 02:31:27.417: INFO: netserver-0 started at 2023-01-01 02:25:27 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:27.417: INFO: Container webserver ready: true, restart count 0 Jan 1 02:31:27.417: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-l9hnv started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:27.417: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:31:27.417: INFO: hostexec-ip-172-20-33-66.eu-west-3.compute.internal-qm7jt started at 2023-01-01 02:27:26 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:27.417: INFO: Container agnhost-container ready: true, restart count 0 Jan 1 02:31:27.417: INFO: local-client started at 2023-01-01 02:31:07 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:27.417: INFO: Container local-client ready: false, restart count 0 Jan 1 02:31:27.417: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-vnv2x started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:27.417: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:31:27.417: INFO: csi-mockplugin-0 started at 2023-01-01 02:26:15 +0000 UTC (0+4 container statuses recorded) Jan 1 02:31:27.417: INFO: Container busybox ready: true, restart count 0 Jan 1 02:31:27.417: INFO: Container csi-provisioner ready: true, restart count 0 Jan 1 02:31:27.417: INFO: Container driver-registrar ready: true, restart count 0 Jan 1 02:31:27.417: INFO: Container mock ready: true, restart count 0 Jan 1 02:31:27.417: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-csnzw started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:27.417: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: false, restart count 0 Jan 1 02:31:27.417: INFO: ebs-csi-node-vsxjc started at 2023-01-01 02:20:26 +0000 UTC (0+3 container statuses recorded) Jan 1 02:31:27.417: INFO: Container ebs-plugin ready: true, restart count 0 Jan 1 02:31:27.417: INFO: Container liveness-probe ready: true, restart count 0 Jan 1 02:31:27.417: INFO: Container node-driver-registrar ready: true, restart count 0 Jan 1 02:31:27.417: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-h4nvd started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:27.417: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:31:27.417: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-m6vq2 started at 2023-01-01 02:25:17 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:27.417: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:31:27.417: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-h6mmt started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:27.417: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:31:27.417: INFO: cilium-vbcgc started at 2023-01-01 02:20:26 +0000 UTC (1+1 container statuses recorded) Jan 1 02:31:27.417: INFO: Init container clean-cilium-state ready: true, restart count 0 Jan 1 02:31:27.417: INFO: Container cilium-agent ready: true, restart count 0 Jan 1 02:31:27.417: INFO: affinity-clusterip-timeout-q9qt2 started at 2023-01-01 02:27:32 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:27.417: INFO: Container affinity-clusterip-timeout ready: false, restart count 0 Jan 1 02:31:27.417: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-dkfpd started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:27.417: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:31:27.417: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-546bc started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:27.417: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:31:27.417: INFO: csi-mockplugin-attacher-0 started at 2023-01-01 02:27:56 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:27.417: INFO: Container csi-attacher ready: false, restart count 0 Jan 1 02:31:27.417: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-q8pgp started at 2023-01-01 02:25:17 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:27.417: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:31:27.417: INFO: csi-mockplugin-0 started at 2023-01-01 02:27:56 +0000 UTC (0+3 container statuses recorded) Jan 1 02:31:27.417: INFO: Container csi-provisioner ready: false, restart count 0 Jan 1 02:31:27.417: INFO: Container driver-registrar ready: false, restart count 0 Jan 1 02:31:27.417: INFO: Container mock ready: false, restart count 0 Jan 1 02:31:27.417: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-2v87j started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:27.417: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:31:27.869: INFO: Latency metrics for node ip-172-20-33-66.eu-west-3.compute.internal Jan 1 02:31:27.870: INFO: Logging node info for node ip-172-20-38-238.eu-west-3.compute.internal Jan 1 02:31:27.975: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-38-238.eu-west-3.compute.internal 0eb713b4-64bf-47b7-8090-bab8111218ee 9308 0 2023-01-01 02:18:51 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:c5.large beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:eu-west-3 failure-domain.beta.kubernetes.io/zone:eu-west-3a kops.k8s.io/kops-controller-pki: kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-38-238.eu-west-3.compute.internal kubernetes.io/os:linux kubernetes.io/role:master node-role.kubernetes.io/control-plane: node-role.kubernetes.io/master: node.kubernetes.io/exclude-from-external-load-balancers: node.kubernetes.io/instance-type:c5.large topology.ebs.csi.aws.com/zone:eu-west-3a topology.kubernetes.io/region:eu-west-3 topology.kubernetes.io/zone:eu-west-3a] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-0f70ebb30a9076648"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kubelet Update v1 2023-01-01 02:18:50 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {protokube Update v1 2023-01-01 02:19:16 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kops.k8s.io/kops-controller-pki":{},"f:node-role.kubernetes.io/control-plane":{},"f:node.kubernetes.io/exclude-from-external-load-balancers":{}}}} } {kops-controller Update v1 2023-01-01 02:19:21 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/master":{}}}} } {kube-controller-manager Update v1 2023-01-01 02:19:31 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.0.0/24\"":{}},"f:taints":{}}} } {kubelet Update v1 2023-01-01 02:20:36 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.0.0/24,DoNotUseExternalID:,ProviderID:aws:///eu-west-3a/i-0f70ebb30a9076648,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/master,Value:,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[100.96.0.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{50478186496 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3918811136 0} {<nil>} 3826964Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{45430367772 0} {<nil>} 45430367772 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3813953536 0} {<nil>} 3724564Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-01-01 02:31:26 +0000 UTC,LastTransitionTime:2023-01-01 02:18:45 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-01-01 02:31:26 +0000 UTC,LastTransitionTime:2023-01-01 02:18:45 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-01-01 02:31:26 +0000 UTC,LastTransitionTime:2023-01-01 02:18:45 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-01-01 02:31:26 +0000 UTC,LastTransitionTime:2023-01-01 02:19:31 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.38.238,},NodeAddress{Type:ExternalIP,Address:15.237.26.58,},NodeAddress{Type:Hostname,Address:ip-172-20-38-238.eu-west-3.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-38-238.eu-west-3.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-15-237-26-58.eu-west-3.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2e6c64b5b811bf0c78b2aa7ccdb943,SystemUUID:ec2e6c64-b5b8-11bf-0c78-b2aa7ccdb943,BootID:9db01fa3-d136-4230-b21a-b03abbfb6bff,KernelVersion:4.19.0-23-cloud-amd64,OSImage:Debian GNU/Linux 10 (buster),ContainerRuntimeVersion:docker://20.10.17,KubeletVersion:v1.23.15,KubeProxyVersion:v1.23.15,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[registry.k8s.io/etcdadm/etcd-manager@sha256:66a453db625abb268f4b3bbefc5a34a171d81e6e8796cecca54cfd71775c77c4 registry.k8s.io/etcdadm/etcd-manager:v3.0.20221209],SizeBytes:662209237,},ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:456569528,},ContainerImage{Names:[k8s.gcr.io/kube-apiserver-amd64:v1.23.15 registry.k8s.io/kube-apiserver-amd64:v1.23.15],SizeBytes:135240444,},ContainerImage{Names:[k8s.gcr.io/kube-controller-manager-amd64:v1.23.15 registry.k8s.io/kube-controller-manager-amd64:v1.23.15],SizeBytes:125046097,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.23.15 registry.k8s.io/kube-proxy-amd64:v1.23.15],SizeBytes:112336036,},ContainerImage{Names:[quay.io/cilium/operator@sha256:a6d24a006a6b92967ac90786b49bc1ac26e5477cf028cd1186efcfc2466484db quay.io/cilium/operator:v1.12.5],SizeBytes:110016956,},ContainerImage{Names:[registry.k8s.io/kops/kops-controller:1.27.0-alpha.1],SizeBytes:105197757,},ContainerImage{Names:[registry.k8s.io/kops/dns-controller:1.27.0-alpha.1],SizeBytes:102577531,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:10e231cf07c82dfab5141c1afe548dc734e5fc1f67665ec3982a325d1bd31a9a registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.0],SizeBytes:94471559,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:122bfb8c1edabb3c0edd63f06523e6940d958d19b3957dc7b1d6f81e9f1f6119 registry.k8s.io/sig-storage/csi-provisioner:v3.1.0],SizeBytes:57736070,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:9ebbf9f023e7b41ccee3d52afe39a89e3ddacdbb69269d583abfc25847cfd9e4 registry.k8s.io/sig-storage/csi-resizer:v1.4.0],SizeBytes:55464778,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:8b9c313c05f54fb04f8d430896f5f5904b6cb157df261501b29adc04d2b2dc7b registry.k8s.io/sig-storage/csi-attacher:v3.4.0],SizeBytes:54848595,},ContainerImage{Names:[k8s.gcr.io/kube-scheduler-amd64:v1.23.15 registry.k8s.io/kube-scheduler-amd64:v1.23.15],SizeBytes:53521430,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:19582112,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:17611980,},ContainerImage{Names:[registry.k8s.io/kops/kube-apiserver-healthcheck:1.27.0-alpha.1],SizeBytes:8787111,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:682696,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Jan 1 02:31:27.975: INFO: Logging kubelet events for node ip-172-20-38-238.eu-west-3.compute.internal Jan 1 02:31:28.082: INFO: Logging pods the kubelet thinks is on node ip-172-20-38-238.eu-west-3.compute.internal Jan 1 02:31:28.197: INFO: ebs-csi-node-k87k6 started at 2023-01-01 02:19:20 +0000 UTC (0+3 container statuses recorded) Jan 1 02:31:28.197: INFO: Container ebs-plugin ready: true, restart count 0 Jan 1 02:31:28.197: INFO: Container liveness-probe ready: true, restart count 0 Jan 1 02:31:28.197: INFO: Container node-driver-registrar ready: true, restart count 0 Jan 1 02:31:28.197: INFO: dns-controller-7db8b74b99-58zzw started at 2023-01-01 02:19:20 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:28.197: INFO: Container dns-controller ready: true, restart count 0 Jan 1 02:31:28.197: INFO: cilium-operator-d47487c76-j84nq started at 2023-01-01 02:19:20 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:28.197: INFO: Container cilium-operator ready: true, restart count 0 Jan 1 02:31:28.197: INFO: etcd-manager-events-ip-172-20-38-238.eu-west-3.compute.internal started at 2023-01-01 02:18:11 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:28.197: INFO: Container etcd-manager ready: true, restart count 0 Jan 1 02:31:28.197: INFO: etcd-manager-main-ip-172-20-38-238.eu-west-3.compute.internal started at 2023-01-01 02:18:11 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:28.197: INFO: Container etcd-manager ready: true, restart count 0 Jan 1 02:31:28.197: INFO: cilium-qk59v started at 2023-01-01 02:19:20 +0000 UTC (1+1 container statuses recorded) Jan 1 02:31:28.197: INFO: Init container clean-cilium-state ready: true, restart count 0 Jan 1 02:31:28.197: INFO: Container cilium-agent ready: true, restart count 0 Jan 1 02:31:28.197: INFO: kops-controller-26gzg started at 2023-01-01 02:19:20 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:28.197: INFO: Container kops-controller ready: true, restart count 0 Jan 1 02:31:28.197: INFO: ebs-csi-controller-6dc545b75-5qqh8 started at 2023-01-01 02:19:20 +0000 UTC (0+5 container statuses recorded) Jan 1 02:31:28.197: INFO: Container csi-attacher ready: true, restart count 0 Jan 1 02:31:28.197: INFO: Container csi-provisioner ready: true, restart count 0 Jan 1 02:31:28.197: INFO: Container csi-resizer ready: true, restart count 0 Jan 1 02:31:28.197: INFO: Container ebs-plugin ready: true, restart count 0 Jan 1 02:31:28.197: INFO: Container liveness-probe ready: true, restart count 0 Jan 1 02:31:28.197: INFO: kube-controller-manager-ip-172-20-38-238.eu-west-3.compute.internal started at 2023-01-01 02:18:11 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:28.197: INFO: Container kube-controller-manager ready: true, restart count 2 Jan 1 02:31:28.197: INFO: kube-scheduler-ip-172-20-38-238.eu-west-3.compute.internal started at 2023-01-01 02:18:11 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:28.197: INFO: Container kube-scheduler ready: true, restart count 0 Jan 1 02:31:28.197: INFO: kube-apiserver-ip-172-20-38-238.eu-west-3.compute.internal started at 2023-01-01 02:18:11 +0000 UTC (0+2 container statuses recorded) Jan 1 02:31:28.197: INFO: Container healthcheck ready: true, restart count 0 Jan 1 02:31:28.197: INFO: Container kube-apiserver ready: true, restart count 1 Jan 1 02:31:28.546: INFO: Latency metrics for node ip-172-20-38-238.eu-west-3.compute.internal Jan 1 02:31:28.546: INFO: Logging node info for node ip-172-20-45-112.eu-west-3.compute.internal Jan 1 02:31:28.652: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-45-112.eu-west-3.compute.internal 48eed937-05bb-4d8a-bddd-aa94a2e6ab92 9109 0 2023-01-01 02:20:27 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:eu-west-3 failure-domain.beta.kubernetes.io/zone:eu-west-3a kubelet_cleanup:true kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-45-112.eu-west-3.compute.internal kubernetes.io/os:linux kubernetes.io/role:node node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:eu-west-3a topology.hostpath.csi/node:ip-172-20-45-112.eu-west-3.compute.internal topology.kubernetes.io/region:eu-west-3 topology.kubernetes.io/zone:eu-west-3a] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-0778adf2085fa7f34"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kube-controller-manager Update v1 2023-01-01 02:20:27 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.3.0/24\"":{}}}} } {kubelet Update v1 2023-01-01 02:20:27 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {kops-controller Update v1 2023-01-01 02:20:28 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}} } {e2e.test Update v1 2023-01-01 02:25:15 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubelet_cleanup":{}}}} } {kube-controller-manager Update v1 2023-01-01 02:30:49 +0000 UTC FieldsV1 {"f:status":{"f:volumesAttached":{}}} status} {kubelet Update v1 2023-01-01 02:30:52 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:capacity":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.3.0/24,DoNotUseExternalID:,ProviderID:aws:///eu-west-3a/i-0778adf2085fa7f34,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.3.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{50478186496 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4078190592 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{45430367772 0} {<nil>} 45430367772 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3973332992 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-01-01 02:31:12 +0000 UTC,LastTransitionTime:2023-01-01 02:20:27 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-01-01 02:31:12 +0000 UTC,LastTransitionTime:2023-01-01 02:20:27 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-01-01 02:31:12 +0000 UTC,LastTransitionTime:2023-01-01 02:20:27 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-01-01 02:31:12 +0000 UTC,LastTransitionTime:2023-01-01 02:20:48 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.45.112,},NodeAddress{Type:ExternalIP,Address:13.38.32.14,},NodeAddress{Type:Hostname,Address:ip-172-20-45-112.eu-west-3.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-45-112.eu-west-3.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-13-38-32-14.eu-west-3.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2dbdf846e8ff614f4555e5bf9dbbf4,SystemUUID:ec2dbdf8-46e8-ff61-4f45-55e5bf9dbbf4,BootID:10a054ae-621a-47ce-9566-b0048cc5101b,KernelVersion:4.19.0-23-cloud-amd64,OSImage:Debian GNU/Linux 10 (buster),ContainerRuntimeVersion:docker://20.10.17,KubeletVersion:v1.23.15,KubeProxyVersion:v1.23.15,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:456569528,},ContainerImage{Names:[nginx@sha256:0047b729188a15da49380d9506d65959cce6d40291ccfb4e039f5dc7efd33286 nginx:latest],SizeBytes:141812353,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:7e8bdd271312fd25fc5ff5a8f04727be84044eb3d7d8d03611972a6752e2e11e k8s.gcr.io/e2e-test-images/agnhost:2.39],SizeBytes:126872991,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:1b9d1b2f36cb2dbee1960e82a9344aeb11bd4c4c03abf5e1853e0559c23855e3 k8s.gcr.io/e2e-test-images/httpd:2.4.38-2],SizeBytes:123781643,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.23.15 registry.k8s.io/kube-proxy-amd64:v1.23.15],SizeBytes:112336036,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:10e231cf07c82dfab5141c1afe548dc734e5fc1f67665ec3982a325d1bd31a9a registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.0],SizeBytes:94471559,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:c8e03f60afa90a28e4bb6ec9a8d0fc36d89de4b7475cf2d613afa793ec969fe0 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.0],SizeBytes:56382329,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:4e74c0492bceddc598de1c90cc5bc14dcda94cb49fa9c5bad9d117c4834b5e08 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.1],SizeBytes:56379269,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:36c31f7e1f433c9634d24f876353e8646246d81a03c4e351202c2644daff1620 k8s.gcr.io/sig-storage/csi-resizer:v1.2.0],SizeBytes:53956308,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:ed98431376c9e944e19a465fe8ea944806714dd95416a0821096c78d66b579bd k8s.gcr.io/sig-storage/csi-snapshotter:v4.1.1],SizeBytes:53531990,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:c5be65d6679efabb969d9b019300d187437ae876f992c40911fd2892bbef3b36 k8s.gcr.io/sig-storage/csi-attacher:v3.2.0],SizeBytes:53500826,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:60ab9b3e6a030d3038c87c0d6bca2930f58d1d72823e6a4af09767dc83b696a2 k8s.gcr.io/sig-storage/csi-attacher:v3.2.1],SizeBytes:53499570,},ContainerImage{Names:[k8s.gcr.io/sig-storage/hostpathplugin@sha256:232fe80174d60d520d36043103853a1d7ab4b7f3782cf43e45034f04ccda58ce k8s.gcr.io/sig-storage/hostpathplugin:v1.7.1],SizeBytes:29977921,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:a61d309da54641db41fb8f35718f744e9f730d4d0384f8c4b186ddc9f06cbd5f k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.1.0],SizeBytes:19662887,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:19582112,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:2dee3fe5fe861bb66c3a4ac51114f3447a4cd35870e0f2e2b558c7a400d89589 k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.2.0],SizeBytes:18699436,},ContainerImage{Names:[k8s.gcr.io/sig-storage/mock-driver@sha256:a7b517f9e0f42ffade802eef9cefa271372386b85f55b702b493241e58459793 k8s.gcr.io/sig-storage/mock-driver:v4.1.0],SizeBytes:17680993,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:17611980,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:529be2c9770add0cdd0c989115222ea9fc1be430c11095eb9f6dafcf98a36e2b k8s.gcr.io/sig-storage/livenessprobe:v2.4.0],SizeBytes:17175488,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:13616070e3f29de4417eee434a8ef472221c9e51b3d037b5a6b46cef08eb7443 k8s.gcr.io/e2e-test-images/nginx:1.14-2],SizeBytes:16032814,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf k8s.gcr.io/e2e-test-images/busybox:1.29-2],SizeBytes:1154361,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:39e1e963e5310e9c313bad51523be012ede7b35bb9316517d19089a010356592 k8s.gcr.io/e2e-test-images/busybox:1.29-1],SizeBytes:1154361,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db k8s.gcr.io/pause:3.6 registry.k8s.io/pause:3.6],SizeBytes:682696,},},VolumesInUse:[kubernetes.io/csi/ebs.csi.aws.com^vol-03d4513454f149737 kubernetes.io/csi/ebs.csi.aws.com^vol-08601772399ba2131],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-03d4513454f149737,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-08601772399ba2131,DevicePath:,},},Config:nil,},} Jan 1 02:31:28.652: INFO: Logging kubelet events for node ip-172-20-45-112.eu-west-3.compute.internal Jan 1 02:31:28.759: INFO: Logging pods the kubelet thinks is on node ip-172-20-45-112.eu-west-3.compute.internal Jan 1 02:31:28.872: INFO: pod-1 started at 2023-01-01 02:31:26 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:28.873: INFO: Container donothing ready: false, restart count 0 Jan 1 02:31:28.873: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-jh8bc started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:28.873: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:31:28.873: INFO: cilium-slx98 started at 2023-01-01 02:20:28 +0000 UTC (1+1 container statuses recorded) Jan 1 02:31:28.873: INFO: Init container clean-cilium-state ready: true, restart count 0 Jan 1 02:31:28.873: INFO: Container cilium-agent ready: true, restart count 0 Jan 1 02:31:28.873: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-4lrj2 started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:28.873: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:31:28.873: INFO: pod-0 started at 2023-01-01 02:31:26 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:28.873: INFO: Container donothing ready: false, restart count 0 Jan 1 02:31:28.873: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-4mml9 started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:28.873: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:31:28.873: INFO: affinity-clusterip-timeout-lv5vx started at 2023-01-01 02:27:32 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:28.873: INFO: Container affinity-clusterip-timeout ready: true, restart count 0 Jan 1 02:31:28.873: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-srf5k started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:28.873: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:31:28.873: INFO: pod-d05d599f-1b25-4249-b9c7-c64a79abebd3 started at 2023-01-01 02:30:59 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:28.873: INFO: Container write-pod ready: false, restart count 0 Jan 1 02:31:28.873: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-x5cp6 started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:28.873: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:31:28.873: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-xnfzj started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:28.873: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:31:28.873: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-gph2d started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:28.873: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:31:28.873: INFO: netserver-1 started at 2023-01-01 02:25:27 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:28.873: INFO: Container webserver ready: true, restart count 0 Jan 1 02:31:28.873: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-rmdrr started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:28.873: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:31:28.873: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-kjg2m started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:28.873: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:31:28.873: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-59bdj started at 2023-01-01 02:25:17 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:28.873: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:31:28.873: INFO: ebs-csi-node-7clrr started at 2023-01-01 02:20:28 +0000 UTC (0+3 container statuses recorded) Jan 1 02:31:28.873: INFO: Container ebs-plugin ready: true, restart count 0 Jan 1 02:31:28.873: INFO: Container liveness-probe ready: true, restart count 0 Jan 1 02:31:28.873: INFO: Container node-driver-registrar ready: true, restart count 0 Jan 1 02:31:28.873: INFO: image-pull-tested8f64cd-03c5-4c0a-9ed6-c4e25c9622f4 started at 2023-01-01 02:31:23 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:28.873: INFO: Container image-pull-test ready: false, restart count 0 Jan 1 02:31:28.873: INFO: e2e-dns-utils started at 2023-01-01 02:31:03 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:28.873: INFO: Container agnhost-container ready: false, restart count 0 Jan 1 02:31:28.873: INFO: inline-volume-tester-fpr45 started at 2023-01-01 02:31:04 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:28.873: INFO: Container csi-volume-tester ready: false, restart count 0 Jan 1 02:31:29.288: INFO: Latency metrics for node ip-172-20-45-112.eu-west-3.compute.internal Jan 1 02:31:29.288: INFO: Logging node info for node ip-172-20-47-178.eu-west-3.compute.internal Jan 1 02:31:29.394: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-47-178.eu-west-3.compute.internal 835e5337-d2b9-45ad-bf0e-9d32514f71ba 8456 0 2023-01-01 02:20:25 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:eu-west-3 failure-domain.beta.kubernetes.io/zone:eu-west-3a kubelet_cleanup:true kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-47-178.eu-west-3.compute.internal kubernetes.io/os:linux kubernetes.io/role:node node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:eu-west-3a topology.kubernetes.io/region:eu-west-3 topology.kubernetes.io/zone:eu-west-3a] map[csi.volume.kubernetes.io/nodeid:{"csi-mock-csi-mock-volumes-5429":"csi-mock-csi-mock-volumes-5429","ebs.csi.aws.com":"i-00e1e60199f5a4d04"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kops-controller Update v1 2023-01-01 02:20:25 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}} } {kube-controller-manager Update v1 2023-01-01 02:20:25 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.2.0/24\"":{}}}} } {kubelet Update v1 2023-01-01 02:20:25 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {e2e.test Update v1 2023-01-01 02:25:15 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubelet_cleanup":{}}}} } {kubelet Update v1 2023-01-01 02:25:42 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:capacity":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.2.0/24,DoNotUseExternalID:,ProviderID:aws:///eu-west-3a/i-00e1e60199f5a4d04,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.2.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{50478186496 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4078190592 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{45430367772 0} {<nil>} 45430367772 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3973332992 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-01-01 02:30:48 +0000 UTC,LastTransitionTime:2023-01-01 02:20:25 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-01-01 02:30:48 +0000 UTC,LastTransitionTime:2023-01-01 02:20:25 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-01-01 02:30:48 +0000 UTC,LastTransitionTime:2023-01-01 02:20:25 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-01-01 02:30:48 +0000 UTC,LastTransitionTime:2023-01-01 02:20:46 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.47.178,},NodeAddress{Type:ExternalIP,Address:13.36.177.101,},NodeAddress{Type:Hostname,Address:ip-172-20-47-178.eu-west-3.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-47-178.eu-west-3.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-13-36-177-101.eu-west-3.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec243eda226c96d5573f3eaa3203df0d,SystemUUID:ec243eda-226c-96d5-573f-3eaa3203df0d,BootID:15dccaa7-d7c7-419d-b7de-76be5781a51a,KernelVersion:4.19.0-23-cloud-amd64,OSImage:Debian GNU/Linux 10 (buster),ContainerRuntimeVersion:docker://20.10.17,KubeletVersion:v1.23.15,KubeProxyVersion:v1.23.15,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:456569528,},ContainerImage{Names:[nginx@sha256:0047b729188a15da49380d9506d65959cce6d40291ccfb4e039f5dc7efd33286 nginx:latest],SizeBytes:141812353,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:7e8bdd271312fd25fc5ff5a8f04727be84044eb3d7d8d03611972a6752e2e11e k8s.gcr.io/e2e-test-images/agnhost:2.39],SizeBytes:126872991,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.23.15 registry.k8s.io/kube-proxy-amd64:v1.23.15],SizeBytes:112336036,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:10e231cf07c82dfab5141c1afe548dc734e5fc1f67665ec3982a325d1bd31a9a registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.0],SizeBytes:94471559,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:c8e03f60afa90a28e4bb6ec9a8d0fc36d89de4b7475cf2d613afa793ec969fe0 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.0],SizeBytes:56382329,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:c5be65d6679efabb969d9b019300d187437ae876f992c40911fd2892bbef3b36 k8s.gcr.io/sig-storage/csi-attacher:v3.2.0],SizeBytes:53500826,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:017727efcfeb7d053af68e51436ce8e65edbc6ca573720afb4f79c8594036955 registry.k8s.io/coredns/coredns:v1.10.0],SizeBytes:50249337,},ContainerImage{Names:[registry.k8s.io/cpa/cluster-proportional-autoscaler@sha256:68d396900aeaa072c1f27289485fdac29834045a6f3ffe369bf389d830ef572d registry.k8s.io/cpa/cluster-proportional-autoscaler:1.8.6],SizeBytes:47910609,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:a61d309da54641db41fb8f35718f744e9f730d4d0384f8c4b186ddc9f06cbd5f k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.1.0],SizeBytes:19662887,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:19582112,},ContainerImage{Names:[k8s.gcr.io/sig-storage/mock-driver@sha256:a7b517f9e0f42ffade802eef9cefa271372386b85f55b702b493241e58459793 k8s.gcr.io/sig-storage/mock-driver:v4.1.0],SizeBytes:17680993,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:17611980,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:13616070e3f29de4417eee434a8ef472221c9e51b3d037b5a6b46cef08eb7443 k8s.gcr.io/e2e-test-images/nginx:1.14-2],SizeBytes:16032814,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf k8s.gcr.io/e2e-test-images/busybox:1.29-2],SizeBytes:1154361,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db k8s.gcr.io/pause:3.6 registry.k8s.io/pause:3.6],SizeBytes:682696,},},VolumesInUse:[kubernetes.io/csi/csi-mock-csi-mock-volumes-5429^4],VolumesAttached:[]AttachedVolume{},Config:nil,},} Jan 1 02:31:29.394: INFO: Logging kubelet events for node ip-172-20-47-178.eu-west-3.compute.internal Jan 1 02:31:29.501: INFO: Logging pods the kubelet thinks is on node ip-172-20-47-178.eu-west-3.compute.internal Jan 1 02:31:29.617: INFO: cilium-kk2c2 started at 2023-01-01 02:20:26 +0000 UTC (1+1 container statuses recorded) Jan 1 02:31:29.617: INFO: Init container clean-cilium-state ready: true, restart count 0 Jan 1 02:31:29.617: INFO: Container cilium-agent ready: true, restart count 0 Jan 1 02:31:29.617: INFO: local-injector started at 2023-01-01 02:31:07 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:29.617: INFO: Container local-injector ready: false, restart count 0 Jan 1 02:31:29.617: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-6vhtn started at 2023-01-01 02:25:17 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:29.617: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: false, restart count 0 Jan 1 02:31:29.617: INFO: coredns-867df8f45c-wzntb started at 2023-01-01 02:20:46 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:29.617: INFO: Container coredns ready: true, restart count 0 Jan 1 02:31:29.617: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-wkclp started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:29.617: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:31:29.617: INFO: coredns-autoscaler-557ccb4c66-k4lzb started at 2023-01-01 02:20:46 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:29.617: INFO: Container autoscaler ready: true, restart count 0 Jan 1 02:31:29.617: INFO: csi-mockplugin-attacher-0 started at 2023-01-01 02:28:39 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:29.617: INFO: Container csi-attacher ready: false, restart count 0 Jan 1 02:31:29.617: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-pgkm2 started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:29.617: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:31:29.617: INFO: netserver-2 started at 2023-01-01 02:25:27 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:29.617: INFO: Container webserver ready: false, restart count 0 Jan 1 02:31:29.617: INFO: hostexec-ip-172-20-47-178.eu-west-3.compute.internal-k7k4h started at 2023-01-01 02:30:47 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:29.617: INFO: Container agnhost-container ready: true, restart count 0 Jan 1 02:31:29.617: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-hrgrq started at 2023-01-01 02:25:17 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:29.617: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:31:29.617: INFO: csi-hostpathplugin-0 started at 2023-01-01 02:31:02 +0000 UTC (0+7 container statuses recorded) Jan 1 02:31:29.617: INFO: Container csi-attacher ready: false, restart count 0 Jan 1 02:31:29.617: INFO: Container csi-provisioner ready: false, restart count 0 Jan 1 02:31:29.617: INFO: Container csi-resizer ready: false, restart count 0 Jan 1 02:31:29.617: INFO: Container csi-snapshotter ready: false, restart count 0 Jan 1 02:31:29.617: INFO: Container hostpath ready: false, restart count 0 Jan 1 02:31:29.617: INFO: Container liveness-probe ready: false, restart count 0 Jan 1 02:31:29.617: INFO: Container node-driver-registrar ready: false, restart count 0 Jan 1 02:31:29.617: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-grrsl started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:29.617: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:31:29.617: INFO: csi-hostpathplugin-0 started at 2023-01-01 02:30:56 +0000 UTC (0+7 container statuses recorded) Jan 1 02:31:29.617: INFO: Container csi-attacher ready: false, restart count 0 Jan 1 02:31:29.617: INFO: Container csi-provisioner ready: false, restart count 0 Jan 1 02:31:29.617: INFO: Container csi-resizer ready: false, restart count 0 Jan 1 02:31:29.617: INFO: Container csi-snapshotter ready: false, restart count 0 Jan 1 02:31:29.618: INFO: Container hostpath ready: false, restart count 0 Jan 1 02:31:29.618: INFO: Container liveness-probe ready: false, restart count 0 Jan 1 02:31:29.618: INFO: Container node-driver-registrar ready: false, restart count 0 Jan 1 02:31:29.618: INFO: csi-mockplugin-0 started at 2023-01-01 02:28:39 +0000 UTC (0+3 container statuses recorded) Jan 1 02:31:29.618: INFO: Container csi-provisioner ready: false, restart count 0 Jan 1 02:31:29.618: INFO: Container driver-registrar ready: false, restart count 0 Jan 1 02:31:29.618: INFO: Container mock ready: false, restart count 0 Jan 1 02:31:29.618: INFO: ebs-csi-node-vcprc started at 2023-01-01 02:20:26 +0000 UTC (0+3 container statuses recorded) Jan 1 02:31:29.618: INFO: Container ebs-plugin ready: true, restart count 0 Jan 1 02:31:29.618: INFO: Container liveness-probe ready: true, restart count 0 Jan 1 02:31:29.618: INFO: Container node-driver-registrar ready: true, restart count 0 Jan 1 02:31:29.618: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-49brr started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:29.618: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:31:29.618: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-z87gr started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:29.618: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:31:29.618: INFO: csi-mockplugin-attacher-0 started at 2023-01-01 02:25:20 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:29.618: INFO: Container csi-attacher ready: false, restart count 0 Jan 1 02:31:29.618: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-wjz96 started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:29.618: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:31:29.618: INFO: csi-mockplugin-0 started at 2023-01-01 02:25:20 +0000 UTC (0+3 container statuses recorded) Jan 1 02:31:29.618: INFO: Container csi-provisioner ready: true, restart count 0 Jan 1 02:31:29.618: INFO: Container driver-registrar ready: true, restart count 0 Jan 1 02:31:29.618: INFO: Container mock ready: true, restart count 0 Jan 1 02:31:29.618: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-w26gj started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:29.618: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:31:29.618: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-s4rqf started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:29.618: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:31:29.618: INFO: pvc-volume-tester-w4w4d started at 2023-01-01 02:25:38 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:29.618: INFO: Container volume-tester ready: false, restart count 0 Jan 1 02:31:29.974: INFO: Latency metrics for node ip-172-20-47-178.eu-west-3.compute.internal Jan 1 02:31:29.974: INFO: Logging node info for node ip-172-20-49-224.eu-west-3.compute.internal Jan 1 02:31:30.079: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-49-224.eu-west-3.compute.internal d7d57146-c54a-4155-94f8-0f322216bfcf 8317 0 2023-01-01 02:20:28 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:eu-west-3 failure-domain.beta.kubernetes.io/zone:eu-west-3a kubelet_cleanup:true kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-49-224.eu-west-3.compute.internal kubernetes.io/os:linux kubernetes.io/role:node node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:eu-west-3a topology.hostpath.csi/node:ip-172-20-49-224.eu-west-3.compute.internal topology.kubernetes.io/region:eu-west-3 topology.kubernetes.io/zone:eu-west-3a] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-089a107939b27c28a"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kops-controller Update v1 2023-01-01 02:20:28 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}} } {kube-controller-manager Update v1 2023-01-01 02:20:28 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.4.0/24\"":{}}}} } {kubelet Update v1 2023-01-01 02:20:28 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {e2e.test Update v1 2023-01-01 02:25:15 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubelet_cleanup":{}}}} } {kubelet Update v1 2023-01-01 02:25:35 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:capacity":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.4.0/24,DoNotUseExternalID:,ProviderID:aws:///eu-west-3a/i-089a107939b27c28a,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.4.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{50478186496 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4078206976 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{45430367772 0} {<nil>} 45430367772 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3973349376 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-01-01 02:30:42 +0000 UTC,LastTransitionTime:2023-01-01 02:20:28 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-01-01 02:30:42 +0000 UTC,LastTransitionTime:2023-01-01 02:20:28 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-01-01 02:30:42 +0000 UTC,LastTransitionTime:2023-01-01 02:20:28 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-01-01 02:30:42 +0000 UTC,LastTransitionTime:2023-01-01 02:20:48 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.49.224,},NodeAddress{Type:ExternalIP,Address:15.237.95.101,},NodeAddress{Type:Hostname,Address:ip-172-20-49-224.eu-west-3.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-49-224.eu-west-3.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-15-237-95-101.eu-west-3.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2d0a1bb33dd26f5a12c3896530131b,SystemUUID:ec2d0a1b-b33d-d26f-5a12-c3896530131b,BootID:9f64e91d-26f0-45aa-aec4-a54433e67034,KernelVersion:4.19.0-23-cloud-amd64,OSImage:Debian GNU/Linux 10 (buster),ContainerRuntimeVersion:docker://20.10.17,KubeletVersion:v1.23.15,KubeProxyVersion:v1.23.15,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:456569528,},ContainerImage{Names:[nginx@sha256:0047b729188a15da49380d9506d65959cce6d40291ccfb4e039f5dc7efd33286 nginx:latest],SizeBytes:141812353,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:7e8bdd271312fd25fc5ff5a8f04727be84044eb3d7d8d03611972a6752e2e11e k8s.gcr.io/e2e-test-images/agnhost:2.39],SizeBytes:126872991,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:1b9d1b2f36cb2dbee1960e82a9344aeb11bd4c4c03abf5e1853e0559c23855e3 k8s.gcr.io/e2e-test-images/httpd:2.4.38-2],SizeBytes:123781643,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.23.15 registry.k8s.io/kube-proxy-amd64:v1.23.15],SizeBytes:112336036,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:10e231cf07c82dfab5141c1afe548dc734e5fc1f67665ec3982a325d1bd31a9a registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.0],SizeBytes:94471559,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:4e74c0492bceddc598de1c90cc5bc14dcda94cb49fa9c5bad9d117c4834b5e08 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.1],SizeBytes:56379269,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:36c31f7e1f433c9634d24f876353e8646246d81a03c4e351202c2644daff1620 k8s.gcr.io/sig-storage/csi-resizer:v1.2.0],SizeBytes:53956308,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:ed98431376c9e944e19a465fe8ea944806714dd95416a0821096c78d66b579bd k8s.gcr.io/sig-storage/csi-snapshotter:v4.1.1],SizeBytes:53531990,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:60ab9b3e6a030d3038c87c0d6bca2930f58d1d72823e6a4af09767dc83b696a2 k8s.gcr.io/sig-storage/csi-attacher:v3.2.1],SizeBytes:53499570,},ContainerImage{Names:[k8s.gcr.io/sig-storage/hostpathplugin@sha256:232fe80174d60d520d36043103853a1d7ab4b7f3782cf43e45034f04ccda58ce k8s.gcr.io/sig-storage/hostpathplugin:v1.7.1],SizeBytes:29977921,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:19582112,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:2dee3fe5fe861bb66c3a4ac51114f3447a4cd35870e0f2e2b558c7a400d89589 k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.2.0],SizeBytes:18699436,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:17611980,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:529be2c9770add0cdd0c989115222ea9fc1be430c11095eb9f6dafcf98a36e2b k8s.gcr.io/sig-storage/livenessprobe:v2.4.0],SizeBytes:17175488,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:13616070e3f29de4417eee434a8ef472221c9e51b3d037b5a6b46cef08eb7443 k8s.gcr.io/e2e-test-images/nginx:1.14-2],SizeBytes:16032814,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf k8s.gcr.io/e2e-test-images/busybox:1.29-2],SizeBytes:1154361,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db k8s.gcr.io/pause:3.6 registry.k8s.io/pause:3.6],SizeBytes:682696,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Jan 1 02:31:30.080: INFO: Logging kubelet events for node ip-172-20-49-224.eu-west-3.compute.internal Jan 1 02:31:30.187: INFO: Logging pods the kubelet thinks is on node ip-172-20-49-224.eu-west-3.compute.internal Jan 1 02:31:30.302: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-tv984 started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:30.302: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:31:30.302: INFO: affinity-clusterip-timeout-pxjtk started at 2023-01-01 02:27:32 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:30.302: INFO: Container affinity-clusterip-timeout ready: true, restart count 0 Jan 1 02:31:30.302: INFO: annotationupdate3b35e9ec-7ad7-4bd9-8137-1bd2bf1a899a started at 2023-01-01 02:28:25 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:30.302: INFO: Container client-container ready: false, restart count 0 Jan 1 02:31:30.302: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-2gndb started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:30.302: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:31:30.302: INFO: csi-mockplugin-0 started at 2023-01-01 02:31:26 +0000 UTC (0+3 container statuses recorded) Jan 1 02:31:30.302: INFO: Container csi-provisioner ready: false, restart count 0 Jan 1 02:31:30.302: INFO: Container driver-registrar ready: false, restart count 0 Jan 1 02:31:30.302: INFO: Container mock ready: false, restart count 0 Jan 1 02:31:30.302: INFO: pod-3e35f616-b2af-42fe-8148-bfe7bf04221b started at 2023-01-01 02:31:05 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:30.302: INFO: Container write-pod ready: false, restart count 0 Jan 1 02:31:30.302: INFO: dns-test-a8277dcf-764d-4d97-a381-813a8d1f903c started at 2023-01-01 02:31:17 +0000 UTC (0+3 container statuses recorded) Jan 1 02:31:30.302: INFO: Container jessie-querier ready: false, restart count 0 Jan 1 02:31:30.302: INFO: Container querier ready: false, restart count 0 Jan 1 02:31:30.302: INFO: Container webserver ready: false, restart count 0 Jan 1 02:31:30.302: INFO: csi-mockplugin-attacher-0 started at 2023-01-01 02:31:26 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:30.302: INFO: Container csi-attacher ready: false, restart count 0 Jan 1 02:31:30.302: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-cz8qh started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:30.302: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:31:30.302: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-6n5j9 started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:30.302: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:31:30.302: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-h6dxb started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:30.302: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:31:30.302: INFO: test-ss-0 started at 2023-01-01 02:27:56 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:30.302: INFO: Container webserver ready: false, restart count 0 Jan 1 02:31:30.302: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-pgc9g started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:30.302: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:31:30.302: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-29cdj started at 2023-01-01 02:25:17 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:30.302: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: false, restart count 0 Jan 1 02:31:30.302: INFO: hostexec-ip-172-20-49-224.eu-west-3.compute.internal-hsfbk started at 2023-01-01 02:31:01 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:30.302: INFO: Container agnhost-container ready: true, restart count 0 Jan 1 02:31:30.302: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-lrzjq started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:30.302: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:31:30.302: INFO: netserver-3 started at 2023-01-01 02:25:27 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:30.302: INFO: Container webserver ready: true, restart count 0 Jan 1 02:31:30.302: INFO: echo-sourceip started at 2023-01-01 02:26:27 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:30.302: INFO: Container agnhost-container ready: false, restart count 0 Jan 1 02:31:30.302: INFO: pod-2 started at 2023-01-01 02:31:27 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:30.302: INFO: Container donothing ready: false, restart count 0 Jan 1 02:31:30.302: INFO: pod-service-account-nomountsa-mountspec started at 2023-01-01 02:29:39 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:30.302: INFO: Container token-test ready: false, restart count 0 Jan 1 02:31:30.302: INFO: e2e-configmap-dns-server-7551ef4f-8c37-4168-bf54-91644ec7fb8f started at 2023-01-01 02:29:40 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:30.302: INFO: Container agnhost-container ready: true, restart count 0 Jan 1 02:31:30.302: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-7z6bp started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:30.302: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:31:30.302: INFO: ebs-csi-node-tzntf started at 2023-01-01 02:20:29 +0000 UTC (0+3 container statuses recorded) Jan 1 02:31:30.302: INFO: Container ebs-plugin ready: true, restart count 0 Jan 1 02:31:30.302: INFO: Container liveness-probe ready: true, restart count 0 Jan 1 02:31:30.302: INFO: Container node-driver-registrar ready: true, restart count 0 Jan 1 02:31:30.302: INFO: pod-subpath-test-secret-87r5 started at 2023-01-01 02:30:51 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:30.302: INFO: Container test-container-subpath-secret-87r5 ready: false, restart count 0 Jan 1 02:31:30.302: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-4jf6d started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:30.302: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:31:30.302: INFO: dns-test-f6141192-e7ab-4530-bedd-b1f3b48a9237 started at 2023-01-01 02:30:56 +0000 UTC (0+3 container statuses recorded) Jan 1 02:31:30.302: INFO: Container jessie-querier ready: false, restart count 0 Jan 1 02:31:30.302: INFO: Container querier ready: false, restart count 0 Jan 1 02:31:30.302: INFO: Container webserver ready: false, restart count 0 Jan 1 02:31:30.302: INFO: cilium-78jll started at 2023-01-01 02:20:29 +0000 UTC (1+1 container statuses recorded) Jan 1 02:31:30.302: INFO: Init container clean-cilium-state ready: true, restart count 0 Jan 1 02:31:30.302: INFO: Container cilium-agent ready: true, restart count 0 Jan 1 02:31:30.725: INFO: Latency metrics for node ip-172-20-49-224.eu-west-3.compute.internal Jan 1 02:31:30.725: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready �[1mSTEP�[0m: Destroying namespace "kubectl-1532" for this suite.
Filter through log files
go run hack/e2e.go -v --test --test_args='--ginkgo.focus=Kubernetes\se2e\ssuite\s\[sig\-network\]\sDNS\sshould\ssupport\sconfigurable\spod\sresolv\.conf$'
/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/network/dns.go:458 Jan 1 02:36:03.503: failed to wait for pod e2e-dns-utils to be running Unexpected error: <*errors.errorString | 0xc00023a2b0>: { s: "timed out waiting for the condition", } timed out waiting for the condition occurred /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/network/dns.go:520from junit_09.xml
[BeforeEach] [sig-network] DNS /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:185 �[1mSTEP�[0m: Creating a kubernetes client Jan 1 02:29:39.825: INFO: >>> kubeConfig: /root/.kube/config �[1mSTEP�[0m: Building a namespace api object, basename dns �[1mSTEP�[0m: Waiting for a default service account to be provisioned in namespace �[1mSTEP�[0m: Waiting for kube-root-ca.crt to be provisioned in namespace [It] should support configurable pod resolv.conf /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/network/dns.go:458 �[1mSTEP�[0m: Preparing a test DNS service with injected DNS names... Jan 1 02:29:40.767: INFO: Created pod &Pod{ObjectMeta:{e2e-configmap-dns-server-7551ef4f-8c37-4168-bf54-91644ec7fb8f dns-5931 9bfc782f-6769-48d1-894d-6411828cbf4c 7975 0 2023-01-01 02:29:40 +0000 UTC <nil> <nil> map[] map[] [] [] [{e2e.test Update v1 2023-01-01 02:29:40 +0000 UTC FieldsV1 {"f:spec":{"f:containers":{"k:{\"name\":\"agnhost-container\"}":{".":{},"f:args":{},"f:command":{},"f:image":{},"f:imagePullPolicy":{},"f:name":{},"f:resources":{},"f:securityContext":{},"f:terminationMessagePath":{},"f:terminationMessagePolicy":{},"f:volumeMounts":{".":{},"k:{\"mountPath\":\"/etc/coredns\"}":{".":{},"f:mountPath":{},"f:name":{},"f:readOnly":{}}}}},"f:dnsPolicy":{},"f:enableServiceLinks":{},"f:restartPolicy":{},"f:schedulerName":{},"f:securityContext":{},"f:terminationGracePeriodSeconds":{},"f:volumes":{".":{},"k:{\"name\":\"coredns-config\"}":{".":{},"f:configMap":{".":{},"f:defaultMode":{},"f:name":{}},"f:name":{}}}}} }]},Spec:PodSpec{Volumes:[]Volume{Volume{Name:coredns-config,VolumeSource:VolumeSource{HostPath:nil,EmptyDir:nil,GCEPersistentDisk:nil,AWSElasticBlockStore:nil,GitRepo:nil,Secret:nil,NFS:nil,ISCSI:nil,Glusterfs:nil,PersistentVolumeClaim:nil,RBD:nil,FlexVolume:nil,Cinder:nil,CephFS:nil,Flocker:nil,DownwardAPI:nil,FC:nil,AzureFile:nil,ConfigMap:&ConfigMapVolumeSource{LocalObjectReference:LocalObjectReference{Name:e2e-coredns-configmap-vcgrk,},Items:[]KeyToPath{},DefaultMode:*420,Optional:nil,},VsphereVolume:nil,Quobyte:nil,AzureDisk:nil,PhotonPersistentDisk:nil,PortworxVolume:nil,ScaleIO:nil,Projected:nil,StorageOS:nil,CSI:nil,Ephemeral:nil,},},Volume{Name:kube-api-access-dzlmn,VolumeSource:VolumeSource{HostPath:nil,EmptyDir:nil,GCEPersistentDisk:nil,AWSElasticBlockStore:nil,GitRepo:nil,Secret:nil,NFS:nil,ISCSI:nil,Glusterfs:nil,PersistentVolumeClaim:nil,RBD:nil,FlexVolume:nil,Cinder:nil,CephFS:nil,Flocker:nil,DownwardAPI:nil,FC:nil,AzureFile:nil,ConfigMap:nil,VsphereVolume:nil,Quobyte:nil,AzureDisk:nil,PhotonPersistentDisk:nil,PortworxVolume:nil,ScaleIO:nil,Projected:&ProjectedVolumeSource{Sources:[]VolumeProjection{VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:nil,ServiceAccountToken:&ServiceAccountTokenProjection{Audience:,ExpirationSeconds:*3607,Path:token,},},VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:&ConfigMapProjection{LocalObjectReference:LocalObjectReference{Name:kube-root-ca.crt,},Items:[]KeyToPath{KeyToPath{Key:ca.crt,Path:ca.crt,Mode:nil,},},Optional:nil,},ServiceAccountToken:nil,},VolumeProjection{Secret:nil,DownwardAPI:&DownwardAPIProjection{Items:[]DownwardAPIVolumeFile{DownwardAPIVolumeFile{Path:namespace,FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,Mode:nil,},},},ConfigMap:nil,ServiceAccountToken:nil,},},DefaultMode:*420,},StorageOS:nil,CSI:nil,Ephemeral:nil,},},},Containers:[]Container{Container{Name:agnhost-container,Image:k8s.gcr.io/e2e-test-images/agnhost:2.39,Command:[/coredns],Args:[-conf /etc/coredns/Corefile],WorkingDir:,Ports:[]ContainerPort{},Env:[]EnvVar{},Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{},},VolumeMounts:[]VolumeMount{VolumeMount{Name:coredns-config,ReadOnly:true,MountPath:/etc/coredns,SubPath:,MountPropagation:nil,SubPathExpr:,},VolumeMount{Name:kube-api-access-dzlmn,ReadOnly:true,MountPath:/var/run/secrets/kubernetes.io/serviceaccount,SubPath:,MountPropagation:nil,SubPathExpr:,},},LivenessProbe:nil,ReadinessProbe:nil,Lifecycle:nil,TerminationMessagePath:/dev/termination-log,ImagePullPolicy:IfNotPresent,SecurityContext:&SecurityContext{Capabilities:nil,Privileged:nil,SELinuxOptions:nil,RunAsUser:nil,RunAsNonRoot:nil,ReadOnlyRootFilesystem:nil,AllowPrivilegeEscalation:nil,RunAsGroup:nil,ProcMount:nil,WindowsOptions:nil,SeccompProfile:nil,},Stdin:false,StdinOnce:false,TTY:false,EnvFrom:[]EnvFromSource{},TerminationMessagePolicy:File,VolumeDevices:[]VolumeDevice{},StartupProbe:nil,},},RestartPolicy:Always,TerminationGracePeriodSeconds:*0,ActiveDeadlineSeconds:nil,DNSPolicy:Default,NodeSelector:map[string]string{},ServiceAccountName:default,DeprecatedServiceAccount:default,NodeName:,HostNetwork:false,HostPID:false,HostIPC:false,SecurityContext:&PodSecurityContext{SELinuxOptions:nil,RunAsUser:nil,RunAsNonRoot:nil,SupplementalGroups:[],FSGroup:nil,RunAsGroup:nil,Sysctls:[]Sysctl{},WindowsOptions:nil,FSGroupChangePolicy:nil,SeccompProfile:nil,},ImagePullSecrets:[]LocalObjectReference{},Hostname:,Subdomain:,Affinity:nil,SchedulerName:default-scheduler,InitContainers:[]Container{},AutomountServiceAccountToken:nil,Tolerations:[]Toleration{Toleration{Key:node.kubernetes.io/not-ready,Operator:Exists,Value:,Effect:NoExecute,TolerationSeconds:*300,},Toleration{Key:node.kubernetes.io/unreachable,Operator:Exists,Value:,Effect:NoExecute,TolerationSeconds:*300,},},HostAliases:[]HostAlias{},PriorityClassName:,Priority:*0,DNSConfig:nil,ShareProcessNamespace:nil,ReadinessGates:[]PodReadinessGate{},RuntimeClassName:nil,EnableServiceLinks:*true,PreemptionPolicy:*PreemptLowerPriority,Overhead:ResourceList{},TopologySpreadConstraints:[]TopologySpreadConstraint{},EphemeralContainers:[]EphemeralContainer{},SetHostnameAsFQDN:nil,OS:nil,},Status:PodStatus{Phase:Pending,Conditions:[]PodCondition{},Message:,Reason:,HostIP:,PodIP:,StartTime:<nil>,ContainerStatuses:[]ContainerStatus{},QOSClass:BestEffort,InitContainerStatuses:[]ContainerStatus{},NominatedNodeName:,PodIPs:[]PodIP{},EphemeralContainerStatuses:[]ContainerStatus{},},} Jan 1 02:31:03.081: INFO: testServerIP is 172.20.42.169 �[1mSTEP�[0m: Creating a pod with dnsPolicy=None and customized dnsConfig... Jan 1 02:31:03.190: INFO: Created pod &Pod{ObjectMeta:{e2e-dns-utils dns-5931 de9effe6-c9c4-40b1-b1dc-ffacad9bd485 9000 0 2023-01-01 02:31:03 +0000 UTC <nil> <nil> map[] map[] [] [] [{e2e.test Update v1 2023-01-01 02:31:03 +0000 UTC FieldsV1 {"f:spec":{"f:containers":{"k:{\"name\":\"agnhost-container\"}":{".":{},"f:args":{},"f:image":{},"f:imagePullPolicy":{},"f:name":{},"f:resources":{},"f:securityContext":{},"f:terminationMessagePath":{},"f:terminationMessagePolicy":{}}},"f:dnsConfig":{".":{},"f:nameservers":{},"f:options":{},"f:searches":{}},"f:dnsPolicy":{},"f:enableServiceLinks":{},"f:restartPolicy":{},"f:schedulerName":{},"f:securityContext":{},"f:terminationGracePeriodSeconds":{}}} }]},Spec:PodSpec{Volumes:[]Volume{Volume{Name:kube-api-access-48zgx,VolumeSource:VolumeSource{HostPath:nil,EmptyDir:nil,GCEPersistentDisk:nil,AWSElasticBlockStore:nil,GitRepo:nil,Secret:nil,NFS:nil,ISCSI:nil,Glusterfs:nil,PersistentVolumeClaim:nil,RBD:nil,FlexVolume:nil,Cinder:nil,CephFS:nil,Flocker:nil,DownwardAPI:nil,FC:nil,AzureFile:nil,ConfigMap:nil,VsphereVolume:nil,Quobyte:nil,AzureDisk:nil,PhotonPersistentDisk:nil,PortworxVolume:nil,ScaleIO:nil,Projected:&ProjectedVolumeSource{Sources:[]VolumeProjection{VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:nil,ServiceAccountToken:&ServiceAccountTokenProjection{Audience:,ExpirationSeconds:*3607,Path:token,},},VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:&ConfigMapProjection{LocalObjectReference:LocalObjectReference{Name:kube-root-ca.crt,},Items:[]KeyToPath{KeyToPath{Key:ca.crt,Path:ca.crt,Mode:nil,},},Optional:nil,},ServiceAccountToken:nil,},VolumeProjection{Secret:nil,DownwardAPI:&DownwardAPIProjection{Items:[]DownwardAPIVolumeFile{DownwardAPIVolumeFile{Path:namespace,FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,Mode:nil,},},},ConfigMap:nil,ServiceAccountToken:nil,},},DefaultMode:*420,},StorageOS:nil,CSI:nil,Ephemeral:nil,},},},Containers:[]Container{Container{Name:agnhost-container,Image:k8s.gcr.io/e2e-test-images/agnhost:2.39,Command:[],Args:[pause],WorkingDir:,Ports:[]ContainerPort{},Env:[]EnvVar{},Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{},},VolumeMounts:[]VolumeMount{VolumeMount{Name:kube-api-access-48zgx,ReadOnly:true,MountPath:/var/run/secrets/kubernetes.io/serviceaccount,SubPath:,MountPropagation:nil,SubPathExpr:,},},LivenessProbe:nil,ReadinessProbe:nil,Lifecycle:nil,TerminationMessagePath:/dev/termination-log,ImagePullPolicy:IfNotPresent,SecurityContext:&SecurityContext{Capabilities:nil,Privileged:nil,SELinuxOptions:nil,RunAsUser:nil,RunAsNonRoot:nil,ReadOnlyRootFilesystem:nil,AllowPrivilegeEscalation:nil,RunAsGroup:nil,ProcMount:nil,WindowsOptions:nil,SeccompProfile:nil,},Stdin:false,StdinOnce:false,TTY:false,EnvFrom:[]EnvFromSource{},TerminationMessagePolicy:File,VolumeDevices:[]VolumeDevice{},StartupProbe:nil,},},RestartPolicy:Always,TerminationGracePeriodSeconds:*0,ActiveDeadlineSeconds:nil,DNSPolicy:None,NodeSelector:map[string]string{},ServiceAccountName:default,DeprecatedServiceAccount:default,NodeName:,HostNetwork:false,HostPID:false,HostIPC:false,SecurityContext:&PodSecurityContext{SELinuxOptions:nil,RunAsUser:nil,RunAsNonRoot:nil,SupplementalGroups:[],FSGroup:nil,RunAsGroup:nil,Sysctls:[]Sysctl{},WindowsOptions:nil,FSGroupChangePolicy:nil,SeccompProfile:nil,},ImagePullSecrets:[]LocalObjectReference{},Hostname:,Subdomain:,Affinity:nil,SchedulerName:default-scheduler,InitContainers:[]Container{},AutomountServiceAccountToken:nil,Tolerations:[]Toleration{Toleration{Key:node.kubernetes.io/not-ready,Operator:Exists,Value:,Effect:NoExecute,TolerationSeconds:*300,},Toleration{Key:node.kubernetes.io/unreachable,Operator:Exists,Value:,Effect:NoExecute,TolerationSeconds:*300,},},HostAliases:[]HostAlias{},PriorityClassName:,Priority:*0,DNSConfig:&PodDNSConfig{Nameservers:[172.20.42.169],Searches:[resolv.conf.local],Options:[]PodDNSConfigOption{PodDNSConfigOption{Name:ndots,Value:*2,},},},ShareProcessNamespace:nil,ReadinessGates:[]PodReadinessGate{},RuntimeClassName:nil,EnableServiceLinks:*true,PreemptionPolicy:*PreemptLowerPriority,Overhead:ResourceList{},TopologySpreadConstraints:[]TopologySpreadConstraint{},EphemeralContainers:[]EphemeralContainer{},SetHostnameAsFQDN:nil,OS:nil,},Status:PodStatus{Phase:Pending,Conditions:[]PodCondition{},Message:,Reason:,HostIP:,PodIP:,StartTime:<nil>,ContainerStatuses:[]ContainerStatus{},QOSClass:BestEffort,InitContainerStatuses:[]ContainerStatus{},NominatedNodeName:,PodIPs:[]PodIP{},EphemeralContainerStatuses:[]ContainerStatus{},},} Jan 1 02:36:03.503: FAIL: failed to wait for pod e2e-dns-utils to be running Unexpected error: <*errors.errorString | 0xc00023a2b0>: { s: "timed out waiting for the condition", } timed out waiting for the condition occurred Full Stack Trace k8s.io/kubernetes/test/e2e/network.glob..func2.11() /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/network/dns.go:520 +0xc67 k8s.io/kubernetes/test/e2e.RunE2ETests(0x24c66d7) _output/dockerized/go/src/k8s.io/kubernetes/test/e2e/e2e.go:133 +0x697 k8s.io/kubernetes/test/e2e.TestE2E(0xc) _output/dockerized/go/src/k8s.io/kubernetes/test/e2e/e2e_test.go:136 +0x19 testing.tRunner(0xc0004a31e0, 0x735e880) /usr/local/go/src/testing/testing.go:1259 +0x102 created by testing.(*T).Run /usr/local/go/src/testing/testing.go:1306 +0x35a Jan 1 02:36:03.504: INFO: Deleting pod e2e-dns-utils... Jan 1 02:36:03.615: INFO: Deleting pod e2e-configmap-dns-server-7551ef4f-8c37-4168-bf54-91644ec7fb8f... Jan 1 02:36:03.730: INFO: Deleting configmap e2e-coredns-configmap-vcgrk... [AfterEach] [sig-network] DNS /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:186 �[1mSTEP�[0m: Collecting events from namespace "dns-5931". �[1mSTEP�[0m: Found 25 events. Jan 1 02:36:03.940: INFO: At 2023-01-01 02:29:40 +0000 UTC - event for e2e-configmap-dns-server-7551ef4f-8c37-4168-bf54-91644ec7fb8f: {default-scheduler } Scheduled: Successfully assigned dns-5931/e2e-configmap-dns-server-7551ef4f-8c37-4168-bf54-91644ec7fb8f to ip-172-20-49-224.eu-west-3.compute.internal Jan 1 02:36:03.940: INFO: At 2023-01-01 02:29:43 +0000 UTC - event for e2e-configmap-dns-server-7551ef4f-8c37-4168-bf54-91644ec7fb8f: {kubelet ip-172-20-49-224.eu-west-3.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "c8f3caa6758053615e58cdf9b5ec34a369701c6b978c9bde83ebc5d37e700117" network for pod "e2e-configmap-dns-server-7551ef4f-8c37-4168-bf54-91644ec7fb8f": networkPlugin cni failed to set up pod "e2e-configmap-dns-server-7551ef4f-8c37-4168-bf54-91644ec7fb8f_dns-5931" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 1 02:36:03.940: INFO: At 2023-01-01 02:29:44 +0000 UTC - event for e2e-configmap-dns-server-7551ef4f-8c37-4168-bf54-91644ec7fb8f: {kubelet ip-172-20-49-224.eu-west-3.compute.internal} SandboxChanged: Pod sandbox changed, it will be killed and re-created. Jan 1 02:36:03.940: INFO: At 2023-01-01 02:29:46 +0000 UTC - event for e2e-configmap-dns-server-7551ef4f-8c37-4168-bf54-91644ec7fb8f: {kubelet ip-172-20-49-224.eu-west-3.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "e349c82ade483f58fa9444fe45c06f3a4b0c063b398468ff216278d733bc4ba2" network for pod "e2e-configmap-dns-server-7551ef4f-8c37-4168-bf54-91644ec7fb8f": networkPlugin cni failed to set up pod "e2e-configmap-dns-server-7551ef4f-8c37-4168-bf54-91644ec7fb8f_dns-5931" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 1 02:36:03.940: INFO: At 2023-01-01 02:29:50 +0000 UTC - event for e2e-configmap-dns-server-7551ef4f-8c37-4168-bf54-91644ec7fb8f: {kubelet ip-172-20-49-224.eu-west-3.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "3711a857c2134e4302dbf202a92200e3a875c2c41b6cd2cb08d13969d9b1f560" network for pod "e2e-configmap-dns-server-7551ef4f-8c37-4168-bf54-91644ec7fb8f": networkPlugin cni failed to set up pod "e2e-configmap-dns-server-7551ef4f-8c37-4168-bf54-91644ec7fb8f_dns-5931" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 1 02:36:03.940: INFO: At 2023-01-01 02:29:53 +0000 UTC - event for e2e-configmap-dns-server-7551ef4f-8c37-4168-bf54-91644ec7fb8f: {kubelet ip-172-20-49-224.eu-west-3.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "723486b1066aa3f7055e7bf5b7bbb98f6cc44e96bd361b0b922178c1826fc3bf" network for pod "e2e-configmap-dns-server-7551ef4f-8c37-4168-bf54-91644ec7fb8f": networkPlugin cni failed to set up pod "e2e-configmap-dns-server-7551ef4f-8c37-4168-bf54-91644ec7fb8f_dns-5931" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 1 02:36:03.940: INFO: At 2023-01-01 02:29:56 +0000 UTC - event for e2e-configmap-dns-server-7551ef4f-8c37-4168-bf54-91644ec7fb8f: {kubelet ip-172-20-49-224.eu-west-3.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "bffb574aeb54d8180a9b2b4218e519fd61f96946ced76c6fee55dc48162a1fc0" network for pod "e2e-configmap-dns-server-7551ef4f-8c37-4168-bf54-91644ec7fb8f": networkPlugin cni failed to set up pod "e2e-configmap-dns-server-7551ef4f-8c37-4168-bf54-91644ec7fb8f_dns-5931" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 1 02:36:03.940: INFO: At 2023-01-01 02:29:58 +0000 UTC - event for e2e-configmap-dns-server-7551ef4f-8c37-4168-bf54-91644ec7fb8f: {kubelet ip-172-20-49-224.eu-west-3.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "d13552119c6d1cc3f79dea10c82e568ac04f0a315d824b5f9df987a7e2f89f9a" network for pod "e2e-configmap-dns-server-7551ef4f-8c37-4168-bf54-91644ec7fb8f": networkPlugin cni failed to set up pod "e2e-configmap-dns-server-7551ef4f-8c37-4168-bf54-91644ec7fb8f_dns-5931" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 1 02:36:03.940: INFO: At 2023-01-01 02:30:00 +0000 UTC - event for e2e-configmap-dns-server-7551ef4f-8c37-4168-bf54-91644ec7fb8f: {kubelet ip-172-20-49-224.eu-west-3.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "f83cb874e7fc2b0f3e0eed728b09a00794996e65e780ffb77cdea0bbfe72ccbb" network for pod "e2e-configmap-dns-server-7551ef4f-8c37-4168-bf54-91644ec7fb8f": networkPlugin cni failed to set up pod "e2e-configmap-dns-server-7551ef4f-8c37-4168-bf54-91644ec7fb8f_dns-5931" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 1 02:36:03.940: INFO: At 2023-01-01 02:30:01 +0000 UTC - event for e2e-configmap-dns-server-7551ef4f-8c37-4168-bf54-91644ec7fb8f: {kubelet ip-172-20-49-224.eu-west-3.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "3499d986be243f5c33a6b95e85d170a82734acbcda46a44aee4c37af07943128" network for pod "e2e-configmap-dns-server-7551ef4f-8c37-4168-bf54-91644ec7fb8f": networkPlugin cni failed to set up pod "e2e-configmap-dns-server-7551ef4f-8c37-4168-bf54-91644ec7fb8f_dns-5931" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 1 02:36:03.940: INFO: At 2023-01-01 02:30:03 +0000 UTC - event for e2e-configmap-dns-server-7551ef4f-8c37-4168-bf54-91644ec7fb8f: {kubelet ip-172-20-49-224.eu-west-3.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "59dc500c687df8e3a6a9a3251be8590e99fe4f16dee18ebfa0f53dd6f1d8b869" network for pod "e2e-configmap-dns-server-7551ef4f-8c37-4168-bf54-91644ec7fb8f": networkPlugin cni failed to set up pod "e2e-configmap-dns-server-7551ef4f-8c37-4168-bf54-91644ec7fb8f_dns-5931" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 1 02:36:03.940: INFO: At 2023-01-01 02:30:05 +0000 UTC - event for e2e-configmap-dns-server-7551ef4f-8c37-4168-bf54-91644ec7fb8f: {kubelet ip-172-20-49-224.eu-west-3.compute.internal} FailedCreatePodSandBox: (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "970d453884e183ad6aaa2d0f763a5aa3d0d19fbdb62365db41a4cc7601bbe7da" network for pod "e2e-configmap-dns-server-7551ef4f-8c37-4168-bf54-91644ec7fb8f": networkPlugin cni failed to set up pod "e2e-configmap-dns-server-7551ef4f-8c37-4168-bf54-91644ec7fb8f_dns-5931" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 1 02:36:03.940: INFO: At 2023-01-01 02:31:03 +0000 UTC - event for e2e-dns-utils: {kubelet ip-172-20-45-112.eu-west-3.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "0fd6f2cb0718b66a8fab21ef10ea2f0efa6c2628ebed0b670f63f4f09320b785" network for pod "e2e-dns-utils": networkPlugin cni failed to set up pod "e2e-dns-utils_dns-5931" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 1 02:36:03.940: INFO: At 2023-01-01 02:31:03 +0000 UTC - event for e2e-dns-utils: {default-scheduler } Scheduled: Successfully assigned dns-5931/e2e-dns-utils to ip-172-20-45-112.eu-west-3.compute.internal Jan 1 02:36:03.940: INFO: At 2023-01-01 02:31:04 +0000 UTC - event for e2e-dns-utils: {kubelet ip-172-20-45-112.eu-west-3.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "0722dbeabf7b40fc779e9d05da7d951d94baed886e4227208d842cb3a6744f84" network for pod "e2e-dns-utils": networkPlugin cni failed to set up pod "e2e-dns-utils_dns-5931" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 1 02:36:03.940: INFO: At 2023-01-01 02:31:04 +0000 UTC - event for e2e-dns-utils: {kubelet ip-172-20-45-112.eu-west-3.compute.internal} SandboxChanged: Pod sandbox changed, it will be killed and re-created. Jan 1 02:36:03.940: INFO: At 2023-01-01 02:31:05 +0000 UTC - event for e2e-dns-utils: {kubelet ip-172-20-45-112.eu-west-3.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "af9a33ff2e30a79c525a0fcfe9139e4fdc566b358f299d7cc6e5716d3b878d24" network for pod "e2e-dns-utils": networkPlugin cni failed to set up pod "e2e-dns-utils_dns-5931" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 1 02:36:03.940: INFO: At 2023-01-01 02:31:06 +0000 UTC - event for e2e-dns-utils: {kubelet ip-172-20-45-112.eu-west-3.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "28e882fa018acb2b60f9524ec3ed8ad4f76bbffff3114ce36d640719ec7ad390" network for pod "e2e-dns-utils": networkPlugin cni failed to set up pod "e2e-dns-utils_dns-5931" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 1 02:36:03.940: INFO: At 2023-01-01 02:31:07 +0000 UTC - event for e2e-dns-utils: {kubelet ip-172-20-45-112.eu-west-3.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "653e82bd7e396aad81e10f1a9837c19ce4876c98bb58af63b79b4b9c9223832c" network for pod "e2e-dns-utils": networkPlugin cni failed to set up pod "e2e-dns-utils_dns-5931" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 1 02:36:03.941: INFO: At 2023-01-01 02:31:08 +0000 UTC - event for e2e-dns-utils: {kubelet ip-172-20-45-112.eu-west-3.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "2efa125db6509e88d23e18310691b345f35d5e34497def79880605b8799e9698" network for pod "e2e-dns-utils": networkPlugin cni failed to set up pod "e2e-dns-utils_dns-5931" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 1 02:36:03.941: INFO: At 2023-01-01 02:31:09 +0000 UTC - event for e2e-dns-utils: {kubelet ip-172-20-45-112.eu-west-3.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "21fda78dc76813988eea3cadf1aa8e06320c7b9faa3c4f79a48182795c85549a" network for pod "e2e-dns-utils": networkPlugin cni failed to set up pod "e2e-dns-utils_dns-5931" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 1 02:36:03.941: INFO: At 2023-01-01 02:31:10 +0000 UTC - event for e2e-dns-utils: {kubelet ip-172-20-45-112.eu-west-3.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "ad49df951b1ba2065e891bf7e8bac0817b529b9776d6272b4b6219158aa71477" network for pod "e2e-dns-utils": networkPlugin cni failed to set up pod "e2e-dns-utils_dns-5931" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 1 02:36:03.941: INFO: At 2023-01-01 02:31:11 +0000 UTC - event for e2e-dns-utils: {kubelet ip-172-20-45-112.eu-west-3.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "5628ab16848c17e64a7de084e783b9eaff4a14c2e370d48cc8609ba613777a9b" network for pod "e2e-dns-utils": networkPlugin cni failed to set up pod "e2e-dns-utils_dns-5931" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 1 02:36:03.941: INFO: At 2023-01-01 02:31:12 +0000 UTC - event for e2e-dns-utils: {kubelet ip-172-20-45-112.eu-west-3.compute.internal} FailedCreatePodSandBox: (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "008ca7e801808d42ce30e5519b60873b87c7874e44a8fc8fc691fd0544ea2bb7" network for pod "e2e-dns-utils": networkPlugin cni failed to set up pod "e2e-dns-utils_dns-5931" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 1 02:36:03.941: INFO: At 2023-01-01 02:36:03 +0000 UTC - event for e2e-configmap-dns-server-7551ef4f-8c37-4168-bf54-91644ec7fb8f: {kubelet ip-172-20-49-224.eu-west-3.compute.internal} Killing: Stopping container agnhost-container Jan 1 02:36:04.044: INFO: POD NODE PHASE GRACE CONDITIONS Jan 1 02:36:04.044: INFO: Jan 1 02:36:04.150: INFO: Logging node info for node ip-172-20-33-66.eu-west-3.compute.internal Jan 1 02:36:04.254: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-33-66.eu-west-3.compute.internal ec4b59b3-62db-4040-aad2-4c3249f80fe8 10705 0 2023-01-01 02:20:25 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:eu-west-3 failure-domain.beta.kubernetes.io/zone:eu-west-3a io.kubernetes.storage.mock/node:some-mock-node kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-33-66.eu-west-3.compute.internal kubernetes.io/os:linux kubernetes.io/role:node node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:eu-west-3a topology.kubernetes.io/region:eu-west-3 topology.kubernetes.io/zone:eu-west-3a] map[csi.volume.kubernetes.io/nodeid:{"csi-mock-csi-mock-volumes-2206":"csi-mock-csi-mock-volumes-2206","csi-mock-csi-mock-volumes-8129":"csi-mock-csi-mock-volumes-8129","ebs.csi.aws.com":"i-028313673124ac4ad"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kops-controller Update v1 2023-01-01 02:20:25 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}} } {kube-controller-manager Update v1 2023-01-01 02:20:25 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.1.0/24\"":{}}}} } {kubelet Update v1 2023-01-01 02:20:25 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {kubelet Update v1 2023-01-01 02:31:50 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:io.kubernetes.storage.mock/node":{},"f:topology.ebs.csi.aws.com/zone":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.1.0/24,DoNotUseExternalID:,ProviderID:aws:///eu-west-3a/i-028313673124ac4ad,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.1.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{50478186496 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4078190592 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{45430367772 0} {<nil>} 45430367772 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3973332992 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-01-01 02:31:50 +0000 UTC,LastTransitionTime:2023-01-01 02:20:25 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-01-01 02:31:50 +0000 UTC,LastTransitionTime:2023-01-01 02:20:25 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-01-01 02:31:50 +0000 UTC,LastTransitionTime:2023-01-01 02:20:25 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-01-01 02:31:50 +0000 UTC,LastTransitionTime:2023-01-01 02:20:46 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.33.66,},NodeAddress{Type:ExternalIP,Address:52.47.140.112,},NodeAddress{Type:Hostname,Address:ip-172-20-33-66.eu-west-3.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-33-66.eu-west-3.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-52-47-140-112.eu-west-3.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2ad2ce1a87d508c178acf6215c5767,SystemUUID:ec2ad2ce-1a87-d508-c178-acf6215c5767,BootID:b3b60783-d786-412b-a36a-802bcfae604d,KernelVersion:4.19.0-23-cloud-amd64,OSImage:Debian GNU/Linux 10 (buster),ContainerRuntimeVersion:docker://20.10.17,KubeletVersion:v1.23.15,KubeProxyVersion:v1.23.15,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:456569528,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/jessie-dnsutils@sha256:11e6a66017ba4e4b938c1612b7a54a3befcefd354796c04e1dba76873a13518e k8s.gcr.io/e2e-test-images/jessie-dnsutils:1.5],SizeBytes:253371792,},ContainerImage{Names:[nginx@sha256:0047b729188a15da49380d9506d65959cce6d40291ccfb4e039f5dc7efd33286 nginx:latest],SizeBytes:141812353,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:7e8bdd271312fd25fc5ff5a8f04727be84044eb3d7d8d03611972a6752e2e11e k8s.gcr.io/e2e-test-images/agnhost:2.39],SizeBytes:126872991,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:1b9d1b2f36cb2dbee1960e82a9344aeb11bd4c4c03abf5e1853e0559c23855e3 k8s.gcr.io/e2e-test-images/httpd:2.4.38-2],SizeBytes:123781643,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.23.15 registry.k8s.io/kube-proxy-amd64:v1.23.15],SizeBytes:112336036,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:10e231cf07c82dfab5141c1afe548dc734e5fc1f67665ec3982a325d1bd31a9a registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.0],SizeBytes:94471559,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:c8e03f60afa90a28e4bb6ec9a8d0fc36d89de4b7475cf2d613afa793ec969fe0 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.0],SizeBytes:56382329,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:c5be65d6679efabb969d9b019300d187437ae876f992c40911fd2892bbef3b36 k8s.gcr.io/sig-storage/csi-attacher:v3.2.0],SizeBytes:53500826,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:017727efcfeb7d053af68e51436ce8e65edbc6ca573720afb4f79c8594036955 registry.k8s.io/coredns/coredns:v1.10.0],SizeBytes:50249337,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:a61d309da54641db41fb8f35718f744e9f730d4d0384f8c4b186ddc9f06cbd5f k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.1.0],SizeBytes:19662887,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:19582112,},ContainerImage{Names:[k8s.gcr.io/sig-storage/mock-driver@sha256:a7b517f9e0f42ffade802eef9cefa271372386b85f55b702b493241e58459793 k8s.gcr.io/sig-storage/mock-driver:v4.1.0],SizeBytes:17680993,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:17611980,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:13616070e3f29de4417eee434a8ef472221c9e51b3d037b5a6b46cef08eb7443 k8s.gcr.io/e2e-test-images/nginx:1.14-2],SizeBytes:16032814,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nonewprivs@sha256:8ac1264691820febacf3aea5d152cbde6d10685731ec14966a9401c6f47a68ac k8s.gcr.io/e2e-test-images/nonewprivs:1.3],SizeBytes:7107254,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf k8s.gcr.io/e2e-test-images/busybox:1.29-2],SizeBytes:1154361,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:39e1e963e5310e9c313bad51523be012ede7b35bb9316517d19089a010356592 k8s.gcr.io/e2e-test-images/busybox:1.29-1],SizeBytes:1154361,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db k8s.gcr.io/pause:3.6 registry.k8s.io/pause:3.6],SizeBytes:682696,},},VolumesInUse:[kubernetes.io/csi/csi-mock-csi-mock-volumes-2206^4],VolumesAttached:[]AttachedVolume{},Config:nil,},} Jan 1 02:36:04.254: INFO: Logging kubelet events for node ip-172-20-33-66.eu-west-3.compute.internal Jan 1 02:36:04.360: INFO: Logging pods the kubelet thinks is on node ip-172-20-33-66.eu-west-3.compute.internal Jan 1 02:36:04.597: INFO: ebs-csi-node-vsxjc started at 2023-01-01 02:20:26 +0000 UTC (0+3 container statuses recorded) Jan 1 02:36:04.597: INFO: Container ebs-plugin ready: true, restart count 0 Jan 1 02:36:04.597: INFO: Container liveness-probe ready: true, restart count 0 Jan 1 02:36:04.597: INFO: Container node-driver-registrar ready: true, restart count 0 Jan 1 02:36:04.597: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-h4nvd started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:36:04.597: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:36:04.597: INFO: cilium-vbcgc started at 2023-01-01 02:20:26 +0000 UTC (1+1 container statuses recorded) Jan 1 02:36:04.597: INFO: Init container clean-cilium-state ready: true, restart count 0 Jan 1 02:36:04.597: INFO: Container cilium-agent ready: true, restart count 0 Jan 1 02:36:04.597: INFO: pvc-volume-tester-8t67k started at 2023-01-01 02:31:45 +0000 UTC (0+1 container statuses recorded) Jan 1 02:36:04.597: INFO: Container volume-tester ready: false, restart count 0 Jan 1 02:36:04.597: INFO: affinity-clusterip-timeout-q9qt2 started at 2023-01-01 02:27:32 +0000 UTC (0+1 container statuses recorded) Jan 1 02:36:04.597: INFO: Container affinity-clusterip-timeout ready: false, restart count 0 Jan 1 02:36:04.597: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-m6vq2 started at 2023-01-01 02:25:17 +0000 UTC (0+1 container statuses recorded) Jan 1 02:36:04.597: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:36:04.597: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-h6mmt started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:36:04.597: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:36:04.597: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-dkfpd started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:36:04.597: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:36:04.597: INFO: pvc-volume-tester-462k4 started at 2023-01-01 02:32:02 +0000 UTC (0+1 container statuses recorded) Jan 1 02:36:04.597: INFO: Container volume-tester ready: false, restart count 0 Jan 1 02:36:04.597: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-546bc started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:36:04.597: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:36:04.597: INFO: csi-mockplugin-attacher-0 started at 2023-01-01 02:27:56 +0000 UTC (0+1 container statuses recorded) Jan 1 02:36:04.597: INFO: Container csi-attacher ready: false, restart count 0 Jan 1 02:36:04.597: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-q8pgp started at 2023-01-01 02:25:17 +0000 UTC (0+1 container statuses recorded) Jan 1 02:36:04.597: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:36:04.597: INFO: netserver-0 started at 2023-01-01 02:31:31 +0000 UTC (0+1 container statuses recorded) Jan 1 02:36:04.597: INFO: Container webserver ready: false, restart count 0 Jan 1 02:36:04.597: INFO: csi-mockplugin-0 started at 2023-01-01 02:27:56 +0000 UTC (0+3 container statuses recorded) Jan 1 02:36:04.597: INFO: Container csi-provisioner ready: true, restart count 0 Jan 1 02:36:04.597: INFO: Container driver-registrar ready: true, restart count 0 Jan 1 02:36:04.597: INFO: Container mock ready: true, restart count 0 Jan 1 02:36:04.597: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-2v87j started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:36:04.597: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:36:04.597: INFO: coredns-867df8f45c-cqmp7 started at 2023-01-01 02:21:18 +0000 UTC (0+1 container statuses recorded) Jan 1 02:36:04.597: INFO: Container coredns ready: true, restart count 0 Jan 1 02:36:04.597: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-l9hnv started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:36:04.597: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:36:04.597: INFO: local-client started at 2023-01-01 02:31:07 +0000 UTC (0+1 container statuses recorded) Jan 1 02:36:04.597: INFO: Container local-client ready: false, restart count 0 Jan 1 02:36:04.597: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-vnv2x started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:36:04.597: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:36:04.597: INFO: csi-mockplugin-0 started at 2023-01-01 02:26:15 +0000 UTC (0+4 container statuses recorded) Jan 1 02:36:04.597: INFO: Container busybox ready: true, restart count 0 Jan 1 02:36:04.597: INFO: Container csi-provisioner ready: true, restart count 0 Jan 1 02:36:04.597: INFO: Container driver-registrar ready: true, restart count 0 Jan 1 02:36:04.597: INFO: Container mock ready: true, restart count 0 Jan 1 02:36:04.597: INFO: hostexec-ip-172-20-33-66.eu-west-3.compute.internal-qm7jt started at 2023-01-01 02:27:26 +0000 UTC (0+1 container statuses recorded) Jan 1 02:36:04.597: INFO: Container agnhost-container ready: true, restart count 0 Jan 1 02:36:04.597: INFO: netserver-0 started at 2023-01-01 02:33:25 +0000 UTC (0+1 container statuses recorded) Jan 1 02:36:04.597: INFO: Container webserver ready: false, restart count 0 Jan 1 02:36:04.597: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-csnzw started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:36:04.597: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: false, restart count 0 Jan 1 02:36:05.045: INFO: Latency metrics for node ip-172-20-33-66.eu-west-3.compute.internal Jan 1 02:36:05.045: INFO: Logging node info for node ip-172-20-38-238.eu-west-3.compute.internal Jan 1 02:36:05.149: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-38-238.eu-west-3.compute.internal 0eb713b4-64bf-47b7-8090-bab8111218ee 9308 0 2023-01-01 02:18:51 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:c5.large beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:eu-west-3 failure-domain.beta.kubernetes.io/zone:eu-west-3a kops.k8s.io/kops-controller-pki: kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-38-238.eu-west-3.compute.internal kubernetes.io/os:linux kubernetes.io/role:master node-role.kubernetes.io/control-plane: node-role.kubernetes.io/master: node.kubernetes.io/exclude-from-external-load-balancers: node.kubernetes.io/instance-type:c5.large topology.ebs.csi.aws.com/zone:eu-west-3a topology.kubernetes.io/region:eu-west-3 topology.kubernetes.io/zone:eu-west-3a] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-0f70ebb30a9076648"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kubelet Update v1 2023-01-01 02:18:50 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {protokube Update v1 2023-01-01 02:19:16 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kops.k8s.io/kops-controller-pki":{},"f:node-role.kubernetes.io/control-plane":{},"f:node.kubernetes.io/exclude-from-external-load-balancers":{}}}} } {kops-controller Update v1 2023-01-01 02:19:21 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/master":{}}}} } {kube-controller-manager Update v1 2023-01-01 02:19:31 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.0.0/24\"":{}},"f:taints":{}}} } {kubelet Update v1 2023-01-01 02:20:36 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.0.0/24,DoNotUseExternalID:,ProviderID:aws:///eu-west-3a/i-0f70ebb30a9076648,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/master,Value:,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[100.96.0.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{50478186496 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3918811136 0} {<nil>} 3826964Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{45430367772 0} {<nil>} 45430367772 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3813953536 0} {<nil>} 3724564Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-01-01 02:31:26 +0000 UTC,LastTransitionTime:2023-01-01 02:18:45 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-01-01 02:31:26 +0000 UTC,LastTransitionTime:2023-01-01 02:18:45 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-01-01 02:31:26 +0000 UTC,LastTransitionTime:2023-01-01 02:18:45 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-01-01 02:31:26 +0000 UTC,LastTransitionTime:2023-01-01 02:19:31 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.38.238,},NodeAddress{Type:ExternalIP,Address:15.237.26.58,},NodeAddress{Type:Hostname,Address:ip-172-20-38-238.eu-west-3.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-38-238.eu-west-3.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-15-237-26-58.eu-west-3.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2e6c64b5b811bf0c78b2aa7ccdb943,SystemUUID:ec2e6c64-b5b8-11bf-0c78-b2aa7ccdb943,BootID:9db01fa3-d136-4230-b21a-b03abbfb6bff,KernelVersion:4.19.0-23-cloud-amd64,OSImage:Debian GNU/Linux 10 (buster),ContainerRuntimeVersion:docker://20.10.17,KubeletVersion:v1.23.15,KubeProxyVersion:v1.23.15,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[registry.k8s.io/etcdadm/etcd-manager@sha256:66a453db625abb268f4b3bbefc5a34a171d81e6e8796cecca54cfd71775c77c4 registry.k8s.io/etcdadm/etcd-manager:v3.0.20221209],SizeBytes:662209237,},ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:456569528,},ContainerImage{Names:[k8s.gcr.io/kube-apiserver-amd64:v1.23.15 registry.k8s.io/kube-apiserver-amd64:v1.23.15],SizeBytes:135240444,},ContainerImage{Names:[k8s.gcr.io/kube-controller-manager-amd64:v1.23.15 registry.k8s.io/kube-controller-manager-amd64:v1.23.15],SizeBytes:125046097,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.23.15 registry.k8s.io/kube-proxy-amd64:v1.23.15],SizeBytes:112336036,},ContainerImage{Names:[quay.io/cilium/operator@sha256:a6d24a006a6b92967ac90786b49bc1ac26e5477cf028cd1186efcfc2466484db quay.io/cilium/operator:v1.12.5],SizeBytes:110016956,},ContainerImage{Names:[registry.k8s.io/kops/kops-controller:1.27.0-alpha.1],SizeBytes:105197757,},ContainerImage{Names:[registry.k8s.io/kops/dns-controller:1.27.0-alpha.1],SizeBytes:102577531,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:10e231cf07c82dfab5141c1afe548dc734e5fc1f67665ec3982a325d1bd31a9a registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.0],SizeBytes:94471559,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:122bfb8c1edabb3c0edd63f06523e6940d958d19b3957dc7b1d6f81e9f1f6119 registry.k8s.io/sig-storage/csi-provisioner:v3.1.0],SizeBytes:57736070,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:9ebbf9f023e7b41ccee3d52afe39a89e3ddacdbb69269d583abfc25847cfd9e4 registry.k8s.io/sig-storage/csi-resizer:v1.4.0],SizeBytes:55464778,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:8b9c313c05f54fb04f8d430896f5f5904b6cb157df261501b29adc04d2b2dc7b registry.k8s.io/sig-storage/csi-attacher:v3.4.0],SizeBytes:54848595,},ContainerImage{Names:[k8s.gcr.io/kube-scheduler-amd64:v1.23.15 registry.k8s.io/kube-scheduler-amd64:v1.23.15],SizeBytes:53521430,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:19582112,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:17611980,},ContainerImage{Names:[registry.k8s.io/kops/kube-apiserver-healthcheck:1.27.0-alpha.1],SizeBytes:8787111,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:682696,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Jan 1 02:36:05.150: INFO: Logging kubelet events for node ip-172-20-38-238.eu-west-3.compute.internal Jan 1 02:36:05.256: INFO: Logging pods the kubelet thinks is on node ip-172-20-38-238.eu-west-3.compute.internal Jan 1 02:36:05.367: INFO: etcd-manager-events-ip-172-20-38-238.eu-west-3.compute.internal started at 2023-01-01 02:18:11 +0000 UTC (0+1 container statuses recorded) Jan 1 02:36:05.367: INFO: Container etcd-manager ready: true, restart count 0 Jan 1 02:36:05.367: INFO: etcd-manager-main-ip-172-20-38-238.eu-west-3.compute.internal started at 2023-01-01 02:18:11 +0000 UTC (0+1 container statuses recorded) Jan 1 02:36:05.367: INFO: Container etcd-manager ready: true, restart count 0 Jan 1 02:36:05.367: INFO: cilium-qk59v started at 2023-01-01 02:19:20 +0000 UTC (1+1 container statuses recorded) Jan 1 02:36:05.367: INFO: Init container clean-cilium-state ready: true, restart count 0 Jan 1 02:36:05.367: INFO: Container cilium-agent ready: true, restart count 0 Jan 1 02:36:05.367: INFO: ebs-csi-node-k87k6 started at 2023-01-01 02:19:20 +0000 UTC (0+3 container statuses recorded) Jan 1 02:36:05.367: INFO: Container ebs-plugin ready: true, restart count 0 Jan 1 02:36:05.367: INFO: Container liveness-probe ready: true, restart count 0 Jan 1 02:36:05.367: INFO: Container node-driver-registrar ready: true, restart count 0 Jan 1 02:36:05.367: INFO: dns-controller-7db8b74b99-58zzw started at 2023-01-01 02:19:20 +0000 UTC (0+1 container statuses recorded) Jan 1 02:36:05.367: INFO: Container dns-controller ready: true, restart count 0 Jan 1 02:36:05.367: INFO: cilium-operator-d47487c76-j84nq started at 2023-01-01 02:19:20 +0000 UTC (0+1 container statuses recorded) Jan 1 02:36:05.367: INFO: Container cilium-operator ready: true, restart count 0 Jan 1 02:36:05.367: INFO: kube-controller-manager-ip-172-20-38-238.eu-west-3.compute.internal started at 2023-01-01 02:18:11 +0000 UTC (0+1 container statuses recorded) Jan 1 02:36:05.367: INFO: Container kube-controller-manager ready: true, restart count 2 Jan 1 02:36:05.367: INFO: kube-scheduler-ip-172-20-38-238.eu-west-3.compute.internal started at 2023-01-01 02:18:11 +0000 UTC (0+1 container statuses recorded) Jan 1 02:36:05.367: INFO: Container kube-scheduler ready: true, restart count 0 Jan 1 02:36:05.367: INFO: kube-apiserver-ip-172-20-38-238.eu-west-3.compute.internal started at 2023-01-01 02:18:11 +0000 UTC (0+2 container statuses recorded) Jan 1 02:36:05.367: INFO: Container healthcheck ready: true, restart count 0 Jan 1 02:36:05.367: INFO: Container kube-apiserver ready: true, restart count 1 Jan 1 02:36:05.367: INFO: kops-controller-26gzg started at 2023-01-01 02:19:20 +0000 UTC (0+1 container statuses recorded) Jan 1 02:36:05.367: INFO: Container kops-controller ready: true, restart count 0 Jan 1 02:36:05.367: INFO: ebs-csi-controller-6dc545b75-5qqh8 started at 2023-01-01 02:19:20 +0000 UTC (0+5 container statuses recorded) Jan 1 02:36:05.367: INFO: Container csi-attacher ready: true, restart count 0 Jan 1 02:36:05.367: INFO: Container csi-provisioner ready: true, restart count 0 Jan 1 02:36:05.367: INFO: Container csi-resizer ready: true, restart count 0 Jan 1 02:36:05.367: INFO: Container ebs-plugin ready: true, restart count 0 Jan 1 02:36:05.367: INFO: Container liveness-probe ready: true, restart count 0 Jan 1 02:36:05.698: INFO: Latency metrics for node ip-172-20-38-238.eu-west-3.compute.internal Jan 1 02:36:05.698: INFO: Logging node info for node ip-172-20-45-112.eu-west-3.compute.internal Jan 1 02:36:05.802: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-45-112.eu-west-3.compute.internal 48eed937-05bb-4d8a-bddd-aa94a2e6ab92 10711 0 2023-01-01 02:20:27 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:eu-west-3 failure-domain.beta.kubernetes.io/zone:eu-west-3a kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-45-112.eu-west-3.compute.internal kubernetes.io/os:linux kubernetes.io/role:node node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:eu-west-3a topology.hostpath.csi/node:ip-172-20-45-112.eu-west-3.compute.internal topology.kubernetes.io/region:eu-west-3 topology.kubernetes.io/zone:eu-west-3a] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-0778adf2085fa7f34"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kube-controller-manager Update v1 2023-01-01 02:20:27 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.3.0/24\"":{}}}} } {kubelet Update v1 2023-01-01 02:20:27 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {kops-controller Update v1 2023-01-01 02:20:28 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}} } {kube-controller-manager Update v1 2023-01-01 02:30:49 +0000 UTC FieldsV1 {"f:status":{"f:volumesAttached":{}}} status} {kubelet Update v1 2023-01-01 02:30:52 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:capacity":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.3.0/24,DoNotUseExternalID:,ProviderID:aws:///eu-west-3a/i-0778adf2085fa7f34,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.3.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{50478186496 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4078190592 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{45430367772 0} {<nil>} 45430367772 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3973332992 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-01-01 02:32:03 +0000 UTC,LastTransitionTime:2023-01-01 02:20:27 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-01-01 02:32:03 +0000 UTC,LastTransitionTime:2023-01-01 02:20:27 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-01-01 02:32:03 +0000 UTC,LastTransitionTime:2023-01-01 02:20:27 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-01-01 02:32:03 +0000 UTC,LastTransitionTime:2023-01-01 02:20:48 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.45.112,},NodeAddress{Type:ExternalIP,Address:13.38.32.14,},NodeAddress{Type:Hostname,Address:ip-172-20-45-112.eu-west-3.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-45-112.eu-west-3.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-13-38-32-14.eu-west-3.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2dbdf846e8ff614f4555e5bf9dbbf4,SystemUUID:ec2dbdf8-46e8-ff61-4f45-55e5bf9dbbf4,BootID:10a054ae-621a-47ce-9566-b0048cc5101b,KernelVersion:4.19.0-23-cloud-amd64,OSImage:Debian GNU/Linux 10 (buster),ContainerRuntimeVersion:docker://20.10.17,KubeletVersion:v1.23.15,KubeProxyVersion:v1.23.15,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:456569528,},ContainerImage{Names:[nginx@sha256:0047b729188a15da49380d9506d65959cce6d40291ccfb4e039f5dc7efd33286 nginx:latest],SizeBytes:141812353,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:7e8bdd271312fd25fc5ff5a8f04727be84044eb3d7d8d03611972a6752e2e11e k8s.gcr.io/e2e-test-images/agnhost:2.39],SizeBytes:126872991,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:1b9d1b2f36cb2dbee1960e82a9344aeb11bd4c4c03abf5e1853e0559c23855e3 k8s.gcr.io/e2e-test-images/httpd:2.4.38-2],SizeBytes:123781643,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.23.15 registry.k8s.io/kube-proxy-amd64:v1.23.15],SizeBytes:112336036,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:10e231cf07c82dfab5141c1afe548dc734e5fc1f67665ec3982a325d1bd31a9a registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.0],SizeBytes:94471559,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:c8e03f60afa90a28e4bb6ec9a8d0fc36d89de4b7475cf2d613afa793ec969fe0 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.0],SizeBytes:56382329,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:4e74c0492bceddc598de1c90cc5bc14dcda94cb49fa9c5bad9d117c4834b5e08 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.1],SizeBytes:56379269,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:36c31f7e1f433c9634d24f876353e8646246d81a03c4e351202c2644daff1620 k8s.gcr.io/sig-storage/csi-resizer:v1.2.0],SizeBytes:53956308,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:ed98431376c9e944e19a465fe8ea944806714dd95416a0821096c78d66b579bd k8s.gcr.io/sig-storage/csi-snapshotter:v4.1.1],SizeBytes:53531990,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:c5be65d6679efabb969d9b019300d187437ae876f992c40911fd2892bbef3b36 k8s.gcr.io/sig-storage/csi-attacher:v3.2.0],SizeBytes:53500826,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:60ab9b3e6a030d3038c87c0d6bca2930f58d1d72823e6a4af09767dc83b696a2 k8s.gcr.io/sig-storage/csi-attacher:v3.2.1],SizeBytes:53499570,},ContainerImage{Names:[k8s.gcr.io/sig-storage/hostpathplugin@sha256:232fe80174d60d520d36043103853a1d7ab4b7f3782cf43e45034f04ccda58ce k8s.gcr.io/sig-storage/hostpathplugin:v1.7.1],SizeBytes:29977921,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:a61d309da54641db41fb8f35718f744e9f730d4d0384f8c4b186ddc9f06cbd5f k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.1.0],SizeBytes:19662887,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:19582112,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:2dee3fe5fe861bb66c3a4ac51114f3447a4cd35870e0f2e2b558c7a400d89589 k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.2.0],SizeBytes:18699436,},ContainerImage{Names:[k8s.gcr.io/sig-storage/mock-driver@sha256:a7b517f9e0f42ffade802eef9cefa271372386b85f55b702b493241e58459793 k8s.gcr.io/sig-storage/mock-driver:v4.1.0],SizeBytes:17680993,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:17611980,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:529be2c9770add0cdd0c989115222ea9fc1be430c11095eb9f6dafcf98a36e2b k8s.gcr.io/sig-storage/livenessprobe:v2.4.0],SizeBytes:17175488,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:13616070e3f29de4417eee434a8ef472221c9e51b3d037b5a6b46cef08eb7443 k8s.gcr.io/e2e-test-images/nginx:1.14-2],SizeBytes:16032814,},ContainerImage{Names:[gcr.io/authenticated-image-pulling/alpine@sha256:7ff177862cb50c602bfe81f805969412e619c054a2bbead977d0c276988aa4a0 gcr.io/authenticated-image-pulling/alpine:3.7],SizeBytes:4206620,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf k8s.gcr.io/e2e-test-images/busybox:1.29-2],SizeBytes:1154361,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:39e1e963e5310e9c313bad51523be012ede7b35bb9316517d19089a010356592 k8s.gcr.io/e2e-test-images/busybox:1.29-1],SizeBytes:1154361,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db k8s.gcr.io/pause:3.6 registry.k8s.io/pause:3.6],SizeBytes:682696,},},VolumesInUse:[kubernetes.io/csi/ebs.csi.aws.com^vol-03d4513454f149737 kubernetes.io/csi/ebs.csi.aws.com^vol-08601772399ba2131],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-03d4513454f149737,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-08601772399ba2131,DevicePath:,},},Config:nil,},} Jan 1 02:36:05.802: INFO: Logging kubelet events for node ip-172-20-45-112.eu-west-3.compute.internal Jan 1 02:36:05.908: INFO: Logging pods the kubelet thinks is on node ip-172-20-45-112.eu-west-3.compute.internal Jan 1 02:36:06.023: INFO: pod-1 started at 2023-01-01 02:31:26 +0000 UTC (0+1 container statuses recorded) Jan 1 02:36:06.023: INFO: Container donothing ready: false, restart count 0 Jan 1 02:36:06.023: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-jh8bc started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:36:06.023: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:36:06.023: INFO: cilium-slx98 started at 2023-01-01 02:20:28 +0000 UTC (1+1 container statuses recorded) Jan 1 02:36:06.023: INFO: Init container clean-cilium-state ready: true, restart count 0 Jan 1 02:36:06.023: INFO: Container cilium-agent ready: true, restart count 0 Jan 1 02:36:06.023: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-4lrj2 started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:36:06.023: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:36:06.023: INFO: netserver-1 started at 2023-01-01 02:33:25 +0000 UTC (0+1 container statuses recorded) Jan 1 02:36:06.023: INFO: Container webserver ready: false, restart count 0 Jan 1 02:36:06.023: INFO: affinity-clusterip-timeout-lv5vx started at 2023-01-01 02:27:32 +0000 UTC (0+1 container statuses recorded) Jan 1 02:36:06.023: INFO: Container affinity-clusterip-timeout ready: true, restart count 0 Jan 1 02:36:06.023: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-srf5k started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:36:06.023: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:36:06.023: INFO: pod-0 started at 2023-01-01 02:31:26 +0000 UTC (0+1 container statuses recorded) Jan 1 02:36:06.023: INFO: Container donothing ready: true, restart count 0 Jan 1 02:36:06.023: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-4mml9 started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:36:06.023: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:36:06.023: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-x5cp6 started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:36:06.023: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:36:06.023: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-xnfzj started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:36:06.023: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:36:06.023: INFO: pod-d05d599f-1b25-4249-b9c7-c64a79abebd3 started at 2023-01-01 02:30:59 +0000 UTC (0+1 container statuses recorded) Jan 1 02:36:06.023: INFO: Container write-pod ready: false, restart count 0 Jan 1 02:36:06.023: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-gph2d started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:36:06.023: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:36:06.023: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-59bdj started at 2023-01-01 02:25:17 +0000 UTC (0+1 container statuses recorded) Jan 1 02:36:06.023: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:36:06.023: INFO: ebs-csi-node-7clrr started at 2023-01-01 02:20:28 +0000 UTC (0+3 container statuses recorded) Jan 1 02:36:06.023: INFO: Container ebs-plugin ready: true, restart count 0 Jan 1 02:36:06.023: INFO: Container liveness-probe ready: true, restart count 0 Jan 1 02:36:06.023: INFO: Container node-driver-registrar ready: true, restart count 0 Jan 1 02:36:06.023: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-rmdrr started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:36:06.023: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:36:06.023: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-kjg2m started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:36:06.023: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:36:06.023: INFO: inline-volume-tester-fpr45 started at 2023-01-01 02:31:04 +0000 UTC (0+1 container statuses recorded) Jan 1 02:36:06.023: INFO: Container csi-volume-tester ready: false, restart count 0 Jan 1 02:36:06.023: INFO: suspend-false-to-true-kb2jl started at 2023-01-01 02:31:57 +0000 UTC (0+1 container statuses recorded) Jan 1 02:36:06.023: INFO: Container c ready: false, restart count 0 Jan 1 02:36:06.023: INFO: netserver-1 started at 2023-01-01 02:31:32 +0000 UTC (0+1 container statuses recorded) Jan 1 02:36:06.023: INFO: Container webserver ready: false, restart count 0 Jan 1 02:36:06.503: INFO: Latency metrics for node ip-172-20-45-112.eu-west-3.compute.internal Jan 1 02:36:06.503: INFO: Logging node info for node ip-172-20-47-178.eu-west-3.compute.internal Jan 1 02:36:06.607: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-47-178.eu-west-3.compute.internal 835e5337-d2b9-45ad-bf0e-9d32514f71ba 10714 0 2023-01-01 02:20:25 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:eu-west-3 failure-domain.beta.kubernetes.io/zone:eu-west-3a kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-47-178.eu-west-3.compute.internal kubernetes.io/os:linux kubernetes.io/role:node node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:eu-west-3a topology.hostpath.csi/node:ip-172-20-47-178.eu-west-3.compute.internal topology.kubernetes.io/region:eu-west-3 topology.kubernetes.io/zone:eu-west-3a] map[csi.volume.kubernetes.io/nodeid:{"csi-hostpath-ephemeral-1708":"ip-172-20-47-178.eu-west-3.compute.internal","ebs.csi.aws.com":"i-00e1e60199f5a4d04"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kops-controller Update v1 2023-01-01 02:20:25 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}} } {kube-controller-manager Update v1 2023-01-01 02:20:25 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.2.0/24\"":{}}}} } {kubelet Update v1 2023-01-01 02:20:25 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {kube-controller-manager Update v1 2023-01-01 02:32:52 +0000 UTC FieldsV1 {"f:status":{"f:volumesAttached":{}}} status} {kubelet Update v1 2023-01-01 02:33:00 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:capacity":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.2.0/24,DoNotUseExternalID:,ProviderID:aws:///eu-west-3a/i-00e1e60199f5a4d04,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.2.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{50478186496 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4078190592 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{45430367772 0} {<nil>} 45430367772 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3973332992 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-01-01 02:33:00 +0000 UTC,LastTransitionTime:2023-01-01 02:20:25 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-01-01 02:33:00 +0000 UTC,LastTransitionTime:2023-01-01 02:20:25 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-01-01 02:33:00 +0000 UTC,LastTransitionTime:2023-01-01 02:20:25 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-01-01 02:33:00 +0000 UTC,LastTransitionTime:2023-01-01 02:20:46 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.47.178,},NodeAddress{Type:ExternalIP,Address:13.36.177.101,},NodeAddress{Type:Hostname,Address:ip-172-20-47-178.eu-west-3.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-47-178.eu-west-3.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-13-36-177-101.eu-west-3.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec243eda226c96d5573f3eaa3203df0d,SystemUUID:ec243eda-226c-96d5-573f-3eaa3203df0d,BootID:15dccaa7-d7c7-419d-b7de-76be5781a51a,KernelVersion:4.19.0-23-cloud-amd64,OSImage:Debian GNU/Linux 10 (buster),ContainerRuntimeVersion:docker://20.10.17,KubeletVersion:v1.23.15,KubeProxyVersion:v1.23.15,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:456569528,},ContainerImage{Names:[nginx@sha256:0047b729188a15da49380d9506d65959cce6d40291ccfb4e039f5dc7efd33286 nginx:latest],SizeBytes:141812353,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:7e8bdd271312fd25fc5ff5a8f04727be84044eb3d7d8d03611972a6752e2e11e k8s.gcr.io/e2e-test-images/agnhost:2.39],SizeBytes:126872991,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.23.15 registry.k8s.io/kube-proxy-amd64:v1.23.15],SizeBytes:112336036,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:10e231cf07c82dfab5141c1afe548dc734e5fc1f67665ec3982a325d1bd31a9a registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.0],SizeBytes:94471559,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:c8e03f60afa90a28e4bb6ec9a8d0fc36d89de4b7475cf2d613afa793ec969fe0 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.0],SizeBytes:56382329,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:4e74c0492bceddc598de1c90cc5bc14dcda94cb49fa9c5bad9d117c4834b5e08 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.1],SizeBytes:56379269,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:36c31f7e1f433c9634d24f876353e8646246d81a03c4e351202c2644daff1620 k8s.gcr.io/sig-storage/csi-resizer:v1.2.0],SizeBytes:53956308,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:ed98431376c9e944e19a465fe8ea944806714dd95416a0821096c78d66b579bd k8s.gcr.io/sig-storage/csi-snapshotter:v4.1.1],SizeBytes:53531990,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:c5be65d6679efabb969d9b019300d187437ae876f992c40911fd2892bbef3b36 k8s.gcr.io/sig-storage/csi-attacher:v3.2.0],SizeBytes:53500826,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:60ab9b3e6a030d3038c87c0d6bca2930f58d1d72823e6a4af09767dc83b696a2 k8s.gcr.io/sig-storage/csi-attacher:v3.2.1],SizeBytes:53499570,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:017727efcfeb7d053af68e51436ce8e65edbc6ca573720afb4f79c8594036955 registry.k8s.io/coredns/coredns:v1.10.0],SizeBytes:50249337,},ContainerImage{Names:[registry.k8s.io/cpa/cluster-proportional-autoscaler@sha256:68d396900aeaa072c1f27289485fdac29834045a6f3ffe369bf389d830ef572d registry.k8s.io/cpa/cluster-proportional-autoscaler:1.8.6],SizeBytes:47910609,},ContainerImage{Names:[k8s.gcr.io/sig-storage/hostpathplugin@sha256:232fe80174d60d520d36043103853a1d7ab4b7f3782cf43e45034f04ccda58ce k8s.gcr.io/sig-storage/hostpathplugin:v1.7.1],SizeBytes:29977921,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:a61d309da54641db41fb8f35718f744e9f730d4d0384f8c4b186ddc9f06cbd5f k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.1.0],SizeBytes:19662887,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:19582112,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:2dee3fe5fe861bb66c3a4ac51114f3447a4cd35870e0f2e2b558c7a400d89589 k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.2.0],SizeBytes:18699436,},ContainerImage{Names:[k8s.gcr.io/sig-storage/mock-driver@sha256:a7b517f9e0f42ffade802eef9cefa271372386b85f55b702b493241e58459793 k8s.gcr.io/sig-storage/mock-driver:v4.1.0],SizeBytes:17680993,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:17611980,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:529be2c9770add0cdd0c989115222ea9fc1be430c11095eb9f6dafcf98a36e2b k8s.gcr.io/sig-storage/livenessprobe:v2.4.0],SizeBytes:17175488,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:13616070e3f29de4417eee434a8ef472221c9e51b3d037b5a6b46cef08eb7443 k8s.gcr.io/e2e-test-images/nginx:1.14-2],SizeBytes:16032814,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf k8s.gcr.io/e2e-test-images/busybox:1.29-2],SizeBytes:1154361,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db k8s.gcr.io/pause:3.6 registry.k8s.io/pause:3.6],SizeBytes:682696,},},VolumesInUse:[kubernetes.io/csi/csi-hostpath-ephemeral-1708^957f1b8b-897c-11ed-afb4-7606258f5a7f],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/csi-hostpath-ephemeral-1708^957f1b8b-897c-11ed-afb4-7606258f5a7f,DevicePath:,},},Config:nil,},} Jan 1 02:36:06.608: INFO: Logging kubelet events for node ip-172-20-47-178.eu-west-3.compute.internal Jan 1 02:36:06.714: INFO: Logging pods the kubelet thinks is on node ip-172-20-47-178.eu-west-3.compute.internal Jan 1 02:36:06.849: INFO: ebs-csi-node-vcprc started at 2023-01-01 02:20:26 +0000 UTC (0+3 container statuses recorded) Jan 1 02:36:06.849: INFO: Container ebs-plugin ready: true, restart count 0 Jan 1 02:36:06.849: INFO: Container liveness-probe ready: true, restart count 0 Jan 1 02:36:06.849: INFO: Container node-driver-registrar ready: true, restart count 0 Jan 1 02:36:06.849: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-49brr started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:36:06.849: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:36:06.849: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-z87gr started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:36:06.849: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:36:06.849: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-wjz96 started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:36:06.849: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:36:06.849: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-w26gj started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:36:06.849: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:36:06.849: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-s4rqf started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:36:06.849: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:36:06.849: INFO: cilium-kk2c2 started at 2023-01-01 02:20:26 +0000 UTC (1+1 container statuses recorded) Jan 1 02:36:06.849: INFO: Init container clean-cilium-state ready: true, restart count 0 Jan 1 02:36:06.849: INFO: Container cilium-agent ready: true, restart count 0 Jan 1 02:36:06.849: INFO: local-injector started at 2023-01-01 02:31:07 +0000 UTC (0+1 container statuses recorded) Jan 1 02:36:06.849: INFO: Container local-injector ready: false, restart count 0 Jan 1 02:36:06.849: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-6vhtn started at 2023-01-01 02:25:17 +0000 UTC (0+1 container statuses recorded) Jan 1 02:36:06.849: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: false, restart count 0 Jan 1 02:36:06.849: INFO: netserver-2 started at 2023-01-01 02:33:25 +0000 UTC (0+1 container statuses recorded) Jan 1 02:36:06.849: INFO: Container webserver ready: false, restart count 0 Jan 1 02:36:06.849: INFO: coredns-867df8f45c-wzntb started at 2023-01-01 02:20:46 +0000 UTC (0+1 container statuses recorded) Jan 1 02:36:06.849: INFO: Container coredns ready: true, restart count 0 Jan 1 02:36:06.849: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-wkclp started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:36:06.849: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:36:06.849: INFO: netserver-2 started at 2023-01-01 02:31:32 +0000 UTC (0+1 container statuses recorded) Jan 1 02:36:06.849: INFO: Container webserver ready: false, restart count 0 Jan 1 02:36:06.849: INFO: inline-volume-tester-t46bx started at 2023-01-01 02:32:51 +0000 UTC (0+1 container statuses recorded) Jan 1 02:36:06.849: INFO: Container csi-volume-tester ready: false, restart count 0 Jan 1 02:36:06.849: INFO: coredns-autoscaler-557ccb4c66-k4lzb started at 2023-01-01 02:20:46 +0000 UTC (0+1 container statuses recorded) Jan 1 02:36:06.849: INFO: Container autoscaler ready: true, restart count 0 Jan 1 02:36:06.849: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-pgkm2 started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:36:06.849: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:36:06.849: INFO: hostexec-ip-172-20-47-178.eu-west-3.compute.internal-k7k4h started at 2023-01-01 02:30:47 +0000 UTC (0+1 container statuses recorded) Jan 1 02:36:06.849: INFO: Container agnhost-container ready: true, restart count 0 Jan 1 02:36:06.849: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-hrgrq started at 2023-01-01 02:25:17 +0000 UTC (0+1 container statuses recorded) Jan 1 02:36:06.849: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:36:06.849: INFO: csi-hostpathplugin-0 started at 2023-01-01 02:31:02 +0000 UTC (0+7 container statuses recorded) Jan 1 02:36:06.849: INFO: Container csi-attacher ready: false, restart count 0 Jan 1 02:36:06.849: INFO: Container csi-provisioner ready: false, restart count 0 Jan 1 02:36:06.849: INFO: Container csi-resizer ready: false, restart count 0 Jan 1 02:36:06.849: INFO: Container csi-snapshotter ready: false, restart count 0 Jan 1 02:36:06.849: INFO: Container hostpath ready: false, restart count 0 Jan 1 02:36:06.849: INFO: Container liveness-probe ready: false, restart count 0 Jan 1 02:36:06.849: INFO: Container node-driver-registrar ready: false, restart count 0 Jan 1 02:36:06.849: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-grrsl started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:36:06.849: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:36:06.850: INFO: csi-hostpathplugin-0 started at 2023-01-01 02:30:56 +0000 UTC (0+7 container statuses recorded) Jan 1 02:36:06.850: INFO: Container csi-attacher ready: true, restart count 0 Jan 1 02:36:06.850: INFO: Container csi-provisioner ready: true, restart count 0 Jan 1 02:36:06.850: INFO: Container csi-resizer ready: true, restart count 0 Jan 1 02:36:06.850: INFO: Container csi-snapshotter ready: true, restart count 0 Jan 1 02:36:06.850: INFO: Container hostpath ready: true, restart count 0 Jan 1 02:36:06.850: INFO: Container liveness-probe ready: true, restart count 0 Jan 1 02:36:06.850: INFO: Container node-driver-registrar ready: true, restart count 0 Jan 1 02:36:07.507: INFO: Latency metrics for node ip-172-20-47-178.eu-west-3.compute.internal Jan 1 02:36:07.507: INFO: Logging node info for node ip-172-20-49-224.eu-west-3.compute.internal Jan 1 02:36:07.611: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-49-224.eu-west-3.compute.internal d7d57146-c54a-4155-94f8-0f322216bfcf 10718 0 2023-01-01 02:20:28 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:eu-west-3 failure-domain.beta.kubernetes.io/zone:eu-west-3a kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-49-224.eu-west-3.compute.internal kubernetes.io/os:linux kubernetes.io/role:node node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:eu-west-3a topology.hostpath.csi/node:ip-172-20-49-224.eu-west-3.compute.internal topology.kubernetes.io/region:eu-west-3 topology.kubernetes.io/zone:eu-west-3a] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-089a107939b27c28a"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kops-controller Update v1 2023-01-01 02:20:28 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}} } {kube-controller-manager Update v1 2023-01-01 02:20:28 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.4.0/24\"":{}}}} } {kubelet Update v1 2023-01-01 02:20:28 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {kube-controller-manager Update v1 2023-01-01 02:34:44 +0000 UTC FieldsV1 {"f:status":{"f:volumesAttached":{}}} status} {kubelet Update v1 2023-01-01 02:34:48 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:capacity":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.4.0/24,DoNotUseExternalID:,ProviderID:aws:///eu-west-3a/i-089a107939b27c28a,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.4.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{50478186496 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4078206976 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{45430367772 0} {<nil>} 45430367772 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3973349376 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-01-01 02:34:48 +0000 UTC,LastTransitionTime:2023-01-01 02:20:28 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-01-01 02:34:48 +0000 UTC,LastTransitionTime:2023-01-01 02:20:28 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-01-01 02:34:48 +0000 UTC,LastTransitionTime:2023-01-01 02:20:28 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-01-01 02:34:48 +0000 UTC,LastTransitionTime:2023-01-01 02:20:48 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.49.224,},NodeAddress{Type:ExternalIP,Address:15.237.95.101,},NodeAddress{Type:Hostname,Address:ip-172-20-49-224.eu-west-3.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-49-224.eu-west-3.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-15-237-95-101.eu-west-3.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2d0a1bb33dd26f5a12c3896530131b,SystemUUID:ec2d0a1b-b33d-d26f-5a12-c3896530131b,BootID:9f64e91d-26f0-45aa-aec4-a54433e67034,KernelVersion:4.19.0-23-cloud-amd64,OSImage:Debian GNU/Linux 10 (buster),ContainerRuntimeVersion:docker://20.10.17,KubeletVersion:v1.23.15,KubeProxyVersion:v1.23.15,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:456569528,},ContainerImage{Names:[nginx@sha256:0047b729188a15da49380d9506d65959cce6d40291ccfb4e039f5dc7efd33286 nginx:latest],SizeBytes:141812353,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:7e8bdd271312fd25fc5ff5a8f04727be84044eb3d7d8d03611972a6752e2e11e k8s.gcr.io/e2e-test-images/agnhost:2.39],SizeBytes:126872991,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:1b9d1b2f36cb2dbee1960e82a9344aeb11bd4c4c03abf5e1853e0559c23855e3 k8s.gcr.io/e2e-test-images/httpd:2.4.38-2],SizeBytes:123781643,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.23.15 registry.k8s.io/kube-proxy-amd64:v1.23.15],SizeBytes:112336036,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:10e231cf07c82dfab5141c1afe548dc734e5fc1f67665ec3982a325d1bd31a9a registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.0],SizeBytes:94471559,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:4e74c0492bceddc598de1c90cc5bc14dcda94cb49fa9c5bad9d117c4834b5e08 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.1],SizeBytes:56379269,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:36c31f7e1f433c9634d24f876353e8646246d81a03c4e351202c2644daff1620 k8s.gcr.io/sig-storage/csi-resizer:v1.2.0],SizeBytes:53956308,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:ed98431376c9e944e19a465fe8ea944806714dd95416a0821096c78d66b579bd k8s.gcr.io/sig-storage/csi-snapshotter:v4.1.1],SizeBytes:53531990,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:c5be65d6679efabb969d9b019300d187437ae876f992c40911fd2892bbef3b36 k8s.gcr.io/sig-storage/csi-attacher:v3.2.0],SizeBytes:53500826,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:60ab9b3e6a030d3038c87c0d6bca2930f58d1d72823e6a4af09767dc83b696a2 k8s.gcr.io/sig-storage/csi-attacher:v3.2.1],SizeBytes:53499570,},ContainerImage{Names:[k8s.gcr.io/sig-storage/hostpathplugin@sha256:232fe80174d60d520d36043103853a1d7ab4b7f3782cf43e45034f04ccda58ce k8s.gcr.io/sig-storage/hostpathplugin:v1.7.1],SizeBytes:29977921,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:19582112,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:2dee3fe5fe861bb66c3a4ac51114f3447a4cd35870e0f2e2b558c7a400d89589 k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.2.0],SizeBytes:18699436,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:17611980,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:529be2c9770add0cdd0c989115222ea9fc1be430c11095eb9f6dafcf98a36e2b k8s.gcr.io/sig-storage/livenessprobe:v2.4.0],SizeBytes:17175488,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:13616070e3f29de4417eee434a8ef472221c9e51b3d037b5a6b46cef08eb7443 k8s.gcr.io/e2e-test-images/nginx:1.14-2],SizeBytes:16032814,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf k8s.gcr.io/e2e-test-images/busybox:1.29-2],SizeBytes:1154361,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db k8s.gcr.io/pause:3.6 registry.k8s.io/pause:3.6],SizeBytes:682696,},},VolumesInUse:[kubernetes.io/csi/ebs.csi.aws.com^vol-06c2c39b58fd301cd],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-06c2c39b58fd301cd,DevicePath:,},},Config:nil,},} Jan 1 02:36:07.611: INFO: Logging kubelet events for node ip-172-20-49-224.eu-west-3.compute.internal Jan 1 02:36:07.720: INFO: Logging pods the kubelet thinks is on node ip-172-20-49-224.eu-west-3.compute.internal Jan 1 02:36:07.851: INFO: ebs-csi-node-tzntf started at 2023-01-01 02:20:29 +0000 UTC (0+3 container statuses recorded) Jan 1 02:36:07.851: INFO: Container ebs-plugin ready: true, restart count 0 Jan 1 02:36:07.851: INFO: Container liveness-probe ready: true, restart count 0 Jan 1 02:36:07.851: INFO: Container node-driver-registrar ready: true, restart count 0 Jan 1 02:36:07.851: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-4jf6d started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:36:07.851: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:36:07.851: INFO: dns-test-f6141192-e7ab-4530-bedd-b1f3b48a9237 started at 2023-01-01 02:30:56 +0000 UTC (0+3 container statuses recorded) Jan 1 02:36:07.851: INFO: Container jessie-querier ready: false, restart count 0 Jan 1 02:36:07.851: INFO: Container querier ready: false, restart count 0 Jan 1 02:36:07.851: INFO: Container webserver ready: false, restart count 0 Jan 1 02:36:07.851: INFO: pod-projected-secrets-fedc3ea5-055e-4010-aa5a-0d89b2f8b6ac started at 2023-01-01 02:31:36 +0000 UTC (0+1 container statuses recorded) Jan 1 02:36:07.851: INFO: Container projected-secret-volume-test ready: false, restart count 0 Jan 1 02:36:07.851: INFO: pod-configmaps-f941fae8-7952-461d-b186-629d475d0f88 started at 2023-01-01 02:33:31 +0000 UTC (0+1 container statuses recorded) Jan 1 02:36:07.851: INFO: Container agnhost-container ready: false, restart count 0 Jan 1 02:36:07.851: INFO: cilium-78jll started at 2023-01-01 02:20:29 +0000 UTC (1+1 container statuses recorded) Jan 1 02:36:07.851: INFO: Init container clean-cilium-state ready: true, restart count 0 Jan 1 02:36:07.851: INFO: Container cilium-agent ready: true, restart count 0 Jan 1 02:36:07.851: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-tv984 started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:36:07.851: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:36:07.851: INFO: affinity-clusterip-timeout-pxjtk started at 2023-01-01 02:27:32 +0000 UTC (0+1 container statuses recorded) Jan 1 02:36:07.851: INFO: Container affinity-clusterip-timeout ready: true, restart count 0 Jan 1 02:36:07.851: INFO: netserver-3 started at 2023-01-01 02:33:25 +0000 UTC (0+1 container statuses recorded) Jan 1 02:36:07.851: INFO: Container webserver ready: false, restart count 0 Jan 1 02:36:07.851: INFO: liveness-b61a07c8-7116-4bf3-bf05-31114b2bce91 started at 2023-01-01 02:36:05 +0000 UTC (0+1 container statuses recorded) Jan 1 02:36:07.851: INFO: Container agnhost-container ready: false, restart count 0 Jan 1 02:36:07.851: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-2gndb started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:36:07.851: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:36:07.851: INFO: csi-mockplugin-0 started at 2023-01-01 02:31:26 +0000 UTC (0+3 container statuses recorded) Jan 1 02:36:07.851: INFO: Container csi-provisioner ready: false, restart count 0 Jan 1 02:36:07.851: INFO: Container driver-registrar ready: false, restart count 0 Jan 1 02:36:07.851: INFO: Container mock ready: false, restart count 0 Jan 1 02:36:07.851: INFO: pod-3e35f616-b2af-42fe-8148-bfe7bf04221b started at 2023-01-01 02:31:05 +0000 UTC (0+1 container statuses recorded) Jan 1 02:36:07.851: INFO: Container write-pod ready: false, restart count 0 Jan 1 02:36:07.851: INFO: dns-test-a8277dcf-764d-4d97-a381-813a8d1f903c started at 2023-01-01 02:31:17 +0000 UTC (0+3 container statuses recorded) Jan 1 02:36:07.851: INFO: Container jessie-querier ready: false, restart count 0 Jan 1 02:36:07.851: INFO: Container querier ready: false, restart count 0 Jan 1 02:36:07.851: INFO: Container webserver ready: false, restart count 0 Jan 1 02:36:07.851: INFO: csi-mockplugin-attacher-0 started at 2023-01-01 02:31:26 +0000 UTC (0+1 container statuses recorded) Jan 1 02:36:07.851: INFO: Container csi-attacher ready: true, restart count 0 Jan 1 02:36:07.851: INFO: test-ss-0 started at 2023-01-01 02:31:35 +0000 UTC (0+1 container statuses recorded) Jan 1 02:36:07.851: INFO: Container webserver ready: false, restart count 0 Jan 1 02:36:07.851: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-cz8qh started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:36:07.851: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:36:07.851: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-6n5j9 started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:36:07.851: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:36:07.851: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-h6dxb started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:36:07.851: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:36:07.851: INFO: inline-volume-tester-tf6rn started at 2023-01-01 02:34:41 +0000 UTC (0+1 container statuses recorded) Jan 1 02:36:07.851: INFO: Container csi-volume-tester ready: false, restart count 0 Jan 1 02:36:07.851: INFO: test-ss-0 started at 2023-01-01 02:27:56 +0000 UTC (0+1 container statuses recorded) Jan 1 02:36:07.851: INFO: Container webserver ready: false, restart count 0 Jan 1 02:36:07.851: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-pgc9g started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:36:07.851: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:36:07.851: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-29cdj started at 2023-01-01 02:25:17 +0000 UTC (0+1 container statuses recorded) Jan 1 02:36:07.851: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: false, restart count 0 Jan 1 02:36:07.851: INFO: suspend-false-to-true-g6ml8 started at 2023-01-01 02:31:57 +0000 UTC (0+1 container statuses recorded) Jan 1 02:36:07.851: INFO: Container c ready: false, restart count 0 Jan 1 02:36:07.851: INFO: hostexec-ip-172-20-49-224.eu-west-3.compute.internal-hsfbk started at 2023-01-01 02:31:01 +0000 UTC (0+1 container statuses recorded) Jan 1 02:36:07.851: INFO: Container agnhost-container ready: true, restart count 0 Jan 1 02:36:07.851: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-lrzjq started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:36:07.851: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:36:07.851: INFO: pod-secrets-6516282b-cc4c-4ad6-9124-72e1a9d3d0dd started at 2023-01-01 02:36:06 +0000 UTC (0+1 container statuses recorded) Jan 1 02:36:07.851: INFO: Container secret-test ready: false, restart count 0 Jan 1 02:36:07.851: INFO: pod-2 started at 2023-01-01 02:31:27 +0000 UTC (0+1 container statuses recorded) Jan 1 02:36:07.851: INFO: Container donothing ready: false, restart count 0 Jan 1 02:36:07.851: INFO: netserver-3 started at 2023-01-01 02:31:32 +0000 UTC (0+1 container statuses recorded) Jan 1 02:36:07.851: INFO: Container webserver ready: false, restart count 0 Jan 1 02:36:07.851: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-7z6bp started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:36:07.851: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:36:08.308: INFO: Latency metrics for node ip-172-20-49-224.eu-west-3.compute.internal Jan 1 02:36:08.308: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready �[1mSTEP�[0m: Destroying namespace "dns-5931" for this suite.
Find e2e-dns-utils mentions in log files
go run hack/e2e.go -v --test --test_args='--ginkgo.focus=Kubernetes\se2e\ssuite\s\[sig\-network\]\sNetworking\sGranular\sChecks\:\sServices\sshould\sfunction\sfor\snode\-Service\:\sudp$'
/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/network/networking.go:204 Jan 1 02:31:20.707: Unexpected error: <*errors.errorString | 0xc0002422b0>: { s: "timed out waiting for the condition", } timed out waiting for the condition occurred /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/network/utils.go:858from junit_05.xml
[BeforeEach] [sig-network] Networking /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:185 �[1mSTEP�[0m: Creating a kubernetes client Jan 1 02:25:26.383: INFO: >>> kubeConfig: /root/.kube/config �[1mSTEP�[0m: Building a namespace api object, basename nettest �[1mSTEP�[0m: Waiting for a default service account to be provisioned in namespace �[1mSTEP�[0m: Waiting for kube-root-ca.crt to be provisioned in namespace [It] should function for node-Service: udp /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/network/networking.go:204 �[1mSTEP�[0m: Performing setup for networking test in namespace nettest-5021 �[1mSTEP�[0m: creating a selector �[1mSTEP�[0m: Creating the service pods in kubernetes Jan 1 02:25:27.116: INFO: Waiting up to 10m0s for all (but 0) nodes to be schedulable Jan 1 02:25:27.867: INFO: The status of Pod netserver-0 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:25:29.973: INFO: The status of Pod netserver-0 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:25:31.973: INFO: The status of Pod netserver-0 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:25:33.971: INFO: The status of Pod netserver-0 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:25:35.975: INFO: The status of Pod netserver-0 is Running (Ready = false) Jan 1 02:25:37.975: INFO: The status of Pod netserver-0 is Running (Ready = false) Jan 1 02:25:39.972: INFO: The status of Pod netserver-0 is Running (Ready = false) Jan 1 02:25:41.974: INFO: The status of Pod netserver-0 is Running (Ready = false) Jan 1 02:25:43.971: INFO: The status of Pod netserver-0 is Running (Ready = false) Jan 1 02:25:45.972: INFO: The status of Pod netserver-0 is Running (Ready = false) Jan 1 02:25:47.971: INFO: The status of Pod netserver-0 is Running (Ready = true) Jan 1 02:25:48.180: INFO: The status of Pod netserver-1 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:25:50.288: INFO: The status of Pod netserver-1 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:25:52.288: INFO: The status of Pod netserver-1 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:25:54.286: INFO: The status of Pod netserver-1 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:25:56.287: INFO: The status of Pod netserver-1 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:25:58.289: INFO: The status of Pod netserver-1 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:26:00.285: INFO: The status of Pod netserver-1 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:26:02.287: INFO: The status of Pod netserver-1 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:26:04.286: INFO: The status of Pod netserver-1 is Running (Ready = false) Jan 1 02:26:06.287: INFO: The status of Pod netserver-1 is Running (Ready = false) Jan 1 02:26:08.286: INFO: The status of Pod netserver-1 is Running (Ready = false) Jan 1 02:26:10.287: INFO: The status of Pod netserver-1 is Running (Ready = false) Jan 1 02:26:12.286: INFO: The status of Pod netserver-1 is Running (Ready = false) Jan 1 02:26:14.286: INFO: The status of Pod netserver-1 is Running (Ready = false) Jan 1 02:26:16.285: INFO: The status of Pod netserver-1 is Running (Ready = false) Jan 1 02:26:18.286: INFO: The status of Pod netserver-1 is Running (Ready = false) Jan 1 02:26:20.287: INFO: The status of Pod netserver-1 is Running (Ready = true) Jan 1 02:26:20.496: INFO: The status of Pod netserver-2 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:26:22.602: INFO: The status of Pod netserver-2 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:26:24.602: INFO: The status of Pod netserver-2 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:26:26.601: INFO: The status of Pod netserver-2 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:26:28.601: INFO: The status of Pod netserver-2 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:26:30.601: INFO: The status of Pod netserver-2 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:26:32.602: INFO: The status of Pod netserver-2 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:26:34.601: INFO: The status of Pod netserver-2 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:26:36.602: INFO: The status of Pod netserver-2 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:26:38.601: INFO: The status of Pod netserver-2 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:26:40.602: INFO: The status of Pod netserver-2 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:26:42.602: INFO: The status of Pod netserver-2 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:26:44.601: INFO: The status of Pod netserver-2 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:26:46.601: INFO: The status of Pod netserver-2 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:26:48.602: INFO: The status of Pod netserver-2 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:26:50.603: INFO: The status of Pod netserver-2 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:26:52.601: INFO: The status of Pod netserver-2 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:26:54.602: INFO: The status of Pod netserver-2 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:26:56.601: INFO: The status of Pod netserver-2 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:26:58.601: INFO: The status of Pod netserver-2 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:27:00.601: INFO: The status of Pod netserver-2 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:27:02.601: INFO: The status of Pod netserver-2 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:27:04.602: INFO: The status of Pod netserver-2 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:27:06.604: INFO: The status of Pod netserver-2 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:27:08.603: INFO: The status of Pod netserver-2 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:27:10.603: INFO: The status of Pod netserver-2 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:27:12.601: INFO: The status of Pod netserver-2 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:27:14.602: INFO: The status of Pod netserver-2 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:27:16.601: INFO: The status of Pod netserver-2 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:27:18.601: INFO: The status of Pod netserver-2 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:27:20.601: INFO: The status of Pod netserver-2 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:27:22.602: INFO: The status of Pod netserver-2 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:27:24.603: INFO: The status of Pod netserver-2 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:27:26.601: INFO: The status of Pod netserver-2 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:27:28.602: INFO: The status of Pod netserver-2 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:27:30.601: INFO: The status of Pod netserver-2 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:27:32.614: INFO: The status of Pod netserver-2 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:27:34.602: INFO: The status of Pod netserver-2 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:27:36.601: INFO: The status of Pod netserver-2 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:27:38.602: INFO: The status of Pod netserver-2 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:27:40.603: INFO: The status of Pod netserver-2 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:27:42.602: INFO: The status of Pod netserver-2 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:27:44.601: INFO: The status of Pod netserver-2 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:27:46.602: INFO: The status of Pod netserver-2 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:27:48.601: INFO: The status of Pod netserver-2 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:27:50.602: INFO: The status of Pod netserver-2 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:27:52.601: INFO: The status of Pod netserver-2 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:27:54.605: INFO: The status of Pod netserver-2 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:27:56.601: INFO: The status of Pod netserver-2 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:27:58.601: INFO: The status of Pod netserver-2 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:28:00.603: INFO: The status of Pod netserver-2 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:28:02.601: INFO: The status of Pod netserver-2 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:28:04.602: INFO: The status of Pod netserver-2 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:28:06.602: INFO: The status of Pod netserver-2 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:28:08.601: INFO: The status of Pod netserver-2 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:28:10.602: INFO: The status of Pod netserver-2 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:28:12.601: INFO: The status of Pod netserver-2 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:28:14.602: INFO: The status of Pod netserver-2 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:28:16.603: INFO: The status of Pod netserver-2 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:28:18.601: INFO: The status of Pod netserver-2 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:28:20.602: INFO: The status of Pod netserver-2 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:28:22.601: INFO: The status of Pod netserver-2 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:28:24.602: INFO: The status of Pod netserver-2 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:28:26.601: INFO: The status of Pod netserver-2 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:28:28.601: INFO: The status of Pod netserver-2 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:28:30.601: INFO: The status of Pod netserver-2 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:28:32.602: INFO: The status of Pod netserver-2 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:28:34.601: INFO: The status of Pod netserver-2 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:28:36.603: INFO: The status of Pod netserver-2 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:28:38.606: INFO: The status of Pod netserver-2 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:28:40.601: INFO: The status of Pod netserver-2 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:28:42.602: INFO: The status of Pod netserver-2 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:28:44.602: INFO: The status of Pod netserver-2 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:28:46.602: INFO: The status of Pod netserver-2 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:28:48.601: INFO: The status of Pod netserver-2 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:28:50.602: INFO: The status of Pod netserver-2 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:28:52.602: INFO: The status of Pod netserver-2 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:28:54.603: INFO: The status of Pod netserver-2 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:28:56.603: INFO: The status of Pod netserver-2 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:28:58.601: INFO: The status of Pod netserver-2 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:29:00.603: INFO: The status of Pod netserver-2 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:29:02.602: INFO: The status of Pod netserver-2 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:29:04.603: INFO: The status of Pod netserver-2 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:29:06.601: INFO: The status of Pod netserver-2 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:29:08.601: INFO: The status of Pod netserver-2 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:29:10.603: INFO: The status of Pod netserver-2 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:29:12.601: INFO: The status of Pod netserver-2 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:29:14.601: INFO: The status of Pod netserver-2 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:29:16.601: INFO: The status of Pod netserver-2 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:29:18.602: INFO: The status of Pod netserver-2 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:29:20.603: INFO: The status of Pod netserver-2 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:29:22.601: INFO: The status of Pod netserver-2 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:29:24.602: INFO: The status of Pod netserver-2 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:29:26.601: INFO: The status of Pod netserver-2 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:29:28.601: INFO: The status of Pod netserver-2 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:29:30.601: INFO: The status of Pod netserver-2 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:29:32.602: INFO: The status of Pod netserver-2 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:29:34.601: INFO: The status of Pod netserver-2 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:29:36.601: INFO: The status of Pod netserver-2 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:29:38.602: INFO: The status of Pod netserver-2 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:29:40.602: INFO: The status of Pod netserver-2 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:29:42.602: INFO: The status of Pod netserver-2 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:29:44.602: INFO: The status of Pod netserver-2 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:29:46.601: INFO: The status of Pod netserver-2 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:29:48.602: INFO: The status of Pod netserver-2 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:29:50.601: INFO: The status of Pod netserver-2 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:29:52.601: INFO: The status of Pod netserver-2 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:29:54.601: INFO: The status of Pod netserver-2 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:29:56.601: INFO: The status of Pod netserver-2 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:29:58.601: INFO: The status of Pod netserver-2 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:30:00.602: INFO: The status of Pod netserver-2 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:30:02.602: INFO: The status of Pod netserver-2 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:30:04.601: INFO: The status of Pod netserver-2 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:30:06.602: INFO: The status of Pod netserver-2 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:30:08.601: INFO: The status of Pod netserver-2 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:30:10.602: INFO: The status of Pod netserver-2 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:30:12.602: INFO: The status of Pod netserver-2 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:30:14.603: INFO: The status of Pod netserver-2 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:30:16.602: INFO: The status of Pod netserver-2 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:30:18.602: INFO: The status of Pod netserver-2 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:30:20.601: INFO: The status of Pod netserver-2 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:30:22.602: INFO: The status of Pod netserver-2 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:30:24.602: INFO: The status of Pod netserver-2 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:30:26.602: INFO: The status of Pod netserver-2 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:30:28.602: INFO: The status of Pod netserver-2 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:30:30.603: INFO: The status of Pod netserver-2 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:30:32.601: INFO: The status of Pod netserver-2 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:30:34.601: INFO: The status of Pod netserver-2 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:30:36.601: INFO: The status of Pod netserver-2 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:30:38.601: INFO: The status of Pod netserver-2 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:30:40.602: INFO: The status of Pod netserver-2 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:30:42.601: INFO: The status of Pod netserver-2 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:30:44.603: INFO: The status of Pod netserver-2 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:30:46.601: INFO: The status of Pod netserver-2 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:30:48.601: INFO: The status of Pod netserver-2 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:30:50.602: INFO: The status of Pod netserver-2 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:30:52.601: INFO: The status of Pod netserver-2 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:30:54.602: INFO: The status of Pod netserver-2 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:30:56.601: INFO: The status of Pod netserver-2 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:30:58.601: INFO: The status of Pod netserver-2 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:31:00.601: INFO: The status of Pod netserver-2 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:31:02.601: INFO: The status of Pod netserver-2 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:31:04.603: INFO: The status of Pod netserver-2 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:31:06.602: INFO: The status of Pod netserver-2 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:31:08.602: INFO: The status of Pod netserver-2 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:31:10.603: INFO: The status of Pod netserver-2 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:31:12.602: INFO: The status of Pod netserver-2 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:31:14.603: INFO: The status of Pod netserver-2 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:31:16.602: INFO: The status of Pod netserver-2 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:31:18.601: INFO: The status of Pod netserver-2 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:31:20.602: INFO: The status of Pod netserver-2 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:31:20.706: INFO: The status of Pod netserver-2 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:31:20.707: FAIL: Unexpected error: <*errors.errorString | 0xc0002422b0>: { s: "timed out waiting for the condition", } timed out waiting for the condition occurred Full Stack Trace k8s.io/kubernetes/test/e2e/framework/network.(*NetworkingTestConfig).createNetProxyPods(0xc006b7e7e0, {0x70da7f5, 0x9}, 0xc003f402d0) /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/network/utils.go:858 +0x1d4 k8s.io/kubernetes/test/e2e/framework/network.(*NetworkingTestConfig).setupCore(0xc006b7e7e0, 0xc003f402d0) /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/network/utils.go:760 +0x5b k8s.io/kubernetes/test/e2e/framework/network.(*NetworkingTestConfig).setup(0xc006b7e7e0, 0x34) /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/network/utils.go:775 +0x3b k8s.io/kubernetes/test/e2e/framework/network.NewNetworkingTestConfig(0xc0011ecf20, {0xc006a43180, 0x1, 0x0}) /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/network/utils.go:129 +0x125 k8s.io/kubernetes/test/e2e/network.glob..func20.6.5() /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/network/networking.go:205 +0x51 k8s.io/kubernetes/test/e2e.RunE2ETests(0x24c66d7) _output/dockerized/go/src/k8s.io/kubernetes/test/e2e/e2e.go:133 +0x697 k8s.io/kubernetes/test/e2e.TestE2E(0x0) _output/dockerized/go/src/k8s.io/kubernetes/test/e2e/e2e_test.go:136 +0x19 testing.tRunner(0xc00037e820, 0x735e880) /usr/local/go/src/testing/testing.go:1259 +0x102 created by testing.(*T).Run /usr/local/go/src/testing/testing.go:1306 +0x35a [AfterEach] [sig-network] Networking /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:186 �[1mSTEP�[0m: Collecting events from namespace "nettest-5021". �[1mSTEP�[0m: Found 46 events. Jan 1 02:31:20.815: INFO: At 2023-01-01 02:25:27 +0000 UTC - event for netserver-0: {default-scheduler } Scheduled: Successfully assigned nettest-5021/netserver-0 to ip-172-20-33-66.eu-west-3.compute.internal Jan 1 02:31:20.815: INFO: At 2023-01-01 02:25:27 +0000 UTC - event for netserver-1: {default-scheduler } Scheduled: Successfully assigned nettest-5021/netserver-1 to ip-172-20-45-112.eu-west-3.compute.internal Jan 1 02:31:20.815: INFO: At 2023-01-01 02:25:27 +0000 UTC - event for netserver-2: {default-scheduler } Scheduled: Successfully assigned nettest-5021/netserver-2 to ip-172-20-47-178.eu-west-3.compute.internal Jan 1 02:31:20.815: INFO: At 2023-01-01 02:25:27 +0000 UTC - event for netserver-3: {default-scheduler } Scheduled: Successfully assigned nettest-5021/netserver-3 to ip-172-20-49-224.eu-west-3.compute.internal Jan 1 02:31:20.815: INFO: At 2023-01-01 02:25:28 +0000 UTC - event for netserver-0: {kubelet ip-172-20-33-66.eu-west-3.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "98eb16666a9a3507dc12792e604f003c6ddc2188194c526bf82f95f9942e5dd6" network for pod "netserver-0": networkPlugin cni failed to set up pod "netserver-0_nettest-5021" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 1 02:31:20.815: INFO: At 2023-01-01 02:25:29 +0000 UTC - event for netserver-0: {kubelet ip-172-20-33-66.eu-west-3.compute.internal} SandboxChanged: Pod sandbox changed, it will be killed and re-created. Jan 1 02:31:20.815: INFO: At 2023-01-01 02:25:29 +0000 UTC - event for netserver-3: {kubelet ip-172-20-49-224.eu-west-3.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "8d1fa9750d246f218b733227adb16f3c44012386e3c565ff79cd0cc568d5bd40" network for pod "netserver-3": networkPlugin cni failed to set up pod "netserver-3_nettest-5021" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 1 02:31:20.815: INFO: At 2023-01-01 02:25:30 +0000 UTC - event for netserver-1: {kubelet ip-172-20-45-112.eu-west-3.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "f244ef968cb86d1ca1df0891c352e45f03fd04828c60d5da9ef3473b7f707dc6" network for pod "netserver-1": networkPlugin cni failed to set up pod "netserver-1_nettest-5021" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 1 02:31:20.815: INFO: At 2023-01-01 02:25:30 +0000 UTC - event for netserver-2: {kubelet ip-172-20-47-178.eu-west-3.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "5483c0823faf8c345f1698f7c0e38bef6135248ced3c95004d2b62b5f64ddd8f" network for pod "netserver-2": networkPlugin cni failed to set up pod "netserver-2_nettest-5021" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 1 02:31:20.815: INFO: At 2023-01-01 02:25:30 +0000 UTC - event for netserver-3: {kubelet ip-172-20-49-224.eu-west-3.compute.internal} SandboxChanged: Pod sandbox changed, it will be killed and re-created. Jan 1 02:31:20.815: INFO: At 2023-01-01 02:25:31 +0000 UTC - event for netserver-0: {kubelet ip-172-20-33-66.eu-west-3.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "f7713f09fd4a78b99e22efdc719b140d4cdf87819265dd69f82857e914f4137a" network for pod "netserver-0": networkPlugin cni failed to set up pod "netserver-0_nettest-5021" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 1 02:31:20.815: INFO: At 2023-01-01 02:25:31 +0000 UTC - event for netserver-1: {kubelet ip-172-20-45-112.eu-west-3.compute.internal} SandboxChanged: Pod sandbox changed, it will be killed and re-created. Jan 1 02:31:20.815: INFO: At 2023-01-01 02:25:31 +0000 UTC - event for netserver-2: {kubelet ip-172-20-47-178.eu-west-3.compute.internal} SandboxChanged: Pod sandbox changed, it will be killed and re-created. Jan 1 02:31:20.815: INFO: At 2023-01-01 02:25:31 +0000 UTC - event for netserver-2: {kubelet ip-172-20-47-178.eu-west-3.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "301118ce6929a2d55c604d6be14eb00912b7451d85dbb023cf4cac9978549471" network for pod "netserver-2": networkPlugin cni failed to set up pod "netserver-2_nettest-5021" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 1 02:31:20.815: INFO: At 2023-01-01 02:25:32 +0000 UTC - event for netserver-3: {kubelet ip-172-20-49-224.eu-west-3.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "eae41dc290b4ac6fdeba052ee7f688061d0caabf380587a6e0d2d15e2fe58a40" network for pod "netserver-3": networkPlugin cni failed to set up pod "netserver-3_nettest-5021" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 1 02:31:20.815: INFO: At 2023-01-01 02:25:33 +0000 UTC - event for netserver-0: {kubelet ip-172-20-33-66.eu-west-3.compute.internal} Pulled: Container image "k8s.gcr.io/e2e-test-images/agnhost:2.39" already present on machine Jan 1 02:31:20.815: INFO: At 2023-01-01 02:25:33 +0000 UTC - event for netserver-0: {kubelet ip-172-20-33-66.eu-west-3.compute.internal} Created: Created container webserver Jan 1 02:31:20.815: INFO: At 2023-01-01 02:25:33 +0000 UTC - event for netserver-2: {kubelet ip-172-20-47-178.eu-west-3.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "61e8ad979bca25d46a0fc69ffe3213c6503a4c3420fa3f3a947f19f72d7e666c" network for pod "netserver-2": networkPlugin cni failed to set up pod "netserver-2_nettest-5021" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 1 02:31:20.815: INFO: At 2023-01-01 02:25:34 +0000 UTC - event for netserver-1: {kubelet ip-172-20-45-112.eu-west-3.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "33941acdc7345be3b9f3f71bc751a630723efadb6282d8f2d815bb4a9490d1d6" network for pod "netserver-1": networkPlugin cni failed to set up pod "netserver-1_nettest-5021" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 1 02:31:20.815: INFO: At 2023-01-01 02:25:35 +0000 UTC - event for netserver-0: {kubelet ip-172-20-33-66.eu-west-3.compute.internal} Started: Started container webserver Jan 1 02:31:20.815: INFO: At 2023-01-01 02:25:35 +0000 UTC - event for netserver-2: {kubelet ip-172-20-47-178.eu-west-3.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "6178aa73d8b4cb35a9c21ecdfd10437d3feb3cbd9a8435a8e0701f52049cdf4d" network for pod "netserver-2": networkPlugin cni failed to set up pod "netserver-2_nettest-5021" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 1 02:31:20.815: INFO: At 2023-01-01 02:25:35 +0000 UTC - event for netserver-3: {kubelet ip-172-20-49-224.eu-west-3.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "7b947ba02f53b3c2a9db5738cf57163dc82e44089e94a944460a97982af60198" network for pod "netserver-3": networkPlugin cni failed to set up pod "netserver-3_nettest-5021" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 1 02:31:20.815: INFO: At 2023-01-01 02:25:38 +0000 UTC - event for netserver-1: {kubelet ip-172-20-45-112.eu-west-3.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "9d28c743838b72570d6458b2c58f829e492fda6bfc4fe94744f53b35cffd38f9" network for pod "netserver-1": networkPlugin cni failed to set up pod "netserver-1_nettest-5021" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 1 02:31:20.815: INFO: At 2023-01-01 02:25:38 +0000 UTC - event for netserver-2: {kubelet ip-172-20-47-178.eu-west-3.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "c91f3ae7ae707b179aa7ef5931779a907e3bb3f26e060adcae687a1cbef1a599" network for pod "netserver-2": networkPlugin cni failed to set up pod "netserver-2_nettest-5021" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 1 02:31:20.815: INFO: At 2023-01-01 02:25:38 +0000 UTC - event for netserver-3: {kubelet ip-172-20-49-224.eu-west-3.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "930216a40187354e3162f7deed9f489acb3cce62741ec4b7b026c92d6391924c" network for pod "netserver-3": networkPlugin cni failed to set up pod "netserver-3_nettest-5021" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 1 02:31:20.815: INFO: At 2023-01-01 02:25:39 +0000 UTC - event for netserver-1: {kubelet ip-172-20-45-112.eu-west-3.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "8a86c7322cbefb76e41437e4718dc6d9523942bf9df8f076de82b26821e0e30d" network for pod "netserver-1": networkPlugin cni failed to set up pod "netserver-1_nettest-5021" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 1 02:31:20.815: INFO: At 2023-01-01 02:25:40 +0000 UTC - event for netserver-2: {kubelet ip-172-20-47-178.eu-west-3.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "abc51985df3b8aa50638ff640b827b02f3f5bb1c0933fc8908c797d376e065cc" network for pod "netserver-2": networkPlugin cni failed to set up pod "netserver-2_nettest-5021" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 1 02:31:20.815: INFO: At 2023-01-01 02:25:40 +0000 UTC - event for netserver-3: {kubelet ip-172-20-49-224.eu-west-3.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "8b771aa35ffc6726d0db5deebf4f7c23253c33878f3cf2292fa536729f980e88" network for pod "netserver-3": networkPlugin cni failed to set up pod "netserver-3_nettest-5021" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 1 02:31:20.815: INFO: At 2023-01-01 02:25:41 +0000 UTC - event for netserver-1: {kubelet ip-172-20-45-112.eu-west-3.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "e018332a40ec40025a7cb0e80fe5249f487df78d75dc34cefd78321805ad4a08" network for pod "netserver-1": networkPlugin cni failed to set up pod "netserver-1_nettest-5021" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 1 02:31:20.815: INFO: At 2023-01-01 02:25:41 +0000 UTC - event for netserver-2: {kubelet ip-172-20-47-178.eu-west-3.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "5ec76919e93bdad9b0eaf5689fb014ca6b6fc8df0e9352a88563bbd9e1ddbf0d" network for pod "netserver-2": networkPlugin cni failed to set up pod "netserver-2_nettest-5021" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 1 02:31:20.815: INFO: At 2023-01-01 02:25:42 +0000 UTC - event for netserver-2: {kubelet ip-172-20-47-178.eu-west-3.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "d1b8894dc2229d172bc00c24d576b44a041162baf579321891d832fc7af5f21a" network for pod "netserver-2": networkPlugin cni failed to set up pod "netserver-2_nettest-5021" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 1 02:31:20.815: INFO: At 2023-01-01 02:25:43 +0000 UTC - event for netserver-1: {kubelet ip-172-20-45-112.eu-west-3.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "81cb9306bd892c5de19594ca35e7671f110197355664dd5da8f4e8a421a3ce2b" network for pod "netserver-1": networkPlugin cni failed to set up pod "netserver-1_nettest-5021" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 1 02:31:20.815: INFO: At 2023-01-01 02:25:43 +0000 UTC - event for netserver-3: {kubelet ip-172-20-49-224.eu-west-3.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "39a64495f568fd11a574d5886c28c37c0946bfa96a3a02f724c5c364b1a96239" network for pod "netserver-3": networkPlugin cni failed to set up pod "netserver-3_nettest-5021" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 1 02:31:20.815: INFO: At 2023-01-01 02:25:44 +0000 UTC - event for netserver-2: {kubelet ip-172-20-47-178.eu-west-3.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "aee801b8746d6c241c57d955d46047453b265095f9ea0667933dfe3f4578c794" network for pod "netserver-2": networkPlugin cni failed to set up pod "netserver-2_nettest-5021" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 1 02:31:20.815: INFO: At 2023-01-01 02:25:45 +0000 UTC - event for netserver-1: {kubelet ip-172-20-45-112.eu-west-3.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "eb6923bb1e8cec3b376bab1c663314dfb282e4f03c27d2003f69b41448d70547" network for pod "netserver-1": networkPlugin cni failed to set up pod "netserver-1_nettest-5021" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 1 02:31:20.815: INFO: At 2023-01-01 02:25:45 +0000 UTC - event for netserver-3: {kubelet ip-172-20-49-224.eu-west-3.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "c05e86e3022ad792c6f33c724bf41ed43f33a9fe88690d1cee363a779e085ae2" network for pod "netserver-3": networkPlugin cni failed to set up pod "netserver-3_nettest-5021" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 1 02:31:20.815: INFO: At 2023-01-01 02:25:48 +0000 UTC - event for netserver-3: {kubelet ip-172-20-49-224.eu-west-3.compute.internal} Pulled: Container image "k8s.gcr.io/e2e-test-images/agnhost:2.39" already present on machine Jan 1 02:31:20.815: INFO: At 2023-01-01 02:25:48 +0000 UTC - event for netserver-3: {kubelet ip-172-20-49-224.eu-west-3.compute.internal} Created: Created container webserver Jan 1 02:31:20.815: INFO: At 2023-01-01 02:25:49 +0000 UTC - event for netserver-1: {kubelet ip-172-20-45-112.eu-west-3.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "5b36c022f54590406ace738aafed79ebb73ff893211779153a74642f2c4a992c" network for pod "netserver-1": networkPlugin cni failed to set up pod "netserver-1_nettest-5021" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 1 02:31:20.815: INFO: At 2023-01-01 02:25:49 +0000 UTC - event for netserver-3: {kubelet ip-172-20-49-224.eu-west-3.compute.internal} Started: Started container webserver Jan 1 02:31:20.815: INFO: At 2023-01-01 02:25:53 +0000 UTC - event for netserver-1: {kubelet ip-172-20-45-112.eu-west-3.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "807a38b300e5ef4a731a496c92b3a80b4ddf3c14d843ef3a55e250778f7614e1" network for pod "netserver-1": networkPlugin cni failed to set up pod "netserver-1_nettest-5021" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 1 02:31:20.815: INFO: At 2023-01-01 02:25:53 +0000 UTC - event for netserver-2: {kubelet ip-172-20-47-178.eu-west-3.compute.internal} FailedCreatePodSandBox: (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "d6953e55f25db0f040e651a5905e6753dd3da59c00dc40718da1abc20942b2f8" network for pod "netserver-2": networkPlugin cni failed to set up pod "netserver-2_nettest-5021" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 1 02:31:20.815: INFO: At 2023-01-01 02:26:01 +0000 UTC - event for netserver-1: {kubelet ip-172-20-45-112.eu-west-3.compute.internal} Pulled: Container image "k8s.gcr.io/e2e-test-images/agnhost:2.39" already present on machine Jan 1 02:31:20.815: INFO: At 2023-01-01 02:26:01 +0000 UTC - event for netserver-1: {kubelet ip-172-20-45-112.eu-west-3.compute.internal} Created: Created container webserver Jan 1 02:31:20.815: INFO: At 2023-01-01 02:26:01 +0000 UTC - event for netserver-1: {kubelet ip-172-20-45-112.eu-west-3.compute.internal} Started: Started container webserver Jan 1 02:31:20.815: INFO: At 2023-01-01 02:26:02 +0000 UTC - event for netserver-1: {kubelet ip-172-20-45-112.eu-west-3.compute.internal} FailedCreatePodSandBox: (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "56158c3cd4af4c8907cc6b091ecb04a84973d086199b014cbf31bb425a7c5f85" network for pod "netserver-1": networkPlugin cni failed to set up pod "netserver-1_nettest-5021" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 1 02:31:20.966: INFO: POD NODE PHASE GRACE CONDITIONS Jan 1 02:31:20.966: INFO: netserver-0 ip-172-20-33-66.eu-west-3.compute.internal Running [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2023-01-01 02:25:27 +0000 UTC } {Ready True 0001-01-01 00:00:00 +0000 UTC 2023-01-01 02:25:47 +0000 UTC } {ContainersReady True 0001-01-01 00:00:00 +0000 UTC 2023-01-01 02:25:47 +0000 UTC } {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2023-01-01 02:25:27 +0000 UTC }] Jan 1 02:31:20.966: INFO: netserver-1 ip-172-20-45-112.eu-west-3.compute.internal Running [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2023-01-01 02:25:27 +0000 UTC } {Ready True 0001-01-01 00:00:00 +0000 UTC 2023-01-01 02:26:18 +0000 UTC } {ContainersReady True 0001-01-01 00:00:00 +0000 UTC 2023-01-01 02:26:18 +0000 UTC } {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2023-01-01 02:25:27 +0000 UTC }] Jan 1 02:31:20.966: INFO: netserver-2 ip-172-20-47-178.eu-west-3.compute.internal Pending [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2023-01-01 02:25:27 +0000 UTC } {Ready False 0001-01-01 00:00:00 +0000 UTC 2023-01-01 02:25:27 +0000 UTC ContainersNotReady containers with unready status: [webserver]} {ContainersReady False 0001-01-01 00:00:00 +0000 UTC 2023-01-01 02:25:27 +0000 UTC ContainersNotReady containers with unready status: [webserver]} {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2023-01-01 02:25:27 +0000 UTC }] Jan 1 02:31:20.966: INFO: netserver-3 ip-172-20-49-224.eu-west-3.compute.internal Running [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2023-01-01 02:25:27 +0000 UTC } {Ready True 0001-01-01 00:00:00 +0000 UTC 2023-01-01 02:26:08 +0000 UTC } {ContainersReady True 0001-01-01 00:00:00 +0000 UTC 2023-01-01 02:26:08 +0000 UTC } {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2023-01-01 02:25:27 +0000 UTC }] Jan 1 02:31:20.966: INFO: Jan 1 02:31:21.303: INFO: Unable to fetch nettest-5021/netserver-2/webserver logs: the server rejected our request for an unknown reason (get pods netserver-2) Jan 1 02:31:21.548: INFO: Logging node info for node ip-172-20-33-66.eu-west-3.compute.internal Jan 1 02:31:21.652: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-33-66.eu-west-3.compute.internal ec4b59b3-62db-4040-aad2-4c3249f80fe8 9096 0 2023-01-01 02:20:25 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:eu-west-3 failure-domain.beta.kubernetes.io/zone:eu-west-3a io.kubernetes.storage.mock/node:some-mock-node kubelet_cleanup:true kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-33-66.eu-west-3.compute.internal kubernetes.io/os:linux kubernetes.io/role:node node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:eu-west-3a topology.kubernetes.io/region:eu-west-3 topology.kubernetes.io/zone:eu-west-3a] map[csi.volume.kubernetes.io/nodeid:{"csi-mock-csi-mock-volumes-8129":"csi-mock-csi-mock-volumes-8129","ebs.csi.aws.com":"i-028313673124ac4ad"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kops-controller Update v1 2023-01-01 02:20:25 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}} } {kube-controller-manager Update v1 2023-01-01 02:20:25 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.1.0/24\"":{}}}} } {kubelet Update v1 2023-01-01 02:20:25 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {e2e.test Update v1 2023-01-01 02:25:15 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubelet_cleanup":{}}}} } {kubelet Update v1 2023-01-01 02:31:10 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:io.kubernetes.storage.mock/node":{},"f:topology.ebs.csi.aws.com/zone":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.1.0/24,DoNotUseExternalID:,ProviderID:aws:///eu-west-3a/i-028313673124ac4ad,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.1.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{50478186496 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4078190592 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{45430367772 0} {<nil>} 45430367772 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3973332992 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-01-01 02:30:38 +0000 UTC,LastTransitionTime:2023-01-01 02:20:25 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-01-01 02:30:38 +0000 UTC,LastTransitionTime:2023-01-01 02:20:25 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-01-01 02:30:38 +0000 UTC,LastTransitionTime:2023-01-01 02:20:25 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-01-01 02:30:38 +0000 UTC,LastTransitionTime:2023-01-01 02:20:46 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.33.66,},NodeAddress{Type:ExternalIP,Address:52.47.140.112,},NodeAddress{Type:Hostname,Address:ip-172-20-33-66.eu-west-3.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-33-66.eu-west-3.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-52-47-140-112.eu-west-3.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2ad2ce1a87d508c178acf6215c5767,SystemUUID:ec2ad2ce-1a87-d508-c178-acf6215c5767,BootID:b3b60783-d786-412b-a36a-802bcfae604d,KernelVersion:4.19.0-23-cloud-amd64,OSImage:Debian GNU/Linux 10 (buster),ContainerRuntimeVersion:docker://20.10.17,KubeletVersion:v1.23.15,KubeProxyVersion:v1.23.15,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:456569528,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/jessie-dnsutils@sha256:11e6a66017ba4e4b938c1612b7a54a3befcefd354796c04e1dba76873a13518e k8s.gcr.io/e2e-test-images/jessie-dnsutils:1.5],SizeBytes:253371792,},ContainerImage{Names:[nginx@sha256:0047b729188a15da49380d9506d65959cce6d40291ccfb4e039f5dc7efd33286 nginx:latest],SizeBytes:141812353,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:7e8bdd271312fd25fc5ff5a8f04727be84044eb3d7d8d03611972a6752e2e11e k8s.gcr.io/e2e-test-images/agnhost:2.39],SizeBytes:126872991,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:1b9d1b2f36cb2dbee1960e82a9344aeb11bd4c4c03abf5e1853e0559c23855e3 k8s.gcr.io/e2e-test-images/httpd:2.4.38-2],SizeBytes:123781643,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.23.15 registry.k8s.io/kube-proxy-amd64:v1.23.15],SizeBytes:112336036,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:10e231cf07c82dfab5141c1afe548dc734e5fc1f67665ec3982a325d1bd31a9a registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.0],SizeBytes:94471559,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:c8e03f60afa90a28e4bb6ec9a8d0fc36d89de4b7475cf2d613afa793ec969fe0 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.0],SizeBytes:56382329,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:c5be65d6679efabb969d9b019300d187437ae876f992c40911fd2892bbef3b36 k8s.gcr.io/sig-storage/csi-attacher:v3.2.0],SizeBytes:53500826,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:017727efcfeb7d053af68e51436ce8e65edbc6ca573720afb4f79c8594036955 registry.k8s.io/coredns/coredns:v1.10.0],SizeBytes:50249337,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:a61d309da54641db41fb8f35718f744e9f730d4d0384f8c4b186ddc9f06cbd5f k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.1.0],SizeBytes:19662887,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:19582112,},ContainerImage{Names:[k8s.gcr.io/sig-storage/mock-driver@sha256:a7b517f9e0f42ffade802eef9cefa271372386b85f55b702b493241e58459793 k8s.gcr.io/sig-storage/mock-driver:v4.1.0],SizeBytes:17680993,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:17611980,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:13616070e3f29de4417eee434a8ef472221c9e51b3d037b5a6b46cef08eb7443 k8s.gcr.io/e2e-test-images/nginx:1.14-2],SizeBytes:16032814,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nonewprivs@sha256:8ac1264691820febacf3aea5d152cbde6d10685731ec14966a9401c6f47a68ac k8s.gcr.io/e2e-test-images/nonewprivs:1.3],SizeBytes:7107254,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf k8s.gcr.io/e2e-test-images/busybox:1.29-2],SizeBytes:1154361,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db k8s.gcr.io/pause:3.6 registry.k8s.io/pause:3.6],SizeBytes:682696,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Jan 1 02:31:21.653: INFO: Logging kubelet events for node ip-172-20-33-66.eu-west-3.compute.internal Jan 1 02:31:21.760: INFO: Logging pods the kubelet thinks is on node ip-172-20-33-66.eu-west-3.compute.internal Jan 1 02:31:22.003: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-546bc started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:22.003: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:31:22.003: INFO: csi-mockplugin-attacher-0 started at 2023-01-01 02:27:56 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:22.003: INFO: Container csi-attacher ready: false, restart count 0 Jan 1 02:31:22.003: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-q8pgp started at 2023-01-01 02:25:17 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:22.003: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:31:22.003: INFO: csi-mockplugin-0 started at 2023-01-01 02:27:56 +0000 UTC (0+3 container statuses recorded) Jan 1 02:31:22.003: INFO: Container csi-provisioner ready: false, restart count 0 Jan 1 02:31:22.003: INFO: Container driver-registrar ready: false, restart count 0 Jan 1 02:31:22.003: INFO: Container mock ready: false, restart count 0 Jan 1 02:31:22.003: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-2v87j started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:22.003: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:31:22.003: INFO: coredns-867df8f45c-cqmp7 started at 2023-01-01 02:21:18 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:22.003: INFO: Container coredns ready: true, restart count 0 Jan 1 02:31:22.003: INFO: netserver-0 started at 2023-01-01 02:25:27 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:22.003: INFO: Container webserver ready: true, restart count 0 Jan 1 02:31:22.003: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-l9hnv started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:22.003: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:31:22.003: INFO: local-client started at 2023-01-01 02:31:07 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:22.003: INFO: Container local-client ready: false, restart count 0 Jan 1 02:31:22.003: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-vnv2x started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:22.003: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:31:22.003: INFO: csi-mockplugin-0 started at 2023-01-01 02:26:15 +0000 UTC (0+4 container statuses recorded) Jan 1 02:31:22.003: INFO: Container busybox ready: true, restart count 0 Jan 1 02:31:22.003: INFO: Container csi-provisioner ready: true, restart count 0 Jan 1 02:31:22.003: INFO: Container driver-registrar ready: true, restart count 0 Jan 1 02:31:22.003: INFO: Container mock ready: true, restart count 0 Jan 1 02:31:22.003: INFO: hostexec-ip-172-20-33-66.eu-west-3.compute.internal-qm7jt started at 2023-01-01 02:27:26 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:22.003: INFO: Container agnhost-container ready: true, restart count 0 Jan 1 02:31:22.003: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-csnzw started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:22.003: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: false, restart count 0 Jan 1 02:31:22.003: INFO: ebs-csi-node-vsxjc started at 2023-01-01 02:20:26 +0000 UTC (0+3 container statuses recorded) Jan 1 02:31:22.003: INFO: Container ebs-plugin ready: true, restart count 0 Jan 1 02:31:22.003: INFO: Container liveness-probe ready: true, restart count 0 Jan 1 02:31:22.003: INFO: Container node-driver-registrar ready: true, restart count 0 Jan 1 02:31:22.003: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-h4nvd started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:22.003: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:31:22.003: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-h6mmt started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:22.003: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:31:22.003: INFO: cilium-vbcgc started at 2023-01-01 02:20:26 +0000 UTC (1+1 container statuses recorded) Jan 1 02:31:22.003: INFO: Init container clean-cilium-state ready: true, restart count 0 Jan 1 02:31:22.003: INFO: Container cilium-agent ready: true, restart count 0 Jan 1 02:31:22.003: INFO: affinity-clusterip-timeout-q9qt2 started at 2023-01-01 02:27:32 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:22.003: INFO: Container affinity-clusterip-timeout ready: false, restart count 0 Jan 1 02:31:22.003: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-m6vq2 started at 2023-01-01 02:25:17 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:22.003: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:31:22.003: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-dkfpd started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:22.003: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:31:22.421: INFO: Latency metrics for node ip-172-20-33-66.eu-west-3.compute.internal Jan 1 02:31:22.421: INFO: Logging node info for node ip-172-20-38-238.eu-west-3.compute.internal Jan 1 02:31:22.526: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-38-238.eu-west-3.compute.internal 0eb713b4-64bf-47b7-8090-bab8111218ee 6771 0 2023-01-01 02:18:51 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:c5.large beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:eu-west-3 failure-domain.beta.kubernetes.io/zone:eu-west-3a kops.k8s.io/kops-controller-pki: kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-38-238.eu-west-3.compute.internal kubernetes.io/os:linux kubernetes.io/role:master node-role.kubernetes.io/control-plane: node-role.kubernetes.io/master: node.kubernetes.io/exclude-from-external-load-balancers: node.kubernetes.io/instance-type:c5.large topology.ebs.csi.aws.com/zone:eu-west-3a topology.kubernetes.io/region:eu-west-3 topology.kubernetes.io/zone:eu-west-3a] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-0f70ebb30a9076648"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kubelet Update v1 2023-01-01 02:18:50 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {protokube Update v1 2023-01-01 02:19:16 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kops.k8s.io/kops-controller-pki":{},"f:node-role.kubernetes.io/control-plane":{},"f:node.kubernetes.io/exclude-from-external-load-balancers":{}}}} } {kops-controller Update v1 2023-01-01 02:19:21 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/master":{}}}} } {kube-controller-manager Update v1 2023-01-01 02:19:31 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.0.0/24\"":{}},"f:taints":{}}} } {kubelet Update v1 2023-01-01 02:20:36 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.0.0/24,DoNotUseExternalID:,ProviderID:aws:///eu-west-3a/i-0f70ebb30a9076648,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/master,Value:,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[100.96.0.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{50478186496 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3918811136 0} {<nil>} 3826964Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{45430367772 0} {<nil>} 45430367772 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3813953536 0} {<nil>} 3724564Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-01-01 02:26:20 +0000 UTC,LastTransitionTime:2023-01-01 02:18:45 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-01-01 02:26:20 +0000 UTC,LastTransitionTime:2023-01-01 02:18:45 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-01-01 02:26:20 +0000 UTC,LastTransitionTime:2023-01-01 02:18:45 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-01-01 02:26:20 +0000 UTC,LastTransitionTime:2023-01-01 02:19:31 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.38.238,},NodeAddress{Type:ExternalIP,Address:15.237.26.58,},NodeAddress{Type:Hostname,Address:ip-172-20-38-238.eu-west-3.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-38-238.eu-west-3.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-15-237-26-58.eu-west-3.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2e6c64b5b811bf0c78b2aa7ccdb943,SystemUUID:ec2e6c64-b5b8-11bf-0c78-b2aa7ccdb943,BootID:9db01fa3-d136-4230-b21a-b03abbfb6bff,KernelVersion:4.19.0-23-cloud-amd64,OSImage:Debian GNU/Linux 10 (buster),ContainerRuntimeVersion:docker://20.10.17,KubeletVersion:v1.23.15,KubeProxyVersion:v1.23.15,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[registry.k8s.io/etcdadm/etcd-manager@sha256:66a453db625abb268f4b3bbefc5a34a171d81e6e8796cecca54cfd71775c77c4 registry.k8s.io/etcdadm/etcd-manager:v3.0.20221209],SizeBytes:662209237,},ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:456569528,},ContainerImage{Names:[k8s.gcr.io/kube-apiserver-amd64:v1.23.15 registry.k8s.io/kube-apiserver-amd64:v1.23.15],SizeBytes:135240444,},ContainerImage{Names:[k8s.gcr.io/kube-controller-manager-amd64:v1.23.15 registry.k8s.io/kube-controller-manager-amd64:v1.23.15],SizeBytes:125046097,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.23.15 registry.k8s.io/kube-proxy-amd64:v1.23.15],SizeBytes:112336036,},ContainerImage{Names:[quay.io/cilium/operator@sha256:a6d24a006a6b92967ac90786b49bc1ac26e5477cf028cd1186efcfc2466484db quay.io/cilium/operator:v1.12.5],SizeBytes:110016956,},ContainerImage{Names:[registry.k8s.io/kops/kops-controller:1.27.0-alpha.1],SizeBytes:105197757,},ContainerImage{Names:[registry.k8s.io/kops/dns-controller:1.27.0-alpha.1],SizeBytes:102577531,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:10e231cf07c82dfab5141c1afe548dc734e5fc1f67665ec3982a325d1bd31a9a registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.0],SizeBytes:94471559,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:122bfb8c1edabb3c0edd63f06523e6940d958d19b3957dc7b1d6f81e9f1f6119 registry.k8s.io/sig-storage/csi-provisioner:v3.1.0],SizeBytes:57736070,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:9ebbf9f023e7b41ccee3d52afe39a89e3ddacdbb69269d583abfc25847cfd9e4 registry.k8s.io/sig-storage/csi-resizer:v1.4.0],SizeBytes:55464778,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:8b9c313c05f54fb04f8d430896f5f5904b6cb157df261501b29adc04d2b2dc7b registry.k8s.io/sig-storage/csi-attacher:v3.4.0],SizeBytes:54848595,},ContainerImage{Names:[k8s.gcr.io/kube-scheduler-amd64:v1.23.15 registry.k8s.io/kube-scheduler-amd64:v1.23.15],SizeBytes:53521430,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:19582112,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:17611980,},ContainerImage{Names:[registry.k8s.io/kops/kube-apiserver-healthcheck:1.27.0-alpha.1],SizeBytes:8787111,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:682696,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Jan 1 02:31:22.527: INFO: Logging kubelet events for node ip-172-20-38-238.eu-west-3.compute.internal Jan 1 02:31:22.633: INFO: Logging pods the kubelet thinks is on node ip-172-20-38-238.eu-west-3.compute.internal Jan 1 02:31:22.746: INFO: kops-controller-26gzg started at 2023-01-01 02:19:20 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:22.746: INFO: Container kops-controller ready: true, restart count 0 Jan 1 02:31:22.746: INFO: ebs-csi-controller-6dc545b75-5qqh8 started at 2023-01-01 02:19:20 +0000 UTC (0+5 container statuses recorded) Jan 1 02:31:22.746: INFO: Container csi-attacher ready: true, restart count 0 Jan 1 02:31:22.746: INFO: Container csi-provisioner ready: true, restart count 0 Jan 1 02:31:22.746: INFO: Container csi-resizer ready: true, restart count 0 Jan 1 02:31:22.746: INFO: Container ebs-plugin ready: true, restart count 0 Jan 1 02:31:22.746: INFO: Container liveness-probe ready: true, restart count 0 Jan 1 02:31:22.746: INFO: kube-controller-manager-ip-172-20-38-238.eu-west-3.compute.internal started at 2023-01-01 02:18:11 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:22.746: INFO: Container kube-controller-manager ready: true, restart count 2 Jan 1 02:31:22.746: INFO: kube-scheduler-ip-172-20-38-238.eu-west-3.compute.internal started at 2023-01-01 02:18:11 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:22.746: INFO: Container kube-scheduler ready: true, restart count 0 Jan 1 02:31:22.746: INFO: kube-apiserver-ip-172-20-38-238.eu-west-3.compute.internal started at 2023-01-01 02:18:11 +0000 UTC (0+2 container statuses recorded) Jan 1 02:31:22.746: INFO: Container healthcheck ready: true, restart count 0 Jan 1 02:31:22.746: INFO: Container kube-apiserver ready: true, restart count 1 Jan 1 02:31:22.746: INFO: ebs-csi-node-k87k6 started at 2023-01-01 02:19:20 +0000 UTC (0+3 container statuses recorded) Jan 1 02:31:22.746: INFO: Container ebs-plugin ready: true, restart count 0 Jan 1 02:31:22.746: INFO: Container liveness-probe ready: true, restart count 0 Jan 1 02:31:22.746: INFO: Container node-driver-registrar ready: true, restart count 0 Jan 1 02:31:22.746: INFO: dns-controller-7db8b74b99-58zzw started at 2023-01-01 02:19:20 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:22.746: INFO: Container dns-controller ready: true, restart count 0 Jan 1 02:31:22.746: INFO: cilium-operator-d47487c76-j84nq started at 2023-01-01 02:19:20 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:22.746: INFO: Container cilium-operator ready: true, restart count 0 Jan 1 02:31:22.746: INFO: etcd-manager-events-ip-172-20-38-238.eu-west-3.compute.internal started at 2023-01-01 02:18:11 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:22.746: INFO: Container etcd-manager ready: true, restart count 0 Jan 1 02:31:22.746: INFO: etcd-manager-main-ip-172-20-38-238.eu-west-3.compute.internal started at 2023-01-01 02:18:11 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:22.746: INFO: Container etcd-manager ready: true, restart count 0 Jan 1 02:31:22.746: INFO: cilium-qk59v started at 2023-01-01 02:19:20 +0000 UTC (1+1 container statuses recorded) Jan 1 02:31:22.746: INFO: Init container clean-cilium-state ready: true, restart count 0 Jan 1 02:31:22.746: INFO: Container cilium-agent ready: true, restart count 0 Jan 1 02:31:23.079: INFO: Latency metrics for node ip-172-20-38-238.eu-west-3.compute.internal Jan 1 02:31:23.079: INFO: Logging node info for node ip-172-20-45-112.eu-west-3.compute.internal Jan 1 02:31:23.183: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-45-112.eu-west-3.compute.internal 48eed937-05bb-4d8a-bddd-aa94a2e6ab92 9109 0 2023-01-01 02:20:27 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:eu-west-3 failure-domain.beta.kubernetes.io/zone:eu-west-3a kubelet_cleanup:true kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-45-112.eu-west-3.compute.internal kubernetes.io/os:linux kubernetes.io/role:node node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:eu-west-3a topology.hostpath.csi/node:ip-172-20-45-112.eu-west-3.compute.internal topology.kubernetes.io/region:eu-west-3 topology.kubernetes.io/zone:eu-west-3a] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-0778adf2085fa7f34"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kube-controller-manager Update v1 2023-01-01 02:20:27 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.3.0/24\"":{}}}} } {kubelet Update v1 2023-01-01 02:20:27 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {kops-controller Update v1 2023-01-01 02:20:28 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}} } {e2e.test Update v1 2023-01-01 02:25:15 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubelet_cleanup":{}}}} } {kube-controller-manager Update v1 2023-01-01 02:30:49 +0000 UTC FieldsV1 {"f:status":{"f:volumesAttached":{}}} status} {kubelet Update v1 2023-01-01 02:30:52 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:capacity":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.3.0/24,DoNotUseExternalID:,ProviderID:aws:///eu-west-3a/i-0778adf2085fa7f34,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.3.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{50478186496 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4078190592 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{45430367772 0} {<nil>} 45430367772 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3973332992 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-01-01 02:31:12 +0000 UTC,LastTransitionTime:2023-01-01 02:20:27 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-01-01 02:31:12 +0000 UTC,LastTransitionTime:2023-01-01 02:20:27 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-01-01 02:31:12 +0000 UTC,LastTransitionTime:2023-01-01 02:20:27 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-01-01 02:31:12 +0000 UTC,LastTransitionTime:2023-01-01 02:20:48 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.45.112,},NodeAddress{Type:ExternalIP,Address:13.38.32.14,},NodeAddress{Type:Hostname,Address:ip-172-20-45-112.eu-west-3.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-45-112.eu-west-3.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-13-38-32-14.eu-west-3.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2dbdf846e8ff614f4555e5bf9dbbf4,SystemUUID:ec2dbdf8-46e8-ff61-4f45-55e5bf9dbbf4,BootID:10a054ae-621a-47ce-9566-b0048cc5101b,KernelVersion:4.19.0-23-cloud-amd64,OSImage:Debian GNU/Linux 10 (buster),ContainerRuntimeVersion:docker://20.10.17,KubeletVersion:v1.23.15,KubeProxyVersion:v1.23.15,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:456569528,},ContainerImage{Names:[nginx@sha256:0047b729188a15da49380d9506d65959cce6d40291ccfb4e039f5dc7efd33286 nginx:latest],SizeBytes:141812353,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:7e8bdd271312fd25fc5ff5a8f04727be84044eb3d7d8d03611972a6752e2e11e k8s.gcr.io/e2e-test-images/agnhost:2.39],SizeBytes:126872991,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:1b9d1b2f36cb2dbee1960e82a9344aeb11bd4c4c03abf5e1853e0559c23855e3 k8s.gcr.io/e2e-test-images/httpd:2.4.38-2],SizeBytes:123781643,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.23.15 registry.k8s.io/kube-proxy-amd64:v1.23.15],SizeBytes:112336036,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:10e231cf07c82dfab5141c1afe548dc734e5fc1f67665ec3982a325d1bd31a9a registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.0],SizeBytes:94471559,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:c8e03f60afa90a28e4bb6ec9a8d0fc36d89de4b7475cf2d613afa793ec969fe0 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.0],SizeBytes:56382329,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:4e74c0492bceddc598de1c90cc5bc14dcda94cb49fa9c5bad9d117c4834b5e08 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.1],SizeBytes:56379269,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:36c31f7e1f433c9634d24f876353e8646246d81a03c4e351202c2644daff1620 k8s.gcr.io/sig-storage/csi-resizer:v1.2.0],SizeBytes:53956308,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:ed98431376c9e944e19a465fe8ea944806714dd95416a0821096c78d66b579bd k8s.gcr.io/sig-storage/csi-snapshotter:v4.1.1],SizeBytes:53531990,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:c5be65d6679efabb969d9b019300d187437ae876f992c40911fd2892bbef3b36 k8s.gcr.io/sig-storage/csi-attacher:v3.2.0],SizeBytes:53500826,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:60ab9b3e6a030d3038c87c0d6bca2930f58d1d72823e6a4af09767dc83b696a2 k8s.gcr.io/sig-storage/csi-attacher:v3.2.1],SizeBytes:53499570,},ContainerImage{Names:[k8s.gcr.io/sig-storage/hostpathplugin@sha256:232fe80174d60d520d36043103853a1d7ab4b7f3782cf43e45034f04ccda58ce k8s.gcr.io/sig-storage/hostpathplugin:v1.7.1],SizeBytes:29977921,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:a61d309da54641db41fb8f35718f744e9f730d4d0384f8c4b186ddc9f06cbd5f k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.1.0],SizeBytes:19662887,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:19582112,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:2dee3fe5fe861bb66c3a4ac51114f3447a4cd35870e0f2e2b558c7a400d89589 k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.2.0],SizeBytes:18699436,},ContainerImage{Names:[k8s.gcr.io/sig-storage/mock-driver@sha256:a7b517f9e0f42ffade802eef9cefa271372386b85f55b702b493241e58459793 k8s.gcr.io/sig-storage/mock-driver:v4.1.0],SizeBytes:17680993,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:17611980,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:529be2c9770add0cdd0c989115222ea9fc1be430c11095eb9f6dafcf98a36e2b k8s.gcr.io/sig-storage/livenessprobe:v2.4.0],SizeBytes:17175488,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:13616070e3f29de4417eee434a8ef472221c9e51b3d037b5a6b46cef08eb7443 k8s.gcr.io/e2e-test-images/nginx:1.14-2],SizeBytes:16032814,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf k8s.gcr.io/e2e-test-images/busybox:1.29-2],SizeBytes:1154361,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:39e1e963e5310e9c313bad51523be012ede7b35bb9316517d19089a010356592 k8s.gcr.io/e2e-test-images/busybox:1.29-1],SizeBytes:1154361,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db k8s.gcr.io/pause:3.6 registry.k8s.io/pause:3.6],SizeBytes:682696,},},VolumesInUse:[kubernetes.io/csi/ebs.csi.aws.com^vol-03d4513454f149737 kubernetes.io/csi/ebs.csi.aws.com^vol-08601772399ba2131],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-03d4513454f149737,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-08601772399ba2131,DevicePath:,},},Config:nil,},} Jan 1 02:31:23.184: INFO: Logging kubelet events for node ip-172-20-45-112.eu-west-3.compute.internal Jan 1 02:31:23.291: INFO: Logging pods the kubelet thinks is on node ip-172-20-45-112.eu-west-3.compute.internal Jan 1 02:31:23.405: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-jh8bc started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:23.406: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:31:23.406: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-4lrj2 started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:23.406: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:31:23.406: INFO: cilium-slx98 started at 2023-01-01 02:20:28 +0000 UTC (1+1 container statuses recorded) Jan 1 02:31:23.406: INFO: Init container clean-cilium-state ready: true, restart count 0 Jan 1 02:31:23.406: INFO: Container cilium-agent ready: true, restart count 0 Jan 1 02:31:23.406: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-srf5k started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:23.406: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:31:23.406: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-4mml9 started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:23.406: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:31:23.406: INFO: affinity-clusterip-timeout-lv5vx started at 2023-01-01 02:27:32 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:23.406: INFO: Container affinity-clusterip-timeout ready: true, restart count 0 Jan 1 02:31:23.406: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-xnfzj started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:23.406: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:31:23.406: INFO: pod-d05d599f-1b25-4249-b9c7-c64a79abebd3 started at 2023-01-01 02:30:59 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:23.406: INFO: Container write-pod ready: false, restart count 0 Jan 1 02:31:23.406: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-x5cp6 started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:23.406: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:31:23.406: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-gph2d started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:23.406: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:31:23.406: INFO: image-pull-tested8f64cd-03c5-4c0a-9ed6-c4e25c9622f4 started at 2023-01-01 02:31:23 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:23.406: INFO: Container image-pull-test ready: false, restart count 0 Jan 1 02:31:23.406: INFO: netserver-1 started at 2023-01-01 02:25:27 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:23.406: INFO: Container webserver ready: true, restart count 0 Jan 1 02:31:23.406: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-rmdrr started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:23.406: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:31:23.406: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-kjg2m started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:23.406: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:31:23.406: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-59bdj started at 2023-01-01 02:25:17 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:23.406: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:31:23.406: INFO: ebs-csi-node-7clrr started at 2023-01-01 02:20:28 +0000 UTC (0+3 container statuses recorded) Jan 1 02:31:23.406: INFO: Container ebs-plugin ready: true, restart count 0 Jan 1 02:31:23.406: INFO: Container liveness-probe ready: true, restart count 0 Jan 1 02:31:23.406: INFO: Container node-driver-registrar ready: true, restart count 0 Jan 1 02:31:23.406: INFO: inline-volume-tester-fpr45 started at 2023-01-01 02:31:04 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:23.406: INFO: Container csi-volume-tester ready: false, restart count 0 Jan 1 02:31:23.406: INFO: e2e-dns-utils started at 2023-01-01 02:31:03 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:23.406: INFO: Container agnhost-container ready: false, restart count 0 Jan 1 02:31:23.942: INFO: Latency metrics for node ip-172-20-45-112.eu-west-3.compute.internal Jan 1 02:31:23.942: INFO: Logging node info for node ip-172-20-47-178.eu-west-3.compute.internal Jan 1 02:31:24.050: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-47-178.eu-west-3.compute.internal 835e5337-d2b9-45ad-bf0e-9d32514f71ba 8456 0 2023-01-01 02:20:25 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:eu-west-3 failure-domain.beta.kubernetes.io/zone:eu-west-3a kubelet_cleanup:true kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-47-178.eu-west-3.compute.internal kubernetes.io/os:linux kubernetes.io/role:node node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:eu-west-3a topology.kubernetes.io/region:eu-west-3 topology.kubernetes.io/zone:eu-west-3a] map[csi.volume.kubernetes.io/nodeid:{"csi-mock-csi-mock-volumes-5429":"csi-mock-csi-mock-volumes-5429","ebs.csi.aws.com":"i-00e1e60199f5a4d04"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kops-controller Update v1 2023-01-01 02:20:25 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}} } {kube-controller-manager Update v1 2023-01-01 02:20:25 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.2.0/24\"":{}}}} } {kubelet Update v1 2023-01-01 02:20:25 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {e2e.test Update v1 2023-01-01 02:25:15 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubelet_cleanup":{}}}} } {kubelet Update v1 2023-01-01 02:25:42 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:capacity":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.2.0/24,DoNotUseExternalID:,ProviderID:aws:///eu-west-3a/i-00e1e60199f5a4d04,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.2.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{50478186496 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4078190592 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{45430367772 0} {<nil>} 45430367772 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3973332992 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-01-01 02:30:48 +0000 UTC,LastTransitionTime:2023-01-01 02:20:25 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-01-01 02:30:48 +0000 UTC,LastTransitionTime:2023-01-01 02:20:25 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-01-01 02:30:48 +0000 UTC,LastTransitionTime:2023-01-01 02:20:25 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-01-01 02:30:48 +0000 UTC,LastTransitionTime:2023-01-01 02:20:46 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.47.178,},NodeAddress{Type:ExternalIP,Address:13.36.177.101,},NodeAddress{Type:Hostname,Address:ip-172-20-47-178.eu-west-3.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-47-178.eu-west-3.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-13-36-177-101.eu-west-3.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec243eda226c96d5573f3eaa3203df0d,SystemUUID:ec243eda-226c-96d5-573f-3eaa3203df0d,BootID:15dccaa7-d7c7-419d-b7de-76be5781a51a,KernelVersion:4.19.0-23-cloud-amd64,OSImage:Debian GNU/Linux 10 (buster),ContainerRuntimeVersion:docker://20.10.17,KubeletVersion:v1.23.15,KubeProxyVersion:v1.23.15,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:456569528,},ContainerImage{Names:[nginx@sha256:0047b729188a15da49380d9506d65959cce6d40291ccfb4e039f5dc7efd33286 nginx:latest],SizeBytes:141812353,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:7e8bdd271312fd25fc5ff5a8f04727be84044eb3d7d8d03611972a6752e2e11e k8s.gcr.io/e2e-test-images/agnhost:2.39],SizeBytes:126872991,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.23.15 registry.k8s.io/kube-proxy-amd64:v1.23.15],SizeBytes:112336036,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:10e231cf07c82dfab5141c1afe548dc734e5fc1f67665ec3982a325d1bd31a9a registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.0],SizeBytes:94471559,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:c8e03f60afa90a28e4bb6ec9a8d0fc36d89de4b7475cf2d613afa793ec969fe0 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.0],SizeBytes:56382329,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:c5be65d6679efabb969d9b019300d187437ae876f992c40911fd2892bbef3b36 k8s.gcr.io/sig-storage/csi-attacher:v3.2.0],SizeBytes:53500826,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:017727efcfeb7d053af68e51436ce8e65edbc6ca573720afb4f79c8594036955 registry.k8s.io/coredns/coredns:v1.10.0],SizeBytes:50249337,},ContainerImage{Names:[registry.k8s.io/cpa/cluster-proportional-autoscaler@sha256:68d396900aeaa072c1f27289485fdac29834045a6f3ffe369bf389d830ef572d registry.k8s.io/cpa/cluster-proportional-autoscaler:1.8.6],SizeBytes:47910609,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:a61d309da54641db41fb8f35718f744e9f730d4d0384f8c4b186ddc9f06cbd5f k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.1.0],SizeBytes:19662887,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:19582112,},ContainerImage{Names:[k8s.gcr.io/sig-storage/mock-driver@sha256:a7b517f9e0f42ffade802eef9cefa271372386b85f55b702b493241e58459793 k8s.gcr.io/sig-storage/mock-driver:v4.1.0],SizeBytes:17680993,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:17611980,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:13616070e3f29de4417eee434a8ef472221c9e51b3d037b5a6b46cef08eb7443 k8s.gcr.io/e2e-test-images/nginx:1.14-2],SizeBytes:16032814,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf k8s.gcr.io/e2e-test-images/busybox:1.29-2],SizeBytes:1154361,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db k8s.gcr.io/pause:3.6 registry.k8s.io/pause:3.6],SizeBytes:682696,},},VolumesInUse:[kubernetes.io/csi/csi-mock-csi-mock-volumes-5429^4],VolumesAttached:[]AttachedVolume{},Config:nil,},} Jan 1 02:31:24.050: INFO: Logging kubelet events for node ip-172-20-47-178.eu-west-3.compute.internal Jan 1 02:31:24.157: INFO: Logging pods the kubelet thinks is on node ip-172-20-47-178.eu-west-3.compute.internal Jan 1 02:31:24.282: INFO: csi-mockplugin-attacher-0 started at 2023-01-01 02:25:20 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:24.282: INFO: Container csi-attacher ready: false, restart count 0 Jan 1 02:31:24.282: INFO: ebs-csi-node-vcprc started at 2023-01-01 02:20:26 +0000 UTC (0+3 container statuses recorded) Jan 1 02:31:24.282: INFO: Container ebs-plugin ready: true, restart count 0 Jan 1 02:31:24.282: INFO: Container liveness-probe ready: true, restart count 0 Jan 1 02:31:24.282: INFO: Container node-driver-registrar ready: true, restart count 0 Jan 1 02:31:24.282: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-49brr started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:24.282: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:31:24.282: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-z87gr started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:24.282: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:31:24.282: INFO: csi-mockplugin-0 started at 2023-01-01 02:25:20 +0000 UTC (0+3 container statuses recorded) Jan 1 02:31:24.282: INFO: Container csi-provisioner ready: true, restart count 0 Jan 1 02:31:24.282: INFO: Container driver-registrar ready: true, restart count 0 Jan 1 02:31:24.282: INFO: Container mock ready: true, restart count 0 Jan 1 02:31:24.282: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-wjz96 started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:24.282: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:31:24.282: INFO: pvc-volume-tester-w4w4d started at 2023-01-01 02:25:38 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:24.282: INFO: Container volume-tester ready: false, restart count 0 Jan 1 02:31:24.282: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-w26gj started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:24.282: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:31:24.282: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-s4rqf started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:24.282: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:31:24.282: INFO: cilium-kk2c2 started at 2023-01-01 02:20:26 +0000 UTC (1+1 container statuses recorded) Jan 1 02:31:24.282: INFO: Init container clean-cilium-state ready: true, restart count 0 Jan 1 02:31:24.282: INFO: Container cilium-agent ready: true, restart count 0 Jan 1 02:31:24.282: INFO: local-injector started at 2023-01-01 02:31:07 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:24.282: INFO: Container local-injector ready: false, restart count 0 Jan 1 02:31:24.282: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-6vhtn started at 2023-01-01 02:25:17 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:24.282: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: false, restart count 0 Jan 1 02:31:24.282: INFO: coredns-867df8f45c-wzntb started at 2023-01-01 02:20:46 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:24.282: INFO: Container coredns ready: true, restart count 0 Jan 1 02:31:24.282: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-wkclp started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:24.282: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:31:24.282: INFO: netserver-2 started at 2023-01-01 02:25:27 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:24.282: INFO: Container webserver ready: false, restart count 0 Jan 1 02:31:24.282: INFO: coredns-autoscaler-557ccb4c66-k4lzb started at 2023-01-01 02:20:46 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:24.282: INFO: Container autoscaler ready: true, restart count 0 Jan 1 02:31:24.282: INFO: csi-mockplugin-attacher-0 started at 2023-01-01 02:28:39 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:24.282: INFO: Container csi-attacher ready: false, restart count 0 Jan 1 02:31:24.282: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-pgkm2 started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:24.282: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:31:24.282: INFO: hostexec-ip-172-20-47-178.eu-west-3.compute.internal-k7k4h started at 2023-01-01 02:30:47 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:24.282: INFO: Container agnhost-container ready: true, restart count 0 Jan 1 02:31:24.282: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-hrgrq started at 2023-01-01 02:25:17 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:24.282: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:31:24.282: INFO: csi-hostpathplugin-0 started at 2023-01-01 02:31:02 +0000 UTC (0+7 container statuses recorded) Jan 1 02:31:24.282: INFO: Container csi-attacher ready: false, restart count 0 Jan 1 02:31:24.282: INFO: Container csi-provisioner ready: false, restart count 0 Jan 1 02:31:24.282: INFO: Container csi-resizer ready: false, restart count 0 Jan 1 02:31:24.282: INFO: Container csi-snapshotter ready: false, restart count 0 Jan 1 02:31:24.282: INFO: Container hostpath ready: false, restart count 0 Jan 1 02:31:24.282: INFO: Container liveness-probe ready: false, restart count 0 Jan 1 02:31:24.282: INFO: Container node-driver-registrar ready: false, restart count 0 Jan 1 02:31:24.282: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-grrsl started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:24.282: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:31:24.282: INFO: csi-hostpathplugin-0 started at 2023-01-01 02:30:56 +0000 UTC (0+7 container statuses recorded) Jan 1 02:31:24.282: INFO: Container csi-attacher ready: false, restart count 0 Jan 1 02:31:24.282: INFO: Container csi-provisioner ready: false, restart count 0 Jan 1 02:31:24.282: INFO: Container csi-resizer ready: false, restart count 0 Jan 1 02:31:24.282: INFO: Container csi-snapshotter ready: false, restart count 0 Jan 1 02:31:24.282: INFO: Container hostpath ready: false, restart count 0 Jan 1 02:31:24.282: INFO: Container liveness-probe ready: false, restart count 0 Jan 1 02:31:24.282: INFO: Container node-driver-registrar ready: false, restart count 0 Jan 1 02:31:24.282: INFO: csi-mockplugin-0 started at 2023-01-01 02:28:39 +0000 UTC (0+3 container statuses recorded) Jan 1 02:31:24.282: INFO: Container csi-provisioner ready: false, restart count 0 Jan 1 02:31:24.282: INFO: Container driver-registrar ready: false, restart count 0 Jan 1 02:31:24.282: INFO: Container mock ready: false, restart count 0 Jan 1 02:31:24.650: INFO: Latency metrics for node ip-172-20-47-178.eu-west-3.compute.internal Jan 1 02:31:24.650: INFO: Logging node info for node ip-172-20-49-224.eu-west-3.compute.internal Jan 1 02:31:24.755: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-49-224.eu-west-3.compute.internal d7d57146-c54a-4155-94f8-0f322216bfcf 8317 0 2023-01-01 02:20:28 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:eu-west-3 failure-domain.beta.kubernetes.io/zone:eu-west-3a kubelet_cleanup:true kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-49-224.eu-west-3.compute.internal kubernetes.io/os:linux kubernetes.io/role:node node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:eu-west-3a topology.hostpath.csi/node:ip-172-20-49-224.eu-west-3.compute.internal topology.kubernetes.io/region:eu-west-3 topology.kubernetes.io/zone:eu-west-3a] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-089a107939b27c28a"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kops-controller Update v1 2023-01-01 02:20:28 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}} } {kube-controller-manager Update v1 2023-01-01 02:20:28 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.4.0/24\"":{}}}} } {kubelet Update v1 2023-01-01 02:20:28 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {e2e.test Update v1 2023-01-01 02:25:15 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubelet_cleanup":{}}}} } {kubelet Update v1 2023-01-01 02:25:35 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:capacity":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.4.0/24,DoNotUseExternalID:,ProviderID:aws:///eu-west-3a/i-089a107939b27c28a,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.4.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{50478186496 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4078206976 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{45430367772 0} {<nil>} 45430367772 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3973349376 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-01-01 02:30:42 +0000 UTC,LastTransitionTime:2023-01-01 02:20:28 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-01-01 02:30:42 +0000 UTC,LastTransitionTime:2023-01-01 02:20:28 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-01-01 02:30:42 +0000 UTC,LastTransitionTime:2023-01-01 02:20:28 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-01-01 02:30:42 +0000 UTC,LastTransitionTime:2023-01-01 02:20:48 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.49.224,},NodeAddress{Type:ExternalIP,Address:15.237.95.101,},NodeAddress{Type:Hostname,Address:ip-172-20-49-224.eu-west-3.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-49-224.eu-west-3.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-15-237-95-101.eu-west-3.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2d0a1bb33dd26f5a12c3896530131b,SystemUUID:ec2d0a1b-b33d-d26f-5a12-c3896530131b,BootID:9f64e91d-26f0-45aa-aec4-a54433e67034,KernelVersion:4.19.0-23-cloud-amd64,OSImage:Debian GNU/Linux 10 (buster),ContainerRuntimeVersion:docker://20.10.17,KubeletVersion:v1.23.15,KubeProxyVersion:v1.23.15,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:456569528,},ContainerImage{Names:[nginx@sha256:0047b729188a15da49380d9506d65959cce6d40291ccfb4e039f5dc7efd33286 nginx:latest],SizeBytes:141812353,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:7e8bdd271312fd25fc5ff5a8f04727be84044eb3d7d8d03611972a6752e2e11e k8s.gcr.io/e2e-test-images/agnhost:2.39],SizeBytes:126872991,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:1b9d1b2f36cb2dbee1960e82a9344aeb11bd4c4c03abf5e1853e0559c23855e3 k8s.gcr.io/e2e-test-images/httpd:2.4.38-2],SizeBytes:123781643,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.23.15 registry.k8s.io/kube-proxy-amd64:v1.23.15],SizeBytes:112336036,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:10e231cf07c82dfab5141c1afe548dc734e5fc1f67665ec3982a325d1bd31a9a registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.0],SizeBytes:94471559,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:4e74c0492bceddc598de1c90cc5bc14dcda94cb49fa9c5bad9d117c4834b5e08 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.1],SizeBytes:56379269,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:36c31f7e1f433c9634d24f876353e8646246d81a03c4e351202c2644daff1620 k8s.gcr.io/sig-storage/csi-resizer:v1.2.0],SizeBytes:53956308,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:ed98431376c9e944e19a465fe8ea944806714dd95416a0821096c78d66b579bd k8s.gcr.io/sig-storage/csi-snapshotter:v4.1.1],SizeBytes:53531990,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:60ab9b3e6a030d3038c87c0d6bca2930f58d1d72823e6a4af09767dc83b696a2 k8s.gcr.io/sig-storage/csi-attacher:v3.2.1],SizeBytes:53499570,},ContainerImage{Names:[k8s.gcr.io/sig-storage/hostpathplugin@sha256:232fe80174d60d520d36043103853a1d7ab4b7f3782cf43e45034f04ccda58ce k8s.gcr.io/sig-storage/hostpathplugin:v1.7.1],SizeBytes:29977921,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:19582112,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:2dee3fe5fe861bb66c3a4ac51114f3447a4cd35870e0f2e2b558c7a400d89589 k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.2.0],SizeBytes:18699436,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:17611980,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:529be2c9770add0cdd0c989115222ea9fc1be430c11095eb9f6dafcf98a36e2b k8s.gcr.io/sig-storage/livenessprobe:v2.4.0],SizeBytes:17175488,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:13616070e3f29de4417eee434a8ef472221c9e51b3d037b5a6b46cef08eb7443 k8s.gcr.io/e2e-test-images/nginx:1.14-2],SizeBytes:16032814,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf k8s.gcr.io/e2e-test-images/busybox:1.29-2],SizeBytes:1154361,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db k8s.gcr.io/pause:3.6 registry.k8s.io/pause:3.6],SizeBytes:682696,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Jan 1 02:31:24.756: INFO: Logging kubelet events for node ip-172-20-49-224.eu-west-3.compute.internal Jan 1 02:31:24.863: INFO: Logging pods the kubelet thinks is on node ip-172-20-49-224.eu-west-3.compute.internal Jan 1 02:31:24.977: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-tv984 started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:24.977: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:31:24.977: INFO: affinity-clusterip-timeout-pxjtk started at 2023-01-01 02:27:32 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:24.977: INFO: Container affinity-clusterip-timeout ready: true, restart count 0 Jan 1 02:31:24.977: INFO: annotationupdate3b35e9ec-7ad7-4bd9-8137-1bd2bf1a899a started at 2023-01-01 02:28:25 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:24.977: INFO: Container client-container ready: false, restart count 0 Jan 1 02:31:24.977: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-cz8qh started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:24.977: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:31:24.977: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-2gndb started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:24.977: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:31:24.977: INFO: httpd started at 2023-01-01 02:26:24 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:24.978: INFO: Container httpd ready: false, restart count 0 Jan 1 02:31:24.978: INFO: pod-3e35f616-b2af-42fe-8148-bfe7bf04221b started at 2023-01-01 02:31:05 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:24.978: INFO: Container write-pod ready: false, restart count 0 Jan 1 02:31:24.978: INFO: dns-test-a8277dcf-764d-4d97-a381-813a8d1f903c started at 2023-01-01 02:31:17 +0000 UTC (0+3 container statuses recorded) Jan 1 02:31:24.978: INFO: Container jessie-querier ready: false, restart count 0 Jan 1 02:31:24.978: INFO: Container querier ready: false, restart count 0 Jan 1 02:31:24.978: INFO: Container webserver ready: false, restart count 0 Jan 1 02:31:24.978: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-6n5j9 started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:24.978: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:31:24.978: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-h6dxb started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:24.978: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:31:24.978: INFO: test-ss-0 started at 2023-01-01 02:27:56 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:24.978: INFO: Container webserver ready: false, restart count 0 Jan 1 02:31:24.978: INFO: hostexec-ip-172-20-49-224.eu-west-3.compute.internal-hsfbk started at 2023-01-01 02:31:01 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:24.978: INFO: Container agnhost-container ready: true, restart count 0 Jan 1 02:31:24.978: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-pgc9g started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:24.978: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:31:24.978: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-29cdj started at 2023-01-01 02:25:17 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:24.978: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: false, restart count 0 Jan 1 02:31:24.978: INFO: e2e-configmap-dns-server-7551ef4f-8c37-4168-bf54-91644ec7fb8f started at 2023-01-01 02:29:40 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:24.978: INFO: Container agnhost-container ready: true, restart count 0 Jan 1 02:31:24.978: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-7z6bp started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:24.978: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:31:24.978: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-lrzjq started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:24.978: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:31:24.978: INFO: netserver-3 started at 2023-01-01 02:25:27 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:24.978: INFO: Container webserver ready: true, restart count 0 Jan 1 02:31:24.978: INFO: echo-sourceip started at 2023-01-01 02:26:27 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:24.978: INFO: Container agnhost-container ready: false, restart count 0 Jan 1 02:31:24.978: INFO: pod-service-account-nomountsa-mountspec started at 2023-01-01 02:29:39 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:24.978: INFO: Container token-test ready: false, restart count 0 Jan 1 02:31:24.978: INFO: cilium-78jll started at 2023-01-01 02:20:29 +0000 UTC (1+1 container statuses recorded) Jan 1 02:31:24.978: INFO: Init container clean-cilium-state ready: true, restart count 0 Jan 1 02:31:24.978: INFO: Container cilium-agent ready: true, restart count 0 Jan 1 02:31:24.978: INFO: ebs-csi-node-tzntf started at 2023-01-01 02:20:29 +0000 UTC (0+3 container statuses recorded) Jan 1 02:31:24.978: INFO: Container ebs-plugin ready: true, restart count 0 Jan 1 02:31:24.978: INFO: Container liveness-probe ready: true, restart count 0 Jan 1 02:31:24.978: INFO: Container node-driver-registrar ready: true, restart count 0 Jan 1 02:31:24.978: INFO: pod-subpath-test-secret-87r5 started at 2023-01-01 02:30:51 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:24.978: INFO: Container test-container-subpath-secret-87r5 ready: false, restart count 0 Jan 1 02:31:24.978: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-4jf6d started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:24.978: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:31:24.978: INFO: dns-test-f6141192-e7ab-4530-bedd-b1f3b48a9237 started at 2023-01-01 02:30:56 +0000 UTC (0+3 container statuses recorded) Jan 1 02:31:24.978: INFO: Container jessie-querier ready: false, restart count 0 Jan 1 02:31:24.978: INFO: Container querier ready: false, restart count 0 Jan 1 02:31:24.978: INFO: Container webserver ready: false, restart count 0 Jan 1 02:31:25.476: INFO: Latency metrics for node ip-172-20-49-224.eu-west-3.compute.internal Jan 1 02:31:25.476: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready �[1mSTEP�[0m: Destroying namespace "nettest-5021" for this suite.
Filter through log files
go run hack/e2e.go -v --test --test_args='--ginkgo.focus=Kubernetes\se2e\ssuite\s\[sig\-network\]\sNetworking\sGranular\sChecks\:\sServices\sshould\supdate\sendpoints\:\sudp$'
/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/network/networking.go:343 Jan 1 02:30:34.430: Unexpected error: <*errors.errorString | 0xc0001c62c0>: { s: "timed out waiting for the condition", } timed out waiting for the condition occurred /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/network/utils.go:858from junit_20.xml
[BeforeEach] [sig-network] Networking /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:185 �[1mSTEP�[0m: Creating a kubernetes client Jan 1 02:25:32.737: INFO: >>> kubeConfig: /root/.kube/config �[1mSTEP�[0m: Building a namespace api object, basename nettest �[1mSTEP�[0m: Waiting for a default service account to be provisioned in namespace �[1mSTEP�[0m: Waiting for kube-root-ca.crt to be provisioned in namespace [It] should update endpoints: udp /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/network/networking.go:343 �[1mSTEP�[0m: Performing setup for networking test in namespace nettest-3615 �[1mSTEP�[0m: creating a selector �[1mSTEP�[0m: Creating the service pods in kubernetes Jan 1 02:25:33.469: INFO: Waiting up to 10m0s for all (but 0) nodes to be schedulable Jan 1 02:25:34.220: INFO: The status of Pod netserver-0 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:25:36.325: INFO: The status of Pod netserver-0 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:25:38.325: INFO: The status of Pod netserver-0 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:25:40.326: INFO: The status of Pod netserver-0 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:25:42.325: INFO: The status of Pod netserver-0 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:25:44.327: INFO: The status of Pod netserver-0 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:25:46.325: INFO: The status of Pod netserver-0 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:25:48.324: INFO: The status of Pod netserver-0 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:25:50.326: INFO: The status of Pod netserver-0 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:25:52.326: INFO: The status of Pod netserver-0 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:25:54.325: INFO: The status of Pod netserver-0 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:25:56.325: INFO: The status of Pod netserver-0 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:25:58.325: INFO: The status of Pod netserver-0 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:26:00.327: INFO: The status of Pod netserver-0 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:26:02.325: INFO: The status of Pod netserver-0 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:26:04.325: INFO: The status of Pod netserver-0 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:26:06.326: INFO: The status of Pod netserver-0 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:26:08.325: INFO: The status of Pod netserver-0 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:26:10.325: INFO: The status of Pod netserver-0 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:26:12.325: INFO: The status of Pod netserver-0 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:26:14.325: INFO: The status of Pod netserver-0 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:26:16.326: INFO: The status of Pod netserver-0 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:26:18.324: INFO: The status of Pod netserver-0 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:26:20.334: INFO: The status of Pod netserver-0 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:26:22.338: INFO: The status of Pod netserver-0 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:26:24.324: INFO: The status of Pod netserver-0 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:26:26.326: INFO: The status of Pod netserver-0 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:26:28.324: INFO: The status of Pod netserver-0 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:26:30.326: INFO: The status of Pod netserver-0 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:26:32.325: INFO: The status of Pod netserver-0 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:26:34.325: INFO: The status of Pod netserver-0 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:26:36.326: INFO: The status of Pod netserver-0 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:26:38.325: INFO: The status of Pod netserver-0 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:26:40.324: INFO: The status of Pod netserver-0 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:26:42.325: INFO: The status of Pod netserver-0 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:26:44.324: INFO: The status of Pod netserver-0 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:26:46.324: INFO: The status of Pod netserver-0 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:26:48.325: INFO: The status of Pod netserver-0 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:26:50.324: INFO: The status of Pod netserver-0 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:26:52.324: INFO: The status of Pod netserver-0 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:26:54.324: INFO: The status of Pod netserver-0 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:26:56.325: INFO: The status of Pod netserver-0 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:26:58.325: INFO: The status of Pod netserver-0 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:27:00.324: INFO: The status of Pod netserver-0 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:27:02.326: INFO: The status of Pod netserver-0 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:27:04.326: INFO: The status of Pod netserver-0 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:27:06.324: INFO: The status of Pod netserver-0 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:27:08.325: INFO: The status of Pod netserver-0 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:27:10.324: INFO: The status of Pod netserver-0 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:27:12.325: INFO: The status of Pod netserver-0 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:27:14.324: INFO: The status of Pod netserver-0 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:27:16.325: INFO: The status of Pod netserver-0 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:27:18.325: INFO: The status of Pod netserver-0 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:27:20.324: INFO: The status of Pod netserver-0 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:27:22.324: INFO: The status of Pod netserver-0 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:27:24.324: INFO: The status of Pod netserver-0 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:27:26.324: INFO: The status of Pod netserver-0 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:27:28.326: INFO: The status of Pod netserver-0 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:27:30.325: INFO: The status of Pod netserver-0 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:27:32.325: INFO: The status of Pod netserver-0 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:27:34.325: INFO: The status of Pod netserver-0 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:27:36.325: INFO: The status of Pod netserver-0 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:27:38.325: INFO: The status of Pod netserver-0 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:27:40.325: INFO: The status of Pod netserver-0 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:27:42.325: INFO: The status of Pod netserver-0 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:27:44.325: INFO: The status of Pod netserver-0 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:27:46.324: INFO: The status of Pod netserver-0 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:27:48.325: INFO: The status of Pod netserver-0 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:27:50.326: INFO: The status of Pod netserver-0 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:27:52.325: INFO: The status of Pod netserver-0 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:27:54.325: INFO: The status of Pod netserver-0 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:27:56.324: INFO: The status of Pod netserver-0 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:27:58.327: INFO: The status of Pod netserver-0 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:28:00.325: INFO: The status of Pod netserver-0 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:28:02.324: INFO: The status of Pod netserver-0 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:28:04.324: INFO: The status of Pod netserver-0 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:28:06.326: INFO: The status of Pod netserver-0 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:28:08.325: INFO: The status of Pod netserver-0 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:28:10.325: INFO: The status of Pod netserver-0 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:28:12.325: INFO: The status of Pod netserver-0 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:28:14.325: INFO: The status of Pod netserver-0 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:28:16.326: INFO: The status of Pod netserver-0 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:28:18.324: INFO: The status of Pod netserver-0 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:28:20.325: INFO: The status of Pod netserver-0 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:28:22.324: INFO: The status of Pod netserver-0 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:28:24.324: INFO: The status of Pod netserver-0 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:28:26.326: INFO: The status of Pod netserver-0 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:28:28.325: INFO: The status of Pod netserver-0 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:28:30.326: INFO: The status of Pod netserver-0 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:28:32.325: INFO: The status of Pod netserver-0 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:28:34.325: INFO: The status of Pod netserver-0 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:28:36.326: INFO: The status of Pod netserver-0 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:28:38.325: INFO: The status of Pod netserver-0 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:28:40.324: INFO: The status of Pod netserver-0 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:28:42.325: INFO: The status of Pod netserver-0 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:28:44.324: INFO: The status of Pod netserver-0 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:28:46.327: INFO: The status of Pod netserver-0 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:28:48.325: INFO: The status of Pod netserver-0 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:28:50.326: INFO: The status of Pod netserver-0 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:28:52.326: INFO: The status of Pod netserver-0 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:28:54.325: INFO: The status of Pod netserver-0 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:28:56.326: INFO: The status of Pod netserver-0 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:28:58.325: INFO: The status of Pod netserver-0 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:29:00.324: INFO: The status of Pod netserver-0 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:29:02.325: INFO: The status of Pod netserver-0 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:29:04.325: INFO: The status of Pod netserver-0 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:29:06.326: INFO: The status of Pod netserver-0 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:29:08.325: INFO: The status of Pod netserver-0 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:29:10.324: INFO: The status of Pod netserver-0 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:29:12.326: INFO: The status of Pod netserver-0 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:29:14.324: INFO: The status of Pod netserver-0 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:29:16.326: INFO: The status of Pod netserver-0 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:29:18.325: INFO: The status of Pod netserver-0 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:29:20.326: INFO: The status of Pod netserver-0 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:29:22.324: INFO: The status of Pod netserver-0 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:29:24.325: INFO: The status of Pod netserver-0 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:29:26.324: INFO: The status of Pod netserver-0 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:29:28.325: INFO: The status of Pod netserver-0 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:29:30.325: INFO: The status of Pod netserver-0 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:29:32.325: INFO: The status of Pod netserver-0 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:29:34.331: INFO: The status of Pod netserver-0 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:29:36.326: INFO: The status of Pod netserver-0 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:29:38.324: INFO: The status of Pod netserver-0 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:29:40.324: INFO: The status of Pod netserver-0 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:29:42.325: INFO: The status of Pod netserver-0 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:29:44.324: INFO: The status of Pod netserver-0 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:29:46.326: INFO: The status of Pod netserver-0 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:29:48.325: INFO: The status of Pod netserver-0 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:29:50.325: INFO: The status of Pod netserver-0 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:29:52.326: INFO: The status of Pod netserver-0 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:29:54.324: INFO: The status of Pod netserver-0 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:29:56.324: INFO: The status of Pod netserver-0 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:29:58.324: INFO: The status of Pod netserver-0 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:30:00.325: INFO: The status of Pod netserver-0 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:30:02.325: INFO: The status of Pod netserver-0 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:30:04.324: INFO: The status of Pod netserver-0 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:30:06.325: INFO: The status of Pod netserver-0 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:30:08.325: INFO: The status of Pod netserver-0 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:30:10.324: INFO: The status of Pod netserver-0 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:30:12.325: INFO: The status of Pod netserver-0 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:30:14.324: INFO: The status of Pod netserver-0 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:30:16.324: INFO: The status of Pod netserver-0 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:30:18.325: INFO: The status of Pod netserver-0 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:30:20.324: INFO: The status of Pod netserver-0 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:30:22.325: INFO: The status of Pod netserver-0 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:30:24.327: INFO: The status of Pod netserver-0 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:30:26.324: INFO: The status of Pod netserver-0 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:30:28.325: INFO: The status of Pod netserver-0 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:30:30.325: INFO: The status of Pod netserver-0 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:30:32.325: INFO: The status of Pod netserver-0 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:30:34.325: INFO: The status of Pod netserver-0 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:30:34.430: INFO: The status of Pod netserver-0 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:30:34.430: FAIL: Unexpected error: <*errors.errorString | 0xc0001c62c0>: { s: "timed out waiting for the condition", } timed out waiting for the condition occurred Full Stack Trace k8s.io/kubernetes/test/e2e/framework/network.(*NetworkingTestConfig).createNetProxyPods(0xc004deab60, {0x70da7f5, 0x9}, 0xc004f92180) /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/network/utils.go:858 +0x1d4 k8s.io/kubernetes/test/e2e/framework/network.(*NetworkingTestConfig).setupCore(0xc004deab60, 0xc004f92180) /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/network/utils.go:760 +0x5b k8s.io/kubernetes/test/e2e/framework/network.(*NetworkingTestConfig).setup(0xc004deab60, 0x34) /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/network/utils.go:775 +0x3b k8s.io/kubernetes/test/e2e/framework/network.NewNetworkingTestConfig(0xc000e35a20, {0x0, 0x0, 0x6c}) /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/network/utils.go:129 +0x125 k8s.io/kubernetes/test/e2e/network.glob..func20.6.12() /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/network/networking.go:344 +0x36 k8s.io/kubernetes/test/e2e.RunE2ETests(0x24c66d7) _output/dockerized/go/src/k8s.io/kubernetes/test/e2e/e2e.go:133 +0x697 k8s.io/kubernetes/test/e2e.TestE2E(0x243a8f9) _output/dockerized/go/src/k8s.io/kubernetes/test/e2e/e2e_test.go:136 +0x19 testing.tRunner(0xc00091d1e0, 0x735e880) /usr/local/go/src/testing/testing.go:1259 +0x102 created by testing.(*T).Run /usr/local/go/src/testing/testing.go:1306 +0x35a [AfterEach] [sig-network] Networking /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:186 �[1mSTEP�[0m: Collecting events from namespace "nettest-3615". �[1mSTEP�[0m: Found 32 events. Jan 1 02:30:34.537: INFO: At 2023-01-01 02:25:33 +0000 UTC - event for netserver-0: {default-scheduler } Scheduled: Successfully assigned nettest-3615/netserver-0 to ip-172-20-33-66.eu-west-3.compute.internal Jan 1 02:30:34.537: INFO: At 2023-01-01 02:25:33 +0000 UTC - event for netserver-1: {default-scheduler } Scheduled: Successfully assigned nettest-3615/netserver-1 to ip-172-20-45-112.eu-west-3.compute.internal Jan 1 02:30:34.537: INFO: At 2023-01-01 02:25:33 +0000 UTC - event for netserver-2: {default-scheduler } Scheduled: Successfully assigned nettest-3615/netserver-2 to ip-172-20-47-178.eu-west-3.compute.internal Jan 1 02:30:34.537: INFO: At 2023-01-01 02:25:34 +0000 UTC - event for netserver-3: {default-scheduler } Scheduled: Successfully assigned nettest-3615/netserver-3 to ip-172-20-49-224.eu-west-3.compute.internal Jan 1 02:30:34.537: INFO: At 2023-01-01 02:25:35 +0000 UTC - event for netserver-0: {kubelet ip-172-20-33-66.eu-west-3.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "cbf831c457ab8132930f4997b9c0697e16e0c2960450c7aef0f05292a81958f5" network for pod "netserver-0": networkPlugin cni failed to set up pod "netserver-0_nettest-3615" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 1 02:30:34.537: INFO: At 2023-01-01 02:25:35 +0000 UTC - event for netserver-2: {kubelet ip-172-20-47-178.eu-west-3.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "5aa57716b84972f473eaf9c87897a2bb086e27d86969695d988574cc80a7b486" network for pod "netserver-2": networkPlugin cni failed to set up pod "netserver-2_nettest-3615" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 1 02:30:34.537: INFO: At 2023-01-01 02:25:36 +0000 UTC - event for netserver-0: {kubelet ip-172-20-33-66.eu-west-3.compute.internal} SandboxChanged: Pod sandbox changed, it will be killed and re-created. Jan 1 02:30:34.537: INFO: At 2023-01-01 02:25:36 +0000 UTC - event for netserver-1: {kubelet ip-172-20-45-112.eu-west-3.compute.internal} Created: Created container webserver Jan 1 02:30:34.537: INFO: At 2023-01-01 02:25:36 +0000 UTC - event for netserver-1: {kubelet ip-172-20-45-112.eu-west-3.compute.internal} Pulled: Container image "k8s.gcr.io/e2e-test-images/agnhost:2.39" already present on machine Jan 1 02:30:34.537: INFO: At 2023-01-01 02:25:37 +0000 UTC - event for netserver-0: {kubelet ip-172-20-33-66.eu-west-3.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "97ab28e33efc4421759ae9139eefa21df31a455d6108de7205019a43cae74a8e" network for pod "netserver-0": networkPlugin cni failed to set up pod "netserver-0_nettest-3615" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 1 02:30:34.537: INFO: At 2023-01-01 02:25:37 +0000 UTC - event for netserver-1: {kubelet ip-172-20-45-112.eu-west-3.compute.internal} Started: Started container webserver Jan 1 02:30:34.537: INFO: At 2023-01-01 02:25:37 +0000 UTC - event for netserver-2: {kubelet ip-172-20-47-178.eu-west-3.compute.internal} SandboxChanged: Pod sandbox changed, it will be killed and re-created. Jan 1 02:30:34.537: INFO: At 2023-01-01 02:25:38 +0000 UTC - event for netserver-0: {kubelet ip-172-20-33-66.eu-west-3.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "01477a80943efde118cabd32fd46fbf9b268870e23d7c53ee8848f4969185f2f" network for pod "netserver-0": networkPlugin cni failed to set up pod "netserver-0_nettest-3615" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 1 02:30:34.537: INFO: At 2023-01-01 02:25:38 +0000 UTC - event for netserver-2: {kubelet ip-172-20-47-178.eu-west-3.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "65f491d3bead0add12a50d0b6cc657f96bc21bf6dacbe05a0a74c8f176aa4600" network for pod "netserver-2": networkPlugin cni failed to set up pod "netserver-2_nettest-3615" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 1 02:30:34.537: INFO: At 2023-01-01 02:25:38 +0000 UTC - event for netserver-3: {kubelet ip-172-20-49-224.eu-west-3.compute.internal} Started: Started container webserver Jan 1 02:30:34.537: INFO: At 2023-01-01 02:25:38 +0000 UTC - event for netserver-3: {kubelet ip-172-20-49-224.eu-west-3.compute.internal} Created: Created container webserver Jan 1 02:30:34.537: INFO: At 2023-01-01 02:25:38 +0000 UTC - event for netserver-3: {kubelet ip-172-20-49-224.eu-west-3.compute.internal} Pulled: Container image "k8s.gcr.io/e2e-test-images/agnhost:2.39" already present on machine Jan 1 02:30:34.537: INFO: At 2023-01-01 02:25:39 +0000 UTC - event for netserver-0: {kubelet ip-172-20-33-66.eu-west-3.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "85f5b2e02ddd95d25b8ce99e3435e9fb51dc6bc183ccfdaa8d980afd36ac47fe" network for pod "netserver-0": networkPlugin cni failed to set up pod "netserver-0_nettest-3615" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 1 02:30:34.537: INFO: At 2023-01-01 02:25:40 +0000 UTC - event for netserver-2: {kubelet ip-172-20-47-178.eu-west-3.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "93ae9cf5c06795c349d933e3bd5db2288a94693210c1af7cff329a46aa53b518" network for pod "netserver-2": networkPlugin cni failed to set up pod "netserver-2_nettest-3615" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 1 02:30:34.537: INFO: At 2023-01-01 02:25:41 +0000 UTC - event for netserver-0: {kubelet ip-172-20-33-66.eu-west-3.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "fe100f4f6082f3c22ff664f38788baf5dcf133c4abe423a9fe26701071c209cd" network for pod "netserver-0": networkPlugin cni failed to set up pod "netserver-0_nettest-3615" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 1 02:30:34.537: INFO: At 2023-01-01 02:25:43 +0000 UTC - event for netserver-0: {kubelet ip-172-20-33-66.eu-west-3.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "e09e0c977509780471a19dd22036c3946e294701e592f1ee7cd673759dab501c" network for pod "netserver-0": networkPlugin cni failed to set up pod "netserver-0_nettest-3615" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 1 02:30:34.537: INFO: At 2023-01-01 02:25:43 +0000 UTC - event for netserver-2: {kubelet ip-172-20-47-178.eu-west-3.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "c9cd5e9a4426372441b26e2570589f67a180eb481c3f8e11077c5e91ccde71d6" network for pod "netserver-2": networkPlugin cni failed to set up pod "netserver-2_nettest-3615" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 1 02:30:34.537: INFO: At 2023-01-01 02:25:45 +0000 UTC - event for netserver-2: {kubelet ip-172-20-47-178.eu-west-3.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "0134d73e75e342e3c179c6268e92eb5305b544bb577b8744cf249e19726b6efb" network for pod "netserver-2": networkPlugin cni failed to set up pod "netserver-2_nettest-3615" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 1 02:30:34.537: INFO: At 2023-01-01 02:25:46 +0000 UTC - event for netserver-0: {kubelet ip-172-20-33-66.eu-west-3.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "9f853cf013838e12e2c9034dbad58c25d815d86f58495ddea36fb48f9345ee6a" network for pod "netserver-0": networkPlugin cni failed to set up pod "netserver-0_nettest-3615" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 1 02:30:34.537: INFO: At 2023-01-01 02:25:48 +0000 UTC - event for netserver-0: {kubelet ip-172-20-33-66.eu-west-3.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "48bc8d21a86371aa23488193c6553fced110f4f30c4f8e633e19bb35084540ae" network for pod "netserver-0": networkPlugin cni failed to set up pod "netserver-0_nettest-3615" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 1 02:30:34.537: INFO: At 2023-01-01 02:25:48 +0000 UTC - event for netserver-2: {kubelet ip-172-20-47-178.eu-west-3.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "07c126f413915146e54e6f6e30d4fc27d63ea2a82eab46285c0c5c75c2c72510" network for pod "netserver-2": networkPlugin cni failed to set up pod "netserver-2_nettest-3615" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 1 02:30:34.537: INFO: At 2023-01-01 02:25:48 +0000 UTC - event for netserver-2: {kubelet ip-172-20-47-178.eu-west-3.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "9a65e937e39e3d36394d3305beacb9f29b0600b2d4c8d64f7b5ff33c21febfdd" network for pod "netserver-2": networkPlugin cni failed to set up pod "netserver-2_nettest-3615" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 1 02:30:34.537: INFO: At 2023-01-01 02:25:50 +0000 UTC - event for netserver-2: {kubelet ip-172-20-47-178.eu-west-3.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "7d70c588be310ffc1673e9af4f9b3e03625a5a427a814547c64d2199a99d64dc" network for pod "netserver-2": networkPlugin cni failed to set up pod "netserver-2_nettest-3615" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 1 02:30:34.537: INFO: At 2023-01-01 02:25:51 +0000 UTC - event for netserver-0: {kubelet ip-172-20-33-66.eu-west-3.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "731b92ae91d6873abfaeb5ac1b2f7ae86f099873467cdee7182ca814cf0184c3" network for pod "netserver-0": networkPlugin cni failed to set up pod "netserver-0_nettest-3615" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 1 02:30:34.537: INFO: At 2023-01-01 02:25:51 +0000 UTC - event for netserver-2: {kubelet ip-172-20-47-178.eu-west-3.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "3f06c10dbede8b314a0ac11411d6a248fc52ad0fe651eeb67e1c455cf73a9195" network for pod "netserver-2": networkPlugin cni failed to set up pod "netserver-2_nettest-3615" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 1 02:30:34.537: INFO: At 2023-01-01 02:25:55 +0000 UTC - event for netserver-0: {kubelet ip-172-20-33-66.eu-west-3.compute.internal} FailedCreatePodSandBox: (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "081572f241bf189a116a4e36b20493fbe9187e855b05e9e0d5d7b209736b1266" network for pod "netserver-0": networkPlugin cni failed to set up pod "netserver-0_nettest-3615" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 1 02:30:34.537: INFO: At 2023-01-01 02:26:00 +0000 UTC - event for netserver-2: {kubelet ip-172-20-47-178.eu-west-3.compute.internal} FailedCreatePodSandBox: (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "1408096d6f27894fe28a81408b8b407d14b14604060df73f93541687eb33756f" network for pod "netserver-2": networkPlugin cni failed to set up pod "netserver-2_nettest-3615" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 1 02:30:34.642: INFO: POD NODE PHASE GRACE CONDITIONS Jan 1 02:30:34.642: INFO: netserver-0 ip-172-20-33-66.eu-west-3.compute.internal Pending [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2023-01-01 02:25:33 +0000 UTC } {Ready False 0001-01-01 00:00:00 +0000 UTC 2023-01-01 02:25:33 +0000 UTC ContainersNotReady containers with unready status: [webserver]} {ContainersReady False 0001-01-01 00:00:00 +0000 UTC 2023-01-01 02:25:33 +0000 UTC ContainersNotReady containers with unready status: [webserver]} {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2023-01-01 02:25:33 +0000 UTC }] Jan 1 02:30:34.642: INFO: netserver-1 ip-172-20-45-112.eu-west-3.compute.internal Running [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2023-01-01 02:25:33 +0000 UTC } {Ready True 0001-01-01 00:00:00 +0000 UTC 2023-01-01 02:25:54 +0000 UTC } {ContainersReady True 0001-01-01 00:00:00 +0000 UTC 2023-01-01 02:25:54 +0000 UTC } {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2023-01-01 02:25:33 +0000 UTC }] Jan 1 02:30:34.642: INFO: netserver-2 ip-172-20-47-178.eu-west-3.compute.internal Pending [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2023-01-01 02:25:33 +0000 UTC } {Ready False 0001-01-01 00:00:00 +0000 UTC 2023-01-01 02:25:33 +0000 UTC ContainersNotReady containers with unready status: [webserver]} {ContainersReady False 0001-01-01 00:00:00 +0000 UTC 2023-01-01 02:25:33 +0000 UTC ContainersNotReady containers with unready status: [webserver]} {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2023-01-01 02:25:33 +0000 UTC }] Jan 1 02:30:34.642: INFO: netserver-3 ip-172-20-49-224.eu-west-3.compute.internal Running [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2023-01-01 02:25:34 +0000 UTC } {Ready True 0001-01-01 00:00:00 +0000 UTC 2023-01-01 02:25:55 +0000 UTC } {ContainersReady True 0001-01-01 00:00:00 +0000 UTC 2023-01-01 02:25:55 +0000 UTC } {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2023-01-01 02:25:34 +0000 UTC }] Jan 1 02:30:34.642: INFO: Jan 1 02:30:34.756: INFO: Unable to fetch nettest-3615/netserver-0/webserver logs: the server rejected our request for an unknown reason (get pods netserver-0) Jan 1 02:30:34.974: INFO: Unable to fetch nettest-3615/netserver-2/webserver logs: the server rejected our request for an unknown reason (get pods netserver-2) Jan 1 02:30:35.210: INFO: Logging node info for node ip-172-20-33-66.eu-west-3.compute.internal Jan 1 02:30:35.315: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-33-66.eu-west-3.compute.internal ec4b59b3-62db-4040-aad2-4c3249f80fe8 6147 0 2023-01-01 02:20:25 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:eu-west-3 failure-domain.beta.kubernetes.io/zone:eu-west-3a kubelet_cleanup:true kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-33-66.eu-west-3.compute.internal kubernetes.io/os:linux kubernetes.io/role:node node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:eu-west-3a topology.kubernetes.io/region:eu-west-3 topology.kubernetes.io/zone:eu-west-3a] map[csi.volume.kubernetes.io/nodeid:{"csi-mock-csi-mock-volumes-4089":"csi-mock-csi-mock-volumes-4089","ebs.csi.aws.com":"i-028313673124ac4ad"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kops-controller Update v1 2023-01-01 02:20:25 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}} } {kube-controller-manager Update v1 2023-01-01 02:20:25 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.1.0/24\"":{}}}} } {kubelet Update v1 2023-01-01 02:20:25 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {e2e.test Update v1 2023-01-01 02:25:15 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubelet_cleanup":{}}}} } {kube-controller-manager Update v1 2023-01-01 02:25:34 +0000 UTC FieldsV1 {"f:status":{"f:volumesAttached":{}}} status} {kubelet Update v1 2023-01-01 02:25:41 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.1.0/24,DoNotUseExternalID:,ProviderID:aws:///eu-west-3a/i-028313673124ac4ad,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.1.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{50478186496 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4078190592 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{45430367772 0} {<nil>} 45430367772 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3973332992 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-01-01 02:25:41 +0000 UTC,LastTransitionTime:2023-01-01 02:20:25 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-01-01 02:25:41 +0000 UTC,LastTransitionTime:2023-01-01 02:20:25 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-01-01 02:25:41 +0000 UTC,LastTransitionTime:2023-01-01 02:20:25 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-01-01 02:25:41 +0000 UTC,LastTransitionTime:2023-01-01 02:20:46 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.33.66,},NodeAddress{Type:ExternalIP,Address:52.47.140.112,},NodeAddress{Type:Hostname,Address:ip-172-20-33-66.eu-west-3.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-33-66.eu-west-3.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-52-47-140-112.eu-west-3.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2ad2ce1a87d508c178acf6215c5767,SystemUUID:ec2ad2ce-1a87-d508-c178-acf6215c5767,BootID:b3b60783-d786-412b-a36a-802bcfae604d,KernelVersion:4.19.0-23-cloud-amd64,OSImage:Debian GNU/Linux 10 (buster),ContainerRuntimeVersion:docker://20.10.17,KubeletVersion:v1.23.15,KubeProxyVersion:v1.23.15,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:456569528,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/jessie-dnsutils@sha256:11e6a66017ba4e4b938c1612b7a54a3befcefd354796c04e1dba76873a13518e k8s.gcr.io/e2e-test-images/jessie-dnsutils:1.5],SizeBytes:253371792,},ContainerImage{Names:[nginx@sha256:0047b729188a15da49380d9506d65959cce6d40291ccfb4e039f5dc7efd33286 nginx:latest],SizeBytes:141812353,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:7e8bdd271312fd25fc5ff5a8f04727be84044eb3d7d8d03611972a6752e2e11e k8s.gcr.io/e2e-test-images/agnhost:2.39],SizeBytes:126872991,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:1b9d1b2f36cb2dbee1960e82a9344aeb11bd4c4c03abf5e1853e0559c23855e3 k8s.gcr.io/e2e-test-images/httpd:2.4.38-2],SizeBytes:123781643,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.23.15 registry.k8s.io/kube-proxy-amd64:v1.23.15],SizeBytes:112336036,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:10e231cf07c82dfab5141c1afe548dc734e5fc1f67665ec3982a325d1bd31a9a registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.0],SizeBytes:94471559,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:c8e03f60afa90a28e4bb6ec9a8d0fc36d89de4b7475cf2d613afa793ec969fe0 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.0],SizeBytes:56382329,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:c5be65d6679efabb969d9b019300d187437ae876f992c40911fd2892bbef3b36 k8s.gcr.io/sig-storage/csi-attacher:v3.2.0],SizeBytes:53500826,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:017727efcfeb7d053af68e51436ce8e65edbc6ca573720afb4f79c8594036955 registry.k8s.io/coredns/coredns:v1.10.0],SizeBytes:50249337,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:a61d309da54641db41fb8f35718f744e9f730d4d0384f8c4b186ddc9f06cbd5f k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.1.0],SizeBytes:19662887,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:19582112,},ContainerImage{Names:[k8s.gcr.io/sig-storage/mock-driver@sha256:a7b517f9e0f42ffade802eef9cefa271372386b85f55b702b493241e58459793 k8s.gcr.io/sig-storage/mock-driver:v4.1.0],SizeBytes:17680993,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:17611980,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:13616070e3f29de4417eee434a8ef472221c9e51b3d037b5a6b46cef08eb7443 k8s.gcr.io/e2e-test-images/nginx:1.14-2],SizeBytes:16032814,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nonewprivs@sha256:8ac1264691820febacf3aea5d152cbde6d10685731ec14966a9401c6f47a68ac k8s.gcr.io/e2e-test-images/nonewprivs:1.3],SizeBytes:7107254,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf k8s.gcr.io/e2e-test-images/busybox:1.29-2],SizeBytes:1154361,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db k8s.gcr.io/pause:3.6 registry.k8s.io/pause:3.6],SizeBytes:682696,},},VolumesInUse:[kubernetes.io/csi/csi-mock-csi-mock-volumes-4089^4],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/csi-mock-csi-mock-volumes-4089^4,DevicePath:,},},Config:nil,},} Jan 1 02:30:35.315: INFO: Logging kubelet events for node ip-172-20-33-66.eu-west-3.compute.internal Jan 1 02:30:35.422: INFO: Logging pods the kubelet thinks is on node ip-172-20-33-66.eu-west-3.compute.internal Jan 1 02:30:35.643: INFO: hostexec-ip-172-20-33-66.eu-west-3.compute.internal-cr6qn started at 2023-01-01 02:25:58 +0000 UTC (0+1 container statuses recorded) Jan 1 02:30:35.643: INFO: Container agnhost-container ready: true, restart count 0 Jan 1 02:30:35.643: INFO: local-injector started at 2023-01-01 02:26:06 +0000 UTC (0+1 container statuses recorded) Jan 1 02:30:35.643: INFO: Container local-injector ready: false, restart count 0 Jan 1 02:30:35.643: INFO: csi-mockplugin-0 started at 2023-01-01 02:24:59 +0000 UTC (0+3 container statuses recorded) Jan 1 02:30:35.643: INFO: Container csi-provisioner ready: true, restart count 0 Jan 1 02:30:35.643: INFO: Container driver-registrar ready: true, restart count 0 Jan 1 02:30:35.643: INFO: Container mock ready: true, restart count 0 Jan 1 02:30:35.643: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-csnzw started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:30:35.643: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: false, restart count 0 Jan 1 02:30:35.643: INFO: ebs-csi-node-vsxjc started at 2023-01-01 02:20:26 +0000 UTC (0+3 container statuses recorded) Jan 1 02:30:35.643: INFO: Container ebs-plugin ready: true, restart count 0 Jan 1 02:30:35.643: INFO: Container liveness-probe ready: true, restart count 0 Jan 1 02:30:35.643: INFO: Container node-driver-registrar ready: true, restart count 0 Jan 1 02:30:35.643: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-h4nvd started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:30:35.643: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:30:35.643: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-m6vq2 started at 2023-01-01 02:25:17 +0000 UTC (0+1 container statuses recorded) Jan 1 02:30:35.643: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: false, restart count 0 Jan 1 02:30:35.643: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-h6mmt started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:30:35.643: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:30:35.643: INFO: cilium-vbcgc started at 2023-01-01 02:20:26 +0000 UTC (1+1 container statuses recorded) Jan 1 02:30:35.643: INFO: Init container clean-cilium-state ready: true, restart count 0 Jan 1 02:30:35.643: INFO: Container cilium-agent ready: true, restart count 0 Jan 1 02:30:35.643: INFO: affinity-clusterip-timeout-q9qt2 started at 2023-01-01 02:27:32 +0000 UTC (0+1 container statuses recorded) Jan 1 02:30:35.643: INFO: Container affinity-clusterip-timeout ready: false, restart count 0 Jan 1 02:30:35.643: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-dkfpd started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:30:35.643: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:30:35.643: INFO: local-injector started at 2023-01-01 02:27:36 +0000 UTC (0+1 container statuses recorded) Jan 1 02:30:35.643: INFO: Container local-injector ready: false, restart count 0 Jan 1 02:30:35.643: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-546bc started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:30:35.643: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:30:35.643: INFO: csi-mockplugin-attacher-0 started at 2023-01-01 02:27:56 +0000 UTC (0+1 container statuses recorded) Jan 1 02:30:35.643: INFO: Container csi-attacher ready: false, restart count 0 Jan 1 02:30:35.643: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-q8pgp started at 2023-01-01 02:25:17 +0000 UTC (0+1 container statuses recorded) Jan 1 02:30:35.643: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:30:35.643: INFO: csi-mockplugin-0 started at 2023-01-01 02:27:56 +0000 UTC (0+3 container statuses recorded) Jan 1 02:30:35.643: INFO: Container csi-provisioner ready: false, restart count 0 Jan 1 02:30:35.643: INFO: Container driver-registrar ready: false, restart count 0 Jan 1 02:30:35.643: INFO: Container mock ready: false, restart count 0 Jan 1 02:30:35.644: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-2v87j started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:30:35.644: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:30:35.644: INFO: csi-mockplugin-attacher-0 started at 2023-01-01 02:25:00 +0000 UTC (0+1 container statuses recorded) Jan 1 02:30:35.644: INFO: Container csi-attacher ready: true, restart count 0 Jan 1 02:30:35.644: INFO: pvc-volume-tester-bc9c6 started at 2023-01-01 02:25:33 +0000 UTC (0+1 container statuses recorded) Jan 1 02:30:35.644: INFO: Container volume-tester ready: false, restart count 0 Jan 1 02:30:35.644: INFO: coredns-867df8f45c-cqmp7 started at 2023-01-01 02:21:18 +0000 UTC (0+1 container statuses recorded) Jan 1 02:30:35.644: INFO: Container coredns ready: true, restart count 0 Jan 1 02:30:35.644: INFO: netserver-0 started at 2023-01-01 02:25:27 +0000 UTC (0+1 container statuses recorded) Jan 1 02:30:35.644: INFO: Container webserver ready: true, restart count 0 Jan 1 02:30:35.644: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-l9hnv started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:30:35.644: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:30:35.644: INFO: hostexec-ip-172-20-33-66.eu-west-3.compute.internal-qm7jt started at 2023-01-01 02:27:26 +0000 UTC (0+1 container statuses recorded) Jan 1 02:30:35.644: INFO: Container agnhost-container ready: true, restart count 0 Jan 1 02:30:35.644: INFO: netserver-0 started at 2023-01-01 02:25:33 +0000 UTC (0+1 container statuses recorded) Jan 1 02:30:35.644: INFO: Container webserver ready: false, restart count 0 Jan 1 02:30:35.644: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-vnv2x started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:30:35.644: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:30:35.644: INFO: csi-mockplugin-0 started at 2023-01-01 02:26:15 +0000 UTC (0+4 container statuses recorded) Jan 1 02:30:35.644: INFO: Container busybox ready: false, restart count 0 Jan 1 02:30:35.644: INFO: Container csi-provisioner ready: false, restart count 0 Jan 1 02:30:35.644: INFO: Container driver-registrar ready: false, restart count 0 Jan 1 02:30:35.644: INFO: Container mock ready: false, restart count 0 Jan 1 02:30:36.214: INFO: Latency metrics for node ip-172-20-33-66.eu-west-3.compute.internal Jan 1 02:30:36.214: INFO: Logging node info for node ip-172-20-38-238.eu-west-3.compute.internal Jan 1 02:30:36.319: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-38-238.eu-west-3.compute.internal 0eb713b4-64bf-47b7-8090-bab8111218ee 6771 0 2023-01-01 02:18:51 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:c5.large beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:eu-west-3 failure-domain.beta.kubernetes.io/zone:eu-west-3a kops.k8s.io/kops-controller-pki: kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-38-238.eu-west-3.compute.internal kubernetes.io/os:linux kubernetes.io/role:master node-role.kubernetes.io/control-plane: node-role.kubernetes.io/master: node.kubernetes.io/exclude-from-external-load-balancers: node.kubernetes.io/instance-type:c5.large topology.ebs.csi.aws.com/zone:eu-west-3a topology.kubernetes.io/region:eu-west-3 topology.kubernetes.io/zone:eu-west-3a] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-0f70ebb30a9076648"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kubelet Update v1 2023-01-01 02:18:50 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {protokube Update v1 2023-01-01 02:19:16 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kops.k8s.io/kops-controller-pki":{},"f:node-role.kubernetes.io/control-plane":{},"f:node.kubernetes.io/exclude-from-external-load-balancers":{}}}} } {kops-controller Update v1 2023-01-01 02:19:21 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/master":{}}}} } {kube-controller-manager Update v1 2023-01-01 02:19:31 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.0.0/24\"":{}},"f:taints":{}}} } {kubelet Update v1 2023-01-01 02:20:36 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.0.0/24,DoNotUseExternalID:,ProviderID:aws:///eu-west-3a/i-0f70ebb30a9076648,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/master,Value:,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[100.96.0.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{50478186496 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3918811136 0} {<nil>} 3826964Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{45430367772 0} {<nil>} 45430367772 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3813953536 0} {<nil>} 3724564Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-01-01 02:26:20 +0000 UTC,LastTransitionTime:2023-01-01 02:18:45 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-01-01 02:26:20 +0000 UTC,LastTransitionTime:2023-01-01 02:18:45 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-01-01 02:26:20 +0000 UTC,LastTransitionTime:2023-01-01 02:18:45 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-01-01 02:26:20 +0000 UTC,LastTransitionTime:2023-01-01 02:19:31 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.38.238,},NodeAddress{Type:ExternalIP,Address:15.237.26.58,},NodeAddress{Type:Hostname,Address:ip-172-20-38-238.eu-west-3.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-38-238.eu-west-3.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-15-237-26-58.eu-west-3.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2e6c64b5b811bf0c78b2aa7ccdb943,SystemUUID:ec2e6c64-b5b8-11bf-0c78-b2aa7ccdb943,BootID:9db01fa3-d136-4230-b21a-b03abbfb6bff,KernelVersion:4.19.0-23-cloud-amd64,OSImage:Debian GNU/Linux 10 (buster),ContainerRuntimeVersion:docker://20.10.17,KubeletVersion:v1.23.15,KubeProxyVersion:v1.23.15,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[registry.k8s.io/etcdadm/etcd-manager@sha256:66a453db625abb268f4b3bbefc5a34a171d81e6e8796cecca54cfd71775c77c4 registry.k8s.io/etcdadm/etcd-manager:v3.0.20221209],SizeBytes:662209237,},ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:456569528,},ContainerImage{Names:[k8s.gcr.io/kube-apiserver-amd64:v1.23.15 registry.k8s.io/kube-apiserver-amd64:v1.23.15],SizeBytes:135240444,},ContainerImage{Names:[k8s.gcr.io/kube-controller-manager-amd64:v1.23.15 registry.k8s.io/kube-controller-manager-amd64:v1.23.15],SizeBytes:125046097,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.23.15 registry.k8s.io/kube-proxy-amd64:v1.23.15],SizeBytes:112336036,},ContainerImage{Names:[quay.io/cilium/operator@sha256:a6d24a006a6b92967ac90786b49bc1ac26e5477cf028cd1186efcfc2466484db quay.io/cilium/operator:v1.12.5],SizeBytes:110016956,},ContainerImage{Names:[registry.k8s.io/kops/kops-controller:1.27.0-alpha.1],SizeBytes:105197757,},ContainerImage{Names:[registry.k8s.io/kops/dns-controller:1.27.0-alpha.1],SizeBytes:102577531,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:10e231cf07c82dfab5141c1afe548dc734e5fc1f67665ec3982a325d1bd31a9a registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.0],SizeBytes:94471559,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:122bfb8c1edabb3c0edd63f06523e6940d958d19b3957dc7b1d6f81e9f1f6119 registry.k8s.io/sig-storage/csi-provisioner:v3.1.0],SizeBytes:57736070,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:9ebbf9f023e7b41ccee3d52afe39a89e3ddacdbb69269d583abfc25847cfd9e4 registry.k8s.io/sig-storage/csi-resizer:v1.4.0],SizeBytes:55464778,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:8b9c313c05f54fb04f8d430896f5f5904b6cb157df261501b29adc04d2b2dc7b registry.k8s.io/sig-storage/csi-attacher:v3.4.0],SizeBytes:54848595,},ContainerImage{Names:[k8s.gcr.io/kube-scheduler-amd64:v1.23.15 registry.k8s.io/kube-scheduler-amd64:v1.23.15],SizeBytes:53521430,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:19582112,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:17611980,},ContainerImage{Names:[registry.k8s.io/kops/kube-apiserver-healthcheck:1.27.0-alpha.1],SizeBytes:8787111,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:682696,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Jan 1 02:30:36.319: INFO: Logging kubelet events for node ip-172-20-38-238.eu-west-3.compute.internal Jan 1 02:30:36.426: INFO: Logging pods the kubelet thinks is on node ip-172-20-38-238.eu-west-3.compute.internal Jan 1 02:30:36.537: INFO: etcd-manager-main-ip-172-20-38-238.eu-west-3.compute.internal started at 2023-01-01 02:18:11 +0000 UTC (0+1 container statuses recorded) Jan 1 02:30:36.537: INFO: Container etcd-manager ready: true, restart count 0 Jan 1 02:30:36.537: INFO: cilium-qk59v started at 2023-01-01 02:19:20 +0000 UTC (1+1 container statuses recorded) Jan 1 02:30:36.537: INFO: Init container clean-cilium-state ready: true, restart count 0 Jan 1 02:30:36.537: INFO: Container cilium-agent ready: true, restart count 0 Jan 1 02:30:36.537: INFO: ebs-csi-node-k87k6 started at 2023-01-01 02:19:20 +0000 UTC (0+3 container statuses recorded) Jan 1 02:30:36.537: INFO: Container ebs-plugin ready: true, restart count 0 Jan 1 02:30:36.537: INFO: Container liveness-probe ready: true, restart count 0 Jan 1 02:30:36.537: INFO: Container node-driver-registrar ready: true, restart count 0 Jan 1 02:30:36.537: INFO: dns-controller-7db8b74b99-58zzw started at 2023-01-01 02:19:20 +0000 UTC (0+1 container statuses recorded) Jan 1 02:30:36.537: INFO: Container dns-controller ready: true, restart count 0 Jan 1 02:30:36.537: INFO: cilium-operator-d47487c76-j84nq started at 2023-01-01 02:19:20 +0000 UTC (0+1 container statuses recorded) Jan 1 02:30:36.537: INFO: Container cilium-operator ready: true, restart count 0 Jan 1 02:30:36.537: INFO: etcd-manager-events-ip-172-20-38-238.eu-west-3.compute.internal started at 2023-01-01 02:18:11 +0000 UTC (0+1 container statuses recorded) Jan 1 02:30:36.537: INFO: Container etcd-manager ready: true, restart count 0 Jan 1 02:30:36.537: INFO: kube-scheduler-ip-172-20-38-238.eu-west-3.compute.internal started at 2023-01-01 02:18:11 +0000 UTC (0+1 container statuses recorded) Jan 1 02:30:36.537: INFO: Container kube-scheduler ready: true, restart count 0 Jan 1 02:30:36.537: INFO: kube-apiserver-ip-172-20-38-238.eu-west-3.compute.internal started at 2023-01-01 02:18:11 +0000 UTC (0+2 container statuses recorded) Jan 1 02:30:36.537: INFO: Container healthcheck ready: true, restart count 0 Jan 1 02:30:36.537: INFO: Container kube-apiserver ready: true, restart count 1 Jan 1 02:30:36.537: INFO: kops-controller-26gzg started at 2023-01-01 02:19:20 +0000 UTC (0+1 container statuses recorded) Jan 1 02:30:36.537: INFO: Container kops-controller ready: true, restart count 0 Jan 1 02:30:36.537: INFO: ebs-csi-controller-6dc545b75-5qqh8 started at 2023-01-01 02:19:20 +0000 UTC (0+5 container statuses recorded) Jan 1 02:30:36.537: INFO: Container csi-attacher ready: true, restart count 0 Jan 1 02:30:36.537: INFO: Container csi-provisioner ready: true, restart count 0 Jan 1 02:30:36.537: INFO: Container csi-resizer ready: true, restart count 0 Jan 1 02:30:36.537: INFO: Container ebs-plugin ready: true, restart count 0 Jan 1 02:30:36.537: INFO: Container liveness-probe ready: true, restart count 0 Jan 1 02:30:36.537: INFO: kube-controller-manager-ip-172-20-38-238.eu-west-3.compute.internal started at 2023-01-01 02:18:11 +0000 UTC (0+1 container statuses recorded) Jan 1 02:30:36.537: INFO: Container kube-controller-manager ready: true, restart count 2 Jan 1 02:30:36.967: INFO: Latency metrics for node ip-172-20-38-238.eu-west-3.compute.internal Jan 1 02:30:36.967: INFO: Logging node info for node ip-172-20-45-112.eu-west-3.compute.internal Jan 1 02:30:37.072: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-45-112.eu-west-3.compute.internal 48eed937-05bb-4d8a-bddd-aa94a2e6ab92 6756 0 2023-01-01 02:20:27 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:eu-west-3 failure-domain.beta.kubernetes.io/zone:eu-west-3a kubelet_cleanup:true kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-45-112.eu-west-3.compute.internal kubernetes.io/os:linux kubernetes.io/role:node node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:eu-west-3a topology.hostpath.csi/node:ip-172-20-45-112.eu-west-3.compute.internal topology.kubernetes.io/region:eu-west-3 topology.kubernetes.io/zone:eu-west-3a] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-0778adf2085fa7f34"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kube-controller-manager Update v1 2023-01-01 02:20:27 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.3.0/24\"":{}}}} } {kubelet Update v1 2023-01-01 02:20:27 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {kops-controller Update v1 2023-01-01 02:20:28 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}} } {e2e.test Update v1 2023-01-01 02:25:15 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubelet_cleanup":{}}}} } {kubelet Update v1 2023-01-01 02:26:05 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:capacity":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.3.0/24,DoNotUseExternalID:,ProviderID:aws:///eu-west-3a/i-0778adf2085fa7f34,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.3.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{50478186496 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4078190592 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{45430367772 0} {<nil>} 45430367772 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3973332992 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-01-01 02:26:05 +0000 UTC,LastTransitionTime:2023-01-01 02:20:27 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-01-01 02:26:05 +0000 UTC,LastTransitionTime:2023-01-01 02:20:27 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-01-01 02:26:05 +0000 UTC,LastTransitionTime:2023-01-01 02:20:27 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-01-01 02:26:05 +0000 UTC,LastTransitionTime:2023-01-01 02:20:48 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.45.112,},NodeAddress{Type:ExternalIP,Address:13.38.32.14,},NodeAddress{Type:Hostname,Address:ip-172-20-45-112.eu-west-3.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-45-112.eu-west-3.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-13-38-32-14.eu-west-3.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2dbdf846e8ff614f4555e5bf9dbbf4,SystemUUID:ec2dbdf8-46e8-ff61-4f45-55e5bf9dbbf4,BootID:10a054ae-621a-47ce-9566-b0048cc5101b,KernelVersion:4.19.0-23-cloud-amd64,OSImage:Debian GNU/Linux 10 (buster),ContainerRuntimeVersion:docker://20.10.17,KubeletVersion:v1.23.15,KubeProxyVersion:v1.23.15,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:456569528,},ContainerImage{Names:[nginx@sha256:0047b729188a15da49380d9506d65959cce6d40291ccfb4e039f5dc7efd33286 nginx:latest],SizeBytes:141812353,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:7e8bdd271312fd25fc5ff5a8f04727be84044eb3d7d8d03611972a6752e2e11e k8s.gcr.io/e2e-test-images/agnhost:2.39],SizeBytes:126872991,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:1b9d1b2f36cb2dbee1960e82a9344aeb11bd4c4c03abf5e1853e0559c23855e3 k8s.gcr.io/e2e-test-images/httpd:2.4.38-2],SizeBytes:123781643,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.23.15 registry.k8s.io/kube-proxy-amd64:v1.23.15],SizeBytes:112336036,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:10e231cf07c82dfab5141c1afe548dc734e5fc1f67665ec3982a325d1bd31a9a registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.0],SizeBytes:94471559,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:c8e03f60afa90a28e4bb6ec9a8d0fc36d89de4b7475cf2d613afa793ec969fe0 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.0],SizeBytes:56382329,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:4e74c0492bceddc598de1c90cc5bc14dcda94cb49fa9c5bad9d117c4834b5e08 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.1],SizeBytes:56379269,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:36c31f7e1f433c9634d24f876353e8646246d81a03c4e351202c2644daff1620 k8s.gcr.io/sig-storage/csi-resizer:v1.2.0],SizeBytes:53956308,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:ed98431376c9e944e19a465fe8ea944806714dd95416a0821096c78d66b579bd k8s.gcr.io/sig-storage/csi-snapshotter:v4.1.1],SizeBytes:53531990,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:c5be65d6679efabb969d9b019300d187437ae876f992c40911fd2892bbef3b36 k8s.gcr.io/sig-storage/csi-attacher:v3.2.0],SizeBytes:53500826,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:60ab9b3e6a030d3038c87c0d6bca2930f58d1d72823e6a4af09767dc83b696a2 k8s.gcr.io/sig-storage/csi-attacher:v3.2.1],SizeBytes:53499570,},ContainerImage{Names:[k8s.gcr.io/sig-storage/hostpathplugin@sha256:232fe80174d60d520d36043103853a1d7ab4b7f3782cf43e45034f04ccda58ce k8s.gcr.io/sig-storage/hostpathplugin:v1.7.1],SizeBytes:29977921,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:a61d309da54641db41fb8f35718f744e9f730d4d0384f8c4b186ddc9f06cbd5f k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.1.0],SizeBytes:19662887,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:19582112,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:2dee3fe5fe861bb66c3a4ac51114f3447a4cd35870e0f2e2b558c7a400d89589 k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.2.0],SizeBytes:18699436,},ContainerImage{Names:[k8s.gcr.io/sig-storage/mock-driver@sha256:a7b517f9e0f42ffade802eef9cefa271372386b85f55b702b493241e58459793 k8s.gcr.io/sig-storage/mock-driver:v4.1.0],SizeBytes:17680993,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:17611980,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:529be2c9770add0cdd0c989115222ea9fc1be430c11095eb9f6dafcf98a36e2b k8s.gcr.io/sig-storage/livenessprobe:v2.4.0],SizeBytes:17175488,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:13616070e3f29de4417eee434a8ef472221c9e51b3d037b5a6b46cef08eb7443 k8s.gcr.io/e2e-test-images/nginx:1.14-2],SizeBytes:16032814,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf k8s.gcr.io/e2e-test-images/busybox:1.29-2],SizeBytes:1154361,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:39e1e963e5310e9c313bad51523be012ede7b35bb9316517d19089a010356592 k8s.gcr.io/e2e-test-images/busybox:1.29-1],SizeBytes:1154361,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db k8s.gcr.io/pause:3.6 registry.k8s.io/pause:3.6],SizeBytes:682696,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Jan 1 02:30:37.072: INFO: Logging kubelet events for node ip-172-20-45-112.eu-west-3.compute.internal Jan 1 02:30:37.179: INFO: Logging pods the kubelet thinks is on node ip-172-20-45-112.eu-west-3.compute.internal Jan 1 02:30:37.300: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-gph2d started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:30:37.300: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:30:37.300: INFO: pod-39bf2777-905c-4d8b-97f4-bf3626087647 started at 2023-01-01 02:25:44 +0000 UTC (0+1 container statuses recorded) Jan 1 02:30:37.300: INFO: Container write-pod ready: false, restart count 0 Jan 1 02:30:37.300: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-59bdj started at 2023-01-01 02:25:17 +0000 UTC (0+1 container statuses recorded) Jan 1 02:30:37.300: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:30:37.300: INFO: ebs-csi-node-7clrr started at 2023-01-01 02:20:28 +0000 UTC (0+3 container statuses recorded) Jan 1 02:30:37.300: INFO: Container ebs-plugin ready: true, restart count 0 Jan 1 02:30:37.300: INFO: Container liveness-probe ready: true, restart count 0 Jan 1 02:30:37.300: INFO: Container node-driver-registrar ready: true, restart count 0 Jan 1 02:30:37.300: INFO: netserver-1 started at 2023-01-01 02:25:27 +0000 UTC (0+1 container statuses recorded) Jan 1 02:30:37.300: INFO: Container webserver ready: true, restart count 0 Jan 1 02:30:37.300: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-rmdrr started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:30:37.300: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:30:37.300: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-kjg2m started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:30:37.300: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:30:37.300: INFO: netserver-1 started at 2023-01-01 02:25:33 +0000 UTC (0+1 container statuses recorded) Jan 1 02:30:37.300: INFO: Container webserver ready: true, restart count 0 Jan 1 02:30:37.300: INFO: security-context-2da5e8a9-2026-481a-8c39-7b479c31bda0 started at 2023-01-01 02:25:47 +0000 UTC (0+1 container statuses recorded) Jan 1 02:30:37.300: INFO: Container test-container ready: false, restart count 0 Jan 1 02:30:37.300: INFO: hostexec-ip-172-20-45-112.eu-west-3.compute.internal-22hvx started at 2023-01-01 02:25:34 +0000 UTC (0+1 container statuses recorded) Jan 1 02:30:37.300: INFO: Container agnhost-container ready: true, restart count 0 Jan 1 02:30:37.300: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-jh8bc started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:30:37.300: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:30:37.300: INFO: cilium-slx98 started at 2023-01-01 02:20:28 +0000 UTC (1+1 container statuses recorded) Jan 1 02:30:37.300: INFO: Init container clean-cilium-state ready: true, restart count 0 Jan 1 02:30:37.300: INFO: Container cilium-agent ready: true, restart count 0 Jan 1 02:30:37.300: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-4lrj2 started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:30:37.300: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:30:37.300: INFO: affinity-clusterip-timeout-lv5vx started at 2023-01-01 02:27:32 +0000 UTC (0+1 container statuses recorded) Jan 1 02:30:37.300: INFO: Container affinity-clusterip-timeout ready: false, restart count 0 Jan 1 02:30:37.300: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-srf5k started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:30:37.300: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:30:37.300: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-4mml9 started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:30:37.300: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:30:37.300: INFO: pod-exec-websocket-4b20549e-25f0-4f45-bfef-409447a30721 started at 2023-01-01 02:25:37 +0000 UTC (0+1 container statuses recorded) Jan 1 02:30:37.300: INFO: Container main ready: false, restart count 0 Jan 1 02:30:37.300: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-x5cp6 started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:30:37.300: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:30:37.300: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-xnfzj started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:30:37.300: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:30:37.300: INFO: test-recreate-deployment-594f666cd9-5pbpl started at 2023-01-01 02:27:48 +0000 UTC (0+1 container statuses recorded) Jan 1 02:30:37.300: INFO: Container agnhost ready: false, restart count 0 Jan 1 02:30:37.659: INFO: Latency metrics for node ip-172-20-45-112.eu-west-3.compute.internal Jan 1 02:30:37.659: INFO: Logging node info for node ip-172-20-47-178.eu-west-3.compute.internal Jan 1 02:30:37.769: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-47-178.eu-west-3.compute.internal 835e5337-d2b9-45ad-bf0e-9d32514f71ba 6172 0 2023-01-01 02:20:25 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:eu-west-3 failure-domain.beta.kubernetes.io/zone:eu-west-3a kubelet_cleanup:true kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-47-178.eu-west-3.compute.internal kubernetes.io/os:linux kubernetes.io/role:node node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:eu-west-3a topology.kubernetes.io/region:eu-west-3 topology.kubernetes.io/zone:eu-west-3a] map[csi.volume.kubernetes.io/nodeid:{"csi-mock-csi-mock-volumes-5429":"csi-mock-csi-mock-volumes-5429","ebs.csi.aws.com":"i-00e1e60199f5a4d04"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kops-controller Update v1 2023-01-01 02:20:25 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}} } {kube-controller-manager Update v1 2023-01-01 02:20:25 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.2.0/24\"":{}}}} } {kubelet Update v1 2023-01-01 02:20:25 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {e2e.test Update v1 2023-01-01 02:25:15 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubelet_cleanup":{}}}} } {kubelet Update v1 2023-01-01 02:25:42 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:capacity":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.2.0/24,DoNotUseExternalID:,ProviderID:aws:///eu-west-3a/i-00e1e60199f5a4d04,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.2.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{50478186496 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4078190592 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{45430367772 0} {<nil>} 45430367772 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3973332992 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-01-01 02:25:42 +0000 UTC,LastTransitionTime:2023-01-01 02:20:25 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-01-01 02:25:42 +0000 UTC,LastTransitionTime:2023-01-01 02:20:25 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-01-01 02:25:42 +0000 UTC,LastTransitionTime:2023-01-01 02:20:25 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-01-01 02:25:42 +0000 UTC,LastTransitionTime:2023-01-01 02:20:46 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.47.178,},NodeAddress{Type:ExternalIP,Address:13.36.177.101,},NodeAddress{Type:Hostname,Address:ip-172-20-47-178.eu-west-3.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-47-178.eu-west-3.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-13-36-177-101.eu-west-3.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec243eda226c96d5573f3eaa3203df0d,SystemUUID:ec243eda-226c-96d5-573f-3eaa3203df0d,BootID:15dccaa7-d7c7-419d-b7de-76be5781a51a,KernelVersion:4.19.0-23-cloud-amd64,OSImage:Debian GNU/Linux 10 (buster),ContainerRuntimeVersion:docker://20.10.17,KubeletVersion:v1.23.15,KubeProxyVersion:v1.23.15,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:456569528,},ContainerImage{Names:[nginx@sha256:0047b729188a15da49380d9506d65959cce6d40291ccfb4e039f5dc7efd33286 nginx:latest],SizeBytes:141812353,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:7e8bdd271312fd25fc5ff5a8f04727be84044eb3d7d8d03611972a6752e2e11e k8s.gcr.io/e2e-test-images/agnhost:2.39],SizeBytes:126872991,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.23.15 registry.k8s.io/kube-proxy-amd64:v1.23.15],SizeBytes:112336036,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:10e231cf07c82dfab5141c1afe548dc734e5fc1f67665ec3982a325d1bd31a9a registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.0],SizeBytes:94471559,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:c8e03f60afa90a28e4bb6ec9a8d0fc36d89de4b7475cf2d613afa793ec969fe0 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.0],SizeBytes:56382329,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:c5be65d6679efabb969d9b019300d187437ae876f992c40911fd2892bbef3b36 k8s.gcr.io/sig-storage/csi-attacher:v3.2.0],SizeBytes:53500826,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:017727efcfeb7d053af68e51436ce8e65edbc6ca573720afb4f79c8594036955 registry.k8s.io/coredns/coredns:v1.10.0],SizeBytes:50249337,},ContainerImage{Names:[registry.k8s.io/cpa/cluster-proportional-autoscaler@sha256:68d396900aeaa072c1f27289485fdac29834045a6f3ffe369bf389d830ef572d registry.k8s.io/cpa/cluster-proportional-autoscaler:1.8.6],SizeBytes:47910609,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:a61d309da54641db41fb8f35718f744e9f730d4d0384f8c4b186ddc9f06cbd5f k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.1.0],SizeBytes:19662887,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:19582112,},ContainerImage{Names:[k8s.gcr.io/sig-storage/mock-driver@sha256:a7b517f9e0f42ffade802eef9cefa271372386b85f55b702b493241e58459793 k8s.gcr.io/sig-storage/mock-driver:v4.1.0],SizeBytes:17680993,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:17611980,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:13616070e3f29de4417eee434a8ef472221c9e51b3d037b5a6b46cef08eb7443 k8s.gcr.io/e2e-test-images/nginx:1.14-2],SizeBytes:16032814,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf k8s.gcr.io/e2e-test-images/busybox:1.29-2],SizeBytes:1154361,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db k8s.gcr.io/pause:3.6 registry.k8s.io/pause:3.6],SizeBytes:682696,},},VolumesInUse:[kubernetes.io/csi/csi-mock-csi-mock-volumes-5429^4],VolumesAttached:[]AttachedVolume{},Config:nil,},} Jan 1 02:30:37.770: INFO: Logging kubelet events for node ip-172-20-47-178.eu-west-3.compute.internal Jan 1 02:30:37.876: INFO: Logging pods the kubelet thinks is on node ip-172-20-47-178.eu-west-3.compute.internal Jan 1 02:30:37.991: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-grrsl started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:30:37.991: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:30:37.991: INFO: exec-volume-test-preprovisionedpv-w6ww started at 2023-01-01 02:25:36 +0000 UTC (0+1 container statuses recorded) Jan 1 02:30:37.991: INFO: Container exec-container-preprovisionedpv-w6ww ready: false, restart count 0 Jan 1 02:30:37.991: INFO: csi-mockplugin-0 started at 2023-01-01 02:28:39 +0000 UTC (0+3 container statuses recorded) Jan 1 02:30:37.991: INFO: Container csi-provisioner ready: false, restart count 0 Jan 1 02:30:37.991: INFO: Container driver-registrar ready: false, restart count 0 Jan 1 02:30:37.991: INFO: Container mock ready: false, restart count 0 Jan 1 02:30:37.991: INFO: ebs-csi-node-vcprc started at 2023-01-01 02:20:26 +0000 UTC (0+3 container statuses recorded) Jan 1 02:30:37.991: INFO: Container ebs-plugin ready: true, restart count 0 Jan 1 02:30:37.991: INFO: Container liveness-probe ready: true, restart count 0 Jan 1 02:30:37.991: INFO: Container node-driver-registrar ready: true, restart count 0 Jan 1 02:30:37.991: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-49brr started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:30:37.991: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:30:37.991: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-z87gr started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:30:37.991: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:30:37.991: INFO: csi-mockplugin-attacher-0 started at 2023-01-01 02:25:20 +0000 UTC (0+1 container statuses recorded) Jan 1 02:30:37.991: INFO: Container csi-attacher ready: false, restart count 0 Jan 1 02:30:37.991: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-wjz96 started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:30:37.991: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:30:37.991: INFO: csi-mockplugin-0 started at 2023-01-01 02:25:20 +0000 UTC (0+3 container statuses recorded) Jan 1 02:30:37.991: INFO: Container csi-provisioner ready: true, restart count 0 Jan 1 02:30:37.991: INFO: Container driver-registrar ready: true, restart count 0 Jan 1 02:30:37.991: INFO: Container mock ready: true, restart count 0 Jan 1 02:30:37.991: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-w26gj started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:30:37.991: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:30:37.991: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-s4rqf started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:30:37.991: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:30:37.991: INFO: netserver-2 started at 2023-01-01 02:25:33 +0000 UTC (0+1 container statuses recorded) Jan 1 02:30:37.991: INFO: Container webserver ready: false, restart count 0 Jan 1 02:30:37.991: INFO: pvc-volume-tester-w4w4d started at 2023-01-01 02:25:38 +0000 UTC (0+1 container statuses recorded) Jan 1 02:30:37.991: INFO: Container volume-tester ready: false, restart count 0 Jan 1 02:30:37.991: INFO: pod-subpath-test-preprovisionedpv-jphj started at 2023-01-01 02:25:52 +0000 UTC (1+1 container statuses recorded) Jan 1 02:30:37.991: INFO: Init container init-volume-preprovisionedpv-jphj ready: false, restart count 0 Jan 1 02:30:37.991: INFO: Container test-container-subpath-preprovisionedpv-jphj ready: false, restart count 0 Jan 1 02:30:37.991: INFO: cilium-kk2c2 started at 2023-01-01 02:20:26 +0000 UTC (1+1 container statuses recorded) Jan 1 02:30:37.991: INFO: Init container clean-cilium-state ready: true, restart count 0 Jan 1 02:30:37.991: INFO: Container cilium-agent ready: true, restart count 0 Jan 1 02:30:37.991: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-6vhtn started at 2023-01-01 02:25:17 +0000 UTC (0+1 container statuses recorded) Jan 1 02:30:37.991: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: false, restart count 0 Jan 1 02:30:37.991: INFO: hostexec-ip-172-20-47-178.eu-west-3.compute.internal-qnn5v started at 2023-01-01 02:25:18 +0000 UTC (0+1 container statuses recorded) Jan 1 02:30:37.991: INFO: Container agnhost-container ready: true, restart count 0 Jan 1 02:30:37.991: INFO: coredns-867df8f45c-wzntb started at 2023-01-01 02:20:46 +0000 UTC (0+1 container statuses recorded) Jan 1 02:30:37.991: INFO: Container coredns ready: true, restart count 0 Jan 1 02:30:37.991: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-wkclp started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:30:37.991: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:30:37.991: INFO: coredns-autoscaler-557ccb4c66-k4lzb started at 2023-01-01 02:20:46 +0000 UTC (0+1 container statuses recorded) Jan 1 02:30:37.991: INFO: Container autoscaler ready: true, restart count 0 Jan 1 02:30:37.991: INFO: csi-mockplugin-attacher-0 started at 2023-01-01 02:28:39 +0000 UTC (0+1 container statuses recorded) Jan 1 02:30:37.991: INFO: Container csi-attacher ready: false, restart count 0 Jan 1 02:30:37.991: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-pgkm2 started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:30:37.991: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:30:37.991: INFO: hostexec-ip-172-20-47-178.eu-west-3.compute.internal-nf87h started at 2023-01-01 02:25:17 +0000 UTC (0+1 container statuses recorded) Jan 1 02:30:37.991: INFO: Container agnhost-container ready: true, restart count 0 Jan 1 02:30:37.991: INFO: netserver-2 started at 2023-01-01 02:25:27 +0000 UTC (0+1 container statuses recorded) Jan 1 02:30:37.991: INFO: Container webserver ready: false, restart count 0 Jan 1 02:30:37.991: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-hrgrq started at 2023-01-01 02:25:17 +0000 UTC (0+1 container statuses recorded) Jan 1 02:30:37.991: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:30:38.335: INFO: Latency metrics for node ip-172-20-47-178.eu-west-3.compute.internal Jan 1 02:30:38.335: INFO: Logging node info for node ip-172-20-49-224.eu-west-3.compute.internal Jan 1 02:30:38.439: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-49-224.eu-west-3.compute.internal d7d57146-c54a-4155-94f8-0f322216bfcf 6403 0 2023-01-01 02:20:28 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:eu-west-3 failure-domain.beta.kubernetes.io/zone:eu-west-3a kubelet_cleanup:true kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-49-224.eu-west-3.compute.internal kubernetes.io/os:linux kubernetes.io/role:node node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:eu-west-3a topology.hostpath.csi/node:ip-172-20-49-224.eu-west-3.compute.internal topology.kubernetes.io/region:eu-west-3 topology.kubernetes.io/zone:eu-west-3a] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-089a107939b27c28a"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kops-controller Update v1 2023-01-01 02:20:28 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}} } {kube-controller-manager Update v1 2023-01-01 02:20:28 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.4.0/24\"":{}}}} } {kubelet Update v1 2023-01-01 02:20:28 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {e2e.test Update v1 2023-01-01 02:25:15 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubelet_cleanup":{}}}} } {kubelet Update v1 2023-01-01 02:25:35 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:capacity":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.4.0/24,DoNotUseExternalID:,ProviderID:aws:///eu-west-3a/i-089a107939b27c28a,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.4.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{50478186496 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4078206976 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{45430367772 0} {<nil>} 45430367772 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3973349376 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-01-01 02:25:35 +0000 UTC,LastTransitionTime:2023-01-01 02:20:28 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-01-01 02:25:35 +0000 UTC,LastTransitionTime:2023-01-01 02:20:28 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-01-01 02:25:35 +0000 UTC,LastTransitionTime:2023-01-01 02:20:28 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-01-01 02:25:35 +0000 UTC,LastTransitionTime:2023-01-01 02:20:48 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.49.224,},NodeAddress{Type:ExternalIP,Address:15.237.95.101,},NodeAddress{Type:Hostname,Address:ip-172-20-49-224.eu-west-3.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-49-224.eu-west-3.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-15-237-95-101.eu-west-3.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2d0a1bb33dd26f5a12c3896530131b,SystemUUID:ec2d0a1b-b33d-d26f-5a12-c3896530131b,BootID:9f64e91d-26f0-45aa-aec4-a54433e67034,KernelVersion:4.19.0-23-cloud-amd64,OSImage:Debian GNU/Linux 10 (buster),ContainerRuntimeVersion:docker://20.10.17,KubeletVersion:v1.23.15,KubeProxyVersion:v1.23.15,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:456569528,},ContainerImage{Names:[nginx@sha256:0047b729188a15da49380d9506d65959cce6d40291ccfb4e039f5dc7efd33286 nginx:latest],SizeBytes:141812353,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:7e8bdd271312fd25fc5ff5a8f04727be84044eb3d7d8d03611972a6752e2e11e k8s.gcr.io/e2e-test-images/agnhost:2.39],SizeBytes:126872991,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:1b9d1b2f36cb2dbee1960e82a9344aeb11bd4c4c03abf5e1853e0559c23855e3 k8s.gcr.io/e2e-test-images/httpd:2.4.38-2],SizeBytes:123781643,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.23.15 registry.k8s.io/kube-proxy-amd64:v1.23.15],SizeBytes:112336036,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:10e231cf07c82dfab5141c1afe548dc734e5fc1f67665ec3982a325d1bd31a9a registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.0],SizeBytes:94471559,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:4e74c0492bceddc598de1c90cc5bc14dcda94cb49fa9c5bad9d117c4834b5e08 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.1],SizeBytes:56379269,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:36c31f7e1f433c9634d24f876353e8646246d81a03c4e351202c2644daff1620 k8s.gcr.io/sig-storage/csi-resizer:v1.2.0],SizeBytes:53956308,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:ed98431376c9e944e19a465fe8ea944806714dd95416a0821096c78d66b579bd k8s.gcr.io/sig-storage/csi-snapshotter:v4.1.1],SizeBytes:53531990,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:60ab9b3e6a030d3038c87c0d6bca2930f58d1d72823e6a4af09767dc83b696a2 k8s.gcr.io/sig-storage/csi-attacher:v3.2.1],SizeBytes:53499570,},ContainerImage{Names:[k8s.gcr.io/sig-storage/hostpathplugin@sha256:232fe80174d60d520d36043103853a1d7ab4b7f3782cf43e45034f04ccda58ce k8s.gcr.io/sig-storage/hostpathplugin:v1.7.1],SizeBytes:29977921,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:19582112,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:2dee3fe5fe861bb66c3a4ac51114f3447a4cd35870e0f2e2b558c7a400d89589 k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.2.0],SizeBytes:18699436,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:17611980,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:529be2c9770add0cdd0c989115222ea9fc1be430c11095eb9f6dafcf98a36e2b k8s.gcr.io/sig-storage/livenessprobe:v2.4.0],SizeBytes:17175488,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:13616070e3f29de4417eee434a8ef472221c9e51b3d037b5a6b46cef08eb7443 k8s.gcr.io/e2e-test-images/nginx:1.14-2],SizeBytes:16032814,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf k8s.gcr.io/e2e-test-images/busybox:1.29-2],SizeBytes:1154361,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db k8s.gcr.io/pause:3.6 registry.k8s.io/pause:3.6],SizeBytes:682696,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Jan 1 02:30:38.440: INFO: Logging kubelet events for node ip-172-20-49-224.eu-west-3.compute.internal Jan 1 02:30:38.547: INFO: Logging pods the kubelet thinks is on node ip-172-20-49-224.eu-west-3.compute.internal Jan 1 02:30:38.661: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-pgc9g started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:30:38.661: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:30:38.661: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-29cdj started at 2023-01-01 02:25:17 +0000 UTC (0+1 container statuses recorded) Jan 1 02:30:38.661: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: false, restart count 0 Jan 1 02:30:38.661: INFO: echo-sourceip started at 2023-01-01 02:26:27 +0000 UTC (0+1 container statuses recorded) Jan 1 02:30:38.661: INFO: Container agnhost-container ready: false, restart count 0 Jan 1 02:30:38.661: INFO: pod-service-account-nomountsa-mountspec started at 2023-01-01 02:29:39 +0000 UTC (0+1 container statuses recorded) Jan 1 02:30:38.661: INFO: Container token-test ready: false, restart count 0 Jan 1 02:30:38.661: INFO: e2e-configmap-dns-server-7551ef4f-8c37-4168-bf54-91644ec7fb8f started at 2023-01-01 02:29:40 +0000 UTC (0+1 container statuses recorded) Jan 1 02:30:38.661: INFO: Container agnhost-container ready: false, restart count 0 Jan 1 02:30:38.661: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-7z6bp started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:30:38.661: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:30:38.661: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-lrzjq started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:30:38.661: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:30:38.661: INFO: netserver-3 started at 2023-01-01 02:25:27 +0000 UTC (0+1 container statuses recorded) Jan 1 02:30:38.661: INFO: Container webserver ready: true, restart count 0 Jan 1 02:30:38.662: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-4jf6d started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:30:38.662: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:30:38.662: INFO: cilium-78jll started at 2023-01-01 02:20:29 +0000 UTC (1+1 container statuses recorded) Jan 1 02:30:38.662: INFO: Init container clean-cilium-state ready: true, restart count 0 Jan 1 02:30:38.662: INFO: Container cilium-agent ready: true, restart count 0 Jan 1 02:30:38.662: INFO: ebs-csi-node-tzntf started at 2023-01-01 02:20:29 +0000 UTC (0+3 container statuses recorded) Jan 1 02:30:38.662: INFO: Container ebs-plugin ready: true, restart count 0 Jan 1 02:30:38.662: INFO: Container liveness-probe ready: true, restart count 0 Jan 1 02:30:38.662: INFO: Container node-driver-registrar ready: true, restart count 0 Jan 1 02:30:38.662: INFO: test-webserver-a9c3c1dd-2bb7-4667-876b-bad20aff393e started at 2023-01-01 02:25:41 +0000 UTC (0+1 container statuses recorded) Jan 1 02:30:38.662: INFO: Container test-webserver ready: false, restart count 0 Jan 1 02:30:38.662: INFO: hostexec-ip-172-20-49-224.eu-west-3.compute.internal-zzhhr started at 2023-01-01 02:25:53 +0000 UTC (0+1 container statuses recorded) Jan 1 02:30:38.662: INFO: Container agnhost-container ready: true, restart count 0 Jan 1 02:30:38.662: INFO: affinity-clusterip-timeout-pxjtk started at 2023-01-01 02:27:32 +0000 UTC (0+1 container statuses recorded) Jan 1 02:30:38.662: INFO: Container affinity-clusterip-timeout ready: true, restart count 0 Jan 1 02:30:38.662: INFO: annotationupdate3b35e9ec-7ad7-4bd9-8137-1bd2bf1a899a started at 2023-01-01 02:28:25 +0000 UTC (0+1 container statuses recorded) Jan 1 02:30:38.662: INFO: Container client-container ready: false, restart count 0 Jan 1 02:30:38.662: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-tv984 started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:30:38.662: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:30:38.662: INFO: netserver-3 started at 2023-01-01 02:25:34 +0000 UTC (0+1 container statuses recorded) Jan 1 02:30:38.662: INFO: Container webserver ready: true, restart count 0 Jan 1 02:30:38.662: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-cz8qh started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:30:38.662: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:30:38.662: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-2gndb started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:30:38.662: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:30:38.662: INFO: httpd started at 2023-01-01 02:26:24 +0000 UTC (0+1 container statuses recorded) Jan 1 02:30:38.662: INFO: Container httpd ready: false, restart count 0 Jan 1 02:30:38.662: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-h6dxb started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:30:38.662: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:30:38.662: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-6n5j9 started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:30:38.662: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:30:38.662: INFO: local-injector started at 2023-01-01 02:26:06 +0000 UTC (0+1 container statuses recorded) Jan 1 02:30:38.662: INFO: Container local-injector ready: false, restart count 0 Jan 1 02:30:38.662: INFO: test-ss-0 started at 2023-01-01 02:27:56 +0000 UTC (0+1 container statuses recorded) Jan 1 02:30:38.662: INFO: Container webserver ready: false, restart count 0 Jan 1 02:30:39.369: INFO: Latency metrics for node ip-172-20-49-224.eu-west-3.compute.internal Jan 1 02:30:39.369: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready �[1mSTEP�[0m: Destroying namespace "nettest-3615" for this suite.
Filter through log files
go run hack/e2e.go -v --test --test_args='--ginkgo.focus=Kubernetes\se2e\ssuite\s\[sig\-network\]\sServices\sshould\shave\ssession\saffinity\stimeout\swork\sfor\sservice\swith\stype\sclusterIP\s\[LinuxOnly\]\s\[Conformance\]$'
/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:633 Jan 1 02:36:02.926: failed to create replication controller with service in the namespace: services-9461 Unexpected error: <*errors.errorString | 0xc002938270>: { s: "only 2 pods started out of 3", } only 2 pods started out of 3 occurred /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/network/service.go:3198from junit_21.xml
[BeforeEach] [sig-network] Services /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:185 �[1mSTEP�[0m: Creating a kubernetes client Jan 1 02:27:27.715: INFO: >>> kubeConfig: /root/.kube/config �[1mSTEP�[0m: Building a namespace api object, basename services �[1mSTEP�[0m: Waiting for a default service account to be provisioned in namespace �[1mSTEP�[0m: Waiting for kube-root-ca.crt to be provisioned in namespace [BeforeEach] [sig-network] Services /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/network/service.go:752 [It] should have session affinity timeout work for service with type clusterIP [LinuxOnly] [Conformance] /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:633 �[1mSTEP�[0m: creating service in namespace services-9461 Jan 1 02:27:28.654: INFO: The status of Pod kube-proxy-mode-detector is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:27:30.759: INFO: The status of Pod kube-proxy-mode-detector is Running (Ready = true) Jan 1 02:27:30.863: INFO: Running '/home/prow/go/src/k8s.io/kops/_rundir/08fd59e0-897a-11ed-b47a-e2b55b2f9198/kubectl --server=https://api.e2e-e2e-kops-grid-cilium-eni-deb10-k23-docker.test-cncf-aws.k8s.io --kubeconfig=/root/.kube/config --namespace=services-9461 exec kube-proxy-mode-detector -- /bin/sh -x -c curl -q -s --connect-timeout 1 http://localhost:10249/proxyMode' Jan 1 02:27:32.088: INFO: rc: 7 Jan 1 02:27:32.199: INFO: Waiting for pod kube-proxy-mode-detector to disappear Jan 1 02:27:32.302: INFO: Pod kube-proxy-mode-detector no longer exists Jan 1 02:27:32.302: INFO: Couldn't detect KubeProxy mode - test failure may be expected: error running /home/prow/go/src/k8s.io/kops/_rundir/08fd59e0-897a-11ed-b47a-e2b55b2f9198/kubectl --server=https://api.e2e-e2e-kops-grid-cilium-eni-deb10-k23-docker.test-cncf-aws.k8s.io --kubeconfig=/root/.kube/config --namespace=services-9461 exec kube-proxy-mode-detector -- /bin/sh -x -c curl -q -s --connect-timeout 1 http://localhost:10249/proxyMode: Command stdout: stderr: + curl -q -s --connect-timeout 1 http://localhost:10249/proxyMode command terminated with exit code 7 error: exit status 7 �[1mSTEP�[0m: creating service affinity-clusterip-timeout in namespace services-9461 �[1mSTEP�[0m: creating replication controller affinity-clusterip-timeout in namespace services-9461 I0101 02:27:32.521418 6805 runners.go:193] Created replication controller with name: affinity-clusterip-timeout, namespace: services-9461, replica count: 3 I0101 02:27:35.672226 6805 runners.go:193] affinity-clusterip-timeout Pods: 3 out of 3 created, 0 running, 3 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0101 02:27:38.673386 6805 runners.go:193] affinity-clusterip-timeout Pods: 3 out of 3 created, 0 running, 3 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0101 02:27:41.674114 6805 runners.go:193] affinity-clusterip-timeout Pods: 3 out of 3 created, 0 running, 3 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0101 02:27:44.674423 6805 runners.go:193] affinity-clusterip-timeout Pods: 3 out of 3 created, 0 running, 3 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0101 02:27:47.674751 6805 runners.go:193] affinity-clusterip-timeout Pods: 3 out of 3 created, 0 running, 3 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0101 02:27:50.675136 6805 runners.go:193] affinity-clusterip-timeout Pods: 3 out of 3 created, 0 running, 3 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0101 02:27:53.676396 6805 runners.go:193] affinity-clusterip-timeout Pods: 3 out of 3 created, 0 running, 3 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0101 02:27:56.678772 6805 runners.go:193] affinity-clusterip-timeout Pods: 3 out of 3 created, 0 running, 3 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0101 02:27:59.679379 6805 runners.go:193] affinity-clusterip-timeout Pods: 3 out of 3 created, 0 running, 3 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0101 02:28:02.680525 6805 runners.go:193] affinity-clusterip-timeout Pods: 3 out of 3 created, 0 running, 3 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0101 02:28:05.680886 6805 runners.go:193] affinity-clusterip-timeout Pods: 3 out of 3 created, 0 running, 3 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0101 02:28:08.681929 6805 runners.go:193] affinity-clusterip-timeout Pods: 3 out of 3 created, 0 running, 3 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0101 02:28:11.682684 6805 runners.go:193] affinity-clusterip-timeout Pods: 3 out of 3 created, 0 running, 3 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0101 02:28:14.683527 6805 runners.go:193] affinity-clusterip-timeout Pods: 3 out of 3 created, 0 running, 3 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0101 02:28:17.683675 6805 runners.go:193] affinity-clusterip-timeout Pods: 3 out of 3 created, 0 running, 3 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0101 02:28:20.684063 6805 runners.go:193] affinity-clusterip-timeout Pods: 3 out of 3 created, 0 running, 3 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0101 02:28:23.684959 6805 runners.go:193] affinity-clusterip-timeout Pods: 3 out of 3 created, 0 running, 3 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0101 02:28:26.685362 6805 runners.go:193] affinity-clusterip-timeout Pods: 3 out of 3 created, 0 running, 3 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0101 02:28:29.686847 6805 runners.go:193] affinity-clusterip-timeout Pods: 3 out of 3 created, 1 running, 2 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0101 02:28:32.687841 6805 runners.go:193] affinity-clusterip-timeout Pods: 3 out of 3 created, 1 running, 2 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0101 02:28:35.688182 6805 runners.go:193] affinity-clusterip-timeout Pods: 3 out of 3 created, 1 running, 2 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0101 02:28:38.688401 6805 runners.go:193] affinity-clusterip-timeout Pods: 3 out of 3 created, 1 running, 2 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0101 02:28:41.690082 6805 runners.go:193] affinity-clusterip-timeout Pods: 3 out of 3 created, 1 running, 2 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0101 02:28:44.691024 6805 runners.go:193] affinity-clusterip-timeout Pods: 3 out of 3 created, 1 running, 2 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0101 02:28:47.691296 6805 runners.go:193] affinity-clusterip-timeout Pods: 3 out of 3 created, 1 running, 2 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0101 02:28:50.693739 6805 runners.go:193] affinity-clusterip-timeout Pods: 3 out of 3 created, 1 running, 2 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0101 02:28:53.694348 6805 runners.go:193] affinity-clusterip-timeout Pods: 3 out of 3 created, 1 running, 2 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0101 02:28:56.696656 6805 runners.go:193] affinity-clusterip-timeout Pods: 3 out of 3 created, 1 running, 2 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0101 02:28:59.697831 6805 runners.go:193] affinity-clusterip-timeout Pods: 3 out of 3 created, 1 running, 2 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0101 02:29:02.698790 6805 runners.go:193] affinity-clusterip-timeout Pods: 3 out of 3 created, 1 running, 2 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0101 02:29:05.700962 6805 runners.go:193] affinity-clusterip-timeout Pods: 3 out of 3 created, 1 running, 2 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0101 02:29:08.701265 6805 runners.go:193] affinity-clusterip-timeout Pods: 3 out of 3 created, 1 running, 2 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0101 02:29:11.701610 6805 runners.go:193] affinity-clusterip-timeout Pods: 3 out of 3 created, 1 running, 2 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0101 02:29:14.701936 6805 runners.go:193] affinity-clusterip-timeout Pods: 3 out of 3 created, 1 running, 2 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0101 02:29:17.702772 6805 runners.go:193] affinity-clusterip-timeout Pods: 3 out of 3 created, 1 running, 2 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0101 02:29:20.703106 6805 runners.go:193] affinity-clusterip-timeout Pods: 3 out of 3 created, 1 running, 2 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0101 02:29:23.704307 6805 runners.go:193] affinity-clusterip-timeout Pods: 3 out of 3 created, 1 running, 2 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0101 02:29:26.706623 6805 runners.go:193] affinity-clusterip-timeout Pods: 3 out of 3 created, 1 running, 2 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0101 02:29:29.707902 6805 runners.go:193] affinity-clusterip-timeout Pods: 3 out of 3 created, 1 running, 2 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0101 02:29:32.708946 6805 runners.go:193] affinity-clusterip-timeout Pods: 3 out of 3 created, 1 running, 2 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0101 02:29:35.709249 6805 runners.go:193] affinity-clusterip-timeout Pods: 3 out of 3 created, 1 running, 2 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0101 02:29:38.709573 6805 runners.go:193] affinity-clusterip-timeout Pods: 3 out of 3 created, 1 running, 2 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0101 02:29:41.710692 6805 runners.go:193] affinity-clusterip-timeout Pods: 3 out of 3 created, 1 running, 2 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0101 02:29:44.711001 6805 runners.go:193] affinity-clusterip-timeout Pods: 3 out of 3 created, 1 running, 2 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0101 02:29:47.711754 6805 runners.go:193] affinity-clusterip-timeout Pods: 3 out of 3 created, 1 running, 2 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0101 02:29:50.712123 6805 runners.go:193] affinity-clusterip-timeout Pods: 3 out of 3 created, 1 running, 2 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0101 02:29:53.713314 6805 runners.go:193] affinity-clusterip-timeout Pods: 3 out of 3 created, 1 running, 2 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0101 02:29:56.715593 6805 runners.go:193] affinity-clusterip-timeout Pods: 3 out of 3 created, 1 running, 2 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0101 02:29:59.715956 6805 runners.go:193] affinity-clusterip-timeout Pods: 3 out of 3 created, 1 running, 2 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0101 02:30:02.716237 6805 runners.go:193] affinity-clusterip-timeout Pods: 3 out of 3 created, 1 running, 2 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0101 02:30:05.716659 6805 runners.go:193] affinity-clusterip-timeout Pods: 3 out of 3 created, 1 running, 2 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0101 02:30:08.717210 6805 runners.go:193] affinity-clusterip-timeout Pods: 3 out of 3 created, 1 running, 2 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0101 02:30:11.719493 6805 runners.go:193] affinity-clusterip-timeout Pods: 3 out of 3 created, 1 running, 2 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0101 02:30:14.720190 6805 runners.go:193] affinity-clusterip-timeout Pods: 3 out of 3 created, 1 running, 2 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0101 02:30:17.720512 6805 runners.go:193] affinity-clusterip-timeout Pods: 3 out of 3 created, 1 running, 2 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0101 02:30:20.720807 6805 runners.go:193] affinity-clusterip-timeout Pods: 3 out of 3 created, 1 running, 2 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0101 02:30:23.722003 6805 runners.go:193] affinity-clusterip-timeout Pods: 3 out of 3 created, 1 running, 2 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0101 02:30:26.722367 6805 runners.go:193] affinity-clusterip-timeout Pods: 3 out of 3 created, 1 running, 2 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0101 02:30:29.723356 6805 runners.go:193] affinity-clusterip-timeout Pods: 3 out of 3 created, 1 running, 2 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0101 02:30:32.723761 6805 runners.go:193] affinity-clusterip-timeout Pods: 3 out of 3 created, 1 running, 2 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0101 02:30:35.724063 6805 runners.go:193] affinity-clusterip-timeout Pods: 3 out of 3 created, 1 running, 2 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0101 02:30:38.724550 6805 runners.go:193] affinity-clusterip-timeout Pods: 3 out of 3 created, 1 running, 2 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0101 02:30:41.725730 6805 runners.go:193] affinity-clusterip-timeout Pods: 3 out of 3 created, 1 running, 2 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0101 02:30:44.726119 6805 runners.go:193] affinity-clusterip-timeout Pods: 3 out of 3 created, 1 running, 2 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0101 02:30:47.726928 6805 runners.go:193] affinity-clusterip-timeout Pods: 3 out of 3 created, 1 running, 2 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0101 02:30:50.727221 6805 runners.go:193] affinity-clusterip-timeout Pods: 3 out of 3 created, 1 running, 2 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0101 02:30:53.727436 6805 runners.go:193] affinity-clusterip-timeout Pods: 3 out of 3 created, 1 running, 2 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0101 02:30:56.727788 6805 runners.go:193] affinity-clusterip-timeout Pods: 3 out of 3 created, 1 running, 2 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0101 02:30:59.728134 6805 runners.go:193] affinity-clusterip-timeout Pods: 3 out of 3 created, 1 running, 2 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0101 02:31:02.729327 6805 runners.go:193] affinity-clusterip-timeout Pods: 3 out of 3 created, 2 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0101 02:31:05.729737 6805 runners.go:193] affinity-clusterip-timeout Pods: 3 out of 3 created, 2 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0101 02:31:08.730293 6805 runners.go:193] affinity-clusterip-timeout Pods: 3 out of 3 created, 2 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0101 02:31:11.732632 6805 runners.go:193] affinity-clusterip-timeout Pods: 3 out of 3 created, 2 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0101 02:31:14.733554 6805 runners.go:193] affinity-clusterip-timeout Pods: 3 out of 3 created, 2 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0101 02:31:17.733825 6805 runners.go:193] affinity-clusterip-timeout Pods: 3 out of 3 created, 2 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0101 02:31:20.734189 6805 runners.go:193] affinity-clusterip-timeout Pods: 3 out of 3 created, 2 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0101 02:31:23.734443 6805 runners.go:193] affinity-clusterip-timeout Pods: 3 out of 3 created, 2 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0101 02:31:26.734665 6805 runners.go:193] affinity-clusterip-timeout Pods: 3 out of 3 created, 2 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0101 02:31:29.734940 6805 runners.go:193] affinity-clusterip-timeout Pods: 3 out of 3 created, 2 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0101 02:31:32.735388 6805 runners.go:193] affinity-clusterip-timeout Pods: 3 out of 3 created, 2 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0101 02:31:35.735774 6805 runners.go:193] affinity-clusterip-timeout Pods: 3 out of 3 created, 2 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0101 02:31:38.736159 6805 runners.go:193] affinity-clusterip-timeout Pods: 3 out of 3 created, 2 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0101 02:31:41.737643 6805 runners.go:193] affinity-clusterip-timeout Pods: 3 out of 3 created, 2 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0101 02:31:44.739229 6805 runners.go:193] affinity-clusterip-timeout Pods: 3 out of 3 created, 2 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0101 02:31:47.739432 6805 runners.go:193] affinity-clusterip-timeout Pods: 3 out of 3 created, 2 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0101 02:31:50.741863 6805 runners.go:193] affinity-clusterip-timeout Pods: 3 out of 3 created, 2 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0101 02:31:53.742202 6805 runners.go:193] affinity-clusterip-timeout Pods: 3 out of 3 created, 2 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0101 02:31:56.742519 6805 runners.go:193] affinity-clusterip-timeout Pods: 3 out of 3 created, 2 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0101 02:31:59.744077 6805 runners.go:193] affinity-clusterip-timeout Pods: 3 out of 3 created, 2 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0101 02:32:02.745097 6805 runners.go:193] affinity-clusterip-timeout Pods: 3 out of 3 created, 2 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0101 02:32:05.747306 6805 runners.go:193] affinity-clusterip-timeout Pods: 3 out of 3 created, 2 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0101 02:32:08.747467 6805 runners.go:193] affinity-clusterip-timeout Pods: 3 out of 3 created, 2 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0101 02:32:11.747866 6805 runners.go:193] affinity-clusterip-timeout Pods: 3 out of 3 created, 2 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0101 02:32:14.750640 6805 runners.go:193] affinity-clusterip-timeout Pods: 3 out of 3 created, 2 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0101 02:32:17.751429 6805 runners.go:193] affinity-clusterip-timeout Pods: 3 out of 3 created, 2 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0101 02:32:20.753728 6805 runners.go:193] affinity-clusterip-timeout Pods: 3 out of 3 created, 2 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0101 02:32:23.754319 6805 runners.go:193] affinity-clusterip-timeout Pods: 3 out of 3 created, 2 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0101 02:32:26.756578 6805 runners.go:193] affinity-clusterip-timeout Pods: 3 out of 3 created, 2 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0101 02:32:29.756912 6805 runners.go:193] affinity-clusterip-timeout Pods: 3 out of 3 created, 2 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0101 02:32:32.757952 6805 runners.go:193] affinity-clusterip-timeout Pods: 3 out of 3 created, 2 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0101 02:32:35.759854 6805 runners.go:193] affinity-clusterip-timeout Pods: 3 out of 3 created, 2 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0101 02:32:38.760303 6805 runners.go:193] affinity-clusterip-timeout Pods: 3 out of 3 created, 2 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0101 02:32:41.762619 6805 runners.go:193] affinity-clusterip-timeout Pods: 3 out of 3 created, 2 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0101 02:32:44.762924 6805 runners.go:193] affinity-clusterip-timeout Pods: 3 out of 3 created, 2 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0101 02:32:47.763948 6805 runners.go:193] affinity-clusterip-timeout Pods: 3 out of 3 created, 2 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0101 02:32:50.764279 6805 runners.go:193] affinity-clusterip-timeout Pods: 3 out of 3 created, 2 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0101 02:32:53.764569 6805 runners.go:193] affinity-clusterip-timeout Pods: 3 out of 3 created, 2 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0101 02:32:56.766881 6805 runners.go:193] affinity-clusterip-timeout Pods: 3 out of 3 created, 2 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0101 02:32:59.768189 6805 runners.go:193] affinity-clusterip-timeout Pods: 3 out of 3 created, 2 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0101 02:33:02.768911 6805 runners.go:193] affinity-clusterip-timeout Pods: 3 out of 3 created, 2 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0101 02:33:05.769968 6805 runners.go:193] affinity-clusterip-timeout Pods: 3 out of 3 created, 2 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0101 02:33:08.770158 6805 runners.go:193] affinity-clusterip-timeout Pods: 3 out of 3 created, 2 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0101 02:33:11.770435 6805 runners.go:193] affinity-clusterip-timeout Pods: 3 out of 3 created, 2 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0101 02:33:14.770770 6805 runners.go:193] affinity-clusterip-timeout Pods: 3 out of 3 created, 2 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0101 02:33:17.771979 6805 runners.go:193] affinity-clusterip-timeout Pods: 3 out of 3 created, 2 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0101 02:33:20.772281 6805 runners.go:193] affinity-clusterip-timeout Pods: 3 out of 3 created, 2 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0101 02:33:23.772575 6805 runners.go:193] affinity-clusterip-timeout Pods: 3 out of 3 created, 2 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0101 02:33:26.773501 6805 runners.go:193] affinity-clusterip-timeout Pods: 3 out of 3 created, 2 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0101 02:33:29.775019 6805 runners.go:193] affinity-clusterip-timeout Pods: 3 out of 3 created, 2 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0101 02:33:32.776060 6805 runners.go:193] affinity-clusterip-timeout Pods: 3 out of 3 created, 2 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0101 02:33:35.776377 6805 runners.go:193] affinity-clusterip-timeout Pods: 3 out of 3 created, 2 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0101 02:33:38.776834 6805 runners.go:193] affinity-clusterip-timeout Pods: 3 out of 3 created, 2 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0101 02:33:41.777158 6805 runners.go:193] affinity-clusterip-timeout Pods: 3 out of 3 created, 2 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0101 02:33:44.778911 6805 runners.go:193] affinity-clusterip-timeout Pods: 3 out of 3 created, 2 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0101 02:33:47.779920 6805 runners.go:193] affinity-clusterip-timeout Pods: 3 out of 3 created, 2 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0101 02:33:50.782225 6805 runners.go:193] affinity-clusterip-timeout Pods: 3 out of 3 created, 2 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0101 02:33:53.783191 6805 runners.go:193] affinity-clusterip-timeout Pods: 3 out of 3 created, 2 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0101 02:33:56.783927 6805 runners.go:193] affinity-clusterip-timeout Pods: 3 out of 3 created, 2 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0101 02:33:59.785545 6805 runners.go:193] affinity-clusterip-timeout Pods: 3 out of 3 created, 2 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0101 02:34:02.785787 6805 runners.go:193] affinity-clusterip-timeout Pods: 3 out of 3 created, 2 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0101 02:34:05.786144 6805 runners.go:193] affinity-clusterip-timeout Pods: 3 out of 3 created, 2 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0101 02:34:08.786507 6805 runners.go:193] affinity-clusterip-timeout Pods: 3 out of 3 created, 2 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0101 02:34:11.788870 6805 runners.go:193] affinity-clusterip-timeout Pods: 3 out of 3 created, 2 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0101 02:34:14.789172 6805 runners.go:193] affinity-clusterip-timeout Pods: 3 out of 3 created, 2 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0101 02:34:17.789466 6805 runners.go:193] affinity-clusterip-timeout Pods: 3 out of 3 created, 2 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0101 02:34:20.792125 6805 runners.go:193] affinity-clusterip-timeout Pods: 3 out of 3 created, 2 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0101 02:34:23.792965 6805 runners.go:193] affinity-clusterip-timeout Pods: 3 out of 3 created, 2 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0101 02:34:26.793278 6805 runners.go:193] affinity-clusterip-timeout Pods: 3 out of 3 created, 2 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0101 02:34:29.794978 6805 runners.go:193] affinity-clusterip-timeout Pods: 3 out of 3 created, 2 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0101 02:34:32.795947 6805 runners.go:193] affinity-clusterip-timeout Pods: 3 out of 3 created, 2 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0101 02:34:35.796212 6805 runners.go:193] affinity-clusterip-timeout Pods: 3 out of 3 created, 2 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0101 02:34:38.797015 6805 runners.go:193] affinity-clusterip-timeout Pods: 3 out of 3 created, 2 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0101 02:34:41.798177 6805 runners.go:193] affinity-clusterip-timeout Pods: 3 out of 3 created, 2 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0101 02:34:44.798515 6805 runners.go:193] affinity-clusterip-timeout Pods: 3 out of 3 created, 2 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0101 02:34:47.799702 6805 runners.go:193] affinity-clusterip-timeout Pods: 3 out of 3 created, 2 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0101 02:34:50.801981 6805 runners.go:193] affinity-clusterip-timeout Pods: 3 out of 3 created, 2 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0101 02:34:53.803168 6805 runners.go:193] affinity-clusterip-timeout Pods: 3 out of 3 created, 2 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0101 02:34:56.804139 6805 runners.go:193] affinity-clusterip-timeout Pods: 3 out of 3 created, 2 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0101 02:34:59.804478 6805 runners.go:193] affinity-clusterip-timeout Pods: 3 out of 3 created, 2 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0101 02:35:02.804753 6805 runners.go:193] affinity-clusterip-timeout Pods: 3 out of 3 created, 2 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0101 02:35:05.807299 6805 runners.go:193] affinity-clusterip-timeout Pods: 3 out of 3 created, 2 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0101 02:35:08.807519 6805 runners.go:193] affinity-clusterip-timeout Pods: 3 out of 3 created, 2 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0101 02:35:11.807836 6805 runners.go:193] affinity-clusterip-timeout Pods: 3 out of 3 created, 2 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0101 02:35:14.808171 6805 runners.go:193] affinity-clusterip-timeout Pods: 3 out of 3 created, 2 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0101 02:35:17.808467 6805 runners.go:193] affinity-clusterip-timeout Pods: 3 out of 3 created, 2 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0101 02:35:20.810855 6805 runners.go:193] affinity-clusterip-timeout Pods: 3 out of 3 created, 2 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0101 02:35:23.811145 6805 runners.go:193] affinity-clusterip-timeout Pods: 3 out of 3 created, 2 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0101 02:35:26.811490 6805 runners.go:193] affinity-clusterip-timeout Pods: 3 out of 3 created, 2 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0101 02:35:29.811834 6805 runners.go:193] affinity-clusterip-timeout Pods: 3 out of 3 created, 2 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0101 02:35:32.812055 6805 runners.go:193] affinity-clusterip-timeout Pods: 3 out of 3 created, 2 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0101 02:35:35.812359 6805 runners.go:193] affinity-clusterip-timeout Pods: 3 out of 3 created, 2 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0101 02:35:38.812553 6805 runners.go:193] affinity-clusterip-timeout Pods: 3 out of 3 created, 2 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0101 02:35:41.813662 6805 runners.go:193] affinity-clusterip-timeout Pods: 3 out of 3 created, 2 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0101 02:35:44.813926 6805 runners.go:193] affinity-clusterip-timeout Pods: 3 out of 3 created, 2 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0101 02:35:47.814985 6805 runners.go:193] affinity-clusterip-timeout Pods: 3 out of 3 created, 2 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0101 02:35:50.815287 6805 runners.go:193] affinity-clusterip-timeout Pods: 3 out of 3 created, 2 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0101 02:35:53.816193 6805 runners.go:193] affinity-clusterip-timeout Pods: 3 out of 3 created, 2 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0101 02:35:56.816745 6805 runners.go:193] affinity-clusterip-timeout Pods: 3 out of 3 created, 2 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0101 02:35:59.818347 6805 runners.go:193] affinity-clusterip-timeout Pods: 3 out of 3 created, 2 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0101 02:36:02.818740 6805 runners.go:193] affinity-clusterip-timeout Pods: 3 out of 3 created, 2 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0101 02:36:02.926330 6805 runners.go:193] Pod affinity-clusterip-timeout-lv5vx ip-172-20-45-112.eu-west-3.compute.internal Running <nil> I0101 02:36:02.926458 6805 runners.go:193] Pod affinity-clusterip-timeout-pxjtk ip-172-20-49-224.eu-west-3.compute.internal Running <nil> I0101 02:36:02.926473 6805 runners.go:193] Pod affinity-clusterip-timeout-q9qt2 ip-172-20-33-66.eu-west-3.compute.internal Pending <nil> Jan 1 02:36:02.926: FAIL: failed to create replication controller with service in the namespace: services-9461 Unexpected error: <*errors.errorString | 0xc002938270>: { s: "only 2 pods started out of 3", } only 2 pods started out of 3 occurred Full Stack Trace k8s.io/kubernetes/test/e2e/network.execAffinityTestForSessionAffinityTimeout(0xc0001ddb80, {0x7b06bd0, 0xc00139f500}, 0xc000ba8c80) /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/network/service.go:3198 +0x30d k8s.io/kubernetes/test/e2e/network.glob..func24.25() /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/network/service.go:2083 +0x8b k8s.io/kubernetes/test/e2e.RunE2ETests(0x24c66d7) _output/dockerized/go/src/k8s.io/kubernetes/test/e2e/e2e.go:133 +0x697 k8s.io/kubernetes/test/e2e.TestE2E(0x0) _output/dockerized/go/src/k8s.io/kubernetes/test/e2e/e2e_test.go:136 +0x19 testing.tRunner(0xc0004c0d00, 0x735e880) /usr/local/go/src/testing/testing.go:1259 +0x102 created by testing.(*T).Run /usr/local/go/src/testing/testing.go:1306 +0x35a [AfterEach] [sig-network] Services /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:186 �[1mSTEP�[0m: Collecting events from namespace "services-9461". �[1mSTEP�[0m: Found 44 events. Jan 1 02:36:03.033: INFO: At 2023-01-01 02:27:28 +0000 UTC - event for kube-proxy-mode-detector: {default-scheduler } Scheduled: Successfully assigned services-9461/kube-proxy-mode-detector to ip-172-20-49-224.eu-west-3.compute.internal Jan 1 02:36:03.033: INFO: At 2023-01-01 02:27:29 +0000 UTC - event for kube-proxy-mode-detector: {kubelet ip-172-20-49-224.eu-west-3.compute.internal} Created: Created container agnhost-container Jan 1 02:36:03.033: INFO: At 2023-01-01 02:27:29 +0000 UTC - event for kube-proxy-mode-detector: {kubelet ip-172-20-49-224.eu-west-3.compute.internal} Pulled: Container image "k8s.gcr.io/e2e-test-images/agnhost:2.39" already present on machine Jan 1 02:36:03.033: INFO: At 2023-01-01 02:27:29 +0000 UTC - event for kube-proxy-mode-detector: {kubelet ip-172-20-49-224.eu-west-3.compute.internal} Started: Started container agnhost-container Jan 1 02:36:03.033: INFO: At 2023-01-01 02:27:32 +0000 UTC - event for affinity-clusterip-timeout: {replication-controller } SuccessfulCreate: Created pod: affinity-clusterip-timeout-lv5vx Jan 1 02:36:03.033: INFO: At 2023-01-01 02:27:32 +0000 UTC - event for affinity-clusterip-timeout: {replication-controller } SuccessfulCreate: Created pod: affinity-clusterip-timeout-q9qt2 Jan 1 02:36:03.033: INFO: At 2023-01-01 02:27:32 +0000 UTC - event for affinity-clusterip-timeout: {replication-controller } SuccessfulCreate: Created pod: affinity-clusterip-timeout-pxjtk Jan 1 02:36:03.033: INFO: At 2023-01-01 02:27:32 +0000 UTC - event for affinity-clusterip-timeout-lv5vx: {default-scheduler } Scheduled: Successfully assigned services-9461/affinity-clusterip-timeout-lv5vx to ip-172-20-45-112.eu-west-3.compute.internal Jan 1 02:36:03.033: INFO: At 2023-01-01 02:27:32 +0000 UTC - event for affinity-clusterip-timeout-pxjtk: {default-scheduler } Scheduled: Successfully assigned services-9461/affinity-clusterip-timeout-pxjtk to ip-172-20-49-224.eu-west-3.compute.internal Jan 1 02:36:03.033: INFO: At 2023-01-01 02:27:32 +0000 UTC - event for affinity-clusterip-timeout-q9qt2: {default-scheduler } Scheduled: Successfully assigned services-9461/affinity-clusterip-timeout-q9qt2 to ip-172-20-33-66.eu-west-3.compute.internal Jan 1 02:36:03.033: INFO: At 2023-01-01 02:27:33 +0000 UTC - event for affinity-clusterip-timeout-lv5vx: {kubelet ip-172-20-45-112.eu-west-3.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "bee9ef2dd5f74a67d79bc40e05297b31020f9f856b423c98220b0792309453ce" network for pod "affinity-clusterip-timeout-lv5vx": networkPlugin cni failed to set up pod "affinity-clusterip-timeout-lv5vx_services-9461" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 1 02:36:03.033: INFO: At 2023-01-01 02:27:33 +0000 UTC - event for affinity-clusterip-timeout-pxjtk: {kubelet ip-172-20-49-224.eu-west-3.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "80257b8bcabd67b69b4cac27a6e5bba70181dfa0f2689082abf22c55fdc1cbf2" network for pod "affinity-clusterip-timeout-pxjtk": networkPlugin cni failed to set up pod "affinity-clusterip-timeout-pxjtk_services-9461" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 1 02:36:03.033: INFO: At 2023-01-01 02:27:33 +0000 UTC - event for kube-proxy-mode-detector: {kubelet ip-172-20-49-224.eu-west-3.compute.internal} Killing: Stopping container agnhost-container Jan 1 02:36:03.033: INFO: At 2023-01-01 02:27:34 +0000 UTC - event for affinity-clusterip-timeout-lv5vx: {kubelet ip-172-20-45-112.eu-west-3.compute.internal} SandboxChanged: Pod sandbox changed, it will be killed and re-created. Jan 1 02:36:03.033: INFO: At 2023-01-01 02:27:34 +0000 UTC - event for affinity-clusterip-timeout-q9qt2: {kubelet ip-172-20-33-66.eu-west-3.compute.internal} SandboxChanged: Pod sandbox changed, it will be killed and re-created. Jan 1 02:36:03.033: INFO: At 2023-01-01 02:27:34 +0000 UTC - event for affinity-clusterip-timeout-q9qt2: {kubelet ip-172-20-33-66.eu-west-3.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "2224cd3d7f83f9a7736d8952ae274d78a60254c486f6264d5b26eaf9f461a6ed" network for pod "affinity-clusterip-timeout-q9qt2": networkPlugin cni failed to set up pod "affinity-clusterip-timeout-q9qt2_services-9461" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 1 02:36:03.033: INFO: At 2023-01-01 02:27:35 +0000 UTC - event for affinity-clusterip-timeout-lv5vx: {kubelet ip-172-20-45-112.eu-west-3.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "bdf86f98d6d89c1084740c68b54d3507e2f77741d277730ef5198cf7ee62db56" network for pod "affinity-clusterip-timeout-lv5vx": networkPlugin cni failed to set up pod "affinity-clusterip-timeout-lv5vx_services-9461" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 1 02:36:03.033: INFO: At 2023-01-01 02:27:35 +0000 UTC - event for affinity-clusterip-timeout-pxjtk: {kubelet ip-172-20-49-224.eu-west-3.compute.internal} SandboxChanged: Pod sandbox changed, it will be killed and re-created. Jan 1 02:36:03.033: INFO: At 2023-01-01 02:27:36 +0000 UTC - event for affinity-clusterip-timeout-q9qt2: {kubelet ip-172-20-33-66.eu-west-3.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "a6aacc7e7689c430dbb0417c970981f2225b46d5bb7f5d59b4beb5f2b9e6b6c3" network for pod "affinity-clusterip-timeout-q9qt2": networkPlugin cni failed to set up pod "affinity-clusterip-timeout-q9qt2_services-9461" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 1 02:36:03.033: INFO: At 2023-01-01 02:27:37 +0000 UTC - event for affinity-clusterip-timeout-pxjtk: {kubelet ip-172-20-49-224.eu-west-3.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "c6844738e86c4cd9a5712c01ca4daf68c2c182303403b6946671fec7bfe2cfc0" network for pod "affinity-clusterip-timeout-pxjtk": networkPlugin cni failed to set up pod "affinity-clusterip-timeout-pxjtk_services-9461" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 1 02:36:03.033: INFO: At 2023-01-01 02:27:38 +0000 UTC - event for affinity-clusterip-timeout-lv5vx: {kubelet ip-172-20-45-112.eu-west-3.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "4abe44e239b8a42549c29dd774fe7411d416595fdadc473775d1b6b0b9ce1a41" network for pod "affinity-clusterip-timeout-lv5vx": networkPlugin cni failed to set up pod "affinity-clusterip-timeout-lv5vx_services-9461" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 1 02:36:03.033: INFO: At 2023-01-01 02:27:39 +0000 UTC - event for affinity-clusterip-timeout-lv5vx: {kubelet ip-172-20-45-112.eu-west-3.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "0a10907f404e97b964ca0ce1cc2a066cba523c0803ee4b6f8d8392c99d2b2dc0" network for pod "affinity-clusterip-timeout-lv5vx": networkPlugin cni failed to set up pod "affinity-clusterip-timeout-lv5vx_services-9461" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 1 02:36:03.033: INFO: At 2023-01-01 02:27:40 +0000 UTC - event for affinity-clusterip-timeout-pxjtk: {kubelet ip-172-20-49-224.eu-west-3.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "7e261f6c20be4bc28122751e765af46c687c846085cc2e2900a4c9f79d37cf4c" network for pod "affinity-clusterip-timeout-pxjtk": networkPlugin cni failed to set up pod "affinity-clusterip-timeout-pxjtk_services-9461" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 1 02:36:03.033: INFO: At 2023-01-01 02:27:40 +0000 UTC - event for affinity-clusterip-timeout-q9qt2: {kubelet ip-172-20-33-66.eu-west-3.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "c5dcde989f1c98875d2d97f070b8cdd0277229ceb0fbed2089cccd9dd14ac261" network for pod "affinity-clusterip-timeout-q9qt2": networkPlugin cni failed to set up pod "affinity-clusterip-timeout-q9qt2_services-9461" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 1 02:36:03.033: INFO: At 2023-01-01 02:27:41 +0000 UTC - event for affinity-clusterip-timeout-lv5vx: {kubelet ip-172-20-45-112.eu-west-3.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "5d4b8431d93fe4f7941a877087e14f265818be53f88dac90b4574fcfb940507b" network for pod "affinity-clusterip-timeout-lv5vx": networkPlugin cni failed to set up pod "affinity-clusterip-timeout-lv5vx_services-9461" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 1 02:36:03.033: INFO: At 2023-01-01 02:27:43 +0000 UTC - event for affinity-clusterip-timeout-pxjtk: {kubelet ip-172-20-49-224.eu-west-3.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "07e90d1f01b479d12e5606f26f987f5c40c3e959facc453561b786624f1244cb" network for pod "affinity-clusterip-timeout-pxjtk": networkPlugin cni failed to set up pod "affinity-clusterip-timeout-pxjtk_services-9461" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 1 02:36:03.033: INFO: At 2023-01-01 02:27:44 +0000 UTC - event for affinity-clusterip-timeout-lv5vx: {kubelet ip-172-20-45-112.eu-west-3.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "626827938b99c3f0f7539b1cc649b6ca0477114d2bd2f3c152d80b7747c4dc43" network for pod "affinity-clusterip-timeout-lv5vx": networkPlugin cni failed to set up pod "affinity-clusterip-timeout-lv5vx_services-9461" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 1 02:36:03.033: INFO: At 2023-01-01 02:27:44 +0000 UTC - event for affinity-clusterip-timeout-q9qt2: {kubelet ip-172-20-33-66.eu-west-3.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "8a880d9900c424f65297421305051bfc097707ae27199e47cb71f74fbd3fa82c" network for pod "affinity-clusterip-timeout-q9qt2": networkPlugin cni failed to set up pod "affinity-clusterip-timeout-q9qt2_services-9461" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 1 02:36:03.033: INFO: At 2023-01-01 02:27:45 +0000 UTC - event for affinity-clusterip-timeout-pxjtk: {kubelet ip-172-20-49-224.eu-west-3.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "3eec3d7af547be3b397b93aa56fa1d98e590ab8aa361cc1e28c17596fb0ada31" network for pod "affinity-clusterip-timeout-pxjtk": networkPlugin cni failed to set up pod "affinity-clusterip-timeout-pxjtk_services-9461" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 1 02:36:03.033: INFO: At 2023-01-01 02:27:46 +0000 UTC - event for affinity-clusterip-timeout-lv5vx: {kubelet ip-172-20-45-112.eu-west-3.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "9b9c4f5da3ccbdc0611ddae2577eab09bbe608ad1fd3c5c283da43aebb79e302" network for pod "affinity-clusterip-timeout-lv5vx": networkPlugin cni failed to set up pod "affinity-clusterip-timeout-lv5vx_services-9461" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 1 02:36:03.033: INFO: At 2023-01-01 02:27:46 +0000 UTC - event for affinity-clusterip-timeout-q9qt2: {kubelet ip-172-20-33-66.eu-west-3.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "b802cb40f258480e9cde159c067bfd39098b1d3aa12850b5093520865ce9a792" network for pod "affinity-clusterip-timeout-q9qt2": networkPlugin cni failed to set up pod "affinity-clusterip-timeout-q9qt2_services-9461" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 1 02:36:03.033: INFO: At 2023-01-01 02:27:47 +0000 UTC - event for affinity-clusterip-timeout-pxjtk: {kubelet ip-172-20-49-224.eu-west-3.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "ba0a10e2cd7dc416f53251cab36927f4389ffd18002e8ffef950debea2a251ea" network for pod "affinity-clusterip-timeout-pxjtk": networkPlugin cni failed to set up pod "affinity-clusterip-timeout-pxjtk_services-9461" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 1 02:36:03.033: INFO: At 2023-01-01 02:27:48 +0000 UTC - event for affinity-clusterip-timeout-q9qt2: {kubelet ip-172-20-33-66.eu-west-3.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "26bbb50274f2e0646218cb90058867823815a54c7bd11535c819894904a34b67" network for pod "affinity-clusterip-timeout-q9qt2": networkPlugin cni failed to set up pod "affinity-clusterip-timeout-q9qt2_services-9461" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 1 02:36:03.033: INFO: At 2023-01-01 02:27:49 +0000 UTC - event for affinity-clusterip-timeout-lv5vx: {kubelet ip-172-20-45-112.eu-west-3.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "48fe9a92178635e007960112133c8d42a50fcb2b3d14f48ff74b56aa0ed47976" network for pod "affinity-clusterip-timeout-lv5vx": networkPlugin cni failed to set up pod "affinity-clusterip-timeout-lv5vx_services-9461" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 1 02:36:03.033: INFO: At 2023-01-01 02:27:49 +0000 UTC - event for affinity-clusterip-timeout-pxjtk: {kubelet ip-172-20-49-224.eu-west-3.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "2115ca527ab43f13f13e64af7c6e14e2d2ca3ae98c821e9d9d73fff37a7644da" network for pod "affinity-clusterip-timeout-pxjtk": networkPlugin cni failed to set up pod "affinity-clusterip-timeout-pxjtk_services-9461" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 1 02:36:03.033: INFO: At 2023-01-01 02:27:50 +0000 UTC - event for affinity-clusterip-timeout-pxjtk: {kubelet ip-172-20-49-224.eu-west-3.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "6741b7017e56297c331cc0edd702ae766e202fbe63f19f2659dea664536b8490" network for pod "affinity-clusterip-timeout-pxjtk": networkPlugin cni failed to set up pod "affinity-clusterip-timeout-pxjtk_services-9461" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 1 02:36:03.033: INFO: At 2023-01-01 02:27:51 +0000 UTC - event for affinity-clusterip-timeout-q9qt2: {kubelet ip-172-20-33-66.eu-west-3.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "248208bfba03f2e837565ac8873962faf1ca1d2a86c93e7578d242ab67388c97" network for pod "affinity-clusterip-timeout-q9qt2": networkPlugin cni failed to set up pod "affinity-clusterip-timeout-q9qt2_services-9461" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 1 02:36:03.033: INFO: At 2023-01-01 02:27:52 +0000 UTC - event for affinity-clusterip-timeout-lv5vx: {kubelet ip-172-20-45-112.eu-west-3.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "894bd34d8a13751a2dd29304440d1a6c8ee0425a268f5b9ea26a4e386ba299ca" network for pod "affinity-clusterip-timeout-lv5vx": networkPlugin cni failed to set up pod "affinity-clusterip-timeout-lv5vx_services-9461" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 1 02:36:03.033: INFO: At 2023-01-01 02:27:52 +0000 UTC - event for affinity-clusterip-timeout-pxjtk: {kubelet ip-172-20-49-224.eu-west-3.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "d869b167075f9d6c93a1521efcd46620587902ae96b211d6c45a95194c140d6e" network for pod "affinity-clusterip-timeout-pxjtk": networkPlugin cni failed to set up pod "affinity-clusterip-timeout-pxjtk_services-9461" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 1 02:36:03.033: INFO: At 2023-01-01 02:27:55 +0000 UTC - event for affinity-clusterip-timeout-lv5vx: {kubelet ip-172-20-45-112.eu-west-3.compute.internal} FailedCreatePodSandBox: (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "b451bc8486d9044fb0856b30bad77b8b5f4cb9fe0116e13b95b6302d0c791346" network for pod "affinity-clusterip-timeout-lv5vx": networkPlugin cni failed to set up pod "affinity-clusterip-timeout-lv5vx_services-9461" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 1 02:36:03.033: INFO: At 2023-01-01 02:27:55 +0000 UTC - event for affinity-clusterip-timeout-pxjtk: {kubelet ip-172-20-49-224.eu-west-3.compute.internal} FailedCreatePodSandBox: (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "f5680cd0662811f02aa5fcdc2dc8a23be2c19b7a419cee6759b1b028e891ca13" network for pod "affinity-clusterip-timeout-pxjtk": networkPlugin cni failed to set up pod "affinity-clusterip-timeout-pxjtk_services-9461" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 1 02:36:03.033: INFO: At 2023-01-01 02:27:55 +0000 UTC - event for affinity-clusterip-timeout-q9qt2: {kubelet ip-172-20-33-66.eu-west-3.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "d4a48600714a1c4e677226458f6577d3c28466c89cd3d52dc8dc30e9af895992" network for pod "affinity-clusterip-timeout-q9qt2": networkPlugin cni failed to set up pod "affinity-clusterip-timeout-q9qt2_services-9461" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 1 02:36:03.033: INFO: At 2023-01-01 02:27:59 +0000 UTC - event for affinity-clusterip-timeout-q9qt2: {kubelet ip-172-20-33-66.eu-west-3.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "6418a09e9cf14f94fe184b51d84aa89456e8054e607fed697906ae9a92ef61e0" network for pod "affinity-clusterip-timeout-q9qt2": networkPlugin cni failed to set up pod "affinity-clusterip-timeout-q9qt2_services-9461" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 1 02:36:03.033: INFO: At 2023-01-01 02:28:02 +0000 UTC - event for affinity-clusterip-timeout-q9qt2: {kubelet ip-172-20-33-66.eu-west-3.compute.internal} FailedCreatePodSandBox: (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "8b0e1affddb2e29454ce270d11dca4f541921f2e86283e3174d19f52ad7a277a" network for pod "affinity-clusterip-timeout-q9qt2": networkPlugin cni failed to set up pod "affinity-clusterip-timeout-q9qt2_services-9461" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 1 02:36:03.138: INFO: POD NODE PHASE GRACE CONDITIONS Jan 1 02:36:03.138: INFO: affinity-clusterip-timeout-lv5vx ip-172-20-45-112.eu-west-3.compute.internal Running [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2023-01-01 02:27:32 +0000 UTC } {Ready True 0001-01-01 00:00:00 +0000 UTC 2023-01-01 02:31:00 +0000 UTC } {ContainersReady True 0001-01-01 00:00:00 +0000 UTC 2023-01-01 02:31:00 +0000 UTC } {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2023-01-01 02:27:32 +0000 UTC }] Jan 1 02:36:03.138: INFO: affinity-clusterip-timeout-pxjtk ip-172-20-49-224.eu-west-3.compute.internal Running [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2023-01-01 02:27:32 +0000 UTC } {Ready True 0001-01-01 00:00:00 +0000 UTC 2023-01-01 02:28:27 +0000 UTC } {ContainersReady True 0001-01-01 00:00:00 +0000 UTC 2023-01-01 02:28:27 +0000 UTC } {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2023-01-01 02:27:32 +0000 UTC }] Jan 1 02:36:03.138: INFO: affinity-clusterip-timeout-q9qt2 ip-172-20-33-66.eu-west-3.compute.internal Pending [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2023-01-01 02:27:32 +0000 UTC } {Ready False 0001-01-01 00:00:00 +0000 UTC 2023-01-01 02:27:32 +0000 UTC ContainersNotReady containers with unready status: [affinity-clusterip-timeout]} {ContainersReady False 0001-01-01 00:00:00 +0000 UTC 2023-01-01 02:27:32 +0000 UTC ContainersNotReady containers with unready status: [affinity-clusterip-timeout]} {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2023-01-01 02:27:32 +0000 UTC }] Jan 1 02:36:03.138: INFO: Jan 1 02:36:03.495: INFO: Unable to fetch services-9461/affinity-clusterip-timeout-q9qt2/affinity-clusterip-timeout logs: the server rejected our request for an unknown reason (get pods affinity-clusterip-timeout-q9qt2) Jan 1 02:36:03.602: INFO: Logging node info for node ip-172-20-33-66.eu-west-3.compute.internal Jan 1 02:36:03.706: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-33-66.eu-west-3.compute.internal ec4b59b3-62db-4040-aad2-4c3249f80fe8 10705 0 2023-01-01 02:20:25 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:eu-west-3 failure-domain.beta.kubernetes.io/zone:eu-west-3a io.kubernetes.storage.mock/node:some-mock-node kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-33-66.eu-west-3.compute.internal kubernetes.io/os:linux kubernetes.io/role:node node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:eu-west-3a topology.kubernetes.io/region:eu-west-3 topology.kubernetes.io/zone:eu-west-3a] map[csi.volume.kubernetes.io/nodeid:{"csi-mock-csi-mock-volumes-2206":"csi-mock-csi-mock-volumes-2206","csi-mock-csi-mock-volumes-8129":"csi-mock-csi-mock-volumes-8129","ebs.csi.aws.com":"i-028313673124ac4ad"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kops-controller Update v1 2023-01-01 02:20:25 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}} } {kube-controller-manager Update v1 2023-01-01 02:20:25 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.1.0/24\"":{}}}} } {kubelet Update v1 2023-01-01 02:20:25 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {kubelet Update v1 2023-01-01 02:31:50 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:io.kubernetes.storage.mock/node":{},"f:topology.ebs.csi.aws.com/zone":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.1.0/24,DoNotUseExternalID:,ProviderID:aws:///eu-west-3a/i-028313673124ac4ad,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.1.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{50478186496 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4078190592 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{45430367772 0} {<nil>} 45430367772 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3973332992 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-01-01 02:31:50 +0000 UTC,LastTransitionTime:2023-01-01 02:20:25 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-01-01 02:31:50 +0000 UTC,LastTransitionTime:2023-01-01 02:20:25 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-01-01 02:31:50 +0000 UTC,LastTransitionTime:2023-01-01 02:20:25 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-01-01 02:31:50 +0000 UTC,LastTransitionTime:2023-01-01 02:20:46 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.33.66,},NodeAddress{Type:ExternalIP,Address:52.47.140.112,},NodeAddress{Type:Hostname,Address:ip-172-20-33-66.eu-west-3.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-33-66.eu-west-3.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-52-47-140-112.eu-west-3.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2ad2ce1a87d508c178acf6215c5767,SystemUUID:ec2ad2ce-1a87-d508-c178-acf6215c5767,BootID:b3b60783-d786-412b-a36a-802bcfae604d,KernelVersion:4.19.0-23-cloud-amd64,OSImage:Debian GNU/Linux 10 (buster),ContainerRuntimeVersion:docker://20.10.17,KubeletVersion:v1.23.15,KubeProxyVersion:v1.23.15,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:456569528,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/jessie-dnsutils@sha256:11e6a66017ba4e4b938c1612b7a54a3befcefd354796c04e1dba76873a13518e k8s.gcr.io/e2e-test-images/jessie-dnsutils:1.5],SizeBytes:253371792,},ContainerImage{Names:[nginx@sha256:0047b729188a15da49380d9506d65959cce6d40291ccfb4e039f5dc7efd33286 nginx:latest],SizeBytes:141812353,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:7e8bdd271312fd25fc5ff5a8f04727be84044eb3d7d8d03611972a6752e2e11e k8s.gcr.io/e2e-test-images/agnhost:2.39],SizeBytes:126872991,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:1b9d1b2f36cb2dbee1960e82a9344aeb11bd4c4c03abf5e1853e0559c23855e3 k8s.gcr.io/e2e-test-images/httpd:2.4.38-2],SizeBytes:123781643,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.23.15 registry.k8s.io/kube-proxy-amd64:v1.23.15],SizeBytes:112336036,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:10e231cf07c82dfab5141c1afe548dc734e5fc1f67665ec3982a325d1bd31a9a registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.0],SizeBytes:94471559,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:c8e03f60afa90a28e4bb6ec9a8d0fc36d89de4b7475cf2d613afa793ec969fe0 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.0],SizeBytes:56382329,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:c5be65d6679efabb969d9b019300d187437ae876f992c40911fd2892bbef3b36 k8s.gcr.io/sig-storage/csi-attacher:v3.2.0],SizeBytes:53500826,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:017727efcfeb7d053af68e51436ce8e65edbc6ca573720afb4f79c8594036955 registry.k8s.io/coredns/coredns:v1.10.0],SizeBytes:50249337,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:a61d309da54641db41fb8f35718f744e9f730d4d0384f8c4b186ddc9f06cbd5f k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.1.0],SizeBytes:19662887,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:19582112,},ContainerImage{Names:[k8s.gcr.io/sig-storage/mock-driver@sha256:a7b517f9e0f42ffade802eef9cefa271372386b85f55b702b493241e58459793 k8s.gcr.io/sig-storage/mock-driver:v4.1.0],SizeBytes:17680993,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:17611980,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:13616070e3f29de4417eee434a8ef472221c9e51b3d037b5a6b46cef08eb7443 k8s.gcr.io/e2e-test-images/nginx:1.14-2],SizeBytes:16032814,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nonewprivs@sha256:8ac1264691820febacf3aea5d152cbde6d10685731ec14966a9401c6f47a68ac k8s.gcr.io/e2e-test-images/nonewprivs:1.3],SizeBytes:7107254,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf k8s.gcr.io/e2e-test-images/busybox:1.29-2],SizeBytes:1154361,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:39e1e963e5310e9c313bad51523be012ede7b35bb9316517d19089a010356592 k8s.gcr.io/e2e-test-images/busybox:1.29-1],SizeBytes:1154361,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db k8s.gcr.io/pause:3.6 registry.k8s.io/pause:3.6],SizeBytes:682696,},},VolumesInUse:[kubernetes.io/csi/csi-mock-csi-mock-volumes-2206^4],VolumesAttached:[]AttachedVolume{},Config:nil,},} Jan 1 02:36:03.707: INFO: Logging kubelet events for node ip-172-20-33-66.eu-west-3.compute.internal Jan 1 02:36:03.813: INFO: Logging pods the kubelet thinks is on node ip-172-20-33-66.eu-west-3.compute.internal Jan 1 02:36:04.027: INFO: coredns-867df8f45c-cqmp7 started at 2023-01-01 02:21:18 +0000 UTC (0+1 container statuses recorded) Jan 1 02:36:04.027: INFO: Container coredns ready: true, restart count 0 Jan 1 02:36:04.027: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-l9hnv started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:36:04.027: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:36:04.027: INFO: local-client started at 2023-01-01 02:31:07 +0000 UTC (0+1 container statuses recorded) Jan 1 02:36:04.027: INFO: Container local-client ready: false, restart count 0 Jan 1 02:36:04.027: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-vnv2x started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:36:04.027: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:36:04.027: INFO: csi-mockplugin-0 started at 2023-01-01 02:26:15 +0000 UTC (0+4 container statuses recorded) Jan 1 02:36:04.027: INFO: Container busybox ready: true, restart count 0 Jan 1 02:36:04.027: INFO: Container csi-provisioner ready: true, restart count 0 Jan 1 02:36:04.027: INFO: Container driver-registrar ready: true, restart count 0 Jan 1 02:36:04.027: INFO: Container mock ready: true, restart count 0 Jan 1 02:36:04.027: INFO: hostexec-ip-172-20-33-66.eu-west-3.compute.internal-qm7jt started at 2023-01-01 02:27:26 +0000 UTC (0+1 container statuses recorded) Jan 1 02:36:04.027: INFO: Container agnhost-container ready: true, restart count 0 Jan 1 02:36:04.027: INFO: netserver-0 started at 2023-01-01 02:33:25 +0000 UTC (0+1 container statuses recorded) Jan 1 02:36:04.027: INFO: Container webserver ready: false, restart count 0 Jan 1 02:36:04.027: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-csnzw started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:36:04.027: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: false, restart count 0 Jan 1 02:36:04.027: INFO: ebs-csi-node-vsxjc started at 2023-01-01 02:20:26 +0000 UTC (0+3 container statuses recorded) Jan 1 02:36:04.027: INFO: Container ebs-plugin ready: true, restart count 0 Jan 1 02:36:04.027: INFO: Container liveness-probe ready: true, restart count 0 Jan 1 02:36:04.027: INFO: Container node-driver-registrar ready: true, restart count 0 Jan 1 02:36:04.027: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-h4nvd started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:36:04.027: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:36:04.027: INFO: cilium-vbcgc started at 2023-01-01 02:20:26 +0000 UTC (1+1 container statuses recorded) Jan 1 02:36:04.027: INFO: Init container clean-cilium-state ready: true, restart count 0 Jan 1 02:36:04.027: INFO: Container cilium-agent ready: true, restart count 0 Jan 1 02:36:04.027: INFO: pvc-volume-tester-8t67k started at 2023-01-01 02:31:45 +0000 UTC (0+1 container statuses recorded) Jan 1 02:36:04.027: INFO: Container volume-tester ready: false, restart count 0 Jan 1 02:36:04.027: INFO: affinity-clusterip-timeout-q9qt2 started at 2023-01-01 02:27:32 +0000 UTC (0+1 container statuses recorded) Jan 1 02:36:04.027: INFO: Container affinity-clusterip-timeout ready: false, restart count 0 Jan 1 02:36:04.027: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-m6vq2 started at 2023-01-01 02:25:17 +0000 UTC (0+1 container statuses recorded) Jan 1 02:36:04.027: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:36:04.027: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-h6mmt started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:36:04.027: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:36:04.028: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-dkfpd started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:36:04.028: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:36:04.028: INFO: pvc-volume-tester-462k4 started at 2023-01-01 02:32:02 +0000 UTC (0+1 container statuses recorded) Jan 1 02:36:04.028: INFO: Container volume-tester ready: false, restart count 0 Jan 1 02:36:04.028: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-546bc started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:36:04.028: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:36:04.028: INFO: csi-mockplugin-attacher-0 started at 2023-01-01 02:27:56 +0000 UTC (0+1 container statuses recorded) Jan 1 02:36:04.028: INFO: Container csi-attacher ready: false, restart count 0 Jan 1 02:36:04.028: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-q8pgp started at 2023-01-01 02:25:17 +0000 UTC (0+1 container statuses recorded) Jan 1 02:36:04.028: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:36:04.028: INFO: netserver-0 started at 2023-01-01 02:31:31 +0000 UTC (0+1 container statuses recorded) Jan 1 02:36:04.028: INFO: Container webserver ready: false, restart count 0 Jan 1 02:36:04.028: INFO: csi-mockplugin-0 started at 2023-01-01 02:27:56 +0000 UTC (0+3 container statuses recorded) Jan 1 02:36:04.028: INFO: Container csi-provisioner ready: true, restart count 0 Jan 1 02:36:04.028: INFO: Container driver-registrar ready: true, restart count 0 Jan 1 02:36:04.028: INFO: Container mock ready: true, restart count 0 Jan 1 02:36:04.028: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-2v87j started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:36:04.028: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:36:04.434: INFO: Latency metrics for node ip-172-20-33-66.eu-west-3.compute.internal Jan 1 02:36:04.434: INFO: Logging node info for node ip-172-20-38-238.eu-west-3.compute.internal Jan 1 02:36:04.538: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-38-238.eu-west-3.compute.internal 0eb713b4-64bf-47b7-8090-bab8111218ee 9308 0 2023-01-01 02:18:51 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:c5.large beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:eu-west-3 failure-domain.beta.kubernetes.io/zone:eu-west-3a kops.k8s.io/kops-controller-pki: kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-38-238.eu-west-3.compute.internal kubernetes.io/os:linux kubernetes.io/role:master node-role.kubernetes.io/control-plane: node-role.kubernetes.io/master: node.kubernetes.io/exclude-from-external-load-balancers: node.kubernetes.io/instance-type:c5.large topology.ebs.csi.aws.com/zone:eu-west-3a topology.kubernetes.io/region:eu-west-3 topology.kubernetes.io/zone:eu-west-3a] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-0f70ebb30a9076648"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kubelet Update v1 2023-01-01 02:18:50 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {protokube Update v1 2023-01-01 02:19:16 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kops.k8s.io/kops-controller-pki":{},"f:node-role.kubernetes.io/control-plane":{},"f:node.kubernetes.io/exclude-from-external-load-balancers":{}}}} } {kops-controller Update v1 2023-01-01 02:19:21 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/master":{}}}} } {kube-controller-manager Update v1 2023-01-01 02:19:31 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.0.0/24\"":{}},"f:taints":{}}} } {kubelet Update v1 2023-01-01 02:20:36 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.0.0/24,DoNotUseExternalID:,ProviderID:aws:///eu-west-3a/i-0f70ebb30a9076648,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/master,Value:,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[100.96.0.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{50478186496 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3918811136 0} {<nil>} 3826964Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{45430367772 0} {<nil>} 45430367772 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3813953536 0} {<nil>} 3724564Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-01-01 02:31:26 +0000 UTC,LastTransitionTime:2023-01-01 02:18:45 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-01-01 02:31:26 +0000 UTC,LastTransitionTime:2023-01-01 02:18:45 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-01-01 02:31:26 +0000 UTC,LastTransitionTime:2023-01-01 02:18:45 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-01-01 02:31:26 +0000 UTC,LastTransitionTime:2023-01-01 02:19:31 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.38.238,},NodeAddress{Type:ExternalIP,Address:15.237.26.58,},NodeAddress{Type:Hostname,Address:ip-172-20-38-238.eu-west-3.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-38-238.eu-west-3.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-15-237-26-58.eu-west-3.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2e6c64b5b811bf0c78b2aa7ccdb943,SystemUUID:ec2e6c64-b5b8-11bf-0c78-b2aa7ccdb943,BootID:9db01fa3-d136-4230-b21a-b03abbfb6bff,KernelVersion:4.19.0-23-cloud-amd64,OSImage:Debian GNU/Linux 10 (buster),ContainerRuntimeVersion:docker://20.10.17,KubeletVersion:v1.23.15,KubeProxyVersion:v1.23.15,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[registry.k8s.io/etcdadm/etcd-manager@sha256:66a453db625abb268f4b3bbefc5a34a171d81e6e8796cecca54cfd71775c77c4 registry.k8s.io/etcdadm/etcd-manager:v3.0.20221209],SizeBytes:662209237,},ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:456569528,},ContainerImage{Names:[k8s.gcr.io/kube-apiserver-amd64:v1.23.15 registry.k8s.io/kube-apiserver-amd64:v1.23.15],SizeBytes:135240444,},ContainerImage{Names:[k8s.gcr.io/kube-controller-manager-amd64:v1.23.15 registry.k8s.io/kube-controller-manager-amd64:v1.23.15],SizeBytes:125046097,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.23.15 registry.k8s.io/kube-proxy-amd64:v1.23.15],SizeBytes:112336036,},ContainerImage{Names:[quay.io/cilium/operator@sha256:a6d24a006a6b92967ac90786b49bc1ac26e5477cf028cd1186efcfc2466484db quay.io/cilium/operator:v1.12.5],SizeBytes:110016956,},ContainerImage{Names:[registry.k8s.io/kops/kops-controller:1.27.0-alpha.1],SizeBytes:105197757,},ContainerImage{Names:[registry.k8s.io/kops/dns-controller:1.27.0-alpha.1],SizeBytes:102577531,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:10e231cf07c82dfab5141c1afe548dc734e5fc1f67665ec3982a325d1bd31a9a registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.0],SizeBytes:94471559,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:122bfb8c1edabb3c0edd63f06523e6940d958d19b3957dc7b1d6f81e9f1f6119 registry.k8s.io/sig-storage/csi-provisioner:v3.1.0],SizeBytes:57736070,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:9ebbf9f023e7b41ccee3d52afe39a89e3ddacdbb69269d583abfc25847cfd9e4 registry.k8s.io/sig-storage/csi-resizer:v1.4.0],SizeBytes:55464778,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:8b9c313c05f54fb04f8d430896f5f5904b6cb157df261501b29adc04d2b2dc7b registry.k8s.io/sig-storage/csi-attacher:v3.4.0],SizeBytes:54848595,},ContainerImage{Names:[k8s.gcr.io/kube-scheduler-amd64:v1.23.15 registry.k8s.io/kube-scheduler-amd64:v1.23.15],SizeBytes:53521430,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:19582112,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:17611980,},ContainerImage{Names:[registry.k8s.io/kops/kube-apiserver-healthcheck:1.27.0-alpha.1],SizeBytes:8787111,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:682696,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Jan 1 02:36:04.538: INFO: Logging kubelet events for node ip-172-20-38-238.eu-west-3.compute.internal Jan 1 02:36:04.644: INFO: Logging pods the kubelet thinks is on node ip-172-20-38-238.eu-west-3.compute.internal Jan 1 02:36:04.755: INFO: kube-controller-manager-ip-172-20-38-238.eu-west-3.compute.internal started at 2023-01-01 02:18:11 +0000 UTC (0+1 container statuses recorded) Jan 1 02:36:04.755: INFO: Container kube-controller-manager ready: true, restart count 2 Jan 1 02:36:04.755: INFO: kube-scheduler-ip-172-20-38-238.eu-west-3.compute.internal started at 2023-01-01 02:18:11 +0000 UTC (0+1 container statuses recorded) Jan 1 02:36:04.755: INFO: Container kube-scheduler ready: true, restart count 0 Jan 1 02:36:04.755: INFO: kube-apiserver-ip-172-20-38-238.eu-west-3.compute.internal started at 2023-01-01 02:18:11 +0000 UTC (0+2 container statuses recorded) Jan 1 02:36:04.755: INFO: Container healthcheck ready: true, restart count 0 Jan 1 02:36:04.755: INFO: Container kube-apiserver ready: true, restart count 1 Jan 1 02:36:04.755: INFO: kops-controller-26gzg started at 2023-01-01 02:19:20 +0000 UTC (0+1 container statuses recorded) Jan 1 02:36:04.755: INFO: Container kops-controller ready: true, restart count 0 Jan 1 02:36:04.755: INFO: ebs-csi-controller-6dc545b75-5qqh8 started at 2023-01-01 02:19:20 +0000 UTC (0+5 container statuses recorded) Jan 1 02:36:04.755: INFO: Container csi-attacher ready: true, restart count 0 Jan 1 02:36:04.755: INFO: Container csi-provisioner ready: true, restart count 0 Jan 1 02:36:04.755: INFO: Container csi-resizer ready: true, restart count 0 Jan 1 02:36:04.755: INFO: Container ebs-plugin ready: true, restart count 0 Jan 1 02:36:04.755: INFO: Container liveness-probe ready: true, restart count 0 Jan 1 02:36:04.755: INFO: etcd-manager-events-ip-172-20-38-238.eu-west-3.compute.internal started at 2023-01-01 02:18:11 +0000 UTC (0+1 container statuses recorded) Jan 1 02:36:04.755: INFO: Container etcd-manager ready: true, restart count 0 Jan 1 02:36:04.755: INFO: etcd-manager-main-ip-172-20-38-238.eu-west-3.compute.internal started at 2023-01-01 02:18:11 +0000 UTC (0+1 container statuses recorded) Jan 1 02:36:04.755: INFO: Container etcd-manager ready: true, restart count 0 Jan 1 02:36:04.755: INFO: cilium-qk59v started at 2023-01-01 02:19:20 +0000 UTC (1+1 container statuses recorded) Jan 1 02:36:04.755: INFO: Init container clean-cilium-state ready: true, restart count 0 Jan 1 02:36:04.755: INFO: Container cilium-agent ready: true, restart count 0 Jan 1 02:36:04.755: INFO: ebs-csi-node-k87k6 started at 2023-01-01 02:19:20 +0000 UTC (0+3 container statuses recorded) Jan 1 02:36:04.755: INFO: Container ebs-plugin ready: true, restart count 0 Jan 1 02:36:04.755: INFO: Container liveness-probe ready: true, restart count 0 Jan 1 02:36:04.755: INFO: Container node-driver-registrar ready: true, restart count 0 Jan 1 02:36:04.755: INFO: dns-controller-7db8b74b99-58zzw started at 2023-01-01 02:19:20 +0000 UTC (0+1 container statuses recorded) Jan 1 02:36:04.755: INFO: Container dns-controller ready: true, restart count 0 Jan 1 02:36:04.755: INFO: cilium-operator-d47487c76-j84nq started at 2023-01-01 02:19:20 +0000 UTC (0+1 container statuses recorded) Jan 1 02:36:04.755: INFO: Container cilium-operator ready: true, restart count 0 Jan 1 02:36:05.091: INFO: Latency metrics for node ip-172-20-38-238.eu-west-3.compute.internal Jan 1 02:36:05.091: INFO: Logging node info for node ip-172-20-45-112.eu-west-3.compute.internal Jan 1 02:36:05.195: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-45-112.eu-west-3.compute.internal 48eed937-05bb-4d8a-bddd-aa94a2e6ab92 10711 0 2023-01-01 02:20:27 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:eu-west-3 failure-domain.beta.kubernetes.io/zone:eu-west-3a kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-45-112.eu-west-3.compute.internal kubernetes.io/os:linux kubernetes.io/role:node node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:eu-west-3a topology.hostpath.csi/node:ip-172-20-45-112.eu-west-3.compute.internal topology.kubernetes.io/region:eu-west-3 topology.kubernetes.io/zone:eu-west-3a] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-0778adf2085fa7f34"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kube-controller-manager Update v1 2023-01-01 02:20:27 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.3.0/24\"":{}}}} } {kubelet Update v1 2023-01-01 02:20:27 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {kops-controller Update v1 2023-01-01 02:20:28 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}} } {kube-controller-manager Update v1 2023-01-01 02:30:49 +0000 UTC FieldsV1 {"f:status":{"f:volumesAttached":{}}} status} {kubelet Update v1 2023-01-01 02:30:52 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:capacity":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.3.0/24,DoNotUseExternalID:,ProviderID:aws:///eu-west-3a/i-0778adf2085fa7f34,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.3.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{50478186496 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4078190592 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{45430367772 0} {<nil>} 45430367772 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3973332992 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-01-01 02:32:03 +0000 UTC,LastTransitionTime:2023-01-01 02:20:27 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-01-01 02:32:03 +0000 UTC,LastTransitionTime:2023-01-01 02:20:27 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-01-01 02:32:03 +0000 UTC,LastTransitionTime:2023-01-01 02:20:27 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-01-01 02:32:03 +0000 UTC,LastTransitionTime:2023-01-01 02:20:48 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.45.112,},NodeAddress{Type:ExternalIP,Address:13.38.32.14,},NodeAddress{Type:Hostname,Address:ip-172-20-45-112.eu-west-3.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-45-112.eu-west-3.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-13-38-32-14.eu-west-3.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2dbdf846e8ff614f4555e5bf9dbbf4,SystemUUID:ec2dbdf8-46e8-ff61-4f45-55e5bf9dbbf4,BootID:10a054ae-621a-47ce-9566-b0048cc5101b,KernelVersion:4.19.0-23-cloud-amd64,OSImage:Debian GNU/Linux 10 (buster),ContainerRuntimeVersion:docker://20.10.17,KubeletVersion:v1.23.15,KubeProxyVersion:v1.23.15,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:456569528,},ContainerImage{Names:[nginx@sha256:0047b729188a15da49380d9506d65959cce6d40291ccfb4e039f5dc7efd33286 nginx:latest],SizeBytes:141812353,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:7e8bdd271312fd25fc5ff5a8f04727be84044eb3d7d8d03611972a6752e2e11e k8s.gcr.io/e2e-test-images/agnhost:2.39],SizeBytes:126872991,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:1b9d1b2f36cb2dbee1960e82a9344aeb11bd4c4c03abf5e1853e0559c23855e3 k8s.gcr.io/e2e-test-images/httpd:2.4.38-2],SizeBytes:123781643,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.23.15 registry.k8s.io/kube-proxy-amd64:v1.23.15],SizeBytes:112336036,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:10e231cf07c82dfab5141c1afe548dc734e5fc1f67665ec3982a325d1bd31a9a registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.0],SizeBytes:94471559,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:c8e03f60afa90a28e4bb6ec9a8d0fc36d89de4b7475cf2d613afa793ec969fe0 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.0],SizeBytes:56382329,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:4e74c0492bceddc598de1c90cc5bc14dcda94cb49fa9c5bad9d117c4834b5e08 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.1],SizeBytes:56379269,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:36c31f7e1f433c9634d24f876353e8646246d81a03c4e351202c2644daff1620 k8s.gcr.io/sig-storage/csi-resizer:v1.2.0],SizeBytes:53956308,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:ed98431376c9e944e19a465fe8ea944806714dd95416a0821096c78d66b579bd k8s.gcr.io/sig-storage/csi-snapshotter:v4.1.1],SizeBytes:53531990,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:c5be65d6679efabb969d9b019300d187437ae876f992c40911fd2892bbef3b36 k8s.gcr.io/sig-storage/csi-attacher:v3.2.0],SizeBytes:53500826,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:60ab9b3e6a030d3038c87c0d6bca2930f58d1d72823e6a4af09767dc83b696a2 k8s.gcr.io/sig-storage/csi-attacher:v3.2.1],SizeBytes:53499570,},ContainerImage{Names:[k8s.gcr.io/sig-storage/hostpathplugin@sha256:232fe80174d60d520d36043103853a1d7ab4b7f3782cf43e45034f04ccda58ce k8s.gcr.io/sig-storage/hostpathplugin:v1.7.1],SizeBytes:29977921,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:a61d309da54641db41fb8f35718f744e9f730d4d0384f8c4b186ddc9f06cbd5f k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.1.0],SizeBytes:19662887,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:19582112,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:2dee3fe5fe861bb66c3a4ac51114f3447a4cd35870e0f2e2b558c7a400d89589 k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.2.0],SizeBytes:18699436,},ContainerImage{Names:[k8s.gcr.io/sig-storage/mock-driver@sha256:a7b517f9e0f42ffade802eef9cefa271372386b85f55b702b493241e58459793 k8s.gcr.io/sig-storage/mock-driver:v4.1.0],SizeBytes:17680993,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:17611980,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:529be2c9770add0cdd0c989115222ea9fc1be430c11095eb9f6dafcf98a36e2b k8s.gcr.io/sig-storage/livenessprobe:v2.4.0],SizeBytes:17175488,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:13616070e3f29de4417eee434a8ef472221c9e51b3d037b5a6b46cef08eb7443 k8s.gcr.io/e2e-test-images/nginx:1.14-2],SizeBytes:16032814,},ContainerImage{Names:[gcr.io/authenticated-image-pulling/alpine@sha256:7ff177862cb50c602bfe81f805969412e619c054a2bbead977d0c276988aa4a0 gcr.io/authenticated-image-pulling/alpine:3.7],SizeBytes:4206620,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf k8s.gcr.io/e2e-test-images/busybox:1.29-2],SizeBytes:1154361,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:39e1e963e5310e9c313bad51523be012ede7b35bb9316517d19089a010356592 k8s.gcr.io/e2e-test-images/busybox:1.29-1],SizeBytes:1154361,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db k8s.gcr.io/pause:3.6 registry.k8s.io/pause:3.6],SizeBytes:682696,},},VolumesInUse:[kubernetes.io/csi/ebs.csi.aws.com^vol-03d4513454f149737 kubernetes.io/csi/ebs.csi.aws.com^vol-08601772399ba2131],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-03d4513454f149737,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-08601772399ba2131,DevicePath:,},},Config:nil,},} Jan 1 02:36:05.195: INFO: Logging kubelet events for node ip-172-20-45-112.eu-west-3.compute.internal Jan 1 02:36:05.301: INFO: Logging pods the kubelet thinks is on node ip-172-20-45-112.eu-west-3.compute.internal Jan 1 02:36:05.414: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-gph2d started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:36:05.414: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:36:05.414: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-rmdrr started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:36:05.414: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:36:05.414: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-kjg2m started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:36:05.414: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:36:05.414: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-59bdj started at 2023-01-01 02:25:17 +0000 UTC (0+1 container statuses recorded) Jan 1 02:36:05.414: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:36:05.414: INFO: ebs-csi-node-7clrr started at 2023-01-01 02:20:28 +0000 UTC (0+3 container statuses recorded) Jan 1 02:36:05.414: INFO: Container ebs-plugin ready: true, restart count 0 Jan 1 02:36:05.414: INFO: Container liveness-probe ready: true, restart count 0 Jan 1 02:36:05.414: INFO: Container node-driver-registrar ready: true, restart count 0 Jan 1 02:36:05.414: INFO: inline-volume-tester-fpr45 started at 2023-01-01 02:31:04 +0000 UTC (0+1 container statuses recorded) Jan 1 02:36:05.414: INFO: Container csi-volume-tester ready: false, restart count 0 Jan 1 02:36:05.414: INFO: suspend-false-to-true-kb2jl started at 2023-01-01 02:31:57 +0000 UTC (0+1 container statuses recorded) Jan 1 02:36:05.414: INFO: Container c ready: false, restart count 0 Jan 1 02:36:05.414: INFO: netserver-1 started at 2023-01-01 02:31:32 +0000 UTC (0+1 container statuses recorded) Jan 1 02:36:05.414: INFO: Container webserver ready: false, restart count 0 Jan 1 02:36:05.414: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-jh8bc started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:36:05.414: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:36:05.414: INFO: pod-1 started at 2023-01-01 02:31:26 +0000 UTC (0+1 container statuses recorded) Jan 1 02:36:05.414: INFO: Container donothing ready: false, restart count 0 Jan 1 02:36:05.414: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-4lrj2 started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:36:05.414: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:36:05.414: INFO: netserver-1 started at 2023-01-01 02:33:25 +0000 UTC (0+1 container statuses recorded) Jan 1 02:36:05.414: INFO: Container webserver ready: false, restart count 0 Jan 1 02:36:05.414: INFO: cilium-slx98 started at 2023-01-01 02:20:28 +0000 UTC (1+1 container statuses recorded) Jan 1 02:36:05.414: INFO: Init container clean-cilium-state ready: true, restart count 0 Jan 1 02:36:05.414: INFO: Container cilium-agent ready: true, restart count 0 Jan 1 02:36:05.414: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-srf5k started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:36:05.414: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:36:05.414: INFO: pod-0 started at 2023-01-01 02:31:26 +0000 UTC (0+1 container statuses recorded) Jan 1 02:36:05.414: INFO: Container donothing ready: true, restart count 0 Jan 1 02:36:05.414: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-4mml9 started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:36:05.414: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:36:05.414: INFO: affinity-clusterip-timeout-lv5vx started at 2023-01-01 02:27:32 +0000 UTC (0+1 container statuses recorded) Jan 1 02:36:05.414: INFO: Container affinity-clusterip-timeout ready: true, restart count 0 Jan 1 02:36:05.414: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-xnfzj started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:36:05.414: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:36:05.414: INFO: pod-d05d599f-1b25-4249-b9c7-c64a79abebd3 started at 2023-01-01 02:30:59 +0000 UTC (0+1 container statuses recorded) Jan 1 02:36:05.414: INFO: Container write-pod ready: false, restart count 0 Jan 1 02:36:05.414: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-x5cp6 started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:36:05.414: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:36:05.804: INFO: Latency metrics for node ip-172-20-45-112.eu-west-3.compute.internal Jan 1 02:36:05.804: INFO: Logging node info for node ip-172-20-47-178.eu-west-3.compute.internal Jan 1 02:36:05.909: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-47-178.eu-west-3.compute.internal 835e5337-d2b9-45ad-bf0e-9d32514f71ba 10714 0 2023-01-01 02:20:25 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:eu-west-3 failure-domain.beta.kubernetes.io/zone:eu-west-3a kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-47-178.eu-west-3.compute.internal kubernetes.io/os:linux kubernetes.io/role:node node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:eu-west-3a topology.hostpath.csi/node:ip-172-20-47-178.eu-west-3.compute.internal topology.kubernetes.io/region:eu-west-3 topology.kubernetes.io/zone:eu-west-3a] map[csi.volume.kubernetes.io/nodeid:{"csi-hostpath-ephemeral-1708":"ip-172-20-47-178.eu-west-3.compute.internal","ebs.csi.aws.com":"i-00e1e60199f5a4d04"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kops-controller Update v1 2023-01-01 02:20:25 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}} } {kube-controller-manager Update v1 2023-01-01 02:20:25 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.2.0/24\"":{}}}} } {kubelet Update v1 2023-01-01 02:20:25 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {kube-controller-manager Update v1 2023-01-01 02:32:52 +0000 UTC FieldsV1 {"f:status":{"f:volumesAttached":{}}} status} {kubelet Update v1 2023-01-01 02:33:00 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:capacity":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.2.0/24,DoNotUseExternalID:,ProviderID:aws:///eu-west-3a/i-00e1e60199f5a4d04,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.2.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{50478186496 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4078190592 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{45430367772 0} {<nil>} 45430367772 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3973332992 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-01-01 02:33:00 +0000 UTC,LastTransitionTime:2023-01-01 02:20:25 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-01-01 02:33:00 +0000 UTC,LastTransitionTime:2023-01-01 02:20:25 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-01-01 02:33:00 +0000 UTC,LastTransitionTime:2023-01-01 02:20:25 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-01-01 02:33:00 +0000 UTC,LastTransitionTime:2023-01-01 02:20:46 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.47.178,},NodeAddress{Type:ExternalIP,Address:13.36.177.101,},NodeAddress{Type:Hostname,Address:ip-172-20-47-178.eu-west-3.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-47-178.eu-west-3.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-13-36-177-101.eu-west-3.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec243eda226c96d5573f3eaa3203df0d,SystemUUID:ec243eda-226c-96d5-573f-3eaa3203df0d,BootID:15dccaa7-d7c7-419d-b7de-76be5781a51a,KernelVersion:4.19.0-23-cloud-amd64,OSImage:Debian GNU/Linux 10 (buster),ContainerRuntimeVersion:docker://20.10.17,KubeletVersion:v1.23.15,KubeProxyVersion:v1.23.15,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:456569528,},ContainerImage{Names:[nginx@sha256:0047b729188a15da49380d9506d65959cce6d40291ccfb4e039f5dc7efd33286 nginx:latest],SizeBytes:141812353,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:7e8bdd271312fd25fc5ff5a8f04727be84044eb3d7d8d03611972a6752e2e11e k8s.gcr.io/e2e-test-images/agnhost:2.39],SizeBytes:126872991,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.23.15 registry.k8s.io/kube-proxy-amd64:v1.23.15],SizeBytes:112336036,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:10e231cf07c82dfab5141c1afe548dc734e5fc1f67665ec3982a325d1bd31a9a registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.0],SizeBytes:94471559,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:c8e03f60afa90a28e4bb6ec9a8d0fc36d89de4b7475cf2d613afa793ec969fe0 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.0],SizeBytes:56382329,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:4e74c0492bceddc598de1c90cc5bc14dcda94cb49fa9c5bad9d117c4834b5e08 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.1],SizeBytes:56379269,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:36c31f7e1f433c9634d24f876353e8646246d81a03c4e351202c2644daff1620 k8s.gcr.io/sig-storage/csi-resizer:v1.2.0],SizeBytes:53956308,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:ed98431376c9e944e19a465fe8ea944806714dd95416a0821096c78d66b579bd k8s.gcr.io/sig-storage/csi-snapshotter:v4.1.1],SizeBytes:53531990,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:c5be65d6679efabb969d9b019300d187437ae876f992c40911fd2892bbef3b36 k8s.gcr.io/sig-storage/csi-attacher:v3.2.0],SizeBytes:53500826,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:60ab9b3e6a030d3038c87c0d6bca2930f58d1d72823e6a4af09767dc83b696a2 k8s.gcr.io/sig-storage/csi-attacher:v3.2.1],SizeBytes:53499570,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:017727efcfeb7d053af68e51436ce8e65edbc6ca573720afb4f79c8594036955 registry.k8s.io/coredns/coredns:v1.10.0],SizeBytes:50249337,},ContainerImage{Names:[registry.k8s.io/cpa/cluster-proportional-autoscaler@sha256:68d396900aeaa072c1f27289485fdac29834045a6f3ffe369bf389d830ef572d registry.k8s.io/cpa/cluster-proportional-autoscaler:1.8.6],SizeBytes:47910609,},ContainerImage{Names:[k8s.gcr.io/sig-storage/hostpathplugin@sha256:232fe80174d60d520d36043103853a1d7ab4b7f3782cf43e45034f04ccda58ce k8s.gcr.io/sig-storage/hostpathplugin:v1.7.1],SizeBytes:29977921,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:a61d309da54641db41fb8f35718f744e9f730d4d0384f8c4b186ddc9f06cbd5f k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.1.0],SizeBytes:19662887,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:19582112,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:2dee3fe5fe861bb66c3a4ac51114f3447a4cd35870e0f2e2b558c7a400d89589 k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.2.0],SizeBytes:18699436,},ContainerImage{Names:[k8s.gcr.io/sig-storage/mock-driver@sha256:a7b517f9e0f42ffade802eef9cefa271372386b85f55b702b493241e58459793 k8s.gcr.io/sig-storage/mock-driver:v4.1.0],SizeBytes:17680993,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:17611980,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:529be2c9770add0cdd0c989115222ea9fc1be430c11095eb9f6dafcf98a36e2b k8s.gcr.io/sig-storage/livenessprobe:v2.4.0],SizeBytes:17175488,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:13616070e3f29de4417eee434a8ef472221c9e51b3d037b5a6b46cef08eb7443 k8s.gcr.io/e2e-test-images/nginx:1.14-2],SizeBytes:16032814,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf k8s.gcr.io/e2e-test-images/busybox:1.29-2],SizeBytes:1154361,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db k8s.gcr.io/pause:3.6 registry.k8s.io/pause:3.6],SizeBytes:682696,},},VolumesInUse:[kubernetes.io/csi/csi-hostpath-ephemeral-1708^957f1b8b-897c-11ed-afb4-7606258f5a7f],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/csi-hostpath-ephemeral-1708^957f1b8b-897c-11ed-afb4-7606258f5a7f,DevicePath:,},},Config:nil,},} Jan 1 02:36:05.910: INFO: Logging kubelet events for node ip-172-20-47-178.eu-west-3.compute.internal Jan 1 02:36:06.016: INFO: Logging pods the kubelet thinks is on node ip-172-20-47-178.eu-west-3.compute.internal Jan 1 02:36:06.130: INFO: coredns-867df8f45c-wzntb started at 2023-01-01 02:20:46 +0000 UTC (0+1 container statuses recorded) Jan 1 02:36:06.130: INFO: Container coredns ready: true, restart count 0 Jan 1 02:36:06.130: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-wkclp started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:36:06.130: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:36:06.130: INFO: netserver-2 started at 2023-01-01 02:31:32 +0000 UTC (0+1 container statuses recorded) Jan 1 02:36:06.130: INFO: Container webserver ready: false, restart count 0 Jan 1 02:36:06.130: INFO: netserver-2 started at 2023-01-01 02:33:25 +0000 UTC (0+1 container statuses recorded) Jan 1 02:36:06.130: INFO: Container webserver ready: false, restart count 0 Jan 1 02:36:06.130: INFO: coredns-autoscaler-557ccb4c66-k4lzb started at 2023-01-01 02:20:46 +0000 UTC (0+1 container statuses recorded) Jan 1 02:36:06.130: INFO: Container autoscaler ready: true, restart count 0 Jan 1 02:36:06.130: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-pgkm2 started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:36:06.130: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:36:06.130: INFO: inline-volume-tester-t46bx started at 2023-01-01 02:32:51 +0000 UTC (0+1 container statuses recorded) Jan 1 02:36:06.130: INFO: Container csi-volume-tester ready: false, restart count 0 Jan 1 02:36:06.130: INFO: hostexec-ip-172-20-47-178.eu-west-3.compute.internal-k7k4h started at 2023-01-01 02:30:47 +0000 UTC (0+1 container statuses recorded) Jan 1 02:36:06.130: INFO: Container agnhost-container ready: true, restart count 0 Jan 1 02:36:06.130: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-hrgrq started at 2023-01-01 02:25:17 +0000 UTC (0+1 container statuses recorded) Jan 1 02:36:06.130: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:36:06.130: INFO: csi-hostpathplugin-0 started at 2023-01-01 02:31:02 +0000 UTC (0+7 container statuses recorded) Jan 1 02:36:06.130: INFO: Container csi-attacher ready: false, restart count 0 Jan 1 02:36:06.130: INFO: Container csi-provisioner ready: false, restart count 0 Jan 1 02:36:06.130: INFO: Container csi-resizer ready: false, restart count 0 Jan 1 02:36:06.130: INFO: Container csi-snapshotter ready: false, restart count 0 Jan 1 02:36:06.130: INFO: Container hostpath ready: false, restart count 0 Jan 1 02:36:06.130: INFO: Container liveness-probe ready: false, restart count 0 Jan 1 02:36:06.130: INFO: Container node-driver-registrar ready: false, restart count 0 Jan 1 02:36:06.130: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-grrsl started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:36:06.130: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:36:06.130: INFO: csi-hostpathplugin-0 started at 2023-01-01 02:30:56 +0000 UTC (0+7 container statuses recorded) Jan 1 02:36:06.131: INFO: Container csi-attacher ready: true, restart count 0 Jan 1 02:36:06.131: INFO: Container csi-provisioner ready: true, restart count 0 Jan 1 02:36:06.131: INFO: Container csi-resizer ready: true, restart count 0 Jan 1 02:36:06.131: INFO: Container csi-snapshotter ready: true, restart count 0 Jan 1 02:36:06.131: INFO: Container hostpath ready: true, restart count 0 Jan 1 02:36:06.131: INFO: Container liveness-probe ready: true, restart count 0 Jan 1 02:36:06.131: INFO: Container node-driver-registrar ready: true, restart count 0 Jan 1 02:36:06.131: INFO: ebs-csi-node-vcprc started at 2023-01-01 02:20:26 +0000 UTC (0+3 container statuses recorded) Jan 1 02:36:06.131: INFO: Container ebs-plugin ready: true, restart count 0 Jan 1 02:36:06.131: INFO: Container liveness-probe ready: true, restart count 0 Jan 1 02:36:06.131: INFO: Container node-driver-registrar ready: true, restart count 0 Jan 1 02:36:06.131: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-49brr started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:36:06.131: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:36:06.131: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-z87gr started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:36:06.131: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:36:06.131: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-wjz96 started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:36:06.131: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:36:06.131: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-w26gj started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:36:06.131: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:36:06.131: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-s4rqf started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:36:06.131: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:36:06.131: INFO: cilium-kk2c2 started at 2023-01-01 02:20:26 +0000 UTC (1+1 container statuses recorded) Jan 1 02:36:06.131: INFO: Init container clean-cilium-state ready: true, restart count 0 Jan 1 02:36:06.131: INFO: Container cilium-agent ready: true, restart count 0 Jan 1 02:36:06.131: INFO: local-injector started at 2023-01-01 02:31:07 +0000 UTC (0+1 container statuses recorded) Jan 1 02:36:06.131: INFO: Container local-injector ready: false, restart count 0 Jan 1 02:36:06.131: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-6vhtn started at 2023-01-01 02:25:17 +0000 UTC (0+1 container statuses recorded) Jan 1 02:36:06.131: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: false, restart count 0 Jan 1 02:36:06.735: INFO: Latency metrics for node ip-172-20-47-178.eu-west-3.compute.internal Jan 1 02:36:06.735: INFO: Logging node info for node ip-172-20-49-224.eu-west-3.compute.internal Jan 1 02:36:06.840: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-49-224.eu-west-3.compute.internal d7d57146-c54a-4155-94f8-0f322216bfcf 10718 0 2023-01-01 02:20:28 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:eu-west-3 failure-domain.beta.kubernetes.io/zone:eu-west-3a kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-49-224.eu-west-3.compute.internal kubernetes.io/os:linux kubernetes.io/role:node node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:eu-west-3a topology.hostpath.csi/node:ip-172-20-49-224.eu-west-3.compute.internal topology.kubernetes.io/region:eu-west-3 topology.kubernetes.io/zone:eu-west-3a] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-089a107939b27c28a"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kops-controller Update v1 2023-01-01 02:20:28 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}} } {kube-controller-manager Update v1 2023-01-01 02:20:28 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.4.0/24\"":{}}}} } {kubelet Update v1 2023-01-01 02:20:28 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {kube-controller-manager Update v1 2023-01-01 02:34:44 +0000 UTC FieldsV1 {"f:status":{"f:volumesAttached":{}}} status} {kubelet Update v1 2023-01-01 02:34:48 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:capacity":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.4.0/24,DoNotUseExternalID:,ProviderID:aws:///eu-west-3a/i-089a107939b27c28a,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.4.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{50478186496 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4078206976 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{45430367772 0} {<nil>} 45430367772 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3973349376 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-01-01 02:34:48 +0000 UTC,LastTransitionTime:2023-01-01 02:20:28 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-01-01 02:34:48 +0000 UTC,LastTransitionTime:2023-01-01 02:20:28 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-01-01 02:34:48 +0000 UTC,LastTransitionTime:2023-01-01 02:20:28 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-01-01 02:34:48 +0000 UTC,LastTransitionTime:2023-01-01 02:20:48 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.49.224,},NodeAddress{Type:ExternalIP,Address:15.237.95.101,},NodeAddress{Type:Hostname,Address:ip-172-20-49-224.eu-west-3.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-49-224.eu-west-3.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-15-237-95-101.eu-west-3.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2d0a1bb33dd26f5a12c3896530131b,SystemUUID:ec2d0a1b-b33d-d26f-5a12-c3896530131b,BootID:9f64e91d-26f0-45aa-aec4-a54433e67034,KernelVersion:4.19.0-23-cloud-amd64,OSImage:Debian GNU/Linux 10 (buster),ContainerRuntimeVersion:docker://20.10.17,KubeletVersion:v1.23.15,KubeProxyVersion:v1.23.15,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:456569528,},ContainerImage{Names:[nginx@sha256:0047b729188a15da49380d9506d65959cce6d40291ccfb4e039f5dc7efd33286 nginx:latest],SizeBytes:141812353,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:7e8bdd271312fd25fc5ff5a8f04727be84044eb3d7d8d03611972a6752e2e11e k8s.gcr.io/e2e-test-images/agnhost:2.39],SizeBytes:126872991,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:1b9d1b2f36cb2dbee1960e82a9344aeb11bd4c4c03abf5e1853e0559c23855e3 k8s.gcr.io/e2e-test-images/httpd:2.4.38-2],SizeBytes:123781643,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.23.15 registry.k8s.io/kube-proxy-amd64:v1.23.15],SizeBytes:112336036,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:10e231cf07c82dfab5141c1afe548dc734e5fc1f67665ec3982a325d1bd31a9a registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.0],SizeBytes:94471559,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:4e74c0492bceddc598de1c90cc5bc14dcda94cb49fa9c5bad9d117c4834b5e08 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.1],SizeBytes:56379269,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:36c31f7e1f433c9634d24f876353e8646246d81a03c4e351202c2644daff1620 k8s.gcr.io/sig-storage/csi-resizer:v1.2.0],SizeBytes:53956308,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:ed98431376c9e944e19a465fe8ea944806714dd95416a0821096c78d66b579bd k8s.gcr.io/sig-storage/csi-snapshotter:v4.1.1],SizeBytes:53531990,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:c5be65d6679efabb969d9b019300d187437ae876f992c40911fd2892bbef3b36 k8s.gcr.io/sig-storage/csi-attacher:v3.2.0],SizeBytes:53500826,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:60ab9b3e6a030d3038c87c0d6bca2930f58d1d72823e6a4af09767dc83b696a2 k8s.gcr.io/sig-storage/csi-attacher:v3.2.1],SizeBytes:53499570,},ContainerImage{Names:[k8s.gcr.io/sig-storage/hostpathplugin@sha256:232fe80174d60d520d36043103853a1d7ab4b7f3782cf43e45034f04ccda58ce k8s.gcr.io/sig-storage/hostpathplugin:v1.7.1],SizeBytes:29977921,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:19582112,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:2dee3fe5fe861bb66c3a4ac51114f3447a4cd35870e0f2e2b558c7a400d89589 k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.2.0],SizeBytes:18699436,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:17611980,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:529be2c9770add0cdd0c989115222ea9fc1be430c11095eb9f6dafcf98a36e2b k8s.gcr.io/sig-storage/livenessprobe:v2.4.0],SizeBytes:17175488,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:13616070e3f29de4417eee434a8ef472221c9e51b3d037b5a6b46cef08eb7443 k8s.gcr.io/e2e-test-images/nginx:1.14-2],SizeBytes:16032814,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf k8s.gcr.io/e2e-test-images/busybox:1.29-2],SizeBytes:1154361,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db k8s.gcr.io/pause:3.6 registry.k8s.io/pause:3.6],SizeBytes:682696,},},VolumesInUse:[kubernetes.io/csi/ebs.csi.aws.com^vol-06c2c39b58fd301cd],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-06c2c39b58fd301cd,DevicePath:,},},Config:nil,},} Jan 1 02:36:06.840: INFO: Logging kubelet events for node ip-172-20-49-224.eu-west-3.compute.internal Jan 1 02:36:06.947: INFO: Logging pods the kubelet thinks is on node ip-172-20-49-224.eu-west-3.compute.internal Jan 1 02:36:07.065: INFO: inline-volume-tester-tf6rn started at 2023-01-01 02:34:41 +0000 UTC (0+1 container statuses recorded) Jan 1 02:36:07.065: INFO: Container csi-volume-tester ready: false, restart count 0 Jan 1 02:36:07.065: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-6n5j9 started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:36:07.065: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:36:07.065: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-h6dxb started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:36:07.065: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:36:07.065: INFO: test-ss-0 started at 2023-01-01 02:27:56 +0000 UTC (0+1 container statuses recorded) Jan 1 02:36:07.065: INFO: Container webserver ready: false, restart count 0 Jan 1 02:36:07.065: INFO: hostexec-ip-172-20-49-224.eu-west-3.compute.internal-hsfbk started at 2023-01-01 02:31:01 +0000 UTC (0+1 container statuses recorded) Jan 1 02:36:07.065: INFO: Container agnhost-container ready: true, restart count 0 Jan 1 02:36:07.065: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-pgc9g started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:36:07.065: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:36:07.065: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-29cdj started at 2023-01-01 02:25:17 +0000 UTC (0+1 container statuses recorded) Jan 1 02:36:07.065: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: false, restart count 0 Jan 1 02:36:07.065: INFO: suspend-false-to-true-g6ml8 started at 2023-01-01 02:31:57 +0000 UTC (0+1 container statuses recorded) Jan 1 02:36:07.065: INFO: Container c ready: false, restart count 0 Jan 1 02:36:07.065: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-7z6bp started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:36:07.065: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:36:07.065: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-lrzjq started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:36:07.065: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:36:07.065: INFO: pod-secrets-6516282b-cc4c-4ad6-9124-72e1a9d3d0dd started at 2023-01-01 02:36:06 +0000 UTC (0+1 container statuses recorded) Jan 1 02:36:07.065: INFO: Container secret-test ready: false, restart count 0 Jan 1 02:36:07.065: INFO: pod-2 started at 2023-01-01 02:31:27 +0000 UTC (0+1 container statuses recorded) Jan 1 02:36:07.065: INFO: Container donothing ready: false, restart count 0 Jan 1 02:36:07.065: INFO: netserver-3 started at 2023-01-01 02:31:32 +0000 UTC (0+1 container statuses recorded) Jan 1 02:36:07.065: INFO: Container webserver ready: false, restart count 0 Jan 1 02:36:07.065: INFO: cilium-78jll started at 2023-01-01 02:20:29 +0000 UTC (1+1 container statuses recorded) Jan 1 02:36:07.065: INFO: Init container clean-cilium-state ready: true, restart count 0 Jan 1 02:36:07.065: INFO: Container cilium-agent ready: true, restart count 0 Jan 1 02:36:07.065: INFO: ebs-csi-node-tzntf started at 2023-01-01 02:20:29 +0000 UTC (0+3 container statuses recorded) Jan 1 02:36:07.065: INFO: Container ebs-plugin ready: true, restart count 0 Jan 1 02:36:07.065: INFO: Container liveness-probe ready: true, restart count 0 Jan 1 02:36:07.065: INFO: Container node-driver-registrar ready: true, restart count 0 Jan 1 02:36:07.065: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-4jf6d started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:36:07.065: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:36:07.065: INFO: dns-test-f6141192-e7ab-4530-bedd-b1f3b48a9237 started at 2023-01-01 02:30:56 +0000 UTC (0+3 container statuses recorded) Jan 1 02:36:07.065: INFO: Container jessie-querier ready: false, restart count 0 Jan 1 02:36:07.065: INFO: Container querier ready: false, restart count 0 Jan 1 02:36:07.065: INFO: Container webserver ready: false, restart count 0 Jan 1 02:36:07.065: INFO: pod-projected-secrets-fedc3ea5-055e-4010-aa5a-0d89b2f8b6ac started at 2023-01-01 02:31:36 +0000 UTC (0+1 container statuses recorded) Jan 1 02:36:07.065: INFO: Container projected-secret-volume-test ready: false, restart count 0 Jan 1 02:36:07.065: INFO: pod-configmaps-f941fae8-7952-461d-b186-629d475d0f88 started at 2023-01-01 02:33:31 +0000 UTC (0+1 container statuses recorded) Jan 1 02:36:07.065: INFO: Container agnhost-container ready: false, restart count 0 Jan 1 02:36:07.065: INFO: netserver-3 started at 2023-01-01 02:33:25 +0000 UTC (0+1 container statuses recorded) Jan 1 02:36:07.065: INFO: Container webserver ready: false, restart count 0 Jan 1 02:36:07.065: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-tv984 started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:36:07.065: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:36:07.065: INFO: affinity-clusterip-timeout-pxjtk started at 2023-01-01 02:27:32 +0000 UTC (0+1 container statuses recorded) Jan 1 02:36:07.065: INFO: Container affinity-clusterip-timeout ready: true, restart count 0 Jan 1 02:36:07.065: INFO: liveness-b61a07c8-7116-4bf3-bf05-31114b2bce91 started at 2023-01-01 02:36:05 +0000 UTC (0+1 container statuses recorded) Jan 1 02:36:07.065: INFO: Container agnhost-container ready: false, restart count 0 Jan 1 02:36:07.065: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-cz8qh started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:36:07.065: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:36:07.065: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-2gndb started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:36:07.065: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:36:07.065: INFO: csi-mockplugin-0 started at 2023-01-01 02:31:26 +0000 UTC (0+3 container statuses recorded) Jan 1 02:36:07.065: INFO: Container csi-provisioner ready: false, restart count 0 Jan 1 02:36:07.065: INFO: Container driver-registrar ready: false, restart count 0 Jan 1 02:36:07.065: INFO: Container mock ready: false, restart count 0 Jan 1 02:36:07.065: INFO: pod-3e35f616-b2af-42fe-8148-bfe7bf04221b started at 2023-01-01 02:31:05 +0000 UTC (0+1 container statuses recorded) Jan 1 02:36:07.065: INFO: Container write-pod ready: false, restart count 0 Jan 1 02:36:07.065: INFO: dns-test-a8277dcf-764d-4d97-a381-813a8d1f903c started at 2023-01-01 02:31:17 +0000 UTC (0+3 container statuses recorded) Jan 1 02:36:07.065: INFO: Container jessie-querier ready: false, restart count 0 Jan 1 02:36:07.065: INFO: Container querier ready: false, restart count 0 Jan 1 02:36:07.065: INFO: Container webserver ready: false, restart count 0 Jan 1 02:36:07.065: INFO: csi-mockplugin-attacher-0 started at 2023-01-01 02:31:26 +0000 UTC (0+1 container statuses recorded) Jan 1 02:36:07.065: INFO: Container csi-attacher ready: true, restart count 0 Jan 1 02:36:07.065: INFO: test-ss-0 started at 2023-01-01 02:31:35 +0000 UTC (0+1 container statuses recorded) Jan 1 02:36:07.066: INFO: Container webserver ready: false, restart count 0 Jan 1 02:36:07.584: INFO: Latency metrics for node ip-172-20-49-224.eu-west-3.compute.internal Jan 1 02:36:07.584: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready �[1mSTEP�[0m: Destroying namespace "services-9461" for this suite. [AfterEach] [sig-network] Services /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/network/service.go:756
Filter through log files
go run hack/e2e.go -v --test --test_args='--ginkgo.focus=Kubernetes\se2e\ssuite\s\[sig\-network\]\sServices\sshould\spreserve\ssource\spod\sIP\sfor\straffic\sthru\sservice\scluster\sIP\s\[LinuxOnly\]$'
/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/network/service.go:927 Jan 1 02:31:27.387: Unexpected error: <*errors.errorString | 0xc00023a2b0>: { s: "timed out waiting for the condition", } timed out waiting for the condition occurred /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/network/service.go:972from junit_15.xml
[BeforeEach] [sig-network] Services /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:185 �[1mSTEP�[0m: Creating a kubernetes client Jan 1 02:26:22.114: INFO: >>> kubeConfig: /root/.kube/config �[1mSTEP�[0m: Building a namespace api object, basename services �[1mSTEP�[0m: Waiting for a default service account to be provisioned in namespace �[1mSTEP�[0m: Waiting for kube-root-ca.crt to be provisioned in namespace [BeforeEach] [sig-network] Services /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/network/service.go:752 [It] should preserve source pod IP for traffic thru service cluster IP [LinuxOnly] /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/network/service.go:927 Jan 1 02:26:23.065: INFO: The status of Pod kube-proxy-mode-detector is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:26:25.170: INFO: The status of Pod kube-proxy-mode-detector is Running (Ready = true) Jan 1 02:26:25.275: INFO: Running '/home/prow/go/src/k8s.io/kops/_rundir/08fd59e0-897a-11ed-b47a-e2b55b2f9198/kubectl --server=https://api.e2e-e2e-kops-grid-cilium-eni-deb10-k23-docker.test-cncf-aws.k8s.io --kubeconfig=/root/.kube/config --namespace=services-5926 exec kube-proxy-mode-detector -- /bin/sh -x -c curl -q -s --connect-timeout 1 http://localhost:10249/proxyMode' Jan 1 02:26:26.533: INFO: rc: 7 Jan 1 02:26:26.643: INFO: Waiting for pod kube-proxy-mode-detector to disappear Jan 1 02:26:26.747: INFO: Pod kube-proxy-mode-detector no longer exists Jan 1 02:26:26.748: INFO: Couldn't detect KubeProxy mode - test failure may be expected: error running /home/prow/go/src/k8s.io/kops/_rundir/08fd59e0-897a-11ed-b47a-e2b55b2f9198/kubectl --server=https://api.e2e-e2e-kops-grid-cilium-eni-deb10-k23-docker.test-cncf-aws.k8s.io --kubeconfig=/root/.kube/config --namespace=services-5926 exec kube-proxy-mode-detector -- /bin/sh -x -c curl -q -s --connect-timeout 1 http://localhost:10249/proxyMode: Command stdout: stderr: + curl -q -s --connect-timeout 1 http://localhost:10249/proxyMode command terminated with exit code 7 error: exit status 7 �[1mSTEP�[0m: creating a TCP service sourceip-test with type=ClusterIP in namespace services-5926 Jan 1 02:26:26.858: INFO: sourceip-test cluster ip: 100.68.136.26 �[1mSTEP�[0m: Picking 2 Nodes to test whether source IP is preserved or not �[1mSTEP�[0m: Creating a webserver pod to be part of the TCP service which echoes back source ip Jan 1 02:26:27.176: INFO: The status of Pod echo-sourceip is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:26:29.281: INFO: The status of Pod echo-sourceip is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:26:31.282: INFO: The status of Pod echo-sourceip is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:26:33.281: INFO: The status of Pod echo-sourceip is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:26:35.281: INFO: The status of Pod echo-sourceip is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:26:37.282: INFO: The status of Pod echo-sourceip is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:26:39.282: INFO: The status of Pod echo-sourceip is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:26:41.281: INFO: The status of Pod echo-sourceip is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:26:43.281: INFO: The status of Pod echo-sourceip is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:26:45.283: INFO: The status of Pod echo-sourceip is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:26:47.282: INFO: The status of Pod echo-sourceip is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:26:49.282: INFO: The status of Pod echo-sourceip is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:26:51.282: INFO: The status of Pod echo-sourceip is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:26:53.282: INFO: The status of Pod echo-sourceip is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:26:55.282: INFO: The status of Pod echo-sourceip is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:26:57.283: INFO: The status of Pod echo-sourceip is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:26:59.281: INFO: The status of Pod echo-sourceip is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:27:01.282: INFO: The status of Pod echo-sourceip is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:27:03.281: INFO: The status of Pod echo-sourceip is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:27:05.281: INFO: The status of Pod echo-sourceip is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:27:07.282: INFO: The status of Pod echo-sourceip is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:27:09.281: INFO: The status of Pod echo-sourceip is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:27:11.284: INFO: The status of Pod echo-sourceip is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:27:13.281: INFO: The status of Pod echo-sourceip is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:27:15.281: INFO: The status of Pod echo-sourceip is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:27:17.282: INFO: The status of Pod echo-sourceip is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:27:19.282: INFO: The status of Pod echo-sourceip is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:27:21.282: INFO: The status of Pod echo-sourceip is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:27:23.281: INFO: The status of Pod echo-sourceip is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:27:25.282: INFO: The status of Pod echo-sourceip is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:27:27.281: INFO: The status of Pod echo-sourceip is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:27:29.282: INFO: The status of Pod echo-sourceip is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:27:31.282: INFO: The status of Pod echo-sourceip is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:27:33.282: INFO: The status of Pod echo-sourceip is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:27:35.287: INFO: The status of Pod echo-sourceip is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:27:37.283: INFO: The status of Pod echo-sourceip is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:27:39.281: INFO: The status of Pod echo-sourceip is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:27:41.281: INFO: The status of Pod echo-sourceip is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:27:43.281: INFO: The status of Pod echo-sourceip is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:27:45.282: INFO: The status of Pod echo-sourceip is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:27:47.283: INFO: The status of Pod echo-sourceip is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:27:49.282: INFO: The status of Pod echo-sourceip is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:27:51.281: INFO: The status of Pod echo-sourceip is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:27:53.281: INFO: The status of Pod echo-sourceip is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:27:55.283: INFO: The status of Pod echo-sourceip is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:27:57.283: INFO: The status of Pod echo-sourceip is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:27:59.281: INFO: The status of Pod echo-sourceip is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:28:01.282: INFO: The status of Pod echo-sourceip is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:28:03.282: INFO: The status of Pod echo-sourceip is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:28:05.283: INFO: The status of Pod echo-sourceip is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:28:07.283: INFO: The status of Pod echo-sourceip is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:28:09.281: INFO: The status of Pod echo-sourceip is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:28:11.282: INFO: The status of Pod echo-sourceip is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:28:13.282: INFO: The status of Pod echo-sourceip is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:28:15.281: INFO: The status of Pod echo-sourceip is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:28:17.281: INFO: The status of Pod echo-sourceip is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:28:19.282: INFO: The status of Pod echo-sourceip is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:28:21.281: INFO: The status of Pod echo-sourceip is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:28:23.281: INFO: The status of Pod echo-sourceip is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:28:25.282: INFO: The status of Pod echo-sourceip is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:28:27.281: INFO: The status of Pod echo-sourceip is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:28:29.282: INFO: The status of Pod echo-sourceip is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:28:31.281: INFO: The status of Pod echo-sourceip is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:28:33.284: INFO: The status of Pod echo-sourceip is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:28:35.282: INFO: The status of Pod echo-sourceip is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:28:37.283: INFO: The status of Pod echo-sourceip is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:28:39.282: INFO: The status of Pod echo-sourceip is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:28:41.282: INFO: The status of Pod echo-sourceip is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:28:43.281: INFO: The status of Pod echo-sourceip is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:28:45.281: INFO: The status of Pod echo-sourceip is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:28:47.282: INFO: The status of Pod echo-sourceip is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:28:49.282: INFO: The status of Pod echo-sourceip is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:28:51.281: INFO: The status of Pod echo-sourceip is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:28:53.282: INFO: The status of Pod echo-sourceip is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:28:55.283: INFO: The status of Pod echo-sourceip is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:28:57.281: INFO: The status of Pod echo-sourceip is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:28:59.281: INFO: The status of Pod echo-sourceip is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:29:01.283: INFO: The status of Pod echo-sourceip is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:29:03.282: INFO: The status of Pod echo-sourceip is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:29:05.283: INFO: The status of Pod echo-sourceip is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:29:07.282: INFO: The status of Pod echo-sourceip is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:29:09.282: INFO: The status of Pod echo-sourceip is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:29:11.282: INFO: The status of Pod echo-sourceip is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:29:13.282: INFO: The status of Pod echo-sourceip is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:29:15.283: INFO: The status of Pod echo-sourceip is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:29:17.282: INFO: The status of Pod echo-sourceip is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:29:19.281: INFO: The status of Pod echo-sourceip is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:29:21.282: INFO: The status of Pod echo-sourceip is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:29:23.282: INFO: The status of Pod echo-sourceip is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:29:25.282: INFO: The status of Pod echo-sourceip is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:29:27.283: INFO: The status of Pod echo-sourceip is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:29:29.281: INFO: The status of Pod echo-sourceip is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:29:31.282: INFO: The status of Pod echo-sourceip is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:29:33.282: INFO: The status of Pod echo-sourceip is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:29:35.282: INFO: The status of Pod echo-sourceip is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:29:37.281: INFO: The status of Pod echo-sourceip is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:29:39.281: INFO: The status of Pod echo-sourceip is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:29:41.281: INFO: The status of Pod echo-sourceip is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:29:43.282: INFO: The status of Pod echo-sourceip is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:29:45.282: INFO: The status of Pod echo-sourceip is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:29:47.282: INFO: The status of Pod echo-sourceip is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:29:49.282: INFO: The status of Pod echo-sourceip is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:29:51.282: INFO: The status of Pod echo-sourceip is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:29:53.282: INFO: The status of Pod echo-sourceip is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:29:55.283: INFO: The status of Pod echo-sourceip is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:29:57.281: INFO: The status of Pod echo-sourceip is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:29:59.282: INFO: The status of Pod echo-sourceip is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:30:01.282: INFO: The status of Pod echo-sourceip is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:30:03.283: INFO: The status of Pod echo-sourceip is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:30:05.282: INFO: The status of Pod echo-sourceip is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:30:07.282: INFO: The status of Pod echo-sourceip is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:30:09.282: INFO: The status of Pod echo-sourceip is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:30:11.282: INFO: The status of Pod echo-sourceip is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:30:13.281: INFO: The status of Pod echo-sourceip is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:30:15.282: INFO: The status of Pod echo-sourceip is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:30:17.282: INFO: The status of Pod echo-sourceip is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:30:19.282: INFO: The status of Pod echo-sourceip is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:30:21.282: INFO: The status of Pod echo-sourceip is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:30:23.281: INFO: The status of Pod echo-sourceip is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:30:25.282: INFO: The status of Pod echo-sourceip is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:30:27.283: INFO: The status of Pod echo-sourceip is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:30:29.282: INFO: The status of Pod echo-sourceip is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:30:31.282: INFO: The status of Pod echo-sourceip is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:30:33.281: INFO: The status of Pod echo-sourceip is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:30:35.283: INFO: The status of Pod echo-sourceip is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:30:37.282: INFO: The status of Pod echo-sourceip is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:30:39.282: INFO: The status of Pod echo-sourceip is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:30:41.281: INFO: The status of Pod echo-sourceip is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:30:43.282: INFO: The status of Pod echo-sourceip is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:30:45.282: INFO: The status of Pod echo-sourceip is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:30:47.285: INFO: The status of Pod echo-sourceip is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:30:49.281: INFO: The status of Pod echo-sourceip is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:30:51.281: INFO: The status of Pod echo-sourceip is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:30:53.282: INFO: The status of Pod echo-sourceip is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:30:55.281: INFO: The status of Pod echo-sourceip is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:30:57.282: INFO: The status of Pod echo-sourceip is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:30:59.281: INFO: The status of Pod echo-sourceip is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:31:01.281: INFO: The status of Pod echo-sourceip is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:31:03.281: INFO: The status of Pod echo-sourceip is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:31:05.283: INFO: The status of Pod echo-sourceip is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:31:07.282: INFO: The status of Pod echo-sourceip is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:31:09.284: INFO: The status of Pod echo-sourceip is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:31:11.282: INFO: The status of Pod echo-sourceip is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:31:13.282: INFO: The status of Pod echo-sourceip is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:31:15.282: INFO: The status of Pod echo-sourceip is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:31:17.282: INFO: The status of Pod echo-sourceip is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:31:19.281: INFO: The status of Pod echo-sourceip is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:31:21.282: INFO: The status of Pod echo-sourceip is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:31:23.281: INFO: The status of Pod echo-sourceip is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:31:25.283: INFO: The status of Pod echo-sourceip is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:31:27.282: INFO: The status of Pod echo-sourceip is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:31:27.387: INFO: The status of Pod echo-sourceip is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:31:27.387: FAIL: Unexpected error: <*errors.errorString | 0xc00023a2b0>: { s: "timed out waiting for the condition", } timed out waiting for the condition occurred Full Stack Trace k8s.io/kubernetes/test/e2e/network.glob..func24.6() /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/network/service.go:972 +0x58a k8s.io/kubernetes/test/e2e.RunE2ETests(0x24c66d7) _output/dockerized/go/src/k8s.io/kubernetes/test/e2e/e2e.go:133 +0x697 k8s.io/kubernetes/test/e2e.TestE2E(0x243a8f9) _output/dockerized/go/src/k8s.io/kubernetes/test/e2e/e2e_test.go:136 +0x19 testing.tRunner(0xc000e16000, 0x735e880) /usr/local/go/src/testing/testing.go:1259 +0x102 created by testing.(*T).Run /usr/local/go/src/testing/testing.go:1306 +0x35a Jan 1 02:31:27.388: INFO: Cleaning up the sourceip test service [AfterEach] [sig-network] Services /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:186 �[1mSTEP�[0m: Collecting events from namespace "services-5926". �[1mSTEP�[0m: Found 17 events. Jan 1 02:31:27.613: INFO: At 2023-01-01 02:26:22 +0000 UTC - event for kube-proxy-mode-detector: {default-scheduler } Scheduled: Successfully assigned services-5926/kube-proxy-mode-detector to ip-172-20-49-224.eu-west-3.compute.internal Jan 1 02:31:27.613: INFO: At 2023-01-01 02:26:23 +0000 UTC - event for kube-proxy-mode-detector: {kubelet ip-172-20-49-224.eu-west-3.compute.internal} Created: Created container agnhost-container Jan 1 02:31:27.613: INFO: At 2023-01-01 02:26:23 +0000 UTC - event for kube-proxy-mode-detector: {kubelet ip-172-20-49-224.eu-west-3.compute.internal} Pulled: Container image "k8s.gcr.io/e2e-test-images/agnhost:2.39" already present on machine Jan 1 02:31:27.613: INFO: At 2023-01-01 02:26:24 +0000 UTC - event for kube-proxy-mode-detector: {kubelet ip-172-20-49-224.eu-west-3.compute.internal} Started: Started container agnhost-container Jan 1 02:31:27.613: INFO: At 2023-01-01 02:26:27 +0000 UTC - event for echo-sourceip: {default-scheduler } Scheduled: Successfully assigned services-5926/echo-sourceip to ip-172-20-49-224.eu-west-3.compute.internal Jan 1 02:31:27.613: INFO: At 2023-01-01 02:26:27 +0000 UTC - event for kube-proxy-mode-detector: {kubelet ip-172-20-49-224.eu-west-3.compute.internal} Killing: Stopping container agnhost-container Jan 1 02:31:27.613: INFO: At 2023-01-01 02:26:28 +0000 UTC - event for echo-sourceip: {kubelet ip-172-20-49-224.eu-west-3.compute.internal} SandboxChanged: Pod sandbox changed, it will be killed and re-created. Jan 1 02:31:27.613: INFO: At 2023-01-01 02:26:28 +0000 UTC - event for echo-sourceip: {kubelet ip-172-20-49-224.eu-west-3.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "58d6d244232b339b812f4fc3d3a270cb3f956aac411f951638daab9f94de6b62" network for pod "echo-sourceip": networkPlugin cni failed to set up pod "echo-sourceip_services-5926" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 1 02:31:27.613: INFO: At 2023-01-01 02:26:29 +0000 UTC - event for echo-sourceip: {kubelet ip-172-20-49-224.eu-west-3.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "68521a9ce5747df5984d8bc596bc8a9797fe8b076eb85be62c4ea1f8a98c9629" network for pod "echo-sourceip": networkPlugin cni failed to set up pod "echo-sourceip_services-5926" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 1 02:31:27.613: INFO: At 2023-01-01 02:26:32 +0000 UTC - event for echo-sourceip: {kubelet ip-172-20-49-224.eu-west-3.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "277251aaffc2ba52d8d3e7e28c559cfd28b91b6903edef9f37913d554f683b33" network for pod "echo-sourceip": networkPlugin cni failed to set up pod "echo-sourceip_services-5926" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 1 02:31:27.613: INFO: At 2023-01-01 02:26:35 +0000 UTC - event for echo-sourceip: {kubelet ip-172-20-49-224.eu-west-3.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "76a32fb8c265b0ace507c6db437251267c376fed64a9b63943d2bf25b8178fbe" network for pod "echo-sourceip": networkPlugin cni failed to set up pod "echo-sourceip_services-5926" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 1 02:31:27.613: INFO: At 2023-01-01 02:26:36 +0000 UTC - event for echo-sourceip: {kubelet ip-172-20-49-224.eu-west-3.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "f168796a12b9cca8ef7d834aa24989f7617ea1483eaba70c1d5c288296a0a06a" network for pod "echo-sourceip": networkPlugin cni failed to set up pod "echo-sourceip_services-5926" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 1 02:31:27.613: INFO: At 2023-01-01 02:26:38 +0000 UTC - event for echo-sourceip: {kubelet ip-172-20-49-224.eu-west-3.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "a5d0da65ed97ade9dd4f42d53f30bb434d3312a7b0a17f824f4e869c41e4dd44" network for pod "echo-sourceip": networkPlugin cni failed to set up pod "echo-sourceip_services-5926" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 1 02:31:27.613: INFO: At 2023-01-01 02:26:40 +0000 UTC - event for echo-sourceip: {kubelet ip-172-20-49-224.eu-west-3.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "3473852615d1d58554a42fc5d6e02adb1c5772a38981f2584b91ae5c664c35d7" network for pod "echo-sourceip": networkPlugin cni failed to set up pod "echo-sourceip_services-5926" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 1 02:31:27.613: INFO: At 2023-01-01 02:26:41 +0000 UTC - event for echo-sourceip: {kubelet ip-172-20-49-224.eu-west-3.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "106c003e7aea5d793d9168effca67aee6636c8178b73e45615e427a5c7368d2b" network for pod "echo-sourceip": networkPlugin cni failed to set up pod "echo-sourceip_services-5926" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 1 02:31:27.613: INFO: At 2023-01-01 02:26:43 +0000 UTC - event for echo-sourceip: {kubelet ip-172-20-49-224.eu-west-3.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "1ea731ab6737e36b23c64274ca68908dc021bd89d0e885114c33664c8e1354e9" network for pod "echo-sourceip": networkPlugin cni failed to set up pod "echo-sourceip_services-5926" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 1 02:31:27.613: INFO: At 2023-01-01 02:26:45 +0000 UTC - event for echo-sourceip: {kubelet ip-172-20-49-224.eu-west-3.compute.internal} FailedCreatePodSandBox: (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "b321a9a5985f875507c8696d0ee80388b5ee302d066e2489f650e640921526fa" network for pod "echo-sourceip": networkPlugin cni failed to set up pod "echo-sourceip_services-5926" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 1 02:31:27.718: INFO: POD NODE PHASE GRACE CONDITIONS Jan 1 02:31:27.718: INFO: echo-sourceip ip-172-20-49-224.eu-west-3.compute.internal Pending [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2023-01-01 02:26:27 +0000 UTC } {Ready False 0001-01-01 00:00:00 +0000 UTC 2023-01-01 02:26:27 +0000 UTC ContainersNotReady containers with unready status: [agnhost-container]} {ContainersReady False 0001-01-01 00:00:00 +0000 UTC 2023-01-01 02:26:27 +0000 UTC ContainersNotReady containers with unready status: [agnhost-container]} {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2023-01-01 02:26:27 +0000 UTC }] Jan 1 02:31:27.718: INFO: Jan 1 02:31:27.824: INFO: Unable to fetch services-5926/echo-sourceip/agnhost-container logs: the server rejected our request for an unknown reason (get pods echo-sourceip) Jan 1 02:31:27.929: INFO: Logging node info for node ip-172-20-33-66.eu-west-3.compute.internal Jan 1 02:31:28.034: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-33-66.eu-west-3.compute.internal ec4b59b3-62db-4040-aad2-4c3249f80fe8 9096 0 2023-01-01 02:20:25 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:eu-west-3 failure-domain.beta.kubernetes.io/zone:eu-west-3a io.kubernetes.storage.mock/node:some-mock-node kubelet_cleanup:true kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-33-66.eu-west-3.compute.internal kubernetes.io/os:linux kubernetes.io/role:node node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:eu-west-3a topology.kubernetes.io/region:eu-west-3 topology.kubernetes.io/zone:eu-west-3a] map[csi.volume.kubernetes.io/nodeid:{"csi-mock-csi-mock-volumes-8129":"csi-mock-csi-mock-volumes-8129","ebs.csi.aws.com":"i-028313673124ac4ad"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kops-controller Update v1 2023-01-01 02:20:25 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}} } {kube-controller-manager Update v1 2023-01-01 02:20:25 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.1.0/24\"":{}}}} } {kubelet Update v1 2023-01-01 02:20:25 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {e2e.test Update v1 2023-01-01 02:25:15 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubelet_cleanup":{}}}} } {kubelet Update v1 2023-01-01 02:31:10 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:io.kubernetes.storage.mock/node":{},"f:topology.ebs.csi.aws.com/zone":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.1.0/24,DoNotUseExternalID:,ProviderID:aws:///eu-west-3a/i-028313673124ac4ad,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.1.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{50478186496 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4078190592 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{45430367772 0} {<nil>} 45430367772 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3973332992 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-01-01 02:30:38 +0000 UTC,LastTransitionTime:2023-01-01 02:20:25 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-01-01 02:30:38 +0000 UTC,LastTransitionTime:2023-01-01 02:20:25 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-01-01 02:30:38 +0000 UTC,LastTransitionTime:2023-01-01 02:20:25 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-01-01 02:30:38 +0000 UTC,LastTransitionTime:2023-01-01 02:20:46 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.33.66,},NodeAddress{Type:ExternalIP,Address:52.47.140.112,},NodeAddress{Type:Hostname,Address:ip-172-20-33-66.eu-west-3.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-33-66.eu-west-3.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-52-47-140-112.eu-west-3.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2ad2ce1a87d508c178acf6215c5767,SystemUUID:ec2ad2ce-1a87-d508-c178-acf6215c5767,BootID:b3b60783-d786-412b-a36a-802bcfae604d,KernelVersion:4.19.0-23-cloud-amd64,OSImage:Debian GNU/Linux 10 (buster),ContainerRuntimeVersion:docker://20.10.17,KubeletVersion:v1.23.15,KubeProxyVersion:v1.23.15,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:456569528,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/jessie-dnsutils@sha256:11e6a66017ba4e4b938c1612b7a54a3befcefd354796c04e1dba76873a13518e k8s.gcr.io/e2e-test-images/jessie-dnsutils:1.5],SizeBytes:253371792,},ContainerImage{Names:[nginx@sha256:0047b729188a15da49380d9506d65959cce6d40291ccfb4e039f5dc7efd33286 nginx:latest],SizeBytes:141812353,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:7e8bdd271312fd25fc5ff5a8f04727be84044eb3d7d8d03611972a6752e2e11e k8s.gcr.io/e2e-test-images/agnhost:2.39],SizeBytes:126872991,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:1b9d1b2f36cb2dbee1960e82a9344aeb11bd4c4c03abf5e1853e0559c23855e3 k8s.gcr.io/e2e-test-images/httpd:2.4.38-2],SizeBytes:123781643,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.23.15 registry.k8s.io/kube-proxy-amd64:v1.23.15],SizeBytes:112336036,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:10e231cf07c82dfab5141c1afe548dc734e5fc1f67665ec3982a325d1bd31a9a registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.0],SizeBytes:94471559,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:c8e03f60afa90a28e4bb6ec9a8d0fc36d89de4b7475cf2d613afa793ec969fe0 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.0],SizeBytes:56382329,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:c5be65d6679efabb969d9b019300d187437ae876f992c40911fd2892bbef3b36 k8s.gcr.io/sig-storage/csi-attacher:v3.2.0],SizeBytes:53500826,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:017727efcfeb7d053af68e51436ce8e65edbc6ca573720afb4f79c8594036955 registry.k8s.io/coredns/coredns:v1.10.0],SizeBytes:50249337,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:a61d309da54641db41fb8f35718f744e9f730d4d0384f8c4b186ddc9f06cbd5f k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.1.0],SizeBytes:19662887,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:19582112,},ContainerImage{Names:[k8s.gcr.io/sig-storage/mock-driver@sha256:a7b517f9e0f42ffade802eef9cefa271372386b85f55b702b493241e58459793 k8s.gcr.io/sig-storage/mock-driver:v4.1.0],SizeBytes:17680993,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:17611980,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:13616070e3f29de4417eee434a8ef472221c9e51b3d037b5a6b46cef08eb7443 k8s.gcr.io/e2e-test-images/nginx:1.14-2],SizeBytes:16032814,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nonewprivs@sha256:8ac1264691820febacf3aea5d152cbde6d10685731ec14966a9401c6f47a68ac k8s.gcr.io/e2e-test-images/nonewprivs:1.3],SizeBytes:7107254,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf k8s.gcr.io/e2e-test-images/busybox:1.29-2],SizeBytes:1154361,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db k8s.gcr.io/pause:3.6 registry.k8s.io/pause:3.6],SizeBytes:682696,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Jan 1 02:31:28.034: INFO: Logging kubelet events for node ip-172-20-33-66.eu-west-3.compute.internal Jan 1 02:31:28.141: INFO: Logging pods the kubelet thinks is on node ip-172-20-33-66.eu-west-3.compute.internal Jan 1 02:31:28.256: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-vnv2x started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:28.256: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:31:28.256: INFO: csi-mockplugin-0 started at 2023-01-01 02:26:15 +0000 UTC (0+4 container statuses recorded) Jan 1 02:31:28.256: INFO: Container busybox ready: true, restart count 0 Jan 1 02:31:28.256: INFO: Container csi-provisioner ready: true, restart count 0 Jan 1 02:31:28.256: INFO: Container driver-registrar ready: true, restart count 0 Jan 1 02:31:28.256: INFO: Container mock ready: true, restart count 0 Jan 1 02:31:28.256: INFO: hostexec-ip-172-20-33-66.eu-west-3.compute.internal-qm7jt started at 2023-01-01 02:27:26 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:28.256: INFO: Container agnhost-container ready: true, restart count 0 Jan 1 02:31:28.256: INFO: local-client started at 2023-01-01 02:31:07 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:28.256: INFO: Container local-client ready: false, restart count 0 Jan 1 02:31:28.256: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-csnzw started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:28.256: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: false, restart count 0 Jan 1 02:31:28.256: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-h4nvd started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:28.256: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:31:28.256: INFO: ebs-csi-node-vsxjc started at 2023-01-01 02:20:26 +0000 UTC (0+3 container statuses recorded) Jan 1 02:31:28.256: INFO: Container ebs-plugin ready: true, restart count 0 Jan 1 02:31:28.256: INFO: Container liveness-probe ready: true, restart count 0 Jan 1 02:31:28.256: INFO: Container node-driver-registrar ready: true, restart count 0 Jan 1 02:31:28.256: INFO: affinity-clusterip-timeout-q9qt2 started at 2023-01-01 02:27:32 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:28.256: INFO: Container affinity-clusterip-timeout ready: false, restart count 0 Jan 1 02:31:28.256: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-m6vq2 started at 2023-01-01 02:25:17 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:28.256: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:31:28.256: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-h6mmt started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:28.256: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:31:28.256: INFO: cilium-vbcgc started at 2023-01-01 02:20:26 +0000 UTC (1+1 container statuses recorded) Jan 1 02:31:28.256: INFO: Init container clean-cilium-state ready: true, restart count 0 Jan 1 02:31:28.256: INFO: Container cilium-agent ready: true, restart count 0 Jan 1 02:31:28.256: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-dkfpd started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:28.256: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:31:28.256: INFO: csi-mockplugin-attacher-0 started at 2023-01-01 02:27:56 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:28.256: INFO: Container csi-attacher ready: false, restart count 0 Jan 1 02:31:28.256: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-q8pgp started at 2023-01-01 02:25:17 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:28.256: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:31:28.256: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-546bc started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:28.256: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:31:28.256: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-2v87j started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:28.256: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:31:28.256: INFO: csi-mockplugin-0 started at 2023-01-01 02:27:56 +0000 UTC (0+3 container statuses recorded) Jan 1 02:31:28.256: INFO: Container csi-provisioner ready: false, restart count 0 Jan 1 02:31:28.256: INFO: Container driver-registrar ready: false, restart count 0 Jan 1 02:31:28.256: INFO: Container mock ready: false, restart count 0 Jan 1 02:31:28.256: INFO: netserver-0 started at 2023-01-01 02:25:27 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:28.256: INFO: Container webserver ready: true, restart count 0 Jan 1 02:31:28.256: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-l9hnv started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:28.256: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:31:28.256: INFO: coredns-867df8f45c-cqmp7 started at 2023-01-01 02:21:18 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:28.256: INFO: Container coredns ready: true, restart count 0 Jan 1 02:31:28.690: INFO: Latency metrics for node ip-172-20-33-66.eu-west-3.compute.internal Jan 1 02:31:28.691: INFO: Logging node info for node ip-172-20-38-238.eu-west-3.compute.internal Jan 1 02:31:28.795: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-38-238.eu-west-3.compute.internal 0eb713b4-64bf-47b7-8090-bab8111218ee 9308 0 2023-01-01 02:18:51 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:c5.large beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:eu-west-3 failure-domain.beta.kubernetes.io/zone:eu-west-3a kops.k8s.io/kops-controller-pki: kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-38-238.eu-west-3.compute.internal kubernetes.io/os:linux kubernetes.io/role:master node-role.kubernetes.io/control-plane: node-role.kubernetes.io/master: node.kubernetes.io/exclude-from-external-load-balancers: node.kubernetes.io/instance-type:c5.large topology.ebs.csi.aws.com/zone:eu-west-3a topology.kubernetes.io/region:eu-west-3 topology.kubernetes.io/zone:eu-west-3a] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-0f70ebb30a9076648"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kubelet Update v1 2023-01-01 02:18:50 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {protokube Update v1 2023-01-01 02:19:16 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kops.k8s.io/kops-controller-pki":{},"f:node-role.kubernetes.io/control-plane":{},"f:node.kubernetes.io/exclude-from-external-load-balancers":{}}}} } {kops-controller Update v1 2023-01-01 02:19:21 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/master":{}}}} } {kube-controller-manager Update v1 2023-01-01 02:19:31 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.0.0/24\"":{}},"f:taints":{}}} } {kubelet Update v1 2023-01-01 02:20:36 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.0.0/24,DoNotUseExternalID:,ProviderID:aws:///eu-west-3a/i-0f70ebb30a9076648,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/master,Value:,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[100.96.0.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{50478186496 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3918811136 0} {<nil>} 3826964Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{45430367772 0} {<nil>} 45430367772 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3813953536 0} {<nil>} 3724564Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-01-01 02:31:26 +0000 UTC,LastTransitionTime:2023-01-01 02:18:45 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-01-01 02:31:26 +0000 UTC,LastTransitionTime:2023-01-01 02:18:45 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-01-01 02:31:26 +0000 UTC,LastTransitionTime:2023-01-01 02:18:45 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-01-01 02:31:26 +0000 UTC,LastTransitionTime:2023-01-01 02:19:31 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.38.238,},NodeAddress{Type:ExternalIP,Address:15.237.26.58,},NodeAddress{Type:Hostname,Address:ip-172-20-38-238.eu-west-3.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-38-238.eu-west-3.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-15-237-26-58.eu-west-3.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2e6c64b5b811bf0c78b2aa7ccdb943,SystemUUID:ec2e6c64-b5b8-11bf-0c78-b2aa7ccdb943,BootID:9db01fa3-d136-4230-b21a-b03abbfb6bff,KernelVersion:4.19.0-23-cloud-amd64,OSImage:Debian GNU/Linux 10 (buster),ContainerRuntimeVersion:docker://20.10.17,KubeletVersion:v1.23.15,KubeProxyVersion:v1.23.15,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[registry.k8s.io/etcdadm/etcd-manager@sha256:66a453db625abb268f4b3bbefc5a34a171d81e6e8796cecca54cfd71775c77c4 registry.k8s.io/etcdadm/etcd-manager:v3.0.20221209],SizeBytes:662209237,},ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:456569528,},ContainerImage{Names:[k8s.gcr.io/kube-apiserver-amd64:v1.23.15 registry.k8s.io/kube-apiserver-amd64:v1.23.15],SizeBytes:135240444,},ContainerImage{Names:[k8s.gcr.io/kube-controller-manager-amd64:v1.23.15 registry.k8s.io/kube-controller-manager-amd64:v1.23.15],SizeBytes:125046097,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.23.15 registry.k8s.io/kube-proxy-amd64:v1.23.15],SizeBytes:112336036,},ContainerImage{Names:[quay.io/cilium/operator@sha256:a6d24a006a6b92967ac90786b49bc1ac26e5477cf028cd1186efcfc2466484db quay.io/cilium/operator:v1.12.5],SizeBytes:110016956,},ContainerImage{Names:[registry.k8s.io/kops/kops-controller:1.27.0-alpha.1],SizeBytes:105197757,},ContainerImage{Names:[registry.k8s.io/kops/dns-controller:1.27.0-alpha.1],SizeBytes:102577531,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:10e231cf07c82dfab5141c1afe548dc734e5fc1f67665ec3982a325d1bd31a9a registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.0],SizeBytes:94471559,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:122bfb8c1edabb3c0edd63f06523e6940d958d19b3957dc7b1d6f81e9f1f6119 registry.k8s.io/sig-storage/csi-provisioner:v3.1.0],SizeBytes:57736070,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:9ebbf9f023e7b41ccee3d52afe39a89e3ddacdbb69269d583abfc25847cfd9e4 registry.k8s.io/sig-storage/csi-resizer:v1.4.0],SizeBytes:55464778,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:8b9c313c05f54fb04f8d430896f5f5904b6cb157df261501b29adc04d2b2dc7b registry.k8s.io/sig-storage/csi-attacher:v3.4.0],SizeBytes:54848595,},ContainerImage{Names:[k8s.gcr.io/kube-scheduler-amd64:v1.23.15 registry.k8s.io/kube-scheduler-amd64:v1.23.15],SizeBytes:53521430,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:19582112,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:17611980,},ContainerImage{Names:[registry.k8s.io/kops/kube-apiserver-healthcheck:1.27.0-alpha.1],SizeBytes:8787111,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:682696,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Jan 1 02:31:28.796: INFO: Logging kubelet events for node ip-172-20-38-238.eu-west-3.compute.internal Jan 1 02:31:28.908: INFO: Logging pods the kubelet thinks is on node ip-172-20-38-238.eu-west-3.compute.internal Jan 1 02:31:29.021: INFO: etcd-manager-events-ip-172-20-38-238.eu-west-3.compute.internal started at 2023-01-01 02:18:11 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:29.021: INFO: Container etcd-manager ready: true, restart count 0 Jan 1 02:31:29.021: INFO: etcd-manager-main-ip-172-20-38-238.eu-west-3.compute.internal started at 2023-01-01 02:18:11 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:29.021: INFO: Container etcd-manager ready: true, restart count 0 Jan 1 02:31:29.021: INFO: cilium-qk59v started at 2023-01-01 02:19:20 +0000 UTC (1+1 container statuses recorded) Jan 1 02:31:29.021: INFO: Init container clean-cilium-state ready: true, restart count 0 Jan 1 02:31:29.021: INFO: Container cilium-agent ready: true, restart count 0 Jan 1 02:31:29.021: INFO: ebs-csi-node-k87k6 started at 2023-01-01 02:19:20 +0000 UTC (0+3 container statuses recorded) Jan 1 02:31:29.021: INFO: Container ebs-plugin ready: true, restart count 0 Jan 1 02:31:29.021: INFO: Container liveness-probe ready: true, restart count 0 Jan 1 02:31:29.021: INFO: Container node-driver-registrar ready: true, restart count 0 Jan 1 02:31:29.021: INFO: dns-controller-7db8b74b99-58zzw started at 2023-01-01 02:19:20 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:29.021: INFO: Container dns-controller ready: true, restart count 0 Jan 1 02:31:29.021: INFO: cilium-operator-d47487c76-j84nq started at 2023-01-01 02:19:20 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:29.021: INFO: Container cilium-operator ready: true, restart count 0 Jan 1 02:31:29.021: INFO: kube-controller-manager-ip-172-20-38-238.eu-west-3.compute.internal started at 2023-01-01 02:18:11 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:29.021: INFO: Container kube-controller-manager ready: true, restart count 2 Jan 1 02:31:29.021: INFO: kube-scheduler-ip-172-20-38-238.eu-west-3.compute.internal started at 2023-01-01 02:18:11 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:29.021: INFO: Container kube-scheduler ready: true, restart count 0 Jan 1 02:31:29.021: INFO: kube-apiserver-ip-172-20-38-238.eu-west-3.compute.internal started at 2023-01-01 02:18:11 +0000 UTC (0+2 container statuses recorded) Jan 1 02:31:29.021: INFO: Container healthcheck ready: true, restart count 0 Jan 1 02:31:29.021: INFO: Container kube-apiserver ready: true, restart count 1 Jan 1 02:31:29.021: INFO: kops-controller-26gzg started at 2023-01-01 02:19:20 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:29.021: INFO: Container kops-controller ready: true, restart count 0 Jan 1 02:31:29.021: INFO: ebs-csi-controller-6dc545b75-5qqh8 started at 2023-01-01 02:19:20 +0000 UTC (0+5 container statuses recorded) Jan 1 02:31:29.021: INFO: Container csi-attacher ready: true, restart count 0 Jan 1 02:31:29.021: INFO: Container csi-provisioner ready: true, restart count 0 Jan 1 02:31:29.021: INFO: Container csi-resizer ready: true, restart count 0 Jan 1 02:31:29.021: INFO: Container ebs-plugin ready: true, restart count 0 Jan 1 02:31:29.021: INFO: Container liveness-probe ready: true, restart count 0 Jan 1 02:31:29.354: INFO: Latency metrics for node ip-172-20-38-238.eu-west-3.compute.internal Jan 1 02:31:29.354: INFO: Logging node info for node ip-172-20-45-112.eu-west-3.compute.internal Jan 1 02:31:29.459: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-45-112.eu-west-3.compute.internal 48eed937-05bb-4d8a-bddd-aa94a2e6ab92 9109 0 2023-01-01 02:20:27 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:eu-west-3 failure-domain.beta.kubernetes.io/zone:eu-west-3a kubelet_cleanup:true kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-45-112.eu-west-3.compute.internal kubernetes.io/os:linux kubernetes.io/role:node node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:eu-west-3a topology.hostpath.csi/node:ip-172-20-45-112.eu-west-3.compute.internal topology.kubernetes.io/region:eu-west-3 topology.kubernetes.io/zone:eu-west-3a] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-0778adf2085fa7f34"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kube-controller-manager Update v1 2023-01-01 02:20:27 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.3.0/24\"":{}}}} } {kubelet Update v1 2023-01-01 02:20:27 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {kops-controller Update v1 2023-01-01 02:20:28 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}} } {e2e.test Update v1 2023-01-01 02:25:15 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubelet_cleanup":{}}}} } {kube-controller-manager Update v1 2023-01-01 02:30:49 +0000 UTC FieldsV1 {"f:status":{"f:volumesAttached":{}}} status} {kubelet Update v1 2023-01-01 02:30:52 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:capacity":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.3.0/24,DoNotUseExternalID:,ProviderID:aws:///eu-west-3a/i-0778adf2085fa7f34,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.3.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{50478186496 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4078190592 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{45430367772 0} {<nil>} 45430367772 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3973332992 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-01-01 02:31:12 +0000 UTC,LastTransitionTime:2023-01-01 02:20:27 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-01-01 02:31:12 +0000 UTC,LastTransitionTime:2023-01-01 02:20:27 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-01-01 02:31:12 +0000 UTC,LastTransitionTime:2023-01-01 02:20:27 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-01-01 02:31:12 +0000 UTC,LastTransitionTime:2023-01-01 02:20:48 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.45.112,},NodeAddress{Type:ExternalIP,Address:13.38.32.14,},NodeAddress{Type:Hostname,Address:ip-172-20-45-112.eu-west-3.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-45-112.eu-west-3.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-13-38-32-14.eu-west-3.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2dbdf846e8ff614f4555e5bf9dbbf4,SystemUUID:ec2dbdf8-46e8-ff61-4f45-55e5bf9dbbf4,BootID:10a054ae-621a-47ce-9566-b0048cc5101b,KernelVersion:4.19.0-23-cloud-amd64,OSImage:Debian GNU/Linux 10 (buster),ContainerRuntimeVersion:docker://20.10.17,KubeletVersion:v1.23.15,KubeProxyVersion:v1.23.15,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:456569528,},ContainerImage{Names:[nginx@sha256:0047b729188a15da49380d9506d65959cce6d40291ccfb4e039f5dc7efd33286 nginx:latest],SizeBytes:141812353,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:7e8bdd271312fd25fc5ff5a8f04727be84044eb3d7d8d03611972a6752e2e11e k8s.gcr.io/e2e-test-images/agnhost:2.39],SizeBytes:126872991,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:1b9d1b2f36cb2dbee1960e82a9344aeb11bd4c4c03abf5e1853e0559c23855e3 k8s.gcr.io/e2e-test-images/httpd:2.4.38-2],SizeBytes:123781643,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.23.15 registry.k8s.io/kube-proxy-amd64:v1.23.15],SizeBytes:112336036,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:10e231cf07c82dfab5141c1afe548dc734e5fc1f67665ec3982a325d1bd31a9a registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.0],SizeBytes:94471559,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:c8e03f60afa90a28e4bb6ec9a8d0fc36d89de4b7475cf2d613afa793ec969fe0 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.0],SizeBytes:56382329,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:4e74c0492bceddc598de1c90cc5bc14dcda94cb49fa9c5bad9d117c4834b5e08 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.1],SizeBytes:56379269,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:36c31f7e1f433c9634d24f876353e8646246d81a03c4e351202c2644daff1620 k8s.gcr.io/sig-storage/csi-resizer:v1.2.0],SizeBytes:53956308,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:ed98431376c9e944e19a465fe8ea944806714dd95416a0821096c78d66b579bd k8s.gcr.io/sig-storage/csi-snapshotter:v4.1.1],SizeBytes:53531990,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:c5be65d6679efabb969d9b019300d187437ae876f992c40911fd2892bbef3b36 k8s.gcr.io/sig-storage/csi-attacher:v3.2.0],SizeBytes:53500826,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:60ab9b3e6a030d3038c87c0d6bca2930f58d1d72823e6a4af09767dc83b696a2 k8s.gcr.io/sig-storage/csi-attacher:v3.2.1],SizeBytes:53499570,},ContainerImage{Names:[k8s.gcr.io/sig-storage/hostpathplugin@sha256:232fe80174d60d520d36043103853a1d7ab4b7f3782cf43e45034f04ccda58ce k8s.gcr.io/sig-storage/hostpathplugin:v1.7.1],SizeBytes:29977921,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:a61d309da54641db41fb8f35718f744e9f730d4d0384f8c4b186ddc9f06cbd5f k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.1.0],SizeBytes:19662887,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:19582112,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:2dee3fe5fe861bb66c3a4ac51114f3447a4cd35870e0f2e2b558c7a400d89589 k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.2.0],SizeBytes:18699436,},ContainerImage{Names:[k8s.gcr.io/sig-storage/mock-driver@sha256:a7b517f9e0f42ffade802eef9cefa271372386b85f55b702b493241e58459793 k8s.gcr.io/sig-storage/mock-driver:v4.1.0],SizeBytes:17680993,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:17611980,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:529be2c9770add0cdd0c989115222ea9fc1be430c11095eb9f6dafcf98a36e2b k8s.gcr.io/sig-storage/livenessprobe:v2.4.0],SizeBytes:17175488,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:13616070e3f29de4417eee434a8ef472221c9e51b3d037b5a6b46cef08eb7443 k8s.gcr.io/e2e-test-images/nginx:1.14-2],SizeBytes:16032814,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf k8s.gcr.io/e2e-test-images/busybox:1.29-2],SizeBytes:1154361,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:39e1e963e5310e9c313bad51523be012ede7b35bb9316517d19089a010356592 k8s.gcr.io/e2e-test-images/busybox:1.29-1],SizeBytes:1154361,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db k8s.gcr.io/pause:3.6 registry.k8s.io/pause:3.6],SizeBytes:682696,},},VolumesInUse:[kubernetes.io/csi/ebs.csi.aws.com^vol-03d4513454f149737 kubernetes.io/csi/ebs.csi.aws.com^vol-08601772399ba2131],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-03d4513454f149737,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-08601772399ba2131,DevicePath:,},},Config:nil,},} Jan 1 02:31:29.459: INFO: Logging kubelet events for node ip-172-20-45-112.eu-west-3.compute.internal Jan 1 02:31:29.566: INFO: Logging pods the kubelet thinks is on node ip-172-20-45-112.eu-west-3.compute.internal Jan 1 02:31:29.681: INFO: pod-1 started at 2023-01-01 02:31:26 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:29.681: INFO: Container donothing ready: false, restart count 0 Jan 1 02:31:29.681: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-jh8bc started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:29.681: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:31:29.681: INFO: cilium-slx98 started at 2023-01-01 02:20:28 +0000 UTC (1+1 container statuses recorded) Jan 1 02:31:29.681: INFO: Init container clean-cilium-state ready: true, restart count 0 Jan 1 02:31:29.681: INFO: Container cilium-agent ready: true, restart count 0 Jan 1 02:31:29.681: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-4lrj2 started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:29.681: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:31:29.681: INFO: affinity-clusterip-timeout-lv5vx started at 2023-01-01 02:27:32 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:29.681: INFO: Container affinity-clusterip-timeout ready: true, restart count 0 Jan 1 02:31:29.681: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-srf5k started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:29.681: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:31:29.681: INFO: pod-0 started at 2023-01-01 02:31:26 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:29.681: INFO: Container donothing ready: false, restart count 0 Jan 1 02:31:29.681: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-4mml9 started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:29.681: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:31:29.681: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-x5cp6 started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:29.681: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:31:29.681: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-xnfzj started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:29.681: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:31:29.681: INFO: pod-d05d599f-1b25-4249-b9c7-c64a79abebd3 started at 2023-01-01 02:30:59 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:29.681: INFO: Container write-pod ready: false, restart count 0 Jan 1 02:31:29.681: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-gph2d started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:29.681: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:31:29.681: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-59bdj started at 2023-01-01 02:25:17 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:29.681: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:31:29.681: INFO: ebs-csi-node-7clrr started at 2023-01-01 02:20:28 +0000 UTC (0+3 container statuses recorded) Jan 1 02:31:29.681: INFO: Container ebs-plugin ready: true, restart count 0 Jan 1 02:31:29.681: INFO: Container liveness-probe ready: true, restart count 0 Jan 1 02:31:29.681: INFO: Container node-driver-registrar ready: true, restart count 0 Jan 1 02:31:29.681: INFO: image-pull-tested8f64cd-03c5-4c0a-9ed6-c4e25c9622f4 started at 2023-01-01 02:31:23 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:29.681: INFO: Container image-pull-test ready: false, restart count 0 Jan 1 02:31:29.681: INFO: netserver-1 started at 2023-01-01 02:25:27 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:29.681: INFO: Container webserver ready: true, restart count 0 Jan 1 02:31:29.681: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-rmdrr started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:29.681: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:31:29.681: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-kjg2m started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:29.681: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:31:29.681: INFO: e2e-dns-utils started at 2023-01-01 02:31:03 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:29.681: INFO: Container agnhost-container ready: false, restart count 0 Jan 1 02:31:29.681: INFO: inline-volume-tester-fpr45 started at 2023-01-01 02:31:04 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:29.681: INFO: Container csi-volume-tester ready: false, restart count 0 Jan 1 02:31:30.358: INFO: Latency metrics for node ip-172-20-45-112.eu-west-3.compute.internal Jan 1 02:31:30.358: INFO: Logging node info for node ip-172-20-47-178.eu-west-3.compute.internal Jan 1 02:31:30.463: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-47-178.eu-west-3.compute.internal 835e5337-d2b9-45ad-bf0e-9d32514f71ba 8456 0 2023-01-01 02:20:25 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:eu-west-3 failure-domain.beta.kubernetes.io/zone:eu-west-3a kubelet_cleanup:true kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-47-178.eu-west-3.compute.internal kubernetes.io/os:linux kubernetes.io/role:node node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:eu-west-3a topology.kubernetes.io/region:eu-west-3 topology.kubernetes.io/zone:eu-west-3a] map[csi.volume.kubernetes.io/nodeid:{"csi-mock-csi-mock-volumes-5429":"csi-mock-csi-mock-volumes-5429","ebs.csi.aws.com":"i-00e1e60199f5a4d04"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kops-controller Update v1 2023-01-01 02:20:25 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}} } {kube-controller-manager Update v1 2023-01-01 02:20:25 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.2.0/24\"":{}}}} } {kubelet Update v1 2023-01-01 02:20:25 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {e2e.test Update v1 2023-01-01 02:25:15 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubelet_cleanup":{}}}} } {kubelet Update v1 2023-01-01 02:25:42 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:capacity":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.2.0/24,DoNotUseExternalID:,ProviderID:aws:///eu-west-3a/i-00e1e60199f5a4d04,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.2.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{50478186496 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4078190592 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{45430367772 0} {<nil>} 45430367772 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3973332992 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-01-01 02:30:48 +0000 UTC,LastTransitionTime:2023-01-01 02:20:25 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-01-01 02:30:48 +0000 UTC,LastTransitionTime:2023-01-01 02:20:25 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-01-01 02:30:48 +0000 UTC,LastTransitionTime:2023-01-01 02:20:25 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-01-01 02:30:48 +0000 UTC,LastTransitionTime:2023-01-01 02:20:46 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.47.178,},NodeAddress{Type:ExternalIP,Address:13.36.177.101,},NodeAddress{Type:Hostname,Address:ip-172-20-47-178.eu-west-3.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-47-178.eu-west-3.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-13-36-177-101.eu-west-3.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec243eda226c96d5573f3eaa3203df0d,SystemUUID:ec243eda-226c-96d5-573f-3eaa3203df0d,BootID:15dccaa7-d7c7-419d-b7de-76be5781a51a,KernelVersion:4.19.0-23-cloud-amd64,OSImage:Debian GNU/Linux 10 (buster),ContainerRuntimeVersion:docker://20.10.17,KubeletVersion:v1.23.15,KubeProxyVersion:v1.23.15,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:456569528,},ContainerImage{Names:[nginx@sha256:0047b729188a15da49380d9506d65959cce6d40291ccfb4e039f5dc7efd33286 nginx:latest],SizeBytes:141812353,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:7e8bdd271312fd25fc5ff5a8f04727be84044eb3d7d8d03611972a6752e2e11e k8s.gcr.io/e2e-test-images/agnhost:2.39],SizeBytes:126872991,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.23.15 registry.k8s.io/kube-proxy-amd64:v1.23.15],SizeBytes:112336036,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:10e231cf07c82dfab5141c1afe548dc734e5fc1f67665ec3982a325d1bd31a9a registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.0],SizeBytes:94471559,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:c8e03f60afa90a28e4bb6ec9a8d0fc36d89de4b7475cf2d613afa793ec969fe0 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.0],SizeBytes:56382329,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:c5be65d6679efabb969d9b019300d187437ae876f992c40911fd2892bbef3b36 k8s.gcr.io/sig-storage/csi-attacher:v3.2.0],SizeBytes:53500826,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:017727efcfeb7d053af68e51436ce8e65edbc6ca573720afb4f79c8594036955 registry.k8s.io/coredns/coredns:v1.10.0],SizeBytes:50249337,},ContainerImage{Names:[registry.k8s.io/cpa/cluster-proportional-autoscaler@sha256:68d396900aeaa072c1f27289485fdac29834045a6f3ffe369bf389d830ef572d registry.k8s.io/cpa/cluster-proportional-autoscaler:1.8.6],SizeBytes:47910609,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:a61d309da54641db41fb8f35718f744e9f730d4d0384f8c4b186ddc9f06cbd5f k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.1.0],SizeBytes:19662887,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:19582112,},ContainerImage{Names:[k8s.gcr.io/sig-storage/mock-driver@sha256:a7b517f9e0f42ffade802eef9cefa271372386b85f55b702b493241e58459793 k8s.gcr.io/sig-storage/mock-driver:v4.1.0],SizeBytes:17680993,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:17611980,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:13616070e3f29de4417eee434a8ef472221c9e51b3d037b5a6b46cef08eb7443 k8s.gcr.io/e2e-test-images/nginx:1.14-2],SizeBytes:16032814,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf k8s.gcr.io/e2e-test-images/busybox:1.29-2],SizeBytes:1154361,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db k8s.gcr.io/pause:3.6 registry.k8s.io/pause:3.6],SizeBytes:682696,},},VolumesInUse:[kubernetes.io/csi/csi-mock-csi-mock-volumes-5429^4],VolumesAttached:[]AttachedVolume{},Config:nil,},} Jan 1 02:31:30.464: INFO: Logging kubelet events for node ip-172-20-47-178.eu-west-3.compute.internal Jan 1 02:31:30.570: INFO: Logging pods the kubelet thinks is on node ip-172-20-47-178.eu-west-3.compute.internal Jan 1 02:31:30.687: INFO: hostexec-ip-172-20-47-178.eu-west-3.compute.internal-k7k4h started at 2023-01-01 02:30:47 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:30.687: INFO: Container agnhost-container ready: true, restart count 0 Jan 1 02:31:30.687: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-hrgrq started at 2023-01-01 02:25:17 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:30.687: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:31:30.687: INFO: csi-hostpathplugin-0 started at 2023-01-01 02:31:02 +0000 UTC (0+7 container statuses recorded) Jan 1 02:31:30.687: INFO: Container csi-attacher ready: false, restart count 0 Jan 1 02:31:30.687: INFO: Container csi-provisioner ready: false, restart count 0 Jan 1 02:31:30.687: INFO: Container csi-resizer ready: false, restart count 0 Jan 1 02:31:30.687: INFO: Container csi-snapshotter ready: false, restart count 0 Jan 1 02:31:30.687: INFO: Container hostpath ready: false, restart count 0 Jan 1 02:31:30.687: INFO: Container liveness-probe ready: false, restart count 0 Jan 1 02:31:30.687: INFO: Container node-driver-registrar ready: false, restart count 0 Jan 1 02:31:30.687: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-grrsl started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:30.687: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:31:30.687: INFO: csi-hostpathplugin-0 started at 2023-01-01 02:30:56 +0000 UTC (0+7 container statuses recorded) Jan 1 02:31:30.687: INFO: Container csi-attacher ready: false, restart count 0 Jan 1 02:31:30.687: INFO: Container csi-provisioner ready: false, restart count 0 Jan 1 02:31:30.687: INFO: Container csi-resizer ready: false, restart count 0 Jan 1 02:31:30.687: INFO: Container csi-snapshotter ready: false, restart count 0 Jan 1 02:31:30.688: INFO: Container hostpath ready: false, restart count 0 Jan 1 02:31:30.688: INFO: Container liveness-probe ready: false, restart count 0 Jan 1 02:31:30.688: INFO: Container node-driver-registrar ready: false, restart count 0 Jan 1 02:31:30.688: INFO: csi-mockplugin-0 started at 2023-01-01 02:28:39 +0000 UTC (0+3 container statuses recorded) Jan 1 02:31:30.688: INFO: Container csi-provisioner ready: false, restart count 0 Jan 1 02:31:30.688: INFO: Container driver-registrar ready: false, restart count 0 Jan 1 02:31:30.688: INFO: Container mock ready: false, restart count 0 Jan 1 02:31:30.688: INFO: ebs-csi-node-vcprc started at 2023-01-01 02:20:26 +0000 UTC (0+3 container statuses recorded) Jan 1 02:31:30.688: INFO: Container ebs-plugin ready: true, restart count 0 Jan 1 02:31:30.688: INFO: Container liveness-probe ready: true, restart count 0 Jan 1 02:31:30.688: INFO: Container node-driver-registrar ready: true, restart count 0 Jan 1 02:31:30.688: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-49brr started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:30.688: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:31:30.688: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-z87gr started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:30.688: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:31:30.688: INFO: csi-mockplugin-attacher-0 started at 2023-01-01 02:25:20 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:30.688: INFO: Container csi-attacher ready: false, restart count 0 Jan 1 02:31:30.688: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-wjz96 started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:30.688: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:31:30.688: INFO: csi-mockplugin-0 started at 2023-01-01 02:25:20 +0000 UTC (0+3 container statuses recorded) Jan 1 02:31:30.688: INFO: Container csi-provisioner ready: true, restart count 0 Jan 1 02:31:30.688: INFO: Container driver-registrar ready: true, restart count 0 Jan 1 02:31:30.688: INFO: Container mock ready: true, restart count 0 Jan 1 02:31:30.688: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-w26gj started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:30.688: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:31:30.688: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-s4rqf started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:30.688: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:31:30.688: INFO: pvc-volume-tester-w4w4d started at 2023-01-01 02:25:38 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:30.688: INFO: Container volume-tester ready: false, restart count 0 Jan 1 02:31:30.688: INFO: cilium-kk2c2 started at 2023-01-01 02:20:26 +0000 UTC (1+1 container statuses recorded) Jan 1 02:31:30.688: INFO: Init container clean-cilium-state ready: true, restart count 0 Jan 1 02:31:30.688: INFO: Container cilium-agent ready: true, restart count 0 Jan 1 02:31:30.688: INFO: local-injector started at 2023-01-01 02:31:07 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:30.688: INFO: Container local-injector ready: false, restart count 0 Jan 1 02:31:30.688: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-6vhtn started at 2023-01-01 02:25:17 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:30.689: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: false, restart count 0 Jan 1 02:31:30.689: INFO: coredns-867df8f45c-wzntb started at 2023-01-01 02:20:46 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:30.689: INFO: Container coredns ready: true, restart count 0 Jan 1 02:31:30.689: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-wkclp started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:30.689: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:31:30.689: INFO: coredns-autoscaler-557ccb4c66-k4lzb started at 2023-01-01 02:20:46 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:30.689: INFO: Container autoscaler ready: true, restart count 0 Jan 1 02:31:30.689: INFO: csi-mockplugin-attacher-0 started at 2023-01-01 02:28:39 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:30.689: INFO: Container csi-attacher ready: false, restart count 0 Jan 1 02:31:30.689: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-pgkm2 started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:30.689: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:31:30.689: INFO: netserver-2 started at 2023-01-01 02:25:27 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:30.689: INFO: Container webserver ready: false, restart count 0 Jan 1 02:31:31.475: INFO: Latency metrics for node ip-172-20-47-178.eu-west-3.compute.internal Jan 1 02:31:31.476: INFO: Logging node info for node ip-172-20-49-224.eu-west-3.compute.internal Jan 1 02:31:31.580: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-49-224.eu-west-3.compute.internal d7d57146-c54a-4155-94f8-0f322216bfcf 8317 0 2023-01-01 02:20:28 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:eu-west-3 failure-domain.beta.kubernetes.io/zone:eu-west-3a kubelet_cleanup:true kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-49-224.eu-west-3.compute.internal kubernetes.io/os:linux kubernetes.io/role:node node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:eu-west-3a topology.hostpath.csi/node:ip-172-20-49-224.eu-west-3.compute.internal topology.kubernetes.io/region:eu-west-3 topology.kubernetes.io/zone:eu-west-3a] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-089a107939b27c28a"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kops-controller Update v1 2023-01-01 02:20:28 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}} } {kube-controller-manager Update v1 2023-01-01 02:20:28 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.4.0/24\"":{}}}} } {kubelet Update v1 2023-01-01 02:20:28 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {e2e.test Update v1 2023-01-01 02:25:15 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubelet_cleanup":{}}}} } {kubelet Update v1 2023-01-01 02:25:35 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:capacity":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.4.0/24,DoNotUseExternalID:,ProviderID:aws:///eu-west-3a/i-089a107939b27c28a,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.4.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{50478186496 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4078206976 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{45430367772 0} {<nil>} 45430367772 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3973349376 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-01-01 02:30:42 +0000 UTC,LastTransitionTime:2023-01-01 02:20:28 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-01-01 02:30:42 +0000 UTC,LastTransitionTime:2023-01-01 02:20:28 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-01-01 02:30:42 +0000 UTC,LastTransitionTime:2023-01-01 02:20:28 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-01-01 02:30:42 +0000 UTC,LastTransitionTime:2023-01-01 02:20:48 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.49.224,},NodeAddress{Type:ExternalIP,Address:15.237.95.101,},NodeAddress{Type:Hostname,Address:ip-172-20-49-224.eu-west-3.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-49-224.eu-west-3.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-15-237-95-101.eu-west-3.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2d0a1bb33dd26f5a12c3896530131b,SystemUUID:ec2d0a1b-b33d-d26f-5a12-c3896530131b,BootID:9f64e91d-26f0-45aa-aec4-a54433e67034,KernelVersion:4.19.0-23-cloud-amd64,OSImage:Debian GNU/Linux 10 (buster),ContainerRuntimeVersion:docker://20.10.17,KubeletVersion:v1.23.15,KubeProxyVersion:v1.23.15,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:456569528,},ContainerImage{Names:[nginx@sha256:0047b729188a15da49380d9506d65959cce6d40291ccfb4e039f5dc7efd33286 nginx:latest],SizeBytes:141812353,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:7e8bdd271312fd25fc5ff5a8f04727be84044eb3d7d8d03611972a6752e2e11e k8s.gcr.io/e2e-test-images/agnhost:2.39],SizeBytes:126872991,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:1b9d1b2f36cb2dbee1960e82a9344aeb11bd4c4c03abf5e1853e0559c23855e3 k8s.gcr.io/e2e-test-images/httpd:2.4.38-2],SizeBytes:123781643,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.23.15 registry.k8s.io/kube-proxy-amd64:v1.23.15],SizeBytes:112336036,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:10e231cf07c82dfab5141c1afe548dc734e5fc1f67665ec3982a325d1bd31a9a registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.0],SizeBytes:94471559,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:4e74c0492bceddc598de1c90cc5bc14dcda94cb49fa9c5bad9d117c4834b5e08 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.1],SizeBytes:56379269,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:36c31f7e1f433c9634d24f876353e8646246d81a03c4e351202c2644daff1620 k8s.gcr.io/sig-storage/csi-resizer:v1.2.0],SizeBytes:53956308,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:ed98431376c9e944e19a465fe8ea944806714dd95416a0821096c78d66b579bd k8s.gcr.io/sig-storage/csi-snapshotter:v4.1.1],SizeBytes:53531990,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:60ab9b3e6a030d3038c87c0d6bca2930f58d1d72823e6a4af09767dc83b696a2 k8s.gcr.io/sig-storage/csi-attacher:v3.2.1],SizeBytes:53499570,},ContainerImage{Names:[k8s.gcr.io/sig-storage/hostpathplugin@sha256:232fe80174d60d520d36043103853a1d7ab4b7f3782cf43e45034f04ccda58ce k8s.gcr.io/sig-storage/hostpathplugin:v1.7.1],SizeBytes:29977921,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:19582112,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:2dee3fe5fe861bb66c3a4ac51114f3447a4cd35870e0f2e2b558c7a400d89589 k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.2.0],SizeBytes:18699436,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:17611980,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:529be2c9770add0cdd0c989115222ea9fc1be430c11095eb9f6dafcf98a36e2b k8s.gcr.io/sig-storage/livenessprobe:v2.4.0],SizeBytes:17175488,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:13616070e3f29de4417eee434a8ef472221c9e51b3d037b5a6b46cef08eb7443 k8s.gcr.io/e2e-test-images/nginx:1.14-2],SizeBytes:16032814,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf k8s.gcr.io/e2e-test-images/busybox:1.29-2],SizeBytes:1154361,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db k8s.gcr.io/pause:3.6 registry.k8s.io/pause:3.6],SizeBytes:682696,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Jan 1 02:31:31.581: INFO: Logging kubelet events for node ip-172-20-49-224.eu-west-3.compute.internal Jan 1 02:31:31.689: INFO: Logging pods the kubelet thinks is on node ip-172-20-49-224.eu-west-3.compute.internal Jan 1 02:31:31.859: INFO: test-ss-0 started at 2023-01-01 02:27:56 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:31.859: INFO: Container webserver ready: false, restart count 0 Jan 1 02:31:31.859: INFO: hostexec-ip-172-20-49-224.eu-west-3.compute.internal-hsfbk started at 2023-01-01 02:31:01 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:31.859: INFO: Container agnhost-container ready: true, restart count 0 Jan 1 02:31:31.859: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-pgc9g started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:31.859: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:31:31.859: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-29cdj started at 2023-01-01 02:25:17 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:31.859: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: false, restart count 0 Jan 1 02:31:31.859: INFO: echo-sourceip started at 2023-01-01 02:26:27 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:31.859: INFO: Container agnhost-container ready: false, restart count 0 Jan 1 02:31:31.859: INFO: pod-2 started at 2023-01-01 02:31:27 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:31.859: INFO: Container donothing ready: false, restart count 0 Jan 1 02:31:31.859: INFO: pod-service-account-nomountsa-mountspec started at 2023-01-01 02:29:39 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:31.859: INFO: Container token-test ready: false, restart count 0 Jan 1 02:31:31.859: INFO: e2e-configmap-dns-server-7551ef4f-8c37-4168-bf54-91644ec7fb8f started at 2023-01-01 02:29:40 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:31.859: INFO: Container agnhost-container ready: true, restart count 0 Jan 1 02:31:31.859: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-7z6bp started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:31.859: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:31:31.859: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-lrzjq started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:31.859: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:31:31.859: INFO: netserver-3 started at 2023-01-01 02:25:27 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:31.859: INFO: Container webserver ready: true, restart count 0 Jan 1 02:31:31.859: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-4jf6d started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:31.859: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:31:31.859: INFO: dns-test-f6141192-e7ab-4530-bedd-b1f3b48a9237 started at 2023-01-01 02:30:56 +0000 UTC (0+3 container statuses recorded) Jan 1 02:31:31.859: INFO: Container jessie-querier ready: false, restart count 0 Jan 1 02:31:31.859: INFO: Container querier ready: false, restart count 0 Jan 1 02:31:31.859: INFO: Container webserver ready: false, restart count 0 Jan 1 02:31:31.859: INFO: cilium-78jll started at 2023-01-01 02:20:29 +0000 UTC (1+1 container statuses recorded) Jan 1 02:31:31.859: INFO: Init container clean-cilium-state ready: true, restart count 0 Jan 1 02:31:31.859: INFO: Container cilium-agent ready: true, restart count 0 Jan 1 02:31:31.859: INFO: ebs-csi-node-tzntf started at 2023-01-01 02:20:29 +0000 UTC (0+3 container statuses recorded) Jan 1 02:31:31.859: INFO: Container ebs-plugin ready: true, restart count 0 Jan 1 02:31:31.859: INFO: Container liveness-probe ready: true, restart count 0 Jan 1 02:31:31.859: INFO: Container node-driver-registrar ready: true, restart count 0 Jan 1 02:31:31.859: INFO: pod-subpath-test-secret-87r5 started at 2023-01-01 02:30:51 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:31.859: INFO: Container test-container-subpath-secret-87r5 ready: false, restart count 0 Jan 1 02:31:31.859: INFO: affinity-clusterip-timeout-pxjtk started at 2023-01-01 02:27:32 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:31.859: INFO: Container affinity-clusterip-timeout ready: true, restart count 0 Jan 1 02:31:31.859: INFO: annotationupdate3b35e9ec-7ad7-4bd9-8137-1bd2bf1a899a started at 2023-01-01 02:28:25 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:31.859: INFO: Container client-container ready: false, restart count 0 Jan 1 02:31:31.859: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-tv984 started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:31.859: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:31:31.859: INFO: pod-3e35f616-b2af-42fe-8148-bfe7bf04221b started at 2023-01-01 02:31:05 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:31.859: INFO: Container write-pod ready: false, restart count 0 Jan 1 02:31:31.859: INFO: dns-test-a8277dcf-764d-4d97-a381-813a8d1f903c started at 2023-01-01 02:31:17 +0000 UTC (0+3 container statuses recorded) Jan 1 02:31:31.859: INFO: Container jessie-querier ready: false, restart count 0 Jan 1 02:31:31.859: INFO: Container querier ready: false, restart count 0 Jan 1 02:31:31.859: INFO: Container webserver ready: false, restart count 0 Jan 1 02:31:31.859: INFO: csi-mockplugin-attacher-0 started at 2023-01-01 02:31:26 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:31.859: INFO: Container csi-attacher ready: false, restart count 0 Jan 1 02:31:31.859: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-cz8qh started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:31.859: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:31:31.859: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-2gndb started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:31.859: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:31:31.859: INFO: csi-mockplugin-0 started at 2023-01-01 02:31:26 +0000 UTC (0+3 container statuses recorded) Jan 1 02:31:31.859: INFO: Container csi-provisioner ready: false, restart count 0 Jan 1 02:31:31.859: INFO: Container driver-registrar ready: false, restart count 0 Jan 1 02:31:31.859: INFO: Container mock ready: false, restart count 0 Jan 1 02:31:31.859: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-h6dxb started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:31.859: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:31:31.859: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-6n5j9 started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:31:31.859: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:31:33.978: INFO: Latency metrics for node ip-172-20-49-224.eu-west-3.compute.internal Jan 1 02:31:33.978: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready �[1mSTEP�[0m: Destroying namespace "services-5926" for this suite. [AfterEach] [sig-network] Services /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/network/service.go:756
Filter through log files
go run hack/e2e.go -v --test --test_args='--ginkgo.focus=Kubernetes\se2e\ssuite\s\[sig\-node\]\sPods\sExtended\sPod\sContainer\slifecycle\sevicted\spods\sshould\sbe\sterminal$'
/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/node/pods.go:300 Jan 1 02:48:28.674: error waiting for pod to be evicted: Gave up after waiting 5m0s for pod "pod-should-be-evicted46f88c5a-f01f-4b72-94c9-0cddd928f146" to be "terminated due to deadline exceeded" /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/onsi/ginkgo/internal/leafnodes/runner.go:113from junit_24.xml
[BeforeEach] [sig-node] Pods Extended /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:185 �[1mSTEP�[0m: Creating a kubernetes client Jan 1 02:43:26.742: INFO: >>> kubeConfig: /root/.kube/config �[1mSTEP�[0m: Building a namespace api object, basename pods �[1mSTEP�[0m: Waiting for a default service account to be provisioned in namespace �[1mSTEP�[0m: Waiting for kube-root-ca.crt to be provisioned in namespace [BeforeEach] Pod Container lifecycle /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/node/pods.go:226 [It] evicted pods should be terminal /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/node/pods.go:300 �[1mSTEP�[0m: creating the pod that should be evicted �[1mSTEP�[0m: submitting the pod to kubernetes Jan 1 02:43:27.569: INFO: Waiting up to 5m0s for pod "pod-should-be-evicted46f88c5a-f01f-4b72-94c9-0cddd928f146" in namespace "pods-2607" to be "terminated due to deadline exceeded" Jan 1 02:43:27.672: INFO: Pod "pod-should-be-evicted46f88c5a-f01f-4b72-94c9-0cddd928f146": Phase="Pending", Reason="", readiness=false. Elapsed: 103.028304ms Jan 1 02:43:29.776: INFO: Pod "pod-should-be-evicted46f88c5a-f01f-4b72-94c9-0cddd928f146": Phase="Pending", Reason="", readiness=false. Elapsed: 2.206487159s Jan 1 02:43:31.883: INFO: Pod "pod-should-be-evicted46f88c5a-f01f-4b72-94c9-0cddd928f146": Phase="Running", Reason="", readiness=true. Elapsed: 4.31334598s Jan 1 02:43:33.989: INFO: Pod "pod-should-be-evicted46f88c5a-f01f-4b72-94c9-0cddd928f146": Phase="Running", Reason="", readiness=true. Elapsed: 6.419778725s Jan 1 02:43:36.097: INFO: Pod "pod-should-be-evicted46f88c5a-f01f-4b72-94c9-0cddd928f146": Phase="Running", Reason="", readiness=true. Elapsed: 8.528000107s Jan 1 02:43:38.201: INFO: Pod "pod-should-be-evicted46f88c5a-f01f-4b72-94c9-0cddd928f146": Phase="Running", Reason="", readiness=true. Elapsed: 10.631603516s Jan 1 02:43:40.306: INFO: Pod "pod-should-be-evicted46f88c5a-f01f-4b72-94c9-0cddd928f146": Phase="Running", Reason="", readiness=true. Elapsed: 12.736276808s Jan 1 02:43:42.412: INFO: Pod "pod-should-be-evicted46f88c5a-f01f-4b72-94c9-0cddd928f146": Phase="Running", Reason="", readiness=true. Elapsed: 14.842478151s Jan 1 02:43:44.516: INFO: Pod "pod-should-be-evicted46f88c5a-f01f-4b72-94c9-0cddd928f146": Phase="Running", Reason="", readiness=true. Elapsed: 16.947134122s Jan 1 02:43:46.622: INFO: Pod "pod-should-be-evicted46f88c5a-f01f-4b72-94c9-0cddd928f146": Phase="Running", Reason="", readiness=true. Elapsed: 19.052385153s Jan 1 02:43:48.725: INFO: Pod "pod-should-be-evicted46f88c5a-f01f-4b72-94c9-0cddd928f146": Phase="Running", Reason="", readiness=true. Elapsed: 21.15540817s Jan 1 02:43:50.829: INFO: Pod "pod-should-be-evicted46f88c5a-f01f-4b72-94c9-0cddd928f146": Phase="Running", Reason="", readiness=true. Elapsed: 23.259354124s Jan 1 02:43:52.933: INFO: Pod "pod-should-be-evicted46f88c5a-f01f-4b72-94c9-0cddd928f146": Phase="Running", Reason="", readiness=true. Elapsed: 25.363530081s Jan 1 02:43:55.037: INFO: Pod "pod-should-be-evicted46f88c5a-f01f-4b72-94c9-0cddd928f146": Phase="Running", Reason="", readiness=true. Elapsed: 27.467264285s Jan 1 02:43:57.142: INFO: Pod "pod-should-be-evicted46f88c5a-f01f-4b72-94c9-0cddd928f146": Phase="Running", Reason="", readiness=true. Elapsed: 29.572656365s Jan 1 02:43:59.246: INFO: Pod "pod-should-be-evicted46f88c5a-f01f-4b72-94c9-0cddd928f146": Phase="Running", Reason="", readiness=true. Elapsed: 31.676280379s Jan 1 02:44:01.349: INFO: Pod "pod-should-be-evicted46f88c5a-f01f-4b72-94c9-0cddd928f146": Phase="Running", Reason="", readiness=true. Elapsed: 33.779672818s Jan 1 02:44:03.457: INFO: Pod "pod-should-be-evicted46f88c5a-f01f-4b72-94c9-0cddd928f146": Phase="Running", Reason="", readiness=true. Elapsed: 35.887669453s Jan 1 02:44:05.560: INFO: Pod "pod-should-be-evicted46f88c5a-f01f-4b72-94c9-0cddd928f146": Phase="Running", Reason="", readiness=true. Elapsed: 37.990418389s Jan 1 02:44:07.663: INFO: Pod "pod-should-be-evicted46f88c5a-f01f-4b72-94c9-0cddd928f146": Phase="Running", Reason="", readiness=true. Elapsed: 40.093840163s Jan 1 02:44:09.766: INFO: Pod "pod-should-be-evicted46f88c5a-f01f-4b72-94c9-0cddd928f146": Phase="Running", Reason="", readiness=true. Elapsed: 42.197005806s Jan 1 02:44:11.870: INFO: Pod "pod-should-be-evicted46f88c5a-f01f-4b72-94c9-0cddd928f146": Phase="Running", Reason="", readiness=true. Elapsed: 44.30113199s Jan 1 02:44:13.973: INFO: Pod "pod-should-be-evicted46f88c5a-f01f-4b72-94c9-0cddd928f146": Phase="Running", Reason="", readiness=true. Elapsed: 46.403845253s Jan 1 02:44:16.077: INFO: Pod "pod-should-be-evicted46f88c5a-f01f-4b72-94c9-0cddd928f146": Phase="Running", Reason="", readiness=true. Elapsed: 48.508137942s Jan 1 02:44:18.181: INFO: Pod "pod-should-be-evicted46f88c5a-f01f-4b72-94c9-0cddd928f146": Phase="Running", Reason="", readiness=true. Elapsed: 50.611748776s Jan 1 02:44:20.307: INFO: Pod "pod-should-be-evicted46f88c5a-f01f-4b72-94c9-0cddd928f146": Phase="Running", Reason="", readiness=true. Elapsed: 52.737750642s Jan 1 02:44:22.411: INFO: Pod "pod-should-be-evicted46f88c5a-f01f-4b72-94c9-0cddd928f146": Phase="Running", Reason="", readiness=true. Elapsed: 54.841701546s Jan 1 02:44:24.516: INFO: Pod "pod-should-be-evicted46f88c5a-f01f-4b72-94c9-0cddd928f146": Phase="Running", Reason="", readiness=true. Elapsed: 56.946775891s Jan 1 02:44:26.620: INFO: Pod "pod-should-be-evicted46f88c5a-f01f-4b72-94c9-0cddd928f146": Phase="Running", Reason="", readiness=true. Elapsed: 59.050716624s Jan 1 02:44:28.723: INFO: Pod "pod-should-be-evicted46f88c5a-f01f-4b72-94c9-0cddd928f146": Phase="Running", Reason="", readiness=true. Elapsed: 1m1.153581003s Jan 1 02:44:30.827: INFO: Pod "pod-should-be-evicted46f88c5a-f01f-4b72-94c9-0cddd928f146": Phase="Running", Reason="", readiness=true. Elapsed: 1m3.257418296s Jan 1 02:44:32.930: INFO: Pod "pod-should-be-evicted46f88c5a-f01f-4b72-94c9-0cddd928f146": Phase="Running", Reason="", readiness=true. Elapsed: 1m5.360885816s Jan 1 02:44:35.034: INFO: Pod "pod-should-be-evicted46f88c5a-f01f-4b72-94c9-0cddd928f146": Phase="Running", Reason="", readiness=true. Elapsed: 1m7.464920675s Jan 1 02:44:37.137: INFO: Pod "pod-should-be-evicted46f88c5a-f01f-4b72-94c9-0cddd928f146": Phase="Running", Reason="", readiness=true. Elapsed: 1m9.567756228s Jan 1 02:44:39.241: INFO: Pod "pod-should-be-evicted46f88c5a-f01f-4b72-94c9-0cddd928f146": Phase="Running", Reason="", readiness=true. Elapsed: 1m11.671953781s Jan 1 02:44:41.344: INFO: Pod "pod-should-be-evicted46f88c5a-f01f-4b72-94c9-0cddd928f146": Phase="Running", Reason="", readiness=true. Elapsed: 1m13.775159314s Jan 1 02:44:43.447: INFO: Pod "pod-should-be-evicted46f88c5a-f01f-4b72-94c9-0cddd928f146": Phase="Running", Reason="", readiness=true. Elapsed: 1m15.878127188s Jan 1 02:44:45.551: INFO: Pod "pod-should-be-evicted46f88c5a-f01f-4b72-94c9-0cddd928f146": Phase="Running", Reason="", readiness=true. Elapsed: 1m17.981371523s Jan 1 02:44:47.655: INFO: Pod "pod-should-be-evicted46f88c5a-f01f-4b72-94c9-0cddd928f146": Phase="Running", Reason="", readiness=true. Elapsed: 1m20.085856548s Jan 1 02:44:49.759: INFO: Pod "pod-should-be-evicted46f88c5a-f01f-4b72-94c9-0cddd928f146": Phase="Running", Reason="", readiness=true. Elapsed: 1m22.189809337s Jan 1 02:44:51.862: INFO: Pod "pod-should-be-evicted46f88c5a-f01f-4b72-94c9-0cddd928f146": Phase="Running", Reason="", readiness=true. Elapsed: 1m24.292956005s Jan 1 02:44:53.965: INFO: Pod "pod-should-be-evicted46f88c5a-f01f-4b72-94c9-0cddd928f146": Phase="Running", Reason="", readiness=true. Elapsed: 1m26.395925917s Jan 1 02:44:56.069: INFO: Pod "pod-should-be-evicted46f88c5a-f01f-4b72-94c9-0cddd928f146": Phase="Running", Reason="", readiness=true. Elapsed: 1m28.500130015s Jan 1 02:44:58.172: INFO: Pod "pod-should-be-evicted46f88c5a-f01f-4b72-94c9-0cddd928f146": Phase="Running", Reason="", readiness=true. Elapsed: 1m30.603153095s Jan 1 02:45:00.277: INFO: Pod "pod-should-be-evicted46f88c5a-f01f-4b72-94c9-0cddd928f146": Phase="Running", Reason="", readiness=true. Elapsed: 1m32.707281717s Jan 1 02:45:02.380: INFO: Pod "pod-should-be-evicted46f88c5a-f01f-4b72-94c9-0cddd928f146": Phase="Running", Reason="", readiness=true. Elapsed: 1m34.811177588s Jan 1 02:45:04.485: INFO: Pod "pod-should-be-evicted46f88c5a-f01f-4b72-94c9-0cddd928f146": Phase="Running", Reason="", readiness=true. Elapsed: 1m36.915882634s Jan 1 02:45:06.588: INFO: Pod "pod-should-be-evicted46f88c5a-f01f-4b72-94c9-0cddd928f146": Phase="Running", Reason="", readiness=true. Elapsed: 1m39.019216402s Jan 1 02:45:08.692: INFO: Pod "pod-should-be-evicted46f88c5a-f01f-4b72-94c9-0cddd928f146": Phase="Running", Reason="", readiness=true. Elapsed: 1m41.12318652s Jan 1 02:45:10.798: INFO: Pod "pod-should-be-evicted46f88c5a-f01f-4b72-94c9-0cddd928f146": Phase="Running", Reason="", readiness=true. Elapsed: 1m43.228619027s Jan 1 02:45:12.902: INFO: Pod "pod-should-be-evicted46f88c5a-f01f-4b72-94c9-0cddd928f146": Phase="Running", Reason="", readiness=true. Elapsed: 1m45.332491883s Jan 1 02:45:15.010: INFO: Pod "pod-should-be-evicted46f88c5a-f01f-4b72-94c9-0cddd928f146": Phase="Running", Reason="", readiness=true. Elapsed: 1m47.440790927s Jan 1 02:45:17.114: INFO: Pod "pod-should-be-evicted46f88c5a-f01f-4b72-94c9-0cddd928f146": Phase="Running", Reason="", readiness=true. Elapsed: 1m49.544697851s Jan 1 02:45:19.218: INFO: Pod "pod-should-be-evicted46f88c5a-f01f-4b72-94c9-0cddd928f146": Phase="Running", Reason="", readiness=true. Elapsed: 1m51.648900064s Jan 1 02:45:21.322: INFO: Pod "pod-should-be-evicted46f88c5a-f01f-4b72-94c9-0cddd928f146": Phase="Running", Reason="", readiness=true. Elapsed: 1m53.752637952s Jan 1 02:45:23.425: INFO: Pod "pod-should-be-evicted46f88c5a-f01f-4b72-94c9-0cddd928f146": Phase="Running", Reason="", readiness=true. Elapsed: 1m55.85598341s Jan 1 02:45:25.529: INFO: Pod "pod-should-be-evicted46f88c5a-f01f-4b72-94c9-0cddd928f146": Phase="Running", Reason="", readiness=true. Elapsed: 1m57.959900931s Jan 1 02:45:27.637: INFO: Pod "pod-should-be-evicted46f88c5a-f01f-4b72-94c9-0cddd928f146": Phase="Running", Reason="", readiness=true. Elapsed: 2m0.067568516s Jan 1 02:45:29.747: INFO: Pod "pod-should-be-evicted46f88c5a-f01f-4b72-94c9-0cddd928f146": Phase="Running", Reason="", readiness=true. Elapsed: 2m2.177344557s Jan 1 02:45:31.862: INFO: Pod "pod-should-be-evicted46f88c5a-f01f-4b72-94c9-0cddd928f146": Phase="Running", Reason="", readiness=true. Elapsed: 2m4.292487922s Jan 1 02:45:33.965: INFO: Pod "pod-should-be-evicted46f88c5a-f01f-4b72-94c9-0cddd928f146": Phase="Running", Reason="", readiness=true. Elapsed: 2m6.395430319s Jan 1 02:45:36.068: INFO: Pod "pod-should-be-evicted46f88c5a-f01f-4b72-94c9-0cddd928f146": Phase="Running", Reason="", readiness=true. Elapsed: 2m8.499064938s Jan 1 02:45:38.171: INFO: Pod "pod-should-be-evicted46f88c5a-f01f-4b72-94c9-0cddd928f146": Phase="Running", Reason="", readiness=true. Elapsed: 2m10.601950542s Jan 1 02:45:40.275: INFO: Pod "pod-should-be-evicted46f88c5a-f01f-4b72-94c9-0cddd928f146": Phase="Running", Reason="", readiness=true. Elapsed: 2m12.705763489s Jan 1 02:45:42.379: INFO: Pod "pod-should-be-evicted46f88c5a-f01f-4b72-94c9-0cddd928f146": Phase="Running", Reason="", readiness=true. Elapsed: 2m14.809792429s Jan 1 02:45:44.482: INFO: Pod "pod-should-be-evicted46f88c5a-f01f-4b72-94c9-0cddd928f146": Phase="Running", Reason="", readiness=true. Elapsed: 2m16.912553651s Jan 1 02:45:46.586: INFO: Pod "pod-should-be-evicted46f88c5a-f01f-4b72-94c9-0cddd928f146": Phase="Running", Reason="", readiness=true. Elapsed: 2m19.016681192s Jan 1 02:45:48.692: INFO: Pod "pod-should-be-evicted46f88c5a-f01f-4b72-94c9-0cddd928f146": Phase="Running", Reason="", readiness=true. Elapsed: 2m21.122238032s Jan 1 02:45:50.794: INFO: Pod "pod-should-be-evicted46f88c5a-f01f-4b72-94c9-0cddd928f146": Phase="Running", Reason="", readiness=true. Elapsed: 2m23.224761569s Jan 1 02:45:52.897: INFO: Pod "pod-should-be-evicted46f88c5a-f01f-4b72-94c9-0cddd928f146": Phase="Running", Reason="", readiness=true. Elapsed: 2m25.328077265s Jan 1 02:45:55.001: INFO: Pod "pod-should-be-evicted46f88c5a-f01f-4b72-94c9-0cddd928f146": Phase="Running", Reason="", readiness=true. Elapsed: 2m27.432123797s Jan 1 02:45:57.106: INFO: Pod "pod-should-be-evicted46f88c5a-f01f-4b72-94c9-0cddd928f146": Phase="Running", Reason="", readiness=true. Elapsed: 2m29.536474357s Jan 1 02:45:59.215: INFO: Pod "pod-should-be-evicted46f88c5a-f01f-4b72-94c9-0cddd928f146": Phase="Running", Reason="", readiness=true. Elapsed: 2m31.645546122s Jan 1 02:46:01.318: INFO: Pod "pod-should-be-evicted46f88c5a-f01f-4b72-94c9-0cddd928f146": Phase="Running", Reason="", readiness=true. Elapsed: 2m33.749124589s Jan 1 02:46:03.427: INFO: Pod "pod-should-be-evicted46f88c5a-f01f-4b72-94c9-0cddd928f146": Phase="Running", Reason="", readiness=true. Elapsed: 2m35.857658143s Jan 1 02:46:05.538: INFO: Pod "pod-should-be-evicted46f88c5a-f01f-4b72-94c9-0cddd928f146": Phase="Running", Reason="", readiness=true. Elapsed: 2m37.968743027s Jan 1 02:46:07.648: INFO: Pod "pod-should-be-evicted46f88c5a-f01f-4b72-94c9-0cddd928f146": Phase="Running", Reason="", readiness=true. Elapsed: 2m40.078521941s Jan 1 02:46:09.753: INFO: Pod "pod-should-be-evicted46f88c5a-f01f-4b72-94c9-0cddd928f146": Phase="Running", Reason="", readiness=true. Elapsed: 2m42.183364762s Jan 1 02:46:11.856: INFO: Pod "pod-should-be-evicted46f88c5a-f01f-4b72-94c9-0cddd928f146": Phase="Running", Reason="", readiness=true. Elapsed: 2m44.286393344s Jan 1 02:46:13.959: INFO: Pod "pod-should-be-evicted46f88c5a-f01f-4b72-94c9-0cddd928f146": Phase="Running", Reason="", readiness=true. Elapsed: 2m46.389864351s Jan 1 02:46:16.063: INFO: Pod "pod-should-be-evicted46f88c5a-f01f-4b72-94c9-0cddd928f146": Phase="Running", Reason="", readiness=true. Elapsed: 2m48.493581867s Jan 1 02:46:18.167: INFO: Pod "pod-should-be-evicted46f88c5a-f01f-4b72-94c9-0cddd928f146": Phase="Running", Reason="", readiness=true. Elapsed: 2m50.597520308s Jan 1 02:46:20.272: INFO: Pod "pod-should-be-evicted46f88c5a-f01f-4b72-94c9-0cddd928f146": Phase="Running", Reason="", readiness=true. Elapsed: 2m52.70273241s Jan 1 02:46:22.376: INFO: Pod "pod-should-be-evicted46f88c5a-f01f-4b72-94c9-0cddd928f146": Phase="Running", Reason="", readiness=true. Elapsed: 2m54.806461977s Jan 1 02:46:24.480: INFO: Pod "pod-should-be-evicted46f88c5a-f01f-4b72-94c9-0cddd928f146": Phase="Running", Reason="", readiness=true. Elapsed: 2m56.910666119s Jan 1 02:46:26.584: INFO: Pod "pod-should-be-evicted46f88c5a-f01f-4b72-94c9-0cddd928f146": Phase="Running", Reason="", readiness=true. Elapsed: 2m59.014922456s Jan 1 02:46:28.687: INFO: Pod "pod-should-be-evicted46f88c5a-f01f-4b72-94c9-0cddd928f146": Phase="Running", Reason="", readiness=true. Elapsed: 3m1.118012519s Jan 1 02:46:30.790: INFO: Pod "pod-should-be-evicted46f88c5a-f01f-4b72-94c9-0cddd928f146": Phase="Running", Reason="", readiness=true. Elapsed: 3m3.221121415s Jan 1 02:46:32.895: INFO: Pod "pod-should-be-evicted46f88c5a-f01f-4b72-94c9-0cddd928f146": Phase="Running", Reason="", readiness=true. Elapsed: 3m5.326016487s Jan 1 02:46:34.998: INFO: Pod "pod-should-be-evicted46f88c5a-f01f-4b72-94c9-0cddd928f146": Phase="Running", Reason="", readiness=true. Elapsed: 3m7.42909111s Jan 1 02:46:37.102: INFO: Pod "pod-should-be-evicted46f88c5a-f01f-4b72-94c9-0cddd928f146": Phase="Running", Reason="", readiness=true. Elapsed: 3m9.532996087s Jan 1 02:46:39.206: INFO: Pod "pod-should-be-evicted46f88c5a-f01f-4b72-94c9-0cddd928f146": Phase="Running", Reason="", readiness=true. Elapsed: 3m11.637031512s Jan 1 02:46:41.310: INFO: Pod "pod-should-be-evicted46f88c5a-f01f-4b72-94c9-0cddd928f146": Phase="Running", Reason="", readiness=true. Elapsed: 3m13.740534289s Jan 1 02:46:43.418: INFO: Pod "pod-should-be-evicted46f88c5a-f01f-4b72-94c9-0cddd928f146": Phase="Running", Reason="", readiness=true. Elapsed: 3m15.848264001s Jan 1 02:46:45.520: INFO: Pod "pod-should-be-evicted46f88c5a-f01f-4b72-94c9-0cddd928f146": Phase="Running", Reason="", readiness=true. Elapsed: 3m17.950955492s Jan 1 02:46:47.623: INFO: Pod "pod-should-be-evicted46f88c5a-f01f-4b72-94c9-0cddd928f146": Phase="Running", Reason="", readiness=true. Elapsed: 3m20.054187323s Jan 1 02:46:49.735: INFO: Pod "pod-should-be-evicted46f88c5a-f01f-4b72-94c9-0cddd928f146": Phase="Running", Reason="", readiness=true. Elapsed: 3m22.165567647s Jan 1 02:46:51.838: INFO: Pod "pod-should-be-evicted46f88c5a-f01f-4b72-94c9-0cddd928f146": Phase="Running", Reason="", readiness=true. Elapsed: 3m24.268410537s Jan 1 02:46:53.941: INFO: Pod "pod-should-be-evicted46f88c5a-f01f-4b72-94c9-0cddd928f146": Phase="Running", Reason="", readiness=true. Elapsed: 3m26.371957301s Jan 1 02:46:56.055: INFO: Pod "pod-should-be-evicted46f88c5a-f01f-4b72-94c9-0cddd928f146": Phase="Running", Reason="", readiness=true. Elapsed: 3m28.485651907s Jan 1 02:46:58.159: INFO: Pod "pod-should-be-evicted46f88c5a-f01f-4b72-94c9-0cddd928f146": Phase="Running", Reason="", readiness=true. Elapsed: 3m30.589852949s Jan 1 02:47:00.265: INFO: Pod "pod-should-be-evicted46f88c5a-f01f-4b72-94c9-0cddd928f146": Phase="Running", Reason="", readiness=true. Elapsed: 3m32.695669173s Jan 1 02:47:02.373: INFO: Pod "pod-should-be-evicted46f88c5a-f01f-4b72-94c9-0cddd928f146": Phase="Running", Reason="", readiness=true. Elapsed: 3m34.803839503s Jan 1 02:47:04.518: INFO: Pod "pod-should-be-evicted46f88c5a-f01f-4b72-94c9-0cddd928f146": Phase="Running", Reason="", readiness=true. Elapsed: 3m36.948655066s Jan 1 02:47:06.622: INFO: Pod "pod-should-be-evicted46f88c5a-f01f-4b72-94c9-0cddd928f146": Phase="Running", Reason="", readiness=true. Elapsed: 3m39.05284118s Jan 1 02:47:08.725: INFO: Pod "pod-should-be-evicted46f88c5a-f01f-4b72-94c9-0cddd928f146": Phase="Running", Reason="", readiness=true. Elapsed: 3m41.156029887s Jan 1 02:47:10.829: INFO: Pod "pod-should-be-evicted46f88c5a-f01f-4b72-94c9-0cddd928f146": Phase="Running", Reason="", readiness=true. Elapsed: 3m43.260099235s Jan 1 02:47:12.933: INFO: Pod "pod-should-be-evicted46f88c5a-f01f-4b72-94c9-0cddd928f146": Phase="Running", Reason="", readiness=true. Elapsed: 3m45.363421782s Jan 1 02:47:15.037: INFO: Pod "pod-should-be-evicted46f88c5a-f01f-4b72-94c9-0cddd928f146": Phase="Running", Reason="", readiness=true. Elapsed: 3m47.467278678s Jan 1 02:47:17.140: INFO: Pod "pod-should-be-evicted46f88c5a-f01f-4b72-94c9-0cddd928f146": Phase="Running", Reason="", readiness=true. Elapsed: 3m49.571171764s Jan 1 02:47:19.250: INFO: Pod "pod-should-be-evicted46f88c5a-f01f-4b72-94c9-0cddd928f146": Phase="Running", Reason="", readiness=true. Elapsed: 3m51.681050956s Jan 1 02:47:21.354: INFO: Pod "pod-should-be-evicted46f88c5a-f01f-4b72-94c9-0cddd928f146": Phase="Running", Reason="", readiness=true. Elapsed: 3m53.784961657s Jan 1 02:47:23.457: INFO: Pod "pod-should-be-evicted46f88c5a-f01f-4b72-94c9-0cddd928f146": Phase="Running", Reason="", readiness=true. Elapsed: 3m55.88811952s Jan 1 02:47:25.564: INFO: Pod "pod-should-be-evicted46f88c5a-f01f-4b72-94c9-0cddd928f146": Phase="Running", Reason="", readiness=true. Elapsed: 3m57.9947212s Jan 1 02:47:27.668: INFO: Pod "pod-should-be-evicted46f88c5a-f01f-4b72-94c9-0cddd928f146": Phase="Running", Reason="", readiness=true. Elapsed: 4m0.098870314s Jan 1 02:47:29.812: INFO: Pod "pod-should-be-evicted46f88c5a-f01f-4b72-94c9-0cddd928f146": Phase="Running", Reason="", readiness=true. Elapsed: 4m2.242410019s Jan 1 02:47:31.917: INFO: Pod "pod-should-be-evicted46f88c5a-f01f-4b72-94c9-0cddd928f146": Phase="Running", Reason="", readiness=true. Elapsed: 4m4.348034917s Jan 1 02:47:34.020: INFO: Pod "pod-should-be-evicted46f88c5a-f01f-4b72-94c9-0cddd928f146": Phase="Running", Reason="", readiness=true. Elapsed: 4m6.451178247s Jan 1 02:47:36.126: INFO: Pod "pod-should-be-evicted46f88c5a-f01f-4b72-94c9-0cddd928f146": Phase="Running", Reason="", readiness=true. Elapsed: 4m8.556640957s Jan 1 02:47:38.229: INFO: Pod "pod-should-be-evicted46f88c5a-f01f-4b72-94c9-0cddd928f146": Phase="Running", Reason="", readiness=true. Elapsed: 4m10.659905977s Jan 1 02:47:40.344: INFO: Pod "pod-should-be-evicted46f88c5a-f01f-4b72-94c9-0cddd928f146": Phase="Running", Reason="", readiness=true. Elapsed: 4m12.774632633s Jan 1 02:47:42.455: INFO: Pod "pod-should-be-evicted46f88c5a-f01f-4b72-94c9-0cddd928f146": Phase="Running", Reason="", readiness=true. Elapsed: 4m14.8856957s Jan 1 02:47:44.558: INFO: Pod "pod-should-be-evicted46f88c5a-f01f-4b72-94c9-0cddd928f146": Phase="Running", Reason="", readiness=true. Elapsed: 4m16.988993424s Jan 1 02:47:46.663: INFO: Pod "pod-should-be-evicted46f88c5a-f01f-4b72-94c9-0cddd928f146": Phase="Running", Reason="", readiness=true. Elapsed: 4m19.093245034s Jan 1 02:47:48.766: INFO: Pod "pod-should-be-evicted46f88c5a-f01f-4b72-94c9-0cddd928f146": Phase="Running", Reason="", readiness=true. Elapsed: 4m21.196406741s Jan 1 02:47:50.876: INFO: Pod "pod-should-be-evicted46f88c5a-f01f-4b72-94c9-0cddd928f146": Phase="Running", Reason="", readiness=true. Elapsed: 4m23.306507149s Jan 1 02:47:52.979: INFO: Pod "pod-should-be-evicted46f88c5a-f01f-4b72-94c9-0cddd928f146": Phase="Running", Reason="", readiness=true. Elapsed: 4m25.410094981s Jan 1 02:47:55.083: INFO: Pod "pod-should-be-evicted46f88c5a-f01f-4b72-94c9-0cddd928f146": Phase="Running", Reason="", readiness=true. Elapsed: 4m27.51372822s Jan 1 02:47:57.189: INFO: Pod "pod-should-be-evicted46f88c5a-f01f-4b72-94c9-0cddd928f146": Phase="Running", Reason="", readiness=true. Elapsed: 4m29.6192982s Jan 1 02:47:59.292: INFO: Pod "pod-should-be-evicted46f88c5a-f01f-4b72-94c9-0cddd928f146": Phase="Running", Reason="", readiness=true. Elapsed: 4m31.722653182s Jan 1 02:48:01.399: INFO: Pod "pod-should-be-evicted46f88c5a-f01f-4b72-94c9-0cddd928f146": Phase="Running", Reason="", readiness=true. Elapsed: 4m33.82953756s Jan 1 02:48:03.509: INFO: Pod "pod-should-be-evicted46f88c5a-f01f-4b72-94c9-0cddd928f146": Phase="Running", Reason="", readiness=true. Elapsed: 4m35.939934518s Jan 1 02:48:05.613: INFO: Pod "pod-should-be-evicted46f88c5a-f01f-4b72-94c9-0cddd928f146": Phase="Running", Reason="", readiness=true. Elapsed: 4m38.043907175s Jan 1 02:48:07.717: INFO: Pod "pod-should-be-evicted46f88c5a-f01f-4b72-94c9-0cddd928f146": Phase="Running", Reason="", readiness=true. Elapsed: 4m40.147618635s Jan 1 02:48:09.820: INFO: Pod "pod-should-be-evicted46f88c5a-f01f-4b72-94c9-0cddd928f146": Phase="Running", Reason="", readiness=true. Elapsed: 4m42.251186999s Jan 1 02:48:11.924: INFO: Pod "pod-should-be-evicted46f88c5a-f01f-4b72-94c9-0cddd928f146": Phase="Running", Reason="", readiness=true. Elapsed: 4m44.354609637s Jan 1 02:48:14.027: INFO: Pod "pod-should-be-evicted46f88c5a-f01f-4b72-94c9-0cddd928f146": Phase="Running", Reason="", readiness=true. Elapsed: 4m46.457676922s Jan 1 02:48:16.130: INFO: Pod "pod-should-be-evicted46f88c5a-f01f-4b72-94c9-0cddd928f146": Phase="Running", Reason="", readiness=true. Elapsed: 4m48.56065114s Jan 1 02:48:18.233: INFO: Pod "pod-should-be-evicted46f88c5a-f01f-4b72-94c9-0cddd928f146": Phase="Running", Reason="", readiness=true. Elapsed: 4m50.663673326s Jan 1 02:48:20.362: INFO: Pod "pod-should-be-evicted46f88c5a-f01f-4b72-94c9-0cddd928f146": Phase="Running", Reason="", readiness=true. Elapsed: 4m52.792510859s Jan 1 02:48:22.466: INFO: Pod "pod-should-be-evicted46f88c5a-f01f-4b72-94c9-0cddd928f146": Phase="Running", Reason="", readiness=true. Elapsed: 4m54.896245486s Jan 1 02:48:24.569: INFO: Pod "pod-should-be-evicted46f88c5a-f01f-4b72-94c9-0cddd928f146": Phase="Running", Reason="", readiness=true. Elapsed: 4m56.999876834s Jan 1 02:48:26.673: INFO: Pod "pod-should-be-evicted46f88c5a-f01f-4b72-94c9-0cddd928f146": Phase="Running", Reason="", readiness=true. Elapsed: 4m59.104086907s Jan 1 02:48:28.674: FAIL: error waiting for pod to be evicted: Gave up after waiting 5m0s for pod "pod-should-be-evicted46f88c5a-f01f-4b72-94c9-0cddd928f146" to be "terminated due to deadline exceeded" Full Stack Trace k8s.io/kubernetes/test/e2e.RunE2ETests(0x24c66d7) _output/dockerized/go/src/k8s.io/kubernetes/test/e2e/e2e.go:133 +0x697 k8s.io/kubernetes/test/e2e.TestE2E(0x243a8f9) _output/dockerized/go/src/k8s.io/kubernetes/test/e2e/e2e_test.go:136 +0x19 testing.tRunner(0xc000dca820, 0x735e880) /usr/local/go/src/testing/testing.go:1259 +0x102 created by testing.(*T).Run /usr/local/go/src/testing/testing.go:1306 +0x35a �[1mSTEP�[0m: deleting the pod [AfterEach] [sig-node] Pods Extended /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:186 �[1mSTEP�[0m: Collecting events from namespace "pods-2607". �[1mSTEP�[0m: Found 5 events. Jan 1 02:48:28.883: INFO: At 2023-01-01 02:43:27 +0000 UTC - event for pod-should-be-evicted46f88c5a-f01f-4b72-94c9-0cddd928f146: {default-scheduler } Scheduled: Successfully assigned pods-2607/pod-should-be-evicted46f88c5a-f01f-4b72-94c9-0cddd928f146 to ip-172-20-33-66.eu-west-3.compute.internal Jan 1 02:48:28.883: INFO: At 2023-01-01 02:43:29 +0000 UTC - event for pod-should-be-evicted46f88c5a-f01f-4b72-94c9-0cddd928f146: {kubelet ip-172-20-33-66.eu-west-3.compute.internal} Pulled: Container image "k8s.gcr.io/e2e-test-images/busybox:1.29-2" already present on machine Jan 1 02:48:28.883: INFO: At 2023-01-01 02:43:29 +0000 UTC - event for pod-should-be-evicted46f88c5a-f01f-4b72-94c9-0cddd928f146: {kubelet ip-172-20-33-66.eu-west-3.compute.internal} Created: Created container bar Jan 1 02:48:28.883: INFO: At 2023-01-01 02:43:30 +0000 UTC - event for pod-should-be-evicted46f88c5a-f01f-4b72-94c9-0cddd928f146: {kubelet ip-172-20-33-66.eu-west-3.compute.internal} Started: Started container bar Jan 1 02:48:28.883: INFO: At 2023-01-01 02:48:28 +0000 UTC - event for pod-should-be-evicted46f88c5a-f01f-4b72-94c9-0cddd928f146: {kubelet ip-172-20-33-66.eu-west-3.compute.internal} Killing: Stopping container bar Jan 1 02:48:28.986: INFO: POD NODE PHASE GRACE CONDITIONS Jan 1 02:48:28.986: INFO: pod-should-be-evicted46f88c5a-f01f-4b72-94c9-0cddd928f146 ip-172-20-33-66.eu-west-3.compute.internal Running 30s [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2023-01-01 02:43:27 +0000 UTC } {Ready True 0001-01-01 00:00:00 +0000 UTC 2023-01-01 02:43:30 +0000 UTC } {ContainersReady True 0001-01-01 00:00:00 +0000 UTC 2023-01-01 02:43:30 +0000 UTC } {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2023-01-01 02:43:27 +0000 UTC }] Jan 1 02:48:28.986: INFO: Jan 1 02:48:29.198: INFO: Logging node info for node ip-172-20-33-66.eu-west-3.compute.internal Jan 1 02:48:29.301: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-33-66.eu-west-3.compute.internal ec4b59b3-62db-4040-aad2-4c3249f80fe8 42763 0 2023-01-01 02:20:25 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:eu-west-3 failure-domain.beta.kubernetes.io/zone:eu-west-3a io.kubernetes.storage.mock/node:some-mock-node kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-33-66.eu-west-3.compute.internal kubernetes.io/os:linux kubernetes.io/role:node node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:eu-west-3a topology.hostpath.csi/node:ip-172-20-33-66.eu-west-3.compute.internal topology.kubernetes.io/region:eu-west-3 topology.kubernetes.io/zone:eu-west-3a] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-028313673124ac4ad"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kops-controller Update v1 2023-01-01 02:20:25 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}} } {kube-controller-manager Update v1 2023-01-01 02:20:25 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.1.0/24\"":{}}}} } {kubelet Update v1 2023-01-01 02:20:25 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {kube-controller-manager Update v1 2023-01-01 02:47:31 +0000 UTC FieldsV1 {"f:status":{"f:volumesAttached":{}}} status} {kubelet Update v1 2023-01-01 02:47:32 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:io.kubernetes.storage.mock/node":{},"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.1.0/24,DoNotUseExternalID:,ProviderID:aws:///eu-west-3a/i-028313673124ac4ad,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.1.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{50478186496 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4078190592 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{45430367772 0} {<nil>} 45430367772 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3973332992 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-01-01 02:48:02 +0000 UTC,LastTransitionTime:2023-01-01 02:20:25 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-01-01 02:48:02 +0000 UTC,LastTransitionTime:2023-01-01 02:20:25 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-01-01 02:48:02 +0000 UTC,LastTransitionTime:2023-01-01 02:20:25 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-01-01 02:48:02 +0000 UTC,LastTransitionTime:2023-01-01 02:20:46 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.33.66,},NodeAddress{Type:ExternalIP,Address:52.47.140.112,},NodeAddress{Type:Hostname,Address:ip-172-20-33-66.eu-west-3.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-33-66.eu-west-3.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-52-47-140-112.eu-west-3.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2ad2ce1a87d508c178acf6215c5767,SystemUUID:ec2ad2ce-1a87-d508-c178-acf6215c5767,BootID:b3b60783-d786-412b-a36a-802bcfae604d,KernelVersion:4.19.0-23-cloud-amd64,OSImage:Debian GNU/Linux 10 (buster),ContainerRuntimeVersion:docker://20.10.17,KubeletVersion:v1.23.15,KubeProxyVersion:v1.23.15,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:456569528,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/jessie-dnsutils@sha256:11e6a66017ba4e4b938c1612b7a54a3befcefd354796c04e1dba76873a13518e k8s.gcr.io/e2e-test-images/jessie-dnsutils:1.5],SizeBytes:253371792,},ContainerImage{Names:[nginx@sha256:0047b729188a15da49380d9506d65959cce6d40291ccfb4e039f5dc7efd33286 nginx:latest],SizeBytes:141812353,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:20f25f275d46aa728f7615a1ccc19c78b2ed89435bf943a44b339f70f45508e6 k8s.gcr.io/e2e-test-images/httpd:2.4.39-2],SizeBytes:126894770,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:7e8bdd271312fd25fc5ff5a8f04727be84044eb3d7d8d03611972a6752e2e11e k8s.gcr.io/e2e-test-images/agnhost:2.39],SizeBytes:126872991,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:1b9d1b2f36cb2dbee1960e82a9344aeb11bd4c4c03abf5e1853e0559c23855e3 k8s.gcr.io/e2e-test-images/httpd:2.4.38-2],SizeBytes:123781643,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.23.15 registry.k8s.io/kube-proxy-amd64:v1.23.15],SizeBytes:112336036,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:10e231cf07c82dfab5141c1afe548dc734e5fc1f67665ec3982a325d1bd31a9a registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.0],SizeBytes:94471559,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:c8e03f60afa90a28e4bb6ec9a8d0fc36d89de4b7475cf2d613afa793ec969fe0 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.0],SizeBytes:56382329,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:4e74c0492bceddc598de1c90cc5bc14dcda94cb49fa9c5bad9d117c4834b5e08 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.1],SizeBytes:56379269,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:36c31f7e1f433c9634d24f876353e8646246d81a03c4e351202c2644daff1620 k8s.gcr.io/sig-storage/csi-resizer:v1.2.0],SizeBytes:53956308,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:ed98431376c9e944e19a465fe8ea944806714dd95416a0821096c78d66b579bd k8s.gcr.io/sig-storage/csi-snapshotter:v4.1.1],SizeBytes:53531990,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:c5be65d6679efabb969d9b019300d187437ae876f992c40911fd2892bbef3b36 k8s.gcr.io/sig-storage/csi-attacher:v3.2.0],SizeBytes:53500826,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:60ab9b3e6a030d3038c87c0d6bca2930f58d1d72823e6a4af09767dc83b696a2 k8s.gcr.io/sig-storage/csi-attacher:v3.2.1],SizeBytes:53499570,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:017727efcfeb7d053af68e51436ce8e65edbc6ca573720afb4f79c8594036955 registry.k8s.io/coredns/coredns:v1.10.0],SizeBytes:50249337,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:7a5ba58a44e0d749e0767e4e37315bcf6a61f33ce3185c1991848af4db0fb70a k8s.gcr.io/sig-storage/csi-resizer:v1.1.0],SizeBytes:49176472,},ContainerImage{Names:[k8s.gcr.io/sig-storage/hostpathplugin@sha256:232fe80174d60d520d36043103853a1d7ab4b7f3782cf43e45034f04ccda58ce k8s.gcr.io/sig-storage/hostpathplugin:v1.7.1],SizeBytes:29977921,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:a61d309da54641db41fb8f35718f744e9f730d4d0384f8c4b186ddc9f06cbd5f k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.1.0],SizeBytes:19662887,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:19582112,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:2dee3fe5fe861bb66c3a4ac51114f3447a4cd35870e0f2e2b558c7a400d89589 k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.2.0],SizeBytes:18699436,},ContainerImage{Names:[k8s.gcr.io/sig-storage/mock-driver@sha256:a7b517f9e0f42ffade802eef9cefa271372386b85f55b702b493241e58459793 k8s.gcr.io/sig-storage/mock-driver:v4.1.0],SizeBytes:17680993,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:17611980,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:529be2c9770add0cdd0c989115222ea9fc1be430c11095eb9f6dafcf98a36e2b k8s.gcr.io/sig-storage/livenessprobe:v2.4.0],SizeBytes:17175488,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:13616070e3f29de4417eee434a8ef472221c9e51b3d037b5a6b46cef08eb7443 k8s.gcr.io/e2e-test-images/nginx:1.14-2],SizeBytes:16032814,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nonewprivs@sha256:8ac1264691820febacf3aea5d152cbde6d10685731ec14966a9401c6f47a68ac k8s.gcr.io/e2e-test-images/nonewprivs:1.3],SizeBytes:7107254,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf k8s.gcr.io/e2e-test-images/busybox:1.29-2],SizeBytes:1154361,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:39e1e963e5310e9c313bad51523be012ede7b35bb9316517d19089a010356592 k8s.gcr.io/e2e-test-images/busybox:1.29-1],SizeBytes:1154361,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db k8s.gcr.io/pause:3.6 registry.k8s.io/pause:3.6],SizeBytes:682696,},},VolumesInUse:[kubernetes.io/csi/ebs.csi.aws.com^vol-06765546f5213e581 kubernetes.io/csi/ebs.csi.aws.com^vol-0bf141bae4d3e4140],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-0bf141bae4d3e4140,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-06765546f5213e581,DevicePath:,},},Config:nil,},} Jan 1 02:48:29.301: INFO: Logging kubelet events for node ip-172-20-33-66.eu-west-3.compute.internal Jan 1 02:48:29.407: INFO: Logging pods the kubelet thinks is on node ip-172-20-33-66.eu-west-3.compute.internal Jan 1 02:48:29.518: INFO: test-container-pod started at 2023-01-01 02:48:24 +0000 UTC (0+1 container statuses recorded) Jan 1 02:48:29.518: INFO: Container webserver ready: true, restart count 0 Jan 1 02:48:29.518: INFO: netserver-0 started at 2023-01-01 02:47:59 +0000 UTC (0+1 container statuses recorded) Jan 1 02:48:29.518: INFO: Container webserver ready: true, restart count 0 Jan 1 02:48:29.518: INFO: coredns-867df8f45c-cqmp7 started at 2023-01-01 02:21:18 +0000 UTC (0+1 container statuses recorded) Jan 1 02:48:29.518: INFO: Container coredns ready: true, restart count 0 Jan 1 02:48:29.518: INFO: pod-subpath-test-inlinevolume-wkb5 started at 2023-01-01 02:48:28 +0000 UTC (1+1 container statuses recorded) Jan 1 02:48:29.518: INFO: Init container init-volume-inlinevolume-wkb5 ready: false, restart count 0 Jan 1 02:48:29.518: INFO: Container test-container-subpath-inlinevolume-wkb5 ready: false, restart count 0 Jan 1 02:48:29.518: INFO: netserver-0 started at 2023-01-01 02:48:17 +0000 UTC (0+1 container statuses recorded) Jan 1 02:48:29.518: INFO: Container webserver ready: false, restart count 0 Jan 1 02:48:29.518: INFO: pod-ephm-test-projected-klsf started at 2023-01-01 02:47:55 +0000 UTC (0+1 container statuses recorded) Jan 1 02:48:29.518: INFO: Container test-container-subpath-projected-klsf ready: false, restart count 0 Jan 1 02:48:29.518: INFO: inline-volume-tester2-hcvdh started at 2023-01-01 02:47:53 +0000 UTC (0+1 container statuses recorded) Jan 1 02:48:29.518: INFO: Container csi-volume-tester ready: true, restart count 0 Jan 1 02:48:29.518: INFO: ebs-csi-node-vsxjc started at 2023-01-01 02:20:26 +0000 UTC (0+3 container statuses recorded) Jan 1 02:48:29.518: INFO: Container ebs-plugin ready: true, restart count 0 Jan 1 02:48:29.518: INFO: Container liveness-probe ready: true, restart count 0 Jan 1 02:48:29.518: INFO: Container node-driver-registrar ready: true, restart count 0 Jan 1 02:48:29.518: INFO: webserver-pod started at 2023-01-01 02:43:28 +0000 UTC (0+1 container statuses recorded) Jan 1 02:48:29.518: INFO: Container agnhost ready: false, restart count 0 Jan 1 02:48:29.518: INFO: cilium-vbcgc started at 2023-01-01 02:20:26 +0000 UTC (1+1 container statuses recorded) Jan 1 02:48:29.518: INFO: Init container clean-cilium-state ready: true, restart count 0 Jan 1 02:48:29.518: INFO: Container cilium-agent ready: true, restart count 0 Jan 1 02:48:29.518: INFO: pod-should-be-evicted46f88c5a-f01f-4b72-94c9-0cddd928f146 started at 2023-01-01 02:43:27 +0000 UTC (0+1 container statuses recorded) Jan 1 02:48:29.518: INFO: Container bar ready: true, restart count 0 Jan 1 02:48:29.917: INFO: Latency metrics for node ip-172-20-33-66.eu-west-3.compute.internal Jan 1 02:48:29.918: INFO: Logging node info for node ip-172-20-38-238.eu-west-3.compute.internal Jan 1 02:48:30.026: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-38-238.eu-west-3.compute.internal 0eb713b4-64bf-47b7-8090-bab8111218ee 39267 0 2023-01-01 02:18:51 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:c5.large beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:eu-west-3 failure-domain.beta.kubernetes.io/zone:eu-west-3a kops.k8s.io/kops-controller-pki: kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-38-238.eu-west-3.compute.internal kubernetes.io/os:linux kubernetes.io/role:master node-role.kubernetes.io/control-plane: node-role.kubernetes.io/master: node.kubernetes.io/exclude-from-external-load-balancers: node.kubernetes.io/instance-type:c5.large topology.ebs.csi.aws.com/zone:eu-west-3a topology.kubernetes.io/region:eu-west-3 topology.kubernetes.io/zone:eu-west-3a] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-0f70ebb30a9076648"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kubelet Update v1 2023-01-01 02:18:50 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {protokube Update v1 2023-01-01 02:19:16 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kops.k8s.io/kops-controller-pki":{},"f:node-role.kubernetes.io/control-plane":{},"f:node.kubernetes.io/exclude-from-external-load-balancers":{}}}} } {kops-controller Update v1 2023-01-01 02:19:21 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/master":{}}}} } {kube-controller-manager Update v1 2023-01-01 02:19:31 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.0.0/24\"":{}},"f:taints":{}}} } {kubelet Update v1 2023-01-01 02:20:36 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.0.0/24,DoNotUseExternalID:,ProviderID:aws:///eu-west-3a/i-0f70ebb30a9076648,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/master,Value:,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[100.96.0.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{50478186496 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3918811136 0} {<nil>} 3826964Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{45430367772 0} {<nil>} 45430367772 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3813953536 0} {<nil>} 3724564Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-01-01 02:46:44 +0000 UTC,LastTransitionTime:2023-01-01 02:18:45 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-01-01 02:46:44 +0000 UTC,LastTransitionTime:2023-01-01 02:18:45 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-01-01 02:46:44 +0000 UTC,LastTransitionTime:2023-01-01 02:18:45 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-01-01 02:46:44 +0000 UTC,LastTransitionTime:2023-01-01 02:19:31 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.38.238,},NodeAddress{Type:ExternalIP,Address:15.237.26.58,},NodeAddress{Type:Hostname,Address:ip-172-20-38-238.eu-west-3.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-38-238.eu-west-3.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-15-237-26-58.eu-west-3.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2e6c64b5b811bf0c78b2aa7ccdb943,SystemUUID:ec2e6c64-b5b8-11bf-0c78-b2aa7ccdb943,BootID:9db01fa3-d136-4230-b21a-b03abbfb6bff,KernelVersion:4.19.0-23-cloud-amd64,OSImage:Debian GNU/Linux 10 (buster),ContainerRuntimeVersion:docker://20.10.17,KubeletVersion:v1.23.15,KubeProxyVersion:v1.23.15,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[registry.k8s.io/etcdadm/etcd-manager@sha256:66a453db625abb268f4b3bbefc5a34a171d81e6e8796cecca54cfd71775c77c4 registry.k8s.io/etcdadm/etcd-manager:v3.0.20221209],SizeBytes:662209237,},ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:456569528,},ContainerImage{Names:[k8s.gcr.io/kube-apiserver-amd64:v1.23.15 registry.k8s.io/kube-apiserver-amd64:v1.23.15],SizeBytes:135240444,},ContainerImage{Names:[k8s.gcr.io/kube-controller-manager-amd64:v1.23.15 registry.k8s.io/kube-controller-manager-amd64:v1.23.15],SizeBytes:125046097,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.23.15 registry.k8s.io/kube-proxy-amd64:v1.23.15],SizeBytes:112336036,},ContainerImage{Names:[quay.io/cilium/operator@sha256:a6d24a006a6b92967ac90786b49bc1ac26e5477cf028cd1186efcfc2466484db quay.io/cilium/operator:v1.12.5],SizeBytes:110016956,},ContainerImage{Names:[registry.k8s.io/kops/kops-controller:1.27.0-alpha.1],SizeBytes:105197757,},ContainerImage{Names:[registry.k8s.io/kops/dns-controller:1.27.0-alpha.1],SizeBytes:102577531,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:10e231cf07c82dfab5141c1afe548dc734e5fc1f67665ec3982a325d1bd31a9a registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.0],SizeBytes:94471559,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:122bfb8c1edabb3c0edd63f06523e6940d958d19b3957dc7b1d6f81e9f1f6119 registry.k8s.io/sig-storage/csi-provisioner:v3.1.0],SizeBytes:57736070,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:9ebbf9f023e7b41ccee3d52afe39a89e3ddacdbb69269d583abfc25847cfd9e4 registry.k8s.io/sig-storage/csi-resizer:v1.4.0],SizeBytes:55464778,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:8b9c313c05f54fb04f8d430896f5f5904b6cb157df261501b29adc04d2b2dc7b registry.k8s.io/sig-storage/csi-attacher:v3.4.0],SizeBytes:54848595,},ContainerImage{Names:[k8s.gcr.io/kube-scheduler-amd64:v1.23.15 registry.k8s.io/kube-scheduler-amd64:v1.23.15],SizeBytes:53521430,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:19582112,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:17611980,},ContainerImage{Names:[registry.k8s.io/kops/kube-apiserver-healthcheck:1.27.0-alpha.1],SizeBytes:8787111,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:682696,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Jan 1 02:48:30.027: INFO: Logging kubelet events for node ip-172-20-38-238.eu-west-3.compute.internal Jan 1 02:48:30.132: INFO: Logging pods the kubelet thinks is on node ip-172-20-38-238.eu-west-3.compute.internal Jan 1 02:48:30.248: INFO: kube-controller-manager-ip-172-20-38-238.eu-west-3.compute.internal started at 2023-01-01 02:18:11 +0000 UTC (0+1 container statuses recorded) Jan 1 02:48:30.248: INFO: Container kube-controller-manager ready: true, restart count 2 Jan 1 02:48:30.248: INFO: kube-scheduler-ip-172-20-38-238.eu-west-3.compute.internal started at 2023-01-01 02:18:11 +0000 UTC (0+1 container statuses recorded) Jan 1 02:48:30.248: INFO: Container kube-scheduler ready: true, restart count 0 Jan 1 02:48:30.248: INFO: kube-apiserver-ip-172-20-38-238.eu-west-3.compute.internal started at 2023-01-01 02:18:11 +0000 UTC (0+2 container statuses recorded) Jan 1 02:48:30.248: INFO: Container healthcheck ready: true, restart count 0 Jan 1 02:48:30.248: INFO: Container kube-apiserver ready: true, restart count 1 Jan 1 02:48:30.248: INFO: kops-controller-26gzg started at 2023-01-01 02:19:20 +0000 UTC (0+1 container statuses recorded) Jan 1 02:48:30.248: INFO: Container kops-controller ready: true, restart count 0 Jan 1 02:48:30.248: INFO: ebs-csi-controller-6dc545b75-5qqh8 started at 2023-01-01 02:19:20 +0000 UTC (0+5 container statuses recorded) Jan 1 02:48:30.248: INFO: Container csi-attacher ready: true, restart count 0 Jan 1 02:48:30.248: INFO: Container csi-provisioner ready: true, restart count 0 Jan 1 02:48:30.248: INFO: Container csi-resizer ready: true, restart count 0 Jan 1 02:48:30.248: INFO: Container ebs-plugin ready: true, restart count 0 Jan 1 02:48:30.248: INFO: Container liveness-probe ready: true, restart count 0 Jan 1 02:48:30.248: INFO: etcd-manager-events-ip-172-20-38-238.eu-west-3.compute.internal started at 2023-01-01 02:18:11 +0000 UTC (0+1 container statuses recorded) Jan 1 02:48:30.248: INFO: Container etcd-manager ready: true, restart count 0 Jan 1 02:48:30.248: INFO: etcd-manager-main-ip-172-20-38-238.eu-west-3.compute.internal started at 2023-01-01 02:18:11 +0000 UTC (0+1 container statuses recorded) Jan 1 02:48:30.248: INFO: Container etcd-manager ready: true, restart count 0 Jan 1 02:48:30.248: INFO: cilium-qk59v started at 2023-01-01 02:19:20 +0000 UTC (1+1 container statuses recorded) Jan 1 02:48:30.248: INFO: Init container clean-cilium-state ready: true, restart count 0 Jan 1 02:48:30.248: INFO: Container cilium-agent ready: true, restart count 0 Jan 1 02:48:30.248: INFO: ebs-csi-node-k87k6 started at 2023-01-01 02:19:20 +0000 UTC (0+3 container statuses recorded) Jan 1 02:48:30.248: INFO: Container ebs-plugin ready: true, restart count 0 Jan 1 02:48:30.248: INFO: Container liveness-probe ready: true, restart count 0 Jan 1 02:48:30.248: INFO: Container node-driver-registrar ready: true, restart count 0 Jan 1 02:48:30.248: INFO: dns-controller-7db8b74b99-58zzw started at 2023-01-01 02:19:20 +0000 UTC (0+1 container statuses recorded) Jan 1 02:48:30.248: INFO: Container dns-controller ready: true, restart count 0 Jan 1 02:48:30.248: INFO: cilium-operator-d47487c76-j84nq started at 2023-01-01 02:19:20 +0000 UTC (0+1 container statuses recorded) Jan 1 02:48:30.248: INFO: Container cilium-operator ready: true, restart count 0 Jan 1 02:48:30.578: INFO: Latency metrics for node ip-172-20-38-238.eu-west-3.compute.internal Jan 1 02:48:30.579: INFO: Logging node info for node ip-172-20-45-112.eu-west-3.compute.internal Jan 1 02:48:30.686: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-45-112.eu-west-3.compute.internal 48eed937-05bb-4d8a-bddd-aa94a2e6ab92 43751 0 2023-01-01 02:20:27 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:eu-west-3 failure-domain.beta.kubernetes.io/zone:eu-west-3a kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-45-112.eu-west-3.compute.internal kubernetes.io/os:linux kubernetes.io/role:node node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:eu-west-3a topology.hostpath.csi/node:ip-172-20-45-112.eu-west-3.compute.internal topology.kubernetes.io/region:eu-west-3 topology.kubernetes.io/zone:eu-west-3a] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-0778adf2085fa7f34"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kube-controller-manager Update v1 2023-01-01 02:20:27 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.3.0/24\"":{}}}} } {kubelet Update v1 2023-01-01 02:20:27 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {kops-controller Update v1 2023-01-01 02:20:28 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}} } {kubelet Update v1 2023-01-01 02:46:11 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:capacity":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status} {kube-controller-manager Update v1 2023-01-01 02:47:58 +0000 UTC FieldsV1 {"f:status":{"f:volumesAttached":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.3.0/24,DoNotUseExternalID:,ProviderID:aws:///eu-west-3a/i-0778adf2085fa7f34,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.3.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{50478186496 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4078190592 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{45430367772 0} {<nil>} 45430367772 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3973332992 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-01-01 02:48:25 +0000 UTC,LastTransitionTime:2023-01-01 02:20:27 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-01-01 02:48:25 +0000 UTC,LastTransitionTime:2023-01-01 02:20:27 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-01-01 02:48:25 +0000 UTC,LastTransitionTime:2023-01-01 02:20:27 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-01-01 02:48:25 +0000 UTC,LastTransitionTime:2023-01-01 02:20:48 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.45.112,},NodeAddress{Type:ExternalIP,Address:13.38.32.14,},NodeAddress{Type:Hostname,Address:ip-172-20-45-112.eu-west-3.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-45-112.eu-west-3.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-13-38-32-14.eu-west-3.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2dbdf846e8ff614f4555e5bf9dbbf4,SystemUUID:ec2dbdf8-46e8-ff61-4f45-55e5bf9dbbf4,BootID:10a054ae-621a-47ce-9566-b0048cc5101b,KernelVersion:4.19.0-23-cloud-amd64,OSImage:Debian GNU/Linux 10 (buster),ContainerRuntimeVersion:docker://20.10.17,KubeletVersion:v1.23.15,KubeProxyVersion:v1.23.15,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:456569528,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/jessie-dnsutils@sha256:11e6a66017ba4e4b938c1612b7a54a3befcefd354796c04e1dba76873a13518e k8s.gcr.io/e2e-test-images/jessie-dnsutils:1.5],SizeBytes:253371792,},ContainerImage{Names:[nginx@sha256:0047b729188a15da49380d9506d65959cce6d40291ccfb4e039f5dc7efd33286 nginx:latest],SizeBytes:141812353,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:20f25f275d46aa728f7615a1ccc19c78b2ed89435bf943a44b339f70f45508e6 k8s.gcr.io/e2e-test-images/httpd:2.4.39-2],SizeBytes:126894770,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:7e8bdd271312fd25fc5ff5a8f04727be84044eb3d7d8d03611972a6752e2e11e k8s.gcr.io/e2e-test-images/agnhost:2.39],SizeBytes:126872991,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:1b9d1b2f36cb2dbee1960e82a9344aeb11bd4c4c03abf5e1853e0559c23855e3 k8s.gcr.io/e2e-test-images/httpd:2.4.38-2],SizeBytes:123781643,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.23.15 registry.k8s.io/kube-proxy-amd64:v1.23.15],SizeBytes:112336036,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:10e231cf07c82dfab5141c1afe548dc734e5fc1f67665ec3982a325d1bd31a9a registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.0],SizeBytes:94471559,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:c8e03f60afa90a28e4bb6ec9a8d0fc36d89de4b7475cf2d613afa793ec969fe0 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.0],SizeBytes:56382329,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:4e74c0492bceddc598de1c90cc5bc14dcda94cb49fa9c5bad9d117c4834b5e08 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.1],SizeBytes:56379269,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:36c31f7e1f433c9634d24f876353e8646246d81a03c4e351202c2644daff1620 k8s.gcr.io/sig-storage/csi-resizer:v1.2.0],SizeBytes:53956308,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:ed98431376c9e944e19a465fe8ea944806714dd95416a0821096c78d66b579bd k8s.gcr.io/sig-storage/csi-snapshotter:v4.1.1],SizeBytes:53531990,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:c5be65d6679efabb969d9b019300d187437ae876f992c40911fd2892bbef3b36 k8s.gcr.io/sig-storage/csi-attacher:v3.2.0],SizeBytes:53500826,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:60ab9b3e6a030d3038c87c0d6bca2930f58d1d72823e6a4af09767dc83b696a2 k8s.gcr.io/sig-storage/csi-attacher:v3.2.1],SizeBytes:53499570,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:7a5ba58a44e0d749e0767e4e37315bcf6a61f33ce3185c1991848af4db0fb70a k8s.gcr.io/sig-storage/csi-resizer:v1.1.0],SizeBytes:49176472,},ContainerImage{Names:[k8s.gcr.io/sig-storage/hostpathplugin@sha256:232fe80174d60d520d36043103853a1d7ab4b7f3782cf43e45034f04ccda58ce k8s.gcr.io/sig-storage/hostpathplugin:v1.7.1],SizeBytes:29977921,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:a61d309da54641db41fb8f35718f744e9f730d4d0384f8c4b186ddc9f06cbd5f k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.1.0],SizeBytes:19662887,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:19582112,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:2dee3fe5fe861bb66c3a4ac51114f3447a4cd35870e0f2e2b558c7a400d89589 k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.2.0],SizeBytes:18699436,},ContainerImage{Names:[k8s.gcr.io/sig-storage/mock-driver@sha256:a7b517f9e0f42ffade802eef9cefa271372386b85f55b702b493241e58459793 k8s.gcr.io/sig-storage/mock-driver:v4.1.0],SizeBytes:17680993,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:17611980,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:529be2c9770add0cdd0c989115222ea9fc1be430c11095eb9f6dafcf98a36e2b k8s.gcr.io/sig-storage/livenessprobe:v2.4.0],SizeBytes:17175488,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:13616070e3f29de4417eee434a8ef472221c9e51b3d037b5a6b46cef08eb7443 k8s.gcr.io/e2e-test-images/nginx:1.14-2],SizeBytes:16032814,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nonewprivs@sha256:8ac1264691820febacf3aea5d152cbde6d10685731ec14966a9401c6f47a68ac k8s.gcr.io/e2e-test-images/nonewprivs:1.3],SizeBytes:7107254,},ContainerImage{Names:[gcr.io/authenticated-image-pulling/alpine@sha256:7ff177862cb50c602bfe81f805969412e619c054a2bbead977d0c276988aa4a0 gcr.io/authenticated-image-pulling/alpine:3.7],SizeBytes:4206620,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf k8s.gcr.io/e2e-test-images/busybox:1.29-2],SizeBytes:1154361,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:39e1e963e5310e9c313bad51523be012ede7b35bb9316517d19089a010356592 k8s.gcr.io/e2e-test-images/busybox:1.29-1],SizeBytes:1154361,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db k8s.gcr.io/pause:3.6 registry.k8s.io/pause:3.6],SizeBytes:682696,},},VolumesInUse:[kubernetes.io/csi/ebs.csi.aws.com^vol-05e6b795b64c06b6b],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-05e6b795b64c06b6b,DevicePath:,},},Config:nil,},} Jan 1 02:48:30.686: INFO: Logging kubelet events for node ip-172-20-45-112.eu-west-3.compute.internal Jan 1 02:48:30.791: INFO: Logging pods the kubelet thinks is on node ip-172-20-45-112.eu-west-3.compute.internal Jan 1 02:48:30.901: INFO: pvc-tester-kc549 started at 2023-01-01 02:48:23 +0000 UTC (0+1 container statuses recorded) Jan 1 02:48:30.902: INFO: Container write-pod ready: false, restart count 0 Jan 1 02:48:30.902: INFO: ebs-csi-node-7clrr started at 2023-01-01 02:20:28 +0000 UTC (0+3 container statuses recorded) Jan 1 02:48:30.902: INFO: Container ebs-plugin ready: true, restart count 0 Jan 1 02:48:30.902: INFO: Container liveness-probe ready: true, restart count 0 Jan 1 02:48:30.902: INFO: Container node-driver-registrar ready: true, restart count 0 Jan 1 02:48:30.902: INFO: logs-generator started at 2023-01-01 02:48:11 +0000 UTC (0+1 container statuses recorded) Jan 1 02:48:30.902: INFO: Container logs-generator ready: true, restart count 0 Jan 1 02:48:30.902: INFO: pod-3616e4df-4e32-4dea-9ac1-6199f29b4d91 started at <nil> (0+0 container statuses recorded) Jan 1 02:48:30.902: INFO: netserver-1 started at 2023-01-01 02:47:59 +0000 UTC (0+1 container statuses recorded) Jan 1 02:48:30.902: INFO: Container webserver ready: true, restart count 0 Jan 1 02:48:30.902: INFO: netserver-1 started at 2023-01-01 02:48:17 +0000 UTC (0+1 container statuses recorded) Jan 1 02:48:30.902: INFO: Container webserver ready: false, restart count 0 Jan 1 02:48:30.902: INFO: pod-configmaps-0b39cb62-23d3-4ff0-a9d5-a679ff32e71c started at 2023-01-01 02:48:21 +0000 UTC (0+1 container statuses recorded) Jan 1 02:48:30.902: INFO: Container configmap-volume-test ready: false, restart count 0 Jan 1 02:48:30.902: INFO: cilium-slx98 started at 2023-01-01 02:20:28 +0000 UTC (1+1 container statuses recorded) Jan 1 02:48:30.902: INFO: Init container clean-cilium-state ready: true, restart count 0 Jan 1 02:48:30.902: INFO: Container cilium-agent ready: true, restart count 0 Jan 1 02:48:30.902: INFO: pod-c165f026-39f8-479a-af36-6fa50c4706cc started at 2023-01-01 02:48:22 +0000 UTC (0+1 container statuses recorded) Jan 1 02:48:30.902: INFO: Container test-container ready: false, restart count 0 Jan 1 02:48:30.902: INFO: probe-test-4c0a1663-4d3a-479b-b77b-6057ccfcb3d3 started at 2023-01-01 02:48:01 +0000 UTC (0+1 container statuses recorded) Jan 1 02:48:30.902: INFO: Container probe-test-4c0a1663-4d3a-479b-b77b-6057ccfcb3d3 ready: false, restart count 0 Jan 1 02:48:30.902: INFO: test-pod started at <nil> (0+0 container statuses recorded) Jan 1 02:48:32.065: INFO: Latency metrics for node ip-172-20-45-112.eu-west-3.compute.internal Jan 1 02:48:32.065: INFO: Logging node info for node ip-172-20-47-178.eu-west-3.compute.internal Jan 1 02:48:32.167: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-47-178.eu-west-3.compute.internal 835e5337-d2b9-45ad-bf0e-9d32514f71ba 43862 0 2023-01-01 02:20:25 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:eu-west-3 failure-domain.beta.kubernetes.io/zone:eu-west-3a kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-47-178.eu-west-3.compute.internal kubernetes.io/os:linux kubernetes.io/role:node node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:eu-west-3a topology.hostpath.csi/node:ip-172-20-47-178.eu-west-3.compute.internal topology.kubernetes.io/region:eu-west-3 topology.kubernetes.io/zone:eu-west-3a] map[csi.volume.kubernetes.io/nodeid:{"csi-hostpath-ephemeral-2363":"ip-172-20-47-178.eu-west-3.compute.internal","csi-hostpath-volume-3958":"ip-172-20-47-178.eu-west-3.compute.internal","ebs.csi.aws.com":"i-00e1e60199f5a4d04"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kops-controller Update v1 2023-01-01 02:20:25 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}} } {kube-controller-manager Update v1 2023-01-01 02:20:25 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.2.0/24\"":{}}}} } {kubelet Update v1 2023-01-01 02:20:25 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {kube-controller-manager Update v1 2023-01-01 02:47:35 +0000 UTC FieldsV1 {"f:status":{"f:volumesAttached":{}}} status} {kubelet Update v1 2023-01-01 02:47:40 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:capacity":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.2.0/24,DoNotUseExternalID:,ProviderID:aws:///eu-west-3a/i-00e1e60199f5a4d04,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.2.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{50478186496 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4078190592 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{45430367772 0} {<nil>} 45430367772 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3973332992 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-01-01 02:48:31 +0000 UTC,LastTransitionTime:2023-01-01 02:20:25 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-01-01 02:48:31 +0000 UTC,LastTransitionTime:2023-01-01 02:20:25 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-01-01 02:48:31 +0000 UTC,LastTransitionTime:2023-01-01 02:20:25 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-01-01 02:48:31 +0000 UTC,LastTransitionTime:2023-01-01 02:20:46 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.47.178,},NodeAddress{Type:ExternalIP,Address:13.36.177.101,},NodeAddress{Type:Hostname,Address:ip-172-20-47-178.eu-west-3.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-47-178.eu-west-3.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-13-36-177-101.eu-west-3.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec243eda226c96d5573f3eaa3203df0d,SystemUUID:ec243eda-226c-96d5-573f-3eaa3203df0d,BootID:15dccaa7-d7c7-419d-b7de-76be5781a51a,KernelVersion:4.19.0-23-cloud-amd64,OSImage:Debian GNU/Linux 10 (buster),ContainerRuntimeVersion:docker://20.10.17,KubeletVersion:v1.23.15,KubeProxyVersion:v1.23.15,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:456569528,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/jessie-dnsutils@sha256:11e6a66017ba4e4b938c1612b7a54a3befcefd354796c04e1dba76873a13518e k8s.gcr.io/e2e-test-images/jessie-dnsutils:1.5],SizeBytes:253371792,},ContainerImage{Names:[nginx@sha256:0047b729188a15da49380d9506d65959cce6d40291ccfb4e039f5dc7efd33286 nginx:latest],SizeBytes:141812353,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:20f25f275d46aa728f7615a1ccc19c78b2ed89435bf943a44b339f70f45508e6 k8s.gcr.io/e2e-test-images/httpd:2.4.39-2],SizeBytes:126894770,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:7e8bdd271312fd25fc5ff5a8f04727be84044eb3d7d8d03611972a6752e2e11e k8s.gcr.io/e2e-test-images/agnhost:2.39],SizeBytes:126872991,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nautilus@sha256:99c0d6f1ad24a1aa1905d9c6534d193f268f7b23f9add2ae6bb41f31094bdd5c k8s.gcr.io/e2e-test-images/nautilus:1.5],SizeBytes:124758741,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:1b9d1b2f36cb2dbee1960e82a9344aeb11bd4c4c03abf5e1853e0559c23855e3 k8s.gcr.io/e2e-test-images/httpd:2.4.38-2],SizeBytes:123781643,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.23.15 registry.k8s.io/kube-proxy-amd64:v1.23.15],SizeBytes:112336036,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:10e231cf07c82dfab5141c1afe548dc734e5fc1f67665ec3982a325d1bd31a9a registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.0],SizeBytes:94471559,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:c8e03f60afa90a28e4bb6ec9a8d0fc36d89de4b7475cf2d613afa793ec969fe0 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.0],SizeBytes:56382329,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:4e74c0492bceddc598de1c90cc5bc14dcda94cb49fa9c5bad9d117c4834b5e08 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.1],SizeBytes:56379269,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:36c31f7e1f433c9634d24f876353e8646246d81a03c4e351202c2644daff1620 k8s.gcr.io/sig-storage/csi-resizer:v1.2.0],SizeBytes:53956308,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:ed98431376c9e944e19a465fe8ea944806714dd95416a0821096c78d66b579bd k8s.gcr.io/sig-storage/csi-snapshotter:v4.1.1],SizeBytes:53531990,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:c5be65d6679efabb969d9b019300d187437ae876f992c40911fd2892bbef3b36 k8s.gcr.io/sig-storage/csi-attacher:v3.2.0],SizeBytes:53500826,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:60ab9b3e6a030d3038c87c0d6bca2930f58d1d72823e6a4af09767dc83b696a2 k8s.gcr.io/sig-storage/csi-attacher:v3.2.1],SizeBytes:53499570,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:017727efcfeb7d053af68e51436ce8e65edbc6ca573720afb4f79c8594036955 registry.k8s.io/coredns/coredns:v1.10.0],SizeBytes:50249337,},ContainerImage{Names:[registry.k8s.io/cpa/cluster-proportional-autoscaler@sha256:68d396900aeaa072c1f27289485fdac29834045a6f3ffe369bf389d830ef572d registry.k8s.io/cpa/cluster-proportional-autoscaler:1.8.6],SizeBytes:47910609,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nonroot@sha256:b9e2958a3dd879e3cf11142228c6d073d0fc4ea2e857c3be6f4fb0ab5fb2c937 k8s.gcr.io/e2e-test-images/nonroot:1.2],SizeBytes:42321438,},ContainerImage{Names:[k8s.gcr.io/sig-storage/hostpathplugin@sha256:232fe80174d60d520d36043103853a1d7ab4b7f3782cf43e45034f04ccda58ce k8s.gcr.io/sig-storage/hostpathplugin:v1.7.1],SizeBytes:29977921,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:a61d309da54641db41fb8f35718f744e9f730d4d0384f8c4b186ddc9f06cbd5f k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.1.0],SizeBytes:19662887,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:19582112,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:2dee3fe5fe861bb66c3a4ac51114f3447a4cd35870e0f2e2b558c7a400d89589 k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.2.0],SizeBytes:18699436,},ContainerImage{Names:[k8s.gcr.io/sig-storage/mock-driver@sha256:a7b517f9e0f42ffade802eef9cefa271372386b85f55b702b493241e58459793 k8s.gcr.io/sig-storage/mock-driver:v4.1.0],SizeBytes:17680993,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:17611980,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:529be2c9770add0cdd0c989115222ea9fc1be430c11095eb9f6dafcf98a36e2b k8s.gcr.io/sig-storage/livenessprobe:v2.4.0],SizeBytes:17175488,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:13616070e3f29de4417eee434a8ef472221c9e51b3d037b5a6b46cef08eb7443 k8s.gcr.io/e2e-test-images/nginx:1.14-2],SizeBytes:16032814,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf k8s.gcr.io/e2e-test-images/busybox:1.29-2],SizeBytes:1154361,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db k8s.gcr.io/pause:3.6 registry.k8s.io/pause:3.6],SizeBytes:682696,},},VolumesInUse:[kubernetes.io/csi/csi-hostpath-volume-3958^af3f2105-897e-11ed-a663-0277093bc21f kubernetes.io/csi/ebs.csi.aws.com^vol-02700bde5d78619cd],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-02700bde5d78619cd,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/csi-hostpath-volume-3958^af3f2105-897e-11ed-a663-0277093bc21f,DevicePath:,},},Config:nil,},} Jan 1 02:48:32.168: INFO: Logging kubelet events for node ip-172-20-47-178.eu-west-3.compute.internal Jan 1 02:48:32.273: INFO: Logging pods the kubelet thinks is on node ip-172-20-47-178.eu-west-3.compute.internal Jan 1 02:48:32.383: INFO: inline-volume-tester-fs955 started at 2023-01-01 02:47:33 +0000 UTC (0+1 container statuses recorded) Jan 1 02:48:32.383: INFO: Container csi-volume-tester ready: true, restart count 0 Jan 1 02:48:32.383: INFO: cilium-kk2c2 started at 2023-01-01 02:20:26 +0000 UTC (1+1 container statuses recorded) Jan 1 02:48:32.383: INFO: Init container clean-cilium-state ready: true, restart count 0 Jan 1 02:48:32.383: INFO: Container cilium-agent ready: true, restart count 0 Jan 1 02:48:32.383: INFO: inline-volume-tester-vhq8k started at 2023-01-01 02:48:17 +0000 UTC (0+1 container statuses recorded) Jan 1 02:48:32.383: INFO: Container csi-volume-tester ready: true, restart count 0 Jan 1 02:48:32.383: INFO: coredns-867df8f45c-wzntb started at 2023-01-01 02:20:46 +0000 UTC (0+1 container statuses recorded) Jan 1 02:48:32.383: INFO: Container coredns ready: true, restart count 0 Jan 1 02:48:32.383: INFO: netserver-2 started at 2023-01-01 02:47:59 +0000 UTC (0+1 container statuses recorded) Jan 1 02:48:32.383: INFO: Container webserver ready: true, restart count 0 Jan 1 02:48:32.383: INFO: pod-subpath-test-inlinevolume-vgz2 started at 2023-01-01 02:48:28 +0000 UTC (0+2 container statuses recorded) Jan 1 02:48:32.383: INFO: Container test-container-subpath-inlinevolume-vgz2 ready: false, restart count 0 Jan 1 02:48:32.383: INFO: Container test-container-volume-inlinevolume-vgz2 ready: true, restart count 0 Jan 1 02:48:32.383: INFO: coredns-autoscaler-557ccb4c66-k4lzb started at 2023-01-01 02:20:46 +0000 UTC (0+1 container statuses recorded) Jan 1 02:48:32.383: INFO: Container autoscaler ready: true, restart count 0 Jan 1 02:48:32.384: INFO: csi-hostpathplugin-0 started at 2023-01-01 02:48:17 +0000 UTC (0+7 container statuses recorded) Jan 1 02:48:32.384: INFO: Container csi-attacher ready: true, restart count 0 Jan 1 02:48:32.384: INFO: Container csi-provisioner ready: true, restart count 0 Jan 1 02:48:32.384: INFO: Container csi-resizer ready: true, restart count 0 Jan 1 02:48:32.384: INFO: Container csi-snapshotter ready: true, restart count 0 Jan 1 02:48:32.384: INFO: Container hostpath ready: true, restart count 0 Jan 1 02:48:32.384: INFO: Container liveness-probe ready: true, restart count 0 Jan 1 02:48:32.384: INFO: Container node-driver-registrar ready: true, restart count 0 Jan 1 02:48:32.384: INFO: hostpath-client started at 2023-01-01 02:48:12 +0000 UTC (0+1 container statuses recorded) Jan 1 02:48:32.384: INFO: Container hostpath-client ready: true, restart count 0 Jan 1 02:48:32.384: INFO: csi-hostpathplugin-0 started at 2023-01-01 02:47:47 +0000 UTC (0+7 container statuses recorded) Jan 1 02:48:32.384: INFO: Container csi-attacher ready: true, restart count 0 Jan 1 02:48:32.384: INFO: Container csi-provisioner ready: true, restart count 0 Jan 1 02:48:32.384: INFO: Container csi-resizer ready: true, restart count 0 Jan 1 02:48:32.384: INFO: Container csi-snapshotter ready: true, restart count 0 Jan 1 02:48:32.384: INFO: Container hostpath ready: true, restart count 0 Jan 1 02:48:32.384: INFO: Container liveness-probe ready: true, restart count 0 Jan 1 02:48:32.384: INFO: Container node-driver-registrar ready: true, restart count 0 Jan 1 02:48:32.384: INFO: netserver-2 started at 2023-01-01 02:48:17 +0000 UTC (0+1 container statuses recorded) Jan 1 02:48:32.384: INFO: Container webserver ready: false, restart count 0 Jan 1 02:48:32.384: INFO: ebs-csi-node-vcprc started at 2023-01-01 02:20:26 +0000 UTC (0+3 container statuses recorded) Jan 1 02:48:32.384: INFO: Container ebs-plugin ready: true, restart count 0 Jan 1 02:48:32.384: INFO: Container liveness-probe ready: true, restart count 0 Jan 1 02:48:32.384: INFO: Container node-driver-registrar ready: true, restart count 0 Jan 1 02:48:32.872: INFO: Latency metrics for node ip-172-20-47-178.eu-west-3.compute.internal Jan 1 02:48:32.872: INFO: Logging node info for node ip-172-20-49-224.eu-west-3.compute.internal Jan 1 02:48:32.999: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-49-224.eu-west-3.compute.internal d7d57146-c54a-4155-94f8-0f322216bfcf 42954 0 2023-01-01 02:20:28 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:eu-west-3 failure-domain.beta.kubernetes.io/zone:eu-west-3a kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-49-224.eu-west-3.compute.internal kubernetes.io/os:linux kubernetes.io/role:node node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:eu-west-3a topology.hostpath.csi/node:ip-172-20-49-224.eu-west-3.compute.internal topology.kubernetes.io/region:eu-west-3 topology.kubernetes.io/zone:eu-west-3a] map[csi.volume.kubernetes.io/nodeid:{"csi-mock-csi-mock-volumes-8997":"csi-mock-csi-mock-volumes-8997","ebs.csi.aws.com":"i-089a107939b27c28a"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kops-controller Update v1 2023-01-01 02:20:28 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}} } {kube-controller-manager Update v1 2023-01-01 02:20:28 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.4.0/24\"":{}}}} } {kubelet Update v1 2023-01-01 02:20:28 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {kube-controller-manager Update v1 2023-01-01 02:48:02 +0000 UTC FieldsV1 {"f:status":{"f:volumesAttached":{}}} status} {kubelet Update v1 2023-01-01 02:48:09 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:capacity":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.4.0/24,DoNotUseExternalID:,ProviderID:aws:///eu-west-3a/i-089a107939b27c28a,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.4.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{50478186496 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4078206976 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{45430367772 0} {<nil>} 45430367772 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3973349376 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-01-01 02:48:09 +0000 UTC,LastTransitionTime:2023-01-01 02:20:28 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-01-01 02:48:09 +0000 UTC,LastTransitionTime:2023-01-01 02:20:28 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-01-01 02:48:09 +0000 UTC,LastTransitionTime:2023-01-01 02:20:28 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-01-01 02:48:09 +0000 UTC,LastTransitionTime:2023-01-01 02:20:48 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.49.224,},NodeAddress{Type:ExternalIP,Address:15.237.95.101,},NodeAddress{Type:Hostname,Address:ip-172-20-49-224.eu-west-3.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-49-224.eu-west-3.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-15-237-95-101.eu-west-3.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2d0a1bb33dd26f5a12c3896530131b,SystemUUID:ec2d0a1b-b33d-d26f-5a12-c3896530131b,BootID:9f64e91d-26f0-45aa-aec4-a54433e67034,KernelVersion:4.19.0-23-cloud-amd64,OSImage:Debian GNU/Linux 10 (buster),ContainerRuntimeVersion:docker://20.10.17,KubeletVersion:v1.23.15,KubeProxyVersion:v1.23.15,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:456569528,},ContainerImage{Names:[k8s.gcr.io/etcd@sha256:dd75ec974b0a2a6f6bb47001ba09207976e625db898d1b16735528c009cb171c k8s.gcr.io/etcd:3.5.6-0],SizeBytes:299475478,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/jessie-dnsutils@sha256:11e6a66017ba4e4b938c1612b7a54a3befcefd354796c04e1dba76873a13518e k8s.gcr.io/e2e-test-images/jessie-dnsutils:1.5],SizeBytes:253371792,},ContainerImage{Names:[nginx@sha256:0047b729188a15da49380d9506d65959cce6d40291ccfb4e039f5dc7efd33286 nginx:latest],SizeBytes:141812353,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:20f25f275d46aa728f7615a1ccc19c78b2ed89435bf943a44b339f70f45508e6 k8s.gcr.io/e2e-test-images/httpd:2.4.39-2],SizeBytes:126894770,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:7e8bdd271312fd25fc5ff5a8f04727be84044eb3d7d8d03611972a6752e2e11e k8s.gcr.io/e2e-test-images/agnhost:2.39],SizeBytes:126872991,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:1b9d1b2f36cb2dbee1960e82a9344aeb11bd4c4c03abf5e1853e0559c23855e3 k8s.gcr.io/e2e-test-images/httpd:2.4.38-2],SizeBytes:123781643,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.23.15 registry.k8s.io/kube-proxy-amd64:v1.23.15],SizeBytes:112336036,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:10e231cf07c82dfab5141c1afe548dc734e5fc1f67665ec3982a325d1bd31a9a registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.0],SizeBytes:94471559,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/sample-apiserver@sha256:f9c93b92b6ff750b41a93c4e4fe0bfe384597aeb841e2539d5444815c55b2d8f k8s.gcr.io/e2e-test-images/sample-apiserver:1.17.5],SizeBytes:56504541,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:c8e03f60afa90a28e4bb6ec9a8d0fc36d89de4b7475cf2d613afa793ec969fe0 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.0],SizeBytes:56382329,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:4e74c0492bceddc598de1c90cc5bc14dcda94cb49fa9c5bad9d117c4834b5e08 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.1],SizeBytes:56379269,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:36c31f7e1f433c9634d24f876353e8646246d81a03c4e351202c2644daff1620 k8s.gcr.io/sig-storage/csi-resizer:v1.2.0],SizeBytes:53956308,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:ed98431376c9e944e19a465fe8ea944806714dd95416a0821096c78d66b579bd k8s.gcr.io/sig-storage/csi-snapshotter:v4.1.1],SizeBytes:53531990,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:c5be65d6679efabb969d9b019300d187437ae876f992c40911fd2892bbef3b36 k8s.gcr.io/sig-storage/csi-attacher:v3.2.0],SizeBytes:53500826,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:60ab9b3e6a030d3038c87c0d6bca2930f58d1d72823e6a4af09767dc83b696a2 k8s.gcr.io/sig-storage/csi-attacher:v3.2.1],SizeBytes:53499570,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:7a5ba58a44e0d749e0767e4e37315bcf6a61f33ce3185c1991848af4db0fb70a k8s.gcr.io/sig-storage/csi-resizer:v1.1.0],SizeBytes:49176472,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/regression-issue-74839@sha256:b4f1d8d61bdad84bd50442d161d5460e4019d53e989b64220fdbc62fc87d76bf k8s.gcr.io/e2e-test-images/regression-issue-74839:1.2],SizeBytes:44576952,},ContainerImage{Names:[k8s.gcr.io/sig-storage/hostpathplugin@sha256:232fe80174d60d520d36043103853a1d7ab4b7f3782cf43e45034f04ccda58ce k8s.gcr.io/sig-storage/hostpathplugin:v1.7.1],SizeBytes:29977921,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:a61d309da54641db41fb8f35718f744e9f730d4d0384f8c4b186ddc9f06cbd5f k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.1.0],SizeBytes:19662887,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:19582112,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:2dee3fe5fe861bb66c3a4ac51114f3447a4cd35870e0f2e2b558c7a400d89589 k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.2.0],SizeBytes:18699436,},ContainerImage{Names:[k8s.gcr.io/sig-storage/mock-driver@sha256:a7b517f9e0f42ffade802eef9cefa271372386b85f55b702b493241e58459793 k8s.gcr.io/sig-storage/mock-driver:v4.1.0],SizeBytes:17680993,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:17611980,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:529be2c9770add0cdd0c989115222ea9fc1be430c11095eb9f6dafcf98a36e2b k8s.gcr.io/sig-storage/livenessprobe:v2.4.0],SizeBytes:17175488,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:13616070e3f29de4417eee434a8ef472221c9e51b3d037b5a6b46cef08eb7443 k8s.gcr.io/e2e-test-images/nginx:1.14-2],SizeBytes:16032814,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf k8s.gcr.io/e2e-test-images/busybox:1.29-2],SizeBytes:1154361,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db k8s.gcr.io/pause:3.6 registry.k8s.io/pause:3.6],SizeBytes:682696,},},VolumesInUse:[kubernetes.io/csi/csi-mock-csi-mock-volumes-8997^4],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/csi-mock-csi-mock-volumes-8997^4,DevicePath:,},},Config:nil,},} Jan 1 02:48:33.000: INFO: Logging kubelet events for node ip-172-20-49-224.eu-west-3.compute.internal Jan 1 02:48:33.117: INFO: Logging pods the kubelet thinks is on node ip-172-20-49-224.eu-west-3.compute.internal Jan 1 02:48:33.228: INFO: netserver-3 started at 2023-01-01 02:48:17 +0000 UTC (0+1 container statuses recorded) Jan 1 02:48:33.228: INFO: Container webserver ready: false, restart count 0 Jan 1 02:48:33.228: INFO: csi-mockplugin-0 started at 2023-01-01 02:48:30 +0000 UTC (0+3 container statuses recorded) Jan 1 02:48:33.228: INFO: Container csi-provisioner ready: false, restart count 0 Jan 1 02:48:33.228: INFO: Container driver-registrar ready: false, restart count 0 Jan 1 02:48:33.228: INFO: Container mock ready: false, restart count 0 Jan 1 02:48:33.228: INFO: csi-mockplugin-attacher-0 started at 2023-01-01 02:48:30 +0000 UTC (0+1 container statuses recorded) Jan 1 02:48:33.228: INFO: Container csi-attacher ready: false, restart count 0 Jan 1 02:48:33.228: INFO: netserver-3 started at 2023-01-01 02:47:59 +0000 UTC (0+1 container statuses recorded) Jan 1 02:48:33.228: INFO: Container webserver ready: true, restart count 0 Jan 1 02:48:33.228: INFO: cilium-78jll started at 2023-01-01 02:20:29 +0000 UTC (1+1 container statuses recorded) Jan 1 02:48:33.228: INFO: Init container clean-cilium-state ready: true, restart count 0 Jan 1 02:48:33.228: INFO: Container cilium-agent ready: true, restart count 0 Jan 1 02:48:33.228: INFO: startup-1bd43bee-48b5-4ecf-b08a-2cd50bb794bb started at 2023-01-01 02:46:45 +0000 UTC (0+1 container statuses recorded) Jan 1 02:48:33.228: INFO: Container busybox ready: false, restart count 0 Jan 1 02:48:33.228: INFO: csi-mockplugin-0 started at 2023-01-01 02:47:51 +0000 UTC (0+3 container statuses recorded) Jan 1 02:48:33.228: INFO: Container csi-provisioner ready: true, restart count 0 Jan 1 02:48:33.228: INFO: Container driver-registrar ready: true, restart count 0 Jan 1 02:48:33.228: INFO: Container mock ready: true, restart count 0 Jan 1 02:48:33.228: INFO: rs-qwrvr started at 2023-01-01 02:47:13 +0000 UTC (0+1 container statuses recorded) Jan 1 02:48:33.228: INFO: Container donothing ready: false, restart count 0 Jan 1 02:48:33.228: INFO: pvc-volume-tester-nshg2 started at 2023-01-01 02:48:02 +0000 UTC (0+1 container statuses recorded) Jan 1 02:48:33.228: INFO: Container volume-tester ready: true, restart count 0 Jan 1 02:48:33.228: INFO: ebs-csi-node-tzntf started at 2023-01-01 02:20:29 +0000 UTC (0+3 container statuses recorded) Jan 1 02:48:33.228: INFO: Container ebs-plugin ready: true, restart count 0 Jan 1 02:48:33.228: INFO: Container liveness-probe ready: true, restart count 0 Jan 1 02:48:33.228: INFO: Container node-driver-registrar ready: true, restart count 0 Jan 1 02:48:33.228: INFO: csi-mockplugin-attacher-0 started at 2023-01-01 02:47:51 +0000 UTC (0+1 container statuses recorded) Jan 1 02:48:33.228: INFO: Container csi-attacher ready: true, restart count 0 Jan 1 02:48:33.715: INFO: Latency metrics for node ip-172-20-49-224.eu-west-3.compute.internal Jan 1 02:48:33.715: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready �[1mSTEP�[0m: Destroying namespace "pods-2607" for this suite.
Find to mentions in log files
go run hack/e2e.go -v --test --test_args='--ginkgo.focus=Kubernetes\se2e\ssuite\s\[sig\-node\]\sPods\sshould\ssupport\sremote\scommand\sexecution\sover\swebsockets\s\[NodeConformance\]\s\[Conformance\]$'
/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:633 Jan 1 02:30:37.503: Unexpected error: <*errors.errorString | 0xc0002ba2b0>: { s: "timed out waiting for the condition", } timed out waiting for the condition occurred /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/pods.go:107from junit_06.xml
[BeforeEach] [sig-node] Pods /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:185 �[1mSTEP�[0m: Creating a kubernetes client Jan 1 02:25:36.369: INFO: >>> kubeConfig: /root/.kube/config �[1mSTEP�[0m: Building a namespace api object, basename pods �[1mSTEP�[0m: Waiting for a default service account to be provisioned in namespace �[1mSTEP�[0m: Waiting for kube-root-ca.crt to be provisioned in namespace [BeforeEach] [sig-node] Pods /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/common/node/pods.go:189 [It] should support remote command execution over websockets [NodeConformance] [Conformance] /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:633 Jan 1 02:25:37.088: INFO: >>> kubeConfig: /root/.kube/config �[1mSTEP�[0m: creating the pod �[1mSTEP�[0m: submitting the pod to kubernetes Jan 1 02:25:37.297: INFO: The status of Pod pod-exec-websocket-4b20549e-25f0-4f45-bfef-409447a30721 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:25:39.400: INFO: The status of Pod pod-exec-websocket-4b20549e-25f0-4f45-bfef-409447a30721 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:25:41.400: INFO: The status of Pod pod-exec-websocket-4b20549e-25f0-4f45-bfef-409447a30721 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:25:43.401: INFO: The status of Pod pod-exec-websocket-4b20549e-25f0-4f45-bfef-409447a30721 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:25:45.401: INFO: The status of Pod pod-exec-websocket-4b20549e-25f0-4f45-bfef-409447a30721 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:25:47.401: INFO: The status of Pod pod-exec-websocket-4b20549e-25f0-4f45-bfef-409447a30721 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:25:49.401: INFO: The status of Pod pod-exec-websocket-4b20549e-25f0-4f45-bfef-409447a30721 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:25:51.401: INFO: The status of Pod pod-exec-websocket-4b20549e-25f0-4f45-bfef-409447a30721 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:25:53.401: INFO: The status of Pod pod-exec-websocket-4b20549e-25f0-4f45-bfef-409447a30721 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:25:55.400: INFO: The status of Pod pod-exec-websocket-4b20549e-25f0-4f45-bfef-409447a30721 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:25:57.400: INFO: The status of Pod pod-exec-websocket-4b20549e-25f0-4f45-bfef-409447a30721 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:25:59.400: INFO: The status of Pod pod-exec-websocket-4b20549e-25f0-4f45-bfef-409447a30721 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:26:01.400: INFO: The status of Pod pod-exec-websocket-4b20549e-25f0-4f45-bfef-409447a30721 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:26:03.400: INFO: The status of Pod pod-exec-websocket-4b20549e-25f0-4f45-bfef-409447a30721 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:26:05.401: INFO: The status of Pod pod-exec-websocket-4b20549e-25f0-4f45-bfef-409447a30721 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:26:07.401: INFO: The status of Pod pod-exec-websocket-4b20549e-25f0-4f45-bfef-409447a30721 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:26:09.401: INFO: The status of Pod pod-exec-websocket-4b20549e-25f0-4f45-bfef-409447a30721 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:26:11.400: INFO: The status of Pod pod-exec-websocket-4b20549e-25f0-4f45-bfef-409447a30721 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:26:13.401: INFO: The status of Pod pod-exec-websocket-4b20549e-25f0-4f45-bfef-409447a30721 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:26:15.401: INFO: The status of Pod pod-exec-websocket-4b20549e-25f0-4f45-bfef-409447a30721 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:26:17.400: INFO: The status of Pod pod-exec-websocket-4b20549e-25f0-4f45-bfef-409447a30721 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:26:19.400: INFO: The status of Pod pod-exec-websocket-4b20549e-25f0-4f45-bfef-409447a30721 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:26:21.401: INFO: The status of Pod pod-exec-websocket-4b20549e-25f0-4f45-bfef-409447a30721 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:26:23.400: INFO: The status of Pod pod-exec-websocket-4b20549e-25f0-4f45-bfef-409447a30721 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:26:25.400: INFO: The status of Pod pod-exec-websocket-4b20549e-25f0-4f45-bfef-409447a30721 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:26:27.400: INFO: The status of Pod pod-exec-websocket-4b20549e-25f0-4f45-bfef-409447a30721 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:26:29.401: INFO: The status of Pod pod-exec-websocket-4b20549e-25f0-4f45-bfef-409447a30721 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:26:31.401: INFO: The status of Pod pod-exec-websocket-4b20549e-25f0-4f45-bfef-409447a30721 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:26:33.400: INFO: The status of Pod pod-exec-websocket-4b20549e-25f0-4f45-bfef-409447a30721 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:26:35.401: INFO: The status of Pod pod-exec-websocket-4b20549e-25f0-4f45-bfef-409447a30721 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:26:37.400: INFO: The status of Pod pod-exec-websocket-4b20549e-25f0-4f45-bfef-409447a30721 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:26:39.400: INFO: The status of Pod pod-exec-websocket-4b20549e-25f0-4f45-bfef-409447a30721 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:26:41.400: INFO: The status of Pod pod-exec-websocket-4b20549e-25f0-4f45-bfef-409447a30721 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:26:43.401: INFO: The status of Pod pod-exec-websocket-4b20549e-25f0-4f45-bfef-409447a30721 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:26:45.401: INFO: The status of Pod pod-exec-websocket-4b20549e-25f0-4f45-bfef-409447a30721 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:26:47.400: INFO: The status of Pod pod-exec-websocket-4b20549e-25f0-4f45-bfef-409447a30721 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:26:49.401: INFO: The status of Pod pod-exec-websocket-4b20549e-25f0-4f45-bfef-409447a30721 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:26:51.400: INFO: The status of Pod pod-exec-websocket-4b20549e-25f0-4f45-bfef-409447a30721 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:26:53.400: INFO: The status of Pod pod-exec-websocket-4b20549e-25f0-4f45-bfef-409447a30721 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:26:55.401: INFO: The status of Pod pod-exec-websocket-4b20549e-25f0-4f45-bfef-409447a30721 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:26:57.400: INFO: The status of Pod pod-exec-websocket-4b20549e-25f0-4f45-bfef-409447a30721 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:26:59.402: INFO: The status of Pod pod-exec-websocket-4b20549e-25f0-4f45-bfef-409447a30721 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:27:01.400: INFO: The status of Pod pod-exec-websocket-4b20549e-25f0-4f45-bfef-409447a30721 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:27:03.402: INFO: The status of Pod pod-exec-websocket-4b20549e-25f0-4f45-bfef-409447a30721 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:27:05.400: INFO: The status of Pod pod-exec-websocket-4b20549e-25f0-4f45-bfef-409447a30721 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:27:07.400: INFO: The status of Pod pod-exec-websocket-4b20549e-25f0-4f45-bfef-409447a30721 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:27:09.400: INFO: The status of Pod pod-exec-websocket-4b20549e-25f0-4f45-bfef-409447a30721 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:27:11.400: INFO: The status of Pod pod-exec-websocket-4b20549e-25f0-4f45-bfef-409447a30721 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:27:13.401: INFO: The status of Pod pod-exec-websocket-4b20549e-25f0-4f45-bfef-409447a30721 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:27:15.400: INFO: The status of Pod pod-exec-websocket-4b20549e-25f0-4f45-bfef-409447a30721 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:27:17.400: INFO: The status of Pod pod-exec-websocket-4b20549e-25f0-4f45-bfef-409447a30721 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:27:19.400: INFO: The status of Pod pod-exec-websocket-4b20549e-25f0-4f45-bfef-409447a30721 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:27:21.401: INFO: The status of Pod pod-exec-websocket-4b20549e-25f0-4f45-bfef-409447a30721 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:27:23.400: INFO: The status of Pod pod-exec-websocket-4b20549e-25f0-4f45-bfef-409447a30721 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:27:25.400: INFO: The status of Pod pod-exec-websocket-4b20549e-25f0-4f45-bfef-409447a30721 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:27:27.400: INFO: The status of Pod pod-exec-websocket-4b20549e-25f0-4f45-bfef-409447a30721 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:27:29.400: INFO: The status of Pod pod-exec-websocket-4b20549e-25f0-4f45-bfef-409447a30721 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:27:31.400: INFO: The status of Pod pod-exec-websocket-4b20549e-25f0-4f45-bfef-409447a30721 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:27:33.400: INFO: The status of Pod pod-exec-websocket-4b20549e-25f0-4f45-bfef-409447a30721 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:27:35.400: INFO: The status of Pod pod-exec-websocket-4b20549e-25f0-4f45-bfef-409447a30721 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:27:37.400: INFO: The status of Pod pod-exec-websocket-4b20549e-25f0-4f45-bfef-409447a30721 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:27:39.400: INFO: The status of Pod pod-exec-websocket-4b20549e-25f0-4f45-bfef-409447a30721 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:27:41.401: INFO: The status of Pod pod-exec-websocket-4b20549e-25f0-4f45-bfef-409447a30721 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:27:43.400: INFO: The status of Pod pod-exec-websocket-4b20549e-25f0-4f45-bfef-409447a30721 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:27:45.400: INFO: The status of Pod pod-exec-websocket-4b20549e-25f0-4f45-bfef-409447a30721 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:27:47.400: INFO: The status of Pod pod-exec-websocket-4b20549e-25f0-4f45-bfef-409447a30721 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:27:49.401: INFO: The status of Pod pod-exec-websocket-4b20549e-25f0-4f45-bfef-409447a30721 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:27:51.400: INFO: The status of Pod pod-exec-websocket-4b20549e-25f0-4f45-bfef-409447a30721 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:27:53.400: INFO: The status of Pod pod-exec-websocket-4b20549e-25f0-4f45-bfef-409447a30721 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:27:55.400: INFO: The status of Pod pod-exec-websocket-4b20549e-25f0-4f45-bfef-409447a30721 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:27:57.401: INFO: The status of Pod pod-exec-websocket-4b20549e-25f0-4f45-bfef-409447a30721 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:27:59.401: INFO: The status of Pod pod-exec-websocket-4b20549e-25f0-4f45-bfef-409447a30721 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:28:01.400: INFO: The status of Pod pod-exec-websocket-4b20549e-25f0-4f45-bfef-409447a30721 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:28:03.400: INFO: The status of Pod pod-exec-websocket-4b20549e-25f0-4f45-bfef-409447a30721 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:28:05.400: INFO: The status of Pod pod-exec-websocket-4b20549e-25f0-4f45-bfef-409447a30721 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:28:07.400: INFO: The status of Pod pod-exec-websocket-4b20549e-25f0-4f45-bfef-409447a30721 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:28:09.401: INFO: The status of Pod pod-exec-websocket-4b20549e-25f0-4f45-bfef-409447a30721 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:28:11.400: INFO: The status of Pod pod-exec-websocket-4b20549e-25f0-4f45-bfef-409447a30721 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:28:13.400: INFO: The status of Pod pod-exec-websocket-4b20549e-25f0-4f45-bfef-409447a30721 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:28:15.401: INFO: The status of Pod pod-exec-websocket-4b20549e-25f0-4f45-bfef-409447a30721 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:28:17.400: INFO: The status of Pod pod-exec-websocket-4b20549e-25f0-4f45-bfef-409447a30721 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:28:19.401: INFO: The status of Pod pod-exec-websocket-4b20549e-25f0-4f45-bfef-409447a30721 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:28:21.401: INFO: The status of Pod pod-exec-websocket-4b20549e-25f0-4f45-bfef-409447a30721 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:28:23.400: INFO: The status of Pod pod-exec-websocket-4b20549e-25f0-4f45-bfef-409447a30721 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:28:25.402: INFO: The status of Pod pod-exec-websocket-4b20549e-25f0-4f45-bfef-409447a30721 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:28:27.400: INFO: The status of Pod pod-exec-websocket-4b20549e-25f0-4f45-bfef-409447a30721 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:28:29.400: INFO: The status of Pod pod-exec-websocket-4b20549e-25f0-4f45-bfef-409447a30721 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:28:31.400: INFO: The status of Pod pod-exec-websocket-4b20549e-25f0-4f45-bfef-409447a30721 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:28:33.400: INFO: The status of Pod pod-exec-websocket-4b20549e-25f0-4f45-bfef-409447a30721 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:28:35.401: INFO: The status of Pod pod-exec-websocket-4b20549e-25f0-4f45-bfef-409447a30721 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:28:37.399: INFO: The status of Pod pod-exec-websocket-4b20549e-25f0-4f45-bfef-409447a30721 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:28:39.403: INFO: The status of Pod pod-exec-websocket-4b20549e-25f0-4f45-bfef-409447a30721 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:28:41.400: INFO: The status of Pod pod-exec-websocket-4b20549e-25f0-4f45-bfef-409447a30721 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:28:43.400: INFO: The status of Pod pod-exec-websocket-4b20549e-25f0-4f45-bfef-409447a30721 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:28:45.400: INFO: The status of Pod pod-exec-websocket-4b20549e-25f0-4f45-bfef-409447a30721 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:28:47.401: INFO: The status of Pod pod-exec-websocket-4b20549e-25f0-4f45-bfef-409447a30721 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:28:49.400: INFO: The status of Pod pod-exec-websocket-4b20549e-25f0-4f45-bfef-409447a30721 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:28:51.401: INFO: The status of Pod pod-exec-websocket-4b20549e-25f0-4f45-bfef-409447a30721 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:28:53.400: INFO: The status of Pod pod-exec-websocket-4b20549e-25f0-4f45-bfef-409447a30721 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:28:55.400: INFO: The status of Pod pod-exec-websocket-4b20549e-25f0-4f45-bfef-409447a30721 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:28:57.400: INFO: The status of Pod pod-exec-websocket-4b20549e-25f0-4f45-bfef-409447a30721 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:28:59.400: INFO: The status of Pod pod-exec-websocket-4b20549e-25f0-4f45-bfef-409447a30721 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:29:01.400: INFO: The status of Pod pod-exec-websocket-4b20549e-25f0-4f45-bfef-409447a30721 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:29:03.400: INFO: The status of Pod pod-exec-websocket-4b20549e-25f0-4f45-bfef-409447a30721 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:29:05.402: INFO: The status of Pod pod-exec-websocket-4b20549e-25f0-4f45-bfef-409447a30721 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:29:07.400: INFO: The status of Pod pod-exec-websocket-4b20549e-25f0-4f45-bfef-409447a30721 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:29:09.401: INFO: The status of Pod pod-exec-websocket-4b20549e-25f0-4f45-bfef-409447a30721 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:29:11.403: INFO: The status of Pod pod-exec-websocket-4b20549e-25f0-4f45-bfef-409447a30721 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:29:13.400: INFO: The status of Pod pod-exec-websocket-4b20549e-25f0-4f45-bfef-409447a30721 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:29:15.402: INFO: The status of Pod pod-exec-websocket-4b20549e-25f0-4f45-bfef-409447a30721 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:29:17.401: INFO: The status of Pod pod-exec-websocket-4b20549e-25f0-4f45-bfef-409447a30721 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:29:19.401: INFO: The status of Pod pod-exec-websocket-4b20549e-25f0-4f45-bfef-409447a30721 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:29:21.402: INFO: The status of Pod pod-exec-websocket-4b20549e-25f0-4f45-bfef-409447a30721 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:29:23.399: INFO: The status of Pod pod-exec-websocket-4b20549e-25f0-4f45-bfef-409447a30721 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:29:25.401: INFO: The status of Pod pod-exec-websocket-4b20549e-25f0-4f45-bfef-409447a30721 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:29:27.401: INFO: The status of Pod pod-exec-websocket-4b20549e-25f0-4f45-bfef-409447a30721 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:29:29.401: INFO: The status of Pod pod-exec-websocket-4b20549e-25f0-4f45-bfef-409447a30721 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:29:31.400: INFO: The status of Pod pod-exec-websocket-4b20549e-25f0-4f45-bfef-409447a30721 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:29:33.400: INFO: The status of Pod pod-exec-websocket-4b20549e-25f0-4f45-bfef-409447a30721 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:29:35.400: INFO: The status of Pod pod-exec-websocket-4b20549e-25f0-4f45-bfef-409447a30721 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:29:37.400: INFO: The status of Pod pod-exec-websocket-4b20549e-25f0-4f45-bfef-409447a30721 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:29:39.400: INFO: The status of Pod pod-exec-websocket-4b20549e-25f0-4f45-bfef-409447a30721 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:29:41.400: INFO: The status of Pod pod-exec-websocket-4b20549e-25f0-4f45-bfef-409447a30721 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:29:43.400: INFO: The status of Pod pod-exec-websocket-4b20549e-25f0-4f45-bfef-409447a30721 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:29:45.401: INFO: The status of Pod pod-exec-websocket-4b20549e-25f0-4f45-bfef-409447a30721 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:29:47.401: INFO: The status of Pod pod-exec-websocket-4b20549e-25f0-4f45-bfef-409447a30721 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:29:49.400: INFO: The status of Pod pod-exec-websocket-4b20549e-25f0-4f45-bfef-409447a30721 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:29:51.401: INFO: The status of Pod pod-exec-websocket-4b20549e-25f0-4f45-bfef-409447a30721 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:29:53.400: INFO: The status of Pod pod-exec-websocket-4b20549e-25f0-4f45-bfef-409447a30721 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:29:55.401: INFO: The status of Pod pod-exec-websocket-4b20549e-25f0-4f45-bfef-409447a30721 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:29:57.400: INFO: The status of Pod pod-exec-websocket-4b20549e-25f0-4f45-bfef-409447a30721 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:29:59.400: INFO: The status of Pod pod-exec-websocket-4b20549e-25f0-4f45-bfef-409447a30721 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:30:01.400: INFO: The status of Pod pod-exec-websocket-4b20549e-25f0-4f45-bfef-409447a30721 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:30:03.400: INFO: The status of Pod pod-exec-websocket-4b20549e-25f0-4f45-bfef-409447a30721 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:30:05.400: INFO: The status of Pod pod-exec-websocket-4b20549e-25f0-4f45-bfef-409447a30721 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:30:07.400: INFO: The status of Pod pod-exec-websocket-4b20549e-25f0-4f45-bfef-409447a30721 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:30:09.401: INFO: The status of Pod pod-exec-websocket-4b20549e-25f0-4f45-bfef-409447a30721 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:30:11.401: INFO: The status of Pod pod-exec-websocket-4b20549e-25f0-4f45-bfef-409447a30721 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:30:13.401: INFO: The status of Pod pod-exec-websocket-4b20549e-25f0-4f45-bfef-409447a30721 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:30:15.401: INFO: The status of Pod pod-exec-websocket-4b20549e-25f0-4f45-bfef-409447a30721 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:30:17.401: INFO: The status of Pod pod-exec-websocket-4b20549e-25f0-4f45-bfef-409447a30721 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:30:19.400: INFO: The status of Pod pod-exec-websocket-4b20549e-25f0-4f45-bfef-409447a30721 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:30:21.401: INFO: The status of Pod pod-exec-websocket-4b20549e-25f0-4f45-bfef-409447a30721 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:30:23.400: INFO: The status of Pod pod-exec-websocket-4b20549e-25f0-4f45-bfef-409447a30721 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:30:25.401: INFO: The status of Pod pod-exec-websocket-4b20549e-25f0-4f45-bfef-409447a30721 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:30:27.400: INFO: The status of Pod pod-exec-websocket-4b20549e-25f0-4f45-bfef-409447a30721 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:30:29.400: INFO: The status of Pod pod-exec-websocket-4b20549e-25f0-4f45-bfef-409447a30721 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:30:31.400: INFO: The status of Pod pod-exec-websocket-4b20549e-25f0-4f45-bfef-409447a30721 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:30:33.400: INFO: The status of Pod pod-exec-websocket-4b20549e-25f0-4f45-bfef-409447a30721 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:30:35.401: INFO: The status of Pod pod-exec-websocket-4b20549e-25f0-4f45-bfef-409447a30721 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:30:37.400: INFO: The status of Pod pod-exec-websocket-4b20549e-25f0-4f45-bfef-409447a30721 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:30:37.503: INFO: The status of Pod pod-exec-websocket-4b20549e-25f0-4f45-bfef-409447a30721 is Pending, waiting for it to be Running (with Ready = true) Jan 1 02:30:37.503: FAIL: Unexpected error: <*errors.errorString | 0xc0002ba2b0>: { s: "timed out waiting for the condition", } timed out waiting for the condition occurred Full Stack Trace k8s.io/kubernetes/test/e2e/framework.(*PodClient).CreateSync(0xc002ebb200, 0x8) /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/pods.go:107 +0x94 k8s.io/kubernetes/test/e2e/common/node.glob..func14.7() /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/common/node/pods.go:562 +0x387 k8s.io/kubernetes/test/e2e.RunE2ETests(0x24c66d7) _output/dockerized/go/src/k8s.io/kubernetes/test/e2e/e2e.go:133 +0x697 k8s.io/kubernetes/test/e2e.TestE2E(0x243a8f9) _output/dockerized/go/src/k8s.io/kubernetes/test/e2e/e2e_test.go:136 +0x19 testing.tRunner(0xc000d81380, 0x735e880) /usr/local/go/src/testing/testing.go:1259 +0x102 created by testing.(*T).Run /usr/local/go/src/testing/testing.go:1306 +0x35a [AfterEach] [sig-node] Pods /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:186 �[1mSTEP�[0m: Collecting events from namespace "pods-4895". �[1mSTEP�[0m: Found 12 events. Jan 1 02:30:37.608: INFO: At 2023-01-01 02:25:37 +0000 UTC - event for pod-exec-websocket-4b20549e-25f0-4f45-bfef-409447a30721: {default-scheduler } Scheduled: Successfully assigned pods-4895/pod-exec-websocket-4b20549e-25f0-4f45-bfef-409447a30721 to ip-172-20-45-112.eu-west-3.compute.internal Jan 1 02:30:37.608: INFO: At 2023-01-01 02:25:39 +0000 UTC - event for pod-exec-websocket-4b20549e-25f0-4f45-bfef-409447a30721: {kubelet ip-172-20-45-112.eu-west-3.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "f08e97a0bc46d6bdb314083138bde204c00dd60200dabf26be2f339abbfa3ed5" network for pod "pod-exec-websocket-4b20549e-25f0-4f45-bfef-409447a30721": networkPlugin cni failed to set up pod "pod-exec-websocket-4b20549e-25f0-4f45-bfef-409447a30721_pods-4895" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 1 02:30:37.608: INFO: At 2023-01-01 02:25:39 +0000 UTC - event for pod-exec-websocket-4b20549e-25f0-4f45-bfef-409447a30721: {kubelet ip-172-20-45-112.eu-west-3.compute.internal} SandboxChanged: Pod sandbox changed, it will be killed and re-created. Jan 1 02:30:37.608: INFO: At 2023-01-01 02:25:41 +0000 UTC - event for pod-exec-websocket-4b20549e-25f0-4f45-bfef-409447a30721: {kubelet ip-172-20-45-112.eu-west-3.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "eb480ef317d3a5f3d5bc206cce78d1a63c8f0e57e480f03f006c1a6bedfde74e" network for pod "pod-exec-websocket-4b20549e-25f0-4f45-bfef-409447a30721": networkPlugin cni failed to set up pod "pod-exec-websocket-4b20549e-25f0-4f45-bfef-409447a30721_pods-4895" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 1 02:30:37.608: INFO: At 2023-01-01 02:25:43 +0000 UTC - event for pod-exec-websocket-4b20549e-25f0-4f45-bfef-409447a30721: {kubelet ip-172-20-45-112.eu-west-3.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "dd1c7b221d913cbd49d78a9b175733b729eb9e08741d76addf770edb7ade9cf1" network for pod "pod-exec-websocket-4b20549e-25f0-4f45-bfef-409447a30721": networkPlugin cni failed to set up pod "pod-exec-websocket-4b20549e-25f0-4f45-bfef-409447a30721_pods-4895" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 1 02:30:37.608: INFO: At 2023-01-01 02:25:46 +0000 UTC - event for pod-exec-websocket-4b20549e-25f0-4f45-bfef-409447a30721: {kubelet ip-172-20-45-112.eu-west-3.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "97c67b17645e1a58ed2119f4b045bf2425a58f5b6e87e5251618c5f5000172b5" network for pod "pod-exec-websocket-4b20549e-25f0-4f45-bfef-409447a30721": networkPlugin cni failed to set up pod "pod-exec-websocket-4b20549e-25f0-4f45-bfef-409447a30721_pods-4895" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 1 02:30:37.608: INFO: At 2023-01-01 02:25:49 +0000 UTC - event for pod-exec-websocket-4b20549e-25f0-4f45-bfef-409447a30721: {kubelet ip-172-20-45-112.eu-west-3.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "e288bfc8dc47205bf34ed2292a5809ce00b22314866d0dc27e6c29c419daa9e5" network for pod "pod-exec-websocket-4b20549e-25f0-4f45-bfef-409447a30721": networkPlugin cni failed to set up pod "pod-exec-websocket-4b20549e-25f0-4f45-bfef-409447a30721_pods-4895" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 1 02:30:37.608: INFO: At 2023-01-01 02:25:53 +0000 UTC - event for pod-exec-websocket-4b20549e-25f0-4f45-bfef-409447a30721: {kubelet ip-172-20-45-112.eu-west-3.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "e6c14bbfc3b1af8e7f46d823c7a0f8d54bc4c4477bd2e8135f2246293f68a422" network for pod "pod-exec-websocket-4b20549e-25f0-4f45-bfef-409447a30721": networkPlugin cni failed to set up pod "pod-exec-websocket-4b20549e-25f0-4f45-bfef-409447a30721_pods-4895" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 1 02:30:37.608: INFO: At 2023-01-01 02:25:56 +0000 UTC - event for pod-exec-websocket-4b20549e-25f0-4f45-bfef-409447a30721: {kubelet ip-172-20-45-112.eu-west-3.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "3d019cbc03966465836e59704dc9513d58ea793035cabb3591714ba146e2e6e0" network for pod "pod-exec-websocket-4b20549e-25f0-4f45-bfef-409447a30721": networkPlugin cni failed to set up pod "pod-exec-websocket-4b20549e-25f0-4f45-bfef-409447a30721_pods-4895" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 1 02:30:37.608: INFO: At 2023-01-01 02:26:00 +0000 UTC - event for pod-exec-websocket-4b20549e-25f0-4f45-bfef-409447a30721: {kubelet ip-172-20-45-112.eu-west-3.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "86338b9d5b67aea85b2bbb8843ec927da6e9e6f241678be7d88891ba8b968c85" network for pod "pod-exec-websocket-4b20549e-25f0-4f45-bfef-409447a30721": networkPlugin cni failed to set up pod "pod-exec-websocket-4b20549e-25f0-4f45-bfef-409447a30721_pods-4895" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 1 02:30:37.608: INFO: At 2023-01-01 02:26:05 +0000 UTC - event for pod-exec-websocket-4b20549e-25f0-4f45-bfef-409447a30721: {kubelet ip-172-20-45-112.eu-west-3.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "649ce4f132fbdbf828b1b7ee43f9cdb1800458b3b378522c917fa4d205b60e1f" network for pod "pod-exec-websocket-4b20549e-25f0-4f45-bfef-409447a30721": networkPlugin cni failed to set up pod "pod-exec-websocket-4b20549e-25f0-4f45-bfef-409447a30721_pods-4895" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 1 02:30:37.608: INFO: At 2023-01-01 02:26:18 +0000 UTC - event for pod-exec-websocket-4b20549e-25f0-4f45-bfef-409447a30721: {kubelet ip-172-20-45-112.eu-west-3.compute.internal} FailedCreatePodSandBox: (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "9f6410cdf82dc92b7e606575284bb6f7dbf6d112b03d45913e9d57457ca1e547" network for pod "pod-exec-websocket-4b20549e-25f0-4f45-bfef-409447a30721": networkPlugin cni failed to set up pod "pod-exec-websocket-4b20549e-25f0-4f45-bfef-409447a30721_pods-4895" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 1 02:30:37.711: INFO: POD NODE PHASE GRACE CONDITIONS Jan 1 02:30:37.711: INFO: pod-exec-websocket-4b20549e-25f0-4f45-bfef-409447a30721 ip-172-20-45-112.eu-west-3.compute.internal Pending [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2023-01-01 02:25:37 +0000 UTC } {Ready False 0001-01-01 00:00:00 +0000 UTC 2023-01-01 02:25:37 +0000 UTC ContainersNotReady containers with unready status: [main]} {ContainersReady False 0001-01-01 00:00:00 +0000 UTC 2023-01-01 02:25:37 +0000 UTC ContainersNotReady containers with unready status: [main]} {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2023-01-01 02:25:37 +0000 UTC }] Jan 1 02:30:37.711: INFO: Jan 1 02:30:37.815: INFO: Unable to fetch pods-4895/pod-exec-websocket-4b20549e-25f0-4f45-bfef-409447a30721/main logs: the server rejected our request for an unknown reason (get pods pod-exec-websocket-4b20549e-25f0-4f45-bfef-409447a30721) Jan 1 02:30:37.919: INFO: Logging node info for node ip-172-20-33-66.eu-west-3.compute.internal Jan 1 02:30:38.021: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-33-66.eu-west-3.compute.internal ec4b59b3-62db-4040-aad2-4c3249f80fe8 6147 0 2023-01-01 02:20:25 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:eu-west-3 failure-domain.beta.kubernetes.io/zone:eu-west-3a kubelet_cleanup:true kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-33-66.eu-west-3.compute.internal kubernetes.io/os:linux kubernetes.io/role:node node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:eu-west-3a topology.kubernetes.io/region:eu-west-3 topology.kubernetes.io/zone:eu-west-3a] map[csi.volume.kubernetes.io/nodeid:{"csi-mock-csi-mock-volumes-4089":"csi-mock-csi-mock-volumes-4089","ebs.csi.aws.com":"i-028313673124ac4ad"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kops-controller Update v1 2023-01-01 02:20:25 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}} } {kube-controller-manager Update v1 2023-01-01 02:20:25 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.1.0/24\"":{}}}} } {kubelet Update v1 2023-01-01 02:20:25 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {e2e.test Update v1 2023-01-01 02:25:15 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubelet_cleanup":{}}}} } {kube-controller-manager Update v1 2023-01-01 02:25:34 +0000 UTC FieldsV1 {"f:status":{"f:volumesAttached":{}}} status} {kubelet Update v1 2023-01-01 02:25:41 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.1.0/24,DoNotUseExternalID:,ProviderID:aws:///eu-west-3a/i-028313673124ac4ad,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.1.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{50478186496 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4078190592 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{45430367772 0} {<nil>} 45430367772 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3973332992 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-01-01 02:25:41 +0000 UTC,LastTransitionTime:2023-01-01 02:20:25 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-01-01 02:25:41 +0000 UTC,LastTransitionTime:2023-01-01 02:20:25 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-01-01 02:25:41 +0000 UTC,LastTransitionTime:2023-01-01 02:20:25 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-01-01 02:25:41 +0000 UTC,LastTransitionTime:2023-01-01 02:20:46 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.33.66,},NodeAddress{Type:ExternalIP,Address:52.47.140.112,},NodeAddress{Type:Hostname,Address:ip-172-20-33-66.eu-west-3.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-33-66.eu-west-3.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-52-47-140-112.eu-west-3.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2ad2ce1a87d508c178acf6215c5767,SystemUUID:ec2ad2ce-1a87-d508-c178-acf6215c5767,BootID:b3b60783-d786-412b-a36a-802bcfae604d,KernelVersion:4.19.0-23-cloud-amd64,OSImage:Debian GNU/Linux 10 (buster),ContainerRuntimeVersion:docker://20.10.17,KubeletVersion:v1.23.15,KubeProxyVersion:v1.23.15,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:456569528,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/jessie-dnsutils@sha256:11e6a66017ba4e4b938c1612b7a54a3befcefd354796c04e1dba76873a13518e k8s.gcr.io/e2e-test-images/jessie-dnsutils:1.5],SizeBytes:253371792,},ContainerImage{Names:[nginx@sha256:0047b729188a15da49380d9506d65959cce6d40291ccfb4e039f5dc7efd33286 nginx:latest],SizeBytes:141812353,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:7e8bdd271312fd25fc5ff5a8f04727be84044eb3d7d8d03611972a6752e2e11e k8s.gcr.io/e2e-test-images/agnhost:2.39],SizeBytes:126872991,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:1b9d1b2f36cb2dbee1960e82a9344aeb11bd4c4c03abf5e1853e0559c23855e3 k8s.gcr.io/e2e-test-images/httpd:2.4.38-2],SizeBytes:123781643,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.23.15 registry.k8s.io/kube-proxy-amd64:v1.23.15],SizeBytes:112336036,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:10e231cf07c82dfab5141c1afe548dc734e5fc1f67665ec3982a325d1bd31a9a registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.0],SizeBytes:94471559,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:c8e03f60afa90a28e4bb6ec9a8d0fc36d89de4b7475cf2d613afa793ec969fe0 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.0],SizeBytes:56382329,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:c5be65d6679efabb969d9b019300d187437ae876f992c40911fd2892bbef3b36 k8s.gcr.io/sig-storage/csi-attacher:v3.2.0],SizeBytes:53500826,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:017727efcfeb7d053af68e51436ce8e65edbc6ca573720afb4f79c8594036955 registry.k8s.io/coredns/coredns:v1.10.0],SizeBytes:50249337,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:a61d309da54641db41fb8f35718f744e9f730d4d0384f8c4b186ddc9f06cbd5f k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.1.0],SizeBytes:19662887,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:19582112,},ContainerImage{Names:[k8s.gcr.io/sig-storage/mock-driver@sha256:a7b517f9e0f42ffade802eef9cefa271372386b85f55b702b493241e58459793 k8s.gcr.io/sig-storage/mock-driver:v4.1.0],SizeBytes:17680993,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:17611980,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:13616070e3f29de4417eee434a8ef472221c9e51b3d037b5a6b46cef08eb7443 k8s.gcr.io/e2e-test-images/nginx:1.14-2],SizeBytes:16032814,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nonewprivs@sha256:8ac1264691820febacf3aea5d152cbde6d10685731ec14966a9401c6f47a68ac k8s.gcr.io/e2e-test-images/nonewprivs:1.3],SizeBytes:7107254,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf k8s.gcr.io/e2e-test-images/busybox:1.29-2],SizeBytes:1154361,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db k8s.gcr.io/pause:3.6 registry.k8s.io/pause:3.6],SizeBytes:682696,},},VolumesInUse:[kubernetes.io/csi/csi-mock-csi-mock-volumes-4089^4],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/csi-mock-csi-mock-volumes-4089^4,DevicePath:,},},Config:nil,},} Jan 1 02:30:38.022: INFO: Logging kubelet events for node ip-172-20-33-66.eu-west-3.compute.internal Jan 1 02:30:38.127: INFO: Logging pods the kubelet thinks is on node ip-172-20-33-66.eu-west-3.compute.internal Jan 1 02:30:38.341: INFO: coredns-867df8f45c-cqmp7 started at 2023-01-01 02:21:18 +0000 UTC (0+1 container statuses recorded) Jan 1 02:30:38.341: INFO: Container coredns ready: true, restart count 0 Jan 1 02:30:38.341: INFO: netserver-0 started at 2023-01-01 02:25:27 +0000 UTC (0+1 container statuses recorded) Jan 1 02:30:38.341: INFO: Container webserver ready: true, restart count 0 Jan 1 02:30:38.341: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-l9hnv started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:30:38.341: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:30:38.341: INFO: pvc-volume-tester-bc9c6 started at 2023-01-01 02:25:33 +0000 UTC (0+1 container statuses recorded) Jan 1 02:30:38.341: INFO: Container volume-tester ready: false, restart count 0 Jan 1 02:30:38.341: INFO: netserver-0 started at 2023-01-01 02:25:33 +0000 UTC (0+1 container statuses recorded) Jan 1 02:30:38.341: INFO: Container webserver ready: false, restart count 0 Jan 1 02:30:38.341: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-vnv2x started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:30:38.341: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:30:38.341: INFO: csi-mockplugin-0 started at 2023-01-01 02:26:15 +0000 UTC (0+4 container statuses recorded) Jan 1 02:30:38.341: INFO: Container busybox ready: false, restart count 0 Jan 1 02:30:38.341: INFO: Container csi-provisioner ready: false, restart count 0 Jan 1 02:30:38.341: INFO: Container driver-registrar ready: false, restart count 0 Jan 1 02:30:38.341: INFO: Container mock ready: false, restart count 0 Jan 1 02:30:38.341: INFO: hostexec-ip-172-20-33-66.eu-west-3.compute.internal-qm7jt started at 2023-01-01 02:27:26 +0000 UTC (0+1 container statuses recorded) Jan 1 02:30:38.341: INFO: Container agnhost-container ready: true, restart count 0 Jan 1 02:30:38.341: INFO: csi-mockplugin-0 started at 2023-01-01 02:24:59 +0000 UTC (0+3 container statuses recorded) Jan 1 02:30:38.341: INFO: Container csi-provisioner ready: true, restart count 0 Jan 1 02:30:38.341: INFO: Container driver-registrar ready: true, restart count 0 Jan 1 02:30:38.341: INFO: Container mock ready: true, restart count 0 Jan 1 02:30:38.342: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-csnzw started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:30:38.342: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: false, restart count 0 Jan 1 02:30:38.342: INFO: hostexec-ip-172-20-33-66.eu-west-3.compute.internal-cr6qn started at 2023-01-01 02:25:58 +0000 UTC (0+1 container statuses recorded) Jan 1 02:30:38.342: INFO: Container agnhost-container ready: true, restart count 0 Jan 1 02:30:38.342: INFO: local-injector started at 2023-01-01 02:26:06 +0000 UTC (0+1 container statuses recorded) Jan 1 02:30:38.342: INFO: Container local-injector ready: false, restart count 0 Jan 1 02:30:38.342: INFO: ebs-csi-node-vsxjc started at 2023-01-01 02:20:26 +0000 UTC (0+3 container statuses recorded) Jan 1 02:30:38.342: INFO: Container ebs-plugin ready: true, restart count 0 Jan 1 02:30:38.342: INFO: Container liveness-probe ready: true, restart count 0 Jan 1 02:30:38.342: INFO: Container node-driver-registrar ready: true, restart count 0 Jan 1 02:30:38.342: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-h4nvd started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:30:38.342: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:30:38.342: INFO: cilium-vbcgc started at 2023-01-01 02:20:26 +0000 UTC (1+1 container statuses recorded) Jan 1 02:30:38.342: INFO: Init container clean-cilium-state ready: true, restart count 0 Jan 1 02:30:38.342: INFO: Container cilium-agent ready: true, restart count 0 Jan 1 02:30:38.342: INFO: affinity-clusterip-timeout-q9qt2 started at 2023-01-01 02:27:32 +0000 UTC (0+1 container statuses recorded) Jan 1 02:30:38.342: INFO: Container affinity-clusterip-timeout ready: false, restart count 0 Jan 1 02:30:38.342: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-m6vq2 started at 2023-01-01 02:25:17 +0000 UTC (0+1 container statuses recorded) Jan 1 02:30:38.342: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: false, restart count 0 Jan 1 02:30:38.342: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-h6mmt started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:30:38.342: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:30:38.342: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-dkfpd started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:30:38.342: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:30:38.342: INFO: local-injector started at 2023-01-01 02:27:36 +0000 UTC (0+1 container statuses recorded) Jan 1 02:30:38.342: INFO: Container local-injector ready: false, restart count 0 Jan 1 02:30:38.342: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-546bc started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:30:38.342: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:30:38.342: INFO: csi-mockplugin-attacher-0 started at 2023-01-01 02:27:56 +0000 UTC (0+1 container statuses recorded) Jan 1 02:30:38.342: INFO: Container csi-attacher ready: false, restart count 0 Jan 1 02:30:38.342: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-q8pgp started at 2023-01-01 02:25:17 +0000 UTC (0+1 container statuses recorded) Jan 1 02:30:38.342: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:30:38.342: INFO: csi-mockplugin-0 started at 2023-01-01 02:27:56 +0000 UTC (0+3 container statuses recorded) Jan 1 02:30:38.342: INFO: Container csi-provisioner ready: false, restart count 0 Jan 1 02:30:38.342: INFO: Container driver-registrar ready: false, restart count 0 Jan 1 02:30:38.342: INFO: Container mock ready: false, restart count 0 Jan 1 02:30:38.342: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-2v87j started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:30:38.342: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:30:38.342: INFO: csi-mockplugin-attacher-0 started at 2023-01-01 02:25:00 +0000 UTC (0+1 container statuses recorded) Jan 1 02:30:38.342: INFO: Container csi-attacher ready: true, restart count 0 Jan 1 02:30:38.843: INFO: Latency metrics for node ip-172-20-33-66.eu-west-3.compute.internal Jan 1 02:30:38.843: INFO: Logging node info for node ip-172-20-38-238.eu-west-3.compute.internal Jan 1 02:30:38.945: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-38-238.eu-west-3.compute.internal 0eb713b4-64bf-47b7-8090-bab8111218ee 6771 0 2023-01-01 02:18:51 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:c5.large beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:eu-west-3 failure-domain.beta.kubernetes.io/zone:eu-west-3a kops.k8s.io/kops-controller-pki: kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-38-238.eu-west-3.compute.internal kubernetes.io/os:linux kubernetes.io/role:master node-role.kubernetes.io/control-plane: node-role.kubernetes.io/master: node.kubernetes.io/exclude-from-external-load-balancers: node.kubernetes.io/instance-type:c5.large topology.ebs.csi.aws.com/zone:eu-west-3a topology.kubernetes.io/region:eu-west-3 topology.kubernetes.io/zone:eu-west-3a] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-0f70ebb30a9076648"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kubelet Update v1 2023-01-01 02:18:50 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {protokube Update v1 2023-01-01 02:19:16 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kops.k8s.io/kops-controller-pki":{},"f:node-role.kubernetes.io/control-plane":{},"f:node.kubernetes.io/exclude-from-external-load-balancers":{}}}} } {kops-controller Update v1 2023-01-01 02:19:21 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/master":{}}}} } {kube-controller-manager Update v1 2023-01-01 02:19:31 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.0.0/24\"":{}},"f:taints":{}}} } {kubelet Update v1 2023-01-01 02:20:36 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.0.0/24,DoNotUseExternalID:,ProviderID:aws:///eu-west-3a/i-0f70ebb30a9076648,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/master,Value:,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[100.96.0.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{50478186496 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3918811136 0} {<nil>} 3826964Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{45430367772 0} {<nil>} 45430367772 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3813953536 0} {<nil>} 3724564Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-01-01 02:26:20 +0000 UTC,LastTransitionTime:2023-01-01 02:18:45 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-01-01 02:26:20 +0000 UTC,LastTransitionTime:2023-01-01 02:18:45 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-01-01 02:26:20 +0000 UTC,LastTransitionTime:2023-01-01 02:18:45 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-01-01 02:26:20 +0000 UTC,LastTransitionTime:2023-01-01 02:19:31 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.38.238,},NodeAddress{Type:ExternalIP,Address:15.237.26.58,},NodeAddress{Type:Hostname,Address:ip-172-20-38-238.eu-west-3.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-38-238.eu-west-3.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-15-237-26-58.eu-west-3.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2e6c64b5b811bf0c78b2aa7ccdb943,SystemUUID:ec2e6c64-b5b8-11bf-0c78-b2aa7ccdb943,BootID:9db01fa3-d136-4230-b21a-b03abbfb6bff,KernelVersion:4.19.0-23-cloud-amd64,OSImage:Debian GNU/Linux 10 (buster),ContainerRuntimeVersion:docker://20.10.17,KubeletVersion:v1.23.15,KubeProxyVersion:v1.23.15,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[registry.k8s.io/etcdadm/etcd-manager@sha256:66a453db625abb268f4b3bbefc5a34a171d81e6e8796cecca54cfd71775c77c4 registry.k8s.io/etcdadm/etcd-manager:v3.0.20221209],SizeBytes:662209237,},ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:456569528,},ContainerImage{Names:[k8s.gcr.io/kube-apiserver-amd64:v1.23.15 registry.k8s.io/kube-apiserver-amd64:v1.23.15],SizeBytes:135240444,},ContainerImage{Names:[k8s.gcr.io/kube-controller-manager-amd64:v1.23.15 registry.k8s.io/kube-controller-manager-amd64:v1.23.15],SizeBytes:125046097,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.23.15 registry.k8s.io/kube-proxy-amd64:v1.23.15],SizeBytes:112336036,},ContainerImage{Names:[quay.io/cilium/operator@sha256:a6d24a006a6b92967ac90786b49bc1ac26e5477cf028cd1186efcfc2466484db quay.io/cilium/operator:v1.12.5],SizeBytes:110016956,},ContainerImage{Names:[registry.k8s.io/kops/kops-controller:1.27.0-alpha.1],SizeBytes:105197757,},ContainerImage{Names:[registry.k8s.io/kops/dns-controller:1.27.0-alpha.1],SizeBytes:102577531,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:10e231cf07c82dfab5141c1afe548dc734e5fc1f67665ec3982a325d1bd31a9a registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.0],SizeBytes:94471559,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:122bfb8c1edabb3c0edd63f06523e6940d958d19b3957dc7b1d6f81e9f1f6119 registry.k8s.io/sig-storage/csi-provisioner:v3.1.0],SizeBytes:57736070,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:9ebbf9f023e7b41ccee3d52afe39a89e3ddacdbb69269d583abfc25847cfd9e4 registry.k8s.io/sig-storage/csi-resizer:v1.4.0],SizeBytes:55464778,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:8b9c313c05f54fb04f8d430896f5f5904b6cb157df261501b29adc04d2b2dc7b registry.k8s.io/sig-storage/csi-attacher:v3.4.0],SizeBytes:54848595,},ContainerImage{Names:[k8s.gcr.io/kube-scheduler-amd64:v1.23.15 registry.k8s.io/kube-scheduler-amd64:v1.23.15],SizeBytes:53521430,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:19582112,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:17611980,},ContainerImage{Names:[registry.k8s.io/kops/kube-apiserver-healthcheck:1.27.0-alpha.1],SizeBytes:8787111,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:682696,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Jan 1 02:30:38.946: INFO: Logging kubelet events for node ip-172-20-38-238.eu-west-3.compute.internal Jan 1 02:30:39.051: INFO: Logging pods the kubelet thinks is on node ip-172-20-38-238.eu-west-3.compute.internal Jan 1 02:30:39.161: INFO: etcd-manager-events-ip-172-20-38-238.eu-west-3.compute.internal started at 2023-01-01 02:18:11 +0000 UTC (0+1 container statuses recorded) Jan 1 02:30:39.161: INFO: Container etcd-manager ready: true, restart count 0 Jan 1 02:30:39.161: INFO: etcd-manager-main-ip-172-20-38-238.eu-west-3.compute.internal started at 2023-01-01 02:18:11 +0000 UTC (0+1 container statuses recorded) Jan 1 02:30:39.161: INFO: Container etcd-manager ready: true, restart count 0 Jan 1 02:30:39.161: INFO: cilium-qk59v started at 2023-01-01 02:19:20 +0000 UTC (1+1 container statuses recorded) Jan 1 02:30:39.161: INFO: Init container clean-cilium-state ready: true, restart count 0 Jan 1 02:30:39.161: INFO: Container cilium-agent ready: true, restart count 0 Jan 1 02:30:39.161: INFO: ebs-csi-node-k87k6 started at 2023-01-01 02:19:20 +0000 UTC (0+3 container statuses recorded) Jan 1 02:30:39.161: INFO: Container ebs-plugin ready: true, restart count 0 Jan 1 02:30:39.161: INFO: Container liveness-probe ready: true, restart count 0 Jan 1 02:30:39.161: INFO: Container node-driver-registrar ready: true, restart count 0 Jan 1 02:30:39.161: INFO: dns-controller-7db8b74b99-58zzw started at 2023-01-01 02:19:20 +0000 UTC (0+1 container statuses recorded) Jan 1 02:30:39.161: INFO: Container dns-controller ready: true, restart count 0 Jan 1 02:30:39.161: INFO: cilium-operator-d47487c76-j84nq started at 2023-01-01 02:19:20 +0000 UTC (0+1 container statuses recorded) Jan 1 02:30:39.161: INFO: Container cilium-operator ready: true, restart count 0 Jan 1 02:30:39.161: INFO: kube-controller-manager-ip-172-20-38-238.eu-west-3.compute.internal started at 2023-01-01 02:18:11 +0000 UTC (0+1 container statuses recorded) Jan 1 02:30:39.161: INFO: Container kube-controller-manager ready: true, restart count 2 Jan 1 02:30:39.161: INFO: kube-scheduler-ip-172-20-38-238.eu-west-3.compute.internal started at 2023-01-01 02:18:11 +0000 UTC (0+1 container statuses recorded) Jan 1 02:30:39.161: INFO: Container kube-scheduler ready: true, restart count 0 Jan 1 02:30:39.161: INFO: kube-apiserver-ip-172-20-38-238.eu-west-3.compute.internal started at 2023-01-01 02:18:11 +0000 UTC (0+2 container statuses recorded) Jan 1 02:30:39.161: INFO: Container healthcheck ready: true, restart count 0 Jan 1 02:30:39.161: INFO: Container kube-apiserver ready: true, restart count 1 Jan 1 02:30:39.161: INFO: kops-controller-26gzg started at 2023-01-01 02:19:20 +0000 UTC (0+1 container statuses recorded) Jan 1 02:30:39.161: INFO: Container kops-controller ready: true, restart count 0 Jan 1 02:30:39.161: INFO: ebs-csi-controller-6dc545b75-5qqh8 started at 2023-01-01 02:19:20 +0000 UTC (0+5 container statuses recorded) Jan 1 02:30:39.161: INFO: Container csi-attacher ready: true, restart count 0 Jan 1 02:30:39.161: INFO: Container csi-provisioner ready: true, restart count 0 Jan 1 02:30:39.161: INFO: Container csi-resizer ready: true, restart count 0 Jan 1 02:30:39.161: INFO: Container ebs-plugin ready: true, restart count 0 Jan 1 02:30:39.162: INFO: Container liveness-probe ready: true, restart count 0 Jan 1 02:30:39.488: INFO: Latency metrics for node ip-172-20-38-238.eu-west-3.compute.internal Jan 1 02:30:39.488: INFO: Logging node info for node ip-172-20-45-112.eu-west-3.compute.internal Jan 1 02:30:39.591: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-45-112.eu-west-3.compute.internal 48eed937-05bb-4d8a-bddd-aa94a2e6ab92 6756 0 2023-01-01 02:20:27 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:eu-west-3 failure-domain.beta.kubernetes.io/zone:eu-west-3a kubelet_cleanup:true kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-45-112.eu-west-3.compute.internal kubernetes.io/os:linux kubernetes.io/role:node node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:eu-west-3a topology.hostpath.csi/node:ip-172-20-45-112.eu-west-3.compute.internal topology.kubernetes.io/region:eu-west-3 topology.kubernetes.io/zone:eu-west-3a] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-0778adf2085fa7f34"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kube-controller-manager Update v1 2023-01-01 02:20:27 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.3.0/24\"":{}}}} } {kubelet Update v1 2023-01-01 02:20:27 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {kops-controller Update v1 2023-01-01 02:20:28 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}} } {e2e.test Update v1 2023-01-01 02:25:15 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubelet_cleanup":{}}}} } {kubelet Update v1 2023-01-01 02:26:05 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:capacity":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.3.0/24,DoNotUseExternalID:,ProviderID:aws:///eu-west-3a/i-0778adf2085fa7f34,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.3.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{50478186496 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4078190592 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{45430367772 0} {<nil>} 45430367772 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3973332992 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-01-01 02:26:05 +0000 UTC,LastTransitionTime:2023-01-01 02:20:27 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-01-01 02:26:05 +0000 UTC,LastTransitionTime:2023-01-01 02:20:27 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-01-01 02:26:05 +0000 UTC,LastTransitionTime:2023-01-01 02:20:27 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-01-01 02:26:05 +0000 UTC,LastTransitionTime:2023-01-01 02:20:48 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.45.112,},NodeAddress{Type:ExternalIP,Address:13.38.32.14,},NodeAddress{Type:Hostname,Address:ip-172-20-45-112.eu-west-3.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-45-112.eu-west-3.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-13-38-32-14.eu-west-3.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2dbdf846e8ff614f4555e5bf9dbbf4,SystemUUID:ec2dbdf8-46e8-ff61-4f45-55e5bf9dbbf4,BootID:10a054ae-621a-47ce-9566-b0048cc5101b,KernelVersion:4.19.0-23-cloud-amd64,OSImage:Debian GNU/Linux 10 (buster),ContainerRuntimeVersion:docker://20.10.17,KubeletVersion:v1.23.15,KubeProxyVersion:v1.23.15,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:456569528,},ContainerImage{Names:[nginx@sha256:0047b729188a15da49380d9506d65959cce6d40291ccfb4e039f5dc7efd33286 nginx:latest],SizeBytes:141812353,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:7e8bdd271312fd25fc5ff5a8f04727be84044eb3d7d8d03611972a6752e2e11e k8s.gcr.io/e2e-test-images/agnhost:2.39],SizeBytes:126872991,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:1b9d1b2f36cb2dbee1960e82a9344aeb11bd4c4c03abf5e1853e0559c23855e3 k8s.gcr.io/e2e-test-images/httpd:2.4.38-2],SizeBytes:123781643,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.23.15 registry.k8s.io/kube-proxy-amd64:v1.23.15],SizeBytes:112336036,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:10e231cf07c82dfab5141c1afe548dc734e5fc1f67665ec3982a325d1bd31a9a registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.0],SizeBytes:94471559,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:c8e03f60afa90a28e4bb6ec9a8d0fc36d89de4b7475cf2d613afa793ec969fe0 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.0],SizeBytes:56382329,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:4e74c0492bceddc598de1c90cc5bc14dcda94cb49fa9c5bad9d117c4834b5e08 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.1],SizeBytes:56379269,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:36c31f7e1f433c9634d24f876353e8646246d81a03c4e351202c2644daff1620 k8s.gcr.io/sig-storage/csi-resizer:v1.2.0],SizeBytes:53956308,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:ed98431376c9e944e19a465fe8ea944806714dd95416a0821096c78d66b579bd k8s.gcr.io/sig-storage/csi-snapshotter:v4.1.1],SizeBytes:53531990,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:c5be65d6679efabb969d9b019300d187437ae876f992c40911fd2892bbef3b36 k8s.gcr.io/sig-storage/csi-attacher:v3.2.0],SizeBytes:53500826,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:60ab9b3e6a030d3038c87c0d6bca2930f58d1d72823e6a4af09767dc83b696a2 k8s.gcr.io/sig-storage/csi-attacher:v3.2.1],SizeBytes:53499570,},ContainerImage{Names:[k8s.gcr.io/sig-storage/hostpathplugin@sha256:232fe80174d60d520d36043103853a1d7ab4b7f3782cf43e45034f04ccda58ce k8s.gcr.io/sig-storage/hostpathplugin:v1.7.1],SizeBytes:29977921,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:a61d309da54641db41fb8f35718f744e9f730d4d0384f8c4b186ddc9f06cbd5f k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.1.0],SizeBytes:19662887,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:19582112,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:2dee3fe5fe861bb66c3a4ac51114f3447a4cd35870e0f2e2b558c7a400d89589 k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.2.0],SizeBytes:18699436,},ContainerImage{Names:[k8s.gcr.io/sig-storage/mock-driver@sha256:a7b517f9e0f42ffade802eef9cefa271372386b85f55b702b493241e58459793 k8s.gcr.io/sig-storage/mock-driver:v4.1.0],SizeBytes:17680993,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:17611980,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:529be2c9770add0cdd0c989115222ea9fc1be430c11095eb9f6dafcf98a36e2b k8s.gcr.io/sig-storage/livenessprobe:v2.4.0],SizeBytes:17175488,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:13616070e3f29de4417eee434a8ef472221c9e51b3d037b5a6b46cef08eb7443 k8s.gcr.io/e2e-test-images/nginx:1.14-2],SizeBytes:16032814,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf k8s.gcr.io/e2e-test-images/busybox:1.29-2],SizeBytes:1154361,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:39e1e963e5310e9c313bad51523be012ede7b35bb9316517d19089a010356592 k8s.gcr.io/e2e-test-images/busybox:1.29-1],SizeBytes:1154361,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db k8s.gcr.io/pause:3.6 registry.k8s.io/pause:3.6],SizeBytes:682696,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Jan 1 02:30:39.591: INFO: Logging kubelet events for node ip-172-20-45-112.eu-west-3.compute.internal Jan 1 02:30:39.696: INFO: Logging pods the kubelet thinks is on node ip-172-20-45-112.eu-west-3.compute.internal Jan 1 02:30:39.809: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-srf5k started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:30:39.809: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:30:39.809: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-4mml9 started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:30:39.809: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:30:39.809: INFO: pod-exec-websocket-4b20549e-25f0-4f45-bfef-409447a30721 started at 2023-01-01 02:25:37 +0000 UTC (0+1 container statuses recorded) Jan 1 02:30:39.809: INFO: Container main ready: false, restart count 0 Jan 1 02:30:39.809: INFO: affinity-clusterip-timeout-lv5vx started at 2023-01-01 02:27:32 +0000 UTC (0+1 container statuses recorded) Jan 1 02:30:39.809: INFO: Container affinity-clusterip-timeout ready: false, restart count 0 Jan 1 02:30:39.809: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-xnfzj started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:30:39.809: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:30:39.809: INFO: test-recreate-deployment-594f666cd9-5pbpl started at 2023-01-01 02:27:48 +0000 UTC (0+1 container statuses recorded) Jan 1 02:30:39.809: INFO: Container agnhost ready: false, restart count 0 Jan 1 02:30:39.809: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-x5cp6 started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:30:39.809: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:30:39.809: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-gph2d started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:30:39.809: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:30:39.809: INFO: pod-39bf2777-905c-4d8b-97f4-bf3626087647 started at 2023-01-01 02:25:44 +0000 UTC (0+1 container statuses recorded) Jan 1 02:30:39.809: INFO: Container write-pod ready: false, restart count 0 Jan 1 02:30:39.809: INFO: netserver-1 started at 2023-01-01 02:25:27 +0000 UTC (0+1 container statuses recorded) Jan 1 02:30:39.810: INFO: Container webserver ready: true, restart count 0 Jan 1 02:30:39.810: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-rmdrr started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:30:39.810: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:30:39.810: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-kjg2m started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:30:39.810: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:30:39.810: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-59bdj started at 2023-01-01 02:25:17 +0000 UTC (0+1 container statuses recorded) Jan 1 02:30:39.810: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:30:39.810: INFO: ebs-csi-node-7clrr started at 2023-01-01 02:20:28 +0000 UTC (0+3 container statuses recorded) Jan 1 02:30:39.810: INFO: Container ebs-plugin ready: true, restart count 0 Jan 1 02:30:39.810: INFO: Container liveness-probe ready: true, restart count 0 Jan 1 02:30:39.810: INFO: Container node-driver-registrar ready: true, restart count 0 Jan 1 02:30:39.810: INFO: netserver-1 started at 2023-01-01 02:25:33 +0000 UTC (0+1 container statuses recorded) Jan 1 02:30:39.810: INFO: Container webserver ready: true, restart count 0 Jan 1 02:30:39.810: INFO: security-context-2da5e8a9-2026-481a-8c39-7b479c31bda0 started at 2023-01-01 02:25:47 +0000 UTC (0+1 container statuses recorded) Jan 1 02:30:39.810: INFO: Container test-container ready: false, restart count 0 Jan 1 02:30:39.810: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-jh8bc started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:30:39.810: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:30:39.810: INFO: hostexec-ip-172-20-45-112.eu-west-3.compute.internal-22hvx started at 2023-01-01 02:25:34 +0000 UTC (0+1 container statuses recorded) Jan 1 02:30:39.810: INFO: Container agnhost-container ready: true, restart count 0 Jan 1 02:30:39.810: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-4lrj2 started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:30:39.810: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:30:39.810: INFO: cilium-slx98 started at 2023-01-01 02:20:28 +0000 UTC (1+1 container statuses recorded) Jan 1 02:30:39.810: INFO: Init container clean-cilium-state ready: true, restart count 0 Jan 1 02:30:39.810: INFO: Container cilium-agent ready: true, restart count 0 Jan 1 02:30:40.183: INFO: Latency metrics for node ip-172-20-45-112.eu-west-3.compute.internal Jan 1 02:30:40.184: INFO: Logging node info for node ip-172-20-47-178.eu-west-3.compute.internal Jan 1 02:30:40.287: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-47-178.eu-west-3.compute.internal 835e5337-d2b9-45ad-bf0e-9d32514f71ba 6172 0 2023-01-01 02:20:25 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:eu-west-3 failure-domain.beta.kubernetes.io/zone:eu-west-3a kubelet_cleanup:true kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-47-178.eu-west-3.compute.internal kubernetes.io/os:linux kubernetes.io/role:node node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:eu-west-3a topology.kubernetes.io/region:eu-west-3 topology.kubernetes.io/zone:eu-west-3a] map[csi.volume.kubernetes.io/nodeid:{"csi-mock-csi-mock-volumes-5429":"csi-mock-csi-mock-volumes-5429","ebs.csi.aws.com":"i-00e1e60199f5a4d04"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kops-controller Update v1 2023-01-01 02:20:25 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}} } {kube-controller-manager Update v1 2023-01-01 02:20:25 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.2.0/24\"":{}}}} } {kubelet Update v1 2023-01-01 02:20:25 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {e2e.test Update v1 2023-01-01 02:25:15 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubelet_cleanup":{}}}} } {kubelet Update v1 2023-01-01 02:25:42 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:capacity":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.2.0/24,DoNotUseExternalID:,ProviderID:aws:///eu-west-3a/i-00e1e60199f5a4d04,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.2.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{50478186496 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4078190592 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{45430367772 0} {<nil>} 45430367772 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3973332992 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-01-01 02:25:42 +0000 UTC,LastTransitionTime:2023-01-01 02:20:25 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-01-01 02:25:42 +0000 UTC,LastTransitionTime:2023-01-01 02:20:25 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-01-01 02:25:42 +0000 UTC,LastTransitionTime:2023-01-01 02:20:25 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-01-01 02:25:42 +0000 UTC,LastTransitionTime:2023-01-01 02:20:46 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.47.178,},NodeAddress{Type:ExternalIP,Address:13.36.177.101,},NodeAddress{Type:Hostname,Address:ip-172-20-47-178.eu-west-3.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-47-178.eu-west-3.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-13-36-177-101.eu-west-3.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec243eda226c96d5573f3eaa3203df0d,SystemUUID:ec243eda-226c-96d5-573f-3eaa3203df0d,BootID:15dccaa7-d7c7-419d-b7de-76be5781a51a,KernelVersion:4.19.0-23-cloud-amd64,OSImage:Debian GNU/Linux 10 (buster),ContainerRuntimeVersion:docker://20.10.17,KubeletVersion:v1.23.15,KubeProxyVersion:v1.23.15,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:456569528,},ContainerImage{Names:[nginx@sha256:0047b729188a15da49380d9506d65959cce6d40291ccfb4e039f5dc7efd33286 nginx:latest],SizeBytes:141812353,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:7e8bdd271312fd25fc5ff5a8f04727be84044eb3d7d8d03611972a6752e2e11e k8s.gcr.io/e2e-test-images/agnhost:2.39],SizeBytes:126872991,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.23.15 registry.k8s.io/kube-proxy-amd64:v1.23.15],SizeBytes:112336036,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:10e231cf07c82dfab5141c1afe548dc734e5fc1f67665ec3982a325d1bd31a9a registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.0],SizeBytes:94471559,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:c8e03f60afa90a28e4bb6ec9a8d0fc36d89de4b7475cf2d613afa793ec969fe0 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.0],SizeBytes:56382329,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:c5be65d6679efabb969d9b019300d187437ae876f992c40911fd2892bbef3b36 k8s.gcr.io/sig-storage/csi-attacher:v3.2.0],SizeBytes:53500826,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:017727efcfeb7d053af68e51436ce8e65edbc6ca573720afb4f79c8594036955 registry.k8s.io/coredns/coredns:v1.10.0],SizeBytes:50249337,},ContainerImage{Names:[registry.k8s.io/cpa/cluster-proportional-autoscaler@sha256:68d396900aeaa072c1f27289485fdac29834045a6f3ffe369bf389d830ef572d registry.k8s.io/cpa/cluster-proportional-autoscaler:1.8.6],SizeBytes:47910609,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:a61d309da54641db41fb8f35718f744e9f730d4d0384f8c4b186ddc9f06cbd5f k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.1.0],SizeBytes:19662887,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:19582112,},ContainerImage{Names:[k8s.gcr.io/sig-storage/mock-driver@sha256:a7b517f9e0f42ffade802eef9cefa271372386b85f55b702b493241e58459793 k8s.gcr.io/sig-storage/mock-driver:v4.1.0],SizeBytes:17680993,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:17611980,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:13616070e3f29de4417eee434a8ef472221c9e51b3d037b5a6b46cef08eb7443 k8s.gcr.io/e2e-test-images/nginx:1.14-2],SizeBytes:16032814,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf k8s.gcr.io/e2e-test-images/busybox:1.29-2],SizeBytes:1154361,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db k8s.gcr.io/pause:3.6 registry.k8s.io/pause:3.6],SizeBytes:682696,},},VolumesInUse:[kubernetes.io/csi/csi-mock-csi-mock-volumes-5429^4],VolumesAttached:[]AttachedVolume{},Config:nil,},} Jan 1 02:30:40.287: INFO: Logging kubelet events for node ip-172-20-47-178.eu-west-3.compute.internal Jan 1 02:30:40.395: INFO: Logging pods the kubelet thinks is on node ip-172-20-47-178.eu-west-3.compute.internal Jan 1 02:30:40.511: INFO: hostexec-ip-172-20-47-178.eu-west-3.compute.internal-qnn5v started at 2023-01-01 02:25:18 +0000 UTC (0+1 container statuses recorded) Jan 1 02:30:40.511: INFO: Container agnhost-container ready: true, restart count 0 Jan 1 02:30:40.511: INFO: cilium-kk2c2 started at 2023-01-01 02:20:26 +0000 UTC (1+1 container statuses recorded) Jan 1 02:30:40.511: INFO: Init container clean-cilium-state ready: true, restart count 0 Jan 1 02:30:40.511: INFO: Container cilium-agent ready: true, restart count 0 Jan 1 02:30:40.511: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-6vhtn started at 2023-01-01 02:25:17 +0000 UTC (0+1 container statuses recorded) Jan 1 02:30:40.511: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: false, restart count 0 Jan 1 02:30:40.511: INFO: coredns-867df8f45c-wzntb started at 2023-01-01 02:20:46 +0000 UTC (0+1 container statuses recorded) Jan 1 02:30:40.511: INFO: Container coredns ready: true, restart count 0 Jan 1 02:30:40.511: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-wkclp started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:30:40.511: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:30:40.511: INFO: netserver-2 started at 2023-01-01 02:25:27 +0000 UTC (0+1 container statuses recorded) Jan 1 02:30:40.511: INFO: Container webserver ready: false, restart count 0 Jan 1 02:30:40.511: INFO: coredns-autoscaler-557ccb4c66-k4lzb started at 2023-01-01 02:20:46 +0000 UTC (0+1 container statuses recorded) Jan 1 02:30:40.511: INFO: Container autoscaler ready: true, restart count 0 Jan 1 02:30:40.511: INFO: csi-mockplugin-attacher-0 started at 2023-01-01 02:28:39 +0000 UTC (0+1 container statuses recorded) Jan 1 02:30:40.511: INFO: Container csi-attacher ready: false, restart count 0 Jan 1 02:30:40.511: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-pgkm2 started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:30:40.511: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:30:40.511: INFO: hostexec-ip-172-20-47-178.eu-west-3.compute.internal-nf87h started at 2023-01-01 02:25:17 +0000 UTC (0+1 container statuses recorded) Jan 1 02:30:40.511: INFO: Container agnhost-container ready: true, restart count 0 Jan 1 02:30:40.511: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-hrgrq started at 2023-01-01 02:25:17 +0000 UTC (0+1 container statuses recorded) Jan 1 02:30:40.511: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:30:40.511: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-grrsl started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:30:40.511: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:30:40.511: INFO: exec-volume-test-preprovisionedpv-w6ww started at 2023-01-01 02:25:36 +0000 UTC (0+1 container statuses recorded) Jan 1 02:30:40.511: INFO: Container exec-container-preprovisionedpv-w6ww ready: false, restart count 0 Jan 1 02:30:40.511: INFO: csi-mockplugin-0 started at 2023-01-01 02:28:39 +0000 UTC (0+3 container statuses recorded) Jan 1 02:30:40.511: INFO: Container csi-provisioner ready: false, restart count 0 Jan 1 02:30:40.511: INFO: Container driver-registrar ready: false, restart count 0 Jan 1 02:30:40.511: INFO: Container mock ready: false, restart count 0 Jan 1 02:30:40.511: INFO: ebs-csi-node-vcprc started at 2023-01-01 02:20:26 +0000 UTC (0+3 container statuses recorded) Jan 1 02:30:40.511: INFO: Container ebs-plugin ready: true, restart count 0 Jan 1 02:30:40.511: INFO: Container liveness-probe ready: true, restart count 0 Jan 1 02:30:40.511: INFO: Container node-driver-registrar ready: true, restart count 0 Jan 1 02:30:40.511: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-49brr started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:30:40.511: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:30:40.511: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-z87gr started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:30:40.511: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:30:40.511: INFO: csi-mockplugin-attacher-0 started at 2023-01-01 02:25:20 +0000 UTC (0+1 container statuses recorded) Jan 1 02:30:40.511: INFO: Container csi-attacher ready: false, restart count 0 Jan 1 02:30:40.511: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-wjz96 started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:30:40.511: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:30:40.511: INFO: csi-mockplugin-0 started at 2023-01-01 02:25:20 +0000 UTC (0+3 container statuses recorded) Jan 1 02:30:40.511: INFO: Container csi-provisioner ready: true, restart count 0 Jan 1 02:30:40.511: INFO: Container driver-registrar ready: true, restart count 0 Jan 1 02:30:40.511: INFO: Container mock ready: true, restart count 0 Jan 1 02:30:40.511: INFO: pod-subpath-test-preprovisionedpv-jphj started at 2023-01-01 02:25:52 +0000 UTC (1+1 container statuses recorded) Jan 1 02:30:40.511: INFO: Init container init-volume-preprovisionedpv-jphj ready: false, restart count 0 Jan 1 02:30:40.511: INFO: Container test-container-subpath-preprovisionedpv-jphj ready: false, restart count 0 Jan 1 02:30:40.511: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-w26gj started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:30:40.511: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:30:40.511: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-s4rqf started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:30:40.511: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:30:40.511: INFO: netserver-2 started at 2023-01-01 02:25:33 +0000 UTC (0+1 container statuses recorded) Jan 1 02:30:40.511: INFO: Container webserver ready: false, restart count 0 Jan 1 02:30:40.511: INFO: pvc-volume-tester-w4w4d started at 2023-01-01 02:25:38 +0000 UTC (0+1 container statuses recorded) Jan 1 02:30:40.511: INFO: Container volume-tester ready: false, restart count 0 Jan 1 02:30:40.931: INFO: Latency metrics for node ip-172-20-47-178.eu-west-3.compute.internal Jan 1 02:30:40.931: INFO: Logging node info for node ip-172-20-49-224.eu-west-3.compute.internal Jan 1 02:30:41.035: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-49-224.eu-west-3.compute.internal d7d57146-c54a-4155-94f8-0f322216bfcf 6403 0 2023-01-01 02:20:28 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:eu-west-3 failure-domain.beta.kubernetes.io/zone:eu-west-3a kubelet_cleanup:true kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-49-224.eu-west-3.compute.internal kubernetes.io/os:linux kubernetes.io/role:node node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:eu-west-3a topology.hostpath.csi/node:ip-172-20-49-224.eu-west-3.compute.internal topology.kubernetes.io/region:eu-west-3 topology.kubernetes.io/zone:eu-west-3a] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-089a107939b27c28a"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kops-controller Update v1 2023-01-01 02:20:28 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}} } {kube-controller-manager Update v1 2023-01-01 02:20:28 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.4.0/24\"":{}}}} } {kubelet Update v1 2023-01-01 02:20:28 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {e2e.test Update v1 2023-01-01 02:25:15 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubelet_cleanup":{}}}} } {kubelet Update v1 2023-01-01 02:25:35 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:capacity":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.4.0/24,DoNotUseExternalID:,ProviderID:aws:///eu-west-3a/i-089a107939b27c28a,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.4.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{50478186496 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4078206976 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{45430367772 0} {<nil>} 45430367772 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3973349376 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-01-01 02:25:35 +0000 UTC,LastTransitionTime:2023-01-01 02:20:28 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-01-01 02:25:35 +0000 UTC,LastTransitionTime:2023-01-01 02:20:28 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-01-01 02:25:35 +0000 UTC,LastTransitionTime:2023-01-01 02:20:28 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-01-01 02:25:35 +0000 UTC,LastTransitionTime:2023-01-01 02:20:48 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.49.224,},NodeAddress{Type:ExternalIP,Address:15.237.95.101,},NodeAddress{Type:Hostname,Address:ip-172-20-49-224.eu-west-3.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-49-224.eu-west-3.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-15-237-95-101.eu-west-3.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2d0a1bb33dd26f5a12c3896530131b,SystemUUID:ec2d0a1b-b33d-d26f-5a12-c3896530131b,BootID:9f64e91d-26f0-45aa-aec4-a54433e67034,KernelVersion:4.19.0-23-cloud-amd64,OSImage:Debian GNU/Linux 10 (buster),ContainerRuntimeVersion:docker://20.10.17,KubeletVersion:v1.23.15,KubeProxyVersion:v1.23.15,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:456569528,},ContainerImage{Names:[nginx@sha256:0047b729188a15da49380d9506d65959cce6d40291ccfb4e039f5dc7efd33286 nginx:latest],SizeBytes:141812353,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:7e8bdd271312fd25fc5ff5a8f04727be84044eb3d7d8d03611972a6752e2e11e k8s.gcr.io/e2e-test-images/agnhost:2.39],SizeBytes:126872991,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:1b9d1b2f36cb2dbee1960e82a9344aeb11bd4c4c03abf5e1853e0559c23855e3 k8s.gcr.io/e2e-test-images/httpd:2.4.38-2],SizeBytes:123781643,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.23.15 registry.k8s.io/kube-proxy-amd64:v1.23.15],SizeBytes:112336036,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:10e231cf07c82dfab5141c1afe548dc734e5fc1f67665ec3982a325d1bd31a9a registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.0],SizeBytes:94471559,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:4e74c0492bceddc598de1c90cc5bc14dcda94cb49fa9c5bad9d117c4834b5e08 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.1],SizeBytes:56379269,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:36c31f7e1f433c9634d24f876353e8646246d81a03c4e351202c2644daff1620 k8s.gcr.io/sig-storage/csi-resizer:v1.2.0],SizeBytes:53956308,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:ed98431376c9e944e19a465fe8ea944806714dd95416a0821096c78d66b579bd k8s.gcr.io/sig-storage/csi-snapshotter:v4.1.1],SizeBytes:53531990,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:60ab9b3e6a030d3038c87c0d6bca2930f58d1d72823e6a4af09767dc83b696a2 k8s.gcr.io/sig-storage/csi-attacher:v3.2.1],SizeBytes:53499570,},ContainerImage{Names:[k8s.gcr.io/sig-storage/hostpathplugin@sha256:232fe80174d60d520d36043103853a1d7ab4b7f3782cf43e45034f04ccda58ce k8s.gcr.io/sig-storage/hostpathplugin:v1.7.1],SizeBytes:29977921,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:19582112,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:2dee3fe5fe861bb66c3a4ac51114f3447a4cd35870e0f2e2b558c7a400d89589 k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.2.0],SizeBytes:18699436,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:17611980,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:529be2c9770add0cdd0c989115222ea9fc1be430c11095eb9f6dafcf98a36e2b k8s.gcr.io/sig-storage/livenessprobe:v2.4.0],SizeBytes:17175488,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:13616070e3f29de4417eee434a8ef472221c9e51b3d037b5a6b46cef08eb7443 k8s.gcr.io/e2e-test-images/nginx:1.14-2],SizeBytes:16032814,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf k8s.gcr.io/e2e-test-images/busybox:1.29-2],SizeBytes:1154361,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db k8s.gcr.io/pause:3.6 registry.k8s.io/pause:3.6],SizeBytes:682696,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Jan 1 02:30:41.035: INFO: Logging kubelet events for node ip-172-20-49-224.eu-west-3.compute.internal Jan 1 02:30:41.141: INFO: Logging pods the kubelet thinks is on node ip-172-20-49-224.eu-west-3.compute.internal Jan 1 02:30:41.253: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-pgc9g started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:30:41.253: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:30:41.253: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-29cdj started at 2023-01-01 02:25:17 +0000 UTC (0+1 container statuses recorded) Jan 1 02:30:41.253: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: false, restart count 0 Jan 1 02:30:41.253: INFO: echo-sourceip started at 2023-01-01 02:26:27 +0000 UTC (0+1 container statuses recorded) Jan 1 02:30:41.253: INFO: Container agnhost-container ready: false, restart count 0 Jan 1 02:30:41.253: INFO: pod-service-account-nomountsa-mountspec started at 2023-01-01 02:29:39 +0000 UTC (0+1 container statuses recorded) Jan 1 02:30:41.253: INFO: Container token-test ready: false, restart count 0 Jan 1 02:30:41.253: INFO: e2e-configmap-dns-server-7551ef4f-8c37-4168-bf54-91644ec7fb8f started at 2023-01-01 02:29:40 +0000 UTC (0+1 container statuses recorded) Jan 1 02:30:41.253: INFO: Container agnhost-container ready: false, restart count 0 Jan 1 02:30:41.253: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-7z6bp started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:30:41.253: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:30:41.253: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-lrzjq started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:30:41.253: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:30:41.253: INFO: netserver-3 started at 2023-01-01 02:25:27 +0000 UTC (0+1 container statuses recorded) Jan 1 02:30:41.253: INFO: Container webserver ready: true, restart count 0 Jan 1 02:30:41.253: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-4jf6d started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:30:41.253: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:30:41.253: INFO: cilium-78jll started at 2023-01-01 02:20:29 +0000 UTC (1+1 container statuses recorded) Jan 1 02:30:41.253: INFO: Init container clean-cilium-state ready: true, restart count 0 Jan 1 02:30:41.253: INFO: Container cilium-agent ready: true, restart count 0 Jan 1 02:30:41.253: INFO: ebs-csi-node-tzntf started at 2023-01-01 02:20:29 +0000 UTC (0+3 container statuses recorded) Jan 1 02:30:41.253: INFO: Container ebs-plugin ready: true, restart count 0 Jan 1 02:30:41.253: INFO: Container liveness-probe ready: true, restart count 0 Jan 1 02:30:41.253: INFO: Container node-driver-registrar ready: true, restart count 0 Jan 1 02:30:41.253: INFO: test-webserver-a9c3c1dd-2bb7-4667-876b-bad20aff393e started at 2023-01-01 02:25:41 +0000 UTC (0+1 container statuses recorded) Jan 1 02:30:41.253: INFO: Container test-webserver ready: false, restart count 0 Jan 1 02:30:41.253: INFO: hostexec-ip-172-20-49-224.eu-west-3.compute.internal-zzhhr started at 2023-01-01 02:25:53 +0000 UTC (0+1 container statuses recorded) Jan 1 02:30:41.253: INFO: Container agnhost-container ready: true, restart count 0 Jan 1 02:30:41.253: INFO: affinity-clusterip-timeout-pxjtk started at 2023-01-01 02:27:32 +0000 UTC (0+1 container statuses recorded) Jan 1 02:30:41.253: INFO: Container affinity-clusterip-timeout ready: true, restart count 0 Jan 1 02:30:41.253: INFO: annotationupdate3b35e9ec-7ad7-4bd9-8137-1bd2bf1a899a started at 2023-01-01 02:28:25 +0000 UTC (0+1 container statuses recorded) Jan 1 02:30:41.253: INFO: Container client-container ready: false, restart count 0 Jan 1 02:30:41.253: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-tv984 started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:30:41.253: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:30:41.253: INFO: netserver-3 started at 2023-01-01 02:25:34 +0000 UTC (0+1 container statuses recorded) Jan 1 02:30:41.253: INFO: Container webserver ready: true, restart count 0 Jan 1 02:30:41.253: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-cz8qh started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:30:41.253: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:30:41.253: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-2gndb started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:30:41.253: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:30:41.253: INFO: httpd started at 2023-01-01 02:26:24 +0000 UTC (0+1 container statuses recorded) Jan 1 02:30:41.253: INFO: Container httpd ready: false, restart count 0 Jan 1 02:30:41.253: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-h6dxb started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:30:41.253: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:30:41.253: INFO: cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2-6n5j9 started at 2023-01-01 02:25:16 +0000 UTC (0+1 container statuses recorded) Jan 1 02:30:41.253: INFO: Container cleanup40-6f45abc9-e32d-424f-b2c3-d74d506011d2 ready: true, restart count 0 Jan 1 02:30:41.253: INFO: local-injector started at 2023-01-01 02:26:06 +0000 UTC (0+1 container statuses recorded) Jan 1 02:30:41.253: INFO: Container local-injector ready: false, restart count 0 Jan 1 02:30:41.253: INFO: test-ss-0 started at 2023-01-01 02:27:56 +0000 UTC (0+1 container statuses recorded) Jan 1 02:30:41.254: INFO: Container webserver ready: false, restart count 0 Jan 1 02:30:41.940: INFO: Latency metrics for node ip-172-20-49-224.eu-west-3.compute.internal Jan 1 02:30:41.941: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready �[1mSTEP�[0m: Destroying namespace "pods-4895" for this suite.
Filter through log files
go run hack/e2e.go -v --test --test_args='--ginkgo.focus=Kubernetes\se2e\ssuite\s\[sig\-node\]\sProbing\scontainer\sshould\s\*not\*\sbe\srestarted\swith\sa\s\/healthz\shttp\sliveness\sprobe\s\[NodeConformance\]\s\[Conformance\]$'
/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:633 Jan 1 02:30:41.749: starting pod test-webserver-a9c3c1dd-2bb7-4667-876b-bad20aff393e in namespace container-probe-6960 Unexpected error: <*errors.errorString | 0xc0002c22c0>: { s: "timed out waiting for the condition", } timed out waiting for the condition occurred /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/common/node/container_probe.go:904
{"msg":"PASSED [sig-node] Probing container with readiness probe should not be ready before initial delay and never restart [NodeConformance] [Conformance]","total":-1,"completed":2,"skipped":21,"failed":0} [BeforeEach] [sig-node] Probing container /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:185 �[1mSTEP�[0m: Creating a kubernetes client Jan 1 02:25:40.602: INFO: >>> kubeConfig: /root/.kube/config �[1mSTEP�[0m: Building a namespace api object, basename container-probe �[1mSTEP�[0m: Waiting for a default service account to be provisioned in namespace �[1mSTEP�[0m: Waiting for kube-root-ca.crt to be provisioned in namespace [BeforeEach] [sig-node] Probing container /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/common/node/container_probe.go:59 [It] should *not* be restarted with a /healthz http liveness probe [NodeConformance] [Conformance] /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:633 �[1mSTEP�[0m: Creating pod test-webserver-a9c3c1dd-2bb7-4667-876b-bad20aff393e in namespace container-probe-6960 Jan 1 02:30:41.749: FAIL: starting pod test-webserver-a9c3c1dd-2bb7-4667-876b-bad20aff393e in namespace container-probe-6960 Unexpected error: <*errors.errorString | 0xc0002c22c0>: { s: "timed out waiting for the condition", } timed out waiting for the condition occurred Full Stack Trace k8s.io/kubernetes/test/e2e/common/node.RunLivenessTest(0xc00082f8c0, 0xc004be2400, 0x0, 0xe) /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/common/node/container_probe.go:904 +0x3b9 k8s.io/kubernetes/test/e2e/common/node.glob..func2.9() /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/common/node/container_probe.go:214 +0x118 k8s.io/kubernetes/test/e2e.RunE2ETests(0x24c66d7) _output/dockerized/go/src/k8s.io/kubernetes/test/e2e/e2e.go:133 +0x697 k8s.io/kubernetes/test/e2e.TestE2E(0x243a8f9) _output/dockerized/go/src/k8s.io/kubernetes/test/e2e/e2e_test.go:136 +0x19 testing.tRunner(0xc000a0c1a0, 0x735e880) /usr/local/go/src/testing/testing.go:1259 +0x102 created by testing.(*T).Run /usr/local/go/src/testing/testing.go:1306 +0x35a �[1mSTEP�[0m: deleting the pod [AfterEach] [sig-node] Probing container /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:186 �[1mSTEP�[0m: Collecting events from namespace "container-probe-6960". �[1mSTEP�[0m: Found 12 events. Jan 1 02:30:41.966: INFO: At 2023-01-01 02:25:41 +0000 UTC - event for test-webserver-a9c3c1dd-2bb7-4667-876b-bad20aff393e: {default-scheduler } Scheduled: Successfully assigned container-probe-6960/test-webserver-a9c3c1dd-2bb7-4667-876b-bad20aff393e to ip-172-20-49-224.eu-west-3.compute.internal Jan 1 02:30:41.966: INFO: At 2023-01-01 02:25:43 +0000 UTC - event for test-webserver-a9c3c1dd-2bb7-4667-876b-bad20aff393e: {kubelet ip-172-20-49-224.eu-west-3.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "c515bab0d83482c359be8de00506f452027327d9961d45da8ed94cd00c548a5a" network for pod "test-webserver-a9c3c1dd-2bb7-4667-876b-bad20aff393e": networkPlugin cni failed to set up pod "test-webserver-a9c3c1dd-2bb7-4667-876b-bad20aff393e_container-probe-6960" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 1 02:30:41.966: INFO: At 2023-01-01 02:25:44 +0000 UTC - event for test-webserver-a9c3c1dd-2bb7-4667-876b-bad20aff393e: {kubelet ip-172-20-49-224.eu-west-3.compute.internal} SandboxChanged: Pod sandbox changed, it will be killed and re-created. Jan 1 02:30:41.966: INFO: At 2023-01-01 02:25:45 +0000 UTC - event for test-webserver-a9c3c1dd-2bb7-4667-876b-bad20aff393e: {kubelet ip-172-20-49-224.eu-west-3.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "5eb0f6fc72ee6fb387c713aa7f31781a7d1180f51784b92c99e63fab5bb67aae" network for pod "test-webserver-a9c3c1dd-2bb7-4667-876b-bad20aff393e